Development of Interpretable and Efficient Visual Question Answering Models

Abstract

Visual Question Answering (VQA) combines computer vision and natural language processing to enable machines to answer questions about images. This task, essential for applications like personalized education, medical diagnostics, and interactive gaming, requires both image understanding and language comprehension. Despite advances in deep learning, VQA models struggle with dataset biases, diverse question types, and limited annotations, often relying on shortcuts from training data that hinder generalization. This thesis aims to address these challenges, improving the efficiency and power of VQA models. Firstly, we introduce two VQA frameworks to reduce training time and computational requirements without compromising accuracy. The first technique employs automatic mixed precision, using 32-bit precision (FP32) for loss calculation and 16-bit precision (FP16) for gradient computation during backpropagation. The second technique involves question segregation to minimize inference time. By classifying questions and training on reduced, category-specific datasets, we optimize model performance and accuracy measurement using weighted averages based on dataset composition. Secondly, to develop efficient VQA models, we propose a novel model that integrates both local and global image features. Unlike existing models that often overlook critical foreground and background information, our approach uses an ensemble of attention methods including traditional co-attention, encoder-decoder attention, and spatial-channel attention to extract visual features. This integration aims to improve understanding by capturing comprehensive contextual information from images.