A System for Automatic Past Event Search using Visual Question Answering

Loading...
Thumbnail Image

Date

item.page.authors

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

newline From around the world every minute, hundreds of hours of video and huge amount of images are uploaded to the internet and social media sites. Within only a few years, social media internet or smartphone users ends up accumulating dozens of thousands of photos and many hours of video, that capture cherish able moments from one s past, such as weddings, family gatherings or birthday parties. This multimedia data creates a visual record of the experiences of a significant percentage of humanity and can help illuminate how we live in the present moment. When properly analyzed, these video and images can also help users to reconstruct events of interest, including weddings, family gatherings or birthday parties. Even the reconstruction of events of interest can be extended to identify war crimes, human rights violations, and terrorist acts. newlineThe proposed research work addressed various challenges. Photos and videos involve rich information. A user typically has sequences of videos or photos, ordered according to their time stamps. These photos or videos may have text annotations, tags or other metadata. A robust method is designed and developed to control irregularity of multimodal data, such as photos or videos not having annotations. The another challenge regards relevant videos to be displayed in output for justifications in addition to direct answer based on input data. The associated evidence for the answer must be identified to help users with a lot of photos and videos. The proposed system is designed such that it not only give a definite answer, but also relevant evidential photos or text snippets in the output sequence to validate the reasoning process. A substantial amount of time is needed to examine every photo or video or text questions and answers. The relevant photos or videos are required as an evidence to quickly verify the answer. newlineIn this research work long short-term memory (LSTM) based visual question answering model integrating with convolutional layer non-linearity. The proposed adaptive memory network with attention mechanism model is designed and evaluated on common objects in context dataset with weight initialization and regularization, and model optimization. The Event-based Focal Visual-Content Text Attention (EFVCTA) framework is proposed in this research work. The system is designed on the basis of the proposed EFVCTA for automatic past event search using visual question answering (VQA). The system can be used to identify the past event s information from the photos of the various training programmes, workshops, conferences, annual social gatherings, etc. conducted in the academic institutions. newline newline newline newline

Description

Keywords

Citation

item.page.endorsement

item.page.review

item.page.supplemented

item.page.referenced