ConceptBert: Concept-Aware Representation for Visual Question Answering

作者：François Gardères, Maryam Ziaeefard, B. Abeloos, Freddy Lécué · 年份：2020 · DOI：10.18653/v1/2020.findings-emnlp.44 · 被引用次数：147 · 研究领域：Multimodal Machine Learning Applications、Advanced Image and Video Retrieval Techniques、Domain Adaptation and Few-Shot Learning

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Current works in VQA focus on questions which are answerable by direct analysis of the question and image alone. We present a concept-aware algorithm, ConceptBert, for questions which require common sense, or basic factual knowledge from external structured content. Given an image and a question in natural language, ConceptBert requires visual elements of the image and a Knowledge Graph (KG) to infer the correct answer. We introduce a multi-modal representation which learns a joint Concept-Vision-Language embedding. We exploit ConceptNet KG for encoding the common sense knowledge and evaluate our methodology on the Outside Knowledge-VQA (OK-VQA) and VQA datasets. Our code is available at