Knowing What it is: Semantic-Enhanced Dual Attention Transformer

作者：Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, Feiyue Huang, Rongrong Ji · 发表于：IEEE Transactions on Multimedia · 年份：2022 · DOI：10.1109/tmm.2022.3164787 · 被引用次数：37 · 研究领域：Multimodal Machine Learning Applications、Domain Adaptation and Few-Shot Learning、Advanced Image and Video Retrieval Techniques

Attention has become an indispensable component of the models of various multimedia tasks likeImage Captioning(IC) andVisual Question Answering(VQA). However, most existing attention modules are designed for capturing the spatial dependency, and are still insufficient in semantic understanding,e.g., the categories of objects and their attributes, which is also critical for image captioning. To compensate for this defect, we propose a novel attention module termedChannel-wise Attention Block(CAB) to model channel-wise dependency for both visual modality and linguistic modality, thereby improving semantic learning and multi-modal reasoning simultaneously. Specifically, CAB has two novel designs to tackle with the high overhead of channel-wise attention, which are thereduction-reconstruction block structureand thegating-based attention prediction. Based on CAB, we further propose a novelSemantic-enhanced Dual Attention Transformer(termed SDATR), which combines the merits of spatial and channel-wise attentions. To validate SDATR, we conduct extensive experiments on the MS COCO dataset and yield new state-of-the-art performance of 134.5 CIDEr score on COCO Karpathy test split and 136.0 CIDEr score on the official online testing server. To examine the generalization of SDATR, we also apply it to the task of visual question answering, where superior performance gains are also witnessed. The code and models are publicly available athttps://github.com/xmu-xiaoma666/SDATR.