Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

作者：Yan Zhang, Huai Yu, Yujie He, Xinya Wang, Wen Yang · 发表于：IEEE Transactions on Instrumentation and Measurement · 年份：2023 · DOI：10.1109/tim.2023.3251414 · 被引用次数：78 · 研究领域：Industrial Vision Systems and Defect Detection、Advanced Neural Network Applications、Advanced Image and Video Retrieval Techniques

Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this article, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed Triple-I Net (TINet) to adaptively fuse the complementary features extracted from RGB and thermal images. First, we design an illumination-guided feature weighting module to guide the network in learning toward a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.