MetaSegNet: Metadata-Collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images

作者：Libo Wang, Sijun Dong, Ying Chen, Xiaoliang Meng, Shenghui Fang, Songlin Fei · 发表于：IEEE Transactions on Geoscience and Remote Sensing · 年份：2024 · DOI：10.1109/tgrs.2024.3477548 · 被引用次数：22 · 研究领域：Advanced Image and Video Retrieval Techniques、Multimodal Machine Learning Applications、Image Retrieval and Classification Techniques

Semantic segmentation of remote sensing images plays a vital role in a wide range of Earth Observation applications, such as land-use land-cover (LULC) mapping, environment monitoring, and sustainable development. Driven by rapid developments in artificial intelligence, deep learning (DL) has emerged as the mainstream for semantic segmentation and has achieved many breakthroughs in the field of remote sensing. However, most DL-based methods focus on unimodal visual data while ignoring rich multimodal information involved in the real world. Nonvisual data, such as text, can gather extra knowledge from the real world, which can strengthen the interpretability, reliability, and generalization of visual models. Inspired by this, we propose a novel metadata-collaborative segmentation network (MetaSegNet) that applies vision-language representation learning for the semantic segmentation of remote sensing images. Unlike the common model structure that only uses unimodal visual data, we extract the key characteristic (e.g., the climate zone) from freely available remote sensing image metadata and transfer it into geographic text prompts via the generic ChatGPT. Then, we construct an image encoder, a text encoder, and a crossmodal attention fusion subnetwork to extract the image and text feature and apply image-text interaction. Benefiting from such a design, the proposed MetaSegNet not only demonstrates superior generalization in zero-shot testing but also achieves competitive accura...