KAID: Knowledge-Aware Interactive Distillation for Vision-Language Models

作者：Da Zhang, Feiyu Wang, Bingyu Li, Zhiyuan Zhao, Junyu Gao, Xuelong Li · 年份：2025 · DOI：10.1145/3746027.3755008 · 被引用次数：7 · 研究领域：Multimodal Machine Learning Applications、Advanced Image and Video Retrieval Techniques、Advanced Neural Network Applications

Vision-Language Models (VLMs) such as CLIP have demonstrated outstanding performance in cross-modal tasks, but the prohibitive computational cost hinders practical deployment. Although Knowledge Distillation (KD) provides a promising compression paradigm, most existing methods rely heavily on feature imitation and contrastive relations without explicit fine-grained alignment. Additionally, they do not fully leverage the multimodal interaction knowledge from the teacher model, restricting cross-modal semantic alignment. To address these challenges, we propose KAID, a Knowledge-Aware Interactive Distillation method for VLMs. Specifically, we first pretrain a large CLIP teacher model with domain few-shot labels and store text features as category vectors. Then, an Image Feature Matching (IFM) module is introduced to calculate the feature distribution of teacher-student models with improved cosine similarity, which achieves hierarchical knowledge transfer from global to local levels and enhances the fine-grained perception of student model through joint optimization. Moreover, a Pixel-Wise Alignment (PWA) module is constructed between the teacher's text features and the student's image features, employing a cross-modal attention mechanism to establish semantic associations, while a Text-guided Pixel alignment Loss function (TPloss) is concurrently designed to enhance the student's comprehension capabilities. Ultimately, the well-trained student model is used for inference. Extens...