Medical Knowledge-Guided CLIP Adaptation for Fundus Image Diagnosis

作者：Shaolong Wang, Xinyu Zhao, Zhenquan Wu, Guoming Zhang, Tianfu Wang, Baiying Lei · 发表于：IEEE International Conference on Bioinformatics and Biomedicine · 年份：2025 · DOI：10.1109/BIBM66473.2025.11356048 · 研究领域：Computer Science

Fundus image classification plays a crucial role in diagnosing ophthalmic diseases but remains challenging due to the scarcity of annotated data and the subtlety of lesion features, which often resemble surrounding tissues. Vision-language models (VLMs), known for their impressive few-shot learning performance on natural images, offer a promising foundation for medical image analysis. However, directly applying these models to medical tasks is suboptimal, as they lack domain-specific knowledge and struggle to capture fine-grained pathology cues. To address these challenges, we propose a novel framework for few-shot fundus image classification that integrates medical knowledge-driven prompt learning into CLIP. Specifically, our method utilizes a domain-specific prompt bank constructed from clinical terminology to enrich the model's understanding of medical context. Additionally, we introduce a cross-modal alignment loss to improve consistency between visual and textual features and employ a lightweight adapter for efficient task-specific fine-tuning. Extensive experiments across multiple datasets demonstrate that our approach significantly enhances performance, surpassing existing methods in various few-shot scenarios.