Generalizable Prompt Learning via Gradient Constrained Sharpness-Aware Minimization

作者：Liangchen Liu, Nannan Wang, Dawei Zhou, Decheng Liu, Xi Yang, Xinbo Gao, Tongliang Liu · 发表于：IEEE Transactions on Multimedia · 年份：2024 · DOI：10.1109/tmm.2024.3521702 · 被引用次数：5 · 研究领域：Domain Adaptation and Few-Shot Learning、Neural Networks and Applications、Advanced Data Compression Techniques

This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM), i.e., improving the performance on unseen classes while maintaining the performance on seen classes. Comparing with existing generalizable methods that neglect the seen classes degradation, the setting of this problem is stricter and fits more closely with practical applications. To solve this problem, we start from the optimization perspective, and leverage the relationship between loss landscape geometry and model generalization ability. By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to bothloss valueandloss sharpness, while each of them is indispensable. However, we find the optimizing gradient of existing methods cannot maintain high relevance to both loss value and loss sharpness during optimization, which severely affects their trade-off performance. To this end, we propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp), to dynamically constrain the optimizing gradient, thus achieving above two-fold optimization objective simultaneously. Extensive experiments verify the effectiveness of GCSCoOp in the trade-off problem.