Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

作者：Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, Jun Zhu · 年份：2024 · DOI：10.1109/cvpr52733.2024.02508 · 被引用次数：14 · 研究领域：Multimodal Machine Learning Applications、Topic Modeling、Domain Adaptation and Few-Shot Learning

Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on down-stream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require indepen-dent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferabil-ity of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through ex-tensive experiments with 6 modern MLLMs on a wide vari-ety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction.