VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding

作者：Henghao Zhao, Ge-Peng Ji, Rui Yan, Huan Xiong, Zechao Li · 发表于：IEEE Transactions on Circuits and Systems for Video Technology · 年份：2026 · DOI：10.1109/tcsvt.2026.3653742 · 被引用次数：3 · 研究领域：Multimodal Machine Learning Applications、Generative Adversarial Networks and Image Synthesis、Topic Modeling

The core challenge in video understanding lies in perceiving dynamic content changes over time. However, multi-modal large language models (MLLMs) struggle with temporal-sensitive video tasks, such as video temporal grounding, which requires generating timestamps to mark the occurrence of specific events. Existing strategies require MLLMs to generate absolute or relative timestamps directly. We have observed that those MLLMs tend to rely more on language patterns than visual cues when generating timestamps, affecting their performance. To address this problem, we propose VideoExpert, a general-purpose MLLM suitable for several temporal-sensitive video tasks. Inspired by the expert concept, VideoExpert integrates two parallel modules: the Temporal Expert and the Spatial Expert. The Temporal Expert is responsible for modeling time sequences and performing temporal grounding. It processes high-frame-rate yet compressed tokens to capture dynamic variations in videos and includes a lightweight prediction head for precise event localization. The Spatial Expert focuses on content detail analysis and instruction following. It handles specially designed spatial tokens and language input, aiming to generate content-related responses. These two experts collaborate seamlessly via a special token, ensuring coordinated temporal grounding and content generation. Notably, the Temporal and Spatial Experts maintain independent parameter sets. This parameter decoupling design enables specialize...