Selective Structured State-Spaces for Long-Form Video Understanding

作者：Jue Wang, Wenjie Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, Raffay Hamid · 发表于：Computer Vision and Pattern Recognition · 年份：2023 · DOI：10.1109/cvpr52729.2023.00618 · 被引用次数：196 · 研究领域：Computer Science

Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence ($S4$) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all imagetokens equally as done by $S4$ model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective $S4$ (i.e., $S5)$ model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our $S5$ model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated $S4$ model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consi...