Spatio-Temporal Self-Supervision Enhanced Transformer Networks for Action Recognition

作者：Yongkang Zhang, Han Zhang, Guoming Wu, Jun Li · 发表于：2022 IEEE International Conference on Multimedia and Expo (ICME) · 年份：2022 · DOI：10.1109/icme52920.2022.9859741 · 被引用次数：5 · 研究领域：Human Pose and Action Recognition、Gait Recognition and Analysis、Multimodal Machine Learning Applications

With the development of deep neural networks, video action recognition has gradually become a research hotspot in recent years. However, the additional temporal dimension in video makes this task very challenging. In this paper, we propose a novel Spatio-Temporal Self-Supervision enhanced Transformer Networks (STTNet) for video action recognition, which mainly consists of Self-Supervised SpatioTemporal Representation Learning module and Transformer based Spatio-Temporal Aggregator module. Concretely, our STTNet can adaptively encode the spatial and temporal enhanced key features, which are respectively learned through the Temporal and Spatial Self-Supervised sub-module using the unlabeled video data, in a nonlinear and nonlocal manner via the Transformer based SpatioTemporal Aggrega-tor. The extensive experiments on three widely used datasets (HMDB51, UCF101 and Something-Something V1) demon-strate that our proposed STTNet can achieve the state-of-the-art performance. Code is available at https://github.com/ICME2022/STTNet.