STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition

作者：Dasom Ahn, Sangwon Kim, Hyunsu Hong, Byoung Chul Ko · 发表于：2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · 年份：2023 · DOI：10.1109/wacv56688.2023.00333 · 被引用次数：200 · 研究领域：Human Pose and Action Recognition、Gait Recognition and Analysis、Diabetic Foot Ulcer Assessment and Management

In action recognition, although the combination of spatiotemporal videos and skeleton features can improve the recognition performance, a separate model and balancing feature representation for cross-modal data are required. To solve these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can effectively represent two cross-modal features as a recognizable vector. First, from the input video and skeleton sequence, video frames are output as global grid tokens and skeletons are output as joint map tokens, respectively. These tokens are then aggregated into multi-class tokens and input into STAR-transformer. The STAR-transformer encoder consists of a full spatio-temporal attention (FAttn) module and a proposed zigzag spatio-temporal attention (ZAttn) module. Similarly, the continuous decoder consists of a FAttn module and a proposed binary spatio-temporal attention (BAttn) module. STAR-transformer learns an efficient multi-feature representation of the spatio-temporal features by properly arranging pairings of the FAttn, ZAttn, and BAttn modules. Experimental results on the Penn-Action, NTU-RGB+D 60, and 120 datasets show that the proposed method achieves a promising improvement in performance in comparison to previous state-of-the-art methods.