MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

作者：Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer · 发表于：2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 年份：2022 · DOI：10.1109/cvpr52688.2022.00476 · 被引用次数：786 · 研究领域：Advanced Neural Network Applications、Domain Adaptation and Few-Shot Learning、Advanced Image and Video Retrieval Techniques

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 APboxon COCO object detection as well as 86.1% on Kinetics-400 video classification. Code and models are available at https://github.com/facebookresearch/mvit.