Multi-Modal Generative AI: Multi-Modal LLMs, Diffusions, and the Unification

作者：Xin Wang, Yuwei Zhou, Bin Huang, Hong Chen, Wenwu Zhu · 发表于：IEEE Transactions on Circuits and Systems for Video Technology · 年份：2025 · DOI：10.1109/tcsvt.2025.3635224 · 被引用次数：3 · 研究领域：Natural Language Processing Techniques、Topic Modeling

Multi-modal generative AI (Artificial Intelligence) has attracted increasing attention from both academia and industry. Particularly, two dominant families of techniques have emerged: i) Multi-modal large language models (LLMs) demonstrate impressive ability formulti-modal understanding; and ii) Diffusion models exhibit remarkable multi-modal powers in terms ofmulti-modal generation. Therefore, this paper provides a comprehensive overview of multi-modal generative AI, including multi-modal LLMs, diffusions, and the unification for understanding and generation. To lay a solid foundation for unified models, we first provide a detailed review of both multi-modal LLMs and diffusion models, respectively, including their probabilistic modeling procedure, multi-modal architecture design, and advanced applications to image/video LLMs as well as text-to-image/video generation. Furthermore, we explore the emerging efforts toward unified models for understanding and generation. To achieve the unification of understanding and generation, we investigate key designs including autoregressive-based and diffusion-based modeling, as well as dense and Mixture-of-Experts (MoE) architectures. We then introduce several strategies for unified models, analyzing their potential advantages and disadvantages. In addition, we summarize the common datasets widely used for multi-modal generative AI pretraining. Last but not least, we present several challenging future research directions that may contribu...