Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models

作者：Shengqu Cai, Duygu Ceylan, Matheus Gadelha, C. Huang, Tuanfeng Y. Wang, Gordon Wetzstein · 发表于：Computer Vision and Pattern Recognition · 年份：2023 · DOI：10.1109/CVPR52733.2024.00727 · 被引用次数：33 · 研究领域：Computer Science

Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth cor-respondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path. Project page: primecai.github.io/ generative. rendering.