3D-aware Conditional Image Synthesis

作者：Kangle Deng, Gengshan Yang, D. Ramanan, Junchen Zhu · 发表于：Computer Vision and Pattern Recognition · 年份：2023 · DOI：10.1109/CVPR52729.2023.00431 · 被引用次数：38 · 研究领域：Computer Science

We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available posed monocular image and label map pairs, our model learns to assign a label to every 3D point in addition to color and density, which enables it to render the image and pixel-aligned label map simultaneously. Finally, we build an interactive system that allows users to edit the label map from different viewpoints and generate outputs accordingly.