LightIt: Illumination Modeling and Control for Diffusion Models

作者：Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, Yannick Hold-Geoffroy · 年份：2024 · DOI：10.1109/cvpr52733.2024.00894 · 被引用次数：18 · 研究领域：Computer Graphics and Visualization Techniques、Advanced Vision and Imaging、Color Science and Applications

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artis-tic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limi-tations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving re-lighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.