This AI Lets You Move the Sun in Any Photo!

Key Concepts

• AI Relighting: A new technique to change the lighting in 2D photographs.
• De-lighting: Removing the original lighting from a 2D image.
• 3D-ifying: Converting a 2D image into a 3D scene.
• Neural Renderer: A neural network that converts a rough 3D rendering into a realistic image.
• Rough 3D Rendering: An initial, imperfect 3D reconstruction of a 2D image.
• Light Wiggling: Iteratively adjusting light sources in a 3D scene to match the lighting in the original 2D photo.

1. Introduction: The Problem with Relighting 2D Photos

• In 3D modeling programs like Blender, changing lighting is easy.
• In 2D photography, even with AI, moving light sources is difficult; edits are limited to contrast adjustments.
• The traditional constraint: "the light is permanent."

2. The New AI Relighting Technique: Overview

• A new research paper introduces a method to relight 2D photographs.
• The technique allows changing the time of day or adding new light sources to a 2D image.
• Example: Transforming a daytime photo into a nighttime scene with new light sources.

3. Step-by-Step Process

• Step 1: De-lighting: Remove the original lighting from the 2D photo using a previously developed technique.
• Step 2: 3D-ifying: Convert the de-lit 2D image into a 3D scene using existing methods.
  • Initial 3D scene is often rough, with holes and lacking detail.
• Step 3: Beautifying with Neural Rendering: Use a neural network to transform the rough 3D rendering into a realistic image.

4. Neural Renderer Training: The Key Contribution

• Training a neural renderer requires thousands of pairs of rough 3D renderings and corresponding real photos.
• The challenge: Creating accurate rough 3D renderings that match the lighting of the real photos.
• Solution: "Light Wiggling"
  • Place initial lights in the 3D scene and render it.
  • Compare the rendered result to the target photo.
  • Iteratively adjust the light positions and parameters until the 3D scene's lighting closely matches the photo.
  • Repeat this process for thousands of photos to train the neural network.

5. Capabilities and Applications

• The trained system can take a photo and change the lighting from night to daytime or vice versa.
• It supports adding and removing lights, including spotlights and area lights.
• Animated projectors can be used to project images onto the scene.
• The technique enables creative applications, such as placing oneself into various scenes with dynamic lighting.
• "And this is why it’s a bit like the Mona Lisa at the laser-lit party."

6. Performance and Speed

• The relighting process is fast, taking only about three seconds per photo.
• Two seconds for pre-processing and less than one second for relighting.

7. Limitations

• The results are not perfect; blocky artifacts can appear as the lighting changes.
• The resolution of the 3D geometry is not yet fantastic.
• Placing lights in unexpected locations can cause artifacts.
• Images with specular highlights and complex materials like skin are challenging.

8. Future Potential

• The technology is expected to improve with future research.
• "But just think about what we will be capable of two more papers down the line. My goodness."

9. Impact and Significance

• The technique represents a fundamental transformation of the photograph.
• It turns static memories into editable worlds.
• It empowers artists to control reality long after the photo is taken.
• "Today, we are witnessing the fundamental transformation of the photograph, from a static memory into a living, editable world. This isn't just about making cool edits; it's about handing the artist the power to direct reality itself, long after the shutter has closed."

10. Conclusion

• The AI relighting technique is a significant advancement in image editing.
• It enables realistic and dynamic lighting changes in 2D photographs.
• Despite current limitations, the technology has immense potential for future development and creative applications.
• "This paper and the quality of presentation they do in their video needs an award. Full stop."