NVIDIA’s New AI: Impossible Ray Tracing!

Key Concepts:

• Rasterization: A fast rendering technique commonly used in computer games.
• Ray Tracing: A rendering technique that simulates light rays for high-quality reflections and refractions.
• Gaussian Splatting: A technique representing scenes as collections of small Gaussians (bumps).
• 3D Gaussian Unscented Transform (3DGUT): A new method combining Gaussian Splatting with ray tracing.
• Secondary Rays: Rays of light added to Gaussian Splatting to enable reflections and refractions.
• Subsurface Scattering: A rendering technique for translucent objects like skin, marble, and milk.
• Relighting: The process of changing the lighting of an object in a rendered scene.

1. Introduction: Rasterization vs. Ray Tracing

• Rasterization is fast and widely used in computer games but lacks high-quality reflections and refractions.
• Ray tracing simulates light paths, producing realistic reflections, refractions, and volumetric caustics.
• Ray tracing is computationally expensive, taking minutes to weeks to render complex scenes.
• The video introduces a new approach that combines the speed of rasterization with the quality of ray tracing.

2. Gaussian Splatting: A Fast but Limited Technique

• Gaussian Splatting represents scenes as collections of small Gaussians, allowing for real-time rendering.
• Scenes can be captured from real life using cameras and converted into digital versions.
• Limitations of Gaussian Splatting include poor mirror-like specular reflections, high memory usage, and lack of support for advanced camera models (e.g., fisheye, rolling shutter).
• Gaussian Splatting lacks refractions and other advanced effects.

3. 3DGUT: Combining Gaussian Splatting and Ray Tracing

• Researchers at NVIDIA developed 3DGUT, which integrates Gaussian Splatting with secondary rays.
• Secondary rays enable reflections and refractions within the Gaussian Splatting framework.
• 3DGUT allows for real-time rendering of virtual worlds with high-quality reflections and refractions.
• It supports fisheye cameras and rolling shutter effects.

4. Performance and Comparisons

• 3DGUT outperforms previous techniques for extending Gaussian Splats to fisheye cameras, which suffer from artifacts and distortions.
• 3DGUT produces significantly better results, especially for objects close to the camera.
• The source code for 3DGUT is available for free.

5. Subsurface Scattering for Gaussian Splats

• The video discusses subsurface scattering, a technique for rendering translucent objects.
• A rasterization-based method called Separable Subsurface Scattering is mentioned, which is fast and compact (fitting into 4 kilobytes).
• The new work extends subsurface scattering to Gaussian Splats, enabling relighting and material editing.
• Users can change the appearance of objects from skin to glass to wax.

6. Applications and Impact

• 3DGUT has potential applications in computer games, virtual worlds, and self-driving car training.
• The ability to simulate various camera models and rolling shutter is particularly useful for training AI in self-driving cars.
• The presenter expresses surprise that these papers are not receiving more attention.

7. Conclusion

• 3DGUT represents a significant advancement in rendering technology by combining the speed of Gaussian Splatting with the quality of ray tracing.
• The availability of the source code encourages further research and development in this area.
• The integration of subsurface scattering further enhances the realism of virtual worlds.
• The presenter encourages viewers to explore the papers and share their thoughts in the comments.

Notable Quotes:

• "So, rasterization, fast, limited, ray tracing, a complete solution, but slow."
• "This research work does it by taking Gaussian Splatting, and adding to it something they call secondary rays."
• "A proper virtual world running in real time, and high-quality reflections. I am absolutely loving this one."
• "Gaussian Splats are not old at all, it’s about 2 years old. And now, through the power of open research and human ingenuity, and a hint of AI, reflections, refractions, and translucency is already possible for Gaussian Splats."