3D Gaussian Splatting for Realistic Physical AI Simulations

Key Concepts

• 3D Gaussian Splatting: A neural rendering technique for high-fidelity scene reconstruction using optimized 3D Gaussians.
• Neural Reconstruction: Creating 3D models from sparse observations using neural networks.
• Omniverse Neurick: NVIDIA's set of APIs and libraries for generating interactive 3D simulations from real-world data.
• 3D Gaussian Rate Tracing (VDGRT): Introduces ray tracing to Gaussians, enabling secondary lighting and nonlinear camera models.
• 3D Gaussian Unscented Transform (3D GOUT): A hybrid approach using splatting to determine rays for tracing, supporting distorted camera models.
• Cosmos World Foundation Models: NVIDIA's generative AI models leveraging knowledge from millions of physically grounded drives.
• Asset Harvester: A tool that generates novel views of objects using diffusion models and estimates Gaussians from them.
• USD (Universal Scene Description): A format used by NVIDIA to store Gaussian data.
• Physical AI: Using realistic simulations to train and test AI agents in physical environments.

Introduction to 3D Gaussian Splatting

Nick Schneider from NVIDIA discusses how 3D Gaussian splatting has become a key technology for enabling physical AI by providing highly realistic simulations. Gaussian splatting is a neural reconstruction technique that takes sparse observations of a scene as input and trains a model to generate novel views. This is crucial for robotic applications, allowing simulation of various camera paths and sensor configurations.

How Gaussian Splatting Works: The Balloon Analogy

Nick uses an analogy of painting a picture with balloons to explain Gaussian splatting. Each balloon represents a Gaussian, and its position, scale, color, and rotation are optimized to represent the scene. The process involves:

• Placement: Attaching balloons (Gaussians) to represent the scene.
• Runtime: Placing many balloons requires significant processing time.
• Reconstruction Quality: A large number of balloons is needed for good reconstruction.
• Memory: Running out of nail space is analogous to running out of GPU memory.
• Optimization: Rearranging balloons to optimize the view from different perspectives.

In Gaussian splatting, the positions, scale, color, and rotation of Gaussians are optimized. The rendered image is used to calculate a loss, which is then backpropagated to the Gaussians.

Advancements in Gaussian Splatting: VDGRT and 3D GOUT

Two key papers from NVIDIA have expanded the applications of Gaussian splatting:

• 3D Gaussian Rate Tracing (VDGRT): Introduces ray tracing to Gaussians, enabling secondary lighting, nonlinear camera models (like fisheye), and compensation for rolling shutter effects. However, it impacts training and rendering time.
• 3D Gaussian Unscented Transform (3D GOUT): Uses a hybrid approach where splatting determines rays for tracing via sigma points. This allows for nonlinear camera models and rolling shutter consideration in planning processes. 3D GOUT achieves state-of-the-art reconstruction quality with only a slight increase in reconstruction time compared to the original paper. It is particularly beneficial for robotic platforms with diverse cameras and lenses.

A comparison video demonstrates that Gaussian splatting has higher fidelity than NeRF reconstructions. The rolling shutter effect, not accounted for in standard Gaussian splatting, is addressed by 3D GOUT, resulting in less blurred reconstructions. The code for both VDGRT and 3D GOUT is available on GitHub.

Omniverse Neurick: Bringing Real-World Data into Simulation

Omniverse Neurick is a set of APIs and libraries designed to generate interactive 3D simulations from real-world data. It includes reconstruction, rendering, and generative enhancement capabilities.

• Reconstruction: Uses Gaussian-based methods to reconstruct scenes into millions of particles, speeding up reconstruction times compared to NeRF-based methods.
• Rendering: Adds ray tracing to improve the fidelity of the reconstruction using NVIDIA's 3D rendering algorithm.
• Generative Models: Employs models like Neurixer to fill in reconstruction artifacts for high-quality novel view synthesis.

Neurick leverages NVIDIA models and accelerated libraries, including Cosmos World Foundation models for generative AI.

Neurick Workflow

A typical workflow with Neurick involves:

1. Sensor Data: Input from robotic platforms or autonomous vehicles.
2. Encore: Libraries and tools to ensure high data quality at the input stage.
3. 3D Reconstruction: Using 3D GOUT with domain-specific enhancements (e.g., for automotive applications).
4. Neurixer: Refining reconstructions during training or as a post-processing step.
5. Asset Harvester: Extracting objects as Gaussian splats to build asset libraries.
6. USD Storage: Storing the reconstruction output as USD.
7. Rendering: Loading the USD data for rendering via Omniverse or a gRPC service for communication with simulation environments.

This allows for a continuous loop where Neurick renders novel views, the robotic application reacts, generates a new path, and Neurick creates the next frame.

Generative AI Enhancements: Neurixer and Asset Harvester

• Neurixer: Enhances reconstructions by leveraging knowledge from millions of physically grounded drives via the Cosmos World Foundation model. This fills in gaps in observation and improves reconstruction fidelity, especially when generalizing to novel views.
• Asset Harvester: Generates novel views of objects using a diffusion model and estimates Gaussians from them. This allows for the creation of large libraries of vehicle assets for driving applications, even completing shapes from limited observations.

Standardization of Gaussian Data

NVIDIA uses USD to store Gaussian data, but there is currently no industry standard. While standardization is challenging due to the rapid evolution of the field, extensibility is key. NVIDIA is working with the community to define a suitable standard.

Applications and Integrations

Gaussian splatting is already being used by industrial partners in applications like autonomous taxis. Realistic simulation is crucial for safely scaling autonomous driving. NVIDIA's internal AV simulator generates and renders a large set of new reconstructions daily.

Key milestones include:

• Omniverse Neurick Rendering in Kala: Allows developers to test applications in simulation on real-world Gaussian splatting data. Over 80 driving scenes have been reconstructed.
• IS65 Integration with Neurick: Enables users to record real-world environments and turn them into simulations where robots can interact.

Recent research includes:

• GenAI Relighting of Assets: Replaying real-world drives at different times of day.
• Physics Materials for Gaussians: Enabling robots to manipulate and interact with Gaussian objects.
• Large-Scale Scenic Constructions: Rebuilding complete cities.
• Text-to-Environment Generation: Generating environments from text and representing them as Gaussians.

Conclusion

Nick encourages the audience to test the available tools, including VDGRT, Neurixer, Kala, and IS65, and to contribute to further research in this direction. The combination of Gaussian splatting, generative AI, and realistic simulation is paving the way for advancements in physical AI and robotics.