What Is a Humanoid Foundation Model? An Introduction to GR00T N1 - Annika & Aastha

Key Concepts

• Robotics Foundation Model: A general-purpose AI model for robots, capable of adapting to different embodiments and tasks.
• GR101: A specific robotics foundation model developed by Advidia, open-source and customizable.
• Physical AI Life Cycle: The process of generating data, training a model, and deploying it on a robot for real-world operation.
• Data Pyramid: A strategy for collecting and generating data for robotics, balancing real-world, synthetic, and internet data.
• Dual System Architecture (System 1 & System 2): An architecture inspired by "Thinking, Fast and Slow," where System 2 plans and System 1 executes actions quickly.
• Embodiment: The physical form of a robot (e.g., humanoid, robotic arm).
• Imitation Learning: Training a robot by copying a human expert's actions.
• Reinforcement Learning: Training a robot through trial and error, maximizing rewards.
• VLM (Vision Language Model): A model that bridges the gap between visual and textual data, enabling the robot to understand both.
• Diffusion Transformer Block: A key component of the GR101 architecture, using cross-attention and self-attention to process state, action, vision, and text tokens.

The Challenge: Physical AI and the Job Market

The presentation begins with a "hot take" challenging the narrative of AI causing widespread job loss. A McKinsey report indicates a potential surplus of jobs in the world's 30 most advanced economies. The key issue is not a lack of jobs, but a lack of people to fill them, particularly in sectors like leisure, hospitality, healthcare, construction, transportation, and manufacturing. These industries require physical interaction and cannot be automated by language models alone. The core challenge is bridging the gap between the intelligence of language models and their ability to operate in the physical world, requiring "physical AI." Humanoid robots are designed to resemble humans because the world is designed for humans. To create generalist robots capable of performing multiple tasks, mimicking the human form is often the most practical approach.

The Physical AI Life Cycle: Data, Training, and Deployment

The process of creating a functional robot involves three key stages:

1. Data Generation/Collection/Multiplication: Gathering and creating the necessary data to train the robot.
2. Model Training: Training the AI model using the collected data.
3. Deployment: Deploying the trained model on the robot for real-world operation.

Nvidia refers to this as the "three computer problem" due to the different computational requirements of each stage:

• Simulation: Requires powerful simulation capabilities (e.g., OVX Omniverse).
• Training: Requires massive data processing power (e.g., DGX).
• Deployment: Requires efficient edge computing (e.g., AGX).

Project Groot is Nvidia's comprehensive strategy for advancing humanoid and other robotics, encompassing compute infrastructure, software, and research. The GR101 foundation model is a key component of this strategy.

GR101: A Robotics Foundation Model

The GR101 foundation model, announced at GTC, is open-source, highly customizable, and designed for cross-embodiment applications. It is a 2 billion parameter model, relatively small compared to LLMs but substantial for robotics. The model can be adapted for various robot embodiments and use cases.

The Data Pyramid: Addressing the Data Scarcity Problem

The biggest challenge in robotics is data scarcity. Unlike language models that can leverage vast amounts of internet text, robots lack a comparable data source. The "data pyramid" addresses this:

• Top (Smallest): Real-world data collected from robots performing tasks, often through human teleoperation. This data is high-quality but expensive and limited. Data is collected through humans teleoperating a robot, often using devices like Apple Vision Pro and gloves.
• Middle: Synthetic data generated through simulation. While theoretically infinite, creating high-quality simulation environments is labor-intensive. Techniques like multiplying human trajectories using video generation models are also employed.
• Bottom (Largest): Internet data, primarily human videos of task completion. This data is unstructured and not directly relevant to robots but can provide some value.

DreamGen, announced at Computex, is a solution for data generation. Combining simulation data with real-world data is an active area of research.

GR101 Architecture: Dual System and Diffusion Transformer

The GR101 architecture is inspired by Daniel Kahneman's "Thinking, Fast and Slow," employing a dual-system approach:

• System 2 (Planner): The "brain" of the robot, responsible for breaking down complex tasks into simpler steps. It operates slowly and deliberately.
• System 1 (Executor): The "fast" system, executing the tasks planned by System 2 at a high frequency (120 Hz).

The architecture processes the following inputs:

• Robot State: A snapshot of the robot's physical condition and environment at a given time.
• Noised Action: The robot's intended action, with added noise to reflect sensor imperfections.
• Vision Input: Images captured by the robot's cameras.
• Text Input: Language prompts instructing the robot on what to do.

The inputs are processed through the following components:

1. State Encoder & Action Encoder: Generate state and action tokens.
2. Vision Encoder: Processes image input and generates vision tokens.
3. VLM (Vision Language Model): Standardizes the encoding format of vision and text tokens (e.g., using EagleToVLM in GR101).
4. Text Tokenizer: Processes text input and generates text tokens.
5. Diffusion Transformer Block: Uses cross-attention and self-attention to integrate all tokens.
6. Action Decoder: Translates the output tokens into action vectors specific to the robot's embodiment, enabling the model to be a generalist.

The action decoder is crucial for adapting the model to different robot embodiments.

Robot Learning: Imitation vs. Reinforcement

There are two primary methods for training robots:

• Imitation Learning: The robot learns by copying a human expert's actions, minimizing the loss between the robot's actions and the expert's. It is bottlenecked by the availability of expert data.
• Reinforcement Learning: The robot learns through trial and error, maximizing rewards. It overcomes the expert data bottleneck but faces challenges in transferring knowledge from simulation to the real world (sim-to-real gap).

GR101 utilizes both imitation and reinforcement learning techniques.

Capabilities and Conclusion

The trained GR101 model can perform various tasks, including pick-and-place operations in kitchens and industrial settings. It can even be trained to perform more complex tasks with enough training.

The three core principles of GR101 are:

1. Data Pyramid: Addressing data scarcity through a combination of real-world, synthetic, and internet data.
2. Dual System Architecture: Co-training System 1 and System 2 for coherent task execution.
3. Generalist Model: Leveraging foundation knowledge to adapt to different embodiments and tasks, similar to how LLMs can be fine-tuned for specific applications.

The GR101 model can be adapted to any embodiment and any downstream task.