Robotics: why now? - Quan Vuong and Jost Tobias Springberg, Physical Intelligence

Key Concepts

Vision Language Action Models (VLAs), Vision Language Models (VLMs), multimodal LLMs, robot data engine, teleoperation, diffusion models, open world generalization, action expert transformer, high-frequency control, data diversity, model scaling, robot partnerships.

Vision Language Action Models for Robotics: A Deep Dive

Introduction

The presentation outlines the mission to create a universal robot control model capable of performing any task on any robot. This goal requires significant scientific breakthroughs, and the approach involves open research, open-source models, and public discussions. The core focus is on Vision Language Action Models (VLAs) and the challenges and advancements in their development for robotics.

The Evolution of Robotics

• Traditional Robotics: Characterized by constrained environments (e.g., factories), repetitive motions, and structured settings. Real-world applications often face difficulties.
• Modern Robotics: Demonstrates complex physical motions (e.g., humanoid dancing) and the ability to operate in semi-structured environments (e.g., sorting laundry).
• Key Change: The rise of AI, particularly vision language action models, is driving advancements in robotics.

Vision Language Action Models (VLAs) Explained

• VLMs (Vision Language Models): Take text and images as input, embed them, and use a transformer model to generate text-based answers.
• VLAs (Vision Language Action Models): Adapt VLMs for robotics by incorporating robot state information (e.g., joint positions) and outputting actions to directly control the robot.
• Difference between VLM and VLA: Instead of answering questions about the scene, VLA produces actions to control the robot directly.

Challenges in VLA Training

• Data Acquisition: Unlike VLMs that can leverage web data, VLA training requires specialized robot data, posing a "trillion-dollar question" regarding the ideal data source.
• Model Adaptation: VLM backbones need adaptation to enable high-frequency robot control.
• Deployment: Lack of standard solutions for deploying robot policies across various locations (on-premise, on-device).

Building a Robot Data Engine

• Custom Data Engine: PI is building a data engine from the ground up to enable robust and dextrous policies for robots.
• Operationalization: Getting the data pipeline right is considered more than 50% of the work.
• Process:
  1. Task Selection: Choosing tasks to test current capabilities (e.g., folding clothes, bagging groceries).
  2. Teleoperation: Human operators control robots using a custom runtime and teleoperation system, demonstrating intricate tasks. Leader arms are used to trace motions, which are then transferred to the robot.
  3. Data Annotation: Data is annotated in the cloud and filtered for model training.
  4. Policy Training: Trained policies enable autonomous task execution.
• Data Collection Metrics: A dashboard tracks data collection sessions, showing operators performing specific episodes for specific tasks.

Data Scale and Diversity

• Initial State: The largest public dataset (Open Cross Embodiment) contained ~3,800 hours of data.
• Six-Month Progress: PI collected ~10,000 hours of successful episodes in tens of environments.
• Additional Six Months: Data expanded to include mobile manipulation setups, covering hundreds of scenes and environments.
• Impact: Increased scale and diversity enable new capabilities and engineering challenges.

VLA Capability Trends

• VLMs: Constant improvements from conversational agents to RL-trained reasoning models and coding assistants.
• VLAs: Follow a similar trajectory with a time lag.
  • Early VLAs (e.g., RT2): Proofs of concept with some generalization but limited by data availability.
  • Dextrous Multi-Robot VLAs (e.g., Gemini for Robotics, Nvidia's group models, PI Zero): Models that produce actions via diffusion for high-frequency control. PI Zero is open source.

PIO5: VLA with Open World Generalization

• Goal: To study how model capabilities change with increased data diversity.
• Data Expansion: Includes static and mobile robot data, web data, object detection data, and language annotations.
• Architecture:
  • VLM Backbone: Pre-trained transformer model trained for general questions and task subdivision (e.g., "clean my bedroom" -> "pick up a pillow").
  • Action Expert Transformer: Attends to the VLM and produces continuous output actions via diffusion flow matching.
• Performance: PIO5 can perform long-horizon tasks (up to 10 minutes) in unseen homes, demonstrating broad generalization.
• Generalization Testing: Training PIO5 on a fixed amount of data but varying the number of homes used for training. Results showed that performance in a held-out test scene increased with more training locations, eventually matching or surpassing training with the held-out scene specifically.

Robot Partnerships

• Remote Robot Control: Demonstrated controlling a robot in a remote location to make coffee, highlighting the potential for model portability across different hardware platforms.
• Open Collaboration: PI is open to working with other companies, sharing model checkpoints and engaging in technical discussions.
• Bottleneck: The biggest bottleneck is the need for talented individuals to solve scientific, engineering, and operational challenges.

Conclusion

The presentation highlights the progress and challenges in developing VLAs for robotics. The key takeaways are the importance of a robust data engine, the benefits of data diversity for generalization, and the potential for VLAs to enable robots to perform complex tasks in unstructured environments. The emphasis on open research and collaboration underscores the need for collective effort to achieve the goal of a universal robot control model.