Stanford Seminar - Multitask Transfer in TRI’s Large Behavior Models for Dexterous Manipulation

Key Concepts

• Diffusion Policy: Single-task imitation learning pipeline using diffusion models for visual motor control.
• Large Behavior Models (LBMs): Multi-task, language-conditioned visual motor policies, often pre-trained on diverse datasets.
• Vision Language Action Models (VLAs): Models using VLM pre-training or co-training.
• Pre-training: Training a model on a large, diverse dataset before fine-tuning it on a specific task.
• Robustness: The ability of a policy to perform well under varying conditions and disturbances.
• Dexterity: Fine motor skills and manipulation capabilities.
• Sim-to-Real Transfer: Transferring policies trained in simulation to real-world robots.
• Statistical Significance: Ensuring that observed improvements are not due to random chance.
• Adaptive Testing: A framework that allows you to with 95% confidence my policy A is better than policy B the mean is is above for instance.
• P-hacking: Manipulating data analysis to achieve statistical significance.
• Joint Impedance Control: A control method that regulates the robot's resistance to external forces.

1. Introduction and Motivation

• The talk focuses on understanding the mechanisms behind the scaling hypothesis in large models for robotics, moving beyond simply observing that scaling works.
• The goal is to understand what these complex systems are actually learning and how they achieve control.
• The speaker emphasizes the importance of rigorous scientific methodology and avoiding hype.

2. Diffusion Policy and Large Behavior Models

• Diffusion Policy: Defined as a single-task imitation learning pipeline. Requires teleoperating a robot to perform a task (e.g., folding a t-shirt) hundreds of times. The output is a visual motor policy.
• Large Behavior Models (LBMs): The multi-task version of diffusion policy. LBMs are language-conditioned visual motor policies pre-trained on a large and diverse dataset.
• LBMs offer the potential to program robots via imprecise natural language or a few demonstrations.
• A key idea is that LBMs can imbue robots with common sense for physical intelligence, leading to robustness.

3. Starter Examples: Breakfast Making and Apple Cutting

• Breakfast Making: A side-by-side comparison of a single-task diffusion policy (trained on 229 demonstrations) versus an LBM pre-trained on a larger dataset and then fine-tuned on the same 229 demonstrations. Both perform well, but the LBM exhibits smoother trajectories and a higher success rate.
• Apple Cutting: Demonstrates a level of dexterity that was unexpected. The robot performs complex maneuvers, including recovering from errors and sheathing the knife (which is a difficult step).
• These examples highlight the potential benefits of pre-training for dextrous manipulation.

4. Key Questions and Hypotheses

• Main Question: Does pre-training actually improve the quality of dextrous manipulation? If so, how?
• Hypothesis 1: Training on many tasks allows for adding new tasks with fewer demonstrations. The goal is a smooth scaling law where the more skills a model has, the better it becomes at learning new skills.
• Hypothesis 2: Pre-training increases robustness. Even if the initial success rate on a task is low, pre-training on diverse tasks should improve the ability to recover from errors.
• A Deeper Question: Is it better to train a robot on 10,000 demonstrations of a single task, or on a diversity of tasks with the same total number of demonstrations? The idea is that diverse tasks can excite motions and recoveries in different parts of the state space, providing long-tail coverage.

5. Methodology and Architectures

• The primary focus is on bimanual manipulation using Panda robots with grippers and wrist cameras.
• Architectures:
  • Scaling up diffusion policy with a language encoder.
  • Vision Language Action Models (VLAs) using VLM pre-training.
  • Video prediction models.
• The talk focuses on a multitask diffusion policy version, which is simpler to study.
• Scaling Up Diffusion Policy:
  • Replaced ResNet with CLIP for vision encoding.
  • Using a CLIP language encoder (mostly frozen).
  • Using DIT instead of UNET.
  • Sharing image encoders.
• The resulting model is only a few times larger than the original diffusion policy, suggesting that the optimization landscape, rather than model capacity, was the limiting factor.

6. Data and Training

• The model was trained on a dataset of roughly 500 high-quality, long-duration skills.
• The dataset consists mostly of real-world data, with a smaller amount of OXE data (batch balanced) and simulation data.
• The simulation data is primarily used for evaluation, acting as a surrogate for real-world performance.

7. Testing and Evaluation

• AB Testing: Always compare new policies against existing policies by re-running evaluations.
• Blind Randomized Trials: Evaluators should not know which policy they are testing.
• Controlled Initial Conditions: Use GUIs to ensure consistent initial conditions.
• Metrics: Beyond success rate, also measure task completion, jitter, pauses, and incorrect task execution.
• Simulation-Based Testing: Essential for automated evaluation. Run evaluations on every checkpoint in the cloud.
• Simulation Philosophy: Use high-quality, artist-touched assets and physics parameters to minimize the sim-to-real gap.
• Benefits of Simulation: Repeatability, ability to run many more rollouts, and automated evaluation.
• Limitations of Simulation: Strong correspondence with real-world performance, but not perfect.

8. Statistical Rigor

• Confidence Intervals: Use 95% confidence intervals to determine if one checkpoint is significantly better than another.
• Checkpoints vs. Architectures: Statistics are about checkpoints (specific model parameters), not about architectures (model types).
• Sources of Randomness: Different objects, environments, initial conditions, and stochastic policies.
• Variability Across Tasks: Acknowledge that performance varies across tasks.
• Number of Rollouts: Budget 50 rollouts for single-task results to achieve smaller confidence intervals.
• P-hacking: Avoid manipulating data analysis to achieve statistical significance.
• Bayesian Analysis: Use Bayesian analysis to show the distribution of possible outcomes and determine if policies can be statistically separated.

9. Low Data Regime and Difficult Tasks

• The experiments are conducted in a low-data regime (200 demonstrations per task) with intentionally difficult initial conditions.
• The goal is to learn about the policies, not to impress with high success rates.
• The speaker emphasizes the importance of making decisions that favor the weaker player (single-task policy) to ensure unbiased results.

10. Specific Results

• Architectural Changes: Bigger encoders and DIT scale better than UNET. Relative action parameterization improves performance.
• Data Normalization: Data normalization is significantly more important than architecture changes.
• Multitask Pre-training: Safely cuts through the noise and makes a big difference.
• LBMs Can Learn All Tasks: The scaled-up multitask diffusion policy can learn all tasks.
• Fine-tuning into New Tasks: LBMs improve rapidly with fewer demonstrations on unseen tasks.
• Robustness: LBMs achieve comparable performance to single-task policies with only 15% of the data in hard, distribution-shifted scenarios.
• Deployment on New Stations: Pre-training helps deploy policies on new robot stations.
• Importance of Data Quality: Even with pre-training, data quality matters. Demonstrations should include examples of recoveries.
• Beware of In-Distribution Performance: Use more aggressive testing to make better decisions based on sim-based testing.

11. Key Takeaways and Observations

• Validation loss (val action MSSE) does not correspond to closed-loop performance.
• Data quality still matters, even with pre-training.
• Beware of looking just at in-distribution performance, particularly in sim, when evaluating pre-training.
• LBMs don't always accomplish the task, but they always look like manipulation.
• The role of pre-training is to achieve a 5-10% success rate on any possible task, enabling RL post-training to refine the policy.

12. Unvarnished Truth and Future Directions

• The speaker presents the worst rollout ever seen for both the single-task policy and the LBM, highlighting the LBM's persistence and recovery attempts.
• LBMs exhibit more recovery behaviors and a higher task completion rate.
• Future directions include collaboration with Boston Dynamics to integrate LBMs with their dynamic robots.

13. Q&A Highlights

• Discussion on defining "same task" with unseen objects.
• Tips for creating good simulators for evaluation.
• Quantifying the importance of generalization data.
• The role of model-based techniques going forward.
• Trusting simulation as an evaluation tool.
• Quantifying recovery behaviors.
• The impact of video generation models.
• Lessons about quantifying recovery behavior.
• The role of the low-level controller in protecting from singularities.
• The importance of tactile sensors.
• The safety of robots and how to make them safer for home use.
• Speeding up real-time performance.

14. Synthesis/Conclusion

The presentation provides a detailed analysis of the benefits of pre-training large behavior models for robotic manipulation. Through rigorous experimentation and statistical analysis, the speaker demonstrates that LBMs can learn skills more quickly, improve robustness, and exhibit more sophisticated recovery behaviors. The talk emphasizes the importance of careful methodology, data quality, and the need for continued research to fully understand the mechanisms behind these models. While challenges remain, the results suggest a promising path towards more capable and adaptable robots.