Stanford Robotics Seminar ENGR319 | Autumn 2025 | Manipulation Data Pyramid

Key Concepts

• Imitation Learning: Learning by observing and replicating actions.
• Reinforcement Learning (RL): Learning through trial and error with rewards and penalties.
• Pre-training: Initial training of a model on a large dataset to acquire general knowledge.
• Fine-tuning: Adapting a pre-trained model to a specific task or domain.
• Embodiment: The physical form and capabilities of a robot.
• Data Pyramid: A framework for robot manipulation learning, consisting of pre-training, embodiment-specific fine-tuning, and reinforcement learning.
• Motion Level Information: Detailed information about how actions are performed, crucial for manipulation.
• Data Scaling Law: The relationship between the amount of data and model performance.
• Environment Generalization: The ability of a model to perform well in unseen environments.
• Object Generalization: The ability of a model to perform well with unseen objects.
• Foundation Priors: Prior knowledge (policy, value, success signals) used to guide RL.
• Reward Shaping: Modifying the reward function to guide RL exploration.
• Sample Efficiency: The amount of data required for an RL algorithm to learn effectively.

Robot Manipulation Learning: A Data-Driven Approach

This talk outlines a comprehensive approach to enabling robust robot manipulation, drawing parallels with human and animal learning, and large language models (LLMs). The core idea is to build a "manipulation data pyramid" that leverages large-scale data for pre-training, followed by embodiment-specific fine-tuning, and finally, sample-efficient reinforcement learning for skill refinement.

1. The Manipulation Data Pyramid

The proposed framework consists of three stages:

• Bottom Layer: Large-Scale Pre-training: This stage focuses on learning from vast quantities of data, such as human demonstrations or simulation data. The goal is to equip the model with general prior knowledge for manipulation tasks, including object avoidance and path planning.
• Middle Layer: Embodiment-Specific Fine-tuning: Here, the pre-trained model is adapted to the specific physical limitations and capabilities of the robot. This involves collecting robot-specific data to understand constraints like arm reach and payload capacity.
• Top Layer: Reinforcement Learning: This final stage refines the robot's skills through sample-efficient RL in the physical world. The aim is to achieve high success rates, enabling precise actions like inserting a needle into a small hole or preventing frequent object drops.

The speaker acknowledges that significant challenges remain in each stage, including identifying optimal pre-training data and loss functions, determining the data quantity for fine-tuning, and developing sample-efficient RL algorithms for real-world deployment.

2. Pre-training: Learning Motion Information from Human Videos

Key Problem: Existing pre-training methods for robots often rely on static internet data (text and images), which lacks the crucial "motion level of information" needed for manipulation. Robot operation data is scarce, making it difficult to acquire this information.

Proposed Solution: Leverage the vast amount of human video data available online, which inherently contains rich motion information. The central question is whether this motion information can be effectively transferred to robots, given differences in morphology and visual appearance.

Experimental Design:

• Data: Two types of data were used: robot operation data for training tasks and human video data for evaluation tasks.
• Evaluation Strategy: The model was evaluated on tasks not present in the robot training data but present in the human video data. Success on these unseen tasks would indicate successful knowledge transfer.
• Methodology:
  • Human hand poses were extracted from videos.
  • These hand poses were "retargeted" to the robot's morphology.
  • The vision component remained unchanged, using original human video frames.
  • Robot operation data (robot visuals, robot hand motions) and human data (human visuals, retargeted human hand poses) were mixed and used for end-to-end training.
• Results:
  • A non-trivial success rate of approximately 20% was achieved on tasks unseen in the robot data.
  • This success rate, while low, demonstrated the ability to transfer motion information from human videos to robots.
  • A more detailed evaluation using a rubric (scoring individual steps like moving towards an object, grasping, pouring) revealed that even tasks with 0% success rate showed learning of motion information.
• Limitations: The human participants were instructed to perform robotic actions, and the transfer was limited by the robot's current dexterity. Scaling this to "in the wild" internet videos remains a challenge.

3. Data Scaling Law for Robot Manipulation

Key Problem: After pre-training, robots need embodiment-specific data for fine-tuning. The question is: how much data is required to achieve high success rates, and how can data collection be optimized?

Proposed Solution: Investigate the "data scaling law" in imitation learning, characterizing the relationship between data quantity/diversity and generalization performance.

Goals:

1. Understand the relationship between collected data and model performance.
2. Design an efficient data collection schema for new tasks.

Methodology:

• Collected approximately 40,000 demonstrations and performed over 15,000 evaluations.
• Evaluated generalization based on environment generalization (different backgrounds, lighting) and object generalization (different object textures, sizes).

Key Findings:

• Diversity is Crucial: Increasing data diversity (number of environment-object pairs) significantly improves performance.
• Diminishing Returns with Quantity: Once diversity reaches a certain level (e.g., 32 environments/objects), the absolute number of demonstrations per environment has less impact. Collecting data from more diverse environments is more beneficial than collecting many demonstrations in a few environments.
• Data Scaling Law: The relationship between data and performance follows a scaling law, similar to LLMs, with an R-score greater than 0.95. This suggests that imitation learning also obeys these laws.
• Optimized Data Collection Schema: A schema was developed by finding as many environments as possible (e.g., 32) and collecting about 50 demonstrations per environment.
• Real-World Application: This schema, with a total of 1,600 demonstrations, was applied to new tasks (e.g., "f towel," "unplug charger"), achieving 80-90% success rates in novel environments with novel objects and lighting conditions.
• Limitations: This approach did not utilize pre-training, and the success rates had significant standard deviations, suggesting subtle failures.

Speaker's Insight on Diversity: Diversity is defined by the majority of visible pixels being different, not necessarily vastly different physical environments. While data augmentation can help with low-level variations (lighting, texture), it struggles with capturing diverse motions.

4. Reinforcement Learning with Foundation Priors

Key Problem: Achieving very high success rates (e.g., 99%) often requires reinforcement learning, but RL is notoriously sample-inefficient in the real world.

Proposed Solution: Develop a new paradigm for RL that leverages "foundation priors" – prior knowledge about policies, value functions, and success signals – to make RL sample-efficient and effective in the physical world. This approach is inspired by how humans learn complex skills.

Human Learning Analogy: Humans don't start with random trial and error. They have prior knowledge about how to perform a task, how to correct errors (e.g., throwing too far), and when they have succeeded.

Foundation Priors Used:

• Policy Prior: An initial policy obtained from the pre-training and fine-tuning stages (achieving ~80% success).
• Value Prior: Learned from human videos using a method called "value implicit pre-training," estimating the optimal time to reach a goal.
• Success Reward Prior: Generated by querying GPT-4 for a binary success/failure signal for a given state.

Methodology:

• Policy Prior: Constrained using a behavior cloning loss.
• Value Prior: Used for "reward shaping" to guide exploration.
• Success Reward Prior: Added directly to the reward function.
• RL Algorithm: Any standard RL algorithm can be used; a modified Soft Actor-Critic (SAC) was employed.

Key Properties and Results:

• No Manual Rewards: The system learns without requiring manual reward engineering.
• Dense Rewards: Foundation priors provide dense rewards, facilitating learning.
• High Sample Efficiency: Learning improved success rates from 20% to 85% on average within just one hour of real-world training.
• Demonstrated Tasks: The method was successful across various tasks, including watering flowers, picking and placing eggplants, opening drawers, unscrewing bottle caps, and playing mini-golf.
• Time-Bound Trials: Each trial was time-bound (e.g., 30 seconds), though ideally, a "done" signal from the policy would be used.

Speaker's Insight on Failure Data: While failure data is valuable, knowing what is wrong doesn't always tell you what is right. RL inherently uses failure data by providing negative rewards, pushing the policy away from incorrect behaviors.

5. Future Directions and Open Questions

The speaker highlights several areas for future research:

• Pre-training: How to effectively utilize "in the wild" human videos and improve the quality of priors.
• Fine-tuning: Investigating data scaling laws in multi-task settings and how tasks with shared similarities can benefit from each other.
• Reinforcement Learning: Further improving sample efficiency (reducing training time from hours to minutes) and enabling success rate transfer across tasks.
• Hardware Limitations: Addressing how to translate human motion data to better suit specific robot hardware, acknowledging that current hardware (e.g., a 6-DOF hand) can limit gripping capabilities.
• Generated Video: The potential for generated videos to replace real-world videos for pre-training is becoming increasingly plausible.
• Autonomous Operation: Moving towards fully autonomous RL in the real world, addressing safety constraints and the need for multi-task learning.
• Learning from Failure: Exploring more sophisticated ways to learn from failure data beyond standard RL negative rewards.

6. Conclusion

The presented work advocates for a multi-stage learning paradigm for robot manipulation, starting with broad pre-training on diverse data, followed by embodiment-specific adaptation, and culminating in sample-efficient RL guided by foundation priors. This approach aims to bridge the gap between current robot capabilities and the sophisticated manipulation skills observed in humans and animals, paving the way for more general and capable robotic systems.