LeRobot – Lowering the entry barrier to AI for robotics

Robotics and AI: Democratizing Access with Hugging Face

Key Concepts:

• AI for Robotics
• Large Language Models (LLMs)
• Agentic AI
• Action Tokens
• Open-Source Robotics
• Teleoperation
• Humanoid Robots
• Sovereignty Concerns
• Safety in Robotics

Introduction: The Rise of AI and its Application to Robotics

The speaker, Remi Cadene from Hugging Face, reflects on the rapid advancements and adoption of AI, particularly LLMs. He highlights the transformative impact of LLMs on coding productivity and anticipates the emergence of agentic AI capable of automating complex digital tasks. He emphasizes the potential of applying these same AI techniques to the physical world through robotics, opening up vast possibilities.

Pi-Zero: A Case Study in AI-Powered Robotics

Cadene introduces Pi-Zero, an advanced AI model developed by a physical intelligence company. Pi-Zero is trained using PaliGemma, fine-tuned to produce action tokens (motor angles) instead of language tokens. This approach eliminates the need for 3D representation, reconstruction, LiDAR, or depth sensors, relying solely on standard cameras and motor information. The demonstration shows Pi-Zero successfully folding a shirt, a task previously difficult to automate, showcasing the model's ability to correct errors and adapt to real-world conditions. The key takeaway is the simplicity and generality of the approach: show the robot the task a few times, and it learns.

Hugging Face Robot: An Open-Source Initiative

Recognizing the potential of AI in robotics, Hugging Face launched an open-source library called "Hugging Face Robot" to democratize access to this technology. The library provides:

• Datasets: Tools to host, share, and visualize robotics datasets for free on Hugging Face.
• Pre-trained Models: Including Pi-Zero, accessible in Python.
• Middleware: An efficient Python middleware for communicating with various robots.
• Community: A Discord community of 4,000 members for support and collaboration.

The $100 Robot Arm: A Full-Stack Project for Accessibility

To further enhance accessibility, Hugging Face developed a low-cost robot arm in collaboration with Rob Knight from the Robot Studio. The arm costs approximately $100, a significant reduction from the $10,000+ price tag of traditional lab robots.

Step-by-Step Assembly and Training:

1. Assembly: The robot arm can be assembled like LEGO using 3D-printed parts and motors. Tutorials are available on GitHub.
2. Data Recording: Trajectories are recorded using laptop and phone cameras through teleoperation.
3. Training: The AI is trained on a MacBook Pro or Linux machine with NVIDIA GPUs. Generalization can be observed within approximately half an hour.
4. Evaluation: The robot's performance is evaluated by placing LEGO blocks in various locations and observing its ability to grasp them.

Examples from the Discord community demonstrate the robot arm being used to automate tasks at home, such as folding clothes, with a setup costing around $400.

Community Contributions and Hackathons

The open-source nature of the project has fostered community contributions, including attaching a Muppet to the robot arm and connecting it to a VLM to create a language interface for cooking assistance. Hackathons have also been organized, enabling individuals with no prior robotics or AI experience to assemble robots, follow tutorials, and automate tasks within a weekend.

Mobile Robot Base and Humanoid Robots

In collaboration with students from the SIG Robotics Lab at the University of Illinois, Hugging Face developed a mobile base with three wheels, allowing the robot arm to perform tasks requiring a wider range of motion. The speaker also showcases the application of the "tokens in, tokens out" approach to more capable robots, such as the humanoid robot developed by Pollen Robotics. This robot can be instructed to grasp objects through natural language commands, demonstrating its potential to assist individuals with disabilities. Unitree robots are also mentioned as using similar techniques for humanoid manipulation learning.

Humanoid Robot Arm and Hand: Towards Advanced Manipulation

Hugging Face is also developing a low-cost humanoid robot arm and hand, costing around $500. A simple glove with an exoskeleton design is used for teleoperation, enabling the recording of data for training the robot arm. The system utilizes fishing nets for tendon support, showcasing the use of inexpensive components to create capable robots. The next step is the development of legs for the humanoid robot.

Conclusion: Call to Action and Considerations

The speaker concludes by encouraging viewers to build their own robots, follow the tutorials on GitHub, and join the Discord community. He emphasizes the importance of addressing sovereignty concerns, safety, and interface design in the development of robotics. He highlights the reusability of knowledge from LLMs, VLMs, and audio processing in robotics and encourages individuals to join the effort, particularly in Paris. He hopes to see everyone at the next hackathon.

Notable Quotes:

• "Every day, I am impressed by the new capabilities of our AI systems and how fast they are adopted by people." - Remi Cadene
• "It has never been the best time to start." - Remi Cadene

Technical Terms:

• LLMs (Large Language Models): AI models trained on vast amounts of text data, capable of generating human-like text.
• Agentic AI: AI systems capable of autonomously performing tasks and making decisions.
• Action Tokens: Discrete units representing motor actions, used to control robot movements.
• Teleoperation: Controlling a robot remotely, often using a human operator.
• VLMs (Vision Language Models): AI models that combine visual and textual information.

Synthesis:

The presentation highlights Hugging Face's efforts to democratize AI for robotics through open-source tools, low-cost hardware, and community engagement. By simplifying the development process and lowering the barrier to entry, Hugging Face aims to empower individuals and researchers to explore the potential of robotics and address the challenges associated with its development and deployment. The focus on accessibility, combined with the use of advanced AI techniques, positions Hugging Face as a key player in shaping the future of robotics.