Can AI program a robot dog?

Key Concepts

• Frontier AI Models: Advanced artificial intelligence models, such as large language models (LLMs), that are pushing the boundaries of AI capabilities.
• Project Fetch: An experiment designed to measure the impact of Claude (a frontier AI model) on human performance in a robotics task.
• Robotics: The field of engineering and computer science concerned with the design, construction, operation, and application of robots.
• ROS2 (Robot Operating System 2): An open-source framework for developing robot software.
• SDK (Software Development Kit): A set of tools and libraries that allows developers to create applications for a specific platform or system.
• PIP: A package installer for Python, used to install and manage software packages.
• Autonomous Task: A task that a robot can perform without direct human intervention.

Project Fetch: Measuring Claude's Impact on Robotics Tasks

This summary details the findings of "Project Fetch," an experiment conducted by Anthropic to assess how frontier AI models, specifically Claude, can accelerate human proficiency in robotics tasks, particularly for individuals with limited prior experience. The experiment aimed to bridge the gap between software engineering and the physical world through robotics.

Experiment Design and Phases

Project Fetch was a one-day experiment divided into three phases, all centered around the task of controlling a robot dog to fetch a beach ball. Two teams of software and research engineers at Anthropic, who had minimal robotics experience, participated. One team had access to Claude, while the other did not.

Phase 1: Pre-provided Controllers

• Objective: To use pre-programmed controllers to guide the robot dog to retrieve a beach ball and return it to its starting position.
• Team with Claude: Completed the task in approximately 7 minutes.
• Team without Claude: Took approximately 10 minutes to complete the task.
• Observation: Even in this basic phase, the team with Claude showed a slight acceleration.

Phase 2: Programming Custom Controllers

• Objective: To program a custom controller from scratch to control the robot dog, requiring direct interaction with the hardware and software development on a laptop.
• Key Challenge: A significant bottleneck was establishing communication between the laptop and the robot hardware, involving the installation of various software libraries and dependencies (e.g., ROS2 SDK, PIP packages).
• Team with Claude:
  • Claude assisted in identifying and installing necessary software libraries.
  • Claude helped set up a "dog server" to facilitate communication between multiple computers and the robot.
  • This team finished phase two in about 2 hours and 15 minutes.
  • The most significant uplift from Claude was observed in the task of connecting to the robot, which is considered a difficult challenge for anyone trying to interface with arbitrary hardware.
• Team without Claude:
  • Struggled significantly with setting up the development environment and establishing hardware communication.
  • Explored multiple unsuccessful paths.
  • Required intervention from the experiment facilitators to provide a working strategy, which then unlocked progress for the rest of the phase.
• Notable Incident: The team without Claude was humorously "disqualified" for their robot dog hitting another participant.

Phase 3: Greater Degree of Autonomy

• Objective: To write a program that would enable the robot dog to autonomously search for, detect, walk to, and retrieve the beach ball without human intervention. This phase was designed to simulate the future challenges frontier models are expected to solve in enabling autonomous robotic actions.
• Team with Claude:
  • Came "fairly close" to completing the autonomous task.
  • Estimated to be about an hour and a half away from full completion by the end of the experiment.
• Team without Claude:
  • Made progress on individual sub-tasks like tracking the robot's location and detecting the ball.
  • Did not succeed in integrating these components to achieve full autonomy.
  • Expressed a strong sentiment of missing Claude's assistance ("I miss Claude so much").

Overall Results and Implications

• Time Savings: The team with Claude completed all their accomplished tasks approximately two hours faster than the team without Claude.
• Near-Term Impact: AI models like Claude are expected to significantly lower the barrier to entry for individuals with limited robotics experience to engage meaningfully with robots. This was demonstrated by the dramatic acceleration in the participants' ability to work with the robot dog.
• Long-Term Vision: The experiment is seen as a leading indicator of future trends. What currently requires the collaboration of a human and an AI model may eventually be handled solely by advanced AI models.
• Broader Impact: The influence of AI is not confined to software but is extending into hardware and the physical world.

Key Arguments and Perspectives

• AI as an Accelerator: Frontier AI models can dramatically accelerate human learning and task completion, especially in complex technical domains like robotics.
• Democratization of Robotics: AI can make robotics more accessible to a wider range of individuals, reducing the need for deep specialized knowledge.
• Future of Robotics: The future of robotics will likely involve greater autonomy driven by AI, with AI models capable of solving complex problems to direct robotic actions.
• Synergy of Human and AI: Currently, the most effective approach involves a combination of human expertise and AI assistance, but this is expected to evolve towards AI-driven autonomy.

Notable Quotes

• "Project Fetch is this self-contained experiment where we wanted to measure how much does Claude accelerate humans performing a fairly sophisticated technical task that they do not have experience with?"
• "I've never really understood how reliant I am on Claude doing the menial work, finding all the nitty gritty details I don't want to have to figure out."
• "Probably the area where we saw the most uplift from Claude was just in the task of connecting to the robot. We think that's really important because it is, in fact, difficult for anyone to identify an arbitrary piece of hardware in the world and figure out how to talk to it and how to control it."
• "What today requires the combination of a person and an AI model, tomorrow is likely to just require the AI model."

Conclusion

Project Fetch convincingly demonstrated that frontier AI models like Claude can significantly accelerate human performance in complex robotics tasks, even for individuals with no prior experience. The experiment highlighted Claude's ability to overcome technical hurdles, such as software installation and hardware communication, and to facilitate progress towards more autonomous robotic operations. This suggests a future where AI plays a pivotal role in making robotics more accessible and driving greater autonomy in the physical world.