Robots as professional Chefs - Nikhil Abraham, CloudChef

Cloud Chef: Training Robots to be Chefs

Key Concepts:

• Culinary Intelligent Robots: Robots that can act, sense, reason, and behave like a chef.
• Robot Foundation Models: Models fine-tuned for motion primitives like picking and stirring.
• Thermal and Visual Embeddings: Models specific to cooking that help robots reason through unseen environments.
• State Machines: Recipes modeled as state machines based on embedding models.
• Teleoperation: Remote operation of the robot by a human.
• Motion Primitives: Basic movements like picking, stirring, and pouring.
• Line Cooking: The process of cooking food to order in a restaurant.

1. Introduction and Cloud Chef's Mission

• Nikil, co-founder and CEO of Cloud Chef, explains how they transformed a general-purpose robot into a professional chef.
• Cloud Chef's mission is to make high-quality, nutritious food affordable by automating commercial kitchen labor using "culinary intelligent robots."

2. Robot Form Factor and Cost-Effectiveness

• The video contrasts expensive and unreliable humanoid robots (like Tesla Optimus) with more practical, general-purpose robots consisting of two hands on a mobile base.
• These wheeled robots are significantly cheaper than human labor, but require sophisticated software.
• Cloud Chef provides the necessary software to enable these robots to perform chef-like tasks.

3. Culinary School for Robots: Learning Motion and Understanding Food

• "Culinary school" for a robot involves learning motion primitives (picking, stirring, etc.) using fine-tuned robot foundation models and teleoperation for edge cases.
• Robots need to understand food: recognizing when onions are brown enough or shrimp is cooked, despite ingredient variations.
• Cloud Chef uses thermal and visual embeddings specific to cooking to reason through unseen environments.
• Recipes are modeled as state machines based on these embedding models.

4. Adapting to New Kitchens and Recipes

• The robot must adapt to new kitchens, understand recipes from a single expert demonstration, interact with humans, and perform real work.
• Cloud Chef's system performs better than expert chefs in their cuisine of training in certain tasks.

5. Performance Evaluation and Data Collection

• The robot's performance is evaluated by its ability to estimate progress in a cooking process, given live cooking data, an expert demonstration, and a text recipe.
• Cloud Chef's models outperform state-of-the-art models like Gemini 1.5 and 1.0 because they incorporate thermal modality.
• Thermal modality data is collected from sensors installed in active commercial kitchens, capturing hundreds of thousands of live-cooked meals across various cuisines and seasons.
• This private data is combined with public data and self-supervised models to create a robust culinary system.

6. Autonomy and Teleoperation

• The system is currently 95% autonomous and 5% teleoperated.
• The robot can learn a recipe from a chef and execute it, adapting to ingredient and appliance variations.
• Example: The robot cooks a recipe from a two-Michelin-star chef, monitoring onion browning and adjusting accordingly.

7. Real-World Deployment and Examples

• The robots are deployed in real-world kitchens, cooking recipes for actual customers.
• Examples include cooking chicken wings, fetching ingredients, and saucing the wings.
• The robot acts as a weighing scale, precisely measuring ingredients.

8. Hiring and Contact Information

• Cloud Chef is hiring for software, ML, and robotics positions.
• Contact: Nikil at [email protected]

9. Q&A Highlights

• Measuring Success: Success is measured by the robot's ability to understand the cooking process (state estimation) and the speed of its physical motions.
• Taste Consistency: The goal is to achieve taste consistency better than a chef can achieve when recreating a recipe. Blind taste tests are used for evaluation.
• Robot Form Factor: The robot consists of two hands on a mobile base with cameras.
• Human Interaction: Ideally, humans shouldn't need to interact, but currently, they do in some deployments.
• Appliance Variation: The robot can work with arbitrary unseen appliances because of its sensing stack and ability to turn knobs.
• Movement Speed: The robot's motions are 80-95% the speed of a human, limited by the speed of human teleoperators providing training data.
• New Tasks: Adding new tasks like chopping requires collecting data for those tasks.
• Where to Eat: Food cooked by Cloud Chef robots can be found at Wingstop in San Francisco, India's Top 20 in Palo Alto, and Alana in Menlo Park.
• Line Cooking Focus: Cloud Chef is initially focused on line cooking, which accounts for 50% of kitchen labor costs.
• Recipe Speedup: Robots can sometimes speed up recipes by optimizing processes.
• 24/7 Operation: Robots can operate 24/7, unlike human chefs.
• Cross-Contamination: Cross-contamination is addressed with small, washable silicone pads.
• Recipe Modification: Modifying recipes to cook faster is still experimental and depends on the cuisine.

10. Conclusion

• Cloud Chef is automating commercial kitchen labor with robots that can learn, adapt, and perform chef-like tasks.
• The system combines advanced sensing, machine learning, and robotics to achieve high levels of autonomy and performance.
• The robots are deployed in real-world kitchens, cooking food for actual customers.