Stanford Seminar - Towards Open World Robot Safety

Key Concepts

• Robot Safety: Ensuring robots operate reliably and without causing harm in real-world environments.
• Safety Filter: A mechanism to monitor and modify a robot's actions to prevent unsafe outcomes.
• Hamilton-Jacobi Reachability: A control systems technique for computing safety filters.
• Generative World Models: Machine learning models that learn latent representations of the world and can predict future states.
• Latent Space: A compressed, abstract representation of data learned by a machine learning model.
• Vision Language Models (VLMs): Models that combine visual and textual information to understand and reason about the world.
• Policy Steering: Selecting the best action plan from a set of generated options based on a task context.

1. Introduction: The Nuances of Robot Safety

• The speaker highlights the rapid advancements in robotics, citing examples like self-driving cars and robots capable of complex manipulation tasks.
• The increased deployment of robots raises concerns about safety and reliability, prompting research and regulatory efforts.
• The speaker argues that safety is a nuanced concept, especially when robots operate at scale in the open world.
• Example: A Tesla self-driving car causing a multi-car pileup by braking unexpectedly illustrates that even seemingly simple safety specifications like "don't collide" can be complex in practice.
• Quote: "When we deploy robots truly at scale truly in the open world what safety means is actually a really really nuanced concept."

2. Challenges in Defining and Achieving Robot Safety

• Decision-making is highly context-dependent, making it difficult to create universal safety rules.
• Beyond collision avoidance, safety includes considerations like avoiding damage to objects or unintended consequences.
• Example: A robot using a large language model to plan tasks might suggest unsafe actions, such as putting a metal bowl in the microwave.
• Example: A Roomba escaping into the open world demonstrates the need for robots to understand the consequences of their actions.

3. Traditional Approaches to Robot Safety: Robust Control Theory

• Robust control theory offers a mathematical framework for safe control, focusing on constraint satisfaction.
• This framework captures feedback loops, where a robot's actions influence long-term outcomes.
• It couples the detection of unsafe events with mitigation strategies.
• Limitation: Traditional methods rely on hand-designed state representations, dynamics models, and failure states, which are difficult to scale to complex environments.
• Technical Terms:
  • State Space: A mathematical representation of all possible states of a system.
  • Dynamical Systems Model: A mathematical description of how a system evolves over time.
  • Failure Set: The subset of the state space where the system is considered to have failed.

4. Generative AI and Robotics: Learning Latent Representations

• Generative AI, particularly trajectory forecasting, excels at learning latent representations of the world from high-dimensional observations.
• These models can be guided by non-expert feedback, such as thumbs up/thumbs down ratings.
• Limitation: Generative AI often focuses on anomaly detection or uncertainty quantification but doesn't provide robots with specific actions to take in unsafe situations.
• Example: Large language models can detect unusual scenes, like a Cybertruck with a light post on its back, but struggle to determine the appropriate response.

5. Uniting Safe Control and Generative Modeling: A Hybrid Approach

• The speaker's research explores how to combine the strengths of safe control and generative modeling to generalize robot safety.
• The goal is to leverage latent representations learned by generative models for safe control.

6. Foundations: Safety Filters and Reachability Analysis

• A safety filter monitors a robot's policy output and modifies it to prevent future failures.
• Constructing a safety filter requires:
  1. A state space.
  2. A dynamical systems model.
  3. A representation of failure.
  4. A safety specification.
• Hamilton-Jacobi reachability is used to compute the safety filter.
• The value function remembers the closest the robot ever gets to failure under the best strategy.
• The safety filter consists of a safety policy (what to do) and a safety monitor (when to act).
• Technical Terms:
  • Safety Policy: A control strategy designed to keep the robot within safe operating conditions.
  • Safety Monitor: A function that determines when the robot is approaching an unsafe state and needs to activate the safety policy.
  • Hamilton-Jacobi Reachability: A technique for computing the set of states from which a system can reach a target set (e.g., a failure set) despite disturbances.

7. Limitations of Traditional Safety Representations: Collision Avoidance

• Many safety representations focus on collision avoidance, such as distance to obstacles or other agents.
• This is insufficient for addressing more complex safety concerns like tearing, breaking, or spilling.
• The challenge lies in scaling hand-designed state and dynamics representations to these complex scenarios.

8. Generalizing Safety Principles: Safe Control in Latent Space

• The key idea is to perform safe control directly in the latent space learned by generative world models.
• The state space becomes an embedding of observations, the failure specification becomes a classifier on the embedding state, and the world model represents the dynamics.
• Process:
  1. Train an encoder to map observations to a latent state.
  2. Train a dynamics model to predict how the latent state evolves based on the robot's actions.
  3. Train a failure classifier to identify unsafe latent states.
• Technical Terms:
  • Encoder: A neural network that maps high-dimensional input data (e.g., images) to a lower-dimensional latent space.
  • Dynamics Model: A model that predicts the next state of a system given its current state and action.
  • Failure Classifier: A model that classifies a given state as either safe or unsafe.

9. Computing a Safety Filter in Latent Space

• The safety filter is computed by solving an optimization problem in the latent space.
• A latent safety Bellman equation is derived to make the computation more tractable.
• Reinforcement learning is used to approximate the safety value function.
• The resulting safety monitor and safety policy are deployed to protect the robot.

10. Evaluation: Dubins Car and Visual Manipulation Tasks

• The latent safety approach is evaluated on a Dubins car benchmark task.
• The learned failure classifier is more conservative than the ground truth, but the unsafe sets are of high quality.
• The approach is also tested on a visual manipulation task where the robot must pick up a green block without toppling red blocks.
• The latent safety filter outperforms other latent control strategies in balancing success and failure rates.

11. Hardware Experiments: Skittles Example

• The approach is deployed on a real robot tasked with picking up a bag of Skittles without spilling them.
• The robot uses a third-person camera and a wrist camera for perception.
• The world model is trained on 1,300 trajectories, including random, safe, and unsafe demonstrations.
• The safety filter successfully prevents the robot from spilling the Skittles in various scenarios.
• Limitations: The approach can fail if the opening of the bag is not observable or if the dynamics of the bag are significantly different from the training data (e.g., using a bag of M&Ms).

12. Addressing Context-Dependent Safety: VLM-in-the-Loop Policy Steering

• The failure specification is context-dependent, meaning that what is considered safe or unsafe can vary depending on the situation.
• The speaker proposes using knowledge in pre-trained vision language models (VLMs) to create a context-dependent failure detector.
• Process:
  1. The robot generates multiple action plans using a base policy (e.g., an imitation learning policy).
  2. A world model predicts the latent states corresponding to each action plan.
  3. A VLM is aligned to reason about the latent states and evaluate the outcomes based on a task context (e.g., "serve a cup of water to the guest").
  4. The VLM selects the best action plan based on the task context.
• Technical Terms:
  • Policy Steering: The process of selecting the best action plan from a set of generated options based on a task context.
  • Vision Language Model (VLM): A model that combines visual and textual information to understand and reason about the world.

13. Benefits of the Two-Stage Pipeline: World Model and VLM

• The two-stage pipeline (world model for prediction, VLM for evaluation) outperforms an end-to-end VLM model.
• The world model is good at embodied reasoning, while the VLM is good at translation and understanding context.

14. Conclusion

• The speaker summarizes the research, emphasizing the potential of combining safe control and generative modeling to generalize robot safety.
• The work represents a step towards enabling robots to operate safely and reliably in complex, open-world environments.

Main Takeaways

• Robot safety is a complex and nuanced concept that requires more than just collision avoidance.
• Traditional approaches to robot safety, such as robust control theory, are limited by their reliance on hand-designed state and dynamics representations.
• Generative AI offers a promising approach to learning latent representations of the world, which can be used for safe control.
• Combining safe control and generative modeling can lead to more generalizable and adaptable robot safety systems.
• Vision language models can be used to create context-dependent failure detectors, allowing robots to adapt their behavior based on the task at hand.