Stanford AA228V I Validation of Safety Critical Systems I Guest Lecture: Anthony Corso, Terra AI

Key Concepts

• Falsification: Finding failures of systems.
• Adversary: An agent that introduces disturbances to a system to cause it to fail.
• Reinforcement Learning (RL): Training an agent to perform a task by receiving reward feedback for its actions.
• Sample Efficiency: Achieving good performance in RL with the fewest interactions with the environment.
• Adaptive Stress Testing (AST): Falsification with the objective of finding the most likely failure.
• Responsibility Sensitive Safety (RSS): A framework for codifying rules of safe driving into mathematical specifications.
• Deep Generative Models: Neural networks that learn to generate data similar to a given training set.
• Generative Adversarial Networks (GANs): A type of deep generative model consisting of a generator and a discriminator.
• Diffusion Models: A type of deep generative model that learns to reverse a diffusion process to generate data.
• POMDP (Partially Observable Markov Decision Process): A framework for decision-making under uncertainty where the agent's state is not fully observable.
• Surrogate Models: Simplified models that approximate the behavior of more complex systems.
• Induced Seismicity: Earthquakes caused by human activities, such as injecting fluids into the ground.

Reinforcement Learning for Falsification

• Setup: The system under test interacts with an adversary that introduces disturbances at each time step. The system provides a reward to the adversary, where higher rewards indicate the system is closer to failure.
• Key Insight: This setup is analogous to reinforcement learning, where the adversary is the agent, disturbances are the actions, and the reward reflects the system's proximity to failure.
• Benefits:
  • Leverages existing sample-efficient RL algorithms to find failures quickly.
  • Can generalize to different initial states, unlike tree search methods that start from a single initial state.
• Example: Training an RL agent to introduce disturbances in a continuum world, causing trajectories starting from various initial states to collide with an obstacle.
• Adaptive Stress Testing: Using MCTS or RL to search for the most likely failure of a system. This concept originated in the Stanford Intelligent Systems Laboratory (SISL) by Ritchie Lee.

Choosing a Falsification Method

• Domain Specificity: The effectiveness of falsification methods varies across different domains.
• Rarity of Failures:
  • If failures are common, direct sampling may be more efficient than complex methods like MCTS.
  • For rare failures, sample-efficient algorithms like tree search are beneficial.
• Simulator Requirements: The type of simulator limits the applicable falsification methods.
  • Black-box simulators: (Initial state + disturbance sequence -> trajectory) allow for direct sampling, fuzzing, and population methods.
  • Simulators allowing step-by-step interaction: (Initial state + disturbance -> next state) are required for reinforcement learning and tree search-based methods.
• First-order and Second-order Methods: These local descent methods require the gradient and Hessian of the system, limiting their applicability to systems where these can be computed. Direct methods (zero-order methods) do not require gradient information.

Adaptive Stress Testing in Transportation Systems (Anthony Corso)

• Motivation: Autonomous systems offer potential benefits like improved aviation safety and reduced driving accidents, but also pose risks, as demonstrated by real-world collisions involving autonomous vehicles.
• Challenges:
  • Specifying the Objective: Defining failure conditions can be complex, especially in scenarios involving nuanced traffic rules.
  • Modeling the Environment: Accurately modeling human behavior and sensor dynamics is challenging.
  • Optimization: Efficiently searching for failures in complex, long-time-horizon environments is difficult.
• Specifying the Objective (Example):
  • Initial attempt: Training an adversary to cause a vehicle to collide with a pedestrian resulted in the pedestrian intentionally running into the vehicle.
  • Improved approach: Using Responsibility Sensitive Safety (RSS) to define failures where the autonomous vehicle is at fault. This led to finding scenarios where biased sensor noise caused the vehicle to misinterpret the pedestrian's position.
• Modeling the Environment:
  • Using data-driven approaches to model human behavior and sensor dynamics.
  • Employing deep generative models like GANs to imitate real-world driving behavior and sensor noise.
• Optimization (Example: TaxiNet):
  • TaxiNet: An autonomous system for aircraft taxiing.
  • Goal: Find image errors that could cause the aircraft to deviate from the runway.
  • Method: Using neural network verification techniques to find the worst-case noise at each time step.
  • Finding: Applying the worst-case disturbance at every step did not lead to failure.
  • Improved Method: Using Monte Carlo tree search (MCTS) to find sequences of disturbances that lead to failure. This revealed that a combination of left and right biases could cause the aircraft to run off the runway.
• Diffusion Models for Failure Sampling (DiFS):
  • Using conditional diffusion models to sample diverse and likely failure modes.
  • Iteratively training a diffusion model on data generated from random samples, updating the risk threshold to focus on higher-risk areas.
  • Demonstrated on toy problems, inverted pendulum, and F-16 models.

Earth Resource Problems and Safety (Anthony Corso)

• Context: The subsurface is a key resource for the net-zero transition, providing raw materials, renewable energy (geothermal), and carbon storage.
• Carbon Storage: Injecting CO2 into saline aquifers for long-term storage.
• Challenges:
  • Uncertainty about subsurface geology.
  • Complex decision-making in uncertain environments.
• POMDP Formulation: Modeling carbon storage as a POMDP, where the state of the subsurface is partially observed, and actions involve injecting CO2 and taking measurements.
• Surrogate Models: Building fast surrogate models for CO2 migration using deep neural networks trained on data from physics-based simulators.
• Risks:
  • Induced Seismicity: Injecting fluids into the ground can reactivate faults and cause earthquakes.
  • CO2 Leakage: CO2 can leak back to the surface, potentially causing suffocation or negating the benefits of carbon storage.
• Safety Evaluation: Developing methods for safety evaluation of automated decision-making agents in Earth resource problems.

Conclusion

The lecture covers falsification techniques, particularly reinforcement learning and adaptive stress testing, for finding failures in complex systems. It emphasizes the importance of sample efficiency, accurate environment modeling, and appropriate objective specification. Anthony Corso's guest lecture extends these concepts to real-world transportation systems and emerging Earth resource problems, highlighting the challenges and potential risks associated with deploying autonomous systems in safety-critical domains. The use of advanced techniques like diffusion models and POMDPs offers promising avenues for improving safety evaluation and decision-making in these complex environments.