Stanford AA228V I Validation of Safety Critical Systems I Guest Lecture: Somil Bansal, Stanford

Key Concepts

• Vision-based controllers
• Safety-critical failures
• Reachability analysis (Hamilton-Jacobi reachability)
• Backward reachable tube (BRT)
• Stress testing
• Anomaly detection
• Incremental training
• Modular vs. end-to-end testing
• Perception monitors
• SPARQ (Safety Assessment and Repair Network)
• System-level vs. component-level safety

Introduction

Professor Somil Bansal discusses his lab's work on mining failures in vision-based controllers, emphasizing the importance of safety in AI-based autonomous systems. He frames safety as a continuous process integrated throughout the data cycle, from training to deployment, and focuses on stress testing as a method for identifying and mitigating safety-critical failures.

Safety as a Continuous Process

• Three Stages of Safety:
  • Training Time: Incorporating safety requirements programmatically into the training phase to ensure safety within the training data distribution.
  • Operation Time: Adapting safety measures to account for out-of-distribution scenarios encountered during deployment.
  • System Lifecycle: Stress testing systems to identify safety-critical failures and using these failures to improve system design.
• Continual Safety Assurance: This framework provides provisional safety during training, adapts during operation, and improves over the system's lifecycle.

Stress Testing Vision-Based Controllers

• Problem Formulation:
  • Given a system with dynamics (state x, control u) and a visual sensor providing observations (i), the goal is to find visual inputs that lead to safety violations.
  • The system includes a robot with dynamics, a visual sensor, a vision-based controller, and a simulator for stress testing.
• Reachability Problem:
  • Failure discovery is cast as a reachability problem by concatenating the visual observation function with the vision-based controller to obtain an equivalent state-based policy.
  • Existing reachability-based methods are used to compute the backward reachable tube (BRT).
• Backward Reachable Tube (BRT): The set of all initial states that will eventually steer the system into a predefined failure region under the given controller.

Hamilton-Jacobi Reachability Primer

• System Dynamics: The system has state x, control u, and disturbance d (representing modeling uncertainty or external disturbances).
• Backward Reachable Tube: The set of all initial states from which the system will be driven to an undesirable (failure) set, despite the best control effort.
• Safe Set: The converse of the BRT; if the system starts within this set, a policy exists to keep it inside the set at all times (control invariant set).
• Optimal Control Problem: The safety problem is converted into an optimal control problem.
• Implicit Representation of Failure Set: A function L(x) is used to represent the failure set, where L(x) is negative inside the failure set and positive outside. L(x) represents the safety reward at state x.
• Cumulative Reward/Cost: The minimum safety reward incurred along a trajectory. A negative cumulative reward indicates an unsafe trajectory.
• Value Function: Represents the closest the system will ever get to the failure set. Negative values indicate unsafe states (within the BRT).
• Bellman Backup: Dynamic programming is used to solve for the value function, resulting in a Bellman backup (partial differential equation in continuous time).
• Safety Controller: Derived from the value function, the controller attempts to perform gradient ascent in the value function, pushing the system towards safer states.

Case Study: Autonomous Aircraft Taxiing

• Problem: An aircraft uses a vision-based controller to taxi on a runway, avoiding steering off the runway (the failure set).
• Method: The BRT is computed, identifying initial conditions from which the aircraft will steer off the runway.
• Results:
  • Failure images are extracted from the failure region.
  • Analysis revealed that runway markings confused the CNN, causing it to steer the aircraft off course.
• Key Insight: Not all vision failures are equal; the focus is on vision failures that lead to system-level failures.
• Environmental Variations: The BRT can be computed as a function of different environmental factors (e.g., time of day, cloud conditions).

Case Study: Indoor Robot Navigation

• Problem: An indoor robot uses a ResNet-based vision controller to navigate hallways.
• Controller: Trained entirely in simulation using photorealistic simulators.
• Failure Mode: The CNN learned a spurious correlation between light-colored walls and traversability, leading to collisions when encountering light-colored walls and dark-colored floors.

Using Failure Data to Improve Controllers

• Runtime Anomaly Detection:
  • Train an anomaly detector (binary classifier) to predict whether an image is likely to cause a failure.
  • Trigger a fallback controller if a failure is predicted.
• Targeted Incremental Training:
  • Retrain the vision-based controller using the collected failure images.
  • This reduces the size of the BRT, indicating improved safety.

Challenges and Open Questions

• Generalization to New Environments: How to adapt stress testing and anomaly detection to environments not seen during training. Potential solutions include digital twins and vision models (NeRFs, Gaussian splatting).
• Monotonic Improvement: Ensuring that incremental training leads to consistent safety improvements. Neural networks do not guarantee monotonic improvement with added data.
• Identifying Underlying Failure Reasons: The framework identifies failure images but does not provide the underlying reasons for the failures. Manual analysis is currently required.
• Modular vs. End-to-End Testing:
  • End-to-end testing is easier to implement but harder to interpret and use for improving individual components.
  • Modular testing allows for targeted improvements but requires defining failure criteria for each component.
• System-Level vs. Component-Level Testing:
  • Component-level testing (e.g., adversarial robustness) may not correlate with system-level safety.
  • The focus should be on component failures that lead to system-level failures.

SPARQ: Safety Assessment and Repair Network

• Collaboration with NVIDIA: Studying system-level safety given perception failures in autonomous cars.
• Perception Monitor: Provides a probability distribution of missed agent positions.
• SPARQ Network: Takes monitor input, ego plan, and scene information to assess the safety of the plan and suggest repairs.
• Runtime Operation: Designed to run online due to low latency requirements.
• Supervised Learning: Trained offline using simulated data with different perception uncertainties.
• Output: Predicts whether a plan is safe, risky, or critical, and proposes a candidate safe plan.

Conclusion

Stress testing is a crucial mechanism for validating and improving autonomous systems. Control theory provides valuable tools for stress testing, and data-driven reachability methods can efficiently stress test vision-based controllers. Open questions remain regarding generalization, monotonic improvement, identifying failure reasons, and the trade-offs between modular and end-to-end testing. The speaker emphasizes the importance of considering system-level safety and developing methods for propagating uncertainty through complex autonomous systems.