Stanford AA228V I Validation of Safety Critical Systems I Explainability

Key Concepts

• Collision Avoidance: Techniques and systems designed to prevent aircraft from colliding.
• ACAS X: An airborne collision avoidance system.
• Bayesian Network: A probabilistic graphical model representing dependencies between variables.
• Near Midair Collision (NMAC): A situation where two aircraft come dangerously close to colliding.
• Importance Sampling: A variance reduction technique used in Monte Carlo methods to estimate rare event probabilities.
• Explainability: The ability to understand and explain the behavior of a system, especially a complex one like a machine learning model.
• Policy Visualization: Techniques for visualizing the decision-making process of an agent.
• Feature Importance: Methods for determining the contribution of different input features to the output of a system.
• Sensitivity Analysis: A method for assessing how changes in input features affect the output of a system.
• Saliency Maps: Visual representations that highlight the most important input features for a given prediction.
• Integrated Gradients: A technique for computing feature importance that addresses the issue of saturated features.
• Shapley Values: A concept from game theory used to fairly distribute the contribution of each feature to the outcome of a system.
• Surrogate Models: Simplified models used to approximate the behavior of more complex models, making them easier to understand.
• Counterfactuals: Hypothetical scenarios that explore what would have happened if different actions had been taken.
• Failure Mode Characterization: The process of identifying and categorizing the different ways in which a system can fail.
• Clustering Algorithms: Unsupervised machine learning algorithms used to group similar data points together.
• Temporal Logic: A formal language for specifying and reasoning about the behavior of systems over time.

Collision Avoidance and Importance Sampling

• ACAS X Example: The ACAS X airborne collision avoidance system uses a model of the airspace trained on nine months of radar data across the continental United States. This model is represented as a Bayesian network.
• Simulating Encounters: Aircraft encounters are sampled from the Bayesian network to simulate two aircraft flying towards each other. This helps understand if the system prevents near midair collisions (NMACs).
• Probability of NMAC: The goal is to compute the probability of an NMAC given an encounter scenario.
• Inefficiency of Direct Sampling: Directly sampling encounters from the Bayesian network is inefficient because NMACs are rare events.
• Importance Sampling for Efficiency: Importance sampling is used to bias the distribution after sampling from the Bayesian network, focusing on encounters with smaller vertical and horizontal miss distances. This provides more interesting behavior and a better estimate of the probability of NMAC.
• Importance Reweighting: Importance reweighting is applied to get an accurate estimate of the probability of NMAC.

Explainability: Introduction

• Beyond "It Works": The discussion shifts from simply validating that a system works to understanding why it works.
• Importance of Understanding "Why": Knowing why a system works is crucial to avoid being fooled by seemingly correct behavior that might fail under different metrics or scenarios.
• Increased Confidence: Explaining the behavior of a system increases confidence when deploying it in the real world.
• Topics Covered: The lecture covers policy visualization, feature importance, surrogate models, counterfactuals, and failure mode characterization.

Policy Visualization

• Rolling Out the System: Visualizing rollouts of the system to observe its behavior.
  • Example: Collision Avoidance: Observing modes of behavior, such as going above or below the intruder.
  • Example: Inverted Pendulum: Observing the pendulum staying upright most of the time and occasionally falling over.
• Plotting the Policy: Plotting the policy of the agent to sanity check its behavior.
  • Example: Inverted Pendulum: Plotting torque as a function of angle (theta) and angular velocity (omega). Verifying that the policy applies zero torque when the pendulum is upright and applies corrective torque when it deviates.
  • Example: Collision Avoidance: Plotting action (climb, descend, no advisory) as a function of time to collision and relative altitude.
• Policy Slices: Plotting slices of the state space for high-dimensional policies.
  • Example: Collision Avoidance: Plotting action as a function of time to collision and relative altitude, revealing unintuitive features like a "notch" where no advisory is given.
• Partitioning the State Space: Partitioning the state space into regions and tracking the actions taken in each region during rollouts.
  • Advantage: Works for non-Markovian systems (history-dependent).

Feature Importance: Sensitivity Analysis

• Goal: Understand the contribution of various features (state, disturbances, pixels) to the overall behavior of a system.
• Examples:
  • Aircraft Steering: Identifying the most important pixels in an image of a runway for determining the steering angle.
  • Inverted Pendulum: Identifying the most influential disturbances in a trajectory for the robustness of the system.
• Two Approaches: Sensitivity analysis and Shapley values.
• Sensitivity Analysis: Understanding how a particular output changes when a single feature is changed.
  • Example: Perturbing a disturbance at a specific point in an inverted pendulum trajectory and observing the resulting spread of trajectories.
• Computing Sensitivity:
  1. Resample a disturbance from its nominal distribution.
  2. Simulate the trajectory with the resampled disturbance.
  3. Compute the spread (e.g., standard deviation) of the resulting trajectories or robustness values.
  4. Repeat for each disturbance in the trajectory.
• Observation: Disturbances earlier in the trajectory tend to have a greater impact on the final outcome.
• Taxi Problem Example:
  • Passing an image of a runway through a neural network to output a steering angle.
  • Resampling each pixel individually and observing the change in steering angle.
  • Plotting the relative change in steering angle on top of the image to identify the most sensitive pixels.
• Computational Cost: Perturbing each pixel individually is computationally expensive.
• Saliency Maps: Using gradients to identify inputs that are most important in determining a particular outcome.
  • Idea: Inputs with higher gradients have a greater impact on the output.
  • Process: Take the gradient of the robustness of the trajectory with respect to each disturbance.
  • Advantage: Reduces computational cost compared to sensitivity analysis.
• Comparison: Saliency maps provide a similar but slightly different explanation compared to sensitivity analysis.
• Saturated Features: Saliency maps may miss saturated features where the gradient is small even though the feature is important.
• Integrated Gradients: A technique to address saturated features by starting at a baseline point and gradually moving to the current point, averaging the gradients along the way.
  • Process:
    1. Start at a baseline point (e.g., a black image).
    2. Gradually move to the current point (e.g., the actual image).
    3. Average the gradients along the path.
• Taxi Image Example: Starting from a black image and gradually increasing the brightness to the actual image, observing the gradients for different pixels.

Feature Importance: Cautions and Shapley Values

• Word of Warning: Be careful about the information drawn from saliency maps.
• Sanity Checks: Some saliency methods are independent of the model and the data generating process.
• Edge Detection Example: Saliency maps may simply highlight edges in an image, regardless of the model being used.
• Randomized Models: Saliency maps may remain similar even when the model is replaced with a completely random network.
• Importance of Critical Evaluation: Ensure that explanations are faithful and not simply artifacts of the method.
• Shapley Values: Capture interactions between features, unlike sensitivity analysis which considers only one feature at a time.
• Wildfire Scenario Example:
  • Dividing a region into cells that are either burning or not burning.
  • Determining the probability of a specific cell (upper right corner) catching fire.
  • Demonstrating that sensitivity analysis fails to capture the importance of a cell that only contributes to the probability of fire when another neighboring cell is also burning.
• Shapley Value Calculation:
  1. Fix a random subset of cells (blue cells).
  2. Randomly sample the burning status of the remaining cells.
  3. Calculate the probability of the target cell burning.
  4. Repeat the process, but now fix the cell of interest (cell i) to its original value.
  5. Calculate the probability of the target cell burning again.
  6. Take the difference between the two probabilities.
  7. Repeat this process for all possible subsets of cells and average the results.
• Computational Challenge: The number of possible subsets grows exponentially with the number of features, making it computationally intractable to evaluate all subsets.
• Approximation: In practice, randomly sample a subset of possible subsets to estimate the Shapley values.
• Inverted Pendulum Example:
  • Calculating Shapley values for disturbances in an inverted pendulum trajectory.
  • Identifying the disturbances that have the greatest impact on the robustness of the trajectory.
  • Demonstrating that removing the disturbances with the highest Shapley values significantly improves the robustness of the trajectory.

Surrogate Models

• Use Case: Explaining the policies of agents, especially those with complex policies like neural networks.
• Process: Build a surrogate model to approximate the agent's policy.
• Desirable Properties:
  • High Fidelity: Accurately represents the policy it's modeling.
  • Interpretability: Easy to understand and draw conclusions from.
• Trade-off: There is often a trade-off between fidelity and interpretability.
• Linear Models:
  • Represent the output as a linear sum of weighted input features.
  • Interpretability: Weights indicate the relative importance of each feature.
• Local Linear Models:
  • Approximate the policy in a local region of the state space.
  • Example: Using LIME to create local linear models of the features.
• Collision Avoidance Example:
  • Creating a local linear model of the collision avoidance policy in a specific region of the state space.
  • Training the linear model on samples from that region.
  • Analyzing the weights to understand the importance of relative altitude and time to collision in that region.
• Adding More Features: Adding more features (e.g., squared terms, interaction terms) can improve fidelity but reduce interpretability.
• Decision Trees:
  • Another type of interpretable model.
  • Example: Training a decision tree to approximate the collision avoidance policy.
  • Interpretability: The structure of the tree reveals the decision-making process.
• Trade-off: More complex decision trees can achieve higher fidelity but are less interpretable.

Counterfactuals and Failure Mode Characterization

• Counterfactuals: Exploring what would have happened if different actions had been taken.
• Failure Mode Characterization: Identifying and categorizing the different ways in which a system can fail.
• Clustering Algorithms: Used to group similar failure trajectories together.
• K-Means Clustering: An iterative algorithm that finds distinct clusters in the data.
• Feature Selection: The results of clustering depend on the features used for clustering.
  • Examples: Average angle and angular velocity, entire state vector, action trajectories, disturbances.
• Temporal Logic Features: Clustering based on temporal logic features can provide more interpretable clusters.
• Parametric Signal Temporal Logic: Assigns parameters to temporal logic formulas, allowing clustering in the parameter space.

Conclusion

• Variety of Explainability Techniques: A wide range of techniques are available for explaining the behavior of systems.
• Potential for Contentiousness: The interpretation and validity of explanations can be subjective and debated.
• Limitations of Explainability: Some things in the world may be inherently difficult or impossible to fully explain.