Stanford AA228V I Validation of Safety Critical Systems I Runtime Monitoring

Key Concepts

• Offline Validation: Validation methods performed before system deployment.
• Runtime Monitoring: Flagging hazardous situations as they occur to trigger fallback mechanisms.
• Operational Design Domain (ODD): The set of conditions under which a system is designed to operate safely.
• Out-of-Distribution Detection: Identifying data points that fall outside the expected distribution.
• Uncertainty Quantification: Understanding and quantifying uncertainty in system behavior.
• Output Uncertainty (Aleatoric Uncertainty): Uncertainty inherent in the data due to randomness or noise.
• Model Uncertainty (Epistemic Uncertainty): Uncertainty due to limitations of the model, such as lack of data.
• Super Level Set: All points where a function's value is greater than a certain threshold.
• Curse of Dimensionality: The exponential increase in volume that data must cover as dimensionality increases.
• Feature Collapse: The phenomenon where distinct sets of images collapse to the same region in a lower-dimensional space.
• Bayesian Model Averaging: Averaging predictions over a distribution of possible models.
• Ensemble of Models: A collection of models used to approximate Bayesian model averaging.
• Temperature Scaling: A calibration technique for softmax classification models.
• Prediction Sets: Sets of possible outputs with a specified coverage probability.
• Conformal Prediction: A method for generating accurate prediction sets without calibration.
• Failure Monitoring: Monitoring for potential failures even within the operational design domain.
• Test-Time Augmentation (TTA): Augmenting input data at test time to improve model robustness.
• Sensitivity Analysis: Assessing how changes in input data affect model outputs.
• Safety Case: A structured argument, supported by evidence, that a system is acceptably safe.

1. Introduction: The Need for Runtime Monitoring

• Traditional validation methods are "offline," performed before deployment, and rely on models of the world.
• Models are inherently incomplete and cannot capture all possible scenarios, especially in complex environments.
• Examples: Waymo encountering a random fountain, Tesla following a truck transporting traffic lights.
• Runtime monitoring is presented as the solution to catch what offline validation misses.
• The goal is to flag hazardous situations in real-time and trigger fallback mechanisms (safe mode, human operator).

2. Plan for Runtime Monitoring

• Operational Design Domain (ODD) Monitoring
• Uncertainty Quantification
  • Output Uncertainty
  • Model Uncertainty
• Failure Monitoring

3. Operational Design Domain (ODD) Monitoring

• Definition: The set of conditions under which a system is designed to operate safely.
• Operating outside the ODD invalidates validation results and safety guarantees.
• Goal: Flag situations where the system is operating outside its ODD.
• Representing the ODD:
  • Hand-Designed Features: Defining the ODD using explicit conditions (e.g., daytime, no clouds, taxiway A). Requires domain knowledge and can be difficult to enumerate all conditions.
  • Data-Driven Approach: Defining the ODD based on data used for validation (e.g., trajectories, images).
• Data-Driven ODD Monitoring with Continuum World Example:
  • Using states from successful rollouts as data points defining the ODD.
  • Nearest Neighbors Approach: Defining the ODD as the set of points whose nearest neighbor in the data set is within a threshold distance.
    • Varying the threshold affects conservativeness.
    • Requiring multiple neighbors within the threshold can eliminate outliers.
  • Drawbacks of Nearest Neighbors:
    • Computational cost of storing the entire data set in memory.
    • Potential solutions: Clustering data and using cluster centers, using a more compact set representation like a polytope.
  • Polytope Representation:
    • Using the convex hull of the data to define the ODD.
    • Issue: ODDs are often non-convex.
    • Solution: Clustering data and taking the union of convex hulls of each cluster.
  • Super Level Set of a Distribution:
    • Fitting a distribution (e.g., multivariate Gaussian, mixture of Gaussians) to the data.
    • Defining the ODD as the super level set of the probability density.
    • Requires selecting a model class expressive enough to fit the data.
  • Super Level Set of a Trained Model:
    • Training a classifier to output the likelihood of a point being in the ODD.
    • Thresholding the probability to define the ODD.
    • Drawback: Requires data from outside the ODD, which may not be available.

4. High-Dimensional Data and the Curse of Dimensionality

• ODD monitoring works well in low dimensions but becomes challenging in high dimensions (e.g., images).
• Curse of Dimensionality: The volume of space that data must cover increases exponentially with dimension.
• Distance metrics lose meaning in high-dimensional spaces.
• Solutions:
  • Brute force: Get more data and better models (e.g., normalizing flows).
  • Dimensionality Reduction: Assume data lies on a lower-dimensional manifold and define the ODD in that manifold.
    • Example: Using an autoencoder to project taxi images into a 2D space.
    • Feature Collapse: The risk that distinct sets of images collapse to the same region in the lower-dimensional space.

5. Uncertainty Quantification

• Goal: Understand uncertainty in system behavior to flag potentially hazardous situations.
• Two Types of Uncertainty:
  • Output Uncertainty (Aleatoric Uncertainty): Inherent randomness or noise in the data. Irreducible.
  • Model Uncertainty (Epistemic Uncertainty): Limitations of the model due to lack of data or knowledge. Reducible.
• Output Uncertainty Quantification:
  • Training a model to output parameters of a distribution over its prediction (e.g., mean and variance).
  • Deriving a loss function that includes both mean squared error and variance terms.
  • For classification models, softmax outputs a discrete distribution.
  • Calibration Issues: Modern neural networks are often not well-calibrated.
    • Reason: Training is approximate, data is limited, and models have finite capacity.
  • Temperature Scaling: A calibration technique for softmax classification models.
    • Adding a parameter (lambda or temperature T) to adjust softmax probabilities.
    • Picking a value for lambda that minimizes the negative log likelihood on a calibration set.
  • Prediction Sets: Outputting a set of possible outputs with a specified coverage probability.
    • Example: "I am 95% confident that I am here plus or minus 2 meters."
    • Creating prediction sets by taking the 95% probability mass of the output distribution.
    • Conformal Prediction: Generating accurate prediction sets even with uncalibrated uncertainty measures.

6. Model Uncertainty Quantification

• Model uncertainty is due to a lack of data, so output uncertainty methods may not be calibrated.
• Bayesian Model Averaging: Maintaining a distribution over all possible models.
  • Averaging predictions over all models to get the output.
  • Computationally intractable in general.
• Approximate Bayesian Model Averaging with Ensembles:
  • Creating an ensemble of models with high likelihood.
  • Averaging predictions over the models in the ensemble.
  • Training models with different initializations to reach different local minima.
  • Problem: Models may collapse to the same local minima.
  • Solution: Ensure sufficient diversity in the models (e.g., randomized prior functions, different architectures).

7. Failure Monitoring

• Even within the operational design domain, failures can still occur.
• Goal: Monitor for potential failures even within the ODD.
• Methods:
  • Running offline validation algorithms online (e.g., reachability analysis, rollouts).
  • Storing the probability of failure for each state offline and monitoring for high-risk states online.

8. Course Wrap-up and Conclusion

• No silver bullet in safety validation.
• Need to build a safety case using multiple methods.
• Swiss cheese model: Holes in individual methods should not align.
• Many open research questions in validation.
• Encouragement to stay in touch and continue exploring validation techniques.