Stanford AA228V I Validation of Safety Critical Systems I Property Specification 2

Key Concepts

• Composite Metrics: Combining multiple metrics into a single value for decision-making.
• Pareto Optimality: A state where no metric can be improved without worsening another.
• Pareto Frontier: The set of all Pareto optimal solutions.
• Weighted Sum Method: A composite metric where each metric is assigned a weight, and the weighted sum is calculated.
• Goal Distance: A composite metric that measures the distance to a desired goal (e.g., Utopia point).
• Weighted Exponential Sum Metric: A composite metric that combines distance and weights.
• Pairwise Queries: Asking users to choose between two options to infer their preferences.
• Half Space: A region of space defined by an inequality.
• Active Learning: Selecting queries strategically to maximize information gain.
• RLHF (Reinforcement Learning from Human Feedback): A technique used in large language models that involves learning from human preferences.
• Logical Specification: Formally defining system requirements using logical formulas.
• Propositional Logic: A system of logic based on propositions (statements that are either true or false).
• Atomic Propositions: Propositions that cannot be broken down further.
• Logical Operators: Symbols used to combine or modify propositions (e.g., NOT, AND, OR, IMPLIES, BICONDITIONAL).
• First-Order Logic: An extension of propositional logic that includes variables, predicates, and quantifiers.
• Variables: Objects in a domain.
• Predicates: Functions that evaluate propositions over variables.
• Quantifiers: Symbols used to express the quantity of variables that satisfy a proposition (e.g., FOR ALL, THERE EXISTS).
• Temporal Logic: An extension of first-order logic that specifies properties over time.
• Linear Temporal Logic (LTL): Specifies properties over linear sequences of states.
• Temporal Operators (LTL): Symbols used to express temporal relationships (e.g., ALWAYS, EVENTUALLY, UNTIL).
• Signal Temporal Logic (STL): Extends LTL to real-valued signals and allows specifying properties over time intervals.
• Signal: A function that maps time to a real value.
• Robustness: A measure of how well a signal satisfies a temporal logic formula.
• Smooth Robustness: A modified robustness measure that uses softmin and softmax operators to enable gradient-based optimization.
• Softmin/Softmax: Smooth approximations of the min/max functions.

Composite Metrics and Pareto Optimality

The discussion begins with a review of composite metrics and Pareto optimality.

• Pareto Optimality: A design is Pareto optimal if improving one metric necessitates worsening another.
• Pareto Frontier: The set of all Pareto optimal designs.
• Composite Metrics: Used to select a single design from the Pareto frontier.

Three types of composite metrics are reviewed:

1. Weighted Sum Method: Assigns weights to each metric, reflecting their relative importance. The design with the highest weighted sum is chosen.
   • Example: Alert rate weight = 0.8, Collision rate weight = 0.2.
2. Goal Distance: Selects the design closest to a defined goal (e.g., the Utopia point of zero alerts and zero collisions).
3. Weighted Exponential Sum Metric: Combines distance and weights.

Eliciting Weights Using Pairwise Queries

The lecture then focuses on how to determine the weights for the weighted sum method.

• Challenge: Directly asking experts for weights can be difficult.
• Solution: Use pairwise queries, asking experts to choose between two designs with different metric values.
• Methodology:
  1. Present two designs (A and B) with different values for each metric.
  2. Ask the expert which design they prefer.
  3. Infer a constraint on the weight vector based on the expert's choice.
  4. Repeat with multiple pairwise queries to narrow down the possible weight vectors.

Candy Preference Example:

• Goal: Learn a weight vector for M&Ms (W1), Sour Patch Kids (W2), and Skittles (W3), where W1 + W2 + W3 = 1.
• Process:
  1. Present the volunteer with two bags of candy (A and B) with different quantities of each candy type.
  2. The volunteer chooses their preferred bag.
  3. Based on the choice, an inequality is formed: W1 * A1 + W2 * A2 + W3 * A3 > W1 * B1 + W2 * B2 + W3 * B3 (where A1, A2, A3 are the quantities of each candy in bag A, and similarly for bag B).
  4. This inequality defines a half space in the W1-W2 plane (since W3 is determined by W1 and W2).
  5. Repeat steps 1-4 with different bag combinations to create multiple half spaces, progressively narrowing down the possible weight vectors.
  6. Once the space is sufficiently narrowed, choose a weight vector within the remaining region (e.g., the center point).
  7. Validate the inferred weight vector by presenting a final choice and comparing the predicted preference with the volunteer's actual choice.

Inconsistent Preferences:

• If the pairwise queries result in an empty space of possible weights, it indicates inconsistent preferences.
• Solutions:
  • Relax the assumption of perfect rationality.
  • Model human choice probabilistically, using a distribution over weights.
  • Use Bayesian estimation techniques.

Connection to RLHF:

• This process is related to Reinforcement Learning from Human Feedback (RLHF) used in large language models.

Logical Specifications

The lecture transitions to logical specifications, which formally define operating requirements for a system.

Propositional Logic

• Definition: A system of logic based on propositions, which are statements that evaluate to either true or false.
• Atomic Propositions: Propositions that cannot be broken down further.
• Operators:
  • NOT (¬): Negation (reverses the truth value).
  • AND (∧): Conjunction (true only if both operands are true).
  • OR (∨): Disjunction (true if either or both operands are true).
  • IMPLIES (→): Implication (if P is true, then Q must be true).
  • BICONDITIONAL (↔): True if and only if (true only if both operands have the same truth value).

Example:

• Statement: "If the agent is in a safe state, then the agent is not in a collision state."
• Atomic Propositions:
  • S: The agent is in a safe state.
  • C: The agent is in a collision state.
• Logical Formula: S → ¬C

First-Order Logic

• Definition: An extension of propositional logic that includes variables, predicates, and quantifiers.
• Variables: Objects in a domain (e.g., the state of an agent).
• Predicates: Functions that evaluate propositions over variables (e.g., P(x) returns true if the agent is in a safe state x).
• Quantifiers:
  • Universal Quantifier (∀): "For all" (all variables in the domain must satisfy the proposition).
  • Existential Quantifier (∃): "There exists" (at least one variable in the domain must satisfy the proposition).

Example (Gridworld):

• Goal: Agent must avoid obstacles and reach a goal.
• Predicates:
  • O(x): True if x is an obstacle state.
  • G(x): True if x is a goal state.
• Logical Formula: (∀x, ¬O(x)) ∧ (∃x, G(x))

Temporal Logic

• Definition: Extends first-order logic to specify properties over time.
• Linear Temporal Logic (LTL): Specifies properties over linear sequences of states.
• Temporal Operators:
  • ALWAYS (□): The proposition must be true at all time steps in the future.
  • EVENTUALLY (◊): The proposition must be true at some time step in the future.
  • UNTIL (U): Q must be true at some time step in the future, and P must be true at least until Q becomes true.

Example (Gridworld with Checkpoint):

• Goal: Agent must reach the goal after passing through a checkpoint and always avoid the obstacle.
• Predicates:
  • F(s(t)): True if the state at time t contains the obstacle.
  • G(s(t)): True if the state at time t is the goal.
  • C(s(t)): True if the state at time t is the checkpoint.
• Logical Formula: (◊G) ∧ (¬G U C) ∧ (□¬F)

Signal Temporal Logic (STL)

• Definition: Extends LTL to real-valued signals.
• Key Features:
  • Specifies properties over time intervals (e.g., EVENTUALLY between A and B).
  • Maps real-valued signals to truth values using predicates of the form μc(s), where μ is a function applied to the state s, and the result is compared to a constant c.

Example (Aircraft Collision Avoidance):

• Goal: Maintain sufficient vertical separation between aircraft between 40 and 41 seconds before potential collision.
• Predicate: S: The absolute value of the relative altitude between the aircraft is greater than or equal to 50 meters.
• Property: ALWAYS (between 40 and 41 seconds), S.

Robustness in Signal Temporal Logic

• Concept: A measure of how well a signal satisfies a temporal logic formula.
• Interpretation:
  • Positive robustness: The trajectory is a success. Higher values indicate greater success.
  • Negative robustness: The trajectory is a failure. More negative values indicate a greater failure.
• Calculation (Simple Proposition): For a proposition μc(s(t)), the robustness is calculated as μ(s(t)) - c.
• Extending Robustness to Operators:
  • NOT P: Robustness(¬P) = -Robustness(P)
  • P AND Q: Robustness(P ∧ Q) = min(Robustness(P), Robustness(Q))
  • P OR Q: Robustness(P ∨ Q) = max(Robustness(P), Robustness(Q))
  • P IMPLIES Q: (Formula provided, derivation in the book)
• Temporal Operators:
  • EVENTUALLY P: Robustness(◊P) = max(Robustness(P(t)) for all t)
  • ALWAYS P: Robustness(□P) = min(Robustness(P(t)) for all t)
  • UNTIL: (Formula provided)

Smooth Robustness

• Problem: The min and max operators in the robustness calculations are not differentiable, hindering gradient-based optimization.
• Solution: Replace min and max with softmin and softmax operators.
• Softmin/Softmax: Smooth approximations of the min/max functions, controlled by a parameter w.
  • w = 0: Approximates the true min/max.
  • w = infinity: Approximates the mean.
• Benefits: Enables gradient-based optimization by providing a smooth, differentiable robustness measure.

Conclusion

The lecture provides a comprehensive overview of metrics, logical specifications, and robustness in signal temporal logic. It covers various techniques for defining system requirements, evaluating system performance, and enabling gradient-based optimization. The concepts and techniques discussed are essential for the remainder of the course.