Stanford CS329H: ML from Human Preferences | Autumn 2024 | Model-based Preference Optimization

Key Concepts

• Metric Elicitation: The process of identifying and quantifying the most relevant performance metrics for a specific task, particularly in classification problems, by actively learning stakeholder preferences.
• Cost-Sensitive Classification: A classification problem where different types of errors (false positives, false negatives) have different associated costs or consequences.
• Active Learning: A machine learning approach where the algorithm actively selects which data points to label, aiming to improve learning efficiency and reduce sample complexity.
• Utility Estimation: The process of building models to estimate the underlying utility or value that individuals or stakeholders assign to different outcomes or choices.
• Query Complexity: A measure of the number of queries or questions required to achieve a desired level of accuracy in a learning problem.
• Confusion Matrix: A table that summarizes the performance of a classification model by showing the counts of true positives, true negatives, false positives, and false negatives.
• ROC Curve: A graphical representation of the performance of a binary classification model, plotting the true positive rate against the false positive rate at various threshold settings.
• Pareto Frontier: The set of solutions in a multi-objective optimization problem that are non-dominated, meaning that no other solution can improve one objective without worsening another.
• Inverse Decision Theory: A framework for inferring the preferences or utility functions of decision-makers by observing their choices or actions.

Metric Elicitation: A Deep Dive

Introduction

The lecture focuses on metric elicitation, connecting it to previous discussions on active learning and future topics like mechanism design. The goal is to efficiently identify the most relevant performance metrics for classification problems, particularly cost-sensitive ones, by actively learning stakeholder preferences.

Review of Previous Lectures

• Preference Modeling: Choices are governed by underlying utility, and models can estimate these utilities.
• Preference Selection as Machine Learning: Preference choices (binary, rank, etc.) can be framed as classification problems.
• Active Learning for Preference Learning: Applying active learning to preference learning can improve efficiency, reducing the number of questions needed to learn utility functions.

Motivation: The Importance of Metric Selection

• Real-World Costs of Errors: Different kinds of errors in classification have different real-world costs (e.g., false positives in medical diagnosis).
• Asymmetric Costs: Errors are often asymmetric; false positives and false negatives have different consequences.
• Choosing the Right Metric: Selecting the appropriate metric is crucial for building models that align with the specific task and stakeholder preferences.

Example: Model Selection with Different Metrics

• Scenario: Choosing between three models (Nearest Neighbor, SVM) with different accuracy, false positive rate, and false negative rate.
• Decision Factors: The choice depends on the relative importance of overall accuracy versus the cost of specific error types.
• Context Matters: The optimal model depends on the application (e.g., minimizing false negatives in medical diagnosis, minimizing false positives in criminal justice).

The Problem of Metric Selection

• Performance Metric: A quantitative description of how errors are evaluated in a model.
• Choice Dependency: The choice of metric depends on relative costs and benefits, which can be quantified through weighted combinations of different error types.
• Exploding Metric Space: Even in binary classification, the space of possible metrics (weighted combinations) is infinite.

Case Studies: When Metrics Fail

• Netflix Prize: The competition used root mean square error (RMSE) as the metric, but this did not necessarily correlate with improving top-end ranking, which was the actual goal.
  • Lesson: Choosing the wrong metric can lead to algorithms that are not useful for the intended task.
• COMPAS Recidivism Prediction: The algorithm used a notion of fairness based on calibration, but ProPublica analyzed it using false positive and false negative rates, revealing disparities across demographic groups.
  • Lesson: Different metrics can lead to completely different conclusions about the fairness and effectiveness of an algorithm.
  • Impossibility Results: It's impossible to simultaneously optimize all fairness metrics; trade-offs are inevitable.

Metric Elicitation as a Solution

• Stakeholder Preferences: Use human stakeholders to select metrics that align with their preferences.
• Algorithmic Tool: Develop an algorithm that helps stakeholders efficiently select the metric that best aligns with their preferences.
• Utility Estimation and Active Learning: Metric elicitation combines utility estimation with active learning to efficiently identify the most relevant metrics.

Binary Classification Metric Elicitation

• Linear Binary Classification Metrics: Can be reduced to a weighted combination of false positive and false negative errors.
• Problem Formulation: The goal is to determine the relative weights (A1, A2) for false positive and false negative errors.
• Scale Invariance: The scale of the weights doesn't matter; only the relative proportions are important.

Algorithm: Noise-Free Setting

• Optimal Classifier: If the distribution is known, the optimal classifier chooses 1 if P(Y|X) > 0.5, and 0 otherwise.
• Weighted Problem: The optimal classifier for a weighted problem (A1 * False Positives + A2 * False Negatives) is a threshold classifier with a threshold (delta) that depends on A1 and A2.
• Confusion Matrix and Degrees of Freedom: Binary classification metrics can be described with two numbers (e.g., true positives, true negatives).
• ROC Curve and Pareto Frontier: The set of achievable confusion matrices forms a convex space, and the boundaries are given by thresholding the conditional probability.
• Linear Combinations: Weighted measures work out to linear combinations of the confusion matrix elements.
• Binary Search: Use binary search along the ROC curve to find the threshold that best matches stakeholder preferences.
  • Assumption: Requires a weak uniqueness assumption (unimodal preferences).
• Query Complexity: Binary search achieves epsilon accuracy with log(1/epsilon) steps.

Algorithm: Noisy Setting

• Probabilistic Bisection: Use a probabilistic version of binary search to handle noise and uncertainty.
• Prior and Posterior: Build a prior over possible threshold points and update it based on evidence from stakeholder responses.

Real-World Application

• Implemented the metric elicitation approach with stakeholders and showed that it better reflects their preferences than default approaches.

Key Takeaways

• Metric selection is crucial, even in simple classification problems.
• Active learning and preference learning can be combined to efficiently elicit stakeholder preferences for metrics.
• Exploiting the structure of classification problems can simplify the metric selection process.

Notable Quotes

• "Almost always and maybe as a strong claim sort of across most of learning uh choices of evaluation or utility functions can be as important as every other choice that you're making in your model design because they can completely change what you value what you think of as sort of a good result."
• "There is not a right answer but perhaps for particular stakeholders there's a right answer for them."

Technical Terms

• Sample Complexity: The number of samples required to achieve a desired level of accuracy in a learning problem.
• Utility Function: A function that represents the preferences or values of an individual or stakeholder.
• Loss Function: A function that quantifies the cost or error associated with a particular prediction or decision.
• Calibration: A property of a probabilistic model where the predicted probabilities accurately reflect the observed frequencies of events.

Logical Connections

• The lecture builds upon previous discussions of active learning and preference modeling, applying these concepts to the specific problem of metric elicitation.
• The case studies illustrate the importance of metric selection and the potential pitfalls of using inappropriate metrics.
• The algorithm for metric elicitation leverages the geometry of classification problems to efficiently search for the optimal metric.

Data, Research Findings, and Statistics

• The lecture mentions query complexity bounds for binary search, indicating the number of questions needed to achieve a desired level of accuracy.
• The case studies provide examples of real-world applications where metric selection had a significant impact on the outcomes.

Conclusion

The lecture provides a comprehensive overview of metric elicitation, highlighting its importance, challenges, and potential solutions. By actively learning stakeholder preferences and exploiting the structure of classification problems, it's possible to efficiently identify the most relevant metrics and build models that align with the specific needs and values of different stakeholders.