[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

Key Concepts

• Evals: Evaluation of AI model outputs, crucial for training and improvement.
• Scoring System: A comprehensive set of metrics used to evaluate AI model outputs, breaking down subjective assessments into objective signals.
• Metrics: Individual, measurable aspects of AI model outputs used in a scoring system.
• Calibration: Aligning metrics with human judgment or user data to ensure they accurately reflect desired outcomes.
• Vibe Testing: Initial, informal testing of AI model outputs to get a general sense of performance.
• Synthetic Data: Artificially generated data used for training and evaluating AI models, especially for novelty or edge cases.
• LLM as a Judge: Using large language models to automatically evaluate AI model outputs.
• Dimensions/Signals: Individual criteria or factors considered when evaluating AI model outputs.
• Co-pilot: An AI-powered tool to assist in building and iterating on scoring systems.

Main Topics and Key Points

Introduction to Evals and Challenges

• The session aims to provide methodologies and insights into building effective evaluation systems (evals) for AI models.
• Evals are crucial for providing feedback to AI agents, similar to training data for machine learning.
• Challenges in evals include:
  • Defining appropriate metrics.
  • Subjectivity and difficulty in determining the "correct" answer.
  • Labor-intensive and manual processes.
  • The need for custom evals tailored to specific use cases.
  • The difficulty of automating evals, as LLMs often struggle with judging tasks.
• The importance of evals is increasing, with validation potentially taking up 80% of feature development time.

Methodologies and Best Practices

• Benchmarking and Iteration: Continuously setting up benchmarks, exhausting them, and moving on to the next.
• Calibration: Calibrating metrics with human data and user data.
• Scoring System Approach:
  • Start with a few correlated signals (5-10) and build upon them over time.
  • Treat evals as a feedback loop, measuring and improving iteratively.
  • Evals are the primary place where domain knowledge lives.
• Layered Approach: Start with simple methods like vibe testing and tracing, then layer in more sophisticated techniques as the system scales.
• Online Reinforcement Learning: Generate multiple responses from the model, score them online, and select the best one.

The Scoring System in Detail

• A scoring system breaks down complex evaluations into a combination of simpler, objective signals.
• Individual signals are easy to understand and inspect.
• Signals are combined into a single score using a mathematical function.
• The scoring system allows for visibility over the application and finer-grained analysis.
• Examples of signals include SEO, document popularity, title scores, content quality, spam detection, and clickbaitiness.
• The goal is to measure as many relevant things as possible to reduce variance and increase precision.

Workshop Overview and Tools

• The workshop is hands-on, involving coding and experimentation.
• Participants will use a co-pilot tool to build a scoring system for a meeting summarizer application.
• The co-pilot helps generate examples, identify dimensions, and modify the scoring system.
• The scoring system can be integrated into Google Sheets for testing and analysis.
• A Google Colab notebook is provided for more advanced experimentation and automation.
• The Colab notebook includes:
  • Loading data from Hugging Face.
  • Running the scoring system on the data.
  • Comparing different models.
  • Comparing different prompts.
  • Implementing online reinforcement learning.

Key Arguments and Perspectives

• Evals are not just testing; they are the primary place where domain knowledge lives.
• Metrics that work are calibrated metrics, not necessarily good or bad metrics.
• The industry is moving towards more nuanced evals that capture the subtleties of applications.
• Evals should be run online whenever possible.

Notable Quotes

• "Evals are depending upon eval to provide feedback from the world to let them know whether or not they're getting it right."
• "Evals are actually the only place you're going to spend most of your time because that's where domain knowledge is going to live."
• "Metrics that work are not necessarily good metrics or bad metrics they're either calibrated metrics or unccalibrated metrics."

Technical Terms and Concepts

• Stochastic Application: An application whose output is not deterministic, meaning it can vary even with the same input.
• Decoder Models: A type of neural network architecture commonly used in language models.
• Temperature: A parameter that controls the randomness of the model's output. Higher temperatures lead to more diverse and creative outputs.
• Meta Prompts: Prompts designed to optimize or improve other prompts.
• DSPI: (Mentioned as an optimizer)
• Onslaught Integration: (Mentioned as a tool for reinforcement learning)
• PICORE: (Mentioned as a tool for integration)
• Generalized Additive Model (GAM): A statistical model that combines multiple signals or features to make predictions.
• Birectional Attention: A type of attention mechanism used in neural networks that allows the model to attend to both past and future context.
• Regression Head: A component of a neural network that predicts a continuous value.
• Auto Regressively: Generating tokens one at a time, where each token depends on the previous tokens.
• Embeddings: Vector representations of words or phrases that capture their semantic meaning.

Logical Connections

• The discussion starts with the challenges of evals and then moves to methodologies for addressing those challenges.
• The scoring system is presented as a way to break down complex evaluations into simpler, more manageable components.
• The workshop is designed to provide hands-on experience with building and using scoring systems.
• The Colab notebook builds upon the concepts introduced in the first part of the workshop, providing tools for more advanced experimentation.

Data, Research Findings, or Statistics

• Validation can take up 80% of feature development time.
• Google search uses around 300 signals for ranking.
• The scoring system can score 20 dimensions in sub-50 milliseconds.

Synthesis/Conclusion

The workshop focuses on providing practical methodologies and tools for building effective evaluation systems for AI models. The key takeaway is the importance of breaking down complex evaluations into simpler, objective signals and continuously calibrating those signals with human judgment or user data. By adopting a scoring system approach, developers can gain greater visibility over their applications, reduce variance, and improve the overall quality of their AI models. The hands-on exercises and tools provided in the workshop are designed to help participants implement these methodologies in their own projects.