Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Key Concepts

• Evals (Evaluations): Systematic methods to measure the performance of AI models and agents.
• Benchmark Maxing: The practice of optimizing models specifically to score high on standardized tests, often at the expense of real-world utility.
• Vibe Check: The subjective, qualitative assessment of how an AI model "feels" to use.
• Hill Climbing: An iterative process of improving an agent’s performance by measuring results on an eval, identifying failures, and making incremental adjustments.
• Agentic Flows: Workflows where AI agents perform multi-step tasks (e.g., searching docs, installing environments, running tests).
• Terminal Bench: A specialized benchmark suite consisting of 89 real-world coding tasks designed to test agentic capabilities.
• Harbor: An infrastructure framework used to run evaluation tasks in parallel within isolated environments.

---

1. The Critique of Current Evals

The speaker argues that the current approach to evaluations is "broken" due to two extremes:

• The Objective Metrics Camp: People who treat leaderboard numbers as absolute truth. The speaker warns that these are often approximations and can be manipulated through "benchmark maxing," where models are tuned to pass specific tests rather than perform well in real-world scenarios.
• The Taste/Vibes Camp: People who ignore data entirely, relying solely on subjective "vibes" or anthropomorphizing the model.

Key Heuristics for Interpreting Evals:

1. Don't trust the numbers blindly: Treat them as approximations.
2. Avoid the "Bleeding Edge": Don't be the first to switch models. Let new models "settle on fire" for a few weeks to see if they hold up to real-world usage.
3. Seek New, Precise Evals: Standardized benchmarks (like HumanEval) are often outdated and no longer measure frontier capabilities (e.g., they fail at basic tasks like matrix multiplication).

---

2. Building and Improving Agentic Evals

The speaker emphasizes that evaluating agents is significantly harder than evaluating single-turn LLM responses because agents operate in an "infinite search space."

The Process of Building Custom Evals:

• Data Sourcing: Use real-world user data (with consent) to identify actual problems users face.
• Environment Isolation: Use virtual machines or containers to ensure the agent has a clean, reproducible environment (e.g., proper Linux setup, RAM, and CPU).
• Parallelization: Use tools like Harbor to run multiple tasks (e.g., the 89 tasks in Terminal Bench) in parallel to reduce testing time.
• Failure Analysis: After a run, analyze the "traces" (logs of every LLM call) to categorize failures. This allows for targeted "hill climbing."

---

3. The Three Zones of Improvement

When hill climbing on an eval, the speaker identifies three distinct zones of optimization:

1. Zone 1 (Obvious Flaws): Fixing bugs in the harness, resolving rate-limiting issues, or fixing infrastructure crashes.
2. Zone 2 (Nuanced Improvements): The most critical phase. This involves prompt engineering specific to model families (e.g., Anthropic vs. Codex vs. Gemini) and adjusting agent behavior (e.g., tuning how much the model "thinks").
3. Zone 3 (The Danger Zone): Overfitting. This is where you "cheat" to get a high score for the sake of marketing or social media, which the speaker strongly advises against.

---

4. Notable Quotes

• "Evals are not the end-all-be-all. They're also not completely useless. There are right ways to use them and wrong ways to use them."
• "If you're solving stupid problems, it doesn't matter if you score 100% all the time."
• "You have to both pass the vibe check... and at the same time, you also have to have a very, very decent score."

---

5. Synthesis and Conclusion

The core takeaway is that developers should stop relying on public leaderboards and instead build custom, domain-specific evaluation suites. By treating evaluation as an engineering problem—using isolated environments, parallelized infrastructure, and rigorous failure analysis—developers can effectively "hill climb" their agents to achieve superior real-world performance. The ultimate goal is to balance quantitative rigor (the score) with qualitative excellence (the user experience).