Why building eval platforms is hard — Phil Hetzel, Braintrust

Key Concepts

• Evals (Evaluations): The process of testing and measuring the performance of LLM-based agents before deployment.
• Observability: Monitoring agent performance in production to ensure it continues to meet quality standards.
• Agent Quality Platform: A unified system combining offline evaluation and online observability to maintain agent reliability.
• Flywheel Effect: The iterative loop of using production trace data to inform offline evaluations, which then leads to improved agent performance.
• Vibe Coding: A colloquial term for building bespoke, quick-and-dirty UI/UX solutions for internal tooling.
• Trace Data: The logs of agent interactions, which in the LLM context are often large, semi-structured, and high-velocity.
• Unknown Unknowns: Unforeseen failure modes or usage patterns in production that require automated discovery (e.g., via topic modeling).

---

1. The Importance of Evals

LLMs are inherently variable, which is their greatest strength but also their greatest risk. As agents become the standard interface for customer interactions, companies face significant risks—brand damage, compliance failures, and high maintenance costs—if these agents are not rigorously tested. Evals provide the necessary confidence to move from a Proof of Concept (PoC) to a production-ready system.

2. Stages of Building an Eval Platform

Phil Hzel outlines a maturity model for organizations building internal evaluation systems:

• Stage 1: The Spreadsheet Approach:
  • Methodology: Using a spreadsheet (Excel/Google Sheets) and a simple script (for-loop) to run inputs through an agent.
  • Pros: Zero barrier to entry; easy to start.
  • Cons: Difficult to scale; lacks analytics; cumbersome to manage; not conducive to team collaboration.
• Stage 2: The "Vibe Coded" UI:
  • Methodology: Building a custom, bespoke UI and moving data into a proper database (e.g., Neon).
  • Pros: More approachable for non-technical stakeholders; better data persistence.
  • Cons: Still primarily a reporting tool rather than an experimentation engine; requires ongoing maintenance by the creator.
• Stage 3: The Experimentation Platform:
  • Methodology: Implementing a "playground" where users can tweak system instructions and parameters.
  • Key Feature: Running comparative evaluations across different agent configurations to bubble up performance scores.
• Stage 4: The Flywheel (Observability + Evals):
  • Methodology: Connecting production traffic directly to the evaluation environment.
  • Process: Capture production traces $\rightarrow$ Analyze in aggregate $\rightarrow$ Identify failure modes $\rightarrow$ Create offline test cases $\rightarrow$ Iterate.

3. The Systems Engineering Challenge

Building a robust eval platform is a "systems problem" rather than just a UI/UX task. Key technical challenges include:

• Data Volume and Structure: LLM traces are often massive (10–20 MB per span) and highly unstructured.
• Query Patterns: Systems must support both low-latency retrieval (for debugging individual traces) and high-throughput aggregation (for analytics).
• Storage Limitations: Traditional relational databases (like Postgres) struggle with the scale and text-heavy nature of LLM traces.
• Headless Requirements: Advanced users often prefer programmatic access (SDKs/APIs) over UIs to allow coding agents to self-heal or optimize prompts automatically.

4. Future Directions and Best Practices

• Automated Discovery: Moving beyond manual testing to using topic modeling to uncover "unknown unknowns" in production data.
• Agent-to-Agent Interaction: Designing platforms that can be queried and managed by other AI agents, not just humans.
• Centralized Governance: Implementing AI proxies or gateways to force tracing across all LLM calls, ensuring no agent goes unmonitored.
• Non-Functional Requirements: As platforms scale, they must incorporate Role-Based Access Control (RBAC) and data masking to ensure security and compliance.

5. Notable Quotes

• "Evals are a team sport... you want to make sure that you're bringing a ton of people into the fold, not just technical folks."
• "If you build it, you have to manage it... that's going to be your job now."
• "Measuring agent quality... it's actually a systems problem. It's not just a UI/UX problem."

Synthesis

The transition from simple spreadsheets to sophisticated observability platforms is essential for any organization serious about deploying LLM agents. The core takeaway is that evaluation and observability are two sides of the same coin. By creating a "flywheel" that feeds production data back into the testing environment, teams can move from reactive debugging to proactive quality assurance. However, this requires moving away from simple UI-focused tools toward robust data architectures capable of handling the high-velocity, unstructured nature of LLM trace data.