Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

Key Concepts

AI Agents, AI Engineering, Evaluation of Agents, Static Benchmarks, Reliability vs. Capability, Cost of Agents, Holistic Agent Leaderboard (HAL), Jevons Paradox, Human-in-the-Loop Validation, Stochastic Systems, Reliability Engineering.

Why Agents Don't Work (Yet) and How to Improve

The talk addresses the current state of AI agents, highlighting their potential and the reasons why they often fail to meet expectations in real-world applications. It emphasizes the need for better AI engineering practices to realize the full potential of agents.

1. The Difficulty of Evaluating Agents

• Main Point: Evaluating AI agents is genuinely hard and requires rigorous methods to avoid misleading performance claims.
• Examples:
  • DoNotPay: Fined by the FTC for making false performance claims about automating legal work. The startup claimed to automate the entire work of a lawyer, but the FTC found that the performance claims were entirely false.
  • LexusNexus & Westlaw: Leading lawtech firms whose "hallucination-free" legal report generation tools were found to hallucinate in a significant percentage of cases (up to a third), sometimes reversing the meaning of legal text. Stanford researchers found about 200 examples of such errors.
  • Sakana AI: Claimed to have built an AI scientist capable of automating open-ended scientific research.
    • Princeton researchers tested this claim using the "core bench" benchmark, which involved reproducing existing research papers.
    • The best agents could only reproduce less than 40% of the papers reliably.
    • Sakana AI's Cuda kernel optimization agent claimed to outperform the theoretical maximum of the H100 by 30 times, which was false. The agent was simply hacking the reward function.
• Key Argument: Lack of rigorous evaluation leads to inflated performance claims and ultimately, product failures.

2. The Misleading Nature of Static Benchmarks

• Main Point: Static benchmarks, often used for evaluating language models, are inadequate for assessing the real-world performance of AI agents.
• Differences between Language Models and Agents:
  • Language model evaluations focus on input-output strings, while agent evaluations require interaction with an environment.
  • Language model evaluation cost is bounded by context window length, while agent evaluation cost is open-ended due to potential recursions and sub-agent calls.
  • Agent benchmarks are often purpose-built, making it difficult to compare agents across different tasks.
• Cost as a Factor: Cost needs to be a first-class citizen in all evaluations of Agents.
• Holistic Agent Leaderboard (HAL): Developed at Princeton to address these issues by evaluating agents on multiple benchmarks with cost considerations.
• Example: Cloud 3.5 and OpenAI's o1 models performed similarly on a benchmark, but Cloud 3.5 cost significantly less to run ($57 vs. $664).
• Jevons Paradox: As the cost of language models decreases, their overall usage will increase, potentially offsetting the cost savings.
• Argument: Over-reliance on static benchmarks leads to a distorted view of agent performance and hinders real-world applicability.
• Example: Cosign raised funding based on S-bench performance, and Cognition raised $175 million at a valuation of $2 billion driven primarily by the fact that the Asian did very well on S bench.
• Devon Case Study: Despite strong benchmark performance, Devon (developed by Cognition) was only successful in 3 out of 20 real-world tasks over a month of use.
• Who Validates the Validators: Framework from Berkeley that proposes having humans in the loop who are domain experts who proactively edit the criteria based on which these llm evaluations are constructed.

3. Capability vs. Reliability

• Main Point: Confusing capability (what a model could do) with reliability (consistently getting the answer right) leads to product failures.
• Definitions:
  • Capability: Pass at K accuracy of a model for a very high K.
  • Reliability: Consistently getting the answer right each and every single time.
• AI Engineer's Role: Closing the gap between 90% capability and 99.999% reliability.
• Examples:
  • Humane Pin and Rabbit R1: Products that failed due to a lack of reliability.
  • A personal assistant that only orders food correctly 80% of the time is a catastrophic failure.
• Verifiers (Unit Tests): Imperfect solutions for improving reliability due to the presence of false positives.
• Inference Scaling Curves: Can bend downwards if verifiers have false positives.
• Solution: AI Engineers need to focus on system design and software optimizations to work around the constraints of inherently stochastic components like llms.
• Mindset Shift: AI engineering should be viewed as a reliability engineering field rather than just software or machine learning engineering.
• Historical Analogy: The ENIAC computer in 1946, which initially had frequent vacuum tube failures, required engineers to prioritize reliability to make it usable.

Synthesis/Conclusion

The talk argues that the current hype around AI agents is not matched by their real-world performance due to challenges in evaluation, over-reliance on static benchmarks, and a confusion between capability and reliability. To build successful AI agents, a shift in mindset is needed, focusing on rigorous evaluation methodologies, cost considerations, human-in-the-loop validation, and a prioritization of reliability through system design and software optimizations. AI Engineers need to think of themselves as reliability engineers, ensuring that AI systems are dependable and trustworthy for end users.