Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Key Concepts

Software agents, bug detection, bug fixing, software maintenance, SDLC (Software Development Life Cycle), LLMs (Large Language Models), benchmarks, false positives, true positives, code review, program comprehension, cross-file reasoning, bug pattern recognition, thinking models, needle in a haystack, PR review (Pull Request Review).

Benchmark for Software Agents in Bug Detection and Fixing

Introduction

Bismouth, led by Ian (CEO) and Nick (CTO), has developed a benchmark to evaluate the effectiveness of software agents in finding and fixing bugs, particularly in software maintenance tasks. This benchmark aims to address the gap in existing benchmarks that primarily focus on feature development and testing.

The Software Development Life Cycle (SDLC) and Benchmarking Gaps

The SDLC includes:

• Initial Scoping and Planning: Requires business context and knowledge of existing systems.
• Feature Development and Testing: Already covered by existing benchmarks.
• Code Review: Largely unbenchmarked.
• Deployment: Requires configuration, monitoring setup, and integration.
• Software Maintenance: Bug fixes, dependency upgrades, migrations. This is the focus of Bismouth's benchmark.

The Importance of Software Maintenance

Maintenance requires deep reasoning through a system to identify potential bugs. Agents often struggle with holistic evaluation, finding only subsets of bugs per run. Thinking models help, but reasoning remains narrow. While agents can often patch simple bugs, their ability to identify complex bugs is limited.

Limitations of Existing Bug Detection Benchmarks

Existing benchmarks, primarily from the software security space, are designed for classic static analysis tools and have limitations:

• Focus on simplistic bugs (null pointer dereferences, buffer overflows, SQL injection).
• Limited language support (often only Java).
• Bias towards security issues, neglecting other types of bugs (e.g., copy-paste bugs).

Bismouth's SM100 Benchmark

SM100 is a benchmark consisting of 100 triaged, validated, and classified bugs from 84+ public repositories. The goal is to provide a range of issue types, from obvious to requiring senior staff-level engineering knowledge.

Key Features:

• Multi-Language: Python, TypeScript/JavaScript, and Go.
• Objective Bugs: Explicit security or logical issues causing data loss or system crashes. Excludes feature requests, optimizations, formatting, and design decisions.
• Metadata Annotation: Each bug is annotated with:
  • Severity
  • Context
  • System-specific domain knowledge required
  • Difficulty to find
  • Implication of the bug (data loss, crash, security exploit)

Benchmark Metrics

Four key metrics are used to evaluate agent performance:

1. Needle in a Haystack: Can the system discover the bug without prior knowledge?
2. False Positive Rate: Measures the accuracy of reported bugs.
3. Bug Detection at Introduction: Can the agent identify the bug at the time of the pull request or commit?
4. Remediation Suggestion: Can the agent fix the bug without breaking the codebase?

Methodology for "Needle in a Haystack"

To avoid biasing the agents, repositories are divided into subsystems containing interrelated files. The benchmark filters subsystems containing files modified in the "golden" PR/commit and feeds those files to the LLM. This reduces the search space without hinting at the bug's location.

Building an Effective Agent

Basic agent implementations (shell tool + search/replace loop) are insufficient. Building a good agent requires:

• Model selection
• System design
• Prompting strategies
• Information feeding methods
• Navigation strategy

Performance Comparison

Bismouth leads in "needle in a haystack" detection, finding 10 bugs compared to the next leading solution's 7. Codec leads in true positive rate and detection. Devon and Cursor agents find a lot of issues, but with a low true positive rate.

• Needle in a Haystack: Bismouth found 10, the next leading solution found 7.
• True Positive Rate: Claude Code at 16%, Bismouth at 25%, Codeex at 45%.
• PR Review: Codeex found 27% of needles, Devon at 21%, Bismouth at 17%.

Bismouth's solution, running on top of Anthropic models, outperformed Anthropic's own solution (Claude Code) in multiple categories.

Open Source Model Performance

Open-source models (R1, Llama Maverick) have a long way to go, with low true positive rates (1-2%). Sonnet 4 and 03 performed better in a loop, finding 6 and 2 bugs, respectively.

The State of the Industry

The most frequently used agents score poorly (7% true positive rate) on SM100, indicating a problem in the industry's focus. Some agents generate excessive reports (e.g., 70 reports for one issue), making them impractical for engineers. Even simple bugs are often missed.

Narrow Thinking in Agents

Agents exhibit narrow thinking, even with thinking models. They don't evaluate files holistically, and the bugs they find change between runs. Broader and deeper thinking chains are needed.

Implications for Software Management

Despite high scores on simpler tasks, agents struggle with SM100, suggesting they may struggle with managing and fixing software after deployment. This requires targeted search, better program comprehension, cross-file reasoning, and bug pattern recognition.

Conclusion

The most frequently used agents have a high risk of introducing bugs. Newer agents show increased ability to reason through code. Stability is nascent, but improvements are encouraging. The SM100 benchmark can track progress in the field. Bismouth is at the forefront of this new generation of agents.