AI powered entomology: Lessons from millions of AI code reviews — Tomas Reimers, Graphite

AI-Powered Entomology: Finding Bugs with AI

Key Concepts:

• AI-powered code review
• Large Language Models (LLMs) for bug detection
• Categorization of bugs (LLM-detectable vs. human-preferred)
• Measuring the success of AI code review (upvotes/downvotes, comment action rate)
• Diamond: Graphite's AI code review product

Introduction

Tomas, co-founder of Graphite, discusses using AI to find bugs in code, specifically with their product Diamond, an AI-powered code reviewer. The initial observation was that while AI was generating more code, it was also contributing to more bugs. The talk explores whether AI can be used to both create and detect bugs.

Early Experiments and Frustrations

Initial experiments using LLMs like Claude to find bugs in pull requests (PRs) showed promising results, such as identifying uninstantiated database OM classes and division by negative number errors in border radius calculations. However, these early successes were accompanied by frustrating experiences, including irrelevant or incorrect suggestions (e.g., suggesting reverting to previous code, misunderstanding CSS).

Categorizing Bugs: LLM Capability vs. Human Preference

The key insight was that there are different types of bugs and that LLMs have varying capabilities in detecting them. Furthermore, developers have different preferences regarding the types of comments they want to receive from an LLM versus a human.

• LLM Capability: LLMs can mimic code review comments but may not always be accurate or relevant.
• Human Preference: Developers may find certain comments (e.g., "add a comment," "extract logic") pedantic or annoying when generated by an LLM, but more acceptable when coming from a human.

The Bug Quadrant: A Framework for Categorization

Graphite analyzed 10,000 comments from their codebase and open-source projects, categorizing them based on LLM capability and human preference. This resulted in a quadrant-based framework:

• Top Right (Ideal): Bugs, accidentally committed code, performance/security concerns, documentation inconsistencies, stylistic changes. These are bugs that LLMs can reliably detect and that developers are generally receptive to receiving from an AI.
• Bottom Right (Human Domain): Tribal knowledge (e.g., "we used to do it this way because..."). LLMs currently struggle with these types of comments, which often rely on undocumented context and experience.
• Top Left (LLM Overreach): Code cleanliness and best practices (e.g., "comment this function," "add tests," "extract this logic"). While technically correct, these suggestions can be perceived as pedantic and unwelcome from an LLM.
• Bottom Left: (Not explicitly discussed, but implied to be areas where LLMs are not capable and humans don't want to receive)

Measuring Success: Upvotes, Downvotes, and Action Rate

To ensure the AI code reviewer stays within the "ideal" quadrant, Graphite implemented several metrics:

1. Upvotes/Downvotes: Users can react to comments with emojis, providing feedback on the LLM's accuracy and relevance. A spike in downvotes indicates potential hallucinations or overreach. Graphite reports a downvote rate of less than 4%.

2. Comment Action Rate: This measures the percentage of comments that lead to a code change in the PR. The rationale is that the purpose of a code review comment is to prompt action. Graphite found that only about 50% of human comments lead to changes within the PR itself.

   • Caveat: Not all comments require immediate action. Some are fixed in follow-up PRs, provide future guidance, or reflect preferential disagreements.

Achieving Human-Level Fidelity

By carefully prompting the LLM and focusing on comments that are both detectable and welcome, Graphite was able to achieve a comment action rate of 52% as of March, surpassing the average for human code reviewers.

Conclusion

The talk concludes that using LLMs to find bugs is effective when done thoughtfully. By categorizing bugs based on LLM capability and human preference, and by measuring success through metrics like upvotes/downvotes and comment action rate, it's possible to create an AI code reviewer that provides valuable and well-received feedback. Diamond, Graphite's product, embodies these principles.