The New Bottleneck: AI That Codes Faster Than Humans Can Review

Key Concepts

• Generative AI
• Code Review Process
• Pull Requests
• Large Language Models (LLMs)
• Reasoning Models
• Contextual Technologies (RAG)
• Agentic Systems
• Code Graph Analysis
• Evals (Evaluation Metrics)
• Hallucination
• Reliable Agents
• CI/CD Pipelines
• Static Analysis Tools
• Rule-Based Systems

Code Rabbit and the Problem of Code Review

Haraj Gil, CEO and Founder of Code Rabbit, discusses the company's focus on automating the code review process using generative AI. The core problem Code Rabbit addresses is the bottleneck in software development caused by slow and often inconsistent human code reviews.

• Historical Context: Code review has always been a critical step, especially in companies like Google, to ensure code quality, security, and consistency. Issues found early in the code review process are significantly less expensive to fix than those discovered in production.
• The Bottleneck: Code review requires senior talent to understand the implications of code changes on the larger system. In larger companies (50-100+ engineers), this process can take weeks, leading to delays in shipping software.
• AI Code Generation Worsens the Problem: With the rise of AI code generation (e.g., using tools like Cursor), developers are increasingly becoming reviewers of AI-generated code, further exacerbating the bottleneck. Developers often don't fully understand the code generated by AI, making review even more critical.

Limitations of Traditional Automated Code Review

Traditional automated code review relies on rule-based systems (e.g., linters, static analysis tools like SonarQube, Semgrep).

• Rule-Based Systems: These systems are difficult to configure and maintain due to high false positive rates and the need to constantly update rules and clean up the existing codebase.
• Lack of Context: Rule-based systems don't understand the intent of the code change or its implications on the overall architecture. Human reviewers bring tribal knowledge and consider best practices beyond just syntax and coding standards.

Code Rabbit's Approach: Leveraging LLMs and Context

Code Rabbit uses large language models (LLMs), particularly reasoning models, to automate code review.

• LLMs as Infrastructure: LLMs are viewed as infrastructure, similar to serverless computing. The key differentiator is the context and workflow built around them.
• Context is Key: Code Rabbit focuses on providing LLMs with extensive context, including:
  • Code changes (diffs)
  • Entire files
  • Code graph (relationships between files)
  • Learnings from past user interactions with Code Rabbit
• Proactive and Reactive Context: Code Rabbit proactively provides context but also gives the AI "agency" to find additional information when needed.
• Pull Request Integration: Code Rabbit integrates into the pull request workflow on platforms like GitHub, GitLab, Azure DevOps, and Bitbucket. It analyzes the code change and provides human-like feedback in the form of review comments.

The Sandbox Environment

Code Rabbit creates a sandbox environment to provide the AI with a realistic context for code review.

• Repository Cloning: The entire repository is cloned into a sandbox environment.
• Dependency Installation: Necessary dependencies are installed to allow the AI to understand and potentially run the software.
• AI Access: The AI is given access to the sandbox with CLI tools (e.g., cat, grep, rip grep) to navigate the code, read files, and search for information, mimicking how a human reviewer would investigate the code.

Learning from Open Source and User Behavior

Code Rabbit initially offered its product for free to open-source projects to gather user behavior data and improve its models.

• Prompt Engineering: User insights helped refine prompts and guide the AI in the right direction.
• Example: Package Updates: When a package.json file is updated, the AI proactively checks for security vulnerabilities in the new packages using GitHub's security vulnerability database.

Addressing Hallucination and Model Selection

LLMs can hallucinate (generate incorrect or nonsensical information). Code Rabbit addresses this through:

• Contextual Awareness: Providing rich context reduces the likelihood of hallucination.
• Multiple Models: Using a combination of models for different tasks (summarization, context gathering, reasoning, code generation).
• Custom Evals: Developing internal evaluation metrics (evals) tailored to Code Rabbit's specific use case, as external benchmarks often prove unreliable in production.
• Iterative Tuning: Continuously tuning models and contextual technologies based on real-world usage and feedback.

Contextual Technologies and the Evolution of AI

The speaker discusses the evolution of contextual technologies and the challenges of building reliable AI agents.

• Emerging Abstractions: The right abstractions for AI are still emerging, similar to how Kubernetes evolved for container orchestration.
• Custom Use Cases: Existing AI platforms may not be suitable for highly customized use cases like Code Rabbit.
• RAG Maturity: Retrieval-augmented generation (RAG) technologies are maturing, but building reliable agents remains a challenge.
• Compounding Errors in Agents: Errors compound when chaining multiple agents together, making it difficult to maintain accuracy and reliability. A 5% error rate in the first agent can lead to a 10% error rate in the next step.
• Cascading Failures: Debugging agent chains requires tracing errors back to their source, similar to debugging cascading failures in complex systems.

Conclusion

Code Rabbit is tackling the critical bottleneck of code review by leveraging the power of LLMs and providing them with rich context. The company's approach involves creating a sandbox environment, learning from user behavior, and continuously tuning its models and contextual technologies. While challenges remain, particularly in building reliable AI agents, Code Rabbit is at the forefront of automating and improving the software development process.