Automatic code reviews with OpenAI Codex

Key Concepts

• Codex Code Review: An AI-powered feature designed to assist in the code review process by identifying bugs and issues.
• GPT-5 and GPT-5 Codex: Advanced AI models trained for coding tasks, including bug detection and code verification.
• Human Verification Bottleneck: The challenge of human engineers being unable to keep pace with the rapid generation of code by AI.
• Iterative Deployment: OpenAI's philosophy of releasing technology incrementally and gathering real-world feedback.
• Pull Request (PR): A mechanism for submitting code changes for review and merging into a repository.
• Static Analysis: A method of analyzing code without executing it, typically to find syntax errors or potential issues.
• Repository Access: The ability of the AI model to access and understand the entire codebase, not just the changes in a diff.
• Hypothesis Testing: The AI model's ability to form hypotheses about code behavior and write code to test them.
• Precision Rate: A metric for evaluating the accuracy of AI-generated comments, aiming for fewer incorrect suggestions.
• Agent.md: An open format for coding agents, allowing for specific instructions and configurations.
• Steerability: The ability to guide the AI model's behavior through custom instructions or guidelines.
• Codex CLI: A command-line interface for interacting with Codex, enabling local code reviews.

Codex Code Review: Enhancing Engineering Workflows

This video details the capabilities and implementation of Codex's code review feature, highlighting its role in improving engineering team efficiency and code quality. The core premise is that Codex needs to integrate seamlessly with existing tools and team workflows to be an effective coding teammate. Code review is identified as a critical workflow where Codex can provide significant assistance.

1. The Need for AI in Code Review

Maya from OpenAI's alignment team explains that as AI capabilities grow, the volume of generated code increases, creating a bottleneck in human verification. To address this, OpenAI has trained powerful models, including GPT-5 and GPT-5 Codex, specifically for bug detection and code verification. The goal is to ensure that AI's verification abilities scale alongside its code generation capabilities.

2. Enabling and Using Codex Code Review

Enabling the Feature:

• Users must first go to their Codex web settings and enable the code review feature.
• Once enabled, Codex will automatically review any Pull Request (PR) submitted to a repository.

Triggering and Customizing Reviews:

• Automatic Review: PRs marked as "ready for review" are automatically picked up by the code review agents.
• Manual Triggering: Users can manually trigger a review by commenting @codex review this PR.
• Specific Instructions: Additional information or specific areas of focus can be provided in the comment to guide the AI. For example, a user might instruct Codex to "focus on the authentication logic."

3. How Codex Code Review Works: Beyond Static Analysis

A key differentiator of Codex code review is its ability to go beyond static analysis.

• Repository-Wide Context: The model has access to the entire repository, allowing it to track dependencies and understand code in a broader context. This is crucial for complex codebases where individual developers may not have complete knowledge.
• Hypothesis Testing and Execution: Codex can form hypotheses about code behavior, write Python code to test these hypotheses, and verify them with examples. This dynamic approach allows for deeper analysis than traditional static tools.
• Reviewing Logs: Users can review the logs of the AI's review process, understanding the steps it took, including any code it wrote to test its findings.

4. Training and Evaluation of Code Review Models

• Specific Bug-Catching Tasks: The models are trained with specific tasks that prioritize catching bugs that are significant and likely to be fixed by developers.
• High Precision Rate: A key objective during training was to achieve a high precision rate, meaning fewer incorrect comments or false positives. Evaluation against previous model generations showed a significantly lower incorrect comment rate.
• Real-World Evaluation: The most important evaluation is considered to be practical usage by developers, where the model finds real bugs without being overly intrusive with excessive comments.

5. Internal Use Cases at OpenAI

Codex code review has been used internally at OpenAI for some time, yielding valuable results:

• Preventing Critical Issues: It has helped prevent critical issues in training runs that could have delayed important model releases.
• Identifying Configuration Bugs: It has caught configuration errors that might not be apparent from just looking at the code diff.
• Facilitating Contributions to Unfamiliar Codebases: Developers can contribute more confidently to codebases they are not fully familiar with, relying on Codex for an initial review.
• Example: VS Code Extension Bug: Alex, a Codex PM, contributed to a VS Code extension. Codex identified an incorrect implementation, specifically the removal of a React prop with CSS, which was not easily detectable from the diff alone but required understanding of the broader code context.
• Fixing Identified Bugs: After Codex identifies a bug, users can ask it to trigger a new task to fix the issue, creating a seamless workflow.

6. The Vision: Codex as a Teammate

The overarching vision is for Codex to be an AI teammate available at any point and on any surface, capable of performing tasks.

• Augmenting Human Engineers: Codex can handle the time-consuming task of running through various scenarios and checking for bugs, freeing up human engineers to focus on higher-level problem-solving.

7. Agent.md and Steerability

• Agent.md Format: Codex supports the agent.md format, an open standard for coding agents, allowing for specific instructions.
• Customizable Guidelines: Users can add custom code review guidelines to their project's package.json or similar files. These guidelines can inform the model about specific requirements, areas to pay attention to, or issues to ignore.
• Style Customization: Users can even instruct Codex on its response style. For instance, one user requested that Codex reassure them they are still an amazing programmer even when pointing out bugs.

8. Codex Code Review in the CLI

• Local Reviews: Codex code review is now available in the Codex CLI, enabling users to review code directly in their terminal.
• Pre-Commit Reviews: This feature is ideal for reviewing code locally before it's committed to a repository or submitted as a PR.
• Executing Local Commands: The CLI version can execute more commands locally on the user's computer.
• Usage: Users can simply type /review to initiate a review of their current changes. This feature is expected to see significant improvements in the coming weeks.

9. Conclusion and Takeaways

Codex code review offers a powerful AI-powered coding agent that can be used:

• Locally: In the terminal via the CLI.
• In the Cloud: As a teammate to send tasks to.
• For Pre-Coworker Reviews: To review local changes or PRs on GitHub before human colleagues see them.

The primary benefits are catching bugs before they reach production, accelerating team velocity, and ultimately helping teams ship better and safer products.