How to close pull requests faster with Copilot code review | GitHub Checkout

Key Concepts

• GitHub Copilot Code Review: An AI assistant designed to streamline and enhance the code review process.
• Agentic Workflows: A system where an AI agent orchestrates various tools to perform a task, in this case, code review.
• LLM (Large Language Model): The core AI technology powering Copilot's understanding and generation capabilities.
• Deterministic Tools: Tools like linters (ESLint, PMD) and static analysis engines (CodeQL) that provide consistent and predictable results.
• Non-deterministic Tools: LLMs, which can produce varied outputs based on context and internal processing.
• Pull Request (PR): A mechanism for proposing changes to a codebase, which then undergoes review.
• PR Closure Rate: A metric indicating how quickly pull requests are merged.
• Mission Control: A new user experience interface for viewing the progress and actions of the Copilot code review agent.
• Stacked PR: A pull request created on top of another to address feedback or implement suggestions.
• Custom Instructions: Repository or organization-level markdown files that guide Copilot's review behavior.
• CodeQL: A powerful static analysis engine used for security and quality checks.
• Autovalidate: A component that integrates various linters into the review process.
• Public Preview: A phase where a feature is available for testing and feedback before general release.
• Opt-in Preview Features Policy: An enterprise-level setting that allows administrators to enable preview features for their teams.

GitHub Copilot Code Review: Enhancing Efficiency and Quality

This discussion focuses on GitHub Copilot's capabilities in code review, highlighting its role in increasing developer productivity and maintaining code quality. Elle, a member of the code review tools team, explains how Copilot code review automates repetitive tasks, provides context for reviewers, and ensures consistent review quality across teams.

Key Benefits and Performance Metrics

• Faster PR Closure: Businesses using Copilot code review have reported closing Pull Requests (PRs) 43% faster. This is achieved by automating parts of the review process and providing reviewers with essential context.
• Increased Developer Productivity: By handling routine checks, Copilot frees up developers to focus on more complex aspects of their work.
• Consistent Review Quality: Copilot helps standardize the review process, ensuring that all code receives a consistent level of scrutiny, regardless of the reviewer.

Demo and Workflow Explanation

Elle demonstrates the functionality by creating a simple GitHub action that summarizes changed files grouped by language. This action is conceptually similar to how Copilot code review triggers different analyzers.

1. Code Commit and PR Creation: Changes are committed, and a new branch is created, leading to the creation of a new pull request.
2. Automated Review Trigger: Upon PR creation, Copilot code review is automatically added to the review process.
3. Copilot Code Review Architecture: The review is powered by compute and GitHub Actions, utilizing a new workflow. The "agentic" nature of Copilot code review involves orchestrating multiple tools:
   • Agent: The primary LLM that identifies problems in the codebase.
   • Autovalidate: Integrates linters like ESLint and PMD.
   • CodeQL: A static analysis tool, now also included in the review process. These tools run in parallel to review the code and provide feedback directly on the PR.

Mission Control: Agent's Work in Progress

A new user experience, "Mission Control," allows users to view the session of the agent's work in progress. This interface shows the code review agent:

• Calling various tools to gather relevant context.
• Listing directories to look up code.
• Reading and understanding the codebase.
• Making a plan for the review.
• Storing comments.
• Ultimately posting the review feedback.

Review Output and Feedback Mechanisms

Copilot code review provides several key outputs on a pull request:

• PR Overview: A summary of key changes, providing context for reviewers. It includes an "Implement suggestion" button to hand off issue resolution to the Copilot coding agent.
• Specific Issue Identification: Copilot can identify critical mismatches that could break code, such as discrepancies between the action.yml file and its required inputs. It can also detect issues like missing "await" statements that could cause actions to freeze.
• Implementing Suggestions: Users can trust Copilot's suggestions and click "Implement suggestion." This action triggers the Copilot coding agent to create a new pull request based on the feedback, resolving the identified issues.
• Stacked PRs for Resolution: Copilot can create stacked PRs on top of existing reviews to resolve identified issues. This process deduplicates comments and resolves potential conflicts.

Enabling and Configuring Copilot Code Review

• Repository Rule Setup: The easiest way to enable Copilot code review is through repository settings.
  • Navigate to the "Copilot" section in settings.
  • Click on "Code review."
  • A quick button may be available to create a default rule if none exist.
• Rule Configuration: Within the rule settings, the option "Automatically request Copilot code review" can be selected.
• Automation Options: Further automations can be configured:
  • Reviewing based on new pushes to the PR.
  • Automatically reviewing draft PRs before they are exposed to the team.

Customizing Review Behavior

• Repository and Organization Custom Instructions: Users can customize Copilot's review process to align with their organization's standards.
  • This is achieved by leveraging copilot-instructions.md or the instructions.md directory.
  • These natural language markdown files can specify:
    • Focus on specific programming language practices.
    • Types of issues Copilot should never miss.
  • Copilot will consider these instructions during its review.

Integration with CodeQL

• Leveraging CodeQL: Copilot code review now utilizes CodeQL, a premier static analysis tool, for analysis.
• Benefit of Deterministic Tools: By integrating deterministic tools like CodeQL and Autovalidate, Copilot ensures consistent identification of basic and in-depth issues, reducing reliance solely on LLM's non-deterministic nature. This enhances review consistency.

User Feedback and Future Plans

• Overwhelmingly Positive Feedback: Users have reported increased retention and significant quality improvements, indicated by positive feedback on Copilot's comments (thumbs up/thumbs down).
• Future Development:
  • Continued addition of more tools to the LLM agents for more intelligent and in-depth reviews.
  • Expansion of the "code quality feature" to review entire repositories and identify quality improvements.
  • Building out the breadth of impact of Copilot code review.

Access and Enabling for Different User Tiers

• Copilot Pro and Copilot Pro Plus Users: Have Copilot code review enabled by default.
• Enterprise and Business Users: Enabled through a policy called "Opt into preview features." Enterprise administrators can select this policy to auto-enable the feature.

Conclusion

GitHub Copilot code review represents a significant advancement in developer tooling, offering a powerful AI assistant to enhance code review efficiency and quality. By automating repetitive tasks, providing crucial context, and integrating advanced analysis tools like CodeQL, Copilot empowers development teams to deliver higher-quality code faster. The ongoing development and user feedback loop promise further enhancements to this transformative feature.