Scaling code quality in the age of AI

Key Concepts

Scaling Code Quality in the Age of AI: The central theme of the presentation, focusing on how to maintain and improve code quality while leveraging the speed benefits of AI in software development.
Speed vs. Control (Racing Analogy): The analogy of Formula One racing is used to illustrate that true speed in software development requires control and quality, not just raw velocity.
GitHub Code Quality: A new product announced in public preview, designed to provide code quality intelligence, remediation at scale, and policy enforcement across enterprises.
AI and Static Analysis: The core technologies powering GitHub Code Quality, combining the generative capabilities of AI with the analytical rigor of static analysis.
Maintainability and Reliability: Two key pillars of code quality addressed by the current release of GitHub Code Quality.
Test Coverage and AI Era Challenges: Future areas of focus for GitHub Code Quality, acknowledging the evolving landscape of software development.
CodeQL: A powerful static analysis engine used by GitHub Code Quality for structured and deterministic findings.
LLM Detection: Leveraging Large Language Models for code quality analysis.
Auto Fix: Automated suggestions and implementations for code quality issues.
Copilot: GitHub's AI pair programmer, integrated with Code Quality for code generation, review, and remediation.
Technical Debt: Accumulation of code quality issues that can hinder future development and increase costs.
Developer Experience: The importance of integrating code quality tools seamlessly into the developer workflow.
Enterprise-Scale Rollout: Features designed for managing code quality across an entire organization.

Scaling Code Quality in the Age of AI

The Racing Analogy: Speed and Control

Marcelo Oliveira opens the session by drawing a parallel between Formula One racing and software development. He emphasizes that while speed is exhilarating, speed without control is dangerous. This is illustrated by the observation that elite drivers, like Formula One legend Alain Prost, achieve their fastest speeds when they appear smooth and in control, not when they look like they are pushing the limits. The key takeaway is that speed and control are not trade-offs but are essential and complementary.

Software Development as a Race Track

In software development, the "straightaways" represent periods of rapid code creation, often accelerated by AI. The "turns" are analogous to crucial processes like code reviews, where quality, security, and control are rigorously evaluated.

The AI Turbocharge and its Challenges

The presentation acknowledges that AI, exemplified by tools like GitHub Copilot, is significantly accelerating software development. Marcelo shares a firsthand example from GitHub: the use of Copilot enabled his team to increase the number of secret checkers shipped per quarter, delivering massive customer value that was previously impossible.

However, this acceleration introduces challenges. The core problem is the trap of viewing speed and quality as trade-offs.

Focusing too much on speed can lead to accumulating technical debt, bugs slipping into production, loss of customer trust, and ultimately, impact on revenue.
Hitting the brakes too hard by over-emphasizing quality checks can slow down development, break the flow, reduce productivity, and negate the speed gains from AI.

GitHub's Vision: Quality by Default

GitHub's vision is that every piece of code, whether AI-generated or developer-written, should be secure and high-quality by default, not an afterthought or a trade-off. This requires integrating quality into the platform and surfacing it throughout the developer experience. The goal is to achieve smooth, controlled, and therefore, truly fast software development.

Introducing GitHub Code Quality

Marcelo proudly announces the public preview of GitHub Code Quality. This new product aims to provide:

Code quality intelligence
Remediation at scale
Policy enforcement across the entire enterprise.

It leverages the combined power of AI and static analysis to ensure high standards for every team, repository, and line of code. The promise is to accelerate with AI with confidence, knowing that GitHub has your back.

Deep Dive into GitHub Code Quality

Carol, a Product Manager on Marcelo's team, takes the stage to provide a detailed look at the product.

Defining Code Quality

Carol addresses the question of what "code quality" means, noting its contextual nature. Based on internal discussions and customer feedback, they've identified four key categories:

Maintainability: How easy it is for developers to work with and extend the codebase over time.
Reliability: The likelihood of the code crashing or causing issues, especially in critical moments.
Test Coverage: The confidence that the code functions as intended.
AI Era Challenges: New quality considerations arising from AI-generated code.

The current release of GitHub Code Quality focuses on maintainability and reliability. Test coverage is coming soon, and AI era challenges are under active research.

Current Features (Public Preview)

The initial release offers:

One-click enablement at the repository level for quick adoption.
Dual action detection using CodeQL static analysis and LLM detection for a holistic view.
Auto-fix capabilities for both security and quality issues identified by GitHub.

Future Enhancements

Planned features to support platform teams and leaders include:

At-scale rollout with organization-level controls.
Trend reporting for test coverage from pull requests to the organization level.
Integration with APIs, self-hosted runners, and expected governance platform policies.

Rounding Out the Developer Experience

Carol emphasizes that code quality is not new. GitHub has always provided tools for developers. Copilot assists in writing high-quality code and reviewing pull requests. GitHub Code Quality complements this by providing visibility into technical debt and older code, helping teams target their quality initiatives effectively.

Live Demo: GitHub Code Quality in Action

Carol switches to a live demonstration, putting on a "developer hat" for a fictional company called "Rev Linux," which provides analytics for F1 racing.

Developer Workflow with Code Quality

Adding a New Feature: Carol demonstrates adding a "fuel usage panel" to their F1 analytics application.
Pull Request Analysis: Upon opening a pull request for this new feature, the GitHub Code Quality bot has already run.
Identifying Issues: The bot flags issues such as an unused variable and a duplicated function declaration.
Auto-Fix Suggestions: For each issue, the bot provides a recommendation and a direct way to commit the suggestion.
Policy Enforcement: The demo shows that merging is currently blocked due to rule sets associated with Code Quality. Rev Linux has a high bar, preventing merges with any outstanding issues.

Addressing Historical Code Quality

After merging the new feature, Carol demonstrates how to address existing technical debt:

AI Findings Page: Navigating to a new experimental AI findings page under the security tab, which surfaces findings from LLMs in recently changed files.
File with Legacy Debt: A file named "legacy quality debt" is shown to have 25 problems identified by Copilot, with descriptions and fixes provided.
Assigning to Copilot Coding Agent: The option to assign all fixes in a file to the Copilot Coding agent is presented, allowing the developer to focus on other tasks.
CodeQL Findings Page: Carol then shows the standard findings page, familiar to users of GitHub's security code scanning tool. This page displays CodeQL-related findings, tagged with reliability or maintainability and assigned severities (note, warning, error).
Score Visibility: The dashboard shows maintainability and reliability scores. Carol notes disappointment with a low reliability score.
Filtering for Reliability Issues: Using filters, Carol isolates reliability problems, specifically two error-level findings related to assigning values to constants.
Auto-Fix for CodeQL Findings: Auto-fixes are available for these CodeQL findings, with the ability to open a pull request directly. Bulk fixes and assignment to the Copilot Coding agent are mentioned as upcoming features for this page.
Impact of Fixes: After applying fixes (simulated with "demo magic"), the reliability score dramatically improves from "needs improvement" to "good."

Future Roadmap and Data Insights

Carol provides a sneak peek at an organization-level dashboard design, which will show side-by-side comparisons of repository scores for reliability, maintainability, and test coverage, enabling targeted support.

Private Preview Data

The tool has been in private preview with approximately 50 customers across over 300 repositories. This resulted in the identification of 88,000 quality issues, averaging 271 findings per repository.

Carol addresses the potential concern of developers ignoring these alerts by highlighting the power of assigning these issues to Copilot for automated remediation.

Conclusion and Call to Action

Marcelo returns to wrap up the session, reiterating what makes GitHub Code Quality unique:

Built into GitHub: Eliminates disconnected workflows and the need to leave the IDE.
AI and Static Analysis Synergy: Deeper issue detection, greater consistency, and remediation at scale, focusing on driving improvement rather than just finding noise.
Effortless Deployment: From a single repository to the entire enterprise, simplifying management and reducing total cost of ownership.

Call to Action

GitHub Code Quality is available today for free in public preview. Marcelo urges the audience to try it out and provide feedback, as it will directly influence the product's roadmap. He also invites individuals to become design partners for the Code Quality product.

The ultimate goal is to build the future of code quality together and ensure that speed and quality are never a trade-off again.

Scaling code quality in the age of AI

Key Concepts