What Data from 20 Million Pull Requests Reveal About AI Transformation in the Wild — Nich, Jellyfish

Key Concepts

• AI Transformation: The process of integrating Artificial Intelligence into existing companies or founding new AI-native organizations.
• AI Coding Tools: Software applications that assist developers in writing code, such as Copilot, Cursor, and Claude Code.
• Autonomous Coding Agents: AI systems designed to perform coding tasks with minimal human intervention, like Devon and Codeex.
• Developer Adoption Rate: The percentage of time developers spend using AI tools while coding.
• PR Throughput: The number of pull requests an average engineer merges per week, a key productivity metric.
• Cycle Time: The latency or lead time for code to go from the first commit to being deployed.
• PR Size: The net lines of code added in a pull request.
• Code Architecture: The organizational structure of code across repositories, including monorepos vs. polyrepos, and monolithic vs. microservices.
• Active Repos per Engineer: A metric indicating the number of distinct repositories an engineer contributes to weekly, reflecting code distribution.
• Context Engineering: The effort required to provide AI tools with the necessary understanding of code relationships across multiple repositories.

AI Transformation: Real-World Data Insights

Nicholas Arcolano, Head of Research at Jellyfish, presents data-backed insights into the current state of AI transformation in software engineering. The discussion addresses common questions regarding AI adoption, productivity gains, side effects, and troubleshooting when expected results are not achieved.

1. Understanding AI Adoption

1.1. Lines of Code as a Metric: While commonly cited in media, using "lines of code" as a primary metric for AI adoption is deemed insufficient. Data shows a steady increase in companies generating 50% or more of their code with AI, growing from approximately 2% in summer last year to nearly half by the present.

1.2. Developer Adoption Rate: A more insightful metric is the developer adoption rate, defined as the fraction of time developers use AI tools while coding.

• Definition: 100% adoption means using AI tools every time a developer codes. A company's adoption rate is the average of individual developer rates.
• Data Trends:
  • Median company adoption rates have risen significantly from around 22% last summer to nearly 90% currently.
  • Despite high adoption among some, technical, organizational, and cultural barriers prevent universal 100% adoption for many teams.

1.3. Autonomous Coding Agents: The adoption of fully autonomous coding agents (e.g., Devon, Codeex) is still in its nascent stages.

• Current Status: Only about 44% of companies have engaged with autonomous agents in the past three months.
• Usage: The majority of this engagement is experimental, accounting for less than 2% of merged pull requests.

2. Productivity Gains from AI

Despite the early stage of autonomous agents, significant productivity gains are being observed from interactive AI coding tools.

2.1. Defining Productivity Metrics:

• PR Throughput: The average number of pull requests merged per engineer per week. This metric is considered proven and widely accepted, though its absolute level can vary based on work scoping and architecture. Measuring changes in PR throughput is key for tracking gains.
• Cycle Time: The latency from the first commit in a pull request to its merge. This measures how quickly code gets deployed.

2.2. Observed Productivity Improvements:

• PR Throughput: A clear correlation exists between AI adoption and PR throughput. Companies with 100% AI adoption can expect, on average, a 2x increase in PR throughput compared to those not using AI.
• Cycle Time: A 24% decrease in median cycle times is observed as AI adoption moves from 0% to 100%.

3. Side Effects of AI Transformation

AI transformation brings about several changes beyond productivity.

3.1. PR Size:

• Trend: Teams with full AI adoption are pushing 18% larger PRs in terms of net lines of code added.
• Composition: This increase is primarily due to net additions rather than deletions, indicating more new code being introduced.
• File Touched: The average number of files touched per PR remains similar, suggesting that the increased size is due to more thorough or verbose code within existing files, not broader code modifications across the codebase.

3.2. Quality Impact:

• Current Findings: Currently, no statistically significant relationship has been found between AI adoption rates and increased bug creation or PR reverts.
• Bug Resolution: Interestingly, there's an observed increase in bug resolution rates. This is attributed to teams disproportionately using AI to tackle existing bug tickets in their backlog, as bugs are often well-scoped and verifiable tasks suitable for AI.
• Future Outlook: While no "smoking gun" on quality issues exists yet, continued monitoring is planned, especially with the growth of asynchronous agents.

4. Troubleshooting AI Transformation: When Results Lag

If AI transformation is not delivering expected results, even with high adoption, several factors can be at play.

4.1. The Role of Code Architecture: The organization of code across repositories (code architecture) significantly impacts AI productivity gains.

• Metric: Active repos per engineer is a key metric to understand architecture. It measures how many distinct repositories an engineer contributes to weekly and is scale-independent.
• Architectural Regimes:
  • Centralized: Low active repos per engineer.
  • Balanced: Moderate active repos per engineer.
  • Distributed: Higher active repos per engineer.
  • Highly Distributed: Very high active repos per engineer.

4.2. Architecture and Productivity Correlation:

• Centralized & Balanced Architectures: These show significantly higher PR throughput gains, trending towards 4x with AI adoption, exceeding the average 2x.
• Distributed Architectures: These align more closely with the global average of 2x PR throughput gains.
• Highly Distributed Architectures: These exhibit little to no correlation between AI adoption and PR throughput. In some cases, a slightly negative trend is observed.

4.3. Challenges in Highly Distributed Architectures:

• Context Limitations: Most current AI tools are designed to work with a single repository. Combining context across multiple repositories is challenging for both humans and AI.
• Undocumented Relationships: The relationships between repositories and their associated systems are often not clearly documented and are frequently held within the knowledge of senior engineers, making them inaccessible to AI tools.
• Context Engineering: Significant investment in "context engineering" is required to overcome these challenges.

4.4. Microservices and Future Potential: While microservices are often touted as ideal for AI-native development, current context challenges in highly distributed architectures hinder AI gains. However, with advancements in context engineering and autonomous agents, this landscape could shift, potentially making highly distributed architectures the most productive.

4.5. Metric Nuances: The absolute number of PRs can increase in highly distributed architectures due to coordination overhead. This highlights why tracking the change in PR throughput is more reliable for measuring productivity than absolute counts.

Conclusion and Key Takeaways

• AI Coding Tools are Widely Adopted: Interactive AI coding tools are seeing significant usage.
• Autonomous Agents are Emerging: Adoption of fully autonomous agents is still in its early stages.
• Productivity Gains are Real: Expect significant productivity gains, including a 2x increase in PR throughput and a 24% decrease in cycle time, even with interactive AI tools.
• Larger PRs are Expected: Be prepared for larger PR sizes as a side effect.
• Quality Concerns are Not Yet Evident: Current data does not show significant negative impacts on code quality.
• Code Architecture Matters: If productivity gains are not materializing, investigate your code architecture. Highly distributed architectures, without proper context engineering, can limit AI's effectiveness.
• Focus on Adoption and Context: Prioritize driving developer adoption of AI tools and invest in solving context challenges, especially in distributed environments, to unlock full AI productivity potential.