Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford

Key Concepts

• AI-assisted coding
• Developer productivity
• Software engineering productivity measurement
• Greenfield vs. Brownfield tasks
• Task complexity
• Language popularity
• Codebase size
• Context window limitations
• Data-driven decision-making in software engineering

Introduction

The video discusses the impact of AI on developer productivity, addressing the initial hype and subsequent questions raised by Mark Zuckerberg's statement about replacing mid-level engineers with AI. It presents findings from a large-scale study conducted at Stanford, analyzing data from over 100,000 software engineers across 600+ companies. The study aims to provide a more nuanced understanding of AI's effects, moving beyond simplistic metrics like commits and PRs.

Limitations of Existing Studies

The speaker identifies three main limitations of existing studies on AI and developer productivity:

1. Reliance on Commits and PRs: Studies often focus on the number of commits or PRs completed, assuming more commits equal more productivity. However, task size varies, and AI can introduce new tasks like bug fixes, leading to a "spinning wheels" effect.
2. Greenfield Task Bias: Many studies evaluate AI on greenfield tasks (building something from scratch), where AI excels at boilerplate code generation. This doesn't reflect the reality of most software engineering, which involves working with existing codebases and dependencies.
3. Ineffective Surveys: Surveys asking developers to self-evaluate their productivity are unreliable. A small experiment showed little correlation between self-assessment and measured productivity, with people misjudging their productivity by about 30 percentile points. "Asking someone how productive they think they are is almost as good as flipping a coin."

Methodology

The Stanford study uses a model that plugs into Git and analyzes source code changes in each commit. It quantifies these changes based on dimensions like functionality, quality, and maintainability. This approach aims to measure the actual functionality delivered by the code over time, rather than just lines of code or commits. The model automates the process of expert code review, which is accurate but slow and expensive. The model correlates well with expert reviews, is fast, scalable, and affordable.

Results: Overall Impact of AI

• Initial Productivity Boost: Implementing AI initially leads to a perceived increase in productivity, with more code being written and more commits being pushed.
• Rework Increase: However, AI also introduces more rework (altering recently written code), indicating that some of the initial gains are offset by the need to fix AI-generated bugs or issues.
• Average Productivity Gain: The average productivity gain from using AI is roughly 15-20% across all industries and sectors, after accounting for rework. "With AI coding you generate or you increase your productivity by roughly 30 40%...which in turn gives you an average productivity gain...of roughly about 15 to 20%."

Results: Task Complexity and Project Maturity

The study reveals that AI's impact varies depending on task complexity and project maturity (greenfield vs. brownfield):

• Low Complexity Tasks: AI performs better on simpler tasks, leading to higher productivity gains.
• High Complexity Tasks: AI can sometimes decrease productivity on complex tasks.
• Greenfield Tasks: AI is more effective in greenfield projects, where it can generate boilerplate code and accelerate initial development.
• Brownfield Tasks: It's harder to leverage AI to increase productivity in brownfield projects with existing codebases and dependencies.

A matrix summarizes these findings:

• Low Complexity, Greenfield: 30-40% gains
• High Complexity, Greenfield: 10-15% gains
• Low Complexity, Brownfield: 15-20% gains
• High Complexity, Brownfield: 0-10% gains

Results: Language Popularity

AI's effectiveness also depends on the popularity of the programming language:

• Low Popularity Languages (e.g., Cobol, Haskell, Elixir): AI is not very helpful, even for low complexity tasks. In some cases, it can decrease productivity on complex tasks due to its poor performance in these languages.
• High Popularity Languages (e.g., Python, Java, JavaScript, TypeScript): AI provides gains of around 20% for low complexity tasks and 10-15% for high complexity tasks.

Results: Codebase Size and Context Length

The study suggests that the gains from AI decrease as the codebase size increases. This is attributed to:

1. Context Window Limitations: Large language models (LLMs) have limited context windows, meaning they can only process a certain amount of code at a time.
2. Signal-to-Noise Ratio: Larger codebases have more noise, making it harder for the model to understand the relevant context.
3. Dependencies and Domain-Specific Logic: Larger codebases have more dependencies and domain-specific logic, which AI models may struggle to grasp.

The speaker references a paper called "No Lima" which shows that LLM performance on coding tasks decreases as context length increases, even with large context windows like 32,000 tokens.

Conclusion

AI can increase developer productivity in many cases, but it's not a universal solution. Its effectiveness depends on factors like task complexity, codebase maturity, language popularity, codebase size, and context length. Companies should carefully consider these factors when implementing AI-assisted coding tools and avoid relying on simplistic metrics or surveys to measure their impact. "AI does increase developer productivity you should use AI for most cases but it doesn't increase the productivity of developers all the time."