How to measure AI developer productivity in 2025 | Nicole Forsgren

Key Concepts

• Developer Experience (DevX): The overall experience of a developer when building software, encompassing friction, workflows, and support.
• Flow State: A mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity.
• Cognitive Load: The amount of mental effort required to perform a task.
• Feedback Loops: The process of receiving information about the results of an action, which can be used to adjust future actions.
• DORA Metrics: A set of four key metrics (Deployment Frequency, Lead Time for Changes, Mean Time to Restore, Change Failure Rate) used to assess the performance of software development teams.
• SPACE Framework: A framework for measuring developer experience that includes Satisfaction, Performance, Activity, Communication & Collaboration, and Efficiency & Flow.
• Hallucinations (AI): In the context of AI-generated code, this refers to instances where the AI produces incorrect, nonsensical, or fabricated output.
• Frictionless: A book and framework by Nicole Forsgren and Abby Notta focused on removing barriers to improve developer experience and accelerate competition in the age of AI.

Summary

This discussion with Nicole Forsgren, a leading expert in developer experience and author of "Frictionless," delves into the complexities of measuring productivity, particularly in the context of rapidly evolving AI tools. Forsgren argues that traditional productivity metrics are often misleading and that AI, while accelerating coding, introduces new challenges that can hinder actual developer speed and effectiveness.

The Flaws in Traditional Productivity Metrics

Forsgren begins by asserting that "most productivity metrics are a lie." She highlights how easily systems can be gamed, using "lines of code" as a prime example. If the goal is simply more lines of code, AI can be prompted to generate excessively verbose code, including comments, which inflates the metric without necessarily improving quality or efficiency. This can lead to increased complexity and technical debt. While lines of code might offer some downstream insights (e.g., code survivability, quality, impact on retraining AI models), they are fundamentally flawed as a direct productivity measure.

Developer Experience (DevX) and its Components

The conversation pivots to Developer Experience (DevX), defined as "what it's like to build software day-to-day for a developer." Forsgren emphasizes that poor DevX undermines even the best tools and processes. Within DevX, she identifies three reinforcing components:

1. Flow State: The immersive, enjoyable state of deep work. Forsgren notes that AI can both disrupt and potentially enhance flow. While AI-generated code and agent interactions can interrupt focused coding, senior developers are finding ways to leverage AI to maintain flow by offloading detailed tasks and focusing on higher-level goals.
2. Cognitive Load: The mental effort required for tasks. High cognitive load from dealing with complex systems or tedious mechanics leaves less mental space for innovation.
3. Feedback Loops: The speed and quality of information received about work. AI necessitates much faster feedback loops, extending beyond local builds and tests to encompass the entire development pipeline.

AI's Impact on Flow State and Workflows

Forsgren elaborates on how AI is changing the nature of coding. Instead of long, uninterrupted coding sessions, developers now engage in prompt-response cycles, code review, and integration. This can be interruptive. However, she describes advanced workflows where developers use AI agents to design entire systems in parallel, then step back to evaluate and refine. This shifts the developer's focus from line-by-line coding to strategic oversight and architectural design, potentially enabling them to enter a different kind of flow state. This also suggests a rethinking of work structures, where shorter, 45-minute blocks might become more effective due to AI's ability to help re-establish context and facilitate quicker dives back into tasks.

Measuring Productivity Gains from AI: What Companies Get Wrong

Companies often err by relying on outdated metrics like lines of code. Forsgren stresses that while DORA metrics (Deployment Frequency, Lead Time, Mean Time to Restore, Change Failure Rate) remain relevant for assessing pipeline speed and stability, they cannot be blindly applied in the AI era. AI introduces faster feedback loops that need to be leveraged differently. The SPACE framework (Satisfaction, Performance, Activity, Communication & Collaboration, Efficiency & Flow) is presented as more adaptable because it's a framework, not a prescriptive set of metrics. It allows for the inclusion of new dimensions like trust in AI-generated code, which is paramount given AI's non-deterministic nature and potential for hallucinations.

Rethinking Work Structure and the Role of Strategy

The shift towards AI-generated code means a significant portion of time will be spent reviewing rather than writing code. This presents an opportunity to rethink daily and weekly work structures. Forsgren highlights research suggesting humans have limited deep work capacity (around four hours per day). AI can help optimize this by handling more granular tasks, allowing developers to focus their deep work on higher-value activities. Crucially, she emphasizes that speed without strategy is futile. Companies can ship "trash faster every single day" if they don't have smart decisions about what to build. AI can accelerate prototyping and experimentation, but the strategy for what to build and test remains paramount.

The "Frictionless" Framework: Seven Steps to Improve DevX

Forsgren introduces her upcoming book, "Frictionless: Seven Steps to Remove Barriers Along Value and Outpace Your Competition in the Age of AI," co-authored with Abby Notta. The book aims to guide organizations in improving developer experience. The seven steps are:

1. Start the Journey: Conduct listening tours, synthesize learnings, and visualize current workflows.
2. Get a Quick Win: Start small with impactful projects and share successes.
3. Use Data to Optimize Work: Establish a data foundation, collect relevant data (including surveys), and analyze insights.
4. Decide Strategy and Priority: Use evaluation frameworks to determine the most critical areas for improvement.
5. Sell Your Strategy: Gain buy-in by communicating the rationale and benefits.
6. Drive Change at Your Scale: Implement changes effectively, whether with local or global scope of control.
7. Evaluate Progress and Show Value: Measure impact and loop back to refine the process.

The book also emphasizes practices like resourcing, change management, sustainable technology, and applying a product management lens to DevX.

Identifying Signs of Friction and Opportunities for Improvement

Forsgren identifies common "smells" indicating a team could move faster: consistently breaking builds, flaky tests, overly long or difficult processes for environment provisioning or task switching. She notes that most teams can move faster, but speed must be aligned with strategic goals. The most common improvements companies need to make are often process-related, not necessarily tool-driven. This includes simplifying cumbersome multi-step processes and establishing lightweight, effective workflows.

Measuring the Impact of AI and DevX Initiatives

When measuring the impact of AI tools and DevX improvements, Forsgren advises aligning with what leadership cares about most. This could be market share (focus on speed, e.g., idea-to-production time), profit margin (focus on cost savings, e.g., reduced cloud costs from efficient builds, vendor spend reduction), or transformation/disruption. She stresses the importance of clear communication and framing metrics in terms that resonate with leadership.

For those starting with measurement, Forsgren recommends:

• Talking to people: Direct interviews are crucial for understanding pain points.
• Surveys: These provide a quick, quantified overview of the landscape. She provides examples of effective survey questions that focus on satisfaction and barriers to productivity, prioritizing the top three issues and their frequency.
• Careful survey design: Emphasizing the need for clear, singular questions to avoid ambiguous data.

She distinguishes between happiness (broad and influenced by many factors) and satisfaction (specific to tools, work, and teams), advocating for measuring satisfaction as a more actionable metric.

Tools and Future Directions

Forsgren mentions popular AI tools like GitHub Copilot, Cursor, Gemini, and Claude Code, highlighting Claude Code's potential for non-coding use cases. She advocates for a product mindset in DevX initiatives, treating them as products with users, MVPs, feedback loops, and a go-to-market strategy. This includes continuously evaluating the relevance of metrics and sunsetting those that are no longer driving value.

Personal Use of AI and Life Motto

In her personal life, Forsgren uses AI tools like ChatGPT and Gemini for home design visualization, rendering images based on floor plans and desired aesthetics. She also shares a life motto: "Hindsight is 20/20, but it's also really dumb." This emphasizes giving grace to past decisions made with available information, rather than judging them solely by future knowledge.

New Role at Google

Forsgren is now the Senior Director of Developer Intelligence in Core Developer at Google, focusing on improving developer experience, productivity, and velocity across Google's properties. Her role involves measuring these improvements, adapting feedback loops, and driving impactful change within the organization.