From metrics to impact: Turn GitHub Copilot data into business value

Key Concepts

• AI Metrics Journey: A phased approach to measuring the impact of AI, starting with adoption and usage, progressing to proficiency and efficiency, and culminating in business and impact metrics.
• Copilot Metrics Dashboard & API: Tools providing access to usage and adoption metrics for GitHub Copilot across an enterprise.
• Adoption and Usage Metrics: Data indicating who is using Copilot and how.
• Proficiency and Efficiency Metrics: Data showing the effectiveness of AI within an organization and its impact on workflows.
• Business and Impact Metrics: Data translating AI innovation into business outcomes like velocity, quality, and ROI.
• Agent Adoption Rate: The percentage of users within an organization who have adopted an AI agent (e.g., Copilot).
• Active User Trend: The number of users actively engaging with a tool over a period.
• Billing Utilization Metrics: Data related to the cost and usage of a service.
• Lines of Code Accepted: A metric indicating the number of code suggestions from AI that developers have incorporated.
• Pull Request (PR) Creation Rates: The frequency at which developers create pull requests.
• Average Chats per User: The average number of interactions (chats) a user has with an AI tool.
• Requests per Mode: The distribution of requests across different AI interaction modes (e.g., chat, inline edits, agent).
• Suggested vs. Completed Completions: The ratio of code suggestions provided by AI versus those actually accepted by developers.
• Acceptance Rate: The percentage of AI-generated code suggestions that developers accept.
• Model Usage: The breakdown of which AI models are being used and for what purposes.
• Language Usage Trends: The distribution of AI tool usage across different programming languages.
• Lines of Code Data: Metrics related to the amount of code generated, deleted, or accepted with AI assistance.
• AI-Generated Lines of Code: Code suggestions provided by AI.
• Lines Deleted: Code that was removed, potentially after being generated by AI.
• Efficiency Gains: Improvements in productivity or speed attributed to AI.
• Observe, Gather Feedback, Deploy Solutions, Improve Cycle: A continuous loop for enhancing AI adoption and effectiveness.
• AI Contribution: The extent to which AI is involved in code generation or other development tasks.
• Survival Metric of AI Code: The longevity and integration of AI-generated code within a codebase.
• Software Development Life Cycle (SDLC) Phases: Plan, Code, Verify, Deploy, Operate.

---

Introduction to AI Metrics and GitHub Copilot

Sharanya Doddapaneni, VP of Engineering at GitHub, introduces the critical need for measuring the impact of AI investments. She emphasizes that the next challenge for organizations is to connect AI innovation to tangible business outcomes through metrics. The presentation outlines a journey for metrics, starting with adoption and usage, progressing to proficiency and efficiency, and ultimately aiming for business and impact metrics that demonstrate velocity, quality, and ROI. Currently, GitHub is focused on helping customers measure adoption and usage, with plans to expand to business and impact insights.

Public Preview of Refreshed Copilot Metrics Dashboard and API

A significant announcement is the public preview of the refreshed Copilot Metrics dashboard and API, available starting today. This release provides enterprise customers with access to usage and adoption metrics. General Availability (GA) for these metrics is planned for February 2026. Enterprise administrators can enable this feature via Agent Control plane policies.

Customer Journey: Mona, Inc. - A Case Study

The presentation uses a fictional company, Mona, Inc., to illustrate how different roles within an organization leverage Copilot Metrics to drive and measure impact.

1. CTO of Mona, Inc.

• Focus: Strategic decision-making and positioning the company as an "AI-first organization."
• Key Metric: 72% agent adoption rate. This provides a concrete baseline, moving beyond anecdotal feedback.
• Insight: The CTO uses this metric to track progress in the AI transformation journey.

2. VP of Engineering of Mona, Inc.

• Focus: Developer happiness and productivity, strategic engineering leadership questions.
• Key Metric: 85% active user trend.
• Insight: This metric confirms that Copilot has moved beyond early adopter enthusiasm and is integral to how engineers deliver code to production. The VP also analyzes average chats per user to identify onboarding effectiveness and potential needs for coaching. They also examine requests per mode to understand how teams use different AI features and look at suggested vs. completed completions and acceptance rate (around 25-30%) to gauge developer trust and integration.

3. CIO of Mona, Inc.

• Focus: Cost, demonstrating Copilot as a business-critical investment, and connecting business value to spend.
• Key Metrics: Billing utilization metrics connected to lines of code accepted.
• Insight: The CIO uses lines of code accepted as concrete data to translate spend into business value. They analyze billing premium analytics to observe trends at enterprise and user levels. For ROI, they track AI-generated lines of code, lines deleted, and efficiency gains, estimating time and cost saved.

4. Engineering Manager of Mona, Inc.

• Focus: Team productivity and enablement, making day-to-day practical decisions.
• Key Access: User-level downloadable reports via the dashboard and API.
• Insight: Unlike executives, engineering managers look for granular trends. They can identify individuals who are thriving with AI and those who need support, enabling them to identify power users and disseminate knowledge. They segment adoption by team or project to identify workflow issues, training gaps, or tool configuration problems. They also use average chats per user as an onboarding heat map and analyze requests per mode for imbalances, focusing on removing friction and changing workflows to drive merge velocity and reduce review grind. For inline code suggestions, they use the acceptance rate as a coaching opportunity, filtering by team to identify potential issues with complex legacy code or to compare with code review metrics.

Live Demo of Copilot Metrics Dashboard

Shruti Corbett provides a live demonstration of the Copilot Metrics dashboard, highlighting its visualization capabilities, API integration for deeper analysis, and user-level report downloads. The demo reiterates how Mona, Inc. uses these metrics:

• CTO: Uses the 80%+ agent adoption rate as a North Star metric.
• Engineering Manager: Segments adoption by team/project to identify issues.
• CTO: Reviews monthly active user trends and compares daily/weekly patterns.
• VP of Engineering: Analyzes active users in relation to Pull Request creation rates to assess throughput.
• VP of Engineering: Tracks average chats per user for onboarding effectiveness.
• Engineering Manager: Uses average chats per user as an onboarding heat map.
• CTO: Analyzes requests per mode to determine which features drive value and where to invest.
• VP of Engineering: Compares team usage of each mode (ask, edit, agent).
• Engineering Manager: Identifies imbalances in mode usage.
• Inline Code Suggestions: The dashboard shows suggested vs. completed completions and the acceptance rate (25-30% for Mona, Inc.).
• VP of Engineering: Views the acceptance rate as an indicator of prompting skills and coding speed.
• CTO: Uses the acceptance rate to prove sustainable adoption and ROI.
• Engineering Manager: Uses acceptance rate for coaching, filtering by team and comparing with code review metrics.
• Model Usage: Claude Sonnet dominates at ~80%, with GPT 5 and 4.1 making up ~15%.
• VP of Engineering: Observes that most interactions occur in ask and agent modes, indicating AI use for problem-solving.
• Language Usage Trends: Go and Ruby are prominent, but over 70% of usage comes from other languages, demonstrating broad tech stack adoption.
• VP of Engineering: Notes GPT 5 gaining traction with backend Go teams, indicating experimentation.
• Downloadable Report: Demonstrates JSON and Jupyter Notebook formats for lines of code data.
• Engineering Manager: Uses lines of code data for motivation, highlighting team output and hours saved.
• CIO: Tracks AI-generated lines of code, lines deleted, and efficiency gains for ROI calculation.

Current Availability and Future Vision

As of today, the following are available:

• Enterprise-level dashboard and API for usage and adoption metrics.
• User-level downloadable reports via dashboard and API.
• 28-day rolling window for data.
• Billing analytics at enterprise and user levels.

Metrics are presented as tools to reveal stories, provide insights, and guide action. A retail customer used active user tracking to identify thriving teams, turning them into case studies. GitHub itself used internal experiments to discover Copilot's utility in writing documentation, fixing accessibility issues, and addressing tech debt. These stories became contagious, driving new ways of thinking and working.

The AI Efficiency Cycle

To become truly AI efficient, organizations need a continuous cycle: observe, gather feedback, deploy solutions, and improve. Metrics are used for observation, while feedback comes from productivity surveys, direct conversations, and community forums (like GitHub's computer club). Actions include learning sessions, peer pairings, and product fixes.

API Integration and Roadmap

The Copilot Metrics API allows integration into existing custom dashboards and analytical tools. The roadmap for future development includes:

• 2025:
  • Lines of code metrics in the dashboard (API available today).
  • Support for organization-level analytics.
• Early 2026:
  • Fine-grained permissions for roles.
  • Copilot Metrics on Data Resident Solution (January 2026).
  • General Availability (GA) of metrics (February 2026).
• Beyond 2026:
  • Non-IDE sources: Copilot Coding Agent and Copilot Code Review metrics.
  • Next edit suggestions metrics.
  • Increased data window beyond 28 days.
  • Analytics agent for conversational metric gathering.
  • Data Warehouse connectors for integration into custom analytics solutions.
  • Team and Repo level analytics.
  • Individual analytics for Copilot Pro and Proplus customers.
  • More impact metrics.

GitHub aims to deliver end-to-end insights across the SDLC (plan, code, verify, deploy, operate) including metrics like PR velocity, time to production, and quality metrics. Unique GitHub insights will include AI contribution, developer contributions, and the survival metric of AI code.

Conclusion

The presentation concludes by reiterating that metrics are a critical tool for driving Copilot investment and impact. The AI landscape evolves rapidly, and measurement must keep pace. The goal is to make value visible and translate it into action, ultimately leading to ROI. The final poll reinforces the importance of metrics, with a near 100% affirmation.