The latest in managing and auditing GitHub Copilot agents

Key Concepts

• Copilot Dashboard: A centralized view for monitoring Copilot usage and activity within an enterprise.
• Agent Mode: A specific mode of interaction with Copilot, likely involving more autonomous task execution.
• Custom Roles: User-defined roles within an enterprise that can be assigned specific permissions for managing Copilot settings.
• Control Panel: A management interface for configuring and overseeing Copilot features and agents at the enterprise level.
• Coding Agent & Code Review Agent: Primary, pre-defined agents within Copilot for code generation and review.
• Custom Agents: User-created agents defined using markdown files for specialized tasks, deployable at the organization level.
• Push Rule Sets: Rules governing the management and deployment of enterprise-level agent files.
• Audit Logs: Records of actions performed within the enterprise, specifically detailing Copilot's activities and the users who initiated them.
• Model Context Protocol (MCP): A protocol enabling AI agents to access and utilize external models, with associated security considerations.
• MCP Registry: A designated location for discovering and accessing MCP servers.

Copilot Management and Insights

The video details enhancements to managing Copilot's capabilities as its usage grows. A new Copilot dashboard has been integrated into the Insights panel within Enterprises. This dashboard provides an at-a-glance overview of key metrics, including:

• Active Users: The number of users actively utilizing Copilot in the current calendar month.
• Agent Mode Usage: The percentage of developers employing Agent Mode in Chat.
• Most Popular Model: Identification of the most frequently used AI model by developers.

Further down the dashboard, users can find detailed interaction data presented through charts, such as:

• Code Completion Acceptance Rates: The percentage of code suggestions accepted by developers.
• Chat Modes in Use: The distribution of different chat interaction modes.
• Models Utilized: A breakdown of the various AI models being used.

The transcript emphasizes that APIs are available for accessing additional, more granular details beyond what is presented in the dashboard.

Access Control and Role Management

To manage access to Copilot settings, the system now allows for the creation of custom roles at the enterprise level. These custom roles can be assigned specific permissions, such as the ability to view or modify Copilot settings for the entire enterprise. Any user assigned to such a role will inherit these defined permissions.

Enterprise-Level Control Panel

A dedicated control panel is introduced for managing Copilot itself. The landing page of this control panel displays a list of all agents currently active or recently completed across the entire enterprise.

The control panel highlights two primary agents:

• Coding Agent: Likely responsible for code generation and assistance.
• Code Review Agent: Designed for reviewing code for quality and potential issues.

From this panel, administrators can choose to enable or disable these features enterprise-wide or allow individual organizations to make their own decisions regarding their enablement.

Custom Agents and Deployment

GitHub has introduced custom agents, which are defined using markdown files to create more powerful and specialized AI agents. These custom agents can be defined at the organization level within a .githubs-private repository. The enterprise control panel allows administrators to select which organization's custom agents will be made available throughout the entire enterprise.

It is noted that custom agents are typically developed by more experienced developers or those with AI specialization. The control panel also displays the push rule set that impacts enterprise-level agent files, allowing for the creation or modification of these rules. For instance, access to these rules can be granted to the previously created AI admin's enterprise role.

Audit Logs and Accountability

The importance of demonstrating who performed what actions, when, and where is addressed through updated audit logs. These logs provide detailed information about Copilot's activities. When accessing audit logs for an enterprise from the control panel, a filter is automatically applied to show logs where the actor is Copilot. Every task performed by Copilot will have this actor property.

Examining the details of a creation action reveals that Copilot is listed as the actor, and a property indicates the action was performed by an AI agent. Crucially, the user field identifies the individual who requested Copilot's work. This functionality enables administrators to track activities within repositories, understand who initiated what, and demonstrate compliance.

Model Context Protocol (MCP)

A significant enhancement for AI agents is the Model Context Protocol (MCP). MCP opens up new capabilities but also introduces potential security concerns. The control panel allows administrators to specify an MCP registry, which helps developers discover and utilize MCP servers. Furthermore, administrators can control whether developers are allowed to use MCP servers at all, or only those from the specified registry. Currently, MCP server and registry settings are exclusively supported by VS Code.

Conclusion

The video concludes by reiterating that Copilot's feature set is continuously expanding, and with this expansion comes an evolution of the available controls. These controls are designed to ensure that the right people have access to the necessary tools, and that administrators have the insights and control mechanisms to manage how these AI tools are being utilized within the enterprise.