TNS Agents live from Open Source Summit EU in Amsterdam with Lin Sun, head of open source at Solo.io

Key Concepts

• Agent Gateway: An open-source project donated to the Linux Foundation that provides security, governance, and observability for agent-to-agent (A2A), agent-to-MCP (Management Configuration Protocol) tools, and agent-to-large language model (LLM) communication.
• MCP (Management Configuration Protocol): A protocol used for managing and configuring tools, often compared to a USB-C driver or adapter for tools.
• A2A (Agent-to-Agent): A standardized communication protocol between agents.
• Inference Gateway: A smart LLM router that intelligently routes requests based on endpoint availability and model loading status.
• K Gateway: A control plane used to program Agent Gateway within Kubernetes, providing Kubernetes Gateway API support.
• K Agent: An internal Solo.io project inspired by a production outage, aiming to scale expertise within the company by baking knowledge into AI agents.
• Kubernetes Gateway API: The future of networking API in Kubernetes.
• Cloud Native Operation Excellence (COE): An initiative focused on aentic AI for platform engineers, involving Cisco, AWS, and others.

Agent Gateway: Introduction and Purpose

Lynn from Solo.io introduces Agent Gateway, a new open-source project donated to the Linux Foundation. It's positioned as the first and only open-source agent gateway project, designed to address security, governance, and observability challenges in agentic AI environments. The core idea is that as organizations adopt AI agents and MCP tools, these components need to communicate securely and efficiently, not only with each other but also with traditional microservices and LLMs. Agent Gateway aims to mediate this communication, providing a unified control point.

Functionality and Features

Agent Gateway provides security, governance, and observability to agent-to-agent, agent-to-MCP tools, and agent-to-LLM communication. It natively understands MCP and A2A protocols, as well as traditional microservices protocols. It supports Kubernetes Gateway API and Inference Gateway.

• Inference Gateway Details: The Inference Gateway acts as a smart router for LLM requests, intelligently routing them based on factors like model availability and endpoint metrics. This ensures efficient utilization of LLM resources.
• MCP Governance: Agent Gateway allows administrators to control which MCP tools are exposed to specific agents. For example, out of 20+ tools available via an MCP server, only a subset might be exposed to certain AI agents, enhancing security and preventing unintended usage.
• Protocol Support: Agent Gateway supports HTTP, gRPC, and TCP routes, making it compatible with traditional microservices.

Why Agent Gateway? Addressing the Need for a Middleman

While A2A and MCP provide standardized communication protocols, they lack built-in governance and security features. Agent Gateway acts as a "middleman" to enforce policies, provide resilience, and mediate communication between agents, MCP tools, and microservices.

• Security Policy Enforcement: Agent Gateway enables the implementation of security policies to control which agents can access specific MCP tools or resources.
• Observability and Compliance: It provides observability and compliance features for agentic AI environments.
• Beyond Traditional API Gateways: While similar to API gateways, Agent Gateway is tailored for the unique characteristics of MCP and A2A protocols, such as JSON RPC-based communication and context sensitivity. It handles multiplexing, allowing one Agent Gateway to connect to multiple MCP servers and tools.

Real-World Use Cases and Adoption

Several companies are already exploring or using Agent Gateway.

• Internal Use at Solo.io (K Agent): Solo.io developed Agent Gateway for its internal K Agent project, which aims to scale expertise by using AI agents to assist support engineers.
• Customer Interest: Companies like T-Mobile and UBS have expressed interest in Agent Gateway.
• Banking and Regulated Industries: Organizations in the banking and cryptocurrency sectors are particularly interested in Agent Gateway due to the need for strong governance and security in AI deployments.
• Cloud Native Operation Excellence (COE) Initiative: The COE initiative, involving Cisco, AWS, and others, is considering adopting Agent Gateway to manage communication between agents and MCP tools.

Community Contributions and Technical Details

The Agent Gateway project has received contributions from the community.

• Microsoft: Added Windows support for Agent Gateway.
• AWS: Added bad rock examples.
• Rust Implementation: Agent Gateway is built using Rust, chosen for its performance and safety characteristics. This decision was influenced by lessons learned from the Istio Ambient project, where relying on Envoy (C++) proved challenging.

Relationship to Istio and Service Mesh

Solo.io views Agent Gateway as an extension of its cloud-native networking focus, which includes service mesh and API gateway solutions. The company believes that as organizations adopt agentic AI, they will need to integrate AI agents into their cloud-native architectures. Agent Gateway helps to make the mesh "agentic" by providing the necessary security, governance, and observability features.

• Evolving Perspective on Service Mesh: The adoption of service mesh has been slower than expected due to complexity and the availability of simpler alternatives. However, the emergence of agentic AI may change this, as the need for security and observability becomes more critical.
• Agent Security and Auditability: The need for auditability is greater with agents than with microservices, as organizations may not fully trust the behavior of AI agents.

Conclusion

Agent Gateway addresses a critical gap in the agentic AI ecosystem by providing a unified control point for security, governance, and observability. Its support for MCP, A2A, and traditional microservices protocols, combined with its real-world use cases and community contributions, position it as a valuable tool for organizations adopting AI agents. The project's focus on security and auditability is particularly relevant in regulated industries and for applications where trust in AI agents is paramount.