How we built 1,000 AI agents that run a marathon

Key Concepts

• Agentic System: A multi-agent architecture where independent AI entities (runners, planners, simulators) make autonomous decisions.
• Polyglot Stack: A system architecture utilizing multiple programming languages (Go for high-concurrency networking, Python for AI/LLM logic) to leverage the strengths of each.
• A2A (Agent-to-Agent) Protocol: A communication standard allowing agents to discover and interact with one another without hardcoded configurations.
• Dispatch Modes: Two methods of agent communication: Callable (standard HTTP for occasional tasks) and Subscriber (Redis Pub/Sub for high-frequency, low-latency real-time updates).
• ADK (Agent Development Kit): A framework used to manage agent lifecycles, tool calling, telemetry, and LLM interactions.
• Resilient Model: A wrapper around LLM calls that implements automated retries, exponential backoff, and jitter to handle intermittent failures (e.g., 429 quota errors).
• Agent Card: A metadata profile (JSON) defining an agent’s identity, capabilities, skills, and communication protocols.
• Context Window Management: The practice of lazy-loading skills and instructions to keep the LLM context efficient and cost-effective.

---

1. System Architecture Overview

The "Race Condition" project is a large-scale simulation of 1,000 AI agents running a marathon. The architecture is divided into four distinct layers:

• Front End (Angular): Provides the visualization and user interface for the simulation.
• Gateway (Go): Acts as a high-concurrency "traffic cop," handling WebSocket connections and routing messages between the front end and the back end.
• Back End (Python/ADK): Contains the agent logic, including planners, simulators, and runner agents.
• Infrastructure (Redis/Pub-Sub): Provides the messaging backbone for real-time communication.

2. Communication Strategies: Callable vs. Subscriber

To achieve high performance with 1,000 simultaneous agents, the system employs two dispatch modes:

• Subscriber Mode (Redis Pub/Sub): Used for "Runner" agents. The simulator broadcasts a "tick" message to Redis, which all runners receive simultaneously. This eliminates HTTP latency, allowing for sub-millisecond interactions.
• Callable Mode (HTTP): Used for "Planner" and "Simulator" agents. These agents remain idle (scaling to zero) and are only invoked via standard HTTP requests when needed, saving CPU costs.

3. The Role of ADK and LLM Integration

The system uses the ADK framework to avoid building distributed system "plumbing" from scratch. Key benefits include:

• Lifecycle Management: Handling agent state across requests.
• Telemetry: Built-in dashboards to monitor agent health and activity.
• Resilience: The "Resilient Model" utility ensures that if an LLM call fails due to network issues or quota limits, the system automatically retries with exponential backoff and jitter.
• Static vs. Dynamic Instructions: Static instructions are cached server-side to improve speed and reduce costs, while dynamic instructions allow for variable injection based on session state.

4. Dynamic Skill Loading

To manage the context window effectively, the system uses lazy-loading for skills.

• Agents are instructed to load specific skills (e.g., "GIS Spatial Engineering") only when required.
• Each skill includes a skill.markdown file containing instructions and metadata for associated tools.
• This prevents the LLM from being overwhelmed by unnecessary information, keeping the agent focused and efficient.

5. Key Arguments and Design Philosophy

• Separation of Concerns: By separating the "thinker" agents (planners) from the "doer" agents (runners), the system optimizes for both intelligence and raw speed.
• Mesh Networking: The developer designed the system as a mesh network where any agent can theoretically communicate with any other, rather than a rigid directed graph. This allows for future expansion, such as adding "Public Safety" or "Business" agents that react to the marathon's progress.
• Deterministic vs. Probabilistic: While the system uses LLMs for decision-making (introducing "chaotic and unexpected behaviors"), it relies on deterministic infrastructure (Go/Redis) to ensure the simulation remains stable and scalable.

6. Notable Quotes

• "The network becomes the bottleneck... once you optimize your agents and your communication pathways." — Casey West, on the necessity of moving from HTTP to Redis Pub/Sub for high-scale agent communication.
• "ADK gives us all of that [lifecycle, telemetry, tool calling] out of the box... it means that you can focus 100% of your time on intelligence... rather than the plumbing of distributed systems." — Casey West.

Synthesis

The "Race Condition" project demonstrates that building a high-scale, multi-agent system requires a hybrid approach. By combining the flexibility of LLMs (via ADK) with the performance of low-level networking (Go and Redis), developers can create complex, real-time simulations. The core takeaway is that infrastructure design—specifically how agents communicate and manage their context—is just as critical as the intelligence of the models themselves.