How to design a multi-agent system that skips the LLM

Key Concepts

• Agent Development Kit (ADK): A framework for building, managing, and scaling AI agents.
• Deterministic Logic vs. LLM Inference: The strategic choice between using hard-coded algorithms (deterministic) and generative AI (LLM) to optimize for cost, latency, and reliability.
• NP-Hard Problems: Computationally complex problems (like shortest-path routing) where deterministic algorithms are often more efficient and reliable than LLM reasoning.
• Server-Side Ticks: A game-development concept used to synchronize state across multiple independent agents.
• Redis Session Store: A high-performance, in-memory data store used to manage state across stateless, distributed agent instances.
• Before-Model Callbacks: An ADK feature used to intercept agent lifecycles to inject deterministic logic, effectively bypassing LLM calls.

---

1. Architectural Strategy: Hybrid Systems

The core philosophy presented is that not every agent action requires an LLM. By using a hybrid approach, developers can maintain the benefits of an agent framework (telemetry, lifecycle management, observability) while using deterministic code for mechanical or computationally heavy tasks.

• The "Planner" Agent: Instead of asking an LLM to calculate a marathon route (which is an NP-hard problem prone to "hallucinations" or noise), the team used deterministic algorithms.
• The "Simulator" Agent: Uses a "tick" system (similar to game servers) to manage the state of 1,000 runners. By using before_model callbacks, the system intercepts the agent's execution flow to run procedural code, bypassing the LLM entirely.
• Benefits: This approach significantly reduces token consumption, lowers latency, and ensures the system remains unit-testable.

2. Step-by-Step Methodology: Building the Marathon Route

The team avoided using Google Maps or LLMs for the route due to specific constraints (e.g., 26.2-mile requirement, city-approved streets, avoiding walking bridges).

1. Algorithm Selection: Used Gemini in AI Studio to research and generate Python code for pathfinding algorithms.
2. Deterministic Implementation:
   • Phase 1: Used a Dijkstra algorithm with Haversine-weighted edges to connect landmarks.
   • Phase 2: Applied a serpentine waypoint sequence to extend the path.
   • Trimming: Programmatically trimmed the final segment to hit the exact official marathon distance of 26.2188 miles.
3. Validation: The resulting code is fully unit-testable, ensuring reliability that an LLM could not guarantee.

3. Scaling to 1,000 Agents

To manage 1,000 simultaneous, independent agent sessions on Google Cloud Run, the team addressed the challenge of statelessness:

• The Problem: Standard in-memory session stores fail when scaling across 50+ instances of a service. Database-backed stores (like AlloyDB) introduced too much latency due to transaction overhead.
• The Solution: Implemented a Redis Session Service (sourced from the Google ADK Community GitHub repository).
• Optimization: The custom Redis implementation includes event pruning to prevent "blob growth," keeping the session state lean and fast.

4. Key Arguments and Evidence

• Reliability over Generative Power: Casey Vest argues that for complex, constrained problems, "the best answers are not to go with full large language models." The evidence is the system's ability to run 1,000 agents simultaneously without the cost or latency of 1,000 concurrent LLM calls.
• Telemetry Preservation: By using the ADK framework even for deterministic tasks, the team retains full visibility into the agent's lifecycle. "It would be easy to write this where you short-circuit... but then you lose things like visibility and telemetry."

5. Notable Quotes

• "We use the model where it actually helps and we skip the model where it doesn't." — Casey Vest
• "The model is required by the LLM agent in order to create the object, but it's never actually called because the before-model callback intercepts every invocation." — Casey Vest, explaining the bypass mechanism.

6. Synthesis and Conclusion

The project demonstrates that the most effective AI systems are often "agent-orchestrated" rather than "LLM-driven." By treating the LLM as a high-level decision-maker (e.g., choosing which algorithm to use) and the code as the execution engine (e.g., running the pathfinding math), the team achieved a highly scalable, cost-effective, and performant architecture. The key takeaway for developers is to leverage the agent framework for its management capabilities while aggressively offloading repetitive or logic-heavy tasks to deterministic code.