Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft

Key Concepts

• Bitwise Determinism: The expectation that the same input will always yield the exact same output. The speakers argue this is impossible to achieve with hosted LLM APIs.
• Replayability: The ability to record and re-validate a specific execution path of an agent to facilitate debugging.
• Boundary: A conceptual "bounding box" around nodes (LLM calls, tool calls, RAG retrievals) used to capture inputs, outputs, and metadata.
• Chronicle: A proof-of-concept framework that implements boundary-based recording and stubbing for agentic workflows.
• Non-determinism in LLMs: Caused by GPU hardware behavior, floating-point math, batching, and Mixture of Experts (MoE) routing.

---

1. The Problem: Debugging Non-Reproducible Agent Failures

The speakers, Tisha and Sachin, highlight a critical issue in production AI agents: when an agent makes a logical error (e.g., misinterpreting a "sell $1,000 of stock" request as "sell 1,000 shares"), it is often impossible to reproduce the error locally.

• The "Temperature Zero" Fallacy: Developers often set temperature to zero to force determinism. The speakers argue this is a misconception; it does not fix broken reasoning paths and fails to account for underlying hardware non-determinism.
• The "Silent Failure" Scenario: In the provided case study, an agent executed a $190,000 trade instead of a $1,000 trade, yet the API returned a "200 OK" status in 30ms, leaving dashboards green and developers blind to the error.

2. Why Determinism Fails (First Principles)

The speakers identify four technical reasons why LLM outputs vary even with identical prompts:

1. Sampling vs. System Determinism: Argmax (temperature 0) only selects the highest probability token; it does not guarantee that the underlying logit scores remain identical across runs.
2. Floating Point Math: Matrix operations are not associative; tiny shifts in calculation order alter final logits, potentially flipping the winning token.
3. Batch Invariance: Requests are grouped with other traffic on the server. The specific batch context affects the matrix multiplication results.
4. Mixture of Experts (MoE) Routing: Experts have capacity limits. If a batch overflows a subnetwork, tokens are rerouted based on current traffic, leading to different outputs.

3. The Solution: Replayability over Determinism

The speakers argue that chasing bitwise determinism is a "losing battle." Instead, teams should focus on observability through replayability.

• Recording at the Boundary: Rather than logging network packets, developers should record the inputs and outputs at the "boundary" of every node (LLM, tool, or retrieval).
• The Chronicle Framework:
  • Annotation: Developers use a @boundary annotation on methods.
  • State Capture: It records input/output pairs, model versions, and code versions, effectively "freezing" the state of the agent run.
  • Stubbing: During testing, developers can load a recorded trace and "stub" specific nodes. This allows them to isolate a faulty node (e.g., an LLM call) while running the rest of the agent logic live to verify a fix.

4. Testing Methodologies

The presentation distinguishes between two essential types of testing for AI agents:

• Deterministic Testing: Used for guardrails and tool calls. By using recorded traces to stub LLM nodes, developers can create free, repeatable, and deterministic test cases that verify if a fix (like a guardrail) successfully blocks a previously failed action.
• Behavioral Testing: Used for subjective qualities like tone or trajectory. Techniques like "LLM as a judge" are recommended here.

5. Notable Quotes

• "If you can't reproduce it, you can't debug it. And if you can't debug it, you can't promise it won't happen to your next customer."
• "We don't need the model to return the exact same token back every time. We just need our system to execute the exact same state transition."
• "Stop chasing bitwise determinism through the API... the randomness is what makes the model good."

6. Synthesis and Takeaways

The core takeaway is a shift in engineering mindset: stop trying to force the LLM to be deterministic. Instead, build systems that record the "full envelope" of an agent's execution. By capturing the state at every node boundary, developers can transform production failures into reliable, stubbable test cases. This approach allows for rigorous debugging and verification without sacrificing the creative benefits of non-zero temperature generation.