The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Key Concepts

• Observability Gap: The inability to trace and visualize AI decision-making processes in production.
• Evaluation Gap: The lack of defined, measurable business success criteria and automated testing frameworks.
• Governance Gap: The absence of accountability, security protocols, and change management for AI systems.
• LLM-as-a-Judge: Using a secondary LLM to evaluate the output of a primary model based on safety, relevance, and groundedness.
• Golden Data Sets: Curated sets of high-quality, ground-truth examples used to benchmark AI performance.
• Orchestration Patterns: Architectural frameworks (Orchestrator-Worker, Choreography) for managing multiple AI agents.
• RAG (Retrieval-Augmented Generation): A technique where AI retrieves data from external sources (like vector databases) to ground its responses.

---

1. The Five Pillars of AI Production

To move from a successful demo to a scalable production system, the speaker proposes a framework consisting of five essential pillars:

1. Evaluation: Defining success metrics (e.g., deflection rates, accuracy) and building an automated pipeline to measure performance against "golden data sets."
2. Observability: Implementing tracing to capture every step an agent takes, which is critical for debugging and regulatory compliance.
3. Data Foundation: Establishing a robust strategy for both "question data" (context for AI) and "tracking data" (logs for observability).
4. Orchestration: Managing the complexity of multi-agent systems through specific patterns like Orchestrator-Worker or Choreography.
5. Governance: Ensuring accountability, security (PII protection), and rigorous change management for prompts and models.

---

2. Evaluation Framework Layers

The speaker categorizes evaluation into three distinct layers:

• Deterministic: Basic checks (regex, email/phone formats, PII detection) using traditional coding or classic ML.
• Non-Deterministic (Semantic): Using LLM-as-a-judge to evaluate groundedness, safety, and relevance.
• Behavioral: Monitoring agent logic, such as tool-calling efficiency, to prevent redundant API calls and infinite loops.

---

3. Multi-Agent Orchestration Patterns

• Orchestrator-Worker: A centralized controller distributes tasks to specialized agents. This provides high control and easy debugging via central logs.
• Choreography: Agents are autonomous and communicate via a message bus. This is more efficient for parallel tasks but increases complexity in state management.
• Human-in-the-Loop: A fallback mechanism where an agent triggers human intervention if confidence scores fall below a specific threshold.

---

4. Data Strategy and Infrastructure

The speaker emphasizes that data must be "AI-ready." Using Databricks as an example, the architecture involves:

• Delta Lake: Providing database-like properties (ACID transactions) on raw data.
• Unity Catalog: A centralized governance layer for permissions, metadata tagging, and data discovery.
• Tracing Strategy: Centralizing logs from various frameworks (LangChain, CrewAI) into a single location to serve operational dashboards and auditors.

---

5. Case Study: Retail Banking Chatbot

• Problem: 20,000 monthly queries; 60% were simple, yet the bank relied on human agents. A previous 6-month, $85K POC failed due to lack of observability and clear metrics.
• Solution:
  • Weeks 1-2: Built an evaluation data set of 200 human-agent interactions.
  • Weeks 3-6: Implemented tracing and automated evaluation pipelines.
  • Weeks 7-8: Selected the model based on performance against the golden data set.
• Outcome: Successfully automated simple queries. When interest rate policies changed, the system detected a drop in CSAT because the RAG system was retrieving stale documents, allowing for a rapid, data-driven fix.

---

6. Actionable Insights & Best Practices

• Treat Prompts as Code: Implement version control and document the "why" behind every prompt change.
• Living Evaluation System: Your test data set is not static; it must grow as you encounter new edge cases in production.
• Cost Management: Run full evaluation suites only on the main branch; use smaller subsets for CI/CD pipelines to save on compute costs.
• Incident Playbook: Define a clear process: Detect (via dashboard) → Diagnose (via traces) → Contain (via prompt versioning/human fallback) → Fix (via test case library).

"Data was always built for humans, and humans are always forgiving. Agents don't forgive you; they will go, find it wrong, and give you the wrong answer confidently." — Sandy, Technical Lead for Data and AI at Databricks.

Conclusion

Taking AI to production requires shifting focus from "choosing the right model" to "building the right infrastructure." By prioritizing observability, automated evaluation, and rigorous governance, organizations can transform experimental demos into reliable, enterprise-grade systems. The key takeaway is to build a "living" evaluation system that evolves with the data and ensures accountability at every step of the AI's decision-making process.