LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Key Concepts

• Observability: The practice of monitoring and auditing AI systems (agents/harnesses) using telemetry to understand internal operations.
• Evals (Evaluations): Methods to derive signal from AI systems to measure performance, quality, and business impact.
• OpenTelemetry (Otel): An industry-standard framework for collecting traces and spans to audit system behavior.
• Traces and Spans: The fundamental audit records of an agent’s execution path.
• LLM-as-a-Judge: Using an LLM to evaluate the output of another LLM or agent.
• Non-deterministic Systems: AI applications where the same input may yield different outputs, requiring distributional analysis rather than simple unit testing.
• Trajectory Evals: Evaluating the sequence of steps (pathing) an agent takes to complete a task.
• Golden Data Sets: Curated, high-quality data sets used as a benchmark for evaluating and tuning AI performance.

---

1. Observability: Understanding System Behavior

The speaker emphasizes that AI development is "software reimagined." To manage non-deterministic AI agents, developers must move beyond traditional logging to Observability.

• OpenTelemetry (Otel) First: Arize AI utilizes Otel to provide visibility. By using auto-instrumentation (a single line of code), developers can generate traces and spans that act as an audit record for agent actions.
• Sessions and State: Beyond individual calls, observability must track "sessions"—the back-and-forth stateful conversations between users and agents.
• Distributional Views: Instead of looking at single calls, developers should analyze the distribution of traffic across different branches, loops, and paths to identify bottlenecks or latency issues.

2. Evals: Deriving Signal

Evals are categorized into five "flavors" to determine if an application is working as intended:

1. LLM-as-a-Judge: Using an LLM to score outputs.
2. Human Feedback: Utilizing end-user or subject matter expert input.
3. Golden Data Sets: Using trusted, domain-specific data to calibrate and tune the "LLM-as-a-Judge."
4. Deterministic Evals: Logic-based checks (e.g., validating JSON schema or non-null fields).
5. Business Metrics: Measuring ROI, cost savings, or time efficiency.

Scope and Depth of Evals:

• Span Evals: Single input/output evaluation.
• Multi-span Evals: Evaluating data flow across multiple components.
• Trajectory Evals: Analyzing the sequence of tool calls to ensure the agent followed the correct logic.
• Session Evals: Evaluating the entire state machine of a conversation (e.g., "Was the user frustrated?").

3. Experimentation and Improvement

The speaker highlights the "flywheel" of AI development: Observability → Evals → Experimentation → Improvement.

• Regression Management: A critical challenge is that fixing one issue often introduces two or three new regressions.
• Experimentation: Developers can test changes to prompts, models, orchestration, or configurations using either a UI or programmatic CLI tools.
• Automation: The ultimate goal is to automate the entire loop. The speaker introduces "Alex," an AI system that can analyze application data, detect issues (like high latency or errors), and suggest or run evals automatically.

4. Organizational Roles

The speaker notes a shift in team dynamics:

• Technical Users (AI Engineers): Focus on frameworking, automation, and infrastructure.
• Subject Matter Experts (Product Managers): Focus on the "AI experience," prompt engineering, and defining what "quality" looks like.
• Framework: The platform allows non-technical users to select models and run templates, while technical users can attach programmatic evals.

5. Notable Quotes

• "It feels like magic, but it's not magic. It's all just engineering."
• "Code doesn't audit agents or harnesses; it's actually the telemetry that does that."
• "Just because you can eval something doesn't mean you always should... you want to see the minimal set of evals I can get away with."

6. Tools and Platforms

• Arize Phoenix: An open-source, single-container tool for local deployment, designed for engineering-first workflows.
• Arize AX: An enterprise-grade platform for large-scale production environments (e.g., Uber, Reddit, Booking.com).

---

Synthesis

The core takeaway is that managing AI agents requires a shift from traditional software testing to a continuous observability and evaluation loop. By leveraging OpenTelemetry for visibility and implementing a tiered evaluation strategy (from span-level to session-level), teams can effectively manage the non-deterministic nature of AI. The industry is moving toward automating this "flywheel," where AI systems eventually self-diagnose and suggest improvements, reducing the manual burden on developers.