Taming Rogue AI Agents with Observability-Driven Evaluation — Jim Bennett, Galileo

Key Concepts

• AI Evaluation: Using AI to verify the performance and accuracy of other AI systems.
• Observability-Driven Development: Building and improving AI systems based on real-time monitoring and analysis of their behavior.
• Agentic Workflows: Complex AI applications involving multiple agents, LLMs, and tools working together.
• Metrics Granularity: Breaking down evaluations to individual steps within a workflow to pinpoint failure points.
• Continuous Learning by Human Feedback (CLHF): Incorporating human input to refine and improve AI evaluation metrics.
• Hallucination: The tendency of AI models to generate false or nonsensical information.
• RAG (Retrieval-Augmented Generation): A framework for improving the accuracy and reliability of LLMs by grounding them in external knowledge sources.

AI Trust and the Problem of Detection

The speaker begins by highlighting the widespread distrust in AI, citing examples of AI systems making errors and generating false information.

• Examples:
  • The Chicago Sun-Times publishing a summer reading list generated by AI that included a non-existent book ("The Last Algorithm" by Andy Weir).
  • Lawyers using AI for case law research citing false cases.
  • Air Canada's chatbot providing incorrect refund information.

The core problem is that detecting issues in AI systems is difficult due to their non-deterministic nature. Unlike traditional code where unit tests can easily verify specific inputs and outputs, AI systems, especially complex agentic workflows, are hard to evaluate.

• Challenge: Defining "work" in the context of AI, especially in applications like chatbots where human conversation is involved.

Evaluation-Driven Development: Setting a Thief to Catch a Thief

The speaker introduces the concept of using AI to evaluate AI, drawing on the British expression "set a thief to catch a thief." The idea is that AI can be effective at identifying issues in other AI systems.

• Key Point: AIs are about as good as humans at determining whether an AI system is working correctly.

Demo: Chatbot Evaluation

The speaker presents a demo of a chatbot designed for a fintech application. The chatbot is asked to provide the user's account balance.

• Scenario:

  1. Initial question: "What is my account balance?" - Response: "I don't have access to account information." (Failure)
  2. Follow-up question: "What is the balance of my checking account?" - Response: "Please could you let me know the name of your checking account." (Partial Advancement)
  3. User provides the name: "Checking account" - Response: Provides the account balance. (Completion)

• Analysis: The chatbot eventually provided the correct information, but it took three steps, indicating a problem with the initial prompt and workflow.

Metrics and Granularity

The speaker emphasizes the importance of defining metrics to measure the success of AI systems and breaking down evaluations to individual steps within a workflow.

• Key Metrics:

  • Action Completion: Did the AI successfully complete the task it was asked to do?
  • Action Advancement: Did the AI move forward towards the end goal?

• Granularity: It's crucial to evaluate each component of an agentic system (e.g., LLM calls, tool usage, data retrieval) to pinpoint where failures occur.

Using LLMs for Evaluation

The speaker explains how LLMs can be used to evaluate AI systems.

• Process:

  1. Use an LLM (ideally a better one than used in the main application) to score the output of the AI system based on the input and any relevant context (e.g., data from a RAG system).
  2. Define a well-defined set of prompts to guide the LLM in extracting relevant information for evaluation.
  3. Incorporate this evaluation process into workflows from the beginning of development.

• Recommendation: Use a custom-trained LLM specifically designed for evaluations.

Integrating Evaluations into the Development Lifecycle

The speaker stresses the importance of integrating evaluations into the entire development lifecycle, from initial prompt engineering and model selection to CI/CD pipelines and production monitoring.

• Key Point: The best time to add evaluations is during the initial development phase; the second best time is now.

Real-Time Prevention and Alerting

The speaker advocates for real-time monitoring and alerting to detect and prevent issues in production.

• Goal: To be alerted when an AI agent goes rogue, allowing for immediate intervention.

Unstructured Data and AI Insights

The speaker highlights the potential of using AI to analyze unstructured data generated by evaluations and provide insights for improvement.

• Example: An AI system analyzing evaluation data might identify that the LLM frequently fails to use the "get balance" tool when asked about account balances and suggest adding explicit instructions to the system message.

Human in the Loop and Continuous Learning

The speaker emphasizes the importance of human feedback in the evaluation process.

• Key Point: Evaluation metrics are not perfect out of the box and require continuous training and refinement based on human input.

• CLHF (Continuous Learning by Human Feedback): Humans should evaluate the numbers generated by AI evaluations, identify errors, and retune the system accordingly.

Steps to Tame AI Agents with Evaluations

The speaker outlines the following steps:

1. Add evaluations to your agent: Start as early as possible in the development process.
2. Measure precisely what you need: Define specific metrics relevant to your application (e.g., toxicity, hallucinations, incomprehensible output, RAG performance).
3. Keep it going all through production: Continuously monitor and evaluate the system in production to catch unexpected issues.
4. Have real-time prevention: Implement alerting to detect and respond to problems in real-time.

Conclusion

The speaker concludes by emphasizing the importance of evaluations in taming rogue AI agents. By integrating evaluations into the development lifecycle, measuring relevant metrics, and incorporating human feedback, developers can build more reliable and trustworthy AI systems.