Building AI Agents That Work - My Talk at Google NEXT

Key Concepts

• AI Agents: AIdriven applications using language models guided by instructions and tools within a dynamic runtime environment.
• Workflows: LLMs with instructions following a decision graph, offering a balance between rigidity and flexibility.
• Frameworks (Langraph, Crew AI, Autogen, etc.): Tools for rapid prototyping of AI agents, but potentially introducing unnecessary abstractions.
• System Prompts: Instructions that shape the behavior of the agent.
• Tool Descriptions: Descriptions of the tools available to the agent, crucial for decision-making.
• Evaluation Data Set: A set of examples used to observe and iterate on the behavior of the agent.

1. Defining AI Agents

• Agent Definition: An AIdriven application that uses a language model (or visual language model) guided by a set of instructions that shapes its behavior. It has access to additional tools that enhance its capabilities, all running within a dynamic runtime environment that the LLM can control.
• Core Components:
  • Large Language Model (LLM) at the core.
  • Set of instructions to guide behavior.
  • Access to tools for enhanced capabilities.
  • Dynamic runtime environment controlled by the LLM.
• React Agent Diagram: The agent plans actions, executes them using an orchestration layer and tools, observes the results, and modifies its plans accordingly.

2. When NOT to Use Agents: The Four Key Questions

• Main Point: Agents are not always the best solution. Evaluate the problem before implementing an agentic system.
• Four Questions to Ask:
  1. Predictability: How predictable is your task structure? Can you map out all steps beforehand, or does it require dynamic decision-making? If the task is predictable, workflows are preferable.
  2. Control: How critical is consistency and control? Do you need guaranteed behavior patterns, or can you allow flexibility? If consistency is crucial, avoid stochastic agentic systems.
  3. Boundaries and Complexity: How clearly defined are your boundaries? Can you decompose the problem into fixed subtasks, or is it open-ended? If the problem can be decomposed, workflows are better.
  4. Latency and Cost: What is your tolerance for cost and latency? Are you willing to trade higher costs and longer processing time for more adaptability? Agentic loops involve multiple tool and LLM calls, increasing latency and cost.

3. Workflows as an Alternative

• Definition: LLMs with a set of instructions that follow a pre-designed decision graph or tree.
• Benefits: Offer a balance between rigidity and flexibility, often sufficient for many tasks.
• Use Case: Suitable when predictability, control, well-defined boundaries, and latency/cost constraints are important.
• Enthropic's Blog Post: The concept of workflows was introduced by Anthropic in their blog post "Building Effective Agents."
• Examples:
  • Chaining different LLM calls with decision-making steps based on the output of each call.
  • Using a router LLM to decide which LLM to use with a specific set of instructions.

4. Frameworks: Use Cases and Limitations

• Frameworks Mentioned: Langraph, Crew AI, Autogen, Google Agent Development SDK, OpenAI Agent SDK.
• Benefits: Enable rapid prototyping and quick proof of concepts (POCs).
• Limitations:
  • Introduce abstractions that hide internal workings and system instructions.
  • Force design choices before identifying specific limitations in the system.
  • May lead to suboptimal systems for specific problems.
• Recommended Approach: Start with a framework for a quick prototype, then remove unnecessary abstractions and iterate based on specific needs.

5. Building Agentic Systems: Best Practices

• Start Simple: Build a simple, single agent first, even if the goal is a multi-agent system.
• Iterate Based on Observation: Observe the behavior of the system, identify missing functionalities, and iterate to address them.
• Evaluation Data Set: Create a small evaluation data set (10-15 examples) to test and refine the system.
• Functionality Over Complexity: Prioritize usefulness and avoid over-engineering from the beginning.

6. Prompting Strategies

• Clarity in Prompts: Prioritize clarity and specificity in prompts. Avoid assumptions and provide as much guidance as possible.
• Think Like the Model: Put yourself in the shoes of the model and consider how different actions will look to the agent.
• Tool Descriptions: Pay close attention to tool descriptions and parameters. Ensure they are clear and unambiguous.
• Error Analysis: Run evaluation prompts, analyze errors, and refine tool descriptions and model capabilities.

7. Software Development Practices

• Treat Agentic Applications as Software Development Projects: Emphasize testing, feedback, and iterative refinement.
• Fragility of Systems: Acknowledge the fragility of agentic systems and the need for robust software development practices.
• Iterative Refinement: Continuously refine the system based on feedback and proper evals.

8. Key Takeaways (Repeated for Emphasis)

• "Do not use agents if you don't need to."
• "Frameworks are great for initial PCs or proof of concepts, but then you want to move away from frameworks to build optimal systems."
• "Start simple, do not try to overengineer from the beginning."
• "Prompting is one of the most critical pieces; you really need to pay attention to the instructions that you are giving your LLM."

9. Additional Considerations

• Number of Tools: Limit the number of tools and ensure clear differentiation between them to avoid confusion.
• Prompting Guides: Refer to prompting guides from OpenAI, Google, and Anthropic, but be aware that prompting strategies vary depending on the LLM.

10. Synthesis/Conclusion

The talk emphasizes a pragmatic approach to building AI agents. It cautions against overusing agents when simpler workflows suffice and highlights the importance of clear prompting, iterative development, and a focus on functionality over complexity. While frameworks can be useful for initial prototyping, they should not be the end goal. The key is to understand the problem, design a solution that fits the specific needs, and continuously refine it based on feedback and evaluation. The speaker underscores that these are not magical systems and require rigorous software development practices.