My Top 20 Lessons from Building 100s of AI Agents (Super Actionable)

Key Concepts

• AI Agents: Programs using LLMs to interact with environments and achieve goals.
• Hallucinations: AI agents making mistakes or fabricating information confidently.
• Non-Determinism: AI agents producing varying outputs for the same input.
• AI Agent Components: Agent Program (system prompt), LLM, Tools, Memory System (short-term and long-term).
• Guardrails: Logic to prevent or correct hallucinations.
• Specialized Agents: Distributing responsibility among agents for specific tasks.
• System Prompt: Instructions defining agent behavior and tone.
• Context Length: The amount of information an LLM can process at once.
• RAG: Retrieval-Augmented Generation, a technique for improving LLM accuracy by retrieving relevant information from a knowledge base.

High-Level Lessons

1. Use AI to Save Time, Not Replace You Entirely

• Main Point: Share responsibility with AI to catch hallucinations.
• Example: Instead of automatic email replies, have the agent draft emails for review.
• Argument: AI agents are powerful but prone to errors; human oversight is crucial.
• Quote: "With great power comes great responsibility." - Peter Parker's uncle.

2. Don't Skimp on Planning and Prototyping

• Main Point: Thorough planning saves development time.
• Details: Dedicate time to defining goals, tools, and agent behavior before implementation.
• Argument: Investing time upfront prevents costly rework later.
• Roadmap: Refer to the roadmap for building AI agents, emphasizing planning and prototyping phases.

3. Beware the Hallucination Explosion (Compounding Non-Determinism)

• Main Point: Errors compound in multi-agent workflows.
• Details: If multiple agents work together, the overall system accuracy decreases significantly.
• Example: Three agents working at 95% accuracy result in an 86% system accuracy.
• Calculation: 0.95 * 0.95 * 0.95 = 0.857 (approximately 86%).
• Solution: Implement strategies to reduce hallucinations in individual agents.

4. AI Agent Guardrails

• Main Point: Implement logic to prevent or correct hallucinations.
• Details: Guardrails run before or after LLM calls to detect potential issues.
• Example: Travel planning agent:
  • Input Guardrail: Check if the budget is reasonable before planning.
  • Output Guardrail: Verify the itinerary matches the requested duration.
• Process: If a guardrail fails, take corrective action (e.g., inform the user, loop back to the agent).

5. Specialized Agents

• Main Point: Distribute responsibility among specialized agents.
• Argument: Specialization improves accuracy and reduces the load on individual LLMs.
• Example: Separate agents for Slack tool calls and database interactions.
• Consideration: Be mindful of the hallucination explosion when using multiple agents.

6. Examples, Examples, and Examples

• Main Point: Provide concrete examples in the system prompt.
• Details: Examples demonstrate desired behavior and output format.
• Examples: Vzero, Cursor, and Bolt include detailed examples in their system prompts.
• Benefits: Helps the agent understand complex instructions and tool usage.

Lessons for the Agent Program (System Prompt)

7. Avoid Adding Negatives

• Main Point: Phrase instructions positively instead of negatively.
• Bad Example: "Do not use complex language."
• Good Example: "Use fifth-grade level English."
• Reason: LLMs may drop the negative in longer system prompts.

8. Avoid Contradictions in Your System Prompts

• Main Point: Ensure consistency in instructions.
• Example: Avoid telling the agent to be both concise and comprehensive.
• Impact: Contradictions lead to inconsistent results and hallucinations.
• Case Study: A customer support agent told to be flexible but also rigid with calendar tool usage.

9. Version Your Prompts

• Main Point: Track changes to system prompts for easy reversion.
• Reason: New prompts may introduce errors or reduce performance.
• Tools: Langfuse, GitHub repositories.

Lessons for Large Language Models (LLMs)

10. Swapping Large Language Models Can Actually Be Pretty Dangerous

• Main Point: Changing LLMs requires thorough testing and potential prompt adjustments.
• Example: Switching from GPT-4 Turbo to GPT-4o caused unexpected hallucinations.
• Reason: Different LLMs interpret system prompts differently.

11. Your Favorite LLM Isn't Always the Best

• Main Point: Be open to using different LLMs for different tasks.
• Example: Claude 3.7 Sonnet for coding, Gemini 2.5 Pro for creative tasks.
• Argument: Different LLMs excel in different areas.

12. Watch Your Context Lengths

• Main Point: Monitor context length, especially with local LLMs.
• Problem: Exceeding context length can lead to the loss of the system prompt and conversation history.
• Symptoms: The agent forgets instructions and tool usage.
• Solution: Use LLMs with longer context lengths or reduce the amount of information in the prompt.

Lessons for Memory Systems

13. Previous Hallucinations Are Likely to Be Repeated by the Agent

• Main Point: Agents may repeat corrected mistakes in the same conversation.
• Example: An agent initially states that "To Kill a Mockingbird" was published in 1962, is corrected to 1960, but later reverts to the incorrect year.
• Solution: Start new conversations frequently.

14. Long-Term Memory for Your Agents Is Just Another RAG

• Main Point: Long-term memory is essentially another RAG system.
• Details: Strategies for better RAG retrieval apply to long-term memory as well.
• Example: The N8N agent uses the same RAG pipeline for retrieving memories and documents.

15. Include Tool Calls in Your Conversation History

• Main Point: Store tool requests and responses in the conversation history.
• Benefits: Allows the agent to re-reference tool outputs and avoid redundant calls.
• Example: The agent can re-use retrieved chunks from a RAG lookup to answer follow-up questions.

Lessons for Tools

16. The Descriptions That You Give to Your Tools Are Key

• Main Point: Tool descriptions are crucial for guiding LLM usage.
• Details: Include the purpose, usage instructions, and arguments.
• Rule of Thumb: Tool descriptions explain individual tool usage, while the system prompt explains how to use tools together.

17. Give Examples Specifically Examples for What the Parameters Might Look Like

• Main Point: Provide examples of parameter formats for complex tools.
• Example: For a RAG tool, provide examples of query formats.
• Benefits: Helps the agent understand how to use the tool correctly.

18. Catch the Errors and Then Return the Problem to the Agent

• Main Point: Handle tool errors gracefully and inform the agent.
• Process: Wrap tool code in a try-catch block.
• Benefits: Prevents application crashes and allows the agent to correct its mistakes.

19. Be Very Careful to Only Return What the LLM Needs to Know

• Main Point: Return only relevant information to the LLM.
• Example: When using the Shopify API, return only the data, not the metadata.
• Reason: Avoid overwhelming the LLM with unnecessary information.

20. Anatomy of a Good Tool

• Components:
  • Detailed prompt with examples.
  • Try-catch block for error handling.
  • Specific error messages for the agent.
  • Formatted output with only relevant information.

Conclusion

Building effective AI agents requires careful consideration of various factors, including managing hallucinations, planning and prototyping, prompt engineering, LLM selection, memory management, and tool design. By implementing the lessons learned from building hundreds of AI agents, developers can create more reliable, accurate, and efficient AI solutions. The key takeaways emphasize the importance of human oversight, specialization, clear instructions, and continuous monitoring and improvement.