AI Agent Insights From 72 Hours of Expert Panels

AI Agents: Conference Season Cliff Notes

Key Concepts: AI Agents, Vertical AI Agents, Prompt Engineering, Evals (Evaluations), Agent Engineer, Business Workflow Decomposition, Voice Agents, Coding.

Opportunities for AI Agents

• Google's AI Integration: Google is heavily focused on integrating AI into its products, giving them agentic abilities. This trend extends to other major companies as well. 2025 is seen as the beginning of agentic AI products.
• Vertical AI Agents (YC Roundtable): The best opportunity for startups and smaller businesses lies in building specialized AI agents for specific industries and functions (B2B vertical).
  • SAS Boom Analogy:
    • Wave 1 (Obvious Consumer Applications): Incumbents (big companies) won (e.g., online versions of desktop software).
    • Wave 2 (Not Obvious Consumer Applications): Startups won due to risk and innovation (e.g., Uber, Airbnb, Instacart, DoorDash, Coinbase).
    • Wave 3 (B2B Vertical SAS): Largest category, startups succeeded by addressing niche needs incumbents ignored (e.g., Veeva for pharma, Gusto for payroll).
  • Vertical AI Agents Potential: Vertical AI agents can be 10x bigger than their SAS counterparts because they can replace both software and the people operating it.
  • Example: HR-specific AI agent that allows a CEO to have personalized conversations with all employees, potentially replacing the entire HR department.
• Getting Started with Vertical AI Agents:
  1. Choose a Vertical Niche: Select a niche where you have domain knowledge (law, accounting, marketing, etc.).
  2. Identify Repetitive Tasks: Automate repetitive, administrative, and boring tasks using AI agents.
  • Advantages of AI Agents: 24/7 availability, low cost, personalization.
  • Example: AI agent to automatically refresh a government website.

Skills to Learn for Building AI Agents

• Disclaimer: Many skills are needed, but prompt engineering and writing evals are crucial.
• Prompt Engineering: Designing and refining input instructions for generative AI models.
  • Importance:
    1. All instructions must be in a single prompt.
    2. Balance between clear instructions and flexibility for autonomous behavior.
  • Anthropic's Guidance: Start with a short prompt and experiment with different use cases.
  • Six-Component Framework: Define role, task, input, output, constraint, and capabilities/reminders within the prompt.
    • Example: Research assistant agent to summarize AI news and trends.
  • Advanced Techniques (YC Roundtable): Step-by-step reasoning, markdown, and XML tags for structuring prompts.
• Evals (Evaluations): Structured way to measure AI agent performance across tasks and scenarios.
  • Relationship to Prompt Engineering: Evals determine if a prompt is effective.
  • Importance: Thoroughly test AI agent behavior, especially edge cases.
  • Common Eval Categories:
    • Task completion
    • Planning and reasoning quality
    • Tool use accuracy
    • Robustness (error handling)
    • Latency and efficiency
  • Eval Tools: OpenAI's evaltools eval platform.
  • Evals as IP: Companies consider evals their "crown jewels" because they represent the successes and failures of user workflows. Deep domain knowledge is needed to create good evals.

Career Advice: Agent Engineer

• Agent Engineer Role (Harrison Chase, LangChain CEO): A new role focused on building robust AI agents for production environments.
• Required Skills:
  1. Prompting: Effective interaction with LLMs.
  2. Traditional Engineering: Building reliable systems and data pipelines (software and data engineers).
  3. Product: Domain knowledge to understand user workflows.
  4. Machine Learning: Good eval statistics and handling non-determinism (data scientists and researchers).

Nuggets of Insights

• Business Workflow Decomposition Framework:
  1. Observe: Look at what people are doing.
  2. Decompose: Break down the workflow into smaller tasks.
  3. Map the Flow: Identify task relationships and branches.
  4. Prototype: Build a simple agentic version.
  5. Evaluate and Iterate: Use evals to improve the agent.
• Future Focus (Andrew Ng):
  1. Prompt engineering and evals.
  2. Voice agents (underrated).
  3. Coding (everyone should learn).
• Opinion on Coding: Coded implementations are needed, but not everyone needs to code. People with business and domain knowledge can use no-code tools for prototyping and hire engineers for implementation.

Synthesis/Conclusion

The video provides a comprehensive overview of the AI agent landscape, emphasizing the opportunities in vertical AI agents, the crucial skills of prompt engineering and evals, and the emerging role of the agent engineer. It also offers practical advice on workflow decomposition and future trends, highlighting the importance of voice agents and coding skills. The key takeaway is that building successful AI agents requires a combination of technical expertise, domain knowledge, and a focus on solving specific industry problems.