Build Hour: AgentKit

Key Concepts

• Agent Kit: A suite of tools and best practices from OpenAI designed to simplify the development, deployment, and evaluation of AI agents.
• Agent Builder: A visual workflow builder within Agent Kit for creating and orchestrating AI agents.
• Chat Kit: A customizable UI component for deploying and interacting with AI agents, allowing for brand customization and rich user experiences.
• Eval Platform: A system for testing, evaluating, and optimizing AI agents at scale, including features for trace grading and automated prompt optimization.
• Agents SDK: The underlying code library that powers Agent Kit, offering programmatic control over agent creation and orchestration.
• MCP Server (Model-Centric Protocol Server): A protocol that enables agents to securely connect to and interact with external tools and data sources.
• Connector Registry: A feature within Agent Kit for safely connecting data and tools to agents.
• Trace Grading: A feature within the Eval Platform that allows for the evaluation of agent execution traces to identify issues and areas for improvement.
• Automated Prompt Optimization: A tool within the Eval Platform that uses evaluation data to automatically refine agent prompts.
• Vector Stores: A method for storing and retrieving information, often used for knowledge retrieval by agents.

Agent Kit: A Comprehensive Overview

This session introduces OpenAI's Agent Kit, a new suite of tools designed to empower developers in building, deploying, and scaling AI agents. The presentation highlights the significant improvements Agent Kit offers over previous methods, emphasizing a more streamlined and efficient development process.

1. The Evolution of Agent Development

Previously, building AI agents was a complex and time-consuming endeavor. Key challenges included:

• Complex Orchestration: Agents had to be written entirely in code, making updates prone to breaking changes.
• Tool Integration: Securely connecting tools required custom coding.
• Evaluation Difficulties: Testing and trusting agents at scale involved manually extracting data across multiple systems.
• Prompt Optimization: This process was slow and manual, relying on trial and error.
• UI Development: Building a user interface for agents added weeks or months to the development cycle.

Agent Kit aims to address these pain points by introducing a more integrated and user-friendly approach.

2. Agent Kit: Key Components and Features

Agent Kit offers a comprehensive tech stack for agent development:

• Agent Builder:

  • Visual Workflow Builder: Allows for the creation of agent workflows visually, reducing the need for extensive coding.
  • Versioning: Ensures that updates do not introduce breaking changes.
  • Connector Registry: A secure administrative center for connecting data and tools.
  • Built-in Evals: Integrated evaluation tools, including support for third-party models.
  • Automated Prompt Optimization: Tools to automatically refine prompts, saving time and effort.
  • Chat Kit: A customizable UI component for deploying agents, which can be hosted by the user or OpenAI.

• Agent Builder Workflow:

  • Model Selection: Choose the models to deploy agents with.
  • Tool Connection: Integrate various tools and data sources.
  • Prompt Writing and Automation: Develop and optimize prompts.
  • Guardrails: Implement safeguards to ensure agents perform as expected, even with unexpected queries.
  • Deployment: Deploy agents to Chat Kit.
  • Optimization: Use the Eval Platform to optimize agents based on real-world data and human feedback.

3. Use Cases for Agents

Agent Kit is already being utilized by startups and Fortune 500 companies for a wide range of applications, including:

• Customer Support Agents: Triaging and answering chat-based customer support tickets.
• Sales Assistants: Aiding sales teams in various tasks.
• Internal Productivity Tools: Enhancing team efficiency and reducing duplicate work.
• Knowledge Assistants: Providing information and conducting research (e.g., document research).

The Agent Builder includes templates showcasing these popular use cases.

4. Real-World Example: Go-to-Market Assistant

A practical demonstration showcases the creation of a go-to-market assistant to save sales time and increase revenue. This involves building three core agents:

• Data Analysis Agent: Pulls data from sources like Databricks.
• Lead Qualification Agent: Gathers information from the internet.
• Outbound Email Generation Agent: Crafts personalized emails, potentially incorporating campaign details from uploaded files.

4.1. Building the Workflow in Agent Builder

The demo illustrates a step-by-step process:

• Start Node and Agent Node: The fundamental building blocks of a workflow. The Agent Node is powered by the Agents SDK.
• Question Classifier Agent:
  • Purpose: To classify incoming queries into categories like "qualification," "data," or "email."
  • Structured Output: The model is forced to output in a predefined schema (e.g., an enum for "category") to ensure consistent routing.
  • Prompt Engineering: The presenter emphasizes using ChatGPT and GPT-4 for initial prompt generation and refinement within Agent Builder.
• State Management: The set state icon allows for storing output values (e.g., the classified "category") as variables for subsequent steps.
• Conditional Branching: Based on the "category" state, the workflow can branch to different agents (e.g., "data analysis agent" or other logic for email/qualification).
• Data Analysis Agent with External Tools:
  • Tool Integration: Demonstrates connecting to a Databricks MCP server using a personal access token for authentication.
  • MCP Server Capabilities: Supports both read and write actions. Out-of-the-box MCP servers for services like Gmail and SharePoint are available.
  • Tool Selection: The model can be restricted to a subset of available functions from the MCP server.
  • Testing and Iteration: The ability to test queries and see the model's execution steps in real-time is highlighted.
  • Inline Changes: Prompts can be modified on the fly to adjust output formatting (e.g., requesting natural language results).
• Information Gathering Agent (for Research):
  • Purpose: To search the internet for specific company information (legal name, employee count, description, revenue, geography).
  • Structured Output Schema: Defines the expected output format for gathered information.
  • Web Search Tool: The agent is equipped with a web search tool.
• Email Generation Agent:
  • File Uploads: Supports uploading files (e.g., PDFs with campaign information or email writing SOPs) to inform email generation.
  • Vector Store Integration: Allows attaching existing vector stores or adding them via API for knowledge retrieval.
• Lead Enhancement Agent:
  • Output Schematic: Generates a schematic for assigning account executives based on gathered information.
  • Prompt Generation: Agent Builder can output a prompt as a starting point for this agent.
• Rich Widgets: Agent Builder supports rich widgets for multimodal outputs, going beyond plain text or JSON. These can be previewed and exported.
• Testing and Debugging: The ability to test workflows live and debug issues is a key feature. Traces of model execution are saved for detailed analysis.
• Hosting Workflows: Workflows built in Agent Builder can be hosted using Chat Kit or integrated into custom codebases.

4.2. Deployment with Chat Kit

• Customizable UI: Chat Kit allows for full brand customization of the chat interface, including color schemes, font families, and starter prompts.
• Hosting Options: Chat Kit can be hosted by the user or OpenAI.
• Example: Utility Bill Analysis: A demo shows an agent that connects to an MCP server, pulls billing history, analyzes bills, and presents a rich graph visualization to the user, demonstrating the power of custom widgets.
• Email Widget: An email widget can be used to draft emails, with a simple click to send.
• Widget Gallery: A gallery of pre-built widgets is available, with the ability to view their code and even generate new widgets using natural language.
• Real-World Integration: An example demonstrates controlling a JavaScript-rendered globe on a website via natural language commands, showcasing Chat Kit's ability to integrate with existing web applications.

5. Evaluating and Optimizing Agents with the Eval Platform

Henry introduces the Eval Platform, crucial for testing and trusting agents at scale.

• Node-Level Evaluation:
  • Evaluate Button: An "evaluate" button within Agent Builder allows users to open a specific agent node in the Eval Platform.
  • Dataset UI: A visual interface for building simple evaluations.
  • Data Import: Users can import data and ground truth values into datasets.
  • Generation and Evaluation Stages: The process involves running generations and then evaluating them.
  • Annotations and Graders: Users can add free-text feedback and create custom "graders" (evaluators) with specific criteria (e.g., requiring upside/downside arguments, competitor comparisons, and a buy/sell/hold rating).
  • Automated Prompt Optimization: The platform can automatically rewrite prompts based on evaluation data and grader outputs, saving manual prompt engineering effort.
• End-to-End Evaluation (Trace Grading):
  • Traces: Agents emit traces that record every step of their execution.
  • Trace Analysis: Users can click through traces to identify specific issues (e.g., unwanted third-party sources being cited).
  • Grader Rubrics: Sets of criteria can be defined as "grader rubrics" to evaluate a large number of traces at scale.
  • "Grade All" Button: This feature exports traces and graders to a new evaluation, allowing for efficient assessment of many agent runs.
• Best Practices for Evals:
  • Start Simple and Early: Begin with a few inputs and a simple grader at the start of a project.
  • Eval-Driven Development: Rigorously test prototypes and quantitatively measure improvements.
  • Use Human Data: Real user data provides more representative inputs than hypothetical or synthetic data.
  • Annotate Generations and Align Graders: Ensure subject matter expertise is encoded into the system.

6. Real-World Examples and Customer Success

• RAMP: Built a procurement agent prototype 70% faster using Agent Kit, equivalent to two engineering sprints instead of two quarters.
• Ripling: Collaborated on a project, highlighting the value of Agent Builder in aligning subject matter experts and building logically sound workflows.
• HubSpot: Used Chat Kit to enhance their Breeze AI assistant, saving weeks of front-end development time.
• Carlile and Bain: Achieved a 25% efficiency gain in their eval datasets.
• Early Adopters: Agent Kit is powering tech stacks at startups and Fortune 500s for diverse use cases like work assistants, procurement agents, policy agents, merchandising intelligence, and code modernization.

7. Q&A and Future Sessions

• For Loops: Agent Builder supports while loops for conditional iteration, but not traditional for loops. The Agents SDK can be used for more complex loop structures in custom code.
• Agent Kit vs. Agents SDK: Agent Kit is a curated suite of tools, while the Agents SDK is the underlying code library. Agent Kit aims to provide a more opinionated and user-friendly experience, with ongoing efforts to achieve parity.
• MCP Servers: Users can leverage out-of-the-box MCP servers for common services (Gmail, SharePoint) or build their own for custom authentication and write capabilities.
• Classifier Agent Recommendation: Using a classifier agent with branching logic is recommended when an agent's complexity increases, or when different use cases require distinct tool sets or output interpretations. This prevents model confusion and improves performance.
• Multimodal Use Cases: Agent Kit supports file inputs for analyzing images and files, and this behavior propagates to Chat Kit.
• Resources: Links to Agent Kit docs, a cookbook, Chat Studio, and the Build Hour refill on GitHub are provided.
• Upcoming Build Hours:
  • November 5th: Agent RFT (customizing models for tool calling, custom graders).
  • December 3rd: Agent Memory Patterns.

Conclusion

Agent Kit represents a significant leap forward in AI agent development, offering a visual builder, integrated evaluation tools, and a customizable deployment UI. By simplifying complex processes and providing robust testing capabilities, Agent Kit empowers developers to build and scale sophisticated AI agents more efficiently and effectively. The emphasis on iterative development, real-world data, and automated optimization ensures that agents can be trusted and perform at scale.