This Framework Finally Makes Agent Swarms Work

Agency Swarm 1.0 Summary

Key Concepts:

• Multi-agent systems
• OpenAI Agents SDK
• Agency Swarm framework
• Tools (Base Tools, Function Tool Decorators)
• Open Source Model Support (LightLLM)
• Multi-Agent Workflows
• Structured Outputs
• Agent Visualization
• Copilot Integration (GUI)
• RAG (Retrieval-Augmented Generation)
• Connectors (Hosted MCP Tools)
• MCP Servers
• Observability (Tracing)
• Guardrails (Input/Output Validation)
• Hybrid Communication Flows (Handoff, Orchestration)
• Custom Communication Flows
• FastAPI Integration
• Production Deployment

Introduction

The video discusses the limitations of existing multi-agent systems and introduces Agency Swarm 1.0, a framework designed to make multi-agent systems reliable in production. It highlights the framework's foundation on real-world deployment experience and its complete redesign based on that experience.

Background and Motivation

• Early Chatbots (2023-Early 2024): Initial LLM use cases were primarily chatbots with custom data and instructions.
• Shift to AI Agents (Mid-2024): Clients wanted AI to perform actions, coinciding with the release of function calling. The speaker emphasizes that real AI agents were not possible before function calling.
• Hallucination Problem: Agents with numerous tools (e.g., 10-13 tools with GPT-3.5 Turbo) suffered from hallucinations.
• Multi-Agent Solution: Splitting agents (e.g., main agent + database analysis agent) and combining them reduced hallucinations. This led to the development of Agency Swarm.
• Initial Release & YouTube Success: The initial system built for a client was released on YouTube, gaining significant traction and leading to the open-source project.

Transition from Assistants API to OpenAI Agents SDK

• Assistants API Limitations: Limited open-source model support and high latency (2-3 seconds per response).
• OpenAI Agents SDK Advantages: Greater flexibility and control compared to other frameworks.
• OpenAI Agents SDK Drawbacks: Lacked built-in AI agent loop and agent collaboration features (only handoff communication).
• Agency Swarm Extension: Agency Swarm extends OpenAI Agents SDK by adding missing features like collaborative communication, utility methods for file handling and schema parsing, and instructor type-based tool classes.

Key Features and Functionality

Tools

• Base Tools (Pydantic): Defining tools using Pydantic models for validation and reliability. The strict parameter can prevent parameter hallucination.
• Function Tool Decorators: Using function decorators for simpler tools with less complex validation needs.

Open Source Model Support (LightLLM)

• One-Line Integration: Using LightLLM, open-source models can be integrated with a single line of code.
• Model Flexibility: Supports models like Gemini 1.5 Pro and Anthropic Claude Sonnet.
• LightLLM Modify Params: The lightllm_modify_params=True parameter is used to avoid issues with parameter differences between LLM providers.
• Default Tools from Other Providers: The framework allows using default tools from other model providers like Google's web search tool.

Multi-Agent Workflows and Structured Outputs

• Structured Output Models: Defining Pydantic models for agent outputs to ensure consistent response formats.
• Guaranteed Response Format: Agents are guaranteed to provide responses in the defined format, improving reliability in complex workflows.
• Example: Risk analyst agent providing risk assessment with risk level score, key risks, and recommendation. Report generator agent providing executive summary and market position.

Agent Visualization

• Visualization Method: A special visualization method helps understand agent communication flows and tool usage.
• Graphical Representation: Shows which agents use which tools and communicate with other agents.

Copilot Integration (GUI)

• GUI Interface: Supports a GUI interface for quickly connecting agents to various front ends.
• Copilot Demo Launcher: A simple demo can be run with Copilot, a UI library for AI applications.
• Fast Responses: The Responses API enables fast agent responses.

RAG (Retrieval-Augmented Generation)

• File Indexing: Adding a files folder automatically indexes files, uploads them to OpenAI, and adds a file search tool to the agent.
• Citations: The include_search_results=True parameter enables the agent to provide citations.

Connectors

• Hosted MCP Tools: A new way to define MCP services from OpenAI for connecting to services without publicly available MCP servers (e.g., Google Calendar, Gmail).
• Google OAuth Token: Requires obtaining a Google OAuth token through the Google OAuth Playground.
• Replacing Platforms like Zapier: OpenAI connectors are essentially replacing platforms like Zapier by adding connectors to their own API.

MCP Servers

• MCP Server Support: Supports any kind of MCP server, including Python MCP servers and hosted MCP servers.
• Hosted MCP Tool: The recommended approach for hosted MCP servers.

Observability

• Tracing: Supports observability with AgentTaps, LangSmith, and default OpenAI tracing.
• Trace Visualization: Traces can be viewed on platform.openai.com to see agent communication, tool calls, and data flow.

Guardrails

• Input/Output Validation: Guardrails are used to validate agent inputs and outputs, ensuring correct communication between agents.
• Guardrail Functions: Custom guardrail functions can perform string comparisons, database lookups, or LLM-based checks.
• Trip Wire: The trip_wire_triggered parameter indicates whether the guardrail was triggered.

Hybrid Communication Flows

• Combining Communication Types: Combines handoff communication (agent replacement) with orchestration (main agent calls sub-agents).
• Custom Send Message Classes: Creating custom send_message classes to add extra parameters and control communication flow.
• Example: Project manager sends a task to a developer, who then hands off the task to a security expert. The security expert completes the task and responds to the project manager directly.

Custom Communication Flows

• Custom Send Message Classes: Creating custom send_message classes to add custom parameters and descriptions for agent communication.
• Addressing Cognition's Concerns: Addressing concerns about agents missing key moments by including context in the send_message tool.

FastAPI Integration

• One-Line Server Deployment: Using run_fastapi to deploy agents on a server with a single line of code.
• Create Agency Method: Agents are initialized under the create_agency method.
• API Endpoint: The agency is hosted on the /my_agency endpoint.

Conclusion

Agency Swarm 1.0 provides a comprehensive framework for building and deploying reliable multi-agent systems in production. By extending OpenAI Agents SDK and adding features like collaborative communication, structured outputs, and guardrails, it addresses the limitations of existing frameworks and enables developers to create more effective and robust AI agents. The framework's flexibility, combined with its ease of deployment through FastAPI integration, makes it a valuable tool for building AI-powered solutions. The speaker encourages users to experiment with the framework and combine different features to create unique applications. The upcoming platform release will further simplify deployment by allowing users to deploy agents from their GitHub repos with a single click.