Build a Multimodal Live Streaming Agent with ADK

Key Concepts

• Multimodal Live Agents
• Gemini API
• Agent Development Kit (ADK)
• Proxy Server
• Tool Handler
• Live Request Queue
• Events (Interruption, Function Call, Function Response, Data)
• Voice Config
• Runner.runLive

Building Multimodal Live Agents with Gemini API

• Architecture: The initial architecture involves four key components:
  • Client: A UI in the browser for user interaction.
  • Proxy: A Genai Python SDK proxy that establishes the connection to the Multimodal Live API, handles tool usage, and manages authentication. It acts as the central hub, routing requests between the client and the Live API.
  • Live API: The core of the application, provided by Google, handles the AI processing.
  • Tools: Custom-built components, such as a "get weather tool" running as a Cloud Run function, that extend the agent's capabilities.
• Workflow:
  1. The user interacts with the client (e.g., asks "What is the weather in London?").
  2. The client sends the request to the proxy.
  3. The proxy forwards the request to the Gemini Multimodal Live API.
  4. The Live API recognizes the need for a tool (e.g., weather API).
  5. The Live API initiates a function call, specifying the required tool and parameters (e.g., get_weather(city="London")).
  6. The proxy's tool handler receives the function call.
  7. The tool handler matches the function call to the corresponding tool (e.g., the "get weather tool").
  8. The tool executes, interacting with an external API (e.g., OpenWeather API).
  9. The tool returns the data to the Live API.
  10. The Live API formulates a human-readable response (e.g., "The weather in London is scattered clouds with a temperature of 14° and 62% humidity").
  11. The Live API sends the response back to the proxy.
  12. The proxy forwards the response to the client.
  13. The client displays the response to the user.
• Tool Handler: The tool handler is a crucial component on the server side. It receives function calls from the Live API and executes the corresponding tools. The challenge lies in correctly matching the function call with the appropriate tool.
• Example: The "get weather tool" is implemented as a Cloud Function. It interacts with the OpenWeather API to retrieve real-time weather information.
• Demonstration: A live demonstration shows the agent responding to a weather query, illustrating the entire workflow.

Building Multimodal Live Agents with Agent Development Kit (ADK)

• Simplification: The Agent Development Kit (ADK) simplifies the development process by abstracting away much of the complexity involved in managing websockets and interacting directly with the Gemini Live Streaming API.
• Architecture: The ADK-based architecture replaces the proxy server with a streamlined backend implementation.
• Key Functions: Two main functions are required:
  • handle_client_messages: This function receives audio, text, or video data from the client and pushes it into a live_request_queue. ADK handles the wrapping and formatting of these messages for the Gemini Live Streaming API.
  • handle_agent_responses: This function receives events from ADK, which represent responses from the Gemini model. These events can include interruptions, function calls, function responses, and data (text, audio, images). The function processes these events and sends the appropriate responses back to the client.
• Code Snippets:
  • The handle_client_messages function demonstrates how to add audio, image, and text data to the live_request_queue.
  • The handle_agent_responses function shows how to process different types of events, such as interruptions, function calls, and data responses.
• Event Handling: ADK provides events that simplify the handling of different response types from the Gemini model. These events include:
  • Interruption: Indicates that the agent has been interrupted.
  • Function Call: Contains the name and arguments of a function that needs to be executed. ADK triggers the tool behind the scenes.
  • Function Response: Contains the response from a function that has been executed.
  • Data: Contains the actual data from the agent, such as text transcripts or audio data.
• Agent Instantiation: Creating a live agent with ADK involves:
  • Voice Config: Configuring the voice settings, such as the voice type.
  • Runner Configuration: Enabling features like audio transcripts.
  • runner.runLive: Using the runner.runLive method to start the live agent, passing in the configuration.
• Agent Constructor: The agent constructor is similar to that of a regular agent, defining the model type, name, description, instructions, and tools.
• Demonstration: A live demonstration showcases the ADK-powered agent interacting with the user, answering questions about Google Cloud services and the Agent Development Kit. The agent is interrupted multiple times, demonstrating its ability to handle interruptions gracefully.
• Benefits: ADK simplifies the development of live agents by:
  • Abstracting away the complexities of websocket management.
  • Providing a high-level API for interacting with the Gemini Live Streaming API.
  • Handling tool execution automatically.
  • Providing events for easy processing of agent responses.

Synthesis/Conclusion

The video demonstrates two approaches to building multimodal live agents with Google's Gemini API. The first approach involves a more complex architecture with a proxy server and manual handling of websockets and tool execution. The second approach utilizes the Agent Development Kit (ADK), which significantly simplifies the development process by abstracting away much of the complexity. ADK provides a high-level API for interacting with the Gemini Live Streaming API, handles tool execution automatically, and provides events for easy processing of agent responses. The ADK approach requires only two main functions and allows developers to focus on building the agent's core logic and functionality. The video concludes that ADK makes it much easier for developers to build live agents, enabling them to focus on creating intelligent and interactive AI systems.