Build your first Voice Assistant with Agent Development Kit (ADK)

Key Concepts

• Voice Assistant Agent: An AI agent capable of understanding and responding to voice commands, managing tasks like calendar operations.
• ADK (Agent Development Kit): A framework for building AI agents, providing tools for agent creation, model selection, and tool integration.
• Gemini 2.0 Flash (Experimental): A low-latency, bidirectional voice and video model with function calling support, suitable for voice assistants.
• Tools: Specific functions or capabilities that an agent can use, such as listing, creating, editing, and deleting calendar events.
• FastAPI: A modern, high-performance web framework for building APIs, used to create a website interface for the voice agent.
• WebSockets: A communication protocol providing full-duplex communication channels over a single TCP connection, enabling real-time interaction between the agent and the website.
• Sessions: Stateful chat histories that store messages and state information for an agent's interactions.
• Live Mode: A real-time interaction mode for agents, allowing for immediate responses to voice or text input.
• MIME Type: A Multipurpose Internet Mail Extensions (MIME) type indicates the nature and format of a file or byte stream.

Voice Assistant Agent Overview

The video demonstrates how to build a voice assistant using ADK that can manage a user's Google Calendar. The agent, named Jarvis, can list, create, edit, and delete events through voice commands. The tutorial is divided into two phases:

• Phase 1: Covers the basics of creating a voice-assisted agent, including agent setup, model selection, instruction definition, and tool integration.
• Phase 2: Focuses on connecting the voice agent to a website using FastAPI, allowing for more control over voice configurations and a richer user interface.

Phase 1: Building the Basic Voice Agent

Agent Setup

• The agent is named "Jarvis" and its instructions are defined in the agent.py file.
• The model used is gemini-2.0-flash-experimental due to its low latency, bidirectional voice capabilities, and function calling support.
• Instructions: The instructions provide the agent with a high-level overview of its capabilities, including scheduling and calendar operations.
  • Guidelines are provided for each tool to help the agent understand how to use them effectively.
  • The agent is instructed to be concise and avoid outputting raw tool outputs.

Tool Integration

• The agent has access to four tools: list_events, create_events, edit_events, and delete_events.
• Each tool is a wrapper around the Google Calendar API, requiring proper authentication and credential setup.
• List Events: Retrieves events from the user's primary calendar within a specified date range.
  • If no date is provided, it defaults to today's date.
  • It formats the event data for the agent to use.
• Create Events: Creates a new event on the user's calendar with a specified summary, start time, and end time.
  • It uses the local time zone for the event.
• Edit Events: Modifies an existing event on the user's calendar, allowing changes to the title, start time, and end time.
  • Requires the event ID, which can be obtained from the list_events tool.
• Delete Events: Deletes an event from the user's calendar based on the event ID.
  • Requires user confirmation before deleting the event.

Testing with ADK Web

• The agent can be tested locally using adk web, which spins up a website on port 8000.
• This allows for basic text and voice interaction with the agent.
• However, adk web has limitations, such as the inability to see tool calls or customize voice configurations.
• Chrome is the recommended browser for testing.

Phase 2: Connecting the Agent to a Website with FastAPI

FastAPI Setup

• FastAPI is used to create a website that can communicate with the voice agent.
• The main.py file sets up the FastAPI application and defines the endpoints for handling WebSocket connections.
• Static files (HTML, JavaScript, CSS) are served from the static directory.

WebSocket Communication

• WebSockets enable real-time communication between the agent and the website (client).
• The start_agent_session function creates a new session and runs the agent in live mode.
• Agent Configuration:
  • The agent can respond with either audio or text.
  • Voice configurations, such as voice selection (e.g., Puck), can be customized.
  • The audio transcript can be displayed on the website.
• Client to Agent Messaging:
  • The client_to_agent_messaging function handles messages sent from the website to the agent.
  • Text messages are sent as plain text content.
  • Audio messages are encoded as base64 and sent as audio/pcm blobs.
• Agent to Client Messaging:
  • The agent_to_client_messaging function handles messages sent from the agent to the website.
  • It processes live events, such as turn completion and content updates.
  • Text messages are streamed to the website in partial snippets.
  • Audio messages are encoded as base64 and sent to the website for playback.

Website Implementation (app.js)

• The app.js file contains the JavaScript logic for the website.
• It establishes a WebSocket connection with the server.
• It handles incoming messages (audio and text) from the agent.
• It sends text and audio data to the agent.
• Key Functions:
  • startAudio(): Starts the audio recorder and streams audio data to the server.
  • sendMessage(): Sends text messages to the server.
  • WebSocket event handlers: Handle WebSocket open, message, close, and error events.

Demo and Explanation

• The video demonstrates the voice assistant in action, showing how it responds to voice commands and manages calendar events.
• Logs are used to illustrate the communication flow between the client and the agent, including the format of the messages and the data being exchanged.
• The demo highlights the real-time streaming of text and audio data.

Synthesis/Conclusion

The video provides a comprehensive guide to building a voice assistant using ADK and connecting it to a website with FastAPI. It covers the essential steps of agent creation, tool integration, and real-time communication, offering valuable insights into the underlying configurations and functionalities. The tutorial emphasizes the importance of model selection, instruction design, and data formatting for effective voice interaction. By following the steps outlined in the video, developers can create their own custom voice assistants capable of managing various tasks and providing a seamless user experience.