I Built a Voice Agent that Handles my Daily Tasks

Key Concepts:

• Voice Agent API: A unified conversational AI API combining transcription, LLM generation, and speech generation.
• Automatic Speech Recognition (ASR): Converts spoken language into text.
• Large Language Models (LLMs): AI models used for natural language understanding and generation.
• Function Calling/Tool Calling: LLM's ability to use external tools or functions based on user input.
• Deepgram: A company providing speech recognition and conversational AI APIs.
• Vendor Lock-in: Being dependent on a single vendor for technology or services.

1. Introduction to the Voice Agent

• The video introduces a voice agent, "Talia," capable of managing calendars, checking emails, and setting tasks.
• Example interaction: The user asks Talia about an interview on August 7th, 2025, at 6:18 a.m. Talia provides the time and attendees (Bob Smith, Carol Williams, and David Brown).
• The user requests rescheduling to August 22nd at 3 p.m. Talia drafts a message: "Dear Bob, Carol, and David, I hope this message finds you well. Due to a scheduling conflict, I would like to propose rescheduling our interview to August 22nd at 3 p.m. Please let me know."

2. Deepgram's Voice Agent API

• The voice agent is powered by Deepgram's Voice Agent API, a unified conversational AI API.
• The API integrates transcription, LLM generation, and speech generation into a single service.
• LLM Options: Users can bring their custom LLMs, offering flexibility compared to vendor lock-in solutions like OpenAI's advanced voice mode. Supported models include GPT4 mini and Cloud 3 Haiku.
• Automatic Speech Recognition (ASR): The API includes ASR, eliminating the need for a separate voice activity detection layer, enabling interruptions.
• Pricing: Deepgram offers a free tier with $200 of credit upon signup.

3. Setting Up and Configuring the Voice Agent

• The video demonstrates how to set up the voice agent and its components.
• Playground: Deepgram offers a playground for testing the voice agent with pre-configured models (GPT4 mini, Cloud 3 Haiku) and voices.
• Function Calling: The agent can use external tools or functions. Examples include calculation and ending conversations.
• Agent Configuration: Setting up a custom agent involves configuring voice input/output, selecting a speech generation model (e.g., Talia), and defining a system prompt with tool descriptions.
• Workflow: The LLM analyzes user input, selects a tool based on its description, executes the tool, receives results, and generates a response.
• Process: User speech is transcribed, passed to the LLM, which uses available tools, generates text, converts it to speech, and sends it to the client.

4. Code Example and Implementation

• The code example is based on a Deepgram repository, modified for specific functionality. The repository link is in the video description.
• client.py: Implements a Flask API, acting as a bridge between the front end and back end.
• agent_template file: Contains the main agent configurations, including the API endpoint, audio settings, speech-to-text model, LLM (GPT4 mini), and list of functions/tools.
• Functions/Tools: Defined with descriptions, names, and input/output parameters. Examples include getting calendar events, creating meetings, getting email lists, composing emails, and managing to-do lists.
• businesslogic.py: Generates mock data for the agent to use, creating simulated calendar events, emails, and tasks.
• index.html: Implements the user interface.
• Client-Side Functions: Handle microphone setup, streaming data, and agent behavior.

5. Local Setup Instructions

• Install PortAudio: Required for audio input/output. Instructions are provided for macOS.
• Virtual Environment: Set up a virtual environment and install dependencies from requirements.txt using pip install requirements.txt.
• Deepgram API Key: Obtain an API key from the Deepgram account (offering $200 credit) and set it in the terminal or .env file.
• Run client.py: Execute the client.py file to start the voice agent.
• UI: The UI allows selecting different voices and input devices.

6. Demonstration of the Local Setup

• The video demonstrates the locally set up voice agent.
• The agent is asked about the calendar for the day, which includes a tentative meeting from 10:34 a.m. to 11:34 a.m. at the client site.
• The agent is asked what day it is, and it responds with "Tuesday, July 8th, 2025."
• The agent is asked to look at pending tasks and identify the highest priority tasks.

7. Conclusion

• The video encourages viewers to explore Deepgram's Voice Agent API.
• The presenter thanks the viewers and concludes the video.

Key Takeaways:

• Deepgram's Voice Agent API offers a unified solution for building conversational AI applications.
• The API's flexibility allows users to bring their own LLMs and customize the agent's behavior.
• The function calling feature enables the agent to interact with external tools and services.
• The provided code example and setup instructions simplify the process of building and deploying a voice agent.