Master AI Voice Agents with 11Labs + n8n (Beginner to Pro Guide)

Key Concepts

• Voice AI Agents: AI-powered agents that interact with users through voice.
• 11 Labs: A platform used for voice cloning, text-to-speech, and building voice AI agents.
• NAN (n8n): A workflow automation platform used as a backend to process information and connect different services.
• LLM (Large Language Model): The AI model used to generate responses and understand user input. Examples include Google's Gemini, GPT-4, and Claude.
• System Prompt: Instructions given to the LLM to define its persona, behavior, and how to use tools.
• Knowledge Base: A repository of information that the AI agent can use to answer questions.
• Tools: Functions or services that the AI agent can use to perform tasks, such as retrieving information or booking appointments.
• Webhooks: A way for different applications to communicate with each other in real-time.
• Body Parameters: Data sent in the body of a webhook request.
• Query Parameters: Data sent in the URL of a webhook request.
• From AI Functionality: A feature in NAN that allows you to extract data from the output of an AI agent.

1. Introduction to Voice AI Agents with 11 Labs and NAN

• The video introduces a deep dive series on building voice AI agents using 11 Labs for voice interaction and NAN as a backend for data processing and workflow automation.
• The initial focus is on embedding a voice agent widget on websites, with future plans to explore Twilio integration for phone calls.
• Actionable Insight: The series aims to provide practical guidance on creating powerful voice AI agents by combining 11 Labs and NAN.

2. Setting Up 11 Labs

• Account Creation: Users need to create an account on 11labs.io.
• Pricing Tiers: 11 Labs offers various pricing options, including a free tier. The "Starter" or "Creator" plans are recommended for beginners due to the generous credit allocation. The creator plan is highlighted as a good deal with a 50% discount for the first month.
• Voice Cloning: The platform allows users to clone voices with high quality, available in the "Creator" plan.
• Navigation: After logging in, users can access the "Agents" section on the left-hand side to manage their AI agents.

3. Creating and Configuring an AI Agent in 11 Labs

• Agent Creation: Users can create a new agent by clicking the "+" button or "Create an Agent."
• Agent Templates: 11 Labs provides pre-built agent templates like "Support Agent," "Map Tutor Agent," and "Video Game Character Agent," each with pre-selected LLMs and system prompts.
• Blank Template: Users can also start from scratch with a blank template for full customization.
• Agent Tab:
  • Agent Language: Supports multiple languages for voice interaction.
  • First Message: The initial message spoken by the agent when a user initiates a call.
  • System Prompt: Defines the agent's persona and behavior.
  • LLM Selection: Allows users to choose the LLM to power the agent. Google's Gemini 1.5 Flash is the default for its speed in real-time conversations. Other options include GPT-4 Turbo, Mini, and Claude 2.5.
  • Temperature: Controls the creativity and randomness of the LLM's responses.
  • Limit Token Usage: Sets a limit on the number of tokens used by the LLM.
  • Knowledge Base: Enables adding files (PDF, text, docs, HTML, EPUB), URLs, or manual text to provide the agent with information.
  • Tools: Allows integration with external services via webhooks.
  • Secrets: Used for authentication.
• Voice Tab:
  • Voice Selection: Users can choose from a variety of pre-existing voices or clone their own.
  • Use Flash: Option for low-latency use cases.
  • TTS Output Format: Controls the quality of the text-to-speech output.
  • Lexicon Dictionary: Allows uploading files to apply pronunciation replacements.
  • Optimization: Settings for streaming latency, stability, and similarity.
• Analysis Tab:
  • Evaluation Criteria: Allows setting evaluation criteria to collect data during conversations.
• Security Tab:
  • Enabling Overrides: For more advanced features.
• Limitations Tab:
  • Timeout: Sets a timeout for inactivity.
  • Max Conversation Duration: Limits the length of conversations.
• Widget Tab:
  • Customization: Allows customizing the appearance of the embedded widget, including color, font, and avatar.
  • Code Snippet: Provides a code snippet to embed the widget on a website.
• Testing: The "Test AI Agent" button allows users to test the agent's behavior and ensure changes are implemented correctly.
• Actionable Insight: The agent and voice tabs are the most frequently used during initial setup.

4. Building a Tech Support AI Agent

• Customizing the First Message: The initial message can be modified to greet users with a specific name and purpose.
• Modifying the System Prompt: The system prompt can be changed to define the agent's persona and behavior.
• Adding a Knowledge Base: A PDF document containing FAQs about an AI workshop is uploaded to provide the agent with relevant information.
• Instructing the Agent: The system prompt is updated to instruct the agent to rely on its knowledge base when answering questions about the AI workshop.
• Testing the Knowledge Base: The agent is tested to ensure it can answer questions based on the uploaded document.
• Using a URL as a Knowledge Base: A URL is used as a knowledge base, allowing the agent to scrape information from a website.
• Actionable Insight: The system prompt is crucial for defining the agent's behavior and ensuring it uses the knowledge base effectively.

5. Integrating with NAN via Webhooks

• Creating a Tool: A new tool is created using the "Webhook" option.
• Tool Name and Description: The tool is named "test_tool" and given a description that instructs the LLM when to use it.
• Method and URL: The method is set to "POST," and the URL is set to the webhook URL from NAN.
• NAN Setup:
  • A new workflow is created in NAN.
  • A "Webhook" trigger node is added to listen for incoming requests.
  • The HTTP method is set to "POST," and the "Respond To Webhook" option is enabled.
  • A "Respond to Webhook" node is added to send data back to 11 Labs.
• Body Parameters:
  • A body parameter is enabled to send data from 11 Labs to NAN.
  • The parameter is named "name," and its description instructs the agent to ask the customer for their name.
• Testing the Integration:
  • The workflow is tested to ensure that data is being sent from 11 Labs to NAN.
  • The agent is instructed to use the "test_tool" when a customer asks about the weather.
  • When the agent is asked about the weather, it prompts the user for their name, and the name is sent to NAN.
• Actionable Insight: Webhooks enable seamless communication between 11 Labs and NAN, allowing for complex workflows and data processing.

6. Building a Customer Reservation System

• Creating a New Agent: A new agent is created for a restaurant called "Joe's Diner."
• System Prompt: The system prompt is updated to reflect the agent's role as a customer support agent for Joe's Diner.
• Knowledge Base: A PDF document containing FAQs about Joe's Diner and a text file containing the menu items and prices are added to the knowledge base.
• Creating a Reservation Tool: A new tool is created using the "Webhook" option.
• Tool Name and Description: The tool is named "reservations," and its description instructs the LLM to use it to make reservations for Joe's Diner.
• Body Parameters:
  • Body parameters are enabled to collect the customer's name, date of reservation, time of reservation, and number of people.
• Google Sheets Integration:
  • A new Google Sheet is created to store reservation data.
  • A "Google Sheets" node is added to the NAN workflow to append rows to the sheet.
  • The columns in the Google Sheet are mapped to the corresponding body parameters from the webhook.
• Testing the System:
  • The agent is tested to ensure it can answer questions based on the knowledge base and collect reservation information.
  • The workflow is tested to ensure that reservation data is being sent to the Google Sheet.
• Actionable Insight: By integrating 11 Labs with NAN and Google Sheets, a fully automated customer reservation system can be created.

7. Adding an AI Agent to the NAN Workflow

• Replacing Direct Integration: Instead of directly connecting the webhook to the Google Sheet, an AI agent is added to the NAN workflow.
• AI Agent Configuration:
  • A "Tools Agent" node is added to the workflow.
  • A chat model (e.g., GPT-4 Mini, Claude 3.5) is selected.
  • The Google Sheets tool is attached to the AI agent.
• System Prompt:
  • A system prompt is created to instruct the AI agent on its role as a customer support agent for Joe's Diner.
  • The prompt specifies that the agent should use the Google Sheets tool to store reservation details.
• Mapping Columns with From AI Functionality:
  • The "From AI" functionality is used to dynamically extract data from the AI agent's output and map it to the columns in the Google Sheet.
• Testing the System:
  • The agent is tested to ensure it can collect reservation information and send it to the AI agent.
  • The workflow is tested to ensure that the AI agent can extract the data and add it to the Google Sheet.
• Actionable Insight: Adding an AI agent to the NAN workflow allows for more complex data processing and decision-making.

8. Building a More Complex AI Agent for AI Workshop

• Overview of the System: A more complex AI agent is created for an AI workshop, with access to multiple tools, including:
  • SERP API: For general information retrieval.
  • Appointments: For managing calendar appointments.
  • Google Calendar: For accessing and managing calendar events.
  • Google Sheets: For storing contact information.
  • Vector Database: For retrieving information about the AI workshop school community.
• System Prompt: A detailed system prompt is created to instruct the AI agent on its various roles and how to use each tool.
• Query Parameters: Query parameters are used to send additional information to the NAN workflow, allowing the AI agent to determine the purpose of the request.
• Respond to Webhook: The "Respond to Webhook" node is configured to stringify the output from the AI agent and send it back to 11 Labs.
• Actionable Insight: By combining multiple tools and a detailed system prompt, a highly versatile AI agent can be created to handle a wide range of tasks.

9. Key Takeaways

• Voice AI agents can be built by combining 11 Labs for voice interaction and NAN for backend processing.
• The system prompt is crucial for defining the agent's behavior and ensuring it uses the knowledge base and tools effectively.
• Webhooks enable seamless communication between 11 Labs and NAN, allowing for complex workflows and data processing.
• Adding an AI agent to the NAN workflow allows for more complex data processing and decision-making.
• By combining multiple tools and a detailed system prompt, a highly versatile AI agent can be created to handle a wide range of tasks.
• Testing and iteration are essential for optimizing the performance and behavior of voice AI agents.
• The choice of LLM can significantly impact the speed and quality of the agent's responses.
• Analyzing conversation history and collecting data can provide valuable insights into the agent's performance.
• The series aims to provide practical guidance on creating powerful voice AI agents by combining 11 Labs and NAN.