Full Workshop: Realtime Voice AI — Mark Backman, Daily

Key Concepts

• Pipecat: An open-source Python framework for building voice and AI multimodal agents.
• Pipeline: A series of processors (boxes) that receive, process, and stream data (audio, video, text) in Pipecat.
• Transport: The input/output mechanism for data in a Pipecat pipeline (e.g., Daily, WebSockets).
• Processors: Individual components within a Pipecat pipeline that perform specific tasks (e.g., speech-to-text, LLM, text-to-speech).
• Cascaded Model: A traditional pipeline architecture where data flows sequentially through distinct processors.
• Speech-to-Speech Model: A newer type of LLM that natively accepts audio input and produces audio output, simplifying the pipeline.
• Context Aggregation: The process of collecting and formatting conversation history for LLMs.
• Function Schema: A universal schema for defining and translating function calls across different LLMs.
• VAD (Voice Activity Detection): A mechanism for detecting when a user starts speaking.
• WebRTC: A real-time communication protocol suitable for client-server applications.
• WebSockets: A communication protocol suitable for server-to-server applications.
• Gemini Live: Google's real-time audio dialogue model.

Pipecat Overview and Voice AI Challenges

• Pipecat is an open-source Python framework developed by Daily for building voice and AI agents. It has been around for over a year (since March 2024).
• Real-time voice applications are challenging due to high user expectations, requiring:
  • Good listening capabilities.
  • Smart and conversational behavior.
  • Connectivity to data stores.
  • Natural-sounding voice output.
  • Fast end-to-end communication (target: ~800 milliseconds).

Pipecat Pipeline Architecture

• The Pipecat pipeline is a multimedia pipeline consisting of interconnected processors.
• Processors receive input (audio, video), process it, and stream the data to subsequent processors.
• Example pipeline:
  1. Transport (Daily): Captures user audio.
  2. Speech-to-Text Service: Transcribes audio to text.
  3. LLM (Gemini): Processes text and generates output tokens.
  4. Text-to-Speech Service: Converts tokens to audio.
  5. Transport (Daily): Outputs audio to the user.
• Gemini Live simplifies the pipeline by integrating speech-to-text, LLM, and text-to-speech into a single processor.
• Pipecat offers orchestration and abstractions for common utilities like recording, transcription, and artifact generation.
• Modularity: Pipecat allows swapping out individual services (e.g., speech-to-text, LLM) without changing the underlying application code.
• Parallel Pipelines: Pipecat supports branching pipelines for scenarios like failover or parallel processing of audio and video.

Building a Voice Agent with Pipecat and Gemini Live

• A public repository (daily-co/gemini-pipcat-workshop) provides a starting point for building a voice agent.
• The gemini_bot.py file contains the main Pipecat code.
• The code runs within an AIO HTTP session.
• Key components:
  • Daily Transport: Configured with a room URL, token (optional), and parameters (input/output enabled).
  • Gemini Multimodal Live LLM Service: Initialized with system instructions and tools (function calling).
  • Context Aggregation: Collects conversation history for LLMs (using OpenAI as the default).
  • Pipeline Definition: A tuple of services (transport input, LLM, transport output).
  • Event Handlers: Triggered by client connections/disconnections to inject context frames into Gemini.
• The Runner executes the Task, which runs the Pipeline.

Function Calling and Tool Schema

• Function calling allows the LLM to access external tools and data.
• The function_schema provides a universal format for defining functions, enabling portability across different LLMs.
• Tools are passed to the Gemini service, granting it access to execute them.
• Example tools: fetch_weather, restaurant_recommendation.

Transports: WebRTC vs. WebSockets

• WebRTC: Recommended for client-server applications due to error correction and better audio quality.
• WebSockets: Suitable for server-to-server applications (e.g., phone chatbots).
• Pipecat supports both WebRTC and WebSockets.
• Daily provides WebRTC transport.
• FastAPI version available for WebSocket transport.
• Small WebRTC transport: peer-to-peer WebRTC communication that's free.

Voice Activity Detection (VAD)

• VAD detects when a user starts speaking, triggering the user's turn in the conversation.
• Pipecat emits a "user started speaking" frame, interrupting any ongoing bot speech.
• Solero: An open-source, on-device VAD option with low CPU consumption.
• VAD is crucial for accurate and fast turn management.

Telephony Integration

• Pipecat supports various telephony protocols:
  • WebSockets: Native WebSocket connection to providers like Twilio, Telnix, Pivo, or Exotel.
  • PSTN (Public Switched Telephone Network): Dial-in access.
  • SIP (Session Initiation Protocol): Call control via SIP providers like Daily.
• Websocket connections are instantaneous, while SIP offers superior call control but is more complex.

Guardrails and Content Moderation

• Guardrails (content moderation) are not inherently built into Pipecat.
• Latency is a critical factor; avoid unnecessary turns.
• Strategies for managing LLM behavior:
  • Task-oriented conversations: Chunk the conversation into discrete tasks.
  • Context window control: Manage the size of the context window judiciously.
  • Context summarization: Use an out-of-band LLM call to summarize the context.
• Google's live API offers context management through rolling/sliding windows and token caps.

State Management and Context Window Size

• Large context windows can slow down LLM processing.
• Chunking prompts is generally preferable to dumping everything into a large context.
• Function calls in real-time can be slow.
• Managing the context window is crucial for accuracy and performance.

Noisy Environments

• Noisy environments are a significant challenge for voice AI.
• Crisp: A partner that provides excellent noise cancellation, removing ambient and background human noise.
• Noise cancellation can be integrated into the transport layer of the Pipecat pipeline.

LLM Selection and Performance

• Pipecat supports Gemini Multimodal Live, OpenAI Realtime, and AWS Nova Sonic.
• Latency is generally good across all providers.
• System instruction handling varies across LLM providers.

Interruptions and Turn Management

• Human-like turn-taking is challenging for bots.
• Mechanical approach: VAD timeout after the user stops speaking.
• Semantic End-of-Turn Detection: An emerging field that uses audio cues (filler words, pauses, intonation) and text context to determine when a user has finished speaking.
• Smart-Turn: A native audio-in classifier that outputs "complete" or "incomplete" to dynamically adjust the VAD timeout.

Word and Time Stamp Synchronization

• Pipecat supports word and time stamp synchronization with TTS providers like Cartisia, 11 Labs, and Rhyme.
• TTS services output audio streams and TTS text frames.
• Clients can observe these text frames and synchronize word-by-word output with the audio.

Offline Models

• Offline models are feasible depending on the complexity of the task.
• Simple tasks (e.g., restaurant reservations) can be handled with local models like Llama.
• Whisper has challenges as an open-source ST model.
• The quality of the speech-to-text transcription is critical for overall performance.

Conclusion

Pipecat provides a flexible and modular framework for building voice and AI agents. It supports various transports, LLMs, and services, enabling developers to create sophisticated real-time applications. While challenges remain in areas like turn management and noisy environments, ongoing advancements in AI and signal processing are paving the way for more natural and seamless voice interactions. The workshop demonstrated the ease of creating a voice agent with Gemini Live and highlighted the potential of Pipecat for building innovative voice-driven experiences.