Building LiveKit Agents with Gemini Live API

Key Concepts

• Native Audio Model: A model that processes and generates audio directly without intermediate speech-to-text (STT) or text-to-speech (TTS) conversion.
• Gemini 3.1 Flash Audio (EAP): The specific early-access model used for real-time, low-latency voice interaction.
• Speaker Drift: The phenomenon where an AI voice gradually loses its intended persona or accent over long sessions.
• Function Calling: The ability for the model to trigger external code (server-side) or client-side actions (via RPC).
• Multimodal: The capability to process and understand audio, text, and video inputs simultaneously.
• LiveKit Agents: A framework for building and deploying real-time voice and video AI applications.

---

1. Technical Implementation and Setup

The video outlines a process for building a local voice agent using Python and the LiveKit framework.

• Environment Setup: Initialize a project using uv init, then add livekit-agents and the google-genai plugin.
• Configuration: API keys for LiveKit (Cloud) and Google AI Studio must be stored in a .env.local file.
• Agent Initialization: The agent.py file serves as the core, where the Gemini 3.1 Flash Audio model is defined. The session is initiated via the LiveKit server, which handles the WebRTC connection.

2. System Prompt Engineering for Voice

The speaker emphasizes that system prompts for voice agents differ significantly from those for text-based LLMs:

• Persona and Scope: Be explicit about the agent's identity and limitations.
• Audio-First Writing: Avoid bullet points; use short, conversational sentences and natural pauses.
• In-line Guardrails: Define safety and redirection rules directly within the prompt rather than relying on external layers.
• Director Notes: Use the model’s deep audio understanding to shape delivery (e.g., "speak with an Irish accent," "use a slower pace," or "higher energy").

3. Advanced Capabilities

• Multilingual Support: The model natively supports ~70 languages and can switch between them mid-conversation without requiring language code configuration.
• Reduced Speaker Drift: The Gemini 3 architecture is specifically optimized to maintain consistent persona and accent stability during long, multi-turn conversations.
• Multimodality: Beyond audio, the model can process text and video inputs, with the option to output real-time text transcripts alongside audio.

4. Tool Calling Frameworks

The agent utilizes two distinct methods for executing tasks:

• Server-Side Function Calling: Uses the @function_tool decorator. The agent identifies when a tool is needed based on the function description, executes the logic on the server, and returns the result.
  • Example: A custom function to query account status by ID.
  • Built-in Tool: The google.tools.GoogleSearch tool allows the agent to fetch real-time information (e.g., weather, news) without additional API configuration.
• Client-Side RPC (Remote Procedure Call): Allows the agent to trigger actions on the user's device (browser or mobile app). This is used for UI updates or controlling local device state via the WebRTC connection.

5. Testing and Validation

The speaker suggests a structured testing methodology for developers:

1. Latency & Chaining: Test back-to-back tool calls to observe how the model handles complex sequences.
2. Drift Testing: Conduct 10–15 minute conversations to ensure the voice persona remains consistent.
3. Multilingual Switching: Transition between languages (e.g., English to Spanish to German) to verify seamless language detection.
4. Real-time Data: Use the Google Search tool to query volatile data like stock prices or current weather.
5. RPC Integration: Trigger UI-specific functions to ensure the agent can effectively control the host application.

Synthesis and Conclusion

The transition to a native audio model represents a paradigm shift in voice agent development. By eliminating the latency and quality loss associated with traditional STT/TTS pipelines, the Gemini 3.1 Flash Audio model enables more natural, human-like interactions. The combination of LiveKit’s infrastructure and Gemini’s native audio capabilities allows developers to build scalable, production-ready agents that maintain persona consistency and handle complex, multilingual, and multi-tool workflows with minimal configuration.