Building real-time voice applications with Live API

Key Concepts

• Live API: Google's multimodal interface for developers to build real-time, bi-directional applications with Gemini.
• Multimodality: The ability to interact with the API using multiple input/output methods, including audio, video (screen sharing, webcam), and text.
• Native Audio Input/Output: Direct audio processing by the model, enabling more natural-sounding voices and improved dialogue.
• Half Cascade Architecture: An earlier architecture using native audio input but text-to-speech for output.
• Tools: Functions or external resources (e.g., Google Search, code execution) that the Live API can access and use to enhance its capabilities.
• URL Context: A tool that allows the model to retrieve and understand the content of URLs.
• Async Function Calling: A tool that allows the model to start a task in the background while continuing the conversation.
• Proactive Audio: The model's ability to intelligently choose when not to respond to audio input.
• Effective Dialogue: The model's ability to respond in a tone-aware and sentiment-aware manner.
• Turn Detection: The ability of the model to accurately determine when a user has finished speaking.
• Session Length: The duration of a Live API session.
• Proactive Video: The model's ability to proactively identify and respond to specific inputs in the video stream.
• Semantic Turn Detection: The model's ability to understand the context of a conversation and determine when it is appropriate to interject.

Live API Overview

• The Live API was released in December and allows real-time, bi-directional, multimodal interactions with Gemini.
• Initial launch included screen sharing and webcam capabilities, which went viral.
• Updates since December have focused on improving audio input/output, language support, session management, and turn detection.
• Google Next updates included improvements to the half cascade architecture, language support, new voices, session management, and turn detection.
• Google I/O introduced native audio output, enabling more natural-sounding voices and features like proactive audio and effective dialogue.

The Importance of Audio

• Audio is a natural interface modality, as humans learn to talk before they learn to read.
• Audio is a high-information-density modality, as most people talk faster than they type.
• Thinking aloud is a common problem-solving technique, making audio a natural fit for AI applications.
• Native audio needs further development to be production-grade, particularly in terms of precision and latency.
• Models will learn to work around the precision trade-off by understanding the nuances of human speech.

Text-to-Speech (TTS)

• Gemini now offers controllable and promptable text-to-speech capabilities.
• Users can prompt the model to speak with a specific tone and style.
• Multi-speaker text-to-speech is available, similar to the NotebookLM podcast experience.
• Example: A sommelier student uses multi-speaker TTS to create podcasts from study materials.

Use Cases for the Live API

• Software Co-pilots: Real-time feedback on how to navigate complex software (e.g., Photoshop, CloudFlare).
• AI Companions/Assistants: Applications in industrial equipment, debugging, and general assistance.
• Learning Tutors: Specific to learning, including language learning with translation capabilities.
• Gaming Assistants: Providing real-time assistance and information within games.
• Real-Time Interactions in Cars: Answering questions and providing assistance to passengers (e.g., what to do if keys are left in the car).
• Conducting Interviews: Recruiting and user research interviews.
• Full Co-presence: A built-in operating system feature that allows the AI to observe and assist with tasks.

Tools in the Live API

• Initial launch included Google Search and code execution as first-party tools.
• Third-party function calling is also supported.
• Tools can be chained together (e.g., search + code execution + display).
• URL Context: Allows the model to retrieve and understand content from URLs, enabling deep dives on specific topics.
• Async Function Calling: Allows the model to start a task in the background while continuing the conversation, improving latency.

New Features and Feedback

• Proactive Audio: The model chooses not to respond to certain inputs (e.g., sidebars in a conversation).
• Effective Dialogue: The model responds in a tone-aware and sentiment-aware manner, matching the user's emotional state.
• Multilingual Performance: Native audio output has improved multilingual capabilities, but further improvements are needed.
• Session Length: Developers can now control session length by adjusting image resolution, streaming video only when audio is spoken, and configuring session resumption.
• Turn Detection: Developers can configure turn detection sensitivity or bring their own turn detection models.
• Tool Calls/Function Calling: Continuous improvements are being made to function calling and first-party tools.
• Trinity Performance: Ongoing efforts to reduce cost, latency, and hallucinations.

Future Directions

• Thinking UI: Developing the right user experience for when models are "thinking" (i.e., processing information).
• Proactive Video: The model proactively identifies and responds to specific inputs in the video stream (e.g., locating a lost object).
• Coding Agent: A pair programming assistant that can talk with the developer and provide real-time guidance.
• Semantic Turn Detection: The model understands the context of a conversation and determines when it is appropriate to interject.

The Voice AI Market

• The voice AI market has exploded in H2 2024.
• Use cases are being explored across various verticals, including healthcare, finance, B2B, industrial, and media.
• AI can support customer support agents by handling after-hour calls.

Getting Started with Voice AI

• Experiment with voice on Google AI Studio, both in the chat experience and the Live API.
• Explore cookbooks and code samples.
• Utilize the developer relations team for guidance.

Demo Highlights

• The demo showcased Gemini's ability to:
  • Understand and respond to questions.
  • Adopt different personas (e.g., over-enthusiastic blobfish, embarrassed panda, snarky hedgehog, happy cow).
  • Translate between languages (Hindi to English).
  • Maintain coherence throughout a long conversation.
  • Recall previous information and actions.
• Gemini was able to generate a snarky poem in Hindi about Logan's shipping habits and then translate it into English.
• The demo highlighted the range and flexibility of the Live API.

Synthesis/Conclusion

The Live API represents a significant step forward in AI development, enabling developers to build real-time, multimodal applications with Gemini. The focus on audio, particularly native audio input/output, unlocks new possibilities for natural and intuitive interactions. While challenges remain in areas such as precision, latency, and turn detection, ongoing improvements and new features like proactive audio, effective dialogue, and proactive video are paving the way for innovative use cases across various industries. The explosion of the voice AI market underscores the potential of this technology, and Google is actively working to provide developers with the tools and resources they need to build the next generation of AI-powered applications.