Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Kramer (Daily)

Key Concepts

WebRTC, websockets, low latency audio, voice AI, real-time communication, edge-to-cloud audio, packet loss, jitter buffer, bandwidth estimation, serverless WebRTC, peer-to-peer connection, conversational intelligence, generative AI, multilingual education.

Building Natural, Fast, Humanlike Voice Experiences

The core challenge in building successful conversational voice experiences is latency. If the AI responds too slowly, user engagement and satisfaction plummet. Voice AI engineering shares similarities with other AI disciplines, particularly multi-turn agents, but latency is the critical differentiator.

• Target Latency: Aim for a voice-to-voice latency (time between human speech ending and AI response beginning) of under one second, ideally around 500 milliseconds, to mimic natural human conversation.
• Latency Breakdown: A real-world voice AI app running in a web browser on macOS, communicating with a cloud-based voice agent via Pipecat, demonstrates that achieving sub-second latency is possible but requires careful optimization.
• Common Pitfalls: Slow LLMs, inefficient code, and Bluetooth devices can significantly increase latency.

WebRTC vs. Websockets for Real-Time Audio

The most significant mistake developers make is using websockets instead of WebRTC for sending and receiving audio over the network.

• Websockets: Suitable for long-lived connections with small amounts of data and server-to-server communication. Easy to implement and targets all platforms for prototyping.
• WebRTC: Designed for high-quality audio, high bandwidth, and low latency real-time communication. More complex to implement but offers crucial advantages.

TL;DR: Use websockets for server-to-server communication and small data transfers. Use WebRTC for sending audio and video streams from web or native apps.

Why WebRTC is Essential for Real-Time Edge-to-Cloud Audio

WebRTC provides critical features that websockets lack for real-time audio:

1. Best-Effort Networking: WebRTC prioritizes speed and low latency over guaranteed delivery. It uses clever math and buffer management to hide packet loss, sending packets as fast as possible and ignoring those that arrive outside the tight latency budget. Websockets, built on TCP, guarantee in-order delivery, which can lead to delays and audio glitchiness if packets are lost.
   • TCP Behavior: TCP connections will keep trying to send packets until they are acknowledged or the connection times out, which is unsuitable for real-time audio where outdated packets are irrelevant.
   • Real-World Impact: In 10-15% of network connections, websockets can result in audio glitchiness, high latency, or unexpected socket disconnections due to packet loss.
2. Built-in Audio Handling: WebRTC handles resampling, packetization, bandwidth estimation, and provides standard APIs for stats and observability. Implementing these features with websockets requires significant custom development.
3. Code Comparison: The code required to send a bidirectional audio stream using WebRTC is significantly less complex than the equivalent implementation using websockets.

Example: OpenAI's real-time API offers both WebRTC and websockets options, highlighting the importance of choosing the right technology for the specific use case.

Applications and Future of WebRTC

WebRTC enables real-time audio and video in various applications:

• Common Applications: Facebook Messenger, WhatsApp, Zoom, Discord, and other popular platforms use WebRTC for real-time communication.
• Emerging Applications: Surgery over the internet, teleoperation of vehicles, and other advanced applications leverage WebRTC's real-time capabilities.

Voice as the Next UI: Voice is poised to become a core building block of the next generation of user interfaces, particularly for generative AI.

• Analogy to Mobile Revolution: The current state of voice AI is likened to the early days of the mobile revolution (late 2007), with significant innovation and development yet to come.
• Accessibility and Convenience: Voice interfaces offer hands-free interaction and remote computing power, making them suitable for various situations.

Squabbert: A Demonstration of Edge AI with WebRTC

Squabbert, a Raspberry Pi-based device, demonstrates serverless WebRTC and edge AI capabilities.

• Tech Stack: Squabbert uses MLX, Whisper, Gemma 3, and a custom logic sampler.
• Counting Syllables: Squabbert can count syllables, showcasing edge AI capabilities that even large cloud-based LLMs may not possess.
• Poetry Generation: Squabbert generates short poems with specific constraints (e.g., using only two-syllable words).
• Connection Types: Squabbert demonstrates a peer-to-peer WebRTC connection to a laptop over a local area network, highlighting the flexibility of WebRTC. Other options include connecting to a server in the cloud or using Pipecat for multi-party connections.

Yashin's Multilingual Education Project

Yashin, a mom of two bilingual children, is developing a voice AI application to make language education more natural and fun for kids.

• Motivation: To help bilingual children connect with their cultural roots.
• Technology: Developed a prototype with guidance from the WebRTC community, despite having no prior coding experience.
• Early Testers: Parents are eager to test the application.

Conclusion

WebRTC is the superior choice for building low-latency, real-time audio and video applications. Its ability to handle packet loss, manage bandwidth, and provide built-in audio processing capabilities makes it essential for creating natural and responsive voice AI experiences. The future of voice AI is bright, with potential applications ranging from education to remote surgery. The community is encouraged to explore WebRTC and contribute to the development of innovative voice-based solutions.