Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Key Concepts

• Voice AI for Enterprise: Building interactive voice models for improved agent experiences.
• Foundation Models: Large models typically hosted in the cloud and used for batch operations.
• Real-time Multimodal Intelligence: Processing multiple data types (e.g., voice, video) in real-time.
• Latency: The delay between a request and a response, critical for interactive applications.
• Controllability: The ability to customize voice AI to reflect a specific brand or use case.
• State Space Models (SSMs): An alternative to transformers that offer O(1) generation at inference time, enabling low latency.
• Voice Cloning: Replicating a specific person's voice using AI.
• Edge Computing: Running AI models on local devices rather than in the cloud.

Voice AI for Enterprise: Challenges and Solutions

Arjun, co-founder of Cartisia AI, discusses the importance of voice AI for enterprise applications, emphasizing the need for real-time multimodal intelligence. He contrasts traditional foundation models, which are often hosted in the cloud and operate in batch mode (with delays of 500-600 milliseconds), with the requirements of interactive applications like voice, where speed is paramount.

• Challenge: Traditional foundation models are too slow for real-time voice interactions.
• Solution: Cartisia AI focuses on bringing foundation models to real-time, covering multiple modalities (especially voice), and running them on any device, not just the cloud.
• Importance of Speed: In voice interactions, delays of even a second are unacceptable. Voice agents need to respond quickly and accurately to avoid frustrating users.
• Key Considerations: Voice AI systems must handle interruptions, globalization (accents), and background noise. The user experience is highly subjective and requires customization.

Cartisia AI's Approach: Quality, Latency, and Controllability

Cartisia AI builds voice AI solutions from first principles, focusing on three key aspects:

1. Quality: The naturalness of the voice must be excellent.
2. Latency: The first sound of audio should be heard as soon as possible, giving the agent more time to reason.
3. Controllability: The voice AI should be customizable to reflect the brand and specific use case.

• State Space Models (SSMs): Cartisia AI has pioneered the use of state space models as an alternative to transformers. Transformers scale quadratically, meaning their memory and runtime increase quadratically with input length. SSMs offer O(1) generation at inference time, maintaining a state that allows for very low latency.
• Performance: Cartisia AI's SSMs not only offer better latency but also comparable or better quality than transformers.

Customer Challenges and Use Cases

Rohit Turri from AWS asks about the challenges customers face when implementing voice AI.

• Challenge: Integrating voice generation models with language models (LLMs) and speech-to-text models requires careful management of latency. LLMs need sufficient time to process information.
• Solution: Cartisia AI's low-latency TTS models provide more slack for the LLM.
• Challenge: Achieving the desired level of controllability and customization, including voice cloning and capturing realistic background noises.
• Solution: Cartisia AI's platform allows for fine-grained control over voice characteristics, including accents and background sounds.

Customer Use Cases:

• Healthcare: Applications in virtual assistants, patient communication, and remote monitoring.
• Customer Support: Automating customer service interactions and providing efficient support.
• Real-time Gaming: Creating dynamic and interactive non-player characters (NPCs).

The Role of Human Narrators

Arjun emphasizes that Cartisia AI's goal is not to replace voice actors but to provide them with a platform to license their voices and personalities.

• Voice Marketplace: Cartisia AI has a voice marketplace for creators, allowing them to amplify their reach and generate revenue.
• Narration: Many use cases are focused on narration, providing opportunities for voice actors.

Questions from the Audience

• Latency with Claude: A user asks about the high latency experienced when integrating Cartisia AI with the Claude language model. Arjun acknowledges this as a common problem and suggests that dedicated instances of LLMs can improve latency. Rohit adds that AWS aims to provide customers with optionality and is looking for innovative model providers like Cartisia AI to integrate into its ecosystem.
• Data Requirements: A question is raised about whether voice AI models require massive datasets or if the richness of the data is more important. Arjun responds that both are important. High-quality, rich data is essential, but it must be paired with information that caters to diverse preferences.
• Speech-to-Speech Models: A question about the future of speech-to-speech models. Arjun believes that while speech-to-speech models have potential, orchestrated solutions (combining separate speech-to-text, LLM, and text-to-speech models) currently offer more controllability and are better suited for enterprise-grade use cases.
• Edge Computing: A question about local models. Arjun confirms that Cartisia AI builds models that can run locally on edge devices. He emphasizes that edge computing can offer significantly lower latency than cloud-based solutions in certain scenarios.
• Monitoring Agent Performance: A question about tools for monitoring the performance of voice AI agents. Arjun suggests that issues often arise at the LLM stage due to formatting problems. Cartisia AI handles many of these edge cases to improve the robustness of its system.

Vision for the Future

Rohit asks Arjun about his vision for the future of Cartisia AI and voice AI in general.

• Voice AI as the Norm: Arjun believes that voice AI will become ubiquitous across industries, playing a significant role in customer support, gaming, and other applications.
• Interactive World Models: He envisions a future where AI systems can interact with users in real-time, understanding the world around them and providing assistance as co-pilots.

Conclusion

The discussion highlights the growing importance of voice AI for enterprise applications and the challenges of achieving real-time performance, controllability, and high quality. Cartisia AI's innovative approach, using state space models and focusing on edge computing, positions it as a key player in the voice AI landscape. The conversation also emphasizes the continued role of human voice actors and the potential for AI to augment, rather than replace, human creativity.