Voice Agent Engineering — Nik Caryotakis, SuperDial

Key Concepts

Voice AI, Agents as a Service, Voice AI Engineer, Last Mile Problem, Conversation Design, Orchestration Framework, Text-to-Speech (TTS), Speech-to-Text (STT), Latency, Reliability, Generative AI, AI Ethics, Vertical Voice AI.

Superdial's Approach to Voice AI

• Business Focus: Superdial focuses on automating phone calls, specifically in the healthcare administration sector (e.g., insurance calls).
• Agentic Contract: They offer an "agents as a service" model where customers provide call details (who to call, questions to ask) and receive structured results.
• Agentic Loop: Superdial uses an internal agentic loop:
  1. Wait for office hours/call center availability.
  2. Attempt call with voice bot.
  3. Fallback to human agents if the bot fails after a certain number of attempts or requires human intervention.
• Transparency: Customers are informed about the fallback mechanism, which is presented as a benefit ensuring call completion.
• Continuous Learning: The system learns from each call, updating office hours and phone tree traversal strategies.
• Quality Assurance: Random call audits are conducted to ensure system reliability.
• Impact: Superdial has saved over 100,000 hours of human phone calling time and is projected to save millions more in 2025. This was achieved with a lean engineering team of four.

The Voice AI Engineer

• Multimodal Data: Deals with MP3s, audio bytes, and transcripts.
• Technology Stack: Works with transcription models, voice models, and speech-to-speech systems.
• Real-time Constraints: Latency is a critical factor, requiring expertise in asynchronous programming (async) and Python.
• Product Constraint: Voice conversations demand high expectations for natural and contextually appropriate interactions.
• Key Sayings:
  • "Say the right thing at the right time."
  • "Build this plane while we fly it."
• Horizontal Voice AI Stack: Leverages a horizontal stack to customize scripts and use cases for individual customers.
• Inflection Point: The focus is shifting from the technical aspects of voice (e.g., realism, interruption handling) to conversational content, design, and vertical integrations.

AI Ethics in Voice AI

• Potential Biases: Voice AI applications can be biased against individuals with specific accents or dialects.
• Spookiness Factor: Realistic voice AI can be unsettling if it produces unexpected or inappropriate responses.
• Lack of Regulation: The absence of strong AI regulations places responsibility on AI engineers and leaders to address ethical concerns.
• Accessibility and Collaboration: AI development should be accessible and collaborative, involving diverse stakeholders.
• Tooling Choices: Infrastructure and tooling should be selected to facilitate broad participation in the development process.

Last Mile Problems in Voice AI

• From Prescriptive to Descriptive Development: The shift from prescribing bot behavior to describing desired outcomes and relying on generative models.
• Conversation Design:
  • Open-ended vs. Constrained Questions: Superdial prefers open-ended questions to capture more information from call center representatives.
  • Adaptation: Focus on adapting to user responses rather than preventing incorrect statements.
  • Recommendation: Hire conversation designers or conduct table reads (role-playing conversations) to identify gaps and awkwardness.
• Orchestration Framework:
  • Pipe Cat: Superdial uses Pipe Cat, an open-source framework maintained by Daily, for voice AI orchestration.
  • Self-Hosting: Pipe Cat's self-hosting capability allows for custom scaling and deployment.
• LLM Infrastructure:
  • Owning OpenAI Endpoint: Superdial owns its OpenAI endpoint for better control and integration with voice AI tools.
  • Tensor Zero: Uses Tensor Zero for structured and typed LLM endpoints, enabling experimentation in production.
  • Lane Fuse: Self-hosts Lane Fuse for logging, observability, anomaly detection, and evaluations.
  • HIPAA Compliance: Self-hosting is also driven by HIPAA compliance requirements for healthcare data.
• Text-to-Speech (TTS) Challenges:
  • Mismatch: LLM output may not be suitable for TTS engines, and neither may match the actual recording.
  • Pronunciation: Uses tools like Rhyme to specify exact pronunciations.
  • Spelling: Employs custom functions to control pauses and breaks in spoken words.
  • Review: Recordings are reviewed to ensure audio quality.
• Mini Last Mile Problems:
  • Persona: Avoid problematic bot names (e.g., "Billy" sounding like "Billie" over the phone).
  • Don't Build from Scratch: Leverage existing tools like Pipe Cat for rapid prototyping.
  • Track Latency: Monitor time to first byte for each processing step.
  • Upgrade Paths: Plan for easy upgrades to components like speech-to-text engines (e.g., Deep).
  • Fallbacks: Implement fallback mechanisms for each part of the stack (e.g., using Tensor Zero) to handle outages.
  • End-to-End Testing:
    • MP3 Playback: Test basic audio playback.
    • Simulated Voice Tree: Navigate a simulated phone tree.
    • Bot-to-Bot Communication: Use services like Koval and V to test interactions between bots.

Conclusion

The speaker emphasizes the importance of focusing on the "last mile" problems in voice AI, particularly conversation design and integration, to create truly valuable and reliable agents. While new models and technologies are constantly emerging, the key to success lies in understanding the specific needs of the use case and building a robust, ethical, and well-tested system. Choosing the right tools and infrastructure is crucial for enabling rapid iteration and adaptation in this fast-moving field.