Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

Key Concepts

• Voice Agent Pipeline: A cascading architecture consisting of Speech-to-Text (STT), Large Language Models (LLM), and Text-to-Speech (TTS).
• Latency Budgets: The critical time constraints (typically <500ms) required to maintain natural human-like conversation.
• TTFT (Time to First Token): The time taken for an LLM to begin generating text.
• TTFA (Time to First Audio): The time taken for a TTS model to produce the first audible chunk.
• Real-Time Factor (RTF): The ratio of processing time to the duration of audio generated; ideally <1.0 to prevent buffering.
• Co-location: The practice of placing models and orchestrators in the same data center to minimize network latency.
• Full Duplex: The ability of an agent to listen and speak simultaneously, allowing for back-channeling and interruptions.

---

1. The Voice Agent Pipeline Architecture

The current industry standard for production voice agents is a cascading pipeline:

1. Speech-to-Text (STT): Converts audio chunks into text. Key metrics include Word Error Rate (WER) (state-of-the-art is ~6%) and Time to Complete Transcript.
2. LLM (The Brain): Processes text, manages logic, and performs tool calling. Optimal models for latency are typically in the 8B–30B parameter range.
3. Text-to-Speech (TTS): Converts LLM output back into audio. Key metrics include TTFA and RTF.

Evolution in STT: The industry is shifting from batch-based models (like standard Whisper) to streaming-native models. These use encoder look-ahead (80ms–1s) and activation caching to process audio in real-time without waiting for 30-second clips.

2. Engineering Challenges & Trade-offs

• Latency: Human conversation requires a response within ~300ms. Every component (STT, LLM, TTS) and network hop adds to this budget.
• Reliability & Scaling: Unlike asynchronous systems, voice agents require stateful, long-lived connections. Auto-scaling must be handled carefully to avoid killing active conversations.
• Turn Detection: A significant, largely unsolved challenge. Determining if a pause in speech signifies the end of a turn or a momentary hesitation is critical to prevent the agent from interrupting the user.
• Network Latency: Moving from cross-continental data centers to co-location (placing all components in the same building) can reduce latency by ~70ms, representing a ~30% performance gain in optimized systems.

3. Advanced Architectures: Speech-to-Speech

The speaker highlights a shift toward pure speech-to-speech models (e.g., OpenAI’s Real-time API, Nvidia’s Voice Chat).

• Benefits: These models natively understand tone, emotion, and hesitation, and support full-duplex communication (back-channeling like "uh-huh").
• Current Limitations: They currently struggle with complex instruction following and reliable tool calling compared to the pipeline approach. Most production systems currently rely on the pipeline for its robustness in business logic.

4. Observability and Guardrails

• Guardrails: In production, developers often insert classifier models before the LLM (to route requests) or after the LLM (to filter unauthorized content).
• The "Thinker-Talker" Pattern: To manage latency and accuracy, a small, fast LLM handles the immediate conversation, while a larger, more capable model is called asynchronously to handle complex tool execution or guardrail verification.
• Evals: Evaluation for voice agents involves both component-level metrics (WER, TTFA) and conversation-level metrics. For tool calling, developers often fine-tune smaller models on domain-specific data to maintain high accuracy within strict latency budgets.

5. Notable Quotes

• "Voice is just this brand new interface to interact with systems and computers... This is not the domain of science fiction or research anymore. This is primarily an engineering problem today."
• "If you don't get the transcript right... there's no way to fix this. Your LLM will carry forward the mistake. The TTS model will carry forward the mistake."

Synthesis

Building high-quality voice agents is an "and" problem: one must simultaneously solve for low latency, high intelligence (tool calling), natural voice quality, and massive concurrency. While the pipeline architecture is the current production standard due to its modularity and reliability, the future lies in native speech-to-speech models that can handle the nuances of human interaction—such as interruptions and emotional context—without the overhead of text conversion. Success in this field requires deep observability, aggressive latency optimization through co-location, and rigorous evaluation of every component in the stack.