Optimizing inference for voice models in production - Philip Kiely, Baseten

Key Concepts

• TTS (Text-to-Speech) Model Optimization: Improving the speed, efficiency, and cost-effectiveness of TTS models in production environments.
• Orpheus TTS: A specific open-source TTS model based on the Llama 3 architecture, used as a primary example.
• LLM (Large Language Model) Similarities: Recognizing the architectural similarities between TTS models and LLMs to leverage LLM tooling for optimization.
• Time to First Byte (TTFB): A key performance metric for TTS models, focusing on minimizing the delay before the first audio data is received.
• Concurrency: Maximizing the number of simultaneous audio streams a model can handle to reduce GPU resource usage.
• TensorRT-LLM: An NVIDIA library used to optimize LLM inference, applicable to TTS models due to their architectural similarities.
• Quantization (FP8): Reducing the precision of model weights to FP8 (8-bit Floating Point) to decrease model size and increase inference speed.
• Audio Codec Optimization: Optimizing the audio decoding process, including using libraries like Snack and leveraging GPU acceleration.
• Dynamic Batching: Grouping audio processing tasks into batches to improve throughput, balancing latency and efficiency.
• Infrastructure and Client-Side Optimization: Addressing non-runtime factors like network latency and client code efficiency to avoid negating runtime optimizations.

TTS Model Architecture and LLM Similarities

The speaker emphasizes that while it's an oversimplification, viewing TTS models as similar to LLMs is a useful framework for optimization. TTS models, like Orpheus TTS, often use transformer architectures similar to LLMs (e.g., Llama 3). This allows leveraging the extensive tooling and optimization techniques developed for LLMs. Orpheus TTS, specifically, is based on a Llama 3 3B backbone, making it amenable to LLM optimization strategies. Key architectural details of Orpheus TTS include a larger vocabulary size to accommodate speech-specific tokens and an extended context length.

Performance Metrics for TTS Models

Traditional LLM metrics like tokens per second (TPS) are less relevant for TTS. Instead, the focus shifts to:

• Time to First Byte (TTFB): Minimizing the latency before the first audio data is available.
• Concurrency: Maximizing the number of simultaneous audio streams the model can handle.
• Throughput: The number of requests served at a given time.

The goal is to achieve real-time streaming (around 83 tokens per second for Orpheus TTS) and then optimize for TTFB and concurrency to reduce GPU costs. The speaker's goal is to fit all the different voice agents that the model is capable of creating on one or even less than one GPU.

Optimization Techniques

Several optimization techniques are discussed:

• TensorRT-LLM: Using TensorRT-LLM for optimized inference, especially beneficial for models with LLM-like architectures.
• Quantization (FP8): Quantizing the model to FP8, including the KV cache, to reduce model size and improve performance. This worked well for Orpheus TTS, even though quantizing small models can sometimes degrade performance.
• Audio Codec Optimization: Optimizing the audio decoding process using libraries like Snack and leveraging GPU acceleration with torch.compile and PyTorch inference mode.
• Dynamic Batching: Implementing dynamic batching to group audio processing tasks, balancing latency and throughput. Batches are sent out every 15 milliseconds.

Performance Results

The speaker presents performance results achieved with the optimized Orpheus TTS model:

• Concurrency: The optimized implementation supports significantly more simultaneous streams compared to a base implementation. With variable traffic, it supports 16 simultaneous streams, and with constant traffic, it supports 24 simultaneous streams on an H100 MIG (half an H100 GPU).
• Time to First Byte (TTFB): The optimized implementation achieves a TTFB of 150 milliseconds in real-world testing.

These improvements translate to lower costs per hour of conversation compared to per-token APIs.

Infrastructure and Client-Side Considerations

The speaker emphasizes that non-runtime factors, particularly infrastructure and client code, can significantly impact overall performance. Pitfalls to avoid include:

• Sequential Requests: Sending requests sequentially instead of concurrently.
• Session Management: Creating a new session for each request instead of sharing sessions.
• Protocol Overhead: Using inefficient protocols like HTTP streaming instead of websockets or gRPC.

The speaker recommends using a multiprocessing pool, sharing sessions between requests, and using appropriate protocols to maximize performance.

Voice Agent Pipeline

The speaker highlights that TTS models are only one part of a larger voice agent pipeline, which includes listening, thinking, and talking components. Optimizing the infrastructure to connect these components is crucial. Factors like data center proximity and avoiding DNS hair-pinning can significantly reduce latency.

Conclusion

Optimizing TTS models for production involves a combination of runtime optimizations (e.g., TensorRT-LLM, quantization, audio codec optimization) and infrastructure/client-side considerations. While runtime optimizations can significantly improve TTFB and concurrency, non-runtime factors can easily negate these gains. A holistic approach that addresses all aspects of the voice agent pipeline is essential for achieving optimal performance.