Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs

Key Concepts

• Latency: The time delay between a user's input and the system's response.
• Voice-in/Visuals-out: An interaction paradigm where users speak to an AI, and the AI responds via visual interfaces (UI, charts, images).
• Inference Platform: The infrastructure used to run AI models, specifically optimized for speed.
• Prefix Caching: A technique where the initial, static part of a prompt is cached to significantly reduce latency and cost in subsequent requests.
• P95 Latency: A performance metric representing the response time that 95% of requests fall under.
• Agentic Workflow: AI systems that perform tasks autonomously based on user intent.

---

1. The Shift in AI Interaction Paradigms

Alan Pike discusses the evolving nature of human-AI interaction, referencing Andrej Karpathy’s argument that humans prefer voice as input and visuals as output.

• Visuals-out: Modern AI models can now generate rich HTML, interactive controls, and illustrations. Because one-third of the human brain is dedicated to visual processing, this is a highly effective way to communicate complex information.
• Voice-in: While historically plagued by "slow and dumb" interfaces (e.g., early Siri or clunky voice modes), voice remains the highest-bandwidth communication method for humans. Pike argues that when voice agents are "dialed in," they can act on intent without being interruptive, creating a natural, seamless experience.

2. The Tyranny of Latency

The primary barrier to natural AI interaction is latency. Pike outlines the critical thresholds for human perception:

• 100ms: The threshold for a system to feel "instant."
• 1,000ms (1 second): The limit of human attention; beyond this, users lose their train of thought.
• 200ms: The requirement for a truly fluid, conversational "voice-in/voice-out" experience, allowing for interjections and natural turn-taking.

Pike notes that achieving 200ms is technically difficult due to network requests, speech-to-text processing, and model inference. He suggests that voice-in/visuals-out is a more practical current approach because the "visual response envelope" is more forgiving (up to 1 second) than the strict requirements of real-time voice conversation.

3. Strategies for Low-Latency AI Experiences

Pike shares three technical pillars for building responsive, delightful AI agents:

A. Prioritize Fast Models and Inference Platforms

• Model Selection: Use "Haiku-class" models or small, efficient open-source models. Pike notes that while newer "mini" models are intelligent, they often fail to meet P95 latency requirements (sometimes hitting 5,000ms–10,000ms).
• Hybrid Architecture: Use a fast, small model for real-time interaction and offload complex, heavy tasks to a larger, "thinking" model asynchronously.

B. Eager Inference

• Avoid waiting for long pauses or explicit "stop" signals from the user.
• Implement "eager" inference, where the system sends requests every 1–2 seconds while the user is still speaking. This creates a sense of responsiveness and allows the AI to begin processing intent before the user finishes their sentence.

C. Stable Caching Regimen

• Prefix Caching: Leverage platform-specific caching to keep the first 90% of the context window static across requests. This makes inference significantly faster and cheaper.
• Token Optimization: Minimize output tokens to ensure the final response generation is as rapid as possible.

4. Real-World Application

Pike highlights a case study from his company, Forest Walk, where they built an agent that listens to team calls. When a user mentions a bug, the agent automatically files a ticket in Linear. This demonstrates the power of "incidental" voice interaction—where the AI acts on intent without requiring a formal, dedicated command-and-response structure.

5. Conclusion

The future of AI interaction lies in moving away from typing and toward high-bandwidth, low-latency voice inputs paired with rich, visual outputs. By focusing on infrastructure that prioritizes latency—specifically through fast model selection, eager inference, and aggressive prefix caching—developers can create AI experiences that feel like natural, seamless extensions of human collaboration.