ChatGPT is poorly designed. So I fixed it

Key Concepts

• Multimodal AI interaction (voice and text)
• Tool calls for handling different types of requests
• Reasoning models for complex tasks
• "Shipping the org chart" - disjointed user experience due to organizational silos
• Heuristics for determining when to use different models
• Live audio chat integration
• Coding agents for code-related tasks

Main Topics and Key Points

The Problem: Confusing User Experience in ChatGPT

• Despite its popularity, ChatGPT's voice interface feels disconnected from its text interface.
• The voice interface lacks the ability to collaborate on written content (e.g., editing an email draft).
• Scott Hanselman's analogy of "shipping the org chart" is used to describe this disjointed experience, where different features feel like they were built by separate teams without a unified vision. Example: EV car with climate, speedometer, and map all in different fonts.
• The current implementation feels like a "science fair full of potential options" rather than a cohesive product.

Proposed Solution: Integrating Voice and Text, Smart Model Selection

• Two key improvements are proposed:
  • Allowing voice and text interaction simultaneously.
  • Smartly choosing the right model based on the user's request.

Implementation with Off-the-Shelf Tools

• Real-time Audio Chat: Use a library like 40 Realtime for live audio chat functionality.
• Tool Calls: Utilize tool calls to handle different types of requests.
  • Send text for longer details, links, and drafts.
  • Hand off to a smarter reasoning model for complex tasks.

User Interface Design

• A new UI is proposed that resembles iMessage, allowing users to text while on a voice call.
• Call controls (mute, end call) are placed at the top of the screen.
• A reminder of the user's initial question is displayed for context.
• Text responses are shown in the chat panel for detailed information.

Handling Complex Requests with Reasoning Models

• Example: Using Warp terminal, a developer tool, to demonstrate how complex coding tasks can be handled.
• For simple tasks (e.g., "undo my last commit"), a coding agent directly executes commands in the terminal.
• For complex tasks (e.g., "refactor this entire codebase to use Flutter"), the system:
  1. Detects the complexity.
  2. Writes a plan using a reasoning model.
  3. Executes the plan to ensure the code works.
• Heuristics (e.g., asking for details, pros and cons) can be used to determine when to hand off to a reasoning model.

Building the Solution with APIs

• A demonstration is provided of building the proposed solution using off-the-shelf APIs.
• Example conversation:
  • User: "Hey there how's it going"
  • AI: "Hey there I'm doing well thank you how about you how's your day going"
  • User: "pretty good i just landed in California can you send me a link to a park that I should go visit"
  • AI: "of course i'll send you a link to a popular park in California"
  • User: "yeah I've heard about Yuseite can you tell me more about its history go deep"
  • AI: "yoseite National Park's history is rich beginning with its ancient Native American heritage and leading to its establishment as a national park check the chat for more details"
• send_chat_message tool: Used to send details that are easier to explain via text. The description of the tool is sufficient for the model to use it effectively without a system prompt.
• Reasoning Model Tool: Used when the user wants to go deeper on a topic. Details are sent to the model, and the response is either sent back to the model or directly to the client.

Data, Research Findings, or Statistics Mentioned

• GPT is one of the fastest-growing apps in history, with hundreds of millions of users.

Synthesis/Conclusion

The video highlights the disjointed user experience in ChatGPT's voice interface and proposes a solution that integrates voice and text interaction while intelligently selecting the appropriate model for each task. By leveraging off-the-shelf tools like 40 Realtime for audio and implementing tool calls for different request types, a more cohesive and user-friendly experience can be achieved. The use of reasoning models for complex tasks, guided by heuristics, further enhances the system's ability to provide detailed and accurate responses. The demonstration using Warp terminal and the example conversation showcase the practical application of these concepts. The source code is available on GitHub.