The New Stack Agents - Andrew Lee, Co-Founder & CEO at Shortwave

Key Concepts:

• AI Agents
• Large Language Models (LLMs)
• Tool Calling
• Asynchronous Execution
• MCP (Message Communication Protocol)
• Dynamic UI Generation
• Model Switching (OpenAI, Anthropic, Open Source)
• Embeddings
• Vector Databases (Pinecone)
• Full-Text Search (Elasticsearch)

1. Shortwave's Evolution and AI Pivot:

• Initial Goal (2020): Build a better Gmail with improved thread display, search, collaboration, and sharing features.
• Early Progress: Achieved some user adoption and revenue, but growth wasn't explosive.
• AI Pivot (Late 2022): Recognized the potential of LLMs to interpret email data (correspondence, receipts, newsletters, calendar invites, SaaS notifications).
• Shift in Focus: From basic email improvements to leveraging AI for a more powerful user experience.

2. Competing with Gmail:

• Google's Institutional Challenges: Large companies struggle with innovation due to stakeholder conflicts, legacy systems, and established user habits.
• Gmail's Stagnation: Perceived lack of innovation due to internal conflicts and risk aversion.
• Google Inbox Case Study: Google's innovative email client was ultimately killed due to internal resistance, signaling a lack of appetite for radical change within the Gmail team.
• Opportunity: Shortwave aimed to capitalize on Gmail's stagnation by offering a more innovative email experience.

3. Early AI Features and Validation:

• Initial AI Features: Summarization, translation, and action item listing.
• Implementation: On-demand features triggered by user actions (button clicks, menu selections).
• User Response: Generated hype, press coverage, sign-ups, and paid subscriptions, validating the potential of AI in email.
• Limitations: Slow performance due to model limitations and high costs.

4. Current Shortwave Capabilities (Demo):

• Core Functionality: Standard email client features (inbox, threads).
• AI Agent Integration: Claude-like AI assistant integrated into the email client.
• Context Awareness: The AI agent can access and interpret the user's email data, contacts, and calendar.
• Tool Calling: Integrates with other tools via MCP (Asana, Notion, GitHub).
• Example Use Cases:
  • Calendar Integration: "Whose podcast am I on today?" - The AI agent checks the calendar and provides relevant information.
  • Email Composition: "Send Frederick an email and tell him I'm going to be late." - The AI agent composes an email in the user's writing style.
  • Email Analysis: "Based on the emails I've sent and received over the last day, what are my top priorities?" - The AI agent analyzes email data and identifies key priorities.
• Autocomplete: Suggests and completes text based on context.

5. Technical Architecture:

• Data Import: Emails are streamed from Gmail into Shortwave.
• Embedding Generation: Emails are processed by an open-source model to generate embeddings.
• Database Infrastructure:
  • Pinecone: Stores embeddings for semantic search.
  • Elasticsearch: Provides full-text search capabilities.
  • Postgres: Stores metadata for structured queries.
• Query Processing:
  • The LLM translates user queries into database searches.
  • Tool calls are generated to retrieve relevant information.
  • Data from multiple databases is combined to provide comprehensive results.
  • The LLM processes the retrieved data and generates a response.

6. Tool Calling Evolution:

• Early Implementation: Shortwave developed its own tool calling logic using XML parsing before major model vendors offered built-in support.
• OpenAI's Tool Calling: Initial attempts with OpenAI's tool calling resulted in decreased overall intelligence.
• Anthropic's Cloud Sonnet 3.5: Switching to Cloud Sonnet 3.5 in December led to a significant improvement in tool calling capabilities, enabling complex sequences of tool calls.
• Impact: Radically smarter AI agent capable of more sophisticated reasoning and task execution.

7. Model Switching Strategy:

• Anthropic's Dominance: Anthropic's models are considered superior for agentic tool calling use cases.
• Rapid Adoption: Shortwave quickly adopts new Anthropic models through public experiments and live rollouts.
• Validation: User feedback is used to validate the performance of new models.

8. Current Limitations:

• Integration Limits: Adding too many integrations can degrade reasoning and increase errors.
• Need for Scalability: Models need to handle a larger number of tools and integrations without performance degradation.

9. Agent Architecture:

• Single Agent, Extensive Context: Shortwave uses a single, iterative agent with a wide range of tools.
• Rationale: This approach allows the LLM to reason across a broader set of tools and discover creative solutions.
• Alternative Approaches: Hierarchical sub-agents may be more suitable for cost-efficiency or structured tasks.

10. MCP (Message Communication Protocol):

• Potential: Enables AI agents to connect to other tools and services.
• Adoption: Gaining traction with numerous server implementations, client applications, and community formation.
• Shortwave's Perspective: Architected to support multiple protocols, with a current focus on MCP.
• Key Question: How to leverage AI's ability to connect to other tools while ensuring data safety.

11. Future Directions:

• Asynchronous Execution: Enabling AI agents to perform tasks automatically in the background (e.g., processing emails, updating databases).
• AI Filters: Expanding AI filters into asynchronous AI agents capable of complex workflows.
• Event-Driven Automation: Triggering agent execution based on various events (email receipt, recurring tasks, label additions).

12. AI in Development:

• Tools: Cloud Code and Cursor are heavily used for code generation and bug fixing.
• Impact: AI is responsible for a significant portion of the code committed at Shortwave.
• Human Replacement: AI has not yet replaced human developers, but may be influencing hiring decisions.

13. AI in Content Creation:

• Blog Posts and Newsletters: AI is used to generate content based on outlines and contextual information.
• Efficiency: AI assistance saves time and effort in content creation.

14. AI in Business Processes:

• Candidate Screening: AI is used to automate candidate screening processes (resume parsing, email categorization).
• Automation: Shortwave is actively seeking opportunities to streamline and automate business processes using AI.

15. Dynamic UI Generation:

• Potential: LLMs can generate code for UIs, enabling dynamic and purpose-built interfaces.
• Use Case: Reconfiguring the email client interface for specific tasks (e.g., resume reviewing).
• Future Vision: Apps that can dynamically adapt their UI based on user needs and context.

16. Other Areas of Interest:

• Tool Calling Improvements: Enhancing the reliability and efficiency of tool calling.
• Large Context Sizes: Increasing the amount of context that LLMs can process.
• Speed Improvements: Reducing the latency of AI-powered features.
• Cost Reduction: Lowering the cost of running LLMs to enable new use cases.

17. Cost Considerations:

• Token Usage: As AI costs decrease, token usage increases.
• Infrastructure Spending: LLMs account for a significant portion of Shortwave's infrastructure spending.
• Model Selection: Balancing performance and cost when choosing LLMs.

Synthesis/Conclusion:

Shortwave has successfully pivoted from a traditional email client to an AI-powered communication platform. By leveraging LLMs, tool calling, and a robust technical architecture, Shortwave offers a range of intelligent features that enhance productivity and automate tasks. The company is committed to rapid innovation, adopting new models and exploring emerging technologies like MCP and dynamic UI generation. While challenges remain, Shortwave is well-positioned to capitalize on the growing potential of AI in the email and business communication space. The future focus on asynchronous execution and continuous improvements in model performance and cost-efficiency will further expand the capabilities and value of the platform.