Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Key Concepts

• Inference Market: The market for running and utilizing AI models, potentially the largest in software.
• Long Tail of Language Models: The idea that there will be a vast number of specialized, smaller language models, each with unique data and capabilities.
• Model Aggregation: Collecting and providing access to a wide variety of language models through a single API.
• Middleware (AI-Native): A system for augmenting and transforming the inputs and outputs of language models, enabling features like web search and PDF parsing.
• Multimodal AI: AI systems that can process and generate multiple types of data, such as text and images.
• Routing: Directing requests to the optimal model and provider based on factors like price, performance, and geographical location.

Open Router Founding Story and Investigation

The speaker started Open Router in early 2023 to investigate whether the AI inference market would be "winner take all," dominated by a single model like OpenAI's GPT. Initial observations in January 2023 revealed user interest in models with transparent moderation policies, especially for content generation scenarios where OpenAI's restrictions were problematic.

In February 2023, Meta's LLaMA 1 emerged, an open-weights model that surprisingly outperformed GPT-3 on some benchmarks. This was a pivotal moment, suggesting that smaller, locally runnable models could compete with large, server-based models. However, LLaMA 1 was still difficult to use and primarily a text completion model.

The real breakthrough came in March 2023 with Alpaca, a model fine-tuned from LLaMA 1 using GPT-3 generated data. Alpaca demonstrated the transfer of style and knowledge from large models to smaller ones, costing less than $600 to create. This opened the door to creating unique data-as-a-service language models. The speaker realized the need for a platform to discover and understand these specialized models.

Window AI and Open Router Launch

In April 2023, the speaker launched Window AI, an open-source Chrome extension that allowed users to select their preferred model and integrate it into web applications. This demonstrated the feasibility of users bringing their own models to generic websites.

The following month, Open Router was officially launched, co-founded with Lewis from Plasmo. Initially, it served as a collection of models, but it evolved into a marketplace offering better prices, uptime, and model choice.

Open Router as a Marketplace

Open Router has experienced significant growth, with 10-100% month-over-month growth over the last two years. It provides a single API to access various language models, offering near-zero switching costs. The platform boasts over 400 models from 60+ active providers, supporting multiple payment methods, including crypto. Open Router handles the complexities of normalizing tool calls and caching.

The platform initially started with a primary and fallback provider for each open-source model. As more providers emerged with varying prices, performance, and feature support (e.g., min-p sampling, caching, tool calling), Open Router evolved into a marketplace aggregating providers at different price points.

Open Router helps developers improve uptime by aggregating multiple providers for a single model. It also provides real-world data on latency and throughput, enabling informed model selection.

Data and Market Analysis

Open Router's data suggests that the AI market is not "winner take all." Google's Gemini has grown from 2-3% to 34-35% of token processing on the platform over the last 12 months. Anthropic is also a popular model.

The speaker believes the future is multimodal, with customers using different models for different purposes. Inference is becoming a commodity, and Open Router aims to make models from different providers (e.g., Claude from Bedrock vs. Claude from Vertex) interchangeable. Selecting and routing requests to the optimal model will be crucial.

Technical Story: AI-Native Middleware

Open Router developed its own AI-native middleware system to expand inference with new features like web search and PDF parsing. This middleware allows augmenting model outputs on the way back to the user.

The web search plugin, for example, adds web annotations to any language model's output in real-time, within the streaming process.

Technical Challenges and Solutions

Open Router addressed several technical challenges:

• Low Latency: Achieved industry-leading latency (around 30 milliseconds) through custom caching.
• Stream Cancellation: Standardized stream cancellation policies across different providers to avoid unexpected billing.
• Provider Standardization: Developed a robust architecture to standardize diverse providers and models.

Future Directions

Open Router plans to:

• Add more modalities, such as image generation (transfusion models).
• Implement more powerful routing, including geographical routing and enterprise-level optimization.
• Improve prompt observability and model discovery with fine-grained categorization.
• Continue to offer better prices.

Conclusion

Open Router's journey from an experiment to a marketplace highlights the evolving landscape of AI inference. The platform's focus on model aggregation, AI-native middleware, and technical innovation positions it as a key player in enabling a diverse and accessible AI ecosystem. The speaker emphasizes collaboration and building a durable ecosystem with low vendor lock-in.