Inside Harvey AI’s $8 billion AI lawyer app, PLUS How OpenRouter unites the LLMs | E2207

Key Concepts

• Harvey AI: A company providing AI-powered solutions for the legal profession, focusing on creating a unified workspace for legal matters and enhancing collaboration between law firms and clients.
• Open Router: A platform offering a unified API for accessing a wide variety of Large Language Models (LLMs) and inference providers, simplifying model selection and usage for developers.
• LLMs (Large Language Models): AI models trained on vast amounts of text data, capable of understanding and generating human-like text.
• RAG (Retrieval-Augmented Generation): A technique that combines retrieval of relevant information with text generation to improve the accuracy and relevance of LLM outputs.
• Inference Providers: Companies or services that host and run LLMs, making them accessible for use.
• Billable Hour: A traditional legal billing model where clients are charged for the time lawyers spend on their cases.
• TAM (Total Addressable Market): The total market demand for a product or service.
• ARR (Annual Recurring Revenue): The predictable revenue a company expects to receive from its customers over a year.
• Founder Questions: A segment where Jason Calacanis answers questions from startup founders.
• Network Effects: The phenomenon where a product or service becomes more valuable as more people use it.
• Customer Acquisition Cost (CAC): The cost of acquiring a new customer.
• Vendor Lock-in: A situation where a customer is dependent on a vendor for products or services and cannot easily switch to another vendor.
• Distributor: A company that helps other companies reach a wider customer base.
• Disintermediation: The removal of intermediaries in a supply chain or transaction.
• Forward Deployed Engineers: Engineers who work directly with customers to implement and integrate solutions.
• Customer Success: A department focused on ensuring customers achieve their desired outcomes while using a company's product or service.
• M&A (Mergers and Acquisitions): The consolidation of companies or assets through various types of financial transactions.
• E-discovery: The process of identifying, collecting, and producing electronically stored information (ESI) in response to a request for production in a legal case.
• Case Law: The law as established by the outcome of former cases.
• Foundation Models: Large, general-purpose AI models that can be adapted to a wide range of downstream tasks.
• IDE (Integrated Development Environment): A software application that provides comprehensive facilities to computer programmers for software development.
• CISO (Chief Information Security Officer): A senior-level executive responsible for an organization's information security.
• GenAI (Generative AI): A type of artificial intelligence that can create new content, such as text, images, audio, and video.
• Product Market Fit: The degree to which a product satisfies strong market demand.
• MVP (Minimum Viable Product): A version of a product with just enough features to be usable by early customers who can then provide feedback for future product development.
• Red Ocean: A term used to describe highly competitive markets.
• Blue Ocean: A term used to describe markets with little to no competition.
• Network Liquidity: The ease with which participants can enter and exit a marketplace.
• Moat: A sustainable competitive advantage that protects a company from competitors.

---

Harvey AI: Revolutionizing Legal Practice with AI

Introduction to Harvey AI and its Mission

Harvey AI is a prominent startup in the legal tech space, recognized for its significant valuation and rapid growth. The company aims to leverage Artificial Intelligence to transform the legal profession, addressing inefficiencies and enhancing the capabilities of legal professionals. The core problem Harvey seeks to solve is the lack of a unified workspace for managing complex legal matters, known as "client matters."

Harvey's Product Architecture: Vault and Assistant

Harvey's platform is built around two key components:

• Vault: This feature allows law firms to upload and centralize their proprietary data, including case history, client information, and internal documents. This creates a secure and organized repository of firm-specific knowledge.
• Assistant: This tool utilizes the data stored in Vault, along with external legal resources, to assist lawyers with various tasks. This includes drafting legal documents, performing research, and synthesizing information from diverse sources.

Expanding the Legal Workspace Beyond Individual Lawyers

Harvey's vision extends beyond providing tools for individual lawyers. A significant part of their development focuses on:

• Client Matter Management: Creating a single workspace where lawyers or AI agents can manage entire client matters, from litigation to M&A, end-to-end. This involves integrating with existing legal tools like case law databases, data rooms, and e-discovery platforms.
• Governance and Administrative Controls: Addressing the critical need for large law firms to manage and segregate sensitive client data across tens of thousands of ongoing matters and millions of historical ones.
• Client-Law Firm Collaboration: Developing functionalities that enable seamless collaboration between law firms and their clients on client matters. This includes making clients themselves users of the Harvey platform.

The Network Effect in Enterprise Legal Tech

Harvey is experiencing a powerful network effect, particularly when selling to enterprise clients (e.g., private equity firms, Fortune 500 companies).

• Client-Driven Adoption: Enterprises that adopt Harvey often mandate their external law firms to do the same, creating a strong incentive for law firms to integrate with the platform.
• Law Firm-Driven Adoption: Conversely, law firms are increasingly encouraging their clients to adopt Harvey to improve collaboration and efficiency.
• Reduced CAC: This dual-sided adoption significantly reduces customer acquisition costs for Harvey, as deals are often driven by existing relationships and client mandates.

Why AI is a Ripe Domain for Legal

The legal profession is an ideal fit for AI due to several factors:

• Structured Language and Grammar: Legal documents and case law possess a distinct structure and "grammar" that AI can learn and process effectively.
• Vast Corpus of Data: There is an enormous volume of publicly available case law, legal writings, and scholarly analysis that AI models can learn from.
• Information Synthesis: A significant portion of legal work, especially for junior associates, involves synthesizing massive amounts of information from various sources, including industry-specific knowledge, case law, and discovery documents. AI can significantly accelerate this process.
• Analogies with Coding: Similar to coding, legal work involves complex logic, structured data, and a large body of existing work, making it amenable to AI-driven solutions.

The Evolution of AI in Legal: From Consumer to Corporate

Harvey's journey in applying AI to law has evolved:

• Early Consumer Focus (Pre-GPT-4): Initially, Harvey explored using AI for consumer-facing legal issues (e.g., landlord-tenant disputes). Early models were effective for questions with readily available online information. However, regulatory hurdles like the unauthorized practice of law posed challenges.
• Shift to Corporate Law: The core business has shifted to corporate law, which presents more complex challenges due to the highly sensitive and private nature of the data involved.
• Challenges with Legal Research: While AI models demonstrate strong reasoning on public Supreme Court cases (likely due to extensive public analysis), their performance falters on more nuanced, specialized cases where public analysis is scarce. This highlights the need for models to be trained on proprietary firm data.

Technical Architecture and Data Handling

Harvey's approach to integrating AI models is sophisticated and security-conscious:

• Contextual Integration: A major challenge is connecting all the necessary context for a lawyer working on a specific client matter. This involves integrating data from document management systems, e-discovery platforms, case law, and internal documents.
• Multi-Model Approach: Harvey has integrated models from various providers, including OpenAI, Anthropic, and Alphabet, to offer a multimodal solution.
• Data Privacy and Security: Harvey operates under strict "eyes off" policies, never using client data for training. They are building infrastructure to partition data for individual clients, enabling firms to train systems on their specific client expertise with consent.
• Enterprise Client Needs: Clients of law firms are increasingly seeking a unified legal model that accesses all their company's legal data and connects with their various law firms, learning from their collective work without compromising confidentiality.
• Regulatory Compliance: Harvey is addressing the significant technical, security, and legal challenges of providing GenAI experiences that meet stringent regulatory requirements, akin to the healthcare industry.

Expanding Beyond Legal: Adjacent Industries

The underlying technology and problem-solving approach at Harvey are transferable to other regulated industries where sensitive data needs to be shared with advisors.

• Adjacent Verticals: Harvey is already expanding into areas like tax and is exploring applications in investment banking, consulting, audit, HR, and compliance.
• Core Problem: The common thread is enabling companies to securely share sensitive data with advisors for analysis and operational tasks.

Growth and Customer Mix

Harvey has experienced impressive growth:

• ARR Growth: Scaled from $50 million ARR at the end of the previous year to $100 million ARR by August.
• Client Base: Serves approximately 500 law firms.
• Customer Segments:
  • Large Law Firms: Initially focused on the largest firms, now present in over 50 of the Am Law 100 globally.
  • Mid-Market Firms: Growing rapidly.
  • Enterprise Clients: Clients of law firms, including private equity and Fortune 500 companies, are also a significant growth area.
• TAM Expansion: The inclusion of enterprise clients significantly expands Harvey's TAM, as they also utilize AI for internal legal work (contracting, tax, HR, compliance) not involving outside counsel.

Addressing the Billable Hour Tension

The efficiency gains from AI can create tension with the traditional billable hour model.

• Efficiency vs. Billing: While AI makes firms more efficient, leading to potentially lower billable hours for certain tasks, Harvey aims to create win-win scenarios.
• New Business Models: Law firms can leverage AI to offer fixed-fee services, take on more matters at scale, and potentially achieve better profit margins.
• Decoupling Revenue from Headcount: AI enables law firms to decouple revenue from headcount, allowing them to scale beyond traditional staffing limitations and approach software-like margins.

Competitive Landscape

Harvey views its competition from multiple angles:

• Indirect Competition from General Model Providers: Companies like OpenAI are considered indirect competitors as their general models improve. Harvey's differentiation lies in its vertical-specific expertise and infrastructure.
• Startup Competition: While numerous startups are entering the legal AI space, Harvey focuses on building a comprehensive AI-first law firm transformation platform, encompassing governance, change management, and new service delivery models, which horizontal players are unlikely to replicate.
• Self-Build Risk: In-house legal departments may initially consider building their own solutions, but Harvey anticipates that, similar to Salesforce, the complexity of scaling and maintaining such systems will drive adoption of specialized platforms.

The Role of Customer Success and Forward Deployed Engineers

Harvey emphasizes a scalable platform approach:

• Platform-Centricity: The majority of Harvey's platform is built for broad applicability, rather than custom solutions for each client.
• Forward Deployed Engineers for New Verticals: Forward deployed engineers are primarily used to identify and build the necessary "building blocks" for new verticals (e.g., private equity fund operating systems) that can then be integrated into the core platform.
• Data Discovery Challenges: A common client problem is data discovery and integration across disparate systems, requiring some level of client engagement to map and connect data.
• Scalable Implementation: Despite the need for some client-specific integration, Harvey believes its approach is highly scalable, having successfully sold the same core platform to diverse clients.

---

Open Router: Unifying Access to the LLM Ecosystem

The Problem: LLM Fragmentation and Complexity

The rapid proliferation of Large Language Models (LLMs) and inference providers presents a significant challenge for developers and businesses. With new models emerging frequently, navigating the landscape, integrating them, and managing costs can be complex and time-consuming.

Open Router's Solution: A Unified API Gateway

Open Router acts as a unified API gateway, providing a single point of access to a vast array of LLMs and inference providers.

• Comprehensive Model Access: Offers access to hundreds of models from dozens of inference providers, including major players like OpenAI, Google (Gemini), Anthropic, and open-source models from China and Europe (e.g., Mistral, Kimi K2, MiniMAX M2).
• Simplified Integration: Standardizes and normalizes requests across different models and providers, abstracting away technical complexities and "paper cuts" associated with switching or integrating new models.
• Cost and Time Savings: Eliminates the need for extensive vendor setup, billing relationships, and enterprise agreements for each new model, allowing developers to quickly test and deploy new models.

Key Differentiators from Cloud Providers

Open Router distinguishes itself from cloud-based solutions like Amazon Bedrock and Azure AI Foundry by:

• Broader Model Selection: Offering a significantly wider range of models beyond those directly offered by a single cloud provider.
• Provider Agnosticism: Allowing users to bring their own credits and existing deals, providing a single pane of glass for all LLM interactions, regardless of the underlying provider.
• Error Correction and Quality Improvement: Actively fixing errors and improving the quality of inference across all requests.

The "Twilio for AI" Analogy

Open Router is often compared to Twilio, a company that simplified telecommunications for developers.

• Abstraction of Complexity: Just as Twilio abstracted away the complexities of telecom infrastructure, Open Router abstracts away the complexities of the LLM ecosystem.
• Increased LLM Usage: By making LLMs easier to access and experiment with, Open Router is expected to drive increased overall LLM inference and token usage.

Driving Competition and Distribution

Open Router plays a crucial role in fostering competition and distribution within the AI market:

• Leveling the Playing Field: By making it easier to switch between models, Open Router increases competition among LLM providers, preventing vendor lock-in.
• Distributor for Vendors: For model providers, Open Router acts as a distributor, helping them reach new users and gain early traction, especially for new model releases.
• Stealth Model Launches: Open Router facilitates "stealth" model launches, allowing providers to gather early feedback and benchmarks from a diverse user community.

Business Model and Revenue Streams

Open Router generates revenue through several channels:

• Platform Fee: A 5.5% platform fee on non-free access to models.
• Credit Top-up Fee: For off-the-shelf inference.
• Volume Discounts: Negotiating volume discounts with providers and passing some savings to users.
• Inference-Adjacent Software: Charging for enterprise features like observability, enhanced team management, multimodal support, and advanced file context management.

Growth and Future Outlook

Open Router is experiencing rapid growth:

• Token Processing Volume: The platform is on track to process trillions of tokens weekly, with steady and significant growth in inference volumes.
• Underutilized LLMs: The company believes LLMs are still significantly underutilized, with vast potential for automation across various tasks.
• Domain-Specific Improvements: Future LLM development will focus on improving performance in domain-specific tasks (e.g., product management, agent workloads), moving beyond general drafting.
• Dynamic Benchmarking: Open Router emphasizes dynamic, constantly updating benchmarks to reflect real-world usage and model performance, productizing these benchmarks.
• Auto Router and Recommendations: The "Auto Router" feature helps customers select the best model for their needs. While currently unbiased, future iterations may include more sophisticated model recommendations and potentially sponsored placements (though neutrality is a core principle).
• Inference Cost vs. Salaries: The company predicts that inference costs could eclipse salaries as a dominant operating expense for knowledge-based companies in the coming years, as AI models become more capable and widely adopted.

The Future of LLMs: Commoditization and Differentiation

While some LLMs may become commoditized, Open Router believes differentiation will persist:

• Idiosyncratic Skills: Models will continue to shine based on unique capabilities, such as tool calling (e.g., Open Router's "Exacto" endpoints).
• Dynamic Benchmarks: Open Router's dynamic benchmarking approach will help users identify models excelling in specific areas, even as general performance converges.
• Provider Specialization: Inference providers will differentiate themselves through specialized offerings and tool-calling capabilities.

---

Founder Questions with Jason Calacanis

The Challenge of Rapid Competition and "Vibe Coding"

A founder expressed concern that the rapid pace of development, amplified by AI coding tools ("vibe coding"), makes it nearly impossible to build a startup today. They argue that any product launched is quickly copied, collapsing pricing power and making profitability a distant dream.

Jason's Perspective: Bubble Behavior and Marathon Running

Jason Calacanis characterizes this sentiment as "peak bubble behavior," where abundant capital and numerous entrepreneurs chase the same ideas.

• The Nature of Competition: He acknowledges that many companies will start, but only a few will ultimately succeed. This is a natural process of market consolidation.
• Winning the Marathon: Startups that win are those with discipline, a strong team, consistent feature shipping, and robust network liquidity.
• Defensible Moats: Beyond the initial product, defensible moats include:
  • Team: The quality and capability of the founding team.
  • Network Liquidity: For marketplaces, the ease of transactions and availability of participants (e.g., Airbnb's vast inventory vs. a new platform with few listings).
  • Sales Teams: For enterprise software, a strong sales force can be a significant advantage (e.g., Salesforce, Oracle).
  • Capital: The ability to raise more capital than competitors to fund marketing, R&D, and distribution.
• Stages of Startup Success: Jason outlines a progression where competitors are lost at each stage:
  1. Idea: Everyone has an idea.
  2. Product Completion: Few can finish the product.
  3. Distribution & Sales Cadence: Even fewer can maintain consistent go-to-market momentum.
  4. Scaling & Fundraising: A smaller group can raise capital at scale.
  5. Maintenance & Sustainability: Ultimately, only a few can sustain their success.
• Distribution and Marketing: Jason emphasizes that superior technology doesn't always win. Marketing and sales strategies are crucial. He highlights Perplexity's aggressive early partnerships and affiliate programs as an example of building distribution.
• Hot vs. Quiet Markets:
  • Hot Market: Easier to raise money, but harder to get attention due to intense competition.
  • Quiet Market: Harder to raise money, but easier to get attention for a product as there's less noise.
  • Conclusion: Entrepreneurship is always hard, regardless of market conditions.

The "Long Walk" Analogy

Jason uses the analogy of Stephen King's "The Long Walk" to describe the arduous nature of entrepreneurship, where only the last person standing wins, implying a brutal, survival-of-the-fittest dynamic. He also muses on the potential historical inspiration for the book.