It doesn’t need to be a chatbot: Unlocking the product value of smaller AI models

Key Concepts

• AI Integration Reality Check: The common, often unsuccessful, approach of directly integrating large language models (LLMs) like ChatGPT into existing products without proper consideration for data, context, and risks.
• Organic and Incremental AI Integration: A more successful approach that focuses on building smaller, less disruptive AI features into products gradually.
• AI Opportunity Tree: A framework for identifying and structuring AI use cases within a product, categorized by four main benefit areas.
• Internal Enabler: Using AI to improve product operations and decision-making from within, rather than directly exposing it to users.
• User-Facing Features: Adding AI capabilities directly into the product experience for users.
• Smaller AI Models: Utilizing predictive AI models or smaller language models (e.g., with a few billion parameters) for more manageable and cost-effective AI implementation.
• Brownfield Product: An existing product with established data, knowledge, and user context.
• Retrieval Augmented Generation (RAG): A technique to improve LLM responses by retrieving relevant information from external data sources.
• Context Engineering: The process of preparing and providing relevant context to an LLM.
• Hallucinations, Bias, Toxic Behavior: Risks associated with LLMs that can negatively impact product and company reputation.
• AI Agents/Agentic AI: Advanced AI systems designed to perform tasks autonomously, often requiring a foundation of successful, smaller AI features.

Reality Check: The Pitfalls of Large-Scale AI Integration

The presentation begins by highlighting a common scenario in companies where the decision to "use AI" is made spontaneously, often driven by competitive pressure, excitement about new technology, or directives from leadership. This frequently leads to the immediate goal of building a chatbot.

Key Points:

• Default Thinking: The immediate impulse is often to build a chatbot, treating it as a direct equivalent to models like ChatGPT.
• Mismatch: A company's existing product, with its unique data, knowledge, and context, is fundamentally different from a general-purpose LLM.
• Two Substantial Challenges:
  1. Integration: Loading the product's extensive data, context, and user knowledge into an LLM is complex. Simple solutions like advanced context engineering or RAG systems are often insufficient, especially if the necessary databases are not in place. Data is frequently incomplete, disconnected, and contains blind spots.
  2. Risk Exposure: LLMs possess vast capabilities beyond what is needed for a specific product. This excess functionality introduces new risks, including hallucinations (generating false information), bias, and toxic or harmful behavior. These risks often surface later in production, leading to negative consequences.
• High Failure Rate: The presentation cites figures from the Rand Corporation (2024) and similar reports from McKinsey, indicating that approximately 80% of AI initiatives in companies fail. This underscores the low odds of success with a "big bang" approach.

An Alternative Approach: Organic and Incremental AI Integration

In contrast to the "big bang" approach, the speaker advocates for a more organic and incremental strategy, quoting Andrej Karpathy: "When implementing AI, I see more organizations fail by starting too big than starting too small."

Key Points:

• Starting Small: This involves two primary avenues:
  • Smaller AI Models: Utilizing predictive AI models or smaller language models (e.g., with a few billion parameters) that are more manageable, cost-effective, and less risky.
  • Smaller, Organic Features: Building AI features that are incremental, do not disrupt the existing user experience, and have a lower risk profile.
• Brownfield Product Integration: For existing products, AI can be integrated in two ways:
  • Internal Enabler: Using AI to improve internal product operations, such as enhancing product analytics for better development and prioritization decisions.
  • User-Facing Features: Adding AI capabilities directly into the product experience for users, which is considered more glamorous but also riskier.

The AI Opportunity Tree: A Framework for Identifying Use Cases

The AI Opportunity Tree is presented as a tool to identify and structure AI use cases within a product. It starts by defining broad benefits of AI and then drills down into specific product features and use cases.

Four Main Branches of Benefits:

1. Gaining Deeper Insights:

   • Concept: Leveraging existing product data to extract more intelligence.
   • Examples:
     • Clustering customer pain points from support tickets and product reviews.
     • Analyzing user journeys to identify friction points and drop-off areas.
   • Application: Using these insights to inform development and prioritization decisions.

2. Removing Friction:

   • Concept: Addressing existing inefficiencies and limitations in the product's user experience that were previously constrained by technology.
   • Example (Booking.com):
     • Problem: Long, complex filter bars in search-based applications are difficult to design and use, often failing to meet user needs.
     • Solution: Implementing a "smart filter" feature that allows users to input queries in natural language. The AI then translates these queries into structured filter categories.
   • Technical Insight: This is particularly effective with smaller language models in areas with significant structural knowledge and numerous categories, allowing a switch from categorical thinking to natural language interaction.

3. Adding New Functionality:

   • Concept: Observing adjacent user actions outside the product and automating or implementing them with AI.
   • Methodology: This follows a classical product management approach of identifying user needs and potential AI-driven solutions.

4. Personalizing the User Experience:

   • Concept: Tailoring content or the overall user experience to individual users.
   • Examples:
     • Content Personalization: Adapting the style and tone of messages.
     • User Experience Personalization: Prioritizing specific items (e.g., pets, drinks, baggage) in the user flow based on user preferences.

Prioritization within the Tree: The AI Opportunity Tree suggests a prioritization strategy: start with easier, less risky use cases on the left and move towards more complex and risky ones as momentum and experience are gained.

Synthesis and Conclusion: Key Takeaways

The presentation concludes with actionable advice for companies looking to integrate AI.

Key Takeaways:

• Start Small, Aim for Gradual Improvement: Focus on incremental enhancements rather than disruptive changes.
• Iterate Faster, Lower Cost/Risk with Smaller Models: Smaller AI models enable quicker experimentation at a reduced cost and risk. Many are also free to use.
• Be Cautious with User Uncertainty: Introduce AI features gradually into the user experience. Prioritize internal AI enablement before exposing it directly to users.
• Foundation for Complex Applications: A bottom-up approach, building successful smaller AI features, creates a strong foundation for more complex applications like AI agents and agentic AI in the future. It also validates the quality and readiness of the underlying data.