Building AI Products That Actually Work - Ben Hylak, Sid Bendre

Key Concepts

• AI Product Iteration
• Evals (Evaluations) - Offline vs. Online
• Signals (Explicit & Implicit)
• Intents
• Trellis Framework (Discretization, Prioritization, Recursive Refinement)
• Workflows (Semi-deterministic)
• Estimated Achievable Delta

Building AI Products That Actually Work

Introduction

Ben Hilac, CTO of Raindrop, and Sid, co-founder of Aliv, discuss strategies for building successful AI products, emphasizing iteration and a structured approach to managing the inherent chaos of AI. The core argument is that continuous refinement based on user data and signals is crucial for creating reliable and engaging AI experiences.

The Importance of Iteration

Iteration is presented as a critical component of building effective AI products. The speaker argues that you can't fully define the scope of a product's behavior upfront due to the evolving capabilities of AI models and the increasing complexity of integrations.

Debunking Eval Misconceptions

The speaker addresses common misconceptions about evals:

• Lie #1: Evals tell you how good your product is. Evals only capture what you already know and are easily saturated (Goodhart's Law). Real-world performance often differs from eval scores.
• Lie #2: Using LLMs as judges (e.g., "How funny is my joke?") works well. The best companies use highly curated datasets and autogradable evals (deterministic pass/fail).
• Lie #3: Eval production data is straightforward. It can be expensive, inaccurate, and limited to pre-existing knowledge. OpenAI's experience with ChatGPT highlights that real-world usage reveals issues that evals miss.

The Role of Signals

Signals are defined as "ground truthy indicators" of an app's performance. They are essential because AI apps lack concrete errors like exceptions. An AI issue is composed of a combination of signals (implicit and explicit) and user intents.

Defining Signals

• Explicit Signals: Analytics events sent by the app (e.g., thumbs up/down, copying portions of a message, preference data). Examples include tracking syntax errors in coding assistants or the correctness of search results.
• Implicit Signals: Detected behaviors (e.g., refusals, task failures, user frustration). Clustering these signals can reveal patterns and issues.

Exploring Signals

Exploring signals involves analyzing metadata (properties, models, keywords) and user intents to understand the context of issues.

Refining Signals

The process involves constantly looking at data, talking to users, and defining new issues that were not initially anticipated.

Trellis Framework by Aliv

Sid introduces the Trellis framework, designed for continuously refining AI experiences to scale viral products while maintaining reliability.

Core Aims

1. Discretization: Breaking down the infinite output space into specific, manageable buckets.
2. Prioritization: Ranking these buckets based on their potential impact on the business.
3. Recursive Refinement: Repeating the process within the prioritized buckets to create structure and order.

Six Steps of Trellis

1. Initialize Output Space: Launch an MVP agent based on product priors and expectations to collect user data.
2. Classify Intents: Categorize user data into intents based on usage patterns to understand why users are engaged.
3. Convert to Workflows: Transform intents into semi-deterministic workflows (predefined steps to achieve a specific output). Workflows should be broad enough to be useful but narrow enough to be reliable.
4. Prioritize Workflows: Rank workflows using a scoring mechanism tied to company KPIs.
5. Analyze Workflows: Understand failure patterns and sub-intents within each workflow.
6. Recursive Refinement: Continuously iterate and refine the workflows based on analysis.

Prioritization Scoring Mechanisms

• Naive (Volume Only): Focuses on workflows with the highest volume.
• Recommended (Volume x Negative Sentiment): Considers the negative sentiment associated with each workflow.
• Informed (Negative Sentiment x Volume x Estimated Achievable Delta x Strategic Relevance): Incorporates the estimated achievable delta (potential improvement) and strategic relevance.

Benefits of Trellis

• Structured workflows that are self-attributable, deterministic, and self-bound.
• Faster and more reliable team movement because changes are contained within specific workflows.
• Engineered, repeatable, testable, and attributable AI experiences.

Conclusion

Building successful AI products requires a shift from upfront definition to continuous iteration. By focusing on signals, understanding user intents, and employing frameworks like Trellis, developers can manage the chaos of AI and create reliable, engaging, and scalable experiences. The key takeaway is that AI product development is an ongoing process of refinement driven by data and user feedback.