Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates

Key Concepts

• Digital Workforce: A platform for designing and managing AI agents and human-computer interactions.
• Affordances: Focusing on the capabilities and possibilities of AI agents rather than specific, pre-defined features.
• Emergent Behavior: Unexpected functionalities and behaviors that arise from AI agents interacting with their environment.
• Evals: Testing frameworks for probabilistic AI, used to evaluate the performance and behavior of AI agents against desired criteria.
• Vibe Coding: Rapid prototyping and experimentation to get a feel for the user experience and interaction with AI agents.
• Probabilistic AI: AI systems that operate based on probabilities and statistical models, rather than deterministic rules.

1. Introduction: Stacy and the Google Doc Comment

The speaker, a founder and product manager at Teammates, introduces Stacy, an AI teammate (an L3 engineer) who interacts with users through collaboration tools like Google Workspace and Slack. The speaker highlights a customer question about tagging Stacy in a Google Doc comment, which prompted the speaker to realize the uncertainty around the AI teammate's behavior in such scenarios. This uncertainty serves as the central theme of the talk: how to develop and manage products when the underlying AI's capabilities are not fully known.

2. The Transformation of Product Management

The speaker argues that product management is undergoing a profound transformation due to two main factors:

• Unpredictable LLMs: Products are increasingly built on top of Large Language Models (LLMs), whose knowledge and behavior are inherently unpredictable. We don't know what the LLMs know.
• Boundless Customer Expectations: Users expect AI-powered products to handle a wide range of tasks and interactions, leading to a vast and undefined surface area.

3. Shifting Mindsets: Affordances and Emergent Behavior

The speaker proposes a shift in mindset from defining specific requirements to focusing on affordances. Instead of specifying exactly how Stacy should respond to a Google Doc comment, the focus should be on providing her with the affordance to comment, communicate, or collaborate.

• Affordances vs. Specific Requirements: Instead of dictating specific actions, provide the AI with the ability to perform a range of related tasks.
• Trusting the LLM: Rely on the AI's internal mechanisms (agentic workflow, work planning) to determine the appropriate action within the given affordances.

The speaker also emphasizes the importance of recognizing that behavior is emergent. AI agents may exhibit unexpected functionalities and behaviors. The role of product managers is to discover these emergent behaviors and identify the right building blocks (Lego bricks) for users to compose their own solutions.

4. Evals as the New Specification

The speaker advocates for using evals (evaluation frameworks) as the new specification for AI-powered products. Evals are testing frameworks for probabilistic AI, used to evaluate the performance and behavior of AI agents against desired criteria.

• Evals for Probabilistic AI: Evals provide a way to test and measure the behavior of AI agents in scenarios where deterministic testing is not possible (e.g., "was she snarky in Slack?").
• Product People and Evals: Product managers should actively review evals to understand what the product can do and to inform product development decisions.
• Behavior Driven Development (BDD) Analogy: While previous attempts to involve business people in writing tests (BDD) were unsuccessful, evals offer a more meaningful way for product managers to understand and specify AI behavior.

5. Vibe Coding for Intuitive Understanding

The speaker introduces the concept of vibe coding as a crucial tool for product management in the age of AI. Vibe coding involves rapid prototyping and experimentation to get a feel for the user experience and interaction with AI agents.

• Feeling the Experience: Vibe coding allows product managers to experience the interaction with an AI agent firsthand, identifying nuances and potential issues that are difficult to anticipate through traditional specification methods.
• Avoiding Over-Specification: Vibe coding helps avoid the trap of over-specifying AI behavior, which can lead to rigid and unnatural interactions (e.g., the "certainly" issue with the Claude AI).
• Not a Replacement for Engineering: Vibe coding is not meant to replace formal engineering processes but to inform and guide development efforts.

6. Discovering Functionality and Debugging Challenges

The speaker highlights the importance of discovering functionality through experimentation and exploration. This involves trying out different scenarios and use cases to uncover the emergent behaviors of AI agents.

• QA Analogy: The speaker uses a joke about a QA engineer testing a bar to illustrate the unpredictable nature of real-world usage and the need for continuous discovery.
• Redefining Bugs: The speaker discusses the challenges of defining and debugging issues in AI-powered products. It can be difficult to determine whether an unexpected behavior is a bug, a feature, or simply an emergent property of the AI.
• Credibility and Metrics: Establishing clear metrics and acceptance criteria (based on evals) is crucial for building credibility and resolving disagreements about what constitutes a bug.

7. Customer Interactions in the Age of AI

The speaker reflects on the changing dynamics of customer interactions. Traditional product management roles (visionary, honest broker) are difficult to play when the product's capabilities are uncertain.

• The Future is Uncertain: Customers may not believe in the visionary roadmap because the technology seems like "witchcraft."
• The Present is Unknown: Product managers may not be able to provide honest assessments of the product's capabilities because they themselves don't fully understand them.
• Inventing the Future Together: The speaker suggests framing customer interactions as a collaborative effort to "invent the future together." This involves acknowledging the uncertainty and inviting customers to participate in the discovery process.

8. Conclusion: Embracing the Unknown

The speaker concludes by emphasizing the excitement and challenges of building AI-powered products. The product management discipline is undergoing a rapid transformation, requiring a willingness to adapt, experiment, and embrace the unknown. While core principles like listening to customers and solving real problems remain important, the tools and techniques of product management are being fundamentally upended. The speaker encourages product people and AI engineers to work together to build awesome products in this new era.

Notable Quotes

• "How do I ship a product? How do I develop a product? How do I talk to customers? How do I instill trust when I don't know what my own product can do?"
• "I think what's happening is the product management discipline is going to undergo a transformation, a shift, an evolution, whatever you call it, that is super profound."
• "We're actually building things and then discovering what they can do themselves."
• "Evals are a testing framework for probabilistic AI for agents."
• "I've never had more fun building. I've never felt like both more inept and like more excited about what what I'm doing."

Key Takeaways

• Product management is evolving to accommodate the unpredictable nature of AI.
• Focus on providing affordances rather than specifying exact behaviors.
• Embrace emergent behavior and discover new functionalities through experimentation.
• Use evals as the new specification for AI-powered products.
• Employ vibe coding to gain an intuitive understanding of the user experience.
• Frame customer interactions as a collaborative effort to invent the future.
• Be prepared to adapt and forget much of what you used to know about product management.