The insight that grew retention by 20%

Key Concepts:

• Virtual Coach: An AI-driven feature within a chess game designed to provide feedback and guidance to players.
• User Engagement: The level of interaction and involvement users have with a product or feature.
• User Retention: The ability of a product to keep its existing users over a period of time.
• Game Review: The process of analyzing a completed chess game, often with the assistance of the virtual coach, to understand moves and outcomes.
• Counterintuitive Insight: A discovery or observation that goes against initial assumptions, common sense, or expected behavior.
• Blunders: Significant mistakes or poor moves made during a chess game.
• Brilliant Moves: Exceptionally good, insightful, or strategically advantageous moves made during a chess game.
• Product Experience Redesign: The process of re-evaluating and modifying the way users interact with and perceive a product or specific features.
• Cold Patterns: Areas or features within a product that users might avoid, underutilize, or perceive negatively, potentially leading to disengagement.
• Cross-functional Insight Sharing: The practice of disseminating valuable findings, data, and learnings across different teams or departments within an organization.

1. Initial Product Design and User Behavior Hypothesis The core product is a chess game featuring a "virtual coach" whose primary objective is to enhance user engagement and retention. The coach's function is to provide feedback on players' "worst moves" and "best moves" after a game. Initially, the product developers operated under the assumption that users would primarily engage with the game review feature after losses, seeking to identify and learn from their "mistakes" to improve their play.

2. Counterintuitive User Behavior Discovery During an "exploratory phase" aimed at increasing game review activity, a significant and "counterintuitive" user behavior pattern was observed. Data revealed that "80% of people that review their games actually do so after a win." This finding directly contradicted the initial hypothesis that losses would be the primary driver for game review engagement.

3. Product Experience Redesign and Implementation Based on this critical insight, a strategic change was implemented in the product experience, specifically for users who lose a game.

• Previous Approach: After a loss, the system would "surface your blunders," highlighting the player's mistakes.
• New Approach: The experience was "flipped on its head." Instead of focusing on errors, the system now:
  • Highlights "your brilliant moves, your best moves."
  • Features the virtual coach delivering "encouraging" messages, such as "Losing just part of learning, keep it up." This shifts the focus from negative reinforcement to positive encouragement and resilience.

4. Dramatic Impact and Business Outcomes The product change yielded "pretty dramatic" and quantifiable positive results:

• Game Reviews: Increased by "25%."
• Subscriptions: Grew by "20%."
• User Retention: Improved "by a lot." These statistics provide strong evidence for the effectiveness of the redesigned post-loss experience in driving key business metrics.

5. Broader Organizational Implications and Framework for Improvement The speaker emphasizes that the learning "doesn't just stop there." The "insight" – that positive reinforcement significantly drives engagement even after a negative event – must be "share[d] broadly across the company." This enables "adjacent product managers," such as those working on "puzzles," to:

• "Audit these cold patterns in my product": Identify areas or features within their own products where users might disengage, avoid interaction, or have negative experiences.
• "Think about making them more positive": Apply the principle of positive framing and reinforcement to transform these "cold patterns" into more engaging and encouraging experiences, thereby improving overall product stickiness and user satisfaction.

Synthesis/Conclusion: This case study underscores the critical importance of data-driven user behavior analysis over initial assumptions in product development. By observing the "counterintuitive" preference for positive reinforcement even after a loss, the team successfully redesigned the post-game experience, leading to significant increases in game reviews, subscriptions, and user retention. The key takeaway extends beyond this specific feature, advocating for a company-wide approach to identify and transform "cold patterns" across the product ecosystem by applying insights like the power of positive framing, fostering a culture of continuous improvement and cross-functional learning.