Jackson is the Retention Team Lead at Duolingo

Key Concepts:

• Streak extension in Duolingo
• User engagement and retention
• Funnel analysis (user acquisition and completion)
• Unit of measure for streaks
• Target user profile
• Level of commitment

Experiment on Streak Extension

Duolingo conducted an experiment to determine if allowing users to extend their streak by completing just one exercise within a lesson would improve user engagement and retention. The initial hypothesis was based on funnel analysis, which showed a significant drop-off of users who started lessons but didn't finish them, thus failing to extend their streaks and potentially leading to churn.

Unexpected Results

The experiment yielded surprising results: Daily Active Users (DAU) remained unchanged. This indicated that the intervention had no measurable impact on overall user activity.

Analysis of the Outcome

The lack of impact was attributed to several factors:

1. Inappropriate Unit of Measure: The assumption that users would be motivated to complete just one question to maintain their streak proved incorrect. Users don't typically think in terms of completing a single question.
2. Target User Profile: The users who were most likely to take advantage of the "one exercise streak extension" were the least engaged users. These users were characterized by completing only one or two questions before leaving the app.
3. Level of Commitment: The experiment inadvertently targeted users with a very low level of commitment to language learning. These users were unlikely to be retained regardless of the streak extension mechanism.

Key Takeaway

The experiment highlighted the importance of carefully considering the target user profile and the level of commitment when designing streak mechanisms. It's crucial to understand what kind of habit you're building and for whom. Over-indexing on a type of user who is unlikely to be retained can lead to ineffective interventions.

Quote:

"Nobody thinks about I just want to come do one question on Duolingo. Nobody thinks about that." This quote emphasizes the mismatch between the designed intervention and the user's actual behavior and mindset.

Conclusion

The Duolingo experiment serves as a cautionary tale about the importance of aligning streak mechanisms with the target user's behavior and level of commitment. While the initial hypothesis was based on sound funnel analysis, the experiment revealed that the intervention inadvertently targeted the least engaged users, rendering it ineffective. The key takeaway is to carefully consider the user profile and the type of habit being built when designing streak-based engagement strategies.