Meaningful metrics - How data sharpened the focus of product teams

17-02-2023

url: https://blog.duolingo.com/growth-model-duolingo/
rely on organic growth
- business grows because users love the product, offered for free, some user convert to paid subscriptions
defining metrics
- how to decide on metrics
  - movable metrics that matter
- how to advocate to adopt new metrics
- how to evaluate if current metrics are obsolete
case study on daily active users
- problem
  - growth started stagnating, difficult to optimize and improve the metric
- solution
  - how can we focus on metrics that indirectly drive DAU
  - top-down growth model that breaks down topline metrics into smaller user segments
    - all members classified into activity state, monitor rates of transition between states
      - transition probabilities
        
        retention rates
        
        deactivation rates
        
        activation rates
    - 7 mutually-exclusive user states
      - New users: learners who are experiencing Duolingo for the first time ever
      - Current users: learners active today, who were also active in the past week
      - Reactivated users: learners active today, who were also active in the past month (but not the past week)
      - Resurrected users: learners active today, who were last active >30 days ago
      - At-risk Weekly Active Users: learners who have been active within the past week, but not today
      - At-risk Monthly Active Users: learners who were active within the past month, but not the past week
      - Dormant Users: learners who have been inactive for at least 30 days
- considerations
  - aliasing users across devices and platforms
    - prevent double counting, ensure accuracy of results
  - top-down metrics (aggregate)
    - preconceived notion of what matters (this may be incorrect)
      - reduces a diverse group into an average, which become lossy as that group grows
    - current users state became the biggest state (90% of users), so optimizing for this group as a whole leaves a lot on the table
      - not able to see all of the distinct, diverse learners in each state
      - metric may become too big to move or impact
  - bottom-up methods for user segmentation
    - don't reduce user segments down to an average
    - leverage unsupervised learning to identify unexpected patterns in the data
  - defining a ceiling for top-level metrics
    - current-user retention rate
      - cannot be 100%, what is meaningful and how to convey this to the org
- defininig metrics
  - top-level KPI
    - DAU
  - top-level metrics (aggregated)
    - WAU and MAU
- calculations
  - ```
  # KPI
  DAUt = ReactivatedUsert + NewUsert + ResurrectedUsert + CurrentUsert
  
  # high-level aggregates metrics
  WAUt = ReactivatedUsert + NewUsert + ResurrectedUsert + CurrentUsert + AtRiskWAUt
  MAUt = ReactivatedUsert + NewUsert + ResurrectedUsert + CurrentUsert + AtRiskWAUt + AtRiskMAUt
  
  # active states
  ReactivatedUsert = ReactivationRatet * AtRiskMAUt-1
  ResurrectedUsert = ResurrectionRatet * DormantUserst-1
  CurrentUsert = NewUsert-1 * NURRt + ReactivatedUsert-1 * RURRt + ResurrectedUsert-1 * SURRt + CurrentUsert-1 * CURRt + AtRiskWAUt-1 * WAURRt
  
  # inactive users
  DormantUsert = DormantUsert-1 * DormantRRt + AtRiskMAUt-1 * MAULossRatet
  AtRiskMAUt = AtRiskMAUt-1 * ARMAURRt + AtRiskWAUt-1 * WAULossRatet
  AtRiskWAUt = AtRiskWAUt-1 * ARWAURRt + CurrentUsert-1 * (1-CURRt) + ReactivatedUsert-1 * (1-RURRt) + NewUsert-1 * (1-NURRt) + ResurrectedUsert-1 * (1-SURRt)
```