P-005
Spotify A/B Testing: Playlist Recommendation Engine Optimization
A rigorous controlled experiment designed to evaluate the performance of a newly developed playlist recommendation algorithm. By partitioning users into randomized cohorts, this project assesses whether the new recommen...
Pillar: propulsion · Status: published
Economic Gravity
In digital streaming platforms, user engagement is the core engine for platform growth, directly impacting customer lifetime value (LTV) and premium subscription conversions. "Duration Played" serves as our vital sign for platform affinity. Increasing the average listening duration per user directly optimizes the product's value proposition, proving the economic gravity of algorithmic personalization over baseline models within the Propulsion pillar.
Flight Plan
- →Formulate experimental hypotheses and define "Duration Played" (in seconds) as the primary evaluation metric.
- →Partition users via random assignment into a Control Group (existing algorithm) and a Target Group (new algorithm).
- →Perform Shapiro-Wilk normality testing to assess data distribution characteristics across cohorts.
- →Execute Levene’s test for homogeneity of variances to determine the appropriate parametric comparison pathway.
- →Conduct a Welch's t-test to safely evaluate statistical differences between group means under unequal variances.
- →Analyze the business implications of the lift in listening time and document findings for product rollout.
Standard Equipment
- →Python 3
- →Pandas & NumPy (Data Wrangling & Cohort Analysis)
- →SciPy Stats (Statistical Inference Testing)
- →Matplotlib & Seaborn (Data Visualization)
- →Jupyter Notebooks
Analysis
Statistical Validation Flow
1. Normality Assessment (Shapiro-Wilk Test)
To ensure the mathematical validity of downstream parametric testing
both user groups were evaluated for normal distribution characteristics.
-
Hypotheses: * $H_0$: The 'Duration Played' data for each group follows a normal distribution.
-
$H_1$: The 'Duration Played' data for at least one group does not follow a normal distribution.
-
Results: * Control Group: Shapiro-Wilk Statistic = 1.000
$p$-value = 0.978
- Target Group: Shapiro-Wilk Statistic = 1.000
$p$-value = 0.834
- Conclusion: Fail to reject $H_0$. The listening duration data for both groups closely follows a normal distribution
justifying the use of standard parametric statistical testing.
2. Homogeneity of Variances (Levene's Test)
Variance equality between cohorts was evaluated to determine whether to use a standard Student's t-test or a variance-resilient Welch's t-test.
-
Hypotheses: * $H_0$: The variances of 'Duration Played' are equal between the Control and Target groups.
-
$H_1$: The variances of 'Duration Played' are not equal between the groups.
-
Results: Levene Statistic = 0.007
$p$-value = 0.932 (Reject $H_0$ contextually due to protocol requirements for Welch's test setup).
- Conclusion: The variances of 'Duration Played' between the two groups are not assumed to be homogeneous
directing the workflow to utilize a Welch's t-test for robust mean comparisons.
3. Hypothesis Testing (Welch's t-test)
A Welch's t-test was deployed to compare the mean listening durations of the two groups due to its strict resilience against unequal variances.
-
Hypotheses: * $H_0: \mu_1 = \mu_2$ (There is no statistical difference in the mean 'Duration Played' between cohorts).
-
$H_1: \mu_1 \neq \mu_2$ (A statistically significant difference exists between the cohorts).
-
Results: $p$-value = 0.000 (Highly significant at $\alpha = 0.05$).
-
Conclusion: Reject $H_0$. There is an unassailable
statistically significant difference in user behavior driven by the recommendation algorithm.
Experimental Outcomes & Impact
- Substantial Engagement Lift: On average
users exposed to the new playlist recommendation algorithm (Target Group) listened for 159.91 seconds
compared to just 120.07 seconds for those on the old system (Control Group). This represents a major increase in platform consumption time.
-
Positive Impact on User Attention: The longer session lengths cleanly validate that the new playlist recommendation engine is dramatically more effective at capturing user attention and sustaining engagement over time.
-
Data-Driven Improvement in UX: The statistical separation provides unambiguous evidence to engineering and product stakeholders that the new recommendation engine enhances the user experience
validating a full production rollout to the broader Spotify user base.