Can we see churn coming?
A risk score is only useful if it ranks tomorrow's churners above everyone else. Each model was trained on 2016 renewal decisions, tuned on Q4 2016, and scored once on January–February 2017: months it never saw, the way it would be used in practice.
The best model (Gradient boosting) reaches an AUC of 0.944. Contacting its riskiest 10% of expiring members would reach 82% of all churners, against 68% for a hand-written rule score and 10% for random outreach.
How many churners each approach finds
Sort expiring members from highest to lowest risk and work down the list: the curve shows the share of eventual churners reached. The diagonal is random outreach; the higher the curve, the less retention budget is wasted on members who were going to stay.
Show data table
| Top % contacted | Gradient boosting | Rule-based score | Gradient boosting · listening only |
|---|---|---|---|
| 10% | 82% | 68% | 32% |
| 20% | 91% | 89% | 47% |
| 30% | 93% | 90% | 56% |
| 40% | 96% | 91% | 62% |
| 50% | 98% | 93% | 69% |
| 60% | 99% | 94% | 76% |
| 70% | 100% | 95% | 84% |
| 80% | 100% | 97% | 90% |
| 90% | 100% | 98% | 95% |
| 100% | 100% | 100% | 100% |
Test AUC by approach
0.5 is a coin flip, 1.0 is perfect. The model is chosen on validation AUC, never on test.
Show data table
| Approach | Validation AUC | Test AUC | Test PR-AUC |
|---|---|---|---|
| Rule-based score | 0.836 | 0.901 | 0.458 |
| Logistic regression | 0.840 | 0.918 | 0.572 |
| Gradient boosting | 0.874 | 0.944 | 0.613 |
| Logistic regression · listening only | 0.629 | 0.666 | 0.101 |
| Gradient boosting · listening only | 0.661 | 0.676 | 0.127 |
What moves the risk, holding everything else equal
Logistic regression odds ratios per standard deviation (or for yes vs no). Right of the line raises churn odds, left lowers them. These are associations after adjustment, not causal effects.
Show data table
| Feature | Odds ratio |
|---|---|
| Payment method 34 | 0.26× |
| Payment method 41 | 0.28× |
| Plan: 30d | 0.38× |
| Payment method 33 | 0.42× |
| Payment method 40 | 2.35× |
| Auto-renew on | 0.44× |
| Cancelled before expiry | 2.07× |
| Sign-up channel unknown | 0.49× |
| Payment method 39 | 1.74× |
| Plan: 90d | 0.59× |
| Payment method 37 | 1.67× |
| Plan: 180d+ | 0.63× |
Caveats. Every model scores higher on the test months than on validation (0.944 against 0.874 for the best one). Q4 2016 includes December's churn spike, which is harder to predict, so treat the validation figure as the conservative estimate. Payment-method IDs are anonymised in the source data, so the model can say that some channels carry far more risk than others, but not which real-world channels they are.
Code: analysis/churn_model.py. Features come from fct_renewal_decisions and describe only what was known before the renewal month.