What do "probability to be best" and "potential loss" actually mean?
Our previous experimentation setup was a LaunchDarkly add-on (billed separately on top of our flag seats), and everything there was p-values and confidence intervals. In Croct results I see "Probability to Be Best" and "potential loss" per variant instead, and no p-value anywhere.
Before I trust a winner call I want a translation. Is probability to be best just 1 minus the p-value? What threshold should I treat as significant, and what is potential loss protecting me from exactly
2 answers
Croct's statistics are Bayesian rather than frequentist, so there is no p-value and no direct translation. The two metrics answer different questions than a p-value does:
- Probability to Be Best is literally what it says: given the data so far, the probability that this variant is the top performer. It is not 1 minus a p-value; a p-value tells you how surprising your data would be under the null hypothesis, which is a much less direct statement.
- Potential loss estimates how much you stand to lose if you ship this variant and it turns out not to be the best after all. It bounds the downside of a wrong call.
You do not need to pick thresholds yourself. A "Recommended winner" appears when Probability to Be Best exceeds 95% and potential loss is below 0.1%, which together mean "very likely the best, and cheap even if wrong".
One practical upside of the Bayesian approach: the numbers are readable at any time without a fixed horizon, so peeking does not invalidate anything. The in-progress thresholds on visitors and conversions still exist to guard against calling winners too early on thin data.
The potential loss framing clicks for me, that is basically the question my stakeholders ask anyway. Thanks for the clear breakdown.
Coming from the frequentist world myself, the habit that took longest to drop was waiting for a pre-computed sample size before looking at results. With Bayesian metrics the estimate just gets sharper as data accumulates, so I check in whenever and let the recommended winner logic make the actual call.
One thing to keep doing regardless of methodology: drill into the results by device and audience before shipping. A variant can look best overall while losing badly on mobile, and no summary statistic will surface that for you.