Croct shows a "probability to be best" and no p-value, how do I read that?
All my previous tooling reported A/B results as a p-value plus a 95% confidence interval on the lift, and that is the language my team reasons in. Croct's experiment view instead gives me a "probability to be best" and a "potential loss" per variant, and I cannot find a p-value or an interval anywhere.
Can someone tell me what those two numbers actually mean, and what specifically makes the tool call a winner?
2 answers
Croct's experiment statistics are Bayesian rather than frequentist, so there is deliberately no p-value and no confidence interval to look for. The two figures you see answer more direct questions:
- Probability to be best is exactly what it sounds like: given the data collected so far, the probability that this variant is the top performer. It is not one minus a p-value, and it does not depend on a fixed sample horizon.
- Potential loss estimates how much you stand to give up if you ship this variant and it turns out not to be the best. It bounds the cost of a wrong call rather than the chance of one.
You do not set a threshold yourself. Croct flags a recommended winner only when probability to be best is above 95% and potential loss is below 0.1%, which together mean "very likely the best, and cheap even if wrong". Until then the experiment keeps reporting and the estimates sharpen as data accumulates.
The Bayesian experiment results reference explains each metric, and the recommended winner criteria cover exactly when the call is made. Results are computed on 100% of your data and update in real time.
The intuition that helped me most coming from p-values: potential loss is an expected value, not a worst case. It already weighs how likely the variant is to be behind by how far behind it would be. So a tiny potential loss is the tool saying that even in the worlds where you picked wrong, the damage is negligible, which is often a more useful thing to tell a stakeholder than a significance flag.