Does 90% probability to be best mean the same as 90% confidence?
I support a marketing team that is used to seeing "significant at 95% confidence" in every results deck. Before Croct I would run their raw numbers through an online significance calculator and report the p-value.
Now the dashboard shows Probability to Be Best, and I want to be careful with terms before I present anything. Is a 90% Probability to Be Best equivalent to 90% confidence in the frequentist sense, or are these different quantities that happen to both be percentages? My stakeholders will absolutely conflate them if I am not precise
1 answer
They are different quantities, so keep them separate in the deck. Croct's statistics are Bayesian. Probability to Be Best is the direct probability that a variant is the top performer given the data observed so far. A frequentist confidence level is a statement about the procedure, not about the variant, and it answers a two-sided question by default.
In practice, Probability to Be Best is closer to the one-tailed question stakeholders actually ask, which is "how likely is it that B beats A". That makes it easier to present honestly: "there is a 90% probability this variant is the best" is a literally true reading of the number, which you can never say about a confidence level.
Two things to include in your deck framing:
- The recommended-winner rule combines two checks: Probability to Be Best above 95% AND potential loss below 0.1%. A high PBB alone does not trigger a recommendation, because a variant can be probably-best while the downside of being wrong is still material.
- Bayesian results are readable at any time without a fixed horizon, but the in-progress thresholds (1000 visitors, 25 conversions, and 1 week per variant) still guard against early calls.
There is more depth on the metric itself in this thread on how Probability to Be Best is computed.