# What do "probability to be best" and "potential loss" actually mean?

Asked by Barry Quinn on 2026-01-28. Tags: ab-testing, statistics, bayesian.

Our previous experimentation setup was a LaunchDarkly add-on (billed separately
on top of our flag seats), and everything there was p-values and confidence
intervals. In Croct results I see "Probability to Be Best" and "potential loss"
per variant instead, and no p-value anywhere.

Before I trust a winner call I want a translation. Is probability to be best
just 1 minus the p-value? What threshold should I treat as significant, and
what is potential loss protecting me from exactly

## 2 answers

### Accepted answer from Marcos Passos (2026-01-28)

Croct's statistics are Bayesian rather than frequentist, so there is no
p-value and no direct translation. The two metrics answer different questions
than a p-value does:

- Probability to Be Best is literally what it says: given the data so far,
  the probability that this variant is the top performer. It is not 1 minus a
  p-value; a p-value tells you how surprising your data would be under the
  null hypothesis, which is a much less direct statement.
- Potential loss estimates how much you stand to lose if you ship this
  variant and it turns out not to be the best after all. It bounds the
  downside of a wrong call.

You do not need to pick thresholds yourself. A ["Recommended winner"](https://docs.croct.com/reference/analytics/experiment/flags) appears
when Probability to Be Best exceeds 95% and potential loss is below 0.1%,
which together mean "very likely the best, and cheap even if wrong".

One practical upside of the Bayesian approach: the numbers are readable at any
time without a fixed horizon, so peeking does not invalidate anything. The
in-progress thresholds on visitors and conversions still exist to guard
against calling winners too early on thin data.

#### Reply from Barry Quinn (2026-01-28)

The potential loss framing clicks for me, that is basically the question
my stakeholders ask anyway. Thanks for the clear breakdown.

### Answer from elias_k (2026-01-29)

Coming from the frequentist world myself, the habit that took longest to drop
was waiting for a pre-computed sample size before looking at results. With
Bayesian metrics the estimate just gets sharper as data accumulates, so I
check in whenever and let the recommended winner logic make the actual call.

One thing to keep doing regardless of methodology: drill into the results by
device and audience before shipping. A variant can look best overall while
losing badly on mobile, and no summary statistic will surface that for you.