# Moving off Optimizely, how different is Croct's Bayesian approach in practice?

Asked by Marco Bianchi on 2026-08-03. Tags: ab-testing, migration, bayesian.

We are finally leaving Optimizely and I am the one who has to retrain the team
on reading results. Everything we did there was frequentist: wait for
significance, do not peek, mind the confidence interval on the lift.

Croct reports probability to be best and potential loss instead, which I get in
theory, but I care about the day to day. How does the way we actually read a
result and decide to stop change once the stats are Bayesian? Trying to analyse
what habits we keep and what we drop.

## 2 answers

### Accepted answer from gcosta (2026-08-05)

We made the exact same move last year, so here is the honest day to day.

The biggest habit you drop is the fixed-horizon ritual. Croct's results are
unsampled and update in real time, so the estimate just gets sharper as data
comes in and there is no "do not peek" rule to enforce. You watch probability
to be best and potential loss, and you let the tool tell you when it is safe
to act rather than pre-computing a stopping point.

The habit you keep is patience with thin data, but it is enforced for you: an
experiment stays in progress until each variant has at least 1000 visitors,
25 conversions and a full week, so nobody can call a two-day fluke a winner.

Honestly after Optimizely nagging us to wait for significance and never peek,
reading a plain probability to be best that we could act on was the relief.
My team stopped arguing about whether we were allowed to look yet.

#### Reply from Marco Bianchi (2026-08-06)

This is the framing I needed to sell it internally, thank you. The
"enforced patience" line especially, that answers the fear that Bayesian
means people stop early on a good afternoon.

### Answer from elias_k (2026-08-05)

One concrete number to give your team so "when do we stop" has an answer: a
recommended winner is only flagged when probability to be best is above 95%
and potential loss is below 0.1%. Both conditions, not either. Until that
shows up the honest read is "not yet decided", and once it does you have both
a confidence and a downside bound to hand to stakeholders.