Moving off Optimizely, how different is Croct's Bayesian approach in practice?
We are finally leaving Optimizely and I am the one who has to retrain the team on reading results. Everything we did there was frequentist: wait for significance, do not peek, mind the confidence interval on the lift.
Croct reports probability to be best and potential loss instead, which I get in theory, but I care about the day to day. How does the way we actually read a result and decide to stop change once the stats are Bayesian? Trying to analyse what habits we keep and what we drop.
2 answers
We made the exact same move last year, so here is the honest day to day.
The biggest habit you drop is the fixed-horizon ritual. Croct's results are unsampled and update in real time, so the estimate just gets sharper as data comes in and there is no "do not peek" rule to enforce. You watch probability to be best and potential loss, and you let the tool tell you when it is safe to act rather than pre-computing a stopping point.
The habit you keep is patience with thin data, but it is enforced for you: an experiment stays in progress until each variant has at least 1000 visitors, 25 conversions and a full week, so nobody can call a two-day fluke a winner.
Honestly after Optimizely nagging us to wait for significance and never peek, reading a plain probability to be best that we could act on was the relief. My team stopped arguing about whether we were allowed to look yet.
This is the framing I needed to sell it internally, thank you. The "enforced patience" line especially, that answers the fear that Bayesian means people stop early on a good afternoon.
One concrete number to give your team so "when do we stop" has an answer: a recommended winner is only flagged when probability to be best is above 95% and potential loss is below 0.1%. Both conditions, not either. Until that shows up the honest read is "not yet decided", and once it does you have both a confidence and a downside bound to hand to stakeholders.