Can I check Bayesian results daily and stop early without wrecking them?
With frequentist tests I learned the hard way that peeking every day and stopping on the first good result inflates false positives, so we trained ourselves not to look until the planned end.
Croct's results are Bayesian and update in real time, which tempts me to watch them daily again. Does the same peeking trap apply here, and is it actually safe to monitor an experiment every day and stop when it looks decided?
1 answer
You can watch daily, and monitoring does not invalidate anything. The peeking penalty you are describing is specific to frequentist testing, where the p-value is calibrated against a fixed sample size and every extra look is an extra chance to cross the line by luck. Croct's results are Bayesian and unsampled, so the numbers are valid statements about the data at the moment you read them, no matter how often you read them. Looking does not consume anything. The Bayesian methodology goes into why.
Where the discipline still lives is in the stopping rule, and Croct handles that for you. You should not stop just because a variant looks ahead on a given day. A recommended winner is only surfaced when probability to be best is above 95% and potential loss is below 0.1%, and even that is gated behind the in-progress guardrails: each variant needs at least 1000 visitors, 25 conversions and a full week first. Those recommended winner criteria are what protects you from an early call, not a rule against looking.
One real caveat: do not touch the experiment while it runs. Changing the traffic distribution or adding and removing variants mid-flight genuinely does invalidate the statistics, so keep the setup fixed and only your eyes moving.