How long should I let an experiment run before trusting the numbers?
this is my first experiment and after two days variant B is already ahead and it looks really promising. part of me wants to just call it and ship, but i do not want to embarrass myself if teh early lead is noise.
what actually decides when a result is trustworthy on Croct? is there a duration i should wait for, or is it about volume?
1 answer
Two days is too early to trust, and Croct will not let the experiment call a winner yet even if the lead looks large. It is both volume and duration, and the tool gates on both.
An experiment stays "in progress" until each variant independently reaches at least 1000 visitors, at least 25 conversions, and at least one full week of runtime. All three, for every variant. The one-week minimum is not about sample size, it exists so the test spans a complete weekday and weekend cycle, since behavior on a Saturday often differs sharply from a Tuesday and a two-day slice can be badly skewed.
Once those guardrails clear, a recommended winner is only flagged when probability to be best is above 95% and potential loss is below 0.1%. The statistics are Bayesian and unsampled, computed on 100% of your data and updated in real time, so you can watch the estimate sharpen without harming it, you just should not act until both the guardrails and the winner criteria are met.
See the in-progress criteria for the thresholds and the recommended winner rules for when the call is made.