Ran an A/B test on a Sanity hero, how long before Croct calls a winner?
first experiment here. I set up an A/B test on our Sanity hero component and it has been running a few days, but it still shows as in progress and I am getting a little impatient for a call.
What are the actual thresholds Croct waits for before it recommends a winner? I would like the concrete numbers so I know whether I am looking at days or weeks, and whether I have enough traffic to get there at all.
2 answers
There are hard gates before it will even consider a result. An experiment stays "in progress" until every variant has at least 1000 visitors, at least 25 conversions, and it has run for at least 1 week. All three, per variant.
Once those are met, a recommended winner appears when the probability to be best goes above 95 percent and the potential loss drops below 0.1 percent. The stats are Bayesian and unsampled, so it is looking at 100 percent of your data rather than a sample.
So if it is only a few days in, the one-week floor alone means it cannot call it yet regardless of how good the numbers look. Check where each variant sits against the visitor and conversion counts and you will know if traffic is the constraint.
that explains the wait, we are past the visitor count but not the week yet. will leave it alone. thanks
One warning while you wait: do not touch the traffic allocation or add or remove variants mid-test. Changing the split or the variant set during a running experiment is strongly discouraged because it invalidates the statistics, and then the clock effectively resets on trusting the result. If a variant is clearly bad, pause the whole experiment or let it finish and start a fresh one rather than editing it live.