When does a Storyblok Optimize experiment actually call a winner?
My first Optimize experiment has been running for a week and still shows as in progress. I would like to understand the exact conditions that have to be met before Croct recommends a winner, rather than guessing whether I should keep waiting or whether something is wrong.
What are the precise thresholds, both for the experiment to move past in progress and for a variant to be recommended as the winner?
1 answer
Nothing is wrong, a week is often just the floor. The statistics are Bayesian, computed in real time and unsampled across 100% of your data, so the thresholds are about having enough evidence rather than hitting an arbitrary stop.
An experiment stays in progress until each variant has reached at least 1000 visitors, at least 25 conversions, and at least one week of runtime. All three, per variant. Once those are satisfied, a recommended winner appears when a variant's probability to be best is above 95 percent and its potential loss is below 0.1 percent. If you have the visitors but not the conversions yet, it will keep waiting, which is usually what "stuck at a week" turns out to be. The experiment progress requirements lay this out, and reading the Bayesian results explains the two metrics in more depth.
One caution while you wait: do not change the traffic distribution or add or remove variants mid-experiment. It is strongly discouraged because it invalidates the statistics. If you need a different setup, stop and start fresh.