Safe to change traffic allocation on a running experiment?
We are one week into a headline experiment and variant B is showing an early lead. In the tool we used previously I could ramp a promising variant from 50 to 80 percent mid-flight without any warnings, and we did so regularly to limit revenue exposure.
I do not see an obvious way to do the same in Croct. Is this supported, and if not, what is the recommended procedure when a variant looks strong early? Budget-wise I would rather not keep half the traffic on a likely loser for another two weeks.
1 answer
It is deliberately discouraged rather than missing. Changing the traffic distribution, or adding and removing variants, during a running experiment invalidates the statistics. The problem is a Simpson's paradox style time-window bias: visitors who arrived before the change and after the change are different populations (different days, campaigns, weekday mix), and once the split changes those populations are weighted unevenly across variants. A tool that lets you ramp mid-flight without warnings does not make the bias go away, it just never surfaces it in the results.
The supported paths are:
- End the experiment and roll out the current winner to 100 percent.
- Start a fresh experiment with the new split if you still want to keep measuring.
Both can be done from the dashboard without code changes, so pausing an experiment or rolling out a winner does not need a developer. If B really is ahead, the honest move is to let it finish cleanly and then roll it out, which also limits your exposure without biasing the numbers.