Can I bump a variant from 30% to 50% of traffic while the test is already running?
variant B is looking strong quite early and I want to reach significance faster by sending it more traffic. Currently it is a 30/70 split and I am tempted to push it to 50/50 mid-flight.
Is that safe to do while the test is live, or does changing the split mess up the results? I would rather move fast here but not if it invalidates the whole thing.
2 answers
It is strongly discouraged, and for a real reason rather than a policy one. Changing the traffic distribution mid-flight, or adding and removing variants once a test is running, invalidates the statistics. The sample you collected before the change and the sample after it are drawn from different populations (different days, campaigns, weekday mix), so once the split shifts those two groups get weighted unevenly across your variants and the comparison is no longer apples to apples.
Traffic allocation and the set of variants, anywhere from 2 to 5, are meant to be decided at the start and left alone. The Bayesian model assumes the assignment stayed stable, so a mid-test bump quietly breaks that assumption even though the tool will not stop you.
If you really want the 50/50, the clean path is to end this one and start a fresh experiment with the new allocation from the outset. You lose the days already banked, but you get numbers you can actually trust. Otherwise just let the current split run to the thresholds. B being ahead early does not mean it wins, and rushing the split is the fastest way to fool yourself.
fair enough. I will let it ride to the thresholds rather than restart and lose the week. thanks both