Can I bump the traffic allocation partway through a Hydrogen experiment?
I started an experiment on a product page on our Hydrogen storefront at a conservative traffic share because I wanted to limit exposure early. It is behaving well and I would now like to ramp the allocation up to gather data faster.
Is changing the traffic allocation mid-flight safe, or does it compromise the validity of the results? I would rather leave it alone than end up with numbers I cannot trust.
2 answers
Leave it alone. The traffic allocation is a percentage you set on the experiment, say 30% of visitors see the variants across your 2 to 5 variants, and changing that distribution mid-flight is strongly discouraged because it invalidates the statistics. Visitors before and after the change are different populations, and once the split moves those populations get weighted unevenly across variants, which biases the result. Adding or removing a variant during the run has the same problem.
For context on when it is even allowed to end: an experiment stays in progress until each variant has at least 1000 visitors, at least 25 conversions, and has run at least one week. So a conservative allocation mostly just means it reaches those thresholds a little later, not that the run is compromised.
If gathering data faster is the real goal, the clean path is to restart with the allocation you actually want rather than ramp the current one. Plan the split up front on the fresh test and let it run untouched to the thresholds. You lose the early days of data, but you get a result you can defend, which is cheaper than finishing a biased run and not being able to use it.
That settles it. I will stop the current one and relaunch at the higher allocation from the start rather than touch a running test. Thanks for confirming the reasoning, not just the rule.