Should I start an experiment at 10% traffic and ramp it up over time?
Coming from a feature-flag background I am used to rolling changes out slowly, 10% then 25% then 50% then everyone, so if something goes wrong the blast radius is small. I would like to do the same with a Croct experiment, starting the variant at a low share of traffic and ramping it up as confidence grows.
Is ramping the traffic allocation during a running experiment a good idea here or is there a better way to get that gradual exposure without wrecking my numbers?
2 answers
Ramping the allocation mid-run is the one thing to avoid here, because it mixes two different jobs that Croct keeps separate: measuring a change and staging a rollout.
Traffic allocation on an experiment sets what share of visitors enter the test at all, for example 30% see the variants while the rest see the default. That share is meant to be set once and left fixed for the run. Changing the traffic distribution while an experiment is live is strongly discouraged because it invalidates the statistics: visitors from before and after the change are different populations, and reweighting them across variants biases the result.
If your goal is limited exposure, use the allocation to hold most traffic out from the start, say 10% into the experiment, and then keep it there for the whole run. That gives you the small blast radius without touching the split. If instead your goal is a genuine staged release of a finished change, that is a feature flag job, not an experiment. See traffic allocation for how the share works and feature flags for the staged-rollout tooling.
The way we operationalize this: decide the allocation up front based on how much risk we are willing to expose, write it down, and treat it as locked for the duration. If partway through we decide the exposure was wrong, we do not nudge the live one, we end it and start fresh with the new split. Annoying once, but it keeps every result we ever report clean, and it stopped the "just bump it to 50" arguments that used to happen with our old tool.