Starting my first experiment on a Strapi hero, what traffic split should I begin with?
I wired our Strapi hero block to a slot and built two variants, so now I have to actually set the experiment up. I am not sure whether to send all my traffic into the test right away or ramp it slowly, and whether I can adjust it later once it is running.
Thanks in advance.
1 answer
When you set the experiment up you pick three things: a primary goal, which has to be an event you are already tracking, a traffic allocation percentage, and your 2 to 5 variants. The traffic allocation is the share of visitors who enter the test at all, so 30 percent means roughly a third see the variants and the rest stay on your control.
On where to start: pick the allocation you are comfortable committing to and leave it there. The one thing to avoid is changing the traffic distribution, or adding and removing variants, once the experiment is running, because that invalidates the statistics. So it is less "ramp it" and more "choose a share and let it run". Results stay in progress until each variant reaches at least 1000 visitors, 25 conversions, and one week, and the statistics are Bayesian and computed on 100 percent of your data, with no sampling.
Setting up an experiment walks through the goal and allocation, and the winner thresholds cover when results are considered ready.
for anyone else setting their first one up, take the do-not-change-it-midway bit seriously. i bumped the allocation up two days in because a variant looked good early and the numbers went useless, had to restart clean. there is a whole thread on why changing the split mid-run breaks the stats. pick the share and walk away.