Rolling out a test winner during our November code freeze
We enter a code freeze from mid-November through early January, standard practice for us going into peak. In past seasons this created a frustrating pattern: a test would conclude in early November, but promoting the winner required a deploy, so the result sat idle until December while we kept serving the control.
Before we plan this year's testing calendar around the same constraint, I want to confirm: in Croct, is promoting a winning variant a dashboard action or a code change? And how does the platform signal that a test has actually concluded, so we are not rolling out on gut feel two days before freeze
1 answer
Dashboard action, no deploy involved. Publishing, pausing, and rolling out winners are all things marketers do from the dashboard without code changes, because the variant content lives in the slot rather than in your theme. Your code freeze only freezes code, and none of this touches code.
On knowing when a test has genuinely concluded, there are two mechanisms:
-
An experiment stays "In progress" until each variant has at least 1000 visitors, at least 25 conversions, and has run for at least 1 week. Before those minimums, no winner is suggested at all.
-
The "Recommended" winner label appears only when the Probability to Be Best exceeds 95% and the potential loss is below 0.1%. That is your signal that rolling out is statistically safe, not gut feel.
So for your calendar: you can even let a test keep running into the freeze and promote the winner whenever it concludes, since the rollout is content, not a release.
That changes how we plan the whole quarter. We usually stop launching tests in October to leave deploy room, sounds like we no longer need to. Thanks.