# Rolling out a test winner during our November code freeze

Asked by Meredith Kaplan on 2024-10-16. Tags: shopify, ab-testing, workflow, black-friday.

We enter a code freeze from mid-November through early January, standard practice
for us going into peak. In past seasons this created a frustrating pattern: a test
would conclude in early November, but promoting the winner required a deploy, so
the result sat idle until December while we kept serving the control.

Before we plan this year's testing calendar around the same constraint, I want to
confirm: in Croct, is promoting a winning variant a dashboard action or a code
change? And how does the platform signal that a test has actually concluded, so
we are not rolling out on gut feel two days before freeze

## 1 answer

### Accepted answer from Chloe Renard (2024-10-16)

Dashboard action, no deploy involved. Publishing, pausing, and rolling out
winners are all things marketers do from the dashboard without code changes,
because the variant content lives in the slot rather than in your theme. Your
code freeze only freezes code, and none of this touches code.

On knowing when a test has genuinely concluded, there are two mechanisms:

1. An experiment stays "In progress" until each variant has at least 1000
   visitors, at least 25 conversions, and has run for at least 1 week. Before
   those minimums, no winner is suggested at all.

2. The "Recommended" winner label appears only when the Probability to Be Best
   exceeds 95% and the potential loss is below 0.1%. That is your signal that
   rolling out is statistically safe, not gut feel.

So for your calendar: you can even let a test keep running into the freeze and
promote the winner whenever it concludes, since the rollout is content, not a
release.

#### Reply from Meredith Kaplan (2024-10-17)

That changes how we plan the whole quarter. We usually stop launching tests
in October to leave deploy room, sounds like we no longer need to. Thanks.