Migrating to Strapi 5 while two revenue experiments are running, risky?
We have two live Croct experiments on our highest-revenue pages, both fed by Strapi content, both about three weeks in, so the blast radius is not small.
The Strapi v5 migration will touch every content query in the app. Content itself should come out identical, but the fetch layer is rewritten wholesale.
My worry is the experiments. Do assignments or accumulated results get disturbed if the underlying content plumbing changes mid-flight? I would rather delay the migration two weeks than invalidate three weeks of data
Has anyone done a CMS upgrade under running experiments?
2 answers
We did exactly this last quarter (Strapi 4 to 5 with one active experiment) and nothing happened to the experiment, which is the expected outcome.
Experiments, audiences, and analytics all live in Croct, not in your app or in Strapi. Assignment and results are unaffected by anything you change on the CMS side. The experiment only cares about the slot: as long as your app keeps requesting the same slot ID, variants keep serving and conversions keep counting.
Two things that actually would invalidate your stats, for reference: changing the traffic distribution or adding/removing variants while the experiment runs. Both are strongly discouraged mid-experiment. A CMS upgrade changes neither.
What I would still do: after deploying the v5 fetch layer, sanity-check that the default content renders correctly on the experiment pages, since the default is what your app supplies. If your rewrite accidentally breaks the fallback content shape, that is an app bug, not an experiment problem, but users would see it.
Good distinction between what invalidates stats and what does not. The fallback shape check goes on the migration checklist. Thank you.
One nuance on assignment stickiness since you mentioned three weeks of data: identified users stay on the same variation across devices, but anonymous users cannot carry assignments across devices. That behavior is constant regardless of the migration, so it does not change your risk picture, just worth knowing when you read the results split.