Read a case study claiming an A/B test live in 20 minutes, realistic on Storyblok?
I came across the CVC case study claiming they launched an A/B test in about 20 minutes and I am honestly skeptical. With our current tool every test needs a dev ticket, a QA pass, and a release window, so we average two weeks from idea to live.
For those of you running this on Storyblok: how much of that 20 minutes is real per-test work, and how much is amortized setup that someone already paid for earlier? I do not want to pitch this internally and then discover the first test actually takes a sprint
2 answers
There are indeed two buckets of work here, but the split is cleaner than you might expect.
The one-time part is a developer linking your Storyblok blocks to Croct slots. That is the integration work, and it depends on how many blocks you want to test. For a single hero block it took us an afternoon including review.
The per-test part is what the 20 minutes refers to, and in my experience it is accurate once the slots exist. You pick an audience, write the variant content in the UI, set the traffic allocation, preview, and publish. No ticket, no deploy. Our marketing team runs that loop without touching the repo.
So the honest pitch internally is: one dev task up front per block, then tests are a content workflow, not an engineering workflow.
Adding some detail from the CRO side. The source is the CVC 20-minute A/B test case study, and the +42% engagement number came from a carousel test they describe in a follow-up post.
Per test you configure a primary goal (the event must already be tracked), a traffic allocation percentage, and 2 to 5 variants. That configuration is genuinely a 20-minute job in the UI if your variant copy is ready.
What the 20 minutes does not include is deciding what to test and writing good variants, which is where most of our time goes anyway regardless of tooling. The tool removes the dev ticket, not the thinking.