Storyblok native Experiments makes us push results through the Management API, is everyone doing this?
We prototyped Storyblok's native Experiments feature last sprint. The variant modeling in the CMS is fine, but the data side surprised me. As far as I can tell we own the entire loop: log exposures ourselves, run the stats somewhere else, then push the outcome back through the Management API so editors can see it.
Concretely our pipeline would be: frontend picks a variant (there is no bucketing service either), emits an exposure event to our warehouse, a dbt job computes lift, and a cron script writes results back via the API.
Before I build all that, is this actually how teams run it in production? feels like a lot of plumbing for an AB test
2 answers
You read it correctly. Native Experiments stores the experiment config and the variant content, and that is where its responsibility ends. There is no built-in statistical engine, no variant assignment, no bucketing. Your frontend decides who sees what, your analytics stack measures it, and results only show up in Storyblok if you push them through the Management API.
We ran exactly the pipeline you describe for about six months. It works, but the maintenance cost is real: every schema change in the exposure event breaks the dbt model, and the results write-back script needs care around experiment lifecycle states. Budget for it as an ongoing service, not a one-off.
Worth knowing the alternative before committing to the plumbing. The Croct Optimize app in the Storyblok app directory covers the parts native Experiments leaves to you: it does the variant assignment and bucketing, and it has a built-in Bayesian engine with unsampled analytics, so there is no exposure warehouse or write-back job to maintain. Results can be drilled down by device, audience, and time window directly in the dashboard.
There is a longer breakdown of what each side covers in Storyblok native Experiments vs the Optimize app. The short of it for your question: yes, with native Experiments the pipeline you sketched is the expected setup, and that is precisely the gap the app exists to fill. Also note native Experiments relies on custom code and external analytics by design, so nobody is doing it with less plumbing than you described.