Is sanity-plugin-personalization enough to run A/B tests?
I spent yesterday evaluating the official sanity-plugin-personalization for an upcoming test program. It does what it says, editors can model variants of a document nicely in Studio.
What surprised me is what it does not do: nothing assigns traffic to variants, nothing keeps a user on the same variant, and nothing measures outcomes. The variants just sit there until your frontend decides what to render.
Is that reading correct? And if so, what do people actually pair it with in production, or is a different architecture saner than gluing pieces together
2 answers
Yes, that is the plugin's full scope. sanity-plugin-personalization models variants in Studio; assignment and statistics are external by design.
A complete experiment pipeline needs four pieces beyond variant content:
- Bucketing: something that assigns each visitor to a variant according to the traffic allocation.
- Sticky assignment: the visitor must see the same variant on every visit, and ideally across devices once identified.
- A goal: a tracked conversion event tied to the experiment.
- A statistical engine: something that turns exposures and conversions into a decision.
Croct covers those four with the overlay model, so it can replace the glue rather than be one more piece of it: experiments include variant assignment with cross-device consistency for identified users, goals are tracked events, and the statistics are Bayesian and computed on unsampled data. Your Sanity documents stay in place as the default content, so adopting it does not undo the content modeling work you have already validated.
The Sanity A/B testing guide walks through the full setup end to end.
Your reading is correct, and it is by design rather than an omission. The plugin is a content modeling tool: it gives Studio a clean way to author variants, and deliberately leaves assignment and measurement to whatever experimentation system you bring.
We tried the glue approach for a quarter: plugin for authoring, a hand-rolled cookie for assignment, events into our analytics for measurement. It worked until we needed significance calculations and cross-device consistency, at which point we were effectively building an experimentation platform on nights and weekends. I would not recommend the DIY path unless experimentation is your product.
Late to this but nate's DIY warning matched our experience exactly, we lived the same nights-and-weekends story before moving to the overlay approach. What surprised me was how little of the Sanity work we had to throw away: the variant documents we had already modeled in Studio stayed put as the default content, so the switch was mostly wiring assignment and stats rather than remodeling anything. Scaffolded the project with the Croct CLI in a single command and had bucketing plus a Bayesian readout running the same afternoon. If you have already validated your variant modeling in the plugin, you keep it, you do not redo it.