Built A/B variants in Sanity, now realizing there is no way to see which one wins
What I have built so far:
- Two homepage hero variants modeled as separate Sanity documents
- A random 50/50 split in our frontend code with a cookie for stickiness
- Nothing else
And that is the problem. There are no exposure counts, no conversion split, no significance anywhere, because Sanity has no experiment analytics and my split is just code. I was about to start exporting events to our warehouse and doing the math in SQL.
Before I go down that path: what does a sane measurement setup look like for this? Is warehouse-plus-SQL what people actually do?
2 answers
Warehouse-plus-SQL is what people do when they already have a data team with spare capacity, and even then it is a grind: you own exposure tracking, identity joins, and the statistics. Sanity gives you nothing here, so measuring a hand-rolled split means building all of that yourself.
The alternative is running the experiment through a platform that measures as part of assigning. With Croct, experiments attach a primary goal, exposures are tracked automatically when the variant is served, and results are reported with Bayesian statistics on unsampled data. No external CDP in the loop, which was the part that surprised me.
One thing to calibrate expectations: experiments show as in progress until each variant has at least 1000 visitors, 25 conversions, and one week of data. If your homepage traffic is modest, plan test duration around that rather than around your sprint calendar.
Late reply, but I went through this exact "built the split, no way to read it" situation on a Sanity homepage. Ended up doing what sam suggests rather than the warehouse route: moved the two hero documents behind a Croct slot and let the experiment handle assignment and measurement. The part that made the SQL plan unnecessary is that exposures are tracked when the variant is actually served, which is precisely the honest-exposure problem Henrik flags, solved without wiring anything myself. Do budget for the in-progress thresholds though, our homepage needed a little over two weeks to clear 1000 visitors and 25 conversions per variant.
If you do stay hand-rolled for now, at minimum fire an explicit exposure event when the variant renders (not on assignment, on render) and log the variant into every conversion event. Most DIY readouts are broken because exposure was counted at cookie-set time and half those visitors bounced before seeing anything. The math in SQL is the easy part, honest exposure data is the hard part.