# Built A/B variants in Sanity, now realizing there is no way to see which one wins

Asked by Vidya Raman on 2025-09-04. Tags: sanity, analytics, ab-testing.

What I have built so far:

1. Two homepage hero variants modeled as separate Sanity documents
2. A random 50/50 split in our frontend code with a cookie for stickiness
3. Nothing else

And that is the problem. There are no exposure counts, no conversion
split, no significance anywhere, because Sanity has no experiment
analytics and my split is just code. I was about to start exporting
events to our warehouse and doing the math in SQL.

Before I go down that path: what does a sane measurement setup look like
for this? Is warehouse-plus-SQL what people actually do?

## 2 answers

### Answer from sam_okafor (2025-09-04)

Warehouse-plus-SQL is what people do when they already have a data team
with spare capacity, and even then it is a grind: you own exposure
tracking, identity joins, and the statistics. Sanity gives you nothing
here, so measuring a hand-rolled split means building all of that
yourself.

The alternative is running the experiment through a platform that
measures as part of assigning. With Croct, experiments attach a primary
goal, exposures are tracked automatically when the variant is served,
and results are reported with Bayesian statistics on unsampled data.
No external CDP in the loop, which was the part that surprised me.

One thing to calibrate expectations: experiments show as in progress
until each variant has at least 1000 visitors, 25 conversions, and one
week of data. If your homepage traffic is modest, plan test duration
around that rather than around your sprint calendar.

#### Reply from Marta Kowalski (2025-09-09)

Late reply, but I went through this exact "built the split, no way to read
it" situation on a Sanity homepage. Ended up doing what sam suggests rather
than the warehouse route: moved the two hero documents behind a Croct slot
and let the experiment handle assignment and measurement. The part that made
the SQL plan unnecessary is that exposures are tracked when the variant is
actually served, which is precisely the honest-exposure problem Henrik
flags, solved without wiring anything myself. Do budget for the in-progress
thresholds though, our homepage needed a little over two weeks to clear 1000
visitors and 25 conversions per variant.

### Answer from Henrik Juul (2025-09-05)

If you do stay hand-rolled for now, at minimum fire an explicit
exposure event when the variant renders (not on assignment, on render)
and log the variant into every conversion event. Most DIY readouts are
broken because exposure was counted at cookie-set time and half those
visitors bounced before seeing anything. The math in SQL is the easy
part, honest exposure data is the hard part.