How do we measure A/B test results on a Hydrogen store?
Our first Croct experiment on Hydrogen went live yesterday, a two-variant test on the homepage hero with checkout starts as the goal. I read the Croct guide on Hydrogen personalization and A/B testing during evaluation, but it focuses on setup, and my questions now are about reading results.
Where do results show up, what statistical engine is running underneath, and most importantly, what are the objective criteria for calling a winner? I report to a board that will ask exactly that question and "the dashboard says so" wont fly
1 answer
Results appear on the experiment's page in the Croct dashboard, updating in real time as visitors convert.
On the engine: the statistics are Bayesian, not frequentist, and the analytics are unsampled, meaning results are computed from 100% of your data rather than a sample. For board reporting, the Bayesian output reads as "Probability to Be Best" per variant instead of a p-value.
The objective criteria you asked for:
- An experiment stays "In progress" until each variant has at least 1000 visitors, at least 25 conversions, and the experiment has run for at least one week. Before those thresholds, no winner is suggested regardless of how the numbers look.
- A "Recommended winner" appears when a variant's Probability to Be Best exceeds 95% and its potential loss is below 0.1%.
Those two rules give you a defensible, precommitted stopping criterion to cite.
For the analysis itself, results drill down by device, audience, time window, and custom events, so you can verify the lift holds on mobile or within a specific audience before rolling the winner out.