How do I read the experiment results for my Sanity hero test?
Running my first hero test on our Sanity site. Five days in, dashboard still says "in progress". Two questions:
- What exactly unlocks a recommendation? Is it time, sample size, or both?
- How should I read the Bayesian numbers? I am used to p-values from a previous tool and probability to be best is new to me.
Traffic is decent, about 4k visitors per variant so far
2 answers
Both time and sample size gate the recommendation. An experiment stays "in progress" until each variant has at least 1000 visitors, at least 25 conversions, and at least one week has passed. With 4k visitors per variant at day five, you have likely met the volume thresholds and are waiting out the one-week minimum, which exists to cover weekday and weekend behavior at least once.
A "recommended winner" appears when two conditions hold simultaneously:
- Probability to Be Best exceeds 95%.
- Potential loss is below 0.1%.
On reading the numbers: Probability to Be Best is more direct than a p-value. It answers "given the data so far, what is the probability this variant is the best one?", so 92% means exactly that. Potential loss estimates how much conversion you would give up if you shipped this variant and it turned out not to be the best, which is why both must pass before a recommendation.
The statistics are Bayesian and computed on unsampled data, meaning 100% of your traffic, not an extrapolated sample. You can also drill down by device, audience, time window, and custom events to see whether a variant wins overall but loses for a specific segment.
One habit worth building from the p-value world you are leaving behind: resist calling it early when Probability to Be Best crosses 90% on day six. It can still swing, and the whole point of the potential loss guardrail is to keep you from shipping a coin flip. Let the recommendation appear, then act.
The drill-downs are also where the interesting findings hide. Our last hero test won overall but lost on mobile, and we only caught it in the device breakdown.