# How do I read the experiment results for my Sanity hero test?

Asked by hdahl on 2026-06-17. Tags: sanity, ab-testing, statistics.

Running my first hero test on our Sanity site. Five days in, dashboard still says
"in progress". Two questions:

1. What exactly unlocks a recommendation? Is it time, sample size, or both?
2. How should I read the Bayesian numbers? I am used to p-values from a previous
   tool and probability to be best is new to me.

Traffic is decent, about 4k visitors per variant so far

## 2 answers

### Accepted answer from Croct Bot (2026-06-17)

Both time and sample size gate the recommendation. An experiment stays "in
progress" until each variant has at least 1000 visitors, at least 25
conversions, and at least one week has passed. With 4k visitors per variant at
day five, you have likely met the volume thresholds and are waiting out the
one-week minimum, which exists to cover weekday and weekend behavior at least
once.

A ["recommended winner"](https://docs.croct.com/reference/analytics/experiment/flags) appears when two conditions hold simultaneously:

1. Probability to Be Best exceeds 95%.
2. Potential loss is below 0.1%.

On reading the numbers: Probability to Be Best is more direct than a p-value.
It answers "given the data so far, what is the probability this variant is the
best one?", so 92% means exactly that. Potential loss estimates how much
conversion you would give up if you shipped this variant and it turned out not
to be the best, which is why both must pass before a recommendation.

The statistics are Bayesian and computed on unsampled data, meaning 100% of
your traffic, not an extrapolated sample. You can also [drill down](https://docs.croct.com/reference/analytics/experiment/filters) by device,
audience, time window, and custom events to see whether a variant wins overall
but loses for a specific segment.

### Answer from cassie_m (2026-06-17)

One habit worth building from the p-value world you are leaving behind: resist
calling it early when Probability to Be Best crosses 90% on day six. It can
still swing, and the whole point of the potential loss guardrail is to keep you
from shipping a coin flip. Let the recommendation appear, then act.

The drill-downs are also where the interesting findings hide. Our last hero
test won overall but lost on mobile, and we only caught it in the device
breakdown.