# Is it valid to slice a finished experiment by audience to find a winner?

Asked by Renata Alves on 2026-10-29. Tags: ab-testing, segmentation, statistics.

My headline experiment came out flat overall, no recommended winner, the two
variants basically tied. But I have a hunch mobile visitors reacted differently
from desktop, and I can drill the results down by device and audience.

If I slice the finished experiment and find that variant B clearly wins on
mobile, is it statistically sound to act on that, or am I fooling myself by
looking after the fact?

## 2 answers

### Accepted answer from elias_k (2026-10-30)

You are right to be suspicious of yourself here, this is the classic post-hoc
slicing trap and the drill-down makes it easy to fall into.

Two things are working against a slice you did not declare in advance. First,
each slice is a smaller sample, so your mobile-only cut very likely does not
meet the 1000 visitors and 25 conversions guardrails that make a call
trustworthy in the first place. Second, if you carve the data enough ways,
some cut will look like a winner by chance alone, and "mobile" is just the cut
that happened to catch your eye.

The sound way to use that mobile result is as a hypothesis, not a decision.
You noticed a signal, now confirm it deliberately. Treat it exactly the way
you would treat any promising-but-unproven lead.

### Answer from Owen Tran (2026-10-30)

Concretely, the follow-up is to re-run the test scoped to the audience you
suspect. Pre-declare a mobile (or whatever) target audience on the experience
so the experiment only enrolls those visitors, and let it collect its own
1000 visitors and 25 conversions and full week. If B really wins on mobile it
will earn a recommended winner on its own data, and now it is a result you can
defend rather than a slice you went fishing for.

#### Reply from Renata Alves (2026-10-31)

Agreed, that is the honest path. I will log the mobile observation as a
hypothesis and set up a scoped re-run rather than acting on the slice.
Better to spend the extra weeks than to ship a chance artifact.