Is it valid to slice a finished experiment by audience to find a winner?
My headline experiment came out flat overall, no recommended winner, the two variants basically tied. But I have a hunch mobile visitors reacted differently from desktop, and I can drill the results down by device and audience.
If I slice the finished experiment and find that variant B clearly wins on mobile, is it statistically sound to act on that, or am I fooling myself by looking after the fact?
2 answers
You are right to be suspicious of yourself here, this is the classic post-hoc slicing trap and the drill-down makes it easy to fall into.
Two things are working against a slice you did not declare in advance. First, each slice is a smaller sample, so your mobile-only cut very likely does not meet the 1000 visitors and 25 conversions guardrails that make a call trustworthy in the first place. Second, if you carve the data enough ways, some cut will look like a winner by chance alone, and "mobile" is just the cut that happened to catch your eye.
The sound way to use that mobile result is as a hypothesis, not a decision. You noticed a signal, now confirm it deliberately. Treat it exactly the way you would treat any promising-but-unproven lead.
Concretely, the follow-up is to re-run the test scoped to the audience you suspect. Pre-declare a mobile (or whatever) target audience on the experience so the experiment only enrolls those visitors, and let it collect its own 1000 visitors and 25 conversions and full week. If B really wins on mobile it will earn a recommended winner on its own data, and now it is a result you can defend rather than a slice you went fishing for.
Agreed, that is the honest path. I will log the mobile observation as a hypothesis and set up a scoped re-run rather than acting on the slice. Better to spend the extra weeks than to ship a chance artifact.