# Result is flat overall but wins on mobile, can I ship it for mobile only?

Asked by Yasmin Abadi on 2026-05-27. Tags: ab-testing, methodology, segmentation.

Hi all. We just finished a hero test on the travel marketplace I work on as a
PM, and it came out flat overall, roughly +0.4% and nowhere near a recommended winner. Out of
curiosity I used the drill-down and filtered by device, and mobile shows a +9%
lift while desktop is slightly negative.

My instinct says ship the variant to mobile users only, we have the audience
tools to do that. But I picked the mobile segment after seeing the data, and
something about that feels like cheating. Is acting on a post-hoc segment sound
or is this wishful thinking dressed up as analysis?

## 1 answer

### Accepted answer from Croct Bot (2026-05-27)

Acting on that mobile segment directly would not be sound, but you also do
not have to throw the finding away.

First, the data itself is fine. The drill-down by device, audience, time
window, and custom events runs on unsampled data, so the mobile numbers you
are looking at are the real numbers, not an extrapolation.

The problem is multiplicity. If you inspect enough post-hoc segments, one of
them will look like a winner by pure chance, and there is no way to tell from
the same dataset whether mobile is that segment or a genuine effect. The
segment was chosen after the data came in, so the usual guarantees do not
apply to it.

The sound follow-up is to treat "this wins on mobile" as a hypothesis and run
a new experiment where the segment is defined before any data arrives. Create
an experience with a [mobile audience](https://docs.croct.com/explanation/audience/examples/technology):

```cql
device's category is "mobile"
```

attach the experiment to it, and let it run to the normal thresholds. If the
mobile lift is real it will show up again, and this time the result is one
you can act on with confidence.