Result is flat overall but wins on mobile, can I ship it for mobile only?
Hi all. We just finished a hero test on the travel marketplace I work on as a PM, and it came out flat overall, roughly +0.4% and nowhere near a recommended winner. Out of curiosity I used the drill-down and filtered by device, and mobile shows a +9% lift while desktop is slightly negative.
My instinct says ship the variant to mobile users only, we have the audience tools to do that. But I picked the mobile segment after seeing the data, and something about that feels like cheating. Is acting on a post-hoc segment sound or is this wishful thinking dressed up as analysis?
1 answer
Acting on that mobile segment directly would not be sound, but you also do not have to throw the finding away.
First, the data itself is fine. The drill-down by device, audience, time window, and custom events runs on unsampled data, so the mobile numbers you are looking at are the real numbers, not an extrapolation.
The problem is multiplicity. If you inspect enough post-hoc segments, one of them will look like a winner by pure chance, and there is no way to tell from the same dataset whether mobile is that segment or a genuine effect. The segment was chosen after the data came in, so the usual guarantees do not apply to it.
The sound follow-up is to treat "this wins on mobile" as a hypothesis and run a new experiment where the segment is defined before any data arrives. Create an experience with a mobile audience:
device's category is "mobile"attach the experiment to it, and let it run to the normal thresholds. If the mobile lift is real it will show up again, and this time the result is one you can act on with confidence.