# How much do anonymous users switching devices distort experiment results?

Asked by piotr_lw on 2025-11-05. Tags: ab-testing, identity, methodology.

I do data science at an ecommerce group. I understand that anonymous visitors
cannot keep a variant assignment across devices, there is no identifier to link
the two browsers. My question is about the statistical consequence, not the
mechanism.

A meaningful share of our audience browses on mobile during the day and
completes the purchase on desktop in the evening. If those journeys can be
exposed to both arms of a test, what does that do to measured lift? Is the
failure mode false winners, or attenuation, or something worse? I want to know
whether I should be discounting our results and by roughly what logic.

## 1 answer

### Accepted answer from Felix Gruber (2025-11-06)

The failure mode is dilution, which is the benign one. A cross-device journey
that sees both arms contributes conversions that are partially attributable
to the other arm, so the measured difference between variants shrinks. The
bias pushes lift toward zero, it does not manufacture false winners. Your
true effects are likely somewhat larger than measured, and a null result on a
heavily cross-device audience is weaker evidence of no effect than it looks.

The mitigation is identity. Identified users get consistent variant
assignment across devices, so calling `croct.identify` on login as early as
your product allows shrinks the contaminated share directly. Identity
resolution then links the anonymous history to the identified profile without
losing data, so you are not throwing away the mobile half of the journey.
There is more detail on the assignment mechanics in
[Variant stickiness across devices](/answers/variant-stickiness-across-devices).

The second mitigation is goal design: prefer goals reachable in the same
session as exposure. Your mobile-browse, desktop-buy pattern only
contaminates tests whose goal sits on the far side of the device switch. A
test on the mobile browsing experience with a same-session goal is clean
regardless of what happens on desktop later.

Rough discounting logic: if x% of converting journeys are cross-device and
anonymous, expect attenuation on that slice, so treat measured lift as a
floor rather than a point estimate.