How much do anonymous users switching devices distort experiment results?
I do data science at an ecommerce group. I understand that anonymous visitors cannot keep a variant assignment across devices, there is no identifier to link the two browsers. My question is about the statistical consequence, not the mechanism.
A meaningful share of our audience browses on mobile during the day and completes the purchase on desktop in the evening. If those journeys can be exposed to both arms of a test, what does that do to measured lift? Is the failure mode false winners, or attenuation, or something worse? I want to know whether I should be discounting our results and by roughly what logic.
1 answer
The failure mode is dilution, which is the benign one. A cross-device journey that sees both arms contributes conversions that are partially attributable to the other arm, so the measured difference between variants shrinks. The bias pushes lift toward zero, it does not manufacture false winners. Your true effects are likely somewhat larger than measured, and a null result on a heavily cross-device audience is weaker evidence of no effect than it looks.
The mitigation is identity. Identified users get consistent variant assignment across devices, so calling croct.identify on login as early as your product allows shrinks the contaminated share directly. Identity resolution then links the anonymous history to the identified profile without losing data, so you are not throwing away the mobile half of the journey. There is more detail on the assignment mechanics in Variant stickiness across devices.
The second mitigation is goal design: prefer goals reachable in the same session as exposure. Your mobile-browse, desktop-buy pattern only contaminates tests whose goal sits on the far side of the device switch. A test on the mobile browsing experience with a same-session goal is clean regardless of what happens on desktop later.
Rough discounting logic: if x% of converting journeys are cross-device and anonymous, expect attenuation on that slice, so treat measured lift as a floor rather than a point estimate.