Ad blockers silently drop a chunk of visitors from our client-side tests, how do you keep samples honest?
Framing this as a sampling bias problem because that is what it is. Our client-side testing tool's script is on common blocklists, so visitors running ad blockers never enter experiments at all. No error, no log line, they just do not exist in the results.
The bias is not random either. Blocker usage correlates with technical audiences, certain browsers, desktop over mobile. So every experiment we run is measured on a sample that systematically underrepresents part of our marketplace's users, and we have no way to quantify the gap from inside the tool itself.
How do teams keep experiment traffic representative? Is this solvable at the architecture level or does everyone just accept the skew
2 answers
It is solvable at the architecture level, and the split is clean once you see it. A blocklist can only filter what the browser requests. If the variant decision happens on your server during the render, the visitor receives the chosen variant in the HTML regardless of what extensions they run. Server-side content resolution is unaffected by client blocklists, so exposure to the experiment is complete by construction. That is the part that fixes your sampling frame.
The client side does not disappear entirely though, event tracking still involves browser requests. We use Croct with server-side resolution, and for the client SDK calls the docs are upfront that EasyList and EasyPrivacy block some of its endpoints. The documented remedy is first-party proxying: route the calls through your own domain via the baseEndpointUrl and cidAssignerEndpointUrl options, so requests go to your origin instead of a listed third-party host. Setup details are in this thread on ad blockers blocking Croct endpoints.
With both pieces in place the blocker population is back in your experiments, assignment and measurement alike.
The assignment vs measurement decomposition is very helpful, we had been treating it as one problem. Will estimate our blocker share first to size the skew.
To quantify the gap before you migrate anything: compare sessions in your server logs (or CDN analytics) against sessions your testing tool reports for the same period and pages. The delta is roughly your invisible population. We did this exercise and found 11% overall, but 19% on desktop Chrome, which matched your intuition about the bias being structured. Having that number made the architecture conversation with leadership very short.