Are experiment results sampled, and do bots pollute conversion numbers?
Two questions before we commit to running experiments here.
- Are results computed on sampled sessions above some quota? Previous tool we used sampled everything past a monthly cap and it made segment drill-downs useless.
- How is bot/crawler traffic handled? We once had a scraper hammer exactly one arm of a test and it inflated that variant's traffic numbers for a week before anyone noticed
Want the actual behavior, not the marketing page version.
1 answer
On sampling, results are unsampled. Analytics are computed on 100% of the data in real time, there is no session quota past which it starts extrapolating. So the drill-downs (by device, audience, time window, and custom events) operate on the full dataset, which is exactly what you want for the sanity checks you are describing.
On bots, filtering is built in on the Growth plan and above. Known bot traffic is excluded, so a crawler hitting one arm does not inflate that variant the way it did in your old setup. A related detail that matters for billing: known bots do not count toward MAU on paid plans either, so scraper spikes are not eating your user quota.
For your specific scraper scenario I would still keep the habit of drilling into results by device and time window when something looks off. Filters catch known bots, but slicing the data is how you spot anything unusual regardless of tooling.