Enabled bot filtering and my conversion rates jumped, which numbers were right?
Upgraded to Growth last week, turned on bot traffic filtering. Before/after on our main signup experiment:
visitors/day conv ratebefore 4,100 1.9%after 3,300 2.4%Visitor counts dropped ~20%, conversion rates rose. Not shocked, LLM crawlers had been a growing share of our traffic for months (the Croct blog post on LLM bots and web traffic matched what I was seeing).
Two questions. Which period do I trust for reporting? And is the experiment that straddled the change salvageable or do I rerun it
1 answer
Trust the post-filtering numbers. Bots convert at roughly zero, so before the change they were pure denominator inflation: they padded your visitor counts while contributing nothing to conversions. The 2.4% is closer to how your human visitors actually behave, the 1.9% was diluted. Your ~20% drop is consistent with what sites with heavy crawler exposure see.
Two side effects of enabling it that work in your favor. Bot traffic filtering is available from the Growth plan up, and once it is on, known bots also stop counting toward your MAU, so your quota now reflects human traffic only. See how known bots are excluded from your metrics.
On the straddling experiment: rerun it if the result was anywhere near borderline. The problem is not that either arm was treated unfairly, both arms got cleaner at the same moment. The problem is that the time windows are no longer comparable: the pre-change period measures humans plus bots and the post-change period measures humans only, so any trend across the boundary mixes a real effect with a measurement change. If the experiment had a large, stable lead before and after the switch, you can reasonably keep it. Anything close, rerun on clean data. It will resolve faster now anyway since the conversion rate is no longer diluted.
Clear. It was borderline, rerunning. Nice that the bots stop eating MAU too, had not clocked that.