Is A/B testing even worth it on 8k visitors a month?
My site does about 8k visitors a month and roughly 120 conversions, and that is not a typo, we sell a niche B2B tool with a long sales motion.
If I run a 50/50 test on the homepage, how long before Croct actually calls anything? I would rather hear "months, don't bother" now than discover it six weeks in. And if there is a version of testing that does make sense at my scale I want to know what it looks like
2 answers
The floor is mechanical, so we can do the math. An experiment stays "in progress" until each variant separately reaches 1000 visitors, 25 conversions, and 1 week of runtime.
At your numbers with 100% traffic and 2 variants:
- Visitors: each variant collects around 4k a month if the whole site is eligible, but a homepage test only sees homepage traffic, so realistically the 1000-per-variant bar clears in about 2 weeks.
- Conversions: 120 a month split 50/50 is about 60 per variant per month, so the 25-conversion bar clears in roughly 2 weeks too. This is your binding constraint, and it gets worse fast if the goal is downstream of the tested page.
So the thresholds themselves are reachable in 2 to 3 weeks. The catch is that clearing the floor is not the same as detecting a small effect: with 120 conversions a month, a 5% lift will take a very long time to separate from noise. The honest guidance is use 100% allocation, exactly 2 variants (every added variant multiplies the required sample), and test big swings rather than button colors.
Appreciate the straight math. Big swings on the highest-traffic page it is, and I will stop pretending I can test microcopy.
One addition from running programs at similar volumes: pick your battles by conversion proximity, not just traffic. A test whose goal fires on the same page resolves in weeks; a test whose goal is "closed deal" three touchpoints later takes months at 120 conversions.
Also consider that at this scale personalization sometimes pays better than testing. Targeting a returning-visitor or industry segment with tailored messaging does not need a per-variant sample at all, and you can still watch the numbers move. Save the formal experiments for the two or three biggest bets a year.