Is it worth running an A/A test first to check my Croct setup?
before i trust any real results i want to sanity-check the plumbing, so my plan is an A/A test: two identical variants, same content, and just confirm that traffic splits evenly and the goal counts the way i expect.
is that a sane thing to do on Croct, and is there anything specific i should watch for while it runs
2 answers
It is a sane check and I have done it before rolling out a new goal. A couple of things to line up first.
An experiment attaches to an experience with a single primary goal, and that goal has to be an event you are already tracking. So the real value of the A/A run is verifying that the goal fires and gets counted, not just that traffic splits. Configure your allocation and your two identical variants (Croct allows 2 to 5), then let it collect.
What to watch: the experiment stays "in progress" until each variant reaches at least 1000 visitors, at least 25 conversions, and at least one full week. Do not read anything into the numbers before all three clear for both variants, an A/A on thin data will look lopsided just from noise.
One reassurance on the split itself: bot filtering keeps known bots out of the counts, so a crawler wave will not quietly skew one side.
ran it exactly like this. split came out even once both variants cleared the thresholds and the goal counted on both sides, so the plumbing is good. thanks
Adding one expectation so you are not surprised: with two truly identical variants no recommended winner should ever surface. There is no real difference to detect, so probability to be best will hover around 50/50 and neither side will cross the 95% bar. If a winner did get flagged on a pure A/A, that would be the actual red flag worth chasing.