How many variants can one experiment have?
Planning an A/B/n headline test for a client and we brainstormed seven candidate headlines. Before I start cutting, what is the actual ceiling on variants per experiment in Croct?
And a methodology question on top of that. Their traffic is decent but not huge, maybe 40k sessions a month on the target page. Is running that many arms even statistically sane at that level, or am I going to be waiting forever for significance on each arm?
2 answers
The limit is 2 to 5 variants per experiment, and that covers A/B, A/B/n, and multivariate setups on individual elements or full journeys. So seven headlines will not fit in a single test regardless.
On the sample question, keep in mind that every variant you add multiplies the total sample you need, because each variant must independently reach at least 1000 visitors and 25 conversions before a winner can even be considered. Five arms means five buckets each clearing those bars, and your 40k monthly sessions get split five ways on that one page.
Been through this exact situation at our agency. With modest traffic the practical answer is to shortlist hard: pick your 2 or 3 strongest headlines, run a clean test, then iterate with the learnings in a follow-up experiment. Two sequential 3-arm tests will usually get you a confident answer faster than one bloated 5-arm test that crawls toward the thresholds for months.
Seven candidates almost always collapse into 2-3 genuinely different hypotheses anyway, the rest are wording tweaks of each other.