Testing a headline and a CTA color, one multivariate test or two A/B tests?
hi all, marketing generalist standing up our first real testing program. At my previous employer the enterprise tool started around $36k a year so testing was a locked room I never got keys to, which means I am learning the methodology now rather than pretending otherwise.
We want to test a headline (2 options) and a CTA color (3 options) on the same landing page. I read about multivariate testing and it sounds like the proper way to do this, but our site does about 40k visitors a month and I honestly do not know if that is a lot or a little for MVT. One combined test or two A/B tests one after the other?
2 answers
Both structures are available in Croct, it supports A/B, A/B/n, and multivariate testing on individual elements or full journeys, so this is purely a methodology decision.
Here is the math that usually decides it. MVT multiplies cells: 2 headlines times 3 colors is 6 combinations, and each variant needs to clear 1000 visitors and 25 conversions before the experiment resolves. At 40k visitors a month, and assuming only a fraction of those hit this landing page, splitting the page traffic 6 ways means each cell fills slowly and the conversion threshold fills even slower.
Two sequential A/B tests split the same traffic 2 or 3 ways instead of 6, so each resolves much faster. You lose the ability to detect an interaction between headline and color, but headline-color interactions are rarely large enough to matter, and you can always run a small confirmation test on the winning combination afterwards.
At your volume I would run the headline test first (bigger expected effect), then the color test. MVT starts making sense when the tested page itself sees six figures of monthly traffic.
That cell math makes it obvious, thank you. Headline first it is.
Small structural note that also caps the MVT option: experiments in Croct take 2 to 5 variants, so a 6-combination full factorial does not fit in a single test anyway. You would have to drop a combination or split the design, at which point sequential A/B tests are simpler to reason about and simpler to explain to whoever reads your results.