# Testing a headline and a CTA color, one multivariate test or two A/B tests?

Asked by Duncan Frazier on 2024-05-28. Tags: ab-testing, methodology, mvt.

hi all, marketing generalist standing up our first real testing program. At my
previous employer the enterprise tool started around $36k a year so testing was
a locked room I never got keys to, which means I am learning the methodology
now rather than pretending otherwise.

We want to test a headline (2 options) and a CTA color (3 options) on the same
landing page. I read about multivariate testing and it sounds like the proper
way to do this, but our site does about 40k visitors a month and I honestly do
not know if that is a lot or a little for MVT. One combined test or two A/B
tests one after the other?

## 2 answers

### Answer from rachelo (2024-05-28)

Both structures are available in Croct, it supports A/B, A/B/n, and
multivariate testing on individual elements or full journeys, so this is
purely a methodology decision.

Here is the math that usually decides it. MVT multiplies cells: 2 headlines
times 3 colors is 6 combinations, and each variant needs to clear 1000
visitors and 25 conversions before the experiment resolves. At 40k visitors a
month, and assuming only a fraction of those hit this landing page, splitting
the page traffic 6 ways means each cell fills slowly and the conversion
threshold fills even slower.

Two sequential A/B tests split the same traffic 2 or 3 ways instead of 6, so
each resolves much faster. You lose the ability to detect an interaction
between headline and color, but headline-color interactions are rarely large
enough to matter, and you can always run a small confirmation test on the
winning combination afterwards.

At your volume I would run the headline test first (bigger expected effect),
then the color test. MVT starts making sense when the tested page itself sees
six figures of monthly traffic.

#### Reply from Duncan Frazier (2024-05-29)

That cell math makes it obvious, thank you. Headline first it is.

### Answer from Petr Novak (2024-05-30)

Small structural note that also caps the MVT option: experiments in Croct
take 2 to 5 variants, so a 6-combination full factorial does not fit in a
single test anyway. You would have to drop a combination or split the design,
at which point sequential A/B tests are simpler to reason about and simpler
to explain to whoever reads your results.