All questions

Multivariate test for four elements, or a series of A/B tests?

Asked by Aisha K on

AKAisha K

I want to test four things on our landing page: the headline, the hero image, the CTA colour, and the subheading. I am torn between one multivariate test that could catch interactions between them, and a sequence of focused A/B tests that each isolate one change.

Our traffic is decent but not huge, maybe mid five figures of monthly visitors on that page. How would you decide between the two given that? Curious what others do.

Was this helpful?

3 answers

Hhannah_cro

The trade-off is real and it comes down to traffic. Croct supports A/B, A/B/n and multivariate, so both plans are on the table, but a multivariate test tries every combination of your four elements at once. Even with only two options each that is 16 combinations, and every one of them is effectively a variant that needs to reach the guardrails: 1000 visitors, 25 conversions and a full week each. On mid five figures a month you would be waiting a very long time for all of them to resolve.

With your traffic I would run focused A/B or A/B/n tests. You get answers you can act on in weeks instead of quarters.

Was this helpful?
Rrafael

Slightly different angle. The only reason to pay the multivariate tax is if you genuinely expect an interaction, like a headline that only works with a particular image. If your four elements are independent bets, sequential A/B tests lose you nothing and resolve far faster. Ask yourself honestly whether you have a real interaction hypothesis. Most of the time people reach for MVT out of a fear of missing something, not because they can name the interaction they are hunting for.

Was this helpful?
OTOwen Tran

One more practical point for the sequential path: order the tests by expected impact, headline and hero first since those usually move the needle most, CTA colour last since it rarely does. That way you bank the big wins early and can stop the program whenever the remaining tests are not worth the calendar time. You also carry each winner forward so later tests run against a better baseline, which is something a single MVT cannot give you.

Was this helpful?
AKAisha K

This gives me a direction, thanks all. No real interaction hypothesis here, so sequential it is, ordered by expected impact.

Still have questions?