{"id":24079,"date":"2025-10-15T18:57:10","date_gmt":"2025-10-15T16:57:10","guid":{"rendered":"https:\/\/varify.io\/?p=24079"},"modified":"2025-10-19T18:18:31","modified_gmt":"2025-10-19T16:18:31","slug":"multivariate-tests","status":"publish","type":"post","link":"https:\/\/varify.io\/en\/blog\/multivariate-testing\/","title":{"rendered":"Multivariate tests: Why they are (usually) a bad idea - and when they are really worthwhile"},"content":{"rendered":"
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Multivariate tests (MVT) seem powerful at first glance - you can test several elements simultaneously and recognize their interactions.<\/p>

In practice, however, they are traffic-hungry<\/b>, statistically complicated<\/b> and simply too slow<\/b> for most websites.<\/p>

An MVT only makes sense from several hundred thousand visitors per month<\/b> or conversion rates above 5%<\/b>.<\/p>

Until then: Well-planned A\/B\/n tests deliver valid findings more quickly<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What multivariate tests actually do<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A multivariate test checks several elements simultaneously<\/strong>.<\/p>

Example:
You test 2 headlines \u00d7 2 images \u00d7 2 buttons \u2192 this results in 8 combinations<\/strong>.<\/p>

Each combination gets its share of the traffic - only 1\/8 of the visitors if evenly distributed.<\/p>

And this is precisely where the problem lies: The traffic per variant shrinks dramatically<\/strong>, and with it the meaningfulness of your data<\/strong>.<\/p>

Result: Tests take much longer<\/strong>, significance is reached later<\/strong>, and you block other experiments<\/strong> in the meantime.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why multivariate tests are rarely useful<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
\n\t\t\t<\/i>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

1. traffic is spread over too many variants<\/h3>\nMore combinations mean that your visitors are spread across many groups.\n This reduces the statistical power (i.e. the ability to recognize real effects).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
\n\t\t\t<\/i>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

2. the tests take forever<\/h3>

Even with 100,000 visitors per month, it can take months<\/strong> before you see a result - whereas you would have generated several valid learnings long ago with a simple A\/B test.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
\n\t\t\t<\/i>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

3. more complex statistics & higher error rate<\/h3>

Each additional variant increases your risk of randomly finding a false winner<\/strong> (this is called alpha error accumulation<\/strong>).<\/p>

The more you test, the greater the chance that a variant will accidentally look better<\/strong>, even though it is not. This is also known as the \"multiple comparison problem\".<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
\n\t\t\t<\/i>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

4. small effects, big effort<\/h3>\nMVTs often only show fine-tuning effects - e.g. that button A performs minimally better with headline B. In practice, such mini-effects are usually not relevant enough to justify the effort.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What \"power\" really means (simply explained)<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The power<\/strong> of a test describes the probability that you will actually detect a real effect<\/strong>.<\/p>

For example, if a test has a power of 80 %<\/strong>, this means that if a real difference exists, you will also find it statistically in 8 out of 10 cases<\/strong>.<\/p>

In 2 out of 10 cases (20 %) you would falsely believe that there is no effect<\/strong>.<\/p>

The smaller the uplift, the more visitors<\/strong> you need to maintain the same power value.<\/p>

An MVT with 8 combinations therefore needs much more traffic<\/strong> to achieve the same power as a simple A\/B test.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why you need at least 500 conversions per variant<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A rough rule of CRO practice is:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t

\n\t\t\t\tAt least 500 conversions per variant are necessary to be able to make a reasonably stable statement.\t\t\t<\/p>\n\t\t\t\t\t<\/blockquote>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

There are three reasons for this:<\/p>

  1. Conversion rates fluctuate extremely with very small samples - small coincidences can distort the result.<\/li>
  2. Tools cannot perform clean significance tests if the events per variant are too small.<\/li>
  3. From around 500 conversions, the distribution stabilizes in such a way that even small differences become visible.<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Important:<\/strong> This applies per variant<\/strong> - i.e. also for every combination in an MVT.<\/p>

    If you have 8 combinations, you need at least 8 \u00d7 500 = 4000 conversions<\/strong> just to fulfill the basic requirement.<\/p>

    Depending on the conversion rate, this can quickly attract hundreds of thousands of visitors<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

    \n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    The alpha error: Why more variants increase the risk<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    The alpha error (error of the first kind)<\/strong> describes the probability that you mistakenly identify a variant as the winner<\/strong>, although no real difference exists.<\/p>

    An example:
    If you are working with a significance level of 5 % (\u03b1=0.05\\alpha=0{,}05\u03b1=0.05), this means:<\/p>