Create a Multivariate Test

There are many ways to present your email content, and you may wonder which way is best. Use our multivariate testing feature to see how small changes to your emails impact subscriber engagement.

In this article, you'll learn how to create and edit a multivariate test.

Before you start

• This feature is included with the Standard plan or higher.
• Before you create a multivariate test, you may want to familiarize yourself with the terminology and ways multivariate tests can be used.
• We recommend you send each combination to at least 5,000 subscribed contacts to get the most useful data, but smaller tests can still help you learn about your audience’s preferences.
• Wait at least 4 hours after you send a multivariate test to send a winning email. Many subscribed contacts won't engage with your email sooner, which can misinform your test results. Shorter test times may cause a losing combination to go out to the rest of your audience. You can see the test results and the time the winner was selected on the report page.
• In multivariate tests where content is a variable, choose the winner by best click rate, since opens occur before a recipient views your content.

Create a multivariate test

Multivariate tests are built in a series of steps. First, you'll create and name your emails, then you'll choose your recipients, decide which variables to test, choose your setup, and add your content. Last, you'll confirm that everything looks good and send.

To create a multivariate test, follow these steps.

1. Click Create, then click Email.
2. Click Multivariate.
3. Enter an email name and click Begin.

Recipients step

On the Recipients step of the builder, choose the audience or segment you want to send your test to. You can configure a new segment on this screen or choose a saved segment.

When you're ready, click Next.

Variables step

You'll configure the elements you want to test on the Variables step.

Choose whether to test your subject line, From name, content, or send time. Choose to test up to 3 variables at once.

To configure your variables, follow these steps.

1. On the Variables step, click each variable you want to test.
2. Click the plus or minus icons to add or remove additional variations of the variable.

You can test up to 3 of 4 variables, to create up to 8 combinations. The number of variations of each variable multiplied together equals the number of combinations. This number is shown in the Summary section. For example, if you want to test 3 subject lines and 2 variations of content, we'll generate 6 combinations of your email.

1. In the What percentage of your recipients should receive your test combinations section, use the slider to choose how many of your recipients you want to send your test combinations to.

Send combinations to 100% of your recipients if your audience or segment doesn't allow for 5,000 subscribed contacts in each combination. If you're testing Send time, this is your only option.

1. Review the Summary section to see how many of your subscribed contacts will receive each combination.
2. Choose whether to determine a winner based on open rate, click rate, total revenue, or manually. Then, choose how long you want the test to run.
   
   At the end of the test phase, we'll send the winning combination to your remaining recipients.
3. Click Next.

During the test phase, you can convert to manual selection to choose the winner from the Reports page.

Setup step

Depending on which variables you test, the Setup step will present multiple Email subject, From email address and name, and Subject line fields.

To choose your setup, follow these steps.

1. Type in your variation content in the fields provided. Settings for Send time variations also appear on this step.
   
2. Check the boxes next to your preferred settings or tracking options.
3. Click Next.

Content step

The Content step is where you add content to your test. What you see on this step will vary if you choose to test content.

Test subject line, From name, or send time

To add content when your variable is subject line, From name, or send time, follow these steps.

1. On the Select a Template step of the content builder, choose your layout.
2. Design and preview your email in the builder.
3. When everything is how you want it, click Confirm before sending.

Test content

If your variable is content, the Content step shows a preview for each variation.

To add content to each variation, follow these steps.

1. Click Add Content for any variation of your email.
   
2. On the Select a template step of the email builder, choose your layout.
3. Design and preview your email in the builder. When everything is how you want it, click Save and return to content.
4. On the Description step of the content builder, type in text to describe the variation of the content. Your subscribed contacts won't see this description, but it will help you remember the differences between each variation.
   
5. Click Continue to content to return to your content variations.
6. Repeat steps 1-5 for each content variation. On the Select a template step, use a different template or select a variation you already designed in the Previous Variations tab.
7. When everything is how you want it, click Next.

Confirm step

The Confirm step is where you'll review the combinations that make up your multivariate test and send them when ready.

To confirm and send your multivariate test, follow these steps.

1. Review the pre-delivery checklist. Click Resolve or Edit to make changes to any section.
2. After you're finished, click Send to send your test now, or click Schedule to schedule it to for a later date.

Account users with a Manager user level or higher will receive an email when a winner has been automatically chosen, or after 4 hours if you haven't yet picked a winner manually.

• If a subscribed contact opens the archive page for a sent email, they'll always see the test combination they received, even if the winning combination is different.
• The winning combination of your multivariate test will appear in the public email archive for your audience, unless you tested send time.

Manually choose winner

If you chose Manual Selection from the drop-down menu on the Variables step of the builder, you'll be able to view the email report after the test combinations are sent, and choose the winner.

To manually choose a winner, follow these steps.

1. Click Analytics, then click Reports.
2. Click View Report for the multivariate test you want to work with.
3. An alert will tell you how long into the testing phase you are and recommend that you wait at least 4 hours.
   
4. Review the report for each combination. Under Finalize your test, click the radio button next to the combination you want to send to your remaining subscribed contacts.
   
5. Click Send To Remaining Recipients.
6. Click Send Now in the confirmation pop-up modal to send the winning combination.

Convert to manual selection

After you send your test with the choice to have the winner chosen by open rate, click rate, or total revenue, you may decide that you want to manually choose the winning combination instead.

To convert to manual selection after you sent your test, follow these steps.

1. Click Analytics, then click Reports.
2. Click View Report for the multivariate test you want to work with.
3. Click Convert to manual selection in the alert block.
   
4. In the pop-up modal, type CONVERT and click Convert.
   
5. When you're ready, choose a combination to immediately send to your remaining recipients.

In the event that none of your recipients opens or clicks in any of your combinations, the final set of recipients will get the last combination by default. This could happen if your test sends to very few recipients.

If the same number of people click and open your combinations, the first combination will send to the remaining recipients.