Create an A/B Test

Each time you create an email, you may wonder whether your content is presented in the best way for your audience. Use an A/B test to compare how small changes can impact your marketing engagement.

In this article, you'll learn how to create and edit A/B tests.

Before you start

Here are some things to know before you begin this process.

• Email and SMS A/B testing is included with the Essentials plan or higher. The popup form A/B test feature is included with the Standard plan or higher.
• For information about how A/B tests work, read About A/B Tests.
• Create a Multivariate test to compare multiple variables in a single email. This feature is included with the Standard plan or higher.
• When you test content as the variable in an A/B test, choose the winner by click rate, as opposed to open rate, since opens occur before a subscribed contact views your content.

Create an A/B test email

Email A/B tests are built in a series of steps. First you'll create and name your test, then you'll choose your recipients, decide which variable to test, choose your setup, and add your content. Last, you'll confirm that everything looks good and send.

To create a new A/B test, follow these steps.

1. Click the Create icon, then click Email.
2. Click A/B test.
3. Enter an email name and click Begin.

If you see the Multivariate option instead, your plan includes advanced testing options. To learn more, check out Create a Multivariate Test.

Recipients step

On the Recipients step, choose the audience or segment you want to send your test to. Configure a new segment on this screen, or choose a saved segment.

When you're ready, click Next.

Variables step

On the Variables step, choose whether to test your subject line, From name, content, or send time.

To choose the variable you want to test, follow these steps.

1. On the Variables step, click the variable you want to test.
   
2. Click the plus or minus icons to add or remove variations of the variable. Test 1 variable with up to 3 variations.
   
   Each variation generates a combination as shown in the Summary section. For example, to test 3 subject lines, we'll create 3 combinations of your email.
   
3. In the What percentage of your recipients should receive your test combinations? section, use the slider to choose how many of your subscribed contacts you want to send your test combinations to.
   
   Review the number of recipients who'll receive each combination in the Summary section. We require that you send to at least 10% of your recipients for any type of A/B test, so the slider won't go below 10%. If you're testing Send time, you'll always send to 100% of the subscribed email contacts in your chosen audience or segment.
   
   We recommend you send each combination to at least 5,000 subscribed email contacts to get the most useful data. Smaller tests can also help you learn about your audience’s preferences.
4. Choose whether to determine a winner based on open rate, click rate, total revenue, or manually. Then, choose how long you want the test to run.
   
   At the end of the test phase, we'll send the winning combination to your remaining recipients. If you choose to send test combinations to 100% of your recipients, you won't see this option.
5. Click Next.

Setup step

Depending on which variable you test, the Setup step will present multiple Email subject, From email address and name, and Subject line fields.

To choose your setup, follow these steps.

1. Type your Email subject, From email address, and From name in the fields provided. Send time variation settings also appear on this step.
2. Check the boxes next to your preferred settings or tracking options. Track opens and Track clicks can't be disabled, since they're used to determine the winner of the test.
3. Click Next.

Content step

The Content step is where you add content to your test. What you see on this step will vary if you choose to test content.

Test subject line, From name, or send time

To add content when your variable is subject line, From name, or send time, follow these steps.

1. On the Select a template step of the content builder, choose your layout.
2. Design and preview your email in the builder.
3. When everything is how you want it, click Confirm before sending.

Test content

If your variable is content, the Content step shows a preview for each variation.

To add content to each variation, follow these steps.

1. Click Add Content for a variation of your email.
2. On the Select a template step of the content builder, choose your layout.
3. Design and preview your email in the builder. When everything is how you want it, click Save and return to content.
4. On the Description step of the content builder, type in text to describe the variation of the content. Your subscribed email contacts won't see this description, but it will help you remember the differences between each variation.
5. Click Continue to content to return to your content variations.
6. Repeat steps 1-5 for each content variation. On the Select a template step, use a different template or select a variation you already designed in the Previous Variations tab.
7. When everything is how you want it, click Next.

Confirm step

The Confirm step is where you'll review the combinations that make up your A/B test and send them when ready.

To confirm and send your A/B test, follow these steps.

1. Review the pre-delivery checklist. Click Resolve or Edit to make changes.
2. When you're ready, click Send to send your test now, or click Schedule to schedule it for a later date.

Subscribed email contacts may not interact with your email right away, so test data may take some time to populate. We recommend you wait at least 4 hours after you send an A/B test to send the winning combination. Winner notification emails are sent to any user on your account with a user level of Manager or higher.

• If a subscribed contact opens the email in their browser, they'll always see the test combination they received, even if the winning combination is different.
• The winning combination of your A/B test appears in the public email archive for your audience, unless you tested send time.

Manually choose a winner

If you choose Manual selection from the drop-down on the Variables step of the builder, you'll be able to view the report after tests are sent, and choose a winner.

To manually choose a winner, follow these steps.

1. Click Analytics, then click Reports.
2. Click View report for the A/B test you want to work with.
3. An alert will tell you how long into the testing phase you are and recommend that you wait at least 4 hours.
4. Review the report for each combination. Under Finalize your test, click the radio button next to the combination you want to send to your remaining subscribed contacts.
   
5. Click Send To Remaining Recipients.
6. In the confirmation pop-up modal, click Send Now.

Convert to manual selection

After you send your test, you may decide you want to manually choose the winning combination.

To convert to manual selection after you send your test, follow these steps.

1. Click Analytics, then click Reports.
2. Click View Report for the A/B test you want to work with.
3. Click Convert to manual selection in the alert block.
4. In the popup modal, type CONVERT in all caps and click Convert.
5. When you're ready, choose a combination to immediately send to your remaining recipients.

• If you sent test combinations to 100% of recipients, you won't see the option to manually choose a winner because they've already received your email.
• In the event that none of your recipients open or click any test combination, the final set of recipients will get the last combination by default. This could happen if your test sends to very few recipients.
• If the same number of people click and open your combinations, the first combination will send to the remaining recipients.

Create an SMS A/B test

Use an A/B test to align text messages with your audience. Learn about the best message length, whether recipients prefer emojis or plain text, if they prefer images, or offer promo codes. With A/B testing for SMS, we'll automatically choose the winning variation based on the highest click rate. You can also manually select the winner.

• To determine a winning version, each message in your A/B test needs a link.
• To run an A/B test, we need to shorten and track links added to your messages.

To create a new A/B test, follow these steps.

1. Click SMS.
2. Click Create SMS.
3. In the Content section, click Edit.
4. In the SMS builder, click A/B test this message.

The screen displays a dropdown above the Message section with your two variations labeled as “Message A” and “Message B.” There'll also be a section with mockups of the two messages that you can click. When a mockup is clicked on, you’ll have access to the Message field to make changes.

After you’ve designed your message, click Save and continue to go back to the SMS checklist.

A/B test details

After you select to A/B test the message, the A/B test details section is present in the SMS checklist. Here, you can determine the percentage of recipients who'll receive each variation, select the amount of time the test will run for, and what the winning criteria will be.

Once you’ve set the conditions for the A/B test, click Save to go back to the SMS checklist. Click Send or schedule to send the A/B test out now or choose to schedule it for a later date. If you want to send or schedule your A/B test at a later date, click Finish later to go back to the SMS Overview page.

End the test early

After the A/B test has started sending, you may want to manually choose the winning combination.

To end the test early after you send your test, follow these steps.

1. Click Analytics, then click Reports.
2. Click View Report for the A/B test you want to work with.
3. Select the Test Results tab, then click End test early.
4. Next, choose the test version you want to send to the rest of your recipients. You can also end the test here without sending a winning version.
5. If you’ve selected a winning version, click the Send winner to remaining recipients button.
6. In the End test and send selected version pop-up modal, click Send.

Create a popup form A/B test

Use an A/B test to learn which version of your popup form performs better. We’ll create a duplicate of your popup form that you can edit, then test. Update colors, images, text, or change your offer to see what resonates with your website visitors. 50% of recipients will receive each variation. After you publish, your A/B test runs until you choose a winner.

To create a popup form A/B test, follow these steps.

1. Select the draft popup form you want to work with.
2. In the What should it look like? section of the checklist, click Add version to A/B test.
3. Click Edit test title to enter a name for your A/B test.
   The Content A/B test details section shows the percentage of recipients who’ll receive each test version.
4. Click Save.
5. Click the Edit content drop-down to design your content for Version A and Version B. In the Editor, update colors, images, text, or change your offer.
6. When you’re ready, click Save and review.
7. The Preview window in the popup form builder shows your 2 test versions. Click the Edit drop-down to make changes, or click Remove.
   
8. Click Edit title to rename each version. We suggest that you name ‌versions based on what you’re trying to test. For example, if you want to test warm colors and cool colors, name each version accordingly.
9. Click Publish or schedule to publish your A/B test now or schedule it for a later time.
10. On the Form details page, we’ll populate data for unique impressions, email, and SMS opt-in rates for each version. Track analytics for all devices or by device type.
11. Evaluate the test data, then click Select as winner next to the winning version.
    
12. Click Publish popup to publish the winning version live on your website.

After you publish, the A/B test automatically ends. The winning popup form version is live for all website visitors. All performance data starts over and the previous version is archived.