QuickBooks HelpQuickBooksHelpIntuit

About A/B Tests

by Intuit1 Updated 1 week ago

A/B tests compare different versions of a single email to see how small changes can have an impact on your results. Choose what you want to test, like the subject line or content, and compare results to find out what works and what doesn't work for your audience.

In this article, you'll learn about A/B tests.

Things to know

Definitions

Here are some terms you’ll come across when you use A/B tests in Mailchimp.

  • Variable
    The element of your test that you want to compare. With an A/B test, you can test 1 of 4 variables: subject line, From name, content, or send time. Each version of the variable is called a variation.

  • Combination
    Each variation of your test that is created from the variables you choose. If you want to test 3 different From names, we'll create 3 different combinations of your campaign. Combinations sent in the test phase are called test combinations.

  • Test phase
    The period of time after the combinations are sent out and we compare the results. Data collected during the test phase can be used to determine the test's winning combination automatically or manually.

  • Winner or winning combination
    The combination that performs the best. This may be automatically determined by click rate, open rate, or total revenue, or manually chosen based on the reporting data you find the most valuable.

How A/B tests work

Set up the A/B test

You'll choose a single variable type—subject line, From name, content, or send time—and create up to 3 variations. We'll generate all possible combinations and send them to different sets of subscribed contacts, so no one receives more than 1 combination of your test.

The combination that your subscribed contacts receive are chosen at random and tracked solely for the purpose of choosing a winner. This means you won't be able to see which combination a subscribed contact receives.

Choose winner criteria

Send the combinations to all your subscribed contacts at once if you have a small audience or segment, or if you're testing send time. With other variables or a large audience or segment, send your test combinations to a percentage of your subscribed contacts, and send the winning combination to your remaining subscribed contacts.

To choose the winner, use 1 of these options.

  • Automatic: open rate, click rate, or total revenue
    Use this option to send the winning combination to your remaining subscribed contacts after a set amount of time. The winner can be determined by the highest open or click rate, or total revenue if your online store is connected to your account.

Graphic showing how two combinations are sent and the winner is picked based on the higher click rate.

  • Manual: report statistics
    Use this option to choose a winner yourself based on reporting data or other factors that you find to be the most valuable.

Variables you can test

Here’s some more information on the variables you can test along with some ideas to help you get started.

  • Subject line
    Try different phrasing, or sales offers, or emoji to see what gets the most attention.
  • From name
    See if your subscribed contacts are more responsive to emails coming from a person's name or from the name of your company or organization. You'll provide the From name and From email address you want to use for each combination.
  • Content
    Create different versions of your content to see what gets a better response. Use this variable to test small content changes or completely different templates.

    When you test content, you may want to better understand how successful your calls to action, links, or buttons are. Use our link comparison tool in the email report to see how your links performed in each combination.

  • Send time
    Learn when your subscribed contacts are most likely to open your emails. Since this option tests specific days and times, you must send your combinations to all of your subscribed contacts at once because the winning combination can't be sent at a past time. Instead, use this data to inform when to send or schedule future emails.

A/B test ideas

Here are some common ways to use A/B tests to improve your marketing.

  • What day of the week gets better open rates?
  • Does a subject line with an incentive or a teaser work best?
  • Does including your company name in your subject line increase engagement?
  • Is it better to use your name as the From name, or your company's name?
  • Does the time of day an email is sent affect the click rate?
  • Are subscribed contacts more likely to click a linked image or linked text?
  • Do subscribed contacts prefer an email that contains a GIF or one with static images?
Mailchimp

Sign in now for personalized help

See articles customized for your product and join our large community of QuickBooks users.

More like this