A/B Testing Email: Do’s and Don’ts

Tracy Sestili
Aug. 10, 2016 by Tracy Sestili

AB Testing Email

Rules for A/B Testing Email

When businesses A/B test, they’re testing two web pages at the same time to see which one performs better. Similarly, when we’re AB testing email, we’re testing two emails against one another to a subset of our subscriber base to see which performs better so that we can send the winning email to the remainder of our list.

AB testing email can be one of the most profitable and insightful things a company can do to optimize conversions. You want to make sure you:

Do try and do this

  • Pick your variables (decide what you’re testing)
  • Pick what you want to measure (#opens, #clicks, open rate %, deliverability %, etc.)
  • Determine which recipients to send to
  • Select a time frame to run your test
  • Pick a winner

But your success is dependent on you not making one of these common mistakes in AB testing email:

Don’t make these mistakes

  1. Testing the wrong thing
    If you have a low email open rate and want to AB test to increase your open rate, then changing the layout, headlines or imagery within the email is not going to help. However, changing the subject line and time of day you send your email, will. Make sure you are testing the right thing.
  2. Measuring multiple items
    Measure one thing per test. This means if you want to test subject lines, then only swap out the subject line but keep everything else the same — including the layout, number of images, button placements, etc. If you want to test your layout, then keep the subject line and content the same, but change the layout. The more changed elements you introduce, the harder it will be to measure.
  3. Unlimited calls to action with different goals
    Try to limit your calls to action to the same goal. If you have multiple calls to action in one email going toward the same goal (same landing page, sign-up, buy, etc.) that’s fine. But when you mix CTAs or try to introduce multiple CTAs, it clouds the results. For example, if you want them to make a purchase, read a blog post and subscribe to your newsletter, did they not click on the action you were hoping for of “buy” because you mixed CTAs? The point of A/B Testing your email is to make sure that you can accurately measure the results. When you mix variables, it makes the statistical analysis that much harder.
  4. Do not constantly alter the “From Name”
    Figure out what your relationship is to your subscriber. Are they more familiar with your company name, a thought leader in your company — such as the CEO, a mascot, or product name? People are more likely to open emails from names or brands they recognize. If you constantly change the ‘from name’, you may start to see a decline in your email open rate.
  5. Sample size is too small
    You don’t want to draw your conclusion on a samples size that is too small. You can A/B test on any sample size greater than 100. A good rule of thumb to test is usually 20-30% of your subscriber list so that your confidence level is high. (Click for more on sample size and confidence levels.)
  1. Testing different content pieces
    Email is not a good place to test how content performs because it could be that one piece of content performed better simply because it was written better or the headline of the article was more appealing to that particular audience member at that particular time.


sell more with email webinar banner


  • Good to hear!

  • Very good information about email testing. Thanks for sharing.

  • Glad you found it useful, Sanjeet. Thanks for stopping by!

Related Content

3 Strategies for Your Next Financial Services Reengagement Campaign

Discover three great marketing strategies that you should consider when building your next financial services reengagement campaign.

read more

Iterable Shares What Cross-Channel Engagement Means in 2018

Discover some of the marketing insights outlined in Iterable's recently released 2018 Cross-Channel Engagement Benchmark Report.

read more

Trends and Insights for Interactive Email and AMP for Email

Watch the replay of the Trends and Insights for Interactive Email and AMP for Email webinar today to learn more about the future of interactivity in email.

read more

Get started and start sending

Try SparkPost and see how easy it is to deliver your app’s email on time and to the inbox.

Try Free

Send this to a friend