Introduction to A/B Testing for Email Notifications

How to use testing to build more effective SaaS email notifications

Because email notifications are so crucial to building user engagement with SaaS apps and services, their performance shouldn’t be left to chance. A/B testing is one way for product teams to measure the performance of emails like these. In this article, you will learn:

What A/B testing means

A/B testing is a method of experimenting and measuring which of two variants of something like a UX element, ad copy, or an email message generates better results. It often is employed by marketers to determine which creative element performs best in a marketing piece, such as an email or a landing page. They create two versions of one element—such as text, image, layout, or a call-to-action (CTA) button—and leave everything else the same, so they can see which one drove stronger results for their marketing campaign.

Email marketers typically test the subject line, body text, an image, layout, a CTA button, or even the day and time they send messages to determine which version prompts more people to open the email, visit their website, or perform another desired action. After sending their message and giving recipients enough time to decide if they’re going to open the email or ignore or delete it, email marketers check the metrics for the campaign. They look at such numbers as the percentage of emails opened and the number of CTA clicks to determine which version was more successful.

Today, that same approach has been adopted by product growth teams to analyze the effects of in-app UX and other features on user engagement. Does changing the color or placement of a particular button result in more clicks? Does a particular menu configuration result in higher use of particular pages? Do variations on the wording or frequency of particular notifications drive more visits?

A/B tests are most suitable for assessing directly measurable impacts such as clicks or visits. But with the proper data attribution model, they can be used to assess impact on downstream activity, such as product purchases or subscription sign-ups, to determine if their efforts were successful for driving overall business goals. Customer conversions or improvements to customer lifetime value are examples of this sort of effort.

How to A/B test email notifications

Let’s say you’re the product manager for Widget.ly, a fictional SaaS app that helps widget teams manage their widget workflows. Your analytics have shown that frequency of widget updates is strongly correlated to your overall customer retention rate. You’ve also found that notifications are an effective way of reminding your users to return to the site to update the status of their widget. Every marginal improvement in response to a notification message will have a real impact of your business performance.

Accordingly, you decide to run an experiment to see if you can raise the response rate for this particular type of email notification. You could test different CTAs, but perhaps a better first step might be improving the number of times your notifications even get opened, in the theory that increasing the topline is a good way to also improve later metrics.

So you’re going to A/B test the subject line of this particular notification with 10% of your active users and see which email has better open rates in the first 24 hours before putting any change into broader production.

You could see if a subject line that’s personalized with your user’s name will be more successful than the generic subject line you’ve been using. For example:

Subject line 1: There are 4 widgets waiting for approval

Subject line 2: Brent, you have 4 widgets waiting for approval

For A/B tests on existing features, it’s important to keep one of your variants unchanged—just like scientific experiments, maintaining a consistent control group is essential to getting accurate results. Keep the content, CTA, etc., of your email the same vary only the subject line. Once you have those results, then do another experiment to see if an alternate CTA improves results further.

Examples of common A/B tests for email notifications

Aspects of email notifications that are ripe for A/B testing include:

  • Subject line
  • Personalization
  • Images in messages
  • Call to action
  • “From” email address
  • Frequency of notifications
  • Time of day for scheduled notifications
  • Deep linking to your app

Automating your app’s A/B tests

A/B testing is a fundamental part of most marketing automation products. Email marketers, for example, use A/B testing on subject lines, images, calls to action, and other aspects of almost every campaign they execute.

But for product teams and others who don’t need a full-on marketing stack, there also are many more focused ways to run A/B tests and view the data from your experiments. For example, web site optimization tools are a common approach to ensuring an app’s UX is as effective as possible.

Another approach is incorporating native A/B tests into the API for specific services. The SparkPost email API is an illustration of this approach. It includes an A/B test function that allows for testing email notifications and other messages in real-time. After defining the templates to test, sample recipients, sample size, and test duration, the template with the best conversion rate wins and the remaining recipients receive it.

The results of API-driven A/B test might look something like this:

This pseudo-code snippet reflects the results of a very simple test: after 5 emails, version A outperformed version B 4–1; the app that called this test presumably will then send future recipients the winning version A of its notification message.

But the principle easily scales to much more complex scenarios. Building automated A/B tests such as this into an app or service’s notifications framework can be very powerful. Pinterest, for example, continually feeds the results of A/B testing with a variety of engagement metrics into its machine-learning algorithms to ensure that every email it sends its users is as effective as possible.

UP NEXT: How to Build for the Cloud

APIs are essential to software development. But now they’re a key part of SaaS products’ go-to-market strategy, too. Here’s why it matters.

More Essential Email Resources

Develop your email industry expertise and master best practices with SparkPost’s email resources.

Email Tools for Developers

These free tools from the email experts at SparkPost make it easy to build and check your app’s emails.

read more

The Big Rewards of Email Deliverability

Learn how third-party data shows the deliverability difference between SparkPost and also-ran cloud service providers yields hard, bottom-line benefits.

read more

Inside the Email Deliverability Lab

This practical course is a great way to get started understanding email deliverability and how to measure email performance.

read more