How to Get Started with A/B Testing in SparkPost

Isaac Kim
Aug. 31, 2018 by Isaac Kim

Quick Refresher: A/B Testing

A/B testing emails is a common approach for determining which variations in content, subject lines, images, etc., drive more engagement (i.e., higher open and click rates) by recipients. Engagement is a key objective for a variety of emails, but most A/B testing tools aren’t well-suited to notifications and transactional use cases.

Good news, though! SparkPost has a new A/B Testing feature that’s been optimized for transactional email and other app-generated messages and notifications. My colleague Irina recently shared the details in her preview of the feature.

Now, we’ve released A/B Testing, and we’d like you to try it! So don’t be surprised when you see a new option next time you log into SparkPost.

I’ve put together a quick tutorial on how to get started with A/B testing emails in SparkPost in three steps.

A/B Testing in 3… 2… 1.

Let’s get started with SparkPost’s A/B Testing feature. In a nutshell, our implementation is built around the core product features you already use: templates and transmissions. Here’s how it works.

Step 1. Create an A/B Test

First things first: let’s create a new test.

  • Once you’ve logged into SparkPost, click “A/B Testing” in the left navbar.
  • On the A/B Testing page, you’ll see a list of existing A/B tests, if any. Otherwise, you’ll see a blank slate. Let’s change that!
  • Let’s create a new A/B Test. Click on the button on the top right labeled “Create a New A/B Test”.
  • Give your test a name. You’ll note that we automatically suggest a new test ID for you as well, but you’re welcome to override that with any unique ID text you wish. Be sure to note the ID—you’ll need it when you issue a transmission API call later.
  • We’ll leave the subaccount field alone for this tutorial.
  • Select the template you’d like to use as the test’s default. You’ll see this field includes a nice autocomplete dropdown that suggests stored template names you’ve already created, so finding a default template couldn’t be easier.
  • Click the “Create New Test” button to save this new test and go to the next step.

Step 2. Refine and Schedule Your A/B Test

Here’s where we get into the fun part: picking the preferences and variants of your test to determine what drives higher engagement!

  • Start by selecting the date range over which you’d like to run the A/B test.
  • Next, pick between our Bayesian Mode or Learning Mode, including the confidence level. (You can learn more about these modes in our docs. And our data science team recently shared some detail on the Bayesian statistical model that underpins our A/B Testing functionality. Data is beautiful.)
  • As you scroll down, you’re given more control and options at the tip of your hands, including how you want to determine a winner.
  • At the bottom of the screen, you’ll be able to select the variant templates you want to use to compare against your default template.
  • Once you’ve made all your selections, scroll back up to the top and click on the “Finalize and Schedule Test” button in the top right corner.

Step 3. Call Your A/B Test During Email Transmissions

Now that you’ve created and scheduled your new A/B test, all that’s left to do is to call it when you inject messages during email transmissions by including ab_test_id rather than template_id in the content of the JSON object you submit to the Transmissions API.

Here’s an example of a JSON object you’d post to the Transmissions API endpoint:

And that’s it! Your A/B test will execute on messages injected with the ab_test_id over the time period and message volume you’ve selected.

You can view your test’s progress at any time by selecting “View Test” from the test’s “…” more actions popup menu on the A/B Testing Page.

At the bottom of the A/B Test detail screen, you’ll be able to view a comparison of Engagement Rate Achieved for your default and variant templates.

Nicely done.

Full A/B Testing Feature Documentation

We’ve updated our docs to cover the A/B testing feature and the various modes of A/B testing we support—as well as detail on configuring A/B Testing via the API, if you’re into that sort of thing. 🙂

Feedback Wanted!

SparkPost’s A/B Testing feature makes it easy to compare two or more templates to see which one performs best. But today’s just the start—and we need your help.

I’m the product manager for SparkPost’s A/B Testing feature, and I’d love to hear from you about what you think of it. In fact, I’m going to make it really easy for you to help us shape the future of this feature:

  • Schedule a quick 15 minute phone call with me via Calend.ly, and I’ll send you a $50 Amazon gift card for your trouble.
  • Afterwards, if you complete an A/B test and then give me feedback on how it went (hearing about how your engagement improved and which template won would be fantastic too!), I’ll give you another $50. That’s $100 up for grabs.

If free money isn’t your thing (hey, no judgement), but you still want to voice your opinion about A/B tests, tweet us or share your thoughts in our Community Slack.

—Isaac
Technical Product Manager

 

Related Content

New Feature: A/B Testing API for Transactional Email and Notifications

Transactional email has unique testing requirements. SparkPost’s A/B Testing API is optimized for testing real-time use cases like app notifications.

read more

The Bayesian Logic in SparkPost’s A/B Testing API

Our new API feature for A/B testing transactional emails uses a statistical algorithm, Bayesian Logic, for picking the winning variant of the message.

read more

A/B Testing Email: Do's and Don'ts

A/B testing email can be one of the most profitable and insightful things a company can do to optimize conversions. But don't make these six mistakes.

read more

Get started and start sending

Try SparkPost and see how easy it is to deliver your app’s email on time and to the inbox.

Try Free

Send this to a friend