A/B Testing in SparkPost
# A/B Testing in SparkPost
An A/B test, in summary, is a common method of comparing templates to see which one performs better. SparkPost provides the means to create new A/B tests and view completed results through our A/B Testing API.
What benefits do I get from using A/B testing?
An A/B test is a method of comparing templates to see which one performs better by splitting a given recipient list across the number of templates you are comparing. Performing these tests on a limited percentage of a recipient list will allow you to send to the winning template your remaining audience, which can help drive better engagement results for your campaigns.
Creation and review of A/B testing through the API
Please refer to the A/B testing API documentation for specifics on how to create and retrieve A/B tests.
Notes about A/B testing specifics:
- A “group” in the context of A/B testing is the combination of a template and the recipients assigned to receive that template.
- The winning template is automatically selected when the expiration time of the A/B test is reached. There are no manual controls to select a winning template.
- The winning template is chosen by best open rate across all groups. If there is a tie in open rate among templates being tested, a random winner will be selected from the templates that tied. A GET of the A/B test’s ID through the API will list the template winner.
- A minimum of 2 templates is required to run an A/B test. You may have up to a maximum of 10 templates in a single A/B test.
- A recipient list used in an A/B test can have a maximum of 2,000 recipients.
- The assignment of individual recipients into groups is randomized. You can pull which recipients were included in each group by performing a GET call with the A/B test’s ID to the API endpoint.
- The total number of recipients in the recipient list must be greater than the sum of all recipients assigned to test groups. There must be at least 1 remaining recipient not belonging to any test group.
- Templates must be published to be used in A/B testing. You cannot use draft templates.
- The published templates you are testing with must have engagement tracking turned on.
- Reporting by individual groups is available through both the message events API (individual message events) and the metrics API (aggregate statistics), and the UI.
- For the message events API, you must filter by both campaign ID and template ID to pull individual recipient data.
- To pull aggregate statistics by template performance, use the deliverability metrics by template as described in the metrics API link immediately above.
- In the UI, you can filter all reports using the search bar or clickable filters by campaign and template.
Best practices for A/B testing
- Test as large a sample as you can for more accurate results. If possible, use the max that SparkPost allows – 2,000 recipients.
- Test often for the best results – a single test may not be enough to provide a conclusive winner, as results may change over time.
- Only test one variable at a time for best results when using A/B testing. This typically means that you should only change one variable in your template such as:
- Subject line (this is often the testing variable in the majority of tests)
- CTA (call to action)
- Offer language/different offers
- Header image