A/B testing is often used to test variations to web pages and applications in an effort to increase user engagement. But A/B testing can also be used to good effect in the world of email, and the same concepts apply.
A/B testing (also known as split testing) is a randomized experiment with two variants (A and B) to determine which produces the best results. A/B testing is one of the easiest and most effective ways to optimize your email engagement. The goal of A/B testing email is to improve open and click-through rates by identifying phrases, formats, or techniques that subscribers are more likely to respond to. Once identified, they can be combined to significantly improve the success and profitability of your email campaigns.
When A/B testing bulk email, choose one element to test at a time, such as:
- Subject line – Try variations of the same message.
- From name – Determine whether subscribers are more likely to open your email if it comes from a person versus the company or organization.
- Personalization – Address emails to either the subscriber’s first name or their last name (“Mrs. Jones”)
- Delivery dates/times – Send emails at different dates/times to determine when subscribers are most likely to open your email.
- Call to action – Vary the language used to entice readers to click.
- Hyperlinks – Determine whether subscribers are more likely to click on a linked image or linked text.
- Email design – Change the layout of your emails, testing one versus two columns, for example.
When deciding what to test first, consider where you are having the most difficulty. It doesn’t make sense to focus on click-through rates if your open rate is low. If that’s the case, then testing subject lines, ‘from’ name, and delivery dates/times can help you determine how to increase open rates. Regardless of which element you choose, be careful to conduct one test at a time. For example, don’t A/B test a landing page at the same time you’re A/B testing an email that directs to that landing page. You won’t know which element drove an increase (or decrease) in clicks.
It is also important, whenever possible, to test your entire list. This, along with randomly and equally splitting the list will help ensure accurate results. Both variations must also run at the same time, otherwise you are introducing another element (day/time) that can influence your results.
To learn more about SparkPost’s real-time analytics, which you can use to support A/B testing email, see https://blog.sparkpost.com/real-time-analytics-digging-into-email-metrics/.
You’re developing a new web service or mobile app, and you need a way to reach out to new users when they sign up, and then keep notifying and engaging users over time. “I need some kind of mass email capability,” you might be thinking. And you’re right – to an extent.
In the email world, mass email or bulk email refers to a very specific set of capabilities and techniques; namely, the ability to send a very large number of emails all at the same time, often in the context of a marketing campaign. Mass email can be sent via standalone software (like Momentum) or through email services operated by third-party vendors where customers pay on a “per send” or a CPM (cost per mille [thousand]) basis. SparkPost generally falls into the second category, and while it certainly has the power to handle mass email, it’s primarily designed for transactional email, meaning messages that are initiated by the action of the recipient or in response to a significant event (“You have a new friend request.”).
It’s an important distinction. Mass email services provided by Email Service Providers are popular because they often provide better reliability and performance than open source MTA platforms and eliminate the need for customers to deal with technical support issues. But most are designed for marketers. SparkPost is different. It was built with an “API First” approach, using test-driven development, so the services and application are highly stable and can easily support enterprise-level production usage.
Another difference is our approach to data and analytics. Most mass email services, being designed with direct marketers in mind, tend to emphasize campaign performance metrics. Click-to-open rate was X%, conversions were Y%, and you don’t get this data for a few days. SparkPost takes a different approach for your transactional email. Where many ESPs de-emphasize underlying disposition email data – ISP blocks, delays, spam hits – we think immediate, comprehensive visibility into disposition data send is key to successful messaging programs.
Another point to consider is that the data and results you get from transactional email are important in different ways than results are for bulk messages. Take password reset emails: if a user forgets their password and needs a reset link, they don’t want to wait for it and may get frustrated if they don’t receive the reset link quickly. Monitoring latency on these messages is important, and may be difficult to do with other services.
Consider also that more and more companies today are doing sophisticated analyses of their customers. Email engagement information can tell you a lot about someone. Any email service you consider using should have APIs that make it easy to get email information into your own BI or big data systems where you can mix it in with all the other data you analyze.
So, the point is that it’s important to understand what capabilities are essential and what you really need to measure, and not get blinded by bells and whistles that aren’t relevant to you. We built SparkPost so that developers could get up and running with email really fast, and easily scale as traffic volumes grow.