Delivery Metrics: Why Measuring Email Delivery is so Important

Recently, I described how to measure the latency of email notifications, a key metric for measuring how effectively an app can generate and send notifications and similar messages.

In today’s post, I will look at the next stage of an email notification’s journey: delivery. Email delivery sounds like a simple matter, but ensuring your app’s emails actually reach your users’ inboxes is surprisingly complicated. Understanding how to measure email delivery is an important part of measuring the overall effectiveness of your emails for driving user engagement with your app.

Key numbers to analyze at this stage include:

  • Acceptance rate
  • Number (and types) of failures
  • Inbox rate

Acceptance Rate

After your message is submitted, it is queued by the sending MTA and then the sending MTA will attempt to deliver the message to the receiving MTA. Typically this next hop will be to the edge of the ISP’s network (for example, from your MTA to an MTA server at the edge of Gmail’s network). The remote server may accept the message, but first it will run a series of checks; looking to see whether your server is on a blacklist, whether the recipient’s email address is valid and active, and even some initial content filtering.

If all of these tests pass, your message will be accepted by the remote MTA, and the local MTA will record that the message was accepted. Not all mail will be accepted, and there are a number of reasons why remote MTAs reject messages (which will be covered below), but the key metric at this stage in the life of the message is the accept rate.

When senders talk about their deliverability, they are usually referring to their accept rate: the percentage of the email message they submitted that was ultimately accepted by the remote MTA. On average, 72% of all email messages are not delivered, so having a high accept rate is critical to the success of your application. Non-delivery of your messages leads to user dissatisfaction, slowed or blocked user onboarding, support costs, and churn.

Temporary Failures (Soft Bounces)

Messages are not always accepted (or rejected) by the remote MTA on the first attempt. The remote MTA can respond with a temporary failure (commonly known as a soft bounce) for a number of reasons, causing the sending MTA to re-queue the message and wait before attempting another delivery. A temporary failure indicates something is wrong with the mail systems, but not the recipient address.

The original reason for temporary failures is still a common one: the remote MTA gets overwhelmed and issues temporary failures to incoming messages in order to catch up with incoming traffic. This is not unusual and does not require any further intervention on your part, the sending MTA will wait and retry.

Remote MTAs can also send temporary failure responses while they are being shut down, reconfigured, and for other technical reasons, and the same response applies: the sending MTA needs to wait and try again later.

Delays caused by temporary failures can also be a result of your sending reputation. When a remote MTA doesn’t consider your sending reputation to be trustworthy, it can greylist your messages so that it can see whether your sending MTA will follow best practices and try again. In addition, some ISPs will enforce rate limits on sending MTAs based on how reputable they consider your sending to be: when you exceed the hourly limit, they will then start responding with temporary failure messages for the remainder of the hour.

While the right course of action for the sending MTA is to wait and try again, senders need to pay attention to reputation-based temporary failures and review their sending practices and take steps to improve their sending reputation. Note that sometimes you can get this response on only part of your email traffic to a give ISP, in this case they will likely be letting through a small sample of your messages to see how their users respond, and then make a decision on the remainder of your messages later.

Permanent Failures (Hard Bounces)

The third potential response a remote MTA can issue is a permanent failure (also called hard bounce). Remote MTAs issue permanent failures to tell the sending MTA to cease attempting to deliver a message.

The most common permanent failure reason is an invalid recipient address. This can happen for a number of reasons: a new user may typo their email address, an established user may shut down their email account in favor of a new one, a malicious third-party may enter an invalid email address into your signup forms to trigger excess hard bounces (this one can be common when an organization has rivals, such as a political party). The ISPs are generally tolerant of these bounces, but they will be watching to see whether your application pays attention and prevents future sends to addresses identified as being invalid. It’s key that your applications process these bounces and stop sending to the recipients as soon as possible (if your sending platform supports webhooks for bounce data, this is a good use for them).

Some permanent failures, such as the invalid recipient bounce mentioned above, are signals that you should not attempt sending to a given recipient again. Other permanent failures can indicate that while the current message should no longer be attempted, future messages may have a different outcome.

One common permanent failure response is a spam block. These can be related to either the source or the content of the message you send, and either type needs to be investigated and acted on promptly. If you get a surge in message related spam blocks, you need to review what you are sending, if you get a source related spam block, you need to review your content and your sending practices.

Additionally, hard bounces can occur for reasons such as the recipient’s mailbox being full, the message containing an attachment that the ISP considers too large, and for other reasons that may or may not support future attempts at sending. The key is to categorize your hard bounce responses and act on a granular basis, removing bad addresses, reviewing spam blocks, adjusting attachment sizes, and pausing sends to users with full mailboxes.

Inbox Rate

Message acceptance (or failure) is not the end of your message’s journey. Once the MTA at the ISPs has accepted the message from the sending MTA, it will often run additional checks, perhaps perform deeper scanning, or even store the messages temporarily while it monitors how a subset of recipients react to your message before releasing the remainder of your messages, or putting them in the spam folder.

All of this means that you need to look further than the accept rate to see whether your messages have successfully reached your user. The ISPs do not provide data on whether messages reach the inbox, the spam folder, or are dropped after delivery, so there is no definitive way to know whether each of your messages was successfully delivered.

Instead, senders can use sampling to determine how many of their messages reached the inbox at the various ISPs. Like measuring real latency, the only way to get an accurate inbox rate is to look at the inbox itself. Some senders will configure mailboxes at the major ISPs they are interested in to see whether test messages deliver regularly, but many will take advantage of a third-party provider such as 250ok and their large networks of monitored mailboxes.

While it’s impossible to get a precise number for your inbox rate, you can use the data from sampling for trending information. If you see dips in your inbox rate over time, you need to review what you’re sending and who you’re sending it to in order to identify any poor sending practices you may have recently adopted.

Up Next: Opens, Clicks, and User Engagement

Message delivery and inbox placement is a crucial part of any product’s email performance. But what comes next might be even more important: did your users actually take action on the email you sent? Stay tuned for our final installment: understanding how to measure user engagement with notifications and other product emails.

—Mike

Every individual of an organization is responsible for its growth, from the email marketer who creates the campaigns to the engineer who manages the systems that send them. The popularity of “growth marketing” strategy illustrates this desire to accelerate business, but what is growth marketing and how can it be applied to improve email performance?

Having a growth mindset, rather than one that is fixed, means understanding that it is always possible to better understand your customers. People’s preferences are not set in stone, so it’s critical to possess a spirit of self-improvement around your tools and tactics.

In essence, growth marketing is all about coordinating all the channels at your disposal to increase user engagement, and email has been proven time and again to be the best digital channel for ROI.

Much of what is discussed in business is user acquisition, the strategy of gaining new members of your audience. But once an email address has been obtained, it’s up to you to keep users interested.

User Engagement Benchmarks growth marketing

Regardless of your role, you can practice growth marketing by improving these three metrics to increase user engagement.

  1. Open Rate – Obviously, emails can’t be opened if they’re delivered to the wrong place. Technical operations teams and email marketers alike understand that deliverability isn’t just about email acceptance, but also about the right inbox placement. Get started on the right foot with customers by ensuring that your content doesn’t land in the promotions tab, or worse, in the spam folder.
  2. Click-to-Open Rate (CTO) – Unlike click-through rate, which measures the total times a link was clicked, CTO more effectively determines user engagement by only calculating the number of clicks from opened emails. According to industry benchmarks, email pros should expect a 25-40 percent click-to-open rate.
  3. Unsubscribe Rate – Though few of us enjoy seeing unsubscribes to our emails, they do serve as a sort of “tough love” that helps improve overall list quality, as well as a learning opportunity for fine-tuning content. Setting up an email preference center is the key to reducing your unsubscribe rate, which should not exceed 0.4 percent.

These metrics are key to understanding how to communicate with your users, but there are many more that must be considered.

Find out how to implement high-performing campaigns in The Growth Marketer’s Guide to Email Metrics, co-produced by SparkPost and Iterable.

Want to know how Iterable and SparkPost work together to support the email marketing needs of B2C companies? Visit iterable.com to learn more or reach out to us to request a demo. We look forward to hearing from you!

====

—Alyssa Jarrett, Content Marketing Manager at Iterable

Alyssa Jarrett, Iterable

Zillions Guide Blog Footer

 

Introducing SparkPost FEELs

As Q1 winds down we wanted to release some exciting previews of the new metrics coming in Q2.

Over the past few months many of you have said, “What we need from SparkPost is more metrics! You give us so much awesome information about the email delivery process, but what we want to know is how do our customers feel about it? What did I do wrong? How amazing am I?”

Well you asked for it, and we listened!

For weeks we organized working committees, hosted focus groups, created user and marketing personas, and spent lots of time looking up agile development buzzwords to seem busy during standup meetings. We scoured the feedback from our SparkPost Community Slack team and pored over the many user-submitted feature requests. Taking that data, we leveraged machine learning, artificial intelligence, and other buzzwords we saw on reddit to discover the most useful and most requested metrics! We asynchronously polled several leading email clients to get real-world examples of user interaction and then applied deeply artificial machine learning algorithms to the resulting data store. The resulting metrics are insightful, deep, and synergistically aligned across most cross-functional teams.

Our New Social Engagement Metrics!

new feels metrics reports

Number of Forgotten or Missing Attachments

Forgotten Attachments

We’ve all done it.  You craft that masterful message to go out, filled with numerous references to the documents attached.  Documents that painstakingly written and designed by hard working people.  Except they aren’t there. Similar to how Google analyzes your content for potentially forgotten attachments, we can reliably track how many times you meant to attach something but didn’t… and we let everyone know via this handy counter.

Number of Novels Written

Novels

Being succinct is overrated! Why skip on the flourish or the gritty details? Walls of text are important ways to convey deep thoughts and are a clear sign of enhanced intelligence, so throwing around your “Number of Novels” stat at that next meetup will definitely help you make new friends and meaningful business connections.

Number of Emojis Used

Emojis

Text based communication is known for not being great at conveying emotion, but thanks to the introduction of emoticons and emojis, we can show just how we want our message to be interpreted. By referencing the number of emoticons/emojis used, you can tell just how clearly you communicated the emotional intent of your message.

Number of Awkward Emails You Shouldn’t Have Sent

Awkward

Email can be awkward. Sometimes it’s a mis-targeted email sent to the wrong list with the wrong information. Sometimes it includes a message from your CEO to your users but you misspell her name. And sometimes it’s some unintentionally inappropriate wording that made it into the subject line of your newsletter. By analyzing the opens, the forwards, and the responses to your message, we can show you just how many times you may have evoked a feeling of, “Uhh… what?” from your email recipients.

Number of Eye rolls and Sighs Per Open

Eye Rolls

Closely related to the Number of Awkward Emails Sent, eye rolls and sighs provide deep insight into your emotional engagement, message targeting, and how effective your dad jokes are.

Top Performing Clickbait Subject Lines

Clickbait

If you’ve ever gone to the trouble of carefully crafting a subject line that sounds amazing but doesn’t relate to your content whatsoever, someone should acknowledge how amazing you are! We track the g.p.o.’s (groans per open), r.g.m.’s (Ryan Gosling mentions), and the number of opens per message marked as spam to bring you the most cringe-worthy data possible.

How to Use The SparkPost FEELs

The new SparkPost FEELs metrics will provide you with the most relevant, non-alternative facts about how your email recipients feel about your mailings. They also provide vital insight into your own team, letting you know just who the problem is without tedious things like human interaction getting in the way. Who has time for tact and courtesy when we have all these DATA?!

Feedback

We would love to get your feedback on these new metrics before they go live in Q2. So, please reach out here, on our Community Slack team, or via Twitter and let us know what you think!

-Jesse

 

Email Marketing News Digest

The latest Google Analytics and Adwords updates have a focus on multi-channel attribution and providing marketers with better clarity on what’s driving conversions. These features and the increasing emphasis on both cross-channel and multi-channel marketing strategies highlight the increasing complexity of getting the attention of customers.

The Key to the Inbox is Subscriber Love

The key to the inbox? Strong permissions and relevant emails. ISPs measure user feedback based on how many people are hitting the “this is spam” or “this is not spam” button. Here are four factors to examine when trying to determine why you are not making it into the inbox:

  • Permissions
  • Relevancy
  • Frequency
  • Behavioral Targeting

Of course if you want to hear directly from the ISPs themselves and pepper them with questions, there’s always our all star ISP panel at Interact 2013 – The Digital Messaging Industry Conference. If you have a burning question on deliverability in your mind, register today!

Email design change driving a 61% click increase

Or is it? The DMA recently tested two different email designs in a split test and found that the skinny design had 61% more clicks. But wait – there’s more to the story. While on the surface the mobile optimized version seemed like a clear winner, it seemed that the copy and call-to-action from one particular link was driving the increase since the copy was not standardized between the 2 versions. In a second test where the copy was the same, there was no discernable click rate response difference.

Email Split Testing

Boom Times Still On for Emailers

Experian’s Quarterly Email Benchmark Study shows that email volume in the seond quarter was 17.9% greater than in Q2 2012. Unique open rates rose 7.2%, while unique click rates declined slightly by 0.2%. People who opened only on their mobile device averaged 10 cents per transaction while those that opened on mobile with another device averaged $1.28 per transaction. The results of the study present much food for thought when it comes to multichannel attribution. In the words of Bill Tancer, GM of global research at Experian Marketing Services,

“The path to the consumer is getting increasingly complex… We’re just now at the tip of the iceberg in knowing how to get from the initial contact to the end of the transaction.”

4 Easy Questions: Understanding Engagement to Survive and Thrive in Email and Beyond

How do you define engagement? A one-off spike from a seasonal campaign or ongoing long term interaction from subscribers? Margaret Farmakis from Return Path says it should be the latter. Here are four important questions to ask when evaluating how engagement applies to your email program:

  1. How do you define engagement? Goals need to be clearly stated and the difference between an engaged and disengaged subscriber should be clear.
  2. How do you measure engagement? Define metrics such as those from social, user-generated feedback and those from surveys as well.
  3. How do your engagement metrics compare with the competition?
  4. What is your strategy for improving engagement?

Email: Wanted Dead or Alive [Infographic]

Email Wanted Dead or Alive

Find out more about cross-channel banking and how to make shifting customer behavior work for your bank in our webinar!

Cross Channel Banking Webinar