How to Measure Delivery of Email Notifications

Mike Hillyer
Dec. 13, 2017 by Mike Hillyer

Delivery Metrics: Why Measuring Email Delivery is so Important

Recently, I described how to measure the latency of email notifications, a key metric for measuring how effectively an app can generate and send notifications and similar messages.

In today’s post, I will look at the next stage of an email notification’s journey: delivery. Email delivery sounds like a simple matter, but ensuring your app’s emails actually reach your users’ inboxes is surprisingly complicated. Understanding how to measure email delivery is an important part of measuring the overall effectiveness of your emails for driving user engagement with your app.

Key numbers to analyze at this stage include:

  • Acceptance rate
  • Number (and types) of failures
  • Inbox rate

Acceptance Rate

After your message is submitted, it is queued by the sending MTA and then the sending MTA will attempt to deliver the message to the receiving MTA. Typically this next hop will be to the edge of the ISP’s network (for example, from your MTA to an MTA server at the edge of Gmail’s network). The remote server may accept the message, but first it will run a series of checks; looking to see whether your server is on a blacklist, whether the recipient’s email address is valid and active, and even some initial content filtering.

If all of these tests pass, your message will be accepted by the remote MTA, and the local MTA will record that the message was accepted. Not all mail will be accepted, and there are a number of reasons why remote MTAs reject messages (which will be covered below), but the key metric at this stage in the life of the message is the accept rate.

When senders talk about their deliverability, they are usually referring to their accept rate: the percentage of the email message they submitted that was ultimately accepted by the remote MTA. On average, 72% of all email messages are not delivered, so having a high accept rate is critical to the success of your application. Non-delivery of your messages leads to user dissatisfaction, slowed or blocked user onboarding, support costs, and churn.

Temporary Failures (Soft Bounces)

Messages are not always accepted (or rejected) by the remote MTA on the first attempt. The remote MTA can respond with a temporary failure (commonly known as a soft bounce) for a number of reasons, causing the sending MTA to re-queue the message and wait before attempting another delivery. A temporary failure indicates something is wrong with the mail systems, but not the recipient address.

The original reason for temporary failures is still a common one: the remote MTA gets overwhelmed and issues temporary failures to incoming messages in order to catch up with incoming traffic. This is not unusual and does not require any further intervention on your part, the sending MTA will wait and retry.

Remote MTAs can also send temporary failure responses while they are being shut down, reconfigured, and for other technical reasons, and the same response applies: the sending MTA needs to wait and try again later.

Delays caused by temporary failures can also be a result of your sending reputation. When a remote MTA doesn’t consider your sending reputation to be trustworthy, it can greylist your messages so that it can see whether your sending MTA will follow best practices and try again. In addition, some ISPs will enforce rate limits on sending MTAs based on how reputable they consider your sending to be: when you exceed the hourly limit, they will then start responding with temporary failure messages for the remainder of the hour.

While the right course of action for the sending MTA is to wait and try again, senders need to pay attention to reputation-based temporary failures and review their sending practices and take steps to improve their sending reputation. Note that sometimes you can get this response on only part of your email traffic to a give ISP, in this case they will likely be letting through a small sample of your messages to see how their users respond, and then make a decision on the remainder of your messages later.

Permanent Failures (Hard Bounces)

The third potential response a remote MTA can issue is a permanent failure (also called hard bounce). Remote MTAs issue permanent failures to tell the sending MTA to cease attempting to deliver a message.

The most common permanent failure reason is an invalid recipient address. This can happen for a number of reasons: a new user may typo their email address, an established user may shut down their email account in favor of a new one, a malicious third-party may enter an invalid email address into your signup forms to trigger excess hard bounces (this one can be common when an organization has rivals, such as a political party). The ISPs are generally tolerant of these bounces, but they will be watching to see whether your application pays attention and prevents future sends to addresses identified as being invalid. It’s key that your applications process these bounces and stop sending to the recipients as soon as possible (if your sending platform supports webhooks for bounce data, this is a good use for them).

Some permanent failures, such as the invalid recipient bounce mentioned above, are signals that you should not attempt sending to a given recipient again. Other permanent failures can indicate that while the current message should no longer be attempted, future messages may have a different outcome.

One common permanent failure response is a spam block. These can be related to either the source or the content of the message you send, and either type needs to be investigated and acted on promptly. If you get a surge in message related spam blocks, you need to review what you are sending, if you get a source related spam block, you need to review your content and your sending practices.

Additionally, hard bounces can occur for reasons such as the recipient’s mailbox being full, the message containing an attachment that the ISP considers too large, and for other reasons that may or may not support future attempts at sending. The key is to categorize your hard bounce responses and act on a granular basis, removing bad addresses, reviewing spam blocks, adjusting attachment sizes, and pausing sends to users with full mailboxes.

Inbox Rate

Message acceptance (or failure) is not the end of your message’s journey. Once the MTA at the ISPs has accepted the message from the sending MTA, it will often run additional checks, perhaps perform deeper scanning, or even store the messages temporarily while it monitors how a subset of recipients react to your message before releasing the remainder of your messages, or putting them in the spam folder.

All of this means that you need to look further than the accept rate to see whether your messages have successfully reached your user. The ISPs do not provide data on whether messages reach the inbox, the spam folder, or are dropped after delivery, so there is no definitive way to know whether each of your messages was successfully delivered.

Instead, senders can use sampling to determine how many of their messages reached the inbox at the various ISPs. Like measuring real latency, the only way to get an accurate inbox rate is to look at the inbox itself. Some senders will configure mailboxes at the major ISPs they are interested in to see whether test messages deliver regularly, but many will take advantage of a third-party provider such as 250ok and their large networks of monitored mailboxes.

While it’s impossible to get a precise number for your inbox rate, you can use the data from sampling for trending information. If you see dips in your inbox rate over time, you need to review what you’re sending and who you’re sending it to in order to identify any poor sending practices you may have recently adopted.

Up Next: Opens, Clicks, and User Engagement

Message delivery and inbox placement is a crucial part of any product’s email performance. But what comes next might be even more important: did your users actually take action on the email you sent? Stay tuned for our final installment: understanding how to measure user engagement with notifications and other product emails.

—Mike

Related Content

How to Measure SaaS Product Email Latency

Email latency is a critical component of product email performance. Here's how to measure email latency and ensure your emails are being delivered on time.

read more

Email Deliverability Crash Course: Message Acceptance, Seed Lists & Panel Data

Email deliverability is a fundamental metric. But what does it really mean? Here’s a crash course in message acceptance rates, seed lists, and panel data.

read more

Are you Realizing the Big Rewards of Email Deliverability?

We did the math, learn how email deliverability is calculated, how it impacts your bottom line, and how to improve it for better engagement rates and ROI.

read more

Get started and start sending

Try SparkPost and see how easy it is to deliver your app’s email on time and to the inbox.

Try Free

Send this to a friend