A few days ago, my colleague Dan looked at how email headers can tell you a lot about where a message came from, the path it took to an inbox, and even whether it actually was sent by who it claims to be from.

But the information contained in headers really is just a start. There’s more that can be measured about the journey of an email—and how well it’s working at the job you’re giving it.

Understanding Product Email Performance

Let’s look at a typical SaaS application email. What’s the basic reason your app is sending an email? It’s because your user has done something, and you need them to take an action to complete the task.

Account signup is a common example:

  1. A user signs up for an account
  2. You need the user to confirm her or his email address to complete the signup

Sounds pretty straightforward, right? Certainly, making sure that process works is essential to keeping your user happy (and not churning). Whether you’re a product manager or on the developer and devops side, making sure emails like this are working is an important part of measuring and managing your product’s overall performance.

Behind the scenes, though, the path that email takes is not as simple as it might seem. Most teams building SaaS products don’t have visibility into what happens once an email like this account confirmation message is sent.

With this article, I’m beginning a series that looks at the life of an account confirmation email, the stages it passes through on its way to the user’s inbox, and how data can be used to understand its performance at each stage.

Generating and Submitting a Product Email

The first step in our message’s journey is the generation and initial submission of the message. Depending on your application, this could happen in one of several ways:

  • Your app generates a fully-formed email and then submits it via Simple Mail Transfer Protocol (SMTP) to an external mail server for transmission.
  • Your app generates the content of a message and uses a simple mail sending function with a local SMTP server on your application host.
  • Or you can use an API provided by a dedicated email delivery service (in shorthand, an email API). This approach is the most scalable, as it allows your app to submit just the relevant data like name and address, while the specialized service handles message generation and delivery.

In any case, the email message is submitted to a Message Transfer Agent (MTA)—i.e., a mail server—that queues the message and takes responsibility for its delivery to the next hop in your message’s journey.

Why Latency Is a Key Metric for Product Emails

Measuring performance at this stage focuses on two metrics, injection count and latency.

The injection count is how many messages you submitted (injected) to the mail server for a given period of time. By itself this count won’t mean much, but it’s a good baseline against which to measure other data.

On the other hand, latency is a critical metric; when you send messages such as signup confirmations, your user’s experience depends on the timely delivery of the message. That’s especially important for something like an account confirmation email—the longer the message is delayed, the lower the likelihood the user will complete the registration and use your product.

If your message assembly and submission system has latency, you may want to review your code to see if it can be made more efficient, in addition, using an email API service can go a long ways towards improving performance, as it offloads the message generation from your own systems.

Typically, an app will only need to gather the recipient name, email, and other small amounts of personalization data and submit it to the API. The service then takes care of assembling and personalizing the messages and queuing them for delivery.

How to Measure Email Delivery Latency

Another component of email latency is how long it takes for your mail server (MTA) to deliver the message to its next stop, typically the inbound mail server operated by your user’s ISP. (Gmail or iCloud are typical examples.)

At this stage, some factors are out of your control, but latency here is a good indication of the overall quality of your email infrastructure. It’s a function of your infrastructure’s technical architecture and performance as well as its ability to manage delivery to ISPs. Both of these areas are where a high-performing email delivery service typically exhibits a major advantage over in-house email systems.

If your systems are slow at processing messages, or at managing queues, or are simply overworked, you’ll see latency of a minute or more. Efficient systems should be operating with a latency of less than ten seconds on average. There will always be some variation, as is common with all complex systems, but it’s the long-term trends that should concern you.

When you see latency numbers that are consistently high, you need to look at the underlying infrastructure: are there too many systems in the middle of your sending? Even your message generation process could be at fault: pickup directory systems can be relatively straightforward to build, but they are often based on scheduled pickups, which can introduce a long wait for a message submitted just after the last pass of the reader.

Latency after the first attempt is an indication of your reputation with the ISPs. Often an ISP will reject your message on the first attempt as a form of greylisting, refusing the first attempt at delivery to see if your MTA makes a second attempt (many spam bots will try once and move on). This causes the message to be queued and deferred temporarily for a subsequent attempt. The longer it takes to deliver a multi-attempt message, the more likely your sending reputation needs improvement.

How to Measure Latency by Hand

If you don’t have analytics in a dashboard for latency data, you’ll have to gather it yourself. Setting up scheduled jobs that send messages through your infrastructure to dedicated mailboxes that you maintain on the major ISPs, then retrieve the messages and check the timestamps of the first Received header and the Date header to determine the latency (note in the example below the time zones must be accounted for when calculating latency).

Email latency is a very important part of product email performance, but it’s just the start. Stay tuned for our next installment: understanding message delivery and what happens when a product email is delayed or never arrives.


See how @Zillow grew their email engagement by 161% w/ @SparkPost & @Iterable Click to Tweet
If it’s one thing I’ve learned in the past two years by working at SparkPost, it’s that there’s a clear difference between ‘delivered’ and ‘deliverability’ when talking about email. An email that’s delivered simply means an accepted transmission by the ISP has occurred. However, when measuring true email deliverability, we’re talking about measuring the success of getting into the Inbox vs. the spam/junk folder.

Email deliverability is a very important part of the equation for growth marketers wanting to crack the nut on email marketing. However, it’s not everything. Metrics such as open, click-to-open, and forwards are also important. Why? Because an engaged user is an ambassador for your brand. When we recently sat down with our customer, Zillow, they expressed interest in improving user engagement in their email marketing. Through regular conversations with their Technical Account Manager (TAM), we suggested they speak to our partner, Iterable. Iterable has been able to guide them in achieving better segmentation and increased open and click-to-open rates by 161% and 18%, respectively. Being able to test, scale, and nurture customers in real-time is essential to growth for Zillow and all companies.

In our recorded webinar , Beyond the Click: Essential Email Metrics that Drive User Engagement, please join us, Iterable and Zillow as we discuss Zillow’s unique approach to email.

In this upcoming webinar we’ll talk with Zillow to learn:

  • How deliverability helps increase user engagement
  • The approach Zillow took to increase their open rate by 161% and their click-to-open by 18%
  • Which email metrics to measure to boost your ROI
  • Why Zillow is able to make quick decisions based on real-time data

Get the recorded webcast today for Beyond the Click: Essential Email Metrics that Drive User Engagement.

In the meantime, you can read Iterable’s guest blog on how growth marketing can help increase user engagement by improving three email metrics.

~ Tracy Sestili
VP, Growth Marketing

9 Things ISPs Really Want Email Marketers to Know

Many email marketers are mostly concerned with the open and click thru rates of their campaigns. However, that’s just scratching the surface of what savvy email marketers are looking at today.

Email Marketing Beyond The Click - SparkPost BlogYou may have heard a lot about email deliverability, but to the novice marketer, if it’s opened then it must have been delivered, right? Not quite (more on that below), but even so, there’s so much more to understanding email analytics than just open, click-through and bounces rates when you’re seeking to deliver the best customer experience and to give you the best return on your investment.

It’s easy to assume that once an ISP accepts an email transmission, it’s been delivered. Sure, that’s a crucial step. But genuine email deliverability is all about whether your message got into the inbox, not the spam or junk folder. To make matters more complex, there are cases where it can be accepted by the ISP but not necessarily delivered to the inbox because say, the ISP is throttling a connection when the amount of email being received is at too high a rate currently. Thus, even if the ISP shows a high “accepted” rate, your open rate could still be low. As a result you could waste hours A/B testing your email subject lines when the likely reality is that your message is going to the spam folder and you don’t even know it!

(By the way, an additional wrinkle for Gmail is that the Promotional and Social tabs are considered the Inbox unless the end-user does something with that message by explicitly marking it as spam or moving it to the junk folder. That’s an important qualification to consider when you assess inbox performance for Gmail users.)

But why, as a marketer, should you care about deliverability? Because deliverability is a cornerstone of your ability to generate revenue from your email.

Deliverability. Open Rates. Timing.

Even after a message has been delivered, it may not (yet) have been opened. It could be that the subject line didn’t peak the recipients curiosity enough  or the email was sent at an inopportune time of day.

Source: 2015 Email-Marketing Metrics Benchmark Study 2015 by Silverpop
Source: 2015 Email-Marketing Metrics Benchmark Study 2015 by Silverpop

And, of course, the percent opened is directly related to your click rate. (A user can’t click what he or she hasn’t opened!) If the email was opened, but not clicked, a number of factors could be at fault: it could be because you have the wrong product, the wrong promotion, or even poor timing: the offer you’re giving may not be relevant to the user today, even if it might be at a future point in time.

But before you toss your subject lines, remember that a low open rate percent might not be a direct indication that your subject line isn’t contextually meaningful enough, but it could be that subtle deliverability factors are in play. You might have authentication or throttling issues, or the ISP may never have delayed accepting your messages in the first place (soft bounce). For example, in the chart below, you can see that only 1/3 of messages are being accepted by this particular ISP on the first attempt, and although over 80% are being accepted by the fifth attempt. Of course, if your email is time-sensitive, this sort of delay is not the outcome you want to see!

Note: These charts are an example of what you’d see in SparkPost. The 1st/2nd/3rd attempt is a level of reporting not all ESPs offer.

Accepted Rates Low_final

On the contrary, you’d rather see an acceptance rate something like this:

Accepted Rates High Final

ISPs may start to throttle or even suspend your email delivery when they think you’re sending too much email too quickly or otherwise present characteristics that suggest spam. For example, if you send out an email to your list, and it’s been two hours and no one has opened it, it might not be because of timing, but rather because of throttling. Each ISP has different rules when it comes to throttling. If you find you are being throttled (and, by the way, SparkPost can actually tell you which ISP is blocking you), you can contact the ISP or work with your email delivery service. But without those expert deliverability services, as a marketer you won’t know which ISPs are blocking you, leaving it a guessing game and that’s leaving money on the table.

ISPs judge you based on your sender reputation.

Your message deliverability also is tied directly to your “sender reputation.” Your email reputation is a lot like a credit score – including a lot of factors you build up over time. ISPs attempt to predict if the recipient wants the message being considered for delivery through a series of tests. The first being an empirical test of does your message looks like spam? ISPs aggregate data based on what they generally know about spammers and then compare that to your message. For example, if you are a new sender (i.e., have no sending history from a domain or IP address), have a lot of spam clicks, or have bad address bounces from the same IP or sending domain—well, then your message might get throttled or worse, not sent at all.

ISPs also look at how people interact with your message. Do they usually open and click on emails sent by you? How long do they spend on the message? Do they put your email in a folder or forward it onward? These and many more factors are what ISPs look for to identify and block spammers.

By the way, although your own marketing is the main factor in determining engagement and content violations with ISPs, one reason you might consider going with an email delivery service is because of the in-house expertise they provide to handle the technical aspects such as whether or not your mail server is set up correctly. If your mail server isn’t secure, for example, other people can send email through it unbeknownst to you. Moreover, email delivery services (like SparkPost) have expert deliverability teams with proven credibility and existing relationships with anti-spam systems and ISPs. Their expertise makes it much easier to determine and solve the root cause of deliverability issues.

Context and Personalization.

“Contextual, predictive and personalization” are buzzwords you probably have been hearing about email marketing recently. The reason is simple: these concepts really do have a profound effect on your bottom line. For example, we’ve already heard this year from Experian Marketing Services that emails with personalized subject lines boost open average rates by 29.3%! Interestingly, this factor seems to vary widely across industries, as shown in the chart below.

Personalized Subject Lines
Source: Experian Marketing Services for Marketing Sherpa

Personalization doesn’t just help with email open rates however, it also impacts revenue. According to Experian, transaction rates with personalized messages were 49% higher and revenue per email was 73% higher ($0.15 compared to $0.08). Yet only about 35% of the brands surveyed are using personalization. 

Good housekeeping.

As a smart email marketer, you already know that list management housekeeping is a must. You’re savvy in the practice of scrubbing your list on a regular basis to remove hard bounces, unsubscribes and complaints. But good list hygiene goes beyond that to also removing emails that aren’t responding—in other words, pruning inactive segments. Inactive segments are detrimental to your bottom line because their relatively low engagement rate affects your sender reputation, which in turn impacts your deliverability, and which then further impacts your overall engagement (open, click, conversion). It’s a vicious cycle that has one solution: remember that list quality is much more important than list quantity. Don’t keep sending to users with poor engagement rates. It’s just not worth it!

Basic analytics tell you how many people unsubscribed from an email. What you don’t know is if they unsubscribed because you were bombarding them with email or if because you sent something that was contextually not relevant. Industry average benchmarks help you make predictive and smart decisions based on this data. (See the industry average unsubscribe chart below for some ideas.) But that’s when you should look more closely at your sign-up form to see if you’re delivering on your promise. For example, if you choose to have a sign-up form that promises 10% off their first order, but which isn’t up-front that they’ll also get weekly emails from you, then it shouldn’t be surprising if you have a large amount of people who unsubscribe because all they wanted was the discount to begin with, right?

The bottom line.

Lastly, the number one reason you should care more about deliverability is that it is directly tied to revenue. Email has an average ROI of $38 for each $1 spent. If the email doesn’t get delivered to the inbox, then you’re not making any money from it. Email metrics don’t get any simpler than that. 🙂

Industry Avg Unsubscribe Rate MailChimp_FINAL

If you liked this blog, you may also like: 10 Ways to Build Brand Trust & Loyalty Trough Your Transactional Email

10 Ways To Build Brand Trusty and Loyalty Through Transactional Email

Blog-Aside_AnalytisSparkPost offers a real-time analytics metrics dashboard with over 35 different metrics that we actively track for our customers. You can customize your SparkPost reporting interface to reflect any metrics that are important to you at a particular time. This is a powerful set of tools for evaluating how your email performs.

Other providers give you the basics of how many messages you sent, and then how many bounced, opened and clicked. But no other provider gives you as much data to troubleshoot issues. For example, if you sent 100 messages and 10 bounced, you might assume that 90 got delivered. But that’s rarely the case because of how the message traverses the providers’ systems. Some messages get rejected because of malformed email addresses. Others get backed up in long queues because another sender has a large mailing ahead of yours. But how would you know – until it’s too late to do anything about it – if your ESP doesn’t provide you with the data.

If I put on my operations hat on, the metrics I might care about are:

  • Targeted – how many did I intend to send? This is the basis for the rest of my comparison.
  • How many policy rejections did I have? In other words, how many messages did SparkPost not send out because of account limits, or because they hit the suppression list?
  • How many generation failures? If you are using the API to inject lists and templates, how many messages failed to generate because of missing or malformed template data?

I won’t go through every metric, but you can go through this exercise for your messages to get an accurate picture of which messages are actually getting out to the Internet and troubleshoot those that aren’t. Hint: you can scroll over each metric to see what it means or go here for a full list: https://support.sparkpost.com/customer/portal/articles/1929969-sparkpost-deliverability-metrics



Now, if I put on my deliverability hat on, I can start to diagnose which messages are making it to the recipient vs. which aren’t and why. Some metrics to look at are:

  • How many were accepted? This means how many were accepted by the receiving ISP.
  • How many were hard bounces? These are permanent failures – bad or nonexistent email addresses. If this is high, you might need to investigate how you collected those addresses.
  • How many were spam complaints? This happens when recipients click on “this is spam” in their inboxes. A large number here means you’re not getting permission or sending content that’s significantly different from what the recipient expected. A high number of bounces or spam complaints will likely land you in the spam folder next time.

You can get additional detail by going to the Bounces and Rejections reports.


Finally, if I put my Marketing hat on, I’m likely looking for engagement. In the Summary Report, I’m looking at Accepted, Rendered, Clicks, and Unique Clicks. Further detail is available in the Engagement Report, which provides you with a funnel chart that clearly indicates how many unique emails were attempted in the campaign, how many were accepted by the email providers, the number of unique emails that were opened, and the number of unique emails that were clicked.


In addition to looking at all this data in aggregate, SparkPost provides you with a number of ways of looking at this information. For example, you can filter by recipient domain to see where you had a high render rate or where, perhaps, you might need to make adjustments because of a high bounce rate with a particular provider. You can also filter by Campaign and Templates to hone in on what content is working and what isn’t.

SparkPost also gives you access to the raw data, letting you peer deeper into the numbers to see in precise detail how your recipients are engaging (you can download it as a CSV file or access it via webhooks).

I encourage you to dig into the numbers and see the wealth of information available to you in SparkPost, then think about tags that are specific to your business to learn even more about your email engagement.

To learn more:

SparkPost Deliverability Metrics https://support.sparkpost.com/customer/portal/articles/1929969-sparkpost-deliverability-metrics

Consuming SparkPost Webhook Events with Loggly – Part 1 of 3 https://www.sparkpost.com/blog/consuming-sparkpost-webhook-events-loggly-part-1-3



Weekly Email Marketing News Digest

This week’s email marketing series has a Valentine’s Day slant as marketers sift through emails to find out what’s trending in the season where love is in the air. They tell you what they’re in love with… or not.

From Bad First Date to Love: How ProFlowers’ Valentine’s Day Emails Won Me Over

Real life examples are often far more compelling than theoretical talk. Here’s the story of how ProFlowers first got personalization wrong… before they got it right. As they say, practice makes perfect! Here’s the version that got the thumbs up:


Love May Be in the Air, But It’s Not in the Inbox

There’s a little less love in the inbox this year as compared to last. On Feb 5 2013, there were only 11% Valentine’s Day themed emails. Last year on the same day it was 14%. On Feb 9, where you’d expect retailers to be ramping up their activities, the percentage fell further from 13% last year to 6% this year.


What’s increasing this year is the percentage of heart symbols used in email subject lines. On Feb 8, 8% of emails were using this tactic as compared to none last year.

Unsurprisingly, the words “free”, “shipping” and “save” were the three most used words in emails during the last few weeks.

UK DMers Advance New Engagement Metric

Will the open/reach metric be the next big thing in email marketing?

If a group of UK DMA (Direct Marketing Association) members have their way, that metric could become the reality for many emarketers.

The open/reach metric is defined as the number of people who have opened a marketer’s email at least once over a given period of time, from a quarter to a year, depending on the frequency of mailings.

Engagement is currently measured by open, clicks and conversions. The need to drive these metrics have led to marketers trimming their lists when it comes to ‘inactive’ subscribers – perhaps needlessly some claim. While this group of subscribers does not click frequently on mailings, they may well respond sporadically.

Do you remember us?

Our client, Dennis Dayman from Eloqua shares his experience with what he first thought was spam mail. It turned out to be a welcome email practicing one of the tenets of email marketing best practices – The Unsubscribe Option.


Drive Frequency. Drive Subscribers. Drive Marketing

“Anyone without an email address is the digital equivalent of a homeless person.” – Dela Quist, Founder & CEO, Alchemy Worx

Quist’s advice to the audience at the Email Evolution Conference was to organically grow email databases and amplify email frequency. Increasing the frequency of email is something that lies in the grey area as emarketers often fear being labelled a spammer.

Quist urges emarketers to give consumers a chance to get around to opening emails. How so? By giving them more opportunities to open emails with more emails. According to Quist, email frequency can drive engagement between a brand and customers. It also links email to other channels. However, it’s a good idea to understand consumer behavior on those channels before embarking on a cross-channel strategy to optimize the impact of the email.

And there you have it, The Valentine’s Day edition!

Weekly Email Marketing News Digest

Don’t you just love the holiday season? There’s always something new to celebrate. Now that Christmas is over, we’re all gearing up to ring in the new year in just four days!

And as always, at the beginning of a new year, it seems apt to reflect upon your hits and misses and overall individual or company performance. In line with that sentiment, here are a couple of tools to measure your annual email usage statistics.

Know Your Gmail Stats using Gmail Meter

With over 425 million active Gmail users in June 2012, Gmail is arguably one of the leading providers (if not THE leading provider) of free webmail. The Gmail Meter, developed by Romain Vialard, a Google Apps Script Top Contributor, provides users with basic email statistics to better understand their email usage. These include:

  • Volume Statistics
  • Daily Traffic
  • Traffic Pattern
  • Email Categories
  • Time Before First Response
  • Word Count
  • Thread Lengths
  • Top Senders & Recipients

Contactually Launches A Year-End Email Report Card That Goes Deeper Than Google’s Gmail Meter Report

The title says it all.

For a deeper more insightful look at your email usage statistics, get a personalised email report card from Contactually. Interesting metrics include:

  • Email Mood
  • Popular Subject Lines
  • Popular Words
  • Subject Line Length

With the amount of spam and direct marketing messages that one receives every day, it’s hard to see how someone can get a good grade on metrics such as unresponded emails and email totals. Nevertheless, it’s an entertaining and beautifully rendered report card for one’s own amusement – as well as an excellent way to gain brand awareness in the marketplace.

What Marketers Can Learn From a Gaming Company’s Email Tactics

When it comes to gaming, email is key to gamer retention, increasing customer lifetime value as well as brand recall. MarketingProf’s advice to gaming companies is to provide disclosure and value for gamers when requesting for their email. Here are the recommended guidelines for lead nurturing and enhanced customer service:

  • Request for an email early in the gaming process together with details on what players can expect to receive through email – tip & tricks, tutorials, discounts.
  • Provide players with a value-add of some kind immediately after they sign up through their email address. For example, when I signed up for Blizzard Entertainment’s popular MMORPG game World of Warcraft, I was immediately provided with a free downloadable online guide to the game through email.
  • Use of event triggered emails for more effective brand engagement instead of weekly updates. Has the player just conquered Diablo III? Invite them to start playing Starcraft with a free trial.
  • Use analytics to determine the playing habits of individual subscribers, followed by appropriate tailoring of the messages. For example, gamers who play frequently probably wouldn’t mind receiving weekly email updates from your company, but not those who have not played in a while. For these players, reactivation messages with gifts when they resume game play would be more effective.
  • Give players control of the messages they receive – let them set the terms and grounds for communication with a preference center. If they opted in to certain types of communication, they’ll be more likely to appreciate those emails from your brand.

The article also calls for branding consistency by ensuring players who have signed up for a specific game are getting emails from the game they signed up for and not the parent company. I have to say, I’m not sure I entirely agree with this one though. While I am a huge fan of brand consistency, I’ll admit that email cross-selling under the parent company’s name has worked on me. If I am interested in Dragons of Altantis from Kabam for example, I’m also likely to be interested in a 50% offer on Kingdoms of Camelot.

Now? Really? Spamhaus Blacklists Retailers for Typos

Major retailers like Gap and Gilt have found themselves on email blacklists this year – simply for typing errors in subscriber’s email addresses. Such errors typically occur during point-of-sale where customers opt for paperless receipts to be sent to their inbox. Up to 60% of a sender’s email may not make it into inboxes when placed on such blacklists. With the festive season being a prime opportunity for retailers to increase their sales and maximize customer lifetime value, the timing couldn’t be worse.  Its impact on these retailers could be akin to Google’s Florida algorithm update in 2003 right before the holiday season, which left many holiday retailers scrambling.

Why mobile email templates need to disappear

We’ve been highlighting stories about designing for the mobile recently, and with good reason. Consider the incredible escalation in the adoption of mobile devices as seen in John Pinson’s The Internet’s Growing Faster Than You Thought and the fact that 35% of all emails are now viewed on mobile devices.

It’s no longer possible for designers to design specifically for each and every new mobile device that is launched, yet it is imperative to ensure that a companies emails are optimized for the mobile. Hence, here are three tips for mobile design:

  • Use a one column design
  • Simplify call to actions due to lack of screen real estate
  • Use media queries

 Gaming emails aside, there’s plenty you can do with your transactional email to keep subscribers coming back for more. Optimize your transactional email today with the help of the Transactional Messaging Best Practices eBook.

Transactional Messaging Best Practices