Minds (and Rates), They Are a Changin’ !

Irina Doliov
Mar. 14, 2018 by Irina Doliov

A Quick Metrics Calculation Update

“I feel I change my mind all the time. And I sort of feel that’s your responsibility as a person, as a human being—to constantly be updating your positions on as many things as possible. And if you don’t contradict yourself on a regular basis, then you’re not thinking.” ― Malcolm Gladwell

A bit of context for the quote: Way back when, when SparkPost was just a gleam in this Product Manager’s eye, our team investigated how best to define metrics like “open rate”, “click rate,” and so on. That was a surprisingly ambiguous question, given that different ESPs defined them slightly differently, and internally we had as many strongly held opinions of what is “right” as we had people in the discussions. It got heated!

What we finally settled on was this: as an email infrastructure provider, we took the most conservative approach and defined all our rates to be a percentage of “Targeted”.

Fast forward 4+ years and many customer conversations later, and we have collectively changed our minds (hence the Malcolm Gladwell quote.) We’ve realized that calculating bounce rates using a denominator that includes messages that we never attempted to send (i.e. those that were suppressed) or calculating engagement rates using a denominator that includes messages that the ISP never accepted doesn’t make sense.

It also made apples-to-apples comparisons difficult: we’ve had numerous customers ask why our open rates seemed worse than what they experienced on other platforms, when the issue was just that other platforms calculate open rate using “Accepted” (a.k.a “Delivered”) as the denominator — a number that’s always going to be smaller than “Targeted”. Smaller denominator = larger rate.

As a result, we realized it makes sense to change how we calculate and report these rates. Later this month, the calculations for the following rates in the SparkPost user interface (UI) will change as follows. Rather than using “targeted” as the denominator, all metrics will reflect either “sent” or “accepted” as appropriate. You can hover over the metrics in the UI to get the definitions.

  • Accepted Rate = Accepted / Sent
  • All Bounce Rate = Bounces / Sent
  • Block Bounce Rate = Block Bounces / Sent
  • Hard Bounce Rate = Hard Bounces / Sent
  • Soft Bounce Rate = Soft Bounces / Sent
  • Undetermined Bounce Rate = Undetermined Bounces / Sent
  • Spam Complaint Rate = Spam Complaints / Accepted
  • Open Rate = Rendered / Accepted
  • Click-Through Rate = Clicks / Accepted

metrics calculation update

Note that this means the all the reported bounce rates will increase as well. This is not an indication that more messages are bouncing — only that we’re using a different denominator in the calculation.

Last point: none of your webhook events or Message Events API results will change since those report raw numbers, not rates. These rate changes will only be visible in the application UI and in the Deliverability Report Card you receive if you have an active account.

Let us know if you have questions or comments. We always love hearing from you.

—Irina
Sr. Lead Product Manager

Related Content

Product Announcement: Webhooks by Subaccount and Multiple Bounce Domains

Webhooks come with a firehose of data that is hard for ESPs to deal with. In this blog, the Product team announce 2 latest features that solve this problem.

read more

New Feature Announcement: Stored Templates by Subaccount

Introducing one of the most requested enhancements by both our enterprise customers and developer community, stored templates by subaccount.

read more

Benefits of Real-Time Analytics: Digging into Email Metrics

SparkPost offers a real-time analytics metrics dashboard with 35+ different metrics that we actively track for our customers. You can customize your SparkPost reporting interface to reflect any metrics ...

read more

Get started and start sending

Try SparkPost and see how easy it is to deliver your app’s email on time and to the inbox.

Try Free

Send this to a friend