A Quick Metrics Calculation Update
“I feel I change my mind all the time. And I sort of feel that’s your responsibility as a person, as a human being—to constantly be updating your positions on as many things as possible. And if you don’t contradict yourself on a regular basis, then you’re not thinking.” ― Malcolm Gladwell
A bit of context for the quote: Way back when, when SparkPost was just a gleam in this Product Manager’s eye, our team investigated how best to define metrics like “open rate”, “click rate,” and so on. That was a surprisingly ambiguous question, given that different ESPs defined them slightly differently, and internally we had as many strongly held opinions of what is “right” as we had people in the discussions. It got heated!
What we finally settled on was this: as an email infrastructure provider, we took the most conservative approach and defined all our rates to be a percentage of “Targeted”.
Fast forward 4+ years and many customer conversations later, and we have collectively changed our minds (hence the Malcolm Gladwell quote). We’ve realized that calculating bounce rates using a denominator that includes messages that we never attempted to send (i.e. those that were suppressed) or calculating engagement rates using a denominator that includes messages that the ISP never accepted doesn’t make sense.
It also made apples-to-apples comparisons difficult: we’ve had numerous customers ask why our open rates seemed worse than what they experienced on other platforms, when the issue was just that other platforms calculate open rate using “Accepted” (a.k.a “Delivered”) as the denominator — a number that’s always going to be smaller than “Targeted”. Smaller denominator = larger rate.
As a result, we realized it makes sense to change how we calculate and report these rates. Later this month, the calculations for the following rates in the SparkPost user interface (UI) will change as follows. Rather than using “targeted” as the denominator, all metrics will reflect either “sent” or “accepted” as appropriate. You can hover over the metrics in the UI to get the definitions.
- Accepted Rate = Accepted / Sent
- Bounce Rate = Bounces / Sent
- Block Bounce Rate = Block Bounces / Sent
- Hard Bounce Rate = Hard Bounces / Sent
- Soft Bounce Rate = Soft Bounces / Sent
- Undetermined Bounce Rate = Undetermined Bounces / Sent
- Spam Complaint Rate = Spam Complaints / Accepted
- Open Rate = Rendered / Accepted
- Click-Through Rate = Clicks / Accepted
Note that this means the all the reported bounce rates will increase as well. This is not an indication that more messages are bouncing — only that we’re using a different denominator in the calculation.
Last point: none of your webhook events or Message Events API results will change since those report raw numbers, not rates. These rate changes will only be visible in the application UI and in the Deliverability Report Card you receive if you have an active account.
Sr. Lead Product Manager