Everyone in the email industry is surely well aware of the global outage that happened with Gmail and G Suite/Google Workspace this week. You may have even felt the effects of it by not being able to get critical last minute holiday promotions out in time for shipping cutoffs or having order confirmations delayed during the outage. We empathize with the email community on the disruption as well as the team at Google that worked hard to get the service back online.
We first noticed the issue via our anomaly detection at the onset of the outage. Our teams investigated and conferred with others in the email community and understood that this was a widespread issue and not unique to our customers. On Tuesday afternoon, during Google’s most severe 3 hour downtime period, we observed 90% of our customers’ messages being rejected by Google. We quickly realized we needed to help our customer mitigate the situation.
A Closer Look at the Impact
One of the lingering effects that still hovers over many senders is what to do about the erroneous 550-5.1.1 rejection error code, which stands for “The email account that you tried to reach does not exist”. For many large volume email platforms like SparkPost, a 550 error code triggers a hard bounce, or permanently undeliverable status, and adds that email address to a suppression list to prevent customers from sending to a bad address. Additionally, many of our customers follow good list hygiene practices and flag hard bounces in their internal systems to prevent them from being included in list pulls for campaigns sent through SparkPost.
What We’re Doing For Customers
Because Google has stated that the 550 error code was a mistake, we have successfully removed these records from our customers’ suppression lists (for specific details see our Status Page incident). We have also developed a script to help you create a list of impacted records that you can use to roll back this status within your internal systems, since we understand this downhill impact continues beyond our own database.
Recommended Next Steps For Customers
It’s always a good idea to periodically audit your data integrations and rules between your email platform and your source systems. Do a quick audit to understand how you manage hard bounces and see if those rules have led to many of your Gmail and G Suite email addresses being flagged as unsubscribed or undeliverable in your internal source systems during the outage period. If you have been impacted, you can leverage the script we’ve developed for clients by visiting our Github page. This script will save you time and help get these records flagged properly, so they don’t negatively impact your future list pull logic.
As always, If you have any questions or concerns please submit a support ticket or reach out to your assigned TAM or CSM.
December 21st Update
We’ve now broadened our search to include both in-band and out-of-band bounces, as we believe there are a small number of additional people affected by the out-of-band bounce case. This arises with, e.g. people who have set up forwarding from another mailbox to Gmail, such as when using vanity domains.
Additionally, with more information out of Google we have learned that G Suite / Google Workspace email addresses were not a part of this outage as originally thought. We are confident there is no risk related to the removal of the G Suite addresses from the suppression–as they will be added back to the suppression list as soon as they bounce again for the second time.