“It takes many good deeds to build a good reputation, and only one bad one to lose it.”

– Benjamin Franklin

From our handy Getting Started Guide, you know how important it is to bring your suppression list from your old provider with you. Ben Franklin was right – your email reputation will catch a nasty cold if you send to stale, unsubscribed, bounced addresses. This affects whether your messages to your real subscribers are accepted, now and in the future, so it’s best to heed the doctor’s advice.

In this article, we’ll set up an easy-to-use tool to manage your suppression lists. If you want some more background on the “what” and “why” of suppressions, this article is a good starting point.

Scrubbing up

The exported suppression lists we see coming from old providers are often dirty. Duplicate entries, invalid entries with more than one @ sign, invalid characters, telephone numbers instead of email addresses, you name it. We’ve seen files with weird characters in various obscure international alphabets. This might make you consider just amputating those lists, but we’ll explore techniques to preserve as much of them as we can.

Lists can be large, reaching nearly a million entries. Working with small blocks manually is going to take forever. We are going to need a robot surgeon!

Plan for treatment

Let’s set out our needs, and translate them into design goals.

  • Make it easy to get started and simple to use.
  • Make best efforts to understand your file format, even if it appears to contain weird characters.
  • Check and upload any size of a list without manual work.
  • Check the input files up front, with helpful warnings as we go.
  • Checks should be thorough and fast. If there are faults, we want to know exactly where they are in the file, and what’s wrong. Specifically, we need to:
    • Ensure email addresses are well-formed (i.e. follow the RFCs)
    • Check the other field values, such as transactional / non_transactional flags.
  • A “check everything but don’t change anything” mode, to make it easy to find and fix faulty input data.
  • Allow retrieval of your whole suppression list back from SparkPost, or select time-bounded portions.
    • Have time-zone awareness, while accepting times in your locale. In particular, remember that start and end times could fall on either side of a daylight savings time change.
    • Keep it simple. The API supports searching by domain, source, type, description etc – however, that can be done by filtering the retrieved file afterward. If you want these features, raise an issue on the Git repository and we’ll look at it.
  • Work across both master account and subaccounts.
  • Make it easy to supply defaults for missing/optional file information.

That leads us on to making a tool with the following options:

  • Check the format of your files (prior to import). Always a good idea to bring your suppressions with you.
  • Update your suppression list in SparkPost (i.e. create if your list is currently empty).
  • Retrieve your suppression list from SparkPost, for example, if you want to get suppressions back into your upstream campaign management tool.
  • Delete your suppression list from SparkPost, i.e. clean your suppression list out. Maybe you uploaded some entries by mistake. We hope that’s a rare use-case, but it’s there for you.

Time to operate

sparkySuppress is a tool written in Python to help you manage your suppression list. The Github repo includes a comprehensive README file. There’s some help with getting Python 3 installed here if you need it.

You can configure sparkySuppress with the sparkpost.ini file, which is used to set up things you change infrequently, such as your API key, timezone, batch sizes and so on. You can leave everything except API key set to default if you like.

Email addresses from input files are checked as we go, using the excellent email_validator library. We use this to give comprehensive reporting in case of faulty addresses, for example:

The ! marks the entry as having an error. We’ll mark entries that have recoverable problems with a warning w like this:

An excellent character

Text files are not as simple as they appear! Unusual file character encoding can be an obstacle, particularly when you don’t have control over how the suppression list export was created in the first place.

UTF-8 is the most modern and capable encoding, but some systems may not use it. Output files exported from some older versions of Excel will be in Latin-1 for example, rather than UTF-8.

The FileCharacterEncodings setting in the sparkpost.ini provides an easy way to control how your input file will be processed. The tool reads your file using each encoding in turn, and if it finds anomalies, will try in the next encoding and so on. So if you have:

you will see the tool trying each encoding until it finds one that reads the whole file without error. You can select any of the standard encodings shown here.

Your first encoding in the list is used when you’re retrieving entries back from SparkPost into a file.

A good performance

Delete is a bit special – it uses multi-threading because deletes have to be done one per call. Update and retrieve work fast when single-threaded, as each call handles a batch. You should experience good performance with the default batch size and thread settings, but you can tweak them if needed.

Practicing your medicine

In case you don’t have data from your old provider yet, here’s a tool for creating suppression lists that you can use to create a dummy file to practice on.

That’s about it! You are now a skilled suppression list surgeon. You’ll soon have your campaigns in excellent shape.

And finally…

If you are exploring this tool and want to give the author feedback, you’re welcome to visit our Community Slack channel – there’s a channel just for Python, #python. Alternatively, open a Github project issue or pull-request.

If you don’t like Python (whut?) there are some lower-level command-line SparkPost projects that provide a thin “wrapper” over the API and can be used to manipulate suppression lists. Check out Node.js and Go and if you want to know more about the API and UI for suppression lists, here’s a good place to start. There’s also a node.js tool to retrieve your list back again from SparkPost for checking.

If you prefer point-and-click, the SparkPost user interface has a built-in Lists/Suppressions upload feature. This gives you a nice example template and is ideal when you have perfectly formatted files that aren’t too large, with a maximum of 10,000 recipients per file.

— Steve

Suppression List API Updates Gears Laptop

New Suppression List API Functionality Coming Soon

At SparkPost we’re constantly using feedback from our community to enhance our application and APIs. Based on what we’ve heard, we want to let you know about some new functionality that will be coming to our suppression list API in early November 2016.

  • We’ve heard you loud and clear and will be removing rate limiting on all Suppression List API calls. We’re improving the underlying data storage for this API, which allows us to remove the rate limiting that was put in place to guard against poor performance.
  • You will now be able to retrieve your entire suppression list. Each GET call to the API will include pagination so you can page through the results.
  • You will be able to search for suppression list entries by domain or description. For example, searching for all suppressions from gmail.
  • The PUT method on a single recipient is returning. You will no longer have to do a PUT to /api/v1/suppression-list and provide an array. Instead you can now do a PUT to /api/v1/suppression-list/test@example.com to update the test@example.com resource directly.

There will be no change in functionality for the existing create, update, and delete endpoints. While we always make our best effort to prevent breaking changes, we had to make an exception and will introduce a breaking change for GET calls to the Suppression List API.

Currently, GET calls return a response where each recipient contains two fields – transactional and non_transactional:

With the updates to the API, GET calls will now return a response where a recipient will show up for each type of suppression. For example, if a recipient is suppressed for both transactional and non-transactional emails, they will show up in the list twice with transactional or non-transactional set to true.  We also added type to reflect the boolean – set to either transactional or non_transactional:

Why we are making this change

Our initial implementation has several limitations:

  • Requests to the Suppression List API are rate limited.
  • Downloading the entire suppression list is not possible as the number of records returned is capped at 100,000.
  • It is not possible to distinguish why a recipient was added to the suppression list (e.g. hard bounce or spam complaint) for recipients that were suppressed for both transactional and non-transactional emails because there is only a single entry for each recipient. This data is contained in the source field. The same can be said about the updated and description fields.
  • It is not possible to make a PUT call to update a single entry in the suppression list.
  • The Web UI is limited. It does not load an initial page of suppressions and you cannot download your list.

Currently it takes several hours for a recipient to be suppressed and we agree this isn’t a great experience. In order to support a more real-time suppression, we swapped out the backing database and changed the data model to tune it for real-time lookups. With separate entries for each recipient for each type of suppression, it allows us to support more granular suppressions by more than just transactional/non-transactional flags.

Who’s affected

This change will only impact users that have suppressed a recipient for both transactional and non-transactional messages. More specifically, have code that make GET calls to the Suppression List API and are not iterating over the entire array to check the transactional/non-transactional booleans. Although we cannot read everyone’s code that has integrated against this API, we feel pretty confident that the number of users affected are very low, if any at all.

Necessary actions

If you have code that parses the response of the GET endpoints /api/v1/suppression-list or /api/v1/suppression-list/foo@bar.com, you may have to add some additional code to iterate over your array to ensure you are parsing both the transactional and non-transactional records. If you are using subaccounts, you may already be iterating over the result but you may have to add additional logic to check for the transactional/non-transactional booleans. You can view the updated Suppression List API documentation with the improved response format here.

Making a tough decision

At SparkPost, we take backwards-compatibility very seriously. We don’t make these types of changes lightly and appreciate that our community has code in production using our API. We wanted to version the API but in this case it wouldn’t be feasible because the entire underlying architecture was changing. As a result, Taking all of the above feedback and issues into account, we believe the updates will provide a much better experience for our community in the long run.

Finally, if you’d like to discuss this update, or anything SparkPost related, please feel free to join us on our community Slack channel or reach out to us via Twitter.

—Bob Evans
Director of Application Engineering, SparkPost