“It takes many good deeds to build a good reputation, and only one bad one to lose it.”

– Benjamin Franklin

From our handy Getting Started Guide, you know how important it is to bring your suppression list from your old provider with you. Ben Franklin was right – your email reputation will catch a nasty cold if you send to stale, unsubscribed, bounced addresses. This affects whether your messages to your real subscribers are accepted, now and in the future, so it’s best to heed the doctor’s advice.

In this article, we’ll set up an easy-to-use tool to manage your suppression lists. If you want some more background on the “what” and “why” of suppressions, this article is a good starting point.

Scrubbing up

The exported suppression lists we see coming from old providers are often dirty. Duplicate entries, invalid entries with more than one @ sign, invalid characters, telephone numbers instead of email addresses, you name it. We’ve seen files with weird characters in various obscure international alphabets. This might make you consider just amputating those lists, but we’ll explore techniques to preserve as much of them as we can.

Lists can be large, reaching nearly a million entries. Working with small blocks manually is going to take forever. We are going to need a robot surgeon!

Plan for treatment

Let’s set out our needs, and translate them into design goals.

  • Make it easy to get started and simple to use.
  • Make best efforts to understand your file format, even if it appears to contain weird characters.
  • Check and upload any size of a list without manual work.
  • Check the input files up front, with helpful warnings as we go.
  • Checks should be thorough and fast. If there are faults, we want to know exactly where they are in the file, and what’s wrong. Specifically, we need to:
    • Ensure email addresses are well-formed (i.e. follow the RFCs)
    • Check the other field values, such as transactional / non_transactional flags.
  • A “check everything but don’t change anything” mode, to make it easy to find and fix faulty input data.
  • Allow retrieval of your whole suppression list back from SparkPost, or select time-bounded portions.
    • Have time-zone awareness, while accepting times in your locale. In particular, remember that start and end times could fall on either side of a daylight savings time change.
    • Keep it simple. The API supports searching by domain, source, type, description etc – however, that can be done by filtering the retrieved file afterward. If you want these features, raise an issue on the Git repository and we’ll look at it.
  • Work across both master account and subaccounts.
  • Make it easy to supply defaults for missing/optional file information.

That leads us on to making a tool with the following options:

  • Check the format of your files (prior to import). Always a good idea to bring your suppressions with you.
  • Update your suppression list in SparkPost (i.e. create if your list is currently empty).
  • Retrieve your suppression list from SparkPost, for example, if you want to get suppressions back into your upstream campaign management tool.
  • Delete your suppression list from SparkPost, i.e. clean your suppression list out. Maybe you uploaded some entries by mistake. We hope that’s a rare use-case, but it’s there for you.

Time to operate

sparkySuppress is a tool written in Python to help you manage your suppression list. The Github repo includes a comprehensive README file. There’s some help with getting Python 3 installed here if you need it.

You can configure sparkySuppress with the sparkpost.ini file, which is used to set up things you change infrequently, such as your API key, timezone, batch sizes and so on. You can leave everything except API key set to default if you like.

Email addresses from input files are checked as we go, using the excellent email_validator library. We use this to give comprehensive reporting in case of faulty addresses, for example:

The ! marks the entry as having an error. We’ll mark entries that have recoverable problems with a warning w like this:

An excellent character

Text files are not as simple as they appear! Unusual file character encoding can be an obstacle, particularly when you don’t have control over how the suppression list export was created in the first place.

UTF-8 is the most modern and capable encoding, but some systems may not use it. Output files exported from some older versions of Excel will be in Latin-1 for example, rather than UTF-8.

The FileCharacterEncodings setting in the sparkpost.ini provides an easy way to control how your input file will be processed. The tool reads your file using each encoding in turn, and if it finds anomalies, will try in the next encoding and so on. So if you have:

you will see the tool trying each encoding until it finds one that reads the whole file without error. You can select any of the standard encodings shown here.

Your first encoding in the list is used when you’re retrieving entries back from SparkPost into a file.

A good performance

Delete is a bit special – it uses multi-threading because deletes have to be done one per call. Update and retrieve work fast when single-threaded, as each call handles a batch. You should experience good performance with the default batch size and thread settings, but you can tweak them if needed.

Practicing your medicine

In case you don’t have data from your old provider yet, here’s a tool for creating suppression lists that you can use to create a dummy file to practice on.

That’s about it! You are now a skilled suppression list surgeon. You’ll soon have your campaigns in excellent shape.

And finally…

If you are exploring this tool and want to give the author feedback, you’re welcome to visit our Community Slack channel – there’s a channel just for Python, #python. Alternatively, open a Github project issue or pull-request.

If you don’t like Python (whut?) there are some lower-level command-line SparkPost projects that provide a thin “wrapper” over the API and can be used to manipulate suppression lists. Check out Node.js and Go and if you want to know more about the API and UI for suppression lists, here’s a good place to start. There’s also a node.js tool to retrieve your list back again from SparkPost for checking.

If you prefer point-and-click, the SparkPost user interface has a built-in Lists/Suppressions upload feature. This gives you a nice example template and is ideal when you have perfectly formatted files that aren’t too large, with a maximum of 10,000 recipients per file.

— Steve

A Simple Recipe for Scheduled Mailings

Do you need to send batches of emails, synchronized to go at a set time? Are you unsure whether to develop your own campaign management tools, or buy off-the-shelf? Have you been through our Getting Started Guide, and are inspired to send your first campaign, but are feeling a bit nervous about writing your own code?

A customer of ours recently had this same need: sending out email batches of a few million emails each morning. Fully-featured campaign management tools from SparkPost partners such as Iterable, Ongage, and Cordial handle this task (as well as much more complicated scenarios) easily. When you have many different campaign types, triggered emails, complex “customer journey” campaigns, and integrated WYSIWYG editors, sophisticated marketing tools are essential.

If, however, you’re just looking for something plain and simple (and don’t mind getting your hands dirty with a little code), there is another way. And for that you’re in the right place! SparkPost’s Python library and our built-in scheduled sending feature make it easy to put something together.

We’ll put ourselves in the shoes of our friendly fictional company, Avocado Industries, and follow them through setting up a campaign. This article takes you through various features of SparkPost’s Python client library, and links to the final code here.

So What Do I Need?

You’re sending out a newsletter to your subscribers. You’ve created a nice looking template and uploaded it to SparkPost. You have your recipient list at hand, or can export it easily enough from your database. You want SparkPost to mail-merge the recipient personalization details in, and get your awesome send out.

These needs translate into the following design goals for the code we’re going to write:

  • Specify everything about your send using parameters and a small text file. You don’t want to change the code for each campaign.
  • Leverage the SparkPost stored template and personalization features, without doing any programming yourself.
  • Use local flat files for the recipient lists. The same format used by SparkPost’s stored recipient list uploads is a good fit.
  • Gather recipients from your list into API-call batches for efficiency, with no upper limits to the overall size of your send.
  • Support timezones for the scheduled send, and also support “just send it now.”

Data Guacamole

The SparkPost recipient-list format looks like this, with all the fields populated. Here we see Jerome’s details for the Avocado Industries loyalty scheme. We can see he’s a gold-card member, lives in Washington State, and likes several different avocado varieties.

Everything apart from the email address is optional, so it would be nice to have the tool also accept just a plain old list of email addresses.  It would also be nice if the tool is happy if we omit the header line.  That’s easily done.

Taco Me To The Start

The tool is written for python3.  SparkPost relies on the pip installer, and we’ll need git to obtain the tool.  You can check if you already have these tools, using the following commands.

If you already have them, continue to “Add SparkPost Python Library sauce” below.  Otherwise here is a simple install sequence for Amazon EC2 Linux:

If you are using another platform, check out installation instructions for your platform here.

Add SparkPost Python Library Sauce

We use pip3 to install, as follows.

Get the sparkySched code from Github using:

Gotta Be .ini To Win It

We now set up some attributes such as your API key, campaign, and certain substitution data in a text file, as they will be the same each time you send. An example is provided in the project called sparkpost.ini.example. Rename this to sparkpost.ini, and replace <YOUR API KEY> with a key you’ve created in your own SparkPost account.

And, Send!

There’s a sample file of 1000 safe test recipients included in the project that we can send to.  Change the template name below from avocado-goodness  to one you have in your account, and set the sending time to suit you:

If all is well, you should see the “OK” line, and your mailing is sent.  That’s all you need to do.  Happy sending!

Code Salsa

In this section, we take a deeper look inside the code.  You can skip this if you just want to use the tool instead of changing it.  Here’s how we call the SparkPost API to send messages, using the SparkPost Python library:

After some helpful on-screen output about what we’re trying to send, the function composes the recipients with the other passed-in ingredients and mixes in some sensible defaults, using sendObj.update().

The SparkPost library call is wrapped in a try/except clause, as it can return errors at the application level (such as having an incorrect API key), or at the transport level (such as your Internet connection being down). This is generally good practice with any code that’s communicating with a remote service, and follows the examples packaged with our library.

We use startT, endT, and the time()  function to measure how long the API call actually takes. While not strictly necessary, it’s interesting to see how performance varies with batch size, routing distance from client to server, etc.

We will now craft the code to read parameters from the .ini file and use them in the API sends. Let’s read and check the mandatory parameters:

The Python library configParser does the heavy lifting. You’ve got to have an API key, so we exit if unset. baseUri defaults to the sparkpost.com API endpoint if it’s unset. The other parameters from the .ini file are read in the same way, and are described in the project README file.

There are other ways to set things up in Python, such as using environment variables. My preference is for .ini files, because the file is right there, staring at you. It’s easy to store, communicate, change and check, right there in your project.

Chickens Go In, Pies Come Out …

Let’s look at how to read that .csv format recipient list. Python provides a nice library, csv. All the reading of double-quoted stuff that .csv files need to carry JSON objects like "{""custID"": 60525717}" is taken care of for us.

We could use csv to read the whole recipient-list into a Python array object – but that’s not a great idea, if we have squillions of addresses in our list. The client will be perfectly fast enough for our purposes, and we’ll use less client memory, if we read in just enough to give us a nice sized batch to cook each time around.

We’ll also handle line 1 of the file specially, to meet our ‘go easy on the optional file header’ requirement. Recall that a well-formed header should look like this:

If it’s got the single word email somewhere on line 1, let’s assume it really is a header, and we’ll take our field layouts from that line.  The tool will be happy if you omit optional fields, or have them in a different order on the line.  The only one you absolutely need is the email field.

We then check if it’s really a headerless file with just a bunch of email addresses in it, by checking for a single entry with an @ sign.

In the main loop for i,h in enumerate(hdr) we use some nice Python language features to conform the data to the JSON object that SparkPost is expecting. The name field needs to be put inside the address.name JSON attribute. Return_path is added, if present. Metadata, substitution_data and tags all come in to us as JSON-formatted strings, so we unpack them using json.loads().

All that’s left to do, is to chew through the list, sending each time we gather in a full sized batch. We send any final batch at the end, and we’re done.

Command-line Garnish – A Pinch Of Thyme Time

The last part we need is some command-line argument parsing. The recipient-list, the template-ID, and the sending date/time are the things you might want to vary each time the tool is run. Python munges your arguments using argv[] in much the same way as other languages.

There are all kinds of nonsense possible with input date and time – such as February 30th, 24:01 and so on. Mix in timezone offsets, so the user can schedule in their local time, and no-one would seriously want to write their own time parsing code! SparkPost’s API will of course be the final arbiter on what’s good, and what’s not – but it’s better to do some initial taste-tests before we try to send.

Python’s strptime() function does mostly what we want. The format string can be made like SparkPost format, except Python has no : separator in the %z timezone. Python’s elegant negative indexing into strings (working backwards from the end of the string) makes it easy to write a small checking function.

Plat Du Jour

If you don’t want to schedule a future start_time, you can just give today’s date and time. Times in the past are sent immediately.

Depending on where your code is running (I happen to be using an AWS virtual machine), you should see each batch get sent in a few seconds. Even though it’s single-threaded, you can schedule a million emails in around ten minutes. The actual send will proceed (at the scheduled start_time) as fast as it can.

And that’s pretty much it. The full code, which is just over 100 actual lines, is here with easy installation instructions.

A Small Digestif …

What’s the best way to test this out for real? A tool to generate dummy recipient-list with sinkhole addresses could be handy. Keep an eye out for a follow-up blog post. I’ve included a couple of ready-made recipes recipient files in the github project to get you started.

Is this your first dining experience with Python and SparkPost? Did I add too much seasoning? Should the author be pun-ished for these bad jokes? Let us know!

– Steve Tuck
Senior Messaging Engineer

P.S. Want to talk more Python with us? Join us in our Community Slack.

Here at SparkPost, I spend most of my time writing JavaScript. However, the first programming language I really enjoyed was Python. As a member of the dev relations team, I get to write some Python here and there. Whether it’s crunching data and generating reports with jupyter notebook or maintaining our python-sparkpost library, I look forward to spending some quality time with one of my favorite languages. If you are here, chances are you’ve got similar feelings about Python, and you probably want to know how you can use it with SparkPost. So let’s get started!

Installing & Setup

Regardless of which library or framework you’re using, you’ll need to install python-sparkpost:

pip install sparkpost

Once installed you can import and create an instance of the SparkPost class:

If you’d like to avoid putting your API key in code (we highly recommend storing it outside your code), the client library will detect the SPARKPOST_API_KEY environment variable.

Sending Email

Now that you have a SparkPost instance, you’re ready to send. There are quite a few options available for sending, but we’ll start with a simple example. Let’s send an email to a single recipient, specifying inline content:

The recipients argument can be a list containing a mix of strings and dictionaries or a single dictionary with one recipient. Following are some examples of valid values that can be passed to the recipients argument.

String with email only:

String with friendly from and email:

Dictionary:

It’s worth noting that under the hood, the library parses strings and ultimately converts them into the dictionary format before calling the SparkPost API.

There are more options available with transmissions, including specifying stored templates or recipient lists, cc and bcc, adding attachments, specifying a campaign, using substitution data, and much more. Check out the examples, docs for the Transmission class, and the Transmissions API documentation for more info.

Error Handling

We intercept any non-2xx responses from the SparkPost API and wrap them using the SparkPostAPIException  class. You can use this class in your code to catch API exceptions and extract information about the underlying error. We use the excellent requests library under the hood and expose requests’ response object through err.response.

The above code would output the following:

Other Capabilities

The Transmissions class is essentially a wrapper with some syntactic sugar. It’s a subclass of the Resource class, which can be used directly to access the API. This comes in handy when you need to access an underlying API that we haven’t wrapped. Let’s use webhooks as an example:

Usage with Django

There are two ways to use SparkPost through Django. The python-sparkpost library includes a Django email backend (docs).

To configure Django to use SparkPost, put the following configuration in settings.py file.

Replace  API_KEY with an actual API key with the “Send via SMTP” permission. We recommend reading the API key from the SPARKPOST_API_KEY environment variable using  os.environ.get.

You can also use SPARKPOST_OPTIONS to set options that will apply to every transmission. For example:

Django is now configured to use the SparkPost email backend. You can send mail using Django’s send_mail method:

You can also use the EmailMessage or EmailMultiAlternatives classes directly. That will give you access to more specific fields like template:

Alternatively, you can use the excellent django-anymail library, which provides a layer of abstraction on top of several email service providers. For usage instructions, consult the django-anymail documentation for SparkPost.

Usage with Flask

Using SparkPost with Flask is simple—use it like you would with any Python script. This simple example code shows the bare minimum for a Flask app that uses python-sparkpost, catching and returning any exceptions from the API:

Conclusion

We’ve tried to make using SparkPost with Python as simple and enjoyable as writing any other Python code. If you run into any issues or have any questions, feel free to submit an issue on the github repo or join us on our community Slack team in the #python channel. I’d love to hear from you and, selfishly, get the chance to talk about Python.

—Rich Leland
Director of Growth Engineering