A Simple Recipe for Scheduled Mailings

Do you need to send batches of emails, synchronized to go at a set time? Are you unsure whether to develop your own campaign management tools, or buy off-the-shelf? Have you been through our Getting Started Guide, and are inspired to send your first campaign, but are feeling a bit nervous about writing your own code?

A customer of ours recently had this same need: sending out email batches of a few million emails each morning. Fully-featured campaign management tools from SparkPost partners such as Iterable, Ongage, and Cordial handle this task (as well as much more complicated scenarios) easily. When you have many different campaign types, triggered emails, complex “customer journey” campaigns, and integrated WYSIWYG editors, sophisticated marketing tools are essential.

If, however, you’re just looking for something plain and simple (and don’t mind getting your hands dirty with a little code), there is another way. And for that you’re in the right place! SparkPost’s Python library and our built-in scheduled sending feature make it easy to put something together.

We’ll put ourselves in the shoes of our friendly fictional company, Avocado Industries, and follow them through setting up a campaign. This article takes you through various features of SparkPost’s Python client library, and links to the final code here.

So What Do I Need?

You’re sending out a newsletter to your subscribers. You’ve created a nice looking template and uploaded it to SparkPost. You have your recipient list at hand, or can export it easily enough from your database. You want SparkPost to mail-merge the recipient personalization details in, and get your awesome send out.

These needs translate into the following design goals for the code we’re going to write:

  • Specify everything about your send using parameters and a small text file. You don’t want to change the code for each campaign.
  • Leverage the SparkPost stored template and personalization features, without doing any programming yourself.
  • Use local flat files for the recipient lists. The same format used by SparkPost’s stored recipient list uploads is a good fit.
  • Gather recipients from your list into API-call batches for efficiency, with no upper limits to the overall size of your send.
  • Support timezones for the scheduled send, and also support “just send it now.”

Data Guacamole

The SparkPost recipient-list format looks like this, with all the fields populated. Here we see Jerome’s details for the Avocado Industries loyalty scheme. We can see he’s a gold-card member, lives in Washington State, and likes several different avocado varieties.

Everything apart from the email address is optional, so it would be nice to have the tool also accept just a plain old list of email addresses.  It would also be nice if the tool is happy if we omit the header line.  That’s easily done.

Taco Me To The Start

The tool is written for python3.  SparkPost relies on the pip installer, and we’ll need git to obtain the tool.  You can check if you already have these tools, using the following commands.

If you already have them, continue to “Add SparkPost Python Library sauce” below.  Otherwise here is a simple install sequence for Amazon EC2 Linux:

If you are using another platform, check out installation instructions for your platform here.

Add SparkPost Python Library Sauce

We use pip3 to install, as follows.

Get the sparkySched code from Github using:

Gotta Be .ini To Win It

We now set up some attributes such as your API key, campaign, and certain substitution data in a text file, as they will be the same each time you send. An example is provided in the project called sparkpost.ini.example. Rename this to sparkpost.ini, and replace <YOUR API KEY> with a key you’ve created in your own SparkPost account.

And, Send!

There’s a sample file of 1000 safe test recipients included in the project that we can send to.  Change the template name below from avocado-goodness  to one you have in your account, and set the sending time to suit you:

If all is well, you should see the “OK” line, and your mailing is sent.  That’s all you need to do.  Happy sending!

Code Salsa

In this section, we take a deeper look inside the code.  You can skip this if you just want to use the tool instead of changing it.  Here’s how we call the SparkPost API to send messages, using the SparkPost Python library:

After some helpful on-screen output about what we’re trying to send, the function composes the recipients with the other passed-in ingredients and mixes in some sensible defaults, using sendObj.update().

The SparkPost library call is wrapped in a try/except clause, as it can return errors at the application level (such as having an incorrect API key), or at the transport level (such as your Internet connection being down). This is generally good practice with any code that’s communicating with a remote service, and follows the examples packaged with our library.

We use startT, endT, and the time()  function to measure how long the API call actually takes. While not strictly necessary, it’s interesting to see how performance varies with batch size, routing distance from client to server, etc.

We will now craft the code to read parameters from the .ini file and use them in the API sends. Let’s read and check the mandatory parameters:

The Python library configParser does the heavy lifting. You’ve got to have an API key, so we exit if unset. baseUri defaults to the sparkpost.com API endpoint if it’s unset. The other parameters from the .ini file are read in the same way, and are described in the project README file.

There are other ways to set things up in Python, such as using environment variables. My preference is for .ini files, because the file is right there, staring at you. It’s easy to store, communicate, change and check, right there in your project.

Chickens Go In, Pies Come Out …

Let’s look at how to read that .csv format recipient list. Python provides a nice library, csv. All the reading of double-quoted stuff that .csv files need to carry JSON objects like "{""custID"": 60525717}" is taken care of for us.

We could use csv to read the whole recipient-list into a Python array object – but that’s not a great idea, if we have squillions of addresses in our list. The client will be perfectly fast enough for our purposes, and we’ll use less client memory, if we read in just enough to give us a nice sized batch to cook each time around.

We’ll also handle line 1 of the file specially, to meet our ‘go easy on the optional file header’ requirement. Recall that a well-formed header should look like this:

If it’s got the single word email somewhere on line 1, let’s assume it really is a header, and we’ll take our field layouts from that line.  The tool will be happy if you omit optional fields, or have them in a different order on the line.  The only one you absolutely need is the email field.

We then check if it’s really a headerless file with just a bunch of email addresses in it, by checking for a single entry with an @ sign.

In the main loop for i,h in enumerate(hdr) we use some nice Python language features to conform the data to the JSON object that SparkPost is expecting. The name field needs to be put inside the address.name JSON attribute. Return_path is added, if present. Metadata, substitution_data and tags all come in to us as JSON-formatted strings, so we unpack them using json.loads().

All that’s left to do, is to chew through the list, sending each time we gather in a full sized batch. We send any final batch at the end, and we’re done.

Command-line Garnish – A Pinch Of Thyme Time

The last part we need is some command-line argument parsing. The recipient-list, the template-ID, and the sending date/time are the things you might want to vary each time the tool is run. Python munges your arguments using argv[] in much the same way as other languages.

There are all kinds of nonsense possible with input date and time – such as February 30th, 24:01 and so on. Mix in timezone offsets, so the user can schedule in their local time, and no-one would seriously want to write their own time parsing code! SparkPost’s API will of course be the final arbiter on what’s good, and what’s not – but it’s better to do some initial taste-tests before we try to send.

Python’s strptime() function does mostly what we want. The format string can be made like SparkPost format, except Python has no : separator in the %z timezone. Python’s elegant negative indexing into strings (working backwards from the end of the string) makes it easy to write a small checking function.

Plat Du Jour

If you don’t want to schedule a future start_time, you can just give today’s date and time. Times in the past are sent immediately.

Depending on where your code is running (I happen to be using an AWS virtual machine), you should see each batch get sent in a few seconds. Even though it’s single-threaded, you can schedule a million emails in around ten minutes. The actual send will proceed (at the scheduled start_time) as fast as it can.

And that’s pretty much it. The full code, which is just over 100 actual lines, is here with easy installation instructions.

A Small Digestif …

What’s the best way to test this out for real? A tool to generate dummy recipient-list with sinkhole addresses could be handy. Keep an eye out for a follow-up blog post. I’ve included a couple of ready-made recipes recipient files in the github project to get you started.

Is this your first dining experience with Python and SparkPost? Did I add too much seasoning? Should the author be pun-ished for these bad jokes? Let us know!

– Steve Tuck
Senior Messaging Engineer

P.S. Want to talk more Python with us? Join us in our Community Slack.

Twice a year, SparkPost hosts an employee hackathon. The idea is to give our hard-working technical staff a chance to step outside their usual areas of focus and to have free rein to explore their creative ideas that push our technology in all sorts of cool directions. These demonstrations, and the exercise itself, are fun and really inspiring to us all and an important part of our culture of innovation.

During the hackathon, our engineering and technical operations teams spend a day and a half playing with code and turning ideas into working proof of concepts. Though exploration is the main goal, many of the ideas end up being refined and incorporated into our products or to our internal toolsets.

As usual, the Fall hackathon teams impressed us with their orthogonal thinking and technical chops. The group produced a crop of innovative ideas that show the depth and breadth of our capabilities. Let’s take a look at some of the great hacks that came out of the event this time around.

Guidelines

Each hackathon event has a theme with several areas of focus. Our Fall event featured the following categories:

  • Product: a new or improved product or feature, including end-user tools and add-ons
  • Tools: an internal tool or infrastructure that can improve how we design, build, test, ship, or support our products
  • Partner Integrations: a value added integration of our products and services with a third party
  • Fun with Data: making more of the data we have in our cloud environments
  • We Deliver: innovations in abuse detection, compliance, and deliverability

The rules are simple: use whatever tech you want, research as much or as little as you want, and don’t write any code ahead of time. The hackathon starts at 10 AM on day one and ends with presentations starting at 1 PM the following day. A panel of judges reviews the entries and picks a winning team for each category.

Results

Best in Show

Our judges award best in show to the project that stands out to them the most. In this hackathon, it was a proof of concept for a new iteration of our Adaptive Delivery functionality, which auto-tunes outbound email delivery parameters and traffic shaping in real-time to avoid blocks, safeguard reputation and optimize delivery.

This functionality is currently a part of the core technology that powers SparkPost—the Momentum platform. The project used a combination of Vertica, Redis, and Node.js to power the rules system for Adaptive Delivery, allowing us to offload some of the work to more efficient services. The main benefits here are scalability, performance, and abstraction. Another potential benefit could be sharing Adaptive Delivery data to improve deliverability for all of our customers.

winners-best-in-show

Product

For our product category, two projects shared top honors: an in-application API explorer for SparkPost, and a hot backup for our PowerMTA product. The interactive discoverability made possible by in-application API explorer was so awesome that we’ve added it to the SparkPost UI! This interactive tool complements our traditional API documentation.

explorer-2

 

The PowerMTA project did a proof-of-concept system that enables one instance of PowerMTA to be the hot backup for another instance. Also addressed was a potential solution for recovery from failures when using hot backups. The team relied on their knowledge of database replication and SMTP to build the replication protocol and used TCP socket programming to implement the details.

Tools

The tools category produced a project allowing us to automate testing against our SparkPost Elite environments. Our tech ops team routinely spins up new SparkPost Elite customer environments with the requisite architecture and needs to test that everything is set up correctly and ready to send. To aid in automation of this process, one of the teams created autoscott, a command line utility that allows teams to run commands to perform actions like sending a single transmission, sending multiple transmissions, creating recipient lists, creating templates, and more.

Fun With Data

In the fun with data category, we had a team create a prototype for a weekly report card. This report would be delivered to our customers and contain a summary of their sending including number of emails sent, open rate, and click rate. It would also deliver helpful tips on how to improve your mailings based on trends in data. For example, if your open rate went down week-over-week, we’d let you know how to potentially correct this issue. Additionally the email contains a breakdown of volume by day for the past few weeks and some information about your sending habits as they relate to other senders on SparkPost.

Partner Integrations

Signing up for and signing in to SparkPost is simple. The winning entry in our partner integrations category wanted to take this a step further, so they built the ability to sign up and sign in via Github, Twitter, or Google. The project made use of the Node.js library passport and the adapters for GithubTwitter, and Google. As an added bonus, there is also a passport adapter for SAML, which we’re using to power our upcoming SSO functionality for our SparkPost Elite customers.

We Deliver

Deliverability is a big deal at SparkPost. We pride ourselves on keeping the bad actors out and the good actors sending reliably. In our we deliver category, the winning team worked on refining our ability to scan outgoing email for spam. One particular enhancement was a performance optimization to make scanning of large transmissions more efficient. This group worked using the fine CSDMC2010 spam corpus, which includes a training collection of labeled spam and ham emails, as well as an additional unlabeled test set. The group also worked to visualize the resulting data using SumoLogic, a commercial product which we use in our data analysis.

winners-we-deliver

Another Successful Hackathon

Hosting employee hackathons has become a great tradition here at SparkPost. Our hackathons support our culture of innovation…and ultimately help our customers succeed. As you can see, our teams are constantly striving to improve our product and our tools. The Fall hackathon was no exception.

We’re always looking for bright people to join our team. If solving complex problems, using awesome tech, and participating in Hackathons sounds fun to you, check out our open positions.

Error Logging

Mistakes happen. It’s a fact of life. But, mistakes are often a precursor to growth…as long as you learn from them!

Early on after launching our service, the SparkPost support team occasionally received feedback from customers who tried to sign up for one of our paid tiers, explaining that the transaction failed. Not only did that stink for our customers, but it was a real pain point for us as well. We knew that sort of error couldn’t continue unchecked, so we decided to get to the bottom of this frustrating problem.

We integrate with a third party for payment processing, and the third-party system sometimes responds to a request with what’s essentially an unknown error. Super helpful! In most cases, we were able to surface a more helpful error to the user based on logic that parses the response. But up until a few months ago, we had zero visibility into the unknown errors. We had no way to help our customers, and no way to learn from the problem! After looking at a few more third-party solutions, we decided to leverage functionality our current stack already provided.

Some updates to our Angular-based UI and our Nginx web server was all it took. And while we were at it, we went ahead and used the approach to log uncaught exceptions, too. In the front-end portion of our application, we built a simple service for posting errors to Nginx. We just have to get the username from the session, along with the error response and data posted, and post it back to our logs. (Of course, for compliance reasons, we never deal with actual credit card information; our third-party service takes care of that.) No need to catch a rejected promise in the service either, because for our case, we don’t have a need for handling failed log attempts.

From the Nginx side, we’re able to customize our log format by leveraging the built in directives and variables available in the core module as well as the log and proxy modules. In our case, we’re using the access_log and log_format directives, and the $msec variable from the log module. Then $proxy_add_x_forwarded_for from the proxy module. We get $http_user_agent and $request_body from the core module. With that in place, we can easily see information about the request, in addition to the data being sent from the UI.

Finally, back to our front-end code and the case of uncaught exceptions. Angular comes with a global application exception handler $exceptionHandler that does exactly that. In our case, we use the $provide service’s $decorator method to add additional behavior. It’s called in the config phase callback, so that the decorator can intercept the service creation, and we use the $delegate method to retain the exception handlers base functionality.

As any JavaScript developer will know, we have a lot of new tools at our disposal today. But, by knowing your existing tools well, you can arrive at simple, clean solution that addresses your problem without having to pull in extra dependencies, and without the need to learn additional APIs.

Now, whenever our support team needs to assist customers with their subscriptions, we’re able to quickly access additional details, and as an added bonus, we have an easy way to gain insight into exceptions within our application to prevent future occurrences! Learning from this previously frustrating error turned out to be an opportunity for improving the customer experience—and for making the internals of our application better, as well.

php client library

We are proud to announce a new release of the php-sparkpost library at version 1.0.0!  After receiving feedback from the community, we have dramatically overhauled the library to be easier to use and more flexible.  

One of the biggest changes that we’ve made in this release is the use of HTTP adapters.  Since the library is built on top of the SparkPost REST API, one of the biggest challenges that we faced in previous versions was the ability to use our library with other libraries that also made HTTP requests.  We had originally chosen to ship our client library with Guzzle, a popular HTTP request library, but this unfortunately came with the side effect of version mismatches and dependency hell for our users.

Enter HTTP adapters.  HTTP adapters allow us to present the same interface to the user and allows us to use the same interface within the library for any HTTP request library or tool that the user decides to implement.  The user simply has to instantiate an adapter for the library that they are using and pass it into the SparkPost constructor when they instantiate it.  We use the Ivory\HttpAdapter library which is PSR-7 compliant.  PSR-7 is a specification for HTTP message interfaces in PHP.

For example if you use a library like Guzzle:

Or if you’re using something like CURL:

The second major change is that we switched from a static interface to an instance based one. This doesn’t change much for the end user of the library besides only needing to include one package instead of two and swapping a few ‘::’ operators for ‘->’ operators, but between this and the adapter changes mentioned above, this version isn’t a simple drop in replacement for previous versions (which is why we’re releasing as v1.0). However, as the library’s toolset grows, we hope these changes offer developers the ability to access all of its functionality from a single reference and with a shared configuration across all of those tools.

Here is a simple example of sending an email with SparkPost after setting it up above: 

We also added a new feature to be able to use the library to take advantage of other API endpoints that aren’t currently supported.  We do this with a new ‘setupUnwrapped’ method passing in the endpoint name. Doing so exposes basic create, update, get, and delete methods which will support bodies and parameters outlined in our API documentation.  This is a little hard to explain but it may be easier to see with an example:

The composer packagist repository for the library resides here

It can be installed using composer:

Please feel free to try it out in your projects. If you have feedback or would like to make a contribution please feel free to submit an issue or a pull request on the GitHub repository.