**We are excited to announce our new partnership with Rebel. Rebel is disrupting the email marketing space and allows users to send interactive emails with no plug-ins required. This guest blog post, written by Kevin Dutra highlights Rebel’s ease of use and their competitive edge in the market.

Integrate your email channel and app with in-email interactivity.

Email is one of the oldest communication protocols on the web. Unfortunately for the billions of people that use it to communicate, up until a few years ago, almost nothing has dramatically improved the user experience. After years of very little innovation, interactive email has created a flurry of exciting changes.

At Rebel, we don’t see stodgy, old email clients that are a nightmare to develop against. We see a brand new surface for your application to live on. Interactive email allows you to show a user the exact action you want them to take, without the need for a click or leave their inbox.

Taking advantage of these new opportunities for user actions is easier said than done. Coding normal emails remains a challenge for most developers due to the deep fragmentation in email clients. That’s why we believe the best way to take advantage is through our interactive email API.

It’s the only way of generating interactive emails on the fly, personalized to the user you are sending to. You don’t need to know the latest hacks to get the email rendering perfectly or the quirks of gathering data from inside an email client. All you need to know is the information you want to get from a user, and with a simple JSON request, you can get back ready-to-send interactive email HTML.

But what does that really mean?

If the above sounds fanciful and far-fetched to you, you’re not alone. However, it is that simple. All that you need to do is describe the functionality that you want to achieve in JSON and our API will convert it to HTML for you.

To prove that point we’re going to build a quick email that asks a user for some feedback on your app. It will contain stars for rating 1-5 and a text input for collecting general feedback.

The API call structure.

For this example, we’ll be looking at an abbreviated API call (I’ve taken out some of the options so that the information is clear), that will return to you HTML with two form fields, and a submit button.

As you can see below the JSON call includes an array at context.components.body.formfields , in that array there are two objects. Each describes one piece of functionality. One for the rating module and one for the text area. Here you place all the information you want to be covered displayed in the email and adjust and field-specific settings.

 

The above example is abbreviated for illustrative purposes, but generally, our API allows you to set the values and styles of any element inside our interactive modules.

What you get back.

Our  /campaigns/:id/html  endpoint will return to you HTML, completely ready to be delivered. No need to do anything to it, just send it out. Here is the email from the full request above as seen in Apple Mail:

Interactive email analytics.

Of course what good is new technology if you can’t tell if it’s adding value to your business. That’s why we built a robust analytics platform for recording every possible action a user takes within an email. If a user clicks a radio button or star rating, we record it. If they open the email on Outlook and see a beautifully rendered static fallback message, we record it.

All that actionable data is only good if you can consume it. That’s why we created a number of ways to have our data pushed to your servers (such as a webhook).

Go forth and interact.

These kinds of forms are one excellent way to allow users to interact with your app from within an email client, and there is a myriad of ways to apply this same technology. Responding to comments, prompting inactive users to post again for the first time, or allowing users to complete portions of their onboarding while verifying their email address, are all possible.

You can do even more than forms with our flexible API, Thanks to the various modules we have created, your users can take quizzes, research or filter products, and even shop directly in the inbox. All while our robust analytics allows you to gather every action and use it for things like progressive profiling and user recommendations.

 

 

“It takes many good deeds to build a good reputation, and only one bad one to lose it.”

– Benjamin Franklin

From our handy Getting Started Guide, you know how important it is to bring your suppression list from your old provider with you. Ben Franklin was right – your email reputation will catch a nasty cold if you send to stale, unsubscribed, bounced addresses. This affects whether your messages to your real subscribers are accepted, now and in the future, so it’s best to heed the doctor’s advice.

In this article, we’ll set up an easy-to-use tool to manage your suppression lists. If you want some more background on the “what” and “why” of suppressions, this article is a good starting point.

Scrubbing up

The exported suppression lists we see coming from old providers are often dirty. Duplicate entries, invalid entries with more than one @ sign, invalid characters, telephone numbers instead of email addresses, you name it. We’ve seen files with weird characters in various obscure international alphabets. This might make you consider just amputating those lists, but we’ll explore techniques to preserve as much of them as we can.

Lists can be large, reaching nearly a million entries. Working with small blocks manually is going to take forever. We are going to need a robot surgeon!

Plan for treatment

Let’s set out our needs, and translate them into design goals.

  • Make it easy to get started and simple to use.
  • Make best efforts to understand your file format, even if it appears to contain weird characters.
  • Check and upload any size of a list without manual work.
  • Check the input files up front, with helpful warnings as we go.
  • Checks should be thorough and fast. If there are faults, we want to know exactly where they are in the file, and what’s wrong. Specifically, we need to:
    • Ensure email addresses are well-formed (i.e. follow the RFCs)
    • Check the other field values, such as transactional / non_transactional flags.
  • A “check everything but don’t change anything” mode, to make it easy to find and fix faulty input data.
  • Allow retrieval of your whole suppression list back from SparkPost, or select time-bounded portions.
    • Have time-zone awareness, while accepting times in your locale. In particular, remember that start and end times could fall on either side of a daylight savings time change.
    • Keep it simple. The API supports searching by domain, source, type, description etc – however, that can be done by filtering the retrieved file afterward. If you want these features, raise an issue on the Git repository and we’ll look at it.
  • Work across both master account and subaccounts.
  • Make it easy to supply defaults for missing/optional file information.

That leads us on to making a tool with the following options:

  • Check the format of your files (prior to import). Always a good idea to bring your suppressions with you.
  • Update your suppression list in SparkPost (i.e. create if your list is currently empty).
  • Retrieve your suppression list from SparkPost, for example, if you want to get suppressions back into your upstream campaign management tool.
  • Delete your suppression list from SparkPost, i.e. clean your suppression list out. Maybe you uploaded some entries by mistake. We hope that’s a rare use-case, but it’s there for you.

Time to operate

sparkySuppress is a tool written in Python to help you manage your suppression list. The Github repo includes a comprehensive README file. There’s some help with getting Python 3 installed here if you need it.

You can configure sparkySuppress with the sparkpost.ini file, which is used to set up things you change infrequently, such as your API key, timezone, batch sizes and so on. You can leave everything except API key set to default if you like.

Email addresses from input files are checked as we go, using the excellent email_validator library. We use this to give comprehensive reporting in case of faulty addresses, for example:

The ! marks the entry as having an error. We’ll mark entries that have recoverable problems with a warning w like this:

An excellent character

Text files are not as simple as they appear! Unusual file character encoding can be an obstacle, particularly when you don’t have control over how the suppression list export was created in the first place.

UTF-8 is the most modern and capable encoding, but some systems may not use it. Output files exported from some older versions of Excel will be in Latin-1 for example, rather than UTF-8.

The FileCharacterEncodings setting in the sparkpost.ini provides an easy way to control how your input file will be processed. The tool reads your file using each encoding in turn, and if it finds anomalies, will try in the next encoding and so on. So if you have:

you will see the tool trying each encoding until it finds one that reads the whole file without error. You can select any of the standard encodings shown here.

Your first encoding in the list is used when you’re retrieving entries back from SparkPost into a file.

A good performance

Delete is a bit special – it uses multi-threading because deletes have to be done one per call. Update and retrieve work fast when single-threaded, as each call handles a batch. You should experience good performance with the default batch size and thread settings, but you can tweak them if needed.

Practicing your medicine

In case you don’t have data from your old provider yet, here’s a tool for creating suppression lists that you can use to create a dummy file to practice on.

That’s about it! You are now a skilled suppression list surgeon. You’ll soon have your campaigns in excellent shape.

And finally…

If you are exploring this tool and want to give the author feedback, you’re welcome to visit our Community Slack channel – there’s a channel just for Python, #python. Alternatively, open a Github project issue or pull-request.

If you don’t like Python (whut?) there are some lower-level command-line SparkPost projects that provide a thin “wrapper” over the API and can be used to manipulate suppression lists. Check out Node.js and Go and if you want to know more about the API and UI for suppression lists, here’s a good place to start. There’s also a node.js tool to retrieve your list back again from SparkPost for checking.

If you prefer point-and-click, the SparkPost user interface has a built-in Lists/Suppressions upload feature. This gives you a nice example template and is ideal when you have perfectly formatted files that aren’t too large, with a maximum of 10,000 recipients per file.

— Steve

Hello everyone 👋 Avi here. I’ve got exciting news about my role at SparkPost!

TL;DR:

 

Last summer, I joined SparkPost as an intern on the Growth Engineering team. I was lucky enough drop the “intern” part of my title in September of last year, and I’ve gotten to work on some pretty interesting stuff.

Now, one year later (drumroll please) I’m psyched to say that I’m joining the Developer Relations team as the West Coast Developer Advocate in San Francisco. 🎉 I’ve gotten to work with Mary and Aydrian during my time here and I’m thrilled to join them full time.

I visited SF for the first time after AngelHack Silicon Valley a couple of weeks ago and I’m already a little in love with the city.

I’ve got a ton of ideas that I hope to share about email and making it a little more fun. I’ve got some tooling for front-end work that I’m playing with because let’s face it — HTML emails are hard. (If you want to help, let me know 💌)

After that, who knows! I want to work on email testing and validation tools, integrations with some awesome projects, and some fun demos and guides. Right now, I’m toying with the idea of checkers built with interactive email and SparkPost.

I’m going to be traveling to more events now as well (though I’ve managed to go a good number as an engineer) so if you see me, say hi! I absolutely love going to hackathons, so if you have any suggestions there let me know!

In any case, I’m new to the Developer Advocacy side of things, but I’m super excited for this adventure. If you have any tips or tricks for living in SF or traveling, tweet em at me 😬

If you want to talk, you can find me on Twitter, Github, or email ❤, and I’m always up for a cup of coffee.

— Avi

Big Rewards Blog Footer

Top 10 Blogs: Our Year in Review

We’re finishing out the year with a roundup of our top 10 blogs from 2016. The Mandrill announcement in April impacted our community, and as a result our blog, in a big way. We’re recapping that along with other top posts on deliverability tips and email marketing best practices down below. As always, our ears are open, so if there’s a certain topic you’d like to see on the blog, leave us a comment, tweet us, or ping us in slack.

Without further ado, we give you the top 10 blogs of 2016:

#1 Mandrill Alternatives

It’s no surprise that our Mandrill alternative blogs dominated our top 10 list (5 out of our top 10). We responded in real-time to the Mandrill crisis, and our CEO even weighed in and made you a promise he intends to stick by for the long haul. The Mandrill incident also inspired us to create SendGrid and Mailgun migration guides, check them out when you have a chance.

Mandrill Template Migration top 10 blogs

#2 PHP

But beyond Mandrill, we also had some other top posts. Coming in second was using SparkPost in PHP. Believe it or not, many of you use PHP through our WordPress plugin.

PHP in SparkPost

#3 Advanced Email Templates

For developers who want to get the most out of SparkPost templating capabilities, this post was meant for you! In this straight-forward post, Chris Wilson makes sending email easy and gives you some pro tips along the way.

 

advanced email templates

 

#4 What Recruiters Look for in a Dev Candidate

Everyone wants to know how to interview well. In this post we told you about what four tech recruiters look for when hiring developer and engineering candidates.

Recruiter for Dev Candidate

#5 Webhooks!

One of the most useful elements of SparkPost are our webhooks and in this post, Ewan Dennis walks you through the basics and beyond. Knowing what to expect functionally beyond the raw API spec is half the battle when consuming new data sources like SparkPost webhooks.

webhooks: beyond the basics

#6 Outlook and Hotmail Email Deliverability

The Outlook inbox is one of the major destinations for most email senders, especially those with large numbers of consumer subscribers. It also has a reputation for being somewhat tricky to get into. In this post, one of our deliverability experts, Tonya Gordon, shares what senders need to know in order to get the best Hotmail/Outlook deliverability and ensure their messages reach the inbox.

#7 Announcing Subaccounts!

Thanks to your feedback, the Mandrill event helped us expedite our release of subaccounts ahead of schedule. Our VP of Product told you about how we process your feedback and what’s available with subaccounts.

SparkPost #WeLoveDevelopers

#8 Are You an Email Rookie?

Sometimes you need to go beyond a top 10 list and in this case we did — 17 tips on how not to be labeled an email rookie. In this post we put together a list of common mistakes, with a heavy dose of snark, on how to avoid being labeled an email marketing rookie.

Email Marketing Rookie Kid Missing Steering Wheel

#9 Retail Marketing Stats You Need to Know

Do you know what the lowest e-commerce order generators are? In this post, we give you five tips and stats for mastering retail marketing. From social media, to mobile and beacon triggered emails.

Retail Marketing statistics mobile 2016

#10 Setting Up SparkPost as your SMTP Relay

You know you need to send email, but you don’t want to spend a lot of time or effort on it — you just want something that works out of the box. It’s not too much to ask! Many frameworks, languages, and tools come with SMTP support, but the last step is the most important – an SMTP server. In this post, we walk you through how to set up SparkPost as your SMTP Relay.

And that rounds out our Top 10 Blogs for 2016! Any industry trends or topics you think were under-represented? Leave us a comment below, or tweet us!

-Tracy

run a hackathon winter hackathon tips and tricks

Have you ever thought about organizing a hackathon at your company? If you have, we’ve put together some tips to help inspire you to run a hackathon of your own. Trust me – it is easier to organize than you might think, a lot of fun for everyone involved, and definitely worthwhile!

A SparkPost Tradition

Since 2013 I have had the honor and pleasure of organizing our company-wide hackathons which we hold twice a year. Our most recent one was earlier this month. These hackathons provide a fantastic opportunity for engineers to take a break from the normal day to day priorities of shipping new features and otherwise supporting our cloud service. In turn, they are able to experiment with new technology, explore a creative way of solving an interesting problem, or prototype the next killer feature. It’s a good morale booster and team building exercise, and we find it also helps with our recruitment efforts.

run a hackathon group picture

Hackathon Format

Our SparkPost hackathon is a two day event — we run from Thursday morning through demos and awards on Friday afternoon. We usually have about fifteen teams that add their group to the sign-up sheet. Each team has 3 or 4 staff members, though there are those lone wolf entries that can be quite brilliant, and exceptions for teams of 5 are possible. All members of the CTO organization are encouraged to participate – including technical staff associated with our various brands including Message Systems, Port25, and SparkPost – and it is left up the staff members to form their own teams. While the primary site is in our office in Columbia, MD, we have several teams that are partially or wholly remote which seems to work well. In some rare cases we will get an prerecorded entry if someone can’t stick around for the demos.

There are no restrictions on technology used in the entries–it’s all fair game. Examples of what teams have done:

  • write something from scratch
  • build on our existing code base
  • incorporate open source or a trial version of commercial software or service
  • use a new or familiar language

We use an honor system where no pre-work or coding is allowed! Research and brainstorming is OK to do in advance.

The hackathon format itself is quite simple. We have made some adjustments over the years but have settled on this basic schedule:

  • First thing in the morning we distribute hackathon t-shirts designed by our creative genius Lynn.
  • Teams can get started immediately afterwards.
  • Around 12PM we cater lunch for the office.
  • Work continues through the afternoon as late as the teams want and picks up again the next morning.
  • At 12:00 PM on the second day, everyone stops coding and can get lunch which is also catered.
  • Starting at 1pm, each team has 10 minutes, and no more, to present their work to the audience (including any Q&A).
  • After presentations, the judges deliberate for 30 minutes and then the awards presentations take place.
  • We usually wrap up by 4pm and invariably some staff head out for an early happy hour, while others go back to their desks to wrap up that last thing before the weekend.

Hackathon Demos and Judging

Presentations should include a live demo, but may include other communication methods such as a slide-deck, and often include some good humor. With 15 teams it’s important to be strict with timekeeping. A panel of three judges will pick the winners in 5 separate categories:

  • Product: A new or improved product or feature, including end-user tools and add-ons.
  • Tools: An internal tool or infrastructure experiment that can improve how we design, build, test, ship, or support our products.
  • Partner Integrations: A value added integration of our products/services with a third party
  • Fun with Data: making more of the data we have in our cloud environments
  • We Deliver: innovations in abuse detection, compliance, and deliverability

Though the categories are deliberately broad, we get the most entries under Product and Tools. We’ve experimented with themes like “mobile” in the past, but decided to let the hackathon remain as open ended as possible. While each team can suggest one or more categories, the judges are also free to re-categorize entries to make the distribution a little more even. This was the very first time that I was a judge and it was a lot of fun. The other two judges were veterans including Josh Aberant, our head of Marketing, and Laura Rose, who leads our awesome Technical Account Manager team. In the past we have had other great judges including the CEO, CTO, head of Product, head of Sales, and other leaders from throughout the company.

And the Winner Is…

All category winners receive a commemorative mug or water bottle. One of the 5 category winners is also selected as an overall “Best in Show” winner. Besides bragging rights, the “Best in Show” winning team gets the perpetual trophy cup which has their names engraved on it. The judges may pick a “Runner Up” or “Honorable Mention” for any project that didn’t win the category. The Best in Show winner this time was “J3” which included Jacob Kleidman, Jason Sorensen, and Jordan Nornhold (pictured below). This innovative team also won the Product category by building an multivariate “A/B” test API which dynamically sent emails to a sample set of recipients using two different templates and then – based on the engagement results of the samples – would pick the winning template to use for sending to the remaining recipients.

run a hackahon winning team

As for what happens post-hackathon, it’s left to the judge’s discretion if any of the winning ideas are considered for inclusion in any product. They will coordinate any follow up with Product as appropriate. There’s usually a fair amount of effort to convert an innovated hackathon product idea into a production feature. Often tool ideas find their way into use. All winning projects will get a 2 minute video feature at our upcoming all hands company meeting.

Run a Hackathon: Start Your Own Tradition

Our employee hackathons are a great tradition here at SparkPost. Our hackathons cultivate innovation and we have successfully applied learnings to how we build and support a better product. I hope you find this inspiring enough for you to start a hackathon at your company. Feel free to reach out on Twitter if you have questions or want advice on how to run a hackathon. And, as always we are looking for bright people to join our team. If solving problems with super smart people using awesome tech appeals to you – check out our open positions!

Chris McFadden, VP Engineering

Community Driven Development

The Community Driven Development Approach

April 2015 was an exciting time to be an engineer at SparkPost. Our service had just launched. We were seeing the results of all the hard work and dedication from previous quarters spent building our functionality, infrastructure, and teams.

Fast-forward to late February 2016. SparkPost had been generally available for ten months. Our community was growing steadily. We continued to make refinements to our service and infrastructure. We didn’t know it at the time, but MailChimp had just announced some important changes to Mandrill that would change the way we interact with our community and how we improve and support our product.

In the days following the announcement, we’d go on to set up our community Slack and revise our product and engineering processes to embrace community driven development. We used the feedback and experiences to drive our decision making processes to execute on what really makes our community successful.

Channels for information

Several mechanisms for feedback from our community exist at SparkPost. I mentioned our community Slack. There, our community has access to SparkPost team members and, more importantly, each other. We learn from you not only through direct interaction but by observing, listening, and taking notes. Any member of our team can take feedback from community Slack and post it directly to a feedback channel in our internal Slack. There, our engineers, product managers, and executives have direct access to your comments.

community driven development slack screengrab

Support tickets and Github issues are another essential method for understanding ways we can make our community’s lives easier. Our support team does a great job of escalating issues internally to our engineering team as well as bringing up issues that are occurring repeatedly and brainstorming ways to address. Github issues allow us to understand how our client libraries are being used and where we can make improvements.

Attending meet-ups, conferences, and hackathons has been a major source of learning for us too. We get to spend quality time face-to-face with our community, learn about how you use SparkPost and what improvements might help you and your business be more successful. After each event we make a point to share learnings internally so that we can discuss ideas and issues that arise during our conversations with attendees.

What happens behind the scenes?

Collecting all this valuable input wouldn’t mean much if we didn’t act on it. In the past three quarters we’ve acted on a little over 50 items that were a direct result of the information we gathered from our community. These smaller items are what we consider quick wins – things like updating language in our UI, improving documentation, bug fixes, and minor feature improvements. Here are some examples:

  • Click to see event details in Message Events UI
  • Allow customers to clone Templates in UI
  • Create code samples for cc/bcc in the client libraries
  • Surface all 4xx API errors in the web UI
  • Add information about rate limiting and error responses to the docs
  • Fix click tracking on <area> elements

Additionally, the engineering, product, and support teams meet weekly to discuss customer impacting issues, solutions, and action items. Like many software companies, we also have a prioritized list of larger initiatives that we work our way through. Some examples of work completed from the past few quarters include:

Prioritization

Prioritization of new features and improvements is a constant work in progress. We do our best to compile all of the great feedback we receive and maintain a balance between addressing current issues and the planning and execution of new functionality and improvements. Some additional items we’ve heard our community request include:

  • Deliver webhooks by subaccount
  • View the contents of sent emails in the UI
  • View billing history in the UI
  • Add the ability to use a test API key
  • Reporting by geography in the UI
  • Support for stored templates with subaccounts

Lastly, the best way to stay up-to-date with what’s new as well as fixes we deploy is to join the #announcements channel in our community Slack or keep an eye on our change log article. We update both every Friday when we deploy changes that we want to make sure you know about.

Onward & Upward

With 2017 nearly upon us, we’re looking forward to implementing some of the improvements and features you’ve told us are important to you. Our community plays a major role in the development and evolution of SparkPost. We love to hear from you, so keep the feedback coming. And keep pushing us to deliver the best service we can.

—Rich Leland, Director of Growth Engineering

ps: Thoughts on community driven development? How do your teams prioritize and process feedback? Leave us a comment below.

developer networking tips

There are currently 18.5 million software developers in the world. A number that is only steadily increasing even as you read this sentence. This is a positive reassurance that our profession is headed toward technological advancements that bring continuous change in the way we live and think. It’s also a testimonial of how competitive getting into the field can be. While there are tons of studies and stories of how careers advance by speaking to the right person at the right time, things can be a little more tricky for developers.

Here are the top 5 developer networking tips that have helped me the most in my journey as a young software engineer:

1. Being proactive

This has to be first on my networking tips list. I’ve personally witnessed the importance of always taking the first step. Recruiters meet with and speak to hundreds of developers about a potential job on a daily basis, so it’s easy to go unnoticed. You have to be the one to initiate that LinkedIn connection request, that conversation at an information session, or to start an email thread regarding your interest. Your attitude needs to project your passion for programming. Your enthusiasm should drive the conversation and reflect your ambition and drive.

2. Socializing in the right groups

The amount of programming you study is almost always inversely proportional to your extraversion. Half the battle is won when you market yourself right, but it may not be the only way of going about it in the developers circles. Instead, what takes precedence here is finding your fit and meeting more people like you. You will always find someone who knows more about a skill than you do. Some good places to start are hackathons, student clubs, meetup groups, and social media. It will only help you stay focused and get a little more knowledgeable about your career direction. And who knows, you might even meet your future employer in one of these places.

3. All that talk

It’s one thing to meet recruiters and engineers from a company, it’s another ball game to hold a conversation. In most cases, recruiters only have a few minutes to chat before they move on to the next in line. What I have learnt from experience is knowing what skills and qualities to project. Speaking about your personal project in intricate detail with an HR person does not make for interesting candidature. Prepare and practice exactly what parts of your resumé you would stress over in each conversation. Then estimate the follow-up questions and prepare for them too. Sound confident, not rehearsed.

4. Maintain your contacts

You may have met a recruiter at a career fair, or have a friend who introduced you to somebody at a company you’d like to work at. Ask to keep in touch. I know a lot of friends who met someone at a career fair and had everything going for them except that they didn’t have their email to reach out for a follow up. Not only is it smart to ask for email addresses or to ask to follow them on LinkedIn, it is also a way to make the otherwise detached process of networking a little personal. Touch base with recruiters you met last semester. Let them know your progress and passion for the field, connect with people by sending them personalized notes about what interests you, point out a specific conversation detail and look to build a relationship.

5. Keep swimming

It’s very easy to find yourself lost in this sea of developers who seem smarter than you. I only took to programming after I started grad school and was one of the odd ones that was either too late to the party or was a complete outsider. What really set me apart in the end was the fact that I did not give up. I know of people who took rejections too personally and stopped their applications. Sure the hiring process is tough to endure, but it is important to keep your head above the water. All those hours in the library, all those speeches in front of a mirror, all the ups and downs that you will face in the process were only put there to make you better. And in the end, guaranteed, it will all pay off and it will be worth it all.

Although the ways to go about networking may have changed, it still remains an essential ingredient to make a career move. No matter how many software engineers this world churns up, the significance of a productive human interaction would still remain.

Do you have any developer networking tips that you want to share? Let us know on twitter or slack!

–Sneha Vurrakula

Suppression List API Updates Gears Laptop

New Suppression List API Functionality Coming Soon

At SparkPost we’re constantly using feedback from our community to enhance our application and APIs. Based on what we’ve heard, we want to let you know about some new functionality that will be coming to our suppression list API in early November 2016.

  • We’ve heard you loud and clear and will be removing rate limiting on all Suppression List API calls. We’re improving the underlying data storage for this API, which allows us to remove the rate limiting that was put in place to guard against poor performance.
  • You will now be able to retrieve your entire suppression list. Each GET call to the API will include pagination so you can page through the results.
  • You will be able to search for suppression list entries by domain or description. For example, searching for all suppressions from gmail.
  • The PUT method on a single recipient is returning. You will no longer have to do a PUT to /api/v1/suppression-list and provide an array. Instead you can now do a PUT to /api/v1/suppression-list/test@example.com to update the test@example.com resource directly.

There will be no change in functionality for the existing create, update, and delete endpoints. While we always make our best effort to prevent breaking changes, we had to make an exception and will introduce a breaking change for GET calls to the Suppression List API.

Currently, GET calls return a response where each recipient contains two fields – transactional and non_transactional:

With the updates to the API, GET calls will now return a response where a recipient will show up for each type of suppression. For example, if a recipient is suppressed for both transactional and non-transactional emails, they will show up in the list twice with transactional or non-transactional set to true.  We also added type to reflect the boolean – set to either transactional or non_transactional:

Why we are making this change

Our initial implementation has several limitations:

  • Requests to the Suppression List API are rate limited.
  • Downloading the entire suppression list is not possible as the number of records returned is capped at 100,000.
  • It is not possible to distinguish why a recipient was added to the suppression list (e.g. hard bounce or spam complaint) for recipients that were suppressed for both transactional and non-transactional emails because there is only a single entry for each recipient. This data is contained in the source field. The same can be said about the updated and description fields.
  • It is not possible to make a PUT call to update a single entry in the suppression list.
  • The Web UI is limited. It does not load an initial page of suppressions and you cannot download your list.

Currently it takes several hours for a recipient to be suppressed and we agree this isn’t a great experience. In order to support a more real-time suppression, we swapped out the backing database and changed the data model to tune it for real-time lookups. With separate entries for each recipient for each type of suppression, it allows us to support more granular suppressions by more than just transactional/non-transactional flags.

Who’s affected

This change will only impact users that have suppressed a recipient for both transactional and non-transactional messages. More specifically, have code that make GET calls to the Suppression List API and are not iterating over the entire array to check the transactional/non-transactional booleans. Although we cannot read everyone’s code that has integrated against this API, we feel pretty confident that the number of users affected are very low, if any at all.

Necessary actions

If you have code that parses the response of the GET endpoints /api/v1/suppression-list or /api/v1/suppression-list/foo@bar.com, you may have to add some additional code to iterate over your array to ensure you are parsing both the transactional and non-transactional records. If you are using subaccounts, you may already be iterating over the result but you may have to add additional logic to check for the transactional/non-transactional booleans. You can view the updated Suppression List API documentation with the improved response format here.

Making a tough decision

At SparkPost, we take backwards-compatibility very seriously. We don’t make these types of changes lightly and appreciate that our community has code in production using our API. We wanted to version the API but in this case it wouldn’t be feasible because the entire underlying architecture was changing. As a result, Taking all of the above feedback and issues into account, we believe the updates will provide a much better experience for our community in the long run.

Finally, if you’d like to discuss this update, or anything SparkPost related, please feel free to join us on our community Slack channel or reach out to us via Twitter.

—Bob Evans
Director of Application Engineering, SparkPost

How-SparkPost-Built-An-Email-API_Dev-Blog_600x300_0716

Using community feedback to build the best email API

Steve Jobs once said, “Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple.” This is certainly true when it comes to building a simple and easy-to-use API, which for SparkPost is the primary user interface that developers interact with. So how did we create the best email API that developers love? Read on to learn some tips from the SparkPost Engineering team.

RESTful Architecture

One of our first decisions was to adopt a RESTful architectural style. Representational state transfer (REST) based web APIs have been used since the early 2000s. In the last several years, it has become the de-facto standard of cloud APIs. RESTful APIs, including the SparkPost API, has a collection of resources (https://api.sparkpost.com/api/v1/sending-domains/mydomain.com for example). These resources, described using JSON, allow clients to interact with it over the network using the standard HTTP verbs (GET, POST, PUT, DELETE), and get standard HTTP response codes (200, 500, 400, etc ) back.

The RESTful architecture style improves the simplicity, portability, interoperability, and modifiability of an API. Using HTTP and JSON makes it easy for developers, regardless of which programming language they use – PHP, Ruby, and Java – to integrate with our API without knowing our underlying technology. Other styles that were more common in the past like SOAP and JSON-RPC are generally obsolete. Today, more developer-friendly RESTful APIs replace them.

API Governance

Important as it is to decide on an architectural style, there are many choices to make when implementing the API’s interface. Oscar Wilde once said, “consistency is the last refuge of the unimaginative,” but if your choices are not a consistent set of conventions, then the API will be very hard to understand and use. At SparkPost, we have multiple teams working on different parts of our API. Within those teams, people work on different APIs at different times. Therefore, it’s very important that our API is consistent in the use of conventions. To achieve this, we established a governance group including a few engineers representing each team and a member of the Product Management team. This group is responsible for establishing, documenting, and enforcing our API conventions across all teams.

API First

We made a decision early on that our entire application would be built “API first.” This means that we did not bolt on our APIs to the application after the fact – the APIs are the application. Even our web user interface uses the APIs. It’s a client-side Javascript application which makes calls against the same APIs developers integrate directly against. This sometimes means that we build and publicize the API even before there is equivalent functionality in the web UI. To bridge that gap, we provide Postman collections for all our APIs so you can easily interact with all the functionality they provide.

Documentation First

When we build a new API endpoint or enhance an existing one, we always document and peer review changes first before a single line of code is written. This approach helps enforce our conventions, keeps things consistent, and ensures the API is very user friendly. It also cuts down on the amount of rework, bugs, and breaking changes. We use an API documentation markdown language called API Blueprint and manage our docs in Github. We encourage our community to contribute and improve upon these open source docs. Initially, we published our docs to Apiary, a great tool for prototyping and publishing API docs. However, embedding Apiary into our website doesn’t work on mobile devices so we now use Jekyll to generate static docs. Eventually, we may switch over to Swagger (now Open API) as the markdown format since this new standard continues to gain traction.

Technical Excellence and Transparency

We have a strong engineering culture and are proud of what we build. It is not sufficient to just have an idea for a great API. We need to build APIs quickly, iterate on them rapidly, and verify that they’re reliable and stable. For this reason, we embrace Continuous Delivery methodologies so that we can make changes and deploy them quickly and reliably. It is important to developers that we provide a rock solid service.

We peer review all API code changes, run them through unit and functional tests, periodic performance tests, and automatically deploy and smoke test them. We roll back changes automatically without impacting our users if any failures are detected. We also have extensive monitoring using a variety of tools including Circonus and OpsGenie. Thanks to these tools, we are immediately alerted when there are availability issues that need attention. When production issues affecting customers arise, we can respond quickly since we have engineers on-call 24/7. We also use our Status Page to keep our developer community updated.

Listen and Respond to the Developer Community

When we first released our API, it was quite minimal, supporting the most important features of sending email via API and SMTP as well as some reporting with the Metrics API. We collected feedback through customer interviews, user testing, surveys, and developer conferences early on. Members of our developer community recently sent us a lot of feedback on our Slack channel, which we launched in response to the Mandrill announcement. Subaccount support and IP pool management and our developer friendly freemium pricing are examples of improvements to the API in response to developer feedback, and on a weekly basis we make many improvements based on suggestions from the community.

Developer Relations

We have a small team of developer advocates that focus on listening to developers and improving the developer experience. This team helps maintain our API docs, client libraries, and community Slack channel. In addition, they attend and sponsor developer meet-ups like CharmCityJS, hackathons such as Bitcamp, and conferences including Fluent. This team also acts as internal advocates for the developer community and helps drive improvements across the service including the APIs and our support process. Beyond the small team, we have many who contribute on a part-time basis to the developer relations mission.

APIs Are For Integration

While RESTful APIs are easy to integrate against with any language, it is very convenient to have a client library that you can use to get up and running quickly. We are proud of the wide range of client libraries and integrations available for SparkPost. These include Node.js, Java, Go, Python, and others listed on our Developer Hub. We also have integrations for WordPress, Zapier, Heroku, and have several community contributed libraries for Ruby and C#.

Versioning and No Breaking Changes

We released v1 of our API, with the public beta of SparkPost, upon two deliberate decisions. The first is that versioning will be in the URI rather than as a header. This is easier than requiring an API or schema version to passed in as a header. Secondly, by explicitly putting the version into the URI, we committed to no breaking changes to the API. Breaking changes occur when the behavior of an API call changes, which breaks a customer’s integration. We take this seriously and prevent them through code reviews and automated tests. However, we can and do extend the API by adding new endpoints, parameters and inputs, and response values. Will we ever need a v2 of the API? Perhaps down the road when we wish to change the whole semantics for the API. However, we don’t foresee a need for that soon.

Don’t Forget About SMTP

We know that most developers still like the ease and portability of using SMTP to send their transactional or marketing email. Therefore, we added the ability to have much of the same functionality over SMTP as you have with the RESTful API. The X-MSYS-API header allows you to pass in campaign, meta data, and control data such as turning on engagement tracking or setting the IP Pool. The SMTP API also supports subaccounts.

 

I hope you learned a few things from how SparkPost built the best email API that developers love. Take what you learned and apply it to your own API development. Do you love building awesome APIs? Come join us, we are always hiring.  If you have any questions or comments please mention us on Twitter or find us on our community Slack channel.

—Chris McFadden, VP Engineering