DKIM Validator

DKIM Validator by SparkPost

You might have heard about our shiny new DKIM Validator, since we announced recently in our newsletter. In this post, we’re going to show you how we built it.

We talked about the “why” and “how” of DKIM in Ewan’s blog post last week. I’m guessing that a very common reaction was “it’s complicated”, which it is, because crypto. We’ve written before about why DKIM exists and how it works, even publishing a shiny infographic.

So let’s dive into the details. How did we build our DKIM Validator?

It Takes Two to DKIM

DKIM protects messages between Alice and Bob from Mallory, who wants to modify them. When Alice sends a message to Bob that includes a DKIM signature, Bob can prove the received message is exactly what Alice sent. Proving that a message hasn’t changed is a similar operation to signing it. However, it’s different enough that you need to use different tools.

We use quite a lot of Node.js under the hood of our service. As it turns out, there wasn’t a Node.js module that could verify messages signed with DKIM. Several options existed for signing messages. So we picked the one that seemed most active, with the most thorough tests, and got to work adding what we needed. We made sure to return intuitive, diagnostic-quality error messages when validation fails, since our goal is helping you fix the right problem when something’s busted.

the server answer is meaningless

Testing code involving crypto is hard. Validating messages with the same module used to sign them would result in a sort of echo chamber effect where bugs get ignored, because they’re our bugs. We settled on signing messages using OpenDKIM’s testing tools, then validating those signed messages in our test suite to avoid as many echoes as possible. That ended up resulting in some pull… er, merge requests to OpenDKIM to make their testing tools able to easily generate a wider variety of signatures.

SMTP, API, UI, Oh My!

Since SparkPost runs primarily on Amazon’s cloud, we quickly spun up some PostgreSQL RDS instances to store generated test addresses, and any associated DKIM verification results. We use flyway to apply any changes to our database schema. Bamboo deploys the API components to an existing tier of servers, which listen for HTTP requests coming in from the tool’s interface.

there is no cloud sad face

With the hard part out of the way, next we wired our shiny new DKIM validator up to a SparkPost account, because dogfooding your own service is good for everybody. That account has an Inbound Domain and Relay Webhook configured to accept your test messages via SMTP, transform to JSON, and pass them through to the validator’s back end for storage and processing.

Validating your DKIM configuration with a test message is simple:

  1. Generate a test address
  2. Send mail from the system being tested, to that address
  3. View results (on failure, fix & GOTO 1)

And that’s it!

With a few clicks, including one on a “send” button, you’ve confirmed that your messages are being DKIM signed! And because they were sent only to us at SparkPost, you didn’t have to worry about any damage to your domain reputation if there was an error. Mailbox providers will now see your DKIM signatures and know it’s you sending that email. And no, nobody injected any spam or malware-ified any links along the way. Because if they try, DKIM protects your subscribers and customers by raising a red flag.

So go ahead, test your email with our DKIM testing tool – you know you want to! If you have any questions about DKIM, or email, or ideas for improvements, tweet @SparkPostDev. Feel free to also join our slack community!

Happy testing!

–Dave

Email Tools in an API World Laptop w Gears

Modern Email Tools Available To You

Email has a long heritage of standards and specifications, many of them are truly great and some are even followed. Implementing these standards requires considerable knowledge. Tools ease our work by encompassing that knowledge, packaging it up and making it available to all. One strong measure of a tool’s worth is its effectiveness at embodying knowledge and reducing cognitive load on the wielder.

The world of email tools is almost as rich as its body of standards. We have myriad services, applications, libraries and commands for configuring, testing, orchestrating and tracking our various messaging activities. The email industry is hugely advanced in the orchestration, delivery and tracking spaces, since those are where the broadest commercial interest lies. Unfortunately, tooling for email service configuration and testing is less advanced, possibly since those tools would merely improve engineers’ lives.

Suffice to say, there is still considerable knowledge required to configure, verify and troubleshoot one’s email estate.

Let’s look at a typical setup task that most email senders have grappled with: configuring DKIM signing for an email sending domain. DKIM allows you to take responsibility for the messages you send by associating your sending domain with your mail. Don’t worry too much if you don’t 100% follow all these steps. The intent is to show level of effort.

DKIM Setup: The Old School Way

Here are some outline steps we might follow to start signing our email.

  1. Generate an RSA key pair:

2. Extract just the public key:

3. Form our DKIM DNS record from public key:

4. Have our DNS provider publish the DKIM record under

5. Configure our email service to sign using your private key.

With that done, these ancillary points are left as an exercise for the reader:

DKIM Setup: The SparkPost Way

Ok, let’s try that again, this time using the SparkPost API.

  1. Ask SparkPost to create our sending domain and a matching DKIM key pair
    (we can also do this in the app):

2. Form our DKIM DNS record using API result
(Hint: SparkPost will show us the fully-formed record here):

3. Have our DNS provider publish the DKIM record

4. Verify our setup by clicking the DKIM record test button for your sending domain in SparkPost

Side note: Here are all the tasty the details on using SparkPost to manage and verify sending domains and also using the SparkPost API sending domains endpoint itself.

Much Better…

Now we’re a tiny bit biased but we do love our email API and it certainly reduces the user’s cognitive load here. Questions we no longer need to worry about:

  • What size should the RSA key be?
  • What even is an RSA key pair?
  • How do we invoke the openSSL toolkit correctly?
  • How do we handle our private key material safely?
  • How do we format our DKIM DNS record?

That single API call executes sending domain configuration, creates our DKIM keys and formats our DNS record all at once. There’s less to do, less to understand and fewer places to stub our toes along the way.

…But Not Perfect (Yet)

…and yet there are further submerged rocks to run aground on here. The DNS record could be fat fingered during publication, the DNS provider may truncate the record and what do you do if there’s an existing DKIM DNS record for your domain? How do we diagnose each of these and how do we verify the end-to-end DKIM signing and verification process?

These are all issues and questions we’ve seen in production use. Clearly, there’s more we could do, as a tool-loving developer community, to help each other be more reliably successful, faster and just plain happier in our work. Just as SparkPost has an email API, DNS services have record management APIs. Combine these with the modern web and we could build far better experiences for each other. For starters, how about:

  • A DKIM verification tool explicitly designed for diagnostic and remedial use (stay tuned for next week’s post!)
  • A modern validating SPF record editor
  • Open DMARC as a service

Against a background of modern API-driven workflows, email tooling looks a generation or so out of date. In the SaaS email infrastructure community, we can do better. Isn’t it about time for a new generation of messaging tools?

If you liked, hated, agreed, disagreed or just want to chat about email infrastructure, tooling, APIs or anything else, come join our burgeoning Slack Community. We’d love to hear from you.

—Ewan

Clipboard Briefcase Clock image Developer Productivity Tools

Developer Productivity Tools You Might Like

Staying productive in a modern software firm can be a challenge. You might be physically remote, in a different timezone or maybe you just have an excitingly active user community. Whatever your situation, we each have productivity tools, techniques and coping mechanisms for maintaining momentum in the modern world.  I often learn from others on this topic so here is very quick overview of my own thinking.

Slack

Slack is one of the most useful developer productivity tools available to me today. Slack’s modern grown-up take on IRC makes traditional corporate IM completely obsolete. I could not function without live group comms with all my colleagues and our user community. Our founder and CTO George Schlossnagle says this and more here.

I also can’t imagine, due to Slack, how many fewer hours of meetings, memos, dry papers and otherwise slow-boat conversations we now have, but it’s alot.

an alot developer productivity tools blog Image c/o: an alot

My home within SparkPost is the Developer Relations group. We’re spread over 8 time zone hours and some of us move around quite a bit. Slack’s backscroll, the body of messages sent while I was away, lets me catch up when travelling, eating, asleep or whatever else. There is no substitute for backscroll to stay connected and in sync with the team.

So much of the work we do is visible in backscroll and it’s searchable too! I sound a little like an advert but Slack is an unprecedented new capability for me. I can now answer a great many of my own questions with Slack search – thats huge win for group productivity. I’ll touch on that a little more in the anti-patterns section later.

Of course Slack is not a replacement for deliberate documentation but it certainly raises the knowledge floor, does it not?

Calendar

Ok, yes of course we all have calendars. We use Google Apps at SparkPost and Google Calendar helps my productivity in a few specific ways. I like to put solo tasks in my calendar. This lets me block out my days and remind “future me” (read: confused and lazy me) to do things on time. For example, I have a calendar entry entitled “Draft blog post” for writing this article with a couple of hours allocated. I leave the calendar notification on screen while I work (I use Chrome) so I can see it right beside my clock.

calendar reminder to draft blog post developer productivity tools

I reschedule tasks as needed but I try to work as dictated by my calendar. “Past me” had a plan to make progress on multiple fronts and keep things rolling along.  I can also reduce my cognitive load by separating the planning from the doing. In an convenient twist, Google Calendar’s “find a time” feature also meshes my tasks with actual meetings.

Anti-Patterns

Alright. So far, so obvious. I have also identified a few productivity anti-patterns (behaviors that negatively affect my performance) which I try to avoid or at least manage. Here they are now.

Slack (again)

I know. I just advertised Slack like I was getting paid for it but I also feel it’s possible to be too connected. In larger teams, Slack lets you watch the whole organization function in real-time. That can feel a little like staring directly up the Niagara Falls and it’s as alluring as it is unproductive.

In response, I have learned to limit my channel membership, to make liberal use of channel muting and to re-think things when I’m reading backscroll for longer than a few minutes at a time.

Twitch Culture

Have you ever heard: “Can you just quickly do X for me? It’ll only take a moment.”?

Our lives are made up of moments and they’re all precious. Technology can also exert unwanted pressure to be ever available and responsive throughout the working day. For tasks in engineering, education, research and writing, this “twitch culture” impacts our performance because interruptions affect our flow. They can be jarring and difficult to recover from.

As a coping mechanism, I tend to just be less available during intensive task periods (like now). It takes a little getting used to but it works. I also try to answer my own questions (Slack search, lmgtfy, …) before interrupting others. It’s a balance so I’m sure everyone’s is different.

We live in a time of explosive growth in information availability. With a little awareness and more of these excellent developer productivity tools, I hope we can all continue to have rich, productive, interesting and happy lives.

Productivity is an extremely important, ever changing and yet very personal topic. We’ve been writing about it throughout July and frankly we’re exhausted. Leave us a comment or tweet about your own developer productivity tools, tips and techniques.

– Ewan

3 sparkpost tools to improve productivity

Email Testing Tools: 3 Handy Tricks

Signing up for email newsletters is fun. I usually spell my email address correctly. Sometimes, when I don’t, someone else with a clever typo in their email doesn’t understand why they’re getting the messages I signed up for. With SparkPost’s handy testing tools (and double opt-in), be protected from Mayhem like me.

We protect our users by detecting negative feedback from email receivers and automatically suppressing future messages from your account to those addresses. It can be difficult to test your integration since SparkPost only emits the events in question under specific difficult to reproduce circumstances. Let’s see how to use a couple of simple tools that recreate those test conditions for you to avoid future Mayhem by testing to make sure you’re applying those suppressions to your list as well.

Negative Feedback

Two types of negative feedback (AKA Mayhem) that can be tricky to test are 1) out-of-band (OOB) bounces, where a receiver initially accepts the message then later sends a bounce notification and 2) feedback loop (FBL) reports which typically mean a recipient clicked a “this is spam” button.

Clicking “this is spam” on your own messages is dangerous because that can damage your sending reputation and composing an OOB bounce is tricky to get right. But don’t worry, we’ve got you covered with some, you guessed it, handy tools!

Simulating Negative Feedback

The first step of testing both types of Mayhem is to send yourself a message through SparkPost. We’ve got a tool for that too! Once you’ve got the message in your Inbox, save the message, including all of the headers, and deleting any blank lines at the very top. You’ll be using this file shortly.

Both of these tests will add the receiving address to your suppression list! Afterwards, you’ll need to manually remove the address to continue to receive emails from SparkPost. I usually use our Postman collection for this. Here are some instructions showing how to get that up and running.

Now, unless you’re connecting from somewhere that blocks outbound connections on port 25 (behind a firewall, residential ISP), you’re all ready to test! The install instructions for both tools suggest some ways to get around blocked ports.

Fake an FBL

The fblgen tool lets you generate and send an FBL report. Here’s a dry run:

Running the same command and adding the –send flag will attempt to deliver the message to the detected MX. This will trigger an FBL event of type spam_complaint, which will flow through to message events and any configured webhooks.  This tool will also cause an increase (of 1) in the count_spam_complaint metric and the “spam complaints” value shown in the SparkPost summary report.

Instead of waiting around for an FBL or risking damage to your sending reputation by triggering one yourself, this tool gives you a way to predictably trigger an FBL event from your own systems.

Bogus OOB Bounces

The oobgen tool lets you generate and send an out-of-band bounce message. Here’s a dry run:

Again, running the same command with the –send flag will attempt to deliver the message to the detected MX. This will trigger an OOB bounce event of type out_of_band, which will flow through to message events and webhooks.  This tool will also cause an increase (of 1) in the count_outofband_bounce metric and the “out-of-band” bounce value shown in the SparkPost bounces report.

Bonus Tool!

Once you’ve gotten a collection of events in JSON format, perhaps by using Postman and our message events endpoint, what’s the easiest way to do some analysis on that data? If you’ve made it this far, then you’re probably comfortable on the command line, and the answer is to use jq!

This handy tool can extract the parts you care about from a big blob of JSON, for example counting the number of events of each type that were received in the last two hours:

Conclusion

With these handy email testing tools, testing those uncommon, hard-to-trigger events is now easy. Making sure these types of events are handled correctly in your integration with SparkPost will help you prevent future Mayhem from striking, by keeping your list clean.

Did you get a chance to try out any of the tools mentioned in this post? Let us know if and how they fit into your testing processes, and feel free to create an issue or submit a PR on Github if there are any other features you’d like to see.

—Dave

Twice a year, SparkPost hosts an employee hackathon. The idea is to give our hard-working technical staff a chance to step outside their usual areas of focus and to have free rein to explore their creative ideas that push our technology in all sorts of cool directions. These demonstrations, and the exercise itself, are fun and really inspiring to us all and an important part of our culture of innovation.

During the hackathon, our engineering and technical operations teams spend a day and a half playing with code and turning ideas into working proof of concepts. Though exploration is the main goal, many of the ideas end up being refined and incorporated into our products or to our internal toolsets.

As usual, the Fall hackathon teams impressed us with their orthogonal thinking and technical chops. The group produced a crop of innovative ideas that show the depth and breadth of our capabilities. Let’s take a look at some of the great hacks that came out of the event this time around.

Guidelines

Each hackathon event has a theme with several areas of focus. Our Fall event featured the following categories:

  • Product: a new or improved product or feature, including end-user tools and add-ons
  • Tools: an internal tool or infrastructure that can improve how we design, build, test, ship, or support our products
  • Partner Integrations: a value added integration of our products and services with a third party
  • Fun with Data: making more of the data we have in our cloud environments
  • We Deliver: innovations in abuse detection, compliance, and deliverability

The rules are simple: use whatever tech you want, research as much or as little as you want, and don’t write any code ahead of time. The hackathon starts at 10 AM on day one and ends with presentations starting at 1 PM the following day. A panel of judges reviews the entries and picks a winning team for each category.

Results

Best in Show

Our judges award best in show to the project that stands out to them the most. In this hackathon, it was a proof of concept for a new iteration of our Adaptive Delivery functionality, which auto-tunes outbound email delivery parameters and traffic shaping in real-time to avoid blocks, safeguard reputation and optimize delivery.

This functionality is currently a part of the core technology that powers SparkPost—the Momentum platform. The project used a combination of Vertica, Redis, and Node.js to power the rules system for Adaptive Delivery, allowing us to offload some of the work to more efficient services. The main benefits here are scalability, performance, and abstraction. Another potential benefit could be sharing Adaptive Delivery data to improve deliverability for all of our customers.

winners-best-in-show

Product

For our product category, two projects shared top honors: an in-application API explorer for SparkPost, and a hot backup for our PowerMTA product. The interactive discoverability made possible by in-application API explorer was so awesome that we’ve added it to the SparkPost UI! This interactive tool complements our traditional API documentation.

explorer-2

 

The PowerMTA project did a proof-of-concept system that enables one instance of PowerMTA to be the hot backup for another instance. Also addressed was a potential solution for recovery from failures when using hot backups. The team relied on their knowledge of database replication and SMTP to build the replication protocol and used TCP socket programming to implement the details.

Tools

The tools category produced a project allowing us to automate testing against our SparkPost Elite environments. Our tech ops team routinely spins up new SparkPost Elite customer environments with the requisite architecture and needs to test that everything is set up correctly and ready to send. To aid in automation of this process, one of the teams created autoscott, a command line utility that allows teams to run commands to perform actions like sending a single transmission, sending multiple transmissions, creating recipient lists, creating templates, and more.

Fun With Data

In the fun with data category, we had a team create a prototype for a weekly report card. This report would be delivered to our customers and contain a summary of their sending including number of emails sent, open rate, and click rate. It would also deliver helpful tips on how to improve your mailings based on trends in data. For example, if your open rate went down week-over-week, we’d let you know how to potentially correct this issue. Additionally the email contains a breakdown of volume by day for the past few weeks and some information about your sending habits as they relate to other senders on SparkPost.

Partner Integrations

Signing up for and signing in to SparkPost is simple. The winning entry in our partner integrations category wanted to take this a step further, so they built the ability to sign up and sign in via Github, Twitter, or Google. The project made use of the Node.js library passport and the adapters for GithubTwitter, and Google. As an added bonus, there is also a passport adapter for SAML, which we’re using to power our upcoming SSO functionality for our SparkPost Elite customers.

We Deliver

Deliverability is a big deal at SparkPost. We pride ourselves on keeping the bad actors out and the good actors sending reliably. In our we deliver category, the winning team worked on refining our ability to scan outgoing email for spam. One particular enhancement was a performance optimization to make scanning of large transmissions more efficient. This group worked using the fine CSDMC2010 spam corpus, which includes a training collection of labeled spam and ham emails, as well as an additional unlabeled test set. The group also worked to visualize the resulting data using SumoLogic, a commercial product which we use in our data analysis.

winners-we-deliver

Another Successful Hackathon

Hosting employee hackathons has become a great tradition here at SparkPost. Our hackathons support our culture of innovation…and ultimately help our customers succeed. As you can see, our teams are constantly striving to improve our product and our tools. The Fall hackathon was no exception.

We’re always looking for bright people to join our team. If solving complex problems, using awesome tech, and participating in Hackathons sounds fun to you, check out our open positions.