user agents metrics identify user groups

We love our developer community and I personally love learning about you. How you use SparkPost, what tools you favor, where you are and what you need. In a recent post, we took a brief peek into the software stacks you collectively use. This time, we’ll go a little deeper into a one of the key community data sources that we follow: the User-Agent HTTP request header.

For those familiar with it, the User-Agent header might not seem like a very rich source of information. I often find it easy to go a little non-linear while analysing large data sets so it’s instructive to stick to just a single bite-sized chunk of data at a time: in this case, your user agent strings.

What’s In User Agents?

When your app hits the SparkPost API, it can optionally include a User-Agent HTTP header in its request. User-Agent includes some details on your HTTP client, your software stack and occasionally even your host operating system. Web admins commonly use it to learn about what browsers people use on their site. However, it’s also very useful with a developer-centric email API service like SparkPost because it shows what client libraries you all use with which versions, and some other interesting tidbits too. With that out of the way, let’s dig into some data.

Client Library Usage By Version

We like to follow the usage patterns of our “Big 3” client libraries: PHP, Python and Node.js. You can see how they stack up in my last post on the topic. We find it useful to break down client library usage of each one by version so our library maintainers can see the spread of versions you all use.

We’ll look at PHP first because it forms the largest single group with 44% of transmission requests in the last week.

php sparkpost client library usage

Most requests come from php-sparkpost/1.x with only 13% coming from our 2.x series client which was released in June. The 2.0.0 release introduces a refactored client library with a new interface so it’s possible that mostly new PHP users will pick it up since 1.x still functions nicely and is presumably integrated with many customer stacks at this point. Still, those folks will also get a nicer experience. PHP was the first client library to bump to 2.0 – uptake is farthest along there.

Python is next on our leaderboard with 33% of transmission traffic. Python users are more spread across versions with about half using either last or last-but-one releases.

python sparkpost client library usage

Last but by no means least, we have node-sparkpost where version 2 was released just a few weeks ago.

node sparkpost client library usage

Node.js users seem to keep up quite well with client library releases. Even though node-sparkpost/2.x was released 2 weeks ago, we are already seeing traffic from it. Given that our Node.js client is 2 years old (happy birthday node-sparkpost!), the 2.x release was overdue so if you’re a Node coder, go check it out. With features like in-built promise support and a slew of cleanups and usability improvements, we predict some quick uptake here.

It’s Not Just The Software

Beyond identifying our own client libraries, there are other useful details in our trusty little User-Agent header. For example, we can identify subcommunities or at least users with common needs and once we do that, we can learn about their needs and how best to support them.

One such subcommunity we recently discovered is the Google App Engine user-base. After fielding questions from the occasional community member, we found that App Engine users live under quite specific constraints. For instance, python App Engine users need special attention due to the nature of the App Engine python environment. Also it’s interesting to see that App Engine adds its own details to its customers’ outbound HTTP requests. That means we can gauge popularity of the platform with our users. On the back of these discoveries, we’ll be paying close attention to our App Engine user-base from now on.

The Imperfection Of One-Dimensional Data

No data set is without its foibles and ours has some ‘unique’ features worth mentioning. We collect User-Agent information from calls to /api/v1/transmissions since we use it to send emails. Interestingly, a good half of requests to transmissions have no User-Agent header at all. That privacy is your right but it does reduce our sample size so we’re calling it out here.

Further, we’re comparing call volumes with a given User-Agent string. It’s tempting but not 100% correct to consider “call volume” as a proxy for popularity.

For instance, if we see 1 million calls marked with ‘User-Agent: captain-clown/1.0’, we might assume the captain-clown HTTP client to be quite popular when it might just be very inefficient. Maybe those 1 million calls could be condensed into 2 or 10. This case is exaggerated to show the issue. Call volume as popularity still has some value, especially across groups of User-Agents as we have done in this article.

Summary

I hope you’ve enjoyed our brief look into the SparkPost community. Our community is hugely diverse, using widely varied tools and technologies and with an exciting and ever changing spread of business models. We’re still discovering our community’s make-up and looking for ways to serve more people, better, quicker. We also strive to be as open and welcoming as possible. Want to learn more about our community, share your own experiences, or draw our attention to your own subcommunities? Come find us on Slack or fire us a tweet.

–Ewan

client-libraries_600x300-01

I have an old-school pocket dictionary at home (literally, old school… it’s the Primary School Dictionary of the English Language from 1874). As a child, I was fascinated with the “pictorial illustrations” on the back. Everything from flags and musical instruments to animals and forms of punishment. My favorite page, however, was always the “Vehicles for Land and Aërial Locomotion.”

screen-shot-2016-10-28-at-12-59-41-pm

It was always fascinating to me how differently everyone chose to travel. They might all be going to the same place, but how they got there was the true story in my mind. Was it a dog-cart, and how big were the dogs?!? Or a Calash, which always resembled Cinderella’s pumpkin carriage in my mind, or maybe even a balloon? My imagination could go wild, and it often did.

The journey-story continues to intrigue me, to this day, and that’s something that we at SparkPost understand. We know that not all of you are coming from the same place, or even trying to solve the same problems. And we certainly know that not all of you are savvy when it comes to JavaScript, and we don’t expect you to be!

We want you to be as successful as possible in all of your email-sending endeavors, with whatever stack you happen to be using. So this is where our client libraries come into play. They’re designed to drop into your existing environment and allow you to access our API in a native-to-you programming language, meaning your lives are presto chango easier (almost) instantly! (You’re welcome 😉)

Since we know how important this service is to you, there’s a lot of process that goes into these libraries. We make sure that they’re tested and ready for you to use on a regular basis. I sat down with a few members of our team to find out about this process and the thought that goes into developing these libraries.

Why Client Libraries?

If you’re like our Developer Advocate Aydrian Howard, you live and breathe JavaScript, which means interacting with our API is fairly straightforward. But let’s say you prefer Python, Java, or some good ol’ PHP.

“It’s really easy for someone using JavaScript, since we pass JSON to our API,” said Aydrian. “But not every programming language has that concept of JSON, so we do that manipulation and pass it over. It makes it more like a function call and less like an API call.”

This means no matter if you’re comfortable using JavaScript or not, you can be up and running with SparkPost quickly and easily.

easy peasy lemon squeezy graphic

But all of this takes work, and we want that to be on us, not on you – easy peasy, remember? Early on in our client library days, we established a set of guidelines for testing, samples, code style, and an explanation of how you, our community, could contribute (wink wink, nudge nudge). These guidelines set us all up for success. For example, making sure that everything is 100% tested before it’s released. They also help make the process easier for you. Our requirement for consistent code style,  based on community best practice, keeps our libraries readable and, thus more maintainable.

If It Ain’t Broke…Right?

When the libraries were first created, each author decided how they would interface with the SparkPost API. Did they want it to be lightweight and simply a wrapper? Or did they want to offer a variety of conveniences for the user, flattening out the parameters? It was the wild west, and we were grateful to any and all who stepped up to the siren call of client libraries.

But then the feedback started coming…

stampede horses

Rich Leland, Director of Growth Engineering, said, “We put our first cut of these libraries out just after having a product launch, and quickly ran into a good problem: we had so much community feedback that we couldn’t reply fast enough!”

We quickly learned that a thin wrapper allowed us to maintain the libraries more easily, as well as keep the libraries relatively lightweight for the community. The heavier libraries were bogging us down. They forced us to completely change each of the client libraries every time we made any changes or additions to the API.

Software Engineer Avi Goldman joined our team just after we noticed this exact problem occurring with our PHP library. He spent time identifying the good bits and then worked on refactoring the rest of it.

“There was already a really good framework for testing,” he said. “We settled on a simple request function and everything revolves around that one central point. Now it’s simply a thin wrapper with some syntactic sugar around the things that are difficult to do, like cc and bcc.”

Basically, there’s now a base resource that can be used to call out to the SparkPost API with a one-to-one mapping that’s consistent between payload parameters in PHP-land, and what is listed in our API documentation. In other words, we’re back to easy peasy lemon squeezy!

Our End Goal: Your Happiness

Ultimately, our goal is to make the lives of our community member’s easier – no matter where they’re coming from. With that goal in mind, we’re always working to improve our API as well as the client libraries. In addition, we’re building various tools and resources to help you along the SparkPost path.

So, what vehicles… err, languages, do you use to get to SparkPost? And how can we, as community advocates, make your journey easier? Come talk to us in Slack, or leave a comment below.

– Mary

p.s. Want to talk to us live? We’ll be talking about client libraries in our next episode of All Things Delivered. Tweet your questions to #SparkPostLive and tune in at 3pm ET on November 16.

Email Tools in an API World Laptop w Gears

Modern Email Tools Available To You

Email has a long heritage of standards and specifications, many of them are truly great and some are even followed. Implementing these standards requires considerable knowledge. Tools ease our work by encompassing that knowledge, packaging it up and making it available to all. One strong measure of a tool’s worth is its effectiveness at embodying knowledge and reducing cognitive load on the wielder.

The world of email tools is almost as rich as its body of standards. We have myriad services, applications, libraries and commands for configuring, testing, orchestrating and tracking our various messaging activities. The email industry is hugely advanced in the orchestration, delivery and tracking spaces, since those are where the broadest commercial interest lies. Unfortunately, tooling for email service configuration and testing is less advanced, possibly since those tools would merely improve engineers’ lives.

Suffice to say, there is still considerable knowledge required to configure, verify and troubleshoot one’s email estate.

Let’s look at a typical setup task that most email senders have grappled with: configuring DKIM signing for an email sending domain. DKIM allows you to take responsibility for the messages you send by associating your sending domain with your mail. Don’t worry too much if you don’t 100% follow all these steps. The intent is to show level of effort.

DKIM Setup: The Old School Way

Here are some outline steps we might follow to start signing our email.

  1. Generate an RSA key pair:

2. Extract just the public key:

3. Form our DKIM DNS record from public key:

4. Have our DNS provider publish the DKIM record under

5. Configure our email service to sign using your private key.

With that done, these ancillary points are left as an exercise for the reader:

DKIM Setup: The SparkPost Way

Ok, let’s try that again, this time using the SparkPost API.

  1. Ask SparkPost to create our sending domain and a matching DKIM key pair
    (we can also do this in the app):

2. Form our DKIM DNS record using API result
(Hint: SparkPost will show us the fully-formed record here):

3. Have our DNS provider publish the DKIM record

4. Verify our setup by clicking the DKIM record test button for your sending domain in SparkPost

Side note: Here are all the tasty the details on using SparkPost to manage and verify sending domains and also using the SparkPost API sending domains endpoint itself.

Much Better…

Now we’re a tiny bit biased but we do love our email API and it certainly reduces the user’s cognitive load here. Questions we no longer need to worry about:

  • What size should the RSA key be?
  • What even is an RSA key pair?
  • How do we invoke the openSSL toolkit correctly?
  • How do we handle our private key material safely?
  • How do we format our DKIM DNS record?

That single API call executes sending domain configuration, creates our DKIM keys and formats our DNS record all at once. There’s less to do, less to understand and fewer places to stub our toes along the way.

…But Not Perfect (Yet)

…and yet there are further submerged rocks to run aground on here. The DNS record could be fat fingered during publication, the DNS provider may truncate the record and what do you do if there’s an existing DKIM DNS record for your domain? How do we diagnose each of these and how do we verify the end-to-end DKIM signing and verification process?

These are all issues and questions we’ve seen in production use. Clearly, there’s more we could do, as a tool-loving developer community, to help each other be more reliably successful, faster and just plain happier in our work. Just as SparkPost has an email API, DNS services have record management APIs. Combine these with the modern web and we could build far better experiences for each other. For starters, how about:

  • A DKIM verification tool explicitly designed for diagnostic and remedial use (stay tuned for next week’s post!)
  • A modern validating SPF record editor
  • Open DMARC as a service

Against a background of modern API-driven workflows, email tooling looks a generation or so out of date. In the SaaS email infrastructure community, we can do better. Isn’t it about time for a new generation of messaging tools?

If you liked, hated, agreed, disagreed or just want to chat about email infrastructure, tooling, APIs or anything else, come join our burgeoning Slack Community. We’d love to hear from you.

—Ewan

How-SparkPost-Built-An-Email-API_Dev-Blog_600x300_0716

Using community feedback to build the best email API

Steve Jobs once said, “Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple.” This is certainly true when it comes to building a simple and easy-to-use API, which for SparkPost is the primary user interface that developers interact with. So how did we create the best email API that developers love? Read on to learn some tips from the SparkPost Engineering team.

RESTful Architecture

One of our first decisions was to adopt a RESTful architectural style. Representational state transfer (REST) based web APIs have been used since the early 2000s. In the last several years, it has become the de-facto standard of cloud APIs. RESTful APIs, including the SparkPost API, has a collection of resources (https://api.sparkpost.com/api/v1/sending-domains/mydomain.com for example). These resources, described using JSON, allow clients to interact with it over the network using the standard HTTP verbs (GET, POST, PUT, DELETE), and get standard HTTP response codes (200, 500, 400, etc ) back.

The RESTful architecture style improves the simplicity, portability, interoperability, and modifiability of an API. Using HTTP and JSON makes it easy for developers, regardless of which programming language they use – PHP, Ruby, and Java – to integrate with our API without knowing our underlying technology. Other styles that were more common in the past like SOAP and JSON-RPC are generally obsolete. Today, more developer-friendly RESTful APIs replace them.

API Governance

Important as it is to decide on an architectural style, there are many choices to make when implementing the API’s interface. Oscar Wilde once said, “consistency is the last refuge of the unimaginative,” but if your choices are not a consistent set of conventions, then the API will be very hard to understand and use. At SparkPost, we have multiple teams working on different parts of our API. Within those teams, people work on different APIs at different times. Therefore, it’s very important that our API is consistent in the use of conventions. To achieve this, we established a governance group including a few engineers representing each team and a member of the Product Management team. This group is responsible for establishing, documenting, and enforcing our API conventions across all teams.

API First

We made a decision early on that our entire application would be built “API first.” This means that we did not bolt on our APIs to the application after the fact – the APIs are the application. Even our web user interface uses the APIs. It’s a client-side Javascript application which makes calls against the same APIs developers integrate directly against. This sometimes means that we build and publicize the API even before there is equivalent functionality in the web UI. To bridge that gap, we provide Postman collections for all our APIs so you can easily interact with all the functionality they provide.

Documentation First

When we build a new API endpoint or enhance an existing one, we always document and peer review changes first before a single line of code is written. This approach helps enforce our conventions, keeps things consistent, and ensures the API is very user friendly. It also cuts down on the amount of rework, bugs, and breaking changes. We use an API documentation markdown language called API Blueprint and manage our docs in Github. We encourage our community to contribute and improve upon these open source docs. Initially, we published our docs to Apiary, a great tool for prototyping and publishing API docs. However, embedding Apiary into our website doesn’t work on mobile devices so we now use Jekyll to generate static docs. Eventually, we may switch over to Swagger (now Open API) as the markdown format since this new standard continues to gain traction.

Technical Excellence and Transparency

We have a strong engineering culture and are proud of what we build. It is not sufficient to just have an idea for a great API. We need to build APIs quickly, iterate on them rapidly, and verify that they’re reliable and stable. For this reason, we embrace Continuous Delivery methodologies so that we can make changes and deploy them quickly and reliably. It is important to developers that we provide a rock solid service.

We peer review all API code changes, run them through unit and functional tests, periodic performance tests, and automatically deploy and smoke test them. We roll back changes automatically without impacting our users if any failures are detected. We also have extensive monitoring using a variety of tools including Circonus and OpsGenie. Thanks to these tools, we are immediately alerted when there are availability issues that need attention. When production issues affecting customers arise, we can respond quickly since we have engineers on-call 24/7. We also use our Status Page to keep our developer community updated.

Listen and Respond to the Developer Community

When we first released our API, it was quite minimal, supporting the most important features of sending email via API and SMTP as well as some reporting with the Metrics API. We collected feedback through customer interviews, user testing, surveys, and developer conferences early on. Members of our developer community recently sent us a lot of feedback on our Slack channel, which we launched in response to the Mandrill announcement. Subaccount support and IP pool management and our developer friendly freemium pricing are examples of improvements to the API in response to developer feedback, and on a weekly basis we make many improvements based on suggestions from the community.

Developer Relations

We have a small team of developer advocates that focus on listening to developers and improving the developer experience. This team helps maintain our API docs, client libraries, and community Slack channel. In addition, they attend and sponsor developer meet-ups like CharmCityJS, hackathons such as Bitcamp, and conferences including Fluent. This team also acts as internal advocates for the developer community and helps drive improvements across the service including the APIs and our support process. Beyond the small team, we have many who contribute on a part-time basis to the developer relations mission.

APIs Are For Integration

While RESTful APIs are easy to integrate against with any language, it is very convenient to have a client library that you can use to get up and running quickly. We are proud of the wide range of client libraries and integrations available for SparkPost. These include Node.js, Java, Go, Python, and others listed on our Developer Hub. We also have integrations for WordPress, Zapier, Heroku, and have several community contributed libraries for Ruby and C#.

Versioning and No Breaking Changes

We released v1 of our API, with the public beta of SparkPost, upon two deliberate decisions. The first is that versioning will be in the URI rather than as a header. This is easier than requiring an API or schema version to passed in as a header. Secondly, by explicitly putting the version into the URI, we committed to no breaking changes to the API. Breaking changes occur when the behavior of an API call changes, which breaks a customer’s integration. We take this seriously and prevent them through code reviews and automated tests. However, we can and do extend the API by adding new endpoints, parameters and inputs, and response values. Will we ever need a v2 of the API? Perhaps down the road when we wish to change the whole semantics for the API. However, we don’t foresee a need for that soon.

Don’t Forget About SMTP

We know that most developers still like the ease and portability of using SMTP to send their transactional or marketing email. Therefore, we added the ability to have much of the same functionality over SMTP as you have with the RESTful API. The X-MSYS-API header allows you to pass in campaign, meta data, and control data such as turning on engagement tracking or setting the IP Pool. The SMTP API also supports subaccounts.

 

I hope you learned a few things from how SparkPost built the best email API that developers love. Take what you learned and apply it to your own API development. Do you love building awesome APIs? Come join us, we are always hiring.  If you have any questions or comments please mention us on Twitter or find us on our community Slack channel.

—Chris McFadden, VP Engineering

Email Evolved Iguana Email API

An overview of the email API

With apologies to all who appreciate the particular genius of the 1967 film The Graduate, “I want to say one word to you. Just one word. Are you listening? APIs.”

These days, if you belly up to any bar (or spirits-free meet-up, if you’re so inclined) frequented by tech industry types, chances are you’re going to hear certain buzzwords. I can almost guarantee that “the cloud” and “API” are going to be among them. Sure, roll your eyes. I’ve been there, and I’ve been among those that smiled at a recent Internet meme that declared, “There is no cloud; it’s just someone else’s computer.”

But, when I really think about how tech and its use in the real world has evolved over the past decade, I stop dismissing those words as mere jargon. After all, almost every digital service I use today actually lives “in the cloud.” The bits that make up this blog actually live in the cloud, not a specific server my team maintains. So do the emails I get from readers and the music I listen to while writing. So do most of the web sites I visit and do business with. Unlike the early days of the Internet, when I first began working in this industry, server outages are all but a memory. In fact, the notion of a discrete “server” has all but been eliminated for most applications.

How did this happen? A lot of things made it possible, but the evolution of APIs are key. APIs (“application programming interfaces”) are the fundamental method by which all the virtual infrastructure embodied by the cloud is interconnected. The cloud could not exist without them. Even among business users, “API” has become a ubiquitous part of today’s technology vernacular. But, how APIs actually work, and their key role in the cloud revolution sometimes is taken for granted. We assume they just work.

A simple analogy is with the power outlets in the wall of your office or home. These receptacles provide a standardized interface to connect an appliance to the power network. Simple, right? Yes, it is. But look a little more closely, and that standard gets a little more complicated. If you have an older house, some of your outlets might not support the grounding function of three-pronged plugs or the modern, polarized version of two-prong plugs. Moreover, your electric clothes drier may require a special, oversized plug that connects to circuits that pull more power. Of course, if you’re traveling to the UK, you’ll need a special adapter to plug in your laptop, because it’s a different standard over there. Oh, heck, might as well bring the whole adapter kit with seven types of plugs to accommodate the other countries you’ll be visiting, too. And there are all kinds of extra connection standards for industrial applications that go well beyond what you or I encounter in our everyday experience.

APIs have a similar quality. They are the standard way for one piece of software to plug into—to invoke the functionality of—another piece of software. APIs connect disparate systems, services, and technologies. They are, in short, what makes the virtual infrastructure of the cloud possible. And email APIs are how any app or service can add email without reinventing the wheel.

However, APIs historically were highly idiosyncratic, with very little standardization among platforms. They were clumsy to code, difficult to invoke, and often poorly performing with limited scalability. It’s not a surprise, then that programmers and IT decision-makers alike often treated them as an afterthought to their overall technology implementation and were loathe to rely upon them in mission-critical contexts. In light of the constraints of both hardware resources and API performance, most developers chose to keep everything under one roof in a monolithic codebase optimized for a specific hardware environment.

So what changed that moved APIs from little-loved feature of monolithic applications to the all-but-invisible linchpin of the modern cloud? Three major developments are responsible for the upending of this dynamic and making today’s architecture possible:

  • The rise of the Internet as a ubiquitous web that connects nearly every computer (or other electronic devices, whether televisions, phones, refrigerators, and thermostats, inventory control systems, or factory equipment) removes one historical constraint: “always on” connectivity.
  • The exponential growth in the performance and capacity (and mirrored by plummeting costs) of computer hardware and storage devices removes another limit: economies of scale.
  • A codification (both formal and de facto) of several design patterns and best practices for describing, invoking, and transmitting information among diverse software systems provided the final piece of the puzzle: API standardization.

Together, these forces have enabled massively scaled cloud platforms. In turn, these platform-as-a-service offerings form the basis of virtualized computing stacks for countless applications across every industry and consumer market. And, yes, a good email API makes it possible to add email to nearly any app and to have some confidence that it just works.

OK, so what does that mean in the real world? Well, it means on the subway home from work, I can open my mobile app and expect to see the exact same data I touched at the office. It means the data I entered in one app get distributed to several other systems without requiring duplicate entry or manually synchronize records. It means the store I just visited doesn’t need to worry about hosting their own infrastructure just to email me a purchase receipt. It all just works, and it’s all in the cloud.

That’s pretty amazing, if you really think about it. And that’s why an email API matters.

How is your day touched by APIs and the cloud? I’d love to hear about it!

—Brent
@brentsleeper

By the way, want to learn a little more about the role of an email API and how they’ve changed the way businesses use email? Read “Email Evolved: Why the Cloud and Modern APIs Matter for the Future of Data-Driven Marketing.” And, if you’re wondering why a good email API and cloud architecture makes a difference, check out this blog post Why ESPs Struggle to Deliver Data-Driven Email. (Spoiler: it’s because ESPs aren’t really API-driven.)

Dev Survival Guide Blog Footer