Continuous Integration and Deployment at SparkPost

Rich Leland
Aug. 19, 2016 by Rich Leland

Continuous Integration gears laptop mobile phone

Some two years ago, a small team met in a conference room to discuss building a self-service offering on top of Momentum, the world’s best email delivery platform. Since then, SparkPost has gone from an idea to a developer-focused service with an automated release cycle built on a culture of testing and constant iteration. So we figured it was time to share what we’ve learned and how we handle continuous integration and deployment.

Why We’ve Embraced Continuous Integration and Deployment

Before we dive into the how, you need to know why we’ve embraced continuous integration and deployment. We have 20 components that make up our service and we routinely deploy 15-20 times a week. In fact, deploying frequently allows us to focus on creating a better experience for our users iteratively. Since we can deploy small changes to any component of our service independently, we can respond quickly based on what we learn from our community. We’ve found that releasing discrete pieces of functionality for specific components lowers the risk of deployments because we can quickly verify the work and move on.

Testing is at the core of being able to continuously deploy features. The testing culture at SparkPost gives us the confidence to deploy at will. We don’t have an enforced or preferred method of testing like BDD or TDD. We have a simple rule – write tests to cover the functionality you are building. Every engineer writes unit, functional, and smoke tests using mocha, nock, sinon, and protractor. Each type of test is critical to the deployment pipeline.

Our deployment pipeline orchestration is done using Atlassian Bamboo for our private projects. We have three types of plans chained together: test, package, and deploy. During the test plan, we clone both of our automation scripts. We house all our continuous integration bash scripts, and the component we’re working on (e.g. our metrics API) in them. Bamboo then runs the unit and functional tests for that component, generating test results and coverage reports. Upon successful build, the packaging plan is triggered, generating any necessary RPM packages and uploading them to a yum repo. Once the packaging is complete, it triggers the deployment of the package. Deploy plans are responsible for installing/upgrading the component and any related configuration using Ansible, running smoke tests using protractor, and, if necessary, rolling back to a previous version.

Open source work, like our client libraries, Slack bots, and API documentation, is run through TravisCI. Check out the .travis.yml files for our Python library, PHP library, API docs, and developer hub to see what they do.

continuous integration slack screenshot

Slack and Additional Ways We Use Automation

You most likely know about our obsession with Slack by now. We use it for both manual and automated notifications related to deploying features. Before we merge/deploy code, we announce the component and the environment it will be going to. Merges to develop branches trigger deployments to UAT. Merges to master (hotfixes or develop branch promotions) trigger deployments to staging. Deployments to production are push button to allow for proper communication and timed releases of features. Once merged, it triggers the deployment pipeline outlined above. Bamboo sends email notifications upon successful plan builds, the start of a deployment, and the success or failure of a deployment. This email is sent to an internal address which is consumed by a process that posts a message in Slack.

Some additional ways we use automation include:

  • Deploying the Web UI
  • Deploying Momentum, our core platform written using C and Lua
  • Testing and upgrading Node.js
  • Making nginx configuration changes
  • Deploying one of our 18 APIs
  • Pushing customer acquisition and community data into dashboards
  • Deploying cron jobs that run cleanup tasks and reports
  • Deploying Fauxmentum, our internal tool for generating test data against our various environments

Continuous integration and deployment are vital parts of SparkPost’s ability to listen and respond to what our community members asks for. To sum up, we hope that we’ve given you some insight that will help you improve upon your own ability to build, test, and deliver features by sharing some of our experience and process. If you’d like to see some of our pipeline in action then you can sign up for an account here. Also, feel free to join our community Slack channel, and chat with us about your experiences with SparkPost. We’d love to hear from you!

—Rich Leland, Director of Growth Engineering

Share your Thoughts

Your email address will not be published.

Related Content

Burnout: What Happens When You Take On Too Much

Burnout has become a very relevant topic in the tech industry. Our community manager offers several tips on how to avoid being overworked and stressed.

read more

Creating the Next Generation of Interactive Emails with Rebel Mail

Meet our partner, RebelMail! Today they're demonstrating how to use their interactive email API to keep customers engaged with the inbox.

read more

Using the SparkPost Metrics API with C#

Learn how to query the SparkPost API for information and metrics on the emails you’re sending using the C# wrapper and gain valuable insights.

read more

Start sending email in minutes!

The world’s most powerful email delivery solution is now yours in a developer-friendly, quick to set up cloud service. Open a SparkPost account today and get started for free.

Get Started

Send this to a friend