A frequent theme for SparkPost messaging engineers is to help customers to gradually migrate traffic across to SparkPost. We have this article showing a simple PowerMTA relay configuration example. My colleague Tom Mairs wrote this article on diversifying your message streams, and this article on strategies for Momentum and PowerMTA migrations. We’ll build on this work here. 

Your email programs are business-critical, so you need a simple way of selecting which traffic streams are delivered via SparkPost, and which continue to be delivered by PowerMTA.

This article describes how to:

  • Leverage Virtual MTAs in PowerMTA, and subaccounts in SparkPost, to map message streams with load-sharing between PowerMTA and SparkPost;
  • Plan your migration, including transitioning from accounting files to SparkPost event reporting;
  • Configure SparkPost resources:
    • Sending domains, including DKIM keys
    • Bounce domains
    • Tracking domains
    • IP Pools, including auto-warmup
    • Subaccounts;
  • Change PowerMTA configuration:
    • Back up first
    • Add the per-VMTA routing to SparkPost
    • Disable PowerMTA DKIM signing on relayed domains
    • Consider: SMTP hostnames, bounce processing, campaign_id
  • Test your setup – with  suggested steps:
  • Set up alerts;
  • Post-migration tidy-ups.

OK, let’s dive in.

Virtual MTAs and Subaccounts

If you’re already using virtual MTAs (abbreviated from now on to VMTA) and VMTA pools in your configuration, you can route traffic based on those; otherwise, we can use domain-based selection as shown here.

Let’s build on these foundations:

  • The VMTA pool provides selective routing of traffic streams with load-sharing and a smooth migration path, with the option of rollback if needed;
  • Subaccounts provide mapping of traffic streams to delivery IP pools, with Auto IP Warmup;
  • SparkPost Signals Analytics provides visibility of migrated traffic streams with clear reporting.

We’ll use PowerMTA as a relay to route traffic via SparkPost, without the need to rework any of your message-generation stack. Later, you could update your message-generation to connect directly to SparkPost, rather than via PowerMTA; in the meantime, PowerMTA keeps your message streams flowing and helps to warm your new SparkPost IPs.

Each VMTA pool can load-share between direct delivery and relay to SparkPost; more on that in a moment. We’ll also use Auto IP Warmup to protect your SparkPost dedicated IP pools from sending too much traffic at the start. The remainder will overflow to your SparkPost account default pool, or to our global shared pool (on request).

Planning your migration

PowerMTA Accounting Files vs. SparkPost events

PowerMTA creates various accounting files for messages received, delivered, bounced, remote-bounced, FBLs etc. Migrated streams will show only a “delivery” event in the accounting logs, which really indicates that the relay was successful, not the final delivery to the recipient.

Your SparkPost relayed traffic will trigger events within SparkPost according to this model. SparkPost events can be viewed on the web UI, accessed via API and received in near real-time via webhooks.

You’ll need to prepare your own systems to handle events coming via SparkPost – this article describes how to use webhooks.

Configure SparkPost resources

Prepare your new message streams by setting up resources in SparkPost, using the articles linked here.

For simplicity, we can create one subaccount per VMTA. You can also easily map streams from several VMTAs to one subaccount, if that makes more sense for your situation (i.e. many → one mapping is allowed).

Before we go further into the various SparkPost configuration choices, here’s a high level view of our message flow.

The message generation code controls the MAIL FROM (sending domain), RCPT TO and X-Job values. Let’s follow the messages.

Subaccount selection

PowerMTA’s VMTA configuration routes mail to SparkPost, setting the subaccount value. Here’s the relevant SparkPost configuration screen.

We won’t need individual API keys per subaccount (more on that later).

Subaccounts are available to you as a reporting “filter” for the Summary Report, Bounce Report, Signals Health Score, and so on. This enables you to provide graphical reports for each message stream.

The sending domain(s) available for that message stream are either:

  • master account / shared with subaccounts; or
  • specific to that subaccount.

IP Pool mapping

When it comes to selecting the SparkPost IP pool, we have another choice – 

  1. Specify in PowerMTA configuration, using the sparkpost-ip-pool directive;
  2. specify in SparkPost configuration of each subaccount.

There are advantages to 2):

  • Takes us closer to the desired end-state of having the SparkPost account ready for direct injection;
  • Frankly, it’s just easier and less error-prone. SparkPost configuration is done over a web user interface (or API), with selection made from a drop-down menu!

A many → one mapping from subaccount to IP pool is also allowed. For example, you might want to group traffic streams with similar reputation (e.g. “new, medium, good”), rather than a separate pool per message-stream.

Each pool has a name, an (optional) Default Signing Domain, Overflow Pool, and one or more Sending IPs.

IP warming settings

As mentioned earlier, we set each dedicated IP to automatically warm, so that it only carries a small amount of traffic to begin with.

Configuration is described here.

Sending domain & bounce domain selection

Sending domains can be held at Master Account level (with access granted to all subaccounts); or you can also specify at time of creation, to restrict a particular sending domain to a subaccount. Bear in mind it’s not currently possible to switch a master-account sending domain to a specific subaccount after creation via the web UI. If you’re specifying your own DKIM private/public key pair (see below) then you could delete/create again as a workaround.

The bounce domain(s) (aka Return-Path:) can be selected via subaccount default, or master account default. In this case we’ll keep it simple and just use subaccount defaults.

DKIM keys in SparkPost

The DKIM key in SparkPost can either be:

  1. A new key pair, generated by SparkPost as the sending domain is created. This can happily exist alongside your current keys, as the selector value will be unique. You will, however have to create a DNS record; or
  2. An existing key pair that you already have.

If you have many domains, and no other reason to generate new keys, then go with 2). You’ll need to use the SparkPost API to load this in (see thedkim attribute defined here), but here’s a simple Python tool to help you.

X-Job mapping to Campaign ID

PowerMTA can pass its X-Job header value into the Campaign ID used by SparkPost using this configuration setting. That ensures the job name our message-generation stack applied to the message is also available in reports.

Engagement Tracking domains in SparkPost

SparkPost open & click tracking (on / off) can be controlled at the main account level, using the setting described here, or controlled using per-message headers (described here). You can have as many custom Tracking Domains as needed.

PowerMTA config changes

Backing up your PowerMTA config

Configuration management is important, particularly during a time of change. We strongly recommend you archive your PowerMTA config file each time, prior to making any changes. If you have a test / staging environment, use that instead of working on Production servers!

Here’s a simple command-line method I use to make an archive with the current date/time embedded in the file name:

sudo su -
cd /etc/pmta
tar -cvzf config_$(date +%y-%m-%d-%H_%M_%S).tar.gz config

Config file(s) and DKIM private/public key files are visible from the PowerMTA web monitor under “Configuration”, however these are more efficiently exported in bulk from the command line.

Per-VMTA routing to SparkPost

The <domain {sparkpost}> directive allows configuration of various parameters including API key.

We wish to have specific routing configurations, either in the VMTA scope, VMTA pool scope, or in the domain scope referred to by the queue-to directive.

The subaccount can either be selected with:

  1. A specific API key per subaccount; or
  2. The “username” numeric subaccount selection shown here, e.g. SMTP_Injection:X-MSYS-SUBACCOUNT=123, used with a master account API key.

Option 1) is appropriate if you wish to open up your service for end-customers to make SMTP injections directly; but we are not doing that here. Where injection traffic comes from your own systems, option 2) is more manageable. There is only one API key to maintain, and subaccounts are referred to using a human-readable numeric ID that corresponds to the SparkPost UI presentation.

Disabling PowerMTA DKIM signing on relayed domains 

You’ll want to retain your existing DKIM configuration in PowerMTA for directly-delivered message streams, and disable DKIM signing for the “relayed to SparkPost” streams. The dkim-sign directive (User Guide link) is used to control this.

SMTP hostnames

Your PowerMTA config may be setting specific host-names. This is not applicable to the SparkPost cloud service, which will apply its own host-names.

Bounce processing

SparkPost will insert its own Return-Path: domain, and its own FBL loop. Out-of-band (remote) bounces and FBL responses will be picked up and handled by SparkPost directly (following the event reporting sequence shown here).

Campaign_id

The add-x-msys-api-header directive sets campaign_id to the “JobID” (PowerMTA’s equivalent of a campaign). This is desirable because it enables “campaign” to be an active reporting facet on the SparkPost Signals Analytics reports, such as the Summary Report and the Health Score. In fact this is enabled by default by the <domain {sparkpost}> scope.

Example configuration

# Rules that declare the traffic sources look like this.

<source customerXYZ>
   # other directives ...
   default-virtual-mta customerXYZ-pool
</source>

#
# This virtual MTA pool used mta1 to deliver traffic.
# Now load-shares between SparkPost relay and MTA.
# Adjust the load-share percentage split by repeating lines.
#
# Repeat for as many relayed virtual MTA pools required.
# Each is the same, apart from scope name and the auth-username directive.
# the old VMTA definitions can be left in place then gradually retired.
#
<virtual-mta-pool customerXYZ-pool>
    virtual-mta sp-relayXYZ
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
    virtual-mta mta1
</virtual-mta-pool>

<virtual-mta sp-relayXYZ>
    <domain *>
       queue-to {sparkpost}
    </domain>
    <domain {sparkpost}>
       auth-username SMTP_Injection:X-MSYS-SUBACCOUNT=1053
       # sparkpost-ip-pool default # set only if need to override
    </domain>
</virtual-mta>

#
# SparkPost Account configuration
#
<domain {sparkpost}>
    auth-password <<MASTER-API-KEY>> # Replace with your actual key
    use-starttls yes
    require-starttls yes
    dkim-sign no
</domain>

Load-sharing and warm-up

The customerXYZ-poolsplits the traffic between mta1 and sp-relayXYZ. By repeating lines, you can achieve a percentage split, in my case 90% direct, 10% relayed. You’ll want to begin with a low percentage going to SparkPost; the ideal is for each individual message stream to follow this plan for daily volume changes. Sparkpost auto IP warm-up (see above) will also protect us from damaging our reputation by accidentally sending too much, too soon via the relay.

TLS

SparkPost will always offer STARTTLS as an option to incoming connections. Here, to ensure tight security, we explicitly require STARTTLS on the connection.

It is important to ensure PowerMTA has access to a valid TLS certificate. If you don’t have one issued by a Certificate Authority, PowerMTA includes a tool to generate a self-signed certificate. We can also ensure modern TLS versions are used, and older ones deprecated with the following directives. 

smtp-server-tls-certificate /etc/pmta/mycert.pem
smtp-server-tls-allow-tlsv1   false
smtp-server-tls-allow-tlsv1.1 false # these should not be used any more
smtp-server-tls-allow-tlsv1.2 true
smtp-server-tls-allow-tlsv1.3 true  # For future

 

It’s also good practice to check which ciphers and cipher suites are in use (see directives smtp-server-tls-ciphers and smtp-server-tls-v1.3-ciphersuites). OWASP (Open Web Application Security Project) has some advice on TLS and ciphers here.

Testing your setup

1. Verify message injection direct to SparkPost

Inject traffic directly into SparkPost using “swaks” (i.e. not using PowerMTA) and confirm messages delivered with correct DKIM signature, bounce domain, and tracking domain applied. For example:

swaks --server smtp.sparkpostmail.com:587 --from steve@acme.thetucks.com --to <<YOUR-INBOX>> --auth-user SMTP_Injection:X-MSYS-SUBACCOUNT=1053 --auth-pass <<MASTER-API-KEY>> --tls --tls-protocol tlsv1_2
acme.thetucks.com My verified sending domain on SparkPost, assigned to subaccount 1053
<<YOUR-INBOX>> Please put your own inbox test address here. You could also use the SparkPost sink or Bouncy Sink, but you won’t get to see the message.
<<MASTER-API-KEY>> My SparkPost API key, with (at least) “Send with SMTP” permission set

If your SparkPost subaccount and sending domain is correctly configured, you’ll see a message exchange ending with:

<~  250 2.0.0 OK 76/DA-08533-C66B7EE5
 ~> QUIT
<~  221 2.3.0 ab.mta1vsmtp.cc.aws-usw2a.prd.sparkpost closing connection
=== Connection closed with remote host.

The message should appear in your test inbox.

2. Create PowerMTA relay setup for a single VMTA pool with this domain

See example config above. This is best done with no Production traffic flowing, so you can initially route all traffic on this VMTA to SparkPost. We’ll adjust load-sharing in a later step.

3. Check that PowerMTA is negotiating STARTTLS with SparkPost

It’s important to ensure that your API key is transmitted securely. PowerMTA includes a nifty tool to trace outbound connection establishment via a specific VMTA. Specify the sending domain and your SparkPost relay VMTA, for example:

pmta trace acme.thetucks.com/sp-relayXYZ

You should see lines like this in your console output.

:
2020-06-15 18:45:54 tls: starting TLS
2020-06-15 18:45:54 tls: TLSv1.2 connected with 256-bit ECDHE-RSA-AES256-GCM-SHA384
2020-06-15 18:45:54 tls: cert: /CN=*.mx.e.sparkpost.com; ..
:

You can also use the log-tlsdirective, or use tcpdump / Wireshark to sniff the outgoing connection.

4. Inject traffic into PowerMTA for this domain

Here I’ll use “swaks” again.

swaks --server pmta.signalsdemo.trymsys.net:587 --from steve@acme.thetucks.com --to bob.lumreeker@gmail.com --auth-user customerXYZ --auth-pass <<YOUR-POWERMTA-INJECTION-PASSWORD-HERE>> --tls --tls-protocol tlsv1_2

If you don’t need TLS to reach your PowerMTA, other tools such as smtp-source can be used. Confirm you see the “250 OK” response, and that messages are relayed through PowerMTA, via SparkPost to your test inbox.

5. End-to-end test

Check messages injected by your message generation stack are relayed through PowerMTA and delivered by SparkPost to a recipient mailbox. The message should have the DKIM signature, bounce domain, and tracking domain applied as you’d expect by SparkPost, because we’ve already tested it back in step 1.

6. Set up the load-sharing percentage split within your VMTA pool

Introduce a small amount of traffic from your message generation stack. Check that PowerMTA is relaying some of this to SparkPost, by looking for the VMTA name in the accounting log file.

You should see some (relayed) such as

sp-relayXYZ,,,{sparkpost}/sp-relayXYZ,customerXYZ-pool,

and some (directly delivered) such as

,mta1,,,<<your-inbox-domain>>/mta1,customerXYZ-pool,

You should see a spread of directly-delivered and relayed messages, according to your chosen load-balancing split.

7. Check the SparkPost Summary Report

It should show traffic on your delivery IPs according to your auto warm-up settings.

8. Check your event reporting

This step is dependent on your specific setup. Check it’s correctly handling injection, delivery, bounce, out-of-band, and FBL events. The Bouncy Sink can be used for moderate test volumes.

Set up alerts 

You can also set up individual filtered Alerts for subaccounts, if the (for example) the Block Bounce rate increases or the Health Score deteriorates:

Post-migration tidy-ups

Once all (or nearly all) traffic is being relayed through SparkPost, your PowerMTA configuration can be significantly cut down and simplified. The IP connection limits, MX rollups, host-name declarations, and bounce processor handling (the reply patterns) can eventually be removed once you’re confident SparkPost is handling these aspects.

You can also plan to move your message generation stack to send messages directly to SparkPost, but consider the following:

  • For very high volume messaging, delivering from your own MTA is likely to remain the lowest-cost option, although SparkPost provides rich analytics and other features not present in an MTA.
  • Your message generation stack will need to provide TLS security. Some older message generation stacks (particularly Windows-based ones) are known to not support TLS. This is fine when you connect to your MTAs over a private network. Connecting to any public cloud SaaS endpoint means you need to include security. PowerMTA relay is one way of achieving that.
  • Retaining PowerMTA in your message path gives you the option to deliver directly to mailboxes should you need that, providing routing resilience and future flexibility for a modest license cost.
  • PowerMTA is acting as a fast, dedicated, low-latency local queue for your outbound message bursts. Injecting to any public cloud endpoint will have some more latency (at the sub-second level). You will need to scale more threads/processes in your message generation stack to allow for this.
  • SparkPost cloud service accepts multiple injection threads/processes and is highly performant (see our recommendations here). PowerMTA relay is an easy way to get the most out of this, with tuning parameters such as max-smtp-out, max-msg-per-connection. When injecting directly, you’ll need to tune the concurrency in your own stack.
  • You’ll get the lowest latency if you select the SparkPost service region closest to your injection processes. Your choices are currently US-West2 = Oregon, vs EU-West1 =  Ireland.

Summary

We’ve covered a lot of ground, outlining some of the choices you’ll come across as you plan your migration. Some areas such as Accounting files / event reporting will need to be addressed as part of this plan, as well as the actual message routing itself.

There’s a lot of flexibility in both PowerMTA and SparkPost. If what’s described here doesn’t fit your situation, it’s likely some slight variations will get you there.

I’d love to hear your thoughts on this – you can reach us on Twitter @sparkpost!

See also

  1. https://www.sparkpost.com/docs/integrations/power-mta/
  2. https://download.port25.com/files/UsersGuide-5.0.html#sparkpost-traffic-redirection-support-directives

~ Steve