Saving Time with Substitution Data and Templates

If you’ve been around systems development for any amount of time, you know there’s a lot of inertia when it comes to how transactional email or other messages are generated by business systems. Even today, many companies that use services like SparkPost or any email service provider (ESP) as a “pipe” (be it dumb or smart), build the full email in one of their back-end applications then connect to the ESP via SMTP in order to send the email. If the email is something like an invoice, password reset, shipping information or abandoned cart, the email is personalized to that user. If the communication is more akin to an advertisement of products it may or may not be personalized to leverage the user’s search or purchasing patterns. Either way, these emails are typically built by homegrown systems that construct the full email content into a single SMTP call to the ESP. As discussed earlier, this connection between code and the content creation makes it very difficult to make changes.

Over the last few years, ESPs have started to provide REST APIs that allow companies the ability to create content structures, or templates, via the ESP’s UI (or submitted via another REST API). Following the creation of these templates the application sends data to the ESP, identifying which template to inject the data into.

The Old Way [Well, Current Painful Way for Many]
Development or IT builds and maintains email creative and personalization content


Any changes to the email must go through the change request process, development process, QA process, and fight all other development projects!

The new way is to separate out the creation/updates of the creative and obtaining the data for personalizing the content.


The high level changes that need to take place from a development perspective are:

  1. Gather the data to be sent and format appropriately for the ESP. For SparkPost, this would be a json structure (see example below).
  2. Add supporting information:
    a. Which template to use and/or campaign name
    b. Turn on/off options like click or open tracking
    c. Optional metadata and tags that can be used to help understand user email behavior
  3. Send json formatted substitution data via REST API

One of the cool options that you can do with many of these REST APIs is that you can send more than one email at a time. For instance, SparkPost allows you to send the request for thousands of emails all at the same time, as long as they use the same template. This allows the developer to batch emails in groups instead of making a REST API connection for every email.

SparkPost also supports the concepts of metadata and tags into the system. Tags are high level identifiers that describe the overall email campaign; for example, Newsletter, Invoice or Welcome would be good tags. On the other hand, metadata is a key-value pair of data that is often used to describe that individual email. For example, if you wish to track what category of products are in the email being sent to this user you may have a key value pair of “category” : “electronics” (see example below). When the user opens the email or clicks on a link, the metadata (and tags) are added to the information logged for that email which then in turn can be sent to your back end databases via webhooks which are small chunks of data that is sent/pushed from the mail system to your applications via HTTPS for processing. SparkPost has about 19 different webhooks that handle everything from email injection, creation, bounces, feedback, opens and clicks. Our developer hub has more information about SparkPost’s webhooks support.

Hint: Metadata fields can also be a multi-level array AND they can be used in your email as substitution fields.

So how does the HTML template (email) actually use the data being sent by your applications? Great question. Many template systems, including SparkPost, use a familiar handlebars-style syntax. Start and end markers are defined as double curly braces; for example, {{billing_zip}} will replace the user’s actual zip code that was sent as substitution data via the REST API into the HTML (or text) email before sending out. Of course more complex replacements can take place if there are arrays of data which would be common for emails that might list a set of features within a list of products but that description will come in a later blog post. Right now, if you want more info, please refer to our substitutions reference documentation.

A real example of using substitutions within an HTML email with the data below might look like:

<td id="Right" valign="top" width="50%">
        <h3 style="font-family: Arial; font-size: 18px;">Billing Address</h3>
         <div style="font-family: Arial; font-size: 12px;">
                   <strong>{{first_name}} {{last_name}}</strong><br>
                   {{billing_city}}, {{billing_state}} {{billing_zip}}<br>

With this new approach, marketing can take charge of the creative and let development focus on pulling the data together.  When legal or marketing need to make a change to the wording, it’s all up to them now and not development.  That sounds like a win-win to me!  To further simplify a development team’s life, a good practice is to overload the JSON entry with lots of information about the user and not worry about matching the exact information that the template is looking to use.  If the template doesn’t use the data, so what?  This simplifies the coding for development tremendously.  I wouldn’t go crazy with sending every stitch of info you have on a user, but adding extra info in order to minimize a bunch of code branches won’t hurt anyone.

Sample Json for a single email recipient

	"open_tracking": true,
	"click_tracking": true
	"template_id": "shipping_address_validation”
"campaign_id": "Address Validation",
"return_path": "",
	"address": {"email": ""},
		“category” : “electronics”
	"tags": ["validation"],
		"user_name": "jeffreyca",
		"first_name": "Jeff",
		"last_name": "Goldstein",
		"shipping_street_addr1": "123 Main Ln",
		"shipping_street_addr2": "",
		"shipping_city": "Pleasanton",
		"shipping_state": "California",
		"shipping_zip": "94566",
		"shipping_country": "United States",
		"shipping_phone": "(925) 462-5333",
		"billing_street_addr1": "123 Main Ln",
		"billing_street_addr2": "",
		"billing_city": "Pleasanton",
		"billing_state": "California",
		"billing_zip": "94566",
		"billing_country": "United States",
		"billing_phone": "(925) 426-5483"

Sample web hook event for a bounce event on the mail system

"msys": {
      "message_event": {
       	 "type": "bounce",
        	"bounce_class": "1",
       	 "campaign_id": "Address Validation",
       	 "customer_id": "1",
       	 "delv_method": "esmtp",
       	 "error_code": "554",
        	"event_id": "92356927693813856",
        	"friendly_from": "",
        	"ip_address": "",
       	 "ip_pool": "Example-Ip-Pool",
     	 "message_id": "000443ee14578172be22",
        	"msg_from": "",
        	"msg_size": "1337",
        	"num_retries": "2",
        	"rcpt_meta": {
		“category”: “electronics”
        	"rcpt_tags": [
        	"rcpt_to": "",
        	"raw_rcpt_to": "",
        	"rcpt_type": "cc",
        	"raw_reason": "MAIL REFUSED - IP ( is in black list",
        	"reason": "MAIL REFUSED - IP (a.b.c.d) is in black list",
        	"routing_domain": "",
        	"sending_ip": "",
        	"subaccount_id": "101",
        	"subject": "New Shipment Address, Please Verify!",
       	"template_id": "templ-1234",
        	"template_version": "1",
        	"timestamp": "1454442600",
        	"transmission_id": "65832150921904138"

So that is the introduction into leveraging substitution data and templates to separate the work of obtaining data for personalized emails and the creative for those emails.

In the next blog, I’m going to talk about ways to use substitution fields at a global level which can be used to change colors for branding or white labeling, as well as to simplify management of headers, footers, and multi-language issues.

Happy Sending

Jeff Goldstein
Senior Messaging Engineer

Questions on using substitution data and templates in email personalization? Give us a shout on twitter or comment below.