Four years ago, Message Systems had a single product offering in Momentum version 2. While this was a groundbreaking MTA, it was really built for the singular function of processing email very, very quickly. Momentum 2 stored a number of logs but had no local database and did not really need one.

When Momentum version 3 was introduced, it came with a new dashboard that used an embedded PostgreSQL database. Shortly after this release, the product offering expanded to include content creation and campaign tools that also made use of the PostgreSQL database. Intelligent adaptive message shaping was also added utilizing a Riak database. Most recently, introspection and tracing tools were added that make use of a Lucene database.

So why use all these databases? Each of these types of data stores are particularly good at certain functions so we use different databases to make the overall platform solution as efficient as possible. The PostgreSQL database is a powerful SQL relational database with the ability to store complex tables as well as procedures for advanced processing. This is used for the bulk of the data workload in the Message Systems platform.

Riak is a non-SQL database that is designed to work in a distributed environment. It is essentially a key-value pair data store, similar in concept to Hadoop, that is extremely fast and durable. It replicates natively across a cluster to provide fast access to data to any node in a cluster. We use this primarily for the storage and retrieval of message shaping settings for the Adaptive Delivery® component.

Lucene is an Apache project based in Java that delivers extremely high performance text search capabilities. In our case, we record message transaction logs to a Lucene data store so that we can search them at a later date. When necessary, a customer service representative can execute a search against this Lucene data store for any message using a wide variety of factors.

So you can see that each of these data store technologies are widely different and serve very different purposes. Message Systems includes each because they are the best at what they do and they work together to provide the most efficient platform overall.

More Information:

Message Systems Documentation (client login required):

Learn more about why the world’s best brands are using Momentum in the Total Economic White Paper by Forrrester!

Blog_Ads_TEI-Momentum_100413

On Tuesday, September 25, a completely redesigned and updated Message Systems website was unveiled to the world. It was an exciting development for several reasons, but especially for those of us in the marketing department because — aside from some search engine optimization hired guns — the new content, navigation architecture, design and coding were handled entirely by our in-house team.

What’s New

Of course, internally we all had our own reasons for wanting to update the website: make it easier to update and maintain, improve performance, and a chance to try out some cool new toys. But more importantly, for the outside world it served as a first glimpse at our newest packaged offering: the Customer Conversation Hub (CCH). We’ll have a lot more to say about the CCH soon, but as you’ll see on the solutions overview page, it’s a comprehensive customer engagement solution that enables you to connect your data sources, business processes and applications with your messaging (email, text and more) to coordinate customer interactions across business units and channels. A major step forward in harnessing digital messaging for customer engagement, the CCH provides all the capabilities needed to initiate and sustain the kinds of real-time, conversation-style interactions that customers increasingly demand in today’s communication environment.

Simplified Navigation and Information

One thing we learned in our site update research was that our customers, prospects and other site visitors wished for a more intuitive path to key information. To that end, we streamlined and organized the content into more logical sections. We also put a lot of work into highlighting the Message Systems story in a way that tells you more clearly what we do and why we do it. We believe that once you understand how truly visionary our solutions are, you’ll see how they can help you drive your business forward, right now and on into the future. Pay us a visit — here are a few things you’ll find:

  • A Community section that’s home to our blog, events listings, partners, industry alliance information and lots more. Be sure to check out the new Featured Clients page that provides quick snapshots of our wonderful customers.
  • A dedicated Resources section with all of our product collateral, customer case studies, white papers and more, all in once place. Don’t miss the updated Videos page with clips from our clients, including US Airways and TC Transcontinental; and highlights from our Interact user conferences. We’ll be posting many more client videos in the weeks ahead — so check back again soon.
  • A refreshed Solutions section with information on how Message Systems customers use our technology to help achieve business objectives like Customer Engagement, Revenue Generation and Deliverability.

This recent website launch is just the first in a series of improvements and updates that will be rolled out in the coming months. We look forward to providing you with more resources in multiple formats so that no matter how you like to receive information, relevant content will be available.

Sign up for our newsletter for our latest company updates!

CEO George Schlossnagle has been named a finalist in the Ernst & Young Entrepreneur Of The Year® 2012 program for the Maryland, U.S. region. The award recognises outstanding entrepreneurs who demonstrate excellence and extraordinary success in such areas as innovation, financial performance and personal commitment to their businesses and communities. Awards will be presented at a special gala on June 28 in Baltimore, Maryland, U.S.

“It’s an honour to be recognised as a finalist for an Ernst & Young Entrepreneur of the Year Award,” said Schlossnagle. “At Message Systems, we are committed to delivering the best messaging technology solutions possible. It’s very exciting to be acknowledged for our innovation, growth achievements and genuine commitment to our business.”

Now in its 26th year, the Entrepreneur Of The Year Program has expanded to recognise business leaders in more than 140 cities in more than 50 countries throughout the world. Regional award winners are eligible for consideration for the Ernst & Young National Entrepreneur Of The Year Program. Award winners in several national categories, as well as the Ernst & Young National Entrepreneur Of The Year Overall Award winner, will be announced at the annual awards gala in Palm Springs, California, U.S., on November 17, 2012. The awards are the culminating event of the Ernst & Young Strategic Growth Forum, the nation’s most prestigious gathering of high-growth, market-leading companies.

Starting with Momentum Version 3.0, Message Systems has included an embedded copy of the Lua language in the bundle as a replacement for Sieve. For years Message Systems customers used Sieve as a policy engine to script actions based on message details, but Sieve, even in its modified Sieve++ form, is limited. Lua is a big improvement, offering the ability to iterate over data and work with tables, among other features. Just like Sieve, we have modified our embedded version of Lua to extend its functionality in numerous ways.

Not surprisingly, Lua has a large following in the programming world. It’s a well-developed, multi-purpose language that is often used by game developers for its simplicity and speed, as well as its inherent ability to expose and use C modules as well as its own native scripting. In Momentum, Lua has been extended with a number of local functions specifically to help with messaging rules. Aside from the extensive documentation on our custom functions, there is also a whole community of Lua users and developers on the web (links at the end of the article).

You will see in the sample below that the code itself is quite simple to follow, and anyone who has written a policy script in Sieve will see this as manna from heaven. So what does Lua look like in Momentum? I can show you using a very basic policy script that will assign messages to a binding based on an X-Header value. Here is the heavily commented policy script:

— This policy script is a common, but very simple example
–[[
You can use comments in Lua with a “–” for a single line or multi-line comments can be framed as this one is.
]]–
— As with many languages, you can “require” helper modules
— We have developed dozens of helpers that make message management easier

require(“msys.core”)
require(‘msys.extended.message’);


— We define the following variable as “local” and make it a “table” using braces
— IE: the variable “mod” is a local table

local mod = {};


— We have predefined function names for each message phase.
— This one is active specifically in the set_binding phase.

function mod:validate_set_binding(msg)


— getting the value of the X-Binding header is this easy:

   local mybinding = msg:header(“X-Binding” );


— Assigning a binding of the same name as the header is just as easy:

   local err = msg:binding(mybinding);


— proper etiquette requires us to “end” the function

end;


— and finally we register the script so Momentum can use it.

msys.registerModule(“policy”, mod);

Toggle Comments »

Enabling the script to run with Momentum just requires adding a reference to the ecelerity.conf file:

scriptlet scriptlet {
   script policy {
      source = “/opt/msys/ecelerity/etc/conf/default/lua/policy.lua”
   }
}

That is all there is to it. This is a very basic example but is probably the most common use of scripting in Momentum. Our clients have installed some extremely complex functionality with Lua including auto-responders, message cadence (recipient fatigue) protection and automated database list hygiene systems. If you do a little research you will find that Lua is embedded or core to many popular pieces of familiar code including Angry Birds, Civilization V, World of Warcraft, Far Cry, AutumnOut, the Sputnik wiki engine, and the Cisco Adaptive Security Appliance. We are not alone in thinking Lua is awesome.

If you’re interested in gaining a better understanding of the Lua language, I recommend the book Programming in Lua, Second Edition by Roberto Ierusalimschy — A.K.A. the Lua Bible. There’s also a ton of links on the Internet, and I’ve listed a few of them below. By the way, Lua is a proper name, and is not an acronym. Portuguese for “moon,” Lua was originally developed by a team at the Pontifical Catholic University of Rio de Janeiro in Brazil.

Stepping in a new direction, Message Systems recently announced a new line of software solutions that extend and enhance the messaging capabilities of legacy CRM systems. The first module in this new line is tailored to boost the email and SMS/MMS capabilities of Oracle’s Siebel® CRM application. Momentum for CRM empowers B2C and B2B organizations relying on Oracle’s Siebel® CRM to unleash the full potential of Siebel® as a cross-channel marketing platform for engaging today’s increasingly mobile customers.

CEO George Schlossnagle explained the impetus behind the new solutions as a response to a particular market need: “We’re experiencing strong customer demand for immediate solutions to specific messaging challenges, and an issue that we’ve been asked about repeatedly is how to boost the effectiveness of Oracle’s Siebel® CRM for marketing and customer care. With so many global brands relying on legacy systems, such as Oracle’s Siebel CRM, to engage in multinational markets, we believe this kind of targeted solution offers great value and is one whose time has come.”

In an early trial project, a Momentum for CRM implementation enabled an EMEA-based manufacturer of mobile electronics to facilitate a far greater message delivery capacity across its international marketing operations, improving campaign effectiveness and driving increased revenue. It achieved these goals while improving its deliverability and sending reputation with ISPs. At the same time, this company was able to significantly cut the costs of its mobile marketing operations through policy-based controls in Momentum for CRM that prevented unauthorized sending of SMS/MMS messages from internal users, and also routed messages to its global aggregator network in the most cost-efficient way possible. Get the whole story on that implementation here (PDF), or learn more about Momentum for CRM here.

Oracle and Siebel are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

For most of us, when we hear the phrase social network we think of Facebook. In the U.S., of course, Facebook completely dominates the discussion and the marketplace when it comes to social media. Outside the U.S. it’s a different story. China, especially, has a highly competitive social networking scene, with at least five major players scrambling to capture and build market share within the world’s largest population. These companies provide offerings that mix Twitter-like microblogging services, social games, mobile services and straight-up imitations of Facebook features. This Mashable article with a handy infographic provides a great overview of the players. Right in the middle of the action is Message Systems client Kaixin001.

We first started working with Kaixin001 in early 2011 as they were initially seeking ways to improve their email deliverability, but it became apparent as the relationship progressed that a Momentum implementation could help the company advance its business objectives in multiple ways. Wei Guo, Vice President at Kaixin001 put it this way: “our challenge at Kaixin001 is to innovate, to create a platform that continually engages our member base with better services, new offerings and fun activities. With state-of-the-art messaging powered by the Momentum platform, we have the capacity now to roll out any number of new services, from social and mobile Internet offerings to games and other kinds of opportunities.”

With their Momentum implementation in place, Kaixin001 immediately boosted deliverability consistently above 90%, but it also opened up new possibilities for additional revenue streams. Perhaps most importantly, with a vastly expanded messaging capability and deliverability problems solved, Kaixin001 was able to increase its use of automated status emails (alerts and notifications) for member outreach. And that’s exactly what the company did, creating new services based on message-driven workflows that improved member engagement and strengthened loyalties. Building on its relationship with Message Systems, Kaixin001 is set up to compete more energetically than ever before in the intensely competitive Chinese social media market. Get the full story in English or in Chinese (PDF).

Tech Tips: Momentum Performance Tuning Tips

by Oleksiy Kovyrin, Senior Technical Operations Engineer, LivingSocial

We’re pleased to present Oleksiy Kovyrin as our first guest columnist for Tech Tips. This article originally appeared in Oleksiy’s blog and he shared it in the Message Systems LinkedIn group, which is how we first learned about it. If you’re digging into Momentum or any of our products, we urge you to join this smart and highly vocal group. Take it away Oleksiy:

One of my first tasks as part of the technical operations team at LivingSocial was to figure out a way to make our messaging software perform better and deliver faster. We use Momentum, and it is really fast, but I’m always looking for ways to squeeze as much speed out of our system as possible.

While working on it I’ve created a set of scripts to integrate Momentum with Graphite, for all kinds of crazy stats graphing. Those scripts will be opensourced soon, but for now I’ve decided to share a few tips about performance-related changes we’ve made to improve our performance at least 2x:

  • Use EXT2 Filesystem for the spool storage — After a lot of benchmarking we noticed that we’ve been doing way too much I/O compared to our throughput. Some investigation showed that the EXT3 filesystem we were using for the spool partition resulted in way too high metadata-update overhead because the spool storage uses a lot of really small files. Switching to EXT2 helped us gain at least 50-75% additional performance. Additional performance gain was caused by turning on the noatime option for our spool.There are some sources who claim that using XFS for spool directories is a better option, but we’ve decided to stick with EXT2 for now.
  • Do not use %h{X} macro in your custom logs —Custom logs is an awesome feature of Momentum and we use it to log our bounces along with some information from mail headers. Unfortunately, the most straightforward thing to do (using %h{X} macro) was not the best option for I/O-loaded servers because every time Momentum needs to log a bounce it needs to swap message body in from the disk and parse it to get you the header value.To solve this issue we’ve created a Sieve+ policy script that would extract the headers we need from a message during the initial spooling phase (when the message is still in memory) and put those values into the message metadata. This way, when we need to log those values we won’t have to swap message body in from the disk. Here is the Sieve script to extract header value:SEE FORMATTING FOR THE SCRIPT HERE: http://kovyrin.net/2012/01/07/momentum-ecelerity-tuning-tips/

    require [ “ec_header_get”, “vctx_mess_set”, “ec_log” ];

    # Extract x-ls-send-id header to LsSendId context variable
    # (later used in deliver log)

    ($send_id) = ec_header_get “x-ls-send-id”;

    vctx_mess_set “LsSendId” $send_id;

    Toggle Comments »

    After this we could use it in a custom logger like this:

    custom_logger “custom_logger1”

    {

    delivery_logfile = “cluster:///var/log/ecelerity/ls-delivery_log.cluster=>master”

    delivery_format =”%t@%BI@%i@%CI@D@%r@%R@%m@%M@%H@%p@%g@%b@%vctx_mess{LsSendId}”

    delivery_log_mode = 0664

    }

    Editor’s Note: For those who are not familiar with Momentum, it was formerly named Ecelerity, and this is still how the product is designated in the code, as in line 3.

  • Give more RAM to Momentum —When Momentum receives a message, it stores it to the disk (as required by the SMTP standard) and then tries to deliver the copy it has in memory, and if delivery succeeds, the on-disk copy is unliked. The problem with a really heavy outbound traffic load is that momentum needs to keep tons of emails in memory, but by default it can only hold 250. With a load of 250-500 messages a second this is just too small.To change this limit we’ve increased the Max_Resident_Active_Queue parameter to 1000000 (of course we made sure to have enough RAM to hold that many messages if needed) and Max_Resident_Messages to 0 (which means unlimited). This allows Momentum to keep as many messages resident as possible and reduce the load caused by swap-in operations required for re-delivery attempts, etc.Editor’s Note (IMPORTANT): Changing the Max_Resident_Active_Queue and Max_Resident_Messages are advanced settings which require careful planning and a thorough understanding of how those changes will impact the memory in your system. The changes listed above have worked for Living Social, but they won’t work for everyone. In particular, the amount of memory available to you, which features you are making use of in the system, and your average message size, can all affect the sizing of these two parameters and their impact on the system. We recommend you carefully review the documentation on these settings, and follow up with support on any further questions or concerns you may have before implementing changes to these settings.
  • Choose a proper size for your I/O-related thread pools —In the default Momentum configuration the SwapIn and SwapOut thread pool sizes are set to 20. Under a really high load even on our 4xSAS15k RAID10 this tends to be too high a value. We’ve switched those pools to 8 threads each, and this helped to reduce I/O contention and overall I/O throughput.In summary, as with any optimizations, before tuning your system, it really helps to set up as much monitoring for your Momentum servers as possible: Cacti graphs, Graphite (mentioned above), Ganglia or something else — it doesn’t matter. Just make sure you can observe all the aspects of your system performance and understand what is going on with your system before changing any performance-related settings.

I wanted to get a note into our first newsletter of 2012 because in so many ways, 2011 was a watershed year for Message Systems. First, we’re growing like gangbusters. We’ve added 25 entirely new brands to our client list in 2011, and expanded our relationships with many of our existing clients. To keep that dizzying pace going through 2012 and beyond we’ve expanded our headcount by nearly 70% over the past 12 months. Not surprisingly this required us to expand our offices: We secured a new suite in a building adjacent to our headquarters in Columbia, so we now have all of our engineers in one location. And, our San Francisco offices have reached capacity, so we’ll be almost doubling our space there in the coming months.

2011 also saw us hosting our first-ever user conference in San Diego, Interact2011, which was a resounding success, attracting more than a hundred attendees. The chance to hear from our customers first-hand was invaluable, and we had a darn good time, too. If you didn’t attend Interact2011, I urge you not to miss Interact2012. You can register now, in fact, for the August event.

Another aspect of our growth story is that we’ve been able to accelerate our product development. As such, we’re moving to a quarterly release schedule. We believe this is going to help us to respond more rapidly to customer needs, and we’ll be working to keep customers better apprised of our progress as we push the envelope on what’s possible with digital messaging.

To that end, we’re covering Momentum 3.4.0 in our Product Spotlight in this issue of the newsletter. With Momentum 3.4.0, you can now experiment with transforming message content on the fly between not only email and SMS – which you can do now with our Mobile Momentum product – but also IM. Imagine, being able to keep a conversation going with a customer or partner over email, SMS and IM chat, bouncing from one channel to another throughout the day without losing the context of the conversation. We’re making cross-channel messaging a reality.

We’ve also got a featured customer story on our friends at Infusionsoft, a marketing automation firm based in Arizona that’s using Momentum’s Adaptive Delivery® feature to drive deliverability rates consistently up into the 99% range. And, we’ve turned our Tech Tips column this month over to Oleksiy Kovyrin, Senior Technical Operations Engineer at LivingSocial, who was kind enough to share his tips on tuning Momentum to your specific environment for best performance.

In 2012, we’re focused on one thing: Helping you, our customers or potential customers, to be successful. Have an idea on how we can do that? Let us know!

Momentum features a configuration repository that resides on the Cluster Manager (CM). This allows Momentum server nodes to pull configuration updates every minute (from cron). Local copies of this configuration reside in a “working copy (WC) directory” of the same name on EACH cluster node (server and manager):

/opt/msys/ecelerity/etc/conf/default

The Momentum Web UI provides customers with the ability to make configuration changes from within the UI. While some customers use this interface, many choose to make configuration modifications from the OS command line. This article describes a best practice for making configuration changes to ecelerity.conf from the command line in a single subcluster Momentum environment.

  1. Login to one your Momentum server nodes (not the CM)
  2. Be sure that you’re using the latest version of the configuration. Invoking this from the command line:
    /opt/msys/ecelerity/bin/eccfg pull –u username –p user_password
  3. Edit the configuration. Use your favorite editor such as vim or nano to make changes to ecelerity.conf from within the local WC directory of the Momentum server and save.
  4. Check the syntax. Do this by invoking
    /opt/msys/ecelerity/bin/validate_config
    This application will return “Configuration valid” for valid configurations. However, this will “return quietly” if the configuration is NOT valid. Running ec_dump_config will give you verbose output and give you a “good hint” relative to your configuration error.
  5. Check the configuration by running “config reload” from within ec_console.
  6. Commit the configuration to the repository. Use the eccfg application:
    /opt/msys/ecelerity/bin/eccfg commit –u username –p user_password

NOTE: There are some cases where validate_config may give you a false positive. One example is when using “node local” configurations (see below). In these cases, you’ll need to run validate_config and config reload on EACH Momentum server to be sure that your local configuration is valid.

Other configuration-related tips:

  1. Momentum allows “node local” configuration: Create a directory such as
    /opt/msys/ecelerity/etc/conf/serverhostname
    where “serverhostname” is the hostname. Any configuration files that are saved in this directory will be included in ecelerity.conf (assuming you’ve used the include directive within ecelerity.conf) because these “node local” directories are part of Momentum’s default search path for configuration files.
  2. There is also a global repository that contains configuration that is common to every node in every subcluster (eg. mbus.conf). That checkout resides here:
    /opt/msys/ecelerity/etc/conf/global
  3. Check sieve syntax. Use this ec_console directive (for example):
    sieve:sieve testfile each_rcpt_phase1 /path/to/each_rcpt.siv
  4. Check Lua syntax. Use rcluac
    /opt/msys/3rdParty/bin/rcluac    file.lua

Tom Mairs
Manager, Solution Engineering

Tom Cain
Manager, Technical Training