Log Formats for Version 3.x

March 26, 2020 Contributors

This section documents the log formats used in Momentum version 3.0 and higher.

The mainlog.ec Format

The mainlog.ec file logs reception, delivery, transient failure, permanent failure and heartbeat events. The log entry format differs depending upon the event type. The different formats are described in the following sections.

E.1.1.1. Reception Records

Every reception that Momentum performs is written to the mainlog.ec file as a single line as shown in the example below.

1064868656@00/00-25004-31B987F3@00/00-03736-F4101B54@00/00-04532-A3456B54@R »
@bob@example.fict@info@postalengine.com@10.0.1.1@201@esmtp@default@default

As with all log file entries, the fields are delimited by @ signs. The fields are:

Offset Field Meaning
0 1064868656 The date of reception in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 00/00-25004-31B987F3 The message’s message-id. This is a unique value per message. This also corresponds with its location on disk.
2 00/00-03736-F4101B54 The batch ID
3 00/00-04532-A3456B54 The connection ID
4 R R’ indicates a reception.
5 bob The localpart of the recipient.
6 example.fict The domain of the recipient.
7 info The localpart of the envelope sender.
8 postalengine.com The domain of the envelope sender.
9 10.0.1.1 The IP address the message was received from.
10 201 The size of the message in bytes.
11 esmtp The protocol the message was received over (either esmtp or ecstream). If the message was transferred from another node, the "protocol" will be xfer. The value internal indicates an internally generated bounce or a message generated by Sieve.
12 default The MultiVIP® binding group that the message was assigned to.
13 default The MultiVIP® binding that the message was assigned to.

E.1.1.2. Delivery Records

A delivery line is written to mainlog.ec for every message that is delivered by Momentum. In a cluster configuration, transfers between nodes are also logged.

The log entry is an @ delimited string such as the following:

1064871280@20/00-25593-945A87F3@00/00-03736-F4101B54@00/00-04532-A3456B54@D@ »
postalengine.com@266@group-a@binding-a@0@0.393@10.0.0.1

A description of the fields follows:

Offset Field Meaning
0 1064871280 The date of delivery in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 20/00-25593-945A87F3 The message’s message-id. This is a unique value per message.
2 00/00-03736-F4101B54 The batch ID
3 00/00-04532-A3456B54 The connection ID
4 D D’ marks this line as a successful delivery. In a cluster configuration, transfers between nodes are indicated by an ‘X’. ‘X’ entries appear in the delivery log on the transferring node indicating that the message left for another cluster node. The node receiving the message will have an ‘R’ entry in its log and the "protocol" of the message will be xfer.
5 postalengine.com The destination domain.
6 266 The size in bytes of the delivered message.
7 group-a The MultiVIP® binding group for this message.
8 binding-a The MultiVIP® binding for this message.
9 0 The number of times the message has been retried. Since this was the initial attempt, the value is 0.
10 0.393 The amount of time, in seconds, between reception and delivery.
11 10.0.0.1 The IP address that accepted the message.

E.1.1.3. Transient Failure Records

A transient failure is a failure which allows retry of the same message. The ec_logger module allows for optional logging of transient failures.

A transient failure line is an @ delimited string like the following:

1064869327@00/00-25593-CBD987F3@00/00-03736-F4101B54@00/00-04532-A3456B54@T@ »
example.fict@0@group-a@binding-a@15@0@18.53@10.0.0.1@421 no adequate servers

A description of the fields follows:

Offset Field Meaning
0 1064869327 The date of transient failure in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 00/00-25004-31B987F3 The message’s message-id. This is a unique value per message. This also corresponds with its location on disk.
2 00/00-03736-F4101B54 The batch ID
3 00/00-04532-A3456B54 The connection ID
4 T T’ marks this log line as a transient failure.
5 example.fict The destination domain.
6 0 The number of bytes of data transferred before the failure occurred. Since no connection could be made for bob@example.fict (it’s a bogus domain), 0 bytes where transferred.
7 group-a The MultiVIP® binding group for this message.
8 binding-a The MultiVIP® binding for this message.
9 15 The stage of the message. Stage descriptions can be found at “Connection Stages”.
10 0 The number of times the message has been retried. Since this was the initial attempt, the value is 0.
11 18.53 The amount of time, in seconds, between reception and this transient failure.
12 10.0.0.1 The IP address of the server which rejected the message.
13 421 no adequate servers The error message associated with the failure.

E.1.1.4. Permanent Failure Records

A permanent failure indicates that the message that was attempted failed in such a way that it should not be retried further. After logging a permanent failure the message will be discarded.

A permanent failure line is an ‘@’ delimited string like the following:

1064870847@10/00-25593-393A87F3@00/00-03736-F4101B54@00/00-04532-A3456B54@P@ »
postalengine.com@31@group-a@binding-a@5@1@3.89@10.0.0.1@552 No such account

A description of the fields follows:

Offset Field Meaning
0 1064870847 The date of permanent failure in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 10/00-25593-393A87F3 The message’s message-id. This is a unique value per message. This also corresponds with its location on disk.
2 00/00-03736-F4101B54 The batch ID
3 00/00-04532-A3456B54 The connection ID
4 P P’ marks the log line as a permanent failure.
5 postalengine.com The destination domain.
6 31 The number of bytes of data transferred before the failure occurred. Here, 31 bytes were transferred before the remote server returned its error.
7 group-a The MultiVIP® binding group for this message.
8 binding-a The MultiVIP® binding for this message.
9 5 The stage of the message. Stage descriptions can be found at “Connection Stages”.
10 1 The number of times the message has been retried.
11 3.89 The amount of time, in seconds, between reception and the time of permanent failure.
12 10.0.0.1 The IP address of the server that rejected the message.
13 552 No such account The error message returned from the remote host.

E.1.1.5. Heartbeat Records

A heartbeat is written periodically to the log, to indicate that Momentum is still active and may log further data. The heartbeats are used by the web console for real-time reporting.

1251470342@@@@M1

A description of the fields follows:

Offset Field Meaning
0 1191248269 The date of delivery in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 Unused.
2 Unused.
3 Unused.
4 M1 M1’ marks this line as a heartbeat. Having a different type identifier in version 3.x, helps differentiate log entries. This is useful when a log has both version 2.2 and version 3.x entries.

Having three unused fields ensures that, like other logs, the fifth field is the log entry type. This makes parsing easier.

The rejectlog.ec Format

This section documents the rejectlog.ec format. Every rejection that Momentum performs is written to the rejectlog.ec file as a single line as shown in the following example:

1236672125: R="10.79.25.142:42601" L="10.79.25.142:25" C="18/00-07149-D7E16B94" PATH="default" »
PATH_GRP="default" P="awaiting mailfrom" E=550 M=scriptlet »
CTXCONN=[pathway=default,ehlo_string="EHLO rh52-node1",ehlo_domain=rh52-node1] »
CTXMESS=[mailfrom_domain=,mailfrom_string="MAIL FROM:<>",mailfrom_localpart=] relaying denied

In this case, the fields are delimited by spaces. The fields are:

Offset Field Meaning
0 1236672125: The date of delivery in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 R="10.79.25.142:42601" Remote IP:port
2 L="10.79.25.142:25" Local IP:port
3 C="18/00-07149-D7E16B94" Connection ID
4 PATH="default" Pathway
5 PATH_GRP="default" Pathway Group
6 P="awaiting mailfrom" Phase
7 E=550 Error Code
8 M=scriptlet Last Module run
9 CTXCONN=[pathway=default,ehlo_string="EHLO rh52-node1",ehlo_domain=rh52-node1] Validation Context for connection
10 CTXMESS=[mailfrom_domain=,mailfrom_string="MAIL FROM:<>",mailfrom_localpart=] Validation Context for message
11 relaying denied Error message

The rejectlog.ec file also has heartbeat entries in the following format:

1252064908: Marker 1

The "Marker 1" entry follows the Unix timestamp and is separated from it by a space.

The acctlog.ec Format

The acctlog.ec log contains both authentication entries and authorization entries for the SMTP and Control Listeners. For more information on configuring listeners for logging authentication and authorization events see “Authentication, Authorization and Accounting”.

E.1.3.1. Authentication Records

If enabled for the listener, authentication events for Unix domain sockets are logged one per line as shown below:

1160503808@N@/tmp/2025@@ec-user@1

If enabled for the listener, authentication events for TCP/IP listeners are logged one per line as shown below:

1160172232@N@*:2025@10.80.116.126:37164@ec_user@1

The fields are delimited by @ signs. @, \, \n and other control characters appearing in a field are escaped by adding an \ before them. The fields are:

Offset Field Meaning
0 1160172232 The date of authentication in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 N N’ indicates that this is a log for authentication.
2 *:2025 The listener endpoint on which the event occurred.
3 10.80.116.126:37164 The IP and port of the peer. For Unix domain connections, this field will be empty.
4 ec_user The user name used for the authentication.
5 1 1 means the authentication is successful. 0 otherwise.

E.1.3.2. Authorization Records

A line is written to acctlog.ec for every authorization event. The line is an @ delimited string. @, \, \n and other control characters appearing in a field are escaped by adding an \ before them.

Authorization entries might look like the following:

1160503811@Z@/tmp/2025@@ec-user@1@summary@users
1160504707@Z@/tmp/2025@@ec-user@0@shutdown
1160172223@Z@*:2025@10.80.116.126:37162@ec-user@1@summary@users
1160172219@Z@*:2025@10.80.116.126:37162@ec-user@0@shutdown

A description of the fields follows:

Offset Field Meaning
0 1160172219 The date of authorization in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 Z Z’ indicates that this is an authorization entry.
2 /tmp/2025 or *:2025 The listener endpoint on which the event occurred.
3 10.80.116.126:37162 The IP and port of the peer. For Unix domain connections this field will be blank.
4 ec_user The user name used for the authentication.
5 1 1 means the user is authorized to run the command. 0 means the authorization failed. -1 means some error occurred during authorization.
6 summary The command that was requested to run.
7 users The role that matched during successful authorization.

Note

The ‘T’ type indicator denotes an authentication timeout and the ‘?’ type indicator denotes an unknown acctlog type.

The importlog Format

You can record the outcome of a spool import operation (see spool import) by configuring a path for the importlog. To do this add the following line to the ec_logger module:

...
importlog = /var/log/ecelerity/importlog.ec

By default no log rotation is performed for the importlog.

The format for the importlog is:

Offset Field Meaning
0 1064869327 The date of the event in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 00/00-25004-31B987F3 The message’s original in-spool message-id. This is a unique value per message. This corresponds with its location in the spool being imported.
2 I I’ indicates that this log line is an import event.
3 00/00-25004-31B987F3 The message’s new in-spool message-id. This is usually the same as the message-id recorded in field 1, but may be altered during import to avoid collisions with existing messages with the same identifier.
4 1 The result indicator. This is a number between 1 and 4 with the following meanings. 1 is complete success, 2 indicates failure during the read of metadata, 3 is failure reading the message from the spool and 4 is failure writing the message into the main Momentum spool.
5 /var/spool/my-alternative-spool The base directory containing the spool being imported.

The bounce_logger

bounce_logger writes a single logfile, bouncelog.ec, which records both in-band (or protocol-time) and out-of-band bounces. The format was designed to be a concise, machine-parseable, computationally inexpensive and complete format for writing information about every bounce message.

E.1.5.1. The bounce_logger Format

The bouncelog.ec file logs bounces and heartbeats. The log entry format differs depending upon the event type.

E.1.5.1.1. Bounce Records

Every bounce that Momentum witnesses is written to the bouncelog.ec file as a single line as show in the example below. (this example has been wrapped for display purposes; it will be a single line in the log file).

1064868656@91/6D-07914-E67BC044@00/00-03736-F4101B54@00/00-04532-A3456B54@B@ »
johndoe@example.fict@info@postalengine.com@group-a@binding-a@21@24@1223@10.0.0.1@554 »
 5.4.7 [internal] exceeded max time without delivery

As in all ec_logger-formatted entries, the fields are delimited by @ signs. The fields are:

Offset Field Meaning
0 1064868656 The date of reception in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 91/6D-07914-E67BC044 The message’s message-id. This is a unique value per message. This also corresponds with its location on disk.
2 00/00-03736-F4101B54 The batch ID
3 00/00-04532-A3456B54 The connection ID
4 B The log type. ‘B’ indicates a bounce. If the log_transient_failures option is on, transient failures are indicated by ‘T’.
5 johndoe The localpart of the recipient of the original message.
6 example.fict The domain of the recipient.
7 info The localpart of the envelope sender.
8 postalengine.com The domain of the envelope sender.
9 group-a The binding group the bounce email was bound to, if available.
10 binding-a The binding the bounce email was bound to, if available.
11 21 The phase the message occurred in. See “Connection Stages” for details.
12 24 The classification code for the message. 24 is a Timeout-Before-Delivery error. See “Bounce Classification Codes” for the full code list.
13 1223 The message size.
14 10.0.0.1 The IP address of the server that bounced the message.
15 554 5.4.7 [internal] exceeded max time without delivery The raw bounce message from the server.

The bounce logs also have heartbeat entries in the following format:

1251222268@@@@M1

A description of the fields follows:

Offset Field Meaning
0 1191248269 The date of delivery in Unix timestamp format (seconds since 00:00:00 Jan 1, 1970).
1 Unused.
2 Unused.
3 Unused.
4 M1 M1’ marks this line as a heartbeat. Having a different type identifier in version 3.x, helps differentiate log entries. This is useful when a log has both version 2.2 and version 3.x entries.

Having three unused fields ensures that, like other logs, the fifth field is the log entry type. This makes parsing easier.

E.1.5.1.2. Bounce Classification Codes

The bounce classification codes are identical for all versions of Momentum. For a list of the codes and their meanings see “Bounce Classification Codes”.