Nagios Log Server - Sending Multiline Log Files Using Syslog

Overview

This KB article explains how to configure syslog to send multi-line logs to Nagios Log Server (NLS).

What is a multi-line log? This is when the data that encompasses the entire event is spread across multiple lines in the log file. For example:

2017.02.13 14:42:07:198 EST | Audit | write2216 | express_admin | System | UserDeleteRequest

    <?xml version="1.0" encoding="UTF-8"?>
<BroadsoftDocument protocol="OCI" xmlns="C">
  <userId xmlns="">express_admin</userId>
  <command xsi:type="UserDeleteRequest" xmlns="" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <userId>2136658700</userId>
  </command>
</BroadsoftDocument>

Normally when you configure syslog to send a custom log file to NLS it is sent on a line-by-line basis. This can make it complicated to review the logs on NLS as it will be displayed as multiple events.

In the example above, you can see the first line starts with the date time format yyyy.mm.dd hh:mm:ss:ms. Every entry recorded in this log file will always have this first line formatted this way.  Syslog can be configured to identify this string and then send the entire data to NLS as a multi-line log.

NLS will also have an extra configuration input added to handle the incoming multi-line data.

This KB article will walk you through the steps to implement this.

Scenario Details

To properly demonstrate how this works, the following KB article will use the log file /var/log/AuditLog.log to send to NLS.

To simulate a multiple line log entry being added to the log, a second file will be created called /var/log/test.log with the following contents:

2017.02.13 14:42:07:198 EST | Audit | write2216 | express_admin | System | UserDeleteRequest

    <?xml version="1.0" encoding="UTF-8"?>
<BroadsoftDocument protocol="OCI" xmlns="C">
  <userId xmlns="">express_admin</userId>
  <command xsi:type="UserDeleteRequest" xmlns="" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <userId>2136658700</userId>
  </command>
</BroadsoftDocument>

The following command in a terminal session will append the data to the /var/log/AuditLog.log:

more /var/log/test.log >> /var/log/AuditLog.log

Using those steps you will be able to successfully follow this KB article and confirm the functionality works. Every time the command is executed above, a multi-line entry is added to the /var/log/AuditLog.log log file. Even though technically the date is incorrect this will not matter, it's simply an example.

Configure Nagios Log Server

The first step is to configure the NLS input to identify multi-line logs.

Log into your NLS web interface and navigate to Administration > Global Configuration.

Under Inputs use the + Add input input drop down list and click Custom. This will add a new block underneath.

Give it a name such as Syslog Multiline.

In the text area add the following:

tcp {
    codec => multiline {
        pattern => '^[0-9]{4}.[0-9]{2}.[0-9]{2}'
        negate => true
        what => previous
    }
    port => 6677
    type => 'syslog_multiline'
}

Click the Save button.

Then click the Verfiy button above to ensure this is a valid configuration.

Once the verification process is OK, in the left pane under Configuration Editor click Apply Configuration.

Click the Apply button.

Click Yes, Apply Now

Once this process has finished you can continue onto the next section. You'll return back to NLS once syslog has been configured.

The input that was just added is listening on port 6677, this will be used in the steps below when configuring syslog.

What was does all of that mean?

The line codec => multiline { tells Logstash to use the multiline codec filter.

The pattern tells Logstash that the following sting format is first line of a log entry:

pattern => '^[0-9]{4}.[0-9]{2}.[0-9]{2}'

• This is a regular expression (regex)

• The ^ means that the line begins with this pattern

• [0-9]{4} indicates that there are four digits in the range 0 to 9 (the year)

• The . period matches a single character, without caring what that character is. This represents the character that separates the year and the month, it's just co-incidental that this character is actually a . period.

• [0-9]{2} indicates that there are two digits in the range 0 to 9 (the month)

• The . period matches a single character.

• [0-9]{2} indicates that there are two digits in the range 0 to 9 (the day)

Basically it's saying this is the format of the string which needs to be matched:

dddd.dd.dd

Remember the example we have:

2017.02.13 14:42:07:198 EST | Audit | write2216 | express_admin | System | UserDeleteRequest

The negate line:

negate => true

true means that a received message not matching the pattern will constitute a match of the multiline filter and the what will be applied.

The what line:

what => previous

previous says that any line not starting with the pattern should be merged with the previous line.

The type line:

type => 'syslog_multiline'

Is how this received entry is indexed as, it will help with searches later.

 

Configure syslog

Establish a terminal session to the Linux server that has the log file in question.

Syslog is going to be configured to watch the /var/log/AuditLog.log log file and send it to NLS. This example will use the NLS with the address 10.25.5.99 and it is listening on port 5566.

In your terminal session execute these commands:

cd /tmp
curl -s -O http://10.25.5.99/nagioslogserver/scripts/setup-linux.sh
sudo bash setup-linux.sh -s 10.25.5.99 -p 6677 -f "/var/log/AuditLog.log" -t syslog_multiline

This will create a syslog configuration file called /etc/rsyslog.d/90-nagioslogserver_var_log_AuditLog.log.conf.

A change needs to be made to this configuration file, edit the file in the vi editor using the following command:

vi /etc/rsyslog.d/90-nagioslogserver_var_log_AuditLog.log.conf

When using the vi editor, to make changes press i on the keyboard first to enter insert mode. Press Esc to exit insert mode.

Make the changes highlighted in bold:

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for syslog_multiline
$InputFileName /var/log/AuditLog.log
$InputFileTag syslog_multiline:
$InputFileStateFile nls-state-var_log_AuditLog.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template clean,"%rawmsg%"

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'syslog_multiline' then @@10.25.5.99:6677;clean
if $programname == 'syslog_multiline' then ~

You can see that the line $template clean,"%rawmsg%" was added and ;clean was added to the end of the second last line.

This will change the rsyslog configuration to apply the clean filter, which is just sending the raw message using rsyslog's %rawmsg% macro.

When you have finished, save the changes in vi by typing:

:wq

and press Enter.

Finally you need to restart the rsyslog service with the following command:

service rsyslog restart

Test

Now you can test that it is working by executing the following command on your Linux machine:

more /var/log/test.log >> /var/log/AuditLog.log

Now open the NLS web interface and navigate to Dashboards.

You should now see an entry like the following:

 

Summary

This KB article showed you how to use syslog to send multi-line log files to Nagios Log Server. Armed with this information you should be able to apply this to your situation.

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/