Nagios Log Server - Configuration Overview

Log Server Configuration Overview

Nagios Log Server utilizes logstash for collecting logs from various systems. Logstash has an extensive list of different inputs, filters and outputs available, and Log Server gives users the ability to configure these through the web UI under Administration menu. Users can set Global Configurations or per Instance configurations. All instances use the Global Configurations, and Per Instance configurations are only applied to the instance they are configured for.

Formatting and Creating Blocks

When creating inputs, filters and outputs, users can create named blocks in each section that will be added to the configuration files in the order they are in the web UI. Once blocks are created they can be dragged and dropped to re-order.

Standard logstash configuration syntax can be used with one exception, each section, inputs, filters and outputs will be wrapped with their respective start and end tags and should not be entered. For example, if you saw the following block on the logstash documentation site:

input {
    file {
        path => "/tmp/access_log"
        start_position => beginning
    }
}

To duplicate this in a configuration block you would just use the following:

file {
    path => "/tmp/access_log"
    start_position => beginning
    }

When creating inputs, there are a couple considerations that should be taken into account:

1. Do not use the same port more than once for items that listed on TCP or UDP ports

2. Ports under 1024 are privileged and will require following this documentation to run logstash as a privileged user

Saving and Applying Configuration

At any point during configuration you can save your changes by clicking the red Save button, however this just saves the current configuration to the datastore and does not verify it is valid, nor does it apply the configuration to make sure that it is valid.

Before applying the configuration it is good practice to click the Verify button to allow the configuration files to be validated. If there is an error in the configuration if will display in the popup, otherwise you will receive an OK message.

Once the configuration has been verified, you can click Apply configuration that have the configuration files written form the datastore to all the instances in the cluster. During an Apply configuration process, each Log Server instance will do the following:

1. Take a snapshot of the current configuration files that can be used to restore previous configurations

2. Read all the global and per instance configuration files out of the datastore and write them to the filesystem

3. Restart the logstash service on each instance to start using the new configuration files

Configuration files are written on each instance to /usr/local/nagioslogserver/logstash/etc/conf.d/

-rwxrwxr-x. 1 nagios nagios 777 Jan 26 13:38 000_inputs.conf
-rwxrwxr-x. 1 nagios nagios 978 Jan 26 13:38 500_filters.conf
-rwxrwxr-x. 1 nagios nagios 493 Jan 26 13:38 999_outputs.conf

Outputs

Nagios Log Server automatically adds a default output of the local Elasticsearch datastore, and this will not be displayed to end users as it is not optional and not user configurable. Additional output may however be added too the system by clicking the "show Outputs" icon on the configuration page and using any of the standard logstash outputs and the block formatting outlined above. This can be useful if a user would also like a text archive of their logs in addition to the default elasticsearch output.

Conditional Syntax

Nagios Log Server configuration blocks also allow for standard logstash conditional syntax giving users the ability for example to only apply filters to some of the logs going through. Conditional syntax may be see in use in the Apache (Default) filter that comes pre-shipped with Log Server, but lets consider the following syntax in the filter section, we will walk through what is happening.

if [program] == 'apache_access' {
    grok {
        match => [ 'message', '%{COMBINEDAPACHELOG}']
    }
}

This syntax is saying, if there is a field named program and the value of that field is apache_access we are going to take the message filed and pass it to the grok filter to explode the message into different parts. If the program field doesn't exist, or doesn't equal apache_access this filter will be skipped.

Troubleshooting Problems

Problem: The logstash process is dying or not listening on the port added

Potential Causes:

• New configuration files were not able to be written to the config directory /usr/local/nagioslogserver/logstash/etc/conf.d/

• There may be a problem with the server already utilizing the same port.

• Check the /var/log/logstash/logstash.log log file for errors

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/