Introduction

Nagios Log Server R2 is a powerful tool for centralized log management and analysis, but like any logging system, it may encounter indexing issues that cause slow searches, missing logs, or delayed data processing. Proper log indexing ensures logs are efficiently stored and quickly retrievable when needed for monitoring, security analysis, or compliance reporting.

In this guide, we’ll walk through the common causes of log indexing issues in Nagios Log Server R2 and how to troubleshoot them effectively.

1. Identifying the Root Cause of Indexing Issues

Before applying a fix, determine whether the issue is related to:

• Slow searches – Log queries take too long to return results.
• Missing logs – Some logs are not appearing in searches.
• Delayed log indexing – Logs arrive but are indexed late.

To check indexing status and shard health, run:

curl -X GET "http://localhost:9200/_cluster/health?pretty"

curl -X GET "http://localhost:9200/_cluster/health?pretty"

A green status means the cluster is healthy, while yellow or red indicates indexing problems.

2. Fixing Slow Log Searches in Nagios Log Server 2024R2

A. Check System Resources

Run the following command to check CPU, memory, and disk usage:

top
df -h

top
df -h

If CPU or memory usage is consistently high, consider:

• Adding more system resources (RAM, CPU)
• Restarting the Nagios Log Server service to free up memory: systemctl restart nagioslogserver

B. Optimize Shard Allocation

Too many small shards slow down queries. Check the number of shards:

curl -X GET "http://localhost:9200/_cat/shards?v"

curl -X GET "http://localhost:9200/_cat/shards?v"

If you have too many shards, consider merging indices or increasing shard size.

To adjust shard count for a new index:

curl -X PUT "http://localhost:9200/_settings" -H "Content-Type: application/json" -d '
{
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}'

curl -X PUT "http://localhost:9200/_settings" -H "Content-Type: application/json" -d '
{
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}'

For existing indices, use the force merge command:

curl -X POST "http://localhost:9200/_forcemerge?max_num_segments=1"

curl -X POST "http://localhost:9200/_forcemerge?max_num_segments=1"

This reduces overhead and speeds up searches.

C. Limit Search Query Scope

Narrow down search queries to specific time ranges or log sources:

• Use shorter time frames instead of searching all logs
• Filter by specific log sources (firewall, Linux, Windows logs)

3. Fixing Missing Logs in Nagios Log Server R2

A. Verify Log Input Sources

Check if log sources (servers, firewalls, applications) are sending data.

1. List active log inputs: curl -X GET "http://localhost:9200/_cat/indices?v"
2. If an expected index is missing, restart log collection: systemctl restart rsyslog

B. Check Log Collection Services

If Nagios Log Server isn’t receiving logs, restart the Logstash service:

systemctl restart logstash

systemctl restart logstash

Check the Logstash logs for errors:

tail -f /var/log/logstash/logstash.log

tail -f /var/log/logstash/logstash.log

If Logstash is down, re-enable it:

systemctl enable --now logstash

systemctl enable --now logstash

C. Verify Firewall and Network Connectivity

If logs are not arriving, check firewall rules on the source system:

sudo firewall-cmd --list-all

sudo firewall-cmd --list-all

Make sure the firewall allows traffic on port 514 (Syslog) or 9200 (Opensearch).

To open ports:

sudo firewall-cmd --add-port=514/udp --permanent
sudo firewall-cmd --reload

sudo firewall-cmd --add-port=514/udp --permanent
sudo firewall-cmd --reload

4. Fixing Delayed Log Indexing

A. Check Indexing Queue Size

If logs are delayed, check the queue size:

curl -X GET "http://localhost:9200/_cluster/pending_tasks"

curl -X GET "http://localhost:9200/_cluster/pending_tasks"

If there are too many pending tasks, restart the OpenSearch to clear the backlog:

systemctl restart opensearch

systemctl restart opensearch

B. Increase Java Heap Size for OpenSearch

If Elasticsearch is running out of memory, increase heap size:

1. Edit the jvm.options file: nano /etc/opensearch/jvm.options
2. Increase memory allocation (e.g., from 8gb to 16gb): -Xms4g -Xmx4g
3. Restart opensearch: systemctl restart opensearch

C. Reduce Log Retention Period

If old logs consume too much storage, delete outdated logs:

curl -X DELETE "http://localhost:9200/logs-2023.01*"

curl -X DELETE "http://localhost:9200/logs-2023.01*"

Or set an automatic log retention policy:

curl -X PUT "http://localhost:9200/_ilm/policy/log_cleanup" -H "Content-Type: application/json" -d '
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

curl -X PUT "http://localhost:9200/_ilm/policy/log_cleanup" -H "Content-Type: application/json" -d '
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

This automatically deletes logs older than 90 days, freeing up space.

5. Verifying Cluster Health and Performance

To ensure Nagios Log Server R2 is functioning correctly, regularly check:

• Cluster health: curl -X GET "http://localhost:9200/_cluster/health?pretty"
• Disk space usage: df -h
• Logstash service status: systemctl status logstash

Conclusion

Properly managing log indexing in Nagios Log Server R2 ensures faster searches, real-time log analysis, and efficient storage utilization. Regular monitoring and maintenance will keep Nagios Log Server 2024R2 running efficiently. If issues persist, check back on Nagios documentation, consider maintenance and support, or ask the Nagios Support Forum.

Happy monitoring!

Step 1: Investigate Database Issues

Nagios XI relies on MySQL or MariaDB to store configuration data. A corrupted database or missing entries can cause services to disappear.

1. Log in to the Database Server:

mysql -u root -p

mysql -u root -p

2. Check Database Integrity:

USE nagios; 
CHECK TABLE nagios_services; 

USE nagios; 
CHECK TABLE nagios_services; 

If errors are detected, repair the table:

REPAIR TABLE nagios_services;

REPAIR TABLE nagios_services;

3. Confirm Service Entries Exist:

SELECT service_description, host_name FROM nagios_services;

SELECT service_description, host_name FROM nagios_services;

Verify that your missing service is listed. If it’s not, it may have been deleted or never saved.

4. Restart the Database Service:

systemctl restart mariadb
# or, for MySQL:
systemctl restart mysql

systemctl restart mariadb
# or, for MySQL:
systemctl restart mysql

5. Optimize for Prevention:

OPTIMIZE TABLE nagios_services;

OPTIMIZE TABLE nagios_services;

This improves performance and helps prevent future database issues.

Step 2: Ensure Adequate Disk Space

Low disk space can disrupt database writes, causing services to vanish from Nagios XI.

1. Check Disk Usage:

df -h

df -h

If any partition is at 100%, free up space immediately.

2. Inspect MySQL Storage:

du -sh /var/lib/mysql

du -sh /var/lib/mysql

Excessive size here might indicate log or data buildup.

3. Clean Up Excess Files:

• Remove old Nagios logs:

rm -rf /var/log/nagios/*.log

rm -rf /var/log/nagios/*.log

• Clear temporary files:

rm -rf /tmp/*

rm -rf /tmp/*

• Purge MySQL binary logs (use with caution):

mysql -u root -p -e "PURGE BINARY LOGS BEFORE NOW();"

mysql -u root -p -e "PURGE BINARY LOGS BEFORE NOW();"

4. Expand Storage if Needed:
If space remains critically low, resize the partition or add more storage.

Step 3: Validate Configuration Files

A misconfiguration or syntax error can silently drop services from Nagios XI.

1. Test the Configuration:

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Review and fix any errors in the output.

2. Search for the Missing Service:

grep -r "service_name" /usr/local/nagios/etc/services/

grep -r "service_name" /usr/local/nagios/etc/services/

Replace service_name with your missing service’s name. If it’s not found, it may have been removed.

3. Apply Manual Changes Properly:
If you’ve edited configs manually, reapply them:

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

4. Restart Nagios:

systemctl restart nagios

systemctl restart nagios

Step 4: Check Audit Logs

Nagios XI includes audit logs that track changes to configurations and settings.

1. View Audit Logs: Navigate to Admin > System Information > Audit Log

• Look for any recent deletions or changes to service configurations.
• If you find an unintended deletion, re-add the service.

2. Check Event Logs for Errors:

tail -f /usr/local/nagiosxi/var/eventman.log

tail -f /usr/local/nagiosxi/var/eventman.log

This helps identify potential background processing failures.

Step 5: Restore from a Backup

If the service remains missing, a backup restore might be your best bet.

1. Locate Backups:

ls -l /store/backups/nagiosxi

ls -l /store/backups/nagiosxi

2. Perform the Restore:

/usr/local/nagiosxi/scripts/restore_xi.sh /store/backups/nagiosxi/backup_file.tar.gz

/usr/local/nagiosxi/scripts/restore_xi.sh /store/backups/nagiosxi/backup_file.tar.gz

Replace backup_file.tar.gz with the desired backup file.

3. Confirm Success: Log into Nagios XI and check if the missing service reappears.

Step 6: Check Permissions and System Health

Permission issues or resource strain could be to blame.

1. Fix Ownership:

chown -R nagios:nagios /usr/local/nagios
chown -R nagios:nagios /var/lib/nagios

chown -R nagios:nagios /usr/local/nagios
chown -R nagios:nagios /var/lib/nagios

2. Restart Related Services:

systemctl restart nagios
systemctl restart httpd
systemctl restart ndo2db

systemctl restart nagios
systemctl restart httpd
systemctl restart ndo2db

3. Monitor Resources:

top

top

High CPU or memory usage by Nagios XI might signal a deeper issue.

Conclusion

By following these troubleshooting steps, you should be able to restore missing services in Nagios XI. Regular database maintenance, disk space monitoring, and proper backups can help prevent such issues in the future. Additionally, keeping Nagios XI updated and ensuring proper system resource allocation can minimize potential disruptions. If problems persist, consider checking Nagios XI logs or reaching out to Nagios support for further assistance.

This document is intended for admins who are having issues with the NRPE agent on Linux platforms. It assumes that you have some basic knowledge of how NRPE works, that you have installed and created a base configuration.

NRPE Troubleshooting and Common Solutions