How to fix SYSLOG failed message

If you are receiving the following messages from your cPanel server, an ongoing rsyslog bug has updated the configuration in such a way that it disabled local logging:

  lfd on host.domain.tld: SYSLOG Check Failed
  Time: Thu Jun 13 02:57:23 2019 -0400
  Error: Failed to detect code [YHt8VYSQnJ6KBc6ELafzIsUeNw] in SYSLOG_LOG [/var/log/messages]
  SYSLOG may not be running correctly on host.domain.tld

First, you should go ahead and check the current log and the rotated logs to see if maybe the log rotated at the same time the check was being done:

  grep $code /var/log/messages
  zgrep $code /var/log/messages-*

The following example demonstrates that the code has not been logged at all:

  [root@host ~]# grep YHt8VYSQnJ6KBc6ELafzIsUeNw /var/log/messages
  [root@host ~]# grep YHt8VYSQnJ6KBc6ELafzIsUeNw /var/log/messages-20190
  messages-20190519  messages-20190526  messages-20190602  messages-20190609  
  [root@host ~]# grep YHt8VYSQnJ6KBc6ELafzIsUeNw /var/log/messages-20190*
  [root@host ~]# 

Had the code been logged, you would want to check the iowait and load on the server to determine if the server is just running too slowly to have completed the check within 5 minutes time.

Next, we should check to see that the rsyslog service is running since it isn’t writing to the file:

  [root@host ~]# /scripts/restartsrv_rsyslogd --check 
  The 'rsyslog' service passed the check: 
    rsyslog (/usr/sbin/rsyslogd -n) is running as root with PID 594 (systemd+/proc check method).
  [root@host ~]# ps auxf | grep rsyslog
  root     31446  0.0  0.0   9040   820 pts/0    S+   04:35   0:00          \_ grep --color=auto rsyslog
  root       594  0.0  0.0 356936  2464 ?        Ssl  Jun11   0:05 /usr/sbin/rsyslogd -n
  [root@host ~]# 

It is running according to the checks done. Next, we should check to see if the PID of the running process actually matches the PID in the PID file (it occasionally happens when this is the cause of such failures, such as when there are two instances of the same service running):

  [root@host ~]# cat /var/run/syslogd.pid 
  594[root@host ~]# 

The PID matches. Now, let’s check to see if we can force rsyslog to log a message for us:

  [root@host ~]# logger -p auth.notice "test123"
  [root@host ~]# grep "test123" /var/log/messages
  [root@host ~]#

It failed to write the message to the log for us. You should see this instead:

  [root@host support]# logger -p auth.notice "test123"
  [root@host support]# grep "test123" /var/log/messages
  Jun 13 04:38:23 host root: test123

This does appear to be the bug that causes local logging to be disabled/misconfigured. Let’s check the configuration to confirm:

  [root@host ~]# grep -i "OmitLocalLogging\|ModLoad imjournal\|IMJournalStateFile" /etc/rsyslog.conf
  $ModLoad imjournal # provides access to the systemd journal
  $OmitLocalLogging on
  $IMJournalStateFile imjournal.state
  [root@host ~]# 

The output confirms this. Now, we will make a backup of the configuration file, correct the configuration files, and then restart the service to correct this issue:

  [root@host ~]# cp /etc/rsyslog.conf /etc/rsyslog.conf.bak 
  [root@host ~]# sed -i 's/OmitLocalLogging on/OmitLocalLogging off/' /etc/rsyslog.conf
  [root@host ~]# sed -i 's/$IMJournalStateFile imjournal.state/#$IMJournalStateFile imjournal.state/' 
    /etc/rsyslog.conf
  [root@host ~]# sed -i 's/$ModLoad imjournal/#$ModLoad imjournal/' /etc/rsyslog.conf
  [root@host ~]# /scripts/restartsrv_rsyslog
  Waiting for “rsyslog” to restart ………waiting for “rsyslog” to initialize ………finished.
  Service Status
	rsyslog (/usr/sbin/rsyslogd -n) is running as root with PID 31621 (systemd+/proc check method).
  Startup Log
	Jun 13 08:43:40 host systemd[1]: Starting System Logging Service...
	Jun 13 08:43:40 host rsyslogd[31621]:  
          [origin software="rsyslogd" swVersion="8.24.0-34.el7" x-pid="31621" x- 
 info="http://www.rsyslog.com"] start
	Jun 13 08:43:40 host systemd[1]: Started System Logging Service.
  rsyslog restarted successfully.
  [root@host ~]# systemctl restart systemd-journald
  [root@host ~]# 

Now, retest. You should now see your test logging message:

  [root@host ~]# logger -p auth.notice "testing123" && grep testing /var/log/messages
  Jun 13 04:44:08 host root: testing123
  [root@host ~]# 

You can see now that the code was indeed detected correctly.