rsyslog inside docker containers => "rsyslogd is not running ... failed" - docker

Question

I am running rsyslog within docker containers to send UDP messages to logstash.
When I log into the docker container, and type:
service rsyslog status
shows:
rsyslogd is not running ... failed!
However, while I am in the container, if I type:
service rsyslog start
It starts up perfectly with no errors and no real sign of why it failed at the start
I CAN NOT FIGURE OUT WHY IT IS FAILING!!!!
*The rsyslog conf file has not been modified except the Modules to allow for imfile. The rsyslog.conf is as follows:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#################
#### MODULES ####
#################
module(load="imfile" PollingInterval="10")
module(load="imuxsock" ) # provides support for local system logging
module(load="immark") #provides --MARK-- message capability
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
*I have a script file that starts rsyslog
if [[ -z "$(pgrep rsyslog)" ]]; then
echo "starting rsyslog"
service rsyslog start
fi
My conf file is as follows:
##Get Nginx Error Logs
$InputFileName /var/log/nginx/error.log
$InputFileTag http-error
$InputFileStateFile stat-nginx-error
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor
#GRAB PHP-FPM ACCESS LOGS
$InputFileName /var/log/php-fpm/access_log
$InputFileTag php-fpm-access
$InputFileStateFile stat-php-fpm-access
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
#GRAB PHP-FPM ERROR LOGS
$InputFileName /var/log/php-fpm/error_log
$InputFileTag php-fpm-error
$InputFileStateFile stat-php-fpm-error
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor
#Json Template
template(name="json_temp" type="list")
{ constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timegenerated" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"severity_label\":\"") property(name="syslogseverity-text")
constant(value="\",\"severity\":\"") property(name="syslogseverity")
constant(value="\",\"facility_label\":\"") property(name="syslogfacility-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"pid\":\"") property(name="procid")
constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}\n")
}
if $programname == 'http-error' then #ip.address:port;json_temp
if $programname == 'http-error' then stop
if $programname == 'php-fpm-access' then #ip.address:port;json_temp
if $programname == 'php-fpm-access' then stop
if $programname == 'php-fpm-error' then #ip.address:port;json_temp
if $programname == 'php-fpm-error' then stop
*.* #ip.address:port;json_temp
Any help would be awesome because I do not understand why it is not starting up.
Cheers

Answer 1

We bumped upon the same issue on a Docker 17.03.2-ce image created on CentOS 7.3.1611. The solution is in verifying /etc/rsyslog.conf as per this documentation. Basically, in /etc/rsyslog.conf:
Remove $ModLoad imjournal
Set $OmitLocalLogging to off
Make sure $ModLoad imuxsock is present
Comment out: $IMJournalStateFile imjournal.state
Finally, note that running the rsyslogd or anything else, is the responsibility of the program that is being run inside the container. It is not going to get launched, automatically.

Answer 2

This is my rsyslog.conf in docker container(centos7):
$> cat /etc/rsyslog.conf |grep -vE '^$|^#'
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging off
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
Also i changed /etc/rsyslog.d/listen.conf
#$SystemLogSocketName /run/systemd/journal/syslog
And then
$> rsyslogd -n
Thanks #elinax
More info at https://www.projectatomic.io/blog/2014/09/running-syslog-within-a-docker-container/

Answer 3

Add this line in file entrypoint.sh:
sudo service rsyslog start