Rsyslog collect logs from different timezones - timezone

Question

Im using rsyslog on server to collect logs from remote hosts.
Collect server config:
# timedatectl
Local time: Wed 2022-04-27 16:02:43 MSK
Universal time: Wed 2022-04-27 13:02:43 UTC
RTC time: n/a
Time zone: Europe/Moscow (MSK, +0300)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
# cat /etc/rsyslog.d/20_external.conf
$CreateDirs on
$PreserveFQDN on
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
template(
name="external"
type="string"
string="/var/log/external/%HOSTNAME%/%syslogfacility-text%.%programname%.%syslogseverity-text%.log"
)
action(
type="omfile"
dirCreateMode="0775"
FileCreateMode="0644"
dynaFile="external"
)
On remote host
# timedatectl
Local time: Wed 2022-04-27 13:04:03 UTC
Universal time: Wed 2022-04-27 13:04:03 UTC
RTC time: n/a
Time zone: UTC (UTC, +0000)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
# cat /etc/rsyslog.d/10-external.conf
*.* #rserver
# logger "hello, local time $(date)"
And get on rsyslogserver:
cat /var/log/external/ruser.home.xmu/user.root.notice.log
2022-04-27T13:07:06+03:00 ruser.home.xmu root: hello, local time 2022-04-27T13:07:06 UTC
# date
2022-04-27T16:08:56 MSK
What i can do for change time zone settings for some remote hosts on collect-server?
When i reserch incedents from all servers the time does not match in logs. I want the time on the collector in the logs to be in his time zone.
2022-04-27T16:07:06+03:00 ruser.home.xmu root: hello, local time 2022-04-27T13:07:06 UTC

Answer 1

You can define the timezone in rsyslog on the client - which in my opinion is the cleaner solution.
In /etc/rsyslog.conf do the following:
Comment/remove the current template
# Use default timestamp format
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
Then add the timezone, as well as a custom log template:
timezone(id="MSK" offset="+03:00")
# Custom time format
$template myTemplate,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"
$ActionFileDefaultTemplate myTemplate
However, if you can't access the remote client which is sending the logs, it's possible to use the timestamp when the log was received on the server.
$template myTemplate,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"
ruleset(name="myRuleset"){
$ActionFileDefaultTemplate myTemplate
# Do some other stuff
}
module(load="imtcp")
input(type="imtcp" port="5000" ruleset="myRuleset")
module(load="imudp")
input(type="imudp" port="5000" ruleset="myRuleset")
NOTE: Don't forget to restart the rsyslog service after applying the changes.
sudo service rsyslog restart
EDIT:
Creating a template using the advanced syntax would look like the following:
template (name="myTemplate" type="string"
string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\n")
The string is the actual template of the messages that should be logged, not the destination to which the messages should be logged.

Answer 2

I ran into a similar problem recently. I think the simplest way to synchronize data coming from multiple sources is to use UTC timestamps everywhere, this way you avoid any timezone and summer time/winter time problems.
I think the problem here (and the problem I had) is that the syslog ouput uses the rfc3164 format which does not include the timezone information:
<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Hello!
There's no way of knowing at what time that message actually happened without the timezone information. Since rsyslog version 8.18.0 what you can do though is define custom rsyslog templates that print the timestamps in UTC. If you use UTC everywhere you don't need that timestamp information anymore.
I'm sending my logs to fluentd, I use the following configuration in my rsyslog.conf file on my hosts:
template(
name="UTCTraditionalForwardFormat"
type="string"
string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)
*.* action(
type="omfwd"
Target="10.5.0.9"
Port="5145"
Protocol="udp"
Template="UTCTraditionalForwardFormat"
)
The UTCTraditionalForwardFormat here is the same as the default template RSYSLOG_TraditionalForwardFormat (rsyslog templates) but with an added :::date-utc to the TIMESTAMP property.
On your rsyslog that collects logs, looking at the input module's doc there an experimental parameter DefaultTZ which should let you define the source timezone, something like this should work (I haven't tested):
input(
type="imudp"
port="514"
DefaultTZ="+00:00"
)
Assuming this DefaultTZ parameter works, this should work regardless of your hosts timezone.
Little word about fluentd, the fluentd syslog module also accepts the rfc5424 message format which does encode timezone information. I decided not to go this way though because rsyslog default templates does not seem to have a real rfc5424 version (the RSYSLOG_SyslogProtocol23Format template is very close to the rfc5424, but I don't know how close).