Parsing linux audit logs with fluent.d - fluentd

Question

I'm attempting send to Linux audit logs to an elastic endpoint. I've installed it via the RPM package. For context I am using CentOS Linux release 8.3.2011. My Linux audit logs are under: /var/log/audit/audit.log. I've checked and double check that the audit logs exist.
The logs never indicate that I'm ever tailing the logs. Here's my configuration:
<source>
#type tail
tag linux_logs.raw
path /var/log/audit/audit.log
read_from_head true
pos_file /etc/td-agent/test.pos
<parse>
#type regexp
expression /(?<message>.+)/
time_format %Y-%m-%d %H:%M:%S
utc true
</parse>
</source>
####
## Filter descriptions:
##
<filter **>
#type record_transformer
<record>
hostname "${hostname}"
timestamp "${time}"
</record>
</filter>
####
## Output descriptions:
##
<match **>
#type http
endpoint https://myendpoint/
open_timeout 2
headers {"Authorization":"Bearer <token> <token2>"}
<format>
#type json
</format>
<buffer>
#type memory
flush_interval 10s
compress gzip
</buffer>
</match>
The logs never indicate I'm ever tailing the audit.log file.
2021-06-14 14:42:59 -0400 [info]: starting fluentd-1.12.3 pid=10725 ruby="2.7.3"
2021-06-14 14:42:59 -0400 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2021-06-14 14:43:00 -0400 [info]: adding filter pattern="**" type="record_transformer"
2021-06-14 14:43:00 -0400 [info]: adding match pattern="**" type="http"
2021-06-14 14:43:00 -0400 [warn]: #0 Status code 503 is going to be removed from default `retryable_response_codes` from fluentd v2. Please add it by yourself if you wish
2021-06-14 14:43:00 -0400 [info]: adding source type="tail"
2021-06-14 14:43:00 -0400 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label #FLUENT_LOG> instead
2021-06-14 14:43:00 -0400 [info]: #0 starting fluentd worker pid=10734 ppid=10731 worker=0
2021-06-14 14:43:00 -0400 [info]: #0 fluentd worker is now running worker=0
Is this a permissions issue?? The tailing works if I do a tmp file so it seems to be a permissions issue. Any ideas?

Answer 1

Yes it is a permission issue. Fluentd is installed by RPM, so the daemon run with "td-agent" user and "td-agent" group.
You need to check the "/var/log/audit/audit.log" file permissions and, in case you have:
-rw-------
I suggest you to run Fluentd as root. To do this, you need to change the "/lib/systemd/system/td-agent.service" file from:
[Service]
User=td-agent
Group=td-agent
to
[Service]
User=root
Group=root
Finally, do a daemon-reload and a service (Fluentd) restart