Fluentd - Kubernetes - ParserError error="pattern not matched with data" - docker

Question

I'm trying to redirect Kubernetes logs from containers to OpenSearch.
But there is always some error with the date. What am I doing wrong?
Docker logs example:
{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230191143Z"}
{"log":"E0401 10:02:31.230146 1 backup_controller.go:153] longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\n","stream":"stderr","time":"2022-04-01T10:02:31.230214608Z"}
{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Dropping Longhorn backup longhorn-system/backup-989764daba094e0d out of the queue\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230218285Z"}
Fluentd Output:
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524389862 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524404952 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.538539106 +0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
Config:
<source>
#type tail
#id tail_all_container_logs
#label #FLUENTD.OPENSEARCH
path /var/log/containers/longhorn*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
exclude_path "#{ENV['FLUENT_ALL_CONTAINERS_TAIL_EXCLUDE_PATHS']}"
<parse>
#type json
</parse>
</source>
<filter kubernetes.**>
#type parser
key_name log
<parse>
#type json
time_format %Y-%m-%dT%H:%M:%S.%N%z
timezone +00:00
</parse>
</filter>

Answer 1

The date/time from your logs is 2022-04-01T09:54:33Z note no milliseconds.
While in your config has time_format %Y-%m-%dT%H:%M:%S.%N%z
%N - Fractional seconds digits, default is 9 digits (nanosecond).
Try to remove the .%N part to match with your time format from logs. Which would be:
time_format %Y-%m-%dT%H:%M:%S%z
For more information about the time format syntax kindly refer to this page.

Answer 2

#piotr-malec - Thanks for the answer.
I've tried many different date/time formats but it's always the same.
<filter kubernetes.**>
#type parser
key_name log
<parse>
#type json
time_format %Y-%m-%dT%H:%M:%S.%N%z
#time_format %Y-%m-%dT%H:%M:%S.%z
#time_format %Y-%m-%dT%H:%M:%S%z
#time_format %Y-%m-%dT%H:%M:%Sz
#time_format %Y-%m-%dT%H:%M:%SZ
#time_format %Y-%m-%dT%H:%M:%S%Z
#time_format %Y-%m-%dT%H:%M:%S.%z
timezone +00:00
</parse>
</filter>
Now I have a similar case with a different log.
fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:46:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep\n'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306044198 +0000 record={"log"=>"[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep\n", "stream"=>"stdout"}
fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:51:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep\n'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306370871 +0000 record={"log"=>"[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep\n", "stream"=>"stdout"}
I have tried the following formats. Each of them does not work.
<filter opensearch-master>
#type parser
key_name log
<parse>
#type json
time_key #timestamp
time_format %Y-%m-%dT%H:%M:%S,%3N
#time_format %Y-%m-%dT%H:%M:%S,%L
timezone +02:00
</parse>
</filter>
Ruby Documentation:
https://docs.ruby-lang.org/en/2.4.0/Time.html#method-i-strftime