filebeat-logstash with clickhouse for docker container logs are not pushing - docker

Question

Below is my configs for FileBeat and Logstash, i just followed this code. but i'm trying to push the docker container logs to ClickHouse.
Running the filebeat with below docker cmd. I think the Clickhouse log table and logstash log is not matching.
docker run --rm -it --name=filebeat --user=root \
--volume="/var/log/:/var/log/" \
--volume="/var/lib/docker/containers/:/var/lib/docker/containers/" \
--volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml" \
docker.elastic.co/beats/filebeat:5.6.3 filebeat -e -strict.perms=false
filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*.log
- /var/lib/docker/containers/*.log
exclude_files: ['.gz$']
output.logstash:
hosts: ['localhost:5044']
logstash.conf
input {
beats {
port => 5044
}
}
filter {
grok {
match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
remove_field => ["#version", "host", "message", "beat", "offset", "type", "tags", "input_type", "source"]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp", "#timestamp" ]
target => [ "logdatetime" ]
}
ruby {
code => "tstamp = event.get('logdatetime').to_i
event.set('logdatetime', Time.at(tstamp).strftime('%Y-%m-%d %H:%M:%S'))
event.set('logdate', Time.at(tstamp).strftime('%Y-%m-%d'))"
}
useragent {
source => "agent"
}
prune {
interpolate => true
whitelist_names => ["^logdate$" ,"^logdatetime$" ,"^request$" ,"^agent$" ,"^os$" ,"^minor$" ,"^auth$" ,"^ident$" ,"^verb$" ,"^patch$" ,"^referrer$" ,"^major$" ,"^build$" ,"^response$","^bytes$","^clientip$" ,"^name$" ,"^os_name$" ,"^httpversion$" ,"^device$" ]
}
}
output {
clickhouse {
http_hosts => ["http://localhost:8123"]
table => "nginx_access"
request_tolerance => 1
flush_size => 1000
pool_max => 1000
}
}
filebeat logs
2023/02/09 09:45:22.247315 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2023/02/09 09:45:22.247331 prospector.go:124: INFO Starting prospector of type: log; id: 7589403446011535719
2023/02/09 09:45:22.247356 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2023/02/09 09:45:22.247929 log.go:91: INFO Harvester started for file: /var/log/falcond.log
2023/02/09 09:45:22.247982 log.go:91: INFO Harvester started for file: /var/log/yum.log
2023/02/09 09:45:22.248088 log.go:91: INFO Harvester started for file: /var/log/falconctl.log
2023/02/09 09:45:22.248622 log.go:91: INFO Harvester started for file: /var/log/boot.log
2023/02/09 09:45:22.248665 log.go:91: INFO Harvester started for file: /var/log/falcon-sensor.log
2023/02/09 09:45:52.239459 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=5 filebeat.harvester.running=5 filebeat.harvester.started=5 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=24 libbeat.logstash.publish.write_bytes=3764 libbeat.logstash.published_and_acked_events=77 libbeat.publisher.published_events=77 publish.events=82 registrar.states.current=5 registrar.states.update=82 registrar.writes=2
2023/02/09 09:46:22.239403 metrics.go:34: INFO No non-zero metrics in the last 30s
2023/02/09 09:46:52.239518 metrics.go:34: INFO No non-zero metrics in the last 30s
2023/02/09 09:47:22.239436 metrics.go:34: INFO No non-zero metrics in the last 30s

Answer 1

I'm interested why you have chosen Logstash and not something Fluentbit / Vector for log tailing...both of those have native ClickHouse integrations.
https://clickhouse.com/blog/storing-log-data-in-clickhouse-fluent-bit-vector-open-telemetry