We have some windows servers, linux server and some switches, each group sends his independent syslog format to the fluentd syslog server.
which is better:
Each group send his logs to a certain fluentd tcp port, so that each group will have his logs parsed inside an individual source tag
`
<source> # for linux
type syslog
**port 12000**
<parse>
<grok>
</grok>
</parse>
</source>
<source> # for windows
type syslog
**port 12001**
<parse>
<grok>
</grok>
</parse>
</source>
<source> # for some switches
type syslog
**port 12002**
<parse>
<grok>
</grok>
</parse>
</source>
`
or
All groups send their logs to the flentd syslog server and the parsing will be done inside one source tag
`
<source>
#type tail
path /var/log/syslog
tag test
<parse>
<grok>
pattern
</grok>
<grok>
pattern
</grok>
<grok>
pattern
</grok>
<parse>
</source>
`
we are using EFK for log monitoring, Please feel free to recommend a better solution if available
thanks
Related
the config on the host server, I need some way with the two servers to put the logs in /tmp/task/<hostname/<file_name> , for example /tmp/task/app1/auth.log or /tmp/task/app2/auth.log
on servers app1 and app2 all messages are marked with the tag .var.log.*, where * is the file name, and - hostname of the source of logs
<source>
#type forward
</source>
<match *.localfile>
#type copy
<store>
#type file
path /tmp/task/*
<buffer>
timekey 1m
</buffer>
</store>
</match>
I need to send my application logs into a FluentD which is part of an EFK service. so I tried to config another FluentD to do that.
my-fluent.conf:
<source>
#type kafka_group
consumer_group cgrp
brokers "#{ENV['KAFKA_BROKERS']}"
scram_mechanism sha512
username "#{ENV['KAFKA_USERNAME']}"
password "#{ENV['KAFKA_PASSWORD']}"
ssl_ca_certs_from_system true
topics "#{ENV['KAFKA_TOPICS']}"
format json
</source>
<filter TOPIC>
#type parser
key_name log
reserve_data false
<parse>
#type json
</parse>
</filter>
<match TOPIC>
#type copy
<store>
#type stdout
</store>
<store>
#type forward
<server>
host "#{ENV['FLUENTD_HOST']}"
port "#{ENV['FLUENTD_PORT']}"
shared_key "#{ENV['FLUENTD_SHARED_KEY']}"
</server>
</store>
</match>
I am able to see the output of stdout correctly
2021-07-06 07:36:54.376459650 +0000 TOPIC: {"foo":"bar", ...}
But I'm unable to see the logs from kibana. after tracing I figured it out that the second fluentd is throwing error when receiving data:
{"time":"2021-07-05 11:21:41 +0000","level":"error","message":"unexpected error on reading data host="X.X.X.X" port=58548 error_class=MessagePack::MalformedFormatError error="invalid byte"","worker_id":0}
{"time":"2021-07-05 11:21:41 +0000","level":"error","worker_id":0,"message":"/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_forward.rb:262:in feed_each'\n/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_forward.rb:262:in block (2 levels) in read_messages'\n/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_forward.rb:271:in block in read_messages'\n/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin_helper/server.rb:613:in on_read_without_connection'\n/usr/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:123:in on_readable'\n/usr/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in on_readable'\n/usr/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run_once'\n/usr/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run'\n/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin_helper/event_loop.rb:93:in block in start'\n/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'"}
The problem was missing security tag in first fluentd.
<match TOPIC>
#type copy
<store>
#type stdout
</store>
<store>
#type forward
<server>
host "#{ENV['FLUENTD_HOST']}"
port "#{ENV['FLUENTD_PORT']}"
shared_key "#{ENV['FLUENTD_SHARED_KEY']}"
</server>
<security>
self_hostname HOSTNAME
shared_key "#{ENV['FLUENTD_SHARED_KEY']}"
</security>
</store>
</match>
I have Flask app which is streaming some logs in stdout on localhost:5555.
I want to listen these logs by dockerized Fluentd, but I'm a bit confused which plugin I should use: in_tcp or in_forward?
Config like this results in error: "Address not available - bind(2) for \"my_ip\" port 5555"
<source>
#type tcp
tag "tcp.events"
format none
bind my_ip
port 5555
log-level debug
</source>
<filter **>
#type stdout
</filter>
Config examples for in_forward always have port 24224 in config, so they seem to listen the other fluentds, not to listen an application.
Could you please advice?
For the ones which will follow:
Use fluent-logger-language to export your logs to Fluentd server.
Here are all the links:
https://github.com/fluent
Fluentd server config
<source>
#type forward
port 24224
host <if remote>
</source>
<filter **>
#type stdout
</filter>
I write
<source>
#type tail
format nginx
path /home/work/opt/nginx/var/log/access.log
tag nginx.access
</source>
in my fluent.conf file and match to stdout,but when i make a post call to my website,nginx log access into its log,but fluentd_ui suggest
[warn]: no patterns matched tag="nginx.access"
how to add tag to my http request?
<source>
#type tail
path /var/log/nginx/access.log
pos_file /var/log/access.log.pos
tag apache.access
format nginx
</source>
<match apache.access>
#type stdout
</match>
Try running this conf file code. I think you will be able to solve your problem!!
I have fluentd + InfluxDB + Graphite + Grafana.
I need to apply math operations with number data, but InfluxDB or Grafana regard my numeric data like a string. So I can't compare with WHERE statements or color with grafana.
How I can set data type?
My configuration is like this:
<source>
#type http
port 12102
format tsv
keys string1,string2,number1,number2
delimiter |
</source>
<match test>
#type copy
<store>
#type graphite
tag_for prefix
name_keys number1,number2
host localhost
port 2003
</store>
<store>
#type influxdb
dbname test
flush_interval 10s
host localhost
port 8086
</store>
</match>
And the input is like this:
curl -X POST -d "text1|text2|764.2|57" "http://localhost:12102/test?time=1461940658"
On graphite it's all OK.