Question:
Is it possible to use stdout/stderr as fluentd source?
If not, are there some sort of workaround to implement this?
Background:
I have to containerize a NodeJS web server that uses json-log as a logging resource.
Since containers are ephemeral, I want to extract it's logs for debugging purposes.
To do this, I've decided to use EFK stack.
However, since...
The philosophy of json-log is...
Write to stdout/err
I can only get the logs of the web server from stdout.
After going through the fluentd documentation, I didn't find a way to use stdout/stderr as a source.
Related question:
Is it possible to use stdout as a fluentd source to capture specific logs for write to elasticsearch?
The question has an answer but it is inapplicable in my case.
See https://www.npmjs.com/package/json-log#write-to-stdouterr
You can send logs from json-log to syslog.
So you can use fluent-plugin-syslog to receive logs from json-log, and send them to Fluentd.
Related
I'm having issues connecting Fluentd to Kafka for a centralized logging PoC I'm working on.
I'm currently using the following configuration:
Minikube
Fluentd
fluent/fluentd-kubernetes-daemonset:v1.14.3-debian-kafka2-1.0 (docker)
Configuration: I have the FLUENT_KAFKA2_BROKERS=<INTERNAL KAFKA BOOTSTRAP IP>:9092 and FLUENT_KAFKA2_DEFAULT_TOPIC=logs env set in my yaml for fluentd daemonset.
Kafka
I was sort of expecting to see the logs appear in a Kafka consumer running against the same broker listening on the "logs" topic. No dice.
Could anyone recommend next steps for troubleshooting and or a good reference? I've done a good bit of searching and have only found a few people posting about setting up with the fluentd-kafka plugin. Also would it make sense for me to explore Fluent Bit Kafka setup as an alternative?
In general, to configure forwarding of log events to Kafka topic you would definitely need to use output plugins for Fluentd.
Fluentd delivers fluent-plugin-kafka plugin, as specified in Fluentd docs, for both input and output use cases. For output case, this plugin has Kafka Producer functions to publishes messages into topics. kafka-connect-fluentd plugin can also be used as an alternative.
Fluent Bit - being the sub-project of Fluentd - a good lightweight alternative for Fluentd, but which one to use depends on your particular use case.
Fluent Bit has limited amount of filtering options, it is not as pluggable and flexible as Fluentd. The later has more configuration options and filters, it can be integrated with a much larger amount of input and output sources. It is essentially designed to deal with heavy throughput — aggregating from multiple inputs, processing data and routing to different outputs. More on comparison here and here.
In a Docker environment my Java-App logs on STDOUT via log4j, the messages will be sent to a Graylog instance. There is no special logging config besides configuring the Console-Appender to use JsonLayout.
My docker-compose.yml snippet:
logging:
driver: gelf
options:
gelf-address: "tcp://[GRAYLOG_HOST]:[PORT]"
tag: "[...]"
Everything works fine there. But we are thinking about changing this environment to K8s.
There will be a Graylog instance in K8s, too. It looks like that there is no K8s equivalent for the docker-compose.yml logging settings. It seems that I have to use some kind of logging agent, e.g. fluent-bit. But the documentation of fluent-bit looks like that it only can collect logs from a log file as input (and some more), but not from STDOUT.
I have the following questions:
Is there another possibility to read the logs directly from STDOUT and send them into Graylog?
If I have to log the log messages into a log file to be read from fluent-bit: Do I have to configure log4j to do some roll-over strategies to prevent, that the log file will be bigger and bigger? I do not want to "waste" my resources "just" for logging.
How do you handle application logs in K8s?
Maybe I misunderstand the logging principles in K8s. Feel free to explain it to me.
Is there another possibility to read the logs directly from STDOUT and send them into Graylog?
Fluent Bit allows for data collection through STDIN. Redirect your application STDOUT to Fluent Bit's STDIN and you are set.
If I have to log the log messages into a log file to be read from fluent-bit: Do I have to configure log4j to do some roll-over strategies to prevent, that the log file will be bigger and bigger? I do not want to "waste" my resources "just" for logging.
In this case you can use logrotate
How do you handle application logs in K8s?
Three possible ways:
Application directly output their traces in external systems (eg. databases).
Sidecar container with embedded logging agent that collect application traces and send them to a store (again database for example).
Cluster-wide centralized logging (eg. ELK stack)
I'd recommend you to use sidecar container for log collection. This is probably most widely used solution.
We re trying to eliminate Datadog agents from our infrastructure. I am trying to find a solution to forward the containers standard output logs to be visualised on datadog but without the agents and without changing the dockerfiles because there are hundreds of them.
I was thinking about trying to centralize the logs with rsyslog but I dont know if its a good idea. Any suggestions ?
This doc will show you a comprehensive list of all integrations that involve log collection. Some of these include other common log shippers, which can also be used to forward logs to Datadog. Among these you'd find...
Fluentd
Logstash
Rsyslog (for linux)
Syslog-ng (for linux, windows)
nxlog (for windows)
That said, you can still just use the Datadog agent to collect logs only (they want you to collect everything with their agent, that's why they warn you against collecting just their logs).
If you want to collect logs from docker containers, the Datadog agent is an easy way to do that, and it has the benefit of adding lots of relevant docker-metadata as tags to your logs. (Docker log collection instructions here.)
If you don't want to do that, I'd look at Fluentd first on the list above -- it has a good reputation for containerized log collection, promotes JSON log formatting (for easier processing), and scales reasonably well.
I want to send logs to multiple locations from a docker logging driver, is it possible with any logging driver?
For php you can use Monolog.
Find monolog here,
https://github.com/Seldaek/monolog
Monolog is not a driver, its a php package.
It depends on your setup.Can you elaborate more?
I noticed that the fluentd engine uses the out_forward output to send logs. Meaning all logs are sent in the clear. Is there a way to specify the output type? I'd like to be able to have Docker send logs with out_secure_forward instead.
Are there plans to enable more configuration? Should I use a different logging driver if I want security? Perhaps use the JSON file engine and then use fluentd to ship those securely?
IMO the best option to do what you want is:
introduce an additional docker container (A) to run Fluentd in it
configure your docker containers to send logs (over fluentd log drivers) to that container (A)
send these logs to another site from the fluentd in container (A) by using secure-forward