In a Docker environment my Java-App logs on STDOUT via log4j, the messages will be sent to a Graylog instance. There is no special logging config besides configuring the Console-Appender to use JsonLayout.
My docker-compose.yml snippet:
logging:
driver: gelf
options:
gelf-address: "tcp://[GRAYLOG_HOST]:[PORT]"
tag: "[...]"
Everything works fine there. But we are thinking about changing this environment to K8s.
There will be a Graylog instance in K8s, too. It looks like that there is no K8s equivalent for the docker-compose.yml logging settings. It seems that I have to use some kind of logging agent, e.g. fluent-bit. But the documentation of fluent-bit looks like that it only can collect logs from a log file as input (and some more), but not from STDOUT.
I have the following questions:
Is there another possibility to read the logs directly from STDOUT and send them into Graylog?
If I have to log the log messages into a log file to be read from fluent-bit: Do I have to configure log4j to do some roll-over strategies to prevent, that the log file will be bigger and bigger? I do not want to "waste" my resources "just" for logging.
How do you handle application logs in K8s?
Maybe I misunderstand the logging principles in K8s. Feel free to explain it to me.
Is there another possibility to read the logs directly from STDOUT and send them into Graylog?
Fluent Bit allows for data collection through STDIN. Redirect your application STDOUT to Fluent Bit's STDIN and you are set.
If I have to log the log messages into a log file to be read from fluent-bit: Do I have to configure log4j to do some roll-over strategies to prevent, that the log file will be bigger and bigger? I do not want to "waste" my resources "just" for logging.
In this case you can use logrotate
How do you handle application logs in K8s?
Three possible ways:
Application directly output their traces in external systems (eg. databases).
Sidecar container with embedded logging agent that collect application traces and send them to a store (again database for example).
Cluster-wide centralized logging (eg. ELK stack)
I'd recommend you to use sidecar container for log collection. This is probably most widely used solution.
Related
We have application deployed in K8S pod and all logs are being monitored in ELK stack. Now we have one application which is using external *.jar which is writing logs in one file local to container path. How I can send this logs to kubernetes console so that it will come to elastic search monitoring.
Any help is much appreciated!.
Now we have one application which is using external *.jar which is writing logs in one file local to container path. How I can send this logs to kubernetes console so that it will come to elastic search monitoring.
There are three ways, in increasing order of complexity:
Cheat and symlink the path it tries to log to as /dev/stdout (or /proc/1/fd/0); sometimes it works and it's super cheap, but if the logging system tries to seek to the end of the file, or rotate it, or catches on that it's not actually a "file", then you'll have to try other tricks
If the app uses a "normal" logging framework, such as log4j, slf4j, logback, etc, you have a better-than-average chance of being able to influence the app's logging behavior via some well placed configuration files or in some cases environment variables
Actually, you know, ask your developers to configure their application according to the 12 Factor App principles and log to stdout (and stderr!) like a sane app
Without more specifics we can't offer more specific advice, but that's the gist of it
We re trying to eliminate Datadog agents from our infrastructure. I am trying to find a solution to forward the containers standard output logs to be visualised on datadog but without the agents and without changing the dockerfiles because there are hundreds of them.
I was thinking about trying to centralize the logs with rsyslog but I dont know if its a good idea. Any suggestions ?
This doc will show you a comprehensive list of all integrations that involve log collection. Some of these include other common log shippers, which can also be used to forward logs to Datadog. Among these you'd find...
Fluentd
Logstash
Rsyslog (for linux)
Syslog-ng (for linux, windows)
nxlog (for windows)
That said, you can still just use the Datadog agent to collect logs only (they want you to collect everything with their agent, that's why they warn you against collecting just their logs).
If you want to collect logs from docker containers, the Datadog agent is an easy way to do that, and it has the benefit of adding lots of relevant docker-metadata as tags to your logs. (Docker log collection instructions here.)
If you don't want to do that, I'd look at Fluentd first on the list above -- it has a good reputation for containerized log collection, promotes JSON log formatting (for easier processing), and scales reasonably well.
Question:
Is it possible to use stdout/stderr as fluentd source?
If not, are there some sort of workaround to implement this?
Background:
I have to containerize a NodeJS web server that uses json-log as a logging resource.
Since containers are ephemeral, I want to extract it's logs for debugging purposes.
To do this, I've decided to use EFK stack.
However, since...
The philosophy of json-log is...
Write to stdout/err
I can only get the logs of the web server from stdout.
After going through the fluentd documentation, I didn't find a way to use stdout/stderr as a source.
Related question:
Is it possible to use stdout as a fluentd source to capture specific logs for write to elasticsearch?
The question has an answer but it is inapplicable in my case.
See https://www.npmjs.com/package/json-log#write-to-stdouterr
You can send logs from json-log to syslog.
So you can use fluent-plugin-syslog to receive logs from json-log, and send them to Fluentd.
is it possible to use multiple logging drivers for the same container - say fluentd and json?
Thank you.
As of 18.03, Docker Engine Enterprise(EE) supports multiple log drivers, but it is not in the Community Edition(CE):
https://docs.docker.com/ee/engine/release-notes/#18031-ee-1-2018-06-27
No, you can only specify a single logging driver/container.
To have separate sinks for your logs, you'd have to rely on something like fluentd to receive the logs (or read the json log files) and configure a pipeline to distribute them.
Dual logging is available in docker CE since version 20.10.1.
The feature was previously only available in Docker Enterprise since version 18.03.1-ee-1.
The official documentation chapter "Dual Logging" doesn't reflect this (as of 2021-01-04 ).
The feature has been open-sourced in pull request #40543 and was merged into master on 2020-02-27.
The related GitHub issue #17910 in moby/moby was closed with the following comment:
The upcoming Docker 20.10 release will come with the feature described above ("dual logging"), which uses the local logging driver as a ring-buffer, which makes docker logs work when using a logging driver that does not have "read" support (for example, logging drivers that send logs to a remote logging aggregator).
No you can specify a single logging driver as stated in the official documentation :
You cannot specify more than one log driver.
The log-driver documentation indicates that too :
To configure the Docker daemon to default to a specific logging
driver, set the value of log-driver to the name of the logging driver
in the daemon.json file...`
{
"log-driver": "syslog"
}
You can see that "log-driver" expects a string and not an array.
In fact, since Docker Engine Enterprise 18.03.1-ee-1, Docker has "just" enable a dual logging feature that allows to configure any logging driver log while being still the possibility to read them with docker logs.
For example before that feature, specifying that driver in the daemon.json :
{
"log-driver": "syslog"
}
allowed to redirect the logs to a syslog server but that also made Docker to not publish any longer logs to the local logging driver.
Now that is not the case, the information is available in both destination.
Starting with Docker Engine Enterprise 18.03.1-ee-1, you can use
docker logs to read container logs regardless of the configured
logging driver or plugin. This capability, sometimes referred to as
dual logging, allows you to use docker logs to read container logs
locally in a consistent format, regardless of the remote log driver
used, because the engine is configured to log information to the
“local” logging driver.
I noticed that the fluentd engine uses the out_forward output to send logs. Meaning all logs are sent in the clear. Is there a way to specify the output type? I'd like to be able to have Docker send logs with out_secure_forward instead.
Are there plans to enable more configuration? Should I use a different logging driver if I want security? Perhaps use the JSON file engine and then use fluentd to ship those securely?
IMO the best option to do what you want is:
introduce an additional docker container (A) to run Fluentd in it
configure your docker containers to send logs (over fluentd log drivers) to that container (A)
send these logs to another site from the fluentd in container (A) by using secure-forward