I would like to start to monitor our dockers with Splunk. To do it I want to add it to the docker buildfile.
The issue is that if its the first time you start the forwarder, it prompts you to agree for the license agreement, which breaks the script.
Is there a way to start the Splunk forwarder without being prompted?
The arguments you are looking for are --accept-license --answer-yes --no-prompt.
A Splunk employee publishes some Docker images on his own time that can cater for this already. Have a look at SPLUNK_START_ARGS in the forwarders entrypoint.sh.
Also, Docker natively supports sending its own logs to Splunk. This includes the stdout and stderr of all containers so you may not need your own forwarder.
Related
I am working with logs in my system.
I want to use a log sidecar to collect business container's log.
And my business container's log will write to its STDOUT.
So I want to redirect this STDOUT to pod's volume file, because in a pod all containers share the same volume, so my sidecar can collect log from volume.
How should I configuer this?
I mean maybe I should write some configuration in my k8s yaml so k8s will automaticlly redirect the container's STDOUT to pod's volume?
Adding this 2>&1 > /<your_path_to_volume_inside_pod>/file.log to your command would redirect STDOUT and STDERR to a file
You could use a sidecar container with a logging agent
Streaming sidecar container
By having your sidecar containers write to their own stdout and stderr streams, you can take advantage of the kubelet and the logging agent that already run on each node.
The sidecar containers read logs from a file, a socket, or journald. Each sidecar container prints a log to its own stdout or stderr stream.
This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr.
The logic behind redirecting logs is minimal, so it's not a significant overhead.
Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.
In your case, it depends on how your application pod can be configured (for intance, with the journald service active, in order to record logs)
And the backend would be your common volume file.
Docker logging drivers are specified online, and these limitations.
Limitations of logging drivers
Users of Docker Enterprise can make use of “dual logging”, which enables you to use the docker logs command for any logging driver. Refer to reading logs when using remote logging drivers for information about using docker logs to read container logs locally for many third party logging solutions, including:
syslog
gelf
fluentd
awslogs
splunk
etwlogs
gcplogs
Logentries
When using Docker Community Engine, the docker logs command is only available on the following drivers:
local
json-file
journald
Reading log information requires decompressing rotated log files, which causes a temporary increase in disk usage (until the log entries from the rotated files are read) and an increased CPU usage while decompressing.
The capacity of the host storage where the Docker data directory resides determines the maximum size of the log file information.
I am using Docker CE, but I have a question about this documentation. Does this mean, using CE, I cant do syslog at all? or just that I cant do syslog and have docker logs?
There is nothing stopping you from using syslog within the container, but you can't read those logs using the 'docker logs' command. There is also nothing stopping you from writing your logs to stdout and piping your logs to as many log shippers as you want.
Here's an article that explains how to do syslog in a docker container: https://medium.com/better-programming/docker-centralized-logging-with-syslog-97b9c147bd30
I think that fluentd and fluent-bit are better choices than syslog these days given the structure they provide to the msg field, though syslog-ng looks interesting. Fluent-bit is incredibly good though, so you might want to take a look at it.
I am trying to understand the interaction between Docker and Fluentd in a K8s cluster. I have seen places where you need to configure Docker to output to a logging driver, and Fluentd can be used as logging driver, like here.
On the other hand, I have seen posts (like this or this) where Docker does not know the existence of Fluentd as a DaemonSet.
My whole intention is to do log rotation, however I am not sure if having Fluentd in place will actually rotate the logs Docker writes on, so I do not end up with the whole storage space in the node taken up by the logs over time. Is it enough to use FluentD DaemonSet without Docker knowing the existence of Fluentd?, o I need to somehow connect Docker to Fluentd with a driver as well?
Per official k8s logging architecture docker (or any other runtime) does not need to know about FluentBit, Fluentd, Filebeat, or any other log collector you use. In fact, you can use multiple log collectors a time!
The same document states that k8s is not responsible for log rotation, so you set up a logrotate yourself. Fluentd/FluentBit daemon on the other end also does not rotate log files, but it does able to track log rotation and adjust the tail cursor accordingly (by default).
By far the easiest way to implement the architecture is
Leave kubelet & docker settings at default
Ensure the app logs stdout/stderr
Ensure there's logrotate: many k8s worker AMIs, e.g. EKS already have it.
Setup FluentBit log collector daemonset https://github.com/helm/charts/tree/master/stable/fluent-bit
We're interested in forwarding the logs from a node.js server running in a Docker container to Splunk.
Some options we've considered include a side-car container running a Splunk forwarder. The side-car would write to a shared volume that the side-car would observe and send on.
Ideally, we would just use a syslog drain or another mechanism, but I can't seem to find any documentation on how to set that up?
There are a lot of options to send logs from containers to Splunk.
For logs, sent to Standard Output and Error:
Splunk Logging Driver https://docs.docker.com/v17.09/engine/admin/logging/splunk/
Splunk Docker logging plugin https://github.com/splunk/docker-logging-plugin - an improved version of Splunk Logging Driver
For application logs (logs written inside of the container):
Sidecars with UF
Our company (https://www.outcoldsolutions.com) offers one solution that can simply forward container (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/) and application logs (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#application-logs) from the Docker hosts, and collect metrics. We also provide you with an application in Splunk for tracking the health and performance of your clusters https://splunkbase.splunk.com/app/3723/. Our application is not free, but cheap compared to the time you can spend building something similar.
Another option is using fluentd as an intermediary.
Fluentd exists as docker logging driver as well, but you can use it to redirect the logs to several backends (Splunk, Elasticsearch). You are not as tightly coupled to Splunk.
Additionally that's the way proposed by Openshift.
It looks like Docker has a logging driver that handles this
https://docs.docker.com/v17.09/engine/admin/logging/splunk/
I am using gelf as log driver for my docker container. In log options i provided udp endpoint.
Now when i start the container, everything is working as expected.
My question is, if it is possible to see the container logs in the host where it is running(not at UDP endpoint)?
This depends on Docker version.
Docker 20.10 and up introduces “dual logging”, which uses a local buffer that allows you to use the docker logs command for any logging driver.
If you are talking about seeing the logs via docker logs command on the machine running the docker containers, its not possible to do so when using other logging drivers.
See limitations of logging drivers.
If you know where the log is at inside the container, a work around would be to write a script which copies the log file from the container and displays it, or maybe just exec's into the container and displays it. But I really wouldn't recommend that.
Something like:
#!/bin/bash
docker cp mycontainer:/var/log/mylog.log $(pwd)/logs/mylog.log
tail -f $(pwd)/logs/mylog.log