When we run the same process in docker and in host system, how it differentiates one from the other, from the perspective of audit logs?
Can I view the process running in docker in host system?
You would not run the same process (same pid) in docker and in host, since the purpose of a container is to provide isolation (both processes and filesystem)
I mentioned in your previous question "Docker Namespace in kernel level" that the pid of a process run in a container could be made visible from the host.
But in term of audit log, you can configure logging drivers in order to follow only containers, and ignore processes running directly on host.
For instance, in this article, Mark configures rsyslog to isolate the Docker logs into their own file.
To do this create /etc/rsyslog.d/10-docker.conf and copy the following content into the file using your favorite text editor.
# Docker logging
daemon.* {
/var/log/docker.log
stop
}
In summary this will write all logs for the daemon category to /var/log/docker.log then stop processing that log entry so it isn’t written to the systems default syslog file.
That should be enough to clearly differentiate the host processes logs (in regular syslog) from the ones running in containers (in /var/log/docker.log)
Update May 2016: issue 10163 and --pid=container:id is closed by PR 22481 for docker 1.12, allowing to join another container's PID namespace.
Related
Currently we have multiple docker containers(host A).
We send the logs from each docker container to logger(which is runs on docker container on another server).
Here is my daemon.json:
{
"log-driver":"gelf",
"log-opts":{
"gelf-address":"tcp://10.*.*.*:12201"
},
"dns":[
"10.*.*.*"
],
"icc":false
}
The problem is that if logger docker is not running and i restarting host A, they not starting because cannot connect to logger.
Is there any way to configure docker containers to start even if they cannot connect to logger configured in daemon.json?
Thank you.
With this you are not configuring docker containers, but the daemon itself. If you restart you host, you restart the daemon and on startup it reads the config. If the config is invalid or parts of it are not working, it doesn't start up. You can manually start up the docker daemon with a manual configuration like
dockerd --debug \
--tls=true \
--tlscert=/var/docker/server.pem \
--tlskey=/var/docker/serverkey.pem \
--host tcp://192.168.59.3:2376
see: Docker daemon documentation
Keep in mind, that it will keep running with those options, until it's restarted.
The logging settings in daemon.json are defaults for newly created containers. Changing this file will not change existing containers being restarted.
You may want to reconsider your logging design. One option is to swap out the logging driver for a logging forwarder, leaving the logs in the default json driver, and having another process monitor those and forward the logs to the remote server. This avoids blocking at the cost of missing some logs written just as the container is deleted (or very short lived containers). The other option is to improve the redundancy of your logging system since it is a single point of failure that blocks your workloads from running.
I have one question, Is there any way to ship the logs of each container where the log files are located inside the containers. Actually, the current flow will help to ship the log files which is located in the default path(var/lib/docker/containers//.log). I want to customize the filebeat.yaml to ship the logs from each container to logstash instead of the default path.
If you can set your containers to log to stdout rather than to files, it looks like filebeat has an autodiscover mode which will capture the docker logs of every container.
Another common setup in an ELK world is to configure logstash on your host, and set up Docker's logging options to send all output on containers' stdout into logstash. This makes docker logs not work, but all of your log output is available via Kibana.
If your container processes always write to log files, you can use the docker run -v option or the Docker Compose volumes: option to mount a host directory on to an individual container's /var/log directory. Then the log files will be visible on the host, and you can use whatever file-based collector to capture them. This is in the realm of routine changes that will require you to stop and delete your existing containers before starting them with different options.
I have a server that is the host OS for multiple docker containers. Each of the containers contains an application that is creating logs. I want these logs to be sent to a single place by using the syslog daemon, and then I want filebeat to transmit this data to another server. Is it possible to install filebeat on the HOST OS (without making another container for filebeat), and make the containers applications' log data be collected by the syslog daemon and then consolidated in /var/log on the host OS? Thanks.
You need to share a volume with every container in order to get your logs in the host filesystem.
Then, you can install filebeat on the host and forward the logs where you want, as they were "standard" log files.
Please be aware that usually docker containers do not write they logs to real log files, but to stdout. That means that you'll probably need custom images in order to fix this logging problem.
How docker namespace is different from Host namespace and how the pid can be mapped between these two? Can anyone give me an idea that helps to make easy way of mapping pid's between host n docker using source code?
You can find the mapping in /proc/PID/status file. It contains a line like:
NSpid: 16950 24
Which means that 16950 on the host is 24 inside the container.
As I mentioned in "Running docker securely":
Currently, Docker uses five namespaces to alter processes view of the system: Process, Network, Mount, Hostname, Shared Memory.
The fact that, by default, as I mentioned in your previous question "Docker Namespace in kernel level" the container pid are isolated from the host (unless you run them with --pid host) is by design.
If you are using --pid=host, then those container pids are visible from the host, but not easily matched to a particular container, not until issue 10163 and --pid=container:id is resolved.
Update May 2016: issue 10163 and --pid=container:id is actually resolved by PR 22481 for docker 1.12, allowing to join another container's PID namespace.
I created a customize Docker image based on ubuntu 14.04 with the Sensu-Client package inside.
Everything's went fine but now I'm wondering how can I trigger the checks to run from the hosts machine.
For example, I want to be able to check the processes that are running on the host machine and not only the ones running inside the container.
Thanks
It depends on what checks you want to run. A lot of system-level checks work fine if you run sensu container with --net=host and --privileged flags.
--net=host not just allows you to see the same hostname and IP as host system, but also all the tcp connections and interface metric will match for container and host.
--privileged gives container full access to system metrics like hdd, memory, cpu.
Tricky thing is checking external process metrics, as docker isolates it even from privileged container, but you can share host's root filesystem as docker volume ( -v /:/host) and patch check to use chroot or use /host/proc instead of /proc.
Long story short, some checks will just work, for others you need to patch or develop your own way, but sensu in docker is one possible way.
an unprivileged docker container cannot check processes outside of it's container because docker uses kernel namespaces to isolate it from all other processes running on the host. This is by design: docker security documentation
If you would like to run a super privileged docker container that has this namespace disabled you can run:
docker run -it --rm --privileged --pid=host alpine /bin/sh
Doing so removes an important security layer that docker provides and should be avoided if possible. Once in the container, try running ps auxf and you will see all processes on the host.
I don't think this is possible right now.
If the processes in the host instance are running inside docker, you can mount the socket and get the status from the sensu container
Add a sensu-client to the host machine? You might want to split it out so you have granulation between problems in the containers VS problems with your hosts
Else - You would have to set up some way to report from the inside - Either using something low level (system calls etc) or set up something from outside to catch the the calls and report back status.
HTHs
Most if not all sensu plugins hardcode the path to the proc files. One option is to mount the host proc files to a different path inside of the docker container and modify the sensu plugins to support this other location.
This is my base docker container that supports modifying the sensu plugins proc file location.
https://github.com/sstarcher/docker-sensu