I currently use docker for my backend, and when I first start them up with
docker-compose up
I get log outputs of all 4 dockers at once, so I can see how they are interacting with each other when a request comes in. Looking like this, one request going from nginx to couchdb
The issue is now that I am running on GCE with load balancing, when a new VM spins up, it auto starts the dockers and runs normally, I would like to be able to access a load balanced VM and view the live logs, but I can not get docker to allow me this style, when I use logs, it gives me normal all white font with no label of where it came from.
Using
docker events
does nothing, it won't return any info.
tldr; what is the best way to obtain a view, same as the log output you get when running "docker-compose up"
If using docker-compose, you use
docker-compose logs --tail=0 --follow
instead of
docker logs --tail=0 --follow
This will get the output I was originally looking for.
You can see the logs for all running containers with
docker ps -q | xargs -L 1 docker logs
In theory this might work for the --follow too if xargs is ran with -P <count>, where the count is higher than the number of running containers.
I use a variation of this to live tail (--follow) all logs and indicate which log is tailing at the time. This bash includes both stdout and stderr. Note you may need to purge the /tmp dir of *.{log,err} afterwards.
for c in $(docker ps -a --format="{{.Names}}")
do
docker logs -f $c > /tmp/$c.log 2> /tmp/$c.err &
done
tail -f /tmp/*.{log,err}
Hope this helps. Logging has become so problematic these days, and other get-off-my-lawn old man rants...
Try "watch"
Here's a quick and dirty multitail/xtail for docker containers.
watch 'docker ps --format "{{.Names}}" | sort | xargs --verbose --max-args=1 -- docker logs --tail=8 --timestamps'
How this works:
watch to run every few seconds
docker ps --format "{{.Names}}" to get the names of all running containers
sort to sort them
xargs to give these names to docker logs:
docker logs to print the actual logs
Adjust parameter "--tail=8" as needed so that everything still fits on one screen.
The "xargs" methods listed above (in another user's answer) will stop working as containers are stopped and restarted. This "watch" method here does not have that problem. (But it's not great either.)
If you are using Docker Swarm, you can find your services by
docker service ls
Grap the id, and then run
docker service logs $ID -f
if the service is defined with tty: true, then you must run with the --raw flag. Notice, this wont tell you which container is giving the outputted log entry.
Related
I use Dokku to run my app, and for some reason, the container is dying every few hours and recreates itself.
In order to investigate the issue, I am willing to read the error logs to this container and understand why it's crashing. Since Docker clears logs of dead containers, this is impossible.
I turned on docker events and it shows many events (like container update, container kill, container die, etc.) But no sign of what triggered this kill.
How can I investigate the issue?
Versions:
Docker version 19.03.13, build 4484c46d9d
dokku version 0.25.1
Logs are deleted when the container is deleted. If you want the logs to persist, then you need to avoid deleting the container. Make sure you aren't running the container with an option like --rm that automatically deletes it on exit. And check for the obvious issues like running out of disk space.
There are several things you can do to investigate the issue:
You can run the container in the foreground and allow it to log to your console.
If you were previously starting the container in the background with docker run -d (or docker-compose up -d), just remove the -d from the command line and allow the container to log to your terminal. When it crashes, you'll be able to see the most recent logs and scroll back to the limits of your terminal's history buffer.
You can even capture this output to a file using e.g. the script tool:
script -c 'docker run ...`
This will dump all the output to a file named typescript, although you can of course provide a different output name on the command line.
You can change the log driver.
You can configure your container to use a different logging driver. If you select something like syslog or journald, your container logs will be sent to the corrresponding service, and will continue to be available even after the container has been deleted.
I like use the journald logging driver because this allows searching for output by container id. For example, if I start a container like this:
docker run --log-driver journald --name web -p 8080:8080 -d docker.io/alpinelinux/darkhttpd
I can see logs from that container by running:
$ journalctl CONTAINER_NAME=web
Feb 25 20:50:04 docker 0bff1aec9b65[660]: darkhttpd/1.13, copyright (c) 2003-2021 Emil Mikulic.
These logs will persist even after the container exits.
(You can also search by container id instead of name by using CONTAINER_ID_FULL (the full id) or CONTAINER_ID (the short id), or even by image name with IMAGE_NAME.)
$ docker run --rm -it busybox
/ # who
<empty>
In the next session I'm trying to attach to this docker container and expecting second user will appear, but no luck again:
$ docker attach `docker container ls | grep busybox | cut -d" " -f1`
/ # who
<empty again>
So the question is - why there are no logons happened not by first run-and-attach, not by consequent attaches? And why there is no even a single logon into this container?
who reads the list of users from /var/run/utmp. On a regular Linux system, the login program prompts for the username and password and then starts the user's shell. It also updates /var/run/utmp with the new user.
The same thing happens for SSH and Telnet servers. They are expected to update /var/run/utmp.
In a Docker container, login is usually not executed. Docker isolates resources from the host system with Linux Namespaces, it does not provide a complete Linux system. When you enter a Docker container, the given entrypoint or command is executed with PID 1.
Subsequent docker exec calls are handled in a similar way. Docker enters the namespace of the container and executes the given command.
EDIT: after some reading I see Alexander answer as more to the point. Couple of useful links I've read along that way:
https://docs.docker.com/engine/security/security/
https://lwn.net/Articles/531114
As far as I understand busybox docker container is very basic and does not support all functionality of full-fledged Linux.
Here I thought I understood Docker until I saw the BusyBox docker image there is a discussion about what that image is and what it is for.
I can view the list of running containers with docker ps or equivalently docker container ls (added in Docker 1.13). However, it doesn't display the user who launched each Docker container. How can I see which user launched a Docker container? Ideally I would prefer to have the list of running containers along with the user for launched each of them.
You can try this;
docker inspect $(docker ps -q) --format '{{.Config.User}} {{.Name}}'
Edit: Container name added to output
There's no built in way to do this.
You can check the user that the application inside the container is configured to run as by inspecting the container for the .Config.User field, and if it's blank the default is uid 0 (root). But this doesn't tell you who ran the docker command that started the container. User bob with access to docker can run a container as any uid (this is the docker run -u 1234 some-image option to run as uid 1234). Most images that haven't been hardened will default to running as root no matter the user that starts the container.
To understand why, realize that docker is a client/server app, and the server can receive connections in different ways. By default, this server is running as root, and users can submit requests with any configuration. These requests may be over a unix socket, you could sudo to root to connect to that socket, you could expose the API to the network (not recommended), or you may have another layer of tooling on top of docker (e.g. Kubernetes with the docker-shim). The big issue in that list is the difference between the network requests vs a unix socket, because network requests don't tell you who's running on the remote host, and if it did, you'd be trusting that remote client to provide accurate information. And since the API is documented, anyone with a curl command could submit a request claiming to be a different user.
In short, every user with access to the docker API is an anonymized root user on your host.
The closest you can get is to either place something in front of docker that authenticates users and populates something like a label. Or trust users to populate that label and be honest (because there's nothing in docker validating these settings).
$ docker run -l "user=$(id -u)" -d --rm --name test-label busybox tail -f /dev/null
...
$ docker container inspect test-label --format '{{ .Config.Labels.user }}'
1000
Beyond that, if you have a deployed container, sometimes you can infer the user by looking through the configuration and finding volume mappings back to that user's home directory. That gives you a strong likelihood, but again, not a guarantee since any user can set any volume.
I found a solution. It is not perfect, but it works for me.
I start all my containers with an environment variable ($CONTAINER_OWNER in my case) which includes the user. Then, I can list the containers with the environment variable.
Start container with environment variable
docker run -e CONTAINER_OWNER=$(whoami) MY_CONTAINER
Start docker compose with environment variable
echo "CONTAINER_OWNER=$(whoami)" > deployment.env # Create env file
docker-compose --env-file deployment.env up
List containers with the environment variable
for container_id in $(docker container ls -q); do
echo $container_id $(docker exec $container_id bash -c 'echo "$CONTAINER_OWNER"')
done
As far as I know, docker inspect will show only the configuration that
the container started with.
Because of the fact that commands like entrypoint (or any init script) might change the user, those changes will not be reflected on the docker inspect output.
In order to work around this, you can to overwrite the default entrypoint set by the image with --entrypoint="" and specify a command like whoami or id after it.
You asked specifically to see all the containers running and the launched user, so this solution is only partial and gives you the user in case it doesn't appear with the docker inspect command:
docker run --entrypoint "" <image-name> whoami
Maybe somebody will proceed from this point to a full solution (:
Read more about entrypoint "" in here.
If you are used to ps command, running ps on the Docker host and grep with parts of the process your process is running. For example, if you have a Tomcat container running, you may run the following command to get details on which user would have started the container.
ps -u | grep tomcat
This is possible because containers are nothing but processes managed by docker. However, this will only work on single host. Docker provides alternatives to get container details as mentioned in other answer.
this command will print the uid and gid
docker exec <CONTAINER_ID> id
ps -aux | less
Find the process's name (the one running inside the container) in the list (last column) and you will see the user ran it in the first column
I have multiple docker instance running and I want to push the log outputs to a GUI.
I wrote python script which basically does
docker logs $containerId --tail 50 -f
for each container.
The result is nice and it works fine, but I have a very high cpu load from dockerd.
What can I do to prevent the high load? Are there better ways to get the log output?
Thanks in advance.
I fixed it. Docker also saves the output to a file:
docker inspect --format='{{.LogPath}}' $containerId
Then I just tail this file:
tail -f $path
I have a simple code for which I have created a docker container and the status shows it running fine. Inside the code I have used some print() commands to print the data. I wanted to see that print command output.
For this I have seen docker logs . But it seems not to be working as it shows no logs. How to check logs.?
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a3b3fd261b94 myfirstdocker "python3 ./my_script…" 22 minutes ago Up 22 minutes elegant_darwin
$ sudo docker logs a3b3fd261b94
<shows nothing>
The first point you need to print your logs to stdout.
To check docker logs just use the following command:
docker logs --help
Usage: docker logs [OPTIONS] CONTAINER
Fetch the logs of a container
Options:
--details Show extra details provided to logs
-f, --follow Follow log output
--help Print usage
--since string Show logs since timestamp
--tail string Number of lines to show from the end of the logs (default "all")
-t, --timestamps Show timestamps
Some example:
docker logs --since=1h <container_id>
4
Let's try using that docker create start and then logs command again and see what happens.
sudo docker create busybox echo hi there
output of the command
now I will take the ID and run a docker start and paste the ID that starts up the container it executes echo high there inside of it and then immediately exits.
Now I want to go back to that stopped container and get all the logs that have been emitted inside of it.
To do so I can run at docker logs and then paste the ID in and I will see that when the container had been running it had printed out the string Hi there.
One thing to be really clear about is that by running docker logs I am not re-running or restarting the container to in any way shape or form, I am just getting a record of all the logs that have been emitted from that container.
docker logs container_id
If there's not so much supposed output (e.g. script just tries to print few bytes), I'd suspect python is buffering it.
Try adding more data to the output to be sure that buffer is flushed, and also using PYTHONUNBUFFERED=1 (although, python3 still may do some buffering despite of this setting).