I am running several containers using docker-compose. I can see application logs with command docker-compose logs. However I would like to access raw log file to send it somewhere for example? Where is it located? I guess it's separate log per each container (inside container?) but where I can find it?
A container's logs can be found in :
/var/lib/docker/containers/<container id>/<container id>-json.log
(if you use the default log format which is json)
You can docker inspect each container to see where their logs are:
docker inspect --format='{{.LogPath}}' $INSTANCE_ID
And, in case you were trying to figure out where the logs were to manage their collective size, or adjust parameters of the logging itself you will find the following relevant.
Fixing the amount of space reserved for the logs
This is taken from Request for the ability to clear log history (issue 1083)):
Docker 1.8 and docker-compose 1.4 there is already exists a method to limit log size using docker compose log driver and log-opt max-size:
mycontainer:
...
log_driver: "json-file"
log_opt:
# limit logs to 2MB (20 rotations of 100K each)
max-size: "100k"
max-file: "20"
In docker compose files of version '2' , the syntax changed a bit:
version: '2'
...
mycontainer:
...
logging:
#limit logs to 200MB (4rotations of 50M each)
driver: "json-file"
options:
max-size: "50m"
max-file: "4"
(note that in both syntaxes, the numbers are expressed as strings, in quotes)
Possible issue with docker-compose logs not terminating
issue 1866: command logs doesn't exit if the container is already stopped
To see how much space each container's log is taking up, use this:
docker ps -qa | xargs docker inspect --format='{{.LogPath}}' | xargs ls -hl
(you might need a sudo before ls).
docker inspect <containername> | grep log
On Windows, the default location is: C:\ProgramData\Docker\containers\<container-id>-json.log.
Here is the location for
Windows 10 + WSL 2 (Ubuntu 20.04), Docker version 20.10.2, build 2291f61
Lets say
DOCKER_ARTIFACTS == \\wsl$\docker-desktop-data\version-pack-data\community\docker
Location of container logs can be found in
DOCKER_ARTIFACTS\containers\[Your_container_ID]\[Your_container_ID]-json.log
Here is an example
To directly view the logfile in less, I use:
docker inspect $1 | grep 'LogPath' | sed -n "s/^.*\(\/var.*\)\",$/\1/p" | xargs sudo less
run as ./viewLogs.sh CONTAINERNAME
As of 8/22/2018, the logs can be found in :
/data/docker/containers/<container id>/<container id>-json.log
To see the size of logs per container, you can use this bash command :
for cont_id in $(docker ps -aq); do cont_name=$(docker ps | grep $cont_id | awk '{ print $NF }') && cont_size=$(docker inspect --format='{{.LogPath}}' $cont_id | xargs sudo ls -hl | awk '{ print $5 }') && echo "$cont_name ($cont_id): $cont_size"; done
Example output:
container_name (6eed984b29da): 13M
elegant_albattani (acd8f73aa31e): 2.3G
Related
I am doing some work with docker container management, and need to get docker container status.
My current approach is using ssh client by execute shell cmd to grab.
For example i can get container stats by execute cmd:
docker stats --no-stream --format '{"container":"{{ .Name }}","memory":{"raw":"{{ .MemUsage }}","percent":"{{ .MemPerc }}"},"cpu":"{{ .CPUPerc }}","networkIO":"{{.NetIO}}","BlockIO":"{{.BlockIO}}"}'
output:
{"container":"postgresql","memory":{"raw":"255.4MiB / 31.21GiB","percent":"0.80%"},"cpu":"0.00%","networkIO":"1.03GB / 476MB,"BlockIO":"545MB / 7.67GB"}
{"container":"pgadmin","memory":{"raw":"146.1MiB / 31.21GiB","percent":"0.46%"},"cpu":"0.03%","networkIO":"26.2kB / 0B,"BlockIO":"200MB / 8.19kB"}
{"container":"pis_middle_layer_flask","memory":{"raw":"849.9MiB / 31.21GiB","percent":"2.66%"},"cpu":"13.48%","networkIO":"26.4kB / 0B,"BlockIO":"65.9MB / 0B"}
So how can i get similar text with docker system df -v?
Cause i want to get each container size and their volume size.
I did with same cmd:
docker system df -v --format '{"container":"{{ .Name }}","memory":{"raw":"{{ .MemUsage }}","percent":"{{ .MemPerc }}"},"cpu":"{{ .CPUPerc }}","networkIO":"{{.NetIO}}","BlockIO":"{{.BlockIO}}"}'
but occurred with error:
{"container":"template: :1:17: executing "" at <.Name>: can't evaluate field Name in type *formatter.diskUsageContext
I know i got wrong go template keywords, but really can't find anything doc about it.
Al right...just got it.
docker system df -v --format "{{ json . }}"
this cmd will return as json so i can parse it with json.loads
Rancher system started to use a heavy amount of disksspace. Kubernetes was setup by Rancher's RKE.
Diskusage is already over 5TB however I have only 10-12 replicaset, their real data is binded to PV which uses nfs (which has only a size of 10gb).
df -h --total clearly shows which one takes up so many space:
Filesystem Size Used Avail Use% Mounted on
overlay 98G 78G 16G 84% /var/lib/docker/overlay2/84db..somehash/merged
I have ~50-60 entry from these.
How can I cleanup these? Is there any maintenance feature in rancher for this? Couldn't find any though.
Kubernetes's garbage collection should be cleaning up your nodes.
This looks a lot an issue I saw with some log collectors like Splunk and Datadog.
If the following usage numbers do not match up. Then using the script below to release the file descriptors.
df -h /var/lib/docker
docker system df
Workaround:
ps aux | grep dockerd <<== This pid
cd /proc/`pid of dockerd`/fd
ls -l |grep var.log.journal |grep deleted.$ |awk '{print $9}' |while read x; do :> $x; done;
I have a simple code in Dotnet core console and it simply counted forever :
class Program
{
static void Main(string[] args)
{
int counter = 1;
while (true)
{
counter++;
Console.WriteLine(counter);
}
}
}
I contained it with docker and ran it via VS Docker run.
but every time that I execute > docker logs -f dockerID it starts and shows counting from scratch! (1,2,3,....). I expected that whenever I run this command it shows me logs from the last integer that it counts!
Is "docker logs -f" cause to run a new instance of my application every time?
When running docker help logs you see this:
--tail string Number of lines to show from the end of the logs
(default "all")
So you should do:
docker logs --tail 100 -f dockerID
Additional reference: docker logs documentation > Options
docker logs -f should work similar to tail -f, so, it should show just new logs.
Therefore, if you're watching logs repeated, it looks your program is being executed in loop.
The only thing that comes to my mind is a little trick, although it's not an elegant solution, it can be useful for other similar situations:
Define in your system history time format according to docker logs
export HISTTIMEFORMAT="%Y-%m-%dT%H:%M:%S "
Get your last docker logs command date with a simple script.
LAST_DOCKER_LOG=`history | grep " docker logs" | tac | head -1 | awk '{print $2}'`
Get logs after your last docker logs execution:
docker logs --since $LAST_DOCKER_LOG <your_docker>
In one line:
docker logs -t --since `history | grep " docker logs" | tac | head -1 | awk '{print $2}'` <your_docker>
Note: history time and docker log times have to be in the same timezone.
In spite of everything, I'd have a look to your loop execution. Maybe, docker log -t timestamp show option can help you to solve it.
Fisrt, docker run -d busybox sleep 2000
Seconde, ps aux | grep {container id} to find the pid;
Third, pstree -paAl {pid} shows the nine strange docker-containe;
What does these containes mean? 5214-5222
Fedora 29, docker.
I don't know how to format the output, but you can try it on your pc.I
only know the 5213 docker-containe is my container...
[emacsliu#localhost ~]$ pstree -aAlp 5213
docker-containe,5213
a82c3cf17ed83570758a80ff84fc9f6ff9e649b3407f4f0bc371f2e3aa5f351e /var/run/docker/libcontainerd/a82c3cf17ed83570758a80ff84fc9f6ff9e649b340
7f4f0bc371f2e3aa5f351e /usr/libexec/docker/docker-runc-current
|-sleep,5232 2000
|-{docker-containe},5214
|-{docker-containe},5215
|-{docker-containe},5216
|-{docker-containe},5217
|-{docker-containe},5218
|-{docker-containe},5219
|-{docker-containe},5220
|-{docker-containe},5221
-{docker-containe},5222
docker-containe is actually abbreviated docker-containerd-shim (or containerd-shim on some systems) and it is Docker component. Use other tools, which will show a full process name. For example htop tree view:
Ref:
http://alexander.holbreich.org/docker-components-explained/
dockerd vs docker-containerd vs docker-runc vs docker-containerd-ctr vs docker-containerd-shim
I have a Docker Compose v2 file which starts a container. I locally run a service on port 3001. I want to reach this service from the Docker container.
The Docker Compose file looks like this:
version: '2'
services:
my-thingy:
image: my-image:latest
#network_mode: host #DOES not help
environment:
- THE_HOST_I_WANT_TO_CONNECT_TO=http://127.0.0.1:3001
ports:
- "3010:3010"
Now, how can I reach THE_HOST_I_WANT_TO_CONNECT_TO?
What I tried is:
Setting network_mode to host. This did not work. 127.0.0.1 could not be reached.
I can also see that I can reach the host from the container if I use the local IP of the host. A quick hack would be to use something like ifconfig | grep broadcast | awk '{print $2}' to obtain the IP and substitute that in Docker Compose. Since this IP can change on reconnect and different setups can have different ifconfig results, I am looking for a better solution.
I've used another hack/workarkound from comments in the docker issue #1143. Seems to Work For Me™ for the time being... Specifically, I've added the following lines in my Dockerfile:
# - net-tools contains netstat, used to discover IP of Docker host server.
# NOTE: the netstat trick is to make Docker host server accessible
# from inside Docker container under name 'dockerhost'. Unfortunately,
# as of 2016.10, there's no official/robust way to do this when Docker host
# has no public IP/DNS entry. What is used here is built based on:
# - https://github.com/docker/docker/issues/1143#issuecomment-39364200
# - https://github.com/docker/docker/issues/1143#issuecomment-46105218
# See also:
# - http://stackoverflow.com/q/38936738/98528
# - https://github.com/docker/docker/issues/8395#issuecomment-200808798
# - https://github.com/docker/docker/issues/23177
RUN apt-get update && apt-get install -y net-tools
CMD (netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2" dockerhost"}' >> /etc/hosts) && \
...old CMD...
With this, I can use dockerhost as the name of the host where Docker is installed. As mentioned above, this is based on:
https://github.com/docker/docker/issues/1143#issuecomment-39364200
(...) One way is to rely on the fact that the Docker host is reachable through the address of the Docker bridge, which happens to be the default gateway for the container. In other words, a clever parsing of ip route ls | grep ^default might be all you need in that case. Of course, it relies on an implementation detail (the default gateway happens to be an IP address of the Docker host) which might change in the future. (...)
https://github.com/docker/docker/issues/1143#issuecomment-46105218
(...) A lot of people like us are looking for a little tidbit like this
netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'
where netstat -nr means:
Netstat prints information about the Linux networking subsystem.
(...)
--route , -r
Display the kernel routing tables.
(...)
--numeric , -n
Show numerical addresses instead of trying to determine symbolic host, port or user names.
This is a known issue with Docker Compose: see Document how to connect to Docker host from container #1143. The suggested solution of a dockerhost entry in /etc/hosts is not implemented.
I went for the solution with a shell variable as also suggested in a comment by amcdl on the issue:
Create a LOCAL_XX_HOST variable: export LOCAL_XX_HOST="http://$(ifconfig en0 inet | grep "inet " | awk -F'[: ]+' '{ print $2 }'):3001".
Then, for example, refer to this variable in docker-compose like this:
my-thingy:
image: my-image:latest
environment:
- THE_HOST_I_WANT_TO_CONNECT_TO=${LOCAL_XX_HOST}