I am trying to measure total cpu_usage for the docker container using GET /containers/$CONTAINER_ID/stats API and while the container is running the endpoint returns meaningful data:
{"cpu_usage":{"total_usage":213788430,"percpu_usage":[1917539,288587,145139232,1911156,5966276,4931989,11474144,1252778,593573,847283,39465873,0],"usage_in_kernelmode":20000000,"usage_in_usermode":160000000}
but once it exits everything is zeroed out:
{"cpu_usage":{"total_usage":0,"usage_in_kernelmode":0,"usage_in_usermode":0}
How can I get the total CPU usage for a docker container which has finished it's work?
Related
We are using docker swarm on the server. using openjdk8. If do :
docker service ls
see the result :
ID NAME MODE REPLICAS IMAGE PORTS
7l89205dje61 integration_api replicated 1/1 docker.repo1.tomba.com/koppu/koppu-api:3.1.2.96019dc
.................
I am trying to update jvm heap size for this service so I tried :
docker service update --env-add JAVA_OPTS="-Xms3G -Xmx3G -XX:MaxPermSize=1024m" integration_api
Saw this result:
integration_api
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
Now I am trying to see the heap size and not finding a way as when tried to get inside the container taking the id above as :
docker exec -it 7l89205dje61 bash
getting error :
this container does not exit.
Any suggestion?
Perhaps you can exec into the running container and display the current heap size with something like this?
# get the name of a container within your service with
docker exec -it <CONTAINER-ID> bash
# after execing into the container,
java -XX:+PrintFlagsFinal -version | grep HeapSize
Use this Stack Orerflow post to figure out how to exec into a service
Got the java code to print heap settings from this Stack Overflow post
Note these ideas don't have a good example out there as yet to my knowledge. However, one good way of doing it is to implement a "healthcheck" process that would query the JVM statistics like heap and other things and report it to another system.
Another way is exposing the Spring Boot Actuator API so that Prometheus can read and track it over time.
We are using docker swarm on the server for orchestration. using openjdk8. My backend application is a rest service named "api". On the master, if do :
docker service ls
see the result :
ID NAME MODE REPLICAS IMAGE PORTS
7l89205dje61 integration_api replicated 1/1 docker.repo1.tomba.com/koppu/koppu-api:3.1.2.96019dc
.................
Time to time I am seeing an error in this docker service log (docker service logs integration_api):
java.lang.OutOfMemoryError: Java heap space
And hence, I am trying to update jvm heap size for this docker service. so I tried :
docker service update --env-add JAVA_OPTS="-Xms3G -Xmx3G -XX:MaxPermSize=1024m" integration_api
Saw this result:
integration_api
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
However, it does not actually seem to increase the heap in the container. What should I do differently?
I started running a celery beat worker in a dedicated container. This works fine sometimes, but now I get the following error trying to remove or re-deploy my containers:
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
Also, I cannot access the container anymore and the following commands just get stuck:
docker restart beat
docker logs beat
docker exec beat bash
I have created a new docker services and am determining its required resources. Since applying the RAM to a new service is greedy--saying the container can have 8GB of RAM it will get them--I don't want to waste the cluster's resources.
Now I am trying to find out how much RAM a docker run took at its peak.
For example, I created a httpie-image (for the rightfully paranoid, the Dockerfile is also on dockerhub that I execute via:
docker run -it k0pernikus/httpie-docker-alpine HEAD https://stackoverflow.com/
I know that there is a docker stats command, yet it appears to show the current memory usage, and I don't really want to monitor that.
If I run it after the container ended, it will show 0. (To get the container id, I use the d flag.)
$ docker run -itd k0pernikus/httpie-docker-alpine HEAD https://stackoverflow.com/
132a93ffc9e297250b8ca37b2563aa2b5e423e146890fe3383a91a7f26ef990c
$ docker stats 132a93ffc9e297250b8ca37b2563aa2b5e423e146890fe3383a91a7f26ef990c
it will show:
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
132a93ffc9e297250b8ca37b2563aa2b5e423e146890fe3383a91a7f26ef990c 0.00% 0 B / 0 B 0.00% 0 B / 0 B 0 B / 0 B 0
Yet how much RAM did it consume at maximum?
tl;dr:
Enable memory accounting
cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.max_usage_in_bytes
Docker uses cgroups under the hood. We only have to ask the kernel the right question, that is to cat the correct file. For this to work memory accounting has to be enabled (as noted in the docs).
On systemd based systems this is quite straight forward. Create a drop-in config for the docker daemon:
systemctl set-property docker.service MemoryAccounting=yes
systemctl daemon-reload
systemctl restart docker.service
(This will add a little overhead though as each time the container allocates RAM a counter has to be updated. Further details: https://lwn.net/Articles/606004/)
Then by using the full container id that you can discover via docker inspect:
docker inspect --format="{{.Id}}" $CONTAINER_NAME
you can get the maximum memory used:
cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.max_usage_in_bytes
The container has to be running for this to work.
I have a Docker container running with this command in my Jenkins job:
docker run --name="mydoc" reportgeneration:1.0 start=$START end=$END config=$myfile
This works very well. The image is created from a DockerFile which is executing a shell script with ENTRYPOINT.
Now I want to know how much CPU and memory has been utilized by this container. I am using a Jenkins job, where in the "execute shell command", I am running the above Docker run command.
I saw about 'docker stats' command. It works very well in my Ubuntu machine. But I want it to run via Jenkins as my container is running via Jenkins console. So here follows the limitations I have.
I don't know if there is any way to stop docker stats command. In Ubuntu command line, we hit 'ctrl+c' to stop it. How will I do it in Jenkins?
Even if I figure out a way to stop docker stats, once the 'docker run' command gets executed, the container will not be active and will be exited. For exited container, CPU and memory utilisation will be zero.
docker run 'image'
docker stats container id/name
With the above two lines, docker stats command will only get an exited container and I don't think docker stats will even work with Jenkins console as it cannot be stopped.
Is there any way that I can get container's resource utilization (CPU, memory) in a better way via Jenkins console?
Suggestion is to not run docker stats interactively, but have a piece of a shell script with a loop like this:
#!/bin/sh
# First, start the container
CONTAINER_ID=$(docker run -d ...)
# Then start watching that it's running (with inspect)
while [ "$(docker inspect -f {{.State.Running}} $CONTAINER_ID 2>/dev/null)" = "true" ]; do
# And while it's running, check stats
docker stats --no-stream $CONTAINER_ID
sleep 1
done
# When the script reaches this point, the container had stopped.
# For example, let's clean it up (assuming you haven't used --rm in run).
docker rm $CONTAINER_ID
The condition checks whenever the container is running or not, and docker stats --no-stream prints stats once then exits, making it suitable for non-interactive use.
I believe you can use a variant of such shell script file (obviously, updated to do something useful, rather than just starting the container and watching its stats) as a build step.
But if you need/want/have an interactive process that you want to stop, kill is the command you're looking for. Ctrl-C in a terminal just sends a SIGINT to the process.
You need to know an PID, of course. I'm not sure about Jenkins, but if you've just started a child process from a shell script with child-process & (e.g. docker stats &), then its PID would be in the $! variable. Or you can try to figure it using pidof or ps commands, but that may be error-prone in case of concurrent jobs (unless they're all isolated).
Here I've assumed that your Jenkins jobs are shell scripts that do the actual work. If your setup is different (e.g. if you use some plugins so Jenkins talk to Docker directly), things may be different and more complicated.