I am building a project inside the docker container without any resource limitation on creating the container. when I am monitoring it, I see the different results for CPU usage.
from ctop
From the Grafana (Full Node Exporter Chart)
And from the cAdvisor
I do not understand why the results are different, specially with ctop command.
but my main question is, does Docker really use all CPUs? this machine has 16 vCPU and 16GB RAM
It's not exactly clear in the node exporter what instance or container you are monitoring, though it seems like the node exporter it showing the total machine CPU usage in 0-100 format and ctop shows in 100% per vCPU format.
Also try using docker stats, that should show all running containers resource usage, from cpu to network and disk usage, when using it each vCPU would as 100% so your total will be 1600% for 16 vCPU.
Regarding the cAdvisor output it doesn't show the same time range as the grafana node exporter so it would be hard to make a hard conclusion but it seems like similarly to ctop and docker stats it shows on a per core basis but instead of percentage it shows in 'cores' unit of measurement
Related
I have a C program running in an alipine docker container. The image size is 10M on both OSX and on ubuntu.
On OSX, when I run this image, using the 'docker stats' I see it uses 1M RAM and so in the docker compose file I allocate a max of 5M within my swarm.
However, on Ubuntu 16.04.4 LTS the image is also 10M but when running it uses about 9M RAM, and I have to increase the max allocated memory in my compose file.
Why is there a such a difference in RAM usage between OSX and Ubuntu?
Even though we have different OSs, I would have thought once you are running inside a framework, then you would behave similarly on different machines. So I would have thought there should be comparable memory usage.
Update:
Thanks for the comments. So 'stats' may be inaccurate, and there are differences so best to baseline on linux. As an aside, but I think interesting, the reason for asking this question is to understand the 'under the hood' in order to tune my setup for a large number of deployed programs. Originally, when I tested I tried to allocate the smallest amount of maximum RAM on ubuntu, this resulted in a lot of disk thrashing something I didn't see or hear on my Macbook, (no hard disks!).
Some numbers which are completely my setup but also I think are interesting.
1000 docker containers, 1 C program each, 20M RAM MAX per container, Server load of 98, Server runs 4K processes in total, [1000 C programs total]
20 docker containers, 100 C programs each, 200M RAM MAX per container, Server load of 5 to 50, Server runs 2.3K processes in total, [2000 C programs total].
This all points at give your docker images a good amount of MAX RAM and it is nicer to your server to have fewer docker containers running.
I'm relatively new to docker. I'm trying to get the percent of cpu quota (actually) used by a container. Is there a default metric emitted by one of the endpoints or is it something that I will have to calculate with other metrics? Thanks!
docker stats --no-stream
CONTAINER ID NAME CPU % (rest of line truncated)
949e2a3724e6 practical_shannon 8.32% (truncated)
As mentioned in the comment from #asuresh4, above, docker stats appears to give the ACTUAL cpu utilization, not the configured values. The output here is from Docker version 17.12.1-ce, build 7390fc6
--no-stream means run stats once, not continuously as it normally does. As you might guess, you can also ask for stats on a single container (specify the container name or id).
In addition to CPU %, MEM USAGE / LIMIT, MEM %, NET I/O, and BLOCK I/O are also shown.
Is there any way to monitor disk usage of docker containers in DataDog?
I can see in DataDog web all the CPU, RAM and IO metrics for my containers.
But I can't see any of disk space related metrics.
Their page https://docs.datadoghq.com/integrations/docker/ says about:
docker.disk.used (now reported as docker.data.used)
docker.disk.free (now reported as docker.data.free)
docker.disk.total (now reported as docker.data.total)
I can't find these neither in Dashboards > Docker nor in Metrics > Explorer
I'm new to DataDog, so possibly missing something obvious here.
There are 2 relevant options in /etc/dd-agent/conf.d/docker_daemon.yaml:
collect_disk_stats
If you use devicemapper-backed storage (which is default in ECS but not in vanilla Docker or Kubernetes), docker.data.* and docker.metadata.* statistics should do what you are looking for.
collect_container_size
A generic way, using the docker API but virtually running df in every container. This enables the docker.container.* metrics.
See more here:
https://help.datadoghq.com/hc/en-us/articles/115001786703-How-to-report-host-disk-metrics-when-dd-agent-runs-in-a-docker-container-
and here:
https://github.com/DataDog/docker-dd-agent/blob/master/conf.d/docker_daemon.yaml#L46
I'm trying to build a system which runs pieces of code in consistent conditions, and one way I imagine this being possible is to run the various programs in docker containers with the same layout, reserving the same amount of memory, etc. However, I can't seem to figure out how to keep CPU usage consistent.
The closest thing I can seem to find are "cpu shares," which, if I understand the documentation, limit cpu usage with respect to what other containers/other processes are running on the system, and what's available on the system. They do not seem to be capable of limiting the container to an absolute amount of cpu usage.
Ideally, I'd like to set up docker containers that would be limited to using a single cpu core. Is this at all possible?
If you use a newer version of Docker, you can use --cpuset-cpus="" in docker run to specify the CPU cores you want to allocate:
docker run --cpuset-cpus="0" [...]
If you use an older version of Docker (< 0.9), which uses LXC as the default execution environment, you can use --lxc-conf to configure the allocated CPU cores:
docker run --lxc-conf="lxc.cgroup.cpuset.cpus = 0" [...]
In both of those cases, only the first CPU core will be available to the docker container. Both of these options are documented in the docker help.
I've tried to provide a tutorial on container resource alloc.
https://gist.github.com/afolarin/15d12a476e40c173bf5f
Supposed I want to run several Docker containers at the same time.
Is there any formula I can use to find out in advance on how many containers can be run at the same time by a single Docker host? I.e., how much CPU, memory & co. do I have to take into account for the containers themselves?
It's not a formula per se, but you can gather information about resource usage in the container by examining Linux control groups in /sys/fs/cgroup.
Links
See this excellent post by Jérôme Petazzoni of Docker, Inc on the subject.
See also Google's cAdvisor tool to view container resource usage.
This IBM research paper documents that Docker performance is higher than KVM in every measurement.
docker stats is also useful for getting a rough idea for how CPU and Memory your containers use.
cAdvisor is going to provide resource usage and other interesting stats about all containers o a host. We have a preliminary setup for root usage, but we are adding a lot more this week.