I have a C program running in an alipine docker container. The image size is 10M on both OSX and on ubuntu.
On OSX, when I run this image, using the 'docker stats' I see it uses 1M RAM and so in the docker compose file I allocate a max of 5M within my swarm.
However, on Ubuntu 16.04.4 LTS the image is also 10M but when running it uses about 9M RAM, and I have to increase the max allocated memory in my compose file.
Why is there a such a difference in RAM usage between OSX and Ubuntu?
Even though we have different OSs, I would have thought once you are running inside a framework, then you would behave similarly on different machines. So I would have thought there should be comparable memory usage.
Update:
Thanks for the comments. So 'stats' may be inaccurate, and there are differences so best to baseline on linux. As an aside, but I think interesting, the reason for asking this question is to understand the 'under the hood' in order to tune my setup for a large number of deployed programs. Originally, when I tested I tried to allocate the smallest amount of maximum RAM on ubuntu, this resulted in a lot of disk thrashing something I didn't see or hear on my Macbook, (no hard disks!).
Some numbers which are completely my setup but also I think are interesting.
1000 docker containers, 1 C program each, 20M RAM MAX per container, Server load of 98, Server runs 4K processes in total, [1000 C programs total]
20 docker containers, 100 C programs each, 200M RAM MAX per container, Server load of 5 to 50, Server runs 2.3K processes in total, [2000 C programs total].
This all points at give your docker images a good amount of MAX RAM and it is nicer to your server to have fewer docker containers running.
Related
I am using a Docker Setup that consists of 14 different containers. Every container gets a cpu_limit of 2 and a mem_limit of 2g.
To create and run these containers, I've written a Python script that uses the docker-py library. As of now, the containers are created sequentially, which takes approximately 2 minutes.
Now I'm thinking about parallelizing the process. So now instead of doing (its pseudocode):
for container in containers_to_start:
create_container(container)
I do
from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(4)
pool.map(create_container, containers_to_start)
And as a result the 14 containers are created 2x faster. BUT: The applications within the containers take a significant longer time to boot. At the end of the day, i dont gain really much, the time until every application is reachable is more or less the same, no matter if with or without multithreading.
But I don't really know why, because every container gets the same amount of CPU and memory resources, so I would expect the same boot time no matter how many containers are starting at the same time. Clearly this is not the case. Maybe I'm missing some knowledge here, any explanation would be greatly appreciated.
System Specs
CPU: intel i7 # 2.90 GHz
32GB RAM
I am using Windows 10 with Docker installed in WSL2 backend.
I am working on Hyperledger Fabric ver 1.4, my server freeze when many transactions are been done. My understanding is that the old transactions or versions which are committed do not get removed but stay in the docker using disk space ( I might be wrong on this assumption ) therefore the only solution I have yet found is increasing my virtual server disk space to 120GB. 100GB for docker and 20GB to run my Front-end development.
I have looked into Alpine images but right now I do not want to take that route.
With Current Configurations I have 75GB SSD and 4GB of RAM
Is there a way to decrease the Max Disk size been used in Ubuntu
16.04 LTS 64bit Minimal?
If not, is the Maximum container size 100GB in Docker?
In GUI interface I can increase and decrease the size as show in the attached image.
GUI Interface Docker
We are in the process of trying to migrate a windows desktop app to docker. We have created a light-weight Ubuntu based container with wine + vnc and the app is running well.
We need to run a large quantity of these apps on a given host, circa 500 - 600 per host. The host its self is a high spec, 4 x 8 Core CPU.
When testing under load, dockerd is using a very high amount of the sys cpu, and by high i mean for every 1% of user CPU in use, its using around 1% of sys cpu.
The problem this is causing is that compared to running the same app under Windows / Hyper-V, we can only get 50% quantity wise of the same application running, which is clearly an issue. If we were to factor out the sys CPU load, then they are pretty much equal.
Networking wise, we are using MACVLAN where each container has its own IP address that is mapped directly into the network.
First of all, is this normal for dockerd to be using so much CPU?
Cheers in advance!
Im using the "gci" container optimised vm image running on GCP.
My program has a spike in disk reads, and I think RAM, and then crashes.
The problem is I cannot see RAM usage, only disk and CPU.
I cannot install any utilities on the "gci" vm, I can only run tools inside a Debian based container "toolbox".
How do I record RAM usage?
There are several commands in Linux that can be used to check RAM. For example vmstat, top, free, /prox/meminfo. See this link: https://www.linux.com/blog/5-commands-check-memory-usage-linux
If I am running multiple docker containers with bursty memory and CPU utilization, will they be able to use the full capacity of the host machine? Or will they be limited to their CPU and memory limits of the individual container definitions?
For example:
If I were running 3 containers that burst to 1GB of memory once per day, at disjoint times.
And similarly if those same containers instead were CPU heavy, and bursted to 1CPU unit per day at disjoint times.
Could I run those 3 containers on a box with only 1.1GB of memory, or 1.1 CPU unit respectively?
Docker containers are not VM's,
They run in a cage over the host OS kernel, so there's no hypervisor magic behind.
Processes running inside a container are not much different from host processes from a kernel point of view. They are just highly isolated.
Memory and cpu scheduling will be handled by the "host". What you set on docker settings are CPU shares, to give priority and bounds to some containers.
So yes, containers with sleeping processes won't consume much cpu/memory if the used memory is correctly freed after the processing spike, otherwise, that memory would be swapped out, with no much performance impact.
Instantiating a docker container will only consume memory resources. As long as no process is running, you will see zero cpu usage by it.
I would recommend reviewing cgroups documentation, and actually docs for cgroups v2, since they are better structured that v1 docs. See chapter 5 for cpu and memory controllers: https://www.kernel.org/doc/Documentation/cgroup-v2.txt
When you don't need to explicitly specify --memory and --cpu-shares option at the container startup, the container will have all the cpu share and memory available for use on the instance. If no other process is consuming the resources,then the container can use all the cpu and memory available.
In theory you should be able to run the 3 containers on the instance.
Make sure non of the containers tie up the memory or cpu resources.