we're running several containers on a single docker host, mainly to run R and Python apps for data analysis. So when I load a big table into one of the containers, its memory footprint on the docker host increases.
However, when I close the Jupyter Notebook or R session, the container's memory footprint appears to remain unchanged on the host. It seems that the memory consumption of a docker container can only go up, and not down.
So I know that Linux in general occupies memory which is not needed by other applications (stuff is cached). However, how is this dealt with in the case of Docker containers? From the individual containers' perspectives there is a lot of memory (we don't want to limit the memory available to containers), and even if it is not needed inside this particular container, it would remain "occupied" in the container, and therefore inaccessible by other containers. And the host doesn't know if this memory is really needed or simply used for caching.
So how is this dealt with? I can imagine a situation where several people have started containers in which they have loaded or generated big data sets, but this was only temporary, and now the host's memory is all occupied because the memory is not freed.
I'm pretty sure that this is not how it works, so can someone explain this to me, please?
Many thanks,
Enno
In the Docker documentation, under resource constraints, there is an explanation about limiting memory to containers. When running a container, the memory is not freed based on the processes running in the container. The docs explain how the host system manages memory:
It is important not to allow a running container to consume too much of the host machine’s memory. On Linux hosts, if the kernel detects that there is not enough memory to perform important system functions, it throws an OOME, or Out Of Memory Exception, and starts killing processes to free up memory. Any process is subject to killing, including Docker and other important applications. This can effectively bring the entire system down if the wrong process is killed.
Docker attempts to mitigate these risks by adjusting the OOM priority on the Docker daemon so that it is less likely to be killed than other processes on the system...
The Docker containers can use memory but is prevented by the Docker daemon from crashing the host system. The memory allotted to Docker containers can also be limited:
Docker can enforce hard memory limits, which allow the container to use no more than a given amount of user or system memory, or soft limits, which allow the container to use as much memory as it needs unless certain conditions are met, such as when the kernel detects low memory or contention on the host machine.
We do not want to limit memory to the containers, but there are options to do so, like --memory=<value> , --memory-swap , and --memory-reservation. So no, the host cannot free up memory of a container that is running, but it does prevent the risk of all memory being occupied and making the kernel potentially kill a crucial system process.
Please excuse the formatting. Hope this helps; I also linked the related documentation. Also, not completely related, but maybe you can check this out about using a Java application in a container:
Why the docker container memory usage doesn't decrease?
Related
I am writing and testing a cpp program in a docker container. And I did not designate the max memory size for the container.
docker run -it xxx:latest /bin/bash
And cpp program will sometimes cause memory leak such as not free the allocated heap memory.
So I am curious that if the memory leak will disappear in the host linux when I kill the container?
Or this memory leak caused by the program in the container still exists in the host?
A Docker container is a wrapper around a single process. Killing the container also kills that process; conversely, if the process exits on its own, that causes the container to exit too.
Ending a process will release all of the memory that process used. So, if you have a C++ program, and it calls new without a corresponding delete, it will leak memory, but ending the process will reclaim all of the process's memory, even space the application has lost track of. This same rule still applies if the process is running in a container.
This also applies to other leak-like behavior and in other languages; for example, appending a very large number of values to a list and then ignoring them, so they're still "in use" unnecessarily. Some other OS resources like file descriptors are cleaned up this way, but some are not. In particular, if you fork(2) a subprocess, you must clean up after it yourself. Similarly, if you have access to the Docker API, you must clean up any related containers you spawn yourself.
What is the best practice to select memory size for a container running on an ec2?
My ec2 has 8gig of ram. It runs 2 containers.
PHP container
NGINX container
The NGINX container is set to 512 Mb.
How large is the recommendation to make the PHP container? It was set by some sort of default to 2 gigs, I want to make it 6gigs but was interested to hear what the recommendations are?
The basis for this question is that our container ran out of memory, and died. I believe we can alleviate this by upping the memory.
Best way to set memory limits is based on monitored metrics for memory usage. If you don't have metrics, then increase in increments and observe. Leave enough memory, say 1-2G, for the operating system itself.
Side note: Setup the container to auto restart via --restart=always, so even if the container is OOMKilled the app is restarted and continues to function.
I'm working on project where I divided the application in multiple docker images and I'm running around 5 containers where each one has its own image. Following the "One process per container" rule.
For that I'm using a beaglebone black which has only 480Mb of memory. Sometimes when the application runs for some time it crashes due to Out of memory exception.
So I was wondering if I make the images smaller would it consume less memory? How is the memory allocated for each container?
What if I group some images/containers into a single running container with more than one process? Would it use less memory?
When a process is killed with an OOM exception this is not related to the docker image size, this is the amount of memory the process is trying to use.
You can specify some memory limits on each container when you run them.
For example this will limit your container to 100MB of memory.
docker run -m 100M busybox
However if your applications exceed their allocated memory they will be killed with an OOM exception. The problem you are having is likely because the applications you are running have a minimum requirement which is higher than your beaglebone black.
Grouping processes into one container will not help, they will still use the same amount of memory.
If I am running multiple docker containers with bursty memory and CPU utilization, will they be able to use the full capacity of the host machine? Or will they be limited to their CPU and memory limits of the individual container definitions?
For example:
If I were running 3 containers that burst to 1GB of memory once per day, at disjoint times.
And similarly if those same containers instead were CPU heavy, and bursted to 1CPU unit per day at disjoint times.
Could I run those 3 containers on a box with only 1.1GB of memory, or 1.1 CPU unit respectively?
Docker containers are not VM's,
They run in a cage over the host OS kernel, so there's no hypervisor magic behind.
Processes running inside a container are not much different from host processes from a kernel point of view. They are just highly isolated.
Memory and cpu scheduling will be handled by the "host". What you set on docker settings are CPU shares, to give priority and bounds to some containers.
So yes, containers with sleeping processes won't consume much cpu/memory if the used memory is correctly freed after the processing spike, otherwise, that memory would be swapped out, with no much performance impact.
Instantiating a docker container will only consume memory resources. As long as no process is running, you will see zero cpu usage by it.
I would recommend reviewing cgroups documentation, and actually docs for cgroups v2, since they are better structured that v1 docs. See chapter 5 for cpu and memory controllers: https://www.kernel.org/doc/Documentation/cgroup-v2.txt
When you don't need to explicitly specify --memory and --cpu-shares option at the container startup, the container will have all the cpu share and memory available for use on the instance. If no other process is consuming the resources,then the container can use all the cpu and memory available.
In theory you should be able to run the 3 containers on the instance.
Make sure non of the containers tie up the memory or cpu resources.
I'm running npm inside a docker container and every so often it aborts because it cannot allocate enough memory. I see some flags like --memory (How do I set resources allocated to a container using docker?) for the docker run command that seem to limit the maximum amount of memory that a container can consume, but haven't seen anything yet that would allow me to reserve an amount of memory for the container and abort immediately if it cannot be allocated.
This is not how memory management works under Linux.
If you run full virtualization, like QEMU, then all memory can be allocated and passed down into the VM. That VM then boots the kernel and the memory is managed by the kernel in the VM.
In Docker, or any other container/namespace system, the memory is managed by the kernel that runs docker and the "containers". The process that is run in container still runs like a normal process but in a different cgroup. Each cgroup has limits, like how much memory the kernel will hand out to userland, or what network interfaces it sees, but it still runs on same kernel.
An analogy of this is that docker is a "glorified ulimit". Processes under this limit still behave as normal Linux processes
they allocate memory as-needed
they will cause OOM issues if they pass some limit, or host runs out of memory
And just like you can't pre-allocate memory for Firefox, you can't pre-allocate memory for a Docker container.
You can't reserve memory in docker, only limit it with --memory.
See: https://docs.docker.com/engine/reference/run/ for more detail.
Specifically look at the user memory constraints section.
memory=inf, memory-swap=inf (default) >>>> There is no memory limit for
the container. The container can use as much memory as needed.
Note that's the default. So like other processes on the system npm will use all it can get/need.
So either free up some memory or add more.
As others have said, you cannot reserve memory for processes, and therefore containers. However, you could have the node app called from a script that will check the available memory and exit if it is below a certain threshold.