I would like to set limitation of cpu resource, and found this docker-compose setting file below.
version: '3'
services:
redis:
image: redis:alpine
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
reservations:
cpus: '0.0001'
memory: 20M
BUT I don't know what 0.001 mean in here. It will use 0.1% of total cpu capacity? and some document said it can be set more than 1. what happen If i set more than 1?
and What is default value of resource.limits.cpus and resource.limits.cpus.memory?
By default a container has limitless access to the host's cpu. Limits allow you to configure resources that are available to a container.
A cpus value of 0.001 will mean for each one second, a 1 millisecond will be available to the container.
A value greater than 1 will make sense when you think of hosts with multi-cpus. A value of 2 for instance will mean the container will be given access for at most 2 cpus.
The limits are the maximum docker will use for the container. The reservations are how much it will set aside for the container i.e. prevent other containers from using.
CPU is measured in percentage of a second on a single CPU. So the example you have will limit you to 0.1% of a CPU (1ms/s), which might be useful for some I/O bound tasks. '1.0' would allow one CPU to be used entirely, and higher values would allow multiple CPUs to be allocated (or reserved) to the container.
This answer has some graphs showing the effects.
Related
Upon first launch and after a period of inactivity, our custom-speech-to-text docker container takes a long time to reply (10+ seconds vs sub 1 second response times for subsequent invocations).
Here is the docker-compose lines used to run the container:
msasr:
image: mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text:2.6.0-amd64
volumes:
- ./MS_ASR_20201202:/usr/local/models
restart: always
deploy:
resources:
limits:
cpus: '4'
memory: 4G
reservations:
cpus: '4'
memory: 4G
command: Eula=accept Billing=${MS_ENDPOINT_URI} ApiKey=${MS_API_KEY}
docker-compose is run with the --compatibility flag to for the memory allocations
This problem appeared after moving to the custom-speech-to-text:2.6.0-amd64 from the cognitive-services-custom-speech-to-text:2.2.0-amd64-preview
I suppose (no proof here) that there are some elements that need to be loaded into ram and are discarded when no in use, but this has a large QOS impact later on. Is there a way to tell the instance to keep them loaded, or re-prime the system?
This symptom suggests page-outs are occurring. In the past, the following actions have very effectively solved this:
Use 8G to ensure that all locales have the resources they need without any risk of requiring swap space.
Disable swap on the host entirely if permissible.
Ensure that there are not other processes that could lead to physical memory oversubscription. In other words, ensure the sum of reservations does not exceed the physical host.
Please let me know how this goes.
Andrew
When setting up a dask cluster using Helm, there are a set of variables in the config.yaml file for customizing the number of workers, and I'm hoping for some help with the terminology. For example, if I set up an Kubernetes cluster with 16 virtual machines, 8 cores/machine and 32GB/virtual machine, I end up with 128 vCPUs and 512GB memory. If I pass the "helm ... update -f config.yaml"
worker:
name: worker
allowed-failures: 2
replicas: 48
resources:
limits:
cpu: 2
memory: 8G
requests:
cpu: 2
memory: 8G
It seems like I should be able to create 64 workers with 2 cpus each, and use all of my 512 GB RAM. (Minus the resources dedicated to the scheduler). However, in practice, the distributed client tops out at 40 workers, 80 cores and 320 GB of total RAM.
Are there best practices around setting up pods to maximize the utilization of the cluster? I know from this post that the workload comes first, in terms of the use of threads and processes per worker, but should the number of workers == the number of cores == number of pods? If so, what is the role of the cpu keyword in the above .yaml file?
My first guess is that other things are running on your nodes, and so Kubernetes doesn't feel comfortable giving everything that you've asked for. For example, Kubernetes itself takes up some memory.
I am running my application as a docker container in both Docker Swarm UCP (using compose.yml file) as well as in Mesos (using marathon.json file).
I have added the resource constraints in both the files.
compose.yml:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
marathon.json:
"cpus": 0.50,
"mem": 128.0,
"disk": 5.0
What I found out is memory is the hard limit and cpu is the soft limit. i.e. cpu limit is only for weight and priority. If mesos cpu is 1 and if two applications are running one with 0.4 cpu and the other with 0.6 cpu then app one will get 40% of the cpu cycles and app two will get 60% of the cpu cycles.
Then what is the use of limit and reservation in compose.yml file here?
Now I am trying to understand below things
How does this resource constraints work exactly?
What happens when the container exceeds these values?
reservations means that the container won't start on a node of that node doesn't have enough free resources to respect this constraint.
limits means that when a process in the container reaches that limit and tries to allocate more it will be forcefully killed.
I'm trying to limit my container so that it doesn't take up all the RAM on the host. From the Docker docs I understand that --memory limits the RAM and --memory-swap limits (RAM+swap). From the docker-compose docs it looks like the terms for those are mem_limit and memswap_limit, so I've constructed the following docker-compose file:
> cat docker-compose.yml
version: "2"
services:
stress:
image: progrium/stress
command: '-m 1 --vm-bytes 15G --vm-hang 0 --timeout 10s'
mem_limit: 1g
memswap_limit: 2g
The progrium/stress image just runs stress, which in this case spawns a single thread which requests 15GB RAM and holds on to it for 10 seconds.
I'd expect this to crash, since 15>2. (It does crash if I ask for more RAM than the host has.)
The kernel has cgroups enabled, and docker stats shows that the limit is being recognised:
> docker stats
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7624a9605c70 0.00% 1024MiB / 1GiB 99.99% 396B / 0B 172kB / 0B 2
So what's going on? How do I actually limit the container?
Update:
Watching free, it looks like the RAM usage is effectively limited (only 1GB of RAM is used) but the swap is not: the container will gradually increase swap usage until it's eaten though all of the swap and stress crashes (it takes about 20secs to get through 5GB of swap on my machine).
Update 2:
Setting mem_swappiness: 0 causes an immediate crash when requesting more memory than mem_limit, regardless of memswap_limit.
Running docker info shows WARNING: No swap limit support
According to https://docs.docker.com/engine/installation/linux/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities this is disabled by default ("Memory and swap accounting incur an overhead of about 1% of the total available memory and a 10% overall performance degradation.") You can enable it by editing the /etc/default/grub file:
Add or edit the GRUB_CMDLINE_LINUX line to add the following two key-value pairs:
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
then update GRUB with update-grub and reboot.
I used docker-compose to run a service but it crashed, I entered the container and got resource info by 'top' as below.
top - 13:43:25 up 1 day, 6:46, 0 users, load average: 1.82, 0.74, 0.52
Tasks: 3 total, 1 running, 2 sleeping, 0 stopped, 0 zombie
%Cpu(s): 32.2 us, 22.4 sy, 0.0 ni, 40.1 id, 3.0 wa, 0.0 hi, 2.3 si, 0.0 st
KiB Mem: 2047040 total, 1976928 used, 70112 free, 172 buffers
KiB Swap: 1048572 total, 1048572 used, 0 free. 14588 cached Mem
So I think my docker is out of memory.
I've tried add
mem_limit: 5g
memswap_limit: 5g
mem_reservation: 5g
into docker-compose.yml
But it seems not work. My question is, how to enlarge docker's memory limit by docker-compose.
The docker engine has a compatiblity mode which aims to make transition from compose v2 files to v3 easier. As a result of that, it is possible to partially use the swarm config (deploy) to specifiy resource limits for standard docker-compose usage:
To run in compatiblity mode just add the --compatiblity flag like this:
docker-compose --compatibility up myservice
and you can use a compose file like this:
version: '3.5'
services:
myservice:
image: postgres:12-alpine
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
If this is on a machine running Docker Desktop, then you would have to open the Docker Desktop preferences and go to Resources section to tweak how much of your host resources the Docker Engine can use.
As stated in the documentation the following fields can be used in docker-compose to control memory and cpu resources.
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
reservations:
cpus: '0.0001'
memory: 20M
Note however, by default there are no limits for the container memory usage, so setting the memory flags will unlikely help.