I'm doing an internship focused on Docker and I have to load-balance an application which have a client, a server and a database. My goal is to dynamically scale the number of server containers according their CPU usage. For instance if the CPU usage is over 60% I add a new container on the fly to divide the CPU usage. My problem is that my simulation does not get the CPU usage higher than 20%, it is a very simple simulation where a random users register and go to random pages.
Question : How can I lower the CPU capacity of my server containers using my docker-compose file in order to artificially make the CPU go higher ? I tried to use the cpu_quota and cpu_shares instructions but it's not very documented and I don't know how it works or affects my containers.
Related
I'm trying to understand if the following is possible. I have a single docker container (container_A) running on a host machine that also hosts a non docker service (service_B). I would like to set minimum and maximum CPU & memory resource usage for container_A however I want priority to go to service_B.
If service_B is receiving a lot of requests, I want container_A's resource limits dynamically reduced. I service_B is relatively quiet, I want container_A's limits dynamically increased.
I will probably be using a simple metric such as HOST_CPU_UTILISATION - CONTAINER_UTILISATION to determine if service_B should be getting priority, but what I don't know is if I can dynamically restrict container_A when service_B is busy.
In other words can I implement a rule that states container_A can utilise CPU & Memory that is available up to 80% of total CPU & Memory?
While I'm not currently using Docker Swarm or k8, I'd consider them if they gave me what I needed.
For example, I have a 4vCPU, 8GB mem VM. At first, I ran a Nginx container on it and then used a stress test tool to continuously send requests to it and got some information like QPS, average latency. Then I ran three same Nginx containers on the VM and parallelly send the same requests above to these containers.I found that the respective QPS all decreased, and average latency all increased.
So what factors can affect different containers processing on one machine at the same time? I think the CPU and memory are enough to provide resources to these containers. What factors below the docker can affect these, firstly I think is network, but what else? And Specifically, why can network affect these QPS, average latency metrics?
When running a simple Linux container on ACI there is a huge discrepancy between the 'graphed' CPU usage in the portal compared to running 'top' in the container itself.
I can see my process running in 'top' and the cpu usage stays at around 5% and the load on the machine is below 0.10 but the portal reports around 60% usage. It's a single processor container.
Under heavier loads I have seen CPU usage of 300-400 % which feels like an issue related to the number of processors but even this does not add up and as previously stated it's a single processor container
Any thoughts ??
the ACI CPU Usage metrics seems to be in millicores, not in %. So when you see 300-400 it would be in fact .3 to .4 CPU which for a single CPU would represent 30-40%.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-monitor#available-metrics
Hoping this helps.
One of my containers is always busy, and is taking CPU away from other containers (webservers) that need to be responsive and are only active from time to time.
I would like to lower the CPU priority of the CPU-consuming container, so that whenever the other containers need the CPU, it is not clogged.
How do I do this? I have been searching the web for a while now, but I can't find the answer.
I have tried running the container with --entrypoint='nice 10 mybinary', but it turns out --entrypoint can only run binaries, not shell commands.
You can limit CPU resources on the container level. I recommend to use --cpu-shares 512 for your case.
https://docs.docker.com/config/containers/resource_constraints/:
Set this flag to a value greater or less than the default of 1024 to increase or reduce the container’s weight, and give it access to a greater or lesser proportion of the host machine’s CPU cycles. This is only enforced when CPU cycles are constrained. When plenty of CPU cycles are available, all containers use as much CPU as they need. In that way, this is a soft limit. --cpu-shares does not prevent containers from being scheduled in swarm mode. It prioritizes container CPU resources for the available CPU cycles. It does not guarantee or reserve any specific CPU access.
Setting the CPU shares is the most direct answer to your request, and typically preferred over adding capabilities to the container could be used by a malicious actor inside of the container to impact the host. The only reason I can think of to add the SYS_NICE capability to the container is if you have multiple processes inside the container and want to give different priorities to them, or need to change the priority while the container is running.
The more traditional solution to noisy neighbors is to configure each container with a limit on how much CPU and memory it is allowed to use. This is an upper bound, so realize there may be idle CPU resources if you set this low and do not have any other tasks available for the CPU to run.
The easiest way to set the limit on containers from the docker run command line is with --cpus which allows you to configure a fractional number of cores to be available to the container. Passing an option like --cpus 2.5 allows the container to use as many as 2.5 cores before the kernel scheduler throttles the process. If you had a 4 core host, that would ensure that at least 1.5 cores are always available to other processes.
Related to these limits, with Swarm Mode you can also configure a reservation for CPU (and memory). The reservation is a lower limit that Docker ensures has not been reserved for any other containers. This is used to select nodes to schedule containers, and may prevent some containers from being scheduled when there are not enough resources available, rather than scheduling so many jobs on a single node that it fails.
--cpu-shares looks like a good answer, although it's not clear to me how to verify it's working. I'm also curious what the max value is? Document doesn't say.
But, as an alternative for trusted containers, that same document also shows --cap-add=sys_nice that will allow changing process priorities within a container. i.e., if the nice or renice command is available within the container, it should work when you add the sys_nice capability. You'll only want to allow this capability for trusted containers because you don't want untrusted programs changing their own priorities willy nilly.
You can verify by inspecting the NI column for the process in question using top or ps -efl on the host.
So I have a worker docker images. I want to spin up a network of 500-50000 nodes to emulate what happens to a private blockchain such as etherium on different scales. What would be a recomendation for an opensource tool/library for such job:
a) one that would make sure that even on a low-endish (say one 40 cores node) all workers will be moved forward in time equaly (not realtime)
b) would allow (a) in a distributed setting (say 10 low-endish nodes on a single lan)
In other words I do not seek for realtime network emulation, so I can wait for 10 hours to simulate 1 minute and it would be good enough fro me. I thought about Kathara yet a problem still stands - how to make sure that say 10000 containers are given the same amount of ticks in a round-robin manner?
So how to emulate a complex network of docker workers?
I'm taking the assumption that you will run each inside of a container. To ensure each container runs with similar CPU access, you can configure CPU reservations and limits on each replica. These numbers get computed down to fractional slices of a core, so on an 8 core system, you could give each container 0.01 of a core to run upwards of 800 containers. See the compose documentation on how to set resource constraints. And with swarm mode, you could spread these replicas across multiple nodes, sharing a network.
That said, I think the advice to run shorter simulations on more hardware is good. You will find a significant portion of the time is spent in context switching between each process, possibly invalidating any measurements you want to take.
You will also encounter scalability issues with docker and the orchestration tool you choose. For example, you'll need to adjust the subnet size for any shared network which defaults to a /24 with around 253 available IP's. The docker engine itself will likely be spending a non-trivial amount of CPU time maintaining the state for all of the running containers.