I am using Rancher. I have deployed a cluster with 1 master & 3 worker nodes.
All Machines are VPSes with 2 vCPU, 8GB RAM and 80GB SSD.
After the cluster was set up, the CPU reserved figure on Rancher dashboard was 15%. After metrics were enabled, I could see CPU used figures too and now CPU reserved had become 44% and CPU used was 16%. I find those figures too high. Is it normal for Kubernetes a cluster to consume this much CPU by itself?
Drilling down into the metrics, if find that the networking solution that Rancher uses - Canal - consumes almost 10% of CPU resources. Is this normal?
Rancher v2.3.0
User Interface v2.3.15
Helm v2.10.0-rancher12
Machine v0.15.0-rancher12-1
This "issue" is known for some time now and it affects smaller clusters. Kuberenetes is very CPU hungry relative to small clusters and this is currently by design. I have found multiple threads reporting this for different kind of setups. Here is an example.
So the short answer is: yes, Kubernetes setup consumes these amounts of CPU when used with relative small clusters.
I hope it helps.
Related
For example, I have a 4vCPU, 8GB mem VM. At first, I ran a Nginx container on it and then used a stress test tool to continuously send requests to it and got some information like QPS, average latency. Then I ran three same Nginx containers on the VM and parallelly send the same requests above to these containers.I found that the respective QPS all decreased, and average latency all increased.
So what factors can affect different containers processing on one machine at the same time? I think the CPU and memory are enough to provide resources to these containers. What factors below the docker can affect these, firstly I think is network, but what else? And Specifically, why can network affect these QPS, average latency metrics?
When running a simple Linux container on ACI there is a huge discrepancy between the 'graphed' CPU usage in the portal compared to running 'top' in the container itself.
I can see my process running in 'top' and the cpu usage stays at around 5% and the load on the machine is below 0.10 but the portal reports around 60% usage. It's a single processor container.
Under heavier loads I have seen CPU usage of 300-400 % which feels like an issue related to the number of processors but even this does not add up and as previously stated it's a single processor container
Any thoughts ??
the ACI CPU Usage metrics seems to be in millicores, not in %. So when you see 300-400 it would be in fact .3 to .4 CPU which for a single CPU would represent 30-40%.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-monitor#available-metrics
Hoping this helps.
I understand the use of replicas in Docker Swarm mode. It is mainly to eliminate points of failure and reduce the amount of downtime. It is well explained in this post.
Since having more replicas is more useful for a system as a whole, why don't companies just initialise as many replicas as possible e.g 1000 replicas for a docker service? I can imagine a large corporation running a back-end system may face multiple points of failures at any given time and they would benefit from having more instances of the particular service.
I would like to know how many replicas are considered TOO MUCH and what are the factors affecting the performance of a Docker Swarm?
I can think of hardware overhead being a limiting factor.
Lets say your running Rails app. Each instance required 128Mb of RAM, and 10% CPU usage. 9 instances is a touch over 1Gb of memory and 1 entire CPU.
While that does not sounds like a lot, image an organization has 100 + teams each with 3,4,5 applications each. The hardware requirements to operation an application at acceptable levels quickly ramp up.
Then there is network chatter. 10MB/s is typical in big org/corp settings. While a heartbeat check for a couple instances is barely noticeable, heartbeat on 100's of instances could jam up the network.
At the end of the day it comes down the constraints. What are the boundaries within the software, hardware, environment, budgetary, and support systems? It is often hard to imagine the pressures present when (technical) decisions are made.
Hie,
I have recently come across virtualization called containers.
this is regarding auto-scaling.
I have read that when current container is out of resources then it will add a new container to the host. How does adding new container saves in resources?
can any explain the scale of
scenario 1: container running all of the resources (leaving resources for host os)
vs
Scenario 2: the scale of running two containers running in the same
host (where the container is using half of the resources consumed in
the previous case)
if the scale is more in scenario 2 then can anyone explain how scale has increased by having two containers even through total resources are same?
Consider following scenario regarding workload of an application (Only CPU and memory are considered here for example) -
Normal Workload
It requires 4 Cores , 16 GB memory.
Maximum Workload
It requires 8 Cores, 32 GB Memory.
Assuming that the burst of activities (max workload) happens only 2 hours per day.
Case 1 - When application is not containerized
The application has to reserve 8 Cores and 32 GB of memory for it's working so that it could handle max workload with expected performance.
But the 4 Cores and 16 GB of memory is going to be wasted for 22 hours in a day.
Case 2 - When application is containerized
Lets assume that a container of 4 Cores and 16 GB memory is spawned. Then remaining resources, 4 Cores and 16 GB memory are available for other applications in cluster and another container of same configuration will be spawned for this application only for 2 hours in a day when max workload
is required.
Therefore the resources in cluster are used optimally when applications are containerized.
What if single machine does not have all the resources required for the application ?
In such cases, if application is containerized then containers/resources can be allocated from multiple machines in the cluster.
Increase fault tolerance
If application is running on single machine and the machine goes down then the whole application become unavailable. But in case of containerized application
running on different machines in cluster. If a machine fails then only few containers will not be available.
Regarding your question, if the application's workload is going to be uniform throughout the lifetime then there is no benefit in breaking the application in smaller containers in terms of scale. But you may still consider containerizing applications for other benefits. In terms of scale, an application is containerized only when there is varying workload or more workload is anticipated in future.
how scale has increased by having two containers even through total resources are same?
It usually doesn't. If you have a single threaded application, and you have a multi core host, then scaling multiple instances of the container in the same host will give you access to more cores by that application. This could also help with a multi threaded application if it's limited by internal resource contention and not using all of the cores. But for most multi threaded processes, scaling the application up on a single host will not improve performance.
What scaling does help with in an multi node environment is allowing the application to run in other hosts in the cluster that are not fully utilized. This is the horizontal scaling that most target with 12 factor apps. It allows apps deployed to cloud environments to scale out with more replicas/shards by adding more nodes rather than trying to find more resources for a single large node.
I'm doing an internship focused on Docker and I have to load-balance an application which have a client, a server and a database. My goal is to dynamically scale the number of server containers according their CPU usage. For instance if the CPU usage is over 60% I add a new container on the fly to divide the CPU usage. My problem is that my simulation does not get the CPU usage higher than 20%, it is a very simple simulation where a random users register and go to random pages.
Question : How can I lower the CPU capacity of my server containers using my docker-compose file in order to artificially make the CPU go higher ? I tried to use the cpu_quota and cpu_shares instructions but it's not very documented and I don't know how it works or affects my containers.