I've successfully deployed AKS with virtual nodes, where it automatically creates Azure Container Instances to support the number of pods requested, and I can manually scale up with:
kubectl scale --replicas=20 deployment.apps/blah
And sure enough I see 20 container instances get created in a special resource group and they're all running my app. When I scale it down, they disappear. This is great.
So then I try setting up autoscaling. I set limits/requests for CPU in my yaml and I say:
kubectl autoscale deployment blah --min=1 --max=20 --cpu-percent=50
But no new pods get created. To find out more I say:
kubectl describe hpa
And I get:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 3s (x12 over 2m49s) horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning FailedComputeMetricsReplicas 3s (x12 over 2m49s) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
According to these docs the metrics-server is automatically available in AKS since 1.8 (mine is 1.12, newly created).
Is this specifically a problem with virtual nodes, i.e. do they lack the ability to expose resource utilization via metrics-server in the way required by autoscale? Or is there something else I need to set up?
Metric-Server should be able to gather metrics from Virtual Kubelet (ACI)
Here's an example repo that shows that HPA with ACI is possible.
https://github.com/Azure-Samples/virtual-node-autoscale
Related
I have below resource limits in kubernates pod. where as my image starts with jvm args .
we have added resource limit less than min/max heap . java process started to run for some time and pod got killed abruptly with OutOfmemory.
How can pod start if memory specified in resource limit is 3 times less value ? Could some one help on this ?
cmd:
java -Xmx1512m -Xms1512m $LOGARGS -jar test-ms.jar
pod resourcelimits:
resources:
limits:
cpu: 300m
memory: 500Mi
requests:
cpu: 150m
memory: 300Mi
/start.sh: line 19: 7 Killed java -Xmx1512m -Xms1512m $LOGARGS -jar test-ms.jar
At 500Mi your but is evicted. If Java requires 1500, this cant work. Raise the memory value in the limit section, play with the value. Your container does not only need memory for Java.
Update to your comment:
Means that when the container reach 500Mi (500 MiB = 524.288 MB) the pod is restarted. To avoid that the pod use to much memory (ex. 10GB) because something inaspect happens. Memoryleaks for example. You limit the memory so that also other pods can run on the node. You must get what is a normal scenario for memory inside your container. As you are setting memory requirements for java, you can check if you really need them. If your cluster has metrics installed like prometheus.
https://github.com/kubernetes-sigs/metrics-server
You can install:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml
Then to analyse Cpu and memory:
kubectl top node // shows values for the nodes
kubectl top pod // shows values for the pods
Refere to the documentation if you want to use it.
While your container is running, you can also get inside and execute the normal top linux command.
I created a cluster with autopilot mode. When I try to install an app inside this cluster using helm, workloads fail with this error Does not have minimum availability. If I click on this error, I get Cannot schedule pods: Insufficient cpu and Cannot schedule pods: Insufficient memory.
If I do kubectl describe node <name> I find 0/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu.
Isn't GKE autopilot mode supposed to allocate sufficient memory and cpu?
I found where my mistake was. It had nothing to do with cpu or memomry. It was a mistake inside my yaml file (wrong host for database).
Noob here. I want to have a Dask install with a worker pool that can grow and shrink based on current demands. I followed the instructions in zero to jupyterhub to install on GKE, and then went through the install instructions for dask-kubernetes: https://kubernetes.dask.org/en/latest/.
I originally ran into some permissions issues, so I created a service account with all permissions and changed my config.yaml to use this service account. That got rid of the permissions issues, but now when I run this script, with the default worker-spec.yml, I get no workers:
cluster = KubeCluster.from_yaml('worker-spec.yml')
cluster.scale_up(4) # specify number of nodes explicitly
client = distributed.Client(cluster)
client
Cluster
Workers: 0
Cores: 0
Memory: 0 B
When I list my pods, I see a lot of workers in the pending state:
patrick_mineault#cloudshell:~ (neuron-264716)$ kubectl get pod --namespace jhub
NAME READY STATUS RESTARTS AGE
dask-jovyan-24034fcc-22qw7w 0/1 Pending 0 45m
dask-jovyan-24034fcc-25h89q 0/1 Pending 0 45m
dask-jovyan-24034fcc-2bpt25 0/1 Pending 0 45m
dask-jovyan-24034fcc-2dthg6 0/1 Pending 0 45m
dask-jovyan-25b11132-52rn6k 0/1 Pending 0 26m
...
And when I describe each pod, I see that there's an insufficient memory, cpu error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 69s (x22 over 30m) default-scheduler 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.
Do I need to manually create a new autoscaling pool in GKE or something? I only have one pool now, the one which runs jupyterlab, and that pool is already fully committed. I can't figure out what piece of configuration causes dask to figure out in which pool to put the workers.
I indeed needed to create a flexible, scalable worker pool to host the workers - there's an example of this in the Pangeo setup guide: https://github.com/pangeo-data/pangeo/blob/master/gce/setup-guide/1_create_cluster.sh. This is the relevant line:
gcloud container node-pools create worker-pool --zone=$ZONE --cluster=$CLUSTER_NAME \
--machine-type=$WORKER_MACHINE_TYPE --preemptible --num-nodes=$MIN_WORKER_NODES
I have been trying to set up an Kubernetes 1.13 AKS deployment to use HPA, and I keep running into a problem:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
dev-hpa-poc Deployment/dev-hpa-poc <unknown>/50% 1 4 2 65m
Describing the HPA gives me these events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 5m4s (x200 over 55m) horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu
Warning FailedGetResourceMetric 3s (x220 over 55m) horizontal-pod-autoscaler missing request for cpu
It doesn't appear to be able to actually retrieve CPU usage. I have specified cpu and memory usage in the deployment YAML:
resources:
requests:
cpu: 250m
memory: 128Mi
limits:
cpu: 800m
memory: 1024Mi
The system:metrics-server is running and healthy, too, so that's not it. I can monitor pod health and CPU usage from the Azure portal. Any ideas as to what I'm missing? Could this potentially be a permissions issue?
for missing request for [x] make sure that all the containers in the pod have requests declared.
In my case the reason was that other deployment haven't resource limits. You should add resources for each pod and deployment in namespace.
Adding to #nakamume's answer, make sure to double check sidecar containers.
For me, I forgot to declare requests for GCP cloud-sql-proxy sidecar which had me pulling hairs for couple of hours.
I'm having some trouble configuring my deployments on GKE.
I want to decrease the default(110) number of pods allowed on a node and I can't do it through the console. I read that the max number of pods can be set by kubelet --max-pods=<int32> however I don't know how to do this with GKE.
Any help is much appreciated.
The kubelet --max-pods=<int32> approach you mentioned is actually deprecated as per the Kubernetes documentation:
This parameter should be set via the config file specified by the
Kubelet's --config flag. See
https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
for more information.
However, in Google Kubernetes Engine, you can modify this by going to the Developer Console > Kubernetes Engine > Clusters, click on your cluster's name and click on Add Node Pool at the top. You'll be able to set a maximum number of pods per node for that new node pool.
Once done, you'll be able to deploy your services to specific node pools by following the instructions here.