Docker Kubernetes (Mac) - Autoscaler unable to find metrics - docker

Question

I have installed a local instance of Kubernetes via Docker on my Mac.
Following the walkthrough on how to activate autoscaling on a deployment I have experienced an issue. The autoscaler can't read the metrics.
When I am running kubectl describe hpa the current cpu usage comes back as unknown / 50% with the warnings:
Warning FailedGetResourceMetric:
horizontal-pod-autoscaler unable to get metrics for resource cpu:
unable to fetch metrics from API: the server could not find the
requested resource (get pods.metrics.k8s.io)
Warning FailedComputeMetricsReplicas
horizontal-pod-autoscaler failed to get cpu utilization: unable to
get metrics for resource cpu: unable to fetch metrics from API: the
server could not find the requested resource (get pods.metrics.k8s.io)
I have installed the metrics-server via git clone https://github.com/kubernetes-incubator/metrics-server.gitand installed it with kubectl create -f deploy/1.8+

Answer 1

I finally got it working..
Here are the full steps I took to get things working:
Have Kubernetes running within Docker
Delete any previous instance of metrics-server from your Kubernetes instance with kubectl delete -n kube-system deployments.apps metrics-server
Clone metrics-server with git clone https://github.com/kubernetes-incubator/metrics-server.git
Edit the file deploy/1.8+/metrics-server-deployment.yaml to override the default command by adding a command section that didn't exist before. The new section will instruct metrics-server to allow for an insecure communications session (don't verify the certs involved). Do this only for Docker, and not for production deployments of metrics-server:
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.1
command:
- /metrics-server
- --kubelet-insecure-tls
Add metrics-server to your Kubernetes instance with kubectl create -f deploy/1.8+ (if errors with the .yaml, write this instead: kubectl apply -f deploy/1.8+)
Remove and add the autoscaler to your deployment again. It should now show the current cpu usage.
EDIT July 2020:
Most of the above steps hold true except the metrics-server has changed and that file does not exist anymore.
The repo now recommends installing it like this:
apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
So we can now download this file,
curl -L https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml --output components.yaml
add --kubelet-insecure-tls under args (L88) to the metrics-server deployment and run
kubectl apply -f components.yaml

Answer 2

For who are use Internal-IP here may work for you. Follow #Mr.Turtle above at step 4. add more one command.
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.3
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

Answer 3

We upgraded to AWS EKS version 1.13.7 and that's when we started having problems with HPA, It turns out on my deployment I had to specified a value for resources.requests.cpu=200m and the HPA started working for me.

Answer 4

Had same issue while using my kubernetes kubeadm lab and the updated procedure is here
https://github.com/kubernetes-sigs/metrics-server
This solved the issue:
horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Answer 5

If someone still has problems fixing this issue, this helped me fix it on minikube:
I had 2 deployments with the same label, something like this:
kind: Deployment
metadata:
name: webserver
spec:
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
---
kind: Deployment
metadata:
name: database
spec:
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
I renamed the label and matchLabels of the database (e.g. to app: db), then deleted both deployments and applied the new config - et voilà it worked. (after hours of trying to solve the problem..)
Further informations to this issue: https://github.com/kubernetes/kubernetes/issues/79365

Answer 6

I have deployed over EKS and i was facing the same issue.
muhasan#admins-MacBook-Pro devops % kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
backend-iam-deployment Deployment/backend-iam-deployment 36278272/100Mi, <unknown>/50% 1 10 1 10m
in my deployment i just specified resources and that helped me to get HPA running
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-iam-deployment
labels:
app: backend-iam-deployment
spec:
replicas: 1
selector:
matchLabels:
app: backend-iam-deployment
template:
metadata:
labels:
app: backend-iam-deployment
spec:
containers:
- name: backend-iam-deployment
image: <imagename>
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
startupProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 10
failureThreshold: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 5
imagePullSecrets:
- name: us-east-1-ecr-registry
After applying resource limit to my deployment HPA started working for me.
muhasan#admins-MacBook-Pro fx.identitymanagement % kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
backend-iam-deployment Deployment/backend-iam-deployment 23216128/100Mi, 1%/50% 1 10 1 24m