Running kubernetes autoscalar - docker

Question

I have a replication controller running with the following spec:
apiVersion: v1
kind: ReplicationController
metadata:
name: owncloud-controller
spec:
replicas: 1
selector:
app: owncloud
template:
metadata:
labels:
app: owncloud
spec:
containers:
- name: owncloud
image: adimania/owncloud9-centos7
ports:
- containerPort: 80
volumeMounts:
- name: userdata
mountPath: /var/www/html/owncloud/data
resources:
requests:
cpu: 400m
volumes:
- name: userdata
hostPath:
path: /opt/data
Now I run a hpa using autoscale command.
$ kubectl autoscale rc owncloud-controller --max=5 --cpu-percent=10
I have also started heapster using kubernetes run command.
$ kubectl run heapster --image=gcr.io/google_containers/heapster:v1.0.2 --command -- /heapster --source=kubernetes:http://192.168.0.103:8080?inClusterConfig=false --sink=log
After all this, the autoscaling never kicks in. From logs, it seems that the actual CPU utilization is not getting reported.
$ kubectl describe hpa owncloud-controller
Name: owncloud-controller
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Thu, 26 May 2016 14:24:51 +0530
Reference: ReplicationController/owncloud-controller/scale
Target CPU utilization: 10%
Current CPU utilization: <unset>
Min replicas: 1
Max replicas: 5
ReplicationController pods: 1 current / 1 desired
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
44m 8s 92 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: metrics obtained for 0/1 of pods
44m 8s 92 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods
What am I missing here?

Answer 1

Most probably heapster is running in a wrong namespace ("default"). HPA expects heapster to be in "kube-system" namespace. Please, add --namespace=kube-system to kubectl run heapster command.

Answer 2

I installed hepaster under the name space "kube-system" and it worked. After running heapster, make sure it's running before you use HPA for your application.
How to run Heapster with Kubernetes cluster
I put all files here https://gitlab.com/abushoeb/kubernetes/tree/master/heapster. They are collected from the official Kubernetes Repository and made minor changes.
How to run Heapster
Go to the directory heapster where you have grafana.yaml, heapster.yaml and influxdb.yaml and run following command
$ kubectl create -f .
How to stop Heapster
Go to the same heapster directory and then run following command
$ kubectl delete -f .
How to check Heapster is running
You can access heapster metric model from the pod where heapster is running to make sure heapster is working. It can be accessed via web browser by accessing http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/. The same result can be seen by executing following command.
$ curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/
If you see the list of metrics then heapster is running correctly. You can also browse grafana dashboard to see it (find the ip of the pod where grafana is running and the access it http://grafana-pod-ip:grafana-service-port).
Full documentation of Heapster Metric Model are available here.
Also just run ($ kubectl cluster-info) and see if it shows results like this:
Kubernetes master is running at https://cluster-ip:6443
Heapster is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/heapster
kubernetes-dashboard is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
monitoring-influxdb is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
Check influxdb
You can also check influxdb if it has data in it. Install Influxdb Client on your local machine to get connected to infuxdb database.
$ influx -host <cluster-ip> -port <influxdb-service-port>
Some Sample influxdb queries
show databases
use db-name
show measurements
select value from "cpu/node_capacity"
Reference and Help
https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md
https://github.com/kubernetes/heapster/blob/master/docs/debugging.md
https://blog.kublr.com/how-to-utilize-the-heapster-influxdb-grafana-stack-in-kubernetes-for-monitoring-pods-4a553f4d36c9
http://www.dasblinkenlichten.com/installing-cadvisor-and-heapster-on-bare-metal-kubernetes/
http://blog.arungupta.me/kubernetes-monitoring-heapster-influxdb-grafana/