I am trying out Openshift Origin version 3 (using Docker and Kubernetes) and I can't find how to automate horizontal pod scaling.
I know vertical scaling is automated and horizontal scaling is possible (ex: oc scale test frontend --replicas=3)
What I want is additionnal replicas (pod) being created when the application load is going up and these replicas being terminated when the load is going down.
Anyone knows how to do that?
It's not there yet - the initial work is being done in Kubernetes now (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/autoscaling.md and https://github.com/GoogleCloudPlatform/kubernetes/pull/9612).
now horizontal auto scaling is supported from OC v 3.1
requirments:
1- need to start horizontal auto-scale HorizontalPodAutoscaler object
2- enable metrics through Heapster
the auto-scale uses CPU utilization currently to decide if the pods needs to be scaled-up or down. to create an auto-scale object we need a yaml file like the following:
apiVersion: extensions/v1beta1
kind: HorizontalPodAutoscaler
metadata:
name: frontend-scaler
spec:
scaleRef:
kind: DeploymentConfig
name: welcome-php
apiVersion: v1
subresource: scale
minReplicas: 1
maxReplicas: 10
cpuUtilization:
targetPercentage: 70
in the example above the target percentage is set to 70% CPU utilization and if the pod reaches this limit then the horizontal auto-scale will spin up a new pod.
the next step is to create the object:
$ oc create -f scaler.yaml
horizontalpodautoscaler "frontend-scaler" created
https://docs.openshift.com/enterprise/3.1/install_config/cluster_metrics.html#install-config-cluster-metrics
Related
Hey I have a wider problem as when I update secrets in kubernetes they are not implemented in pods unless they are ugprades/reschedules or just re-deployed; I saw the other stackoverflow post about it but noone of the solutions fit me Update kubernetes secrets doesn't update running container env vars
Also so the in-app solution of python script on pod to update its secret automatically https://medium.com/analytics-vidhya/updating-secrets-from-a-kubernetes-pod-f3c7df51770d but it seems like a long shot and I came up with solution to adding annotation to deployment manifest - and hoping it would re-schedule pods everytime a helm chart would put a new timestamp in it - it does put it but it doesn't reschedule - any thought how to force that behaviour ?
apiVersion: apps/v1
kind: Deployment
metadata:
name: xxx
namespace: xxx
labels: xxx
annotations:
lastUpdate: {{ now }}
also I dont feel like adding this patch command to ci/cd deployment, as its arbitraty and - well doesnt feel like right solution
kubectl patch deployment mydeployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"mycontainer","env":[{"name":"RESTART_","value":"'$(date +%s)'"}]}]}}}}'
didn't anyone else find better solution to re-deploy pods on changed secrets ?
Kubernetes by itself does not do rolling update of a deployment automatically when a secret is changed. So there needs to a controller which will do that for you automatically. Take a look at Reloader which is a controller that watches if some change happens in ConfigMap and/or Secret; then perform a rolling upgrade on relevant DeploymentConfig, Deployment, Daemonset and Statefulset.
Add reloader.stakater.com/auto annotation to the deployment with name xxx and have a ConfigMap called xxx-configmap or Secret called xxx-secret.
This will discover deployments/daemonsets/statefulset automatically where xxx-configmap or xxx-secret is being used either via environment variable or from volume mount. And it will perform rolling upgrade on related pods when xxx-configmap or xxx-secret are updated
apiVersion: apps/v1
kind: Deployment
metadata:
name: xxx
namespace: xxx
labels: xxx
annotations:
reloader.stakater.com/auto: "true"
On Minikube using KubeCtl, I run an image created by Docker using the following command:
kubectl run my-service --image=my-service-image:latest --port=8080 --image-pull-policy Never
But on Minukube, a different configuration is to be applied to the application. I prepared some environment variables in a deployment file and want to apply them to the images on Minikube. Is there a way to tell KubeCtl to run those images using a given deployment file or even a different way to provide the images with those values?
I tried the apply verb of KubeCtl for example, but it tries to create the pod instead of applying the configuration on it.
In minukube/kubernetes you need to apply the environment variables in the yaml file of your pod/deployment.
Here is a example of how you can configure the environment variables in a deployment spec:
apiVersion: apps/v1
kind: Pod
metadata:
name: envar-demo
labels:
purpose: demonstrate-envars
spec:
containers:
- name: envar-demo-container
image: gcr.io/google-samples/node-hello:1.0
env:
- name: DEMO_GREETING
value: "Hello from the environment"
- name: DEMO_FAREWELL
value: "Such a sweet sorrow"
Here you can find more information abour environment variables.
In this case, if you want to change any value, you need to delete the pod and apply it again. But if you use deployment all modification can be done using kubectl apply command.
After building a docker image named my-http I can create a deployment from it with
kubectl create deploy http-deployment --image=my-http
This will not pull the image because imagePullPolicy is Always.
So then run
kubectl edit deploy http-deployment
and change the imagePullPolicy to Never, then it runs.
But for automation purposes I've created a yaml to create the deployment and set the imagePullPolicy at the same time.
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-deployment
spec:
replicas: 3
selector:
matchLabels:
app: http
template:
metadata:
labels:
app: http
spec:
containers:
- name: my-http
image: my-http
imagePullPolicy: Never
ports:
- containerPort: 8080
Then apply -f and the pods start running but after a while a Crashloopbackoff starts with the message
container image my-http already present on machine
Apparently it has something to do with the container port but what to use for that port to get it running? There is no container running...
edit: the image already present is just informational, this is the last line in the pod description
Warning BackOff 7s (x8 over 91s) kubelet, minikube Back-off
restarting failed container
If you using kubernetes cluster your images only available on the nodes that you build the images.
You have to push images to container registries then the kubernetes will try to pull the image to node that will running the container.
If you want to run the container in the nodes that you build the images you have to use NodeSelector, or PodAffinity.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
Your image is probably private image which Kubernetes can't pull if you didn't specify imagePullSecrets.
This shouldn't be the problem however, because imagePullPolicy: Never would just use the image on the nodes. You can diagnose real problem by either kubectl describe pod pod_name or getting logs of the previous pod with --previous flag because newer pod may not have encountered the problem.
I've worked quite a lot with Docker in the past years, but I'm a newbie when it comes to Kubernetes. I'm starting today and I am struggling with the usefulness of the Pod concept in comparison with the way I used to do thinks with Docker swarm.
Let's say that I have a cluster with 7 powerful machines and I have the following stack:
I want three Cassandra replicas each running in a dedicated machine (3/7)
I want two Kafka replicas each running in a dedicated machine (5/7)
I want a MyProducer replica running on its own machine, receiving messages from the web and pushing them into Kafka (6/7)
I want 3 MyConsumer replicas all running in the last machine (7/7), which pull from Kafka and insert in Cassandra.
With docker swarm I used to handle container distribution with node labels, e.g. I would label three machines and Cassandra container configuration as C_HOST, 2 machines and Kafka configuration as K_HOST,... The swarm deployment would place each container correctly.
I have the following questions:
Does Kubernetes pods bring any advantage comparing to my previous approach (e.g. simplicity)? I understood that I am still required to configure labels, if so, I don't see the appeal.
What would be the correct way to configure these pods? Would it be one pod for Cassandra replicas, one pod for Kafka replicas, one pod for MyConsumer replicas and one pod for MyProducer?
Using pod anti-affinity, you can ensure that a pod is not co-located with other pods with specific labels.
So say your have a label "app" with values "cassandra", "kafka", "my-producer" and "my-consumer".
Since you want to have cassandra, kafka and my-producer on dedicated nodes all by themselves, you simply configure an anti-affinity to ALL the existing labels:
(see https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ for full schema)
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cassandra
- kafka
- my-producer
- my-consumer
This is for a "Pod" resource, so you'd define this in a deployment (where you also define how many replicas) in the pod template.
Since you want three instances of my-consumer running on the same node (or really, you don't care where they run, since by now only one node is left), you do not need to define anything about affinity or anti-affinity:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-consumer
namespace: default
labels:
app: my-consumer
spec:
selector:
matchLabels:
app: my-consumer
replicas: 3 # here you set the number of replicas that should run
template: # this is the pod template
metadata:
labels:
app: my-consumer # now this is the label you can set an anti-affinity to
spec:
containers:
- image: ${IMAGE}
name: my-consumer
# affinity:
# now here below this you'd put the affinity-settings from above
# for the other deployments
You can still use node labels and use nodeSelector parameter.
You can add node labels by using kubectl...
kubectl label nodes <node-name> <label-key>=<label-value> to add a label to the node you’ve chosen.
But more advanced way is use affinity for pod distribution...
I have saved a docker image as a tar file locally using the command,
docker save -o ./dockerImage:version.tar docker.io/image:latest-1.0
How to specify this file in my pod.yaml to use this tarball and start the pod instead of pulling / already pulled image to launch the container.
Current pod.yaml file:
apiVersion: myApp/v1
kind: myKind
metadata:
name: myPod2
spec:
baseImage: docker.io/image
version: latest-1.0
I want similar to this
apiVersion: myApp/v1
kind: myKind
metadata:
name: myPod2
spec:
baseImage: localDockerImage.tar:latest-1.0
version: latest-1.0
There's no direct way to achieve that in Kubernetes.
See the discussions here: https://github.com/kubernetes/kubernetes/issues/1668
They have finally closed that issue because of the following reasons:
Given that there are a number of ways to do this (your own cluster startup scripts, run a daemonset to side load your custom images, create VM images with images pre-loaded, run a cluster-local docker registry), and the fact that there have been no substantial updates in over two years, I'm going to close this as obsolete.