My goal is to have a pod with a working Kubectl binary inside.
Unfortunatly every kubectl image from docker hub I booted using basic yaml resulted in CrashLoopbackOff or else.
Has anyone got some yaml (deployment, pod, etc) that would get me my kubectl ?
I tried a bunch of images with this basic yaml there:
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubectl-demo
labels:
app: deploy
role: backend
spec:
replicas: 1
selector:
matchLabels:
app: deploy
role: backend
template:
metadata:
labels:
app: deploy
role: backend
spec:
containers:
- name: kubectl-demo
image: <SOME_IMAGE>
ports:
- containerPort: 80
Thx
Or, you can do this. It works in my context, with kubernetes on VMs, where I know where is kubeconfig file. You would need to make the necessary changes, to make it work in your environment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubectl
spec:
replicas: 1
selector:
matchLabels:
role: kubectl
template:
metadata:
labels:
role: kubectl
spec:
containers:
- image: viejo/kubectl
name: kubelet
tty: true
securityContext:
privileged: true
volumeMounts:
- name: kube-config
mountPath: /root/.kube/
volumes:
- name: kube-config
hostPath:
path: /home/$USER/.kube/
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
This is the result:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
kubectl-cb8bfc6dd-nv6ht 1/1 Running 0 70s
$ kubectl exec kubectl-cb8bfc6dd-nv6ht -- kubectl get no
NAME STATUS ROLES AGE VERSION
kubernetes-1-17-master Ready master 16h v1.17.3
kubernetes-1-17-worker Ready <none> 16h v1.17.3
As Suren already explained in the comments that kubectl is not a daemon so kubectl will run, exit and cause the container to restart.
There are a couple of workarounds for this. One of these is to use sleep command with infinity argument. This would keep the Pod alive, prevent it from restarting and allow you to exec into it.
Here`s an example how to do that:
spec:
containers:
- image: bitnami/kubectl
command:
- sleep
- "infinity"
name: kctl
Let me know if this helps.
Related
I have deployed a service on Knative. I iterated on the service code/Docker image and I try to redeploy it at the same address. I proceeded as follow:
Pushed the new Docker image on our private Docker repo
Updated the service YAML file to point to the new Docker image (see YAML below)
Delete the service with the command: kubectl -n myspacename delete -f myservicename.yaml
Recreate the service with the command: kubectl -n myspacename apply -f myservicename.yaml
During the deployment, the service shows READY = Unknown and REASON = RevisionMissing, and after a while, READY = False and REASON = ProgressDeadlineExceeded. When looking at the logs of the pod with the following command kubectl -n myspacename logs revision.serving.knative.dev/myservicename-00001, I get the message:
no kind "Revision" is registered for version "serving.knative.dev/v1" in scheme "pkg/scheme/scheme.go:28"
Here is the YAML file of the service:
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: myservicename
namespace: myspacename
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/metric: concurrency
autoscaling.knative.dev/target: '1'
autoscaling.knative.dev/minScale: '0'
autoscaling.knative.dev/maxScale: '5'
autoscaling.knative.dev/scaleDownDelay: 60s
autoscaling.knative.dev/window: 600s
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
volumes:
- name: nfs-volume
persistentVolumeClaim:
claimName: myspacename-models-pvc
imagePullSecrets:
- name: myrobotaccount-pull-secret
containers:
- name: myservicename
image: quay.company.com/project/myservicename:0.4.0
ports:
- containerPort: 5000
name: user-port
protocol: TCP
resources:
limits:
cpu: "4"
memory: 36Gi
nvidia.com/gpu: 1
requests:
cpu: "2"
memory: 32Gi
volumeMounts:
- name: nfs-volume
mountPath: /tmp/static/
securityContext:
privileged: true
env:
- name: CLOUD_STORAGE_PASSWORD
valueFrom:
secretKeyRef:
name: myservicename-cloud-storage-password
key: key
envFrom:
- configMapRef:
name: myservicename-config
The protocol I followed above is correct, the problem was because of a bug in the code of the Docker image that Knative is serving. I was able to troubleshoot the issue by looking at the logs of the pods as follow:
First run the following command to get the pod name: kubectl -n myspacename get pods. Example of pod name = myservicename-00001-deployment-56595b764f-dl7x6
Then get the logs of the pod with the following command: kubectl -n myspacename logs myservicename-00001-deployment-56595b764f-dl7x6
I am trying to deploy few pods on GKE cluster created using image "Ubuntu with docker" and they are giving the error below. I did not find any solution on the internet. Any help would be greatly appreciated.
Error response from daemon: OCI runtime create failed: invalid mount {Destination:[/sys/fs/cgroup Type:bind Source:/var/lib/docker/volumes/d9e3b871f4cc210e3dba6471f326dcbf7b404daad7906ed9fc669e207c093ec2/_data Options:[rbind]}: mount destination [/sys/fs/cgroup not absolute: unknown
The spec file
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
diamanti.com/app: armada
diamanti.com/control-plane: 'true'
name: armada
namespace: diamanti-system
spec:
selector:
matchLabels:
diamanti.com/app: armada
diamanti.com/control-plane: 'true'
template:
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
labels:
diamanti.com/app: armada
diamanti.com/control-plane: 'true'
spec:
containers:
- envFrom:
- configMapRef:
name: armada-config
image: 'diamanti/armada:v3.3.1-197'
name: armada
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
spektra.diamanti.io/node: "true"
hostNetwork: true
restartPolicy: Always
schedulerName: default-scheduler
serviceAccount: diamanti-node-runner
serviceAccountName: diamanti-node-runner
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
The serviceaccount diamanti-node-runner is bound to cluster-admin role.
As Kubernetes is removing the support for docker runtime you can use the other container runtime. Use their default, it works fine. You do not need to change anything at your end related to docker images.
There is a kubernetes cluster with 100 nodes, I have to clean the specific images manually, I know the kubelet garbage collect may help, but it isn't applied in my case.
After browsing the internet , I found a solution - docker in docker, to solve my problem.
I just wanna remove the image in each node one time, is there any way to run a job in each node one time?
I checked the kubernetes labels and podaffinity, but still no ideas, any body could help?
Also, I tried to use daemonset to solve the problem, but turns out that it can only remove the image for a part of nodes instead of all nodes, I don't what might be the problem...
here is the daemonset example:
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: test-ds
labels:
k8s-app: test
spec:
selector:
matchLabels:
k8s-app: test
template:
metadata:
labels:
k8s-app: test
spec:
containers:
- name: test
env:
- name: DELETE_IMAGE_NAME
value: "nginx"
image: busybox
command: ['sh', '-c', 'curl --unix-socket /var/run/docker.sock -X DELETE http://localhost/v1.39/images/$(DELETE_IMAGE_NAME)']
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-sock-volume
ports:
- containerPort: 80
volumes:
- name: docker-sock-volume
hostPath:
# location on host
path: /var/run/docker.sock
If you want to run you job on single specific Node you can us the Nodeselector in POD spec
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: test
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
nodeSelector:
name: node3
daemon set ideally should resolve your issues, as it creates the PODs on each available Node in the cluster.
You can read more about the affinity at here : https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
nodeSelector provides a very simple way to constrain pods to nodes
with particular labels. The affinity/anti-affinity feature, greatly
expands the types of constraints you can express. The key enhancements
are
The affinity/anti-affinity language is more expressive. The language
offers more matching rules besides exact matches created with a
logical AND operation;
You can use the Affinity in Job YAML something like
apiVersion: v1
kind: Pod
metadata:
name: with-node-affinity
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
containers:
- name: with-node-affinity
image: k8s.gcr.io/pause:2.0
Update
Now if you have issue with the Deamon affinity with the Job is also useless, as Job will create the Single POD which will get schedule to Single node as per affinity. Either create 100 job with different affinity rules or you use Deployment + Affinity to schedule the Replicas on different nodes.
We will create one Deployment with POD affinity and make sure, multiple PODs of a single deployment won't get scheduled on one Node.
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
labels:
app: test
spec:
replicas: 100
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: <Image>
ports:
- containerPort: 80
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- test
topologyKey: "kubernetes.io/hostname"
Try using this deployment template and replace your image here. You can reduce replicas first to 10 instead of 100 to check it's spreading PODs or not.
Read more at : https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#an-example-of-a-pod-that-uses-pod-affinity
Extra :
You can also write and use your custom CRD : https://github.com/darkowlzz/daemonset-job which will behave as daemon set and job
I'm trying to deploy a MicroServices system on my local machine using Skaffold.
ingress-srv.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: ticketing.dot
http:
paths:
- path: /api/users/?(.*)
backend:
serviceName: auth-srv
servicePort: 3000
auth-depl.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: ****MYDOCKERID****/auth
env:
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: jwt-secret
key: JWT_KEY
---
apiVersion: v1
kind: Service
metadata:
name: auth-srv
spec:
selector:
app: auth
ports:
- name: auth
protocol: TCP
port: 3000
targetPort: 3000
auth-mongo-depl.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-mongo-depl
spec:
replicas: 1
selector:
matchLabels:
app: auth-mongo
template:
metadata:
labels:
app: auth-mongo
spec:
containers:
- name: auth-mongo
image: mongo
---
apiVersion: v1
kind: Service
metadata:
name: auth-mongo-srv
spec:
selector:
app: auth-mongo
ports:
- name: db
protocol: TCP
port: 27017
targetPort: 27017
I've followed through the guidelines in the manual:
https://kubernetes.github.io/ingress-nginx/deploy/
and hit:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.0/deploy/static/provider/cloud/deploy.yaml
However Skaffold keep terminating the deployment:
[34mListing files to watch...[0m
[34m - ****MYDOCKERID****/auth
[0m[34mGenerating tags...[0m
[34m - ****MYDOCKERID****/auth -> [0m****MYDOCKERID****/auth:683e8db
[34mChecking cache...[0m
[34m - ****MYDOCKERID****/auth: [0m[32mFound Locally[0m
[34mTags used in deployment:[0m
[34m - ****MYDOCKERID****/auth -> [0m****MYDOCKERID****/auth:3c4bb66ff693320b5fac3fde91906768f8b54b968813b226822d057d1dd3a995
[34mStarting deploy...[0m
- deployment.apps/auth-depl created
- service/auth-srv created
- deployment.apps/auth-mongo-depl created
- service/auth-mongo-srv created
- ingress.extensions/ingress-service created
[34mWaiting for deployments to stabilize...[0m
- deployment/auth-depl:
- deployment/auth-mongo-depl:
- deployment/auth-depl: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/auth-mongo-depl: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/auth-mongo-depl is ready. [1/2 deployment(s) still pending]
- deployment/auth-depl failed. Error: could not stabilize within 2m0s: context deadline exceeded.
[34mCleaning up...[0m
- deployment.apps "auth-depl" deleted
- service "auth-srv" deleted
- deployment.apps "auth-mongo-depl" deleted
- service "auth-mongo-srv" deleted
- ingress.extensions "ingress-service" deleted
[31mexiting dev mode because first deploy failed: 1/2 deployment(s) failed[0m
How can we fix this annoying issue?
EDIT 9:44 AM ISRAEL TIME :
C:\Development-T410\Micro Services - JAN>kubectl get pods
NAME READY STATUS RESTARTS AGE
auth-depl-645bbf7b9d-llp2q 0/1 CreateContainerConfigError 0 115s
auth-depl-c6c765d7c-7wvcg 0/1 CreateContainerConfigError 0 28m
auth-mongo-depl-6b594c4847-4kzzt 1/1 Running 0 115s
client-depl-5888f95b59-vznh6 1/1 Running 0 114s
nats-depl-7dfccdf5-874vm 1/1 Running 0 114s
orders-depl-74f4d48559-cbwlp 0/1 CreateContainerConfigError 0 114s
orders-depl-78fc845b4-9tfml 0/1 CreateContainerConfigError 0 28m
orders-mongo-depl-688676d675-lrvhp 1/1 Running 0 113s
tickets-depl-7cc7ddbbff-z9pvc 0/1 CreateContainerConfigError 0 113s
tickets-depl-8574fc8f9b-tm6p4 0/1 CreateContainerConfigError 0 28m
tickets-mongo-depl-b95f45947-hf6wq 1/1 Running 0 113s
C:\Development-T410\Micro Services>kubectl logs auth-depl-c6c765d7c-7wvcg
Error from server (BadRequest): container "auth" in pod "auth-depl-c6c765d7c-7wvcg" is waiting to start: CreateContainerConfigError
Looks look your auth-depl deployment is failing. Possibly the container is crashing or erroring out. To debug you can see the pod logs
$ kubectl logs auth-depl-xxxxxxxxxx-xxxxx
Make sure you run skaffold with the --cleanup=false option so that you can debug. For example,
$ skaffold dev --cleanup=false
Update:
Based on the logs it looks like it's an issue with your Kubernetes Secret and how it's defined, possibly the format or YAML format. This answer sheds some details on what the problem may be: Pod status as `CreateContainerConfigError` in Minikube cluster
You should add the environment variable for the mongo image in your deployment file
env:
- name: MONGO_INITDB_ROOT_USERNAME
value: root
- name: MONGO_INITDB_ROOT_PASSWORD
value: "rootuser"
I have my controller.yaml that looks like this:
apiVersion: v1
kind: ReplicationController
metadata:
name: hmrcaction
labels:
name: hmrcaction
spec:
replicas: 1
selector:
name: hmrcaction
template:
metadata:
labels:
name: hmrcaction
version: 0.1.4
spec:
containers:
- name: hmrcaction
image: ccc-docker-docker-release.someartifactory.com/hmrcaction:0.1.4
ports:
- containerPort: 9000
imagePullSecrets:
- name: fff-artifactory
and service yaml that looks like this:
apiVersion: v1
kind: Service
metadata:
name: hmrcaction
labels:
name: hmrcaction
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
type: LoadBalancer
ports:
# the port that this service should serve on
- port: 9000
selector:
name: hmrcaction
and I have a kubernetes cluster, so I wanted to use this rc to upload my docker to the cluster and I did it like this:
kubectl create -f controller.yaml
but I get some weird status, when I run the command kubectl get pods I get:
NAME READY STATUS RESTARTS AGE
hmrcaction-k9bb6 0/1 ImagePullBackOff 0 40s
what is this?? before the status was ErrImagePull...
please help :)
thanks!
kubectl describe pods -l name=hmrcaction should give you more useful information.