I'm following this Link to setup Jenkins on Kubernetes cluster.
The environment information is mentioned below,
Environment:-
On-Premise Physical Server
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-server Ready master 2d23h v1.19.16
node-server1 Ready worker1 2d23h v1.19.16
node-server2 Ready worker2 2d23h v1.19.16
node-server3 Ready worker3 2d23h v1.19.16
I have below yaml files.
deploy-jenkins.yaml
sa-jenkins.yaml
service-jenkins.yaml
volume-jenkins.yaml
PersistentVolume i want to use my master server local path, So in the volume-jenkins.yaml file I have updated path and values as below.
local:
path: /home/linux-user/kubernetes/jenkins
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- master-server
When i apply the yaml files, My jenkins pod remain in pending status always.
Jenkins Pod status:-
# kubectl get pods -n jenkins
NAME READY STATUS RESTARTS AGE
jenkins-69b8564b9f-gm48n 0/1 Pending 0 102m
Jenkins Pod describe Status:-
# kubectl describe pod jenkins-69b8564b9f-gm48n -n jenkins
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m45s (x68 over 104m) default-scheduler 0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.
PV describe details:-
# kubectl describe pv jenkins-pv -n jenkins
Name: jenkins-pv
Labels: type=local
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: jenkins/jenkins-pvc
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [master-server]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /home/linux-user/kubernetes/jenkins
Events: <none>
What is wrong with my yaml files? and let me know the way to solve the node conflict issue. Thanks in advance.
...i want to use my master server local path
Add nodeSelector and tolerations to your deployment spec:
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
nodeSelector:
node-role.kubernetes.io/master: ""
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
containers:
- name: jenkins
...
Related
I wrote a simple node.js app that listens on a port and returns HTML. I can docker run the node.js app and, with port forwarding in place, hit it happily.
+ docker run -p 7081:7081 split-server
Now I want to run the app in kubernetes. I am on a mac and set up minikube and virtual box. I also set up a local docker registry for my local app, using instructions found here.
It doesn't work no matter what combination of things I try. Pending. The describe is below. I think I'm close, but I just can't get useful debugging output from kubectl:
+ kubectl describe pod split-server
Name: split-server-68fc6cdcd-gpk5m
Namespace: default
Priority: 0
Node: <none>
Labels: app=split-server
pod-template-hash=68fc6cdcd
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/split-server-68fc6cdcd
Containers:
app:
Image: split-server:latest
Port: 7081/TCP
Host Port: 0/TCP
Environment:
SPLIT_API_KEY: <API KEY>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f8lzd (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-f8lzd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m27s (x3 over 13m) default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 1 node(s) were unschedulable. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
My YAML is...
apiVersion: v1
kind: Service
metadata:
name: split-server
spec:
selector:
app: split-server
ports:
- port: 7081
targetPort: 7081
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: split-server
spec:
replicas: 1
selector:
matchLabels:
app: split-server
template:
metadata:
labels:
app: split-server
spec:
containers:
- name: app
image: 192.168.4.26:5000/split-server:latest
#image: split-server:latest
ports:
- containerPort: 7081
env:
- name: SPLIT_API_KEY
value: <API KEY>
imagePullPolicy: Always
And here is what docker has for its list of images:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
split-server latest d2caa2d0c693 45 minutes ago 1.01GB
192.168.4.26:5000/local/split-server latest d2caa2d0c693 45 minutes ago 1.01GB
Where should I be hunting? What tools am I missing? kubectl logs comes back empty every time... should have a single line of logging if the app had come up properly.
The minikube node is marked as unschedulable for some reason (manually or there is a problem), you can try to remove the taint:
kubectl taint nodes --all node.kubernetes.io/unschedulable-
or add a toleration on your pod:
apiVersion: v1
kind: Pod
metadata:
name: ...
...
spec:
containers:
- name: ...
...
tolerations:
- key: "node.kubernetes.io/unschedulable"
operator: "Exists"
effect: "NoSchedule"
Way to troubleshoot a pending pod is by looking at the events that you get when you describe the pod. In your case master node is marked unschedulable hence you are facing the issue.
The command to fix would be like what Hussein said also refer this page to get an idea how to troubleshoot a pending pod:
Troubleshooting pending pods
Environment information:
Computer detail: One master node and four slave nodes. All are CentOS Linux release 7.8.2003 (Core).
Kubernetes version: v1.18.0.
Zero to JupyterHub version: 0.9.0.
Helm version: v2.11.0
I recently try to deploy an online code environment(like Google Colab) in new lab servers via Zero to JupyterHub. Unfortunately, I failed to deploy Persistent Volume(PV) for JupyterHub and I got a failure message such below:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4s (x27 over 35m) default-scheduler running "VolumeBinding" filter plugin for pod "hub-7b9cbbcf59-747jl": pod has unbound immediate PersistentVolumeClaims
I followed the installing process by the tutorial of JupyterHub, and I was used Helm to install JupyterHub on k8s. That config file such below:
config.yaml
proxy:
secretToken: "2fdeb3679d666277bdb1c93102a08f5b894774ba796e60af7957cb5677f40706"
singleuser:
storage:
dynamic:
storageClass: local-storage
Here, I was config a local-storage for JupyterHub, the local-storage was observed k8s: Link. And its yaml file
such like that:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
Then I use kubectl get storageclass to check it work, I got the message below:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 64m
So, I thought I deployed a storage for JupyterHub, but I so naive. I am so disappointed about that because my other Pods(JupyterHub) are all running. And I have been search some solutions so long, but also failed.
So now, my problems are:
What is the true way to solve the PV problems? (Better using local storage.)
Is the local storage way will using other nodes disk not only master?
In fact, my lab had a could storage service, so if Q2 answer is No, and how I using my lab could storage service to deploy PV?
I had been addressed above problem with #Arghya Sadhu's solution. But now, I got a new problem is the Pod hub-db-dir also pending, it result my service proxy-public pending.
The description of hub-db-dir such below:
Name: hub-7b9cbbcf59-jv49z
Namespace: jhub
Priority: 0
Node: <none>
Labels: app=jupyterhub
component=hub
hub.jupyter.org/network-access-proxy-api=true
hub.jupyter.org/network-access-proxy-http=true
hub.jupyter.org/network-access-singleuser=true
pod-template-hash=7b9cbbcf59
release=jhub
Annotations: checksum/config-map: c20a64c7c9475201046ac620b057f0fa65ad6928744f7d265bc8705c959bce2e
checksum/secret: 1beaebb110d06103988476ec8a3117eee58d97e7dbc70c115c20048ea04e79a4
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/hub-7b9cbbcf59
Containers:
hub:
Image: jupyterhub/k8s-hub:0.9.0
Port: 8081/TCP
Host Port: 0/TCP
Command:
jupyterhub
--config
/etc/jupyterhub/jupyterhub_config.py
--upgrade-db
Requests:
cpu: 200m
memory: 512Mi
Readiness: http-get http://:hub/hub/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
PYTHONUNBUFFERED: 1
HELM_RELEASE_NAME: jhub
POD_NAMESPACE: jhub (v1:metadata.namespace)
CONFIGPROXY_AUTH_TOKEN: <set to the key 'proxy.token' in secret 'hub-secret'> Optional: false
Mounts:
/etc/jupyterhub/config/ from config (rw)
/etc/jupyterhub/cull_idle_servers.py from config (rw,path="cull_idle_servers.py")
/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
/etc/jupyterhub/secret/ from secret (rw)
/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
/srv/jupyterhub from hub-db-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from hub-token-vlgwz (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: hub-config
Optional: false
secret:
Type: Secret (a volume populated by a Secret)
SecretName: hub-secret
Optional: false
hub-db-dir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: hub-db-dir
ReadOnly: false
hub-token-vlgwz:
Type: Secret (a volume populated by a Secret)
SecretName: hub-token-vlgwz
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 61s (x43 over 56m) default-scheduler 0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 4 node(s) didn't find available persistent volumes to bind.
The information with kubectl get pv,pvc,sc.
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/hub-db-dir Pending local-storage 162m
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/local-storage (default) kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 8h
So, how to fix it?
In addition to #Arghya Sadhu answer, in order to make it work using local storage you have to create a PersistentVolume manually.
For example:
apiVersion: v1
kind: PersistentVolume
metadata:
name: hub-db-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: <path_to_local_volume>
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <name_of_the_node>
Then you can deploy the chart:
helm upgrade --install $RELEASE jupyterhub/jupyterhub \
--namespace $NAMESPACE \
--version=0.9.0 \
--values config.yaml
The config.yaml file can be left as is:
proxy:
secretToken: "<token>"
singleuser:
storage:
dynamic:
storageClass: local-storage
I think you need to make local-storage as default storage class
kubectl patch storageclass local-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Local storage will use the local disk storage of the node where the pod get scheduled.
Hard to tell without more details. You can either create PV manually or use a storage class which does dynamic volume provisioning.
I am using the YAML file to deploy the container on Kubernetes with some replication factor on a hosted machine.
YAML File
apiVersion: apps/v1
kind: Deployment
metadata:
name: mojo-deployment
labels:
app: mojo
spec:
selector:
matchLabels:
app: mojo
replicas: 3
template:
metadata:
labels:
app: mojo
spec:
containers:
- name: mojo
image: mojo:1.0.1
ports:
- containerPort: 9000
---
#Services Info
apiVersion: v1
kind: Service
metadata:
name: mojo-services
spec:
selector:
app: mojo
ports:
- protocol: TCP
port: 80
targetPort: 9376
---
#Ingress Configuration
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: mojo-ingress
annotations:
kubernetes.io/ingress.class: mojo
spec:
backend:
serviceName: mojo-services
servicePort: 80
Steps:
Build Docker image using `docker build -t mojo:1.0 .
docker image ls show me an image id.
Skipping docker build command to deploy image on container. Do I need to do it? or kubectl service will take care of it.
Run kubectl apply -f Prod.yaml. It shows
deployment.apps/mojo-deployment created
service/mojo-services created
ingress.networking.k8s.io/mojo-ingress created
kubectl get service returns
kubectl get pod returns
kubectl get deployment returns
Questions?
Do I need to build the container before deploying YAML file? I tried it but still kubernetes not running.
Why all pods are showing Pending status.
Deployment is also showing pending status.
Though I am trying to access the Ingress with :80 and cannot access it.
Edit
pod description
Name: mojo-deployment-6665bdc557-s57m7
Namespace: default
Priority: 0
Node: <none>
Labels: app=mojo
pod-template-hash=6665bdc557
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/mojo-deployment-6665bdc557
Containers:
mojo:
Image: mojo:1.0
Port: 9000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tjx6p
(ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-tjx6p:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tjx6p
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From
Message
---- ------ ---- ---- -------
Warning FailedScheduling 70s (x45 over 67m) default-scheduler 0/1
nodes are available: 1 node(s) were unschedulable.
Edit 2
After removing the taint from the master node.
1. kubectl get node returns
kubectl get pod returns
kubectl describe node : https://gist.github.com/amixpal/333bffd6ab91def749267f30d4ffb079
If you have only one node (master) , then usually a Taint will be added to it which will make master node unschedulable. Remove taint from the master (and all other nodes, if there is more than one) using below.
kubectl taint nodes --all node-role.kubernetes.io/master-
Edit: Based on the node describe output, the CNI not ready.
Please make sure all CNI related Pods are running and healthy
Your container manifest should include downloadable docker image or k8s node should already contain docker image:
containers:
- name: mojo
image: mojo:1.0.1
ports:
- containerPort: 9000
Please answer: How your mojo:1.0.1 docker image appears on kubernetes nodes?
All pods wait to image be available.
Deployment wait for all pods will be in status Running.
K8s services make ingress be available after deployment be ready.
I want to use ceph rbd with kubernetes.
I have a kubernetes 1.9.2 and ceph 12.2.5 cluster and on my k8s nodes I have installed ceph-common pag.
[root#docker09 manifest]# ceph auth get-key client.admin|base64
QVFEcmxmcGFmZXlZQ2hBQVFJWkExR0pXcS9RcXV4QmgvV3ZFWkE9PQ==
[root#docker09 manifest]# cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
data:
key: QVFEcmxmcGFmZXlZQ2hBQVFJWkExR0pXcS9RcXV4QmgvV3ZFWkE9PQ==
kubectl create -f ceph-secret.yaml
Then:
[root#docker09 manifest]# cat ceph-pv.yaml |grep -v "#"
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- 10.211.121.61:6789
- 10.211.121.62:6789
- 10.211.121.63:6789
pool: rbd
image: ceph-image
user: admin
secretRef:
name: ceph-secret
fsType: ext4
readOnly: false
persistentVolumeReclaimPolicy: Recycle
[root#docker09 manifest]# rbd info ceph-image
rbd image 'ceph-image':
size 2048 MB in 512 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.341d374b0dc51
format: 2
features: layering
flags:
create_timestamp: Fri Jun 15 15:58:04 2018
[root#docker09 manifest]# cat task-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
[root#docker09 manifest]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv/ceph-pv 2Gi RWO Recycle Bound default/ceph-claim 54m
pv/host 10Gi RWO Retain Bound default/hostv 24d
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc/ceph-claim Bound ceph-pv 2Gi RWO 53m
pvc/hostv Bound host 10Gi RWO 24d
I create a pod use this pvc .
[root#docker09 manifest]# cat ceph-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: ceph-pod2
spec:
containers:
- name: ceph-busybox
image: busybox
command: ["sleep", "60000"]
volumeMounts:
- name: ceph-vol1
mountPath: /usr/share/busybox
readOnly: false
volumes:
- name: ceph-vol1
persistentVolumeClaim:
claimName: ceph-claim
[root#docker09 manifest]# kubectl get pod ceph-pod2 -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ceph-pod2 0/1 ContainerCreating 0 14m <none> docker10
The pod is still in ContainerCreating status.
[root#docker09 manifest]# kubectl describe pod ceph-pod2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned ceph-pod2 to docker10
Normal SuccessfulMountVolume 15m kubelet, docker10 MountVolume.SetUp succeeded for volume "default-token-85rc7"
Warning FailedMount 1m (x6 over 12m) kubelet, docker10 Unable to mount volumes for pod "ceph-pod2_default(56af9345-7073-11e8-aeb6-1c98ec29cbec)": timeout expired waiting for volumes to attach/mount for pod "default"/"ceph-pod2". list of unattached/unmounted volumes=[ceph-vol1]
I don't know why this happening, need your help... Best regards.
There's no need to reinvent a wheel here. There's already project called ROOK, which deploys ceph on kubernetes and it's super easy to run.
https://rook.io/
rbd -v (included in ceph-common) should return the same version as your cluster. You should also check the messages of kubelet.
I've deployed a private registry and can pull from it with docker pull x.x.x/name. The thing is that I can't make Kubernetes pull from that repository. I think I've followed all the answers on other topics, but they don't seem to do the trick.
.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: private-image-test-1
spec:
containers:
- name: uses-private-image
image: x.x.x/nginx_1
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
imagePullSecrets:
- name: registrypullsecret
kubectl get pods:
NAME READY STATUS RESTARTS AGE
private-image-test-1 0/1 Image: x.x.x/nginx_1 is ready, container is creating 0 4m
kubectl describe pods private-image-test-1
Name: private-image-test-1
Namespace: default
Node: 37.72.163.69/37.72.163.69
Start Time: Fri, 06 May 2016 08:04:45 +0000
Labels: <none>
Status: Pending
IP:
Controllers: <none>
Containers:
uses-private-image:
Container ID:
Image: x.x.x/nginx_1
Image ID:
Port:
Command:
echo
SUCCESS
QoS Tier:
cpu: BestEffort
memory: BestEffort
State: Waiting
Reason: Image: x.x.x/nginx_1 is ready, container is creating
Ready: False
Restart Count: 0
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
default-token-zrn4n:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-zrn4n
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
4m 4m 1 {scheduler } scheduled Successfully assigned private-image-test-1 to 37.72.163.69
4m 8s 30 {kubelet 37.72.163.69} implicitly required container POD pulled Successfully pulled image "gcr.io/google_containers/pause:0.8.0"
4m 8s 30 {kubelet 37.72.163.69} implicitly required container POD failed Failed to create docker container with error: no such image
4m 8s 30 {kubelet 37.72.163.69} failedSync Error syncing pod, skipping: no such image
Any help is welcome at this point, thanks!
In most cases where I've come across this issue, it is almost always your credential secret being incorrect. The proper format should be along the lines of
apiVersion: v1
kind: Secret
metadata:
name: registrypullsecret
data:
.dockerconfigjson: {BASE64 encoding of your config}
type: kubernetes.io/dockerconfigjson
From memory, the type field has changed in recent versions of k8s so definitely check that you have the correct type listed.
Also, your yaml example has bad indenting, but thats likely a SO editor issue.