Jenkins unable to initialize using Kubernetes - jenkins

Question

From yesterday I started having problems with jenkins pod - it is unable to be initialized. I haven't update any configuration in the meantime.
This is how my pod deployment configuration looks like
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: eed56a3d795865e4432dea721435a777ee100059998724f0d57bf1f9378dbb88
creationTimestamp: 2020-09-17T14:14:12Z
generateName: jenkins-74cc957b47-
labels:
app: jenkins
chart: jenkins-0.35.0
component: jenkins-jenkins-master
heritage: Tiller
pod-template-hash: "3077513603"
release: jenkins
name: jenkins-74cc957b47-zf67f
namespace: infrastructure
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: jenkins-74cc957b47
uid: 77b5d3d7-f0f1-11ea-acd2-02be15828c0e
resourceVersion: "158370354"
selfLink: /api/v1/namespaces/infrastructure/pods/jenkins-74cc957b47-zf67f
uid: 0fcefd0d-f8f0-11ea-acd2-02be15828c0e
spec:
containers:
- args:
- --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
- --argumentsRealm.roles.$(ADMIN_USER)=admin
env:
- name: JAVA_OPTS
- name: JENKINS_OPTS
- name: JENKINS_SLAVE_AGENT_PORT
value: "50000"
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: jenkins-admin-password
name: jenkins
- name: ADMIN_USER
valueFrom:
secretKeyRef:
key: jenkins-admin-user
name: jenkins
image: jenkins/jenkins:2.247
imagePullPolicy: Always
livenessProbe:
failureThreshold: 12
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 90
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: jenkins
ports:
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 50000
name: slavelistener
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1280m"
memory: 3Gi
requests:
cpu: 50m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /var/jenkins_config
name: jenkins-config
readOnly: true
- mountPath: /usr/share/jenkins/ref/plugins/
name: plugin-dir
- mountPath: /usr/share/jenkins/ref/secrets/
name: secrets-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5tbbb
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- /var/jenkins_config/apply_config.sh
env:
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: jenkins-admin-password
name: jenkins
- name: ADMIN_USER
valueFrom:
secretKeyRef:
key: jenkins-admin-user
name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: Always
name: copy-default-config
resources:
limits:
cpu: "1280m"
memory: 3Gi
requests:
cpu: 50m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /var/jenkins_config
name: jenkins-config
- mountPath: /var/jenkins_plugins
name: plugin-dir
- mountPath: /usr/share/jenkins/ref/secrets/
name: secrets-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5tbbb
readOnly: true
nodeName: ip-172-20-62-226.eu-west-1.compute.internal
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 0
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: jenkins
name: jenkins-config
- emptyDir: {}
name: plugin-dir
- emptyDir: {}
name: secrets-dir
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins
- name: default-token-5tbbb
secret:
defaultMode: 420
secretName: default-token-5tbbb
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2020-09-17T14:15:03Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2020-09-17T14:17:11Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: null
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: 2020-09-17T14:14:12Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://688f4ad7dde842c2b5d6a0f1fd3cdd7ca156c8457336ca07f1d11270c2df0779
image: jenkins/jenkins:lts
imageID: docker-pullable://jenkins/jenkins#sha256:a3e7b2b6efbc2c252608b028bb844e419d44ad5e3974770c4543ab7ae6e8eb27
lastState: {}
name: jenkins
ready: true
restartCount: 0
state:
running:
startedAt: 2020-09-17T14:15:05Z
hostIP: 172.20.62.226
initContainerStatuses:
- containerID: docker://6761eab1b990aa42c7ec21ee84d1e2362eeddf9373f595ccb13b0e59c0462505
image: jenkins/jenkins:lts
imageID: docker-pullable://jenkins/jenkins#sha256:a3e7b2b6efbc2c252608b028bb844e419d44ad5e3974770c4543ab7ae6e8eb27
lastState: {}
name: copy-default-config
ready: true
restartCount: 0
state:
terminated:
containerID: docker://6761eab1b990aa42c7ec21ee84d1e2362eeddf9373f595ccb13b0e59c0462505
exitCode: 0
finishedAt: 2020-09-17T14:15:02Z
reason: Completed
startedAt: 2020-09-17T14:14:41Z
phase: Running
podIP: 100.105.185.69
qosClass: Burstable
startTime: 2020-09-17T14:14:12Z
I have tried to edit it and set the specific jenkins image version, for example image: jenkins/jenkins:2.219, but it is still not able to initialize.
When I run kubectl logs jenkins-df87c46d5-52dtt -c copy-default-config -n infrastructure I can see the following log:
11:21:05 Failed in the last attempt (curl -sSfL --connect-timeout 20 --retry 3 --retry-delay 0 --retry-max-time 60 https://updates.jenkins.io/dynamic-2.248//latest/workflow-cps.hpi -o /usr/share/jenkins/ref/plugins/workflow-cps.jpi)
Downloading plugin: workflow-cps-plugin from https://updates.jenkins.io/dynamic-2.248//latest/workflow-cps-plugin.hpi
curl: (28) Resolving timed out after 20527 milliseconds
11:21:05 Failure (28) Retrying in 1 seconds...
curl: (28) Resolving timed out after 20526 milliseconds
11:21:08 Failure (28) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
11:21:14 Failure (22) Retrying in 1 seconds...
Full output is available here: https://justpaste.it/8h30t

Answer 1

Try this. I have removed the plugin directiory which might cause the issue
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: eed56a3d795865e4432dea721435a777ee100059998724f0d57bf1f9378dbb88
creationTimestamp: 2020-09-17T14:14:12Z
generateName: jenkins-74cc957b47-
labels:
app: jenkins
chart: jenkins-0.35.0
component: jenkins-jenkins-master
heritage: Tiller
pod-template-hash: "3077513603"
release: jenkins
name: jenkins-74cc957b47-zf67f
namespace: infrastructure
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: jenkins-74cc957b47
uid: 77b5d3d7-f0f1-11ea-acd2-02be15828c0e
resourceVersion: "158370354"
selfLink: /api/v1/namespaces/infrastructure/pods/jenkins-74cc957b47-zf67f
uid: 0fcefd0d-f8f0-11ea-acd2-02be15828c0e
spec:
containers:
- args:
- --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
- --argumentsRealm.roles.$(ADMIN_USER)=admin
env:
- name: JAVA_OPTS
- name: JENKINS_OPTS
- name: JENKINS_SLAVE_AGENT_PORT
value: "50000"
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: jenkins-admin-password
name: jenkins
- name: ADMIN_USER
valueFrom:
secretKeyRef:
key: jenkins-admin-user
name: jenkins
image: jenkins/jenkins:2.247
imagePullPolicy: Always
livenessProbe:
failureThreshold: 12
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 90
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: jenkins
ports:
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 50000
name: slavelistener
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1280m"
memory: 3Gi
requests:
cpu: 50m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /var/jenkins_config
name: jenkins-config
readOnly: true
- mountPath: /usr/share/jenkins/ref/secrets/
name: secrets-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5tbbb
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- /var/jenkins_config/apply_config.sh
env:
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: jenkins-admin-password
name: jenkins
- name: ADMIN_USER
valueFrom:
secretKeyRef:
key: jenkins-admin-user
name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: Always
name: copy-default-config
resources:
limits:
cpu: "1280m"
memory: 3Gi
requests:
cpu: 50m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /var/jenkins_config
name: jenkins-config
- mountPath: /usr/share/jenkins/ref/secrets/
name: secrets-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5tbbb
readOnly: true
nodeName: ip-172-20-62-226.eu-west-1.compute.internal
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 0
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: jenkins
name: jenkins-config
- emptyDir: {}
name: secrets-dir
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins
- name: default-token-5tbbb
secret:
defaultMode: 420
secretName: default-token-5tbbb