I have some code which i have copied from the internet, below code having two pod container template maven and golang, when i am running my jenkins job, my build got stuck with following error message.
podTemplate(containers: [
containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')
]) {
node(POD_LABEL) {
stage('Get a Maven project') {
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
stage('Build a Maven project') {
sh 'mvn -B clean install'
}
}
}
stage('Get a Golang project') {
git url: 'https://github.com/hashicorp/terraform.git'
container('golang') {
stage('Build a Go project') {
sh """
mkdir -p /go/src/github.com/hashicorp
ln -s `pwd` /go/src/github.com/hashicorp/terraform
cd /go/src/github.com/hashicorp/terraform && make core-dev
"""
}
}
}
}
}
Still waiting to schedule task
All nodes of label ‘fuck_25-g75b6’ are offline
meanwhile when i am checking my master machine, and i verified pod status
kubectl get pods
NAME READY STATUS RESTARTS AGE
fuck-25-g75b6-37hzw-gns85 2/3 Terminating 0 11s
fuck-25-g75b6-37hzw-wz5x6 2/3 Terminating 0 31s
Please find the below pod describe logs
root#jenkins-linux-vm:/usr/share/jenkins# kubectl describe pod fuck-25-g75b6-37hzw-7q2mh
Name: fuck-25-g75b6-37hzw-7q2mh
Namespace: default
Priority: 0
Node: homework-poc-worker2/10.0.0.5
Start Time: Sat, 14 Dec 2019 12:37:55 +0000
Labels: jenkins=slave
jenkins/fuck_25-g75b6=true
Annotations: buildUrl: http://10.0.0.5:9090/job/fuck/25/
Status: Terminating (lasts <invalid>)
Termination Grace Period: 30s
IP: 10.32.0.2
IPs:
IP: 10.32.0.2
Containers:
maven:
Container ID: docker://c6f1acba6c340ea1708d0a13d72d1fc2742d47efeb9db94d38ea1c3ee4269fc7
Image: maven:3.3.9-jdk-8-alpine
Image ID: docker-pullable://maven#sha256:3ab854089af4b40cf3f1a12c96a6c84afe07063677073451c2190cdcec30391b
Port: <none>
Host Port: <none>
Command:
cat
State: Running
Started: Sat, 14 Dec 2019 12:37:59 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8wgjn (ro)
golang:
Container ID: docker://077e06837a4928266f842f1ada7c989e75dfd1069b78d5648c2eb24e433193b0
Image: golang:1.8.0
Image ID: docker-pullable://golang#sha256:70317de915e3414b6bd8e1a0f714d6c4e2d11262abf6815fe9895495a1b3d635
Port: <none>
Host Port: <none>
Command:
cat
State: Running
Started: Sat, 14 Dec 2019 12:38:01 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8wgjn (ro)
jnlp:
Container ID: docker://e89ff0b06c72988427b963253e4958950a9c458d4b92dd1e041a406395b07d7b
Image: jenkins/jnlp-slave:3.35-5-alpine
Image ID: docker-pullable://jenkins/jnlp-slave#sha256:b21cf35bf498b48fdca62ecaf8800ea9e6bacb8ffd68a45b1d78143368533ae1
Port: <none>
Host Port: <none>
State: Terminated
Reason: Error
Exit Code: 255
Started: Sat, 14 Dec 2019 12:38:02 +0000
Finished: Sat, 14 Dec 2019 12:38:03 +0000
Ready: False
Restart Count: 0
Environment:
JENKINS_SECRET: 1de8f650bdeb80df7e4896310a1a1ae8e1e65690f92858218952c93cb86c063f
JENKINS_TUNNEL: http://10.0.0.5:50000
JENKINS_AGENT_NAME: fuck-25-g75b6-37hzw-7q2mh
JENKINS_NAME: fuck-25-g75b6-37hzw-7q2mh
JENKINS_AGENT_WORKDIR: /home/jenkins/agent
JENKINS_URL: http://10.0.0.5:9090/
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8wgjn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-8wgjn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8wgjn
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/fuck-25-g75b6-37hzw-7q2mh to homework-poc-worker2
Normal Pulled 29s kubelet, homework-poc-worker2 Container image "maven:3.3.9-jdk-8-alpine" already present on machine
Normal Created 27s kubelet, homework-poc-worker2 Created container maven
Normal Started 27s kubelet, homework-poc-worker2 Started container maven
Normal Pulled 27s kubelet, homework-poc-worker2 Container image "golang:1.8.0" already present on machine
Normal Created 26s kubelet, homework-poc-worker2 Created container golang
Normal Started 25s kubelet, homework-poc-worker2 Started container golang
Normal Pulled 25s kubelet, homework-poc-worker2 Container image "jenkins/jnlp-slave:3.35-5-alpine" already present on machine
Normal Created 25s kubelet, homework-poc-worker2 Created container jnlp
Normal Started 24s kubelet, homework-poc-worker2 Started container jnlp
Normal Killing 20s kubelet, homework-poc-worker2 Stopping container maven
Normal Killing 20s kubelet, homework-poc-worker2 Stopping container golang
Note: i have two node master/worker both are up and running. let me know if any other details are required.
Can you please some one help me how to fix this issue, this would be really helpful to understand the basic workflow.
Related
I got error when creating deployment.
This is my Dockerfile that i have run and test it on local, i also push it to DockerHub
FROM node:14.15.4
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
RUN npm install pm2 -g
COPY . .
EXPOSE 3001
CMD [ "pm2-runtime", "server.js" ]
In my raspberry pi 3 model B, i have install k3s using curl -sfL https://get.k3s.io | sh -
Here is my controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller-deployment
labels:
app: controller
spec:
replicas: 1
selector:
matchLabels:
app: controller
template:
metadata:
labels:
app: controller
spec:
containers:
- name: controller
image: akirayorunoe/node-controller-server
ports:
- containerPort: 3001
After run this file the pod is error
When i log the pod, it said
standard_init_linux.go:219: exec user process caused: exec format error
Here is the reponse from describe pod
Name: controller-deployment-8669c9c864-sw8kh
Namespace: default
Priority: 0
Node: raspberrypi/192.168.0.30
Start Time: Fri, 21 May 2021 11:21:05 +0700
Labels: app=controller
pod-template-hash=8669c9c864
Annotations: <none>
Status: Running
IP: 10.42.0.43
IPs:
IP: 10.42.0.43
Controlled By: ReplicaSet/controller-deployment-8669c9c864
Containers:
controller:
Container ID: containerd://439edcfdbf49df998e3cabe2c82206b24819a9ae13500b0 13b9bac1df6e56231
Image: akirayorunoe/node-controller-server
Image ID: docker.io/akirayorunoe/node-controller-server#sha256:e1c5115 2f9d596856952d590b1ef9a486e136661076a9d259a9259d4df314226
Port: 3001/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 21 May 2021 11:24:29 +0700
Finished: Fri, 21 May 2021 11:24:29 +0700
Ready: False
Restart Count: 5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-txm85 (ro )
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-txm85:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-txm85
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m33s default-scheduler Successfully ass igned default/controller-deployment-8669c9c864-sw8kh to raspberrypi
Normal Pulled 5m29s kubelet Successfully pul led image "akirayorunoe/node-controller-server" in 3.072053213s
Normal Pulled 5m24s kubelet Successfully pul led image "akirayorunoe/node-controller-server" in 3.018192177s
Normal Pulled 5m6s kubelet Successfully pul led image "akirayorunoe/node-controller-server" in 3.015959209s
Normal Pulled 4m34s kubelet Successfully pul led image "akirayorunoe/node-controller-server" in 2.921116157s
Normal Created 4m34s (x4 over 5m29s) kubelet Created containe r controller
Normal Started 4m33s (x4 over 5m28s) kubelet Started containe r controller
Normal Pulling 3m40s (x5 over 5m32s) kubelet Pulling image "a kirayorunoe/node-controller-server"
Warning BackOff 30s (x23 over 5m22s) kubelet Back-off restart ing failed container
Here is the error images
You are trying to launch a container built for x86 (or x86_64, same difference) on an ARM machine. This does not work. Containers for ARM must be built specifically for ARM and contain ARM executables. While major projects are slowly adding ARM support to their builds, most random images you find on Docker Hub or whatever will not work on ARM.
I'm following the doc in Jenkins page, I'm running with 2 node K8s cluster (1 master 1 worker), setting service type to nodeport, for some reason the init container crashes and never comes up.
kubectl describe pod jenkins-0 -n jenkins
Name: jenkins-0
Namespace: jenkins
Priority: 0
Node: vlab048009.dom047600.lab/10.204.110.35
Start Time: Wed, 09 Dec 2020 23:19:59 +0530
Labels: app.kubernetes.io/component=jenkins-controller
app.kubernetes.io/instance=jenkins
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=jenkins
controller-revision-hash=jenkins-c5795f65f
statefulset.kubernetes.io/pod-name=jenkins-0
Annotations: checksum/config: 2a4c2b3ea5dea271cb7c0b8e8582b682814d39f8e933e0348725b0b9a7dbf258
Status: Pending
IP: 10.244.1.28
IPs:
IP: 10.244.1.28
Controlled By: StatefulSet/jenkins
Init Containers:
init:
Container ID: docker://95e3298740bcaed3c2adf832f41d346e563c92add728080cfdcfcac375e0254d
Image: jenkins/jenkins:lts
Image ID: docker-pullable://jenkins/jenkins#sha256:1433deaac433ce20c534d8b87fcd0af3f25260f375f4ee6bdb41d70e1769d9ce
Port: <none>
Host Port: <none>
Command:
sh
/var/jenkins_config/apply_config.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 09 Dec 2020 23:41:28 +0530
Finished: Wed, 09 Dec 2020 23:41:29 +0530
Ready: False
Restart Count: 9
Limits:
cpu: 2
memory: 4Gi
Requests:
cpu: 50m
memory: 256Mi
Environment: <none>
Mounts:
/usr/share/jenkins/ref/plugins from plugins (rw)
/var/jenkins_config from jenkins-config (rw)
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_plugins from plugin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-ppfw7 (ro)
Containers:
jenkins:
Container ID:
Image: jenkins/jenkins:lts
Image ID:
Ports: 8080/TCP, 50000/TCP
Host Ports: 0/TCP, 0/TCP
Args:
--httpPort=8080
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 4Gi
Requests:
cpu: 50m
memory: 256Mi
Liveness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=3
Startup: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=12
Environment:
POD_NAME: jenkins-0 (v1:metadata.name)
JAVA_OPTS: -Dcasc.reload.token=$(POD_NAME)
JENKINS_OPTS:
JENKINS_SLAVE_AGENT_PORT: 50000
CASC_JENKINS_CONFIG: /var/jenkins_home/casc_configs
Mounts:
/run/secrets/chart-admin-password from admin-secret (ro,path="jenkins-admin-password")
/run/secrets/chart-admin-username from admin-secret (ro,path="jenkins-admin-user")
/usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
/var/jenkins_config from jenkins-config (ro)
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_home/casc_configs from sc-config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-ppfw7 (ro)
config-reload:
Container ID:
Image: kiwigrid/k8s-sidecar:0.1.275
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
POD_NAME: jenkins-0 (v1:metadata.name)
LABEL: jenkins-jenkins-config
FOLDER: /var/jenkins_home/casc_configs
NAMESPACE: jenkins
REQ_URL: http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
REQ_METHOD: POST
REQ_RETRY_CONNECT: 10
Mounts:
/var/jenkins_home from jenkins-home (rw)
/var/jenkins_home/casc_configs from sc-config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-ppfw7 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
jenkins-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: jenkins
Optional: false
plugin-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
jenkins-home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
sc-config-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
admin-secret:
Type: Secret (a volume populated by a Secret)
SecretName: jenkins
Optional: false
jenkins-token-ppfw7:
Type: Secret (a volume populated by a Secret)
SecretName: jenkins-token-ppfw7
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 22m default-scheduler Successfully assigned jenkins/jenkins-0 to vlab048009.dom047600.lab
Normal Pulled 22m kubelet Successfully pulled image "jenkins/jenkins:lts" in 4.648858149s
Normal Pulled 21m kubelet Successfully pulled image "jenkins/jenkins:lts" in 1.407161762s
Normal Pulled 21m kubelet Successfully pulled image "jenkins/jenkins:lts" in 4.963056101s
Normal Created 21m (x4 over 22m) kubelet Created container init
Normal Started 21m (x4 over 22m) kubelet Started container init
Normal Pulled 21m kubelet Successfully pulled image "jenkins/jenkins:lts" in 8.0749493s
Normal Pulling 20m (x5 over 22m) kubelet Pulling image "jenkins/jenkins:lts"
Warning BackOff 2m1s (x95 over 21m) kubelet Back-off restarting failed container
[
kubectl logs -f jenkins-0 -c init -n jenkins
Error from server: Get "https://10.204.110.35:10250/containerLogs/jenkins/jenkins-0/init?follow=true": dial tcp 10.204.110.35:10250: connect: no route to host
kubectl get events -n jenkins
LAST SEEN TYPE REASON OBJECT MESSAGE
23m Normal Scheduled pod/jenkins-0 Successfully assigned jenkins/jenkins-0 to vlab048009.dom047600.lab
21m Normal Pulling pod/jenkins-0 Pulling image "jenkins/jenkins:lts"
23m Normal Pulled pod/jenkins-0 Successfully pulled image "jenkins/jenkins:lts" in 4.648858149s
22m Normal Created pod/jenkins-0 Created container init
22m Normal Started pod/jenkins-0 Started container init
23m Normal Pulled pod/jenkins-0 Successfully pulled image "jenkins/jenkins:lts" in 1.407161762s
3m30s Warning BackOff pod/jenkins-0 Back-off restarting failed container
23m Normal Pulled pod/jenkins-0 Successfully pulled image "jenkins/jenkins:lts" in 4.963056101s
22m Normal Pulled pod/jenkins-0 Successfully pulled image "jenkins/jenkins:lts" in 8.0749493s
23m Normal SuccessfulCreate statefulset/jenkins create Pod jenkins-0 in StatefulSet jenkins successful
Every 2.0s: kubectl get all -n jenkins Wed Dec 9 23:48:31 2020
NAME READY STATUS RESTARTS AGE
pod/jenkins-0 0/2 Init:CrashLoopBackOff 10 28m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/jenkins NodePort 10.103.209.122 <none> 8080:32323/TCP 28m
service/jenkins-agent ClusterIP 10.103.195.120 <none> 50000/TCP 28m
NAME READY AGE
statefulset.apps/jenkins 0/1 28m
Using helm3 to deploy jenkins, pretty much changes done as per doc.
Not sure how to debug this issues wrt init container crashing, any leads or a solution would be appreciated, Thanks
Firstly make sure that you had executed command:
$ helm repo update
Execute also command:
$ kubectl logs <pod-name> -c <init-container-name>
to inspect init container. Then you will be able to properly debug this setup.
This might be a connection issue to the Jenkins update site. You can build an image which contains required plugins and disable plugin download. Take a look: jenkins-kubernetes.
See more: jenkins-helm-issues - in this case problem lays in plug-in compatibility.
I want to trigger a manual workflow in Argo. I am using Openshift and ArgoCD, have scheduled workflows that are running successfully in Argo but failing when triggering a manual run for one workflow.
The concerned workflow is:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: "obslytics-data-exporter-manual-workflow-"
labels:
owner: "obslytics-remote-reader"
app: "obslytics-data-exporter"
pipeline: "obslytics-data-exporter"
spec:
arguments:
parameters:
- name: start_timestamp
value: "2020-11-18T20:00:00Z"
entrypoint: manual-trigger
templates:
- name: manual-trigger
steps:
- - name: trigger
templateRef:
name: "obslytics-data-exporter-workflow-triggers"
template: trigger-workflow
volumes:
- name: "obslytics-data-exporter-workflow-secrets"
secret:
secretname: "obslytics-data-exporter-workflow-secrets"
When I run the command:
argo submit trigger.local.yaml
The build pod is completed but the rest pods fail:
➜ dh-workflow-obslytics git:(master) ✗ oc get pods
NAME READY STATUS RESTARTS AGE
argo-ui-7fcf5ff95-9k8cc 1/1 Running 0 3d
gateway-controller-76bb888f7b-lq84r 1/1 Running 0 3d
obslytics-data-exporter-1-build 0/1 Completed 0 3d
obslytics-data-exporter-calendar-gateway-fbbb8d7-zhdnf 2/2 Running 1 3d
obslytics-data-exporter-manual-workflow-m7jdg-1074461258 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1477271209 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1544087495 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1979266120 0/2 Completed 0 4m
obslytics-data-exporter-sensor-6594954795-xw8fk 1/1 Running 0 3d
opendatahub-operator-8994ddcf8-v8wxm 1/1 Running 0 3d
sensor-controller-58bdc7c4f4-9h4jw 1/1 Running 0 3d
workflow-controller-759649b79b-s69l7 1/1 Running 0 3d
The pods starting with obslytics-data-exporter-manual-workflow are the concerned pods that are failing. When I attempt to debug by describing pods:
➜ dh-workflow-obslytics git:(master) ✗ oc describe pods/obslytics-data-exporter-manual-workflow-4hzqz-3278280317
Name: obslytics-data-exporter-manual-workflow-4hzqz-3278280317
Namespace: dh-dev-argo
Priority: 0
PriorityClassName: <none>
Node: avsrivas-dev-ocp-3.11/10.0.111.224
Start Time: Tue, 24 Nov 2020 07:27:57 -0500
Labels: workflows.argoproj.io/completed=true
workflows.argoproj.io/workflow=obslytics-data-exporter-manual-workflow-4hzqz
Annotations: openshift.io/scc=restricted
workflows.argoproj.io/node-message=timeout after 0s
workflows.argoproj.io/node-name=obslytics-data-exporter-manual-workflow-4hzqz[0].trigger[1].run[0].metric-split(0:cluster_version)[0].process-metric(0)
workflows.argoproj.io/template={"name":"run-obslytics","arguments":{},"inputs":{"parameters":[{"name":"metric","value":"cluster_version"},{"name":"start_timestamp","value":"2020-11-18T20:00:00Z"},{"na...
Status: Failed
IP: 10.128.0.69
Controlled By: Workflow/obslytics-data-exporter-manual-workflow-4hzqz
Init Containers:
init:
Container ID: docker://25b95c684ef66b13520ba9deeba353082142f3bb39bafe443ee508074c58047e
Image: argoproj/argoexec:v2.4.2
Image ID: docker-pullable://docker.io/argoproj/argoexec#sha256:4e393daa6ed985cf680bcf0ecf04f7b0758940f0789505428331fcfe99cce06b
Port: <none>
Host Port: <none>
Command:
argoexec
init
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Nov 2020 07:27:59 -0500
Finished: Tue, 24 Nov 2020 07:27:59 -0500
Ready: True
Restart Count: 0
Environment:
ARGO_POD_NAME: obslytics-data-exporter-manual-workflow-4hzqz-3278280317 (v1:metadata.name)
ARGO_CONTAINER_RUNTIME_EXECUTOR: k8sapi
Mounts:
/argo/podmetadata from podmetadata (rw)
/argo/staging from argo-staging (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
Containers:
wait:
Container ID: docker://a94e7f1bc1cfec4c8b549120193b697c91760bb8f3af414babef1d6f7ccee831
Image: argoproj/argoexec:v2.4.2
Image ID: docker-pullable://docker.io/argoproj/argoexec#sha256:4e393daa6ed985cf680bcf0ecf04f7b0758940f0789505428331fcfe99cce06b
Port: <none>
Host Port: <none>
Command:
argoexec
wait
State: Terminated
Reason: Completed
Message: timeout after 0s
Exit Code: 0
Started: Tue, 24 Nov 2020 07:28:00 -0500
Finished: Tue, 24 Nov 2020 07:28:01 -0500
Ready: False
Restart Count: 0
Environment:
ARGO_POD_NAME: obslytics-data-exporter-manual-workflow-4hzqz-3278280317 (v1:metadata.name)
ARGO_CONTAINER_RUNTIME_EXECUTOR: k8sapi
Mounts:
/argo/podmetadata from podmetadata (rw)
/mainctrfs/argo/staging from argo-staging (rw)
/mainctrfs/etc/obslytics-data-exporter from obslytics-data-exporter-workflow-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
main:
Container ID: docker://<some_id>
Image: docker-registry.default.svc:5000/<some_id>
Image ID: docker-pullable://docker-registry.default.svc:5000/<some_id>
Port: <none>
Host Port: <none>
Command:
/bin/sh
-e
Args:
/argo/staging/script
State: Terminated
Reason: Error
Exit Code: 126
Started: Tue, 24 Nov 2020 07:28:01 -0500
Finished: Tue, 24 Nov 2020 07:28:01 -0500
Ready: False
Restart Count: 0
Limits:
memory: 1Gi
Requests:
memory: 1Gi
Environment: <none>
Mounts:
/argo/staging from argo-staging (rw)
/etc/obslytics-data-exporter from obslytics-data-exporter-workflow-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
podmetadata:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations -> annotations
obslytics-data-exporter-workflow-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: obslytics-data-exporter-workflow-secrets
Optional: false
argo-staging:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-qpggm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qpggm
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27m default-scheduler Successfully assigned dh-dev-argo/obslytics-data-exporter-manual-workflow-4hzqz-3278280317 to avsrivas-dev-ocp-3.11
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Container image "argoproj/argoexec:v2.4.2" already present on machine
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Container image "argoproj/argoexec:v2.4.2" already present on machine
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
Normal Pulling 27m kubelet, avsrivas-dev-ocp-3.11 pulling image "docker-registry.default.svc:5000/dh-dev-argo/obslytics-data-exporter:latest"
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Successfully pulled image "docker-registry.default.svc:5000/dh-dev-argo/obslytics-data-exporter:latest"
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
The only thing I learn from the above description is that the pods fail due to an error. I am unable to see any error in order to debug this issue.
When I attempt to read the Argo watch logs:
Name: obslytics-data-exporter-manual-workflow-8wzcc
Namespace: dh-dev-argo
ServiceAccount: default
Status: Running
Created: Tue Nov 24 08:01:10 -0500 (8 minutes ago)
Started: Tue Nov 24 08:01:10 -0500 (8 minutes ago)
Duration: 8 minutes 10 seconds
Progress:
Parameters:
start_timestamp: 2020-11-18T20:00:00Z
STEP TEMPLATE PODNAME DURATION MESSAGE
● obslytics-data-exporter-manual-workflow-8wzcc manual-trigger
└───● trigger obslytics-data-exporter-workflow-triggers/trigger-workflow
├───✔ get-labels(0) obslytics-data-exporter-workflow-template/get-labels obslytics-data-exporter-manual-workflow-8wzcc-2604296472 6s
└───● run obslytics-data-exporter-workflow-template/init
└───● metric-split(0:cluster_version) metric-worker
└───● process-metric run-obslytics
├─✖ process-metric(0) run-obslytics obslytics-data-exporter-manual-workflow-8wzcc-4222496183 6s failed with exit code 126
└─◷ process-metric(1) run-obslytics obslytics-data-exporter-manual-workflow-8wzcc-531670266 7m PodInitializing
I am trying to upgraded pods with 7.7 version of elasticsearch. Unable to do so.
below is values.yaml. Referring to here https://www.docker.elastic.co/r/elasticsearch/elasticsearch-oss:7.7.1 for offical docker image.
cluster:
name: elastic-x-pack
replicaCount:
client: 2
data: 2
master: 3
minimum_master_nodes: 2
image:
registry: docker.elastic.co
name: elasticsearch/elasticsearch-oss
tag: 7.7.1
pullPolicy: Always
service:
type: NodePort
http:
externalPort: 30000
internalPort: 9200
tcp:
externalPort: 30112
internalPort: 9300
opts: -Xms256m -Xmx256m
resources: {}
global:
elasticsearch:
storage:
data:
class: standard
size: 3Gi
snapshot:
class: standard
size: 5Gi
accessModes: [ ReadWriteMany ]
name: data-snapshot
cluster:
features:
DistributedTracing: test
ignite:
registry: test
But pods are not running and are in CrashLoopBackOff state.
below is the description of the pod
Name: elastic-cluster-elasticsearch-cluster-client-685d698-jf7bb
Namespace: default
Priority: 0
Node: ip-172-31-38-123.us-west-2.compute.internal/172.31.38.123
Start Time: Fri, 26 Jun 2020 09:31:23 +0000
Labels: app=elasticsearch-cluster
component=elasticsearch
pod-template-hash=685d698
release=elastic-cluster
role=client
Annotations: <none>
Status: Running
IP: 10.233.68.58
Controlled By: ReplicaSet/elastic-cluster-elasticsearch-cluster-client-685d698
Init Containers:
init-sysctl:
Container ID: docker://d83c3be3f4d7ac1362599d115813d6cd1b1356959a5a2784c1f90f3ed74daa69
Image: busybox:1.27.2
Image ID: docker-pullable://busybox#sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 26 Jun 2020 09:31:24 +0000
Finished: Fri, 26 Jun 2020 09:31:24 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d7b79 (ro)
Containers:
elasticsearch-cluster:
Container ID: docker://74822d62d876b798c1518c0e42da071d661b2ccdbeb1fe40487044a9cc07e6f4
Image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.7.1
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch-oss#sha256:04f0a377e55fcc41f3467e8a222357a7a5ef0b1e3ec026b6d63a59465870bd8e
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 78
Started: Fri, 26 Jun 2020 09:32:26 +0000
Finished: Fri, 26 Jun 2020 09:32:33 +0000
Ready: False
Restart Count: 3
Liveness: tcp-socket :transport delay=300s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/_cluster/health delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
NAMESPACE: default (v1:metadata.namespace)
NODE_NAME: elastic-cluster-elasticsearch-cluster-client-685d698-jf7bb (v1:metadata.name)
CLUSTER_NAME: elastic-x-pack
ES_JAVA_OPTS: -Xms256m -Xmx256m
NODE_DATA: false
HTTP_ENABLE: true
NETWORK_HOST: _site_,_lo_
NODE_MASTER: false
Mounts:
/data from storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d7b79 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-d7b79:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-d7b79
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 112s default-scheduler Successfully assigned default/elastic-cluster-elasticsearch-cluster-client-685d698-jf7bb to ip-172-31-38-123.us-west-2.compute.internal
Normal Pulled 111s kubelet, ip-172-31-38-123.us-west-2.compute.internal Container image "busybox:1.27.2" already present on machine
Normal Created 111s kubelet, ip-172-31-38-123.us-west-2.compute.internal Created container init-sysctl
Normal Started 111s kubelet, ip-172-31-38-123.us-west-2.compute.internal Started container init-sysctl
Normal Pulled 49s (x4 over 110s) kubelet, ip-172-31-38-123.us-west-2.compute.internal Container image "docker.elastic.co/elasticsearch/elasticsearch-oss:7.7.1" already present on machine
Normal Created 49s (x4 over 110s) kubelet, ip-172-31-38-123.us-west-2.compute.internal Created container elasticsearch-cluster
Normal Started 49s (x4 over 110s) kubelet, ip-172-31-38-123.us-west-2.compute.internal Started container elasticsearch-cluster
Warning BackOff 5s (x9 over 94s) kubelet, ip-172-31-38-123.us-west-2.compute.internal Back-off restarting failed container
PLEASE READ UPDATE 2
I have a very simple EventHubClient app. It will just listen to an EventHub messages.
I get it running with the Docker support given in Visual Studio 2017 (Linux Container).
But when I try to deploy it in Kubernetes, I get "Back-off restarting failed container"
C# Code:
public static void Main(string[] args)
{
// Init Mapper
AutoMapper.Mapper.Initialize(cfg =>
{
cfg.AddProfile<AiElementProfile>();
});
Console.WriteLine("Registering EventProcessor...");
var eventProcessorHost = new EventProcessorHost(
EventHubPath,
ConsumerGroupName,
EventHubConnectionString,
AzureStorageConnectionString,
ContainerName
);
// Registers the Event Processor Host and starts receiving messages
eventProcessorHost.RegisterEventProcessorAsync<EventProcessor>();
Console.WriteLine("Receiving. Press ENTER to stop worker.");
Console.ReadLine();
}
Kubernetes Manifest file (.yaml):
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: historysvc-deployment
spec:
selector:
matchLabels:
app: historysvc
replicas: 2
template:
metadata:
labels:
app: historysvc
spec:
containers:
- name: historysvc
image: vncont.azurecr.io/historysvc:v1
ports:
- containerPort: 80
imagePullSecrets:
- name: acr-auth
kubectl get pods:
NAME READY STATUS RESTARTS AGE
historysvc-deployment-558fc5649f-bln8f 0/1 CrashLoopBackOff 17 1h
historysvc-deployment-558fc5649f-jgjvq 0/1 CrashLoopBackOff 17 1h
kubectl describe pod historysvc-deployment-558fc5649f-bln8f
Name: historysvc-deployment-558fc5649f-bln8f
Namespace: default
Node: aks-nodepool1-81522366-0/10.240.0.4
Start Time: Tue, 24 Jul 2018 10:15:37 +0200
Labels: app=historysvc
pod-template-hash=1149712059
Annotations: <none>
Status: Running
IP: 10.244.0.11
Controlled By: ReplicaSet/historysvc-deployment-558fc5649f
Containers:
historysvc:
Container ID: docker://59e66f1e6420146f6eca4f19e2801a4ee0435a34c7ac555a8d04f699a1497f35
Image: vncont.azurecr.io/historysvc:v1
Image ID: docker-pullable://vncont.azurecr.io/historysvc#sha256:636d81435bd421ec92a0b079c3841cbeb3ad410509a6e37b1ec673dc4ab8a444
Port: 80/TCP
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Jul 2018 10:17:10 +0200
Finished: Tue, 24 Jul 2018 10:17:10 +0200
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Jul 2018 10:16:29 +0200
Finished: Tue, 24 Jul 2018 10:16:29 +0200
Ready: False
Restart Count: 4
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mt8mm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-mt8mm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mt8mm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned historysvc-deployment-558fc5649f-bln8f to aks-nodepool1-81522366-0
Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-81522366-0 MountVolume.SetUp succeeded for volume "default-token-mt8mm"
Normal Pulled 8s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Container image "vncont.azurecr.io/historysvc:v1" already present on machine
Normal Created 7s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Created container
Normal Started 6s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Started container
Warning BackOff 6s (x8 over 1m) kubelet, aks-nodepool1-81522366-0 Back-off restarting failed container
What am I missing?
UPDATE 1
kubectl describe pod historysvc-deployment-558fc5649f-jgjvq
Name: historysvc-deployment-558fc5649f-jgjvq
Namespace: default
Node: aks-nodepool1-81522366-0/10.240.0.4
Start Time: Tue, 24 Jul 2018 10:15:37 +0200
Labels: app=historysvc
pod-template-hash=1149712059
Annotations: <none>
Status: Running
IP: 10.244.0.12
Controlled By: ReplicaSet/historysvc-deployment-558fc5649f
Containers:
historysvc:
Container ID: docker://ccf83bce216276450ed79d67fb4f8a66daa54cd424461762478ec62f7e592e30
Image: vncont.azurecr.io/historysvc:v1
Image ID: docker-pullable://vncont.azurecr.io/historysvc#sha256:636d81435bd421ec92a0b079c3841cbeb3ad410509a6e37b1ec673dc4ab8a444
Port: 80/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 25 Jul 2018 09:32:34 +0200
Finished: Wed, 25 Jul 2018 09:32:35 +0200
Ready: False
Restart Count: 277
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mt8mm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-mt8mm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mt8mm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 2m (x6238 over 23h) kubelet, aks-nodepool1-81522366-0 Back-off restarting failed container
UPDATE 2
When I run it localy with:
docker run <image>
it ends instantly (ignores the read line) (completes), which seems to be the problem.
I have to write
docker run -it <image>
-it at the end for it to do the read line.
How does kubernetes runs the docker image? Where can I set that?
This can be done by attaching an argument to run with your deployment.
In your case the Kubernetes Manifest file (.yaml) should look like this:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: historysvc-deployment
spec:
selector:
matchLabels:
app: historysvc
replicas: 2
template:
metadata:
labels:
app: historysvc
spec:
containers:
- name: historysvc
image: vncont.azurecr.io/historysvc:v1
ports:
- containerPort: 80
args: ["-it"]
imagePullSecrets:
- name: acr-auth
You can find this explained in k8s docs inject-data-application/define-command-argument-container
When you create a Pod, you can define a command and arguments for the containers that run in the Pod. To define a command, include the command field in the configuration file. To define arguments for the command, include the args field in the configuration file. The command and arguments that you define cannot be changed after the Pod is created.
The command and arguments that you define in the configuration file override the default command and arguments provided by the container image. If you define args, but do not define a command, the default command is used with your new arguments.