Jenkins sh script hangs when run in specific container - docker

Question

I'm trying to automate deployments using the official ArgoCD docker image (https://hub.docker.com/r/argoproj/argocd/dockerfile)
I've created a declarative jenkins pipeline using the kubernetes plugin for the agents and define the pod using yaml, the container definition looks like this:
pipeline {
agent {
kubernetes {
yaml """
kind: Pod
metadata:
name: agent
spec:
containers:
- name: maven
image: maven:slim
command:
- cat
tty: true
volumeMounts:
- name: jenkins-maven-cache
mountPath: /root/.m2/repository
- name: argocd
image: argoproj/argocd:latest
command:
- cat
tty: true
...
I'm trying to run commands inside that container, that step in the pipeline looks like this:
stage('Build') {
steps {
container('maven') {
sh 'echo testing' // this works just fine
}
}
}
stage('Deploy') {
steps {
container('argocd') {
sh "echo testing" // this does not work
// more deploy scripts here, once sh works
}
}
}
So I have two containers, one where the sh script works just fine and another where it doesn't. The sh scripts in the "argocd" container just hangs for 5 minutes and then Jenkins kills it, the exit message is:
process apparently never started in /home/jenkins/agent/workspace/job-name#tmp/durable-46cefcae (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
I can't do echo a simple string in this particular container.
It works fine in other containers like the official for Maven from Docker, I use to build the spring boot application. I can also run commands directly in the argocd container manually from commandline with docker exec, but jenkins just won't in the pipeline for some reason.
What could it be?
I am running the latest version (1.33) of the durable task plugin.
Update:
Turns out that the image for argo-cd (continuous deployment tool) argoproj/argocd:latest does not include other commands except argocd, so the issue was with the container image I tried to use and not Jenkins itself. My solution was to install the Argo-CD CLI into a custom docker container and use that instead of the official one.

Answer 1

I've just encountered a similar issue with a custom docker image created by myself.
It turns out, I was using USER nobody in Dockerfile of that image and somehow, this way Jenkins agent pod was unable to run cat command or any other shell command from my pipeline script. Running specific container with root user worked for me.
So in your case I would add securityContext: runAsUser: 0 like below.
...
- name: argocd
image: argoproj/argocd:latest
command:
- cat
tty: true
securityContext:
runAsUser: 0
...
Kubernetes reference: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container

Answer 2

If the issue is Jenkins related here are some things that may help to solve the problem:
Issues with the working directory, if you updated Jenkins from some older version the workdir was /home/jenkins while in the recent versions it should be /home/jenkins/agent
or if you are running it in Windows the path should start with C:\dir and not with /dir
You can try a new clean install with apt-get --purge remove jenkins and then apt-get install jenkins
This is not your case as you run latest version of durable task plugin. But for other people reference versions prior to 1.28-1.30 caused the same issue.
If your Jenkins is clean the issue should be investigated in a different way, it seems that it's not returning an exit code to the sh command and/or the script is executed in a different shell.
I would try to do an sh file to be placed in the working directory of the container
#!/bin/bash
echo "testing"
echo $?
and try to run it with source my_script.sh
or with bash my_script.sh
$? is the exit code of the latest bash operation, having it printed will make sure that your script is terminated correctly. The source command to run the script will make it run in the same shell that is calling it so the shell variables are accessible. Bash command will run it in another subshell instead.