"For loop" bash command does not execute in Jenkins build - jenkins

I try to execute such a scenery via Jenkins "execute shell" build step:
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt";
for file in $(<"_dpatch/deploy/files.txt"); do cp --parents "$file" "_dpatch"; done;
whoami
Build ends successfully with console output:
[Deploy to production] $ /bin/sh -xe /tmp/hudson8315034696077699718.sh
+ rm -r -f _dpatch
+ mkdir _dpatch
+ mkdir _dpatch/deploy
+ from_revision=HEAD
+ to_revision=2766920
+ git diff --name-only HEAD 2766920
+
+ whoami
jenkins
Finished: SUCCESS
The problem is line "for file in" is just ignored, I do not understand why.
Content of files.txt is not empty and looks like this:
addons/tiny_mce/plugins/image/plugin.min.org.js
addons/webrtc/adapter-latest.js
templates/standard/style/review.css
More over, when I execute via ssh the same script in the same jenkins workspace folder under the same user (jenkins) - "for file in" line executes normally and creates files in "_dpatch" subfolder as it should.
My environment:
Debian 8,
Jenkins 2.45
Thanks

Possibly your /bin/sh is a POSIX bourne shell. It think that the $(< construct is a bash-ism, so it will not work with /bin/sh.
Try to replace
$(<"_dpatch/deploy/files.txt")
with
$(cat "_dpatch/deploy/files.txt")
Alternatively, prepend your build step with #!/bin/bash.
If your login shell is bash, then this also explains why everything works fine via ssh.

Try substituting for with while loop. And also add some more logging
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt" && echo "git diff finished"
while IFS= read -r line; do
echo $line
cp --parent $line $_dpatch
done < _dpatch/deploy/files.txt
whoami

Related

Jenkins Jenkinsfile Groovy bash command no such file or directory

The file validates and I look to have the proper syntax.
script {
sh """
summon -f folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
open folder/file.yml: no such file or directory
I confirmed the existence of the file and workspace location.
Try using full path with workspace variable:
script {
sh """
summon -f ${WORKSPACE}/folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
}
so what I see happening is I wrapped the file into a script. ran git add , git commit, git push. I updated the jenkins file to ls -l the folder and I notice that file is missing. so not sure if this is a git issue or jenkins or etc

Build and Run Docker Container in Jenkins

I need to run docker container in Jenkins so that installed libraries like pycodestyle can be runnable in the following steps.
I successfully built Docker Container (in Dockerfile)
How do I access to the container so that I can use it in the next step? (Please look for >> << code in Build step below)
Thanks
stage('Build') {
// Install python libraries from requirements.txt (Check Dockerfile for more detail)
sh "docker login -u '${DOCKER_USR}' -p '${DOCKER_PSW}' ${DOCKER_REGISTRY}"
sh "docker build \
--tag '${DOCKER_REGISTRY}/${DOCKER_TAG}:latest' \
--build-arg HTTPS_PROXY=${PIP_PROXY} ."
>> sh "docker run -ti ${DOCKER_REGISTRY}/${DOCKER_TAG}:latest sh" <<<
}
}
stage('Linting') {
sh '''
awd=$(pwd)
echo '===== Linting START ====='
for file in $(find . -name '*.py'); do
filename=$(basename $file)
if [[ ${file:(-3)} == ".py" ]] && [[ $filename = *"test"* ]] ; then
echo "perform PEP8 lint (python pylint blah) for $filename"
cd $awd && cd $(dirname "${file}") && pycodestyle "${filename}"
fi
done
echo '===== Linting END ====='
'''
}
You need to mount the workspace of your Jenkins job (containing your python project) as volume (see "docker run -v" option) to your container and then run the "next step" build step inside this container. You can do this by providing a shell script as part of your project's source code, which does the "next step" or write this script in a previous build stage.
It would be something like this:
sh "chmod +x build.sh"
sh "docker run -v $WORKSPACE:/workspace ${DOCKER_REGISTRY}/${DOCKER_TAG}:latest /workspace/build.sh"
build.sh is an executable script, which is part of your project's workspace and performans the "next step".
$WORKSPACE is the folder that is used by your jenkins job (normally /var/jenkins_home/jobs//workspace - it is provided by Jenkins as a build variable.
Please note: This solution requires that the Docker daemon is running on the same host as Jenkins! Otherwise the workspace will not be available to your container.
Another solution would be to run Jenkins as Docker container, so you can share the Jenkins home/workspaces easily with the containers you run within your build jobs, like described here:
Running Jenkins tests in Docker containers build from dockerfile in codebase

unzip cannot delete directory in Jenkins Pipeline

This is what I'm getting when I run AWS terraform plan with Jenkins. Below code that we are using
Error: error: cannot delete old terraform
Is a directory
Code :
sh '''set +x
curl -L 'https://releases.hashicorp.com/terraform/0.11.10/terraform_0.11.10_linux_amd64.zip' --output terraform.zip
unzip -o terraform.zip
echo "Using $(terraform -version) from: $(which terraform)"
'''
sh "terraform init -backend-config='bucket=${bucketName}'"
Jenkins Error:
+ set +x
after terraform download
Archive: terraform.zip
error: cannot delete old terraform
Is a directory
[Pipeline] End of Pipeline
ERROR: script returned exit code 50
Finished: FAILURE
Please suggest some better solution.
Unzip refuses to overwrite the terraform/ directory that seems to be still lying around in your workspace from the previous run.
Run either a sh "rm -rf terraform/" before the unzip (or cleanWs())
unzip -f terraform.zip
Use -f instead of -o
-f freshen existing files, create none i.e unzip to replace the new files only
-n never overwrite existing files
-q quiet mode (-qq => quieter)
-o overwrite files WITHOUT prompting

How can i execute an shell script in my own jenkins pipeline plugin?

my problem is that i want to execute an script inside my jenkins pipeline plugin, and the 'perf script' command do not work.
My script is:
#! /bin/bash
if test $# -lt 2
then
sudo perf record -F 99 -a -g -- sleep 20
sudo perf script > info.perf
echo "voila"
fi
exit 0
My Jenkins can execute sudo so this is not the problem, and in my own Linux Shell this script works perfectly..
How can i solve this?
I solved this adding the -i option to perf script command:
sudo perf record -F 99 -a -g -- sleep 20
sudo perf script -i perf.data > info.perf
echo "voila"
Seems like Jenkins is not able to read perf.data without -i option
If the redirection does not work within the script, try and see if it is working within the DSL Jenkinsfile.
If you call that script with the sh step supports returnStdout (JENKINS-26133):
res = sh(returnStdout: true, script: '/path/to/your/bash/script').trim()
You could process the result directly in res, bypassing the need for a file.

jenkins pipeline: multiline shell commands with pipe

I am trying to create a Jenkins pipeline where I need to execute multiple shell commands and use the result of one command in the next command or so. I found that wrapping the commands in a pair of three single quotes ''' can accomplish the same. However, I am facing issues while using pipe to feed output of one command to another command. For example
stage('Test') {
sh '''
echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r '.public_url'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r '.code'`
echo $RESULT
'''
}
Commands with pipe are not working properly. Here is the jenkins console output:
+ echo Executing Tests
Executing Tests
+ curl -s http://localhost:4040/api/tunnels/command_line
+ jq -r .public_url
+ URL=null
+ echo null
null
+ curl -sPOST https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=null
I tried entering all these commands in the jenkins snippet generator for pipeline and it gave the following output:
sh ''' echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r \'.public_url\'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r \'.code\'`
echo $RESULT
'''
Notice the escaped single quotes in the commands jq -r \'.public_url\' and jq -r \'.code\'. Using the code this way solved the problem
UPDATE: : After a while even that started to give problems. There were certain commands executing prior to these commands. One of them was grunt serve and the other was ./ngrok http 9000. I added some delay after each of these commands and it solved the problem for now.
The following scenario shows a real example that may need to use multiline shell commands. Which is, say you are using a plugin like Publish Over SSH and you need to execute a set of commands in the destination host in a single SSH session:
stage ('Prepare destination host') {
sh '''
ssh -t -t user#host 'bash -s << 'ENDSSH'
if [[ -d "/path/to/some/directory/" ]];
then
rm -f /path/to/some/directory/*.jar
else
sudo mkdir -p /path/to/some/directory/
sudo chmod -R 755 /path/to/some/directory/
sudo chown -R user:user /path/to/some/directory/
fi
ENDSSH'
'''
}
Special Notes:
The last ENDSSH' should not have any characters before it. So it
should be at the starting position of a new line.
use ssh -t -t if you have sudo within the remote shell command
I split the commands with &&
node {
FOO = world
stage('Preparation') { // for display purposes
sh "ls -a && pwd && echo ${FOO}"
}
}
The example outputs:
- ls -a (the files in your workspace
- pwd (location workspace)
- echo world

Resources