I am trying to create a Jenkins pipeline where I need to execute multiple shell commands and use the result of one command in the next command or so. I found that wrapping the commands in a pair of three single quotes ''' can accomplish the same. However, I am facing issues while using pipe to feed output of one command to another command. For example
stage('Test') {
sh '''
echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r '.public_url'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r '.code'`
echo $RESULT
'''
}
Commands with pipe are not working properly. Here is the jenkins console output:
+ echo Executing Tests
Executing Tests
+ curl -s http://localhost:4040/api/tunnels/command_line
+ jq -r .public_url
+ URL=null
+ echo null
null
+ curl -sPOST https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=null
I tried entering all these commands in the jenkins snippet generator for pipeline and it gave the following output:
sh ''' echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r \'.public_url\'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r \'.code\'`
echo $RESULT
'''
Notice the escaped single quotes in the commands jq -r \'.public_url\' and jq -r \'.code\'. Using the code this way solved the problem
UPDATE: : After a while even that started to give problems. There were certain commands executing prior to these commands. One of them was grunt serve and the other was ./ngrok http 9000. I added some delay after each of these commands and it solved the problem for now.
The following scenario shows a real example that may need to use multiline shell commands. Which is, say you are using a plugin like Publish Over SSH and you need to execute a set of commands in the destination host in a single SSH session:
stage ('Prepare destination host') {
sh '''
ssh -t -t user#host 'bash -s << 'ENDSSH'
if [[ -d "/path/to/some/directory/" ]];
then
rm -f /path/to/some/directory/*.jar
else
sudo mkdir -p /path/to/some/directory/
sudo chmod -R 755 /path/to/some/directory/
sudo chown -R user:user /path/to/some/directory/
fi
ENDSSH'
'''
}
Special Notes:
The last ENDSSH' should not have any characters before it. So it
should be at the starting position of a new line.
use ssh -t -t if you have sudo within the remote shell command
I split the commands with &&
node {
FOO = world
stage('Preparation') { // for display purposes
sh "ls -a && pwd && echo ${FOO}"
}
}
The example outputs:
- ls -a (the files in your workspace
- pwd (location workspace)
- echo world
Related
I have password that looks like this KMqJH9OL?LoNw:w=ZgD1;?zLrH<c. Now when I am trying to input this password into the groovy script it breaks because of ; and <.
I have tried different ways to escape this non of it worked. In the below code BITBUCKET_PASS is the one having the above password.
sh '''
yarn cov-report -c ${BUILD_VERSION} -u ${BITBUCKET_USER} -p ${BITBUCKET_PASS} $PWD/backend/test_report/lcov.info
'''
Here's the code that I have tried and it didn't work.
sh '''
yarn cov-report -c ${BUILD_VERSION} -u ${BITBUCKET_USER} -p "${BITBUCKET_PASS}" $PWD/backend/test_report/lcov.info
'''
Please look at the String interpolation in Groovy scripts :
Instead using {} try like this :
sh '''
yarn cov-report -c $BUILD_VERSION -u $BITBUCKET_USER -p $BITBUCKET_PASS $PWD/backend/test_report/lcov.info
'''
I'm writing a jenkins pipeline jenkinsfile and within the script clause I have to ssh to a box and run some commands. I think the problem has to do with the env vars that I'm using within the quotes. I'm getting a ENDSSH command not found error and I'm at a loss. Any help would be much appreciated.
stage("Checkout my-git-repo"){
steps {
script {
sh """
ssh -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -i ${JENKINS_KEY} ${JENKINS_KEY_USR}#${env.hostname} << ENDSSH
echo 'Removing current /opt/my-git-repo directory'
sudo rm -rf /opt/my-git-repo
echo 'Cloning new my-git-repo repo into /opt'
git clone ssh://${JENKINS_USR}#git.gitbox.com:30303/my-git-repo
sudo mv /home/jenkins/my-git-repo /opt
ENDSSH
"""
}
}
}
-bash: line 6: ENDSSH: command not found
I'm personally not familiar with jenkins, but I'd guess the issue is the whitespace before ENDSSH
White space in front of the delimiter is not allowed.
(https://linuxize.com/post/bash-heredoc/)
Try either removing the indentation:
stage("Checkout my-git-repo"){
steps {
script {
sh """
ssh -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -i ${JENKINS_KEY} ${JENKINS_KEY_USR}#${env.hostname} << ENDSSH
echo 'Removing current /opt/my-git-repo directory'
sudo rm -rf /opt/my-git-repo
echo 'Cloning new my-git-repo repo into /opt'
git clone ssh://${JENKINS_USR}#git.gitbox.com:30303/my-git-repo
sudo mv /home/jenkins/my-git-repo /opt
ENDSSH
"""
}
}
}
OR ensure that the whitespace is only tabs and replace << with <<-:
Appending a minus sign to the redirection operator <<-, will cause all
leading tab characters to be ignored. This allows you to use
indentation when writing here-documents in shell scripts. Leading
whitespace characters are not allowed, only tab.
I am having problems figuring out how to pass some variables into the parallel runs in the Jenkins groovy script below:
#!/usr/bin/env groovy
def call(version, project) {
sh '''#!/bin/bash
[[ ! -e ${WORKSPACE}/target/rpm/${project}/RPMS/ ]] && mkdir -p ${WORKSPACE}/target/rpm/${project}/RPMS/
(( $(ls ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm | wc -l) != 0 )) && rm ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm
cd ${WORKSPACE}/scripts/fpm_requirements && bundle install && bundle show fpm
'''
parallel (
"package foo": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
},
"package bar": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
}
)
}
So the version and project variables are populated in the first shell that is called but when they hit the two parallel runs they are not being pulled in.
I have tried a few different options to pass them in but none have worked.
Does anyone have any relevant ideas that might help?
You should change the ''' to """. In Groovy, string inside single/triple quote won't trigger string interpolation, but string inside single/triple double quote will do that.
So the ${version} and ${project} in your Shell script will be treated as variable from Shell context, but actually they are exist in Groovy context.
More about Groovy String at here, Below option 2 more suitable for your issue.
Option 1) using "" or """
"package foo": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
},
"package bar": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
}
Attention: need to escape the $ ahead of ${WORKSPACE} and $(echo ..), because we hope $ be kept after interpolation.
Option 2) using ' or ''' and inject version and project into Environment Variables of Shell context.
def call(version, project) {
env.version=version
env.project=project
// Groovy env api used to inject groovy value into environment variable
// so that you can refer groovy value later in shell script
// still use ''' in following code, no need to change
...
my problem is that i want to execute an script inside my jenkins pipeline plugin, and the 'perf script' command do not work.
My script is:
#! /bin/bash
if test $# -lt 2
then
sudo perf record -F 99 -a -g -- sleep 20
sudo perf script > info.perf
echo "voila"
fi
exit 0
My Jenkins can execute sudo so this is not the problem, and in my own Linux Shell this script works perfectly..
How can i solve this?
I solved this adding the -i option to perf script command:
sudo perf record -F 99 -a -g -- sleep 20
sudo perf script -i perf.data > info.perf
echo "voila"
Seems like Jenkins is not able to read perf.data without -i option
If the redirection does not work within the script, try and see if it is working within the DSL Jenkinsfile.
If you call that script with the sh step supports returnStdout (JENKINS-26133):
res = sh(returnStdout: true, script: '/path/to/your/bash/script').trim()
You could process the result directly in res, bypassing the need for a file.
I try to execute such a scenery via Jenkins "execute shell" build step:
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt";
for file in $(<"_dpatch/deploy/files.txt"); do cp --parents "$file" "_dpatch"; done;
whoami
Build ends successfully with console output:
[Deploy to production] $ /bin/sh -xe /tmp/hudson8315034696077699718.sh
+ rm -r -f _dpatch
+ mkdir _dpatch
+ mkdir _dpatch/deploy
+ from_revision=HEAD
+ to_revision=2766920
+ git diff --name-only HEAD 2766920
+
+ whoami
jenkins
Finished: SUCCESS
The problem is line "for file in" is just ignored, I do not understand why.
Content of files.txt is not empty and looks like this:
addons/tiny_mce/plugins/image/plugin.min.org.js
addons/webrtc/adapter-latest.js
templates/standard/style/review.css
More over, when I execute via ssh the same script in the same jenkins workspace folder under the same user (jenkins) - "for file in" line executes normally and creates files in "_dpatch" subfolder as it should.
My environment:
Debian 8,
Jenkins 2.45
Thanks
Possibly your /bin/sh is a POSIX bourne shell. It think that the $(< construct is a bash-ism, so it will not work with /bin/sh.
Try to replace
$(<"_dpatch/deploy/files.txt")
with
$(cat "_dpatch/deploy/files.txt")
Alternatively, prepend your build step with #!/bin/bash.
If your login shell is bash, then this also explains why everything works fine via ssh.
Try substituting for with while loop. And also add some more logging
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt" && echo "git diff finished"
while IFS= read -r line; do
echo $line
cp --parent $line $_dpatch
done < _dpatch/deploy/files.txt
whoami