How can I export some variables out of an sh block so they can be used in later stages?
The following does not give me any errors but the values are never available as environment variables in later stages.
steps {
sh """
ASSUME_ROLE_RESPONSE=\$(aws sts assume-role --role-arn "arn:aws:iam::${env.NON_PROD_ACCOUNT_ID}:role/${env.AWS_ROLE}" --role-session-name "${env.AWS_ROLE_SESSION}" --duration-seconds 3600)
${env.ACCESS_KEY_ID}=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.AccessKeyId')
${env.SECRET_ACCESS_KEY}=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.SecretAccessKey')
${env.SESSION_TOKEN}=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.SessionToken')
echo "AWS_ACCESS_KEY_ID=${ACCESS_KEY_ID},AWS_SECRET_ACCESS_KEY=${SECRET_ACCESS_KEY},AWS_SESSION_TOKEN=${SESSION_TOKEN}"
printenv | sort
"""
}
I have got this working but I cannot say it is elegant, if someone has a better\cleaner answer I would happily accept it.
Here is my solution:
stage("Authenticate To Non-Prod Account") {
steps {
script {
aws_credentials = sh(script: """
ASSUME_ROLE_RESPONSE=\$(aws sts assume-role --role-arn "arn:aws:iam::${env.NON_PROD_ACCOUNT_ID}:role/${env.AWS_ROLE}" --role-session-name "${env.AWS_ROLE_SESSION}" --duration-seconds 3600)
ACCESS_KEY_ID=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.AccessKeyId')
SECRET_ACCESS_KEY=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.SecretAccessKey')
SESSION_TOKEN=\$(echo \$ASSUME_ROLE_RESPONSE | jq --raw-output '.Credentials.SessionToken')
echo "AWS_ACCESS_KEY_ID=\$ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY=\$SECRET_ACCESS_KEY,AWS_SESSION_TOKEN=\$SESSION_TOKEN"
""", returnStdout: true)
env.ACCESS_KEY_ID = aws_credentials.split(',')[0].split('=')[1].trim()
env.AWS_SECRET_KEY = aws_credentials.split(',')[1].split('=')[1].trim()
env.SESSION_TOKEN = aws_credentials.split(',')[2].split('=')[1].trim()
}
}
}
I am answering as I have read a heap of posts which have suggested ideas that didnt work for me so this, for now, is the best option I have for authentication to AWS and ensuring the credentials are available in subsequent stages.
To export vars out of sh try using this:
env.var = sh (returnStdout: true, script: ''' SOME SH COMMAND ''').trim()
This will export your bash values to groovy variables in fact it can push the vars to the environment.
Related
I have created several bash scripts that work perfect in the Linux shell, but when I try to incorporate them in a Jenkins Pipeline I get multiple errors, I attach an example of my Pipeline where I just want to show the value of my variables, the pipeline works fine except when I added in line 5 the environment, you can see that there are special characters that are not interpreted by Groovy as the Bash does.
pipeline {
agent {
label params.LABS == "any" ? "" : params.LABS
}
environment{
PORT_INSTANCE="${docker ps --format 'table {{ .Names }} \{{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf "%s\n"; sort -k 3) | grep web | tail -1 | sed 's/.*0.0.0.0.0://g'|sed 's/->.*//g'}"
}
stages {
stage('Setup parameters') {
steps {
script {
properties([
parameters([
choice(
choices: ['LAB-2', 'LAB-3'],
name: 'LABS'
),
string(
defaultValue: 'cliente-1',
name: 'INSTANCE_NAME',
trim: true
),
string(
defaultValue: '8888',
name: 'PORT_NUMBER',
trim: true
),
string(
defaultValue: 'lab.domain.com',
name: 'DOMAIN_NAME',
trim: true
)
])
])
}
sh """
echo '${params.INSTANCE_NAME}'
echo '${params.PORT_NUMBER}'
echo '${params.DOMAIN_NAME}'
echo '${PORT_INSTANCE}
"""
}
}
}
}
I already tried the same thing from the sh section """ command """ and they throw the same errors.
Can someone help me to know how to run advanced commands that work in the linux shell (bash), that is, is there any way to migrate scripts from bash to Jenkins?
Thank you very much for your help ;)
I want to be able to create a variable from a bash script command from the Pipeline in Jenkins
PORT_INSTANCE="${docker ps --format 'table {{ .Names }} {{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf "%s\n"; sort -k 3) | grep web | tail -1 | sed 's/.0.0.0.0.0://g'|sed 's/->.//g'}"
I believe that you can't execute a bash script in the environment step based on the documentation.
You can create a variable from a bash script using the sh step with returnStdout set to true. Declarative pipeline doesn't allow you to assign the retrun value to a variable, so you will need to call sh inside a script like this:
stage('Calculate port') {
steps {
script {
// When you don't use `def` in front of a variable, you implicitly create a global variable
// This means that the variable will exist with a value, and can be used in any following line in your scipt
PORT_INSTANCE = sh returnStdout: true, script: "docker ps --format 'table {{ .Names }} \{{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf \"%s\n\"; sort -k 3) | grep web | tail -1 | sed 's/.*0.0.0.0.0://g'|sed 's/->.*//g'"
// Shell output will contain a new line character at the end, remove it
PORT_INSTANCE = PORT_INSTANCE.trim()
}
}
}
I would add a stage like this, as the first stage in my pipeline.
Note that I didn't run the same shell command as you when I was testing this, so my command may have issues like un-escaped quotes.
Hi i have a following jenkins pipeline like below.
pipeline {
agent any
environment {
//JSON_NAME = sh(returnStdout: true, script: "sed -n '2 p' package.json | awk '{print \$2}' | sed 's/\\,//g'").trim()
JSON_NAME = sh(returnStdout: true, script: "sed -n '2 p' package.json | awk '{print \$2}' | sed 's/\\,//g' | awk -F "/" '{print \$2}'").trim()
}
stages {
stage ('Update Italy.json') {
when {expression { fileExists('italy.json')}}
steps {
sh "echo ${JSON_NAME}"
}
}
}
}
As you can see i have to env in ENVIRONMENT block.
First one is commented, and its working.
But when i try use my second ENV it gives me error.
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.lang.String.div() is applicable for argument types: (java.lang.String) values: [ '{print $2}']
I couldnt figure out what wrong with this env. Any ideas ? Thanks in advance
I am having problems figuring out how to pass some variables into the parallel runs in the Jenkins groovy script below:
#!/usr/bin/env groovy
def call(version, project) {
sh '''#!/bin/bash
[[ ! -e ${WORKSPACE}/target/rpm/${project}/RPMS/ ]] && mkdir -p ${WORKSPACE}/target/rpm/${project}/RPMS/
(( $(ls ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm | wc -l) != 0 )) && rm ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm
cd ${WORKSPACE}/scripts/fpm_requirements && bundle install && bundle show fpm
'''
parallel (
"package foo": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
},
"package bar": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
}
)
}
So the version and project variables are populated in the first shell that is called but when they hit the two parallel runs they are not being pulled in.
I have tried a few different options to pass them in but none have worked.
Does anyone have any relevant ideas that might help?
You should change the ''' to """. In Groovy, string inside single/triple quote won't trigger string interpolation, but string inside single/triple double quote will do that.
So the ${version} and ${project} in your Shell script will be treated as variable from Shell context, but actually they are exist in Groovy context.
More about Groovy String at here, Below option 2 more suitable for your issue.
Option 1) using "" or """
"package foo": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
},
"package bar": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
}
Attention: need to escape the $ ahead of ${WORKSPACE} and $(echo ..), because we hope $ be kept after interpolation.
Option 2) using ' or ''' and inject version and project into Environment Variables of Shell context.
def call(version, project) {
env.version=version
env.project=project
// Groovy env api used to inject groovy value into environment variable
// so that you can refer groovy value later in shell script
// still use ''' in following code, no need to change
...
Script in package.json:
"scripts": {
"version": "echo $npm_package_version"
},
One of the stage in Jenkins pipeline:
stage('Build'){
sh 'npm install'
def packageVersion = sh 'npm run version'
echo $packageVersion
sh 'VERSION=${packageVersion} npm run build'
}
I got version from npm script output, but following line
echo $packageVersion
return null
Is packageVersion value not correctly assigned?
Edited:
when using Pipeline Utility Steps with following code
stage('Build'){
def packageJSON = readJSON file: 'package.json'
def packageJSONVersion = packageJSON.version
echo packageJSONVersion
sh 'VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build'
}
I get
[Pipeline] echo
1.1.0
[Pipeline] sh
[...] Running shell script + VERSION=_16_SOME_BRANCH npm run build
So I am able to extract version, but still cannot pass it when running script
After your edit using readJSON, you now get string interpolation wrong. Variables within single quotes are not replaced in Groovy, only within double quotes.
sh 'VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build'
must be
sh "VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build"
The sh step by default returns nothing, so packageVersion should be null.
To return the output of the executed command, use it like this:
sh(script: 'npm run version', returnStdout: true)
This variant of sh returns the output instead of printing it.
Actually, I am wondering, why echo $packageVersion doesn't fail with an error, as this variable is not defined, but should be echo packageVersion.
For my it this:
sh(script: "grep \"version\" package.json | cut -d '\"' -f4 | tr -d '[[:space:]]'", returnStdout: true)
This worked for me:
Full version number:
PACKAGE_VERSION = sh returnStdout: true, script: '''grep 'version' package.json | cut -d '"' -f4 | tr '\n' '\0''''
echo "Current package version: $PACKAGE_VERSION"
$ > 1.2.3
Major version only:
PACKAGE_VERSION = sh returnStdout: true, script: '''grep 'version' package.json | cut -d '"' -f4 | cut -d '.' -f1 | tr '\n' '\0''''
echo "Current package Major version: $PACKAGE_VERSION"
$ > 1
stage('Read JSON') {
steps {
script {
def packageJson = readJSON file: 'package.json'
def packageVersion = packageJSON.version
echo "${packageJSONVersion}"
}
}
}
Below snippet worked for me: Credits to #Ferenc Takacs
version = sh(returnStdout: true, script: "grep 'version' package.json | cut -d '"' -f4 | tr '\n' '\0'")
This command will take exact property version: ... in package.json and can work on both Mac and Linux. The other solutions using grep will not give you correct answer in case of you have > 1 version keyword in your package.json (it'll return all of them instead of just the one you want)
awk -F'"' '/"version": ".+"/{ print $4; exit; }' package.json
To access an npm environment variable outside the scope of a run-script, parse the variable with bash:
$ npm run env | grep npm_package_version | cut -d '=' -f 2
Author: https://remarkablemark.org/blog/2018/08/14/package-json-version/
I am using a Jenkinsfile in a pipeline on version 2.32.2.
For various reasons I want to extract the version string from the pom. I was hoping I wouldn't have to add the maven help plugin and use evaluate.
I quickly came up with a little sed expression to get it out of the pom which uses pipes and works on the commandline in the jenkins workspace on the executor.
$ sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
1.0.0-SNAPSHOT
It could probably be optimized, but I want to understand why the pipeline seems to be failing on piped sh commands. I've played with various string formats and am currently using a dollar slashy string.
The pipeline step looks like the following to allow for easy output of the command string:
script {
def ver_script = $/sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
When run in the jenkins pipeline I get the following console output where it seems to be separating the piped commands into separate commands:
[Pipeline] script
[Pipeline] {
[Pipeline] echo
sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
[Pipeline] sh
[FRA-198-versioned-artifacts-44SD6DBQOGOI54UEF7NYE4ECARE7RMF7VQYXDPBVFOHS5CMSTFLA] Running shell script
+ sed -n /<version>/,/<version/p pom.xml
+ head -1
+ sed s/[[:blank:]]*<\/*version>//g
sed: couldn't write 89 items to stdout: Broken pipe
[Pipeline] }
[Pipeline] // script
Any guidance out there on how to properly use piped commands in a jenkinsfile ?
I finally put some thought into it and realized that pipe subshells are probably causing the issue. I know some of the evils of eval but I ended up wrappping this in an eval:
script {
def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
I know this kind of late answer, but whoever you who needs the solution without eval you can use /bin/bash -c "script" to make pipe works
script {
POM_VERSION = sh(script: "/bin/bash -c 'sed -n \'/<version>/,/<version/p\' pom.xml | head -1 | sed \'s/[[:blank:]]*<\/*version>//g\'\''", returnStdout: true)
echo "${POM_VERSION}"
}
The only problem with this method is hellish escape yet this way the subshell of pipe will be handled by our boy /bin/bash -c
If your environment allows it, I've found a simple solution to this problem to be to place your script containing pipes into a file, and then run that with sh, like so:
script.sh:
#!/bin/sh
kubectl exec --container bla -i $(kubectl get pods | awk '/foo-/{ print $1 }') -- php /code/dostuff
Jenkinsfile:
stage('Run script with pipes') {
steps {
sh "./script.sh"
}
}
The pipeline-utility-steps plugin nowadays includes a readMavenPom step, which allows to access the version as follows:
version = readMavenPom.getVersion()
So nothing detailed above worked for me using the scripted Jenkinsfile syntax with Groovy. I was able to get it working, however. The type of quotations you use are important. In the example below, I am trying to fetch the latest git tag from GitHub.
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
Notice how the shell execution to assign LATEST_TAG works as expected (assigning the variable to v2.1.0). If we were to try the same thing (with single quotes) to assign VERSION_NUMBER, it would NOT work - the pipe messes everything up. Instead, we wrap the script in double quotes.
The first echo prints VERSION_NUMBER: 2.1.0 but the second prints VERSION_NUMBER:. If you want VERSION_NUMBER to be available in the shell commands, you have to assign the output of the shell command to env.VERSION_NUMBER as shown below:
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
env.VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
The first echo prints VERSION_NUMBER: 2.1.0 and the second prints VERSION_NUMBER: 2.1.0.
I am also struggling with the usage of pipe inside my jenkins pipeline but as a side note, if you want a simple way to extract the version of a maven pom, here's a very clean one I found in another post and that I'm using :
stage('Preparation') {
version = getVersion()
print "version : " + version
}
def getVersion() {
def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
matcher ? matcher[0][1] : null
}
gives you :
[Pipeline] echo
releaseVersion : 0.1.24
[Pipeline] sh