I'm attempting to update ECS task and service by using the following stage in my pipeline:
stage('Update ECS Task Definition & Service') {
steps {
dir('foo') {
sh """
TASK_DEFINITION=\$(aws ecs describe-task-definition --task-definition \"${env.TASK_NAME}\" --region \"${env.AWS_REGION}\")
NEW_TASK_DEFINITION=\$(echo \${TASK_DEFINITION} | jq '.taskDefinition | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities) | del(.registeredAt) | del(.registeredBy)')
NEW_REVISION=\$(aws ecs register-task-definition --region \"${env.AWS_REGION}\" --cli-input-json \"${NEW_TASK_DEFINITION}\")
NEW_REVISION_DATA=\$(echo \${NEW_REVISION} | jq '.taskDefinition.revision')
echo "New Revision for \${env.TASK_NAME} is: \${NEW_REVISION_DATA}"
"""
}
}
}
This produces the following error:
groovy.lang.MissingPropertyException: No such property: NEW_TASK_DEFINITION for class: groovy.lang.Binding
Am I not escaping correctly? I'm not sure the cleanest way to write this.
Note: env.TASK_NAME and env.AWS_REGION are environment variables earlier declared in the pipeline.
This should work. Take a look at the following working pipeline.
pipeline {
agent any
stages {
stage('Example') {
steps {
script {
sh """
VAR1=\$(echo "abcd123")
VAR2=\$(echo \$VAR1)
echo \$VAR1
echo \$VAR2
"""
}
}
}
}
}
If you are not passing any Pipeline variables into the shell block, you can simply use three single quotes(''') instead of """. In that case, you don't have to escape the $.
pipeline {
agent any
stages {
stage('Example') {
steps {
script {
sh '''
VAR1=$(echo "abcd123")
VAR2=$VAR1
echo $VAR1
echo $VAR2
'''
}
}
}
}
}
Update
You should be escaping the $ before the NEW_TASK_DEFINITION check the following.
NEW_REVISION=\$(aws ecs register-task-definition --region \"${env.AWS_REGION}\" --cli-input-json \"\${NEW_TASK_DEFINITION}\")
Related
def pname = "netstat -ntlp|grep 8080|awk '{printf \$7}'|cut -d/ -f2"
sh "echo $pname" \ java
if ("java".equals(pname)) { sh "echo 1111" }
The process corresponding to port 8080 is a java process, and the 2nd line print "java". But the body of the if statement just doesn't execute.
You seem to be not executing the command correctly. Please refer to the following sample. Please note the returnStdout: true to return output of the command.
pipeline {
agent any
stages {
stage('Test') {
steps {
script {
def pname = sh(returnStdout: true, script: "netstat -ntlp|grep 8080|awk '{printf \$7}'|cut -d/ -f2").trim()
if (pname == "java") {
echo "echo 1111"
}
}
}
}
}
}
try
"==" for equal
or you can read doc.
https://groovy-lang.org/operators.html#_relational_operators
I am looking to use a database username/password in my config.ini file. I have the following withCredentials line in my Jenkinsfile:
withCredentials([usernamePassword(credentialsId: 'database', usernameVariable: 'DATABASE_USER', passwordVariable: 'DATABASE_PASSWORD')])
I don't explicitly call this config.ini file in my Jenkinsfile, however I do use a bash script to:
export CONFIG_FILE='config.ini'
Is there any way to set these accordingly in my config.ini:
DB_USERNAME = {DATABASE_USER}
DB_PASSWORD = {DATABASE_PASSWORD}
Bash can do this for you. You have two options:
Use envsubst. You'll need to install it on all of your nodes (it's usually part of the gettext package).
Use evil eval
Full example:
pipeline {
agent {
label 'linux' // make sure we're running on Linux
}
environment {
USER = 'theuser'
PASSWORD = 'thepassword'
}
stages {
stage('Write Config') {
steps {
sh 'echo -n "user=$USER\npassword=$PASSWORD" > config.ini'
}
}
stage('Envsubst') {
steps {
sh 'cat config.ini | envsubst > config_envsubst.ini'
sh 'cat config_envsubst.ini'
}
}
stage('Eval') {
steps {
sh 'eval "echo \"$(cat config.ini)\"" > config_eval.ini'
sh 'cat config_eval.ini'
}
}
}
}
This this Stackexchange question for more options.
I'm trying to optimize my pipeline. I'm using the pipeline to generate and deploy some docs. At the end I clear my document root and write the newly generated docs into the document root. I'm doing this for several stages in parallel.
o-----o-----o--+--o--+---+--o--+-----o
| | | |
+--o--+ +--o--+
| | | |
+--o--+ +--o--+
this is the pipeline exerpt for the parallel stages
stage("clear nfs directory") {
steps {
parallel(
test: {
sh "rm -rf /mnt/nfs/test/docs/$pipelineParams.groupname"
sh "mkdir /mnt/nfs/test/docs/$pipelineParams.groupname"
},
rele: {
sh "rm -rf /mnt/nfs/rele/docs/$pipelineParams.groupname"
sh "mkdir /mnt/nfs/rele/docs/$pipelineParams.groupname"
},
prod: {
sh "rm -rf /mnt/nfs/prod/docs/$pipelineParams.groupname"
sh "mkdir /mnt/nfs/prod/docs/$pipelineParams.groupname"
}
)
}
}
stage("copy generated docs to nfs directory") {
steps {
parallel(
test: {
dir("target/public") {
sh "cp -r * /mnt/nfs/test/docs/$pipelineParams.groupname"
}
},
rele: {
dir("target/public") {
sh "cp -r * /mnt/nfs/rele/docs/$pipelineParams.groupname"
}
},
prod: {
dir("target/public") {
sh "cp -r * /mnt/nfs/prod/docs/$pipelineParams.groupname"
}
}
)
}
}
Since clear and write should depend on each other I would like to refactor the pipeline into a more sequential design (running multiple steps in sequence in less parallel steps)
o-----o-----o--+--o---o--+-----o
| |
+--o---o--+
| |
+--o---o--+
I'm not sure how to run multiple steps in the same parallel block ... can anyone give me a hint? Thanks guys
Please see below reference which will allow you to run multiple steps in same parallel block.
You would need to use sequential stages which will give below output :
o-----o-----o--+--o---o--+-----o
| |
+--o---o--+
| |
+--o---o--+
pipeline {
agent { label 'master' }
stages {
stage('Build and Test') {
parallel {
stage("Build and Test Linux") {
stages {
stage("Build (Linux)") {
agent any
steps {
echo "Inside for loop 1"
}
}
stage("Test (Linux)") {
agent any
steps {
echo "Inside for loop 2"
}
}
}
}
stage("Build and Test Windows") {
stages {
stage("Build (Windows)") {
agent any
steps {
echo "Inside for loop 3"
}
}
stage("Test (Windows)") {
agent any
steps {
echo "Inside for loop 4"
}
}
}
}
}
}
}
}
For more info see:-
https://www.jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/
Below link gives reference example:
https://issues.jenkins.io/browse/JENKINS-55438
I have a Jenkins pipeline which needs to run on a slave node. I curently have issues with passing Variables set by plugin withCredentials. When I try to use them on the slave node they are empty, but they work on the master.
Here is the pipeline snippet.
#!groovy
#Library('sharedPipelineLib#master') _
pipeline {
agent { node
{ label 'jenkins-slave-docker' }
}
options {
skipDefaultCheckout(true)
}
environment {
sonar = credentials('SONAR')
}
stages {
stage('Checkout') {
steps {
cleanWs()
script {
checkout scm
}
}
}
stage('Deploy backend') {
steps {
script {
withCredentials([
[
$class : 'AmazonWebServicesCredentialsBinding',
credentialsId : 'AWS_ACCOUNT_ID_DEV',
accessKeyVariable: 'AWS_ACCESS_KEY_ID_DEV',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY_DEV'
],
[
$class : 'AmazonWebServicesCredentialsBinding',
credentialsId : 'AWS_ACCOUNT_ID_DNS',
accessKeyVariable: 'AWS_ACCESS_KEY_ID_DNS',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY_DNS'
]
]){
sh '''
echo "$AWS_ACCESS_KEY_ID_DEV\\n$AWS_SECRET_ACCESS_KEY_DEV\\n\\n" | aws configure --profile profile_705229686812
echo "$AWS_ACCESS_KEY_ID_DNS\\n$AWS_SECRET_ACCESS_KEY_DNS\\n\\n" | aws configure --profile profile_417752960097
'''
}
}
}
}
}
}
And the log
[Pipeline] withCredentials
Masking supported pattern matches of $AWS_ACCESS_KEY_ID_DEV or $AWS_SECRET_ACCESS_KEY_DEV or $AWS_SECRET_ACCESS_KEY_DNS or $AWS_ACCESS_KEY_ID_DNS
[Pipeline] {
[Pipeline] sh
echo '\n\n\n'
aws configure --profile profile_705229686812
AWS Access Key ID [None]: AWS Secret Access Key [None]:
EOF when reading a line
the issue was again echo cmd. I had to uses printf instead, cause echo adds newline which causes to fail.
I have defined global variable in Jenkins pipeline
def BUILDNRO = '0'
pipeline { ...
Then i manipulate variable with shell script to enable running builds parallel by using job build number as identifier so we don't mix different docker swarms.
stage('Handle BUILD_NUMBER') {
steps {
script {
BUILDNRO = sh( script: '''#!/bin/bash
Build=`echo ${BUILD_NUMBER} | grep -o '..$'`
# Check if BUILD first character is 0
if [[ $Build:0:1 == "0" ]]; then
# replace BUILD first character from 0 to 5
Build=`echo $Build | sed s/./5/1`
fi
echo $Build
''',returnStdout: true).trim()
}
}
}
i get value out from previos stage and trying to get global variable on next stage
stage('DOCKER: Init docker swarm') {
steps {
echo "BUILDNRO is: ${BUILDNRO}" --> Value is here.
sh '''#!/bin/bash
echo Buildnro is: ${BUILDNRO} --> This is empty.
...
}
}
This will out give global variable empty. why? in previous stage there was value in it.
EDIT 1.
Modified code blocks to reflect current status.
I managed to figure it out. Here is solution how i managed to did it.
BUILDNRO is groovy variable and if wanting to used in bash variable it have to pass using withEnv. BUILD_NUMBER in first stage is bash variable hence it can be used directly script in first stage.
def BUILDNRO = '0'
pipeline {
....
stages {
stage('Handle BUILD_NUMBER') {
steps {
script {
BUILDNRO = sh( script: '''#!/bin/bash
Build=`echo ${BUILD_NUMBER} | grep -o '..$'`
''',returnStdout: true).trim()
}
}
}
stage('DOCKER: Init docker swarm') {
steps {
dir("prose_env/prose_api_dev_env") {
withEnv(["MYNRO=${BUILDNRO}"]) {
sh(returnStdout: false, script: '''#!/bin/bash
echo Buildnro is: ${MYNRO}`
'''.stripIndent())
}
}
}
}
}
}
If you are using single quotes(```) in the shell module, Jenkins treats every variable as a bash variable. The solution is using double quotes(""") but then if you made bash variable you have to escape it. Below an example with working your use case and escaped bash variable
pipeline {
agent any
stages {
stage('Handle BUILD_NUMBER') {
steps {
script {
BUILDNRO = sh(script: 'pwd', returnStdout: true).trim()
echo "BUILDNRO is: ${BUILDNRO}"
}
}
}
stage('DOCKER: Init docker swarm') {
steps {
sh """#!/bin/bash
echo Buildnro is: ${BUILDNRO}
variable=world
echo "hello \${variable}"
sh """
}
}
}
}
output of the second stage:
Buildnro is: /var/lib/jenkins/workspace/stack1
hello world