I can run this command from the command line to get the value for the latest release in a GitHub repo:
curl --silent "https://api.github.com/repos/MyOrg/MyRepo/releases/latest"|grep "tag_name"|sed -E 's/."([^"]+)"./\1/'
I'd like to use this in a Jenkinsfile, but I can't seem to properly escape the special characters.
pipeline {
agent any
stages {
stage('Test') {
steps {
sh """
latest=`curl --silent \\\"https://api.github.com/repos/MyOrg/MyRepo/releases/latest\\\"|
grep \\\"tag_name\\\"|
sed -E \\\'s/.*\\\"([^\\\"]+)\\\".*/\1/\\\'`
""".stripIndent()
}
}
}
}
When run in a pipeline, I get the following error:
[Pipeline] sh
curl --silent "https://api.github.com/repos/MyOrg/MyRepo/releases/latest"
grep "tag_name"
sed -E 's/.*"
sed: -e expression #1, char 1: unknown command: `''
instead of wrapping the large expression in backticks, try instead wrapping it in
$(command_1 | command_2 | command_3)
Also add, at the start of the shell command:
set -x;
and you'll get debug output in the console output (possibly). You should then see how your script is being evaluated.
Related
I am not sure if its appropriate and if there is a way to continue posting question the existing thread as this:
sed error: "invalid reference \1 on `s' command's RHS"
and
Invalid reference \1 using sed when trying to print matching expression
The issue I am facing is with the Jenkins (2.249.1) pipeline script:
pipeline {
agent any
stages {
stage("exp") {
steps {
script {
def output=sh(returnStdout: true, script: "echo ab4d | sed 's/b\\(([0-9]\\))/B\\1/' ").trim()
echo "output=$output";
}
}
}
}
}
When run the build, I get this o/p where I was expecting only "B4"
+ echo ab4d
+ sed 's/b\(([0-9]\))/B\1/'
[Pipeline] echo
output=ab4d
The second version that I tried was :
def output=sh(returnStdout: true, script: "echo ab4d | sed 's/b\([0-9]\)/B\1/' ").trim()
+ echo ab4d
+ sed 's/b\([0-9]\)/B\1/'
[Pipeline] echo
output=aB4d
With the third version: def output=sh(returnStdout: true, script: "echo ab4d | sed 's/b([0-9])/B\1/' ").trim()
+ echo ab4d
+ sed 's/b([0-9])/B\1/'
sed: -e expression #1, char 15: invalid reference \1 on `s' command's RHS
The output I want is just B4. Please let me know the right approach to fix this issue, and help me understand sed in Pipeline
PS: I am not sure what my Jenkins supports and what it doesn't with respective to using -r option, etc... So I purposely avoiding it until I know exactly if its a must.
I am tring to run an ansible shell script thorugh a groovy script in Jenkins. But I am getting weird syntax error when I use special characters like $ or . I tried to use the escape sequence but still getting an error. It works fine if I remove the JAVA_OPTS variable.
batch_service_url="http://DEV:8080/test"
JAVA_OPTS="\$JAVA_OPTS -Dactivemq.tcp.url=failover:\(tcp://DEV1:61616,tcp://DEV1:61616\)?nested.wireFormat.maxInactivityDuration=30000"
def test(){
sh """sudo w360ansibleint <<EOF
ansible-playbook -i ansible/ANS-5.2.0/hosts ansible/ANS-5.2.0/app_config.yml -e '{
"ansible_hostname":"${ansible_hostname}",
"tomcat_app_parameters":"base",
"batch_service_url":"${batch_service_url}",
"tomcat_setenv_extra": ["\\$JAVA_OPTS"]
}'
EOF
"""
}
I have a file 'README.txt' which contains the line -
"version": "1.0.0-alpha-test.7"
Using a Jenkins Pipeline, I want to replace this line with
"version": "1.0.0-alpha-test.{BUILD_NUMBER}"
The following sed command works when I try it on a linux cluster
sed -i -E "s#(\"version\"[ ]*:[ ]*\".+alpha-test\.)[0-9]+\"#\1${BUILD_NUMBER}#g" README.txt
The same command does not work using a Jenkins Pipeline.
Tried with the following query but it doesn't work -
sh """
sed -i -E "s|([\"]version[\"][ ]*:[ ]*[\"].+alpha-test\\.)[0-9]+\"|\1${BUILD_NUMBER}|g" README.txt
cat README.txt
"""
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
3:
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
Syntax error: ")" unexpected
Best to use perl command instead...
script{
old_version = (sh (returnStdout: true, script:'''old_version=`cat version.cfg |grep VERSION=|cut -d "=" -f2`
echo $old_version''')).toString().trim()
sh """
if [ "$old_version" != $new_version ]; then
perl -pi -e "s,$old_version,$new_version,g" version.cfg
##-- git push operation --##
fi
"""
}
So writing Groovy with basic shell scripts seem to be much more difficult than it really should be.
I have a pipeline that needs to replace an entry in a file after running a packer command. It seems sensible to do this in the same shell script as the packer command as the variables are not available outside of the shell script even when exported.
The problem is that the sed command needs escape upon escape and still doesn't work. So this is what the Jenkins Pipeline Syntax generator suggested:
parallel (
"build my-application" : {
sh '''#!/bin/bash
export PATH=$PATH:~/bin
cd ${WORKSPACE}/platform/packer
packer build -machine-readable template.json | tee packer.out
AMI_APP=$(grep amazon-ebs,artifact,0,id,eu-west-2:ami- packer.out | awk -F: \'{ print $NF }\')
[[ ! ${AMI_APP} ]] && exit 1
sed -i.bak \'s!aws_ami_app = \\".*\\"!aws_ami_app = \\"\'"${AMI_APP}"\'\\"!\' ${WORKSPACE}/platform/terraform/env-${ENV}/env.auto.tfvars
'''
},
"build some-more-apps" : {
sh ''' *** same again different name ***
'''
}
)
What is the correct way to get a variable is a sed command working in a bash script running in groovy?
Any tips for the correct syntax going forward with Jenkins, groovy and bash - any documentation that actually helps?
EDIT
The original sed command that is running in a Jenkins Job shell is:
sed -i.bak 's!aws_ami_app = \".*\"!aws_ami_app = \"'"${AMI_APP}"'\"!' ${WORKSPACE}/platform/terraform/env-${ENV}/env.auto.tfvars
Because you put the shell script inside ''' which won't trigger Groovy String interpolation.
So you no need to escape any character, write the script as when you typing in Shell cmd window.
Below is example:
sh '''#!/bin/bash +x
echo "aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.23" > test.txt
echo "cpu = 512" >> test.txt
cat test.txt
AMI_APP=docker.xy.com/xy-ap123/conn:7et45u.1.25
sed -i 's,aws_ami_app.*,aws_ami_app = '"$AMI_APP"',' test.txt
cat test.txt
'''
Output in jenkins console:
[Pipeline] sh
[poc] Running shell script
aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.23
cpu = 512
aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.25
cpu = 512
I am trying to create a Jenkins pipeline where I need to execute multiple shell commands and use the result of one command in the next command or so. I found that wrapping the commands in a pair of three single quotes ''' can accomplish the same. However, I am facing issues while using pipe to feed output of one command to another command. For example
stage('Test') {
sh '''
echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r '.public_url'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r '.code'`
echo $RESULT
'''
}
Commands with pipe are not working properly. Here is the jenkins console output:
+ echo Executing Tests
Executing Tests
+ curl -s http://localhost:4040/api/tunnels/command_line
+ jq -r .public_url
+ URL=null
+ echo null
null
+ curl -sPOST https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=null
I tried entering all these commands in the jenkins snippet generator for pipeline and it gave the following output:
sh ''' echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r \'.public_url\'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r \'.code\'`
echo $RESULT
'''
Notice the escaped single quotes in the commands jq -r \'.public_url\' and jq -r \'.code\'. Using the code this way solved the problem
UPDATE: : After a while even that started to give problems. There were certain commands executing prior to these commands. One of them was grunt serve and the other was ./ngrok http 9000. I added some delay after each of these commands and it solved the problem for now.
The following scenario shows a real example that may need to use multiline shell commands. Which is, say you are using a plugin like Publish Over SSH and you need to execute a set of commands in the destination host in a single SSH session:
stage ('Prepare destination host') {
sh '''
ssh -t -t user#host 'bash -s << 'ENDSSH'
if [[ -d "/path/to/some/directory/" ]];
then
rm -f /path/to/some/directory/*.jar
else
sudo mkdir -p /path/to/some/directory/
sudo chmod -R 755 /path/to/some/directory/
sudo chown -R user:user /path/to/some/directory/
fi
ENDSSH'
'''
}
Special Notes:
The last ENDSSH' should not have any characters before it. So it
should be at the starting position of a new line.
use ssh -t -t if you have sudo within the remote shell command
I split the commands with &&
node {
FOO = world
stage('Preparation') { // for display purposes
sh "ls -a && pwd && echo ${FOO}"
}
}
The example outputs:
- ls -a (the files in your workspace
- pwd (location workspace)
- echo world