I'm running into a weird issues with a pipeline script. I have a multi line sh blob like
sh """
git tag -fa \\"${version}\\" -m \\"Release of ${version}\\"
"""
And this somehow runs as:
+ git tag -fa '"1.0-16-959069f'
error: Terminal is dumb, but EDITOR unset
Please supply the message using either -m or -F option.
So its dropping the -m and message. I've tried single escapes, double escapes, nothing seems to work.
I have no idea why this worked but this did
def tagGithub(String version) {
def exec = """
git tag -d ${version} || true
git push origin :refs/tags/${version}
# tag new version
git tag -fa ${version} -m "Release of ${version}"
git push origin --tags
"""
sh exec
}
Something with the inline jenkins groovy interpolation seems busted, doing the interpolation in another var and then executing it worked
Related
Currently, I choose what image to push to registry and i use a " complex " method to set the image tag to manifest before push the files to Git repos.
This is my code
stage("Push to Repo "){
steps {
script {
def filename = 'Path/to/file/deploy.yaml'
def data = readYaml file: filename
data.spec[0].template.spec.containers[0].image = "XXXXXXXXXXXXXXXXXXXXX:${PROJECT_VERSION}"
sh "rm $filename"
writeYaml file: filename, data: data
sh "sed -ie 's/- apiVersion/ apiVersion/g' Path/to/file/deploy.yaml "
sh "sed -i '/^ - c.*/a ---' Path/to/file/deploy.yaml "
sh ''' cd Path/to/file/
git add .
git commit -m "[0000] [update] update manifest to version: ${PROJECT_VERSION} "
git push -u origin HEAD:branche_name '''
}}}
I'am looking for a another way to parse the image tag directly to manifest.
Is there a Jenkins plugin to do that ?
I use YQ tool to do this, it's an image used to edit yaml files.
Example (just docker run):
docker run --rm --user="root"
-e TAG=dev-123456
-v "${PWD}":/workspace
-w /workspace mikefarah/yq
eval '.spec.spec.containers.image.tag = strenv(TAG)'
-i values.yaml
This replaces tag dev-123456 for the current tag in deployment.
I write on multiple lines to make it easier to see, you can write on one line if you want.
Link for details:
https://hub.docker.com/r/mikefarah/yq
This question is a follow up to this question
How to pass jenkins credentials into docker build command?
I am getting the ssh key file from jenkins credential store in my groovy pipeline and
passing it into docker build command via --build-arg so that I can checkout and build artifacts from the private git repos from within my docker container
credentials store id : cicd-user, which works for checking out my private works as expected from my groovy Jenkinsfile
checkout([$class: 'GitSCM',
userRemoteConfigs: [[credentialsId: 'cicd-user', url:'ssh://git#bitbucket.myorg.co:7999/A/software.git']]
I access it and try to pass the same to docker build command:
withCredentials([sshUserPrivateKey(credentialsId: 'cicd-user', keyFileVariable: 'FILE')]) {
sh "cd ${WORKSPACE} && docker build -t ${some-name} --build-arg USERNAME=cicd-user --build-arg PRIV_KEY_FILE=\$FILE --network=host -f software/tools/jenkins/${some-name}/Dockerfile ."
}
in Dockerfile I do
RUN echo "$PRIV_KEY_FILE" > /home/"$USERNAME"/.ssh/id_rsa && \
chmod 700 /home/"$USERNAME"/.ssh/id_rsa
RUN echo "Host bitbucket.myorg.co\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
But I am seeing the following issue
"Load key "/home/cicd-user/.ssh/id_rsa" :(invalid format)
"git#Bitbucket.mycomp.co:Permission denied( Public key)
"fatal: could not read from remote repository"
In the past I have passed the ssh priv key as --build-arg from outside by cat'ing like below
--build-arg ssh_prv_key="$(cat ~/.ssh/id_rsa)"
Should I do something similar
--build-arg PRIV_KEY_FILE="$(cat $FILE)"
Any idea on what might be going wrong or where I should be looking for debugging this correctly ?
I ran into the same issue yesterday and I think I've come up with a workable solution.
Here are the basic steps I took - using the sshagent plugin to manage the sshagent within the Jenkins job. You could probably use withCredentials as well, though that's not what I ended up finding success with.
The ssagent (or alternatively the key) can be made available to specific build steps using the docker build commands --ssh flag. (Feature reference) It's important to note that for this to work (at the current time) you need to set DOCKER_BUILDKIT=1. If you forget to do this, then it seems like it ignores this configuration and the ssh connection will fail. Once that's set, the sshagent
Cut down look at the pipeline:
pipeline {
agent {
// ...
}
environment {
// Necessary to enable Docker buildkit features such as --ssh
DOCKER_BUILDKIT = "1"
}
stages {
// other stages
stage('Docker Build') {
steps {
// Start ssh agent and add the private key(s) that will be needed in docker build
sshagent(['credentials-id-of-private-key']) {
// Make the default ssh agent (the one configured above) accessible in the build
sh 'docker build --ssh default .'
}
}
// other stages
}
}
}
In the Dockerfile it's necessary to explicitly give lines that need it access to the ssh agent. This can be done by including mount=type=ssh in the relevant RUN command.
For me, this looked roughly like this:
FROM node:14
# Retrieve bitbucket host key
RUN mkdir -p -m -0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
...
# Mount ssh agent for install
RUN --mount=type=ssh npm i
...
With this configuration, the npm install was able to install a private git repo stored on Bitbucket by utilizing the SSH private key within docker build via sshagent.
After spending one week I found some how reasonable way to do.
just add
RUN git config --global url."https://${GIT_ACCESS_TOKEN}#github.com".insteadOf "ssh://git#github.com"
into your docker file and it will install if it needs to install private packages as well.
add pass your GIT_ACCESS_TOKEN (you can have it in your github settings account with setting proper permissions) where you are building your image. Like
docker build --build-arg GIT_ACCESS_TOKEN=yourtoken -t imageNameAndTag .
My Jenkins pipeline code successfully checks out my private git repo from bitbucket using
checkout([$class: 'GitSCM',
userRemoteConfigs: [[credentialsId: 'cicd-user', url:'ssh://git#bitbucket.myorg.co:7999/A/software.git']]
in same software.git I have a Dockerfile that I want to use to build various build targets present in software.git on Kubernetes and I am trying the below to pass jenkins credentials into a docker container that I want to build and run.
So in the same jenkins pipeline when I checked out software.git (above code), I try to do the following to get the docker container built
withCredentials([sshUserPrivateKey(credentialsId: 'cicd-user', keyFileVariable: 'FILE')]) {
sh "cd ${WORKSPACE} && docker build -t ${some-name} --build-arg USERNAME=cicd-user --build-arg PRIV_KEY_FILE=$FILE --network=host -f software/tools/jenkins/${some-name}/Dockerfile ."
}
in Dockerfile I do
RUN echo "$PRIV_KEY_FILE" > /home/"$USERNAME"/.ssh/id_rsa && \
chmod 700 /home/"$USERNAME"/.ssh/id_rsa
RUN echo "Host bitbucket.myorg.co\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
But still from my Docker container I am not able to successfully checkout my private repo(s). What am I missing ? Any comments, suggestions ? Thanks.
Please read about Groovy String Interpolation.
In your expression
sh "cd ${WORKSPACE} && docker build -t ${some-name} \
--build-arg USERNAME=cicd-user \
--build-arg PRIV_KEY_FILE=$FILE --network=host \
-f software/tools/jenkins/${some-name}/Dockerfile ."
you use double quotes so Groovy interpolates all the variables in the string. This includes $FILE so Groovy replaces that with the value of Groovy variable named FILE. You don't have any Groovy variable with that name (but rather bash variable which is different from Groovy) so this gets replaced with an empty string.
To prevent interpolating that particular variable, you need to hint Groovy not to interpolate this particular one, by escaping this $ with \:
sh "cd ${WORKSPACE} && docker build -t ${some-name}\
--build-arg USERNAME=cicd-user \
--build-arg PRIV_KEY_FILE=\$FILE --network=host \
-f software/tools/jenkins/${some-name}/Dockerfile ."
The file validates and I look to have the proper syntax.
script {
sh """
summon -f folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
open folder/file.yml: no such file or directory
I confirmed the existence of the file and workspace location.
Try using full path with workspace variable:
script {
sh """
summon -f ${WORKSPACE}/folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
}
so what I see happening is I wrapped the file into a script. ran git add , git commit, git push. I updated the jenkins file to ls -l the folder and I notice that file is missing. so not sure if this is a git issue or jenkins or etc
I am tring to run an ansible shell script thorugh a groovy script in Jenkins. But I am getting weird syntax error when I use special characters like $ or . I tried to use the escape sequence but still getting an error. It works fine if I remove the JAVA_OPTS variable.
batch_service_url="http://DEV:8080/test"
JAVA_OPTS="\$JAVA_OPTS -Dactivemq.tcp.url=failover:\(tcp://DEV1:61616,tcp://DEV1:61616\)?nested.wireFormat.maxInactivityDuration=30000"
def test(){
sh """sudo w360ansibleint <<EOF
ansible-playbook -i ansible/ANS-5.2.0/hosts ansible/ANS-5.2.0/app_config.yml -e '{
"ansible_hostname":"${ansible_hostname}",
"tomcat_app_parameters":"base",
"batch_service_url":"${batch_service_url}",
"tomcat_setenv_extra": ["\\$JAVA_OPTS"]
}'
EOF
"""
}