Jenkins Copy Artifact unable to find folder/multiProjectPipeline/branchWithSlash - jenkins

I have Jenkins LTS 2.60.2 on Windows Server 2016 and using these plugins:
Folders plugin (6.1.0)
Copy Artifact plugin (1.38.1)
Pipeline plugin (2.5) + all dependent pipeline sub-plugins
Various other dependent plugins...
See Pipeline to use artifacts from 2 projects associated by the same git branch name for more details about my setup, but to sum it up I have these items:
playground (a folder created with the Folders plugin to group all these following items)
frontend (multibranch pipeline)
backend (multibranch pipeline)
configure (pipeline with a parameter called BRANCH_NAME)
The frontend and backend git repos, both have a branch called master and one called release/2017.2.
The idea is to call the configure pipeline automatically after each successful build, passing the git branch name. Automatically triggering the configure pipeline works.
What doesn't work and I need your help to fix, is the step inside the configure pipeline to copy the artifacts from a multibranchPipeline/specificBranch.
If for the BRANCH_NAME parameter (or the upstream pipeline) is master it works. If BRANCH_NAME is: release/2017.2 I get this error:
ERROR: Unable to find project for artifact copy:
playground/frontend/release%2f2017.2 This may be due to incorrect project
name or permission settings; see help for project name in job
configuration. Finished: FAILURE
The configure pipeline looks like this:
node {
stage('Prepare') {
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2f")
step([$class: 'CopyArtifact', projectName: "${projectname}", selector: [$class: 'StatusBuildSelector', stable: false]])
}
stage('Archive') {
archiveArtifacts '**'
}
}
As you can see I already replace / with %2f (it's needed).
If I don't use the "playground" folder (all my pipelines as is, not inside a folder item), it works. If I use the folder and use the master branch, it works. It doesn't work if I use the folder and a branch name like 2017.2. What am I doing wrong? Can you help making it work? Of well if it's a bug (I searched in https://issues.jenkins-ci.org and found some bugs where a similar setup with folder doesn't work, but they have been fixed... so I really wonder...) in the copy artifact plugin, please file the bug and share the link here, so we can all monitor its progress...
Thank you.

I finally found the issue. The configure pipeline was failing to find a branch with a slash because the encoding was incorrect.
So, in my question, in the configure pipeline:
this (replace / with %2f) is wrong and generates the error:
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2f")
this is the proper way to encode the slash, and it works:
def projectname = "playground/frontend/" + URLEncoder.encode("${BRANCH_NAME}", "UTF-8").replace("+", "%20")
Credits to: http://www.pipegrep.se/copy-artifacts-from-jenkins-pipeline-jobs.html
UPDATE: actually, I investigated a bit further and added echo "${projectname}" just before step, with the previous and fixed projectname, and I noticed that the difference was %2f lowercase.
Uppercase, like this: %2F works:
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2F")
So, the fixed configure pipeline looks like this (I kept my replace function, which was enough for my case):
node {
stage('Prepare') {
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2F")
step([$class: 'CopyArtifact', projectName: "${projectname}", selector: [$class: 'StatusBuildSelector', stable: false]])
}
stage('Archive') {
archiveArtifacts '**'
}
}

I created a sample project to try and recreate what you were seeing, and I was able to do so, after a fashion, except that the build that I was having trouble on was master instead of release/2017.2. Eventually, I realized that I was doing the build job incorrectly from the frontend project, and it was giving me the same error as you because I hadn't ever completed a successful build of the frontend/master branch (I had completed a successful build of the release/2017.2 branch because I didn't have it triggering the configure build initially, so it didn't give me the same error once I did configure it to trigger the configure build).
What worked was changing the build job in the frontend Jenkinsfile to this:
build job: 'playground/configure', parameters: [[$class: 'StringParameterValue', name: 'BRANCH_NAME', value: env.BRANCH_NAME]], quietPeriod: 2, wait: false
Adding in the quietPeriod gives a couple seconds of quiet time between completing the previous job (I'm not certain that this is critical, but it seems like it might be a nice fail-safe, to try and make sure there's enough time for the build to complete), but the important part is the wait: false, which instructs Jenkins that this build shouldn't wait for the triggered build to complete. Once I changed that, the frontend/master branch completed successfully, and the configure build that it triggered also completed successfully.
Hopefully this helps. I was able to get both my master and release/2017.2 branches to build properly, so I don't believe there's any intrinsic problem with the / in the project name. You can see my simple Jenkinsfiles in the referenced repo, and I used the same pipeline script as you posted in your question.

Related

How do I get an artifact from the PR that is getting merged into master in Jenkiins

Scenario:
A developer creates a PR against the master branch. Jenkins (Cloudbees) goes to work building and validating that PR as well as generating a text file containing a build_info.txt file as an artifact.
When the PR is merged into master I need to be able to access the artifact that was created in the PR validation step, extract the version information it contains and commit that version information into master along with PR code changes.
Problem:
I've printed out the env vars during the merge to master script (run_pr_merge), but I haven't seen information (It might be there, I just don't recognize it) that would allow me to link back to the PR job that is being merged or a way to say "give me the artifacts that were created during this PR's build and validation job"
My script looks something like this:
if (isMasterBranch()) {
// run this when code is pushed to master
sh "bash ./run_pr_merge.sh" // which requires build_info.txt from PR build
} else {
// run this for each PR build
sh "bash ./build_and_validate.sh"
archiveArtifacts 'build_info.txt'
}
Am not super familiar with Jenkins/Cloudbees, so there might be a better way of structuring the pipeline to achieve what I need but am hoping there's a relatively easy way to get hold of the PR info being merged int master.
Have looked at copyArtifacts but again I'm not sure how to to reference the PR being merged. Any help greatly appreciated.
There's a CopyArtifact plugin for Jenkins that can be used to copy artifacts from another job. It supports both "Freestyle" jobs and pipeline. You need to provide job URL, and, optionally, build number (by default it will use last successful build). You can craft the URL of the job provided you have the PR number.
In our environment, the URL for a specific job looks something like:
https://jenkins.mycompany.com/job/mycompany/job/myproj/job/PR-<number>/<build_no>/
Provided I have the PR number, I can build the needed URL with e.g.
def pr_number = 12345
def job_url = "https://jenkins.mycompany.com/job/mycompany/job/myproj/job/PR-${pr_number}/"
step([
$class: 'CopyArtifact',
filter: 'build_info.txt',
fingerprintArtifacts: true,
optional: true,
projectName: job_url
// default selector is "last successful build"
// selector: [$class: 'SpecificBuildSelector',
// buildNumber: build_number]
])
// check the file is there
sh "cat ${WORKSPACE}/build_info.txt"

Jenkins: How to start, in a Jenkinsfile, a multibranch job inside a Bitbucket folder?

In a Jenkinsfile, to start a parameterized pipeline job from another job, I have this code snippet:
build job: 'build-sharpen-branch', parameters: [
[$class: 'StringParameterValue', name: 'BRANCHNAME', value: mergeBranchname]
]
This already works as expected, and it will start a job at URL https://$JENKINS_URL/job/build-sharpen-branch/.
Now I want to start a job, that is one branch of a multibranch project inside a Bitbucket folder. The URL of the job is https://$JENKINS_URL/job/iText%207%20.NET/job/sharpen/job/feature%2FQA-10738/.
iText%207%20.NET is the name of the Bitbucket project.
sharpen is the name of the Multibranch job.
feature%2FQA-10738 is the name of the branch, urlencoded.
I read the following questions about starting a multibranch job NOT inside a folder:
Trigger Multibranch Job from another
Triggering a multibranch pipeline job from an other multibranch pipeline
How to trigger Multibranch Pipeline Jenkins Job within regular pipeline job?
From the answers there, I gather that the syntax is $JOB/$BRANCH (where $BRANCH is URL-encoded to rename branches like feature/foo to feature%2Ffoo).
From Jenkins pipeline with folder plugin. How to build a job located different folder I gather that the syntax for a job inside a folder is $FOLDER/$JOB.
Combining the two, I conclude that the syntax for folder+job+branch is most likely $FOLDER/$JOB/$BRANCH.
So I tried with this code:
build job: "iText%207%20.NET/sharpen/${java.net.URLEncoder.encode branchName, 'UTF-8'}"
with
folder = iText%207%20.NET
job = sharpen
branch = ${java.net.URLEncoder.encode branchName, 'UTF-8'} (URLEncoder to change / in the branch name to %2F)
To my surprise, when I ran this, I got an error:
ERROR: No item named iText%207%20.NET/sharpen/feature%2FQA-10738 found
As already stated above, a job exists on URL https://$JENKINS_URL/job/iText%207%20.NET/job/sharpen/job/feature%2FQA-10738/.
What is the correct syntax for a multibranch job inside a Bitbucket folder?
The problem was with the folder name iText%207%20.NET, which is an urlencoded version of iText 7 .NET. Apparently Jenkins can't handle urlencoded spaces in folder names.
I renamed the folder to itext_7_dotnet and then used
build job: "itext_7_dotnet/sharpen/${java.net.URLEncoder.encode branchName, 'UTF-8'}"
This works.
Moral of the story: never use spaces in your folder names!
For us, this works:
build job: "${folder}/${repo}/${branch.replace('/','%2F')}", wait: false, propagate: false
You can also trigger build by using curl, with crumbs and all that.

Jenkins Pipeline and jboss deploy

I'm currently using Jenkins FreeStyle Project in my project,trying to migrate to Jenkins Pipeline, but I'm facing some issues:
1) I need to commit jenkinsfile in my project, but my deploy phase is just copy from target/project.war to jboss deployment folder, as shown below:
stage('Deploy') {
steps {
sh 'cp /var/lib/jenkins/workspace/project/project.war /opt/jboss/standalone/deployment/project.war'
}
}
The problem: currently the path is fixed and tomorrow if a change occurs and there is a need to deploy to another machine, An update should be made to the source code which should be avoided. In FreeStyle project i just update the JOB and everything works.
2) The project has 3 modules. The FreeStyle project was configured so that JOB A will call JOB B on finish. In pipeline how can this order be achieved:
- Start JOB A --> JOB B --> JOB C.
You can add the following into your script
1.Issue with copying:
Firstly, you avoid using the actual path(Location of file in workspace) to a
relative Path i.e. using project/*.war or **/*.war it will take it
from the workspace itself.
Second, Coming to the issue that you have
to change the target location like you said you have to change it
FreeStyle Project :) so you have to change it in the
JenkinsFile also :)
2.To call other jobs in your pipeline and the following
build job: 'Job2', parameters: [
new org.jvnet.jenkins.plugins.nodelabelparameter.NodeParameterValue
("TARGET_NODE", "description", nodeName)
]
If it does not have any parameter remove that section out.
There is something called Jenkins workflow which provides more power and control if your interested in it, you can look it up here https://dzone.com/refcardz/continuous-delivery-with-jenkins-workflow?chapter=1
you can use sshPulissher:send build artifacts over ssh
add this code in your jenkins pipeline
and configure your sshServer in manage jenkins
finally your war is transfert in your destination

Jenkins how to create pipeline manual step

Prior Jenkins2 I was using Build Pipeline Plugin to build and manually deploy application to server.
Old configuration:
That works great, but I want to use new Jenkins pipeline, generated from groovy script (Jenkinsfile), to create manual step.
So far I came up with input jenkins step.
Used jenkinsfile script:
node {
stage 'Checkout'
// Get some code from repository
stage 'Build'
// Run the build
}
stage 'deployment'
input 'Do you approve deployment?'
node {
//deploy things
}
But this waits for user input, noting that build is not completed. I could add timeout to input, but this won't allow me to pick/trigger a build and deploy it later on:
How can I achive same/similiar result for manual step/trigger with new jenkins-pipeline as prior with Build Pipeline Plugin?
This is a huge gap in the Jenkins Pipeline capabilities IMO. Definitely hard to provide due to the fact that a pipeline is a single job. One solution might be to "archive" the workspace as an "artifact" (tar and archive **/* as 'workspace.tar.gz'), and then have another pipeline copy the artifact and and untar it into the new workspace. This allows the second pipeline to pickup where the previous one left off. Of course there is no way to gauentee that the second pipeline cannot be executed out of turn or more than once. Which is too bad. The Delivery Pipeline Plugin really shines here. You execute a new pipeline right from the view - instead of the first job. Anyway - not much of an answer - but its the path I'm going to try.
EDIT: This plugin looks promising:
https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/doc/PIPELINE_EXAMPLES.md

Jenkins multibranch pipeline with Jenkinsfile from different repository

I have a Git repository with code I'd like to build but I'm not "allowed" to add a Jenkinsfile in its root (it is a Debian package so I can't add files to upstream source). Is there a way to store the Jenkinsfile in one repository and have it build code from another repository? Since my code repository has several branches to build (one for each Debian release) this should be a multibranch pipeline. Commits in either the code or Jenkinsfile repositories should trigger a build.
Bonus complexity: I have several code/packaging repositories like this and I'd like to reuse the same Jenkinsfile for all of them. Thus it should somehow dynamically fetch the right Git URL to use. The branches to build have the same names across all repositories.
Short answer is : you cannot do that with a multibranch pipeline. Multibranch pipelines are only designed (at least for now) to execute a specific pipeline in Pipeline script from SCM style, with a fixed Jenkinsfile at the root of the project.
You can however use the Multi-Branch Project plugin made for multibranch freestyle projects. First, you need to define your multibranch freestyle configuration just like you would with a multibranch pipeline configuration.
Select this new item like shown below :
This type of configuration will behave exactly same as the multibranch pipeline type, i.e. it will create you a folder with the name of your configuration and a sub-project for each branch it automatically detected.
The implementation should then be a piece of cake :
Specify your SCM repository in the multibranch configuration
Call another build as part of your build/post-build as you would do in a standard freestyle project, except that you have to call a parameterized job (let's call it build-job) and give it your repository information, i.e. Git URL and current branch (you can use the pre-defined variables $GIT_URL and $GIT_BRANCH for this purpose)
In your build-job, just define either an inline pipeline or a pipeline script checked out from SCM, and inside this script do a SCM checkout and go on with the steps you need to build. Example of build-job pipeline content :
.
node() {
stage 'Checkout'
checkout scm: [$class: 'GitSCM', branches: [[name: '*/${GIT_BRANCH}']], userRemoteConfigs: [[url: '${GIT_URL}']]]
stage 'Build'
// Build steps...
}
Of course if your different multibranches projects need to be treated a bit differently, you could also use intermediate projects (let's say build-project-A, build-project-B, ...) that would in turn call the generic build-job pipeline)
The one, major drawback of this solution is that you will only have one job responsible for all of your builds, making it harder to debug. You would still have your multibranch projects going blue/red in case of success/error but you will have to go back to called build-job to find the real problem of your build.
The best way I have found is to use the Remote Jenkinsfile Provider plugin. https://plugins.jenkins.io/remote-file/
This will add an option "by Remote Jenkinsfile Provider plugin" under Build Configuration>Mode then you can point to another repo where the Jenkinsfile is. I find this to be a much better solution than the Pipeline Multibranch Defaults Plugin, which makes you store the Jenkins file in Jenkins itself, rather than in source control.
U can make use of this plugin
https://github.com/jenkinsci/pipeline-multibranch-defaults-plugin/blob/master/README.md
Where we need to configure the jenkinsfile on jenkins rather than having it on each branch of your repo
I have version 2.121 and you can do this two ways:
Way 1
In the multibranch pipeline configuration > Build Configuration > Mode > Select "Custom Script" and put in "Marker File" below the name of a file you will use to identify branches that you want to have builds for.
Then, below that in Pipeline > Definition select "Pipeline Script from SCM" and enter the "SCM" information for how to find the "Jenkinsfile" that holds the script you want to run. It can be in the same repo you are finding branches in to create the jobs (if you put in the same GitHub repo's info) but I can't find a way to indicate that you just use the same branch for the file.
Way 2
Same as above, in the multibranch pipeline configuration > Build Configuration > Mode > Select "Custom Script" and put in "Marker File" below the name of a file you will use to identify branches that you want to have builds for.
Then, below that in Pipeline > Definition select "Pipeline Script" and put a bit of Groovy in the text box to load whatever you want or to run some script that already got loaded into the workspace.
In my case, i have an escenario whith a gitlab project based on gradle who has dependencies on another gitlab preject based on gradle too (same dashboard, but differents commits, differents developers).
I have added the following lines into my Jenkinsfile (the one which depends)
stage('Build') {
steps {
git branch: 'dev', credentialsId: 'jenkins-generated-ssh-key', url: 'git#gitlab.project.com:root/coreProject.git'
sh './gradlew clean'
}
}
Note: Be awark on the order on the sentences.
If you have doubt on how to create jenkins-generated-ssh-key please ask me

Resources