Reuse artifacts at a later stage in the same Jenkins project - jenkins

I have a Jenkins pipeline whose Build step has an archiveArtifacts command.
After the Build step there is Unit test, Integration test and Deploy.
In Deploy step, I want to use one of the artifacts. I thought I could find it in the same place the Build step generated it, but apparently the archiveArtifacts has deleted them.
As a workaround I can copy the artifact before it is archived, but it doesn't look elegant to me. Is there any better way?

As I understand it, archiveArtifacts is more for saving artifacts for use by something (or someone) after the build has finished. I would recommend looking at using "stash" and "unstash" for transferring files between stages or nodes.
You just go...
stash include: 'globdescribingfiles', name: 'stashnameusedlatertounstash'
and when you want to later retrieve that artifact...
unstash 'stashnameusedlatertounstash'
and the stashed files will be put into the current working directory.
Here's the example of that given in the Jenkinsfile docs (https://jenkins.io/doc/book/pipeline/jenkinsfile/#using-multiple-agents):
pipeline {
agent none
stages {
stage('Build') {
agent any
steps {
checkout scm
sh 'make'
stash includes: '**/target/*.jar', name: 'app'
}
}
stage('Test on Linux') {
agent {
label 'linux'
}
steps {
unstash 'app'
sh 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
stage('Test on Windows') {
agent {
label 'windows'
}
steps {
unstash 'app'
bat 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
}
}

Related

Jenkins: unable to access the artifacts on the initial run

My setup: main node runs on Linux and an agent on Windows. I want to compile a library on an agent, archive those artifacts and copy them on the main node to create a release togather with the Linux compiled binaries.
This is my Jenkinsfile:
pipeline {
agent none
stages {
stage('Build-Windows') {
agent {
dockerfile {
filename 'docker/Dockerfile-Windows'
label 'windows'
}
}
steps {
bat "tools/ci/build.bat"
archiveArtifacts artifacts: 'build_32/bin/mylib.dll'
}
}
}
post {
success {
node('linux') {
copyArtifacts filter: 'build_32/bin/mylib.dll', flatten: true, projectName: '${JOB_NAME}', target: 'Win32'
}
}
}
}
My problem is, when I run this project for the first time, I get the following error
Unable to find project for artifact copy: mylib
But when I comment the copyArtifacts block and rerun the project, it is successful and I have artifacts vivible in the project overview. After this I can reenable the copyArtifacts and then the artifacts will be copied as expected.
How to configure the pipeline so it can access the artifacts on the initial run?
The copyArtifacts capability is usually used to copy artifacts between different builds and not between agents on the same build. Instead, to achieve what you want you can use the stash and unstash keywords which are designed exactly for passing artifacts from different agents in the same pipeline execution:
stash: Stash some files to be used later in the build.
Saves a set of files for later use on any node/workspace in the same Pipeline run. By default, stashed files are discarded at the end of a pipeline run
unstash: Restore files previously stashed.
Restores a set of files previously stashed into the current workspace.
In your case it can look like:
pipeline {
agent none
stages {
stage('Build-Windows') {
agent {
dockerfile {
filename 'docker/Dockerfile-Windows'
label 'windows'
}
}
steps {
bat "tools/ci/build.bat"
// dir is used to control the path structure of the stashed artifact
dir('build_32/bin'){
stash name: "build_artifact" ,includes: 'mylib.dll'
}
}
}
}
post {
success {
node('linux') {
// dir is used to control the output location of the unstash keyword
dir('Win32'){
unstash "build_artifact"
}
}
}
}

Sharing files between Jenkins pipelines

A lot of the examples I see like How can I use the Jenkins Copy Artifacts Plugin from within the pipelines (jenkinsfile)? share a file within the SAME pipeline. I want to share a file between two different pipelines.
I tried to use the Copy Artifacts plugin like so
Pipeline1:
node {'linux-0') {
stage("Create file") {
sh "echo \"hello world\" > hello.txt"
archiveArtifacts artifact: 'hello.txt', fingerprint: true
}
}
Pipeline2:
node('linux-1') {
stage("copy") {
copyArtifacts projectName: 'Pipeline1',
fingerprintArtifacts: true,
filter: 'hello.txt'
}
}
and I get the following error for Pipeline2
ERROR: Unable to find project for artifact copy: Pipeline1
This may be due to incorrect project name or permission settings; see help for project name in job configuration.
Finished: FAILURE
What am I missing?
NOTE: My real scripted, i.e., not declarative, pipelines are more complicated than these so I can't readily convert them to declarative pipelines.
I just tested this and it worked fine for me. Here is my code, pipeline1:
pipeline {
agent any
stages {
stage('Hello') {
steps {
sh "echo \"hello world\" > hello.txt"
archiveArtifacts artifacts: 'hello.txt', fingerprint: true
}
}
}
}
pipeline2
pipeline {
agent any
stages {
stage('Hello') {
steps {
copyArtifacts projectName: 'pipeline1',
fingerprintArtifacts: true,
filter: 'hello.txt'
}
}
}
}
copyArtifacts projectName: 'pipeline1'
Ensure that the project name is exactly same as the first pipleline's name (
and there are no conflicts on that name). If you have conflicts or use folder plugin ensure to look at this link to refer the project accordingly:
https://wiki.jenkins.io/display/JENKINS/How+to+reference+another+project+by+name

What are the #tmp folders in a Jenkins workspace and how to clean them up

I have a Jenkins pipeline, for a PHP project in a Docker container. This is my Jenkinsfile:
pipeline {
agent any
stages {
stage('Build') {
agent any
steps {
sh 'docker-compose up -d'
sh 'docker exec symfony composer install'
}
}
stage('Test') {
steps {
sh 'docker exec symfony php ./bin/phpunit --coverage-clover=\'reports/coverage/coverage.xml\' --coverage-html=\'reports/coverage\' --coverage-crap4j=\'reports/crap4j.xml\''
}
}
stage('Coverage') {
steps {
step([$class: 'CloverPublisher', cloverReportDir: '/reports/coverage', cloverReportFileName: 'coverage.xml'])
}
}
}
post {
cleanup {
sh 'docker-compose down -v'
cleanWs()
}
}
}
After running the pipeline, the var/lib/jenkins/workspace folder contains 4 folders (assuming my project name is Foo):
Foo
Foo#2
Foo#2#tmp
Foo#tmp
What are these, and how do I clean them up? cleanWs does not remove any except the first of them after the build.
EDIT: This is not a duplicate of this question because
That question does not answer my question: what are these files.
The answers to that question suggest using deleteDir, which is not recommended when using Docker containers.
There is an opened Jenkins issue about deleteDir() not deleting the #tmp/#script/#... directories.
A workaround to delete those:
post {
always {
cleanWs()
dir("${env.WORKSPACE}#tmp") {
deleteDir()
}
dir("${env.WORKSPACE}#script") {
deleteDir()
}
dir("${env.WORKSPACE}#script#tmp") {
deleteDir()
}
}
}
There is also a comment on the issue describing what #tmp is:
It [#tmp folder] contains the content of any library that was loaded at
run time. Without a copy, Replay can't work reliably.
The
Foo#2
Foo#2#tmp
folders were created because the agent was defined 2 times. Once it was defined at the top level inside the pipeline block. And once inside the stage called build.
The working folder of stage 'build' is the Foo#2 folder.

Jenkins pipeline: how to download a archived artifact in a later stage in a Jenkins pipeline

I have a Jenkins pipeline.
In stage A, I have a step in which I need to archive or to save my artifacts because I need to reuse those in a different stage on a different slave:
stage('Save artifacts'){
steps {
archiveArtifacts artifacts: '**/**/target/app*.ear'
}
}
Archiving seems to work. I see the artifacts in the UI when the build finishes and can download them. But how can I access/download these artifacts in a later stage?
Instead of archiveArtifacts you should use stash and unstash. E.g.:
stage("Build") {
steps {
// ...
stash(name: "ear", includes: '**/**/target/app*.ear')
}
}
stage("Deploy") {
steps {
unstash("ear")
// ...
}
}
Not that stash does not only stash the files, but also their paths. So unstash will put the files exactly in the same places they were (e.g. my-service/target/app.ear).
you can get artifact in this path: $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER/archive/xxxx/appxx.ear

Jenkins Pipeline Wipe Out Workspace

We are running Jenkins 2.x and love the new Pipeline plugin. However, with so many branches in a repository, disk space fills up quickly.
Is there any plugin that's compatible with Pipeline that I can wipe out the workspace on a successful build?
Like #gotgenes pointed out with Jenkins Version. 2.74, the below works, not sure since when, maybe if some one can edit and add the version above
cleanWs()
With, Jenkins Version 2.16 and the Workspace Cleanup Plugin, that I have, I use
step([$class: 'WsCleanup'])
to delete the workspace.
You can view it by going to
JENKINS_URL/job/<any Pipeline project>/pipeline-syntax
Then selecting "step: General Build Step" from Sample step and then selecting "Delete workspace when build is done" from Build step
The mentioned solutions deleteDir() and cleanWs() (if using the workspace cleanup plugin) both work, but the recommendation to use it in an extra build step is usually not the desired solution. If the build fails and the pipeline is aborted, this cleanup-stage is never reached and therefore the workspace is not cleaned on failed builds.
=> In most cases you should probably put it in a post-built-step condition like always:
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
}
}
}
post {
always {
cleanWs()
}
}
}
You can use deleteDir() as the last step of the pipeline Jenkinsfile (assuming you didn't change the working directory).
In fact the deleteDir function recursively deletes the current directory and its contents. Symbolic links and junctions will not be followed but will be removed.
To delete a specific directory of a workspace wrap the deleteDir step in a dir step.
dir('directoryToDelete') {
deleteDir()
}
Using the following pipeline script:
pipeline {
agent { label "master" }
options { skipDefaultCheckout() }
stages {
stage('CleanWorkspace') {
steps {
cleanWs()
}
}
}
}
Follow these steps:
Navigate to the latest build of the pipeline job you would like to clean the workspace of.
Click the Replay link in the LHS menu.
Paste the above script in the text box and click Run
I used deleteDir() as follows:
post {
always {
deleteDir() /* clean up our workspace */
}
}
However, I then had to also run a Success or Failure AFTER always but you cannot order the post conditions.
The current order is always, changed, aborted, failure, success and then unstable.
However, there is a very useful post condition, cleanup which always runs last, see https://jenkins.io/doc/book/pipeline/syntax/
So in the end my post was as follows :
post {
always {
}
success{
}
failure {
}
cleanup{
deleteDir()
}
}
Hopefully this may be helpful for some corner cases
If you have used custom workspace in Jenkins then deleteDir() will not delete #tmp folder.
So to delete #tmp along with workspace use following
pipeline {
agent {
node {
customWorkspace "/home/jenkins/jenkins_workspace/${JOB_NAME}_${BUILD_NUMBER}"
}
}
post {
cleanup {
/* clean up our workspace */
deleteDir()
/* clean up tmp directory */
dir("${workspace}#tmp") {
deleteDir()
}
/* clean up script directory */
dir("${workspace}#script") {
deleteDir()
}
}
}
}
This snippet will work for default workspace also.
Using the 'WipeWorkspace' extension seems to work as well. It requires the longer form:
checkout([
$class: 'GitSCM',
branches: scm.branches,
extensions: scm.extensions + [[$class: 'WipeWorkspace']],
userRemoteConfigs: scm.userRemoteConfigs
])
More details here: https://support.cloudbees.com/hc/en-us/articles/226122247-How-to-Customize-Checkout-for-Pipeline-Multibranch-
Available GitSCM extensions here: https://github.com/jenkinsci/git-plugin/tree/master/src/main/java/hudson/plugins/git/extensions/impl
For Jenkins 2.190.1 this works for sure:
post {
always {
cleanWs deleteDirs: true, notFailBuild: true
}
}
pipeline {
agent any
tools {nodejs "node"}
environment {
}
parameters {
string(name: 'FOLDER', defaultValue: 'ABC', description: 'FOLDER', trim: true)
}
stages {
stage('1') {
steps{
}
}
stage("2") {
steps {
}
}
}
post {
always {
echo "Release finished do cleanup and send mails"
deleteDir()
}
success {
echo "Release Success"
}
failure {
echo "Release Failed"
}
cleanup {
echo "Clean up in post work space"
cleanWs()
}
}
}
We make sure we are working with a clean workspace by using a feature of the git plugin. You can add additional behaviors like 'Clean before checkout'. We use this as well for 'Prune stale remote-tracking branches'.
In my case, I want to clear out old files at the beginning of the build, but this is problematic since the source code has been checked out.
My solution is to ask git to clean out any files (from the last build) that it doesn't know about:
sh "git clean -x -f"
That way I can start the build out clean, and if it fails, the workspace isn't cleaned out and therefore easily debuggable.
Cleaning up : Since the post section of a Pipeline is guaranteed to run at the end of a Pipeline’s execution, we can add some notification or other steps to perform finalization, notification, or other end-of-Pipeline tasks.
pipeline {
agent any
stages {
stage('No-op') {
steps {
sh 'ls'
}
}
}
post {
cleanup {
echo 'One way or another, I have finished'
deleteDir() /* clean up our workspace */
}
}
}
Currently both deletedir() and cleanWs() do not work properly when using Jenkins kubernetes plugin, the pod workspace is deleted but the master workspace persists
it should not be a problem for persistant branches, when you have a step to clean the workspace prior to checkout scam. It will basically reuse the same workspace over and over again: but when using multibranch pipelines the master keeps the whole workspace and git directory
I believe this should be an issue with Jenkins,
any enlightenment here?
I usually use this:
post {
success {
cleanWs()
}
}

Resources