I'm using a Release Pipeline to execute Cypress tests in a docker container. I've gotten to the point where I have been able to push the /videos folder outside the container to the agent itself.
I've tried using the Publish Artifacts task but it only seems to apply to a Build Pipeline and I'm using a Release Pipeline since we need an environment to execute the tests against.
Is there some other pipeline task or something I'm missing that I can use now to publish this folder since I'd like to be able to view the videos for debugging purposes?
I actually tried the publish task but it's a release pipeline not a
build.
For this issue , you can try this extension: Publish and Download Artifacts from Release Pipelines
The tasks contained in this package allow on one side to upload files to the logs of the release to create Release Artifacts. Another task allows to fetch those artifacts again, for example if they are needed in a different agent phase.
Publish Release Artifact Task :
The Publish Release Artifact Task is very straight forward and only needs one parameter to work properly. Specify the path to the file or the folder you want to have uploaded to the logs.
In case a folder is specified, this folder will be zipped and is available later on as foldername.zip in the logs.
There is a job controlled by Development team which built in a different node. I am on Testing team who want to take the artifacts and deploy on test device.
I can see those Artifacts from dev are stored in some path in dev's node. Does it means it must first archived in Jenkins master before I can copy it to my job?
I am using Copy Artifact plugin and constantly getting the error
Failed to copy artifacts from <dev-job> with filter: <path-in-dev-node>
*Some newbie question since i just moved from TeamCity
You probably want to use: Copy Artifact plugin.
Adds a build step to copy artifacts from another project.
Consider also, the Jenkins post-buid step "Archive the artifacts".
If you copy from the other job's workspace, what happens if another job is in progress or the workspace is wiped? That step copies them from the node to the master and stores a copy along with the build logs, etc. That makes them available via the UI as long as the build logs remain. It can take up space tho.
If you do use archive artifacts, consider using the system property jenkins.model.Jenkins.buildsDir to store all the build logs (and artifacts) outside of the jobs config directory. Some downtime and work required to separate the two (config / logs) .
You may also want to consider using a proper repository manager (Nexus / artifactory)
Finally, you may want to learn about using a Jenkins pipeline rather the relying on chained jobs, triggers or users and so forth. Why? 'cos it's much more controlled and easier to maintain.
ps: I'm not a huge fan of artifactDeployer, but it may work for you.
pps: you may want to review this in depth answer: Jenkis downstream job fails to find upstream artifacts
I installed the Deploy Plugin on my Jenkins in order to automate the deployment of my Maven built war packages to Tomcat 7. The problem is that I am able to use the plugin to deploy to a remote Tomcat server only if they are made within the same job that uses the deploy plugin. In other words, I have not been able to set up a standalone job that deploys artifacts made by a different job.
For example, I have a job named pack.foo. It uses the source code in /var/lib/project/module to create module.war and put it in /var/lib/project/module/target. However, because of the Maven version setup, the artifact posted on pack.foo's artifact page is something like module-2.0.0-SNAPSHOT.war.
The only way I am able to deploy module.war is if I add a Post-build Action to pack.foo and specify **/module.war to be a remote Tomcat manager URL (provided I have the manager's credentials in Jenkins config). Then the job's console output logs that /var/lib/project/module/target/module.war was deployed to that URL:
Deploying /var/lib/project/module/target/module.war to container Tomcat 7.x Remote with context
[/var/lib/project/module/target/module.war] is not deployed. Doing a fresh deployment.
Deploying [/var/lib/project/module/target/module.war]
How can I use this, or another plugin, to deploy a WAR artifact that was made in a separate Jenkins job? I would like to have separate jobs for artifact creation and deployment. The plugin wasn't finding **/module-2.0.0-SNAPSHOT.war or even **/module.war built by another job even though there was definitely a file on disk that matched that pattern.
See the paragraph on the Deploy Plugin's page you linked:
How to rollback or redeploy a previous build
There may be several ways to accomplish this, but here is one suggested method:
Install the Copy Artifact Plugin
Create a new job that you will trigger manually only when needed
Configure this job with a build parameter of type "Build selector for Copy Artifact", and a copy artifact build step using "Specified by build parameter" to select the build.
Add a post-build action to deploy the artifact that was copied from the other job
Now when you trigger this job you can enter the build number (or use any other available selector) to select which build to redeploy. Thanks to Helge Taubert for this idea.
I would like to share byproducts of one jenkins job, with another one that run after.
I am aware that I can set "use custom workspace", but that would merge the jobs together; which is not what I want. I just need to move few files in a location, that are read by the next job.
So far I can't find out how you actually tell Jenkins jobs to look for a specific folder; since it does not have a concept of file system, beyond what is going on in the job workspace folder.
Is there a way to access the host file system, or declare a shared folder inside jenkins (like in the main workspace folder, which contains all the other jobs?), so I can copy and read files in it, from different jobs?
Where possible I would like to avoid plugins and extras; I would like to use what is included with Jenkins base.
I realize you want to avoid plugins, but the Jenkins-y way to accomplish this is to use the Copy Artifacts plugin, which does exactly what you want.
There are a variety of problems that you may run into when trying to manage the filesystem yourself. (How do you publish to a common location when running on different build nodes? How do you handle unsuccessful builds?) This solution uses Jenkins to track builds and artifacts. In the absence of a separate artifact repository, its a lot better than trying to manage it yourself.
To use Copy Artifacts:
As a Post-Build step, choose "Archive Artifacts" in the first job and enter the path(s) to the generated files.
Then in the second job, add a "Copy Artifacts from another project" build step to grab some or all files marked as artifacts in your first job. (By default, Jenkins will re-create the paths of the generated files in the second job's workspace, which may or may not be what you want, but you can change this behavior.)
Configure the Jenkins to run a Maven build, and deploy your artifacts with "mvn clean deploy" This will push it to an "artifact server" which you probably have, or if not, need to add / configure.
Then in your downstream job, also a Maven job, you configure it to depend on the same artifact that was published in the upstream job. This will trigger a download of the artifact from the artifact server and make it available to the build.
The setup is used to build and deploy to Adobe AEM.
Master Build job pulls from git repository, builds and packages, run the tests and then fires downstream jobs that should use the built packages from upstream job.
The issue is that downstream job fail with the message:
Unable to access upstream artifacts area /var/lib/jenkins/jobs/PROJECTNAME-Master-Branch/builds/2014-10-22_11-33-46/archive. Does source project archive artifacts?
It seems to me that somehow CopyArtifacts plugin, triggered by the downstream job, is looking for the artifacts in wrong location. The correct location would be
/var/lib/jenkins/jobs/PROJECTNAME-Master-Branch/workspace/PROJECTNAME-*/**/*.jar,/var/lib/jenkins/jobs/PROJECTNAME-Master-Branch/workspace/PROJECTNAME-*/**/*.zip
But then, it complains about
java.io.IOException: Expecting Ant GLOB pattern, but saw '/var/lib/jenkins/jobs/PROJECTNAME-Master-Branch/workspace/PROJECTNAME-*/**/*.jar,/var/lib/jenkins/jobs/PROJECTNAME-Master-Branch/workspace/PROJECTNAME-*/**/*.zip'. See http://ant.apache.org/manual/Types/fileset.html for syntax
The downstream job copies artifacts from another project, and then the build was either "Upstream build that triggered this job" or "Copy from workspace of latest completed build". And none works.
Any ideas?
TL;DR
You are trying to use artifacts without archiving them first.
You are trying to use absolute paths, but they should be relative to $WORKSPACE and/or "archive location".
Full Answer
You are misunderstanding the concept of "Artifacts" as it relates to Jenkins.
What are Jenkins Artifacts
Artifacts are files that are specifically preserved after the build with the help of Archive the Artifacts post-build action.
When the build runs, it runs within:
$WORKSPACE, which on filesystem usually resides within
$JENKINS_HOME/jobs/$JOB_NAME/workspace
Inside there, you can have your SCM checkout folders, temporary build files, final built files, binaries, etc.
The contents of $WORKSPACE is volatile, you should never rely on it, outside of the build timeframe (and downstream jobs are outside of the build timeframe). The contents of $WORKSPACE could be different between different master/slave nodes, it could be deleted at any time by admin, or by SCM update/cleanup/checkout.
It's also important to understand that there is only one $WORKSPACE for the whole Job.
But now pay attention to your Build History, there are several entries in that list, referenced by build number (#) and date timestamp.
These are stored under:
$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID
with $BUILD_ID being the date-timestamp of the build, like 2014-10-22_11-33-46
The $WORKSPACE contains the information relevant to current or last (and the problem is: you can never be sure if it's "current" or "last") build;
The builds folder contains a record of all past (retained) build executions (this is what makes up the Build History list on your left), per build.
By default, it contains only what Jenkins itself needs: build.xml copy, changelog information, console log. When you go to URL http://$JENKINS_URL/job/$JOB_NAME/[nn]/ where [nn] is a numeric job build/run number (#), it's reading this information from the builds folder on the filesystem.
To preserve artifacts of a build (to avoid them being overwritten by the next build, wiped out worskpace, or just to access older builds), you need to Archive the Artifacts (with same post-build action with the same title). When you archive the artifacts, you indicate which files within $WORKSPACE you want to preserve. When Jenkins does the archiving, it will place those files (keeping paths [relative to $WORKSPACE] preserved) into:
$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive/.
This way, you can have multiple sets of artifacts preserved for previous builds, not just "latest/last" from $WORKSPACE.
For the sake of completeness, I will mention that Jenkins's "permalinks", such as http://$JENKINS_URL/job/$JOB_NAME/lastSuccessfulBuild and /lastFailedBuild, etc are in fact symlinks on the filesystem to one of the preserved builds/$BUILD_ID folders.
Lastly, you control how many build runs and how many artifacts are retained (can be configured separately) through "Discard old builds" checkmark on job configuration. By default, all are retained, but if you start retaining artifacts, you need to think of hard-disk space capacity.
Solutions to your problem
So with the information above, and looking at your error messages, you should now see that the Copy Artifacts plugin is correctly looking for artifacts under the /archive/ section of a build.
You should also notice that Copy Artifacts plugin does not let you pick "current build" when selecting which build to copy from. It has permalinks (like "last successful" or "last build"), and specific build numbers, all of which translate to preserved builds under $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive/
Even "Upstream Build that triggered this job" will link to a specific $BUILD_ID.
In either of below options
Configuration for Archiving Artifacts is relative to $WORKSPACE.
Configuration for Copy Artifacts is relative to "archive location", that is $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive/.
Since "Copy Artifacts" is relative to "archive location", and "archive location" is relative to $WORKSPACE, then for all intensive purposes, the relative paths of both configurations can be same and relative to $WORKSPACE
Option 1
First Archive the Artifacts with the post-build action, otherwise you have nothing to copy from.
If you have your files in the root of $WORKSPACE, it should be:
PROJECTNAME-*/**/*.jar,PROJECTNAME-*/**/*.zip
(Note, not full paths in here)
Then use Upstream Build that triggered this job for Copy Artifacts selection.
For Artifacts to copy field use either:
** or blank to copy all archived artifacts, or
PROJECTNAME-*/**/*.jar,PROJECTNAME-*/**/*.zip (same as the archiving section)
Option 2
If you don't want to archive, you can use $WORKSPACE directly, with Copy from workspace of latest completed build, however you must ensure that no second upstream build can run while downstream build is executing, else you risk getting a partial file from a partial build, because as previously explained, $WORKSPACE is volatile.
Again, for the Copy Artifacts step, under Artifacts to copy field, use path relative to $WORKSPACE, that is:
PROJECTNAME-*/**/*.jar,PROJECTNAME-*/**/*.zip
Option 3
If you really want to copy the whole WORKSPACE between different jobs, use either
Clone Workspace SCM plugin or
Shared Workspace plugin
The fix may be this simple: disable or remove Compress Artifacts plugin and reboot Jenkins.
This workaround was deduced from a long-standing bug report: "Copy Artifacts Plugin" should support ArtifactManager.
The solution is about the configuration of the builder.
The root cause sits on the configuration of the downstream job. Once "Copy from workspace of latest completed build" is chosen for the build to be copied, and the path of artifacts to copy is set to relative path, such as projectname-//.jar,projectname-//.zip then the build succeeds.
Furthemore, in the parent job configuration, downstream job needs to be allowed to CopyArtifact and Projects to allow copy artifacts field should specify the downstream job.
Edit: Now I see that you responded in the meantime. Great answer and basically clears up some of the questions I had.
The one unclear thing about option 1 is that archiving of the files happens after the parent job completes.
Waiting for the completion of projectname-Deploy
projectname-Deploy #19 completed. Result was SUCCESS
Waiting for the completion of projectname-Deploy
projectname-Deploy #20 completed. Result was SUCCESS
Build step 'Trigger/call builds on other projects' changed build result to SUCCESS
Strings match run condition: string 1=[lab2b], string 2=[both]
Run condition [Strings match] preventing perform for step [BuilderChain]
Archiving artifacts
Once I changed the approach to option two it worked for me, but I would like to understand first option as well.