I was recently put in charge of all Jenkins-related work at my job, and was tasked with storing build artifacts from our declarative pipelines in a place where:
- They are accessible to everyone on the team
- They can be stored for long periods of time
Ideally they would be visible on the Jenkins interface, where they appear when using the default 'archiveArtifacts' command. I know this saves them in the JENKINS_HOME directory. The problem is that I have to discard old builds to avoid running out of space and the artifacts are deleted with them. Furthermore, I don't have access to the server that Jenkins runs on because it's managed by a separate team, so I can't go into JENKINS_HOME.
I looked into a few ARMs like Nexus and Artifactory, but from my understanding those are only supposed to be used for full releases. I'm looking to save artifacts after every new merge, which can happen multiple times a day.
I'm currently saving them on a functional user's home directory, but I'm the only one with direct access to it so that's no good. I also looked into plugins like ArtifactDeployer, which doesn't support pipelines and only does as much as a 'cp' command as far as I could tell.
I ended up creating some freestyle jobs that copy artifacts from the pipelines and save them directly in their workspace. This way they're stored on our Jenkins slaves and visible through the interface to anyone who has permission to view job workspaces.
Nexus does not care what kind of artifacts you drop there. Its a good idea to use it.
Related
Is it possible in Jenkins to only delete specific artifacts created by a build job?
In my situation I create 4 artifacts - 3 of which are 2kb .txt files and one of which is 0.5gb tar.gz. I'd like to keep the 3 smaller artifacts indefinitely, but aggressively remove the tar.gz artifacts after 5 builds.
Using logrotator this doesn't seem to be an option (it can only remove all artifacts or none) but I was hoping there might be a way to write a Groovy library that could be called after each new build, or a pluggin that could handle this for me?
See: https://javadoc.jenkins.io/hudson/tasks/LogRotator.html
Does anyone have any pointers on how to get started with this?
You can safely remove build artifacts directly on filesystem level.
If you want to trigger this directly from within Jenkins, the easiest way is probably to write a Groovy post-build script that takes care of the deletion.
I've used jenkins for quite a few years but have never set it up myself which I did at my new job. There are a couple questions and issues that I ran into.
Default workspace location - It seems like the latest Jenkins has the default workspace in Jenkins\jobs[projectName]\workspace and is overwritten (or wiped if selected) for every build. I thought that it should instead be in Jenkins\jobs[projectName]\builds[build_id]\ so that it would be able to store the workspace state for every build for future reference?
Displaying workspace on the project>Build_ID page - This goes along with the previous as I expected each 'workspace' for previous builds to show here. Currently in my setup this individual page gives you nothing except the Git revision and what repo changes triggered the build. As well as console output. Where are the artifacts? Where is the link to this build's workspace that was used?
Archiving Artifacts in builds - When choosing artifacts, the filter doesn't seem to work. My build creates a filestructure with the artifacts in it inside workspace. I want to store this and the artifacts filter says it starts at workspace. So I put in 'artifacts' and nothing gets stores (also where would this get stored?). I have also tried '/artifacts' and 'artifacts/*'.
Any help would be great! Thanks!
It does seem like you are confused about several aspects of Jenkins.. I think your question basically boils down to the following.
What is a difference between a workspace and a build?
So, here are some thoughts on this topic:
Builds are historical data. They (usually) don't change like a workspace does during building/checkout.
Builds contain information about a run (e.g. its status, build number, change log, etc) and any artifacts that you tell it to archive (logs, test results, etc). They (usually) don't contain source code like a workspace.
Builds are stored in the Jenkins\jobs\[projectName]\builds\[build_id]\ directory. This is a directory managed by Jenkins and you (usually) do not need to modify anything in this directory. However, workspaces are directories meant for the build and you can do pretty much anything with them and place them anywhere (it does not need to be in the default Jenkins\jobs\[projectName]\workspace directory.
Workspaces should be able to be wiped at any given time. To restore it, just rebuild the job with the same parameters/revision. If you need to keep something after a build, tell Jenkins to archive it before the build is done.
In regard to saving the entire state, I don't think you need to do that. As mentioned in #4, you should be able to reproduce the same build by kicking off the same revision/parameters as the build in question. If you cannot get back to the original state from the same revision/parameters, then that might be something to strive for as debugging is going to be a nightmare. :)
A workspace is an aspect of the project and not a build and that is why there is no link to the workspace from that page. Again, a build is just saved data from a previous run. A project uses the workspace to build stuff and that is why you can get to the workspace from that page.
In regard to how to save artifacts, you must specify the names of the files you want to save. Unless you are trying to save a file called "artifacts", then you should probably use something else. How about **/*.log for all log files? or **/*.xml for all xml files?
Hope this helps.
I'd like to get a hint how (which plugin) it is possible run SINGLE Jenkins job by the user chosen way. User MUST be able to choose the job he/she wants to run and choose the rule of execution:
E.g:
Create only jar files;
Create jars and send them over ssh
Create jars, generate documentation, etc...
I've found out a few plugins (Artifactory, Release plugin) but seems they don't support such logic.
I know that such thing can be implemented by creating several jobs, but this would require additional disk space.
Many Thanks!
In order to solve my issue, I've decided to create a few Jenkins jobs with the same custom workspace. So that, when a IT engineer runs any of these "connected" (which have the same workspace) jobs the workspace is updated (have a look at the CVS rules for your job) and that's why we avoid wasting of space.
Additionally, its (job) behaviour can be configured easily => the sets of rules (shell scripts, gradle, batch etc) and their sequence in order to achieve the desired result.
The last advantage, but not the least one, is that the security (access control) is still very easy to configure.
I think, that is the correct way.
My understanding
As far as I understood artifacts up to now, is that it was used to know if my project refers to an unfinished build task's artifacts.
Problem
I tried reading more (on jenkins site too) but I'm not sure I understand so easily what they do now. I know that when I promote a build, I can fingerprint artifacts. What does it mean then?
Artifacts - anything produced during the build process.
Fingerprinting artifacts - recording the MD5 checksum of selected artifacts.
You can then use the MD5 checksum to track a particular artifact back to the build it came from.
Adding to #Slav answer, Fingerprinting help Jenkins in keeping track of which version of a file is used by which version of a dependency.
Quoting an example and how it works from Jenkins Page:
For example:
Suppose you have the TOP project that depends on the MIDDLE project, which in turn depends on the BOTTOM project.
You are working on the BOTTOM project. The TOP team reported that bottom.jar that they are using causes an Null Pointer Exception, which you (a member of the BOTTOM team) thought you fixed in BOTTOM #32.
Jenkins can tell you which MIDDLE builds and TOP builds are using (or not using) your bottom.jar #32.
How does it work?
The fingerprint of a file is simply a MD5 checksum. Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.
To avoid the excessive disk usage, Jenkins does not store the actual file. Instead, it just stores md5sum and their usages. These files can be seen in
$JENKINS_HOME/fingerprints
I'm trying to work out a way to backup and restore jenkins so we can provision a new jenkins automatically.
I cannot work out a way to backup+restore /path/to/jenkins/plugins without including the binaries. We would like the backup to be in xml file format, just as everything else in jenkins. My assumption was that we could somehow backup xml files, and jenkins could restore the plugin binaries if they are missing, since it has access to maven.
I would prefer to avoid using config management tools to install plugins, as I then I have to manage versions of plugins in a way that feels too controlled. I'm happy to just backup what ever is there, and restore it elsewhere when needed. The developers should be free to install plugins, without involving me or puppet.
Googl'ing the issue is difficult since "plugin" is used in so many other contexts.
Below link says it governs plugins as well, but I cannot see how this is - maybe I'm missing something.
http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion
I have ported the idea to use git and it generally works, except that plugins do not re-appear by magic on the new machine - only the default plugins come back.
Can anyone suggest?
If you don't want to back up the plugin binary files, you can use the Jenkins REST API to get the list of current plugins:
http://jenkins:8080/pluginManager/api/json?tree=plugins[shortName,version]&pretty=true.
(You can use tree=plugins[*] to see a more complete list of fields in the API.)
Save this data as part of your configuration backup and use the Jenkins API to restore the plugins when you're re-deploying.
There's additional documentation and how to update plugins on the pluginManager API page: http://jenkins:8080/pluginManager/api
The best idea I've come up with to date, is to split the instance into an OS disk and a Jenkins disk mounted on /var/lib/jenkins. Use your cloud provider's snapshot feature to backup the jenkins disk periodically. Jenkins is for many organisations I believe, always going to be a "flake" server, or a pet, that needs nurturing and does not benefit much from automation, other than what is used to maintain the OS.
To backup Jenkins components, you can use Handy Backup. A best practice is to set up at least daily backup in a differential or mixed (full/differential) mode. This is an advantage before using any plug-in, due to assured regularity of backups.