My understanding
As far as I understood artifacts up to now, is that it was used to know if my project refers to an unfinished build task's artifacts.
Problem
I tried reading more (on jenkins site too) but I'm not sure I understand so easily what they do now. I know that when I promote a build, I can fingerprint artifacts. What does it mean then?
Artifacts - anything produced during the build process.
Fingerprinting artifacts - recording the MD5 checksum of selected artifacts.
You can then use the MD5 checksum to track a particular artifact back to the build it came from.
Adding to #Slav answer, Fingerprinting help Jenkins in keeping track of which version of a file is used by which version of a dependency.
Quoting an example and how it works from Jenkins Page:
For example:
Suppose you have the TOP project that depends on the MIDDLE project, which in turn depends on the BOTTOM project.
You are working on the BOTTOM project. The TOP team reported that bottom.jar that they are using causes an Null Pointer Exception, which you (a member of the BOTTOM team) thought you fixed in BOTTOM #32.
Jenkins can tell you which MIDDLE builds and TOP builds are using (or not using) your bottom.jar #32.
How does it work?
The fingerprint of a file is simply a MD5 checksum. Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.
To avoid the excessive disk usage, Jenkins does not store the actual file. Instead, it just stores md5sum and their usages. These files can be seen in
$JENKINS_HOME/fingerprints
Related
As is good practise, I've got Jenkins set up at work to automatically build everything for continuous integration, pulling files from our Git repositories. On our development branches, builds get kicked off automatically whenever anyone commits a change. When we want to do formal testing, we pull the build from Jenkins and use that; and when we want to sign off a change request, we quote the Jenkins build number where the change went in. So far, so good.
The problem we have is that builds are a significant size. For our SDK, we have to build across multiple platforms so that we can check it works on all of them. At maybe 50MB per build, this starts to mount up! Short term I can keep asking IT to give me more storage space, but longer term I'd like a more strategic solution
The obvious answer in Jenkins is to set up deletion rules, whether deleting after some time or after some number of builds. The problem then though is that if we delete that older development build, we lose the traceability of what we tested. I'm sure most engineers at one time or another have had to do a binary chop through older builds to find an obscure bug/regression which was only spotted some time later. For me, it is unacceptable to lose that history.
The important feature of build history though is not the binary build artifacts, but the build log recording what Git commits (or anything else; toolchain versions for example) went into each build. That's what lets us go back to investigate older builds and recreate them if required. The build log is relatively small (and highly compressible, being a text file). We do still need to keep build artifacts for recent builds though, so that testers can use them. So I'm thinking a better alternative would be to preserve the build log in Jenkins for all builds, but to have Jenkins automatically delete build artifacts after some time.
Does anyone know of a way in Jenkins (perhaps a plugin?) which would let us automatically delete/archive build artifacts from older builds, but still keep the build details and log for those builds? I'm happy to do a Jenkins upgrade if necessary to get this feature. And of course this needs to be only for selected development build jobs - all release build jobs need their build artifacts to be preserved forever, as do any builds which have the "keep forever" button ticked.
If it's absolutely necessary, I could set up a separate cron job to do this on the Jenkins file area. That's a nasty hack though, and I suspect it's likely to cause some issues with Jenkins, so I'd rather not do something that brute-force if there's a better alternative.
I think you need this option in your jenkinsfile
buildDiscarder(logRotator(artifactNumToKeepStr: '10'))
artifactNumToKeepStr: This number of builds have their artifacts kept.
I was recently put in charge of all Jenkins-related work at my job, and was tasked with storing build artifacts from our declarative pipelines in a place where:
- They are accessible to everyone on the team
- They can be stored for long periods of time
Ideally they would be visible on the Jenkins interface, where they appear when using the default 'archiveArtifacts' command. I know this saves them in the JENKINS_HOME directory. The problem is that I have to discard old builds to avoid running out of space and the artifacts are deleted with them. Furthermore, I don't have access to the server that Jenkins runs on because it's managed by a separate team, so I can't go into JENKINS_HOME.
I looked into a few ARMs like Nexus and Artifactory, but from my understanding those are only supposed to be used for full releases. I'm looking to save artifacts after every new merge, which can happen multiple times a day.
I'm currently saving them on a functional user's home directory, but I'm the only one with direct access to it so that's no good. I also looked into plugins like ArtifactDeployer, which doesn't support pipelines and only does as much as a 'cp' command as far as I could tell.
I ended up creating some freestyle jobs that copy artifacts from the pipelines and save them directly in their workspace. This way they're stored on our Jenkins slaves and visible through the interface to anyone who has permission to view job workspaces.
Nexus does not care what kind of artifacts you drop there. Its a good idea to use it.
In our builds there is sometimes a need (for one or another reason) to copy/create some files that are required during the build, but are not actually artifacts.
As it is a bit problematic to guarantee cleanup/deletion of those files after the build in all cases, they should obviously be stored in some directory that is managed by the build pipeline, so when the build completes or thereafter that directory will be automatically cleaned up.
So far we have been using Build.ArtifactStagingDirectory to get the expected behavior as it is the only one that is
purged before each new build, so you don't have to clean it up yourself.
Still, as such files are not really artifacts, it seems to be slightly inappropriate.
Perhaps there is a better alternative?
$(Build.BinariesDirectory) can also be configured to be purged before each build. That may be more appropriate for temporary build outputs.
However, there's no right answer to this.
I've used jenkins for quite a few years but have never set it up myself which I did at my new job. There are a couple questions and issues that I ran into.
Default workspace location - It seems like the latest Jenkins has the default workspace in Jenkins\jobs[projectName]\workspace and is overwritten (or wiped if selected) for every build. I thought that it should instead be in Jenkins\jobs[projectName]\builds[build_id]\ so that it would be able to store the workspace state for every build for future reference?
Displaying workspace on the project>Build_ID page - This goes along with the previous as I expected each 'workspace' for previous builds to show here. Currently in my setup this individual page gives you nothing except the Git revision and what repo changes triggered the build. As well as console output. Where are the artifacts? Where is the link to this build's workspace that was used?
Archiving Artifacts in builds - When choosing artifacts, the filter doesn't seem to work. My build creates a filestructure with the artifacts in it inside workspace. I want to store this and the artifacts filter says it starts at workspace. So I put in 'artifacts' and nothing gets stores (also where would this get stored?). I have also tried '/artifacts' and 'artifacts/*'.
Any help would be great! Thanks!
It does seem like you are confused about several aspects of Jenkins.. I think your question basically boils down to the following.
What is a difference between a workspace and a build?
So, here are some thoughts on this topic:
Builds are historical data. They (usually) don't change like a workspace does during building/checkout.
Builds contain information about a run (e.g. its status, build number, change log, etc) and any artifacts that you tell it to archive (logs, test results, etc). They (usually) don't contain source code like a workspace.
Builds are stored in the Jenkins\jobs\[projectName]\builds\[build_id]\ directory. This is a directory managed by Jenkins and you (usually) do not need to modify anything in this directory. However, workspaces are directories meant for the build and you can do pretty much anything with them and place them anywhere (it does not need to be in the default Jenkins\jobs\[projectName]\workspace directory.
Workspaces should be able to be wiped at any given time. To restore it, just rebuild the job with the same parameters/revision. If you need to keep something after a build, tell Jenkins to archive it before the build is done.
In regard to saving the entire state, I don't think you need to do that. As mentioned in #4, you should be able to reproduce the same build by kicking off the same revision/parameters as the build in question. If you cannot get back to the original state from the same revision/parameters, then that might be something to strive for as debugging is going to be a nightmare. :)
A workspace is an aspect of the project and not a build and that is why there is no link to the workspace from that page. Again, a build is just saved data from a previous run. A project uses the workspace to build stuff and that is why you can get to the workspace from that page.
In regard to how to save artifacts, you must specify the names of the files you want to save. Unless you are trying to save a file called "artifacts", then you should probably use something else. How about **/*.log for all log files? or **/*.xml for all xml files?
Hope this helps.
Having real problems creating artifacts in teamcity 6.5 (using TFS & MSBuild as the buildrunner if it makes any odds, which it probably does as any examples I find seem to use SVN...).
The Build works, so long as I enter no checkout rules.
If I understand it, I'll need to set up some artifacts, that themselves rely on checkout rules(?).
I have two builds that are identical other than the way they are kicked off.
One is initiated on check-in
One is initiated manually from within TC. This build is the Test Build
Assembly version numbers come from a single versioninfo.cs file that is a linked file in all projects in the solution. This method is detailed here : http://www.codeproject.com/Articles/328977/The-Right-Way-to-Version-Your-Assemblies and holds the version number thus:
[assembly: AssemblyFileVersion("9.1.0.0")]
Ultimately, I'm unable to copy the output of the test build to another location.
As it stands, the only output of a build is in the teamcity data directory, for example :
C:\TeamCity\buildAgent\work\ceaaf65dc87ff856\Project1\bin\Debug
C:\TeamCity\buildAgent\work\ceaaf65dc87ff856\Project2\bin\Debug
etc
I'd like to copy the output files (exes and DLLs) to an output folder which has the build number of the build on it
For arguments sake, lets say for the version number above, this would be to
c:\BuildServer_Output\SolutionName\9.1.0.0
Currently I have not been able to create artifact paths that actually do anything - i.e. to copy anything anywhere.
For instance I have acoupe of artifact paths, but nothing ever gets put into C:\BuildServer_TestBuilds -
+:Accounts\bin\debug* => C:\BuildServer_TestBuilds
+:BackOffice\bin\debug* => C:\BuildServer_TestBuilds
Am I getting no artifacts (and my artifact paths therefore ignored) because I have no checkout rules?
Any help would be appreciated.
I am pretty sure artifacts and checkout rules are completely independent. Artifacts just deal with what has been built. Checkout rules tell teamcity how to react to and checkout changes in the VCS.
It looks like your artifact paths are beginning with absolute paths. I have always found it easier to use relative paths with wildcards. That way I don't need to worry about where teamcity put the build. We use the following to get all dlls and exes to one folder
**\bin\Debug\*.*=>deploymentdir
Our build configuration page has an artifacts link and when we open it it will have things like
deploymentdir\common\bin\debug\common.dll
deploymentdir\common\bin\debug\common.pdb
deploymentdir\runner\bin\debug\runner.exe
In one of our other builds we use an msbuild script to flatten our output before putting it through the artifact process.
We do use checkout rules but we have not had to change our artifact paths to accommodate them.