I'm running an on-prem TFS instance with two agents. Agent 1 has a local path where we store our artifacts. Agent 2 has to access that path over a network path (\agent1\artifacts...).
Downloading the artifacts from agent 1 takes 20-30 seconds. Downloading the artifacts from agent 2 takes 4-5 minutes. If from agent 2 I copy the files using explorer, it takes about 20-30 seconds.
I've tried adding other agents on other machines. All of them perform equally poorly when downloading the artifacts but quick when copying manually.
Anyone else experience this or offer some ideas of what might work to fix this?
Yes It's definitely the v2 that's causing the problem.
Our download artifacts step has gone from 2mins to 36mins. Which is completely unacceptable. Im going to try out agent v2.120.2 to see if that's any better...
Agent v2.120.2
I think it's because of the amount of files in our artifacts, we have 3.71GB across 12,042 files in 2,604 Folders!
The other option I will look into it zipping or creating a nuget package for each public artifact and then after the drop, unzipping! Not the ideal solution but something I've done before when needing to use RoboCopy which is apparently what this version of the Agent uses.
RoboCopy is not great at handling lots of small files, and having to create a handle for each file across the network adds a lot of overhead!
Edit:
The change to the newest version made no difference. We've decided to go a different route and use an Artifact type of "Server" rather than "File Share" which has sped it up from 26 minutes to 4.5 minutes.
I've found the source of my problem and it seems to be the v2 agent.
Going off of Marina's comment I tried to install a 2nd agent on 01 and it had the exact same behavior as 02. I tried to figure out what was different and then I noticed 01's agent version is 1.105.7 and the new test instance is 2.105.7. I took a stab in the dark and installed the 1.105.7 on my second server and they now have comparable artifact download times.
I appreciate your help.
Related
I was recently put in charge of all Jenkins-related work at my job, and was tasked with storing build artifacts from our declarative pipelines in a place where:
- They are accessible to everyone on the team
- They can be stored for long periods of time
Ideally they would be visible on the Jenkins interface, where they appear when using the default 'archiveArtifacts' command. I know this saves them in the JENKINS_HOME directory. The problem is that I have to discard old builds to avoid running out of space and the artifacts are deleted with them. Furthermore, I don't have access to the server that Jenkins runs on because it's managed by a separate team, so I can't go into JENKINS_HOME.
I looked into a few ARMs like Nexus and Artifactory, but from my understanding those are only supposed to be used for full releases. I'm looking to save artifacts after every new merge, which can happen multiple times a day.
I'm currently saving them on a functional user's home directory, but I'm the only one with direct access to it so that's no good. I also looked into plugins like ArtifactDeployer, which doesn't support pipelines and only does as much as a 'cp' command as far as I could tell.
I ended up creating some freestyle jobs that copy artifacts from the pipelines and save them directly in their workspace. This way they're stored on our Jenkins slaves and visible through the interface to anyone who has permission to view job workspaces.
Nexus does not care what kind of artifacts you drop there. Its a good idea to use it.
I have Jenkins version 1.6.11 installed onto a windows server, the number of jobs configured are huge and the load is distributed among multiple Master & slave's. There are couple of issues occurring very frequently,
The whole Jenkins UI becomes so slow, either Jenkins server needs to restarted or the whole server needs to restarted to bring it back to normal.
Certain jobs are taking way too much time to load. To fix this, that particular job has to be abandoned and new ones has to be created for the same.
It would be really helpful if you could provide possible solutions for the two issues.
Use this syntax /node_modules/ to remove node modules folder, but before you do this, you should exclude .git folder using /.git/
We are migrating 50+ .net project from TFS to GitHub, at the same, we want to use Jenkins to automate the build. Currently all the builds are done inside the Visual Studio manually. I know how to automate this build using MSBuild and we already have a lot of these projects building inside Jenkins.
My question: is there a way to set up these 50+ project quickly w/o creating them one by one manually? Anyway to script them? e.g. a Jenkins project has everything inside a folder, I can copy a sample project/folder to create a new one and modify something. Or create a Jenkins project using a script reading a config file? Any idea can save some time is appreciated.
Not a direct answer but too long for a comment so here it goes anyway. Following the Joel test (which in no way is dogmatic for me but does make a lot of good points), and in my experience, you should already have an msbuild file now to build all those projects 'in one click'. Then, setting up a build server, in fact any build server, is just a matter of making it build that single parent project. This might not work for everyone, but for several projects I've worked on this had the following advantages:
the entire build process gets defined by developpers, working locally on their machine, using 'standard' tools
as such they don't need to spend hours in a web interface figuring out the appropriate build steps, dependencies and whatnot (also those hours would have been worthless in the end if switching to a different build server)
since a complete build is now just a matter of msbuild master.proj, possibly along with some options to define configuration/platform/output directories getting this running on any build server should be painless and quick
in the same manner this makes it easy to test different build servers with a minimum of time and migrate between them (also no need to ask SO questions on how to set everything up :)
this also makes it easy for other developpers to get complete builds as well without having to go round via a build server
Anecdote: we once had Jenkins running on multiple different projects as well. It took us days to get everything running, with the templates etc, and we found the web intercae slow and cumbersome (and getting to know the API would have taken even more days). Then one day I got sick of this and made a bunch of msbuild scripts which could build everything from one msbuild command. That took much less time than setting up Jenkins, a couple of hours or so. Then I took a TeamCity installation we already had and made it build the new master project. Took like an hour and everything worked. Just recently I took the same project and got it working on Visual Studio Online, again in no time.
If those projects are more or less similar to build, you will probably be interested in using the template plug-in for jenkins. There you configure a dummy project such that it does what is common to (most of) the 50+ projects.
Afterwards you create a separate project for each: Create the first project and make it use the template project for each of the steps which can be shared with the template project (use build step from other project). All subsequent projects can be created as slightly adopted copy of this first 'real' project.
I use it such that the variable $JOB_NAME (the actual project name in jenkins that is) is part of the repository path and I can thus clone from http://example.org/$JOB_NAME/
Configured that way, I can include the source code management step in the templating job and use it unmodified. Similar with the build step and post-build step: they are run by a script which is somewhat universal accross all my projects (mostly calling make and guessing deployment / publication paths upon $JOB_NAME again).
My understanding
As far as I understood artifacts up to now, is that it was used to know if my project refers to an unfinished build task's artifacts.
Problem
I tried reading more (on jenkins site too) but I'm not sure I understand so easily what they do now. I know that when I promote a build, I can fingerprint artifacts. What does it mean then?
Artifacts - anything produced during the build process.
Fingerprinting artifacts - recording the MD5 checksum of selected artifacts.
You can then use the MD5 checksum to track a particular artifact back to the build it came from.
Adding to #Slav answer, Fingerprinting help Jenkins in keeping track of which version of a file is used by which version of a dependency.
Quoting an example and how it works from Jenkins Page:
For example:
Suppose you have the TOP project that depends on the MIDDLE project, which in turn depends on the BOTTOM project.
You are working on the BOTTOM project. The TOP team reported that bottom.jar that they are using causes an Null Pointer Exception, which you (a member of the BOTTOM team) thought you fixed in BOTTOM #32.
Jenkins can tell you which MIDDLE builds and TOP builds are using (or not using) your bottom.jar #32.
How does it work?
The fingerprint of a file is simply a MD5 checksum. Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.
To avoid the excessive disk usage, Jenkins does not store the actual file. Instead, it just stores md5sum and their usages. These files can be seen in
$JENKINS_HOME/fingerprints
There is some Jenkins plugin to ZIP old builds? I don't want to package just the generated archive (I am deleting those). I want to ZIP just the log data and the data used by tools, like FindBugs, Checkstyle, Surefire, Cobertura, etc.
Currently I am running out of disk space due to Jenkins. There are some build log files that reach 50 Mb due running 3000+ unit tests (most of these are severely broken builds full of stacktraces in which everything fails). But this happens frequently in some of my projects, so I get this for every bad build. Good builds are milder and may get up to around 15 Mb, but that is still a bit costly.
The surefile XML files for these are huge too. Since they tend to contain very repetitive data, I could save a lot of disk space by zipping them. But I know no Jenkins plugin for this.
Note: I am already deleting old builds not needed anymore.
The 'Compress Buildlog' plugin does pretty much exactly what you're asking for, for the logs themselves at least.
https://github.com/daniel-beck/compress-buildlog-plugin
For everything else, you'll probably want an unconditional step after your build completes that manually applies compression to other generated files that will stick around.
The administering Jenkins guide gives some guidance on how to do this manually. There are also links to the following plugins
Shelve Project
thinBackup
The last one is really designed to backup Jenkins configuration, but there are also options for build results.
Although this question is early 3 years ago there may be other people search the same question
here is my answer
If you want to compress the current build job's log using This jenkins plugin
If you want to compress the old jenkins jobs using the following script mtime +5 means the file change time is 5 days ago
cd "$JENKINS_HOME/jobs"
find * -name "log" -mtime +5|xargs gzip -9v '{}'