I have a large solution, with many projects and many files, and only one build configuration, Release. I am using TFS, and the complete rebuild takes like 2 hours.
Is it possible to distribute the build across several agents, so that they will compile different projects, or, even better, different files? Something like dictcc? I can distribute the build on up to 10+ different machines, but the build only works on one.
For now, my impression is that agents can only have specialized jobs, like build, run tests, etc, but not split and distribute only one build.
I already tried optimizing the build, but still the project is big and can benefit on parallel build
You can but but must roll-up your sleeves: there is no built-in template that helps, but Jim explains how to make one.
Do not forget that you can also leverage multi-CPU/Core as explained in Building Multiple Projects in Parallel with MSBuild.
Your best option would be to break your solution down into defunct components that can be built separately.
If you seperate each bit and build and test before publishing as Nuget you can distribute easily across build servers and even only build the bits that have changed.
This process will also work in the new build system coming in 2015 that does not use XAML.
Related
I'm looking for a way to use Jenkins to build a single code base for multiple CPU architectures. at the moment this is amd_64 and armhf, although this may expand in the future. The ideal situation would be to run the build over a number of different jenkins slaves with a different CPU architectures.
These build jobs are not compiler based (maven, gradle ext.) but system independ shell scripts (bash and python) which auto detect their CPU architecture and produce build artifacts to match the CPU.
I may be missing something really obvious, but I don't see a way to automatically run a build a number of times over different architectures or bind a specific build to a specific architecture.
Could anyone point me in the right direction?
Funny you ask this question now. Published last Friday (2019-11-22) ...
You should review the Jenkins blog: Welcome to the Matrix
I often find myself needing to run the same actions on a bunch of
different configurations. Up to now, that meant I had to make multiple
copies of the same stages in my pipelines. When I needed to make
changes, I had to make the same changes in multiple places throughout
my pipeline. Maintaining even a small number of configuration was
difficult for larger pipelines.
Single configuration pipeline
Pipeline for multiple platforms and browsers
Excluding invalid combinations
We are migrating 50+ .net project from TFS to GitHub, at the same, we want to use Jenkins to automate the build. Currently all the builds are done inside the Visual Studio manually. I know how to automate this build using MSBuild and we already have a lot of these projects building inside Jenkins.
My question: is there a way to set up these 50+ project quickly w/o creating them one by one manually? Anyway to script them? e.g. a Jenkins project has everything inside a folder, I can copy a sample project/folder to create a new one and modify something. Or create a Jenkins project using a script reading a config file? Any idea can save some time is appreciated.
Not a direct answer but too long for a comment so here it goes anyway. Following the Joel test (which in no way is dogmatic for me but does make a lot of good points), and in my experience, you should already have an msbuild file now to build all those projects 'in one click'. Then, setting up a build server, in fact any build server, is just a matter of making it build that single parent project. This might not work for everyone, but for several projects I've worked on this had the following advantages:
the entire build process gets defined by developpers, working locally on their machine, using 'standard' tools
as such they don't need to spend hours in a web interface figuring out the appropriate build steps, dependencies and whatnot (also those hours would have been worthless in the end if switching to a different build server)
since a complete build is now just a matter of msbuild master.proj, possibly along with some options to define configuration/platform/output directories getting this running on any build server should be painless and quick
in the same manner this makes it easy to test different build servers with a minimum of time and migrate between them (also no need to ask SO questions on how to set everything up :)
this also makes it easy for other developpers to get complete builds as well without having to go round via a build server
Anecdote: we once had Jenkins running on multiple different projects as well. It took us days to get everything running, with the templates etc, and we found the web intercae slow and cumbersome (and getting to know the API would have taken even more days). Then one day I got sick of this and made a bunch of msbuild scripts which could build everything from one msbuild command. That took much less time than setting up Jenkins, a couple of hours or so. Then I took a TeamCity installation we already had and made it build the new master project. Took like an hour and everything worked. Just recently I took the same project and got it working on Visual Studio Online, again in no time.
If those projects are more or less similar to build, you will probably be interested in using the template plug-in for jenkins. There you configure a dummy project such that it does what is common to (most of) the 50+ projects.
Afterwards you create a separate project for each: Create the first project and make it use the template project for each of the steps which can be shared with the template project (use build step from other project). All subsequent projects can be created as slightly adopted copy of this first 'real' project.
I use it such that the variable $JOB_NAME (the actual project name in jenkins that is) is part of the repository path and I can thus clone from http://example.org/$JOB_NAME/
Configured that way, I can include the source code management step in the templating job and use it unmodified. Similar with the build step and post-build step: they are run by a script which is somewhat universal accross all my projects (mostly calling make and guessing deployment / publication paths upon $JOB_NAME again).
I have a TFS 2010 with some projects and a common library used in 5 of them. We use VS 2013 and we have Rolling Builds enabled in most if not all build definitions. When the common library is checked in, all of the projects referencing it are recompiled - but the order is poor, the most commonly used project is compiled as the last one. Is there a way to change that so it gets compiled first?
This question hints at a lot of problems and possible solutions. The simplest answer is probably to just add more build servers to run all the builds in parallel.
Otherwise you need to consider turning off rolling builds and writing your own build scheduler. That or other strategies such as building the DLL once and checking it in as source to the other builds or running the build on a branch that is outside of the other 5 builds and merging the source into those builds when they want to pick up the change.
If you are otherwise happy with how things are now and don't want to do a lot of work to solve the problem.... then just scale out your build farm with additional agents.
There's a "Priority in queue" on the build definition, but it sounds like you might want to change your solution's > Project Dependencies' > Build Order?
I am in the process of setting up continuous integration in our TFS system. One major part of our system are the development of about 50 DotNetNuke modules to support our CMS infrastructure. Right now, each of those projects have their own solution since their code bases are mostly siloed (with common code in 1 or 2 common projects). Keeping them in their own solution is done because it makes the development process faster (loading, compiling, etc....)
However, this has proven difficult to maintain when setting up TFS team build as each solution has to be manually added to the build definition and MSBuild seems unable to take advantage of parallel compiling due to each project being in its own solution. This causes about 5 minute full build times, which while isn't horrible isn't ideal. Mostly though, it's not ideal from a build definition maintenance aspect.
To solve this I creating a global solution that included all projects. The idea being that if you want your project to be automatically compiled and deployed by TFS you will have to include your project in the global solution. This seems works well, as it's easy to maintain from a build definition standpoint and brings the total build time down to 70 seconds.
The one problem is that the displayed TFS build log groups all warnings and errors together under the solution instead of separating them out by project. This makes it difficult to quickly see what project caused which errors and warnings.
Is there a good way to see project level error/warning messages in the build log summary view without delving into the cluttered build log?
To answer your direct question, I believe the answer is no (at least not without some heavy customization).
For me this is never a big concern as I am pretty aggressive about getting my teams to bring errors/warnings down to zero, then enforcing it via TFS Build (/p:TreatWarningsAsErrors=true). This means you should never have to wade through hundreds of warnings in the build summary.
If you add all your individual solutions to the build definition, you can always use the TFS Power Tools to "clone" a build def to make maintenance easier. You could also modify the Build Template to build the solutions/projects in parallel, although this runs the risk of having file contention issues.
How would you manage the lifecycle and automated build process when some of the projects (C# .csproj projects) are part of the actual build system?
Example:
A .csproj is a project that uses MSBuild tasks that are implemented in BuildEnv.csproj.
Both projects are part of the same product (meaning, BuildEnv.csproj frequently changes as the product is being developed and not a 3rd party that is rarely updated)
You must factor this out into two separate "projects" otherwise you'll spend ages chasing your tail trying to find out if a broken build is due to changes in the build system or chages in the code being developed.
Previously we've factored the two systems out into separate projects in CVS.
You want to be able to vary one thing while keeping the other constant to limit what you would have to look at when performing forensic analysis.
Hope that helps.