I have created a vNext build definition in TFS 2018. I'm able to queue this
build definition many times and they were processed parallel.
This behaviour is not acceptable. If they are processed in order, I can live with it.
Does anyone know which setting I have to setup in the definition?
I'm running a tfs in following environment:
TFS 2018.2 On-Premise
4 Build-Agents with different capabilities
1 Build-Definition which fulfills the capabilities of all agents
It is planned, to have more build definitions. Today, we have only one because we are migrating our xaml-builds to vNext.
On the Triggers tab, check the "Batch Changes While Build is in Progress" option and set the number of concurrent builds to 1.
However, I would strongly recommend fixing whatever prevents your builds from running in parallel. Builds should ideally be stateless and have no impact on one another.
You can also set a custom Demand/Capability pair such that only one of your agents satisfies your build definition's requirements.
My company has a bunch of different builds and half a dozen different build agents, and I need to update some software for one of the builds. I don't want to break any other builds that are using said agent. I would like to get a list of all builds that use said agent so that I can validate them after my software updates on the agent. I would prefer not to individually review each build, as there are dozens, if not hundreds of them. Is there some way to get this information quickly? Either from the agent, or from TFS somehow?
By default, builds are tied to controllers, not agents and could therefore run on any of the agents bound to the controller. Unless as suggested by Daniel Mann, you have your builds tagged to specific agents you won't be able to get that level of detail. Without tagging, your report would be limited to a list of machines that each build could possibly run on.
What I do in this situation is to have a separate, private build controller for build software testing. Upgrade the software on this and then queue test builds for the affected definitions, changing the controller to your test controller in the Queue Build parameters. Once you've verified that your changes won't break the builds you can then schedule downtime to upgrade the production agent machines.
On Jenkins is there a way to setup a job to have two distinct builds that can be triggers for different reasons.
As an example
(1) First build handles releases and is only started manually.
(2) Second build runs automatically every time a commit occurs.
I know that I can split this into two different jobs each with its only build and triggers however I prefer not to.
Do the builds have anything in common? If they share the same build process, but the release requires extra pre/post-build steps, then Release Plugin is exactly what you need.
It allows to define additional pre/post-build steps that are executed only when manually triggering a release build.
However this goes against the CI concept. In CI, your eventually-released-build would have been promoted from a regular automatic CI build.
Background
I am using Jenkins with the Build Pipeline plugin to build some fairly complicated projects that require multiple compilation steps:
Build source RPM.
Build binary RPM (this is performed twice, once for each platform).
Deploy to YUM repository.
My strategy for solving build requirements involves splitting the common work into parameterized jobs that can be reused across projects and branches, with each job representing one stage in the pipeline. Each stage is triggered with parameters, and build artifacts passed along to the next job in the pipeline. However, I'm having some trouble with this strategy, and could really use some tips on how to go about solving this problem in the most elegant and flexible way possible.
To be more specific, there are two common libraries, which are shared by other projects (but not all projects). The libraries are built differently from the dependent projects, but it should not be necessary to specify in Jenkins what the dependent projects are.
There are multiple branches, the master branch (rebuilt nightly), the develop branch (polled for changes), feature branches (also polled), and release branches (polled, but built for release). The branches are built the same way across multiple projects.
We create multiple repositories every month, and whilst it is feasible to expect a little setup for a new project, generally we want this to be as simple and automated as possible.
The Problems
I have many projects with multiple branches, and I do not wish to build all branches or even all projects in the same way. Because most of the build steps are similar I can turn these common steps into parameterized build jobs, and get each job to trigger the next in the chain, passing parameters and build artifacts along the chain. However, this falls apart if one of the steps needs to be skipped, because I don't know of a way to conditionally skip a build step. This implies I would need to copy the build jobs so that I can customise them for each pipeline, resulting in a very large number of build jobs. I could use a combination of plugins to create a job generator (eg. dsl flow, dsl job, etc), and hide as much as possible from the users, but what's the most elegant Jenkins solution to this? Are there any plugins, or examples that I might have missed? What's your experience of doing this?
Because step 2 can be split into two jobs that can be run in parallel, this introduces a complexity that is causing me problems with my pipeline. My first attempt would trigger a parameterized build job twice with different parameters, and then join the jobs afterwards using the join plugin, but it was beginning to look like it would be complicated to copy in the build artifacts from the two upstream jobs. This is significant, because I need the build artifacts from both jobs for stage 3. What's the most elegant solution to join parallel jobs and copy artifacts from them all? Are there any examples that I might have missed?
I need to combine test results generated from both of the jobs in stage 2, and copy them to the job that triggers the build. What's the best way to handle this?
I'm happy to read articles, presentations, technical articles, reference documentation, write scripts and whatever else necessary to make this work nicely, but I'm not a Jenkins expert. If anyone can give me some advice on these 3 problems then that would be helpful. Additionally, I would appreciate any constructive advice on how to get the best out of pipeline CI builds in Jenkins, if relevant.
For the first point, the Job Generator plugin I wrote has been developed to address this use case. You can find more info on the wiki page of Job Generator.
There is also the same type of plugin with a different approach (job generator as a build step), it is called Jobcopy Builder.
The other approaches you mentioned require some kind of DSL and can be a good choice too.
Why are there two kinds of jobs for Jenkins, both the multi-configuration project and the free-style project project? I read somewhere that once you choose one of them, you can't convert to the other (easily). Why wouldn't I always pick the multi-configuration project in order to be safe for future changes?
I would like to setup a build for a project building both on Windows and Unix (and other platforms as well). I found this question), which asks the same thing, but I don't really get the answer. Why would I need three matrix projects (and not three free-style projects), one for each platform? Why can't I keep them all in one matrix, with platforms AND (for example) gcc version on one axis and (my) software versions on the other?
I also read this blog post, but that builds everything on the same machine, with just different Python versions.
So, in short: how do most people configure a multi-configuration project targeting many different platforms?
The two types of jobs have separate functions:
Free-style jobs: these allow you to build your project on a single computer or label (group of computers, for eg "Windows-XP-32").
Multi-configuration jobs: these allow you to build your project on multiple computers or labels, or a mix of the two, for eg Windows-XP, Windows-Vista, Windows-7 and RedHat - useful for checking compatibility or building for multiple platforms (qt programs?)
If you have a project which you want to build on Windows & Unix, you have two options:
Create a separate free-style job for each configuration, in which case you have to maintain each one individually
You have one multi-configuration job, and you select 2 (or more) labels/computers/slaves - 1 for Windows and 1 for Unix. In this case, you only have to maintain one job for the build
You can keep the your gcc versions on one axis, and software versions on another. There is no reason you should not be able to.
The question that you link has a fair point, but one that does not relate to your question directly: in his case, he had a multi-configuration job A, which - on success - triggered another job B. Now, in a multi-configuration job, if one of the configuration fails, the entire job fails (obviously, since you want your project to build successfully on all your configurations).
IMHO, for building the same project on multiple platforms, the better way to go is to use a multi-configuration style job.
Another option is to use a python build step to check the current OS and then call an appropriate setup or build script. In the python script, you can save the updated environment to a file and inject the environment again using the EnvInject plugin for subsequent build steps. Depending on the size of your build environment, you could also use a multi-platform build tool like SCons.
You could create a script (e.g. build) and a batch file (e.g. build.bat) that get checked in with your source code. In Jenkins in your build step you can call $WORKSPACE/build - Windows will execute build.bat whereas Linux will run build.
An option is to use user-defined axis combined with slaves(windows, linux, ...), so you need to add a filter for each combination and use the Conditional BuildStep Plugin to set the build step specific for each plataform(Executar shell, Windows command, ...)
This link has a tutorial but it is in portuguese, but it's easy to work it out based on image...
http://manhadalasanha.wordpress.com/2013/06/20/projeto-de-multiplas-configuracoes-matrix-no-jenkins/
You could use the variable that jenkins create when you define a configuration matrix axis. For example:
You create a slave axis with name OSTYPE and check the two slaves (Windows and Linux). Then you create two separate build steps and check for the OSTYPE environment variable.
You could use a improved script language instead, like python, which is multi-platform and can achieve the same functionality independent of the slaves' name and in just one build step.
If you go the matrix route with Windows and something else, you'll want the XShell plugin. You just create your two build scripts such as "build.bat" for cmd and "build" for bash, and tell XShell to run "build". The right one will be run in each case.
A hack to have batch files run on Windows and shell scripts on Unix:
On Unix, make batch files exit with 0 exit status:
ln -s /bin/true /bin/cmd
On Windows, either find a true.exe, name it sh.exe and place it somewhere in the PATH.
Alternatively, if you have any sh.exe installed on Windows (From Cygwin, Git, or other source), add this to the top of the shell script in Jenkins:
[ -n "$WINDIR" ] && exit 0
Why wouldn't you always pick the multi-configuration job type?
Some reasons come to mind:
Because jobs should be easy to create and configure. If it is hard to configure any job in your environment, you are probably doing something wrong outside the scope of the jenkins job. If you are happy that you managed to create that one job and it finally runs, and you are reluctant to do this whole work again, that's where you should try to improve.
Because multi configuration jobs are more complex. They usually require you to think about both the main job and the different sub job variables, and they tend to grow in complexity to a level beyond being manageable. So in a single job scenario, you'd probably waste thoughts on not using that complexity, and when extending the build variables, things might grow in the wrong direction. I'd suggest using the simple jobs as default, and the multi configuration jobs only if there is a need for multiple configurations.
Because executing multi configuration jobs might need more job slots on the slaves than single jobs. There will always be a master job that is executed on a special, invisible slot (that's no problem by itself) and triggers the sub jobs, but if these sub jobs do themselves trigger sub jobs, you might easily end in a deadlock if there are more sub jobs than slots, and some sub jobs trigger again sub jobs that then cannot execute because there are no more open slots. This problem might be circumvented by using some configuration setup on the slaves, but it is present and might only occur if several multi jobs run concurrently.
So in essence: The multi configuration job is a more complex thing, and because complexity should be avoided unless necessary, the regular freestyle job is a better default.
If you want to select on which slave you run the job, you need to use multi-configuration project (otherwise you won't be able to select/limit slaves on which you run it – there are three ways to do it, however I've tried them all (Tie plugin works only for master job, Restrict in Advanced Project Options is not rock-safe trigger as well so you want to use Slave axis that is proven to work correctly today.)