Exploring upgrade to on premise TFS 2017.3
We have a large project that takes about 20-30 minutes to compile. I'm planning on setting up multiple build agents on different servers to handle compilations on every check-in.
I enabled Continues Integration trigger and can run builds on all 10 agents in parallel (triggered by check-in). When I continue to check in files, the builds just put in queue waiting for agents to complete previous builds.
Selecting "Batch changes while a build is in progress" checkbox allows only one build to run even if I have 9 more agents ready for work.
How can I use all 10 agents and butch changes only when all of them are being used?
thanks.
This is the expected behavior, take a look at the behavior of "Batch changes" of Continuous Integration
Batch changes
Select this check box if you have a lot of team members uploading
changes often and you want to reduce the number of builds you are
running. If you select this option, when a build is running, the
system waits until the build is completed and then queues another
build of all changes that have not yet been built.
You can batch changes when your code is in Git in the project or on
GitHub. This option is not available if your code is in a remote Git
repo or in Subversion.
If you selected this option, then you will have to wait the build completed and queue another build. More details please take a look at the official link.
Related
We run an environment were multiple features are divided into branches. Those branches are configured to a channel in Octopus Deploy.
I would like to see if there is a way to check in a build definition if another build is queued for a branch.
For context - I have a Octopus Deploy step as the last step in our build definition. To deploy the app - it takes approximately 10 minutes. Our build process is also about 10 minutes (it is a big application).
When a team working on one feature checks in - we end up with 2 and 3 builds queued and waiting for each check-in and build to complete.
What I would like to do is have the deploy task run only if there is NOT another build queued for the same branch. This would ensure we don't waste 30 minutes of unnecessary deployments and only deploy the latest code.
What type of build do you use? XAML build or Vnext build. It seems that you use the CI trigger in your build definition, so it will trigger a build each time you do a check in.
VNext build:
You could select the Batch changes checkbox. According to this document,
If you select this option, when a build is running, the system waits until the build is completed and then queues another build of all changes that have not yet been built.
This will combin the changes into one build when you have a lot build queued.
XAML build:
In the build definition, you could use the Rolling builds trigger. This has the same function like the Batch changes mentioned above.
Note:
You also could use this REST API to get if there're builds of a build definition that are queued.
Http method: Get
http://servername:8080/tfs/DefaultCollection/TeamprojectName/_apis/build/builds?definitions=10&statusFilter=notStarted&api-version=2.0
It ended up being easier than I thought - but not exactly what I tried to do in the question.
The Octopus Create Release Task for TFS and VSTS has an option to "Show Deployment Progress". Checking this causes the build definition to wait - periodically receiving feedback from Octopus through the Octo.exe tool.
The deployment for this is actually a wrapper for several deployments (15 separate projects) - so it would take a long time.
Unchecking this option causes TFS to not wait - but send the create release / deploy command to Octopus.
Now this didn't solve my initial question - but Tingting0929-MFST did help me out in that I explored using the TFS Rest API. I wrote a PS script the effectively did what I asked - not deploy if there was a another build queued up for the same branch. I got it working - but it introduced a different problem in that the release notes from TFS and associated changesets / work items were getting swallowed in between releases.
In other words if three checkins came in fairly close separation, the last one would get deployed. Octopus would show only the release notes for last deployment and not the first two.
For those using Octopus Deploy in a feature branch set-up (one project deploying to multiple channels) - this is an option to get you the Continuous Deployment for each channel (branch).
I have one build controller and it has 4 build agents in it. When I run more than one build at a time it queues the build. Why it doesn't run 4 builds parallel? Is there any settings that I should change?
I am running TFS 2012.
running gated checkin
it also runs all the tests after the build is done.
Thanks for the help!
This is 'by design'. From Use a gated check-in build process to validate changes
Each gated check-in build definition can have only one running build at a time.
This is necessary to preserve the logic coherence of you version control.
If this is slowing your dev organization, you have many options (some are in the doc linked):
Change from Gated to Batched gated
Have multiple Build definitions (mapped on different portions of you sources)
Minimize the checks during build
Feed more resources to the Build server
Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run
On TFS 2010 we set up a build script to automatically run and deploy to our test environment no more than every 4 hours. So the trigger of the build definition is "Rolling builds - accumulate check-ins until the prior build finishes" with the check box checked for "Build no more often than every [240] minutes". The way I'm interpreting this is that if a build happens at 16:25, and there's 3 check-ins in the next 30 minutes (the last happening at 16:52), then the next build will happen at 20:25.
The issue we're seeing is that if this sequence of events happens on a Friday, then the next triggered build doesn't occur until Monday morning. We do have two build agents on their own separate virtual machine from TFS, but all machines are up and running 24/7.
There's nothing that anybody on the team is aware of that would cause this behavior, so we're at a loss for why we're seeing it. Are we simply misunderstanding the meaning of Rolling Build, or is there configuration setting we forgot to check?
EDIT: For what it's worth, the entirety of the build & deployment process takes all of 90-120 seconds. So there were not any check-ins that occurred while it was still building.
The Accumulate check-ins means that when you check-in a changeset, if there is no build of this build definition in 240 minutes, TFS will queue a build automatically. But if do not check-in a changeset, no builds will be queued.
Builds should be triggered as you wrote. 3rd build should be triggered after the first check-in after 20:25 build not before 0:25 on Saturday. :)
Try to check if changesets were not checked into the location cloaked in the build definition workspace (or checked into the location not mapped by build definition workspace).
Consider TFS 2010's ability for a Build Controller to have 1+ build agents. Since builds are a subjective topic to the team/environment, consider an environment where builds are performed on commit/check-in. Each Project Collection will have 10+ Team Projects, but perhaps only 1 or 2 are being committed to in a day.
When should a TFS administrator consider creating a new build agent?
Do multiple agents run in parallel?
When a single agent is defined to a Build Controller, does it run serially?
MSDN states: "if you set up your agents to have specialized capabilities..." . What does this mean? A technology/platform differentiator? How can you setup your agents to have specialized capabilities?
How can 'tagging' build agents be used effectively in an environment where builds are (typically) performed on each check-in.
You use multiple build agents to support multiple build machines (I work currently with a build farm with 3 build machines - and thus 3 build agents - to distribute the load).
You also might want to have multiple build agents to be able to run builds in parallel. This is a nice feature to share resources, but a requirement when you start working with Test/Lab Management features.
With the capabilities: for example you can setup a build agent with version 1 of a 3rd party component, and a second build agent with version 2. With tagging you can specify in the build definition which build agent it will choose from out of the pool of build agents.
We use 2 build agents on the same machine at work, since we only have one build machine.
The first one handles our CI builds, and is tag with CI. The build definition for the CI build is set up to only use agents that have the CI tag.
The second one is for manually queued builds, mostly for the release branch builds.
I specialized the CI build agent, because it was not uncommon when we were preparing a new build for QA to have several developers check into the development branch, which would slowing down being able to release the build to QA.
One of our builds takes 9 minutes. Its nice that you don't have to be in the Queue behind it if you happen to deploy at the wrong time.