When I have defined a Gated Build, when somebody checks in code the Integration Build field of a work item changes to the Gated Build number (if the developer associates his check in with work items, of course). Once a CI build is triggered this field changes to the CI build number.
My question is: Is there any way of not changing the Integration Build field of a work item once a Gated Build is triggered?
EDIT
Let me be more clear about how we work.
We have several work itens (some are user stories and some are bugs). When a developer checks in code he or she associates his/her check in with those user stories that gets the Resolved state and a "Gated x.x.x.x" in the integration build field. We never test gated builds. Instead, every night we manually trigger a build and those work itens gets updated again, but this time with a "Release x.x.x.x" in the integration build field. In the next day we test those work itens but the process continues and developers keep check in more US or Bugs (that will have the Gated ...).
Sometime we get confused and we test work itens that should not be tested because they are in the "Gated state".
Even if we have branches that will not solve our problem because the developer associates a check in with work itens and we cant change that.
We do not test gated builds because our QA team is small. The dev team have 20 developers and the QA team have only 2. The process of deploy the application takes about 10 minutes and it can be a pain to wait 10 minutes on every developer check in. Also changing the code while we are testing is never a good idea because it can mess up with our test.
Somebody can think that our process is wrong and suggest a new approach. This will be very welcome, but what we do is working very well besides that small issue.
Related
I've always worked with the Continuous Integration (CI) build in TFS. However, in my last project we started to use the gated check-in trigger.
Are there any disadvantages when using a gated check-in? Because if it prevents the team from checking in broken code, what's the purpose of a CI trigger?
Gated checkin is a form of continuous integration build. In TFS, it creates a shelveset containing the code that's being validated, then runs a build of that code. Only if that code builds successfully and all configured unit tests pass does the code actually get committed.
Continuous integration is different -- in CI, the code is committed regardless of what happens as a result of the build. If a CI build fails due to bad code being committed, the code is still there, in source control, available for everyone to grab.
Now for the opinion-based part:
Gated checkin is great if you have a large number of developers of varying levels of skill/experience, since it prevents broken code from going into source control. The downside is that it increases the time between code being committed and code being available for others, and thus can lead to situations where people are sitting around twiddling their thumbs waiting for the build to complete successfully.
I recommend using gated check-in as a stopgap. If you have a ton of gated check-in build failing, then it's doing its job and preventing bad code from getting committed. If, over time, the team matures and gated check-in builds fail infrequently, then it's serving less purpose and should be switched over to continuous integration and correcting failing builds as they come, instead of delaying every commit in the off chance there's a problem.
Gated check-ins are fundamentally flawed because they validate dirty local state, not versioned state, so they cannot replace independent checks based on a clean slate (such as in a CI pipeline). Similarly QA often needs to be done with a matrix of environments (different compiler version, different browsers, different OS, ...), the cost to set up are better invested in a central CI.
Gated checkins also make committing harder and slower. That is commonly a bad thing, because:
In TDD, members should be able to push commits with failing tests
Reporting bugs as failing tests is super useful
When cooperating on a WIP (work in progress) branch, members should be able to push dirty changes to make them available quickly to others
When working on a big change, it can be useful to let other members review unfinished work before investing the time to finish
Many people like regularly committing incomplete work as a form of snapshot/backup
committing incomplete work is great when frequently switching between branches (stashing only of limited help in particular for new files)
QA cannot be time-limited, but committing should not take long
So scenarios where gated checked are ok as workaround or quick and dirty mitigation:
Your VCS makes branching hard, or your company does not allow branching
The project is tiny
Only one developer
No CI present
Only specific long-lived branches are protected by the gates (but that's not an alternative to CI)
I'm having some trouble preventing a Scheduled Build from being automatically kicked off after the same build has been trigger manually and no changes are present.
Problem: The Build is scheduled to go out at 3am everyday but only if any changes have been made (the Settings option ‘Build even if nothing has changed since the previous build’ is Unchecked). The problem is that if you trigger a manual build and no changes are made afterwards, the Scheduled Build is still going to get triggered even if there are no changes. Please follow the scenario bellow for an example:
Scenario:
Scheduled build gets kicked off today at 3am - assembly version changes to 1.0.0.1
On the same day, several changes are checked-in before noon.
The same Build is kicked off manually at 1pm - assembly version changes to 1.0.0.2
No more check-ins are made after the manual build from 1pm
The next day at 3am the Scheduled build gets triggered which it shouldn't have (assembly version changes to 1.0.0.3) since there were no check-ins since 1pm (triggered manually).
It seems like the 1pm build is not even being considered and accounted for.
It looks like TFS is taking in count check-ins since the last “Scheduled Build” rather than “the last completed Build” (same build definition) which in this case was the manually triggered one.
So my question is: “Is there any way we can prevent TFS from triggering Scheduled build if there were no check-ins after a manual build has been kicked off?”
Currently, it looks like TFS does not have that option anywhere in the Build Definition
Thank you in advance
In order to do this you would need to write your own scheduler that checked the previous manual build for you. You could write a TfsJob for this or even as a a scheduled powershell that runs at the allotted time...
I ran into this issue today as well.
It seems this is currently "by design".
Someone else posted a suggestion to "fix" this here:
http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/5702884--build-even-if-nothing-has-changed-since-the-previ
We just started using Visual Studio Release Management for one of our projects, and we're already having some problems with how we are doing things.
For now, we've created a single release stage, which is responsible for deploying our build artifacts to a dedicated virtual machine for testing. We intend to use this machine to run our integration tests later on.
Right now, we have a gated checkin build process: each checkin fires all the unit tests and we configured the release trigger to happen on this build also. At first, it seemed plausible that, after each checkin, the project was deployed and the integration tests were executed. We noticed that all released builds were polluting the console on Release Management, and that all builds were being marked as "Retain Indefinitely" and our drop folder location was growing fast (after seeing that, it makes sense that the tool automatically does this, since one could promote any build to another stage and the artifacts need to be persisted).
The question then is: what are we doing wrong? I've been thinking about this and it really does not make any sense to "release" every checkin. We should probably be starting this release process when a sprint ends, a point that can be considered a "release candidate".
If we do that though, how and when would we run our automated integration tests? I mean, a deployment process is required for running those in our case, and if we try to use other means to achieve that (like the LabTemplate build process) we will end up duplicating deployment code.
What is the best approach here?
It's tough to say without being inside your organization and looking at how you do things, but I'll take a stab.
First, I generally avoid gated checkin builds unless there's a frequent problem with broken builds. If broken builds aren't a pain point, don't use gated checkin. Why? Simple: If your build/test process takes 10 minutes to run, that's 10 minutes that I have to wait to know whether I can keep working, or if I'm going to get my changes kicked back out at me. It discourages small, frequent checkins and encourages giant, contextless checkins.
It's also 10 minutes that Developer B has to wait to grab Developer A's latest changes. If Developer B needs that checkin to keep working, that's wasted time. Trust your CI process to catch a broken build and your developers to take responsibility and fix them on the rare occasions when they occur.
It's more appropriate (depending on your branching strategy) to do a gated checkin against your trunk, and then CI builds against your dev/feature branches. Of course, that opens up the whole "how do I build once/deploy many when I have multiple branches?" can of worms. :)
If your integration tests are slow and require a deployment to succeed, they're probably not good candidates to run as part of CI. Have a CI/gated checkin build that just:
Builds
Runs fast unit tests
Runs high-priority, non-deployment-based integration tests
Then, have a second build (either scheduled, or rolling) that actually deploys and runs the whole test suite. You can schedule it according to your tastes -- I usually go with one at noon (or whatever passes for "lunch break" among the team), and one at midnight. That way you get a tested build from the morning's work, and one from the afternoon's work.
Using the Release Default Template, you can target your scheduled builds to just go as far as your "dev" (/test/integration/whatever you call it) stage. When you're ready to actually release a build, you can kick off a new release using that specific build that targets Production and let it go through all your stages normally.
Don't get tripped up on the 'Release' word. In MS Release Management (RM), creating a Release does not necessarily mean you will have this code delivered to your customers / not even that it has the quality to move out of dev. It only means you are putting a version of the code on your Release Path. This version/release can stop right in the first stage and that is ok.
Let's say you have a Release Path consisting of Dev, QA, Prod. In the course of a month, you may end up releasing 100 times in Dev, but only 5 times in QA and once in Prod.
You should drive to get each check-in deployed and integration tested. If tests takes a long time, only do the minimal during (gated or not) check-in (for example, unit tests + deployment), and the rest in your second stage of Release Path (which should be automatically triggered after first stage completes). It does not matter if second stage takes a long time. As a dev, check-in, once build completes successfully (and first stage), expect the rest to go smoothly and continue on your next task. (Note that only result of the first stage impacts your TFS build).
Most of the time, deployment and rest will run fine and so there won't be any impact to dev. Every now and then, you will have a failure in first stage, now the dev will interrupt his new work and get a resolution asap.
As for the issue that every build is kept indefinitely, for the time being, that is a side effect of RM. Current customers need to do the clean up manually (or script it). In the coming releases, a new retention policy for releases/builds will be put in place to improve this. This has not been worked on yet, but the intention would be to, for example, instruct RM to keep all releases that went to Prod, keep only the last 5 that went to QA and keep only the last 2 that went to Dev.
This is not a simple question, so also the answer must be articulated.
First of all, you will never keep all of your builds; the older a build, the less interesting to anyone; a build that doesn't get deployed in production is overtaken by builds that reaches that stage.
A team must agree on the criteria that makes a build interesting to keep around and how long to keep it. Define a policy for builds shipped to production or customers: how long do you support them? Until the next release, until the following one, for five years? Potentially shippable builds, still not in your customers' hands, are superseded by newer, so you can use a numeric or a temporal criteria (TFS implements only the first, as the second is more error-prone). Often you have more than one shippable build, when you want a safety net option and being able select from a pool which deliver (the one with more manageable bugs).
The TFS "Retain Indefinitely" should be used when you cannot automate the previous criteria, so you switch to a manually implemented policy. Indefinitely is not forever, means for an unknown time interval.
We set up TFS 2013 recently and tried to set up gated check-in. In our experiment, it correctly failed the build and rejected the bad check-in. However, both the build notification tray icon and the build tab on the TFS web access show the failure, and it is this way for all users. This will make everyone think "the" build is broken when it's just one person's "gate." It will skew metrics, too.
A) Why would this be the default behavior? It seems very counter-productive and counter-intuitive. [Or maybe this isn't the default behavior and our setup is hosed?]
B) Is there a configuration for the build tab where a rejected gated check-in/build is visible only to the person who broke it?
C) How can we make the build notifier tool ignore gated failures?
a) This is a public build. Why should it be failing. If the gated build is failing it shows a lack of care by the developers. Are they checking in any old thing? Are they running their unit tests first? It really sounds like you have a quality issue there.
That is why the gated-builds are still public builds.
b) no - Anyone can however send a Private build to the build server if they think that there may be an issue that they can't catch locally..
c) Each user can uncheck the monitoring of the gated build. I would however suggest that they should concentrate on not failing the build however rather than sweeping it under the carpet...
A filed build, even a gated one, should be the exception and not the rule...
MrHinsh answered my question from a largely philosophical standpoint, and that led me to this article:
[http://adamstephensen.com/2012/11/01/gated-checkins-mask-dysfunction/]
I agree in principle with the article. However, because we are attacking agile processes in baby steps, we still want this extra safety check for the time being.
Below isn't the exact solution we wanted, but it is a hybrid of the approach in the article and my original question. It doesn't force gated check-ins, but it allows them to be done, giving risky check-ins a safety net. Code quality beyond build-ability will be addressed with other mechanisms outside the scope of this question.
The compromise solution is that "vanilla" changes would be checked in normally (after local testing, of course), and then the CI build would kick in. Fixing broken builds would be high priority to be addressed immediately.
But anything out of the ordinary (configuration changes, referencing new/different external dependencies, or just anything a developer is unsure of) would use a private gated check-in, but not on the main build, because we don't want everyone to be tricked into thinking the build is broken upon rejections. This build definition is a clone of the main build definition, but with the addition that upon successful check-in, it chains a build of the main build definition using this:
[How to chain TFS builds?
While the main build is redundant in most cases (race conditions may apply), it solves the immediate need.
I checked in some code and my build was added to the build queue. I can't go home until the build passes.
There is nobody in front of me, but it's been well over half an hour since my checkin and my build hasn't started (plus it's a half an hour to build the 49 large projects against the build environment). I've been at work since morning and just want to go home before midnight (it's past 8:30 already). My checkin's just sitting their in queue, not being fired off.
How do I force the build to start in an empty build machine if there is nobody in front of me in queue and nothing else going on?
edit: This is what I get for breaking my own rule of never checking in after 3pm unless it's a mandate.
Open the Team Explorer pane and navigate to your team project. Expand the 'Builds' node and find the build definition that you need to build. You should then be able to right-click it an choose 'Queue new build..'. (This does require a separate permission, and you may not have that permission).
This will then bring up the confirmation dialog where you can specify the shelveset to build (if its a gated build).
If the Builds screen shows that there are queued builds, and none running - it could be that there are no available Build Agents or they are marked offline. See Manage Your Build System for more information about how to check the status of your build agents.