We set up TFS 2013 recently and tried to set up gated check-in. In our experiment, it correctly failed the build and rejected the bad check-in. However, both the build notification tray icon and the build tab on the TFS web access show the failure, and it is this way for all users. This will make everyone think "the" build is broken when it's just one person's "gate." It will skew metrics, too.
A) Why would this be the default behavior? It seems very counter-productive and counter-intuitive. [Or maybe this isn't the default behavior and our setup is hosed?]
B) Is there a configuration for the build tab where a rejected gated check-in/build is visible only to the person who broke it?
C) How can we make the build notifier tool ignore gated failures?
a) This is a public build. Why should it be failing. If the gated build is failing it shows a lack of care by the developers. Are they checking in any old thing? Are they running their unit tests first? It really sounds like you have a quality issue there.
That is why the gated-builds are still public builds.
b) no - Anyone can however send a Private build to the build server if they think that there may be an issue that they can't catch locally..
c) Each user can uncheck the monitoring of the gated build. I would however suggest that they should concentrate on not failing the build however rather than sweeping it under the carpet...
A filed build, even a gated one, should be the exception and not the rule...
MrHinsh answered my question from a largely philosophical standpoint, and that led me to this article:
[http://adamstephensen.com/2012/11/01/gated-checkins-mask-dysfunction/]
I agree in principle with the article. However, because we are attacking agile processes in baby steps, we still want this extra safety check for the time being.
Below isn't the exact solution we wanted, but it is a hybrid of the approach in the article and my original question. It doesn't force gated check-ins, but it allows them to be done, giving risky check-ins a safety net. Code quality beyond build-ability will be addressed with other mechanisms outside the scope of this question.
The compromise solution is that "vanilla" changes would be checked in normally (after local testing, of course), and then the CI build would kick in. Fixing broken builds would be high priority to be addressed immediately.
But anything out of the ordinary (configuration changes, referencing new/different external dependencies, or just anything a developer is unsure of) would use a private gated check-in, but not on the main build, because we don't want everyone to be tricked into thinking the build is broken upon rejections. This build definition is a clone of the main build definition, but with the addition that upon successful check-in, it chains a build of the main build definition using this:
[How to chain TFS builds?
While the main build is redundant in most cases (race conditions may apply), it solves the immediate need.
Related
I've always worked with the Continuous Integration (CI) build in TFS. However, in my last project we started to use the gated check-in trigger.
Are there any disadvantages when using a gated check-in? Because if it prevents the team from checking in broken code, what's the purpose of a CI trigger?
Gated checkin is a form of continuous integration build. In TFS, it creates a shelveset containing the code that's being validated, then runs a build of that code. Only if that code builds successfully and all configured unit tests pass does the code actually get committed.
Continuous integration is different -- in CI, the code is committed regardless of what happens as a result of the build. If a CI build fails due to bad code being committed, the code is still there, in source control, available for everyone to grab.
Now for the opinion-based part:
Gated checkin is great if you have a large number of developers of varying levels of skill/experience, since it prevents broken code from going into source control. The downside is that it increases the time between code being committed and code being available for others, and thus can lead to situations where people are sitting around twiddling their thumbs waiting for the build to complete successfully.
I recommend using gated check-in as a stopgap. If you have a ton of gated check-in build failing, then it's doing its job and preventing bad code from getting committed. If, over time, the team matures and gated check-in builds fail infrequently, then it's serving less purpose and should be switched over to continuous integration and correcting failing builds as they come, instead of delaying every commit in the off chance there's a problem.
Gated check-ins are fundamentally flawed because they validate dirty local state, not versioned state, so they cannot replace independent checks based on a clean slate (such as in a CI pipeline). Similarly QA often needs to be done with a matrix of environments (different compiler version, different browsers, different OS, ...), the cost to set up are better invested in a central CI.
Gated checkins also make committing harder and slower. That is commonly a bad thing, because:
In TDD, members should be able to push commits with failing tests
Reporting bugs as failing tests is super useful
When cooperating on a WIP (work in progress) branch, members should be able to push dirty changes to make them available quickly to others
When working on a big change, it can be useful to let other members review unfinished work before investing the time to finish
Many people like regularly committing incomplete work as a form of snapshot/backup
committing incomplete work is great when frequently switching between branches (stashing only of limited help in particular for new files)
QA cannot be time-limited, but committing should not take long
So scenarios where gated checked are ok as workaround or quick and dirty mitigation:
Your VCS makes branching hard, or your company does not allow branching
The project is tiny
Only one developer
No CI present
Only specific long-lived branches are protected by the gates (but that's not an alternative to CI)
I am new in configuration of TFS.
Currently our project is 50% done but we found that we have very bad code. We consider the need for static code analysis like Resharper or another product like StyleCop, CodeAnalysis and FxCop.
We want configure the TFS to reject a checkin when that check in contains code that triggers code analysis warnings.
But for the previous code we want to suppress the existing warning to prevent the code from becoming worse than it already is.
As Ivan mentions, your root cause it not in the lack of analysis tools, but probably in the level of quality and rigor agreed (or currently being enforeced between team members) between the development team and their project's sponsor. It may be that the pressure on the team is too high, causing important review actions to be skipped, or that the team (or the sponsor!) doesn't have the same desire to quality as you or the sponsor. Or that the team doesn't have the right level of knowledge to prevent these issues from happening.
The best way out of this is to fix as much as you can in a short period of time.
Warning: I've experienced with a number of teams the effect of turning on too many rules all at once. Generally, there is a reluctance for people to concede that their work hasn't been up to par and turning on rules that do not directly cause bug ("The identifier is cased incorrectly" for example) can cause frustration that can severely hamper your momentum. Carefully selecting which rules need to be solved now and which can wait for later worked much better in my experience. Once the team has developed a way to solve these kinds of problem, you can easily apply more.
Turning on Tools like configuring Code Analysis for your solution or using the Solution Wide Analysis feature of Resharper, can help you spot issues, but it won't solve them or prevent similar issues from popping up in the future unless your team stops creating them.
Tip: Note that you can turn on Resharper during your build as well using the Resharper CLI features.
StyleCop I would not enforce on this team (just yet) if the code itself is bad enough to trigger massive warnings that may hold bugs and issues. Fix these problems first, make the code it pretty later. Your priorities are now to remove any possible bugs.
CodeAnalysis and FxCop are the same things, so you won't need to turn on both. A tool like Resharper can help your developers to quickly remove a lot of the issues by using the magic-key ALT+ENTER.
If you want to create a clean baseline you can run code analysis once, then select all warnings that are generated and then select Suppress in global suppression file. This will work for Code Analysis issues, but won't suppress any Compiler Warnings, there is no easy way to quickly suppress all current compiler warnings.
Tip: It sometimes helps to temporarily rename any existing globalsupressions.cs files, so that this "baseline" is stored separately. You then know which warnings you'll have to fix at a later point in time.
Tip: When a developer suppresses a warning, have them add a Justification="Reason for suppression" to the suppression that is generated, that way you can distinguish between carefully considered suppression and temporary ones.
Depending on whether you already have a build server your next step is to install Team Build and once you have a build server you'll need to setup a Build Definition. This blog post covers most of the steps.
In the build definition set the trigger to "Gated Checkin" and on the Process tab make sure you set Code Analysis to "Always". If you want to fail your build based on Code Analysis errors, you need to create a custom ruleset and configure that for your solution.
To have compiler errors fail the build you can also enable the "Treat Warnings as Errors
Once you have enabled your gated check-in build all developers changes will be prompted to wait for their build to finish. You can turn on alerts (using Web access) or use the Build Notification Tool to get notified when the changes were successfully submitted.
Tip: Instead of turning on all rules at once (or switching them all to cause an ERROR during builds) you can also opt to turn on rules a couple at a time and fix them. Turning on rules by category gives you a nice opportunity to teach people the importance of the rules being turned on and possible solutions for fixing them.
A far more advanced solution would be to install and configure SonarQube alongside your Team Build environment. The ALM Rangers and Sonar have recently worked together to create installation guidance and a number of extensions to enable Team Build and SonarQube integration. You can find the installation guide here.
We just started using Visual Studio Release Management for one of our projects, and we're already having some problems with how we are doing things.
For now, we've created a single release stage, which is responsible for deploying our build artifacts to a dedicated virtual machine for testing. We intend to use this machine to run our integration tests later on.
Right now, we have a gated checkin build process: each checkin fires all the unit tests and we configured the release trigger to happen on this build also. At first, it seemed plausible that, after each checkin, the project was deployed and the integration tests were executed. We noticed that all released builds were polluting the console on Release Management, and that all builds were being marked as "Retain Indefinitely" and our drop folder location was growing fast (after seeing that, it makes sense that the tool automatically does this, since one could promote any build to another stage and the artifacts need to be persisted).
The question then is: what are we doing wrong? I've been thinking about this and it really does not make any sense to "release" every checkin. We should probably be starting this release process when a sprint ends, a point that can be considered a "release candidate".
If we do that though, how and when would we run our automated integration tests? I mean, a deployment process is required for running those in our case, and if we try to use other means to achieve that (like the LabTemplate build process) we will end up duplicating deployment code.
What is the best approach here?
It's tough to say without being inside your organization and looking at how you do things, but I'll take a stab.
First, I generally avoid gated checkin builds unless there's a frequent problem with broken builds. If broken builds aren't a pain point, don't use gated checkin. Why? Simple: If your build/test process takes 10 minutes to run, that's 10 minutes that I have to wait to know whether I can keep working, or if I'm going to get my changes kicked back out at me. It discourages small, frequent checkins and encourages giant, contextless checkins.
It's also 10 minutes that Developer B has to wait to grab Developer A's latest changes. If Developer B needs that checkin to keep working, that's wasted time. Trust your CI process to catch a broken build and your developers to take responsibility and fix them on the rare occasions when they occur.
It's more appropriate (depending on your branching strategy) to do a gated checkin against your trunk, and then CI builds against your dev/feature branches. Of course, that opens up the whole "how do I build once/deploy many when I have multiple branches?" can of worms. :)
If your integration tests are slow and require a deployment to succeed, they're probably not good candidates to run as part of CI. Have a CI/gated checkin build that just:
Builds
Runs fast unit tests
Runs high-priority, non-deployment-based integration tests
Then, have a second build (either scheduled, or rolling) that actually deploys and runs the whole test suite. You can schedule it according to your tastes -- I usually go with one at noon (or whatever passes for "lunch break" among the team), and one at midnight. That way you get a tested build from the morning's work, and one from the afternoon's work.
Using the Release Default Template, you can target your scheduled builds to just go as far as your "dev" (/test/integration/whatever you call it) stage. When you're ready to actually release a build, you can kick off a new release using that specific build that targets Production and let it go through all your stages normally.
Don't get tripped up on the 'Release' word. In MS Release Management (RM), creating a Release does not necessarily mean you will have this code delivered to your customers / not even that it has the quality to move out of dev. It only means you are putting a version of the code on your Release Path. This version/release can stop right in the first stage and that is ok.
Let's say you have a Release Path consisting of Dev, QA, Prod. In the course of a month, you may end up releasing 100 times in Dev, but only 5 times in QA and once in Prod.
You should drive to get each check-in deployed and integration tested. If tests takes a long time, only do the minimal during (gated or not) check-in (for example, unit tests + deployment), and the rest in your second stage of Release Path (which should be automatically triggered after first stage completes). It does not matter if second stage takes a long time. As a dev, check-in, once build completes successfully (and first stage), expect the rest to go smoothly and continue on your next task. (Note that only result of the first stage impacts your TFS build).
Most of the time, deployment and rest will run fine and so there won't be any impact to dev. Every now and then, you will have a failure in first stage, now the dev will interrupt his new work and get a resolution asap.
As for the issue that every build is kept indefinitely, for the time being, that is a side effect of RM. Current customers need to do the clean up manually (or script it). In the coming releases, a new retention policy for releases/builds will be put in place to improve this. This has not been worked on yet, but the intention would be to, for example, instruct RM to keep all releases that went to Prod, keep only the last 5 that went to QA and keep only the last 2 that went to Dev.
This is not a simple question, so also the answer must be articulated.
First of all, you will never keep all of your builds; the older a build, the less interesting to anyone; a build that doesn't get deployed in production is overtaken by builds that reaches that stage.
A team must agree on the criteria that makes a build interesting to keep around and how long to keep it. Define a policy for builds shipped to production or customers: how long do you support them? Until the next release, until the following one, for five years? Potentially shippable builds, still not in your customers' hands, are superseded by newer, so you can use a numeric or a temporal criteria (TFS implements only the first, as the second is more error-prone). Often you have more than one shippable build, when you want a safety net option and being able select from a pool which deliver (the one with more manageable bugs).
The TFS "Retain Indefinitely" should be used when you cannot automate the previous criteria, so you switch to a manually implemented policy. Indefinitely is not forever, means for an unknown time interval.
Jenkins says a build succeeded or failed, but can it identify the exact commit (and author!) that caused a build to fail?
This issue would seem to indicate no.
Edit: From my exchange with Pace:
What I see is "include culprits", which is everyone since the last
build. I don't want that. I want THE culprit, with Jenkins doing the
binary search. If Jenkins does two builds 10 commits apart, I don't
want 10 possible culprits, I want it to find the one.
I haven't yet heard how to do that.
That page was talking about the "find bugs" plugin, not the normal build cycle. Depending on how things are setup Jenkins can identify the exact commit and author that caused a failure. If Jenkins has the appropriate source control plugins installed and is configured to know about the repository the build is tied to then for every build it will list the changes since the last build.
In addition, Jenkins has the capability in many of its reporting plugins to blame the faulty committer. It can, for example, send an e-mail notification on a failed build to the developer that made the faulty commit.
However, many setups make it difficult for Jenkins to know. For example, if Jenkins is configured for daily builds then there are likely many commits which could have caused the issue. It's also possible that Jenkins isn't configured to know about the source control repository, or there is no source control repository. All of these issues could cause Jenkins to be unable to identify the build breaker.
Specifically for e-mailing faulty committers you can use the email-ext plugin which has options to send e-mails to everyone that committed since the last successful build.
For a humorous take on this subject check out this approach.
I think what you're asking for is impossible in some cases. Determining who the culprit is requires insight into conflict resolution that only a human can decide. Even still, sometimes a manager has to be involved in order to arbitrate. Say for instance you get 3 commits (A,B,C) that depend on a preexisting definition. However, another commit (D) modifies the behavior of that function. Which do you revert? Perhaps it's the business plan to keep A,B,C as is and return D to its original state. The opposite, modifying A,B,C to adapt to the changes of D, is also possible.
In the cases where a machine can handle the arbitration, it is the responsibility of unit tests, and static analyzers, to determine the culprit (although still imperfect). Static analyzers sometimes have built in features that email the person who committed a violation. Unit tests can be written that notify teams or team members responsible for a failed test. Both could work in the same way that identifies who was the last committer on a particular line that failed. Still, if it is a problem with linking, then perhaps some members should be associated with the particular makefile.
I recently migrated from VSS to TFS 2010 and I've been absolutely loving it, but there's something I haven't yet been able to get working the way I think it should.
GOALS
I'd like to quickly know when a change to Development breaks a build. If we find out after-the-fact, it's no big deal. Since a lot of check-ins happen throughout the day, we don't want to wait on the build to finish, so it should be asynchronous.
With our Main branch, I'd like to ensure that any time a merge happens into it, we make sure it's not going to break the build. I want immediate feedback on this. The wait time is fine, since we won't be merging into Main often.
CURRENT SETUP
My solution is under a folder called Main. I've made a branch off of that called Development. The workspace I'm working in is tied to the top-level, which includes both Main and Development branches. I tried adjusting my workspace to only point to Development, in case that was the problem. That didn't seem to fix my issue, so I set it back to the way I had it originally -- with both Main and Development.
Within the workspace's Build definitions, I have two definitions defined -- one for the Main branch and another for Development.
The first definition is for building the Main branch. It has a Gated Check-In trigger and "Items to Build" points to the solution file within Main.
The second definition is for building the solution in the Development branch. It has a Rolling Builds trigger and "Items to Build" points to the solution file within Development.
RESULT
Currently, when a change is made in Development and a check-in is performed, the gated check-in to Main is triggered. This not only causes confusion, but it also tends to slow down our overall process.
NEXT STEP
I've looked at some of the TFS guides to branching, general TFS usage, etc. Unfortunately, I haven't yet found a solution to my issue. If you've run across this issue before, I'd appreciate any suggestions you can give.
I'm not 100% sure at the moment, but the build should be triggered by check-ins into the part of the source tree that is covered by the build definition workspace.
So please have a look into the build definition and see what part of the source tree each build definition gets. Your CI build should only get the Dev branch, the gated build should only get the Main branch.