Does anyone know how to enforce that all tests pass before a TFS check in?
TFS provides so-called Gated check-ins. That means: the complete build (which includes the test runs) is run and has to pass, otherwise the check-in is rejected. It's quite performance-demanding and somewhat time-consuming, when working in a larger team, but otherwise works like a charm.
HTH.
You can use the default "Testing checkin policy".
If you want to create your own testing policy, you can use this.
Related
My managers want we to determine which tests might have to be run, based on coding changes that were made to the application we are testing.
But, it is hard to know which tests are actually needed to be re-verified as a result of a code change. What we have done is common to test the entire area where the code change occurred / or the entire proj, solution.
We were told this could be achieved by TFS build or MTM tools. Could someone share the details?
PM:We are running on TFS 2015 update4,VS2017.
There is a concept of Test Impact Analysis which helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
The Test Impact Analysis (TIA) feature specifically enables this – TIA
is all about incremental validation by automatic test selection. For a
given code commit entering the pipeline TIA will select and run only
the relevant tests required to validate that commit. Thus, that test
run is going to complete faster, if there is a failure you will get to
know about it faster, and because it is all scoped by relevance,
analysis will be faster as well.
Test Impact Analysis for managed automated tests is available via a checkbox in the 2.* preview version of the VSTest task.
If enabled, only the relevant set of managed automated tests that need to be run to validate a given code change will run. Test Impact Analysis requires the latest version of Visual Studio, and is presently supported in CI for managed automated tests.
However this is only available with TFS2017 update1(need 2.* preview version of VSTS task). More details please refer this blog: Accelerated Continuous Testing with Test Impact Analysis
In TFS there is an option in Visual Studio to "Keep Items checked out when checking in" what is the purpose of giving such an option?
I am trying to build a reporting tool to find out the list of files that are checked out, so in the case the user has set the above option would my report be accurate, since the developer can always claim that "Hey, all my changes are checked in"
How do i reflect those kind of checkout in my report?
I use this feature when part of what I'm working on is needed by another developer, but I haven't actually finished the whole task.
Your report would still be accurate, as it's a true reflection of the system - the code really is checked out, and the developer is probably still working on the file. The only way you can truly know if all changes are checked in is by comparing the current version from source control with each developer's locally checked out copy, which isn't going to be feasible for reporting purposes, and is probably of limited value.
In a modern CI environment it's very common to commit changes and simply keep changing the very same modules.On the other hand, once a milestone is reached, the developer will simply commit the changes & start working on something else.So, I think it's very natural for TFS to provide with this configurable flag.Another major feature in the TFS ecosystem is gated-checkin: is this mode the commit is shelved, built & committed once all that has succeeded. If it weren't for this option, the developer would have to stand still & wait until the process has finished.I would disagree with a developer stating "Hey, all my changes are checked in": our principle in the team is that everything that is checked out is being developed, anything else is/should be either shelved or committed.You could consider a rule that all pending changes of each developer get shelved at the end of the working day. I am personally against any such measures, but would surely examine them as a option if my devs wouldn't adapt to the principle.If you would agree to the above, your second question would become rather obsolete: In my opinion there aren't several 'kinds' of checkouts.
Our build process on Team Foundation Server 2010 does several check-ins. I don't want these check-ins to generate automatic e-mails et cetera.
I know there is a magical incantation to make this happen (if I recall correctly, a special naming convention for the check-in), but I cannot find it in the documentation to save my life.
So please, save my life.
Try using ***NO_CI*** as a checkin comment, that will prevent another build being triggered. Not sure if it does what you need though.
If I create a bug in TFS, is it possible to associate it with a unit test and be notified if this bug is fixed/broken?
Not exactly, but the workflow can go the other way: instead of creating a bug up-front, you can let Team Foundation Server handle the work item creation for you. If you create a continuous integration build that runs your unit tests, you can create a bug assigned to the user who submitted the changeset if that build fails (including because the unit tests fail.)
A good article on configuring continuous integration to do this is in MSDN Magazine.
A new feature in TFS 2010 is Gated Check-in, which will prevent this failing code from getting checked-in by requiring the build and test steps to complete successfully before they get committed.
There is, currently, no mechanism to do what you desire. You can force users to check in changes tagged with a work item, which you can then manually examine. Provided the user either alters a test class or creates a new one, and checks in with the fix, you can see whether it is done or not. But, there is no automagic "if this test is changed, alert me".
If you need some type of "change and alert" functionality, I would examine CodePlex and see if there are any tools of this type. It could be facilitated at the database layer, as well, but I am not sure what kind of monkeying that would take. I have not looked at the service endpoints for the latest TFS, so there may be a way to poll to see if there are changes.
Wish I had a better answer, but this is an interesting ASK, so I would consider going to Microsoft Connect for TFS and requesting the feature. It could be included in a future rev if enough people are interested in this feature.
Some background: I am not a tfs guy and I dont know much about build scripts etc.
1 - Is there a way to run tests for every check-in TFS? What I'm dreaming is, if any of the tests is failing then build server rejects the changeset. Is it possible with TFS or should it be some other tool like Hudson, Cruise Control etc? What are the other powerful tools?
2 - Does using such a tool make it possible to run only portion of tests, not all of them (ie only unit tests, not integration tests)?
I am not interested in technical details like how it can be done technically, as it is our tfs team's job. Rather I am after some high level info about the possibilities?
In TFS you have what's called check-in policies. With those in place you could forbid checking-in something without all of the unit tests passing. You could even enforce FxCop rules, etc... but that would be cruel to your developers.
If you already have the Continuous Integration build set up, then change the trigger to Gated Check In, this will do exactly what you want. When a developer tries to commit, TFS will start the build, if the build fails then the check in will be aborted and instead TFS will create a shelveset of the changes.
As for running a portion of the tests, you would probably need to create a test list in your VSMDI that defines the tests you want to run and then configure the build to use that list.