Some background: I am not a tfs guy and I dont know much about build scripts etc.
1 - Is there a way to run tests for every check-in TFS? What I'm dreaming is, if any of the tests is failing then build server rejects the changeset. Is it possible with TFS or should it be some other tool like Hudson, Cruise Control etc? What are the other powerful tools?
2 - Does using such a tool make it possible to run only portion of tests, not all of them (ie only unit tests, not integration tests)?
I am not interested in technical details like how it can be done technically, as it is our tfs team's job. Rather I am after some high level info about the possibilities?
In TFS you have what's called check-in policies. With those in place you could forbid checking-in something without all of the unit tests passing. You could even enforce FxCop rules, etc... but that would be cruel to your developers.
If you already have the Continuous Integration build set up, then change the trigger to Gated Check In, this will do exactly what you want. When a developer tries to commit, TFS will start the build, if the build fails then the check in will be aborted and instead TFS will create a shelveset of the changes.
As for running a portion of the tests, you would probably need to create a test list in your VSMDI that defines the tests you want to run and then configure the build to use that list.
Related
I have 2 executables, an app and an automation test app who will perform actions on the app. My automation tests are basically NUNIT tests who calls Chrome Web Driver.
Everything is hosted at TFS. In my build definition, i run sanity checks for every PR. I wanted to expand that. The automation tests are divided into many different categories (being Sanity a category). I've been seeing some stuff related to TFS Test Suits, and my idea was to make in a way whenever when someone makes a PR, he could choose some test category to run on that PR using that build. So in a easyer way of sayng, if my PR changes how 'blue buttons work' ill run the 'blue buttons test suit' on my pr.
Would using Test Suits be the best solution for this ? Has any1 done this or have any nice information on how to achieve this ?
Thanks for any responses !! Best regards !
You could be able to use the Visual Studio Test Agent Deploy and Run Functional tests steps in your build definition to run auto tests on build agents.
Associated test methods with test cases in Visual Studio.
Create a build definition to build your project and add the 2 steps I mentioned above. In the Run Functional Test step, select the test suites which contains those test cases in step1.
More details please refer this blog: Executing Automated tests in Build vNext using Test Plan, Test Suites
My managers want we to determine which tests might have to be run, based on coding changes that were made to the application we are testing.
But, it is hard to know which tests are actually needed to be re-verified as a result of a code change. What we have done is common to test the entire area where the code change occurred / or the entire proj, solution.
We were told this could be achieved by TFS build or MTM tools. Could someone share the details?
PM:We are running on TFS 2015 update4,VS2017.
There is a concept of Test Impact Analysis which helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
The Test Impact Analysis (TIA) feature specifically enables this – TIA
is all about incremental validation by automatic test selection. For a
given code commit entering the pipeline TIA will select and run only
the relevant tests required to validate that commit. Thus, that test
run is going to complete faster, if there is a failure you will get to
know about it faster, and because it is all scoped by relevance,
analysis will be faster as well.
Test Impact Analysis for managed automated tests is available via a checkbox in the 2.* preview version of the VSTest task.
If enabled, only the relevant set of managed automated tests that need to be run to validate a given code change will run. Test Impact Analysis requires the latest version of Visual Studio, and is presently supported in CI for managed automated tests.
However this is only available with TFS2017 update1(need 2.* preview version of VSTS task). More details please refer this blog: Accelerated Continuous Testing with Test Impact Analysis
In TeamCity, you can create build dependencies where one build will not start until another finishes successfully. Is that possible with TFS 2012? Where can I find more information about how to set that up?
The short answer is that TFS doesn't have equivalent functionality, but you can achieve the same goals with a little work.
A common scenario I encounter is a team wants to do a build when they check-in that does some quick stuff (compile, fast unit tests), then immediately after wants to do another build that runs some slower stuff (integration tests, test deployments, etc). I do this often with my teams, and I'll setup a Gated Build that runs in say 5 mins, then have a CI build that is kicked off as soon as the Gated Build checks-in, which may take an hour to run. I like this approach as it gets the developers some feedback quickly, then more detailed feedback shortly thereafter.
Another supported scenario is having a build explicitly kickoff it's dependencies. If you look a the Lab Build Template it does exactly this, it will first kickoff the application TFS Build, and the Lab Build will sit and wait for it to finish, then the Lab Build will continue. In theory you could have Build A kickoff build B which kicks off C & D, etc.
If your needs are more complex than that (e.g. you have multiple applications that you have a build for each, then a Product that includes some applications that needs to be built after each application changes, then maybe a Product Suite build that needs to kickoff whenever a Product changes - this is the scenario I dealt with). I basically implemented a custom build dependency system to handle this. We made an XML file that described the build dependencies, then wrote a simple ISubscriber plug-in that we would deploy to TFS, and it would listen for Build Completed events then consult the dependency config and kickoff the appropriate build(s).
I am working with Visual Studio 2012 .NET 4.5 ASP.NET MVC 4 project that uses TFS for source control and TFS Build for continuous integration (CI).
I want to create functionality that on each check in the build number gets updated prior to the CI build is kicked off.
From research it seems that a custom activity can be created and integrated in TFS 2010 build template.
I have also seen examples of this can be achieved with MSBuild task.
I haven't done work in this area before, so I am wondering which is the better approach or the recommended approach based upon the options open to me? In general when would I use MSBuild tasks as oppose to custom activity? For example, I will be looking to run FxCop and StyleCop against check ins also in the future, so I would like a common approach to this.
In the case of incrementing the build number, I'd vote for the TFS Build Activity so that the implementation is not tied to your msbuild implementation. This allows you to easily apply the TFS workflow activity to any number of branches without tying it to the branches directly. In addition, it keeps your MSBuild project files clean of the task so that it isn't mistakenly executed on developer machines.
Holistically, I'd say that you need to take a variety of factors into account when deciding between MSBuild and Workflow activities:
1 - Does MSBuild support the functionality out of the box (like Code Analysis/FxCop)?
2 - Does the build step need to run on developer boxes as well as servers (StyleCop/FxCop)?
3 - Does the build step need to interact with the TFS API or source control directly (checking out/in a version file for incrementing)?
4 - Are you going to change build job schedulers later to something free (for example, Jenkins)?
It's the combination of these things that determines the implementation of any given tool integration in my book. I'd implement FxCop, StyleCop and any other tool that should be run on a developer box build via MSBuild. I'd implement build steps such as version incrementing, bin-placing and CI deployment invocation (for example, deployment of a SharePoint webpart as a post-build step) via a Code Activity or some scriptware.
Currently we are using StarTeam to perform the build as well as versioning. we planned to migrate startteam to TFS2010. We have some script for perform the build. i wanted to change this script according to my requirement. i gone through the TFS but i had lot more confusion.
in StarTeam, we will get a files from "Ready to Build" label and perform the build. In TFS, how we are going to get a files from TFS? What concept should i use to get a files and perform the build? i have gone through the lot of commands like get, check-in, checkout etc..
If we use "tf get" command, we can get all the files from TFS but i have a clarification on that. shall i get all the files from TFS for every build? i hope, this is unnecessary headache.. correct me if i am wrong..
how we perform the build in TFS? i have read some types of build such as manual, gatedcheckin, Continuous Integration and schedule.
Is there any relationship between branch and build activities?
In TFS, What is the meaning of Workspace?
As said, many questions in one. Hope this helps along the way:
A workspace is a mapping between the server and a local storage,
similar to checkout in Subversion, view in ClearCase, etc.
"TF get" normally only fetches those files that have changed since
last update. You can force it to fetch everything - and sometimes
have to - but its not normally done.
Team Build is the recommended system to build with when using TFS. It can take some time to get into (Windows Workflow-based), but is quite powerful. There are default process definitions that set up the most common actions for you.
By default, you can't control whether to build by setting a certain label, but you can define that only this label should be used when builds are triggered. Labels in TFS work a little differently compared to other VCS, though, so maybe there's an 'opportunity' to re-think your build process along the way. If you're set on using a label as before, you'll need to build a Custom build activity.