I am using TFS and have a Test Plan. All of the tests in the plan are automated. When creating a test plan, a Release must be created with a Visual Studio Test task included. That has worked until now, but there are a few things that we wanted to do along with running the tests. Specifically, we are replacing some tokens in a .runsettings file and then deleting the file after the tests have run.
I wanted to encapsulate all of that functionality in a Task Group since it seems like a common task that we will want to use often. Rather than having to add each task and configure them, a user could add the Task Group, set some parameters, and it just works.
When I try to execute a Test Run, I get this error
Validating environment:The Visual Studio Test task in the environment you selected is not the correct version, or has incorrect settings.
I am assuming that the issue is that I dont have a Visual Studio Test task included in my Release definition, but that is because the Task is included in my Task Group. Is there any way for this to work? I want to run automated tests while the Visual Studio Test task is part of a Task Group.
It looks like someone has requested this functionality on UserVoice, but I was hoping that there was a workaround.
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/33921664--test-run-step-not-detected-in-test-hub-when-in-t
Thanks
Current task groups for planned automation testing is not supported.
As you could see the user voice in our main forum for product suggestions:
"Test Run" step not detected in Test hub when in Task group
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/33921664--test-run-step-not-detected-in-test-hub-when-in-t
You could vote up and monitor it. TFS PM and Admin will kindly review your suggestion. Thank you for helping us build a better product.
Unfortunately, there is not any workaround for now. Suggest you exclude the test related in your task group for now.
Related
I have 2 executables, an app and an automation test app who will perform actions on the app. My automation tests are basically NUNIT tests who calls Chrome Web Driver.
Everything is hosted at TFS. In my build definition, i run sanity checks for every PR. I wanted to expand that. The automation tests are divided into many different categories (being Sanity a category). I've been seeing some stuff related to TFS Test Suits, and my idea was to make in a way whenever when someone makes a PR, he could choose some test category to run on that PR using that build. So in a easyer way of sayng, if my PR changes how 'blue buttons work' ill run the 'blue buttons test suit' on my pr.
Would using Test Suits be the best solution for this ? Has any1 done this or have any nice information on how to achieve this ?
Thanks for any responses !! Best regards !
You could be able to use the Visual Studio Test Agent Deploy and Run Functional tests steps in your build definition to run auto tests on build agents.
Associated test methods with test cases in Visual Studio.
Create a build definition to build your project and add the 2 steps I mentioned above. In the Run Functional Test step, select the test suites which contains those test cases in step1.
More details please refer this blog: Executing Automated tests in Build vNext using Test Plan, Test Suites
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace
I am creating a build definition in visual studio 2013. As you know, while creating build definition we can also set the assembly (in "Automated Tests" section) which contains the unit test methods to run. I am able to create the definition till this point. Now, what I want is once the test run is over, I want to call an exe that will parse the Trx file generated as a result of test run and send the run results in an email.
Can anyone have an idea about where I can set the path to the exe and pass some arguments to it while creating build definition? If this cannot be done while creating definition, what else is the other way to achieve it?
Thanks!
What you ask involves multiple steps, especially if you want to pull the tool from version control (which is IMHO the best option).
See the ALM Rangers' Team Foundation Build Customization Guide for help and guidance on this topic.
Or you can just setup an email notification when the TFS Build completes. That should include the test run results, and a clickable link to go to the full build report.
I'm trying to setup TFS 2010 Lab Management based Automated Testing system and I have a "Test suite" (Created in TFS Test Manager) with following tests:
1) Start and Login
2) Create group
3) Create user and add to the group
These have order specified properly in Testing Center->Plan->(Test suite)->Order column
However, when they are executed (on TFS Lab Management Build) by Test Agent they run out of order: 2nd then 1st and then 3rd.
Is there a way to make Test Agent run these tests in order?
Thanks.
Create an Ordered Test, it's a simple list of which test methods should run and in what order.
I also had the same issue. I solved it by installing VS2010 Service Pack and TFS 2010 Service Pack in both the Test controller machine and Test agent machines. Please go through the following link for more details
http://social.msdn.microsoft.com/Forums/en-US/vsmantest/thread/29ac3c7b-b0cd-40f3-9992-f5f3c2285331
Unfortunately, There's a blog regarding ordering test cases which says:
Please note this only applies to the manual tests but not automated
tests. For automated tests, the order you set here will not be
respected during test execution.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/11/21/run-your-tests-in-the-order-you-want.aspx
I don't know if it's a new feature but you can create Ordered tests. Just right click on your project in Solution Explorer and choose Add -> Ordered Test. Select you tests in order.
In Test Explorer, run you ordered test.
Details are in the link below but all I had to do is what I'd said above. I am using VS 2015 Enterprise.
https://msdn.microsoft.com/en-us/library/ms182631.aspx
Some background: I am not a tfs guy and I dont know much about build scripts etc.
1 - Is there a way to run tests for every check-in TFS? What I'm dreaming is, if any of the tests is failing then build server rejects the changeset. Is it possible with TFS or should it be some other tool like Hudson, Cruise Control etc? What are the other powerful tools?
2 - Does using such a tool make it possible to run only portion of tests, not all of them (ie only unit tests, not integration tests)?
I am not interested in technical details like how it can be done technically, as it is our tfs team's job. Rather I am after some high level info about the possibilities?
In TFS you have what's called check-in policies. With those in place you could forbid checking-in something without all of the unit tests passing. You could even enforce FxCop rules, etc... but that would be cruel to your developers.
If you already have the Continuous Integration build set up, then change the trigger to Gated Check In, this will do exactly what you want. When a developer tries to commit, TFS will start the build, if the build fails then the check in will be aborted and instead TFS will create a shelveset of the changes.
As for running a portion of the tests, you would probably need to create a test list in your VSMDI that defines the tests you want to run and then configure the build to use that list.