I have a TFS build definition setup to run the task "Visual Studio Test" using the Nunit3 Test Adapter. This runs successfully and publishes test results. However, neither the TestFixture name or the Test's Fully Qualified Name are available in the test reuslts. This makes it difficult to determine what exactly what test failed without using lengthy test names.
Is there a way to include the Test Fixture names in the published test results from the build?
Please refer to the Tests summary tab of your build. It shows the detail information of test results of your Nunit3 tests.
You could use the Filter to find out those Failed or Succeeded tests.
You could also click the link in the picture below to check detail data for this test run.
Related
Good day.
I have a TFS build and a Visual Studio Test step there. All test are run, but I can't get detailed info about them, only quantity, passed\failed\others, run duration.
Also, I see "Outcome: Failed" and "This build doesn't have test results for the outcome you've selected".
How to configure this outcome?
You can't directly use the build-in outcome to analysis the detail test result.
You need to run the Publish Test Result task. First generate an XML file which contains the results. And then use the publish results step and pointing to that file so that the test results will show up in the build output.
More details about how to use this, please refer this similar tutorial with xUnit test: Execute and publish xUnit Tests results with .NET Core and VSTS
Note: For now VSTS/TFS does not support NUnit3 format. Source Link: Support NUnit2 format
Using the default build template on TFS 2013, we use the Post-test script to collect trx files from protractor tests, in order to include those in the build report summary.
We use vs2015 MsTest:
MSTest.exe
/publishresultsfile:...\TRXTestResult\CHROME-54.0.2840.71-results.trx
/publish:https://.../tfs/defaultcollection
/teamproject:...
/publishbuild:...
/platform:AnyCpu
/flavor:Release
Result:
Waiting to publish...
Publishing results of test run (test name) to
https://.../tfs/defaultcollection ...
Publish completed successfully.
However, the published test result wont show up in the build report (only the default from unit test).
This can't be achieved by a simple test script. You need to create a custom activity to handle this.
Inside this activity you can handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity you are also able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.
Then you need to create the custom build template and integration into a TFS Build. More details please refer the tutorial from this blog: Automated UI testing of AngularJS applications using TFS
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace
Today we run BDT test case and face following error from TFS MTM:
There are no test cases matching the criteria specified. Use Microsoft
Test Manager to view the test cases
We have only moved same test case to another test plan and it runs pass.
From post say it may be caused by permission issue.
Later we found out the root cause was that we have not included test configuration in test plan.
But the test configuration is already configured and used by BDT build definition.
So solution is just to include the test configuration into test plan.
Did you update your build definition to point to the new Test Plan and Suite? The error sounds like you have your BDT build pointing to a Test Plan/Suite that no longer exists.
I'm trying to setup TFS 2010 Lab Management based Automated Testing system and I have a "Test suite" (Created in TFS Test Manager) with following tests:
1) Start and Login
2) Create group
3) Create user and add to the group
These have order specified properly in Testing Center->Plan->(Test suite)->Order column
However, when they are executed (on TFS Lab Management Build) by Test Agent they run out of order: 2nd then 1st and then 3rd.
Is there a way to make Test Agent run these tests in order?
Thanks.
Create an Ordered Test, it's a simple list of which test methods should run and in what order.
I also had the same issue. I solved it by installing VS2010 Service Pack and TFS 2010 Service Pack in both the Test controller machine and Test agent machines. Please go through the following link for more details
http://social.msdn.microsoft.com/Forums/en-US/vsmantest/thread/29ac3c7b-b0cd-40f3-9992-f5f3c2285331
Unfortunately, There's a blog regarding ordering test cases which says:
Please note this only applies to the manual tests but not automated
tests. For automated tests, the order you set here will not be
respected during test execution.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/11/21/run-your-tests-in-the-order-you-want.aspx
I don't know if it's a new feature but you can create Ordered tests. Just right click on your project in Solution Explorer and choose Add -> Ordered Test. Select you tests in order.
In Test Explorer, run you ordered test.
Details are in the link below but all I had to do is what I'd said above. I am using VS 2015 Enterprise.
https://msdn.microsoft.com/en-us/library/ms182631.aspx