Using the default build template on TFS 2013, we use the Post-test script to collect trx files from protractor tests, in order to include those in the build report summary.
We use vs2015 MsTest:
MSTest.exe
/publishresultsfile:...\TRXTestResult\CHROME-54.0.2840.71-results.trx
/publish:https://.../tfs/defaultcollection
/teamproject:...
/publishbuild:...
/platform:AnyCpu
/flavor:Release
Result:
Waiting to publish...
Publishing results of test run (test name) to
https://.../tfs/defaultcollection ...
Publish completed successfully.
However, the published test result wont show up in the build report (only the default from unit test).
This can't be achieved by a simple test script. You need to create a custom activity to handle this.
Inside this activity you can handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity you are also able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.
Then you need to create the custom build template and integration into a TFS Build. More details please refer the tutorial from this blog: Automated UI testing of AngularJS applications using TFS
Related
We have a release pipeline which deploys our product, then runs some cucumber tests which produce junit output. We then use PublishTestResults to publish this junit results.
This will then show the results in the Test Tab of the pipeline.
These tests are checked in to TFS source control but do not exist in the Test Tab of TFS.
We would like to create an empty TFS Test for each cucumber tests, and have the result of the test, updated after each pipeline is deployed, from the junit test report. Is this possible with PublishTestResults or another method.
It's not able to do this through publish test result test task.
This task publishes test results to Azure Pipelines or TFS when tests
are executed to provide a comprehensive test reporting and analytics
experience. You can use the test runner of your choice that supports
the results format you require. Supported results formats include
CTest, JUnit (including PHPUnit), NUnit 2, NUnit 3, Visual Studio
Test (TRX), and xUnit 2.
The published test results are displayed in the Tests tab in the pipeline summary and help you to measure pipeline quality, review
traceability, troubleshoot failures, and drive failure ownership.
You could also take a look at the concept of TFS test case. Sorry not similar with cucumber tests, but after google, it looks like also not suit for build-in tfs test case.
So it's easy to publish the generated test result to tests tab in pipeline summary. But it's not possible to publish cucumber tests to TFS Test Manager with PublishTestResults or another method.
I have 2 executables, an app and an automation test app who will perform actions on the app. My automation tests are basically NUNIT tests who calls Chrome Web Driver.
Everything is hosted at TFS. In my build definition, i run sanity checks for every PR. I wanted to expand that. The automation tests are divided into many different categories (being Sanity a category). I've been seeing some stuff related to TFS Test Suits, and my idea was to make in a way whenever when someone makes a PR, he could choose some test category to run on that PR using that build. So in a easyer way of sayng, if my PR changes how 'blue buttons work' ill run the 'blue buttons test suit' on my pr.
Would using Test Suits be the best solution for this ? Has any1 done this or have any nice information on how to achieve this ?
Thanks for any responses !! Best regards !
You could be able to use the Visual Studio Test Agent Deploy and Run Functional tests steps in your build definition to run auto tests on build agents.
Associated test methods with test cases in Visual Studio.
Create a build definition to build your project and add the 2 steps I mentioned above. In the Run Functional Test step, select the test suites which contains those test cases in step1.
More details please refer this blog: Executing Automated tests in Build vNext using Test Plan, Test Suites
I have some unit tests in single class "test.cs" and which runs on Visual Studio 2015. Now I want to configure the tests on TFS build. I tried with below configuration and not success.
Builds->Edit Build Definition->Process->Test->1.Automated Tests->1.Test source->Test Source spec=**\*test.cs
How to configure the build definition to run single class tests on TFS build?
Make sure you specify the test assembly as the test source as Deepa mentioned:
You could then use the criteria to specify the specific class you want to run tests from using:
FullyQualifiedName~NameSpace.Test
However pinning your build on a single class to run tests isn't a good practice. If you are using MSTest, consider using test categories:
If you have created automated tests using Visual Studio you can manage
these tests by categorizing them with test categories. When you run
these automated tests using Visual Studio, Team Foundation Build, or
mstest.exe, you can use these test categories to easily select which
tests you run. You add test categories as attributes to your test
methods.
source: MSDN
These will give you some more flexibility and prevent the build from no longer running test if you rename the class or anything
In your criteria specify the use the category to filter the tests that will run:
TestCategory=MyCategory
There is some more info about the criteria you can use in this blog post. The screenshot are taken from a build definition using the GitTemplate.12.xaml template.
Good day.
I have a TFS build and a Visual Studio Test step there. All test are run, but I can't get detailed info about them, only quantity, passed\failed\others, run duration.
Also, I see "Outcome: Failed" and "This build doesn't have test results for the outcome you've selected".
How to configure this outcome?
You can't directly use the build-in outcome to analysis the detail test result.
You need to run the Publish Test Result task. First generate an XML file which contains the results. And then use the publish results step and pointing to that file so that the test results will show up in the build output.
More details about how to use this, please refer this similar tutorial with xUnit test: Execute and publish xUnit Tests results with .NET Core and VSTS
Note: For now VSTS/TFS does not support NUnit3 format. Source Link: Support NUnit2 format
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace