We have a release pipeline which deploys our product, then runs some cucumber tests which produce junit output. We then use PublishTestResults to publish this junit results.
This will then show the results in the Test Tab of the pipeline.
These tests are checked in to TFS source control but do not exist in the Test Tab of TFS.
We would like to create an empty TFS Test for each cucumber tests, and have the result of the test, updated after each pipeline is deployed, from the junit test report. Is this possible with PublishTestResults or another method.
It's not able to do this through publish test result test task.
This task publishes test results to Azure Pipelines or TFS when tests
are executed to provide a comprehensive test reporting and analytics
experience. You can use the test runner of your choice that supports
the results format you require. Supported results formats include
CTest, JUnit (including PHPUnit), NUnit 2, NUnit 3, Visual Studio
Test (TRX), and xUnit 2.
The published test results are displayed in the Tests tab in the pipeline summary and help you to measure pipeline quality, review
traceability, troubleshoot failures, and drive failure ownership.
You could also take a look at the concept of TFS test case. Sorry not similar with cucumber tests, but after google, it looks like also not suit for build-in tfs test case.
So it's easy to publish the generated test result to tests tab in pipeline summary. But it's not possible to publish cucumber tests to TFS Test Manager with PublishTestResults or another method.
Related
I have 2 executables, an app and an automation test app who will perform actions on the app. My automation tests are basically NUNIT tests who calls Chrome Web Driver.
Everything is hosted at TFS. In my build definition, i run sanity checks for every PR. I wanted to expand that. The automation tests are divided into many different categories (being Sanity a category). I've been seeing some stuff related to TFS Test Suits, and my idea was to make in a way whenever when someone makes a PR, he could choose some test category to run on that PR using that build. So in a easyer way of sayng, if my PR changes how 'blue buttons work' ill run the 'blue buttons test suit' on my pr.
Would using Test Suits be the best solution for this ? Has any1 done this or have any nice information on how to achieve this ?
Thanks for any responses !! Best regards !
You could be able to use the Visual Studio Test Agent Deploy and Run Functional tests steps in your build definition to run auto tests on build agents.
Associated test methods with test cases in Visual Studio.
Create a build definition to build your project and add the 2 steps I mentioned above. In the Run Functional Test step, select the test suites which contains those test cases in step1.
More details please refer this blog: Executing Automated tests in Build vNext using Test Plan, Test Suites
Good day.
I have a TFS build and a Visual Studio Test step there. All test are run, but I can't get detailed info about them, only quantity, passed\failed\others, run duration.
Also, I see "Outcome: Failed" and "This build doesn't have test results for the outcome you've selected".
How to configure this outcome?
You can't directly use the build-in outcome to analysis the detail test result.
You need to run the Publish Test Result task. First generate an XML file which contains the results. And then use the publish results step and pointing to that file so that the test results will show up in the build output.
More details about how to use this, please refer this similar tutorial with xUnit test: Execute and publish xUnit Tests results with .NET Core and VSTS
Note: For now VSTS/TFS does not support NUnit3 format. Source Link: Support NUnit2 format
Using the default build template on TFS 2013, we use the Post-test script to collect trx files from protractor tests, in order to include those in the build report summary.
We use vs2015 MsTest:
MSTest.exe
/publishresultsfile:...\TRXTestResult\CHROME-54.0.2840.71-results.trx
/publish:https://.../tfs/defaultcollection
/teamproject:...
/publishbuild:...
/platform:AnyCpu
/flavor:Release
Result:
Waiting to publish...
Publishing results of test run (test name) to
https://.../tfs/defaultcollection ...
Publish completed successfully.
However, the published test result wont show up in the build report (only the default from unit test).
This can't be achieved by a simple test script. You need to create a custom activity to handle this.
Inside this activity you can handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity you are also able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.
Then you need to create the custom build template and integration into a TFS Build. More details please refer the tutorial from this blog: Automated UI testing of AngularJS applications using TFS
I have a problem of comprehension.
Unit tests are coded by developers in order to test a class (Java).
Integration tests are aimed to know if the different classes work together.
My problem is:
Based on continuous integration: I have Subversion (SVN) linked to Jenkins, and Sonar linked to Jenkins.
How are the integration tests created? Who does them? Are these tests already available in Sonar, or developers have to code them? Sonar launches integration tests thanks to Jenkins? How does it work...?
Integration tests are also coded by developers, to test multiple classes at one time, conceptually a "module", whatever that means in your world.
In my world, unit tests are tests that exercise one class, and have no dependencies externally. We allow file system access for mock data and logging, but that's all.
If a test exercises an actual database, or a running executable somewhere (e.g. web service) it is an integration test. We write them with junit, same as a unit test.
We find it works best for us to have separate Jenkins jobs linked in a pipeline to build, execute unit tests, execute integration tests, and load Sonar. While SonarQube is able to run tests for you, we prefer the separation which allows us to manually execute either set of tests via Jenkins without updating Sonar at the same time.
Has anybody had any success getting Team Build to show xUnit.net test results and code coverage in the build report? The MSBuild runner is running the tests just fine and the results are in the log file, but the test results and code coverage results areas of the build report say "No test result" and "No coverage result" respectively. Is Team Build looking for a certain location / format for the test results to be exported by the xUnit.net runner?
TFS/TeamBuild definitely requires the test results in a particular format, they also need to be specifically published to TFS as well.
I'm currently looking at this problem for the Gallio test runner, and can offer two potential options:
Try wrapping the xUnit tests inside the 'Generic Test' project type
that VSTS offers - this is where MSTest executes the xUnit test
runner.
Come up with an XSLT transform (or similar) that can map the
xUnit results to the MSTest schema (see
http://www.codeplex.com/nunit4teambuild for an example of this
approach for NUnit tests).
See also http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/