Is it possible to link the test results from a build defined in vNext with automated test cases defined in TFS?
Here is our current set-up:
TFS 2015 with update 1 (on premises)
User stories defined, acceptance criteria defined as scenarios which translate to test cases.
Code set-up with automated tests using MsTest (via SpecFlow)
Tests in MsTest linked with test cases using the test case manager tool:
> tcm testcase /import /collection:http://tfs:8080/tfs /teamproject:project1 /syncsuite:121 /storage:Domain1\Domain1.Spec\bin\Debug\Domain1.Spec.dll
2 builds created with vNext:
Continuous integration build where the Visual Studio Test build step used
Periodic delivery build which runs heavier tests using the Visual Studio Test using Test Agent build step
The problem is that we do not see any link to the test case in the test results that were run for the builds.
Having this link would provide useful information like
% of test cases running in CI build
% of test cases running for a delivery build
Run history for each automated test case with respect to the builds
etc.
There is not test case related information in build result. But you can get test run result on how many tests are passed.
Related
I am pretty new to Azure Devops. Currently i am trying to run The Automated Selenium (C#) testcases from Test Plans in Azure Devops.
I am referring to the following link to set up the Release Plan to Run Automated Tests from Test Plan
https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I am able to link the Release Pipeline with the testplan and execute the test by creating a new Release. But when the testcases are executed from testplan, Below error is displayed. Its able to identify the Automated test. But in the second stage (Validating Stage) below error is displayed.
The error:
Validating stage:The selected stage does not have the right version or
settings of the Visual Studio Test task to run tests.
Steps Followed:
Created a TestPlan under TestPlan in Azure Devops
Added the testsuite and the testcases. Associated the Automated testscripts with the testcase.
Created a Release Plan from the Run automated tests from Test Manager template.
Task Added to the Release Pipeline:
Visual Studio Test Task
Configuration of Visual Studio Test Task:
Select tests using: TestPlan
Test suite: Autopopulated the Tetplan to which the Release Plan is linked
Test configuration: 1 - Windows 10 (default value)
Search folder: $(Agent.ReleaseDirectory)
The Build Artifacts for the Build Pipelines publish the Build artifacts in the $(Agent.ReleaseDirectory) folder.
Agent Used: Self Hosted Agent.
To run the automated testSripts by selecting the test cases from Test Plan.
Follow the instructions here https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
Note: For Visual Studio Test Task in release pipeline, Under the test
selection group, for Select test using option, select "Test run" by
default Test Assembies are selected
The documentation indicates you should verify that Select tests using is set to Test run. You can modify the setting and try again.
In my builds I trigger some 'mini' builds that run tests.
Each build that finish summarize to me how many tests in this run failed/passed.
I'm looking for a way to summarize all the builds that were triggered and their results in the main build.
Does there is a way or a feature that support its?
In TFS 2018/2019.
For example, I have 3 builds:
Build Tests (compile etc)
Run parallel tests
Run non parallel tests
The last 2 builds are triggered from the first build.
Each of them has their tests summarize.
I want to see a summarize of both builds (how many tests in total from all builds past, failed, did not run) in the main build.
I am using visual studio test runner task inside the deployment group phase in the release definition. I defined test assemblies to run but all the agents run the exact same tests in parallel. What have i missed in the configuration?
If you specified the same test assemblies on all agents, then of course they will run the same tests.
If you want to speed up the Test Execution, then you need to distribute runs to multiple machines.
Please refer to below articles for parallel testing:
Parallel Test Execution
Speeding up Unit Test Execution in TFS
Running Unit Tests on different machine during TFS 2015 build
Enable parallel execution of tests using the Visual Studio Test
Runner 2015.1
I have few Coded UI tests which are not in TFS. I was running these tests as ordered test from the CommandLine using the command:
vstest.console OrderedTest2.orderedtest
Now, I want to run this through Jenkins but i am seeing the below error:
Error Message:
Error calling Initialization method for test class SAClientAutomation.testsuites.SABVTSuite: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I have installed Test Agent and Test Controller but i don't what to next.
The code is not in TFS. Is TFS & MTM(Microsoft Test Manager) compulsory to achieve this?
Yes, TFS & MTM can run Coded UI tests. If you use TFS 2015.1 and previous versions, you need MTM to run the Coded UI tests, check: https://msdn.microsoft.com/en-us/library/ff472576(v=vs.120).aspx.
If you use TFS 2015.2 and later versions, you can use Run Functional Tests task directly in TFS build definition to run Coded UI tests. Check https://blogs.msdn.microsoft.com/visualstudioalm/2016/03/31/executing-automated-tests-in-build-vnext-using-test-plan-test-suites/.
In order to run coded UI tests in TFS 2015.2 and later, you need to deploy build agent and test agent in Interactive Process:
https://www.visualstudio.com/en-us/docs/build/actions/agents/v1-windows
https://www.visualstudio.com/en-us/docs/build/steps/test/visual-studio-test-agent-deployment
I'm trying to get a bunch of MSpec tests to run on multiple cores in TFS 2013. It doesn't appear to do it out of the box. It can run MSpec, but only in sequence and it takes a over an hour.
I am following this guide, but in step 4 he says replace the Foreach Xaml element with ParallelForEach to get the tests to run in parallel. I downloaded the default build template in TFS 2013. It is a lot simpler, but it doesn't have this tag.
It has:
<mtba:RunAgileTestRunner
DisplayName="Run VS Test Runner"
Enabled="[Not AdvancedTestSettings.GetValue(Of Boolean("DisableTests", false)]"
TestSpecs="[AutomatedTests]"
ConfigurationsToTest="[ConfigurationsToBuild]" />
The default MSpec test runner cannot run tests in parallel. That's why you see the reimplementation of a parallel test runner.
I doubt that TFS is implementing an MSpec test runner from the framework source code (although that would be possible). That parallel test runner is using internal classes, like ISpecificationRunner, and running them in parallel.
Your only options, if you must stick with MSpec and TFS, are
Split your tests into multiple projects/assemblies and feed them to a TFS parallel task that shell-executes the default test runner
Use a TFS shell-execute task to run your tests through the parallel runner
I am assuming that if you want to run tests in parallel they are integration tests that take a long time to run.
If that is the cas then you should move all non Unit Tests out of the build and push them further down the pipeline.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can use Release Management to deploy your application and run your integration tests. Here you can run a larger number of long running tests without locking your build servers.