I am using visual studio test runner task inside the deployment group phase in the release definition. I defined test assemblies to run but all the agents run the exact same tests in parallel. What have i missed in the configuration?
If you specified the same test assemblies on all agents, then of course they will run the same tests.
If you want to speed up the Test Execution, then you need to distribute runs to multiple machines.
Please refer to below articles for parallel testing:
Parallel Test Execution
Speeding up Unit Test Execution in TFS
Running Unit Tests on different machine during TFS 2015 build
Enable parallel execution of tests using the Visual Studio Test
Runner 2015.1
Related
I am pretty new to Azure Devops. Currently i am trying to run The Automated Selenium (C#) testcases from Test Plans in Azure Devops.
I am referring to the following link to set up the Release Plan to Run Automated Tests from Test Plan
https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I am able to link the Release Pipeline with the testplan and execute the test by creating a new Release. But when the testcases are executed from testplan, Below error is displayed. Its able to identify the Automated test. But in the second stage (Validating Stage) below error is displayed.
The error:
Validating stage:The selected stage does not have the right version or
settings of the Visual Studio Test task to run tests.
Steps Followed:
Created a TestPlan under TestPlan in Azure Devops
Added the testsuite and the testcases. Associated the Automated testscripts with the testcase.
Created a Release Plan from the Run automated tests from Test Manager template.
Task Added to the Release Pipeline:
Visual Studio Test Task
Configuration of Visual Studio Test Task:
Select tests using: TestPlan
Test suite: Autopopulated the Tetplan to which the Release Plan is linked
Test configuration: 1 - Windows 10 (default value)
Search folder: $(Agent.ReleaseDirectory)
The Build Artifacts for the Build Pipelines publish the Build artifacts in the $(Agent.ReleaseDirectory) folder.
Agent Used: Self Hosted Agent.
To run the automated testSripts by selecting the test cases from Test Plan.
Follow the instructions here https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
Note: For Visual Studio Test Task in release pipeline, Under the test
selection group, for Select test using option, select "Test run" by
default Test Assembies are selected
The documentation indicates you should verify that Select tests using is set to Test run. You can modify the setting and try again.
As per the documentation, in order to run Coded-UI as part of the build you need to have your build agent running as interactive Mode (Not as a service) also your Test Agent need to be deployed as Interactive Mode.
Now the problem is when you deploy your test agent your build machine will restart automatically so your Build agent will not restart cause its not a a service.
How can you work around this ?
Two options:
Don't use the build server as a test agent. Depending on the complexity and size of the test suite, you can either run the tests directly on the deployed environment or have a dedicated group of test machines to parallelize the run.
Install the test agent outside of a build process and just use the "run functional tests" task to connect to the already-running agent. The "Install test agent" task is really for scenarios closer to option #1, especially if you're doing things like provisioning test machines in the cloud on-demand.
Since you would like to run Coded UI test, you have to configure the test agent to interact with the Desktop and build agent running as interactive Mode.
If you insist on installing the TFS Test agent on the same machine as the Build agent. You need to manually install and configure the agents first. Then specify the test agent installation media path in test agent deploy task and use Run Functional Test task to run Automated Test on agent machine.
I have few Coded UI tests which are not in TFS. I was running these tests as ordered test from the CommandLine using the command:
vstest.console OrderedTest2.orderedtest
Now, I want to run this through Jenkins but i am seeing the below error:
Error Message:
Error calling Initialization method for test class SAClientAutomation.testsuites.SABVTSuite: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I have installed Test Agent and Test Controller but i don't what to next.
The code is not in TFS. Is TFS & MTM(Microsoft Test Manager) compulsory to achieve this?
Yes, TFS & MTM can run Coded UI tests. If you use TFS 2015.1 and previous versions, you need MTM to run the Coded UI tests, check: https://msdn.microsoft.com/en-us/library/ff472576(v=vs.120).aspx.
If you use TFS 2015.2 and later versions, you can use Run Functional Tests task directly in TFS build definition to run Coded UI tests. Check https://blogs.msdn.microsoft.com/visualstudioalm/2016/03/31/executing-automated-tests-in-build-vnext-using-test-plan-test-suites/.
In order to run coded UI tests in TFS 2015.2 and later, you need to deploy build agent and test agent in Interactive Process:
https://www.visualstudio.com/en-us/docs/build/actions/agents/v1-windows
https://www.visualstudio.com/en-us/docs/build/steps/test/visual-studio-test-agent-deployment
Is it possible to link the test results from a build defined in vNext with automated test cases defined in TFS?
Here is our current set-up:
TFS 2015 with update 1 (on premises)
User stories defined, acceptance criteria defined as scenarios which translate to test cases.
Code set-up with automated tests using MsTest (via SpecFlow)
Tests in MsTest linked with test cases using the test case manager tool:
> tcm testcase /import /collection:http://tfs:8080/tfs /teamproject:project1 /syncsuite:121 /storage:Domain1\Domain1.Spec\bin\Debug\Domain1.Spec.dll
2 builds created with vNext:
Continuous integration build where the Visual Studio Test build step used
Periodic delivery build which runs heavier tests using the Visual Studio Test using Test Agent build step
The problem is that we do not see any link to the test case in the test results that were run for the builds.
Having this link would provide useful information like
% of test cases running in CI build
% of test cases running for a delivery build
Run history for each automated test case with respect to the builds
etc.
There is not test case related information in build result. But you can get test run result on how many tests are passed.
I'm trying to get a bunch of MSpec tests to run on multiple cores in TFS 2013. It doesn't appear to do it out of the box. It can run MSpec, but only in sequence and it takes a over an hour.
I am following this guide, but in step 4 he says replace the Foreach Xaml element with ParallelForEach to get the tests to run in parallel. I downloaded the default build template in TFS 2013. It is a lot simpler, but it doesn't have this tag.
It has:
<mtba:RunAgileTestRunner
DisplayName="Run VS Test Runner"
Enabled="[Not AdvancedTestSettings.GetValue(Of Boolean("DisableTests", false)]"
TestSpecs="[AutomatedTests]"
ConfigurationsToTest="[ConfigurationsToBuild]" />
The default MSpec test runner cannot run tests in parallel. That's why you see the reimplementation of a parallel test runner.
I doubt that TFS is implementing an MSpec test runner from the framework source code (although that would be possible). That parallel test runner is using internal classes, like ISpecificationRunner, and running them in parallel.
Your only options, if you must stick with MSpec and TFS, are
Split your tests into multiple projects/assemblies and feed them to a TFS parallel task that shell-executes the default test runner
Use a TFS shell-execute task to run your tests through the parallel runner
I am assuming that if you want to run tests in parallel they are integration tests that take a long time to run.
If that is the cas then you should move all non Unit Tests out of the build and push them further down the pipeline.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can use Release Management to deploy your application and run your integration tests. Here you can run a larger number of long running tests without locking your build servers.