I'm trying to get a bunch of MSpec tests to run on multiple cores in TFS 2013. It doesn't appear to do it out of the box. It can run MSpec, but only in sequence and it takes a over an hour.
I am following this guide, but in step 4 he says replace the Foreach Xaml element with ParallelForEach to get the tests to run in parallel. I downloaded the default build template in TFS 2013. It is a lot simpler, but it doesn't have this tag.
It has:
<mtba:RunAgileTestRunner
DisplayName="Run VS Test Runner"
Enabled="[Not AdvancedTestSettings.GetValue(Of Boolean("DisableTests", false)]"
TestSpecs="[AutomatedTests]"
ConfigurationsToTest="[ConfigurationsToBuild]" />
The default MSpec test runner cannot run tests in parallel. That's why you see the reimplementation of a parallel test runner.
I doubt that TFS is implementing an MSpec test runner from the framework source code (although that would be possible). That parallel test runner is using internal classes, like ISpecificationRunner, and running them in parallel.
Your only options, if you must stick with MSpec and TFS, are
Split your tests into multiple projects/assemblies and feed them to a TFS parallel task that shell-executes the default test runner
Use a TFS shell-execute task to run your tests through the parallel runner
I am assuming that if you want to run tests in parallel they are integration tests that take a long time to run.
If that is the cas then you should move all non Unit Tests out of the build and push them further down the pipeline.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can use Release Management to deploy your application and run your integration tests. Here you can run a larger number of long running tests without locking your build servers.
Related
I am using visual studio test runner task inside the deployment group phase in the release definition. I defined test assemblies to run but all the agents run the exact same tests in parallel. What have i missed in the configuration?
If you specified the same test assemblies on all agents, then of course they will run the same tests.
If you want to speed up the Test Execution, then you need to distribute runs to multiple machines.
Please refer to below articles for parallel testing:
Parallel Test Execution
Speeding up Unit Test Execution in TFS
Running Unit Tests on different machine during TFS 2015 build
Enable parallel execution of tests using the Visual Studio Test
Runner 2015.1
I have few Coded UI tests which are not in TFS. I was running these tests as ordered test from the CommandLine using the command:
vstest.console OrderedTest2.orderedtest
Now, I want to run this through Jenkins but i am seeing the below error:
Error Message:
Error calling Initialization method for test class SAClientAutomation.testsuites.SABVTSuite: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I have installed Test Agent and Test Controller but i don't what to next.
The code is not in TFS. Is TFS & MTM(Microsoft Test Manager) compulsory to achieve this?
Yes, TFS & MTM can run Coded UI tests. If you use TFS 2015.1 and previous versions, you need MTM to run the Coded UI tests, check: https://msdn.microsoft.com/en-us/library/ff472576(v=vs.120).aspx.
If you use TFS 2015.2 and later versions, you can use Run Functional Tests task directly in TFS build definition to run Coded UI tests. Check https://blogs.msdn.microsoft.com/visualstudioalm/2016/03/31/executing-automated-tests-in-build-vnext-using-test-plan-test-suites/.
In order to run coded UI tests in TFS 2015.2 and later, you need to deploy build agent and test agent in Interactive Process:
https://www.visualstudio.com/en-us/docs/build/actions/agents/v1-windows
https://www.visualstudio.com/en-us/docs/build/steps/test/visual-studio-test-agent-deployment
Is it possible to link the test results from a build defined in vNext with automated test cases defined in TFS?
Here is our current set-up:
TFS 2015 with update 1 (on premises)
User stories defined, acceptance criteria defined as scenarios which translate to test cases.
Code set-up with automated tests using MsTest (via SpecFlow)
Tests in MsTest linked with test cases using the test case manager tool:
> tcm testcase /import /collection:http://tfs:8080/tfs /teamproject:project1 /syncsuite:121 /storage:Domain1\Domain1.Spec\bin\Debug\Domain1.Spec.dll
2 builds created with vNext:
Continuous integration build where the Visual Studio Test build step used
Periodic delivery build which runs heavier tests using the Visual Studio Test using Test Agent build step
The problem is that we do not see any link to the test case in the test results that were run for the builds.
Having this link would provide useful information like
% of test cases running in CI build
% of test cases running for a delivery build
Run history for each automated test case with respect to the builds
etc.
There is not test case related information in build result. But you can get test run result on how many tests are passed.
I am using Jenkins for my project CI. And several automation test cases developed by Coded UI(C# language) will be run when the new build is deployed. I created a job on the master(windows) that MSTest those test cases(note: with MSTest under cmd, test cases can be run successfully) but saw the below issue:
Error calling Initialization method for test class QuickUI.SmokeTests: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=159018)
Therefore,
1) Need I use Jenkins slave to run test cases?
2) If so, how to config since the dlls needs to be copied to slave firstly?
3) Is there any account or jenkins service configuration needs to be taken care?
Thanks in advance.
Sounds like you need to get mstest setup on your build machine. The easiest way is to use test agents (http://www.microsoft.com/en-us/download/details.aspx?id=38186) when installing chose test agents.
(below is copied from my answer in Coded UI build server integration process)
You going to want to put your CodedUI tests inside an orderedTest. Right Click on the solution -> add -> orderedTest.
You going to want to install https://wiki.jenkins-ci.org/display/JENKINS/MSTestRunner+Plugin it's not a requirement, but it makes working with mstest in jenkins a little easier.
At this point you just need to configure the plugin in jenkins to run your orderedTest. It will need you to point to mstest and the location of your orderedtest.
1) Recommnd to run test on slave
2) To run GUI test, don't run the slave as windows service
3) Theoratically you only need the slave agent running. However the windows account logged in should have access to all the resources that your tests need.
Some other threads you could reference, pay special attention to this if you want to run the tests in unattended way.
Jenkins on Windows and GUI Tests without RDC
I am using Jenkins for integration testing.
Just to give the context. At the moment I have a separate build server which produces the build daily and Jenkins is not used as the build server. The build server executes the unit testing in my case.
When build process is complete it invokes the Jenkins job. In that job Jenkins start to deploy the build into the Virtual machine. I have a script for doing this.
Followed to that my plan is to run several scripts for doing the end-to-end testing.
Now I have several question in this regard:
How to parallelize the execution of the end-to-end tests?
As I am adding scripts after script I am getting worried how manageable it will be?
I am always using the web interface for adding and changing the scripts. How to do this from the command line?
Any ideas for a good tutorial? Any pointers from all of you? Thanks!
Looks like Build Flow Plugin is what I need.
https://github.com/jenkinsci/build-flow-plugin
You might want to try and see if you can use the Build Pipeline plugin before build flow. Much better visualization of what is going on, less scripting.
I link Build and deploy jobs in one sequence and then have unit and integration test jobs linked separately off the build job. You can then use Fail The Build plugin to have downstream jobs fail upstream ones.