We use TFS 2010 and Automated builds.
We also make use of MSTests.
I would like some concrete information about the build server's test execution method.
Will the test engine (on build server) run the unit tests sequentially or in parallel?
By default it will run them sequentially. You can customize the build workflow by adding a Parallel activity and running different sets of tests in each. Or if you want to parallelize the test run across multiple build machines you can have the build use multiple RunOnAgent activities to do so (http://blogs.msdn.com/b/jimlamb/archive/2010/09/14/parallelized-builds-with-tfs2010.aspx).
Note: If you execute the tests across multiple test runs you will end up with multiple test reports (.trx files) that will not be merged together without further customization of the build.
#Dylan Smiths answer is correct, but does not cover option # 3.
Executing Unit Tests in parallel on a multi-CPU/core machine
DANGER WILL ROBINSON: This is only applicable to VS2010 and mstest.exe. VS2012 has a new test runner that does not support parallel test execution Visual Studio UserVoice Run unit tests in parallel The VS2012 test system can use the legacy testrunner, and you can make it work if you specify a .testsettings file using the MSTest/SettingsFile element. Configuring Unit Tests by using a .runsettings File
How to: Enable parallel test execution
Ensure you have a multi-core/CPU machine
Ensure you are running only unit tests
Ensure your tests are thread-safe
Ensure you do not have any data adapters on
Ensure you are running locally (cannot use TestController/TestAgent)
Modify your test settings file.
Right-click the test setting file and select "Open With" -> Open as Xml
Set the parallelTestCount attribute on the Execution element
Options are:
blank = 1 CPU/Core - this is the default
0 = Auto configure: We will use as many tests as we can based on your CPU and core count
n = The number of tests to run in parallel
Save your settings file
Related
I am trying to write an integration test for my iOS app & our API server. Both are in the same repo, so this test would prevent merging any changes that would break their ability to communicate.
The server team has created a bash script that I need to call to start a local copy of the API server. That script will run until it is terminated. How can I run that script when I run my iOS integration test? I can't put it as part of the build phase as the build will wait for it to terminate before it lets the build finish. I can't use Process directly in my tests to launch the script because that's only available on the Mac, not to iOS targets. Refactoring all my networking & model code to it's own framework that supports iOS & Mac targets would work, but that's a sizable refactor to do. Are there any other ways to run commands along with tests? I'm just using XCTest for all my tests.
As mentioned in the comments I found a way to run scripts on test start & finish. You have to do it for the entire test target not the individual test cases.
You have to edit the scheme, expand the tests and add pre-actions & post-actions.
I am running my test in TFS (Nunit plus Visual Studio with Adapter) and I have set the build definition as below
Build succeeds but no test result file was generated
Does TFS writes this Xml file ?
Log
2017-02-08T08:08:40.8151428Z Executing the powershell script: D:\A1\agent\tasks\PublishTestResults\1.0.20\PublishTestResults.ps1
2017-02-08T08:08:41.0963795Z ##[warning]No test result files were found using search pattern 'D:\A1_work\1\s**\TEST-*.xml'.
If the Nunit plus Visual Studio with Adapter means you have two test steps: one for unit tests and the other one for vs tests.
Please also add two "Publish Test Results" step one for Nunit format.
Also run your test manually on the build server to see if test result file .trx generated on the machine.
we are currently using Wallaby.js for javascript unit testing. Works fine and is great. But within our development pipeline we of course want to run the same tests on the build server - in our case a tfs.
Is it possible to use wallaby on a tfs build server? Anf if yes how?
If not, what is the way to go to run the wallaby configured unit tests on the build server?
As we used the karma test runner earlier, I tried to execute the new test configuration with it but then I get
Can't find variable: wallaby
as in our main/ starting test file it is written
wallaby.delayStart();
require.config({
baseUrl: 'app',
(Originally from a karma/ requirejs configuration)
How to get around this?
Has anyone experience in this scenario?
Wallaby.js main idea is to integrate with editors, run tests for the code that you change and display the results in the editor. You can't use Wallaby.js in a CI build.
You may consider invoking other test runners, or use grunt/gulp task instead for javascript unit testing.
In TFS 2012 and later (might work in 2010 but not sure) you can extend the testing capabilities of the build system.
Check out these posts -
http://www.aspnetperformance.com/post/Unit-testing-JavaScript-as-part-of-TFS-Build.aspx
https://blogs.msdn.microsoft.com/visualstudioalm/2012/07/09/javascript-unit-tests-on-team-foundation-service-with-chutzpah/
I use selenium WebDriver with junit, ant and jenkins.
I set up jenkins to use ant build.xml to run my tests. But currently I run only one tests. In build.xml I set variable which is used in each test. So to run test in Jenkins I set in Targets:
build MyTest1 -Dvariable="value"
I want to run all tests in sequence one after another. I try this:
build MyTest1 -Dvariable="value" MyTest2 -Dvariable="value"
But 2 tests began run in browser at the same time. How can I organize needed sequence. Maybe there are some ways to do it in build.xml? I guess I can create target, in which call targets which runs tests, but how set my variable in that case? I'm new to ant so please advice me solution.
I need to clarify - my tests are independent, I won't run them in some stable sequence. The problem is that tests are running in parallel in browser. I need to run first test and only after it finish - run second test.
First of all, tests should have little dependencies. In your case, your tests depend on a global variable - try to get rid of it. Use a "configuration" object that you can modify from tests in a safe way and which your application code then uses to configure itself.
Which reduces the problem above to "how do I collect a number of tests" to which the answer is: Use a test suite:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
MyTest1.class,
MyTest2.class
})
public class JunitTestSuite {
}
I have the following sequence of things to do in my team build script after the sources are compiled:
-) Sources compiled
-) Run BVT tests (all tests in dlls *.bvt.dll)
-) Set build quality to BVTs Passed (or Rejected, if tests fail)
-) Run Unit tests (all tests in dlls *.unittests.dll)
-) Set build quality to Unit Tests Passed (or Rejected, if tests fail)
Can I accomplish this using team build script for VS 2008? If so, how?
EDIT: I have found a way to edit the build quality. Now all I need is to figure out how to run my tests in two stages. Anyone out there?
Thanks in advance!
A possible way to run accomplish is to write custom build tasks/workflow activities. Since there are no more alternatives presented, this is being marked as the answer. If there are other answers, they will be considered.