I have a suite of regression tests that I am trying to convert from the NUnit runner to use the SpecFlow+ Runner. I've gotten them more or less running, with this major gotcha:
Many of my rests end with an Nunit Assert.Pass(). When I run those tests in SpecFlow runner, it reports them as test failures.
What can I do to get those reported as passing?
Related
I am using visual studio test runner task inside the deployment group phase in the release definition. I defined test assemblies to run but all the agents run the exact same tests in parallel. What have i missed in the configuration?
If you specified the same test assemblies on all agents, then of course they will run the same tests.
If you want to speed up the Test Execution, then you need to distribute runs to multiple machines.
Please refer to below articles for parallel testing:
Parallel Test Execution
Speeding up Unit Test Execution in TFS
Running Unit Tests on different machine during TFS 2015 build
Enable parallel execution of tests using the Visual Studio Test
Runner 2015.1
In Grails 3.1.4: If I use the Grails create-functional-test command and create a Geb test, when I use the Gradle test task it does not run the Geb test, it only runs the unit and integration tests, not the functional.
If I use the Grails test-app command it runs everything. I want to build with Gradle on my Bamboo 5.9.7 integration server.
How do I use Gradle to run all tests, unit, integration, and functional Geb tests?
The test task is only supposed to run the unit tests. You probably want to run integrationTest or iT for short.
In order to get grails3 gradle tests to mimic the way that grails test-app works in grails2 you might try using
gradlew --continue test iT
This will cause both unit and integration tests to run. Without the --continue option, if you have a failing unit test the integration tests will not execute.
Unfortunately I don't see how to add the "--continue" option to gradle commands when running from intellij.
Is it possible to link the test results from a build defined in vNext with automated test cases defined in TFS?
Here is our current set-up:
TFS 2015 with update 1 (on premises)
User stories defined, acceptance criteria defined as scenarios which translate to test cases.
Code set-up with automated tests using MsTest (via SpecFlow)
Tests in MsTest linked with test cases using the test case manager tool:
> tcm testcase /import /collection:http://tfs:8080/tfs /teamproject:project1 /syncsuite:121 /storage:Domain1\Domain1.Spec\bin\Debug\Domain1.Spec.dll
2 builds created with vNext:
Continuous integration build where the Visual Studio Test build step used
Periodic delivery build which runs heavier tests using the Visual Studio Test using Test Agent build step
The problem is that we do not see any link to the test case in the test results that were run for the builds.
Having this link would provide useful information like
% of test cases running in CI build
% of test cases running for a delivery build
Run history for each automated test case with respect to the builds
etc.
There is not test case related information in build result. But you can get test run result on how many tests are passed.
I had configured MSTest plugin in jenkins, but I am not able to run the unit automation tests. Actually the project is in TFS, where the unit tests are written. I had specified all .dll's path in Jenkins but Console output says No tests to execute
I'm trying to get a bunch of MSpec tests to run on multiple cores in TFS 2013. It doesn't appear to do it out of the box. It can run MSpec, but only in sequence and it takes a over an hour.
I am following this guide, but in step 4 he says replace the Foreach Xaml element with ParallelForEach to get the tests to run in parallel. I downloaded the default build template in TFS 2013. It is a lot simpler, but it doesn't have this tag.
It has:
<mtba:RunAgileTestRunner
DisplayName="Run VS Test Runner"
Enabled="[Not AdvancedTestSettings.GetValue(Of Boolean("DisableTests", false)]"
TestSpecs="[AutomatedTests]"
ConfigurationsToTest="[ConfigurationsToBuild]" />
The default MSpec test runner cannot run tests in parallel. That's why you see the reimplementation of a parallel test runner.
I doubt that TFS is implementing an MSpec test runner from the framework source code (although that would be possible). That parallel test runner is using internal classes, like ISpecificationRunner, and running them in parallel.
Your only options, if you must stick with MSpec and TFS, are
Split your tests into multiple projects/assemblies and feed them to a TFS parallel task that shell-executes the default test runner
Use a TFS shell-execute task to run your tests through the parallel runner
I am assuming that if you want to run tests in parallel they are integration tests that take a long time to run.
If that is the cas then you should move all non Unit Tests out of the build and push them further down the pipeline.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
You can use Release Management to deploy your application and run your integration tests. Here you can run a larger number of long running tests without locking your build servers.