I use selenium WebDriver with junit, ant and jenkins.
I set up jenkins to use ant build.xml to run my tests. But currently I run only one tests. In build.xml I set variable which is used in each test. So to run test in Jenkins I set in Targets:
build MyTest1 -Dvariable="value"
I want to run all tests in sequence one after another. I try this:
build MyTest1 -Dvariable="value" MyTest2 -Dvariable="value"
But 2 tests began run in browser at the same time. How can I organize needed sequence. Maybe there are some ways to do it in build.xml? I guess I can create target, in which call targets which runs tests, but how set my variable in that case? I'm new to ant so please advice me solution.
I need to clarify - my tests are independent, I won't run them in some stable sequence. The problem is that tests are running in parallel in browser. I need to run first test and only after it finish - run second test.
First of all, tests should have little dependencies. In your case, your tests depend on a global variable - try to get rid of it. Use a "configuration" object that you can modify from tests in a safe way and which your application code then uses to configure itself.
Which reduces the problem above to "how do I collect a number of tests" to which the answer is: Use a test suite:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
MyTest1.class,
MyTest2.class
})
public class JunitTestSuite {
}
Related
I'm new to gradle, and I'm sure that this can be done somehow, but I can't figure out how - project does use dockerCompose task. It's spinning some docker dependencies up for tests. The problem is, it done in very dummy way:
dockerCompose.isRequiredBy( test )
which means that every single time I run tests from IDE, even if I'm running single unit test, that does not need docker dependencies at all, it's still starting up some docker containers(or failing miserably if docker agent is down).
I would like to run dockerCompose step only for some tests, only the ones that really need it. I don't have full conception, how I would like to mark those tests, but probably the easiest and most readable solution would be to use name, let's say ***DockerTest or ***IntegrationTest. And only for those ones I would run dockerCompose step.
I tried few different approaches, f.e. using system property in test's Before method and if statement in build.gradle(but it's already too late, dockerCompose already started before we went to test step), I've tried registering new task that extends Test and then to use 'filter' command, but it does not seem to work either(the task is started regardless from the test name, so it does not help me at all).
What would be the best way to do it, if that's even possible?
We just hit an issue with yeoman-generator tests when they would pass when run in isolation but fail when run in parallel with other tests.
Specifically, we call require('yeoman-generator').test.run() to run the generator and then use require('yeoman-generator').assert.file to check that the correct files were generated, which is what the documentation says. However, the assert would sometimes fail saying the files don't exist.
How does the interaction between test.run() and assert.file work? Where are the files written? Is is a global variable / temp file that is always the same and therefore can be overwritten by other tests running at the same time?
This is the test, and an example of a failing build.
There's a github issue with detailed discussion and here's a discussion on how the tests suddenly started passing when run in isolation.
We are using the Jest testing framework which runs tests in parallel.
Looks like Yeoman tests can't be run in parallel.
require('yeoman-generator').test.run() does create a temp directory but then changes the current working directory to that directory. This interferes with other tests that also rely on the CWD and therefore the Yeoman tests can't be run in parallel with other tests.
Relevant comment in run-context.js and process.chdir in helpers.js.
I am using Spock plug-in in my grails-2.3.4 application for automated unit and integration tests. When I run grails test-app, all the test cases run two times. Also in test report, every spec file is listed twice. As the application grew, number of test cases also grew, and all of them run twice. This takes double time to execute all of the test cases while development and deploying through Jenkins. Can anyone help me fix it (any help will be appreciated)?
http://grails.github.io/grails-doc/2.3.4/guide/upgradingFromPreviousVersionsOfGrails.html -> Spock included by default
You no longer need to add the Spock plugin to your projects. Simply
create Spock specifications as before and they will be run as unit
tests. In fact, don't install the Spock plugin, otherwise your
specifications will run twice [...].
In some other testing frameworks I'm used to tagging tests, eg #really_slow, #front_end
And then running different batches of tests, like I might want to set up a build slave to run all the really_slow tests, and might want to run all the tests tagged as front end but none that are marked as really slow.
To run my spock+geb tests in grails at the moment I just run grails test-app functional:
How do I tell it to run a subset?
You could use JUnit suites with #Category. Or you could use a SpockConfig.groovy with the following contents:
runner {
include foo.bar.FrontEnd, foo.bar.BackEnd
exclude foo.bar.Slow
}
Here, foo.bar.FrontEnd, foo.bar.BackEnd, and foo.bar.Slow are your own annotations. To activate the configuration file, you have to set a spock.configuration system property pointing to it.
We use TFS 2010 and Automated builds.
We also make use of MSTests.
I would like some concrete information about the build server's test execution method.
Will the test engine (on build server) run the unit tests sequentially or in parallel?
By default it will run them sequentially. You can customize the build workflow by adding a Parallel activity and running different sets of tests in each. Or if you want to parallelize the test run across multiple build machines you can have the build use multiple RunOnAgent activities to do so (http://blogs.msdn.com/b/jimlamb/archive/2010/09/14/parallelized-builds-with-tfs2010.aspx).
Note: If you execute the tests across multiple test runs you will end up with multiple test reports (.trx files) that will not be merged together without further customization of the build.
#Dylan Smiths answer is correct, but does not cover option # 3.
Executing Unit Tests in parallel on a multi-CPU/core machine
DANGER WILL ROBINSON: This is only applicable to VS2010 and mstest.exe. VS2012 has a new test runner that does not support parallel test execution Visual Studio UserVoice Run unit tests in parallel The VS2012 test system can use the legacy testrunner, and you can make it work if you specify a .testsettings file using the MSTest/SettingsFile element. Configuring Unit Tests by using a .runsettings File
How to: Enable parallel test execution
Ensure you have a multi-core/CPU machine
Ensure you are running only unit tests
Ensure your tests are thread-safe
Ensure you do not have any data adapters on
Ensure you are running locally (cannot use TestController/TestAgent)
Modify your test settings file.
Right-click the test setting file and select "Open With" -> Open as Xml
Set the parallelTestCount attribute on the Execution element
Options are:
blank = 1 CPU/Core - this is the default
0 = Auto configure: We will use as many tests as we can based on your CPU and core count
n = The number of tests to run in parallel
Save your settings file