We just hit an issue with yeoman-generator tests when they would pass when run in isolation but fail when run in parallel with other tests.
Specifically, we call require('yeoman-generator').test.run() to run the generator and then use require('yeoman-generator').assert.file to check that the correct files were generated, which is what the documentation says. However, the assert would sometimes fail saying the files don't exist.
How does the interaction between test.run() and assert.file work? Where are the files written? Is is a global variable / temp file that is always the same and therefore can be overwritten by other tests running at the same time?
This is the test, and an example of a failing build.
There's a github issue with detailed discussion and here's a discussion on how the tests suddenly started passing when run in isolation.
We are using the Jest testing framework which runs tests in parallel.
Looks like Yeoman tests can't be run in parallel.
require('yeoman-generator').test.run() does create a temp directory but then changes the current working directory to that directory. This interferes with other tests that also rely on the CWD and therefore the Yeoman tests can't be run in parallel with other tests.
Relevant comment in run-context.js and process.chdir in helpers.js.
Related
I have set up TeamCity to run tests for our codebase. Every now and then, some tests fail with Minitest::UnexpectedError: Errno::ENOENT: No such file or directory # dir_initialize. This only happens in tests usin the tmp directory (cache or creating files there), and these tests fails stochastically (90+% of the time they are fine). I've never encountered this error locally.
I tried adding mkdir tmp to a console build step, but that just made the build fail as the directory existed. I wonder if TeamCity is messing with the directory in builds somehow. We're on Rails 4.2.11.1, Ruby 2.4.6, minitest 5.11.3 and TeamCity 2019.1.1
I'm not sure how to go about fixing this - any ideas?
What I have seen usually is that /tmp doesn't clear up locally even when the Rails server reboots. However, running on CI is different as they would be usually allocating different machines for each CI run. Some cases where the tests might fail:
If your tests directly depend on some files in `/tmp'.
If some of your tests depend on o/p of other test cases which generate files in /tmp
Remember that CI might be running your suite parallely on multiple servers, so that is one more thing to consider. This is the reason why running your tests randomly is considered a good practice.
Hope this helps!
I am using Spock plug-in in my grails-2.3.4 application for automated unit and integration tests. When I run grails test-app, all the test cases run two times. Also in test report, every spec file is listed twice. As the application grew, number of test cases also grew, and all of them run twice. This takes double time to execute all of the test cases while development and deploying through Jenkins. Can anyone help me fix it (any help will be appreciated)?
http://grails.github.io/grails-doc/2.3.4/guide/upgradingFromPreviousVersionsOfGrails.html -> Spock included by default
You no longer need to add the Spock plugin to your projects. Simply
create Spock specifications as before and they will be run as unit
tests. In fact, don't install the Spock plugin, otherwise your
specifications will run twice [...].
I use selenium WebDriver with junit, ant and jenkins.
I set up jenkins to use ant build.xml to run my tests. But currently I run only one tests. In build.xml I set variable which is used in each test. So to run test in Jenkins I set in Targets:
build MyTest1 -Dvariable="value"
I want to run all tests in sequence one after another. I try this:
build MyTest1 -Dvariable="value" MyTest2 -Dvariable="value"
But 2 tests began run in browser at the same time. How can I organize needed sequence. Maybe there are some ways to do it in build.xml? I guess I can create target, in which call targets which runs tests, but how set my variable in that case? I'm new to ant so please advice me solution.
I need to clarify - my tests are independent, I won't run them in some stable sequence. The problem is that tests are running in parallel in browser. I need to run first test and only after it finish - run second test.
First of all, tests should have little dependencies. In your case, your tests depend on a global variable - try to get rid of it. Use a "configuration" object that you can modify from tests in a safe way and which your application code then uses to configure itself.
Which reduces the problem above to "how do I collect a number of tests" to which the answer is: Use a test suite:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
MyTest1.class,
MyTest2.class
})
public class JunitTestSuite {
}
I have a bunch of jasmine tests that I would like to run on a jenkins CI server.
At the moment, we use an html page that runs the specs, that a developper can open in a browser on its own machine.
The transition to CI would be easy if I had access to some kind of server side test runner (like karma), however for some undisclosable reasons, I can not run nodejs on our CI server.
So in the spirit of creativity-under-constraints, what could I use to automate jasmine tests without node ? (But anything that can run with maven and a jdk is probably fine...)
You can make your test automatically spawn a browser with the page that runs your unit test. The tricky part tough is to get the result back to the main test runner. The solution that I have found for that is to use a custom jasmine reporter (you just need to implement the same function has to other reporter) and when a spec has finished to run you do an AJAX call to write that result in a file. The main runner just needs to wait until something is written in that file to see the results. Once the test are finish, just don't forget to kill the browser, otherwise your CI server will be flooded by window.
In some other testing frameworks I'm used to tagging tests, eg #really_slow, #front_end
And then running different batches of tests, like I might want to set up a build slave to run all the really_slow tests, and might want to run all the tests tagged as front end but none that are marked as really slow.
To run my spock+geb tests in grails at the moment I just run grails test-app functional:
How do I tell it to run a subset?
You could use JUnit suites with #Category. Or you could use a SpockConfig.groovy with the following contents:
runner {
include foo.bar.FrontEnd, foo.bar.BackEnd
exclude foo.bar.Slow
}
Here, foo.bar.FrontEnd, foo.bar.BackEnd, and foo.bar.Slow are your own annotations. To activate the configuration file, you have to set a spock.configuration system property pointing to it.