The test suite for my Lua project is split up into multiple files inside the spec/ directory.
Is there a way to ask busted to run those tests in parallel? If I call busted without any arguments it runs all the tests sequentially.
One thing that seems to work is to use GNU Parallel to run multiple test scripts at once.
parallel busted -o utfTerminal ::: spec/*_spec.lua
The -o utfTerminal is to tell busted to use the familiar "green circles" output instead of the simplified text output it uses when its stdout is redirected.
I don't understand much about the Busted library, but apparently what you want is working with multiple threads
Threads are basically the process where the code executes line by line until the end. When we create more threads for a code, multiple loops, functions, etc ... within this new Thread, they executed simultaneously with the original code, without interfering in the process, that is, more than one thing is executed in parallel.
Unfortunately Lua does not contain a way to perform multiple threads, the most it has to work with threads is coroutines. However, there are libraries like lua-llthreads that perform this task, try it out and see what you think. By joining it with your code with Busted, you will be able to perform parallel tasks
Related
I have google test based test suite. Since the tests manipulate the filesystem and do other things that I don't want to be left behind in case of a test crash, besides just not playing nicely with running tests on parallel, I want to run each test case in a new container. I am currently using CTest (aka. CMake test) to run the gtest binary, but I am not very attached to either of these, so if the best option is some other tool, I can accept that.
Can anyone suggest a way to automate this? Right now I am adding each individual test case manually to CTest with a call to docker run as part of the test command, but it is brittle and time consuming. Maybe I am doing this wrong?
You can run your GTest runner with --gtest_list_tests to list all tests.
You can then loop through this list and call your GTest runner with --gtest_filter set to the name of specific test.
The format of the list is a bit awkward to parse though, so need some shell scripting skills to get the actual test names.
Check the exit code of the GTest runner the know whether the test succeeded or failed.
I do not know if this integrates well with CTest.
In Bazel, you can re-run a test multiple times with:
bazel test --runs_per_test=<n> <target>
This is useful to reproduce conditions that lead a flaky test to fail.
The shortcoming of this approach, though, is that the tests will be run n times regardless of the results.
For tests that are flaky in very rare conditions, this means that you have to set n high, but this means you may have to scroll through a lot of text before you find the failing test's output.
Is there a built-in way in Bazel to run a test until it fails? Right now I'm using a while loop in Bash, which is good enough for my use case but not portable:
while bazel test --test_output=errors -t- <target_name>; do :; done
Passing --notest_keep_going will cause Bazel to exit as soon as a test fails. So, you could use --notest_keep_going in combination with an absurdly high --runs_per_test.
I have two Jenkins jobs, COMPILE and TEST, COMPILE triggers TEST, COMPILE is quick, TEST is slow. COMPILE re-creates data which is used by TEST, so if COMPILE runs while TEST is running, TEST might fail due to the necessary data being temporarily unavailable or incomplete.
Sometimes, COMPILE gets triggered a lot (via CMS, busy development). The standard way would be to synchronize COMPILE and TEST via a lock, so that both never run at the same time but instead wait for the other to finish before starting. This waiting does not really suit me as it delays the COMPILE jobs too much.
An alternative might be to turn TEST to concurrent running, but in my case TEST requires too many resources to be able to run concurrently.
So my approach now is to configure COMPILE so that it first aborts a running TEST job (in case one is running) and then starts its work (eventually triggering TEST again in the end). Several quickly performed COMPILE builds will then each start a TEST build which will all be aborted (gray bubble). Only the last TEST build will be completed and show a decent red or green (or yellow) bubble. But that will be enough in my case (and I accept the drawback that this way I cannot detect exactly which commit broke the build).
Now the question is: How can I make COMPILE abort a TEST build? (Preferably in an elegant way.)
I only found a way to generally abort a job from the outside using Jenkins's REST interface, but that doesn't seem to be very elegant. Is there a more elegant way, maybe using a Jenkins-internal feature I don't know or maybe by using a suitable plugin?
I would suggest consolidating the two jobs into a single job. That would allow for multiple simultaneous executions and will still fail if the compile fails.
You can set the number of Executors for the node to "1" . By this we can make sure only one jobs run at a time in the node , even if it is executed in parallel it will wait for the 1st job to complete and then start the second
I have a build file that has different variety of tasks. Some of these are in-house tasks that I am able to control the amount of logging/output generated.
The other tasks are libraries that I have no control over. They do not provide a way to control the amount of output. There is one very trivial task and I am comfortable with turning off the output of the task all together.
My question is if there a way to turn off this specific tasks output in the ant execution. Or does ant provide a way to wrap this task in another task that has echo set to 'off' or something similar?
-Syam
Ant has no builtin feature to turn off output for specific task, but there are possibilities via buildlisteners. See Make ant quiet without the -q flag? for answers
outputproperty="devnull"
It works fine for me and allways you can print this var if you need.
When running unit tests, Gradle can execute multiple tests in parallel without any changes to the tests themselves (i.e. special annotations, test runners, etc.). I'd like to achieve the same thing with ant, but I'm not sure how.
I've seen this question but none of the answers really appeal to me. They either involve hacks with ant-contrib, special runners set up with the #RunWith annotation, some other special annotations, etc. I'm also aware of TestNG, but I can't make the Eclipse plug-in migrate our tests - and we have around 10,000 of them so I'm not doing it by hand!
Gradle doesn't need any of this stuff, so how do I do it in ant? I guess Gradle uses a special runner, but if so, it's set up as part of the JUnit setup, and not mentioned on every single test. If that's the case, then that's fine. I just don't really want to go and modify c. 10,000 unit tests!
Gradle doesn't use a special JUnit runner in the strict sense of the word. It "simply" has a sophisticated test task that knows how to spin up multiple JVMs, run a subset of test classes in each of them (by invoking JUnit), and report back the results to the JVM that executes the build. There the results get aggregated to make it look like a single-JVM, single-threaded test execution. This even works for builds that define their own test listeners.
To get parallel test execution in Ant, you would need an Ant task that supports this feature (not sure if one exists). An alternative is to import your Ant build into Gradle (ant.importBuild "build.xml") and add a test task on the Gradle side.