I would like to run a JUnit regression Test Suite from within an ANT build script. The test suite has a known number of failures which we are working towards fixing. I want the build to fail if the number of failing tests increases (or changes at all if that's easier to code) - this value can be hard coded in the ANT script.
I'm looking for details on how to implement this.
Even though I'm a bit late to the party:
Another option would be to use JUnit 4 and annotate the failing tests with #Ignore.
That way they will not run, and not count as failures, but the number of ignored tests will still be printed, reminding you there is work left to do.
If a test fails for the first time, you'll get a regular failure; then you can decide whether to fix or to ignore it.
More information:
http://www.devx.com/Java/Article/31983/1954
The junit task creates and XML file with the failing tests. There's no reason you couldn't use XMLTask to count the number of failures and have a conditional fail if that value is too high.
Related
When compilation succeeds or a test passes, Bazel caches the result so if we repeat the build / test with the exact same code we get the result immediately.
That's great.
However, if the compilation fails - and I repeat the build with the exact same code - Bazel will attempt to recompile the code (and will fail again, with the exact same result)
Same for tests - if a test fails, and I rerun the test with the exact same code - Bazel will repeat the test.
Is there a way to tell Bazel to cache test / compilation failures as well as successes?
Usecase example:
I changed a lot of code in multiple files
I run bazel test //...:all
100 tests run, 4 different tests fail
I fix the code of one of the tests and rerun bazel test //...:all
All the failing tests run again, even though 3 of the failing tests have no dependency change and there's no point rerunning them
I have to wait 4x longer than necessary for the tests to finish, and I'm sad :(
Something similar for the build failures. Sometimes a failed build can take many minutes to run on our codebase. If I rebuild without changing the files - it's a waste of time for bazel to rerun the failing build if it can use the cache...
I can see why this might be useful, but I'm not aware of a flag or other option that does something like this. At least in the case of a failing test, Bazel re-runs the failed test even if the inputs haven't changed to account for flaky tests. I suppose the same could be said for actions (a flaky compiler sounds pretty scary though). You might consider filing a feature request: https://github.com/bazelbuild/bazel/issues.
My advice would be to test only the specific target you're working on after running //...:all. You can also combine that with --test_filter to test only a specific test method within the test target.
Partial answer:
If we want to cache test failures, we can add --cache_test_results=yes (as opposed to the default auto which only caches successes). Personally, I've added it to my .bazelrc...
I haven't found a way to cache compilation failures yet, unfortunately...
In Bazel, you can re-run a test multiple times with:
bazel test --runs_per_test=<n> <target>
This is useful to reproduce conditions that lead a flaky test to fail.
The shortcoming of this approach, though, is that the tests will be run n times regardless of the results.
For tests that are flaky in very rare conditions, this means that you have to set n high, but this means you may have to scroll through a lot of text before you find the failing test's output.
Is there a built-in way in Bazel to run a test until it fails? Right now I'm using a while loop in Bash, which is good enough for my use case but not portable:
while bazel test --test_output=errors -t- <target_name>; do :; done
Passing --notest_keep_going will cause Bazel to exit as soon as a test fails. So, you could use --notest_keep_going in combination with an absurdly high --runs_per_test.
How do I label flaky tests in junit xml syntax that jenkins uses to give us a report? jenkins gives me a nice report of tests that succeeeded and failed. I would like to know which tests are known to be flaky.
Certainly a flaky test is something that needs to be avoided and fixed. Tests should always pass and if not they need to be worked on.
However we don't live in a perfect world and thus it might be helpful to identify tests that have failed every now and then in the past.
Since Jenkins keeps track of test results you can step through the history of a particular test case (using the "Previous/Next build" links).
Additionally there are two Jenkins plugins that might be helpful:
Test Results Analyzer Plugin:
this plugin lets you see the history of a particular test case (or test suite) at one glance (plus adds nice charts)
Flaky Test Handler Plugin: this plugin has deeper support for failing tests (i.e. rerunning them). It's a bit restricted to maven and git though.
You label such tests not helpful. And your primary goal is to eliminate the "flakiness" of such tests - by identifying the root cause of the problem and either fixing production or test code; or both; or worst case by deleting or #Ignore'ing those tests.
Disclaimer of course: jenkins can't tell you about flaky testcases. How could it?!
If you assume that you have flaky testcases, you could spent some time to create a jenkins setup were all tests are run 5, 10, 50 times against the same build output; to then compare statistics. But this is nothing that would come for free - you will have to implement that yourself.
I have inherited a quite rotten test suite with over 1000 tests, where several hundreds are failing. There are several people fixing the tests, and I would like to quickly see which tests have been fixed between test job runs.
Currently, I copy the lists of failed tests from two runs, sort and diff them. I hope there is a better way.
The only thing I could find that mentions a better possibility is this old thread.
Thank you.
Have you considered using Sonar (now SonarQube)? You can pipe test results over to Sonar and keep test history in a searchable format. We use PMD/Findbugs/etc and test results to red/yellow/green a job for further CI deployment, but use Sonar for trends and analysis.
When running unit tests, Gradle can execute multiple tests in parallel without any changes to the tests themselves (i.e. special annotations, test runners, etc.). I'd like to achieve the same thing with ant, but I'm not sure how.
I've seen this question but none of the answers really appeal to me. They either involve hacks with ant-contrib, special runners set up with the #RunWith annotation, some other special annotations, etc. I'm also aware of TestNG, but I can't make the Eclipse plug-in migrate our tests - and we have around 10,000 of them so I'm not doing it by hand!
Gradle doesn't need any of this stuff, so how do I do it in ant? I guess Gradle uses a special runner, but if so, it's set up as part of the JUnit setup, and not mentioned on every single test. If that's the case, then that's fine. I just don't really want to go and modify c. 10,000 unit tests!
Gradle doesn't use a special JUnit runner in the strict sense of the word. It "simply" has a sophisticated test task that knows how to spin up multiple JVMs, run a subset of test classes in each of them (by invoking JUnit), and report back the results to the JVM that executes the build. There the results get aggregated to make it look like a single-JVM, single-threaded test execution. This even works for builds that define their own test listeners.
To get parallel test execution in Ant, you would need an Ant task that supports this feature (not sure if one exists). An alternative is to import your Ant build into Gradle (ant.importBuild "build.xml") and add a test task on the Gradle side.