I have a Java based test suite running Bazel 4.2.2, and my goal is to collect code coverage regardless of test flakiness. I tried to add these options:
bazel coverage ... --runs_per_test=3 --cache_test_results=no ...
but it looks like if 1/3 of those fail, then the test is failed and coverage data is not collected for failing tests.
Does Bazel have any flags to take the first passing result, and retry only on failures?
The full command I've tried is
bazel coverage --jobs=6 --runs_per_test=3 --cache_test_results=no --combined_report=lcov --coverage_report_generator="#bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main" -- //$TARGET/... 2>&1
Thanks!
Answer to my question (can't accept it yet): there's an option in the documentation I found
https://docs.bazel.build/versions/0.25.0/command-line-reference.html
--flaky_test_attempts=<a positive integer, the string "default", or test_regex#attempts. This flag may be passed more than once> multiple uses are accumulated
Each test will be retried up to the specified number of times in case of any test failure. Tests that required more than one attempt to pass would be marked as 'FLAKY' in the test summary. If this option is set, it should specify an int N or the string 'default'. If it's an int, then all tests will be run up to N times. If it is not specified or its value is ' default', then only a single test attempt will be made for regular tests and three for tests marked explicitly as flaky by their rule (flaky=1 attribute).
Another option is using the flaky attribute on your test rules for the problematic tests. That will run them up to 3 times even with normal bazel test, without needing any flags.
Related
I have many test code lines annotated with [<Fact(Skip="~~")>] expression.
However, when I run test with dotnet test command, they are not considered as active test cases.
How can I easily convert them into active test cases?
Since Ranorex does not provide re-run functionality from under the hood, I have to write my own and before I started, just want to ask for advice from people who've done it or maybe possible existing solution on the market.
Goal is:
In the end of the run, to re-run failed test cases.
Requirements:
Amount of recursive iterations should be customized
If Data binding is used, should include only Iterations for Data binding that failed
I would use the Ranorex command line argument possiblities to achieve this. Main thing would be to structure the suit accordingly that each test-case could be run seperately.
During the test I would log down the failed test cases either into a text file, database or any other solution that you can later on read the data from (even parse it from the xml result if you want to).
And from that data you'll just insert the test-case name as a command line argument while running the suite again:
testSuite.exe /testcase:TestCaseName
or
testSuite.exe /tc:TestCaseName
The full command line args reference can be found here:
https://www.ranorex.com/help/latest/lesson-4-ranorex-test-suite
Possible solutions:
1a. Based on the report xml: Parse report and collect info about all failed TC.
Cons:
Parse will be tricky
or:
1b. Or create list of failed TC on runtime : If failure occurs on tear-down add this iteration to the re-run list (could be file or DB table).
Using for example:
string testCaseName = TestCaseNode.Current.Name;
int testCaseIndex = TestSuite.Current.GetTestCase(testCaseName).DataContext.CurrentRowIndex;
then:
2a. Based on the list, run executable with parameters, looping though each record.
like this:
testSuite.exe /tc:testCaseName tcdr:testCaseIndex
or:
2b. Or generate new TestSuite file .rxtxt and recompile solution to created updated executable.
and last part:
3a. In the end repeat process, checking that failedTestCases == 0 || currentRerunIterations < expectedRerunIterations with script through CI run executable
or:
3b. Wrap whole Test Suite into Rerun test module and do the same check for failedTestCases == 0 || currentRerunIterations < expectedRerunIterations and run Ranorex from TestModule
Please let me know what you think about it.
In Fitnesse Commands: http://<host>:<port>/<suite path and test name>?responder=suite&startTest=TestTwo. I tried to execute. It is executing the test case which is passed in the url. If we pass the suite path and remove the test name, it is executing the whole suite. Is there any way we can run all tests coming after TestTwo?
No, you can run an entire suite or you can run an individual test. Perhaps you can break the suite into smaller sub-suites?
You can use firstTest parameters when calling the url. In your case, you can do:
http://<host>:<port>/<suite path>?responder=suite&firstTest=TestTwo
Note that, this only works based on alphabetical order of full path names of the test cases as mentioned in the fitnesse wiki here
firstTest: if present, only tests whose full path names are lexigraphically greater (later in alphabetical order) will be run. This is of questionable use since tests are not guaranteed to be run in any particular order.
Alternatively, you can tag certain test cases with tags, and execute them using parameter suiteFilter. You can find the relevant documentation in the same wiki page.
I started a new project in VS 2013 and TFS 2013. After doing a little coding I ran some tests and created test cases in the Test Manager. The tests pass and the Test result files show the impact XML files attached. However, all the subsequent builds show no Tests Impacted. After checking the build logs I see that the Test Impact entry has a Match pattern that ends in:
"\bin***.dll", but the application is a windows form app.
Is there something I have missed in setting up the project that would cause this?
This is the output log section:
...
Run VS Test Runner00:00:00
There were no matches for the search pattern ...\bin\**\*test*.dll
There were no matches for the search pattern ...\bin\**\*test*.appx
Run optional script after Test Runner00:00:00
Inputs
EnvironmentVariables:
Enabled: True
Arguments:
FilePath:
Outputs
Result: 0
Get Impacted Tests00:00:00
There were no matches for the search pattern ...\bin\**\*.dll
A baseline build could not be located. Test impact analysis will not be
performed for this build.
Publish Symbols
...
I found the issue. For the Test Impact Analysis the build must have a drop location: The "copy output to the server" option does not count as a "drop location"!
I was wondering if any of you guys had any experience generating code coverage reports in TFS Build Server 2010 while running NUnit tests.
I know it can be easily done with the packaged alternative (MSTest + enabling coverage on the testrunconfig file), but things are a little more involved when using NUnit. I've found some info here and there pointing to NCover, but it seems outdated. I wonder if there are other alternatives and whether someone has actually implemented this or not.
Here's more info about our environment/needs:
- TFS Build Server 2010
- Tests are in plain class libraries (not Test libraries - i.e., no testrunconfig files associated), and are implemented in NUnit. We have no MSTests.
- We are interested in running coverage reports as part of each build and if possible setting coverage threshold requirements for pass/fail criteria.
We 've done it with NUnit-NCover and are pretty happy with our results. NUnit execution is followed by NUnitTfs execution in order to get our testing results published in the Build Log. Then NCover kicks in, generating our code coverage results.
One major thing that poses as a disadvantage is fact that setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it.
Two things could pose as disadvantages:
NUnitTfs doesn't work well with NCover (at least I couldn't find a way to execute both in the same step, so (since NCover invokes NUnit) I have to run Unit tests twice: (1) to get the test results and (2) to get coverage results over NCover. Naturally, that makes my builds last longer.
Setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it .
In any case, the resulting reporting (especially the Trend aspect) is very useful in monitoring how our code evolves within time. Especially if you 're working on a Platform (as opposed to short-timed Projects), Trend reports are of great value.
EDIT
I 'll try to present in a quick & dirty manner how I 've implemented this, I hope it can be useful. We currently have NCover 3.4.12 on our build server.
Our simple naming convention regarding our NUnit assemblies is that if we have a production assembly "123.dll", then another assembly named "123_nunit.dll" exists that implements its tests. So, each build has several *_nunit.dll assemblies that are of interest.
The part in the build process template under "If not disable tests" is the one that has been reworked in order to achieve our goals, in particular the section that was named "Run MSTest for Test Assemblies". The whole implementation is here, after some cleanups to make the flow easier to be understood (pic was too large to be directly inserted here).
At first, some additional Arguments are implemented in the Build Process Template & are then available to be set in each build definition:
We then form the NUnit args in "Formulate nunitCommandLine":
String.Format("{0} /xml={1}\\{2}.xml", nunitDLL, TestResultsDirectory, Path.GetFileNameWithoutExtension(nunitDLL))
This is then used in the "Invoke NUnit"
In case this succeeds & we have set coverage for this build we move to "Generate NCover NCCOV" (the coverage file for this particular assembly). For this we invoke NCover.Console.exe with the following as Args:
String.Format("""{0}"" ""{1}"" //w ""{2}"" //x ""{3}\{4}"" //literal //ias {5} //onlywithsource //p ""{6}""",
NUnitPath,
Path.GetFileName(nunitDLL),
Path.GetDirectoryName(nunitDLL),
Path.GetDirectoryName(Path.GetDirectoryName(nunitDLL)),
Path.GetFileName(nunitDLL).Replace("_nunit.dll", ".nccov"),
Path.GetFileNameWithoutExtension(nunitDLL).Replace("_nunit", ""),
BuildDetail.BuildNumber)
All these run in the foreach loop "For all nunit dlls". When we exit the loop, we enter "Final NCover Activities" & at first the part "Merge NCCovs", where NCover.Console.exe is executed again - this time with different args:
String.Format("""{0}\*.nccov"" //s ""{0}\{1}.nccov"" //at ""{2}\{3}\{3}.trend"" //p {1} ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject
)
When this has run, we have reached the point where all NCCOV files of this build are merged into one NCCOV-file named after the build + the Trend file (that monitors the build throughout its life) has been updated with the elements of this current build.
We now have to only generate the final HTML report, this is done in "Generate final NCover rep" where we invoke NCover.reporting with the following args:
String.Format(" ""{0}\{1}.nccov"" //or FullCoverageReport //op ""{2}\{1}_NCoverReport.html"" //p ""{1}"" //at ""{3}\{4}\{4}_{5}.trend"" ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
PathForNCoverResults,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject,
BuildType
)