I'm running NUnit tests using Jenkins (and the XUnit plugin), and Email-Ext to send out build result summaries.
I'd like to be able to email out something like "3 new test failures: [Names of tests that failed]." I can't work out how to get which tests changed from a previous run.
So far I have:
${TEST_COUNTS,var="total"} tests: ${TEST_COUNTS,var="pass"} pass,
${TEST_COUNTS,var="fail"} fail, ${TEST_COUNTS,var="skip"} skipped
giving
1914 tests: 1903 pass, 10 fail, 1 skipped
and ${FAILED_TESTS} giving the details of all tests failing - but I can't work out how to get just the changes from the previous run.
Viewing the job in Jenkins gives the information I need, so it ought to be possible.
Try this one:
============================
TESTS
There are ${TEST_COUNTS, var="total"} total tests of which ${TEST_COUNTS, var="fail"} test(s) failed.
$FAILED_TESTS
Try this:
CHANGES (All changes since first failure)
${CHANGES_SINCE_LAST_SUCCESS, reverse=true}
Related
Failing tests resulted in green balls in our "Open Blue Ocean" pipeline overview. When I read the manual (https://doc.froglogic.com/squish/latest/rg-cmdline.html) this is according to specification, but using the --exitCodeOnFail should result in our desired behavior. In our Jenkinsfile we scripted the following:
squish([extraOptions: """--tags
${tag}
--retry
2
--config
addAppPath
${squishsrcdir}
--config
addAUT
startSimProApp.bat
${squishsrcdir}
--exitCodeOnFail
-666
--config
setResponseTimeout
30""", squishPackageName: 'squish for qt 6.5.2', testSuite: "${squishsrcdir}", unstableBuildOnError: true])
Unfortunately this results in the following error:
com.froglogic.squish.SquishException: unknown option --exitCodeOnFail
The squish plug-in version is: 8.1.1
What are my options to get red balls when a test fails under squish?
The --exitCodeOnFail option is not supported by the Squish plugin.
Take a look at https://doc.froglogic.com/squish/latest/ao-hudson.html#ao-jenkins-example-pipeline-jobs
The squish step sets neither build nor stage result. It returns the execution results as a string instead. Your pipeline may act based on the returned value. You can find an example on the last screenshot in the linked above chapter.
Squish has a known issue (reported and expected to be solved) in matching returned Squish test suite execution status with retries with the final result of the job. For example, if your test fails during first retry and get passed in the next retry, the final status of the job will remain as unstable/failed.
I am running a ct test and my setup is failing, and thus the test suite is skipped.
*** FAILED {test_suite_name_here_SUITE,init_per_suite} ***
My question is how to debug the run of the test suite? And secondly how to get verbose output from the test run which tells me what went wrong.
You can try to use ct:comment("Output some variable when success ~p", [SomeVar]) and ct:fail("Output some variable when fail ~p", [SomeVar]) - this should helpful for output for success and fail cases. But some time you cannot do it, by reason that something wrong with template of tests, so in this case you need investigate the template what you try to use, for more info you can take a look to awesome online book where you can get a lot of info about Common Tests and not only.
I have a TFS 2017 (version 15.105.25910.0) build which also runs tests, but one test is taking a very long time and the whole build is cancelled due to a timeout set in the 'general' tab of the build edit page. TFS log is included below. How can I check which test is faulty?
Notice the time difference between the first and second log. I assume a faulty test is being run after ReportAnalyzer_Blabla_SomethingTest, but with over 1k tests it's hard to guess which it is.
2017-08-30T11:30:09.7614471Z Passed ReportAnalyzer_Blabla_SomethingTest
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles(rootFolder = D:\TfsBuildAgents\RmsBuild\_work\8\s\TestResults, matchPattern = *.trx, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.GetMatchingItems(includePatterns.Count = 1, excludePatterns.Count = 0, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles - Found 0 matches
2017-08-30T11:53:52.1581687Z ##[debug]Processed: ##vso[task.logissue type=warning;code=002003;]
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1737949Z ##[warning]No results found to publish.
2017-08-30T11:53:52.1737949Z ##[debug]Processed: ##vso[task.logissue type=warning]No results found to publish.
2017-08-30T11:53:52.2050485Z ##[error]The operation was canceled.
2017-08-30T11:53:52.2050485Z ##[debug]System.OperationCanceledException: The operation was canceled.
Normally the faulty test should be the first test after ReportAnalyzer_Blabla_SomethingTest. But as you said, it with over 1k tests, according to the log you posted, if you didn't split the tests we can not exactly identify which test is the faulty one. In this case, I'm afraid that you have to debug that one by one.
So, yo can try to split the tests then debug them accordingly.
You can also try to check if there are any other detailed logs to track that.
See Review continuous test results after a build for more information.
I've found a messy workaround which helped me find the failing test. In all test classes (the messy part) I've added a code which appends the currently running unit test name to a file - the last entry was what I was interested in.
[ClassInitialize]
public static void ClassInitialzie(TestContext testContext)
{
// This is just an example!
File.AppendAllText("testRunLog.txt", testContext.TestName + Environment.NewLine);
}
The closes thing to "run the code before each test in the whole test project" seems to be the ClassInitialize attribute.
https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.classinitializeattribute.aspx
How to get Integration Test Results with additional output like number of: total tests, failed tests etc in Sonarqube?
My setup has jenkins integrated to sonarqube and I'm looking to get sonarqube display, the following results:
Number of Tests
Number of Tests Passed
Number of Test Failed
Number of Test Skipped
Total Execution time
As of now the Integration Test Results widget gives me only Coverage Results Output.
if you have tests unit and/or It, they sould be 'upload' on your sonarqube project dashboard. ( as the same way you should see then on your jenkins project dashboard )
Then, in 'Mesures' menu, at 'coverage' tab, there is a lots of displayed informations like nb test in failure, nb tests success, total exec time, nb tests skipped ...
I am facing the issue of a test that has undefined step(s) not being flagged as a failed test.
In the Java code we use Selenium 2/WebDriver and tests are driven by Ant and run in a Continuous Integration environment.
For the following scenario:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application //working
And I view the test example //working
Then the tree panel exists in the layout //undefined step
The following is a snippet of what is seen in the console:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application
And I view the test example
Then the tree panel exists in the layout
1 scenario (1 undefined)
3 steps (1 undefined, 2 passed)
The ant target used to run the test:
ant test.cuke.firefox -Dwebtest.server="http://localhost:9944" -Dwebtest.cuke.options="--tags #test1"|wac
I read that using the --strict flag gets the tests to fail.
But I've no idea of where I need to mention the flag.
Is it in the build.xml file? If so, where exactly - as wherever I've tried hasn't helped.
Is it in the cucumber.yml file?
There are 2 such files:
i) \lib\cucumber.jruby\gems\cucumber-0.8.7
ii) \lib\cucumber.jruby\gems\gherkin-2.1.5-java
If not in these files, where else?
Could you please point to where and how the flag needs to be set?
I've tried looking up the help but nothing has helped (probably I'm looking in all the wrong places!)
Thanks!
You need to set the strict option:
http://cukes.info/api/cucumber/jvm/javadoc/cucumber/api/junit/Cucumber.Options.html#strict()
Edit: You can set this flag in the RunCukesTest like:
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"html:target/cucumber-html-report"},
strict = true)
public class RunCukesTest {
}