I am running a ct test and my setup is failing, and thus the test suite is skipped.
*** FAILED {test_suite_name_here_SUITE,init_per_suite} ***
My question is how to debug the run of the test suite? And secondly how to get verbose output from the test run which tells me what went wrong.
You can try to use ct:comment("Output some variable when success ~p", [SomeVar]) and ct:fail("Output some variable when fail ~p", [SomeVar]) - this should helpful for output for success and fail cases. But some time you cannot do it, by reason that something wrong with template of tests, so in this case you need investigate the template what you try to use, for more info you can take a look to awesome online book where you can get a lot of info about Common Tests and not only.
Related
I'm using the Jenkins test harness (starting with this example usage: https://github.com/testcookbook/jenkins_harness) and this warning is killing me
Was this fixed? I tried following the issues but it's left me more confused. I just want to NOT see it in my output: https://issues.jenkins-ci.org/browse/JENKINS-60725?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel, https://issues.jenkins-ci.org/browse/JENKINS-60725, https://issues.jenkins-ci.org/browse/JENKINS-60725
I'm using this version of Jenkins and the harness:
testCompile "org.jenkins-ci.main:jenkins-test-harness:2.64"
testCompile "org.jenkins-ci.main:jenkins-war:2.235"
I set testLogging.showStandardStreams true in the build.gradle file because I want to see output of the script (I also write debug messages with print). This warning is making the test harness practically unusable because it's flooding the output. What options do I have for filtering it out?
Edit
Setting events 'standard_out' doesn't show the warnings but now I'm missing a lot of stuff.
When writing tests with dart:test how to print info messages such that they appear interleaved with tests output?
If I use print then it prints in the end, after all tests output.
Looking for analog of info() in ScalaTest.
After clarification with guys from the dart:test dev team it looks like regular print can be used. You just need to pass the --reporter=expanded argument when running
pub run test test/shimlaw_tests_test.dart --reporter=expanded
By default a compact single-line reporter is used which places output of print in the end of the test runner output. While the expanded reporter prints appropriately.
If I have a syntax error in one of my Common Test suites, ct_run simply waits for 15 seconds and then continues. It displays this message:
{error,make_failed}
Failed to compile or locate one or more test suites
Press 'c' to continue or 'a' to abort.
Will continue in 15 seconds if no answer is given!
(c/a)
If I use -noshell, then it displays:
{error,make_failed}
...and continues anyway.
How do I make it abort if compilation fails?
Answer is here. When ct_run encounters compilation error, it tries to ask you, whether you want to continue. continue/2 function does a dirty trick in order to determine if tests are run interactively. But you are able to cheat this function passing -noshell emulator option. In this case, continue/2 function will make decision based on -abort_if_missing_suites.
So, you need to ct_run -abort_if_missing_suites -erl_args -noshell.
I'm running NUnit tests using Jenkins (and the XUnit plugin), and Email-Ext to send out build result summaries.
I'd like to be able to email out something like "3 new test failures: [Names of tests that failed]." I can't work out how to get which tests changed from a previous run.
So far I have:
${TEST_COUNTS,var="total"} tests: ${TEST_COUNTS,var="pass"} pass,
${TEST_COUNTS,var="fail"} fail, ${TEST_COUNTS,var="skip"} skipped
giving
1914 tests: 1903 pass, 10 fail, 1 skipped
and ${FAILED_TESTS} giving the details of all tests failing - but I can't work out how to get just the changes from the previous run.
Viewing the job in Jenkins gives the information I need, so it ought to be possible.
Try this one:
============================
TESTS
There are ${TEST_COUNTS, var="total"} total tests of which ${TEST_COUNTS, var="fail"} test(s) failed.
$FAILED_TESTS
Try this:
CHANGES (All changes since first failure)
${CHANGES_SINCE_LAST_SUCCESS, reverse=true}
I am facing the issue of a test that has undefined step(s) not being flagged as a failed test.
In the Java code we use Selenium 2/WebDriver and tests are driven by Ant and run in a Continuous Integration environment.
For the following scenario:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application //working
And I view the test example //working
Then the tree panel exists in the layout //undefined step
The following is a snippet of what is seen in the console:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application
And I view the test example
Then the tree panel exists in the layout
1 scenario (1 undefined)
3 steps (1 undefined, 2 passed)
The ant target used to run the test:
ant test.cuke.firefox -Dwebtest.server="http://localhost:9944" -Dwebtest.cuke.options="--tags #test1"|wac
I read that using the --strict flag gets the tests to fail.
But I've no idea of where I need to mention the flag.
Is it in the build.xml file? If so, where exactly - as wherever I've tried hasn't helped.
Is it in the cucumber.yml file?
There are 2 such files:
i) \lib\cucumber.jruby\gems\cucumber-0.8.7
ii) \lib\cucumber.jruby\gems\gherkin-2.1.5-java
If not in these files, where else?
Could you please point to where and how the flag needs to be set?
I've tried looking up the help but nothing has helped (probably I'm looking in all the wrong places!)
Thanks!
You need to set the strict option:
http://cukes.info/api/cucumber/jvm/javadoc/cucumber/api/junit/Cucumber.Options.html#strict()
Edit: You can set this flag in the RunCukesTest like:
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"html:target/cucumber-html-report"},
strict = true)
public class RunCukesTest {
}