Test with undefined steps not flagged as a failed test - ant

I am facing the issue of a test that has undefined step(s) not being flagged as a failed test.
In the Java code we use Selenium 2/WebDriver and tests are driven by Ant and run in a Continuous Integration environment.
For the following scenario:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application //working
And I view the test example //working
Then the tree panel exists in the layout //undefined step
The following is a snippet of what is seen in the console:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application
And I view the test example
Then the tree panel exists in the layout
1 scenario (1 undefined)
3 steps (1 undefined, 2 passed)
The ant target used to run the test:
ant test.cuke.firefox -Dwebtest.server="http://localhost:9944" -Dwebtest.cuke.options="--tags #test1"|wac
I read that using the --strict flag gets the tests to fail.
But I've no idea of where I need to mention the flag.
Is it in the build.xml file? If so, where exactly - as wherever I've tried hasn't helped.
Is it in the cucumber.yml file?
There are 2 such files:
i) \lib\cucumber.jruby\gems\cucumber-0.8.7
ii) \lib\cucumber.jruby\gems\gherkin-2.1.5-java
If not in these files, where else?
Could you please point to where and how the flag needs to be set?
I've tried looking up the help but nothing has helped (probably I'm looking in all the wrong places!)
Thanks!

You need to set the strict option:
http://cukes.info/api/cucumber/jvm/javadoc/cucumber/api/junit/Cucumber.Options.html#strict()
Edit: You can set this flag in the RunCukesTest like:
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"html:target/cucumber-html-report"},
strict = true)
public class RunCukesTest {
}

Related

Xcode does not simply run tests - no scheme and/or test plan

Objective
Running simple unit-test for an iOS app.
Setup
Xcode (Version 14.1)
Sample Test:
import XCTest
#testable import SDKTestAppSwift
class SDKTestAppSwiftTests: XCTestCase {
func testExample() throws {
print("Test: testExample")
// This is an example of a functional test case.
let base = "asdfghjkl123"
XCTAssert(base == "asdfghjkl123")
}
}
Next to the ..
func testExample() throws {
.. Xcode provides a nice little 'start single test'-button.
I used to klick it to start the test, locally (Xcode IDE on mac).
This used to work properly.
Problem
Klicking the 'start single test'-button, an overlay appears:
For the sake of this question to be machine-readable:
There is no scheme and/or test plan that contains every test you are trying to run.
Create a new scheme and/or test plan containing the tests you want to run.
To close the overlay an 'OK'-Button is delivered only.
Misleading solution (?!)
Just before there was no such scheme needed.
I wonder why would that changes 'suddenly'.
From my point ov view Scheme/Testplans are executed 'on startup' of the device-emulator (or likewise), which is not my objective in the first place.
However - creating a TestPlan via:
File > New > File... > TestPlan
Adding the SDKTestAppSwiftTests-class using the +-Button.
... no success ... :-/
Several other attempts
It seems my system is not the only one facing likewise issues.
Problems with naming of test plans
xcodebuild: Tests cannot be run because the test plan “Scheme” could not be read
No matter if there is a 'test-plan' and whatever name I assigned it - this is no solution.
Default test plan
https://developer.apple.com/forums/thread/133495
I seriously do not know how to create a 'default test plan' if it was not the way decribed before.
Quit Xcode and delete 'old configurations'
Xcode: No Scheme
xcodebuild says does not contain scheme
Something about files with old configurations which have to be deleted
In this case I am not sure which file and where to find it exactly.
Conclusion
Even though one of the obove may help you, for me the issue is not solved, yet.
Anyways
Thanks for reading & sharing - any help is appreciated :)
For me I fixed this through my scheme → Edit Scheme → Test → Plus Sign (Add Test Target) and then adding my unit test target.
There is another solution ...
First - Manage Schemes
Second - Create new Scheme
Third -> Celebrate, it's done .. :)

How to create a new demo in automotive?

I'm trying to separate out some code from drake/automotive/automotive_demo.cc. As a first step, I'm trying to copy automotive_demo.cc and automotive_demo.py into differently named files (test.cc and test.py) and then running bazel run automotive:test -- --num_simple_cars=1. I modified automotive/BUILD.bazel and test.py to take into account the new dependencies.
The problem is that after I bazel run, the simulator window opens but no car gets rendered. Eventually it just crashes with the following errors:
[lcm-spy] ClassDiscoverer: java.lang.NoClassDefFoundError: apple/laf/AquaPopupMenuUI
[lcm-spy] jar: ../com_jidesoft_jide_oss/jide-oss-2.9.7.jar
[lcm-spy] class: com/jidesoft/plaf/aqua/AquaJidePopupMenuUI.class
...
[drake_visualizer] Qt WebEngine seems to be initialized from a plugin. Please set Qt::AA_ShareOpenGLContexts using QCoreApplication::setAttribute before constructing QGuiApplication.
...
[lcm-spy] LCM: Disabling IPV6 support
[lcm-spy] LCM: TTL set to zero, traffic will not leave localhost.
[lcm-spy] java.net.SocketException: Can't assign requested address
Here is an (unresolved) Github issue that points to the problem being that test is a "custom plug-in". But if automotive_demo can work, surely there's a way to reproduce that behavior for test? I also tried grepping for QGuiApplication and only found a series of binary files, so I didn't know how to follow the error message's suggestion.
when trying out your steps on Mac I unfortunately cannot reproduce your specific errors. I do not think that having test as a target name should cause problems (at least I did not experience issues).
Could you please make sure:
You're able to run bazel run automotive:demo -- --num_simple_car=1?
After having renamed automotive_demo.* to test.*, in your BAZEL.build, test.py files the following are mapped correctly: demo -> test and automotive_demo -> test_cc (or whatever unique name you choose)?

TFS 2017 How do I know which test is being run (before it finishes)?

I have a TFS 2017 (version 15.105.25910.0) build which also runs tests, but one test is taking a very long time and the whole build is cancelled due to a timeout set in the 'general' tab of the build edit page. TFS log is included below. How can I check which test is faulty?
Notice the time difference between the first and second log. I assume a faulty test is being run after ReportAnalyzer_Blabla_SomethingTest, but with over 1k tests it's hard to guess which it is.
2017-08-30T11:30:09.7614471Z Passed ReportAnalyzer_Blabla_SomethingTest
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles(rootFolder = D:\TfsBuildAgents\RmsBuild\_work\8\s\TestResults, matchPattern = *.trx, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.GetMatchingItems(includePatterns.Count = 1, excludePatterns.Count = 0, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles - Found 0 matches
2017-08-30T11:53:52.1581687Z ##[debug]Processed: ##vso[task.logissue type=warning;code=002003;]
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1737949Z ##[warning]No results found to publish.
2017-08-30T11:53:52.1737949Z ##[debug]Processed: ##vso[task.logissue type=warning]No results found to publish.
2017-08-30T11:53:52.2050485Z ##[error]The operation was canceled.
2017-08-30T11:53:52.2050485Z ##[debug]System.OperationCanceledException: The operation was canceled.
Normally the faulty test should be the first test after ReportAnalyzer_Blabla_SomethingTest. But as you said, it with over 1k tests, according to the log you posted, if you didn't split the tests we can not exactly identify which test is the faulty one. In this case, I'm afraid that you have to debug that one by one.
So, yo can try to split the tests then debug them accordingly.
You can also try to check if there are any other detailed logs to track that.
See Review continuous test results after a build for more information.
I've found a messy workaround which helped me find the failing test. In all test classes (the messy part) I've added a code which appends the currently running unit test name to a file - the last entry was what I was interested in.
[ClassInitialize]
public static void ClassInitialzie(TestContext testContext)
{
// This is just an example!
File.AppendAllText("testRunLog.txt", testContext.TestName + Environment.NewLine);
}
The closes thing to "run the code before each test in the whole test project" seems to be the ClassInitialize attribute.
https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.classinitializeattribute.aspx

Jenkins - summarising test result changes from XUnit

I'm running NUnit tests using Jenkins (and the XUnit plugin), and Email-Ext to send out build result summaries.
I'd like to be able to email out something like "3 new test failures: [Names of tests that failed]." I can't work out how to get which tests changed from a previous run.
So far I have:
${TEST_COUNTS,var="total"} tests: ${TEST_COUNTS,var="pass"} pass,
${TEST_COUNTS,var="fail"} fail, ${TEST_COUNTS,var="skip"} skipped
giving
1914 tests: 1903 pass, 10 fail, 1 skipped
and ${FAILED_TESTS} giving the details of all tests failing - but I can't work out how to get just the changes from the previous run.
Viewing the job in Jenkins gives the information I need, so it ought to be possible.
Try this one:
============================
TESTS
There are ${TEST_COUNTS, var="total"} total tests of which ${TEST_COUNTS, var="fail"} test(s) failed.
$FAILED_TESTS
Try this:
CHANGES (All changes since first failure)
${CHANGES_SINCE_LAST_SUCCESS, reverse=true}

Running F# xUnit Fact from TestDriven.NET reporting "It looks like you're trying to execute an xUnit.net unit test."

I am trying to run xUnit tests (from an F# module, if it makes any difference) using TestDriven.NET, but whatever I do I get this error:
It looks like you're trying to execute an xUnit.net unit test.
For xUnit 1.5 or above (recommended):
Please ensure that the directory containing your 'xunit.dll' reference also contains xUnit's
test runner files ('xunit.dll.tdnet', 'xunit.runner.tdnet.dll' etc.)
For earlier versions:
You need to install support for TestDriven.Net using xUnit's 'xunit.installer.exe' application.
You can find xUnit.net downloads and support here:
http://www.codeplex.com/xunit
I tried following the suggestions, i.e. I copied the files
xunit.dll.tdnet
xunit.extensions.dll
xunit.gui.clr4.exe
xunit.runner.tdnet.dll
xunit.runner.utility.dll
xunit.runner.utility.xml
xunit.xml
to the folder with xunit.dll and I ran xunit.installer.exe. How can I get it to work?
I just figured out that I forgot to make the test a function in F# (so it was just a value). The error message can't be more misleading though!
You have two problems:
your Fact is broken:-
If you hover over the
please work
bit, you'll see something like: unit -> int
For a Fact to be picked up by an xUnit runner, it needs to yield `unit (void).
Hence, one key thing to get right first is to not return anything. In other words, replace your 123 with () (or an Assertion).
You can guard against this by putting a :unit stipulation on the test:-
[<Fact>]
let ``please work`` () : unit = 123
This will force a compilation error.
TestDriven.NET is reporting it cannot find the xunit.tdnet modules
It's critical to get step 1 right first. Then retry and the problem should be gone
If it remains...
Either try the VS-based runner which should work as long as it's installed and xunit.dll is getting to your output dir or look at the docs for your version of TD.NET for detailed troubleshooting notes (exec summary is if the .tdnet file was in your out dir or you undo and redo the xunit.installer from the folder containing the packages it should just work, esp if you are on latest)

Resources