I am doing XCUI Testing using the Cucumberish Framework. Sometimes when the test case starts it halts at the very first step
This is intermittent. Most of the times it runs seamlessly. I am are very confused what's going wrong.
I tried hard but can't get a stable solution.
Related
This has been happening with xcode 9 and xcode10 now, using Quick and XCTest for unit tests they will run all the way fine but throw an exception/crash at the end. If I continue the tests will succeed.
I've spent so many hours trying to figure this out with no luck. I don't see anything helpful in the logs or stack trace. It's a big code base with 2,000+ tests so it's hard to figure out if it's just one causing it. I've tried:
Removing all files from test target, adding 1 by 1, at some point it breaks again but moving back to previous OK version now starts to fail.
deleting all async tests from both Quick and XCTest
Deleting all XCTEst cases, to see if it's an issue of both in 1 target.
When I try to test my iOS app the Xcode 7 test navigator shows (null) the second time I run my tests.
The first time it looks normal:
However, as I run the tests again the 12 tests turn into a single test named (null) and it feels like the test cases aren't really run anymore, they just succeed every time after 1 millisecond.
I'm using Kiwi for the tests, but it feels more like something going wrong in Xcode7. I don't remember having this problem in Xcode6. The only thing that consistently solves the problem seems to be an Xcode reboot.
Has anyone had this issue as well?
I have just experienced the same issue. Commenting out the test class name so the test would not build and then putting the test class name back fixed it for me.
In my simple app I have a few files that are tested by unit testing and uitesting.
I am pretty sure that in UITests I tested EVERY line in WLLoginViewController. The test passed, but it is not considered as 100% in code coverage for that file. Sometimes when I restart Xcode, run app again, clean app, hard clean app. Do some accidental things, it works. But run tests once again, and again it is not considered. Why? Any ideas how to fix it? Is it a bug?
As you already mentioned, sometimes Xcode says that you are testing 100% of your code and sometimes it doesn't. So it's really inconsistent. That is an indicator for a bug.
Since you are still using an Xcode-beta, it's most likely a bug in Xcode. I wouldn't worry to much since it's still a beta-version and would check later, when the final version is released.
Ok, I've found temporary workaround.
Remove derived data for your project (window->projects)
Quit Xcode and Simulator.
Launch Xcode with your project, and run tests.
Run tests again.
Remember to wait until indexing is finished:
Weird, but try and see:)
I'm having many issues with unit testing in XCode 5 and want to verify if anybody else if having the same problems (and any possible solutions...) I'm using the Kiwi framework developing for ios7 and its a new project therefore is using XCUnit underneath. I'm on a 2011 macbook air on OSX 10.8
The first time I run a test on startup, it runs all tests, even if I only select one case/test class - it even runs all the disabled ones.
Upon adding new unit tests, there is ~1/3 chance that XCode crashes on running all tests.
Clicking on failed unit tests very rarely takes me to the failed unit tests. It acts as if the code has been deleted.
When unit tests have been fixed, errors are often left over both in the issue navigator and in the editor however test is now reporting to succeed.
Upon successful running of all unit tests Xcode often reports that Tests have failed however all show as successful in the navigator.
Unit tests quite regularly get 'stuck' and execute forever. Its then impossible to run/build anything else until have restarted xcode.
Anybody with a similar setup having the same issues? Any solutions? What a bloody mess.
It seems that Xcode 5.1 (DP) will solve your issues.
Yes I get similar problems, also using Kiwi. No idea if its Kiwi related, but certainly I experience crashes when running unit tests, perhaps about 1 in 5 runs, and more often if the program hits a breakpoint and I run the tests again with Cmd-U.
I haven't been using the 'test single' option, so can't comment there.
The errors left over problem appears to be generic to unit tests, not just Kiwi. If you delete the block that contains the left over error, wait a few secs for Xcode to recompile in the background and paste it back it, you'll get rid of it. Or exit and restart Xcode if you have a bunch of them.
I have enabled test coverage with no problem using the Generate Test Coverage Files and Instrument Program Flow with the fopen$UNIX2003 and fwrite$UNIX2003 hack.
But the problem is, when you use XCode to run tests it ends up launching the simulator which launches your app. When that happens the output for the test coverage is not truly correct because it thinks certain parts of the code are touched because they are executed when the app launches, not because a test touched them.
Is there a better way to see what code was actually touched by a test?
EDIT:
So not really an answer to my question, but a somewhat "solution" that will at least get better coverage numbers can be found here: Run logic tests in Xcode 4 without launching the simulator