I was using KIF Framework until now for iOS UI Automation. KIF (also Unit Test) targets runs the Unit/UI testing code in the same process as your application is running and we can easily Mock different objects/classes used by the app to show mock data.
When switching to Xcode 7 based new UI Automation, I found that UI Unit Test target runs in separate process and it launches the application in separate process. And thus it's not possible to access App classes/objects through unit tests and mock them.
Did anyone faced the same issue, and know about any workaround?
Please let me know if any more details are needed.
Answer: Please see the detailed answer below. Also here is link to reply from Apple devs: https://forums.developer.apple.com/thread/23552
Because you are running in a different process, there isn't really a way to inject mocks into your application via your tests. Having said that, you do get to control your App's process, and can thus have some effects on it.
I've gone back and forth on how I feel about this (I'm currently undecided), but you could add some code to your app that reads an environment variable (via NSProcessInfo) that makes the app behave differently (i.e., change out what your dependency injection is injecting into your class so it uses mocks).
The downside to this is the mock code isn't contained strictly in the test bundle, and you end up with that code in your app (unless you use #if statements to isolate it). Obviously it's also important to be prudent with your code branching, so you don't invalidate your testing.
You could setup an environmental variable prior to launching the app in your UI test:
let app = XCUIApplication()
app.launchEnvironment = ["UITestUseMocks" : "true"]
app.launch()
Then in your application you can check for that, possibly in a #if statement and alter your behavior:
#if TEST_TARGET
if let useMocks = NSProcessInfo().environment["UITestUseMocks"] where useMocks == "true" {
// Alter services used in dependency injection or some other testing behavior
}
#endif
I've been considering this approach for implementing a mock service layer that just replays some canned server responses so I can make my UI tests not depend on server responses (other things test the server, after all).
Related
There are three basic layers of testing:
Unit test — making sure each function/unit of code works as expected
Functional test — making sure units interact with each other as expected
Integration test — making sure our app integrate with other app/api/services as expected
I can handle cases 1. and 2. using Xcode's iOS Unit Testing Bundle and iOS UI Testing Bundle I don't know how to write tests from point 3.
I would like to write to tests to check if me app correctly integrate with backend, BLE device, etc.
Note that I have already written unit tests which I run using CI.
Integration test should not be run in CI because it depends on external environment. I would like to run its only manually.
How to handle it in Xcode? Second Unit Testing Target? Any advice? How are you dealing with it in yours systems?
Depending on the style of integration test that you are after you may want to use dedicated a unit or UI tests target.
From your question I'm guessing the style you want to have leans towards black box testing with other real systems. If that's the case a dedicated UI tests target would do the job.
With UI tests you can do stuff like opening other apps on the device and test how your app interacts with them. For example, this post shows how to write a UI test that uses the Messages app to test Universal Links.
I'd reach for using a dedicated unit tests target only if you need to control part of the code in your app, for example stubbing network requests. But that doesn't really fit with the definition of integration test you have given.
I'm having trouble getting my test suite to pass in full when it runs on CircleCI. Everything will pass locally when run in the suite and when run individually. I eventually came across the issue that was causing problems, the Apple Notification/Location permission dialog. The tests still run, but any test that is expecting an alert to show fails because the Apple dialog is still on the screen. This basically happens every time you run the full suite on a new device or delete the app.
My question, what is the best way to deal with these dialogs? I'm pretty sure it is impossible to write the UI tests around it since all UI tests are pretty strict about ordering and what is on screen. The tests are randomized and the dialog only shows for the first test that calls for it, the rest won't need to worry about it.
My current thinking, but unsure how to proceed:
Mock the request for location/push notifications so it never triggers the dialog. Difficult for me so far as I have a class that does all the work for push and location. Do I mock against Apple classes or my own?
Write a seperate target for tests that only has 1 test that triggers the dialog and can close it. This would only be used when run on the CI server. This may not work as the same simulator may not be used between test runs. I would think it would, but no guarantees.
Add debug code to live code to bypass some of these dialogs and permissions. I already have code when run in DEBUG for notifications since you never actually get a successful token on the simulator. I just stub a fake return when run in debug so I can continue testing. What's a a little more? I'm not honestly considering this one if I can absolutely help it. There is already enough "testing" code in the live codebase that I'd like to prevent any more if at all possible.
Would love some feedback or examples on how to proceed if anyone has any ideas.
Setup Details:
ObjC
Latest version of Xcode, supports iOS 8 and up
Using KIF and OCMock for testing
We had a couple ways to workaround this:
In beforeEach/setUp, check the existence of the alert box and then use KIF's API to ack.
https://github.com/plu/JPSimulatorHacks just works!
Use a XCUITest to trigger permission flows and turn all permission on before moving to the target test bundle.
This answer is woefully late to my own question, but we did manage to solve this by using wix/AppleSimulatorUtils. Locally this was easy to resolve by running the app and approving any pop-ups for Location or Notifications. They won't come back until you run on a new sim. For command line/CI runs, we would use this as part of the setup. Pre-approve any permissions for the device and they won't show up during the test run. This tool has saved me many countless hours and I recommend it completely.
How I do I automate Xamarin.iOS unit test project.
For Android, I found this link which worked fine.
https://developer.xamarin.com/guides/android/troubleshooting/questions/automate-android-nunit-test/
Is there any references like this for Xamarin.iOS too?
At the time of writing this, I don't believe what you'd like to do is possible. If I take your meaning, you'd probably like to say "Run All" with some test runner (presumably in XS or VS) and then get the results immediately, but that's not how it works with Xamarin.iOS. I'm sure you've already glanced at Xamarin's iOS testing quick-start, but if not here that is.
You have to set up a Unit Test app that uses the Touch.Unit framework, fire up the test app, and "touch" the tests you'd like to run. My experience doing this has not been so great. The runner itself seems buggy and you're limited with in what other tools you can use (e.g., mocking frameworks won't work, assertions made with Shouldly won't register). I guess it's better than nothing, though!
I've got an Erlang project comprising a bunch of different applications. I'm using Common Test to do some of the testing.
apps/foo/suites/foo_SUITE.erl
apps/bar/suites/bar_SUITE.erl
I'm starting to see duplication of utility code in those suites.
Where should I put my utility code so that it can be shared between the two suites?
I've considered adding another application:
apps/test_stuff
...but I can't make the CT suites depend on this without making the application under test depend on this (or can I?). I don't want to do that, because test_stuff is only needed when testing.
I have a similar problem with my eunit tests, both between applications (apps/foo/test vs. apps/bar/test), and where I'm using similar functionality between the eunit and CT tests in the same application (apps/bar/suites vs apps/bar/test). Can I use the same solution for this case as well? Or do I need to ask another question about that?
Do you think ct:require/1,2 could help you so that foo and bar SUITE would require test_stuff before it gets executed? For more information http://www.erlang.org/doc/man/ct.html#require-1
It depends on how you are packaging your final releases. For example, I use rebar for relase management. I have Cowboy fetched along with other dependencies for testing purposes, but in my reltool.config, I omit it, so it doesn't get packaged with the final product. I use rebar to run Common Test, and it's able to add Cowboy to the path without having it bundled as a lib with everything else or added as a dependency to the app I'm testing.
However, if you have another process which infers your release configuration from your dependencies, you'll have to find a way to exclude your test code when you generate a release.
I would like to perform the following iPad/iPhone testing scenario automatically:
Tap Edit box A
Type text "abcd"
Verify button B is high-lightened
I understand UIAutomation 4.0 allow you to write a simple JavaScript to perform the above steps. However, UIAutomation does not have test infrastructure ready. For example it lacks testing macros to show if any tests failed and does not have a clear way to run setup and shutdown for each test cases.
That is why I look back to XCode unit testing. Logic tests won't work for me. How about Application tests?
Basically, I am looking for something that can do GUI testing and at the same time has test infrastructure. It is even better if it can be integrated to continuous build environment.
Check out Zucchini. It's just come out and I saw a demo at a recent YOW! conference. It's basically a BDD testing framework that uses coffeescript for scripting and runs against an actual device. It's also fully runnable from CI servers which makes it perfect for agile teams.
I haven't run it myself yet, but it seems to exactly what I'm looking for and No I don't work for PlayUp :-)