I am trying to run my UI Tests all at once. All the tests work fine when I run them individually, but when I run them automatically all together the tests fail because Xcode does not update where it is at in the app to successfully run the other tests.
So just to clarify, my first test is the signUpTest, so I'll run that test and it will work and then it will go to the second test which is the signUpMyInfo test. My issue is that this second test will only run once the app has moved to the second view controller, which is where the first unit test ends, however when the second test begins for some reason Xcode throws the app back to the first view controller, causing the second test to fail.
In short, I am unclear why the app returns to the first view controller after the first test successfully passes rather than remaining on the second view controller and running the second test from there.
Also worth noting, that this problem does not always happen, Xcode does sometime successfully throw me to the next view controller where the second test passes fine, but this problem is happening enough that I have to bother someone on stack to help me solve it :p
At the end of each test Xcode runs, it will close the app, and you'll need to launch the app again at the beginning of each test. This is to encourage test independence, which allows each test to stand alone, so that you know that if a test fails, it fails because of something that test did, not as a result of something another test did. This makes test failures easier to diagnose and makes your test results more informative and accurate.
You will need to add code in your second test to move from the first view controller to the second. This may seem like duplication at first glance, but as explained above, this is for the greater good of your test suite.
Related
I have created a custom framework and inside that all the logs are recorded when we initialise the framework.
Now I have integrate that framework inside a demo project and write UI test case for the same.
When I add the breakpoint inside the method then only logs are getting printed in the console while running the UI test case .Otherwise it don't print the logs in the console when break point is not added.
I don't know why thing is happening. I need logs for analysis purpose.
Could you please help me .
Print statement is only working when adding break point in the view controller class.
![2]
Your UI test code runs in a separate process outside of your app process. The default state shows the debugger for the UI test process, and not your app process, and will only show print statements written in the UI test code. The print statement you wrote seems like it’s a part of your app code.
If you hit a breakpoint in your app code, the Xcode console switches to show the debugger for your app, as opposed to the debugger for your UI test, which is why you see the print statement there.
I am new to unit tests in Xcode and Swift and have some trouble to understand the life cycle of XCTestCase.
How/where to add setup code which is executed before the actual app is launched?
Problem is, that first the host app is launched before any of the test setup methods (class func setUp(), func setUp(), func setUpWithError()) are executed.
It is even possible to run test code before the host app launches?
Details:
As described in a previous question my app uses a SQLite database to persist some data. When the app launches a database connection is created and data is read from the database.
To make tests consistent and repeatable I would like to use a fresh database with some well defined data every time a run the test. To archive this I tried to override setUpWithError remove the existing db file and move a file with pre-defined data in place instead.
Unfortunately this does not work, because setUpWithError is executed only after the host app was launched. The same is true for all other test setup methods.
Moving a fresh database file in place before running the tests is only an example. The problem is the same for all local data which should be in place before the host app launches to ensure repeatable tests.
An answer to my previous question included a UIApplication extension with a isTesting method which can be used to check if a test is performed. While I could use this in my app code to setup the test data, I would consider this a bad solution. I would like to keep the code completely separated from the production code. Is this possible?
There are several approaches to set up data before running a test case
NSPrincipalClass
As described here you can create a class, and the init method of that class is executed before running any test. This helps setting up dependencies used by many test cases. I don't think this is the way to go in your case.
isTesting
Instead of setting up the code in your app target, you can also check for isTesting early in the didFinishLaunchingWithOptions method of your AppDelegate and simply return there. In this case the regular code is not executed and you can run you custom database setup code in the setup of your test class.
Dependency Injection
Right now you are doing integration testing in my opinion. If you want to have proper unit tests, they should not operate against the database of the underlying app. Instead create an extra (in memory) database and inject that into the code you are testing. I think this is the way you should follow. The benefits are
Your app is not affected by your tests. Next time you start the app to manually test, your original data is still there.
Your unit tests are not affected by manual changes to the app
Your unit tests are also independent of each other. Order of execution doesn't change the results when setting up the database fresh for each test case.
It is even possible to run test code before the host app launches
If you have to ask that, your tests are not unit tests. Unit tests test code, not the app. A test that requires the app to be running would be a UI test.
In fact, the best approach for unit tests is to put all your testable code into a framework and give the framework unit tests; that way, your tests run much faster because the app never has to launch at all.
It sounds to me like the problem lies deeper in your app's code: you have evidently not written your code in such a way as to be testable. So that would be your first move. Writing testable code, and writing unit tests, is an art; you have to separate out the "system under test", which should be your code alone, and make sure you are not testing anything that doesn't belong to you and whose workings are already known — like Core Data.
My company switched from idea-131 to idea-171 a few months back (which I've been keeping updated ever since). One thing I've noticed is that I can no longer run Spock unit tests individually anymore. Instead, when I right click on a test, it only offers me the ability to run the whole file. That's kind of annoying if a page has 40 tests! Is there a config somewhere that I need to tick?
Nevermind, I answered that question as soon as i asked it! they've moved the option to a little play button headed beside each test method. I'm not a fan of iconicizing little commands but ... there you go.
I have a number of geb functional tests for a grails application.
The tests are working as expected when executed from terminal or IDE.
Although the tests need to be executed by hudson, so they are run in headless mode using Xvfb.
The problem is that the tests keep failing, or behaving unexpectedly, returning errors like RequiredPageContentNotPresent and Stale Element Reference Exception in places that doesn't make sense.
For example:
(at LicencePage is verified above, and page isn't changed)
when:
addDocument(Data.Test_Doc_name,Data.Test_Doc_file)
sometimes throws
Failure: Add Actual Licence (HomePageSpec)
| geb.error.RequiredPageContentNotPresent: The required page content 'addDocument - SimplePageContent (owner: LicencePage, args: [Functional Test Doc, /var/lib/hudson/jobs/KB-Functional_Tests/workspace/app/../manual_test_data/so_v3/os_test_1], value: null)' is not present
at geb.content.TemplateDerivedPageContent.require(TemplateDerivedPageContent.groovy:61)
at geb.content.PageContentTemplate.create_closure1(PageContentTemplate.groovy:63)
at geb.content.PageContentTemplate.create(PageContentTemplate.groovy:82)
at geb.content.PageContentTemplate.get(PageContentTemplate.groovy:54)
at geb.content.NavigableSupport.getContent(NavigableSupport.groovy:45)
at geb.content.NavigableSupport.methodMissing(NavigableSupport.groovy:121)
at geb.Browser.methodMissing(Browser.groovy:194)
at geb.spock.GebSpec.methodMissing(GebSpec.groovy:51)
at HomePageSpec.Add Actual Licence (HomePageSpec.groovy:228)
The method addDocument() is defined on an 'abstract' page, which LicencePage is extending. In most cases like this, if I copy the method code directly into my Spec, it is going to work, although its ruining all the structure I have on my test pages.
Anyone has experience running geb tests with Xvfb? Have you faced these issues?
All tests are passing when executed locally, and this not a data issue as the DB is always cleared
Also, without making any changes, the tests are behaving non-deterministic (on hudson) so the above exception is not always thrown. Without any changes at all, tests are sometimes successful and sometimes fail.
The description you gave seems to be the symptom of a flackey test-suite. we were facing this problem as well some time ago. A good starting point for this is this presentation (around min. 35) and the documentation about the wait stuff in geb.
If you think, it could have something to do with xvfb (where i have no experiences with), you could try to use phantomjs as the test-runner and check if it works correctly.
I have a test in fitnesse which used to work, but when I got into work today the test did not start at all. As soon as I press test I get the "0 errors 0 warrings..." text on the top of the test. Looking in the source control software, I can not find any changes to the test, or to anything related to it. I have noticed that the runner process does not start when I run the test. Other tests seem to work fine, and I can copy the tables from the test which is not working into an other test and everything is fine. Any ideas on what could be wrong?
My standard answer in this situation is, "have you checked your classpath?"
I say this, as typically when this sort of thing happens, it is that the !path doesn't point to the stuff you need, whether it be FitNesse.jar or your own custom code.
Also, do you get an output page where you can check the classpath? i doubt it, but that can help diagnose classpath issues.