We would like to integrate a tests suite in our iOS app. To have an overall idea, this app is using web services, is saving files on the device and has some complex navigation parts. The requirements:
- run the tests suite with Jenkins
- being able to launch the test suite on a set of devices and iOS versions
- we don't expressly need something readable by non devs like Calabash proposes
We were thinking about doing some workflow testing (interaction bringing from one screen to another) and unit testing.
We googled a bit but articles are often at least 2 years old which is like the Paleolithic period in the mobile world. But still, it gives some nice first inputs.
For Unit Testing, we were thinking about:
GHUnit
XCTest
Kiwi
For workflow testing, we were thinking about:
Zucchini
Calabash
Regarding all the mentionned requirements, does anybody see any advice to chose one framework or the other (or another proposition)?
Thank you for any feedback.
ps: by the way, some interesting articles we have found on the subject that can help:
http://blog.lesspainful.com/2012/03/07/Calabash-iOS/
http://iosunittesting.com/faq/
I recommend Kiwi for unit testing for several reasons:
It supports nested contexts that can each have their own setup and teardown blocks. This allows your tests to be more DRY. XCTest only has one setup and teardown method which is used for all the tests in a file. With Kiwi's nested contexts, you can have setup code that is performed before some but not all of your tests based on how you define the contexts.
Kiwi has great support for mocking/stubbing dependencies when needed. I have found it to be more robust than OCMock (mocked class methods with OCMock stuck around after the test if you forgot to call -stopMocking whereas Kiwi always makes sure to teardown mocked class methods after each test).
I am not experienced in workflow/application testing but I plan to eventually check out KIF and UIAutomation.
I would highly recommend looking at iOS Tests/Specs TDD/BDD and Integration & Acceptance Testing for a more comprehensive discussion of testing libraries.
Related
Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
Typically I find that nearly all my most important test cases for iPhone development revolve around UI testing rather than business logic or data testing. I'm not very familiar with automated UI testing in the XCode environment. Can someone point me to a good tutorial or book?
UPDATE
This question was written several years ago and UI testing has come a long way since then. Using the UI Automation is still an option, but the KIF Framework is a much better solution for functional testing now, IMO.
From KIF's github page:
KIF, which stands for Keep It Functional, is an iOS integration test
framework. It allows for easy automation of iOS apps by leveraging the
accessibility attributes that the OS makes available for those with
visual disabilities.
KIF builds and performs the tests using a standard XCTest testing
target. Testing is conducted synchronously in the main thread (running
the run loop to force the passage of time) allowing for more complex
logic and composition. This also allows KIF to take advantage of the
Xcode 5 Test Navigator, command line build tools, and Bot test
reports.
Your best bet will be to use the UI Automation instrument that debuted with iOS 4.0. This can be scripted to test out many aspects of your user interface.
Apple provides a great introduction to this tool in the video for the WWDC 2010 session 306 - "Automating User Interface Testing with Instruments". I demonstrate how to set up scripts and do testing in the video for the "Testing" session of my advanced iOS development course on iTunes U. My notes on UI Automation from that class, including sample scripts, can be found here.
Additionally, you might want to look at James Turner's article "How to use UIAutomation to create iPhone UI tests" and Alex Vollmer's "Working with UIAutomation".
What are you really trying to test? I agree that testing view interaction is important but find that I can usually construct a unit or integration test which tests the view or the view and it's controller to validate this behavior without attempting to drive the entire app via the UI.
For example http://blog.carbonfive.com/2010/03/10/testing-view-controllers/ has examples of tests which test that UIKit views are bound to IBActions or have appropriate delegate's and that a controller responds to those action or delegate messages appropriately. There's no need to actually drive a UIButton or UITableView since I'm willing to trust that those classes actually work and instead only need to test my use and configuration of them.
For custom view components you can send UIResponder messages directly in your tests and assert that they behave as expected, again without driving the entire UI.
Can you provide an example of the sort of behavior you want to test?
This link could be helpful take a look https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
Seems like the internet doesn't have a definitive answer, or set of principles to help me answer the question. So I turn to the great folk on SO to help me find answers or guiding thoughts :)
SpecFlow is very useful for BDD in .NET. But when we talk about BDD are we just talking integration/acceptance tests, or are we also talking unit tests - a total replacement for TDD?
I've only used it on small projects, but I find that even for my unit tests, SpecFlow improves code documentation and thinking in terms of language. Converseley, I can't see the full code for a test in one place - as the steps are fragmented.
Now to you..........
EDIT: I forgot to mention that I see RSpec in the RoR community which uses BDD-style syntax for unit testing.
I've recently started to use SpecFlow for my BDD testing, but also, I still use unit and integration tests.
Basically, I split the tests into seperate projects:
Specs
Integration
Unit
My unit tests are for testing a single method and do not perform any database calls, or external references whatsoever. I use integration tests for single method calls (maybe sometime two) which do interact with an external resources, such as a database, or web service, etc.
I use BDD to describe tests which mimick the business/domain requirements of the project. For example, I would have specs for the invoice generation feature of a project; or for working with a shopping basket. These tests follow the
As a user, I want, In order to
type of semantics.
My advise is to split your tests based on your needs. Avoid trying to perform unit testing using SpecFlow.
We have started using Specflow even for our unit tests.
The main reason (and benefit) for this is that we find that it forces you to write the tests from a behavior point of view, which in turn forces you to write in a more implementation agnostic way and this ultimately results in tests which are less brittle and more refactoring friendly.
Sure this can also be done with standard unit testing frameworks, but you aren't guided that way as easily as we have found we are using specflow and the gherkin syntax.
There is some overhead setting things up for specflow, but we find this is quickly repaid when you have quite a few tests (due to the significant step reusability that you can get with specflow) or you need refactor your implementation.
Plus you get nice readable specs that are easy for newcomers to the team to understand.
Given:
Unit tests are test of (small) "units of code"
The customer of most “units of codes” are other programmers.
Part of the reason for having a unit test is to provide an example of how to call the code.
Therefore:
Clearly unit tests should normally be written in the programming language that the users of the “unit of code” will be calling it with.
However:
Sometimes data tables are needed to setup the conditions a unit test runs in.
Most unit test frameworks are not good at using tables of data.
Therefore:
Specflow may be the best option for some unit test, but should not be your default choose.
I see it as an integration testing which mean it doesn't replace your unit test cases written as part of your TDD process. Someone will have different opinion about this. IMHO unit test case only test the methods/functions and all the dependencies should be mocked and injected. When in it comes to integration testing, you will be injecting real dependencies instead of mocked one. You could do the same integration testing with any of the unit testing frameworks, but the BDD provides you cleaner way of explaining the integration test use case in a Domain Specific Language which is a plain English(or any localized language).
Ta,
Rajeesh
I used specflow for BDD testing on two different good sized applications. Once we worked through the kinks of the sentence naming conventions, it worked out pretty good. BA's and QA's, and even interns could write BDD tests for the application.
However, I ALSO used it for unit tests. Heresy! I can hear some of you scream. However, there were VERY good reasons for it. The system was responsible for making many calculations or determinations based off a lot of different data. With lots of unit tests that require all this data to be input for test purposes, it makes it a LOT easier to manage that data used for the unit tests via the table format provided by specflow. Effectively mocking the data repository in table format, allowing the different components to be vigorously tested.
I don't know if I would do it in every case, but in the ones I used it for, it made laying out the volumes of data necessary for for performing the unit tests so much easier and clearer.
In the end we are trying to deliver to the customer exactly what the customer wants and as such I really don't see the need to write unit tests in addition to SpecFlow. After all, it exercises the same code base. I am fairly new to BDD/ATDD/TDD but other than being "complete" and strictly adhering to TDD I'm finding it unnecessary to write more unit tests.
Now I suppose if the team was dispersed and the developer was not able to run the entire application then separate unit tests would be necessary but where the developer(s) has access to the entire code base and is able to run the application, then why bother write more tests.
I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:
Acceptance tests based on validating results generated by controllers;
Integration tests using MvcIntegrationTestFramework;
Automated acceptance tests using Selenium;
Manual acceptance tests when tester is prompted to manually validate results.
I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:
Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.
If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.
Thanks in advance.
It's all behaviour.
Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.
The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)
I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.
At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.
After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.
Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.
Here's an example in the same codebase showing Given, When, Then in class-level example comments.
I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.
The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!
Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).
So this is a trade off, that can be different for each situation (that's why we provided the alternatives).
I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)