There are two third-party frameworks for dealing with the iOS UI Automation instrument: Alex Volmer's tuneup.js and the Jasmine framework.
What are the differences between these two frameworks and what are their advantages and disadvantages?
tuneup.js and Jasmine both provide nice wrappers around UIAutomation's rather verbose and arduous assertion API.
A plus of tuneup.js is that it not only includes the test/assertion abstractions, but also a number of handy convenience methods for performing various common tasks in UIAutomation (typing text into a textfield, for example). The downside to tuneup.js (in comparison to Jasmine), is that it lacks some of the nice delineation that Jasmine provides via nested describe / it spec definitions.
As noted above, Jasmine gives a rich way to define various specs of your app. You can demarkate various tests quite easily. Additionally, if you've used Jasmine in regular ol' Javascript testing, then it'll be very familiar.
In either case, if your app has even a remotely deep element hierarchy, I'd recommend pairing either tuneup or Jasmine with a framework of my own, mechanic.js. Mechanic makes traversing your app's 'DOM' and interacting with elements really easy. Plain ol' UIAutomation can get a bit verbose; mechanic is sort of the JQuery of UIAutomation.
Anyways, my personal choice would be Jasmine + Mechanic in most cases. I prefer the cadence of Jasmine's spec definition API, and paired with Mechanic, makes for terse, readable UI tests.
Jasmine and tuneup.js both provide a similar simple syntax for you to write your iOS automation tests with.
One advantage of Jasmine is that it has a good reporting structure so if you plan on running your automation tests from a CI server such as Jenkins or Hudson, it'll be much easier for your CI server to parse the XML test results from Jasmine versus the .plist format you get when using tuneup.js
Related
I'm currently looking at various integration/functional/acceptance testing frameworks for iOS and they all seem to rely heavily on the UI and accessibility labels. To me a lot of these seem more based towards UI tests as opposed to integration testing. Are there any dedicated frameworks that don't rely on this, especially when working with modules/frameworks that aren't built with a UI?
Is XCTest the way to go here when testing this kind of functionality? I've seen fitnesse and this seems to be the only other option I've found so far.
So in this instance I could have something like an API framework that is purely responsible for making network requests, parsing the response and creating relevant objects. Whilst I would write unit tests to test the individual pieces of functionality, I would also like to test an end to end flow.
Are there any suitable testing frameworks for this type of thing? Preferably swift based
thanks
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
I am learning Behavior Driven Development with ASP.NET MVC and, based on a post from Steve Sanderson, understand that BDD can mean, at least, the following test types: individual units of code & UI interactions. Something similar is mentioned in this post. Do I need two different test frameworks if I want both unit and integration testing?
Unit testing repositories, controllers, & services using a context/specification framework, like MSpec. The results of testing with this will be useful to the development team.
Testing complete behaviors (integration) using a given/when/then framework, like SpecFlow with Watin. The results of this testing will be useful for my client.
The videos I have seen so far on using BDD have only been limited to testing the behaviour of entities without testing the behaviour of repositories, controllers, etc... Is there a sample project where I can see both automated Unit and Integration testing using a BDD approach?
Personally I use SpecFlow for building feature specific tests (i.e. "User creates new company record") where I'll sometimes (but not always) use Watin. For testing my respositories, or service classes, I'll use unit/integration tests with NUnit. Integration tests are for when I need to talk to the database during the test, unit is for when I simply run code in the target object under test without external interactions.
I would say that you don't need to use a BDD framework for your non UI tests. You can if you want, but there is no hard and fast rule on this. If you are going to do this, then I highly recommend creating more then one project for your tests. Keeping them split is a good idea, rather then mixing all the test into one project. You could name them:
MyProject.Tests.Features <-- For BDD
SpecFlow tests.
MyProject.Tests.Integration <-- For
tests that access an
external resource i.e. database.
MyProject.Tests.Unit
If you're not wanting to use two BDD frameworks, you can still use MSTest/NUnit in a BDD way. For example, this blog article describes a nice naming convention which is close to BDD, but aimed at MSTest/NUnit unit tests. You could use this for your non SpecFlow tests when your testing things like repositories.
In summary - you don't have to use SpecFlow and MSpec in your testing, but if you do, then I recommend separate test projects.
I generally agree with what Jason posted.
You might want to divide your specs into two categories, system/integration and unit-level tests. You can describe both categories with any framework, but keep in mind that code-only approaches (NUnit, MSpec, etc.) require a business analyst to be capable of writing C#. SpecFlow/Gherkin can be a better approach if you want to involve analysts and users in writing specifications. Since the syntax and rules (Given, When, Then) are easy to understand and writing specifications from a user's perspective are easy to jot down after little training. It's all about bridging the communication gap and having users helping your team form the ubiquitous language of your domain.
I recommend having specifications support both working "outside in" and "inside out". You may start with an "outside in" SpecFlow specification written by the user/analyst/product owner and work your way from "unimplemented" towards "green" writing the actual code. The code supporting the feature is developed using TDD with a more technically oriented framework like MSpec (the "inside out" part).
Here's a repository that use MSpec for both unit and integration tests: https://github.com/agross/duplicatefinder.
Seems like the internet doesn't have a definitive answer, or set of principles to help me answer the question. So I turn to the great folk on SO to help me find answers or guiding thoughts :)
SpecFlow is very useful for BDD in .NET. But when we talk about BDD are we just talking integration/acceptance tests, or are we also talking unit tests - a total replacement for TDD?
I've only used it on small projects, but I find that even for my unit tests, SpecFlow improves code documentation and thinking in terms of language. Converseley, I can't see the full code for a test in one place - as the steps are fragmented.
Now to you..........
EDIT: I forgot to mention that I see RSpec in the RoR community which uses BDD-style syntax for unit testing.
I've recently started to use SpecFlow for my BDD testing, but also, I still use unit and integration tests.
Basically, I split the tests into seperate projects:
Specs
Integration
Unit
My unit tests are for testing a single method and do not perform any database calls, or external references whatsoever. I use integration tests for single method calls (maybe sometime two) which do interact with an external resources, such as a database, or web service, etc.
I use BDD to describe tests which mimick the business/domain requirements of the project. For example, I would have specs for the invoice generation feature of a project; or for working with a shopping basket. These tests follow the
As a user, I want, In order to
type of semantics.
My advise is to split your tests based on your needs. Avoid trying to perform unit testing using SpecFlow.
We have started using Specflow even for our unit tests.
The main reason (and benefit) for this is that we find that it forces you to write the tests from a behavior point of view, which in turn forces you to write in a more implementation agnostic way and this ultimately results in tests which are less brittle and more refactoring friendly.
Sure this can also be done with standard unit testing frameworks, but you aren't guided that way as easily as we have found we are using specflow and the gherkin syntax.
There is some overhead setting things up for specflow, but we find this is quickly repaid when you have quite a few tests (due to the significant step reusability that you can get with specflow) or you need refactor your implementation.
Plus you get nice readable specs that are easy for newcomers to the team to understand.
Given:
Unit tests are test of (small) "units of code"
The customer of most “units of codes” are other programmers.
Part of the reason for having a unit test is to provide an example of how to call the code.
Therefore:
Clearly unit tests should normally be written in the programming language that the users of the “unit of code” will be calling it with.
However:
Sometimes data tables are needed to setup the conditions a unit test runs in.
Most unit test frameworks are not good at using tables of data.
Therefore:
Specflow may be the best option for some unit test, but should not be your default choose.
I see it as an integration testing which mean it doesn't replace your unit test cases written as part of your TDD process. Someone will have different opinion about this. IMHO unit test case only test the methods/functions and all the dependencies should be mocked and injected. When in it comes to integration testing, you will be injecting real dependencies instead of mocked one. You could do the same integration testing with any of the unit testing frameworks, but the BDD provides you cleaner way of explaining the integration test use case in a Domain Specific Language which is a plain English(or any localized language).
Ta,
Rajeesh
I used specflow for BDD testing on two different good sized applications. Once we worked through the kinks of the sentence naming conventions, it worked out pretty good. BA's and QA's, and even interns could write BDD tests for the application.
However, I ALSO used it for unit tests. Heresy! I can hear some of you scream. However, there were VERY good reasons for it. The system was responsible for making many calculations or determinations based off a lot of different data. With lots of unit tests that require all this data to be input for test purposes, it makes it a LOT easier to manage that data used for the unit tests via the table format provided by specflow. Effectively mocking the data repository in table format, allowing the different components to be vigorously tested.
I don't know if I would do it in every case, but in the ones I used it for, it made laying out the volumes of data necessary for for performing the unit tests so much easier and clearer.
In the end we are trying to deliver to the customer exactly what the customer wants and as such I really don't see the need to write unit tests in addition to SpecFlow. After all, it exercises the same code base. I am fairly new to BDD/ATDD/TDD but other than being "complete" and strictly adhering to TDD I'm finding it unnecessary to write more unit tests.
Now I suppose if the team was dispersed and the developer was not able to run the entire application then separate unit tests would be necessary but where the developer(s) has access to the entire code base and is able to run the application, then why bother write more tests.
I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:
Acceptance tests based on validating results generated by controllers;
Integration tests using MvcIntegrationTestFramework;
Automated acceptance tests using Selenium;
Manual acceptance tests when tester is prompted to manually validate results.
I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:
Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.
If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.
Thanks in advance.
It's all behaviour.
Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.
The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)
I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.
At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.
After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.
Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.
Here's an example in the same codebase showing Given, When, Then in class-level example comments.
I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.
The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!
Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).
So this is a trade off, that can be different for each situation (that's why we provided the alternatives).
I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)