spinach vs cucumber for BDD in rails - ruby-on-rails

I am starting on BDD. Was wondering which would be better to start with Cucumber or Spinach. My impression is that Spinach is new off the block. Look here
Which one should I start with. The criteria would be -
Support across the board.
Flexibility of use
Third party tool and APIs integration.
Again it might be ignorant question of the newbie: Where does capybara fit into the picture.

For some context, I've been a long time user of Cucumber, but always wished it was Spinach since day one. I'm switching all my projects to Spinach despite its shortcomings because it uses the new, hot-off-the-block PORO technique (Plain Old Ruby Objects ;). Now I can expand my steps however I want, because it's just Ruby.
To answer your question, as of this writing:
Support across the board.
Cucumber
Spinach is still developing some features, including 'Background' blocks, and I'm currently trying to get it to recognize tables.
Flexibility of use
Spinach
Cucumber encourages bad step design from the start, IMO. If you create feature-specific steps, you'll trip over them later, and if you create reusable global steps, your feature definitions will be long, generic, and boring to read. I've heard people claim they can walk a balance successfully and be just specific enough but still have reusable steps; I consider myself well versed enough that if I can't do it reliably, it's too hard.
Third party tool and APIs integration
Cucumber, assuming the bullet point could be interpreted as community.
If it's really "third party tools and API integration" you're after, Spinach supports capybara and rspec, which is most of what you're after. Cucumber has 3rd party reusable step libraries, but as noted in my earlier point, I think this is bad. In regard to 3rd party & integrations, even if it's not there yet, your really can't get any better than plain old ruby objects.
Where does capybara fit into the picture
Capybara is your test interface to your site, aka a testing mouse & keyboard. You could start it up in a console and drive your app, but that'd get repetitive. Cucumber/Spinach (or rspec/test-unit/minitest) all could use capybara to automate testing your app. People prefer Cucumber/Spinach because they help you step out of the code for a bit to think like a user.
Overall, you'd probably be best off getting an rspec/cucumber book and doing what it says. Just be aware that testing takes a while to get good at, so don't stop there. Maybe check out Spinach somewhere in the process; if you like Cucumber, you might find you'll really like Spinach.

DISCLAIMER: I'm a Spinach mantainer.
If you're starting with BDD I'd highly recommend two books:
The RSpec book
The Cucumber book
I think it's important to learn all the BDD and TDD process (outside-in etc..) and then choose the tool you feel more comfortable with.
Having said that, Cucumber has a huge community, but a lot of things are also aplicable to Spinach, since what they have in common is Gherkin.
As for flexibility of use I would say both are really flexible, but I (obviously) prefer Spinach as every feature it's just a Ruby class, where you can include modules, inherit from other classes and so on (this also applies to APIs integration).
I you want, you can take a look at the spinach-rails-demo and see how everything works.

If you're stuck with Cucumber and you don't want global steps, you can work around the problem by tagging the steps with some sort of scenario ID:
# features/1_greetings.feature
Scenario: Formal greeting
Given I have an empty array [#1]
And I append my first name and my last name to it [#1]
When I pass it to my super-duper method [#1]
Then the output should contain a formal greeting [#1]
The #1 scenario id can be any value. I like to use ticket numbers for future reference.
You can then place all the steps in one step definition file. It's close enough to the look of Spinach::FeatureSteps. No regex arguments too!
# features/step_definitions/1_greetings.rb
Given 'I have an empty array [#1]' do
#...
end
And 'I append my first name and my last name to it [#1]' do
#...
end
When 'I pass it to my super-duper method [#1]' do
#...
end
Then 'the output should contain a formal greeting [#1]' do
#...
end
I posted more about the workaround at github.

I can't really speak for Spinach, as I've never used it, but Cucumber definitely has a huge community support with loads of external libraries.
Capybara allows you to easily test web applications
When I fill in "username" with "foo"
And I click on "login"
Then I should see "enter your password"

Related

What is difference between BDD and specifications by example?

I know BDD and also read Specification by Example the great book by Gojko Adzic; As I understand they are almost the same and have many in common; But I was not able to understand their main difference. I mean Can we use them interchangeably?
General idea
BDD is the format of describing tests in a form of "this functionality does this and that". If requirements are written in the same format, then we end up with the whole team using BDD. Because these requirements can be reused for testing.
Specs by example is.. well, putting a particular example into the requirements.
Testing formats
BDD-for-testing can come in different forms:
Given When Then
Gherkin (a more strict dialect of G/W/T used by Cucumber family of frameworks)
Describe-it (Jasmine, Mocha, etc)
Or something custom. Could be same old JUnit with tests named as "adminHasAccessToProfileOfOthers" instead of older style "testAdminCanAccessProfileOfOthers"
Requirements
As for BDD-for-requirements, here's an example:
If user has sufficient amounts of money, they can buy the product
And here is Specs-by-example:
If user has 10$ and the product is 9$, user can buy that product
See, in the 1st example we used abstract ideas, while in the latter one we used particular values.
Oftentimes when people talk about BDD/Specs-by-example, they would write requirements using Given-When-Then. But as an abstract idea this isn't compulsory - it's just what most testing frameworks support it.
Overall, BDD is a very good idea for testing, especially if it's not a Given-When-Then :) But when it comes to requirements - I haven't seen any project in my experience that really benefited from it.
As for other common misconceptions: BDD and how it does NOT relate to TDD.

Why did the examples of specflow always Uses the UI

I am newbie to BDD trying to understand it.. My understanding about BDD was ..
"Its a Kind of test where the user specfications are used to generate the Ubquitious language from the business"
But the examples i am able to see only the example of UI.. Like when the button is pressed .. when the user enters the text ... this wont form a language which i can use in my code..
Am i wrong in understanding this concept
BDD (Behavior Driven Design) was a term coined by Dan North and the best source to understand his intent is this excellent blog post
Here you can read that Dan want to shift focus from testing details to describe behavior instead. Of course this can (and sure has been) interpreted in oh so many ways :). So where ever you'll turn to you'll get an opinionated view - here is mine.
The idea with Cucumber-based tools, like SpecFlow, is to write down the teams shared understanding of a feature in a language and tool that all involved can read and understand. This is done (again in Cucumber-based tools) by writing down a couple of scenarios or examples on how the feature can be used. Some people call this specfication by example.
Some goodness comes out of writing the specifications by the use of examples in this way:
you can discuss the behavior of the feature before it's implemented
the specifications gives you a great specification to code after, using the outside-in approach
the specifications over time becomes regression tests that verifies that the system behaves as specified
the specification also comes through on a old TDD promise and becomes a living documentation for the system. You can easily see what the current state of an feature is by looking at the executable specification that the feature has passed
So now finally to your question (which by the way is a great one that I often have asked myself and others). Excuse me for rephrasing it, I hope I catch your intent:
Do my scenarios have to (or should they) be run against the UI?
You sure don't have to run the scenarios against the UI - the principles of BDD and the tooling works great going against your domain in any layer.
But to get the most out of your specifications you should consider my (inconclusive) list above. If you don't include the GUI (or the database, or services etc.) then you cannot be sure that the whole application stack works correctly together. So very often the specifications are run End-to-end.
And this makes these "test" something very different than your unit-tests (which you want fast as lightning, mocking out external dependencies, not hitting the database etc.). They take longer time to execute, all of them should not be run on every check-in, the don't use mock etc.
Often you start out with a step of a scenario and as a driver for a behavior and then use ordinary TDD to drive out the details of the internal of the system. This is outside-in programming.
Finally to your example above. So I recommend you to run your specifications against the UI end-to-end all the way to the database; but I would advise describing the UI in technical terms as above (using buttons, links and textboxes for example). When I asked this question on the BDD Google Group I got a great tip from Elisabeth Keogh:
Don't describe the UI. Describe instead what you're trying to achieve with the UI.
So to describe a login feature don't write:
Scenario: Login (describing the UI)
Given I am on the Login-page
When I enter 'AUser' in the textbox 'UserName'
And I enter 'APassword' in the textbox 'Password'
And I click the 'Login' button
Then I should see the following text 'You are logged in'
rather write it something like this:
Scenario: Login (describing what we want to achieve)
Given I am not logged in
When I log in using 'AUser' and 'APassword'
Then I should be logged in
That leaves the complexity on how this is done (click buttons, filling out forms, checking messages etc.) is done in the step definitions that you write in code.
I hope this was helpful. Also I am bracing myself for some "bashing" that can come from other, more experienced BDD-people. But hey, this is my two cents :)
I'm still pretty new to BDD and Specflow too. We've been using it to establish behaviour and code out the controllers, which in turn drive out the views. I don't have a code example in front of me, but I'll try to find something to post later. Cheers!
edit - BTW, if you're looking for a good book on using Cucumber (it uses the same language as Specflow - Gherkin? I'm still getting all the pieces straight), I can highly recommend The RSpec Book from Pragmatic Programmers. The code is Ruby based, but there are some higher level chapters on project definition and running. Well worth the price.

Should I be using RSpec or Cucumber

I'm new to unit/func testing in Rails 3. So I'm starting now, better late than never.
I have a method in /lib/mailingjob.rb called find_reply(body)
Right now I'm using cucumber to test this but given that this is all backend, no web interface to this, should I not be using cucumber?
I'm finding when to use RPSEC vs cucumber confusing.
Thanks
In my opinion, you need to use both. Rspec is very good for unit testing, that is testing models, controllers, views. On the other hand, cucumber is a very nice tool to check full scenarios like a user logs in, clicks a link and he is supposed to view this.
I highly advise that you take a look at the cucumber railscast from railscasts.com. Also, make sure that you use webrat and maybe something to auto load your specs like watchr(which i prefer).
I'd advise you to never use Cucumber unless you are specifically employed as a business analyst. That makes your life much easier: all your tests will be in Rspec.
Cucumber takes a toll on your productivity, and your sanity. Running parallel test environments, adding layers of abstraction and breaking your editor with its odd syntax make for lost time. I've written about this extensively on my blog:
Why Bother With Cucumber Testing?
If I didn't already have a Stack Overflow account, I would have made one JUST to write this post. I urge you to re-read #Jack Kinsella's answer
I got in to BDD about 3 months ago, fell in love with it, and quickly started using Cucumber for everything. I forced it in to every project, no matter what
Recently, I've had to learn a yet another language, and my first step was to get the testing environment set up. I came across these 3 articles:
http://robots.thoughtbot.com/post/25650434584/writing-better-cucumber-scenarios-or-why-were
http://robots.thoughtbot.com/post/25653308407/describe-the-users-perspective-ddd-acceptance
http://37signals.com/svn/posts/3159-testing-like-the-tsa
Honestly, their content isn't that important; what's important is that they hinted at an overall idea. I began to suspect that I'd been using Cucumber wrong all along, and I went hunting for answers. This morning I found your SO question, and #Jack Kinsella's blog post
#Jack comes right out and says the idea I believe those articles were tentatively circling. He also gives me the language I was looking for. I now consider his article to be the final word on the subject :)
According to him, what we've ACTUALLY been using Cucumber for is "integration testing". I never understood what that meant before
Unit testing:
How the code works internally. Math.Add(1,1) should be 2, but the website user doesn't care. Just give me a web page!
-> Use RSpec or an equivalent
Integration testing:
How different branches of the code work together to make the website. I type my name and click Log In, and should be taken to the Home page
-> ALSO use RSpec!
(Inside of RSpec, add whatever you need to handle multiple-technologies-touching-eachother. For the code-to-webBrowsers example: Capybara, Watir, etc)
Acceptance testing:
Many of us don't need it. Someone signed a contract with you saying "I'm going to write 'I can add subpages to my micro-site.' in a text file". You write me code that makes it turn green
-> Use Cucumber. Only if you have to do this kind of testing. Which you almost certainly don't
What an elegant solution. No 2nd testing environment. No steps directory and swarm of extra files!
I had a moment of "But I love how Cucumber seperates English from the code". Regular bdd does that too. "rspec --format nested" or Jasmine test results
#Jack is right. Cucumber isn't adding anything; not the way we've been using it. And it's costing a lot. To quote him:
Why not admit to yourself that you don’t do acceptance testing and that you do not need it in your projects? Swap Cucumber for pure integration tests using Capybara, and you’ll be surprised by how much more productive you can be
To complete SpyrosP's answer, there is an awesome blog post from Sarah Mei which describes a scenario where you use both Rspec and Cucumber. It is called oustide-in Behavior Driven Development and you can find it here.
However, I think most of Jack's problem can be solved by a good cucumber editor. So the problem is not Cucumber. The problem is that there wasn't a capable editor when he wrote this two years ago. But today RubyMine has full cucumber editing capability. While it is not free, it can solve most of Jack's problem: Auto completing, pointing out errors, refactoring. So you probably want to go back and check the points he raised, and see if they are still valid today.
My take so this issue many years later is very simple: If you do not know what to use now, just start with RSpec, because it is the central stone for all your testing that you will always need.
With time you will discover that two big testing needs:
Is my code still working?
Can my users still perform the actions already implemented?
As a developer I tend to think that (1) is for every single code change and (2) when I finish a feature and/or deploy.
Having two test suites is nice because during a deployment, you can sit and run a very generic test suite: Are my user features still working? Can I finally deploy this?
Indeed, it should be so generic that if something breaks in (2) you need to do some heavy research (one row missing in your tables) while a fail in (1) should pin-point the exact class with the problem.
Normally I find myself in the situation where I want separate integration+acceptance testing without the Cucumber burden. First I use RSpec tags to filter out these tests. Whenever I need more I like going for Steak+Capybara since it enables Cucumber like objectives on top of something I already know quite well.
Definitely use both. Use cucumber for user experience tests(views), use rspec for testing internals(models, etc.). You may write controller rspec tests but I think they are not necessary(try to move the logic to models).
Whatever your system is, I strongly advise you to use Cucumber because scenarios will let you and your stakeholders know the functions of your system in a clear written fashion.
You will also have your documentation ready and you'll always know where you are and where you are heading. You'll also relate scenarios with support issues in the future.
Cuke them right by the way don't use variables in scenarios, they should be understandable.
And if you want them to run fast as rspecs use poltergeist(phantomjs).
I've been tempted not use cucumber because in many cases it's not simple as rspec. However every time I understood that I needed scenarios to take full control on the project.

Which testing method to go with? [Rails]

I am starting a new project for a client today.
I have done some rails projects before but never bothered writing tests for them.
I'd like to change that starting with this new project.
I am aware there are several testing tools, but am a bit confused as to which I should be using.
I heard of RSpec, Mocha, Webrat, and Cucumber.
Please keep in mind I never really wrote any regular tests, so my knowledge of testing in general is quite limited.
How would you suggest I get started?
Thanks!
Thank you for all the responses! I posted a related question that may interest those who view this question in the future. You can find it here.
I prefer using Rspec and Cucumber in conjunction. I also prefer Capybara over Webrat.
Rspec is wonderful for it's rails integration and structure. It's technically behavior testing, but in practice it ends up feeling like unit testing. It also allows you to group and define "specs", or rspec tests, in any structure you like with 'describe' blocks.
Cucumber is higher level. It is what you will translate your user stories into. For instance, you may decide that your users should be able to change their password, so you'll write something like this:
Scenario: Change password
Given I am logged in
And I am on the change password page
When I fill in "p4ssw0rd" for "old_password"
And I fill in "newp4ssw0rd$$" for "new_password"
Then my password should be "newp4ssw0rd$$"
If you are working with a QA dept, you can show your tests to them, and they'll instantly know the status of any given feature, and how the feature is supposed to work. Additionally, if you have enterprising testers, they can write tests themselves, even if they are not programmers in general.
They all serve different function and if you really care about the quality of your software, you would use them all.
RSpec is more for unit testing. Unit Testing is testing the smallest part, usually your models methods.
WebRat is more for functionality testing as end user would test it from a browser. You are actually testing the interaction and test whether it returns the correct view/data.
Cucumber is more like for behaviour driven testing or end-to-end testing. It is more to test the business logic according to many different scenarios or data.
Of course you could use all of them together in conjunction. For example, I can use Cucumber to do end-to-end testing and it would involve functional testing using WebRat and Rspec as well. Hopefully this will get the idea for you to start off.
You can have a look at the most popular testing frameworks here http://ruby-toolbox.com/
Cheers

How to learn/teach Gherkin for Cucumber

I'd like to enable the business analysts to be able to write all of their specs for features, scenarios and steps that is Cucumber friendly using Gherkin.
I've read some of the basic info on the github site for Cucumber and from doing a quick Google search but wanted to know if there were recommended resources for getting non-technical folks to be able to write comprehensive BDD using Gherkin (I assume that's the preferred language for Cucumber tests to be created in).
Thanks.
What I did with the business analysts in our company was to teach them the structure by giving them the keywords: Given, When, Then, And for Scenarios and In order to, As a and I want to for Features.
Then I gave them a simple example and told them to write down their own features as they thought they should be written. Surprisingly enough the structure was self explanatory and the features they wrote became a great start.
The only big problem was that they had contained to much logic in each scenario step. I solved that by iteratively asking "why?" which in most cases revealed the core functionality they were after and we re-wrote the scenarios accordantly.
By giving them the guidelines and letting them write the features themselves they got their hands dirty and were forced to think about what they wrote. Today they have a much better understanding and the "why?" iterations are not that common anymore.
Ofcourse you need to have the business analysts and the developers to work closely together and the features the analysts write should only act as a start. Remember that the Cucumber features are just a common language between the analysts and the developers. They still need to sit together often to be able to speak with each other :)
http://cukes.info is a great resource for teaching people how to write them. Ben Mabey also made a great presentation on Cucumber at Mountain West Ruby Conference 2009.
Having just worked on an agile project using cucumber for the first time I think that the best way to learn Cucumber and Gherkin is to get your hands dirty.
I may be wrong but I get the impression from your question you are wanting to train your BAs to write Gherkin; then they will write a bunch of features and hand them to developers.
This is definitely not the way to go. It is much better to have BA's devs and users (if possible) working together to write your scenarios and build them as you go. Then you all learn together what works and what doesn't.
We tried having a BA write entire features and hand them over. We (the devs) ended up having to do major rewrites because the implementation ended up different to that originally envisioned by the BA. We also had to change the syntax of the steps and do find and replace through the whole file.
Do one scenario at a time, get it working then move on to the next. An iterative approach reduces wasted effort and makes sure you all understand how you want the app to behave.
In terms of how to write steps it is best to start with the ones that come with Cucumber and copy and adapt them as you work on your project to fit your particular application. There is no right or wrong, it is what works for you. The documentation on the cucumber sites is generally good and will be a valuable resource as you learn more.
We are teaching Gherkin (for SpecFlow) in a similar way, how mrD has described it.
I think it is very important though, that the audience is familiar with the main intention of "Specification by Example", agile requirement analysis and BDD, so we usually start discussing the background first. We also show a sample Gherkin scenario and explain the very basics (like Given/When/Then/But and tables).
Than we take a simple example story (that is quite familiar to everyone), like "add items to shopping cart" (with some orientation, of course) and let them formulate the acceptance criteria in small groups.
After that every team shows / explains their solutions and we discuss the good and bad practices that were present. After the second team, you can see almost all of the most important (good or bad) practices appearing.
I also type in the concluded solution, and show here alternative ways of describing the scenarios (background, scenario outline, etc.). If there is enough time, I also show how to automate & implement the imagined functionality based on that. This also help to understand some important rules to follow, that makes the automation much easier.
Although, I never know upfront what will happen, usually this exercise is the best part of our BDD training.
The RSpec book has a couple of chapters in it that are relevant to Business Analysts:
http://pragprog.com/book/achbd/the-rspec-book
I think the best way to learn is to start writing. Gherkin & Cucumber are easy to learn but difficult to master, so it’s important to get to practical examples as soon as possible.
While it’s important to get started by writing your first scenarios, you also need some resources to establish good habits and understand key practices. I wrote a book that could help. “Writing Great Specifications” is, I hope, a good way to learn Gherkin and Cucumber. It covers patterns and antipatterns as well as key techniques for writing great scenarios. :) If you have any questions, you can always hit me up on Twitter.
If you are interested in buying “Writing Great Specifications,” you can save 39% with the promo code 39nicieja2 :)
Other great resources:
“Specification by Example” by Gojko Adzic if you’re interested in software development processes and high-level engineering practices.
“BDD in Action” by John Smart if you don’t mind reading testing code in Java. It’s a comprehensive end-to-end view on defining and testing software requirements.
“Behaviour-Driven Development” by Liz Keogh if automated testing doesn’t ring a bell, but you want to understand how specifications with examples affect your business analysis processes.
“The Cucumber Book: Behaviour-Driven Development for Testers and Developers” by Matt Wynne and Aslak Hellesøy
“The RSpec Book: Behaviour-Driven Development with RSpec, Cucumber, and Friends” by David Chelimsky, Dave Astels, Zach Dennis, Aslak Hellesøy, Bryan Helmkamp, Dan North
Best way to learn Gherkin is to read the Behat documents first: http://behat.readthedocs.org
Then read official documents from cucumber site: https://cucumber.io/docs/reference
Also, you can read Guru99 article to write your first cucumber script: http://www.guru99.com/your-first-cucumber-script.html

Resources