I am working on a Rails app that allows you to create a configuration and then launch a server at EC2 with this configuration.
So far I've been using cucumber for BDD and was very happy with that. However, now I want to pick a configuration and actually launch the server. Due to cost and performance issues I don't want to actually launch a server every time I am running the cucumber features.
Are there any best practices for cases like this? I would like to keep the BDD up, but also don't want to spend too much time on working on an elaborate solution just to have feature descriptions for this. On the other hand I will have the same problems, when I will have to write Unit test for this.
When working on a rails app that required twitter integration, I found fakeweb to be extremely helpful. I've used it in conjunction with cucumber successfully.
I found that to support the BDD outside-in development style, I set fakeweb to disallow all web traffic and then added my faked-out web calls one at a time, as my tests failed. It seemed to fit right in with my BDD workflow.
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
I have to preface all my posts about rails: I'm a novice.
Is it necessary to write tests for my app to work properly or is it strictly for finding breaks?
Testing your app is not necessary for it to work, but it is highly recommended and a very good practice. Testing your application will help you develop a better piece of software and a much more solid application. In the Rails world BDD(Behavior Driven Development) is very used as a testing and development technique.
I recommend you two diferent testing suites:
The first one is Rspec that will help you with all your controllers, and models unit testing
Cucumber is a testing suite that will test your application as a whole(integration test), this one is great for a more "real life" testing approach
I encourage you to check both Rspec and Cucumber, there are also other great testing suites like Test Unit.
Remember, testing your application will give you great benefits!
NOTE: Rspec and Cucumber are not mutually exclusive, actually they are recommended to use in conjunction
Writing tests is not necessary in the sense that your application won't operate without them, but they are not only for keeping bugs away. If you're a novice, writing the tests should also help you to understand how everything works.
There's no reason not to write tests. Just write good tests, and don't waste time testing things that don't need to be tested (like generated attr_accessors).
It's not necessary at all but its considered by almost every ruby developer I know to be standard procedure.
I did a couple "rails apps" without testing, but as soon as I needed some real backend logic in ruby, testing helped me to understand what I was doing.
No, writing tests is not required for your application to run. It is a good practice, though, so if you are not used to writing tests I'd recommend that you start learning. It's easy and it will save you a lot of headaches on whatever platform you are using.
Just to chime in with everyone else, I'd suggest you find yourself an article about BDD and test first development. And then read up on mocking and stubbing. Wrapping your head around the why and how of it will probably convince you it's worth the time and effort.
When I first dove into the XP & RoR world I live in now I was daunted by what felt to me like test-mania but it really pays off in spades.
The first time someone had me write the test first THEN write the code we were testing I was mind blown. But I've never gone back to my previous evil ways.
I'm trying to sift through the myriad of test solutions out there, and I'm not even sure if I'm headed the right direction. The story is: we're running a RESTful Web service, implemented as a Rails app, which backs our mobile clients. We're unit testing the Web service (of course), but that involves mocking out many parts of the application, for instance, the search stack (Apache SOLR).
Moreover, our tests don't (i.e. cannot!) cover critical routes such as the mobile sign-in/sign-on process, since that involves communication between the API application and the mobile website, where the user can enter credentials, e.g. for SSO (Janrain Engage). Hence, a standard Rails integration test won't do.
I realize that in theory, if the test suite is very well designed, where mocking only happens strictly at the those join points where the tests for the next layer start, then by unit or functional testing the service API and the mobile website separately, one could get the same test coverage. I find that in practice though, this is an illusion if you have several developers working on the test suite independently; I just admit that our unit tests are simply not that well designed. Particularly when exercising TDD I found that while the tests can drive the application code, the test code design is only tailored to the unit under test, resulting in a rather wildly grown test suite.
Another thing I found is that sometimes we didn't detect regressions purely using unit tests, where e.g. bad queries were sent to the SOLR server due to knock-on effects. That's why I thought the only true way to ensure that the entire stack works along the critical routes is to automatically end-to-end test it on a staging server before every deployment, i.e. having actual HTTP requests sent to the app.
My questions would be:
do you think this is a reasonable thing to do at all? I found very little information about end-to-end testing a live API on the Web, leaving me wondering whether I'm making any sense at all
which tools/setup would you suggest? We use Watir to run acceptance tests for our website, but it seems to be overkill for a Web service (no browser environment needed, no JS or anything UI-ish). Something as simple as a Ruby script even?
any general best practices or advice you can give me w.r.t. to designing such tests?
This might be of interest to you: http://groups.google.com/group/ruby-capybara/browse_thread/thread/5c27bf866eb7fad3
What you might want to to try is combining Cucumber (or similar) with one of the tools mentioned in the link above, so you can do something like
Given I have 2 posts
When I send "DELETE" to post with id 1
Then I should have 1 post
This way you can test the API's full stack and check the result.
We've faced similar problems lately, like testing only on the unit level was fast but not enough to spot integration related anomalies.
I really like Cucumber's gherkin syntax, the value for me in it is that we can run the exercise once and then make several expectations after with separate titles, like if they would be separate RSpec it blocks.
There is a gem called Turnip, which makes possible to write RSpec tests with the gherkin syntax. We have been using it for a while and I'm quite happy with it, we even started having less integration level defects.
I collected my experience regarding in a blog post: https://blog.kalina.tech/2019/10/end-to-end-testing-for-api-only-rails-apps-with-cucumber-syntax.html
I would like to write some automatic tests for my Rails 3 application.
I wonder how to start with that.
I've heard about Selenium/RSpec/Cucumber, and I guess there are many more options.
What are the advantages/disadvantages of these testing frameworks ? Which of them has the best documentation ? Which one is the most popular in the Ruby world ? And in the industry at all ?
I have the general knowledge of how to write tests. I just want to learn the appropriate testing framework(s) for testing Rails applications.
Please help me to decide with which testing framework to start.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
I wouldn't say there is any one best set of testing tools out there; and the community definitely has not decided on any being standard in any way.
I highly recommend The RSpec Book from the Pragmatic Programmers-- about half of the book is on testing with Rails with Cucumber, RSpec, and browser simulators like Selenium. It's a comprehensive overview of the different situations you will find yourself in with Rails and what tools you might want to use in each situation.
Selenium is a framework to automate testing of user interface. Selenium script launches a browser window, goes to a web-page and manipulates page elements.
RSpec and Cucumber are tools for what is called "behavior driven development (BDD)". BDD is a development process where you have description of a separate feature, and some script (can be a Selenium script) that checks this feature. RSpec and Cucumber basically connect these two, so that when you run a test script, you see the list of features written in plain language and their testing status.
Selenium scripts can be written in a number of languages, while Cucumber scripts use Ruby. Documentation is OK for all of them, some knowledge of programming language is required though.
Starting links:
Cucumber;
Selenium;
Testing rails applications
The easiest way to get started is using TestUnit. When you generate a model, view, controller using scaffolding it will automatically generate a testing folder for you containing a test environment setup file and tests, or you can add tests to existing models with a rails generator.
It is then a matter of reading the documentation on TestUnit, of which there is a fair amount.
For a next step, my personal preference is to use RSpec over Cucumber - I've found Cucumber hard to maintain, and whilst it is wonderful to have tests that are readable by your boss, I've found that most bosses don't really want to read or contribute to tests, and it makes for time-consuming development and is very 'bitty' - lots of small bits of text scattered in various files.
If you want to jump right in and begin using RSpec for Rails, then I'd recommend Rspec-Rails, which includes many helpers for asserting the correct behaviour of your application.
I could go on, but you've asked for 'where do I start' - there's two good places.
There are a bunch of frameworks that help you test your ruby/rails code... the great thing is the dynamic nature of Ruby gives you a lot of flexibility.
I like to use RSpec to test the internals of my code, and Cucumber to test the application's behavior. So generally, RSpec will test models and methods, and Cucumber tests user interaction through the browser.
I highly recommend this pragmatic programmers ebook on BDD using RSpec on Cucumber.
Railscast on getting started in cucumber (also called cukes)
Cucumber project
Rails project
I'm currently writing integration tests for a RoR REST API using cucumber.
I'd also like to stress-test the API and was wondering how I might be able to re-use my cucumber scenarios to do this.
I've used JMeter and httperf before for stress-tests, and of course with both tools had to write or record the tests that I wanted to include in the stress testing. Since I already have the scenarios in Cucumber that I want to test, I'm trying to avoid having to rewrite what would be largely the same scenarios in the stress-testing tool's syntax.
Can anybody recommend a way that I could use my cuke tests for this? Ideally, I'd like to simulate hundreds of concurrent users hitting the API.
You very well may have to create your own load harness. Essentially, you'll need to start up a number of threads, each calling the process to start your scenarios. You will want to have a delay to stagger when each thread starts because there will be some overhead involved with loading the Ruby interpreter. You may also have to spread the load across more than one machine for the same reasons. It would probably be best to do this in separate processes. Perhaps this might be good for a new project?
Hmm, what about creating some steps that perform a system call to httpref ?