I'm currently writing integration tests for a RoR REST API using cucumber.
I'd also like to stress-test the API and was wondering how I might be able to re-use my cucumber scenarios to do this.
I've used JMeter and httperf before for stress-tests, and of course with both tools had to write or record the tests that I wanted to include in the stress testing. Since I already have the scenarios in Cucumber that I want to test, I'm trying to avoid having to rewrite what would be largely the same scenarios in the stress-testing tool's syntax.
Can anybody recommend a way that I could use my cuke tests for this? Ideally, I'd like to simulate hundreds of concurrent users hitting the API.
You very well may have to create your own load harness. Essentially, you'll need to start up a number of threads, each calling the process to start your scenarios. You will want to have a delay to stagger when each thread starts because there will be some overhead involved with loading the Ruby interpreter. You may also have to spread the load across more than one machine for the same reasons. It would probably be best to do this in separate processes. Perhaps this might be good for a new project?
Hmm, what about creating some steps that perform a system call to httpref ?
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
I am writing an MVC2 asp.net web app. I would like to have some performance tests within the test project, where I emulate hundreds of users all accessing the database layer at the same time. Is there a way that I can write a test that will make lots of concurrent calls to different controllers? Do I need to spawn lots of threads to make the multiuser aspect truly concurrent?
I would recommend you using a specific tool for performing load tests instead of writing this logic manually by spawning new threads. For example if you have the full version of Visual Studio you could create web tests which represent different user scenarios and then execute those scenarios in a load test. Another useful tool is Apache JMeter.
Yes you need multiple threads to do a full integration testing simulating concurrent user behavior.
You can either create spawn threads yourself or use something Parallel.Foreach() for a slightly more convenient way.
However these shouldn't be part of your unit test suite because they're a different kind of test than unit tests. Best to not mix them with your regular unit tests.
There is an existing project that is already deployed in production. We want to add some tests on it (the sooner the better) and I have to choose between going the BDD way (rspec/cucumber) or the TDD way (TestUnit). I am really starting with BDD and I am wondering what could be the best decision to take? I am affraid that using rspec/cucumber on an existing rails project (which was deployed this week and requires really fast iterations) will be quite hard to do (especially that it is not supposed to be used this way, I mean we are supposed to write stories/features first and iterate from there).
TestUnit could be more reasonable, may be.
Do you have any thoughts on that? An experience to share? Some advices?
I believe the easiest way to get coverage for an existing application is to use cucumber. This will allow to describe and document how to the website/application should work (and will keep working).
Because it works from the outside in, this also has the advantage you do not need to comprehend the inner workings completely yet. At the same time, you test all layers of the application (model-view-controller) in one test.
When you start actually changing code, then I would start adding the unit-tests for the code you are changing, using your favourite testing framework. I personally favour rspec, but as you know this is a personal choice :)
I'm trying to sift through the myriad of test solutions out there, and I'm not even sure if I'm headed the right direction. The story is: we're running a RESTful Web service, implemented as a Rails app, which backs our mobile clients. We're unit testing the Web service (of course), but that involves mocking out many parts of the application, for instance, the search stack (Apache SOLR).
Moreover, our tests don't (i.e. cannot!) cover critical routes such as the mobile sign-in/sign-on process, since that involves communication between the API application and the mobile website, where the user can enter credentials, e.g. for SSO (Janrain Engage). Hence, a standard Rails integration test won't do.
I realize that in theory, if the test suite is very well designed, where mocking only happens strictly at the those join points where the tests for the next layer start, then by unit or functional testing the service API and the mobile website separately, one could get the same test coverage. I find that in practice though, this is an illusion if you have several developers working on the test suite independently; I just admit that our unit tests are simply not that well designed. Particularly when exercising TDD I found that while the tests can drive the application code, the test code design is only tailored to the unit under test, resulting in a rather wildly grown test suite.
Another thing I found is that sometimes we didn't detect regressions purely using unit tests, where e.g. bad queries were sent to the SOLR server due to knock-on effects. That's why I thought the only true way to ensure that the entire stack works along the critical routes is to automatically end-to-end test it on a staging server before every deployment, i.e. having actual HTTP requests sent to the app.
My questions would be:
do you think this is a reasonable thing to do at all? I found very little information about end-to-end testing a live API on the Web, leaving me wondering whether I'm making any sense at all
which tools/setup would you suggest? We use Watir to run acceptance tests for our website, but it seems to be overkill for a Web service (no browser environment needed, no JS or anything UI-ish). Something as simple as a Ruby script even?
any general best practices or advice you can give me w.r.t. to designing such tests?
This might be of interest to you: http://groups.google.com/group/ruby-capybara/browse_thread/thread/5c27bf866eb7fad3
What you might want to to try is combining Cucumber (or similar) with one of the tools mentioned in the link above, so you can do something like
Given I have 2 posts
When I send "DELETE" to post with id 1
Then I should have 1 post
This way you can test the API's full stack and check the result.
We've faced similar problems lately, like testing only on the unit level was fast but not enough to spot integration related anomalies.
I really like Cucumber's gherkin syntax, the value for me in it is that we can run the exercise once and then make several expectations after with separate titles, like if they would be separate RSpec it blocks.
There is a gem called Turnip, which makes possible to write RSpec tests with the gherkin syntax. We have been using it for a while and I'm quite happy with it, we even started having less integration level defects.
I collected my experience regarding in a blog post: https://blog.kalina.tech/2019/10/end-to-end-testing-for-api-only-rails-apps-with-cucumber-syntax.html
I am working on a Rails app that allows you to create a configuration and then launch a server at EC2 with this configuration.
So far I've been using cucumber for BDD and was very happy with that. However, now I want to pick a configuration and actually launch the server. Due to cost and performance issues I don't want to actually launch a server every time I am running the cucumber features.
Are there any best practices for cases like this? I would like to keep the BDD up, but also don't want to spend too much time on working on an elaborate solution just to have feature descriptions for this. On the other hand I will have the same problems, when I will have to write Unit test for this.
When working on a rails app that required twitter integration, I found fakeweb to be extremely helpful. I've used it in conjunction with cucumber successfully.
I found that to support the BDD outside-in development style, I set fakeweb to disallow all web traffic and then added my faked-out web calls one at a time, as my tests failed. It seemed to fit right in with my BDD workflow.