I want to use SpecFlow to test a console app. I want SpecFlow to start the console app and interact with it in exactly the same manner a user would via standard in/out.
Is this possible?
Yes, in fact I have one such solution open on my machine right now. My advice is "Don't do it!".
I open the solution and nCrunch (which is simply a hyper efficient test runner) starts up some tests for a scheduling system we use to co-ordinate multiple servers. Some of these tests check timescales and ensure that processes start and stop as they are supposed to. You can tell this because everytime I try and type something a calc.exe window pops up to steal my focus, and it really gets in the way. Is that what you really want to do?
Don't forget SpecFlow is really a business requirements automation system, and this is "a good thing". But so far you've only focused on the technical issues here.
I'd suggest that you think about your requirements again. Where you want to start a process and check it interacts with you, then simply test its arguments ande results. Use mock's if you need to isolate its functionality (like I didn't do when I wrote my tests, oops). Try and make your tests as simple as possible.
Think of it this way.
- Do you really need to test that Process.Start opens a Window? Surely MS got that right? :-)
- Do you really need to test that Console.ReadLine gets a string?
- And won't your tests will be simpler if you seperately test MyArgumentParser and MyBusinessLogic with mocks splitting them up?
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
We have implemented SSO using rubycas server, we want to do load test on this and we are looking for open source tools which are the good for load testing.
There is pretty good set of open source tools for the load tests. You can check some of them here.
Is not easy to choose proper tool. All depends on the specific case what kind of staff you would like to test. If you would like to do simple throughput testing or just measure the responsiveness of you system, I would recommend one of these:
Apache Bench
httperf
but if you would like to test some more advance "user stories" then I will suggest:
JMeter
Trample
There is an existing project that is already deployed in production. We want to add some tests on it (the sooner the better) and I have to choose between going the BDD way (rspec/cucumber) or the TDD way (TestUnit). I am really starting with BDD and I am wondering what could be the best decision to take? I am affraid that using rspec/cucumber on an existing rails project (which was deployed this week and requires really fast iterations) will be quite hard to do (especially that it is not supposed to be used this way, I mean we are supposed to write stories/features first and iterate from there).
TestUnit could be more reasonable, may be.
Do you have any thoughts on that? An experience to share? Some advices?
I believe the easiest way to get coverage for an existing application is to use cucumber. This will allow to describe and document how to the website/application should work (and will keep working).
Because it works from the outside in, this also has the advantage you do not need to comprehend the inner workings completely yet. At the same time, you test all layers of the application (model-view-controller) in one test.
When you start actually changing code, then I would start adding the unit-tests for the code you are changing, using your favourite testing framework. I personally favour rspec, but as you know this is a personal choice :)
I'm currently writing integration tests for a RoR REST API using cucumber.
I'd also like to stress-test the API and was wondering how I might be able to re-use my cucumber scenarios to do this.
I've used JMeter and httperf before for stress-tests, and of course with both tools had to write or record the tests that I wanted to include in the stress testing. Since I already have the scenarios in Cucumber that I want to test, I'm trying to avoid having to rewrite what would be largely the same scenarios in the stress-testing tool's syntax.
Can anybody recommend a way that I could use my cuke tests for this? Ideally, I'd like to simulate hundreds of concurrent users hitting the API.
You very well may have to create your own load harness. Essentially, you'll need to start up a number of threads, each calling the process to start your scenarios. You will want to have a delay to stagger when each thread starts because there will be some overhead involved with loading the Ruby interpreter. You may also have to spread the load across more than one machine for the same reasons. It would probably be best to do this in separate processes. Perhaps this might be good for a new project?
Hmm, what about creating some steps that perform a system call to httpref ?