For testing purposes for Ruby/Rails I use RSpec and I have well over 1000 tests that are already written in my application. Once the app goes live I want to have the tests run once a day on the production server (since all the code is there) and see if anything fails. The test itself will be run against test data and now production data. This is an effort so that any new code that is added in the future will not cause anything to unknowingly break.
I have found a few solutions:
Travis-CI (only open-source ... not suitable for closed-source projects)
Jenkins-CI (not sure if it works with or well with Ruby/RSpec/Rails)
Watir (not sure if Ruby/Rspec works with it, but the tool itself is written in Ruby).
Preferably something that checks the codebase daily and then emails me when something isn't working.
I also plan on integrating JavaScript testing with the a testing library of my choice (I just need the automation platform for testing it).
Can someone provide me some insight as to which tool to use? Or does anyone have any other tools to recommend?
Jenkins-CI works great with rspec, and can run your jasmine javascript testing, and your cucumber javascript tests as well.
The only thing I'd recommend is not to test on your production server itself. When you push changes to your source-control repository, Jenkins will download the new code and run your tests there. When you're green (tests pass), push the code to production.
Related
I'm a PHP dev mostly and just starting out with Ruby, so please correct me if I say something dumb.
I'm working on fixing some bugs in a "legacy" app written in rails. The app itself has never been unit-tested. I can see the test scaffolding generated by rails but tests are nowhere to be found.
The app is pretty big and the code quality is very bad, so I wanted to write some units tests for the functionality I'll be fixing, before writing any code.
The problem is that when I run rake test command, the testing DB is created, but if I write any tests it keeps crashing on me. There are several problems with some relations and keys which I tried to fix, but more problems just keep appearing. I do understand that the DB is created with schema.rb file, but I'm sure it is just outdated by now. It is another issue I will maybe fix, but for now I just want to write some basic unit tests not even using the DB itself.
So the question is: is it possible to write just unit tests for some methods without invoking all the test DB scripts? I'm aware that this is maybe not the best practice, but I will feel better modifying the app with some test coverage and starting with fixing the DB I do not yet understand seems like a bad idea to me.
I'm using Ruby version 2.1.10 and the app is written in Rails 4.0.4 - these seem to be the latest versions I managed to run the app on.
I have a Rails app that interacts with some Mac software and I need to write some tests for it. How on earth do I do that? Where do I even start?
The Rails app connects to the Mac app through AppleScript and Terminal. Any ideas?
Update
Found this gem to help with Shell expectations. Is that as good as I'm gonna get?
https://github.com/matthijsgroen/rspec-shell-expectations
Testing external dependencies can certainly be a challenge.
Remember that tests you write should test the behavior of the Rails app, not the external dependency. You should have integration tests to verify that the code actually works with the application, but they should be "smoke tests", not a full suite to test every feature.
Write unit tests to verify the behavior of the code that relies on the dependency, and mock out the interactions. Typically with command line apps that means:
the app wrote to STDOUT
the app wrote to STDERR
the app read from STDIN
the app exited with a particular status code
The gem you mentioned is a good start, but you may find it worthwhile to look at rolling your own helper code using Open3 from the Ruby standard library, which can be useful for all the items in the list above.
Use a tag on the specs that need to use the Mac application to make it easier to filter out those specs as necessary.
You may already be familiar with vcr for mocking out HTTP interactions; its "playback" feature is a good source of inspiration.
On edx.org there is Software as a Service course that grades all submitted assignments.
You upload a zip-file with a Rails project and they run a bunch of integration and unit tests.
How do they do it?
My thoughts are they mount an uploaded application as Rails engine. Is it possible to test one Rails app within another? I'd like to create a similar service, but I don't know what to start with.
I would imagine you could do this in a similar way that Jenkins runs continuous testing on a project. Where each project uploaded to your site gets expanded into a workspace, and then you shell out and execute commands. But that allows for a lot of variable configurations and complexity that probably doesn't make sense in the scope you've proposed. It also doesn't protect your underlying OS from the projects your testing.
You could also probably use an application container like docker and manage each uploaded application that way as well, which would keep everything self-contained, and isolate the application from the OS. It also puts the onus on the developer to package and manage dependencies correctly. I'm guessing they are probably using docker or something similar, here's an example Using Docker for MOOCS
At the point you want to capture the test results and report them back, I'd think they probably use something similar to the Junit formatter for Rspec or they just parse the rspec output directly.
I've noticed that when I deployed my project to Heroku that its able to find coding errors where I had forgotten to change calls to methods that I had refactored (ie changes to method name or signature). It just failed to deploy until the code has been fixed.
This seems to suggest that Heroku has some sort of preprocessor or interpreter built in that does a pseudo "compile" on the code before it allows it finish deployment. Is anyone else aware of this?
Is there a gem out there that will allow me to run a quick sanity check on my code?
Naturally, there needs to be specs / unit tests of course, but having another tool in the toolbox can't hurt.
I'd guess that it's more the differing behaviour of how a Rails application is loaded/started in production mode as opposed to development mode.
If you start you local application in production mode locally you should see the same behaviour as on Heroku.
We've recently upgraded our Rails application.
To be extra sure everything works, I've tried to get the tests and specs of the various used plugins (26 at current count) to work, thinking then to add those to our continuous integration, which only runs the main application's specs.
I've run into a lot of problems even getting the specs/tests to run at all, not even getting to any individual test failures. For example, I've run across this problem: http://rails_security.lighthouseapp.com/projects/15332/tickets/7-rake-spec-plugin-fails-on-rails-2-1 (thanks by the way for that ticket, even though the issue wasn't fixed).
So the question is: Are we unusual in that we've ever cared about running plugin tests ? It doesn't seem to feature much here on SO. My nagging feeling is that they should be run as much as the main specs, but you could also argue that since the main specs work, the plugins must also work.
Alot of it depends on the plugin/gem being used.
If I know the author/community of the gem is competant I will skip the tests and simply use the latest stable release and freeze that gem. I will then track the progress of the development using github.
If the plugin/gem is written by an unknown party I will run the tests and freeze the gem/plugin and again monitor the development.
Sometimes however I will write my own contributions to the gem and fork the code. I will clone the repo in github and base my installations from that. At which point any and all changes result in a complete test run.
With all things in the open source world there is an element of trust between the creator and the users of those pieces of code. The tests themselves don't tell me much about the codebase, it shows there are tests and thats it. Do they test everything ? Are there edge cases ? . Its this element of trust I have with certain developers in the community that means I forgo worrying over running tests for those gems.
Its a slippery slope testing everything, where does it stop ? Would you test rails every release ? No, you assume the community has done this for you already.