As title says, my travis-ci instance is telling me that my build is failing when I have cucumber step definitions that are pending (not undefined) - is there a way to stop this behaviour?
I can understand the reason why, but the undefined behaviour I have is undefined by purpose - I have no intention on doing that feature RIGHT NOW, but I know I want to keep the step definition down so I don't forget what I came up with. The reason I ask is because this behaviour (with it failing when I have a pending task) could end up masking real failures that actually matter.
There is a similar answer here which should be useful for you, the answer that he gives is:
I think cucumber gives non-zero process exit code either because of
skipped or because of pending tests. Try to get it to not run any
skipped, then any pending, then any skipped or pending tests and see
what exit codes it gives. To see the exit code (in Unix), run it with
something like:
cucumber ...args to select tests... ; echo $?
So basically you want to figure out what args you can provide to cucumber to have it pass and then provide a .travis.yml that runs that command. an example rake task for cucumber jasmine and rspec is here:
task :travis do
["rspec spec", "rake jasmine:ci", "rake cucumber"].each do |cmd|
puts "Starting to run #{cmd}..."
system("export DISPLAY=:99.0 && bundle exec #{cmd}")
raise "#{cmd} failed!" unless $?.exitstatus == 0
end
end
And read the docs for more info on creating a .travis.yml.
If all of that doesn't work, just create a bash script that catches the output and returns 0 in the right cases, I really doubt this is required though since I'm sure there's an option to cucumber to make it ignore pending.
EDIT: the cucumber docs say:
Pending steps
When a Step Definition’s Proc invokes the #pending method, the step is
marked as yellow (as with undefined ones), reminding you that you have
work to do. If you use --strict this will cause Cucumber to exit with
1.
Which seems to imply that they will not exit with 0 if you don't call --strict
Related
I'm using whenever to schedule a task in Rails, and I would like to find a way to test the behavior being executed.
Is there a way to trigger the event in a test (like RSpec) so I can make assertions about the results? The executed task is a class method that by itself works because I've tested it manually in the Rails console, but is there a way to trigger the event so that this behavior happens and I can assert it works the same way within the config/schedule.rb?
I used a simple trick to assert my expectations like below
require 'spec_helper'
require 'whenever'
describe 'Schedule' do
it 'sends email 10th of each month before 08 AM EST' do
expected = "0 7 10 * * /bin/bash -l -c 'cd && script/rails runner -e production '\\''Delayed::Job.enqueue(MyMailerJob.new([2,3,4]), priority: 10)'"
expect(cron_output).to include(expected)
end
end
def cron_output
Whenever::JobList.new(file: Rails.root.join("config", "schedule.rb").to_s).
generate_cron_output.
gsub(Dir.pwd, '')
end
The "whenever event"
The only thing that changes in what whenever wraps (ie, cron) compared to your regular spec run is system environment, and more specifically environment variables.
You can see the commands whenever will install in cron using its executable :
$ whenever
Cron always start with a raw environment. If you check your crontab content (using crontab -e) you'll see on top variables that are set. You can set additional variables, there. For example :
SHELL="/bin/bash"
PATH="/sbin:/bin:/usr/sbin:/usr/bin"
MAILTO="your_user"
HOME="/home/your_user"
RAILS_ENV="production"
You should not need those, though, because whenever uses a nice trick : it calls all commands using /bin/bash -l -c. This ensures it uses a login shell, which loads ~/.bashrc file. So any variable that is exported in your bashrc file will be accessible within cron execution.
Also note some distributions (like gentoo) have a check in the skel bashrc file that quit if shell is not interactive (so you have to set your environment variables before it if they are needed in cron).
Tests with similar env
To answer about testing, this is probably not a good idea. There's nothing related to your codebase here, only related to your system (similarly, you don't test which version of postgres or imagemagick is installed on your system in your specs).
You can somewhat simulate, for debugging sake, what happens in cron while invoking rspec doing such :
env - HOME=/home/<your_user>/ /bin/bash -l -c 'rspec spec/'
This will ensure environment is emptied (env -), then set HOME as a minimal env, then call a login shell just like whenever commands do.
I frequently run "rake", see some tests fail, and have to manually cut-and-paste failures into a new command, "ruby SomeTest -n some_test_method" to run individual tests. Is there any way to automate that?
It feels like standard behaviour for IDEs to show errors and allow quick re-playing, so I wonder if anyone has figured out how to do it quickly on the command line.
Guard is what you're looking for:
https://github.com/guard/guard
Guard watches the filesystem for changes and automatically triggers a command. With guard you can automatically run tests the second they're saved.
For minitest use:
https://github.com/guard/guard-minitest
For test unit:
https://github.com/guard/guard-test
For Rspec:
https://github.com/guard/guard-rspec
We are using the mock out method for making system calls:
before do
subject.stub(:`) { `(exit 0)` }
end
...
#when running the tests we check the result of the last shell command using $?
if $?.exitstatus == 0
# some code
end
This works locally using jruby 1.7.9 on mac osx. But when the builders execute these same tests using jenkins CI we have an intermittent result of $?.exitstatus == 127. Since the stub points off to a subshell that should only return exit code 0 this is extremely bizarre.
My questions are:
What are the possible causes of this discrepancy?
Is there some difference between the linux variable $? and the ruby call to $?.exitstatus that could lead to getting inconsistent results?
We have since worked around the issue by implementing a stub on how we retrieve exitstatus so this is more a question to satisfy my curiosity.
We've just set up a Jenkins CI server for our app, HiringThing
Everything works well, with builds starting automatically when code is checked into our Github repo.
The problem is detecting build failures. I have the following rake task being run from the command line by Jenkins.
rake test:browser
runs the following
desc "Run browser tests."
task :browser => :environment do
start = Time.now()
puts "Stopping apache if running"
system 'sudo apache2ctl stop'
puts "Running selenium tests"
Dir.glob('./test/browser/*.rb').each { |r|
puts r
system 'rvmsudo ruby ' + r
}
system 'echo -e "\a"'
puts "All browser tests, elapsed: " + (Time.now() - start).to_s + " seconds"
end
I've confirmed that the tests are running properly (using headless Firefox with Xvfb) The problem is that Jenkins doesn't detect any build failures in the browser tests. It seems to work for the usual Rails unit, functional and integration tests (I run "rake test" in Jenkins before this task.)
As far as I can tell, this script above doesn't pass the failed exit code of the ruby tasks to Jenkins, no idea why. Right now the only workaround I can think of is to redirect all the test output to a buffer and use grep to look for failure keywords and pass the failed exit code if found, but that feels hacky to me.
What do I need to change to get Jenkins to detect when the "system 'rvmsudo ruby ' + r" commands return a failed exit code?
UPDATE
robertodecurnex is correct, system is always returning success. The rake command 'sh', however, allows you to capture accurate codes. So I changed this:
system 'rvmsudo ruby ' + r
to this:
sh 'rvmsudo ruby ' + r do |ok, res|
raise "Failed test." if !ok
end
Which seems to be doing the trick by exiting Rake with a fail code and alerting Jenkins that the build did not succeed.
Even if the system call fails your task is returning a normal execution status.
In order to get you CI server notified that there's something wrong the process should be returning an error execution code.
Note that, as an example, cucumber and rspec will blow if there's an error in any of the cases.
You should either check the system call return value and make your task return the same or run the tests inside the same ruby process and not as a system call to another ruby script.
Is there a cucumber command that will print out just the feature info and the scenario names?
I recently began a project and want to print out the cucumber features and scenarios I wrote to describe the scope of the project and get it confirmed with the client.
Do you mean cucumber -d?
$ cucumber -h
-d, --dry-run Invokes formatters without executing the steps.
This also omits the loading of your support/env.rb file if it exists.
Implies --no-snippets.
Implies --dry-run --formatter pretty.