We are using the mock out method for making system calls:
before do
subject.stub(:`) { `(exit 0)` }
end
...
#when running the tests we check the result of the last shell command using $?
if $?.exitstatus == 0
# some code
end
This works locally using jruby 1.7.9 on mac osx. But when the builders execute these same tests using jenkins CI we have an intermittent result of $?.exitstatus == 127. Since the stub points off to a subshell that should only return exit code 0 this is extremely bizarre.
My questions are:
What are the possible causes of this discrepancy?
Is there some difference between the linux variable $? and the ruby call to $?.exitstatus that could lead to getting inconsistent results?
We have since worked around the issue by implementing a stub on how we retrieve exitstatus so this is more a question to satisfy my curiosity.
Related
My question in general is this: when I run automated tests in parallel how can I know which of them were run at the same time?
My issue is this: I'm running tests in parallel (4 tests per time) but I have no idea which of them are executing together (at the same time) and which are waiting.
After my tests are executed there are some failures and it could be that the cause of the failures is other tests that were executing in the same time with the failed test. So I want to know which tests were executing together so I can run them together again to debug the failed test.
Technologies I use: NUnit 3.0, C#, Selenium WebDriver, Jenkins
I'm glad to hear any possible solution (doesn't matter if it's hard to apply)
For debug purposes, you can run the NUnit console with the --trace=Verbose option.
This will write a log to the directory you are executing in, which will look a little like this:
14:56:13.972 Info [13] TestWorker: Worker#4 executing TestsSuite1
14:56:13.973 Debug [13] WorkItemDispatcher: Directly executing TestA
14:56:13.976 Debug [12] WorkItemDispatcher: Directly executing TestB
14:56:13.976 Debug [16] WorkItemDispatcher: Directly executing TestC
14:56:13.980 Debug [12] WorkItemDispatcher: Directly executing TestD
14:56:13.982 Debug [16] WorkItemDispatcher: Directly executing TestE(False,False,False,0)
14:56:13.989 Debug [16] WorkItemDispatcher: Directly executing TestE(False,False,True,0)
I believe the numbers in the square brackets identify different threads, so here, you can see that workers 12 and 16 where running TestC and TestD simultaneously.
Of course, it's never as simple as just seeing which tests where running concurrently, but this may help narrow issues down.
The simplest approach is to disable parallelism selectively until you find the combination that causes the problem.
Disable all parallelism by running with --workers=0. If the same tests fail, then you don't have a parallelism problem.
Selectively rerun the failed tests together, using the --test or --where option to select them. Edit the fixtures to enable/disable parallelism until you hit the combination that does the job.
I've been writing some shell tasks that can't run unless there are secure environment variables present in a Travis CI PR build. There is an auth token that must be present to push some information up about the build, so for builds originating from forks, I'd like to simply skip these parts. They're not critical.
How can I tell if a build is originating from a fork?
From the documentation around "Environment Variables":
TRAVIS_SECURE_ENV_VARS: Whether or not secure environment vars are being used. This value is either "true" or "false".
This is a little ambiguous. Does it mean that secure environment variables are being used anywhere (as in, present in .travis.yml)? That they are being exported as environment variables in the current build? I'm not sure this is a good way to guarantee that I'm testing a pull request that has originated from a fork, but I didn't see any other way to do it.
My first attempted had code that looked something like
[ ${TRAVIS_SECURE_ENV_VARS} = "false" ] && exit 0; # more shell code here...
but this appears to continue ahead and push up without an auth token, failing the task (and the build). Further complicating matters is that should the command fail, the output may contain the auth token...so everything from stderr and stdout is redirected to /dev/null. Considering that builds don't start for several minutes, I'm stuck waiting in a long debug cycle.
My next attempt simply bypassed this built-in environment variable, in favor of instead trying to grab a secure environment variable directly.
[ ${ghToken} -n ] && exit 0;
This fails in the same way as above. I'm beginning to wonder if [ $COND ] && exit 0; really works the way I'm expecting it to in this context. It seems to work just fine when I run equivalent shell scripts locally (Mac OSX and bash).
Does Travis offer a built-in way to determine if the pull request being built originated from the original repository versus a fork?
Here is my current work around.
screenshotsClone: {
command: ['[ ${ghToken} ] &&',
'git submodule add -f', screenshotPullTemplate, 'screenshots > /dev/null 2>&1;'].join(' '),
options: {
stdout: false,
failOnError: false
}
}
I'd rather not silently pass on errors should there be a legitimate problem with the shell task. At this point I might as well remove the environment variable check that precedes it.
Stop using grunt-shell for things in your .travis.yml. Put those shell tasks into their own files so that you can stop using /bin/sh and start using /bin/bash, which is what you've been testing against locally.
Something like this will fix your problems.
Don't forget to mark the shebang as #! /bin/bash, which is crucial for the kinds of checks that should be happening on Travis.
I think what you want is to check if one of your secure environment variables is null to detect that you are running a build from a fork and in that case stop the build script prematurely.
Hence I would suggest you use the -z comparison operator in BASH to detect null strings because the -n operator detects non null string. A non null secure environment variable would mean that you are not running the build from a fork.
My suggestion is to change your line for:
[ -z "${ghToken}" ] && exit 0;
Hope this helps.
As title says, my travis-ci instance is telling me that my build is failing when I have cucumber step definitions that are pending (not undefined) - is there a way to stop this behaviour?
I can understand the reason why, but the undefined behaviour I have is undefined by purpose - I have no intention on doing that feature RIGHT NOW, but I know I want to keep the step definition down so I don't forget what I came up with. The reason I ask is because this behaviour (with it failing when I have a pending task) could end up masking real failures that actually matter.
There is a similar answer here which should be useful for you, the answer that he gives is:
I think cucumber gives non-zero process exit code either because of
skipped or because of pending tests. Try to get it to not run any
skipped, then any pending, then any skipped or pending tests and see
what exit codes it gives. To see the exit code (in Unix), run it with
something like:
cucumber ...args to select tests... ; echo $?
So basically you want to figure out what args you can provide to cucumber to have it pass and then provide a .travis.yml that runs that command. an example rake task for cucumber jasmine and rspec is here:
task :travis do
["rspec spec", "rake jasmine:ci", "rake cucumber"].each do |cmd|
puts "Starting to run #{cmd}..."
system("export DISPLAY=:99.0 && bundle exec #{cmd}")
raise "#{cmd} failed!" unless $?.exitstatus == 0
end
end
And read the docs for more info on creating a .travis.yml.
If all of that doesn't work, just create a bash script that catches the output and returns 0 in the right cases, I really doubt this is required though since I'm sure there's an option to cucumber to make it ignore pending.
EDIT: the cucumber docs say:
Pending steps
When a Step Definition’s Proc invokes the #pending method, the step is
marked as yellow (as with undefined ones), reminding you that you have
work to do. If you use --strict this will cause Cucumber to exit with
1.
Which seems to imply that they will not exit with 0 if you don't call --strict
We've just set up a Jenkins CI server for our app, HiringThing
Everything works well, with builds starting automatically when code is checked into our Github repo.
The problem is detecting build failures. I have the following rake task being run from the command line by Jenkins.
rake test:browser
runs the following
desc "Run browser tests."
task :browser => :environment do
start = Time.now()
puts "Stopping apache if running"
system 'sudo apache2ctl stop'
puts "Running selenium tests"
Dir.glob('./test/browser/*.rb').each { |r|
puts r
system 'rvmsudo ruby ' + r
}
system 'echo -e "\a"'
puts "All browser tests, elapsed: " + (Time.now() - start).to_s + " seconds"
end
I've confirmed that the tests are running properly (using headless Firefox with Xvfb) The problem is that Jenkins doesn't detect any build failures in the browser tests. It seems to work for the usual Rails unit, functional and integration tests (I run "rake test" in Jenkins before this task.)
As far as I can tell, this script above doesn't pass the failed exit code of the ruby tasks to Jenkins, no idea why. Right now the only workaround I can think of is to redirect all the test output to a buffer and use grep to look for failure keywords and pass the failed exit code if found, but that feels hacky to me.
What do I need to change to get Jenkins to detect when the "system 'rvmsudo ruby ' + r" commands return a failed exit code?
UPDATE
robertodecurnex is correct, system is always returning success. The rake command 'sh', however, allows you to capture accurate codes. So I changed this:
system 'rvmsudo ruby ' + r
to this:
sh 'rvmsudo ruby ' + r do |ok, res|
raise "Failed test." if !ok
end
Which seems to be doing the trick by exiting Rake with a fail code and alerting Jenkins that the build did not succeed.
Even if the system call fails your task is returning a normal execution status.
In order to get you CI server notified that there's something wrong the process should be returning an error execution code.
Note that, as an example, cucumber and rspec will blow if there's an error in any of the cases.
You should either check the system call return value and make your task return the same or run the tests inside the same ruby process and not as a system call to another ruby script.
I'm writing a quick Rails app and was wondering how I can validate the success an exec'd command. The two commands I'm running are and SVN update, and a cp from one directory to another.
If you use the Kernel.system() method it will return a boolean indicating the success of the command.
result = system("cp -r dir1 dir2")
if(result)
#do the next thing
else
# handle the error
There is a good comparison of different ruby system commands here.
How are you executing the external commands? The Ruby system() function returns true or false depending on whether the command was successful. Additionally, $? contains an error status.
Just to be pedantic, you can't validate an exec'd command because exec replaces the current program with the exec'd command, so the command would never return to Ruby for validation.
For the cp, at least, you would probably be better of using the FileUtils module (part of the Ruby Standard Library), rather than dropping to the shell.
As noted above, the $? predefined variable will give you the return code of the last command to be executed by system() or the backtick operator.
For SVN update, check the version number before and after the update.
svn_start_version = IO.popen("svn info").readlines[4]
`svn update`
svn_end_version = IO.popen("svn info").readlines[4]
if svn_end_version > svn_start_version
"success"
end
For the cp, you could do a filesize check on the original file being equal to the copied file.
source_file_size = IO.popen("du file1").readlines
`cp file1 file2`
dest_file_size = IO.popen("du file2").readlines
if dest_file_size == source_file_size
"success"
end