Unable to get JSCover and PhantomJS to run Jasmine test on Cloudbees - jenkins

I am currently trying to run JSCover in web server mode to determine the coverage of my Jasmine tests that are executed in the PhantomJS headless browser. I am also using grunt+nodejs to kick off the tests.
The code I use in my gruntfile to start the JSCover server and execute phantomJS is:
// Start JSCover Server
var childProcess = require('child_process'),
var JSCOVER_PORT = "43287";
var JAVA_HOME = process.env.JAVA_HOME;
var jsCoverChildArgs = [
"-jar", "src/js/test/tools/JSCover-all.jar",
"-ws",
"--branch",
"--port="+JSCOVER_PORT,
"--document-root=./",
"--report-dir=target/",
"--no-instrument=src/js/lib/",
"--no-instrument=src/js/test/",
"--no-instrument=src/js/test/lib/"
];
var jsCoverProc = childProcess.spawn(JAVA_HOME + "/bin/java", jsCoverChildArgs);
// Start PhantomJS
var phantomjs = require('phantomjs'),
var binPath = phantomjs.path,
var childArgs = [
'src/js/test/lib/phantomjs_jasminexml_runner.js',
'http://localhost:'+JSCOVER_PORT+'/src/js/test/SpecRunner.html',
'target/surefire-reports'
];
runner = childProcess.execFile(binPath, childArgs);
runner.on('exit', function (code) {
// Tests have finished, so clean up the process
var success = (code === 0) ? true : false;
jsCoverProc.kill(); // kill the JSCover server now that we are done with it
done(success);
});
However, when I run the web server on a Jenkins node in cloudbees and then run phantomjs against it, I get one of the following errors:
Some tests start to run, but then the process fails:
A spec : should be able to have a mock lo-dash ...
Warning: Task "test" failed. Use --force to continue.
Aborted due to warnings.
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE
PhantomJS is unable to access the JSCover server:
Running "test" task
phantomjs> Could not load 'http://127.0.0.1:43287/src/js/test/SpecRunner.html'.
Warning: Task "test" failed. Use --force to continue.
For the second error, I have tried to use different ports and hostnames that I set (e.g. 127.0.0.1 or localhost for hostnames, and 4327, 43287, etc. for ports). The ports are not being dynamically set at build time - I have them hardcoded in my grunt script.
Any thoughts on why the errors above might be occurring or why I am having issues running and accessing the JSCover server on a Cloudbees Jenkins node (but never on my local machine)?

So when you execute JSCover with any process, it takes time to be up. If we expect it to be up earlier that it is, the errors are bound to come.
Quoting from the great article: http://blog.johnryding.com/post/46757192364/javascript-code-coverage-with-phantomjs-jasmine-and
Now that I had a code coverage tool that met all of my requirements,
the last part was to get this code to run as part of our Jenkins build
(which utilizes a grunt script). This was easy to get running, but I
encountered two errors that consistently broke my builds:
Sometimes phantomJS would fail to connect to the JSCover server
Sometimes phantomJS would connect to the server, but then give up executing my tests at a random point during the run.
These were really weird issues that only occurred on my team’s Jenkins nodes and were hard to diagnose - even though they turned out to be simple fixes.
For issue 1, that error was the result of my grunt script not waiting for JSCover to start before I executed phantomJS.
For the second issue, it turns out that my team was using a special jasmine test runner to help with producing XML files after tests completed. The problem with this file was that it had a function that waited for Jasmine to complete its execution, but utilized an extremely short timeout before it gave up running the tests. This was a problem with Jenkins + JSCover because it took a longer time for the tests to load and run now that they had to be loaded from a web server instead of straight from the file system. Fortunately, this fix was as easy as increasing the timeout.

I would say that you need to wait for a while after spawning JSCover - in the past I have done things with webdriver when I have spawned, and then waited for it to be available (ideally you can look for a response and sleep, repeat, until the spawned process is ready).
Ie look for a valid http reponse from 127.0.0.1:43287 before continuing (whatever "valid" means that the server is up).

Related

JMeter execute ant test local failed

I'm following JMeter wiki page of CommittingChanges which was updated.
I'm running
ant checkstyle
ant package
ant test
Test Failed with error in differences from Bug52310.csv:
BUILD FAILED
C:\jmeter\jmeter\build.xml:2681: The following error occurred while executing th
is line:
C:\jmeter\jmeter\build.xml:2621: CSV Files are not identical.
C:\jmeter\jmeter\bin\testfiles\Bug52310.csv
ComputeIPAddr,200,OK,TG2 1-1,text,true
HTTP-Request-HC31,200,OK,TG2 1-1,text,true
HTTP-Request-HC4,200,OK,TG2 1-1,text,true
C:\jmeter\jmeter\bin\Bug52310.csv
ComputeIPAddr,200,OK,TG2 1-1,text,true
HTTP-Request-HC31,Non HTTP response code: java.net.SocketException,Non HTTP resp
onse message: Network is unreachable: connect,TG2 1-1,text,false
HTTP-Request-HC4,Non HTTP response code: java.net.SocketException,Non HTTP respo
nse message: Network is unreachable: connect,TG2 1-1,text,false
Total time: 3 minutes 9 seconds
Is it a bug or configuration/network issue? can all JMeter tests be execute in local environment?
EDIT
Also next step is to run different test
ant test-headless or ant test-headed (whichever was not run by the ant test)
Why not execute ant test-both in the first place?
when I execute ant test-headed and even and test-both it worked successfully
Regarding your first issue, it looks like a temporary connectivity issue from your machine or more probably that test computes an IP address for your machine that is not reachable .
This test exists to test ip-spoofing feature by setting the ip source to your machine address.
Run it in gui and see what IP is computed on your machine then try to resolve it.
Regarding ant test-both, yes it would be better, report a bug please.

Run RobotFramework tests with Sikuli Library with Jenkins on VM (RDC)

I have automation tests based on RobotFramework with SikuliLibrary, which are for Image Compare. I'm using Jenkins to run on external server (VM) the tests.
If I open the VM - image compare script works. The screenshot is created.
If I close the VM session and run the test, there is problem. Here is the log from keyword "Get Match Score":
INFO Could not find C:\Images\image.png
INFO ${scoreFromImage} = 0.0
Is look like, when the VM session is not active (opened), "Get Match Score" cannot take a snapshot from the browser for comparing.
Is there any idea, how to fix this?
The Code:
Compare Process Diagram Image
[Arguments] ${ImageName} ${ImageScore}
${scoreFromImage} = Get Match Score ${ImagesDirectory}${ImageName}.png
${scoreToString} = Convert To String ${scoreFromImage}
${scoreNumberPrecision} = Get Substring ${scoreToString} 0 6
Run Keyword If ${scoreNumberPrecision} == ${ImageScore} Log Successful ELSE Log Fail
Solved:
The VM must be not closed by "X" button. The correct way:
https://support.microsoft.com/en-us/kb/302801
I have .bat file, which contains tscon.exe %SESSIONNAME% /dest:console, after run - the VM is closed correctly.
Also "Jenkins slave jar" must be running.
Running Sikuli Test on VM is possible but need to keep session open. We cannot run Sikuli script on locked PC. When you close VM , it get locked and Test fail to run. Sikuli needs images for comparing and clicking, if session is locked there are no images so Test fail to run.
So how we overcome this ? :
https://support.smartbear.com/viewarticle/85926/
refer this URL for setting . By applying this you can run sikuli Test with minimized window of VM. (Still you cannot close WM window)

Calling Snapshot in Jenkins results in Time out for Simulator

I am using Snapshot from the FastLane suite.
For my purposes I am calling the various tools from scripts and pass in the appropriate environment variables I am using as the inputs.
I am having trouble when I call my script in Jenkins vs from the command line. When I call the script within a Build Step in Jenkins the result is a message from Snapshot saying the process has timed out after waiting 120 seconds for the simulator to boot. If I run this same script from the terminal Snapshot runs as expected without error.
Example:
snapshot \
--workspace "MyWorkspace.xcworkspace" \
--scheme "MyScheme" \
--output_directory "MyOutputDirectory" \
--clear_previous_screenshots \
--stop_after_first_error
(--devices --languages can be found in ./Snapfile)
Snapfile:
devices([
"iPhone 4s"
])
languages([
"en-US"
])
Am I missing something here?
Configuring Jenkins to work for iOS testing and automation is not a simple task, there are a lot of gotchas.
Jenkins the result is a message from Snapshot saying the process has timed out after waiting 120 seconds for the simulator to boot.
This suggests that your Jenkins machine is not able to run the Simulator. This can happen if the jenkins user is not able to start a UI session.
These two posts have useful information on how to configure Jenkins for iOS development:
https://blog.pivotal.io/labs/labs/ios-ci-jenkins
http://staxmanade.com/2015/01/setting-jenkins-up-to-run-xctool-and-xcode-simulator-tests/
The second in particular addresses the issue of Jenkins not running as a GUI user.
Good luck.

Unable to run functional tests on server. [geb spock]

Hi I am running my functional tests using geb.
I am able to run tests on my local computer correctly. but as I deploy my application to server. the build for functional tests fails.
Here is my console output
|Running 10 spock tests... 1 of 10
Failure: |
sign in with voucher
|
geb.driver.DriverCreationException: failed to create driver from callback 'script14007213321291157436758$_run_closure1#77068fce'
at geb.driver.CallbackDriverFactory.getDriver(CallbackDriverFactory.groovy:35)
at geb.driver.CachingDriverFactory.getDriver_closure3(CachingDriverFactory.groovy:80)
at geb.driver.CachingDriverFactory$SimpleCache.get(CachingDriverFactory.groovy:30)
at geb.driver.CachingDriverFactory.getDriver(CachingDriverFactory.groovy:79)
at geb.Configuration.createDriver(Configuration.groovy:346)
at geb.Configuration.getDriver(Configuration.groovy:335)
at geb.Browser.getDriver(Browser.groovy:105)
at geb.Browser.go(Browser.groovy:377)
at geb.Page.to(Page.groovy:171)
at geb.Browser.via(Browser.groovy:454)
at geb.Browser.to(Browser.groovy:413)
at geb.Browser.to(Browser.groovy:391)
at geb.spock.GebSpec.methodMissing(GebSpec.groovy:51)
at VoucherSpec.sign in with voucher(VoucherSpec.groovy:14)
Caused by: org.openqa.selenium.WebDriverException: Failed to connect to binary FirefoxBinary(/usr/local/bin/firefox) on port 7056; process output follows:
Error: no display specified
Error: no display specified
Can anyone guide me to solve this issue.
The server you're running the tests on is 'headless' so doesn't have a display to start Firefox on to run the tests. You've got a couple of options:
Switch from Firefox to a headless browser such as HTMLUnit.
Configure a virtual display on the server.
Use a remote browser service such as SauceLabs.
If you need to test directly on Firefox then HTMLUnit isn't an option for you.
Using a remote browser service such as SauceLabs or BrowserStack has a couple of advantages, for example they record a video of the session and take screenshots, but we found the overhead of passing commands & traffic over the network made the tests unacceptably slow. If you need to test a wide variety of browsers then the overhead diminishes because you can run in parallel...
Option 2, using a virtual display, is the simplest to configure on most servers. If you're using Linux the X Virtual Frame Buffer (XVFB) will get you up and running quickly. It's worth reading up on what's going on but the short answer is:
Install XVFB (sudo apt-get install xvfb)
Install Firefox (sudo apt-get install firefox)
Start XVFB (sudo Xvfb :10 -ac -screen 0 1024x768x8 &). You may want to add an init script so this happens every time the server starts
In your CI server add export DISPLAY=:10 as a step before the tests are run
Run your tests
The XVFB creates a virtual display on :10, which you then set as the default display. When you start Firefox it's completely unaware that it's on a virtual display, so things like getting Geb to take screenshots of failing tests will work as normal.
For more information about the steps see:
http://www.installationpage.com/selenium/how-to-run-selenium-headless-firefox-in-ubuntu/
https://github.com/tomaslin/grails-test-recipes
http://www.semicomplete.com/blog/geekery/xvfb-firefox.html
http://www.labelmedia.co.uk/blog/setting-up-selenium-server-on-a-headless-jenkins-ci-build-machine.html
http://manpages.ubuntu.com/manpages/lucid/man1/xvfb-run.1.html
If you need an init script to get it to start/stop, then there are quite a few to choose from, such as this one.

How can get my rake task to let Jenkins know that the build has failed?

We've just set up a Jenkins CI server for our app, HiringThing
Everything works well, with builds starting automatically when code is checked into our Github repo.
The problem is detecting build failures. I have the following rake task being run from the command line by Jenkins.
rake test:browser
runs the following
desc "Run browser tests."
task :browser => :environment do
start = Time.now()
puts "Stopping apache if running"
system 'sudo apache2ctl stop'
puts "Running selenium tests"
Dir.glob('./test/browser/*.rb').each { |r|
puts r
system 'rvmsudo ruby ' + r
}
system 'echo -e "\a"'
puts "All browser tests, elapsed: " + (Time.now() - start).to_s + " seconds"
end
I've confirmed that the tests are running properly (using headless Firefox with Xvfb) The problem is that Jenkins doesn't detect any build failures in the browser tests. It seems to work for the usual Rails unit, functional and integration tests (I run "rake test" in Jenkins before this task.)
As far as I can tell, this script above doesn't pass the failed exit code of the ruby tasks to Jenkins, no idea why. Right now the only workaround I can think of is to redirect all the test output to a buffer and use grep to look for failure keywords and pass the failed exit code if found, but that feels hacky to me.
What do I need to change to get Jenkins to detect when the "system 'rvmsudo ruby ' + r" commands return a failed exit code?
UPDATE
robertodecurnex is correct, system is always returning success. The rake command 'sh', however, allows you to capture accurate codes. So I changed this:
system 'rvmsudo ruby ' + r
to this:
sh 'rvmsudo ruby ' + r do |ok, res|
raise "Failed test." if !ok
end
Which seems to be doing the trick by exiting Rake with a fail code and alerting Jenkins that the build did not succeed.
Even if the system call fails your task is returning a normal execution status.
In order to get you CI server notified that there's something wrong the process should be returning an error execution code.
Note that, as an example, cucumber and rspec will blow if there's an error in any of the cases.
You should either check the system call return value and make your task return the same or run the tests inside the same ruby process and not as a system call to another ruby script.

Resources