I invoke our gtest suite for iOS in Jenkins using the shell script
#!/bin/sh
pkill -a "iPhone Simulator"
ios-sim launch ${WORKSPACE}/source/apple/build/Debug-iphonesimulator/MyAppTest.app --args --gtest_output=xml:${WORKSPACE}/JUnitTestResultsIOS.xml
exit $?
This always successfully runs the tests, and when the tests pass the xml file gets generated as expected. However, when the tests fail, no xml file is generated, and the "Execute shell command" build step terminates but does not fail the job. I echoed the exit code and it comes back 0 even when the tests fail.
This is even more confusing to me since we have a basically identical script in the same job for running tests on our OSX version. This always writes the xml and successfully fails the job when the tests fail.
This behavior seems totally arbitrary and everything about our configuration seems to be exactly as it should be. What am I missing?
Thanks!
There were two things at work here.
First of all, we had the break_on_failure gtest option enabled, which works great when running tests on a local machine but isn't useful within Jenkins, so we disabled it on the build machine.
The second issue was around how we used the exit code. Since ios-sim launch ... always succeeds we were always getting an exit code of 0, regardless of whether the tests passed or failed. I ended up using grep to determine if the resulting xml file indicated any failures, and generated an exit code based on that.
Related
I'm having trouble running my tests with NUnit in my Jenkins build. Where the tests all succeed when I run them manually (via a Cake build script) - even directly on the build server -, they fail when run during a build in Jenkins that calls that very same build script.
As output I get the following message:
An error occurred when executing task 'Test'.
Error:
NUnit3: Unrecognised error (exit code -1073740940).
When running NUnit with TraceLevel = TraceLevel.Debug, I get a trace file that looks fine until it simply stops after the same line:
15:49:14.342 Debug [ 6] Dispatcher: Using Direct strategy for <my failing test>
I'm running NUnit with InProcess and a single worker.
It's always the same test failing - but only when run in the CI build.
I get that it's nearly impossible to diagnose the exact problem from here - what I'm asking is for is if there's any way to get any more information on what exactly is failing?
Based on the error code, this seems to be a heap corruption error (C0000374). For more info see https://blogs.msdn.microsoft.com/calvin_hsia/2015/01/30/heap-corruption-exception-0xc0000374/ for some examples of how such an error is generated.
My guess is that your test is doing something bad and getting away with it in most circumstances but that running under Jenkins may change the environment enough so that it throws the exception.
This is very hard to diagnose. First thing I would want to ascertain is whether any of your test code gets executed at all. The NUnit internal trace appears to indicate that the test was dispatched (enqueued) but not actually dequeued and run. However, we sometimes lose a few events when the runner crashes.
You might try running with --workers=0 to see if the simplified execution code (no queues are used) gets you more information.
I am trying to run Jmeter with ant, (since I want to display the results of test, with all the steps).
The issue is that after I managed to run the command the Jtl file is empty, I am trying to run the basic test.jmx default test.
I run the command ant -Detest=Test run
the build is successful but the jtl results are empty. moreover it is finished after one second, while if I run via UI mode it should take more time.
Can someone please advise how to use ant with Jmeter, or how to get fully reports like in csv in html out put?
[][build results]
[][build.xml from ant\bin location]
[][location of test.jtl results file]
[][results are empty]
Most probably something is wrong with your JMeter test itself, i.e.
JMeter failed to start (installation or configuration issues)
it has 0 threads in Thread Group
there is an If Controller condition which prevent test execution
the test relies on a JMeter Plugin which is not installed
etc.
So I would recommend amending your build.xml file and enable writing JMeter log file by adding the next line to <jmeter> section: jmeterlogfile="${testpath}/jmeter-ant.log >
So it would look like
<jmeter
jmeterhome="${jmeter.home}"
testplan ="${testpath}/${test}.jmx"
resultlog="${testpath}/${test}.jtl"
jmeterlogfile="${testpath}/jmeter-ant.log>
When you run your test one more time you should see jmeter-ant.log file in the folder where your .jmx file lives.
More information: JMeter Ant Task
I have my selenium tests written using SpecFlow(+SpecRun) and NUnit framework (v.3.8.1.0). I've configured Jenkins to run these tests. My Jenkins Windows Batch Command is as follows:
"C:\Program Files (x86)\NUnit.ConsoleRunner\3.7.0\tools\nunit3-console.exe"
C:\Projects\Selenium\ClassLibrary1\PortalTests\bin\Debug\PortalTests.dll
--test=TransactionTabTest;result="%WORKSPACE%\TestResults\TestR.xml";format=nunit3
When I trigger build test seems to start running as I'm getting as far as end of NUNIT3-CONSOLE [inputfiles] [options] with spinner indicating that test is running but it actually never ends and estimated remaining time is: N/A.
Now, when I run this script with windows cmd.exe:
"[PATH to Console.exe]\nunit3-console.exe" PortalTests.dll -- test=TransactionTabTest
this test pass successfully and so does in VS.
Now, I know this is very generic question but any clues will be much appreciated.
As you are using SpecFlow+Runner/Specrun, you can find the documentation how to configure it for the different build servers here: http://specflow.org/plus/documentation/SpecFlowPlus-and-Build-Servers/
I finally got Hudson to build my project and the corresponding test project (using the XMLTestRunner2 unit provided in the Embarcadero forum). Running the test executable manually correclty produces a "dunit-report.xml" file with the test results.
I can't get Hudson to call my executable and produce this file though.
What I did is to create a build step as a Windows batch command and just call the executable. I tried several things:
bin\Test.exe
start bin\Test.exe
start /wait bin\Test.exe
start /wait /b bin\Test.exe
I cannot get it to work. It either returns immediately with some random exit code or it does not produce the XML output file. This must be very simple but I'm a little bit frustrated by now, because I don't get it to work.
What's the right way to execute the unit tests from Hudson?
I can't seem to run a build execute shell step in Jenkins. I've worked with Hudson in the past on windows and I was able to create shell/batch steps without a problem but I seem to be be missing something here.
It's a fresh jenkins install and I go to "add build step", "execute shell" and enter "echo hi" in the command. I run the build and when I look in the console output, nothing happens.
I've also tried executing a test.sh file which also just echoes hi. I've tested this in both a linux install and an os X installed Jenkins server.
What am I missing in the configuration to run a shell script?
The console output shows that the shell script steps were skipped completely
Started by user admin
Finished: SUCCESS
It looks like Jenkins is not being able to redirect the output from the system. What version of Java are you using? If you're using OpenJDK, could you try with Sun Java/Sun JDK?
First test to try to check if anything is executing at all: add the following to your "Execute Shell"
#!/bin/bash
echo "HELLO WORLD" > /tmp/testfile
Run this and check if there is a /tmp/testfile in on your Linux system, and if it contains the HELLO WORLD text, it means your script is in fact executing.
Which version of Jenkins do you have?
The last good version that I can attest to (last one I know works well at least for us) is 1.447. If you're not using that one, would you be able to try with it?
Also, could you add #!/bin/sh or #!/bin/bash before echo hi on your "Execute Shell" for the Linux system and see if that works.
Also, try running a script using source /path/to/script and see if that works. The script should contain #!/bin/sh or #!/bin/bash as the first line, just to see if that makes a difference.
Note: none of this should be required, but is helpful just to get more information on what's going on. Couldn't fit all this into a comment. I'll update my answer based on your answers to the above, or delete if I can't get anything..
Putting this here for posterity.
I had a Jenkins project configured with Maven running clean test and a execute shell in the pre steps. The logs from Maven where not coming through and the script was not executing. Once I unchecked Build modules in parallel under the Maven build options my logs and scripts started working.
Make sure its in a location where Jenkins can see it, check permissions.