I have a bunch of PHPUnit tests that I invoke with ANT in jenkins. All of a sudden
[exec] ............................................................... 63 / 644 ( 9%)
[exec] ............................................................... 126 / 644 ( 19%)
[exec] ............................................................... 189 / 644 ( 29%)
[exec] ............................................................... 252 / 644 ( 39%)
[exec] ............................................................... 315 / 644 ( 48%)
[exec] .............................................................
BUILD FAILED
/etc/jenkins/build.xml:188: exec returned: 139
Total time: 57 seconds
Build step 'Invoke Ant' marked build as failure
Funny thing is that when I manually run the tests, everything runs (at the very least) ie: goes to 100%.
I already specify Jenkins to use the ant binary from
which ant
in Jenkin's config menu and force this particular build to use it.
None of this makes sense to me because earlier today the build was succeeding. This just started to happen a few days ago and I resolved it by restarting the machine but I really want to figure out why this spontaneously happens.
Does it fail only on the Jenkins server, or does it fail on development machines too? What command is failing?
Exit code 139 is a segmentation violation. You'll have to determine what's failing.
Since you're using Jenkins, and your using PHP and your using <exec> why not try to do the tests outside of Ant? Jenkins can run commands as part of the build process. That will eliminate Ant as a possible cause.
Try playing around with <exec> settings. Setting spawn to true will spawn a subshell that won't have the memory limit that running an exec in Java would have. Running out of memory could be causing the segmentation faults.
I also had this problem. For me it ended up in a problem with xdebug, which is needed for the codecoverage report. Loading xdebug via another php config file seemed to fix the problem.
Related
Summary: A month or two ago we started noticing our Jenkins builds stalling indefinitely. When a build spins up, our Jenkins Shared Library creates a new docker container on a virtual machine and executes the library code to build our service, usually including a maven build. We sporadically see a build or two stall indefinitely on one of various Maven goals (and only maven goals, as seen below). Sometimes this is happens on single build on a virtual machine, but it is most egregious when we run multiple builds at the same time. Some will build, others will stall. All builds on the same virtual machine share an .m2 repository so we don't download dependencies every time.
We are using maven 3.8.1 running in an ubuntu 18.04 container. We're looking for possible leads as we've exhausted all of ours.
We tried upgrading to maven 3.8.2 with no luck, we tried looking into this similar issue but its solution did not apply to us: Jenkins docker container simply hangs and never executes steps
We've also tried having each node have its own maven repository and that didn't work, so we ruled out read/write conflicts.
Details:
Some of the places it stalls:
Mvn help:evaluate after "Picking up Java_Tools_Options...",
Mvn set:property,
On checking on updates from virtual,
On actually downloading from artifactory.
These are a couple example of where it stalls:
[2022-02-22T17:39:15.452Z] + mvn versions:set-property -Dproperty=my.version -DnewVersion=4.0.1 -DgenerateBackupPoms=false -X --debug
[2022-02-22T17:39:15.452Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400
And
[2022-03-03T13:27:01.046Z] + mvn clean install -P snapshot -Dmaven.test.skip=true -Dbuild.number=32 -U
[2022-03-03T13:27:01.046Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400
[2022-03-03T13:27:02.425Z] [INFO] Scanning for projects...
[2022-03-03T13:27:04.913Z] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) # sample-child-java ---
[2022-03-03T13:27:05.481Z] Downloading from virtual: https://my.artifactory/org/codehaus/plexus/plexus-java/0.9.10/plexus-java-0.9.10.pom
When we turn on debug mode for maven, we only get a little extra information before it stalls:
02-22T17:39:20.880Z] [DEBUG] Reading global settings from /opt/maven/conf/settings.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading user settings from /home/myUser/.m2/settings.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading global toolchains from /opt/maven/conf/toolchains.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading user toolchains from /home/myUser/.m2/toolchains.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Using local repository at /home/myUser/.m2/repository
[2022-02-22T17:39:21.144Z] [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/myUser/.m2/repository
After about 6 hours of stalling, the logs will usually show one of these statements:
Cannot contact myVirtualMachine6: Remote call on myVirtualMachine6 failed. The channel is closing down or has closed down
Wrapper script does not seem to be touching the log file in /home/jenkins/workspace/myService#tmp/durable-7db8f58b
(JENKINS-48300: if on an extremely laggy filesystem, consider - HEARBEAT_CHECK_INTERVAL=86400)
We tried using the withMaven() from the Maven Integration Plugin, but even that did not solve our problem:
11:14:07 + mvn versions:set-property -Dproperty=myService.version -DnewVersion=9.9.9-RC1 -DgenerateBackupPoms=false
11:14:07 Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/myService#tmp/withMaven368661c2/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/myService#tmp/withMaven368661c2"
11:14:10 [INFO] [jenkins-event-spy] Generate /home/jenkins/workspace/myService#tmp/withMaven368661c2/maven-spy-20220308-161411-4457772303541115170084.log.tmp ...
When we add a tee command to our withMaven() maven commands, the output where it stalls is just:
[INFO] [jenkins-event-spy] Generated /home/jenkins/workspace/myService#tmp/withMavena1de258e/maven-spy-20220308-192941-7666252427488201541499.log
[INFO] [jenkins-event-spy] Generate /home/jenkins/workspace/myService#tmp/withMavena1de258e/maven-spy-20220308-193003-7826277450333225168004.log.tmp ...
Our Jenkinsfile is just this and calling our maven script:
agent {
docker {
label 'myVirtualMachine6'
image 'myImage'
registryUrl 'xxxx'
registryCredentialsId 'yyyy'
args '-v maven-repo:/home/myUser/.m2/repository/ -v /var/run/docker.sock:/var/run/docker.sock'
}
}
I am facing very weird issue. I am running cucumber jvm for java project on Jenkin with fork count 10.
I have more than 100 features files. Sometimes some features files (not specific) stop their execution without giving any error or exception. So because of that their json files is not getting created and records also get missed in the cucumber html report. Like it is not consistent it occurs 1 out of 8 times. But it is not giving any exceptions, warning or error so hard to track. Any one know what is issue and can help me with this issue.?
Thanks.
Please split your execution, sometimes JVM getting peak. if our TCs running for long time, this will break our jenkins. so run your test by 20 to 30 features at a time.
like
max 30 features / one jenkins build
for Eg: if you have 100 features with 500 scebarios
Build 1 - mvn test -DCucumber.Options="--tags #first0to20"
Build 2 - mvn test -DCucumber.Options="--tags #first21to40"
Build 3 - mvn test -DCucumber.Options="--tags #first41to60"
Build 4 - mvn test -DCucumber.Options="--tags #first61to80"
Build 5 - mvn test -DCucumber.Options="--tags #first81to100"
I'm trying to build bazel on a Windows machine, but bazel build //src:bazel does not find any target to build:
$ bazel build //src:bazel
Loading:
Loading: 0 packages loaded
INFO: Analysed 0 targets (0 packages loaded).
INFO: Found 0 targets...
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
INFO: Elapsed time: 0.389s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action
To be clear: This is happening only with master, I was able to build the 0.10.1 dist version of Bazel (from github) on my Windows machine.
To make sure my 0.10.1 binary is okay I built the cpp example (//examples/cpp:hello-world). It worked as expected - no failures.
I was able to build a stable dist version (namely, 0.10.1)bazel master on a Linux and macOS machines.
Any help here would be greatly appreciated.
It looks like you are running Bazel from MSYS Bash. This is no longer necessary, because Bazel is a native Windows binary since version 0.5.0.
MSYS thinks the argument //src:bazel is a Unix-style path (because it starts with /) so it converts the "path" for Bazel to a Windows path. IIRC the conversion logic just removes one / and the target pattern ends up being /src:bazel.
As a solution, I suggest one of the following:
Run Bazel from cmd.exe or from Powershell. This is how I use Bazel and it works as expected.
Disable the path conversion heuristic in MSYS:
export MSYS_NO_PATHCONV=1
export MSYS2_ARG_CONV_EXCL="*"
Use /// instead of // in the target path. This only works under MSYS and I don't think it's reliable because I don't understand MSYS's path conversion logic, so I don't recommend this solution but here it is anyway:
bazel build ///src:bazel
I invoke our gtest suite for iOS in Jenkins using the shell script
#!/bin/sh
pkill -a "iPhone Simulator"
ios-sim launch ${WORKSPACE}/source/apple/build/Debug-iphonesimulator/MyAppTest.app --args --gtest_output=xml:${WORKSPACE}/JUnitTestResultsIOS.xml
exit $?
This always successfully runs the tests, and when the tests pass the xml file gets generated as expected. However, when the tests fail, no xml file is generated, and the "Execute shell command" build step terminates but does not fail the job. I echoed the exit code and it comes back 0 even when the tests fail.
This is even more confusing to me since we have a basically identical script in the same job for running tests on our OSX version. This always writes the xml and successfully fails the job when the tests fail.
This behavior seems totally arbitrary and everything about our configuration seems to be exactly as it should be. What am I missing?
Thanks!
There were two things at work here.
First of all, we had the break_on_failure gtest option enabled, which works great when running tests on a local machine but isn't useful within Jenkins, so we disabled it on the build machine.
The second issue was around how we used the exit code. Since ios-sim launch ... always succeeds we were always getting an exit code of 0, regardless of whether the tests passed or failed. I ended up using grep to determine if the resulting xml file indicated any failures, and generated an exit code based on that.
i try to integrate my Robotium Tests on our Jenkins Server. I implemented an android Test Project that based on our app Project. Everything works like a charm when i run the Tests on my local Machine on Windows out of Eclipse and also by calling from shell by call
'adb shell am instrument -w <our-package>/android.test.InstrumentationTestRunner'
So far, so good, but after i set up a build Job in Jenkins, let the Projects build by maven clean install sign etc. none of my tests will be proceeded. I also tried to call 'adb shell am instrument -w <our-package>/android.test.InstrumentationTestRunner' as shell command after the build step but also no result. The build ended up with just 2 Tests shown log below
[workspace] $ /bin/sh -xe /tmp/hudson3571502822112946903.sh
+ /home/jenkins/tools/android-sdk-linux/platform-tools/adb shell pm list instrumentation
instrumentation:com.android.emulator.connectivity.test/android.test.InstrumentationTestRunner (target=com.android.emulator.connectivity.test)
instrumentation:com.android.emulator.gps.test/android.test.InstrumentationTestRunner (target=com.android.emulator.gps.test)
instrumentation:com.zeppelin.zemos.test.addispo/android.test.InstrumentationTestRunner (target=com.zeppelin.zemos.addispo)
+ /home/jenkins/tools/android-sdk-linux/platform-tools/adb shell am instrument -w com.zeppelin.zemos.test.addispo/android.test.InstrumentationTestRunner
android.test.AndroidTestCase:.
android.test.LoaderTestCase:.
Test results for InstrumentationTestRunner=..
Time: 0.031
OK (2 tests)
Just the 2 Tests AndroidTestCase and LoaderTestCase are shown up and it seems that all of my other Testcases (i've implemented 11) are not processed by Jenkins.
I spent a long time googling around but found no solution for this. Could this be a ant Problem? I have Version 1.8 local and 1.6 on the Jenkins Server. Or is there another Problem i cannot see.....
Thanks a lot
You can try uninstalling the package from device/ emulator, using
"adb uninstall your_package"
Then, do a clean debug install again and then run the tests.