How to set up SBT build to return zero exit code on test failure for Jenkins? - jenkins

Question

When I am running my Specs2 tests in Jenkins via SBT, then the build is marked as a failure as soon as one test fails. Since Jenkins usually distinguishes between failure to build and test failures, I want to change this.
I know that the build failure in Jenkins is detected by the exit code of the call to SBT, which appears to return 1 as soon as at least one test fails.
What are the options I have assuming I want to avoid changing my build.sbt (or the project in general) just to fix this inconvenience?
Somehow I think it should be possible to put a standard sbt project into a standard Jenkins install and have it work as intended.

Answer 1

tl;dr Use testResultLogger with a custom test result logger that doesn't throw TestsFailedException that in turn sets the non-0 exit code.
Just noticed that I missed that requirement "to avoid changing build.sbt. You can use any other *.sbt file, say exitcodezero.sbt or ~/.sbt/0.13/default.sbt with the custom testResultLogger.
It turns out that since sbt 0.13.5 there's a way to have such behaviour - see Added setting 'testResultLogger' which allows customisation of test reporting where testResultLogger was born.
> help testResultLogger
Logs results after a test task completes.
As it may have been read in the implementation of TestResultLogger.SilentWhenNoTests that's the default value of testResultLogger:
results.overall match {
case TestResult.Error | TestResult.Failed => throw new TestsFailedException
case TestResult.Passed =>
}
It means that when there's an issue executing tests, TestsFailedException exception is thrown that's in turn caught to report it as follows:
[error] Failed: Total 3, Failed 1, Errors 0, Passed 2
[error] Failed tests:
[error] HelloWorldSpec
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
My idea is to disable throwing the exception regardless of the outcome of executing tests. Add the following to build.sbt and have the exit code always 0:
testResultLogger in (Test, test) := new TestResultLogger {
import sbt.Tests._
def run(log: Logger, results: Output, taskName: String): Unit = {
println("Exit code always 0...as you wish")
// uncomment to have the default behaviour back
// TestResultLogger.SilentWhenNoTests.run(log, results, taskName)
}
}
Uncomment TestResultLogger.SilentWhenNoTests.run to have the default behaviour back.
➜ failing-tests-dont-break-build xsbt test; echo $?
JAVA_HOME=/Library/Java/JavaVirtualMachines/java8/Contents/Home
SBT_OPTS= -Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -Dfile.encoding=UTF-8
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Set current project to failing-tests-dont-break-build (in build file:/Users/jacek/sandbox/failing-tests-dont-break-build/)
[info] HelloWorldSpec
[info]
[info] The 'Hello world' string should
[info] x contain 11 characters
[error] 'Hello world' doesn't have size 12 but size 11 (HelloWorldSpec.scala:7)
[info]
[info] + start with 'Hello'
[info] + end with 'world'
[info]
[info] Total for specification HelloWorldSpec
[info] Finished in 15 ms
[info] 3 examples, 1 failure, 0 error
Exit code always 0...as you wish
[success] Total time: 1 s, completed Sep 19, 2014 9:58:09 PM
0

Answer 2

You could run the part of the build that runs the tests in a wrapper script that always returns 0. (If you run both the compile and the tests in one run you'd have to split that so you don't ignore build errors)

Answer 3

Based Jacek Laskowski's solution you can do (at least in sbt >= 1.2.8):
testResultLogger in (Test, test) := TestResultLogger {
(log, results, taskName) =>
try {
(testResultLogger in (Test, test)).value.run(log, results, taskName)
} catch {
case _: TestsFailedException =>
println("Ignore TestsFailedException to get exit code 0")
}
}
If you have multi-module project, you can implement it as a plugin:
object TestExitCodePlugin extends AutoPlugin {
override def requires = JvmPlugin
override def trigger = allRequirements
override def projectSettings: Seq[Def.Setting[_]] = Seq(
testResultLogger in(Test, test) := TestResultLogger {
(log, results, taskName) =>
try {
(testResultLogger in(Test, test)).value.run(log, results, taskName)
} catch {
case _: TestsFailedException =>
println("Ignore TestsFailedException to get exit code 0")
}
}
)
}