We have a CI pipeline on Jenkins. Under test is a REST api running in a Docker container. It is tested with frisby.js which also uses jasmine.
Test are successful
Finished in 0.168 seconds
1 test, 4 assertions, 0 failures, 0 skipped
but xUnit sets the build status to FAILURE:
[xUnit] [INFO] - Converting '/path/to/TEST-FrisbyTestMyApiTest.xml' .
[xUnit] [INFO] - Check 'Failed Tests' threshold.
[xUnit] [INFO] - Check 'Skipped Tests' threshold.
[xUnit] [INFO] - Setting the build status to FAILURE
Why is that? How can I get it to be SUCCESS?
The paths are validated and correct. The tests run successful. I'm running the Jenkins xUnit plugin with JUnit pattern. All xUnit thresholds are set to 0.
Related
jenkins pipeline junit use a weird metric for the health of a job,
https://www.jenkins.io/doc/pipeline/steps/junit/ and if one test fail will mark the stage/job unstable
is it possible to tell the stage if 20% of the test fail then mark it as unstable else is sucesfull?
i created a python script that read the result xml, and got the failed test and the totals
and run pytest inside a python script and not from the cli, that way i can control the error code. since junit addon will mark it unstable if some tests fail since the error code will be other than zero, so my script
pytest_result = pytest.main([my tests, "--junitxml", f"{args.reports_folder}/xml/*name*.xml", "-o", "junit_suite_name=*name*"] )
parse the xml with import xml.etree.ElementTree as ET to get the amount of fail and total amount of tests
percentage_fail = (failed_tests/total_tests)*100
after i compare it to the threshold
if percentage_fail > threshold:
raise SystemExit(1)
else:
print("All test working")
raise SystemExit(0)
junit will not let you change the status of the build if become 'UNSTABLE' the trick is control the error code that comes from running the tests. before passing the report.xml to junit
There is a similar issue here but with no answers and no progress in solving it.
I am running Selenium tests with Cucumber and with Gradle as the build tool. In Jenkins.
This is my build.gradle file:
task cucumber() {
dependsOn assemble, compileTestJava
doLast{
javaexec {
main = "cucumber.api.cli.Main"
classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
args = ['--plugin', 'pretty', '--glue', 'stepmethods', 'src/main/java/features', '--tags', 'not #proba and not #test and not #rucno and not #nedovrseno']
}
}
}
I wanted to exclude some scenarios from the feature I am testing so I added the '--tags', 'not #proba and not #test and not #rucno and not #nedovrseno' part to Gradle task arguments.
After that (and I'm not 100% sure this is the cause) tests run fine in a Jenkins job, but the reports with Cucumber reports plugin are not getting generated.
Instead, this is the Jenkins console output:
15:09:02 BUILD SUCCESSFUL in 1m 58s
15:09:02 3 actionable tasks: 3 executed
15:09:02 Build step 'Invoke Gradle script' changed build result to SUCCESS
15:09:02 [CucumberReport] Using Cucumber Reports version 4.6.0
15:09:02 [CucumberReport] JSON report directory is ""
15:09:02 [CucumberReport] Copied 1 json files from workspace "C:\Users\me\.jenkins\workspace\Project Name" to reports directory "C:\Users\me\.jenkins\jobs\Project Name\builds\12\cucumber-html-reports\.cache"
15:09:02 [CucumberReport] Copied 4 properties files from workspace "C:\Users\me\.jenkins\workspace\Project Name" to reports directory "C:\Users\me\.jenkins\jobs\Project Name\builds\12\cucumber-html-reports\.cache"
15:09:02 [CucumberReport] Processing 1 json files:
15:09:02 [CucumberReport] C:\Users\me\.jenkins\jobs\Project Name\builds\12\cucumber-html-reports\.cache\report.json
15:09:02 [CucumberReport] Missing report result - report was not successfully completed
15:09:02 [CucumberReport] Build status is left unchanged
15:09:02 Finished: SUCCESS
So, the files are processed, but the report is missing. What could be the problem?
EDIT
Added second '--plugin' to string to account for the OP's finding.
Removed additional examples.
Try
task cucumber() {
dependsOn assemble, compileTestJava
doLast{
javaexec {
main = "cucumber.api.cli.Main"
classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
args = ['--plugin', 'pretty', '--plugin', 'html:some/dir', '--glue', 'stepmethods', 'src/main/java/features', '--tags', 'not #proba and not #test and not #rucno and not #nedovrseno']
}
}
}
You didn't specify which report you wanted nor where you wanted it generated. This gives you pretty and html.
I have this Jenkinsfile:
node{
stage ('Checkout')
{
checkout scm
}
stage ('Build')
{
try {
sh '''
mvn clean -B org.jacoco:jacoco-maven-plugin:prepare-agent install
'''
} catch (err) {
// do nothing
currentBuild.result = 'FAILED'
} finally {
//step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])
step([$class: 'XUnitBuilder', testTimeMargin: '3000', thresholdMode: 1,
thresholds: [
[$class: 'FailedThreshold', unstableThreshold: '2'],
[$class: 'SkippedThreshold', unstableThreshold: '']],
tools: [
[$class: 'JUnitType', deleteOutputFiles: false, failIfNotNew: false, pattern: '**/target/surefire-reports/TEST-*.xml', skipNoTestFiles: true, stopProcessingIfError: false]]
])
}
}
}
I want to mark the build as UNSTABLE when : some tests are failed using threshold of xUnit plugin, and I want to mark it FAILED if there is any error in the code like syntax error.
The problem here is that catch is executed in both situations: code error and test failure. How can I configure the Jenkinsfile to make the difference ?
After running a build that has 3 test failures ( threshold is 2 ), I expected the build to be UNSTABLE but it turned to FAILED:
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing JUnit
[xUnit] [INFO] - [JUnit] - 1 test report file(s) were found with the pattern '**/target/surefire-reports/TEST-*.xml' relative to '/var/lib/jenkins/workspace/jenkins-failure_master-EHKMAGGDJWHA7FNGHQYY2VJNJHBX2RXI3XJY4NU5EB2PYT4ERTZQ' for the testing framework 'JUnit'.
[xUnit] [INFO] - Check 'Failed Tests' threshold.
[xUnit] [INFO] - The total number of tests for this category exceeds the specified 'unstable' threshold value.
[xUnit] [INFO] - Setting the build status to FAILURE
[xUnit] [INFO] - Stopping recording.
New case: I commented the line currentBuild.result = 'FAILED' and I added a syntax error in the test class. The build is successed even with the COMPILATION ERROR and it skipped the tests:
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing JUnit
[xUnit] [INFO] - [JUnit] - No test report file(s) were found with the pattern '**/target/surefire-reports/TEST-*.xml' relative to '/var/lib/jenkins/workspace/jenkins-failure_master-EHKMAGGDJWHA7FNGHQYY2VJNJHBX2RXI3XJY4NU5EB2PYT4ERTZQ' for the testing framework 'JUnit'. Did you enter a pattern relative to the correct directory? Did you generate the result report(s) for 'JUnit'?
[xUnit] [WARNING] - No test reports found for the metric 'JUnit' with the resolved pattern '**/target/surefire-reports/TEST-*.xml'.
[xUnit] [INFO] - Skipping the metric tool processing.
[xUnit] [INFO] - There are errors when processing test results.
[xUnit] [INFO] - Skipping tests recording.
From Reporting test results and storing artifacts (declarative pipeline):
pipeline {
agent { docker "java" }
stages {
stage("build") {
steps {
sh 'mvn clean install -Dmaven.test.failure.ignore=true'
}
}
}
post {
always {
archive "target/**/*"
junit 'target/surefire-reports/*.xml'
}
}
}
Use -Dmaven.test.failure.ignore=true to prevent failing the build on test failures. The junit step then collects the test results and marks the build as unstable.
This is also how it is done under the covers when using a non-pipeline Maven job, see JENKINS-24655.
I have a build that calls Yslow test. The build fails due to some of the yslow tests fail (which I am aware of).
Is there a way to set it so Jenkins does not mark the build "failed" regardless of the yslow tests?
Shell:
/usr/local/bin/phantomjs yslow.js -i grade -t 50 -f junit http://www.website.com > yslow.xml
Console Output:
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE
You could use the groovyPostbuild plugin for this. Something like the following should work:
if(manager.logContains(".*Build step 'Execute shell' marked build as failure.*")) {
manager.build.#result = hudson.model.Result.SUCCESS
}
I successfully setup a casperjs test exporting a "result.xml" file.
In Jenkins, I execute the following shell command:
casperjs /home/testing-radu/generated-test.js
This produces "results.xml" in my build's workspace.
In post-build actions I added "Publish XUnit test result report". I don't know how and where to setup the path to the .xml file.
When my build console I see the following:
[37;42;1mPASS 2 tests executed in 9.96s, 2 passed, 0 failed. [0m
[32;1mResult log stored in results.xml [0m
[xUnit] [INFO] - Starting to record.
ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
/var/lib/jenkins/jobs/17live2/workspace/generatedJUnitFiles does not exist.
I added full rights to the workspace directory. I tried creating "generatedJUnitFiles" folder and here is what I'm getting:
[37;42;1mPASS 2 tests executed in 5.296s, 2 passed, 0 failed. [0m
[32;1mResult log stored in results.xml [0m
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Setting the build status to FAILURE
[xUnit] [INFO] - Stopping recording.
Build step 'Publish xUnit test result report' changed build result to FAILURE
Finished: FAILURE
Am I missing something?
Allright, here's the deal:
Under publish XUnit test results I chose JUnit (also setup the pattern to *.xml) and then
created a shell script to be executed by Jenkins:
# Auth tests
for f in auth/*.js ; do casperjs "$f"; done;
# Clean the old results and place the new ones
rm /var/lib/jenkins/jobs/17live2/workspace/*.xml
mv *.xml /var/lib/jenkins/jobs/17live2/workspace/
In Jenkins config I just do: ./run.sh
***You need read/write rights over your folders.