Yesterday, I wrote and ran a Katalon test suite, and today, I'm trying to integrate Katalon with Jenkins. I successfully setup Jenkins, created a new job for the Katalon testing, as per these instructions, but when I went to build it, I get failing builds.
In particular, this is the error message I keep getting :
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Finished: FAILURE
I went ahead and copied the Reports folder structure from the project directory that I specified to the Jenkins workspace. Upon later inspection, I found that, when Jenkins was running the Katalon tests, the JUnit_Report.xml file was actually getting created in the project's Reports folder, instead of at %JENKINS_HOME%\workspace\[project name]\Reports. I explicitly told it to generate test reports to : Reports/LoginSuite/*/JUnit_Report.xml.
NOTE: I'm on a Windows machine.
How can I fix this so that I can display test results from Jenkins?!
UPDATE : I have revised my Windows shell code to the following
C:
cd C:\Katalon
katalon -runMode=console -projectPath="C:\Users\mwarren\Katalon Studio\TestProject" -reportFolder="../../.jenkins/workspace/Katalon Studio Tests/Reports" -reportFileName="report" -retry=0 -testSuitePath="Test Suites/LoginSuite" -browserType="Chrome"
and it's still giving me the same error, even though now the tests are being created there.
Report folder is generated in the jenkins job folder.
Reports/**/JUnit_Report.xml
PEBKAC, apparently. I should have, from the getgo, listened to Jenkins and set my Test Report XMLs as */JUnit_Report.xml
Copy all reports to a temp folder, rename each xml with test case name and then copy it back to junit folder.
testCasesTxt = sh (
script: 'sudo find $WORKSPACE -name "*.ts*" -type f -printf "%f\n"',
returnStdout: true
).trim()
testCasesTxt = testCasesTxt.replace(".ts", "")
testCases = testCasesTxt.split("\n")
for (int i = 0; i < testCases.size(); i++) {
script {
try {
wrap([$class: 'Xvfb']) {
sh """
cd /opt/katalon
./katalon -noSplash -consoleLog -runMode=console -projectPath=$WORKSPACE/"katalon-project.prj" -reportFolder="Reports" -reportFileName="report" -retry=0 -testSuitePath="Test Suites/${testCases[i]}" -executionProfile="qa" -browserType="Chrome"
"""
}
} catch (any) {
currentBuild.result = 'FAILURE'
throw any //rethrow exception to prevent the build from proceeding
} finally {
sh """
cd /home/environment/tmp/
cd Reports
mkdir ${testCases[i]}
cd $WORKSPACE
cp -r Reports/ /home/environment/tmp/Reports/${testCases[i]}
cd /home/environment/tmp/Reports/${testCases[i]}/Reports
mv JUnit_Report.xml JUnit_Report_${testCases[i]}.xml
cd $WORKSPACE
cp -r "Data Files/" "/home/environment/tmp/"
"""
//
}
}
}
According to this thread, this can happen when trying to execute two Katalon instances at the same moment.
If that's the case, try changing Jenkins number of executors 1.
If your report is at \Reports\20200611_172240\TestSuite1\20200611_172240/JUnit_Report.xml location in your project folder, then configure the folder path as below - /Reports///*/JUnit_Report.xml
as the 3 folders after report folder name are always going to change after each execution.
Please use the Test report XMLs path as like this
**/target/surefire-reports/*.xml
According to Jenkins Pipeline Docs, I should be able to use pipeline steps while working with Docker. However, "archiveArtifacts" does not appear to work:
def container = docker.image("some_image")
container.inside {
sh 'date > /tmp/test.txt'
sh 'cat /tmp/test.txt' //works, shows file
def fileContents = readFile '/tmp/test.txt' //works
echo "Contents: ${fileContents}" //works, shows file
archiveArtifacts '/tmp/*.txt' //FAILS
}
"ERROR: No artifacts found that match the file pattern "/tmp/*.txt". Configuration error?".
Things I've tried:
Adding sync and sleep(5) before the archive step in case this is a file sync issue.
Trying to archive '/*' and '*' in case it's running on the host (same error).
Any suggestions on archiving files generated in a Docker container?
PS: I opened a bug report... It looks like archiveArtifacts will only work on files in $WORKSPACE in docker containers.
You seem to have found the solution as reported in the same Jira ticket so I'll post here for everyone:
this works fine:
def image = docker.image("alpine")
image.inside {
sh 'date > /tmp/test.txt'
sh "cp /tmp/test.txt ${WORKSPACE}"
archiveArtifacts 'test.txt'
}
I'm trying to execute a jar file built in a different jenkins job. For that I use the CopyArtifact plugin.
Copying the artefact seems to work. The following code
step ([$class: 'CopyArtifact',
projectName: '/BuildJob/master',
filter: 'target/mytool.jar',
target: './']);
finishes with
Copied 1 artifact from "BuildJob » master" build number 1
But how can I access this copied jar file? I tried the following:
sh(returnStdout: true, script: "java -jar mytool.jar")
but this ended up in the following error message:
Error: Unable to access jarfile mytool.jar
I have a build that calls Yslow test. The build fails due to some of the yslow tests fail (which I am aware of).
Is there a way to set it so Jenkins does not mark the build "failed" regardless of the yslow tests?
Shell:
/usr/local/bin/phantomjs yslow.js -i grade -t 50 -f junit http://www.website.com > yslow.xml
Console Output:
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE
You could use the groovyPostbuild plugin for this. Something like the following should work:
if(manager.logContains(".*Build step 'Execute shell' marked build as failure.*")) {
manager.build.#result = hudson.model.Result.SUCCESS
}
I successfully setup a casperjs test exporting a "result.xml" file.
In Jenkins, I execute the following shell command:
casperjs /home/testing-radu/generated-test.js
This produces "results.xml" in my build's workspace.
In post-build actions I added "Publish XUnit test result report". I don't know how and where to setup the path to the .xml file.
When my build console I see the following:
[37;42;1mPASS 2 tests executed in 9.96s, 2 passed, 0 failed. [0m
[32;1mResult log stored in results.xml [0m
[xUnit] [INFO] - Starting to record.
ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
/var/lib/jenkins/jobs/17live2/workspace/generatedJUnitFiles does not exist.
I added full rights to the workspace directory. I tried creating "generatedJUnitFiles" folder and here is what I'm getting:
[37;42;1mPASS 2 tests executed in 5.296s, 2 passed, 0 failed. [0m
[32;1mResult log stored in results.xml [0m
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Setting the build status to FAILURE
[xUnit] [INFO] - Stopping recording.
Build step 'Publish xUnit test result report' changed build result to FAILURE
Finished: FAILURE
Am I missing something?
Allright, here's the deal:
Under publish XUnit test results I chose JUnit (also setup the pattern to *.xml) and then
created a shell script to be executed by Jenkins:
# Auth tests
for f in auth/*.js ; do casperjs "$f"; done;
# Clean the old results and place the new ones
rm /var/lib/jenkins/jobs/17live2/workspace/*.xml
mv *.xml /var/lib/jenkins/jobs/17live2/workspace/
In Jenkins config I just do: ./run.sh
***You need read/write rights over your folders.