Allure results are generated (xmls) and can be viewed by running allure serve outside the allure-results directory but when configured in Jenkins using the allure results directory with relative path from Jenkins workspace, the allure results are not found. Presumably there is something wrong with the path
[allure_test] $ /Users/me/.jenkins/tools/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation/Allure_2.7.0/bin/allure generate -c -o /Users/me/.jenkins/workspace/allure_test/allure-report
allure-results does not exists
Report successfully generated to /Users/me/.jenkins/workspace/allure_test/allure-report
Allure report was successfully generated.
Relative Path defined for allure report is
../../repos/partner-portal/target/allure-results
The report is generated but with no results
Here is the solution:
1. Inside your work space (For me it is D:\m\Automation Project\Tests), Create a folder named "target" containing two subfolders "allure-results" and "allure-reports".
In Post Build Actions of Jenkins Project provide these entries.
Results: target/allure-results
Report Path: target/allure-reports
Now run your tests and allure reports will be generated without any issues.
I struggled for over 24 hours and could not sleep whole night just because of this issue. There were no clear instructions related to this and how it works. Finally, I figured it out.
May be its little too late for this answer but could you please create the "allure-results" folder in your workspace (like /Users/me/.jenkins/workspace/allure_test/allure-results)?
After I configure Allure report in jenkins, and using behave command, I finally see following output and saw the working report in my jenkins build plan:
.
.
.
.
D:\PythonProject\PythonBehave>behave -f allure_behave.formatter:AllureFormatter -o C:\Users\Alex\.jenkins\workspace\PythonAllure\allure-results D:\PythonProject\PythonBehave\features
Failing scenarios:
features/example.feature:17 user can search text in google -- #2.2 Incorrect
features/github_login.feature:13 User attempt to login with wrong username and password -- #1.1 InCorrect
features/github_login.feature:14 User attempt to login with wrong username and password -- #1.2 InCorrect
features/github_login.feature:18 User attempt to login with correct username and password
0 features passed, 2 failed, 0 skipped
3 scenarios passed, 4 failed, 0 skipped
12 steps passed, 4 failed, 8 skipped, 0 undefined
Took 3m24.651s
D:\PythonProject\PythonBehave>exit 1
Build step 'Custom Python Builder' marked build as failure
[PythonAllure] $ C:\Users\Alex\.jenkins\tools\ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation\Allure_2.13.1\bin\allure.bat generate C:\Users\Alex\.jenkins\workspace\PythonAllure\allure-results -c -o C:\Users\Alex\.jenkins\workspace\PythonAllure\allure-report
Report successfully generated to C:\Users\Alex\.jenkins\workspace\PythonAllure\allure-report
Allure report was successfully generated.
Creating artifact for the build.
Artifact was added to the build.
Finished: FAILURE
And before I used to get this output when there was no allure-results folder in my workspace:
[PythonAllure] $ C:\Users\Alex\.jenkins\tools\ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation\Allure_2.13.1\bin\allure.bat generate -c -o C:\Users\Alex\.jenkins\workspace\PythonAllure\allure-report
we are not using any post-build steps on UI but using jenkins pipeline groovy script as below:
def allureReportsGenerationTask() {
try {
allure([includeProperties: false, jdk: '', properties: [], reportBuildPolicy: 'ALWAYS', results: [[path: 'target/allure-results']]])
} catch(Exception error) {
println("Caught Exception: ${error}")
}
}
The project structure was like that only
ProjectRepo
- src
- target
-alure-results
- pom.xml
- testng.xml
In this case, we are getting the allure-results getting generated but it was not getting picked in the Jenkins pipeline allure plugin. ON UI below command was running always
/opt/jenkins/tools/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation/allure/bin/allure generate -c -o /opt/jenkins/workspace/ProjectRepoPipeline/allure-report
you can see Jenkins was not able to find allure-results for generating parameters in the above Jenkins pipeline command
We added the below parameter in pom.xml maven-surefire-plugin
<systemPropertyVariables>
<allure.results.directory>../target/allure-results</allure.results.directory>
</systemPropertyVariables>
Now project structure becomes
-ProjectRepo
- src
- pom.xml
- testng.xml
target
-allure-results
Now on running the same via pipeline script, we were able to run the below command
/opt/jenkins/tools/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation/allure/bin/allure generate /opt/jenkins/workspace/ProjectRepoPipeline/target/allure-results -c -o /opt/jenkins/workspace/ProjectRepoPipeline/allure-report
ProjectRepoPipeline - This is the Jenkins pipeline name that is running CI/CD
Related
Could you please help with the follow?
Running the protractor test.
Jenkins plugin is not able to generate report.
Error: 404 not found.
Allure plugin: 2.19
Allure commandline: 1.54
Protractor: 5.1.2
Jenkins is run locally: 2.60.1
NodeJS: 6.9.4
NPM: 4.0.5
Error stack trace: allure-report for the report is already in use, add a '--clean' option to overwrite.
Stack trace:
[11:56:19] I/launcher - 0 instance(s) of WebDriver still running
[11:56:19] I/launcher - chrome #01 failed 1 test(s)
[11:56:19] I/launcher - overall: 1 failed spec(s)
[11:56:19] E/launcher - Process exited with error code 1
npm ERR! Test failed. See above for more details.
Build step 'Execute Windows batch command' marked build as failure
[protractor-allure-reporting-example] $ D:\Jenkins\tools\ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation\allure-commandline\allure-2.2.1\bin\allure.bat generate
D:\Jenkins\workspace\protractor-allure-reporting-example\first-result -o
D:\Jenkins\workspace\protractor-allure-reporting-example\allure-report
Allure: Target directory D:\Jenkins\workspace\protractor-allure-reporting-
example\allure-report for the report is already in use, add a '--clean' option to overwrite
ERROR: Build step failed with exception ru.yandex.qatools.allure.jenkins.exception.AllurePluginException: Can not generate Allure Report, exit code: 1
at ru.yandex.qatools.allure.jenkins.AllureReportPublisher.generateReport(AllureReportPublisher.java:173)
at ru.yandex.qatools.allure.jenkins.AllureReportPublisher.perform(AllureReportPublisher.java:94)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Allure Report' marked build as failure
Finished: FAILURE
Seems like you need to clean your workspace before build. Or you can add one more step before report generation that will remove allure-report directory.
Add it in package.json scripts
"posttest": "allure generate allure-results --clean -o allure-report || true && allure report open -o allure-report"
try to check that in the docker container where your allure report generated not able to detect java_home. So try to set your report path where your container ends and then your allure report will able to find java_home.
and able to generate report.
Use below command to check the same:
sh "/var/lib/.../ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation/../bin/allure --version"
I've got a custom Ant task that I'm using successfully from gradle on my local machine:
task fetchRelMod {
doLast {
println 'Fetching the RelMod'
ant.taskdef(name:'relmod',
classpath:'retrievePBSInfo.jar:hsjt400-4-9.jar',
classname:"com.myco.ant.tasks.RetrievePBSRelModString")
ant.relmod(user:project.ext.props.getProperty('fetchrelmod.username'),
password:project.ext.props.getProperty('fetchrelmod.password'),
prodCode:project.ext.props.getProperty('profile.pbs.product.code'),
branch:project.ext.props.getProperty('profile.pbs.branch'),
state:project.ext.props.getProperty('profile.pbs.relmod.selector'))
project.ext.set('iseries_relmod',ant.relmodStub)
project.ext.set('iseries_relmodAndDate', ant.relmod)
}
}
I've got the jar files sitting next to build.gradle for now, out of simplicity... they exist in the same location on the build server. Works great locally. When I run my build from my build server (either through Jenkins or going on the box and running Gradle directly), I get the following:
sudo /var/lib/jenkins/tools/hudson.plugins.gradle.GradleInstallation/gradle214/bin/gradle all -DisQUABuild=true
Building My App
Loading Properties files...
QUA Build. Using build-qua.props
:fetchRelMod
Fetching the RelMod
:fetchRelMod FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/var/lib/jenkins/workspace/MyApp/build.gradle' line: 141
* What went wrong:
Execution failed for task ':fetchRelMod'.
> taskdef class com.myco.ant.tasks.RetrievePBSRelModString cannot be found
using the classloader AntClassLoader[/var/lib/jenkins/workspace/myApp/hsjt400-4-9.jar]
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 3.104 secs
What concerns me is that there are two jar files in the classpath and it only mentions one in the error. Does anyone have any ideas as to what might be going on?
While trying to execute any vNext build with a Publish Artifact task we end up getting the following error
##[error]Unable to associate artifact with resource at location:[location]
example log output
1
2016-01-21T19:29:49.9838278Z Executing the powershell script: D:\Agent\tasks\PublishBuildArtifacts\1.0.4\PublishBuildArtifacts.ps1
2
2016-01-21T19:29:50.1713333Z Preparing artifact content in staging folder D:\Agent\_work\1\a...
3
2016-01-21T19:29:50.2338323Z Copying all files from D:\Agent\_work\1\s\ASC\ASC-Dev\CSDChangeControlDispatcher to D:\Agent\_work\1\a\CSDChangeControlDispatcher\CSDChangeControlDispatcher
4
2016-01-21T19:29:50.4838343Z Copying all files from D:\Agent\_work\1\s\ASC\ASC-Dev\PrecompiledWeb\CSDChangeControlDispatcher to D:\Agent\_work\1\a\CSDChangeControlDispatcher\PrecompiledWeb\CSDChangeControlDispatcher
5
2016-01-21T19:29:50.5150903Z Copying artifact content to D:\BuildDrop\ASC Dev vNext\20160121.7...
6
2016-01-21T19:29:50.9369728Z Start: AssociateArtifact
7
2016-01-21T19:29:50.9369728Z ##[error]Unable to associate artifact with resource at location:D:\BuildDrop\ASC Dev vNext\20160121.7.
8
2016-01-21T19:29:50.9369728Z End: AssociateArtifact
The issue is you specify the Path of Publish Artifact as "D:\BuildDrop\ASC Dev vNext\20160121.7".
You should use "\ \ my\share" in Path item instead of a absolute path.
You will also get this error if the path has a folder name with brackets in it. For example \\servername\afolder\[foldername].
Currently we are using IntelliJ, Scala, SBT to kick off our tests in our local enviroment. With SBT command line, we can specify specific test, suites, wildcards, as instructed here:
ScalaTest.org Page
Such as "test-only *RedSuite"
However on our CI Jenkins server, with the SBT pluging, when specifying this, it gives an error.
In the action field, the following values were used:
Action:compile test-only test.package.name
Using the following does work for ALL tests:
Action:compile test
[success] Total time: 240 s, completed Apr 28, 2014 12:29:36 PM
[error] Expected ID character
[error] Not a valid command: org (similar: export)
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Not a valid key: org (similar: fork, run, doc)
[error] org.company.scalatest.abc.regressionsuite
[error] ^
Build step 'Build using sbt' changed build result to FAILURE
Build step 'Build using sbt' marked build as failure
Recording test results
Does anyone know if there is a way we can pass these parameters through the jenkins SBT plugin?
This is a quoting problem, your Action field is parsed as 3 commands:
compile
test-only
org.company.scalatest.abc.regressionsuite
And it chokes because that org is not a valid command.
Using compile "test-only org.company.scalatest.abc.regressionsuite" should fix that.
I successfully run my unit test with google test in Jenkins, but I don't know how to show the .xml file generated by gtest. It is said that gtest satisfies the JUnit format, so what I set is as follows:
But it ends up with errors after a building.
No test report files were found. Configuration error?
Build step 'Publish JUnit test result report' changed build result to FAILURE
Finished: FAILURE
Fraser's answer is good and you need some extra processing to convert the gtest XML to proper JTest format.
First you ask gtest to output the result to XML using:
mygtestapp --gtest_output=xml:gtestresults.xml
Then in a script you need to add extra elements to properly flag skipped tests as such. Jenkin's JTest processor requires that a skipped test contains the <skipped> element and not just setting status to "notrun":
awk '{ if ($1 == "<testcase" && match($0, "notrun")) print substr($0,0,length($0)-2) "><skipped/></testcase>"; else print $0;}' gtestresults.xml > gtestresults-skipped.xml
mv gtestresults.xml gtestresults.off
If running this on a windows batch file, put the awk action inside a file to avoid problems with the quotes. awk.progfile:
{ if ($1 == "<testcase" && match($0, "notrun")) print substr($0,0,length($0)-2) "><skipped/></testcase>"; else print $0;}
And create add in your bat file:
awk -f awk.progfile gtestresults.xml > gtestresults-skipped.xml
Lastly you point the JTest processor as a Post-Build step to read the converted XML:
# Publish JUnit Test Result Report
Test Report XMLs: gtestresults-skipped.xml
Are you running your test executable with the correct flags (i.e. --gtest_output=xml[:DIRECTORY_PATH\|:FILE_PATH])?
From the --help output:
--gtest_output=xml[:DIRECTORY_PATH\|:FILE_PATH]
Generate an XML report in the given directory or with the given file
name. FILE_PATH defaults to test_details.xml.
The error on the Jenkins configuration page is a bit of a red herring.
Essentially, what's happening is that the test report xml file hasn't been generated by the build job. So, you then get this error:
Recording test results
No test report files were found. Configuration error?
Of course, the location must be configured correctly. For that, see this post:
How to configure test-reports on Jenkins CI working with grails?
So, how to fix the error? The key is to study the console output to check whether the tests did successfully run. Chances are they didn't, and that's why the error has happened.
Once you get the tests running successfully, assuming that you correctly configured the location, you should be ok.
You're using JUnit so it'll be a Java project. I'll note here in case it may help others, that we were running Xcode. The tests weren't being run.
Buried in Jenkins console output just above the error was this note:
note: RunUnitTests exited without running tests because TEST_AFTER_BUILD was set to NO.
Popping back into Xcode, and setting the UnitTests target's Test After Build flag to YES did the trick. That's under the Unit Testing section. You can also of course set the flag at the Project level, and change the target's to 'Other', setting a value of $(inherited).
Your results file is not stored at correct location and Jenkins plugin cannot find it. After your tests are executed and XML file is generated do you store it anywhere?
I suggest try make it working by replacing result.xml with '*' (assuming this is the only XML file that is supposed to be stored there) and if this is going to work then start working on correct file name.
We had the same issue in our configuration. Making sure that generated result XML is stored where the plugin expect it was the key. You can determine workspace root from your project config.
Jenkins has xunit plugin that converts googletest xml to junit format: https://plugins.jenkins.io/xunit/.
Example of pipeline
pipeline {
agent any
stages {
stage('Test'){
steps {
sh "run_tests.bash"
}
}
}
post {
always{
xunit (
thresholds: [ skipped(failureThreshold: '0'), failed(failureThreshold: '0') ],
tools: [ GoogleTest(pattern: 'reports/*.xml') ]
)
}
}
}
Other useful links:
https://jenkins.io/doc/pipeline/steps/xunit/
https://jenkins.io/blog/2016/10/31/xunit-reporting/
Here is a windows batch version for converting the google-test "notRun" to junit "skipped" via windows batch. I know that there are more elegant ways, but this one only requires windows batch and does the job
rem convert gtest xml to junit compatible format (replace notRun by skipped)
IF EXIST %INTEXTFILE% (
IF EXIST %OUTTEXTFILE% (
del %OUTTEXTFILE%
waitfor fileSystemToDoItsStuff /t 1
)
FOR /f "tokens=1,* delims=ΒΆ" %%A IN ( '"type %INTEXTFILE%"') DO (
ECHO."%%A" | findstr /C:"DISABLED_">nul & IF ERRORLEVEL 1 (
SET modified=%%A
) ELSE (
SET string=%%A
SET modified=!string:/^>=^>^<skipped /^>^</testcase^>!
)
ECHO !modified!>> %OUTTEXTFILE%
)
del %INTEXTFILE%
waitfor fileSystemToDoItsStuff /t 1
move %OUTTEXTFILE% %INTEXTFILE%
)