I want to integrate our Polymer test build process into Jenkins CI/CD pipeline using polymer-cli. I want to automate this steps to build our web-components test cases. At the moment, I am doing this manually by executing "polymer test" on the terminal. I also want the automated process to provide the reports that indicate failed test cases. Please provide step if you have implemented this in your CI/CD pipeline.
We have the following working setup for this case, using Jenkins2 Pipelines:
node{
....
stage('build'){
testresult = bat returnStatus: true, script: 'polymer test --plugin xunit-reporter -l chrome'
step([$class: 'XUnitBuilder', testTimeMargin: '3000', thresholdMode: 1, thresholds: [[$class: 'FailedThreshold', failureNewThreshold: '5', failureThreshold: '10', unstableNewThreshold: '2', unstableThreshold: '5'], [$class: 'SkippedThreshold', failureNewThreshold: '', failureThreshold: '', unstableNewThreshold: '', unstableThreshold: '']], tools: [[$class: 'XUnitDotNetTestType', deleteOutputFiles: true, failIfNotNew: true, pattern: 'TEST-wct.xml', skipNoTestFiles: false, stopProcessingIfError: true]]])
if (testresult != 0) {
currentBuild.result = "UNSTABLE"
echo "ALERT: Some Unittests failed, see log above."
}
}
....
}
The testresult is written by the wct-plugin https://github.com/garcus/wct-xunit-reporter to "/TEST-wct.xml".
The XUnit Builder step is configured as such, that it picks that file up and creates the testresult page. You can lookup the syntax for that in the Jenkins Pipeline syntax help page.
See also How to display the XUnit test output on Jenkins
and JUnit formatted test output
Related
In continuation to jenkins-pipeline-syntax-for-p4sync - I am not able to get the "Poll SCM" option work for my pipeline job.
Here is my configuration:
"Poll SCM" is checked and set to poll every 10 minutes
Pipeline script contains the following:
node ('some-node') // not actual value
{
stage ('checkout')
{
checkout([
$class: 'PerforceScm',
credential: '11111111-1111-1111-1111-11111111111', // not actual value
populate: [
$class: 'AutoCleanImpl',
delete: true,
modtime: false,
parallel: [
enable: false,
minbytes: '1024',
minfiles: '1',
path: '/usr/local/bin/p4',
threads: '4'
],
pin: '',
quiet: true,
replace: true
],
workspace: [
$class: 'ManualWorkspaceImpl',
charset: 'none',
name: 'jenkins-${NODE_NAME}-${JOB_NAME}',
pinHost: false,
spec: [
allwrite: false,
clobber: false,
compress: false,
line: 'LOCAL',
locked: false,
modtime: false,
rmdir: false,
streamName: '',
view: '//Depot/subfolder... //jenkins-${NODE_NAME}-${JOB_NAME}/...' // not actual value
]
]
]
)
}
stage ('now do something')
{
sh 'ls -la'
}
}
Ran the job manually once
Still, polling does not work and job does not have a "Perforce Software Polling Log" link like a non-pipelined job has when configuring the perforce source and Poll SCM in the GUI.
It's like the PerforceSCM is missing a poll: true setting - or i'm doing something wrong.
Currently I have a workaround in which I poll perforce in a non-pipelined job which triggers a pipelined job, but then I have to pass the changelists manually and I would rather the pipeline job to do everything.
edit: versions
jenkins - 2.7.4
P4 plugin - 1.4.8
Pipeline plugin - 2.4
Pipeline SCM Step plugin - 2.2
If you go to the Groovy snippet generator and check the "include in polling" checkbox, you'll see that the generated code includes a line item for it:
checkout([
poll: true,
As an aside, you may run into problems at the moment using ${NODE_NAME} in your workspace name. The polling runs on the master, so it might not properly find the change number of your previous build. If that's the case, I know a fix for it should be coming shortly.
After updating all the plugins to latest (as of this post date) and restarting the jenkins server - the polling appears to be working with the exact same configuration (job now has the poll log link).
I'm not sure what exactly resolved the issue - but I consider it resolved.
I have started porting my existing Jenkins job to Jenkins Pipeline.
When it came to port the Scan for compiler warning in Post-build action, I started getting issues.
First I tried to port PC-Lint. Used the following code
stage ('Warnings pclint') {
steps {
timeout(time: 5, unit: 'MINUTES') {
sh 'npm run lint:ci'
step([$class: 'WarningsPublisher',
parserConfigurations: [[
parserName: 'PC-Lint',
pattern: 'pmd.xml'
]],
unstableTotalAll: '0',
usePreviousBuildAsReference: true
])
}
}
}
But it didn't work. What is wrong in this ?
Is there any other way to do this ?
After searching a lot finally I got a working syntax.
step([$class: 'WarningsPublisher',
consoleParsers: [[parserName:'PC-Lint']],
defaultEncoding: '',
excludePattern: '',
healthy: '',
includePattern: '',
messagesPattern: '',
unHealthy: '',
useStableBuildAsReference: true
])
It is good to have this in the post build section of the Pipeline
I am using jenkins with the pipeline plugin, the XUnitBuilder plugin and the stash notifier plugin. I have the problem that it reports the wrong build status. My pipeline:
pipeline {
agent none
stages {
stage('Build') {
steps {
parallel(
failFast: true,
'One': {
node( 'windows' ){
run tests ...
step([$class: 'XUnitBuilder',
thresholds: [
[$class: 'FailedThreshold', failureNewThreshold: '0', failureThreshold: '0', unstableNewThreshold: '0', unstableThreshold: '0']
],
tools: [
[$class: 'CTestType', deleteOutputFiles: true, failIfNotNew: true, pattern: 'Testing/**/Test.xml']
]
])
}
},
'Two': {
node( 'windows' ){
run tests ...
step([$class: 'XUnitBuilder',
thresholds: [
[$class: 'FailedThreshold', failureNewThreshold: '0', failureThreshold: '0', unstableNewThreshold: '0', unstableThreshold: '0']
],
tools: [
[$class: 'CTestType', deleteOutputFiles: true, failIfNotNew: true, pattern: 'Testing/**/Test.xml']
]
])
}
}
)
}
}
}
}
Problem: As soon as 'One' is finished, the XUnitBuilder sets the build result to success which is reported to stash and my pull request can be merged. This is obviously wrong since 'Two' may be not even started.
I need to
* prevent the XUnitPublisher from setting the build result if it is success or
* prevent stash from being notified before the whole pipeline finished
Sure, i could stash the test results and run the XUnitBuilder in a post section but this is getting quite complicated since my pipeline has several stages each with several parallel steps running tests.
Does anyone know hot to fix this?
This guy seems to have the same problem.
Here is what I am doing:
stage('Test') {
steps {
bat "\"${tool 'VSTestRunner'}\" %WORKSPACE%\\MyApp\\bin\\Debug\\MyApp.dll /logger:trx & exit 0"
// The test publish is responsible for decided whether the test stage failed
step([$class : 'XUnitPublisher',
testTimeMargin: '3000',
thresholdMode: 1,
thresholds: [
[$class: 'FailedThreshold', failureNewThreshold: '', failureThreshold: '1', unstableNewThreshold: '', unstableThreshold: ''],
[$class: 'SkippedThreshold', failureNewThreshold: '', failureThreshold: '', unstableNewThreshold: '', unstableThreshold: '']
],
tools : [[$class: 'MSTestJunitHudsonTestType',
deleteOutputFiles: true,
failIfNotNew: false,
pattern: "TestResults\\*.trx",
skipNoTestFiles: false,
stopProcessingIfError: true
]]
])
}
}
If the tests fail the pipeline still continues.
I would like to be able to control whether the pipeline continues or not via the xunit settings. Is there a more elegant way of doing this than this:
if (currentBuild.result.equals("FAILURE")) {
throw "Test results did not pass thresholds"
}
Also, what does "stopProcessingIfError" do? I thought it would stop the pipeline if the error threshold was exceeded but it doesn't. Is there a param to the xunit service that does that?
FAILED, SUCCESS, and UNSTABLE are jenkins build status and not exit status. exit status should come from the process that is being executed. So, your solution is to set an exit 1 (or something non-zero), if the test fails. You batch step seem to be setting an exit 0. Check for the ERRORLEVEL as you seem to be in windows, and if its non-zero, break out from the pipeline by forcing an `exit 1'
note: it can be anything but should not be 0. (i am referring to the 1 in exit code)
I am running karma.js for unit testing and integrating with jenkins pipeline. My goal is to read the type of error thrown from Karma, if it is error related to percentage i want to terminate the job, otherwise continue even if there are other errors like unit test failures etc. (this is a requirement and there are reasons for it.)
I didn't find a way to do this. Any thoughts are appreciated!
karma start ibx-test/olb/karma.conf.js --browsers PhantomJS --log-level warn --single-run
coverageReporter: {
type: 'lcov',
dir: 'unit-tests/coverage/',
check: {
global: {
lines: 100 //This is just for testing
}
}
}
16:17:43 [Unit Test] 09 03 2017 21:17:43.024:ERROR [coverage]:
PhantomJS 2.1.1 (Linux 0.0.0): Coverage for lines (90.33%) does not
meet global threshold (100%)
EDIT: I found "Process xUnit test result report" in pipeline syntax under "Build step", can i use this somehow? Is there correlation between karma reports and xUnit?
i found a way to do this. The "Process xUnit test result report" helps doing exactly this. I checkedout the Pipeline syntax and it gave me the below script and it worked.
step([$class: 'XUnitBuilder', testTimeMargin: '3000', thresholdMode: 1,
thresholds: [
[$class: 'FailedThreshold', failureNewThreshold: '',
failureThreshold: '2', unstableNewThreshold: '',
unstableThreshold: '1'],
[$class: 'SkippedThreshold', failureNewThreshold: '',
failureThreshold: '', unstableNewThreshold: '',
unstableThreshold: '']],
tools: [[$class: 'JUnitType', deleteOutputFiles: false,
failIfNotNew: false, pattern: 'ibx-test/reports/unit-tests/PhantomJS_2.1.1_(Linux_0.0.0)/ibx-test/reports/unit-tests/*.xml',
skipNoTestFiles: false, stopProcessingIfError: false]]])
thresholdMode: means number of (failed or skipped) tests will be used for threshold. 1 for number and 2 for percent. I used 1, so i could just make one test fail and i get the desired result.
FailedThreshold: Is the class to be used for threshold for failures.
SkippedThreshold: can be used for skipped tests. I am not using it yet.
I am not paying attention on other parameters as of now for this testing.
As you can see my value is 2 (failureThreshold: '2'). As soon as i have 2 tests failing, the build fails and terminates.