How do I display "npm lint" results in Jenkins? I know how to display "pytest" results in Jenkins:
Run pytest like this: pytest --junitxml=./path/to/pytestFile.xml ./path/to/code
Run this Jenkins command: junit testResults: "./path/to/pytestFile.xml"
Once I do this, my test results appear in a "Test Results" window in Jenkins. I want to do something similar with my angular lint results. I tried doing this, but it didn't work:
Run linter like this: npm lint --junitxml=./path/to/lintFile.xml
Run this Jenkins command: junit testResults: "./path/to/lintFile.xml"
The problem is that npm doesn't have a --junitxml switch. How can I do this? I'm using the Jenkins declarative pipeline.
JUnit is a format for test results, not lint results. It makes no sense to try to use it.
The best plugin to use is Warnings Next Generation. It lists tslint as a supported format.
steps {
sh 'npm run-script --silent -- ng lint --format=checkstyle >checkstyle-result.xml'
}
post {
always {
recordIssues tool: tsLint(pattern: 'checkstyle-result.xml'),
enableForFailure: true
}
}
If there are multiple projects in your workspace, you may be affected by #14659. Either specify a single project or pipe the output through split -l 1.
As you can see, you then get a report specifically for TSLint, separate from other analysis issues and test results. It also understands the different severity levels, which cannot be expressed with JUnit.
Related
I'm running some JUnit5 tests in Jenkins, using a freestyle job and a Jenkins pipeline. These tests run fine using gradlew locally.
The pipeline groovy script (Jenkinsfile) uses
sh './gradlew clean test'
The code checks out fine and I see the Welcome to Gradle 5.5.1! message
Then, all of the tests which run fine locally with gradlew clean test fail with:
15:50:22
uk.co.motests.mfldirect.tests.UploadDataTest >
dropDownsAreDisplayed() FAILED
15:50:22 java.lang.IllegalStateException
Any idea where / how to start debugging this? This is literally the first time I've tried to run using a groovy pipeline script.
I'm posting this in the hope there is a common, beginners mistake that crops up frequently. I can post further details of the groovy file if required.
To print tests' stdout and stacktrace, enable Gradle INFO logging with sh './gradlew clean test -i'
To view DEBUG logging use sh './gradlew clean test -d'
All this data is already present in ./build/test-results/test/*.xml so you could browse the workspace of the failed Jenkins job.
I am running two sets of pytest tests, one for py27 and one for py37 (inside tox framework if it can help)
The test results are saved in two files: test_27.xml and test_37.xml
I'd like to differentiate between the Python versions in the Jenkins interface.
Is there any way to do that with Junit Jenkins plugin ?
Could you think of any workaround ?
Thanks !
I don't use Jenkins myself and have no idea what the Junit Jenkins plugin does, but as you mention tox: You can run the two environments in different pipelines running in parallel. They will each have their own reports then. You would define one pipeline calling tox -e py27 and the other calling tox -e py37 and you would have it all neatly separated.
The testenv would look something like this (see substitutions):
[testenv]
commands = pytest --junitxml=junit-{envname}.xml
Each pipeline would have their own reporting step, details about this can be glimpsed from this article by Dave Hunt.
This is the best solution I found so far: Change pytest testsuite name in xml report
On my Jenkins pipleline I run unit tests on both: debug and release configurations. Each test configuration generates separate JUnit XML results file. Test names on both: debug and release configuration are same. Currently I use the following junit command in order to show test results:
junit allowEmptyResults: true, healthScaleFactor: 0.0, keepLongStdio: true, testResults: 'Test-Dir/Artifacts/test_xml_reports_*/*.xml'
The problem is that on Jenkins UI both: debug and release tests results are shown together and it is not possible to know which test (from debug or release configuration) is failed.
Is it possible to show debug and release tests results separately? If yes, how can I do that?
We run the same integration tests against two different configurations with different DB types. We use maven and the failsafe plugin, so I take advantage of the -Dsurefire.reportNameSuffix so I can see the difference between the two runs.
The following is an example block of our Jenkinsfile:
stage('Integration test MySql') {
steps {
timeout(75) {
sh("mvn -e verify -DskipUnitTests=true -DtestConfigResource=conf/mysql-local.yaml " +
"-DintegrationForkCount=1 -DdbInitMode=migrations -Dmaven.test.failure.ignore=false " +
"-Dsurefire.reportNameSuffix=MYSQL")
}
}
post {
always {
junit '**/failsafe-reports/*MYSQL.xml'
}
}
}
In the report, the integration tests run against mysql then show up with MYSQL appended to their name.
It looks no solution for my question.
As a workaround I changed JUnit XML report format and included build variant name (debug/release) as a package name.
Trying to use the nifty Jenkins Pipeline, I had problems finding out how to publish NUnit test results.
I am able to run the tests by specifying the following command in the pipeline script:
stage 'Test'
bat '"C:\\Program Files (x86)\\NUnit 2.6.4\\bin\\nunit-console-x86.exe" "ProjectName\\bin\\Release\\UnitTests.net.dll"'
But how to make Jenkins "publish" the test results is not obvious. The Snippet Generator only suggests junit, and that does not seem to work.
I used nunit plugin version 0.21 and was able to publish results using
stage("PublishTestReport"){
nunit testResultsPattern: 'TestResult.xml'
}
(TestResult.xml is at the root of jenkins workspace in this above example)
Investigating the NUnit plugin for Jenkins led me on to this issue, where I found the solution:
step([$class: 'NUnitPublisher', testResultsPattern: 'TestResult.xml', debug: false,
keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
Adding this to the pipeline script worked for me!
However, it seemed the following should also work (but at the present, apparently, it does not): Using the Snippet Generator, select this:
step: General Build Step
Publish NUnit test result report
This generates the following in the Pipeline script:
step <object of type hudson.plugins.nunit.NUnitPublisher>
This fails!
Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.