I am using Jenkins to run unit-tests on a Django project via django-jenkins.
Recently, I found that some of the unit-tests wasn't being executed (someone had mistakenly changed an import, causing Jenkins to miss some of the unit-test files).
Is there a way to assert on the number of tests that Jenkins executes?, or the number of test-packages?
It could be tricky in python.
But you could add additional shell command that just looks into junit report.
cat reports/junit.xml | grep 'classname="your.missing.test.name"'
command would fail if grep doesn't found test record.
Or do the sudo apt-get install libxml-xpath-perl and assert on tests attibute value
[ $(xpath -q -e '//#tests' reports/junit.xml) == 'tests="1"' ]
Related
I am running two sets of pytest tests, one for py27 and one for py37 (inside tox framework if it can help)
The test results are saved in two files: test_27.xml and test_37.xml
I'd like to differentiate between the Python versions in the Jenkins interface.
Is there any way to do that with Junit Jenkins plugin ?
Could you think of any workaround ?
Thanks !
I don't use Jenkins myself and have no idea what the Junit Jenkins plugin does, but as you mention tox: You can run the two environments in different pipelines running in parallel. They will each have their own reports then. You would define one pipeline calling tox -e py27 and the other calling tox -e py37 and you would have it all neatly separated.
The testenv would look something like this (see substitutions):
[testenv]
commands = pytest --junitxml=junit-{envname}.xml
Each pipeline would have their own reporting step, details about this can be glimpsed from this article by Dave Hunt.
This is the best solution I found so far: Change pytest testsuite name in xml report
In a Jenkins freestyle job (on an older 1.6x version, no support for 2.x pipeline jobs) I would like to run a shell command (curl -XPOST ...) as a post build step if the build status recovered(!) from FAILED to SUCCESS.
However, all plugins for determining the build status I am aware of can only do something if the current build status IS FAILED or SUCCESS but don't take into account whether it recovered in comparison to the last build.
Is there any way how to achieve this, e.g. using the Groovy Post build plugin and some lines of scripting?
I found that something like this is a good way to go.
You can build up some interesting logic, and the "currentBuild" variable has some decent documentation here: currentBuild variable doc
script {
if ( ( currentBuild.resultIsBetterOrEqualTo("SUCCESS") && currentBuild.previousBuild.resultIsWorseOrEqualTo("UNSTABLE") ) || currentBuild.resultIsWorseOrEqualTo("UNSTABLE")) {
echo "If current build is good, and last build is bad, or current build is bad"
}
}
Meanwhile I found a way to achieve this. It is not necessarily pretty and I still appreciate alternative solutions :)
First of all, a plugin is needed which lets you execute shell commands in a Post Build step. There might be different ones, I am using the PostBuildScript plugin for that.
Then, create a "Execute a set of scripts" post build step, set the step to execute to Build step and select Execute shell, for me this looks like this:
In there, I run the following shell script lines which use my Jenkins server's REST API in combination with a Python one-liner (you could also use jq or something else for this) to determine the status of the current build as well as of the last completed build:
statusOfCurrentBuild=$(curl --silent "${BUILD_URL}api/json" | python -c "import sys, json; print json.load(sys.stdin)['result']")
statusOfLastBuild=$(curl --silent "${JOB_URL}/lastCompletedBuild/api/json" | python -c "import sys, json; print json.load(sys.stdin)['result']")
if [ "${statusOfCurrentBuild}" == "SUCCESS" ] && [ "${statusOfLastBuild}" == "FAILURE" ]
then
echo "Build was fixed"
# do something interesting here
fi
Depending on your Jenkins settings, using the REST API might require authentication.
I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would like to run them again few times and let them the chance to pass so my build won't be unstable/failed.
Is there any plugin I can use?
I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite or smething which is local.
But there is also a plugin which can retry a build:
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.
If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?
Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:
The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test
The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test
At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps.
#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do
echo "Running failed tests $i time(s)"
for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do
class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
ant -Dtest.class.pattern=${class} test
done
done
After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.
I am using jenkins to build a bunch of legacy code. The legacy code comes with some complex build scripts, written in csh.
The build scripts do not check for or exit on errors. The user is expected to scan the output for error messages. However, this does not work well with Jenkins.
I am executing the csh build scripts in a jenkins "shell execution" build step. For example:
export PATH=`pwd`/ALL/bin:/usr/local/bin:/usr/bin:/bin:$PATH
cd ATLb2.00/expt_02.0
csh 020.com
When I run this from the command line, I can also use the -e option:
csh -e 020.com
In this case, as I expect, the script is run, but when the first error is encountered, the script stops and returns a non-zero code. However, when I try this in Jenkins, the build fails as soon as it gets to the csh -e command, without executing any of the script.
The error I get in Jenkins is:
+ csh -e 020.com
Build step 'Execute shell' marked build as failure
On the command line, the script is run and I see all kinds of output, until something fails, and then the script exits. On Jenkins the script seems to fail without even running. There is no output, and even scripts with no failures will not run for me under jenkins with the -e option.
What's up?
I recommend that you specify csh on a more global level and then execute the commands in a Jenkins build step.
If you want to use csh for all jobs, you can set the default shell using Jenkins > Manage Jenkins > shell executable.
If you want to use csh for only a particular job, begin the Execute shell build step with a shebang, such as:
#!/usr/bin/tcsh -e -x
command1
command2
...
Since I have tested only tcsh, that is what I use in the example.
Beware that a space is not allowed after the #!:
#! /usr/bin/tcsh # Wrong
This will give the error,
java.io.IOException: Cannot run program ""
I tested the above on Jenkins 1.625.3
Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.