Rerun flaky JUnit test in case they failed - jenkins

I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would like to run them again few times and let them the chance to pass so my build won't be unstable/failed.
Is there any plugin I can use?

I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite or smething which is local.
But there is also a plugin which can retry a build:
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.
If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?

Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:
The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test
The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test
At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps.
#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do
echo "Running failed tests $i time(s)"
for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do
class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
ant -Dtest.class.pattern=${class} test
done
done
After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.

Related

How can I get jenkins to fail a build if the tests fail, after the reports are generated?

So I'm trying to learn more about jenkins and I developed a shell script that execute the tests for my iOS project and generates a report. I set the script to exit as soon as failures are detected (Since this is the only way I know how to get the build to fail if the tests fail).
That's the reason why the reports were not being generated. Because I exit the script as soon as the tests failed. But I'm not familiar with how to get the build to fail, without exiting prior to the generation of the reports. What can I do to circumvent this? Here's my current shells script:
set -o pipefail && xcodebuild -project "Tests.xcodeproj"
-scheme "Testing"
-sdk "iphonesimulator12.2"
-destination "platform=iOS Simulator,OS=latest,name=iPhone 7"
test -only-testing:"UITests/UITests"
-resultBundlePath TestResults | xcpretty
xchtmlreport -r TestResults
Typically you would execute all your tests and report each test case state in a result file (in my case this is in JUnit format for Java/JavaScript).
After all tests have run you publish the result file using an appropriate Jenkins plugin step (in my case junit).
you can use
always() { //some code here }
section to specific stage OR pipeline. Put the report generation in that block. Refer to https://jenkins.io/doc/book/pipeline/syntax/#post-conditions
In freestyle project you can add Post-build action
Pipeline/Workflow Job in Jenkins using fastlane: What I did, is I set the post block to run three different lanes depending on the the status of the build. The post block has sublocks called success, unstable and failure. The lanes in the post block's subblocks post a slack message with SUCCESS, UNSTABLE, or FAILED for the three possibilities (successful build and all tests pass, successful build and 1 or more tests fail, and unsuccessful build) respectively. I use danger gem to post test result summary back to my pull request in the repository's comments. See here --> https://www.jenkins.io/doc/book/pipeline/syntax/#post

Jenkins Job fails when pytest test fails

Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.

Asserting Number of executed unit-tests

I am using Jenkins to run unit-tests on a Django project via django-jenkins.
Recently, I found that some of the unit-tests wasn't being executed (someone had mistakenly changed an import, causing Jenkins to miss some of the unit-test files).
Is there a way to assert on the number of tests that Jenkins executes?, or the number of test-packages?
It could be tricky in python.
But you could add additional shell command that just looks into junit report.
cat reports/junit.xml | grep 'classname="your.missing.test.name"'
command would fail if grep doesn't found test record.
Or do the sudo apt-get install libxml-xpath-perl and assert on tests attibute value
[ $(xpath -q -e '//#tests' reports/junit.xml) == 'tests="1"' ]

YSlow Phantomjs and Jenkins jobs failing, but analysis successful

I'm going through the tutorial on YSlow and Phantom js in Jenkins here: http://yslow.org/phantomjs/
Everything appears to be working great except the Jenkins builds are failing. I think this is due to the violations that YSlow is finding (6 for the particular site I am measuring). I'd rather have the build be successful (or unstable) vs. failed though
Is that possible with this or will I have to resort to something like the postgroovy or text finder plugin?
This is the console output:
phantomjs.exe yslow.js -i grade -t 50 --format junit http://www.somesite.com 1>yslow.xml
D:\Apps\Jenkins\workspace\YSlow_Test>exit 6
Build step 'Execute Windows batch command' marked build as failure
Thanks
Any non-zero exit code at the end of your Execute Windows batch command build step will result in build step being marked as failure.
To have the build step marked as success, you need an exit code of 0. I don't know anything about "yslow" or "phantomjs" and why they are giving you exit code of non-zero, but from "batch" side of things, you need only write exit 0 at the end of your build step if you want to overwrite the exit code of your phantomjs command.
You can then use Text Finder plugin to parse the console log and mark build as unstable when certain conditions are met.
Reading over this answer, Configuring yslow on Jenkins looks like you need TAP plugin to have the functionality of unit testing marking the build as unstable automatically

Jenkins - execute a script before building, then have the user confirm to build

I have a bash script that looks like this:
#!/bin/sh
previousRelease=`git describe --tags --match "release*" origin/release`
git diff --name-status $previousRelease..origin/release
Is there a way of having Jenkins execute it as part of a build process? The intention is to see a list of files that have changed since the last release, as a manual step to confirm that the release should go up. The user who has triggered the build needs to read the output and then confirm the release should go ahead.
Most things are possible to do in Jenkins but if it is the best way of doing it is another question.
To solve this I would use an approach with two jobs one for checking the diff (hock that one on to the git repository) The other job for doing the actual release.
The check diff job
1 Create a job of the type freestyle project with build type "execute shell" and run your script above. Add some prints at the end of the log to create a clickable link to manually start the release job with current git-id as argument.
Just printing an URL in console output will make it clickable so:
export GITID=`git log -n| grep and sed or awk something`
echo http://jenkins.example.com:8888/job/releaseme/buildWithParameters?label=$GITID&parameters=build
will create the accept changes user interface you requested.
The release job
2 Create another job(above I assumed you named it releaseme) let the job have one parameter as argument (tick "This build is parameterized") make let the argument be the git-id you would like to release. Create your release script in this job.

Resources