Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.
Related
I have a Jenkins project in which i run a sonarqube analysis in a windows OS.
In jenkins , I created a batch step more or less like this:
mycommand test --machine --coverage > tests.output
sonar-scanner
mycommand is a 3rd party plugin which i can't modify , and, based on the content of the project , this step can fail and I want the jenkins queue to go on with the other command .
Now if mycommand return an error jenkins stop.
How can I achieve this?
The magic here is in the return code from your command. Without knowing what it is and what it does, the important thing is that if any command returns non-zero exit code Jenkins will see the command as failed. To prevent any errors from being registered by Jenkins add exit 0 to the command.
For example
bat """
mycommand test --machine --coverage > tests.output || exit /b 0
"""
bat """
sonar-scanner
"""
Should work for this.
Alternatively you can also chain the commands together with & so Jenkins doesn't check the exit codes before everything is completed, but then the second command will also always show as passed regardless of its real status. Also it can become difficult to read with lengthy command chains.
bat """
mycommand test --machine --coverage > tests.output & sonar-scanner & exit /b 0
"""
I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would like to run them again few times and let them the chance to pass so my build won't be unstable/failed.
Is there any plugin I can use?
I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite or smething which is local.
But there is also a plugin which can retry a build:
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.
If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?
Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:
The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test
The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test
At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps.
#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do
echo "Running failed tests $i time(s)"
for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do
class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
ant -Dtest.class.pattern=${class} test
done
done
After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.
I am using jenkins to build a bunch of legacy code. The legacy code comes with some complex build scripts, written in csh.
The build scripts do not check for or exit on errors. The user is expected to scan the output for error messages. However, this does not work well with Jenkins.
I am executing the csh build scripts in a jenkins "shell execution" build step. For example:
export PATH=`pwd`/ALL/bin:/usr/local/bin:/usr/bin:/bin:$PATH
cd ATLb2.00/expt_02.0
csh 020.com
When I run this from the command line, I can also use the -e option:
csh -e 020.com
In this case, as I expect, the script is run, but when the first error is encountered, the script stops and returns a non-zero code. However, when I try this in Jenkins, the build fails as soon as it gets to the csh -e command, without executing any of the script.
The error I get in Jenkins is:
+ csh -e 020.com
Build step 'Execute shell' marked build as failure
On the command line, the script is run and I see all kinds of output, until something fails, and then the script exits. On Jenkins the script seems to fail without even running. There is no output, and even scripts with no failures will not run for me under jenkins with the -e option.
What's up?
I recommend that you specify csh on a more global level and then execute the commands in a Jenkins build step.
If you want to use csh for all jobs, you can set the default shell using Jenkins > Manage Jenkins > shell executable.
If you want to use csh for only a particular job, begin the Execute shell build step with a shebang, such as:
#!/usr/bin/tcsh -e -x
command1
command2
...
Since I have tested only tcsh, that is what I use in the example.
Beware that a space is not allowed after the #!:
#! /usr/bin/tcsh # Wrong
This will give the error,
java.io.IOException: Cannot run program ""
I tested the above on Jenkins 1.625.3
I am new to Jenkins and need some help..
I have 4 shell scripts : test1.sh, test2.sh, test3.sh and test4.sh
I want test2.sh to only run if test1.sh runs successfully and test4.sh to only run if test3.sh runs successfully.
I also want test1.sh and test3.sh to run in parallel.
How could I achieve it in Jenkins?
I am using "Execute shell script on remote host using ssh" and "Conditional steps(multiple)" (just exploring). I have also set up keys so as to communicate with remote server.
Illustration using screen shot or other way would be helpful.
Thank you!
First, ensure that test1.sh and test3.sh return the standard success code when they succeed (0). Then the simple way, which works in any command line, not just Jenkins, is to use this command line:
((test1.sh && test2.sh) &) ; ((test3.sh && test4.sh) &)
Each pair of parentheses forms a subshell, double-amperands mean "if first succeeds then run second", and single ampersand mean "run in backgorund". So you get the equivalent of two backgrounded shells each running two scripts, which will exit if the first script doesn't return 0.
The Jenkins-specific solution is to have a node with two (or more) runners. Create two jobs, and tie both to that node. Each job runs a single shell, either test1.sh && test2.sh, or test3.sh && test4.sh.
In Jenkins declarative or scripted pipelines you can create parallel executions for each shell script or any other command or program execution you like.
stage('run-parallel-branches') {
steps {
parallel(
a: {
echo "This is branch a"
},
b: {
echo "This is branch b"
}
)
}
}
Reference: https://www.jenkins.io/blog/2017/09/25/declarative-1/
I have a lots of tests on several server. I use Jenkins to manage all of them.
On one server (Slave Windows), when I launch a test in cmd, I got something like :
c:/tests/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m31.761s
echo %errorlevel%
0
No error in the tests, and cucumber seems good.
When jenkins launch exactly the same tests, I get :
c:/jenkins_folder/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m28.453s
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
The test passed, but is marked failed by jenkins.
The command "echo %errorlevel%" is aborted : the job fail before this point.
The same job played on another slave work.
Same problem with all profile and all tags.
Same problem when I replace profile by real value
I don't use the --strict flag
Jenkins, plugins : all up-to-date
Code of the windows batch :
cd /test8folder
cucumber --tag #dev -p ie
echo %errorlevel%
What did I missed ?
There doesn't seem to be anything wrong with the configuration.
The fact that you get Build step 'Execute Windows batch command' marked build as failure after your cucumber command without actually seeing the echo %errorlevel% executed only reaffirms that there was an error in cucumber (more on that later).
However, in Execute Windows Batch Command, even a command in error would not exit the batch script (unlike Jenkins's default Execute Shell implementation). You should be seeing at least some exit code, 0, 1, or anything.
The only time this would happen is if something within your buildstep executed exit /b [exit_code_num]. I don't know "cucumber", but if that is in-fact a cucubmer.bat and inside there is an exit /b statement, this is what's causing it drop out of the buildstep without continuing.
Solution
You can use call cucumber [whateverparams] so that even when it quits with exit /b, the control will return to the calling process, the Execute Windows Batch command script.
Try that first. And you will probably see that your echo %errorlevel% will probably return a non-zero value when executed under Jenkins, but at least you will see it now.
Now, as for why it succeeds on command prompt, but fails within Jenkins, there could be a lot of reasons, the most common one being environment variables and paths. We can tackle that later once we actually see the exit code of cucumber. You also said it worked on another node: even more reason to believe this is an environment issue, maybe a non-existent folder...
Edit:
The reason that even "successful" test execution exits the calling script is because exit /b 0 would still quit the calling script, even though cucumber exited with "success" 0