I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
I am using Jenkins Post build task plugin to execute a bash script. The script is running fine, but the entire raw script is printed on the Jenkins console. Check section 3 in the image.
How can I avoid the script from getting printed on the Jenkins console?
I'm using the Jenkins JUnit Plugin to collect my tests result and according to the documentation:
Allow empty results: If checked, the default behavior of failing a
build on missing test result files or empty test results is changed to
not affect the status of the build. Please note that this setting make
it harder to spot misconfigured jobs or build failures where the test
tool does not exit with an error code when not producing test report
files.
This is how I use it:
junit testResults: '**/reports/junit*.xml', testDataPublishers: [[$class: 'StabilityTestDataPublisher']]
So the behavior should fail if the test file is empty but I get "UNSTABLE" result with the message
Test report file /opt/workspace/integrations_develop/reports/junit_integrations.xml was length 0
How can I make it fail if the test file is empty?
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue
Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.