Using Protractor/Jasmine conjunction automation framework I want to run test suite. When my Jenkins job runs I do not want to fail my job if any test case fails.
Pytest in python provides #pytest.mark.xfail feature to mark expected failures and this does not impact the jenkins job.
Is there any such feature in Protractor which can mark test cases as expected to fail?
I saw xit and xdescribe features but it skips the test case rather then expected failure
Related
I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
I am using the Junit plugin for the tests report. My problem is most of the builds will mark as unstable which should be failed. Is there any way to mark the build as failed when having more than one test failed?
Example1:
Example2:
Example3:
As of now, there is no directly configurable threshold option for the JUnit plugin in Jenkins (Here is the open feature request). However, you can add an if condition after reporting the test results and set the build status.
def junitTestSummary = junit testResults: "**/*.xml"
// junitTestSummary.[failCount, totalCount, skipCount, passCount]
if(junitTestSummary.failCount>0){
error("Failing the pipeline. Because, there are ${junitTestSummary.failCount} tests failed")
}
I'm using this command to run my tests:
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd"
but sometimes I want to run a specific test. How can I do it?
I read that I can use "This project is parameterized" but didn't understand how to use it.
I also saw this - https://plugins.jenkins.io/selected-tests-executor/ but it's not good enough since it required an external file.
If you use the maven-surefire-plugin you can simply run
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd -Dtest=com.example.MyJavaTestClass"
or
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd -Dtest=com.example.MyJavaTestClass#myTestMethod"
I suggest to add a parameter for the test class/method to your pipeline definition.
pipeline {
agent any
parameters {
string defaultValue: '', description: 'Test Name', name: 'TEST_NAME', trim: false
}
stages {
stage('run tests') {
steps {
script {
def optionalParameters = ""
if (params.TEST_NAME != null) {
optionalParameters += " -Dtest=" + params.TEST_NAME
}
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd" + optionalParameters
}
}
}
...
}
...
}
Jenkins isn't really the tool for that. It's typical use case us running all tests. If you really want to run only one test, you should do that in your development environment
But the simplest thing to do, if you want the results for one of the tests, us to have Jenkins run all your tests and simply ignore the results for the other tests.
Generally, you should run (quick, cheap) unit tests in your development environment, and submit code for (expensive, slow) integration tests by Jenkins only once the code passes the unit tests (And Jenkins should run the unit tests again, just in case).
I suspect your real question is "how do I debug failure of an integration test run by Jenkins". Jenkins is a build and test tool not a debugging tool. It is not itself a suitable tool for debugging test failures. But the way you use Jenkins can help debugging.
Do not use integration tests as a substitute for unit tests.
If your software fails an integration test, but no unit tests, as usual with debugging a test failure you should be making hypotheses about the kinds of defect in your software that could cause the failure. Then check whether a unit test could be added that would detect that kind of defect. If so, add that unit test.
Ensure that your tests produce useful diagnostic messages on failure. Test assertions should have helpful messages. Test a should have descriptive names.
If an integration test checks a sequence of actions, ensure you also have tests for the individual actions.
I am running pytest tests in a Jenkins scheduled job and generating a junitxml report as the following
pytest --junitxml=report.xml
Then I am using a post build action of publish junit test result and I can see the results but I don't see output of the passed tests (Even if I check the checkbox of "Retain long standard output/error"
Anyone succeeded in showing the output of passed tests when using pytest + junitxml publisher in Jenkins ?
It's due to Pytest that doesn't add end of line in the .xml.
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue