I am using zephyr scale to do some testing associated with jira tickets (use cases). I am using the pytest integration by uploading junit reports to zephyr scale.
e.g.
# will match with test case with key DEV-T21
def test_method_3_DEV_T21(self):
assert 5 == 5
then outputting the report as xml
pytest --junitxml=output/junitxml_report.xml
then uploading to zephyr scale
curl -H "Authorization: Bearer ${TOKEN}" -F "file=#TestSuites.xml;type=application/xml" https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey="JQA"&autoCreateTestCases=true
This provides the test results well however I want to incorporate the code execution through JIRA as opposed to uploading the results from my local machine. Is there a way that when I click execute test
that I can call my pytest code as part of the execution?
Related
I am using Jmeter for API's functional testing. For this, have added Response Assertion.
Even though it's failing but Jenkin's build appeared as Succeed.
Is there any way to mark Jenkin build as Failed when our Assertions are failed?
Please help out on this, let me know if any more info is required.
It depends on how do you launch JMeter in Jenkins, if it's just a command-line non-GUI execution like jmeter -n -t test.jmx -l result.jtl then it doesn't produce any error exit status code and this is something Jenkins checks.
The options are in:
Migrating to JMeter Maven Plugin which provides jmeter-check-results goal
Migrating to Taurus which provides Pass/Fail Criteria subsystem
And finally you can add a JSR223 Listener to your Test Plan and force JMeter to exit by adding the next code in the "Script" area:
if (!prev.isSuccessful()) {
System.exit(1)
}
In my JMeter setup, I have the following config in my Test Aggregate Report (named loadTestAggregate) in order to list all the erroring URLs in my load test :
When I run my JMeter test locally, this Test Aggregate Report correctly displays the results (as a .csv) and the URLs are displayed as required (i.e. the response code, response message and URL are all displayed).
I then set up this test to run via Jenkins (with the Performance plugin installed).
Below is the Jenkinsfile used to run this test via Jenkins;
sh "jmeter -Jjmeter.save.saveservice.url=true -n -t loadTest.jmx -JTest_Url=${env.TEST_URL} -l jmeter_results.jtl"
perfReport errorFailedThreshold: 5.00, sourceDataFiles:'loadTestAggregate.csv'
However, when the test is completed and I look in Jenkins via the Performance plugin, the URLs are not displayed.
All that's displayed is the following;
Is it possible to get these (erroring) URLs to be listed via the Jenkins Performance plugin?
If so, am I missing something here as my config for the Test Aggregate report works locally but not via Jenkins, so I can only presume that it's a Performance plugin issue.
Many thanks
They're "aggregate" charts per Sampler label
As a workaround you can change the Sampler label (HTTP Request) to the Sampler URL, it can be done via JSR223 PostProcessor and the following Groovy code:
prev.setSampleLabel(prev.getUrlAsString())
where prev stands for the previous SampleResult, see Top 8 JMeter Java Classes You Should Be Using with Groovy article for more information.
Demo:
I am running pytest tests in a Jenkins scheduled job and generating a junitxml report as the following
pytest --junitxml=report.xml
Then I am using a post build action of publish junit test result and I can see the results but I don't see output of the passed tests (Even if I check the checkbox of "Retain long standard output/error"
Anyone succeeded in showing the output of passed tests when using pytest + junitxml publisher in Jenkins ?
It's due to Pytest that doesn't add end of line in the .xml.
Using Protractor/Jasmine conjunction automation framework I want to run test suite. When my Jenkins job runs I do not want to fail my job if any test case fails.
Pytest in python provides #pytest.mark.xfail feature to mark expected failures and this does not impact the jenkins job.
Is there any such feature in Protractor which can mark test cases as expected to fail?
I saw xit and xdescribe features but it skips the test case rather then expected failure
Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.