I have similar issue, that blocks me. I need to run my JMeter tests with Jenkins. But Jenkins validates the JMeter build as successful when it’s actually failed. I wonder what I am doing wrong so that the jmeter won't return fail when an assertion fails.
I run a Windows Batch script calling jMeter. Here's how:
COMMAND LINE
cd C:\apache-jmeter-3.1\bin
jmeter -n -c -t C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx -l C:\Users\maria\Desktop\Automation\WIP\Test-page.xml
And saves the results in .xml to validate the report with the Performance plugin (v 2.0) and Jenkins ver. 2.32.1
CONSOLE
Created the tree successfully using C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx
Starting the test # Fri Dec 30 11:05:58 GMT 2016 (1483095958952)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary = 1 in 00:00:00 = 3.1/s Avg: 206 Min: 206 Max: 206 Err: 1 (100.00%)
Tidying up ... # Fri Dec 30 11:05:59 GMT 2016 (1483095959326)
... end of run
Performance: Recording JMeter reports 'C:\Users\maria\Desktop\Automation\WIP\Test-page.xml'
Performance: Parsing JMeter report file 'C:\Jenkins\jobs\Test demo\builds\3\performance-reports\JMeter\Test-page.xml'.
Performance: Percentage of errors greater or equal than 0% sets the build as unstable
Performance: Percentage of errors greater or equal than 0% sets the build as failure
Finished: SUCCESS
As you can see I have 1 error. But the Performance plugin sets the build to success.
PLUGIN SETTINGS
performance plugin settings
Jmeter properties
jmeter.save.saveservice.assertion_results=all
jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.response_data=true
TEST CASE
Test-page.jmx
Attempts I tried and didn’t work for me:
older version of the Performance plugin (v 1.11 and v.16)
.jtl format
.xml
.csv
adding "echo" to the command line in the end
setting jmeter.save.saveservice.response_data=true
Ran out of ideas how to make the Performance plugin count the errors and raise the "Failed" flag :(
Set "Use error thresholds on single build" from 0 to 1.
This should mean that the jenkins build fails if at least 1 error is in your JMeter test case, where a 0 ignores the error count (If I remember this correctly).
Sounds like a Performance Plugin issue, I believe you should report it via Jenkins Bug Tracker
In the meantime to work this around I would suggest returning non-zero exit code for the JMeter build step, the fastest and the easiest way is running your JMeter test using Taurus tool as a wrapper. Taurus has powerful Pass/Fail Criteria subsystem where you can define the conditions of your build failure. If the condition will be met Taurus execution step will be finished with code 3. Jenkins is smart enough to automatically fail command-line tasks with non-zero exit code.
Minimal working Taurus config file:
---
execution:
scenario: my-test
scenarios:
my-test:
script: C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx
services:
- module: passfail
criteria:
- succ<100%, stop as failed
More information, if needed: Meetup Recap on Using Taurus to Automate JMeter and Jenkins Tests
You can also change JMeter exit code directly form the JMeter test via i.e. JSR223 Scripting Elements like:
if (some condition) {
System.exit(1);
}
but in this case you won't have much flexibility in terms of setting failure criteria
I set my Select evaluation mode to standard mode, Set the radio button on Error Threshold and set Unstable and Failed to 1
Related
I am using Jmeter for API's functional testing. For this, have added Response Assertion.
Even though it's failing but Jenkin's build appeared as Succeed.
Is there any way to mark Jenkin build as Failed when our Assertions are failed?
Please help out on this, let me know if any more info is required.
It depends on how do you launch JMeter in Jenkins, if it's just a command-line non-GUI execution like jmeter -n -t test.jmx -l result.jtl then it doesn't produce any error exit status code and this is something Jenkins checks.
The options are in:
Migrating to JMeter Maven Plugin which provides jmeter-check-results goal
Migrating to Taurus which provides Pass/Fail Criteria subsystem
And finally you can add a JSR223 Listener to your Test Plan and force JMeter to exit by adding the next code in the "Script" area:
if (!prev.isSuccessful()) {
System.exit(1)
}
Cannot detect file type because of error, and : Failed to copy Build step 'Publish Performance test result report' changed build result to FAILURE Finished: FAILURE. whenever i run the script always i face this..
Take a look into Jenkins Console Output - it should give you the reason for the failure.
Most probably the Performance Plugin fails to find JMeter's .jtl results file in Jenkins Workspace, either the .jtl results file is missing or you're pointing the Performance Plugin to the incorrect location.
If you have a Script step to run a JMeter test like:
jmeter -n -t test.jmx -l result.jtl
You should be able to use simply result.jtl in the Performance Plugin.
Check out:
Performance Trend Reporting
How to Use the Jenkins Performance Plugin
and Running Performance Tests articles for more details on various aspects of Jenkins Performance Plugin use cases.
The Setup - A jenkins job using jenkins parameters testApp and testEnv. The Execution Batch looks like this:
C:\jmeter\apache-jmeter-3.2\bin\jmeter.bat -n -t
C:\JMeter\Scripts\API_scripts\%testApp%.jmx -Jtestenv=%testEnv% -JtestApp=%testApp% -JtestBrowser=NA -l
C:\AUTO_Results\jtl\%testApp%_%testEnv%.jtl
Post-build Actions
Console output (build lob) parsing with a global rule so that the Failures that are logged in the Jenkins Console window will consider the JMeter script failing. (discussed Jenkins shows JMeter script failure even though the script actually passed)
Triggered parameterized build - this is a separate jmeter script that updates a wiki page with either PASS/FAIL and uploads the JMeter report.
The Issue - How do I get the downstream Triggered build to use the parameters from the upstream script? I set the Parameter = Current build parameters but it's not applying those. Also, I wont know the value of the testResult parameter until the upstream build finishes. I tried adding %testResult%=PASS to the 'Predefined parameters' box
As per Parameterized Trigger Plugin page:
The parameters section can contain a combination of one or more of the following:
a set of predefined properties
properties from a properties file read from the workspace of the triggering build
the parameters of the current build
Subversion revision: makes sure the triggered projects are built with the same revision(s) of the triggering build. You still have to make sure those projects are actually configured to checkout the right Subversion URLs.
Restrict matrix execution to a subset: allows you to specify the same combination filter expression as you use in the matrix project configuration and further restricts the subset of the downstream matrix builds to be run.
So you basically need to copy over the parameters you would like to have in the "downstream" job from the current one.
As a workaround to current performance plugin limitations you can consider running JMeter using Taurus tool as a wrapper, it has flexible and powerful pass/fail criteria subsystem which will basically return to Jenkins non-zero exit code triggering build failure in case of issue in the test. If everything goes well Taurus exit code will be 0 which is considered successful by Jenkins. Check out How to Run Taurus with the Jenkins Performance Plugin article for more details.
I use Jmeter to validate functional test for our API's. I have created an HTML file which contains the total number of tests executed and the passing/failure ones. This is a customized report.
Can I set a condition such that the build fails if there is a single failure in the report?
Please not that I have already tried performance plugin. But my assertion here is to see if the test case fails or passes and not the reponse/throughtput time.
Not sure if there is a plugin which will enable me to read the HTML and then fail the report if it sees a failing test case.
The easiest option would be running your JMeter test using Taurus tool as a wrapper. Taurus has powerful and flexible Pass/Fail Criteria subsystem where you can configure any combination of test failure thresholds and if any failure occurs - Taurus will exit with non-zero exit status. Jenkins is smart enough to catch this non-zero exit code and fail the build.
JMeter-only solution will be adding a JSR223 Listener with the script like:
if (!prev.isSuccessful()) {
System.exit(1)
}
Which is basically doing the same - in case of any failure the test will immediately exit with status of 1 triggering the build failure.
You could just use a shell job, grep for the failures and exit if found, something like this :
#!/bin/sh
if grep 'failure' file.html; then
exit 1
fi
I don't know what is the structure of your HTML file, but since it's a test pass / fail report then I suppose it's pretty simple.
You may try adding "Execute Python script" (needs "Python plugin") to the job. With that you can easily parse HTML doc and check the results using a simple script.
I was having a similar issue with Jenkins and JMeter. Here's what I did:
Install this plugin: https://wiki.jenkins.io/display/JENKINS/Log+Parser+Plugin
for your logParserfile, add this:
error /FAILED/
Create a BeanShell Assertion for each sampler you are testing that contains something like this:
String testscenario = "TEST SCENARIO: ";
String requestName = "getRequest";
String ver = "${My_ver}";
String respData = new String(ResponseData);
if (respData.contains("OK"))
{
log.info(testscenario+"Passed. "+requestName+" version ="+ver);
print(testscenario+"Passed. "+requestName+" version ="+ver);
} else
{
log.error(testscenario+"*FAILED. "+requestName);
print(testscenario+"*FAILED. "+requestName);
}
When your Jenkins job runs and anything fails, the FAILED log entry will show
I'm going through the tutorial on YSlow and Phantom js in Jenkins here: http://yslow.org/phantomjs/
Everything appears to be working great except the Jenkins builds are failing. I think this is due to the violations that YSlow is finding (6 for the particular site I am measuring). I'd rather have the build be successful (or unstable) vs. failed though
Is that possible with this or will I have to resort to something like the postgroovy or text finder plugin?
This is the console output:
phantomjs.exe yslow.js -i grade -t 50 --format junit http://www.somesite.com 1>yslow.xml
D:\Apps\Jenkins\workspace\YSlow_Test>exit 6
Build step 'Execute Windows batch command' marked build as failure
Thanks
Any non-zero exit code at the end of your Execute Windows batch command build step will result in build step being marked as failure.
To have the build step marked as success, you need an exit code of 0. I don't know anything about "yslow" or "phantomjs" and why they are giving you exit code of non-zero, but from "batch" side of things, you need only write exit 0 at the end of your build step if you want to overwrite the exit code of your phantomjs command.
You can then use Text Finder plugin to parse the console log and mark build as unstable when certain conditions are met.
Reading over this answer, Configuring yslow on Jenkins looks like you need TAP plugin to have the functionality of unit testing marking the build as unstable automatically