When we are running parallel test from TestNG on perfecto cloud for browsers, once the execution is over, we are seeing that many failures are reported on perfecto dashboard with RemoteWebDriver name on it.
Did you observed additional results only with data driven test? I observed additional report entry for each testcase with RemoteWebDriver name and status unknown in addition to report with actual testcase name and pass/fail result. That looks like bug in perfecto report. In either case you should open perfecto support ticket.
Related
Which assertion has to be use to check latency,throughput,received & sent bytes and error rate ?How to impliment assertion in total avg value of each script in JMeter to automate?
JMeter per se doesn't provide this functionality so you can go for one of the following approaches:
AutoStop Listener is capable of automatically stopping the test if one of the following metrics exceeds defined thresholds:
Response Time
Latency
Error Rate
If you are running your test in Jenkins you can use Performance Plugin to mark build as failed or unstable basing on various metrics
And finally you can run your JMeter test via Taurus having very powerful and flexible pass/fail criteria subsystem so if there will be a clash between expected and actual metrics it will basically return non-zero exit code which you can use in shell scripts or Jenkins freestyle jobs and/or pipelines.
I'm using Jenkins to start a jmeter performance test that sends thousands request over one hour.
To avoid storing lots of report data for each successful request and send it all over the network I only log errors on my View Results Tree element.
I was planning on using that errors only file to decide whether the test passed or failed.
But by using Jenkins Performance Plugin it reports that 100% of the tests failed (of course, because all of the requests on that file were related to errors).
Is there a way of using this plugin (or any other) to achieve my goal?
I am using JMeter do API test, and also integrated with CI tool Jenkins.
Now I want to parse the JMeter test report in Jenkins. But I don't need performance test report.
I need the report to list total number of cases, how many cases passed and how many cases failed.
Anyone have this solution?
JMeter was designed for performance testing, and the world of performance testing still heavily relies on human interpretation of test results. Therefore "case" or "test" is not natively supported by JMeter. However, you group your requests under transaction controller(s) and use "generate parent sample" option. This should make the transaction controller and all requests it groups effectively a "test" and either pass or fail it as a whole.
Now, the other question is how to get number of executed and failed "tests". Currently jmeter-maven-plugin can provide information about number of JMX files processed and number of failed "cases". I doesn't provide information about total number of "cases" which were exected. There is a pull request opened which will improve it - reports will contain number of JMX files processed, total number of tests executed and number of failed tests.
EDIT
The functionality you are looking for is currently provided by Lightning. It will parse JMeter output and report total number of transactions and number of failed transactions.
You could use Performance Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin).
Also you could save execution log file, download it after test execution and open it in some JMeter listener (e.g. Summary report). There you will see all the report data:
export OUTPUT=/jenkins/userContent/Load_tests
jmeter -t testscript.jmx -l $OUTPUT/results.jtl
echo "Download a log file http://<url of your Jenkins>/userContent/Load_tests/results.jtl"
We have a service to pick up custom tests in XML and convert those to CodedUI tests. We then start a process for MSTest to load the tests into the Test Controller which then distributes the tests across various Agents. We run regression tests at night, so no one is around to fix a system if something goes wrong. When certain exceptions occur in the test program, it pops open an error window and no more test can run on the system. Subsequent tests are loaded into the agent and fail immediately because they can not perform their assigned tasks. Thousands of tests that should take all night on multiple systems now fail in minutes.
We can detect that an error occurred by how quickly a test is returned, but we don't know how to disable the agent so as not to allow it to pick up any more tests.
addendum:
If the test has failed so miserably that no more tests can attempt a successful run (as noted, we may not have an action to handle some, likely new, popup), then we want to disable that agent as no more tests need to run on it: they will all fail. As we have many agents running concurrently, if one fails (and gets disabled), the load can still be distributed without a long string of failures. These other regression tests can still have a chance to succeed (everything works) or fail (did we miss another popup, or is this an actual regression failure).
2000 failures in 20 seconds doesn't say anything except 1 system had an problem that no one realized it would have and now we wasted a whole night of tests. 2 failures (1 natural, 1 caused by issue from previous failure) and 1 system down means the total nights run might be extended by an hour or two and we have useful data on how to start the day: fix 1 test and rerun both failures.
One would need to abort the testrun in that case. If you are running mstest yourself, you would need to inject a ^c into the command line process. But: if no-one is around to fix it, why does it matter that the consequenting test fail ? if its just to see which test was the cause of the error quickly, why not generate a code ui check to see if the message box is there and mark the test inconclusive with Assert.inconclusive. The causing test would stand out like a flag.
If you can detect the point at which you want to disable the agent then you can disable the agent by running the "TestAgentConfig.exe delete" which will reset the agent to unconfigured state.
I have installed the jMeter plug-in for Jenkins and I can see the performance report based upon the jtl file that my jmx file generates. In my jmx file, I have several Response Assertion listeners. Based upon the jtl file that it generates, the assertion responses are false. However, when I look at the performance report in Jenkins, it is showing 0 in the percentage of errors. My first question would be, what constitutes an error in the performance report? And secondly, how can I see which of the response assertion listeners returns false in the performance report in Jenkins?
What constitute an error in the performance report is determined by the percentage of errors that the samples find
To see what the error is go to
last build->Peformance Report
you should see a listing of all of the samples in your jmeter test plan and the results
Then select the sample name and you should see a the results for the sample taken