I have Integrated Jmeter with Jenkins. Sometimes due to Jmeter assertion failure,the Whole Jenkins build gets fail which is expected for my requirement.
Now, Is there any way to display the Jmeter failed request (which is an assertion failure one mentioned above) on Jenkins Console output?
Add a JSR223 Listener at the same level of requests you want to capture (or a level higher, see JMeter Scoping Rules - The Ultimate Guide article to learn why does the placement matter
Put the following code into "Script" area:
if (!prev.isSuccessful()) {
prev.getAssertionResults().each { assertionResult ->
if (assertionResult.failure) {
println('Request ' + prev.getSampleLabel() + ' failed with ' + assertionResult.failureMessage)
}
}
}
That's it, whenever a request fails due to failed assertion you will have the relevant line printed to STDOUT (therefore to Jenkins console log)
Related
We have a Jenkins declarative pipeline which, after deploying our product on a cloud vm, needs to run some product tests on it. Tests are implemented as a separate job on another jenkins and tests will be run by main pipeline by triggering remote job on 2nd jenkins using parameterized remote trigger plugin parameterized remote trigger plugin.
While this plugin works great, when using option blockBuildUntilComplete it blocks for remote job to finish but doesn't release the jenkins executor. Since tests can take a lot of time to complete(upto 2 days), all this time executor will be blocked just waiting for another job to complete. When setting blockBuildUntilComplete as false it returns a job handle which can be used to fetch build status and result etc. Example here:
while( !handle.isFinished() ) {
echo 'Current Status: ' + handle.getBuildStatus().toString();
sleep 5
handle.updateBuildStatus()
}
But this still keeps consuming the executor, so we still have the same problem.
Based on comments in the article, we tried with waitForWebhook, but even when waiting for webhook it still keeps using the executor.
Based on article we tried with input, and we observed that it wasn't using executor when you have input block in stage and none agent for pipeline :
pipeline {
agent none
stages {
stage("test"){
input {
message "Should we continue?"
}
agent any
steps {
script{
echo "hello"
}
}
}
}
}
so input block does what we want to do with waitForWebhook, at least as far as not consuming an executor while waiting.
What our original idea was to have waitForWebhook inside a timeout surrounded by try catch surrounded by a while loop waiting for remote job to finish :
while(!handle.isFinished()){
try{
timeout(1) {
data = waitForWebhook hook
}
} catch(Exception e){
log.info("timeout occurred")
}
handle.updateBuildStatus()
}
This way we could avoid using executor for long period of time and also benefit from job handle returned by plugin. But we cannot find a way to do this with the input step. input doesn't use executor only when it's separate block inside stage, but it uses one if inside steps or steps->script. So we cannot use try catch, cannot check handle status andcannot loop on it. Is there a way to do this?
Even if waitForWebhook works like input is working we could use that.
Our main pipeline runs on a jenkins inside corporate network and test jenkins runs on cloud and cannot communicate in with our corp jenkins. so when using waitForWebhook, we would publish a message on a message pipeline which a consumer will read, fetch webhook url from db corresponding to job and post on it. We were hoping avoid this with our solution of using while and try-catch.
Failing tests resulted in green balls in our "Open Blue Ocean" pipeline overview. When I read the manual (https://doc.froglogic.com/squish/latest/rg-cmdline.html) this is according to specification, but using the --exitCodeOnFail should result in our desired behavior. In our Jenkinsfile we scripted the following:
squish([extraOptions: """--tags
${tag}
--retry
2
--config
addAppPath
${squishsrcdir}
--config
addAUT
startSimProApp.bat
${squishsrcdir}
--exitCodeOnFail
-666
--config
setResponseTimeout
30""", squishPackageName: 'squish for qt 6.5.2', testSuite: "${squishsrcdir}", unstableBuildOnError: true])
Unfortunately this results in the following error:
com.froglogic.squish.SquishException: unknown option --exitCodeOnFail
The squish plug-in version is: 8.1.1
What are my options to get red balls when a test fails under squish?
The --exitCodeOnFail option is not supported by the Squish plugin.
Take a look at https://doc.froglogic.com/squish/latest/ao-hudson.html#ao-jenkins-example-pipeline-jobs
The squish step sets neither build nor stage result. It returns the execution results as a string instead. Your pipeline may act based on the returned value. You can find an example on the last screenshot in the linked above chapter.
Squish has a known issue (reported and expected to be solved) in matching returned Squish test suite execution status with retries with the final result of the job. For example, if your test fails during first retry and get passed in the next retry, the final status of the job will remain as unstable/failed.
I am running jmeter using jenkins, i have duration assertion of 300ms in my script. The assertion is working fine, as my jmeter result is showing error but the jenkins build still passes.
Is there any way to fail my build in case of error in jmeter result ?
You need to define Error Threshold
The Error Threshold marks the the build as unstable or failed if the amount of errors will exceed the specified value.
Use error thresholds on single build – Define error thresholds in % for the current build.
The options are in:
Configure error threshold in Performance Plugin GUI
Jenkins considers build passed when it returns 0 exit status code, you can "tell" JMeter to exit with non-zero exit code by adding JSR223 Listener and using the following code:
if (!prev.isSuccessful()) {
System.out.println("Test failure, exiting...")
System.exit(1)
}
where prev stands for parent SampleResult class instance which gives you control over parent Sampler response code, message, data, etc. Check out Top 8 JMeter Java Classes You Should Be Using with Groovy article for more information on JMeter API shortcuts available for JSR223 Test Elements
it will result in the following behaviour:
and this 1 status code will cause Jenkins build failure
Alternative solution for point 2 would be using Taurus tool as a wrapper for your JMeter test, it provides handly Pass/Fail Criteria subsystem which provides flexible way of defining custom thresholds for considering your test as failed
I read How can I set the job timeout using the Jenkins DSL. That sets the timeout for one job. I want to set it for all jobs, and with slightly different settings: 150%, averaged over 10 jobs, with a max of 30 minutes.
According to the relevant job-dsl-plugin documentation I should use this syntax:
job('example-3') {
wrappers {
timeout {
elastic(150, 10, 30)
failBuild()
writeDescription('Build failed due to timeout after {0} minutes')
}
}
}
I tested in http://job-dsl.herokuapp.com/ and this is the relevant XML part:
<buildWrappers>
<hudson.plugins.build__timeout.BuildTimeoutWrapper>
<strategy class='hudson.plugins.build_timeout.impl.ElasticTimeOutStrategy'>
<timeoutPercentage>150</timeoutPercentage>
<numberOfBuilds>10</numberOfBuilds>
<timeoutMinutesElasticDefault>30</timeoutMinutesElasticDefault>
</strategy>
<operationList>
<hudson.plugins.build__timeout.operations.FailOperation></hudson.plugins.build__timeout.operations.FailOperation>
<hudson.plugins.build__timeout.operations.WriteDescriptionOperation>
<description>Build failed due to timeout after {0} minutes</description>
</hudson.plugins.build__timeout.operations.WriteDescriptionOperation>
</operationList>
</hudson.plugins.build__timeout.BuildTimeoutWrapper>
</buildWrappers>
I verified with a job I edited manually before, and the XML is correct. So I know that the Jenkins DSL syntax up to here is correct.
Now I want to apply this to all jobs. First I tried to list all the job names:
import jenkins.model.*
jenkins.model.Jenkins.instance.items.findAll().each {
println("Job: " + it.name)
}
This works too, all job names are printed to console.
Now I want to plug it all together. This is the full code I use:
import jenkins.model.*
jenkins.model.Jenkins.instance.items.findAll().each {
job(it.name) {
wrappers {
timeout {
elastic(150, 10, 30)
failBuild()
writeDescription('Build failed due to timeout after {0} minutes')
}
}
}
}
When I push this code and Jenkins runs the DSL seed job, I get this error:
ERROR: Type of item "jobname" does not match existing type, item type can not be changed
What am I doing wrong here?
The Job-DSL plugin can only be used to maintain jobs that have been created by that plugin before. You're trying to modify the configuration of jobs that have been created in some other way -- this will not work.
For mass-modification of existing jobs (like, in your case, adding the timeout) the most straightforward way is to change the job's XML specification directly,
either by changing the config.xml file on disk, or
using the REST or CLI API
xmlstarlet is a powerful tool for performing such tasks directly on shell level.
Alternatively, it is possible to perform the change via a Groovy script from the "Script Console" -- but for that you need some understanding of Jenkins' internal workings and data structures.
I have a Jenkins configured on Windows Server for Nightly builds, compilations etc.. and UI Test (E2E Tests) automation which I'am having some problems.
We have a Web application runs on Chrome and need Jenkins to run these tests using protractor (with Selenium).
With the manual run from Windows cmd console, the tests finish successfully.
-------Here is the console output;
***C:\Jenkins\jobs\UI Automation\workspace> protractor e2e-tests/protractor.conf.js
Using ChromeDriver directly...
[launcher] Running 1 instances of WebDriver
WARNING - more than one element found for locator by.repeater("attachment in pos
t_item.attachments") - the first result will be used
.
Finished in 68.394 seconds
1 test, 8 assertions, 0 failures
[launcher] 0 instance(s) of WebDriver still running
[launcher] chrome #1 passed***
-----Here is the output of the run from Jenkins;
C:\Jenkins\jobs\UI Automation\workspace>protractor e2e-tests/protractor.conf.js
Using ChromeDriver directly...
[launcher] Running 1 instances of WebDriver
WARNING - more than one element found for locator by.repeater("attachment in post_item.attachments") - the first result will be used
[31mF[0m
Failures:
1) new_message_with_attachment should send new message with attachment
Message:
[31mElementNotVisibleError: element not visible
(Session info: chrome=47.0.2526.111)
(Driver info: chromedriver=2.13.307647 (5a7d0541ebc58e69994a6fb2ed930f45261f3c29),platform=Windows NT 6.0 SP2 x86_64)[0m
Stacktrace:
ElementNotVisibleError: element not visible
(Session info: chrome=47.0.2526.111)
(Driver info: chromedriver=2.13.307647 (5a7d0541ebc58e69994a6fb2ed930f45261f3c29),platform=Windows NT 6.0 SP2 x86_64)
at new bot.Error (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\atoms\error.js:113:18)
at Object.bot.response.checkResponse (C:\Users\x\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\atoms\response.js:106:9)
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\webdriver.js:362:20
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\goog\base.js:1582:15
at [object Object].webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1654:20)
at notify (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:465:12)
at notifyAll (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:442:7)
at resolve (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:420:7)
at [object Object].fulfill (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:535:5)
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1520:10
==== async task ====
WebElement.click()
at [object Object].webdriver.WebDriver.schedule (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\webdriver.js:345:15)
at [object Object].webdriver.WebElement.schedule_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\webdriver.js:1727:23)
at [object Object].webdriver.WebElement.click (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\webdriver.js:1832:15)
at actionFn (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:75:32)
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:393:17
at Array.forEach (native)
at actionResults.getWebElements.then.then.e.stack (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:392:9)
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\goog\base.js:1582:15
at [object Object].webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1654:20)
==== async task ====
Asynchronous test function: it()
at [object Object].<anonymous> (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\jasminewd\index.js:93:33)
at [object Object].<anonymous> (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\async-callback.js:45:37)
at [object Object].jasmine.Block.execute (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:1174:17)
at [object Object].jasmine.Queue.next_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:2209:31)
at [object Object]._onTimeout (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:2199:18)
Error
at [object Object].ElementArrayFinder.applyAction_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:389:21)
at [object Object].self.(anonymous function) [as click] (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:77:19)
at [object Object].self.(anonymous function) [as click] (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\lib\element.js:697:11)
at [object Object].<anonymous> (C:\Jenkins\jobs\UI Automation\workspace\e2e-tests\new_message_with_attachment.js:223:62)
at C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\jasminewd\index.js:94:14
at [object Object].webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1654:20)
at [object Object].webdriver.promise.ControlFlow.runEventLoop_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1518:8)
Error
at [object Object].<anonymous> (C:\Jenkins\jobs\UI Automation\workspace\e2e-tests\new_message_with_attachment.js:76:3)
at [object Object].jasmine.Env.describe_ (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:913:21)
at [object Object].jasmine.Env.describe (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:898:15)
at describe (C:\Users\X\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:658:27)
at Object.<anonymous> (C:\Jenkins\jobs\UI Automation\workspace\e2e-tests\new_message_with_attachment.js:75:1)
Finished in 59.756 seconds
[31m1 test, 8 assertions, 2 failures
[0m
[launcher] 0 instance(s) of WebDriver still running
Build step 'Execute Windows batch command' marked build as failure
Discard old builds...
Sending e-mails to: me#home.com
Finished: FAILURE*
-------Here is the protractor.conf.js;
*exports.config = {
allScriptsTimeout: 20000,
specs: [
'new_message_with_attachment.js',
],
capabilities: {
'browserName': 'chrome',
'chromeOptions': {
prefs: {'profile.managed_default_content_settings.notifications': 1}},
shardTestFiles: true,
maxSessions: 1
},
directConnect: true,
jasmineNodeOpts: {
defaultTimeoutInterval: 1000000
}
};*
Any comments?
It has been happened once for me when integrate Protractor with Jenkins CI. So according to your question, I will make some suggestions that will help to increase the stability of your e2e test.
As it seem like you are not just having a single problems here. And I am sure this will solve some of your problems but may not solve all of it.
and some errors like "element not found" are most probably because of the uncompleted webpage install..
...I am getting different errors on every run
This is the most common issue with protractor, there are many cause for this to happen. But fortunately it can be avoid with very simple method combine with Page Object.
This is page object:
var MyPage = function () {
// SET ng-view or ui-view
this.view = $('.view');
// SET interactable components
this.button = element(by.css('button'));
};
MyPage.prototype.get = function () {
browser.setLocation('myRoute/something');
};
MyPage.prototype.waitForPresent = function() {
var EC = protractor.ExpectedConditions;
browser.wait(EC.presenceOf(this.button), 10000);
};
In your test suite/spec:
it("should test something", function() {
var page = new MyPage();
page.get();
browser.waitForAngular();
page.waitForPresent();
page.button.click();
});
The trick here is using Expected conditions and wrote a prototype for your page object waitForPresent , waiting for the component(s) to be presented. Pick one that you think it will be the last one to got rendered on your view/page.
(NOTE: you don't need to use page objects, but that will be quite a pain to manage when you suites getting bigger)
But the tests are running very fast, and some errors like "element not found" are most probably because of the uncompleted webpage install..Sometimes it can even enter the login username with some missing characters.
It seem like you does have very good resources there (CPU and RAM), but the point you said that sometimes it missed some characters when logging in. This mean you are still lack of resource else where, and suspecting is your Graphic's resources. Graphic reources is lacking almost all of servers, because it don't always need to render something. And to deal with this, there is no code adjustment, trick or work-around to use. You must at least try either of these (or do both):
Find away to run headless test. Because render browser is quite heavy. (use Chrome or FireFox, because PhantomJS is not recommended for protractor)
Try to increase your graphic resource.
EDIT : there is an issue on github Calling element.sendKeys() seems to not send all the keys it was first suspecting on an old version of Webdriver. But recently people who used latest version of Webdriver got it too. Unknow cause, but I am highly doubt it was due to lacking of some sort of resources. Anyways this is just for your informations. Hope it is helped :)