I have my jmeter script running from a jenkins job but it is always reporting it as failed even though the script actually passed. I am using the 'Publish Performance test result report Post-build Action. Please see screenshot. What am I doing wrong? Why does it always show an error even though the actual jmeter script is passing?
You set the threshold for Unstable/Failure to 0, so even 0 consider as failure.
Increase the thresholds for Unstable and Failed
I found a way to do this! Use this plugin: https://wiki.jenkins.io/display/JENKINS/Log+Parser+Plugin
In a rules file containing this line:
error /FAILED/
In the script add a beanshell Assertion for each HTTP Request that will log messages for successes and failures. Here's one example:
String testscenario = "TEST SCENARIO: ";
String requestName = "Find Stuff";
String findStuff = "${MyStuff}";
String custName = "${CustName}";
String respData = new String(ResponseData);
if (respData.contains(findStuff))
{
log.info(testscenario+"Passed. "+requestName+" MyStuff for Cust: " +
custName + " containing " + findStuff + " returned.");
print(testscenario+"Passed. "+requestName+" MyStuff for Customer " +
custName + " containing " + findStuff + " returned.");
} else
{
log.error(testscenario+"*FAILED. "+requestName);
print(testscenario+"*FAILED. "+requestName);
}
Related
I have a Jenkins instance with multiple teams using it. It gets cluttered frequently and people don't delete their test jobs after testing. Is there a way to delete unused jobs (not run in the last 6 months) including their workspace?
You can use the following Groovy script for this. Comment out the delete part and test it before using just to make sure it's what you need :)
def deleteBefore = "2022/07/01"
Jenkins.instance.getAllItems(Job.class).each { jobitem ->
def jobName = jobitem.getFullName()
def deleteBeforeTime = new Date(deleteBefore).getTime()
def build = jobitem.getLastBuild()
if(build == null || build.getTimeInMillis() <= deleteBeforeTime){ // If no builds, build is null
println build == null ? "Job " + jobName + " has never run, deleting the Job" : "Job " + jobName + " last ran on (" + build.getTime() + ") hence deleting"
jobitem.delete()
}
}
I have groovy application, and I am using the following code to make some json validations work:
String method = "{\n" +
"\"method\":\"filesContentReplacer\"," +
"\"filesContentReplacer\":{\n" +
"\"files\":[" +
"{" +
"\"path\":\"pom.xml\"," +
"\"target\":\"1.9.0\"," +
"\"replacement\":\"STF_SomeNumber\"" +
"}" +
"]" +
"}" +
"}"
def json = new JsonSlurper().parseText(method)
use(JsonSchema){
assert json instanceof Map
json.schema = (Map) executeReadJson([text: readFromResources('methods/filesContentReplacer.json')])
Log.instance.debug( "json.schema:${json.schema}")
assert json.conformsSchema()
}
That is pure groovy code, based from this git repo: https://github.com/alexdeleon/groovy-json-schema
The problem is when I run the code on a jenkins pipeline, it throws the following error:
expected to call org.codehaus.groovy.runtime.GroovyCategorySupport.use but wound up catching org.jenkinsci.plugins.workflow.cps.CpsClosure2.call; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/
After some research, it turns out, that the guys who made this pipeline groovy language, have not made everything they should have, and the keyword 'use' is not supported.
Now I need somehow to make it work. Can you help me?
Also, if you need any more code, I will provide. But I basically just copied the 2 classes from the git into my project.
Thank you for your time.
don't have jenkins to test but here what you could try:
1/
org.codehaus.groovy.runtime.GroovyCategorySupport.use(JsonSchema){...}
2/
#NonCPS
def myConformsSchema(String json, String schema){
...here put your code to validate json against schema
}
3/
you could try to use static methods from https://github.com/alexdeleon/groovy-json-schema/blob/master/src/main/groovy/com/lumata/os/groovy/jsonschema/JsonSchema.groovy
like this:
def schema = executeReadJson(...)
assert JsonSchema.conformsSchema(json, schema)
I am looking for a way to get a list of build and source code management section in the configuration.
For Example: I have jobs with Testcomplete and SoupUI Pro plug ins, in each jobs, in the configuration section, we input the git repo name, testsuite name, directory etc ...
Currently, I am having to go to each job, click on configuration and get the values I need, would be nice where I can get all this information for all the jobs. I looked in configuration slicing, but it does not have the section I need.
Thank you for your help in advance.
I have not used either plugin you mention, but this crude little groovy script will find every Freestyle job and if it has a gitSCM step, report the primary Git repo url, then for every "Invoke top-level Maven target", report the POM value if set.
Run from <JENKINS_URL>/script or in a Jenkins job with an "Execute System Groovy Script" (not an "Execute Groovy script").
You can modify to find your plugin's builder step and properties. You can get the values by examining <job>/config.xml instead of <job>/configure.
Updated example to specifically include also looking up TestComplete plugin (com.smartbear.jenkins.plugins.testcomplete.TcTestBuilder) values.
Similar approach left to OP for ReadyAPI Functional Testing plugin - (com.smartbear.ready.jenkins.JenkinsSoapUIProTestRunner)
WARNING: soapui-pro-functional-testing:1.6 transmits passwords in clear text. So too does SmartBear's Zephyr products. Does not inspire confidence in a company who's tagline is "quality matters more than ever.... we'll help you get there."
Jenkins.instance.allItems.findAll() {
it instanceof hudson.model.FreeStyleProject
}.each { job ->
if (job.scm instanceof hudson.plugins.git.GitSCM) {
println job.fullName + ' | ' + job.scm.userRemoteConfigs.url[0]
job.builders.findAll() {
it instanceof hudson.tasks.Maven
}.each {step ->
println ' : ' + step?.pom
}
job.builders.findAll() {
it instanceof com.smartbear.jenkins.plugins.testcomplete.TcTestBuilder
}.each {step ->
if (step?.getLaunchType() == 'lcRoutine') {
println ' : ' + step?.getProject() + ' : ' + step?.getUnit() + ' : ' + step?.getRoutine()
}
}
}
}
return
ps: I'm sure there's a cleaner way to iterate but I can't do all the work. Be sure to use the "?" to handle null values
I've got a regex with sample text that is working on regex101, but doesn't seem to work in my Jenkins pipeline scenario. So I'm assuming I've got a mistake in my pipeline script, but I fail to see where.
Here's a repro:
pipeline {
agent any
stages {
stage ('Test') {
steps {
script {
echo ("Test")
output = "Some text. \n\n 12 scenarios (3 failed, 2 success) plus text \n\n and some more text"
def hasSummaryMatch = (output ==~ /\d+ scenarios \([^()]+\)/)
echo ("hasSummaryMatch = " + hasSummaryMatch)
if (!hasSummaryMatch) {
error ("No summary!")
}
}
}
}
}
}
I've tested this with Jenkins 2.60.2 running in the official Docker container.
This provides the following (abbreviated) output:
Started by user Administrator
Running on master in /var/jenkins_home/workspace/Test001
Test
hasSummaryMatch = false
ERROR: No summary!
Finished: FAILURE
The expected output is no error because there should be a match.
What am I doing wrong?
Just use =~ (the find operator) instead of ==~ (the match operator):
def hasSummaryMatch = (output =~ /\d+ scenarios \([^()]+\)/)
When match operator ==~ is used, then a strict match of the input string required
I have tests automation framework, and the test runs are driven through Jenkins. After the test run is completed, it generates a few custom test reports, as HTML file. In the report, I have a table with a total number of errors as one of the columns. I basically want to parse this HTML file and send out an email to a mailing list if the error is greater than 0.
Here is what I have done till now, in my Jenkins job:
a. Configured the job to run the tests [this is in an execute shell option and running my jar, as the test framework is a java application]
b. I have a post-build action, in which I execute a python code, which does the following:
Step 1. Get the latest HTML report.
Step 2. Parse this file.
Step 3. If the error is > 0:
then email_flag = true
else:
email_flag = false.
Now is there a way I can use the above flags and send email notifications, via Jenkins plugins? I just want to understand the best way to do this. Any help is appreciated.
P.S. I did some additional research and I see the Email-ext plugin has a pre-send script option, but I just don't know how to use this to run the python code and handle the email messaging.
During your build action - ie while executing the tests, when you know the test fails in java, immediately or after executing all the tests , you can make java return a value to the shell.
System.exit(0) - Success
System.exit(1) - Failure
Whenever it is not 0, jenkins jobs is marked as Failed by default.
You can use Email-ext plugin to send the email only when the job is failed by selecting the appropriate trigger - Failure - Any
Below is my Extended Email Publisher Jenkins plugin pre-send Groovy script. Hope you are able to adjust it to your needs.
It is designed for parsing Serenity-JUnit XML files but with little effort it can parse HTML. It will look for any file in given sourceDirPath which match pattern "SERENITY-JUNIT-*xml". It will then file.readToString(), extract test results from that xmlString and aggregate those results into variables tests, passed, failures, errors, skipped. Finally it will compose a msgSubject and append it to msg itself.
NOTE that build, logger and msg are objects made available by the plugin. You should not change assign them and/or change their names.
Script:
import static groovy.io.FileType.FILES
import javax.xml.parsers.DocumentBuilderFactory
import javax.xml.parsers.DocumentBuilder
import org.w3c.dom.Document
import org.w3c.dom.NodeList
import org.w3c.dom.Node
import org.w3c.dom.Element
import org.xml.sax.InputSource
import java.io.StringReader
logger.println "------------------------------------------------------------------------------------------------"
logger.println "Parsing SERENITY-JUNIT XML files for results to update Editable Email Notification."
// determine if build is remote, list folder and files
def sourceDirPath
if(build.workspace.isRemote()){
channel = build.workspace.channel
logger.println "Using channel: " + channel
sourceDirPath = new FilePath(channel, build.workspace.toString() + "\\target\\site\\serenity\\")
} else {
sourceDirPath = new FilePath(new File(build.workspace.toString() + "\\target\\site\\serenity\\"))
}
logger.println "Source dir: " + sourceDirPath
def files = sourceDirPath.list("SERENITY-JUNIT-*xml")
logger.println "Found XML files: \n" + files
// initialize result variables
def tests = 0
def failures = 0
def errors = 0
def skipped = 0
def passed = 0
// parse files
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance()
files.each{
logger.println "Starting to parse file: " + it
channel = build.workspace.channel
def file = new FilePath(channel, it.getRemote())
def xmlString = file.readToString()
logger.println "Extracted XML:\n" + xmlString
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder()
Document doc = dBuilder.parse(new InputSource(new StringReader(xmlString)))
doc.getDocumentElement().normalize()
Element testsuiteNode = doc.getDocumentElement()
logger.println testsuiteNode
tests += Integer.parseInt(testsuiteNode.getAttribute("tests"))
failures += Integer.parseInt(testsuiteNode.getAttribute("failures"))
errors += Integer.parseInt(testsuiteNode.getAttribute("errors"))
skipped += Integer.parseInt(testsuiteNode.getAttribute("skipped"))
logger.println "Completed parsing file: " + it
}
// calculate passed tests amount
passed = tests - failures - errors - skipped
// aggregated results
logger.println "Aggregated results:"
logger.println "Tests: " + tests
logger.println "Passed: " + passed
logger.println "Failures: " + failures
logger.println "Errors: " + errors
logger.println "Skipped: " + skipped
// compose message subject
def msgSubject
if(tests == 4){
if(failures > 0 || errors > 0){
msgSubject = "FAILED - Passed ${passed}/${tests}, Failures ${failures}, Errors ${errors}, Skipped ${skipped}."
}else if(skipped > 0){
msgSubject = "OK (w/ skipped tests) - Passed ${passed}/${tests}, Failures ${failures}, Errors ${errors}, Skipped ${skipped}."
}else{
msgSubject = "OK - Passed ${passed}/${tests}, Failures ${failures}, Errors ${errors}, Skipped ${skipped}."
}
}else{
msgSubject = "FAILED - Not all tests or too many tests executed (${tests})."
}
// set message subject
msg.setSubject(msg.getSubject() + ": " + msgSubject)
logger.println "------------------------------------------------------------------------------------------------"
You may of course dynamically change any element of the email (To, From, MsgText, etc.). Here I am only changing the subject.
The result is something like:
Wellness Check: OK - Passed 4/4, Failures 0, Errors 0, Skipped 0.