TestCase properties access with TestRunner - jenkins

My testCases have custom properties defined as TestCase properties. When i launch the test in the SoapUI environment the properties value changes, but when i use the testrunner doesn't.
I have a "current" var containing the current user info to use in the testCase. Each time the TestCase executes the current var increases in one but when using the testrunner (in jenkins & soapui) it doesn't. The increase code is the following and it's saved in a Groovy Script contained in the testCase.
def i = context.testCase.getPropertyValue("_current");
if(Integer.parseInt(i)+1 < Integer.parseInt(total)) {
context.testCase.setPropertyValue("_current",String.valueOf(Integer.parseInt(i)+1));
log.info("Current user "+testRunner.testCase.getPropertyValue("_current"));
} else {
context.testCase.testSuite.setPropertyValue("_current","0");
log.info "###__!!! Reached limit of users !!!__###";
}
I think the testRunner doesn't execute the groovy scripts contained in the testCase. How can I make it work?

I figured it out
The TestCase and TestSuite properties are embedded in the XML project file. For the changes to persist save is needed. Just need to add the -S flag in the testrunner command.

Related

Select a different runner for cucumber.api.cli.Main?

Is it possible to define/specify a runner when starting tests from cucumber's command line(cucumber.api.cli.Main)?
My reason for this is so i can generate xml reports in Jenkins and push the results to ALM Octane.
I kind of inherited this project and its using gradle to do a javaexect and call cucumber.api.cli.Main
I know its possible to do this with #RunWith(OctaneCucumber.class) when using JUnit runner + maven (or only JUnit runner), otherwise that tag is ignored. I have the custom runner with that tag but when i run from cucumber.api.cli.Main i can't find a way to run with it and my tag just gets ignored.
What #Grasshopper suggested didn't exactly work but it made me look in the right direction.
Instead of adding the code as a plugin, i managed to "hack/load" the octane reporter by creating a copy of the cucumber.api.cli.Main, using it as a base to run the cli commands and change a bit the run method and add the plugin at runtime. Needed to do this because the plugin required quite a few parameters in its constructor. Might not be the perfect solution, but it allowed me to keep the gradle build process i initially had.
public static byte run(String[] argv, ClassLoader classLoader) throws IOException {
RuntimeOptions runtimeOptions = new RuntimeOptions(new ArrayList<String>(asList(argv)));
ResourceLoader resourceLoader = new MultiLoader(classLoader);
ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
Runtime runtime = new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
//====================Added the following lines ================
//Hardcoded runner(?) class. If its changed, it will need to be changed here also
OutputFile outputFile = new OutputFile(Main.class);
runtimeOptions.addPlugin(new HPEAlmOctaneGherkinFormatter(resourceLoader, runtimeOptions.getFeaturePaths(), outputFile));
//==============================================================
runtime.run();
return runtime.exitStatus();
}

Jenkins: read an existing job's plugin config via Groovy

We are in the early phases of using Jenkins DSL. One challenge we have come across is being able to read in an existing jobs plugin settings so that we retain it before running the DSL. This allows us the ability to give the Jenkins users the option to keep some of their settings. We have successfully retained the schedule settings for our jobs but the newest challenge is being able to retain a plugin setting. Specifically a setting in the "ExtendedEmailPublisher" plugin. We would like to retain the value:
In the config.xml file for this job in the ExtendedEmailPublisher tags we see the following:
<publishers>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>Our_Team#Our_Team.com</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList/>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<recipientProviders>
<hudson.plugins.emailext.plugins.recipients.ListRecipientProvider/>
</recipientProviders>
<attachmentsPattern/>
<attachBuildLog>false</attachBuildLog>
<compressBuildLog>false</compressBuildLog>
<replyTo>$PROJECT_DEFAULT_REPLYTO</replyTo>
<contentType>project</contentType>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern/>
<presendScript>$DEFAULT_PRESEND_SCRIPT</presendScript>
<classpath/>
<attachBuildLog>false</attachBuildLog>
<compressBuildLog>false</compressBuildLog>
<replyTo>$DEFAULT_REPLYTO</replyTo>
<saveOutput>false</saveOutput>
<disabled>false</disabled>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
The value we would like to extract/preserve from this XML is:
<disabled>false</disabled>
We have tried getting the existing values using groovy but cant seem to find the right code. Our first idea was to try to read the value from the config.xml using the XmlSlurper. We ran this from the Jenkins Script Console:
def projectXml = new XmlSlurper().parseText("curl http://Server_Name:8100/job/Job_Name/api/xml".execute().text);
*we use 8100 for our Jenkins port
Unfortunately while this does return some config info it does not return plugin info.
Then, we also tried running the following to read/replace the existing settings:
def oldJob = hudson.model.Hudson.instance.getItem("Job_Name")
def isDisabled = false // Default Value
for(publisher in oldJob.publishersList) {
if (publisher instanceof hudson.plugins.emailext.ExtendedEmailPublisher) {
isDisabled = publisher.disabled
}
}
And while this works if executed from the Jenkins Script Console, when we try to use it in a DSL Job we get the message:
Processing provided DSL script
ERROR: startup failed:
script: 25: unable to resolve class
hudson.plugins.emailext.ExtendedEmailPublisher
# line 25, column 37.
if (publisher instanceof hudson.plugins.emailext.ExtendedEmailPublisher)
{
1 error
Finished: FAILURE
SOLUTION UPDATE:
Using url #aflat's URL suggestion for getting the raw XML config info, I was able to use the XML Slurper and then use the getProperty method to assign the property I wanted to a variable.
def projectXml = new XmlSlurper().parseText("curl http://Server_Name:8100/job/Job_Name/config.xml".execute().text);
def emailDisabled = projectXml.publishers."hudson.plugins.emailext.ExtendedEmailPublisher".getProperty("disabled");
If you want to parse the config.xml, use
def projectXml = new XmlSlurper().parseText("curl http://Server_Name:8100/job/Job_Name/config.xml");
That should return your raw config.xml data
Under "Manage Jenkins->Configure Global Security" did you try disabling "Enable script security for Job DSL scripts"?

Using the ez-template plugin for Jenkins through the Jenkins Job DSL doesn't apply the template after creation

I am working on automating the creation of Jenkins jobs by using the Jenkins Job DSL (Groovy). Right now, I am trying to automate the creation of a job that uses the ez-template plugin to use an already existing template and apply that to my newly created job. However, after I am done writing the necessary configuration:
job('foo') {
properties {
templateImplementationProperty {
exclusions(['ez-templates', 'job-params', 'disabled', 'description'])
syncAssignedLabel(true)
syncBuildTriggers(true)
syncDescription(false)
syncDisabled(false)
syncMatrixAxis(true)
syncOwnership(true)
syncScm(true)
syncSecurity(true)
templateJobName('template')
}
}
}
the job gets created alright... except the template is never applied until AFTER I manually hit the save button on the UI in the newly created job. Checking the config.xml of the created job I can see that the xml contains the configuration I specified, but it was never applied.
Looking at the ez-template code, I can see that this is due to the silentSave feature that was implemented in that plugin - it writes configuration to disk without triggering any save events.
I've tried methods available to the Jenkins API but I've had no success there. Any ideas on how I can apply my configuration?
Full disclosure: I'm a co-worker, and was able to help shredmasteryjm solve this. I figured it'd be best to put this out on the net for others.
The Groovy code needed to trigger template implementation contents to be updated is:
import hudson.model.*;
import jenkins.model.*;
import com.joelj.jenkins.eztemplates.utils.TemplateUtils;
import com.joelj.jenkins.eztemplates.TemplateImplementationProperty;
Jenkins j = Jenkins.getInstance()
Item job = j.getItemByFullName('foo')
TemplateImplementationProperty template = TemplateUtils.getTemplateImplementationProperty(job)
TemplateUtils.handleTemplateImplementationSaved(job, template)
This utilizes the EZ-Templates TemplateUtils class to trigger the actual save event, using the template that the job uses. Of note, if job 'foo' doesn't implement a template, then the 'template' variable will be null, causing this code to error. YMMV
In our case, we needed to also add in some useful information from another question: Access to build environment variables from a groovy script in a Jenkins build step ( Windows)
in order to utilize a parameterized job name. As such our completed script looks like this:
import hudson.model.*;
import jenkins.model.*;
import com.joelj.jenkins.eztemplates.utils.TemplateUtils;
import com.joelj.jenkins.eztemplates.TemplateImplementationProperty;
// get current thread / Executor
def thr = Thread.currentThread()
// get current build
def build = thr?.executable
def hardcoded_param = "parameter_job_name"
def resolver = build.buildVariableResolver
def hardcoded_param_value = resolver.resolve(hardcoded_param)
Jenkins j = Jenkins.getInstance()
Item job = j.getItemByFullName(hardcoded_param_value)
TemplateImplementationProperty template = TemplateUtils.getTemplateImplementationProperty(job)
TemplateUtils.handleTemplateImplementationSaved(job, template)
FYI ez-templates 1.3.0 now triggers off additional save events such that you do not need the above trick.

Jenkins, how to check regressions against another job

When you set up a Jenkins job various test result plugins will show regressions if the latest build is worse than the previous one.
We have many jobs for many projects on our Jenkins and we wanted to avoid having a 'job per branch' set up. So currently we are using a parameterized build to build eg different development branches using a single job.
But that means when I build a new branch any regressions are measured against the previous build, which may be for a different branch. What I really want is to measure regressions in a feature branch against the latest build of the master branch.
I thought we should probably set up a separate 'master' build alongside the parameterized 'branches' build. But I still can't see how I would compare results between jobs. Is there any plugin that can help?
UPDATE
I have started experimenting in the Script Console to see if I could write a post-build script... I have managed to get the latest build of master branch in my parameterized job... I can't work out how to get to the test results from the build object though.
The data I need is available in JSON at
http://<jenkins server>/job/<job name>/<build number>/testReport/api/json?pretty=true
...if I could just get at this data structure it would be great!
I tried using JsonSlurper to load the json via HTTP but I get 403, I guess because my script has no auth session.
I guess I could load the xml test results from disk and parse them in my script, it just seems a bit stupid when Jenkins has already done this.
I eventually managed to achieve everything I wanted, using a Groovy script in the Groovy Postbuild Plugin
I did a lot of exploring using the script console http://<jenkins>/script and also the Jenkins API class docs are handy.
Everyone's use is going to be a bit different as you have to dig down into the build plugins to get the info you need, but here's some bits of my code which may help.
First get the build you want:
def getProject(projectName) {
// in a postbuild action use `manager.hudson`
// in the script web console use `Jenkins.instance`
def project = manager.hudson.getItemByFullName(projectName)
if (!project) {
throw new RuntimeException("Project not found: $projectName")
}
project
}
// CloudBees folder plugin is supported, you can use natural paths:
project = getProject('MyFolder/TestJob')
build = project.getLastCompletedBuild()
The main test results (jUnit etc) seem to be available directly on the build as:
result = build.getTestResultAction()
// eg
failedTestNames = result.getFailedTests().collect{ test ->
test.getFullName()
}
To get the more specialised results from eg Violations plugin or Cobertura code coverage you have to look for a specific build action.
// have a look what's available:
build.getActions()
You'll see a list of stuff like:
[hudson.plugins.git.GitTagAction#2b4b8a1c,
hudson.scm.SCMRevisionState$None#40d6dce2,
hudson.tasks.junit.TestResultAction#39c99826,
jenkins.plugins.show_build_parameters.ShowParametersBuildAction#4291d1a5]
These are instances, the part in front of the # sign is the class name so I used that to make this method for getting a specific action:
def final VIOLATIONS_ACTION = hudson.plugins.violations.ViolationsBuildAction
def final COVERAGE_ACTION = hudson.plugins.cobertura.CoberturaBuildAction
def getAction(build, actionCls) {
def action = build.getActions().findResult { act ->
actionCls.isInstance(act) ? act : null
}
if (!action) {
throw new RuntimeException("Action not found in ${build.getFullDisplayName()}: ${actionCls.getSimpleName()}")
}
action
}
violations = getAction(build, VIOLATIONS_ACTION)
// you have to explore a bit more to find what you're interested in:
pylint_count = violations?.getReport()?.getViolations()?."pylint"
coverage = getAction(build, COVERAGE_ACTION)?.getResults()
// if you println it looks like a map but it's really an Enum of Ratio objects
// convert to something nicer to work with:
coverage_map = coverage.collectEntries { key, val -> [key.name(), val.getPercentageFloat()] }
With these building blocks I was able to put together a post-build script which compared the results for two 'unrelated' build jobs, then using the Groovy Postbuild plugin's helper methods to set the build status.
Hope this helps someone else.

How create Jenkins Build Pipeline depends on test result?

I have 3 Jenkins jobs. Smoke tests, critical path test (part 1), critical path test (part 2).
Now it starts one by one. I need create build PipeLine depends on test result. I need to take into account the results of a single test (#Test annotation in TestNG), ignoring the overall result of test suite.
I want to get the configuration like this:
Smoke tests -> If specified test passed then run critical path test Part 1 and Part 2 on different nodes
So, please tell me how depends in Jenkins only on one tests result (not all suite)?
You can try to use some of build log analysis plugins:
https://wiki.jenkins-ci.org/display/JENKINS/Text-finder+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task
Scan build output, and downgrade build result to failure on specific text.
Next in downstream item check option "Build after other projects are built" in build triggers section. Set proper upstream item name and set proper trigger result.
I solved that task by using 2 Jenkins extensions:
https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin
Create properties file from test. File contains property, that indicate result of test step status
With EnvInject Plugin add new step into Jenkins Job (step must be after test run) and inject parameter value (from file created at first step)
Create build flow with Build Flow Plugin
Write groovy script:
smokeTest = build( "Run_Smoke_Test" )
def isTestStepSuccessful = smokeTest.environment.get( "TestStepSuccessful" )
if (isTestStepSuccessful != "false") {
parallel (
{
build("Run_Critical_Path_Part_1_Test")
build("Run_Critical_Path_Part_3_Test")
},
{
build("Run_Critical_Path_Part_2_Test")
}
)
}
build( "Run_Critical_Path_Final_Test" )

Resources