We are using Jenkins for automated mobile device testing. However, sometimes the devices cannot be connected to and time is wasted running useless tests. To remedy this, we have instituted a connection check that only runs the test flow if a simple test can pass first.
Currently, this set up requires us to use 2 VMs, one to run the test and one to build the first and then wait around until its done to run the condition (we can't downstream for other reasons).
This is a pretty big resource hog, so I wanted to change the condition handling to the jenkins flow using the Build-Flow Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin).
Does the plug-in handle conditionals as below? If so, what would be the correct syntax for the conditional statement?
build("Connection-check")
if(Connection-check.lastBuild == Pass) //Guessed Pseudo-expression
{build("Rest-of-Tests")}
Okay, so in the end I constructed a secondary flow that ran first, and only kicked off the main flow if a connection could be made using the standard conditional tools.
A little bit less compact than I would have liked, but at the least it was clear as to what it did.
Try this condition:
if(hudson.model.Hudson.instance.getItem("Connection-check").getLastBuild().result == hudson.model.Result.SUCCESS)
{
build("Rest-of-Tests")
}
I think you can use also something like that:
import hudson.model.*
def connCheck = build("Connection-check") //Returns hudson.model.AbstractBuild
if(connCheck.result == Result.SUCCESS) {
...
}
Related
I am looking for way to speed up our Build-Pipeline.
The biggest impact would be to do certain things only if they have not been done.
So basically, I have already parameterized some optional stages which works fine, but there are some things I'd like to skip if there was a execution before which was successfull.
I have searched the docs, especially the section ref. when but the was nothing that did the job.
So the question is: Is there something I can do in a declarative pipeline to always skip a stage, except when
It has never been run successfully before
The last time it ran was not successful (eg. if I allow for its execution to be forced by passing a parameter)
I though about using the Build Number (eg only run it on the first execution of the pipeline), but this does't cut it for 2 reasons
I'm using milestones to prevent multiple executions of the pipe for a given branch/PR
If the first run fails, it would never be tried again
Oh, and I also thought about putting all of the logic in post -> regression and forcing the stage to fail on the first pipeline run, but this doesn't seem to be a good idea either.
One option ,although not ideal for large scale, can be to store the diffrent states as Global Environment Variables. (Manage Jenkins -> Configure System -> Global properties -> Environment variables).
These parameters are available for all jobs and can store parameters in a 'server wide' scope.
You can then update or set them via code using the following groovy method:
#NonCPS
def updateGlobalEnvVariable(String name, String value) {
def globalNodeProperties = Jenkins.getInstance().getGlobalNodeProperties()
def envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class)
if (envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0) {
def envVarsNodePropertyClass = this.class.classLoader.loadClass('hudson.slaves.EnvironmentVariablesNodeProperty')
globalNodeProperties.add(envVarsNodePropertyClass.newInstance())
}
envVarsNodePropertyList.get(0).getEnvVars().put(name, value)
}
This function will create the parameter if it does not exists or update its value in case it already exists. Also this function should better be placed in a Shared Library from which it will be available for all pipelines.
A nice advantage for this technique is that you can always 'reset' the different stages from the configuration page, However if you need to store multiple stages it can overflow the configuration page with a bit too much information.
Maybe you could use custom shared-workspace and then create/store some state file or something that could be used by your next executions like lastexecfailed.state and then try to locate the file on the shared workspace at the beginning of your execution.
How do you automatically parse ansible warnings and errors in your jenkins pipeline jobs?
I greatly enjoy the power of leveraging in ansible in jenkins when it works. Upon a failure, the hunt to locate the actual error can be challenging.
I use WarningsNG which supports custom parsers (and allows their programmatic generation)
Do you know of any plugins or addons that already transform these logs into the kind charts similar to WarningsNG?
I figured I'd ask as I go off into deep regex land and make my own.
One good way to achieve this seems to be the following:
select an existing structured output ansible callback plugin (json, junit and yaml are all viable) . I selected junit as I can play with the format to get a really nice view into the playbook with errors reported in a very obvious way.
fork that GPL file (yes, so be careful with that license) to augment with the following:
store output as file
implement the missing callback methods (the three mentioned above do not implement the v2...item callbacks.
forward events to the default or debug callback to ensure operators see something when they execute the plan
add a secrets cleaner - if you use jenkins credentials-binding-plugin it will hide secrets from the console, it will not not hide secrets within stored files. You'll need to handle that in your playbook or via some groovy code (if groovy, try{...} finally { clean } seems a good pattern)
Snippet - forewarding to default callback
from ansible.plugins.callback.default import CallbackModule as CallbackModule_default
...
class CallbackModule(CallbackBase):
CALLBACK_VERSION = 2.0
CALLBACK_TYPE = 'stdout'
CALLBACK_NAME = 'json'
def __init__(self, display=None):
super(CallbackModule, self).__init__(display)
self.default_callback = CallbackModule_default()
...
def v2_on_file_diff(self, result):
self.default_callback.v2_on_file_diff(result)
... do whatever you'd want to ensure the content appears in the json file
I'm setting up a UFT Testcase, where I need to login to Jenkins and perform few operations like creating new jobs. Though I'm able to identify objects using object spy, I'm not able to identify while running the scripts. I'm using descriptive programming. If there are any related material or solution for this, it would be helpful. Also, please let me know if there are any limitations in automating Jenkins. Thanks in advance!
Associated Add-ins: ActiveX,Web,Java
Jenkins version: 2.150.2
UFT version: 14.03
I'm able to automate login if I'm declaring each field as an object. But If I hard code it, then the script is not identifying the objects. I tried inputting all the parameters for each field, even then it's not working.
Please find the code below, where I just highlight and set username field to XXXX
Code
wait(2)
If Browser("title:=Sign in [Jenkins]").Page("title:=Sign in [Jenkins]").WebEdit("name:=j_username","html tag:=INPUT","html id:=j_username").Exist(10) Then
Browser("title:=Sign in [Jenkins]").Page("title:=Sign in [Jenkins]").WebEdit("name:=j_username","html tag:=INPUT","html id:=j_username").highlight
Browser("title:=Sign in [Jenkins]").Page("title:=Sign in [Jenkins]").WebEdit("name:=j_username","html tag:=INPUT","type:=text").Set "XXXX"
else
print("Object was not recognized")
End IF
I have been scouring the internet for days, I have a problem similar to this.
I need to retrieve the console output in raw (plain) text. But if I can get it in HTML that is fine too, I can always parse it. The only thing is that I need to get it during the build step, which is a problem since the location where it should be available is truncated...
I have tried retrieving the console output from the following URL's (relative to the job):
/consoleText
/logText/progressiveText
/logText/progressiveHTML
The two text ones are plain text and would be perfect if not for the truncation, same goes for the HTML one... exactly what I need - only its truncated....
I am sure it is possible to retrieve this information somehow, since when viewing /consoleFull there is a real-time update of the console, without truncating or buffering.
However, upon examining that web page, instead of finding the content I desired, I found this code where it should have been (I did not include the full pages code, since it would be mostly irrelevant, and I believe those answering would be able to find out and know what should be there on their own)
new Ajax.Request(href,{
method: "post",
parameters: {"start":e.fetchedBytes},
requestHeaders: headers,
onComplete: function(rsp,_) {
var stickToBottom = scroller.isSticking();
var text = rsp.responseText;
if(text!="") {
var p = document.createElement("DIV");
e.appendChild(p); // Needs to be first for IE
// Use "outerHTML" for IE; workaround for:
// http://www.quirksmode.org/bugreports/archives/2004/11/innerhtml_and_t.html
if (p.outerHTML) {
p.outerHTML = '<pre>'+text+'</pre>';
p = e.lastChild;
}
else p.innerHTML = text;
Behaviour.applySubtree(p);
if(stickToBottom) scroller.scrollToBottom();
}
e.fetchedBytes = rsp.getResponseHeader("X-Text-Size");
e.consoleAnnotator = rsp.getResponseHeader("X-ConsoleAnnotator");
if(rsp.getResponseHeader("X-More-Data")=="true")
setTimeout(function(){fetchNext(e,href);},1000);
else
$("spinner").style.display = "none";
}
});
Specifically, I am hoping there is a way for me to get the content from text whatever it may be. I am not familiar with this language and so am not sure how I might be able to get the content I want. Plugins won't help since I want to retrieve this content as part of my script during the build step
You did pretty much good investigation already. I can only add the following: all console related plug-ins I know are designed as a post build actions.
The Log Trigger plugin provides a post-build action that allows Hudson
builds to search their console log for a given regular expression and
if found, trigger additional downstream jobs.
So it looks like there is no straightforward solution to your problem. I can see the following options:
1. Use tee or something similar (applicable to shell build steps only)
This solution is far from being universal, but it can provide quick access to the latest console output, produced by a command or set of command.
tee - read from standard input and write to standard output and files
Using synonyms on the system level other Jenkins build steps can modified in order to produce console output. File with console output can be referenced through Jenkins or using any other way.
2. Modify Jenkins code
You can just do a quick fix for internal usage or provide a patch introducing specific system-wide setting.
3. Mimic /console behavior
Code in your example is used to request updates from the Jenkins server. As you may expect the server side can return piece of information starting with some offset. Let me show.
Periodically console page sends requests to the server:
Parameters are straightforward:
Response is a chunk of information to be added:
Another request with updated offset (start) value
You can easily understand there is no data by analyzing Content-Length
So the answer is: use url/job-name/build-number/logText/progressiveHtml, specify start offset, send request and receive console update.
I had a similar issue, the last part of my Jenkinsfile build script needs to parse the ConsoleLog for particular error messages to put in an email build report.
First attempt: http request.
It felt like a hack, it mostly worked, but ran into issues when we locked down access to the Jenkins server & my build nodes could no longer perform annon http gets on the page
Second attempt: use the APIs to enumerate the log lines.
It felt like the right thing to do, but it failed horribly as my nodes would take 30 minutes to get through the 100 meg log files. My presumption is that the Jenkins server was not caching the file, so each request involved a re-reading of the entire file up until the point of the last read.
Third and most successful solution: run grep on the server.
node('master') {
sh 'grep some_criteria $JENKINS_HOME/workspace/path/to/job/console.log'
}
it was fast, reliable, and it didn't matter how big the log files were.
Yes, this required trust of the Jenkins admin and knowledge of the directory paths on the Jenkins server - but since I was the admin, I trusted myself to do the right thing. Your mileage may vary.
To add some insight: when the Jenkins build was in progress, the response for the .../consoleText URL maxed out at 10000 lines, exactly.
I was using 'requests()' package in Python. I have tried the same URL with curl and again received only the first 10K lines.
Only after the build has finished both methods returned the full log (>22K lines in my case).
I will research further and hope to report back.
[2015-08-18] Update: It seems that this is a known issue (see here) and it's fixed in Jenkins 1.618 and later. I am still running 1.615 so I cannot verify.
Amir
I have a Jenkins server that builds/tests about 50 projects. Unfortunately, some of these builds fail, but I don't have a good way to measure whether build failures are increasing or decreasing in frequency over time.
What I'd like is something along these lines:
A report that shows me, over the course of a month, how many jobs were unstable/failed
A report that says "X Days without a broken build" (kind of like at construction sites)
A "Red/Green calendar", that would show on a per-day basis whether any builds were broken
I didn't see any plugins that visualized data in any of these ways, but I'm willing to scrape the Jenkins logs to get the information. Is there a better way to see data similar to this?
I think this work pretty decent using the API. You can get all jobs from your view, then go into the job details and get the build numbers and build date. With those build numbers you can get the corresponding status. You would have to do some coding to collect and display the data, but this would be a possible way.
Another possibility would be using a Groovy script over the console in Manage Jenkins. I do not have much experience working with that feature though, but as you have access to the internal representation it should be pretty easy to get some data out of there.
Finally, the optimal solution would be to write a plugin that does the work, but this is of course also the solution that requires the most effort and know-how.
The Global Build Stats plugin might provide the reporting you're looking for.
(And if you already considered this plugin, I'm curious what problems you ran into.)
As #pushy mentions, the Groovy script console is a good tool to use for these types of statistics gathering. You can use the groovy script in the remote API as well. Here is a starting point for gathering information from all jobs matching a pattern.
def jobPattern='pattern'
Hudson.instance.getItems(Project).each {project ->
def results = [:]
if (project.name.contains(jobPattern)) {
results."$project.name" = [SUCCESS:0,UNSTABLE:0,FAILURE:0,ABORTED:0]
def build = project.getLastBuild()
while (build){
//println "$project.name;$build.id;$build.result"
results."$project.name"."$build.result" = results."$project.name"."$build.result" +1
build=build.getPreviousBuild()
}
}
results.each{name,map->
map.each{result,count->
println "$name : $result = $count"
}
}
}
"Done"
Use this as a start and modify according to your specific requirements.
Try build metric plugin along with Global Build Stat plugin.