I am working on a requirement where I need to show terraform plan output in a nicer way so that user can understand what resources it is going to create etc.
Currently we are printing terraform plan output in jenkins console in json format but we need this output in graphical format like we have Blue Ocean plugin in jenkins. I did research for various plugins bit not find any.
Do we have any kind of Jenkins plugin which shows terraform plan in the form of any visual representation like graph or do we have any alternate option where we can show the terraform plan output anywhere where user can understand it easily as we are looking for good visual representation.
Here is a link for output options of the terraform plan command. I don't think there are any visual representations regarding the plan. We are using json format and find it very readable.
You can install the AnsiColor plugin and enable the 'Color ANSI Console Output' checkbox (under Build Environments) in job configuration.
I found a few workarounds that are not jenkins specific but still work nicely in a pipeline. For simple stuff you can pipe through dot and render it using:
terraform graph -var-file=config/$TARGET.tfvars | dot -Tpng > qa-graph.png
It ain't pretty, but it works...
Now: graphviz will take that dot output and render it a little nicer. It even gives you some pretty sexy rendering options - but even though you can put lipstick on a pig, it's still a pig! The problem is the graph, itself... I think programmers have been trying to solve this for like 30 or 40 years. The '(not-so) old-school' solution is here --> stackoverflow://1494492/graphviz-how-to-go-from-dot-to-a-graph
Currently, I have been playing with graph beautifier which takes the output of tf-graph and renders it as javascript. I just push the html to nginx instead of the output from graphviz. Then, I just print the url to the output of jenkins, and open the link in a new tab. I know it's kinda-hacky, but I couldn't find any cleaner way - which brought me here, today...
ANYWAY: depending on how modular your tf-code is, the graph-beautifier creates a nice little spider-web screenshot of graph-beautifier
Below is a bash snip-it from the end of my jenkinsfile which rendered that screenshot
#------------------------------------------------------------------------------
# * VISUALIZER: data analysis tool to help make deployments & audits faster
# creates a tf-graph file & publish to the web using local nginx server
#> requires nginx & graphviz -- https://graphviz.org
#------------------------------------------------------------------------------
WEBHOST='myserver.example.com'
WEBROOT='/usr/share/nginx/html'
WEBPATH="graphviz/$BRANCH"
WEBDEST="$WEBROOT/$WEBPATH"
WEBPAGE="index.html"
# * create the target dir if it doesn't exist
[ -d $WEBDEST ] || mkdir -vp $WEBDEST
##------------------------------------------------------------------------------
## 21-1124JN - unhappy with just an image I switched to an interactive visualizer
## ? -- https://github.com/pcasteran/terraform-graph-beautifier
##------------------------------------------------------------------------------
terraform graph | /usr/share/go/bin/terraform-graph-beautifier \
--exclude="module.root.provider" \
--output-type=cyto-html \
--embed-modules=true \
> $WEBDEST/$WEBPAGE
echo -en "\n\n tf-graph --> http://$WEBHOST/$WEBPATH \n\n"
##------------------------------------------------------------------------------
## NOTE: the beautifier is just a POC. it's something I have used before and
##> threw together in a few hours for demo. There are newer projects that
##> are actively maintained & seem to have a lot cleaner ways of working with
##> larger data-sets & graphs. One is called the Rover - Terraform Visualizer
##> and is available here --> https://github.com/im2nguyen/rover check it out
##------------------------------------------------------------------------------
last but not least (as I mention in my code comments) is the Rover Terraform Visualizer - a new one that came out of the brain of one of the hashicorp-kids which is super-promising and has recently been released to open source. It is also written in go, but I haven't gotten it to work yet in my pipeline. Check out his demo.
Related
How do you automatically parse ansible warnings and errors in your jenkins pipeline jobs?
I greatly enjoy the power of leveraging in ansible in jenkins when it works. Upon a failure, the hunt to locate the actual error can be challenging.
I use WarningsNG which supports custom parsers (and allows their programmatic generation)
Do you know of any plugins or addons that already transform these logs into the kind charts similar to WarningsNG?
I figured I'd ask as I go off into deep regex land and make my own.
One good way to achieve this seems to be the following:
select an existing structured output ansible callback plugin (json, junit and yaml are all viable) . I selected junit as I can play with the format to get a really nice view into the playbook with errors reported in a very obvious way.
fork that GPL file (yes, so be careful with that license) to augment with the following:
store output as file
implement the missing callback methods (the three mentioned above do not implement the v2...item callbacks.
forward events to the default or debug callback to ensure operators see something when they execute the plan
add a secrets cleaner - if you use jenkins credentials-binding-plugin it will hide secrets from the console, it will not not hide secrets within stored files. You'll need to handle that in your playbook or via some groovy code (if groovy, try{...} finally { clean } seems a good pattern)
Snippet - forewarding to default callback
from ansible.plugins.callback.default import CallbackModule as CallbackModule_default
...
class CallbackModule(CallbackBase):
CALLBACK_VERSION = 2.0
CALLBACK_TYPE = 'stdout'
CALLBACK_NAME = 'json'
def __init__(self, display=None):
super(CallbackModule, self).__init__(display)
self.default_callback = CallbackModule_default()
...
def v2_on_file_diff(self, result):
self.default_callback.v2_on_file_diff(result)
... do whatever you'd want to ensure the content appears in the json file
I have verified that I can run both normal office and python macros from within office but I still haven't figured out how to run one (even the hello world one) from the command line.
I have googled and looked at other answers here but I'm still not entirely clear how to run an open office macro from the command line:
https://forum.openoffice.org/en/forum/viewtopic.php?f=20&t=8232
suggests to use:
office writer.odt "macro://Standard.Module1.Macro1()"
I have also seen:
office "macro://Standard.Module1.Macro1()" writer.odt
Either way around this just opens the document and and neither runs a macro nor reports an error.
Whereas How to call an existing LibreOffice python macro from a python script
suggests to run office listening on a port and communicate via that.
If I can get that far I still need to find API documentation that will explain how to insert an anchor (as per my other question
asciidoc: is there a way to create an anchor that will be visible in libreoffice writer?)
I'm using RHEL7 for context.
update
oowriter "foo.odt" macro:///Standard.Module1.addXref
works with a office basic macro.
I still haven't figured out the python one.
One issue is I can't find any debug information to look at. Are there any log files anywhere?
Another issue is which version of python to use.
The RHEL package installs site packages for python 2.7.
>rpm -q --whatprovides /usr/bin/writer
libreoffice-writer-4.3.7.2-5.el7_2.1.x86_64
>rpm -ql libreoffice-pyuno-4.3.7.2-5.el7_2.1
...
/usr/lib64/python2.7/site-packages/uno.py
Libreoffice5.1 includes 3.5 with the distro:
>/opt/libreoffice5.1/program/python --version
Python 3.5.0
So to start with I am looking for a hello world python example that pairs a known version of python with a known version of office. Preferably either of the two above (writer 4.3.7 & python 2.7? or writer 5.1 & python 3.5).
update2
Using python3.5 as installed with office5.1 I have a working hello world using the following:
import uno
# get the uno component context from the PyUNO runtime
localContext = uno.getComponentContext()
# create the UnoUrlResolver
resolver = localContext.ServiceManager.createInstanceWithContext(
"com.sun.star.bridge.UnoUrlResolver", localContext )
# connect to the running office
ctx = resolver.resolve( "uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext" )
smgr = ctx.ServiceManager
# get the central desktop object
desktop = smgr.createInstanceWithContext( "com.sun.star.frame.Desktop",ctx)
# access the current writer document
model = desktop.getCurrentComponent()
# access the document's text property
text = model.Text
# create a cursor
cursor = text.createTextCursor()
# insert the text into the document
text.insertString( cursor, "Hello World", 0 )
This works with either version of open office via:
/usr/bin/oowriter --accept="socket,host=localhost,port=2002;urp;StarOffice.ServiceManager"
So the final part is to add the cross reference.
It looks like the command needs an extra slash. This worked for me on Ubuntu:
lowriter "Untitled 1.odt" macro:///Standard.Module1.SayHello
That calls this method in the module named Module1 under My Macros & Dialogs / Standard:
Sub SayHello
MsgBox("Hello, World!")
End Sub
The above approach only works for Basic macros. For Python macros, the standard command line approach is to connect to a listening instance of Office. Warning: This will be much (perhaps 10x) slower than running from within Office.
The link you suggested shows how to call a Python macro from a different Python script, which is more complex than what we need here. Instead, put the connecting code (starting with localContext = uno.getComponentContext()) and macro code in the same script. For an example of what should go in the script, see "First play with the Python shell to get familiar" at http://christopher5106.github.io/office/2015/12/06/openoffice-libreoffice-automate-your-office-tasks-with-python-macros.html.
As far as creating anchors, there are a number of different objects in LibreOffice that can function as anchors:
Headings
Bookmarks - example code to create them
Tables and Frames
Sections
Images and OLE Objects
This list was copied from How do I check for broken internal links in Star Basic?. In your other question you also asked about checking for broken links, so hopefully that question is helpful.
One way to create a hyperlink is to edit the HyperLinkURL property of some text. For example, say there is a bookmark called MyBookmark. Then the following code changes the currently selected text into a hyperlink:
viewcursor = currentController.getViewCursor()
viewcursor.HyperLinkURL = "#MyBookmark"
EDIT:
Regarding which version of python to use, currently LibreOffice uses python 3 and OpenOffice uses python 2.
For debugging information, you can set a checkpoint in the Basic IDE. For python, I use the logging module. OpenOffice also has various log files but I normally do not find them helpful.
Regarding problems with python, did you try the link I posted? If so, how far did you get?
I do not think you will find many RHEL examples. Try to get it working on a desktop distro like Ubuntu first, and then adapt that approach to RHEL.
I have been scouring the internet for days, I have a problem similar to this.
I need to retrieve the console output in raw (plain) text. But if I can get it in HTML that is fine too, I can always parse it. The only thing is that I need to get it during the build step, which is a problem since the location where it should be available is truncated...
I have tried retrieving the console output from the following URL's (relative to the job):
/consoleText
/logText/progressiveText
/logText/progressiveHTML
The two text ones are plain text and would be perfect if not for the truncation, same goes for the HTML one... exactly what I need - only its truncated....
I am sure it is possible to retrieve this information somehow, since when viewing /consoleFull there is a real-time update of the console, without truncating or buffering.
However, upon examining that web page, instead of finding the content I desired, I found this code where it should have been (I did not include the full pages code, since it would be mostly irrelevant, and I believe those answering would be able to find out and know what should be there on their own)
new Ajax.Request(href,{
method: "post",
parameters: {"start":e.fetchedBytes},
requestHeaders: headers,
onComplete: function(rsp,_) {
var stickToBottom = scroller.isSticking();
var text = rsp.responseText;
if(text!="") {
var p = document.createElement("DIV");
e.appendChild(p); // Needs to be first for IE
// Use "outerHTML" for IE; workaround for:
// http://www.quirksmode.org/bugreports/archives/2004/11/innerhtml_and_t.html
if (p.outerHTML) {
p.outerHTML = '<pre>'+text+'</pre>';
p = e.lastChild;
}
else p.innerHTML = text;
Behaviour.applySubtree(p);
if(stickToBottom) scroller.scrollToBottom();
}
e.fetchedBytes = rsp.getResponseHeader("X-Text-Size");
e.consoleAnnotator = rsp.getResponseHeader("X-ConsoleAnnotator");
if(rsp.getResponseHeader("X-More-Data")=="true")
setTimeout(function(){fetchNext(e,href);},1000);
else
$("spinner").style.display = "none";
}
});
Specifically, I am hoping there is a way for me to get the content from text whatever it may be. I am not familiar with this language and so am not sure how I might be able to get the content I want. Plugins won't help since I want to retrieve this content as part of my script during the build step
You did pretty much good investigation already. I can only add the following: all console related plug-ins I know are designed as a post build actions.
The Log Trigger plugin provides a post-build action that allows Hudson
builds to search their console log for a given regular expression and
if found, trigger additional downstream jobs.
So it looks like there is no straightforward solution to your problem. I can see the following options:
1. Use tee or something similar (applicable to shell build steps only)
This solution is far from being universal, but it can provide quick access to the latest console output, produced by a command or set of command.
tee - read from standard input and write to standard output and files
Using synonyms on the system level other Jenkins build steps can modified in order to produce console output. File with console output can be referenced through Jenkins or using any other way.
2. Modify Jenkins code
You can just do a quick fix for internal usage or provide a patch introducing specific system-wide setting.
3. Mimic /console behavior
Code in your example is used to request updates from the Jenkins server. As you may expect the server side can return piece of information starting with some offset. Let me show.
Periodically console page sends requests to the server:
Parameters are straightforward:
Response is a chunk of information to be added:
Another request with updated offset (start) value
You can easily understand there is no data by analyzing Content-Length
So the answer is: use url/job-name/build-number/logText/progressiveHtml, specify start offset, send request and receive console update.
I had a similar issue, the last part of my Jenkinsfile build script needs to parse the ConsoleLog for particular error messages to put in an email build report.
First attempt: http request.
It felt like a hack, it mostly worked, but ran into issues when we locked down access to the Jenkins server & my build nodes could no longer perform annon http gets on the page
Second attempt: use the APIs to enumerate the log lines.
It felt like the right thing to do, but it failed horribly as my nodes would take 30 minutes to get through the 100 meg log files. My presumption is that the Jenkins server was not caching the file, so each request involved a re-reading of the entire file up until the point of the last read.
Third and most successful solution: run grep on the server.
node('master') {
sh 'grep some_criteria $JENKINS_HOME/workspace/path/to/job/console.log'
}
it was fast, reliable, and it didn't matter how big the log files were.
Yes, this required trust of the Jenkins admin and knowledge of the directory paths on the Jenkins server - but since I was the admin, I trusted myself to do the right thing. Your mileage may vary.
To add some insight: when the Jenkins build was in progress, the response for the .../consoleText URL maxed out at 10000 lines, exactly.
I was using 'requests()' package in Python. I have tried the same URL with curl and again received only the first 10K lines.
Only after the build has finished both methods returned the full log (>22K lines in my case).
I will research further and hope to report back.
[2015-08-18] Update: It seems that this is a known issue (see here) and it's fixed in Jenkins 1.618 and later. I am still running 1.615 so I cannot verify.
Amir
I am trying to get a report on ITM 6.2.1 regarding each equipment and situations running with some of the configuration info.
I need to list each equipment, each situation, formula and the system command with the mail send. Is there a way to get this info without having to go manually into each equipment, situation, etc?
Example:
Equip: equip01
Agent: LinuxOS
Situations: LINUX_FILE_SIZE, LINUX_UNIX_FS_CRITICAL, etc
Formula: FILE: '/local/file.err' SIZE: !=0,000
Action: System command: usr/bin/mail oper#mail.com
Many thanks!
I think there are many ways to do this, but I would look into some shell scripting using tacmd commands, like "tacmd listsit -m AGENT" and "tacmd viewsit -s SITUATION" you can automate the work by combining the outputs of these commands and create a report that way.
Also, there is a cool tool called "ITMSUPER" that connects to you ITM environment through SOAP calls and creates really useful reports about the entire environment, you should definitely take a look:
https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Use%20ITMSUPER%20to%20Solve%20ITM%20Issues/page/Some%20Useful%20Examples%20of%20ITMSUPER%20for%20Beginners?lang=en
I have a Jenkins server that builds/tests about 50 projects. Unfortunately, some of these builds fail, but I don't have a good way to measure whether build failures are increasing or decreasing in frequency over time.
What I'd like is something along these lines:
A report that shows me, over the course of a month, how many jobs were unstable/failed
A report that says "X Days without a broken build" (kind of like at construction sites)
A "Red/Green calendar", that would show on a per-day basis whether any builds were broken
I didn't see any plugins that visualized data in any of these ways, but I'm willing to scrape the Jenkins logs to get the information. Is there a better way to see data similar to this?
I think this work pretty decent using the API. You can get all jobs from your view, then go into the job details and get the build numbers and build date. With those build numbers you can get the corresponding status. You would have to do some coding to collect and display the data, but this would be a possible way.
Another possibility would be using a Groovy script over the console in Manage Jenkins. I do not have much experience working with that feature though, but as you have access to the internal representation it should be pretty easy to get some data out of there.
Finally, the optimal solution would be to write a plugin that does the work, but this is of course also the solution that requires the most effort and know-how.
The Global Build Stats plugin might provide the reporting you're looking for.
(And if you already considered this plugin, I'm curious what problems you ran into.)
As #pushy mentions, the Groovy script console is a good tool to use for these types of statistics gathering. You can use the groovy script in the remote API as well. Here is a starting point for gathering information from all jobs matching a pattern.
def jobPattern='pattern'
Hudson.instance.getItems(Project).each {project ->
def results = [:]
if (project.name.contains(jobPattern)) {
results."$project.name" = [SUCCESS:0,UNSTABLE:0,FAILURE:0,ABORTED:0]
def build = project.getLastBuild()
while (build){
//println "$project.name;$build.id;$build.result"
results."$project.name"."$build.result" = results."$project.name"."$build.result" +1
build=build.getPreviousBuild()
}
}
results.each{name,map->
map.each{result,count->
println "$name : $result = $count"
}
}
}
"Done"
Use this as a start and modify according to your specific requirements.
Try build metric plugin along with Global Build Stat plugin.