At my workplace, I've been tasked to look into some metrics that the Jenkins tool provides and somehow pull them programatically and display them in some presentable format. The metrics that I need to pull are:
How many unit tests are passing? Failing? Skipping? The total % of passing?
How many integration tests are passing? Failing? Skipping? The total % of passing?
How many acceptance tests are passing? Failing? Skipping? The total % of passing?
How long does it take to execute the test? Make the build?
What is the number of tests executing in pipelines?
... the list goes on
Now I have a very small 1000 ft understanding of Jenkins, and an even smaller understanding of the steps that I need to take to make this program come to life. I am an intern with not much programming experience either, but after some research, I learned that I can navigate through the Jenkins API by adding '.../api' to the link that I want to find API elements for, and I know that I'm going to need to develop a plugin. Aside from that I don't have much direction at all. I don't know what environment I need to develop these plugins (Maven? Never heard of it)... I don't know what languages are supported (I only know C++, Java, and JS)... I don't know how to even install a plugin or get to the plugin on the Jenkins site. I feel like I'm drinking from a firehose with this task and need some guidance.
Does anyone have good guides, advice, tips, tricks, videos... anything that might help me get started on Jenkins plugin development? Any insight into how I might solve this problem too would be much appreciated.
Thanks so much.
First of all there are tools out there which generates HTML reports. You can start there.
For example: MSTest report (.trx) can be converted to HTML by TRXER
and can be published using the HTML Publisher Plugin
However if you're into building your own plugin use NetBeans (I have tried it; and it works)
But creating Jenkins graphs you have to google and see.
Related
This is sort of an open-ended question/request (hope that's allowed).
On my team we are using Karate API testing for our project, which we love. The tests are easy to write and fairly understandable to people without coding backgrounds. The biggest problem we're facing is that these API tests have some inherent degree of flakiness (since the code we're testing makes calls to other systems). When running the tests locally on my machine, it's easy to see where the test failed. However, we're also using a Jenkins pipeline, and when the tests fail in Jenkins it's difficult to see why/how they failed. By default we get a message like this:
com.company.api.OurKarateTests > [crossdock] Find Crossdock Location.[1:7] LPN is invalid FAILED
com.intuit.karate.exception.KarateException
Basically all this tells us is the file name and starting line of the scenario that failed. We do have our pipeline set up so that we can pass in a debug flag and get more information. There are two problems with this however; one is that you have to remember to put in this flag in every commit you want to see info on; the other is that we go from having not enough information to too much (reading through a 24MB file of the whole build).
What I'm looking for is suggestions on how to improve this process, preferably without making changes to the Jenkins pipeline (another team manages this, and it will likely take a long time). Though if changing the pipeline is the only way to do this, I'd like to know that. I'm willing to "think outside the box" and entertain unorthodox solutions (like, posting to a slack integration).
We're currently on Karate version 0.9.3, but I will probably plan to upgrade to 0.9.5 as part of this effort. I've read a bit about the changes. Would the "ExecutionHook" thing be a good way to do this? I will be experimenting with this on my own a bit.
Have other teams/devs faced this issue? What were your solutions? Again we really love Karate, just struggling with the integration of it to Jenkins.
Aren't you using the Cucumber Reporting library as described here: https://github.com/intuit/karate/tree/master/karate-demo#example-report
If you do - you will get an HTML report with all traffic (and anything you print) in-line with the test-steps, and of-course error traces, and most teams find this sufficient for build troubleshooting, there is no need to dig through logs.
Do try upgrade as well, because we keep trying to improve the usefulness of the logs, and you may see improvements if you had failed in a JS block or karate-config.js.
Else, yes the ExecutionHook would be a good thing to explore - but I would be really surprised if the HTML report does not give you what you need.
I am trying to track the page speed of certain urls of my project on each merging of the pull requests in Github and output the results of report in HTML format or JSON file. On the CI side, I am going to use Jenkins. I have no prior knowledge on performance testing. I want to know about the best approach to automate the speed test, integrate it with Jenkins and output the result.
On researching over the internet, I noted few possibilities which could be done to achieve this goal.
Installing "Page Speed Insights (psi) node package", creating the script that uses the psi for fetching the speed of certain pages, generating the test reports for use with Jenkins. (Referred to this link by Oxagile)
Performance testing using Jmeter and integrating with Jenkins.
Performance analysis using LightHouse. (Referred to this link by Timo Stollenwerk)
Choosing the right approach is very important. Therefore, I would be very grateful if anyone can suggest me different approaches and thus the right one to use(with examples if possible)in my case to achieve this goal.
Thank you in advance.
After quite a bit of research, I found out that sitespeed.io is the best solution for achieving this goal. It is a complete web performance tool that helps us to measure the performance of the website. It is best for running in the continuous integration to find web performance regressions on commits and monitoring them in production and alerting on regressions.
It is easy to find simple examples for declarative or scripted pipeline. But when the point comes where you go deep into scripting you need so much more information. When you're not familiar to the world of web, java and groovy you are running out of questions which can be asked to go future. Googeling helps you find some magic "hudson.model.Hudson..." or .methods and e.g. #NonCPS-operators solutions. Those solutions work, but I'm searching for the bigger context to work my self from the bottom up. Not from the top down. I'm looking for the knowledge, which is obvious to the insiders.
I'm looking for links/books/api-references or introductions to learn to find the entrance to knowledge around the jenkins scripted pipeline. e.g. like this one =).
I am not looking for answers to those questions below from the stackoverflow communety. This would be to much! I am looking for links of documentation to get deep into the topic. I assume that for an insider it's insider knowledge is not obvious. So I'm stating here some questions to make it obvious what I would describe as insider knowledge.
Example questions:
like : "hudson.model.Hudson..." but where do I get those magical dot.separated strings?
Is there a documentation of the Jenkins Api?
How can I find documentation of the classes and methods usable in jenkins like e.g. X.Y.collect?
Is there a way to debug a pipeline?
Is there a faster way in testing code than every time run it in a pipeline?
How does the inner mechanism work?
Is the Knowledge more about groovy or is it about general Jenkins? Or is it java?
Why println MyArrayList.getClass() class java.util.ArrayList which is a java class? does grooy inherit the types from java, or does the pipeline inherit the types from jenkins, which is java?
...
Asking one question at a time:
where do I get those magical dot.separated strings?
Those are inner java classes at the Jenkins core (or plugins). For the former, Javadoc is available, the latter have their code at Github
classes and methods usable in jenkins
Mostly every Java and Groovy class/method is usable
debug a pipeline?
You can only replay it, issuing changes on each Run
testing
you have two approaches: LesFurets one and the real-unit-one
innards
wide question and wider answer. pipelines are loaded, transformed and run as a near to groovy code (#NonCPS annotation alters this behaviour).
Knowledge about Java, Groovy and Jenkins will apply.
Groovy indeed extends Java hence, both languages apply
Does anyobdy know a good solid CI service that provides the common features of build parallelization BUT also support for Junit reports?
The current ones that we have looked at (semaphoreapp, circleCI, travisCI,...) are good but relatively useless as we have to manually investigate what tests failed, since when, and how often, thus negating a lot of the benefits of a hosted service.
Things that we're looking to know (and are all provided by JUnit / Jenkins):
If the build failed, because of what test cases?
Total Number of Failures / Total Number of Tests (trends to better analyze things)
Individual Track record of any test (so we know exactly when it was broken, whether it's intermittent,...)
You mentioned the most famous CI services but there are alternatives where you can get a higher customization level, like installing plugins, fine configuration, etc.
CloudBees and ClinkerHQ are both based on Jenkins offered as a service. You can also get very useful metrics (coverage, failures, graphs, execution times, etc.) thanks to Jenkins Plugins and SonarQube. I think Jenkins and SonarQube are a perfect couple for you.
Notifications are very important too. You want to be notified when something is wrong. This feature is available on both.
Regards,
Antonio.
DISCLAIMER: I'm deeply involved in ClinkerHQ
I LOVE cucumber, my clients love it too.
As far as I know currently there isn't a nice way to share your features with your clients. Us nerds have TextMate or NetBean bundles that give us nice syntax highlighting -- my clients not so much.
What I would love is to be have something hosted at features.myclientsapp.com that would be a organized nice marked up view of the features of the application. Maybe as a bonus an overview page with % coverage, which steps are passing. Ideally this will be exposed as a Rack Engine.
If I am getting greedy -- git integration to see version control, and a way to solicit feedback from the clients.
Does anyone know of anything that does can do this? What other strategies do people have on sharing there features files with their clients/users?
I have been working on this and this is what I have come up with. Its a less known feature that cucumber can be output in pretty nice html. I have this task namespaced as part of a bigger task list that is run with rake doc:features and includes all the rdoc for the app and the README for the app, etc.
desc "runs cucumber features and outputs them to doc/cucumber as html"
task :html do
Dir.glob("features/*.feature").each do |story|
system("cucumber",
"#{story}",
"--format=html",
"-o",
"doc/cucumber/#{story.gsub(/features\/(\w*).feature/, '\1.html')}")
end
end
then its up to you how you want to serve them up. I've been writing some tasks that hook this task in with others to build the documentation and then serve it up with the serve gem. http://github.com/jlong/serve but there are a lot of other options too. other options include running the features on a ci server and putting these feature files in a directory to be browsed, etc.
I agree with you, it would be nice if there was a dashboard page that gave pass fail, etc. and links to each feature file output, etc. If anyone would like share the workload implementing this as part of cucumber core, I would be happy to contribute. I personally think the html formatting should be more robust and part of the central cucumber feature set.
I really like this idea. What do you think about using this-fork of metric_fu that claims to combine cucumber with the rcov and other nice pretty graphs.
As far as formatting the feature themselves, I really like how Chargify uses cucumber features as documentation. They appear to wrap them in a 'pre' tag to be pre-formatted.
I just found Viewcucumber. I haven't used it yet since it currently does not support cucumber 0.10.0, but I will be monitoring it -- looks great.
I new service that looks promising is Relish
Though it is a closed beta, and I couldn't get access. But one to keep an eye on.
features2html is a script that creates a self-contained HTML file from all Cucumber feature files in a folder.
P.S. Self promotion alert :) D.S.