We have our software product which is written in perl. We have setup a Jenkins build for that which runs the automated tests, to get the coverage values we run the unit tests under Devel::Cover, and get the coverage values for each module. But I find it very difficult to go and check the coverge values for each module individually, also it doesn't tell us where are we heading as far as coverage is concerned (increasing/decreasing trend). So what I'd like is to have a history or graph of those values, so that I can have a better picture of where our coverage values are and how do they compare to, say a month ago.
Is there a plug-in available that would do this thing?
I searched the Internet and found some plugins like cobertura and emma that I hope would achieve that purpose, but they are for Java. Is there a good alternative for Perl?
Also, would it be worth it to develop our own tool to display such plots? We already have the coverage data in Jenkins, and all we need is just to get that data and plot it.
You can with luck use Devel::Cover's HTML output formatting, put by default in the cover_db directory by:
$ cover -test
The install and configure the HTML Publisher Plugin.
Configure it as so:
HTML directory: cover_db
index page: coverage.html
Report title: Coverage Report
I written up my notes for setting this up, and they are available at:
https://logiclab.jira.com/wiki/display/OPEN/Continuous+Integration#ContinuousIntegration-CoverageTests
I am currently looking into the Jenkins Plugin Clover, since Devel::Cover can also output in Clover format. I will update my notes soon.
You can with luck use my CPAN distribution Task::Jenkins, installing all of the CPAN dependencies needed for the setup on the referenced wiki page.
Related
I'm trying to get reporting working for Karate DSL, and it's proven a challenge because my team uses Circle CI instead of Jenkins. Cucumber reporting seems to only work for Jenkins.
I've had a look at this documentation, here:
https://github.com/intuit/karate/tree/master/karate-demo#example-report
https://github.com/jenkinsci/cucumber-reports-plugin
I was wondering if there is a circle friendly equivalent you could recommend? It'd be even better if the reports could be generated in the terminal. It's going to be a hard sell to convince my team to change CI tools just so I can implement a test framework.
Thanks!
Here's what I suggest:
If you follow the demo / doc instructions - you will get the HTML reports in say target/cucumber-html-reports, and this is "pure Maven and Java", no dependency on CircleCI at all so far.
Now all you need to do is somehow make these HTML reports accessible via the web. In Jenkins, there is an HTML Publisher Plugin. I am not familiar with CircleCI but a quick search suggests that there is a way to expose links to build artifacts.
Also note that when you follow the demo, Java JUnit XML reports would also be output to target/cucumber-reports. It looks like CircleCI has support for these which means that it should be able to derive the build pass/fail status and stats if configured right.
Also note that Karate now enables you to write custom reports: https://stackoverflow.com/a/66773839/143475
I'm calculating code coverage on every build done in Jenkins and producing a coverage XML report. That's recorded really nicely inside of Jenkins with Cobertura, but what I'd really like is to be able to somehow get at the total branch coverage number so I can automatically publish to a medium the rest of my team can easily consume (i.e. Slack).
As a bonus, getting the difference in coverage from the last run would be even better. I don't see any environment variables that hold this, and haven't found anything detailing a simple way to do this in Cobertura docs. I know I can hack some code together to do this myself, but if there's a simple way I'm missing or something someone else has already built, I'd much rather do that.
I believe you'll have to do this yourself.
The existing Slack plugin just sends build start/success/failure notifications etc.
The Cobertura plugin unfortunately doesn't seem to be built on top of the static code analysis plugin, so there probably isn't much in the way of graphs, difference reports and all that.
You could try adding /api/json to the end of a Cobertura report URL for a Jenkins build — most endpoints reveal some information in this way. If there's some useful information, that could be a basis for whatever you want to hack together.
Reasking my older question:
Java test coverage: who covers what?
Background: I look at sonar's coverage report for a class and want to know, which test contributes to the coverage of a specific line / branch, so that it easy to got to that test and add the test for the newly introduced if-branch.
Are there other (preferably free) alternatives to clover in the IDE? Perhaps even such that they can be included into sonar ?
Or maybe tricks to enhance, accumulate information with some scripting in emma-reports ?
Or even further, patch emma or cobertura to log the required info (instead of logging a "1" for counting, one could well log the names of class under test and the test, I assume)
Thanks!
You should definitely give a try to JaCoCo. Its integration with Sonar allows to benefit from new features, for example :
merge coverage by unit and integration tests. See http://www.sonarsource.org/sonar-3-3-in-screenshots/
track the relations between tests and tested code (since sonar 3.5). You can find a screenshot on the documentation page: http://docs.codehaus.org/display/SONAR/Resource+Viewer#ResourceViewer-CoverageTab
I am using sonar for the last few months and want to know that do sonar works in an incremental way or not i.e if i do soanr analysis for the first time on my project code it will definitely analyze all the code but if i do some enhancement on my core source code and update some files then after updation do sonar analysis again on the same code then will sonar analyze all the files OR only analyze files which i have updated?
I am using "Sonar way with Findbugs" as my default quality profile.
Is there any way to use sonar in an incremental way, to analyze only updated files?
Is this possible in sonar or not?
Kindly revert your help will be appreciated..
Thanks in advance..
Sonar does not currently handle incremental analyses.
If you want, you can watch and vote for the following ticket: http://jira.codehaus.org/browse/SONAR-2815
If your task if code quality metrics in a general way, then you can use these tools directly on Jenkins. See FindBugs Plugin , PMD plugin or Checkstyle Plugin. They independently give what you need in the form of "incremental code" reports. But if you have to use SONAR only, then Fabrice's answer is the correct one.
Here the post where the ticket that #fabrice-sonar-team comments was born:
does Sonar support Incremental code quality analysis
You can read Freddy Mallet's explanation about why this functionality is not trivial to be implemented. It requires lot of effort, so as Fabrice said, vote for the ticket! :)
Just adding on since this is the first google result for 'Sonar incremental preview analysis' in stack, and the answers are way too old.
Sonar has plugins available for IDEs such as Eclipse and NetBeans that can run incremental analysis on the changed files alone. This still needs a connection the SonarQube server though.
You may also run the analysis right from Jenkins by passing an additional sonar analysis property - sonar.analysis.mode=incrementalin your Sonar scan build step. This will report the code quality in a full report - will all code issues as well as a light report containing only new issues (since last full scan as recorded in server).
To take one step further use sonar.issuesReport.html.enable=true and publish the generated html reports to your Jenkins build page - Neat and Trim
Sonar documentation here
We've written a framework to test the performance of our Java application (none of the existing frameworks, eg JMeter, were appropriate). The framework produces various metrics, e.g. mean/min/max transactions per second.
We'd like each Jenkins build to display these metrics so that we can keep track of whether a commit has improved performance or not.
I can't figure out how to do this.
One idea is to modify our performance test framework to output a HTML file, then somehow make Jenkins display/link to it on the build results page.
Any advice gratefully received.
The Peformance Plugin can show the results of JMeter and JUnit test in the nice, graphical fashion. And on the plugin page there is a description on how to use it.
This is an open-source plugin hosted on GitHub. The JUnit and JMeter parser are already there, but You can implement your own just by subclassing PerformanceReportParser. It's pretty easy and you can just fork the repo and start your implementation.
I agree that it is hard (if not impossible) to squeeze all the information into standard formats, like JUnit. They are good for quick identification of problems. Once you know there is a problem - you need more information that is usually free-form or custom-formatted to fit your particular needs. So we use both: JUnit that can be immediately processed by Jenkins to decide if the build is stable or not, draw the nice trend graph, etc. We also produce an HTML report that is much more detailed.
Now to your immediate question: you can simply archive your HTML file as an artifact (there is a standard post-build step to do that). Then a link to it will be displayed among the artifacts for the build. There are permalinks to the latest artifacts and latest successful build artifacts:
http://[server]/job/[job_name]/lastCompletedBuild/artifact/foo.html
http://[server]/job/[job_name]/lastSuccessfulBuild/artifact/foo.html
You may bookmark those links and have quick and easy one-click access to your results.
You could use the HTML Publisher Plugin to publish the resulting HTML page. That would be pretty straightforward.
If you want better integration you could try to create output that follows the same format JMeter produces, and use the Performance Plugin.
For best result you could take Łukasz advice and modify the Performance Plugin to your needs. That requires the most effort on your part, of course.