we are using pitest plugin in sonarqube and set up a quality gate on mutation coverage, since it is much more valuable than other coverage measurements.
Only, when a project is pushed that does not use pitest, the quality gate is not triggerd at all.
Is it possible to define a quality gate to detect that pitest is not defined for a project at all?
Thank you
Simple answer to your question: no, it's not. Quality gate conditions can not be triggered by the absence of a metric. So a metric has to be computed.
But the most recent version of sonar-pitest-plugin (0.9) does only compute coverage information when Mutation Analysis data is present (= a pitest report exists).
The change however is not overly complicated, if you need an urgent fix, check the PitestComputer class at line 84 and add the following else block:
if (mutationsTotal != null) {
...
} else {
context.addMeasure(PitestMetrics.MUTATIONS_KILLED_PERCENT_KEY, 0.0);
}
Compile it and install it manually to your instance.
That said, I want to give you a short heads up, that there is a newer plugin addressing Mutation Analysis in SonarQube (full disclosure: I'm the author), with several new features, rules etc.
The plugin is available via the market place (named "Mutation Analysis").
The plugin has the same limitation as the sonar-pitest-plugin, but I just created a new issue addressing your problem:
https://github.com/devcon5io/mutation-analysis-plugin/issues/13
Edit:
This feature is implemented in version 1.3
Related
My managers want we to determine which tests might have to be run, based on coding changes that were made to the application we are testing.
But, it is hard to know which tests are actually needed to be re-verified as a result of a code change. What we have done is common to test the entire area where the code change occurred / or the entire proj, solution.
We were told this could be achieved by TFS build or MTM tools. Could someone share the details?
PM:We are running on TFS 2015 update4,VS2017.
There is a concept of Test Impact Analysis which helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
The Test Impact Analysis (TIA) feature specifically enables this – TIA
is all about incremental validation by automatic test selection. For a
given code commit entering the pipeline TIA will select and run only
the relevant tests required to validate that commit. Thus, that test
run is going to complete faster, if there is a failure you will get to
know about it faster, and because it is all scoped by relevance,
analysis will be faster as well.
Test Impact Analysis for managed automated tests is available via a checkbox in the 2.* preview version of the VSTest task.
If enabled, only the relevant set of managed automated tests that need to be run to validate a given code change will run. Test Impact Analysis requires the latest version of Visual Studio, and is presently supported in CI for managed automated tests.
However this is only available with TFS2017 update1(need 2.* preview version of VSTS task). More details please refer this blog: Accelerated Continuous Testing with Test Impact Analysis
I am using Jenkins to kick off Sonar-runner for analyzing projects.
Now I use the Build Breaker plugin.
This breaks a build when a quality gate is reporting that the quality is below/above given values.
I'd like to change the quality gate used by the Sonar-Runner, on a per-job basis in Jenkins.
Looking up at Analysis Parameters documentation:
**sonar.profile** parameter is to change the language to analyse
I would thought that for the gate: sonar.gate would work (it doesn't)
Question:
What parameter for sonar-runner has to be used to change the "Quality Gate"?
Used versions:
Jenkins 1.598
Sonar 4.5.1 (LTS)
The property is sonar.qualitygate.
It was not documented because this is generally not a good idea to change the quality gate that should be used during a standard analysis - but it's true that it can be useful with the build breaker on preview analyses.
I updated the documentation.
I tried above solution but it did not work for me. Finally I figured out that with new version of sonar-scanner we can achieve it only from Sonar-Qube UI.
Follow below steps:
Login to Sonar Quber server as administrator and select projects. Search for the project name you want to add the quality gate for. Select the searched project and below screen screen would appear.
Navigate to Administration > Quality Gates
Select the quality gate from the drop-down which you want to use for this project (A default quality gate is assigned by-default for all the projects).
Hope this helps. Thanks!
I'm running a number of static analysis tools and I want to track the results from build to build. For example, if a commit to a branch increases the number of security vulnerabilities, I want to send an email to the committer. I know there are plugins like Sonar and Analysis Collector, but they don't cover all of the areas of analysis I want and they don't seem to have the ability to trigger actions based on build trends (correct me if I'm wrong).
You can use the Groovy Postbuild Plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Postbuild+Plugin
It lets you extract data (such as number of vulnerabilities detected) from the current build's buildlog with num_vul = manager.getLogMatcher(regexp)
And compare that to previous builds, by extracting info from their buildlog, e.g.:
currentBuildNumber = manager.build.number
manager.setBuildNumber(currentBuildNumber - 1)
prev_num_vul = manager.getLogMatcher(regexp)
Then, if the number of vulnerabilities had gone up, I would call manager.buildFailure() which sets the build status to FAILURE, and then have the next PostBuild step be to the Email-ext plugin which allows you to send email to the committer in the event of a failure.
I would recommend the SonarQube tool, which does just what you describe. You mention that you already looked at it, but maybe you missed the Notifications feature or the Build Breaker Plugin. There are more SonarQube features centered around Jenkins integration. SonarQube is free to use.
If you are still missing something, it might be worthwhile asking specifically how that aspect could be covered by SonarQube. Just my two cents.
I am using sonar for the last few months and want to know that do sonar works in an incremental way or not i.e if i do soanr analysis for the first time on my project code it will definitely analyze all the code but if i do some enhancement on my core source code and update some files then after updation do sonar analysis again on the same code then will sonar analyze all the files OR only analyze files which i have updated?
I am using "Sonar way with Findbugs" as my default quality profile.
Is there any way to use sonar in an incremental way, to analyze only updated files?
Is this possible in sonar or not?
Kindly revert your help will be appreciated..
Thanks in advance..
Sonar does not currently handle incremental analyses.
If you want, you can watch and vote for the following ticket: http://jira.codehaus.org/browse/SONAR-2815
If your task if code quality metrics in a general way, then you can use these tools directly on Jenkins. See FindBugs Plugin , PMD plugin or Checkstyle Plugin. They independently give what you need in the form of "incremental code" reports. But if you have to use SONAR only, then Fabrice's answer is the correct one.
Here the post where the ticket that #fabrice-sonar-team comments was born:
does Sonar support Incremental code quality analysis
You can read Freddy Mallet's explanation about why this functionality is not trivial to be implemented. It requires lot of effort, so as Fabrice said, vote for the ticket! :)
Just adding on since this is the first google result for 'Sonar incremental preview analysis' in stack, and the answers are way too old.
Sonar has plugins available for IDEs such as Eclipse and NetBeans that can run incremental analysis on the changed files alone. This still needs a connection the SonarQube server though.
You may also run the analysis right from Jenkins by passing an additional sonar analysis property - sonar.analysis.mode=incrementalin your Sonar scan build step. This will report the code quality in a full report - will all code issues as well as a light report containing only new issues (since last full scan as recorded in server).
To take one step further use sonar.issuesReport.html.enable=true and publish the generated html reports to your Jenkins build page - Neat and Trim
Sonar documentation here
I am putting together a bunch of applications to create an automatic building for microsoft platform (the products I chose and the software I will build, both, runs on windows). The products I've chosen are:
Code repository: SubVersion
Continuous integration: CruiseControl
Unit testing: NUnit
Test coverage: NCover
Static code analysis: FXCop
Now I need to choose a bug/issue tracking system (free if possible) that can be, in some way, integrated with the previous products.
What I mean by integration? Well, all these products have a file as output I want to be able to publish errors and bugs found by them into the tracking system.
Do you know some product, some technique or trick that can help me to do this?
Thanks in advance.
First off, these are all tools I have experience with and congratulate you on your choices - these tools will serve you well if you use them wisely.
The most common usage of these tools is that CC would fail the build if certain criteria are not met, e.g.:
A unit test fails
Code coverage falls below a certain threshold
FXCop detects a violation of a certain severity
Because the build would fail and in continuous integration a failed build should be fixed immediately, you wouldn't really need to put those issues into a bug tracking system. Think of build-failing errors as being as severe as the code not compiling - you drop everything and fix right away.