I am building a simple CI pipeline for my python code in Jenkins using Jenkinsfile, which basically does the following things:
Creating test environment and installing dependencies.
Running static code metrics:
various raw metrics : SLOC, comment lines, blank lines, Cyclomatic Complexity, the Maintainability Index etc.
tests coverage reports using coverage
errors and style check using pylint
Testing pulled source code (Unit testing)
So lets say i have this stage for example, for "Static code metrics":
...
stage('Static code metrics') {
steps {
echo "Raw metrics"
sh ''' radon raw --json my_python_repo/ > raw_report.json
radon cc --json my_python_repo/ > cc_report.json
radon mi --json my_python_repo/ > mi_report.json
//TODO: add conversion and HTML publisher step
'''
}
}
...
As you see above, the reports are saved in .json format. I need to figure out a way to publish these reports in a pretty visual way on Jenkins dashboard.
One of the steps is to convert the .json files to HTML and then use HTML publisher plugin to publish reports but i don't know what tool to use.
If there is way to solve this or any other way to publish these reports on jenkins dashboard, please provide the solution.
Lets say content of mi_report.json is:
{"my_python_repo/code.py": {"mi": 16.42950884051172, "rank": "B"}, "my_python_repo/test.py": {"mi": 33.532817596089814, "rank": "A"}}
There is no Jenkins-ready tool to publish custom JSON as HTML.
But you can use warnings plugin to publish your metrics in the Jenkins way.
Here small instruction on how to configure Jenkins to parse radon reports.
Related
I developed automated tests in Java. The XML test report is generated with junit 5 and xray-junit-extension. This XML is currently being integrated in Jira/Xray, but unfortunately the labels are not being added to the issue.
I believe that the labels could be integrated in two different ways, 1) through this XML test report, or, alternatively, 2) through the Jenkins pipeline itself.
My XML contains the following property :
Click here to see the screenshot
Similarly to what is written in the Xray documentation :
Click in order to see the screenshot of the documentation
https://docs.getxray.app/display/XRAY/Taking+advantage+of+JU...
The only difference is that in the Xray documentation there is a wrap around the tags property. In my XML I do not have that wrap.
Do you happen to have any idea on why the label is not being added in Jira/Xray?
The second approach would be using the XrayImportBuilder to add a label, using importInfo
step([$class: 'XrayImportBuilder',
endpointName: '/junit',
importFilePath: '/reports/.xml',
projectKey: 'P34AMA',
importToSameExecution: 'false',
//testExecKey: 'TSTLKS-753',
serverInstance: '3146a388-d399-4e55-ae28-8c65404d6f9d',
credentialId:'55287529-194d-4e91-9964-7d740d8d2f61',
importInfo: "{ "fields": {"labels": ["label"]}",
//importInfo = '{"fields": {"labels": ["EOD"]} }'
])
Problem using XrayImportBuilder in Jenkins
But when adding importInfo to my pipeline it ends with an issue :
Click here to see the Jenkins logs
Clcik here to see the Jenkins Import Step
Is anyone aware of any other way to add a label to jira automatically without using the hudson.plugins.jira.JiraIssueUpdater ?
Thank you very much for your help!
I have a Pylint running in a Jenkins pipeline. To implement it, I used Gerrit trigger plugin and Next Generation Warnings plugin. Everything is working as expected - Jenkins is joining the review, checks change with pylint and generates report.
Now, I'd like to post pylint score in a custom "Build successful" message. I wanted to pass the pylint score to a environment variable and use it in dedicated window for Gerrit plugin message.
Unfortunately no matter what I try, I cannot pass any "new" variable to the message. Passing parameters embedded in pipeline works (e.g. patchset number).
I created new environment variable in Configure Jenkins menu, tried exporting to shell, writing to it (via $VAR and env. syntax) but nothing works - that is, build message displays raw string like $VAR instead of what variable contains.
What should I do to pass local pylint score (distinct for every pipeline occurence) to the custom build message for Gerrit?
I don't think the custom message can be used for this. This is just supposed to be a static message.
They way I do this is to use the SSH command to perform the review. You can also achieve the same using the REST API.
First I run my linting and white space checking script that will generate a json file with the information I would like to pass to Gerrit. Next I send it to Gerrit using SSH. See below my pipeline script and an example json file.
As a bonus I have added the robot comments. This will now show up in your review as a remark from Jenkins that line 8 of my Jenkins file has a trailing white space. You can easily replace this with your lint result of you like or just ignore it and only put the message. It is easier to use a json file as it will make it easier to create multi line messages
node('master') {
sh """
cat lint_change.json | ssh -p ${env.GERRIT_PORT} ${env.GERRIT_HOST} gerrit review ${env.GERRIT_PATCHSET_REVISION} --json
"""
}
Example json file:
{
"labels": {
"Code-Style": "-1"
},
"message": "Lint Bot Review\nLint Results:\n Errors: 0\n Warnings: 0\n\nWhitespace results:\n Errors: 1",
"robot_comments": {
"Jenkinsfile": [
{
"robot_id": "lint-bot",
"line": "8",
"message": "trailing whitespace."
}
]
}
}
Alternatively, you may want to look at a new gerrit-code-review-plugin that should make this things even easier. However, I have not tried this yet.
How can we get result of summary report parameters (throughput, received & sent bytes) in JMeter script through non gui mode? I have to implement the benchmarking on whole script rather than each thread to mark the status of script pass/fail by comparing the result to the static .csv file which contains the value of parameters .Kindly let me know the approach to opt.
The easiest way is going for JMeterPluginsCMD Command Line Tool, it can generate various tables and charts out of JMeter's .jtl results file.
So you will need to add the following command as the Post Build Step:
JMeterPluginsCMD --generate-csv SummaryReport.csv --input-jtl result.jtl --plugin-type AggregateReport
You can install JMeterPluginsCMD Command Line Tool using JMeter Plugins Manager
I have two runners in my automation project as follows:
Main runner - Executes all the #ui-test tagged test cases and if a scenario is failed target/rerun.txt will be populated with the scenario location (e.g. features/Dummy.feature:22):
#RunWith(Cucumber.class)
#CucumberOptions(
features = "classpath:features",
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber.json", "rerun:target/rerun.txt"},
tags = {"#ui-test", "~#ignore"}
)
public class RunCukesTest {
}
Secondary runner - Re-executes the scenarios from target/rerun.txt:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "#target/rerun.txt",
plugin = {"pretty", "html:target/cucumber-html-report-rerun", "json:target/cucumber_rerun.json"}
)
public class ReRunFailedCukesTest {
}
When the execution is performed two result json files are created:
cucumber.json
cucumber_rerun.json
Jenkins will collect the results via Cucumber-JVM Reports plugin and will create a combined report.
The problem is, even if all the target/rerun.txt tests are passed in the second run, the report status will remain failed because of the cucumber.json.
Is there a way (to set up Cucumber-JVM Reports plugin or modify the upper presented runners) to overwrite cucumber.json with the results from cucumber_rerun.json and to publish only the modified cucumber.json?
Another sub-keywords: maven, java, cucumber-java8, cucumber-junit, junit
I had problem similar to yours, though, I've used single runner, handled re-runs from testNG(re-runs was one of the reasons I've switched from JUnit to TestNG) directly and as a results I had increased amount of tests in my json report.
My solution was to clean json files afterwards, despite the fact that Jenkins knows about failed tests it won't mark build as failed or as unstable.
In your particular case you may try to somehow match tests from rerun.json and exclude them from regular json report.
For parsing jsons I may recommend using Jackson FasterXML
I use Jenkins cucumber reporting latest release with below config in Jenkins.
Image Of Config In Jenkins
1st Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="FolderFeature",
glue={"Gluefolder"},
plugin={"html:target/cucumberpf-html-report",
"json:target/cucumberpf.json"}
)
public class RunPF {
}
2nd Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="Blah/Test.feature",
glue={"mygluefolder"},
plugin={"html:target/cucumber-html-report",
"json:target/cucumber.json"}
)
public class RunRA {
}
I had failed in both .json files and when it passed both were merged and updated correctly in one cucumber report.
Here is the error:
[CucumberReport] Preparing Cucumber Reports
[CucumberReport] JSON report directory is "C:\Users\ajacobs\workspace\com.mytest.framework\target\"
[CucumberReport] Copied 2 json files from workspace "C:\Users\admin\workspace\yourtest\target" to
reports directory "C:\Users\admin\.jenkins\jobs\Regression\builds\21\cucumber-html-reports\.cache"
[CucumberReport] Processing 2 json files:
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumber.json
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumberpf.json
Finished: SUCCESS
I've setup teamcity with my sln file and got the unit tests to show up with the CppUnit plugin that teamcity has. And I get test results in the TeamCity UI.
Now I'm trying to get trending reports to show up for my unit tests and code coverage.
As of code coverage, we're using vsinstr.exe and vsperfmon.exe which produces an XML file.
I'm not quite sure as of what steps I should be taking to make the trending reports and code coverage(not as important) to show up.
I've already seen this post, but the answer seems to require editing the build script, which I don't think would work for my case since I'm building through MSBuild and the .sln file, and the tests are being ran through that build.
So basically I'm trying to get the Statistics tab to show up, and I'm not sure where to begin.
Just add simple Powershell step into your build configuration. Something like this:
function TeamCity-SetBuildStatistic([string]$key, [string]$value) {
Write-Output "##teamcity[buildStatisticValue key='$key' value='$value']"
}
$outputFile = 'MetricsResults.xml'
$xml = [xml] (Get-Content $outputFile)
$metrics = $xml.CodeMetricsReport.Targets.Target[0].Modules.Module.Metrics
$metrics.Metric
| foreach { TeamCity-SetBuildStatistic "$($_.Name)" $_.Value.Replace(',', '') }
It uses XML output from FxCop Metrics. You have to update the script for your actual schema.