Recently migrated from JUnit4 to JUnit5. As per the current gradle settings I no longer see HTML reports getting generated. Have googled but could find a way to generate these reports. Has anyone tried to generate HTML reports of test under JUnit5?
When I run the test under JUnit5, I see a binary folder with files
output.bin
output.bin.idx
results.bin
And hence resultant html report is not generated.
gradle.build code that helps club results from the test task in all subprojects
task testReport(type: TestReport) {
destinationDir = file("$buildDir/reports/allTests")
reportOn subprojects*.test
doLast {
println "Test reports for all test complete $env"
}
}
If there is need for more code to share, please let me know.
Related
I am building a simple CI pipeline for my python code in Jenkins using Jenkinsfile, which basically does the following things:
Creating test environment and installing dependencies.
Running static code metrics:
various raw metrics : SLOC, comment lines, blank lines, Cyclomatic Complexity, the Maintainability Index etc.
tests coverage reports using coverage
errors and style check using pylint
Testing pulled source code (Unit testing)
So lets say i have this stage for example, for "Static code metrics":
...
stage('Static code metrics') {
steps {
echo "Raw metrics"
sh ''' radon raw --json my_python_repo/ > raw_report.json
radon cc --json my_python_repo/ > cc_report.json
radon mi --json my_python_repo/ > mi_report.json
//TODO: add conversion and HTML publisher step
'''
}
}
...
As you see above, the reports are saved in .json format. I need to figure out a way to publish these reports in a pretty visual way on Jenkins dashboard.
One of the steps is to convert the .json files to HTML and then use HTML publisher plugin to publish reports but i don't know what tool to use.
If there is way to solve this or any other way to publish these reports on jenkins dashboard, please provide the solution.
Lets say content of mi_report.json is:
{"my_python_repo/code.py": {"mi": 16.42950884051172, "rank": "B"}, "my_python_repo/test.py": {"mi": 33.532817596089814, "rank": "A"}}
There is no Jenkins-ready tool to publish custom JSON as HTML.
But you can use warnings plugin to publish your metrics in the Jenkins way.
Here small instruction on how to configure Jenkins to parse radon reports.
I'm trying to figure out how to show Coverage in TFS. with .net CORE.
Below is my build definition. I have a "Test .NET Core" task.
https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore21
From above url, I found this hint:
Run the tests in the project in the current directory and generate a
test results file in the trx format:
dotnet test --logger:trx
And I have test results working.
(Details of Test-Task and the Publish-Test-Results below)
Publish Test Results (task).
With the following arguments
Test Result Format:
VSTest
Test Results Files
**/*.trx
So I can see Test Results :) (below image)
But how do you get Code Coverage working? (above image, red circle...there is a place holder for Coverage)
What I've tried:
There is a "Coverage" task..but doesn't seem right to me. (image below)
I have two runners in my automation project as follows:
Main runner - Executes all the #ui-test tagged test cases and if a scenario is failed target/rerun.txt will be populated with the scenario location (e.g. features/Dummy.feature:22):
#RunWith(Cucumber.class)
#CucumberOptions(
features = "classpath:features",
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber.json", "rerun:target/rerun.txt"},
tags = {"#ui-test", "~#ignore"}
)
public class RunCukesTest {
}
Secondary runner - Re-executes the scenarios from target/rerun.txt:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "#target/rerun.txt",
plugin = {"pretty", "html:target/cucumber-html-report-rerun", "json:target/cucumber_rerun.json"}
)
public class ReRunFailedCukesTest {
}
When the execution is performed two result json files are created:
cucumber.json
cucumber_rerun.json
Jenkins will collect the results via Cucumber-JVM Reports plugin and will create a combined report.
The problem is, even if all the target/rerun.txt tests are passed in the second run, the report status will remain failed because of the cucumber.json.
Is there a way (to set up Cucumber-JVM Reports plugin or modify the upper presented runners) to overwrite cucumber.json with the results from cucumber_rerun.json and to publish only the modified cucumber.json?
Another sub-keywords: maven, java, cucumber-java8, cucumber-junit, junit
I had problem similar to yours, though, I've used single runner, handled re-runs from testNG(re-runs was one of the reasons I've switched from JUnit to TestNG) directly and as a results I had increased amount of tests in my json report.
My solution was to clean json files afterwards, despite the fact that Jenkins knows about failed tests it won't mark build as failed or as unstable.
In your particular case you may try to somehow match tests from rerun.json and exclude them from regular json report.
For parsing jsons I may recommend using Jackson FasterXML
I use Jenkins cucumber reporting latest release with below config in Jenkins.
Image Of Config In Jenkins
1st Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="FolderFeature",
glue={"Gluefolder"},
plugin={"html:target/cucumberpf-html-report",
"json:target/cucumberpf.json"}
)
public class RunPF {
}
2nd Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="Blah/Test.feature",
glue={"mygluefolder"},
plugin={"html:target/cucumber-html-report",
"json:target/cucumber.json"}
)
public class RunRA {
}
I had failed in both .json files and when it passed both were merged and updated correctly in one cucumber report.
Here is the error:
[CucumberReport] Preparing Cucumber Reports
[CucumberReport] JSON report directory is "C:\Users\ajacobs\workspace\com.mytest.framework\target\"
[CucumberReport] Copied 2 json files from workspace "C:\Users\admin\workspace\yourtest\target" to
reports directory "C:\Users\admin\.jenkins\jobs\Regression\builds\21\cucumber-html-reports\.cache"
[CucumberReport] Processing 2 json files:
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumber.json
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumberpf.json
Finished: SUCCESS
I have a Jenkins job that is using the "Invoke standalone Sonar analysis" for a javascript project.
I thought it was working fine with the following parameters:
sonar.sources=src
sonar.language=js
sonar.dynamicAnalysis=reuseReports
sonar.javascript.jstestdriver.coveragefile=target/test-coverage/jscover.lcov
sonar.javascript.lcov.reportPath=target/test-coverage/jscover.lcov
But then I noticed that the numbers that are being reported in Sonar do not match the number in the lcov file.
When I log into to Sonar I see the code coverage number as 30%.
But when I examine the lcov file, I get completely different numbers:
$lcov --summary target/test-coverage/jscover.lcov
...
lines......: 48.1%
functions..: 41.7%
branches...: no data found
And in fact, when I view the jscover.html report file, I see the total coverage at 48%.
Sonar reports it at 30%.
And drilling down into the individual files, Sonar's results do not match the results in the lcov file either.
For instance:
Just by looking at a particular file, /src/js/models/Call.js, lcov says it’s at 97% code coverage.
But Sonar displays this:
49.0% by unit tests Line coverage:97.0% (97/100)Branch coverage:0.0% (0/98)
It’s as if Sonar is using the Branch Coverage AND the Line Coverage Stats to get the final code coverage results at 49.0%.
Do you know what I am doing wrong? Do you know why Sonar is not using the coverage results from the lcov file? Is it because the Branch Coverage has no data?
Thanks for any insight on this.
Code coverage is recomputed by SonarQube. SonarQube just retrieves from the report whether a line is covered or not by unit tests. Example:
DA:10,0 => it means that line 10 is not covered
DA:20,1 => it means that line 20 is covered
DA:30,5 => it means that line 30 is covered
Then SonarQube recomputes the code coverage:
Number of covered lines / (Number of covered lines + Number of uncovered lines)
I've setup teamcity with my sln file and got the unit tests to show up with the CppUnit plugin that teamcity has. And I get test results in the TeamCity UI.
Now I'm trying to get trending reports to show up for my unit tests and code coverage.
As of code coverage, we're using vsinstr.exe and vsperfmon.exe which produces an XML file.
I'm not quite sure as of what steps I should be taking to make the trending reports and code coverage(not as important) to show up.
I've already seen this post, but the answer seems to require editing the build script, which I don't think would work for my case since I'm building through MSBuild and the .sln file, and the tests are being ran through that build.
So basically I'm trying to get the Statistics tab to show up, and I'm not sure where to begin.
Just add simple Powershell step into your build configuration. Something like this:
function TeamCity-SetBuildStatistic([string]$key, [string]$value) {
Write-Output "##teamcity[buildStatisticValue key='$key' value='$value']"
}
$outputFile = 'MetricsResults.xml'
$xml = [xml] (Get-Content $outputFile)
$metrics = $xml.CodeMetricsReport.Targets.Target[0].Modules.Module.Metrics
$metrics.Metric
| foreach { TeamCity-SetBuildStatistic "$($_.Name)" $_.Value.Replace(',', '') }
It uses XML output from FxCop Metrics. You have to update the script for your actual schema.