What's Working - We are currently using testng emailable format and allure to generate formatted reports for our current test execution. These are working fine with our local. The /target/report structure can be seen in the image to depict 2 different folders for allure(/site) and testng(/surefire) reports respectively :
Trying to - While we are trying to implement a CI using Jenkins using the same steps as in our local, the tests are executed fine and the respective reports are getting generated as well.
Using TestNG plugin
and specifying the pattern **/target/surefire-reports/testng-results.xml works just fine to display the testNG result graph.
Also using the Email ext plugin I can attach the .html reports to the mail sent to the recipients specifying the attachment field details as :
**/target/surefire-reports/emailable-report.html, **/target/surefire-reports/index.html
What doesn't work - We end up getting emails with HTML reports but these are not formatted, probably because all the CSS linked to these are left behind. Is there a way to overcome this?
Note - Have tried these:
Attaching all the .css files along with .html file in attachments but, one that's brute force and second it still doesn't work.
One way also is to scp the report (/target) directory to another host from Jenkins instance and share the path of the reports on that machine over the email notification and get the formatted reports shared. But then this needs an additional resource and dependency on it, that we would want to avoid.
While posting this, I see an HTML publisher plugin that seems to be doing something similar. Tried installing the same and use it. But I am assuming since we are using Jenkins 2.6 and the plugin note reads
Starting in versions 1.625.3 and 1.641, Jenkins restricted what kind
of content could be displayed when serving static files. This can
impact how HTML files archived using this plugin are displayed. See
Configuring Content Security Policy for more information.
We are not getting the option to Publish HTML Reports in the post build actions.
Any suggestions are more than welcome and please do ask for any more information required over this.
Edit : Adding to the Note 2 above, the Jenkins instances used in our setup are docker slaves, apparently making the reports or targets generated not being persistent.
Here's what you can consider doing.
Option 1
Build an implementation for IExecutionListener wherein you create the logic to basically zip up all the reports that you want and have it sent as an e-mail.
Wire in this listener and that should take care.
PS : Currently implementations of IExecutionListener are invoked "before" the report generation phase. I have changed this as part of this commit. So if you would like to go ahead with this approach then you might want to wait till TestNG goes out with a new release (should happen in a couple of days )
Option 2
Build a wrapper reporter (which implements IReporter) and wire in ONLY this reporter.
Within this reporter, you explicitly instantiate all the reporters that you would like to be called in for your reporting phase. See below for a possible sample.
public class ChainedReporter implements IReporter {
private List<IReporter> reporters = new ArrayList<>;
public ChainedReporter() {
reporters.add(new FooReporter() );//Here FooReporter is a custom reporter. Replace it with yours.
reporters.add(new BarReporter() );//Here BarReporter is a custom reporter. Replace it with yours.
}
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
for (IReporter reporter : reporters) {
reporter.generateReport(xmlSuites, suites, outputDirectory);
}
//By now we have ensured that all the reporting logic has been triggered and we have reports generated.
zipReports(); // This method would take care of creating zipped files of all the reports.
emailReports(); // This emthod would take care of emailing the actual reports.
}
}
Related
Previously, I used testlink. This plugin generated a nice table with the test results in the build summary view.
Now, I don't use testlink anymore, so this plugin cannot be used. But instead, I want to feed a html report (which I take care of generating) to the build summary view.
I tried HTML Publisher Plugin, but it makes a link instead of displaying in the build summary view.
Is there such a plugin that allows me to specify that a certain .html file from the workspace will be included?
Bonus question: or even a plugin that allows a .html fragment to avoid the use of i-frames?
Try using the Summary Display Plugin. It work pretty good.
The configuration is a bit tricky so I'm adding an example screenshot of my configuration.
I am using jenkins to run jobs that uses maven with cucumber . my jobs are running succesfully. I have also managed the plugin Cucumber report in jenkins. But when i view the report i am getting the message as shown in below picture.
I have tried to install various plugins like Jenkins plugin, standalone but with no joy. These are shown as options in the bottom right of the report.
Any clue is helpful. I have tried to check this error in stackover flow also. I didnt get any clue
I could see the reports file in xml which is created by surefire plugin. Any way i can achive reporting part in my project
I managed to get reports in jenkins now.
Previously Jenkins is unable to find the json file in my project.
I have set the path of json file as target and i could able to see the reports.
This might help for those who are looking for answer.
#CucumberOptions(
features= "C:\\Users\\dd pc\\workspace\\PracticeCucumber\\src\\main\\java\\com\\qa\\feature\\Test.feature",
glue= {"com\\qa\\stepDefinition"},
format= {"pretty","html:test-output_1",
"json:target/cucumber-reports/CucumberTestReport.json"},
plugin = ("json:target/cucumber-reports/CucumberTestReport.json"),
monochrome= true,
dryRun= false
)
I am able to solve by providing **/*.json
under Post-build Actions/Cucumber Report/JSON Report Location/ File Include Pattern = **/*.json
First of all in runner file we need to provide following plugin as follows: plugin= {"json:target/cucumber.json" }
Now in Jenkins job, in "post build" section during configuration, add cucumber reports, click on advanced option and provide 'target' text in "JSON Reports Path" text field. Save the job, it will run and will work fine.
Note: target text is the name of folder which we have provided in runner file for generating reports in Json, also cucumber report plugin works on json so need to generate report in json.
I my case the problem was in two jobs over the same test repository (one for Chrome one for Firefox), where one deletes report files after the job while second job was already running and vice versa.
I guess the title is pretty self-explanatory. The reason I want that is so that I can make a live custom HTML reporter for my tests.
My test suite takes hours to complete, and although the tests generate HTML reports as soon as each test step is executed, it's only at post-build time that those report files get published.
Being able to see them as they get generated would reduce the time it takes for me and my teammates to analyze and act upon issues revealed by our test runs.
All I need is that Jenkins let me access the build files as the build executes. Nothing fancy; I can take care of the rest. Is that possible? How?
In our setup there is always an intermediate file (typically XML) but the HTML files are created at the end of the job.
What you can do, is use the progressive output (http://jenkins/job/jobName/buildNumber/logText/progressiveText?start=0). Although you don't state which framework you use, most of them output something that would be easy to parse. e.g. "Test xxx failed".
My Jenkins job runs many tests that create log files. In case of failure, I want to look at the log of the failed test. I'd rather use Jenkins web-server to do it, even have a link in the email it sends me.
Is there any plugin that can do it? Or maybe another way?
You provide few details in your question, so it is impossible to give specific advice. In a general level: this is already possible. When your test framework creates JUnit XML files with test results, the test output can be included between the <failure> and </failure> tags. Usually test frameworks should take care of this automatically, so you are probably not using a test framework and are manually generating the XML files containing test results?
I recommend you adopt some test framework. It is usually well worth the effort.
My company is developing a web application that builds in ant. I've been tasked with getting CruiseControl.net to differentiate between a build failure and a unit test failure, which it can't do natively. ( It currently lumps both together but doesn't help developers understand what's broken )
I have CC.net call a script that returns specific exit codes depending on the nature of an ant task failure. I'd like these exit codes to be reflected in the CC.net failure report / dashboard but am having some trouble finding resources on how this might be done.
Any suggestions?
Not directly. All the reports and display works from information in the logs which are XML files. The display and reports work by applying XSLT to these XML files.
Take a look at your build logs and unit test logs, to see if each of those process write the failure information to their respective log files.
If they do, you should be able to write a custom XSLT or modify the existing XSLT to display that information.
Edit:
A different approach based on your comment. You could probably redirect the ANT error code to a file. Then you could have a seperate ccnet task that takes the error code from that file and re-format and display it (depending on how/where you want it displayed)