I guess the title is pretty self-explanatory. The reason I want that is so that I can make a live custom HTML reporter for my tests.
My test suite takes hours to complete, and although the tests generate HTML reports as soon as each test step is executed, it's only at post-build time that those report files get published.
Being able to see them as they get generated would reduce the time it takes for me and my teammates to analyze and act upon issues revealed by our test runs.
All I need is that Jenkins let me access the build files as the build executes. Nothing fancy; I can take care of the rest. Is that possible? How?
In our setup there is always an intermediate file (typically XML) but the HTML files are created at the end of the job.
What you can do, is use the progressive output (http://jenkins/job/jobName/buildNumber/logText/progressiveText?start=0). Although you don't state which framework you use, most of them output something that would be easy to parse. e.g. "Test xxx failed".
Related
I am new around here, I have done a lot of googling, searching on this site and asking around and have not found a satisfactory answer.
I develop automated tests, UI as well as API. These are then run by TFS and the results are put into a JUnit xml document, which is then read by TFS. But alas the formatting is atrocious and leaves one unable to use the output for anything.
Vis:
There is no information about the Test Suite (which is there in the XML), the actual request sent (which is in the log) or response received and so one is left with absolutely no context to understand what has actually taken place (which request was sent, what test group/suite it belongs to and what any potential error was).
As far as I have been able to uncover, TFS simply has little to no support of proper test result formatting when it comes to automated testing. I am very surprised by this in 2018. Not even any documentation that allows me to develop my report structure/format in some kind of script.
What alternatives do I have? Can I automatically attach a generated HTML report somehow in TFS? Can I output more info anywhere?
You can group by Test Suite, Owner, Priority etc...
And you can double click the specific test result and navigate to the test run Summary to see more information there, also you can attach files there.
More information please see Review continuous test results after a build
What's Working - We are currently using testng emailable format and allure to generate formatted reports for our current test execution. These are working fine with our local. The /target/report structure can be seen in the image to depict 2 different folders for allure(/site) and testng(/surefire) reports respectively :
Trying to - While we are trying to implement a CI using Jenkins using the same steps as in our local, the tests are executed fine and the respective reports are getting generated as well.
Using TestNG plugin
and specifying the pattern **/target/surefire-reports/testng-results.xml works just fine to display the testNG result graph.
Also using the Email ext plugin I can attach the .html reports to the mail sent to the recipients specifying the attachment field details as :
**/target/surefire-reports/emailable-report.html, **/target/surefire-reports/index.html
What doesn't work - We end up getting emails with HTML reports but these are not formatted, probably because all the CSS linked to these are left behind. Is there a way to overcome this?
Note - Have tried these:
Attaching all the .css files along with .html file in attachments but, one that's brute force and second it still doesn't work.
One way also is to scp the report (/target) directory to another host from Jenkins instance and share the path of the reports on that machine over the email notification and get the formatted reports shared. But then this needs an additional resource and dependency on it, that we would want to avoid.
While posting this, I see an HTML publisher plugin that seems to be doing something similar. Tried installing the same and use it. But I am assuming since we are using Jenkins 2.6 and the plugin note reads
Starting in versions 1.625.3 and 1.641, Jenkins restricted what kind
of content could be displayed when serving static files. This can
impact how HTML files archived using this plugin are displayed. See
Configuring Content Security Policy for more information.
We are not getting the option to Publish HTML Reports in the post build actions.
Any suggestions are more than welcome and please do ask for any more information required over this.
Edit : Adding to the Note 2 above, the Jenkins instances used in our setup are docker slaves, apparently making the reports or targets generated not being persistent.
Here's what you can consider doing.
Option 1
Build an implementation for IExecutionListener wherein you create the logic to basically zip up all the reports that you want and have it sent as an e-mail.
Wire in this listener and that should take care.
PS : Currently implementations of IExecutionListener are invoked "before" the report generation phase. I have changed this as part of this commit. So if you would like to go ahead with this approach then you might want to wait till TestNG goes out with a new release (should happen in a couple of days )
Option 2
Build a wrapper reporter (which implements IReporter) and wire in ONLY this reporter.
Within this reporter, you explicitly instantiate all the reporters that you would like to be called in for your reporting phase. See below for a possible sample.
public class ChainedReporter implements IReporter {
private List<IReporter> reporters = new ArrayList<>;
public ChainedReporter() {
reporters.add(new FooReporter() );//Here FooReporter is a custom reporter. Replace it with yours.
reporters.add(new BarReporter() );//Here BarReporter is a custom reporter. Replace it with yours.
}
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
for (IReporter reporter : reporters) {
reporter.generateReport(xmlSuites, suites, outputDirectory);
}
//By now we have ensured that all the reporting logic has been triggered and we have reports generated.
zipReports(); // This method would take care of creating zipped files of all the reports.
emailReports(); // This emthod would take care of emailing the actual reports.
}
}
My Jenkins job runs many tests that create log files. In case of failure, I want to look at the log of the failed test. I'd rather use Jenkins web-server to do it, even have a link in the email it sends me.
Is there any plugin that can do it? Or maybe another way?
You provide few details in your question, so it is impossible to give specific advice. In a general level: this is already possible. When your test framework creates JUnit XML files with test results, the test output can be included between the <failure> and </failure> tags. Usually test frameworks should take care of this automatically, so you are probably not using a test framework and are manually generating the XML files containing test results?
I recommend you adopt some test framework. It is usually well worth the effort.
My company is developing a web application that builds in ant. I've been tasked with getting CruiseControl.net to differentiate between a build failure and a unit test failure, which it can't do natively. ( It currently lumps both together but doesn't help developers understand what's broken )
I have CC.net call a script that returns specific exit codes depending on the nature of an ant task failure. I'd like these exit codes to be reflected in the CC.net failure report / dashboard but am having some trouble finding resources on how this might be done.
Any suggestions?
Not directly. All the reports and display works from information in the logs which are XML files. The display and reports work by applying XSLT to these XML files.
Take a look at your build logs and unit test logs, to see if each of those process write the failure information to their respective log files.
If they do, you should be able to write a custom XSLT or modify the existing XSLT to display that information.
Edit:
A different approach based on your comment. You could probably redirect the ANT error code to a file. Then you could have a seperate ccnet task that takes the error code from that file and re-format and display it (depending on how/where you want it displayed)
Currently the msbuild logs for team build are appalling as they are just plain text and are very difficult to read. Also the ones created by my build are approx 30Mb and take quite a while to download (our TFS server is in our datacentre).
Does anyone know any way of being able to view these logs easier, prefereably integrated with either TFS itself or TFS WebAccess?
Take a look at the following blog post I did a while ago:
http://www.woodwardweb.com/teamprise/000415.html
This describes how to create a simple ASP.NET page that will stream the contents of your log file to you over HTTP. The advantage of doing it this way is that you don't have to wait for the entire page to load before the log starts to render for you in Visual Studio.
Also - you can add some simple formatting to the file while streaming. In the example on my blog I simply make the start of each target appear in bold to make them stand out a bit more, but you can see how you could go crazy with this approach if you wanted.
If increasing bandwidth isn't an option then I would suggest you to write your own html logger and attach it to the build process. Splitting the html build log into minor parts (definded by targets and/or projects) and having one index file pointing to all the minor parts with appropiate information whether a given part failed or succeeded. Then you only need to parse the index file and any requested part over the link.
A third possibility is to compress the log-file after the build completes.