I'm setting up automated regression testing for an FPGA project, almost exactly as described here:
Continuous integration of complex reconfigurable systems
Now I want to get test results (from VHDL REPORT statements in ModelSim simulation) to appear in Jenkins testing reports. My understanding is that Jenkins only natively supports jUnit format, and I looked for plugins supporting non-XML formats but didn't see any.
Generating valid XML from VHDL REPORT statements would be very difficult, since the simulation may immediately terminate depending on the severity. Which means that the closing tags would have to be duplicated in every single possible exit path for every single test -- not the most maintainable approach.
So, do you know of any straightforward way to convert plain text into jUnit (or another format, if supported by Jenkins)? If something doesn't already exist, is there an advantage to writing a Jenkins plugin vs just throwing together a perl script? Any other suggestions?
You should take a look at the XUnit Plugin. The Plugin reads test results from a number of tools, and seems adaptable to custom formats. From the documentation the plugin is able to read not only xml, but also csv and txt. For custom format you need to specify some style sheet for the transformation, I am not quite sure if this will go all the way for you. But even if it does not, I suppose the plugin should be easy to extend for your own format.
Old post but today there is a unit testing framework for VHDL that we've developed. It solves the problem by generating a report on the JUnit format. It also handles the case when the simulation stops due to a severe error. The tool is free and open source and can be found at https://github.com/LarsAsplund/vunit
Related
I am new in Karate but know almost all the basics. I want to generate a report for my test cases which will be kept in the folder for history purposes. currently, I am using the HTML report which overrides or updates every time I run my test cases.
If you can recommend me a step-by-step tutorial or a link, or simply give me a hint on how to approach this.
I have tried to use the allure framework and I am struggling to set it up.
Thought it would be the best approach to solve my problem.
Attached is my HTML report I am talking about. Thanks in advance
The JUnit HTML report which you have referred to as an image is designed for easy development, so when working on a test, you re-run and re-fresh the browser.
What we recommend is to use the parallel runner reports. Most of the time, 3rd party frameworks will solve this problem for you - especially if you use them integrated with CI. See this for example.
But finally, Karate will output the industry-standard Cucumber JSON report format, so you can always write a reporting solution yourself. Note that others have had success integrating Allure - so please refer this answer: https://stackoverflow.com/a/54527955/143475
I want to evaluate the performance of Rascal for a given rewrite system that I've written. I'm wondering if there's a good way of doing it?
Ideally, I'd generate some compiled Java classes from the system and then run them manually against my inputs. Is there an easy or recommended way to do it?
Cheers,
One way to do this is to use the functions in the library util::Benchmark. Typically, you could write something like
cpuType( (){ call_the_function_I_want_to_observe(); } ). This will execute your function and print the cpu time used.
Note that Rascal can be executed in two ways: interpreted and compiled which makes a big difference when measuring performance. We are working hard at the moment to fully integrate the compiler in the Eclipse IDE, but a stand alone version is available as well. This can be called as java -Xss8m -jar rascal-0.8.4-SNAPSHOT.jar --compiledREPL followed by at least values for directories for sources (--src), and binaries (--bin). Here rascal-0.8.4-SNAPSHOT.jar (but most likely named differently) is downloaded from the https://update.rascal-mpl.org/console/rascal-shell-unstable.jar.
If you need more information, don't hesitate to ask for more details: this part of our tool chain is unfortunately still undocumented.
I want to enforce coding standards for our Delphi codebase.
A few colleagues have suggested Code Healer and Pascal Analyzer. I've had a look at these tools and they aren't suitable.
I was hoping to be able to do the same thing that CheckStyle for Java or StyleCop for C# can do
Some newer editions of Delphi offer Audits and Metrics in the Model view, which can also be configured to set allowed limits. They do not run from command line for build integration afaik so I found them not very helpful.
I know the highly customizable Java (and .Net tools) like PMD, FindBugs and CheckStyle which generate XML or HTML statistic reports, and also integrate very well with build tools (Ant, Maven, Hudosn) - but for Delphi nothing comparable has crossed my road so far.
It seems those 2 are the most used. You can also try:
http://jedicodeformat.sourceforge.net/
The best one is Pascal Analyzer (PAL) by Peganza, which you said you tried and found unsuitable, but did not say why. I will say a bit in its favor: It's Commercial, inexpensive, and so worth it. They recently released version 5, and if version 5 doesn't do what you want, you should tell them what you want, because they have always answered my requests whenever I have mentioned a feature I wish the product would add.
We use it instead of the high-end SKUs of Delphi's metrics because it costs less and does more than the built-in $3000 stuff. I think it costs about $160 us.
I am a happy customer. Here is a sample of some of the metric areas that I like:
convention compliance - class names that don't start with T, exception types don't start with E, class fields not in private, identifiers with goofy names, class visibility confusion or bad order, local identifier/unit outer scope identifier clashes. Inconsistent case, Many many many more!
The weakness is that the output is plain text in a "TMemo" control. Of course, I have found a lot of ways to take that output and write my own small sort/filter utilities to mine even more useful stuff from the reports. A powerful tool that you won't be able to live without once you try it.
I realize you said in your answer that you tried that already, but if it's not what you want, it's already the best LINT like tool for Pascal that currently exists.
If you're into writing your own style checking, you can write a .exe in Delphi to look for bad things being committed. Call that in a pre-commit hook into your repository.
You can examine the differences of a checkin by using SVNLOOK.
ex:
excerpt from pre-commit.bat
SVNLOOK diff -t "%2" "%1" | MyCustomFilter.exe
IF %ERRORLEVEL% == 0 GOTO EOJ
EXIT 1
:EOJ
EXIT 0
The team I work for manages a large collection of technical documentation which is written in LaTeX.
Currently all the documentation we have is manually built by the editors and then checked into a version control system. Sometimes people forget to compile their documents so we have a situation where the PDF and .tex files are often out of step. Unfortunately when this happens our users find themselves reading old versions of our document.
I've managed to hack a simple script to build PDFs using Make - it's rather clumsy.
I was wondering if there was a better way to do it? Most people in our department use Eclipse + Pydev for a Python project which means we are all very familiar with this IDE. I know that Ant plays nicely with Eclipse, so might we be able to use this tool for our doc building?
So what's the best way of doing this? I hope I will not have to learn everything there is to know about a new build-system in order to automate the building of some quite simple docs.
There is an external Ant task for LaTeX PDF generation, though the site is in German.
To use it, download the jar to a location on your machine, then define a taskdef as follows:
<taskdef name="latex" classname="de.dokutransdata.antlatex.LaTeX"
classpath="/path/to/ant/lib/ant_latex.jar"/>
Then to use it, define a target like this:
<target name="doLaTeX">
<latex
latexfile="${ltx2.file}"
verbose="on"
clean="on"
pdftex="off"
workingDir="${basedir}"
/>
</target>
Where ltx2.file is the file to process.
This is a link to the howto page listing the parameters. If you need any more options, my German is just about passable enough to explain, maybe.
There is also a maven plugin for LaTeX, but I can't find any documentation.
Haven't tried it, but I remember seeing a blog post about it.
If you know python, this blog post might be interesting
EDIT: Also, I would assume that you're using some kind of version control system, and I can't say for sure, but I use git to manage all my latex docs, and it might be possible to use some kind of post-commit hook to execute a script to rebuild the document. This would depend on how your repository is structured... just thinking out loud, so to speak.
I went into great detail on a large number of build systems for latex in this question, but its slightly different in your case. I think you want rubber or latexmk. The latex-makefile seems a good idea, but only supports building via postscript, which might not be your build process.
In general, its a good idea to keep generated files outside of version control for just this reason. A good exception is when specialist build tools are not widely available, and your situation sounds similar. You might do better with a commit-hook to build automatically upon commit.
I guess I should also point out that committing something without first building it and checking it is a deadly sin, so a better solution might be to stamp that out.
Maven is a better alternative as build system compared to Ant. So I would recommend a maven-plugin to generate PDF from LaTeX sources. Have a look at mathan-latex-maven-plugin
Is there any way to measure code coverage with DUnit? Or are there any free tools accomplishing that? What do you use for that? What code coverage do you usually go for?
Jim McKeeth: Thanks for the detailed answer. I am talking about unit testing in the sense of a TDD approach, not only about unit tests after a failure occured. I'm interested in the code coverage I can achieve with some basic prewritten unit tests.
I have just created a new open source project on Google Code with a basic code coverage tool for Delphi 2010. https://sourceforge.net/projects/delphicodecoverage/
Right now it can measure line coverage but I'm planning to add class and method coverage too.
It generates html reports with a summary as well as marked up source showing you what lines are covered (green), which were not (red) and the rest of the lines that didn't have any code generated for them.
Update:
As of version 0.3 of Delphi Code Coverage you can generate XML reports compatible with the Hudson EMMA plugin to display code coverage trends within Hudson.
Update:
Version 0.5 brings bug fixes, increased configurability and cleaned up reports
Update:
Version 1.0 brings support for emma output, coverage of classes and methods and coverage of DLLs and BPLs
I don't know of any free tools. AQTime is almost the defacto standard for profiling Delphi. I haven't used it, but a quick search found Discover for Delphi, which is now open source, but just does code coverage.
Either of these tools should give you an idea of how much code coverage your unit tests are getting.
Are you referring to code coverage from unit tests or stale code? Generally I think only testable code that has a failure should be covered with a unit test (yes I realize that may be starting a holy war, but that is where I stand). So that would be a pretty low percentage.
Now stale code on the other hand is a different story. Stale code is code that doesn't get used. You most likely don't need a tool to tell you this for a lot of your code, just look for the little Blue Dots after you compile in Delphi. Anything without a blue dot is stale. Generally if code is not being used then it should be removed. So that would be 100% code coverage.
There are other scenarios for stale code, like if you have special code to handle if the date ever lands on the 31st of February. The compiler doesn't know it can't happen, so it compiles it in and gives it a blue dot. Now you can write a unit test for that, and test it and it might work, but then you just wasted your time a second time (first for writing the code, second for testing it).
There are tools to track what code paths get used when the program runs, but that is only simi-reliable since not all code paths will get used every time. Like that special code you have to handle leap year, it will only run every four years. So if you take it out then your program will be broken every four years.
I guess I didn't really answer your question about DUnit and Code Coverage, but I think I may have left you with more questions then you started with. What kind of code coverage are you looking for?
UPDATE: If you are taking a TDD approach then no code is written until you write a test for it, so by nature you have 100 test coverage. Of course just because each method is exercised by a test does not mean that its entire range of behaviors is exercised. SmartInspect provides a really easy method to measure which methods are called along with timing, etc. It is a little less then AQTime, but not free. With some more work on your part you can add instrumentation to measure every code path (branches of "if" statements, etc.) Of course you can also just add your own logging to your methods to achieve a coverage report, and that is free (well, expect for your time, which is probably worth more then the tools). If you use JEDI Debug then you can get a call stack too.
TDD really cannot easily be applied retroactively to existing code without a lot of refactoring. Although the newer Delphi IDEs have the ability to generate unit test stubs for each public method, which then gives you 100% coverage of your public methods. What you put in those stubs determines how effective that coverage is.
I use Discover for Delphi and it does the work, for unit testing with DUnit and Functional testing with TestComplete.
Discover can be configured to run from the command line for automation.
As in:
Discover.exe Project.dpr -s -c -m
Discover works great for me. It hardly slows down your application, unlike AQTime. This may not be a problem for you anyway, of course. I think the recent versions of AQTime perform better in this respect.
I've been using Discover" for years, worked excellently up to and including BDS2006 (which was the last pre-XE* version of Delphi i used and still use), but its current opensourced state, it's unclear how to make it work with XE* versions of Delphi. A shame really, because I loved this tool, fast and convenient in almost every way.
So now I'm moving to delphi-code-coverage...