Tool to create automatically charts from CSV files (in ANT) - ant
I am currently testing a software that writes a couple of CSV files. Now I am looking for a free tool to generate automatically charts (images) from that files. I am working with eclipse and I am working with ANT to build my project. So a tool (perhaps in Java) to use it with ANT would be great.
Currently, I have to do many steps by hand:
go to the folder of the files
open and edit the files (replace . by , in numbers)
open the file in open office
create a diagram
export it
Has anybody an idea how to automate this process (even some steps of it)?
Option 1
A very useful program on unix for generating graphs is gnuplot.
I also found a csv2gnuplot script which you could use in a shell script to batch process files.
Option 2
If you're married to an ANT/Java solution (perhaps you're on windows), then I adapted the following example using jfreechart
src/sample.csv
Max Pause Goal,Minor Collections,Major Collections,Pause Count,Max Pause,GC Time,Total Time, Throughput
0,49,0,49,0.005,0.081,1.831,95.599
100,49,0,49,0.005,0.077,1.828,95.785
200,49,0,49,0.005,0.081,1.829,95.550
300,47,0,47,0.009,0.089,1.837,95.145
400,48,0,48,0.005,0.081,1.835,95.598
500,48,0,48,0.005,0.078,1.825,95.729
600,49,0,49,0.005,0.081,1.830,95.600
700,48,0,48,0.005,0.081,1.828,95.564
800,44,0,44,0.017,0.094,1.857,94.919
900,49,0,49,0.006,0.082,1.833,95.533
1000,49,0,49,0.005,0.088,1.840,95.224
build.xml
<project name="demo" default="chart" xmlns:ivy="antlib:org.apache.ivy.ant">
<property name="build.dir" location="build"/>
<property name="input.file" location="src/sample.csv"/>
<property name="output.file" location="${build.dir}/sample.png"/>
<target name="init">
<ivy:resolve/>
<ivy:cachepath pathid="build.path" conf="build"/>
<mkdir dir="${build.dir}"/>
</target>
<target name="chart" depends="init">
<taskdef name="groovy" classname="org.codehaus.groovy.ant.Groovy" classpathref="build.path"/>
<groovy>
import java.io.*;
import java.util.StringTokenizer;
import org.jfree.chart.*;
import org.jfree.chart.plot.*;
import org.jfree.data.xy.*;
boolean SHOW_LEGEND = false;
boolean SHOW_TOOLTIPS = false;
boolean GENERATE_URLS = false;
FileReader fr = new FileReader(properties["input.file"]);
BufferedReader br = new BufferedReader(fr);
// Get the x-axis label from the first token in the first line
// and the y-axis label from the last token in the first line.
String line = br.readLine();
StringTokenizer st = new StringTokenizer(line, ",");
String xLabel = st.nextToken();
String yLabel = st.nextToken();
while (st.hasMoreTokens()) yLabel = st.nextToken();
String title = yLabel + " by " + xLabel;
// Get the data to plot from the remaining lines.
float minY = Float.MAX_VALUE;
float maxY = -Float.MAX_VALUE;
XYSeries series = new XYSeries("?");
while (true) {
line = br.readLine();
if (line == null) break;
st = new StringTokenizer(line, ",");
// The first token is the x value.
String xValue = st.nextToken();
// The last token is the y value.
String yValue = "";
while (st.hasMoreTokens()) yValue = st.nextToken();
float x = Float.parseFloat(xValue);
float y = Float.parseFloat(yValue);
series.add(x, y);
minY = Math.min(y, minY);
maxY = Math.max(y, maxY);
}
XYSeriesCollection dataset = new XYSeriesCollection();
dataset.addSeries(series);
JFreeChart chart = ChartFactory.createXYLineChart(
title, xLabel, yLabel, dataset,
PlotOrientation.VERTICAL,
SHOW_LEGEND, SHOW_TOOLTIPS, GENERATE_URLS);
XYPlot plot = chart.getXYPlot();
plot.getRangeAxis().setRange(minY, maxY);
int width = 500;
int height = 300;
ChartUtilities.saveChartAsPNG(new File(properties["output.file"]), chart, width, height);
</groovy>
</target>
<target name="clean">
<delete dir="${build.dir}"/>
</target>
</project>
ivy.xml
The ivy plug-in enables ANT to talk to Maven repositories.
<ivy-module version="2.0">
<info organisation="org.myspotontheweb" module="demo"/>
<configurations defaultconfmapping="build->default">
<conf name="build" description="ANT tasks"/>
</configurations>
<dependencies>
<dependency org="org.codehaus.groovy" name="groovy-all" rev="1.8.2"/>
<dependency org="jfree" name="jfreechart" rev="1.0.13"/>
<dependency org="jfree" name="jcommon" rev="1.0.15" force="true"/>
</dependencies>
</ivy-module>
Note:
The latest version of jfreechart has a broken dependency on version 1.0.16 of jcommon (not present in Maven Central)
You could use XChart for this, as creating Charts from CSV files is a new feature since version 2.2.0. The code would look something like this:
// import chart from a folder containing CSV files
Chart chart = CSVImporter.getChartFromCSVDir("./path/to/dir/with/csv/files/", DataOrientation.Columns, 600, 600);
// Show it
new SwingWrapper(chart).displayChart();
// Save it
BitmapEncoder.savePNG(chart, "./Sample_Chart.png");
BitmapEncoder.savePNGWithDPI(chart, "./Sample_Chart_300_DPI.png", 300);
The data in the CSV files can either be horizontal or vertical, and each CSV file should contain the data for a single series.
You can also do the reverse as well and export the Chart data into CSV files:
CSVExporter.writeCSVColumns(chart.getSeriesMap().get(0), "./path/to/dir//");
You could use the Open Office API, there are also some samples for Java in the wiki
Related
Ant: Rename files to include their MD5
The question is likely VERY trivial for anyone familiar with ant, of which I only use the basics thus far. I know how to rename files, e.g. I already use: <copy todir="build/css/"> <fileset dir="css/"> <include name="*.css"/> </fileset> <globmapper from="*.css" to="*-min.css"/> </copy> I know how to calculate an MD5: <checksum file="foo.bar" property="foobarMD5"/> I don't know how to include the second into the first, to rename all those files to include their MD5 - the purpose is to serve as webbrowser cache buster. The other cache-busting option, to append "?[something]" is not as good, as is explained on some Google webmaster pages, having the MD5 as part of the name is better.
I managed to produce a somewhat strange solution using for from ant contrib. But you have to install ant contrib first. The copy in the sequential does not seem to accept/evaluate mappers (it wouldn't work, I tried with ant 1.7.0), so I had to create an extra move with a filtermapper to create the results. It does the following: for each file create an md5sum and save it in property foobarMD5 the property has to be unset before each iteration I create a new file in the same dir named example.java_foobarMD5.java (Notice that the filename contains the fileextension) I move all files with .java_ in its name to a new Folder and remove the .java_ I leave this example with .java. <for param="file"> <path> <fileset dir="src/" includes="**/*.java"/> </path> <sequential> <echo>Letter #{file}</echo> <var name="foobarMD5" unset="true"/> <checksum file="#{file}" property="foobarMD5"/> <echo>${foobarMD5}</echo> <copy file="#{file}" tofile="#{file}_${foobarMD5}.java"/> </sequential> </for> <move todir="teststack" verbose="true"> <fileset dir="src/"> <include name="**/*java_*"/> </fileset> <filtermapper> <replacestring from=".java_" to="-"/> </filtermapper> </move>
You could do this without having to include ant contrib. I had to implement this for work and was not allowed to introduce that extension for security reasons. The solution I came to was this: <target name="appendMD5"> <copy todir="teststack"> <fileset dir="css/" includes="**/*.css"/> <scriptmapper language="javascript"><![CDATA[ var File = Java.type('java.io.File'); var Files = Java.type('java.nio.file.Files'); var MessageDigest = Java.type('java.security.MessageDigest'); var DatatypeConverter = Java.type('javax.xml.bind.DatatypeConverter'); var buildDir = MyProject.getProperty('builddir'); var md5Digest = MessageDigest.getInstance('MD5'); var file = new File(buildDir, source); var fileContents = FIles.readAllBytes(file.toPath()); var hash = DatatypeConverter.printHexBinary(md5Digest.digest(fileContents)); var baseName = source.substring(0, source.lastIndexOf('.')); var extension = source.substring(source.lastIndexOf('.')); self.addMappedName(baseName + '-' + hash + extension); ]]></scriptmapper> </copy> </target> It is worth noting that I wrote this for Java 8 but with some minor tweaks it could work on Java 7. Sadly this won't work for earlier versions of Java without more effort.
Parse HTML using with an Ant Script
I need to retrieve some values from an HTML file. I need to use Ant so I can use these values in other parts of my script. Can this even be achieved in Ant?
As stated in the other answers you can't do this in "pure" XML. You need to embed a programming language. My personal favourite is Groovy, it's integration with ANT is excellent. Here's a sample which retrieves the logo URL, from the groovy homepage: parse: print: [echo] [echo] Logo URL: http://groovy.codehaus.org/images/groovy-logo-medium.png [echo] build.xml Build uses the ivy plug-in to retrieve all 3rd party dependencies. <project name="demo" default="print" xmlns:ivy="antlib:org.apache.ivy.ant"> <target name="resolve"> <ivy:resolve/> <ivy:cachepath pathid="build.path" conf="build"/> </target> <target name="parse" depends="resolve"> <taskdef name="groovy" classname="org.codehaus.groovy.ant.Groovy" classpathref="build.path"/> <groovy> import org.htmlcleaner.* def address = 'http://groovy.codehaus.org/' // Clean any messy HTML def cleaner = new HtmlCleaner() def node = cleaner.clean(address.toURL()) // Convert from HTML to XML def props = cleaner.getProperties() def serializer = new SimpleXmlSerializer(props) def xml = serializer.getXmlAsString(node) // Parse the XML into a document we can work with def page = new XmlSlurper(false,false).parseText(xml) // Retrieve the logo URL properties["logo"] = page.body.div[0].div[1].div[0].div[0].div[0].img.#src </groovy> </target> <target name="print" depends="parse"> <echo> Logo URL: ${logo} </echo> </target> </project> The parsing logic is pure groovy programming. I love the way you can easily walk the page's DOM tree: // Retrieve the logo URL properties["logo"] = page.body.div[0].div[1].div[0].div[0].div[0].img.#src ivy.xml Ivy is similar to Maven. It manages your dependencies on 3rd party software. Here it's being used to pull down groovy and the HTMLCleaner library the groovy logic is using: <ivy-module version="2.0"> <info organisation="org.myspotontheweb" module="demo"/> <configurations defaultconfmapping="build->default"> <conf name="build" description="ANT tasks"/> </configurations> <dependencies> <dependency org="org.codehaus.groovy" name="groovy-all" rev="1.8.2"/> <dependency org="net.sourceforge.htmlcleaner" name="htmlcleaner" rev="2.2"/> </dependencies> </ivy-module> How to install ivy Ivy is a standard ANT plugin. Download it's jar and place it in one of the following directories: $HOME/.ant/lib $ANT_HOME/lib I don't know why the ANT project doesn't ship with ivy.
Yes this is very possible. Note that in order to use this solution you will need to set your JAVA_HOME variable to JRE 1.6 or later. <project name="extractElement" default="test"> <!--Extract element from html file--> <scriptdef name="findelement" language="javascript"> <attribute name="tag" /> <attribute name="file" /> <attribute name="property" /> <![CDATA[ var tag = attributes.get("tag"); var file = attributes.get("file"); var regex = "<" + tag + "[^>]*>(.*?)</" + tag + ">"; var patt = new RegExp(regex,"g"); project.setProperty(attributes.get("property"), patt.exec(file)); ]]> </scriptdef> <!--Only available target...--> <target name="test"> <!--Load html file into property--> <loadfile srcFile="D:\Tools\CruiseControl\Build\artifacts\RECO\20110831100942\RECO_merged_report.html" property="html.file"/> <!--Find element with specific tag and save it to property element--> <findelement tag="title" file="${html.file}" property="element"/> <echo message="File : ${html.file}"/> <echo message="Title : ${element}"/> </target> </project> Output : [echo] Title : <title>Test Report</title>,Test Report As I don't know what exactly variables you were looking for this particular solution will find all elements that you specify in the tag attribute. Of course you could modify the regex to suit your own specific needs. Also this is pure build.xml ant with no external dependencies whatsoever.
Sure, but you have to write your own task for it. Visit http://ant.apache.org/manual/develop.html#writingowntask for more information about writing own tasks for Ant. In your Ant task you may parse your HTML file as needed. I claim, that it is not directly possible with "pure" XML (build.xml) to achieve what you want.
Take a look at the (http://ant.apache.org/manual/Tasks/xmlproperty.html) task and see if it'll work for you. It's pretty straight forward: <xmlProperty file="${html.file}" prefix="html."/> After all, HTML is just a subset of XML. I've used it before to do this very task. No need to write your own task or script.
Get XML Documentation files from TFS Team Build script
I have a dependency replication scheme setup in our TFS environment based on http://geekswithblogs.net/jakob/archive/2009/03/05/implementing-dependency-replication-with-tfs-team-build.aspx. This uses the CompilationOutputs item group to get the built DLL files and branch/merge them into dependent projects. My problem is that the CompilationOutputs item group only contains the DLLs, and I'd like to also include the XML documentation files, so I can get intellisense documentation tips when using these libraries. Is there a different item group that contains these, or a different approach? Do I need to manually find the xml files and add them to an item group? We're on TFS 2010 now, so if there's something new there, we can try to take advantage of it (though it'd be nice if I didn't have to convert this whole scheme to use a Workflow process...)
According to the article you copy and checkin the outputs: <Copy SourceFiles="#(CompilationOutputs)" DestinationFolder="$(ReplicateSourceFolder)"/> <Exec Command="$(TF) checkin /comment:"Checking in file from build" "$(ReplicateSourceFolder)" /recursive"/> Couldn't you add a second copy line before the checkin to copy the xml files using the metadata? <Copy SourceFiles="%(CompilationOutputs.RootDir)%(CompilationOutputs.Directory)\%(CompilationOutputs.Filename).xml" DestinationFolder="$(ReplicateSourceFolder)"/> Here is another option using an inline task that builds another item group changing the extension so that it only adds doc files that actually exist: <Target Name="Test"> <ChangeExtension InputFiles="#(CompilationOutputs)" Extension=".xml"> <Output TaskParameter="OutputFiles" ItemName="DocFiles" /> </ChangeExtension> <Copy SourceFiles="#(CompilationOutputs)" DestinationFolder="$(ReplicateSourceFolder)"/> <Copy SourceFiles="#(DocFiles)" DestinationFolder="$(ReplicateSourceFolder)"/> </Target> <UsingTask TaskName="ChangeExtension" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll"> <ParameterGroup> <InputFiles ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true"/> <Extension ParameterType="System.String" Required="true"/> <OutputFiles ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true"/> </ParameterGroup> <Task> <Code Type="Fragment" Language="cs"> <![CDATA[ if (InputFiles.Length > 0) { List<TaskItem> results = new List<TaskItem>(); for (int i = 0; i < InputFiles.Length; i++) { ITaskItem item = InputFiles[i]; string path = item.GetMetadata("FullPath"); string docfile = Path.ChangeExtension(path, Extension); if (File.Exists(docfile)) { results.Add(new TaskItem(docfile)); } } OutputFiles = results.ToArray(); } ]]> </Code> </Task> </UsingTask>
Ant replaceTokens with line numbers and file names
Is there any way in an Ant script to use the <copy> task (replaceTokens) to preprocess Java files and replace a "__FILE__" string with the file name of the copied file and "__LINE__" with the line number in that file? This would go a long way toward solving Java's lack of a preprocessor. The docs on replaceTokens show examples of fixed replacement strings, but I can't find any way to have Ant put in the file name and line number, respectively. Do I need to write my own task handler?
Here's what I came up with. It does __LINE__, but not __FILE__. Oh well :-(. <target name="preprocess" description="Preprocess the source code"> <mkdir dir="${target.generated.dir}"/> <copy todir="${target.generated.dir}" includeemptydirs="true" failonerror="true" verbose="true"> <fileset dir="${source.dir}"> <exclude name="${source.main.resources.dir}"/> </fileset> <filterchain> <tokenfilter> <filetokenizer/> <scriptfilter language="beanshell" byline="true"><![CDATA[ import java.io.BufferedReader; import java.io.StringReader; int count = 1; BufferedReader br = new BufferedReader(new StringReader(self.getToken())); StringBuilder builder = new StringBuilder(); String line; while ((line = br.readLine()) != null) { builder.append(line.replace("\"__LINE__\"", Integer.toString(count))).append('\n'); count++; } self.setToken(builder.toString()); ]]></scriptfilter> </tokenfilter> </filterchain> </copy> </target>
Ant: Exclude files from merged jar file
For a java project I'd like to merge all third-party jars it depends on into the main jar created by Apache Ant, which I already managed to do. The problem is that some of these jar-files have signature-files in their META-INF-directories, so when I try to run my jar-file, I get the error message "Invalid signature file digest for Manifest main attributes". After I delete the signature-files manually the error is gone. I tried to filter the signature files out in my ant-file with an excludes-attribute or an exclude-tag, but nothing seems to have any effect. This is the ant-task: <target name="jar" description="Creates the jar file"> <mkdir dir="${jar}"/> <jar destfile="${jar}/${ant.project.name}.jar" level="9" filesetmanifest="mergewithoutmain"> <zipgroupfileset dir="${lib}" includes="*.jar"/> <zipfileset dir="${class}"/> <manifest> <attribute name="Main-Class" value="${mainclass}"/> </manifest> </jar> </target> How can I filter files from the resulting jar in this ant-task? Thanks for your help!
carej is right. I've been trying to do this, merging other jars into my application jar excluding some files, and there is no way to use <zipgroupfileset> for it. My solution is a variant of the unzip/clean-up/jar method: I first merge all the external library jars into one with <zipgroupfileset>, then merge it into mine with <zipfileset> which does allow filtering. In my case it works noticeably faster and is cleaner than unzipping the files to disk: <jar jarfile="${dist}/lib/external-libs.jar"> <zipgroupfileset dir="lib/"> <include name="**/*.jar"/> </zipgroupfileset> </jar> <sleep seconds="1"/> <jar jarfile="${dist}/lib/historadar-${DSTAMP}.jar" manifest="Manifest.txt"> <fileset dir="${build}" includes="**/*.*"/> <zipfileset src="${dist}/lib/external-libs.jar"> <exclude name="*"/> </zipfileset> </jar> The first <jar> puts all the jars it finds in lib/ into external-libs.jar, then I make it wait for one second to avoid getting warnings about the files having modification dates in the future, then I merge my class files from the build/ directory with the content of external-libs.jar excluding the files in its root, which in this case were README files and examples. Then I have my own README file that lists all information needed about those libraries I include in my application, such as license, website, etc.
To the best of my knowledge there's no way to filter when using <zipgroupfileset>: the include/excludes used there apply to the zips to be merged, not the content within them. If you have a well-known set of JARs to merge you could use individual <zipset> entries for each one; this approach allows using include/exclude to filter the contents of the source archive. An alternative approach is to simply unzip everything into a temporary location, remove/modify the unwanted bits, then zip everything back up.
You can use the exclude parameter in zipfileset tag to remove content from merged external JAR files, as this: <jar jarfile="${dist}/lib/external-libs.jar"> <zipgroupfileset dir="lib/" excludes="META-INF/**/*"> <include name="**/*.jar"/> </zipgroupfileset> </jar> The resulting JAR file will be unsigned.
Alberto's answer works fine but takes time to unzip&rezip the archive. I implemented a new Ant task to use built-in filtering functions, that results in much faster execution: public class FilterZipTask extends Task { private Zip zipTask; private List<FileSet> groupfilesets = new ArrayList<FileSet>(); private String excludes; public void setExcludesInZips(String excludes) { this.excludes = excludes; } public void addZipGroupFileset(FileSet set) { groupfilesets.add(set); } public void addZip(Zip zipTask) { this.zipTask = zipTask; } #Override public void execute() throws BuildException { for (FileSet fileset : groupfilesets) { Iterator<FileResource> iterator = fileset.iterator(); while (iterator.hasNext()) { ZipFileSet zfs = new ZipFileSet(); FileResource resource = iterator.next(); zfs.setSrc(resource.getFile()); zfs.setExcludes(excludes); zipTask.addZipfileset(zfs); } } zipTask.execute(); } } And use it in build file as follows: <taskdef name="filterzip" classname="FilterZipTask"/> <filterzip excludesInZips="META-INF/*.*"> <zipgroupfileset dir="${deps.dir}" includes="*.jar" /> <zip destfile="${destjar}" /> </filterzip>
Have been struggling with the same issue for a few hours, so ended up writing a new task by extending the existing one: import java.util.HashSet; import java.util.Set; import java.util.StringTokenizer; import org.apache.tools.ant.taskdefs.Jar; import org.apache.tools.ant.types.ResourceCollection; import org.apache.tools.ant.types.ZipFileSet; public class CustomizedJarTask extends Jar { protected Set<String> filters; #Override public void add(ResourceCollection resources) { if (filters != null && resources instanceof ZipFileSet) { ZipFileSet set = ZipFileSet.class.cast(resources); for (String filter : filters) set.createExclude().setName(filter); } super.add(resources); } public void setFilters(String patterns) { StringTokenizer tokenizer = new StringTokenizer(patterns, ", ", false); while (tokenizer.hasMoreTokens()) { if (filters == null) filters = new HashSet<>(); filters.add(tokenizer.nextToken()); } } } With the above, all I need in the build file is: <taskdef name="customized-jar" classname="CustomizedJarTask" classpath="${basedir}/bin/" /> <customized-jar jarfile="${directory.build}/external-libs.jar" duplicate="fail" filters="META-INF/**"> <zipgroupfileset dir="${directory.libs}"> <include name="**/*.jar" /> </zipgroupfileset> </customized-jar>
I was also facing same problem. Googled a lot and found something that worked for me. Un-jar you jar file delete . META-INF/.SF , META-INF/.DSA files. Jar it again and run it should not show the error message. Cause of error is explained here: http://qe-cafe.blogspot.in/2010/06/invalid-signature-file-digest-for.html