Avoiding re-building prerequisites in Ant - ant

I have an existing Ant project and would like to speed up the build process
by avoiding re-building components that are already up to date.
Ant permits you to specify that one target depends on another, but by
default every prerequisite is always rebuilt, even if it is already up to
date. (This is a key difference between Ant and make. By default, make
only re-builds a target when needed -- that is, if some prerequisite is
newer.)
<uptodate property="mytarget.uptodate"> // in set.mytarget.uptodate task
...
</uptodate>
<!-- The prerequisites are executed before the "unless" is checked. -->
<target name="mytarget" depends="set.mytarget.uptodate" unless="mytarget.uptodate">
...
</target>
To make Ant re-build prerequisites only if necessary, there seem to be two
general approaches within Ant.
The first approach is to use the uptodate task to set a property. Then,
your task can test the property and build only if the property is (not)
set.
<uptodate property="mytarget.uptodate"> // in set.mytarget.uptodate task
...
</uptodate>
<!-- The prerequisites are executed before the "unless" is checked. -->
<target name="mytarget" depends="set.mytarget.uptodate" unless="mytarget.uptodate">
...
</target>
An alternate first approach is to use the outofdate task from ant contrib.
It's nicer in that it is just one target without a separate property being
defined; by contrast, outofdate requires separate targets to set and to
test the property.
The second approach is to create a <fileset> using the <modified>
selector. It calculates MD5 hashes for files and selects files whose MD5
differs from earlier stored values. It's optional to set
<param name="cache.cachefile" value="cache.properties"/>
inside the selector; it defaults to "cache.properties". Here is
an example that copies all files from src to dest whose content has
changed:
<copy todir="dest">
<fileset dir="src">
<modified/>
</fileset>
</copy>
Neither of these is very satisfactory, since it requires me to write Ant
code for a process (avoiding re-building) that ought to be automatic.
There is also Ivy, but I can't tell from its documentation whether it
provides this feature. The key use case in the Ivy documentation seems to
be downloading subprojects from the Internet rather than avoiding wasted
work by staging the parts of a single project. Maven provides similar
functionality, with the same use case highlighted in its documentation.
(Moving an existing non-trivila project to Maven is said to be a nightmare;
by contrast, starting greenfield development with Maven is more palatable.)
Is there a better way?

This conditional compilation of a large build is a feature of make that I initally missed in ANT. Rather than use target dependencies, I'd suggest dividing your large project into smaller modules, each publishing to a common shared repository.
Ivy can then be used to control the component versions used by the main module of the project.
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="multi_module_project"/>
<publications>
<artifact name="main" type="jar"/>
</publications>
<dependencies>
<dependency org="com.myspotontheweb" name="component1" rev="latest.integration"/>
<dependency org="com.myspotontheweb" name="component2" rev="latest.integration"/>
<dependency org="com.myspotontheweb" name="component3" rev="latest.integration"/>
<dependency org="com.myspotontheweb" name="component4" rev="latest.integration"/>
</dependencies>
</ivy-module>
The ivy:retrieve task will only download/copy one of the sub-modules if they have changed (published from their build files)
It all sounds more complicated but maybe you're already sub-dividing the project within your build file.... For example, if your ANT uptodate task is being made dependent on one the build artifacts.

Related

Sync Task's Overwrite Attribute Copies Identical Files

I've come across an issue with Ant's Sync task where files are being copied unnecessarily. The goal is to update everything in the ${destination} directory with the contents of the ${source} directory, even if the file in the ${destination} is newer. Based on Ant's documentation, I've added an overwrite attribute to to ensure the ${destination} is overwritten.
<target name="test">
<sync todir="${destination}" overwrite="true" granularity="5000">
<fileset dir="${source}">
</fileset>
</sync>
</target>
This task correctly overwrites the ${destination}, but the file is always copied, even when the source and destination are identical. This leads to a lot of unnecessary traffic.
Based on the documentation, I attempted to configure the granularity attribute, but this doesn't appear to have any effect. I'm also running this test between two directories on the same machine, so I wouldn't expect timestamp differences (certainly not of more than 5 seconds).
Any thoughts about why the Sync task and overwrite attribute function in this way? Are there any solutions using the default set of Ant tasks to prevent the unnecessary file copying?
If you use the sync task with overwrite="true", you will get this behavior.
You could use it with overwrite="false", and then follow up with a copy task that only copy the files that are existing but different, with the different selector, e.g.:
<copy todir="${destination}">
<fileset dir="${source}">
<different targetdir="${destination}" ignoreFileTimes="true"/>
</fileset>
</copy>

Findbugs ant task submitting several dynamically detected JAR files for analysis

I'm currently looking to run static analysis over a pre-existing project. As the project is created and supplied by an off-site company, I cannot change the build process radically.
The project is split into a lot of sub-modules, located in various places. For other analyisi tools (JDepend, Google Testability Explorer, etc.), I have dynamically detected all build JAR files into a path element as follows:
<path id="built-libs">
<fileset dir="${overall-base}">
<include name="${some-common-base}/**/lib/*.jar" />
</fileset>
</path>
<property name="built-libs-string" refid="built-libs" />
For some tools, I use the build-libs, for others I use the string (in classpath form; x.jar;y.jar).
The trouble is, FindBugs uses a completely different format to any other;
<class location="x.jar"/>
<class location="y.jar"/>
...
Now, I could list all the JAR files manually, but then run the risk of this list going out of synch with the other tool's lists, or of introducing typos.
Another complication is that I also want to run the reports in Jenkins, in this case the extract directory for individual modules will depend on the job that has previously built the module (pipeline builds, modules extracted from SCM and built in parallel, the reporting occurring at the end of the pipline).
I could make a call out to the OS to run FindBugs, passing in the JARs in a space separated list (as in Invoking FindBugs from Ant: passing a space-separated list of files to java). However, I prefer a, Ant solution to an OS <exec... hack.
Note I know I have a similar problem for the sourcepath element, however, I'm assuming that solving the class element problem also solves the sourcepath one.
Ideally, FindBugs should be taking a resource collection rather than separate class elements. I'm not familiar with FindBugs, so I can't comment on why they have chose to go the class element route instead of a resource collection, however your comment about using exec implies that using a resource collection is a valid design alternative.
I would try rolling your own Ant macro, which invokes FindBugs directly using the java task. This should give you the control you need and avoiding the redundancy that the FindBugs Ant task would introduce.
Another option (which is an ugly hack) is to use the fileset to write a mini ant file with a FindBugs target, which you then invoke using the ant task. shudders
The Findbugs Ant task allows you to specify a filelist which can be used to specify multiple files. Quoting from the Findbugs documentation
"In addition to or instead of specifying a class element, the FindBugs
task can contain one or more fileset element(s) that specify files to
be analyzed. For example, you might use a fileset to specify that all
of the jar files in a directory should be analyzed."
Example that includes all jars at ${lib.dir}:
<findbugs home="${findbugs.home}" output="xml" outputFile="findbugs.xml" >
<auxClasspath path="${basedir}/lib/Regex.jar" />
<sourcePath path="${basedir}/src/java" />
<fileset dir="${lib.dir}">
<include name="*.jar"/>
</fileset>
</findbugs>

Published artifact patterns in Ivy

When I resolve artifacts from my repository (e.g. filesystem), I use two artifact patterns:
<artifact pattern="${location}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
<artifact pattern="${location}/[organisation]/[module]/[revision]/[artifact]-[revision]-[type]s.[ext]"/>
The first one is for jar files, and the second one is for sources or other types of artifacts.
I'd like to be able to publish artifacts the same way, but I don't know how.
Using just the patterns above, the publish task seems to consider only the first one, thus removing the type. If multiple artifacts have the same name and extension, they will be overwritten.
If I just use the second pattern, then for jar artifacts it makes ${artifact}-${revision}-jars.jar which is really ugly.
Finally, it seems to be possible to have optional parts in patterns, such as:
<artifact pattern="${location}/[organisation]/[module]/[revision]/[artifact]-[revision](-[type]s).[ext]"/>
But the -[type]s part is omitted only if the type is null or empty, and I'd like the type to remain "jar", in which case the part is not omitted.
So is there any other way?
Why don't you use ivy.xml files for artifacts? You should need to create ivy.xml and place them to your module folder, near jar files. ivy.xml example:
<ivy-module version="1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="com.organisation" module="foo" revision="1.0"/>
<publications>
<artifact name="foo"/>
<artifact name="foo-sources" type="source" ext="zip"/>
</publications>
</ivy-module>
Then you should define ivy pattern in your resolver:
<ivy pattern="${location}/[organisation]/[module]/[revision]/ivy.xml"/>
Now if you use <dependency org="com.organisation" name="foo" rev="1.0"/> you will get all artifacts described in ivy.xml. There is a way to select only the needed artifacts.
Not really a solution, but a slightly better way is:
<artifact pattern="${location}/[organisation]/[module]/[revision]/[type]s/[artifact]-[revision].[ext]"/>
I struggled with the same.
I found the solution, you can use:
[artifact](-[classifier]).[ext]
-classifier will be null/empty on the jar on sources/javadoc-jars it contains -sources/-javadoc.jar
I know its been a while, but I found this question by google. So for any future person it will be helpful I think.

ant iterate over files

I want to iterate over a list of jars (undefined number) and add them all to the jar file.
To add them I plan to use something like this:
<jar id="files" jarfile="all.jar">
<zipfileset src="first.jar" includes="**/*.java **/*.class"/>
<zipfileset src="second.jar" includes="**/*.java **/*.class"/>
</jar>
but how do I iterate over them? I don't have ant-contrib
Thanks!
Just use zipgroupfileset with the Ant Zip task
<zip destfile="out.jar">
<zipgroupfileset dir="lib" includes="*.jar"/>
</zip>
This will flatten all included jar libraries' content.
If you do not have access to ant-contrib For task, you may end up to have to define your custom Task for doing what you need...
If you have ant1.6 and above, you can also try subant (see New Ant 1.6 Features for Big Projects):
If you use <subant>'s genericantfile attribute it kind of works like <antcall> invoking a target in the same build file that contains the task.
Unlike <antcall>, <subant> takes a list or set of directories and will invoke the target once for each directory setting the project's base directory.
This is useful if you want to perform the exact same operation in an arbitrary number of directories.

Filter a fileset referenced using a refid

I have a fileset (which is returned from the Maven Ant task), and it contains all the jars I need to repack. This fileset is referenced by a refid. I only want to include our own jars, so I would like to filter that. But Ant filesets don't support any further attributes or nested tags if a refid is used.
For example, if the fileset is:
org.foo.1.jar
org.foo.2.jar
log4j.jar
and I want to have a fileset which contains only
org.foo*.jar
How would I do that?
Try using a restrict resource collection, which you can use like a fileset in any task that uses resource collections to select the groups of files to operate on.
For example, for a fileset returned from your Maven task referenced via an id called dependency.fileset you can declare a restrict resource collection like so:
<restrict id="filtered.dependencies">
<fileset refid="dependency.fileset"/>
<rsel:name name="org.foo*.jar"/>
</restrict>
Note you'll have to declare the resource selector namespace as it isn't part of the built-in Ant namespace:
<project xmlns:rsel="antlib:org.apache.tools.ant.types.resources.selectors">
...
</project>
From here you can reference your restrict resource collection in a similar fashion to how you would reference your fileset. For example, to create backups of your filtered set of files:
<copy todir=".">
<restrict refid="filtered.dependencies"/>
<globmapper from="*" to="*.bak"/>
</copy>
Of course you can inline your restrict resource collection if you so desire:
<copy todir=".">
<restrict>
<fileset refid="dependency.fileset"/>
<rsel:name name="org.foo*.jar"/>
</restrict>
<globmapper from="*" to="*.bak"/>
</copy>
Have a look at the Ant documentation on resource collections for further information.
I think you'll need to write an ant task for that. They're pretty easy to write though.
See http://ant.apache.org/manual/develop.html#writingowntask
In your task, you'll need to call getProject() and ask it to give you the fileset, walk through it, and create a new one.
I 'm using Ant with Ivy. With the help of Ivy it is possible to filter dependencies for retrieval, with the following code in ivy.xml:
<dependency name="Project1" rev="latest.integration" transitive="true" conf="modlibs">
<exclude name="${exclusionRegEx}" matcher="regexp" />
</dependency>
<dependency name="Project2" rev="latest.integration" transitive="false" conf="modules"/>
Maybe a quick look at the Ivy source 'll help?
If you are using a sufficiently recent version of Ant and the JDK, for example, Ant 1.7 and JDK 6, then you can use the optional script task to do what you want. (Earlier versions may also work.) The page I linked to, if you scroll down to the text "The goal is to list the filesizes" then you'll see a sample script that creates a Fileset.
This isn't for the faint of heart, and a custom ant task you write yourself will probably be more flexible. But I wanted to point out the option.

Resources