Ant: Parallel build with external compiler - ant

Question

I need some help on ant. How can I make ant behave similarly to make -j <n>?
Note that I am not building java, but invoking an external compiler for a proprietary language.
While there is the task <parallel> (and an analogous attribute for <for>), they do not seem to support dependencies, or rather I haven't found a way to apply dependencies on these.
However, I do have the dependencies available in a properties file, formatted like this:
fileA=fileB fileC
fileB=fileC
fileA/B/C are the basenames of my target files.
Currently there are some 850 files to build. Using a <for> I walk over this list (properly sorted by ironing out the dependencies) to invoke the compiler.
I was already advised to use <sequential>, however this would mean to translate the dependencies into blocks of <sequential> which is kind of bad.
Any ideas on how to make the <for ... parallel="yes"> loop respect dependencies, or any other idea on how to solve this?

Answer 1

I don't think there's anything in native Ant that can do this, but this example is an absolutely ideal use case for the "dataflow concurrency" model supported by Groovy's GPars, so you can do something with a script task:
<project name="gpars-test">
<path id="groovy.path">
<pathelement location="groovy-all-1.8.8.jar" />
<pathelement location="gpars-0.12.jar" />
</path>
<!-- Target to build one file - expects a property "filetobuild"
containing the name of the file to build -->
<target name="buildOneFile">
<echo>Imagine I just built ${filetobuild}...</echo>
</target>
<target name="main">
<script language="groovy" classpathref="groovy.path"><![CDATA[
import static groovyx.gpars.dataflow.Dataflow.task
import groovyx.gpars.dataflow.Dataflows
// load dependencies
def deps = new Properties()
new File(basedir, 'dependencies.properties').withInputStream {
deps.load(it)
}
def df = new Dataflows()
// spawn one "task" per file to be compiled
deps.each { file, dependencies ->
task {
if(dependencies) {
// wait for dependencies - reading df.something will suspend
// this task until another task has written the same variable
dependencies.split(/ /).each { dep ->
def dummy = df."${dep}"
}
}
// now we know all our dependencies are done, call out to build
// this one.
def compiler = project.createTask('antcall')
compiler.target = "buildOneFile"
def prop = compiler.createParam()
prop.name = "filetobuild"
prop.value = file
try {
compiler.perform()
} catch(Exception e) {
// do something
} finally {
// we're done - this will release any other tasks that are blocked
// depending on us
df."${file}" = "done"
}
}
}
// now wait for all tasks to complete
deps.each { file, dependencies ->
def dummy = df."${file}"
}
println "finished"
]]></script>
</target>
</project>
Note that this does assume that all the files you want to build are listed in the dependencies file, including those with no dependencies, i.e.
fileA=fileB fileC
fileB=fileC
fileC=
If the no-deps file names aren't listed in the properties then you'll have to do some more fiddling to put them in there - immediately before the def df = new Dataflows() you'd need to add something along the lines of
deps.values().collect().each { depString ->
depString.split(/ /).each {
// if we find a fileX that something depends on but which does not
// itself appear in the deps map, assume it has no dependencies
if(!deps.containsKey(it)) {
deps.setProperty(it, "")
}
}
}