After a recent juggling with our ant scripts I've started to wonder if something better is possible.
I need a builder that will know to recompile all required .java files for me.
For ex. for this structure
public class A { ]
public class B extends A {}
public class C {
B b;
}
For: Compile('C') Will know to compile A, B, C.
For: B changed, Compile('C') will know to recompile just B.
I know of several alternatives, Ivy which seems like an extension of ant which is our current java builder. Scons which we are currently using for building C++ code, scons is excellent in doing the above described behavior for C code. Then there are reports of Maven being almost but not quite there.
What would you suggest? What tools are you using Free Software / Commercial for you build system?
Thank you,
Maxim.
Ant, with 'depend' task and with 'closure' option turned on
'make', from IDEA ide
None of ivy, scons or maven will help you with your problem as stated.
What do you mean by "for Compile('C')"? I don't think this is what you have in your ant file.
For this case, Ant should be working as desired: you have described its default behaviour. In the same javac element, Ant will only recompile changed classes. See the Ant manual entry for the javac task, especially the 'includeDestClasses' attribute.
You should probably post an example ant file that you are finding inadequate.
maven, both for my personal and my commercial products
In your question you describe inter-class dependencies. Most build systems, in particular Maven, are aimed more at inter-project dependencies. I believe most systems just recompile all the classes in a project and most of the benefits of these build systems is in building as few projects as possible.
Both Maven and Ivy will allow you to easily specify both external and internal dependencies of your project, including which version of the project you depend on. They will both also automatically download external libraries (such as apache commons) to your local machine as part of the build process if they are not already locally cached, saving a lot of work manually downloading and organizing third party jar files.
Ivy is an extension of ant, like you mention. I recommend Maven. It is a convention oriented build system that I've used successfully and feel is quite mature. Maven requires far less up front effort to start using and is quite extensible.
Related
By default, Maven standard directory layout has two Java source folders:
src/main/java
src/test/java
For my purposes, I need a third one src/junit/java which should be packaged into a JAR with the classifier junit.
If possible, the new source folder should have it's own classpath (compile + everything with scope junit).
My guess is that for this, I will have to modify at least the resource and compile plugins.
Or is there an easier way?
I have a workaround as explained here but for that, I have to put things like Mockito and JUnit on the compile classpath which violates my sense of purity.
For all people who doubt the wisdom of my approach: I have support code that help to write unit tests when you work with code from src/main/java. Since I'm using the same support code in the tests for the project itself, this code needs to be compiled after src/main/java and before src/test/java.
Specifically, my support code needs to import code from src/main/java and the tests need to be able to import the support code.
I've seen a couple of Maven setups, which bundle test code in an own Maven module. You could then create a simple main-module <- support-module <- test-module dependency chain with that. But then main-module would compile fine, if you build it on it's own without test-module. Ofc you could aggreate them together with a reactor-pom and just build the project via this pom.
Edit:
If you have problems with this setup regarding code coverage, you can use the Jacoco Maven plugin to aggregate the test coverage generated by test-module to main-module. See this for further information: http://www.petrikainulainen.net/programming/maven/creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoco-maven-plugin/
I am making use of a library project which uses ant to build. My project however is using SCons because I need a far more complex build setup. Now I would like to use ant via SCons but NOT impose the problematic CLASSPATH issues and installation that ant requires.
So I am currently thinking of writing a build.xml parser, which turns the ant into SCons tasks.
Does anyone know whether this has been done before?
As far as I can tell there is no such parser in existence, which I partly believe is because there is great difference in how SCons and ant work. Especially when it comes to dependency resolution. It should be possible, but the translated file output will be very little SCons like, quite unreadable and probably quite difficult to maintain. Which pretty much defeats the whole reason to use SCons in the first place.
Since the library already uses ant, it would probably be a good idea to just incorporate the running of ant into SCons. If SCons can use ant, then you won't have to maintain the library build script (unless it is you that maintain the ant also)
Have you seen this: http://geosoft.no/development/android.html? We're also looking at converting an ANT based android build into our over-arching SCONS build and this looks like a good starting point.
I've been doing some research on the maven source and javadoc plugins, and I wanted to inquire a bit about the usage of each.
I understand conceptually how the plugins work, and what they do.
What I'm confused about, is why you would want to bundle sources or javadoc along with your artifact. Doesn't the javadoc get published when you do site:deploy? If I am creating a JAR library that will be used as a dependency of another project in eclipse, will attaching javadoc or sources enable me to see the javadoc in eclipse when using functions in that library, whereas if I fail to use the javadoc plugin, they won't be available?
What is "forked-path" and "jar-no-fork"? They seem to be relevant to this. Like I said I've done a lot of researching, I just can't tie it all together. Thanks!
Eclipse and other tools know how to download source and javadoc artifacts and use them to show you doc and source of your dependencies.
Forked-path and jar-no-fork are just about not running out of memory.
I have few own APIs with around 2000 classes overall. Some of them use the new Path API from JDK7. Most other classes, however, do not rely on any new JDK APIs or new language features. So most classes could be used in a JDK6 environment (which I plan to do). Let's assume, I've annotated all JDK7-only classes with #Java7Only.
What I need now, is a way to create a JDK6-only subset of all my projects more-or-less automatically, without introducing new version branching or product lines (would be too complicated to maintain).
All projects are created using Netbeans, thus using Ant. Many projects depend on others.
Please help me evaluate, which ideas according to my problem is most appropriate. Which problems could occur with each idea?
Common first step for all ideas
Let an annotation processor search for #Java7Only-annotated classes and store the list to a properties file.
Idea 1 (specific)
Write a tool which would use the properties file to recursively copy the whole project, except JDK7-only files.
Build the copied project using JDK6 by invoking ant, thus getting a JDK6-compliant jar.
Idea 2 (specific)
Write a second annotation processor which would use the properties file to pass everything except JDK7-only files to a JavaCompiler instance.
Either build a jar using Java APIs or use Ant API for that.
(This would be a Java-only idea, but probably too complicated)
Idea X (abstract)
Somehow influence the Ant build process (by overwriting some targets?) and for each JDK6-compliant class: let Ant compile two versions of it (one time with JDK6 compiler, another time with JDK7 compiler).
(JDK7-only classes would be compiled only once, using the JDK7 compiler, of course)
Package each bunch to a separate jar.
Possible common problems to the ideas
Some projects dependent on others, so some actions (such as packaging) should consider this.
Remember: the JDK7 compiler generates downward incompatible class files, that's why every possible idea has to happen on sources-level (before or during the build process, not afterwards).
My thoughts on Idea 2:
Essentially this is invoking a compiler within a compiler. Annotation processors are run as part of compilation. Can this be done safely? Is there any static state in Sun's javac that would cause problems. (I don't know the answer but from memory there might be some static state that could cause problems in this scenario).
Idea 1 seems simpler and better to me.
But taking a step back, is it possible to separate out all the JDK 7 specific stuff into a separate module and compile it separately, into a different JAR?
Have the 'main' project, compiled using JDK 6 (which JDK 7 would have no problems reading because it is backwards compatible)
The JDK 7 specific module(s), with source in a different directory, which includes the 'main' JAR on the compilation classpath, could be built separately, with a different build.xml if necessary.
This only partially applies but I'd thought I'd mention it anyway.
The problem with just using -source 1.6 -target 1.6 options for validation is that you can still use Java 7 API when compiled using JDK 7.
I've used the Animal Sniffer Maven Plugin for a few projects now and it has proved quite useful. This plugin scans byte-code of your classes for JDK API usage. That is, you can tell it to fail the build if you attempt to use JDK 7 API when you are targeting JDK 6. This wont help much for separating out classes as you need but it could be useful as a final validation step combined with -source 1.6 -target 1.6 compiler options.
There is also an animal sniffer Ant plugin, as mentioned from the Animal Sniffer main page.
Apache Maven is a very popular build and dependency management tool in the Java open source ecosphere. I did some tests to find out if it can handle compiled Free Pascal / Delphi units and found it easy to implement. So it would be possible to
release open source libraries precompiled for Free Pascal (or Delphi) in a public Maven repository
include metadata in this repository which contains dependency information
use Maven on the command line to download the open source library from the public repository, and automatically resolve all dependencies
local repositories, working as proxies, could be used to cache frequently used binaries
automatic checksum generation and verification (provided by Maven) would reduce the risk of downloading corrupted binaries
source code and even documentation files could be provided with the binaries
binaries can be provided with or without debug information
continuous integration servers like Hudson, TeamCity or CruiseControl can be used to build projects whenever changes have been submitted to the source control system and notify developers about build errors
This way of dependency management could be very beneficial for open source projects which use many third party libraries with complex dependencies. It would avoid typical conflicts caused by using wrong versions.
For the developer, the workflow for editing and building a project would be reduced to a minimum:
checkout the project source from internal version control system
edit source file(s)
run mvn package to automatically download all required third party libraries (precompiled units) if they are not yet in the workstation's local repository
compile and run
The only additional file for Apache Maven which is required in the project folder is the POM.XML file containing the project information.
Edit: while Maven is usable for some of the required tasks, implementing a solution like Maven in native Free Pascal would have some advantages: no Java SDK required, support for all development platforms where Free Pascal is available, maintenance and plugin development in Pascal.
Usage of a Maven-like tool would not be helpful for open source projects only - commercial projects could access and use the artifacts in public Maven repositories in the same way as well.
Maven features are listed at http://maven.apache.org/maven-features.html
Update:
one use case could be the build of Lazarus, where Maven would download all required libraries and invoke the compiler with the necessary build path arguments. Changes in the dependencies on lower levels would be propagated automatically up to the parent build.
Possible benefits:
less time needed to set up a new work
station, no manual installation of
third party libraries required
less errors caused by wrong library
versions, detection of version
conflicts (for example if two
libraries depend on different
versions of a third library)
artifacts which are created inhouse
can be added to the local maven
repository and shared between
developers and project, central
storage of all artifacts with
metadata
builds are reproducible, just by
using the same source and project
metadata file (pom.xml)
can reduce development time and
increase project stability
Update #2: FPMake
the FPMake build system for Free Pascal seems to be a tool with much potential, in many details it is quite similar to Maven:
FPMake is a pascal based build system developed for and distributed with FPC
FPMake standardizes the building by defining some limits like standard directories
the command fppkg <packagename> will look in a database for the package, extract it, and then compile fpmake.pp and run it
it has standard build targets (clean, build, install, ...)
it can create a 'manifest' file suitable for import into a repository (like mvn deploy or mvn install), the manifest is an XML file which looks very similar to a pom.xml in Maven:
FPMake manifest file:
<packages>
<package name="my-package">
<version major="0" minor="7" micro="6" build="1"/>
<filename>my-package-0.7.6-1.zip</filename>
<author>my name</author>
<license>GPL</license>
<homepageurl>http://www.freepascal.org/</homepageurl>
<email>myname#freepascal.org</email>
<description>this is the package description</description>
<dependencies>
<dependency>
<package packagename="rtl"/>
</dependency>
</dependencies>
</package>
</packages>
Freepascal has been working on a package system of its own in a cross between apt-get and freebsd ports style. (download source/build/install automatically), called fppkg.
However work has stalled. People investing time are the bottleneck, not people wanting to choose tools.
As far as Maven goes, I don't like auxilary tools that need installation of huge external runtimes. It might be fine for a big major app (like Open Office), but not for an util.
I also prefer a tool that is designed to the FPC reality and workflow.
Documentation tools, build tools, download systems, testsuite systems are already all there, it just need a person that dedicates a lot of time into it to make it happen.
Some typical problems when introducing a new technology in a project as FPC, and why it has a tendency to make its own tools:
need to train 20+ committers in parttime.
The only COMMON programming language you can assume is Free Pascal. Even Delphi inner workings can't be taken for granted to be known (many committers came directly to FPC or even still via TP or a Mac Pascal)
Obviously that makes something with plugins in a different language annoying.
Bash script is a close second. (g)make third, but already a magnitude less.
All servers are *nix-like (FreeBSD, OS X, Linux), but not all run Apache. (e.g. my FreeBSD mirror runs XSHTTPD)
somebody most knowledgable must be dedicated maintainer for a long time. Fix problems, update/ do migrations etc. Perferably more than one for obvious reasons.
a major pain are Linux distributions (and FreeBSD to a lesser degree), most maintainers of *nix packages are not capable of more than "./configure;make;make install", and must be spoonfed with a near buildable repository and auxilary files.
In-distribution packaging of FPC/Lazarus has always been important, and is still increasing
All distributions have their own special rules about metadata, depedancies, and how sources must be published. Particularly Debian/Ubuntu is very bureaucratic and slow.
Most don't like third party auto-installers on top of their systems (since that bypasses their dependancy control)
This all leads to the effective practice that own tools in Pascal with minimal scripting work best. Some tools used:
Gmake is mainly used to parameterise the build process on a per directory level, a successor, fpcmake (not really a make derivative despite the name) has begun, but the migration hasn't completed.
Latex and a latex to html conversion (tex4ht, but debian uses hevea) are used in the documentation building (the non library documentation)
The community site (netscape community server which uses TCL scripting, a heavy complex application server) has been a trouble ever since it started, but specially lately since the maintainer became less active.
Mantis has been a problem (specially the email module would crash or lame the server due to the volume), but it has been whipped into shape during successive updates and hard work of several lazarus devels. Currently it is a decent workhorse.
lazarus.freepascal.org PHPBB forum OTOH is relatively painless since a lot of younger people know how to deal with it.
The same goes for subversions (though the more advanced scale needs some adjusting, not everybody is deep into the ins and outs of mergetracking)
If somebody was really serious about Maven, I usually would ask him:
to CRITICIALLY investigate the use for the project. In a very concrete way, with schedule and time estimates. Birds-eye level "everything's possible" overviews are essentialy worthless.
Give some thought on future change of used technologies. Every technology is eventually replaced, even the in-house ones, in 18 year+ projects. A new technology must not make migrations of other infrastructural components hard or involved. The new technology to end all new technologies doesn't exist.
Make a migration plan. Migration is often underrated and underestimated.
And in the end, there is always the 1000000 Euro question, who will do the daily maintenance?
Keep in mind that in a company you just kick the person responsible for the application server. But in an informal environment this is way harder, specially long term, since people's lives, occupations and time spent on the project vary.
Sounds like an interesting plan, but the Delphi community (and FPC even more so, I'd imagine!) values libraries as source far more than precompiled libraries. The general consensus is that anyone who uses a binary-only library is a fool, for two reasons: You can't fix any bugs you find in it, and compiler changes will break compatibility.