What would be, in your opinion, the best way that a project can depend on another pre-compiled project that is distributed over debian packages?
A custom (new_)debian_package() workspace rule?
Indeed, I'd introduce a new debian_package rule that mimics what maven_jar does.
Related
A few years ago I wrote a set of wrappers for Bazel that enabled me to use it to build FPGA code. The FPGA bit is only relevant because the full clean build takes many CPU days so I really care about caching and minimizing rebuilds.
Using Bazel v0.28 I never found a way to have my Bazel package depend on a single source file from somewhere else in the git repo. It felt like this wasn't something Bazel was designed for.
We want to do this because we have a library of VHDL source files that are parameterized and the parameters are set in the instantiating VHDL source. (VHDL generics). If we declare this library as a Bazel package in its own right then a change to one library file would rebuild everything (at huge time cost) when in practice only a couple of steps might need to be rebuilt.
I worked around this with a python script to copy all the individual source files into a subdirectory and then generate the BUILD file to reference these copies. The resulting build process is:
call python preparation script
bazel build //:allfpgas
call python result extractor
This is clearly quite ugly but the benefits were huge so we live with it.
Now we want to leverage Bazel to build our Java, C++ etc so I wanted to revisit and try and make everything work with Bazel alone.
In the latest Bazel is there a way to have a BUILD package depend on individual source files outside of the package directory? If Bazel cant, would buck pants or please.build work better for our use case?
The Bazel rules for most languages support doing something like this already. For example, the Python rules bundle source files from multiple packages together, and the C++ rules manage include files from other packages. Somehow the rule has to pass the source files around in providers, so that another rule can generate actions which use them. Hard to be more specific without knowing which rules you're using.
If you just want to copy the files, you can do that in bazel with a genrule. In the package with the source file:
exports_files(["templated1.vhd", "templated2.vhd"])
In the package that uses it:
genrule(
name = "copy_templates",
srcs = ["//somewhere:templated1.vhd", "//somewhere:templated2.vhd"],
outs = ["templated1.vhd", "templated2.vhd"],
cmd = "cp $(SRCS) $(RULEDIR)",
)
some_library(
srcs = ["templated1.vhd", "templated2.vhd", "other.vhd"],
)
If you want to deduplicate that across multiple packages that use it, put the filenames in a list and write a macro to create the genrule.
I'm wondering if there is a way to selectively branch files and folders in TFS. We currently branch the trunk but it is a bit cumbersome.
Lets say I have the following structure:
--trunk
--base code
--internal code that relies on base code
--client code that relies on base code
I would like to branch base and internal and base and client.
if this is not possible what would be a good folder structure to facilitate our needs?
Rather than including base (the dependency) in the source for your internal/client, a typical approach is to have base in it's own branch. Then have internal and client take a binary dependency rather than a source dependency on base.
This is realized by having a lib folder in each internal and client, that contains a copy of the specific version of base on which they have a dependency. This allows you to evolve base and the applications that depend on it independently, and it allows the internal/client developers to make an explicit decision about which version of base to depend on.
I am considering switching a Maven project that I manage to Apache-Ant/Ivy. I need more control over the build process and am getting very frustrated with Maven. Please no comments about how great Maven is. My question is about Ivy.
I would like to set up a "standard" Ant build template that can later be used for other projects with minimal changes.
I will set up a central "enterprise" repository where we can place third-party libraries that are not available in the public Maven repositories (e.g. commercial libraries, Sun libraries, proprietary libraries, etc.). This enterprise repository will be available on our local LAN, but may not be available from outside the office.
Each developer will have a private repository in ~/.ivy/repository. I would like the Ant build to automatically update this private repository with changed versions of libraries from the enterprise repository.
In ~/.ivy/ant, I plan on placing "standard" modules for including in the individual project build.xml files, using the include task in Ant 1.8. These modules will provide things like Scala and Clojure compilation targets with different versions for different Scala and Clojure versions (e.g.: scala-compile-2.9.1.xml, clojure-compile-1.3.xml, etc.) The build modules will be available in the enterprise repository and should be updated automatically in the private repositories if they change.
Each project will follow a standard Maven directory structure: ${project}/src/main/java, ${project}/target/classes, etc.
In the past, I tried using Ivy but the Ant build files got to be very large (> 500 lines for the template build file) and hard to manage/edit. I am hoping that by putting standard targets in their own build modules in the ~/.ivy/ant directory, I can avoid that code bloat.
Can this be done? Am I way off base? The only documentation I can find on Ivy is at the Apache web site (http://ant.apache.org/ivy). Is there any other documentation available, including books?
Rather sensible idea about dividing template build file into includable helper files. Personally, now i'm switchin' a really large project from ant (no dependency managment at all - only copying files from ftp) to ant/ivy solution. So i've done this way - i have a file with milestones targets - i.e ready-to-compile, compiled, ready-to-archiving, archived - so on. I think u got the idea. I've configured dependencies of all this targets ( dependencies in terms of ant, do't get me wrong). In that way - compiled depends on ready-to-compile, ready-to-compilede depends on initialized - smth like this. This targets don't have body - they are for including in every build-file of every module of your multi-module project. The sole purpose of this targets for maintaining the STATE of build, because of this import stuff things become rather tricky and it's hard time to know what target was overriden, and when this target would be run. But with this file i can easily change state of vy build on every sensible milestone. I want in one module to compile help files with exteran exe. No problem - in this project i just do this - ready-to-archiving depends on the target for compiling help. And as this milestones targets are included - i can override only some of them - all others would presere the desired way of building project.
Another part of my strategy - mixins build files - for every specific area. So i have a file for ivy. There i put initializing, resolving, publishing and so on. When i want to use ivy - i just include this file and manage depdendencies through my milestones targets. If the build is typical - i only include this file and have a convention-over-configuration functionality. All out of the box. How?? Just combining with other mixins. Mixins may include other mixins to depend on them. So each mixin is a reusable part of my build strategy. The stuff from OOP - single-concerned unit. In your case it's scala mixin with targets specific for scala stuff.
Then i have delegate.xml that delegates child projects common build activities. I have dist, all, test and whatever u want for multimodule project. The build order is evaluated with ant-ivy task buildlist.
There also some other files - but this are the strategically basic files that helped me to have a reusable and maintanable build with this BIG and VERY Conservative project. So, if u are interested about details, don't be shy and contact me. I will be very pleased to help you, because ivy docs are really comlicated and incomplete.
EDIT: About books - Ant in Action may help you, i took several ideas from this book, and i really highly recommend it everyone to read. There u can find ivy stuff, also. And about ivy docs - sorry, it's all that is available. But when i was in trouble with this cumbersome ivy+ant - i found several interesting articles on private blogs. So ... that may fill the gap some way.
For those of you that use Ant with multiple projects, where do you put the build.xml files? Do you put one in each project, or do you put them in a separate project that contains all your Ant-related files?
The usual recommendation is to put a build.xml in each project. But this has a few drawbacks:
It makes it hard to reuse common targets in multiple projects.
Sometimes you want to use Ant to export a project from source control and deploy it. Obviously you can't do this if the build file is in the project itself.
But if you put them all in a common location:
People need to be aware of their location to use them; they can't just use "ant -find" to find the current project's file.
You can't have different build instructions for different branches of the project.
What do you guys do?
EDIT: Thanks for the good suggestions so far. As far Maven, these aren't Java projects, and I get the impression that Maven is only meant for Java.
Place the Ant files with the project. That is the de facto standard and recommended by the creator of Ant. I will try to address some of the issues you have brought up:
Reuse of common targets should be done using techniques as described by Eric Hatcher in his book Java Development with Ant. Basically, you extract all commonality into a some top level files that all other Ant files "inherit" from.
Using Ant to export a project from source control seems odd to me, but if you want to do this, use a different Ant file :-) You can make a target like ant export -Dproject=foo/bar.
For Ant, I recommend you grab that book - it has a ton of helpful techniques.
The real recommendation I would make though is to drop Ant and convert over to Maven - like the Apache Software Foundation did (they maintain both Ant and Maven).
If you're working with independent projects, you can:
put your build.xml at the top level
place common Ant definitions (Antlib) into another project (e.g. config)
use svn:externals to import the common Antlib definition (from 'config') into your project
EDIT The trick with svn:externals is that if you link to the HEAD of some common files, it may happen that they will change after a couple of months/years. So each time you tag, you should change the svn:externals to point to a fix version of the included project. This may come handy when a project has to be rebuild years after it was last built.
My rule of thumb is to put the build.xml file in the directory under which all files are referenced. In other words, no relative paths should start with "../". Where I live, that usually means putting it in the "trunk" directory, which has src, lib, build, docs, etc underneath it.
Doing this makes the paths much cleaner in the file, and it makes it obvious how to build the project.
Where I have multiple projects that need to build, I will create a separate build.xml for each project, and a central build.xml in the directory all the project are in that calls those other build.xml files. That gives you a lot of flexibility with very little work.
I'd expect an Ant build file to be located at the top of a project (it's already a pain to have to look at a the build file to "discover" how to build the project, so if I have to locate it first, it'll drive me totally crazy). Now, regarding all the drawbacks you mentioned, I'm tempted to say: why don't you use Maven?
The way I have done this is in the past (Now I just use Maven):
Have a build.xml in the root of each project
Create an overarching build.xml
for all projects and place it in
the trunk of my repository
The overarching buid.xml has
checkout tasks for each project.
I am guessing when you mentioned
export from repository, you
actually meant import.
The overarching build file also
defines the dependencies, if any
You may update individual projects using each project's individual build file
If you do have common tasks defined, you may inherit from a common build file as well as someone else suggested.
Looks like your set of projects might be a good candidate for migration to Maven, I realize it is not always possible but if you have time, you might want to look into it.
It seems that ant dependency management isn't great to say the least... But, is there hope?
Today there are better choices for build or project management such as maven, ivy but I'm stuck with a bunch of ant project that depend on one another so I was wondering if there's an "ant best practice" for managing dependencies.
Specifically in my case I have:
w.war
a.jar
b.jar
c.jar
w.war is the war file I deploy to the web server. In the war file I use a.jar. a.jar depends on b.jar, so I happily package b.jar into w as well... That's sort of OK until...
The problem starts when b.jar depends on c.jar. The author of a.jar knows about its dependency on b.jar so it can package b.jar into w.war, but it is not aware of the dependency of b.jar on c.jar. Moreover, the author of b.jar could later add even more dependencies such as "b.jar depends on e.jar", so the author of a.jar has no chance following these dependencies as they add up.
What I'd like to have is define "a depends on b" and "b depends on c" (in a different build.xml file) and use ant magic to compile them all into w.war.
Is that possible? If not, is there a Best-Practice?
I'm this-close to just rewriting is all in maven, but it's a lot of work... Is there hope to ant?
If you already use Ant, then your best bet is to use Ivy for dependency management.
http://ant.apache.org/ivy/
It provides a rich set of ant tasks for dependency manipulation.
Just to disillusion you: dependency management in combination with a build system is hard! Unless you have a really simple example, this is not a no-brainer, and you will have to invest some work.
If you want to or have to use Ant, Ivy is definitely a good choice. Being an official sub-project of Ant, it integrates relatively smoothly.
I've been using Ivy for the last 4 years (way before it joined Apache) and haven't had any regret.
Without knowing much about your Ant files, it's a little bit difficult to give a definitive answer but I guess that the projects are build in one go. This means that they can't individually publish their artifacts (or jar files) into a central repository.
The solution is to use both a central (for common libraries) and local (for your project) repositories. You can take a look at my public projects and especially the ivysettings.xml file because they use exactly this very principle.
Here's my take: use the best of both worlds: Maven just for dependencies and Ant for your day-to-day build heavy-lifting:
Why you should use the Maven Ant Tasks instead of Maven or Ivy
There's a new open source build system for Java called EBuild (features) that is a great alternative to Ant/Ivy as it is especially good at dependency management.
There's some detailed articles on the deficiencies of Ant and also Maven on the site.