I have a problem with a build where I have to resolve non-standard artifacts through Apache Ivy.
Problem:
I have dependencies on two artifacts (a.jar and a-lib.jar).
The two dependencies come only as part of a single installer (a_installer.jar).
The installer can be downloaded, the embedded artifacts themselves not.
It's possible to manipulate the installer to unpack the needed dependencies.
Requirements:
I have to resolve/download the artifacts during the build (I cannot keep the installer or the extracted artifacts with my code).
I cannot use a repository to store the extracted artifacts.
Subclassing/Extending Ivy/whatever is perfectly fine.
Has anyone solved a similar problem, or some helpful information to share?
Or maybe I'm approaching the problem in the wrong way? From what I found so far on the web, people seem to use Ivy just to download files and post-process them manually (with Ant/whatever) after the fact, and not actually resolving more complicated dependencies within Ivy.
Thanks
PS: I don't care whether the installer is also put into the ivy download cache, but I would like to download the installer only one time (and not for both dependencies).
The problem with a call to "ivy:retrieve" is that you need to also add an "artifact" tag in your ivy.xml (complete with URL) in order to retrieve a dependency not found in a Maven respository...
I don't like this for two reasons
The ivy.xml should just declare your dependencies, not their locations.
Need additonal custom logic in the build.xml to handle the 3rd party package
Ideally it should be your repository settings that decide how to download the various jars, that is why I like the packager resolver. Even if the library I want is not in Maven, I can configure ivy to handle it.
The following is an example of turning the jreleaseinfo project into an ivy dependency (hosted in sourceforge, I couldn't find it in Maven)
ivy.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="ivy_packager"/>
<dependencies>
<dependency org="ch.oscg" name="jreleaseinfo" rev="1.3.0"/>
</dependencies>
</ivy-module>
Declare two resolvers. Default is Maven2, the other is a packager configured to look locally for instructions. (See also the Ivy Roundup project)
ivysettings.xml
<ivysettings>
<settings defaultResolver="maven2"/>
<resolvers>
<ibiblio name="maven2" m2compatible="true"/>
<packager name="repackage" buildRoot="${user.home}/.ivy2/packager/build" resourceCache="${user.home}/.ivy2/packager/cache">
<ivy pattern="file:///${basedir}/repository/[organisation]/[module]/[revision]/ivy.xml"/>
<artifact pattern="file:///${basedir}/repository/[organisation]/[module]/[revision]/packager.xml"/>
</packager>
</resolvers>
<modules>
<module organisation="ch.oscg" name="jreleaseinfo" resolver="repackage"/>
</modules>
</ivysettings>
The magic is containing in the "packager" file. At resolve time this will be used to generate an ANT script that both downloads and extracts the required jars.
(No need to put this logic into your build.xml)
repository/ch.oscg/jreleaseinfo/1.3.0/packager.xml
<packager-module version="1.0">
<property name="name" value="${ivy.packager.module}"/>
<property name="version" value="${ivy.packager.revision}"/>
<property name="zipname" value="${name}-${version}"/>
<resource dest="archive" url="http://sourceforge.net/projects/jreleaseinfo/files/jreleaseinfo/jreleaseinfo%201.3.0/jreleaseinfo-1.3.0.zip/download" sha1="9386d92758e627d04c2480b820731fd538b13a3f" type="zip"/>
<build>
<move file="archive/${zipname}/${zipname}.jar" tofile="artifacts/jars/${name}.jar"/>
</build>
</packager-module>
To reduce the number of files I omitted the module's ivy.xml. This appears to be optional unless you want to declare it's licence and other attributes that should be present in a public repository.
I think this is very straightforward: 'ivy:retrieve' a_installer and then unzip a.j and a-lib into your lib directory (or wherever you want it). This should be easily to do with ant?
I have to wonder if there is some complication you haven't mentioned that prevents you from doing this.
Related
I have a huge dependency which exports a number of dependencies. I wish to restrict my retriev to couple of them. The pattern is [artifact]-[revision].[ext].
How do i specify this in ivy:retrieve task call
Configurations in ivy is the mechanism for controlling groups of dependencies within ivy.
Once these configurations have been setup in your ivy file it becomes simple to retrieve them within your ANT build as follows:
<ivy:retrieve pattern="lib/[artifact].[ext]" conf="my_custom_conf"/>
Perhaps you could supply some more details of what you want to achieve and someone can demonstrate how to setup a configuration for this purpose. (I'd also recommend searching the Stackoverflow ivy tag, for other examples)
Update
If an ivy module publishes more than one artifact it's possible to restrict the dependency in your ivy file as follows:
ivy.xml
<configurations>
..
<conf name="archives" description="Configuration containing only archive files"/>
</configurations>
<dependencies>
..
<dependency org="acme" name="foo" rev="2.0" conf="archives->default">
<artifact name="a1" type="tar"/>
<artifact name="an" type="zip"/>
</dependency>
</dependencies>
Alternatively..
Look into the remote modules's ivy.xml. There may already be a configuration setup for these files, in which case it becomes a lot simpler (because it's been pre-setup)
<dependency org="acme" name="foo" rev="2.0" conf="archives->remotearchives"/>
The "conf" part of the dependency is mapping the remote configuration onto your local one.
So, let's assume that I have an already installed SVN and installed ANT / Ivy locally.
I want to have the "shared" part of the ivy config point to some kind of share on a server. How would I need to set this up?
I know I have to dig through the ivy jar and pull out the ivysettings file and modify shared repositories.
So let's assume that I have a server on my intranet at MyServer.intranet.net and my team's folder was under /path/to/NetAdmin (thus the full path would be MyServer.intranet.net/path/to/NetAdmin ) How would I get this set up as a team repository for shared libraries? Would I just specify it and when I package the projects it writes the dependencies there?
Thanks
Here what I did:
I created a Subversion project called ivy.dir.
In this ivy.dir project, I have the latest ivy.jar.
In the ivy.dir, I have the ivysettings.xml setup for our environment. For example, we use a local Artifactory Maven repository for our own jars. The ivysettings.xml in the ivy.dir project points to that.
I created a file called ivy.tasks.xml. This is an Ant build file.
The ivy.tasks.xml looks like this:
<project name="Ivy.Tasks"
xmlns:ivy="http://ant.apache.org/ivy"
xmlns:jacoco="antlib:org.jacoco.ant">
<property environment="env"/>
<!-- Add Ivy Tasks -->
<taskdef uri="http://ant.apache.org/ivy"
resource="org/apache/ivy/ant/antlib.xml">
<classpath>
<fileset dir="${ivy.dir}">
<include name="ivy*.jar"/>
</fileset>
</classpath>
</taskdef>
<ivy:settings file="${ivy.dir}/ivysettings.xml"/>
</project>
Notice that I have my own Ivy settings, thank you. I didn't have to munge up the one in the ivy.jar (although I could have since everyone will use my ivy.jar file!). My ivysettings.xml looks like this:
<ivysettings>
<!-- I'll explain this part below -->
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<!-- Just the standard stuff you find in the `ivysettings.xml in the ivy.jar -->
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/> <!-- This one is different -->
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
The big change is the ivysetting-public.xml file:
<ivysettings>
<resolvers>
<ibiblio name="public"
m2compatible="true"
checkmodified="true"
root="http://repos.vegicorp.com/artifactory/libs-release" />
</resolvers>
</ivysettings>
It's pointing to my local Maven repository -- my Artifactory server.
Now, for a developer to use Ivy, all they have to do is:
In the root of their project in Subversion, add a svn:external. This svn:external will be used to bring my ivy.dir project into their Subversion project.
In the build.xml
Add an Ivy namespace definition to their build.xml in the <project> definition.
Set the property ivy.dir to `${basedir}/ivy.dir.
Use the <import> task to import ${ivy.dir}/ivy.tasks.xml into their build.xml file.
Something like this:
<project name="post-a-matic" default="package" basedir="."
xmlns:ivy="http://ant.apache.org/ivy">
<property name="ivy.dir" value="${basedir}/ivy.dir"/>
<import file="${ivy.dir}/ivy.tasks.xml"/>
<!-- A whole bundle of properties are set -->
<target name="clean">
<delete dir="${target.dir}"/>
<ivy:cleancache/> <!-- Look: They have access to Ivy! -->
</target>
<target name="-resolve">
<ivy:resolve/>
</target>
<target name="compile"
depends="-resolve">
<ivy:cachpath
pathid="main.classpath"
conf="compile,provided"/>
<!-- Boy that's easy! -->
<javac srcdir="${main.srcdir}"
destdir="${main.destdir}"
classpathref="main.classpath"/>
</target>
<!-- On and on -->
This solves a lot of problems:
You can update the ivy.settings and everyone will have the updated settings. This ended up being very important to us because we use Jenkins and I wanted Jenkins to clean the ivy cache on each build. Whoops! That cleans out the ivy cache on builds that are being executed at the same time! I solved the problem by changing the ivysettings.xml file to define a different Ivy cache for each Jenkins build executor. One the Jenkins server, you have Ivy caches called $HOME/.ivy2/cache-0, $HOME/.ivy2/cache-1, etc. Each executor can delete it's own Ivy cache without affecting the others. Users, meanwhile will just have $HOME/.ivy2/cache-0.
You also can update Ivy when a new jar comes out. You update your Ivy jar file, and everyone gets the lated.
Big one of course is that Ivy installs itself when a project is checked out.
And an extra special bonus: You could use your ivy.dir and ivy.tasks.xml file to install other tasks. For example, each of our projects must run itself through Findbugs, PMD, CPD (part of the PMD project, Checkstyle, and use JaCoCo. for test coverage.
Each one of these projects consist of a jar file, and a <taskdef> to pull the task definitions into Ant. And, how do you use these tasks too? They're not defined in the standard Ant model. Developers don't know how to use them.
I've added these jars into my ivy.dir project, and installed all of those task definitions into my ivy.tasks.xml file. I also defined easy to use <macrodef> for most of these tasks, so it's easy for the developers to use them. In fact, I've even included the old Ant-Contrib tasks just for fun.
Now, once you add ivy.dir into your project, you have all of these extra tasks, and you have nothing to install on your machine.
You don't need to change the ivy jar. Just create a filesystem resolver in an ivysettings file and publish to this. Here's an example:
good ivy tutorial for local repository?
You'll find that ivy is very flexible and can support pretty much any mechanism for hosting files.
Personally, I'd consider installing a Maven repository manager like Nexus or Artifactory and use this to host both your builds dependencies and build outputs. In the long run it's a lot easier, especially if you're doing Java development.
I want to use the url resolver to download depends. The depends are JS, or XML files. So, I used:
<url name="urlresolver">
<artifact pattern="http://[organisation]/[module]-[revision].[ext]" />
</url>
And
<ivy:retrieve pattern="${build}/[module]-[revision].[ext]"/>
the file is saved in .jar extension.
It's worth digging around the following ivy docs:
Terminology
Main Concepts
Best Practices
The first problem is that your url resolver is not configured to read ivy files for the remote modules (ignoring the first recommendation, in the ivy best practices, to use an ivy file with each module). Without module meta-data ivy will assume you're attempting to download JAR files.
A second problem is that you don't appear to be using an ivy repository to store your files. The following dependency declaration:
<dependency org="yourorg" name="module1" rev="9.1"/>
would be translated into the following URL, using your current settings:
http://yourorg/module1-9.1.jar
The "org" field is designed to specify the organisational unit publishing the module, not the server hostname.
I suspect that you're not really interested in building a repository of files and just want to persuade ivy to download and cache the files? In that case I'd recommend reading the following answer which is using extra attributes on the dependency artifacts to do something similar:
Resolving XSD's using Ivy
Example
ivy.xml
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
..
..
<dependency org="yourorg" name="yourmodule1" rev="9.1">
<artifact name="file1" e:hostname="www.server1.com" type="xml"/>
<artifact name="file2" e:hostname="www.server1.com" type="xml"/>
</dependency>
<dependency org="yourorg" name="yourmodule2" rev="9.1">
<artifact name="file3" e:hostname="www.server2.com" type="xml"/>
<artifact name="file4" e:hostname="www.server2.com" type="xml"/>
</dependency>
..
Note:
Each dependency declares the type of each artifact, plus the additional "hostname" attribute.
If the remote modules had ivy.xml, the publications section would alternatively store this artifact information.
Extra artifacts demonstrate ivy's power in enabling any sort of custom attribute meta-data.
ivysettings.xml
..
<url name="urlresolver">
<artifact pattern="http://[hostname]/files/[organisation]/[module]-[revision].[ext]" />
</url>
..
Demonstrates how the resolver makes use of both the standard attributes and the custom "hostname".
Can anybody give me a hint on how to use nested artifact (and artifacts) elements for publishing modules use the ivy ant task. Unfortunately the official documentation does not specify how to use the attributes.
http://ant.apache.org/ivy/history/latest-milestone/use/publish.html
Maybe there is some documentation or some examples that I could not find?
I am aware of the artifact element in ivy.xml files. this is NOT what this question is about.
Does this answer to this question help?
Issues using ivy:publish task
The nested "artifact" elements in the publish task are used to identify the location(s) of the artifacts you've specified as to be published by your module.
So, for example, let's say your module publishes two files:
<ivy-module version="2.0">
<info organisation="someorganisation" module="myapp"/>
<publications>
<artifact name="myapp" type="jar"/>
<artifact name="license" type="txt"/>
</publications>
..
Your publish task might need to source these files from two different locations within the build workspace:
<ivy:publish resolver="${publish.resolver}" pubrevision="${publish.revision}" status="${publish.status}">
<artifacts pattern="${build.dir}/[artifact].[ext]"/>
<artifacts pattern="${src.dir}/licenses/[organisation]/[artifact].[ext]"/>
</ivy:publish>
I'm using Ivy to manage the dependencies on my project.
So far, I've specified a dependency on Hibernate and servlet-api. However, the hibernate jar itself has a lot of dependencies that aren't really needed, such as jaas and jacc.
This becomes a show-stopper because jaas and jaac are Sun libraries and therefore their licenses forbid to place them in the Maven repos, so Ivy can't find them there.
How do I make Ivy download Hibernate but not these two ?
As a bonus, if I actually needed those and downloaded their Jars from Sun, in which folder in my machine would Ivy look for them ?
Another option for not downloading any dependencies is to disable them with the transitive attribute. So if you wanted hibernate-core, but none of its dependencies, you could do this:
<dependencies>
<dependency org="org.hibernate" name="hibernate-core"
rev="3.3.1.GA" conf='..'
transitive="false" />
</dependencies>
How do I make Ivy download Hibernate but not these two?
Ivy does this using what it calls "configurations." Your ivy.xml that represents Hibernate will need to provide different configurations to represent different use-cases for hibernate. (There is obviously some use of hibernate that does require jaas and jacc, but apparently you don't make use of that case.)
Here is the documentation on configurations. If you want to provide the ivy.xml you are using for hibernate, I can provide pointers on building configurations that will remove the specific libraries you want removed.
If I actually needed those and downloaded their Jars from Sun, in which folder in my machine would Ivy look for them?
The "directories" that ivy looks in for ivy files and artifacts are specified by the list of resolvers you are using. The list of resolvers is specified in the ivy settings file (usually named ivysettings.xml.) Typically, these aren't local directories, but remote URLs. There is; however, a local-file resolver type that will work for this.
If you do this, you will need to provide both ivy files and the artifacts (jars), each with file-names that match the resolvers patterns. Details on that are in the documentation.
Here is an example local-file resolver from an ivy settings file:
<filesystem name="myfiles" checkconsistency="false" checksums="" transactional="false">
<ivy pattern="/data/repo/[organisation]/[module]-[revision].ivy.xml"/>
<artifact pattern="/data/repo/[organisation]/[module]-[revision].[ext]"/>
</filesystem>
Also note that you will need to point your ivy tasks to the correct resolver. You can do this with the resolver attribute on the ant tasks, or with the defaultResolver attribute on the settings element in the ivy settings file.
Here is the documentation on resolvers.
EDIT: The OP found a less-time intensive workaround for his specific original problem. The "exclude" child-tag of the dependency tag did the job for him:
<dependencies>
<dependency org="org.hibernate" name="hibernate-core" rev="3.3.1.GA" conf='..'>
<exclude name='jaas' />
<exclude name='jacc' />
</dependency>
</dependencies>
Browsing the web and blogs, I found the following ivy-settings to work at grabbing jaas/jacc and hibernate
<ivysettings>
<settings defaultResolver="chained" checkUpToDate="true" />
<resolvers>
<chain name="chained">
<url name="com.springsource.repository.bundles.release">
<ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<url name="com.springsource.repository.bundles.external">
<ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<ibiblio name="ibiblio" m2compatible="true"/>
<ibiblio name="jboss" root="http://repository.jboss.org/maven2/" m2compatible="true"/>
<ibiblio name="java-net-maven1" root="http://download.java.net/maven/1" pattern="${java.net.maven.pattern}" m2compatible="false"/>
<ibiblio name="java-net-maven2" root="http://download.java.net/maven/2/" m2compatible="true"/>
<ibiblio name="compass" m2compatible="true" root="http://repo.compass-project.org" />
</chain>
</resolvers>
The jboss ibibilio resolver is what did the trick at grabbing JAAS/JAAC
My ivy.xml then can then pull it in with
<ivy-module version="2.0">
<info organisation="foo" module="Bar"/>
<dependencies>
<dependency org="com.h2database" name="h2" rev="1.2+"/>
<dependency org="org.hibernate" name="hibernate-annotations" rev="3.4.0.GA"/>
</dependencies>
To answer your second sub-question literally, which nobody has done so far, "in which folder in my machine would Ivy look for JARs?" That depends. Assuming you haven't changed the location in ivysettings.xml or another configuration file: for JAAS,
this would be: (user home)/.ivy2/cache/javax.security/jaas/jars. If Ivy already unsuccessfully tried to find JAAS in the Maven Central or other repo's, that directory tree should already exist for the most part, and all you need to do is create the "jars" directory and place jaas-1.0.01.jar in it. Ivy will no longer complain about the missing dependency in its next invocation.
EDIT: Then again, see the discussion below to see considerations to not do it like this.
((user home) is C:/Users/(username) on Windows 7).