I would be very thankful for your help.
We use ant\ivy to manage our dependencies.
And we want to download a minimum quantity of modules: only those are needed for build.
I try to use a configuration intersection, but I got a not expecting ivy behavior.
https://ant.apache.org/ivy/history/latest-milestone/ivyfile/dependency.html
since 2.1 It is also possible to define dependencies on configurations intersection. A configuration intersection is defined using a '+' sign to separate the configuration (eg 'A+B' means the intersection of configuration 'A' and 'B'). In that case only artifacts and dependencies defined in both configurations in the dependency will be part of the master configuration defining the dependency on the configuration intersection.
Configuration intersections can also be used when specifying the confs to resolve.
build.xml file:
<project xmlns:ivy="antlib:org.apache.ivy.ant" xmlns:e="http://ant.apache.org/ivy/extra" name="test" default="get">
<target name="get" description="-->Downloads all artifacts and prepare it">
<ivy:resolve file="ivy.xml" conf="${conf}" />
</target>
</project>
ivy file:
<?xml version="1.0" encoding="utf-8"?>
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
<info module="test" organisation="main" revision="1.0"/>
<configurations>
<conf name="release" e:axis="download_type" />
<conf name="dev" e:axis="download_type" />
<conf name="client" e:axis="product"/>
<conf name="server" e:axis="product"/>
</configurations>
<dependencies>
<dependency org="main" name="zlib" branch="trunk" rev="1.2.8.+" conf="dev->*;client->*"/>
</dependencies>
</ivy-module>
Case 1:
ant -Dconf="dev+server"
Case 2:
ant -Dconf="server+dev"
I expect that the results are same in first and second case: zlib shouldn't be downloaded, because the intersection of "Dev" and "Server" confs is nothing.
But I get:
in case 1: Zlib is not dowloaded
in case 2: Zlib is dowloaded
Where is my misunderstanding?
Thanks
PS. Version of ivy is 2.4.0
Related
Within a Jenkins groovy script I'm trying to download a dependency using the following:
#Grab(group='myorg', module='SuiteCreator', version='1.16.1', conf='jar', transitive=false)
import myorg.myorgAPI
I have a /home/jenkins/.groovy/grapeConfig.xml file with the following:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-settings>
<settings defaultResolver="downloadGrapes"/>
<resolvers>
<chain name="downloadGrapes">
<sftp user="admin" userPassword="pw" host="ivy.myorg.com" name="myrepository" checkmodified="true">
<ivy pattern="/data/ivy/repo/[organisation]/[module]/[branch]/[revision]/ivy-[revision].xml"/>
<artifact pattern="/data/ivy/repo/[organisation]/[module]/[branch]/[revision]/[artifact]-[revision].[ext]"/>
</sftp>
</chain>
</resolvers>
</ivy-settings>
The ivy-1.16.1.xml of the Module I've trying to grab:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="1.0">
<info organisation="myorg" module="SuiteCreator" branch="master" revision="1.16.1" status="release" publication="20190417105814"/>
<configurations>
<conf name="jar" description="Distribution jar"/>
</configurations>
<publications>
<artifact name="myorg-suitecreator" type="jar" ext="jar" conf="jar"/>
</publications>
</ivy-module>
So I'm just trying to grab the artifact: myorg-suitecreator-1.16.1.jar.
When I run my groovy script in Jenkins I get the following error:
2019.07.09 18:06:15 org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed:
2019.07.09 18:06:15 General error during conversion: Error grabbing Grapes -- [download failed:
myorg#SuiteCreator#master;1.16.1!SuiteCreator.jar]
2019.07.09 18:06:15
2019.07.09 18:06:15 java.lang.RuntimeException: Error grabbing Grapes -- [download failed: myorg#SuiteCreator#master;1.16.1!SuiteCreator.jar]
2019.07.09 18:06:15 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
From the error it seems Grape is assuming the Ivy Artifact Name is the same as the Module name. The ivy-1.16.1.xml Artifact Name at: /ivy-module/publications/artifact/#name is defined as myorg-suitecreator However Grab appears to be attempt to download: SuiteCreator.jar.
The artifact pattern in grapeConfig.xml is:
<artifact pattern="/data/ivy/repo/[organisation]/[module]/[branch]/[revision]/[artifact]-[revision].[ext]"/>
And so the file I'm trying to grab is actually: /data/ivy/repo/myorg/SuiteCreator/1.16.1/myorg-suitecreator-1.16.1.jar
Does anyone have any suggestions on how to get this to work (or if Grab can download artifacts with different artifact name to the module name from Ivy?).
I gave up trying to use Grab to achieve this. I found another limitation of Grab is that it doesn't allow the specification of the branch of the artifact you wish to retrieve. I realise that not having releases on a master branch or a single release branch may not be best practice, but we do have this requirement in our dev environment.
Instead I simply used an Invoke Ant build step within Jenkins to retrieve my Ivy artifact. We use ANT already in our dev process so this was not difficult.
The ANT build.xml script is located in the same git repository as the Groovy script I wish to run. The retrieve-suite-creator target is simply an ivy-retrieve
<target name="retrieve-suite-creator" depends="clean, install-ivy">
<ivy:retrieve conf="suite-creator" type="jar" pattern="${build.dir}/[artifact].[ext]" log="${ivy.resolve.log}" settingsRef="${ivy.build.settings}"/>
</target>
Using my ivy.xml (again in the same repo as the groovy script):
<ivy-module version="1.0">
<info organisation="myorg" module="MyAutomation" status="integration" branch="${ivy.branch}"/>
<configurations>
<conf name="suite-creator" description="Configuration for Suite Creator"/>
</configurations>
<dependencies>
<dependency org="myorg" name="SuiteCreator" branch="mybranch" rev="1.16.1" conf="suite-creator->suite-creator" changing="true"/>
</dependencies>
</ivy-module>
I had to add the suite-creator ivy configuration to the SuiteCreator module's ivy.xml (in a separate SuiteCreator Git repo). I couldn't use the existing jar configuration as this also downloaded all the transitive dependencies which I didn't need.
<ivy-module version="1.0">
<info organisation="myorg" module="SuiteCreator" status="integration" branch="${ivy.branch}"/>
<configurations>
<!-- Build configurations -->
<conf name="build" description="Classes used in jar compilation"/>
<conf name="jar" description="Distribution jar"/>
<conf name="suite-creator" description="Just the myorg-suitecreator.jar"/>
</configurations>
<publications>
<artifact name="myorg-suitecreator" type="jar" ext="jar" conf="jar,suite-creator"/>
</publications>
<dependencies>
...
</dependencies>
</ivy-module>
Finally in my Jenkins job after the Invoke Ant build step, I then had an Execute Groovy Script build step, where I had to add the downloaded jar to my Class path.
I've read all the tutorials and examples, and still cannot publish a set of custom jars in my local Ivy repository.
Edit: Basically I want the same behavior as maven-install-plugin.
Here's my setup. I have an Ant task which produces the jars in a given folder. The folder name is not fixed but rather passed as a property in file. I want to get all the jars in this folder and install them in my local Ivy repo so that I can use them on a next step.
Here is my Ant from where I call the ivy:publish:
<project name="Install Ivy Dependencies" xmlns:ivy="antlib:org.apache.ivy.ant" basedir="." default="publish">
<loadproperties srcFile="path_to_folder.properties"/>
<property name="file_pattern" value="${path_to_folder}/[artifact].[ext]" />
<property name="pub_revision" value="1.0.0" />
<target name="resolve">
<ivy:configure file="ivysettings.xml" />
<ivy:resolve file="ivy.xml" />
</target>
<target name="retrieve-all" depends="resolve">
<ivy:retrieve pattern="${file_pattern}" conf="*" />
</target>
<target name="publish" depends="retrieve-all">
<ivy:publish resolver="local" organisation="myOrg" update="true" overwrite="true" pubrevision="${pub_revision}">
<artifacts pattern="${file_pattern}"/>
</ivy:publish>
</target>
</project>
Here's my ivysettings.xml:
<ivysettings>
<resolvers>
<filesystem name="local" local="true"/>
</resolvers>
</ivysettings>
And the ivy.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="myOrg" module="myModule" revision="1.0.0"/>
<publications>
<artifact name="my-custom-jar" ext="jar" type="jar"/>
<artifact name="my-custom-jar-source" ext="jar" type="source"/>
</publications>
</ivy-module>
The error that I am getting when I call the ant task is:
impossible to publish artifacts for myOrg#myModule;1.0.0: java.lang.IllegalStateException: impossible to publish myOrg#myModule;1.0.0!my-custom-jar.jar using local: no artifact pattern defined
I've managed to run my scenario and to resolve my issues using this tutorial There were two major issues in my code/integration.
First one is that you cannot tell Ivy to publish the artifacts in its repository without providing a path to it. I did this with the filesystem resolver:
<filesystem name="local" local="true" transactional="local">
<ivy pattern="${ivy.default.ivy.user.dir}/local/[module]/ivy-[revision].xml" />
<artifact pattern="${ivy.default.ivy.user.dir}/local/[module]/[artifact]-[revision].[ext]" />
</filesystem>
The stupid think about it is this should be build in. If you copy it as is, then everything works. If the config is different, or pointing to a different location - nothing works and you are not told why. I read tons of docs about Apache Ivy and it was nowhere mentioned that these patterns should point to the local Ivy repository. I thought these were the paths from where the jars should be taken. I actually complained about this, but the Ivy documentation is very confusing. Also I think the examples there are wrong. Who would like to publish the artifacts in their ivy.settings.dir. In my case this directory was in my repository!
There was a second issue. It is a smaller one and again very hard to see and fix. There's something wrong the revision param and again the documentation is messed up. If you specify one and the same string for the revision and pub revision the artifacts aren't publish without any explanation why. I fixed it by removing the revision from ivy.xml file.
Last, but not least, I didn't manage to run successfully the "thing" as Ant task, but with java -jar $IVY_JAR ... Maybe the issue was because of the versions, but I was too tired to try it with the fix.
P.S.#cantSleepNow thanks for the help.
You need to add artifact pattern to resolver in ivysettings.xml, something like (example from ivy documentation):
<ivysettings>
<resolvers>
<filesystem name="local" local="true">
<ivy pattern="${ivy.settings.dir}/1/[organisation]/[module]/ivys/ivy-[revision].xml"/>
<artifact pattern="${ivy.settings.dir}/1/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
</filesystem>
</resolvers>
</ivysettings>
I am dealing with a project using using ANT to build the source code into a EAR. The project over couple of years has grown to a mammoth size, more than fifty modules, and not surprisingly it takes 2 hours to build the source code.
The obvious decision that I made was to migrate this to a modular build using IVY for dependency management between sub-modules of the EAR, so I build only the modules that have changed and then package a new EAR every time. I am stuck trying to find out a good way of figuring out dependencies between these sub-modules. Note that 3rd party dependencies were easy to crack. I simply move all that is in the lib folder of these sub-modules to IVY. But, the former is where I am stuck.
This is what they do:
Copy source from all modules to a directory
Compile "everything" into a global.jar
Add this global.jar to the classpath
Build every individual module thereafter
Now how do I figure out that for example Module C depends on module A and B and so A and B should go as ivy dependency in the former? One way could be to add one module at a time in eclipse and try building, and then eleminate failures because of missing classes; but there has to be a better way than this, I cannot imagine spending the next one month of life figuring that out :)
First of all you need to create ivy repository. I don't know how to do it. Try to find it on ivy site. After that you need to create build.xml and ivy.xml file.
Example build.xml which builds some product which depends on module1
<project name="ivy.test" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="resolve">
<!-- this line tells ivy to use ${ivy.settings.file} where are ivy repositories; more info http://ant.apache.org/ivy/history/latest-milestone/settings.html -->
<ivy:configure file="${ivy.settings.file}"/>
<!-- resolve ivy dependencies from ivy.xml and put them in ivy cache dir -->
<ivy:resolve file="${build.dir}/ivy.xml" transitive="false" />
<!-- finally copy resolved dependencies to ${ivy.lib.dir} -->
<ivy:retrieve conf="default" pattern="${ivy.lib.dir}/[artifact].[ext]" />
</target>
</project>
Example ivy.xml used to resolve dependencies (will try to find module1 in one of ivy repository defined in ivysettings file)
<ivy-module version="2.0">
<info organisation="your.organisation" module="modulename">
<ivyauthor name="yourname" />
</info>
<configurations>
<conf name="default" description="default conf" />
</configurations>
<dependencies>
<dependency name="module1" org="your.organisation" rev="latest.release" conf="default->default" />
</dependencies>
</ivy-module>
Example ivy.xml used by module1 project (this project depends on module2 and module3; module1 is also published in repository)
<ivy-module version="2.0">
<info organisation="your.organisation" module="module1"> <!-- module name is used in dependency -->
<ivyauthor name="yourname"/>
</info>
<configurations>
<conf name="default" description="default configuration"/>
</configurations>
<publications>
<artifact name="module" ext="dll" type="dll" conf="default"/>
</publications>
<dependencies>
<dependency name="module2" org="your.organisation" rev="latest.release" conf="default->default" />
<dependency name="module3" org="your.organisation" rev="latest.release" conf="default->default" />
</dependencies>
</ivy-module>
I have the following ivy file:
<configurations defaultconfmapping="buildtime">
<conf name="buildtime" visibility="private" description="Libraries needed only for compilation" />
<conf name="runtime" description="Libraries only needed at runtime" />
<conf name="test" description="Libraries only needed for testing" />
</configurations>
<dependencies>
<dependency org="net.java.dev" name="jvyaml" rev="0.2.1" conf="runtime" />
<dependency org="org.apache.solr" name="solr-core" rev="3.6.0" conf="runtime" />
</dependencies>
and I have an ant retrieve task that looks like this:
<target name="retrieve-all" depends="resolve">
<ivy:retrieve pattern="lib/[conf]/[artifact]-[revision].[ext]" conf="*" />
</target>
The weird thing is, that all the solr dependencies download into lib/runtime as I'd expect, but the jvyaml module does not! It 'resolves', but will not download into the lib/runtime directory unless I change the dependency declaration to:
<dependency org="net.java.dev" name="jvyaml" rev="0.2.1" conf="runtime->master" />
What is this master configuration and why is it needed to pull the jvyaml jar, but not solr?
Thanks
I would suggest restructuring your configurations as follows:
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Libraries needed only for compilation" />
<conf name="runtime" description="Libraries only needed at runtime" extends="compile" />
<conf name="test" description="Libraries only needed for testing" extends="runtime" />
</configurations>
<dependencies>
<dependency org="net.java.dev" name="jvyaml" rev="0.2.1" conf="runtime->default" />
<dependency org="org.apache.solr" name="solr-core" rev="3.6.0" conf="runtime->default" />
</dependencies>
</ivy-module>
Important changes introduced:
Use the more standard "compile" configuration
Configuration inheritance using the "extends" attribute. Compile dependencies can then be automatically included in both the runtime and test configurations.
Use configuration mappings, for example: conf="runtime->default". This makes it obvious which local configuration is associated with which remote configuration.
Configuration mappings explained
Configurations are a powerful ivy feature. When ivy downloads Maven modules it performs an internal translation and assigns a standard set of configurations, listed in this answer:
How are maven scopes mapped to ivy configurations by ivy
When declaring a dependency it's a good idea to always make use of a configuration mapping, so that there is no doubt where the dependencies artifacts are assigned.
For example:
<dependency org="??" name="??" rev="??" conf="runtime->default" />
Here we're saying we want the remote module's default dependencies associated with our local runtime configuration.
In practice, there are only two remote configuration mappings you'll actually need:
default: The remote module's artifact and all it's runtime transitive dependencies
master: The remote module's artifact only (No transitive dependencies)
In conclusion, I think your problem was caused by the fact that the remote Maven module's "runtime" scope does not include Maven module's artifact, instead you were getting the non-existant transitive dependencies of the module jvyaml :-(
Some additional advice
I'd also suggest generating an ivy dependency management report, as follows:
<target name="init" description="Resolve dependencies and populate lib dir">
<ivy:resolve/>
<ivy:report todir="${build.dir}/ivy-report" graph="false"/>
<ivy:retrieve pattern="lib/[conf]/[artifact]-[revision].[ext]"/>
</target>
The report will help explain how each dependency ends up on different configurations. Also really useful for determining how transitive dependencies are being managed.
And finally, here's where the configuration inheritance pays off, creating ivy managed ANT classpaths:
<target name="init" description="Resolve dependencies and set classpaths">
<ivy:resolve/>
<ivy:report todir="${build.dir}/ivy-report" graph="false"/>
<ivy:cachepath pathid="compile.path" conf="compile"/>
<ivy:cachepath pathid="runtime.path" conf="runtime"/>
<ivy:cachepath pathid="test.path" conf="test"/>
</target>
Notice that the original solr-core is not retrieved either.
After your resolve, go the cache and check the ivy.xml files for both modules.
You will see that they publish their artifacts in conf=master only
<artifact name="jvyaml" type="jar" ext="jar" conf="master"/>
<artifact name="solr-core" type="jar" ext="jar" conf="master"/>
Which means, you have to do explicit configuration mapping to denote that your builtime configuration should evoke the 'master' configuration of your dependencies. (check configuration mapping).
HOWEVER, the dependencies of the solr-core, have the configuration mapping as you could see in the ivy.xml file:
<dependency org="org.apache.solr" name="solr-solrj" rev="3.6.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
I think it's the master(*) thingy.
What I usually do is in my own ivy.xml file when I declare dependencies I do the mapping:
<dependency org="net.java.dev" name="jvyaml" rev="0.2.1" conf="runtime->master" />
This one says the runtime be evoking the master configuration in the designated dependency.
You could do
conf="runtime,test->master"
as well
I have a fairly large ivy.xml containing a number of configurations which are the same for a number of projects.
I would like to break out this large repetitive section in to a common include file. Somehow I can't find any documentation describing that this can be done.
Anyone who has an idea whether this is doable?
EDIT: After some further thinking, I think this is not doable on purpose. An Ivy file is meant to be one cohesive unit and should contain no file based references, only references to other ivy modules...
You could create an ivy meta-module, which depends upon all of those common packages, and then have all your other projects resolve the common libraries through transitive dependency:
<?xml version="1.0"?>
<ivy-module version="2.0">
<info organisation="com.example" module="common-libs"/>
<configurations>
<conf name="runtime" transitive="true" visibility="public" />
<conf name="master" transitive="true" visibility="public" />
<conf name="compile" transitive="true" visibility="public" />
<conf name="default" transitive="true" visibility="public" extends="master" />
</configurations>
<dependencies>
<dependency org="oracle" name="ojdbc14_g" rev="10.2.0.3"
conf="compile->compile(*),master(*);runtime->runtime(*);master->master(*)"/>
<dependency org="tomcat" name="servlet-api" rev="6.0.16"
conf="compile->compile(*),master(*);runtime->runtime(*);master->master(*)"/>
<dependency org="junit" name="junit" rev="4.3"
conf="compile->compile(*),master(*);runtime->runtime(*);master->master(*)"/>
</dependencies>
</ivy-module>
And for a typical project:
<?xml version="1.0"?>
<ivy-module version="2.0">
<info organisation="com.example" module="myproject"/>
<configurations>
<conf name="runtime" transitive="true" visibility="public" />
<conf name="master" transitive="true" visibility="public" extends="runtime"/>
<conf name="compile" transitive="true" visibility="public" />
<conf name="default" transitive="true" visibility="public" extends="master" />
</configurations>
<dependencies>
<dependency org="com.example" name="common-libs" rev="latest.release"
conf="compile->compile(*),master(*);runtime->runtime(*);master->master(*)"/>
</dependencies>
</ivy-module>
Here I'm using the traditional configuration naming conventions from the POM->Ivy translations of the Maven resolver, though you could map the configuration names in any way that made sense to you. I tend to use the ivy:install task to copy Maven modules into our Ivy repository, so I use the default ivy.xmls for the most part. If you're using IvyRoundup, you'll primarily want to map the "default" configuration transitively.
If you are using ivy from ant and you are running Ant 1.6 or later, you could use the <import> task to include build file fragments within your ant build file. The referenced files have to be complete Ant build files, though:
<?xml version="1.0"?>
<project name="my-project" default="usage" basedir=".">
<target name="setup">
...
</target>
<import file="./common.xml"/>
...
</project>
You could also use standard XML syntax from within your ivy.xml or build.xml file:
<?xml version="1.0"?>
<!DOCTYPE project [
<!ENTITY common SYSTEM "common.xml">
]>
<project name="my-project" default="usage" basedir=".">
<target name="setup">
...
</target>
&common;
...
</project>
This will literally include the contents of common.xml where you've placed the &common; entity.
(The filename common.xml in this example is resolved relative to the containing XML file by the XML parser. You may also use an absolute file: protocol URI.)