I'm trying to include some jars required by my topology in the jar that I deploy to Storm. When I create the jar file using Ant with the manifestclasspath task, I get a manifest that looks like this:
Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: ../build/lib/kafka-clients-0.8.2.1.jar ../build/lib/kafka_
2.11-0.8.2.1.jar ../build/lib/log4j-1.2.16.jar ../build/lib/metrics-c
ore-2.2.0.jar ../build/lib/scala-library-2.11.5.jar ../build/lib/stor
m-kafka-0.9.5.jar ../build/lib/zookeeper-3.4.6.jar
The paths in the Class-Path entry look wrong. The topology deploys successfully, but when I send data to Kafka, I get a ClassNotFoundException: storm.kafka.KafkaSpout in the Storm logs.
I modified the manifest to look like this:
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: . lib/kafka-clients-0.8.2.1.jar lib/kafka_2.11-0.8.2.1.jar
lib/log4j-1.2.16.jar lib/metrics-core-2.2.0.jar lib/scala-library-2.1
1.5.jar lib/storm-kafka-0.9.5.jar lib/zookeeper-3.4.6.jar
This also deploys successfully but gives the same error when I try to pass messages.
How can I create a jar file that contains these dependencies and runs properly on Storm? I don't want to have to add the jars to the Storm lib directory, but that's the only configuration I've found that works.
Apparently Matthias is correct and Storm cannot handle nested jar files. The fix is to unpack the jar files and include the class files in the topology jar. This took two changes to my Ant build.xml file. First, put the dependencies in a staging directory and build a single jar from them:
<target name="staging">
<copy todir="${build.staging}">
<fileset dir="${kafka.lib}">
<include name="kafka-clients-0.8.2.1.jar"/>
<include name="kafka_2.11-0.8.2.1.jar"/>
<include name="log4j-1.2.16.jar"/>
<include name="metrics-core-2.2.0.jar"/>
<include name="scala-library-2.11.5.jar"/>
</fileset>
<fileset dir="${storm.kafka.lib}">
<include name="storm-kafka-0.9.5.jar"/>
</fileset>
<fileset dir="${zookeeper.lib}">
<include name="zookeeper-3.4.6.jar"/>
</fileset>
</copy>
<jar jarfile="${staging.jar}">
<zipgroupfileset dir="${build.staging}" includes="**/*.jar"/>
</jar>
</target>
Then pull those .class files into the topology jar, excluding the manifests:
<jar basedir="${build.root}"
jarfile="${topology.jar}">
<manifest>
<attribute name="Main-Class" value="${topology.fullname}"/>
</manifest>
<zipfileset src="${staging.jar}" excludes="META-INF/**/*"/>
</jar>
This jar successfully deploys to Storm.
Related
I am using ftp task from ant. For ftp to work, I need commons-net.jar as dependency at
ANT_HOME/lib folder.
As best practice, I am following the folder structure to keep all external folders under customized External Jars folder. Is there a way to keep commons-net.jar at customized folder instead of
ANT_HOME/lib folder?
Another option is to place plugin jars in the "$HOME/.ant/lib" directory.
You can automate the install of your dependent jars as follows:
<project name="demo" default="build">
<available classname="org.apache.commons.net.ftp.FTP" property="ftp.installed"/>
<target name="init" unless="ftp.installed">
<mkdir dir="${user.home}/.ant/lib"/>
<get dest="${user.home}/.ant/lib/commons-net.jar" src="http://search.maven.org/remotecontent?filepath=commons-net/commons-net/3.3/commons-net-3.3.jar"/>
<get dest="${user.home}/.ant/lib/oro.jar" src="http://search.maven.org/remotecontent?filepath=oro/oro/2.0.8/oro-2.0.8.jar"/>
<fail message="FTP task installed. Run ANT again"/>
</target>
<target name="build" depends="init">
<ftp server="ftp.apache.org" userid="anonymous" password="me#myorg.com">
<fileset dir="htdocs/manual"/>
</ftp>
</target>
</project>
I have been trying to use Ant to compile and ready a project for distribution. I have encountered several problems along the way that I have been finally able to solve but the solution leaves me very unsatisfied. First, let me explain the set-up of the project and its dependencies.
I have a project, lets call it Primary which depends on a couple of libraries such as the fantastic Guava. It also depends on another project of mine, lets call it Secondary. The Secondary project also features some dependencies, for example, JDOM2. I have referenced the Jar I build with Ant in Primary.
Let me give you the interesting bits of the build.xml so you can get a picture of what I am doing:
<project name="Primary" default="all" basedir=".">
<property name='build' location='dist' />
<property name='application.version' value='1.0'/>
<property name='application.name' value='Primary'/>
<property name='distribution' value='${application.name}-${application.version}'/>
<path id='compile.classpath'>
<fileset dir='libs'>
<include name='*.jar'/>
</fileset>
</path>
<target name='compile' description='Compile source files.'>
<javac includeantruntime="false" srcdir="src" destdir="bin">
<classpath refid='compile.classpath'/>
</javac>
<target>
<target name='jar' description='Create a jar file for distribution.' depends="compile">
<jar destfile='${build}/${distribution}.jar'>
<fileset dir="bin"/>
<zipgroupfileset dir="libs" includes="*.jar"/>
</jar>
</target>
The Secodnary project's build.xml is nearly identical except that it features a manifest as it needs to run:
<target name='jar' description='Create a jar file for distribution.' depends="compile">
<jar destfile='${dist}/${distribution}.jar' basedir="${build}" >
<fileset dir="${build}"/>
<zipgroupfileset dir="libs" includes="*.jar"/>
<manifest>
<attribute name="Main-Class" value="lu.tudor.ssi.kiss.climate.ClimateChange"/>
</manifest>
</jar>
</target>
After I got it working, trying for many hours to not include that dependencies as class files but as Jars, I don't have the time or insight to go back and try to figure out what I did wrong. Furthermore, I believe that including these libraries as class files is bad practice as it could give rise to licensing issues while not packaging them and merely including them in a directory along the build Jar would most probably not (And if it would you could choose not to distribute them yourself).
I think my inability to correctly assemble the class path, I always received NoClassDefFoundError for classes or libraries in the Primary project when launching Second's Jar, is that I am not very experienced with Ant. Would I require to specify a class path for both projects? Specifying the class path as . should have allowed me to simply add all dependencies to the same folder as Secondary's Jar, should it not?
You may use the MANIFEST.MF "Class-Path: " to cross-reference your jars.
If they are all in the same directory this will probably work as follows (using it in both projects!):
<target name='jar' description='Create a jar file for distribution.' depends="compile">
<pathconvert property="manifest.classpath" pathsep=" ">
<path refid="compile.classpath" />
<flattenmapper />
</pathconvert>
<jar destfile='${build}/${distribution}.jar'>
<fileset dir="bin"/>
<manifest>
<attribute name="Class-Path" value="${manifest.classpath}"/>
</manifest>
</jar>
</target>
This way you can tell the java runtime environment that your jar needs others to work, expecting them to be in the same directory as the jar you are trying to run.
As a result your primary.jar should have secondary.jar in it's classpath and secondary.jar should have guava.jar in it's classpath.
Another way to create the string may be ants manifestclasspath task (https://ant.apache.org/manual/Tasks/manifestclasspath.html) that can handle subdirectories.
If you are goin to use more and more libraries, you may want to have a closer look at ivy or even maven.
I'm trying to generate a executable jar for my JavaFX application with Ant, and the difference between my jar and the one generated by the JavaFX Packager is that the latter include classes from com.javafx.main package.
How can I tell in my Ant script to include these classes in the jar as well ?
The ant file you're using must have the special fx-tasks to deploy the jar, and not the ant built-in jar tasks. Here's a sample ant target for generating a jar with JavaFX:
<target name="jar" depends="compile">
<echo>Creating the main jar file</echo>
<mkdir dir="${distro.dir}" />
<fx:jar destfile="${distro.dir}/main.jar" verbose="true">
<fx:platform javafx="2.1+" j2se="7.0"/>
<fx:application mainClass="${main.class}"/>
<!-- What to include into result jar file?
Everything in the build tree-->
<fileset dir="${classes.dir}"/>
<!-- Define what auxilary resources are needed
These files will go into the manifest file,
where the classpath is defined -->
<fx:resources>
<fx:fileset dir="${distro.dir}" includes="main.jar"/>
<fx:fileset dir="." includes="${lib.dir}/**" type="jar"/>
<fx:fileset dir="." includes="."/>
</fx:resources>
<!-- Make some updates to the Manifest file -->
<manifest>
<attribute name="Implementation-Vendor" value="${app.vendor}"/>
<attribute name="Implementation-Title" value="${app.name}"/>
<attribute name="Implementation-Version" value="1.0"/>
</manifest>
</fx:jar>
</target>
Note, that you must have a taskdef defined somewhere in the script:
<taskdef resource="com/sun/javafx/tools/ant/antlib.xml"
uri="javafx:com.sun.javafx.tools.ant"
classpath="${javafx.sdk.path}/lib/ant-javafx.jar"/>
and the project tag must have the fx xmlns reference:
<project name = "MyProject" default ="compile" xmlns:fx="javafx:com.sun.javafx.tools.ant">
The generated jar file should now include the classes from javafx.main and the manifest will include them as an entry point into the application. More info:
http://docs.oracle.com/javafx/2/deployment/packaging.htm
I am executing lot of scp and sshexec and other remote commands from an ant build script. These commands don't work if jsch.jar isn't in the ant lib directory. To make it work, I copied the JAR into the ant lib directory, but this is not a good solution, as anyone else wanting to run the script would have to do the same thing. To run the ant target from Teamcity, we will have to explicitly set the path of the lib file.
Is there a way I can specify the path of the JAR in the ant build XML itself?
Thanks all for your answers. I am managed to get it work with classloader task. This is what I did.
<project basedir="." >
<property environment="env"/>
<taskdef resource="net/jtools/classloadertask/antlib.xml">
<classpath>
<fileset dir="${basedir}/lib" includes="ant-classloader*.jar"/>
</classpath>
</taskdef>
<!--Add JSCH jar to the classpath-->
<classloader loader="system">
<classpath>
<fileset dir="${basedir}/lib" includes="jsch*.jar"/>
</classpath>
</classloader>
<target name="Test">
<scp todir="user1:pass1#server1:/tmp" trust="true" >
<fileset dir="dir1">
<include name="test.txt" />
</fileset>
</scp>
</target>
</project>
As you can see here, I didn't have to give any dependant target for my "Test" target, it just works. It uses classloader, which appends jsch.jar to the system classloader.
One possible work around would be to use the -lib command line option to tell ant where to look for additional jars. Perhaps you could create a wrapper script that calls ant with this option set.
Another way would be to move the ant-jsch.jar file (this is the jar that comes with ant that defines the tasks, not the jsch.jar file you need to download separately) out of your ant lib directory, and create a taskdef for your ssh task separate to the built in one, then set the classpath for this task to the jsch.jar and the ant-jsch.jar:
<taskdef name="sshexec"
classname="org.apache.tools.ant.taskdefs.optional.ssh.SSHExec">
<classpath>
<pathelement location="jsch-0.1.44.jar"/>
<pathelement location="ant-jsch.jar" />
</classpath>
</taskdef>
I'm not sure this will help you though, since it also involves making changes to the lib directory.
As far as I'm aware, it's not currently possible to specify the extra jars required for the built in tasks in the build file itself in general. There are some special cases, like junit for instance.
To ensure your build is more cross platform I'd suggest using dependency management. The ivy plug-in can automatically install the version of your build's plugin at build-time.
This approach means the last jar you'll ever need to install into your ANT lib is ivy-2.2.0.jar :-)
First declare your project's dependencies in the file ivy.xml
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="anttask" description="Jars implementing ANT tasks"/>
</configurations>
<dependencies>
<dependency org="com.jcraft" name="jsch" rev="0.1.42" conf="anttask->default"/>
</dependencies>
</ivy-module>
Within your build.xml run ivy and use it to populate a custom classpath based on the ivy configuration:
<target name='init' description='Resolve project dependencies and set classpaths'>
<ivy:resolve/>
<ivy:cachepath pathid="anttask.path" conf="anttask"/>
</target>
Finally, elsewhere in your build declare your ANT tasks using the class path now automatically populated by ivy.
<target name='dosomething' depends="init">
<taskdef name="sshexec"
classname="org.apache.tools.ant.taskdefs.optional.ssh.SSHExec"
classpathref="anttask.path"/>
..
..
</target>
This approach works for all ANT plug-ins, most of which are available in the central Maven repository. The second benefit is that it's easy to upgrade the plug-in versions across all builds.
I would like to unjar multiple JAR files and then rebuild into one JAR using an ant build script. Is this possible?
Yes, it's possible with ant. A jar file is basically a zip with a special manifest file. So to unjar, we need to unzip the jars. Ant includes an unzip task.
To unzip/unjar all the jar files in your project:
<target name="unjar_dependencies" depends="clean">
<unzip dest="${build.dir}">
<fileset dir="${lib.dir}">
<include name="**/*.jar" />
</fileset>
</unzip>
</target>
Obviously you need to declare ${build.dir} and ${lib.dir} first. The line <include name="**/*.jar" /> tells ant to include all files that end up with the jar extension, you can tweak that include to suit your needs.
To pack everything into a jar, you use the jar task:
<target name="make_jar" depends="compile, unjar_dependencies">
<jar basedir="${build.dir}"
destfile="${dist.dir}/${project_name}.jar">
<manifest>
<attribute name="Main-Class" value="${mainclass}" />
</manifest>
<fileset dir="${build.dir}">
<include name="**/*.class" />
</fileset>
<fileset dir="${src.dir}">
<include name="applicationContext.xml" />
<include name="log4j.properties" />
</fileset>
</jar>
</target>
In this example, we include different filesets. In one fileset we are including all compiled classes. In another fileset we include two config files that this particular project depends upon.
Yes it is !
You have two possibilities :
Espen answer :
One possible solution that creates one
jar file from all the jar files in a
given directory:
<target name="dependencies.jar">
<jar destfile="WebContent/dependencies.jar">
<zipgroupfileset dir="lib/default/" includes="*.jar"
excludes="*.properties" />
</jar>
</target>
This is useful if you don't need to exclude content that are in some jars (like for example some properties configuration file that might override yours, etc). Here the excludes properties is filtering out files from the dir property.
Use zipfileset
The other solution is to use the zipfileset tag where the excludes property this time will filter out content from the jar to be merged.
<jar destfile="your_final_jar.jar" filesetmanifest="mergewithoutmain">
<manifest>
<attribute name="Main-Class" value="main.class"/>
<attribute name="Class-Path" value="."/>
</manifest>
<zipfileset
excludes="META-INF/*.SF"
src="/path/to/first/jar/to/include.jar"/>
</jar>
Of course you can combine the two tags (zipfileset and zipgroupfileset) inside the same jar tag to get the best of the two.
Yes, it's possible.
One possible solution that creates one jar file from all the jar files in a given directory:
<target name="dependencies.jar">
<jar destfile="WebContent/dependencies.jar">
<zipgroupfileset dir="lib/default/" includes="*.jar"
excludes="*.properties" />
</jar>
</target>
There is also a project devoted to repackage jars called JarJar. You can use it to repackage mutiple Jars into one. Depending on your requirements, you can even rename classes to prevent version conflicts.
From their getting started page:
In this example we include classes from jaxen.jar and add a rule that changes any class name starting with "org.jaxen" to start with "org.example.jaxen" instead (in our imaginary world we control the example.org domain):
<target name="jar" depends="compile">
<taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="lib/jarjar.jar"/>
<jarjar jarfile="dist/example.jar">
<fileset dir="build/main"/>
<zipfileset src="lib/jaxen.jar"/>
<rule pattern="org.jaxen.**" result="org.example.#1"/>
</jarjar>
</target>