I'm trying to diagnose a failure in my ant file when it runs a Worklight task. I have a small ant file containing a simple reference to a Worklight task. This works correctly.
<target name="rawBuildApplication" >
<app-builder
applicationFolder="${applicationSource}"
environments="mobilewebapp"
nativeProjectPrefix="xxx"
outputFolder="${outputFolder}"/>
</target>
However when I invoke this ant file from a build control ant file (actually from the RTC build system) I get a failure (below), showing worklight failing to find an apache Java class. I assume there's some simple environmental difference, perhaps a classpath. It might help to figure it out if I knew where Worklight loaded the apache commons from. Right now I can't see anything in my environment in the case that works that points any apache Jar.
myAntFile.xml:146: java.lang.NoSuchMethodError: org.apache.commons.io.FileUtils.deleteQuietly(Ljava/io/File;)Z
at com.worklight.builder.util.BuilderUtils.<clinit>(BuilderUtils.java:672)
at com.worklight.builder.config.UserBuildConfiguration$Builder.<init>(UserBuildConfiguration.java:203)
at com.worklight.ant.builders.ApplicationBuilderTask.createBuildConfiguration(ApplicationBuilderTask.java:149)
at com.worklight.ant.builders.ApplicationBuilderTask.execute(ApplicationBuilderTask.java:80)
Edited: the cause is the use of -lib to add the RTC toolkit directory, exactly why this clashes and how to work around yet to be determined
Usually means you have version of the commons jar in your classpath, and its overriding the one packaged in the worklight-ant.jar. the apache commons files are inside the worklight-ant.jar file
Additional info from djna: I can confirm that when adding the Rational Team Concert (RTC) 3.0 toolkit to the ant classpath, either explicitly with -lib, or when selecting that option in the RTC Build definition some conflicting commons jars are added to the classpath. Worklight packages the classes it needs in its jar, but the -lib folder seems to take precedence.
My workaround is to replace the conflicting jars with later ones. I used these jars
commons-io-2.4.jar
commons-codec-1.8.jar
httpclient-4.2.5.jar
httpcore-4.2.4.jar
httpmime-4.2.5.jar
I guess the other alternative is to upgrade to a newer RTC, but in our environment that's not currently possible.
Related
We have a build job where the console output shows many strange messages like below, so we cannot load a full log and have to remove the -X option in the build configuration to get rid of them. I think it happened after I upgraded the Jenkins version.
Any idea of what might be causing this?
[DEBUG] http-outgoing-0 >> "j[0x5]q4/J[0x18]^di[0x86][0xbf]C_[0xd6]G[0x1d]
[0xd4][0x7][0xf3][0xc7][0x14][0xdf][0x8d][0xe1][0x13][0xd8]$y|#[0x1e][0xbf]
[0xe6][0x81]<[0xca][0x6][0xa1]~j[0x81]3[0xbc][0x98][0xd1][0x83][0xa7]
[0xc5]8[0xfa]>[0xd9]edZ[0xb2][0xc][0xe0][0x5][0xab][0xf3][0x1a]M[0xb3][0xe7]
[0x1][0xf4][\n]"
[DEBUG] http-outgoing-0 >> "[0xcd][0x9d][0x86]Zjp[0xb4][0x8d][0x87]
[0x8f]cn[0xe7][0xab]oL.[0xb2]}[0x86][0xf8]D[0x87][0xba][0x9d][0xcc]j[0x15]
[0xa4][0xe6]![0x9f]_BBC[0xbf]j[0xab]Rl[0x10][0x92][0xc5])[0xb2][0xc5]i[0xc2]
I encountered this exact problem right around the same time (April 2018), and found that it was being triggered by Artifactory plugin 2.15.0 (and later). For over a year, I had left that plugin downgraded in order to avoid having DEBUG logging in my build logs. Although this worked for me (until I was forced to upgrade last week due to an incompatibility with new versions of Artifactory), it is possible that a different plugin or jar file in your classpath is causing the problem in your case.
After spending an entire weekend troubleshooting this problem, I finally found the underlying cause in my build environment.
The important points are:
This is a problem with the classpath of the build process (e.g., Ant).
This is not a configuration problem with Jenkins.
This cannot be fixed via the project configuration.
The trigger (a Jenkins or plugin update) is not necessarily the underlying cause of the DEBUG logs.
When this problem is triggered, there may be multiple causes, which makes this problem very difficult to troubleshoot.
The Dormant Problem
In my case, the underlying cause of the DEBUG logging was that I had cobertura-2.1.1.jar in my test dependencies, which contains a logback.xml file and also brings logback-classic.jar in as a dependency (widely regarded as a mistake, see Issue 2, Issue 14, Issue 36). The logback.xml file, when found in the classpath, overrides any other logback settings in Jenkins (and the project being built). However, since logback was not the logging framework selected by Apache Commons Logging (JCL), this log setting had never been triggered.
The Trigger
Upgrading the Artifactory plugin from 2.14.0 to 2.15.0 switched its logging from: commons-logging-1.1.1.jar (log4j-1.2.17.jar) to: jcl-over-slf4j-1.7.25.jar (slf4j-api-1.7.25.jar). FYI, log4j 1.x uses a default root logging level of DEBUG, while log4j 2.x uses a global logging level of ERROR, which may be an entirely different source of DEBUG logging (but not in my case). My build environment (ant) was supplying log4j 2.10.0 and logback on the classpath, which served only to confound my testing by producing inconsistent results depending on which plugins were running within the build process.
When using Artifactory plugin v2.15.0+, the logging framework switches to logback, which gives the logback.xml file in cobertura-2.1.1.jar permission to set the root log level to DEBUG, forcing all of the subsequent parts of the build process to record at the DEBUG level. This includes wire-logging for Apache Commons HttpClient, which produces http-outgoing-0 (serialized hexadecimal and interleaved Base64 encoded content from every HTTP/S message - as you show in your question). Even for a small project, recording a single PUT of a JAR file in this way will increase both the build time and the size of your build logs by several orders of magnitude (this is what I was experiencing in my build environment), which can easily cripple your entire Jenkins server. This is a huge problem, and as you can see from the Cobertura GitHub issues above, even though it is an easy fix, no steps have been taken to produce a fixed version in four years.
The Fix
To fix this, I had to make several changes on my Jenkins server:
Replace logback-classic.jar and logback-core.jar with slf4j-simple-1.7.26.jar in my .ant/lib folder (this is the classpath that Ant uses when building and testing my projects). This change prevents the use of logback at all in Ant (therefore any logback.xml file in the classpath becomes irrelevant), while still allowing your build to perform logging over the SLF4J API (via slf4j-simple).
Remove any dependencies on redundant logging versions (e.g., don't include both commons-logging-1.1.1 and commons-logging-1.2 in the classpath). This is especially important if using log4j, where version 1.1 defaulted to DEBUG, and version 1.2 defaulted to ERROR. You never know which underlying framework JCL will choose, so you want to give it as few options as possible.
Finally, in order for the test environment to match the Ant environment, I adjusted the dependencies of all of my projects to specifically exclude logback-classic (I use Ivy for dependency resolution, so maven or gradle will have a different syntax):
<dependency org="net.sourceforge.cobertura" name="cobertura" rev="latest.release" conf="test->default">
<exclude org="org.apache.ant" />
<exclude name="jaxen" />
<exclude name="jetty" />
<exclude name="jetty-util" />
<exclude name="servlet-api-2.5" />
<exclude name="logback-classic" /> <!-- IMPORTANT -->
</dependency>
For reference, my broken .ant/lib folder contained these logging-related jar files (2 possible sources of DEBUG logs):
commons-logging-1.1.1.jar (redundant JCL version, not certain which was used)
commons-logging-1.2.jar
slf4j-api-1.7.21.jar (logging framework used by new artifactory plugin)
logback-classic-1.0.13.jar (logging framework included from cobertura-2.1.1)
logback-core-1.0.13.jar
while my fixed .ant/lib folder contains the following logging jars:
commons-logging-1.2.jar (now the only available JCL version)
slf4j-api-1.7.26.jar (now the only available logging framework)
slf4j-simple-1.7.26.jar (now the only SLF4J implementation)
In my fixed Ant classpath, commons-logging-1.2 can only select the SLF4J API (or JUL), and only a single SLF4J implementation is available (slf4j-simple).
TL;DR
For three years, Cobertura 2.1.1 was telling logback to "DEBUG All the Things!", but nobody was listening until a new version of the Artifactory plugin changed the JCL version and brought along an SLF4J implementation that allowed logback to be selected as the "best available" logging framework. When the JCL was introduced to logback, logback took Cobertura's advice and threw a party in my build logs. This can be prevented by removing logback from the environment and supplying a well-behaved logging framework for the JCL and SLF4J API (such as slf4j-simple).
I want to open Nutch 2.1 source file (http://www.eu.apache.org/dist/nutch/2.1/) at Intellij IDEA. Here is an explanation of how to open it at Eclipse: http://wiki.apache.org/nutch/RunNutchInEclipse
However I am not familiar with Ant (I use Maven) and when I open that source file many classes are not known by Intellij.
i.e.:
org.apache.hadoop.mapreduce.JobContext
org.apache.gora.mapreduce.GoraMapper
How can I add them to library or what should I do?
I finally figure out how to do it. Now our team can dev nutch in IntellIJ
The process we do
Get nutch source from apache.org
wget http://www.eu.apache.org/dist/nutch/2.3/apache-nutch-2.3-src.tar.gz
Import nutch source in intellij
Get Dependencies by Ant
Run ant runtime
Run ant test
Import dependencies into Intellij
File > Project Structures
Library > Ivy
Click to Plus button
Select all libraries in apache-nutch-2.3/build/lib
Now we have a project with nutch source and all dependencies
I think, you should use ant build for nutch project. Because, pom.xml is problematical for nutch. If you want to use it anyway maven, you check maven dependency in pom.xml .
I think, the problem can be solved with the following:
You create new project via idea and add nutch source. Idea is not
supported ivy ant project. You can install ivy plugin for idea, I
suppose, Idea12 does not support it.
or
You can create ant project for nutch via eclipse and then save
project. Then open the project on idea via eclipse classpath.
or
You can configure classpath. If you use ant build, you should add jars : File->ProjectStructure as follows:
Create a library
Attach Files
Then, if you use ant build, select jars from NUTCH_HOME/build/lib/* after build.
If you use maven build, select jars from ~/.m2/* (MAVEN_REPO)
Although Intellij supports running ant scripts directly through the IDE via the ant plugin, the Intellij editor will not figure out the classpath using your build.xml file by itself.This will result in the editor showing the code littered with errors.
Unfortunately the only solution I found to this is to add all the jars manually.You can get all the jars needed by building the ant project using the ant jar command.
So these are the steps you need to follow :->
1.Run ant jar command in the root of your program
2.Copy all the jars created inside the build folder(they will not be directly inside build folder.I had to go from build->ivy->lib->{project name})
3.Paste these jars to a new folder created somewhere outside the directory
4.Go to File->Project Structure->Libraries. Press the + button.
5.Select all the copied jars and press ok to any prompts which appear.
6.Press finish and wait for indexing to complete.Any dependency errors in the editor should resolve now.
Note: Since you are manually adding jars, you would need to keep them in sync if you change your build.xml file for any dependency change.
It seems that dependencies are managed using Ivy (see the ivy folder in the sources archives), so you could try to install the Ivy plugin, which would allow you to fix your classpath issues.
My problem is the following:
I would like to use the propertyregex task in ant. The project I am working on is built on various different servers and I don't want to configure (install the ant-nodeps.jar) every server. The source needs to include everything, that is not installed on the system by default.
So now I would need to add the ant-nodeps.jar to the ant classpath from within the build file. Does somebody know how to do that?
Cheers,
Robert
The propertyregex task is part of ant-contrib and can be installed as part of your build using Apache ivy
Checkout the following example, which demonstrates how to download and use the "for" task (also from the ant-contrib project):
Problems getting my ANT builds to work after OS upgrade
The one downside is that ivy does not come pre-packaged with ANT, so the following answer has a tip on how to bootstrap your ANT builds. Once ivy is started it can be used to pull down everything else your build needs.
Ivy fails to resolve a dependency, unable to find cause
Update
While I understand you requirement to have no change on the target platforms, it's a very difficult problem to solve if you must also match several old versions of the build software. I have found incompatibilities between the latest ANT and 5 year old versions like 1.7 (ANT 1.6.5 is now 8 years old....)
What I do is install a very limited number of ANT versions on my Jenkins slave nodes. Build jobs can then only choose from these and then use ivy to download all other 3rd party software dependencies (This setup emulates how you'd manage a set of Maven projects).
I suspect you're using ANT to run your deployments? If that is the case I would suggest switching to something like Groovy, which can be deployed as a single jar file and can pull down dependencies on the fly, using Grape.
I am trying to use the Ant <scp> task in my buildfile. It gives me an error every time I run the target:
cannot be found: com/jcraft/jsch/UserInfo
I have read some posts that talk about having this issue with Ant 1.7. Some of the posts are talking about removing the ant-jsch.jar files from the ant_home lib. Unfortunately I can't do that as the ant_home is common across lot of other projects and I can't remove or update any jar files from the lib directory.
Is there a way that I can solve this issue purely using Ant configuration?
ant-jsch.jar contains ssh related tasks developed by Apache Ant team, while jsch.jar contains the jsch library which is the dependency of those ssh tasks.
So do a little check: check if jsch.jar is in your $ANT_HOME/lib and also check if the jar contains UserInfo.class.
If one of above happens, download the latest version of jsch.jar and put it in $ANT_HOME/lib.
I am currently using Ant 1.8.4 with a jsch.jar downloaded last year without any problem.
Update:
Check this page first:
http://ant.apache.org/manual/install.html#optionalTasks
Please note that <scp> and other ssh tasks (the ant-jsch.jar) are included in the Ant binary distribution by default. You don't need to write <taskdef> to use <scp> because it is included in Ant's lib directory by default. I recommend you to put jsch.jar in that directory with the following points:
These ssh tasks don't support classpath def, that is to say, you can't define a <classpath> pointing to jsch.jar and then reference to it in an ssh task.
Also you can put jsch.jar in any directory included in your system's CLASSPATH environment variable, but it is not a good practice.
We usually put a task and its dependency in the same directory.
What approach has worked well for you combining IVY + msbuild?
Our goal is to integrate IVY into the C#/C++ build process for dependency resolution and publishing. We have tried adding it to custom tasks at the beginning and end of the build and we have tried wrapping the msbuild calls with ant+ apache-ant-dotnet.
Other options might be gradle, buildr, rake.
What do you use?
Thanks
Peter
Most build technologies can use libraries found in a local directory. I'd suggest using the command-line ivy program to populate this, at the start of your build:
java -jar ivy.jar -ivy ivy.xml -settings ivysettings.xml -retrieve "lib/[conf]/[artifact].[ext]"
Your dependencies are listed in a standard ivy file called ivy.xml. The protocol, location and layout of your remote repository is described in ivysettings.xml
The advantage of this approach (as opposed to switching to Gradle, etc) is that you're not trying to replace your existing build tool. Ivy is solely concerned with managing dependencies.
My team has been using Ivy for .NET for a couple of years very successfully. I know several more that give it a vote of confidence.
Use it standalone. Wrap calls into msbuild tasks. No need to use Ant integration.