I'm talking about ivy 2, not some newer version.
Switching off the cache affects performance. Switching it off loses one of the big benefits of dependency management, because you'd force each build to download everything....
Having said that, over time the cache sometimes become stale, especially in situations where the remote modules might have changing dependencies (Can occur during development). To counter this problem I always include an extra build target that purges my ivy cache:
<target name="clean-all" depends="clean">
<ivy:cleancache />
</target>
This gives you the best of both worlds. Disabling the cache is accomplished by first running a purge as follows:
ant clean-all build
Related
We have a build job where the console output shows many strange messages like below, so we cannot load a full log and have to remove the -X option in the build configuration to get rid of them. I think it happened after I upgraded the Jenkins version.
Any idea of what might be causing this?
[DEBUG] http-outgoing-0 >> "j[0x5]q4/J[0x18]^di[0x86][0xbf]C_[0xd6]G[0x1d]
[0xd4][0x7][0xf3][0xc7][0x14][0xdf][0x8d][0xe1][0x13][0xd8]$y|#[0x1e][0xbf]
[0xe6][0x81]<[0xca][0x6][0xa1]~j[0x81]3[0xbc][0x98][0xd1][0x83][0xa7]
[0xc5]8[0xfa]>[0xd9]edZ[0xb2][0xc][0xe0][0x5][0xab][0xf3][0x1a]M[0xb3][0xe7]
[0x1][0xf4][\n]"
[DEBUG] http-outgoing-0 >> "[0xcd][0x9d][0x86]Zjp[0xb4][0x8d][0x87]
[0x8f]cn[0xe7][0xab]oL.[0xb2]}[0x86][0xf8]D[0x87][0xba][0x9d][0xcc]j[0x15]
[0xa4][0xe6]![0x9f]_BBC[0xbf]j[0xab]Rl[0x10][0x92][0xc5])[0xb2][0xc5]i[0xc2]
I encountered this exact problem right around the same time (April 2018), and found that it was being triggered by Artifactory plugin 2.15.0 (and later). For over a year, I had left that plugin downgraded in order to avoid having DEBUG logging in my build logs. Although this worked for me (until I was forced to upgrade last week due to an incompatibility with new versions of Artifactory), it is possible that a different plugin or jar file in your classpath is causing the problem in your case.
After spending an entire weekend troubleshooting this problem, I finally found the underlying cause in my build environment.
The important points are:
This is a problem with the classpath of the build process (e.g., Ant).
This is not a configuration problem with Jenkins.
This cannot be fixed via the project configuration.
The trigger (a Jenkins or plugin update) is not necessarily the underlying cause of the DEBUG logs.
When this problem is triggered, there may be multiple causes, which makes this problem very difficult to troubleshoot.
The Dormant Problem
In my case, the underlying cause of the DEBUG logging was that I had cobertura-2.1.1.jar in my test dependencies, which contains a logback.xml file and also brings logback-classic.jar in as a dependency (widely regarded as a mistake, see Issue 2, Issue 14, Issue 36). The logback.xml file, when found in the classpath, overrides any other logback settings in Jenkins (and the project being built). However, since logback was not the logging framework selected by Apache Commons Logging (JCL), this log setting had never been triggered.
The Trigger
Upgrading the Artifactory plugin from 2.14.0 to 2.15.0 switched its logging from: commons-logging-1.1.1.jar (log4j-1.2.17.jar) to: jcl-over-slf4j-1.7.25.jar (slf4j-api-1.7.25.jar). FYI, log4j 1.x uses a default root logging level of DEBUG, while log4j 2.x uses a global logging level of ERROR, which may be an entirely different source of DEBUG logging (but not in my case). My build environment (ant) was supplying log4j 2.10.0 and logback on the classpath, which served only to confound my testing by producing inconsistent results depending on which plugins were running within the build process.
When using Artifactory plugin v2.15.0+, the logging framework switches to logback, which gives the logback.xml file in cobertura-2.1.1.jar permission to set the root log level to DEBUG, forcing all of the subsequent parts of the build process to record at the DEBUG level. This includes wire-logging for Apache Commons HttpClient, which produces http-outgoing-0 (serialized hexadecimal and interleaved Base64 encoded content from every HTTP/S message - as you show in your question). Even for a small project, recording a single PUT of a JAR file in this way will increase both the build time and the size of your build logs by several orders of magnitude (this is what I was experiencing in my build environment), which can easily cripple your entire Jenkins server. This is a huge problem, and as you can see from the Cobertura GitHub issues above, even though it is an easy fix, no steps have been taken to produce a fixed version in four years.
The Fix
To fix this, I had to make several changes on my Jenkins server:
Replace logback-classic.jar and logback-core.jar with slf4j-simple-1.7.26.jar in my .ant/lib folder (this is the classpath that Ant uses when building and testing my projects). This change prevents the use of logback at all in Ant (therefore any logback.xml file in the classpath becomes irrelevant), while still allowing your build to perform logging over the SLF4J API (via slf4j-simple).
Remove any dependencies on redundant logging versions (e.g., don't include both commons-logging-1.1.1 and commons-logging-1.2 in the classpath). This is especially important if using log4j, where version 1.1 defaulted to DEBUG, and version 1.2 defaulted to ERROR. You never know which underlying framework JCL will choose, so you want to give it as few options as possible.
Finally, in order for the test environment to match the Ant environment, I adjusted the dependencies of all of my projects to specifically exclude logback-classic (I use Ivy for dependency resolution, so maven or gradle will have a different syntax):
<dependency org="net.sourceforge.cobertura" name="cobertura" rev="latest.release" conf="test->default">
<exclude org="org.apache.ant" />
<exclude name="jaxen" />
<exclude name="jetty" />
<exclude name="jetty-util" />
<exclude name="servlet-api-2.5" />
<exclude name="logback-classic" /> <!-- IMPORTANT -->
</dependency>
For reference, my broken .ant/lib folder contained these logging-related jar files (2 possible sources of DEBUG logs):
commons-logging-1.1.1.jar (redundant JCL version, not certain which was used)
commons-logging-1.2.jar
slf4j-api-1.7.21.jar (logging framework used by new artifactory plugin)
logback-classic-1.0.13.jar (logging framework included from cobertura-2.1.1)
logback-core-1.0.13.jar
while my fixed .ant/lib folder contains the following logging jars:
commons-logging-1.2.jar (now the only available JCL version)
slf4j-api-1.7.26.jar (now the only available logging framework)
slf4j-simple-1.7.26.jar (now the only SLF4J implementation)
In my fixed Ant classpath, commons-logging-1.2 can only select the SLF4J API (or JUL), and only a single SLF4J implementation is available (slf4j-simple).
TL;DR
For three years, Cobertura 2.1.1 was telling logback to "DEBUG All the Things!", but nobody was listening until a new version of the Artifactory plugin changed the JCL version and brought along an SLF4J implementation that allowed logback to be selected as the "best available" logging framework. When the JCL was introduced to logback, logback took Cobertura's advice and threw a party in my build logs. This can be prevented by removing logback from the environment and supplying a well-behaved logging framework for the JCL and SLF4J API (such as slf4j-simple).
I want to unleash the power of parallelism by running some Ivy-related Ant tasks on local Bamboo agents. Our Bamboo machine has plenties of CPU horse power and RAM.
If I split my build task into parallel jobs, each that will produce a different artifact based on the result of ivy-retrieve, I have solved my problem in the theory.
In practice, unfortunately, if two Ant tasks happen, for some reason, to run simultaneously on the same machine and the same organization-artifact, they will clash and one gets an XML error.
I don't have the exact error message with me because 1) the problem is randomic to reproduce and 2) I have already done lots of work to put all the jobs into a sequential one. But I have a clear idea of what is happening.
When Ant runs an ivy-retrieve, code below, it will use local user's cache directory, which happens to be /home/bamboo/.ivy2/cache. There I can find lots of resolved-[org]-[artifact]-[version].xml files (each is a different build version of my project). The problem occurs when I want to run the ivy-retrieve task twice, one for example for compile configuration and another for runtime. The two XMLs will clash and Ivy will report a SAX error in reading one of the files, because it looks like it's being written at the moment.
If I run the job on remote agents I expect no problem, but hey I already have 5 local agents and Bamboo won't fire remote agents if the locals are free.
Unfortunately all of my jobs, independent each other, require a different ivy-retrieve. Currently I run them in sequence.
The question is
Is it possible to tell Ivy, running on Bamboo agent, to use a temporary unique cache directory for its work on dependencies.xml files rather than to use global cache? Or at most to synchronize access to files?
The second option would have parallel Ant process read&write the cached dependencies.xml file mutually exclusively. So what they read will be always a consistent file (being the same exact file, I don't care if a process overwrites another)
Ivy has 2 caches - repository cache and resolution cache. The second one is overwritten each resolution, and should never be used by multiple processes at the same time.
Set an environment variable pointing to a temporary directory in
your bamboo agent.
Create a separate ivysettings.xml file for your project.
User an environment variable in project's ivysettings.xml to setup a cache directory.
Here is an example of ivysettings.xml:
<ivysettings>
<properties environment="env" />
<caches resolutionCacheDir="${env.TEMP_RESOLUTION_CACHE}" />
<settings defaultResolver="local" />
<statuses default="development">
<status name="release" integration="false"/>
<status name="integration" integration="true"/>
<status name="development" integration="true"/>
</statuses>
...
</ivysettings>
Or you can give a try to a lock-strategy. I haven't tried it.
We are going to be using Ivy with Ant, and we'll have Jenkins do our builds. I originally thought that having Jenkins do a <ivy:cleancache/> before running a build would be a good idea. (It would be part of the mandatory "clean" target).
However, I now see that <ivy:cleancache> doesn't simply clean up stuff from <ivy:cachepath>, but really removes the entire $HOME/.ivy/cache directory.
My concern is that if Jenkins does a <ivy:cleancache> on all builds before they start, it will interfere with other builds that Jenkins could be executing.
Is doing an <ivy:cleancache> a good idea, especially if a single user might be doing multiple builds at the same time?
In fact, what happens when you do a <ivy:cachepath pathid="compile.path"/> in multiple projects? Does this also affect something like Jenkins? Will Jenkins become confused if multiple builds are building the compile.cachepath at the same time?
In my opinion running the ivy cleancache task with every build is overkill and does away of one of the main benefits of using ivy, intelligent downloading of 3rd party dependencies.
Having said that as stated in the following related Maven question, all caches can become dirty and should be periodically purged:
When is it safe to delete the local Maven repository?
Couple of recommendations:
Use dedicated Jenkins job(s) to purge ivy cache
My first recommendation is to create a periodic Jenkins job that calls the following clean-all target in your build:
<target name="clean-all" depends="clean">
<ivy:cleancache/>
</target>
This ensures that Jenkins decides when the cache is purged and you can schedule it to happen outside of the normal build times (for example 2am on 1st of every month)
Isolate each project by using multiple caches
My second recommendation increases the isolation between your project builds. Configure each project to have it's own private cache, using the caches directive. in you ivy settings file.
Here's what I've decided to do:
I've modified my ivysettings.xml file to have the following:
<ivysettings>
<properties environment="env." override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
This does two things:
It defines the Ivy local cache as $HOME/.ivy/cache-$EXECUTOR_NUMBER where $EXECUTOR_NUMBER is the Jenkins executor. This means that each executor gets their own Ivy cache. Thus, if Jenkins is executing more than a one job at a time, each job will be picked up with a different executor, so it will have its own cache. If a job wants to clean the cache, it can go right ahead.
I've defined the resolve cache to ${basedir}/target/ivy.cache. This gives each job its own resolver cache which is quite small. But, this way ivy resolving doesn't interfere with other jobs if Jenkins is building multiple revisions of the same Ivy project.
The only drawback is that the user's default cache directory is called $HOME/.ivy/cache-$env.EXECUTOR_NUMBER which is not a pretty site. I'd love to make it a more reasonable $HOME/.ivy/cache-0, but I haven't figured that out. However, it doesn't really affect anything at this point.
Now, a developer has a single Ivy cache which contains all of the jars they've downloaded. This way, jars can be shared between projects which speeds up things for developers.
Meanwhile, Jenkins can clean the Ivy cache as often as it is configured. This could be done for each job, or once per day, or per month. However, since the cache is done per executor, I won't have an issue of the cache being cleaned while another job (which would be running on another executor) is depending upon that cache.
This should solve all of the particular issues. The only thing I'd like to do is figure out how to set a default EXECUTOR_NUMBER variable if one isn't already set. I've tried various things like this:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<properties environment="env." override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
But, to no avail. I've trued changing the override parameters on both the <property> and <properties> file all different ways, but it doesn't quite do what I want.
Just something I have been doing a lot to solve the last problem you have.
You can add this to the properties of the Jenkins Ant Build Steps
another.less.obtrusive.name=${EXECUTOR_NUMBER}
and add to ivysettings.xml.
So it will be "0" for everybody, except for Jenkins because it will inject this property to ANT.
Something on the main question:
On Jenkins, I always start new. CI builds should be robust, thorough. Fast is a welcomed by-product, but not a motivation.
I seem to be getting run-time errors in my project when using javac for incremental builds. Is this type of workflow supported? For example, if A.java depends on B.java, and B.java is modified; will javac recompile A.java because its dependency changed?
Right now I'm using a javac ant build-task for compiling:
<javac destdir="${classes.dir}"
srcdir="${src.dir}"
source="${javac.version}"
debug="${javac.debug}"
deprecation="${javac.deprecation}"
includeantruntime="build.sysclasspath=last">
<classpath refid="compile.classpath" />
<classpath refid="junit.classpath" />
</javac>
Since you're using ant, check out the depend task.
The javac command line compiler will compile every source file given on the command line, and additionally everything on which these depend, if they don't have newer class files.
The ant javac task tries to be a bit smarter, to avoid compiling always everything - it recompiles only those files which have changed (i.e. are newer than their respective class files). This does not pay attention to the case that maybe the dependency of some class changed, and thus other classes need to be recompiled too.
In my current project, I simply do ant clean whenever I have problems on tests (and of course before any production deployment), which deletes all class files. But as vanza said, there is the depend task whose task is to find and delete all classes who depend on your changed classes - run this before your javac task and you should be good.
It depends on what has changed in B.java. If nothing changed that affected how the class is presented to A, then javac doesn't need to recompile A.java for the changes to take effect.
That said, if you are seeing behavior where you believe old code is being loaded and run, I'd be more suspicious of the deployment/packaging process than the compilation process. YMMV.
Do you have some predefined set of targets which all build.xml files you create contain?
For example, a lot of ant manuals suggest the following list of targets:
init
clean
compile
build
jar
test
javadoc
dist
deploy
webapp
What is the most large build file you met in your life? How many targets did it have and what are they? How often do you need more than predefined set of targets?
The goal is to develop some conventions to have standard buildfile template for any project having the notion of the maven-like approach in mind (when a lot of work happens under the cover, convention over configuration). Also it would be great if you know the place where one can find collection of different buildfiles to choose or to get inspired from.
I also use these targets in all ant files
init
clean
compile
build
test
javadoc
The build targets always creates the artefact, no matter whether it is a jar or war or whateveer.
You should also include structural things in your conventions like a shared repository for all libraries (versioned by some VCS).
Another thing would be to define properties for your jar versions i.g.:
lib.commons-collections=commons-collections-2.1.jar
lib.commons-io=commons-io-1.4.jar
which are referenced in all ant files, common.jar is a place where artifacts are placed in case other projects depend on them.
<path id="local-cp">
<pathelement path="${dir.common.jar}/${lib.shared}" />
<pathelement path="${dir.lib}/${lib.commons-logging}" />
<pathelement path="${dir.lib}/${lib.commons-io}" />
...
For deployment I use another set of ant files deploy_component-name.xml
After years with ant I would recommend to keep the number of targets limited, sometimes you may have a few more steps for code generation etc.
To see how others handle bigger projects you could download the source distribution of an application server to examine how they do this job.
int
build
jar
deploy
package
clean
test