I want to unleash the power of parallelism by running some Ivy-related Ant tasks on local Bamboo agents. Our Bamboo machine has plenties of CPU horse power and RAM.
If I split my build task into parallel jobs, each that will produce a different artifact based on the result of ivy-retrieve, I have solved my problem in the theory.
In practice, unfortunately, if two Ant tasks happen, for some reason, to run simultaneously on the same machine and the same organization-artifact, they will clash and one gets an XML error.
I don't have the exact error message with me because 1) the problem is randomic to reproduce and 2) I have already done lots of work to put all the jobs into a sequential one. But I have a clear idea of what is happening.
When Ant runs an ivy-retrieve, code below, it will use local user's cache directory, which happens to be /home/bamboo/.ivy2/cache. There I can find lots of resolved-[org]-[artifact]-[version].xml files (each is a different build version of my project). The problem occurs when I want to run the ivy-retrieve task twice, one for example for compile configuration and another for runtime. The two XMLs will clash and Ivy will report a SAX error in reading one of the files, because it looks like it's being written at the moment.
If I run the job on remote agents I expect no problem, but hey I already have 5 local agents and Bamboo won't fire remote agents if the locals are free.
Unfortunately all of my jobs, independent each other, require a different ivy-retrieve. Currently I run them in sequence.
The question is
Is it possible to tell Ivy, running on Bamboo agent, to use a temporary unique cache directory for its work on dependencies.xml files rather than to use global cache? Or at most to synchronize access to files?
The second option would have parallel Ant process read&write the cached dependencies.xml file mutually exclusively. So what they read will be always a consistent file (being the same exact file, I don't care if a process overwrites another)
Ivy has 2 caches - repository cache and resolution cache. The second one is overwritten each resolution, and should never be used by multiple processes at the same time.
Set an environment variable pointing to a temporary directory in
your bamboo agent.
Create a separate ivysettings.xml file for your project.
User an environment variable in project's ivysettings.xml to setup a cache directory.
Here is an example of ivysettings.xml:
<ivysettings>
<properties environment="env" />
<caches resolutionCacheDir="${env.TEMP_RESOLUTION_CACHE}" />
<settings defaultResolver="local" />
<statuses default="development">
<status name="release" integration="false"/>
<status name="integration" integration="true"/>
<status name="development" integration="true"/>
</statuses>
...
</ivysettings>
Or you can give a try to a lock-strategy. I haven't tried it.
Related
Background
I have a Jenkins process where I build a Windows service and then use MSBuild to compile an .msi from a WiX project. The service itself does different processing based on what configuration it is running as, with this configuration determined by a number assigned to the service via the app.config. So, for example, FooService1 processes certain records, FooService2 other records, etc. The different versions are running the exact same compiled dlls, the only difference being the server number that is assigned in the app.config.
I will be running these different versions of the same service on the same machine, and would like to have the build process for the different variations of the service automated via Jenkins. I am using a powershell script to assign the service number in the app.config, and then passing in that service number via MSBuild and using a WiX variable to modify the name of the Windows service and the name of the folder the service operates out of.
Question/Issue
So the first MSBuild of the .wixproj goes fine. But then the next MSBuild reports that it is
Skipping target "Compile" because all output files are up-to-date with respect to the input files
Is there a way to "fake" MSBuild into thinking that it should compile the .wixproj again? My end goal is that if I want 3 versions of the service, Jenkins will compile the solution once and then build the WiX project 3 times, resulting in a /bin that contains FooService1.msi, FooService2.msi, and FooService3.msi. Each installer will contain the same dlls, but the app.config would be different.
Build the service and MSI once, transform the config at install time.
Either bundle multiple config files in the MSI and overwrite at deployment time like in this answer. Or pass in the desired value at install time and use a property and XML transform to update the file. Building multiple MSI's for a single config value is overkill.
To pass the value at install time you can do something like this
Say you have a value in an app settings section
<appSettings>
<add key="ServiceType" value="1" />
</appSettings>
Add the wixUtil library
<Wix
xmlns="http://schemas.microsoft.com/wix/2006/wi"
xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
In the Product element in your Wix add a property element
<Property Id="SERVICETYPE" Value="25"/>
Note that the property Id should be in upper case (this makes the property public)
Then add this in your product element
<util:XmlFile
Id="UpdateServiceType"
Action="setValue"
File="[INSTALLFOLDER]MyService.exe.config"
SelectionLanguage="XPath"
Permanent="yes"
ElementPath="/configuration/appSettings/add[\[]#key='ServiceType'[\] ]/#value"
Value="[SERVICETYPE]" />
You can then pass in SERVICETYPE at install time ie.
MyMSI.msi SERVICETYPE=2
I am invoking a windows batch command from Jenkins, after i get the latest version of my project from SVN. the windows batch command just performs certain file copying, after the all the files are retrieved from SVN and runs an ANT build. In the ANT build process, i am generating a JSP file where i have tried to capture the in the following fashion.
%BUILD_TAG%-%BUILD_NUMBER%-%BUILD_ID%-%SVN_REVISION%
Unfortunately none of this information is understood by the build process and it just writes %BUILD_TAG%-%BUILD_NUMBER%-%BUILD_ID%-%SVN_REVISION% into the file.
Could you please let me know if there is a way to capture these information into a file in the way i am trying to do? if not, could you direct me on how these information could be captured into a JSP file during the process that we are following?
BUILD_TAG, SVN_REVISION, etc are all environment variables that are present during a Jenkins build, and to use them in Ant, you would use them as any other environment variable from Ant
First, add a line:
<property environment="env"/>
Then you can reference any environment variable with this prefix, like:
${env.VAR_NAME}
So in your case, you'd do:
${env.BUILD_TAG}-${env.BUILD_NUMBER}-${env.BUILD_ID}-${env.SVN_REVISION}
I'm running an ant build through Jenkins and on the stage where it is deploying to windows-share its returning the following error:
Failed to copy FILE to FILE2 due to failed to create the parent directory for FILE2 (I've taken the paths out to keep the question shorter).
I'm guessing that there might be some problem with permissions with the jenkins default user but this problem has only just started occurring, and any help would be great.
Thanks
This is a pretty old question but I thought I'd come back and complete it with a short update as to what was actually going on. Someone had changed the password for the user that was logged on to the vm that jenkins was running on, and when it was trying to create the directory to stick the files into it was running into permissions errors. Only problem was the error message wasn't very descriptive.
So in the end this was an infrastructure problem rather than anything to do with the ant script.
I take it you're doing something like this:
<copy file="${from.dir}/${from.file}"
tofile="${to.dir}/${to.file}"/>
And, you're getting an error that ${to.dir} doesn't exist.
In earlier versions of Ant, you definitely had to create the directory before doing a copy:
<mkdir dir="${to.dir}"/>
<copy file="${from.dir}/${from.file}"
tofile="${to.dir}/${to.file}"/>
I think I also noticed that later versions of Ant will create the directory for you when that directory didn't exist. I've always been in the habit of putting <mkdir/> in front of any task that creates a new file in a new directory, including things like <zip/> and <tar/>.
Here are some questions:
Do your users also use Ant to run their builds? I know that this isn't always the case. Many users use Eclipse and don't bother with Ant.
Is the version of Ant your users have vs. what your users have the same?
Do you do a clean? Jenkins can emulate a clean either by doing an update, reverting and removing unversioned element, or by simply creating a brand new working directory. If your users don't remove the destination directory where ${to.file} is being placed, it might work locally, but not on Jenkins.
Can you manually run Ant from the Jenkins working directory. If so, what results do you get? (Remember to disable this Jenkins job before doing this. You don't want Jenkins to do a build while you're experimenting in the working directory.)
We are going to be using Ivy with Ant, and we'll have Jenkins do our builds. I originally thought that having Jenkins do a <ivy:cleancache/> before running a build would be a good idea. (It would be part of the mandatory "clean" target).
However, I now see that <ivy:cleancache> doesn't simply clean up stuff from <ivy:cachepath>, but really removes the entire $HOME/.ivy/cache directory.
My concern is that if Jenkins does a <ivy:cleancache> on all builds before they start, it will interfere with other builds that Jenkins could be executing.
Is doing an <ivy:cleancache> a good idea, especially if a single user might be doing multiple builds at the same time?
In fact, what happens when you do a <ivy:cachepath pathid="compile.path"/> in multiple projects? Does this also affect something like Jenkins? Will Jenkins become confused if multiple builds are building the compile.cachepath at the same time?
In my opinion running the ivy cleancache task with every build is overkill and does away of one of the main benefits of using ivy, intelligent downloading of 3rd party dependencies.
Having said that as stated in the following related Maven question, all caches can become dirty and should be periodically purged:
When is it safe to delete the local Maven repository?
Couple of recommendations:
Use dedicated Jenkins job(s) to purge ivy cache
My first recommendation is to create a periodic Jenkins job that calls the following clean-all target in your build:
<target name="clean-all" depends="clean">
<ivy:cleancache/>
</target>
This ensures that Jenkins decides when the cache is purged and you can schedule it to happen outside of the normal build times (for example 2am on 1st of every month)
Isolate each project by using multiple caches
My second recommendation increases the isolation between your project builds. Configure each project to have it's own private cache, using the caches directive. in you ivy settings file.
Here's what I've decided to do:
I've modified my ivysettings.xml file to have the following:
<ivysettings>
<properties environment="env." override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
This does two things:
It defines the Ivy local cache as $HOME/.ivy/cache-$EXECUTOR_NUMBER where $EXECUTOR_NUMBER is the Jenkins executor. This means that each executor gets their own Ivy cache. Thus, if Jenkins is executing more than a one job at a time, each job will be picked up with a different executor, so it will have its own cache. If a job wants to clean the cache, it can go right ahead.
I've defined the resolve cache to ${basedir}/target/ivy.cache. This gives each job its own resolver cache which is quite small. But, this way ivy resolving doesn't interfere with other jobs if Jenkins is building multiple revisions of the same Ivy project.
The only drawback is that the user's default cache directory is called $HOME/.ivy/cache-$env.EXECUTOR_NUMBER which is not a pretty site. I'd love to make it a more reasonable $HOME/.ivy/cache-0, but I haven't figured that out. However, it doesn't really affect anything at this point.
Now, a developer has a single Ivy cache which contains all of the jars they've downloaded. This way, jars can be shared between projects which speeds up things for developers.
Meanwhile, Jenkins can clean the Ivy cache as often as it is configured. This could be done for each job, or once per day, or per month. However, since the cache is done per executor, I won't have an issue of the cache being cleaned while another job (which would be running on another executor) is depending upon that cache.
This should solve all of the particular issues. The only thing I'd like to do is figure out how to set a default EXECUTOR_NUMBER variable if one isn't already set. I've tried various things like this:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<properties environment="env." override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
But, to no avail. I've trued changing the override parameters on both the <property> and <properties> file all different ways, but it doesn't quite do what I want.
Just something I have been doing a lot to solve the last problem you have.
You can add this to the properties of the Jenkins Ant Build Steps
another.less.obtrusive.name=${EXECUTOR_NUMBER}
and add to ivysettings.xml.
So it will be "0" for everybody, except for Jenkins because it will inject this property to ANT.
Something on the main question:
On Jenkins, I always start new. CI builds should be robust, thorough. Fast is a welcomed by-product, but not a motivation.
I would like to optimize my scp deployment which currently copies all files to only copy files that have changed since the last build. I believe it should be possible with the current setup somehow, but I don't know how to do this.
I have the following:
Project/src/blah/blah/ <---- files I am editing (mostly PHP in this case, some static assets)
Project/build <------- I have a local build step that I use to copy the files to here
I have an scp task right now that copies all of Project/build out to a remote server when I need it.
Is it possible to somehow take advantage of this extra "build" directory to accomplish what I want -- meaning I only want to upload the "diff" between src/** and build/**. Is it possible to somehow retrieve this as a fileset in ANT and then scp that?
I do realize that what it means is that if I somehow delete/mess around with files on the server in between, the ANT script would not notice, but for me this is okay.
You can tell ant scp to only copy files which have been modified since the last push using the modified tag like so:
<scp trust="true" sftp="true"... >
<fileset dir="${local.dir}">
<modified>
<param name="cache.cachefile" value="localdev.cache"/>
</modified>
</fileset>
</scp>
The first time you use this, it will send all files and cache the timestamps in the cachefile declared in the param. After that, it will only send the modified ones.
Tested and verified in sftp mode.
I think you need to use rsync instead. I found the following article that answers your question.
In a nutshell rsync will resume where it left off and it should be possible to tunnel it over ssh.