I have a script that compile all the projects (around 50) of my solution like following
msbuild "myProjName.dproj" /t:build /p:config="Release" /fileLogger /flp:ErrorsOnly /nologo
This works just fine but takes forever to compile. In order to make it faster to build i've been trying to leverage all the potential of our modern multi-core machines using the '/maxcpucount' switch explained here : http://msdn.microsoft.com/library/bb651793.aspx
I get about the same compilation time on my 4-core CPU dev machine. No perf gains.
Apparently this can only work when projects need dependencies to be built. The others "workers" would then build these dependencies projects in parallel as the main proj.
So i tried to build a project group in delphi and adding all my projects to it and than run the msbuild command on this .groupproj but it is still as slow as it has always been.
Did any of you achieved to build multiple projets at the same time with msbuild?
If yes can you provide me an explanation?
Thanks!
The following applies to RAD Studio XE4, but it may also apply to earlier or later versions. Also, the dependencies defined in the .groupproj will not be honored with this method. The .groupproj I was trying to parallelize had no inter-project dependencies, so I didn't figure out how to handle this.
When you build a .groupproj file with MSBuild using the Build, Clean or Make target, the build doesn't run in parallel because these targets use the CallTarget task to execute other targets, but CallTarget doesn't execute its targets in parallel.
In order to build separate projects in parallel, the MSBuild project must use a single MSBuild task to build multiple projects at once. The targets must be defined like this:
<Target Name="Build">
<MSBuild Projects="#(Projects)" BuildInParallel="true"/>
</Target>
<Target Name="Clean">
<MSBuild Targets="Clean" Projects="#(Projects)" BuildInParallel="true"/>
</Target>
<Target Name="Make">
<MSBuild Targets="Make" Projects="#(Projects)" BuildInParallel="true"/>
</Target>
Add these to the .groupproj, then remove the other <Target> directives as well as the <Import> directive. (CodeGear.Group.Targets defines some targets to build the projects in the proper order and to build dependencies when you ask to build only a subset of the projects, but it overrides the Build, Clean and Make targets defined in the .groupproj.) Note that this only allows you to build all projects, not just a subset.
BuildInParallel was added in MSBuild 3.5. However, since .groupproj files don't specify the ToolsVersion attribute, MSBuild will use the MSBuild task as defined in version 2.0, which didn't support BuildInParallel. There are two options to fix this:
Add ToolsVersion="3.5" (or a later version) to the root <Project> element of your .groupproj file.
Run MSBuild with the /toolsversion:3.5 (or /tv:3.5 for short) command-line parameter (/toolsversion overrides the ToolsVersion specified in all project files.)
After doing this, run MSBuild with the /maxcpucount (or /m) argument and your projects should build in parallel. However, RAD Studio doesn't handle this transformed project group correctly, so you may want to give the file a different extension to make it clear that it's not a standard RAD Studio project group (any extension that ends in proj will do).
The following XSLT stylesheet performs the transformation described above:
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
exclude-result-prefixes="msbuild"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
xmlns:msbuild="http://schemas.microsoft.com/developer/msbuild/2003"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:template match="//msbuild:Project">
<xsl:copy>
<xsl:attribute name="ToolsVersion">3.5</xsl:attribute>
<xsl:apply-templates select="#* | node()"/>
<Target Name="Build">
<MSBuild Projects="#(Projects)" BuildInParallel="true"/>
</Target>
<Target Name="Clean">
<MSBuild Targets="Clean" Projects="#(Projects)" BuildInParallel="true"/>
</Target>
<Target Name="Make">
<MSBuild Targets="Make" Projects="#(Projects)" BuildInParallel="true"/>
</Target>
</xsl:copy>
</xsl:template>
<xsl:template match="//msbuild:Target">
<!-- Do not copy -->
</xsl:template>
<xsl:template match="//msbuild:Import">
<!-- Do not copy -->
</xsl:template>
<xsl:template match="#* | node()">
<xsl:copy>
<xsl:apply-templates select="#* | node()"/>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
You can apply this stylesheet with MSBuild (4.0 or later: XslTransformation was added in MSBuild 4.0) using this project file (where groupproj2parallel.xslt is the XSLT file above):
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="Build" Inputs="$(InputPaths)" Outputs="$(OutputPaths)">
<XslTransformation
XmlInputPaths="$(InputPaths)"
XslInputPath="groupproj2parallel.xslt"
OutputPaths="$(OutputPaths)" />
</Target>
</Project>
You need to specify InputPaths and OutputPaths explicitly on the command line with /p:InputPaths="..." /p:OutputPaths="...", or by specifying them on the Properties parameter of an MSBuild task. (Alternatively, you can just hardcode the file names in the project file.)
The target definitions provided with MSBuild for C# and Visual Basic projects handle dependencies by using the <ProjectReference> items defined in project files, instead of defining dependencies in the solution file. Delphi .dproj files and C++ Builder .cbproj files don't support this, as the underlying CodeGear.Common.Targets doesn't reuse the machinery defined in Microsoft.Common.Targets for <ProjectReference>.
There are two ways to build a Delphi projects: MSBuild or DCC32.exe. MSBuild is recommended as the project files (dproj and groupproj) encapsulate all configuration settings.
However, there are extra over head using MSBuild compare to plain old DCC32.exe. Furthermore, using MSBuild to build Delphi Project Group (.groupproj) doesn't bring any benefifs for multi-core CPUs. The build performance is same as single core CPU.
Here are my statistics to build a 290 dproj files in one single groupproj:
MSBuild a `groupproj` contains 290 `dproj` on 2C/4T CPU: ~100s
MSBuild a `groupproj` contains 290 `dproj` on 4C/8T CPU: ~100s
MSBuild 290 `dproj` run in multi-threads on 2C/4T CPU: ~121s
MSBuild 290 `dproj` run in multi-threads on 4C/8T CPU: ~50s
DCC 290 `dproj` run in multi-threads on 2C/4T CPU: ~37s
DCC 290 `dproj` run in multi-threads on 4C/8T CPU: ~24s
From the reading, we can conclude that MSBuild introduce extra overhead compare to DCC32. To fully utilize CPU cores and threads available, DCC32 is the way to go by sacrifice the convenient of project configuration encapsulation design for .DPROJ.
A msbuild script to build Delphi groupproj in parallel is available at https://github.com/ccy/msbuild.delphi.parallel
Little bit off-topic: you could try the fastdcc part of the IDE fix pack to get faster builds:
http://andy.jgknet.de/blog/ide-tools/ide-fix-pack/
For example, I got a build time of 1 minute going down to 22s!
Related
how can I tell ant to find Ivy's jar in my own lib? ant just kept looking at it's home folder even when I've explicitly told it to find the jar somewhere else.
I would recommend removing the ivy jar from the ANT home directory. (For some very odd reason it's not normally packaged with ANT).
Instead I recommend including a special task to ensure ivy is installed.
<available classname="org.apache.ivy.Main" property="ivy.installed"/>
<target name="install-ivy" description="Install ivy" unless="ivy.installed">
<mkdir dir="${user.home}/.ant/lib"/>
<get dest="${user.home}/.ant/lib/ivy.jar" src="http://search.maven.org/remotecontent?filepath=org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar"/>
<fail message="Ivy has been installed. Run the build again"/>
</target>
Analysis
The ANT manual outlines the order in which jars a loaded by ANT at startup.
-lib jars in the order specified by the -lib elements on the command line
jars from ${user.home}/.ant/lib (unless -nouserlib is set)
jars from ANT_HOME/lib
This will always happen and unfortunately it won't matter what you do inside your build file.....
Jars in the ANT_HOME/lib
In my opinion, putting jars in the ANT_HOME effectively creates a bespoke installation of ANT. It makes your projects less portable across machines, and the customizations are frequently forgotten and undocumented.
So if you have control over the build server I would recommend removing any ANT tasks your find here.
Jars in the ${user.home}/.ant/lib
Placing jars here is less objectionable for the following reasons
Directory owned by the user running the build
Can be ignored at run-time by by using the commandline option -nouserlib
The only jar I put here is ivy... All other jars exist in the ivy cache (including ANT tasks)
You can place Ivy binaries in some folder inside you project folder. For example, in my case, it's etc/build/. I put where ivy.jar and jsch.jar.
After that provide the correct namespace in project defenfition and load Ivy.
<project name="somename" basedir="." xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="ivy-load">
<path id="ivy.lib.path">
<pathelement location="${basedir}/etc/build/ivy.jar"/>
<pathelement location="${basedir}/etc/build/jsch.jar"/>
</path>
<taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
</target>
<target name="ivy-init" depends="ivy-load">
<ivy:settings file="${basedir}/etc/ivysettings/ivysettings.xml"/>
<ivy:resolve conf="${ivy.conf}"/>
</target>
...
</project>
I'm attempting to use TFS and MSBuild as a build and source control for a non-.NET project. This project contains a series of individual called .skbsrc files, that each compile into a .skb file. I'm trying to figure out if it's possible to use MSBuild in a way to build these files.
Say I'm using the example on the msdn website:
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Compile Include="testfile.skbsrc" />
</ItemGroup>
<Target Name="Build">
<Csc Sources="#(Compile)"/>
</Target>
</Project>
Csc is clearly used for C# code, and runs the csc.exe program. Is there a way I can make a block like this for my own compiler (skbuilder) so I could run like:
<Skbuilder Sources="#(Compile)" />
which in turn would run
>skbuilder testfile.skbsrc
If this is possible with msbuild, could anyone post an example? I've been unable to find anything in my searches.
Thank you so much.
You can create a customized task in msbuild to do it. "CSC" is a default task in C#, so what you need is to implement a similar task for your exe.
This MSDN article explains how to write a task for msbuild.
Take a look at the Exec Task.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<SourceFiles Include="*.skbsrc"/>
</ItemGroup>
<Target Name="Build">
<Exec Command="skbuilder "%(SourceFiles.Identity)""/>
</Target>
</Project>
You'll also need to pass an argument to your skbuilder program telling it to output it's files to the $(OutDir) directory or use Copy Task to pick up *.skb to $(OutDir). Use quotes around the argument incase the TFS workspace path has spaces in it.
Finally you can test this on your own machine without using TFS by creating a simple .BAT file:
set PATH=%CD%;C:\Windows\Microsoft.NET\Framework\v4.0.30319
msbuild build.proj > build.log
I have a shared ant script b.ant which internally use antcall. It calculates a property that the client scripts use. I use include instead of import client scripts to avoid unintentional overwriting of targets, but this gives me a problem with the antcall.
When using include all targets in b are prefixes, and depends attributes in b are updated accordingly. This is however not true for antcall. Is there a there are way to handle this, i.e. make antcall always call the "local" ant target?
I can workaround this by using import, but then I'll get all the overwrite problems. It is not possible to use depends instead of antcall.
Example files
I have two files:
a.ant
<project>
<include file="b.ant" as="b" />
<target name="test-depends" depends="b.depend">
<echo>${calculated-property}</echo>
</target>
<target name="test-call" depends="b.call">
<echo>${calculated-property}</echo>
</target>
</project>
b.ant
<project>
<target name="depend" depends="some-target">
<property name="calculated-property" value="Hello World"/>
</target>
<target name="call">
<antcall target="some-target" inheritrefs="true"/>
<property name="calculated-property" value="Hello World"/>
</target>
<target name="some-target"/>
</project>
Example output
Calling test-depend works as expected but test-call fails with this output:
b.call:
BUILD FAILED
D:\ws\rambo2\ws-dobshl\ant-test\b.ant:6: The following error occurred while executing this line:
Target "some-target" does not exist in the project "null".
Total time: 258 milliseconds
Ant is a dependency matrix specification language. Usually a bunch of <antcall/>, <ant/>, <include/> and <import/> is a sign of a poorly written build script. It's a developer trying to force Ant to act like a programming language.
For developer, it makes sense to break up a program into smaller files. Even Python and Perl scripts can benefit from this. However, breaking up an Ant build script usually causes problems. We had a developer who went through every project and broke up all the build.xml files into six or seven separate build files in order to improve the process. It basically broke the whole Ant dependency mechanism. To fix it, he then tossed in a bunch of <ant/> calls and <include> tasks. In the end, it meant that each target was called between 12 to 20 times.
Not using <import/> and <antcall/> isn't a hard and fast rule. But, I've been using Ant for years and rarely ever used these mechanisms. When I do, it's usually for a shared build file that multiple projects will use (which sounds like what you have) but instead of defining targets in my shared build file, I define macros. This eliminates the target namespace issues that you are having, and the macros work better because they act more like Ant tasks. This is especially true with the introduction of <local/> in Ant 1.8.
See if you can restructure the shared build file into using <macrodef/> instead of targets. It will make it much easier to include your shared build file.
Give a name to the <project> in b.ant and then change the target of the <antcall>:
<project name="b"> <!-- Give the project a name -->
<target name="depend" depends="some-target">
<property name="calculated-property" value="In b.depend"/>
</target>
<target name="call">
<!-- Specify the name of the project containing the target -->
<antcall target="b.some-target" inheritrefs="true"/>
<property name="calculated-property" value="In b.call"/>
</target>
<target name="some-target"/>
</project>
The result of ant -f a.ant test-call:
b.call:
b.some-target:
test-call:
[echo] In b.call
BUILD SUCCESSFUL
With the changes to b.ant, the <include> in a.ant can be simplified by removing the as attribute:
<include file="b.ant" />
Hi guys : I noticed that there is an ant task for checkstyle
http://checkstyle.sourceforge.net/anttask.html
I want checkstyle to run in my Ant build, which is on jenkins.
Unfortunately, the instructions are somewhat cryptic - with references to enabling project dependencies , modules, and other ant-specific configurations. I have a massive build file and I'm not really a build engineer - so I want to keep it simple without adding too much bload to the script.
Jenkins has a nice little button which supports displaying the checkstyle results, however, jenkins requires that you run the checkstyle and configure it yourself when you run a build.
What is the simplest way to modify my build.xml and ivy.xml ( i assume i will need to add checkstyle to ivy to get the jar remotely) to enable a basic checkstyle analysis of all the code base when builds are run ?
The sample of how to do it with help of just Ant:
<?xml version="1.0" encoding="UTF-8"?>
<project name="Build" default="build" basedir=".">
<property file="props.properties"/>
<taskdef resource="checkstyletask.properties" classpath="${checkstyle.jar.path}"/>
<target name="build" depends="checkstyle">
<echo>Starting build</echo>
<echo>Build finished</echo>
</target>
<target name="checkstyle">
<echo>Starting checkstyle</echo>
<checkstyle config="rules/sun_checks.xml" failOnViolation="false">
<fileset dir="src" includes="**/*.java"/>
<formatter type="plain"/>
<formatter type="xml" toFile="build/checkstyle_errors.xml"/>
</checkstyle>
<echo>Checkstyle finished</echo>
</target>
</project>
The quote from Checkstyle site:
failOnViolation - Specifies whether the build will continue even if
there are violations. Defaults to "true".
You can download checkstyle-5.4-bin.zip from here.
The distribution package contains sun_checks.xml - checkstyle configuration that checks the sun coding conventions and checkstyle-x.x-all.jar library with task engine.
I have a target named test and I want to do some tests.
I put here the important parts in build.xml. It includes:
<property name='lib.dir' value='lib' />
<path id='classpath'>
<fileset dir="${lib.dir}" includes="**/*.jar" />
</path>
And I have put the junit.jar and ant-junit.jar(is it a must?) in the lib directory.
However, if I run
ant test.
The output error is:
test:
BUILD FAILED
/home/xiaohan/EclipseWorkSpace/AntTest/build.xml:82: Problem: failed to create task or type junit
Cause: the class org.apache.tools.ant.taskdefs.optional.junit.JUnitTask was not found.
This looks like one of Ant's optional components.
Action: Check that the appropriate optional JAR exists in
-/usr/share/ant/lib
-/home/xiaohan/.ant/lib
-a directory added on the command line with the -lib argument
Do not panic, this is a common problem.
The commonest cause is a missing JAR.
This is not a bug; it is a configuration problem
Additionally, if I put the two jar files in /usr/share/ant/lib with the $ANT_HOME set, it still does not work.
Really thanks for any hints
In my case (using Mint, based on Debian)
sudo apt-get install ant-optional
was the only thing that worked.
The issue was solved on RHEL-based systems once ant-junit was installed:
$ sudo yum install ant-junit
<property name='lib.dir' value='lib' />
<path id='classpath'>
<fileset dir="${lib.dir}" includes="**/*.jar" />
</path>
this has nothing to do with Ant classpath itself. It is properties you can use in your tasks. You have to put jars to the suggested dirs or add command line argument.
Try running it like this:
ant -lib /path/to/the/ant-junit.jar/ test
I was seeing this message because I had missed to include the ant-junit.jar from my IDE's Classpath, e.g. in Eclipse > Right click on your project > Run as > Run Configurations.. > Classpath (tab) > Ensure the ant-junit.jar is there.
When you run JUnit tasks, you must make sure both your classpath that was used for your builds, and the path for the junit jars are added to your classpath.
I prefer to put the junit jars inside my project, so others don't have to install them in their machines for my build to work. I would install them in ${basedir}/antlib/junit
I use ${basedir}/antlib to store all the various Ant build related jars such as the Ant-Contrib jars, JaCoCo, Findbugs, etc. Since this is inside my project, a checkout will always include these jars and the build will work for everyone:
<classpath id="junit.path">
<fileset dir="${basedir}/antlib/junit"/>
</classpath>
This will create a path that contains your JUnit jars. Now to compile your junit tests:
<javac dest="${target.dir}/test-classes"
src=${src.test.java}">
<classpath refid="javac.classpath"/> <!-- Your main build classpath -->
<classpath refid="junit.path"/> <!-- Your JUnit classpath -->
<classpath path="${main.destdir}"/> <!-- These the classes you've built-->
Note that placing the JUnit jars in $ANT_HOME/lib main not work because it depends whether <javac> has includeAntRuntime set or not. It's always highly recommended not to have this set in order to make your build more platform independent.
Also remember to include that junit.path in your <junit> task too.