I'm maintaining a quite large legacy application. The source tree is a real mess.
I'm trying to set up a build server.
On the source tree, I've third party component with sources (also in the project's include path). These components are also installed within the IDE.
My question is :
How to manage those components ?
I thought to manage this way:
Install the IDE on the build server
Install all the third party component
Remove the component sources from the project sources tree (and keep them on the project root on a dedicated folder each zipped)
Each time we need to customize (or debug) a third party component we re-build the package and re-install it in the IDE of the build server (and on each developers workstation)
What's the difference between having the components installed in the IDE and having the sources in the include path ? How the linker handle that case ?
We have set up our daily builds using simple command files.
Each project (.dpr) has an associated Build.cmd file.
All Build.cmd files are called from our main BuildServerRun.cmd file.
The BuildServerRun.cmd file takes care of
Deleting the entire source tree on the buildserver.
Getting a latest version from our source control repository.
Call each Build.cmd and pipe the output to a file.
Mail the results to all developers.
All paths to external components are configured in the dcc32.cfg file
..
-u"c:\Program files\Developer Express Inc\ExpressInplaceEditors\Delphi 5\Lib"
-u"c:\Program files\Developer Express Inc\ExpressQuantumGrid\Delphi 5\Lib"
..
-r"c:\Program Files\Borland\Delphi5\Lib"
-r"C:\Program Files\jvcl\jvcl\resources"
..
-i"C:\Program Files\jvcl\jvcl\run"
-i"C:\Program Files\jvcl\jcl\source"
Example of a Build.cmd.
Note: we have a policy to compile to bin\dcu, exe to bin, hence the -N, -E directives.
#echo on
dcc32speed -B -Q -W -H -Nbin\dcu -Ebin BpABA.dpr
#echo off
Example of snipped BuildServerRun.cmd
SET %Drive%=E:
:BuildServer
REM *************************************************
REM Clear files
REM *************************************************
ECHO. > "%Temp%\BuildLieven.txt"
ECHO. > "%Temp%\TestRunLieven.txt"
REM *************************************************
REM Set start time
REM *************************************************
echo | TIME | FIND "Huidige tijd" > "%Temp%\ResultLieven.txt"
REM *************************************************
REM Get latest versions
REM *************************************************
IF %LatestVersion%==1 CALL %Drive%\buildserver\latestversion.cmd
ECHO "Latest versions opgehaald" >> "%Temp%\ResultLieven.txt"
REM *************************************************
REM Build projects
REM *************************************************
CD %Drive%\Projects\
ECHO ***************************************************************** >> "%Temp%\BuildLieven.txt"
ECHO BpABA >> "%Temp%\BuildLieven.txt"
ECHO ***************************************************************** >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\production
ECHO Building BPABA\production
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\test
ECHO Building BPABA\test
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\test\dunit
ECHO Building BPABA\test\dunit
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
ECHO BPABATests >> "%Temp%\TestRunLieven.txt"
ECHO Running BPABATests
CALL bin\BPABATests >> "%Temp%\TestRunLieven.txt"
CD %Drive%\Projects
ECHO. >> "%Temp%\BuildLieven.txt"
ECHO. >> "%Temp%\BuildLieven.txt"
ECHO. >> "%Temp%\BuildLieven.txt"
REM *****************************************************************
REM Gather (Fatal)Errors/Hints/Warnings & Failures
REM *****************************************************************
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO (Fatal)Errors/Hints/Warnings en Failures >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Fatal errors during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Fatal:" >> "%Temp%\ResultLieven.txt"
ECHO Errors during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Error:" >> "%Temp%\ResultLieven.txt"
ECHO Warnings during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Warning:" >> "%Temp%\ResultLieven.txt"
ECHO Hints during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Hint:" >> "%Temp%\ResultLieven.txt"
ECHO Failures during test >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND /c "Failures:" >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Controle #Projecten = #Compiles >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO #Projecten >> "%Temp%\ResultLieven.txt"
TYPE "%Drive%\buildserver\buildserverrun.cmd" | FIND /i "cmd >> " | FIND /i "Lieven" | FIND /i /v /c "FIND /i /v /c" >> "%Temp%\ResultLieven.txt"
ECHO #Compiles >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\buildLieven.txt" | FIND /i /c "dcc32" >> "%Temp%\ResultLieven.txt"
ECHO #Tests expected to run >> "%Temp%\ResultLieven.txt"
TYPE "%Drive%\buildserver\buildserverrun.cmd" | FIND /i "TestRunLieven" | FIND /i "CALL" | FIND /i /v /c "FIND /i /v /c" >> "%Temp%\ResultLieven.txt"
ECHO #Tests actually run >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND /i /c "DUnit / Testing" >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Detail (Fatal)Errors/Hints/Warnings en Failures >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Fatal:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Error:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Warning:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Hint:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND "Failures:" >> "%Temp%\ResultLieven.txt"
REM *************************************************
REM Set stop time
REM *************************************************
ECHO | TIME | FIND "Huidige tijd" >> "%Temp%\ResultLieven.txt"
REM *************************************************
REM Send results
REM *************************************************
CALL %drive%\buildserver\Blat.cmd
My answer is more general than Lieven's answer, which is Delphi-specific. I wrote this one shortly after the question, but went to a co-worker before submitting ;)
I refuse to install any IDE on our main Windows build agent. Sounds like a nightmare to me. The MSBuild engine handles all build scenarios well, and other than .NET, you just need the Windows SDK installed. Or you could use NAnt and even CMake, whatever. Just don't install IDEs. It's not fun on build servers.
Now you have tagged this as Delphi. I don't know how good it works there, but as Lieven wrote, Delphi comes with a command line compiler. I just don't have any experience how it works with third-party compnents, but I think Delphi supports MSBuild in the latest version.
I'm also unsure whether including thiry-party components into version control is a good thing, because of the space it takes - though you can also put them somewhere else and include them as external, which makes it much smaller, but also imposes the problem that upgrading the components for one app will upgrade them for all - so you better have good integration tests. But that is the point of a build server anyway.
Apart from that, it is always a good thing to check-out and have all components required to build the application available. You don't need to install components into an IDE if they were made well. Depending on what components they are, in many cases you don't even need to install them on developer machines. Many .NET components, for example, are available in the designer when you add a reference to them. And licensing is typically no more than "put the license file into the same directory". Well, that's how it should be, at least. If that's not how it works in Delphi today, that's likely one of the reasons Delphi is on its way out. Other than the Borland/Inprise/DevCo/Codegear/Embarcadero hassle.
Similar situation here, fortunately it did not start with a big mess. It is true that the real problem lies in the IDE configuration, which needs to point to the correct versions of the third party components if you check out an older version. The only solution I have heard of is the use of different registry branches for the different product release configurations.
The components are kept in a separate directory structure, and use version numbers in the directory names when possible. This makes it easier to check out old versions and have the build scripts point to the correct version.
We use Apache Ant as the main build tool for years now and it really does everything we need, including unit test invocation and Innosetup script generation.
Compilation of packages is only necessary you ship the executable with BPLs, otherwise it is not necessary on a build server.
Installing the components in the IDE is also not necessary on the build server.
You can use Owly CI tool.
It allow build projects in easy way by defining manifest file.
It also allows handle dependencies - you can wrap 3rd party components in owlyci packages and mark them as dependencies to the main project.
There is an example, how to use it with Jenkins CI system.
Related
I work on a Maven project that need to copy mode than 10 GB of artifacts in a target repository from a maven local repository (after downloaded them).
In some cases (e.g. for tests), I'd like to replace this copy by a symlink creation in order to save few minutes.
My question is: Is there a way to ask to plugin maven-dependency-plugin goal copy-dependencies to create a symlink OR is there any maven plugin that can do it.
The copy-dependencies goal cannot, to my knowledge, do this out of the box. However, you can use a shell script:
#!/bin/sh
outputDir=target/dependency
mkdir -p "$outputDir"
mvn dependency:resolve |
grep ':\(compile\|runtime\)' | sed 's/\[INFO\] *//' |
while read gav
do
case "$gav" in
*:*:*:*:*:*) # G:A:P:C:V:S
g="${gav%%:*}"; remain="${gav#*:}"
a="${remain%%:*}"; remain="${remain#*:}"
p="${remain%%:*}"; remain="${remain#*:}"
c="${remain%%:*}"; remain="${remain#*:}"
v="${remain%%:*}"
s="${remain#*:}"
;;
*:*:*:*:*) # G:A:P:V:S
g="${gav%%:*}"; remain="${gav#*:}"
a="${remain%%:*}"; remain="${remain#*:}"
p="${remain%%:*}"; remain="${remain#*:}"
c=""
v="${remain%%:*}"
s="${remain#*:}"
;;
esac
g=$(echo "$g" | sed 's/\./\//g')
test -n "$c" && artName="$a-$v-$c" || artName="$a-$v"
ln -s "$HOME/.m2/repository/$g/$a/$v/$artName.$p" "$outputDir"
done
I want to ask how to find all the executable names of a package in ROS (Robot Operating System)? For example, find spawn_model in gazebo_ros package. When I inspect the package in my system, it just shows some .xml, .cmake files, without any executables. But I can run it, such as: rosrun gazebo_ros spawn_model.
Thank you!
An easy way to do this is to type: "rosrun name_of_package " and then press tab two times, it should show you all the executables built.
After looking in the bash autocompletion script for rosrun, it looks like the command catkin_find is used to find the location of the executables for a package, and the executables are filtered with a find command.
If you want to create a script to give you a list of the executables follow the instructions below:
Save the following script in a file called rospack-list-executables:
#!/bin/bash
if [[ $# -lt 1 ]]; then
echo "usage: $(basename $0) <pkg_name>"
echo ""
echo " To get a list of all package names use the command"
echo " 'rospack list-names'"
exit
fi
pkgname=${1}
pkgdir="$(catkin_find --first-only --without-underlays --libexec ${pkgname})"
if [[ -n "${pkgdir}" ]]; then
find -L "${pkgdir}" -executable -type f ! -regex ".*/[.].*" ! -regex ".*${pkgdir}\/build\/.*" -print0 | tr '\000' '\n' | sed -e "s/.*\/\(.*\)/\1/g" | sort
else
echo "Cannot find executables for package '${pkgname}'." >&2
exit 1
fi
Then make the rospack-list-executables script executable (chmod +x rospack-list-executables) and place it in a directory that can be found in your $PATH environment variable.
Run the script:
$ rospack-list-executables gazebo_ros
debug
gazebo
gdbrun
gzclient
gzserver
libcommon.sh
perf
spawn_model
You should get the same result that you get when you type the rosrun <pkgname> command and press Tab:
$ rosrun gazebo_ros
debug gazebo gdbrun gzclient gzserver libcommon.sh perf spawn_model
You can check the executables for all packages with the following bash code:
rospack list-names | while read pkgname; do
echo "Executables for package '${pkgname}':";
rospack-list-executables $pkgname; echo "";
done
To enable package autocompletion for your newly created command, type the following:
complete -F _roscomplete rospack-list-executables
If you do not want to have to type the complete command every time you login, you can append it to your .bashrc file:
echo "complete -F _roscomplete rospack-list-executables" >> ~/.bashrc
Now when you type the command rospack-list-executables and press the Tab key, you should get a list of all the available packages to choose from.
catkin_find --first-only --without-underlays --libexec <your package name>)
should give you the folder where the executables are
I have a pre-build PowerShell script defined in my Build Definition. The PowerShell script execute a program:
& "c:\Program Files\nodejs\npm.cmd" "run" "build"
When npm runs, npm writes things to stdout/console. This output doesn't show up in my logs for the build though.
What do I need to do to make sure the build logs capture this console output?
In other words, is there something I can add to & "c:\Program Files\nodejs\npm.cmd" "run" "build" that will pipe the output to the build log?
EDIT:
I think it's one of these three but not sure which:
& "c:\Program Files\nodejs\npm.cmd" "run" "build-release" 2>&1 | Out-Host
& "c:\Program Files\nodejs\npm.cmd" "run" "build-release" 2>&1 | Write-Host
& "c:\Program Files\nodejs\npm.cmd" "run" "build-release" 2>&1 | Write-Output
If you set logging to verbose it will log everything...
I normally add a -Verbose at the end of every line
Now I am compiling my MetaTrader .mq4 files to .ex4 files with MetaEditor.
But my .mq4 files are generated by a Java-process, and I would like to automate the compilation process.
Is there a command-line compiler tool I could call programmatically?
To compile a source code file from a command line, you can use MetaEditor for that. For example:
metaeditor.exe /compile:"C:\Program Files\Platform\MQL5\Scripts\myscript.mq5"
For 64-bit use metaeditor64.exe instead.
In Linux/macOS, this can be achieved using Wine, e.g.:
wine metaeditor.exe /compile:"MQL4/Experts/MACD Sample.mq4"
For mass compilation, you can specify folder, like:
metaeditor.exe" /compile:"MQL5\Scripts"
To specify custom MQL5/MQL4 folder with include files, you can use /inc parameter, for example:
metaeditor.exe /compile:"./Scripts" /inc:"C:\Program Files\TradingPlatform 2\MQL5"
For additional information about the compilation process, you can use /log:
metaeditor.exe /compile:"C:\Program Files\Platform\MQL5\Scripts\myscript.mq5" /log
To check for the syntax only, add extra /s.
If the compilation fails, the MQL4.log file would be created in the platform folder with the relevant details. It's going to be in UTF-16 format, so you may need a special tool for it (such as Vim, Ruby, findstr or rg).
To specify the custom compilation log file, use /log:file.log parameter, e.g.
metaeditor.exe /log:errors.log /compile:.
Note: Display to the standard output is not supported (although on Linux you can use: /log:CON).
For more information, check: Compilation from the Command Line
Some time ago you could download the compiler of MQL4/MQL5 programs that runs separately from MetaEditor — MQL.exe. It was distributed separately from the terminal and you could download it at the following addresses:
https://download.mql5.com/cdn/web/metaquotes.software.corp/mt5/mql.exe
https://download.mql5.com/cdn/web/metaquotes.software.corp/mt5/mql64.exe
Usage (as per MQL4/MQL5 Compiler build 1162 from 02 Jul 2015):
mql.exe [<flags>] filename.mq5
/mql5 - compile mql5 source
/mql4 - compile mql4 source
/s - syntax check only
/i:<path> - set working directory
/o - use code optimizer
However the standalone compiler was intentionally removed, so now links point to the installer in favor of MetaEditor.
Much older version of MetaTrader prior to build 600 had metalang.exe included with the platform.
However in build 616, MetaQuotes intentionally has removed the compiler (mql.exe/mql64.exe) from the standard MetaTrader installation.
This means if you upgrade your MT platform (>616), the compiler executable will be removed.
This is a little late, but since I wrote a little script for UltraEdit/UEStudio and have received heaps of help from stackoverflow, here is my script. It compiles then copies the ex4 to a number of test MT4 installations:
The "Compile" button on UE does:
"MT4Compile.bat" "%FilePath" "%FileName"
Start in path eg: D:\Development\MQ4 (Location of MT4Compile.bat)
Normally my source code is in a library tree under D:\Development\MQ4[Group][ExpertName][FileName].mq4
Contents of D:\Development\MQ4\MT4Compile.bat:
#echo off
rem Version: 1.1
rem Date: 24 Sep 2013
rem Author: Shawky
rem Refer to HELP: for info
SET XC=xcopy /D /Y /V /F /I
SET PROGDIR=D:\Development\Go Pro Demo (MQ4 Testing)
SET DSTPATH=%PROGDIR%\experts
SET SIMPATH1=G:\Apps\MT4\BackTest IC (Recent)\experts
SET SIMPATH2=G:\Apps\MT4\BackTest IC (All)\experts
SET SIMPATH3=G:\Apps\MT4\BackTest Go (All)\experts
SET DEPLOYPATH=D:\Development\Deployment\experts
SET SRCPATH=%1
SET SRCPATH=%SRCPATH:"=%
IF "%SRCPATH%"=="" (
SET SRCPATH=[Arg1]
)
SET APPNAME=%2
SET APPNAME=%APPNAME:"=%
IF "%APPNAME%"=="" (
SET APPNAME=[Arg2]
)
SET SRCFILE=%APPNAME%.mq4
SET DSTFILE=%APPNAME%.ex4
SET CMD="%PROGDIR%\metalang.exe" "%SRCFILE%" "%DSTFILE%"
IF "%SRCPATH%"=="[Arg1]" GOTO HELP
IF "%APPNAME%"=="[Arg2]" GOTO HELP
cd %SRCPATH%
IF NOT EXIST "%SRCFILE%" (
SET ERROR=Error: File "%SRCFILE%" does not exist in %SRCPATH%
GOTO HELP
)
echo .
echo Compiling %SRCFILE% to %DSTPATH%\%DSTFILE%
echo .
DEL *.log
%CMD%
IF EXIST "%DSTFILE%" (
echo .
echo Distributing executable to SIM and Deployment paths
%XC% "%DSTFILE%" "%DSTPATH%\"
IF EXIST "%SIMPATH1%" %XC% "%DSTFILE%" "%SIMPATH1%\"
IF EXIST "%SIMPATH2%" %XC% "%DSTFILE%" "%SIMPATH2%\"
IF EXIST "%SIMPATH3%" %XC% "%DSTFILE%" "%SIMPATH3%\"
IF EXIST "%DEPLOYPATH%" copy /B /Y "%DSTFILE%" "%DEPLOYPATH%\%APPNAME% (Dev).ex4"
)
goto END
:HELP
echo . Metatrader 4 Command Line utility for compiling MT4 programmes.
echo .
echo . This batch files allows MT4 applications to be compiled from a directory other than .\experts.
echo . The output will be copied to experts after compilation.
echo .
echo . [Arg1] = Path to MT4 application directory
echo . [Arg2] = Name (without extension) of the main MQ4 source code to compile.
echo .
echo . Example:
echo . MT4Compile.bat "D:\Development\MQ4\MyExpert\" "PrimaryMQ4FileName"
echo .
echo . Programme Directory: %PROGDIR%
echo . Source Path: %SRCPATH%
echo . Source File: %SRCFILE%
echo . Destination File: %DSTFILE%
echo . Target Path: %DSTPATH%
echo .
echo . Argument 1: %SRCPATH%
echo . Argument 2: %APPNAME%
echo .
echo . Commands to execute would be:
echo .
echo . %CMD%
echo . %XC% "%DSTFILE%" "%DSTPATH%\"
echo .
echo . %ERROR%
echo .
pause
:END
All the best.
Shawky
Yes, there is an executable in the install directory of the terminal. It is called metalang.exe.
so here is my script to loop through specific video extensions » add a manual profile » generate necessary *.bat & finally a final 'loader' batch file to execute previous *.bat files sequentially & necessary logging (this gives quiet a deal of freedom if you so want)
::==
:: gets lines into vars c1 v2 v...
#echo off
:: user input required
cd /d "d:\Trainers\out\"
setLocal EnableDelayedExpansion
dir /B /O:N | findstr ".wmv$ " >filename.txt
echo. >log.txt
:: user input required
for /f "tokens=* delims= " %%a in ('type filename.txt ^|findstr ".wmv$"') do (
set /a n+=1
echo. >file!n!.bat
set in=%in%%%a
:: user input required
set out=!in:.wmv=.mp4!
:: user input required
set v=x264 --crf 23 --level 3.1 --tune film -o "d:\Trainers\out\!in!" "d:\Trainers\out\!out!"
echo. !v!>file!n!.bat
)
dir /B /O:N | findstr ".bat$ " >x264_home.txt
for /f "tokens=* delims= " %%a in (x264_home.txt) do (
set /a n+=1
:: mtee is an external library Google it
set "z=call %%a | mtee /d/c/t/+ log.txt"
echo. !z! >> x264_home.bat
)
echo. #echo off > newFile.bat
type x264_home.bat >> newFile.bat
type newFile.bat > x264_home.bat
del newFile.bat,x264_home.txt,filename.txt
echo. pause >> x264_home.bat
echo. #echo All Operation done... >> x264_home.bat
:: user input required
move "d:\Trainers\out\*.bat" "d:\Program Files\x264_auto\test\"
:: user input required
move "d:\Trainers\out\log.txt" "d:\Program Files\x264_auto\test\"
::==
Now the above code which is fairly easy to understand (bcz its written by a noob) run perfectly & create necessary files. For instance one of the file1.bat looks like this:
x264 --crf 23 --level 3.1 --tune film --preset veryslow --deblock -2:-1 --zones 24233,25324,q=20 --acodec aac --abitrate 80 -o "d:\Trainers\out\1.wmv" "d:\Trainers\out\1.mp4"
...& the loader .bat file looks like
#echo off
call file1.bat | mtee /d/c/t/+ log.txt
call file2.bat | mtee /d/c/t/+ log.txt
call file3.bat | mtee /d/c/t/+ log.txt
#echo All Operation done...
You see this is a quiet flexible approach in that you can use special filestr » set another loop » set another profile. Furthermore every batch file can be latter edited especiialy when you heavily use --zone x264 feature
I am successful because there is no error in any output ...but its the x264.exe (provider/compiler x264GUI) throws error which it otherwise don't?
d:\Program Files\x264_auto\test>x264 --crf 23 --level 3.1 --tune film --preset
veryslow --deblock -2:-1 --zones 24233,25324,q=20 --acodec aac --abitrate 80 -o
"d:\Trainers\out\1.wmv" "d:\Trainers\out\1.mp4"
ffms [error]: could not create index
lavf [error]: could not open input file
raw [error]: raw input requires a resolution.
x264 [error]: could not open input file `d:\Trainers\out\1.mp4' via any method!
its the x264 thats the culprit perhaps a senior guide is required here
Is your x264 compiled with mp4 input support? (I believe that needs lavc/lavformat, just download precompiled x264 from x264.nl which has all extras)
Do you get the same error if you run the same command directly? (not through bat files)
If yes, does it only happen when you use zones? (if it does, then post an example of your command line as x264 bug to x264-devel mailing list)
If no, are you sure you are running the exact same x264? (perhaps there are several in different places on your system)
I recommend doing what you're doing either (a) in python with subprocess.call(...) or (b) in cygwin/bash/shell script, or . bat files are pretty much the wrong answer to any problem :) The nice thing about either of those two is that they have simple, regular escaping rules for program arguments.