For the development/testing of our CFD code I like to frequently switch between Clang (for its strictness/warnings) and GCC (for performance), but this requires some of its dependencies (like e.g. NetCDF) to be compiled with the same compiler.
I know that Homebrew has the option to install multiple versions of software side-by-side and switch between them, but is it possible to do something similar using the same software version, but compiled with different compilers (by setting HOMEBREW_CC and HOMEBREW_CXX)?
Something like (wishful thinking, after somehow installing NetCDF with both Clang and GCC):
brew switch netcdf 4.3.3-gcc
brew switch netcdf 4.3.3-clang
I think this is possible only if you explicitly have different version numbers like in the example you used "4.3.3-gcc" and "4.3.3-clang".
If the version number is identical then there is no difference in the builds and brew can't differentiate them.
Also I wouldn't do this.
By compiling the same library multiple different ways you now start going into a dependency nightmare.
Dependency conflicts. Because even if you swap out "netcdf", how do you know you swapped out all it's dependencies too? If they are not compiled with the same compiler bad things can happen, for example conflicts can arise due to the differences in how calls are made or due to options enabled in one compiler for both the app and it's dependencies that weren't enabled in another build.
I don't recommend doing this, it's too much of a hassle.
But, if you really need two builds (such as for testing) then I would build them to isolated folder trees outside of your system path and do any testing against them there. Brew is not the best way to address this issue, as this is a non-standard use-case.
Related
I got some strange glibc-related linker errors for builds with distributed build cache configured on build nodes running different Linux distributions.
Now I somehow suspect build artifacts from those machines with different glibc versions getting mixed up, but I don't know how to investigate this.
How do I find out what Bazel takes into account when building the hash for a certain build artifact?
I know I can explicitly set environment variables which then will affect the hash. But how can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
And how do I check/compare what's been taken into account?
This is a complex topic and a multi-facet question. I am going to answer in the following order:
How do I check/compare what's been taken into account?
How to investigate against which glibc a build linked?
How can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
How do I check/compare what's been taken into account?
To answer this, you should look into the the execution look, specifically you can read up on https://bazel.build/remote/cache-remote#compare-logs. The *.json execution log should contain everything you need to know (granted, it might be a bit verbose) and is a little easier to process with shell-magic/your editor.
How to investigate against which glibc a build linked?
From the execution log, you can get all the required hashes to retrieve cached artifacts/binaries from your remote cache. Given these files, you should be able to use standard tools to get to the glibc version (ldd -r -v binary | grep GLIBC).
How can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
This depends on the way you have setup for compilation toolchain. The best case would be a fully hermetic compilation toolchain, where all necessary files are declared using attributes like https://bazel.build/reference/be/c-cpp#cc_toolchain.compiler_files.
But this would also mean to lock-down the compiler sysroot. This should include all libraries you are linking against if you want full hermeticity. If you want to use some system libraries, you need to tell bazel where to find them and to factor in their hash: https://stackoverflow.com/a/43419786/20546409 or https://www.stevenengelhardt.com/2021/09/22/practical-bazel-depending-on-a-system-provided-c-cpp-library/
If you use the auto-detected compiler toolchain, some tricks are used to lock-down the sysroot paths, but expect some non-hermiticity. https://github.com/limdor/bazel-examples/tree/master/linux_toolchain is a nice write-up how to move from the auto-detected toolchain to something more hermetic.
The hack
Of course, you can hack around this. Note, this is inherently a bad idea:
create a script that inspects the system, determines everything important like the glibc version, maybe the linux distribution (flavor)
creates a string describing this variation and hash-summing it
use that as the instance key/name for your remote cache
Problem
I've been developing a game in C++ in my spare time and I've opted to use Bazel as my build tool since I have never had a ton of luck (or fun) working with make or cmake. I also have dependencies in other languages (python for some of the high level scripting). I'm using glfw for basic window handling and high level graphics support and that works well enough but now comes the problem. I'm uncertain on how I should handle dependencies like glfw in a Bazel world.
For some of my dependencies (like gtest and fruit) I can just reference them in my WORKSPACE file and Bazel handles them automagically but glfw hasn't adopted Bazel. So all of this leads me to ask, what should I do about dependencies that don't use Bazel inside a Bazel project?
Current approach
For many of the simpler dependencies I have, I simply created a new_git_repository entry in my WORKSPACE file and created a BUILD file for the library. This works great until you get to really complicated libraries like glfw that have a number of dependencies on their own.
When building glfw for a Linux machine running X11 you now have a dependency on X11 which would mean adding X11 to my Bazel setup. X11 Comes with its own set of dependencies (the X11 libraries like X11Cursor) and so on.
glfw also tries to provide basic joystick support which is provided by default in Linux which is great! Except that this is provided by the kernel which means that the kernel is also a dependency of my project. Now I shouldn't need anything more than the kernel headers this still seems like a lot to bring in.
Alternative Options
The reason I took the approach I've taken so far is to make the dependencies required to spin up a machine that can successfully build my game very minimal. In theory they just need a C/C++ compiler, Java 8, and Bazel and they're off to the races. This is great since it also means I can create a Docker container that has Bazel installed and do CI/CD really easily.
I could sacrifice this ease and just say that you need to have libraries like glfw installed before attempting to compile the game but that brings the whole which version is installed and how is it all configured problem back up that Bazel is supposed to help solve.
Surely there is a simpler solution and I'm overthinking this?
If the glfw project has no BUILD files, then you have the following options:
Build glfw inside a genrule.
If glfw supports some other build system like make, you could create a genrule that runs the tool. This approach has obvious drawbacks, like the not-to-be-underestimated impracticality of having to declare all inputs of that genrule, but it'd be the simplest way of Bazel'izing glfw.
Pre-build glfw.o and check it into your source tree.
You can create a cc_library rule for it, and put the .o file in the srcs. Even though this solution is the least flexible of all because you not only restrict the target platform to whatever the .o was built for, but also make it harder to reproduce the whole build, the benefits are sometimes worth the costs.
I view this approach as a last resort. Even in Bazel's own source code there's one cc_library.srcs that includes a raw object file, because it was worth it, as the commit message of 92caf38 explains.
Require that glfw be installed.
You already considered this option. Some people may prefer this to the other approaches.
I'm trying to start a purescript project that uses the purescript signal and drawing libraries. When I try to bower install with both of them as dependencies, bower's interactive prompt asks to resolve several dependency versions. I admittedly pick the versions somewhat arbitrarily since most of the issues are several layers of dependencies deep. Then when I pulp build a basic hello world piece of code that doesn't even use the dependencies, pulp gives around 40 errors depending on which versions I selected.
How do I resolve these dependency versions correctly? Sifting through forty errors and all of the dependencies for both projects doesn't seem feasible. Is there some non-arbitrary way of picking the versions that will resolve the conflicts? From the choices it gave me, it didn't seem like there was any clear resolution. Or are their dependencies too conflicting and you just can't use them together?
This may actually be more of an question about bower, but it may also be something about purescript and pulp that I am just unaware of. They both seem like libraries that should be able to work together, and I can't imagine that other people are having too much difficulty using two or more libraries in a single project.
Purescript 0.10 has breaking changes. If you have the latest compiler, it's likely that it's not compatible with some libraries. From what I see, signal lib is updated but the drawing is not. You can update the lib and use your own version. Or, send a PR, Phil and purescript community in general, is really fast on responding to pull requests.
Another temporary solution is to use the old compiler (0.9) and wait a couple months till everything is back to normal.
also see 0.10 migration guide.
I'm writing a script that will install some of the optional postgresql modules. Is there a way to programmatically figure out where the contrib directory is or do I have to prompt for the path? I've looked at a few examples and it seems inconsistent; doesn't appear in pg_config.
(The script might be run on OS X or linux, and I can't make assumptions about how postgresql was installed).
There is no easy works everywhere answer to this, so it's a matter of how much intelligence you want to try and put into your installer. And for some people it will be impossible to do what you hope for, the best you'll be able to do is offer guidance about what's missing. There is a lot of variation on packaging here between Linux distributions, versions of PostgreSQL, and the various ways you can install PostgreSQL on OS X (MacPorts, homebrew, etc.)
First off, only source code installs will have a contrib directory with source code in it that allows building the optional modules. In the packaged builds for Linux, all the contrib binaries may only be available via an optional package, called something like postgresql-contrib. That's the only way to make the optional modules that come with the database available: install the package the binaries are included in. You may see some variation in the OS X builds here too.
If you want to install extensions (what these are officially called as of PostgreSQL 9.1 now, rather than 'modules') using binaries you provide instead, what you then want to know is where to put the resulting shared libraries and matching SQL files that reference them. What pgconfig returns for pkglibdir will tell you where the binaries go, while sharedir points toward the default place to put the SQL. Providing binaries is a losing game though; the job of syncing with every platform to build them is a huge one.
And here are the sort of additional complications you'll run into in this area, if you wanted to ship source code and try to build things yourself in an automatic way:
PostgreSQL 9.1 now installs these using the CREATE EXTENSION
mechanism, so you'll need to handle both the pre-9.1 method and the
new one introduced there.
Not all PostgreSQL installations will have
pg_config. It's considered a development tool, and which package it
gets bundled with (and whether that package is mandatory or not)
varies. Debian/Ubuntu put it into the optional liqpq-dev, RedHat
derived RPMs have it in postgresql-devel or
postgresql-[version]-devel.
Due to pg_config being necessary for
compiling the new 9.1 extensions, packagers have started
reconsidering where pg_config goes; it's considered a lot more
important now than it used to be. 9.1 or later packages might alter which package it's contained in. That doesn't really change what you can and can't do though. It just impacts what advice you might offer for correcting situations your program can't deal with.
I've been describing the standard
Linux packaging here when I talk about that OS. There are also installers for both Linux and
OS X from EnterpriseDB, what they call their "one-click installers".
These use a different standard altogether for what people do and don't get installed in this area. I don't follow the commercial packaging to know what is actually different, but it's another variable you can expect people to encounter.
Recent OS X versions may
have some system PostgreSQL components floating around too. No idea
yet how this handle extensions though.
Basically, all three of version/packager/platform can vary how this will work, and the idea that you'll find any solution that handles even the majority of those permutations is optimistic. Installing extensions is known to be difficult in PostgreSQL, which is one thing that motivated all the 9.1 changes to turn it into a simple CREATE EXTENSION for many of them. But for now, those changes have just added another whole set of variation into the mix, actually making this harder during the transition period. It will be a while until PostgreSQL versions supporting that are the only ones in popular use.
With the desire to be able to reproduce a given revision of a project that is utilizing 3rd party visual component packages, what goes in SVN and what's the best way to implement/structure the SVN repos?
For non-visual components, the rule seems simple to ensure no reliance on outside repos - "no svn-externals reference to any outside repo allowed". I have a shared repo that I control, which is the only 'svn-externals' reference allowed. This makes it easy to implement and share these types of runtime itemss with sourcecode in different SVN projects. Any reference this internal shared repo is by 'svn-externals' using a specific revision number.
Visual packages seem to go counter to being able to be version controlled easily as they may have to be reinstalled at each revision. How to best create a SVN project which is able to be recreated later at a specific revision number...is there a recommended solution?
Previously we didn't worry about 3rd party components as they don't change often and we never had a real good solution. I was wondering if others have figure out the best way to handle this problem as I'm doing a spring cleaning/internal reorganization and wanted to do it 'better' than before.
Technically, the RTL/VCL source should also be in the SVN repo as well (if there's a Delphi hotfix/service pack released.)
My solution will likely be to create a virtual machine with a particular release of the Delphi environment with all visual controls installed. As we add/update visual controls, or update Delphi with hotfixes/service packs then we create a new version of the virtual machine. We then store an image of this VM revision on a shelf somewhere. Is this what you do? Does the Delphi activation/licensing work well (or at all) in this scenario?
Thanks,
Darian
You can prepare "start IDE" (and possibly "build") scripts for your projects and maintain them as project evolves in repository.
Regardless of your decision about keeping components in separate repositories and using externals, or including them in a single repository with possible branching, you should also include compiled bpl files for every component build and for every branch prepared for a specific Delphi version.
You should definitely try to keep most (if not all) of paths relative, in a worst case use environment variables to point to your root project dir.
Start IDE script allows you to keep each project and Delphi version environment spearately configured on a single Windows installation.
It should include necessary registry keys for your project and Delphi:
Windows Registry Editor Version 5.00
[-${DelphiRegKey}\Disabled Packages]
[-${DelphiRegKey}\Known Packages]
[-${DelphiRegKey}\Library]
[${DelphiRegKey}\Known Packages]
"$(BDS)\\Bin\\dclstd${CompilerVersion}.bpl"="Borland Standard Components"
"$(BDS)\\Bin\\dclie${CompilerVersion}.bpl"="Internet Explorer Components"
"$(BDS)\\Bin\\dcldb${CompilerVersion}.bpl"="Borland Database Components"
(...)
"${CustomComponentPack}"="Custom Components"
[${DelphiRegKey}\Library]
"Search Path"="${YourLibrarySourceFolder1};${YourLibrarySourceFolder2}"
(...)
You can then prepare batch file:
regedit /s project.reg
%DelphiPath%\bin\bds -rProjectRegKey Project.dpr
Where ${DelphiRegKey} is HKEY_CURRENT_USER\Software\Borland(or CodeGear in newer versions)\ProjectRegKey.
Basically it is easier when you will dump your current working configuration from registry, strip it from unnecessary keys, change paths to relative and then adapt to make it work with your project.
In such configuration, switching between projects and their branches which have different sets of components (and/or possibly using different Delphi version) is a matter of checking out a repository only and running the script.
Fortunately for us, we don't have to worry about a hotfix/service pack; we're still on Delphi 5. :D
Sigh, there was a time when an entire application (settings and all) would exist within a single directory - making this a non-issue. But, the world has moved on, and we have various parts of an application scattered all over the place:
registry
Windows\System
Program Files
Sometimes even User folders in "Application Data" or "Local Settings"
You are quite right to consider the impact of hotfixes/service packs. It's not only RTL/VCL that could be affected, but the compiler itself could have been slightly changed. Note also that running on the same line of thought, even when you upgrade Delphi versions, you need to build using the correct version. Admittedly this is a little easier because you can run different Delphi versions alongside each other.
However, I'm going to advise that it's probably not worth going to too much effort. Remember, working on old versions is always more expensive than working on the current version.
Ideally you want all your dev to be be on main branch code, you want to minimise patch-work on older versions.
So strive to keep the majority of your users on the latest version as much as possible.
Admittedly this isn't always possible.
You wouldn't want to jump over to the 'new version' without some testing first in any case.
Certain agile processes do tend to make this easier.
By using a separate build machine or VM, you already have a measure of control.
TIP: I would also suggest that the build process automtically copy build output to a different machine, or at least a different hard-drive.
Once you're satisfied with the service pack, you can plan when you want to roll it to your build machine.
It is extremely important to keep record of the label at which the build configuration changed. (Just in case.)
If your build scripts are also kept in source control, this happens implicitly.
When you've rolled out the hotfix/service pack, fixes to older versions should be actively discouraged.
Of course, they probably can't be eliminated, but if it's rare enough, then even manual reconfiguration could be feasible.
Instead of a VM option to keep your old configuration, you can also consider drive-imaging.
To save on the $$$ of VMWare LabManager, look for a command-line driven VM Player.
You might have to keep 2 "live" machines/VMs, but should never need more than that.
It's okay for an automatic build script to fail because the desired configuration isn't available. This will remind you to set it up manually.
Remember, working on old versions is always more expensive than working on the current version.
Third Party Packages
We went to a little bit more effort here. One of our main motivations though was the fact that we use about 8 third party packages. So doing something to standardise this in itslef made sense. We also decided running 8 installation programs was a PITA, so we devised an easy way to manually install all required packages from source-control.
Key Considerations
The build environment doesn't need any packages installed, provided the object and/or source files are accessible.
It would help if developers could fairly easily ensure they're building with the same version of third party libraries when necessary.
However, dev environments usually must install packages into the IDE.
This can sometimes cause problems with source compatibility.
For example new properties that get written to IDE maintained files.
Which of course brings us back to the second point.
Since Third Party packages are infrequently updated, they are placed within a slightly different area of source-control.
But, NB must still be referenced via relative paths.
We created the following folder structure:
...\ThirdParty\_DesignTimePackages //The actual package files only are copied here
...\ThirdParty\_RunTimePackages //As above, for any packages "required" by those above
...\ThirdParty\Suite1
...\ThirdParty\Suite2
...\ThirdParty\Suite3
As a result of this it's quite easy to configure a new environment:
Get latest version of all ThirdParty files.
Add _DesignTimePackages and _RunTimePackages to Windows Path
Open Delphi
Select Install Components
Select all packages from _DesignTimePackages.
Done!
Edit: Darian was concerned about the possibility of errors when switching switching versions of Design Packages. However, this approach avoids those kinds of problems.
By adding _DesignTimePackages and _RunTimePackages to the Windows Path, Delphi will always find required packages in the same place.
As a result, you're less likely to encounter the 'package nightmare' of incompatible versions.
Of course, if you do something silly like rebuild some of your packages and check-in the new version, you can expect problems - no matter what approach you follow.
I usually structure my repository in SVN like this:
/trunk/app1
/trunk/comp/thirdparty1
/trunk/comp/thirdparty2
/trunk/comp/thirdparty3...
I have, right in the root folder (trunk) a project group (.groupproj, or .bpg on old delphi) that contains all my components. (allcomponents.groupproj).
Installing on a new machine, means opening that package, and installing the designtime components. That's a drag on all versions of Delphi older than 2010, but 2010 and XE have a lovely feature so you can see at a glance, which components are designtime components.
I also, sometimes, will save myself the trouble of installing those components by hand, by making a build.bat file, and a regcomponents.bat file. The regcomponents just runs regedit , and imports the keys needed to register all those components, after build.bat has built them, and everything else.
When you move up from one delphi version to another, it's sure good to have both a batch and reg file, and a group project, to help you. Especially if you have to go through and do a lot of opening of project/packages and saving them as MyComponent3.dpk instead of MyComponent2.dpk, or updating the package extension from 150 to 160, or whatever your packages do.