I'm trying to use Make to ... make modular Dockerfiles. Long story short, I want to centralize certain elements and make the composable and reusable, like classes and functions really, but the Dockerfile syntax does not - and according to the developers, will not - offer any facilities in the image of C's #include or similar composability solutions. Not to worry, #include and friends to the rescue!
Except...
I have the following Makefile in my project:
BUILD_DIR := ${CI_PROJECT_DIR}/build
TEMPLATE_FILES := $(shell find ${CI_PROJECT_DIR} -name '*.build')
TEMPLATE_FILENAMES := $(foreach file,$(TEMPLATE_FILES),$(BUILD_DIR)/$(notdir $(file)).built)
BUILT_TEMPLATES := $(TEMPLATE_FILENAMES:.build.built=.built)
DOCKER_FILES := $(shell find ${CI_PROJECT_DIR} -name '*.Dockerfile')
DOCKER_OBJS := $(foreach file,$(DOCKER_FILES),$(BUILD_DIR)/$(notdir $(file)))
all: $(BUILT_TEMPLATES) $(DOCKER_OBJS)
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR) # build any templated Dockerfiles
cpp -E -P -o $(BUILD_DIR)/$(notdir $#) -I ${CI_PROJECT_DIR}/modules $<
sed -i 's/__NL__ /\n/g' $(BUILD_DIR)/$(notdir $#)
$(BUILD_DIR)/%.Dockerfile: $(DOCKER_FILES) $(BUILD_DIR)
cp $< $(BUILD_DIR)/$(notdir $(#))
$(BUILD_DIR):
mkdir -p $(BUILD_DIR)
.PHONY: clean
clean:
-rm -r $(BUILD_DIR)
The objective is to run the templated Dockerfiles through GCC to compile the #includes in them into proper Docker instructions, and just copy the rest of the files. Sounds simple enough.
Except that it looks like all the target files are "offset" from their sources - like the file names are correct, but the contents are from a file elsewhere in the list, and with no discernible order either.
One thing that I'm fairly sure is wrong - but even more wrong otherwise - is the line
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR) # build any templated Dockerfiles
By all manuals and documentation, it ought to be
$(BUILD_DIR)/%.built: %.build $(BUILD_DIR) # build any templated Dockerfiles
but that's even worse, because then Make just says make: *** No rule to make target '/docker/build/runner-dart-2-18-firebase.built', needed by 'all'. Stop.
I'm out of ideas here, along with my limited knowledge of Make. What am I missing to make Make make - sorry - my Dockerfiles?
This line:
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR)
Says that if make wants to build a target that matches that pattern, and it can find all the prerequisites, then the pattern rule matches and the recipe can be used. Let's ignore BUILD_DIR (note that it's always a bad idea to list a directory as a prerequisite, but that's not causing this problem). Suppose TEMPLATE_FILES is set to the value ./foo/foo.build ./bar/bar.build. Now the above rule expands to:
./build/%.built: ./foo/foo.build ./foo/bar.build ./build
What is the recipe?
cpp -E -P -o $(BUILD_DIR)/$(notdir $#) -I ${CI_PROJECT_DIR}/modules $<
First it's always wrong to create a file that is not exactly $# so you should use just $# not $(BUILD_DIR)/$(notdir $#). But more importantly, what will $< be set to? It is always set to the first prerequisite, and the first prerequisite is always ./foo/foo.build. So every time you run this recipe, regardless of which .built file you're trying to create, you will always be preprocessing the first .build file.
Your idea that you want this instead:
$(BUILD_DIR)/%.built: %.build $(BUILD_DIR)
is correct, in general. Why do you get the error? Because if you are trying to build the target ./build/foo.built, then the stem (part that matches %) is foo. Then make will look to see if the prerequisite foo.build exists or can be created, because you said the prerequisite is %.build. That file does NOT exist and CANNOT be created (make doesn't know how to create it), because the file is ./foo/foo.build not foo.build which is a totally different file.
You have three options. You can either write separate rules for each source directory:
$(BUILD_DIR)/%.built: foo/%.build
...
$(BUILD_DIR)/%.built: bar/%.build
...
Or, you can change your generated files so they are not all in the same directory but instead keep the source directory structure; you would change this:
TEMPLATE_FILENAMES := $(foreach file,$(TEMPLATE_FILES),$(BUILD_DIR)/$(notdir $(file)).built)
BUILT_TEMPLATES := $(TEMPLATE_FILENAMES:.build.built=.built)
to just this:
BUILT_TEMPLATES := $(patsubst %.build,$(BUILD_DIR)/%.built,$(TEMPLATE_FILES))
then create the output directory as part of the recipe:
#mkdir -p $(#D)
cpp -E -P -o $# -I ${CI_PROJECT_DIR}/modules $<
sed -i 's/__NL__ /\n/g' $#
Or finally, you could use VPATH to tell make what directories to look in to find the *.build files:
VPATH := $(sort $(dir $(TEMPLATE_FILES)))
(note, you should choose only one of these options).
Related
Suppose there is a directory /opt/dvp/gmake/, and in this directory there are three files: a.mk, a-b.mk and a-b-c.mk.
Just to clarify, in a-b-c.mk there are definitions and configurations to a specific program, library, etc., which I generic call artifact. In a-b.mk there are difinitions to all artifacts in a-b group, and in a.mk there are definitions to all artifacts in a group.
I want to write a makefile where I call make ARTIFACT=/opt/dvp/gmake/a-b-c.mk, and in the makefile to automatically -include /opt/dvp/gmake/a-b-c.mk, -include /opt/dvp/gmake/a-b.mk and -include /opt/dvp/gmake/a.mk.
I looked at here, here, here, at the GNU Make documentantion, tried many forms of foreach in combination with call and shell functions, but I was not able to do it.
TIA
I'm not really getting how exactly the programs etc get hold of the "specifics" in the .mk-files. Perhaps adding a sample of the rules to the question would help.
One way might be to use . as separator instead of -, and remove suffixes one by one using basename:
# $ make foo.bar.baz
# foo.bar.baz
# foo.bar
# foo
%:
#echo $#
#echo $(basename $#)
#echo $(basename $(basename $#))
In the case of your files (other naming schemes can make it prettier):
-include $(ARTIFACT)
# remove .mk, then .c, then append .mk
-include $(basename $(basename $(ARTIFACT))).mk
# remove .mk, then .c, then .b, then append .mk
-include $(basename $(basename $(basename $(ARTIFACT)))).mk
I have been working on a custom patch for my car with MIB2 unit for some time and would like to try my modified MIBRoot file.
The command I used to repack everything is:
mkxfs -t ifs -nn -o ./ -r / ./mkifs_attributes.txt ./ifs_extracted ./patched_ifs.ifs
But the problem is that the repackaging is not working properly, because the mkifs_attribute.txt file used is probably not fine for my MIB2 or I have a version that is not correct.
In fact, when I do a:
diff -r ./original/ifs_extracted ./repack/ifs_extracted
all files in ./repack/ifs_extracted are different from the originals, when instead only the MIBRoot file should be different.
my mkifs_attribute contain this:
#Porsche PCM4/MIB2 ifs-root (stage 2) attributes file
#Before packing FS: EXPORT QNX_TARGET="/"
#Command to pack: mkxfs -t ifs -nn -o ./ -r / attributes.txt /unpacked_fs /packed_fs.ifs
[-followlink] #Do not resolve links
[compress=2] #LZO Compression
[-bigendian] #Little Endian (ARMLE)
[-autolink] #Do not auto link shared objects
[perms=777] #chmod 777
[uid=0 gid=0] #chown root:root
#You can add a script to be executed after mount here, one line only
#[+script] .script = {
#touch /dev/shmem/AudioFadedIn; touch /dev/shmem/production_mode}
#Symlinks. Add lines here to match your dumpifs symlinks
#[type=link] dest_file=source_file
[type=link] /lib/SPC_configuration.so=/lib/libSPC_configuration.so
[type=link] /lib/AudioConfig_ARM.so=/lib/libAudioConfig_ARM.so
[type=link] /lib/AudioConfig_DSP.so=/lib/libAudioConfig_DSP.so
[type=link] /usr/bin/flashunlock=/usr/bin/flashlock
[type=link] /usr/bin/libi2c-inic-master.so=/lib/libi2c-inic-master.so.1.0
[type=link] /lib/librdshbfpga.so.1.0.0=/lib/librdshbfpga.so
[type=link] /lib/libSysMoCCAFrameworkSharedSo.so=/lib/libSysMoCCAFrameworkSharedSo.so.6
[type=link] /lib/AudioConfig_DSP.so=/lib/libAudioConfig_DSP.so
[type=link] /lib/liba2itodspipc.so=/lib/liba2itodspipc.so.1
Or am i wrong something?
If anyone has a correct mkxfs and mkxfs_attribute.txt could you please provide it to me so I can give it a try?
(From https://groups.google.com/d/msg/bazel-discuss/XrtKLhH1bgI/B9xZn_aVAAAJ)
In our project that uses Bazel for building, I'm using the remote cache (--spawn_strategy=remote), and we are having to do some fine-tuning in order to turn off the cache for certain actions. For example, when we generate tar files, we don't want to use the remote cache, because (a) building tar files locally is just as fast as downloading them, and (b) some of our tar files can be really enormous.
So I'd like to have our .bazelrc files specify a different strategy for some actions, e.g. --strategy=PackageTar=standalone.
Here's the tricky part: In order to override the strategy, you need to know the mnemonic. E.g. for the pkg_tar rule, the relevant action has the mnemonic PackageTar. I found that in the bazel source. Some others are quite a bit trickier to figure out.
Is there any way I can get bazel to tell me the mnemonics of the actions it is executing? I looked into a variety of options, such as --profile, --explain, --verbose_explanations, --subcommands, but couldn't figure out a way.
You can do this with bazel aquery:
$ cat BUILD
load("#bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
pkg_tar(
name = "my_archive",
srcs = ["hello.txt"],
)
$ bazel aquery :my_archive 2>/dev/null
action 'Writing file my_archive.args'
Mnemonic: FileWrite
Target: //:my_archive
Configuration: k8-fastbuild
ActionKey: 3dafce6be7ba0023b7eaae485085f977
Inputs: []
Outputs: [bazel-out/k8-fastbuild/bin/my_archive.args]
action 'PackageTar my_archive.tar'
Mnemonic: PackageTar
Target: //:my_archive
Configuration: k8-fastbuild
ActionKey: 86bd5d0e112232cf0224fd0e3534f553
Inputs: [<snip>]
Outputs: [bazel-out/k8-fastbuild/bin/my_archive.tar]
Command Line: (exec bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar \
--flagfile \
bazel-out/k8-fastbuild/bin/my_archive.args)
Note the Mnemonic: PackageTar line in the second action.
You can use the --output=textproto option to bazel aquery to
get machine-readable output.
You can also use queries like bazel aquery 'outputs(".*\.tar", //...)'
to narrow down the action graph; consult the aquery docs for more
details.
Unfortunately Bazel doesn't seem to report mnemonics anywhere.
You can find a list of available mnemonics by grepping for them in the source.
In Skylark rules:
cd <bazel_src_dir>
find -name '*.bzl' -not -path '*.git/*' -and -not -path '*/test/*' -type f | xargs grep 'mnemonic\s*='
In native rules:
cd <bazel_src_dir>
find src/main/java -name '*.java' -type f | xargs grep -A1 'String getMnemonic()'
I seem to have hit a quirk regarding Makefiles, list processing, and pattern rules. When my source is installed in a directory whose name includes '%', it fails to build correctly. I've boiled it down to a simple test case anybody could run.
I have the following Makefile:
INSTALL_DIR=$(shell pwd)/idir
INSTALL_TREE =
INSTALL_TREE += $(INSTALL_DIR)/dir1
INSTALL_TREE += $(INSTALL_DIR)/dir2
INSTALL_TREE += $(INSTALL_DIR)/dir3
create_tree: $(INSTALL_TREE)
$(INSTALL_TREE):
mkdir -p $#
I put this in a directory called test1, and when I run it I get exactly what I want... all the directories listed in INSTALL_TREE are created:
% make -n
mkdir -p /home/christopher.arguin/test/test1/idir/dir1
mkdir -p /home/christopher.arguin/test/test1/idir/dir2
mkdir -p /home/christopher.arguin/test/test1/idir/dir3
Now use the exact same Makefile in a directory called test%2, and this is what I get:
% make -n
mkdir -p /home/christopher.arguin/test/test2%/idir/dir1
It stops after the first directory. Interestingly, if the first directory does not have a % sign, they all get generated..., e.g.,
INSTALL_TREE =
INSTALL_TREE += ./idir/dir4
INSTALL_TREE += $(INSTALL_DIR)/dir1
INSTALL_TREE += $(INSTALL_DIR)/dir2
INSTALL_TREE += $(INSTALL_DIR)/dir3
gives me:
mkdir -p dir4
mkdir -p /home/christopher.arguin/test/test2%/idir/dir1
mkdir -p /home/christopher.arguin/test/test2%/idir/dir2
mkdir -p /home/christopher.arguin/test/test2%/idir/dir3
The other interesting thing is that the files names are correct... if the % were being misinterpreted as a prefix rule or something you might expect some substitution, but if I let it create the directories it does the absolute right thing.
I tried escaping the % character, but to no avail. Adding this code:
PERCENT := %
ITREE = $(subst $(PERCENT),$(PERCENT)$(PERCENT),$(INSTALL_TREE))
create_tree2: $(ITREE)
$(ITREE):
mkdir -p $#
Just yielded
mkdir -p /home/christopher.arguin/test/test2%%/idir/dir1
The substitution worked, but it didn't actually escape the percent sign. Neither did replacing "%" with "$(PERCENT)", as the filename just contained $(PERCENT) in it afterward.
I found two related questions,
Does GNU Make support '%' in a filename?
and How to correctly escape "%" sign when using pattern rules and patsubst in GNU make?
but neither of those suffered the basic problem I have... that the directory my source is in is out of my control.
Background:
The reason is this an issue for me is that I am trying to migrate to Jenkins Multibranch Pipeline. When Jenkins detects a commit, it creates a workspace based on the name of the branch. If your branch naming convention happens to have slashes in it, Jenkins does the right thing and converts those to "%2F". My makefiles are running afoul of those.
A target containing % defines a pattern rule, and pattern rules with multiple targets are processed differently:
https://www.gnu.org/software/make/manual/make.html#Pattern-Intro
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets.
You can fix this by quoting % when defining the path rule. Note that this isn't necessary for the dependencies of create_tree as that is a regular rule, in fact it won't work if you do, as make will look for targets with a literal \%.
INSTALL_DIR := $(abspath .)/idir
INSTALL_TREE += $(INSTALL_DIR)/dir1
INSTALL_TREE += $(INSTALL_DIR)/dir2
INSTALL_TREE += $(INSTALL_DIR)/dir3
.PHONY: create_tree
create_tree: $(INSTALL_TREE)
$(subst %,\%,$(INSTALL_TREE)):
mkdir -p $#
I'm creating a tarball of a large codebase managed in ClearCase. Every directory has a sub-directory named ".CC". I'd like to exclude these from my tarball.
I've found Excluding directory when creating a .tar.gz file, but excluding that would appear to require passing each and every .CC directory on the commndline. This is impractical in my case.
Is there a way to exclude directories that meet a particular pattern?
EDIT:
I am not asking how to exclude a specific finite list of directories. I am asking how to exclude all directories that end in a particular pattern.
Instead of manually typing --exclude 'root/a/.CC' --exclude 'root/b/.CC' ... you can type $(find root -type d -name .CC -exec echo "--exclude \'{}\'" \;|xargs)
You can use whatever patterns find supports, or even use something like grep inbetween find and xargs.
The following bash script should do the trick. It uses the answer given by #Marcus Sundman.
#!/bin/bash
echo -n "Please enter the name of the tar file you wish to create with out extension "
read nam
echo -n "Please enter the path to the directories to tar "
read pathin
echo tar -czvf $nam.tar.gz
excludes=`find $pathin -iname "*.CC" -exec echo "--exclude \'{}\'" \;|xargs`
echo $pathin
echo tar -czvf $nam.tar.gz $excludes $pathin
This will print out the command you need and you can just copy and paste it back in. There is probably a more elegant way to provide it directly to the command line.
*.CC could be exchanged for any other common extension and this should still work.