Suppose there is a directory /opt/dvp/gmake/, and in this directory there are three files: a.mk, a-b.mk and a-b-c.mk.
Just to clarify, in a-b-c.mk there are definitions and configurations to a specific program, library, etc., which I generic call artifact. In a-b.mk there are difinitions to all artifacts in a-b group, and in a.mk there are definitions to all artifacts in a group.
I want to write a makefile where I call make ARTIFACT=/opt/dvp/gmake/a-b-c.mk, and in the makefile to automatically -include /opt/dvp/gmake/a-b-c.mk, -include /opt/dvp/gmake/a-b.mk and -include /opt/dvp/gmake/a.mk.
I looked at here, here, here, at the GNU Make documentantion, tried many forms of foreach in combination with call and shell functions, but I was not able to do it.
TIA
I'm not really getting how exactly the programs etc get hold of the "specifics" in the .mk-files. Perhaps adding a sample of the rules to the question would help.
One way might be to use . as separator instead of -, and remove suffixes one by one using basename:
# $ make foo.bar.baz
# foo.bar.baz
# foo.bar
# foo
%:
#echo $#
#echo $(basename $#)
#echo $(basename $(basename $#))
In the case of your files (other naming schemes can make it prettier):
-include $(ARTIFACT)
# remove .mk, then .c, then append .mk
-include $(basename $(basename $(ARTIFACT))).mk
# remove .mk, then .c, then .b, then append .mk
-include $(basename $(basename $(basename $(ARTIFACT)))).mk
Related
I'm trying to use Make to ... make modular Dockerfiles. Long story short, I want to centralize certain elements and make the composable and reusable, like classes and functions really, but the Dockerfile syntax does not - and according to the developers, will not - offer any facilities in the image of C's #include or similar composability solutions. Not to worry, #include and friends to the rescue!
Except...
I have the following Makefile in my project:
BUILD_DIR := ${CI_PROJECT_DIR}/build
TEMPLATE_FILES := $(shell find ${CI_PROJECT_DIR} -name '*.build')
TEMPLATE_FILENAMES := $(foreach file,$(TEMPLATE_FILES),$(BUILD_DIR)/$(notdir $(file)).built)
BUILT_TEMPLATES := $(TEMPLATE_FILENAMES:.build.built=.built)
DOCKER_FILES := $(shell find ${CI_PROJECT_DIR} -name '*.Dockerfile')
DOCKER_OBJS := $(foreach file,$(DOCKER_FILES),$(BUILD_DIR)/$(notdir $(file)))
all: $(BUILT_TEMPLATES) $(DOCKER_OBJS)
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR) # build any templated Dockerfiles
cpp -E -P -o $(BUILD_DIR)/$(notdir $#) -I ${CI_PROJECT_DIR}/modules $<
sed -i 's/__NL__ /\n/g' $(BUILD_DIR)/$(notdir $#)
$(BUILD_DIR)/%.Dockerfile: $(DOCKER_FILES) $(BUILD_DIR)
cp $< $(BUILD_DIR)/$(notdir $(#))
$(BUILD_DIR):
mkdir -p $(BUILD_DIR)
.PHONY: clean
clean:
-rm -r $(BUILD_DIR)
The objective is to run the templated Dockerfiles through GCC to compile the #includes in them into proper Docker instructions, and just copy the rest of the files. Sounds simple enough.
Except that it looks like all the target files are "offset" from their sources - like the file names are correct, but the contents are from a file elsewhere in the list, and with no discernible order either.
One thing that I'm fairly sure is wrong - but even more wrong otherwise - is the line
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR) # build any templated Dockerfiles
By all manuals and documentation, it ought to be
$(BUILD_DIR)/%.built: %.build $(BUILD_DIR) # build any templated Dockerfiles
but that's even worse, because then Make just says make: *** No rule to make target '/docker/build/runner-dart-2-18-firebase.built', needed by 'all'. Stop.
I'm out of ideas here, along with my limited knowledge of Make. What am I missing to make Make make - sorry - my Dockerfiles?
This line:
$(BUILD_DIR)/%.built: $(TEMPLATE_FILES) $(BUILD_DIR)
Says that if make wants to build a target that matches that pattern, and it can find all the prerequisites, then the pattern rule matches and the recipe can be used. Let's ignore BUILD_DIR (note that it's always a bad idea to list a directory as a prerequisite, but that's not causing this problem). Suppose TEMPLATE_FILES is set to the value ./foo/foo.build ./bar/bar.build. Now the above rule expands to:
./build/%.built: ./foo/foo.build ./foo/bar.build ./build
What is the recipe?
cpp -E -P -o $(BUILD_DIR)/$(notdir $#) -I ${CI_PROJECT_DIR}/modules $<
First it's always wrong to create a file that is not exactly $# so you should use just $# not $(BUILD_DIR)/$(notdir $#). But more importantly, what will $< be set to? It is always set to the first prerequisite, and the first prerequisite is always ./foo/foo.build. So every time you run this recipe, regardless of which .built file you're trying to create, you will always be preprocessing the first .build file.
Your idea that you want this instead:
$(BUILD_DIR)/%.built: %.build $(BUILD_DIR)
is correct, in general. Why do you get the error? Because if you are trying to build the target ./build/foo.built, then the stem (part that matches %) is foo. Then make will look to see if the prerequisite foo.build exists or can be created, because you said the prerequisite is %.build. That file does NOT exist and CANNOT be created (make doesn't know how to create it), because the file is ./foo/foo.build not foo.build which is a totally different file.
You have three options. You can either write separate rules for each source directory:
$(BUILD_DIR)/%.built: foo/%.build
...
$(BUILD_DIR)/%.built: bar/%.build
...
Or, you can change your generated files so they are not all in the same directory but instead keep the source directory structure; you would change this:
TEMPLATE_FILENAMES := $(foreach file,$(TEMPLATE_FILES),$(BUILD_DIR)/$(notdir $(file)).built)
BUILT_TEMPLATES := $(TEMPLATE_FILENAMES:.build.built=.built)
to just this:
BUILT_TEMPLATES := $(patsubst %.build,$(BUILD_DIR)/%.built,$(TEMPLATE_FILES))
then create the output directory as part of the recipe:
#mkdir -p $(#D)
cpp -E -P -o $# -I ${CI_PROJECT_DIR}/modules $<
sed -i 's/__NL__ /\n/g' $#
Or finally, you could use VPATH to tell make what directories to look in to find the *.build files:
VPATH := $(sort $(dir $(TEMPLATE_FILES)))
(note, you should choose only one of these options).
Content of testfile.txt
/path1/abc.txt
/path2/abc.txt.1
/path3/abc.txt123
Content of pattern.txt
abc.txt$
Bash Command
grep -i -f pattern.txt testfile.txt
Output:
/path1/abc.txt
This is a working solution, but currently the $ in the pattern is manually added to each line and this edited pattern file is uploaded to users. I am trying to avoid the manual amendment.
Alternate solution to loop and read line by line, but required scripting skills or upload scripts to user environment.
Want to keep the original pattern files in an audited environment, users just login and run simple cut-n-paste commands.
Any one liner solution?
You can use sed to add $ to pattern.txt and then use grep, but you might run into issues due to regexp metacharacters like the . character. For example, abc.txt$ will also match abc1txt. And unless you take care of matching only the basename from the file path, abc.txt$ will also match /some/path/foobazabc.txt.
I'd suggest to use awk instead:
$ awk '!f{a[$0]; next} $NF in a' pattern.txt f=1 FS='/' testfile.txt
/path1/abc.txt
pattern.txt f=1 FS='/' testfile.txt here a flag f is set between the two files and field separator is also changed to / for the second file
!f{a[$0]; next} if flag f is not set (i.e. for the first file), build an array a with line contents as the key
$NF in a for the second file, if the last field matches a key in array a, print the line
Just noticed that you are also using -i option, so use this for case insensitive matching:
awk '!f{a[tolower($0)]; next} tolower($NF) in a'
Since pattern.txt contains only a single pattern, and you don't want to change it, since it is an audited file, you could do
grep -i -f "$(<pattern.txt)'$' testfile.txt
instead. Note that this would break, if the maintainer of the file one day decided to actually write there a terminating $.
IMO, it would make more sense to explain to the maintainer of pattern.txt that he is supposed to place there a simple regular expression, which is going to match your testfile. In this case s/he can decide whether the pattern really should match only the right edge or some inner part of the lines.
If pattern.txt contains more than one line, and you want to add the $ to each line, you can likewise do a
grep -i -f <(sed 's/$/$/' <pattern.txt) testfile.txt
As the '$' symbol indicates pattern end. The following script should work.
#!/bin/bash
file_pattern='pattern.txt' # path to pattern file
file_test='testfile.txt' # path to test file
while IFS=$ read -r line
do
echo "$line"
grep -wn "$line" $file_test
done < "$file_pattern"
You can remove the IFS descriptor if the pattern file comes with leading/trailing spaces.
Also the grep option -w matches only whole word and -n provides with line number.
I'm creating a tarball of a large codebase managed in ClearCase. Every directory has a sub-directory named ".CC". I'd like to exclude these from my tarball.
I've found Excluding directory when creating a .tar.gz file, but excluding that would appear to require passing each and every .CC directory on the commndline. This is impractical in my case.
Is there a way to exclude directories that meet a particular pattern?
EDIT:
I am not asking how to exclude a specific finite list of directories. I am asking how to exclude all directories that end in a particular pattern.
Instead of manually typing --exclude 'root/a/.CC' --exclude 'root/b/.CC' ... you can type $(find root -type d -name .CC -exec echo "--exclude \'{}\'" \;|xargs)
You can use whatever patterns find supports, or even use something like grep inbetween find and xargs.
The following bash script should do the trick. It uses the answer given by #Marcus Sundman.
#!/bin/bash
echo -n "Please enter the name of the tar file you wish to create with out extension "
read nam
echo -n "Please enter the path to the directories to tar "
read pathin
echo tar -czvf $nam.tar.gz
excludes=`find $pathin -iname "*.CC" -exec echo "--exclude \'{}\'" \;|xargs`
echo $pathin
echo tar -czvf $nam.tar.gz $excludes $pathin
This will print out the command you need and you can just copy and paste it back in. There is probably a more elegant way to provide it directly to the command line.
*.CC could be exchanged for any other common extension and this should still work.
I'm trying to setup a grep command, that searches my current directory, but excludes a directory, only if it's the root directory.
So for the following directories, I want #1 to be excluded, and #2 to be included
1) vendor/phpunit
2) app/views/vendor
I originally started with the below command
grep -Ir --exclude-dir=vendor keywords *
I tried using ^vendor, ^vendor/, ^vendor/, ^vendor, but nothing seems to work.
Is there a way to do this with grep? I was looking to try to do it with one grep call, but if I have to, I can pipe the results to a second grep.
With pipes:
grep -Ir keywords * | grep -v '^vendor/'
The problem with exclude-dir is, it tests the name of the directory and not the path before going into it, so it is not possible to distinguish between two vendor directories based on their depths.
Here is a better solution, which will actually ignore the specified directory:
function grepex(){
excludedir="$1"
shift;
for i in *; do
if [ "$i" != "$excludedir" ]; then
grep $# "$i"
fi
done
}
You use it as a drop-in replacement to grep, just have the excluded dir as the first argument and leave the * off the end. So, your command would look like:
grepex vendor -Ir keywords
It's not perfect, but as long as you don't have any really weird folders (e.g. with names like -- or something), it will cover most use cases. Feel free to refine it if you want something more elaborate.
I have a directory named XYZ which has directories ABC, DEF, GHI inside it. I want to search for a pattern 'writeText' in all *.c in all directories (i.e XYZ, XYZ/ABC, XYZ/DEF and XYZ/GHI)
What grep command can I use?
Also if I want to search only in XYZ, XYZ/ABC, XYZ/GHI and not XYZ/DEF, what grep command can I use?
Thank you!
grep -R --include="*.c" --exclude-dir={DEF} writeFile /path/to/XYZ
-R means recursive, so it will go into subdirectories of the directory you're grepping through
--include="*.c" means "look for files ending in .c"
--exclude-dir={DEF} means "exclude directories named DEF. If you want to exclude multiple directories, do this: --exclude-dir={DEF,GBA,XYZ}
writeFile is the pattern you're grepping for
/path/to/XYZ is the path to the directory you want to grep through.
Note that these flags apply to GNU grep, might be different if you're using BSD/SysV/AIX grep. If you're using Linux/GNU grep utils you should be fine.
You can use the following command to answer at least the first part of your question.
find . -name *.c | xargs grep "writeText"