How do environment variables and macros interact in GNU Make? - environment-variables

In GNU Makefiles, there are often interactions between environment variables and parameters/variables/macros. There are also several different assignment operators, there are "override" variables that can be provided on the command line, and there is an override directive.
What are the precedence rules for these interactions?
Consider this Makefile:
# Use := to avoid infinite recursion of macro expansion
CFLAGS := -g -O2 $(CFLAGS)
hello: src/hello.c
cc $(CFLAGS) -o $# $<
If you invoke CFLAGS=-pipe make hello, you get cc -g -O2 -pipe -o hello src/hello.c. If you invoke make CFLAGS=-pipe hello, you get cc -pipe -o hello src/hello.c. What's the difference?
And if you change the Makefile to:
CFLAGS ?= -g -O2 $(CFLAGS)
hello: src/hello.c
cc $(CFLAGS) -o $# $<
You get cc -pipe -o hello src/hello.c in both cases.

This answer is based on information in the comments of https://stackoverflow.com/a/63187052/2954547, as well as the GNU Make manual.
The precedence is, in order of strongest to weakest, is:
The override directive
Command-line override
=, :=
Environment variable
?=
There are several individual rules that interact to create this precedence:
A macro is created for every environment variable. If e.g. HOME is set, there will be a $(HOME) macro.
If a macro is already defined, = and := overwrite the previous macro definition. This includes environment variables, which are (conceptually) created before the rest of the Makefile is interpreted. So = or := will overwrite an environment variable. But if the environment variable is defined, it can still be referenced on the right-hand side of the assignment:
CFLAGS := -g -O2 $(CFLAGS)
Note that you had to use := in this case, because if you used = you would get a macro that tried to expand itself recursively, forever, at run time.
As per section 6.5 of the manual, VAR ?= val is equivalent to:
ifeq ($(origin VARIABLE), undefined)
VAR = val
endif
Environment variables have their origin set to 'environment', so ?= will always be overridden by an environment variable. Unlike = and :=, if the environment variable is set, its corresponding assignment with ?= will never even be invoked.
Command-line "override variables" override all variables. This means that make VAR=val overrides the environment variable VAR, and the assignments VAR = val, VAR := val, and VAR ?= val.
The override directive overrides everything, including command-line overrides.
Note that GNU Make has the -e/--environment-overrides option, which causes environment variables to override = and :=. This causes them to behave the same as command-line override variables with respect to macro definition , but (I think) the latter will still take precedence.

Related

Is it possible to define CXXFLAGS with respect to the current compiler in bazel?

I'm trying to convert a Make project to Bazel. Currently, makefile detects the compiler being used and sets certain CXXFLAGS accordingly (not all flags are understood by all compilers).
For example:
ifeq ($(shell $(CC) -dM -E -x c++ /dev/null | grep __clang__),)
CXXFLAGS=-DUSE_GCC
else
CXXFLAGS=-DUSE_CLANG
endif
(Similarly code for compiler versions.)
How can I achieve this in bazel? The relation between a compiler and the flags to be used should be defined somewhere within the project (not on the commandline).
I know the --define option, but is there a way to directly relate the configration to the compiler being used?
Using the copts argument in your cc_library or cc_binary will set compiler flags.
https://docs.bazel.build/versions/master/be/c-cpp.html#cc_binary_args

gnu-parallel fails when command involves environment variables

I am trying to chain a few commands with gnu-parallel. Those commands involve environment variables defined before. For instance, consider the command below, which receives a list of *.c files, emit llvm bytecode with clang to .bc files and optimize it into .rbc files with opt.
parallel --halt now,fail=1 'clang -Xclang -disable-O0-optnone -S -c -emit-llvm {} -o {.}.bc ; opt -S -mem2reg {.}.bc -o {.}.rbc ' ::: "${source_files[#]}"
The program above works just fine, but if I change clang by an ambient variable, the command stops working.
parallel --halt now,fail=1 '$COMPILER -Xclang ... ' ::: "${source_files[#]}"
The shell does not expand environment variables inside single quotes, and parallel does not do so either (hence the env_parallel suggestion in the comments).
The easiest solution here is to use different quotes around the variable:
parallel --halt now,fail=1 "$COMPILER"' -Xclang ... ' ::: "${source_files[#]}"
I've left the single quotes around the rest of the command, just in case, but you may not need to. Quotes placed back-to-back like that are automatically concatenated into one argument by the shell.

How is -fPIE passed along to llc?

I'm working on an LLVM backend for a new architecture and we need to have position independent executables. I can pass '-fPIE' on the clang command line, but I don't see any indication of this show up in the resulting LLVM IR. For example, if I run:
clang -v -emit-llvm -fPIC -O0 -S global_dat.c -o global_dat_x86_pic.ll
And then take a look at the resulting global_dat_x86_pic.ll file, I see the following near the bottom:
!0 = !{i32 1, !"PIC Level", i32 2}
Ok, makes sense.
However if I run:
clang -v -emit-llvm -fPIE -O0 -S global_dat.c -o global_dat_x86_pie.ll
I see that the two .ll files are identical. Near the bottom of global_cat_x86_pie.ll I see:
!0 = !{i32 1, !"PIC Level", i32 2}
Which is identical to the case where I ran with -fPIE. There's no indication of "PIE Level" in the .ll file. If this .ll file were passed on to llc how would llc know that -fPIE had been set on the clang command line?
I have run in gdb and see that in fact in the second case with -fPIE on the clang commandline there is an Opts.PIELevel (in $LLVM_HOME/tools/clang/lib/Frontend/CompilerInvocation.cpp) that gets set to 2 (in fact, both Opts.PIELevel and Opts.PICLevel are set to 2 in that case whereas in -fPIC is passed to clang only Opts.PICLevel is set to 2)
This depends on your default target triple, which I can't tell from your question. You can see what happens if you cut off the default architecture (or run a native clang on an arch that does support PIE)
For example, bare X86-64 shows this,
$ clang -c hello.c -target x86_64 -fPIE -emit-llvm -###
if you run that command, you'll find "-pie-level" "2" in the output, which is how llc (well, it's internal equivalent) knows about it.
The key here is that you'll have to arrange for your backend to do something with this flag. Certain platforms (like Darwin) just ignore it. If you happen to be experiment on an OSX host, you won't see -pie-level in the bogbrush output.

autotools for pthreads not setting correct linker flags

I am adding some pthreads code into my Linux application that I'm building with autotools. I was getting an error about not linking in libpthreads. So I want to specify the pthreads dependency and compiler/linker flags in autotools.
I found some references that say use an ACX_PTHREAD macro. GNU provides an AX_PTHREAD macro. Both are very similar in concept. But I've tried both (on Ubuntu 13.04 64-bit), and found that they set -pthread in $PTHREAD_CFLAGS, but for some reason they don't set the -lpthread linker flag in $PTHREAD_LIBS.
The build fails. When I run make, I get:
...
/bin/sh ../libtool --tag=CXX --mode=link g++ -g -O2 -o myapp main.o ... -lconfuse -llog4cpp -lnsl -lpopt -lfuse -L/usr/local/lib -lrt
libtool: link: g++ -g -O2 -o .libs/myapp main.o ... -lconfuse -llog4cpp -lnsl /usr/lib/x86_64-linux-gnu/libpopt.so -lfuse -L/usr/local/lib -lrt
/usr/bin/ld: app-fuse.o: undefined reference to symbol 'pthread_kill##GLIBC_2.2.5'
/usr/bin/ld: note: 'pthread_kill##GLIBC_2.2.5' is defined in DSO /lib/x86_64-linux-gnu/libpthread.so.0 so try adding it to the linker command line
/lib/x86_64-linux-gnu/libpthread.so.0: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
...
In this case, the ./configure step shows:
...
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
...
I notice it checks for -lpthreads, but shouldn't it be checking for -lpthread?
I've found that I can use:
AC_CHECK_LIB(pthread, pthread_create, [PTHREAD_LIBS+=-lpthread])
and then the build succeeds. But I assume this isn't the best way to make it work on the widest variety of platforms.
I see Ubuntu also has a package libpthread-stubs0-dev. But I'm not sure what it's for.
What is the "right way" to use pthreads with autotools?
Thanks to Peter Simons who asked on the autoconf mailing list, we have a somewhat official answer:
Compiler flags and linker flags are not mutually-exclusive sets, not
least because linking is typically done via the compiler frontend (cc)
and not by invoking the linker (ld) directly. Any flags that you can
use in the compile step (e.g. -O2, -DFOO, -I/tmp/include) will
generally be accepted in the linking step, even if it's not applicable
then. (The reverse may not be true, e.g. -lfoo.)
Given that, it's a lot less error-prone to use PTHREAD_CFLAGS (and
other CFLAGS variables) when linking, rather than duplicating the
applicable flags into PTHREAD_LIBS/LDFLAGS/etc. variables and not
using any CFLAGS variables then.
So just use PTHREAD_CFLAGS for your linker, too.
I bumped into this same issue when I added a first C++ source to an otherwise working C project (a shared library). Adding this C++ file caused libtool to switch from linking with gcc to linking with g++. Seems that linking with gcc a '-pthread' is enough to add the dynamic dependency to libpthread, but when linking with g++, it is not.
I tried with the above patch to a local ax_pthread.m4, but this didn't help. Passing '-lpthread' to g++ would fix the issue.
Edit: for some reason, ax_pthread.m4 forces C as the test language even if the AC_LANG is set as C++. This patch makes things work for me:
--- m4/ax_pthread.m4_orig 2013-06-15 20:03:36.000000000 +0300
+++ m4/ax_pthread.m4 2013-06-15 20:03:51.000000000 +0300
## -87,7 +87,6 ##
AU_ALIAS([ACX_PTHREAD], [AX_PTHREAD])
AC_DEFUN([AX_PTHREAD], [
AC_REQUIRE([AC_CANONICAL_HOST])
-AC_LANG_PUSH([C])
ax_pthread_ok=no
# We used to check for pthread.h first, but this fails if pthread.h
## -313,5 +312,4 ##
ax_pthread_ok=no
$2
fi
-AC_LANG_POP
])dnl AX_PTHREAD
It seems that the AX_PTHREAD macro is finding the -pthread compiler flag, and using that. But it looks as though for that particular flag, it should be specified to the linker as well (it apparently does the equivalent of -lpthread in the linker). I modified the macro as follows, so that the -pthread flag is specified as a linker flag too:
diff --git a/m4/ax_pthread.m4 b/m4/ax_pthread.m4
index 6d400ed..f426654 100644
--- a/m4/ax_pthread.m4
+++ b/m4/ax_pthread.m4
## -172,6 +172,12 ## for flag in $ax_pthread_flags; do
AC_MSG_CHECKING([whether pthreads work without any flags])
;;
+ -pthread)
+ AC_MSG_CHECKING([whether pthreads work with $flag])
+ PTHREAD_CFLAGS="$flag"
+ PTHREAD_LIBS="$flag"
+ ;;
+
-*)
AC_MSG_CHECKING([whether pthreads work with $flag])
PTHREAD_CFLAGS="$flag"
I guess I should submit this to the macro authors.
Expanding on a suggestion above (https://stackoverflow.com/a/20253318/221802) with exact script, this error went away for me after updating my PKbuild.sh script with pthread args:
./bootstrap && \
CPPFLAGS=" -g3 -Wall -pthread "\
CFLAGS=" -pthread -g3 -Wall "\
LDFLAGS=" -lpthread "\
./configure --enable-maintainer-mode \
--enable-debug \
--prefix=/usr \
--mandir=/usr/share/man \
--enable-pie \
--prefix=/usr \
--enable-library \
--enable-test \
......... [and so on]
I used the advice from another post: autoconf with -pthread
Here they mentioned you could download this file:
http://svn.sleuthkit.org/repos/sleuthkit/trunk/configure.ac
Place it into your m4 directory.
Then include this in your configure.ac:
ACX_PTHREAD
Finally, add this to your Makefile.am:
bin_PROGRAMS = main
main_SOURCES = main.c
main_CFLAGS = $(PTHREAD_CFLAGS)
main_LDADD = $(PTHREAD_LIBS)

How to add directories to ld search path for a cross-compilation to ARM?

I am trying to configure util-linux to cross compile using arm-none-linux-gnueabi from CodeSourcery. My only problem so far is that it can't find my ncurses library which I compiled.
How can I add a directory to the ld search path? I've tried adding to my LIBRARY_PATH and LD_LIBRARY_PATH variables, but neither does anything. I know that I can add the -L flag to gcc and it will add to the linker path, but is there any way to do this globally, so that I can do it once, and not have to worry about it again?
Here is the output of arm-none-linux-gnueabi-gcc -print-search-dirs | grep libraries | sed 's/:/\n/g':
libraries
=/tools/bin/../lib/gcc/arm-none-linux-gnueabi/4.6.1/
/tools/bin/../lib/gcc/
/tools/bin/../lib/gcc/arm-none-linux-gnueabi/4.6.1/../../../../arm-none-linux-gnueabi/lib/arm-none-linux-gnueabi/4.6.1/
/tools/bin/../lib/gcc/arm-none-linux-gnueabi/4.6.1/../../../../arm-none-linux-gnueabi/lib/
/tools/bin/../arm-none-linux-gnueabi/libc/lib/arm-none-linux-gnueabi/4.6.1/
/tools/bin/../arm-none-linux-gnueabi/libc/lib/
/tools/bin/../arm-none-linux-gnueabi/libc/usr/lib/arm-none-linux-gnueabi/4.6.1/
/tools/bin/../arm-none-linux-gnueabi/libc/usr/lib/
I would like to add /arm/usr/lib and /arm/usr/local/lib to my ld search path.
If you need output from any other commands, just ask!
EDIT: I just found out about the CFLAGS environment variable--do all configure scripts/makefiles honor it?
Thank you!
If the ncurses library you compiled are going to be linked to the ARM binary you are cross-compiling you can not use LD_LIBRARY_PATH! LD_LIBRARY_PATH is only used by the current run-time and is in no way used by the compiler or linker when building your application.
The use of CFLAGS depends on creator of Makefile. CFLAGS are not automatically used even if they are defined as an environment variable. Only tools like the autoconf tools can pick them up from the environment and use them automagically. In the Makefiles find something like:
$(CC) $(CFLAGS) ....
if this fragment exists then the Makefile uses the CFLAGS variable. LDFLAGS is the more appropriate environment variable to use for link-time options.

Resources