Building a C library (GMP) for arm64 iOS - ios

I'm trying to build a C library (GMP 6.0.0) for arm64 for use on iOS. I'm running the configure script with the invocation below (compiler is as found using xcrun --find).
./configure \
CC="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" \
CPP="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -E" \
CPPFLAGS="-target arm64-apple-darwin -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.1.sdk/ -miphoneos-version-min=7.0" \
--host=aarch64-apple-darwin
However this fails at the following line ("long long reliability test 1"):
checking compiler /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -O2 -pedantic -target arm64-apple-darwin -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.1.sdk/ -miphoneos-version-min=7.0... no, long long reliability test 1
configure: error: could not find a working compiler, see config.log for details
Full config.log available here. It shows multiple warning and errors for the long long reliability test compile, including the following:
conftest.c:9:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
f(){static const struct{t1 n;t1 src[9];t1 want[9];}d[]={{1,{0},{1}},};t1 got[9];int i;
^
conftest.c:10:44: error: implicit declaration of function 'h' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
for(i=0;i<1;i++){if(e(got,got,9,d[i].n)==0)h();g(i,d[i].src,d[i].n,got,d[i].want,9);if(d[i].n)h();}}
^
conftest.c:10:48: error: implicit declaration of function 'g' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
for(i=0;i<1;i++){if(e(got,got,9,d[i].n)==0)h();g(i,d[i].src,d[i].n,got,d[i].want,9);if(d[i].n)h();}}
^
conftest.c:10:100: warning: control reaches end of non-void function [-Wreturn-type]
for(i=0;i<1;i++){if(e(got,got,9,d[i].n)==0)h();g(i,d[i].src,d[i].n,got,d[i].want,9);if(d[i].n)h();}}
^
Using --host=none it works fine, but I'd really like to figure out how to build it with assembly optimized for arm64.
My system is x86_64-apple-darwin13.1.0 (or coreisbr-apple-darwin13.1.0 according to config.guess), an early '11 Core i7 MBP running OS X v10.9.2. I'm using Xcode 5.1 (5B130a).
Any help appreciated.
EDIT 1
Compiling for ARMv7 passes configure, but fails on make (full configure/make output here), apparently while compiling some assembly:
tmp-dive_1.s:165:18: error: unexpected token in '.section' directive
.section .rodata
^
EDIT 2
#MarcGlisse: By forcing clang to ignore the errors as suggested (-Wno-...) arm64 passes configure, but then fails on make (full output here):
tmp-mul_1.s:59:2: error: unrecognized instruction mnemonic
bcc Lfi1
^
tmp-mul_1.s:60:2: error: unrecognized instruction mnemonic
beq Lfi2
^
As a side note: these commits, I assume intended to remove the need for suppressing the error, don't seem to work i.e. I get the same error when removing the supression.
For armv7, using these commits as suggested fixes the .section error, but make fails later on with the following (full output here):
tmp-mode1o.s:64:2: error: unknown directive
.protected ___gmp_binvert_limb_table
^
EDIT 3
Using the suggested edits, armv7, armv7s, i386 and x86_64 now all compile with assembly!
For arm64 the edits get it past the previous error, but now gives several errors about an invalid input constraint 'rZ', all in the same file (full output here):
divrem_1.c:237:5: error: invalid input constraint 'rZ' in asm
udiv_qrnnd_preinv (*qp, r, r, nshift, d, dinv);
^
../gmp-impl.h:3062:2: note: expanded from macro 'udiv_qrnnd_preinv'
add_ssaaaa (_qh, _ql, _qh, _ql, (nh) + 1, (nl)); \
^
../longlong.h:551:7: note: expanded from macro 'add_ssaaaa'
: "rZ" (ah), "rZ" (bh), "%r" (al), "rI" (bl) __CLOBBER_CC)
EDIT 4
After commenting out add_ssaaaa and sub_ddmmss in longlong.h and editing some more assembly instructions in gcd_1.asm (blo to b.lo etc.), it now fails with several of the following errors (full output here):
tmp-invert_limb.s:75:22: error: immediate value expected for shifter operand
add x1, x1, x2, lsr 1
^
tmp-invert_limb.s:75:22: error: invalid operand for instruction
add x1, x1, x2, lsr 1
^
I'll post a total diff later.
EDIT 5
Ok, that gets us another step further, but it now hits into (full output here):
tmp-invert_limb.s:52:2: error: ADR/ADRP relocations must be GOT relative
adrp x1, approx_tab
^
If this keeps going it might be better to continue this via email.

"Support for ARM64 alias Aarch64 alias ARMv8"
https://gmplib.org/gmp6.0

Copyright issues aside...
GMP may not be compatible with Apple's proprietary CPU, and may not be compatible with Clang/LLVM. Being a GNU project it is probably more thoroughly tested with GCC on non-proprietary chipsets.
A quick search shows historically at least there have been issues compiling GMP with clang.
You really should email the GMP community and ask for help on this one.

Related

qemu-irix fails to build under Alpine 3.10 (GCC 8.3)

Just for fun, I'm attempting to build the irixxxx's qemu-irix as a Docker image.
It builds successfuly under Debian Buster (using GCC 8 container). However, it fails to build under Alpine 3.10 (it fails only when I selected the Irix targets), with the following errors (at the make stage):
CC util/oslib-posix.o
In file included from util/oslib-posix.c:39:
/usr/include/sys/signal.h:1:2: warning: #warning redirecting incorrect #include <sys/signal.h> to <signal.h> [-Wcpp]
#warning redirecting incorrect #include <sys/signal.h> to <signal.h>
^~~~~~~
CC util/qemu-openpty.o
util/qemu-openpty.c: In function 'qemu_openpty_raw':
util/qemu-openpty.c:123:9: warning: implicit declaration of function 'openpty'; did you mean 'openat'? [-Wimplicit-function-declaration]
if (openpty(&amaster, aslave, pty_buf, NULL, NULL) < 0) {
^~~~~~~
openat
util/qemu-openpty.c:123:9: warning: nested extern declaration of 'openpty' [-Wnested-externs]
(...)
CC irix-linux-user/linux-user/syscall.o
/qemu-irix/linux-user/syscall.c:6784:22: error: 'F_EXLCK' undeclared here (not in a function); did you mean 'F_RDLCK'?
TRANSTBL_CONVERT(F_EXLCK),
^~~~~~~
/qemu-irix/linux-user/syscall.c:6779:51: note: in definition of macro 'TRANSTBL_CONVERT'
#define TRANSTBL_CONVERT(a) { -1, TARGET_##a, -1, a }
^
/qemu-irix/linux-user/syscall.c:6785:22: error: 'F_SHLCK' undeclared here (not in a function); did you mean 'F_RDLCK'?
TRANSTBL_CONVERT(F_SHLCK),
^~~~~~~
/qemu-irix/linux-user/syscall.c:6779:51: note: in definition of macro 'TRANSTBL_CONVERT'
#define TRANSTBL_CONVERT(a) { -1, TARGET_##a, -1, a }
^
In file included from /qemu-irix/linux-user/syscall.c:121:
/qemu-irix/linux-user/syscall.c: In function 'target_to_host_sigevent':
/qemu-irix/linux-user/syscall.c:7422:27: error: 'struct sigevent' has no member named '_sigev_un'; did you mean 'sigev_value'?
__get_user(host_sevp->_sigev_un._tid, &target_sevp->_sigev_un._tid);
^~~~~~~~~
/qemu-irix/linux-user/qemu.h:501:5: note: in definition of macro '__get_user_e'
((x) = (typeof(*hptr))( \
^
/qemu-irix/linux-user/syscall.c:7422:5: note: in expansion of macro '__get_user'
__get_user(host_sevp->_sigev_un._tid, &target_sevp->_sigev_un._tid);
^~~~~~~~~~
/qemu-irix/linux-user/syscall.c:7422:36: error: '(const bitmask_transtbl *)&<erroneous-expression>' is a pointer; did you mean to use '->'?
__get_user(host_sevp->_sigev_un._tid, &target_sevp->_sigev_un._tid);
^
/qemu-irix/linux-user/qemu.h:501:5: note: in definition of macro '__get_user_e'
((x) = (typeof(*hptr))( \
^
/qemu-irix/linux-user/syscall.c:7422:5: note: in expansion of macro '__get_user'
__get_user(host_sevp->_sigev_un._tid, &target_sevp->_sigev_un._tid);
^~~~~~~~~~
/qemu-irix/linux-user/qemu.h:506:13: warning: left-hand operand of comma expression has no effect [-Wunused-value]
(hptr)), (void)0)
^
/qemu-irix/linux-user/qemu.h:510:31: note: in expansion of macro '__get_user_e'
# define __get_user(x, hptr) __get_user_e(x, hptr, be)
^~~~~~~~~~~~
/qemu-irix/linux-user/syscall.c:7422:5: note: in expansion of macro '__get_user'
__get_user(host_sevp->_sigev_un._tid, &target_sevp->_sigev_un._tid);
^~~~~~~~~~
/qemu-irix/linux-user/syscall.c: In function 'do_syscall':
/qemu-irix/linux-user/syscall.c:13545:25: warning: implicit declaration of function 'sethostid'; did you mean 'gethostid'? [-Wimplicit-function-declaration]
ret = get_errno(sethostid(arg1));
^~~~~~~~~
gethostid
/qemu-irix/linux-user/syscall.c:13545:25: warning: nested extern declaration of 'sethostid' [-Wnested-externs]
make[1]: *** [/qemu-irix/rules.mak:66: linux-user/syscall.o] Error 1
make: *** [Makefile:472: subdir-irix-linux-user] Error 2
The command '/bin/sh -c make && DESTDIR=/tmp/qemu make install' returned a non-zero code: 2
I researched about the differences between Debian's and Alpine's GCC (in fact, libc6 and musl) about similar failures, but I haven't found any information about how to solve this without modifying the code.
So, where can I find more information about this issue and which packages may solve it? Thanks.
Dockerfiles and full log here. Docker container image (Debian Buster) here.
The difference seems to be caused by the underlying libc implementations: glibc of Debian, vs musl-libc of Alpine.
While GNU libc is the defacto standard libc implementation in Linux, musl libc is used by a handful of distributions, such as Alpine Linux and Void Linux. musl is a minimalistic strict-POSIX libc implementation, and is generally not compatible with glibc. Usually, software projects have to be ported to musl libc to be supported on Alpine, especially non trivial applications.
The compilation of syscall.c breaks on several places, the first being:
/qemu-irix/linux-user/syscall.c:6784:22: error: 'F_EXLCK' undeclared here (not in a function); did you mean 'F_RDLCK'?
TRANSTBL_CONVERT(F_EXLCK)
The F_EXLCK is macro is not defined in musl libc's fcntl.h. However, it could be easily patched by defining it manually, for example with make CFLAGS='"-DF_EXLCK=4"'. This is how qemu is patched for musl for non-irix targets (patch link).
However, there are more undefined macros down the road, such as __SIGRTMIN and __SIGRTMAX, the macro TRANSTBL_CONVERT, and probably others. Patching them ad-hoc may not be enough - so it appears to be that the qemu-irix project has to be properly ported for Alpine and musl libc.
If you're willing, you could try following existing qemu musl patches, and attempt to patch it yourself for qemu-irix:
- https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg04773.html
- https://github.com/NixOS/nixpkgs/pull/46449/files
Off topic comment, use make -j to build parallely, which will end much faster.

How to exclude old version library in /usr/local?

Disclamer: using -nostdcincl isn't possible because it excludes needed system libraries. Here instead the problem seems to be that tthe compiler ignores my -I directives
I have installed a library (OpenCV) in ~/local on a remote machine, since I don't have sudo access there. Notice that an older version of the same library is installed in /usr/local.
I'm trying to compile this code:
g++ -DCC_DISABLE_CUDA -I/home/spm1428/CloudCache -I/home/spm1428/local/include/opencv -I/home/spm1428/local/include/opencv2 -I/usr/include/boost -I/home/spm1428/vlfeat -O3 -g -Wall -c -fopenmp -std=c++11 -c -o Descriptor.o ../Descriptors/Descriptor.cpp
However, the returned error is:
In file included from /usr/local/include/opencv2/opencv.hpp:77:0,
from /home/spm1428/CloudCache/Utilities/Utility.hpp:11,
from ../Descriptors/Descriptor.cpp:17:
/usr/local/include/opencv2/highgui/highgui.hpp:165:25: error: redeclaration of ‘IMREAD_UNCHANGED’
IMREAD_UNCHANGED =-1,
^
In file included from ../Descriptors/Descriptor.cpp:13:0:
/home/spm1428/local/include/opencv2/imgcodecs.hpp:65:8: note: previous declaration ‘cv::ImreadModes IMREAD_UNCHANGED’
IMREAD_UNCHANGED = -1, //!< If set, return the loaded image as is (with alpha channel,
^
In file included from /usr/local/include/opencv2/opencv.hpp:77:0,
from /home/spm1428/CloudCache/Utilities/Utility.hpp:11,
from ../Descriptors/Descriptor.cpp:17:
/usr/local/include/opencv2/highgui/highgui.hpp:167:24: error: redeclaration of ‘IMREAD_GRAYSCALE’
IMREAD_GRAYSCALE =0,
I think that this happens because there is another version installed. How can I solve this?
I think this error happens for the same reason (the old version doesn't have cv::xfeatures2d::SURF).

clang-3.8 and compiler-rt vs libgcc

I have been using clang-3.5 to happily build bitcode versions of musl libc and
use the result to produce nice stand alone executables.
Recent attempts with clang-3.8 have not been so happy. It seems that
the bitcode clang-3.8 generates uses functions defined in
compiler-rt/lib/builtins
Typical examples of functions I find polluting the bitcode are mulxc3, mulsc3, and muldc3. I can solve this by linking against libgcc, or even the llvm alternative if I had any clear idea of what that was. Though I would rather prevent the problem from happening in the first place.
I have seen mention of flags like rtlib=compiler-rt etc, but have found precious little documentation on the subject.
So here are some simple questions.
Is it possible to prevent clang from using the compiler-rt/lib/builtins
in the emitted bitcode? Or if not
Does llvm produce a version of libgcc that I could use. Actually I would
probably build a bitcode version of it, but that is besides the point.
Love to hear some guidance on this.
Added 12/8/2016: So I will illustrate my issues with a particular workflow that
people can reproduce if they wish, or, more likely, just point out where I am being stupid.
So start by checking out:
musllv
and follow the instructions in the README.to compile (here I am using clang-3.8 on ubuntu 14.04)
WLLVM_CONFIGURE_ONLY=1 CC=wllvm ./configure --target=LLVM --build=LLVM
make
cd lib
extract-bc -b libc.a
you will also need the bitcode of a simple executable. I will use nweb.c here.
wllvm nweb.c -o nweb
extract-bc nweb
Now we can do things like:
clang -static -nostdlib nweb.bc libc.a.bc crt1.o libc.a -o nweb
This workflow goes smoothly for clang-3.5 but for clang-3.8 we get:
clang -static -nostdlib nweb.bc libc.a.bc crt1.o libc.a -o nweb
/tmp/libc-f734a3.o: In function `cpowl':
libc.a.bc:(.text+0xbb9a): undefined reference to `__mulxc3'
/tmp/libc-f734a3.o: In function `cpowf':
libc.a.bc:(.text+0x38f7d): undefined reference to `__mulsc3'
/tmp/libc-f734a3.o: In function `csqrt':
libc.a.bc:(.text+0x78fc3): undefined reference to `__muldc3'
/tmp/libc-f734a3.o: In function `cpow':
libc.a.bc:(.text+0xafafc): undefined reference to `__muldc3'
clang-3.8: error: linker command failed with exit code 1 (use -v to seeinvocation)
So as #paul-brannan points out we could try
clang -static -nostdlib --rtlib=compiler-rt nweb.bc libc.a.bc crt1.o libc.a -o nweb
But this is where I am probably being stupid, because I get:
clang-3.8: warning: argument unused during compilation: '--rtlib=compiler-rt'
irregardless of whether I use it as a linking or compiling flag.
OK so I finally managed to make headway on this. I built llvm-3.8.1 together with the compiler-rt project using wllvm and wllvm++.
One of the build products was libclang_rt.builtins-x86_64.a,
and from this archive I was able to extract the bitcode module
libclang_rt.builtins-x86_64.bc
using the command:
extract-bc -b libclang_rt.builtins-x86_64.a
This bitcode module has definitions for those pesky instrinsics like
__mulxc3, __mulsc3, and __muldc3.
Hallelujah!

Rebar: Cross Compilation Options

I'm trying use rebar to generate a 32-bit arch release on 64-bit arch system. It's unclear to me how I need to modify an application's rebar.config to be able to do this.
I have taken a look at the rebar source to see how they are detecting my environment: rebar_utils.erl gets the architecture as "x86_64-unknown-linux-gnu" and this is set in rebar_port_compiler as the "default_env". I'm wondering how I can get rebar to compile for another target architecture.
I have tried the following port_env options
{port_env, [{"CFLAGS", "$CFLAGS -fPIC -m32"},{"LDFLAGS", "-arch i386"}]}.
With those options (and a 32-bit Erlang installation in my path), when I run rebar get-deps compile my dependencies are still being built as 64-bit and thus ld skips over my 32-bit libraries (and ultimately fails because it cant find a 64-bit implementation)
/usr/bin/ld: skipping incompatible <PATH TO 32-bit erlang install>/lib/erlang/lib/erl_interface-3.7.14/lib/liberl_interface.a when searching for -lerl_interface
/usr/bin/ld: cannot find -lerl_interface
collect2: ld returned 1 exit status
ERROR: sh(cc c_src/epam.o $LDFLAGS -shared -L"<PATH TO 32-bit erlang install>/lib/erlang/lib/erl_interface-3.7.14/lib" -lerl_interface -lei -o priv/lib/epam.so)
What do I need to do to force my dependencies to compile as 32 bit? My attempt here isn't working.
The port_env settings in rebar.config can make use of the ERLANG_ARCH environment variable to determine whether the Erlang runtime was built for a 32- or 64-bit system. For example, the following port_env definition sets either -m32 or -m64 as appropriate for the C compiler for the x86_64, i686, and i386 chip architectures:
{port_env, [{"x86_64", "CFLAGS", "$CFLAGS -m$ERLANG_ARCH"},
{"i[36]86", "CFLAGS", "$CFLAGS -m$ERLANG_ARCH"}]}.
The first string in each tuple is a regular expression matched against the system architecture string of the Erlang runtime as returned by the erlang:system_info(system_architecture) function. In this example, the additional -m$ERLANG_ARCH option is added only when the regular expression matches, and all other architectures get the default CFLAGS setting.

Compiling x264 for iOS 7

I'm getting error on compiling x264 for iOS.
I have Xcode Version 5.0 (5A1413) with Apple LLVM version 5.0 (clang-500.2.75) (based on LLVM 3.3svn). I'm compiling x264-snapshot-20130925-2245.
Config:
CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang ./configure \
--host=arm-apple-darwin \
--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.0.sdk \
--prefix=armv7 \
--extra-cflags='-arch armv7' \
--extra-ldflags="-L/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.0.sdk/usr/lib/system -arch armv7" \
--enable-pic \
--enable-static
Getting error:
common/arm/cpu-a.S:29:7: error: unknown token in expression
.align
^
common/arm/cpu-a.S:139:5: error: instruction 'suble' can not set flags, but 's' suffix specified
subles ip, ip, #1
^
The relevant change in the Xcode 5 toolchain is that the LLVM compiler now defaults to using the built-in assembler, and the built-in assembler requires more strict adherence to the ARM Unified Assembly Language.
There are two ways to get it to compile with the Xcode 5 toolchain:
Give clang the flag -no-integrated-as. Adding it to --extra-cflags ought to work. (The flag worked for me compiling individual files but I never worked it into configure.) Consider this a workaround.
Fix the assembly source code in x264's common/arm subdirectory. This is pretty easy actually, and it's what I did. This is the right fix. BTW, I'm about to submit a patch to x264 with these changes.
The assembler emits many errors and they fall into four categories:
In cpu-a.S, the ".align" directive should be ".align 2". (Apparently it used to default to 2, now the 2 must be explicit.)
Several subles and sublts instructions in multiple files. These are variants of "sub" (subtract), followed by a condition (2 characters) and the "s" suffix. Now the "s" has to precede the condition. Thus "subles" => "subsle" and "sublts" => "subslt".
A fair number of ldrd instructions in various files. This instruction means "load register, double (from memory)". It loads 2 32-bit words from memory into registers. It used to be OK to name only the first register; now both need to be named. They're always adjacent. So "ldrd r2, whatever" needs to become "ldrd r2, r3, whatever". "ldrd r6, something" becomes "ldrd r6, r7, something". Etc.
In pixel-a.S, there's an instruction "vmov.32 r0, r1, d0". This is incorrect. vmov.32 means move a 32-bit quantity, yet the arguments say to move d0 (64 bits) into r0 and r1. Apparently the old compiler took the ".32" part as a hint. I believe it should be "vmov r0, r1, d0" and that change works for me - but I don't have absolute proof that is the correct instruction.
Many thanks to gparker on the Apple Developer Forum! I could not have figured this out without his/her help. Link to forum discussion, Apple ID required.
So far the only solution seems to be --disable-asm.
I think disabling assembler optimizations is a bad solution.
After a long research I've found the root of the problem: clang during assembler compilation uses ASFLAGS not CFLAGS, so adding --extra-asflags="-arch armv7" solves the problem
./configure \
--host=arm-apple-darwin \
--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk \
--prefix=armv7 \
--extra-cflags="-arch armv7" \
--extra-asflags="-arch armv7" \
--extra-ldflags="-arch armv7" \
--enable-pic \
--enable-static
NOTE: For bitcode support just add -fembed-bitcode to all extra flags parameters

Resources