How to get a smaller toolchain from scratch? - openwrt

Question

I have built a custom toolchain from scratch for a MIPS 24kc (dragino) target platform, using gcc-6.3.0, musl-1.1.16 and binutils-2.27. That toolchain is completely functional.
However, the size of my-custom toolchain is five times bigger than size of equivalent OpenWRT generated toolchain (557M vs 113M). Toolchain binaries generated by OpenWRT (mips-openwrt-linux-musl-*) are dynamically linked against libstdc++ and libgcc_s libraries. However, my toolchain binaries are linked with these libs statically:
$ ldd mips-openwrt-linux-musl-gcc-5.3.0
linux-vdso.so.1 => (0x00007ffc4d534000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f878936f000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8789159000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8788d8f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8788a86000)
/lib64/ld-linux-x86-64.so.2 (0x000055df3ef8b000)
$ ldd mips-linux-musl-gcc-6.3.0
linux-vdso.so.1 => (0x00007ffd40940000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2de4b8f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2de47c6000)
/lib64/ld-linux-x86-64.so.2 (0x0000560bba4b6000)
The same linking issue happens with cc1 and cc1plus. Sizes of these two binaries are huge (cc1 is 124MB, cc1plus is 134MB).
My question is: what is the proper way to setup the cross-toolchain generation to get a cross-platform toolchain as small as the OpenWRT generated toolchain?
Bellow you can find the steps I'm doing to build my-custom toolchain for reference:
Download sources:
wget http://ftpmirror.gnu.org/binutils/binutils-2.27.tar.bz2
wget http://ftpmirror.gnu.org/gcc/gcc-6.3.0/gcc-6.3.0.tar.bz2
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.12.6.tar.bz2
wget https://www.musl-libc.org/releases/musl-1.1.16.tar.gz
wget http://ftpmirror.gnu.org/mpfr/mpfr-3.1.5.tar.bz2
wget http://ftpmirror.gnu.org/gmp/gmp-6.1.2.tar.bz2
wget http://ftpmirror.gnu.org/mpc/mpc-1.0.2.tar.gz
Setup build environment:
export ROOTDIR="${HOME}/custom-toolchains/MIPS"
export NATIVE_PREFIX="${ROOTDIR}/opt/native"
export CROSS_PREFIX="${ROOTDIR}/opt/cross"
export TARGET_MACHINE=mips
export CPU=mips
export ARCH=24kc
export CLIB=musl
export TARGET_TRIPLET=${CPU}-linux-${CLIB}
Build native binutils:
cd ${ROOTDIR}/src
mkdir build-binutils
cd build-binutils
../binutils-2.27/configure --prefix="${NATIVE_PREFIX}" --disable-nls --disable-werror --disable-multilib
make
make install
Build native gcc:
cd ${ROOTDIR}/src/gcc-6.3.0
ln -s ../mpfr-3.1.5 mpfr
ln -s ../gmp-6.1.2 gmp
ln -s ../mpc-1.0.3 mpc
cd ..
mkdir build-gcc
cd build-gcc
../gcc-6.3.0/configure --prefix=${NATIVE_PREFIX} --disable-nls --enable-languages=c --disable-multilib
make
make install
Build cross-binutils:
cd ${ROOTDIR}/src
mkdir build-${CPU}-binutils
cd build-${CPU}-binutils
../binutils-2.27/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --with-sysroot --disable-nls --disable-werror --disable-multilib
make
make install
Install kernel headers:
cd ${ROOTDIR}/src
cd linux-3.12.6
make ARCH=${TARGET_MACHINE} INSTALL_HDR_PATH=${CROSS_PREFIX}/${TARGET_TRIPLET} headers_install
Build cross-gcc (stage 1)
cd ${ROOTDIR}/src
mkdir build-bootstrap-${CPU}-gcc
cd build-bootstrap-${CPU}-gcc
../gcc-6.3.0/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --disable-nls --enable-languages=c --disable-multilib --disable-threads --disable-shared --with-float=soft --with-arch=${ARCH}
make all-gcc install-gcc
make all-target-libgcc install-target-libgcc
Build cross-musl
cd ${ROOTDIR}/src
mkdir build-${CLIB}
cd build-${CLIB}
CC=${TARGET_TRIPLET}-gcc CFLAGS=-Wa,-msoft-float ../musl-1.1.16/configure --prefix=${CROSS_PREFIX}/${TARGET_TRIPLET}/ --enable-optimize CROSS_COMPILE=${TARGET_TRIPLET}-
make
make install
Build cross-gcc (stage 2)
cd ${ROOTDIR}/src
mkdir build-${CPU}-gcc
cd build-${CPU}-gcc
../gcc-6.3.0/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --disable-nls --enable-languages=c,c++ --disable-multilib --enable-threads --enable-shared --with-float=soft --with-arch=${ARCH} --enable-target-optspace --disable-libgomp --disable-libmudflap --without-isl --without-cloog --disable-decimal-float --disable-libssp --disable-libsanitizer --enable-lto --with-host-libstdcxx=-lstdc++
make
make install
Thank you for all your comments in advance!

Answer 1

Fixing this issue has been simpler than expected... All toolchain binaries are installed non-stripped. So this can be fixed calling
$ make install-strip
instead of
$ make install
when you install cross-gcc (stage 2).
Size of generated toolchain is 122MB, what is nice compared with 557M of the original toolchain. So this issue is fixed! Hope this information will be useful for somebody else in the future. Thank you!