Index: docs/HowToCrossCompileBuiltinsOnArm.rst =================================================================== --- docs/HowToCrossCompileBuiltinsOnArm.rst +++ docs/HowToCrossCompileBuiltinsOnArm.rst @@ -19,7 +19,7 @@ Prerequisites ============= -In this use case we'll be using CMake on a Debian-based Linux system, +In this use case we'll be using cmake on a Debian-based Linux system, cross-compiling from an x86_64 host to a hard-float Armv7-A target. We'll be using as many of the LLVM tools as we can, but it is possible to use GNU equivalents. @@ -28,17 +28,21 @@ * ``The qemu-arm user mode emulator`` * ``An arm-linux-gnueabihf sysroot`` +In this example we will be using ninja. + See https://compiler-rt.llvm.org/ for more information about the dependencies on clang and LLVM. ``qemu-arm`` should be available as a package for your Linux distribution. The most complicated of the prequisites to satisfy is the arm-linux-gnueabihf -sysroot. The :doc:`HowToCrossCompileLLVM` has information about how to use the -Linux distributions multiarch support to fulfill the dependencies for building -LLVM. Alternatively, as building and testing just the compiler-rt builtins -requires fewer dependencies than LLVM, it is possible to use the Linaro -arm-linux-gnueabihf gcc installation as our sysroot. +sysroot. In theory it is possible to use the Linux distributions multiarch +support to fulfill the dependencies for building but unfortunately due to +/usr/local/include being added some host includes are selected. The easiest way +to supply a sysroot is to download the arm-linux-gnueabihf toolchain. This can +be found at: +* https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads for gcc 8 and above +* https://releases.linaro.org/components/toolchain/binaries/ for gcc 4.9 to 7.3 Building compiler-rt builtins for Arm ===================================== @@ -46,6 +50,7 @@ options. * ``path/to/llvm/projects/compiler-rt`` +* ``-G Ninja`` * ``-DCOMPILER_RT_BUILD_BUILTINS=ON`` * ``-DCOMPILER_RT_BUILD_SANITIZERS=OFF`` * ``-DCOMPILER_RT_BUILD_XRAY=OFF`` @@ -57,22 +62,38 @@ * ``-DCMAKE_RANLIB=/path/to/llvm-ranlib`` * ``-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld"`` * ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"`` +* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"`` * ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON`` * ``-DLLVM_CONFIG_PATH=/path/to/llvm-config`` * ``-DCMAKE_C_FLAGS="build-c-flags"`` +* ``-DCMAKE_ASM_FLAGS="build-c-flags"`` -The build-c-flags need to be sufficient to pass the C-make compiler check and -to compile compiler-rt. When using a GCC 7 Linaro arm-linux-gnueabihf -installation the following flags are needed: +The ``build-c-flags`` need to be sufficient to pass the C-make compiler check, +compile compiler-rt, and if you are running the tests, compile and link the +tests. When cross-compiling with clang we will need to pass sufficient +information to generate code for the Arm architecture we are targeting. We will +need to select the Arm target, select the Armv7-A architecture and choose +between using Arm or Thumb. +instructions. For example: * ``--target=arm-linux-gnueabihf`` -* ``--march=armv7a`` +* ``-march=armv7a`` +* ``-mthumb`` + +When using a GCC arm-linux-gnueabihf toolchain the following flags are +needed to pick up the includes and libraries: + * ``--gcc-toolchain=/path/to/dir/toolchain`` * ``--sysroot=/path/to/toolchain/arm-linux-gnueabihf/libc`` -Depending on how your sysroot is laid out, you may not need ``--gcc-toolchain``. -For example if you have added armhf as an architecture using your Linux -distributions multiarch support then you should be able to use ``--sysroot=/``. +In this example we will be adding all of the command line options to both +``CMAKE_C_FLAGS`` and ``CMAKE_ASM_FLAGS``. There are cmake flags to pass some of +these options individually which can be used to simplify the ``build-c-flags``: + +* ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"`` +* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"`` +* ``-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=/path/to/dir/toolchain`` +* ``-DCMAKE_SYSROOT=/path/to/dir/toolchain/arm-linux-gnueabihf/libc`` Once cmake has completed the builtins can be built with ``ninja builtins`` @@ -90,12 +111,72 @@ The ``/path/to/armhf/sysroot`` should be the same as the one passed to ``--sysroot`` in the "build-c-flags". -The "test-c-flags" can be the same as the "build-c-flags", with the addition -of ``"-fuse-ld=lld`` if you wish to use lld to link the tests. +The "test-c-flags" need to include the target, architecture, gcc-toolchain, +sysroot and arm/thumb state. The additional cmake defines such as +``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` do not apply when building the tests. If +you have put all of these in "build-c-flags" then these can be repeated. If you +wish to use lld to link the tests then add ``"-fuse-ld=lld``. Once cmake has completed the tests can be built and run using ``ninja check-builtins`` +Troubleshooting +=============== + +The cmake try compile stage fails +--------------------------------- +At an early stage cmake will attempt to compile and link a simple C program to +test if the toolchain is working. + +This stage can often fail at link time if the ``--sysroot`` and +``--gcc-toolchain`` options are not passed to the compiler. Check the +``CMAKE_C_FLAGS`` and ``CMAKE_C_COMPILER_TARGET`` flags. + +It can be useful to build a simple example outside of cmake with your toolchain +to make sure it is working. For example: ``clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c`` + +Clang uses the host header files +-------------------------------- +On debian based systems it is possible to install multiarch support for +arm-linux-gnueabi and arm-linux-gnueabihf. In many cases clang can successfully +use this multiarch support when -gcc-toolchain and --sysroot are not supplied. +Unfortunately clang adds ``/usr/local/include`` before +``/usr/include/arm-linux-gnueabihf`` leading to errors when compiling the hosts +header files. + +The multiarch support is not sufficient to build the builtins you will need to +use a separate arm-linux-gnueabihf toolchain. + +No target passed to clang +------------------------- +If clang is not given a target it will typically use the host target, this will +not understand the Arm assembly language files resulting in error messages such +as ``error: unknown directive .syntax unified``. + +You can check the clang invocation in the error message to see if there is no +``--target`` or if it is set incorrectly. The cause is usually +``CMAKE_ASM_FLAGS`` not containing ``--target`` or ``CMAKE_ASM_COMPILER_TARGET`` not being present. + +Arm architecture not given +-------------------------- +The ``--target=arm-linux-gnueabihf`` will default to arm architecture v4t which +cannot assemble the barrier instructions used in the synch_and_fetch source +files. + +The cause is usually a missing ``-march=armv7a`` from the ``CMAKE_ASM_FLAGS``. + +Compiler-rt builds but the tests fail to build +---------------------------------------------- +The flags used to build the tests are not the same as those used to build the +builtins. The c flags are provided by ``COMPILER_RT_TEST_COMPILE_CFLAGS`` and +the ``CMAKE_C_COMPILER_TARGET``, ``CMAKE_ASM_COMPILER_TARGET``, +``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` and ``CMAKE_SYSROOT`` flags are not +applied. + +Make sure that ``COMPILER_RT_TEST_COMPILE_CFLAGS`` contains all the necessary +information. + + Modifications for other Targets =============================== @@ -112,6 +193,8 @@ instructions, and ``-mfloat-abi=soft -mfpu=none`` for software floating-point emulation. +You will need to use an arm-linux-gnueabi GNU toolchain for soft-float. + AArch64 Target -------------- The instructions for Arm can be used for AArch64 by substituting AArch64 @@ -134,30 +217,34 @@ The libraries can be built with the cmake options: -* ``-DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/sysroot`` -* ``-DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/sysroot`` -* ``-DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/sysroot`` +* ``-DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` +* ``-DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` +* ``-DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` * ``-C /path/to/llvm/source/tools/clang/cmake/caches/BaremetalARM.cmake`` +* ``/path/to/llvm`` **Note** that for the recipe to work the compiler-rt source must be checked out -into the directory llvm/runtimes and not llvm/projects. +into the directory llvm/runtimes and not llvm/projects. You will also need +clang and lld checked out. To build and test the libraries using a similar method to Armv7-A is possible but more difficult. The main problems are: * There isn't a ``qemu-arm`` user-mode emulator for bare-metal systems. The ``qemu-system-arm`` can be used but this is significantly more difficult to setup. -* The target to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won't find the libraries needed to pass the cmake compiler check. +* The targets to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won't find the libraries needed to pass the cmake compiler check. As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions that are supported on Armv7-A we can still get most of the value of running the tests using the same ``qemu-arm`` that we used for Armv7-A by building and running the test cases for Armv7-A but using the builtins compiled for -Armv6-M, Armv7-M or Armv7E-M. This will not catch instructions that are -supported on Armv7-A but not Armv6-M, Armv7-M and Armv7E-M. - -To get the cmake compile test to pass the libraries needed to successfully link -the test application will need to be manually added to ``CMAKE_CFLAGS``. -Alternatively if you are using version 3.6 or above of cmake you can use +Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked +into a binary and execute the tests correctly but it will not catch if the +builtins use instructions that are supported on Armv7-A but not Armv6-M, +Armv7-M and Armv7E-M. + +To get the cmake compile test to pass you will need to pass the libraries +needed to successfully link the cmake test via ``CMAKE_CFLAGS``. It is +strongly recommended that you use version 3.6 or above of cmake so you can use ``CMAKE_TRY_COMPILE_TARGET=STATIC_LIBRARY`` to skip the link step. * ``-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY`` @@ -169,6 +256,7 @@ * ``-DCOMPILER_RT_BUILD_PROFILE=OFF`` * ``-DCMAKE_C_COMPILER=${host_install_dir}/bin/clang`` * ``-DCMAKE_C_COMPILER_TARGET="your *-none-eabi target"`` +* ``-DCMAKE_ASM_COMPILER_TARGET="your *-none-eabi target"`` * ``-DCMAKE_AR=/path/to/llvm-ar`` * ``-DCMAKE_NM=/path/to/llvm-nm`` * ``-DCMAKE_RANLIB=/path/to/llvm-ranlib`` @@ -176,7 +264,7 @@ * ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON`` * ``-DLLVM_CONFIG_PATH=/path/to/llvm-config`` * ``-DCMAKE_C_FLAGS="build-c-flags"`` -* ``-DCMAKE_ASM_FLAGS="${arm_cflags}"`` +* ``-DCMAKE_ASM_FLAGS="build-c-flags"`` * ``-DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armv7-A/sysroot"`` * ``-DCOMPILER_RT_INCLUDE_TESTS=ON`` * ``-DCOMPILER_RT_TEST_COMPILER="/path/to/clang"`` @@ -186,16 +274,8 @@ Armv7-A we must include ``"-mthumb -mfloat-abi=soft -mfpu=none"`` in the test-c-flags. We must use an Armv7-A soft-float abi sysroot for ``qemu-arm``. -Unfortunately at time of writing the Armv7-M and Armv7E-M builds of -compiler-rt will always include assembler files including floating point -instructions. This means that building for a cpu without a floating point unit -requires something like removing the arm_Thumb1_VFPv2_SOURCES from the -arm_Thumb1_SOURCES in builtins/CMakeLists.txt. The float-abi of the compiler-rt -library must be matched by the float abi of the Armv7-A sysroot used by -qemu-arm. - Depending on the linker used for the test cases you may encounter BuildAttribute mismatches between the M-profile objects from compiler-rt and the A-profile -objects from the test. The lld linker does not check the BuildAttributes so it -can be used to link the tests by adding -fuse-ld=lld to the +objects from the test. The lld linker does not check the profile +BuildAttribute so it can be used to link the tests by adding -fuse-ld=lld to the ``COMPILER_RT_TEST_COMPILER_CFLAGS``.