[docs][ARM] Improve How to Cross Compile Compiler-rt Builtins For Arm

[peter.smith]

Summary

Some recent experience on llvm-dev pointed out some errors in the document such as:

• Assumption of ninja without -GNinja
• Use of --march rather than -march
• Problems with host include files when a multiarch setup was used
• Insufficient flags given to clang when acting as the assembler
• Instructions on using the cmake cache file BaremetalARM.cmake were incomplete

There was also insufficient troubleshooting information for when a step went wrong. I've made an effort to improve the document a bit based on this experience. Main changes:

• Fixed the problems described above
• Added a troubleshooting section
• Cleared up one "at time of writing" that is no longer a problem.

There are quite a few changes in the doc. I'm happy to break this up into smaller changes.

[efriedma]

docs/HowToCrossCompileBuiltinsOnArm.rst

227 ↗

(On Diff #178240)

For "A" targets, we recommend putting compiler-rt into llvm/projects, but for "M" targets we recommend putting it into llvm/runtimes? That seems kind of confusing... if it's intentional, please call it out explicitly.

[peter.smith]

Thanks for the comments. I used projects as that was what was in the LLVM Getting Started guide. I do agree that it is confusing though. I've put a bit more text at the start about where to get compiler-rt source code and where to place it. I've also moved the Baremetal CMake cache to the end as it is different to the other two sections.

[efriedma]

LGTM

(It should be possible to build baremetal compiler-rt without an existing sysroot in theory, since there is no libc to link against, but I wouldn't be surprised if it doesn't work in practice; we should look into simplifying it at some point.)

llvm/trunk/docs/HowToCrossCompileBuiltinsOnArm.rst

 
 The instructions in this document depend on libraries and programs external to
 LLVM, there are many ways to install and configure these dependencies so you
 may need to adapt the instructions here to fit your own local situation.
 
 Prerequisites
 =============
 
-In this use case we'll be using CMake on a Debian-based Linux system,
+In this use case we'll be using cmake on a Debian-based Linux system,
 cross-compiling from an x86_64 host to a hard-float Armv7-A target. We'll be
 using as many of the LLVM tools as we can, but it is possible to use GNU
 equivalents.
 
  * ``A build of LLVM/clang for the llvm-tools and llvm-config``
+ * ``A clang executable with support for the ARM target``
+ * ``compiler-rt sources``
  * ``The qemu-arm user mode emulator``
  * ``An arm-linux-gnueabihf sysroot``
 
+In this example we will be using ninja.
+
 See https://compiler-rt.llvm.org/ for more information about the dependencies
 on clang and LLVM.
 
+See https://llvm.org/docs/GettingStarted.html for information about obtaining
+the source for LLVM and compiler-rt. Note that the getting started guide
+places compiler-rt in the projects subdirectory, but this is not essential and
+if you are using the BaremetalARM.cmake cache for v6-M, v7-M and v7-EM then
+compiler-rt must be placed in the runtimes directory.
+
 ``qemu-arm`` should be available as a package for your Linux distribution.
 
 The most complicated of the prequisites to satisfy is the arm-linux-gnueabihf
-sysroot. The :doc:`HowToCrossCompileLLVM` has information about how to use the
-Linux distributions multiarch support to fulfill the dependencies for building
-LLVM. Alternatively, as building and testing just the compiler-rt builtins
-requires fewer dependencies than LLVM, it is possible to use the Linaro
-arm-linux-gnueabihf gcc installation as our sysroot.
+sysroot. In theory it is possible to use the Linux distributions multiarch
+support to fulfill the dependencies for building but unfortunately due to
+/usr/local/include being added some host includes are selected. The easiest way
+to supply a sysroot is to download the arm-linux-gnueabihf toolchain. This can
+be found at:
+* https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads for gcc 8 and above
+* https://releases.linaro.org/components/toolchain/binaries/ for gcc 4.9 to 7.3
 
 Building compiler-rt builtins for Arm
 =====================================
 We will be doing a standalone build of compiler-rt using the following cmake
 options.
 
-* ``path/to/llvm/projects/compiler-rt``
+* ``path/to/compiler-rt``
+* ``-G Ninja``
 * ``-DCOMPILER_RT_BUILD_BUILTINS=ON``
 * ``-DCOMPILER_RT_BUILD_SANITIZERS=OFF``
 * ``-DCOMPILER_RT_BUILD_XRAY=OFF``
 * ``-DCOMPILER_RT_BUILD_LIBFUZZER=OFF``
 * ``-DCOMPILER_RT_BUILD_PROFILE=OFF``
 * ``-DCMAKE_C_COMPILER=/path/to/clang``
 * ``-DCMAKE_AR=/path/to/llvm-ar``
 * ``-DCMAKE_NM=/path/to/llvm-nm``
 * ``-DCMAKE_RANLIB=/path/to/llvm-ranlib``
 * ``-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld"``
 * ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"``
+* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"``
 * ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON``
 * ``-DLLVM_CONFIG_PATH=/path/to/llvm-config``
 * ``-DCMAKE_C_FLAGS="build-c-flags"``
+* ``-DCMAKE_ASM_FLAGS="build-c-flags"``
 
-The build-c-flags need to be sufficient to pass the C-make compiler check and
-to compile compiler-rt. When using a GCC 7 Linaro arm-linux-gnueabihf
-installation the following flags are needed:
+The ``build-c-flags`` need to be sufficient to pass the C-make compiler check,
+compile compiler-rt, and if you are running the tests, compile and link the
+tests. When cross-compiling with clang we will need to pass sufficient
+information to generate code for the Arm architecture we are targeting. We will
+need to select the Arm target, select the Armv7-A architecture and choose
+between using Arm or Thumb.
+instructions. For example:
 
 * ``--target=arm-linux-gnueabihf``
-* ``--march=armv7a``
+* ``-march=armv7a``
+* ``-mthumb``
+
+When using a GCC arm-linux-gnueabihf toolchain the following flags are
+needed to pick up the includes and libraries:
+
 * ``--gcc-toolchain=/path/to/dir/toolchain``
 * ``--sysroot=/path/to/toolchain/arm-linux-gnueabihf/libc``
 
-Depending on how your sysroot is laid out, you may not need ``--gcc-toolchain``.
-For example if you have added armhf as an architecture using your Linux
-distributions multiarch support then you should be able to use ``--sysroot=/``.
+In this example we will be adding all of the command line options to both
+``CMAKE_C_FLAGS`` and ``CMAKE_ASM_FLAGS``. There are cmake flags to pass some of
+these options individually which can be used to simplify the ``build-c-flags``:
+
+* ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"``
+* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"``
+* ``-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=/path/to/dir/toolchain``
+* ``-DCMAKE_SYSROOT=/path/to/dir/toolchain/arm-linux-gnueabihf/libc``
 
 Once cmake has completed the builtins can be built with ``ninja builtins``
 
 Testing compiler-rt builtins using qemu-arm
 ===========================================
 To test the builtins library we need to add a few more cmake flags to enable
 testing and set up the compiler and flags for test case. We must also tell
 cmake that we wish to run the tests on ``qemu-arm``.
 
 * ``-DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armhf/sysroot``
 * ``-DCOMPILER_RT_INCLUDE_TESTS=ON``
 * ``-DCOMPILER_RT_TEST_COMPILER="/path/to/clang"``
 * ``-DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"``
 
 The ``/path/to/armhf/sysroot`` should be the same as the one passed to
 ``--sysroot`` in the "build-c-flags".
 
-The "test-c-flags" can be the same as the "build-c-flags", with the addition
-of ``"-fuse-ld=lld`` if you wish to use lld to link the tests.
+The "test-c-flags" need to include the target, architecture, gcc-toolchain,
+sysroot and arm/thumb state. The additional cmake defines such as
+``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` do not apply when building the tests. If
+you have put all of these in "build-c-flags" then these can be repeated. If you
+wish to use lld to link the tests then add ``"-fuse-ld=lld``.
 
 Once cmake has completed the tests can be built and run using
 ``ninja check-builtins``
 
+Troubleshooting
+===============
+
+The cmake try compile stage fails
+---------------------------------
+At an early stage cmake will attempt to compile and link a simple C program to
+test if the toolchain is working.
+
+This stage can often fail at link time if the ``--sysroot`` and
+``--gcc-toolchain`` options are not passed to the compiler. Check the
+``CMAKE_C_FLAGS`` and ``CMAKE_C_COMPILER_TARGET`` flags.
+
+It can be useful to build a simple example outside of cmake with your toolchain
+to make sure it is working. For example: ``clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c``
+
+Clang uses the host header files
+--------------------------------
+On debian based systems it is possible to install multiarch support for
+arm-linux-gnueabi and arm-linux-gnueabihf. In many cases clang can successfully
+use this multiarch support when -gcc-toolchain and --sysroot are not supplied.
+Unfortunately clang adds ``/usr/local/include`` before
+``/usr/include/arm-linux-gnueabihf`` leading to errors when compiling the hosts
+header files.
+
+The multiarch support is not sufficient to build the builtins you will need to
+use a separate arm-linux-gnueabihf toolchain.
+
+No target passed to clang
+-------------------------
+If clang is not given a target it will typically use the host target, this will
+not understand the Arm assembly language files resulting in error messages such
+as ``error: unknown directive .syntax unified``.
+
+You can check the clang invocation in the error message to see if there is no
+``--target`` or if it is set incorrectly. The cause is usually
+``CMAKE_ASM_FLAGS`` not containing ``--target`` or ``CMAKE_ASM_COMPILER_TARGET`` not being present.
+
+Arm architecture not given
+--------------------------
+The ``--target=arm-linux-gnueabihf`` will default to arm architecture v4t which
+cannot assemble the barrier instructions used in the synch_and_fetch source
+files.
+
+The cause is usually a missing ``-march=armv7a`` from the ``CMAKE_ASM_FLAGS``.
+
+Compiler-rt builds but the tests fail to build
+----------------------------------------------
+The flags used to build the tests are not the same as those used to build the
+builtins. The c flags are provided by ``COMPILER_RT_TEST_COMPILE_CFLAGS`` and
+the ``CMAKE_C_COMPILER_TARGET``, ``CMAKE_ASM_COMPILER_TARGET``,
+``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` and ``CMAKE_SYSROOT`` flags are not
+applied.
+
+Make sure that ``COMPILER_RT_TEST_COMPILE_CFLAGS`` contains all the necessary
+information.
+
+
 Modifications for other Targets
 ===============================
 
 Arm Soft-Float Target
 ---------------------
 The instructions for the Arm hard-float target can be used for the soft-float
 target by substituting soft-float equivalents for the sysroot and target. The
 target to use is:
 
 * ``-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi``
 
 Depending on whether you want to use floating point instructions or not you
 may need extra c-flags such as ``-mfloat-abi=softfp`` for use of floating-point
 instructions, and ``-mfloat-abi=soft -mfpu=none`` for software floating-point
 emulation.
 
+You will need to use an arm-linux-gnueabi GNU toolchain for soft-float.
+
 AArch64 Target
 --------------
 The instructions for Arm can be used for AArch64 by substituting AArch64
 equivalents for the sysroot, emulator and target.
 
 * ``-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu``
 * ``-DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot``
 
 The CMAKE_C_FLAGS and COMPILER_RT_TEST_COMPILER_CFLAGS may also need:
 ``"--sysroot=/path/to/aarch64/sysroot --gcc-toolchain=/path/to/gcc-toolchain"``
 
 Armv6-m, Armv7-m and Armv7E-M targets
 -------------------------------------
-If you wish to build, but not test compiler-rt for Armv6-M, Armv7-M or Armv7E-M
-then the easiest way is to use the BaremetalARM.cmake recipe in
-clang/cmake/caches.
-
-You will need a bare metal sysroot such as that provided by the GNU ARM
-Embedded toolchain.
-
-The libraries can be built with the cmake options:
-
-* ``-DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/sysroot``
-* ``-DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/sysroot``
-* ``-DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/sysroot``
-* ``-C /path/to/llvm/source/tools/clang/cmake/caches/BaremetalARM.cmake``
-
-**Note** that for the recipe to work the compiler-rt source must be checked out
-into the directory llvm/runtimes and not llvm/projects.
-
 To build and test the libraries using a similar method to Armv7-A is possible
 but more difficult. The main problems are:
 
 * There isn't a ``qemu-arm`` user-mode emulator for bare-metal systems. The ``qemu-system-arm`` can be used but this is significantly more difficult to setup.
-* The target to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won't find the libraries needed to pass the cmake compiler check.
+* The targets to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won't find the libraries needed to pass the cmake compiler check.
 
 As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions
 that are supported on Armv7-A we can still get most of the value of running the
 tests using the same ``qemu-arm`` that we used for Armv7-A by building and
 running the test cases for Armv7-A but using the builtins compiled for
-Armv6-M, Armv7-M or Armv7E-M. This will not catch instructions that are
-supported on Armv7-A but not Armv6-M, Armv7-M and Armv7E-M.
-
-To get the cmake compile test to pass the libraries needed to successfully link
-the test application will need to be manually added to ``CMAKE_CFLAGS``.
-Alternatively if you are using version 3.6 or above of cmake you can use
+Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked
+into a binary and execute the tests correctly but it will not catch if the
+builtins use instructions that are supported on Armv7-A but not Armv6-M,
+Armv7-M and Armv7E-M.
+
+To get the cmake compile test to pass you will need to pass the libraries
+needed to successfully link the cmake test via ``CMAKE_CFLAGS``. It is
+strongly recommended that you use version 3.6 or above of cmake so you can use
 ``CMAKE_TRY_COMPILE_TARGET=STATIC_LIBRARY`` to skip the link step.
 
 * ``-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY``
 * ``-DCOMPILER_RT_OS_DIR="baremetal"``
 * ``-DCOMPILER_RT_BUILD_BUILTINS=ON``
 * ``-DCOMPILER_RT_BUILD_SANITIZERS=OFF``
 * ``-DCOMPILER_RT_BUILD_XRAY=OFF``
 * ``-DCOMPILER_RT_BUILD_LIBFUZZER=OFF``
 * ``-DCOMPILER_RT_BUILD_PROFILE=OFF``
 * ``-DCMAKE_C_COMPILER=${host_install_dir}/bin/clang``
 * ``-DCMAKE_C_COMPILER_TARGET="your *-none-eabi target"``
+* ``-DCMAKE_ASM_COMPILER_TARGET="your *-none-eabi target"``
 * ``-DCMAKE_AR=/path/to/llvm-ar``
 * ``-DCMAKE_NM=/path/to/llvm-nm``
 * ``-DCMAKE_RANLIB=/path/to/llvm-ranlib``
 * ``-DCOMPILER_RT_BAREMETAL_BUILD=ON``
 * ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON``
 * ``-DLLVM_CONFIG_PATH=/path/to/llvm-config``
 * ``-DCMAKE_C_FLAGS="build-c-flags"``
-* ``-DCMAKE_ASM_FLAGS="${arm_cflags}"``
+* ``-DCMAKE_ASM_FLAGS="build-c-flags"``
 * ``-DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armv7-A/sysroot"``
 * ``-DCOMPILER_RT_INCLUDE_TESTS=ON``
 * ``-DCOMPILER_RT_TEST_COMPILER="/path/to/clang"``
 * ``-DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"``
 
 The Armv6-M builtins will use the soft-float ABI. When compiling the tests for
 Armv7-A we must include ``"-mthumb -mfloat-abi=soft -mfpu=none"`` in the
 test-c-flags. We must use an Armv7-A soft-float abi sysroot for ``qemu-arm``.
 
-Unfortunately at time of writing the Armv7-M and Armv7E-M builds of
-compiler-rt will always include assembler files including floating point
-instructions. This means that building for a cpu without a floating point unit
-requires something like removing the arm_Thumb1_VFPv2_SOURCES from the
-arm_Thumb1_SOURCES in builtins/CMakeLists.txt. The float-abi of the compiler-rt
-library must be matched by the float abi of the Armv7-A sysroot used by
-qemu-arm.
-
 Depending on the linker used for the test cases you may encounter BuildAttribute
 mismatches between the M-profile objects from compiler-rt and the A-profile
-objects from the test. The lld linker does not check the BuildAttributes so it
-can be used to link the tests by adding -fuse-ld=lld to the
+objects from the test. The lld linker does not check the profile
+BuildAttribute so it can be used to link the tests by adding -fuse-ld=lld to the
 ``COMPILER_RT_TEST_COMPILER_CFLAGS``.
+
+Alternative using a cmake cache
+-------------------------------
+If you wish to build, but not test compiler-rt for Armv6-M, Armv7-M or Armv7E-M
+the easiest way is to use the BaremetalARM.cmake recipe in clang/cmake/caches.
+
+You will need a bare metal sysroot such as that provided by the GNU ARM
+Embedded toolchain.
+
+The libraries can be built with the cmake options:
+
+* ``-DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi``
+* ``-DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi``
+* ``-DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi``
+* ``-C /path/to/llvm/source/tools/clang/cmake/caches/BaremetalARM.cmake``
+* ``/path/to/llvm``
+
+**Note** that for the recipe to work the compiler-rt source must be checked out
+into the directory llvm/runtimes. You will also need clang and lld checked out.
+