[draft] runtimes doc wip

[thakis]

Summary

This will become documentation for the runtimes build. Not quite ready for a look yet.

[mstorsjo]

FWIW, libc++ has some amount of docs on the same matter, see e.g. https://libcxx.llvm.org/BuildingLibcxx.html#the-default-build, also CC @ldionne. Regarding naming of the different build configurations, in the libc++ docs, the setup originally called "runtimes build" is now called "bootstrapping build", as "runtimes build" easily can be confused with manually building using the llvm-project/runtimes directory.

llvm/docs/RuntimeBuild.rst

24

AFAIK, 2. here isn't true. Clang doesn't get the value of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR hardcoded in anywhere (and depending on the --target option, there might be different expectations for where to find the runtimes).

Additionally, Clang doesn't consistently check for runtimes in in the per-target runtime directory, but code for doing that is enabled in a handful of targets (some of the ones in clang/lib/Driver/ToolChains), but not all of them; the code for doing it is scattered around the subclasses. The ones that support it look for both kinds of layouts and add the directories if they are found - so afaik, Clang is pretty much agnostic.

52

These examples cover the case of compiler-rt runtimes, but e.g. libc++ usually is installed in <toolchain>/lib right? And with the per-target runtimes, it's installed in <toolchain>/lib/<triple>.

It also affects how libc++'s __config_site is installed; most libc++ headers are installed in a target-independent directory, while __config_site is kept in a per-target directory.

81

Hmm, I'm pretty sure the version number is picked up from the host OS. (I get the same version number, matching my version of macOS, when I run the clang binary from a bunch of differing old and new Xcode versions.)

87

Just as a tangentially related sidenote; arm64 on darwin is a tricky case, as LLVM internally normalizes it to aarch64 even on that OS - or at least it used to. clang -arch arm64 -dumpmachine did print aarch64-apple-darwin... still in Xcode 12; in Xcode 13 it keeps it as arm64-apple-darwin... though.

100

In the normalized form, mingw (any OS string starting with mingw32 iirc) corresponds to windows-gnu, not windows-mingw.

141

FWIW, at least for libc++, building it with LLVM_ENABLE_PROJECTS is discouraged/deprecated these days. I think both options are still meant to work for e.g. compiler-rt though. (One usecase I've heard about there, is for other languages, who build LLVM and compiler-rt using the existing host C compiler, but don't actually build Clang at all.)

Here it could also be useful to mention building by pointing cmake at llvm-project/runtimes, which is a standalone-ish build, where you manually configure cmake to use your desired (cross)compiler. Similar to pointing cmake directly at e.g. llvm-project/libc++ before, but allowing processing more than one runtime in one cmake invocation.

149

This sentence could be clarified that it is for the runtimes build mode (as the previous bullet covers both).

[thakis]

Thanks for the feedback! As background, I started this doc to try and understand this build config better. (What goals does it serve? Why does it have basically 0 adoption 3 years in? etc)

Then I got distracted and did something else, but aeubanks asked me to upload what I had so far so he could read it. So it's pretty incomplete and not fully researched. But I'm glad I uploaded it, since I got all these valuable comments :)

I'll update it in a bit.

llvm/docs/RuntimeBuild.rst

24

True clang always looks in the place where LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON puts the libraries first, and then in the place where LVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF puts them second.

Will update.

52

Yes, this is incomplete. Will fix.

81

Will check and update, thanks.

87

As a tangent to the tangent, (32-bit) arm is also weird in that (e.g.) --target=arm-linux-androideabi leads to a cc1 -triple armv4t-unknown-linux-android, but --target=arm-linux-androideabi -march=armv7a leads to a cc1 -triple armv7-unknown-linux-android – i.e. march flags affect the cc1 level triple. You can get the same cc triple by passing --armv7a-linux-androideabi without -march flag.

On the flipside, --target=i386-linux and --target=i386-unknown-linux result in the same cc1 triple (i386-unknown-linux).

I don't fully understand the motivation behind LLVM_ENABLE_PER_TARGET_RUNTIME_DIR yet. I thought it's so it's possible to build compiler-rt for different target configs, but it imho it currently doesn't work well because:

1. It uses the canonicalized driver-level triple.
   • This makes it impossible to build a toolchain that just works for users, since users can use a --target flag that canonicalizes to the same triple that you've built libraries for, but since clang looks in the non-canonicalized location you have to guess all possible --target spellings users might use and include symlinks or what
   • likewise, you have to guess if users will pick arm level via -march or via triple (or have symlinks for both)
   • before D115049, you had to have a dedicated symlink for every single android abi level
   • In the other direction, it makes it impossible to have different compiler-rt prebuilts for (say) avx512 on and off, since that's not in the triple

2. It's not really clear (to me) what the goal even is

100

Thanks, wil fix.

141

Thanks, will add.

Side note: It's unfortunate that cmake doesn't have a way to achieve this (yet?) without building the runtimes in a sub-ninja-invocation. I tried LLVM_ENABLE_PROJECTS the other day, and one issue I ran into while debugging something was that ninja -v (at the outer level) didn't print verbose commands for the runtimes build. The GN build conceptually only supports a "runtimes build" (in that compiler-rt etc are always built with just-built clang), but it can do that in single build graph.

[thakis]

@phosek @MaskRay fyi

[mstorsjo]

In D121761#3385773, @thakis wrote:

Thanks for the feedback! As background, I started this doc to try and understand this build config better. (What goals does it serve? Why does it have basically 0 adoption 3 years in? etc)

Do you mean LLVM_ENABLE_RUNTIMES or LLVM_ENABLE_PER_TARGET_RUNTIME here - I presume the latter?

llvm/docs/RuntimeBuild.rst

87

(Speaking as someone who have touched these things quite a bit - not as someone arguing specifically for or against the feature.)

I think the main motivation is to better support multiarch setups, and with better granularity. For compiler-rt, that's already supported in various ways with filenames such as libclang_rt.asan-x86_64.a. But for e.g. libc++, in the current release tarballs, it's provided in <prefix>/lib/libc++.so which clearly doesn't scale when targeting even 32+64 bit x86 on the same OS.

Additionally, it's also supposed to handle other variability than just plain different target architectures - e.g. if building with -fno-exceptions, it's supposed to be able to find a copy of libc++ that is built without exceptions, and if building with asan, it can give you an asan-instrumented libc++. AFAIK the fuchsia cache files build a bunch of runtimes (for fuchsia and possibly other OSes too) in this way.

I totally agree that the variability in mostly-equivalent triples is the main practical issue wrt this.

In my mingw toolchains (which are structured around cross compiling - native compiling is just a special case), I don't use this feature (yet). All arch-specific files are available in <prefix>/<triple>/{include,lib} which are implicitly picked up as sysroots, so there's none of that in <prefix>/lib, and the compiler-rt files contain the arch in the file name already.

[ldionne]

This is awesome -- I think the lack of documentation for the bootstrapping build is one of the main reasons for its slow adoption. Even as a very frequent libc++ contributor, I still don't understand some parts of it (for example how to pass options to specific runtimes, e.g. how to pass LIBCXX_TEST_CONFIG to the libc++ build through the bootstrapping build). It would be nice if we could publish this documentation.

llvm/docs/RuntimeBuild.rst

1

😄

5

I would love if we could officially start calling it the "bootstrapping build"!

63

Also watchos and tvos -- perhaps it's easier to just drop this parenthesis so it doesn't become outdated.

87

FWIW I'd be supportive of a plan to ship libc++ and libc++abi in a per-target runtime directory by default, if someone wanted to take over that task.

146

We should remove any reference to LLVM_ENABLE_PROJECTS for the runtimes, and instead use LLVM_ENABLE_RUNTIMES but root the CMake invocation at <monorepo>/runtimes instead of <monorepo>/llvm/runtimes.

Yes, that's a bit confusing, we could improve that.

[mstorsjo]

llvm/docs/RuntimeBuild.rst

87

Re using per-target runtime directories by default - it's not so much about an all-or-nothing situation, but I think the intent is to maybe enable it by default for some specific targets. (Clang doesn't support using it on all targets anyway yet.)

llvm/docs/RuntimeBuild.rst

+reviewers: collinbaker, jdionne, phosek, maskray, danalbert, mstorsjo
+cc: rnk
+
+==================
+The runtimes build
+==================
+
+.. contents::
+   :local:
+
+Introduction
+============
+
+This document describes the effect of setting
+``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR``, and the difference between
+``LLVM_ENABLE_PROJECTS`` and ``LLVM_ENABLE_RUNTIMES``.
+
+``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR``
+======================================
+
+``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR`` controls:
+
+1. Where LLVM's build system puts the runtime libraries it builds
+2. Where clang looks for runtime libraries
+
+
+Where LLVM's build system puts the runtime libraries it builds
+--------------------------------------------------------------
+
+As of early 2022, ``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR`` defaults to off
+everywhere. The "ON" setting is newer and it might become the default once
+kinks are worked out. For perspective, ``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR``
+was initially added mid-2018.
+
+The problem that the new setting tries to solve is XXX.
+
+With ``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF``, runtimes are placed in
+``lib/clang/$version/lib/$host_os/$runtime_name-$optional_suffix.so``. Here
+are a few concrete examples of runtime library names with this setup:
+
+* ``lib/clang/14.0.0/lib/darwin/libclang_rt.osx.a``
+* ``lib/clang/6.0.0/lib/linux/libclang_rt.asan-x86_64.a``
+* ``lib/clang/6.0.0/lib/linux/libclang_rt.asan-arm-android.so``
+* ``lib/clang/12.0.0/lib/windows/clang_rt.profile-x86_64.lib``
+
+With ``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON``, runtimes are instead
+organized by triple:``lib/clang/$version/lib/$triple/$runtime_name.so``. Here
+are a few concrete examples of this setup:
+
+* ``lib/clang/14.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a``
+* ``lib/clang/14.0.0/lib/aarch64-unknown-fuchsia/libclang_rt.builtins.a``
+
+Where clang looks for runtime libraries
+---------------------------------------
+
+clang's behavior depends on the current triple.
+
+Triples
++++++++
+
+If ``--target=`` is passed, its value is the current triple. Else, the current
+triple is inferred from ``-m32`` / ``-m64`` flags, ``-arch`` flags on darwin
+(in practice, "darwin" means "iOS or macOS"), or if no flags are passed, by the
+value of ``LLVM_DEFAULT_TARGET_TRIPLE`` used at cmake time.
+``LLVM_DEFAULT_TARGET_TRIPLE`` defaults to the triple of the system that clang
+was built on.
+
+``clang -v -c foo.c`` prints the current triple as ``"Target:
+$current_triple"``.
+
+``clang -dumpmachine`` prints the default triple.
+
+Here are a few examples of flags and the triples they imply:
+
+* ``--target=x86_64-unknown-linux-gnu`` sets the current triple to
+  ``x86_64-unknown-linux-gnu``.
+
+* ``-arch arm64`` when run with a clang built on macOS 12.2 sets the current
+  triple to ``arm64-apple-darwin21.2.0``. (If you use Xcode's clang, you can
+  discover the macOS version of the machine that built Xcode's ``clang`` binary
+  this way!)
+
+A triple has the general form of ``$cpu-$vendor-$os-$environment``. The fourth
+component (the environment) is optional.
+
+* "``cpu``" is something like ``x86_64`` (64-bit intel), ``i386`` (32-bit
+  intel), ``armv7a`` (32-bit arm v7 application profile), ``arm64`` (64-bit arm
+  on darwin), ``aarch64`` (64-bit arm on non-darwin). It determines which CPU
+  clang generates code for.
+
+* "``vendor``" is something like ``pc`` or ``apple``. It usually doesn't have
+  a big effect, and at least when targeting Linux it's often just ``unknown``.
+
+* "``os``" is the operating system to target, for example ``linux``,
+  ``windows``, ``darwin``, and so on. This component can have a number suffix,
+  for example ``darwin21.2.0``.
+
+* "``environment``" can communicate extra information. For example, when
+  targeting Windows, ``x86_64-pc-windows-msvc`` picks the Microsoft-compatible
+  ABI, while ``x86_64-pc-windows-mingw`` picks the gcc-compatible ABI. As
+  another example, when targeting Android, this is set to ``android`` (or,
+  for 32-bit arm, ``androideabi``). This can also end in a number, for example
+  to communicate Android NDK minimum supported OS version
+  (``armv7a-unknown-linux-androideabi23``) or Microsoft ABI version
+  (``x86_64-pc-windows-msvc15``).
+
+In practice, some of the components are sometimes omitted. For example,
+``--target=armv7a-linux-androideabi23`` omits the vendor. LLVM's triple parsing
+detects this. The interpreted triple with missing parts filled in is called
+the "canonical triple" and is ``armv7a-unknown-linux-androideabi23`` in this
+case. As another example, ``--target=x86_64-linux`` has
+``x86_64-unknown-linux`` as canonical target. However, if clang was built on
+a 64-bit Linux system, the default triple that's used when no flags are passed
+is ``x86_64-unknown-linux-gnu``.
+
+XXX -march, -mfloat-abi
+
+cc1 triple vs driver-level triple
+
+Runtime lookup
+++++++++++++++
+
+Implications
+------------
+
+* "tends to fallback too often, in rare cases"
+
+This has several implications: XXX
+
+* old way is stable, works well, can handle "fallback"
+
+* new way allows supporting multiple ABIs per ISA
+
+* "tends to fallback too rarely, in common cases"
+
+``LLVM_ENABLE_PROJECTS`` vs ``LLVM_ENABLE_RUNTIMES``
+====================================================
+
+LLVM's runtime libraries like compiler-rt (builtins, asan runtime, ...)
+and others can be built either by including them in ``LLVM_ENABLE_PROJECTS``
+or in ``LLVM_ENABLE_RUNTIMES`` ("runtimes build").
+
+The difference between the two modes are:
+
+* In the runtimes build, the runtimes are built with just-built clang.
+  In the ``LLVM_ENABLE_PROJECTS`` mode, they are built with the host compiler.
+
+* ``LLVM_ENABLE_PER_TARGET_RUNTIME_DIR`` (documented further up) defaults to
+  ``ON``.
+
+* More flexibility
+  * different sysroots per triple
+
+* Sub-cmakes
+  * Doesn't do anything you couldn't do yourself by running cmake several times