[Docs] Multilib design

[amilendra]

Thanks for the design docs. At a high-level this gives a good summary on what you intend the multilib feature to do. Couple of suggestions.

clang/docs/Multilib.rst

87–90

Would it be possible to give more details on how the toolchain makes this choice?

98

Would it be worth to also add a flag to activate/deactivate the multilib feature for the duration that it is experimental?
IIUC, a similar mechanism was used before switching to a new pass manager.

[amilendra]

clang/docs/Multilib.rst

90

What would be the effect of layering? Does this mean multiple choices for --sysroot, -isystem and -L` will be used?

[michaelplatings]

Made changes requested by @amilendra

[michaelplatings]

clang/docs/Multilib.rst

87–90

Added below: "This decision is hard-coded per ToolChain subclass"

90

Added a section below on multilib layering

98

I think no because the feature is implicitly disabled by a lack of multilib.yaml. I added some text explaining what's necessary to enable the feature which hopefully makes that clearer.

[peter.smith]

Thanks very much for writing this. Will be worth updating the RFC with a link as I think that this is an approachable place to comment on the format and selection model without the implementation detail. I'm happy with what has been written so far. My suggestions are mostly additions rather than modifications.

Could be worth adding a design principles section at the end, this may help someone looking back rationalise the choices made. For example I think that it is an important decision to expand the command-line options in clang prior to multi-lib selection to avoid another implementation of the complex architecture feature selection code in multilib.

May be worth a quick description of how we might model some awkward cases like position independent and non-position independent code. This is analagous to soft and softfp.

Position independence is an awkward case to model, which may be worth showing as an example.
* if my incoming flags include position independent code [PIC] then I need to select the [PIC] libraries, which implies that we want a [PIC] flag.
* if my incoming flags do not include position independent code then I can use [PIC] libraries, but I would prefer to use not PIC if it were available.
If we did have separate PIC and non PIC choices then I think this would need two features [PIC] and [non-PIC] with incoming non-position independent code having both [PIC, non-PIC] so it could match both, but position indendent code would only have [PIC]. Presumably we'd need the [non-PIC] variant sorted after.

clang/docs/Multilib.rst

103

May be worth indicating that this is to support compatibility with existing clang multilib, we expect most implementations to use all matching variants.

106

Considering the BareMetal ToolChain is used by RISC-V, Arm and AArch64 is there scope for a different choice per toolchain. I guess that we could put a predicate function for the ToolChain that could change due to architecture.

123

Although it can be worked out by looking at the example, which I think is your intention from the comment. It may be worth a section on Syntax that extracts some of these.

Syntax of multilib.yaml
===============

Some suggestions/questions:

• Does the file have to contain both variants: and flagmap: are either of them optional?
• Do variants have to come before flagmap, or is order not important.
• Will be worth highlighting that the order of the variants may be significant depending on the toolchain.
• Each variant is made up of dir: (path added to sysroot?), flags: [comma separated list of flags] and printArgs: [comma separated list of command line options for -print-multi-lib and -print-multi-directory, but not important for multilib selection]
• Each flagmap is made up of a POSIX extended syntax regex: that matches a flag, and (one of?) matchFlags/noMatchFlags which either adds or removes respecively a [comma separated list of flags]

170

Although very Arm specific. I'd use [V6M] here as ArmV6 (pre-cortex so no explicit profile) is very different to ArmV6-m. Also in regex section below.

[michaelplatings]

Incorporate suggestions from @peter.smith

[michaelplatings]

Could be worth adding a design principles section at the end

That was a very helpful suggestion, thank you. Writing them out helped me examine the compatibility story more carefully. Consequently I added a requirement for a "minimumClangVersion" property, although I haven't implemented that in the other patches yet.

May be worth a quick description of how we might model some awkward cases

Not done yet. This certainly can be modelled but I agree an example would be helpful to provide.

clang/docs/Multilib.rst

106

For the BareMetal ToolChain the only one RISC-V multilibs can ever match a given set of command line options so this isn't a problem in practice.

123

I opted to keep the spec, such as it is, in the example, but I've expanded it to answer your questions, unless it's implicit in it being YAML (e.g. lists can be comma separated but there are other ways to express them in YAML).

[peter.smith]

Thanks for the update, one nit in the language, otherwise looks good to me. It is a good reflection of the implementation.

clang/docs/Multilib.rst

251

"The API need only multilib selection" doesn't look right. Perhaps "The API can do multilib selection"

[peter.smith]

clang/docs/Multilib.rst

126

Although implicit in the mechanism, is it worth highlighting that layered multib.yaml authors will need to make sure that the includes and the libraries in the layer need to be complete enough to mask any incompatibilities?

For example if in the -fno-exceptions case there exists a library libX.a that is affected by -fno-exceptions but is present in the "Core directory" but not the layered no-exceptions directory then the exceptions libX.a will be selected.

/fp/libX.a (should have a noexcept variant in noexcept but multilib implementer gets it wrong!)
/fp/libY.a
/fp/lib/noexecpt/libY.a

execeptions is perhaps not the best example here as including a file with exceptions when the intention is -fno-exceptions is not optimal, but it will work. If there is layering for floating point calling conventions then a mix could result in a broken program.

[michaelplatings]

Tweak multilib.yaml spec slightly

[MaskRay]

clang/docs/Multilib.rst

42

s/clang/Clang/

43

Where is MultilibBuilder?

44

lib/Driver/ToolChains/Gnu.cpp

163

Generally the comments in this example feel verbose. Consider changing some sentences with briefer but equivalent wording.

For example, "It is required to be present." is repeated several times. It can be combined with a previous sentence like "This is required and ..." or "This required key defines ..."

230

I think Example of noMatchFlags - is verbose.

238

What does "API" refer to? A function defined in llvm-project/clang/lib?

[michaelplatings]

Rebase and apply changes requested by @MaskRay and @peter.smith

clang/docs/Multilib.rst

238

Changed "API" to "interface" and reworded to hopefully make it clearer.

[michaelplatings]

PrintArgs -> PrintOptions

[michaelplatings]

• Make "experimental" more obvious.
• Demonstrate using a Clang-generated flag directly in Flags

[michaelplatings]

flags -> tags

[peter.smith]

I've set approved from the Arm side. Please leave some time for people in the US time zone to leave any final comments or ask for extensions. I've left a suggestion that can be applied prior to commit if you want to take it up.

clang/docs/Multilib.rst

57

Can we add a forward reference to the Stable Interface design principle, or perhaps move it here so that it is more obvious?

[michaelplatings]

Reference Multilib.rst from comment in Multilib.h

[michaelplatings]

tags->flags

[MaskRay]

clang/docs/Multilib.rst

242

Add quotes

[michaelplatings]

Rebase

[michaelplatings]

Add quotes

[michaelplatings]

Rebase

[michaelplatings]

Rebase

[michaelplatings]

Update docs to reflect recent changes

[michaelplatings]

FlagMap->Mappings

[phosek]

LGTM

clang/docs/Multilib.rst

232

Use backticks to render the flag name as code, same for the other instances in this paragraph.

clang/docs/Multilib.rst

+========
+Multilib
+========
+
+Introduction
+============
+
+This document describes how multilib is implemented in Clang.
+
+What is multilib and why might you care?
+If you're :doc:`cross compiling<CrossCompilation>` then you can't use native
+system headers and libraries. To address this, you can use a combination of
+``--sysroot``, ``-isystem`` and ``-L`` options to point Clang at suitable
+directories for your target.
+However, when there are many possible directories to choose from, it's not
+necessarily obvious which one to pick.
+Multilib allows a toolchain designer to imbue the toolchain with the ability to
+pick a suitable directory automatically, based on the options the user provides
+to Clang. For example, if the user specifies
+``--target=arm-none-eabi -mcpu=cortex-m4`` the toolchain can choose a directory
+containing headers and libraries suitable for Armv7E-M, because it knows that's
+a suitable architecture for Arm Cortex-M4.
+Multilib can also choose between libraries for the same architecture based on
+other options. For example if the user specifies ``-fno-exceptions`` then a
+toolchain could select libraries built without exception support, thereby
+reducing the size of the resulting binary.
+
+Design
+======
+
+Clang supports GCC's ``-print-multi-lib`` and ``-print-multi-directory``
+options. These are described in
+`GCC Developer Options <https://gcc.gnu.org/onlinedocs/gcc-12.2.0/gcc/Developer-Options.html>`_.
+
+There are two ways to configure multilib in Clang: hard-coded or via a
+configuration file.
+
+Hard-coded Multilib
+===================
+
+The available libraries can be hard-coded in Clang. Typically this is done
+using the ``MultilibBuilder`` interface in
+``clang/include/clang/Driver/MultilibBuilder.h``.
+There are many examples of this in ``lib/Driver/ToolChains/Gnu.cpp``.
+The remainder of this document will not focus on this type of multilib.
+
+Multilib via configuration file
+===============================
+
+Some Clang toolchains support loading multilib configuration from a
+``multilib.yaml`` configuration file.
+
+A ``multilib.yaml`` configuration file specifies which multilib variants are
+available, their relative location, what compilation options were used to build
+them, and the criteria by which they are selected.
+
+Multilib processing
+===================
+
+Clang goes through the following steps to use multilib from a configuration
+file:
+#. Convert command line arguments to flags. Clang can accept the same
+   information via different arguments - for example,
+   ``--target=arm-none-eabi -march=armv7-m`` and
+   ``--target=armv7m-none-eabi`` are equivalent. Clang can also accept many
+   independent pieces of information within a single flag - for example
+   ``-march=armv8.1m.main+fp+mve`` specifies the architecture and two
+   extensions in a single command line argument.
+   To make it easier for the multilib system, Clang converts the command line
+   arguments into a standard set of simpler "flags". In many cases these flags
+   will look like a command line argument with the leading ``-`` stripped off,
+   but where a suitable form for the flag doesn't exist in command line
+   arguments then its form will be different. For example, an Arm architecture
+   extension is represented like ``march=+mve`` since there's no way to specify
+   it in isolation in a command line argument.
+   To see what flags are emitted for a given set of command line arguments, use
+   the ``-print-multi-selection-flags-experimental`` command line argument
+   along with the rest of the arguments you want to use.
+#. Load ``multilib.yaml`` from sysroot.
+#. Generate additional flags. ``multilib.yaml`` contains a ``flagMap`` section,
+   which specifies how to generate additional flags based on the flags derived
+   from command line arguments. Flags are matched using regular expressions.
+   These regular expressions shall use the POSIX extended regular expression
+   syntax.
+#. Match flags against multilib variants. If the generated flags are a superset
+   of the flags specified for a multilib variant then the variant is considered
+   a match.
+   If more than one variant matches then a toolchain may opt to either use only
+   the *last* matching multilib variant, or may use all matching variants,
+   thereby :ref:`layering<Multilib layering>` them.
+#. Generate ``-isystem`` and ``-L`` arguments. Iterate in reverse order over
+   the matching multilib variants, and generate ``-isystem`` and ``-L``
+   arguments based on the multilib variant's directory.
+
+Multilib layering
+=================
+
+When Clang selects multilib variants, it may find that more than one variant
+matches.
+
+It is up to the ToolChain subclass to decide what to do in this case.
+There are two options permitted:
+#. Use only the *last* matching multilib variant. This option exists primarily
+   for compatibility with the previous multilib design.
+#. Use all matching variants, thereby layering them.
+
+This decision is hard-coded per ToolChain subclass. The latter option is
+preferred for ToolChain subclasses without backwards compatibility
+requirements.
+
+If the latter option is chosen then ``-isystem`` and ``-L`` arguments will be
+generated for each matching multilib variant, in reverse order.
+
+This means that the compiler or linker will find files in the last matching
+multilib variant that has the given file.
+This behaviour permits multilib variants with only a partial set of files.
+This means a toolchain can be distributed with one base multilib variant
+containing all system headers and includes, and more specialised multilib
+variants containing only files that are different to those in the base variant.
+
+For example, a multilib variant could be compiled with ``-fno-exceptions``.
+This option doesn't affect the content of header files, nor does it affect the
+C libraries. Therefore if multilib layering is supported by the ToolChain
+subclass and a suitable base multilib variant is present then the
+``-fno-exceptions`` multilib variant need only contain C++ libraries.
+
+It is the responsibility of layered multilib authors to ensure that headers and
+libraries in each layer are complete enough to mask any incompatibilities.
+
+Stability
+=========
+
+Multilib via configuration file shall be considered an experimental feature
+until LLVM 18, at which point ``-print-multi-selection-flags-experimental``
+should be renamed to ``-print-multi-selection-flags``.
+A toolchain can opt in to using this feature by including a ``multilib.yaml``
+file in its distribution, once support for it is added in relevant ToolChain
+subclasses.
+Once stability is reached, flags emitted by ``-print-multi-selection-flags``
+should not be removed or changed, although new flags may be added.
+
+Restrictions
+============
+
+Despite the name, multilib is used to locate both ``include`` and ``lib``
+directories. Therefore it is important that consistent options are passed to
+the Clang driver when both compiling and linking. Otherwise inconsistent
+``include`` and ``lib`` directories may be used, and the results will be
+undefined.
+
+multilib.yaml
+=============
+
+The below example serves as a small of a possible multilib, and documents
+the available options.
+
+For a more comprehensive example see
+``clang/test/Driver/baremetal-multilib.yaml`` in the ``llvm-project`` sources.
+
+.. code-block:: yaml
+  # multilib.yaml
+
+  # Syntax is YAML 1.2
+
+  # This required field defines the minimum version of Clang required
+  # to use this file.
+  # Clang will emit an error if this number is greater than its version, but
+  # will accept lesser versions.
+  ClangMinimumVersion: 17.0.0
+
+  # The rest of this file is in two parts:
+  # 1. A list of multilib variants.
+  # 2. A list of regular expressions that may match flags generated from
+  #    command line arguments, and further flags that shall be added if the
+  #    regex matches.
+  # It is acceptable for the file to contain properties not documented here,
+  # and these will be ignored by Clang.
+
+  # List of multilib variants. Required.
+  # The ordering of Variants is important if more than one variant can match
+  # the same set of flags. See the docs on multilib layering for more info.
+  Variants:
+
+  # Example of a multilib variant targeting Arm v6-M.
+  # Dir is the relative location of the directory containing the headers
+  # and/or libraries.
+  # Exactly how Dir is used is left up to the ToolChain subclass to define, but
+  # typically it will be joined to the sysroot.
+  - Dir: thumb/v6-m
+    # List of one or more "flags", as generated by Clang from the command line
+    # arguments and subsequently from the flagMap.
+    # Here, if the flags are a superset of {V6M} then this multilib variant
+    # will be considered a match.
+    Flags: [V6M]
+    # If a user invokes Clang with -print-multi-lib then the options it
+    # prints will be derived from PrintOptions. For example:
+    #   thumb/v6-m;@-target=thumbv6m-none-eabi
+    # PrintOptions is not used by Clang otherwise.
+    PrintOptions: [--target=thumbv6m-none-eabi]
+
+  # Similarly, a multilib variant targeting Arm v7-M with an FPU (floating
+  # point unit).
+  - Dir: thumb/v7-m
+    # Here, the flags generated by Clang must be a superset of
+    # {V7MorLater, hasFPU} for this multilib variant to be a match.
+    Flags: [V7MorLater, hasFPU]
+    PrintOptions: [--target=thumbv7m-none-eabi, -mfpu=fpv4-sp-d16]
+
+
+  # The second section of the file is a list of regular expressions that are
+  # used to map from auto-detected flags to custom flags. This is optional.
+  # Each regular expression must match a whole flag string.
+  # One or both of "MatchFlags" & "NoMatchFlags" must be specified.
+  # "MatchFlags" flags will be added if an argument matches, while
+  # "NoMatchFlags" flags will be added otherwise.
+  FlagMap:
+
+  # Set a "V6M" flag if the regular expression matches any of the flags
+  # generated from the command line arguments.
+  # Regex is a POSIX extended regular expression strings.
+  - Regex: target=thumbv6m-.*
+    # MatchFlags is a list of one or more strings.
+    MatchFlags: [V6M]
+
+  # Set a "V7MorLater" flag if the regular expression matches any of the flags
+  # generated from the command line arguments.
+  - Regex: target=thumbv[7-9].*
+    MatchFlags: [V7MorLater]
+
+  # Set hasFPU if mfpu=none *doesn't* match.
+  - Regex: mfpu=none
+    # NoMatchFlags is a list of one or more strings.
+    NoMatchFlags: [hasFPU]
+
+Design principles
+=================
+
+Stable interface
+----------------
+
+multilib.yaml and -print-multi-selection-flags-experimental are new interfaces
+to Clang. In order for them to be usable over time and across LLVM versions
+their interfaces should be stable.
+The new multilib system will be considered experimental in LLVM 17, but in
+LLVM 18 it will be stable. In particular this is important to which multilib
+selection flags Clang generates from command line arguments. Once a flag is
+generated by a released version of Clang it may be used in multilib.yaml
+files that exist independently of the LLVM release cycle, and therefore
+ceasing to generate the flag would be a breaking change and should be
+avoided.
+
+Incomplete interface
+--------------------
+
+The new multilib system does multilib selection based on only a limited set of
+command line arguments, and limits which flags can be used for multilib
+selection. This is in order to avoid committing to too large an interface.
+Later LLVM versions can add support for multilib selection from more command
+line arguments as needed.
+
+Extensible
+----------
+
+It is likely that the configuration format will need to evolve in future to
+adapt to new requirements.
+Using a format like YAML that supports key-value pairs helps here as it's
+trivial to add new keys alongside existing ones.
+
+Backwards compatibility
+-----------------------
+
+New versions of Clang should be able to use configuration written for earlier
+Clang versions.
+To avoid behaving in a way that may be subtly incorrect, Clang should be able
+to detect if the configuration is too new and emit an error.
+
+Forwards compatibility
+----------------------
+
+As an author of a multilib configuration, it should be possible to design the
+configuration in such a way that it is likely to work well with future Clang
+versions. For example, if a future version of Clang is likely to add support
+for newer versions of an architecture and the architecture is known to be
+designed for backwards compatibility then it should be possible to express
+compatibility for such architecture versions in the multilib configuration.
+
+Not GNU spec files
+------------------
+
+The GNU spec files standard is large and complex and there's little desire to
+import that complexity to LLVM. It's also heavily oriented towards processing
+command line argument strings which is hard to do correctly, hence the large
+amount of logic dedicated to that task in the Clang driver. While compatibility
+with GNU would bring benefits, the cost in this case is deemed too high.
+
+Avoid re-inventing feature detection in the configuration
+---------------------------------------------------------
+
+A large amount of logic in the Clang driver is dedicated to inferring which
+architectural features are available based on the given command line arguments.
+It is neither desirable nor practical to repeat such logic in each multilib
+configuration. Instead the configuration should be able to benefit from the
+heavy lifting Clang already does to detect features.
+
+Low maintenance
+---------------
+
+Multilib is a relatively small feature in the scheme of things so supporting it
+should accordingly take little time. Where possible this should be achieved by
+implementing it in terms of existing features in the LLVM codebase.
+
+Minimal additional API surface
+------------------------------
+
+The greater the API surface, the greater the difficulty of keeping it stable.
+Where possible the additional API surface should be kept small by defining it
+in relation to existing APIs. An example of this is keeping a simple
+relationship between flag names and command line arguments where possible.
+Since the command line arguments are part of a stable API they are unlikely
+to change, and therefore the flag names get the same stability.
+
+Low compile-time overhead
+-------------------------
+
+If the process of selecting multilib directories must be done on every
+invocation of the Clang driver then it must have a negligible impact on
+overall compile time.

clang/docs/index.rst

 ================
 
 .. toctree::
    :maxdepth: 1
 
    CodeOwners
    InternalsManual
    DriverInternals
+   Multilib
    OffloadingDesign
    PCHInternals
    ItaniumMangleAbiTags
    HardwareAssistedAddressSanitizerDesign.rst
    ConstantInterpreter
 
 
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`search`