diff --git a/clang/docs/InternalsManual.rst b/clang/docs/InternalsManual.rst --- a/clang/docs/InternalsManual.rst +++ b/clang/docs/InternalsManual.rst @@ -572,6 +572,275 @@ The Frontend library contains functionality useful for building tools on top of the Clang libraries, for example several methods for outputting diagnostics. +Compiler Invocation +------------------- + +One of the classes provided by the Frontend library is ``CompilerInvocation``, +which holds information that describe current invocation of the Clang frontend. +The information typically comes from the command line constructed by the Clang +driver, or directly from clients performing custom initialization. The data +structure is split into logical units used by different parts of the compiler, +for example ``PreprocessorOptions``, ``LanguageOptions`` or ``CodeGenOptions``. + +Command Line Interface +---------------------- + +The command line interface of the Clang ``-cc1`` frontend is defined alongside +the driver options in ``clang/Driver/Options.td``. The information making up an +option definition include the name and prefix (for example ``-std=``), form and +position of the option value, help text, aliases and more. Each option may +belong to a certain group and can be marked with zero or more flags. Options +accepted by the ``-cc1`` frontend are marked with the ``CC1Option`` flag. + +Command Line Parsing +-------------------- + +Option definitions are processed by the ``-gen-opt-parser-defs`` tablegen +backend, transformed into a list of invocations of the ``OPTION`` macro and +stored in ``clang/Driver/Driver.inc``. This file is then used to create instance +of ``llvm::opt::OptTable``, which acts as a command line preprocessor. The +preprocessed command line is stored in ``llvm::opt::ArgList``, an object that +provides an API for performing simple queries on the contents of the command +line. + +Finally, the ``CompilerInvocation::CreateFromArgs`` function is responsible for +the actual parsing of command line arguments. It maps the contents of the +``ArgList`` onto fields of ``CompilerInvocation``, normalizing the values in the +process. + +Command Line Generation +----------------------- + +Any valid ``CompilerInvocation`` created from a ``-cc1`` command line can be +also serialized back into semantically equivalent command line in a +deterministic manner. This enables features such as implicitly discovered, +explicitly built modules. + +.. + TODO: Create and link corresponding section in Modules.rst. + +Option Marshalling infrastructure +--------------------------------- + +The obvious way to parse command line arguments is to write the code that +queries ``ArgList`` and constructs ``CompilerInvocation`` manually. However, +given that behavior of most command line options is the same, this approach +would result in lots of repetitive code. This is the reason why most of the +code for parsing and generating arguments is automatically generated from +``Marshalling`` annotations on the tablegen option definitions. This section +goes through the basics of the automatic marshalling system. + +To read and modify contents of ``CompilerInvocation``, the system uses key +paths, which are declared in two steps. First, a tablegen definition for the +``CompilerInvocation`` member is created by inheriting from ``KeyPathAndMacro``. + +.. code-block:: + + // Options.td + + class LangOpts : KeyPathAndMacro<"LangOpts->", field, "LANG_"> {} + // CompilerInvocation member ^^^^^^^^^^ + // OPTION_WITH_MARSHALLING prefix ^^^^^ + +The first argument to the parent class is the beginning of the key path that +references the ``CompilerInvocation`` member. This argument ends with ``->`` if +the member is a pointer type or with ``.`` if it's a value type. The second +argument is the only parameter passed to the child class. The child class can be +used like so: ``LangOpts<"IgnoreExceptions">``, constructing a key path to the +field ``LangOpts->IgnoreExceptions``. The third argument passed to the parent +class is a string that the tablegen backend uses as a prefix to the +``OPTION_WITH_MARSHALLING``. Using the key path then instructs the backend to +generate the following code. + +.. code-block:: c++ + + // Options.inc + + #ifdef LANG_OPTION_WITH_MARSHALLING + LANG_OPTION_WITH_MARSHALLING([...], LangOpts->IgnoreExceptions, [...]) + #endif // LANG_OPTION_WITH_MARSHALLING + +Such definition can be used used in the parsing and generating functions: + +.. code-block:: c++ + + // CompilerInvoation.cpp + + bool CompilerInvocation::ParseLangArgs(LangOptions *LangOpts, ArgList &Args, + DiagnosticsEngine &Diags) { + bool Success = true; + + #define LANG_OPTION_WITH_MARSHALLING( \ + PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM, \ + HELPTEXT, METAVAR, VALUES, SPELLING, SHOULD_PARSE, ALWAYS_EMIT, KEYPATH, \ + DEFAULT_VALUE, IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, DENORMALIZER, \ + MERGER, EXTRACTOR, TABLE_INDEX) \ + PARSE_OPTION_WITH_MARSHALLING(Args, Diags, Success, ID, FLAGS, PARAM, \ + SHOULD_PARSE, KEYPATH, DEFAULT_VALUE, \ + IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, \ + MERGER, TABLE_INDEX) + #include "clang/Driver/Options.inc" + #undef LANG_OPTION_WITH_MARSHALLING + + // ... + + return Success; + } + + void CompilerInvocation::GenerateLangArgs(LangOptions *LangOpts, + SmallVectorImpl &Args, + StringAllocator SA) { + #define LANG_OPTION_WITH_MARSHALLING( \ + PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM, \ + HELPTEXT, METAVAR, VALUES, SPELLING, SHOULD_PARSE, ALWAYS_EMIT, KEYPATH, \ + DEFAULT_VALUE, IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, DENORMALIZER, \ + MERGER, EXTRACTOR, TABLE_INDEX) \ + GENERATE_OPTION_WITH_MARSHALLING( \ + Args, SA, KIND, FLAGS, SPELLING, ALWAYS_EMIT, KEYPATH, DEFAULT_VALUE, \ + IMPLIED_CHECK, IMPLIED_VALUE, DENORMALIZER, EXTRACTOR, TABLE_INDEX) + #include "clang/Driver/Options.inc" + #undef LANG_OPTION_WITH_MARSHALLING + + // ... + } + +How does the tablegen backend know what to put in place of ``[...]`` in the +generated file? This is specified by the ``Marshalling`` utilities described +below. All of them take a key path argument and possibly other information +required for parsing or generating the command line argument. + +**Positive Flag** + +The key path defaults to ``false`` and is set to ``true`` when the flag is +present on command line. + +.. code-block:: + + def fignore_exceptions : Flag<["-"], "fignore-exceptions">, Flags<[CC1Option]>, + MarshallingInfoFlag>; + +**Negative Flag** + +The key path defaults to ``true`` and is set to ``false`` when the flag is +present on command line. + +.. code-block:: + + def fno_verbose_asm : Flag<["-"], "fno-verbose-asm">, Flags<[CC1Option]>, + MarshallingInfoNegativeFlag>; + +**Negative and Positive Flag** + +The key path defaults to the specified value (``false``, ``true`` or some value +that's statically unknown in the tablegen file). Then, the key path is set to +the value associated with the flag that appears last on command line. + +.. code-block:: + + defm legacy_pass_manager : BoolOption<"f", "legacy-pass-manager", + CodeGenOpts<"LegacyPassManager">, DefaultFalse, + PosFlag, + NegFlag, + BothFlags<[CC1Option]>>; + +With most such pair of flags, the ``-cc1`` frontend accepts only the flag that +changes the default key path value. The Clang driver is responsible for +accepting both and either forwarding the changing flag or discarding the flag +that would just set the key path to its default. + +The first argument to ``BoolOption`` is a prefix that is used to construct the +full names of both flags. The positive flag would then be named +``flegacy-pass-manager`` and the negative ``fno-legacy-pass-manager``. +``BoolOption`` also implies the ``-`` prefix for both flags. It's also possible +to use ``BoolFOption`` that implies the ``"f"`` prefix and ``Group``. + +**String** + +The key path defaults to the specified string, or an empty one, if omitted. When +the option appears on the command line, the argument value is simply copied. + +.. code-block:: + + def isysroot : JoinedOrSeparate<["-"], "isysroot">, Flags<[CC1Option]>, + MarshallingInfoString, [{"/"}]>; + +**List of Strings** + +The key path defaults to an empty ``std::vector``. Values specified +with each appearance of the option on command line are appended to the vector. + +.. code-block:: + + def frewrite_map_file : Separate<["-"], "frewrite-map-file">, Flags<[CC1Option]>, + MarshallingInfoStringVector>; + +**Integer** + +The key path defaults to the specified integer value, or ``0`` if omitted. When +the option appears on the command line, its value gets parsed by ``llvm::APInt`` +and the result is assigned to the key path on success. + +.. code-block:: + + def mstack_probe_size : Joined<["-"], "mstack-probe-size=">, Flags<[CC1Option]>, + MarshallingInfoStringInt, "4096">; + +**Enumeration** + +The key path defaults to the value specified in ``MarshallingInfoEnum`` prefixed +by the contents of ``NormalizedValuesScope`` and ``::``. This ensures correct +reference to an enum case is formed even if the enum resides in different +namespace or is an enum class. If the value present on command line does not +match any of the comma-separated values from ``Values``, an error diagnostics is +issued. Otherwise, the corresponding element from ``NormalizedValues`` at the +same index is assigned to the key path (also correctly scoped). The number of +comma-separated string values and elements of the array within +``NormalizedValues`` must match. + +.. code-block:: + + def mthread_model : Separate<["-"], "mthread-model">, Flags<[CC1Option]>, + NormalizedValuesScope<"LangOptions::ThreadModelKind">, + MarshallingInfoEnum, "POSIX">, + Values<"posix,single">, + NormalizedValues<["POSIX", "Single"]>; + +.. + Intentionally omitting MarshallingInfoBitfieldFlag. It's adding some + complexity to the marshalling infrastructure and might be removed. + +It's also possible to define relationships between options. + +**Implication** + +The key path defaults to the default value from the primary ``Marshalling`` +annotation. Then, if any of the elements of ``ImpliedByAnyOf`` evaluate to true, +the key path value is changed to the specified value or ``true`` if missing. +Finally, the command line is parsed according to the primary annotation. + +.. code-block:: + + def fms_extensions : Flag<["-"], "fms-extensions">, Flags<[CC1Option]>, + MarshallingInfoFlag>, + ImpliedByAnyOf<[fms_compatibility.KeyPath], "true">; + +**Condition** + +The option is parsed only if the expression in ``ShouldParseIf`` evaluates to +true. + +.. code-block:: + + def fopenmp_enable_irbuilder : Flag<["-"], "fopenmp-enable-irbuilder">, Flags<[CC1Option]>, + MarshallingInfoFlag>, + ShouldParseIf; + +The building blocks provided by the marshalling infrastructure cover most of the +options accepted by the ``-cc1`` frontend. It's recommended to design new +command line options in such a way that the infrastructure can be reused and +manual implementation of parsing and generation can be avoided. However, manual +implementation is still possible. + The Lexer and Preprocessor Library ==================================