[docs][Remarks] Add documentation for remarks in LLVM

[thegameg]

Summary

This adds documentation that describes remarks in LLVM.

It aims at explaining what remarks are, how to enable them, and what users can do with the different modes.

It lists all the available flags in LLVM (excluding clang), and describes the expected YAML structure as well as the tools that support the YAML format today.

[paquette]

llvm/docs/Remarks.rst

36 ↗

(On Diff #208458)

Probably meant to write foo here? (Even though it's a dummy name, and foz is pretty good anyway)

72 ↗

(On Diff #208458)

Should probably mention that this is a POSIX regex.

119 ↗

(On Diff #208458)

Link to PGO documentation so that people can look at terminology for profiling etc.?

240 ↗

(On Diff #208458)

opt-diff is really *only* intended for a changing compiler + fixed source, right? "E.g." doesn't really pack enough punch here; we should probably just say that the tool isn't intended for a changing source + changing compiler.

[Meinersbur]

Thank your for working on the documentation!

Are you completing the FIXME's before committing?

llvm/docs/Remarks.rst

97 ↗

(On Diff #208458)

Clang does as well using -mllvm

282 ↗

(On Diff #208458)

Is this meant to use = (assignment instead of comparison)?

[thegameg]

Address Jessica and Michael's comments. Thanks!

llvm/docs/Remarks.rst

97 ↗

(On Diff #208458)

I initially didn't want to mention clang at all in the docs here, but if you think it's useful I can add the missing clang flags as well.

119 ↗

(On Diff #208458)

I tried looking for a link, but I can't find anything else than https://llvm.org/docs/HowToBuildWithPGO.html...

282 ↗

(On Diff #208458)

It is indeed!

[thegameg]

In D64355#1574026, @Meinersbur wrote:

Are you completing the FIXME's before committing?

I was going to leave that for later. If it's an issue I can hide the empty sections from the final output.

[thegameg]

llvm/docs/Remarks.rst

240 ↗

(On Diff #208458)

I'll mention both use-cases:

• changing compiler + fixed source
• changing source + fixed compiler

[thegameg]

Update the opt-diff use cases.

[paquette]

Hiding the empty sections sounds like a good call to me.

[paquette]

Added a few style suggestions (prefixed with "Suggestion") which aren't critical.

Also added a couple comments on things that are missing documentation or are ambiguous.

llvm/docs/Remarks.rst

57–58 ↗

(On Diff #208488)

Suggestion:

It might seem redundant, but I would add a sentence just saying what those two modes are. That would tell the reader about what they're going to find in this section.

63–65 ↗

(On Diff #208488)

Suggestion:

I think this sentence is somewhat confusing. The first part kind of jumbled my brains a bit and had me thinking that optimization remarks aren't often produced by the middle/back end.

Maybe just drop the first half of the sentence? E.g.

"Optimization remarks can be emitted as diagnostics. These diagnostics will..."

87 ↗

(On Diff #208488)

Should we call this an "optimization record" instead of serialized remarks? Later in the document, (in the opt-viewer section), you mention optimization records.

It would be good to adopt one term, or at least define both.

94–95 ↗

(On Diff #208488)

Suggestion:

I think that it would be good to mention the later YAML and object file sections here, and link to them (even though they are directly below.) That would improve the flow of the section by giving the reader a heads-up.

97–98 ↗

(On Diff #208488)

Suggestion:

I think there are two themes here:

1. Output style, format, etc.
2. Controlling the contents of the output via filtering, thresholding etc.

Would there be any nice way to split this up into those two themes? I think it would make it really easy to skim the options, even though there aren't many of them right now.

102 ↗

(On Diff #208488)

This is always a YAML file, right? Might be good to mention that here.

164–167 ↗

(On Diff #208488)

Can you add descriptions for these?

169 ↗

(On Diff #208488)

Suggestion: Drop the "all" to be consistent with the other lists below.

180 ↗

(On Diff #208488)

Suggestion: Replace "argument" with "arg entry", like above. "Argument" looks like it could be something else.

190–191 ↗

(On Diff #208488)

2/3 of the tools listed in this section do not produce HTML output.

Maybe say that the opt-viewer directory contains a collection of tools that visualize and summarize optimization records?

llvm/docs/index.rst

187 ↗

(On Diff #208488)

I think it would be better to call this a reference. It's not really notes, and it's not just diagnostics.

[thegameg]

Address Jessica's comments. Thanks!

[paquette]

LGTM!

llvm/trunk/docs/CodeGenerator.rst

 A section containing metadata on function stack sizes will be emitted when
 ``TargetLoweringObjectFile::StackSizesSection`` is not null, and
 ``TargetOptions::EmitStackSizeSection`` is set (-stack-size-section). The
 section will contain an array of pairs of function symbol values (pointer size)
 and stack sizes (unsigned LEB128). The stack size values only include the space
 allocated in the function prologue. Functions with dynamic stack allocations are
 not included.
 
-Emitting remark diagnostics in the object file
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A section containing metadata on remark diagnostics will be emitted when
--remarks-section is passed. The section contains:
-
-* a magic number: "REMARKS\\0"
-* the version number: a little-endian uint64_t
-* the total size of the string table (the size itself excluded):
-  little-endian uint64_t
-* a list of null-terminated strings
-* the absolute file path to the serialized remark diagnostics: a
-  null-terminated string.
-
 VLIW Packetizer
 ---------------
 
 In a Very Long Instruction Word (VLIW) architecture, the compiler is responsible
 for mapping instructions to functional-units available on the architecture. To
 that end, the compiler creates groups of instructions called *packets* or
 *bundles*. The VLIW packetizer in LLVM is a target-independent mechanism to
 enable the packetization of machine instructions.

llvm/trunk/docs/Remarks.rst

+=======
+Remarks
+=======
+
+.. contents::
+   :local:
+
+Introduction to the LLVM remark diagnostics
+===========================================
+
+LLVM is able to emit diagnostics from passes describing whether an optimization
+has been performed or missed for a particular reason, which should give more
+insight to users about what the compiler did during the compilation pipeline.
+
+There are three main remark types:
+
+``Passed``
+
+    Remarks that describe a successful optimization performed by the compiler.
+
+    :Example:
+
+    ::
+
+        foo inlined into bar with (cost=always): always inline attribute
+
+``Missed``
+
+    Remarks that describe an attempt to an optimization by the compiler that
+    could not be performed.
+
+    :Example:
+
+    ::
+
+        foo not inlined into bar because it should never be inlined
+        (cost=never): noinline function attribute
+
+``Analysis``
+
+    Remarks that describe the result of an analysis, that can bring more
+    information to the user regarding the generated code.
+
+    :Example:
+
+    ::
+
+        16 stack bytes in function
+
+    ::
+
+        10 instructions in function
+
+Enabling optimization remarks
+=============================
+
+There are two modes that are supported for enabling optimization remarks in
+LLVM: through remark diagnostics, or through serialized remarks.
+
+Remark diagnostics
+------------------
+
+Optimization remarks can be emitted as diagnostics. These diagnostics will be
+propagated to front-ends if desired, or emitted by tools like :doc:`llc
+<CommandGuide/llc>` or :doc:`opt <CommandGuide/opt>`.
+
+.. option:: -pass-remarks=<regex>
+
+  Enables optimization remarks from passes whose name match the given (POSIX)
+  regular expression.
+
+.. option:: -pass-remarks-missed=<regex>
+
+  Enables missed optimization remarks from passes whose name match the given
+  (POSIX) regular expression.
+
+.. option:: -pass-remarks-analysis=<regex>
+
+  Enables optimization analysis remarks from passes whose name match the given
+  (POSIX) regular expression.
+
+Serialized remarks
+------------------
+
+While diagnostics are useful during development, it is often more useful to
+refer to optimization remarks post-compilation, typically during performance
+analysis.
+
+For that, LLVM can serialize the remarks produced for each compilation unit to
+a file that can be consumed later.
+
+By default, the format of the serialized remarks is :ref:`YAML
+<yamlremarks>`, and it can be accompanied by a :ref:`section <remarkssection>`
+in the object files to easily retrieve it.
+
+:doc:`llc <CommandGuide/llc>` and :doc:`opt <CommandGuide/opt>` support the
+following options:
+
+
+``Basic options``
+
+    .. option:: -pass-remarks-output=<filename>
+
+      Enables the serialization of remarks to a file specified in <filename>.
+
+      By default, the output is serialized to :ref:`YAML <yamlremarks>`.
+
+    .. option:: -pass-remarks-format=<format>
+
+      Specifies the output format of the serialized remarks.
+
+      Supported formats:
+
+      * :ref:`yaml <yamlremarks>` (default)
+
+``Content configuration``
+
+    .. option:: -pass-remarks-filter=<regex>
+
+      Only passes whose name match the given (POSIX) regular expression will be
+      serialized to the final output.
+
+    .. option:: -pass-remarks-with-hotness
+
+      With PGO, include profile count in optimization remarks.
+
+    .. option:: -pass-remarks-hotness-threshold
+
+      The minimum profile count required for an optimization remark to be
+      emitted.
+
+Other tools that support remarks:
+
+:program:`llvm-lto`
+
+    .. option:: -lto-pass-remarks-output=<filename>
+    .. option:: -lto-pass-remarks-filter=<regex>
+    .. option:: -lto-pass-remarks-format=<format>
+    .. option:: -lto-pass-remarks-with-hotness
+    .. option:: -lto-pass-remarks-hotness-threshold
+
+:program:`gold-plugin` and :program:`lld`
+
+    .. option:: -opt-remarks-filename=<filename>
+    .. option:: -opt-remarks-filter=<regex>
+    .. option:: -opt-remarks-format=<format>
+    .. option:: -opt-remarks-with-hotness
+
+.. _yamlremarks:
+
+YAML remarks
+============
+
+A typical remark serialized to YAML looks like this:
+
+.. code-block:: yaml
+
+    --- !<TYPE>
+    Pass: <pass>
+    Name: <name>
+    DebugLoc: { File: <file>, Line: <line>, Column: <column> }
+    Function: <function>
+    Hotness: <hotness>
+    Args:
+      - <key>: <value>
+        DebugLoc: { File: <arg-file>, Line: <arg-line>, Column: <arg-column> }
+
+The following entries are mandatory:
+
+* ``<TYPE>``: can be ``Passed``, ``Missed``, ``Analysis``,
+  ``AnalysisFPCommute``, ``AnalysisAliasing``, ``Failure``.
+* ``<pass>``: the name of the pass that emitted this remark.
+* ``<name>``: the name of the remark coming from ``<pass>``.
+* ``<function>``: the mangled name of the function.
+
+If a ``DebugLoc`` entry is specified, the following fields are required:
+
+* ``<file>``
+* ``<line>``
+* ``<column>``
+
+If an ``arg`` entry is specified, the following fields are required:
+
+* ``<key>``
+* ``<value>``
+
+If a ``DebugLoc`` entry is specified within an ``arg`` entry, the following
+fields are required:
+
+* ``<arg-file>``
+* ``<arg-line>``
+* ``<arg-column>``
+
+opt-viewer
+==========
+
+The ``opt-viewer`` directory contains a collection of tools that visualize and
+summarize serialized remarks.
+
+.. _optviewerpy:
+
+opt-viewer.py
+-------------
+
+Output a HTML page which gives visual feedback on compiler interactions with
+your program.
+
+    :Examples:
+
+    ::
+
+        $ opt-viewer.py my_yaml_file.opt.yaml
+
+    ::
+
+        $ opt-viewer.py my_build_dir/
+
+
+opt-stats.py
+------------
+
+Output statistics about the optimization remarks in the input set.
+
+    :Example:
+
+    ::
+
+        $ opt-stats.py my_yaml_file.opt.yaml
+
+        Total number of remarks           3
+
+
+        Top 10 remarks by pass:
+          inline                         33%
+          asm-printer                    33%
+          prologepilog                   33%
+
+        Top 10 remarks:
+          asm-printer/InstructionCount   33%
+          inline/NoDefinition            33%
+          prologepilog/StackSize         33%
+
+opt-diff.py
+-----------
+
+Produce a new YAML file which contains all of the changes in optimizations
+between two YAML files.
+
+Typically, this tool should be used to do diffs between:
+
+* new compiler + fixed source vs old compiler + fixed source
+* fixed compiler + new source vs fixed compiler + old source
+
+This diff file can be displayed using :ref:`opt-viewer.py <optviewerpy>`.
+
+    :Example:
+
+    ::
+
+        $ opt-diff.py my_opt_yaml1.opt.yaml my_opt_yaml2.opt.yaml -o my_opt_diff.opt.yaml
+        $ opt-viewer.py my_opt_diff.opt.yaml
+
+.. _remarkssection:
+
+Emitting remark diagnostics in the object file
+==============================================
+
+A section containing metadata on remark diagnostics will be emitted when
+-remarks-section is passed. The section contains:
+
+* a magic number: "REMARKS\\0"
+* the version number: a little-endian uint64_t
+* the total size of the string table (the size itself excluded):
+  little-endian uint64_t
+* a list of null-terminated strings
+* the absolute file path to the serialized remark diagnostics: a
+  null-terminated string.
+
+The section is named:
+
+* ``__LLVM,__remarks`` (MachO)
+* ``.remarks`` (ELF)
+
+C API
+=====
+
+LLVM provides a library that can be used to parse remarks through a shared
+library named ``libRemarks``.
+
+The typical usage through the C API is like the following:
+
+.. code-block:: c
+
+    LLVMRemarkParserRef Parser = LLVMRemarkParserCreateYAML(Buf, Size);
+    LLVMRemarkEntryRef Remark = NULL;
+    while ((Remark = LLVMRemarkParserGetNext(Parser))) {
+       // use Remark
+    }
+    bool HasError = LLVMRemarkParserHasError(Parser);
+    LLVMRemarkParserDispose(Parser);
+
+.. FIXME: add documentation for llvm-opt-report.
+.. FIXME: add documentation for Passes supporting optimization remarks
+.. FIXME: add documentation for IR Passes
+.. FIXME: add documentation for CodeGen Passes

llvm/trunk/docs/index.rst

.. toctree::
    MCJITDesignAndImplementation
    ORCv2DesignAndImplementation
    CodeOfConduct
    CompileCudaWithLLVM
    ReportingGuide
    Benchmarking
    Docker
    BuildingADistribution
+   Remarks
 
 :doc:`GettingStarted`
    Discusses how to get up and running quickly with the LLVM infrastructure.
    Everything from unpacking and compilation of the distribution to execution
    of some tools.
 
 :doc:`CMake`
    An addendum to the main Getting Started guide for those using the `CMake

 
 :doc:`Docker`
    A reference for using Dockerfiles provided with LLVM.
 
 :doc:`BuildingADistribution`
   A best-practices guide for using LLVM's CMake build system to package and
   distribute LLVM-based tools.
 
+:doc:`Remarks`
+   A reference on the implementation of remarks in LLVM.
 
 Programming Documentation
 =========================
 
 For developers of applications which use LLVM as a library.
 
 .. toctree::
    :hidden: