Add documentation for MIP

[ellis]

Summary

Add documentation for MIP instrumentation and the llvm-mipdata tool.

llvm/docs/CommandGuide/index.rst

.. toctree::
    llvm-diff
    llvm-dis
    llvm-dwarfdump
    llvm-lib
    llvm-libtool-darwin
    llvm-link
    llvm-lipo
    llvm-mca
+   llvm-mipdata
    llvm-otool
    llvm-profdata
    llvm-readobj
    llvm-stress
    llvm-symbolizer
    opt
 
 GNU binutils replacements

llvm/docs/CommandGuide/llvm-mipdata.rst

+llvm-mipdata - MIP tool
+=======================
+
+.. program:: llvm-mipdata
+
+SYNOPSIS
+--------
+
+:program:`llvm-mipdata` *command* [*args...*]
+
+DESCRIPTION
+-----------
+
+The :program:`llvm-mipdata` tool creates, manipulates, and reports
+information about :doc:`MIP <../MachineProfile>` profiles (``.mip`` files).
+
+General Options
+---------------
+
+.. option:: --help
+
+  Print a summary of command line options.
+
+.. option:: --profile=<profile>, -p <profile>
+
+  Specify the profile to work on.
+
+.. option:: --output=<output>, -o <output>
+
+  Specify the output file name.
+
+.. option:: --version
+
+  Print the current version of this tool.
+
+
+COMMANDS
+--------
+
+* :ref:`create <create>`
+* :ref:`merge <merge>`
+* :ref:`show <show>`
+
+.. _create:
+
+CREATE
+------
+
+SYNOPSIS
+^^^^^^^^
+
+:program:`llvm-mipdata create` [*options*] **-p <profile>** <map_file>
+
+DESCRIPTION
+^^^^^^^^^^^
+
+:program:`llvm-mipdata create` creates an empty <profile> from a <map_file>
+(``.mipmap`` file).
+
+.. _merge:
+
+MERGE
+-----
+
+SYNOPSIS
+^^^^^^^^
+
+:program:`llvm-mipdata merge` [*options*] **-p <profile>** <raw_files>
+
+DESCRIPTION
+^^^^^^^^^^^
+
+:program:`llvm-mipdata merge` merges raw profiles (``.mipraw`` files) into the
+profile.
+
+OPTIONS
+^^^^^^^
+
+.. option:: --strict
+
+  With strict mode, merging a corrupt raw profile will fail the command.
+  Disabled by default.
+
+.. _show:
+
+SHOW
+----
+
+SYNOPSIS
+^^^^^^^^
+
+:program:`llvm-mipdata show` [*options*] **-p <profile>**
+
+DESCRIPTION
+^^^^^^^^^^^
+
+:program:`llvm-mipdata show` prints the data from the given profile.
+
+OPTIONS
+^^^^^^^
+
+.. option:: --debug=<debug_info>, -d <debug_info>
+
+  Use <debug_info> to symbolicate addresses in the profile.
+
+.. option:: --regex=<regex>, -r <regex>
+
+  Only process function names that match <regex>.
+
+
+EXAMPLE
+-------
+
+.. code-block:: console
+
+  $ llvm-mipdata create -p default.mip default.mipmap
+  $ llvm-mipdata merge -p default.mip default-0000.mipraw default-0001.mipraw default-0002.mipraw
+  $ llvm-mipdata show -p default.mip --debug a.out

llvm/docs/MIPBinaryFormat.rst

+======================================
+Machine IR Profile (MIP) Binary Format
+======================================
+
+.. contents::
+  :local:
+
+Introduction
+============
+MIP, which is described in :doc:`MachineProfile <MachineProfile>`, uses three file types to hold
+profile data: ``.mip``, ``.mipmap``, ``.mipraw``. Each format has a common
+header and the data is layed out in little endian format.
+
+Header Format
+=============
+
+.. code-block:: c
+
+  Header {
+    i8[4] : Magic = {0xFB, 'M', 'I', 'P'}  // 0xFB 0x4D 0x49 0x50
+    i16 : Version = 0x8
+    i16 : FileType
+    i32 : ProfileType
+    i32 : ModuleHash                       // MD5Hash(LinkUnitName)
+    i32 : Reserved
+    i32 : OffsetToData = 0x18              // 24 bytes
+  }
+
+``FileType``
+------------
+=========== ======
+Type        Value
+----------- ------
+``.mipraw`` 0x0001
+``.mipmap`` 0x0002
+``.mip``    0x0003
+=========== ======
+
+``ProfileType``
+---------------
+============================== ==========
+Type                           Value
+------------------------------ ----------
+Function Coverage              0x00000001
+Block Coverage                 0x00000002
+Function Timestamp             0x00000004
+Function Call Count            0x00000008
+Return Address Instrumentation 0x00000010
+============================== ==========
+
+``.mipraw``
+===========
+If block coverage is not enabled then ``BlockCount`` is zero for all profiles
+and padding is not necessary.
+
+Function Coverage
+-----------------
+.. code-block:: c
+
+  Header
+  RawProfile[] {
+    i8 : IsCovered
+    i8[BlockCount] : IsCovered
+    // Pad to 8 bits
+  }
+
+Function Timestamp + Function Call Count
+----------------------------------------
+.. code-block:: c
+
+  Header
+  RawProfile[] {
+    i32 : CallCount
+    i32 : Timestamp
+    i8[BlockCount] : IsCovered
+    // Pad to 32 bits
+  }
+
+``.mipmap``
+===========
+.. code-block:: c
+
+  Header
+  FunctionProfile[] {
+    i32 : RawSectionStartPCOffset
+    i32 : RawProfilePCOffset
+    i32 : FunctionPCOffset
+    i32 : FunctionSize
+    i32 : ControlFlowGraphSignature
+    i32 : NonEntryBlockCount
+    BlockProfile[NonEntryBlockCount] {
+      i32 : Offset
+    }
+    i32 : FunctionNameLength
+    char[FunctionNameLength] : FunctionName
+    // Pad to 64 bits
+  }
+
+``.mip``
+========
+
+.. code-block:: c
+
+  Header
+  i64 : ProfileCount
+  FunctionProfile[ProfileCount] {
+    i64 : FunctionSignature  // MD5Hash(FunctionName)
+    i32 : RawProfileDataOffset
+    i32 : FunctionStartOffset
+    i32 : FunctionSize
+    i32 : ControlFlowGraphSignature
+    i32 : NonEntryBlockCount
+    i32 : MergeCount
+    i64 : CallCount
+    i64 : TimestampSum
+    BlockProfile[NonEntryBlockCount] {
+      i32 : Offset
+      i8  : IsCovered
+    }
+    i32 : CallEdgeCount
+    CallEdges[CallEdgeCount] {
+      // Reserved
+    }
+  }
+  i64 : FunctionNamesLength
+  char[FunctionNamesLength] : FunctionNames // nullbyte separated
 No newline at end of file

llvm/docs/MachineProfile.rst

+========================
+Machine IR Profile (MIP)
+========================
+
+.. contents::
+   :local:
+
+Introduction
+============
+MIP is an instrumentation framework for collecting runtime statistics and using
+the profiles for analysis or as input to the compiler for optimization. The
+metrics we collect include function call count, function call order, and a
+sample of the dynamic call graph. There is also a lightweight mode where
+function coverage and basic block coverage are collected with minimal impact
+on the performance and size of the instrumented binaries. MIP is currently
+implemented for both ELF and MachO with the following architectures:
+
+* x86_64
+* aarch64
+* armv7 (thumb1 and thumb2)
+
+MIP uses three file types (``.mip``, ``.mipmap``, ``.mipraw``) which are
+described in :doc:`MIPBinaryFormat <MIPBinaryFormat>`.
+
+Motivation
+----------
+Existing instrumentation frameworks such as :doc:`XRay <XRay>` and
+`Clang Coverage <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>`
+produce instrumented binaries that are too large or too slow to be used in
+many scenarios without sacrificing runtime behavior. Our goals for MIP are to
+
+Usage
+=====
+Building
+--------
+To build an instrumented binary, use the ``-fmachine-profile-generate`` clang
+flag.
+
+.. code-block:: console
+
+  $ clang -fmachine-profile-generate main.cpp
+
+
+Running
+-------
+Run the instrumented binary like normal to cover as many use cases as
+possible. When the program exits the default behavior is to write a raw
+profile to ``default.mipraw``. This path can be overwritten with the
+``LLVM_MIP_PROFILE_FILENAME`` environment variable.
+
+.. code-block:: console
+
+  $ LLVM_MIP_PROFILE_FILENAME=/path/to/profile.mipraw ./a.out
+
+
+Post Processing
+---------------
+After collecting one or more raw profiles (``.mipraw`` files) we can merge
+them into one profile (``.mip`` file) for analysis or optimization. First, we
+need to extract and create an empty profile from our instrumented binary.
+
+.. code-block:: console
+
+  $ llvm-objcopy --dump-section=__llvm_mipmap=default.mipmap <binary> /dev/null
+  $ llvm-mipdata create -p default.mip default.mipmap
+
+Then we can use the :doc:`llvm-mipdata <CommandGuide/llvm-mipdata>` tool to
+merge our raw profiles.
+
+.. code-block:: console
+
+  $ llvm-mipdata merge -p default.mip default.mipraw [default2.mipraw ...]
+
+The resulting ``default.mip`` file can be used as input to the
+``-fmachine-profile-use=default.mip`` frontent flag, or by the
+:doc:`llvm-mipdata <CommandGuide/llvm-mipdata>` tool for analysis.
+
+.. code-block:: console
+
+  $ llvm-mipdata show -p default.mip

llvm/docs/Reference.rst

.. toctree::
    GlobalISel/index
    GwpAsan
    HowToSetUpLLVMStyleRTTI
    HowToUseAttributes
    InAlloca
    LangRef
    LibFuzzer
    MarkedUpDisassembly
+   MachineProfile
    MIRLangRef
+   MIPBinaryFormat
    OptBisect
    PDB/index
    ScudoHardenedAllocator
    MemTagSanitizer
    Security
    SegmentedStacks
    StackMaps
    SpeculativeLoadHardening

 
 :doc:`TestSuiteGuide`
   Describes how to compile and run the test-suite benchmarks.
 
 
 :doc:`GwpAsan`
   A sampled heap memory error detection toolkit designed for production use.
 
-====
-XRay
-====
+===============
+Instrumentation
+===============
+
+:doc:`MachineProfile`
+  Describes how to use instrument binaries with MIP.
 
 :doc:`XRay`
   High-level documentation of how to use XRay in LLVM.
 
 :doc:`XRayExample`
   An example of how to debug an application with XRay.
 
 =================