diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst --- a/llvm/docs/CommandGuide/index.rst +++ b/llvm/docs/CommandGuide/index.rst @@ -29,6 +29,7 @@ llvm-link llvm-lipo llvm-mca + llvm-mipdata llvm-otool llvm-profdata llvm-readobj diff --git a/llvm/docs/CommandGuide/llvm-mipdata.rst b/llvm/docs/CommandGuide/llvm-mipdata.rst new file mode 100644 --- /dev/null +++ b/llvm/docs/CommandGuide/llvm-mipdata.rst @@ -0,0 +1,118 @@ +llvm-mipdata - MIP tool +======================= + +.. program:: llvm-mipdata + +SYNOPSIS +-------- + +:program:`llvm-mipdata` *command* [*args...*] + +DESCRIPTION +----------- + +The :program:`llvm-mipdata` tool creates, manipulates, and reports +information about :doc:`MIP <../MachineProfile>` profiles (``.mip`` files). + +General Options +--------------- + +.. option:: --help + + Print a summary of command line options. + +.. option:: --profile=, -p + + Specify the profile to work on. + +.. option:: --output=, -o + + Specify the output file name. + +.. option:: --version + + Print the current version of this tool. + + +COMMANDS +-------- + +* :ref:`create ` +* :ref:`merge ` +* :ref:`show ` + +.. _create: + +CREATE +------ + +SYNOPSIS +^^^^^^^^ + +:program:`llvm-mipdata create` [*options*] **-p ** + +DESCRIPTION +^^^^^^^^^^^ + +:program:`llvm-mipdata create` creates an empty from a +(``.mipmap`` file). + +.. _merge: + +MERGE +----- + +SYNOPSIS +^^^^^^^^ + +:program:`llvm-mipdata merge` [*options*] **-p ** + +DESCRIPTION +^^^^^^^^^^^ + +:program:`llvm-mipdata merge` merges raw profiles (``.mipraw`` files) into the +profile. + +OPTIONS +^^^^^^^ + +.. option:: --strict + + With strict mode, merging a corrupt raw profile will fail the command. + Disabled by default. + +.. _show: + +SHOW +---- + +SYNOPSIS +^^^^^^^^ + +:program:`llvm-mipdata show` [*options*] **-p ** + +DESCRIPTION +^^^^^^^^^^^ + +:program:`llvm-mipdata show` prints the data from the given profile. + +OPTIONS +^^^^^^^ + +.. option:: --debug=, -d + + Use to symbolicate addresses in the profile. + +.. option:: --regex=, -r + + Only process function names that match . + + +EXAMPLE +------- + +.. code-block:: console + + $ llvm-mipdata create -p default.mip default.mipmap + $ llvm-mipdata merge -p default.mip default-0000.mipraw default-0001.mipraw default-0002.mipraw + $ llvm-mipdata show -p default.mip --debug a.out diff --git a/llvm/docs/MIPBinaryFormat.rst b/llvm/docs/MIPBinaryFormat.rst new file mode 100644 --- /dev/null +++ b/llvm/docs/MIPBinaryFormat.rst @@ -0,0 +1,126 @@ +====================================== +Machine IR Profile (MIP) Binary Format +====================================== + +.. contents:: + :local: + +Introduction +============ +MIP, which is described in :doc:`MachineProfile `, uses three file types to hold +profile data: ``.mip``, ``.mipmap``, ``.mipraw``. Each format has a common +header and the data is layed out in little endian format. + +Header Format +============= + +.. code-block:: c + + Header { + i8[4] : Magic = {0xFB, 'M', 'I', 'P'} // 0xFB 0x4D 0x49 0x50 + i16 : Version = 0x8 + i16 : FileType + i32 : ProfileType + i32 : ModuleHash // MD5Hash(LinkUnitName) + i32 : Reserved + i32 : OffsetToData = 0x18 // 24 bytes + } + +``FileType`` +------------ +=========== ====== +Type Value +----------- ------ +``.mipraw`` 0x0001 +``.mipmap`` 0x0002 +``.mip`` 0x0003 +=========== ====== + +``ProfileType`` +--------------- +============================== ========== +Type Value +------------------------------ ---------- +Function Coverage 0x00000001 +Block Coverage 0x00000002 +Function Timestamp 0x00000004 +Function Call Count 0x00000008 +Return Address Instrumentation 0x00000010 +============================== ========== + +``.mipraw`` +=========== +If block coverage is not enabled then ``BlockCount`` is zero for all profiles +and padding is not necessary. + +Function Coverage +----------------- +.. code-block:: c + + Header + RawProfile[] { + i8 : IsCovered + i8[BlockCount] : IsCovered + // Pad to 8 bits + } + +Function Timestamp + Function Call Count +---------------------------------------- +.. code-block:: c + + Header + RawProfile[] { + i32 : CallCount + i32 : Timestamp + i8[BlockCount] : IsCovered + // Pad to 32 bits + } + +``.mipmap`` +=========== +.. code-block:: c + + Header + FunctionProfile[] { + i32 : RawSectionStartPCOffset + i32 : RawProfilePCOffset + i32 : FunctionPCOffset + i32 : FunctionSize + i32 : ControlFlowGraphSignature + i32 : NonEntryBlockCount + BlockProfile[NonEntryBlockCount] { + i32 : Offset + } + i32 : FunctionNameLength + char[FunctionNameLength] : FunctionName + // Pad to 64 bits + } + +``.mip`` +======== + +.. code-block:: c + + Header + i64 : ProfileCount + FunctionProfile[ProfileCount] { + i64 : FunctionSignature // MD5Hash(FunctionName) + i32 : RawProfileDataOffset + i32 : FunctionStartOffset + i32 : FunctionSize + i32 : ControlFlowGraphSignature + i32 : NonEntryBlockCount + i32 : MergeCount + i64 : CallCount + i64 : TimestampSum + BlockProfile[NonEntryBlockCount] { + i32 : Offset + i8 : IsCovered + } + i32 : CallEdgeCount + CallEdges[CallEdgeCount] { + // Reserved + } + } + i64 : FunctionNamesLength + char[FunctionNamesLength] : FunctionNames // nullbyte separated \ No newline at end of file diff --git a/llvm/docs/MachineProfile.rst b/llvm/docs/MachineProfile.rst new file mode 100644 --- /dev/null +++ b/llvm/docs/MachineProfile.rst @@ -0,0 +1,80 @@ +======================== +Machine IR Profile (MIP) +======================== + +.. contents:: + :local: + +Introduction +============ +MIP is an instrumentation framework for collecting runtime statistics and using +the profiles for analysis or as input to the compiler for optimization. The +metrics we collect include function call count, function call order, and a +sample of the dynamic call graph. There is also a lightweight mode where +function coverage and basic block coverage are collected with minimal impact +on the performance and size of the instrumented binaries. MIP is currently +implemented for both ELF and MachO with the following architectures: + +* x86_64 +* aarch64 +* armv7 (thumb1 and thumb2) + +MIP uses three file types (``.mip``, ``.mipmap``, ``.mipraw``) which are +described in :doc:`MIPBinaryFormat `. + +Motivation +---------- +Existing instrumentation frameworks such as :doc:`XRay ` and +`Clang Coverage ` +produce instrumented binaries that are too large or too slow to be used in +many scenarios without sacrificing runtime behavior. Our goals for MIP are to + +Usage +===== +Building +-------- +To build an instrumented binary, use the ``-fmachine-profile-generate`` clang +flag. + +.. code-block:: console + + $ clang -fmachine-profile-generate main.cpp + + +Running +------- +Run the instrumented binary like normal to cover as many use cases as +possible. When the program exits the default behavior is to write a raw +profile to ``default.mipraw``. This path can be overwritten with the +``LLVM_MIP_PROFILE_FILENAME`` environment variable. + +.. code-block:: console + + $ LLVM_MIP_PROFILE_FILENAME=/path/to/profile.mipraw ./a.out + + +Post Processing +--------------- +After collecting one or more raw profiles (``.mipraw`` files) we can merge +them into one profile (``.mip`` file) for analysis or optimization. First, we +need to extract and create an empty profile from our instrumented binary. + +.. code-block:: console + + $ llvm-objcopy --dump-section=__llvm_mipmap=default.mipmap /dev/null + $ llvm-mipdata create -p default.mip default.mipmap + +Then we can use the :doc:`llvm-mipdata ` tool to +merge our raw profiles. + +.. code-block:: console + + $ llvm-mipdata merge -p default.mip default.mipraw [default2.mipraw ...] + +The resulting ``default.mip`` file can be used as input to the +``-fmachine-profile-use=default.mip`` frontent flag, or by the +:doc:`llvm-mipdata ` tool for analysis. + +.. code-block:: console + + $ llvm-mipdata show -p default.mip diff --git a/llvm/docs/Reference.rst b/llvm/docs/Reference.rst --- a/llvm/docs/Reference.rst +++ b/llvm/docs/Reference.rst @@ -31,7 +31,9 @@ LangRef LibFuzzer MarkedUpDisassembly + MachineProfile MIRLangRef + MIPBinaryFormat OptBisect PDB/index ScudoHardenedAllocator @@ -139,9 +141,12 @@ :doc:`GwpAsan` A sampled heap memory error detection toolkit designed for production use. -==== -XRay -==== +=============== +Instrumentation +=============== + +:doc:`MachineProfile` + Describes how to use instrument binaries with MIP. :doc:`XRay` High-level documentation of how to use XRay in LLVM.