[Remarks] Add an LLVM-bitstream-based remark serializer

[thegameg]

Summary

Add a new serializer, using a binary format based on the LLVM bitstream format.

This format provides a way to serialize the remarks in two modes:

1. Separate mode: the metadata is separate from the remark entries.
2. Standalone mode: the metadata and the remark entries are in the same file.

The format contains:

• a meta block: container version, container type, string table, external file path, remark version
• a remark block: type, remark name, pass name, function name, debug file, debug line, debug column, hotness, arguments (key, value, debug file, debug line, debug column)

A string table is required for this format, which will be dumped in the meta block to be consumed before parsing the remark blocks.

On clang itself, we noticed a size reduction of 13.4x compared to YAML, and a compile-time reduction of between 1.7% and 3.5% on CTMark.

[paquette]

llvm/include/llvm/Remarks/BitstreamRemarkContainer.h

32–34 ↗

(On Diff #205228)

A couple things:

1. Is remark metadata explicitly documented anywhere? The comment in BitstreamRemarkSerializer.h seems to be about half way there. Maybe it would be good to expand a bit upon that, and then reference that file here? (I think it might be worth adding to the LLVM docs on remarks eventually.)

2. The "separate" point only talks about SeparateRemarksFile?

52 ↗

(On Diff #205228)

extra comma

55 ↗

(On Diff #205228)

Can we have a documentation comment on this?

63 ↗

(On Diff #205228)

A documentation comment would be useful here as well.

llvm/include/llvm/Remarks/BitstreamRemarkSerializer.h

62–63 ↗

(On Diff #205228)

Is it possible to verify this with a test?

E.g. add a debug dumper for the available abbrev IDs and do something similar to llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir?

76 ↗

(On Diff #205228)

Grammar nit: Setup is a noun. I would use set up here. English is weird.

81 ↗

(On Diff #205228)

This, and the other ones should probably say "set up" like setupMetaBlockInfo

llvm/lib/Remarks/BitstreamRemarkSerializer.cpp

23–26 ↗

(On Diff #205228)

Whenever this function is called, there's a call to Bitstream.EmitRecord following it. Would it make sense to pull that into the function and give it a more descriptive name?

30–32 ↗

(On Diff #205228)

This is a pretty common pattern as well. Could this be pulled out into a function?

[thegameg]

llvm/include/llvm/Remarks/BitstreamRemarkContainer.h

32–34 ↗

(On Diff #205228)

1. Yes, I documented it right above the SeparateRemarksMeta enum entry. I agree that I should add something to the LLVM docs.
2. Actually, it only talks about SeparateRemarksMeta. I will make this more explicit, but the idea for the separate model is:
   • the meta part will eventually go into a section. It will contain one BLOCK_META with some records.
   • the remark part will go into a separate file (with its path saved in the meta part). It will also contain one BLOCK_META (for more versioning) and multiple BLOCK_REMARK blocks.

llvm/include/llvm/Remarks/BitstreamRemarkSerializer.h

62–63 ↗

(On Diff #205228)

Nothing obvious comes to mind right now for iterating on the available abbrev IDs without clobbering the code with debug stuff. Technically, the BitstreamWriter will assert if the abbrev id's width is bigger than what specified when entering the block.

[thegameg]

• Address Jessica's comments.
• Rename BLOCK_* to *_BLOCK_ID.
• Teach llvm-bcanalyzer to recognize the magic number.
• Add a unit test to check for the values that shouldn't change without a version change.

[paquette]

llvm/lib/Remarks/BitstreamRemarkSerializer.cpp

342 ↗

(On Diff #206982)

This should only run when DidSetUp == true, correct?

Would it make sense to move DidSetUp into the Helper? Then you could set it in setupBlockInfo and either assert or throw an error if DidSetUp == false in appropriate Helper functions.

llvm/test/CodeGen/X86/remarks-bitstream.ll

11 ↗

(On Diff #206982)

Are these outputs guaranteed to be stable? Judging by the opt runline, it looks like this might rely on inlining behaviour + remark formatting?

llvm/tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

975 ↗

(On Diff #206982)

"Bitstream" seems a little redundant here. Just "LLVM Remarks" instead?

[thegameg]

llvm/lib/Remarks/BitstreamRemarkSerializer.cpp

342 ↗

(On Diff #206982)

Correct.

I wouldn't want to move DidSetUp into the Helper because I intend to use the Helper in other places, like the generation of the remark section where emitRemarkBlock won't be called.

Maybe I should just assert(DidSetUp == true) to avoid code getting in between the initialization and the emission of the remark?

llvm/test/CodeGen/X86/remarks-bitstream.ll

11 ↗

(On Diff #206982)

You're right, they rely on inlining behavior and on the emission of the remarks from the passes. Ideally we would have unit tests for this, but I'm afraid at that point we'll have to check for byte-by-byte correctness, while here we can sort of have a textual check.

Maybe we should move the whole analysis behavior from llvm-bcanalyzer to lib/Bitcode? That way we can have unit tests that come from actual Remark objects and can at least compare strings instead of raw bytes.

Let me know if you have any ideas for testing this differently.

[thegameg]

This adds a lot more testing.

1. BitstreamRemarksFormatTest.cpp: test for the stability of the versions and IDs.
2. BitstreamRemarksSerializerTest.cpp: don't depend on actual codegen to generate remarks. Manually create remark objects that get serialized then analyzed by the bitcode analyzer. This tests the multiple remark container types for the metadata blocks.
3. Gets rid of the codegen test: llvm/test/CodeGen/X86/remarks-bitstream.ll.

[thegameg]

Add documentation.

[paquette]

LGTM!

llvm/trunk/docs/Remarks.rst

``Basic options``
     .. option:: -pass-remarks-format=<format>
 
       Specifies the output format of the serialized remarks.
 
       Supported formats:
 
       * :ref:`yaml <yamlremarks>` (default)
       * :ref:`yaml-strtab <yamlstrtabremarks>`
+      * :ref:`bitstream <bitstreamremarks>`
 
 ``Content configuration``
 
     .. option:: -pass-remarks-filter=<regex>
 
       Only passes whose name match the given (POSIX) regular expression will be
       serialized to the final output.
 
* the absolute file path to the serialized remark diagnostics: a
   null-terminated string.
 
 When the metadata is serialized separately from the remarks, the file path
 should be present and point to the file where the remarks are serialized to.
 
 In case the metadata only acts as a header to the remarks, the file path can be
 omitted.
 
+.. _bitstreamremarks:
+
+LLVM bitstream remarks
+======================
+
+This format is using :doc:`LLVM bitstream <BitCodeFormat>` to serialize remarks
+and their associated metadata.
+
+A bitstream remark stream can be identified by the magic number ``"RMRK"`` that
+is placed at the very beginning.
+
+The format for serializing remarks is composed of two different block types:
+
+.. _bitstreamremarksmetablock:
+
+META_BLOCK
+----------
+
+The block providing information about the rest of the content in the stream.
+
+Exactly one block is expected. Having multiple metadata blocks is an error.
+
+This block can contain the following records:
+
+.. _bitstreamremarksrecordmetacontainerinfo:
+
+``RECORD_META_CONTAINER_INFO``
+
+    The container version and type.
+
+    Version: u32
+
+    Type:    u2
+
+.. _bitstreamremarksrecordmetaremarkversion:
+
+``RECORD_META_REMARK_VERSION``
+
+    The version of the remark entries. This can change independently from the
+    container version.
+
+    Version: u32
+
+.. _bitstreamremarksrecordmetastrtab:
+
+``RECORD_META_STRTAB``
+
+    The string table used by the remark entries. The format of the string table
+    is a sequence of strings separated by ``\0``.
+
+.. _bitstreamremarksrecordmetaexternalfile:
+
+``RECORD_META_EXTERNAL_FILE``
+
+    The external remark file path that contains the remark blocks associated
+    with this metadata. This is an absolute path.
+
+.. _bitstreamremarksremarkblock:
+
+REMARK_BLOCK
+------------
+
+The block describing a remark entry.
+
+0 or more blocks per file are allowed. Each block will depend on the
+:ref:`META_BLOCK <bitstreamremarksmetablock>` in order to be parsed correctly.
+
+This block can contain the following records:
+
+``RECORD_REMARK_HEADER``
+
+    The header of the remark. This contains all the mandatory information about
+    a remark.
+
+    +---------------+---------------------------+
+    | Type          | u3                        |
+    +---------------+---------------------------+
+    | Remark name   | VBR6 (string table index) |
+    +---------------+---------------------------+
+    | Pass name     | VBR6 (string table index) |
+    +---------------+---------------------------+
+    | Function name | VBR6 (string table index) |
+    +---------------+---------------------------+
+
+``RECORD_REMARK_DEBUG_LOC``
+
+    The source location for the corresponding remark. This record is optional.
+
+    +--------+---------------------------+
+    | File   | VBR7 (string table index) |
+    +--------+---------------------------+
+    | Line   | u32                       |
+    +--------+---------------------------+
+    | Column | u32                       |
+    +--------+---------------------------+
+
+``RECORD_REMARK_HOTNESS``
+
+    The hotness of the remark. This record is optional.
+
+    +---------------+---------------------+
+    | Hotness | VBR8 (string table index) |
+    +---------------+---------------------+
+
+``RECORD_REMARK_ARG_WITH_DEBUGLOC``
+
+    A remark argument with an associated debug location.
+
+    +--------+---------------------------+
+    | Key    | VBR7 (string table index) |
+    +--------+---------------------------+
+    | Value  | VBR7 (string table index) |
+    +--------+---------------------------+
+    | File   | VBR7 (string table index) |
+    +--------+---------------------------+
+    | Line   | u32                       |
+    +--------+---------------------------+
+    | Column | u32                       |
+    +--------+---------------------------+
+
+``RECORD_REMARK_ARG_WITHOUT_DEBUGLOC``
+
+    A remark argument with an associated debug location.
+
+    +--------+---------------------------+
+    | Key    | VBR7 (string table index) |
+    +--------+---------------------------+
+    | Value  | VBR7 (string table index) |
+    +--------+---------------------------+
+
+The remark container
+--------------------
+
+Bitstream remarks are designed to be used in two different modes:
+
+``The separate mode``
+
+    The separate mode is the mode that is typically used during compilation. It
+    provides a way to serialize the remark entries to a stream while some
+    metadata is kept in memory to be emitted in the product of the compilation
+    (typically, an object file).
+
+``The standalone mode``
+
+    The standalone mode is typically stored and used after the distribution of
+    a program. It contains all the information that allows the parsing of all
+    the remarks without having any external dependencies.
+
+In order to support multiple modes, the format introduces the concept of a
+bitstream remark container type.
+
+.. _bitstreamremarksseparateremarksmeta:
+
+``SeparateRemarksMeta: the metadata emitted separately``
+
+    This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
+
+    * :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
+    * :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
+    * :ref:`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
+
+    Typically, this is emitted in a section in the object files, allowing
+    clients to retrieve remarks and their associated metadata directly from
+    intermediate products.
+
+``SeparateRemarksFile: the remark entries emitted separately``
+
+    This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
+
+    * :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
+    * :ref:`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
+
+    This container type expects 0 or more :ref:`REMARK_BLOCK <bitstreamremarksremarkblock>`.
+
+    Typically, this is emitted in a side-file alongside an object file, and is
+    made to be able to stream to without increasing the memory consumption of
+    the compiler. This is referenced by the :ref:`RECORD_META_EXTERNAL_FILE
+    <bitstreamremarksrecordmetaexternalfile>` entry in the
+    :ref:`SeparateRemarksMeta <bitstreamremarksseparateremarksmeta>` container.
+
+When the parser tries to parse a container that contains the metadata for the
+separate remarks, it should parse the version and type, then keep the string
+table in memory while opening the external file, validating its metadata and
+parsing the remark entries.
+
+The container versions from the separate container should match in order to
+have a well-formed file.
+
+``Standalone: the metadata and the remark entries emitted together``
+
+    This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
+
+    * :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
+    * :ref:`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
+    * :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
+
+    This container type expects 0 or more :ref:`REMARK_BLOCK <bitstreamremarksremarkblock>`.
+
+A complete output of :program:`llvm-bcanalyzer` on the different container types:
+
+``SeparateRemarksMeta``
+
+.. code-block:: none
+
+    <BLOCKINFO_BLOCK/>
+    <Meta BlockID=8 NumWords=13 BlockCodeSize=3>
+      <Container info codeid=1 abbrevid=4 op0=5 op1=0/>
+      <String table codeid=3 abbrevid=5/> blob data = 'pass\\x00key\\x00value\\x00'
+      <External File codeid=4 abbrevid=6/> blob data = '/path/to/file/name'
+    </Meta>
+
+``SeparateRemarksFile``
+
+.. code-block:: none
+
+    <BLOCKINFO_BLOCK/>
+    <Meta BlockID=8 NumWords=3 BlockCodeSize=3>
+      <Container info codeid=1 abbrevid=4 op0=0 op1=1/>
+      <Remark version codeid=2 abbrevid=5 op0=0/>
+    </Meta>
+    <Remark BlockID=9 NumWords=8 BlockCodeSize=4>
+      <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>
+      <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>
+      <Remark hotness codeid=7 abbrevid=6 op0=999999999/>
+      <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 op3=11 op4=66/>
+    </Remark>
+
+``Standalone``
+
+.. code-block:: none
+
+    <BLOCKINFO_BLOCK/>
+    <Meta BlockID=8 NumWords=15 BlockCodeSize=3>
+      <Container info codeid=1 abbrevid=4 op0=5 op1=2/>
+      <Remark version codeid=2 abbrevid=5 op0=30/>
+      <String table codeid=3 abbrevid=6/> blob data = 'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x00'
+    </Meta>
+    <Remark BlockID=9 NumWords=8 BlockCodeSize=4>
+      <Remark header codeid=5 abbrevid=4 op0=2 op1=1 op2=0 op3=2/>
+      <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>
+      <Remark hotness codeid=7 abbrevid=6 op0=999999999/>
+      <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 op3=11 op4=66/>
+    </Remark>
+
 opt-viewer
 ==========
 
 The ``opt-viewer`` directory contains a collection of tools that visualize and
 summarize serialized remarks.
 
 The tools only support the ``yaml`` format.
 

llvm/trunk/include/llvm/Bitcode/BitcodeAnalyzer.h

 namespace llvm {
 
 /// CurStreamTypeType - A type for CurStreamType
 enum CurStreamTypeType {
   UnknownBitstream,
   LLVMIRBitstream,
   ClangSerializedASTBitstream,
   ClangSerializedDiagnosticsBitstream,
+  LLVMBitstreamRemarks
 };
 
 struct BCDumpOptions {
   /// The stream.
   raw_ostream &OS;
   /// Print per-code histogram.
   bool Histogram = false;
   /// Don't emit numeric info in dump if symbolic info is available.

llvm/trunk/include/llvm/Remarks/BitstreamRemarkContainer.h

+//===-- BitstreamRemarkContainer.h - Container for remarks --------------*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides declarations for things used in the various types of
+// remark containers.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_REMARKS_REMARK_CONTAINER_H
+#define LLVM_REMARKS_REMARK_CONTAINER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitCodes.h"
+#include <cstdint>
+
+namespace llvm {
+namespace remarks {
+
+/// The current version of the remark container.
+/// Note: this is different from the version of the remark entry.
+constexpr uint64_t CurrentContainerVersion = 0;
+/// The magic number used for identifying remark blocks.
+constexpr StringRef ContainerMagic("RMRK", 4);
+
+/// Type of the remark container.
+/// The remark container has two modes:
+/// * separate: the metadata is separate from the remarks and points to the
+///             auxiliary file that contains the remarks.
+/// * standalone: the metadata and the remarks are emitted together.
+enum class BitstreamRemarkContainerType {
+  /// The metadata emitted separately.
+  /// This will contain the following:
+  /// * Container version and type
+  /// * String table
+  /// * External file
+  SeparateRemarksMeta,
+  /// The remarks emitted separately.
+  /// This will contain the following:
+  /// * Container version and type
+  /// * Remark version
+  SeparateRemarksFile,
+  /// Everything is emitted together.
+  /// This will contain the following:
+  /// * Container version and type
+  /// * Remark version
+  /// * String table
+  Standalone,
+  First = SeparateRemarksMeta,
+  Last = Standalone,
+};
+
+/// The possible blocks that will be encountered in a bitstream remark
+/// container.
+enum BlockIDs {
+  /// The metadata block is mandatory. It should always come after the
+  /// BLOCKINFO_BLOCK, and contains metadata that should be used when parsing
+  /// REMARK_BLOCKs.
+  /// There should always be only one META_BLOCK.
+  META_BLOCK_ID = bitc::FIRST_APPLICATION_BLOCKID,
+  /// One remark entry is represented using a REMARK_BLOCK. There can be
+  /// multiple REMARK_BLOCKs in the same file.
+  REMARK_BLOCK_ID
+};
+
+constexpr StringRef MetaBlockName = StringRef("Meta", 4);
+constexpr StringRef RemarkBlockName = StringRef("Remark", 6);
+
+/// The possible records that can be encountered in the previously described
+/// blocks.
+enum RecordIDs {
+  // Meta block records.
+  RECORD_META_CONTAINER_INFO = 1,
+  RECORD_META_REMARK_VERSION,
+  RECORD_META_STRTAB,
+  RECORD_META_EXTERNAL_FILE,
+  // Remark block records.
+  RECORD_REMARK_HEADER,
+  RECORD_REMARK_DEBUG_LOC,
+  RECORD_REMARK_HOTNESS,
+  RECORD_REMARK_ARG_WITH_DEBUGLOC,
+  RECORD_REMARK_ARG_WITHOUT_DEBUGLOC,
+  // Helpers.
+  RECORD_FIRST = RECORD_META_CONTAINER_INFO,
+  RECORD_LAST = RECORD_REMARK_ARG_WITHOUT_DEBUGLOC
+};
+
+constexpr StringRef MetaContainerInfoName = StringRef("Container info", 14);
+constexpr StringRef MetaRemarkVersionName = StringRef("Remark version", 14);
+constexpr StringRef MetaStrTabName = StringRef("String table", 12);
+constexpr StringRef MetaExternalFileName = StringRef("External File", 13);
+constexpr StringRef RemarkHeaderName = StringRef("Remark header", 13);
+constexpr StringRef RemarkDebugLocName = StringRef("Remark debug location", 21);
+constexpr StringRef RemarkHotnessName = StringRef("Remark hotness", 14);
+constexpr StringRef RemarkArgWithDebugLocName =
+    StringRef("Argument with debug location", 28);
+constexpr StringRef RemarkArgWithoutDebugLocName = StringRef("Argument", 8);
+
+} // end namespace remarks
+} // end namespace llvm
+
+#endif /* LLVM_REMARKS_REMARK_CONTAINER_H */

llvm/trunk/include/llvm/Remarks/BitstreamRemarkSerializer.h

+//===-- BitstreamRemarkSerializer.h - Bitstream serializer ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides an implementation of the serializer using the LLVM
+// Bitstream format.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H
+#define LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H
+
+#include "llvm/Bitstream/BitstreamWriter.h"
+#include "llvm/Remarks/BitstreamRemarkContainer.h"
+#include "llvm/Remarks/RemarkSerializer.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm {
+namespace remarks {
+
+/// Serialize the remarks to LLVM bitstream.
+/// This class provides ways to emit remarks in the LLVM bitstream format and
+/// its associated metadata.
+///
+/// * The separate model:
+///   Separate meta:        | Container info
+///                         | String table
+///                         | External file
+///
+///   Separate remarks:     | Container info
+///                         | Remark version
+///                         | Remark0
+///                         | Remark1
+///                         | Remark2
+///                         | ...
+///
+/// * The standalone model: | Container info
+///                         | String table
+///                         | Remark version
+///                         | Remark0
+///                         | Remark1
+///                         | Remark2
+///                         | ...
+///
+struct BitstreamSerializerHelper {
+  /// Buffer used for encoding the bitstream before writing it to the final
+  /// stream.
+  SmallVector<char, 1024> Encoded;
+  /// Buffer used to construct records and pass to the bitstream writer.
+  SmallVector<uint64_t, 64> R;
+  /// The Bitstream writer.
+  BitstreamWriter Bitstream;
+  /// The type of the container we are serializing.
+  BitstreamRemarkContainerType ContainerType;
+
+  /// Abbrev IDs initialized in the block info block.
+  /// Note: depending on the container type, some IDs might be uninitialized.
+  /// Warning: When adding more abbrev IDs, make sure to update the
+  /// BlockCodeSize (in the call to EnterSubblock).
+  uint64_t RecordMetaContainerInfoAbbrevID = 0;
+  uint64_t RecordMetaRemarkVersionAbbrevID = 0;
+  uint64_t RecordMetaStrTabAbbrevID = 0;
+  uint64_t RecordMetaExternalFileAbbrevID = 0;
+  uint64_t RecordRemarkHeaderAbbrevID = 0;
+  uint64_t RecordRemarkDebugLocAbbrevID = 0;
+  uint64_t RecordRemarkHotnessAbbrevID = 0;
+  uint64_t RecordRemarkArgWithDebugLocAbbrevID = 0;
+  uint64_t RecordRemarkArgWithoutDebugLocAbbrevID = 0;
+
+  BitstreamSerializerHelper(BitstreamRemarkContainerType ContainerType);
+
+  // Disable copy and move: Bitstream points to Encoded, which needs special
+  // handling during copy/move, but moving the vectors is probably useless
+  // anyway.
+  BitstreamSerializerHelper(const BitstreamSerializerHelper &) = delete;
+  BitstreamSerializerHelper &
+  operator=(const BitstreamSerializerHelper &) = delete;
+  BitstreamSerializerHelper(BitstreamSerializerHelper &&) = delete;
+  BitstreamSerializerHelper &operator=(BitstreamSerializerHelper &&) = delete;
+
+  /// Set up the necessary block info entries according to the container type.
+  void setupBlockInfo();
+
+  /// Set up the block info for the metadata block.
+  void setupMetaBlockInfo();
+  /// The remark version in the metadata block.
+  void setupMetaRemarkVersion();
+  void emitMetaRemarkVersion(uint64_t RemarkVersion);
+  /// The strtab in the metadata block.
+  void setupMetaStrTab();
+  void emitMetaStrTab(const StringTable &StrTab);
+  /// The external file in the metadata block.
+  void setupMetaExternalFile();
+  void emitMetaExternalFile(StringRef Filename);
+
+  /// The block info for the remarks block.
+  void setupRemarkBlockInfo();
+
+  /// Emit the metadata for the remarks.
+  void emitMetaBlock(uint64_t ContainerVersion,
+                     Optional<uint64_t> RemarkVersion,
+                     Optional<const StringTable *> StrTab = None,
+                     Optional<StringRef> Filename = None);
+
+  /// Emit a remark block. The string table is required.
+  void emitRemarkBlock(const Remark &Remark, StringTable &StrTab);
+  /// Finalize the writing to \p OS.
+  void flushToStream(raw_ostream &OS);
+  /// Finalize the writing to a buffer.
+  /// The contents of the buffer remain valid for the lifetime of the object.
+  /// Any call to any other function in this class will invalidate the buffer.
+  StringRef getBuffer();
+};
+
+/// Implementation of the remark serializer using LLVM bitstream.
+struct BitstreamSerializer : public RemarkSerializer {
+  /// The file should contain:
+  /// 1) The block info block that describes how to read the blocks.
+  /// 2) The metadata block that contains various information about the remarks
+  ///    in the file.
+  /// 3) A number of remark blocks.
+
+  /// We need to set up 1) and 2) first, so that we can emit 3) after. This flag
+  /// is used to emit the first two blocks only once.
+  bool DidSetUp = false;
+  /// The helper to emit bitstream.
+  BitstreamSerializerHelper Helper;
+
+  /// Construct a serializer that will create its own string table.
+  BitstreamSerializer(raw_ostream &OS, SerializerMode Mode);
+  /// Construct a serializer with a pre-filled string table.
+  BitstreamSerializer(raw_ostream &OS, SerializerMode Mode, StringTable StrTab);
+
+  /// Emit a remark to the stream. This also emits the metadata associated to
+  /// the remarks based on the SerializerMode specified at construction.
+  /// This writes the serialized output to the provided stream.
+  void emit(const Remark &Remark) override;
+  /// The metadata serializer associated to this remark serializer. Based on the
+  /// container type of the current serializer, the container type of the
+  /// metadata serializer will change.
+  std::unique_ptr<MetaSerializer>
+  metaSerializer(raw_ostream &OS,
+                 Optional<StringRef> ExternalFilename = None) override;
+};
+
+/// Serializer of metadata for bitstream remarks.
+struct BitstreamMetaSerializer : public MetaSerializer {
+  /// This class can be used with [1] a pre-constructed
+  /// BitstreamSerializerHelper, or with [2] one that is owned by the meta
+  /// serializer. In case of [1], we need to be able to store a reference to the
+  /// object, while in case of [2] we need to store the whole object.
+  Optional<BitstreamSerializerHelper> TmpHelper;
+  /// The actual helper, that can point to \p TmpHelper or to an external helper
+  /// object.
+  BitstreamSerializerHelper *Helper = nullptr;
+
+  Optional<const StringTable *> StrTab;
+  Optional<StringRef> ExternalFilename;
+
+  /// Create a new meta serializer based on \p ContainerType.
+  BitstreamMetaSerializer(raw_ostream &OS,
+                          BitstreamRemarkContainerType ContainerType,
+                          Optional<const StringTable *> StrTab = None,
+                          Optional<StringRef> ExternalFilename = None)
+      : MetaSerializer(OS), TmpHelper(None), Helper(nullptr), StrTab(StrTab),
+        ExternalFilename(ExternalFilename) {
+    TmpHelper.emplace(ContainerType);
+    Helper = &*TmpHelper;
+  }
+
+  /// Create a new meta serializer based on a previously built \p Helper.
+  BitstreamMetaSerializer(raw_ostream &OS, BitstreamSerializerHelper &Helper,
+                          Optional<const StringTable *> StrTab = None,
+                          Optional<StringRef> ExternalFilename = None)
+      : MetaSerializer(OS), TmpHelper(None), Helper(&Helper), StrTab(StrTab),
+        ExternalFilename(ExternalFilename) {}
+
+  void emit() override;
+};
+
+} // end namespace remarks
+} // end namespace llvm
+
+#endif /* LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H */

llvm/trunk/include/llvm/Remarks/Remark.h

 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CBindingWrapping.h"
 #include <string>
 
 namespace llvm {
 namespace remarks {
 
-constexpr uint64_t Version = 0;
+/// The current version of the remark entry.
+constexpr uint64_t CurrentRemarkVersion = 0;
 
 /// The debug location used to track a remark back to the source file.
 struct RemarkLocation {
   /// Absolute path of the source file corresponding to this remark.
   StringRef SourceFilePath;
   unsigned SourceLine;
   unsigned SourceColumn;
 };

 enum class Type {
   Unknown,
   Passed,
   Missed,
   Analysis,
   AnalysisFPCommute,
   AnalysisAliasing,
   Failure,
-  LastTypeValue = Failure
+  First = Unknown,
+  Last = Failure
 };
 
 /// A remark type used for both emission and parsing.
 struct Remark {
   /// The type of the remark.
   Type RemarkType = Type::Unknown;
 
   /// Name of the pass that triggers the emission of this remark.

llvm/trunk/include/llvm/Remarks/RemarkFormat.h

 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace remarks {
 
 constexpr StringRef Magic("REMARKS", 7);
 
 /// The format used for serializing/deserializing remarks.
-enum class Format { Unknown, YAML, YAMLStrTab };
+enum class Format { Unknown, YAML, YAMLStrTab, Bitstream };
 
 /// Parse and validate a string for the remark format.
 Expected<Format> parseFormat(StringRef FormatStr);
 
 } // end namespace remarks
 } // end namespace llvm
 
 #endif /* LLVM_REMARKS_REMARK_FORMAT_H */

llvm/trunk/lib/Bitcode/Reader/BitcodeAnalyzer.cpp

if (Signature[2] == 'C' && Signature[3] == 'H')
       return ClangSerializedASTBitstream;
   } else if (Signature[0] == 'D' && Signature[1] == 'I') {
     if (Error Err = tryRead(Signature[2], 8))
       return std::move(Err);
     if (Error Err = tryRead(Signature[3], 8))
       return std::move(Err);
     if (Signature[2] == 'A' && Signature[3] == 'G')
       return ClangSerializedDiagnosticsBitstream;
+  } else if (Signature[0] == 'R' && Signature[1] == 'M') {
+    if (Error Err = tryRead(Signature[2], 8))
+      return std::move(Err);
+    if (Error Err = tryRead(Signature[3], 8))
+      return std::move(Err);
+    if (Signature[2] == 'R' && Signature[3] == 'K')
+      return LLVMBitstreamRemarks;
   } else {
     if (Error Err = tryRead(Signature[2], 4))
       return std::move(Err);
     if (Error Err = tryRead(Signature[3], 4))
       return std::move(Err);
     if (Error Err = tryRead(Signature[4], 4))
       return std::move(Err);
     if (Error Err = tryRead(Signature[5], 4))
case LLVMIRBitstream:
     O.OS << "LLVM IR\n";
     break;
   case ClangSerializedASTBitstream:
     O.OS << "Clang Serialized AST\n";
     break;
   case ClangSerializedDiagnosticsBitstream:
     O.OS << "Clang Serialized Diagnostics\n";
     break;
+  case LLVMBitstreamRemarks:
+    O.OS << "LLVM Remarks\n";
+    break;
   }
   O.OS << "  # Toplevel Blocks: " << NumTopBlocks << "\n";
   O.OS << "\n";
 
   // Emit per-block stats.
   O.OS << "Per-block Summary:\n";
   for (std::map<unsigned, PerBlockIDStats>::iterator I = BlockIDStats.begin(),
                                                      E = BlockIDStats.end();

llvm/trunk/lib/IR/RemarkStreamer.cpp

 // LLVMContext.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/IR/RemarkStreamer.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalValue.h"
+#include "llvm/Remarks/BitstreamRemarkSerializer.h"
 #include "llvm/Remarks/RemarkFormat.h"
 #include "llvm/Remarks/RemarkSerializer.h"
 
 using namespace llvm;
 
 RemarkStreamer::RemarkStreamer(
     StringRef Filename,
     std::unique_ptr<remarks::RemarkSerializer> RemarkSerializer)

llvm/trunk/lib/Remarks/BitstreamRemarkSerializer.cpp

+//===- BitstreamRemarkSerializer.cpp --------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides the implementation of the LLVM bitstream remark serializer
+// using LLVM's bitstream writer.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Remarks/BitstreamRemarkSerializer.h"
+
+using namespace llvm;
+using namespace llvm::remarks;
+
+BitstreamSerializerHelper::BitstreamSerializerHelper(
+    BitstreamRemarkContainerType ContainerType)
+    : Encoded(), R(), Bitstream(Encoded), ContainerType(ContainerType) {}
+
+static void push(SmallVectorImpl<uint64_t> &R, StringRef Str) {
+  for (const char C : Str)
+    R.push_back(C);
+}
+
+static void setRecordName(unsigned RecordID, BitstreamWriter &Bitstream,
+                          SmallVectorImpl<uint64_t> &R, StringRef Str) {
+  R.clear();
+  R.push_back(RecordID);
+  push(R, Str);
+  Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, R);
+}
+
+static void initBlock(unsigned BlockID, BitstreamWriter &Bitstream,
+                      SmallVectorImpl<uint64_t> &R, StringRef Str) {
+  R.clear();
+  R.push_back(BlockID);
+  Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, R);
+
+  R.clear();
+  push(R, Str);
+  Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, R);
+}
+
+void BitstreamSerializerHelper::setupMetaBlockInfo() {
+  // Setup the metadata block.
+  initBlock(META_BLOCK_ID, Bitstream, R, MetaBlockName);
+
+  // The container information.
+  setRecordName(RECORD_META_CONTAINER_INFO, Bitstream, R,
+                MetaContainerInfoName);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(RECORD_META_CONTAINER_INFO));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version.
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 2));  // Type.
+  RecordMetaContainerInfoAbbrevID =
+      Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
+}
+
+void BitstreamSerializerHelper::setupMetaRemarkVersion() {
+  setRecordName(RECORD_META_REMARK_VERSION, Bitstream, R,
+                MetaRemarkVersionName);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(RECORD_META_REMARK_VERSION));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version.
+  RecordMetaRemarkVersionAbbrevID =
+      Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
+}
+
+void BitstreamSerializerHelper::emitMetaRemarkVersion(uint64_t RemarkVersion) {
+  // The remark version is emitted only if we emit remarks.
+  R.clear();
+  R.push_back(RECORD_META_REMARK_VERSION);
+  R.push_back(RemarkVersion);
+  Bitstream.EmitRecordWithAbbrev(RecordMetaRemarkVersionAbbrevID, R);
+}
+
+void BitstreamSerializerHelper::setupMetaStrTab() {
+  setRecordName(RECORD_META_STRTAB, Bitstream, R, MetaStrTabName);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(RECORD_META_STRTAB));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Raw table.
+  RecordMetaStrTabAbbrevID =
+      Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
+}
+
+void BitstreamSerializerHelper::emitMetaStrTab(const StringTable &StrTab) {
+  // The string table is not emitted if we emit remarks separately.
+  R.clear();
+  R.push_back(RECORD_META_STRTAB);
+
+  // Serialize to a blob.
+  std::string Buf;
+  raw_string_ostream OS(Buf);
+  StrTab.serialize(OS);
+  StringRef Blob = OS.str();
+  Bitstream.EmitRecordWithBlob(RecordMetaStrTabAbbrevID, R, Blob);
+}
+
+void BitstreamSerializerHelper::setupMetaExternalFile() {
+  setRecordName(RECORD_META_EXTERNAL_FILE, Bitstream, R, MetaExternalFileName);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(RECORD_META_EXTERNAL_FILE));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Filename.
+  RecordMetaExternalFileAbbrevID =
+      Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
+}
+
+void BitstreamSerializerHelper::emitMetaExternalFile(StringRef Filename) {
+  // The external file is emitted only if we emit the separate metadata.
+  R.clear();
+  R.push_back(RECORD_META_EXTERNAL_FILE);
+  Bitstream.EmitRecordWithBlob(RecordMetaExternalFileAbbrevID, R, Filename);
+}
+
+void BitstreamSerializerHelper::setupRemarkBlockInfo() {
+  // Setup the remark block.
+  initBlock(REMARK_BLOCK_ID, Bitstream, R, RemarkBlockName);
+
+  // The header of a remark.
+  {
+    setRecordName(RECORD_REMARK_HEADER, Bitstream, R, RemarkHeaderName);
+
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HEADER));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 3)); // Type
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6));   // Remark Name
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6));   // Pass name
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));   // Function name
+    RecordRemarkHeaderAbbrevID =
+        Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
+  }
+
+  // The location of a remark.
+  {
+    setRecordName(RECORD_REMARK_DEBUG_LOC, Bitstream, R, RemarkDebugLocName);
+
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_DEBUG_LOC));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7));    // File
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column
+    RecordRemarkDebugLocAbbrevID =
+        Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
+  }
+
+  // The hotness of a remark.
+  {
+    setRecordName(RECORD_REMARK_HOTNESS, Bitstream, R, RemarkHotnessName);
+
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HOTNESS));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Hotness
+    RecordRemarkHotnessAbbrevID =
+        Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
+  }
+
+  // An argument entry with a debug location attached.
+  {
+    setRecordName(RECORD_REMARK_ARG_WITH_DEBUGLOC, Bitstream, R,
+                  RemarkArgWithDebugLocName);
+
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITH_DEBUGLOC));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7));    // Key
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7));    // Value
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7));    // File
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column
+    RecordRemarkArgWithDebugLocAbbrevID =
+        Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
+  }
+
+  // An argument entry with no debug location attached.
+  {
+    setRecordName(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, Bitstream, R,
+                  RemarkArgWithoutDebugLocName);
+
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Key
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Value
+    RecordRemarkArgWithoutDebugLocAbbrevID =
+        Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
+  }
+}
+
+void BitstreamSerializerHelper::setupBlockInfo() {
+  // Emit magic number.
+  for (const char C : ContainerMagic)
+    Bitstream.Emit(static_cast<unsigned>(C), 8);
+
+  Bitstream.EnterBlockInfoBlock();
+
+  // Setup the main metadata. Depending on the container type, we'll setup the
+  // required records next.
+  setupMetaBlockInfo();
+
+  switch (ContainerType) {
+  case BitstreamRemarkContainerType::SeparateRemarksMeta:
+    // Needs a string table that the separate remark file is using.
+    setupMetaStrTab();
+    // Needs to know where the external remarks file is.
+    setupMetaExternalFile();
+    break;
+  case BitstreamRemarkContainerType::SeparateRemarksFile:
+    // Contains remarks: emit the version.
+    setupMetaRemarkVersion();
+    // Contains remarks: emit the remark abbrevs.
+    setupRemarkBlockInfo();
+    break;
+  case BitstreamRemarkContainerType::Standalone:
+    // Contains remarks: emit the version.
+    setupMetaRemarkVersion();
+    // Needs a string table.
+    setupMetaStrTab();
+    // Contains remarks: emit the remark abbrevs.
+    setupRemarkBlockInfo();
+    break;
+  }
+
+  Bitstream.ExitBlock();
+}
+
+void BitstreamSerializerHelper::emitMetaBlock(
+    uint64_t ContainerVersion, Optional<uint64_t> RemarkVersion,
+    Optional<const StringTable *> StrTab, Optional<StringRef> Filename) {
+  // Emit the meta block
+  Bitstream.EnterSubblock(META_BLOCK_ID, 3);
+
+  // The container version and type.
+  R.clear();
+  R.push_back(RECORD_META_CONTAINER_INFO);
+  R.push_back(ContainerVersion);
+  R.push_back(static_cast<uint64_t>(ContainerType));
+  Bitstream.EmitRecordWithAbbrev(RecordMetaContainerInfoAbbrevID, R);
+
+  switch (ContainerType) {
+  case BitstreamRemarkContainerType::SeparateRemarksMeta:
+    assert(StrTab != None && *StrTab != nullptr);
+    emitMetaStrTab(**StrTab);
+    assert(Filename != None);
+    emitMetaExternalFile(*Filename);
+    break;
+  case BitstreamRemarkContainerType::SeparateRemarksFile:
+    assert(RemarkVersion != None);
+    emitMetaRemarkVersion(*RemarkVersion);
+    break;
+  case BitstreamRemarkContainerType::Standalone:
+    assert(RemarkVersion != None);
+    emitMetaRemarkVersion(*RemarkVersion);
+    assert(StrTab != None && *StrTab != nullptr);
+    emitMetaStrTab(**StrTab);
+    break;
+  }
+
+  Bitstream.ExitBlock();
+}
+
+void BitstreamSerializerHelper::emitRemarkBlock(const Remark &Remark,
+                                                StringTable &StrTab) {
+  Bitstream.EnterSubblock(REMARK_BLOCK_ID, 4);
+
+  R.clear();
+  R.push_back(RECORD_REMARK_HEADER);
+  R.push_back(static_cast<uint64_t>(Remark.RemarkType));
+  R.push_back(StrTab.add(Remark.RemarkName).first);
+  R.push_back(StrTab.add(Remark.PassName).first);
+  R.push_back(StrTab.add(Remark.FunctionName).first);
+  Bitstream.EmitRecordWithAbbrev(RecordRemarkHeaderAbbrevID, R);
+
+  if (const Optional<RemarkLocation> &Loc = Remark.Loc) {
+    R.clear();
+    R.push_back(RECORD_REMARK_DEBUG_LOC);
+    R.push_back(StrTab.add(Loc->SourceFilePath).first);
+    R.push_back(Loc->SourceLine);
+    R.push_back(Loc->SourceColumn);
+    Bitstream.EmitRecordWithAbbrev(RecordRemarkDebugLocAbbrevID, R);
+  }
+
+  if (Optional<uint64_t> Hotness = Remark.Hotness) {
+    R.clear();
+    R.push_back(RECORD_REMARK_HOTNESS);
+    R.push_back(*Hotness);
+    Bitstream.EmitRecordWithAbbrev(RecordRemarkHotnessAbbrevID, R);
+  }
+
+  for (const Argument &Arg : Remark.Args) {
+    R.clear();
+    unsigned Key = StrTab.add(Arg.Key).first;
+    unsigned Val = StrTab.add(Arg.Val).first;
+    bool HasDebugLoc = Arg.Loc != None;
+    R.push_back(HasDebugLoc ? RECORD_REMARK_ARG_WITH_DEBUGLOC
+                            : RECORD_REMARK_ARG_WITHOUT_DEBUGLOC);
+    R.push_back(Key);
+    R.push_back(Val);
+    if (HasDebugLoc) {
+      R.push_back(StrTab.add(Arg.Loc->SourceFilePath).first);
+      R.push_back(Arg.Loc->SourceLine);
+      R.push_back(Arg.Loc->SourceColumn);
+    }
+    Bitstream.EmitRecordWithAbbrev(HasDebugLoc
+                                       ? RecordRemarkArgWithDebugLocAbbrevID
+                                       : RecordRemarkArgWithoutDebugLocAbbrevID,
+                                   R);
+  }
+  Bitstream.ExitBlock();
+}
+
+void BitstreamSerializerHelper::flushToStream(raw_ostream &OS) {
+  OS.write(Encoded.data(), Encoded.size());
+  Encoded.clear();
+}
+
+StringRef BitstreamSerializerHelper::getBuffer() {
+  return StringRef(Encoded.data(), Encoded.size());
+}
+
+BitstreamSerializer::BitstreamSerializer(raw_ostream &OS, SerializerMode Mode)
+    : RemarkSerializer(OS, Mode),
+      Helper(BitstreamRemarkContainerType::SeparateRemarksFile) {
+  assert(Mode == SerializerMode::Separate &&
+         "For SerializerMode::Standalone, a pre-filled string table needs to "
+         "be provided.");
+  // We always use a string table with bitstream.
+  StrTab.emplace();
+}
+
+BitstreamSerializer::BitstreamSerializer(raw_ostream &OS, SerializerMode Mode,
+                                         StringTable StrTabIn)
+    : RemarkSerializer(OS, Mode),
+      Helper(Mode == SerializerMode::Separate
+                 ? BitstreamRemarkContainerType::SeparateRemarksFile
+                 : BitstreamRemarkContainerType::Standalone) {
+  StrTab = std::move(StrTabIn);
+}
+
+void BitstreamSerializer::emit(const Remark &Remark) {
+  if (!DidSetUp) {
+    // Emit the metadata that is embedded in the remark file.
+    // If we're in standalone mode, serialize the string table as well.
+    bool IsStandalone =
+        Helper.ContainerType == BitstreamRemarkContainerType::Standalone;
+    BitstreamMetaSerializer MetaSerializer(
+        OS, Helper,
+        IsStandalone ? &*StrTab : Optional<const StringTable *>(None));
+    MetaSerializer.emit();
+    DidSetUp = true;
+  }
+
+  assert(DidSetUp &&
+         "The Block info block and the meta block were not emitted yet.");
+  Helper.emitRemarkBlock(Remark, *StrTab);
+
+  Helper.flushToStream(OS);
+}
+
+std::unique_ptr<MetaSerializer>
+BitstreamSerializer::metaSerializer(raw_ostream &OS,
+                                    Optional<StringRef> ExternalFilename) {
+  assert(Helper.ContainerType !=
+         BitstreamRemarkContainerType::SeparateRemarksMeta);
+  bool IsStandalone =
+      Helper.ContainerType == BitstreamRemarkContainerType::Standalone;
+  return llvm::make_unique<BitstreamMetaSerializer>(
+      OS,
+      IsStandalone ? BitstreamRemarkContainerType::Standalone
+                   : BitstreamRemarkContainerType::SeparateRemarksMeta,
+      &*StrTab, ExternalFilename);
+}
+
+void BitstreamMetaSerializer::emit() {
+  Helper->setupBlockInfo();
+  Helper->emitMetaBlock(CurrentContainerVersion, CurrentRemarkVersion, StrTab,
+                       ExternalFilename);
+  Helper->flushToStream(OS);
+}

llvm/trunk/lib/Remarks/CMakeLists.txt

 add_llvm_library(LLVMRemarks
+  BitstreamRemarkSerializer.cpp
   Remark.cpp
   RemarkFormat.cpp
   RemarkParser.cpp
   RemarkSerializer.cpp
   RemarkStringTable.cpp
   YAMLRemarkParser.cpp
   YAMLRemarkSerializer.cpp
 )

llvm/trunk/lib/Remarks/RemarkFormat.cpp

 #include "llvm/ADT/StringSwitch.h"
 
 using namespace llvm;
 using namespace llvm::remarks;
 
 Expected<Format> llvm::remarks::parseFormat(StringRef FormatStr) {
   auto Result = StringSwitch<Format>(FormatStr)
                     .Cases("", "yaml", Format::YAML)
-                    .Cases("", "yaml-strtab", Format::YAMLStrTab)
+                    .Case("yaml-strtab", Format::YAMLStrTab)
+                    .Case("bitstream", Format::Bitstream)
                     .Default(Format::Unknown);
 
   if (Result == Format::Unknown)
     return createStringError(std::make_error_code(std::errc::invalid_argument),
                              "Unknown remark format: '%s'",
                              FormatStr.data());
 
   return Result;
 }

llvm/trunk/lib/Remarks/RemarkParser.cpp

 llvm::remarks::createRemarkParser(Format ParserFormat, StringRef Buf) {
   switch (ParserFormat) {
   case Format::YAML:
     return llvm::make_unique<YAMLRemarkParser>(Buf);
   case Format::YAMLStrTab:
     return createStringError(
         std::make_error_code(std::errc::invalid_argument),
         "The YAML with string table format requires a parsed string table.");
+  case Format::Bitstream:
+    return createStringError(std::make_error_code(std::errc::invalid_argument),
+                             "Parsing bitstream remarks is not supported.");
   case Format::Unknown:
     return createStringError(std::make_error_code(std::errc::invalid_argument),
                              "Unknown remark parser format.");
   }
   llvm_unreachable("unhandled ParseFormat");
 }
 
 Expected<std::unique_ptr<RemarkParser>>
 llvm::remarks::createRemarkParser(Format ParserFormat, StringRef Buf,
                                   ParsedStringTable StrTab) {
   switch (ParserFormat) {
   case Format::YAML:
     return createStringError(std::make_error_code(std::errc::invalid_argument),
                              "The YAML format can't be used with a string "
                              "table. Use yaml-strtab instead.");
   case Format::YAMLStrTab:
     return llvm::make_unique<YAMLStrTabRemarkParser>(Buf, std::move(StrTab));
+  case Format::Bitstream:
+    return createStringError(std::make_error_code(std::errc::invalid_argument),
+                             "Parsing bitstream remarks is not supported.");
   case Format::Unknown:
     return createStringError(std::make_error_code(std::errc::invalid_argument),
                              "Unknown remark parser format.");
   }
   llvm_unreachable("unhandled ParseFormat");
 }
 
 Expected<std::unique_ptr<RemarkParser>>
 llvm::remarks::createRemarkParserFromMeta(Format ParserFormat, StringRef Buf,
                                           Optional<ParsedStringTable> StrTab) {
   switch (ParserFormat) {
   // Depending on the metadata, the format can be either yaml or yaml-strtab,
   // regardless of the input argument.
   case Format::YAML:
   case Format::YAMLStrTab:
     return createYAMLParserFromMeta(Buf, std::move(StrTab));
+  case Format::Bitstream:
+    return createStringError(std::make_error_code(std::errc::invalid_argument),
+                             "Parsing bitstream remarks is not supported.");
   case Format::Unknown:
     return createStringError(std::make_error_code(std::errc::invalid_argument),
                              "Unknown remark parser format.");
   }
   llvm_unreachable("unhandled ParseFormat");
 }
 
 // Wrapper that holds the state needed to interact with the C API.

llvm/trunk/lib/Remarks/RemarkSerializer.cpp

 //===- RemarkSerializer.cpp -----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file provides tools for serializing remarks.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Remarks/RemarkSerializer.h"
+#include "llvm/Remarks/BitstreamRemarkSerializer.h"
 #include "llvm/Remarks/YAMLRemarkSerializer.h"
 
 using namespace llvm;
 using namespace llvm::remarks;
 
 Expected<std::unique_ptr<RemarkSerializer>>
 remarks::createRemarkSerializer(Format RemarksFormat, SerializerMode Mode,
                                 raw_ostream &OS) {
   switch (RemarksFormat) {
   case Format::Unknown:
     return createStringError(std::errc::invalid_argument,
                              "Unknown remark serializer format.");
   case Format::YAML:
     return llvm::make_unique<YAMLRemarkSerializer>(OS, Mode);
   case Format::YAMLStrTab:
     return llvm::make_unique<YAMLStrTabRemarkSerializer>(OS, Mode);
+  case Format::Bitstream:
+    return llvm::make_unique<BitstreamSerializer>(OS, Mode);
   }
   llvm_unreachable("Unknown remarks::Format enum");
 }
 
 Expected<std::unique_ptr<RemarkSerializer>>
 remarks::createRemarkSerializer(Format RemarksFormat, SerializerMode Mode,
                                 raw_ostream &OS, remarks::StringTable StrTab) {
   switch (RemarksFormat) {
   case Format::Unknown:
     return createStringError(std::errc::invalid_argument,
                              "Unknown remark serializer format.");
   case Format::YAML:
     return createStringError(std::errc::invalid_argument,
                              "Unable to use a string table with the yaml "
                              "format. Use 'yaml-strtab' instead.");
   case Format::YAMLStrTab:
     return llvm::make_unique<YAMLStrTabRemarkSerializer>(OS, Mode,
                                                          std::move(StrTab));
+  case Format::Bitstream:
+    return llvm::make_unique<BitstreamSerializer>(OS, Mode, std::move(StrTab));
   }
   llvm_unreachable("Unknown remarks::Format enum");
 }

llvm/trunk/lib/Remarks/YAMLRemarkParser.cpp

 static Expected<uint64_t> parseVersion(StringRef &Buf) {
   if (Buf.size() < sizeof(uint64_t))
     return createStringError(std::errc::illegal_byte_sequence,
                              "Expecting version number.");
 
   uint64_t Version =
       support::endian::read<uint64_t, support::little, support::unaligned>(
           Buf.data());
-  if (Version != remarks::Version)
+  if (Version != remarks::CurrentRemarkVersion)
     return createStringError(std::errc::illegal_byte_sequence,
                              "Mismatching remark version. Got %" PRId64
                              ", expected %" PRId64 ".",
-                             Version, remarks::Version);
+                             Version, remarks::CurrentRemarkVersion);
   Buf = Buf.drop_front(sizeof(uint64_t));
   return Version;
 }
 
 static Expected<uint64_t> parseStrTabSize(StringRef &Buf) {
   if (Buf.size() < sizeof(uint64_t))
     return createStringError(std::errc::illegal_byte_sequence,
                              "Expecting string table size.");

llvm/trunk/lib/Remarks/YAMLRemarkSerializer.cpp

static void emitMagic(raw_ostream &OS) {
   OS << remarks::Magic;
   // Explicitly emit a '\0'.
   OS.write('\0');
 }
 
 static void emitVersion(raw_ostream &OS) {
   // Emit the version number: little-endian uint64_t.
   std::array<char, 8> Version;
-  support::endian::write64le(Version.data(), remarks::Version);
+  support::endian::write64le(Version.data(), remarks::CurrentRemarkVersion);
   OS.write(Version.data(), Version.size());
 }
 
 static void emitStrTab(raw_ostream &OS, const Optional<StringTable> &StrTab) {
   // Emit the string table in the section.
   uint64_t StrTabSize = StrTab ? StrTab->SerializedSize : 0;
   // Emit the total size of the string table (the size itself excluded):
   // little-endian uint64_t.

llvm/trunk/test/Bitcode/stream-types.c

 // Tests that llvm-bcanalyzer recognizes the correct "stream type" for various
 // common bitstream formats.
 
 // RUN: llvm-bcanalyzer -dump %s.ast | FileCheck %s -check-prefix=CHECK-AST
 // CHECK-AST: Stream type: Clang Serialized AST
 
 // RUN: llvm-bcanalyzer -dump %s.dia | FileCheck %s -check-prefix=CHECK-DIAG
 // CHECK-DIAG: Stream type: Clang Serialized Diagnostics
 
 // RUN: not llvm-bcanalyzer -dump %s.ast.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE
 // RUN: not llvm-bcanalyzer -dump %s.dia.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE
 // CHECK-INCOMPLETE: Bitcode stream should be a multiple of 4 bytes in length
+
+// RUN: llvm-bcanalyzer -dump %s.opt.bitstream | FileCheck %s -check-prefix=CHECK-REMARKS
+// CHECK-REMARKS: Stream type: LLVM Remarks

llvm/trunk/unittests/Remarks/BitstreamRemarksFormatTest.cpp

+//===- unittest/Support/BitstreamRemarksFormatTest.cpp - BitCodes tests ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Remarks/BitstreamRemarkContainer.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+// The goal for this test is to observe test failures and carefully update the
+// constants when they change.
+
+// This should not change over time.
+TEST(BitstreamRemarksFormat, Magic) {
+  EXPECT_EQ(remarks::ContainerMagic, "RMRK");
+}
+
+// This should be updated whenever any of the tests below are modified.
+TEST(BitstreamRemarksFormat, ContainerVersion) {
+  EXPECT_EQ(remarks::CurrentContainerVersion, 0UL);
+}
+
+// The values of the current blocks should not change over time.
+// When adding new blocks, make sure to append them to the enum.
+TEST(BitstreamRemarksFormat, BlockIDs) {
+  EXPECT_EQ(remarks::META_BLOCK_ID, 8);
+  EXPECT_EQ(remarks::REMARK_BLOCK_ID, 9);
+}
+
+// The values of the current records should not change over time.
+// When adding new records, make sure to append them to the enum.
+TEST(BitstreamRemarksFormat, RecordIDs) {
+  EXPECT_EQ(remarks::RECORD_FIRST, 1);
+  EXPECT_EQ(remarks::RECORD_META_CONTAINER_INFO, 1);
+  EXPECT_EQ(remarks::RECORD_META_REMARK_VERSION, 2);
+  EXPECT_EQ(remarks::RECORD_META_STRTAB, 3);
+  EXPECT_EQ(remarks::RECORD_META_EXTERNAL_FILE, 4);
+  EXPECT_EQ(remarks::RECORD_REMARK_HEADER, 5);
+  EXPECT_EQ(remarks::RECORD_REMARK_DEBUG_LOC, 6);
+  EXPECT_EQ(remarks::RECORD_REMARK_HOTNESS, 7);
+  EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITH_DEBUGLOC, 8);
+  EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, 9);
+  EXPECT_EQ(remarks::RECORD_LAST, 9);
+}

llvm/trunk/unittests/Remarks/BitstreamRemarksSerializerTest.cpp

+//===- unittest/Support/BitstreamRemarksSerializerTest.cpp ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Bitcode/BitcodeAnalyzer.h"
+#include "llvm/Remarks/BitstreamRemarkSerializer.h"
+#include "llvm/Support/raw_ostream.h"
+#include "gtest/gtest.h"
+#include <string>
+
+// We need to supprt Windows paths as well. In order to have paths with the same
+// length, use a different path according to the platform.
+#ifdef _WIN32
+#define EXTERNALFILETESTPATH "C:/externalfi"
+#else
+#define EXTERNALFILETESTPATH "/externalfile"
+#endif
+
+using namespace llvm;
+
+static void checkAnalyze(StringRef Input, StringRef Expected) {
+  std::string OutputBuf;
+  raw_string_ostream OutputOS(OutputBuf);
+  BCDumpOptions O(OutputOS);
+  O.ShowBinaryBlobs = true;
+  BitcodeAnalyzer BA(Input);
+  EXPECT_FALSE(BA.analyze(O)); // Expect no errors.
+  EXPECT_EQ(OutputOS.str(), Expected);
+}
+
+static void check(remarks::SerializerMode Mode, const remarks::Remark &R,
+                  StringRef ExpectedR, Optional<StringRef> ExpectedMeta,
+                  Optional<remarks::StringTable> StrTab) {
+  // Emit the remark.
+  std::string InputBuf;
+  raw_string_ostream InputOS(InputBuf);
+  Expected<std::unique_ptr<remarks::RemarkSerializer>> MaybeSerializer = [&] {
+    if (StrTab)
+      return createRemarkSerializer(remarks::Format::Bitstream, Mode, InputOS,
+                                    std::move(*StrTab));
+    else
+      return createRemarkSerializer(remarks::Format::Bitstream, Mode, InputOS);
+  }();
+  EXPECT_FALSE(errorToBool(MaybeSerializer.takeError()));
+  std::unique_ptr<remarks::RemarkSerializer> Serializer =
+      std::move(*MaybeSerializer);
+  Serializer->emit(R);
+
+  // Analyze the serialized remark.
+  checkAnalyze(InputOS.str(), ExpectedR);
+
+  // Analyze the serialized metadata if it's not in standalone mode.
+  if (ExpectedMeta) {
+    std::string MetaBuf;
+    raw_string_ostream MetaOS(MetaBuf);
+    std::unique_ptr<remarks::MetaSerializer> MetaSerializer =
+        Serializer->metaSerializer(MetaOS, StringRef(EXTERNALFILETESTPATH));
+    MetaSerializer->emit();
+    checkAnalyze(MetaOS.str(), *ExpectedMeta);
+  }
+}
+
+static void check(const remarks::Remark &R, StringRef ExpectedR,
+                  StringRef ExpectedMeta,
+                  Optional<remarks::StringTable> StrTab = None) {
+  return check(remarks::SerializerMode::Separate, R, ExpectedR, ExpectedMeta,
+               std::move(StrTab));
+}
+
+static void checkStandalone(const remarks::Remark &R, StringRef ExpectedR,
+                            Optional<remarks::StringTable> StrTab = None) {
+  return check(remarks::SerializerMode::Standalone, R, ExpectedR,
+               /*ExpectedMeta=*/None, std::move(StrTab));
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileNoOptionals) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=1 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileNoOptionalsSeparateStrTab) {
+  remarks::StringTable StrTab;
+  StrTab.add("function");
+  StrTab.add("pass");
+  StrTab.add("remark");
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=1 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=2 op2=1 op3=0/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'function\\x00pass\\x00remark\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n",
+        std::move(StrTab));
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileDebugLoc) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Loc.emplace();
+  R.Loc->SourceFilePath = "path";
+  R.Loc->SourceLine = 99;
+  R.Loc->SourceColumn = 55;
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=4 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "  <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=15 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00path\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileHotness) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Hotness.emplace(999999999);
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=3 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "  <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgNoDebugLoc) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Args.emplace_back();
+  R.Args.back().Key = "key";
+  R.Args.back().Val = "value";
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=2 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "  <Argument codeid=9 abbrevid=8 op0=3 op1=4/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=16 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00key\\x00value\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgDebugLoc) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Args.emplace_back();
+  R.Args.back().Key = "key";
+  R.Args.back().Val = "value";
+  R.Args.back().Loc.emplace();
+  R.Args.back().Loc->SourceFilePath = "path";
+  R.Args.back().Loc->SourceLine = 99;
+  R.Args.back().Loc->SourceColumn = 55;
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=4 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "  <Argument with debug location codeid=8 abbrevid=7 op0=3 op1=4 op2=5 "
+        "op3=99 op4=55/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=17 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00key\\x00value\\x00path\\x00'\n"
+        "  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, SeparateRemarkFileAll) {
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Loc.emplace();
+  R.Loc->SourceFilePath = "path";
+  R.Loc->SourceLine = 99;
+  R.Loc->SourceColumn = 55;
+  R.Hotness.emplace(999999999);
+  R.Args.emplace_back();
+  R.Args.back().Key = "key";
+  R.Args.back().Val = "value";
+  R.Args.back().Loc.emplace();
+  R.Args.back().Loc->SourceFilePath = "argpath";
+  R.Args.back().Loc->SourceLine = 11;
+  R.Args.back().Loc->SourceColumn = 66;
+  check(R,
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
+        "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+        "</Meta>\n"
+        "<Remark BlockID=9 NumWords=8 BlockCodeSize=4>\n"
+        "  <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
+        "  <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
+        "  <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
+        "  <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 "
+        "op3=11 op4=66/>\n"
+        "</Remark>\n",
+        "<BLOCKINFO_BLOCK/>\n"
+        "<Meta BlockID=8 NumWords=19 BlockCodeSize=3>\n"
+        "  <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
+        "  <String table codeid=3 abbrevid=5/> blob data = "
+        "'remark\\x00pass\\x00function\\x00path\\x00key\\x00value\\x00argpa"
+        "th\\x00'\n  <External File codeid=4 abbrevid=6/> blob data = "
+        "'" EXTERNALFILETESTPATH"'\n"
+        "</Meta>\n");
+}
+
+TEST(BitstreamRemarkSerializer, Standalone) {
+  // Pre-populate the string table.
+  remarks::StringTable StrTab;
+  StrTab.add("pass");
+  StrTab.add("remark");
+  StrTab.add("function");
+  StrTab.add("path");
+  StrTab.add("key");
+  StrTab.add("value");
+  StrTab.add("argpath");
+  remarks::Remark R;
+  R.RemarkType = remarks::Type::Missed;
+  R.PassName = "pass";
+  R.RemarkName = "remark";
+  R.FunctionName = "function";
+  R.Loc.emplace();
+  R.Loc->SourceFilePath = "path";
+  R.Loc->SourceLine = 99;
+  R.Loc->SourceColumn = 55;
+  R.Hotness.emplace(999999999);
+  R.Args.emplace_back();
+  R.Args.back().Key = "key";
+  R.Args.back().Val = "value";
+  R.Args.back().Loc.emplace();
+  R.Args.back().Loc->SourceFilePath = "argpath";
+  R.Args.back().Loc->SourceLine = 11;
+  R.Args.back().Loc->SourceColumn = 66;
+  checkStandalone(
+      R,
+      "<BLOCKINFO_BLOCK/>\n"
+      "<Meta BlockID=8 NumWords=15 BlockCodeSize=3>\n"
+      "  <Container info codeid=1 abbrevid=4 op0=0 op1=2/>\n"
+      "  <Remark version codeid=2 abbrevid=5 op0=0/>\n"
+      "  <String table codeid=3 abbrevid=6/> blob data = "
+      "'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x0"
+      "0'\n"
+      "</Meta>\n"
+      "<Remark BlockID=9 NumWords=8 BlockCodeSize=4>\n"
+      "  <Remark header codeid=5 abbrevid=4 op0=2 op1=1 op2=0 op3=2/>\n"
+      "  <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
+      "  <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
+      "  <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 "
+      "op3=11 op4=66/>\n"
+      "</Remark>\n",
+      std::move(StrTab));
+}

llvm/trunk/unittests/Remarks/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  BitReader
   Remarks
   Support
   )
 
 add_llvm_unittest(RemarksTests
+  BitstreamRemarksFormatTest.cpp
+  BitstreamRemarksSerializerTest.cpp
   RemarksStrTabParsingTest.cpp
   YAMLRemarksParsingTest.cpp
   YAMLRemarksSerializerTest.cpp
   )