Index: llvm/docs/Remarks.rst =================================================================== --- llvm/docs/Remarks.rst +++ llvm/docs/Remarks.rst @@ -112,6 +112,7 @@ Supported formats: * :ref:`yaml ` (default) + * :ref:`bitstream ` ``Content configuration`` @@ -191,6 +192,250 @@ * ```` * ```` +.. _bitstreamremarks: + +LLVM bitstream remarks +====================== + +This format is using :doc:`LLVM bitstream ` to serialize remarks +and their associated metadata. + +A bitstream remark stream can be identified by the magic number ``"RMRK"`` that +is placed at the very beginning. + +The format for serializing remarks is composed of two different block types: + +.. _bitstreamremarksmetablock: + +META_BLOCK +---------- + +The block providing information about the rest of the content in the stream. + +Exactly one block is expected. Having multiple metadata blocks is an error. + +This block can contain the following records: + +.. _bitstreamremarksrecordmetacontainerinfo: + +``RECORD_META_CONTAINER_INFO`` + + The container version and type. + + Version: u32 + + Type: u2 + +.. _bitstreamremarksrecordmetaremarkversion: + +``RECORD_META_REMARK_VERSION`` + + The version of the remark entries. This can change independently from the + container version. + + Version: u32 + +.. _bitstreamremarksrecordmetastrtab: + +``RECORD_META_STRTAB`` + + The string table used by the remark entries. The format of the string table + is a sequence of strings separated by ``\0``. + +.. _bitstreamremarksrecordmetaexternalfile: + +``RECORD_META_EXTERNAL_FILE`` + + The external remark file path that contains the remark blocks associated + with this metadata. This is an absolute path. + +.. _bitstreamremarksremarkblock: + +REMARK_BLOCK +------------ + +The block describing a remark entry. + +0 or more blocks per file are allowed. Each block will depend on the +:ref:`META_BLOCK ` in order to be parsed correctly. + +This block can contain the following records: + +``RECORD_REMARK_HEADER`` + + The header of the remark. This contains all the mandatory information about + a remark. + + +---------------+---------------------------+ + | Type | u3 | + +---------------+---------------------------+ + | Remark name | VBR6 (string table index) | + +---------------+---------------------------+ + | Pass name | VBR6 (string table index) | + +---------------+---------------------------+ + | Function name | VBR6 (string table index) | + +---------------+---------------------------+ + +``RECORD_REMARK_DEBUG_LOC`` + + The source location for the corresponding remark. This record is optional. + + +--------+---------------------------+ + | File | VBR7 (string table index) | + +--------+---------------------------+ + | Line | u32 | + +--------+---------------------------+ + | Column | u32 | + +--------+---------------------------+ + +``RECORD_REMARK_HOTNESS`` + + The hotness of the remark. This record is optional. + + +---------------+---------------------+ + | Hotness | VBR8 (string table index) | + +---------------+---------------------+ + +``RECORD_REMARK_ARG_WITH_DEBUGLOC`` + + A remark argument with an associated debug location. + + +--------+---------------------------+ + | Key | VBR7 (string table index) | + +--------+---------------------------+ + | Value | VBR7 (string table index) | + +--------+---------------------------+ + | File | VBR7 (string table index) | + +--------+---------------------------+ + | Line | u32 | + +--------+---------------------------+ + | Column | u32 | + +--------+---------------------------+ + +``RECORD_REMARK_ARG_WITHOUT_DEBUGLOC`` + + A remark argument with an associated debug location. + + +--------+---------------------------+ + | Key | VBR7 (string table index) | + +--------+---------------------------+ + | Value | VBR7 (string table index) | + +--------+---------------------------+ + +The remark container +-------------------- + +Bitstream remarks are designed to be used in two different modes: + +``The separate mode`` + + The separate mode is the mode that is typically used during compilation. It + provides a way to serialize the remark entries to a stream while some + metadata is kept in memory to be emitted in the product of the compilation + (typically, an object file). + +``The standalone mode`` + + The standalone mode is typically stored and used after the distribution of + a program. It contains all the information that allows the parsing of all + the remarks without having any external dependencies. + +In order to support multiple modes, the format introduces the concept of a +bitstream remark container type. + +.. _bitstreamremarksseparateremarksmeta: + +``SeparateRemarksMeta: the metadata emitted separately`` + + This container type expects only a :ref:`META_BLOCK ` containing only: + + * :ref:`RECORD_META_CONTAINER_INFO ` + * :ref:`RECORD_META_STRTAB ` + * :ref:`RECORD_META_EXTERNAL_FILE ` + + Typically, this is emitted in a section in the object files, allowing + clients to retrieve remarks and their associated metadata directly from + intermediate products. + +``SeparateRemarksFile: the remark entries emitted separately`` + + This container type expects only a :ref:`META_BLOCK ` containing only: + + * :ref:`RECORD_META_CONTAINER_INFO ` + * :ref:`RECORD_META_REMARK_VERSION ` + + This container type expects 0 or more :ref:`REMARK_BLOCK `. + + Typically, this is emitted in a side-file alongside an object file, and is + made to be able to stream to without increasing the memory consumption of + the compiler. This is referenced by the :ref:`RECORD_META_EXTERNAL_FILE + ` entry in the + :ref:`SeparateRemarksMeta ` container. + +When the parser tries to parse a container that contains the metadata for the +separate remarks, it should parse the version and type, then keep the string +table in memory while opening the external file, validating its metadata and +parsing the remark entries. + +The container versions from the separate container should match in order to +have a well-formed file. + +``Standalone: the metadata and the remark entries emitted together`` + + This container type expects only a :ref:`META_BLOCK ` containing only: + + * :ref:`RECORD_META_CONTAINER_INFO ` + * :ref:`RECORD_META_REMARK_VERSION ` + * :ref:`RECORD_META_STRTAB ` + + This container type expects 0 or more :ref:`REMARK_BLOCK `. + +A complete output of :program:`llvm-bcanalyzer` on the different container types: + +``SeparateRemarksMeta`` + +.. code-block:: none + + + + + blob data = 'pass\\x00key\\x00value\\x00' + blob data = '/path/to/file/name' + + +``SeparateRemarksFile`` + +.. code-block:: none + + + + + + + + + + + + + +``Standalone`` + +.. code-block:: none + + + + + + blob data = 'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x00' + + + + + + + + opt-viewer ========== Index: llvm/include/llvm/Bitcode/BitcodeAnalyzer.h =================================================================== --- llvm/include/llvm/Bitcode/BitcodeAnalyzer.h +++ llvm/include/llvm/Bitcode/BitcodeAnalyzer.h @@ -30,6 +30,7 @@ LLVMIRBitstream, ClangSerializedASTBitstream, ClangSerializedDiagnosticsBitstream, + LLVMBitstreamRemarks }; struct BCDumpOptions { Index: llvm/include/llvm/IR/RemarkStreamer.h =================================================================== --- llvm/include/llvm/IR/RemarkStreamer.h +++ llvm/include/llvm/IR/RemarkStreamer.h @@ -90,7 +90,7 @@ using RemarkSetupErrorInfo::RemarkSetupErrorInfo; }; -enum class RemarksSerializerFormat { Unknown, YAML }; +enum class RemarksSerializerFormat { Unknown, YAML, Bitstream }; Expected parseSerializerFormat(StringRef Format); Index: llvm/include/llvm/Remarks/BitstreamRemarkContainer.h =================================================================== --- /dev/null +++ llvm/include/llvm/Remarks/BitstreamRemarkContainer.h @@ -0,0 +1,104 @@ +//===-- BitstreamRemarkContainer.h - Container for remarks --------------*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file provides declarations for things used in the various types of +// remark containers. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_REMARKS_REMARK_CONTAINER_H +#define LLVM_REMARKS_REMARK_CONTAINER_H + +#include "llvm/ADT/StringRef.h" +#include "llvm/Bitstream/BitCodes.h" +#include + +namespace llvm { +namespace remarks { + +/// The current version of the remark container. +/// Note: this is different from the version of the remark entry. +constexpr uint64_t CurrentContainerVersion = 0; +/// The magic number used for identifying remark blocks. +constexpr StringRef Magic("RMRK", 4); + +/// Type of the remark container. +/// The remark container has two modes: +/// * separate: the metadata is separate from the remarks and points to the +/// auxiliary file that contains the remarks. +/// * standalone: the metadata and the remarks are emitted together. +enum class BitstreamRemarkContainerType { + /// The metadata emitted separately. + /// This will contain the following: + /// * Container version and type + /// * String table + /// * External file + SeparateRemarksMeta, + /// The remarks emitted separately. + /// This will contain the following: + /// * Container version and type + /// * Remark version + SeparateRemarksFile, + /// Everything is emitted together. + /// This will contain the following: + /// * Container version and type + /// * Remark version + /// * String table + Standalone +}; + +/// The possible blocks that will be encountered in a bitstream remark +/// container. +enum BlockIDs { + /// The metadata block is mandatory. It should always come after the + /// BLOCKINFO_BLOCK, and contains metadata that should be used when parsing + /// REMARK_BLOCKs. + /// There should always be only one META_BLOCK. + META_BLOCK_ID = bitc::FIRST_APPLICATION_BLOCKID, + /// One remark entry is represented using a REMARK_BLOCK. There can be + /// multiple REMARK_BLOCKs in the same file. + REMARK_BLOCK_ID +}; + +constexpr StringRef MetaBlockName = StringRef("Meta", 4); +constexpr StringRef RemarkBlockName = StringRef("Remark", 6); + +/// The possible records that can be encountered in the previously described +/// blocks. +enum RecordIDs { + // Meta block records. + RECORD_META_CONTAINER_INFO = 1, + RECORD_META_REMARK_VERSION, + RECORD_META_STRTAB, + RECORD_META_EXTERNAL_FILE, + // Remark block records. + RECORD_REMARK_HEADER, + RECORD_REMARK_DEBUG_LOC, + RECORD_REMARK_HOTNESS, + RECORD_REMARK_ARG_WITH_DEBUGLOC, + RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, + // Helpers. + RECORD_FIRST = RECORD_META_CONTAINER_INFO, + RECORD_LAST = RECORD_REMARK_ARG_WITHOUT_DEBUGLOC +}; + +constexpr StringRef MetaContainerInfoName = StringRef("Container info", 14); +constexpr StringRef MetaRemarkVersionName = StringRef("Remark version", 14); +constexpr StringRef MetaStrTabName = StringRef("String table", 12); +constexpr StringRef MetaExternalFileName = StringRef("External File", 13); +constexpr StringRef RemarkHeaderName = StringRef("Remark header", 13); +constexpr StringRef RemarkDebugLocName = StringRef("Remark debug location", 21); +constexpr StringRef RemarkHotnessName = StringRef("Remark hotness", 14); +constexpr StringRef RemarkArgWithDebugLocName = + StringRef("Argument with debug location", 28); +constexpr StringRef RemarkArgWithoutDebugLocName = StringRef("Argument", 8); + +} // end namespace remarks +} // end namespace llvm + +#endif /* LLVM_REMARKS_REMARK_CONTAINER_H */ Index: llvm/include/llvm/Remarks/BitstreamRemarkSerializer.h =================================================================== --- /dev/null +++ llvm/include/llvm/Remarks/BitstreamRemarkSerializer.h @@ -0,0 +1,136 @@ +//===-- BitstreamRemarkSerializer.h - Bitstream serializer ------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file provides an implementation of the serializer using the LLVM +// Bitstream format. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H +#define LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H + +#include "llvm/Bitstream/BitstreamWriter.h" +#include "llvm/Remarks/BitstreamRemarkContainer.h" +#include "llvm/Remarks/RemarkSerializer.h" +#include "llvm/Support/raw_ostream.h" + +namespace llvm { +namespace remarks { + +/// Serialize the remarks to LLVM bitstream. +/// This class provides ways to emit remarks in the LLVM bitstream format and +/// its associated metadata. +/// +/// * The separate model: +/// Separate meta: | Container info +/// | String table +/// | External file +/// +/// Separate remarks: | Container info +/// | Remark version +/// | Remark0 +/// | Remark1 +/// | Remark2 +/// | ... +/// +/// * The standalone model: | Container info +/// | String table +/// | Remark version +/// | Remark0 +/// | Remark1 +/// | Remark2 +/// | ... +/// +struct BitstreamSerializerHelper { + /// Buffer used for encoding the bitstream before writing it to the final + /// stream. + SmallVector Encoded; + /// Buffer used to construct records and pass to the bitstream writer. + SmallVector R; + /// The Bitstream writer. + BitstreamWriter Bitstream; + /// The type of the container we are serializing. + BitstreamRemarkContainerType ContainerType; + + /// Abbrev IDs initialized in the block info block. + /// Note: depending on the container type, some IDs might be uninitialized. + /// Warning: When adding more abbrev IDs, make sure to update the + /// BlockCodeSize (in the call to EnterSubblock). + uint64_t RecordMetaContainerInfoAbbrevID = 0; + uint64_t RecordMetaRemarkVersionAbbrevID = 0; + uint64_t RecordMetaStrTabAbbrevID = 0; + uint64_t RecordMetaExternalFileAbbrevID = 0; + uint64_t RecordRemarkHeaderAbbrevID = 0; + uint64_t RecordRemarkDebugLocAbbrevID = 0; + uint64_t RecordRemarkHotnessAbbrevID = 0; + uint64_t RecordRemarkArgWithDebugLocAbbrevID = 0; + uint64_t RecordRemarkArgWithoutDebugLocAbbrevID = 0; + + BitstreamSerializerHelper(BitstreamRemarkContainerType ContainerType); + + /// Set up the necessary block info entries according to the container type. + void setupBlockInfo(); + + /// Set up the block info for the metadata block. + void setupMetaBlockInfo(); + /// The remark version in the metadata block. + void setupMetaRemarkVersion(); + void emitMetaRemarkVersion(uint64_t RemarkVersion); + /// The strtab in the metadata block. + void setupMetaStrTab(); + void emitMetaStrTab(const StringTable &StrTab); + /// The external file in the metadata block. + void setupMetaExternalFile(); + void emitMetaExternalFile(StringRef Filename); + + /// The block info for the remarks block. + void setupRemarkBlockInfo(); + + /// Emit the metadata for the remarks. + void emitMetaBlock(uint64_t ContainerVersion, + Optional RemarkVersion, + Optional StrTab = None, + Optional Filename = None); + + /// Emit a remark block. The string table is required. + void emitRemarkBlock(const Remark &Remark, StringTable &StrTab); + /// Finalize the writing to \p OS. + void flushToStream(raw_ostream &OS); + /// Finalize the writing to a buffer. + /// The contents of the buffer remain valid for the lifetime of the object. + /// Any call to any other function in this class will invalidate the buffer. + StringRef getBuffer(); +}; + +/// Implementation of the remark serializer using LLVM bitstream. +struct BitstreamSerializer : public Serializer { + /// The file should contain: + /// 1) The block info block that describes how to read the blocks. + /// 2) The metadata block that contains various information about the remarks + /// in the file. + /// 3) A number of remark blocks. + + /// We need to set up 1) and 2) first, so that we can emit 3) after. This flag + /// is used to emit the first two blocks only once. + bool DidSetUp = false; + /// The helper to emit bitstream. + BitstreamSerializerHelper Helper; + + BitstreamSerializer(raw_ostream &OS); + + /// Emit a remark to the stream. This also emits the metadata associated to + /// the remarks and assumes the container type is + /// BitstreamRemarkContainerType::SeparateRemarksFile. + /// This writes the serialized output to the provided stream. + void emit(const Remark &Remark) override; +}; + +} // end namespace remarks +} // end namespace llvm + +#endif /* LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H */ Index: llvm/include/llvm/Remarks/Remark.h =================================================================== --- llvm/include/llvm/Remarks/Remark.h +++ llvm/include/llvm/Remarks/Remark.h @@ -23,8 +23,8 @@ namespace llvm { namespace remarks { -constexpr uint64_t Version = 0; -constexpr StringRef Magic("REMARKS", 7); +/// The current version of the remark entry. +constexpr uint64_t CurrentRemarkVersion = 0; /// The debug location used to track a remark back to the source file. struct RemarkLocation { Index: llvm/lib/Bitcode/Reader/BitcodeAnalyzer.cpp =================================================================== --- llvm/lib/Bitcode/Reader/BitcodeAnalyzer.cpp +++ llvm/lib/Bitcode/Reader/BitcodeAnalyzer.cpp @@ -432,6 +432,13 @@ return std::move(Err); if (Signature[2] == 'A' && Signature[3] == 'G') return ClangSerializedDiagnosticsBitstream; + } else if (Signature[0] == 'R' && Signature[1] == 'M') { + if (Error Err = tryRead(Signature[2], 8)) + return std::move(Err); + if (Error Err = tryRead(Signature[3], 8)) + return std::move(Err); + if (Signature[2] == 'R' && Signature[3] == 'K') + return LLVMBitstreamRemarks; } else { if (Error Err = tryRead(Signature[2], 4)) return std::move(Err); @@ -622,6 +629,9 @@ case ClangSerializedDiagnosticsBitstream: O.OS << "Clang Serialized Diagnostics\n"; break; + case LLVMBitstreamRemarks: + O.OS << "LLVM Remarks\n"; + break; } O.OS << " # Toplevel Blocks: " << NumTopBlocks << "\n"; O.OS << "\n"; Index: llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp =================================================================== --- llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp +++ llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp @@ -1356,14 +1356,14 @@ OutStreamer->SwitchSection(RemarksSection); // Emit the magic number. - OutStreamer->EmitBytes(remarks::Magic); + OutStreamer->EmitBytes("REMARKS"); // Explicitly emit a '\0'. OutStreamer->EmitIntValue(/*Value=*/0, /*Size=*/1); // Emit the version number: little-endian uint64_t. // The version number is located at the offset 0x0 in the section. std::array Version; - support::endian::write64le(Version.data(), remarks::Version); + support::endian::write64le(Version.data(), remarks::CurrentRemarkVersion); OutStreamer->EmitBinaryData(StringRef(Version.data(), Version.size())); // Emit the string table in the section. Index: llvm/lib/IR/RemarkStreamer.cpp =================================================================== --- llvm/lib/IR/RemarkStreamer.cpp +++ llvm/lib/IR/RemarkStreamer.cpp @@ -15,6 +15,7 @@ #include "llvm/IR/DiagnosticInfo.h" #include "llvm/IR/Function.h" #include "llvm/IR/GlobalValue.h" +#include "llvm/Remarks/BitstreamRemarkSerializer.h" using namespace llvm; @@ -119,6 +120,8 @@ return nullptr; case RemarksSerializerFormat::YAML: return llvm::make_unique(OS); + case RemarksSerializerFormat::Bitstream: + return llvm::make_unique(OS); }; } @@ -126,6 +129,7 @@ llvm::parseSerializerFormat(StringRef StrFormat) { auto Format = StringSwitch(StrFormat) .Cases("", "yaml", RemarksSerializerFormat::YAML) + .Case("bitstream", RemarksSerializerFormat::Bitstream) .Default(RemarksSerializerFormat::Unknown); if (Format == RemarksSerializerFormat::Unknown) Index: llvm/lib/Remarks/BitstreamRemarkSerializer.cpp =================================================================== --- /dev/null +++ llvm/lib/Remarks/BitstreamRemarkSerializer.cpp @@ -0,0 +1,347 @@ +//===- BitstreamRemarkSerializer.cpp --------------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file provides the implementation of the LLVM bitstream remark serializer +// using LLVM's bitstream writer. +// +//===----------------------------------------------------------------------===// + +#include "llvm/Remarks/BitstreamRemarkSerializer.h" + +using namespace llvm; +using namespace llvm::remarks; + +BitstreamSerializerHelper::BitstreamSerializerHelper( + BitstreamRemarkContainerType ContainerType) + : Encoded(), R(), Bitstream(Encoded), ContainerType(ContainerType) {} + +static void push(SmallVectorImpl &R, StringRef Str) { + for (const char C : Str) + R.push_back(C); +} + +static void setRecordName(unsigned RecordID, BitstreamWriter &Bitstream, + SmallVectorImpl &R, StringRef Str) { + R.clear(); + R.push_back(RecordID); + push(R, Str); + Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, R); +} + +static void initBlock(unsigned BlockID, BitstreamWriter &Bitstream, + SmallVectorImpl &R, StringRef Str) { + R.clear(); + R.push_back(BlockID); + Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, R); + + R.clear(); + push(R, Str); + Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, R); +} + +void BitstreamSerializerHelper::setupMetaBlockInfo() { + // Setup the metadata block. + initBlock(META_BLOCK_ID, Bitstream, R, MetaBlockName); + + // The container information. + setRecordName(RECORD_META_CONTAINER_INFO, Bitstream, R, + MetaContainerInfoName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_META_CONTAINER_INFO)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version. + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 2)); // Type. + RecordMetaContainerInfoAbbrevID = + Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev); +} + +void BitstreamSerializerHelper::setupMetaRemarkVersion() { + setRecordName(RECORD_META_REMARK_VERSION, Bitstream, R, + MetaRemarkVersionName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_META_REMARK_VERSION)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version. + RecordMetaRemarkVersionAbbrevID = + Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev); +} + +void BitstreamSerializerHelper::emitMetaRemarkVersion(uint64_t RemarkVersion) { + // The remark version is emitted only if we emit remarks. + R.clear(); + R.push_back(RECORD_META_REMARK_VERSION); + R.push_back(RemarkVersion); + Bitstream.EmitRecordWithAbbrev(RecordMetaRemarkVersionAbbrevID, R); +} + +void BitstreamSerializerHelper::setupMetaStrTab() { + setRecordName(RECORD_META_STRTAB, Bitstream, R, MetaStrTabName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_META_STRTAB)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Raw table. + RecordMetaStrTabAbbrevID = + Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev); +} + +void BitstreamSerializerHelper::emitMetaStrTab(const StringTable &StrTab) { + // The string table is not emitted if we emit remarks separately. + R.clear(); + R.push_back(RECORD_META_STRTAB); + + // Serialize to a blob. + std::string Buf; + raw_string_ostream OS(Buf); + StrTab.serialize(OS); + StringRef Blob = OS.str(); + Bitstream.EmitRecordWithBlob(RecordMetaStrTabAbbrevID, R, Blob); +} + +void BitstreamSerializerHelper::setupMetaExternalFile() { + setRecordName(RECORD_META_EXTERNAL_FILE, Bitstream, R, MetaExternalFileName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_META_EXTERNAL_FILE)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Filename. + RecordMetaExternalFileAbbrevID = + Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev); +} + +void BitstreamSerializerHelper::emitMetaExternalFile(StringRef Filename) { + // The external file is emitted only if we emit the separate metadata. + R.clear(); + R.push_back(RECORD_META_EXTERNAL_FILE); + Bitstream.EmitRecordWithBlob(RecordMetaExternalFileAbbrevID, R, Filename); +} + +void BitstreamSerializerHelper::setupRemarkBlockInfo() { + // Setup the remark block. + initBlock(REMARK_BLOCK_ID, Bitstream, R, RemarkBlockName); + + // The header of a remark. + { + setRecordName(RECORD_REMARK_HEADER, Bitstream, R, RemarkHeaderName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HEADER)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 3)); // Type + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Remark Name + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Pass name + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Function name + RecordRemarkHeaderAbbrevID = + Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev); + } + + // The location of a remark. + { + setRecordName(RECORD_REMARK_DEBUG_LOC, Bitstream, R, RemarkDebugLocName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_DEBUG_LOC)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // File + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column + RecordRemarkDebugLocAbbrevID = + Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev); + } + + // The hotness of a remark. + { + setRecordName(RECORD_REMARK_HOTNESS, Bitstream, R, RemarkHotnessName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HOTNESS)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Hotness + RecordRemarkHotnessAbbrevID = + Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev); + } + + // An argument entry with a debug location attached. + { + setRecordName(RECORD_REMARK_ARG_WITH_DEBUGLOC, Bitstream, R, + RemarkArgWithDebugLocName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITH_DEBUGLOC)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Key + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Value + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // File + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column + RecordRemarkArgWithDebugLocAbbrevID = + Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev); + } + + // An argument entry with no debug location attached. + { + setRecordName(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, Bitstream, R, + RemarkArgWithoutDebugLocName); + + auto Abbrev = std::make_shared(); + Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC)); + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Key + Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Value + RecordRemarkArgWithoutDebugLocAbbrevID = + Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev); + } +} + +void BitstreamSerializerHelper::setupBlockInfo() { + // Emit magic number. + for (const char C : Magic) + Bitstream.Emit(static_cast(C), 8); + + Bitstream.EnterBlockInfoBlock(); + + // Setup the main metadata. Depending on the container type, we'll setup the + // required records next. + setupMetaBlockInfo(); + + switch (ContainerType) { + case BitstreamRemarkContainerType::SeparateRemarksMeta: + // Needs a string table that the separate remark file is using. + setupMetaStrTab(); + // Needs to know where the external remarks file is. + setupMetaExternalFile(); + break; + case BitstreamRemarkContainerType::SeparateRemarksFile: + // Contains remarks: emit the version. + setupMetaRemarkVersion(); + // Contains remarks: emit the remark abbrevs. + setupRemarkBlockInfo(); + break; + case BitstreamRemarkContainerType::Standalone: + // Contains remarks: emit the version. + setupMetaRemarkVersion(); + // Needs a string table. + setupMetaStrTab(); + // Contains remarks: emit the remark abbrevs. + setupRemarkBlockInfo(); + break; + } + + Bitstream.ExitBlock(); +} + +void BitstreamSerializerHelper::emitMetaBlock( + uint64_t ContainerVersion, Optional RemarkVersion, + Optional StrTab, Optional Filename) { + // Emit the meta block + Bitstream.EnterSubblock(META_BLOCK_ID, 3); + + // The container version and type. + R.clear(); + R.push_back(RECORD_META_CONTAINER_INFO); + R.push_back(ContainerVersion); + R.push_back(static_cast(ContainerType)); + Bitstream.EmitRecordWithAbbrev(RecordMetaContainerInfoAbbrevID, R); + + switch (ContainerType) { + case BitstreamRemarkContainerType::SeparateRemarksMeta: + assert(StrTab != None && *StrTab != nullptr); + emitMetaStrTab(**StrTab); + assert(Filename != None); + emitMetaExternalFile(*Filename); + break; + case BitstreamRemarkContainerType::SeparateRemarksFile: + assert(RemarkVersion != None); + emitMetaRemarkVersion(*RemarkVersion); + break; + case BitstreamRemarkContainerType::Standalone: + assert(RemarkVersion != None); + emitMetaRemarkVersion(*RemarkVersion); + assert(StrTab != None && *StrTab != nullptr); + emitMetaStrTab(**StrTab); + break; + } + + Bitstream.ExitBlock(); +} + +void BitstreamSerializerHelper::emitRemarkBlock(const Remark &Remark, + StringTable &StrTab) { + Bitstream.EnterSubblock(REMARK_BLOCK_ID, 4); + + R.clear(); + R.push_back(RECORD_REMARK_HEADER); + R.push_back(static_cast(Remark.RemarkType)); + R.push_back(StrTab.add(Remark.RemarkName).first); + R.push_back(StrTab.add(Remark.PassName).first); + R.push_back(StrTab.add(Remark.FunctionName).first); + Bitstream.EmitRecordWithAbbrev(RecordRemarkHeaderAbbrevID, R); + + if (const Optional &Loc = Remark.Loc) { + R.clear(); + R.push_back(RECORD_REMARK_DEBUG_LOC); + R.push_back(StrTab.add(Loc->SourceFilePath).first); + R.push_back(Loc->SourceLine); + R.push_back(Loc->SourceColumn); + Bitstream.EmitRecordWithAbbrev(RecordRemarkDebugLocAbbrevID, R); + } + + if (Optional Hotness = Remark.Hotness) { + R.clear(); + R.push_back(RECORD_REMARK_HOTNESS); + R.push_back(*Hotness); + Bitstream.EmitRecordWithAbbrev(RecordRemarkHotnessAbbrevID, R); + } + + for (const Argument &Arg : Remark.Args) { + R.clear(); + unsigned Key = StrTab.add(Arg.Key).first; + unsigned Val = StrTab.add(Arg.Val).first; + bool HasDebugLoc = Arg.Loc != None; + R.push_back(HasDebugLoc ? RECORD_REMARK_ARG_WITH_DEBUGLOC + : RECORD_REMARK_ARG_WITHOUT_DEBUGLOC); + R.push_back(Key); + R.push_back(Val); + if (HasDebugLoc) { + R.push_back(StrTab.add(Arg.Loc->SourceFilePath).first); + R.push_back(Arg.Loc->SourceLine); + R.push_back(Arg.Loc->SourceColumn); + } + Bitstream.EmitRecordWithAbbrev(HasDebugLoc + ? RecordRemarkArgWithDebugLocAbbrevID + : RecordRemarkArgWithoutDebugLocAbbrevID, + R); + } + Bitstream.ExitBlock(); +} + +void BitstreamSerializerHelper::flushToStream(raw_ostream &OS) { + OS.write(Encoded.data(), Encoded.size()); + Encoded.clear(); +} + +StringRef BitstreamSerializerHelper::getBuffer() { + return StringRef(Encoded.data(), Encoded.size()); +} + +BitstreamSerializer::BitstreamSerializer(raw_ostream &OS) + : Serializer(OS), + Helper(BitstreamRemarkContainerType::SeparateRemarksFile) { + // We always use a string table with bitstream. + StrTab.emplace(); +} + +void BitstreamSerializer::emit(const Remark &Remark) { + if (!DidSetUp) { + Helper.setupBlockInfo(); + Helper.emitMetaBlock(/*ContainerVersion=*/CurrentContainerVersion, + /*RemarkVersion=*/CurrentRemarkVersion, + /*StrTab=*/&*StrTab, + /*Filename=*/None); + DidSetUp = true; + } + + assert(DidSetUp && + "The Block info block and the meta block were not emitted yet."); + Helper.emitRemarkBlock(Remark, *StrTab); + + Helper.flushToStream(OS); +} Index: llvm/lib/Remarks/CMakeLists.txt =================================================================== --- llvm/lib/Remarks/CMakeLists.txt +++ llvm/lib/Remarks/CMakeLists.txt @@ -1,4 +1,5 @@ add_llvm_library(LLVMRemarks + BitstreamRemarkSerializer.cpp Remark.cpp RemarkParser.cpp RemarkStringTable.cpp Index: llvm/lib/Remarks/RemarkStringTable.cpp =================================================================== --- llvm/lib/Remarks/RemarkStringTable.cpp +++ llvm/lib/Remarks/RemarkStringTable.cpp @@ -29,9 +29,6 @@ } void StringTable::serialize(raw_ostream &OS) const { - // Emit the number of strings. - uint64_t StrTabSize = SerializedSize; - support::endian::write(OS, StrTabSize, support::little); // Emit the sequence of strings. for (StringRef Str : serialize()) { OS << Str; Index: llvm/test/Bitcode/stream-types.c =================================================================== --- llvm/test/Bitcode/stream-types.c +++ llvm/test/Bitcode/stream-types.c @@ -10,3 +10,6 @@ // RUN: not llvm-bcanalyzer -dump %s.ast.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE // RUN: not llvm-bcanalyzer -dump %s.dia.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE // CHECK-INCOMPLETE: Bitcode stream should be a multiple of 4 bytes in length + +// RUN: llvm-bcanalyzer -dump %s.opt.bitstream | FileCheck %s -check-prefix=CHECK-REMARKS +// CHECK-REMARKS: Stream type: LLVM Remarks Index: llvm/unittests/Remarks/BitstreamRemarksFormatTest.cpp =================================================================== --- /dev/null +++ llvm/unittests/Remarks/BitstreamRemarksFormatTest.cpp @@ -0,0 +1,46 @@ +//===- unittest/Support/BitstreamRemarksFormatTest.cpp - BitCodes tests ---===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Remarks/BitstreamRemarkContainer.h" +#include "gtest/gtest.h" + +using namespace llvm; + +// The goal for this test is to observe test failures and carefully update the +// constants when they change. + +// This should not change over time. +TEST(BitstreamRemarksFormat, Magic) { EXPECT_EQ(remarks::Magic, "RMRK"); } + +// This should be updated whenever any of the tests below are modified. +TEST(BitstreamRemarksFormat, ContainerVersion) { + EXPECT_EQ(remarks::CurrentContainerVersion, 0UL); +} + +// The values of the current blocks should not change over time. +// When adding new blocks, make sure to append them to the enum. +TEST(BitstreamRemarksFormat, BlockIDs) { + EXPECT_EQ(remarks::META_BLOCK_ID, 8); + EXPECT_EQ(remarks::REMARK_BLOCK_ID, 9); +} + +// The values of the current records should not change over time. +// When adding new records, make sure to append them to the enum. +TEST(BitstreamRemarksFormat, RecordIDs) { + EXPECT_EQ(remarks::RECORD_FIRST, 1); + EXPECT_EQ(remarks::RECORD_META_CONTAINER_INFO, 1); + EXPECT_EQ(remarks::RECORD_META_REMARK_VERSION, 2); + EXPECT_EQ(remarks::RECORD_META_STRTAB, 3); + EXPECT_EQ(remarks::RECORD_META_EXTERNAL_FILE, 4); + EXPECT_EQ(remarks::RECORD_REMARK_HEADER, 5); + EXPECT_EQ(remarks::RECORD_REMARK_DEBUG_LOC, 6); + EXPECT_EQ(remarks::RECORD_REMARK_HOTNESS, 7); + EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITH_DEBUGLOC, 8); + EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, 9); + EXPECT_EQ(remarks::RECORD_LAST, 9); +} Index: llvm/unittests/Remarks/BitstreamRemarksSerializerTest.cpp =================================================================== --- /dev/null +++ llvm/unittests/Remarks/BitstreamRemarksSerializerTest.cpp @@ -0,0 +1,267 @@ +//===- unittest/Support/BitstreamRemarksSerializerTest.cpp ----------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Remarks/BitstreamRemarkSerializer.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Bitcode/BitcodeAnalyzer.h" +#include "gtest/gtest.h" +#include + +using namespace llvm; + +static void checkAnalyze(StringRef Input, StringRef Expected) { + std::string OutputBuf; + raw_string_ostream OutputOS(OutputBuf); + BCDumpOptions O(OutputOS); + O.ShowBinaryBlobs = true; + BitcodeAnalyzer BA(Input); + EXPECT_FALSE(BA.analyze(O)); // Expect no errors. + EXPECT_EQ(OutputOS.str(), Expected); +} + +static void check(const remarks::Remark &R, StringRef ExpectedStrTab, + StringRef Expected) { + // Emit the remark. + std::string InputBuf; + raw_string_ostream InputOS(InputBuf); + remarks::BitstreamSerializer Serializer(InputOS); + Serializer.emit(R); + + // Analyze the serialized remark. + checkAnalyze(InputOS.str(), Expected); + + // Analyze the string table. + std::string StrTabBuf; + raw_string_ostream StrTabOS(StrTabBuf); + Serializer.StrTab->serialize(StrTabOS); + EXPECT_EQ(StrTabOS.str(), ExpectedStrTab); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileNoOptionals) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + check(R, StringRef("remark\0pass\0function\0", 21), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileDebugLoc) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + R.Loc.emplace(); + R.Loc->SourceFilePath = "path"; + R.Loc->SourceLine = 99; + R.Loc->SourceColumn = 55; + check(R, StringRef("remark\0pass\0function\0path\0", 26), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileHotness) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + R.Hotness.emplace(999999999); + check(R, StringRef("remark\0pass\0function\0", 21), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgNoDebugLoc) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + SmallVector Args; + Args.emplace_back(); + Args.back().Key = "key"; + Args.back().Val = "value"; + R.Args = Args; + check(R, StringRef("remark\0pass\0function\0key\0value\0", 31), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgDebugLoc) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + SmallVector Args; + Args.emplace_back(); + Args.back().Key = "key"; + Args.back().Val = "value"; + Args.back().Loc.emplace(); + Args.back().Loc->SourceFilePath = "path"; + Args.back().Loc->SourceLine = 99; + Args.back().Loc->SourceColumn = 55; + R.Args = Args; + check(R, StringRef("remark\0pass\0function\0key\0value\0path\0", 36), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkFileAll) { + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + R.Loc.emplace(); + R.Loc->SourceFilePath = "path"; + R.Loc->SourceLine = 99; + R.Loc->SourceColumn = 55; + R.Hotness.emplace(999999999); + SmallVector Args; + Args.emplace_back(); + Args.back().Key = "key"; + Args.back().Val = "value"; + Args.back().Loc.emplace(); + Args.back().Loc->SourceFilePath = "argpath"; + Args.back().Loc->SourceLine = 11; + Args.back().Loc->SourceColumn = 66; + R.Args = Args; + check(R, StringRef("remark\0pass\0function\0path\0key\0value\0argpath\0", 44), + "\n" + "\n" + " \n" + " \n" + "\n" + "\n" + " \n" + " \n" + " \n" + " \n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, SeparateRemarkMeta) { + remarks::StringTable StrTab; + StrTab.add("pass"); + StrTab.add("key"); + StrTab.add("value"); + + remarks::BitstreamSerializerHelper MetaSerializer( + remarks::BitstreamRemarkContainerType::SeparateRemarksMeta); + MetaSerializer.setupBlockInfo(); + MetaSerializer.emitMetaBlock(5, /*RemarkVersion=*/None, &StrTab, + StringRef("/path/to/file/name")); + checkAnalyze(MetaSerializer.getBuffer(), + "\n" + "\n" + " \n" + " blob data = " + "'pass\\x00key\\x00value\\x00'\n" + " blob data = " + "'/path/to/file/name'\n" + "\n"); +} + +TEST(BitstreamRemarkSerializer, Standalone) { + // Pre-populate the string table. + remarks::StringTable StrTab; + StrTab.add("pass"); + StrTab.add("remark"); + StrTab.add("function"); + StrTab.add("path"); + StrTab.add("key"); + StrTab.add("value"); + StrTab.add("argpath"); + + // Emit the metadata. + remarks::BitstreamSerializerHelper Serializer( + remarks::BitstreamRemarkContainerType::Standalone); + Serializer.setupBlockInfo(); + Serializer.emitMetaBlock(/*ContainerVersion=*/5, /*RemarkVersion=*/30, + &StrTab); + remarks::Remark R; + R.RemarkType = remarks::Type::Missed; + R.PassName = "pass"; + R.RemarkName = "remark"; + R.FunctionName = "function"; + R.Loc.emplace(); + R.Loc->SourceFilePath = "path"; + R.Loc->SourceLine = 99; + R.Loc->SourceColumn = 55; + R.Hotness.emplace(999999999); + SmallVector Args; + Args.emplace_back(); + Args.back().Key = "key"; + Args.back().Val = "value"; + Args.back().Loc.emplace(); + Args.back().Loc->SourceFilePath = "argpath"; + Args.back().Loc->SourceLine = 11; + Args.back().Loc->SourceColumn = 66; + R.Args = Args; + // Emit the remark. + Serializer.emitRemarkBlock(R, StrTab); + + checkAnalyze( + Serializer.getBuffer(), + "\n" + "\n" + " \n" + " \n" + " blob data = " + "'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x0" + "0'\n" + "\n" + "\n" + " \n" + " \n" + " \n" + " \n" + "\n"); +} Index: llvm/unittests/Remarks/CMakeLists.txt =================================================================== --- llvm/unittests/Remarks/CMakeLists.txt +++ llvm/unittests/Remarks/CMakeLists.txt @@ -1,9 +1,12 @@ set(LLVM_LINK_COMPONENTS + BitReader Remarks Support ) add_llvm_unittest(RemarksTests + BitstreamRemarksFormatTest.cpp + BitstreamRemarksSerializerTest.cpp RemarksStrTabParsingTest.cpp YAMLRemarksParsingTest.cpp )