[XRay] FDR Trace Dump Tool (FDR Trace Loading Refactor)

[dberris]

Summary

This change implements a FDR trace dumping tool which allows us to
inspect the contents of an FDR mode trace in isolation. This is the
beginning of a larger refactoring of the FDR mode trace loading
implementation.

This change breaks up the FDR trace loading implementation into a number
of parts, allowing for better composition, testing, and extension.

The first part of this change is defining an abstract base type "Record"
which branches off into either a set of "MetadataRecord" types or
"FunctionRecord" instances. This allows us to then to isolate the logic
of reading data from a DataExtractor from interpreting the structure of
the log.

Doing this allows us to then more faithfully represent the data from a
trace as we see it from a log, independent of the semantic structure of
the trace.

The other part of this change involves defining a visitor abstraction,
which we use to process the log at a higher level of abstraction. It
allows us to process blocks of the trace more correctly, by grouping
individual records as blocks and arranging them by process+thread, and
interpreting the execution traces on a per-thread basis. To this end, we
implement a number of visitors which isolate the various higher-level
functions that operate on FDR traces. Some of these are:

• BlockIndexer: A visitor that generates an index grouping of blocks.

• RecordPrinter: A visitor that can print an individual Record instance.

• BlockPrinter: A visitor which defines a block-specific

pretty-printer.

• BlockValidator: A visitor that validates whether the log's structure is consistent with the expectations of the FDR mode log format.

We're starting the refactoring on the FDR mode implementation, then
performing a similar refactoring on the basic mode trace loading
implementation.

In this change we also take a chance to define a builder for each of the
record types, to allow us to generate valid and invalid sequences of
records in unit-testing. The builder interface is solely for testing
purposes.

NOTE: This change is rather large, suggestions for potential partitions are most welcome.

[kpw]

Suggestion for splitting this up.

Patch 1: FDRRecords and RecordVisitor and the Record subclasses.
Patch 2: Visitor implementations
Patch 3: fdr-dump command

I only read code from the hypothetical Patch 1 for these comments.

llvm/include/llvm/XRay/BlockPrinter.h

61

newline?

llvm/include/llvm/XRay/BlockVerifier.h

26

If you forward declare this and move the TransitionTable to the cc file, you could use the .inc macro trick to automatically get two string from the list.

45–67

It's hard to tell whether this is right. I didn't notice any problems, but it is dense.

Is there a better way? Especially with regards to new record types?

llvm/include/llvm/XRay/FDRRecords.h

30–34

Could be confusing for name lookup that this has the same name (modulo qualifiers) as BlockVerifier::RecordKind

51

It kind of seems like this member function should be const qualified and the visit method should take a const reference to Record.

229

FnType? Since there are already a few RecordKind types to possibly confuse.

244–246

Mixed casing convention. You have like_this() and likeThat().

258

I think the Record references can and should be const.

284

push_back(&&) of wrap_unique(new R(std::forward<T>(A)...)) would be my preference.

Emplacement coupled with the unique pointer within the vector template parameter is equivalent but less obvious to me.

299–300

Does this replace or append to the Records? Can it even be called multiple times?

llvm/include/llvm/XRay/RecordPrinter.h

49

No newline.

llvm/lib/XRay/Common.h

1 ↗

(On Diff #159695)

This is almost never a good name and this file is doomed to become a miscellaneous catch all.

llvm/lib/XRay/FDRRecords.cpp

2

I'm going to only skim this and ask you whether you made any changes other than just copy and paste to the logic that lived elsewhere other than the state transition map and assigning to fields instead of structs directly.

277–278

Meh. This seems harmless to me. You could declare an enum with the constants if you like as an easy fix.

[kpw]

Thought that occurred to me this morning:

This models a sequence of records and not a more complex structure like a tree. Instead of greedily filling a vector and then calling visitors, this could be better done with a streaming model.

The "driver" code would maintain a sequence of visitors and call them on each record that it reads.

It should be a small refactoring and could dramatically improve memory use especially for stateless visitors like the printer.

[dberris]

In D50441#1194024, @kpw wrote:

Thought that occurred to me this morning:

This models a sequence of records and not a more complex structure like a tree. Instead of greedily filling a vector and then calling visitors, this could be better done with a streaming model.

Unfortunately, it's not that simple.

Blocks that belong to the same threads/processes have a well-defined order and model a sequence of operations bound to time. This is the reason the indexer is a key visitor implementation, and why we can implement a validator that applies to logical blocks. In the process of re-constituting a single execution trace, we need to model the execution context as individual streams of execution that interlace over time in multiple dimensions (cpu, process, thread).

The complication here is that at runtime, since the implementation can be re-using buffers in a circular buffer, we cannot assume that the blocks which appear at the beginning always come before the blocks that follow in the time dimension. The spatial ordering doesn't imply temporal ordering, so we need to somehow re-construct the temporal order (hence the indexer).

What this patch doesn't show is how we're really supposed to sort the blocks in the indexes first, in temporal order (using the block's wallclock record), before reconstituting a full Trace object (which is the ultimate goal anyway) which requires the logical verification first. That's a separate algorithm which we don't implement yet.

There's a bit of foreshadowing going on, which I really should have explained somewhere in the first place. :)

The "driver" code would maintain a sequence of visitors and call them on each record that it reads.

It should be a small refactoring and could dramatically improve memory use especially for stateless visitors like the printer.

For the purposes of printing a log, I agree.

That seems like a simple algorithm which applies visitors for every record we encounter as we encounter it, without validating, i.e. a true "dump" tool. The missing abstraction seems to be a pipelining mechanism, or a sequencing visitor that determines the order of the visitors to apply.

That doesn't obviate the need for a logical in-memory structure for the reconstitution algorithm though for other uses. I suppose those can come later when we end up actually moving the current convert implementation to using this infrastructure for handling FDR mode logs.

[dberris]

• fixup: optimize storage of transition table
• checkpoint: implement producer API and migrate record initialization to a visitor
• checkpoint: implement non-verifying dump mode

[dberris]

I'll be breaking this up into smaller patches, please bear with me. I've also in the process added a few more abstractions, which makes the types cleaner to operate with -- for instance, I've moved the initialization of records into a visitor which is a friend of each of the record types. This way we isolate the logic of loading the data from a DataExtractor from the Record types. There will be more moving around of types and filenames, so please hold off on further review.

llvm/include/llvm/XRay/BlockVerifier.h

26

Unfortunately, it seems I can't really do that with C++11 enum classes.

45–67

I moved it to the implementation file, and added a few comments to represent the transitions. I'm not exactly sure if it's better, but worth the shot.

llvm/include/llvm/XRay/FDRRecords.h

51

Unfortunately, that contract is much tighter and severely limits the kinds of visitors that could be implemented -- there may be some that end up needing to modify a record when it's visited (you can imagine a visitor that adjusts the deltas for FunctionRecord instances, or those that process the data associated with a custom event, etc.).

244–246

fixed -- using camelCase() more consistently for member functions.

258

Explained elsewhere, that limits the kinds of visitors that could be written. I rather keep it like this to allow visitor implementations to modify Record instances should they need to.

llvm/lib/XRay/Common.h

1 ↗

(On Diff #159695)

Good point. Renamed.

llvm/lib/XRay/FDRRecords.cpp

2

This is mostly copy-paste, although the custom event loading implementation also reads the data from the log.

[dberris]

All changes here have now landed in a series of patches. Abandoning this revision now.

llvm/include/llvm/XRay/BlockIndexer.h

+//===- BlockIndexer.h - FDR Block Indexing Visitor ------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// An implementation of the RecordVisitor which generates a mapping between a
+// thread and a range of records representing a block.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIB_XRAY_BLOCKINDEXER_H_
+#define LLVM_LIB_XRAY_BLOCKINDEXER_H_
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/XRay/FDRRecords.h"
+#include <cstdint>
+#include <vector>
+
+namespace llvm {
+namespace xray {
+
+// The BlockIndexer will gather all related records associated with a
+// process+thread and group them by 'Block'.
+class BlockIndexer : public RecordVisitor {
+public:
+  struct Block {
+    uint64_t ProcessID;
+    int32_t ThreadID;
+    std::vector<Record *> Records;
+  };
+
+  // This maps the process + thread combination to a sequence of blocks.
+  using Index = DenseMap<std::pair<uint64_t, int32_t>, std::vector<Block>>;
+
+private:
+  Index &Indices;
+
+  enum class State : unsigned { SeekExtents, ExtentsFound, ThreadIDFound };
+
+  State CurrentState = State::SeekExtents;
+  Block CurrentBlock{0, 0, {}};
+
+public:
+  explicit BlockIndexer(Index &I) : RecordVisitor(), Indices(I) {}
+
+  Error visit(BufferExtents &) override;
+  Error visit(WallclockRecord &) override;
+  Error visit(NewCPUIDRecord &) override;
+  Error visit(TSCWrapRecord &) override;
+  Error visit(CustomEventRecord &) override;
+  Error visit(CallArgRecord &) override;
+  Error visit(PIDRecord &) override;
+  Error visit(NewBufferRecord &) override;
+  Error visit(EndBufferRecord &) override;
+  Error visit(FunctionRecord &) override;
+
+  /// The flush() function will clear out the current state of the visitor, to
+  /// allow for explicitly flushing a block's records to the currently
+  /// recognized thread and process combination.
+  Error flush();
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_LIB_XRAY_BLOCKINDEXER_H_

llvm/include/llvm/XRay/BlockPrinter.h

+//===- BlockPrinter.h - FDR Block Pretty Printer -------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// An implementation of the RecordVisitor which formats a block of records for
+// easier human consumption.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_INCLUDE_LLVM_XRAY_BLOCKPRINTER_H_
+#define LLVM_INCLUDE_LLVM_XRAY_BLOCKPRINTER_H_
+
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/XRay/FDRRecords.h"
+#include "llvm/XRay/RecordPrinter.h"
+
+namespace llvm {
+namespace xray {
+
+class BlockPrinter : public RecordVisitor {
+  enum class State {
+    Start,
+    Preamble,
+    Metadata,
+    Function,
+    Arg,
+    CustomEvent,
+    End,
+  };
+
+  raw_ostream &OS;
+  RecordPrinter &RP;
+  State CurrentState = State::Start;
+
+public:
+  explicit BlockPrinter(raw_ostream &O, RecordPrinter &P)
+      : RecordVisitor(), OS(O), RP(P) {}
+
+  Error visit(BufferExtents &) override;
+  Error visit(WallclockRecord &) override;
+  Error visit(NewCPUIDRecord &) override;
+  Error visit(TSCWrapRecord &) override;
+  Error visit(CustomEventRecord &) override;
+  Error visit(CallArgRecord &) override;
+  Error visit(PIDRecord &) override;
+  Error visit(NewBufferRecord &) override;
+  Error visit(EndBufferRecord &) override;
+  Error visit(FunctionRecord &) override;
+
+  void reset() { CurrentState = State::Start; }
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_INCLUDE_LLVM_XRAY_BLOCKPRINTER_H_

llvm/include/llvm/XRay/BlockVerifier.h

+//===- BlockVerifier.h - FDR Block Verifier -------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// An implementation of the RecordVisitor which verifies a sequence of records
+// associated with a block, following the FDR mode log format's specifications.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_INCLUDE_LLVM_XRAY_BLOCKVERIFIER_H_
+#define LLVM_INCLUDE_LLVM_XRAY_BLOCKVERIFIER_H_
+
+#include "llvm/XRay/FDRRecords.h"
+#include <array>
+#include <bitset>
+
+namespace llvm {
+namespace xray {
+
+class BlockVerifier : public RecordVisitor {
+public:
+  // We force State elements to be size_t, to be used as indices for containers.
+  enum class State : std::size_t {
+    Unknown,
+    BufferExtents,
+    NewBuffer,
+    WallClockTime,
+    PIDEntry,
+    NewCPUId,
+    TSCWrap,
+    CustomEvent,
+    Function,
+    CallArg,
+    EndOfBuffer,
+    StateMax,
+  };
+
+private:
+  // We keep track of the current record seen by the verifier.
+  State CurrentRecord = State::Unknown;
+
+  // Helper function to convert a State into a StringRef.
+  StringRef recordToString(State);
+
+  // Transitions the current record to the new record, records an error on
+  // invalid transitions.
+  Error transition(State To);
+
+  static constexpr inline unsigned long long mask(State S) {
+    return 1uLL << static_cast<std::size_t>(S);
+  }
+
+  static constexpr inline std::size_t number(State S) {
+    return static_cast<std::size_t>(S);
+  }
+
+public:
+  Error visit(BufferExtents &) override;
+  Error visit(WallclockRecord &) override;
+  Error visit(NewCPUIDRecord &) override;
+  Error visit(TSCWrapRecord &) override;
+  Error visit(CustomEventRecord &) override;
+  Error visit(CallArgRecord &) override;
+  Error visit(PIDRecord &) override;
+  Error visit(NewBufferRecord &) override;
+  Error visit(EndBufferRecord &) override;
+  Error visit(FunctionRecord &) override;
+
+  Error verify();
+  void reset();
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_INCLUDE_LLVM_XRAY_BLOCKVERIFIER_H_

llvm/include/llvm/XRay/FDRRecordConsumer.h

+//===- FDRRecordConsumer.h - XRay Flight Data Recorder Mode Records -------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_INCLUDE_LLVM_XRAY_FDRRECORDCONSUMER_H_
+#define LLVM_INCLUDE_LLVM_XRAY_FDRRECORDCONSUMER_H_
+
+#include "llvm/Support/Error.h"
+#include "llvm/XRay/FDRRecords.h"
+#include <algorithm>
+#include <memory>
+#include <vector>
+
+namespace llvm {
+namespace xray {
+
+class RecordConsumer {
+public:
+  virtual Error consume(std::unique_ptr<Record> R) = 0;
+  virtual ~RecordConsumer() = default;
+};
+
+class LogBuilderConsumer : public RecordConsumer {
+  std::vector<std::unique_ptr<Record>> &Records;
+
+public:
+  explicit LogBuilderConsumer(std::vector<std::unique_ptr<Record>> &R)
+      : RecordConsumer(), Records(R) {}
+
+  Error consume(std::unique_ptr<Record> R) override;
+};
+
+// A PipelineConsumer applies a set of visitors to every consumed Record, in the
+// order by which the visitors are added to the pipeline in the order of
+// appearance.
+class PipelineConsumer : public RecordConsumer {
+  std::vector<RecordVisitor *> Visitors;
+
+public:
+  PipelineConsumer(std::initializer_list<RecordVisitor *> V)
+      : RecordConsumer(), Visitors(V) {}
+
+  Error consume(std::unique_ptr<Record> R) override;
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_INCLUDE_LLVM_XRAY_FDRRECORDCONSUMER_H_

llvm/include/llvm/XRay/FDRRecordProducer.h

+//===- FDRRecordProducer.h - XRay FDR Mode Record Producer ----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_INCLUDE_LLVM_XRAY_FDRRECORDPRODUCER_H_
+#define LLVM_INCLUDE_LLVM_XRAY_FDRRECORDPRODUCER_H_
+
+#include "llvm/Support/Error.h"
+#include "llvm/XRay/FDRRecords.h"
+#include "llvm/XRay/XRayRecord.h"
+#include <memory>
+
+namespace llvm {
+namespace xray {
+
+class RecordProducer {
+public:
+  /// All producer implementations must yield either an Error or a non-nullptr
+  /// unique_ptr<Record>.
+  virtual Expected<std::unique_ptr<Record>> produce() = 0;
+  virtual ~RecordProducer() = default;
+};
+
+class FileBasedRecordProducer : public RecordProducer {
+  const XRayFileHeader &Header;
+  DataExtractor &E;
+  uint32_t &OffsetPtr;
+
+public:
+  FileBasedRecordProducer(const XRayFileHeader &FH, DataExtractor &DE,
+                          uint32_t &OP)
+      : Header(FH), E(DE), OffsetPtr(OP) {}
+
+  /// This producer encapsulates the logic for loading a File-backed
+  /// RecordProducer hidden behind a DataExtractor.
+  Expected<std::unique_ptr<Record>> produce() override;
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_INCLUDE_LLVM_XRAY_FDRRECORDPRODUCER_H_

llvm/include/llvm/XRay/FDRRecords.h

+//===- FDRRecords.h - XRay Flight Data Recorder Mode Records --------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Define types and operations on these types that represent the different kinds
+// of records we encounter in XRay flight data recorder mode traces.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIB_XRAY_FDRRECORDS_H_
+#define LLVM_LIB_XRAY_FDRRECORDS_H_
+
+#include "llvm/Support/DataExtractor.h"
+#include "llvm/Support/Error.h"
+#include "llvm/XRay/XRayRecord.h"
+#include <cstdint>
+
+namespace llvm {
+namespace xray {
+
+class RecordVisitor;
+class RecordInitializer;
+
+class Record {
+protected:
+  enum class Type {
+    Unknown,
+    Function,
+    Metadata,
+  };
+
+public:
+  Record(const Record &) = delete;
+  Record(Record &&) = delete;
+  Record &operator=(const Record &) = delete;
+  Record &operator=(Record &&) = delete;
+  Record() = default;
+
+  virtual Type type() const = 0;
+
+  // Each Record should be able to apply an abstract visitor, and choose the
+  // appropriate function in the visitor to invoke, given its own type.
+  virtual Error apply(RecordVisitor &V) = 0;
+
+  virtual ~Record() = default;
+};
+
+class MetadataRecord : public Record {
+protected:
+  static constexpr int kMetadataBodySize = 15;
+  friend class RecordInitializer;
+
+public:
+  enum class MetadataType : unsigned {
+    Unknown,
+    BufferExtents,
+    WallClockTime,
+    NewCPUId,
+    TSCWrap,
+    CustomEvent,
+    CallArg,
+    PIDEntry,
+    NewBuffer,
+    EndOfBuffer,
+  };
+
+  Type type() const override { return Type::Metadata; }
+
+  // All metadata records must know to provide the type of their open
+  // metadata record.
+  virtual MetadataType metadataType() const = 0;
+
+  virtual ~MetadataRecord() = default;
+};
+
+// What follows are specific Metadata record types which encapsulate the
+// information associated with specific metadata record types in an FDR mode
+// log.
+class BufferExtents : public MetadataRecord {
+  uint64_t Size = 0;
+  friend class RecordInitializer;
+
+public:
+  BufferExtents() = default;
+  explicit BufferExtents(uint64_t S) : MetadataRecord(), Size(S) {}
+
+  MetadataType metadataType() const override {
+    return MetadataType::BufferExtents;
+  }
+
+  uint64_t size() const { return Size; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class WallclockRecord : public MetadataRecord {
+  uint64_t Seconds = 0;
+  uint32_t Nanos = 0;
+  friend class RecordInitializer;
+
+public:
+  WallclockRecord() = default;
+  explicit WallclockRecord(uint64_t S, uint32_t N)
+      : MetadataRecord(), Seconds(S), Nanos(N) {}
+
+  MetadataType metadataType() const override {
+    return MetadataType::WallClockTime;
+  }
+
+  uint64_t seconds() const { return Seconds; }
+  uint32_t nanos() const { return Nanos; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class NewCPUIDRecord : public MetadataRecord {
+  uint16_t CPUId = 0;
+  friend class RecordInitializer;
+
+public:
+  NewCPUIDRecord() = default;
+  explicit NewCPUIDRecord(uint16_t C) : MetadataRecord(), CPUId(C) {}
+
+  MetadataType metadataType() const override { return MetadataType::NewCPUId; }
+
+  uint16_t cpuid() const { return CPUId; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class TSCWrapRecord : public MetadataRecord {
+  uint64_t BaseTSC = 0;
+  friend class RecordInitializer;
+
+public:
+  TSCWrapRecord() = default;
+  explicit TSCWrapRecord(uint64_t B) : MetadataRecord(), BaseTSC(B) {}
+
+  MetadataType metadataType() const override { return MetadataType::TSCWrap; }
+
+  uint64_t tsc() const { return BaseTSC; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class CustomEventRecord : public MetadataRecord {
+  uint64_t TSC = 0;
+  int32_t Size = 0;
+  std::string Data{};
+  friend class RecordInitializer;
+
+public:
+  CustomEventRecord() = default;
+  explicit CustomEventRecord(uint64_t S, std::string D)
+      : MetadataRecord(), Size(S), Data(std::move(D)) {}
+
+  MetadataType metadataType() const override {
+    return MetadataType::CustomEvent;
+  }
+
+  int32_t size() const { return Size; }
+  uint64_t tsc() const { return TSC; }
+  StringRef data() const { return Data; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class CallArgRecord : public MetadataRecord {
+  uint64_t Arg;
+  friend class RecordInitializer;
+
+public:
+  CallArgRecord() = default;
+  explicit CallArgRecord(uint64_t A) : MetadataRecord(), Arg(A) {}
+
+  MetadataType metadataType() const override { return MetadataType::CallArg; }
+
+  uint64_t arg() const { return Arg; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class PIDRecord : public MetadataRecord {
+  uint64_t PID = 0;
+  friend class RecordInitializer;
+
+public:
+  PIDRecord() = default;
+  explicit PIDRecord(uint64_t P) : MetadataRecord(), PID(P) {}
+
+  MetadataType metadataType() const override { return MetadataType::PIDEntry; }
+
+  uint64_t pid() const { return PID; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class NewBufferRecord : public MetadataRecord {
+  int32_t TID = 0;
+  friend class RecordInitializer;
+
+public:
+  NewBufferRecord() = default;
+  explicit NewBufferRecord(int32_t T) : MetadataRecord(), TID(T) {}
+
+  MetadataType metadataType() const override { return MetadataType::NewBuffer; }
+
+  int32_t tid() const { return TID; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class EndBufferRecord : public MetadataRecord {
+public:
+  EndBufferRecord() = default;
+
+  MetadataType metadataType() const override {
+    return MetadataType::EndOfBuffer;
+  }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class FunctionRecord : public Record {
+  RecordTypes Kind;
+  int32_t FuncId;
+  uint32_t Delta;
+  friend class RecordInitializer;
+
+  static constexpr unsigned kFunctionRecordSize = 8;
+
+public:
+  FunctionRecord() = default;
+  explicit FunctionRecord(RecordTypes K, int32_t F, uint32_t D)
+      : Record(), Kind(K), FuncId(F), Delta(D) {}
+
+  Type type() const override { return Type::Function; }
+
+  // A function record is a concrete record type which has a number of common
+  // properties.
+  RecordTypes recordType() const { return Kind; }
+  int32_t functionId() const { return FuncId; }
+  uint64_t delta() const { return Delta; }
+
+  Error apply(RecordVisitor &V) override;
+};
+
+class RecordVisitor {
+public:
+  virtual ~RecordVisitor() = default;
+
+  // Support all specific kinds of records:
+  virtual Error visit(BufferExtents &) = 0;
+  virtual Error visit(WallclockRecord &) = 0;
+  virtual Error visit(NewCPUIDRecord &) = 0;
+  virtual Error visit(TSCWrapRecord &) = 0;
+  virtual Error visit(CustomEventRecord &) = 0;
+  virtual Error visit(CallArgRecord &) = 0;
+  virtual Error visit(PIDRecord &) = 0;
+  virtual Error visit(NewBufferRecord &) = 0;
+  virtual Error visit(EndBufferRecord &) = 0;
+  virtual Error visit(FunctionRecord &) = 0;
+};
+
+class RecordInitializer : public RecordVisitor {
+  DataExtractor &E;
+  uint32_t &OffsetPtr;
+
+public:
+  explicit RecordInitializer(DataExtractor &DE, uint32_t &OP)
+      : RecordVisitor(), E(DE), OffsetPtr(OP) {}
+
+  Error visit(BufferExtents &) override;
+  Error visit(WallclockRecord &) override;
+  Error visit(NewCPUIDRecord &) override;
+  Error visit(TSCWrapRecord &) override;
+  Error visit(CustomEventRecord &) override;
+  Error visit(CallArgRecord &) override;
+  Error visit(PIDRecord &) override;
+  Error visit(NewBufferRecord &) override;
+  Error visit(EndBufferRecord &) override;
+  Error visit(FunctionRecord &) override;
+};
+
+/// The LogBuilder class allows for creating ad-hoc collections of records
+/// through the `add<...>(...)` function. An example use of this API is in
+/// crafting arbitrary sequences of records:
+///
+///   auto Records = LogBuilder()
+///       .add<BufferExtents>(256)
+///       .add<NewBufferRecord>(1)
+///       .consume();
+///
+class LogBuilder {
+  std::vector<std::unique_ptr<Record>> Records;
+
+public:
+  template <class R, class... T> LogBuilder &add(T &&... A) {
+    Records.emplace_back(new R(std::forward<T>(A)...));
+    return *this;
+  }
+
+  std::vector<std::unique_ptr<Record>> consume() { return std::move(Records); }
+};
+
+class FDRTrace {
+  XRayFileHeader Header;
+  std::vector<std::unique_ptr<Record>> Records;
+
+public:
+  /// Loads FDR records raw from a @DataExtractor, starting from @OffsetPtr.
+  Error load(DataExtractor &E, uint32_t &OffsetPtr);
+
+  /// Applies a Visitor onto all the records loaded so far.
+  Error apply(RecordVisitor &V) {
+    for (auto &R : Records)
+      if (auto E = R->apply(V))
+        return E;
+    return Error::success();
+  }
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_LIB_XRAY_FDRRECORDS_H_

llvm/include/llvm/XRay/FileHeaderReader.h

+//===- Common.h - XRay Trace Loading Common Utilities ---------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Common header for declarations of helper functions, used throughout the trace
+// loading library.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIB_XRAY_FILEHEADERREADER_H_
+#define LLVM_LIB_XRAY_FILEHEADERREADER_H_
+
+#include "llvm/Support/DataExtractor.h"
+#include "llvm/Support/Error.h"
+#include "llvm/XRay/XRayRecord.h"
+#include <cstdint>
+
+namespace llvm {
+namespace xray {
+
+/// Convenience function for loading the file header given a data extractor at a
+/// specified offset.
+Error readBinaryFormatHeader(DataExtractor &HeaderExtractor,
+                             uint32_t &OffsetPtr, XRayFileHeader &FileHeader);
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_LIB_XRAY_FILEHEADERREADER_H_

llvm/include/llvm/XRay/RecordPrinter.h

+//===- RecordPrinter.h - FDR Record Printer -------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// An implementation of the RecordVisitor which prints an individual record's
+// data in an adhoc format, suitable for human inspection.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_INCLUDE_LLVM_XRAY_RECORDPRINTER_H_
+#define LLVM_INCLUDE_LLVM_XRAY_RECORDPRINTER_H_
+
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/XRay/FDRRecords.h"
+
+namespace llvm {
+namespace xray {
+
+class RecordPrinter : public RecordVisitor {
+  raw_ostream &OS;
+  std::string Delim;
+
+public:
+  explicit RecordPrinter(raw_ostream &O, std::string D)
+      : RecordVisitor(), OS(O), Delim(std::move(D)) {}
+
+  explicit RecordPrinter(raw_ostream &O) : RecordPrinter(O, ""){};
+
+  Error visit(BufferExtents &) override;
+  Error visit(WallclockRecord &) override;
+  Error visit(NewCPUIDRecord &) override;
+  Error visit(TSCWrapRecord &) override;
+  Error visit(CustomEventRecord &) override;
+  Error visit(CallArgRecord &) override;
+  Error visit(PIDRecord &) override;
+  Error visit(NewBufferRecord &) override;
+  Error visit(EndBufferRecord &) override;
+  Error visit(FunctionRecord &) override;
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_INCLUDE_LLVM_XRAY_RECORDPRINTER_H

llvm/lib/XRay/BlockIndexer.cpp

+//===- BlockIndexer.cpp - FDR Block Indexing VIsitor ----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// An implementation of the RecordVisitor which generates a mapping between a
+// thread and a range of records representing a block.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/BlockIndexer.h"
+
+namespace llvm {
+namespace xray {
+
+Error BlockIndexer::visit(BufferExtents &) {
+  if (CurrentState == State::ThreadIDFound) {
+    Index::iterator It;
+    std::tie(It, std::ignore) =
+        Indices.insert({{CurrentBlock.ProcessID, CurrentBlock.ThreadID}, {}});
+    It->second.push_back({CurrentBlock.ProcessID, CurrentBlock.ThreadID,
+                          std::move(CurrentBlock.Records)});
+    CurrentBlock.ProcessID = 0;
+    CurrentBlock.ThreadID = 0;
+    CurrentBlock.Records = {};
+  }
+  CurrentState = State::ExtentsFound;
+  return Error::success();
+}
+
+Error BlockIndexer::visit(WallclockRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a wall clock record before finding an "
+        "extents record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(NewCPUIDRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a CPU ID record before finding an "
+        "extents record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(TSCWrapRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a TSC wrap record before finding an "
+        "extents record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(CustomEventRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a custom event record before finding an "
+        "extents record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(CallArgRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a call arg record before finding an "
+        "extents record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+};
+
+Error BlockIndexer::visit(PIDRecord &R) {
+  if (CurrentState == State::SeekExtents)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a PID record before finding an "
+        "extents record.");
+  CurrentBlock.ProcessID = R.pid();
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(NewBufferRecord &R) {
+  CurrentState = State::ThreadIDFound;
+  CurrentBlock.ThreadID = R.tid();
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::visit(EndBufferRecord &R) {
+  if (CurrentState != State::ThreadIDFound)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found an end buffer record before finding a "
+        "new buffer record.");
+  CurrentBlock.Records.push_back(&R);
+  CurrentState = State::SeekExtents;
+  return Error::success();
+}
+
+Error BlockIndexer::visit(FunctionRecord &R) {
+  if (CurrentState != State::ThreadIDFound)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "Malformed record order, found a function record before finding a new "
+        "buffer record.");
+  CurrentBlock.Records.push_back(&R);
+  return Error::success();
+}
+
+Error BlockIndexer::flush() {
+  Index::iterator It;
+  std::tie(It, std::ignore) =
+      Indices.insert({{CurrentBlock.ProcessID, CurrentBlock.ThreadID}, {}});
+  It->second.push_back({CurrentBlock.ProcessID, CurrentBlock.ThreadID,
+                        std::move(CurrentBlock.Records)});
+  CurrentBlock.ProcessID = 0;
+  CurrentBlock.ThreadID = 0;
+  CurrentBlock.Records = {};
+  CurrentState = State::SeekExtents;
+  return Error::success();
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/BlockPrinter.cpp

+//===- BlockPrinter.cpp - FDR Block Pretty Printer Implementation --------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/BlockPrinter.h"
+
+namespace llvm {
+namespace xray {
+
+Error BlockPrinter::visit(BufferExtents &R) {
+  OS << "\n[New Block]\n";
+  CurrentState = State::Preamble;
+  return RP.visit(R);
+}
+
+// Preamble printing.
+Error BlockPrinter::visit(NewBufferRecord &R) {
+  if (CurrentState == State::Start)
+    OS << "\n[New Block]\n";
+
+  OS << "Preamble: \n";
+  CurrentState = State::Preamble;
+  return RP.visit(R);
+}
+
+Error BlockPrinter::visit(WallclockRecord &R) {
+  OS << "\n";
+  CurrentState = State::Preamble;
+  return RP.visit(R);
+}
+
+Error BlockPrinter::visit(PIDRecord &R) {
+  OS << "\n";
+  CurrentState = State::Preamble;
+  return RP.visit(R);
+}
+
+// Metadata printing.
+Error BlockPrinter::visit(NewCPUIDRecord &R) {
+  if (CurrentState == State::Preamble)
+    OS << "\nBody:\n";
+  if (CurrentState == State::Function)
+    OS << "\nMetadata: ";
+  CurrentState = State::Metadata;
+  OS << " ";
+  auto E = RP.visit(R);
+  OS << ";";
+  return E;
+}
+
+Error BlockPrinter::visit(TSCWrapRecord &R) {
+  if (CurrentState == State::Function)
+    OS << "\nMetadata:";
+  CurrentState = State::Metadata;
+  OS << " ";
+  auto E = RP.visit(R);
+  OS << ";";
+  return E;
+}
+
+// Custom events will be rendered like "function" events.
+Error BlockPrinter::visit(CustomEventRecord &R) {
+  if (CurrentState == State::Metadata)
+    OS << "\n";
+  CurrentState = State::CustomEvent;
+  OS << "*  ";
+  auto E = RP.visit(R);
+  OS << ";\n";
+  return E;
+}
+
+// Function call printing.
+Error BlockPrinter::visit(FunctionRecord &R) {
+  if (CurrentState == State::Metadata)
+    OS << "\n";
+  CurrentState = State::Function;
+  OS << "-  ";
+  auto E = RP.visit(R);
+  OS << ";\n";
+  return E;
+}
+
+Error BlockPrinter::visit(CallArgRecord &R) {
+  CurrentState = State::Arg;
+  OS << " : ";
+  auto E = RP.visit(R);
+  OS << ";\n";
+  return E;
+}
+
+Error BlockPrinter::visit(EndBufferRecord &R) {
+    CurrentState = State::End;
+    OS << " *** ";
+    auto E = RP.visit(R);
+    OS << " ***\n";
+    return E;
+}
+
+} // namespace xray
+} // namespace llvm
 No newline at end of file

llvm/lib/XRay/BlockVerifier.cpp

+//===- BlockVerifier.cpp - FDR Block Verifier -----------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/BlockVerifier.h"
+#include "llvm/Support/Error.h"
+
+namespace llvm {
+namespace xray {
+
+StringRef BlockVerifier::recordToString(BlockVerifier::State R) {
+  switch (R) {
+  case State::BufferExtents:
+    return "BufferExtents";
+  case State::NewBuffer:
+    return "NewBuffer";
+  case State::WallClockTime:
+    return "WallClockTime";
+  case State::PIDEntry:
+    return "PIDEntry";
+  case State::NewCPUId:
+    return "NewCPUId";
+  case State::TSCWrap:
+    return "TSCWrap";
+  case State::CustomEvent:
+    return "CustomEvent";
+  case State::Function:
+    return "Function";
+  case State::CallArg:
+    return "CallArg";
+  case State::EndOfBuffer:
+    return "EndOfBuffer";
+  case State::StateMax:
+  case State::Unknown:
+    return "Unknown";
+  }
+}
+
+Error BlockVerifier::transition(State To) {
+  using ToSet = std::bitset<number(State::StateMax)>;
+  static constexpr std::array<const std::tuple<State, ToSet>,
+                              number(State::StateMax)>
+      TransitionTable{{
+
+          // Unknown -> BufferExtents | NewBuffer
+          {State::Unknown,
+           {mask(State::BufferExtents) | mask(State::NewBuffer)}},
+
+          // BufferExtents -> NewBuffer
+          {State::BufferExtents, {mask(State::NewBuffer)}},
+
+          // NewBuffer -> WallClockTime
+          {State::NewBuffer, {mask(State::WallClockTime)}},
+
+          // WallClockTime -> PIDEntry | NewCPUId
+          {State::WallClockTime,
+           {mask(State::PIDEntry) | mask(State::NewCPUId)}},
+
+          // PIDEntry -> NewCPUId
+          {State::PIDEntry, {mask(State::NewCPUId)}},
+
+          // NewCPUId ->
+          //     NewCPUId | TSCWrap | CustomEvent | Function | EndOfBuffer
+          {State::NewCPUId,
+           {mask(State::NewCPUId) | mask(State::TSCWrap) |
+            mask(State::CustomEvent) | mask(State::Function) |
+            mask(State::EndOfBuffer)}},
+
+          // TSCWrap ->
+          //     TSCWrap | NewCPUId | CustomEvent | Function | EndOfBuffer
+          {State::TSCWrap,
+           {mask(State::TSCWrap) | mask(State::NewCPUId) |
+            mask(State::CustomEvent) | mask(State::Function) |
+            mask(State::EndOfBuffer)}},
+
+          // CustomEvent ->
+          //     CustomEvent | NewCPUId | TSCWrap | Function | EndOfBuffer
+          {State::CustomEvent,
+           {mask(State::CustomEvent) | mask(State::TSCWrap) |
+            mask(State::NewCPUId) | mask(State::Function) |
+            mask(State::EndOfBuffer)}},
+
+          // Function ->
+          //     Function | NewCPUId | TSCWrap | CustomEvent | CallArg |
+          //     EndOfBuffer
+          {State::Function,
+           {mask(State::Function) | mask(State::TSCWrap) |
+            mask(State::NewCPUId) | mask(State::CustomEvent) |
+            mask(State::CallArg) | mask(State::EndOfBuffer)}},
+
+          // CallArg ->
+          //     CallArg | NewCPUId | TSCWrap | CustomEvent | Function |
+          //     EndOfBuffer
+          {State::CallArg,
+           {mask(State::CallArg) | mask(State::Function) |
+            mask(State::TSCWrap) | mask(State::NewCPUId) |
+            mask(State::CustomEvent) | mask(State::EndOfBuffer)}},
+
+          // EndOfBuffer -> <none>
+          {State::EndOfBuffer, {}}}};
+
+  if (CurrentRecord >= State::StateMax)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "BUG (BlockVerifier): Cannot find transition table entry for %s, "
+        "transitioning to %s.",
+        recordToString(CurrentRecord).data(), recordToString(To).data());
+
+  auto &Mapping = TransitionTable[number(CurrentRecord)];
+  auto &From = std::get<0>(Mapping);
+  auto &Destinations = std::get<1>(Mapping);
+  assert(From == CurrentRecord && "BUG: Wrong index for record mapping.");
+  if ((Destinations & ToSet(mask(To))) == 0)
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "BlockVerifier: Invalid transition from %s to %s.",
+        recordToString(CurrentRecord).data(), recordToString(To).data());
+
+  CurrentRecord = To;
+  return Error::success();
+}
+
+Error BlockVerifier::visit(BufferExtents &) {
+  return transition(State::BufferExtents);
+}
+
+Error BlockVerifier::visit(WallclockRecord &) {
+  return transition(State::WallClockTime);
+}
+
+Error BlockVerifier::visit(NewCPUIDRecord &) {
+  return transition(State::NewCPUId);
+}
+
+Error BlockVerifier::visit(TSCWrapRecord &) {
+  return transition(State::TSCWrap);
+}
+
+Error BlockVerifier::visit(CustomEventRecord &) {
+  return transition(State::CustomEvent);
+}
+
+Error BlockVerifier::visit(CallArgRecord &) {
+  return transition(State::CallArg);
+}
+
+Error BlockVerifier::visit(PIDRecord &) { return transition(State::PIDEntry); }
+
+Error BlockVerifier::visit(NewBufferRecord &) {
+  return transition(State::NewBuffer);
+}
+
+Error BlockVerifier::visit(EndBufferRecord &) {
+  return transition(State::EndOfBuffer);
+}
+
+Error BlockVerifier::visit(FunctionRecord &) {
+  return transition(State::Function);
+}
+
+Error BlockVerifier::verify() {
+  // The known terminal conditions are the following:
+  switch (CurrentRecord) {
+  case State::EndOfBuffer:
+  case State::NewCPUId:
+  case State::CustomEvent:
+  case State::Function:
+  case State::CallArg:
+  case State::TSCWrap:
+    return Error::success();
+  default:
+    return createStringError(
+        std::make_error_code(std::errc::executable_format_error),
+        "BlockVerifier: Invalid terminal condition %s, malformed block.",
+        recordToString(CurrentRecord).data());
+  }
+}
+
+void BlockVerifier::reset() { CurrentRecord = State::Unknown; }
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/CMakeLists.txt

 add_llvm_library(LLVMXRay
+  BlockIndexer.cpp
+  BlockPrinter.cpp
+  BlockVerifier.cpp
+  Common.cpp
+  FDRRecordProducer.cpp
+  FDRRecords.cpp
   InstrumentationMap.cpp
+  LogBuilderConsumer.cpp
+  RecordInitializer.cpp
+  RecordPrinter.cpp
   Trace.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/ADT
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/XRay
 
   DEPENDS
   LLVMSupport
   LLVMObject
 
   LINK_LIBS
   LLVMSupport
   LLVMObject
   )

llvm/lib/XRay/Common.cpp

+//===- Common.cpp - XRay Trace Loading Common Utilities -------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Definition of helper functions, used throughout the trace loading library.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/FileHeaderReader.h"
+
+namespace llvm {
+namespace xray {
+
+// Populates the FileHeader reference by reading the first 32 bytes of the file.
+Error readBinaryFormatHeader(DataExtractor &HeaderExtractor,
+                             uint32_t &OffsetPtr, XRayFileHeader &FileHeader) {
+  // FIXME: Maybe deduce whether the data is little or big-endian using some
+  // magic bytes in the beginning of the file?
+
+  // First 32 bytes of the file will always be the header. We assume a certain
+  // format here:
+  //
+  //   (2)   uint16 : version
+  //   (2)   uint16 : type
+  //   (4)   uint32 : bitfield
+  //   (8)   uint64 : cycle frequency
+  //   (16)  -      : padding
+
+  auto PreReadOffset = OffsetPtr;
+  FileHeader.Version = HeaderExtractor.getU16(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Failed reading version from file header at offset %d.", OffsetPtr);
+
+  PreReadOffset = OffsetPtr;
+  FileHeader.Type = HeaderExtractor.getU16(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Failed reading file type from file header at offset %d.", OffsetPtr);
+
+  PreReadOffset = OffsetPtr;
+  uint32_t Bitfield = HeaderExtractor.getU32(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Failed reading flag bits from file header at offset %d.", OffsetPtr);
+
+  FileHeader.ConstantTSC = Bitfield & 1uL;
+  FileHeader.NonstopTSC = Bitfield & 1uL << 1;
+  PreReadOffset = OffsetPtr;
+  FileHeader.CycleFrequency = HeaderExtractor.getU64(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Failed reading cycle frequency from file header at offset %d.",
+        OffsetPtr);
+
+  std::memcpy(&FileHeader.FreeFormData,
+              HeaderExtractor.getData().bytes_begin() + OffsetPtr, 16);
+
+  // Manually advance the offset pointer 16 bytes, after getting a raw memcpy
+  // from the underlying data.
+  OffsetPtr += 16;
+  if (FileHeader.Version != 1 && FileHeader.Version != 2 &&
+      FileHeader.Version != 3)
+    return createStringError(std::make_error_code(std::errc::invalid_argument),
+                             "Unsupported XRay file version: %d at offset %d",
+                             FileHeader.Version, OffsetPtr);
+  return Error::success();
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/FDRRecordProducer.cpp

+//===- FDRRecordProducer.cpp - XRay FDR Mode Record Producer --------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/FDRRecordProducer.h"
+#include "llvm/Support/DataExtractor.h"
+
+namespace llvm {
+namespace xray {
+
+Expected<std::unique_ptr<Record>> FileBasedRecordProducer::produce() {
+  // At the top level, we read one byte to determine the type of the record to
+  // create. This byte will comprise of the following bits:
+  //
+  //   - offset 0: A '1' indicates a metadata record, a '0' indicates a function
+  //     record.
+  //   - offsets 1-7: For metadata records, this will indicate the kind of
+  //     metadata record should be loaded.
+  //
+  // We read first byte, then create the appropriate type of record to consume
+  // the rest of the bytes.
+  auto PreReadOffset = OffsetPtr;
+  uint8_t FirstByte = E.getU8(&OffsetPtr);
+  std::unique_ptr<Record> R;
+
+  // For metadata records, handle especially here.
+  if (FirstByte & 0x01) {
+    // FIXME: Use a registry, and dispatch based on record types more
+    // automatically.
+    switch (FirstByte >> 1) {
+    case 0: // New buffer record.
+      R = llvm::make_unique<NewBufferRecord>();
+      break;
+    case 1: // End of buffer record.
+      if (Header.Version >= 2)
+        return createStringError(
+            std::make_error_code(std::errc::executable_format_error),
+            "End of buffer records are no longer supported starting version "
+            "2 of the log. Found at offset %d.",
+            PreReadOffset);
+      R = llvm::make_unique<EndBufferRecord>();
+      break;
+    case 2: // New CPU Id record.
+      R = llvm::make_unique<NewCPUIDRecord>();
+      break;
+    case 3: // TSC Wrap record.
+      R = llvm::make_unique<TSCWrapRecord>();
+      break;
+    case 4: // Wall Clock record.
+      R = llvm::make_unique<WallclockRecord>();
+      break;
+    case 5: // Custom event marker and record.
+      R = llvm::make_unique<CustomEventRecord>();
+      break;
+    case 6: // Call argument record.
+      R = llvm::make_unique<CallArgRecord>();
+      break;
+    case 7: // Buffer extents record.
+      R = llvm::make_unique<BufferExtents>();
+      break;
+    case 9: // PID record.
+      R = llvm::make_unique<PIDRecord>();
+      break;
+    default: // Unknown record.
+      return createStringError(
+          std::make_error_code(std::errc::executable_format_error),
+          "Encountered an unsupported metadata record (%d) at offset %d.",
+          FirstByte >> 1, PreReadOffset);
+    }
+  } else {
+    R = llvm::make_unique<FunctionRecord>();
+  }
+  RecordInitializer RI(E, OffsetPtr);
+
+  if (auto Err = R->apply(RI))
+    return std::move(Err);
+
+  assert(R != nullptr);
+  return std::move(R);
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/FDRRecords.cpp

+//===- FDRRecords.cpp -  XRay Flight Data Recorder Mode Records -----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Define types and operations on these types that represent the different kinds
+// of records we encounter in XRay flight data recorder mode traces.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/FDRRecords.h"
+#include "llvm/XRay/FDRRecordConsumer.h"
+#include "llvm/XRay/FDRRecordProducer.h"
+#include "llvm/XRay/FileHeaderReader.h"
+
+namespace llvm {
+namespace xray {
+
+Error BufferExtents::apply(RecordVisitor &V) { return V.visit(*this); }
+Error WallclockRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error NewCPUIDRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error TSCWrapRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error CustomEventRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error CallArgRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error PIDRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error NewBufferRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error EndBufferRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+Error FunctionRecord::apply(RecordVisitor &V) { return V.visit(*this); }
+
+Error FDRTrace::load(DataExtractor &E, uint32_t &OffsetPtr) {
+  // First thing we do is load the file header from the extractor.
+  if (auto Err = readBinaryFormatHeader(E, OffsetPtr, Header))
+    return Err;
+
+  FileBasedRecordProducer Producer(Header, E, OffsetPtr);
+  LogBuilderConsumer Consumer(Records);
+  while (E.isValidOffsetForDataOfSize(OffsetPtr, 1)) {
+    auto R = Producer.produce();
+    if (!R)
+      return R.takeError();
+    if (auto E = Consumer.consume(std::move(R.get())))
+      return E;
+  }
+  return Error::success();
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/LogBuilderConsumer.cpp

+//===- FDRRecordConsumer.h - XRay Flight Data Recorder Mode Records -------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/FDRRecordConsumer.h"
+
+namespace llvm {
+namespace xray {
+
+Error LogBuilderConsumer::consume(std::unique_ptr<Record> R) {
+  if (!R)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Must not call RecordConsumer::consume() with a null pointer.");
+  Records.push_back(std::move(R));
+  return Error::success();
+}
+
+Error PipelineConsumer::consume(std::unique_ptr<Record> R) {
+  if (!R)
+    return createStringError(
+        std::make_error_code(std::errc::invalid_argument),
+        "Must not call RecordConsumer::consume() with a null pointer.");
+
+  // We apply all of the visitors in order, and concatenate errors
+  // appropriately.
+  Error Result = Error::success();
+  for (auto *V : Visitors)
+    Result = joinErrors(std::move(Result), R->apply(*V));
+  return Result;
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/RecordInitializer.cpp

+//===- FDRRecordProducer.cpp - XRay FDR Mode Record Producer --------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/FDRRecords.h"
+
+namespace llvm {
+namespace xray {
+
+Error RecordInitializer::visit(BufferExtents &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr, sizeof(uint64_t)))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a buffer extent (%d).",
+                             OffsetPtr);
+
+  auto PreReadOffset = OffsetPtr;
+  R.Size = E.getU64(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read buffer extent at offset %d.",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(WallclockRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a wallclock record (%d).",
+                             OffsetPtr);
+  auto BeginOffset = OffsetPtr;
+  auto PreReadOffset = OffsetPtr;
+  R.Seconds = E.getU64(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::bad_message),
+        "Cannot read wall clock 'seconds' field at offset %d.", OffsetPtr);
+
+  PreReadOffset = OffsetPtr;
+  R.Nanos = E.getU32(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(
+        std::make_error_code(std::errc::bad_message),
+        "Cannot read wall clock 'nanos' field at offset %d.", OffsetPtr);
+
+  // Align to metadata record size boundary.
+  assert(OffsetPtr - BeginOffset <= MetadataRecord::kMetadataBodySize);
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(NewCPUIDRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a new cpu id record (%d).",
+                             OffsetPtr);
+  auto PreReadOffset = OffsetPtr;
+  R.CPUId = E.getU16(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read CPU id at offset %d.", OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(TSCWrapRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a new TSC wrap record (%d).",
+                             OffsetPtr);
+
+  auto PreReadOffset = OffsetPtr;
+  R.BaseTSC = E.getU64(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read TSC wrap record at offset %d.",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(CustomEventRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a custom event record (%d).",
+                             OffsetPtr);
+
+  auto BeginOffset = OffsetPtr;
+  auto PreReadOffset = OffsetPtr;
+  R.Size = E.getSigned(&OffsetPtr, sizeof(int32_t));
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(
+        std::make_error_code(std::errc::bad_message),
+        "Cannot read a custom event record size field offset %d.", OffsetPtr);
+
+  PreReadOffset = OffsetPtr;
+  R.TSC = E.getU64(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(
+        std::make_error_code(std::errc::bad_message),
+        "Cannot read a custom event TSC field at offset %d.", OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
+
+  // Next we read in a fixed chunk of data from the given offset.
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr, R.Size))
+    return createStringError(
+        std::make_error_code(std::errc::bad_address),
+        "Cannot read %d bytes of custom event data from offset %d.", R.Size,
+        OffsetPtr);
+
+  std::vector<uint8_t> Buffer;
+  Buffer.reserve(R.Size);
+  if (E.getU8(&OffsetPtr, Buffer.data(), R.Size) != Buffer.data())
+    return createStringError(
+        std::make_error_code(std::errc::bad_message),
+        "Failed reading data into buffer of size %d at offset %d.", R.Size,
+        OffsetPtr);
+  R.Data.append(Buffer.begin(), Buffer.end());
+  return Error::success();
+}
+
+Error RecordInitializer::visit(CallArgRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a call argument record (%d).",
+                             OffsetPtr);
+
+  auto PreReadOffset = OffsetPtr;
+  R.Arg = E.getU64(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read a call arg record at offset %d.",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(PIDRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a process ID record (%d).",
+                             OffsetPtr);
+
+  auto PreReadOffset = OffsetPtr;
+  R.PID = E.getU64(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read a process ID record at offset %d.",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(NewBufferRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a new buffer record (%d).",
+                             OffsetPtr);
+
+  auto PreReadOffset = OffsetPtr;
+  R.TID = E.getSigned(&OffsetPtr, sizeof(int32_t));
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Cannot read a new buffer record at offset %d.",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
+  return Error::success();
+}
+
+Error RecordInitializer::visit(EndBufferRecord &R) {
+  if (!E.isValidOffsetForDataOfSize(OffsetPtr,
+                                    MetadataRecord::kMetadataBodySize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for an end-of-buffer record (%d).",
+                             OffsetPtr);
+
+  OffsetPtr += MetadataRecord::kMetadataBodySize;
+  return Error::success();
+}
+
+Error RecordInitializer::visit(FunctionRecord &R) {
+  // For function records, we need to retreat one byte back to read a full
+  // unsigned 32-bit value. The first four bytes will have the following
+  // layout:
+  //
+  //   bit  0     : function record indicator (must be 0)
+  //   bits 1..3  : function record type
+  //   bits 4..32 : function id
+  //
+  if (OffsetPtr == 0 || !E.isValidOffsetForDataOfSize(
+                            --OffsetPtr, FunctionRecord::kFunctionRecordSize))
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Invalid offset for a function record (%d).",
+                             OffsetPtr);
+
+  auto BeginOffset = OffsetPtr;
+  auto PreReadOffset = BeginOffset;
+  uint32_t Buffer = E.getU32(&OffsetPtr);
+  if (PreReadOffset == OffsetPtr)
+    return createStringError(std::make_error_code(std::errc::bad_address),
+                             "Cannot read function id field from offset %d.",
+                             OffsetPtr);
+  unsigned FunctionType = (Buffer >> 1) & 0x07;
+  switch (FunctionType) {
+  case static_cast<unsigned>(RecordTypes::ENTER):
+  case static_cast<unsigned>(RecordTypes::ENTER_ARG):
+  case static_cast<unsigned>(RecordTypes::EXIT):
+  case static_cast<unsigned>(RecordTypes::TAIL_EXIT):
+    R.Kind = static_cast<RecordTypes>(FunctionType);
+    break;
+  default:
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Unknown function record type '%d' at offset %d.",
+                             FunctionType, BeginOffset);
+  }
+
+  R.FuncId = Buffer >> 4;
+  PreReadOffset = OffsetPtr;
+  R.Delta = E.getU32(&OffsetPtr);
+  if (OffsetPtr == PreReadOffset)
+    return createStringError(std::make_error_code(std::errc::bad_message),
+                             "Failed reading TSC delta from offset %d.",
+                             OffsetPtr);
+  assert(FunctionRecord::kFunctionRecordSize == (OffsetPtr - BeginOffset));
+  return Error::success();
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/RecordPrinter.cpp

+//===- RecordPrinter.cpp - FDR Record Printer -----------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "llvm/XRay/RecordPrinter.h"
+
+#include "llvm/Support/FormatVariadic.h"
+
+namespace llvm {
+namespace xray {
+
+Error RecordPrinter::visit(BufferExtents &R) {
+  OS << formatv("<Buffer: size = {0} bytes>", R.size()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(WallclockRecord &R) {
+  OS << formatv("<Wall Time: seconds = {0}.{1,0+6}>", R.seconds(), R.nanos())
+     << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(NewCPUIDRecord &R) {
+  OS << formatv("<CPU ID: {0}>", R.cpuid()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(TSCWrapRecord &R) {
+  OS << formatv("<TSC Wrap: base = {0}>", R.tsc()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(CustomEventRecord &R) {
+  OS << formatv("<Custom Event: tsc = {0}, size = {1}, data = '{2}'>", R.tsc(),
+                R.size(), R.data())
+     << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(CallArgRecord &R) {
+  OS << formatv("<Call Argument: data = {0} (hex = {0:x})>", R.arg()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(PIDRecord &R) {
+  OS << formatv("<PID: {0}>", R.pid()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(NewBufferRecord &R) {
+  OS << formatv("<Thread ID: {0}>", R.tid()) << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(EndBufferRecord &R) {
+  OS << "<End of Buffer>" << Delim;
+  return Error::success();
+}
+
+Error RecordPrinter::visit(FunctionRecord &R) {
+  // FIXME: Support symbolization here?
+  switch (R.recordType()) {
+  case RecordTypes::ENTER:
+    OS << formatv("<Function Enter: #{0} delta = +{0}>", R.functionId(),
+                  R.delta());
+    break;
+  case RecordTypes::ENTER_ARG:
+    OS << formatv("<Function Enter With Arg: #{0} delta = +{0}>",
+                  R.functionId(), R.delta());
+    break;
+  case RecordTypes::EXIT:
+    OS << formatv("<Function Exit: #{0} delta = +{0}>", R.functionId(),
+                  R.delta());
+    break;
+  case RecordTypes::TAIL_EXIT:
+    OS << formatv("<Function Tail Exit: #{0} delta = +{0}>", R.functionId(),
+                  R.delta());
+    break;
+  }
+  OS << Delim;
+  return Error::success();
+}
+
+} // namespace xray
+} // namespace llvm

llvm/lib/XRay/Trace.cpp

 // XRay log reader implementation.
 //
 //===----------------------------------------------------------------------===//
 #include "llvm/XRay/Trace.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/DataExtractor.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/XRay/FileHeaderReader.h"
 #include "llvm/XRay/YAMLXRayRecord.h"
 
 using namespace llvm;
 using namespace llvm::xray;
 using llvm::yaml::Input;
 
 namespace {
 using XRayRecordStorage =
     std::aligned_storage<sizeof(XRayRecord), alignof(XRayRecord)>::type;
 
 // This is the number of bytes in the "body" of a MetadataRecord in FDR Mode.
 // This already excludes the first byte, which indicates the type of metadata
 // record it is.
 constexpr auto kFDRMetadataBodySize = 15;
 
-// Populates the FileHeader reference by reading the first 32 bytes of the file.
-Error readBinaryFormatHeader(DataExtractor &HeaderExtractor,
-                             uint32_t &OffsetPtr, XRayFileHeader &FileHeader) {
-  // FIXME: Maybe deduce whether the data is little or big-endian using some
-  // magic bytes in the beginning of the file?
-
-  // First 32 bytes of the file will always be the header. We assume a certain
-  // format here:
-  //
-  //   (2)   uint16 : version
-  //   (2)   uint16 : type
-  //   (4)   uint32 : bitfield
-  //   (8)   uint64 : cycle frequency
-  //   (16)  -      : padding
-
-  auto PreReadOffset = OffsetPtr;
-  FileHeader.Version = HeaderExtractor.getU16(&OffsetPtr);
-  if (OffsetPtr == PreReadOffset)
-    return createStringError(
-        std::make_error_code(std::errc::invalid_argument),
-        "Failed reading version from file header at offset %d.", OffsetPtr);
-
-  PreReadOffset = OffsetPtr;
-  FileHeader.Type = HeaderExtractor.getU16(&OffsetPtr);
-  if (OffsetPtr == PreReadOffset)
-    return createStringError(
-        std::make_error_code(std::errc::invalid_argument),
-        "Failed reading file type from file header at offset %d.", OffsetPtr);
-
-  PreReadOffset = OffsetPtr;
-  uint32_t Bitfield = HeaderExtractor.getU32(&OffsetPtr);
-  if (OffsetPtr == PreReadOffset)
-    return createStringError(
-        std::make_error_code(std::errc::invalid_argument),
-        "Failed reading flag bits from file header at offset %d.", OffsetPtr);
-
-  FileHeader.ConstantTSC = Bitfield & 1uL;
-  FileHeader.NonstopTSC = Bitfield & 1uL << 1;
-  PreReadOffset = OffsetPtr;
-  FileHeader.CycleFrequency = HeaderExtractor.getU64(&OffsetPtr);
-  if (OffsetPtr == PreReadOffset)
-    return createStringError(
-        std::make_error_code(std::errc::invalid_argument),
-        "Failed reading cycle frequency from file header at offset %d.",
-        OffsetPtr);
-
-  std::memcpy(&FileHeader.FreeFormData,
-              HeaderExtractor.getData().bytes_begin() + OffsetPtr, 16);
-
-  // Manually advance the offset pointer 16 bytes, after getting a raw memcpy
-  // from the underlying data.
-  OffsetPtr += 16;
-  if (FileHeader.Version != 1 && FileHeader.Version != 2 &&
-      FileHeader.Version != 3)
-    return createStringError(std::make_error_code(std::errc::invalid_argument),
-                             "Unsupported XRay file version: %d at offset %d",
-                             FileHeader.Version, OffsetPtr);
-  return Error::success();
-}
-
 Error loadNaiveFormatLog(StringRef Data, XRayFileHeader &FileHeader,
                          std::vector<XRayRecord> &Records) {
   if (Data.size() < 32)
     return make_error<StringError>(
         "Not enough bytes for an XRay log.",
         std::make_error_code(std::errc::invalid_argument));
 
   if (Data.size() - 32 == 0 || Data.size() % 32 != 0)

 ///
 /// In Version 3, we make the following changes:
 ///
 /// ThreadBuffer: BufferExtents NewBuffer WallClockTime Pid NewCPUId
 ///               FunctionSequence
 /// EOB: *deprecated*
 Error loadFDRLog(StringRef Data, XRayFileHeader &FileHeader,
                  std::vector<XRayRecord> &Records) {
+
   if (Data.size() < 32)
     return make_error<StringError>(
         "Not enough bytes for an XRay log.",
         std::make_error_code(std::errc::invalid_argument));
 
   DataExtractor Reader(Data, true, 8);
   uint32_t OffsetPtr = 0;
 

llvm/tools/llvm-xray/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   DebugInfoDWARF
   Object
   Support
   Symbolize
   XRay
   )
 
 add_llvm_tool(llvm-xray
   func-id-helper.cpp
   llvm-xray.cpp
   xray-account.cpp
   xray-color-helper.cpp
   xray-converter.cpp
   xray-extract.cpp
+  xray-fdr-dump.cpp
   xray-graph-diff.cpp
   xray-graph.cpp
   xray-registry.cpp
   xray-stacks.cpp
   )

llvm/tools/llvm-xray/xray-fdr-dump.cpp

+//===- xray-fdr-dump.cpp: XRay FDR Trace Dump Tool ------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Implements the FDR trace dumping tool, using the libraries for handling FDR
+// mode traces specifically.
+//
+//===----------------------------------------------------------------------===//
+#include "xray-registry.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/XRay/BlockIndexer.h"
+#include "llvm/XRay/BlockPrinter.h"
+#include "llvm/XRay/BlockVerifier.h"
+#include "llvm/XRay/FDRRecordConsumer.h"
+#include "llvm/XRay/FDRRecordProducer.h"
+#include "llvm/XRay/FDRRecords.h"
+#include "llvm/XRay/FileHeaderReader.h"
+#include "llvm/XRay/RecordPrinter.h"
+
+using namespace llvm;
+using namespace xray;
+
+static cl::SubCommand Dump("fdr-dump", "FDR Trace Dump");
+static cl::opt<std::string> DumpInput(cl::Positional,
+                                      cl::desc("<xray fdr mode log>"),
+                                      cl::Required, cl::sub(Dump));
+static cl::opt<bool> DumpVerify("verify",
+                                cl::desc("verify structure of the log"),
+                                cl::init(false), cl::sub(Dump));
+
+static CommandRegistration Unused(&Dump, []() -> Error {
+  // Open the file provided.
+  int Fd;
+  if (auto EC = sys::fs::openFileForRead(DumpInput, Fd))
+    return createStringError(EC, "Cannot open file '%s' for read.",
+                             DumpInput.c_str());
+
+  uint64_t FileSize;
+  if (auto EC = sys::fs::file_size(DumpInput, FileSize))
+    return createStringError(EC, "Failed to get file size for '%s'.",
+                             DumpInput.c_str());
+
+  std::error_code EC;
+  sys::fs::mapped_file_region MappedFile(
+      Fd, sys::fs::mapped_file_region::mapmode::readonly, FileSize, 0, EC);
+
+  DataExtractor DE(StringRef(MappedFile.data(), MappedFile.size()), true, 8);
+  uint32_t OffsetPtr = 0;
+
+  if (DumpVerify) {
+    RecordPrinter RP(outs());
+    BlockPrinter BP(outs(), RP);
+    FDRTrace T;
+    if (auto E = T.load(DE, OffsetPtr)) {
+      // Show all the records we have so far.
+      if (auto PrintErrors = T.apply(RP))
+        return joinErrors(std::move(PrintErrors), std::move(E));
+      return E;
+    }
+
+    // Once we have a trace, we then index the blocks.
+    BlockIndexer::Index Index;
+    BlockIndexer BI(Index);
+    if (auto E = T.apply(BI))
+      return E;
+
+    if (auto E = BI.flush())
+      return E;
+
+    // Then we validate while printing each block.
+    BlockVerifier BV;
+    for (auto ProcessThreadBlocks : Index) {
+      auto &Blocks = ProcessThreadBlocks.second;
+      for (auto &B : Blocks) {
+        for (auto *R : B.Records) {
+          if (auto E = R->apply(BV))
+            return E;
+          if (auto E = R->apply(BP))
+            return E;
+        }
+        BV.reset();
+        BP.reset();
+      }
+    }
+    outs().flush();
+    return Error::success();
+  }
+
+  XRayFileHeader H;
+  if (auto E = readBinaryFormatHeader(DE, OffsetPtr, H))
+    return E;
+
+  FileBasedRecordProducer P(H, DE, OffsetPtr);
+  RecordPrinter RP(outs(), "\n");
+  PipelineConsumer C({&RP});
+  while (DE.isValidOffsetForDataOfSize(OffsetPtr, 1)) {
+    auto R = P.produce();
+    if (!R)
+      return R.takeError();
+    if (auto E = C.consume(std::move(R.get())))
+      return E;
+  }
+
+  return Error::success();
+});

llvm/unittests/XRay/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
+  XRay
   )
 
 add_llvm_unittest(XRayTests
   GraphTest.cpp
+  FDRRecordsTest.cpp
   )
 
 add_dependencies(XRayTests intrinsics_gen)

llvm/unittests/XRay/FDRRecordsTest.cpp

+//===- FDRRecords.cpp - Unit Tests for XRay FDR Record Loading ------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+#include "llvm/XRay/BlockIndexer.h"
+#include "llvm/XRay/BlockPrinter.h"
+#include "llvm/XRay/BlockVerifier.h"
+#include "llvm/XRay/FDRRecords.h"
+#include "llvm/XRay/RecordPrinter.h"
+
+namespace llvm {
+namespace xray {
+namespace {
+
+using ::testing::Eq;
+using ::testing::Not;
+
+TEST(XRayFDRTest, BuilderAndBlockIndexer) {
+  // We recreate a single block of valid records, then ensure that we find all
+  // of them belonging in the same index. We do this for three blocks, and
+  // ensure we find the same records in the blocks we deduce.
+  auto Block0 = LogBuilder()
+                    .add<BufferExtents>(100)
+                    .add<NewBufferRecord>(1)
+                    .add<WallclockRecord>(1, 1)
+                    .add<PIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+  auto Block1 = LogBuilder()
+                    .add<BufferExtents>(100)
+                    .add<NewBufferRecord>(1)
+                    .add<WallclockRecord>(1, 2)
+                    .add<PIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+  auto Block2 = LogBuilder()
+                    .add<BufferExtents>(100)
+                    .add<NewBufferRecord>(2)
+                    .add<WallclockRecord>(1, 3)
+                    .add<PIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+  BlockIndexer::Index Index;
+  BlockIndexer Indexer(Index);
+  for (auto B : {std::ref(Block0), std::ref(Block1), std::ref(Block2)}) {
+    for (auto &R : B.get())
+      ASSERT_FALSE(errorToBool(R->apply(Indexer)));
+    ASSERT_FALSE(errorToBool(Indexer.flush()));
+  }
+
+  // We have two threads worth of blocks.
+  ASSERT_THAT(Index.size(), Eq(2u));
+  auto T1Blocks = Index.find({1, 1});
+  ASSERT_THAT(T1Blocks, Not(Eq(Index.end())));
+  ASSERT_THAT(T1Blocks->second.size(), Eq(2u));
+  auto T2Blocks = Index.find({1, 2});
+  ASSERT_THAT(T2Blocks, Not(Eq(Index.end())));
+  ASSERT_THAT(T2Blocks->second.size(), Eq(1u));
+}
+
+TEST(XRayFDRTest, BuilderAndBlockVerifier) {
+  auto Block = LogBuilder()
+                   .add<BufferExtents>(48)
+                   .add<NewBufferRecord>(1)
+                   .add<WallclockRecord>(1, 1)
+                   .add<PIDRecord>(1)
+                   .add<NewCPUIDRecord>(1)
+                   .consume();
+  BlockVerifier Verifier;
+  for (auto &R : Block)
+    ASSERT_FALSE(errorToBool(R->apply(Verifier)));
+  ASSERT_FALSE(errorToBool(Verifier.verify()));
+}
+
+TEST(XRayFDRTest, IndexAndVerifyBlocks) {
+  auto Block0 = LogBuilder()
+                    .add<BufferExtents>(64)
+                    .add<NewBufferRecord>(1)
+                    .add<WallclockRecord>(1, 1)
+                    .add<PIDRecord>(1)
+                    .add<NewCPUIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+  auto Block1 = LogBuilder()
+                    .add<BufferExtents>(64)
+                    .add<NewBufferRecord>(1)
+                    .add<WallclockRecord>(1, 1)
+                    .add<PIDRecord>(1)
+                    .add<NewCPUIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+  auto Block2 = LogBuilder()
+                    .add<BufferExtents>(64)
+                    .add<NewBufferRecord>(1)
+                    .add<WallclockRecord>(1, 1)
+                    .add<PIDRecord>(1)
+                    .add<NewCPUIDRecord>(1)
+                    .add<FunctionRecord>(RecordTypes::ENTER, 1, 1)
+                    .add<FunctionRecord>(RecordTypes::EXIT, 1, 100)
+                    .consume();
+
+  // First, index the records in different blocks.
+  BlockIndexer::Index Index;
+  BlockIndexer Indexer(Index);
+  for (auto B : {std::ref(Block0), std::ref(Block1), std::ref(Block2)}) {
+    for (auto &R : B.get())
+      ASSERT_FALSE(errorToBool(R->apply(Indexer)));
+    ASSERT_FALSE(errorToBool(Indexer.flush()));
+  }
+
+  // Next, verify that each block is consistently defined.
+  BlockVerifier Verifier;
+  for (auto &ProcessThreadBlocks : Index) {
+    auto &Blocks = ProcessThreadBlocks.second;
+    for (auto &B : Blocks) {
+      for (auto *R : B.Records)
+        ASSERT_FALSE(errorToBool(R->apply(Verifier)));
+      ASSERT_FALSE(errorToBool(Verifier.verify()));
+      Verifier.reset();
+    }
+  }
+
+  // Then set up the printing mechanisms.
+  std::string Output;
+  raw_string_ostream OS(Output);
+  RecordPrinter RP(OS);
+  BlockPrinter BP(OS, RP);
+  for (auto &ProcessThreadBlocks : Index) {
+    auto &Blocks = ProcessThreadBlocks.second;
+    for (auto &B : Blocks) {
+      for (auto *R : B.Records)
+        ASSERT_FALSE(errorToBool(R->apply(BP)));
+      BP.reset();
+    }
+  }
+
+  OS.flush();
+  EXPECT_THAT(Output, Not(Eq("")));
+}
+
+} // namespace
+} // namespace xray
+} // namespace llvm