[XRay] Implement `llvm-xray convert` -- trace file conversion

[dberris]

Summary

This is the second part of a multi-part change to define additional
subcommands to the llvm-xray tool.

This change defines a conversion subcommand to take XRay log files, and
turns them from one format to another (binary or YAML). This currently
only supports the first version of the log file format, defined in the
compiler-rt runtime.

Depends on D21987.

[dberris]

• Use the command registry instead of hard-coding into main

[dberris]

• Use the command registry instead of hard-coding into main
• Rebase

[dberris]

• Use llvm::Error

[dberris]

• Rebase

[dberris]

Rebase

[dberris]

I'll be adding more tests and sample inputs for this tool, and probably use DataExtractor too for loading the records from the XRay log files. Please hold off on reviewing @dblaikie.

[dberris]

• Add test for converting a simple raw log to YAML

[dberris]

PTAL @dblaikie -- do you think using the DataExtractor here would be a good idea?

[dblaikie]

Before I get into all the details - this seems similar to the extract tool, except it deals with the log rather than the instrumentation map, right?

So pretty much all the same feedback as there (& I don't necessarily remember the original answers). We ended up with a one way conversion last time (to yaml, but not back again), should this one be different? If we only convert one way, is 'convert' the right name? Should be be more explicit about the difference between these two things, or just detect based on file magic & use one command name for both kinds of input files? (check if it's an ELF file then assume the user is trying to extract an instrumentation map, and otherwise assume the file's a log?)

tools/llvm-xray/func-id-helper.cc

41–43

We don't usually bother putting {} on single-line if statements.

50–54

Probably drop the {} here too (maybe even use the conditional operator, if you like)

tools/llvm-xray/xray-record.h

34

Is the alignas attribute supported on all the platforms/compilers we care about?

54

There's LLVM_PACKED you may need to use to make this portable to MSVC, by the looks of it.

(though I'm still intrinsically suspicious of splatting in/out of memory, and would be more comfortable seeing just 'obvious' code to read bytes and shift them into values, etc - but perhaps I'm in the minority/an outlier here and this sort of code is OK with everyone else)

[dblaikie]

A few of the inline comments from my last feedback are probably out of date/not relevant - I had them queued up from weeks/months ago.

That said - taking a guess at it: Yes, I'd use DataExtractor here over splatting/memcpying structs around for the same reasons.

[dberris]

In D24376#581421, @dblaikie wrote:

Before I get into all the details - this seems similar to the extract tool, except it deals with the log rather than the instrumentation map, right?

So pretty much all the same feedback as there (& I don't necessarily remember the original answers). We ended up with a one way conversion last time (to yaml, but not back again), should this one be different? If we only convert one way, is 'convert' the right name? Should be be more explicit about the difference between these two things, or just detect based on file magic & use one command name for both kinds of input files? (check if it's an ELF file then assume the user is trying to extract an instrumentation map, and otherwise assume the file's a log?)

The value of having a convert function is so that we can support other formats specifically for the xray log. Here are some future things we need to support:

• Convert from an old version of the log into a newer version.
• Convert from the XRay log format to another format. One of them is the Chrome Trace Viewer format, something that the Google Performance Tools (https://github.com/gperftools/gperftools) can consume, etc.

For now, the most convenient thing we can do is create YAML files. The important part here is the log file loading library, that we use in the accounting implementation (stacked on top of this change).

So the verb really is to "convert", as opposed to just "extract".

Does this make sense?

That said, I'll apply most of the changes in the 'extract' review here.

Cheers

[dberris]

• Add more tests, fix some bugs
• Use DataExtractor, refactor a bit to use Error properly, and use a simpler implementation of the log reader.

[dberris]

Ready for a look now @dblaikie, thanks in advance!

PS. We no longer support round-tripping in this implementation.

[dblaikie]

tools/llvm-xray/func-id-helper.cc

29–30

Fold ResOrErr into the if condition to reduce/match its scope to its use?

50–51

Fold variable into if condition.

(alternatively - and this applies to the other case of the same - consider a short-exit:

if (!ResOrErr) {
  handleAllErrors(...)
  return F.str();
}

I know this means duplicating the "Return F.str()" code - but allows the main non-error code to be unindented, can make it easier to follow (by keeping the error handling close to the error, too).)

54–58

Drop {} on single line blocks, probably

tools/llvm-xray/func-id-helper.h

40

This should be implicit? Or did you want it to disable the other special members? Any particular reason? (might be worth a comment if that's the case)

tools/llvm-xray/xray-converter.cc

122–125

If we need to support these formats for both input and output - I don't so much mind whether we support the null conversion/roundtrip. The point in the previous review was that we didn't need to handle binary output nor yaml input - but it sounds like you need all the different input and output modes here? (at the very least the native format will be a valid input (for conversion to non-native formats) and output (for upgrading) format, for example.

So, up to you - if supporting roundtripping is convenient/nice generalization, that's fine

127–130

Hmm - now I'm a bit confused given after my last comment.

If we don't support roundtripping, and we don't support binary output - then the only thing we support is binary input and YAML output, so why the switch a few lines below that chooses the input format between binary and yaml? Isn't the yaml reading case in that switch unreachable and thus the YAMLogLoader unused/untested?

Indeed, no test seems to exercise -i/-input-format?

tools/llvm-xray/xray-extract.cc

193–194 ↗

(On Diff #76528)

Roll variable into if condition

203–204 ↗

(On Diff #76528)

Guessing it's mapped rather than streamed because that's the API for DataExtractor (dealing with a buffer)? Or does it have other needs to be mapped?

(might be nice if it didn't have to be - but I suppose there's no strong reason/benefit to it... )

210 ↗

(On Diff #76528)

Might make sense to pass MappedFile.size() rather than fileSize, even though they're the same (& even maybe just having MappedFile have a way to access a StringRef of the range)

tools/llvm-xray/xray-log-reader.cc

88

I probably wouldn't bother with explicit scope here (I usually only use them if I need the dtor behavior) - but I appreciate the benefit to scoping the variables and don't mind it if you prefer it this way.

113

unneeded extra scope here?

tools/llvm-xray/xray-record.h

24

Guessing this doesn't need an alignment attribute anymore?

39

Is there an enum that should be used for this member?

[dberris]

• Address some comments
• Add FIXME to support binary conversion

[dberris]

Thanks @dblaikie, PTAL.

tools/llvm-xray/func-id-helper.cc

50–51

I like just limiting the scope of the variable, so I stuck with that. Also, now using sys::path::filename instead of manually futzing with the string.

tools/llvm-xray/xray-converter.cc

127–130

Right, I think the better message here is "not yet supported". When we need the functionality then it seems better to change this then.

tools/llvm-xray/xray-extract.cc

203–204 ↗

(On Diff #76528)

Yeah, the API for the DataExtractor deals with a buffer. We could be reading things in chunks at a time explicitly, but that seems unnecessary (unless mmapping is undesirable for other reasons).

tools/llvm-xray/xray-log-reader.cc

88

No strong preference, this was a remnant of attempting to use the same variable name for the extractor -- but there's really no good reason for the explicit scope. :)

[dberris]

Just in case it wasn't clear, this one's ready for another look @dblaikie.

[dblaikie]

tools/llvm-xray/func-id-helper.cc

50–51

I'll say, on reflection, that I kind of prefer keeping the diagnostic handling closer - rather than "if (success) { next bit of code } else { error handling for the if condition }"

but I'll leave that up to you - can get a feel for it and always change it later if it starts to get unwieldy

tools/llvm-xray/xray-converter.cc

127–130

Dead/untested code is a to be avoided - so if the YAMLLogLoader is unused/untested, it probably shouldn't be committed yet.

It's difficult to keep track of test coverage if code is committed without coverage - then when the code becomes live it's not obvious that it was previously untested, etc.

153

Spurious semicolon?

tools/llvm-xray/xray-extract.cc

203–204 ↗

(On Diff #76528)

Can be nice to be able to read streaming input (general good for uniformity - lots of tools accept '-' as the filename to read from stdin), etc. Not a strong requirement by any means.

tools/llvm-xray/xray-log-reader.cc

114

Could use 'emplace_back()' if you reckon that's more readable (I don't think it's less readable than {} at least).

tools/llvm-xray/xray-record.h

39

The comment "Usually either ENTER = 0 or EXIT = 1" is no longer needed, as it's implied by the type. (perhaps could be reworded, etc - "Identifies this as an enter or exit record" or somesuch if that's particularly useful - though the enum is only a few lines away)

[dberris]

• Implement conversion to v1 binary format

tools/llvm-xray/xray-converter.cc

127–130

Good point. I've re-implemented the binary output, and the round-tripping.

tools/llvm-xray/xray-extract.cc

203–204 ↗

(On Diff #76528)

Actually, now that I read this part of the code again, it's the YAML Input that requires a full document to be provided in its constructor.

http://llvm.org/docs/YamlIO.html#input

tools/llvm-xray/xray-record.h

39

Reworded to be short and sweet.

[dberris]

Ready for another look now.

[dblaikie]

(I was thinking more going the other direction: removing the dead code, rather than adding a use of it)

What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)

That's my mental model at least.

(this commit seems to add several .bin test files - but I don't see where those are used?)

tools/llvm-xray/xray-converter.cc

176–177

This seems like an odd construct - should Extractor not error-out in the case where ConvertInstrMap is empty?

[dberris]

• Address review comments

[dberris]

In D24376#602808, @dblaikie wrote:

(I was thinking more going the other direction: removing the dead code, rather than adding a use of it)

What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)

That's my mental model at least.

Other things going on concurrently (work on the FDR mode for the runtime library) is making me need to be able to turn versions of the binary log from one form to another. I've found that working with the binary versions is much more convenient from a tool perspective, and that this current simple format is more amenable to analysis than the condensed format I'm working on.

So effectively, version 2 of the log format is turning out to have both non-fixed-size records and more interleaving happening. This tool, that's able to take that complex format into something simpler/different doesn't make sense for generating just YAML files.

(this commit seems to add several .bin test files - but I don't see where those are used?)

Some of the tests are looking at symbolization of the function id's, and associating functions with debug info, etc. for coverage of the various modes by which the YAML output could be generated.

tools/llvm-xray/xray-converter.cc

176–177

Good point.

Technically, that is an error in the use of the extractor's constructor, hence it signalling an error. Think about it as if the construction of the extractor had thrown an exception -- and in this case, the appropriate action is to ignore that exception.

So this is saying, if the filename was empty, then the extractor couldn't be initialised and therefore we ignore the error. In the case that it wasn't empty, then we should emit the error, but still not cause the tool to fail in execution.

Changed both here and the extract sub-command where we check the filename explicitly to not be empty.

[dberris]

It seems Phabricator had been swallowing some of my comments (and not sending out email messages).

This is ready for another look @dblaikie.

[dblaikie]

In D24376#603715, @dberris wrote:

In D24376#602808, @dblaikie wrote:

(I was thinking more going the other direction: removing the dead code, rather than adding a use of it)

What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)

That's my mental model at least.

Other things going on concurrently (work on the FDR mode for the runtime library) is making me need to be able to turn versions of the binary log from one form to another. I've found that working with the binary versions is much more convenient from a tool perspective, and that this current simple format is more amenable to analysis than the condensed format I'm working on.

I'm not sure I'm following here - could you describe this in more detail?

My understanding was that tools would likely use these LLVM APIs for data handling (or they'd use YAML input) - so there wouldn't be any benefit to any particular format (except YAML) since these LLVM APIs could handle all the formats. This would make the binary format(s) fairly niche - just convenient for runtime emission, but not desirable for any other use.

Where have I gone wrong here?

So effectively, version 2 of the log format is turning out to have both non-fixed-size records and more interleaving happening. This tool, that's able to take that complex format into something simpler/different doesn't make sense for generating just YAML files.

(this commit seems to add several .bin test files - but I don't see where those are used?)

Some of the tests are looking at symbolization of the function id's, and associating functions with debug info, etc. for coverage of the various modes by which the YAML output could be generated.

Not quite following, more practically: which tests are these files used in?

[dberris]

In D24376#604850, @dblaikie wrote:

In D24376#603715, @dberris wrote:

In D24376#602808, @dblaikie wrote:

(I was thinking more going the other direction: removing the dead code, rather than adding a use of it)

What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)

That's my mental model at least.

Other things going on concurrently (work on the FDR mode for the runtime library) is making me need to be able to turn versions of the binary log from one form to another. I've found that working with the binary versions is much more convenient from a tool perspective, and that this current simple format is more amenable to analysis than the condensed format I'm working on.

I'm not sure I'm following here - could you describe this in more detail?

Sure, there's a patch I've uploaded recently that has a lot more about this different format (see D27038).

My understanding was that tools would likely use these LLVM APIs for data handling (or they'd use YAML input) - so there wouldn't be any benefit to any particular format (except YAML) since these LLVM APIs could handle all the formats. This would make the binary format(s) fairly niche - just convenient for runtime emission, but not desirable for any other use.

Where have I gone wrong here?

That's one half of the story.

The other half is the new log format (still binary) is not conducive to this simple approach of loading up records with full information. The FDR mode log format is substantially different, in that per-thread buffers are chunked up into fixed-sized blocks and each record might be a metadata record (16 bytes) or a function record (8 bytes). That log has all sorts of information consolidated in very different ways (e.g. the thread ID is stored once, as a metadata record at the beginning of a buffer, CPU id's only show up when the CPU is first encountered, we store TSC deltas instead of full TSCs in function records), and is not conducive to just "load in memory, read records one by one".

So then, the conversion tool ought to be able to convert the FDR mode logs into this simpler log format that we already support in this tool -- then let all the other tools upcoming (accounting, timeline visualisation, etc.) just use the simpler log format.

The closest analogy I can make for existing tools that do something similar would be 'dot' and friends.

So effectively, version 2 of the log format is turning out to have both non-fixed-size records and more interleaving happening. This tool, that's able to take that complex format into something simpler/different doesn't make sense for generating just YAML files.

(this commit seems to add several .bin test files - but I don't see where those are used?)

Some of the tests are looking at symbolization of the function id's, and associating functions with debug info, etc. for coverage of the various modes by which the YAML output could be generated.

Not quite following, more practically: which tests are these files used in?

convert-with-debug-syms.txt
convert-with-standalone-instrmap.txt
convert-with-yaml-instrmap.txt

[dblaikie]

In D24376#604864, @dberris wrote:

In D24376#604850, @dblaikie wrote:

In D24376#603715, @dberris wrote:

In D24376#602808, @dblaikie wrote:

(I was thinking more going the other direction: removing the dead code, rather than adding a use of it)

What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)

That's my mental model at least.

Other things going on concurrently (work on the FDR mode for the runtime library) is making me need to be able to turn versions of the binary log from one form to another. I've found that working with the binary versions is much more convenient from a tool perspective, and that this current simple format is more amenable to analysis than the condensed format I'm working on.

I'm not sure I'm following here - could you describe this in more detail?

Sure, there's a patch I've uploaded recently that has a lot more about this different format (see D27038).

My understanding was that tools would likely use these LLVM APIs for data handling (or they'd use YAML input) - so there wouldn't be any benefit to any particular format (except YAML) since these LLVM APIs could handle all the formats. This would make the binary format(s) fairly niche - just convenient for runtime emission, but not desirable for any other use.

Where have I gone wrong here?

That's one half of the story.

The other half is the new log format (still binary) is not conducive to this simple approach of loading up records with full information. The FDR mode log format is substantially different, in that per-thread buffers are chunked up into fixed-sized blocks and each record might be a metadata record (16 bytes) or a function record (8 bytes). That log has all sorts of information consolidated in very different ways (e.g. the thread ID is stored once, as a metadata record at the beginning of a buffer, CPU id's only show up when the CPU is first encountered, we store TSC deltas instead of full TSCs in function records), and is not conducive to just "load in memory, read records one by one".

So then, the conversion tool ought to be able to convert the FDR mode logs into this simpler log format that we already support in this tool -- then let all the other tools upcoming (accounting, timeline visualisation, etc.) just use the simpler log format.

I'm still a bit confused though - why not just convert to the yaml format at that point? (& why convert at all - if we're talking about other tools built on top of LLVM's library (tools built outside of that would presumably only rely on the YAML?) would use a common parser/reader API that could handle all formats, right? Rather than having the user run a conversion tool, then the tool they want to use)

The closest analogy I can make for existing tools that do something similar would be 'dot' and friends.

So effectively, version 2 of the log format is turning out to have both non-fixed-size records and more interleaving happening. This tool, that's able to take that complex format into something simpler/different doesn't make sense for generating just YAML files.

(this commit seems to add several .bin test files - but I don't see where those are used?)

Some of the tests are looking at symbolization of the function id's, and associating functions with debug info, etc. for coverage of the various modes by which the YAML output could be generated.

Not quite following, more practically: which tests are these files used in?

convert-with-debug-syms.txt
convert-with-standalone-instrmap.txt
convert-with-yaml-instrmap.txt

ah, hmm - I saw the .bin files (some of them) added in nthe last update - so I guess maybe they just hadn't been added previously, but the uses were already there. No worries.

[dberris]

In D24376#604995, @dblaikie wrote:

In D24376#604864, @dberris wrote:

The other half is the new log format (still binary) is not conducive to this simple approach of loading up records with full information. The FDR mode log format is substantially different, in that per-thread buffers are chunked up into fixed-sized blocks and each record might be a metadata record (16 bytes) or a function record (8 bytes). That log has all sorts of information consolidated in very different ways (e.g. the thread ID is stored once, as a metadata record at the beginning of a buffer, CPU id's only show up when the CPU is first encountered, we store TSC deltas instead of full TSCs in function records), and is not conducive to just "load in memory, read records one by one".

So then, the conversion tool ought to be able to convert the FDR mode logs into this simpler log format that we already support in this tool -- then let all the other tools upcoming (accounting, timeline visualisation, etc.) just use the simpler log format.

I'm still a bit confused though - why not just convert to the yaml format at that point? (& why convert at all - if we're talking about other tools built on top of LLVM's library (tools built outside of that would presumably only rely on the YAML?) would use a common parser/reader API that could handle all formats, right? Rather than having the user run a conversion tool, then the tool they want to use)

YAML is a little big and verbose -- the blow-up from the binary format to YAML is a factor of at least 10. In practical terms, dealing with the binary format in terms of storage and transport is a good option.

YAML is convenient for human consumption, and if users ever build tools that use scripting languages or other things already deal with YAML (or JSON, or some other text format) then that's the convenient path to take.

Maybe the solution here is to move the log reading part into something that's part of the LLVM library, but that still has the problem that reconstruction of a consistent timeline representing events on multiple threads is much more complex than one that just takes a linear "simple" binary file. Then the XRay log file that has non-fixed-size records and more complex encoding ought to be converted to the simplified form.

So the compromise/trade-off being made here is that we localize the knowledge of dealing with the different XRay binary log formats to this tool, that can turn it into "the simplest XRay log format", that can be read by simpler libraries for consumption. Given that, and the plan of being able to convert these XRay log files into other non-XRay specific formats (Chrome trace viewer format which is JSON, or the perftools utilities, etc.) is a huge convenience. This convenience is not only for testing purposes but also for building and integrating with other existing tools that deal with all sorts of traces.

Does this make sense?

[dberris]

Rebase

[dberris]

Rebase again

[dberris]

Squash local commits in the hopes of producing a simpler patch.

[dberris]

Ping? @dblaikie

[echristo]

So then, the conversion tool ought to be able to convert the FDR mode logs into this simpler log format that we already support in this tool -- then let all the other tools upcoming (accounting, timeline visualisation, etc.) just use the simpler log format.

I'm still a bit confused though - why not just convert to the yaml format at that point? (& why convert at all - if we're talking about other tools built on top of LLVM's library (tools built outside of that would presumably only rely on the YAML?) would use a common parser/reader API that could handle all formats, right? Rather than having the user run a conversion tool, then the tool they want to use)

YAML is a little big and verbose -- the blow-up from the binary format to YAML is a factor of at least 10. In practical terms, dealing with the binary format in terms of storage and transport is a good option.

I can agree with this.

YAML is convenient for human consumption, and if users ever build tools that use scripting languages or other things already deal with YAML (or JSON, or some other text format) then that's the convenient path to take.

Well, not entirely convenient. It would be nice to have a "dumping" output that produces an easy to consume by normal humans format that isn't the full yaml format for specifying.

Maybe the solution here is to move the log reading part into something that's part of the LLVM library, but that still has the problem that reconstruction of a consistent timeline representing events on multiple threads is much more complex than one that just takes a linear "simple" binary file. Then the XRay log file that has non-fixed-size records and more complex encoding ought to be converted to the simplified form.

I think that the format reading should definitely be a part of the llvm library and not the tool in particular. The tool should just wrap the library.

So the compromise/trade-off being made here is that we localize the knowledge of dealing with the different XRay binary log formats to this tool, that can turn it into "the simplest XRay log format", that can be read by simpler libraries for consumption. Given that, and the plan of being able to convert these XRay log files into other non-XRay specific formats (Chrome trace viewer format which is JSON, or the perftools utilities, etc.) is a huge convenience. This convenience is not only for testing purposes but also for building and integrating with other existing tools that deal with all sorts of traces.

Does this make sense?

Being able to convert seems useful in general. You want to be able to write testcases in something human readable, but you want the default output to be binary for compactness.

-eric

[dberris]

In D24376#634790, @echristo wrote:

I think that the format reading should definitely be a part of the llvm library and not the tool in particular. The tool should just wrap the library.

Cool, thanks Eric -- D28345 does just the log reading library which depends on this change landing.

So the compromise/trade-off being made here is that we localize the knowledge of dealing with the different XRay binary log formats to this tool, that can turn it into "the simplest XRay log format", that can be read by simpler libraries for consumption. Given that, and the plan of being able to convert these XRay log files into other non-XRay specific formats (Chrome trace viewer format which is JSON, or the perftools utilities, etc.) is a huge convenience. This convenience is not only for testing purposes but also for building and integrating with other existing tools that deal with all sorts of traces.

Does this make sense?

Being able to convert seems useful in general. You want to be able to write testcases in something human readable, but you want the default output to be binary for compactness.

Good point. So I think "conversion" is a bit more loaded than just "dump", where "dump" really just makes the representation more human-readable (for debugging, or just turning machine-readable form to human-readable form).

Will renaming this from 'convert' to 'dump' make that better?

-eric

[dblaikie]

Good point. So I think "conversion" is a bit more loaded than just "dump", where "dump" really just makes the representation more human-readable (for debugging, or just turning machine-readable form to human-readable form).

Will renaming this from 'convert' to 'dump' make that better?

I don't think that'd be accurate - this is doing format conversions between semantic preserving representations intended for machine consumption. Our dumping tools are generally intended only for human consumption (or really hacky machine consumption in scripts for quick work sometimes (eg: some scripts I'm writing using objdump/dwarfdump for doing some object size analysis)).

tools/llvm-xray/xray-extract.cc

220 ↗

(On Diff #80031)

push_back? (since you're specifying the type anyway, etc)

tools/llvm-xray/xray-log-reader.cc

166

I'd skip the 'Tmp' temporary, and put the result directly into 'Records' (because it seems simpler - but arguably could have slightly improved performance if 'Records' already has a large capacity that can be reused)

[dberris]

• Address review comments

test/tools/llvm-xray/X86/convert-roundtrip.yaml

+#RUN: llvm-xray convert %s -i=yaml -f=raw -o %t && llvm-xray convert %t -f=yaml -o - | FileCheck %s
+---
+header:
+  version: 1
+  type: 0
+  constant-tsc: true
+  nonstop-tsc: true
+  cycle-frequency: 2601000000
+records:
+  - { type: 0, func-id: 1, cpu: 1, thread: 111, kind: function-enter,
+    tsc: 10001 }
+  - { type: 0, func-id: 1, cpu: 1, thread: 111, kind: function-exit,
+    tsc: 10100 }
+...
+
+#CHECK:       ---
+#CHECK-NEXT:  header:
+#CHECK-NEXT:    version: 1
+#CHECK-NEXT:    type: 0
+#CHECK-NEXT:    constant-tsc: true
+#CHECK-NEXT:    nonstop-tsc: true
+#CHECK-NEXT:    cycle-frequency: 2601000000
+#CHECK-NEXT:  records:
+#CHECK-NEXT:    - { type: 0, func-id: 1, function: '1', cpu: 1, thread: 111, kind: function-enter,
+#CHECK-NEXT:      tsc: 10001 }
+#CHECK-NEXT:    - { type: 0, func-id: 1, function: '1', cpu: 1, thread: 111, kind: function-exit,
+#CHECK-NEXT:      tsc: 10100 }
+#CHECK-NEXT:  ...

tools/llvm-xray/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
+  DebugInfoDWARF
+  Object
   Support
-  Object)
+  Symbolize)
 
 set(LLVM_XRAY_TOOLS
+  func-id-helper.cc
+  xray-converter.cc
   xray-extract.cc
+  xray-extract.cc
+  xray-log-reader.cc
   xray-registry.cc)
 
 add_llvm_tool(llvm-xray llvm-xray.cc ${LLVM_XRAY_TOOLS})

tools/llvm-xray/func-id-helper.h

+//===- func-id-helper.h - XRay Function ID Conversion Helpers -------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Defines helper tools dealing with XRay-generated function ids.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TOOLS_LLVM_XRAY_FUNC_ID_HELPER_H
+#define LLVM_TOOLS_LLVM_XRAY_FUNC_ID_HELPER_H
+
+#include "llvm/DebugInfo/Symbolize/Symbolize.h"
+#include <unordered_map>
+
+namespace llvm {
+namespace xray {
+
+// This class consolidates common operations related to Function IDs.
+class FuncIdConversionHelper {
+public:
+  using FunctionAddressMap = std::unordered_map<int32_t, uint64_t>;
+
+private:
+  std::string BinaryInstrMap;
+  symbolize::LLVMSymbolizer &Symbolizer;
+  const FunctionAddressMap &FunctionAddresses;
+
+public:
+  FuncIdConversionHelper(std::string BinaryInstrMap,
+                         symbolize::LLVMSymbolizer &Symbolizer,
+                         const FunctionAddressMap &FunctionAddresses)
+      : BinaryInstrMap(std::move(BinaryInstrMap)), Symbolizer(Symbolizer),
+        FunctionAddresses(FunctionAddresses) {}
+
+  FuncIdConversionHelper(const FuncIdConversionHelper &) = default;
+
+  // Returns the symbol or a string representation of the function id.
+  std::string SymbolOrNumber(int32_t FuncId) const;
+
+  // Returns the file and column from debug info for the given function id.
+  std::string FileLineAndColumn(int32_t FuncId) const;
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_TOOLS_LLVM_XRAY_FUNC_ID_HELPER_H

tools/llvm-xray/func-id-helper.cc

+//===- xray-fc-account.cc - XRay Function Call Accounting Tool ------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Implementation of the helper tools dealing with XRay-generated function ids.
+//
+//===----------------------------------------------------------------------===//
+
+#include <sstream>
+#include "func-id-helper.h"
+
+using namespace llvm;
+using namespace xray;
+
+std::string FuncIdConversionHelper::SymbolOrNumber(int32_t FuncId) const {
+  std::ostringstream F;
+  auto It = FunctionAddresses.find(FuncId);
+  if (It == FunctionAddresses.end()) {
+    F << "#" << FuncId;
+    return F.str();
+  }
+
+  auto ResOrErr = Symbolizer.symbolizeCode(BinaryInstrMap, It->second);
+  if (!ResOrErr || ResOrErr.get().FunctionName == "<invalid>") {
+    F << "@(" << std::hex << It->second << ")";
+  } else {
+    auto &DI = ResOrErr.get();
+    F << DI.FunctionName;
+  }
+  return F.str();
+}
+
+std::string FuncIdConversionHelper::FileLineAndColumn(int32_t FuncId) const {
+  auto It = FunctionAddresses.find(FuncId);
+  if (It == FunctionAddresses.end()) {
+    return "(unknown)";
+  }
+  std::ostringstream F;
+  auto ResOrErr = Symbolizer.symbolizeCode(BinaryInstrMap, It->second);
+  if (!ResOrErr || ResOrErr.get().FileName == "<invalid>")
+    return "(unknown)";
+  auto &DI = ResOrErr.get();
+  auto LastSlash = DI.FileName.find_last_of('/');
+  if (LastSlash != std::string::npos) {
+    F << DI.FileName.substr(LastSlash + 1);
+  } else {
+    F << DI.FileName;
+  }
+  F << ":" << DI.Line << ":" << DI.Column;
+  return F.str();
+}

tools/llvm-xray/llvm-xray.cc

 //===----------------------------------------------------------------------===//
 //
 // Basic usage:
 //
 //   llvm-xray [options] <subcommand> [subcommand-specific options]
 //
 #include "xray-registry.h"
 #include "llvm/Support/CommandLine.h"
-#include "llvm/Support/FileSystem.h"
-#include "llvm/Support/raw_ostream.h"
 #include <unistd.h>
 
 using namespace llvm;
 using namespace llvm::xray;
 
 int main(int argc, char *argv[]) {
   cl::ParseCommandLineOptions(argc, argv,
                               "XRay Tools\n\n"

tools/llvm-xray/xray-converter.h

+//===- xray-converter.h - XRay Trace Conversion ---------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Defines the TraceConverter class for turning binary traces into
+// human-readable text and vice versa.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TOOLS_LLVM_XRAY_XRAY_CONVERTER_H
+#define LLVM_TOOLS_LLVM_XRAY_XRAY_CONVERTER_H
+
+#include "func-id-helper.h"
+#include "xray-log-reader.h"
+#include "xray-record.h"
+
+namespace llvm {
+namespace xray {
+
+class TraceConverter {
+  FuncIdConversionHelper &FuncIdHelper;
+  bool Symbolize;
+
+public:
+  TraceConverter(FuncIdConversionHelper &FuncIdHelper, bool Symbolize = false)
+      : FuncIdHelper(FuncIdHelper), Symbolize(Symbolize) {}
+
+  void exportAsYAML(const NaiveLogReader &Records, raw_ostream &OS);
+  void exportAsRAW(const YAMLLogReader &Records, int Fd);
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_TOOLS_LLVM_XRAY_XRAY_CONVERTER_H

tools/llvm-xray/xray-converter.cc

+//===- xray-converter.cc - XRay Trace Conversion --------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Implements the trace conversion functions.
+//
+//===----------------------------------------------------------------------===//
+#include "xray-converter.h"
+
+#include "xray-extract.h"
+#include "xray-record-yaml.h"
+#include "xray-registry.h"
+#include "llvm/DebugInfo/Symbolize/Symbolize.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/YAMLTraits.h"
+#include "llvm/Support/raw_ostream.h"
+#include <unistd.h>
+
+using namespace llvm;
+using namespace xray;
+
+// llvm-xray convert
+// ----------------------------------------------------------------------------
+static cl::SubCommand Convert("convert", "Trace Format Conversion");
+static cl::opt<std::string> ConvertInput(cl::Positional,
+                                         cl::desc("<xray log file>"),
+                                         cl::Required, cl::sub(Convert));
+enum class ConvertFormats { BINARY, YAML };
+static cl::opt<ConvertFormats> ConvertInputFormat(
+    "input-format", cl::desc("input format"),
+    cl::values(clEnumValN(ConvertFormats::BINARY, "raw",
+                          "input is in raw binary"),
+               clEnumValN(ConvertFormats::YAML, "yaml", "input is in yaml")),
+    cl::sub(Convert));
+static cl::alias ConvertInputFormat2("i", cl::aliasopt(ConvertInputFormat),
+                                     cl::desc("Alias for -input-format"),
+                                     cl::sub(Convert));
+static cl::opt<ConvertFormats> ConvertOutputFormat(
+    "output-format", cl::desc("output format"),
+    cl::values(clEnumValN(ConvertFormats::BINARY, "raw", "output in binary"),
+               clEnumValN(ConvertFormats::YAML, "yaml", "output in yaml")),
+    cl::sub(Convert));
+static cl::alias ConvertOutputFormat2("f", cl::aliasopt(ConvertOutputFormat),
+                                      cl::desc("Alias for -output-format"),
+                                      cl::sub(Convert));
+static cl::opt<std::string>
+    ConvertOutput("output", cl::value_desc("output file"), cl::init("-"),
+                  cl::desc("output file; use '-' for stdout"),
+                  cl::sub(Convert));
+static cl::alias ConvertOutput2("o", cl::aliasopt(ConvertOutput),
+                                cl::desc("Alias for -output"),
+                                cl::sub(Convert));
+
+static cl::opt<bool>
+    ConvertSymbolize("symbolize",
+                     cl::desc("symbolize function ids from the input log"),
+                     cl::init(false), cl::sub(Convert));
+
+static cl::opt<std::string>
+    ConvertInstrMap("instr_map",
+                    cl::desc("binary with the instrumentation map, or "
+                             "a separate instrumentation map"),
+                    cl::value_desc("binary with xray_instr_map"),
+                    cl::sub(Convert), cl::init(""));
+static cl::alias ConvertInstrMap2("m", cl::aliasopt(ConvertInstrMap),
+                                  cl::desc("Alias for -instr_map"),
+                                  cl::sub(Convert));
+static cl::opt<bool> ConvertSortInput(
+    "sort",
+    cl::desc("determines whether to sort input log records by timestamp"),
+    cl::sub(Convert), cl::init(true));
+static cl::alias ConvertSortInput2("s", cl::aliasopt(ConvertSortInput),
+                                   cl::desc("Alias for -sort"),
+                                   cl::sub(Convert));
+static cl::opt<InstrumentationMapExtractor::InputFormats> InstrMapFormat(
+    "instr-map-format", cl::desc("format of instrumentation map"),
+    cl::values(clEnumValN(InstrumentationMapExtractor::InputFormats::ELF, "elf",
+                          "instrumentation map in an ELF header"),
+               clEnumValN(InstrumentationMapExtractor::InputFormats::YAML,
+                          "yaml", "instrumentation map in YAML")),
+    cl::sub(Convert), cl::init(InstrumentationMapExtractor::InputFormats::ELF));
+static cl::alias InstrMapFormat2("t", cl::aliasopt(InstrMapFormat),
+                                 cl::desc("Alias for -instr-map-format"),
+                                 cl::sub(Convert));
+
+using llvm::yaml::MappingTraits;
+using llvm::yaml::ScalarEnumerationTraits;
+using llvm::yaml::IO;
+using llvm::yaml::Output;
+
+void TraceConverter::exportAsYAML(const NaiveLogReader &Records,
+                                  raw_ostream &OS) {
+  YAMLXRayTrace Trace;
+  const auto &FH = Records.getFileHeader();
+  Trace.Header = {FH.Version, FH.Type, FH.ConstantTSC, FH.NonstopTSC,
+                  FH.CycleFrequency};
+  Trace.Records.reserve(Records.size());
+  for (const auto &R : Records) {
+    Trace.Records.push_back(
+        {R.RecordType, R.CPU,
+         R.Type == 0 ? RecordTypes::ENTER : RecordTypes::EXIT, R.FuncId,
+         Symbolize ? FuncIdHelper.SymbolOrNumber(R.FuncId)
+                   : std::to_string(R.FuncId),
+         R.TSC, R.TId});
+  }
+  Output Out(OS);
+  Out << Trace;
+}
+
+namespace {
+
+void retryingWriteAll(int Fd, const char *Begin, const char *End) {
+  if (Begin == End)
+    return;
+  auto TotalBytes = std::distance(Begin, End);
+  while (auto Written = write(Fd, Begin, TotalBytes)) {
+    if (Written < 0) {
+      if (errno == EINTR)
+        continue;
+      return;
+    }
+    TotalBytes -= Written;
+    if (TotalBytes == 0)
+      break;
+    Begin += Written;
+  }
+  return;
+}
+
+} // namespace
+
+void TraceConverter::exportAsRAW(const YAMLLogReader &Records, int Fd) {
+  retryingWriteAll(Fd, reinterpret_cast<const char *>(&Records.getFileHeader()),
+                   reinterpret_cast<const char *>(&Records.getFileHeader()) +
+                       sizeof(XRayFileHeader));
+
+  // Now we write each record as binary.
+  std::aligned_storage<sizeof(XRayRecord), alignof(XRayRecord)>::type Buffer;
+  for (const auto &Y : Records) {
+    std::ignore = toXRayRecord(Y, Buffer);
+    retryingWriteAll(Fd, reinterpret_cast<const char *>(&Buffer),
+                     reinterpret_cast<const char *>(&Buffer) +
+                         sizeof(XRayRecord));
+  }
+}
+
+namespace llvm {
+namespace xray {
+
+static CommandRegistration Unused(&Convert, [] {
+  if (ConvertInputFormat == ConvertOutputFormat) {
+    errs() << "-input-format and -output-format are the same, bailing out.\n";
+    return 1;
+  }
+  int Fd;
+  auto EC = sys::fs::openFileForRead(ConvertInput, Fd);
+  if (EC) {
+    errs() << "Cannot open file '" << ConvertInput << "' for reading.\n";
+    return 1;
+  }
+  xray::InstrumentationMapExtractor Extractor(ConvertInstrMap, InstrMapFormat,
+                                              EC);
+  const auto &FunctionAddresses = Extractor.getFunctionAddresses();
+  symbolize::LLVMSymbolizer::Options Opts(
+      symbolize::FunctionNameKind::LinkageName, true, true, false, "");
+  symbolize::LLVMSymbolizer Symbolizer(Opts);
+  llvm::xray::FuncIdConversionHelper FuncIdHelper(ConvertInstrMap, Symbolizer,
+                                                  FunctionAddresses);
+  llvm::xray::TraceConverter TC(FuncIdHelper, ConvertSymbolize);
+  switch (ConvertInputFormat) {
+  case ConvertFormats::BINARY: {
+    xray::NaiveLogReader LogReader(Fd, EC, ConvertSortInput);
+    if (EC) {
+      errs() << "Failed loading input file '" << ConvertInput << "'\n";
+      return 1;
+    }
+    if (LogReader.getFileHeader().Version != 1) {
+      errs() << "Unsupported XRay file version: "
+             << LogReader.getFileHeader().Version << '\n';
+      return 1;
+    }
+    raw_fd_ostream OS(ConvertOutput, EC,
+                      ConvertOutputFormat == ConvertFormats::BINARY
+                          ? sys::fs::OpenFlags::F_None
+                          : sys::fs::OpenFlags::F_Text);
+    if (EC) {
+      errs() << "Cannot open file '" << ConvertOutput << "' for writing.\n";
+      return 1;
+    }
+    switch (ConvertOutputFormat) {
+    case ConvertFormats::YAML:
+      TC.exportAsYAML(LogReader, OS);
+      return 0;
+    default:
+      errs() << "input-output conversion for provided formats not supported "
+                "yet.\n";
+      return 1;
+    }
+    break;
+  }
+  case ConvertFormats::YAML: {
+    xray::YAMLLogReader LogReader(Fd, EC, ConvertSortInput);
+    if (EC) {
+      errs() << "Failed loading input file '" << ConvertInput << "'\n";
+      return 1;
+    }
+    if (LogReader.getFileHeader().Version != 1) {
+      errs() << "Unsupported XRay file version: "
+             << LogReader.getFileHeader().Version << '\n';
+      return 1;
+    }
+    int Fd;
+    sys::fs::openFileForWrite(ConvertOutput, Fd, sys::fs::OpenFlags::F_None);
+    if (EC) {
+      errs() << "Cannot open file '" << ConvertOutput << "' for writing.\n";
+      return 1;
+    }
+    switch (ConvertOutputFormat) {
+    case ConvertFormats::BINARY:
+      TC.exportAsRAW(LogReader, Fd);
+      close(Fd);
+      return 0;
+    default:
+      errs() << "input-output conversion for provided formats not supported "
+                "yet.\n";
+      close(Fd);
+      return 1;
+    }
+    break;
+  }
+  }
+  return 0;
+});
+
+} // namespace xray
+} // namespace llvm

tools/llvm-xray/xray-log-reader.h

+//===- xray-log-reader.h - XRay Log Reader Interface ----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Define the interface for an XRay log reader. Currently we only support one
+// version of the log (naive log) with fixed-sized records.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TOOLS_LLVM_XRAY_XRAY_LOG_READER_H
+#define LLVM_TOOLS_LLVM_XRAY_XRAY_LOG_READER_H
+
+#include <cstdint>
+#include <deque>
+#include <vector>
+
+#include "xray-record-yaml.h"
+#include "xray-record.h"
+
+namespace llvm {
+namespace xray {
+
+class NaiveLogReader {
+  XRayFileHeader FileHeader;
+  typedef std::deque<XRayRecord> RecordsDeque;
+  RecordsDeque Records;
+  using iterator = RecordsDeque::iterator;
+  using citerator = RecordsDeque::const_iterator;
+
+public:
+  // Loads the contents of the file into memory, then sorts the events by TSC if
+  // |Sort| is true.
+  explicit NaiveLogReader(int Fd, std::error_code &EC, bool Sort = true);
+
+  const XRayFileHeader &getFileHeader() const { return FileHeader; }
+
+  iterator begin() { return Records.begin(); }
+  iterator end() { return Records.end(); }
+  citerator cbegin() { return Records.cbegin(); }
+  citerator cend() { return Records.cend(); }
+  citerator begin() const { return Records.begin(); }
+  citerator end() const { return Records.end(); }
+  citerator cbegin() const { return Records.begin(); }
+  citerator cend() const { return Records.end(); }
+  size_t size() const { return Records.size(); }
+};
+
+class YAMLLogReader {
+  XRayFileHeader FileHeader;
+  typedef std::vector<YAMLXRayRecord> RecordsVector;
+  RecordsVector Records;
+  using iterator = RecordsVector::iterator;
+  using citerator = RecordsVector::const_iterator;
+
+public:
+  // Loads the contents of the file from |Fd|, parses the YAML and turns those
+  // into XRayRecord entries. Sorts the events by TSC if |Sort| is true.
+  explicit YAMLLogReader(int Fd, std::error_code &EC, bool Sort = true);
+
+  const XRayFileHeader &getFileHeader() const { return FileHeader; };
+  iterator begin() { return Records.begin(); }
+  iterator end() { return Records.end(); }
+  citerator cbegin() { return Records.cbegin(); }
+  citerator cend() { return Records.cend(); }
+  citerator begin() const { return Records.begin(); }
+  citerator end() const { return Records.end(); }
+  citerator cbegin() const { return Records.begin(); }
+  citerator cend() const { return Records.end(); }
+  size_t size() const { return Records.size(); }
+};
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_TOOLS_LLVM_XRAY_XRAY_LOG_READER_H

tools/llvm-xray/xray-log-reader.cc

+//===- xray-log-reader.cc - XRay Log Reader Implementation ----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// XRay log reader implementation.
+//
+//===----------------------------------------------------------------------===//
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "xray-log-reader.h"
+#include "xray-record-yaml.h"
+#include "llvm/Support/FileSystem.h"
+
+using namespace llvm;
+using namespace llvm::xray;
+using llvm::yaml::Input;
+
+NaiveLogReader::NaiveLogReader(int Fd, std::error_code &EC, bool Sort) {
+  off_t FileSize;
+  while ((FileSize = lseek(Fd, 0, SEEK_END)) == -1) {
+    if (errno == EINTR)
+      continue;
+  }
+  assert(FileSize != -1 && "Error seeking to the end of file");
+  sys::fs::mapped_file_region MappedFile(
+      Fd, sys::fs::mapped_file_region::mapmode::readonly, FileSize, 0, EC);
+  if (EC)
+    return;
+
+  // Set up the FileHeader.
+  memcpy(&FileHeader, MappedFile.data(), sizeof(XRayFileHeader));
+
+  // Start copying the data from the MappedFile into the Records data structure.
+  auto I = MappedFile.data() + sizeof(XRayFileHeader);
+  auto E = MappedFile.data() + MappedFile.size();
+  while (I + sizeof(XRayRecord) <= E) {
+    Records.push_back({});
+    memcpy(&Records.back(), I, sizeof(XRayRecord));
+    I += sizeof(XRayRecord);
+  }
+
+  if (Sort)
+    std::sort(
+        Records.begin(), Records.end(),
+        [](const XRayRecord &L, const XRayRecord &R) { return L.TSC < R.TSC; });
+}
+
+YAMLLogReader::YAMLLogReader(int Fd, std::error_code &EC, bool Sort) {
+  off_t FileSize;
+  while ((FileSize = lseek(Fd, 0, SEEK_END)) == -1) {
+    if (errno == EINTR)
+      continue;
+  }
+  assert(FileSize != -1 && "Error seeking to the end of the file");
+  sys::fs::mapped_file_region MappedFile(
+      Fd, sys::fs::mapped_file_region::mapmode::readonly, FileSize, 0, EC);
+  if (EC)
+    return;
+
+  // Load the documents from the MappedFile.
+  YAMLXRayTrace Trace;
+  Input In(StringRef(MappedFile.data(), FileSize));
+  In >> Trace;
+  if (In.error()) {
+    EC = std::make_error_code(std::errc::io_error);
+    return;
+  }
+  // Only read the first trace in the document.
+  FileHeader.Version = Trace.Header.Version;
+  FileHeader.Type = Trace.Header.Type;
+  FileHeader.ConstantTSC = Trace.Header.ConstantTSC;
+  FileHeader.NonstopTSC = Trace.Header.NonstopTSC;
+  FileHeader.CycleFrequency = Trace.Header.CycleFrequency;
+  Records.swap(Trace.Records);
+  if (Sort)
+    std::sort(Records.begin(), Records.end(),
+              [](const YAMLXRayRecord &L, const YAMLXRayRecord &R) {
+                return L.TSC < R.TSC;
+              });
+}

tools/llvm-xray/xray-record-yaml.h

+//===- xray-record-yaml.h - XRay Record YAML Support Definitions ----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Types and traits specialisations for YAML I/O of XRay log entries.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_YAML_H
+#define LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_YAML_H
+
+#include <type_traits>
+
+#include "xray-record.h"
+#include "llvm/Support/YAMLTraits.h"
+
+namespace llvm {
+namespace xray {
+
+struct YAMLXRayFileHeader {
+  uint16_t Version;
+  uint16_t Type;
+  bool ConstantTSC;
+  bool NonstopTSC;
+  uint64_t CycleFrequency;
+};
+
+enum class RecordTypes { ENTER, EXIT };
+
+struct YAMLXRayRecord {
+  uint16_t RecordType;
+  uint8_t CPU;
+  RecordTypes Type;
+  int32_t FuncId;
+  std::string Function;
+  uint64_t TSC;
+  uint32_t TId;
+};
+
+struct YAMLXRayTrace {
+  YAMLXRayFileHeader Header;
+  std::vector<YAMLXRayRecord> Records;
+};
+
+using XRayRecordStorage =
+    std::aligned_storage<sizeof(XRayRecord), alignof(XRayRecord)>::type;
+
+/// Converts a YAMLXRayRecord into an XRayRecord stored in an aligned storage
+/// buffer |Buf| then returns a reference to that object.
+inline XRayRecord &toXRayRecord(const YAMLXRayRecord &Y,
+                                XRayRecordStorage &Buf) {
+  XRayRecord &R = *reinterpret_cast<XRayRecord *>(&Buf);
+  R.RecordType = Y.RecordType;
+  R.CPU = Y.CPU;
+  switch (Y.Type) {
+  case RecordTypes::ENTER:
+    R.Type = 0;
+    break;
+  case RecordTypes::EXIT:
+    R.Type = 1;
+    break;
+  }
+  R.FuncId = Y.FuncId;
+  R.TSC = Y.TSC;
+  R.TId = Y.TId;
+  memset(R.Buffer, 0, sizeof(R.Buffer));
+  return R;
+}
+
+} // namespace xray
+
+namespace yaml {
+
+// YAML Traits
+// -----------
+template <> struct ScalarEnumerationTraits<xray::RecordTypes> {
+  static void enumeration(IO &IO, xray::RecordTypes &Type) {
+    IO.enumCase(Type, "function-enter", xray::RecordTypes::ENTER);
+    IO.enumCase(Type, "function-exit", xray::RecordTypes::EXIT);
+  }
+};
+
+template <> struct MappingTraits<xray::YAMLXRayFileHeader> {
+  static void mapping(IO &IO, xray::YAMLXRayFileHeader &Header) {
+    IO.mapRequired("version", Header.Version);
+    IO.mapRequired("type", Header.Type);
+    IO.mapRequired("constant-tsc", Header.ConstantTSC);
+    IO.mapRequired("nonstop-tsc", Header.NonstopTSC);
+    IO.mapRequired("cycle-frequency", Header.CycleFrequency);
+  }
+};
+
+template <> struct MappingTraits<xray::YAMLXRayRecord> {
+  static void mapping(IO &IO, xray::YAMLXRayRecord &Record) {
+    // FIXME: Make this type actually be descriptive
+    IO.mapRequired("type", Record.RecordType);
+    IO.mapRequired("func-id", Record.FuncId);
+    IO.mapOptional("function", Record.Function);
+    IO.mapRequired("cpu", Record.CPU);
+    IO.mapRequired("thread", Record.TId);
+    IO.mapRequired("kind", Record.Type);
+    IO.mapRequired("tsc", Record.TSC);
+  }
+
+  static constexpr bool flow = true;
+};
+
+template <> struct MappingTraits<xray::YAMLXRayTrace> {
+  static void mapping(IO &IO, xray::YAMLXRayTrace &Trace) {
+    // A trace file contains two parts, the header and the list of all the
+    // trace records.
+    IO.mapRequired("header", Trace.Header);
+    IO.mapRequired("records", Trace.Records);
+  }
+};
+
+} // namespace yaml
+} // namespace llvm
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(xray::YAMLXRayRecord);
+
+#endif // LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_YAML_H

tools/llvm-xray/xray-record.h

+//===- xray-record.h - XRay Trace Record ----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file replicates the record definition for XRay log entries. This should
+// follow the evolution of the log record versions supported in the compiler-rt
+// xray project.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_H
+#define LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_H
+
+#include <cstdint>
+
+namespace llvm {
+namespace xray {
+
+struct alignas(32) XRayFileHeader {
+  uint16_t Version = 0;
+  uint16_t Type = 0;
+  bool ConstantTSC : 1;
+  bool NonstopTSC : 1;
+  alignas(8) uint64_t CycleFrequency = 0;
+} __attribute__((packed));
+
+static_assert(sizeof(XRayFileHeader) == 32, "XRayFileHeader != 32 bytes");
+
+struct alignas(32) XRayRecord {
+  uint16_t RecordType = 0;
+
+  // The CPU where the thread is running. We assume number of CPUs <= 256.
+  uint8_t CPU = 0;
+
+  // The type of the event. Usually either ENTER = 0 or EXIT = 1.
+  uint8_t Type = 0;
+
+  // The function ID for the record.
+  int32_t FuncId = 0;
+
+  // Get the full 8 bytes of the TSC when we get the log record.
+  uint64_t TSC = 0;
+
+  // The thread ID for the currently running thread.
+  uint32_t TId = 0;
+
+  // Use some bytes in the end of the record for buffers.
+  char Buffer[4] = {};
+} __attribute__((packed));
+
+static_assert(sizeof(XRayRecord) == 32, "XRayRecord != 32 bytes");
+
+} // namespace xray
+} // namespace llvm
+
+#endif // LLVM_TOOLS_LLVM_XRAY_XRAY_RECORD_H