[extra] pptrace - preprocessor tracing and testing tool

[jtsoftware]

Summary

pp-trace is a tool for displaying preprocessor activity by means of the PPCallbacks interface. Its primary reason for existence is for testing the PPCallbacks mechanism, but it also might be useful as a tool for understanding preprocessor activity (as an alternative to clang's -P option for looking at preprocessor output).

This is a first cut, for proof of concept, with only two tests for demonstration purposes.

It supports two output formats, YAML and a generic format. Originally I just put in the YAML, but I felt it was kind of verbose, so I added the generic format as a default.

Please see the file comments for details.

Documentation and more tests will follow in a later checkin.

[silvas]

I don't see the purpose of the "generic" output format. It is only 1 character away from being valid YAML. The following is valid YAML (Assuming the ArgumentK values are unique, which I think is true in this case):

- CallbackName:
    Argument1: Value1
    Argument2: Value2

The overhead of maintaining two output formats doesn't seem worth it. For simplicity of code that manipulates this output, it may be best to use something like:

- Kind: CallbackName
  Argument1: Value1
  Argument2: Value2

(i.e., have the callback name just be another field of the record; this is a very common pattern for handling serialized variant data in dynamic languages (and YAML's semantic model is basically wed to dynamic languages)).

Since you don't need to read this information back into the same data structure that produced it, it seems like it might be simpler to forego YAMLIO.

pp-trace/PPCallbacksTracker.cpp

21

http://llvm.org/docs/CodingStandards.html#include-style. In particular, "PPCallbacksTracker.h" should be first.

177

Any particular reason why the capitalization for this is different from the rest?

607–608

This output doesn't seem very useful.

626

Ew. No. Fixed size buffers are evil.

It seems like most of the reason you need this is just to print a string "Foo(Bar)". Better to just have a subroutine taking the two strings "Foo" and "Bar" and using raw_string_ostream or raw_svector_ostream to generate that output.

pp-trace/PPCallbacksTracker.h

119–155

Don't copy the documentation from the base class.

156–163

Use LLVM_OVERRIDE (i.e. C++11 override) instead of prefixing virtual to all these overriden virtual functions. That way, C++11 builds (which many people are doing these days) will benefit from the error checking.

[jtsoftware]

I've revised to use just a YAML output format without YAML I/O, and address the other comments, except the moduleImport name, which I'll fix in later clang+extra checkin.

[jtsoftware]

On second thought, aren't function members supposed start with a lower case? Are the other functions wrong, or is there a convention for virtual/callback functions?

pp-trace/PPCallbacksTracker.cpp

177

It's like this in the base class. I'll fix it in a separate checkin.

[silvas]

pp-trace/PPCallbacksTracker.cpp

26–33

I think that Loc.printToString may sometimes not do quite what you want. See http://clang.llvm.org/doxygen/SourceLocation_8cpp_source.html#l00038. You might want to replicate that logic in a customized way here to make sure that the results are consistently as intended.

257–263

Use raw_string_ostream or raw_svector_ostream here and anywhere else you have used sprintf.

412–413

You can simplify this to just .push_back(Argument(Name, Value))

416–433

This format is annoying to parse downstream. Most of the uses of this seem to be just to tag the type of data. For example, SourceRange's should be just a YAML list of two entries: [ loc1, loc2 ], and enum constants could be just their name (or maybe better EnumName::EN_EnumerantName).

The SourceLocation format that you are using to print them seems fine, since that can be deconstructed with a single call to .split(':') (in Python) or similar.

516–518

This comment is out of date. Please check the comments on the other methods as well.

522–530

I recommend formatting this as a list of YAML flow records (i.e., basically JSON) for easy downstream consumption. So it would be something like:

[{name: foo, loc: sourcelocationstring}, ...]

563–564

This seems like it should produce a YAML list in the end, instead of a YAML string formatted like a function call (which would have to be parsed by a consumer, instead of being handled in the YAML parser itself). The YAML flow style can be used, e.g. [ foo, bar, baz ]. For example, this is easily consumed by a client using a YAML parser without an additional parsing step:

- Callback: Name
  Args: [ foo, bar, baz ]
  Argument2: Value2

The fact that it is a list of macro arguments should be clear from the context.

565–568

Tiny bit of LLVM style guidance:

• unless there's a good reason, a for loop integer induction variable should be called i and the upper bound should be called e (which is inconsistent with the rest of the coding style, which is weird, but that's the style).

• In C++, when you are iterating through a sequence sequentially, the convention is to use != to compare (this generalizes to e.g. linked lists, which can be traversed sequentially, but whose iterators can't have a meaningful <) and ++var (which avoids potentially making a copy of the iterator); obviously it doesn't matter in this case, but for consistency and clarity that is the convention. These conventions originate in the STL. If you aren't familiar with the various conceptual iterator types, you can read a summary here: http://www.sgi.com/tech/stl/Iterators.html.

So I would recommend rewriting this loop header as:

for (int i = 0, e =  Value->getNumArguments(); i != e; ++i) {

• When outputting constructs with "separator" semantics (e.g. a comma-separated list) rather than "terminator semantics" (e.g. a semicolon follows each statement, including the last), the pattern I've seen most commonly in LLVM is
  
  for (int i = ..., e = ...; i != e; ++i) { if (i) OS << ", "; [...] }

and I would recommend following this since it makes it very clear "up front" (at the beginning of the loop) that this is outputting a comma-separated list.

So overall I would recommend writing this loop as:

for (int i = 0, e =  Value->getNumArguments(); i != e; ++i) {
  if (i)
    SS << ", ";
  SS << PP.getSpelling(*Value->getUnexpArgument(i));
}

pp-trace/PPCallbacksTracker.h

83–84

This is not correct use of this macro, which is semantically equivalent to the C++11 override specifier. See http://en.cppreference.com/w/cpp/language/override.

pp-trace/PPTrace.cpp

20–25

I assume that a -- can be used to separate the [compiler options] from the rest. Is that correct? You probably should mention the behavior in that case (especially if what I described is *not* the case, to prevent confusion).

33–34

This is not correct use of "i.e." (which basically means "in other words"). You probably meant "e.g." which basically means "for example".

158–164

I feel like I'm probably nitpicking here, but I've never seen a reference in the LLVM codebase initialized with parentheses. Please use e.g. Argument &Arg = *AI; here and elsewhere.

(hopefully someday clang-tidy will catch and fix these things).

217

Use an early exit to simplify this:

if (!Error.empty()) {
  // print message
  return 1;
}

test/pp-trace/pp-trace-macro.cpp

2

Why do you need -Xclang for -std=c++11?

Also, why does -Xclang -triple=x86_64 need to be there if you are already passing -target x86_64?

[kimgr]

I applied the patch and did a clean build on Clang r193498 on Windows, MSVC 10.

For some reason, I'm getting an assertion failure whenever I #include <stdio.h>:
Assertion failed: Result < Start+NumUnexpArgTokens && "Invalid arg #", file llvm-trunk\llvm\tools\clang\lib\Lex\MacroArgs.cpp, line 124

I've boiled it down to the following repro:

// repro.cpp
#define DECORATED(id) id

enum Enum
{
  DECORATED( One ) = 1
};

Original problem surfaced in C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\CodeAnalysis\sourceannotations.h:56.

clang-check accepts it without complaining, so it seems like there's something amiss in the setup of pp-trace.

[jtsoftware]

Sean,

I revised per your comments, except for the one about -- in the command line, as I don't understand. I think you can use either "-' or "--" for options. I don't know about using "--" to separate the compiler options.

Kim,

What I usually do in the case of strange build errors on Windows is update cmake, delete the CMakeCache.txt file, rerun the initial camke command line ("cmake -G "Visual Studio 10" -DCLANG_BUILD_EXAMPLES=ON ." in my case), run clean in Visual Studio, then a build. I just did this, but didn't see any error, so I'm not sure what to do.

-John

[kimgr]

John,

Thanks for attempting to repro. I've done the same here, and I'm still seeing the problem. I'm using the Ninja CMake generator with cl.exe from MSVC 10.

I can give proper Visual Studio a shot later too and see if the problem persists.

[silvas]

Pending on the issue that Kim is running into, I think this looks good enough to commit.

pp-trace/PPCallbacksTracker.cpp

278–279

It's probably simpler to just use a for (int i = 0, e = Ids.size(); ... style loop here.

529–534

Same here.

539

Is this missing a colon after "Loc"? Try running it through a YAML parser (utils/yaml-bench with the -canonical flag should be sufficient to ensure syntactic validity (see test/YAMLParser/ for example usage)).

test/pp-trace/pp-trace-macro.cpp

24

MD here is a bit cryptic. I see that you are doing this to be maximally consistent with the argument name, but I think that MacroDirective would be better from a readability standpoint.

[jtsoftware]

Sean,

Thanks for the excellent review, and the reference to yaml-bench. It caught a few things so far, and I'll have to use it when I write the other tests too. It didn't like the backslashes in Windows file paths, or the colons in SourceLocations, so I put in some character replacing and quoting.

I revised the for loops as pointed out.

Also, I wanted to ask about the special cases, such as where I use values like (null) to represent when a pointer is null, or (invalid) when a SourceLocation has the invalid flag set. In the former case, passing a null pointer is expected in some situations. The YAML parser doesn't seem to mind (null), but is this reasonable semantically?

Also, again semantically, you probably already noticed I'm not trying to flatten the full data structures for possible later resurrection, just providing what I think is sufficient high-level information to know what the preprocessor is up to.

Kim,

Thanks also for your help. I'll install ninja cmake and give it a shot.

-John

[kimgr]

Still seeing the same assertion when I build with VS10; my steps to repro:

llvm-trunk$ mkdir vcbuild && cd vcbuild
llvm-trunk/vcbuild$ cmake -G "Visual Studio 10" ..\llvm
llvm-trunk/vcbuild$ start LLVM.sln
(right-click and build pp-trace only)
llvm-trunk/vcbuild$ bin\Debug\pp-trace repro.cpp

I can't explain it, but it seems to be consistent :-(

[jtsoftware]

Kim,

I totally misunderstood. I thought you were talking about a problem building pp-trace, but you're talking about an assertion while running pp-trace, which I'm getting too with your repro. I'm looking into it. Thanks for the help.

-John

[jtsoftware]

This revision includes a fix for the assertion Kim saw.

Since we seem to be close on the main idea, after this I'll start adding more test to get better coverage.

-John

[kimgr]

Ah, I completely missed the fact that your code was calling the asserting method. Works fine now, thanks!

Bunch of aesthetic comments added, I'll try this on some bigger preprocessor challenges...

pp-trace/PPCallbacksTracker.cpp

2

You don't need *- C++ -* in .cpp files.

82

Indentation looks funny here, but maybe that's a good thing considering that "0" is apparently not in the enum. :-)

88–91

This duplicates the information from the header. I vote to remove it.

97–98

The comment seems redundant to me.

102

Should these be doxygen comments?

213

PragmaDebug

252

PragmaDiagnostic

282

I saw Sean suggested lower-case index names, but the coding guidelines seem to advocate upper-case I and E.

312

I'm not sure what to think about naming the argument in the output something else than the actual arg. It stands out, but it would be nice if the MD argument was in fact named MacroDirective (though it clashes with the type...)

322

MacroDirective/MD

330

MacroDirective/MD

339

MacroDirective/MD

378

MacroDirective/MD

388

MacroDirective/MD

411

I'll never get used to the fact that SmallSet::count returns a bool :-)

444

This should be a const std::string &, right?

593

const std::string &

pp-trace/PPCallbacksTracker.h

23–24

Should be PPTRACE_CALLBACKS_H or maybe match the filename in full, i.e. PPTRACE_PPCALLBACKSTRACKER_H?

71

Style guide says to avoid repeating class/function names in \brief. Not sure what to say about the ctor in \brief, maybe something to the effect of ownership?

pp-trace/PPTrace.cpp

155–167

I prefer const_iterator and const refs to elements, but I don't know what the general convention for LLVM is.

183

/*MaxSplit=*/-1, /*KeepEmpty=*/false

211–213

Add braces around this block to keep symmetry with else

[kimgr]

Running pp-trace on this file still asserts, unfortunately:

// repro2.cpp
#define X X_IMPL(a,b) X_IMPL2(c)

#define X_IMPL(p1,p2)
#define X_IMPL2(p1)

X

This looks nonsensical, but is based on C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\sal.h:454.

I suspected that Value->getNumArguments() might return zero, which would cause your e to become negative (or very large), but the number in this case is four. You still have a bug if getNumArguments ever returns zero, but maybe it never will (i.e. assert on it)

[kimgr]

OK, I took some time to step through this. Enumerating the arguments of a MacroArgs seems like a difficult problem (tm).

• getNumArguments does not return the number of arguments, it returns the number of tokens in the argument list
• getUnexpArgument takes an index, but skips over eof tokens

Here's a way I've found to enumerate over the argument tokens:

// This can probably be translated into a for loop,
// but it's late here and my head is buzzing...
unsigned ArgTokenCount = Value->getNumArguments();
unsigned I = 0;
while (I < ArgTokenCount) {
    const clang::Token* Current = Value->getUnexpArgument(I);
    unsigned TokenCount = Value->getArgLength(Current) + 1; // include EOF
    I++;
    ArgTokenCount -= TokenCount;
}

One problem is arguments rarely consist of a single token, but we can look at things like:

#define X Y(1 + 6, 2)
#define Y(a,b)

X

You'd need to find a way to rejoin a sequence of tokens into a string. The tokens appear to live in contiguous memory, though, so once you have the first token and the length of every argument, the entire arg is covered by [Current, Current + TokenCount). Not sure how to go from there to something you can render, but it should be doable.

Hope that helps!

[jtsoftware]

Thanks, Kim, for the excellent review, and also for solving the walking issue. I've applied the suggested changes, revised the MacroArgs argument walking code segment, and added another test.

More tests will follow, but first I'm going to run the tool over a set of platform headers, to see if I can ring out any more crashes.

-John

[jtsoftware]

I made one more change to the macro args formatting, as it could produce output with YAML symbols that will confuse a YAML parser. I have it print tokens that are not identifiers or numbers by the results of their getName() call value, enclosed in '<' and '>'.

I ran pp-trace over a full set of platform headers, including running yaml-bench over the output to check for valid YAML, with no errors.

Though I still need to write tests for all the callback types, and user documentation, I think it might be ready for an initial checkin.

-John

[kimgr]

I'm in no position to sign off commits, but this looks good and useful to me. A few small comments/questions, otherwise I'm happy. Thanks!

pp-trace/PPCallbacksTracker.cpp

564–591

Nice! :-)

607

const reference?

pp-trace/PPCallbacksTracker.h

218

Why not a const reference?

pp-trace/PPTrace.cpp

109

Just pass callbacks tracker into addPPCallbacks immediately:

PP.addPPCallbacks(new PPCallbacksTracker(Ignore, CallbackCalls, PP));

112–113

No use keeping this as a member

210–211

Why is the "-" necessary? I just leave out -output and pp-trace dutifully prints to stdout.

[silvas]

(sorry for the delay). As per my earlier comment now that it seems like the
issue Kim was seeing is resolved, feel free to check this in.

More comments inline:

[jtsoftware]

Committed in r193743 (and later fixes in r194440 r194422 r194081 r194079 r193842 r193841 r193746).

pp-trace/PPCallbacksTracker.cpp

82

The comment of the Mapping enum mentions value of 0 meaning "uncomputed". Indentation courtesy of clang-format:-)

102

Sean pointed out that since these are overrides, they shouldn't have the Doxygen comments.

CMakeLists.txt

 add_subdirectory(clang-apply-replacements)
 add_subdirectory(clang-modernize)
 add_subdirectory(clang-tidy)
 add_subdirectory(modularize)
+add_subdirectory(pp-trace)
 add_subdirectory(remove-cstr-calls)
 add_subdirectory(tool-template)
 
 # Add the common testsuite after all the tools.
 add_subdirectory(test)
 add_subdirectory(unittests)

Makefile

 ##===- tools/extra/Makefile --------------------------------*- Makefile -*-===##
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 ##===----------------------------------------------------------------------===##
 
 CLANG_LEVEL := ../..
 
 include $(CLANG_LEVEL)/../../Makefile.config
 
-PARALLEL_DIRS := remove-cstr-calls tool-template modularize
+PARALLEL_DIRS := remove-cstr-calls tool-template modularize pp-trace
 DIRS := clang-apply-replacements clang-modernize clang-tidy unittests
 
 include $(CLANG_LEVEL)/Makefile
 
 ###
 # Handle the nested test suite.
 
 ifneq ($(PROJ_SRC_ROOT),$(PROJ_OBJ_ROOT))

pp-trace/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  asmparser
+  support
+  mc
+  )
+
+add_clang_executable(pp-trace
+  PPTrace.cpp
+  PPCallbacksTracker.cpp
+  )
+
+target_link_libraries(pp-trace
+  clangLex
+  clangParse
+  clangSema
+  clangTooling
+  )

pp-trace/Makefile

+##===- extra/pp-trace/Makefile --------------------------*- Makefile -*---===##
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+##===---------------------------------------------------------------------===##
+
+CLANG_LEVEL := ../../..
+
+TOOLNAME = pp-trace
+
+# No plugins, optimize startup time.
+TOOL_NO_EXPORTS = 1
+
+LINK_COMPONENTS := mcparser bitreader support mc option TransformUtils
+USEDLIBS = clangFrontend.a clangSerialization.a clangDriver.a \
+           clangTooling.a clangParse.a clangSema.a clangAnalysis.a \
+           clangEdit.a clangAST.a clangLex.a clangBasic.a
+
+include $(CLANG_LEVEL)/Makefile

pp-trace/PPCallbacksTracker.h

+//===--- PPCallbacksTracker.h - Preprocessor tracking -*- C++ -*---------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===--------------------------------------------------------------------===//
+///
+/// \file
+/// \brief Classes and definitions for preprocessor tracking.
+///
+/// The core definition is the PPCallbacksTracker class, derived from Clang's
+/// PPCallbacks class from the Lex library, which overrides all the callbacks
+/// and collects information about each callback call, saving it in a
+/// data structure built up of CallbackCall and Argument objects, which
+/// record the preprocessor callback name and arguments in high-level string
+/// form for later inspection.
+///
+//===--------------------------------------------------------------------===//
+
+#ifndef PPTRACE_PPCALLBACKSTRACKER_H
+#define PPTRACE_PPCALLBACKSTRACKER_H
+
+#include "clang/Lex/PPCallbacks.h"
+#include "clang/Lex/Preprocessor.h"
+
+/// \brief This class represents one callback function argument by name
+///   and value.
+class Argument {
+public:
+  Argument(llvm::StringRef Name, llvm::StringRef Value)
+      : Name(Name), Value(Value) {}
+  Argument() {}
+
+  std::string Name;
+  std::string Value;
+};
+
+/// \brief This class represents one callback call by name and an array
+///   of arguments.
+class CallbackCall {
+public:
+  CallbackCall(llvm::StringRef Name) : Name(Name) {}
+  CallbackCall() {}
+
+  std::string Name;
+  std::vector<Argument> Arguments;
+};
+
+/// \brief This class overrides the PPCallbacks class for tracking preprocessor
+///   activity by means of its callback functions.
+///
+/// This object is given a vector for storing the trace information, built up
+/// of CallbackCall and subordinate Argument objects for representing the
+/// callback calls and their arguments.  It's a reference so the vector can
+/// exist beyond the lifetime of this object, because it's deleted by the
+/// preprocessor automatically in its destructor.
+///
+/// This class supports a mechanism for inhibiting trace output for
+/// specific callbacks by name, for the purpose of eliminating output for
+/// callbacks of no interest that might clutter the output.
+///
+/// Following the constructor and destructor function declarations, the
+/// overidden callback functions are defined.  The remaining functions are
+/// helpers for recording the trace data, to reduce the coupling between it
+/// and the recorded data structure.
+class PPCallbacksTracker : public clang::PPCallbacks {
+public:
+  /// \brief Note that all of the arguments are references, and owned
+  /// by the caller.
+  /// \param Ignore - Set of names of callbacks to ignore.
+  /// \param CallbackCalls - Trace buffer.
+  /// \param PP - The preprocessor.  Needed for getting some argument strings.
+  PPCallbacksTracker(llvm::SmallSet<std::string, 4> &Ignore,
+                     std::vector<CallbackCall> &CallbackCalls,
+                     clang::Preprocessor &PP);
+
+  virtual ~PPCallbacksTracker();
+
+  // Overidden callback functions.
+
+  void FileChanged(clang::SourceLocation Loc,
+                   clang::PPCallbacks::FileChangeReason Reason,
+                   clang::SrcMgr::CharacteristicKind FileType,
+                   clang::FileID PrevFID = clang::FileID()) LLVM_OVERRIDE;
+  void FileSkipped(const clang::FileEntry &ParentFile,
+                   const clang::Token &FilenameTok,
+                   clang::SrcMgr::CharacteristicKind FileType) LLVM_OVERRIDE;
+  bool FileNotFound(llvm::StringRef FileName,
+                    llvm::SmallVectorImpl<char> &RecoveryPath) LLVM_OVERRIDE;
+  void InclusionDirective(clang::SourceLocation HashLoc,
+                          const clang::Token &IncludeTok,
+                          llvm::StringRef FileName, bool IsAngled,
+                          clang::CharSourceRange FilenameRange,
+                          const clang::FileEntry *File,
+                          llvm::StringRef SearchPath,
+                          llvm::StringRef RelativePath,
+                          const clang::Module *Imported) LLVM_OVERRIDE;
+  void moduleImport(clang::SourceLocation ImportLoc, clang::ModuleIdPath Path,
+                    const clang::Module *Imported) LLVM_OVERRIDE;
+  void EndOfMainFile() LLVM_OVERRIDE;
+  void Ident(clang::SourceLocation Loc, const std::string &str) LLVM_OVERRIDE;
+  void PragmaDirective(clang::SourceLocation Loc,
+                       clang::PragmaIntroducerKind Introducer) LLVM_OVERRIDE;
+  void PragmaComment(clang::SourceLocation Loc,
+                     const clang::IdentifierInfo *Kind,
+                     const std::string &Str) LLVM_OVERRIDE;
+  void PragmaDetectMismatch(clang::SourceLocation Loc, const std::string &Name,
+                            const std::string &Value) LLVM_OVERRIDE;
+  void PragmaDebug(clang::SourceLocation Loc,
+                   llvm::StringRef DebugType) LLVM_OVERRIDE;
+  void PragmaMessage(clang::SourceLocation Loc, llvm::StringRef Namespace,
+                     clang::PPCallbacks::PragmaMessageKind Kind,
+                     llvm::StringRef Str) LLVM_OVERRIDE;
+  void PragmaDiagnosticPush(clang::SourceLocation Loc,
+                            llvm::StringRef Namespace) LLVM_OVERRIDE;
+  void PragmaDiagnosticPop(clang::SourceLocation Loc,
+                           llvm::StringRef Namespace) LLVM_OVERRIDE;
+  void PragmaDiagnostic(clang::SourceLocation Loc, llvm::StringRef Namespace,
+                        clang::diag::Mapping mapping,
+                        llvm::StringRef Str) LLVM_OVERRIDE;
+  void PragmaOpenCLExtension(clang::SourceLocation NameLoc,
+                             const clang::IdentifierInfo *Name,
+                             clang::SourceLocation StateLoc,
+                             unsigned State) LLVM_OVERRIDE;
+  void PragmaWarning(clang::SourceLocation Loc, llvm::StringRef WarningSpec,
+                     llvm::ArrayRef<int> Ids) LLVM_OVERRIDE;
+  void PragmaWarningPush(clang::SourceLocation Loc, int Level) LLVM_OVERRIDE;
+  void PragmaWarningPop(clang::SourceLocation Loc) LLVM_OVERRIDE;
+  void MacroExpands(const clang::Token &MacroNameTok,
+                    const clang::MacroDirective *MD, clang::SourceRange Range,
+                    const clang::MacroArgs *Args) LLVM_OVERRIDE;
+  void MacroDefined(const clang::Token &MacroNameTok,
+                    const clang::MacroDirective *MD) LLVM_OVERRIDE;
+  void MacroUndefined(const clang::Token &MacroNameTok,
+                      const clang::MacroDirective *MD) LLVM_OVERRIDE;
+  void Defined(const clang::Token &MacroNameTok,
+               const clang::MacroDirective *MD,
+               clang::SourceRange Range) LLVM_OVERRIDE;
+  void SourceRangeSkipped(clang::SourceRange Range) LLVM_OVERRIDE;
+  void If(clang::SourceLocation Loc, clang::SourceRange ConditionRange,
+          bool ConditionValue) LLVM_OVERRIDE;
+  void Elif(clang::SourceLocation Loc, clang::SourceRange ConditionRange,
+            bool ConditionValue, clang::SourceLocation IfLoc) LLVM_OVERRIDE;
+  void Ifdef(clang::SourceLocation Loc, const clang::Token &MacroNameTok,
+             const clang::MacroDirective *MD) LLVM_OVERRIDE;
+  void Ifndef(clang::SourceLocation Loc, const clang::Token &MacroNameTok,
+              const clang::MacroDirective *MD) LLVM_OVERRIDE;
+  void Else(clang::SourceLocation Loc,
+            clang::SourceLocation IfLoc) LLVM_OVERRIDE;
+  void Endif(clang::SourceLocation Loc,
+             clang::SourceLocation IfLoc) LLVM_OVERRIDE;
+
+  // Helper functions.
+
+  /// \brief Start a new callback.
+  void beginCallback(const char *Name);
+
+  /// \brief Append a string to the top trace item.
+  void append(const char *Str);
+
+  /// \brief Format and append a string to the top trace item.
+  void appendFormatted(const char *Format, ...);
+
+  /// \brief Append a bool argument to the top trace item.
+  void appendArgument(const char *Name, bool Value);
+
+  /// \brief Append an int argument to the top trace item.
+  void appendArgument(const char *Name, int Value);
+
+  /// \brief Append a string argument to the top trace item.
+  void appendArgument(const char *Name, const char *Value);
+
+  /// \brief Append a string reference object argument to the top trace item.
+  void appendArgument(const char *Name, llvm::StringRef Value);
+
+  /// \brief Append a string object argument to the top trace item.
+  void appendArgument(const char *Name, const std::string &Value);
+
+  /// \brief Append a token argument to the top trace item.
+  void appendArgument(const char *Name, const clang::Token &Value);
+
+  /// \brief Append an enum argument to the top trace item.
+  void appendArgument(const char *Name, int Value, const char *Strings[]);
+
+  /// \brief Append a FileID argument to the top trace item.
+  void appendArgument(const char *Name, clang::FileID Value);
+
+  /// \brief Append a FileEntry argument to the top trace item.
+  void appendArgument(const char *Name, const clang::FileEntry *Value);
+
+  /// \brief Append a SourceLocation argument to the top trace item.
+  void appendArgument(const char *Name, clang::SourceLocation Value);
+
+  /// \brief Append a SourceRange argument to the top trace item.
+  void appendArgument(const char *Name, clang::SourceRange Value);
+
+  /// \brief Append a CharSourceRange argument to the top trace item.
+  void appendArgument(const char *Name, clang::CharSourceRange Value);
+
+  /// \brief Append a ModuleIdPath argument to the top trace item.
+  void appendArgument(const char *Name, clang::ModuleIdPath Value);
+
+  /// \brief Append an IdentifierInfo argument to the top trace item.
+  void appendArgument(const char *Name, const clang::IdentifierInfo *Value);
+
+  /// \brief Append a MacroDirective argument to the top trace item.
+  void appendArgument(const char *Name, const clang::MacroDirective *Value);
+
+  /// \brief Append a MacroArgs argument to the top trace item.
+  void appendArgument(const char *Name, const clang::MacroArgs *Value);
+
+  /// \brief Append a Module argument to the top trace item.
+  void appendArgument(const char *Name, const clang::Module *Value);
+
+  /// \brief Append a double-quoted argument to the top trace item.
+  void appendQuotedArgument(const char *Name, std::string &Value);
+
+  /// \brief Append a double-quoted file path argument to the top trace item.
+  void appendFilePathArgument(const char *Name, llvm::StringRef Value);
+
+  /// \brief Get the raw source string of the range.
+  llvm::StringRef getSourceString(clang::CharSourceRange Range);
+
+  /// \brief Callback trace information.
+  /// We use a reference so the trace will be preserved for the caller
+  /// after this object is destructed.
+  std::vector<CallbackCall> &CallbackCalls;
+
+  /// \brief Names of callbacks to ignore.
+  llvm::SmallSet<std::string, 4> &Ignore;
+
+  /// \brief Inhibit trace while this is set.
+  bool DisableTrace;
+
+  clang::Preprocessor &PP;
+};
+
+#endif // PPTRACE_PPCALLBACKSTRACKER_H

pp-trace/PPCallbacksTracker.cpp

+//===--- PPCallbacksTracker.cpp - Preprocessor tracker -*--*-------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===--------------------------------------------------------------------===//
+///
+/// \file
+/// \brief Implementations for preprocessor tracking.
+///
+/// See the header for details.
+///
+//===--------------------------------------------------------------------===//
+
+#include "PPCallbacksTracker.h"
+#include "clang/Lex/MacroArgs.h"
+#include "llvm/Support/raw_ostream.h"
+#include <stdarg.h>
+#include <stdio.h>
+
+// Utility functions.
+
+// Get a "file:line:column" source location string.
+static std::string getSourceLocationString(clang::Preprocessor &PP,
+                                           clang::SourceLocation Loc) {
+  if (Loc.isInvalid())
+    return std::string("(none)");
+
+  if (Loc.isFileID()) {
+    clang::PresumedLoc PLoc = PP.getSourceManager().getPresumedLoc(Loc);
+
+    if (PLoc.isInvalid()) {
+      return std::string("(invalid)");
+    }
+
+    std::string Str;
+    llvm::raw_string_ostream SS(Str);
+
+    // The macro expansion and spelling pos is identical for file locs.
+    SS << "\"" << PLoc.getFilename() << ':' << PLoc.getLine() << ':'
+       << PLoc.getColumn() << "\"";
+
+    std::string Result = SS.str();
+
+    // YAML treats backslash as escape, so use forward slashes.
+    std::replace(Result.begin(), Result.end(), '\\', '/');
+
+    return Result;
+  }
+
+  return std::string("(nonfile)");
+}
+
+// Enum string tables.
+
+// FileChangeReason strings.
+static const char *FileChangeReasonStrings[] = {
+  "EnterFile", "ExitFile", "SystemHeaderPragma", "RenameFile"
+};
+
+// CharacteristicKind strings.
+static const char *CharacteristicKindStrings[] = { "C_User", "C_System",
+                                                   "C_ExternCSystem" };
+
+// MacroDirective::Kind strings.
+static const char *MacroDirectiveKindStrings[] = { "MD_Define", "MD_Undefine",
+                                                   "MD_Visibility" };
+
+// PragmaIntroducerKind strings.
+static const char *PragmaIntroducerKindStrings[] = { "PIK_HashPragma",
+                                                     "PIK__Pragma",
+                                                     "PIK___pragma" };
+
+// PragmaMessageKind strings.
+static const char *PragmaMessageKindStrings[] = { "PMK_Message", "PMK_Warning",
+                                                  "PMK_Error" };
+
+// Mapping strings.
+static const char *MappingStrings[] = { "0",           "MAP_IGNORE",
+                                        "MAP_WARNING", "MAP_ERROR",
+                                        "MAP_FATAL" };
+
+// PPCallbacksTracker functions.
+
+PPCallbacksTracker::PPCallbacksTracker(llvm::SmallSet<std::string, 4> &Ignore,
+                                       std::vector<CallbackCall> &CallbackCalls,
+                                       clang::Preprocessor &PP)
+    : CallbackCalls(CallbackCalls), Ignore(Ignore), PP(PP) {}
+
+PPCallbacksTracker::~PPCallbacksTracker() {}
+
+// Callback functions.
+
+// Callback invoked whenever a source file is entered or exited.
+void PPCallbacksTracker::FileChanged(
+    clang::SourceLocation Loc, clang::PPCallbacks::FileChangeReason Reason,
+    clang::SrcMgr::CharacteristicKind FileType, clang::FileID PrevFID) {
+  beginCallback("FileChanged");
+  appendArgument("Loc", Loc);
+  appendArgument("Reason", Reason, FileChangeReasonStrings);
+  appendArgument("FileType", FileType, CharacteristicKindStrings);
+  appendArgument("PrevFID", PrevFID);
+}
+
+// Callback invoked whenever a source file is skipped as the result
+// of header guard optimization.
+void
+PPCallbacksTracker::FileSkipped(const clang::FileEntry &ParentFile,
+                                const clang::Token &FilenameTok,
+                                clang::SrcMgr::CharacteristicKind FileType) {
+  beginCallback("FileSkipped");
+  appendArgument("ParentFile", &ParentFile);
+  appendArgument("FilenameTok", FilenameTok);
+  appendArgument("FileType", FileType, CharacteristicKindStrings);
+}
+
+// Callback invoked whenever an inclusion directive results in a
+// file-not-found error.
+bool
+PPCallbacksTracker::FileNotFound(llvm::StringRef FileName,
+                                 llvm::SmallVectorImpl<char> &RecoveryPath) {
+  beginCallback("FileNotFound");
+  appendFilePathArgument("FileName", FileName);
+  return false;
+}
+
+// Callback invoked whenever an inclusion directive of
+// any kind (#include, #import, etc.) has been processed, regardless
+// of whether the inclusion will actually result in an inclusion.
+void PPCallbacksTracker::InclusionDirective(
+    clang::SourceLocation HashLoc, const clang::Token &IncludeTok,
+    llvm::StringRef FileName, bool IsAngled,
+    clang::CharSourceRange FilenameRange, const clang::FileEntry *File,
+    llvm::StringRef SearchPath, llvm::StringRef RelativePath,
+    const clang::Module *Imported) {
+  beginCallback("InclusionDirective");
+  appendArgument("IncludeTok", IncludeTok);
+  appendFilePathArgument("FileName", FileName);
+  appendArgument("IsAngled", IsAngled);
+  appendArgument("FilenameRange", FilenameRange);
+  appendArgument("File", File);
+  appendFilePathArgument("SearchPath", SearchPath);
+  appendFilePathArgument("RelativePath", RelativePath);
+  appendArgument("Imported", Imported);
+}
+
+// Callback invoked whenever there was an explicit module-import
+// syntax.
+void PPCallbacksTracker::moduleImport(clang::SourceLocation ImportLoc,
+                                      clang::ModuleIdPath Path,
+                                      const clang::Module *Imported) {
+  beginCallback("moduleImport");
+  appendArgument("ImportLoc", ImportLoc);
+  appendArgument("Path", Path);
+  appendArgument("Imported", Imported);
+}
+
+// Callback invoked when the end of the main file is reached.
+// No subsequent callbacks will be made.
+void PPCallbacksTracker::EndOfMainFile() { beginCallback("EndOfMainFile"); }
+
+// Callback invoked when a #ident or #sccs directive is read.
+void PPCallbacksTracker::Ident(clang::SourceLocation Loc,
+                               const std::string &Str) {
+  beginCallback("Ident");
+  appendArgument("Loc", Loc);
+  appendArgument("Path", Str);
+}
+
+// Callback invoked when start reading any pragma directive.
+void
+PPCallbacksTracker::PragmaDirective(clang::SourceLocation Loc,
+                                    clang::PragmaIntroducerKind Introducer) {
+  beginCallback("PragmaDirective");
+  appendArgument("Loc", Loc);
+  appendArgument("Path", Introducer, PragmaIntroducerKindStrings);
+}
+
+// Callback invoked when a #pragma comment directive is read.
+void PPCallbacksTracker::PragmaComment(clang::SourceLocation Loc,
+                                       const clang::IdentifierInfo *Kind,
+                                       const std::string &Str) {
+  beginCallback("PragmaComment");
+  appendArgument("Loc", Loc);
+  appendArgument("Kind", Kind);
+  appendArgument("Str", Str);
+}
+
+// Callback invoked when a #pragma detect_mismatch directive is
+// read.
+void PPCallbacksTracker::PragmaDetectMismatch(clang::SourceLocation Loc,
+                                              const std::string &Name,
+                                              const std::string &Value) {
+  beginCallback("PragmaDetectMismatch");
+  appendArgument("Loc", Loc);
+  appendArgument("Name", Name);
+  appendArgument("Value", Value);
+}
+
+// Callback invoked when a #pragma clang __debug directive is read.
+void PPCallbacksTracker::PragmaDebug(clang::SourceLocation Loc,
+                                     llvm::StringRef DebugType) {
+  beginCallback("PragmaDebug");
+  appendArgument("Loc", Loc);
+  appendArgument("DebugType", DebugType);
+}
+
+// Callback invoked when a #pragma message directive is read.
+void PPCallbacksTracker::PragmaMessage(
+    clang::SourceLocation Loc, llvm::StringRef Namespace,
+    clang::PPCallbacks::PragmaMessageKind Kind, llvm::StringRef Str) {
+  beginCallback("PragmaMessage");
+  appendArgument("Loc", Loc);
+  appendArgument("Namespace", Namespace);
+  appendArgument("Kind", Kind, PragmaMessageKindStrings);
+  appendArgument("Str", Str);
+}
+
+// Callback invoked when a #pragma gcc dianostic push directive
+// is read.
+void PPCallbacksTracker::PragmaDiagnosticPush(clang::SourceLocation Loc,
+                                              llvm::StringRef Namespace) {
+  beginCallback("PragmaDiagnosticPush");
+  appendArgument("Loc", Loc);
+  appendArgument("Namespace", Namespace);
+}
+
+// Callback invoked when a #pragma gcc dianostic pop directive
+// is read.
+void PPCallbacksTracker::PragmaDiagnosticPop(clang::SourceLocation Loc,
+                                             llvm::StringRef Namespace) {
+  beginCallback("PragmaDiagnosticPop");
+  appendArgument("Loc", Loc);
+  appendArgument("Namespace", Namespace);
+}
+
+// Callback invoked when a #pragma gcc dianostic directive is read.
+void PPCallbacksTracker::PragmaDiagnostic(clang::SourceLocation Loc,
+                                          llvm::StringRef Namespace,
+                                          clang::diag::Mapping Mapping,
+                                          llvm::StringRef Str) {
+  beginCallback("PragmaDiagnostic");
+  appendArgument("Loc", Loc);
+  appendArgument("Namespace", Namespace);
+  appendArgument("Mapping", Mapping, MappingStrings);
+  appendArgument("Str", Str);
+}
+
+// Called when an OpenCL extension is either disabled or
+// enabled with a pragma.
+void PPCallbacksTracker::PragmaOpenCLExtension(
+    clang::SourceLocation NameLoc, const clang::IdentifierInfo *Name,
+    clang::SourceLocation StateLoc, unsigned State) {
+  beginCallback("PragmaOpenCLExtension");
+  appendArgument("NameLoc", NameLoc);
+  appendArgument("Name", Name);
+  appendArgument("StateLoc", StateLoc);
+  appendArgument("State", (int)State);
+}
+
+// Callback invoked when a #pragma warning directive is read.
+void PPCallbacksTracker::PragmaWarning(clang::SourceLocation Loc,
+                                       llvm::StringRef WarningSpec,
+                                       llvm::ArrayRef<int> Ids) {
+  beginCallback("PragmaWarning");
+  appendArgument("Loc", Loc);
+  appendArgument("WarningSpec", WarningSpec);
+
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << "[";
+  for (int i = 0, e = Ids.size(); i != e; ++i) {
+    if (i)
+      SS << ", ";
+    SS << Ids[i];
+  }
+  appendArgument("Ids", SS.str());
+}
+
+// Callback invoked when a #pragma warning(push) directive is read.
+void PPCallbacksTracker::PragmaWarningPush(clang::SourceLocation Loc,
+                                           int Level) {
+  beginCallback("PragmaWarningPush");
+  appendArgument("Loc", Loc);
+  appendArgument("Level", Level);
+}
+
+// Callback invoked when a #pragma warning(pop) directive is read.
+void PPCallbacksTracker::PragmaWarningPop(clang::SourceLocation Loc) {
+  beginCallback("PragmaWarningPop");
+  appendArgument("Loc", Loc);
+}
+
+// Called by Preprocessor::HandleMacroExpandedIdentifier when a
+// macro invocation is found.
+void
+PPCallbacksTracker::MacroExpands(const clang::Token &MacroNameTok,
+                                 const clang::MacroDirective *MacroDirective,
+                                 clang::SourceRange Range,
+                                 const clang::MacroArgs *Args) {
+  beginCallback("MacroExpands");
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+  appendArgument("Range", Range);
+  appendArgument("Args", Args);
+}
+
+// Hook called whenever a macro definition is seen.
+void
+PPCallbacksTracker::MacroDefined(const clang::Token &MacroNameTok,
+                                 const clang::MacroDirective *MacroDirective) {
+  beginCallback("MacroDefined");
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+}
+
+// Hook called whenever a macro #undef is seen.
+void PPCallbacksTracker::MacroUndefined(
+    const clang::Token &MacroNameTok,
+    const clang::MacroDirective *MacroDirective) {
+  beginCallback("MacroUndefined");
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+}
+
+// Hook called whenever the 'defined' operator is seen.
+void PPCallbacksTracker::Defined(const clang::Token &MacroNameTok,
+                                 const clang::MacroDirective *MacroDirective,
+                                 clang::SourceRange Range) {
+  beginCallback("Defined");
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+  appendArgument("Range", Range);
+}
+
+// Hook called when a source range is skipped.
+void PPCallbacksTracker::SourceRangeSkipped(clang::SourceRange Range) {
+  beginCallback("SourceRangeSkipped");
+  appendArgument("Range", Range);
+}
+
+// Hook called whenever an #if is seen.
+void PPCallbacksTracker::If(clang::SourceLocation Loc,
+                            clang::SourceRange ConditionRange,
+                            bool ConditionValue) {
+  beginCallback("If");
+  appendArgument("Loc", Loc);
+  appendArgument("ConditionRange", ConditionRange);
+  appendArgument("ConditionValue", ConditionValue);
+}
+
+// Hook called whenever an #elif is seen.
+void PPCallbacksTracker::Elif(clang::SourceLocation Loc,
+                              clang::SourceRange ConditionRange,
+                              bool ConditionValue,
+                              clang::SourceLocation IfLoc) {
+  beginCallback("Elif");
+  appendArgument("Loc", Loc);
+  appendArgument("ConditionRange", ConditionRange);
+  appendArgument("ConditionValue", ConditionValue);
+  appendArgument("IfLoc", IfLoc);
+}
+
+// Hook called whenever an #ifdef is seen.
+void PPCallbacksTracker::Ifdef(clang::SourceLocation Loc,
+                               const clang::Token &MacroNameTok,
+                               const clang::MacroDirective *MacroDirective) {
+  beginCallback("Ifdef");
+  appendArgument("Loc", Loc);
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+}
+
+// Hook called whenever an #ifndef is seen.
+void PPCallbacksTracker::Ifndef(clang::SourceLocation Loc,
+                                const clang::Token &MacroNameTok,
+                                const clang::MacroDirective *MacroDirective) {
+  beginCallback("Ifndef");
+  appendArgument("Loc", Loc);
+  appendArgument("MacroNameTok", MacroNameTok);
+  appendArgument("MacroDirective", MacroDirective);
+}
+
+// Hook called whenever an #else is seen.
+void PPCallbacksTracker::Else(clang::SourceLocation Loc,
+                              clang::SourceLocation IfLoc) {
+  beginCallback("Else");
+  appendArgument("Loc", Loc);
+  appendArgument("IfLoc", IfLoc);
+}
+
+// Hook called whenever an #endif is seen.
+void PPCallbacksTracker::Endif(clang::SourceLocation Loc,
+                               clang::SourceLocation IfLoc) {
+  beginCallback("Endif");
+  appendArgument("Loc", Loc);
+  appendArgument("IfLoc", IfLoc);
+}
+
+// Helper functions.
+
+// Start a new callback.
+void PPCallbacksTracker::beginCallback(const char *Name) {
+  DisableTrace = Ignore.count(std::string(Name));
+  if (DisableTrace)
+    return;
+  CallbackCalls.push_back(CallbackCall(Name));
+}
+
+// Append a bool argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name, bool Value) {
+  appendArgument(Name, (Value ? "true" : "false"));
+}
+
+// Append an int argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name, int Value) {
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << Value;
+  appendArgument(Name, SS.str());
+}
+
+// Append a string argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name, const char *Value) {
+  if (DisableTrace)
+    return;
+  CallbackCalls.back().Arguments.push_back(Argument(Name, Value));
+}
+
+// Append a string object argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        llvm::StringRef Value) {
+  appendArgument(Name, Value.str());
+}
+
+// Append a string object argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const std::string &Value) {
+  appendArgument(Name, Value.c_str());
+}
+
+// Append a token argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::Token &Value) {
+  appendArgument(Name, PP.getSpelling(Value));
+}
+
+// Append an enum argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name, int Value,
+                                        const char *Strings[]) {
+  appendArgument(Name, Strings[Value]);
+}
+
+// Append a FileID argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name, clang::FileID Value) {
+  if (Value.isInvalid()) {
+    appendArgument(Name, "(invalid)");
+    return;
+  }
+  const clang::FileEntry *FileEntry =
+      PP.getSourceManager().getFileEntryForID(Value);
+  if (FileEntry == 0) {
+    appendArgument(Name, "(getFileEntryForID failed)");
+    return;
+  }
+  appendFilePathArgument(Name, FileEntry->getName());
+}
+
+// Append a FileEntry argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::FileEntry *Value) {
+  if (Value == 0) {
+    appendArgument(Name, "(null)");
+    return;
+  }
+  appendFilePathArgument(Name, Value->getName());
+}
+
+// Append a SourceLocation argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        clang::SourceLocation Value) {
+  if (Value.isInvalid()) {
+    appendArgument(Name, "(invalid)");
+    return;
+  }
+  appendArgument(Name, getSourceLocationString(PP, Value).c_str());
+}
+
+// Append a SourceRange argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        clang::SourceRange Value) {
+  if (DisableTrace)
+    return;
+  if (Value.isInvalid()) {
+    appendArgument(Name, "(invalid)");
+    return;
+  }
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << "[" << getSourceLocationString(PP, Value.getBegin()) << ", "
+     << getSourceLocationString(PP, Value.getEnd()) << "]";
+  appendArgument(Name, SS.str());
+}
+
+// Append a CharSourceRange argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        clang::CharSourceRange Value) {
+  if (Value.isInvalid()) {
+    appendArgument(Name, "(invalid)");
+    return;
+  }
+  appendArgument(Name, getSourceString(Value).str().c_str());
+}
+
+// Append a SourceLocation argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        clang::ModuleIdPath Value) {
+  if (DisableTrace)
+    return;
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << "[";
+  for (int I = 0, E = Value.size(); I != E; ++I) {
+    if (I)
+      SS << ", ";
+    SS << "{"
+       << "Name: " << Value[I].first->getName() << ", "
+       << "Loc:" << getSourceLocationString(PP, Value[I].second) << "}";
+  }
+  SS << "]";
+  appendArgument(Name, SS.str());
+}
+
+// Append an IdentifierInfo argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::IdentifierInfo *Value) {
+  if (!Value) {
+    appendArgument(Name, "(null)");
+    return;
+  }
+  appendArgument(Name, Value->getName().str().c_str());
+}
+
+// Append a MacroDirective argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::MacroDirective *Value) {
+  if (!Value) {
+    appendArgument(Name, "(null)");
+    return;
+  }
+  appendArgument(Name, MacroDirectiveKindStrings[Value->getKind()]);
+}
+
+// Append a MacroArgs argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::MacroArgs *Value) {
+  if (!Value) {
+    appendArgument(Name, "(null)");
+    return;
+  }
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << "[";
+  // The argument tokens might include end tokens, so we reflect how
+  // how getUnexpArgument provides the arguments.
+  for (int I = 0, E = Value->getNumArguments(); I < E; ++I) {
+    const clang::Token *Current = Value->getUnexpArgument(I);
+    int TokenCount = Value->getArgLength(Current) + 1; // include EOF
+    E -= TokenCount;
+    if (I)
+      SS << ", ";
+    // We're assuming tokens are contiguous, as otherwise we have no
+    // other way to get at them.
+    --TokenCount;
+    for (int TokenIndex = 0; TokenIndex < TokenCount; ++TokenIndex, ++Current) {
+      if (TokenIndex)
+        SS << " ";
+      // We need to be careful here because the arguments might not be legal in
+      // YAML, so we use the token name for anything but identifiers and
+      // numeric literals.
+      if (Current->isAnyIdentifier() ||
+          Current->is(clang::tok::numeric_constant)) {
+        SS << PP.getSpelling(*Current);
+      } else {
+        SS << "<" << Current->getName() << ">";
+      }
+    }
+  }
+  SS << "]";
+  appendArgument(Name, SS.str());
+}
+
+// Append a Module argument to the top trace item.
+void PPCallbacksTracker::appendArgument(const char *Name,
+                                        const clang::Module *Value) {
+  if (!Value) {
+    appendArgument(Name, "(null)");
+    return;
+  }
+  appendArgument(Name, Value->Name.c_str());
+}
+
+// Append a double-quoted argument to the top trace item.
+void PPCallbacksTracker::appendQuotedArgument(const char *Name,
+                                              std::string &Value) {
+  std::string Str;
+  llvm::raw_string_ostream SS(Str);
+  SS << "\"" << Value << "\"";
+  appendArgument(Name, SS.str());
+}
+
+// Append a double-quoted file path argument to the top trace item.
+void PPCallbacksTracker::appendFilePathArgument(const char *Name,
+                                                llvm::StringRef Value) {
+  std::string Path(Value);
+  // YAML treats backslash as escape, so use forward slashes.
+  std::replace(Path.begin(), Path.end(), '\\', '/');
+  appendQuotedArgument(Name, Path);
+}
+
+// Get the raw source string of the range.
+llvm::StringRef
+PPCallbacksTracker::getSourceString(clang::CharSourceRange Range) {
+  const char *B = PP.getSourceManager().getCharacterData(Range.getBegin());
+  const char *E = PP.getSourceManager().getCharacterData(Range.getEnd());
+  return llvm::StringRef(B, E - B);
+}

pp-trace/PPTrace.cpp

+//===--- tools/pp-trace/PPTrace.cpp - Clang preprocessor tracer -----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements pp-trace, a tool for displaying a textual trace
+// of the Clang preprocessor activity.  It's based on a derivation of the
+// PPCallbacks class, that once registerd with Clang, receives callback calls
+// to its virtual members, and outputs the information passed to the callbacks
+// in a high-level YAML format.
+//
+// The pp-trace tool also serves as the basis for a test of the PPCallbacks
+// mechanism.
+//
+// The pp-trace tool supports the following general command line format:
+//
+//    pp-trace [pp-trace options] (source file(s)) [compiler options]
+//
+// Basically you put the pp-trace options first, then the source file or files,
+// and then any options you want to pass to the compiler.
+//
+// These are the pp-trace options:
+//
+//    -ignore (callback list)     Don't display output for a comma-separated
+//                                list of callbacks, i.e.:
+//                                  -ignore "FileChanged,InclusionDirective"
+//
+//    -output (file|-)            Output trace to the given file (or stdout
+//                                for "-") in a YAML format, e.g.:
+//
+//                                  ---
+//                                  - Callback: Name
+//                                    Argument1: Value1
+//                                    Argument2: Value2
+//                                  (etc.)
+//                                  ...
+//
+//===----------------------------------------------------------------------===//
+
+#include "PPCallbacksTracker.h"
+#include "clang/AST/ASTConsumer.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Driver/Options.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/OwningPtr.h"
+#include "llvm/Config/config.h"
+#include "llvm/Option/Arg.h"
+#include "llvm/Option/ArgList.h"
+#include "llvm/Option/OptTable.h"
+#include "llvm/Option/Option.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include <algorithm>
+#include <fstream>
+#include <iterator>
+#include <string>
+#include <vector>
+
+using namespace clang;
+using namespace clang::driver;
+using namespace clang::driver::options;
+using namespace clang::tooling;
+using namespace llvm;
+using namespace llvm::opt;
+
+// Options:
+
+// Collect the source files.
+cl::list<std::string> SourcePaths(cl::Positional,
+                                  cl::desc("<source0> [... <sourceN>]"),
+                                  cl::OneOrMore);
+
+// Option to specify a list or one or more callback names to ignore.
+cl::opt<std::string> IgnoreCallbacks(
+    "ignore", cl::init(""),
+    cl::desc("Ignore callbacks, i.e. \"Callback1, Callback2...\"."));
+
+// Option to specify the trace output file name.
+cl::opt<std::string> OutputFileName(
+    "output", cl::init(""),
+    cl::desc("Output trace to the given file name or '-' for stdout."));
+
+// Collect all other arguments, which will be passed to the front end.
+cl::list<std::string>
+CC1Arguments(cl::ConsumeAfter,
+             cl::desc("<arguments to be passed to front end>..."));
+
+// Frontend action stuff:
+
+// Consumer is responsible for setting up the callbacks.
+class PPTraceConsumer : public ASTConsumer {
+public:
+  PPTraceConsumer(SmallSet<std::string, 4> &Ignore,
+                  std::vector<CallbackCall> &CallbackCalls, Preprocessor &PP)
+      : CallbacksTracker(new PPCallbacksTracker(Ignore, CallbackCalls, PP)) {
+    PP.addPPCallbacks(CallbacksTracker); // Takes ownership.
+  }
+
+private:
+  PPCallbacksTracker *CallbacksTracker; // Not owned here.
+};
+
+class PPTraceAction : public SyntaxOnlyAction {
+public:
+  PPTraceAction(SmallSet<std::string, 4> &Ignore,
+                std::vector<CallbackCall> &CallbackCalls)
+      : Ignore(Ignore), CallbackCalls(CallbackCalls) {}
+
+protected:
+  virtual clang::ASTConsumer *CreateASTConsumer(CompilerInstance &CI,
+                                                StringRef InFile) {
+    return new PPTraceConsumer(Ignore, CallbackCalls, CI.getPreprocessor());
+  }
+
+private:
+  SmallSet<std::string, 4> &Ignore;
+  std::vector<CallbackCall> &CallbackCalls;
+};
+
+class PPTraceFrontendActionFactory : public FrontendActionFactory {
+public:
+  PPTraceFrontendActionFactory(SmallSet<std::string, 4> &Ignore,
+                               std::vector<CallbackCall> &CallbackCalls)
+      : Ignore(Ignore), CallbackCalls(CallbackCalls) {}
+
+  virtual PPTraceAction *create() {
+    return new PPTraceAction(Ignore, CallbackCalls);
+  }
+
+private:
+  SmallSet<std::string, 4> &Ignore;
+  std::vector<CallbackCall> &CallbackCalls;
+};
+
+// Output the trace given its data structure and a stream.
+int outputPPTrace(std::vector<CallbackCall> &CallbackCalls,
+                  llvm::raw_ostream &OS) {
+  // Mark start of document.
+  OS << "---\n";
+
+  for (std::vector<CallbackCall>::const_iterator I = CallbackCalls.begin(),
+                                                 E = CallbackCalls.end();
+       I != E; ++I) {
+    const CallbackCall &Callback = *I;
+    OS << "- Callback: " << Callback.Name << "\n";
+
+    for (std::vector<Argument>::const_iterator AI = Callback.Arguments.begin(),
+                                               AE = Callback.Arguments.end();
+         AI != AE; ++AI) {
+      const Argument &Arg = *AI;
+      OS << "  " << Arg.Name << ": " << Arg.Value << "\n";
+    }
+  }
+
+  // Mark end of document.
+  OS << "...\n";
+
+  return 0;
+}
+
+// Program entry point.
+int main(int Argc, const char **Argv) {
+
+  // Parse command line.
+  cl::ParseCommandLineOptions(Argc, Argv, "pp-trace.\n");
+
+  // Parse the IgnoreCallbacks list into strings.
+  SmallVector<StringRef, 32> IgnoreCallbacksStrings;
+  StringRef(IgnoreCallbacks).split(IgnoreCallbacksStrings, ",",
+                                   /*MaxSplit=*/ -1, /*KeepEmpty=*/false);
+  SmallSet<std::string, 4> Ignore;
+  for (SmallVector<StringRef, 32>::iterator I = IgnoreCallbacksStrings.begin(),
+                                            E = IgnoreCallbacksStrings.end();
+       I != E; ++I)
+    Ignore.insert(*I);
+
+  // Create the compilation database.
+  SmallString<256> PathBuf;
+  sys::fs::current_path(PathBuf);
+  OwningPtr<CompilationDatabase> Compilations;
+  Compilations.reset(
+      new FixedCompilationDatabase(Twine(PathBuf), CC1Arguments));
+
+  // Store the callback trace information here.
+  std::vector<CallbackCall> CallbackCalls;
+
+  // Create the tool and run the compilation.
+  ClangTool Tool(*Compilations, SourcePaths);
+  int HadErrors =
+      Tool.run(new PPTraceFrontendActionFactory(Ignore, CallbackCalls));
+
+  // If we had errors, exit early.
+  if (HadErrors)
+    return HadErrors;
+
+  // Do the output.
+  // If file name is "-", output to stdout.
+  if (!OutputFileName.size() || (OutputFileName == "-")) {
+    HadErrors = outputPPTrace(CallbackCalls, llvm::outs());
+  } else {
+    // Set up output file.
+    std::string Error;
+    llvm::tool_output_file Out(OutputFileName.c_str(), Error);
+    if (!Error.empty()) {
+      llvm::errs() << "pp-trace: error creating " << OutputFileName << ":"
+                   << Error << "\n";
+      return 1;
+    }
+
+    HadErrors = outputPPTrace(CallbackCalls, Out.os());
+
+    // Tell tool_output_file that we want to keep the file.
+    if (HadErrors == 0)
+      Out.keep();
+  }
+
+  return HadErrors;
+}

test/pp-trace/Input/Level1A.h

+#include "Level2A.h"
+#define MACRO_1A 1

test/pp-trace/Input/Level1B.h

+#define MACRO_1B 1

test/pp-trace/Input/Level2A.h

+#define MACRO_2A 1

test/pp-trace/pp-trace-include.cpp

+// RUN: pp-trace %s -undef -target x86_64 -std=c++11 | FileCheck --strict-whitespace %s
+
+#include "Input/Level1A.h"
+#include "Input/Level1B.h"
+
+// CHECK: ---
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-include.cpp:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "<built-in>:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "<built-in>:1:1"
+// CHECK-NEXT:   Reason: RenameFile
+// CHECK-NEXT:   FileType: C_System
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_HOSTED__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __cplusplus
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_UTF_16__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_UTF_32__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "<command line>:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "<built-in>:1:1"
+// CHECK-NEXT:   Reason: ExitFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-include.cpp:1:1"
+// CHECK-NEXT:   Reason: ExitFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (getFileEntryForID failed)
+// CHECK-NEXT: - Callback: InclusionDirective
+// CHECK-NEXT:   IncludeTok: include
+// CHECK-NEXT:   FileName: "Input/Level1A.h"
+// CHECK-NEXT:   IsAngled: false
+// CHECK-NEXT:   FilenameRange: "Input/Level1A.h"
+// CHECK-NEXT:   File: "{{.*}}{{[/\\]}}Input/Level1A.h"
+// CHECK-NEXT:   SearchPath: "{{.*}}{{[/\\]}}pp-trace"
+// CHECK-NEXT:   RelativePath: "Input/Level1A.h"
+// CHECK-NEXT:   Imported: (null)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}Input/Level1A.h:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: InclusionDirective
+// CHECK-NEXT:   IncludeTok: include
+// CHECK-NEXT:   FileName: "Level2A.h"
+// CHECK-NEXT:   IsAngled: false
+// CHECK-NEXT:   FilenameRange: "Level2A.h"
+// CHECK-NEXT:   File: "{{.*}}{{[/\\]}}Input/Level2A.h"
+// CHECK-NEXT:   SearchPath: "{{.*}}{{[/\\]}}Input"
+// CHECK-NEXT:   RelativePath: "Level2A.h"
+// CHECK-NEXT:   Imported: (null)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}Input/Level2A.h:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: MACRO_2A
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}Input/Level1A.h:2:1"
+// CHECK-NEXT:   Reason: ExitFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: "{{.*}}{{[/\\]}}Input/Level2A.h"
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: MACRO_1A
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-include.cpp:4:1"
+// CHECK-NEXT:   Reason: ExitFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: "{{.*}}{{[/\\]}}Input/Level1A.h"
+// CHECK-NEXT: - Callback: InclusionDirective
+// CHECK-NEXT:   IncludeTok: include
+// CHECK-NEXT:   FileName: "Input/Level1B.h"
+// CHECK-NEXT:   IsAngled: false
+// CHECK-NEXT:   FilenameRange: "Input/Level1B.h"
+// CHECK-NEXT:   File: "{{.*}}{{[/\\]}}Input/Level1B.h"
+// CHECK-NEXT:   SearchPath: "{{.*}}{{[/\\]}}pp-trace"
+// CHECK-NEXT:   RelativePath: "Input/Level1B.h"
+// CHECK-NEXT:   Imported: (null)
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}Input/Level1B.h:1:1"
+// CHECK-NEXT:   Reason: EnterFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: (invalid)
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: MACRO_1B
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: FileChanged
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-include.cpp:5:1"
+// CHECK-NEXT:   Reason: ExitFile
+// CHECK-NEXT:   FileType: C_User
+// CHECK-NEXT:   PrevFID: "{{.*}}{{[/\\]}}Input/Level1B.h"
+// CHECK-NEXT: - Callback: EndOfMainFile
+// CHECK-NEXT: ...

test/pp-trace/pp-trace-macro.cpp

+// RUN: pp-trace -ignore FileChanged %s -undef -target x86_64 -std=c++11 | FileCheck --strict-whitespace %s
+
+#define MACRO 1
+int i = MACRO;
+#if defined(MACRO)
+#endif
+#undef MACRO
+#if defined(MACRO)
+#endif
+#define FUNCMACRO(ARG1) ARG1
+int j = FUNCMACRO(1);
+#define X X_IMPL(a+y,b) X_IMPL2(c)
+#define X_IMPL(p1,p2)
+#define X_IMPL2(p1)
+X
+
+// CHECK: ---
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_HOSTED__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __cplusplus
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_UTF_16__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: __STDC_UTF_32__
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: MACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroExpands
+// CHECK-NEXT:   MacroNameTok: MACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:4:9", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:4:9"]
+// CHECK-NEXT:   Args: (null)
+// CHECK-NEXT: - Callback: Defined
+// CHECK-NEXT:   MacroNameTok: MACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:5:5", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:5:19"]
+// CHECK-NEXT: - Callback: If
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:5:2"
+// CHECK-NEXT:   ConditionRange: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:5:4", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:6:1"]
+// CHECK-NEXT:   ConditionValue: true
+// CHECK-NEXT: - Callback: Endif
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:6:2"
+// CHECK-NEXT:   IfLoc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:5:2"
+// CHECK-NEXT: - Callback: MacroUndefined
+// CHECK-NEXT:   MacroNameTok: MACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: Defined
+// CHECK-NEXT:   MacroNameTok: MACRO
+// CHECK-NEXT:   MacroDirective: (null)
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:5", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:19"]
+// CHECK-NEXT: - Callback: If
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:2"
+// CHECK-NEXT:   ConditionRange: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:4", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:9:1"]
+// CHECK-NEXT:   ConditionValue: false
+// CHECK-NEXT: - Callback: Endif
+// CHECK-NEXT:   Loc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:9:2"
+// CHECK-NEXT:   IfLoc: "{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:2"
+// CHECK-NEXT: - Callback: SourceRangeSkipped
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:8:2", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:9:2"]
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: FUNCMACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroExpands
+// CHECK-NEXT:   MacroNameTok: FUNCMACRO
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:11:9", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:11:20"]
+// CHECK-NEXT:   Args: [1]
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: X
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: X_IMPL
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroDefined
+// CHECK-NEXT:   MacroNameTok: X_IMPL2
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT: - Callback: MacroExpands
+// CHECK-NEXT:   MacroNameTok: X
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: ["{{.*}}{{[/\\]}}pp-trace-macro.cpp:15:1", "{{.*}}{{[/\\]}}pp-trace-macro.cpp:15:1"]
+// CHECK-NEXT:   Args: (null)
+// CHECK-NEXT: - Callback: MacroExpands
+// CHECK-NEXT:   MacroNameTok: X_IMPL
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: [(nonfile), (nonfile)]
+// CHECK-NEXT:   Args: [a plus y, b]
+// CHECK-NEXT: - Callback: MacroExpands
+// CHECK-NEXT:   MacroNameTok: X_IMPL2
+// CHECK-NEXT:   MacroDirective: MD_Define
+// CHECK-NEXT:   Range: [(nonfile), (nonfile)]
+// CHECK-NEXT:   Args: [c]
+// CHECK-NEXT: - Callback: EndOfMainFile
+// CHECK-NEXT: ...