PPCallbacksTest expansion - part 1

[jtsoftware]

Summary

This patch implements a test mechanism for the PPCallbacks mechanism that I've added to the existing tests. It's based on creating a derivation of PPCallbacks that records information passed to each callback in a high-level string format, for later comparison against an array of expected patterns. It allows testing of both the callback order as well as the callback argument content.

This patch provides the basic mechanism, but only implements two new test functions using it, for just the macro-based and conditional-based callbacks, as a proof of concept. If you approve this basic mechanism, I will add additional tests covering other callbacks and situations in subsequent patches.

[silvas]

Turning the preprocessor decisions into a serialized textual format is a great idea!

Though this seems like it would be better done as an independent tool. I.e. one that reads its file on stdin (or as an argument) and prints out this information, for use in conjunction with FileCheck. That would allow idiomatic LLVM FileCheck testing similar to:

; RUN: opt < %s -sroa -S | FileCheck %s

(that was from llvm/test/Transforms/SROA/basictest.ll but there are innumerable other examples).

so the usage would be something like:

; RUN: pp-trace < %s | FileCheck %s

That would also completely offload the burden of the pattern matching/specification to FileCheck, and integrate with developers' existing familiarity with FileCheck.

Also, I'm not a huge fan of this ad-hoc serialization. Could you please use YAMLIO http://llvm.org/docs/YamlIO.html and have the output be YAML? YAMLIO is really simple to use once you get the hang of it. Look at LLVM's r183711 for a pretty minimal example; the code there only uses it for input, but YAMLIO is inherently bidirectional so output is just a matter of using << rather than >>. Feel free to ask me questions too.

YAML output would be *amazing*, since then it would be possible to write ad-hoc simple tools in any language (I'm thinking in particular scripting languages), for e.g. quickly checking assumptions, scoping out the prevalence of a particular pattern, etc.. Many languages have YAML parsers, and turning YAML into equivalent JSON is like a 10-line Python program, and then every language can access it.

[kimgr]

I don't know my way around FileCheck, so I can't spek to the benefit for testing (it sounds nice, though), but a standalone tool would be useful for learning and troubleshooting, PPCallbacks just like -ast-dump is for AST traversals. I literally started writing one a few nights ago, but I never have enough free time to finish anything :-/

I would appreciate it, is what I'm trying to say.

[kimgr]

Sean, John,

[jtsoftware]

I created the tool pptrace as a basis for doing these kinds of tests instead.

PPCallbacksTest.cpp

 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/MacroArgs.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Parse/Parser.h"
 #include "clang/Sema/Sema.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/ASTConsumer.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/Path.h"
 #include "gtest/gtest.h"
+#include <stdarg.h>
 
 using namespace llvm;
 using namespace llvm::sys;
 using namespace clang;
 
 namespace {
 
 // Stub out module loading.
public:
   bool IsAngled;
   CharSourceRange FilenameRange;
   const FileEntry* File;
   SmallString<16> SearchPath;
   SmallString<16> RelativePath;
   const Module* Imported;
 };
 
+// Utility functions.
+
+// Get a "file:line:column" source location string.
+static std::string getSourceLocationString(clang::Preprocessor &PP,
+                                           clang::SourceLocation Loc) {
+  if (Loc.isInvalid())
+    return std::string("(none)");
+  else
+    return Loc.printToString(PP.getSourceManager());
+}
+
+// Enum string tables.
+
+// FileChangeReason strings.
+static const char *FileChangeReasonStrings[] = {
+  "EnterFile", "ExitFile", "SystemHeaderPragma", "RenameFile"
+};
+
+// CharacteristicKind strings.
+static const char *CharacteristicKindStrings[] = { "C_User", "C_System",
+                                                   "C_ExternCSystem" };
+
+// MacroDirective::Kind strings.
+static const char *MacroDirectiveKindStrings[] = { "MD_Define", "MD_Undefine",
+                                                   "MD_Visibility" };
+
+// PragmaIntroducerKind strings.
+static const char *PragmaIntroducerKindStrings[] = { "PIK_HashPragma",
+                                                     "PIK__Pragma",
+                                                     "PIK___pragma" };
+
+// PragmaMessageKind strings.
+static const char *PragmaMessageKindStrings[] = { "PMK_Message", "PMK_Warning",
+                                                  "PMK_Error" };
+
+// Mapping strings.
+static const char *MappingStrings[] = { "0",           "MAP_IGNORE",
+                                        "MAP_WARNING", "MAP_ERROR",
+                                        "MAP_FATAL" };
+
+// PPCallbacks tracking derivation.
+// We use this derivation of PPCallbacks to record a trace from
+// the callbacks, for latter test pattern matching.
+class PPCallbacksTracker : public PPCallbacks {
+public:
+  // Constructor.
+  // Trace - The trace buffer.  One string per callback.
+  // IgnoreCallbacks - Names of callbacks to ignore. Null terminates the list.
+  // PP - The preprocessor.  Needed for getting some argument strings.
+  PPCallbacksTracker(std::vector<std::string> &Trace,
+                     const char **IgnoreCallbacks, Preprocessor &PP)
+      : Trace(Trace), PP(PP), DisableTrace(false) {
+    if (IgnoreCallbacks) {
+      while (*IgnoreCallbacks)
+        Ignore.insert(std::string(*IgnoreCallbacks++));
+    }
+  }
+
+  // Callback functions.
+
+  // Callback invoked whenever a source file is entered or exited.
+  // Loc Indicates the new location.
+  // PrevFID the file that was exited if Reason is ExitFile.
+  void FileChanged(SourceLocation Loc, FileChangeReason Reason,
+                   SrcMgr::CharacteristicKind FileType,
+                   FileID PrevFID = FileID()) {
+    BeginCallback("FileChanged");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Reason", Reason, FileChangeReasonStrings);
+    AppendArgument("FileType", FileType, CharacteristicKindStrings);
+    AppendArgument("PrevFID", PrevFID);
+  }
+
+  // Callback invoked whenever a source file is skipped as the result
+  // of header guard optimization.
+  // ParentFile - The file that #included the skipped file.
+  // FilenameTok - The token in ParentFile that indicates the
+  // skipped file.
+  void FileSkipped(const FileEntry &ParentFile, const Token &FilenameTok,
+                   SrcMgr::CharacteristicKind FileType) {
+    BeginCallback("FileChanged");
+    AppendArgument("ParentFile", &ParentFile);
+    AppendArgument("FilenameTok", FilenameTok);
+    AppendArgument("FileType", FileType, CharacteristicKindStrings);
+  }
+
+  // Callback invoked whenever an inclusion directive results in a
+  // file-not-found error.
+  // FileName - The name of the file being included, as written in the
+  // source code.
+  // RecoveryPath - If this client indicates that it can recover from
+  // this missing file, the client should set this as an additional header
+  // search patch.
+  // Returns true to indicate that the preprocessor should attempt to recover
+  // by adding RecoveryPath as a header search path.
+  bool FileNotFound(StringRef FileName, SmallVectorImpl<char> &RecoveryPath) {
+    BeginCallback("FileNotFound");
+    AppendArgument("FileName", FileName);
+    return false;
+  }
+
+  // Callback invoked whenever an inclusion directive of
+  // any kind (#include, #import, etc.) has been processed, regardless
+  // of whether the inclusion will actually result in an inclusion.
+  // HashLoc - The location of the '#' that starts the inclusion
+  // directive.
+  // IncludeTok - The token that indicates the kind of inclusion
+  // directive, e.g., 'include' or 'import'.
+  // FileName - The name of the file being included, as written in the
+  // source code.
+  // IsAngled - Whether the file name was enclosed in angle brackets;
+  // otherwise, it was enclosed in quotes.
+  // FilenameRange - The character range of the quotes or angle brackets
+  // for the written file name.
+  // File - The actual file that may be included by this inclusion
+  // directive.
+  // SearchPath - Contains the search path which was used to find the file
+  // in the file system. If the file was found via an absolute include path,
+  // SearchPath will be empty. For framework includes, the SearchPath and
+  // RelativePath will be split up. For example, if an include of "Some/Some.h"
+  // is found via the framework path
+  // "path/to/Frameworks/Some.framework/Headers/Some.h", SearchPath will be
+  // "path/to/Frameworks/Some.framework/Headers" and RelativePath will be
+  // "Some.h".
+  // RelativePath - The path relative to SearchPath, at which the include
+  // file was found. This is equal to FileName except for framework includes.
+  // Imported - The module, whenever an inclusion directive was
+  // automatically turned into a module import or null otherwise.
+  void InclusionDirective(SourceLocation HashLoc, const Token &IncludeTok,
+                          StringRef FileName, bool IsAngled,
+                          CharSourceRange FilenameRange, const FileEntry *File,
+                          StringRef SearchPath, StringRef RelativePath,
+                          const Module *Imported) {
+    BeginCallback("InclusionDirective");
+    AppendArgument("IncludeTok", IncludeTok);
+    AppendArgument("FileName", FileName);
+    AppendArgument("IsAngled", IsAngled);
+    AppendArgument("FilenameRange", FilenameRange);
+    AppendArgument("File", File);
+    AppendArgument("SearchPath", SearchPath);
+    AppendArgument("RelativePath", RelativePath);
+    AppendArgument("Imported", Imported);
+  }
+
+  // Callback invoked whenever there was an explicit module-import
+  // syntax.
+  // ImportLoc - The location of import directive token.
+  // Path - The identifiers (and their locations) of the module
+  // "path", e.g., "std.vector" would be split into "std" and "vector".
+  // Imported - The imported module; can be null if importing failed.
+  void moduleImport(SourceLocation ImportLoc, ModuleIdPath Path,
+                    const Module *Imported) {
+    BeginCallback("moduleImport");
+    AppendArgument("ImportLoc", ImportLoc);
+    AppendArgument("Path", Path);
+    AppendArgument("Imported", Imported);
+  }
+
+  // Callback invoked when the end of the main file is reached.
+  // No subsequent callbacks will be made.
+  void EndOfMainFile() { BeginCallback("EndOfMainFile"); }
+
+  // Callback invoked when a #ident or #sccs directive is read.
+  // Loc - The location of the directive.
+  // str - The text of the directive.
+  void Ident(SourceLocation Loc, const std::string &Str) {
+    BeginCallback("Ident");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Path", Str);
+  }
+
+  // Callback invoked when start reading any pragma directive.
+  void PragmaDirective(SourceLocation Loc, PragmaIntroducerKind Introducer) {
+    BeginCallback("PragmaDirective");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Path", Introducer, PragmaIntroducerKindStrings);
+  }
+
+  // Callback invoked when a #pragma comment directive is read.
+  void PragmaComment(SourceLocation Loc, const IdentifierInfo *Kind,
+                     const std::string &Str) {
+    BeginCallback("PragmaComment");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Kind", Kind);
+    AppendArgument("Str", Str);
+  }
+
+  // Callback invoked when a #pragma detect_mismatch directive is
+  // read.
+  void PragmaDetectMismatch(SourceLocation Loc, const std::string &Name,
+                            const std::string &Value) {
+    BeginCallback("PragmaDetectMismatch");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Name", Name);
+    AppendArgument("Value", Value);
+  }
+
+  // Callback invoked when a #pragma clang __debug directive is read.
+  // Loc - The location of the debug directive.
+  // DebugType - The identifier following __debug.
+  void PragmaDebug(SourceLocation Loc, StringRef DebugType) {
+    BeginCallback("PragmaDetectMismatch");
+    AppendArgument("Loc", Loc);
+    AppendArgument("DebugType", DebugType);
+  }
+
+  // Callback invoked when a #pragma message directive is read.
+  // Loc - The location of the message directive.
+  // Namespace - The namespace of the message directive.
+  // Kind - The type of the message directive.
+  // Str - The text of the message directive.
+  void PragmaMessage(SourceLocation Loc, StringRef Namespace,
+                     PragmaMessageKind Kind, StringRef Str) {
+    BeginCallback("PragmaMessage");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Namespace", Namespace);
+    AppendArgument("Kind", Kind, PragmaMessageKindStrings);
+    AppendArgument("Str", Str);
+  }
+
+  // Callback invoked when a #pragma gcc dianostic push directive
+  // is read.
+  void PragmaDiagnosticPush(SourceLocation Loc, StringRef Namespace) {
+    BeginCallback("PragmaDiagnosticPush");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Namespace", Namespace);
+  }
+
+  // Callback invoked when a #pragma gcc dianostic pop directive
+  // is read.
+  void PragmaDiagnosticPop(SourceLocation Loc, StringRef Namespace) {
+    BeginCallback("PragmaDiagnosticPop");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Namespace", Namespace);
+  }
+
+  // Callback invoked when a #pragma gcc dianostic directive is read.
+  void PragmaDiagnostic(SourceLocation Loc, StringRef Namespace,
+                        diag::Mapping Mapping, StringRef Str) {
+    BeginCallback("PragmaDiagnosticPop");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Namespace", Namespace);
+    AppendArgument("Mapping", Mapping, MappingStrings);
+    AppendArgument("Str", Str);
+  }
+
+  // Called when an OpenCL extension is either disabled or
+  // enabled with a pragma.
+  void PragmaOpenCLExtension(SourceLocation NameLoc, const IdentifierInfo *Name,
+                             SourceLocation StateLoc, unsigned State) {
+    BeginCallback("PragmaOpenCLExtension");
+    AppendArgument("NameLoc", NameLoc);
+    AppendArgument("Name", Name);
+    AppendArgument("StateLoc", StateLoc);
+    AppendArgument("State", (int)State);
+  }
+
+  // Callback invoked when a #pragma warning directive is read.
+  void PragmaWarning(SourceLocation Loc, StringRef WarningSpec,
+                     ArrayRef<int> Ids) {
+    BeginCallback("PragmaWarning");
+    AppendArgument("Loc", Loc);
+    AppendArgument("WarningSpec", WarningSpec);
+
+    std::string IdsString;
+    for (ArrayRef<int>::iterator I = Ids.begin(), E = Ids.end(); I != E; ++I) {
+      char Buffer[64];
+      sprintf(Buffer, "%d, ", *I);
+      IdsString.append(Buffer);
+    }
+    AppendArgument("Ids", IdsString);
+  }
+
+  // Callback invoked when a #pragma warning(push) directive is read.
+  void PragmaWarningPush(SourceLocation Loc, int Level) {
+    BeginCallback("PragmaWarningPush");
+    AppendArgument("Loc", Loc);
+    AppendArgument("Level", Level);
+  }
+
+  // Callback invoked when a #pragma warning(pop) directive is read.
+  void PragmaWarningPop(SourceLocation Loc) {
+    BeginCallback("PragmaWarningPop");
+    AppendArgument("Loc", Loc);
+  }
+
+  // Called by Preprocessor::HandleMacroExpandedIdentifier when a
+  // macro invocation is found.
+  void MacroExpands(const Token &MacroNameTok, const MacroDirective *MD,
+                    SourceRange Range, const MacroArgs *Args) {
+    BeginCallback("MacroExpands");
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+    AppendArgument("Range", Range);
+    AppendArgument("Args", Args);
+  }
+
+  // Hook called whenever a macro definition is seen.
+  void MacroDefined(const Token &MacroNameTok, const MacroDirective *MD) {
+    BeginCallback("MacroDefined");
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+  }
+
+  // Hook called whenever a macro #undef is seen.
+  // MD is released immediately following this callback.
+  void MacroUndefined(const Token &MacroNameTok, const MacroDirective *MD) {
+    BeginCallback("MacroUndefined");
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+  }
+
+  // Hook called whenever the 'defined' operator is seen.
+  // MD The MacroDirective if the name was a macro, null otherwise.
+  void Defined(const Token &MacroNameTok, const MacroDirective *MD,
+               SourceRange Range) {
+    BeginCallback("Defined");
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+    AppendArgument("Range", Range);
+  }
+
+  // Hook called when a source range is skipped.
+  // Range - The SourceRange that was skipped. The range begins at the
+  // #if/#else directive and ends after the #endif/#else directive.
+  void SourceRangeSkipped(SourceRange Range) {
+    BeginCallback("SourceRangeSkipped");
+    AppendArgument("Range", Range);
+  }
+
+  // Hook called whenever an #if is seen.
+  // Loc - the source location of the directive.
+  // ConditionRange - The SourceRange of the expression being tested.
+  // ConditionValue - The evaluated value of the condition.
+  void If(SourceLocation Loc, SourceRange ConditionRange, bool ConditionValue) {
+    BeginCallback("If");
+    AppendArgument("Loc", Loc);
+    AppendArgument("ConditionRange", ConditionRange);
+    AppendArgument("ConditionValue", ConditionValue);
+  }
+
+  // Hook called whenever an #elif is seen.
+  // Loc - the source location of the directive.
+  // ConditionRange - The SourceRange of the expression being tested.
+  // ConditionValue - The evaluated value of the condition.
+  // IfLoc - the source location of the #if/#ifdef/#ifndef directive.
+  void Elif(SourceLocation Loc, SourceRange ConditionRange, bool ConditionValue,
+            SourceLocation IfLoc) {
+    BeginCallback("Elif");
+    AppendArgument("Loc", Loc);
+    AppendArgument("ConditionRange", ConditionRange);
+    AppendArgument("ConditionValue", ConditionValue);
+    AppendArgument("IfLoc", IfLoc);
+  }
+
+  // Hook called whenever an #ifdef is seen.
+  // Loc - the source location of the directive.
+  // MacroNameTok - Information on the token being tested.
+  // MD - The MacroDirective if the name was a macro, null otherwise.
+  void Ifdef(SourceLocation Loc, const Token &MacroNameTok,
+             const MacroDirective *MD) {
+    BeginCallback("Ifdef");
+    AppendArgument("Loc", Loc);
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+  }
+
+  // Hook called whenever an #ifndef is seen.
+  // Loc - the source location of the directive.
+  // MacroNameTok - Information on the token being tested.
+  // MD - The MacroDirective if the name was a macro, null otherwise.
+  void Ifndef(SourceLocation Loc, const Token &MacroNameTok,
+              const MacroDirective *MD) {
+    BeginCallback("Ifndef");
+    AppendArgument("Loc", Loc);
+    AppendArgument("MacroNameTok", MacroNameTok);
+    AppendArgument("MD", MD);
+  }
+
+  // Hook called whenever an #else is seen.
+  // Loc - the source location of the directive.
+  // IfLoc - the source location of the #if/#ifdef/#ifndef directive.
+  void Else(SourceLocation Loc, SourceLocation IfLoc) {
+    BeginCallback("Else");
+    AppendArgument("Loc", Loc);
+    AppendArgument("IfLoc", IfLoc);
+  }
+
+  // Hook called whenever an #endif is seen.
+  // Loc - the source location of the directive.
+  // IfLoc - the source location of the #if/#ifdef/#ifndef directive.
+  void Endif(SourceLocation Loc, SourceLocation IfLoc) {
+    BeginCallback("Endif");
+    AppendArgument("Loc", Loc);
+    AppendArgument("IfLoc", IfLoc);
+  }
+
+  // Helper functions.
+
+  // Push a new string on the trace.
+  void Push(const char *Str) { Trace.push_back(std::string(Str)); }
+
+  // Format and push a new string on the trace.
+  void PushFormatted(const char *Format, ...) {
+    char Buffer[256];
+    va_list Args;
+    va_start(Args, Format);
+    vsnprintf(Buffer, sizeof(Buffer) - 1, Format, Args);
+    Push(Buffer);
+    va_end(Args);
+  }
+
+  // Start a new callback.
+  void BeginCallback(const char *Name) {
+    DisableTrace = Ignore.count(std::string(Name));
+    if (DisableTrace)
+      return;
+    PushFormatted("%s\n", Name);
+  }
+
+  // Append a string to the top trace item.
+  void Append(const char *Str) {
+    if (DisableTrace)
+      return;
+    Trace.back().append(Str);
+  }
+
+  // Format and append a string to the top trace item.
+  void AppendFormatted(const char *Format, ...) {
+    if (DisableTrace)
+      return;
+    char Buffer[256];
+    va_list Args;
+    va_start(Args, Format);
+    vsnprintf(Buffer, sizeof(Buffer) - 1, Format, Args);
+    Append(Buffer);
+    va_end(Args);
+  }
+
+  // Append a bool argument to the top trace item.
+  void AppendArgument(const char *Name, bool Value) {
+    AppendFormatted("  %s = %s\n", Name, (Value ? "true" : "false"));
+  }
+
+  // Append an int argument to the top trace item.
+  void AppendArgument(const char *Name, int Value) {
+    AppendFormatted("  %s = %d\n", Name, Value);
+  }
+
+  // Append a string argument to the top trace item.
+  void AppendArgument(const char *Name, const char *Value) {
+    AppendFormatted("  %s = %s\n", Name, Value);
+  }
+
+  // Append a string object argument to the top trace item.
+  void AppendArgument(const char *Name, StringRef Value) {
+    AppendArgument(Name, Value.str());
+  }
+
+  // Append a string object argument to the top trace item.
+  void AppendArgument(const char *Name, std::string Value) {
+    AppendArgument(Name, Value.c_str());
+  }
+
+  // Append a token argument to the top trace item.
+  void AppendArgument(const char *Name, const Token &Value) {
+    AppendArgument(Name, PP.getSpelling(Value));
+  }
+
+  // Append an enum argument to the top trace item.
+  void AppendArgument(const char *Name, int Value, const char *Strings[]) {
+    AppendArgument(Name, Strings[Value]);
+  }
+
+  // Append a FileID argument to the top trace item.
+  void AppendArgument(const char *Name, FileID Value) {
+    if (Value.isInvalid()) {
+      AppendArgument(Name, "(invalid)");
+      return;
+    }
+    const FileEntry *FileEntry = PP.getSourceManager().getFileEntryForID(Value);
+    if (FileEntry == 0) {
+      AppendArgument(Name, "(getFileEntryForID failed)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "FileID(%s)", FileEntry->getName());
+  }
+
+  // Append a FileEntry argument to the top trace item.
+  void AppendArgument(const char *Name, const FileEntry *Value) {
+    if (Value == 0) {
+      AppendArgument(Name, "(null)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "FileEntry(%s)", Value->getName());
+  }
+
+  // Append a SourceLocation argument to the top trace item.
+  void AppendArgument(const char *Name, SourceLocation Value) {
+    if (Value.isInvalid()) {
+      AppendArgument(Name, "(invalid)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "SourceLocation(%s)",
+                            getSourceLocationString(PP, Value).c_str());
+  }
+
+  // Append a SourceRange argument to the top trace item.
+  void AppendArgument(const char *Name, SourceRange Value) {
+    if (Value.isInvalid()) {
+      AppendArgument(Name, "(invalid)");
+      return;
+    }
+    AppendArgumentFormatted(
+        Name, "SourceRange(%s, %s)",
+        getSourceLocationString(PP, Value.getBegin()).c_str(),
+        getSourceLocationString(PP, Value.getEnd()).c_str());
+  }
+
+  // Append a CharSourceRange argument to the top trace item.
+  void AppendArgument(const char *Name, CharSourceRange Value) {
+    if (Value.isInvalid()) {
+      AppendArgument(Name, "(invalid)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "CharSourceRange(%s)",
+                            getSourceString(Value).str().c_str());
+  }
+
+  // Append a SourceLocation argument to the top trace item.
+  void AppendArgument(const char *Name, ModuleIdPath Value) {
+    if (DisableTrace)
+      return;
+    std::string Buffer;
+    for (ModuleIdPath::iterator I = Value.begin(), E = Value.end(); I != E;
+         ++I) {
+      IdentifierInfo *Info = I->first;
+      SourceLocation Loc = I->second;
+      Buffer.append("(");
+      Buffer.append(Info->getName());
+      Buffer.append(", ");
+      Buffer.append(getSourceLocationString(PP, Loc));
+      Buffer.append("), ");
+    }
+    AppendArgument(Name, Buffer.c_str());
+  }
+
+  // Append an IdentifierInfo argument to the top trace item.
+  void AppendArgument(const char *Name, const IdentifierInfo *Value) {
+    if (!Value) {
+      AppendArgument(Name, "(null)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "IdentifierInfo(%s)", Value->getName());
+  }
+
+  // Append a MacroDirective argument to the top trace item.
+  void AppendArgument(const char *Name, const MacroDirective *Value) {
+    if (!Value) {
+      AppendArgument(Name, "(null)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "MacroDirective(%s)",
+                            MacroDirectiveKindStrings[Value->getKind()]);
+  }
+
+  // Append a MacroArgs argument to the top trace item.
+  void AppendArgument(const char *Name, const MacroArgs *Value) {
+    if (!Value) {
+      AppendArgument(Name, "(null)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "(%d)", Value->getNumArguments());
+  }
+
+  // Append a Module argument to the top trace item.
+  void AppendArgument(const char *Name, const Module *Value) {
+    if (!Value) {
+      AppendArgument(Name, "(null)");
+      return;
+    }
+    AppendArgumentFormatted(Name, "Module(%s)", Value->Name.c_str());
+  }
+
+  // Format and append an argument to the top trace item.
+  void AppendArgumentFormatted(const char *Name, const char *Format, ...) {
+    if (DisableTrace)
+      return;
+    char Buffer[256];
+    va_list Args;
+    va_start(Args, Format);
+    vsnprintf(Buffer, sizeof(Buffer) - 1, Format, Args);
+    AppendArgument(Name, Buffer);
+    va_end(Args);
+  }
+
+  // Get the raw source string of the range.
+  StringRef getSourceString(CharSourceRange Range) {
+    const char *B = PP.getSourceManager().getCharacterData(Range.getBegin());
+    const char *E = PP.getSourceManager().getCharacterData(Range.getEnd());
+    return StringRef(B, E - B);
+  }
+
+  std::vector<std::string> &Trace; // Trace strings stored here.
+  Preprocessor &PP;
+  llvm::SmallSet<std::string, 4> Ignore; // Callbacks to ignore stored here.
+  bool DisableTrace;                     // Inhibit trace while this is set.
+};
+
 // Stub to collect data from PragmaOpenCLExtension callbacks.
 class PragmaOpenCLExtensionCallbacks : public PPCallbacks {
 public:
   typedef struct {
     SmallString<16> Name;
     unsigned State;
   } CallbackParameters;
 
PragmaOpenCLExtensionCall(const char* SourceText) {
     }
 
     PragmaOpenCLExtensionCallbacks::CallbackParameters RetVal = {
       Callbacks->Name,
       Callbacks->State
     };
     return RetVal;    
   }
+
+  // Run preprocessor and lexer over SourceText and collect trace strings,
+  // then compare them against the given patterns.
+  // This function uses the PPCallbacksTracker derivation of PPCallbacks
+  // to record trace strings from the callbacks, which this function
+  // then will compare against the given test patterns.
+  // If any callback traces don't match, output messages will show a
+  // comparison of expected and actual traces.
+  // The arguments:
+  // FileName - Name to give the preprocessed buffer.
+  // SourceText - The text to be preprocessed.
+  // HeaderPath - Optional fake header name or null.
+  // SystemHeader - True if optional fake header is system header.
+  // Patterns - Array of per-callback trace patterns to match.
+  //   At present there is no support for wildcards or regular expression
+  //   sequences, but this could be added in the future if necessary.
+  // IgnoreCallbacks - Array of callback names to not trace.
+  // Dump - Display callback traces (for use during development).
+  bool TrackCallbacks(const char *FileName, const char *SourceText,
+                      const char *HeaderPath, bool SystemHeader,
+                      const char *Patterns[], const char **IgnoreCallbacks,
+                      bool Dump) {
+    MemoryBuffer *Buf = MemoryBuffer::getMemBuffer(SourceText, FileName, true);
+    (void)SourceMgr.createMainFileIDForMemBuffer(Buf);
+
+    VoidModuleLoader ModLoader;
+
+    IntrusiveRefCntPtr<HeaderSearchOptions> HSOpts = new HeaderSearchOptions();
+    HeaderSearch HeaderInfo(HSOpts, FileMgr, Diags, LangOpts, Target.getPtr());
+
+    if (HeaderPath)
+      AddFakeHeader(HeaderInfo, HeaderPath, SystemHeader);
+
+    IntrusiveRefCntPtr<PreprocessorOptions> PPOpts = new PreprocessorOptions();
+    Preprocessor PP(PPOpts, Diags, LangOpts, Target.getPtr(), SourceMgr,
+                    HeaderInfo, ModLoader,
+                    /*IILookup =*/0,
+                    /*OwnsHeaderSearch =*/false,
+                    /*DelayInitialization =*/false);
+    std::vector<std::string> Trace;
+    PPCallbacksTracker *Callbacks =
+        new PPCallbacksTracker(Trace, IgnoreCallbacks, PP);
+    PP.addPPCallbacks(Callbacks); // Takes ownership.
+
+    // Lex source text.
+    PP.EnterMainSourceFile();
+
+    // Lex through source.
+    while (true) {
+      Token Tok;
+      PP.Lex(Tok);
+      if (Tok.is(tok::eof))
+        break;
+    }
+
+    if (Dump)
+      DumpTrace(Trace, true);
+
+    return ValidateTrace(Trace, Patterns);
+  }
+
+  // Dump trace.
+  // Trace - The callback trace string vector.
+  // Development - If true, output in string-literal form.
+  void DumpTrace(std::vector<std::string> Trace, bool Development) {
+    if (Development) {
+      for (std::vector<std::string>::iterator I = Trace.begin(),
+                                              E = Trace.end();
+           I != E; ++I) {
+        SmallVector<StringRef, 16> Strings;
+        StringRef(*I).split(Strings, "\n", -1, false);
+        int Index = 0;
+        int Last = (int)Strings.size() - 1;
+        for (SmallVector<StringRef, 16>::iterator II = Strings.begin(),
+                                                  EE = Strings.end();
+             II < EE; ++II, ++Index) {
+          if (Index == Last)
+            errs() << "    \"" << *II << "\\n\",\n";
+          else
+            errs() << "    \"" << *II << "\\n\"\n";
+        }
+      }
+      errs() << "    0\n";
+    } else {
+      for (std::vector<std::string>::iterator I = Trace.begin(),
+                                              E = Trace.end();
+           I != E; ++I) {
+        errs() << *I;
+      }
+    }
+  }
+
+  // Compare PPCallbacks trace against pattern.
+  // If any mismatch, display difference message.
+  // Returns true if match.
+  bool ValidateTrace(std::vector<std::string> &Trace, const char *Patterns[]) {
+    int I, E = Trace.size();
+    bool ReturnValue = true;
+    for (I = 0; Patterns[I] != 0; ++I) {
+      std::string Value;
+      if (I < E)
+        Value = Trace[I];
+      else
+        Value = "-";
+      if (strcmp(Trace[I].c_str(), Patterns[I]) != 0) {
+        ReturnValue = false;
+        errs() << "PPCallbacks trace mismatch:\n"
+               << "Expected:\n" << Patterns[I] << "Received:\n" << Trace[I];
+      }
+    }
+    // If more trace than patterns
+    if (E > I) {
+      ReturnValue = false;
+      errs() << "Too many traces:\n"
+             << "Expected:\n-\n"
+             << "Received:\n" << Trace[I];
+    }
+    return ReturnValue;
+  }
 };
 
 TEST_F(PPCallbacksTest, QuotedFilename) {
   const char* Source =
     "#include \"quoted.h\"\n";
 
   CharSourceRange Range =
     InclusionDirectiveFilenameRange(Source, "/quoted.h", false);
TEST_F(PPCallbacksTest, OpenCLExtensionPragmaDisabled) {
   PragmaOpenCLExtensionCallbacks::CallbackParameters Parameters =
     PragmaOpenCLExtensionCall(Source);
 
   ASSERT_EQ("cl_khr_fp16", Parameters.Name);
   unsigned ExpectedState = 0;
   ASSERT_EQ(ExpectedState, Parameters.State);
 }
 
+// Test macro-related callbacks.
+TEST_F(PPCallbacksTest, MacroCallbacks) {
+  const char *Source = "#define MACRO 1\n"
+                       "int i = MACRO;\n"
+                       "#if defined(MACRO)\n"
+                       "#endif\n"
+                       "#undef MACRO\n"
+                       "#if defined(MACRO)\n"
+                       "#endif\n";
+  const char *Ignore[] = {
+    "FileChanged", "If", "Endif", "SourceRangeSkipped", 0
+  };
+  const char *Patterns[] = {
+    "MacroDefined\n"
+    "  MacroNameTok = MACRO\n"
+    "  MD = MacroDirective(MD_Define)\n",
+    "MacroExpands\n"
+    "  MacroNameTok = MACRO\n"
+    "  MD = MacroDirective(MD_Define)\n"
+    "  Range = SourceRange(File.cpp:2:9, File.cpp:2:9)\n"
+    "  Args = (null)\n",
+    "Defined\n"
+    "  MacroNameTok = MACRO\n"
+    "  MD = MacroDirective(MD_Define)\n"
+    "  Range = SourceRange(File.cpp:3:5, File.cpp:3:19)\n",
+    "MacroUndefined\n"
+    "  MacroNameTok = MACRO\n"
+    "  MD = MacroDirective(MD_Define)\n",
+    "Defined\n"
+    "  MacroNameTok = MACRO\n"
+    "  MD = (null)\n"
+    "  Range = SourceRange(File.cpp:6:5, File.cpp:6:19)\n",
+    0
+  };
+  ASSERT_TRUE(
+      TrackCallbacks("File.cpp", Source, 0, false, Patterns, Ignore, false));
+}
+
+// Test conditional-related callbacks.
+TEST_F(PPCallbacksTest, ConditionalCallbacks) {
+  const char *Source = "// Simple if/endif with true condition.\n"
+                       "#if 1\n"
+                       "#endif\n"
+                       "// Simple if/endif with false condition.\n"
+                       "#if 0\n"
+                       "#endif\n"
+                       "// Simple if/else/endif with true condition.\n"
+                       "#if 1\n"
+                       "#else\n"
+                       "#endif\n"
+                       "// Simple if/else/endif with false condition.\n"
+                       "#if 0\n"
+                       "#else\n"
+                       "#endif\n"
+                       "// Simple if/elif/endif with true/true condition.\n"
+                       "#if 1\n"
+                       "#elif 1\n"
+                       "#endif\n"
+                       "// Simple if/elif/endif with false/true condition.\n"
+                       "#if 0\n"
+                       "#elif 1\n"
+                       "#endif\n"
+                       "// Simple if/elif/endif with true/false condition.\n"
+                       "#if 1\n"
+                       "#elif 0\n"
+                       "#endif\n"
+                       "// Simple if/elif/endif with false/false condition.\n"
+                       "#if 0\n"
+                       "#elif 0\n"
+                       "#endif\n"
+                       "// Some macros to use.\n"
+                       "#define MACRO0 0\n"
+                       "#define MACRO1 1\n"
+                       "// Simple if/endif with macro false condition.\n"
+                       "#if MACRO0\n"
+                       "#endif\n"
+                       "// Simple if/endif with macro true condition.\n"
+                       "#if MACRO1\n"
+                       "#endif\n"
+                       "// Simple if/endif with defined() true condition.\n"
+                       "#if defined(MACRO0)\n"
+                       "#endif\n"
+                       "// Simple if/endif with defined() false condition.\n"
+                       "#if defined(MACROX)\n"
+                       "#endif\n"
+                       "// Simple nested if/endif with true condition.\n"
+                       "#if 1\n"
+                       "#if 1\n"
+                       "#endif\n"
+                       "#endif\n"
+                       "// Simple nested if/endif with false condition.\n"
+                       "#if 0\n"
+                       "#if 1\n"
+                       "#endif\n"
+                       "#endif\n";
+  const char *Ignore[] = { "FileChanged", 0 };
+  const char *Patterns[] = {
+    // Simple if/endif with true condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:2:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:2:4, File.cpp:3:1)\n"
+    "  ConditionValue = true\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:3:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:2:2)\n",
+    // Simple if/endif with false condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:5:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:5:4, File.cpp:6:1)\n"
+    "  ConditionValue = false\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:6:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:5:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:5:2, File.cpp:6:2)\n",
+    // Simple if/else/endif with true condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:8:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:8:4, File.cpp:9:1)\n"
+    "  ConditionValue = true\n",
+    "Else\n"
+    "  Loc = SourceLocation(File.cpp:9:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:8:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:10:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:8:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:9:2, File.cpp:10:2)\n",
+    // Simple if/else/endif with false condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:12:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:12:4, File.cpp:13:1)\n"
+    "  ConditionValue = false\n",
+    "Else\n"
+    "  Loc = SourceLocation(File.cpp:13:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:12:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:12:2, File.cpp:13:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:14:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:12:2)\n",
+    // Simple if/elif/endif with true/true condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:16:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:16:4, File.cpp:17:1)\n"
+    "  ConditionValue = true\n",
+    "Elif\n"
+    "  Loc = SourceLocation(File.cpp:17:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:17:6, File.cpp:18:1)\n"
+    "  ConditionValue = true\n"
+    "  IfLoc = SourceLocation(File.cpp:16:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:18:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:16:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:17:2, File.cpp:18:2)\n",
+    // Simple if/elif/endif with false/true condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:20:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:20:4, File.cpp:21:1)\n"
+    "  ConditionValue = false\n",
+    "Elif\n"
+    "  Loc = SourceLocation(File.cpp:21:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:21:6, File.cpp:22:1)\n"
+    "  ConditionValue = true\n"
+    "  IfLoc = SourceLocation(File.cpp:20:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:20:2, File.cpp:21:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:22:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:20:2)\n",
+    // Simple if/elif/endif with true/false condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:24:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:24:4, File.cpp:25:1)\n"
+    "  ConditionValue = true\n",
+    "Elif\n"
+    "  Loc = SourceLocation(File.cpp:25:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:25:6, File.cpp:26:1)\n"
+    "  ConditionValue = true\n"
+    "  IfLoc = SourceLocation(File.cpp:24:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:26:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:24:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:25:2, File.cpp:26:2)\n",
+    // Simple if/elif/endif with false/false condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:28:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:28:4, File.cpp:29:1)\n"
+    "  ConditionValue = false\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:30:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:28:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:28:2, File.cpp:30:2)\n",
+    // Some macros to use.
+    "MacroDefined\n"
+    "  MacroNameTok = MACRO0\n"
+    "  MD = MacroDirective(MD_Define)\n",
+    "MacroDefined\n"
+    "  MacroNameTok = MACRO1\n"
+    "  MD = MacroDirective(MD_Define)\n",
+    // Simple if/endif with macro false condition.
+    "MacroExpands\n"
+    "  MacroNameTok = MACRO0\n"
+    "  MD = MacroDirective(MD_Define)\n"
+    "  Range = SourceRange(File.cpp:35:5, File.cpp:35:5)\n"
+    "  Args = (null)\n",
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:35:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:35:4, File.cpp:36:1)\n"
+    "  ConditionValue = false\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:36:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:35:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:35:2, File.cpp:36:2)\n",
+    // Simple if/endif with macro true condition.
+    "MacroExpands\n"
+    "  MacroNameTok = MACRO1\n"
+    "  MD = MacroDirective(MD_Define)\n"
+    "  Range = SourceRange(File.cpp:38:5, File.cpp:38:5)\n"
+    "  Args = (null)\n",
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:38:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:38:4, File.cpp:39:1)\n"
+    "  ConditionValue = true\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:39:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:38:2)\n",
+    // Simple if/endif with defined() true condition.
+    "Defined\n"
+    "  MacroNameTok = MACRO0\n"
+    "  MD = MacroDirective(MD_Define)\n"
+    "  Range = SourceRange(File.cpp:41:5, File.cpp:41:20)\n",
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:41:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:41:4, File.cpp:42:1)\n"
+    "  ConditionValue = true\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:42:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:41:2)\n",
+    // Simple if/endif with defined() false condition.
+    "Defined\n"
+    "  MacroNameTok = MACROX\n"
+    "  MD = (null)\n"
+    "  Range = SourceRange(File.cpp:44:5, File.cpp:44:20)\n",
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:44:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:44:4, File.cpp:45:1)\n"
+    "  ConditionValue = false\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:45:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:44:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:44:2, File.cpp:45:2)\n",
+    // Simple nested if/endif with true condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:47:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:47:4, File.cpp:48:1)\n"
+    "  ConditionValue = true\n",
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:48:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:48:4, File.cpp:49:1)\n"
+    "  ConditionValue = true\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:49:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:48:2)\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:50:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:47:2)\n",
+    // Simple nested if/endif with false condition.
+    "If\n"
+    "  Loc = SourceLocation(File.cpp:52:2)\n"
+    "  ConditionRange = SourceRange(File.cpp:52:4, File.cpp:53:1)\n"
+    "  ConditionValue = false\n",
+    "Endif\n"
+    "  Loc = SourceLocation(File.cpp:55:2)\n"
+    "  IfLoc = SourceLocation(File.cpp:52:2)\n",
+    "SourceRangeSkipped\n"
+    "  Range = SourceRange(File.cpp:52:2, File.cpp:55:2)\n",
+    // Mark end of patterns.
+    0
+  };
+  ASSERT_TRUE(
+      TrackCallbacks("File.cpp", Source, 0, false, Patterns, Ignore, false));
+}
+
 } // anonoymous namespace