[analyzer] Introduce MacroExpansionContext to libAnalysis

Authored by steakhal on Feb 22 2021, 2:11 AM.


[analyzer] Introduce MacroExpansionContext to libAnalysis

Introduce MacroExpansionContext to track what and how macros in a translation
unit expand. This is the first element of the patch-stack in this direction.

The main goal is to substitute the current macro expansion generator in the
PlistsDiagnostics, but all the other DiagnosticsConsumer could benefit from

getExpandedText and getOriginalText are the primary functions of this class.
The former can provide you the text that was the result of the macro expansion
chain starting from a SourceLocation.
While the latter will tell you what text was in the original source code
replaced by the macro expansion chain from that location.

Here is an example:

void bar();
#define retArg(x) x
#define retArgUnclosed retArg(bar()
#define BB CC
#define applyInt BB(int)
#define CC(x) retArgUnclosed

void unbalancedMacros() {
  applyInt  );
//^~~~~~~~~~^ is the substituted range
// Original text is "applyInt  )"
// Expanded text is "bar()"

#define expandArgUnclosedCommaExpr(x) (x, bar(), 1
#define f expandArgUnclosedCommaExpr

void unbalancedMacros2() {
  int x =  f(f(1))  ));  // Look at the parenthesis!
//         ^~~~~~^ is the substituted range
// Original text is "f(f(1))"
// Expanded text is "((1,bar(),1,bar(),1"

Might worth investigating how to provide a reusable component, which could be
used for example by a standalone tool eg. expanding all macros to their

I borrowed the main idea from the PrintPreprocessedOutput.cpp Frontend
component, providing a PPCallbacks instance hooking the preprocessor events.
I'm using that for calculating the source range where tokens will be expanded
to. I'm also using the Preprocessor's OnToken callback, via the
Preprocessor::setTokenWatcher to reconstruct the expanded text.

Unfortunately, I concatenate the token's string representation without any
whitespaces except if the token is an identifier when I emit an extra space
to produce valid code for int var token sequences.
This could be improved later if needed.


  1. D93222 (this one) Introduces the MacroExpansionContext class and unittests
  2. D93223 Create MacroExpansionContext member in AnalysisConsumer and pass down to the diagnostics consumers
  3. D93224 Use the MacroExpansionContext for macro expansions in plists It replaces the 'old' macro expansion mechanism.
  4. D94673 API for CTU macro expansions You should be able to get a MacroExpansionContext for each imported TU. Right now it will just return llvm::None as this is not implemented yet.
  5. FIXME: Implement macro expansion tracking for imported TUs as well.

It would also relieve us from bugs like:

  • [fixed] D86135
  • [confirmed] The __VA_ARGS__ and other macro nitty-gritty, such as how to stringify macro parameters, where to put or swallow commas, etc. are not handled correctly.
  • [confirmed] Unbalanced parenthesis are not well handled - resulting in incorrect expansions or even crashes.
  • [confirmed][crashing] https://bugs.llvm.org/show_bug.cgi?id=48358

Reviewed By: martong, Szelethus

Differential Revision: https://reviews.llvm.org/D93222