[clang-format] Add simple macro replacements in formatting.

[klimek]

Summary

Add configuration to specify macros.
Macros will be expanded, and the code will be parsed and annotated
in the expanded state. In a second step, the formatting decisions
in the annotated expanded code will be reconstructed onto the
original unexpanded macro call.

Eventually, this will allow to remove special-case code for
various macro options we accumulated over the years in favor of
one principled mechanism.

[sammccall]

It's happening!

clang/lib/Format/ContinuationIndenter.cpp

743

nit: parentheses

744

This is a bit hard to follow without context, I had to look up MacroCallReconstructor::takeResult() to recall the structure it's referring to, and there's no obvious breadcrumb to point you there. And if you don't follow the details, it's hard to understand what the relevance is.

Maybe something like: this doesn't apply to macro expansion lines, which are MACRO( , , ) with args as children of ( and ,. It does not make sense to align the commas with the opening paren.

(I think describing here how the alignment ends up working in that case might just add confusion: it's not handled by this code, right?)

clang/lib/Format/TokenAnnotator.h

68

this seems like a separate bugfix: previously if the first node in Line.Tokens had children, we wouldn't be adding them.

Is this true? Was there some reason the first node could never have children before?

76

does "add children" here refer to the procedure in addChildren(), or to the token insertion idea introduced in the previous patch?

(i.e. is this just describing that previous/next skip over trees of children rather than including them in the chain, or is there something new going on?)

clang/lib/Format/UnwrappedLineFormatter.cpp

924

comment as to why?

926

why do we no longer finalize the child tree?

clang/lib/Format/UnwrappedLineParser.cpp

224

I'm sure you got this right, but it's hard for me to evaluate this code because I don't know what UnwrappedLineConsumer::finishRun() is for - it's not documented and the implementation is mysterious.

You might consider adding a comment to that interface/method, it's not really related to this change though.

4209–4210

nit: you check this condition a bunch of times, and this patch adds more.

giving it a name like !parsingPPDirective() would be less cryptic

4246

This is a bit vague as to what the "indexes" refer to and what the implications are (between comment & code "indexes" is mentioned 3 times but it's not clear without digging what they are).

AFAICS in practice this only affects indentation of braces in whitesmiths style, which may be worth mentioning. (It'll get stale, but at the moment the comment sounds like arbitrary things are likely to be broken)

4570

it looks like you go into parsing the macro call including parens and arguments regardless of whether the macro is object-like or function-like.

So given

#define DEBUG puts
DEBUG("foo");

you're going to expand to puts; instead of puts("foo");

4579

Calling a macro with the wrong arity might be worth flagging in debug output?

4668

this mishandles:

#define BAR() foo
int y = BAR; // no macro used here, just a variable named BAR

4706

hitting eof is an error condition here, so it seems more natural to handle it as a case in the switch as a kind of early-return, and maybe it's worth adding an LLVM_DEBUG() for that case. If anything it'd be nice to have r_paren break the loop (but too annoying with the switch inside)

clang/lib/Format/UnwrappedLineParser.h

78

maybe slightly more descriptive like resetBlockLineIndexes?

in principle you want to clear any indices you have here, so it's nice to be vague, but the caller also needs to reason about the consequences of destroying this data.

104

you forward declare this here, move the destructor below out-of-line etc
but also include Macros.h and use MacroExpander by value below
Do we want to avoid the dependency or not?

294

if this doesn't need to be incomplete, optional would be clearer

clang/unittests/Format/FormatTest.cpp

70

hmm, testing formatting in this way without messing up the formatting first isn't very convincing from a black-box perspective!

if messUp isn't feasible here (why precisely?) then maybe these tests should specify both input + expected output?

22587

If this is assign_or_return, the treatment of "c" is fairly strange. Also you are mostly calling this with only two args. Maybe drop C and use a different macro for the 3-arg case?

22615

MOCK_METHOD is another motivating case that might be worth testing (even if it's just a more complicated version of the same thing)

[klimek]

Address reviewer comments.

clang/lib/Format/ContinuationIndenter.cpp

744

Excellent wording, applied. And yes, alignment in the macro case is not handled here.

clang/lib/Format/TokenAnnotator.h

68

I think the main problem is that it's impossible to write C++ code that has a child in the first node based on how this worked. That is, previously we only added children for blocks within one line, that is, within a function or lambda. Those need intro tokens. So I really couldn't make this trigger, or I would have fixed it in a separate patch.

76

It's about adding children in general.
When fixing this I realized this is in principle wrong, but then noticed that we have code that depends on this behavior.

clang/lib/Format/UnwrappedLineFormatter.cpp

926

We do - markFinalized() is called for the child tree again. Previously we marked multiple times, which doesn't work with the new algorithm, as it would move ExpandedArg -> UnexpandedArg -> Finalized in one go.

clang/lib/Format/UnwrappedLineParser.cpp

224

Done. Not sure I've put too much info about how it's going to be used into the interface, where it doesn't really belong.

4570

Excellent find. Completely invalidated the way I was doing formatting of expanded lines, but fixed the problem with the line indices needing resetting \o/

Now we're formatting all lines in the first round with the macro calls replaced with expanded lines, and then again format everything including the macro calls. That is more in line with clang-format's assumptions anyway.

4706

Usually we don't see hitting eof as an error, as that can always happen while we're in the process of writing a new file, or formatting "up to cursor". Thus, I think this fits the pattern of "stop where we are when we hit eof and just go with what we have" that's everywhere in the UnwrappedLineParser.

clang/lib/Format/UnwrappedLineParser.h

104

That was a leftover from a previous phase, thanks for catching. Removed forward decl and removed out of line destructor.

294

Excellent idea, done.

clang/unittests/Format/FormatTest.cpp

22587

ASSIGN_OR_RETURN allows 3 args, though; this is basically the thing I'd propose to configure for ASSIGN_OR_RETURN in general; is there anything specific you don't like about this?

[klimek]

Add comment.

[sammccall]

Thanks, this all looks good, at least as far as I understand it! The two-passes-over-the-whole-file approach is easier for me to follow, too.

Mostly just comment nits, but I have one annoying question left about using macros with the wrong arity - I thought this was just an implementation limitation, but it sounds like you want it to be a feature, so I'd like to nail it down a little...

clang/include/clang/Format/Format.h

2750

I find "try to match the structure of the expanded call" a bit hard to follow from a user perspective.

Maybe "Code will be parsed with macros expanded, in order to determine how to interpret and format the macro arguments"?

2753

This is a good example. I think you should spell out both sides, and probably start with the simpler case (no macro definition) to motivate the problem.

For example, the code:
  A(a*b);
will usually be interpreted as a call to a function A, and the multiplication expression will be formatted as `a * b`.

If we specify the macro definition:
  Macros:
    - A(x)=x

the code will now be parsed as a declaration of the variable b of type a*,
and formatted as `a* b` (depending on pointer-binding rules).

2757

type a*, not A*

2762

I think the restrictions should be spelled out here: object-like and function-like macros are both supported, macro args should be used exactly once, macros should not reference other macros.

clang/lib/Format/UnwrappedLineParser.cpp

224

Thanks, comment seems just right to me.

clang/lib/Format/UnwrappedLineParser.h

88

this is a bit of a garden path sentence: for code where ((macros are expanded) and (the code with (the original macro calls) ... ERR_MISSING_PREDICATE))

maybe when macros are involved, for the expanded code and the as-written code?

262

nit: should these added functions be const?

(maybe this class doesn't bother with const - computePPHash is const though)

clang/unittests/Format/FormatTest.cpp

22587

OK, this is a bit sticky.

• ASSIGN_OR_RETURN is almost always called with two args
• some versions of ASSIGN_OR_RETURN support an optional third arg (there's no canonical public version)
• these emulate overloading using variadic macro tricks
• this patch doesn't claim to support either variadic macros or overloading

So the principled options seem to be:

• macros have fixed arity: clang-format can support the two-arg version of ASSIGN_OR_RETURN but not the three-arg version. (This is what I was assuming)
• macros have variable arity: the public docs need to describe how passing too many/too few args is treated, because this isn't obvious and it sounds like we want people to rely on it. This feels like stretching "principled" to me - we're relying on the expansion still parsing correctly when args are missing, which basically seems like coincidence.
• macros support overloading: instead of looking up macros by name, the key is (string Name, optional<int> Arity)

[klimek]

Address review comments.

clang/unittests/Format/FormatTest.cpp

22587

Chose option 3, as this is pretty straight-forward and I believe the best user experience.

[sammccall]

Great!

The overloading impl is a little surprising to me.
I was assuming that object would always win over function (or that it would be disallowed to combine the two).
I think that is simpler to implement, as you know in advance what type of expansion you're attempting.
But what you have isn't that much more complicated, and the extra flexibility may prove useful.

One weird side-effect: if you define an object-like macro FOO and then write FOO( and never close the paren, then we die inside macro-arg-parsing rather than hitting the normal EOF-while-formatting behavior. I left a comment on parseMacroCall() about how to fix this but I'm not sure whether it matters at a high level.

clang/lib/Format/MacroExpander.cpp

162

this assertion failure isn't going to be as useful as it could be!

maybe rather

if (OptionalArgs)
  assert(...);
else
  assert(...);

Also I think we're now ensuring to only call this if arity matches, so I'd make this a precondition and replace the first assert with hasArity to make it easier to reason about

clang/lib/Format/UnwrappedLineParser.cpp

4580

nit: call => Macro call in dbgs()?

4592

Strictly I don't think this comment is true, we hit this path when it's only an object-like macro, used before parens.

For this reason you might *not* want the dbgs() here but rather in the bottom else

4610

(nit: std::move(args)?)

4610

nit: maybe "expansion" is more descriptive than "new"?

4620

I think you may want a comment and/or dbgs() here explaining this case: Didn't expand macro FOO because it was used {with 3|without} args, which doesn't match any definition

4708

I believe this has changed meaning along with the return type: previously it returned an empty vector, now it returns nullopt. Not sure which you intend here.

I think the ideal thing here would be to rewind the token stream back to before the (, and return nullopt. We'll end up performing an object-like substitution if possible, and non-macro parsing will continue.

[klimek]

Address review comments.

clang/lib/Format/MacroExpander.cpp

162

Hopefully done, the comment took me a while to parse :)

clang/lib/Format/UnwrappedLineParser.cpp

4592

I'm pretty sure we hit this when it's overloaded for both, but we call it with the wrong arity.
E.g. A=x, A(x, y, z)=x y z

A(1, 2)

• Macros.objectLike(A) -> true
• Args -> true
• !Macros.hasArity(A, 2) -> true

[sammccall]

Still LG

clang/lib/Format/MacroExpander.cpp

162

Sorry about that, LG though

clang/lib/Format/UnwrappedLineParser.cpp

4592

I agree that (the case described in the comment) => (we get to this point in the code).

I'm disputing that (we get to this point in the code) => (the case described in the comment).

i.e. given A=x, code=A(1, 2), we also get: objectLike(A) -> true, Args -> true, !Macros.hasArity(A, 2) -> true, but this time the comment is lying about the state.

[klimek]

Address review comments.

clang/lib/Format/UnwrappedLineParser.cpp

4592

Ah, your comment made me think it was the former; I think the dbgs() is still what I want, but with an "either or" dbgs().

Adapted the comment in the code and the dbgs() to match.

[MyDeveloperDay]

I know this is like "telling my grandmother to suck eggs" but @klimek the change here to Format.h means you need to regenerate ClangFormatStyleOptions.rst via docs/tools/dump_format_style.py

Impacting the next change to generate - D125171: [clang-format] Add a new clang-format option AlwaysBreakBeforeFunctionParameters

ok fixed your your behalf via rG7a5b95732ade: [clang-format] NFC Format.h and ClangFormatStyleOptions.rst are out of date

[klimek]

I am so sorry, thanks for sending out the patch already and fixing the
layout!

[HazardyKnusperkeks]

clang/include/clang/Format/Format.h

4343

I put a lot of effort into bringing the stuff sorted. And now a change which was completely transparent to me, because of the missing clang-format tag.

[owenpan]

It seems that this patch caused a regression. See https://github.com/llvm/llvm-project/issues/62768.

[rymiel]

NOTE: Clang-Format Team Automated Review Comment

Your review contains a change to clang/include/clang/Format/Format.h but does not contain an update to ClangFormatStyleOptions.rst

ClangFormatStyleOptions.rst is generated via clang/docs/tools/dump_format_style.py, please run this to regenerate the .rst

You can validate that the rst is valid by running.

./docs/tools/dump_format_style.py
mkdir -p html
/usr/bin/sphinx-build -n ./docs ./html

clang/include/clang/Format/Format.h

   /// \endcode
   /// \version 3.7
   std::string MacroBlockBegin;
 
   /// A regular expression matching macros that end a block.
   /// \version 3.7
   std::string MacroBlockEnd;
 
+  /// A list of macros of the form \c <definition>=<expansion> .
+  ///
+  /// Code will be parsed with macros expanded, in order to determine how to
+  /// interpret and format the macro arguments.
+  ///
+  /// For example, the code:
+  /// \code
+  ///   A(a*b);
+  /// \endcode
+  /// will usually be interpreted as a call to a function A, and the
+  /// multiplication expression will be formatted as `a * b`.
+  ///
+  /// If we specify the macro definition:
+  /// \code
+  ///   Macros:
+  ///   - A(x)=x
+  /// \endcode
+  /// the code will now be parsed as a declaration of the variable b of type a*,
+  /// and formatted as `a* b` (depending on pointer-binding rules).
+  ///
+  /// Features and restrictions:
+  /// *  Both function-like macros and object-like macros are supported.
+  /// *  Macro arguments must be used exactly once in the expansion.
+  /// *  No recursive expansion; macros referencing other macros will be
+  ///    ignored.
+  /// *  Overloading by arity is supported: for example, given the macro
+  ///    definitions A=x, A()=y, A(a)=a,
+  ///    'A;' -> 'x;'
+  ///    'A();' -> 'y;'
+  ///    'A(z);' -> 'z;'
+  ///    'A(a, b) will not be expanded.
+  std::vector<std::string> Macros;
+
   /// The maximum number of consecutive empty lines to keep.
   /// \code
   ///    MaxEmptyLinesToKeep: 1         vs.     MaxEmptyLinesToKeep: 0
   ///    int f() {                              int f() {
   ///      int = 1;                                 int i = 1;
   ///                                               i = foo();
   ///      i = foo();                               return i;
   ///                                           }
return AccessModifierOffset == R.AccessModifierOffset &&
            SpacesInLineCommentPrefix.Maximum ==
                R.SpacesInLineCommentPrefix.Maximum &&
            SpacesInParentheses == R.SpacesInParentheses &&
            SpacesInSquareBrackets == R.SpacesInSquareBrackets &&
            Standard == R.Standard &&
            StatementAttributeLikeMacros == R.StatementAttributeLikeMacros &&
            StatementMacros == R.StatementMacros && TabWidth == R.TabWidth &&
            TypenameMacros == R.TypenameMacros && UseTab == R.UseTab &&
-           WhitespaceSensitiveMacros == R.WhitespaceSensitiveMacros;
+           WhitespaceSensitiveMacros == R.WhitespaceSensitiveMacros &&
+           Macros == R.Macros;
   }
 
   std::optional<FormatStyle> GetLanguageStyle(LanguageKind Language) const;
 
   // Stores per-language styles. A FormatStyle instance inside has an empty
   // StyleSet. A FormatStyle instance returned by the Get method has its
   // StyleSet set to a copy of the originating StyleSet, effectively keeping the
   // internal representation of that StyleSet alive.

clang/lib/Format/ContinuationIndenter.cpp

 
 #include "ContinuationIndenter.h"
 #include "BreakableToken.h"
 #include "FormatInternal.h"
 #include "FormatToken.h"
 #include "WhitespaceManager.h"
 #include "clang/Basic/OperatorPrecedence.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TokenKinds.h"
 #include "clang/Format/Format.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
 
 #define DEBUG_TYPE "format-indenter"
 
 namespace clang {
if ((Style.AlignAfterOpenBracket == FormatStyle::BAS_AlwaysBreak ||
       //               caaaaaaaaaaaaaaaaaaaaaaall(aaaaaaaaaaaaaa, aaaaaaaaa))));
       Current.FakeLParens.size() > 0 &&
       Current.FakeLParens.back() > prec::Unknown) {
     CurrentState.NoLineBreak = true;
   }
   if (Previous.is(TT_TemplateString) && Previous.opensScope())
     CurrentState.NoLineBreak = true;
 
+  // Align following lines within parentheses / brackets if configured.
+  // Note: This doesn't apply to macro expansion lines, which are MACRO( , , )
+  // with args as children of the '(' and ',' tokens. It does not make sense to
+  // align the commas with the opening paren.
   if (Style.AlignAfterOpenBracket != FormatStyle::BAS_DontAlign &&
       !CurrentState.IsCSharpGenericTypeConstraint && Previous.opensScope() &&
       Previous.isNot(TT_ObjCMethodExpr) && Previous.isNot(TT_RequiresClause) &&
+      !(Current.MacroParent && Previous.MacroParent) &&
       (Current.isNot(TT_LineComment) || Previous.is(BK_BracedInit))) {
     CurrentState.Indent = State.Column + Spaces;
     CurrentState.IsAligned = true;
   }
   if (CurrentState.AvoidBinPacking && startsNextParameter(Current, Style))
     CurrentState.NoLineBreak = true;
   if (startsSegmentOfBuilderTypeCall(Current) &&
       State.Column > getNewLineColumn(State)) {

clang/lib/Format/Format.cpp

static void mapping(IO &IO, FormatStyle &Style) {
     IO.mapOptional("StatementAttributeLikeMacros",
                    Style.StatementAttributeLikeMacros);
     IO.mapOptional("StatementMacros", Style.StatementMacros);
     IO.mapOptional("TabWidth", Style.TabWidth);
     IO.mapOptional("TypenameMacros", Style.TypenameMacros);
     IO.mapOptional("UseTab", Style.UseTab);
     IO.mapOptional("WhitespaceSensitiveMacros",
                    Style.WhitespaceSensitiveMacros);
+    IO.mapOptional("Macros", Style.Macros);
 
     // If AlwaysBreakAfterDefinitionReturnType was specified but
     // AlwaysBreakAfterReturnType was not, initialize the latter from the
     // former for backwards compatibility.
     if (Style.AlwaysBreakAfterDefinitionReturnType != FormatStyle::DRTBS_None &&
         Style.AlwaysBreakAfterReturnType == FormatStyle::RTBS_None) {
       if (Style.AlwaysBreakAfterDefinitionReturnType ==
           FormatStyle::DRTBS_All) {

clang/lib/Format/FormatToken.h

private:
   unsigned TypeIsFinalized : 1;
   TokenType Type;
 
 public:
   /// Returns the token's type, e.g. whether "<" is a template opener or
   /// binary operator.
   TokenType getType() const { return Type; }
   void setType(TokenType T) {
+    // If this token is a macro argument while formatting an unexpanded macro
+    // call, we do not change its type any more - the type was deduced from
+    // formatting the expanded macro stream already.
+    if (MacroCtx && MacroCtx->Role == MR_UnexpandedArg)
+      return;
     assert((!TypeIsFinalized || T == Type) &&
            "Please use overwriteFixedType to change a fixed type.");
     Type = T;
   }
   /// Sets the type and also the finalized flag. This prevents the type to be
   /// reset in TokenAnnotator::resetTokenMetadata(). If the type needs to be set
   /// to another one please use overwriteFixedType, or even better remove the
   /// need to reassign the type.

clang/lib/Format/MacroExpander.cpp

Buffers.push_back(
       llvm::MemoryBuffer::getMemBufferCopy(Macro, "<scratch space>"));
   clang::FileID FID = SourceMgr.createFileID(Buffers.back()->getMemBufferRef());
   FormatTokenLexer Lex(SourceMgr, FID, 0, Style, encoding::Encoding_UTF8,
                        Allocator, IdentTable);
   const auto Tokens = Lex.lex();
   if (!Tokens.empty()) {
     DefinitionParser Parser(Tokens);
     auto Definition = Parser.parse();
-    Definitions[Definition.Name] = std::move(Definition);
+    if (Definition.ObjectLike) {
+      ObjectLike[Definition.Name] = std::move(Definition);
+    } else {
+      FunctionLike[Definition.Name][Definition.Params.size()] =
+          std::move(Definition);
+    }
   }
 }
 
 bool MacroExpander::defined(llvm::StringRef Name) const {
-  return Definitions.find(Name) != Definitions.end();
+  return FunctionLike.find(Name) != FunctionLike.end() ||
+         ObjectLike.find(Name) != ObjectLike.end();
 }
 
 bool MacroExpander::objectLike(llvm::StringRef Name) const {
-  return Definitions.find(Name)->second.ObjectLike;
+  return ObjectLike.find(Name) != ObjectLike.end();
 }
 
-llvm::SmallVector<FormatToken *, 8> MacroExpander::expand(FormatToken *ID,
-                                                          ArgsList Args) const {
-  assert(defined(ID->TokenText));
+bool MacroExpander::hasArity(llvm::StringRef Name, unsigned Arity) const {
+  auto it = FunctionLike.find(Name);
+  return it != FunctionLike.end() &&
+         (it->second.find(Arity) != it->second.end());
+}
+
+llvm::SmallVector<FormatToken *, 8>
+MacroExpander::expand(FormatToken *ID,
+                      std::optional<ArgsList> OptionalArgs) const {
+  if (OptionalArgs)
+    assert(hasArity(ID->TokenText, OptionalArgs->size()));
+  else
+    assert(objectLike(ID->TokenText));
+  const Definition &Def = OptionalArgs
+                              ? FunctionLike.find(ID->TokenText)
+                                    ->second.find(OptionalArgs.value().size())
+                                    ->second
+                              : ObjectLike.find(ID->TokenText)->second;
+  ArgsList Args = OptionalArgs ? OptionalArgs.value() : ArgsList();
   SmallVector<FormatToken *, 8> Result;
-  const Definition &Def = Definitions.find(ID->TokenText)->second;
-
   // Expand each argument at most once.
   llvm::StringSet<> ExpandedArgs;
 
   // Adds the given token to Result.
   auto pushToken = [&](FormatToken *Tok) {
     Tok->MacroCtx->ExpandedFrom.push_back(ID);
     Result.push_back(Tok);
   };

clang/lib/Format/Macros.h

public:
   /// Macros that cannot be parsed will be silently discarded.
   ///
   MacroExpander(const std::vector<std::string> &Macros,
                 clang::SourceManager &SourceMgr, const FormatStyle &Style,
                 llvm::SpecificBumpPtrAllocator<FormatToken> &Allocator,
                 IdentifierTable &IdentTable);
   ~MacroExpander();
 
-  /// Returns whether a macro \p Name is defined.
+  /// Returns whether any macro \p Name is defined, regardless of overloads.
   bool defined(llvm::StringRef Name) const;
 
-  /// Returns whether the macro has no arguments and should not consume
-  /// subsequent parentheses.
+  /// Returns whetherh there is an object-like overload, i.e. where the macro
+  /// has no arguments and should not consume subsequent parentheses.
   bool objectLike(llvm::StringRef Name) const;
 
+  /// Returns whether macro \p Name provides an overload with the given arity.
+  bool hasArity(llvm::StringRef Name, unsigned Arity) const;
+
   /// Returns the expanded stream of format tokens for \p ID, where
   /// each element in \p Args is a positional argument to the macro call.
-  llvm::SmallVector<FormatToken *, 8> expand(FormatToken *ID,
-                                             ArgsList Args) const;
+  /// If \p Args is not set, the object-like overload is used.
+  /// If \p Args is set, the overload with the arity equal to \c Args.size() is
+  /// used.
+  llvm::SmallVector<FormatToken *, 8>
+  expand(FormatToken *ID, std::optional<ArgsList> OptionalArgs) const;
 
 private:
   struct Definition;
   class DefinitionParser;
 
   void parseDefinition(const std::string &Macro);
 
   clang::SourceManager &SourceMgr;
   const FormatStyle &Style;
   llvm::SpecificBumpPtrAllocator<FormatToken> &Allocator;
   IdentifierTable &IdentTable;
   SmallVector<std::unique_ptr<llvm::MemoryBuffer>> Buffers;
-  llvm::StringMap<Definition> Definitions;
+  llvm::StringMap<llvm::DenseMap<int, Definition>> FunctionLike;
+  llvm::StringMap<Definition> ObjectLike;
 };
 
 /// Converts a sequence of UnwrappedLines containing expanded macros into a
 /// single UnwrappedLine containing the macro calls.  This UnwrappedLine may be
 /// broken into child lines, in a way that best conveys the structure of the
 /// expanded code.
 ///
 /// In the simplest case, a spelled UnwrappedLine contains one macro, and after
 /// expanding it we have one expanded UnwrappedLine.  In general, macro
 /// expansions can span UnwrappedLines, and multiple macros can contribute
 /// tokens to the same line.  We keep consuming expanded lines until:
 /// *   all expansions that started have finished (we're not chopping any macros
 ///     in half)
 /// *   *and* we've reached the end of a *spelled* unwrapped line.
 ///
 /// A single UnwrappedLine represents this chunk of code.
 ///
 /// After this point, the state of the spelled/expanded stream is "in sync"
 /// (both at the start of an UnwrappedLine, with no macros open), so the
-/// Unexpander can be thrown away and parsing can continue.
+/// Reconstructor can be thrown away and parsing can continue.
 ///
 /// Given a mapping from the macro name identifier token in the macro call
 /// to the tokens of the macro call, for example:
 /// CLASSA -> CLASSA({public: void x();})
 ///
 /// When getting the formatted lines of the expansion via the \c addLine method
 /// (each '->' specifies a call to \c addLine ):
 /// -> class A {

clang/lib/Format/TokenAnalyzer.h

public:
   // that the next lines of \p Code should start at \p NextStartColumn, and
   // that \p Code should end at \p LastStartColumn if it ends in newline.
   // See also the documentation of clang::format::internal::reformat.
   Environment(StringRef Code, StringRef FileName, unsigned FirstStartColumn = 0,
               unsigned NextStartColumn = 0, unsigned LastStartColumn = 0);
 
   FileID getFileID() const { return ID; }
 
-  const SourceManager &getSourceManager() const { return SM; }
+  SourceManager &getSourceManager() const { return SM; }
 
   ArrayRef<CharSourceRange> getCharRanges() const { return CharRanges; }
 
   // Returns the column at which the fragment of code managed by this
   // environment starts.
   unsigned getFirstStartColumn() const { return FirstStartColumn; }
 
   // Returns the column at which subsequent lines of the fragment of code

clang/lib/Format/TokenAnalyzer.cpp

 
 std::pair<tooling::Replacements, unsigned>
 TokenAnalyzer::process(bool SkipAnnotation) {
   tooling::Replacements Result;
   llvm::SpecificBumpPtrAllocator<FormatToken> Allocator;
   IdentifierTable IdentTable(getFormattingLangOpts(Style));
   FormatTokenLexer Lex(Env.getSourceManager(), Env.getFileID(),
                        Env.getFirstStartColumn(), Style, Encoding, Allocator,
-
                        IdentTable);
   ArrayRef<FormatToken *> Toks(Lex.lex());
   SmallVector<FormatToken *, 10> Tokens(Toks.begin(), Toks.end());
-  UnwrappedLineParser Parser(Style, Lex.getKeywords(),
-                             Env.getFirstStartColumn(), Tokens, *this);
+  UnwrappedLineParser Parser(Env.getSourceManager(), Style, Lex.getKeywords(),
+                             Env.getFirstStartColumn(), Tokens, *this,
+                             Allocator, IdentTable);
   Parser.parse();
   assert(UnwrappedLines.back().empty());
   unsigned Penalty = 0;
   for (unsigned Run = 0, RunE = UnwrappedLines.size(); Run + 1 != RunE; ++Run) {
     const auto &Lines = UnwrappedLines[Run];
     LLVM_DEBUG(llvm::dbgs() << "Run " << Run << "...\n");
     SmallVector<AnnotatedLine *, 16> AnnotatedLines;
     AnnotatedLines.reserve(Lines.size());

clang/lib/Format/TokenAnnotator.h

AnnotatedLine(const UnwrappedLine &Line)
         FirstStartColumn(Line.FirstStartColumn) {
     assert(!Line.Tokens.empty());
 
     // Calculate Next and Previous for all tokens. Note that we must overwrite
     // Next and Previous for every token, as previous formatting runs might have
     // left them in a different state.
     First->Previous = nullptr;
     FormatToken *Current = First;
+    addChildren(Line.Tokens.front(), Current);
     for (const UnwrappedLineNode &Node : llvm::drop_begin(Line.Tokens)) {
+      if (Node.Tok->MacroParent)
+        ContainsMacroCall = true;
       Current->Next = Node.Tok;
       Node.Tok->Previous = Current;
       Current = Current->Next;
-      Current->Children.clear();
-      for (const auto &Child : Node.Children) {
-        Children.push_back(new AnnotatedLine(Child));
-        Current->Children.push_back(Children.back());
-      }
-    }
-    Last = Current;
-    Last->Next = nullptr;
-  }
+      addChildren(Node, Current);
+      // FIXME: if we add children, previous will point to the token before
+      // the children; changing this requires significant changes across
+      // clang-format.
+    }
+    Last = Current;
+    Last->Next = nullptr;
+  }
+
+  void addChildren(const UnwrappedLineNode &Node, FormatToken *Current) {
+      if (Children.back()->ContainsMacroCall)
+        ContainsMacroCall = true;
 
   ~AnnotatedLine() {
     for (AnnotatedLine *Child : Children)
       delete Child;
     FormatToken *Current = First;
     while (Current) {
       Current->Children.clear();
       Current->Role.reset();
public:
   size_t MatchingClosingBlockLineIndex;
   bool InPPDirective;
   bool InPragmaDirective;
   bool InMacroBody;
   bool MustBeDeclaration;
   bool MightBeFunctionDecl;
   bool IsMultiVariableDeclStmt;
 
+  /// \c True if this line contains a macro call for which an expansion exists.
+  bool ContainsMacroCall = false;
+
   /// \c True if this line should be formatted, i.e. intersects directly or
   /// indirectly with one of the input ranges.
   bool Affected;
 
   /// \c True if the leading empty lines of this line intersect with one of the
   /// input ranges.
   bool LeadingEmptyLinesAffected;
 

clang/lib/Format/TokenAnnotator.cpp

while (Current) {
       if (Style.isVerilog() && Precedence == prec::Comma) {
         VerilogFirstOfType =
             verilogGroupDecl(VerilogFirstOfType, LatestOperator);
       }
 
       // Consume operators with higher precedence.
       parse(Precedence + 1);
 
+      // Do not assign fake parenthesis to tokens that are part of an
+      // unexpanded macro call. The line within the macro call contains
+      // the parenthesis and commas, and we will not find operators within
+      // that structure.
+      if (Current && Current->MacroParent)
+        break;
+
       int CurrentPrecedence = getCurrentPrecedence();
 
       if (Precedence == CurrentPrecedence && Current &&
           Current->is(TT_SelectorName)) {
         if (LatestOperator)
           addFakeParenthesis(Start, prec::Level(Precedence));
         Start = Current;
       }
if (Line.Type == LT_ObjCProperty &&
       (Right.is(tok::equal) || Left.is(tok::equal))) {
     return false;
   }
 
   if (Right.isOneOf(TT_TrailingReturnArrow, TT_LambdaArrow) ||
       Left.isOneOf(TT_TrailingReturnArrow, TT_LambdaArrow)) {
     return true;
   }
-  if (Left.is(tok::comma) && !Right.is(TT_OverloadedOperatorLParen))
+  if (Left.is(tok::comma) && !Right.is(TT_OverloadedOperatorLParen) &&
+      // In an unexpanded macro call we only find the parentheses and commas
+      // in a line; the commas and closing parenthesis do not require a space.
+      (Left.Children.empty() || !Left.MacroParent)) {
     return true;
+  }
   if (Right.is(tok::comma))
     return false;
   if (Right.is(TT_ObjCBlockLParen))
     return true;
   if (Right.is(TT_CtorInitializerColon))
     return Style.SpaceBeforeCtorInitializerColon;
   if (Right.is(TT_InheritanceColon) && !Style.SpaceBeforeInheritanceColon)
     return false;

clang/lib/Format/UnwrappedLineFormatter.cpp

 //===--- UnwrappedLineFormatter.cpp - Format C++ code ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "UnwrappedLineFormatter.h"
+#include "FormatToken.h"
 #include "NamespaceEndCommentsFixer.h"
 #include "WhitespaceManager.h"
 #include "llvm/Support/Debug.h"
 #include <queue>
 
 #define DEBUG_TYPE "format-formatter"
 
 namespace clang {
private:
   const SmallVectorImpl<AnnotatedLine *>::const_iterator End;
 
   SmallVectorImpl<AnnotatedLine *>::const_iterator Next;
   const SmallVectorImpl<AnnotatedLine *> &AnnotatedLines;
 };
 
 static void markFinalized(FormatToken *Tok) {
   for (; Tok; Tok = Tok->Next) {
+    if (Tok->MacroCtx && Tok->MacroCtx->Role == MR_ExpandedArg) {
+      // In the first pass we format all macro arguments in the expanded token
+      // stream. Instead of finalizing the macro arguments, we mark that they
+      // will be modified as unexpanded arguments (as part of the macro call
+      // formatting) in the next pass.
+      Tok->MacroCtx->Role = MR_UnexpandedArg;
+      // Reset whether spaces are required before this token, as that is context
+      // dependent, and that context may change when formatting the macro call.
+      // For example, given M(x) -> 2 * x, and the macro call M(var),
+      // the token 'var' will have SpacesRequiredBefore = 1 after being
+      // formatted as part of the expanded macro, but SpacesRequiredBefore = 0
+      // for its position within the macro call.
+      Tok->SpacesRequiredBefore = 0;
+    } else {
       Tok->Finalized = true;
-    for (AnnotatedLine *Child : Tok->Children)
-      markFinalized(Child->First);
+    }
   }
 }
 
 #ifndef NDEBUG
 static void printLineState(const LineState &State) {
   llvm::dbgs() << "State: ";
   for (const ParenState &P : State.Stack) {
     llvm::dbgs() << (P.Tok ? P.Tok->TokenText : "F") << "|" << P.Indent << "|"
protected:
   /// the closing "}" on yet another new line.
   ///
   /// This enables us to keep the simple structure of the
   /// \c UnwrappedLineFormatter, where we only have two options for each token:
   /// break or don't break.
   bool formatChildren(LineState &State, bool NewLine, bool DryRun,
                       unsigned &Penalty) {
     const FormatToken *LBrace = State.NextToken->getPreviousNonComment();
+    bool HasLBrace = LBrace && LBrace->is(tok::l_brace) && LBrace->is(BK_Block);
     FormatToken &Previous = *State.NextToken->Previous;
-    if (!LBrace || LBrace->isNot(tok::l_brace) || LBrace->isNot(BK_Block) ||
-        Previous.Children.size() == 0) {
+    if (Previous.Children.size() == 0 || (!HasLBrace && !LBrace->MacroParent)) {
       // The previous token does not open a block. Nothing to do. We don't
       // assert so that we can simply call this function for all tokens.
       return true;
     }
 
-    if (NewLine) {
+    if (NewLine || Previous.MacroParent) {
       const ParenState &P = State.Stack.back();
 
       int AdditionalIndent =
           P.Indent - Previous.Children[0]->Level * Style.IndentWidth;
 
       if (Style.LambdaBodyIndentation == FormatStyle::LBI_OuterScope &&
           P.NestedBlockIndent == P.LastSpace) {
         if (State.NextToken->MatchingParen &&
if (ShouldFormat && TheLine.Type != LT_Invalid) {
         bool LastLine = TheLine.First->is(tok::eof);
         formatFirstToken(TheLine, PreviousLine, PrevPrevLine, Lines, Indent,
                          LastLine ? LastStartColumn : NextStartColumn + Indent);
       }
 
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       unsigned ColumnLimit = getColumnLimit(TheLine.InPPDirective, NextLine);
       bool FitsIntoOneLine =
-          TheLine.Last->TotalLength + Indent <= ColumnLimit ||
+          !TheLine.ContainsMacroCall &&
+          (TheLine.Last->TotalLength + Indent <= ColumnLimit ||
            (TheLine.Type == LT_ImportStatement &&
             (!Style.isJavaScript() || !Style.JavaScriptWrapImports)) ||
            (Style.isCSharp() &&
-           TheLine.InPPDirective); // don't split #regions in C#
+            TheLine.InPPDirective)); // don't split #regions in C#
       if (Style.ColumnLimit == 0) {
         NoColumnLimitLineFormatter(Indenter, Whitespaces, Style, this)
             .formatLine(TheLine, NextStartColumn + Indent,
                         FirstLine ? FirstStartColumn : 0, DryRun);
       } else if (FitsIntoOneLine) {
         Penalty += NoLineBreakFormatter(Indenter, Whitespaces, Style, this)
                        .formatLine(TheLine, NextStartColumn + Indent,
                                    FirstLine ? FirstStartColumn : 0, DryRun);

clang/lib/Format/UnwrappedLineParser.h

 /// This file contains the declaration of the UnwrappedLineParser,
 /// which turns a stream of tokens into UnwrappedLines.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_FORMAT_UNWRAPPEDLINEPARSER_H
 #define LLVM_CLANG_LIB_FORMAT_UNWRAPPEDLINEPARSER_H
 
+#include "Encoding.h"
 #include "FormatToken.h"
+#include "Macros.h"
 #include "clang/Basic/IdentifierTable.h"
 #include "clang/Format/Format.h"
+#include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/BitVector.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/Support/Regex.h"
 #include <list>
 #include <stack>
 #include <vector>
 
 namespace clang {
 namespace format {
 
struct UnwrappedLine {
   /// opening line. Otherwise, \c MatchingOpeningBlockLineIndex must be
   /// \c kInvalidIndex.
   size_t MatchingOpeningBlockLineIndex = kInvalidIndex;
 
   /// If this \c UnwrappedLine opens a block, stores the index of the
   /// line with the corresponding closing brace.
   size_t MatchingClosingBlockLineIndex = kInvalidIndex;
 
   static const size_t kInvalidIndex = -1;
 
   unsigned FirstStartColumn = 0;
 };
 
+/// Interface for users of the UnwrappedLineParser to receive the parsed lines.
+/// Parsing a single snippet of code can lead to multiple runs, where each
+/// run is a coherent view of the file.
+///
+/// For example, different runs are generated:
+/// - for different combinations of #if blocks
+/// - when macros are involved, for the expanded code and the as-written code
+///
+/// Some tokens will only be visible in a subset of the runs.
+/// For each run, \c UnwrappedLineParser will call \c consumeUnwrappedLine
+/// for each parsed unwrapped line, and then \c finishRun to indicate
+/// that the set of unwrapped lines before is one coherent view of the
+/// code snippet to be formatted.
 class UnwrappedLineConsumer {
 public:
   virtual ~UnwrappedLineConsumer() {}
   virtual void consumeUnwrappedLine(const UnwrappedLine &Line) = 0;
   virtual void finishRun() = 0;
 };
 
 class FormatTokenSource;
 
 class UnwrappedLineParser {
 public:
-  UnwrappedLineParser(const FormatStyle &Style,
+  UnwrappedLineParser(SourceManager &SourceMgr, const FormatStyle &Style,
                       const AdditionalKeywords &Keywords,
                       unsigned FirstStartColumn, ArrayRef<FormatToken *> Tokens,
-                      UnwrappedLineConsumer &Callback);
+                      UnwrappedLineConsumer &Callback,
+                      llvm::SpecificBumpPtrAllocator<FormatToken> &Allocator,
+                      IdentifierTable &IdentTable);
 
   void parse();
 
 private:
   enum class IfStmtKind {
     NotIf,   // Not an if statement.
     IfOnly,  // An if statement without the else clause.
     IfElse,  // An if statement followed by else but not else if.
private:
   bool tryToParseSimpleAttribute();
   void parseVerilogHierarchyIdentifier();
   void parseVerilogSensitivityList();
   // Returns the number of levels of indentation in addition to the normal 1
   // level for a block, used for indenting case labels.
   unsigned parseVerilogHierarchyHeader();
   void parseVerilogTable();
   void parseVerilogCaseLabel();
+  std::optional<llvm::SmallVector<llvm::SmallVector<FormatToken *, 8>, 1>>
+  parseMacroCall();
 
   // Used by addUnwrappedLine to denote whether to keep or remove a level
   // when resetting the line state.
   enum class LineLevel { Remove, Keep };
 
   void addUnwrappedLine(LineLevel AdjustLevel = LineLevel::Remove);
   bool eof() const;
   // LevelDifference is the difference of levels after and before the current
private:
   // not be taken in this round.
   void conditionalCompilationCondition(bool Unreachable);
   void conditionalCompilationStart(bool Unreachable);
   void conditionalCompilationAlternative();
   void conditionalCompilationEnd();
 
   bool isOnNewLine(const FormatToken &FormatTok);
 
+  // Returns whether there is a macro expansion in the line, i.e. a token that
+  // was expanded from a macro call.
+  bool containsExpansion(const UnwrappedLine &Line) const;
+
   // Compute hash of the current preprocessor branch.
   // This is used to identify the different branches, and thus track if block
   // open and close in the same branch.
   size_t computePPHash() const;
 
+  bool parsingPPDirective() const { return CurrentLines != &Lines; }
+
   // FIXME: We are constantly running into bugs where Line.Level is incorrectly
   // subtracted from beyond 0. Introduce a method to subtract from Line.Level
   // and use that everywhere in the Parser.
   std::unique_ptr<UnwrappedLine> Line;
 
+  // Lines that are created by macro expansion.
+  // When formatting code containing macro calls, we first format the expanded
+  // lines to set the token types correctly. Afterwards, we format the
+  // reconstructed macro calls, re-using the token types determined in the first
+  // step.
+  // ExpandedLines will be reset every time we create a new LineAndExpansion
+  // instance once a line containing macro calls has been parsed.
+  SmallVector<UnwrappedLine, 8> CurrentExpandedLines;
+
+  // Maps from the first token of a top-level UnwrappedLine that contains
+  // a macro call to the replacement UnwrappedLines expanded from the macro
+  // call.
+  llvm::DenseMap<FormatToken *, SmallVector<UnwrappedLine, 8>> ExpandedLines;
+
+  // Map from the macro identifier to a line containing the full unexpanded
+  // macro call.
+  llvm::DenseMap<FormatToken *, std::unique_ptr<UnwrappedLine>> Unexpanded;
+
+  // For recursive macro expansions, trigger reconstruction only on the
+  // outermost expansion.
+  bool InExpansion = false;
+
+  // Set while we reconstruct a macro call.
+  // For reconstruction, we feed the expanded lines into the reconstructor
+  // until it is finished.
+  std::optional<MacroCallReconstructor> Reconstruct;
+
   // Comments are sorted into unwrapped lines by whether they are in the same
   // line as the previous token, or not. If not, they belong to the next token.
   // Since the next token might already be in a new unwrapped line, we need to
   // store the comments belonging to that token.
   SmallVector<FormatToken *, 1> CommentsBeforeNextToken;
   FormatToken *FormatTok;
   bool MustBreakBeforeNextToken;
 
private:
   // IncludeGuardState == IG_IfNdefed.
   FormatToken *IncludeGuardToken;
 
   // Contains the first start column where the source begins. This is zero for
   // normal source code and may be nonzero when formatting a code fragment that
   // does not start at the beginning of the file.
   unsigned FirstStartColumn;
 
+  MacroExpander Macros;
+
   friend class ScopedLineState;
   friend class CompoundStatementIndenter;
 };
 
 struct UnwrappedLineNode {
   UnwrappedLineNode() : Tok(nullptr) {}
-  UnwrappedLineNode(FormatToken *Tok) : Tok(Tok) {}
+  UnwrappedLineNode(FormatToken *Tok,
+                    llvm::ArrayRef<UnwrappedLine> Children = {})
+      : Tok(Tok), Children(Children.begin(), Children.end()) {}
 
   FormatToken *Tok;
   SmallVector<UnwrappedLine, 0> Children;
 };
 
 inline UnwrappedLine::UnwrappedLine()
     : Level(0), PPLevel(0), InPPDirective(false), InPragmaDirective(false),
       InMacroBody(false), MustBeDeclaration(false),
       MatchingOpeningBlockLineIndex(kInvalidIndex) {}
 
 } // end namespace format
 } // end namespace clang
 
 #endif

clang/lib/Format/UnwrappedLineParser.cpp

 //===--- UnwrappedLineParser.cpp - Format C++ code ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// This file contains the implementation of the UnwrappedLineParser,
 /// which turns a stream of tokens into UnwrappedLines.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "UnwrappedLineParser.h"
 #include "FormatToken.h"
+#include "FormatTokenLexer.h"
 #include "FormatTokenSource.h"
+#include "Macros.h"
 #include "TokenAnnotator.h"
 #include "clang/Basic/TokenKinds.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/raw_os_ostream.h"
 #include "llvm/Support/raw_ostream.h"
 
 #include <algorithm>
 #include <utility>
 
 #define DEBUG_TYPE "format-parser"
 
 namespace clang {
public:
   }
   ~CompoundStatementIndenter() { LineLevel = OldLineLevel; }
 
 private:
   unsigned &LineLevel;
   unsigned OldLineLevel;
 };
 
-UnwrappedLineParser::UnwrappedLineParser(const FormatStyle &Style,
-                                         const AdditionalKeywords &Keywords,
-                                         unsigned FirstStartColumn,
-                                         ArrayRef<FormatToken *> Tokens,
-                                         UnwrappedLineConsumer &Callback)
+UnwrappedLineParser::UnwrappedLineParser(
+    SourceManager &SourceMgr, const FormatStyle &Style,
+    const AdditionalKeywords &Keywords, unsigned FirstStartColumn,
+    ArrayRef<FormatToken *> Tokens, UnwrappedLineConsumer &Callback,
+    llvm::SpecificBumpPtrAllocator<FormatToken> &Allocator,
+    IdentifierTable &IdentTable)
     : Line(new UnwrappedLine), MustBreakBeforeNextToken(false),
       CurrentLines(&Lines), Style(Style), Keywords(Keywords),
       CommentPragmasRegex(Style.CommentPragmas), Tokens(nullptr),
       Callback(Callback), AllTokens(Tokens), PPBranchLevel(-1),
       IncludeGuard(Style.IndentPPDirectives == FormatStyle::PPDIS_None
                        ? IG_Rejected
                        : IG_Inited),
-      IncludeGuardToken(nullptr), FirstStartColumn(FirstStartColumn) {}
+      IncludeGuardToken(nullptr), FirstStartColumn(FirstStartColumn),
+      Macros(Style.Macros, SourceMgr, Style, Allocator, IdentTable) {}
 
 void UnwrappedLineParser::reset() {
   PPBranchLevel = -1;
   IncludeGuard = Style.IndentPPDirectives == FormatStyle::PPDIS_None
                      ? IG_Rejected
                      : IG_Inited;
   IncludeGuardToken = nullptr;
   Line.reset(new UnwrappedLine);
   CommentsBeforeNextToken.clear();
   FormatTok = nullptr;
   MustBreakBeforeNextToken = false;
   PreprocessorDirectives.clear();
   CurrentLines = &Lines;
   DeclarationScopeStack.clear();
   NestedTooDeep.clear();
   PPStack.clear();
   Line->FirstStartColumn = FirstStartColumn;
+
+  if (!Unexpanded.empty())
+    for (FormatToken *Token : AllTokens)
+      Token->MacroCtx.reset();
+  CurrentExpandedLines.clear();
+  ExpandedLines.clear();
+  Unexpanded.clear();
+  InExpansion = false;
+  Reconstruct.reset();
 }
 
 void UnwrappedLineParser::parse() {
   IndexedTokenSource TokenSource(AllTokens);
   Line->FirstStartColumn = FirstStartColumn;
   do {
     LLVM_DEBUG(llvm::dbgs() << "----\n");
     reset();
     Tokens = &TokenSource;
     TokenSource.reset();
 
     readToken();
     parseFile();
 
     // If we found an include guard then all preprocessor directives (other than
     // the guard) are over-indented by one.
     if (IncludeGuard == IG_Found) {
       for (auto &Line : Lines)
         if (Line.InPPDirective && Line.Level > 0)
           --Line.Level;
     }
 
     // Create line with eof token.
+    assert(FormatTok->is(tok::eof));
     pushToken(FormatTok);
     addUnwrappedLine();
 
-    for (const UnwrappedLine &Line : Lines)
+    // In a first run, format everything with the lines containing macro calls
+    // replaced by the expansion.
+    if (!ExpandedLines.empty()) {
+      LLVM_DEBUG(llvm::dbgs() << "Expanded lines:\n");
+      for (const auto &Line : Lines) {
+        if (!Line.Tokens.empty()) {
+          auto it = ExpandedLines.find(Line.Tokens.begin()->Tok);
+          if (it != ExpandedLines.end()) {
+            for (const auto &Expanded : it->second) {
+              LLVM_DEBUG(printDebugInfo(Expanded));
+              Callback.consumeUnwrappedLine(Expanded);
+            }
+            continue;
+          }
+        }
+        LLVM_DEBUG(printDebugInfo(Line));
         Callback.consumeUnwrappedLine(Line);
+      }
+      Callback.finishRun();
+    }
 
+    LLVM_DEBUG(llvm::dbgs() << "Unwrapped lines:\n");
+    for (const UnwrappedLine &Line : Lines) {
+      LLVM_DEBUG(printDebugInfo(Line));
+      Callback.consumeUnwrappedLine(Line);
+    }
     Callback.finishRun();
     Lines.clear();
     while (!PPLevelBranchIndex.empty() &&
            PPLevelBranchIndex.back() + 1 >= PPLevelBranchCount.back()) {
       PPLevelBranchIndex.resize(PPLevelBranchIndex.size() - 1);
       PPLevelBranchCount.resize(PPLevelBranchCount.size() - 1);
     }
     if (!PPLevelBranchIndex.empty()) {
FormatToken *UnwrappedLineParser::parseBlock(
   // Bail out if there are too many levels. Otherwise, the stack might overflow.
   if (Line->Level > 300)
     return nullptr;
 
   if (MacroBlock && FormatTok->is(tok::l_paren))
     parseParens();
 
   size_t NbPreprocessorDirectives =
-      CurrentLines == &Lines ? PreprocessorDirectives.size() : 0;
+      !parsingPPDirective() ? PreprocessorDirectives.size() : 0;
   addUnwrappedLine();
   size_t OpeningLineIndex =
       CurrentLines->empty()
           ? (UnwrappedLine::kInvalidIndex)
           : (CurrentLines->size() - 1 - NbPreprocessorDirectives);
 
   // Whitesmiths is weird here. The brace needs to be indented for the namespace
   // block, but the block itself may not be indented depending on the style
void UnwrappedLineParser::parseVerilogCaseLabel() {
   parseStructuralElement();
   // Restore the indentation in both the new line and the line that has the
   // label.
   if (CurrentLines->size() > FirstLine)
     (*CurrentLines)[FirstLine].Level = OrigLevel;
   Line->Level = OrigLevel;
 }
 
+bool UnwrappedLineParser::containsExpansion(const UnwrappedLine &Line) const {
+  for (const auto &N : Line.Tokens) {
+    if (N.Tok->MacroCtx)
+      return true;
+    for (const UnwrappedLine &Child : N.Children)
+      if (containsExpansion(Child))
+        return true;
+  }
+  return false;
+}
+
 void UnwrappedLineParser::addUnwrappedLine(LineLevel AdjustLevel) {
   if (Line->Tokens.empty())
     return;
   LLVM_DEBUG({
-    if (CurrentLines == &Lines)
+    if (!parsingPPDirective()) {
+      llvm::dbgs() << "Adding unwrapped line:\n";
       printDebugInfo(*Line);
+    }
   });
 
   // If this line closes a block when in Whitesmiths mode, remember that
   // information so that the level can be decreased after the line is added.
   // This has to happen after the addition of the line since the line itself
   // needs to be indented.
   bool ClosesWhitesmithsBlock =
       Line->MatchingOpeningBlockLineIndex != UnwrappedLine::kInvalidIndex &&
       Style.BreakBeforeBraces == FormatStyle::BS_Whitesmiths;
 
+  // If the current line was expanded from a macro call, we use it to
+  // reconstruct an unwrapped line from the structure of the expanded unwrapped
+  // line and the unexpanded token stream.
+  if (!parsingPPDirective() && !InExpansion && containsExpansion(*Line)) {
+    if (!Reconstruct)
+      Reconstruct.emplace(Line->Level, Unexpanded);
+    Reconstruct->addLine(*Line);
+
+    // While the reconstructed unexpanded lines are stored in the normal
+    // flow of lines, the expanded lines are stored on the side to be analyzed
+    // in an extra step.
+    CurrentExpandedLines.push_back(std::move(*Line));
+
+    if (Reconstruct->finished()) {
+      UnwrappedLine Reconstructed = std::move(*Reconstruct).takeResult();
+      assert(!Reconstructed.Tokens.empty() &&
+             "Reconstructed must at least contain the macro identifier.");
+      assert(!parsingPPDirective());
+      LLVM_DEBUG({
+        llvm::dbgs() << "Adding unexpanded line:\n";
+        printDebugInfo(Reconstructed);
+      });
+      ExpandedLines[Reconstructed.Tokens.begin()->Tok] = CurrentExpandedLines;
+      Lines.push_back(std::move(Reconstructed));
+      CurrentExpandedLines.clear();
+      Reconstruct.reset();
+    }
+  } else {
+    // At the top level we only get here when no unexpansion is going on, or
+    // when conditional formatting led to unfinished macro reconstructions.
+    assert(!Reconstruct || (CurrentLines != &Lines) || PPStack.size() > 0);
     CurrentLines->push_back(std::move(*Line));
+  }
   Line->Tokens.clear();
   Line->MatchingOpeningBlockLineIndex = UnwrappedLine::kInvalidIndex;
   Line->FirstStartColumn = 0;
   Line->IsContinuation = false;
 
   if (ClosesWhitesmithsBlock && AdjustLevel == LineLevel::Remove)
     --Line->Level;
-  if (CurrentLines == &Lines && !PreprocessorDirectives.empty()) {
+  if (!parsingPPDirective() && !PreprocessorDirectives.empty()) {
     CurrentLines->append(
         std::make_move_iterator(PreprocessorDirectives.begin()),
         std::make_move_iterator(PreprocessorDirectives.end()));
     PreprocessorDirectives.clear();
   }
   // Disconnect the current token from the last token on the previous line.
   FormatTok->Previous = nullptr;
 }
while (!Line->InPPDirective && FormatTok->is(tok::hash) &&
           FirstNonCommentOnLine, *FormatTok, PreviousWasComment);
     }
 
     if (!PPStack.empty() && (PPStack.back().Kind == PP_Unreachable) &&
         !Line->InPPDirective) {
       continue;
     }
 
+    if (FormatTok->is(tok::identifier) &&
+        Macros.defined(FormatTok->TokenText) &&
+        // FIXME: Allow expanding macros in preprocessor directives.
+        !Line->InPPDirective) {
+      FormatToken *ID = FormatTok;
+      unsigned Position = Tokens->getPosition();
+
+      // To correctly parse the code, we need to replace the tokens of the macro
+      // call with its expansion.
+      auto PreCall = std::move(Line);
+      Line.reset(new UnwrappedLine);
+      bool OldInExpansion = InExpansion;
+      InExpansion = true;
+      // We parse the macro call into a new line.
+      auto Args = parseMacroCall();
+      InExpansion = OldInExpansion;
+      assert(Line->Tokens.front().Tok == ID);
+      // And remember the unexpanded macro call tokens.
+      auto UnexpandedLine = std::move(Line);
+      // Reset to the old line.
+      Line = std::move(PreCall);
+
+      LLVM_DEBUG({
+        llvm::dbgs() << "Macro call: " << ID->TokenText << "(";
+        if (Args) {
+          llvm::dbgs() << "(";
+          for (const auto &Arg : Args.value())
+            for (const auto &T : Arg)
+              llvm::dbgs() << T->TokenText << " ";
+          llvm::dbgs() << ")";
+        }
+        llvm::dbgs() << "\n";
+      });
+      if (Macros.objectLike(ID->TokenText) && Args &&
+          !Macros.hasArity(ID->TokenText, Args->size())) {
+        // The macro is either
+        // - object-like, but we got argumnets, or
+        // - overloaded to be both object-like and function-like, but none of
+        //   the function-like arities match the number of arguments.
+        // Thus, expand as object-like macro.
+        LLVM_DEBUG(llvm::dbgs()
+                   << "Macro \"" << ID->TokenText
+                   << "\" not overloaded for arity " << Args->size()
+                   << "or not function-like, using object-like overload.");
+        Args.reset();
+        UnexpandedLine->Tokens.resize(1);
+        Tokens->setPosition(Position);
+        nextToken();
+        assert(!Args && Macros.objectLike(ID->TokenText));
+      }
+      if ((!Args && Macros.objectLike(ID->TokenText)) ||
+          (Args && Macros.hasArity(ID->TokenText, Args->size()))) {
+        // Next, we insert the expanded tokens in the token stream at the
+        // current position, and continue parsing.
+        Unexpanded[ID] = std::move(UnexpandedLine);
+        SmallVector<FormatToken *, 8> Expansion =
+            Macros.expand(ID, std::move(Args));
+        if (!Expansion.empty())
+          FormatTok = Tokens->insertTokens(Expansion);
+
+        LLVM_DEBUG({
+          llvm::dbgs() << "Expanded: ";
+          for (const auto &T : Expansion)
+            llvm::dbgs() << T->TokenText << " ";
+          llvm::dbgs() << "\n";
+        });
+      } else {
+        LLVM_DEBUG({
+          llvm::dbgs() << "Did not expand macro \"" << ID->TokenText
+                       << "\", because it was used ";
+          if (Args)
+            llvm::dbgs() << "with " << Args->size();
+          else
+            llvm::dbgs() << "without";
+          llvm::dbgs() << " arguments, which doesn't match any definition.\n";
+        });
+        Tokens->setPosition(Position);
+        FormatTok = ID;
+      }
+    }
+
     if (!FormatTok->is(tok::comment)) {
       distributeComments(Comments, FormatTok);
       Comments.clear();
       return;
     }
 
     Comments.push_back(FormatTok);
   } while (!eof());
 
   distributeComments(Comments, nullptr);
   Comments.clear();
 }
 
+namespace {
+template <typename Iterator>
+void pushTokens(Iterator Begin, Iterator End,
+                llvm::SmallVectorImpl<FormatToken *> &Into) {
+  for (auto I = Begin; I != End; ++I) {
+    Into.push_back(I->Tok);
+    for (const auto &Child : I->Children)
+      pushTokens(Child.Tokens.begin(), Child.Tokens.end(), Into);
+  }
+}
+} // namespace
+
+std::optional<llvm::SmallVector<llvm::SmallVector<FormatToken *, 8>, 1>>
+UnwrappedLineParser::parseMacroCall() {
+  std::optional<llvm::SmallVector<llvm::SmallVector<FormatToken *, 8>, 1>> Args;
+  assert(Line->Tokens.empty());
+  nextToken();
+  if (!FormatTok->is(tok::l_paren))
+    return Args;
+  unsigned Position = Tokens->getPosition();
+  FormatToken *Tok = FormatTok;
+  nextToken();
+  Args.emplace();
+  auto ArgStart = std::prev(Line->Tokens.end());
+
+  int Parens = 0;
+  do {
+    switch (FormatTok->Tok.getKind()) {
+    case tok::l_paren:
+      ++Parens;
+      nextToken();
+      break;
+    case tok::r_paren: {
+      if (Parens > 0) {
+        --Parens;
+        nextToken();
+        break;
+      }
+      Args->push_back({});
+      pushTokens(std::next(ArgStart), Line->Tokens.end(), Args->back());
+      nextToken();
+      return Args;
+    }
+    case tok::comma: {
+      if (Parens > 0) {
+        nextToken();
+        break;
+      }
+      Args->push_back({});
+      pushTokens(std::next(ArgStart), Line->Tokens.end(), Args->back());
+      nextToken();
+      ArgStart = std::prev(Line->Tokens.end());
+      break;
+    }
+    default:
+      nextToken();
+      break;
+    }
+  } while (!eof());
+  Line->Tokens.resize(1);
+  Tokens->setPosition(Position);
+  FormatTok = Tok;
+  return {};
+}
+
 void UnwrappedLineParser::pushToken(FormatToken *Tok) {
   Line->Tokens.push_back(UnwrappedLineNode(Tok));
   if (MustBreakBeforeNextToken) {
     Line->Tokens.back().Tok->MustBreakBefore = true;
     MustBreakBeforeNextToken = false;
   }
 }
 
 } // end namespace format
 } // end namespace clang

clang/lib/Format/WhitespaceManager.cpp

: Tok(&Tok), CreateReplacement(CreateReplacement),
       PreviousEndOfTokenColumn(0), EscapedNewlineColumn(0),
       StartOfBlockComment(nullptr), IndentationOffset(0), ConditionalsLevel(0) {
 }
 
 void WhitespaceManager::replaceWhitespace(FormatToken &Tok, unsigned Newlines,
                                           unsigned Spaces,
                                           unsigned StartOfTokenColumn,
                                           bool IsAligned, bool InPPDirective) {
-  if (Tok.Finalized)
+  if (Tok.Finalized || (Tok.MacroCtx && Tok.MacroCtx->Role == MR_ExpandedArg))
     return;
   Tok.setDecision((Newlines > 0) ? FD_Break : FD_Continue);
   Changes.push_back(Change(Tok, /*CreateReplacement=*/true, Tok.WhitespaceRange,
                            Spaces, StartOfTokenColumn, Newlines, "", "",
                            IsAligned, InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
 void WhitespaceManager::addUntouchableToken(const FormatToken &Tok,
                                             bool InPPDirective) {
-  if (Tok.Finalized)
+  if (Tok.Finalized || (Tok.MacroCtx && Tok.MacroCtx->Role == MR_ExpandedArg))
     return;
   Changes.push_back(Change(Tok, /*CreateReplacement=*/false,
                            Tok.WhitespaceRange, /*Spaces=*/0,
                            Tok.OriginalColumn, Tok.NewlinesBefore, "", "",
                            /*IsAligned=*/false, InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
 llvm::Error
 WhitespaceManager::addReplacement(const tooling::Replacement &Replacement) {
   return Replaces.add(Replacement);
 }
 
 bool WhitespaceManager::inputUsesCRLF(StringRef Text, bool DefaultToCRLF) {
   size_t LF = Text.count('\n');
   size_t CR = Text.count('\r') * 2;
   return LF == CR ? DefaultToCRLF : CR > LF;
 }
 
 void WhitespaceManager::replaceWhitespaceInToken(
     const FormatToken &Tok, unsigned Offset, unsigned ReplaceChars,
     StringRef PreviousPostfix, StringRef CurrentPrefix, bool InPPDirective,
     unsigned Newlines, int Spaces) {
-  if (Tok.Finalized)
+  if (Tok.Finalized || (Tok.MacroCtx && Tok.MacroCtx->Role == MR_ExpandedArg))
     return;
   SourceLocation Start = Tok.getStartOfNonWhitespace().getLocWithOffset(Offset);
   Changes.push_back(
       Change(Tok, /*CreateReplacement=*/true,
              SourceRange(Start, Start.getLocWithOffset(ReplaceChars)), Spaces,
              std::max(0, Spaces), Newlines, PreviousPostfix, CurrentPrefix,
              /*IsAligned=*/true, InPPDirective && !Tok.IsFirst,
              /*IsInsideToken=*/true));

clang/unittests/Format/FormatTest.cpp

protected:
 
   FormatStyle getGoogleStyleWithColumns(unsigned ColumnLimit) {
     return getStyleWithColumns(getGoogleStyle(), ColumnLimit);
   }
 
   void _verifyFormat(const char *File, int Line, llvm::StringRef Expected,
                      llvm::StringRef Code,
                      const FormatStyle &Style = getLLVMStyle()) {
     ScopedTrace t(File, Line, ::testing::Message() << Code.str());
     EXPECT_EQ(Expected.str(), format(Expected, Style))
         << "Expected code is not stable";
     EXPECT_EQ(Expected.str(), format(Code, Style));
     if (Style.Language == FormatStyle::LK_Cpp) {
       // Objective-C++ is a superset of C++, so everything checked for C++
       // needs to be checked for Objective-C++ as well.
       FormatStyle ObjCStyle = Style;
       ObjCStyle.Language = FormatStyle::LK_ObjC;
TEST_F(FormatTest, MergeLessLessAtEnd) {
   verifyFormat("<<");
   EXPECT_EQ("< < <", format("\\\n<<<"));
   verifyFormat("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
                "aaallvm::outs() <<");
   verifyFormat("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
                "aaaallvm::outs()\n    <<");
 }
 
+TEST_F(FormatTest, UnexpandConfiguredMacros) {
+  FormatStyle Style = getLLVMStyle();
+  Style.Macros.push_back("CLASS=class C {");
+  Style.Macros.push_back("SEMI=;");
+  Style.Macros.push_back("STMT=f();");
+  Style.Macros.push_back("ID(x)=x");
+  Style.Macros.push_back("ID3(x, y, z)=x y z");
+  Style.Macros.push_back("CALL(x)=f([] { x })");
+  Style.Macros.push_back("ASSIGN_OR_RETURN(a, b)=a = (b)");
+  Style.Macros.push_back("ASSIGN_OR_RETURN(a, b, c)=a = (b); if (x) return c");
+  Style.Macros.push_back("MOCK_METHOD(r, n, a, s)=r n a s");
+
+  verifyFormat("ID(nested(a(b, c), d))", Style);
+  verifyFormat("CLASS\n"
+               "  a *b;\n"
+               "};",
+               Style);
+  verifyFormat("SEMI\n"
+               "SEMI\n"
+               "SEMI",
+               Style);
+  verifyFormat("STMT\n"
+               "STMT\n"
+               "STMT",
+               Style);
+  verifyFormat("void f() { ID(a *b); }", Style);
+  verifyFormat(R"(ID(
+    { ID(a *b); });
+)",
+               Style);
+  verifyIncompleteFormat(R"(ID3({, ID(a *b),
+  ;
+  });
+)",
+                         Style);
+
+  verifyFormat("ID(CALL(CALL(return a * b;)));", Style);
+
+  verifyFormat("ASSIGN_OR_RETURN(MySomewhatLongType *variable,\n"
+               "                 MySomewhatLongFunction(SomethingElse()));\n",
+               Style);
+  verifyFormat("ASSIGN_OR_RETURN(MySomewhatLongType *variable,\n"
+               "                 MySomewhatLongFunction(SomethingElse()), "
+               "ReturnMe());\n",
+               Style);
+
+  verifyFormat(R"(
+#define MACRO(a, b) ID(a + b)
+)",
+               Style);
+  EXPECT_EQ(R"(
+int a;
+int b;
+int c;
+int d;
+int e;
+int f;
+ID(
+    namespace foo {
+    int a;
+    }
+) // namespace k
+)",
+            format(R"(
+int a;
+int b;
+int c;
+int d;
+int e;
+int f;
+ID(namespace foo { int a; })  // namespace k
+)",
+                   Style));
+  verifyFormat(R"(ID(
+    //
+    ({ ; }))
+)",
+               Style);
+
+  Style.ColumnLimit = 35;
+  // FIXME: Arbitrary formatting of macros where the end of the logical
+  // line is in the middle of a macro call are not working yet.
+  verifyFormat(R"(ID(
+    void f();
+    void)
+ID(g) ID(()) ID(
+    ;
+    void g();)
+)",
+               Style);
+
+  Style.ColumnLimit = 10;
+  verifyFormat("STMT\n"
+               "STMT\n"
+               "STMT",
+               Style);
+
+  EXPECT_EQ(R"(
+ID(CALL(CALL(
+    a *b)));
+)",
+            format(R"(
+ID(CALL(CALL(a * b)));
+)",
+                   Style));
+
+  // FIXME: If we want to support unbalanced braces or parens from macro
+  // expansions we need to re-think how we propagate errors in
+  // TokenAnnotator::parseLine; for investigation, switching the inner loop of
+  // TokenAnnotator::parseLine to return LT_Other instead of LT_Invalid in case
+  // of !consumeToken() changes the formatting of the test below and makes it
+  // believe it has a fully correct formatting.
+  EXPECT_EQ(R"(
+ID3(
+    {
+    CLASS
+    a *b;
+    };
+    },
+    ID(x *y);
+    ,
+    STMT
+    STMT
+    STMT)
+void f();
+)",
+            format(R"(
+ID3({CLASS a*b; };}, ID(x*y);, STMT STMT STMT)
+void f();
+)",
+                   Style));
+
+  verifyFormat("ID(a(\n"
+               "#ifdef A\n"
+               "    b, c\n"
+               "#else\n"
+               "    d(e)\n"
+               "#endif\n"
+               "    ))",
+               Style);
+  Style.ColumnLimit = 80;
+  verifyFormat(R"(ASSIGN_OR_RETURN(
+    // Comment
+    a b, c);
+)",
+               Style);
+  Style.ColumnLimit = 30;
+  verifyFormat(R"(ASSIGN_OR_RETURN(
+    // Comment
+    //
+    a b,
+    xxxxxxxxxxxx(
+        yyyyyyyyyyyyyyyyy,
+        zzzzzzzzzzzzzzzzzz),
+    f([]() {
+      a();
+      b();
+    }));
+)",
+               Style);
+  verifyFormat(R"(int a = []() {
+  ID(
+      x;
+      y;
+      z;)
+  ;
+}();
+)",
+               Style);
+  EXPECT_EQ(
+      R"(ASSIGN_OR_RETURN((
+====
+#))
+})",
+      format(R"(ASSIGN_OR_RETURN((
+====
+#))
+})",
+             Style, SC_ExpectIncomplete));
+  EXPECT_EQ(R"(ASSIGN_OR_RETURN(
+}
+(
+====
+#),
+    a))",
+            format(R"(ASSIGN_OR_RETURN(
+}
+(
+====
+#),
+a))",
+                   Style, SC_ExpectIncomplete));
+  EXPECT_EQ(R"(ASSIGN_OR_RETURN(a
+//
+====
+#
+                 <))",
+            format(R"(ASSIGN_OR_RETURN(a
+//
+====
+#
+                 <))",
+                   Style));
+  verifyFormat("class C {\n"
+               "  MOCK_METHOD(R, f,\n"
+               "              (a *b, c *d),\n"
+               "              (override));\n"
+               "};",
+               Style);
+}
+
+TEST_F(FormatTest, KeepParensWhenExpandingObjectLikeMacros) {
+  FormatStyle Style = getLLVMStyle();
+  Style.Macros.push_back("FN=class C { int f");
+  verifyFormat("void f() {\n"
+               "  FN(a *b);\n"
+               "  };\n"
+               "}",
+               Style);
+}
+
+TEST_F(FormatTest, DoesNotExpandFunctionLikeMacrosWithoutParens) {
+  FormatStyle Style = getLLVMStyle();
+  Style.Macros.push_back("CLASS()=class C {");
+  verifyFormat("CLASS void f();\n"
+               "}\n"
+               ";",
+               Style);
+}
+
+TEST_F(FormatTest, ContinueFormattingAfterUnclosedParensAfterObjectLikeMacro) {
+  FormatStyle Style = getLLVMStyle();
+  Style.Macros.push_back("O=class {");
+  verifyIncompleteFormat("O(auto x = [](){\n"
+                         "    f();}",
+                         Style);
+}
+
 TEST_F(FormatTest, HandleUnbalancedImplicitBracesAcrossPPBranches) {
   std::string code = "#if A\n"
                      "#if B\n"
                      "a.\n"
                      "#endif\n"
                      "    a = 1;\n"
                      "#else\n"
                      "#endif\n"

clang/unittests/Format/MacroCallReconstructorTest.cpp

public:
   Expansion(TestLexer &Lex, MacroExpander &Macros) : Lex(Lex), Macros(Macros) {}
 
   // Appends the token stream obtained from expanding the macro Name given
   // the provided arguments, to be later retrieved with getTokens().
   // Returns the list of tokens making up the unexpanded macro call.
   TokenList
   expand(llvm::StringRef Name,
          const SmallVector<llvm::SmallVector<FormatToken *, 8>, 1> &Args) {
+    return expandInternal(Name, Args);
+  }
+
+  TokenList expand(llvm::StringRef Name) { return expandInternal(Name, {}); }
+
+  TokenList expand(llvm::StringRef Name, const std::vector<std::string> &Args) {
+    return expandInternal(Name, lexArgs(Args));
+  }
+
+  const UnexpandedMap &getUnexpanded() const { return Unexpanded; }
+
+  const TokenList &getTokens() const { return Tokens; }
+
+private:
+  TokenList expandInternal(
+      llvm::StringRef Name,
+      const std::optional<SmallVector<llvm::SmallVector<FormatToken *, 8>, 1>>
+          &Args) {
     auto *ID = Lex.id(Name);
     auto UnexpandedLine = std::make_unique<UnwrappedLine>();
     UnexpandedLine->Tokens.push_back(ID);
-    if (!Args.empty()) {
+    if (Args && !Args->empty()) {
       UnexpandedLine->Tokens.push_back(Lex.id("("));
-      for (auto I = Args.begin(), E = Args.end(); I != E; ++I) {
-        if (I != Args.begin())
+      for (auto I = Args->begin(), E = Args->end(); I != E; ++I) {
+        if (I != Args->begin())
           UnexpandedLine->Tokens.push_back(Lex.id(","));
         UnexpandedLine->Tokens.insert(UnexpandedLine->Tokens.end(), I->begin(),
                                       I->end());
       }
       UnexpandedLine->Tokens.push_back(Lex.id(")"));
     }
     Unexpanded[ID] = std::move(UnexpandedLine);
 
     auto Expanded = uneof(Macros.expand(ID, Args));
     Tokens.append(Expanded.begin(), Expanded.end());
 
     TokenList UnexpandedTokens;
     for (const UnwrappedLineNode &Node : Unexpanded[ID]->Tokens)
       UnexpandedTokens.push_back(Node.Tok);
     return UnexpandedTokens;
   }
 
-  TokenList expand(llvm::StringRef Name,
-                   const std::vector<std::string> &Args = {}) {
-    return expand(Name, lexArgs(Args));
-  }
-
-  const UnexpandedMap &getUnexpanded() const { return Unexpanded; }
-
-  const TokenList &getTokens() const { return Tokens; }
-
-private:
   llvm::SmallVector<TokenList, 1>
   lexArgs(const std::vector<std::string> &Args) {
     llvm::SmallVector<TokenList, 1> Result;
     for (const auto &Arg : Args)
       Result.push_back(uneof(Lex.lex(Arg)));
     return Result;
   }
   llvm::DenseMap<FormatToken *, std::unique_ptr<UnwrappedLine>> Unexpanded;
auto Expected = line({
               U.consume(")"),
           }),
       }),
       Postfix,
   });
   EXPECT_THAT(std::move(Unexp).takeResult(), matchesLine(Expected));
 }
 
-TEST_F(MacroCallReconstructorTest, UnusedMacroArguments) {
-  auto Macros = createExpander({"X=x"});
-  Expansion Exp(Lex, *Macros);
-  TokenList Call = Exp.expand("X", {"a", "b", "c"});
-
-  MacroCallReconstructor Unexp(0, Exp.getUnexpanded());
-  Unexp.addLine(line(Exp.getTokens()));
-  EXPECT_TRUE(Unexp.finished());
-  Matcher U(Call, Lex);
-  EXPECT_THAT(std::move(Unexp).takeResult(),
-              matchesLine(line(U.consume("X(a, b, c)"))));
-}
-
-TEST_F(MacroCallReconstructorTest, UnusedEmptyMacroArgument) {
-  auto Macros = createExpander({"X=x"});
-  Expansion Exp(Lex, *Macros);
-  TokenList Call = Exp.expand("X", {std::string("")});
-
-  MacroCallReconstructor Unexp(0, Exp.getUnexpanded());
-  Matcher E(Exp.getTokens(), Lex);
-  auto Semi = tokens(";");
-  Unexp.addLine(line({E.consume("x"), Semi}));
-  EXPECT_TRUE(Unexp.finished());
-  Matcher U(Call, Lex);
-  EXPECT_THAT(std::move(Unexp).takeResult(),
-              matchesLine(line({U.consume("X()"), Semi})));
-}
-
 TEST_F(MacroCallReconstructorTest, ChildrenSplitAcrossArguments) {
   auto Macros = createExpander({"CALL(a, b)=f([]() a b)"});
   Expansion Exp(Lex, *Macros);
   TokenList Call = Exp.expand("CALL", {std::string("{ a;"), "b; }"});
 
   MacroCallReconstructor Unexp(0, Exp.getUnexpanded());
   Matcher E(Exp.getTokens(), Lex);
   Unexp.addLine(line({
auto Expected = line({
               Postfix,
           })),
       })),
   });
   EXPECT_THAT(std::move(Unexp).takeResult(), matchesLine(Expected));
 }
 
 TEST_F(MacroCallReconstructorTest, InvalidCodeSplittingBracesAcrossArgs) {
-  auto Macros = createExpander({"M(a, b)=(a) (b)"});
+  auto Macros = createExpander({"M(a, b, c)=(a) (b) c"});
   Expansion Exp(Lex, *Macros);
   TokenList Call = Exp.expand("M", {std::string("{"), "x", ""});
 
   MacroCallReconstructor Unexp(0, Exp.getUnexpanded());
   Matcher E(Exp.getTokens(), Lex);
   auto Prefix = tokens("({");
   Unexp.addLine(line({
       Prefix,

clang/unittests/Format/MacroExpanderTest.cpp

public:
   MacroExpanderTest() : Lex(Allocator, Buffers) {}
   std::unique_ptr<MacroExpander>
   create(const std::vector<std::string> &MacroDefinitions) {
     return std::make_unique<MacroExpander>(MacroDefinitions,
                                            Lex.SourceMgr.get(), Lex.Style,
                                            Lex.Allocator, Lex.IdentTable);
   }
 
+  std::string expand(MacroExpander &Macros, llvm::StringRef Name) {
+    EXPECT_TRUE(Macros.defined(Name))
+        << "Macro not defined: \"" << Name << "\"";
+    return text(Macros.expand(Lex.id(Name), {}));
+  }
+
   std::string expand(MacroExpander &Macros, llvm::StringRef Name,
-                     const std::vector<std::string> &Args = {}) {
-    EXPECT_TRUE(Macros.defined(Name));
+                     const std::vector<std::string> &Args) {
+    EXPECT_TRUE(Macros.defined(Name))
+        << "Macro not defined: \"" << Name << "\"";
     return text(Macros.expand(Lex.id(Name), lexArgs(Args)));
   }
 
   llvm::SmallVector<TokenList, 1>
   lexArgs(const std::vector<std::string> &Args) {
     llvm::SmallVector<TokenList, 1> Result;
     for (const auto &Arg : Args)
       Result.push_back(uneof(Lex.lex(Arg)));
TEST_F(MacroExpanderTest, ExpandsWithoutArguments) {
   });
   EXPECT_TRUE(Macros->objectLike("A"));
   EXPECT_TRUE(Macros->objectLike("B"));
   EXPECT_TRUE(Macros->objectLike("C"));
   EXPECT_TRUE(!Macros->objectLike("D"));
   EXPECT_EQ("", expand(*Macros, "A"));
   EXPECT_EQ("b", expand(*Macros, "B"));
   EXPECT_EQ("c+c", expand(*Macros, "C"));
-  EXPECT_EQ("", expand(*Macros, "D"));
+  EXPECT_EQ("", expand(*Macros, "D", {}));
 }
 
 TEST_F(MacroExpanderTest, ExpandsWithArguments) {
   auto Macros = create({
       "A(x)",
       "B(x, y)=x + y",
   });
   EXPECT_EQ("", expand(*Macros, "A", {"a"}));
   EXPECT_EQ("b1+b2+b3", expand(*Macros, "B", {"b1", "b2 + b3"}));
-  EXPECT_EQ("x+", expand(*Macros, "B", {"x"}));
 }
 
 TEST_F(MacroExpanderTest, AttributizesTokens) {
   auto Macros = create({
       "A(x, y)={ x + y; }",
       "B(x, y)=x + 3 + y",
   });
   auto *A = Lex.id("A");
std::vector<MacroAttributes> Attributes = {
       {tok::identifier, MR_ExpandedArg, 0, 0, {A}},
       {tok::equal, MR_Hidden, 0, 0, {A}},
       {tok::numeric_constant, MR_Hidden, 0, 0, {A}},
       {tok::semi, MR_Hidden, 0, 1, {A}},
   };
   EXPECT_ATTRIBUTES(Result, Attributes);
 }
 
+TEST_F(MacroExpanderTest, Overloads) {
+  auto Macros = create({"A=x", "A()=y", "A(a)=a", "A(a, b)=a b"});
+  EXPECT_EQ("x", expand(*Macros, "A"));
+  EXPECT_EQ("y", expand(*Macros, "A", {}));
+  EXPECT_EQ("z", expand(*Macros, "A", {"z"}));
+  EXPECT_EQ("xy", expand(*Macros, "A", {"x", "y"}));
+}
+
 } // namespace
 } // namespace format
 } // namespace clang

clang/unittests/Format/TestLexer.h

TokenList lex(llvm::StringRef Code) {
     FormatTokenLexer Lex = getNewLexer(Code);
     ArrayRef<FormatToken *> Result = Lex.lex();
     return TokenList(Result.begin(), Result.end());
   }
 
   TokenList annotate(llvm::StringRef Code) {
     FormatTokenLexer Lex = getNewLexer(Code);
     auto Tokens = Lex.lex();
-    UnwrappedLineParser Parser(Style, Lex.getKeywords(), 0, Tokens, *this);
+    UnwrappedLineParser Parser(SourceMgr.get(), Style, Lex.getKeywords(), 0,
+                               Tokens, *this, Allocator, IdentTable);
     Parser.parse();
     TokenAnnotator Annotator(Style, Lex.getKeywords());
     for (auto &Line : UnwrappedLines) {
       AnnotatedLine Annotated(Line);
       Annotator.annotate(Annotated);
       Annotator.calculateFormattingInformation(Annotated);
     }
     UnwrappedLines.clear();