[clang-format] Further improve support for requires expressions

[HazardyKnusperkeks]

Summary

Detect requires expressions in more unusable contexts. This is far from
perfect, but currently we have no good metric to decide between a
requires expression and a trailing requires clause.

[curdeius]

Some nits.

clang/lib/Format/UnwrappedLineParser.cpp

1564–1565

2817

2844

2852

2880

2888

[HazardyKnusperkeks]

Rebased and updated

[• Quuxplusone]

clang/lib/Format/UnwrappedLineParser.cpp

2824–2828

s/doesn't/don't/
IIUC you're talking about, like, void member() && requires ( — is that right?
It might help the reader to give an example snippet right here. (OTOH, it might be "obvious," I don't know. I'm not in the target audience for this code.)
...Ah, I see you give a snippet on line 2837 that's basically what I mean; I just first felt the need for that snippet all the way up here.

2861–2867

I think it's weird that your heuristic parses backward rather than forward. I would think that the next token after the requires keyword tells you what it is with pretty high probability:
requires requires — it's a clause
requires identifier — it's a clause
requires { — it's an expression
requires ( — unclear, apply further heuristics

Or are those heuristics already present in trunk, and this PR is just dealing with the "unclear" case?

2908

[HazardyKnusperkeks]

clang/lib/Format/UnwrappedLineParser.cpp

2861–2867

That would be so much better, but I can't easily look forward. Next is still nullptr, until I call nextToken(), but then I'm already moved along.

But this got me thinking, at least for the easy stuff I can just go forward and don't start on the keyword in parseRequiresClause() and parseRequiresExpression(). The paren case is more tricky, but I will try something.

[HazardyKnusperkeks]

New approach for identifying the expressions.

[HazardyKnusperkeks]

clang/lib/Format/UnwrappedLineParser.cpp

2861–2867

Present in main everything is a clause, except for requires expressions in a constraint expression. So the stuff where you use the requires expression in a "normal" boolean expression are misparsed and thus most likely misformatted.

There is actually a peekToken(), let's see if this is better.

[curdeius]

LGTM!

[klimek]

Hey ho, sorry for the late comment here, but adding peekNextToken(n) is problematic, as this gets in the way of future changes we want to do to handle macros better.
Usually we want to use X = Tokens->getPosition() and FormatTok = Tokens->setPosition(X) pairs when doing look-ahead.
I did a quick attempt at fixing this, but ran into infinite loops later in the annotator :(

[klimek]

Generally, why do we need to have that much information? I.e. why do we need to know the exact type of the "requires" keyword?
I do understand we need to know the brace type, but that seems like it would be easier to figure out in the TokenAnnotator (where we already parsed UnwrappedLines).
Do we ever parse UnwrappedLines differently depending on requires clauses/expressions?
If not, we should really do the annotation in TokenAnnotator, where we already have nice parsing bounds from the parsed UnwrappedLines.

[HazardyKnusperkeks]

In D119138#3951749, @klimek wrote:

Generally, why do we need to have that much information? I.e. why do we need to know the exact type of the "requires" keyword?
I do understand we need to know the brace type, but that seems like it would be easier to figure out in the TokenAnnotator (where we already parsed UnwrappedLines).
Do we ever parse UnwrappedLines differently depending on requires clauses/expressions?
If not, we should really do the annotation in TokenAnnotator, where we already have nice parsing bounds from the parsed UnwrappedLines.

Who is we, I'm not part of that we and haven't heard of some macro improvements. And I don't see how that feature is harming you, but be my guest in changing that. If you look into the history of this change I had a heuristic approach which would only look behind to differentiate.

I don't know if that can be solved in the TokenAnnotator, but you and I have different opinions about that. I'd put more annotating in the UnwrappedLineParser, annotate it as soon as we can.

I'll happily review any changes proposed, but I will not rework this piece of code, unless I can see a big flaw in it (which I can't right now).

[klimek]

I changed it in 49aca00d63e14df8bc68fc4329e6cbc9c9805eb8.

"We" is the people working on clang-format :) I hope that we have a common goal of making clang-format as easy to maintain as we can.

FWIW, I once had the same opinion as you about best doing all parsing as early as possible, but djasper convinced me that the split was a good idea, and in the end, I think it turns out to be significantly less brittle to do more complex annotation in TokenAnnotator. E.g. we now have a lookahead limit of 50, which seems rather arbitrary, while in TokenAnnotator we could simply limit lookahead towards the current UnwrappedLine. Similarly, in TokenAnnotator, we already have all the parens connected, so we could simply look from requires l_paren to the corresponding r_paren and whether the next token is an l_brace. If I can find a bit of time I'll take an attempt at implementing it.

[HazardyKnusperkeks]

In D119138#3951850, @klimek wrote:

I changed it in 49aca00d63e14df8bc68fc4329e6cbc9c9805eb8.

"We" is the people working on clang-format :) I hope that we have a common goal of making clang-format as easy to maintain as we can.

FWIW, I once had the same opinion as you about best doing all parsing as early as possible, but djasper convinced me that the split was a good idea, and in the end, I think it turns out to be significantly less brittle to do more complex annotation in TokenAnnotator. E.g. we now have a lookahead limit of 50, which seems rather arbitrary, while in TokenAnnotator we could simply limit lookahead towards the current UnwrappedLine. Similarly, in TokenAnnotator, we already have all the parens connected, so we could simply look from requires l_paren to the corresponding r_paren and whether the next token is an l_brace. If I can find a bit of time I'll take an attempt at implementing it.

Your commit is in my view a an example of making that maintaining a bit harder, it didn't went through review, had you not posted it here I'd never seen it. LLVM receives to many commits to scan them for changes in clang-format. And as someone who isn't that long involved in clang-format I think there is an overview really missing.

[klimek]

For non-functional clean-ups generally llvm doesn't require pre-commit review - I did communicate here so people involved in the original change wouldn't miss the clean-up. I do agree that what commits to pre-review is a fine line, and usually try to err on the side of pre-review; I'll take your feedback into consideration for future changes.

Regarding a better overview, you're 100% right. This is something we've definitely not been good enough and we need to get better at.

clang/lib/Format/UnwrappedLineParser.h

private:
   void parseSwitch();
   void parseNamespace();
   void parseModuleImport();
   void parseNew();
   void parseAccessSpecifier();
   bool parseEnum();
   bool parseStructLike();
   void parseConcept();
-  void parseRequiresClause();
-  void parseRequiresExpression();
+  bool parseRequires();
+  void parseRequiresClause(FormatToken *RequiresToken);
+  void parseRequiresExpression(FormatToken *RequiresToken);
   void parseConstraintExpression();
   void parseJavaEnumBody();
   // Parses a record (aka class) as a top level element. If ParseAsExpr is true,
   // parses the record as a child block, i.e. if the class declaration is an
   // expression.
   void parseRecord(bool ParseAsExpr = false);
   void parseObjCLightweightGenerics();
   void parseObjCMethod();

clang/lib/Format/UnwrappedLineParser.cpp

public:
   // Returns the token preceding the token returned by the last call to
   // getNextToken() in the token stream, or nullptr if no such token exists.
   virtual FormatToken *getPreviousToken() = 0;
 
   // Returns the token that would be returned by the next call to
   // getNextToken().
   virtual FormatToken *peekNextToken() = 0;
 
+  // Returns the token that would be returned after the next N calls to
+  // getNextToken(). N needs to be greater than zero, and small enough that
+  // there are still tokens. Check for tok::eof with N-1 before calling it with
+  // N.
+  virtual FormatToken *peekNextToken(int N) = 0;
+
   // Returns whether we are at the end of the file.
   // This can be different from whether getNextToken() returned an eof token
   // when the FormatTokenSource is a view on a part of the token stream.
   virtual bool isEOF() = 0;
 
   // Gets the current position in the token stream, to be used by setPosition().
   virtual unsigned getPosition() = 0;
 
public:
   }
 
   FormatToken *peekNextToken() override {
     if (eof())
       return &FakeEOF;
     return PreviousTokenSource->peekNextToken();
   }
 
+  FormatToken *peekNextToken(int N) override {
+    assert(N > 0);
+    if (eof())
+      return &FakeEOF;
+    return PreviousTokenSource->peekNextToken(N);
+  }
+
   bool isEOF() override { return PreviousTokenSource->isEOF(); }
 
   unsigned getPosition() override { return PreviousTokenSource->getPosition(); }
 
   FormatToken *setPosition(unsigned Position) override {
     PreviousToken = nullptr;
     Token = PreviousTokenSource->setPosition(Position);
     return Token;
FormatToken *peekNextToken() override {
     int Next = Position + 1;
     LLVM_DEBUG({
       llvm::dbgs() << "Peeking ";
       dbgToken(Next);
     });
     return Tokens[Next];
   }
 
+  FormatToken *peekNextToken(int N) override {
+    assert(N > 0);
+    int Next = Position + N;
+    LLVM_DEBUG({
+      llvm::dbgs() << "Peeking (+" << (N - 1) << ") ";
+      dbgToken(Next);
+    });
+    return Tokens[Next];
+  }
+
   bool isEOF() override { return Tokens[Position]->is(tok::eof); }
 
   unsigned getPosition() override {
     LLVM_DEBUG(llvm::dbgs() << "Getting Position: " << Position << "\n");
     assert(Position >= 0);
     return Position;
   }
 
case tok::at:
         return;
       default:
         break;
       }
       break;
     case tok::kw_concept:
       parseConcept();
       return;
-    case tok::kw_requires:
-      parseRequiresClause();
+    case tok::kw_requires: {
+      bool ParsedClause = parseRequires();
+      if (ParsedClause)
         return;
+      break;
+    }
     case tok::kw_enum:
       // Ignore if this is part of "template <enum ...".
       if (Previous && Previous->is(tok::less)) {
         nextToken();
         break;
       }
 
       // parseEnum falls through and does not yet add an unwrapped line as an
do {
     case tok::identifier:
       if (Style.isJavaScript() &&
           (FormatTok->is(Keywords.kw_function) ||
            FormatTok->startsSequence(Keywords.kw_async, Keywords.kw_function)))
         tryToParseJSFunction();
       else
         nextToken();
       break;
-    case tok::kw_requires:
-      parseRequiresExpression();
+    case tok::kw_requires: {
+      auto RequiresToken = FormatTok;
+      nextToken();
+      parseRequiresExpression(RequiresToken);
       break;
+    }
     case tok::ampamp:
       if (AmpAmpTokenType != TT_Unknown)
         FormatTok->setType(AmpAmpTokenType);
       LLVM_FALLTHROUGH;
     default:
       nextToken();
       break;
     }
if (!FormatTok->Tok.is(tok::equal))
     return;
   nextToken();
   parseConstraintExpression();
   if (FormatTok->Tok.is(tok::semi))
     nextToken();
   addUnwrappedLine();
 }
 
+/// \brief Parses a requires, decides if it is a clause or an expression.
+/// \pre The current token has to be the requires keyword.
+/// \returns true if it parsed a clause.
+bool clang::format::UnwrappedLineParser::parseRequires() {
+  assert(FormatTok->Tok.is(tok::kw_requires) && "'requires' expected");
+  auto RequiresToken = FormatTok;
+
+  // We try to guess if it is a requires clause, or a requires expression. For
+  // that we first consume the keyword and check the next token.
+  nextToken();
+
+  switch (FormatTok->Tok.getKind()) {
+  case tok::l_brace:
+    // This can only be an expression, never a clause.
+    parseRequiresExpression(RequiresToken);
+    return false;
+  case tok::l_paren:
+    // Clauses and expression can start with a paren, it's unclear what we have.
+    break;
+  default:
+    // All other tokens can only be a clause.
+    parseRequiresClause(RequiresToken);
+    return true;
+  }
+
+  // Looking forward we would have to decide if there are function declaration
+  // like arguments to the requires expression:
+  // requires (T t) {
+  // Or there is a constraint expression for the requires clause:
+  // requires (C<T> && ...
+
+  // But first let's look behind.
+  auto *PreviousNonComment = RequiresToken->getPreviousNonComment();
+
+  if (!PreviousNonComment ||
+      PreviousNonComment->is(TT_RequiresExpressionLBrace)) {
+    // If there is no token, or an expression left brace, we are a requires
+    // clause within a requires expression.
+    parseRequiresClause(RequiresToken);
+    return true;
+  }
+
+  switch (PreviousNonComment->Tok.getKind()) {
+  case tok::greater:
+  case tok::r_paren:
+  case tok::kw_noexcept:
+  case tok::kw_const:
+    // This is a requires clause.
+    parseRequiresClause(RequiresToken);
+    return true;
+  case tok::amp:
+  case tok::ampamp: {
+    // This can be either:
+    // if (... && requires (T t) ...)
+    // Or
+    // void member(...) && requires (C<T> ...
+    // We check the one token before that for a const:
+    // void member(...) const && requires (C<T> ...
+    auto PrevPrev = PreviousNonComment->getPreviousNonComment();
+    if (PrevPrev && PrevPrev->is(tok::kw_const)) {
+      parseRequiresClause(RequiresToken);
+      return true;
+    }
+    break;
+  }
+  default:
+    // It's an expression.
+    parseRequiresExpression(RequiresToken);
+    return false;
+  }
+
+  // Now we look forward and try to check if the paren content is a parameter
+  // list. The parameters can be cv-qualified and contain references or
+  // pointers.
+  // So we want basically to check for TYPE NAME, but TYPE can contain all kinds
+  // of stuff: typename, const, *, &, &&, ::, identifiers.
+
+  int NextTokenOffset = 1;
+  auto NextToken = Tokens->peekNextToken(NextTokenOffset);
+  auto PeekNext = [&NextTokenOffset, &NextToken, this] {
+    ++NextTokenOffset;
+    NextToken = Tokens->peekNextToken(NextTokenOffset);
+  };
+
+  bool FoundType = false;
+  bool LastWasColonColon = false;
+  int OpenAngles = 0;
+
+  for (; NextTokenOffset < 50; PeekNext()) {
+    switch (NextToken->Tok.getKind()) {
+    case tok::kw_volatile:
+    case tok::kw_const:
+    case tok::comma:
+      parseRequiresExpression(RequiresToken);
+      return false;
+    case tok::r_paren:
+    case tok::pipepipe:
+      parseRequiresClause(RequiresToken);
+      return true;
+    case tok::eof:
+      // Break out of the loop.
+      NextTokenOffset = 50;
+      break;
+    case tok::coloncolon:
+      LastWasColonColon = true;
+      break;
+    case tok::identifier:
+      if (FoundType && !LastWasColonColon && OpenAngles == 0) {
+        parseRequiresExpression(RequiresToken);
+        return false;
+      }
+      FoundType = true;
+      LastWasColonColon = false;
+      break;
+    case tok::less:
+      ++OpenAngles;
+      break;
+    case tok::greater:
+      --OpenAngles;
+      break;
+    default:
+      if (NextToken->isSimpleTypeSpecifier()) {
+        parseRequiresExpression(RequiresToken);
+        return false;
+      }
+      break;
+    }
+  }
+
+  // This seems to be a complicated expression, just assume it's a clause.
+  parseRequiresClause(RequiresToken);
+  return true;
+}
+
 /// \brief Parses a requires clause.
-/// \pre The current token needs to be the requires keyword.
+/// \param RequiresToken The requires keyword token, which starts this clause.
+/// \pre We need to be on the next token after the requires keyword.
 /// \sa parseRequiresExpression
 ///
 /// Returns if it either has finished parsing the clause, or it detects, that
 /// the clause is incorrect.
-void UnwrappedLineParser::parseRequiresClause() {
-  assert(FormatTok->Tok.is(tok::kw_requires) && "'requires' expected");
-  assert(FormatTok->getType() == TT_Unknown);
+void UnwrappedLineParser::parseRequiresClause(FormatToken *RequiresToken) {
+  assert(FormatTok->getPreviousNonComment() == RequiresToken);
+  assert(RequiresToken->Tok.is(tok::kw_requires) && "'requires' expected");
+  assert(RequiresToken->getType() == TT_Unknown);
 
   // If there is no previous token, we are within a requires expression,
   // otherwise we will always have the template or function declaration in front
   // of it.
   bool InRequiresExpression =
-      !FormatTok->Previous ||
-      FormatTok->Previous->is(TT_RequiresExpressionLBrace);
+      !RequiresToken->Previous ||
+      RequiresToken->Previous->is(TT_RequiresExpressionLBrace);
 
-  FormatTok->setType(InRequiresExpression
+  RequiresToken->setType(InRequiresExpression
                              ? TT_RequiresClauseInARequiresExpression
                              : TT_RequiresClause);
 
-  nextToken();
   parseConstraintExpression();
 
   if (!InRequiresExpression)
     FormatTok->Previous->ClosesRequiresClause = true;
 }
 
 /// \brief Parses a requires expression.
-/// \pre The current token needs to be the requires keyword.
+/// \param RequiresToken The requires keyword token, which starts this clause.
+/// \pre We need to be on the next token after the requires keyword.
 /// \sa parseRequiresClause
 ///
 /// Returns if it either has finished parsing the expression, or it detects,
 /// that the expression is incorrect.
-void UnwrappedLineParser::parseRequiresExpression() {
-  assert(FormatTok->Tok.is(tok::kw_requires) && "'requires' expected");
-  assert(FormatTok->getType() == TT_Unknown);
+void UnwrappedLineParser::parseRequiresExpression(FormatToken *RequiresToken) {
+  assert(FormatTok->getPreviousNonComment() == RequiresToken);
+  assert(RequiresToken->Tok.is(tok::kw_requires) && "'requires' expected");
+  assert(RequiresToken->getType() == TT_Unknown);
 
-  FormatTok->setType(TT_RequiresExpression);
-  nextToken();
+  RequiresToken->setType(TT_RequiresExpression);
 
   if (FormatTok->is(tok::l_paren)) {
     FormatTok->setType(TT_RequiresExpressionLParen);
     parseParens();
   }
 
   if (FormatTok->is(tok::l_brace)) {
     FormatTok->setType(TT_RequiresExpressionLBrace);
     parseChildBlock(/*CanContainBracedList=*/false,
                     /*NextLBracesType=*/TT_CompoundRequirementLBrace);
   }
 }
 
 /// \brief Parses a constraint expression.
 ///
 /// This is either the definition of a concept, or the body of a requires
 /// clause. It returns, when the parsing is complete, or the expression is
 /// incorrect.
 void UnwrappedLineParser::parseConstraintExpression() {
   do {
     switch (FormatTok->Tok.getKind()) {
-    case tok::kw_requires:
-      parseRequiresExpression();
+    case tok::kw_requires: {
+      auto RequiresToken = FormatTok;
+      nextToken();
+      parseRequiresExpression(RequiresToken);
       break;
+    }
 
     case tok::l_paren:
       parseParens(/*AmpAmpTokenType=*/TT_BinaryOperator);
       break;
 
     case tok::l_square:
       if (!tryToParseLambda())
         return;

clang/unittests/Format/TokenAnnotatorTest.cpp

TEST_F(TokenAnnotatorTest, UnderstandsRequiresClausesAndConcepts) {
   EXPECT_TOKEN(Tokens[13], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[16], tok::ampamp, TT_BinaryOperator);
 
   Tokens = annotate("template <typename T>\n"
                     "concept C = requires(T t) {\n"
                     "  { t.foo() };\n"
                     "} && Bar<T> && Baz<T>;");
   ASSERT_EQ(Tokens.size(), 35u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[9], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[13], tok::l_brace, TT_RequiresExpressionLBrace);
   EXPECT_TOKEN(Tokens[23], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[28], tok::ampamp, TT_BinaryOperator);
 
   Tokens = annotate("template<typename T>\n"
                     "requires C1<T> && (C21<T> || C22<T> && C2e<T>) && C3<T>\n"
                     "struct Foo;");
   ASSERT_EQ(Tokens.size(), 36u) << Tokens;
+  EXPECT_TOKEN(Tokens[5], tok::kw_requires, TT_RequiresClause);
   EXPECT_TOKEN(Tokens[6], tok::identifier, TT_Unknown);
   EXPECT_EQ(Tokens[6]->FakeLParens.size(), 1u);
   EXPECT_TOKEN(Tokens[10], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[16], tok::pipepipe, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[21], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[27], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[31], tok::greater, TT_TemplateCloser);
   EXPECT_EQ(Tokens[31]->FakeRParens, 1u);
   EXPECT_TRUE(Tokens[31]->ClosesRequiresClause);
 
   Tokens =
       annotate("template<typename T>\n"
                "requires (C1<T> && (C21<T> || C22<T> && C2e<T>) && C3<T>)\n"
                "struct Foo;");
   ASSERT_EQ(Tokens.size(), 38u) << Tokens;
+  EXPECT_TOKEN(Tokens[5], tok::kw_requires, TT_RequiresClause);
   EXPECT_TOKEN(Tokens[7], tok::identifier, TT_Unknown);
   EXPECT_EQ(Tokens[7]->FakeLParens.size(), 1u);
   EXPECT_TOKEN(Tokens[11], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[17], tok::pipepipe, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[22], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[28], tok::ampamp, TT_BinaryOperator);
   EXPECT_TOKEN(Tokens[32], tok::greater, TT_TemplateCloser);
   EXPECT_EQ(Tokens[32]->FakeRParens, 1u);
   EXPECT_TOKEN(Tokens[33], tok::r_paren, TT_Unknown);
   EXPECT_TRUE(Tokens[33]->ClosesRequiresClause);
+
+  Tokens = annotate("template <typename T>\n"
+                    "void foo(T) noexcept requires Bar<T>;");
+  ASSERT_EQ(Tokens.size(), 18u) << Tokens;
+  EXPECT_TOKEN(Tokens[11], tok::kw_requires, TT_RequiresClause);
+
+  Tokens = annotate("template <typename T>\n"
+                    "struct S {\n"
+                    "  void foo() const requires Bar<T>;\n"
+                    "  void bar() const & requires Baz<T>;\n"
+                    "  void bar() && requires Baz2<T>;\n"
+                    "  void baz() const & noexcept requires Baz<T>;\n"
+                    "  void baz() && noexcept requires Baz2<T>;\n"
+                    "};\n"
+                    "\n"
+                    "void S::bar() const & requires Baz<T> { }");
+  ASSERT_EQ(Tokens.size(), 85u) << Tokens;
+  EXPECT_TOKEN(Tokens[13], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[25], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[36], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[49], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[61], tok::kw_requires, TT_RequiresClause);
+  EXPECT_TOKEN(Tokens[77], tok::kw_requires, TT_RequiresClause);
+
+  Tokens = annotate("void Class::member() && requires(Constant) {}");
+  ASSERT_EQ(Tokens.size(), 14u) << Tokens;
+  EXPECT_TOKEN(Tokens[7], tok::kw_requires, TT_RequiresClause);
+
+  Tokens = annotate("void Class::member() && requires(Constant<T>) {}");
+  ASSERT_EQ(Tokens.size(), 17u) << Tokens;
+  EXPECT_TOKEN(Tokens[7], tok::kw_requires, TT_RequiresClause);
+
+  Tokens =
+      annotate("void Class::member() && requires(Namespace::Constant<T>) {}");
+  ASSERT_EQ(Tokens.size(), 19u) << Tokens;
+  EXPECT_TOKEN(Tokens[7], tok::kw_requires, TT_RequiresClause);
+
+  Tokens = annotate("void Class::member() && requires(typename "
+                    "Namespace::Outer<T>::Inner::Constant) {}");
+  ASSERT_EQ(Tokens.size(), 24u) << Tokens;
+  EXPECT_TOKEN(Tokens[7], tok::kw_requires, TT_RequiresClause);
+}
+
+TEST_F(TokenAnnotatorTest, UnderstandsRequiresExpressions) {
+  auto Tokens = annotate("bool b = requires(int i) { i + 5; };");
+  ASSERT_EQ(Tokens.size(), 16u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[8], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("if (requires(int i) { i + 5; }) return;");
+  ASSERT_EQ(Tokens.size(), 17u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[7], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("if (func() && requires(int i) { i + 5; }) return;");
+  ASSERT_EQ(Tokens.size(), 21u) << Tokens;
+  EXPECT_TOKEN(Tokens[6], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[7], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[11], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("foo(requires(const T t) {});");
+  ASSERT_EQ(Tokens.size(), 13u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[8], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("foo(requires(const int t) {});");
+  ASSERT_EQ(Tokens.size(), 13u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[8], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("foo(requires(const T t) {});");
+  ASSERT_EQ(Tokens.size(), 13u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[8], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("foo(requires(int const* volatile t) {});");
+  ASSERT_EQ(Tokens.size(), 15u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[10], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("foo(requires(T const* volatile t) {});");
+  ASSERT_EQ(Tokens.size(), 15u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[10], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens =
+      annotate("foo(requires(const typename Outer<T>::Inner * const t) {});");
+  ASSERT_EQ(Tokens.size(), 21u) << Tokens;
+  EXPECT_TOKEN(Tokens[2], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[16], tok::l_brace, TT_RequiresExpressionLBrace);
+
+  Tokens = annotate("template <typename T>\n"
+                    "concept C = requires(T T) {\n"
+                    "  requires Bar<T> && Foo<T>;\n"
+                    "};");
+  ASSERT_EQ(Tokens.size(), 28u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[9], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[13], tok::l_brace, TT_RequiresExpressionLBrace);
+  EXPECT_TOKEN(Tokens[14], tok::kw_requires,
+               TT_RequiresClauseInARequiresExpression);
+
+  Tokens = annotate("template <typename T>\n"
+                    "concept C = requires(T T) {\n"
+                    "  { t.func() } -> std::same_as<int>;"
+                    "  requires Bar<T> && Foo<T>;\n"
+                    "};");
+  ASSERT_EQ(Tokens.size(), 43u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_requires, TT_RequiresExpression);
+  EXPECT_TOKEN(Tokens[9], tok::l_paren, TT_RequiresExpressionLParen);
+  EXPECT_TOKEN(Tokens[13], tok::l_brace, TT_RequiresExpressionLBrace);
+  EXPECT_TOKEN(Tokens[29], tok::kw_requires,
+               TT_RequiresClauseInARequiresExpression);
 }
 
 TEST_F(TokenAnnotatorTest, RequiresDoesNotChangeParsingOfTheRest) {
   auto NumberOfAdditionalRequiresClauseTokens = 5u;
   auto NumberOfTokensBeforeRequires = 5u;
 
   auto BaseTokens = annotate("template<typename T>\n"
                              "T Pi = 3.14;");