[pseudo] Implement a guard to determine function declarator.

[hokein]

Summary

This eliminates some simple-declaration/function-definition false parses.

• implement a function to determine whether a declarator ForestNode is a function declarator;
• extend the standard declarator to two guarded function-declarator and non-function-declarator nonterminals;

[sammccall]

clang-tools-extra/pseudo/gen/Main.cpp

60

This code is copied right?

62

this deserves a comment: "Compared to in the grammar, ; becomes semi and goto becomes kw_goto"?

113

pull out a ruleName() too?

or mangleSymbol()/mangleRule() by analogy with linker name mangling

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

29

This allows/encourages guards that dynamically consider multiple rules, which increases the amount of coupling between rules and grammar.

Can we express this as
function-declarator := declarator [guard=FunctionDeclarator]?

This does mangle the grammar a bit to our implementation, and introduces two names for the same concept (guard & nonterminal).
If this feels ugly and redundant, another option would be to change the guard to be specified by the rule ID instead of an extension ID:

function-declarator := declarator [guard]

DenseMap<RuleID, RuleGuard> Guards;

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

58

The "dfs" doesn't actually branch anywhere, so it's just a linear walk, and I think replacing the recursion with iteration is actually more readable here.

Also, the first known kind on the walk up == the last known kind on the walk down.

So I think this can be written as:

bool IsFunction = false;
// Walk down the declarator chain, innermost one wins.
// e.g. (*x)() is a non function, but *(x()) is a function.
for (;;) {
  if (Declarator->kind() != Sequence) // not well-formed, guess
    return IsFunction;

  switch (Declarator->rule()) {
    case noptr_declarator$$declarator_id: // reached the bottom
      return IsFunction;
     // *X is a nonfunction (unless X is a function).
    case ptr_declarator$$ptr_operator$ptr_declarator:
      Declarator = Declarator->elements()[1];
      IsFunction = false;
      continue;
     // X() is a function (unless X is a pointer or similar)
    case declarator$$noptr_declarator$parameters_and_qualifiers$...:
      Declarator = Declarator->elements()[0];
      IsFunction = true;
      continue;
    // (X) is whatever X is.
    case declarator$$l_paren$ptr_declarator$r_paren:
      Declarator = Declarator->elements()[1];
      continue;
    // ... more cases ...
    default:
      assert(false && "unhandled declarator for IsFunction");
      return IsFunction; 
  }
}

60

yes, we should. We can't descend, so I think we should just treat it as if it were an identifier.

108

if you haven't handled all rule kinds, you're just falling off the end here?
We should have an assertion in debug mode and probably treat it as opaque in production I think

[sammccall]

clang-tools-extra/pseudo/gen/Main.cpp

114

so the dollar signs are a practical problem: at least on my machine the warning is on by default, so we emit thousands of warnings and it's impossible to find the error among them.

At *minimum* we need to use pragmas or something to suppress the warning.

But using nonstandard C++ seems likely to cause other problems and limit portability for not-great reasons.

Some ideas:
-Rule::ptr_declarator__EQUALS__ptr_operator__ptr_declarator (this violates the internal __ rule, though that one is widely ignored)

• Rule::ptr_declarator_0ptr_operator_1ptr_declarator
• Rule::PtrDeclarator_EQ_PtrOperator_PtrDeclarator (yet another spelling variation, but quite readable. We could even change the spellings in the BNF file...)

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

29

Having played with this idea a bit, I do think we should strongly consider dispatching on the rule ID rather than a named extension. I went to add some more guards (on NUMERIC_CONSTANT etc) and they really are 1:1 with rules.

[hokein]

clang-tools-extra/pseudo/gen/Main.cpp

114

so the dollar signs are a practical problem: at least on my machine the warning is on by default, so we emit thousands of warnings and it's impossible to find the error among them.

I didn't see this warning on my machine, I'm using the g++.

Rule::ptr_declaratorEQUALSptr_operatorptr_declarator (this violates the internal rule, though that one is widely ignored)

It is not quite readable to me.

Rule::PtrDeclarator_EQ_PtrOperator_PtrDeclarator (yet another spelling variation, but quite readable. We could even change the spellings in the BNF file...)

It is better, but I'm not willing to change the spelling to CamelCase in bnf grammar, I like the current ptr-declarator name, and it matches the standard style. We could keep them unchanged in the bnf, and do the conversion during the mangling.

Rule::ptr_declarator_0ptr_operator_1ptr_declarator

I'm starting to like it now. I find that encoding the index for RHS is useful, especially when we do some work on the ForestNode based on the rules. So I will vote this one.

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

29

This allows/encourages guards that dynamically consider multiple rules, which increases the amount of coupling between rules and grammar.

Yeah, this is the sad bit of this approach, but IMO it is not that bad.

Can we express this as
function-declarator := declarator [guard=FunctionDeclarator]?
This does mangle the grammar a bit to our implementation, and introduces two names for the same concept (guard & nonterminal).

We can do this (we will also need another non-function-declarator nonterminal).

I feel uncomfortable of it, this seems a non-trivial&non-local change (we're changing an important nonterminal) -- Declarator is the only top-level declarator defined in the standard grammar, and now we add two more, and we need to update all related declaration rules. We're making it more complicated, which might introduce subtle grammar bugs.

There is a 3rd approach -- we could consider make the guard on symbol level rather than on Rule level. The guard corresponds to one of the grammar symbols on RHS.

29

Having played with this idea a bit, I do think we should strongly consider dispatching on the rule ID rather than a named extension. I went to add some more guards (on NUMERIC_CONSTANT etc) and they really are 1:1 with rules.

This is an interesting idea. The annotation syntax [guard] in grammar is not required, as we can just use {Rule::function-declarator$$declarator, guardFunctionDeclarator} to build the map, this means we loose the connection between the grammar annotation and C++ binding code, which seems like a regression to me :(

It has a hard restriction: every grammar rule will have at most 1 guard. It is probably fine, I can't think of a case where we will use multiple guards for a single rule (even we have, we could solve it by introducing a new rule with guard).

[sammccall]

clang-tools-extra/pseudo/gen/Main.cpp

114

I didn't see this warning on my machine, I'm using the g++.

Clang seems to enable -Wdollar-in-identifier-extension by default.

It is better, but I'm not willing to change the spelling to CamelCase in bnf grammar

Fair enough. I find this option the most readable, but the lack of consistency is annoying.

I'm starting to like it now. I find that encoding the index for RHS is useful, especially when we do some work on the ForestNode based on the rules. So I will vote this one.

Ok, i find this least readable, but good in all other respects. Let's go for it.

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

29

This is an interesting idea. The annotation syntax [guard] in grammar is not required ... This means we loose the connection between the grammar annotation and C++ binding code,

I don't really understand this argument: my suggestion was to keep the [guard] annotations to preserve this connection.
If you think that's unnecessary then we can drop it (i don't feel strongly). But it can't be both necessary and unnecessary!

29

Having played with this idea a bit, I do think we should strongly consider dispatching on the rule ID rather than a named extension. I went to add some more guards (on NUMERIC_CONSTANT etc) and they really are 1:1 with rules.

This is an interesting idea. The annotation syntax [guard] in grammar is not required, as we can just use {Rule::function-declarator$$declarator, guardFunctionDeclarator} to build the map, this means we loose the connection between the grammar annotation and C++ binding code, which seems like a regression to me :(

It has a hard restriction: every grammar rule will have at most 1 guard. It is probably fine, I can't think of a case where we will use multiple guards for a single rule (even we have, we could solve it by introducing a new rule with guard).

It's always possible, if the rule imposes two conditions then the rule's guard can check both.

29

Can we express this as
function-declarator := declarator [guard=FunctionDeclarator]?
This does mangle the grammar a bit to our implementation, and introduces two names for the same concept (guard & nonterminal).

We can do this (we will also need another non-function-declarator nonterminal).

I feel uncomfortable of it, this seems a non-trivial&non-local change (we're changing an important nonterminal)

To me this seems like an advantage:
This non-function-declarator is clearly an important concept being processed at grammatical parse time, it should have a name and be visible in the output!

Declarator is the only top-level declarator defined in the standard grammar, and now we add two more, and we need to update all related declaration rules. We're making it more complicated, which might introduce subtle grammar bugs.

This logic needs to live somewhere. We're using a grammar not just because the standard provides one, but because it's easier to reason about and check than imperative code.
Grammar bugs are possible, but they're not made any better by moving them out of the grammar!

There is a 3rd approach -- we could consider make the guard on symbol level rather than on Rule level. The guard corresponds to one of the grammar symbols on RHS.

Yeah. This looks more awkward to implement than rule guards, but i think the biggest difference is (not) representing "this declarator is a function declarator" in the forest.

[hokein]

clang-tools-extra/pseudo/gen/Main.cpp

114

Sounds good, separated it in https://reviews.llvm.org/D129359.

[hokein]

rebase and address the main comment -- encoding the function-declarator into the grammar.

[hokein]

rebase and add comments.

[hokein]

All comments should be addressed now. As discussed offline, I'm going to land it now, happy to address any post comments.

[sammccall]

impl looks good!

clang-tools-extra/pseudo/test/cxx/declarator-function.cpp

7–28

I'm not convinced that rewriting this test (rather than just removing the FIXME and XFAIL) is an improvement.

We're asserting a lot more stuff, but none of it is actually important. I find it tends to make it hard to find the signal among the noise of failures.

8

FWIW: I think that including the full contents of each line is bit unneccesary and brittle.

e.g. you can leave off the := ... without any risk of matching spuriously

clang-tools-extra/pseudo/test/cxx/declarator-var.cpp

6

this is testing essentially the same thing as the new int s2 case you added (though this one shows the subtle difference better IMO).

I think you should keep either one, but not both.

[hokein]

restore the previous and simplified function-declarator test.

[mizvekov]

It seems this change broke build on windows for me:

clang-tools-extra\pseudo\lib\cxx\CXX.cpp(59): error C2838: 'noptr_declarator_0declarator_id': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(59): error C2065: 'noptr_declarator_0declarator_id': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(59): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(59): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(62): error C2838: 'ptr_declarator_0ptr_operator_1ptr_declarator': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(62): error C2065: 'ptr_declarator_0ptr_operator_1ptr_declarator': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(62): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(62): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(68): error C2838: 'declarator_0noptr_declarator_1parameters_and_qualifiers_2trailing_return_type': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(68): error C2065: 'declarator_0noptr_declarator_1parameters_and_qualifiers_2trailing_return_type': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(68): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(68): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(69): error C2838: 'noptr_declarator_0noptr_declarator_1parameters_and_qualifiers': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(69): error C2065: 'noptr_declarator_0noptr_declarator_1parameters_and_qualifiers': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(69): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(69): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(75): error C2838: 'noptr_declarator_0noptr_declarator_1l_square_2constant_expression_3r_square': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(75): error C2065: 'noptr_declarator_0noptr_declarator_1l_square_2constant_expression_3r_square': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(75): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(75): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(76): error C2838: 'noptr_declarator_0noptr_declarator_1l_square_2r_square': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(76): error C2065: 'noptr_declarator_0noptr_declarator_1l_square_2r_square': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(76): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(76): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(81): error C2838: 'noptr_declarator_0l_paren_1ptr_declarator_2r_paren': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(81): error C2065: 'noptr_declarator_0l_paren_1ptr_declarator_2r_paren': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(81): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(81): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(84): error C2838: 'ptr_declarator_0noptr_declarator': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(84): error C2065: 'ptr_declarator_0noptr_declarator': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(84): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(84): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(85): error C2838: 'declarator_0ptr_declarator': illegal qualified name in member declaration
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(85): error C2065: 'declarator_0ptr_declarator': undeclared identifier
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(85): error C2131: expression did not evaluate to a constant
  clang-tools-extra\pseudo\lib\cxx\CXX.cpp(85): note: a non-constant (sub-)expression was encountered
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(59): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(62): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(67): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(69): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(74): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(76): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(81): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(84): error C2051: case expression not constant
clang-tools-extra\pseudo\lib\cxx\CXX.cpp(85): error C2051: case expression not constant

[mizvekov]

I confirm reverting just this patch fixes the build for me.

[sammccall]

In D129222#3666043, @mizvekov wrote:

I confirm reverting just this patch fixes the build for me.

Hi @mizvekov,
Can you provide details about the compiler version?
Neither the presubmit nor postsubmit bots (e.g.) had this error, and I can't relate the error messages to the code (member declaration on line 59?!)
I fear this is a compiler bug we'll need to work around.

@hokein it probably still makes sense to revert...

[mizvekov]

Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31627.1 for x64

As included in Microsoft Visual Studio 2022 Version 17.3.0 Preview 4.0

[mizvekov]

I un-reverted this locally to take a better look.

It does not seem to me to be a compiler bug, because both cl.exe and intellisense are picking this error up, and I believe the MSVC intellisense is based on a completely different frontend (EDG).

What I see is that enum class Rule is empty.

We try to populate it with

enum class Rule : RuleID {
#define RULE(X, Y) X = Y,
#include "CXXSymbols.inc"
#undef RULE
};

But I open CXXSymbols.inc and I see nothing that would expand that RULE macro, the contents of the file don't mention it at all, just NONTERMINAL and EXTENSION.

[mizvekov]

Running clang-pseudo-gen --grammar ..\clang-tools-extra\pseudo\lib\cxx\cxx.bnf --emit-symbol-list does seem to generate something that would work. But even if I delete CXXSymbols.inc, the cmake rule seems to be regenerating a wrong CXXSymbols.inc again, so I suspect this is a build system bug and we have wrong targets or dependecies somewhere.

[mizvekov]

The clang-pseudo-gen that sits in .\llvm\RelWithDebInfo\bin\clang-pseudo-gen generates correct output, the one that sits in .\llvm\NATIVE\Release\bin\clang-pseudo-gen.exe doesn't, which seems to be the one cmake is picking up for this job.

This could have something to do with "LLVM_OPTIMIZED_TABLEGEN": "ON", perhaps, and there is some dependency missing to rebuild the NATIVE target.

[mizvekov]

Yeah, turning off LLVM_OPTIMIZED_TABLEGEN made cmake instantly generate the correct CXXSymbols.inc. So while I think there is a bug out there somewhere, it might not necessarily be in this patch.

[sammccall]

Thanks for digging!
Agreed then that reverting this patch won't really solve anything. I expect I can reproduce on linux by enabling LLVM_OPTIMIZED_TABLEGEN in the same way, looking into it...

[hokein]

I'm sorry for the trouble it causes, and thanks for all details!

This could have something to do with "LLVM_OPTIMIZED_TABLEGEN": "ON", perhaps, and there is some dependency missing to rebuild the NATIVE target.

Yeah, the native clang-pseudo-gen tool didn't rebuild somehow even its source file changes, it is a bug in our cmake config. Should be fixed in 2955192df8ac270515b5fa4aaa9e9380148e7f00 (I verified it locally). Can you retry it with LLVM_OPTIMIZED_TABLEGEN on?

[mizvekov]

In D129222#3667638, @hokein wrote:

Yeah, the native clang-pseudo-gen tool didn't rebuild somehow even its source file changes, it is a bug in our cmake config. Should be fixed in 2955192df8ac270515b5fa4aaa9e9380148e7f00 (I verified it locally). Can you retry it with LLVM_OPTIMIZED_TABLEGEN on?

Yeah I confirm that fixes it, thanks!

Though it's strange that you would have to declare that dependency explicitly, I think it should be implied.

One other small thing I noticed: It seems the pseudo-gen tool generated output takes care of undefining the macros it takes as input to expand, but we don't take advantage of that fact on clang-tools-extra\pseudo\include\clang-pseudo\cxx\CXX.h and are always undefing them. Just a minor thing :)

[sammccall]

In D129222#3668073, @mizvekov wrote:

In D129222#3667638, @hokein wrote:

Yeah, the native clang-pseudo-gen tool didn't rebuild somehow even its source file changes, it is a bug in our cmake config. Should be fixed in 2955192df8ac270515b5fa4aaa9e9380148e7f00 (I verified it locally). Can you retry it with LLVM_OPTIMIZED_TABLEGEN on?

Yeah I confirm that fixes it, thanks!

Though it's strange that you would have to declare that dependency explicitly, I think it should be implied.

I think what's going on here: outer cmake invokes this opaque external command (which just _happens_ to be cmake --build) to produce an exe. The outer cmake has no special knowledge of what the inner cmake is going to use as inputs, so i it never invalidates the exe and invokes inner cmake again. Having the native exe depend on the target exe is a hack: it means that it (indirectly) depends on the right set of sources and is invalidated when they change. (In a perfect world we wouldn't build the target clang-pseudo-gen at all)

[mizvekov]

In D129222#3668158, @sammccall wrote:

I think what's going on here: outer cmake invokes this opaque external command (which just _happens_ to be cmake --build) to produce an exe. The outer cmake has no special knowledge of what the inner cmake is going to use as inputs, so i it never invalidates the exe and invokes inner cmake again. Having the native exe depend on the target exe is a hack: it means that it (indirectly) depends on the right set of sources and is invalidated when they change. (In a perfect world we wouldn't build the target clang-pseudo-gen at all)

Yeah I agree, I just meant that build_native_tool (From https://reviews.llvm.org/rG2955192df8ac270515b5fa4aaa9e9380148e7f00) might as well just always add this dependency itself. I think that anything that uses it and forgets to add this dependency would be a bug anyway.

clang-tools-extra/pseudo/gen/Main.cpp

llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Text =
       llvm::MemoryBuffer::getFile(Path);
   if (std::error_code EC = Text.getError()) {
     llvm::errs() << "Error: can't read grammar file '" << Path
                  << "': " << EC.message() << "\n";
     ::exit(1);
   }
   return Text.get()->getBuffer().str();
 }
+
+std::string symbolName(clang::pseudo::SymbolID SID,
+                       const clang::pseudo::Grammar &G) {
+  static const char *const TokNames[] = {
+#define TOK(X) #X,
+#define KEYWORD(X, Y) #X,
+#include "clang/Basic/TokenKinds.def"
+      nullptr};
+  if (clang::pseudo::isToken(SID))
+    return TokNames[clang::pseudo::symbolToToken(SID)];
+  std::string Name = G.symbolName(SID).str();
+  // translation-unit -> translation_unit
+  std::replace(Name.begin(), Name.end(), '-', '_');
+  return Name;
+}
 } // namespace
 
 int main(int argc, char *argv[]) {
   llvm::cl::ParseCommandLineOptions(argc, argv, "");
 
   std::string GrammarText = readOrDie(Grammar);
   std::vector<std::string> Diags;
   auto G = clang::pseudo::Grammar::parseBNF(GrammarText, Diags);
int main(int argc, char *argv[]) {
   }
 
   switch (Emit) {
   case EmitSymbolList:
     Out.os() << R"cpp(
 #ifndef NONTERMINAL
 #define NONTERMINAL(X, Y)
 #endif
+#ifndef RULE
+#define RULE(X, Y)
+#endif
 #ifndef EXTENSION
 #define EXTENSION(X, Y)
 #endif
     )cpp";
     for (clang::pseudo::SymbolID ID = 0; ID < G.table().Nonterminals.size();
-         ++ID) {
-      std::string Name = G.symbolName(ID).str();
-      // translation-unit -> translation_unit
-      std::replace(Name.begin(), Name.end(), '-', '_');
-      Out.os() << llvm::formatv("NONTERMINAL({0}, {1})\n", Name, ID);
+         ++ID)
+      Out.os() << llvm::formatv("NONTERMINAL({0}, {1})\n", symbolName(ID, G),
+                                ID);
+    for (clang::pseudo::RuleID RID = 0; RID < G.table().Rules.size(); ++RID) {
+      const clang::pseudo::Rule &R = G.table().Rules[RID];
+      // lhs$$rhs$rhs$rhs
+      std::string EnumName = symbolName(R.Target, G) + "$";
+      for (clang::pseudo::SymbolID SeqID : R.seq())
+        EnumName += "$" + symbolName(SeqID, G);
+      Out.os() << llvm::formatv("RULE({0}, {1})\n", EnumName, RID);
     }
     for (clang::pseudo::ExtensionID EID = 1 /*skip the sentinel 0 value*/;
          EID < G.table().AttributeValues.size(); ++EID) {
       llvm::StringRef Name = G.table().AttributeValues[EID];
       assert(!Name.empty());
       Out.os() << llvm::formatv("EXTENSION({0}, {1})\n", Name, EID);
     }
     Out.os() << R"cpp(
 #undef NONTERMINAL
+#undef RULE
 #undef EXTENSION
     )cpp";
     break;
   case EmitGrammarContent:
     for (llvm::StringRef Line : llvm::split(GrammarText, '\n')) {
       Out.os() << '"';
       Out.os().write_escaped((Line + "\n").str());
       Out.os() << "\"\n";
     }
     break;
   }
 
   Out.keep();
 
   return 0;
 }

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

 
 // A guard restricts when a grammar rule can be used.
 //
 // The GLR parser will use the guard to determine whether a rule reduction will
 // be conducted. For example, e.g. a guard may allow the rule
 // `virt-specifier := IDENTIFIER` only if the identifier's text is 'override`.
 //
 // Return true if the guard is satisfied.
 using RuleGuard = llvm::function_ref<bool(
-    llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &)>;
+    RuleID RID, llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &)>;
 
 // A recovery strategy determines a region of code to skip when parsing fails.
 //
 // For example, given `class-def := CLASS IDENT { body [recover=Brackets] }`,
 // if parsing fails while attempting to parse `body`, we may skip up to the
 // matching `}` and assume everything between was a `body`.
 //
 // The provided index is the token where the skipped region begins.

clang-tools-extra/pseudo/include/clang-pseudo/cxx/CXX.h

 // Symbol represents nonterminal symbols in the C++ grammar.
 // It provides a simple uniform way to access a particular nonterminal.
 enum class Symbol : SymbolID {
 #define NONTERMINAL(X, Y) X = Y,
 #include "CXXSymbols.inc"
 #undef NONTERMINAL
 };
 
+enum class Rule : RuleID {
+#define RULE(X, Y) X = Y,
+#include "CXXSymbols.inc"
+#undef RULE
+};
+
 enum class Extension : ExtensionID {
 #define EXTENSION(X, Y) X = Y,
 #include "CXXSymbols.inc"
 #undef EXTENSION
 };
 
 // Returns the Language for the cxx.bnf grammar.
 const Language &getLanguage();
 
 } // namespace cxx
 
 } // namespace pseudo
 } // namespace clang
 
 #endif // CLANG_PSEUDO_CXX_CXX_H

clang-tools-extra/pseudo/lib/GLR.cpp

public:
   }
 
 private:
   bool canReduce(ExtensionID GuardID, RuleID RID,
                  llvm::ArrayRef<const ForestNode *> RHS) const {
     if (!GuardID)
       return true;
     if (auto Guard = Lang.Guards.lookup(GuardID))
-      return Guard(RHS, Params.Code);
+      return Guard(RID, RHS, Params.Code);
     LLVM_DEBUG(llvm::dbgs()
                << llvm::formatv("missing guard implementation for rule {0}\n",
                                 Lang.G.dumpRule(RID)));
     return true;
   }
   // pop walks up the parent chain(s) for a reduction from Head by to Rule.
   // Once we reach the end, record the bases and sequences.
   void pop(const GSS::Node *Head, RuleID RID, const Rule &Rule) {

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

 namespace clang {
 namespace pseudo {
 namespace cxx {
 namespace {
 static const char *CXXBNF =
 #include "CXXBNF.inc"
     ;
 
-bool guardOverride(llvm::ArrayRef<const ForestNode *> RHS,
+bool guardOverride(RuleID RID, llvm::ArrayRef<const ForestNode *> RHS,
                    const TokenStream &Tokens) {
   assert(RHS.size() == 1 &&
          RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
   return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "override";
 }
-bool guardFinal(llvm::ArrayRef<const ForestNode *> RHS,
+bool guardFinal(RuleID RID, llvm::ArrayRef<const ForestNode *> RHS,
                 const TokenStream &Tokens) {
   assert(RHS.size() == 1 &&
          RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
   return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "final";
 }
 
+// Determine the declarator-id of the given declarator is a function.
+//
+// E.g.
+//   `a` in `int a`           => false, a is int
+//   `*p` in `int *p`         => false, p is a pointer to int
+//   `*p()` in `int *p()`     => true, p is a function
+//   `(*p)()` in `int (*p)()` => false, p is a pointer to a function
+//
+// FIXME: though the cost of this check is relatively cheap, we should
+// probably consider caching the results to avoid redundant computations.
+bool isFunctionDeclarator(const ForestNode *DeclaratorFN) {
+  assert(DeclaratorFN->symbol() == (SymbolID)(cxx::Symbol::declarator));
+  enum DeclaratorKind {
+    Unknown = 0,
+    Function,
+    Other, // non-function kinds, pointer etc
+  };
+  DeclaratorKind Kind = Unknown;
+  // Declarators nest "inside out", we perform a DFS on the declarator:
+  //  1) descend into the inner-most node (which is a declarator-id)
+  //  2) walkup the stack, find the first rule that can determine the kind
+  auto DFS = [&Kind](const ForestNode *Declarator, auto DFS) {
+    // !! The declarator defined in the cxx.bnf should be unambiguous.
+    // FIXME: should we consider opaque node?
+    assert(Declarator->kind() == ForestNode::Sequence);
+    Rule R = (Rule)Declarator->rule();
+
+    // Reach to the inner-most node of the declarator, walkup now.
+    if (R == Rule::noptr_declarator$$declarator_id)
+      return;
+    // This is the main entry grammar rule for declarator.
+    if (R == Rule::declarator$$ptr_declarator) {
+      DFS(Declarator->elements()[0], DFS);
+      if (Kind == Unknown)
+        Kind = Other; // simple declarator (e.g. `a` in `int a`).
+      return;
+    }
+    // These rules don't contribute the declarator kind.
+    if (R == Rule::ptr_declarator$$noptr_declarator)
+      return DFS(Declarator->elements()[0], DFS);
+    if (R == Rule::noptr_declarator$$l_paren$ptr_declarator$r_paren)
+      return DFS(Declarator->elements()[1], DFS);
+
+    if (R == Rule::ptr_declarator$$ptr_operator$ptr_declarator) {
+      // a pointer declarator p: `int *p`
+      // vs. a function-declarator p: `int *p()`
+      DFS(Declarator->elements()[1], DFS);
+      if (Kind == Unknown)
+        Kind = Other; //  pointer declarator
+      return;
+    }
+    if (R ==
+            Rule::
+                noptr_declarator$$noptr_declarator$l_square$constant_expression$r_square ||
+        R == Rule::noptr_declarator$$noptr_declarator$l_square$r_square) {
+      // `int s[];` vs `int (*s())[];`
+      DFS(Declarator->elements()[0], DFS);
+      if (Kind == Unknown)
+        Kind = Other; // array declarator
+      return;
+    }
+    if (R ==
+            Rule::
+                declarator$$noptr_declarator$parameters_and_qualifiers$trailing_return_type ||
+        R == Rule::
+                 noptr_declarator$$noptr_declarator$parameters_and_qualifiers) {
+      DFS(Declarator->elements()[0], DFS);
+      if (Kind == Unknown)
+        Kind = Function;
+      return;
+    }
+  };
+  DFS(DeclaratorFN, DFS);
+  assert(Kind != Unknown && "The declarator kind is unknown!");
+  return Kind == Function;
+}
+
+bool guardNotFunctionDeclarator(RuleID RID,
+                                llvm::ArrayRef<const ForestNode *> RHS,
+                                const TokenStream &Tokens) {
+  if ((Rule)(RID) == Rule::init_declarator$$declarator$initializer)
+    return !isFunctionDeclarator(RHS[0]);
+  return true;
+}
+
+bool guardFunctionDeclarator(RuleID RID, llvm::ArrayRef<const ForestNode *> RHS,
+                             const TokenStream &Tokens) {
+  auto R = (Rule)(RID);
+  size_t DeclaratorIndex = 0;
+  switch (R) {
+  case Rule::function_definition$$decl_specifier_seq$declarator$function_body:
+  case Rule::
+      function_definition$$decl_specifier_seq$declarator$requires_clause$function_body:
+  case Rule::
+      function_definition$$decl_specifier_seq$declarator$virt_specifier_seq$function_body:
+    DeclaratorIndex = 1;
+    break;
+  case Rule::function_definition$$declarator$function_body:
+  case Rule::function_definition$$declarator$requires_clause$function_body:
+  case Rule::function_definition$$declarator$virt_specifier_seq$function_body:
+    DeclaratorIndex = 0;
+    break;
+  default:
+    llvm_unreachable("Unhandle function-definition rules.");
+  }
+  return isFunctionDeclarator(RHS[DeclaratorIndex]);
+}
+
 llvm::DenseMap<ExtensionID, RuleGuard> buildGuards() {
-  return {
-      {(ExtensionID)Extension::Override, guardOverride},
+  return {{(ExtensionID)Extension::Override, guardOverride},
           {(ExtensionID)Extension::Final, guardFinal},
-  };
+          {(ExtensionID)Extension::FunctionDeclarator, guardFunctionDeclarator},
+          {(ExtensionID)Extension::NotFunctionDeclarator,
+           guardNotFunctionDeclarator}};
 }
 
 Token::Index recoverBrackets(Token::Index Begin, const TokenStream &Tokens) {
   assert(Begin > 0);
   const Token &Left = Tokens.tokens()[Begin - 1];
   assert(Left.Kind == tok::l_brace || Left.Kind == tok::l_paren ||
          Left.Kind == tok::l_square);
   if (const Token *Right = Left.pair()) {

clang-tools-extra/pseudo/lib/cxx/cxx.bnf

 elaborated-type-specifier := class-key nested-name-specifier TEMPLATE_opt simple-template-id
 elaborated-type-specifier := elaborated-enum-specifier
 elaborated-enum-specifier := ENUM nested-name-specifier_opt IDENTIFIER
 decltype-specifier := DECLTYPE ( expression )
 placeholder-type-specifier := type-constraint_opt AUTO
 placeholder-type-specifier := type-constraint_opt DECLTYPE ( AUTO )
 init-declarator-list := init-declarator
 init-declarator-list := init-declarator-list , init-declarator
-init-declarator := declarator initializer_opt
+init-declarator := declarator initializer_opt [guard=NotFunctionDeclarator]
 init-declarator := declarator requires-clause
 declarator := ptr-declarator
 declarator := noptr-declarator parameters-and-qualifiers trailing-return-type
 ptr-declarator := noptr-declarator
 ptr-declarator := ptr-operator ptr-declarator
 noptr-declarator := declarator-id
 noptr-declarator := noptr-declarator parameters-and-qualifiers
 noptr-declarator := noptr-declarator [ constant-expression_opt ]

 initializer-list-item := initializer-clause ..._opt
 initializer-list-item := designator brace-or-equal-initializer ..._opt
 designator := . IDENTIFIER
 #! Array designators are legal in C, and a common extension in C++.
 designator := [ expression ]
 expr-or-braced-init-list := expression
 expr-or-braced-init-list := braced-init-list
 # dcl.fct
-function-definition := decl-specifier-seq_opt declarator virt-specifier-seq_opt function-body
-function-definition := decl-specifier-seq_opt declarator requires-clause function-body
+function-definition := decl-specifier-seq_opt declarator virt-specifier-seq_opt function-body [guard=FunctionDeclarator]
+function-definition := decl-specifier-seq_opt declarator requires-clause function-body [guard=FunctionDeclarator]
 function-body := ctor-initializer_opt compound-statement
 function-body := function-try-block
 function-body := = DEFAULT ;
 function-body := = DELETE ;
 # dcl.enum
 enum-specifier := enum-head { enumerator-list_opt }
 enum-specifier := enum-head { enumerator-list , }
 enum-head := enum-key enum-head-name_opt enum-base_opt

clang-tools-extra/pseudo/test/cxx/declarator-function.cpp

 // The standard grammar allows an init-list with any declarator, including
 // a function declarator. This creates an ambiguity where a function-definition
 // is misparsed as a simple-declaration.
-// FIXME: eliminate this false parse.
-// XFAIL: *
 
 // RUN: clang-pseudo -grammar=cxx -source=%s --print-forest | FileCheck %s
 void s(){};
-// CHECK-NOT:      simple-declaration
-// CHECK:          function-definition := decl-specifier-seq declarator
-// function-body CHECK-NOT:      simple-declaration
+// CHECK:      ├─declaration-seq~function-definition := decl-specifier-seq declarator function-body                 [guard=FunctionDeclarator]
+// CHECK-NEXT: │ ├─decl-specifier-seq~VOID := tok[0]
+// CHECK-NEXT: │ ├─declarator~noptr-declarator := noptr-declarator parameters-and-qualifiers
+// CHECK-NEXT: │ │ ├─noptr-declarator~IDENTIFIER := tok[1]
+// CHECK-NEXT: │ │ └─parameters-and-qualifiers := ( )
+// CHECK-NEXT: │ │   ├─( := tok[2]
+// CHECK-NEXT: │ │   └─) := tok[3]
+// CHECK-NEXT: │ └─function-body~compound-statement := { }
+// CHECK-NEXT: │   ├─{ := tok[4]
+// CHECK-NEXT: │   └─} := tok[5]
+// CHECK-NEXT: └─declaration~; := tok[6]
+
+// Should not parse as a function-definition.
+int s2{};
+// CHECK-NEXT:  declaration~simple-declaration := decl-specifier-seq init-declarator-list ;
+// CHECK-NEXT:  ├─decl-specifier-seq~INT := tok[7]
+// CHECK-NEXT:  ├─init-declarator-list~init-declarator := declarator initializer [guard=NotFunctionDeclarator]
+// CHECK-NEXT:  │ ├─declarator~IDENTIFIER := tok[8]
+// CHECK-NEXT:  │ └─initializer~braced-init-list := { }
+// CHECK-NEXT:  │   ├─{ := tok[9]
+// CHECK-NEXT:  │   └─} := tok[10]
+// CHECK-NEXT:  └─; := tok[11]

clang-tools-extra/pseudo/test/cxx/declarator-var.cpp

 // The standard grammar allows an function-body to use any declarator, including
 // a non-function declarator. This creates an ambiguity where a
 // simple-declaration is misparsed as a function-definition.
-// FIXME: eliminate this false parse.
-// XFAIL: *
 
 // RUN: clang-pseudo -grammar=cxx -source=%s --print-forest | FileCheck %s
 void (*s)(){};
 // CHECK-NOT:      function-definition
 // CHECK:          init-declarator := declarator initializer
 // CHECK-NOT:      function-definition

clang-tools-extra/pseudo/unittests/GLRTest.cpp

 TEST_F(GLRTest, GuardExtension) {
   build(R"bnf(
     _ := start
 
     start := IDENTIFIER [guard=TestOnly]
   )bnf");
   TestLang.Guards.try_emplace(
       extensionID("TestOnly"),
-      [&](llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &Tokens) {
+      [&](RuleID RID, llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &Tokens) {
         assert(RHS.size() == 1 &&
                RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
         return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "test";
       });
   clang::LangOptions LOptions;
   TestLang.Table = LRTable::buildSLR(TestLang.G);
 
   std::string Input = "test";