[pseudo] Key guards by RuleID, add guards to literals (and 0).

[sammccall]

Summary

After this, NUMERIC_CONSTANT and strings should parse only one way.

There are 8 types of literals, and 24 valid (literal, TokenKind) pairs.
This means adding 8 new named guards (or 24, if we want to assert the token).

It seems fairly clear to me at this point that the guard names are unneccesary
indirection: the guards are in fact coupled to the rule signature.

(Also add the zero guard I forgot in the previous patch.)

[sammccall]

reorder a bit

[sammccall]

rebase on function-declarator guards

[hokein]

it looks good.

clang-tools-extra/pseudo/gen/Main.cpp

102 ↗

(On Diff #446061)

I think this invariant is still true even in this patch, ExtensionID for empty attribute value is 0, and we start from 1 in the loop.

clang-tools-extra/pseudo/include/clang-pseudo/grammar/Grammar.h

31

nit: update the stale comment.

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

31–91

nit: not sure the abbreviation of UD is a clearer, took me a while to understand it is for UserDefined, add a comment?

159–169

we might probably to consider to split it out from the CXX.cpp at some point in the future. I think currently it is fine.

174

nit: add a blank line, I think function-declarator guard is different than the contextual-sensitive guard.

clang-tools-extra/pseudo/tool/ClangPseudo.cpp

48

I like this flag -- I have been wanted for several times.

[sammccall]

clang-tools-extra/pseudo/gen/Main.cpp

102 ↗

(On Diff #446061)

Hmm, the assert was firing...

Ah I see what was going on: we were inserting "" at 0 but were inserting it *again* if it was in UniqueAttributeValues, so we had two copies. Fixed.

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

159–169

Yeah, I'd like to keep this all lumped together with no public interfaces for now until the patterns are stable - I wouldn't be surprised if we have to refactor these macros a few more times.

174

Yeah, good call.

[amyk]

Hi!

It appears that this patch is causing a build failure on a couple PPC bots that build with shared libraries:
https://lab.llvm.org/buildbot/#/builders/57/builds/20179
https://lab.llvm.org/buildbot/#/builders/121/builds/21678

The specific error that occurs looks like this:

2.485 [936/22/19] Linking CXX shared library lib/libclangPseudoCXX.so.15git
FAILED: lib/libclangPseudoCXX.so.15git 
: && /home/buildbots/clang.11.0.0/bin/clang++ --gcc-toolchain=/opt/rh/devtoolset-7/root/usr -fPIC -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/./lib  -Wl,--gc-sections -shared -Wl,-soname,libclangPseudoCXX.so.15git -o lib/libclangPseudoCXX.so.15git tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libclangPseudo.so.15git  lib/libclangPseudoGrammar.so.15git  lib/libLLVMSupport.so.15git  -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/lib && :
tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o:(.toc+0x10): undefined reference to `clang::charinfo::InfoTable'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Would you be able to take a look at this issue (or revert the patch if this requires more time to resolve)? Thank you in advance.

[sammccall]

In D130066#3670745, @amyk wrote:

Hi!

It appears that this patch is causing a build failure on a couple PPC bots that build with shared libraries:
https://lab.llvm.org/buildbot/#/builders/57/builds/20179
https://lab.llvm.org/buildbot/#/builders/121/builds/21678

The specific error that occurs looks like this:

2.485 [936/22/19] Linking CXX shared library lib/libclangPseudoCXX.so.15git
FAILED: lib/libclangPseudoCXX.so.15git 
: && /home/buildbots/clang.11.0.0/bin/clang++ --gcc-toolchain=/opt/rh/devtoolset-7/root/usr -fPIC -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/./lib  -Wl,--gc-sections -shared -Wl,-soname,libclangPseudoCXX.so.15git -o lib/libclangPseudoCXX.so.15git tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libclangPseudo.so.15git  lib/libclangPseudoGrammar.so.15git  lib/libLLVMSupport.so.15git  -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/lib && :
tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o:(.toc+0x10): undefined reference to `clang::charinfo::InfoTable'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Would you be able to take a look at this issue (or revert the patch if this requires more time to resolve)? Thank you in advance.

Sorry about that. d26ee284ded30aff46 links in clangBasic for this dep.
(I think it wasn't showing up in the non-shared builds because these functions were always inlined)

[amyk]

In D130066#3670875, @sammccall wrote:

In D130066#3670745, @amyk wrote:

Hi!

It appears that this patch is causing a build failure on a couple PPC bots that build with shared libraries:
https://lab.llvm.org/buildbot/#/builders/57/builds/20179
https://lab.llvm.org/buildbot/#/builders/121/builds/21678

The specific error that occurs looks like this:

2.485 [936/22/19] Linking CXX shared library lib/libclangPseudoCXX.so.15git
FAILED: lib/libclangPseudoCXX.so.15git 
: && /home/buildbots/clang.11.0.0/bin/clang++ --gcc-toolchain=/opt/rh/devtoolset-7/root/usr -fPIC -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/./lib  -Wl,--gc-sections -shared -Wl,-soname,libclangPseudoCXX.so.15git -o lib/libclangPseudoCXX.so.15git tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libclangPseudo.so.15git  lib/libclangPseudoGrammar.so.15git  lib/libLLVMSupport.so.15git  -Wl,-rpath-link,/home/buildbots/docker-RHEL-buildbot/SetupBot/worker_env/ppc64le-clang-rhel-test/clang-ppc64le-rhel/build/lib && :
tools/clang/tools/extra/pseudo/lib/cxx/CMakeFiles/obj.clangPseudoCXX.dir/CXX.cpp.o:(.toc+0x10): undefined reference to `clang::charinfo::InfoTable'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Would you be able to take a look at this issue (or revert the patch if this requires more time to resolve)? Thank you in advance.

Sorry about that. d26ee284ded30aff46 links in clangBasic for this dep.
(I think it wasn't showing up in the non-shared builds because these functions were always inlined)

Thank you for fixing the build failures, @sammccall, I appreciate it! The bots are back to green now. :-)

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

using RecoveryStrategy =
     llvm::function_ref<Token::Index(Token::Index Start, const TokenStream &)>;
 
 // Specify a language that can be parsed by the pseduoparser.
 struct Language {
   Grammar G;
   LRTable Table;
 
   // Binding extension ids to corresponding implementations.
-  llvm::DenseMap<ExtensionID, RuleGuard> Guards;
+  llvm::DenseMap<RuleID, RuleGuard> Guards;
   llvm::DenseMap<ExtensionID, RecoveryStrategy> RecoveryStrategies;
 
   // FIXME: add clang::LangOptions.
   // FIXME: add default start symbols.
 };
 
 } // namespace pseudo
 } // namespace clang
 
 #endif // CLANG_PSEUDO_LANGUAGE_H

clang-tools-extra/pseudo/include/clang-pseudo/grammar/Grammar.h

 //  attributes which are associated with either a grammar symbol (on the
 //  right-hand side of the symbol) or a grammar rule (at the end of the rule
 //  body).
 //  Attributes provide a way to inject custom code into the GLR parser. Each
 //  unique attribute value creates an extension point (identified by ExtensionID
 //  ), and an extension point corresponds to a piece of native code. For
 //  example, C++ grammar has a rule:
 //
-//    contextual-override := IDENTIFIER [guard=Override]
+//   compound_statement := { statement-seq [recover=Brackets] }
 //
-//  GLR parser only conducts the reduction of the rule if the IDENTIFIER
-//  content is `override`. This Override guard is implemented in CXX.cpp by
-//  binding the ExtensionID for the `Override` value to a specific C++ function
-//  that performs the check.
+//  The `recover` attribute instructs the parser that we should perform error
+//  recovery if parsing the statement-seq fails. The `Brackets` recovery
+//  heuristic is implemented in CXX.cpp by binding the ExtensionID for the
+//  `Recovery` value to a specific C++ function that finds the recovery point.
 //
 //  Notions about the BNF grammar:
 //  - "_" is the start symbol of the augmented grammar;
 //  - single-line comment is supported, starting with a #
 //  - A rule describes how a nonterminal (left side of :=) is constructed, and
 //    it is *per line* in the grammar file
 //  - Terminals (also called tokens) correspond to the clang::TokenKind; they
 //    are written in the grammar like "IDENTIFIER", "USING", "+"
static_assert(SizeBits + SymbolBits <= 16,
                 "Must be able to store symbol ID + size efficiently");
 
   // 16 bits for target symbol and size of sequence:
   // SymbolID : 12 | Size : 4
   SymbolID Target : SymbolBits;
   uint8_t Size : SizeBits; // Size of the Sequence
   SymbolID Sequence[MaxElements];
 
-  // A guard extension controls whether a reduction of a rule will be conducted
-  // by the GLR parser.
-  // 0 is sentinel unset extension ID, indicating there is no guard extension
-  // being set for this rule.
-  ExtensionID Guard = 0;
+  // A guarded rule has extra logic to determine whether the RHS is eligible.
+  bool Guarded = false;
 
   // Specifies the index within Sequence eligible for error recovery.
   // Given stmt := { stmt-seq_opt }, if we fail to parse the stmt-seq then we
   // should recover by finding the matching brace, and forcing stmt-seq to match
   // everything between braces.
   // For now, only a single strategy at a single point is possible.
   uint8_t RecoveryIndex = -1;
   ExtensionID Recovery = 0;
 
   llvm::ArrayRef<SymbolID> seq() const {
     return llvm::ArrayRef<SymbolID>(Sequence, Size);
   }
   friend bool operator==(const Rule &L, const Rule &R) {
-    return L.Target == R.Target && L.seq() == R.seq() && L.Guard == R.Guard;
+    return L.Target == R.Target && L.seq() == R.seq() && L.Guarded == R.Guarded;
   }
 };
 
 struct GrammarTable;
 
 // Grammar that describes a programming language, e.g. C++. It represents the
 // contents of the specified grammar.
 // It is a building block for constructing a table-based parser.

clang-tools-extra/pseudo/lib/GLR.cpp

void operator()(std::vector<const GSS::Node *> &Heads, SymbolID Lookahead) {
     popPending();
     while (!Sequences.empty()) {
       pushNext();
       popPending();
     }
   }
 
 private:
-  bool canReduce(ExtensionID GuardID, RuleID RID,
+  bool canReduce(const Rule &R, RuleID RID,
                  llvm::ArrayRef<const ForestNode *> RHS) const {
-    if (!GuardID)
+    if (!R.Guarded)
       return true;
-    if (auto Guard = Lang.Guards.lookup(GuardID))
+    if (auto Guard = Lang.Guards.lookup(RID))
       return Guard(RHS, Params.Code);
     LLVM_DEBUG(llvm::dbgs()
                << llvm::formatv("missing guard implementation for rule {0}\n",
                                 Lang.G.dumpRule(RID)));
     return true;
   }
   // pop walks up the parent chain(s) for a reduction from Head by to Rule.
   // Once we reach the end, record the bases and sequences.
   void pop(const GSS::Node *Head, RuleID RID, const Rule &Rule) {
     LLVM_DEBUG(llvm::dbgs() << "  Pop " << Lang.G.dumpRule(RID) << "\n");
     Family F{/*Start=*/0, /*Symbol=*/Rule.Target, /*Rule=*/RID};
     TempSequence.resize_for_overwrite(Rule.Size);
     auto DFS = [&](const GSS::Node *N, unsigned I, auto &DFS) {
       TempSequence[Rule.Size - 1 - I] = N->Payload;
       if (I + 1 == Rule.Size) {
         F.Start = TempSequence.front()->startTokenIndex();
         LLVM_DEBUG({
           for (const auto *B : N->parents())
             llvm::dbgs() << "    --> base at S" << B->State << "\n";
         });
-        if (!canReduce(Rule.Guard, RID, TempSequence))
+        if (!canReduce(Rule, RID, TempSequence))
           return;
         // Copy the chain to stable storage so it can be enqueued.
         if (SequenceStorageCount == SequenceStorage.size())
           SequenceStorage.emplace_back();
         SequenceStorage[SequenceStorageCount] = TempSequence;
         Sequence *Seq = &SequenceStorage[SequenceStorageCount++];
 
         Sequences.emplace(F, PushSpec{N, Seq});
bool popAndPushTrivial() {
     const GSS::Node *Base = Head;
     TempSequence.resize_for_overwrite(Rule.Size);
     for (unsigned I = 0; I < Rule.Size; ++I) {
       if (Base->parents().size() != 1)
         return false;
       TempSequence[Rule.Size - 1 - I] = Base->Payload;
       Base = Base->parents().front();
     }
-    if (!canReduce(Rule.Guard, *RID, TempSequence))
+    if (!canReduce(Rule, *RID, TempSequence))
       return true; // reduction is not available
     const ForestNode *Parsed =
         &Params.Forest.createSequence(Rule.Target, *RID, TempSequence);
     auto NextState = Lang.Table.getGoToState(Base->State, Rule.Target);
     assert(NextState.has_value() && "goto must succeed after reduce!");
     Heads->push_back(Params.GSStack.addNode(*NextState, Parsed, {Base}));
     return true;
   }

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

 //===--- CXX.cpp - Define public interfaces for C++ grammar ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang-pseudo/cxx/CXX.h"
 #include "clang-pseudo/Forest.h"
 #include "clang-pseudo/Language.h"
 #include "clang-pseudo/grammar/Grammar.h"
 #include "clang-pseudo/grammar/LRTable.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/TokenKinds.h"
+#include "llvm/ADT/StringSwitch.h"
 #include <utility>
 
 namespace clang {
 namespace pseudo {
 namespace cxx {
 namespace {
 static const char *CXXBNF =
 #include "CXXBNF.inc"
     ;
 
-bool guardOverride(llvm::ArrayRef<const ForestNode *> RHS,
-                   const TokenStream &Tokens) {
-  assert(RHS.size() == 1 &&
-         RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
-  return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "override";
-}
-bool guardFinal(llvm::ArrayRef<const ForestNode *> RHS,
-                const TokenStream &Tokens) {
-  assert(RHS.size() == 1 &&
-         RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
-  return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "final";
+// User-defined string literals look like `""suffix`.
+bool isStringUserDefined(const Token &Tok) {
+  return !Tok.text().endswith("\"");
+}
+bool isCharUserDefined(const Token &Tok) { return !Tok.text().endswith("'"); }
+
+// Combinable flags describing numbers.
+// Clang has just one numeric_token kind, the grammar has 4.
+enum NumericKind {
+  Integer = 0,
+  Floating = 1 << 0,
+  UserDefined = 1 << 1,
+};
+// Determine the kind of numeric_constant we have.
+// We can assume it's something valid, as it has been lexed.
+// FIXME: is this expensive enough that we should set flags on the token
+// and reuse them rather than computing it for each guard?
+unsigned numKind(const Token &Tok) {
+  assert(Tok.Kind == tok::numeric_constant);
+  llvm::StringRef Text = Tok.text();
+  if (Text.size() <= 1)
+    return Integer;
+  bool Hex =
+      Text.size() > 2 && Text[0] == '0' && (Text[1] == 'x' || Text[1] == 'X');
+  uint8_t K = Integer;
+
+  for (char C : Text) {
+    switch (C) {
+    case '.':
+      K |= Floating;
+      break;
+    case 'e':
+    case 'E':
+      if (!Hex)
+        K |= Floating;
+      break;
+    case 'p':
+    case 'P':
+      if (Hex)
+        K |= Floating;
+      break;
+    case '_':
+      K |= UserDefined;
+      break;
+    default:
+      break;
     }
-bool guardModule(llvm::ArrayRef<const ForestNode *> RHS,
-                 const TokenStream &Tokens) {
-  return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "module";
   }
-bool guardImport(llvm::ArrayRef<const ForestNode *> RHS,
+
+  // We would be done here, but there are stdlib UDLs that lack _.
+  // We must distinguish these from the builtin suffixes.
+  unsigned LastLetter = Text.size();
+  while (LastLetter > 0 && isLetter(Text[LastLetter - 1]))
+    --LastLetter;
+  if (LastLetter == Text.size()) // Common case
+    return NumericKind(K);
+  // Trailing d/e/f are not part of the suffix in hex numbers.
+  while (Hex && LastLetter < Text.size() && isHexDigit(Text[LastLetter]))
+    ++LastLetter;
+  return llvm::StringSwitch<int, unsigned>(Text.substr(LastLetter))
+      // std::chrono
+      .Cases("h", "min", "s", "ms", "us", "ns", "d", "y", K | UserDefined)
+      // complex
+      .Cases("il", "i", "if", K | UserDefined)
+      .Default(K);
+}
+
+// RHS is expected to contain a single terminal.
+// Returns the corresponding token.
+const Token &onlyToken(tok::TokenKind Kind,
+                       const ArrayRef<const ForestNode *> RHS,
                        const TokenStream &Tokens) {
-  return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "import";
+  assert(RHS.size() == 1 && RHS.front()->symbol() == tokenSymbol(Kind));
+  return Tokens.tokens()[RHS.front()->startTokenIndex()];
 }
-bool guardExport(llvm::ArrayRef<const ForestNode *> RHS,
+// RHS is expected to contain a single symbol.
+// Returns the corresponding ForestNode.
+const ForestNode &onlySymbol(SymbolID Kind,
+                             const ArrayRef<const ForestNode *> RHS,
                              const TokenStream &Tokens) {
-  return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "export";
+  assert(RHS.size() == 1 && RHS.front()->symbol() == Kind);
+  return *RHS.front();
 }
 
 bool isFunctionDeclarator(const ForestNode *Declarator) {
   assert(Declarator->symbol() == (SymbolID)(cxx::Symbol::declarator));
   bool IsFunction = false;
   using cxx::Rule;
   while (true) {
     // not well-formed code, return the best guess.
while (true) {
 
     default:
       assert(false && "unhandled declarator for IsFunction");
       return IsFunction;
     }
   }
   llvm_unreachable("unreachable");
 }
-bool guardFunction(llvm::ArrayRef<const ForestNode *> RHS,
-                   const TokenStream &Tokens) {
-  assert(RHS.size() == 1 &&
-         RHS.front()->symbol() == (SymbolID)(cxx::Symbol::declarator));
-  return isFunctionDeclarator(RHS.front());
-}
-bool guardNonFunction(llvm::ArrayRef<const ForestNode *> RHS,
-                      const TokenStream &Tokens) {
-  assert(RHS.size() == 1 &&
-         RHS.front()->symbol() == (SymbolID)(cxx::Symbol::declarator));
-  return !isFunctionDeclarator(RHS.front());
-}
 
 llvm::DenseMap<ExtensionID, RuleGuard> buildGuards() {
+#define TOKEN_GUARD(kind, cond)                                                \
+  [](llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &Tokens) {      \
+    const Token &Tok = onlyToken(tok::kind, RHS, Tokens);                      \
+    return cond;                                                               \
+  }
+#define SYMBOL_GUARD(kind, cond)                                               \
+  [](llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &Tokens) {      \
+    const ForestNode &N = onlySymbol((SymbolID)Symbol::kind, RHS, Tokens);     \
+    return cond;                                                               \
+  }
   return {
-      {(ExtensionID)Extension::Override, guardOverride},
-      {(ExtensionID)Extension::Final, guardFinal},
-      {(ExtensionID)Extension::Import, guardImport},
-      {(ExtensionID)Extension::Export, guardExport},
-      {(ExtensionID)Extension::Module, guardModule},
-      {(ExtensionID)Extension::FunctionDeclarator, guardFunction},
-      {(ExtensionID)Extension::NonFunctionDeclarator, guardNonFunction},
+      {(RuleID)Rule::function_declarator_0declarator,
+       SYMBOL_GUARD(declarator, isFunctionDeclarator(&N))},
+      {(RuleID)Rule::non_function_declarator_0declarator,
+       SYMBOL_GUARD(declarator, !isFunctionDeclarator(&N))},
+
+      {(RuleID)Rule::contextual_override_0identifier,
+       TOKEN_GUARD(identifier, Tok.text() == "override")},
+      {(RuleID)Rule::contextual_final_0identifier,
+       TOKEN_GUARD(identifier, Tok.text() == "final")},
+      {(RuleID)Rule::import_keyword_0identifier,
+       TOKEN_GUARD(identifier, Tok.text() == "import")},
+      {(RuleID)Rule::export_keyword_0identifier,
+       TOKEN_GUARD(identifier, Tok.text() == "export")},
+      {(RuleID)Rule::module_keyword_0identifier,
+       TOKEN_GUARD(identifier, Tok.text() == "module")},
+      {(RuleID)Rule::contextual_zero_0numeric_constant,
+       TOKEN_GUARD(numeric_constant, Tok.text() == "0")},
+
+      // The grammar distinguishes (only) user-defined vs plain string literals,
+      // where the clang lexer distinguishes (only) encoding types.
+      {(RuleID)Rule::user_defined_string_literal_chunk_0string_literal,
+       TOKEN_GUARD(string_literal, isStringUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_string_literal_chunk_0utf8_string_literal,
+       TOKEN_GUARD(utf8_string_literal, isStringUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_string_literal_chunk_0utf16_string_literal,
+       TOKEN_GUARD(utf16_string_literal, isStringUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_string_literal_chunk_0utf32_string_literal,
+       TOKEN_GUARD(utf32_string_literal, isStringUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_string_literal_chunk_0wide_string_literal,
+       TOKEN_GUARD(wide_string_literal, isStringUserDefined(Tok))},
+      {(RuleID)Rule::string_literal_chunk_0string_literal,
+       TOKEN_GUARD(string_literal, !isStringUserDefined(Tok))},
+      {(RuleID)Rule::string_literal_chunk_0utf8_string_literal,
+       TOKEN_GUARD(utf8_string_literal, !isStringUserDefined(Tok))},
+      {(RuleID)Rule::string_literal_chunk_0utf16_string_literal,
+       TOKEN_GUARD(utf16_string_literal, !isStringUserDefined(Tok))},
+      {(RuleID)Rule::string_literal_chunk_0utf32_string_literal,
+       TOKEN_GUARD(utf32_string_literal, !isStringUserDefined(Tok))},
+      {(RuleID)Rule::string_literal_chunk_0wide_string_literal,
+       TOKEN_GUARD(wide_string_literal, !isStringUserDefined(Tok))},
+      // And the same for chars.
+      {(RuleID)Rule::user_defined_character_literal_0char_constant,
+       TOKEN_GUARD(char_constant, isCharUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_character_literal_0utf8_char_constant,
+       TOKEN_GUARD(utf8_char_constant, isCharUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_character_literal_0utf16_char_constant,
+       TOKEN_GUARD(utf16_char_constant, isCharUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_character_literal_0utf32_char_constant,
+       TOKEN_GUARD(utf32_char_constant, isCharUserDefined(Tok))},
+      {(RuleID)Rule::user_defined_character_literal_0wide_char_constant,
+       TOKEN_GUARD(wide_char_constant, isCharUserDefined(Tok))},
+      {(RuleID)Rule::character_literal_0char_constant,
+       TOKEN_GUARD(char_constant, !isCharUserDefined(Tok))},
+      {(RuleID)Rule::character_literal_0utf8_char_constant,
+       TOKEN_GUARD(utf8_char_constant, !isCharUserDefined(Tok))},
+      {(RuleID)Rule::character_literal_0utf16_char_constant,
+       TOKEN_GUARD(utf16_char_constant, !isCharUserDefined(Tok))},
+      {(RuleID)Rule::character_literal_0utf32_char_constant,
+       TOKEN_GUARD(utf32_char_constant, !isCharUserDefined(Tok))},
+      {(RuleID)Rule::character_literal_0wide_char_constant,
+       TOKEN_GUARD(wide_char_constant, !isCharUserDefined(Tok))},
+      // clang just has one NUMERIC_CONSTANT token for {ud,plain}x{float,int}
+      {(RuleID)Rule::user_defined_integer_literal_0numeric_constant,
+       TOKEN_GUARD(numeric_constant, numKind(Tok) == (Integer | UserDefined))},
+      {(RuleID)Rule::user_defined_floating_point_literal_0numeric_constant,
+       TOKEN_GUARD(numeric_constant, numKind(Tok) == (Floating | UserDefined))},
+      {(RuleID)Rule::integer_literal_0numeric_constant,
+       TOKEN_GUARD(numeric_constant, numKind(Tok) == Integer)},
+      {(RuleID)Rule::floating_point_literal_0numeric_constant,
+       TOKEN_GUARD(numeric_constant, numKind(Tok) == Floating)},
   };
+#undef TOKEN_GUARD
+#undef SYMBOL_GUARD
 }
 
 Token::Index recoverBrackets(Token::Index Begin, const TokenStream &Tokens) {
   assert(Begin > 0);
   const Token &Left = Tokens.tokens()[Begin - 1];
   assert(Left.Kind == tok::l_brace || Left.Kind == tok::l_paren ||
          Left.Kind == tok::l_square);
   if (const Token *Right = Left.pair()) {

clang-tools-extra/pseudo/lib/cxx/cxx.bnf

 #!      declaration;
 #!   2) an function-body with any declarator, includeing a non-function
 #!      declarator, this creates an ambiguity whwere a simple-declaration is
 #!      misparsed as a function-definition;
 #! We extend the standard declarator to function-declarator and non-function-declarator
 #! to eliminate these false parses.
 init-declarator := non-function-declarator initializer_opt
 init-declarator := function-declarator requires-clause_opt
-function-declarator := declarator [guard=FunctionDeclarator]
-non-function-declarator := declarator [guard=NonFunctionDeclarator]
+function-declarator := declarator [guard]
+non-function-declarator := declarator [guard]
 declarator := ptr-declarator
 declarator := noptr-declarator parameters-and-qualifiers trailing-return-type
 ptr-declarator := noptr-declarator
 ptr-declarator := ptr-operator ptr-declarator
 noptr-declarator := declarator-id
 noptr-declarator := noptr-declarator parameters-and-qualifiers
 noptr-declarator := noptr-declarator [ constant-expression_opt ]
 noptr-declarator := ( ptr-declarator )

 #! needed literals.
 literal := integer-literal
 literal := character-literal
 literal := floating-point-literal
 literal := string-literal
 literal := boolean-literal
 literal := pointer-literal
 literal := user-defined-literal
-integer-literal := NUMERIC_CONSTANT
-character-literal := CHAR_CONSTANT
-character-literal := WIDE_CHAR_CONSTANT
-character-literal := UTF8_CHAR_CONSTANT
-character-literal := UTF16_CHAR_CONSTANT
-character-literal := UTF32_CHAR_CONSTANT
-floating-point-literal := NUMERIC_CONSTANT
-string-literal-chunk := STRING_LITERAL
-string-literal-chunk := WIDE_STRING_LITERAL
-string-literal-chunk := UTF8_STRING_LITERAL
-string-literal-chunk := UTF16_STRING_LITERAL
-string-literal-chunk := UTF32_STRING_LITERAL
+integer-literal := NUMERIC_CONSTANT [guard]
+character-literal := CHAR_CONSTANT [guard]
+character-literal := WIDE_CHAR_CONSTANT [guard]
+character-literal := UTF8_CHAR_CONSTANT [guard]
+character-literal := UTF16_CHAR_CONSTANT [guard]
+character-literal := UTF32_CHAR_CONSTANT [guard]
+floating-point-literal := NUMERIC_CONSTANT [guard]
+string-literal-chunk := STRING_LITERAL [guard]
+string-literal-chunk := WIDE_STRING_LITERAL [guard]
+string-literal-chunk := UTF8_STRING_LITERAL [guard]
+string-literal-chunk := UTF16_STRING_LITERAL [guard]
+string-literal-chunk := UTF32_STRING_LITERAL [guard]
 #! Technically, string concatenation happens at phase 6 which is before parsing,
 #! so it doesn't belong to the grammar. However, we extend the grammar to
 #! support it, to make the pseudoparser fully functional on practical code.
 string-literal := string-literal-chunk
 string-literal := string-literal string-literal-chunk
 user-defined-literal := user-defined-integer-literal
 user-defined-literal := user-defined-floating-point-literal
 user-defined-literal := user-defined-string-literal
 user-defined-literal := user-defined-character-literal
-user-defined-integer-literal := NUMERIC_CONSTANT
-user-defined-string-literal-chunk := STRING_LITERAL
-user-defined-string-literal-chunk := WIDE_STRING_LITERAL
-user-defined-string-literal-chunk := UTF8_STRING_LITERAL
-user-defined-string-literal-chunk := UTF16_STRING_LITERAL
-user-defined-string-literal-chunk := UTF32_STRING_LITERAL
+user-defined-integer-literal := NUMERIC_CONSTANT [guard]
+user-defined-string-literal-chunk := STRING_LITERAL [guard]
+user-defined-string-literal-chunk := WIDE_STRING_LITERAL [guard]
+user-defined-string-literal-chunk := UTF8_STRING_LITERAL [guard]
+user-defined-string-literal-chunk := UTF16_STRING_LITERAL [guard]
+user-defined-string-literal-chunk := UTF32_STRING_LITERAL [guard]
 user-defined-string-literal := user-defined-string-literal-chunk
 user-defined-string-literal := string-literal-chunk user-defined-string-literal
 user-defined-string-literal := user-defined-string-literal string-literal-chunk
-user-defined-floating-point-literal := NUMERIC_CONSTANT
-user-defined-character-literal := CHAR_CONSTANT
-user-defined-character-literal := WIDE_CHAR_CONSTANT
-user-defined-character-literal := UTF8_CHAR_CONSTANT
-user-defined-character-literal := UTF16_CHAR_CONSTANT
-user-defined-character-literal := UTF32_CHAR_CONSTANT
+user-defined-floating-point-literal := NUMERIC_CONSTANT [guard]
+user-defined-character-literal := CHAR_CONSTANT [guard]
+user-defined-character-literal := WIDE_CHAR_CONSTANT [guard]
+user-defined-character-literal := UTF8_CHAR_CONSTANT [guard]
+user-defined-character-literal := UTF16_CHAR_CONSTANT [guard]
+user-defined-character-literal := UTF32_CHAR_CONSTANT [guard]
 boolean-literal := FALSE
 boolean-literal := TRUE
 pointer-literal := NULLPTR
 
 #! Contextual keywords -- clang lexer always lexes them as identifier tokens.
 #! Placeholders for literal text in the grammar that lex as other things.
-contextual-override := IDENTIFIER [guard=Override]
-contextual-final := IDENTIFIER [guard=Final]
-contextual-zero := NUMERIC_CONSTANT [guard=Zero]
-module-keyword := IDENTIFIER [guard=Module]
-import-keyword := IDENTIFIER [guard=Import]
-export-keyword := IDENTIFIER [guard=Export]
+contextual-override := IDENTIFIER [guard]
+contextual-final := IDENTIFIER [guard]
+contextual-zero := NUMERIC_CONSTANT [guard]
+module-keyword := IDENTIFIER [guard]
+import-keyword := IDENTIFIER [guard]
+export-keyword := IDENTIFIER [guard]
 
 #! greatergreater token -- clang lexer always lexes it as a single token, we
 #! split it into two tokens to make the GLR parser aware of the nested-template
 #! case.
 greatergreater := > >
 
 #! C++ predefined identifier, __func__ [dcl.fct.def.general] p8
 #! FIXME: add other (MSVC, GNU extension) predefined identifiers.
 primary-expression := predefined-expression
 predefined-expression := __FUNC__

clang-tools-extra/pseudo/lib/grammar/Grammar.cpp

std::string Grammar::dumpRule(RuleID RID) const {
   llvm::raw_string_ostream OS(Result);
   const Rule &R = T->Rules[RID];
   OS << symbolName(R.Target) << " :=";
   for (unsigned I = 0; I < R.Size; ++I) {
     OS << " " << symbolName(R.Sequence[I]);
     if (R.RecoveryIndex == I)
       OS << " [recover=" << T->AttributeValues[R.Recovery] << "]";
   }
-  if (R.Guard)
-    OS << " [guard=" << T->AttributeValues[R.Guard] << "]";
+  if (R.Guarded)
+    OS << " [guard]";
   return Result;
 }
 
 std::string Grammar::dumpRules(SymbolID SID) const {
   assert(isNonterminal(SID));
   std::string Result;
   const auto &Range = T->Nonterminals[SID].RuleRange;
   for (RuleID RID = Range.Start; RID < Range.End; ++RID)

clang-tools-extra/pseudo/lib/grammar/GrammarBNF.cpp

Grammar build(llvm::StringRef BNF) {
     assert(T->Nonterminals.size() < (1 << (SymbolBits - 1)) &&
            "Too many nonterminals to fit in SymbolID bits!");
     llvm::sort(T->Nonterminals, [](const GrammarTable::Nonterminal &L,
                                    const GrammarTable::Nonterminal &R) {
       return L.Name < R.Name;
     });
     // Add an empty string for the corresponding sentinel unset attribute.
     T->AttributeValues.push_back("");
+    UniqueAttributeValues.erase("");
     llvm::for_each(UniqueAttributeValues, [&T](llvm::StringRef Name) {
       T->AttributeValues.emplace_back();
       T->AttributeValues.back() = Name.str();
     });
     llvm::sort(T->AttributeValues);
     assert(T->AttributeValues.front() == "");
 
     // Build name -> ID maps for nonterminals.
auto LookupExtensionID = [&T](llvm::StringRef Name) {
           T.AttributeValues, [&](llvm::StringRef X) { return X < Name; });
       assert(It != T.AttributeValues.end() && *It == Name &&
              "Didn't find the attribute in AttrValues!");
       return It - T.AttributeValues.begin();
     };
     for (unsigned I = 0; I < Spec.Sequence.size(); ++I) {
       for (const auto &KV : Spec.Sequence[I].Attributes) {
         if (KV.first == "guard") {
-          R.Guard = LookupExtensionID(KV.second);
+          R.Guarded = true;
         } else if (KV.first == "recover") {
           R.Recovery = LookupExtensionID(KV.second);
           R.RecoveryIndex = I;
         } else {
           Diagnostics.push_back(
               llvm::formatv("Unknown attribute '{0}'", KV.first).str());
         }
       }

clang-tools-extra/pseudo/test/cxx/literals.cpp

+// RUN: clang-pseudo -grammar=cxx -source=%s --print-forest -forest-abbrev=0 | FileCheck %s --implicit-check-not=ambiguous
+auto list = {
+  0,      // CHECK: := integer-literal
+  0b1011, // CHECK: := integer-literal
+  0777,   // CHECK: := integer-literal
+  42_u,   // CHECK: := user-defined-integer-literal
+  0LL,    // CHECK: := integer-literal
+  0h,     // CHECK: := user-defined-integer-literal
+  0.,     // CHECK: := floating-point-literal
+  .2,     // CHECK: := floating-point-literal
+  2e1,    // CHECK: := floating-point-literal
+  0x42d,  // CHECK: := integer-literal
+  0x42_d, // CHECK: := user-defined-integer-literal
+  0x42ds, // CHECK: := user-defined-integer-literal
+  0x1.2p2,// CHECK: := floating-point-literal
+  
+  "",               // CHECK: literal := string-literal
+  L"",              // CHECK: literal := string-literal
+  u8"",             // CHECK: literal := string-literal
+  u"",              // CHECK: literal := string-literal
+  U"",              // CHECK: literal := string-literal
+  R"()",            // CHECK: literal := string-literal
+  uR"()",           // CHECK: literal := string-literal
+  "a" "b",          // CHECK: literal := string-literal
+  u8"a" "b",        // CHECK: literal := string-literal
+  u"a" u"b",        // CHECK: literal := string-literal
+  "a"_u "b",        // CHECK: user-defined-literal := user-defined-string-literal
+  "a"_u u"b",       // CHECK: user-defined-literal := user-defined-string-literal
+  R"(a)" "\n",      // CHECK: literal := string-literal
+  R"c(a)c"_u u"\n", // CHECK: user-defined-literal := user-defined-string-literal
+
+  'a',      // CHECK: := character-literal
+  'abc',    // CHECK: := character-literal
+  'abcdef', // CHECK: := character-literal
+  u'a',     // CHECK: := character-literal
+  U'a',     // CHECK: := character-literal
+  L'a',     // CHECK: := character-literal
+  L'abc',   // CHECK: := character-literal
+  U'\u1234',// CHECK: := character-literal
+  '\u1234', // CHECK: := character-literal
+  u'a'_u,   // CHECK: := user-defined-character-literal
+};
+

clang-tools-extra/pseudo/test/cxx/mixed-designator.cpp

 // RUN: clang-pseudo -grammar=cxx -source=%s --print-forest | FileCheck %s
 // FIXME: tighten CHECK to CHECK-NEXT once numeric literals are unambiguous.
 auto x = { 1, .f = 2, [c]{3} };
 // CHECK:      initializer-clause~braced-init-list
 // CHECK-NEXT: ├─{ := tok[3]
 // CHECK-NEXT: ├─initializer-list
 // CHECK-NEXT: │ ├─initializer-list
-// CHECK-NEXT: │ │ ├─initializer-list~literal
-// CHECK:      │ │ ├─, := tok[5]
+// CHECK-NEXT: │ │ ├─initializer-list~NUMERIC_CONSTANT
+// CHECK-NEXT: │ │ ├─, := tok[5]
 // CHECK-NEXT: │ │ └─initializer-list-item
 // CHECK-NEXT: │ │   ├─designator
 // CHECK-NEXT: │ │   │ ├─. := tok[6]
 // CHECK-NEXT: │ │   │ └─IDENTIFIER := tok[7]
 // CHECK-NEXT: │ │   └─brace-or-equal-initializer
 // CHECK-NEXT: │ │     ├─= := tok[8]
-// CHECK-NEXT: │ │     └─initializer-clause~literal
-// CHECK:      │ ├─, := tok[10]
+// CHECK-NEXT: │ │     └─initializer-clause~NUMERIC_CONSTANT
+// CHECK-NEXT: │ ├─, := tok[10]
 // CHECK-NEXT: │ └─initializer-list-item
 // CHECK-NEXT: │   ├─designator
 // CHECK-NEXT: │   │ ├─[ := tok[11]
 // CHECK-NEXT: │   │ ├─expression~IDENTIFIER := tok[12]
 // CHECK-NEXT: │   │ └─] := tok[13]
 // CHECK-NEXT: │   └─brace-or-equal-initializer~braced-init-list
 // CHECK-NEXT: │     ├─{ := tok[14]
-// CHECK-NEXT: │     ├─initializer-list~literal
+// CHECK-NEXT: │     ├─initializer-list~NUMERIC_CONSTANT
 // CHECK:      │     └─} := tok[16]
 // CHECK-NEXT: └─} := tok[17]

clang-tools-extra/pseudo/tool/ClangPseudo.cpp

 static opt<bool>
     PrintDirectiveTree("print-directive-tree",
                       desc("Print directive structure of source code"));
 static opt<bool>
     StripDirectives("strip-directives",
                     desc("Strip directives and select conditional sections"));
 static opt<bool> PrintStatistics("print-statistics", desc("Print GLR parser statistics"));
 static opt<bool> PrintForest("print-forest", desc("Print parse forest"));
+static opt<bool> ForestAbbrev("forest-abbrev", desc("Abbreviate parse forest"),
+                              init(true));
 static opt<std::string> HTMLForest("html-forest",
                                    desc("output file for HTML forest"));
 static opt<std::string> StartSymbol("start-symbol",
                                     desc("specify the start symbol to parse"),
                                     init("translation-unit"));
 
 static std::string readOrDie(llvm::StringRef Path) {
   llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Text =
if (!StartSymID) {
       llvm::errs() << llvm::formatv(
           "The start symbol {0} doesn't exit in the grammar!\n", StartSymbol);
       return 2;
     }
     auto &Root =
         glrParse(clang::pseudo::ParseParams{*ParseableStream, Arena, GSS},
                  *StartSymID, Lang);
     if (PrintForest)
-      llvm::outs() << Root.dumpRecursive(Lang.G, /*Abbreviated=*/true);
+      llvm::outs() << Root.dumpRecursive(Lang.G, /*Abbreviated=*/ForestAbbrev);
 
     if (HTMLForest.getNumOccurrences()) {
       std::error_code EC;
       llvm::raw_fd_ostream HTMLOut(HTMLForest, EC);
       if (EC) {
         llvm::errs() << "Couldn't write " << HTMLForest << ": " << EC.message()
                      << "\n";
         return 2;

clang-tools-extra/pseudo/unittests/GLRTest.cpp

EXPECT_EQ(Parsed.dumpRecursive(TestLang.G),
             "[  1, end) └─test := IDENTIFIER\n"
             "[  1, end)   └─IDENTIFIER := tok[1]\n");
 }
 
 TEST_F(GLRTest, GuardExtension) {
   build(R"bnf(
     _ := start
 
-    start := IDENTIFIER [guard=TestOnly]
+    start := IDENTIFIER [guard]
   )bnf");
   TestLang.Guards.try_emplace(
-      extensionID("TestOnly"),
+      ruleFor("start"),
       [&](llvm::ArrayRef<const ForestNode *> RHS, const TokenStream &Tokens) {
         assert(RHS.size() == 1 &&
                RHS.front()->symbol() == tokenSymbol(clang::tok::identifier));
         return Tokens.tokens()[RHS.front()->startTokenIndex()].text() == "test";
       });
   clang::LangOptions LOptions;
   TestLang.Table = LRTable::buildSLR(TestLang.G);
 
   std::string Input = "test";
   const TokenStream &Succeeded = cook(lex(Input, LOptions), LOptions);
   EXPECT_EQ(glrParse({Succeeded, Arena, GSStack}, id("start"), TestLang)
                 .dumpRecursive(TestLang.G),
-            "[  0, end) start := IDENTIFIER [guard=TestOnly]\n"
+            "[  0, end) start := IDENTIFIER [guard]\n"
             "[  0, end) └─IDENTIFIER := tok[0]\n");
 
   Input = "notest";
   const TokenStream &Failed = cook(lex(Input, LOptions), LOptions);
   EXPECT_EQ(glrParse({Failed, Arena, GSStack}, id("start"), TestLang)
                 .dumpRecursive(TestLang.G),
             "[  0, end) start := <opaque>\n");
 }

clang-tools-extra/pseudo/unittests/GrammarTest.cpp

TEST_F(GrammarTest, RuleIDSorted) {
   EXPECT_LT(ruleFor("z"), ruleFor("y"));
   EXPECT_LT(ruleFor("y"), ruleFor("x"));
   EXPECT_LT(ruleFor("x"), ruleFor("_"));
 }
 
 TEST_F(GrammarTest, Annotation) {
   build(R"bnf(
     _ := x
-
-    x := y [guard=value]
-    y := IDENTIFIER [guard=final]
-
+    x := IDENTIFIER [guard]
   )bnf");
-  ASSERT_TRUE(Diags.empty());
-  EXPECT_EQ(G.lookupRule(ruleFor("_")).Guard, 0);
-  EXPECT_GT(G.lookupRule(ruleFor("x")).Guard, 0);
-  EXPECT_GT(G.lookupRule(ruleFor("y")).Guard, 0);
-  EXPECT_NE(G.lookupRule(ruleFor("x")).Guard, G.lookupRule(ruleFor("y")).Guard);
+  ASSERT_THAT(Diags, IsEmpty());
+  EXPECT_FALSE(G.lookupRule(ruleFor("_")).Guarded);
+  EXPECT_TRUE(G.lookupRule(ruleFor("x")).Guarded);
 }
 
 TEST_F(GrammarTest, MangleName) {
   build(R"bnf(
     _ := declaration
 
     declaration := ptr-declarator ;
     ptr-declarator := * IDENTIFIER