[pseudo] Define a clangPseudoCLI library.

[hokein]

Summary

• define a common data structure ParseLang which is a compiled result of the bnf grammar. It is defined in ParseLang.h;
• creates a clangPseudoCLI lib which defines a grammar commandline flag and expose a function to get the ParseLang. It supports --grammar=cxx, --grammmar=/path/to/file.bnf;
• Use the clangPseudoCLI in clang-pseudo, fuzzer, and benchmark tools (simplify the code and use the prebuilt cxx grammar);

Split out from https://reviews.llvm.org/D127448 with comments addressed

[hokein]

remove a dependency from pseudoCLI lib

[sammccall]

Thanks, the split here looks good!

clang-tools-extra/pseudo/include/clang-pseudo/GLR.h

115

You're already touching the callsites of the glr* functions, it might be worth just doing this already... up to you

clang-tools-extra/pseudo/include/clang-pseudo/ParseLang.h

2

The "ParseLang" name doesn't feel right to me :-(

I think it's a combination of:

• Lang is unneccesarily abbreviated
• "Parse" doesn't actually disambiguate much, as "parse" is the whole project

Do you think clang::pseudo::Language would work?

(Sorry for not realizing this on the previous pass, I know it's a pain... happy to chat more offline)

20

I don't know what this sentence means - is it always true? is it necessary?

25

is this "fix right after this patch" or "fix sometime, maybe" or something in between?

(I think these make a lot of sense, and am worried the structure will be incoherent if we don't actually add them soon)

clang-tools-extra/pseudo/include/clang-pseudo/cli/CLI.h

10

The first sentence should say what it is, rather than what calls it.

If the file by design defines a single thing function I'm not sure a file comment distinct from the function comment helps much, maybe merge them?

28

this should generally be called exactly one from CLI tools - why does it need to be memoized and returned as a reference?

clang-tools-extra/pseudo/lib/cli/CLI.cpp

39

this is extremely surprising.
At minimum it needs to be very clearly documented, but I think it's probably better to return null to the caller

43

this if() isn't needed unless you want to print a header to provide some context (which might be a good idea)

[hokein]

address comments

[hokein]

clang-tools-extra/pseudo/include/clang-pseudo/GLR.h

115

these callsites are trivial to update. I'd prefer to do it in a separate patch.

clang-tools-extra/pseudo/include/clang-pseudo/ParseLang.h

2

That sounds good to me. Regarding the location of this file, I think the subdir grammar will be a better fit.

20

I think it is always true for the grammar and LRTable object, but might not be true for the LangOptions. Removed.

25

I would expect we will add them soon.

clang-tools-extra/pseudo/include/clang-pseudo/cli/CLI.h

10

I'm not sure whether we will add more functions in this library (though I can't come up with one), I will just keep the file comment.

28

yeah, for now, returning an object is ok, but in the future, when we build the ParseLang (LRTable, LRGrammar) for cxx.bnf in a fully-static way, we might have trouble right?

clang-tools-extra/pseudo/lib/cli/CLI.cpp

39

It doesn't seem to be surprising to me, as this function is used for the CLI tools, crashing the program when the input grammar text is invalid seems reasonable (all our exiting CLI tools at the moment behave like this), and we have print the error message.

Returning a nullptr seems annoying IMO -- it requires all callers to handle this unimportant case. Updated the document for this function.

43

I don't get your point of the comment. Not sure what you meant by Print a header?

I think for the CLI tool use-case, printing the diagnostic of the grammar by default is reasonable.

[sammccall]

clang-tools-extra/pseudo/benchmarks/Benchmark.cpp

51

nit: still PLang here and in a bunch of places

clang-tools-extra/pseudo/include/clang-pseudo/ParseLang.h

2

The main purpose of the grammar library is to minimize the amount of stuff we pull into the gen step right?

I'm a bit concerned about mixing clang::LangOptions in there unneccesarily - if grammar doesn't *need* the header, maybe it's OK where it is?

clang-tools-extra/pseudo/include/clang-pseudo/cli/CLI.h

10

nit: "this file implements a CLI library which provides an interface for" is boilerplate.
Just "Provides the Grammar, LRTable etc for a language specified by flags"?

clang-tools-extra/pseudo/lib/cli/CLI.cpp

43

You could replace

if (!Diags.empty())
 for(D : Diags)
    ...

with just:

for (D : Diags)
  ...

unless you would prefer:

if (!Diags.empty()) {
  errs() << "Problems with the grammar:\n";
  for (D : Diags)
    ...
}

(Yesterday I thought the last one would be clearer, today I'm not so sure)

[hokein]

address review comments

clang-tools-extra/pseudo/benchmarks/Benchmark.cpp

51

renamed all to Lang

clang-tools-extra/pseudo/include/clang-pseudo/ParseLang.h

2

as discussed offline, moved back to pseudo/

clang-tools-extra/pseudo/lib/cli/CLI.cpp

43

oops, I see what you mean now.

[sammccall]

Looks good!

clang-tools-extra/pseudo/include/clang-pseudo/Language.h

21 ↗

(On Diff #440929)

LR might be more distinguishing (and shorter!), up to you

clang-tools-extra/pseudo/benchmarks/Benchmark.cpp

 //===----------------------------------------------------------------------===//
 
 #include "benchmark/benchmark.h"
 #include "clang-pseudo/Bracket.h"
 #include "clang-pseudo/DirectiveTree.h"
 #include "clang-pseudo/Forest.h"
 #include "clang-pseudo/GLR.h"
 #include "clang-pseudo/Token.h"
+#include "clang-pseudo/cli/CLI.h"
 #include "clang-pseudo/grammar/Grammar.h"
 #include "clang-pseudo/grammar/LRTable.h"
 #include "clang/Basic/LangOptions.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <string>
 
 using llvm::cl::desc;
 using llvm::cl::opt;
 using llvm::cl::Required;
 
-static opt<std::string> GrammarFile("grammar",
-                                    desc("Parse and check a BNF grammar file."),
-                                    Required);
 static opt<std::string> Source("source", desc("Source file"), Required);
 
 namespace clang {
 namespace pseudo {
 namespace bench {
 namespace {
 
-const std::string *GrammarText = nullptr;
 const std::string *SourceText = nullptr;
-const Grammar *G = nullptr;
+const ParseLang *PLang = nullptr;
 
-void setupGrammarAndSource() {
+void setup() {
   auto ReadFile = [](llvm::StringRef FilePath) -> std::string {
     llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> GrammarText =
         llvm::MemoryBuffer::getFile(FilePath);
     if (std::error_code EC = GrammarText.getError()) {
       llvm::errs() << "Error: can't read file '" << FilePath
                    << "': " << EC.message() << "\n";
       std::exit(1);
     }
     return GrammarText.get()->getBuffer().str();
   };
-  GrammarText = new std::string(ReadFile(GrammarFile));
   SourceText = new std::string(ReadFile(Source));
-  std::vector<std::string> Diags;
-  G = Grammar::parseBNF(*GrammarText, Diags).release();
+  PLang = &getParseLangFromFlags();
 }
 
-static void parseBNF(benchmark::State &State) {
-  std::vector<std::string> Diags;
-  for (auto _ : State)
-    Grammar::parseBNF(*GrammarText, Diags);
-}
-BENCHMARK(parseBNF);
-
 static void buildSLR(benchmark::State &State) {
   for (auto _ : State)
-    LRTable::buildSLR(*G);
+    LRTable::buildSLR(*PLang->G);
 }
 BENCHMARK(buildSLR);
 
 TokenStream lexAndPreprocess() {
   clang::LangOptions LangOpts = genericLangOpts();
   TokenStream RawStream = pseudo::lex(*SourceText, LangOpts);
   auto DirectiveStructure = DirectiveTree::parse(RawStream);
   chooseConditionalBranches(DirectiveStructure, RawStream);
stripComments(
         cook(DirectiveStructure.stripDirectives(RawStream), LangOpts));
   }
   State.SetBytesProcessed(static_cast<uint64_t>(State.iterations()) *
                           SourceText->size());
 }
 BENCHMARK(preprocess);
 
 static void glrParse(benchmark::State &State) {
-  LRTable Table = clang::pseudo::LRTable::buildSLR(*G);
-  SymbolID StartSymbol = *G->findNonterminal("translation-unit");
+  SymbolID StartSymbol = *PLang->G->findNonterminal("translation-unit");
   TokenStream Stream = lexAndPreprocess();
   for (auto _ : State) {
     pseudo::ForestArena Forest;
     pseudo::GSS GSS;
-    pseudo::glrParse(Stream, ParseParams{*G, Table, Forest, GSS}, StartSymbol);
+    pseudo::glrParse(Stream, ParseParams{*PLang->G, PLang->Table, Forest, GSS},
+                     StartSymbol);
   }
   State.SetBytesProcessed(static_cast<uint64_t>(State.iterations()) *
                           SourceText->size());
 }
 BENCHMARK(glrParse);
 
 static void full(benchmark::State &State) {
-  LRTable Table = clang::pseudo::LRTable::buildSLR(*G);
-  SymbolID StartSymbol = *G->findNonterminal("translation-unit");
+  SymbolID StartSymbol = *PLang->G->findNonterminal("translation-unit");
   for (auto _ : State) {
     TokenStream Stream = lexAndPreprocess();
     pseudo::ForestArena Forest;
     pseudo::GSS GSS;
-    pseudo::glrParse(lexAndPreprocess(), ParseParams{*G, Table, Forest, GSS},
+    pseudo::glrParse(lexAndPreprocess(),
+                     ParseParams{*PLang->G, PLang->Table, Forest, GSS},
                      StartSymbol);
   }
   State.SetBytesProcessed(static_cast<uint64_t>(State.iterations()) *
                           SourceText->size());
 }
 BENCHMARK(full);
 
 } // namespace
 } // namespace bench
 } // namespace pseudo
 } // namespace clang
 
 int main(int argc, char *argv[]) {
   benchmark::Initialize(&argc, argv);
   llvm::cl::ParseCommandLineOptions(argc, argv);
-  clang::pseudo::bench::setupGrammarAndSource();
+  clang::pseudo::bench::setup();
   benchmark::RunSpecifiedBenchmarks();
   return 0;
 }

clang-tools-extra/pseudo/benchmarks/CMakeLists.txt

 add_benchmark(ClangPseudoBenchmark Benchmark.cpp)
 
 target_link_libraries(ClangPseudoBenchmark
   PRIVATE
   clangPseudo
+  clangPseudoCLI
   clangPseudoGrammar
   LLVMSupport
   )

clang-tools-extra/pseudo/fuzzer/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   FuzzerCLI
   Support
   )
 
 add_llvm_fuzzer(clang-pseudo-fuzzer
   Fuzzer.cpp
   DUMMY_MAIN Main.cpp
   )
 
 target_link_libraries(clang-pseudo-fuzzer
   PRIVATE
   clangPseudo
+  clangPseudoCLI
   clangPseudoGrammar
   )

clang-tools-extra/pseudo/fuzzer/Fuzzer.cpp

 //===-- Fuzzer.cpp - Fuzz the pseudoparser --------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang-pseudo/DirectiveTree.h"
 #include "clang-pseudo/Forest.h"
 #include "clang-pseudo/GLR.h"
 #include "clang-pseudo/Token.h"
+#include "clang-pseudo/cli/CLI.h"
 #include "clang-pseudo/grammar/Grammar.h"
 #include "clang-pseudo/grammar/LRTable.h"
 #include "clang/Basic/LangOptions.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 
 namespace clang {
 namespace pseudo {
 namespace {
 
 class Fuzzer {
   clang::LangOptions LangOpts = clang::pseudo::genericLangOpts();
-  std::unique_ptr<Grammar> G;
-  LRTable T;
   bool Print;
 
 public:
-  Fuzzer(llvm::StringRef GrammarPath, bool Print) : Print(Print) {
-    llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> GrammarText =
-        llvm::MemoryBuffer::getFile(GrammarPath);
-    if (std::error_code EC = GrammarText.getError()) {
-      llvm::errs() << "Error: can't read grammar file '" << GrammarPath
-                   << "': " << EC.message() << "\n";
-      std::exit(1);
-    }
-    std::vector<std::string> Diags;
-    G = Grammar::parseBNF(GrammarText->get()->getBuffer(), Diags);
-    if (!Diags.empty()) {
-      for (const auto &Diag : Diags)
-        llvm::errs() << Diag << "\n";
-      std::exit(1);
-    }
-    T = LRTable::buildSLR(*G);
-  }
+  Fuzzer(bool Print) : Print(Print) {}
 
   void operator()(llvm::StringRef Code) {
     std::string CodeStr = Code.str(); // Must be null-terminated.
     auto RawStream = lex(CodeStr, LangOpts);
     auto DirectiveStructure = DirectiveTree::parse(RawStream);
     clang::pseudo::chooseConditionalBranches(DirectiveStructure, RawStream);
     // FIXME: strip preprocessor directives
     auto ParseableStream =
         clang::pseudo::stripComments(cook(RawStream, LangOpts));
 
     clang::pseudo::ForestArena Arena;
     clang::pseudo::GSS GSS;
+    const ParseLang &PLang = getParseLangFromFlags();
     auto &Root =
-        glrParse(ParseableStream, clang::pseudo::ParseParams{*G, T, Arena, GSS},
-                 *G->findNonterminal("translation-unit"));
+        glrParse(ParseableStream,
+                 clang::pseudo::ParseParams{*PLang.G, PLang.Table, Arena, GSS},
+                 *PLang.G->findNonterminal("translation-unit"));
     if (Print)
-      llvm::outs() << Root.dumpRecursive(*G);
+      llvm::outs() << Root.dumpRecursive(*PLang.G);
   }
 };
 
 Fuzzer *Fuzz = nullptr;
 
 } // namespace
 } // namespace pseudo
 } // namespace clang
 
 extern "C" {
 
 // Set up the fuzzer from command line flags:
-//  -grammar=<file> (required) - path to cxx.bnf
 //  -print                     - used for testing the fuzzer
 int LLVMFuzzerInitialize(int *Argc, char ***Argv) {
-  llvm::StringRef GrammarFile;
   bool PrintForest = false;
   auto ConsumeArg = [&](llvm::StringRef Arg) -> bool {
-    if (Arg.consume_front("-grammar=")) {
-      GrammarFile = Arg;
-      return true;
-    } else if (Arg == "-print") {
+    if (Arg == "-print") {
       PrintForest = true;
       return true;
     }
     return false;
   };
   *Argc = std::remove_if(*Argv + 1, *Argv + *Argc, ConsumeArg) - *Argv;
 
-  if (GrammarFile.empty()) {
-    fprintf(stderr, "Fuzzer needs -grammar=/path/to/cxx.bnf\n");
-    exit(1);
-  }
-  clang::pseudo::Fuzz = new clang::pseudo::Fuzzer(GrammarFile, PrintForest);
+  clang::pseudo::Fuzz = new clang::pseudo::Fuzzer(PrintForest);
   return 0;
 }
 
 int LLVMFuzzerTestOneInput(uint8_t *Data, size_t Size) {
   (*clang::pseudo::Fuzz)(llvm::StringRef(reinterpret_cast<char *>(Data), Size));
   return 0;
 }
 }

clang-tools-extra/pseudo/include/clang-pseudo/GLR.h

private:
   bool GCParity = false; // All nodes should match this, except during GC.
 
   llvm::BumpPtrAllocator Arena;
   unsigned NodesCreated = 0;
 };
 llvm::raw_ostream &operator<<(llvm::raw_ostream &, const GSS::Node &);
 
 // Parameters for the GLR parsing.
+// FIXME: refine it with the ParseLang struct.
 struct ParseParams {
   // The grammar of the language we're going to parse.
   const Grammar &G;
   // The LR table which GLR uses to parse the input, should correspond to the
   // Grammar G.
   const LRTable &Table;
 
   // Arena for data structure used by the GLR algorithm.

clang-tools-extra/pseudo/include/clang-pseudo/ParseLang.h

+//===--- ParseLang.h ------------------------------------------- -*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_PSEUDO_PARSELANG_H
+#define CLANG_PSEUDO_PARSELANG_H
+
+#include "clang-pseudo/grammar/Grammar.h"
+#include "clang-pseudo/grammar/LRTable.h"
+
+namespace clang {
+namespace pseudo {
+
+// Specify a language that can be parsed by the pseduoparser.
+// Manifest generated from a bnf grammar file.
+struct ParseLang {
+  std::unique_ptr<Grammar> G;
+  LRTable Table;
+
+  // FIXME: add clang::LangOptions.
+  // FIXME: add default start symbols.
+};
+
+} // namespace pseudo
+} // namespace clang
+
+#endif // CLANG_PSEUDO_PARSELANG_H

clang-tools-extra/pseudo/include/clang-pseudo/cli/CLI.h

+//===--- CLI.h - Get grammar from variant sources ----------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// A library shared among different pseudoparser-based tools. It provides a
+// uniform way to get basic pieces of the parser (Grammar, LRTable etc) from
+// variant grammar sources.
+// It defines a `--grammar` CLI flag, which supports 1) using a grammar from a
+// file (--grammar=/path/to/lang.bnf) or using the prebuilt cxx language
+// (--grammar=cxx).
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_PSEUDO_CLI_CLI_H
+#define CLANG_PSEUDO_CLI_CLI_H
+
+#include "clang-pseudo/ParseLang.h"
+
+namespace clang {
+namespace pseudo {
+
+// Returns the corresponding language from the '--grammar' command-line flag.
+const ParseLang &getParseLangFromFlags();
+
+} // namespace pseudo
+} // namespace clang
+
+#endif // CLANG_PSEUDO_CLI_CLI_H

clang-tools-extra/pseudo/include/clang-pseudo/cxx/CXX.h

 //  FIXME: not everything is fully compiled yet. The implementation of the
 //  interfaces are still parsing the grammar file at the runtime.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef CLANG_PSEUDO_CXX_CXX_H
 #define CLANG_PSEUDO_CXX_CXX_H
 
+#include "clang-pseudo/ParseLang.h"
 #include "clang-pseudo/grammar/Grammar.h"
 
 namespace clang {
 namespace pseudo {
-class LRTable;
-
 namespace cxx {
 // Symbol represents nonterminal symbols in the C++ grammar.
 // It provides a simple uniform way to access a particular nonterminal.
 enum class Symbol : SymbolID {
 #define NONTERMINAL(X, Y) X = Y,
 #include "CXXSymbols.inc"
 #undef NONTERMINAL
 };
 
-// Returns the C++ grammar.
-const Grammar &getGrammar();
-// Returns the corresponding LRTable for the C++ grammar.
-const LRTable &getLRTable();
+const ParseLang &getParseLang();
 
 } // namespace cxx
 
 } // namespace pseudo
 } // namespace clang
 
 #endif // CLANG_PSEUDO_CXX_CXX_H

clang-tools-extra/pseudo/lib/CMakeLists.txt

+add_subdirectory(cli)
 add_subdirectory(cxx)
 add_subdirectory(grammar)
 
 set(LLVM_LINK_COMPONENTS Support)
 
 add_clang_library(clangPseudo
   Bracket.cpp
   DirectiveTree.cpp

clang-tools-extra/pseudo/lib/cli/CLI.cpp

+//===--- CLI.cpp -  ----------------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+
+#include "clang-pseudo/cli/CLI.h"
+#include "clang-pseudo/ParseLang.h"
+#include "clang-pseudo/cxx/CXX.h"
+#include "clang-pseudo/grammar/LRTable.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <utility>
+
+static llvm::cl::opt<std::string> Grammar(
+    "grammar",
+    llvm::cl::desc(
+        "Specify a BNF grammar file path, or a builtin language (cxx)."),
+    llvm::cl::init("cxx"));
+
+namespace clang {
+namespace pseudo {
+
+const ParseLang &getParseLangFromFlags() {
+  if (::Grammar == "cxx")
+    return cxx::getParseLang();
+  static ParseLang *PLang = []() {
+    // Read from a bnf grammar file.
+    llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> GrammarText =
+        llvm::MemoryBuffer::getFile(::Grammar);
+    if (std::error_code EC = GrammarText.getError()) {
+      llvm::errs() << "Error: can't read grammar file '" << ::Grammar
+                   << "': " << EC.message() << "\n";
+      std::exit(1);
+    }
+    std::vector<std::string> Diags;
+    auto G = Grammar::parseBNF(GrammarText->get()->getBuffer(), Diags);
+    if (!Diags.empty()) {
+      for (const auto &Diag : Diags)
+        llvm::errs() << Diag << "\n";
+    }
+    return new ParseLang{std::move(G), LRTable::buildSLR(*G)};
+  }();
+  return *PLang;
+}
+
+} // namespace pseudo
+} // namespace clang

clang-tools-extra/pseudo/lib/cli/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+add_clang_library(clangPseudoCLI
+  CLI.cpp
+
+  LINK_LIBS
+  clangPseudo
+  clangPseudoGrammar
+  clangPseudoCXX
+  )

clang-tools-extra/pseudo/lib/cxx/CXX.cpp

 //===--- CXX.cpp - Define public interfaces for C++ grammar ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang-pseudo/cxx/CXX.h"
+#include "clang-pseudo/ParseLang.h"
+#include "clang-pseudo/grammar/Grammar.h"
 #include "clang-pseudo/grammar/LRTable.h"
+#include <utility>
 
 namespace clang {
 namespace pseudo {
 namespace cxx {
-
+namespace {
 static const char *CXXBNF =
 #include "CXXBNF.inc"
     ;
+} // namespace
 
-const Grammar &getGrammar() {
-  static std::vector<std::string> Diags;
-  static Grammar *G = Grammar::parseBNF(CXXBNF, Diags).release();
+const ParseLang &getParseLang() {
+  static const auto &CXXLanguage = []() -> const ParseLang & {
+    std::vector<std::string> Diags;
+    auto G = Grammar::parseBNF(CXXBNF, Diags);
     assert(Diags.empty());
-  return *G;
-}
-
-const LRTable &getLRTable() {
-  static LRTable *Table = new LRTable(LRTable::buildSLR(getGrammar()));
-  return *Table;
+    LRTable Table = LRTable::buildSLR(*G);
+    const ParseLang *PL = new ParseLang{
+        std::move(G),
+        std::move(Table),
+    };
+    return *PL;
+  }();
+  return CXXLanguage;
 }
 
 } // namespace cxx
 } // namespace pseudo
 } // namespace clang

clang-tools-extra/pseudo/tool/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS support)
 
 add_clang_tool(clang-pseudo
   ClangPseudo.cpp
   )
 
 clang_target_link_libraries(clang-pseudo
   PRIVATE
   clangBasic
   )
 
 target_link_libraries(clang-pseudo
   PRIVATE
   clangPseudo
   clangPseudoGrammar
+  clangPseudoCLI
   )
 

clang-tools-extra/pseudo/tool/ClangPseudo.cpp

 //===-- ClangPseudo.cpp - Clang pseudoparser tool -------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang-pseudo/Bracket.h"
 #include "clang-pseudo/DirectiveTree.h"
 #include "clang-pseudo/GLR.h"
+#include "clang-pseudo/ParseLang.h"
 #include "clang-pseudo/Token.h"
+#include "clang-pseudo/cli/CLI.h"
 #include "clang-pseudo/grammar/Grammar.h"
 #include "clang-pseudo/grammar/LRGraph.h"
 #include "clang-pseudo/grammar/LRTable.h"
 #include "clang/Basic/LangOptions.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Signals.h"
 
-using clang::pseudo::Grammar;
 using clang::pseudo::TokenStream;
 using llvm::cl::desc;
 using llvm::cl::init;
 using llvm::cl::opt;
 
-static opt<std::string>
-    Grammar("grammar", desc("Parse and check a BNF grammar file."), init(""));
 static opt<bool> PrintGrammar("print-grammar", desc("Print the grammar."));
 static opt<bool> PrintGraph("print-graph",
                             desc("Print the LR graph for the grammar"));
 static opt<bool> PrintTable("print-table",
                             desc("Print the LR table for the grammar"));
 static opt<std::string> Source("source", desc("Source file"));
 static opt<bool> PrintSource("print-source", desc("Print token stream"));
 static opt<bool> PrintTokens("print-tokens", desc("Print detailed token info"));
if (PrintTokens)
       llvm::outs() << *Stream;
     if (PrintDirectiveTree)
       llvm::outs() << DirectiveStructure;
 
     ParseableStream = clang::pseudo::stripComments(cook(*Stream, LangOpts));
     pairBrackets(*ParseableStream);
   }
 
-  if (Grammar.getNumOccurrences()) {
-    std::string Text = readOrDie(Grammar);
-    std::vector<std::string> Diags;
-    auto G = Grammar::parseBNF(Text, Diags);
-
-    if (!Diags.empty()) {
-      llvm::errs() << llvm::join(Diags, "\n");
-      return 2;
-    }
-    llvm::outs() << llvm::formatv("grammar file {0} is parsed successfully\n",
-                                  Grammar);
-    if (PrintGrammar)
-      llvm::outs() << G->dump();
-    if (PrintGraph)
-      llvm::outs() << clang::pseudo::LRGraph::buildLR0(*G).dumpForTests(*G);
-    auto LRTable = clang::pseudo::LRTable::buildSLR(*G);
-    if (PrintTable)
-      llvm::outs() << LRTable.dumpForTests(*G);
-    if (PrintStatistics)
-      llvm::outs() << LRTable.dumpStatistics();
+  const auto &Lang = clang::pseudo::getParseLangFromFlags();
+    llvm::outs() << Lang.G->dump();
+    llvm::outs() << clang::pseudo::LRGraph::buildLR0(*Lang.G).dumpForTests(
+        *Lang.G);
+
+    llvm::outs() << Lang.Table.dumpForTests(*Lang.G);
+    llvm::outs() << Lang.Table.dumpStatistics();
 
-    if (ParseableStream) {
-      clang::pseudo::ForestArena Arena;
-      clang::pseudo::GSS GSS;
-      llvm::Optional<clang::pseudo::SymbolID> StartSymID =
-          G->findNonterminal(StartSymbol);
-      if (!StartSymID) {
-        llvm::errs() << llvm::formatv(
-            "The start symbol {0} doesn't exit in the grammar!\n", Grammar);
-        return 2;
-      }
-      auto &Root = glrParse(*ParseableStream,
-                            clang::pseudo::ParseParams{*G, LRTable, Arena, GSS},
-                            *StartSymID);
-      if (PrintForest)
-        llvm::outs() << Root.dumpRecursive(*G, /*Abbreviated=*/true);
+        Lang.G->findNonterminal(StartSymbol);
+          "The start symbol {0} doesn't exit in the grammar!\n", StartSymbol);
+    auto &Root =
+        glrParse(*ParseableStream,
+                 clang::pseudo::ParseParams{*Lang.G, Lang.Table, Arena, GSS},
+      llvm::outs() << Root.dumpRecursive(*Lang.G, /*Abbreviated=*/true);
 
-      if (PrintStatistics) {
-        llvm::outs() << "Forest bytes: " << Arena.bytes()
-                     << " nodes: " << Arena.nodeCount() << "\n";
-        llvm::outs() << "GSS bytes: " << GSS.bytes()
-                     << " nodes: " << GSS.nodesCreated() << "\n";
-      }
-    }
-  }
 
   return 0;
 }

clang-tools-extra/pseudo/unittests/GLRTest.cpp

 //===--- GLRTest.cpp - Test the GLR parser ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang-pseudo/GLR.h"
 #include "clang-pseudo/Token.h"
+#include "clang-pseudo/ParseLang.h"
 #include "clang-pseudo/grammar/Grammar.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/TokenKinds.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 #include <memory>
parents(llvm::ArrayRef<const GSS::Node *> Parents) {
   return testing::Property(&GSS::Node::parents,
                            testing::UnorderedElementsAreArray(Parents));
 }
 
 class GLRTest : public ::testing::Test {
 public:
   void build(llvm::StringRef GrammarBNF) {
     std::vector<std::string> Diags;
-    G = Grammar::parseBNF(GrammarBNF, Diags);
+    TestLang.G = Grammar::parseBNF(GrammarBNF, Diags);
+  }
+
+  TokenStream emptyTokenStream() {
+    TokenStream Empty;
+    Empty.finalize();
+    return Empty;
   }
  
   void buildGrammar(std::vector<std::string> Nonterminals,
                     std::vector<std::string> Rules) {
     Nonterminals.push_back("_");
     llvm::sort(Nonterminals);
     Nonterminals.erase(std::unique(Nonterminals.begin(), Nonterminals.end()),
                        Nonterminals.end());
     std::string FakeTestBNF;
     for (const auto &NT : Nonterminals)
       FakeTestBNF += llvm::formatv("{0} := {1}\n", "_", NT);
     FakeTestBNF += llvm::join(Rules, "\n");
     build(FakeTestBNF);
   }
 
   SymbolID id(llvm::StringRef Name) const {
     for (unsigned I = 0; I < NumTerminals; ++I)
-      if (G->table().Terminals[I] == Name)
+      if (TestLang.G->table().Terminals[I] == Name)
         return tokenSymbol(static_cast<tok::TokenKind>(I));
-    for (SymbolID ID = 0; ID < G->table().Nonterminals.size(); ++ID)
-      if (G->table().Nonterminals[ID].Name == Name)
+    for (SymbolID ID = 0; ID < TestLang.G->table().Nonterminals.size(); ++ID)
+      if (TestLang.G->table().Nonterminals[ID].Name == Name)
         return ID;
     ADD_FAILURE() << "No such symbol found: " << Name;
     return 0;
   }
 
   RuleID ruleFor(llvm::StringRef NonterminalName) const {
-    auto RuleRange = G->table().Nonterminals[id(NonterminalName)].RuleRange;
+    auto RuleRange =
+        TestLang.G->table().Nonterminals[id(NonterminalName)].RuleRange;
     if (RuleRange.End - RuleRange.Start == 1)
-      return G->table().Nonterminals[id(NonterminalName)].RuleRange.Start;
+      return TestLang.G->table()
+          .Nonterminals[id(NonterminalName)]
+          .RuleRange.Start;
     ADD_FAILURE() << "Expected a single rule for " << NonterminalName
                   << ", but it has " << RuleRange.End - RuleRange.Start
                   << " rule!\n";
     return 0;
   }
 
 protected:
-  std::unique_ptr<Grammar> G;
+  ParseLang TestLang;
   ForestArena Arena;
   GSS GSStack;
 };
 
 TEST_F(GLRTest, ShiftMergingHeads) {
   // Given a test case where we have two heads 1, 2, 3 in the GSS, the heads 1,
   // 2 have shift actions to reach state 4, and the head 3 has a shift action to
   // reach state 5:
TEST_F(GLRTest, ShiftMergingHeads) {
   auto *GSSNode1 = GSStack.addNode(/*State=*/1, /*ForestNode=*/nullptr,
                                    /*Parents=*/{GSSNode0});
   auto *GSSNode2 = GSStack.addNode(/*State=*/2, /*ForestNode=*/nullptr,
                                    /*Parents=*/{GSSNode0});
   auto *GSSNode3 = GSStack.addNode(/*State=*/3, /*ForestNode=*/nullptr,
                                    /*Parents=*/{GSSNode0});
 
   buildGrammar({}, {}); // Create a fake empty grammar.
-  LRTable T =
-      LRTable::buildForTests(G->table(), /*Entries=*/{
+  TestLang.Table =
+      LRTable::buildForTests(TestLang.G->table(), /*Entries=*/{
                                  {1, tokenSymbol(tok::semi), Action::shift(4)},
                                  {2, tokenSymbol(tok::semi), Action::shift(4)},
                                  {3, tokenSymbol(tok::semi), Action::shift(5)},
                              });
 
   ForestNode &SemiTerminal = Arena.createTerminal(tok::semi, 0);
   std::vector<const GSS::Node *> NewHeads;
   glrShift({GSSNode1, GSSNode2, GSSNode3}, SemiTerminal,
-           {*G, T, Arena, GSStack}, NewHeads);
+           {*TestLang.G, TestLang.Table, Arena, GSStack}, NewHeads);
 
   EXPECT_THAT(NewHeads,
               UnorderedElementsAre(AllOf(state(4), parsedSymbol(&SemiTerminal),
                                          parents({GSSNode1, GSSNode2})),
                                    AllOf(state(5), parsedSymbol(&SemiTerminal),
                                          parents({GSSNode3}))))
       << NewHeads;
 }
 
 TEST_F(GLRTest, ReduceConflictsSplitting) {
   //  Before (splitting due to R/R conflict):
   //    0--1(IDENTIFIER)
   //  After reducing 1 by `class-name := IDENTIFIER` and
   //                      `enum-name := IDENTIFIER`:
   //    0--2(class-name)    // 2 is goto(0, class-name)
   //    └--3(enum-name)     // 3 is goto(0, enum-name)
   buildGrammar({"class-name", "enum-name"},
                {"class-name := IDENTIFIER", "enum-name := IDENTIFIER"});
 
-  LRTable Table = LRTable::buildForTests(
-      G->table(), {
+  TestLang.Table = LRTable::buildForTests(
+      TestLang.G->table(), {
                       {/*State=*/0, id("class-name"), Action::goTo(2)},
                       {/*State=*/0, id("enum-name"), Action::goTo(3)},
                       {/*State=*/1, tokenSymbol(tok::l_brace),
                        Action::reduce(ruleFor("class-name"))},
                       {/*State=*/1, tokenSymbol(tok::l_brace),
                        Action::reduce(ruleFor("enum-name"))},
                   });
 
   const auto *GSSNode0 =
       GSStack.addNode(/*State=*/0, /*ForestNode=*/nullptr, /*Parents=*/{});
   const auto *GSSNode1 =
       GSStack.addNode(1, &Arena.createTerminal(tok::identifier, 0), {GSSNode0});
 
   std::vector<const GSS::Node *> Heads = {GSSNode1};
-  glrReduce(Heads, tokenSymbol(tok::l_brace), {*G, Table, Arena, GSStack});
+  glrReduce(Heads, tokenSymbol(tok::l_brace), {*TestLang.G, TestLang.Table, Arena, GSStack});
   EXPECT_THAT(Heads, UnorderedElementsAre(
                          GSSNode1,
                          AllOf(state(2), parsedSymbolID(id("class-name")),
                                parents({GSSNode0})),
                          AllOf(state(3), parsedSymbolID(id("enum-name")),
                                parents({GSSNode0}))))
       << Heads;
 }
TEST_F(GLRTest, ReduceSplittingDueToMultipleBases) {
   const auto *GSSNode2 =
       GSStack.addNode(/*State=*/2, /*ForestNode=*/ClassNameNode, /*Parents=*/{});
   const auto *GSSNode3 =
       GSStack.addNode(/*State=*/3, /*ForestNode=*/EnumNameNode, /*Parents=*/{});
   const auto *GSSNode4 = GSStack.addNode(
       /*State=*/4, &Arena.createTerminal(tok::star, /*TokenIndex=*/1),
       /*Parents=*/{GSSNode2, GSSNode3});
 
-  LRTable Table = LRTable::buildForTests(
-      G->table(),
+  TestLang.Table = LRTable::buildForTests(
+      TestLang.G->table(),
       {
           {/*State=*/2, id("ptr-operator"), Action::goTo(/*NextState=*/5)},
           {/*State=*/3, id("ptr-operator"), Action::goTo(/*NextState=*/6)},
           {/*State=*/4, tokenSymbol(tok::identifier),
            Action::reduce(ruleFor("ptr-operator"))},
       });
   std::vector<const GSS::Node *> Heads = {GSSNode4};
-  glrReduce(Heads, tokenSymbol(tok::identifier), {*G, Table, Arena, GSStack});
+  glrReduce(Heads, tokenSymbol(tok::identifier), {*TestLang.G, TestLang.Table, Arena, GSStack});
 
   EXPECT_THAT(Heads, UnorderedElementsAre(
                          GSSNode4,
                          AllOf(state(5), parsedSymbolID(id("ptr-operator")),
                                parents({GSSNode2})),
                          AllOf(state(6), parsedSymbolID(id("ptr-operator")),
                                parents({GSSNode3}))))
       << Heads;
TEST_F(GLRTest, ReduceJoiningWithMultipleBases) {
   const auto *GSSNode3 =
       GSStack.addNode(/*State=*/3, /*ForestNode=*/ClassNameNode,
                       /*Parents=*/{GSSNode1});
   const auto *GSSNode4 =
       GSStack.addNode(/*State=*/4, /*ForestNode=*/EnumNameNode,
                       /*Parents=*/{GSSNode2});
 
   // FIXME: figure out a way to get rid of the hard-coded reduce RuleID!
-  LRTable Table = LRTable::buildForTests(
-      G->table(),
+  TestLang.Table = LRTable::buildForTests(
+      TestLang.G->table(),
       {
           {/*State=*/1, id("type-name"), Action::goTo(/*NextState=*/5)},
           {/*State=*/2, id("type-name"), Action::goTo(/*NextState=*/5)},
           {/*State=*/3, tokenSymbol(tok::l_paren),
            Action::reduce(/* type-name := class-name */ 0)},
           {/*State=*/4, tokenSymbol(tok::l_paren),
            Action::reduce(/* type-name := enum-name */ 1)},
       });
   std::vector<const GSS::Node *> Heads = {GSSNode3, GSSNode4};
-  glrReduce(Heads, tokenSymbol(tok::l_paren), {*G, Table, Arena, GSStack});
+  glrReduce(Heads, tokenSymbol(tok::l_paren), {*TestLang.G, TestLang.Table, Arena, GSStack});
 
   // Verify that the stack heads are joint at state 5 after reduces.
   EXPECT_THAT(Heads, UnorderedElementsAre(GSSNode3, GSSNode4,
                                           AllOf(state(5),
                                                 parsedSymbolID(id("type-name")),
                                                 parents({GSSNode1, GSSNode2}))))
       << Heads;
   // Verify that we create an ambiguous ForestNode of two parses of `type-name`.
-  EXPECT_EQ(Heads.back()->Payload->dumpRecursive(*G),
+  EXPECT_EQ(Heads.back()->Payload->dumpRecursive(*TestLang.G),
             "[  1, end) type-name := <ambiguous>\n"
             "[  1, end) ├─type-name := class-name\n"
             "[  1, end) │ └─class-name := <opaque>\n"
             "[  1, end) └─type-name := enum-name\n"
             "[  1, end)   └─enum-name := <opaque>\n");
 }
 
 TEST_F(GLRTest, ReduceJoiningWithSameBase) {
TEST_F(GLRTest, ReduceJoiningWithSameBase) {
   const auto *GSSNode3 =
       GSStack.addNode(/*State=*/3, /*ForestNode=*/StartTerminal,
                       /*Parents=*/{GSSNode1});
   const auto *GSSNode4 =
       GSStack.addNode(/*State=*/4, /*ForestNode=*/StartTerminal,
                       /*Parents=*/{GSSNode2});
 
   // FIXME: figure out a way to get rid of the hard-coded reduce RuleID!
-  LRTable Table = LRTable::buildForTests(
-      G->table(), {
+  TestLang.Table = LRTable::buildForTests(
+      TestLang.G->table(), {
                       {/*State=*/0, id("pointer"), Action::goTo(5)},
                       {3, tokenSymbol(tok::l_paren),
                        Action::reduce(/* pointer := class-name */ 0)},
                       {4, tokenSymbol(tok::l_paren),
                        Action::reduce(/* pointer := enum-name */ 1)},
                   });
   std::vector<const GSS::Node *> Heads = {GSSNode3, GSSNode4};
-  glrReduce(Heads, tokenSymbol(tok::l_paren), {*G, Table, Arena, GSStack});
+  glrReduce(Heads, tokenSymbol(tok::l_paren), {*TestLang.G, TestLang.Table, Arena, GSStack});
 
   EXPECT_THAT(
       Heads, UnorderedElementsAre(GSSNode3, GSSNode4,
                                   AllOf(state(5), parsedSymbolID(id("pointer")),
                                         parents({GSSNode0}))))
       << Heads;
-  EXPECT_EQ(Heads.back()->Payload->dumpRecursive(*G),
+  EXPECT_EQ(Heads.back()->Payload->dumpRecursive(*TestLang.G),
             "[  0, end) pointer := <ambiguous>\n"
             "[  0, end) ├─pointer := class-name *\n"
             "[  0,   1) │ ├─class-name := <opaque>\n"
             "[  1, end) │ └─* := tok[1]\n"
             "[  0, end) └─pointer := enum-name *\n"
             "[  0,   1)   ├─enum-name := <opaque>\n"
             "[  1, end)   └─* := tok[1]\n");
 }
build(R"bnf(
     _ := test
 
     test := { expr
     test := { IDENTIFIER
     test := left-paren expr
     left-paren := {
     expr := IDENTIFIER
   )bnf");
+  TestLang.Table = LRTable::buildSLR(*TestLang.G);
   clang::LangOptions LOptions;
   const TokenStream &Tokens = cook(lex("{ abc", LOptions), LOptions);
-  auto LRTable = LRTable::buildSLR(*G);
 
   const ForestNode &Parsed =
-      glrParse(Tokens, {*G, LRTable, Arena, GSStack}, id("test"));
+      glrParse(Tokens, {*TestLang.G, TestLang.Table, Arena, GSStack}, id("test"));
   // Verify that there is no duplicated sequence node of `expr := IDENTIFIER`
   // in the forest, see the `#1` and `=#1` in the dump string.
-  EXPECT_EQ(Parsed.dumpRecursive(*G),
+  EXPECT_EQ(Parsed.dumpRecursive(*TestLang.G),
             "[  0, end) test := <ambiguous>\n"
             "[  0, end) ├─test := { expr\n"
             "[  0,   1) │ ├─{ := tok[0]\n"
             "[  1, end) │ └─expr := IDENTIFIER #1\n"
             "[  1, end) │   └─IDENTIFIER := tok[1]\n"
             "[  0, end) ├─test := { IDENTIFIER\n"
             "[  0,   1) │ ├─{ := tok[0]\n"
             "[  1, end) │ └─IDENTIFIER := tok[1]\n"
build(R"bnf(
     _ := test
 
     test := IDENTIFIER
     test := foo
     foo := IDENTIFIER
   )bnf");
   clang::LangOptions LOptions;
   const TokenStream &Tokens = cook(lex("IDENTIFIER", LOptions), LOptions);
-  auto LRTable = LRTable::buildSLR(*G);
+  TestLang.Table = LRTable::buildSLR(*TestLang.G);
 
   const ForestNode &Parsed =
-      glrParse(Tokens, {*G, LRTable, Arena, GSStack}, id("test"));
-  EXPECT_EQ(Parsed.dumpRecursive(*G),
+      glrParse(Tokens, {*TestLang.G, TestLang.Table, Arena, GSStack}, id("test"));
+  EXPECT_EQ(Parsed.dumpRecursive(*TestLang.G),
             "[  0, end) test := <ambiguous>\n"
             "[  0, end) ├─test := IDENTIFIER\n"
             "[  0, end) │ └─IDENTIFIER := tok[0]\n"
             "[  0, end) └─test := foo\n"
             "[  0, end)   └─foo := IDENTIFIER\n"
             "[  0, end)     └─IDENTIFIER := tok[0]\n");
 }
 
 TEST_F(GLRTest, NoExplicitAccept) {
   build(R"bnf(
     _ := test
 
     test := IDENTIFIER test
     test := IDENTIFIER
   )bnf");
   clang::LangOptions LOptions;
   // Given the following input, and the grammar above, we perform two reductions
   // of the nonterminal `test` when the next token is `eof`, verify that the
   // parser stops at the right state.
   const TokenStream &Tokens = cook(lex("id id", LOptions), LOptions);
-  auto LRTable = LRTable::buildSLR(*G);
+  TestLang.Table = LRTable::buildSLR(*TestLang.G);
 
   const ForestNode &Parsed =
-      glrParse(Tokens, {*G, LRTable, Arena, GSStack}, id("test"));
-  EXPECT_EQ(Parsed.dumpRecursive(*G),
+      glrParse(Tokens, {*TestLang.G, TestLang.Table, Arena, GSStack}, id("test"));
+  EXPECT_EQ(Parsed.dumpRecursive(*TestLang.G),
             "[  0, end) test := IDENTIFIER test\n"
             "[  0,   1) ├─IDENTIFIER := tok[0]\n"
             "[  1, end) └─test := IDENTIFIER\n"
             "[  1, end)   └─IDENTIFIER := tok[1]\n");
 }
 
 TEST(GSSTest, GC) {
   //      ┌-A-┬-AB