[TableGen] New backend to print fully detailed records

[Paul-C-Anagnostopoulos]

Summary

This new TableGen backend, invoked with --print-detailed-records, prints all the details about the global variables, classes, and records in a TableGen file. It includes more details than the default backend.

The "TableGen Backend Developer's Guide" is updated to describe this new backend.

Add a function to the SourceMgr class to get a string with a source location formatted in the standard way.

[Paul-C-Anagnostopoulos]

I have addressed all the pertinent Lint comments.

[lattner]

Looks nice to me, please tidy up the SourceMgr stuff and I'll take another look.

llvm/lib/Support/SourceMgr.cpp

207

Is it standard to drop the path prefix? Is this what happens when sourcemgr emits errors and warning diagnostics?

210

Shouldn't this also include column numbers?

Making this utility accessible to other clients of SourceMgr is nice, but if so, please change SourceMgr itself to use it for its internal formatting of the location information. This ensures that things stay consistent.

[Paul-C-Anagnostopoulos]

llvm/lib/Support/SourceMgr.cpp

207

No, it is not dropped in messages. However, it adds horrible clutter to the detailed record report, since the paths can be long. I convinced myself that there will rarely, if ever, be a duplicate filename and so the source locations will be clear.

(Check the examples above and picture them with much longer filenames.)

210

While testing the new backend, I found the column numbers to be of no use. The primary purpose of showing the source locations is to help people figure out the path that a record or field took through all the TableGen definitions. Knowing the column number of, say, the value of a field is no more useful than knowing the line number of its definition.

I will investigate using this new function in the SourceMgr itself.

[lattner]

llvm/lib/Support/SourceMgr.cpp

210

Ok, no problem. This just means that this method isn't a general "SourceMgr::getFormattedLocation" method, it is something specific to the tblgen backend. Please move it there, and the concern is addressed, thanks!

[Paul-C-Anagnostopoulos]

llvm/lib/Support/SourceMgr.cpp

210

The formatting of messages is already spread between a SourceMgr function and an SMDiagnostic function. The latter is the most general one, but it writes directly to a stream rather than returning a string. So I don't think the new function is contributing much to the confusion. I will, however, add a comment about this.

One solution is to write an SMDiagnostic low-level message formatter that returns a string, then use it everywhere. However, that requires you to create an SMDiagnostic instance just to format a location. So perhaps the work should be done by a new SMLoc function. The point is that there are SMLoc formatting requirements that have nothing to do with diagnostic messages.

I have added this issue to my list of things to investigate.

[Paul-C-Anagnostopoulos]

llvm/lib/Support/SourceMgr.cpp

210

It would be general if I add a second optional argument to include the column number. Then it can format a source location in the full style, or in various abbreviated styles.

I put the function in SourceMgr so that the new backend wouldn't directly call such functions as FindBufferContaining and GetBufferInfo. Let me check something . . .

So, only two existing TableGen backends call those functions directly: CTagsEmitter and DAGIselMatcherEmitter. So I propose to (1) create this new function we're discussing; (2) use it in those two backends; (3) then address the idea of one general-purpose location formatter.

[lattner]

SGTM!

[Paul-C-Anagnostopoulos]

Further investigation reveals that every every backend that prints source locations does not include the line offset. Further, PrintIncludeStack(), in the source manager itself, also does not include the line offset. So I am going to leave this new function without the line offset.

Once this patch is pushed, I will find all those places and change them to use this new function. That cleans away a bunch of uses of lower-level SourceMgr functions.

Then I will ponder what we really want for a universal source file location formatter.

[Paul-C-Anagnostopoulos]

I changed SourceMgr::getFormattedLocation() to getFormattedLocationNoOffset() to make it clear and to save the former name for a possible general source location formatter.

I will push this patch on Friday.

llvm/docs/TableGen/BackGuide.rst

 ``RecordVal``, and
 ``Init``.
 
-A constant and two helper functions are provided for producing the output
-file.  The constant ``MAX_LINE_LEN`` specifies the maximum length of output
-lines.  The helper function ``printLine`` prints a horizontal line comment.
-The helper function ``emitSourceFileHeader`` prints the header comment that
-should be included at the top of every output file.
+The helper function ``emitSourceFileHeader()`` prints the header comment
+that should be included at the top of every output file. A call to it is
+included in the skeleton backend file ``TableGenBackendSkeleton.cpp``.
 
 Printing Error Messages
 =======================

 fields. Note that anonymous records are named ``anonymous_0``,
 ``anonymous_1``, etc.
 
-
-
 The ``PrintDetailedRecords`` Backend
 ------------------------------------
 
-[to come]
+The TableGen command option ``--print-detailed-records`` invokes a backend
+that prints all the global variables, classes, and records defined in the
+source files. The output looks like this.
+
+.. code-block:: text
+
+  DETAILED RECORDS for file llvm-project\llvm\lib\target\arc\arc.td
+  
+  -------------------- Global Variables (5) --------------------
+  
+  AMDGPUBufferIntrinsics = [int_amdgcn_buffer_load_format, ...
+  AMDGPUImageDimAtomicIntrinsics = [int_amdgcn_image_atomic_swap_1d, ...
+  ...
+  -------------------- Classes (758) --------------------
+  
+  AMDGPUBufferLoad  |IntrinsicsAMDGPU.td:879|
+    Template args:
+      LLVMType AMDGPUBufferLoad:data_ty = llvm_any_ty  |IntrinsicsAMDGPU.td:879|
+    Superclasses: (SDPatternOperator) Intrinsic AMDGPURsrcIntrinsic
+    Fields:
+      list<SDNodeProperty> Properties = [SDNPMemOperand]  |Intrinsics.td:348|
+      string LLVMName = ""  |Intrinsics.td:343|
+  ...
+  -------------------- Records (12303) --------------------
+  
+  AMDGPUSample_lz_o  |IntrinsicsAMDGPU.td:560|
+    Defm sequence: |IntrinsicsAMDGPU.td:584| |IntrinsicsAMDGPU.td:566|
+    Superclasses: AMDGPUSampleVariant
+    Fields:
+      string UpperCaseMod = "_LZ_O"  |IntrinsicsAMDGPU.td:542|
+      string LowerCaseMod = "_lz_o"  |IntrinsicsAMDGPU.td:543|
+  ...
+
+* Global variables defined with outer ``defvar`` statements are shown with
+  their values.
+
+* The classes are shown with their source location, template arguments,
+  superclasses, and fields. 
+
+* The records are shown with their source location, ``defm`` sequence,
+  superclasses, and fields.
+
+Superclasses are shown in the order processed, with indirect superclasses in
+parentheses. Each field is shown with its value and the source location at
+which it was set.
+The ``defm`` sequence gives the locations of the ``defm`` statements that
+were involved in generating the record, in the order they were invoked.
 No newline at end of file

llvm/include/llvm/Support/SourceMgr.h

   std::pair<unsigned, unsigned> getLineAndColumn(SMLoc Loc,
                                                  unsigned BufferID = 0) const;
 
+  /// Get a string with the \p SMLoc filename and line number
+  /// formatted in the standard style.
+  std::string getFormattedLocationNoOffset(SMLoc Loc,
+                                           bool IncludePath = false) const;
+
   /// Given a line and column number in a mapped buffer, turn it into an SMLoc.
   /// This will return a null SMLoc if the line/column location is invalid.
   SMLoc FindLocForLineAndColumn(unsigned BufferID, unsigned LineNo,

llvm/include/llvm/TableGen/Record.h

     return SuperClasses;
   }
 
-  /// Append the direct super classes of this record to Classes.
+  /// Determine whether this record has the specified direct superclass.
+  bool hasDirectSuperClass(const Record *SuperClass) const;
+
+  /// Append the direct superclasses of this record to Classes.
   void getDirectSuperClasses(SmallVectorImpl<Record *> &Classes) const;
 
   bool isTemplateArg(Init *Name) const {

   friend class RecordRecTy;
 
   using RecordMap = std::map<std::string, std::unique_ptr<Record>, std::less<>>;
+  using GlobalMap = std::map<std::string, Init *, std::less<>>;
 
+  std::string InputFilename;
   RecordMap Classes, Defs;
   FoldingSet<RecordRecTy> RecordTypePool;
   std::map<std::string, Init *, std::less<>> ExtraGlobals;
   unsigned AnonCounter = 0;
 
 public:
+  /// Get the main TableGen input file's name.
+  const std::string getInputFilename() const { return InputFilename; }
+
   /// Get the map of classes.
   const RecordMap &getClasses() const { return Classes; }
 
   /// Get the map of records (defs).
   const RecordMap &getDefs() const { return Defs; }
 
+  /// Get the map of global variables.
+  const GlobalMap &getGlobals() const { return ExtraGlobals; }
+
   /// Get the class with the specified name.
   Record *getClass(StringRef Name) const {
     auto I = Classes.find(Name);

     return It == ExtraGlobals.end() ? nullptr : It->second;
   }
 
+  void saveInputFilename(std::string Filename) {
+    InputFilename = Filename;
+  }
+
   void addClass(std::unique_ptr<Record> R) {
     bool Ins = Classes.insert(std::make_pair(std::string(R->getName()),
                                              std::move(R))).second;

   Init *resolve(Init *VarName) override;
 };
 
+void EmitDetailedRecords(RecordKeeper &RK, raw_ostream &OS);
 void EmitJSON(RecordKeeper &RK, raw_ostream &OS);
 
 } // end namespace llvm

llvm/lib/Support/SourceMgr.cpp

 SourceMgr::getLineAndColumn(SMLoc Loc, unsigned BufferID) const {
   if (!BufferID)
     BufferID = FindBufferContainingLoc(Loc);
-  assert(BufferID && "Invalid Location!");
+  assert(BufferID && "Invalid location!");
 
   auto &SB = getBufferInfo(BufferID);
   const char *Ptr = Loc.getPointer();

   return std::make_pair(LineNo, Ptr - BufStart - NewlineOffs);
 }
 
+// FIXME: Note that the formatting of source locations is spread between
+// multiple functions, some in SourceMgr and some in SMDiagnostic. A better
+// solution would be a general-purpose source location formatter
+// in one of those two classes, or possibly in SMLoc.
+
+/// Get a string with the source location formatted in the standard
+/// style, but without the line offset. If \p IncludePath is true, the path
+/// is included. If false, only the file name and extension are included.
+std::string SourceMgr::getFormattedLocationNoOffset(SMLoc Loc,
+                                                    bool IncludePath) const {
+  auto BufferID = FindBufferContainingLoc(Loc);
+  assert(BufferID && "Invalid location!");
+  auto FileSpec = getBufferInfo(BufferID).Buffer->getBufferIdentifier();
+
+  if (IncludePath) {
+    return FileSpec.str() + ":" + std::to_string(FindLineNumber(Loc, BufferID));
+  } else {
+    auto I = FileSpec.find_last_of("/\\");
+    I = (I == FileSpec.size()) ? 0 : (I + 1);
+    return FileSpec.substr(I).str() + ":" +
+           std::to_string(FindLineNumber(Loc, BufferID));
+  }
+}
+
 /// Given a line and column number in a mapped buffer, turn it into an SMLoc.
 /// This will return a null SMLoc if the line/column location is invalid.
 SMLoc SourceMgr::FindLocForLineAndColumn(unsigned BufferID, unsigned LineNo,

llvm/lib/TableGen/CMakeLists.txt

 add_llvm_component_library(LLVMTableGen
+  DetailedRecordsBackend.cpp
   Error.cpp
   JSONBackend.cpp
   Main.cpp

llvm/lib/TableGen/DetailedRecordsBackend.cpp

+//===- DetailedRecordBackend.cpp - Detailed Records Report      -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This Tablegen backend prints a report that includes all the global 
+// variables, classes, and records in complete detail. It includes more
+// detail than the default TableGen printer backend.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Format.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+#include "llvm/TableGen/TableGenBackend.h"
+#include <algorithm>
+#include <set>
+#include <string>
+#include <vector>
+
+#define DEBUG_TYPE "detailed-records-backend"
+
+#define NL "\n"
+
+using namespace llvm;
+
+namespace {
+
+class DetailedRecordsEmitter {
+private:
+  RecordKeeper &Records;
+
+public:
+  DetailedRecordsEmitter(RecordKeeper &RK) : Records(RK) {}
+
+  void run(raw_ostream &OS);
+  void printReportHeading(raw_ostream &OS);
+  void printVariables(raw_ostream &OS);
+  void printClasses(raw_ostream &OS);
+  void printRecords(raw_ostream &OS);
+  void printSectionHeading(std::string Title, int Count, raw_ostream &OS);
+  void printDefms(Record *Rec, raw_ostream &OS);
+  void printTemplateArgs(Record *Rec, raw_ostream &OS);
+  void printSuperclasses(Record *Rec, raw_ostream &OS);
+  void printFields(Record *Rec, raw_ostream &OS);
+  std::string formatLocation(const SMLoc Loc);
+}; // emitter class
+
+} // anonymous namespace
+
+// Print the report.
+void DetailedRecordsEmitter::run(raw_ostream &OS) {
+  printReportHeading(OS);
+  printVariables(OS);
+  printClasses(OS);
+  printRecords(OS);
+}
+
+// Print the report heading, including the source file name.
+void DetailedRecordsEmitter::printReportHeading(raw_ostream &OS) {
+  OS << formatv("DETAILED RECORDS for file {0}\n", Records.getInputFilename());
+}
+
+// Print the global variables.
+void DetailedRecordsEmitter::printVariables(raw_ostream &OS) {
+  const auto GlobalList = Records.getGlobals();
+  printSectionHeading("Global Variables", GlobalList.size(), OS);
+
+  OS << NL;
+  for (const auto &Var : GlobalList) {
+    OS << Var.first << " = " << Var.second->getAsString() << NL;
+  }
+}
+
+// Print the classes, including the template arguments, superclasses,
+// and fields.
+void DetailedRecordsEmitter::printClasses(raw_ostream &OS) {
+  const auto &ClassList = Records.getClasses();
+  printSectionHeading("Classes", ClassList.size(), OS);
+
+  for (const auto &ClassPair : ClassList) {
+    auto *const Class = ClassPair.second.get();
+    OS << formatv("\n{0}  |{1}|\n", Class->getNameInitAsString(),
+                  SrcMgr.getFormattedLocationNoOffset(Class->getLoc().front()));
+    printTemplateArgs(Class, OS);
+    printSuperclasses(Class, OS);
+    printFields(Class, OS);
+  }
+}
+
+// Print the records, including the defm sequences, supercasses,
+// and fields.
+void DetailedRecordsEmitter::printRecords(raw_ostream &OS) {
+  const auto &RecordList = Records.getDefs();
+  printSectionHeading("Records", RecordList.size(), OS);
+
+  for (const auto &RecPair : RecordList) {
+    auto *const Rec = RecPair.second.get();
+    OS << formatv("\n{0}  |{1}|\n", Rec->getNameInitAsString(),
+                  SrcMgr.getFormattedLocationNoOffset(Rec->getLoc().front()));
+    printDefms(Rec, OS);
+    printSuperclasses(Rec, OS);
+    printFields(Rec, OS);
+  }
+}
+
+// Print a section heading with the name of the section and
+// the item count.
+void DetailedRecordsEmitter::printSectionHeading(std::string Title, int Count,
+                                                 raw_ostream &OS) {
+  OS << formatv("\n{0} {1} ({2}) {0}\n", "--------------------", Title, Count);
+}
+
+// Print the record's defm source locations, if any. Note that they
+// are stored in the reverse order of their invocation.
+void DetailedRecordsEmitter::printDefms(Record *Rec, raw_ostream &OS) {
+  const auto &LocList = Rec->getLoc();
+  if (LocList.size() < 2)
+    return;
+
+  OS << "  Defm sequence:";
+  for (unsigned I = LocList.size() - 1; I >= 1; --I) {
+    OS << formatv(" |{0}|", SrcMgr.getFormattedLocationNoOffset(LocList[I]));
+  }
+  OS << NL;
+}
+
+// Print the template arguments of a class.
+void DetailedRecordsEmitter::printTemplateArgs(Record *Rec,
+                                               raw_ostream &OS) {
+  ArrayRef<Init *> Args = Rec->getTemplateArgs();
+  if (Args.empty()) {
+    OS << "  Template args: (none)\n";
+    return;
+  }
+
+  OS << "  Template args:\n";
+  for (const Init *ArgName : Args) {
+    const RecordVal *Value = Rec->getValue(ArgName);
+    assert(Value && "Template argument value not found.");
+    OS << "    ";
+    Value->print(OS, false);
+    OS << formatv("  |{0}|", SrcMgr.getFormattedLocationNoOffset(Value->getLoc()));
+    OS << NL;
+  }
+}
+
+// Print the superclasses of a class or record. Indirect superclasses
+// are enclosed in parentheses.
+void DetailedRecordsEmitter::printSuperclasses(Record *Rec, raw_ostream &OS) {
+  ArrayRef<std::pair<Record *, SMRange>> Superclasses = Rec->getSuperClasses();
+  if (Superclasses.empty()) {
+    OS << "  Superclasses: (none)\n";
+    return;
+  }
+
+  OS << "  Superclasses:";
+  for (const auto &SuperclassPair : Superclasses) {
+    auto *ClassRec = SuperclassPair.first;
+    if (Rec->hasDirectSuperClass(ClassRec))
+      OS << formatv(" {0}", ClassRec->getNameInitAsString());
+    else
+      OS << formatv(" ({0})", ClassRec->getNameInitAsString());
+  }
+  OS << NL;
+}
+
+// Print the fields of a class or record, including their source locations.
+void DetailedRecordsEmitter::printFields(Record *Rec, raw_ostream &OS) {
+  const auto &ValueList = Rec->getValues();
+  if (ValueList.empty()) {
+    OS << "  Fields: (none)\n";
+    return;
+  }
+
+  OS << "  Fields:\n";
+  for (const RecordVal &Value : ValueList)
+    if (!Rec->isTemplateArg(Value.getNameInit())) {
+      OS << "    ";
+      Value.print(OS, false);
+      OS << formatv("  |{0}|\n",
+                    SrcMgr.getFormattedLocationNoOffset(Value.getLoc()));
+    }
+}
+
+namespace llvm {
+
+// This function is called by TableGen after parsing the files.
+
+void EmitDetailedRecords(RecordKeeper &RK, raw_ostream &OS) {
+  // Instantiate the emitter class and invoke run().
+  DetailedRecordsEmitter(RK).run(OS);
+}
+
+} // namespace llvm

llvm/lib/TableGen/Main.cpp

     return reportError(argv0, "Could not open input file '" + InputFilename +
                                   "': " + EC.message() + "\n");
 
+  Records.saveInputFilename(InputFilename);
+
   // Tell SrcMgr about this buffer, which is what TGParser will pick up.
   SrcMgr.AddNewSourceBuffer(std::move(*FileOrErr), SMLoc());
 

llvm/lib/TableGen/Record.cpp

   // this.  See TGParser::ParseDef and TGParser::ParseDefm.
 }
 
+// NOTE for the next two functions:
+// Superclasses are in post-order, so the final one is a direct
+// superclass. All of its transitive superclases immediately precede it,
+// so we can step through the direct superclasses in reverse order.
+
+bool Record::hasDirectSuperClass(const Record *Superclass) const {
+  ArrayRef<std::pair<Record *, SMRange>> SCs = getSuperClasses();
+
+  for (int I = SCs.size() - 1; I >= 0; --I) {
+    const Record *SC = SCs[I].first;
+    if (SC == Superclass)
+      return true;
+    I -= SC->getSuperClasses().size();
+  }
+
+  return false;
+}
+
 void Record::getDirectSuperClasses(SmallVectorImpl<Record *> &Classes) const {
   ArrayRef<std::pair<Record *, SMRange>> SCs = getSuperClasses();
 
-  // Superclasses are in post-order, so the final one is a direct
-  // superclass. All of its transitive superclases immediately precede it,
-  // so we can step through the direct superclasses in reverse order.
   while (!SCs.empty()) {
     Record *SC = SCs.back().first;
     SCs = SCs.drop_back(1 + SC->getSuperClasses().size());

llvm/lib/TableGen/TableGenBackendSkeleton.cpp

   SkeletonEmitter(RecordKeeper &RK) : Records(RK) {}
 
   void run(raw_ostream &OS);
-}; // End emitter class.
+}; // emitter class
 
-} // End anonymous namespace.
+} // anonymous namespace
 
 void SkeletonEmitter::run(raw_ostream &OS) {
   emitSourceFileHeader("Skeleton data structures", OS);

   SkeletonEmitter(RK).run(OS);
 }
 
-} // End llvm namespace.
+} // namespace llvm

llvm/utils/TableGen/TableGen.cpp

 
 enum ActionType {
   PrintRecords,
+  PrintDetailedRecords,
   DumpJSON,
   GenEmitter,
   GenRegisterInfo,

     cl::values(
         clEnumValN(PrintRecords, "print-records",
                    "Print all records to stdout (default)"),
+        clEnumValN(PrintDetailedRecords, "print-detailed-records",
+                   "Print full details of all records to stdout"),
         clEnumValN(DumpJSON, "dump-json",
                    "Dump all records as machine-readable JSON"),
         clEnumValN(GenEmitter, "gen-emitter", "Generate machine code emitter"),

   case PrintRecords:
     OS << Records;           // No argument, dump all contents
     break;
+  case PrintDetailedRecords:
+    EmitDetailedRecords(Records, OS);
+    break;
   case DumpJSON:
     EmitJSON(Records, OS);
     break;