[TableGen] Fix a couple of minor issues regarding the paste operator

[Paul-C-Anagnostopoulos]

Summary

This patch fixes a couple of minor problems with the paste (#) operator. It adds an assertion about the mode in which the operands are parsed. And it corrects an error message that was worded backward.

Most importantly, this patch updates the Programmer's Reference to fully describe and illustrate the paste operator.

[lattner]

very nice Paul, you're a machine :)

[Paul-C-Anagnostopoulos]

Semi-retirement affords one the time to work on something fun and educational.

[MaskRay]

Hi Paul, thanks for working on TableGen documentation. You can install sphinx-build, configure llvm-project with -DLLVM_ENABLE_SPHINX=on, and use docs-llvm-html to test whether the documentation builds. I just fixed some problems in d2c45f66204552177b164095501d560cd950a690 There is also a build bot (http://lab.llvm.org:8011/#/builders/30)

[MaskRay]

llvm/docs/TableGen/ProgRef.rst

16

I am not an expert of sphinx but I suspect this kind of recursive toctree is not supported.

[Paul-C-Anagnostopoulos]

Thank you for fixing those problems. I do build the documentation files with sphinx, but in a separate tree that doesn't match the actual tree.

Could you explain what I need to do to build the documentation in the real build? I'm not sure what "install sphinx-build" means, nor what "use docs-llvm-html" means. Treat me like a system build newbie.

[Paul-C-Anagnostopoulos]

Never mind, I found docs/REAME.txt. I should have read that weeks ago.

Thanks again for the fix and now I will check all the toctrees.

llvm/docs/TableGen/ProgRef.rst

 ===============================
 TableGen Programmer's Reference
 ===============================
 
 .. sectnum::
 
 .. contents::
    :local:
 
+.. toctree::
+   :hidden:
+
+   BackEnds
+   BackGuide
+   Index
+   ../CommandGuide/tblgen
+
 Introduction
 ============
 
 The purpose of TableGen is to generate complex output files based on
 information from source files that are significantly easier to code than the
 output files would be, and also easier to maintain and modify over time. The
 information is coded in a declarative style involving classes and records,
 which are then processed by TableGen. The internalized records are passed on

 records may remain uninitialized.
 
 TableGen provides *multiclasses* to collect a group of record definitions in
 one place. A multiclass is a sort of macro that can be "invoked" to define
 multiple concrete records all at once. A multiclass can inherit from other
 multiclasses, which means that the multiclass inherits all the definitions
 from its parent multiclasses.
 
-`Appendix B: Sample Record`_ illustrates a complex record in the Intel X86
+`Appendix C: Sample Record`_ illustrates a complex record in the Intel X86
 target and the simple way in which it is defined.
 
 Source Files
 ============
 
 TableGen source files are plain ASCII text files. The files can contain
 statements, comments, and blank lines (see `Lexical Analysis`_). The standard file
 extension for TableGen files is ``.td``.

 
 
 Values and Expressions
 ======================
 
 There are many contexts in TableGen statements where a value is required. A
 common example is in the definition of a record, where each field is
 specified by a name and an optional value. TableGen allows for a reasonable
-number of different forms when building up values. These forms allow the
-TableGen file to be written in a syntax that is natural for the application.
+number of different forms when building up value expressions. These forms
+allow the TableGen file to be written in a syntax that is natural for the
+application.
 
 Note that all of the values have rules for converting them from one type to
 another. For example, these rules allow you to assign a value like ``7``
 to an entity of type ``bits<4>``.
 
 .. productionlist::
    Value: `SimpleValue` `ValueSuffix`*
+        :| `Value` "#" `Value`
    ValueSuffix: "{" `RangeList` "}"
               :| "[" `RangeList` "]"
               :| "." `TokIdentifier`
    RangeList: `RangePiece` ("," `RangePiece`)*
    RangePiece: `TokInteger`
              :| `TokInteger` "..." `TokInteger`
              :| `TokInteger` "-" `TokInteger`
              :| `TokInteger` `TokInteger`
*value*\ ``[4...7,17,2...3,4]``
     the brackets). The
     new list contains elements 4, 5, 6, 7, 17, 2, 3, and 4. Elements may be
     included multiple times and in any order.
 
 *value*\ ``.`` *field*
     The final value is the value of the specified *field* in the specified
     record *value*.
 
+The paste operator
+------------------
+
+The paste operator (``#``) is the only infix operator availabe in TableGen
+expressions. It allows you to concatenate strings or lists, but has a few
+unusual features.
+
+The paste operator can be used when specifying the record name in a
+:token:`Def` or :token:`Defm` statement, in which case it must construct a
+string. If an operand is an undefined name (:token:`TokIdentifier`) or the
+name of a global :token:`Defvar` or :token:`Defset`, it is treated as a
+verbatim string of characters. The value of a global name is not used.
+
+The paste operator can be used in all other value expressions, in which case
+it can construct a string or a list. Rather oddly, but consistent with the
+previous case, if the *right-hand-side* operand is an undefined name or a
+global name, it is treated as a verbatim string of characters. The
+left-hand-side operand is treated normally.
+
+`Appendix B: Paste Operator Examples`_ presents examples of the behavior of
+the paste operator.
+
 Statements
 ==========
 
 The following statements may appear at the top level of TableGen source
 files.
 
 .. productionlist::
    TableGenFile: `Statement`*

 
 ``def`` --- define a concrete record
 ------------------------------------
 
 A ``def`` statement defines a new concrete record.
 
 .. productionlist::
    Def: "def" [`NameValue`] `RecordBody`
-   NameValue: `Value`
+   NameValue: `Value` (parsed in a special manner)
 
 The name value is optional. If specified, it is parsed in a special mode
 where undefined (unrecognized) identifiers are interpreted as literal
-strings.  In particular, global identifiers are considered unrecognized.
+strings. In particular, global identifiers are considered unrecognized.
 These include global variables defined by ``defvar`` and ``defset``.
 
 If no name value is given, the record is *anonymous*. The final name of an
 anonymous record is unspecified but globally unique.
 
 Special handling occurs if a ``def`` appears inside a ``multiclass``
 statement. See the ``multiclass`` section below for details.
 
``!srl(``\ *a*\ ``,`` *count*\ ``)``
     This operator shifts *a* right logically by *count* bits and produces the resulting
     value. The operation is performed on a 64-bit integer; the result
     is undefined for shift counts outside 0...63.
 
 ``!strconcat(``\ *str1*\ ``,`` *str2*\ ``, ...)``
     This operator concatenates the string arguments *str1*, *str2*, etc., and
     produces the resulting string.
 
-*str1*\ ``#``\ *str2*
-    The paste operator (``#``) is a shorthand for
-    ``!strconcat`` with two arguments.  It can be used to concatenate operands that
-    are not strings, in which
-    case an implicit ``!cast<string>`` is done on those operands.
-
 ``!sub(``\ *a*\ ``,`` *b*\ ``)``
     This operator subtracts *b* from *a* and produces the arithmetic difference.
 
 ``!subst(``\ *target*\ ``,`` *repl*\ ``,`` *value*\ ``)``
     This operator replaces all occurrences of the *target* in the *value* with
     the *repl* and produces the resulting value. The *value* can
     be a string, in which case substring substitution is performed.
 
     The *value* can be a record name, in which case the operator produces the *repl*
     record if the *target* record name equals the *value* record name; otherwise it
     produces the *value*.
 
 ``!tail(``\ *a*\ ``)``
     This operator produces a new list with all the elements
     of the list *a* except for the zeroth one. (See also ``!head``.)
 
 ``!xor(``\ *a*\ ``,`` *b*\ ``, ...)``
     This operator does a bitwise EXCLUSIVE OR on *a*, *b*, etc., and produces
     the result. A logical XOR can be performed if all the arguments are either
     0 or 1.
 
+Appendix B: Paste Operator Examples
+===================================
+
+Here is an example illustrating the use of the paste operator in record names.
+
+.. code-block:: text
+
+  defvar suffix = "_suffstring";
+  defvar some_ints = [0, 1, 2, 3];
+
+  def name # suffix {
+  }
+
+  foreach i = [1, 2] in {
+  def rec # i {
+  }
+  }
+
+The first ``def`` does not use the value of the ``suffix`` variable. The
+second def does use the value of the ``i`` iterator variable, because it is not a
+global name. The following records are produced.
+
+.. code-block:: text
+
+  def namesuffix {
+  }
+  def rec1 {
+  }
+  def rec2 {
+  }
+
+Here is a second example illustrating the paste operator in field value expressions.
+
+.. code-block:: text
+
+  def test {
+    string strings = suffix # suffix;
+    list<int> integers = some_ints # [4, 5, 6];
+  }
+
+The ``strings`` field expression uses ``suffix`` on both sides of the paste
+operator. It is evaluated normally on the left hand side, but taken verbatim
+on the right hand side. The ``integers`` field expression uses the value of
+the ``some_ints`` variable and a literal list. The following record is
+produced.
+
+.. code-block:: text
+
+  def test {
+    string strings = "_suffstringsuffix";
+    list<int> ints = [0, 1, 2, 3, 4, 5, 6];
+  }
+
 
-Appendix B: Sample Record
+Appendix C: Sample Record
 =========================
 
 One target machine supported by LLVM is the Intel x86. The following output
 from TableGen shows the record that is created to represent the 32-bit
 register-to-register ADD instruction.
 
 .. code-block:: text
 

llvm/lib/TableGen/TGParser.cpp

case tgtok::l_square: {          // Value ::= '[' ValueList ']'
     SmallVector<Init*, 16> Vals;
 
     RecTy *DeducedEltTy = nullptr;
     ListRecTy *GivenListTy = nullptr;
 
     if (ItemType) {
       ListRecTy *ListType = dyn_cast<ListRecTy>(ItemType);
       if (!ListType) {
-        TokError(Twine("Type mismatch for list, expected list type, got ") +
+        TokError(Twine("Encountered a list when expecting a ") +
                  ItemType->getAsString());
         return nullptr;
       }
       GivenListTy = ListType;
     }
 
     if (Lex.getCode() != tgtok::r_square) {
       ParseValueList(Vals, CurRec, nullptr,
case tgtok::paste:
         Error(PasteLoc, "LHS of paste is not typed!");
         return nullptr;
       }
 
       // Check if it's a 'listA # listB'
       if (isa<ListRecTy>(LHS->getType())) {
         Lex.Lex();  // Eat the '#'.
 
+        assert(Mode == ParseValueMode && "encountered paste of lists in name");
+
         switch (Lex.getCode()) {
         case tgtok::colon:
         case tgtok::semi:
         case tgtok::l_brace:
           Result = LHS; // trailing paste, ignore.
           break;
         default:
-          Init *RHSResult = ParseValue(CurRec, ItemType, ParseNameMode);
+          Init *RHSResult = ParseValue(CurRec, ItemType, ParseValueMode);
           if (!RHSResult)
             return nullptr;
           Result = BinOpInit::getListConcat(LHS, RHSResult);
         }
         break;
       }
 
       // Create a !strconcat() operation, first casting each operand to