[TableGen] X86 mnemonic tables backend

[Amir]

Summary

Add tablegen backend that generates X86 mnemonic-based opcode groupings, e.g.
isADD, isTEST, etc.

Addresses https://lists.llvm.org/pipermail/llvm-dev/2022-January/154526.html

[craig.topper]

I was looking at the code in replaceRegWithReg where isAND, isCMP, etc. are used. I don't understand this loop.

for (unsigned Index = InstDesc.getNumDefs() + (HasLHS ? 1 : 0),
              E = InstDesc.getNumOperands();
     Index != E; ++Index) {

HasLHS is set for all AND, ADD, SUB instructions, but some of them are RMW memory operations where getNumDefs() == 0. And there are 5 source address operands. The HasLHS will be 1 and skip the first of those 5 operands. Is that correct?

[Amir]

In D121571#3378638, @craig.topper wrote:

I was looking at the code in replaceRegWithReg where isAND, isCMP, etc. are used. I don't understand this loop.

for (unsigned Index = InstDesc.getNumDefs() + (HasLHS ? 1 : 0),
              E = InstDesc.getNumOperands();
     Index != E; ++Index) {

HasLHS is set for all AND, ADD, SUB instructions, but some of them are RMW memory operations where getNumDefs() == 0. And there are 5 source address operands. The HasLHS will be 1 and skip the first of those 5 operands. Is that correct?

Thanks for flagging, it doesn't seem right. The code may have borrowed from a different function (replaceMemOperandWithImm) where the only opcode group having RMW semantics was TESTmr and it was handled separately. For the purposes of replaceRegWithReg, HasLHS logic is not applicable to RMW instructions. I'll address that separately.

[skan]

llvm/utils/TableGen/X86MnemonicTables.cpp

53–59

Use StringRef::take_until

[skan]

llvm/utils/TableGen/X86MnemonicTables.cpp

45

Need to check the Form is not neither Pseudo or PrefixByte.

48

Need to check ForceDisassemble is not true for isCodeGenOnly

74

Avoid using switch clause when there is only one instruction.

[Amir]

Feedback

[Amir]

llvm/utils/TableGen/X86MnemonicTables.cpp

45

I decided to reuse X86Disassembler::RecognizableInstr as it parses almost all the necessary fields, specifically Form. Otherwise I'd have to copy/paste byteFromBitsInit Record field parsing function which I didn't want.

[skan]

llvm/utils/TableGen/X86MnemonicTables.cpp

70

I'm afraid using StringRef here is not safe, AsmString is destoryed after each iteration.

[Amir]

Handle CC in mnemonic printing

[Amir]

Handle CC in mnemonic printing

[Amir]

llvm/utils/TableGen/X86MnemonicTables.cpp

70

You're right. Moved the CC printing down to the emission steps.

[skan]

llvm/utils/TableGen/X86MnemonicTables.cpp

73

It's not about CC.

std::string AsmString = I->FlattenAsmStringVariants(I->AsmString, Variant);

AsmString is contructed in each iteration and is destroyed at the end of the iteration, Mnemoic references a substring of it while you use it outside the loop.
I'm afraid it's not safe. Correct me if I am wrong.

[Amir]

Store a string in the map

[Amir]

llvm/utils/TableGen/X86MnemonicTables.cpp

73

Now I understand the issue. I think I've addressed the problem by storing a std::string as a key in the new revision.

[skan]

Thanks! LGTM generally.

llvm/utils/TableGen/X86MnemonicTables.cpp

74

We could use StringRef::contains, but it's a minor issue...

[Amir]

Reuse Mnemonic.find("${cond}")

[Amir]

llvm/utils/TableGen/X86MnemonicTables.cpp

74

The result of find is used later to extract the mnemonic part, so I'm keeping it.

[Amir]

Replace $cond in-place

[zero9178]

Hello!

An LLVM build of mine that has UBsan enabled has started failing since this commit. The error itself is:

[5/1719] Building X86GenMnemonicTables.inc...
FAILED: lib/Target/X86/X86GenMnemonicTables.inc
cd /mnt/c/LLVM/llvm-project/llvm/build-wsld && /mnt/c/LLVM/llvm-project/llvm/build-wsld/bin/llvm-tblgen -gen-x86-mnemonic-tables -asmwriternum=1 -I /mnt/c/LLVM/llvm-project/llvm/lib/Target/X86 -I/mnt/c/LLVM/llvm-project/llvm/build-wsld/include -I/mnt/c/LLVM/llvm-project/llvm/include -I /mnt/c/LLVM/llvm-project/llvm/lib/Target /mnt/c/LLVM/llvm-project/llvm/lib/Target/X86/X86.td --write-if-changed -o lib/Target/X86/X86GenMnemonicTables.inc -d lib/Target/X86/X86GenMnemonicTables.inc.d
../utils/TableGen/X86MnemonicTables.cpp:52:29: runtime error: load of value 96, which is not a valid value for type 'bool'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../utils/TableGen/X86MnemonicTables.cpp:52:29 in

It seems like at X86MnemonicTables.cpp:52 you are accessing the IsCodeGenOnly field of RI, yet the constructor in X86RecognizableInstr.cpp:95 only initializes that field conditionally (based on the early return at line 75).
Would you mind taking a look? I'd suggest initializing those bool fields to some sane defaults or at the very least make the access in X86MnemonicTables.cpp:52 conditional.

For the sake of completion: These were the build options I used:

cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/mnt/c/Libraries/WSL/LLVMd -DLLVM_ENABLE_PROJECTS="mlir;lld" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_USE_SANITIZER="Address;Undefined" -DLLVM_BUILD_TOOLS=OFF -DLLVM_BUILD_UTILS=ON -DLLVM_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=clang-12 -DCMAKE_CXX_COMPILER=clang++-12 -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_INSTALL_UTILS=ON -DLLVM_FORCE_ENABLE_STATS=ON -DLLVM_ENABLE_LLD=ON

Thanks

[Amir]

In D121571#3396209, @zero9178 wrote:

Hello!

An LLVM build of mine that has UBsan enabled has started failing since this commit. The error itself is:

[5/1719] Building X86GenMnemonicTables.inc...
FAILED: lib/Target/X86/X86GenMnemonicTables.inc
cd /mnt/c/LLVM/llvm-project/llvm/build-wsld && /mnt/c/LLVM/llvm-project/llvm/build-wsld/bin/llvm-tblgen -gen-x86-mnemonic-tables -asmwriternum=1 -I /mnt/c/LLVM/llvm-project/llvm/lib/Target/X86 -I/mnt/c/LLVM/llvm-project/llvm/build-wsld/include -I/mnt/c/LLVM/llvm-project/llvm/include -I /mnt/c/LLVM/llvm-project/llvm/lib/Target /mnt/c/LLVM/llvm-project/llvm/lib/Target/X86/X86.td --write-if-changed -o lib/Target/X86/X86GenMnemonicTables.inc -d lib/Target/X86/X86GenMnemonicTables.inc.d
../utils/TableGen/X86MnemonicTables.cpp:52:29: runtime error: load of value 96, which is not a valid value for type 'bool'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../utils/TableGen/X86MnemonicTables.cpp:52:29 in

It seems like at X86MnemonicTables.cpp:52 you are accessing the IsCodeGenOnly field of RI, yet the constructor in X86RecognizableInstr.cpp:95 only initializes that field conditionally (based on the early return at line 75).
Would you mind taking a look? I'd suggest initializing those bool fields to some sane defaults or at the very least make the access in X86MnemonicTables.cpp:52 conditional.

For the sake of completion: These were the build options I used:

cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/mnt/c/Libraries/WSL/LLVMd -DLLVM_ENABLE_PROJECTS="mlir;lld" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_USE_SANITIZER="Address;Undefined" -DLLVM_BUILD_TOOLS=OFF -DLLVM_BUILD_UTILS=ON -DLLVM_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=clang-12 -DCMAKE_CXX_COMPILER=clang++-12 -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_INSTALL_UTILS=ON -DLLVM_FORCE_ENABLE_STATS=ON -DLLVM_ENABLE_LLD=ON

Thanks

Thanks for reporting! I'm working on it.

[Amir]

D122140

llvm/utils/TableGen/CMakeLists.txt

add_tablegen(llvm-tblgen LLVM
   SubtargetEmitter.cpp
   SubtargetFeatureInfo.cpp
   TableGen.cpp
   Types.cpp
   VarLenCodeEmitterGen.cpp
   X86DisassemblerTables.cpp
   X86EVEX2VEXTablesEmitter.cpp
   X86FoldTablesEmitter.cpp
+  X86MnemonicTables.cpp
   X86ModRMFilters.cpp
   X86RecognizableInstr.cpp
   WebAssemblyDisassemblerEmitter.cpp
   CTagsEmitter.cpp
   )
 target_link_libraries(llvm-tblgen PRIVATE LLVMTableGenGlobalISel)
 set_target_properties(llvm-tblgen PROPERTIES FOLDER "Tablegenning")

llvm/utils/TableGen/TableGen.cpp

enum ActionType {
   GenOptRST,
   GenCTags,
   GenAttributes,
   GenSearchableTables,
   GenGlobalISel,
   GenGICombiner,
   GenX86EVEX2VEXTables,
   GenX86FoldTables,
+  GenX86MnemonicTables,
   GenRegisterBank,
   GenExegesis,
   GenAutomata,
   GenDirectivesEnumDecl,
   GenDirectivesEnumImpl,
 };
 
 namespace llvm {
cl::values(
         clEnumValN(GenGlobalISel, "gen-global-isel",
                    "Generate GlobalISel selector"),
         clEnumValN(GenGICombiner, "gen-global-isel-combiner",
                    "Generate GlobalISel combiner"),
         clEnumValN(GenX86EVEX2VEXTables, "gen-x86-EVEX2VEX-tables",
                    "Generate X86 EVEX to VEX compress tables"),
         clEnumValN(GenX86FoldTables, "gen-x86-fold-tables",
                    "Generate X86 fold tables"),
+        clEnumValN(GenX86MnemonicTables, "gen-x86-mnemonic-tables",
+                   "Generate X86 mnemonic tables"),
         clEnumValN(GenRegisterBank, "gen-register-bank",
                    "Generate registers bank descriptions"),
         clEnumValN(GenExegesis, "gen-exegesis",
                    "Generate llvm-exegesis tables"),
         clEnumValN(GenAutomata, "gen-automata", "Generate generic automata"),
         clEnumValN(GenDirectivesEnumDecl, "gen-directive-decl",
                    "Generate directive related declaration code (header file)"),
         clEnumValN(GenDirectivesEnumImpl, "gen-directive-impl",
case GenGICombiner:
     EmitGICombiner(Records, OS);
     break;
   case GenRegisterBank:
     EmitRegisterBank(Records, OS);
     break;
   case GenX86EVEX2VEXTables:
     EmitX86EVEX2VEXTables(Records, OS);
     break;
+  case GenX86MnemonicTables:
+    EmitX86MnemonicTables(Records, OS);
+    break;
   case GenX86FoldTables:
     EmitX86FoldTables(Records, OS);
     break;
   case GenExegesis:
     EmitExegesis(Records, OS);
     break;
   case GenAutomata:
     EmitAutomata(Records, OS);

llvm/utils/TableGen/TableGenBackends.h

 void EmitOptRST(RecordKeeper &RK, raw_ostream &OS);
 void EmitCTags(RecordKeeper &RK, raw_ostream &OS);
 void EmitAttributes(RecordKeeper &RK, raw_ostream &OS);
 void EmitSearchableTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS);
 void EmitGICombiner(RecordKeeper &RK, raw_ostream &OS);
 void EmitX86EVEX2VEXTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &OS);
+void EmitX86MnemonicTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitRegisterBank(RecordKeeper &RK, raw_ostream &OS);
 void EmitExegesis(RecordKeeper &RK, raw_ostream &OS);
 void EmitAutomata(RecordKeeper &RK, raw_ostream &OS);
 void EmitDirectivesDecl(RecordKeeper &RK, raw_ostream &OS);
 void EmitDirectivesImpl(RecordKeeper &RK, raw_ostream &OS);
 
 } // End llvm namespace
 
 #endif

llvm/utils/TableGen/X86MnemonicTables.cpp

+//==- X86MnemonicTables.cpp - Generate mnemonic extraction tables. -*- C++ -*-//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This tablegen backend is responsible for emitting tables that group
+// instructions by their mnemonic name wrt AsmWriter Variant (e.g. isADD, etc).
+//
+//===----------------------------------------------------------------------===//
+
+#include "CodeGenInstruction.h"
+#include "CodeGenTarget.h"
+#include "X86DisassemblerTables.h"
+#include "X86RecognizableInstr.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/TableGenBackend.h"
+
+using namespace llvm;
+
+namespace {
+
+class X86MnemonicTablesEmitter {
+  CodeGenTarget Target;
+
+public:
+  X86MnemonicTablesEmitter(RecordKeeper &R) : Target(R) {}
+
+  // Output X86 mnemonic tables.
+  void run(raw_ostream &OS);
+};
+
+void X86MnemonicTablesEmitter::run(raw_ostream &OS) {
+  emitSourceFileHeader("X86 Mnemonic tables", OS);
+  OS << "namespace llvm {\nnamespace X86 {\n\n";
+  Record *AsmWriter = Target.getAsmWriter();
+  unsigned Variant = AsmWriter->getValueAsInt("Variant");
+
+  // Hold all instructions grouped by mnemonic
+  StringMap<SmallVector<const CodeGenInstruction *, 0>> MnemonicToCGInstrMap;
+
+  // Unused
+  X86Disassembler::DisassemblerTables Tables;
+  ArrayRef<const CodeGenInstruction *> NumberedInstructions =
+      Target.getInstructionsByEnumValue();
+  for (unsigned II = 0, IE = NumberedInstructions.size(); II != IE; ++II) {
+    const CodeGenInstruction *I = NumberedInstructions[II];
+    X86Disassembler::RecognizableInstr RI(Tables, *I, II);
+    Record *Def = I->TheDef;
+    bool IsCodeGenOnly = RI.IsCodeGenOnly;
+    bool ForceDisassemble = RI.ForceDisassemble;
+    uint8_t Form = RI.Form;
+    if ( // Filter non-X86 instructions
+        !Def->isSubClassOf("X86Inst") ||
+        // Skip pseudo instructions as they may contain non-alnum characters in
+        // mnemonic
+        (IsCodeGenOnly && !ForceDisassemble) ||
+        // Non-parsable instruction defs contain prefix as part of AsmString
+        Def->getValueAsString("AsmVariantName") == "NonParsable" ||
+        // Skip CodeGenInstructions that are not real standalone instructions
+        Form == X86Local::PrefixByte || Form == X86Local::Pseudo)
+      continue;
+    // Flatten an instruction assembly string.
+    std::string AsmString = I->FlattenAsmStringVariants(I->AsmString, Variant);
+    StringRef Mnemonic(AsmString);
+    // Extract a mnemonic assuming it's separated by \t
+    Mnemonic = Mnemonic.take_until([](char C) { return C == '\t'; });
+
+    // Special case: CMOVCC, JCC, SETCC have "${cond}" in mnemonic.
+    // Replace it with "CC" in-place.
+    size_t CondPos = Mnemonic.find("${cond}");
+    if (CondPos != StringRef::npos)
+      Mnemonic = AsmString.replace(CondPos, StringRef::npos, "CC");
+
+    // It's intentional that we put a std::string to the map (StringRef::upper
+    // returns a string) as AsmString is deallocated at the end of the iteration
+    MnemonicToCGInstrMap[Mnemonic.upper()].push_back(I);
+  }
+
+  OS << "#ifdef GET_X86_MNEMONIC_TABLES_H\n";
+  OS << "#undef GET_X86_MNEMONIC_TABLES_H\n\n";
+  for (StringRef Mnemonic : MnemonicToCGInstrMap.keys())
+    OS << "bool is" << Mnemonic << "(unsigned Opcode);\n";
+  OS << "#endif // GET_X86_MNEMONIC_TABLES_H\n\n";
+
+  OS << "#ifdef GET_X86_MNEMONIC_TABLES_CPP\n";
+  OS << "#undef GET_X86_MNEMONIC_TABLES_CPP\n\n";
+  for (StringRef Mnemonic : MnemonicToCGInstrMap.keys()) {
+    OS << "bool is" << Mnemonic << "(unsigned Opcode) {\n";
+    auto Mnemonics = MnemonicToCGInstrMap[Mnemonic];
+    if (Mnemonics.size() == 1) {
+      const CodeGenInstruction *CGI = *Mnemonics.begin();
+      OS << "\treturn Opcode == " << CGI->TheDef->getName() << ";\n}\n\n";
+    } else {
+      OS << "\tswitch (Opcode) {\n";
+      for (const CodeGenInstruction *CGI : Mnemonics) {
+        OS << "\tcase " << CGI->TheDef->getName() << ":\n";
+      }
+      OS << "\t\treturn true;\n\t}\n\treturn false;\n}\n\n";
+    }
+  }
+  OS << "#endif // GET_X86_MNEMONIC_TABLES_CPP\n\n";
+  OS << "} // end namespace X86\n} // end namespace llvm";
+}
+
+} // namespace
+
+namespace llvm {
+void EmitX86MnemonicTables(RecordKeeper &RK, raw_ostream &OS) {
+  X86MnemonicTablesEmitter(RK).run(OS);
+}
+} // namespace llvm

llvm/utils/TableGen/X86RecognizableInstr.h

 
 class DisassemblerTables;
 
 /// RecognizableInstr - Encapsulates all information required to decode a single
 ///   instruction, as extracted from the LLVM instruction tables.  Has methods
 ///   to interpret the information available in the LLVM tables, and to emit the
 ///   instruction into DisassemblerTables.
 class RecognizableInstr {
-private:
+public:
   /// The opcode of the instruction, as used in an MCInst
   InstrUID UID;
   /// The record from the .td files corresponding to this instruction
   const Record* Rec;
   /// The OpPrefix field from the record
   uint8_t OpPrefix;
   /// The OpMap field from the record
   uint8_t OpMap;
public:
   /// They are not one-to-one with operands listed in the MCInst; for example,
   /// memory operands expand to 5 operands in the MCInst
   const std::vector<CGIOperandList::OperandInfo>* Operands;
 
   /// The description of the instruction that is emitted into the instruction
   /// info table
   InstructionSpecifier* Spec;
 
+private:
   /// insnContext - Returns the primary context in which the instruction is
   ///   valid.
   ///
   /// @return - The context in which the instruction is valid.
   InstructionContext insnContext() const;
 
   /// typeFromString - Translates an operand type from the string provided in
   ///   the LLVM tables to an OperandType for use in the operand specifier.
private:
 
   /// emitDecodePath - Populates the proper fields in the decode tables
   ///   corresponding to the decode paths for this instruction.
   ///
   /// \param tables The DisassemblerTables to populate with the decode
   ///               decode information for the current instruction.
   void emitDecodePath(DisassemblerTables &tables) const;
 
+public:
   /// Constructor - Initializes a RecognizableInstr with the appropriate fields
   ///   from a CodeGenInstruction.
   ///
   /// \param tables The DisassemblerTables that the specifier will be added to.
   /// \param insn   The CodeGenInstruction to extract information from.
   /// \param uid    The unique ID of the current instruction.
   RecognizableInstr(DisassemblerTables &tables,
                     const CodeGenInstruction &insn,
                     InstrUID uid);
-public:
   /// processInstr - Accepts a CodeGenInstruction and loads decode information
   ///   for it into a DisassemblerTables if appropriate.
   ///
   /// \param tables The DiassemblerTables to be populated with decode
   ///               information.
   /// \param insn   The CodeGenInstruction to be used as a source for this
   ///               information.
   /// \param uid    The unique ID of the instruction.