[X86][TableGen] Recommitting the X86 memory folding tables TableGen backend while disabling it by default.

[aymanmus]

Summary

After the original commit (rL304088) was reverted, a discussion in llvm-dev was opened on 'how to accomplish this task'.
In the discussion we concluded that the best way to achieve our goal (which is to automate the folding tables and remove the manually maintained tables) is:

1. Commit the tablegen backend disabled by default.
2. Proceed with an incremental updating of the manual tables - while checking the validity of each added entry.
3. Repeat previous step until we reach a state where the generated and the manual tables are identical. Then we can safely remove the manual tables and include the generated tables instead.
4. Schedule periodical (1 week/2 weeks/1 month) runs of the pass:
   • if changes appear (new entries):
     • make sure the entries are legal
     • If they are not, mark them as illegal to folding
   • Commit the changes (if there are any).

CMake flag added for this purpose is "X86_GEN_FOLD_TABLES". Building with this flags will run the pass and emit the X86GenFoldTables.inc file under build/lib/Target/X86/ directory which is a good reference for any developer who wants to take part in the effort of completing the current folding tables.

[craig.topper]

Why can't we just generate the table by default and not include it yet? Then anyone can look at the table at any time without having to do something extra?

[craig.topper]

Why can't we just generate the table by default and not include it yet? Then anyone can look at the table at any time without having to do something extra?

[aymanmus]

Since the backend would be disabled at the end of the process, I think that it's better to keep it that way from the beginning.
The generated tables do not mean anything now and IMHO should not be visible to anyone who has no interest in contributing to this specific effort.
So I guess the best way to keep the temporary tables "hidden" is by enabling the "run-by-demand" mechanism now.

[craig.topper]

While will the backend be disabled at the end of the process? Won't be using the backend in place of the manual tables?

[aymanmus]

No it won't completely replace it. It would be used as a reference to the manual tables.
It's explained in the description section of this patch.
The decision was made after a discussion in the community's mailing list (http://lists.llvm.org/pipermail/llvm-dev/2017-July/115734.html)

[craig.topper]

Bullet 3 in this patch says “remove the manual tables” and “include the generated tables”.

[aymanmus]

@craig.topper that's right, at the end of this process both the generated and manual tables will hold the same entries. But the backend would be disabled and we would include a copy of the last generated inc file from it.
The main challenge here is how to integrate the TableGen's backend and taking advantage of its capabilities while supervising and validating each change made to them.

[craig.topper]

LGTM

[thakis]

It's been a bit over a year, what's the status here? lib/Target/X86/CMakeLists.txt still has a check for X86_GEN_FOLD_TABLES but nothing sets that from what I can tell.

[craig.topper]

You're correct, nothing sets it. I'm nervous about ever using the autogenerated table by default. I've used it a while back to copy entries into the manual tables. But I had to add more waivers and adjustments to fix bugs in the autogenerated table in the process. I don't think they are still a perfect match. I worry that if we switch to the autogenerated table and as we add new instructions in the future we can silently introduce bugs because the new table entries weren't reviewed.

[thakis]

In D38028#1324522, @craig.topper wrote:

You're correct, nothing sets it. I'm nervous about ever using the autogenerated table by default. I've used it a while back to copy entries into the manual tables. But I had to add more waivers and adjustments to fix bugs in the autogenerated table in the process. I don't think they are still a perfect match. I worry that if we switch to the autogenerated table and as we add new instructions in the future we can silently introduce bugs because the new table entries weren't reviewed.

Should we remove the autogenerated table again then?

[RKSimon]

In D38028#1324542, @thakis wrote:

In D38028#1324522, @craig.topper wrote:

You're correct, nothing sets it. I'm nervous about ever using the autogenerated table by default. I've used it a while back to copy entries into the manual tables. But I had to add more waivers and adjustments to fix bugs in the autogenerated table in the process. I don't think they are still a perfect match. I worry that if we switch to the autogenerated table and as we add new instructions in the future we can silently introduce bugs because the new table entries weren't reviewed.

Should we remove the autogenerated table again then?

It still has merit as a debugging tool to check against the manually created tables - not sure if that's a big enough reason?

[thakis]

In D38028#1324584, @RKSimon wrote:

In D38028#1324542, @thakis wrote:

In D38028#1324522, @craig.topper wrote:

You're correct, nothing sets it. I'm nervous about ever using the autogenerated table by default. I've used it a while back to copy entries into the manual tables. But I had to add more waivers and adjustments to fix bugs in the autogenerated table in the process. I don't think they are still a perfect match. I worry that if we switch to the autogenerated table and as we add new instructions in the future we can silently introduce bugs because the new table entries weren't reviewed.

Should we remove the autogenerated table again then?

It still has merit as a debugging tool to check against the manually created tables - not sure if that's a big enough reason?

Up to the folks using it :-) Out of curiosity, when you do this checking, do you manually run tablegen, or do you set this cmake var somehow?

[thakis]

Up to the folks using it :-) Out of curiosity, when you do this checking, do you manually run tablegen, or do you set this cmake var somehow?

Looks like just passing -DX86_GEN_FOLD_TABLES=ON to cmake is enough, so I'm guessing that's what you do.

But if y'all basically invoke this manually and then copy bits over, maybe you run llvm-tblgen -gen-x86-fold-tables directly? In that case, we could remove the (to me confusing) 4 lines of cmake? (I don't have a strong opinion about this; if you do invoke this via cmake I don't have a problem with keeping it in.)

[craig.topper]

In the past I just added it to my cmake invocation and since I used the same build area for a while it was just always being built. I asked in the original review why we couldn't just unconditionally generate it, but not include it so it was just always there as a reference. Not sure of the compile time cost of generating it.

lib/Target/X86/CMakeLists.txt

 set(LLVM_TARGET_DEFINITIONS X86.td)
 
 tablegen(LLVM X86GenRegisterInfo.inc -gen-register-info)
 tablegen(LLVM X86GenDisassemblerTables.inc -gen-disassembler)
 tablegen(LLVM X86GenInstrInfo.inc -gen-instr-info)
 tablegen(LLVM X86GenAsmWriter.inc -gen-asm-writer)
 tablegen(LLVM X86GenAsmWriter1.inc -gen-asm-writer -asmwriternum=1)
 tablegen(LLVM X86GenAsmMatcher.inc -gen-asm-matcher)
 tablegen(LLVM X86GenDAGISel.inc -gen-dag-isel)
 tablegen(LLVM X86GenFastISel.inc -gen-fast-isel)
 tablegen(LLVM X86GenCallingConv.inc -gen-callingconv)
 tablegen(LLVM X86GenSubtargetInfo.inc -gen-subtarget)
 tablegen(LLVM X86GenEVEX2VEXTables.inc -gen-x86-EVEX2VEX-tables)
 tablegen(LLVM X86GenRegisterBank.inc -gen-register-bank)
 tablegen(LLVM X86GenGlobalISel.inc -gen-global-isel)
 
+if (X86_GEN_FOLD_TABLES)
+  tablegen(LLVM X86GenFoldTables.inc -gen-x86-fold-tables)
+endif()
+
 add_public_tablegen_target(X86CommonTableGen)
 
 set(sources
   X86AsmPrinter.cpp
   X86CallFrameOptimization.cpp
   X86CallLowering.cpp
   X86CmovConversion.cpp
   X86ExpandPseudo.cpp

utils/TableGen/CMakeLists.txt

add_tablegen(llvm-tblgen LLVM
   RegisterInfoEmitter.cpp
   SearchableTableEmitter.cpp
   SubtargetEmitter.cpp
   SubtargetFeatureInfo.cpp
   TableGen.cpp
   Types.cpp
   X86DisassemblerTables.cpp
   X86EVEX2VEXTablesEmitter.cpp
+  X86FoldTablesEmitter.cpp
   X86ModRMFilters.cpp
   X86RecognizableInstr.cpp
   CTagsEmitter.cpp
   )

utils/TableGen/TableGen.cpp

enum ActionType {
   PrintEnums,
   PrintSets,
   GenOptParserDefs,
   GenCTags,
   GenAttributes,
   GenSearchableTables,
   GenGlobalISel,
   GenX86EVEX2VEXTables,
+  GenX86FoldTables,
   GenRegisterBank,
 };
 
 namespace {
   cl::opt<ActionType>
   Action(cl::desc("Action to perform:"),
          cl::values(clEnumValN(PrintRecords, "print-records",
                                "Print all records to stdout (default)"),
Action(cl::desc("Action to perform:"),
                     clEnumValN(GenAttributes, "gen-attrs",
                                "Generate attributes"),
                     clEnumValN(GenSearchableTables, "gen-searchable-tables",
                                "Generate generic binary-searchable table"),
                     clEnumValN(GenGlobalISel, "gen-global-isel",
                                "Generate GlobalISel selector"),
                     clEnumValN(GenX86EVEX2VEXTables, "gen-x86-EVEX2VEX-tables",
                                "Generate X86 EVEX to VEX compress tables"),
+                    clEnumValN(GenX86FoldTables, "gen-x86-fold-tables",
+                               "Generate X86 fold tables"),
                     clEnumValN(GenRegisterBank, "gen-register-bank",
                                "Generate registers bank descriptions")));
 
   cl::OptionCategory PrintEnumsCat("Options for -print-enums");
   cl::opt<std::string>
   Class("class", cl::desc("Print Enum list for this class"),
         cl::value_desc("class name"), cl::cat(PrintEnumsCat));
 
case GenGlobalISel:
     EmitGlobalISel(Records, OS);
     break;
   case GenRegisterBank:
     EmitRegisterBank(Records, OS);
     break;
   case GenX86EVEX2VEXTables:
     EmitX86EVEX2VEXTables(Records, OS);
     break;
+  case GenX86FoldTables:
+    EmitX86FoldTables(Records, OS);
+    break;
   }
 
   return false;
 }
 }
 
 int main(int argc, char **argv) {
   sys::PrintStackTraceOnErrorSignal(argv[0]);

utils/TableGen/TableGenBackends.h

 void EmitSubtarget(RecordKeeper &RK, raw_ostream &OS);
 void EmitMapTable(RecordKeeper &RK, raw_ostream &OS);
 void EmitOptParser(RecordKeeper &RK, raw_ostream &OS);
 void EmitCTags(RecordKeeper &RK, raw_ostream &OS);
 void EmitAttributes(RecordKeeper &RK, raw_ostream &OS);
 void EmitSearchableTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS);
 void EmitX86EVEX2VEXTables(RecordKeeper &RK, raw_ostream &OS);
+void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitRegisterBank(RecordKeeper &RK, raw_ostream &OS);
 
 } // End llvm namespace
 
 #endif

utils/TableGen/X86FoldTablesEmitter.cpp

+//===- utils/TableGen/X86FoldTablesEmitter.cpp - X86 backend-*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This tablegen backend is responsible for emitting the memory fold tables of
+// the X86 backend instructions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "CodeGenDAGPatterns.h"
+#include "CodeGenTarget.h"
+#include "X86RecognizableInstr.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/TableGenBackend.h"
+
+using namespace llvm;
+
+namespace {
+
+// 3 possible strategies for the unfolding flag (TB_NO_REVERSE) of the
+// manual added entries.
+enum UnfoldStrategy {
+  UNFOLD,     // Allow unfolding
+  NO_UNFOLD,  // Prevent unfolding
+  NO_STRATEGY // Make decision according to operands' sizes
+};
+
+// Represents an entry in the manual mapped instructions set.
+struct ManualMapEntry {
+  const char *RegInstStr;
+  const char *MemInstStr;
+  UnfoldStrategy Strategy;
+
+  ManualMapEntry(const char *RegInstStr, const char *MemInstStr,
+                 UnfoldStrategy Strategy = NO_STRATEGY)
+      : RegInstStr(RegInstStr), MemInstStr(MemInstStr), Strategy(Strategy) {}
+};
+
+class IsMatch;
+
+// List of instructions requiring explicitly aligned memory.
+const char *ExplicitAlign[] = {"MOVDQA",  "MOVAPS",  "MOVAPD",  "MOVNTPS",
+                               "MOVNTPD", "MOVNTDQ", "MOVNTDQA"};
+
+// List of instructions NOT requiring explicit memory alignment.
+const char *ExplicitUnalign[] = {"MOVDQU", "MOVUPS", "MOVUPD"};
+
+// For manually mapping instructions that do not match by their encoding.
+const ManualMapEntry ManualMapSet[] = {
+    { "ADD16ri_DB",       "ADD16mi",         NO_UNFOLD  },
+    { "ADD16ri8_DB",      "ADD16mi8",        NO_UNFOLD  },
+    { "ADD16rr_DB",       "ADD16mr",         NO_UNFOLD  },
+    { "ADD32ri_DB",       "ADD32mi",         NO_UNFOLD  },
+    { "ADD32ri8_DB",      "ADD32mi8",        NO_UNFOLD  },
+    { "ADD32rr_DB",       "ADD32mr",         NO_UNFOLD  },
+    { "ADD64ri32_DB",     "ADD64mi32",       NO_UNFOLD  },
+    { "ADD64ri8_DB",      "ADD64mi8",        NO_UNFOLD  },
+    { "ADD64rr_DB",       "ADD64mr",         NO_UNFOLD  },
+    { "ADD16rr_DB",       "ADD16rm",         NO_UNFOLD  },
+    { "ADD32rr_DB",       "ADD32rm",         NO_UNFOLD  },
+    { "ADD64rr_DB",       "ADD64rm",         NO_UNFOLD  },
+    { "PUSH16r",          "PUSH16rmm",       NO_UNFOLD  },
+    { "PUSH32r",          "PUSH32rmm",       NO_UNFOLD  },
+    { "PUSH64r",          "PUSH64rmm",       NO_UNFOLD  },
+    { "TAILJMPr",         "TAILJMPm",        UNFOLD },
+    { "TAILJMPr64",       "TAILJMPm64",      UNFOLD },
+    { "TAILJMPr64_REX",   "TAILJMPm64_REX",  UNFOLD },
+};
+
+
+static bool isExplicitAlign(const CodeGenInstruction *Inst) {
+  return any_of(ExplicitAlign, [Inst](const char *InstStr) {
+    return Inst->TheDef->getName().find(InstStr) != StringRef::npos;
+  });
+}
+
+static bool isExplicitUnalign(const CodeGenInstruction *Inst) {
+  return any_of(ExplicitUnalign, [Inst](const char *InstStr) {
+    return Inst->TheDef->getName().find(InstStr) != StringRef::npos;
+  });
+}
+
+class X86FoldTablesEmitter {
+  RecordKeeper &Records;
+  CodeGenTarget Target;
+
+  // Represents an entry in the folding table
+  class X86FoldTableEntry {
+    const CodeGenInstruction *RegInst;
+    const CodeGenInstruction *MemInst;
+
+  public:
+    bool CannotUnfold = false;
+    bool IsLoad = false;
+    bool IsStore = false;
+    bool IsAligned = false;
+    unsigned int Alignment = 0;
+
+    X86FoldTableEntry(const CodeGenInstruction *RegInst,
+                      const CodeGenInstruction *MemInst)
+        : RegInst(RegInst), MemInst(MemInst) {}
+
+    friend raw_ostream &operator<<(raw_ostream &OS,
+                                   const X86FoldTableEntry &E) {
+      OS << "{ X86::" << E.RegInst->TheDef->getName().str()
+         << ", X86::" << E.MemInst->TheDef->getName().str() << ", ";
+
+      if (E.IsLoad)
+        OS << "TB_FOLDED_LOAD | ";
+      if (E.IsStore)
+        OS << "TB_FOLDED_STORE | ";
+      if (E.CannotUnfold)
+        OS << "TB_NO_REVERSE | ";
+      if (E.IsAligned)
+        OS << "TB_ALIGN_" + std::to_string(E.Alignment) + " | ";
+
+      OS << "0 },\n";
+
+      return OS;
+    }
+  };
+
+  typedef std::vector<X86FoldTableEntry> FoldTable;
+  // std::vector for each folding table.
+  // Table2Addr - Holds instructions which their memory form performs load+store
+  // Table#i - Holds instructions which the their memory form perform a load OR
+  //           a store,  and their #i'th operand is folded.
+  FoldTable Table2Addr;
+  FoldTable Table0;
+  FoldTable Table1;
+  FoldTable Table2;
+  FoldTable Table3;
+  FoldTable Table4;
+
+public:
+  X86FoldTablesEmitter(RecordKeeper &R) : Records(R), Target(R) {}
+
+  // run - Generate the 6 X86 memory fold tables.
+  void run(raw_ostream &OS);
+
+private:
+  // Decides to which table to add the entry with the given instructions.
+  // S sets the strategy of adding the TB_NO_REVERSE flag.
+  void updateTables(const CodeGenInstruction *RegInstr,
+                    const CodeGenInstruction *MemInstr,
+                    const UnfoldStrategy S = NO_STRATEGY);
+
+  // Generates X86FoldTableEntry with the given instructions and fill it with
+  // the appropriate flags - then adds it to Table.
+  void addEntryWithFlags(FoldTable &Table, const CodeGenInstruction *RegInstr,
+                         const CodeGenInstruction *MemInstr,
+                         const UnfoldStrategy S, const unsigned int FoldedInd);
+
+  // Print the given table as a static const C++ array of type
+  // X86MemoryFoldTableEntry.
+  void printTable(const FoldTable &Table, std::string TableName,
+                  raw_ostream &OS) {
+    OS << "static const X86MemoryFoldTableEntry MemoryFold" << TableName
+       << "[] = {\n";
+
+    for (const X86FoldTableEntry &E : Table)
+      OS << E;
+
+    OS << "};\n";
+  }
+};
+
+// Return true if one of the instruction's operands is a RST register class
+static bool hasRSTRegClass(const CodeGenInstruction *Inst) {
+  return any_of(Inst->Operands, [](const CGIOperandList::OperandInfo &OpIn) {
+    return OpIn.Rec->getName() == "RST";
+  });
+}
+
+// Return true if one of the instruction's operands is a ptr_rc_tailcall
+static bool hasPtrTailcallRegClass(const CodeGenInstruction *Inst) {
+  return any_of(Inst->Operands, [](const CGIOperandList::OperandInfo &OpIn) {
+    return OpIn.Rec->getName() == "ptr_rc_tailcall";
+  });
+}
+
+// Calculates the integer value representing the BitsInit object
+static inline uint64_t getValueFromBitsInit(const BitsInit *B) {
+  assert(B->getNumBits() <= sizeof(uint64_t) * 8 && "BitInits' too long!");
+
+  uint64_t Value = 0;
+  for (unsigned i = 0, e = B->getNumBits(); i != e; ++i) {
+    BitInit *Bit = cast<BitInit>(B->getBit(i));
+    Value |= uint64_t(Bit->getValue()) << i;
+  }
+  return Value;
+}
+
+// Returns true if the two given BitsInits represent the same integer value
+static inline bool equalBitsInits(const BitsInit *B1, const BitsInit *B2) {
+  if (B1->getNumBits() != B2->getNumBits())
+    PrintFatalError("Comparing two BitsInits with different sizes!");
+
+  for (unsigned i = 0, e = B1->getNumBits(); i != e; ++i) {
+    BitInit *Bit1 = cast<BitInit>(B1->getBit(i));
+    BitInit *Bit2 = cast<BitInit>(B2->getBit(i));
+    if (Bit1->getValue() != Bit2->getValue())
+      return false;
+  }
+  return true;
+}
+
+// Return the size of the register operand
+static inline unsigned int getRegOperandSize(const Record *RegRec) {
+  if (RegRec->isSubClassOf("RegisterOperand"))
+    RegRec = RegRec->getValueAsDef("RegClass");
+  if (RegRec->isSubClassOf("RegisterClass"))
+    return RegRec->getValueAsListOfDefs("RegTypes")[0]->getValueAsInt("Size");
+
+  llvm_unreachable("Register operand's size not known!");
+}
+
+// Return the size of the memory operand
+static inline unsigned int
+getMemOperandSize(const Record *MemRec, const bool IntrinsicSensitive = false) {
+  if (MemRec->isSubClassOf("Operand")) {
+    // Intrinsic memory instructions use ssmem/sdmem.
+    if (IntrinsicSensitive &&
+        (MemRec->getName() == "sdmem" || MemRec->getName() == "ssmem"))
+      return 128;
+
+    StringRef Name =
+        MemRec->getValueAsDef("ParserMatchClass")->getValueAsString("Name");
+    if (Name == "Mem8")
+      return 8;
+    if (Name == "Mem16")
+      return 16;
+    if (Name == "Mem32")
+      return 32;
+    if (Name == "Mem64")
+      return 64;
+    if (Name == "Mem80")
+      return 80;
+    if (Name == "Mem128")
+      return 128;
+    if (Name == "Mem256")
+      return 256;
+    if (Name == "Mem512")
+      return 512;
+  }
+
+  llvm_unreachable("Memory operand's size not known!");
+}
+
+// Returns true if the record's list of defs includes the given def.
+static inline bool hasDefInList(const Record *Rec, const StringRef List,
+                                const StringRef Def) {
+  if (!Rec->isValueUnset(List)) {
+    return any_of(*(Rec->getValueAsListInit(List)),
+                  [Def](const Init *I) { return I->getAsString() == Def; });
+  }
+  return false;
+}
+
+// Return true if the instruction defined as a register flavor.
+static inline bool hasRegisterFormat(const Record *Inst) {
+  const BitsInit *FormBits = Inst->getValueAsBitsInit("FormBits");
+  uint64_t FormBitsNum = getValueFromBitsInit(FormBits);
+
+  // Values from X86Local namespace defined in X86RecognizableInstr.cpp
+  return FormBitsNum >= X86Local::MRMDestReg && FormBitsNum <= X86Local::MRM7r;
+}
+
+// Return true if the instruction defined as a memory flavor.
+static inline bool hasMemoryFormat(const Record *Inst) {
+  const BitsInit *FormBits = Inst->getValueAsBitsInit("FormBits");
+  uint64_t FormBitsNum = getValueFromBitsInit(FormBits);
+
+  // Values from X86Local namespace defined in X86RecognizableInstr.cpp
+  return FormBitsNum >= X86Local::MRMDestMem && FormBitsNum <= X86Local::MRM7m;
+}
+
+static inline bool isNOREXRegClass(const Record *Op) {
+  return Op->getName().find("_NOREX") != StringRef::npos;
+}
+
+static inline bool isRegisterOperand(const Record *Rec) {
+  return Rec->isSubClassOf("RegisterClass") ||
+         Rec->isSubClassOf("RegisterOperand") ||
+         Rec->isSubClassOf("PointerLikeRegClass");
+}
+
+static inline bool isMemoryOperand(const Record *Rec) {
+  return Rec->isSubClassOf("Operand") &&
+         Rec->getValueAsString("OperandType") == "OPERAND_MEMORY";
+}
+
+static inline bool isImmediateOperand(const Record *Rec) {
+  return Rec->isSubClassOf("Operand") &&
+         Rec->getValueAsString("OperandType") == "OPERAND_IMMEDIATE";
+}
+
+// Get the alternative instruction pointed by "FoldGenRegForm" field.
+static inline const CodeGenInstruction *
+getAltRegInst(const CodeGenInstruction *I, const RecordKeeper &Records,
+              const CodeGenTarget &Target) {
+
+  StringRef AltRegInstStr = I->TheDef->getValueAsString("FoldGenRegForm");
+  Record *AltRegInstRec = Records.getDef(AltRegInstStr);
+  assert(AltRegInstRec &&
+         "Alternative register form instruction def not found");
+  CodeGenInstruction &AltRegInst = Target.getInstruction(AltRegInstRec);
+  return &AltRegInst;
+}
+
+// Function object - Operator() returns true if the given VEX instruction
+// matches the EVEX instruction of this object.
+class IsMatch {
+  const CodeGenInstruction *MemInst;
+  const RecordKeeper &Records;
+
+public:
+  IsMatch(const CodeGenInstruction *Inst, const RecordKeeper &Records)
+      : MemInst(Inst), Records(Records) {}
+
+  bool operator()(const CodeGenInstruction *RegInst) {
+    Record *MemRec = MemInst->TheDef;
+    Record *RegRec = RegInst->TheDef;
+
+    // Return false if one (at least) of the encoding fields of both
+    // instructions do not match.
+    if (RegRec->getValueAsDef("OpEnc") != MemRec->getValueAsDef("OpEnc") ||
+        !equalBitsInits(RegRec->getValueAsBitsInit("Opcode"),
+                        MemRec->getValueAsBitsInit("Opcode")) ||
+        // VEX/EVEX fields
+        RegRec->getValueAsDef("OpPrefix") !=
+            MemRec->getValueAsDef("OpPrefix") ||
+        RegRec->getValueAsDef("OpMap") != MemRec->getValueAsDef("OpMap") ||
+        RegRec->getValueAsDef("OpSize") != MemRec->getValueAsDef("OpSize") ||
+        RegRec->getValueAsBit("hasVEX_4V") !=
+            MemRec->getValueAsBit("hasVEX_4V") ||
+        RegRec->getValueAsBit("hasEVEX_K") !=
+            MemRec->getValueAsBit("hasEVEX_K") ||
+        RegRec->getValueAsBit("hasEVEX_Z") !=
+            MemRec->getValueAsBit("hasEVEX_Z") ||
+        RegRec->getValueAsBit("hasEVEX_B") !=
+            MemRec->getValueAsBit("hasEVEX_B") ||
+        RegRec->getValueAsBit("hasEVEX_RC") !=
+            MemRec->getValueAsBit("hasEVEX_RC") ||
+        RegRec->getValueAsBit("hasREX_WPrefix") !=
+            MemRec->getValueAsBit("hasREX_WPrefix") ||
+        RegRec->getValueAsBit("hasLockPrefix") !=
+            MemRec->getValueAsBit("hasLockPrefix") ||
+        !equalBitsInits(RegRec->getValueAsBitsInit("EVEX_LL"),
+                        MemRec->getValueAsBitsInit("EVEX_LL")) ||
+        !equalBitsInits(RegRec->getValueAsBitsInit("VEX_WPrefix"),
+                        MemRec->getValueAsBitsInit("VEX_WPrefix")) ||
+        // Instruction's format - The register form's "Form" field should be
+        // the opposite of the memory form's "Form" field.
+        !areOppositeForms(RegRec->getValueAsBitsInit("FormBits"),
+                          MemRec->getValueAsBitsInit("FormBits")) ||
+        RegRec->getValueAsBit("isAsmParserOnly") !=
+            MemRec->getValueAsBit("isAsmParserOnly"))
+      return false;
+
+    // Make sure the sizes of the operands of both instructions suit each other.
+    // This is needed for instructions with intrinsic version (_Int).
+    // Where the only difference is the size of the operands.
+    // For example: VUCOMISDZrm and Int_VUCOMISDrm
+    // Also for instructions that their EVEX version was upgraded to work with
+    // k-registers. For example VPCMPEQBrm (xmm output register) and
+    // VPCMPEQBZ128rm (k register output register).
+    bool ArgFolded = false;
+    unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
+    unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
+    unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
+    unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
+
+    // Instructions with one output in their memory form use the memory folded
+    // operand as source and destination (Read-Modify-Write).
+    unsigned RegStartIdx =
+        (MemOutSize + 1 == RegOutSize) && (MemInSize == RegInSize) ? 1 : 0;
+
+    for (unsigned i = 0, e = MemInst->Operands.size(); i < e; i++) {
+      Record *MemOpRec = MemInst->Operands[i].Rec;
+      Record *RegOpRec = RegInst->Operands[i + RegStartIdx].Rec;
+
+      if (MemOpRec == RegOpRec)
+        continue;
+
+      if (isRegisterOperand(MemOpRec) && isRegisterOperand(RegOpRec)) {
+        if (getRegOperandSize(MemOpRec) != getRegOperandSize(RegOpRec) ||
+            isNOREXRegClass(MemOpRec) != isNOREXRegClass(RegOpRec))
+          return false;
+      } else if (isMemoryOperand(MemOpRec) && isMemoryOperand(RegOpRec)) {
+        if (getMemOperandSize(MemOpRec) != getMemOperandSize(RegOpRec))
+          return false;
+      } else if (isImmediateOperand(MemOpRec) && isImmediateOperand(RegOpRec)) {
+        if (MemOpRec->getValueAsDef("Type") != RegOpRec->getValueAsDef("Type"))
+          return false;
+      } else {
+        // Only one operand can be folded.
+        if (ArgFolded)
+          return false;
+
+        assert(isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec));
+        ArgFolded = true;
+      }
+    }
+
+    return true;
+  }
+
+private:
+  // Return true of the 2 given forms are the opposite of each other.
+  bool areOppositeForms(const BitsInit *RegFormBits,
+                        const BitsInit *MemFormBits) {
+    uint64_t MemFormNum = getValueFromBitsInit(MemFormBits);
+    uint64_t RegFormNum = getValueFromBitsInit(RegFormBits);
+
+    if ((MemFormNum == X86Local::MRM0m && RegFormNum == X86Local::MRM0r) ||
+        (MemFormNum == X86Local::MRM1m && RegFormNum == X86Local::MRM1r) ||
+        (MemFormNum == X86Local::MRM2m && RegFormNum == X86Local::MRM2r) ||
+        (MemFormNum == X86Local::MRM3m && RegFormNum == X86Local::MRM3r) ||
+        (MemFormNum == X86Local::MRM4m && RegFormNum == X86Local::MRM4r) ||
+        (MemFormNum == X86Local::MRM5m && RegFormNum == X86Local::MRM5r) ||
+        (MemFormNum == X86Local::MRM6m && RegFormNum == X86Local::MRM6r) ||
+        (MemFormNum == X86Local::MRM7m && RegFormNum == X86Local::MRM7r) ||
+        (MemFormNum == X86Local::MRMXm && RegFormNum == X86Local::MRMXr) ||
+        (MemFormNum == X86Local::MRMDestMem &&
+         RegFormNum == X86Local::MRMDestReg) ||
+        (MemFormNum == X86Local::MRMSrcMem &&
+         RegFormNum == X86Local::MRMSrcReg) ||
+        (MemFormNum == X86Local::MRMSrcMem4VOp3 &&
+         RegFormNum == X86Local::MRMSrcReg4VOp3) ||
+        (MemFormNum == X86Local::MRMSrcMemOp4 &&
+         RegFormNum == X86Local::MRMSrcRegOp4))
+      return true;
+
+    return false;
+  }
+};
+
+} // end anonymous namespace
+
+void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
+                                             const CodeGenInstruction *RegInstr,
+                                             const CodeGenInstruction *MemInstr,
+                                             const UnfoldStrategy S,
+                                             const unsigned int FoldedInd) {
+
+  X86FoldTableEntry Result = X86FoldTableEntry(RegInstr, MemInstr);
+  Record *RegRec = RegInstr->TheDef;
+  Record *MemRec = MemInstr->TheDef;
+
+  // Only table0 entries should explicitly specify a load or store flag.
+  if (&Table == &Table0) {
+    unsigned MemInOpsNum = MemRec->getValueAsDag("InOperandList")->getNumArgs();
+    unsigned RegInOpsNum = RegRec->getValueAsDag("InOperandList")->getNumArgs();
+    // If the instruction writes to the folded operand, it will appear as an
+    // output in the register form instruction and as an input in the memory
+    // form instruction.
+    // If the instruction reads from the folded operand, it well appear as in
+    // input in both forms.
+    if (MemInOpsNum == RegInOpsNum)
+      Result.IsLoad = true;
+    else
+      Result.IsStore = true;
+  }
+
+  Record *RegOpRec = RegInstr->Operands[FoldedInd].Rec;
+  Record *MemOpRec = MemInstr->Operands[FoldedInd].Rec;
+
+  // Unfolding code generates a load/store instruction according to the size of
+  // the register in the register form instruction.
+  // If the register's size is greater than the memory's operand size, do not
+  // allow unfolding.
+  if (S == UNFOLD)
+    Result.CannotUnfold = false;
+  else if (S == NO_UNFOLD)
+    Result.CannotUnfold = true;
+  else if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
+    Result.CannotUnfold = true; // S == NO_STRATEGY
+
+  uint64_t Enc = getValueFromBitsInit(RegRec->getValueAsBitsInit("OpEncBits"));
+  if (isExplicitAlign(RegInstr)) {
+    // The instruction require explicitly aligned memory.
+    BitsInit *VectSize = RegRec->getValueAsBitsInit("VectSize");
+    uint64_t Value = getValueFromBitsInit(VectSize);
+    Result.IsAligned = true;
+    Result.Alignment = Value;
+  } else if (Enc != X86Local::XOP && Enc != X86Local::VEX &&
+             Enc != X86Local::EVEX) {
+    // Instructions with VEX encoding do not require alignment.
+    if (!isExplicitUnalign(RegInstr) && getMemOperandSize(MemOpRec) > 64) {
+      // SSE packed vector instructions require a 16 byte alignment.
+      Result.IsAligned = true;
+      Result.Alignment = 16;
+    }
+  }
+
+  Table.push_back(Result);
+}
+
+void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
+                                        const CodeGenInstruction *MemInstr,
+                                        const UnfoldStrategy S) {
+
+  Record *RegRec = RegInstr->TheDef;
+  Record *MemRec = MemInstr->TheDef;
+  unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
+  unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
+  unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
+  unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
+
+  // Instructions which have the WriteRMW value (Read-Modify-Write) should be
+  // added to Table2Addr.
+  if (hasDefInList(MemRec, "SchedRW", "WriteRMW") && MemOutSize != RegOutSize &&
+      MemInSize == RegInSize) {
+    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0);
+    return;
+  }
+
+  if (MemInSize == RegInSize && MemOutSize == RegOutSize) {
+    // Load-Folding cases.
+    // If the i'th register form operand is a register and the i'th memory form
+    // operand is a memory operand, add instructions to Table#i.
+    for (unsigned i = RegOutSize, e = RegInstr->Operands.size(); i < e; i++) {
+      Record *RegOpRec = RegInstr->Operands[i].Rec;
+      Record *MemOpRec = MemInstr->Operands[i].Rec;
+      if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec)) {
+        switch (i) {
+        case 0:
+          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+          return;
+        case 1:
+          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1);
+          return;
+        case 2:
+          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2);
+          return;
+        case 3:
+          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3);
+          return;
+        case 4:
+          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4);
+          return;
+        }
+      }
+    }
+  } else if (MemInSize == RegInSize + 1 && MemOutSize + 1 == RegOutSize) {
+    // Store-Folding cases.
+    // If the memory form instruction performs performs a store, the *output*
+    // register of the register form instructions disappear and instead a
+    // memory *input* operand appears in the memory form instruction.
+    // For example:
+    //   MOVAPSrr => (outs VR128:$dst), (ins VR128:$src)
+    //   MOVAPSmr => (outs), (ins f128mem:$dst, VR128:$src)
+    Record *RegOpRec = RegInstr->Operands[RegOutSize - 1].Rec;
+    Record *MemOpRec = MemInstr->Operands[RegOutSize - 1].Rec;
+    if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec))
+      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+  }
+
+  return;
+}
+
+void X86FoldTablesEmitter::run(raw_ostream &OS) {
+  emitSourceFileHeader("X86 fold tables", OS);
+
+  // Holds all memory instructions
+  std::vector<const CodeGenInstruction *> MemInsts;
+  // Holds all register instructions - divided according to opcode.
+  std::map<uint8_t, std::vector<const CodeGenInstruction *>> RegInsts;
+
+  ArrayRef<const CodeGenInstruction *> NumberedInstructions =
+      Target.getInstructionsByEnumValue();
+
+  for (const CodeGenInstruction *Inst : NumberedInstructions) {
+    if (!Inst->TheDef->getNameInit() || !Inst->TheDef->isSubClassOf("X86Inst"))
+      continue;
+
+    const Record *Rec = Inst->TheDef;
+
+    // - Do not proceed if the instruction is marked as notMemoryFoldable.
+    // - Instructions including RST register class operands are not relevant
+    //   for memory folding (for further details check the explanation in
+    //   lib/Target/X86/X86InstrFPStack.td file).
+    // - Some instructions (listed in the manual map above) use the register
+    //   class ptr_rc_tailcall, which can be of a size 32 or 64, to ensure
+    //   safe mapping of these instruction we manually map them and exclude
+    //   them from the automation.
+    if (Rec->getValueAsBit("isMemoryFoldable") == false ||
+        hasRSTRegClass(Inst) || hasPtrTailcallRegClass(Inst))
+      continue;
+
+    // Add all the memory form instructions to MemInsts, and all the register
+    // form instructions to RegInsts[Opc], where Opc in the opcode of each
+    // instructions. this helps reducing the runtime of the backend.
+    if (hasMemoryFormat(Rec))
+      MemInsts.push_back(Inst);
+    else if (hasRegisterFormat(Rec)) {
+      uint8_t Opc = getValueFromBitsInit(Rec->getValueAsBitsInit("Opcode"));
+      RegInsts[Opc].push_back(Inst);
+    }
+  }
+
+  // For each memory form instruction, try to find its register form
+  // instruction.
+  for (const CodeGenInstruction *MemInst : MemInsts) {
+    uint8_t Opc =
+        getValueFromBitsInit(MemInst->TheDef->getValueAsBitsInit("Opcode"));
+
+    if (RegInsts.count(Opc) == 0)
+      continue;
+
+    // Two forms (memory & register) of the same instruction must have the same
+    // opcode. try matching only with register form instructions with the same
+    // opcode.
+    std::vector<const CodeGenInstruction *> &OpcRegInsts =
+        RegInsts.find(Opc)->second;
+
+    auto Match = find_if(OpcRegInsts, IsMatch(MemInst, Records));
+    if (Match != OpcRegInsts.end()) {
+      const CodeGenInstruction *RegInst = *Match;
+      // If the matched instruction has it's "FoldGenRegForm" set, map the
+      // memory form instruction to the register form instruction pointed by
+      // this field
+      if (RegInst->TheDef->isValueUnset("FoldGenRegForm")) {
+        updateTables(RegInst, MemInst);
+      } else {
+        const CodeGenInstruction *AltRegInst =
+            getAltRegInst(RegInst, Records, Target);
+        updateTables(AltRegInst, MemInst);
+      }
+      OpcRegInsts.erase(Match);
+    }
+  }
+
+  // Add the manually mapped instructions listed above.
+  for (const ManualMapEntry &Entry : ManualMapSet) {
+    Record *RegInstIter = Records.getDef(Entry.RegInstStr);
+    Record *MemInstIter = Records.getDef(Entry.MemInstStr);
+
+    updateTables(&(Target.getInstruction(RegInstIter)),
+                 &(Target.getInstruction(MemInstIter)), Entry.Strategy);
+  }
+
+  // Print all tables to raw_ostream OS.
+  printTable(Table2Addr, "Table2Addr", OS);
+  printTable(Table0, "Table0", OS);
+  printTable(Table1, "Table1", OS);
+  printTable(Table2, "Table2", OS);
+  printTable(Table3, "Table3", OS);
+  printTable(Table4, "Table4", OS);
+}
+
+namespace llvm {
+
+void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &OS) {
+  X86FoldTablesEmitter(RK).run(OS);
+}
+} // namespace llvm