[X86] Adding new LLVM TableGen backend that generates the X86 backend memory folding tables.

[aymanmus]

Summary

X86 backend holds huge tables in order to map between the register and memory forms of each instruction.
This TableGen Backend automatically generated all these tables with the appropriate flags for each entry.

[RKSimon]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

Please don't delete tests, especially if they've regressed - show the new codegen and add a todo comment

[RKSimon]

A few style comments

utils/TableGen/X86FoldTablesEmitter.cpp

69

Is there a better way to avoid having to strip the trailing " | "? If not, at least comment what this is doing.

265

Use llvm::any_of ?

275

Use llvm::any_of ?

329

if (IntrinsicSensitive &&
    (MemRec->getName() == "sdmem" || MemRec->getName() == "ssmem")) 
  return 128;

360

llvm::any_of

397

return (Op->getName().find("_NOREX") != std::string::npos);

[craig.topper]

utils/TableGen/X86FoldTablesEmitter.cpp

69

Couldn't we just always emit a 0 at the end? We do this in other generated tables already.

110

Can these be StringRef instead of std::string?

118

Why does this need to be vector? Why not a regular array or std::array?

215

This is a string copy. Did you omit an '&'?

[craig.topper]

utils/TableGen/X86FoldTablesEmitter.cpp

223

string copy

234

Don't use 'else' after 'return' per coding standards.

372

Can we extract the X86Local namespace from X86RecognizableInstr.cpp and do this by encodings rather than by string match names?

397

Isn't Op->getName() returning a StringRef? Shouldn't that be String::npos?

654

Again I'd like to see if we can use enum encodings from X86Local here.

691

Why is this uint16_t? Isn't opcode only 8-bits?

720

You use uint64_t for Opc here, but uint16_t on line 729. I think really it should be uint8_t for both right?

[aymanmus]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

The test checks that specific instructions are memory folded.
In the removed test cases, the instructions were removed from the memory folding tables so testing that these instructions are NOT folded in this test, is not the best idea.

utils/TableGen/X86FoldTablesEmitter.cpp

69

Thanks for the suggestion.

118

Defining an array as a class member requires stating it's size (ManualMapSet[18]), while std::vector doesn't. I thought this table might be dynamic and changed from time to time, so this way can be more friendly for future tuning.
What do you think?

372

This is very helpful, thanks for the suggestion.

[craig.topper]

utils/TableGen/X86FoldTablesEmitter.cpp

109

Fix indentation.

118

The number is required even if you put the array contents with it?

215

Should those be String::npos too?

[aymanmus]

utils/TableGen/X86FoldTablesEmitter.cpp

118

Yes, ManualMapSet[] =... would result in defining 0 sized array, and the compiler would give the following messages:

• warning: ISO C++ forbids zero-size array ManualMapSet
• error: too many initializers for const {anonymous}::X86FoldTablesEmitter::ManualMapEntry [0]

[RKSimon]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

OK - so just leave TODO comments for the 4 cases:

; TODO stack_fold_sqrtsd
; TODO stack_fold_sqrtsd_int
; TODO stack_fold_sqrtss
; TODO stack_fold_sqrtss_int

[aymanmus]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

Sorry if i'm being too picky about this, but I cannot understand why to add TODOs for these test cases. what should be done with them later on? what would it indicate?

[RKSimon]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

The SSE scalar math memory folding has always been tricky - I had wondered if these cases could be handled by X86InstrInfo::foldMemoryOperandCustom? The _Int variants will probably have to be treated quite differently but it'd be great to get pass-through of the upper bits from the other operand being properly supported some day.

I notice that you didn't have to alter the SSE versions, nor the rsqrt/rcp equivalents, so in the medium term I think we can get this fixed, which is why I say we keep the TODOs there to help keep track. You can strip the actual test cases, just leave the 4 TODO comments.

[filcab]

I'd prefer to have the tables, string tables, IsMatch, etc, outside of the X86FoldTablesEmitter.
Most (all?) of them don't really anything from there, and it would make the class smaller and easier to understand its workings (the tables and the small utility functions are all self-contained).

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

Adding TODO tests and comments allows anyone reading the source code/tests to notice that there's an optimization that should be done but isn't being done.
It's especially helpful in cases like this. From your comments (and other tests in this diff), it looks like we're not folding some sqrts? instructions after this patch.
We should keep these tests and add a TODO mentioning that folding this instructions is not easy to do with tables and they should be folded elsewhere.

utils/TableGen/X86FoldTablesEmitter.cpp

48

Please specify the default initialization value in the declaration of the member variables.

88

Lose the typedef, just enum UnfoldStrategy { ... };.

105

This isn't needed if you pull the IsMatch class outside of this one.

108

No need to specify the size of the array.

I know Craig mentioned StringRef (which is strictly better than string), but why not a simple const char *? For ExplicitUnalign too. These are only used as parameters to StringRef::find().

113

Array size.

118

Why? Not specifying the first (outermost) dimension is acceptable.
Example: https://godbolt.org/g/HTnCIh
Did I miss something?

160

D28744 was abandoned, and this comment made me think that we were doing something special when optimizing for size, which doesn't seem to be the case, after reading the patch.
Is this a "this should be done" comment?

204

Nit: I'd go with either just S or a full Strategy

248

Don't use inline when defining a function in a class definition.
http://llvm.org/docs/CodingStandards.html#don-t-use-inline-when-defining-a-function-in-a-class-definition

268

Same comment as above?
(Remove inline from below, too)

277

Why end the anonymous namespace here if you end up making a bunch of the following functions static inline?

281

assert(B->getNumBits() <= sizeof(uint64_t)*8);

297

You should probably use cast<BitInit>.... It doesn't seem like we should expect getBit to return something that's not a BitInit.

310

You should probably comment that something ensures size == alignment.

436

You already know the index before calling addEntryWithFlags. Maybe just pass the index along?

634

Please run clang-format on the file before committing.

714

Maybe this?

auto FoundRegInst = RegInsts.find(Opc);
if (FoundRegInst == RegInsts.end())
  continue;
std::vector<const CodeGenInstruction *> &OpcRegInsts = FoundRegInst->second;

728

Should we also do a OpcRegInsts.erase(Match); here?

735

Nit: No need for this empty line

[aymanmus]

test/CodeGen/X86/stack-folding-fp-avx1.ll

1654 ↗

(On Diff #97233)

Fair enough. Thanks for the detailed answer.

utils/TableGen/X86FoldTablesEmitter.cpp

118

In your example you defined the array to be global, in this case it works.
It doesn't work when the array is defined as a class member.
Thanks to your previous comment I moved this table outside the class and updated it accordingly.

160

Yes, some of the listed instructions have DAG patterns for memory folding which are marked with OptForSize predicate. A more "sophisticated" dealing should be added to VEX and EVEX instructions (which is detailed in the abandoned review comments).

310

Thanks for noticing, I planned to change that and somehow forgot. Updated with better approach.

714

The previous check:

if (RegInsts.count(Opc) == 0)
      continue;

makes sure RegInsts has an element with the required key.

[RKSimon]

Where did the rest of the diff go?

[aymanmus]

Correcting last update (uploaded wrong patch).
@RKSimon - Thanks

[craig.topper]

Can we try not to define so many of the methods inside the class itself? It makes for a very large class and an extra indentation level. I'm thinking if you do that most of the static methods in the class can just be static methods at file scope.

utils/TableGen/X86FoldTablesEmitter.cpp

213

This method should be static now right?

235

shouldn't this be static?

243

Shouldn't this be static?

[RKSimon]

utils/TableGen/X86FoldTablesEmitter.cpp

134

Convert to any_of ?

142

Convert to any_of ?

455

for (unsigned i = 0, e = MemInst->Operands.size(); i < e; i++) {

598

for (unsigned i = RegOutSize, e = RegInstr->Operands.size(); i < e; i++) {

[craig.topper]

utils/TableGen/X86FoldTablesEmitter.cpp

76

Can this be an array now instead of a vector?

172

Do we need the temporary string? Can we just stream as we go?

228

This is a copy, can we just do

for (const X86FoldTableEntry &E : Table)

350

Should this be X86RecognizableInstr.h?

562

Can we use the OpEncBits field and compare against the encoding from X86Local?

588

Is the best way we have to detect these instructions?

[RKSimon]

utils/TableGen/X86FoldTablesEmitter.cpp

257

Shouldn't this be dyn_cast if you want it to fail gracefully? Otherwise possibly replace that PrintFatalError with an assertion at the start of the loop?

268

Are we better off with an assertion?

271

Again, use dyn_cast and assertions?

[aymanmus]

utils/TableGen/X86FoldTablesEmitter.cpp

76

We only want to search this list for a match, so I think defining a vector and using the llvm::find function is more suitable.

257

It was a dynamic cast, but then @filcab noted that we can't expect getBit() to return something that is not a BitInit.
I personally agree with him.

268

In this context I think this behavior is more suitable. Here we always try to compare the same fields of two different records, where the types of the fields must be the same.
Trying to compare two BitsInit with different sizes (thus different types) should alert that something wrong is happening.

588

I think it is.

[RKSimon]

utils/TableGen/X86FoldTablesEmitter.cpp

257

OK - but those cast<BitInit> can't fail so the if()'s should be removed

[aymanmus]

Ping. (For dependent reviews also D32683, D32685)

[craig.topper]

LGTM

[chandlerc]

Folks, I know this has gone back and forth a few times, but I'm afraid I have to revert it again.

Ultimately, this patch does two things at once, and I think that caught a lot of people by surprise:

1. It changes *how* the fold tables are built to use tablegen
2. It changes *what* is in the fold tables

#1 should be a no-op change in terms of behavior. But #2 is a really big behavior change that has already caused us to track down three really painful miscompiles due to inappropriate folding. Essentially, the new tables aren't correct yet, and the fact that this new functionality came all at once and coupled with a more benign refactoring makes it hard to cope with.

I also have some design concerns about the mechanisms used here. The only way to fix these is to grow an ever larger table of instructions in the C++ source code for tablegen, which is (IMO) even worse than a big table of instructions in the backend. No one will even think to look here to find out that they should add instructions to this. There is no checking that you have named an instruction correctly. And it isn't even clear what the correct name is in some cases.

All of this is also being done without properly addressing the glaring TODOs in the memory folding tests in the X86 backend. For example, in test/CodeGen/X86/stack-folding-int-sse42.ll we have ; TODO: stack_fold_pextrw. This means we don't currently have any coverage of this instruction when doing folding. I think we need to have this kind of coverage before attempting to do #2, otherwise, we will run into bugs.

As it happens, I just debugged exactly this instruction. If I copy and paste the pextrd test like so:

define i16 @stack_fold_pextrw(<8 x i16> %a0) {
  ;CHECK-LABEL: stack_fold_pextrw
  ;CHECK:       pextrw $1, {{%xmm[0-9][0-9]*}}, {{-?[0-9]*}}(%rsp) {{.*#+}} 4-byte Folded Spill
  ;CHECK:       movl    {{-?[0-9]*}}(%rsp), %eax {{.*#+}} 4-byte Reload
  ; add forces execution domain
  %1 = add <8 x i16> %a0, <i16 1, i16 2, i16 3, i16 4, i16 5, i16 6, i16 7, i16 8>
  %2 = extractelement <8 x i16> %1, i32 1
  %3 = tail call <2 x i64> asm sideeffect "nop", "=x,~{rax},~{rbx},~{rcx},~{rdx},~{rsi},~{rdi},~{rbp},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15}"()
  ret i16 %2
}

It passes. Note that we do pextrw to a memory location and then reload it with movl. This will load garbage from the high 16 bits! We shouldn't be folding a 32-bit memory access into the memory operand of pextrw, but I can't even find where the pattern is that defines this lowering...

I could just blindly guess based on knowing how instructions are encoded in the X86 backend and probably disable folding for it, but it would be the third such disabling that I'm aware of. We can't just play whack-a-mole here.

So my concrete suggestions are:

1. Redesign how the folding works so that it is enabled or disabled by the instruction pattern itself rather than by a table in this file with string literals in it. Or at least by something else in the X86 tablegen files rather than in this C++ code.
2. Add the new tablegen emitter then, with just the flags in instruction patterns that generate the exact same table as was hard coded before.
3. Audit the outstanding TODOs (I'll submit the above test case though as soon as I finish reverting) and fill in the obvious ones so we have better coverage of folding.
4. Then start incrementally enabling more things with commensurate test updates as you go.

Does that make sense?

It'll probably take me an hour just to construct the revert, so shout if folks are really vehemently opposed to this. But it seems better for me to revert back to green than try to guess at how to stamp out yet another miscompile derived from this patch. =[

[chandlerc]

Reverted in r304762 and added another test case in r304763.

See the revert commit log, but I've carefully tried to preserve the added testing which shouldn't change when this comes back in, and I've also preserved as much of the cleanups to other code that also happened to touch this file.

Let me know if I can help in any way with getting this patch re-configured and back in place!

[aymanmus]

Hi @chandlerc,
Sorry for the mess caused by this patch. It indeed included many changes and a big behavior changing.
Thanks for the gentle handling in reverting and adding test for future use.
We will reconsider our approach of solving this issue taking into account the currently known and other possible (yet not known) exceptions.

utils/TableGen/X86FoldTablesEmitter.cpp

 //===- utils/TableGen/X86FoldTablesEmitter.cpp - X86 backend-*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This tablegen backend is responsible for emitting the memory fold tables of
 // the X86 backend instructions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenDAGPatterns.h"
 #include "CodeGenTarget.h"
+#include "X86RecognizableInstr.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/TableGenBackend.h"
-#include "X86RecognizableInstr.h"
 
 using namespace llvm;
 
 namespace {
 
-class X86FoldTablesEmitter {
-  RecordKeeper &Records;
-  CodeGenTarget Target;
-
-  // Represents an entry in the folding table
-  class X86FoldTableEntry {
-    const CodeGenInstruction *RegInst;
-    const CodeGenInstruction *MemInst;
-
-  public:
-    bool CannotUnfold;
-    bool IsLoad;
-    bool IsStore;
-    bool IsAligned;
-    unsigned int Alignment;
-
-    X86FoldTableEntry(const CodeGenInstruction *RegInst,
-                      const CodeGenInstruction *MemInst)
-        : RegInst(RegInst), MemInst(MemInst) {
-      CannotUnfold = false;
-      IsLoad = false;
-      IsStore = false;
-      IsAligned = false;
-      Alignment = 0;
-    }
-
-    friend raw_ostream &operator<<(raw_ostream &OS,
-                                   const X86FoldTableEntry &E) {
-      OS << "{ X86::" << E.RegInst->TheDef->getName().str()
-         << ", X86::" << E.MemInst->TheDef->getName().str() << ", ";
-
-      std::string result = "";
-
-      if (E.IsLoad)
-        result += "TB_FOLDED_LOAD | ";
-      if (E.IsStore)
-        result += "TB_FOLDED_STORE | ";
-      if (E.CannotUnfold)
-        result += "TB_NO_REVERSE | ";
-      if (E.IsAligned)
-        result += "TB_ALIGN_" + std::to_string(E.Alignment) + " | ";
-
-      result += "0";
-
-      OS << result << " },\n";
-      return OS;
-    }
-  };
-
-  typedef std::vector<X86FoldTableEntry> FoldTable;
-  // std::vector for each folding table.
-  // Table2Addr - Holds instructions which their memory form performs load+store
-  // Table#i - Holds instructions which the their memory form perform a load OR
-  //           a store,  and their #i'th operand is folded.
-  FoldTable Table2Addr;
-  FoldTable Table0;
-  FoldTable Table1;
-  FoldTable Table2;
-  FoldTable Table3;
-  FoldTable Table4;
-
-  // 3 possible strategies for the unfolding flag (TB_NO_REVERSE) of the
-  // manual added entries.
-  typedef enum {
-    UNFOLD,     // Allow unfolding
-    NO_UNFOLD,  // Prevent unfolding
-    NO_STRATEGY // Make decision according to operands' sizes
-  } UnfoldStrategy;
+enum UnfoldStrategy {
+};
 
-  // Represents an entry in the manual mapped instructions set.
-  struct ManualMapEntry {
-    const char *RegInstStr;
-    const char *MemInstStr;
-    UnfoldStrategy Strategy;
 
-    ManualMapEntry(const char *RegInstStr, const char *MemInstStr,
-                   UnfoldStrategy Strategy = NO_STRATEGY)
-        : RegInstStr(RegInstStr), MemInstStr(MemInstStr), Strategy(Strategy) {}
-  };
 
-  class IsMatch;
 
-  // List of instructions requiring explicitly aligned memory.
-  const StringRef ExplicitAlign[7] = {"MOVDQA",  "MOVAPS",  "MOVAPD",
-                                      "MOVNTPS", "MOVNTPD", "MOVNTDQ",
-                                      "MOVNTDQA"};
+const char *ExplicitAlign[] = {"MOVDQA",  "MOVAPS",  "MOVAPD",  "MOVNTPS",
+                               "MOVNTPD", "MOVNTDQ", "MOVNTDQA"};
 
-  // List of instructions NOT requiring explicit memory alignment.
-  const StringRef ExplicitUnalign[3] = {"MOVDQU", "MOVUPS", "MOVUPD"};
+const char *ExplicitUnalign[] = {"MOVDQU", "MOVUPS", "MOVUPD"};
 
-  // For manually mapping instructions that do not match by their encoding.
-  const std::vector<ManualMapEntry> ManualMapSet = {
-      { "ADD16ri_DB",       "ADD16mi",         NO_UNFOLD  },
-      { "ADD16ri8_DB",      "ADD16mi8",        NO_UNFOLD  },
-      { "ADD16rr_DB",       "ADD16mr",         NO_UNFOLD  },
-      { "ADD32ri_DB",       "ADD32mi",         NO_UNFOLD  },
-      { "ADD32ri8_DB",      "ADD32mi8",        NO_UNFOLD  },
-      { "ADD32rr_DB",       "ADD32mr",         NO_UNFOLD  },
-      { "ADD64ri32_DB",     "ADD64mi32",       NO_UNFOLD  },
-      { "ADD64ri8_DB",      "ADD64mi8",        NO_UNFOLD  },
-      { "ADD64rr_DB",       "ADD64mr",         NO_UNFOLD  },
-      { "ADD16rr_DB",       "ADD16rm",         NO_UNFOLD  },
-      { "ADD32rr_DB",       "ADD32rm",         NO_UNFOLD  },
-      { "ADD64rr_DB",       "ADD64rm",         NO_UNFOLD  },
-      { "PUSH16r",          "PUSH16rmm",       NO_UNFOLD  },
-      { "PUSH32r",          "PUSH32rmm",       NO_UNFOLD  },
-      { "PUSH64r",          "PUSH64rmm",       NO_UNFOLD  },
-      { "TAILJMPr",         "TAILJMPm",        UNFOLD },
-      { "TAILJMPr64",       "TAILJMPm64",      UNFOLD },
-      { "TAILJMPr64_REX",   "TAILJMPm64_REX",  UNFOLD },
-  };
+const ManualMapEntry ManualMapSet[] = {
 
-  // Do not add these instructions to any of the folding tables.
-  const std::vector<const char*> NoFoldSet = {
-      "TCRETURNri64",
-      "TCRETURNmi64", // Special dealing (in X86InstrCompiler.td under
-      "TCRETURNri",   // "tailcall stuff" section).
-      "TCRETURNmi"
+const std::vector<const char *> NoFoldSet = {
 
-      // Different calculations of the folded operand between
-      // memory and register forms (folding is illegal).
-      // - In their register form, the second register operand's relevant
-      //   bits are only the first 4/5/6 (depending on mode and reg size).
-      // - In their memory form, the second register operand's relevant
-      //   bits are only the first 16/32/64 (depending on mode and reg size).
-      "BT16rr",  "BT32rr",  "BT64rr",
-      "BT16mr",  "BT32mr",  "BT64mr",
-      "BTC16rr", "BTC32rr", "BTC64rr",
-      "BTC16mr", "BTC32mr", "BTC64mr",
-      "BTR16rr", "BTR32rr", "BTR64rr",
-      "BTR16mr", "BTR32mr", "BTR64mr",
-      "BTS16rr", "BTS32rr", "BTS64rr",
-      "BTS16mr", "BTS32mr", "BTS64mr",
 
-      // Memory folding is enabled only when optimizing for size by DAG
-      // patterns only. (issue detailed in D28744 review)
-      "VCVTSS2SDrm",            "VCVTSS2SDrr",
-      "VCVTSS2SDZrm",           "VCVTSS2SDZrr",
-      "VCVTSS2SDZrmk",          "VCVTSS2SDZrrk",
-      "VCVTSS2SDZrmkz",         "VCVTSS2SDZrrkz",
-      "VCVTSS2SDZrm_Int",       "VCVTSS2SDZrr_Int",
-      "VCVTSS2SDZrm_Intk",      "VCVTSS2SDZrr_Intk",
-      "VCVTSS2SDZrm_Intkz",     "VCVTSS2SDZrr_Intkz",
-      "VCVTSD2SSrm",            "VCVTSD2SSrr",
-      "VCVTSD2SSZrm",           "VCVTSD2SSZrr",
-      "VCVTSD2SSZrmk",          "VCVTSD2SSZrrk",
-      "VCVTSD2SSZrmkz",         "VCVTSD2SSZrrkz",
-      "VCVTSD2SSZrm_Int",       "VCVTSD2SSZrr_Int",
-      "VCVTSD2SSZrm_Intk",      "VCVTSD2SSZrr_Intk",
-      "VCVTSD2SSZrm_Intkz",     "VCVTSD2SSZrr_Intkz",
-      "VRCP14SSrm",             "VRCP14SSrr",
-      "VRCP14SDrm",             "VRCP14SDrr",
-      "VRSQRT14SSrm",           "VRSQRT14SSrr",
-      "VRSQRT14SDrm",           "VRSQRT14SDrr",
-      "VSQRTSSm",               "VSQRTSSr",
-      "VSQRTSSm_Int",           "VSQRTSSr_Int",
-      "VSQRTSSZm",              "VSQRTSSZr",
-      "VSQRTSSZm_Int",          "VSQRTSSZr_Int",
-      "VSQRTSSZm_Intk",         "VSQRTSSZr_Intk",
-      "VSQRTSSZm_Intkz",        "VSQRTSSZr_Intkz",
-      "VSQRTSDm",               "VSQRTSDr",
-      "VSQRTSDm_Int",           "VSQRTSDr_Int",
-      "VSQRTSDZm",              "VSQRTSDZr",
-      "VSQRTSDZm_Int",          "VSQRTSDZr_Int",
-      "VSQRTSDZm_Intk",         "VSQRTSDZr_Intk",
-      "VSQRTSDZm_Intkz",        "VSQRTSDZr_Intkz",
-    };
 
+class X86FoldTablesEmitter {
+  RecordKeeper &Records;
+  CodeGenTarget Target;
+
+  // Represents an entry in the folding table
+  class X86FoldTableEntry {
+    const CodeGenInstruction *RegInst;
+    const CodeGenInstruction *MemInst;
+
+  public:
+    bool CannotUnfold = false;
+    bool IsLoad = false;
+    bool IsStore = false;
+    bool IsAligned = false;
+    unsigned int Alignment = 0;
+
+    X86FoldTableEntry(const CodeGenInstruction *RegInst,
+                      const CodeGenInstruction *MemInst)
+        : RegInst(RegInst), MemInst(MemInst) {}
+
+    friend raw_ostream &operator<<(raw_ostream &OS,
+                                   const X86FoldTableEntry &E) {
+      OS << "{ X86::" << E.RegInst->TheDef->getName().str()
+         << ", X86::" << E.MemInst->TheDef->getName().str() << ", ";
+
+      std::string result = "";
+
+      if (E.IsLoad)
+        result += "TB_FOLDED_LOAD | ";
+      if (E.IsStore)
+        result += "TB_FOLDED_STORE | ";
+      if (E.CannotUnfold)
+        result += "TB_NO_REVERSE | ";
+      if (E.IsAligned)
+        result += "TB_ALIGN_" + std::to_string(E.Alignment) + " | ";
+
+      result += "0";
+
+      OS << result << " },\n";
+      return OS;
+    }
+  };
+
+  typedef std::vector<X86FoldTableEntry> FoldTable;
+  // std::vector for each folding table.
+  // Table2Addr - Holds instructions which their memory form performs load+store
+  // Table#i - Holds instructions which the their memory form perform a load OR
+  //           a store,  and their #i'th operand is folded.
+  FoldTable Table2Addr;
+  FoldTable Table0;
+  FoldTable Table1;
+  FoldTable Table2;
+  FoldTable Table3;
+  FoldTable Table4;
+
 public:
   X86FoldTablesEmitter(RecordKeeper &R) : Records(R), Target(R) {}
 
   // run - Generate the 6 X86 memory fold tables.
   void run(raw_ostream &OS);
 
 private:
   // Decides to which table to add the entry with the given instructions.
-  // Strgy sets the strategy of adding the TB_NO_REVERSE flag.
+  // S sets the strategy of adding the TB_NO_REVERSE flag.
   void updateTables(const CodeGenInstruction *RegInstr,
                     const CodeGenInstruction *MemInstr,
-                    const UnfoldStrategy Strgy = NO_STRATEGY);
+                    const UnfoldStrategy S = NO_STRATEGY);
 
   // Generates X86FoldTableEntry with the given instructions and fill it with
   // the appropriate flags - then adds it to Table.
   void addEntryWithFlags(FoldTable &Table, const CodeGenInstruction *RegInstr,
                          const CodeGenInstruction *MemInstr,
-                         const UnfoldStrategy Strgy);
+                         const UnfoldStrategy S, const unsigned int FoldedInd);
 
   bool isExplicitAlign(const CodeGenInstruction *Inst) {
-    for (const StringRef &InstStr : ExplicitAlign) {
+    for (const char *InstStr : ExplicitAlign) {
       if (Inst->TheDef->getName().find(InstStr) != StringRef::npos)
         return true;
     }
     return false;
   }
 
   bool isExplicitUnalign(const CodeGenInstruction *Inst) {
-    for (const StringRef &InstStr : ExplicitUnalign) {
+    for (const char *InstStr : ExplicitUnalign) {
       if (Inst->TheDef->getName().find(InstStr) != StringRef::npos)
         return true;
     }
     return false;
   }
 
-  // Return the index of the folded operand for each table
-  unsigned int getIndexOfFoldedOperand(const FoldTable &Table) {
-    if (&Table == &Table2Addr)
-      return 0;
-    if (&Table == &Table0)
-      return 0;
-    if (&Table == &Table1)
-      return 1;
-    if (&Table == &Table2)
-      return 2;
-    if (&Table == &Table3)
-      return 3;
-    if (&Table == &Table4)
-      return 4;
-
-    llvm_unreachable("No such table!");
-  }
-
   // Print the given table as a static const C++ array of type
   // X86MemoryFoldTableEntry.
-  inline void printTable(const FoldTable &Table, std::string TableName,
-                         raw_ostream &OS) {
+  void printTable(const FoldTable &Table, std::string TableName,
     OS << "static const X86MemoryFoldTableEntry MemoryFold" << TableName
        << "[] = {\n";
 
     for (X86FoldTableEntry E : Table)
       OS << E;
 
     OS << "};\n";
   }
 
   // Return true if one of the instruction's operands is a RST register class
-  inline bool hasRSTRegClass(const CodeGenInstruction *Inst) {
-
+  bool hasRSTRegClass(const CodeGenInstruction *Inst) {
     return llvm::any_of(Inst->Operands,
                         [](const CGIOperandList::OperandInfo &OpIn) {
-                              return OpIn.Rec->getName() == "RST";
-                           });
   }
 
-  // Return true if one of the instruction's operands is a RST register class
-  inline bool hasPtrTailcallRegClass(const CodeGenInstruction *Inst) {
-
+  // Return true if one of the instruction's operands is a ptr_rc_tailcall
+  bool hasPtrTailcallRegClass(const CodeGenInstruction *Inst) {
     return llvm::any_of(Inst->Operands,
                         [](const CGIOperandList::OperandInfo &OpIn) {
-                              return OpIn.Rec->getName() == "ptr_rc_tailcall";
-                           });
   }
 };
-} // end anonymous namespace
 
 // Calculates the integer value representing the BitsInit object
 static inline uint64_t getValueFromBitsInit(const BitsInit *B) {
+  assert(B->getNumBits() <= sizeof(uint64_t) * 8 && "BitInits' too long!");
+
   uint64_t Value = 0;
   for (unsigned i = 0, e = B->getNumBits(); i != e; ++i) {
-    if (BitInit *Bit = dyn_cast<BitInit>(B->getBit(i)))
+    if (BitInit *Bit = cast<BitInit>(B->getBit(i)))
       Value |= uint64_t(Bit->getValue()) << i;
     else
       PrintFatalError("Invalid VectSize bit");
   }
   return Value;
 }
 
 // Returns true if the two given BitsInits represent the same integer value
 static inline bool equalBitsInits(const BitsInit *B1, const BitsInit *B2) {
   if (B1->getNumBits() != B2->getNumBits())
     PrintFatalError("Comparing two BitsInits with different sizes!");
 
   for (unsigned i = 0, e = B1->getNumBits(); i != e; ++i) {
-    if (BitInit *Bit1 = dyn_cast<BitInit>(B1->getBit(i))) {
-      if (BitInit *Bit2 = dyn_cast<BitInit>(B2->getBit(i))) {
+    if (BitInit *Bit1 = cast<BitInit>(B1->getBit(i))) {
+      if (BitInit *Bit2 = cast<BitInit>(B2->getBit(i))) {
         if (Bit1->getValue() != Bit2->getValue())
           return false;
       } else
         PrintFatalError("Invalid BitsInit bit");
     } else
       PrintFatalError("Invalid BitsInit bit");
   }
   return true;
 }
 
 // Return the size of the register operand
 static inline unsigned int getRegOperandSize(const Record *RegRec) {
-  if (RegRec->isSubClassOf("RegisterClass"))
-    return RegRec->getValueAsInt("Alignment");
   if (RegRec->isSubClassOf("RegisterOperand"))
-    return RegRec->getValueAsDef("RegClass")->getValueAsInt("Alignment");
+    RegRec = RegRec->getValueAsDef("RegClass");
+  if (RegRec->isSubClassOf("RegisterClass"))
+    return RegRec->getValueAsListOfDefs("RegTypes")[0]->getValueAsInt("Size");
 
   llvm_unreachable("Register operand's size not known!");
 }
 
 // Return the size of the memory operand
 static inline unsigned int
 getMemOperandSize(const Record *MemRec, const bool IntrinsicSensitive = false) {
   if (MemRec->isSubClassOf("Operand")) {
     // Intrinsic memory instructions use ssmem/sdmem.
     if (IntrinsicSensitive &&
         (MemRec->getName() == "sdmem" || MemRec->getName() == "ssmem"))
       return 128;
 
     StringRef Name =
         MemRec->getValueAsDef("ParserMatchClass")->getValueAsString("Name");
     if (Name == "Mem8")
       return 8;
     if (Name == "Mem16")
       return 16;
     if (Name == "Mem32")
       return 32;
     if (Name == "Mem64")
       return 64;
     if (Name == "Mem80")
       return 80;
     if (Name == "Mem128")
       return 128;
     if (Name == "Mem256")
       return 256;
     if (Name == "Mem512")
       return 512;
   }
 
   llvm_unreachable("Memory operand's size not known!");
 }
 
 // Returns true if the record's list of defs includes the given def.
-static bool hasDefInList(const Record *Rec, const StringRef List,
+static inline bool hasDefInList(const Record *Rec, const StringRef List,
                                 const StringRef Def) {
   if (!Rec->isValueUnset(List)) {
-    return llvm::any_of(*(Rec->getValueAsListInit(List)),
-                       [Def](const Init *I) {return I->getAsString() == Def;});
+    return llvm::any_of(*(Rec->getValueAsListInit(List)), [Def](const Init *I) {
+      return I->getAsString() == Def;
+    });
   }
   return false;
 }
 
 // Return true if the instruction defined as a register flavor.
 static inline bool hasRegisterFormat(const Record *Inst) {
   const BitsInit *FormBits = Inst->getValueAsBitsInit("FormBits");
   uint64_t FormBitsNum = getValueFromBitsInit(FormBits);
 
   // Values from X86Local namespace defined in X86RecognizableInstr.cpp
   return FormBitsNum >= X86Local::MRMDestReg && FormBitsNum <= X86Local::MRM7r;
 }
 
 // Return true if the instruction defined as a memory flavor.
 static inline bool hasMemoryFormat(const Record *Inst) {
   const BitsInit *FormBits = Inst->getValueAsBitsInit("FormBits");
   uint64_t FormBitsNum = getValueFromBitsInit(FormBits);
 
   // Values from X86Local namespace defined in X86RecognizableInstr.cpp
   return FormBitsNum >= X86Local::MRMDestMem && FormBitsNum <= X86Local::MRM7m;
 }
 
 static inline bool isNOREXRegClass(const Record *Op) {
   return Op->getName().find("_NOREX") != StringRef::npos;
 }
 
 static inline bool isRegisterOperand(const Record *Rec) {
   return Rec->isSubClassOf("RegisterClass") ||
          Rec->isSubClassOf("RegisterOperand") ||
          Rec->isSubClassOf("PointerLikeRegClass");
 }
 
 static inline bool isMemoryOperand(const Record *Rec) {
   return Rec->isSubClassOf("Operand") &&
          Rec->getValueAsString("OperandType") == "OPERAND_MEMORY";
 }
 
 static inline bool isImmediateOperand(const Record *Rec) {
   return Rec->isSubClassOf("Operand") &&
          Rec->getValueAsString("OperandType") == "OPERAND_IMMEDIATE";
 }
 
 // Get the alternative instruction pointed by "FoldGenRegForm" field.
 static inline const CodeGenInstruction *
 getAltRegInst(const CodeGenInstruction *I, const RecordKeeper &Records,
               const CodeGenTarget &Target) {
 
   StringRef AltRegInstStr = I->TheDef->getValueAsString("FoldGenRegForm");
   Record *AltRegInstRec = Records.getDef(AltRegInstStr);
   assert(AltRegInstRec &&
          "Alternative register form instruction def not found");
   CodeGenInstruction &AltRegInst = Target.getInstruction(AltRegInstRec);
   return &AltRegInst;
 }
 
-void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
-                                             const CodeGenInstruction *RegInstr,
-                                             const CodeGenInstruction *MemInstr,
-                                             const UnfoldStrategy Strgy) {
-
-  X86FoldTableEntry Result = X86FoldTableEntry(RegInstr, MemInstr);
-  Record *RegRec = RegInstr->TheDef;
-  Record *MemRec = MemInstr->TheDef;
-
-  // Only table0 entries should explicitly specify a load or store flag.
-  if (&Table == &Table0) {
-    unsigned MemInOpsNum = MemRec->getValueAsDag("InOperandList")->getNumArgs();
-    unsigned RegInOpsNum = RegRec->getValueAsDag("InOperandList")->getNumArgs();
-    // If the instruction writes to the folded operand, it will appear as an
-    // output in the register form instruction and as an input in the memory
-    // form instruction.
-    // If the instruction reads from the folded operand, it well appear as in
-    // input in both forms.
-    if (MemInOpsNum == RegInOpsNum)
-      Result.IsLoad = true;
-    else
-      Result.IsStore = true;
-  }
-
-  unsigned int FoldedInd = getIndexOfFoldedOperand(Table);
-  Record *RegOpRec = RegInstr->Operands[FoldedInd].Rec;
-  Record *MemOpRec = MemInstr->Operands[FoldedInd].Rec;
-
-  // Unfolding code generates a load/store instruction according to the size of
-  // the register in the register form instruction.
-  // If the register's size is greater than the memory's operand size, do not
-  // allow unfolding.
-  if (Strgy == UNFOLD)
-    Result.CannotUnfold = false;
-  else if (Strgy == NO_UNFOLD)
-    Result.CannotUnfold = true;
-  else if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
-    Result.CannotUnfold = true; // Strgy == NO_STRATEGY
-
-  if (isExplicitAlign(RegInstr)) {
-    // The instruction require explicitly aligned memory.
-    BitsInit *VectSize = RegRec->getValueAsBitsInit("VectSize");
-    uint64_t Value = getValueFromBitsInit(VectSize);
-    Result.IsAligned = true;
-    Result.Alignment = Value;
-  } else if (!RegRec->isSubClassOf("VEX")) {
-    // Instructions with VEX encoding do not require alignment.
-    if (!isExplicitUnalign(RegInstr) && getMemOperandSize(MemOpRec) > 64 &&
-        !RegInstr->TheDef->isSubClassOf("XOP")) {
-      // SSE packed vector instructions require a 16 byte alignment.
-      Result.IsAligned = true;
-      Result.Alignment = 16;
-    }
-  }
-
-  Table.push_back(Result);
-}
-
-void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
-                                        const CodeGenInstruction *MemInstr,
-                                        const UnfoldStrategy Strgy) {
-
-  Record *RegRec = RegInstr->TheDef;
-  Record *MemRec = MemInstr->TheDef;
-  unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
-  unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
-  unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
-  unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
-
-  // Instructions which have the WriteRMW value (Read-Modify-Write) should be
-  // added to Table2Addr.
-  if (hasDefInList(MemRec, "SchedRW", "WriteRMW") && MemOutSize != RegOutSize &&
-      MemInSize == RegInSize) {
-    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, Strgy);
-    return;
-  }
-
-  if (MemInSize == RegInSize && MemOutSize == RegOutSize) {
-    // Load-Folding cases.
-    // If the i'th register form operand is a register and the i'th memory form
-    // operand is a memory operand, add instructions to Table#i.
-    for (unsigned i = RegOutSize; i < RegInstr->Operands.size(); i++) {
-      Record *RegOpRec = RegInstr->Operands[i].Rec;
-      Record *MemOpRec = MemInstr->Operands[i].Rec;
-      if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec)) {
-        switch (i) {
-        case 0:
-          addEntryWithFlags(Table0, RegInstr, MemInstr, Strgy);
-          return;
-        case 1:
-          addEntryWithFlags(Table1, RegInstr, MemInstr, Strgy);
-          return;
-        case 2:
-          addEntryWithFlags(Table2, RegInstr, MemInstr, Strgy);
-          return;
-        case 3:
-          addEntryWithFlags(Table3, RegInstr, MemInstr, Strgy);
-          return;
-        case 4:
-          addEntryWithFlags(Table4, RegInstr, MemInstr, Strgy);
-          return;
-        }
-      }
-    }
-  } else if (MemInSize == RegInSize + 1 && MemOutSize + 1 == RegOutSize) {
-    // Store-Folding cases.
-    // If the memory form instruction performs performs a store, the *output*
-    // register of the register form instructions disappear and instead a
-    // memory *input* operand appears in the memory form instruction.
-    // For example:
-    //   MOVAPSrr => (outs VR128:$dst), (ins VR128:$src)
-    //   MOVAPSmr => (outs), (ins f128mem:$dst, VR128:$src)
-    Record *RegOpRec = RegInstr->Operands[RegOutSize - 1].Rec;
-    Record *MemOpRec = MemInstr->Operands[RegOutSize - 1].Rec;
-    if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec))
-      addEntryWithFlags(Table0, RegInstr, MemInstr, Strgy);
-  }
-
-  return;
-}
-
 // Function object - Operator() returns true if the given VEX instruction
 // matches the EVEX instruction of this object.
-class X86FoldTablesEmitter::IsMatch {
+class IsMatch {
   const CodeGenInstruction *MemInst;
   const RecordKeeper &Records;
 
 public:
   IsMatch(const CodeGenInstruction *Inst, const RecordKeeper &Records)
       : MemInst(Inst), Records(Records) {}
 
   bool operator()(const CodeGenInstruction *RegInst) {
     Record *MemRec = MemInst->TheDef;
     Record *RegRec = RegInst->TheDef;
 
     // Return false if one (at least) of the encoding fields of both
     // instructions do not match.
     if (RegRec->getValueAsDef("OpEnc") != MemRec->getValueAsDef("OpEnc") ||
         !equalBitsInits(RegRec->getValueAsBitsInit("Opcode"),
                         MemRec->getValueAsBitsInit("Opcode")) ||
if (RegRec->getValueAsDef("OpEnc") != MemRec->getValueAsDef("OpEnc") ||
                         MemRec->getValueAsBitsInit("VEX_WPrefix")) ||
         // Instruction's format - The register form's "Form" field should be
         // the opposite of the memory form's "Form" field.
         !areOppositeForms(RegRec->getValueAsBitsInit("FormBits"),
                           MemRec->getValueAsBitsInit("FormBits")) ||
         RegRec->getValueAsBit("isAsmParserOnly") !=
             MemRec->getValueAsBit("isAsmParserOnly"))
       return false;
 
     // Make sure the sizes of the operands of both instructions suit each other.
     // This is needed for instructions with intrinsic version (_Int).
     // Where the only difference is the size of the operands.
     // For example: VUCOMISDZrm and Int_VUCOMISDrm
     // Also for instructions that their EVEX version was upgraded to work with
     // k-registers. For example VPCMPEQBrm (xmm output register) and
     // VPCMPEQBZ128rm (k register output register).
     bool ArgFolded = false;
     unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
     unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
     unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
     unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
 
     // Instructions with one output in their memory form use the memory folded
     // operand as source and destination (Read-Modify-Write).
     unsigned RegStartIdx =
         (MemOutSize + 1 == RegOutSize) && (MemInSize == RegInSize) ? 1 : 0;
 
     for (unsigned i = 0; i < MemInst->Operands.size(); i++) {
       Record *MemOpRec = MemInst->Operands[i].Rec;
       Record *RegOpRec = RegInst->Operands[i + RegStartIdx].Rec;
 
       if (MemOpRec == RegOpRec)
         continue;
 
       if (isRegisterOperand(MemOpRec) && isRegisterOperand(RegOpRec)) {
         if (getRegOperandSize(MemOpRec) != getRegOperandSize(RegOpRec) ||
for (unsigned i = 0; i < MemInst->Operands.size(); i++) {
         ArgFolded = true;
       }
     }
 
     return true;
   }
 
 private:
   // Return true of the 2 given forms are the opposite of each other.
   bool areOppositeForms(const BitsInit *RegFormBits,
-                           const BitsInit *MemFormBits) {
     uint64_t MemFormNum = getValueFromBitsInit(MemFormBits);
     uint64_t RegFormNum = getValueFromBitsInit(RegFormBits);
 
     if ((MemFormNum == X86Local::MRM0m && RegFormNum == X86Local::MRM0r) ||
         (MemFormNum == X86Local::MRM1m && RegFormNum == X86Local::MRM1r) ||
         (MemFormNum == X86Local::MRM2m && RegFormNum == X86Local::MRM2r) ||
         (MemFormNum == X86Local::MRM3m && RegFormNum == X86Local::MRM3r) ||
         (MemFormNum == X86Local::MRM4m && RegFormNum == X86Local::MRM4r) ||
         (MemFormNum == X86Local::MRM5m && RegFormNum == X86Local::MRM5r) ||
         (MemFormNum == X86Local::MRM6m && RegFormNum == X86Local::MRM6r) ||
         (MemFormNum == X86Local::MRM7m && RegFormNum == X86Local::MRM7r) ||
         (MemFormNum == X86Local::MRMXm && RegFormNum == X86Local::MRMXr) ||
         (MemFormNum == X86Local::MRMDestMem &&
-            RegFormNum == X86Local::MRMDestReg) ||
         (MemFormNum == X86Local::MRMSrcMem &&
-            RegFormNum == X86Local::MRMSrcReg) ||
         (MemFormNum == X86Local::MRMSrcMem4VOp3 &&
-            RegFormNum == X86Local::MRMSrcReg4VOp3) ||
         (MemFormNum == X86Local::MRMSrcMemOp4 &&
-            RegFormNum == X86Local::MRMSrcRegOp4))
       return true;
 
     return false;
   }
 };
 
+} // end anonymous namespace
+
+void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
+                                             const CodeGenInstruction *RegInstr,
+                                             const CodeGenInstruction *MemInstr,
+                                             const UnfoldStrategy S,
+                                             const unsigned int FoldedInd) {
+
+  X86FoldTableEntry Result = X86FoldTableEntry(RegInstr, MemInstr);
+  Record *RegRec = RegInstr->TheDef;
+  Record *MemRec = MemInstr->TheDef;
+
+  // Only table0 entries should explicitly specify a load or store flag.
+  if (&Table == &Table0) {
+    unsigned MemInOpsNum = MemRec->getValueAsDag("InOperandList")->getNumArgs();
+    unsigned RegInOpsNum = RegRec->getValueAsDag("InOperandList")->getNumArgs();
+    // If the instruction writes to the folded operand, it will appear as an
+    // output in the register form instruction and as an input in the memory
+    // form instruction.
+    // If the instruction reads from the folded operand, it well appear as in
+    // input in both forms.
+    if (MemInOpsNum == RegInOpsNum)
+      Result.IsLoad = true;
+    else
+      Result.IsStore = true;
+  }
+
+  Record *RegOpRec = RegInstr->Operands[FoldedInd].Rec;
+  Record *MemOpRec = MemInstr->Operands[FoldedInd].Rec;
+
+  // Unfolding code generates a load/store instruction according to the size of
+  // the register in the register form instruction.
+  // If the register's size is greater than the memory's operand size, do not
+  // allow unfolding.
+  if (S == UNFOLD)
+    Result.CannotUnfold = false;
+  else if (S == NO_UNFOLD)
+    Result.CannotUnfold = true;
+  else if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
+    Result.CannotUnfold = true; // S == NO_STRATEGY
+
+  if (isExplicitAlign(RegInstr)) {
+    // The instruction require explicitly aligned memory.
+    BitsInit *VectSize = RegRec->getValueAsBitsInit("VectSize");
+    uint64_t Value = getValueFromBitsInit(VectSize);
+    Result.IsAligned = true;
+    Result.Alignment = Value;
+  } else if (!RegRec->isSubClassOf("VEX")) {
+    // Instructions with VEX encoding do not require alignment.
+    if (!isExplicitUnalign(RegInstr) && getMemOperandSize(MemOpRec) > 64 &&
+        !RegInstr->TheDef->isSubClassOf("XOP")) {
+      // SSE packed vector instructions require a 16 byte alignment.
+      Result.IsAligned = true;
+      Result.Alignment = 16;
+    }
+  }
+
+  Table.push_back(Result);
+}
+
+void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
+                                        const CodeGenInstruction *MemInstr,
+                                        const UnfoldStrategy S) {
+
+  Record *RegRec = RegInstr->TheDef;
+  Record *MemRec = MemInstr->TheDef;
+  unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
+  unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
+  unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
+  unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
+
+  // Instructions which have the WriteRMW value (Read-Modify-Write) should be
+  // added to Table2Addr.
+  if (hasDefInList(MemRec, "SchedRW", "WriteRMW") && MemOutSize != RegOutSize &&
+      MemInSize == RegInSize) {
+    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0);
+    return;
+  }
+
+  if (MemInSize == RegInSize && MemOutSize == RegOutSize) {
+    // Load-Folding cases.
+    // If the i'th register form operand is a register and the i'th memory form
+    // operand is a memory operand, add instructions to Table#i.
+    for (unsigned i = RegOutSize; i < RegInstr->Operands.size(); i++) {
+      Record *RegOpRec = RegInstr->Operands[i].Rec;
+      Record *MemOpRec = MemInstr->Operands[i].Rec;
+      if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec)) {
+        switch (i) {
+        case 0:
+          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+          return;
+        case 1:
+          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1);
+          return;
+        case 2:
+          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2);
+          return;
+        case 3:
+          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3);
+          return;
+        case 4:
+          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4);
+          return;
+        }
+      }
+    }
+  } else if (MemInSize == RegInSize + 1 && MemOutSize + 1 == RegOutSize) {
+    // Store-Folding cases.
+    // If the memory form instruction performs performs a store, the *output*
+    // register of the register form instructions disappear and instead a
+    // memory *input* operand appears in the memory form instruction.
+    // For example:
+    //   MOVAPSrr => (outs VR128:$dst), (ins VR128:$src)
+    //   MOVAPSmr => (outs), (ins f128mem:$dst, VR128:$src)
+    Record *RegOpRec = RegInstr->Operands[RegOutSize - 1].Rec;
+    Record *MemOpRec = MemInstr->Operands[RegOutSize - 1].Rec;
+    if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec))
+      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+  }
+
+  return;
+}
+
 void X86FoldTablesEmitter::run(raw_ostream &OS) {
   emitSourceFileHeader("X86 fold tables", OS);
 
   // Holds all memory instructions
   std::vector<const CodeGenInstruction *> MemInsts;
   // Holds all register instructions - divided according to opcode.
   std::map<uint8_t, std::vector<const CodeGenInstruction *>> RegInsts;
 
   ArrayRef<const CodeGenInstruction *> NumberedInstructions =
       Target.getInstructionsByEnumValue();
 
   for (const CodeGenInstruction *Inst : NumberedInstructions) {
     if (!Inst->TheDef->getNameInit() || !Inst->TheDef->isSubClassOf("X86Inst"))
       continue;
 
     const Record *Rec = Inst->TheDef;
 
     // - Do not proceed matching if the instruction in NoFoldSet.
     // - Instructions including RST register class operands are not relevant
     //   for memory folding (for further details check the explanation in
     //   lib/Target/X86/X86InstrFPStack.td file).
     // - Some instructions (listed in the manual map above) use the register
     //   class ptr_rc_tailcall, which can be of a size 32 or 64, to ensure
     //   safe mapping of these instruction we manually map them and exclude
     //   them from the automation.
for (const CodeGenInstruction *MemInst : MemInsts) {
 
     if (RegInsts.count(Opc) == 0)
       continue;
 
     // Two forms (memory & register) of the same instruction must have the same
     // opcode. try matching only with register form instructions with the same
     // opcode.
     std::vector<const CodeGenInstruction *> &OpcRegInsts =
         RegInsts.find(Opc)->second;
 
     auto Match = find_if(OpcRegInsts, IsMatch(MemInst, Records));
     if (Match != OpcRegInsts.end()) {
       const CodeGenInstruction *RegInst = *Match;
       // If the matched instruction has it's "FoldGenRegForm" set, map the
       // memory form instruction to the register form instruction pointed by
       // this field
       if (RegInst->TheDef->isValueUnset("FoldGenRegForm")) {
         updateTables(RegInst, MemInst);
-        OpcRegInsts.erase(Match);
       } else {
         const CodeGenInstruction *AltRegInst =
             getAltRegInst(RegInst, Records, Target);
         updateTables(AltRegInst, MemInst);
       }
+      OpcRegInsts.erase(Match);
     }
   }
 
   // Add the manually mapped instructions listed above.
   for (const ManualMapEntry &Entry : ManualMapSet) {
-
     Record *RegInstIter = Records.getDef(Entry.RegInstStr);
     Record *MemInstIter = Records.getDef(Entry.MemInstStr);
 
     updateTables(&(Target.getInstruction(RegInstIter)),
                  &(Target.getInstruction(MemInstIter)), Entry.Strategy);
   }
 
   // Print all tables to raw_ostream OS.
   printTable(Table2Addr, "Table2Addr", OS);
   printTable(Table0, "Table0", OS);
   printTable(Table1, "Table1", OS);
   printTable(Table2, "Table2", OS);
   printTable(Table3, "Table3", OS);
   printTable(Table4, "Table4", OS);
 }
 
 namespace llvm {
 
 void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &OS) {
   X86FoldTablesEmitter(RK).run(OS);
 }
 } // namespace llvm