[RFC][X86][MemFold] Upgrade the mechanism of auto-generated Memory Folding Table

[yubing]

Summary

1. Align ManualMapSet with X86MemoryFoldTableEntry instead of using UnfoldStrategy
2. ManualMapSet able to update the existing record in auto-generated MemFold table

[yubing]

PLAN:

1. Move MemoryFoldTable2Addr MemoryFoldTable0~4 into X86InstrFoldTables.def from llvm/lib/Target/X86/X86InstrFoldTables.cpp https://reviews.llvm.org/D142083
2. Update llvm/lib/Target/X86/X86InstrFoldTables.def with adding records, mainly for avx512fp16 https://reviews.llvm.org/D143149
3. Upgrade mechanism of auto-generated memfolding table https://reviews.llvm.org/D142084
4. Update ManualMapSet in X86FoldTablesEmitter.cpp to make x86-auto-memfold.td pass
5. small modification for remaining different ~30 records

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

35

Where is NO_STRATEGY define? It looks like it was deleted from the left hand of this diff.

[skan]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

63

Don't duplicate the definitions in llvm/lib/Target/X86/X86InstrFoldTables.h, try finding a common place

84

See my comments in https://reviews.llvm.org/D142083, we can change thing to print here

[yubing]

give 0 to default Strategy of ManualMapEntry

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

35

changed to 0

[skan]

llvm/test/TableGen/x86-auto-memfold.td

3

Drop "-asmwriternum=1" "-write-if-changed"

4

Drop this line by using pipleline in the previous line.

5

We should remove all the \t or space in INC file, don't use "tr" for it.

8

Drop the XFAIL, this patch should be merged after fp16 records are added.

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

47–48

Move this into a .inc as a whitelist

[pengfei]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

32

Remove.

52

1?

128

Should be a space here.

130

Add assert to make sure LHS and RHS not null?

381

How about !!(S & TB_NO_REVERSE)?

[yubing]

address all the comments

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

52

i copy the enum's definition from llvm/lib/Target/X86/X86InstrFoldTables.h

128

clang-format can't do this. i observed

[skan]

llvm/test/TableGen/x86-auto-memfold.td

3

Question: what do we skip here?

[skan]

llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

2

This comment does not applies to all the entries in the table, I think we should move it to a suitable place. in this file.
And we need a clarification for each group in this table.

[yubing]

llvm/test/TableGen/x86-auto-memfold.td

3

the first 568 bytes are:

/*===- TableGen'erated file -------------------------------------*- C++ -*-===*\
|*                                                                            *|
|* X86 fold tables                                                            *|
|*                                                                            *|
|* Automatically generated file, do not edit!                                 *|
|*                                                                            *|
\*===----------------------------------------------------------------------===*/

[skan]

Reverse ping @yubing

[yubing]

squash D143149

[RKSimon]

llvm/lib/Target/X86/X86MemFoldTables.inc

261 ↗

(On Diff #495799)

It would be good to get some/all of these diffs committed first to minimize any changes in codegen due to this patch - the patch should only be about the build process.

[skan]

llvm/lib/Target/X86/X86MemFoldTables.inc

261 ↗

(On Diff #495799)

Both methods are okay to me. I suggested the author to merge the table change into this PR b/c we are not confident the update in the table is correct. The new table needs a careful review.

And if the reviewer doubted an entry was incorrect, the author could point out which rule used in the X86FoldTablesEmitter.cpp was "guilty" and get suggestion/feedback from reviewers.

A separate diff would be the best way if we could check the correctness of the table just by looking at the entry themselves.

[yubing]

llvm/lib/Target/X86/X86MemFoldTables.inc

509–510 ↗

(On Diff #495799)

should remove one of them otherwise leads to duplicate entry in unfolding table.
i will remove {X86::MMX_MOVQ64rr, X86::MMX_MOVQ64rm, 0}, and write "{X86::MMX_MOVQ64rr, X86::MMX_MOVQ64rm, noforward || noreverse}," in llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

[craig.topper]

llvm/lib/Target/X86/X86MemFoldTables.inc

614 ↗

(On Diff #495799)

These aren't real load instructions. It doesn't really make sense for them to be in the table. Folding the load would make the load not happen. That would be incorrect for volatile.

4009 ↗

(On Diff #495799)

Would this allow unfolding a VMOVAPDZ128rmk to VMOVAPDZ128rrk + VMOVAPDZ128rm? That would be a bug.

[yubing]

llvm/lib/Target/X86/X86MemFoldTables.inc

4009 ↗

(On Diff #495799)

@craig.topper
https://discourse.llvm.org/t/auto-generate-the-memory-folding-tables/61100/19
i remember we always fold whole load intrinsics into masked arithmetic. does it happen to masked movereg instruction as well?(only wholeload+ movereg => mask.load)

besides, do we need to add TB_NO_REVERSE for {X86::VMOVAPDZ128rr, X86::VMOVAPDZ128rm, TB_ALIGN_16}?

[craig.topper]

llvm/lib/Target/X86/X86MemFoldTables.inc

4009 ↗

(On Diff #495799)

What I said previously applied to arithmetic that has loads folded into them.

VMOVAPDZ128rmk is created from a masked load with an isel pattern.

VMOVAPDZ128rm will never be unfolded. We unfold to separate a loop invariant load from arithmetic. There’s no arithmetic folded in to it.. the only operands are address operands. They would all need to be loop invariant to hoist so the whole instruction is hoistable.

For VMOVAPDZ128rmk there is a mask operand and address operands. Without TB_NO_REVERSE we would try to unfold it if the address was loop invariant but the mask wasn’t. This would be incorrect. The mask may be masking off elements that will fault.

[yubing]

llvm/lib/Target/X86/X86MemFoldTables.inc

4009 ↗

(On Diff #495799)

thanks for explanation. i also check the code in llvm/lib/CodeGen/MachineLICM.cpp

// First check whether we should hoist this instruction.
if (!IsLoopInvariantInst(*MI) || !IsProfitableToHoist(*MI)) {
  // If not, try unfolding a hoistable load.
  MI = ExtractHoistableLoad(MI);
  if (!MI) return false;
}

since VMOVAPDZ128rmk((the address isloop invariant but the mask is not)) is not IsLoopInvariantInst, we can extract a wholeload from it if it is Without TB_NO_REVERSE. so what you said about VMOVAPDZ128rmk and VMOVAPDZ128rm makes sense to me.

[yubing]

make X86::VMOVAPDZ128rrk, X86::VMOVAPDZ128rmk TB_NO_REVERSE

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

422

Is there better way to check no-kz version's isMoveReg?

[yubing]

remove {"MMX_MOVQ64rr", "MMX_MOVQ64mr"}

[yubing]

disable {"MMX_MOVQ64rr", "MMX_MOVQ64rm"} as well

[yubing]

add comments for removing {"MMX_MOVQ64rr", "MMX_MOVQ64rm"}

[yubing]

the patch can let internal testcase pass. no regression was found in benchmark.

[yubing]

small rebase

[yubing]

ping? @skan

[skan]

LGTM in general

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

21

What's header file Error.h used for?

89

Drop the comment?

109–110

Is this operator never used?

133

Why do we need to compare isPseudo?

425

Do we need to assert the result of getDef is not NULL?

432

Why do we need Result.IsStore here?

[skan]

llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

78

unfoldingtable -> unfolding table

81

ditto

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

load such as VMOVAPDZ128rm will never be unfolded.
if VMOVAPDZ128rm 's memoperand is invariant, then VMOVAPDZ128rm will be hoisted directly and won't be unfolded into rm+rr
if VMOVAPDZ128rm 's memoperand is not invariant, we stop doing unfolding since compiler find it is a simple load.

// First check whether we should hoist this instruction.
if (!IsLoopInvariantInst(*MI) || !IsProfitableToHoist(*MI)) {
  // If not, try unfolding a hoistable load.
  MI = ExtractHoistableLoad(MI);
  if (!MI) return false;
}

MachineInstr *MachineLICMBase::ExtractHoistableLoad(MachineInstr *MI) {
  // Don't unfold simple loads.
  if (MI->canFoldAsLoad())
    return nullptr;

@craig.topper , does line429~439 make sense to you as well?

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

133

I copy the code from "bool operator<(const X86FoldTableEntry &RHS) const {",
@craig.topper , i saw the author of "bool operator<(const X86FoldTableEntry &RHS) const {" is you. do you still remember why we need to compare isPseudo here?

[skan]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

133

I think it's a NFC change from me. If we remove this comparison, will any test fail?

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

133

It's the sort order for the enum in X86GenInstrInfo.inc. Instructions with isPseudo set are ordered before other instructions.

[yubing]

does unfolding only happen in LICM?

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

@craig.topper i think my analysis of VMOVAPDZ128rm is only for LICM. but for other passes in codegen, is it possible for VMOVAPDZ128rm to be unfolded?

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

The other 2 places I know that unfold are TwoAddressInstructionPass and SelectionDAG->MachineIR

TwoAddressInstuctionPass should only happen for instructions with tied source and dest. That doesn't apply to moveReg.

I think SelectionDAG->MachineIR case happens if we need to duplicate an instruction that has an EFLAGS def. If the EFLAGS are used by two instructions and EFLAGS is clobbered in between them. We need to duplicate the flag producing instruction to satisfy the second user. But we can't duplicate the folded load so we have to unfold. That doesn't apply to moveReg.

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

i saw two kinds of unfoldMemoryOperand, one of which is for MIR and another is for DAG. Our folding table can't affect unfoldMemoryOperand for DAG, right?
and unfoldMemoryOperand for MIR only happens in LICM, TwoAddress, X86CMOVConversion, X86KCFI, and X86SpeculativeLoadHardening.
i checked VMOVAPDZ128rm won't be handled in those Pasess. so we don't worry if we need to set TB_NO_REVERSE for VMOVAPDZ128rm.

[yubing]

address skan's comments

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

I think the DAG unfoldMemoryOperand uses the same table.

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

The DAG unfold occurs after the isel portion of SelectionDAG. So it's MachineOpcodes

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

oh, you're right. unfoldMemoryOperand is trying to find the MIR according to DAGNode's Machinecode, and then lookupUnfoldTable

bool
X86InstrInfo::unfoldMemoryOperand(SelectionDAG &DAG, SDNode *N,
                                  SmallVectorImpl<SDNode*> &NewNodes) const {
  if (!N->isMachineOpcode())
    return false;

  const X86MemoryFoldTableEntry *I = lookupUnfoldTable(N->getMachineOpcode());

@craig.topper , but where can i find the code of DAG unfold which occurs after the isel portion of SelectionDAG?
i saw unfoldMemoryOperand(for DAG) in llvm/lib/CodeGen/SelectionDAG/ScheduleDAGRRList.cpp

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

432

ScheduleDAGRRList.cpp is the code. The DAG needs to be scheduled into linear basic block for MachineIR.

[yubing]

llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

31

for myself, why " { "TAILJMPr", "TAILJMPm", TB_FOLDED_LOAD }," listed here.

[steven_wu]

llvm/test/TableGen/x86-auto-memfold.td

3

From the manpage of cmp, it should be like:

cmp [OPTIONS] file1 file2

There are some implementation that cannot take --ignore-initial after the files. Can you move it to the front?

[smeenai]

llvm/test/TableGen/x86-auto-memfold.td

3

This was breaking our tests on macOS as well, so I pushed rG7e271c2a8552 to fix.

[vitalybuka]

Breaks this bot https://lab.llvm.org/buildbot/#/builders/5/builds/32220/steps/16/logs/stdio

[yubing]

In D142084#4201278, @vitalybuka wrote:

Breaks this bot https://lab.llvm.org/buildbot/#/builders/5/builds/32220/steps/16/logs/stdio

@vitalybuka , do you know how to reproduce the issue by buildbot_fast.sh?
i tried https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild#try-local-changes
but it set HEAD wrongly at

commit 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a
Author: Nikolas Klauser <[e-mail]>
Date:   Thu Nov 17 21:34:29 2022 +0100

    [libc++] Fix memory leaks when throwing inside std::vector constructors

    Fixes #58392

    Reviewed By: ldionne, #libc

    Spies: alexfh, hans, joanahalili, dblaikie, libcxx-commits

    Differential Revision: https://reviews.llvm.org/D138601

[yubing]

@hctim do you know how to reproduce the ubsan issue elegantly. after i spend some time modify buildbot_functions.sh, i still can't reproduce it locally. https://lab.llvm.org/buildbot/#/builders/5/builds/32220/steps/16/logs/stdio

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

386

Is it possible for S & TB_ALIGN_MASK to be 0? That would cause 1<< -1 which would match the ubsan error

[yubing]

address ubsan issue and ignore-initial issue

[craig.topper]

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

387

Use a variable instead of repeating the same code twice?

[yubing]

address Craig's comments

[fdeazeve]

This commit may be breaking this bot:
https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/52590/console

llvm-project/llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc:4:48: error: use of undeclared identifier 'TB_NO_REVERSE'
    { "ADD16ri_DB",         "ADD16mi",         TB_NO_REVERSE  },

[fdeazeve]

Also note that the pre-merge checks failed to clang format these files:

llvm/include/llvm/Support/X86FoldTablesUtils.h
llvm/lib/Target/X86/X86InstrFoldTables.h
llvm/utils/TableGen/X86FoldTablesEmitter.cpp
llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

[fdeazeve]

llvm/include/llvm/Support/X86FoldTablesUtils.h

12

The anon namespace is causing the bot failure, was a there a particular reason for this change? If not, I'll submit a patch

[skan]

llvm/include/llvm/Support/X86FoldTablesUtils.h

12

I think it's a typo by the author of this patch.

[abhina.sreeskantharajan]

Hi, this is also causing a failure on the clang-ppc64-aix bot https://lab.llvm.org/buildbot/#/builders/214/builds/6519

+ : 'RUN: at line 1'
+ /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/llvm-tblgen -gen-x86-fold-tables -asmwriternum=1 /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/X86.td -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../include -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/ -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../include/ -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/ --write-if-changed -o /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/test/TableGen/Output/x86-auto-memfold.td.tmp1
+ : 'RUN: at line 2'
+ cmp --ignore-initial=0:568 /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/X86MemFoldTables.inc /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/test/TableGen/Output/x86-auto-memfold.td.tmp1
cmp: illegal option -- -
usage: cmp [-l | -s] file1 file2

[yubing]

llvm/include/llvm/Support/X86FoldTablesUtils.h

12

thanks for fixing it.

[yubing]

In D142084#4206765, @abhina.sreeskantharajan wrote:

Hi, this is also causing a failure on the clang-ppc64-aix bot https://lab.llvm.org/buildbot/#/builders/214/builds/6519

+ : 'RUN: at line 1'
+ /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/llvm-tblgen -gen-x86-fold-tables -asmwriternum=1 /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/X86.td -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../include -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/ -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../include/ -I /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/ --write-if-changed -o /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/test/TableGen/Output/x86-auto-memfold.td.tmp1
+ : 'RUN: at line 2'
+ cmp --ignore-initial=0:568 /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/llvm/test/TableGen/../../lib/Target/X86/X86MemFoldTables.inc /scratch/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/test/TableGen/Output/x86-auto-memfold.td.tmp1
cmp: illegal option -- -
usage: cmp [-l | -s] file1 file2

@abhina.sreeskantharajan
seems "--ignore-initial=0:568" doesn't support ppc64? will https://reviews.llvm.org/D146498 works for you?

[dblaikie]

llvm/test/TableGen/x86-auto-memfold.td

1

Could you update/fix this test to not depend on the real .td file - all the other tblgen tests include a small snippet/test case/example instead? (though the test was renamed in 35aeb321c005ed77ef5e5fb4f8dea69a81c81253)

[skan]

llvm/test/TableGen/x86-auto-memfold.td

1

@dblaikie Thank you for the suggestion! The test itself aims to expose the vulnerable rules in X86FoldTablesEmitter.cpp. Due to the complexity of X86ISA, even the folding rules are already complicated, it is hard to predict whether exceptions will occur when new ISAs are introduced. So unfortunately, we need the real .td file as input. I illustrated the usage of the test file in D147527

llvm/include/llvm/Support/X86FoldTablesUtils.h

-//===-- X86InstrFoldTables.h - X86 Instruction Folding Tables ---*- C++ -*-===//
+//===-- X86FoldTablesUtils.h ------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
-//
-// This file contains the interface to query the X86 memory folding tables.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_LIB_TARGET_X86_X86INSTRFOLDTABLES_H
-#define LLVM_LIB_TARGET_X86_X86INSTRFOLDTABLES_H
 
-#include <cstdint>
-
-namespace llvm {
+#ifndef LLVM_SUPPORT_X86FOLDTABLESUTILS_H
+#define LLVM_SUPPORT_X86FOLDTABLESUTILS_H
 
+namespace {
 enum {
   // Select which memory operand is being unfolded.
   // (stored in bits 0 - 2)
   TB_INDEX_0    = 0,
   TB_INDEX_1    = 1,
   TB_INDEX_2    = 2,
   TB_INDEX_3    = 3,
   TB_INDEX_4    = 4,
enum {
   TB_BCAST_D    =   0 << TB_BCAST_TYPE_SHIFT,
   TB_BCAST_Q    =   1 << TB_BCAST_TYPE_SHIFT,
   TB_BCAST_SS   =   2 << TB_BCAST_TYPE_SHIFT,
   TB_BCAST_SD   =   3 << TB_BCAST_TYPE_SHIFT,
   TB_BCAST_MASK = 0x3 << TB_BCAST_TYPE_SHIFT,
 
   // Unused bits 14-15
 };
-
-// This struct is used for both the folding and unfold tables. They KeyOp
-// is used to determine the sorting order.
-struct X86MemoryFoldTableEntry {
-  uint16_t KeyOp;
-  uint16_t DstOp;
-  uint16_t Flags;
-
-  bool operator<(const X86MemoryFoldTableEntry &RHS) const {
-    return KeyOp < RHS.KeyOp;
-  }
-  bool operator==(const X86MemoryFoldTableEntry &RHS) const {
-    return KeyOp == RHS.KeyOp;
-  }
-  friend bool operator<(const X86MemoryFoldTableEntry &TE, unsigned Opcode) {
-    return TE.KeyOp < Opcode;
-  }
-};
-
-// Look up the memory folding table entry for folding a load and a store into
-// operand 0.
-const X86MemoryFoldTableEntry *lookupTwoAddrFoldTable(unsigned RegOp);
-
-// Look up the memory folding table entry for folding a load or store with
-// operand OpNum.
-const X86MemoryFoldTableEntry *lookupFoldTable(unsigned RegOp, unsigned OpNum);
-
-// Look up the memory unfolding table entry for this instruction.
-const X86MemoryFoldTableEntry *lookupUnfoldTable(unsigned MemOp);
-
-} // namespace llvm
-
-#endif
+#endif // LLVM_SUPPORT_X86FOLDTABLESUTILS_H

llvm/lib/Target/X86/X86InstrFoldTables.h

 //===-- X86InstrFoldTables.h - X86 Instruction Folding Tables ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains the interface to query the X86 memory folding tables.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_X86_X86INSTRFOLDTABLES_H
 #define LLVM_LIB_TARGET_X86_X86INSTRFOLDTABLES_H
 
 #include <cstdint>
+#include "llvm/Support/X86FoldTablesUtils.h"
 
 namespace llvm {
 
-enum {
-  // Select which memory operand is being unfolded.
-  // (stored in bits 0 - 2)
-  TB_INDEX_0    = 0,
-  TB_INDEX_1    = 1,
-  TB_INDEX_2    = 2,
-  TB_INDEX_3    = 3,
-  TB_INDEX_4    = 4,
-  TB_INDEX_MASK = 0x7,
-
-  // Do not insert the reverse map (MemOp -> RegOp) into the table.
-  // This may be needed because there is a many -> one mapping.
-  TB_NO_REVERSE   = 1 << 3,
-
-  // Do not insert the forward map (RegOp -> MemOp) into the table.
-  // This is needed for Native Client, which prohibits branch
-  // instructions from using a memory operand.
-  TB_NO_FORWARD   = 1 << 4,
-
-  TB_FOLDED_LOAD  = 1 << 5,
-  TB_FOLDED_STORE = 1 << 6,
-  TB_FOLDED_BCAST = 1 << 7,
-
-  // Minimum alignment required for load/store.
-  // Used for RegOp->MemOp conversion. Encoded as Log2(Align) + 1 to allow 0
-  // to mean align of 0.
-  // (stored in bits 8 - 11)
-  TB_ALIGN_SHIFT = 8,
-  TB_ALIGN_NONE  =   0 << TB_ALIGN_SHIFT,
-  TB_ALIGN_16    =   5 << TB_ALIGN_SHIFT,
-  TB_ALIGN_32    =   6 << TB_ALIGN_SHIFT,
-  TB_ALIGN_64    =   7 << TB_ALIGN_SHIFT,
-  TB_ALIGN_MASK  = 0xf << TB_ALIGN_SHIFT,
-
-  // Broadcast type.
-  // (stored in bits 12 - 13)
-  TB_BCAST_TYPE_SHIFT = 12,
-  TB_BCAST_D    =   0 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_Q    =   1 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_SS   =   2 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_SD   =   3 << TB_BCAST_TYPE_SHIFT,
-  TB_BCAST_MASK = 0x3 << TB_BCAST_TYPE_SHIFT,
-
-  // Unused bits 14-15
-};
-
 // This struct is used for both the folding and unfold tables. They KeyOp
 // is used to determine the sorting order.
 struct X86MemoryFoldTableEntry {
   uint16_t KeyOp;
   uint16_t DstOp;
   uint16_t Flags;
 
   bool operator<(const X86MemoryFoldTableEntry &RHS) const {

llvm/test/TableGen/x86-auto-memfold.td

+// RUN: llvm-tblgen -gen-x86-fold-tables -asmwriternum=1 %p/../../lib/Target/X86/X86.td -I %p/../../include -I %p/../../lib/Target/X86/ -I %p/../../include/ -I %p/../../lib/Target/ --write-if-changed  -o %t1
+// RUN: cmp  --ignore-initial=0:568 %p/../../lib/Target/X86/X86MemFoldTables.inc %t1

llvm/utils/TableGen/X86FoldTablesEmitter.cpp

 // the X86 backend instructions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenInstruction.h"
 #include "CodeGenTarget.h"
 #include "TableGenBackends.h"
 #include "X86RecognizableInstr.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/Support/FormattedStream.h"
+#include "llvm/Support/X86FoldTablesUtils.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 
 using namespace llvm;
 using namespace X86Disassembler;
 
 namespace {
-
-// 3 possible strategies for the unfolding flag (TB_NO_REVERSE) of the
-// manual added entries.
-enum UnfoldStrategy {
-  UNFOLD,     // Allow unfolding
-  NO_UNFOLD,  // Prevent unfolding
-  NO_STRATEGY // Make decision according to operands' sizes
-};
-
 // Represents an entry in the manual mapped instructions set.
 struct ManualMapEntry {
   const char *RegInstStr;
   const char *MemInstStr;
-  UnfoldStrategy Strategy;
+  uint16_t Strategy;
 
   ManualMapEntry(const char *RegInstStr, const char *MemInstStr,
-                 UnfoldStrategy Strategy = NO_STRATEGY)
+                 uint16_t Strategy = 0)
       : RegInstStr(RegInstStr), MemInstStr(MemInstStr), Strategy(Strategy) {}
 };
 
 // List of instructions requiring explicitly aligned memory.
 const char *ExplicitAlign[] = {"MOVDQA",  "MOVAPS",  "MOVAPD",  "MOVNTPS",
                                "MOVNTPD", "MOVNTDQ", "MOVNTDQA"};
 
 // List of instructions NOT requiring explicit memory alignment.
 const char *ExplicitUnalign[] = {"MOVDQU", "MOVUPS", "MOVUPD",
                                  "PCMPESTRM", "PCMPESTRI",
                                  "PCMPISTRM", "PCMPISTRI" };
 
-// For manually mapping instructions that do not match by their encoding.
-const ManualMapEntry ManualMapSet[] = {
-    { "ADD16ri_DB",         "ADD16mi",         NO_UNFOLD  },
-    { "ADD16ri8_DB",        "ADD16mi8",        NO_UNFOLD  },
-    { "ADD16rr_DB",         "ADD16mr",         NO_UNFOLD  },
-    { "ADD32ri_DB",         "ADD32mi",         NO_UNFOLD  },
-    { "ADD32ri8_DB",        "ADD32mi8",        NO_UNFOLD  },
-    { "ADD32rr_DB",         "ADD32mr",         NO_UNFOLD  },
-    { "ADD64ri32_DB",       "ADD64mi32",       NO_UNFOLD  },
-    { "ADD64ri8_DB",        "ADD64mi8",        NO_UNFOLD  },
-    { "ADD64rr_DB",         "ADD64mr",         NO_UNFOLD  },
-    { "ADD8ri_DB",          "ADD8mi",          NO_UNFOLD  },
-    { "ADD8rr_DB",          "ADD8mr",          NO_UNFOLD  },
-    { "ADD16rr_DB",         "ADD16rm",         NO_UNFOLD  },
-    { "ADD32rr_DB",         "ADD32rm",         NO_UNFOLD  },
-    { "ADD64rr_DB",         "ADD64rm",         NO_UNFOLD  },
-    { "ADD8rr_DB",          "ADD8rm",          NO_UNFOLD  },
-    { "MMX_MOVD64from64rr", "MMX_MOVQ64mr",    UNFOLD },
-    { "MMX_MOVD64grr",      "MMX_MOVD64mr",    UNFOLD },
-    { "MOVLHPSrr",          "MOVHPSrm",        NO_UNFOLD  },
-    { "PUSH16r",            "PUSH16rmm",       UNFOLD },
-    { "PUSH32r",            "PUSH32rmm",       UNFOLD },
-    { "PUSH64r",            "PUSH64rmm",       UNFOLD },
-    { "TAILJMPr",           "TAILJMPm",        UNFOLD },
-    { "TAILJMPr64",         "TAILJMPm64",      UNFOLD },
-    { "TAILJMPr64_REX",     "TAILJMPm64_REX",  UNFOLD },
-    { "VMOVLHPSZrr",        "VMOVHPSZ128rm",   NO_UNFOLD  },
-    { "VMOVLHPSrr",         "VMOVHPSrm",       NO_UNFOLD  },
-};
-
-
+#include "X86FoldTablesEmitterManualMapSet.inc"
 static bool isExplicitAlign(const CodeGenInstruction *Inst) {
   return any_of(ExplicitAlign, [Inst](const char *InstStr) {
     return Inst->TheDef->getName().contains(InstStr);
   });
 }
 
 static bool isExplicitUnalign(const CodeGenInstruction *Inst) {
   return any_of(ExplicitUnalign, [Inst](const char *InstStr) {
     return Inst->TheDef->getName().contains(InstStr);
   });
 }
 
 class X86FoldTablesEmitter {
   RecordKeeper &Records;
   CodeGenTarget Target;
 
   // Represents an entry in the folding table
   class X86FoldTableEntry {
     const CodeGenInstruction *RegInst;
     const CodeGenInstruction *MemInst;
 
   public:
     bool CannotUnfold = false;
+    bool CannotFold = false;
     bool IsLoad = false;
     bool IsStore = false;
     bool IsAligned = false;
     unsigned int Alignment = 0;
 
+    X86FoldTableEntry() = default;
     X86FoldTableEntry(const CodeGenInstruction *RegInst,
                       const CodeGenInstruction *MemInst)
         : RegInst(RegInst), MemInst(MemInst) {}
 
     void print(formatted_raw_ostream &OS) const {
+      // Stop printing record if it can't fold and unfold.
+      if(CannotUnfold && CannotFold)
+        return;
       OS.indent(2);
-      OS << "{ X86::" << RegInst->TheDef->getName() << ",";
-      OS.PadToColumn(40);
-      OS  << "X86::" << MemInst->TheDef->getName() << ",";
-      OS.PadToColumn(75);
+      OS << "{X86::" << RegInst->TheDef->getName() << ", ";
+      OS  << "X86::" << MemInst->TheDef->getName() << ", ";
 
       std::string Attrs;
       if (IsLoad)
-        Attrs += "TB_FOLDED_LOAD | ";
+        Attrs += "TB_FOLDED_LOAD|";
       if (IsStore)
-        Attrs += "TB_FOLDED_STORE | ";
+        Attrs += "TB_FOLDED_STORE|";
       if (CannotUnfold)
-        Attrs += "TB_NO_REVERSE | ";
+        Attrs += "TB_NO_REVERSE|";
+      if (CannotFold)
+        Attrs += "TB_NO_FORWARD|";
       if (IsAligned)
-        Attrs += "TB_ALIGN_" + std::to_string(Alignment) + " | ";
+        Attrs += "TB_ALIGN_" + std::to_string(Alignment) + "|";
 
-      StringRef SimplifiedAttrs = StringRef(Attrs).rtrim("| ");
+      StringRef SimplifiedAttrs = StringRef(Attrs).rtrim("|");
       if (SimplifiedAttrs.empty())
         SimplifiedAttrs = "0";
 
-      OS << SimplifiedAttrs << " },\n";
+      OS << SimplifiedAttrs << "},\n";
     }
 
-    bool operator<(const X86FoldTableEntry &RHS) const {
-      bool LHSpseudo = RegInst->TheDef->getValueAsBit("isPseudo");
-      bool RHSpseudo = RHS.RegInst->TheDef->getValueAsBit("isPseudo");
+  };
+
+  struct CodeGenInstructionComparator {
+    // Comparator function
+    bool operator()(const CodeGenInstruction *LHS,
+                    const CodeGenInstruction *RHS) const {
+      assert(LHS && RHS && "LHS and RHS shouldn't be nullptr");
+      bool LHSpseudo = LHS->TheDef->getValueAsBit("isPseudo");
+      bool RHSpseudo = RHS->TheDef->getValueAsBit("isPseudo");
       if (LHSpseudo != RHSpseudo)
         return LHSpseudo;
 
-      return RegInst->TheDef->getName() < RHS.RegInst->TheDef->getName();
+      return LHS->TheDef->getName() < RHS->TheDef->getName();
     }
   };
 
-  typedef std::vector<X86FoldTableEntry> FoldTable;
+  typedef std::map<const CodeGenInstruction *, X86FoldTableEntry,
+                   CodeGenInstructionComparator>
+      FoldTable;
   // std::vector for each folding table.
   // Table2Addr - Holds instructions which their memory form performs load+store
   // Table#i - Holds instructions which the their memory form perform a load OR
   //           a store,  and their #i'th operand is folded.
   FoldTable Table2Addr;
   FoldTable Table0;
   FoldTable Table1;
   FoldTable Table2;
   FoldTable Table3;
   FoldTable Table4;
 
 public:
   X86FoldTablesEmitter(RecordKeeper &R) : Records(R), Target(R) {}
 
   // run - Generate the 6 X86 memory fold tables.
   void run(formatted_raw_ostream &OS);
 
 private:
   // Decides to which table to add the entry with the given instructions.
   // S sets the strategy of adding the TB_NO_REVERSE flag.
   void updateTables(const CodeGenInstruction *RegInstr,
-                    const CodeGenInstruction *MemInstr,
-                    const UnfoldStrategy S = NO_STRATEGY);
+                    const CodeGenInstruction *MemInstr, const uint16_t S = 0,
+                    bool IsManual = false);
 
   // Generates X86FoldTableEntry with the given instructions and fill it with
   // the appropriate flags - then adds it to Table.
   void addEntryWithFlags(FoldTable &Table, const CodeGenInstruction *RegInstr,
-                         const CodeGenInstruction *MemInstr,
-                         const UnfoldStrategy S, const unsigned int FoldedInd);
+                         const CodeGenInstruction *MemInstr, const uint16_t S,
+                         const unsigned int FoldedInd, bool isManual);
 
   // Print the given table as a static const C++ array of type
   // X86MemoryFoldTableEntry.
   void printTable(const FoldTable &Table, StringRef TableName,
                   formatted_raw_ostream &OS) {
     OS << "static const X86MemoryFoldTableEntry MemoryFold" << TableName
        << "[] = {\n";
 
-    for (const X86FoldTableEntry &E : Table)
-      E.print(OS);
+    for (auto &E : Table)
+      E.second.print(OS);
 
     OS << "};\n\n";
   }
 };
 
 // Return true if one of the instruction's operands is a RST register class
 static bool hasRSTRegClass(const CodeGenInstruction *Inst) {
   return any_of(Inst->Operands, [](const CGIOperandList::OperandInfo &OpIn) {
private:
   }
 };
 
 } // end anonymous namespace
 
 void X86FoldTablesEmitter::addEntryWithFlags(FoldTable &Table,
                                              const CodeGenInstruction *RegInstr,
                                              const CodeGenInstruction *MemInstr,
-                                             const UnfoldStrategy S,
-                                             const unsigned int FoldedInd) {
+                                             const uint16_t S,
+                                             const unsigned int FoldedInd,
+                                             bool isManual) {
 
   X86FoldTableEntry Result = X86FoldTableEntry(RegInstr, MemInstr);
   Record *RegRec = RegInstr->TheDef;
   Record *MemRec = MemInstr->TheDef;
 
+  if (isManual) {
+    Result.CannotUnfold = (S & TB_NO_REVERSE) != 0;
+    Result.CannotFold = (S & TB_NO_FORWARD) != 0;
+    Result.IsLoad = (S & TB_FOLDED_LOAD) != 0;
+    Result.IsStore = (S & TB_FOLDED_STORE) != 0;
+    Result.IsAligned = (S & TB_ALIGN_MASK) != 0;
+    auto AlignValue = (S & TB_ALIGN_MASK) >> TB_ALIGN_SHIFT;
+    Result.Alignment = AlignValue > 0 ? (1 << (AlignValue - 1)) : 0;
+    Table[RegInstr] = Result;
+    return;
+  }
+
   // Only table0 entries should explicitly specify a load or store flag.
   if (&Table == &Table0) {
     unsigned MemInOpsNum = MemRec->getValueAsDag("InOperandList")->getNumArgs();
     unsigned RegInOpsNum = RegRec->getValueAsDag("InOperandList")->getNumArgs();
     // If the instruction writes to the folded operand, it will appear as an
     // output in the register form instruction and as an input in the memory
     // form instruction.
     // If the instruction reads from the folded operand, it well appear as in
     // input in both forms.
     if (MemInOpsNum == RegInOpsNum)
       Result.IsLoad = true;
     else
       Result.IsStore = true;
   }
 
   Record *RegOpRec = RegInstr->Operands[FoldedInd].Rec;
   Record *MemOpRec = MemInstr->Operands[FoldedInd].Rec;
 
   // Unfolding code generates a load/store instruction according to the size of
   // the register in the register form instruction.
   // If the register's size is greater than the memory's operand size, do not
   // allow unfolding.
-  if (S == UNFOLD)
-    Result.CannotUnfold = false;
-  else if (S == NO_UNFOLD)
+
+  // the unfolded load size will be based on the register size. If that’s bigger
+  // than the memory operand size, the unfolded load will load more memory and
+  // potentially cause a memory fault.
+  if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
+    Result.CannotUnfold = true;
+
+  // Check no-kz version's isMoveReg
+  Record *BaseDef = nullptr;
+  if (RegRec->getName().ends_with("rkz") &&
+      (BaseDef = Records.getDef(
+           RegRec->getName().substr(0, RegRec->getName().size() - 2)))) {
+    Result.CannotUnfold =
+        Target.getInstruction(BaseDef).isMoveReg ? true : Result.CannotUnfold;
+  } else if (RegRec->getName().ends_with("rk") &&
+             (BaseDef = Records.getDef(
+                  RegRec->getName().substr(0, RegRec->getName().size() - 1)))) {
+    Result.CannotUnfold =
+        Target.getInstruction(BaseDef).isMoveReg ? true : Result.CannotUnfold;
+  } else if (RegInstr->isMoveReg && Result.IsStore)
     Result.CannotUnfold = true;
-  else if (getRegOperandSize(RegOpRec) > getMemOperandSize(MemOpRec))
-    Result.CannotUnfold = true; // S == NO_STRATEGY
 
   uint64_t Enc = getValueFromBitsInit(RegRec->getValueAsBitsInit("OpEncBits"));
   if (isExplicitAlign(RegInstr)) {
     // The instruction require explicitly aligned memory.
     BitsInit *VectSize = RegRec->getValueAsBitsInit("VectSize");
     uint64_t Value = getValueFromBitsInit(VectSize);
     Result.IsAligned = true;
     Result.Alignment = Value;
   } else if (Enc != X86Local::XOP && Enc != X86Local::VEX &&
              Enc != X86Local::EVEX) {
     // Instructions with VEX encoding do not require alignment.
     if (!isExplicitUnalign(RegInstr) && getMemOperandSize(MemOpRec) > 64) {
       // SSE packed vector instructions require a 16 byte alignment.
       Result.IsAligned = true;
       Result.Alignment = 16;
     }
   }
+  // Expand is only ever created as a masked instruction. It is not safe to
+  // unfold a masked expand because we don't know if it came from an expand load
+  // intrinsic or folding a plain load. If it is from a expand load intrinsic,
+  // Unfolding to plain load would read more elements and could trigger a fault.
+  if (RegRec->getName().contains("EXPAND"))
+    Result.CannotUnfold = true;
 
-  Table.push_back(Result);
+  Table[RegInstr] = Result;
 }
 
 void X86FoldTablesEmitter::updateTables(const CodeGenInstruction *RegInstr,
                                         const CodeGenInstruction *MemInstr,
-                                        const UnfoldStrategy S) {
+                                        const uint16_t S, bool IsManual) {
 
   Record *RegRec = RegInstr->TheDef;
   Record *MemRec = MemInstr->TheDef;
   unsigned MemOutSize = MemRec->getValueAsDag("OutOperandList")->getNumArgs();
   unsigned RegOutSize = RegRec->getValueAsDag("OutOperandList")->getNumArgs();
   unsigned MemInSize = MemRec->getValueAsDag("InOperandList")->getNumArgs();
   unsigned RegInSize = RegRec->getValueAsDag("InOperandList")->getNumArgs();
 
   // Instructions which Read-Modify-Write should be added to Table2Addr.
   if (MemOutSize != RegOutSize && MemInSize == RegInSize) {
-    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0);
+    addEntryWithFlags(Table2Addr, RegInstr, MemInstr, S, 0, IsManual);
     return;
   }
 
   if (MemInSize == RegInSize && MemOutSize == RegOutSize) {
     // Load-Folding cases.
     // If the i'th register form operand is a register and the i'th memory form
     // operand is a memory operand, add instructions to Table#i.
     for (unsigned i = RegOutSize, e = RegInstr->Operands.size(); i < e; i++) {
       Record *RegOpRec = RegInstr->Operands[i].Rec;
       Record *MemOpRec = MemInstr->Operands[i].Rec;
       // PointerLikeRegClass: For instructions like TAILJMPr, TAILJMPr64, TAILJMPr64_REX
       if ((isRegisterOperand(RegOpRec) ||
            RegOpRec->isSubClassOf("PointerLikeRegClass")) &&
           isMemoryOperand(MemOpRec)) {
         switch (i) {
         case 0:
-          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+          addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0, IsManual);
           return;
         case 1:
-          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1);
+          addEntryWithFlags(Table1, RegInstr, MemInstr, S, 1, IsManual);
           return;
         case 2:
-          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2);
+          addEntryWithFlags(Table2, RegInstr, MemInstr, S, 2, IsManual);
           return;
         case 3:
-          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3);
+          addEntryWithFlags(Table3, RegInstr, MemInstr, S, 3, IsManual);
           return;
         case 4:
-          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4);
+          addEntryWithFlags(Table4, RegInstr, MemInstr, S, 4, IsManual);
           return;
         }
       }
     }
   } else if (MemInSize == RegInSize + 1 && MemOutSize + 1 == RegOutSize) {
     // Store-Folding cases.
     // If the memory form instruction performs a store, the *output*
     // register of the register form instructions disappear and instead a
     // memory *input* operand appears in the memory form instruction.
     // For example:
     //   MOVAPSrr => (outs VR128:$dst), (ins VR128:$src)
     //   MOVAPSmr => (outs), (ins f128mem:$dst, VR128:$src)
     Record *RegOpRec = RegInstr->Operands[RegOutSize - 1].Rec;
     Record *MemOpRec = MemInstr->Operands[RegOutSize - 1].Rec;
     if (isRegisterOperand(RegOpRec) && isMemoryOperand(MemOpRec) &&
         getRegOperandSize(RegOpRec) == getMemOperandSize(MemOpRec))
-      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0);
+      addEntryWithFlags(Table0, RegInstr, MemInstr, S, 0, IsManual);
   }
 }
 
 void X86FoldTablesEmitter::run(formatted_raw_ostream &OS) {
   emitSourceFileHeader("X86 fold tables", OS);
 
   // Holds all memory instructions
   std::vector<const CodeGenInstruction *> MemInsts;
void X86FoldTablesEmitter::run(formatted_raw_ostream &OS) {
   }
 
   // Add the manually mapped instructions listed above.
   for (const ManualMapEntry &Entry : ManualMapSet) {
     Record *RegInstIter = Records.getDef(Entry.RegInstStr);
     Record *MemInstIter = Records.getDef(Entry.MemInstStr);
 
     updateTables(&(Target.getInstruction(RegInstIter)),
-                 &(Target.getInstruction(MemInstIter)), Entry.Strategy);
+                 &(Target.getInstruction(MemInstIter)), Entry.Strategy, true);
   }
 
-  // Sort the tables before printing.
-  llvm::sort(Table2Addr);
-  llvm::sort(Table0);
-  llvm::sort(Table1);
-  llvm::sort(Table2);
-  llvm::sort(Table3);
-  llvm::sort(Table4);
-
   // Print all tables.
   printTable(Table2Addr, "Table2Addr", OS);
   printTable(Table0, "Table0", OS);
   printTable(Table1, "Table1", OS);
   printTable(Table2, "Table2", OS);
   printTable(Table3, "Table3", OS);
   printTable(Table4, "Table4", OS);
 }
 
 namespace llvm {
 
 void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &o) {
   formatted_raw_ostream OS(o);
   X86FoldTablesEmitter(RK).run(OS);
 }
 } // namespace llvm

llvm/utils/TableGen/X86FoldTablesEmitterManualMapSet.inc

+const ManualMapEntry ManualMapSet[] = {
+    // Part1: These following records are for manually mapping instructions that
+    // do not match by their encoding.
+    { "ADD16ri_DB",         "ADD16mi",         TB_NO_REVERSE  },
+    { "ADD16ri8_DB",        "ADD16mi8",        TB_NO_REVERSE  },
+    { "ADD16rr_DB",         "ADD16mr",         TB_NO_REVERSE  },
+    { "ADD32ri_DB",         "ADD32mi",         TB_NO_REVERSE  },
+    { "ADD32ri8_DB",        "ADD32mi8",        TB_NO_REVERSE  },
+    { "ADD32rr_DB",         "ADD32mr",         TB_NO_REVERSE  },
+    { "ADD64ri32_DB",       "ADD64mi32",       TB_NO_REVERSE  },
+    { "ADD64ri8_DB",        "ADD64mi8",        TB_NO_REVERSE  },
+    { "ADD64rr_DB",         "ADD64mr",         TB_NO_REVERSE  },
+    { "ADD8ri_DB",          "ADD8mi",          TB_NO_REVERSE  },
+    { "ADD8rr_DB",          "ADD8mr",          TB_NO_REVERSE  },
+    { "ADD16rr_DB",         "ADD16rm",         TB_NO_REVERSE  },
+    { "ADD32rr_DB",         "ADD32rm",         TB_NO_REVERSE  },
+    { "ADD64rr_DB",         "ADD64rm",         TB_NO_REVERSE  },
+    { "ADD8rr_DB",          "ADD8rm",          TB_NO_REVERSE  },
+    { "MMX_MOVD64from64rr", "MMX_MOVQ64mr",    TB_FOLDED_STORE },
+    { "MMX_MOVD64grr",      "MMX_MOVD64mr",    TB_FOLDED_STORE },
+    { "MOV64toSDrr",        "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVDI2SSrr",         "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVPQIto64rr",       "MOVPQI2QImr",     TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVSDto64rr",        "MOVSDmr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "MOVSS2DIrr",         "MOVSSmr",         TB_FOLDED_STORE },
+    { "MOVLHPSrr",          "MOVHPSrm",        TB_NO_REVERSE  },
+    { "PUSH16r",            "PUSH16rmm",       TB_FOLDED_LOAD },
+    { "PUSH32r",            "PUSH32rmm",       TB_FOLDED_LOAD },
+    { "PUSH64r",            "PUSH64rmm",       TB_FOLDED_LOAD },
+    { "TAILJMPr",           "TAILJMPm",        TB_FOLDED_LOAD },
+    { "TAILJMPr64",         "TAILJMPm64",      TB_FOLDED_LOAD },
+    { "TAILJMPr64_REX",     "TAILJMPm64_REX",  TB_FOLDED_LOAD },
+    { "TCRETURNri",         "TCRETURNmi",      TB_FOLDED_LOAD | TB_NO_FORWARD },
+    { "TCRETURNri64",       "TCRETURNmi64",    TB_FOLDED_LOAD | TB_NO_FORWARD },
+    { "VMOVLHPSZrr",        "VMOVHPSZ128rm",   TB_NO_REVERSE  },
+    { "VMOVLHPSrr",         "VMOVHPSrm",       TB_NO_REVERSE  },
+    { "VMOV64toSDZrr",      "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOV64toSDrr",       "MOV64mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVDI2SSZrr",       "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVDI2SSrr",        "MOV32mr",         TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVPQIto64Zrr",     "VMOVPQI2QIZmr",   TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVPQIto64rr",      "VMOVPQI2QImr",    TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSDto64Zrr",      "VMOVSDZmr",       TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSDto64rr",       "VMOVSDmr",        TB_FOLDED_STORE | TB_NO_REVERSE },
+    { "VMOVSS2DIZrr",       "VMOVSSZmr",       TB_FOLDED_STORE },
+    { "VMOVSS2DIrr",        "VMOVSSmr",        TB_FOLDED_STORE },
+    { "MMX_MOVD64to64rr",   "MMX_MOVQ64rm",    0 },
+    { "MOV64toPQIrr",       "MOVQI2PQIrm",     TB_NO_REVERSE },
+    { "MOV64toSDrr",        "MOVSDrm_alt",     TB_NO_REVERSE },
+    { "MOVDI2SSrr",         "MOVSSrm_alt",     0 },
+    { "VMOV64toPQIZrr",     "VMOVQI2PQIZrm",    TB_NO_REVERSE },
+    { "VMOV64toPQIrr",      "VMOVQI2PQIrm",     TB_NO_REVERSE },
+    { "VMOV64toSDZrr",      "VMOVSDZrm_alt",   TB_NO_REVERSE },
+    { "VMOV64toSDrr",       "VMOVSDrm_alt",    TB_NO_REVERSE },
+    { "VMOVDI2SSZrr",       "VMOVSSZrm_alt",   0 },
+    { "VMOVDI2SSrr",        "VMOVSSrm_alt",    0 },
+    { "MOVSDrr",            "MOVLPDrm",         TB_NO_REVERSE },
+    { "VMOVSDZrr",          "VMOVLPDZ128rm",    TB_NO_REVERSE },
+    { "VMOVSDrr",           "VMOVLPDrm",        TB_NO_REVERSE },
+
+    // Part2: These following records are for manually mapping instructions that
+    // have same opcode.
+    // INSERTPSrm has no count_s while INSERTPSrr has count_s.
+    // count_s is to indicate which element in dst vector is inserted.
+    // if count_s!=0, we can't fold INSERTPSrr into INSERTPSrm
+    //
+    // the following folding can happen when count_s==0
+    // load xmm0, m32
+    // insertpsrr xmm1, xmm0, imm
+    // =>
+    // insertpsrm xmm1, m32, imm
+    { "INSERTPSrr",         "INSERTPSrm",      TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Lr",              "UD1Lm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Qr",              "UD1Qm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    { "UD1Wr",              "UD1Wm",           TB_NO_REVERSE | TB_NO_FORWARD },
+    // Remove {"MMX_MOVQ64rr", "MMX_MOVQ64mr"} since it will create duplicate in
+    // unfolding table due to the {"MMX_MOVD64from64rr", "MMX_MOVQ64mr"}
+    { "MMX_MOVQ64rr",       "MMX_MOVQ64mr",    TB_NO_FORWARD | TB_NO_REVERSE },
+    // Remove {"MMX_MOVQ64rr", "MMX_MOVQ64rm"} since it will create duplicate in
+    // unfolding table due to the {"MMX_MOVD64from64rr", "MMX_MOVQ64rm"}
+    { "MMX_MOVQ64rr",       "MMX_MOVQ64rm",    TB_NO_FORWARD | TB_NO_REVERSE },
+};
+