[RFC][GISel] Add a way to ignore COPY instructions in InstructionSelector

[Pierre-vh]

Summary

RFC to add a way to ignore COPY instructions when pattern-matching MIR in GISel.

• Add a new "GISelFlags" class to TableGen. Both Pattern and PatFrags defs can use it to alter matching behaviour.
• Flags start at zero and are scoped: the setter returns a SaveAndRestore object so that when the current scope ends, the flags are restored to their previous values. This allows child patterns to modify the flags without affecting the parent pattern.
• Child patterns always reuse the parent's pattern, but they can override its values. For more examples, see GlobalISelEmitterFlags.td tests.
• [AMDGPU] Use the IgnoreCopies flag in BFI patterns, which are known to be bothered by cross-regbank copies.

[Pierre-vh]

Rebase, rework

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

5263

Would it be be possible to set this on a PatFrag level instead of a top-level pattern?

[Pierre-vh]

• Add test

@arsenm for the PatFrags it looks like they don't (always) generate a Try block so the system as-is cannot handle them.
I was thinking of making GIM_SetFlag work only per-instruction and be used as a prefix, so instead of having

GIM_Try ...
  GIM_SetFlag GIMSF_IgnoreCopies

to set flags for a whole block we'd have

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_Try ... # Sets the flag on the next instruction -> Try saves them in the whole block

Then we could have flags set for a single instruction which would allow us to support PatFrags and virtually anything else
after a bit more backend work:

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_RecordInsn ...

[Pierre-vh]

Current design doesn't work on PatFrags as they are inlined. I'm looking for solutions.

[Pierre-vh]

Rework so the system works on both Pattern and PatFrags

[Pierre-vh]

Rebase

This is ready for review, I don't have anymore changes to make for now.

[Petar.Avramovic]

I like the GISelFlags, looks pretty lightweight.
Are you planning to add more flags?
About IgnoreCopies, it looks correct to me. Considering the possibility to break constant bus limit, patch synergies well with GISelPredicateCode that checks for it.
Can you add look through copies to Three Op Pats from VOP3 td file?
Upcoming regbankselect changes should be able to deal with copies but nevertheless I would like to see this one merged.
LGTM but I would wait for feedback from other reviewers.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstructions.td

2062

I think it's cleaner to hide these in an AMDGPUPat subclass, like AMDGPUPatIgnoreCopies

[Pierre-vh]

Address comment

Can this land (once the parent diff does) or does this need to wait for more people to approve?

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

enum {
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   /// - LowerBound - numerically minimum Type ID supported
   /// - UpperBound - numerically maximum + 1 Type ID supported
   /// - Default - failure jump target
   /// - JumpTable... - (UpperBound - LowerBound) (at least 2) jump targets
   GIM_SwitchType,
 
-  /// Record the specified instruction
+  /// Record the specified instruction.
+  /// The IgnoreCopies variant ignores COPY instructions.
   /// - NewInsnID - Instruction ID to define
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   GIM_RecordInsn,
+  GIM_RecordInsnIgnoreCopies,
 
   /// Check the feature bits
   /// - Expected features
   GIM_CheckFeatures,
 
   /// Check the opcode on the specified instruction
   /// - InsnID - Instruction ID
   /// - Expected opcode
enum {
   GIM_CheckIsImm,
 
   /// Check if the specified operand is safe to fold into the current
   /// instruction.
   /// - InsnID - Instruction ID
   GIM_CheckIsSafeToFold,
 
   /// Check the specified operands are identical.
+  /// The IgnoreCopies variant looks through COPY instructions before
+  /// comparing the operands.
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   /// - OtherInsnID - Other instruction ID
   /// - OtherOpIdx - Other operand index
   GIM_CheckIsSameOperand,
+  GIM_CheckIsSameOperandIgnoreCopies,
 
   /// Predicates with 'let PredicateCodeUsesOperands = 1' need to examine some
   /// named operands that will be recorded in RecordedOperands. Names of these
   /// operands are referenced in predicate argument list. Emitter determines
   /// StoreIdx(corresponds to the order in which names appear in argument list).
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   /// - StoreIdx - Store location in RecordedOperands.
enum {
   /// - OldInsnID - Instruction ID to get the matched operand from
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomRenderer,
 
   /// Render operands to the specified instruction using a custom function,
   /// reading from a specific operand.
   /// - InsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to get the matched operand from
-  /// - OpIdx - Operand index in OldInsnID the render function should read from..
+  /// - OpIdx - Operand index in OldInsnID the render function should read
+  /// from..
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomOperandRenderer,
 
   /// Render a G_CONSTANT operator as a sign-extended immediate.
   /// - NewInsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to copy from
   /// The operand index is implicitly 1.
   GIR_CopyConstantAsSImm,

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

while (true) {
     switch (MatcherOpcode) {
     case GIM_Try: {
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": Begin try-block\n");
       OnFailResumeAt.push_back(MatchTable[CurrentIdx++]);
       break;
     }
 
-    case GIM_RecordInsn: {
+    case GIM_RecordInsn:
+    case GIM_RecordInsnIgnoreCopies: {
       int64_t NewInsnID = MatchTable[CurrentIdx++];
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
 
       // As an optimisation we require that MIs[0] is always the root. Refuse
       // any attempt to modify it.
       assert(NewInsnID != 0 && "Refusing to modify MIs[0]");
 
       MachineOperand &MO = State.MIs[InsnID]->getOperand(OpIdx);
       if (!MO.isReg()) {
         DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                         dbgs() << CurrentIdx << ": Not a register\n");
         if (handleReject() == RejectAndGiveUp)
           return false;
         break;
       }
       if (MO.getReg().isPhysical()) {
         DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                         dbgs() << CurrentIdx << ": Is a physical register\n");
         if (handleReject() == RejectAndGiveUp)
           return false;
         break;
       }
 
-      MachineInstr *NewMI = MRI.getVRegDef(MO.getReg());
+      MachineInstr *NewMI;
+      if (MatcherOpcode == GIM_RecordInsnIgnoreCopies)
+        NewMI = getDefIgnoringCopies(MO.getReg(), MRI);
+      else
+        NewMI = MRI.getVRegDef(MO.getReg());
+
       if ((size_t)NewInsnID < State.MIs.size())
         State.MIs[NewInsnID] = NewMI;
       else {
         assert((size_t)NewInsnID == State.MIs.size() &&
                "Expected to store MIs in order");
         State.MIs.push_back(NewMI);
       }
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
case GIM_CheckIsSafeToFold: {
                              << InsnID << "])\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       if (!isObviouslySafeToFold(*State.MIs[InsnID], *State.MIs[0])) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
-    case GIM_CheckIsSameOperand: {
+    case GIM_CheckIsSameOperand:
+    case GIM_CheckIsSameOperandIgnoreCopies: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
       int64_t OtherInsnID = MatchTable[CurrentIdx++];
       int64_t OtherOpIdx = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_CheckIsSameOperand(MIs["
                              << InsnID << "][" << OpIdx << "], MIs["
                              << OtherInsnID << "][" << OtherOpIdx << "])\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       assert(State.MIs[OtherInsnID] != nullptr && "Used insn before defined");
-      if (!State.MIs[InsnID]->getOperand(OpIdx).isIdenticalTo(
-              State.MIs[OtherInsnID]->getOperand(OtherOpIdx))) {
+
+      MachineOperand &Op = State.MIs[InsnID]->getOperand(OpIdx);
+      MachineOperand &OtherOp = State.MIs[OtherInsnID]->getOperand(OtherOpIdx);
+
+      if (MatcherOpcode == GIM_CheckIsSameOperandIgnoreCopies) {
+        if (Op.isReg() && OtherOp.isReg()) {
+          MachineInstr *MI = getDefIgnoringCopies(Op.getReg(), MRI);
+          MachineInstr *OtherMI = getDefIgnoringCopies(OtherOp.getReg(), MRI);
+          if (MI && MI == OtherMI)
+            break;
+        }
+      }
+
+      if (!Op.isIdenticalTo(OtherOp)) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
     case GIM_Reject:
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_Reject\n");

llvm/include/llvm/Target/TargetSelectionDAG.td

class PatFrags<dag ops, list<dag> frags, code pred = [{}],
   // cast<LoadSDNode>(N)->getMemoryVT() == MVT::<VT>;
   // cast<StoreSDNode>(N)->getMemoryVT() == MVT::<VT>;
   ValueType MemoryVT = ?;
   // cast<LoadSDNode>(N)->getMemoryVT().getScalarType() == MVT::<VT>;
   // cast<StoreSDNode>(N)->getMemoryVT().getScalarType() == MVT::<VT>;
   ValueType ScalarMemoryVT = ?;
 }
 
+// Patterns and PatFrags can also subclass GISelFlags to set flags that affect
+// how GlobalISel behaves when matching them.
+class GISelFlags {
+  bit GIIgnoreCopies = ?;
+}
+
 // PatFrag - A version of PatFrags matching only a single fragment.
 class PatFrag<dag ops, dag frag, code pred = [{}],
               SDNodeXForm xform = NOOP_SDNodeXForm>
   : PatFrags<ops, [frag], pred, xform>;
 
 // OutPatFrag is a pattern fragment that is used as part of an output pattern
 // (not an input pattern). These do not have predicates or transforms, but are
 // used to avoid repeated subexpressions in output patterns.

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

class PredicateControl {
   list<Predicate> Predicates = PredConcat<
                                  PredConcat<PredConcat<OtherPredicates,
                                                        SubtargetPredicate>.ret,
                                             AssemblerPredicate>.ret,
                                  WaveSizePredicate>.ret;
 }
 
 class AMDGPUPat<dag pattern, dag result> : Pat<pattern, result>,
-      PredicateControl;
+      PredicateControl, GISelFlags;
+
+let GIIgnoreCopies = 1 in
+class AMDGPUPatIgnoreCopies<dag pattern, dag result> : AMDGPUPat<pattern, result>;
 
 let RecomputePerFunction = 1 in {
 def FP16Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def FP32Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP32Denormals()">;
 def FP64Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def NoFP16Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def NoFP32Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP32Denormals()">;
 def NoFP64Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;

llvm/lib/Target/AMDGPU/SIInstructions.td

def BFIImm32 : PatFrag<
   [{
     auto *X = dyn_cast<ConstantSDNode>(N->getOperand(0)->getOperand(1));
     auto *NotX = dyn_cast<ConstantSDNode>(N->getOperand(1)->getOperand(1));
     return X && NotX &&
       ~(unsigned)X->getZExtValue() == (unsigned)NotX->getZExtValue();
   }]
 >;
 
+
 // Definition from ISA doc:
 // (y & x) | (z & ~x)
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<or> (and i32:$y, i32:$x), (and i32:$z, (not i32:$x))),
   (V_BFI_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
-                 (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32),
-                 (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32))
 >;
 
 // (y & C) | (z & ~C)
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (BFIImm32 i32:$x, i32:$y, i32:$z),
   (V_BFI_B32_e64 VSrc_b32:$x, VSrc_b32:$y, VSrc_b32:$z)
 >;
 
 // 64-bit version
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<or> (and i64:$y, i64:$x), (and i64:$z, (not i64:$x))),
   (REG_SEQUENCE VReg_64,
     (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub0)),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub0))), sub0,
     (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub1)),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub1))), sub1)
 >;
 
 // SHA-256 Ch function
 // z ^ (x & (y ^ z))
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<xor> i32:$z, (and i32:$x, (xor i32:$y, i32:$z))),
   (V_BFI_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
-                 (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32),
-                 (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32))
 >;
 
 // 64-bit version
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<xor> i64:$z, (and i64:$x, (xor i64:$y, i64:$z))),
   (REG_SEQUENCE VReg_64,
     (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub0)),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub0))), sub0,
     (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub1)),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub1))), sub1)
 >;
 
 def : AMDGPUPat <
   (fcopysign f32:$src0, f32:$src1),
   (V_BFI_B32_e64 (S_MOV_B32 (i32 0x7fffffff)), $src0, $src1)
 >;
 
 def : AMDGPUPat <
def : AMDGPUPat <
   (DivergentBinFrag<sra> (shl_oneuse i32:$src, (sub 32, i32:$width)),
                          (sub 32, i32:$width)),
   (V_BFE_I32_e64 $src, (i32 0), $width)
 >;
 
 // SHA-256 Ma patterns
 
 // ((x & z) | (y & (x | z))) -> BFI (XOR x, y), z, y
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<or> (and i32:$x, i32:$z),
                         (and i32:$y, (or i32:$x, i32:$z))),
   (V_BFI_B32_e64 (V_XOR_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
                                 (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32)),
-                 (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32),
-                 (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32))
 >;
 
-def : AMDGPUPat <
+def : AMDGPUPatIgnoreCopies <
   (DivergentBinFrag<or> (and i64:$x, i64:$z),
                         (and i64:$y, (or i64:$x, i64:$z))),
   (REG_SEQUENCE VReg_64,
     (V_BFI_B32_e64 (V_XOR_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
                     (i32 (EXTRACT_SUBREG VReg_64:$y, sub0))),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub0)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub0))), sub0,
     (V_BFI_B32_e64 (V_XOR_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
                     (i32 (EXTRACT_SUBREG VReg_64:$y, sub1))),
-               (i32 (EXTRACT_SUBREG VReg_64:$z, sub1)),
-               (i32 (EXTRACT_SUBREG VReg_64:$y, sub1))), sub1)
 >;
 
 multiclass IntMed3Pat<Instruction med3Inst,
                  SDPatternOperator min,
                  SDPatternOperator max> {
 
   // This matches 16 permutations of
   // min(max(a, b), max(min(a, b), c))

llvm/test/CodeGen/AMDGPU/bfi_int.ll

 ;
 ; GFX10-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, v0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    s_xor_b32 s0, s0, s1
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX8-GISEL-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s1
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    s_xor_b32 s0, s0, s1
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, v0, s0, s1
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

 ;
 ; GFX10-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, s0, v0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX8-GISEL-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, s0, v0, v1
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, s0, v0, s1
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

 ;
 ; GFX10-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v1, s0, v1
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, v0, v1
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s0, v0
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v1, s0, v1
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, v0, v1
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s0, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

llvm/test/TableGen/GlobalISelEmitterFlags.td

+// RUN: llvm-tblgen %s -gen-global-isel -optimize-match-table=false -I %p/../../include -I %p/Common -o - | FileCheck %s
+
+include "llvm/Target/Target.td"
+include "GlobalISelEmitterCommon.td"
+
+def InstThreeOperands : I<(outs GPR32:$dst), (ins GPR32:$cond, GPR32:$src,GPR32:$src2), []>;
+
+class SrlPF<bit IC>: PatFrag<
+  (ops node:$PATFRAG_Src0, node:$src1),
+  (srl $PATFRAG_Src0, $src1)>, GISelFlags {
+  let GIIgnoreCopies = IC;
+}
+
+// GIIgnoreCopies on Pattern
+//  MIs[1] should be using IgnoreCopies variants.
+let GIIgnoreCopies = 1 in
+def : Pat<
+  (i32 (sub (mul i32:$src0, i32:$src0), i32:$src1)),
+  (InstThreeOperands GPR32:$src0, GPR32:$src1, GPR32:$src1)
+>, GISelFlags;
+
+// GIIgnoreCopies set on "root" PatFrag.
+//  MIs[1] and MIs[2] should be using IgnoreCopies variants.
+def : Pat<
+  (i32 (SrlPF<1> (shl (mul i32:$src0, i32:$src0), i32:$src1), i32:$src0)),
+  (InstThreeOperands GPR32:$src0, GPR32:$src1, GPR32:$src1)
+>;
+
+// GIIgnoreCopies set on "root" PatFrag, but a children PatFrag forces it back to zero.
+//    MIs[1] should be using IgnoreCopies variants.
+//    MIs[2] should NOT be using them.
+def : Pat<
+  (i32 (SrlPF<1> (SrlPF<0> (add i32:$src0, i32:$src0), i32:$src1), i32:$src0)),
+  (InstThreeOperands GPR32:$src0, GPR32:$src1, GPR32:$src1)
+>;
+
+// CHECK: GIM_Try
+// CHECK:    GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK:    GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_LSHR
+// CHECK:    GIM_RecordInsn, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK:    GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_ADD
+// CHECK:    GIM_CheckIsSameOperand, /*MI*/2, /*OpIdx*/2, /*OtherMI*/2, /*OtherOpIdx*/1
+// CHECK:    GIM_CheckIsSameOperandIgnoreCopies, /*MI*/0, /*OpIdx*/2, /*OtherMI*/2, /*OtherOpIdx*/1
+// CHECK:    // (srl:{ *:[i32] } (srl:{ *:[i32] } (add:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src0), i32:{ *:[i32] }:$src1), i32:{ *:[i32] }:$src0)
+// CHECK:   GIR_Done
+// CHECK: GIM_Try
+// CHECK:   GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK:   GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_SHL
+// CHECK:   GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK:   GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_MUL
+// CHECK:   GIM_CheckIsSameOperandIgnoreCopies, /*MI*/2, /*OpIdx*/2, /*OtherMI*/2, /*OtherOpIdx*/1
+// CHECK:   GIM_CheckIsSameOperandIgnoreCopies, /*MI*/0, /*OpIdx*/2, /*OtherMI*/2, /*OtherOpIdx*/1
+// CHECK:   // (srl:{ *:[i32] } (shl:{ *:[i32] } (mul:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src0), i32:{ *:[i32] }:$src1), i32:{ *:[i32] }:$src0)
+// CHECK:   GIR_Done
+// CHECK: GIM_Try
+// CHECK:   GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK:   GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_MUL
+// CHECK:   GIM_CheckIsSameOperandIgnoreCopies, /*MI*/1, /*OpIdx*/2, /*OtherMI*/1, /*OtherOpIdx*/1
+// CHECK:   // (sub:{ *:[i32] } (mul:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src0), i32:{ *:[i32] }:$src1)  =>  (InstThreeOperands:{ *:[i32] } GPR32:{ *:[i32] }:$src0, GPR32:{ *:[i32] }:$src1, GPR32:{ *:[i32] }:$src1)
+// CHECK:   GIR_Done
 No newline at end of file

llvm/utils/TableGen/CodeGenDAGPatterns.h

class TreePatternNode {
   std::vector<TreePredicateCall> PredicateCalls;
 
   /// TransformFn - The transformation function to execute on this node before
   /// it can be substituted into the resulting instruction on a pattern match.
   Record *TransformFn;
 
   std::vector<TreePatternNodePtr> Children;
 
+  /// If this was instantiated from a PatFrag node, and the PatFrag was derived
+  /// from "GISelFlags": the original Record derived from GISelFlags.
+  const Record *GISelFlags = nullptr;
+
 public:
   TreePatternNode(Record *Op, std::vector<TreePatternNodePtr> Ch,
                   unsigned NumResults)
       : Operator(Op), Val(nullptr), TransformFn(nullptr),
         Children(std::move(Ch)) {
     Types.resize(NumResults);
     ResultPerm.resize(NumResults);
     std::iota(ResultPerm.begin(), ResultPerm.end(), 0);
public:
   /// TreeHasProperty - Return true if any node in this tree has the specified
   /// property.
   bool TreeHasProperty(SDNP Property, const CodeGenDAGPatterns &CGP) const;
 
   /// isCommutativeIntrinsic - Return true if the node is an intrinsic which is
   /// marked isCommutative.
   bool isCommutativeIntrinsic(const CodeGenDAGPatterns &CDP) const;
 
+  void setGISelFlagsRecord(const Record *R) { GISelFlags = R; }
+  const Record *getGISelFlagsRecord() const { return GISelFlags; }
+
   void print(raw_ostream &OS) const;
   void dump() const;
 
 public:   // Higher level manipulation routines.
 
   /// clone - Return a new copy of this tree.
   ///
   TreePatternNodePtr clone() const;

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

for (unsigned i = 0, e = getNumChildren(); i != e; ++i)
       CChildren.push_back(getChild(i)->clone());
     New = std::make_shared<TreePatternNode>(getOperator(), std::move(CChildren),
                                             getNumTypes());
   }
   New->setName(getName());
   New->setNamesAsPredicateArg(getNamesAsPredicateArg());
   New->Types = Types;
   New->setPredicateCalls(getPredicateCalls());
+  New->setGISelFlagsRecord(getGISelFlagsRecord());
   New->setTransformFn(getTransformFn());
   return New;
 }
 
 /// RemoveAllTypes - Recursively strip all the types of this tree.
 void TreePatternNode::RemoveAllTypes() {
   // Reset to unknown type.
   std::fill(Types.begin(), Types.end(), TypeSetByHwMode());
do {
         NewChildren.push_back(ChildAlternatives[i][Idxs[i]]);
       TreePatternNodePtr R = std::make_shared<TreePatternNode>(
           getOperator(), std::move(NewChildren), getNumTypes());
 
       // Copy over properties.
       R->setName(getName());
       R->setNamesAsPredicateArg(getNamesAsPredicateArg());
       R->setPredicateCalls(getPredicateCalls());
+      R->setGISelFlagsRecord(getGISelFlagsRecord());
       R->setTransformFn(getTransformFn());
       for (unsigned i = 0, e = getNumTypes(); i != e; ++i)
         R->setType(i, getExtType(i));
       for (unsigned i = 0, e = getNumResults(); i != e; ++i)
         R->setResultIndex(i, getResultIndex(i));
 
       // Register alternative.
       OutAlternatives.push_back(R);
if (Frag->getNumArgs())
       FragTree->SubstituteFormalArguments(ArgMap);
 
     // Transfer types.  Note that the resolved alternative may have fewer
     // (but not more) results than the PatFrags node.
     FragTree->setName(getName());
     for (unsigned i = 0, e = FragTree->getNumTypes(); i != e; ++i)
       FragTree->UpdateNodeType(i, getExtType(i), TP);
 
+    if (Op->isSubClassOf("GISelFlags"))
+      FragTree->setGISelFlagsRecord(Op);
+
     // Transfer in the old predicates.
     for (const TreePredicateCall &Pred : getPredicateCalls())
       FragTree->addPredicateCall(Pred);
 
     // The fragment we inlined could have recursive inlining that is needed.  See
     // if there are any pattern fragments in it and inline them as needed.
     FragTree->InlinePatternFragments(FragTree, TP, OutAlternatives);
   }
for (unsigned i = 0, e = ChildVariants.size(); i != e; ++i)
       NewChildren.push_back(ChildVariants[i][Idxs[i]]);
     TreePatternNodePtr R = std::make_shared<TreePatternNode>(
         Orig->getOperator(), std::move(NewChildren), Orig->getNumTypes());
 
     // Copy over properties.
     R->setName(Orig->getName());
     R->setNamesAsPredicateArg(Orig->getNamesAsPredicateArg());
     R->setPredicateCalls(Orig->getPredicateCalls());
+    R->setGISelFlagsRecord(Orig->getGISelFlagsRecord());
     R->setTransformFn(Orig->getTransformFn());
     for (unsigned i = 0, e = Orig->getNumTypes(); i != e; ++i)
       R->setType(i, Orig->getExtType(i));
 
     // If this pattern cannot match, do not include it as a variant.
     std::string ErrString;
     // Scan to see if this pattern has already been emitted.  We can get
     // duplication due to things like commuting:

llvm/utils/TableGen/GlobalISelEmitter.cpp

 #include "CodeGenInstruction.h"
 #include "SubtargetFeatureInfo.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Support/CodeGenCoverage.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/MachineValueType.h"
+#include "llvm/Support/SaveAndRestore.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 #include <numeric>
 #include <string>
 using namespace llvm;
 

 }
 
 //===- Matchers -----------------------------------------------------------===//
 
 class OperandMatcher;
 class MatchAction;
 class PredicateMatcher;
 
+enum {
+  GISF_IgnoreCopies = 0x1,
+};
+
+using GISelFlags = std::uint16_t;
+
 class Matcher {
 public:
   virtual ~Matcher() = default;
   virtual void optimize() {}
   virtual void emit(MatchTable &Table) = 0;
 
   virtual bool hasFirstCondition() const = 0;
   virtual const PredicateMatcher &getFirstCondition() const = 0;
protected:
   unsigned NextInsnVarID;
 
   /// ID for the next output instruction allocated with allocateOutputInsnID()
   unsigned NextOutputInsnID;
 
   /// ID for the next temporary register ID allocated with allocateTempRegID()
   unsigned NextTempRegID;
 
+  /// Current GISelFlags
+  GISelFlags Flags = 0;
+
   std::vector<Record *> RequiredFeatures;
   std::vector<std::unique_ptr<PredicateMatcher>> EpilogueMatchers;
 
   ArrayRef<SMLoc> SrcLoc;
 
   typedef std::tuple<Record *, unsigned, unsigned>
       DefinedComplexPatternSubOperand;
   typedef StringMap<DefinedComplexPatternSubOperand>
       DefinedComplexPatternSubOperandMap;
   /// A map of Symbolic Names to ComplexPattern sub-operands.
   DefinedComplexPatternSubOperandMap ComplexSubOperands;
   /// A map used to for multiple referenced error check of ComplexSubOperand.
   /// ComplexSubOperand can't be referenced multiple from different operands,
   /// however multiple references from same operand are allowed since that is
   /// how 'same operand checks' are generated.
   StringMap<std::string> ComplexSubOperandsParentName;
 
   uint64_t RuleID;
   static uint64_t NextRuleID;
 
+  GISelFlags updateGISelFlag(GISelFlags CurFlags, const Record *R,
+                             StringRef FlagName, GISelFlags FlagBit) {
+    // If the value of a flag is unset, ignore it.
+    // If it's set, it always takes precedence over the existing value so
+    // clear/set the corresponding bit.
+    bool Unset = false;
+    bool Value = R->getValueAsBitOrUnset("GIIgnoreCopies", Unset);
+    if (!Unset)
+      return Value ? (CurFlags | FlagBit) : (CurFlags & ~FlagBit);
+    return CurFlags;
+  }
+
 public:
   RuleMatcher(ArrayRef<SMLoc> SrcLoc)
       : NextInsnVarID(0), NextOutputInsnID(0), NextTempRegID(0), SrcLoc(SrcLoc),
         RuleID(NextRuleID++) {}
   RuleMatcher(RuleMatcher &&Other) = default;
   RuleMatcher &operator=(RuleMatcher &&Other) = default;
 
   uint64_t getRuleID() const { return RuleID; }
 
   InstructionMatcher &addInstructionMatcher(StringRef SymbolicName);
   void addRequiredFeature(Record *Feature);
   const std::vector<Record *> &getRequiredFeatures() const;
 
   template <class Kind, class... Args> Kind &addAction(Args &&... args);
   template <class Kind, class... Args>
   action_iterator insertAction(action_iterator InsertPt, Args &&... args);
 
+  // Update the active GISelFlags based on the GISelFlags Record R.
+  // A SaveAndRestore object is returned so the old GISelFlags are restored
+  // at the end of the scope.
+  SaveAndRestore<GISelFlags> setGISelFlags(const Record *R) {
+    if (!R || !R->isSubClassOf("GISelFlags"))
+      return {Flags, Flags};
+
+    assert((R->isSubClassOf("PatFrags") || R->isSubClassOf("Pattern")) &&
+           "GISelFlags is only expected on Pattern/PatFrags!");
+
+    GISelFlags NewFlags =
+        updateGISelFlag(Flags, R, "GIIgnoreCopies", GISF_IgnoreCopies);
+    return {Flags, NewFlags};
+  }
+
+  GISelFlags getGISelFlags() const { return Flags; }
+
   /// Define an instruction without emitting any code to do so.
   unsigned implicitlyDefineInsnVar(InstructionMatcher &Matcher);
 
   unsigned getInsnVarID(InstructionMatcher &InsnMatcher) const;
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_begin() const {
     return InsnVariableIDs.begin();
   }
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_end() const {

 }
 
 /// Generates code to check that a register operand is defined by the same exact
 /// one as another.
 class SameOperandMatcher : public OperandPredicateMatcher {
   std::string MatchingName;
   unsigned OrigOpIdx;
 
+  GISelFlags Flags;
+
 public:
   SameOperandMatcher(unsigned InsnVarID, unsigned OpIdx, StringRef MatchingName,
-                     unsigned OrigOpIdx)
+                     unsigned OrigOpIdx, GISelFlags Flags)
       : OperandPredicateMatcher(OPM_SameOperand, InsnVarID, OpIdx),
-        MatchingName(MatchingName), OrigOpIdx(OrigOpIdx) {}
+        MatchingName(MatchingName), OrigOpIdx(OrigOpIdx), Flags(Flags) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_SameOperand;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override;
 

 /// would use an InstructionOperandMatcher for operand 1 of the G_MUL to match
 /// the:
 ///   (G_ADD $src1, $src2)
 /// subpattern.
 class InstructionOperandMatcher : public OperandPredicateMatcher {
 protected:
   std::unique_ptr<InstructionMatcher> InsnMatcher;
 
+  GISelFlags Flags;
+
 public:
   InstructionOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             RuleMatcher &Rule, StringRef SymbolicName,
                             bool NumOpsCheck = true)
       : OperandPredicateMatcher(OPM_Instruction, InsnVarID, OpIdx),
-        InsnMatcher(new InstructionMatcher(Rule, SymbolicName, NumOpsCheck)) {}
+        InsnMatcher(new InstructionMatcher(Rule, SymbolicName, NumOpsCheck)),
+        Flags(Rule.getGISelFlags()) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Instruction;
   }
 
   InstructionMatcher &getInsnMatcher() const { return *InsnMatcher; }
 
   void emitCaptureOpcodes(MatchTable &Table, RuleMatcher &Rule) const {
     const unsigned NewInsnVarID = InsnMatcher->getInsnVarID();
-    Table << MatchTable::Opcode("GIM_RecordInsn")
+    const bool IgnoreCopies = Flags & GISF_IgnoreCopies;
+    Table << MatchTable::Opcode(IgnoreCopies ? "GIM_RecordInsnIgnoreCopies"
+                                             : "GIM_RecordInsn")
           << MatchTable::Comment("DefineMI")
           << MatchTable::IntValue(NewInsnVarID) << MatchTable::Comment("MI")
           << MatchTable::IntValue(getInsnVarID())
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(getOpIdx())
           << MatchTable::Comment("MIs[" + llvm::to_string(NewInsnVarID) + "]")
           << MatchTable::LineBreak;
   }
 

 void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
   if (DefinedOperands.find(SymbolicName) == DefinedOperands.end()) {
     DefinedOperands[SymbolicName] = &OM;
     return;
   }
 
   // If the operand is already defined, then we must ensure both references in
   // the matcher have the exact same node.
+  RuleMatcher &RM = OM.getInstructionMatcher().getRuleMatcher();
   OM.addPredicate<SameOperandMatcher>(
-      OM.getSymbolicName(), getOperandMatcher(OM.getSymbolicName()).getOpIdx());
+      OM.getSymbolicName(), getOperandMatcher(OM.getSymbolicName()).getOpIdx(),
+      RM.getGISelFlags());
 }
 
 void RuleMatcher::definePhysRegOperand(Record *Reg, OperandMatcher &OM) {
   if (PhysRegOperands.find(Reg) == PhysRegOperands.end()) {
     PhysRegOperands[Reg] = &OM;
     return;
   }
 }
bool OperandPredicateMatcher::isHigherPriorityThan(
   return Kind < B.Kind;
 }
 
 void SameOperandMatcher::emitPredicateOpcodes(MatchTable &Table,
                                               RuleMatcher &Rule) const {
   const OperandMatcher &OtherOM = Rule.getOperandMatcher(MatchingName);
   unsigned OtherInsnVarID = Rule.getInsnVarID(OtherOM.getInstructionMatcher());
   assert(OtherInsnVarID == OtherOM.getInstructionMatcher().getInsnVarID());
-
-  Table << MatchTable::Opcode("GIM_CheckIsSameOperand")
+  const bool IgnoreCopies = Flags & GISF_IgnoreCopies;
+  Table << MatchTable::Opcode(IgnoreCopies
+                                  ? "GIM_CheckIsSameOperandIgnoreCopies"
+                                  : "GIM_CheckIsSameOperand")
         << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
         << MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
         << MatchTable::Comment("OtherMI")
         << MatchTable::IntValue(OtherInsnVarID)
         << MatchTable::Comment("OtherOpIdx")
-        << MatchTable::IntValue(OtherOM.getOpIdx())
-        << MatchTable::LineBreak;
+        << MatchTable::IntValue(OtherOM.getOpIdx()) << MatchTable::LineBreak;
 }
 
 //===- GlobalISelEmitter class --------------------------------------------===//
 
 static Expected<LLTCodeGen> getInstResultType(const TreePatternNode *Dst) {
   ArrayRef<TypeSetByHwMode> ChildTypes = Dst->getExtTypes();
   if (ChildTypes.size() != 1)
     return failedImport("Dst pattern child has multiple results");
Expected<InstructionMatcher &> GlobalISelEmitter::addBuiltinPredicates(
   }
   HasAddedMatcher = false;
   return InsnMatcher;
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
     RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
     const TreePatternNode *Src, unsigned &TempOpIdx) {
+  const auto SavedFlags = Rule.setGISelFlags(Src->getGISelFlagsRecord());
+
   Record *SrcGIEquivOrNull = nullptr;
   const CodeGenInstruction *SrcGIOrNull = nullptr;
 
   // Start with the defined operands (i.e., the results of the root operator).
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (isa<IntInit>(SrcInit)) {
       InsnMatcher.addPredicate<InstructionOpcodeMatcher>(
Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // naming being the same. One possible solution would be to have
   // explicit operator for operation capture and reference those.
   // The plus side is that it would expose opportunities to share
   // the capture accross rules. The downside is that it would
   // introduce a dependency between predicates (captures must happen
   // before their first use.)
   InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
   unsigned TempOpIdx = 0;
+
+  const auto SavedFlags = M.setGISelFlags(P.getSrcRecord());
+
   auto InsnMatcherOrError =
       createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
   if (auto Error = InsnMatcherOrError.takeError())
     return std::move(Error);
   InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
 
   if (Dst->isLeaf()) {
     Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());