[RFC][GISel] Add a way to ignore COPY instructions in InstructionSelector

[Pierre-vh]

Summary

RFC to add a way to ignore COPY instructions when pattern-matching MIR in GISel.

• Add a new "GISelFlags" class to TableGen. Both Pattern and PatFrags defs can use it to alter matching behaviour.
• Flags start at zero and are scoped: the setter returns a SaveAndRestore object so that when the current scope ends, the flags are restored to their previous values. This allows child patterns to modify the flags without affecting the parent pattern.
• Child patterns always reuse the parent's pattern, but they can override its values. For more examples, see GlobalISelEmitterFlags.td tests.
• [AMDGPU] Use the IgnoreCopies flag in BFI patterns, which are known to be bothered by cross-regbank copies.

[Pierre-vh]

Rebase, rework

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

5231

Would it be be possible to set this on a PatFrag level instead of a top-level pattern?

[Pierre-vh]

• Add test

@arsenm for the PatFrags it looks like they don't (always) generate a Try block so the system as-is cannot handle them.
I was thinking of making GIM_SetFlag work only per-instruction and be used as a prefix, so instead of having

GIM_Try ...
  GIM_SetFlag GIMSF_IgnoreCopies

to set flags for a whole block we'd have

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_Try ... # Sets the flag on the next instruction -> Try saves them in the whole block

Then we could have flags set for a single instruction which would allow us to support PatFrags and virtually anything else
after a bit more backend work:

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_RecordInsn ...

[Pierre-vh]

Current design doesn't work on PatFrags as they are inlined. I'm looking for solutions.

[Pierre-vh]

Rework so the system works on both Pattern and PatFrags

[Pierre-vh]

Rebase

This is ready for review, I don't have anymore changes to make for now.

[Petar.Avramovic]

I like the GISelFlags, looks pretty lightweight.
Are you planning to add more flags?
About IgnoreCopies, it looks correct to me. Considering the possibility to break constant bus limit, patch synergies well with GISelPredicateCode that checks for it.
Can you add look through copies to Three Op Pats from VOP3 td file?
Upcoming regbankselect changes should be able to deal with copies but nevertheless I would like to see this one merged.
LGTM but I would wait for feedback from other reviewers.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstructions.td

2046–2054

I think it's cleaner to hide these in an AMDGPUPat subclass, like AMDGPUPatIgnoreCopies

[Pierre-vh]

Address comment

Can this land (once the parent diff does) or does this need to wait for more people to approve?

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

public:
 
   PredicateBitsetImpl(std::initializer_list<unsigned> Init) {
     for (auto I : Init)
       std::bitset<MaxPredicates>::set(I);
   }
 };
 
 enum {
+  /// Ignore COPY when looking at operands.
+  /// Affects:
+  ///   GIM_RecordIsns
+  ///   GIM_CheckIsSameOperand
+  GIMSF_IgnoreCopies = 0x1,
+};
+
+enum {
   /// Begin a try-block to attempt a match and jump to OnFail if it is
   /// unsuccessful.
   /// - OnFail - The MatchTable entry at which to resume if the match fails.
   ///
   /// FIXME: This ought to take an argument indicating the number of try-blocks
   ///        to exit on failure. It's usually one but the last match attempt of
   ///        a block will need more. The (implemented) alternative is to tack a
   ///        GIM_Reject on the end of each try-block which is simpler but
   ///        requires an extra opcode and iteration in the interpreter on each
   ///        failed match.
   GIM_Try,
 
+  /// Sets SelectorFlags (GIMSF_*) to alter instruction selection behaviour in
+  /// the
+  /// current try-block.
+  /// Note that the change is propagated downwards to nested try-blocks, but
+  /// never to parent try-blocks. Flags are saved on a stack and new try-blocks
+  /// duplicate the latest stack entry, and when a try-block ends (is rejected),
+  /// the stack is popped once.
+  /// - Flag - SelectorFlag to set (GIMSF_*)
+  GIM_SetFlag,
+
   /// Switch over the opcode on the specified instruction
   /// - InsnID - Instruction ID
   /// - LowerBound - numerically minimum opcode supported
   /// - UpperBound - numerically maximum + 1 opcode supported
   /// - Default - failure jump target
   /// - JumpTable... - (UpperBound - LowerBound) (at least 2) jump targets
   GIM_SwitchOpcode,
 
enum {
   /// - OldInsnID - Instruction ID to get the matched operand from
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomRenderer,
 
   /// Render operands to the specified instruction using a custom function,
   /// reading from a specific operand.
   /// - InsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to get the matched operand from
-  /// - OpIdx - Operand index in OldInsnID the render function should read from..
+  /// - OpIdx - Operand index in OldInsnID the render function should read
+  /// from..
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomOperandRenderer,
 
   /// Render a G_CONSTANT operator as a sign-extended immediate.
   /// - NewInsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to copy from
   /// The operand index is implicitly 1.
   GIR_CopyConstantAsSImm,

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

bool InstructionSelector::executeMatchTable(
     const int64_t *MatchTable, const TargetInstrInfo &TII,
     MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
     const RegisterBankInfo &RBI, const PredicateBitset &AvailableFeatures,
     CodeGenCoverage &CoverageInfo) const {
 
   uint64_t CurrentIdx = 0;
   SmallVector<uint64_t, 4> OnFailResumeAt;
 
+  // Keep track of selector flags per try-block. Keep an empty value at the
+  // bottom of the stack so calling .back() is always valid.
+  SmallVector<uint64_t, 4> SelectorFlags = {0};
+
   // Bypass the flag check on the instruction, and only look at the MCInstrDesc.
   bool NoFPException = !State.MIs[0]->getDesc().mayRaiseFPException();
 
   const uint16_t Flags = State.MIs[0]->getFlags();
 
   enum RejectAction { RejectAndGiveUp, RejectAndResume };
+
+  auto handleTryBlock = [&](uint64_t ResumeAt) {
+    OnFailResumeAt.push_back(ResumeAt);
+    SelectorFlags.push_back(SelectorFlags.back());
+  };
+
   auto handleReject = [&]() -> RejectAction {
     DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                     dbgs() << CurrentIdx << ": Rejected\n");
     if (OnFailResumeAt.empty())
       return RejectAndGiveUp;
     CurrentIdx = OnFailResumeAt.pop_back_val();
+    (void)SelectorFlags.pop_back_val();
     DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
-                    dbgs() << CurrentIdx << ": Resume at " << CurrentIdx << " ("
+                    dbgs() << CurrentIdx << ": Resume at " << CurrentIdx
+                           << " (SelectorFlags: "
+                           << format_hex(SelectorFlags.back(), 8) << ", "
                            << OnFailResumeAt.size() << " try-blocks remain)\n");
     return RejectAndResume;
   };
 
   auto propagateFlags = [=](NewMIVector &OutMIs) {
     for (auto MIB : OutMIs) {
       // Set the NoFPExcept flag when no original matched instruction could
       // raise an FP exception, but the new instruction potentially might.
       uint16_t MIBFlags = Flags;
       if (NoFPException && MIB->mayRaiseFPException())
         MIBFlags |= MachineInstr::NoFPExcept;
       MIB.setMIFlags(MIBFlags);
     }
 
     return true;
   };
 
   while (true) {
     assert(CurrentIdx != ~0u && "Invalid MatchTable index");
     int64_t MatcherOpcode = MatchTable[CurrentIdx++];
     switch (MatcherOpcode) {
     case GIM_Try: {
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": Begin try-block\n");
-      OnFailResumeAt.push_back(MatchTable[CurrentIdx++]);
+      handleTryBlock(MatchTable[CurrentIdx++]);
+      break;
+    }
+    case GIM_SetFlag: {
+      uint64_t Value = uint64_t(MatchTable[CurrentIdx++]);
+      DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
+                      dbgs() << CurrentIdx << ": GIM_SetFlag "
+                             << format_hex(Value, 8) << ": "
+                             << format_hex(SelectorFlags.back(), 8) << " -> ");
+      SelectorFlags.back() |= Value;
+      DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
+                      dbgs() << format_hex(SelectorFlags.back(), 8) << "\n");
       break;
     }
-
     case GIM_RecordInsn: {
       int64_t NewInsnID = MatchTable[CurrentIdx++];
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
 
       // As an optimisation we require that MIs[0] is always the root. Refuse
       // any attempt to modify it.
       assert(NewInsnID != 0 && "Refusing to modify MIs[0]");
case GIM_RecordInsn: {
       if (MO.getReg().isPhysical()) {
         DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                         dbgs() << CurrentIdx << ": Is a physical register\n");
         if (handleReject() == RejectAndGiveUp)
           return false;
         break;
       }
 
-      MachineInstr *NewMI = MRI.getVRegDef(MO.getReg());
+      MachineInstr *NewMI;
+      if (SelectorFlags.back() & GIMSF_IgnoreCopies)
+        NewMI = getDefIgnoringCopies(MO.getReg(), MRI);
+      else
+        NewMI = MRI.getVRegDef(MO.getReg());
+
       if ((size_t)NewInsnID < State.MIs.size())
         State.MIs[NewInsnID] = NewMI;
       else {
         assert((size_t)NewInsnID == State.MIs.size() &&
                "Expected to store MIs in order");
         State.MIs.push_back(NewMI);
       }
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
case GIM_SwitchOpcode: {
         CurrentIdx = Default;
         break;
       }
       CurrentIdx = MatchTable[CurrentIdx + (Opcode - LowerBound)];
       if (!CurrentIdx) {
         CurrentIdx = Default;
         break;
       }
-      OnFailResumeAt.push_back(Default);
+      handleTryBlock(Default);
       break;
     }
 
     case GIM_SwitchType: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
       int64_t LowerBound = MatchTable[CurrentIdx++];
       int64_t UpperBound = MatchTable[CurrentIdx++];
case GIM_SwitchType: {
         CurrentIdx = Default;
         break;
       }
       CurrentIdx = MatchTable[CurrentIdx + (TypeID - LowerBound)];
       if (!CurrentIdx) {
         CurrentIdx = Default;
         break;
       }
-      OnFailResumeAt.push_back(Default);
+      handleTryBlock(Default);
       break;
     }
 
     case GIM_CheckNumOperands: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t Expected = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_CheckNumOperands(MIs["
case GIM_CheckIsSameOperand: {
       int64_t OtherInsnID = MatchTable[CurrentIdx++];
       int64_t OtherOpIdx = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_CheckIsSameOperand(MIs["
                              << InsnID << "][" << OpIdx << "], MIs["
                              << OtherInsnID << "][" << OtherOpIdx << "])\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       assert(State.MIs[OtherInsnID] != nullptr && "Used insn before defined");
-      if (!State.MIs[InsnID]->getOperand(OpIdx).isIdenticalTo(
-              State.MIs[OtherInsnID]->getOperand(OtherOpIdx))) {
+
+      MachineOperand &Op = State.MIs[InsnID]->getOperand(OpIdx);
+      MachineOperand &OtherOp = State.MIs[OtherInsnID]->getOperand(OtherOpIdx);
+
+      if (SelectorFlags.back() & GIMSF_IgnoreCopies) {
+        if (Op.isReg() && OtherOp.isReg()) {
+          MachineInstr *MI = getDefIgnoringCopies(Op.getReg(), MRI);
+          MachineInstr *OtherMI = getDefIgnoringCopies(OtherOp.getReg(), MRI);
+          if (MI && MI == OtherMI)
+            break;
+        }
+      }
+
+      if (!Op.isIdenticalTo(OtherOp)) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
     case GIM_Reject:
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_Reject\n");

llvm/include/llvm/Target/TargetSelectionDAG.td

 // their GPRs).
 //
 
 class Pattern<dag patternToMatch, list<dag> resultInstrs> {
   dag             PatternToMatch  = patternToMatch;
   list<dag>       ResultInstrs    = resultInstrs;
   list<Predicate> Predicates      = [];  // See class Instruction in Target.td.
   int             AddedComplexity = 0;   // See class Instruction in Target.td.
+
+  // When this is set, the GISel matcher will look through COPY instructions
+  // before checking opcodes for instruction operands.
+  bit GISelIgnoreCopies = ?;
 }
 
 // Pat - A simple (but common) form of a pattern, which produces a simple result
 // not needing a full list.
 class Pat<dag pattern, dag result> : Pattern<pattern, [result]>;
 
 //===----------------------------------------------------------------------===//
 // Complex pattern definitions.

llvm/lib/Target/AMDGPU/SIInstructions.td

def BFIImm32 : PatFrag<
   [{
     auto *X = dyn_cast<ConstantSDNode>(N->getOperand(0)->getOperand(1));
     auto *NotX = dyn_cast<ConstantSDNode>(N->getOperand(1)->getOperand(1));
     return X && NotX &&
       ~(unsigned)X->getZExtValue() == (unsigned)NotX->getZExtValue();
   }]
 >;
 
+let GISelIgnoreCopies = 1 in {
   // Definition from ISA doc:
   // (y & x) | (z & ~x)
   def : AMDGPUPat <
     (DivergentBinFrag<or> (and i32:$y, i32:$x), (and i32:$z, (not i32:$x))),
     (V_BFI_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
                   (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32),
                   (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32))
   >;
 
   // (y & C) | (z & ~C)
   def : AMDGPUPat <
     (BFIImm32 i32:$x, i32:$y, i32:$z),
     (V_BFI_B32_e64 VSrc_b32:$x, VSrc_b32:$y, VSrc_b32:$z)
   >;
 
   // 64-bit version
   def : AMDGPUPat <
     (DivergentBinFrag<or> (and i64:$y, i64:$x), (and i64:$z, (not i64:$x))),
     (REG_SEQUENCE VReg_64,
       (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub0)),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub0))), sub0,
       (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub1)),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub1))), sub1)
   >;
 
   // SHA-256 Ch function
   // z ^ (x & (y ^ z))
   def : AMDGPUPat <
     (DivergentBinFrag<xor> i32:$z, (and i32:$x, (xor i32:$y, i32:$z))),
     (V_BFI_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
                   (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32),
                   (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32))
   >;
 
   // 64-bit version
   def : AMDGPUPat <
     (DivergentBinFrag<xor> i64:$z, (and i64:$x, (xor i64:$y, i64:$z))),
     (REG_SEQUENCE VReg_64,
       (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub0)),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub0))), sub0,
       (V_BFI_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub1)),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub1))), sub1)
   >;
+}
 
 def : AMDGPUPat <
   (fcopysign f32:$src0, f32:$src1),
   (V_BFI_B32_e64 (S_MOV_B32 (i32 0x7fffffff)), $src0, $src1)
 >;
 
 def : AMDGPUPat <
   (fcopysign f32:$src0, f64:$src1),
def : AMDGPUPat <
   (DivergentBinFrag<sra> (shl_oneuse i32:$src, (sub 32, i32:$width)),
                          (sub 32, i32:$width)),
   (V_BFE_I32_e64 $src, (i32 0), $width)
 >;
 
 // SHA-256 Ma patterns
 
 // ((x & z) | (y & (x | z))) -> BFI (XOR x, y), z, y
+let GISelIgnoreCopies = 1 in {
   def : AMDGPUPat <
     (DivergentBinFrag<or> (and i32:$x, i32:$z),
                           (and i32:$y, (or i32:$x, i32:$z))),
     (V_BFI_B32_e64 (V_XOR_B32_e64 (COPY_TO_REGCLASS VSrc_b32:$x, VGPR_32),
                                   (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32)),
                   (COPY_TO_REGCLASS VSrc_b32:$z, VGPR_32),
                   (COPY_TO_REGCLASS VSrc_b32:$y, VGPR_32))
   >;
 
   def : AMDGPUPat <
     (DivergentBinFrag<or> (and i64:$x, i64:$z),
                           (and i64:$y, (or i64:$x, i64:$z))),
     (REG_SEQUENCE VReg_64,
       (V_BFI_B32_e64 (V_XOR_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub0)),
                       (i32 (EXTRACT_SUBREG VReg_64:$y, sub0))),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub0)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub0))), sub0,
       (V_BFI_B32_e64 (V_XOR_B32_e64 (i32 (EXTRACT_SUBREG VReg_64:$x, sub1)),
                       (i32 (EXTRACT_SUBREG VReg_64:$y, sub1))),
                 (i32 (EXTRACT_SUBREG VReg_64:$z, sub1)),
                 (i32 (EXTRACT_SUBREG VReg_64:$y, sub1))), sub1)
   >;
+}
 
 multiclass IntMed3Pat<Instruction med3Inst,
                  SDPatternOperator min,
                  SDPatternOperator max> {
 
   // This matches 16 permutations of
   // min(max(a, b), max(min(a, b), c))
   def : AMDGPUPat <

llvm/test/CodeGen/AMDGPU/bfi_int.ll

 ;
 ; GFX10-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, v0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    s_xor_b32 s0, s0, s1
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX8-GISEL-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s1
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: v_s_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    s_xor_b32 s0, s0, s1
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, v0, s0, s1
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

 ;
 ; GFX10-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, s0, v0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX8-GISEL-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, s0, v0, v1
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: s_v_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, s0, v0
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s1, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, s0, v0, s1
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

 ;
 ; GFX10-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-GISEL-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX8-GISEL:       ; %bb.0: ; %entry
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v1, s0, v1
-; GFX8-GISEL-NEXT:    v_and_b32_e32 v0, v0, v1
-; GFX8-GISEL-NEXT:    v_xor_b32_e32 v0, s0, v0
+; GFX8-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX8-GISEL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-GISEL-LABEL: v_v_s_bfi_sha256_ch:
 ; GFX10-GISEL:       ; %bb.0: ; %entry
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v1, s0, v1
-; GFX10-GISEL-NEXT:    v_and_b32_e32 v0, v0, v1
-; GFX10-GISEL-NEXT:    v_xor_b32_e32 v0, s0, v0
+; GFX10-GISEL-NEXT:    v_bfi_b32 v0, v0, v1, s0
 ; GFX10-GISEL-NEXT:    ; return to shader part epilog
 entry:
   %xor0 = xor i32 %y, %z
   %and = and i32 %x, %xor0
   %xor1 = xor i32 %z, %and
   %cast = bitcast i32 %xor1 to float
   ret float %cast
 }

llvm/test/TableGen/GlobalISelEmitterCustomPredicate.td

def and_or_pat : PatFrag<
   (ops node:$x, node:$y, node:$z),
   (and (or node:$x, node:$y), node:$z), [{ return foo(); }]> {
   let GISelPredicateCode = [{
     return doesComplexCheck(MI);
   }];
   let PredicateCodeUsesOperands = 1;
 }
 
-// CHECK: GIM_Try, /*On fail goto*//*Label 0*/ 99, // Rule ID 6 //
+// CHECK: GIM_Try, /*On fail goto*//*Label 0*/ 99, // Rule ID 7 //
 // CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/3,
 // CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
 // CHECK-NEXT: // MIs[0] dst
 // CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
 // CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/0, /*Op*/0, /*RC*/Test::DRegsRegClassID,
 // CHECK-NEXT: // MIs[0] src2
 // CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
 // CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/0, /*Op*/1, /*StoreIdx*/2, // Name : pred:3:z

 }
 
 // CHECK: GIM_Try, /*On fail goto*//*Label 3*/ 372, // Rule ID 1 //
 // CHECK: // (xor:{ *:[i32] } (add:{ *:[i32] } i32:{ *:[i32] }:$src0:$pred:2:x, i32:{ *:[i32] }:$src1:$pred:2:y), i32:{ *:[i32] }:$src2:$pred:2:z)<<P:2:Predicate_patfrags_test_pat>>  =>  (PATFRAGS:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src1, i32:{ *:[i32] }:$src2)
 
 // CHECK: GIM_Try, /*On fail goto*//*Label 4*/ 459, // Rule ID 2 //
 // CHECK: // (xor:{ *:[i32] } (sub:{ *:[i32] } i32:{ *:[i32] }:$src0:$pred:2:x, i32:{ *:[i32] }:$src1:$pred:2:y), i32:{ *:[i32] }:$src2:$pred:2:z)<<P:2:Predicate_patfrags_test_pat>>  =>  (PATFRAGS:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src1, i32:{ *:[i32] }:$src2)
 
-// CHECK: GIM_Try, /*On fail goto*//*Label 5*/ 546, // Rule ID 4 //
+// CHECK: GIM_Try, /*On fail goto*//*Label 5*/ 546, // Rule ID 5 //
 // CHECK: // (xor:{ *:[i32] } i32:{ *:[i32] }:$src2:$pred:2:z, (add:{ *:[i32] } i32:{ *:[i32] }:$src0:$pred:2:x, i32:{ *:[i32] }:$src1:$pred:2:y))<<P:2:Predicate_patfrags_test_pat>>  =>  (PATFRAGS:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src1, i32:{ *:[i32] }:$src2)
 
-// CHECK: GIM_Try, /*On fail goto*//*Label 6*/ 633, // Rule ID 5 //
+// CHECK: GIM_Try, /*On fail goto*//*Label 6*/ 633, // Rule ID 6 //
 // CHECK: // (xor:{ *:[i32] } i32:{ *:[i32] }:$src2:$pred:2:z, (sub:{ *:[i32] } i32:{ *:[i32] }:$src0:$pred:2:x, i32:{ *:[i32] }:$src1:$pred:2:y))<<P:2:Predicate_patfrags_test_pat>>  =>  (PATFRAGS:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src1, i32:{ *:[i32] }:$src2)
 
 
 // Test a commutative pattern using multiple patterns using PatFrags.
 def PATFRAGS : I<(outs DRegs:$dst),
   (ins DOP:$src0, DOP:$src1, DOP:$src2),
   [(set DRegs:$dst, (patfrags_test_pat i32:$src0, i32:$src1, i32:$src2))]
 >;
+
+// Test a pattern that uses GIMSF_* flags on the whole pattern.
+// CHECK:       GIM_Try, /*On fail goto*//*Label 7*/ 708, // Rule ID 4 //
+// CHECK-NEXT:    GIM_SetFlag, GIMSF_IgnoreCopies,
+// CHECK-NOT:     GIR_Done
+// CHECK:         // (shl:{ *:[i32] } (shl:{ *:[i32] } i32:{ *:[i32] }:$src0, i32:{ *:[i32] }:$src1), i32:{ *:[i32] }:$src0)  =>  (AND_OR:{ *:[i32] } DOP:{ *:[i32] }:$src0, DOP:{ *:[i32] }:$src1, DOP:{ *:[i32] }:$src1)
+// CHECK:         GIR_Done
+let GISelIgnoreCopies = 1 in
+def : Pat<
+  (i32 (shl (shl i32:$src0, i32:$src1), i32:$src0)),
+  (AND_OR DOP:$src0, DOP:$src1, DOP:$src1)
+>;
+
+// TODO
+// Test a pattern that uses GIMSF_* flags on a PatFrag.
+// let GISelIgnoreCopies = 1 in
+// def IgnoreCopiesSRL: PatFrag<
+//   (ops node:$src0, node:$src1),
+//   (srl $src0, $src1)>;
+//
+// def : Pat<
+//   (i32 (shl (IgnoreCopiesSRL i32:$src0, i32:$src1), i32:$src0)),
+//   (AND_OR DOP:$src1, DOP:$src1, DOP:$src0)
+// >;

llvm/utils/TableGen/GlobalISelEmitter.cpp

 
 //===- Matchers -----------------------------------------------------------===//
 
 class OperandMatcher;
 class MatchAction;
 class PredicateMatcher;
 
 class Matcher {
+  bool IgnoreCopies = false;
+
 public:
   virtual ~Matcher() = default;
   virtual void optimize() {}
   virtual void emit(MatchTable &Table) = 0;
 
   virtual bool hasFirstCondition() const = 0;
   virtual const PredicateMatcher &getFirstCondition() const = 0;
   virtual std::unique_ptr<PredicateMatcher> popFirstCondition() = 0;
+
+  void setIgnoreCopies(bool Value = true) { IgnoreCopies = Value; }
+  bool getIgnoreCopies() const { return IgnoreCopies; }
+
+  void emitSetFlags(MatchTable &Table) {
+    // Note: If more flags are added, we can just emit one SetFlag and & the
+    // flags together.
+    if (IgnoreCopies) {
+      Table << MatchTable::Opcode("GIM_SetFlag")
+            << MatchTable::NamedValue("GIMSF_IgnoreCopies")
+            << MatchTable::LineBreak;
+    }
+  }
 };
 
 MatchTable MatchTable::buildTable(ArrayRef<Matcher *> Rules,
                                   bool WithCoverage) {
   MatchTable Table(WithCoverage);
   for (Matcher *Rule : Rules)
     Rule->emit(Table);
 
void RuleMatcher::emit(MatchTable &Table) {
   assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
 
   unsigned LabelID = Table.allocateLabelID();
   Table << MatchTable::Opcode("GIM_Try", +1)
         << MatchTable::Comment("On fail goto")
         << MatchTable::JumpTarget(LabelID)
         << MatchTable::Comment(("Rule ID " + Twine(RuleID) + " //").str())
         << MatchTable::LineBreak;
+  emitSetFlags(Table);
 
   if (!RequiredFeatures.empty()) {
     Table << MatchTable::Opcode("GIM_CheckFeatures")
           << MatchTable::NamedValue(getNameForFeatureBitset(RequiredFeatures))
           << MatchTable::LineBreak;
   }
 
   Matchers.front()->emitPredicateOpcodes(Table, *this);
Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // naming being the same. One possible solution would be to have
   // explicit operator for operation capture and reference those.
   // The plus side is that it would expose opportunities to share
   // the capture accross rules. The downside is that it would
   // introduce a dependency between predicates (captures must happen
   // before their first use.)
   InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
   unsigned TempOpIdx = 0;
+
+  if (P.getSrcRecord()->isSubClassOf("Pattern")) {
+    bool Unset = false;
+    M.setIgnoreCopies(
+        P.getSrcRecord()->getValueAsBitOrUnset("GISelIgnoreCopies", Unset));
+  }
+
   auto InsnMatcherOrError =
       createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
   if (auto Error = InsnMatcherOrError.takeError())
     return std::move(Error);
   InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
 
   if (Dst->isLeaf()) {
     Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());
std::stable_sort(F, T, [](Matcher *A, Matcher *B) {
       auto *L = static_cast<RuleMatcher *>(A);
       auto *R = static_cast<RuleMatcher *>(B);
       return L->getFirstConditionAsRootType() <
              R->getFirstConditionAsRootType();
     });
     if (T != E)
       F = ++T;
   }
+
+  // Optimize flags: If all RuleMatchers have the same flag set, unset it and
+  // set it here instead so GIM_SetFlag is called only once.
+  if (!Matchers.empty()) {
+    const bool AllHaveIgnoreCopies =
+        all_of(Matchers, [&](Matcher *M) { return M->getIgnoreCopies(); });
+
+    if (AllHaveIgnoreCopies) {
+      setIgnoreCopies();
+      for (Matcher *M : Matchers)
+        M->setIgnoreCopies(false);
+    }
+  }
+
   GlobalISelEmitter::optimizeRules<GroupMatcher>(Matchers, MatcherStorage)
       .swap(Matchers);
   GlobalISelEmitter::optimizeRules<SwitchMatcher>(Matchers, MatcherStorage)
       .swap(Matchers);
 }
 
 void GlobalISelEmitter::run(raw_ostream &OS) {
   if (!UseCoverageFile.empty()) {

 
 void GroupMatcher::emit(MatchTable &Table) {
   unsigned LabelID = ~0U;
   if (!Conditions.empty()) {
     LabelID = Table.allocateLabelID();
     Table << MatchTable::Opcode("GIM_Try", +1)
           << MatchTable::Comment("On fail goto")
           << MatchTable::JumpTarget(LabelID) << MatchTable::LineBreak;
+    emitSetFlags(Table);
   }
   for (auto &Condition : Conditions)
     Condition->emitPredicateOpcodes(
         Table, *static_cast<RuleMatcher *>(*Matchers.begin()));
 
   for (const auto &M : Matchers)
     M->emit(Table);