AMDGPU/GlobalISel: Add tablegen operator that looks through copies

[Petar.Avramovic]

Summary

Introduce g_maybe_cross_reg_bank_copy, operator that can be used in patterns to ignore cross register bank copies.
Operator will be recognized by global-isel emitter and ignored by sdag emitter (sdag uses types not register banks).
Generates GIM_CheckOpcodeIgnoreCopies instead of GIM_CheckOpcode.
Does not affect complexity of the pattern.

[foad]

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

419

Can't this still be inside the class? I.e. move it down two or three lines?

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

2855–2856 ↗

(On Diff #286821)

Why not call AdjustInstrPostInstrSelectionBase on each MI in OutMIs?

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

4535 ↗

(On Diff #286821)

Forcing a function to be shared that refers to an SDNode is also pretty awkward

4538 ↗

(On Diff #286821)

I was trying to avoid doing things this way. This is selecting instructions incorrectly, and then relying on something else to fix them up. I think we should select trivially correct instructions, and a follow on pass could do a better job figuring out the optimal SGPR operands. This is why RegBankSelect is forcing everything to VGPRs for VALU instructions. The later pass will have an easier time if everything is expected to consistently be in VGPRs, rather than having to figure out if some operands are already scalar and moving them back to VGPRs if it decided another copy was better

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

4538 ↗

(On Diff #286821)

We're also incorrectly letting some constant bus violations through with ThreeOp_i32_Pats, which do need a complex predicate to avoid the constant bus violations. Technically the later folding pass could form these cases, but it would be nice to keep these in patterns. I haven't thought of a good solution for this yet

[foad]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

4538 ↗

(On Diff #286821)

How do we "select trivially correct instructions" if regbankselect has already run, and it has no foreknowledge of what complex patterns we might want to match as part of instruction selection? Some of the current failures to select v_lshl_add are because the inputs to the G_SHL are uniformn (or constant) and regbankselect doesn't know that we might want to combine it into a G_ADD whose other operand is divergent.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

4538 ↗

(On Diff #286821)

It's trivial if you look at a single instruction at a time. The constant bus violations only start showing up when you start looking through copies when pattern matching multiple input operations.

The patterns have to make sure they don't violate the constant bus restriction. As I mentioned, ThreeOp_i32_Pats is broken and really should fail to import. ThreeOpFrag needs a non-trivial predicate that uses PredicateCodeUsesOperands, but GlobalISelEmitter doesn't implement it (or even check if it's in use).

[foad]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

4538 ↗

(On Diff #286821)

It's trivial if you look at a single instruction at a time. The constant bus violations only start showing up when you start looking through copies when pattern matching multiple input operations.

The patterns have to make sure they don't violate the constant bus restriction. As I mentioned, ThreeOp_i32_Pats is broken and really should fail to import. ThreeOpFrag needs a non-trivial predicate that uses PredicateCodeUsesOperands, but GlobalISelEmitter doesn't implement it (or even check if it's in use).

4538 ↗

(On Diff #286821)

OK, makes sense. We can look at implementing PredicateCodeUsesOperands for GlobalISel in tablegen.

[Petar.Avramovic]

Attempt to teach emitter to collect operands that predicate with 'let PredicateCodeUsesOperands = 1' requires.

[Petar.Avramovic]

Move support for predicate code that uses operands to a separate patch.

[arsenm]

I think copies that can be ignored should be taken care of by a combiner pass, and the selector should not have to do this

[foad]

In D86294#2261887, @arsenm wrote:

I think copies that can be ignored should be taken care of by a combiner pass, and the selector should not have to do this

Can you explain how you think this should work for selecting e.g. v_add_lshl where some of the operands can be sgprs? A combiner pass can't remove cross-register-bank copies, can it?

[arsenm]

In D86294#2262832, @foad wrote:

In D86294#2261887, @arsenm wrote:

I think copies that can be ignored should be taken care of by a combiner pass, and the selector should not have to do this

Can you explain how you think this should work for selecting e.g. v_add_lshl where some of the operands can be sgprs? A combiner pass can't remove cross-register-bank copies, can it?

A combiner could, but under the current scheme it should not. I want to minimize the amount of places/code that need to worry about the constant bus restriction. The pattern for these instructions should search through the copies if it can. We may need a new tablegen operator or something to look through copies

[Petar.Avramovic]

Allow patterns to explicitly ask to look through copies.

[arsenm]

llvm/include/llvm/Target/TargetSelectionDAG.td

313

I think the name needs work. I don't think it should have a g_ prefix. Maybe ignore_copies?

[Pierre-vh]

@Petar.Avramovic this could be relevant for https://discourse.llvm.org/t/cleanly-addressing-globalisel-dag-pattern-matching-differences/64817/14, do you think you have time to resume work on this patch & get it landed, or may I take over it?
Thank you

[Petar.Avramovic]

You can take over. I am not so sure this will be useful as is. Just looking through copies will break constant bus limit. Then there are constants and their register banks.
I didn't like g_maybe_cross_reg_bank_copy it would be better to have some field in PatFrag recognized by the global-isel emitter (it would generate GIM_CheckOpcodeIgnoreCopies instead of GIM_CheckOpcode for all opcode checks in pattern).
But that can break constant bus limit. There is a mechanism to collect operands with
PredicateCodeUsesOperands = 1
Maybe you could add another field in PatFrag that will be a signal to generate jump-table entries that will go through collected operands and check constant bus limit for collected operands.
Just to clarify by have some field in PatFrag recognized by the global-isel emitter I meant something like HasNoUse but sdag emitter will ignore it.

[Pierre-vh]

D136234 landed and is an alternative implementation of this

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

enum {
   /// - Expected features
   GIM_CheckFeatures,
 
   /// Check the opcode on the specified instruction
   /// - InsnID - Instruction ID
   /// - Expected opcode
   GIM_CheckOpcode,
 
+  /// Check the opcode on the specified instruction, if instruction was copy
+  /// check the opcode of the instruction that was source of the copy.
+  /// - InsnID - Instruction ID
+  /// - Expected opcode
+  GIM_CheckOpcodeIgnoreCopies,
+
   /// Check the opcode on the specified instruction, checking 2 acceptable
   /// alternatives.
   /// - InsnID - Instruction ID
   /// - Expected opcode
   /// - Alternative expected opcode
   GIM_CheckOpcodeIsEither,
 
   /// Check the instruction has the right number of operands

 };
 
 enum {
   /// Indicates the end of the variable-length MergeInsnID list in a
   /// GIR_MergeMemOperands opcode.
   GIU_MergeMemOperands_EndOfList = -1,
 };
 
 /// Provides the logic to select generic machine instructions.
 class InstructionSelector {
 public:
   virtual ~InstructionSelector() = default;
 
   /// Select the (possibly generic) instruction \p I to only use target-specific
   /// opcodes. It is OK to insert multiple instructions, but they cannot be
   /// generic pre-isel instructions.
   ///

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

case GIM_CheckFeatures: {
           ISelInfo.FeatureBitsets[ExpectedBitsetID]) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
 
     case GIM_CheckOpcode:
+    case GIM_CheckOpcodeIgnoreCopies:
     case GIM_CheckOpcodeIsEither: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t Expected0 = MatchTable[CurrentIdx++];
       int64_t Expected1 = -1;
       if (MatcherOpcode == GIM_CheckOpcodeIsEither)
         Expected1 = MatchTable[CurrentIdx++];
 
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       unsigned Opcode = State.MIs[InsnID]->getOpcode();
+      if (MatcherOpcode == GIM_CheckOpcodeIgnoreCopies &&
+          Opcode == TargetOpcode::COPY) {
+        Register CopySrc = State.MIs[InsnID]->getOperand(1).getReg();
+        if (!CopySrc.isPhysical()) {
+          auto MI2 = State.MIs[InsnID] = getDefIgnoringCopies(CopySrc, MRI);
+          if (MI2) {
+            State.MIs[InsnID] = MI2;
+            Opcode = MI2->getOpcode();
+          }
+        }
+      }
 
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
         dbgs() << CurrentIdx << ": GIM_CheckOpcode(MIs[" << InsnID
         << "], ExpectedOpcode=" << Expected0;
         if (MatcherOpcode == GIM_CheckOpcodeIsEither)
           dbgs() << " || " << Expected1;
         dbgs() << ") // Got=" << Opcode << "\n";
       );

llvm/include/llvm/Target/TargetSelectionDAG.td

class SDNode<string opcode, SDTypeProfile typeprof,
   SDTypeProfile TypeProfile = typeprof;
 }
 
 // Special TableGen-recognized dag nodes
 def set;
 def implicit;
 def node;
 def srcvalue;
+// Used by global-isel, looks through cross register bank copies.
+// Does not increase complexity of the pattern. Ignored by sdag emitter.
+def g_maybe_cross_reg_bank_copy;
 
 def imm        : SDNode<"ISD::Constant"  , SDTIntLeaf , [], "ConstantSDNode">;
 def timm       : SDNode<"ISD::TargetConstant",SDTIntLeaf, [], "ConstantSDNode">;
 def fpimm      : SDNode<"ISD::ConstantFP", SDTFPLeaf  , [], "ConstantFPSDNode">;
 def vt         : SDNode<"ISD::VALUETYPE" , SDTOther   , [], "VTSDNode">;
 def bb         : SDNode<"ISD::BasicBlock", SDTOther   , [], "BasicBlockSDNode">;
 def cond       : SDNode<"ISD::CONDCODE"  , SDTOther   , [], "CondCodeSDNode">;
 def undef      : SDNode<"ISD::UNDEF"     , SDTUNDEF   , []>;

llvm/lib/Target/AMDGPU/VOP3Instructions.td

 
 } // End Predicates = [Has16BitInsts, isGFX10Plus]
 
 class ThreeOpFrag<SDPatternOperator op1, SDPatternOperator op2> : PatFrag<
   (ops node:$x, node:$y, node:$z),
   // When the inner operation is used multiple times, selecting 3-op
   // instructions may still be beneficial -- if the other users can be
   // combined similarly. Let's be conservative for now.
-  (op2 (HasOneUseBinOp<op1> node:$x, node:$y), node:$z),
+  (op2 (g_maybe_cross_reg_bank_copy (HasOneUseBinOp<op1> node:$x, node:$y)), node:$z),
   [{
     // Only use VALU ops when the result is divergent.
     if (!N->isDivergent())
       return false;
 
     // Check constant bus limitations.
     //
     // Note: Use !isDivergent as a conservative proxy for whether the value

llvm/test/CodeGen/AMDGPU/GlobalISel/add_shl.ll

 ; GFX9-LABEL: add_shl_vgpr_c:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_add_i32 s2, s2, s3
 ; GFX9-NEXT:    v_lshlrev_b32_e64 v0, v0, s2
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: add_shl_vgpr_c:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_add_i32 s2, s2, s3
+; GFX10-NEXT:    v_add_lshl_u32 v0, s2, s3, v0
 ; GFX10-NEXT:    ; implicit-def: $vcc_hi
-; GFX10-NEXT:    v_lshlrev_b32_e64 v0, v0, s2
 ; GFX10-NEXT:    ; return to shader part epilog
   %x = add i32 %a, %b
   %result = shl i32 %x, %c
   %bc = bitcast i32 %result to float
   ret float %bc
 }
 
 define amdgpu_ps float @add_shl_vgpr_ac(i32 %a, i32 inreg %b, i32 %c) {

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-pattern-and-or.mir

bb.0:
     ; GFX9: [[V_OR_B32_e64_:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[COPY3]], [[COPY2]], implicit $exec
     ; GFX9: S_ENDPGM 0, implicit [[V_OR_B32_e64_]]
     ; GFX10-LABEL: name: and_or_s32_sgpr_sgpr_vgpr
     ; GFX10: liveins: $sgpr0, $sgpr1, $vgpr0
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; GFX10: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr1
     ; GFX10: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX10: [[S_AND_B32_:%[0-9]+]]:sreg_32 = S_AND_B32 [[COPY]], [[COPY1]], implicit-def $scc
-    ; GFX10: [[COPY3:%[0-9]+]]:vgpr_32 = COPY [[S_AND_B32_]]
-    ; GFX10: [[V_OR_B32_e64_:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[COPY3]], [[COPY2]], implicit $exec
-    ; GFX10: S_ENDPGM 0, implicit [[V_OR_B32_e64_]]
+    ; GFX10: [[V_AND_OR_B32_:%[0-9]+]]:vgpr_32 = V_AND_OR_B32 [[COPY]], [[COPY1]], [[COPY2]], implicit $exec
+    ; GFX10: S_ENDPGM 0, implicit [[V_AND_OR_B32_]]
     %0:sgpr(s32) = COPY $sgpr0
     %1:sgpr(s32) = COPY $sgpr1
     %2:vgpr(s32) = COPY $vgpr0
     %3:sgpr(s32) = G_AND %0, %1
     %4:vgpr(s32) = COPY %3
     %5:vgpr(s32) = G_OR %4, %2
     S_ENDPGM 0, implicit %5
 ...

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-pattern-xor3.mir

bb.0:
     ; GFX9: [[V_XOR_B32_e64_:%[0-9]+]]:vgpr_32 = V_XOR_B32_e64 [[COPY3]], [[COPY2]], implicit $exec
     ; GFX9: S_ENDPGM 0, implicit [[V_XOR_B32_e64_]]
     ; GFX10-LABEL: name: xor_s32_sgpr_sgpr_vgpr_copy
     ; GFX10: liveins: $sgpr0, $sgpr1, $vgpr0
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; GFX10: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr1
     ; GFX10: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX10: [[S_XOR_B32_:%[0-9]+]]:sreg_32 = S_XOR_B32 [[COPY]], [[COPY1]], implicit-def $scc
-    ; GFX10: [[COPY3:%[0-9]+]]:vgpr_32 = COPY [[S_XOR_B32_]]
-    ; GFX10: [[V_XOR_B32_e64_:%[0-9]+]]:vgpr_32 = V_XOR_B32_e64 [[COPY3]], [[COPY2]], implicit $exec
-    ; GFX10: S_ENDPGM 0, implicit [[V_XOR_B32_e64_]]
+    ; GFX10: [[V_XOR3_B32_:%[0-9]+]]:vgpr_32 = V_XOR3_B32 [[COPY]], [[COPY1]], [[COPY2]], implicit $exec
+    ; GFX10: S_ENDPGM 0, implicit [[V_XOR3_B32_]]
     %0:sgpr(s32) = COPY $sgpr0
     %1:sgpr(s32) = COPY $sgpr1
     %2:vgpr(s32) = COPY $vgpr0
     %3:sgpr(s32) = G_XOR %0, %1
     %4:vgpr(s32) = COPY %3
     %5:vgpr(s32) = G_XOR %4, %2
     S_ENDPGM 0, implicit %5
 ...
bb.0:
     ; GFX9: [[V_XOR_B32_e64_:%[0-9]+]]:vgpr_32 = V_XOR_B32_e64 [[COPY2]], [[COPY3]], implicit $exec
     ; GFX9: S_ENDPGM 0, implicit [[V_XOR_B32_e64_]]
     ; GFX10-LABEL: name: xor_s32_sgpr_sgpr_vgpr_copy_commute
     ; GFX10: liveins: $sgpr0, $sgpr1, $vgpr0
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; GFX10: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr1
     ; GFX10: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX10: [[S_XOR_B32_:%[0-9]+]]:sreg_32 = S_XOR_B32 [[COPY]], [[COPY1]], implicit-def $scc
-    ; GFX10: [[COPY3:%[0-9]+]]:vgpr_32 = COPY [[S_XOR_B32_]]
-    ; GFX10: [[V_XOR_B32_e64_:%[0-9]+]]:vgpr_32 = V_XOR_B32_e64 [[COPY2]], [[COPY3]], implicit $exec
-    ; GFX10: S_ENDPGM 0, implicit [[V_XOR_B32_e64_]]
+    ; GFX10: [[V_XOR3_B32_:%[0-9]+]]:vgpr_32 = V_XOR3_B32 [[COPY]], [[COPY1]], [[COPY2]], implicit $exec
+    ; GFX10: S_ENDPGM 0, implicit [[V_XOR3_B32_]]
     %0:sgpr(s32) = COPY $sgpr0
     %1:sgpr(s32) = COPY $sgpr1
     %2:vgpr(s32) = COPY $vgpr0
     %3:sgpr(s32) = G_XOR %0, %1
     %4:vgpr(s32) = COPY %3
     %5:vgpr(s32) = G_XOR %2, %4
     S_ENDPGM 0, implicit %5
 ...

llvm/test/TableGen/GlobalISelEmitterCustomPredicateIgnoreCrossRegBankCopies.td

+// RUN: llvm-tblgen %s -gen-global-isel -optimize-match-table=false -I %p/../../include -I %p/Common -o - | FileCheck %s
+
+include "llvm/Target/Target.td"
+include "GlobalISelEmitterCommon.td"
+
+// Boilerplate code for setting up some registers with subregs.
+class MyReg<string n, list<Register> subregs = []>
+  : Register<n> {
+  let SubRegs = subregs;
+}
+
+class MyClass<int size, list<ValueType> types, dag registers>
+  : RegisterClass<"Test", types, size, registers> {
+  let Size = size;
+}
+
+def sub0 : SubRegIndex<16>;
+def sub1 : SubRegIndex<16, 16>;
+def S0 : MyReg<"s0">;
+def S1 : MyReg<"s1">;
+def SRegs : MyClass<16, [i16], (sequence "S%u", 0, 1)>;
+
+let SubRegIndices = [sub0, sub1] in {
+def D0 : MyReg<"d0", [S0, S1]>;
+}
+
+def DRegs : MyClass<32, [i32], (sequence "D%u", 0, 0)>;
+def DOP : RegisterOperand<DRegs>;
+def AND_OR : I<(outs DRegs:$dst), (ins DOP:$src0, DOP:$src1, DOP:$src2), []>;
+
+def and_or_pat : PatFrag<
+  (ops node:$x, node:$y, node:$z),
+  (and (g_maybe_cross_reg_bank_copy (or node:$x, node:$y)), node:$z), [{ return foo(); }]> {
+  let GISelPredicateCode = [{
+    return doesComplexCheck(MI);
+  }];
+  let PredicateCodeUsesOperands = 1;
+}
+
+// CHECK: GIM_Try, /*On fail goto*//*Label 0*/ 99, // Rule ID 1 //
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/3,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/0, /*Op*/0, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[0] src2
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/0, /*Op*/1, /*StoreIdx*/2, // Name : pred:1:z
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/0, /*Op*/1, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[0] Operand 2
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordInsn, /*DefineMI*/1, /*MI*/0, /*OpIdx*/2, // MIs[1]
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/1, /*Expected*/3,
+// CHECK-NEXT: GIM_CheckOpcodeIgnoreCopies, /*MI*/1, TargetOpcode::G_OR,
+// CHECK-NEXT: // MIs[1] Operand 0
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/0, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[1] src0
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/1, /*Op*/1, /*StoreIdx*/0, // Name : pred:1:x
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/1, /*Op*/1, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[1] src1
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/1, /*Op*/2, /*StoreIdx*/1, // Name : pred:1:y
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/1, /*Op*/2, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GIPFP_MI_Predicate_and_or_pat,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: // (and:{ *:[i32] } DOP:{ *:[i32] }:$src2:$pred:1:z, (g_maybe_cross_reg_bank_copy:{ *:[i32] } (or:{ *:[i32] } DOP:{ *:[i32] }:$src0:$pred:1:x, DOP:{ *:[i32] }:$src1:$pred:1:y)))<<P:1:Predicate_and_or_pat>>  =>  (AND_OR:{ *:[i32] } DOP:{ *:[i32] }:$src0, DOP:{ *:[i32] }:$src1, DOP:{ *:[i32] }:$src2)
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/MyTarget::AND_OR,
+
+// CHECK: GIM_Try, /*On fail goto*//*Label 1*/ 198, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/3,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/0, /*Op*/0, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[0] Operand 1
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordInsn, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/1, /*Expected*/3,
+// CHECK-NEXT: GIM_CheckOpcodeIgnoreCopies, /*MI*/1, TargetOpcode::G_OR,
+// CHECK-NEXT: // MIs[1] Operand 0
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/0, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[1] src0
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/1, /*Op*/1, /*StoreIdx*/0, // Name : pred:1:x
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/1, /*Op*/1, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[1] src1
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/1, /*Op*/2, /*StoreIdx*/1, // Name : pred:1:y
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/1, /*Op*/2, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: // MIs[0] src2
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordNamedOperand, /*MI*/0, /*Op*/2, /*StoreIdx*/2, // Name : pred:1:z
+// CHECK-NEXT: GIM_CheckRegBankForClass, /*MI*/0, /*Op*/2, /*RC*/Test::DRegsRegClassID,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GIPFP_MI_Predicate_and_or_pat,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: // (and:{ *:[i32] } (g_maybe_cross_reg_bank_copy:{ *:[i32] } (or:{ *:[i32] } DOP:{ *:[i32] }:$src0:$pred:1:x, DOP:{ *:[i32] }:$src1:$pred:1:y)), DOP:{ *:[i32] }:$src2:$pred:1:z)<<P:1:Predicate_and_or_pat>>  =>  (AND_OR:{ *:[i32] } DOP:{ *:[i32] }:$src0, DOP:{ *:[i32] }:$src1, DOP:{ *:[i32] }:$src2)
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/MyTarget::AND_OR,
+
+// Test g_maybe_cross_reg_bank_copy.
+def : Pat<
+  (i32 (and_or_pat DOP:$src0, DOP:$src1, DOP:$src2)),
+  (AND_OR DOP:$src0, DOP:$src1, DOP:$src2)
+>;

llvm/utils/TableGen/CodeGenDAGPatterns.cpp

 }
 
 /// getPatternSize - Return the 'size' of this pattern.  We want to match large
 /// patterns before small ones.  This is used to determine the size of a
 /// pattern.
 static unsigned getPatternSize(const TreePatternNode *P,
                                const CodeGenDAGPatterns &CGP) {
   unsigned Size = 3;  // The node itself.
+  if (!P->isLeaf() &&
+      P->getOperator()->getName().equals("g_maybe_cross_reg_bank_copy")) {
+    return getPatternSize(P->getChild(0), CGP);
+  }
   // If the root node is a ConstantSDNode, increases its size.
   // e.g. (set R32:$dst, 0).
   if (P->isLeaf() && isa<IntInit>(P->getLeafValue()))
     Size += 2;
 
   if (const ComplexPattern *AM = P->getComplexPatternInfo(CGP)) {
     Size += AM->getComplexity();
     // We don't want to count any children twice, so return early.

 // TreePatternNode implementation
 //
 
 static unsigned GetNumNodeResults(Record *Operator, CodeGenDAGPatterns &CDP) {
   if (Operator->getName() == "set" ||
       Operator->getName() == "implicit")
     return 0;  // All return nothing.
 
+  if (Operator->getName() == "g_maybe_cross_reg_bank_copy")
+    return 1;
+
   if (Operator->isSubClassOf("Intrinsic"))
     return CDP.getIntrinsic(Operator).IS.RetVTs.size();
 
   if (Operator->isSubClassOf("SDNode"))
     return CDP.getSDNodeInfo(Operator).getNumResults();
 
   if (Operator->isSubClassOf("PatFrags")) {
     // If we've already parsed this pattern fragment, get it.  Otherwise, handle
if (getOperator()->isSubClassOf("ComplexPattern")) {
     bool MadeChange = false;
 
     for (unsigned i = 0; i < getNumChildren(); ++i)
       MadeChange |= getChild(i)->ApplyTypeConstraints(TP, NotRegisters);
 
     return MadeChange;
   }
 
+  if (getOperator()->getName() == "g_maybe_cross_reg_bank_copy") {
+    bool MadeChange = getChild(0)->ApplyTypeConstraints(TP, NotRegisters);
+    MadeChange |= this->UpdateNodeType(0, getChild(0)->getExtType(0),TP);
+    return MadeChange;
+  }
+
   assert(getOperator()->isSubClassOf("SDNodeXForm") && "Unknown node type!");
 
   // Node transforms always take one operand.
   if (getNumChildren() != 1) {
     TP.error("Node transform '" + getOperator()->getName() +
              "' requires one operand!");
     return false;
   }
bool TreePatternNode::canPatternMatch(std::string &Reason,
   if (getOperator()->isSubClassOf("Intrinsic")) {
     // TODO:
     return true;
   }
 
   if (getOperator()->isSubClassOf("ComplexPattern"))
     return true;
 
+  if (getOperator()->getName() == "g_maybe_cross_reg_bank_copy")
+    return true;
+
   // If this node is a commutative operator, check that the LHS isn't an
   // immediate.
   const SDNodeInfo &NodeInfo = CDP.getSDNodeInfo(getOperator());
   bool isCommIntrinsic = isCommutativeIntrinsic(CDP);
   if (NodeInfo.hasProperty(SDNPCommutative) || isCommIntrinsic) {
     // Scan all of the operands of the node and make sure that only the last one
     // is a constant node, unless the RHS also is.
     if (!OnlyOnRHSOfCommutative(getChild(getNumChildren()-1))) {
TreePatternNodePtr TreePattern::ParseTreePattern(Init *TheInit,
   // Verify that this is something that makes sense for an operator.
   if (!Operator->isSubClassOf("PatFrags") &&
       !Operator->isSubClassOf("SDNode") &&
       !Operator->isSubClassOf("Instruction") &&
       !Operator->isSubClassOf("SDNodeXForm") &&
       !Operator->isSubClassOf("Intrinsic") &&
       !Operator->isSubClassOf("ComplexPattern") &&
       Operator->getName() != "set" &&
-      Operator->getName() != "implicit")
+      Operator->getName() != "implicit" &&
+      Operator->getName() != "g_maybe_cross_reg_bank_copy")
     error("Unrecognized node '" + Operator->getName() + "'!");
 
   //  Check to see if this is something that is illegal in an input pattern.
   if (isInputPattern) {
     if (Operator->isSubClassOf("Instruction") ||
         Operator->isSubClassOf("SDNodeXForm"))
       error("Cannot use '" + Operator->getName() + "' in an input pattern!");
   } else {

 /// GenerateVariantsOf - Given a pattern N, generate all permutations we can of
 /// the (potentially recursive) pattern by using algebraic laws.
 ///
 static void GenerateVariantsOf(TreePatternNodePtr N,
                                std::vector<TreePatternNodePtr> &OutVariants,
                                CodeGenDAGPatterns &CDP,
                                const MultipleUseVarSet &DepVars) {
   // We cannot permute leaves or ComplexPattern uses.
-  if (N->isLeaf() || N->getOperator()->isSubClassOf("ComplexPattern")) {
+  if (N->isLeaf() || N->getOperator()->isSubClassOf("ComplexPattern") ||
+      N->getOperator()->getName() == "g_maybe_cross_reg_bank_copy") {
     OutVariants.push_back(N);
     return;
   }
 
   // Look up interesting info about the node.
   const SDNodeInfo &NodeInfo = CDP.getSDNodeInfo(N->getOperator());
 
   // If this node is associative, re-associate.

llvm/utils/TableGen/DAGISelMatcherGen.cpp

for (const std::string &Name : Names)
     VariableMap[Name] = Entry;
 
   return NewRecord;
 }
 
 void MatcherGen::EmitMatchCode(const TreePatternNode *N,
                                TreePatternNode *NodeNoTypes,
                                unsigned ForceMode) {
+  // This operator is used by global-isel, skip it for sdag.
+  if (!N->isLeaf() &&
+      N->getOperator()->getName().equals("g_maybe_cross_reg_bank_copy"))
+    return EmitMatchCode(N->getChild(0), NodeNoTypes->getChild(0), ForceMode);
+
   // If N and NodeNoTypes don't agree on a type, then this is a case where we
   // need to do a type check.  Emit the check, apply the type to NodeNoTypes and
   // reinfer any correlated types.
   SmallVector<unsigned, 2> ResultsToTypeCheck;
 
   for (unsigned i = 0, e = NodeNoTypes->getNumTypes(); i != e; ++i) {
     if (NodeNoTypes->getExtType(i) == N->getExtType(i)) continue;
     NodeNoTypes->setType(i, N->getExtType(i));

llvm/utils/TableGen/GlobalISelEmitter.cpp

protected:
   // Allow matching one to several, similar opcodes that share properties. This
   // is to handle patterns where one SelectionDAG operation maps to multiple
   // GlobalISel ones (e.g. G_BUILD_VECTOR and G_BUILD_VECTOR_TRUNC). The first
   // is treated as the canonical opcode.
   SmallVector<const CodeGenInstruction *, 2> Insts;
 
   static DenseMap<const CodeGenInstruction *, unsigned> OpcodeValues;
 
+  bool SkipCopies;
 
   MatchTableRecord getInstValue(const CodeGenInstruction *I) const {
     const auto VI = OpcodeValues.find(I);
     if (VI != OpcodeValues.end())
       return MatchTable::NamedValue(I->Namespace, I->TheDef->getName(),
                                     VI->second);
     return MatchTable::NamedValue(I->Namespace, I->TheDef->getName());
   }
 
 public:
   static void initOpcodeValuesMap(const CodeGenTarget &Target) {
     OpcodeValues.clear();
 
     unsigned OpcodeValue = 0;
     for (const CodeGenInstruction *I : Target.getInstructionsByEnumValue())
       OpcodeValues[I] = OpcodeValue++;
   }
 
   InstructionOpcodeMatcher(unsigned InsnVarID,
-                           ArrayRef<const CodeGenInstruction *> I)
+                           ArrayRef<const CodeGenInstruction *> I,
+                           bool SkipCopies = false)
       : InstructionPredicateMatcher(IPM_Opcode, InsnVarID),
-        Insts(I.begin(), I.end()) {
+        Insts(I.begin(), I.end()), SkipCopies(SkipCopies) {
     assert((Insts.size() == 1 || Insts.size() == 2) &&
            "unexpected number of opcode alternatives");
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_Opcode;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
-           Insts == cast<InstructionOpcodeMatcher>(&B)->Insts;
+           Insts == cast<InstructionOpcodeMatcher>(&B)->Insts &&
+           SkipCopies == cast<InstructionOpcodeMatcher>(&B)->SkipCopies;
   }
 
   bool hasValue() const override {
     return Insts.size() == 1 && OpcodeValues.count(Insts[0]);
   }
 
   // TODO: This is used for the SwitchMatcher optimization. We should be able to
   // return a list of the opcodes to match.
   MatchTableRecord getValue() const override {
     assert(Insts.size() == 1);
 
     const CodeGenInstruction *I = Insts[0];
     const auto VI = OpcodeValues.find(I);
     if (VI != OpcodeValues.end())
       return MatchTable::NamedValue(I->Namespace, I->TheDef->getName(),
                                     VI->second);
     return MatchTable::NamedValue(I->Namespace, I->TheDef->getName());
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
-    StringRef CheckType = Insts.size() == 1 ?
-                          "GIM_CheckOpcode" : "GIM_CheckOpcodeIsEither";
+    StringRef CheckType = SkipCopies
+                              ? "GIM_CheckOpcodeIgnoreCopies"
+                              : (Insts.size() == 1 ? "GIM_CheckOpcode"
+                                                   : "GIM_CheckOpcodeIsEither");
     Table << MatchTable::Opcode(CheckType) << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID);
 
     for (const CodeGenInstruction *I : Insts)
       Table << getInstValue(I);
     Table << MatchTable::LineBreak;
   }
 
if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (isa<IntInit>(SrcInit)) {
       InsnMatcher.addPredicate<InstructionOpcodeMatcher>(
           &Target.getInstruction(RK.getDef("G_CONSTANT")));
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");
   } else {
+    bool SkipCopies = false;
+    if (Src->getOperator()->getName() == "g_maybe_cross_reg_bank_copy") {
+      Src = Src->getChild(0);
+      SkipCopies = true;
+    }
     SrcGIEquivOrNull = findNodeEquiv(Src->getOperator());
     if (!SrcGIEquivOrNull)
       return failedImport("Pattern operator lacks an equivalent Instruction" +
                           explainOperator(Src->getOperator()));
     SrcGIOrNull = getEquivNode(*SrcGIEquivOrNull, Src);
 
     // The operators look good: match the opcode
-    InsnMatcher.addPredicate<InstructionOpcodeMatcher>(SrcGIOrNull);
+    InsnMatcher.addPredicate<InstructionOpcodeMatcher>(SrcGIOrNull, SkipCopies);
   }
 
   unsigned OpIdx = 0;
   for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
     // Results don't have a name unless they are the root node. The caller will
     // set the name if appropriate.
     OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
     if (auto Error = OM.addTypeCheckPredicate(VTy, false /* OperandIsAPointer */))