AMDGPU/GlobalISel/Emitter Recognize additional 'same operand checks'

[Petar.Avramovic]

Summary

The "name" of a non-leaf complex pattern (MY_PAT $op1, $op2) is
"MY_PAT:op1:op2" and the ones with same "name" represent same operand.
Add 'same operand check' for this case.

[foad]

How does the new test case relate to the tablegen change? Perhaps you could pre-commit it so we can see the test case diffs?

[Petar.Avramovic]

Show how this emitter change affects isel, and update tablegen tests.
Add additional checks for complex suboperands for test added in D57980. Neither tablegen nor sdag have check for such pattern and it looks like an error to me (there is no way for tablegen to know that some c++ code that generates two operands will generate same operand x (first out operand) from different input operands - these are operand 1 and 2 of G_ADD instruction). Also, if I checked correctly, there were no patterns like that in llvm upstream. Skip such patterns anyway in GlobalIselEmitter.

[arsenm]

Can you add tests for the min3/max3/med3 patterns? I think this is what was blocking those

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-fract.f64.mir

27–28

Is this pattern really correct? It should probably be moved into a combine instead of a selection pattern if so

[Petar.Avramovic]

min3/max3 patterns are handled in combiner, inducing med3 patterns with constants. Patch imports med3 isel patterns: min(max(a, b), max(min(a, b), c)) but they need some combines/legalize fixes to be selected. From new imports only V_FRACT can be selected at the moment.

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-fract.f64.mir

27–28

It is correctly selected since -enable-unsafe-fp-math flag was on.

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-fract.f64.mir

27–28

I think this should be moved into a combine and not depend on the global flag, but that's a separate change

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-fract.f64.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=amdgcn -mcpu=gfx1010 -enable-unsafe-fp-math -run-pass=instruction-select -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name:            fract_f64_neg
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+
+body: |
+  bb.1:
+    liveins: $sgpr0_sgpr1
+
+    ; CHECK-LABEL: name: fract_f64_neg
+    ; CHECK: liveins: $sgpr0_sgpr1
+    ; CHECK: $vcc_hi = IMPLICIT_DEF
+    ; CHECK: [[COPY:%[0-9]+]]:sreg_64 = COPY $sgpr0_sgpr1
+    ; CHECK: [[S_LOAD_DWORDX4_IMM:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[COPY]], 36, 0, 0 :: (dereferenceable invariant load 16, align 4, addrspace 4)
+    ; CHECK: [[COPY1:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX4_IMM]].sub0_sub1
+    ; CHECK: [[COPY2:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX4_IMM]].sub2_sub3
+    ; CHECK: [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY2]], 0, 0, 0 :: (load 8, addrspace 1)
+    ; CHECK: [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 0
+    ; CHECK: [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 -2147483648
+    ; CHECK: [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[S_MOV_B32_]], %subreg.sub0, [[S_MOV_B32_1]], %subreg.sub1
+    ; CHECK: [[COPY3:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]]
+    ; CHECK: [[COPY4:%[0-9]+]]:vreg_64 = COPY [[S_LOAD_DWORDX2_IMM]]
+    ; CHECK: %12:vreg_64 = nofpexcept V_ADD_F64 0, [[COPY3]], 1, [[COPY4]], 0, 0, implicit $mode, implicit $exec
+    ; CHECK: %15:vreg_64 = nofpexcept V_FRACT_F64_e64 0, %12, 0, 0, implicit $mode, implicit $exec
+    ; CHECK: [[COPY5:%[0-9]+]]:vreg_64 = COPY [[COPY1]]
+    ; CHECK: GLOBAL_STORE_DWORDX2 [[COPY5]], %15, 0, 0, 0, 0, implicit $exec :: (store 8, addrspace 1)
+    ; CHECK: S_ENDPGM 0
+    %2:sgpr(p4) = COPY $sgpr0_sgpr1
+    %7:sgpr(s64) = G_CONSTANT i64 36
+    %8:sgpr(p4) = G_PTR_ADD %2, %7(s64)
+    %9:sgpr(<2 x s64>) = G_LOAD %8(p4) :: (dereferenceable invariant load 16, align 4, addrspace 4)
+    %10:sgpr(s64) = G_EXTRACT %9(<2 x s64>), 0
+    %13:sgpr(s64) = G_EXTRACT %9(<2 x s64>), 64
+    %15:sgpr(p1) = G_INTTOPTR %13(s64)
+    %18:sgpr(s64) = G_LOAD %15(p1) :: (load 8, addrspace 1)
+    %19:sgpr(s64) = G_FCONSTANT double -0.000000e+00
+    %24:sgpr(s64) = G_FNEG %18
+    %25:vgpr(s64) = COPY %19(s64)
+    %26:vgpr(s64) = COPY %24(s64)
+    %20:vgpr(s64) = G_FADD %25, %26
+    %21:vgpr(s64) = G_FFLOOR %20
+    %23:vgpr(s64) = G_FNEG %21
+    %22:vgpr(s64) = G_FADD %20, %23
+    %12:sgpr(p1) = G_INTTOPTR %10(s64)
+    %27:vgpr(p1) = COPY %12(p1)
+    G_STORE %22(s64), %27(p1) :: (store 8, addrspace 1)
+    S_ENDPGM 0
+...
+
+---
+name:            fract_f64_neg_abs
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+
+body: |
+  bb.1:
+    liveins: $sgpr0_sgpr1
+
+    ; CHECK-LABEL: name: fract_f64_neg_abs
+    ; CHECK: liveins: $sgpr0_sgpr1
+    ; CHECK: $vcc_hi = IMPLICIT_DEF
+    ; CHECK: [[COPY:%[0-9]+]]:sreg_64 = COPY $sgpr0_sgpr1
+    ; CHECK: [[S_LOAD_DWORDX4_IMM:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[COPY]], 36, 0, 0 :: (dereferenceable invariant load 16, align 4, addrspace 4)
+    ; CHECK: [[COPY1:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX4_IMM]].sub0_sub1
+    ; CHECK: [[COPY2:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX4_IMM]].sub2_sub3
+    ; CHECK: [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY2]], 0, 0, 0 :: (load 8, addrspace 1)
+    ; CHECK: [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 0
+    ; CHECK: [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 -2147483648
+    ; CHECK: [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[S_MOV_B32_]], %subreg.sub0, [[S_MOV_B32_1]], %subreg.sub1
+    ; CHECK: [[COPY3:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]]
+    ; CHECK: [[COPY4:%[0-9]+]]:vreg_64 = COPY [[S_LOAD_DWORDX2_IMM]]
+    ; CHECK: %13:vreg_64 = nofpexcept V_ADD_F64 0, [[COPY3]], 3, [[COPY4]], 0, 0, implicit $mode, implicit $exec
+    ; CHECK: %16:vreg_64 = nofpexcept V_FRACT_F64_e64 0, %13, 0, 0, implicit $mode, implicit $exec
+    ; CHECK: [[COPY5:%[0-9]+]]:vreg_64 = COPY [[COPY1]]
+    ; CHECK: GLOBAL_STORE_DWORDX2 [[COPY5]], %16, 0, 0, 0, 0, implicit $exec :: (store 8, addrspace 1)
+    ; CHECK: S_ENDPGM 0
+    %2:sgpr(p4) = COPY $sgpr0_sgpr1
+    %7:sgpr(s64) = G_CONSTANT i64 36
+    %8:sgpr(p4) = G_PTR_ADD %2, %7(s64)
+    %9:sgpr(<2 x s64>) = G_LOAD %8(p4) :: (dereferenceable invariant load 16, align 4, addrspace 4)
+    %10:sgpr(s64) = G_EXTRACT %9(<2 x s64>), 0
+    %13:sgpr(s64) = G_EXTRACT %9(<2 x s64>), 64
+    %15:sgpr(p1) = G_INTTOPTR %13(s64)
+    %18:sgpr(s64) = G_LOAD %15(p1) :: (load 8, addrspace 1)
+    %19:sgpr(s64) = G_FABS %18
+    %20:sgpr(s64) = G_FCONSTANT double -0.000000e+00
+    %25:sgpr(s64) = G_FNEG %19
+    %26:vgpr(s64) = COPY %20(s64)
+    %27:vgpr(s64) = COPY %25(s64)
+    %21:vgpr(s64) = G_FADD %26, %27
+    %22:vgpr(s64) = G_FFLOOR %21
+    %24:vgpr(s64) = G_FNEG %22
+    %23:vgpr(s64) = G_FADD %21, %24
+    %12:sgpr(p1) = G_INTTOPTR %10(s64)
+    %28:vgpr(p1) = COPY %12(p1)
+    G_STORE %23(s64), %28(p1) :: (store 8, addrspace 1)
+    S_ENDPGM 0
+...

llvm/utils/TableGen/GlobalISelEmitter.cpp

public:
   }
 
   void defineOperand(StringRef SymbolicName, OperandMatcher &OM);
 
   void definePhysRegOperand(Record *Reg, OperandMatcher &OM);
 
   Error defineComplexSubOperand(StringRef SymbolicName, Record *ComplexPattern,
                                 unsigned RendererID, unsigned SubOperandID) {
-    if (ComplexSubOperands.count(SymbolicName))
-      return failedImport(
-          "Complex suboperand referenced more than once (Operand: " +
-          SymbolicName + ")");
-
+    if (!ComplexSubOperands.count(SymbolicName))
       ComplexSubOperands[SymbolicName] =
           std::make_tuple(ComplexPattern, RendererID, SubOperandID);
-
+    // Complex suboperand referenced more than once is used to generate 'same
+    // operand check' emitting of GIR_ComplexSubOperandRenderer for them is
+    // already handled.
     return Error::success();
   }
 
   Optional<DefinedComplexPatternSubOperand>
   getComplexSubOperand(StringRef SymbolicName) const {
     const auto &I = ComplexSubOperands.find(SymbolicName);
     if (I == ComplexSubOperands.end())
       return None;

 }
 
 Error GlobalISelEmitter::importChildMatcher(
     RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
     const TreePatternNode *SrcChild, bool OperandIsAPointer,
     bool OperandIsImmArg, unsigned OpIdx, unsigned &TempOpIdx) {
 
   Record *PhysReg = nullptr;
-  StringRef SrcChildName = getSrcChildName(SrcChild, PhysReg);
+  std::string SrcChildName = std::string(getSrcChildName(SrcChild, PhysReg));
+  if (!SrcChild->isLeaf() &&
+      SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
+    // The "name" of a non-leaf complex pattern (MY_PAT $op1, $op2) is
+    // "MY_PAT:op1:op2" and the ones with same "name" represent same operand.
+    std::string PatternName = std::string(SrcChild->getOperator()->getName());
+    for (unsigned i = 0; i < SrcChild->getNumChildren(); ++i) {
+      PatternName += ":";
+      PatternName += SrcChild->getChild(i)->getName();
+    }
+    SrcChildName = PatternName;
+  }
 
   OperandMatcher &OM =
-      PhysReg
-          ? InsnMatcher.addPhysRegInput(PhysReg, OpIdx, TempOpIdx)
-          : InsnMatcher.addOperand(OpIdx, std::string(SrcChildName), TempOpIdx);
+      PhysReg ? InsnMatcher.addPhysRegInput(PhysReg, OpIdx, TempOpIdx)
+              : InsnMatcher.addOperand(OpIdx, SrcChildName, TempOpIdx);
   if (OM.isSameAsAnotherOperand())
     return Error::success();
 
   ArrayRef<TypeSetByHwMode> ChildTypes = SrcChild->getExtTypes();
   if (ChildTypes.size() != 1)
     return failedImport("Src pattern child has multiple results");
 
   // Check MBB's before the type check since they are not a known type.