[SDag][AMDGPU] Maintain DAG divergence through instruction selection

[foad]

Summary

Maintain and verify the IsDivergent property of SDNodes until after
instruction selection. This requires target support for identifying
MachineSDNodes that are known to be divergent or uniform.

[foad]

Currently there are no users of the post-instruction-selection divergence information, but does this seem potentially useful for anything?

I used it for analyzing post-isel DAG dumps, looking for places where we might have selected VALU instructions for uniform nodes for no good reason.

[arsenm]

In D88485#2300583, @foad wrote:

Currently there are no users of the post-instruction-selection divergence information, but does this seem potentially useful for anything?

I used it for analyzing post-isel DAG dumps, looking for places where we might have selected VALU instructions for uniform nodes for no good reason.

I think putting any effort into DAG infrastructure is a waste of time at this point

[madhur13490]

I think this is useful to have until we completely switch to GlobalISel. I agree with any new effort being low !/$ but since you have made it, I'd vote to get this in.

BTW, I think we can have lit tests to verify this. Dump DAG before and after isel and check with file-check. I am not sure if just dumping with existing dump() routine would dump all the required information but its worthwhile to have a test, IMHO.

[rampitec]

That is beneficial to know if an instruction is divergent or uniform even after selection because it cannot be determined purely on a VALU vs SALU basis or register classes. However, I do not like that list of opcodes in the isSDNodeSourceOfDivergence(), it is error prone.

[foad]

In D88485#2301154, @rampitec wrote:

However, I do not like that list of opcodes in the isSDNodeSourceOfDivergence(), it is error prone.

These lists are the SDNode equivalents of GCNTTIImpl::isSourceOfDivergence (which uses some data defined in .td files) and GCNTTIImpl::isAlwaysUniform. I don't know if there's a better way to generate them.

[rampitec]

In D88485#2301282, @foad wrote:

In D88485#2301154, @rampitec wrote:

However, I do not like that list of opcodes in the isSDNodeSourceOfDivergence(), it is error prone.

These lists are the SDNode equivalents of GCNTTIImpl::isSourceOfDivergence (which uses some data defined in .td files) and GCNTTIImpl::isAlwaysUniform. I don't know if there's a better way to generate them.

It can use a flag in Desc I think.

[alex-t]

It is not clear to me why do we need to query divergence information for MachineSDNode?
After unstruction selection is done we should have all the instructions selected correctly to VALU vs SALU basing on the information that is available at the selection stage.
Thus, we can use isDivergent bit value set for the MachineSDNode in case we need to recompute or update divergence information after selection.
So, instead of adding machine opcodes to isSDNodeSourceOfDivergence it is better to mark that opcodes right away as they are selected.

After the selection is done we'd rather interested in VALU or SALU property instead of divergence. We have lots of opcodes that are VALU even if uniform.
For that we may like to have theseparate analysis pass that could work similar to the divergence analysis but propagate the VALU/SALU property instead of divergence.
BUt creating such a "one more divergence analysis" upon the MIR requires a lot of efforts and strong reasons.

[foad]

It is not clear to me why do we need to query divergence information for MachineSDNode?

We don't need it. It is just something I found useful when I was doing some debugging and analysis of the code we generate.

[arsenm]

I guess this is technically correct but I wouldn't spend time getting this accurate

[foad]

Abandoning. It was just an experiment. There is no real use case for it.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

for (op_iterator I = op_begin(), E = op_end(); I != E; ) {
     Use.set(SDValue());
   }
 }
 
 void SelectionDAG::setNodeMemRefs(MachineSDNode *N,
                                   ArrayRef<MachineMemOperand *> NewMemRefs) {
   if (NewMemRefs.empty()) {
     N->clearMemRefs();
-    return;
-  }
-
-  // Check if we can avoid allocating by storing a single reference directly.
-  if (NewMemRefs.size() == 1) {
+  } else if (NewMemRefs.size() == 1) {
+    // Avoid allocating by storing a single reference directly.
     N->MemRefs = NewMemRefs[0];
     N->NumMemRefs = 1;
-    return;
-  }
-
+  } else {
     MachineMemOperand **MemRefsBuffer =
         Allocator.template Allocate<MachineMemOperand *>(NewMemRefs.size());
     llvm::copy(NewMemRefs, MemRefsBuffer);
     N->MemRefs = MemRefsBuffer;
     N->NumMemRefs = static_cast<int>(NewMemRefs.size());
   }
+  updateDivergence(N);
+}
 
 /// SelectNodeTo - These are wrappers around MorphNodeTo that accept a
 /// machine opcode.
 ///
 SDNode *SelectionDAG::SelectNodeTo(SDNode *N, unsigned MachineOpc,
                                    EVT VT) {
   SDVTList VTs = getVTList(VT);
   return SelectNodeTo(N, MachineOpc, VTs, None);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

// code to the MachineBasicBlock.
     DoInstructionSelection();
   }
 
   LLVM_DEBUG(dbgs() << "Selected selection DAG: "
                     << printMBBReference(*FuncInfo->MBB) << " '" << BlockName
                     << "'\n";
              CurDAG->dump());
 
+#ifndef NDEBUG
+  if (TTI.hasBranchDivergence())
+    CurDAG->VerifyDAGDiverence();
+#endif
+
   if (ViewSchedDAGs && MatchFilterBB)
     CurDAG->viewGraph("scheduler input for " + BlockName);
 
   // Schedule machine code.
   ScheduleDAGSDNodes *Scheduler = CreateScheduler();
   {
     NamedRegionTimer T("sched", "Instruction Scheduling", GroupName,
                        GroupDescription, TimePassesIsEnabled);

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

case ISD::INTRINSIC_WO_CHAIN: {
     return false;
   }
   case ISD::LOAD:
     if (cast<LoadSDNode>(N)->getMemOperand()->getAddrSpace() ==
         AMDGPUAS::CONSTANT_ADDRESS_32BIT)
       return true;
     return false;
   }
+  if (auto *MN = dyn_cast<MachineSDNode>(N)) {
+    switch (MN->getMachineOpcode()) {
+    case AMDGPU::V_READFIRSTLANE_B32:
+    case AMDGPU::V_READLANE_B32:
+      // TODO: add more opcodes
+      // TODO: generate from .td files?
+      return true;
+    }
+    for (auto *MMO : MN->memoperands()) {
+      if (MMO->getAddrSpace() == AMDGPUAS::CONSTANT_ADDRESS_32BIT)
+        return true;
+    }
+  }
   return false;
 }
 
 SDValue AMDGPUTargetLowering::getNegatedExpression(
     SDValue Op, SelectionDAG &DAG, bool LegalOperations, bool ForCodeSize,
     NegatibleCost &Cost, unsigned Depth) const {
 
   switch (Op.getOpcode()) {

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

case ISD::CALLSEQ_END:
     return true;
   case ISD::INTRINSIC_WO_CHAIN:
     return AMDGPU::isIntrinsicSourceOfDivergence(
         cast<ConstantSDNode>(N->getOperand(0))->getZExtValue());
   case ISD::INTRINSIC_W_CHAIN:
     return AMDGPU::isIntrinsicSourceOfDivergence(
         cast<ConstantSDNode>(N->getOperand(1))->getZExtValue());
   }
+  if (auto *MN = dyn_cast<MachineSDNode>(N)) {
+    switch (MN->getMachineOpcode()) {
+    case AMDGPU::DS_SWIZZLE_B32:
+    case AMDGPU::GLOBAL_ATOMIC_CSUB_RTN:
+    case AMDGPU::SI_PS_LIVE:
+    case AMDGPU::V_INTERP_MOV_F32:
+    case AMDGPU::V_INTERP_P1LL_F16:
+    case AMDGPU::V_INTERP_P1_F32:
+    case AMDGPU::V_INTERP_P2_F16:
+    case AMDGPU::V_INTERP_P2_F32:
+    case AMDGPU::V_MBCNT_HI_U32_B32_e64:
+    case AMDGPU::V_MBCNT_LO_U32_B32_e64:
+    case AMDGPU::V_MFMA_F32_16X16X16F16:
+    case AMDGPU::V_MFMA_F32_16X16X1F32:
+    case AMDGPU::V_MFMA_F32_16X16X2BF16:
+    case AMDGPU::V_MFMA_F32_16X16X4F16:
+    case AMDGPU::V_MFMA_F32_16X16X4F32:
+    case AMDGPU::V_MFMA_F32_16X16X8BF16:
+    case AMDGPU::V_MFMA_F32_32X32X1F32:
+    case AMDGPU::V_MFMA_F32_32X32X2BF16:
+    case AMDGPU::V_MFMA_F32_32X32X2F32:
+    case AMDGPU::V_MFMA_F32_32X32X4BF16:
+    case AMDGPU::V_MFMA_F32_32X32X4F16:
+    case AMDGPU::V_MFMA_F32_32X32X8F16:
+    case AMDGPU::V_MFMA_F32_4X4X1F32:
+    case AMDGPU::V_MFMA_F32_4X4X2BF16:
+    case AMDGPU::V_MFMA_F32_4X4X4F16:
+    case AMDGPU::V_MFMA_I32_16X16X16I8:
+    case AMDGPU::V_MFMA_I32_16X16X4I8:
+    case AMDGPU::V_MFMA_I32_32X32X4I8:
+    case AMDGPU::V_MFMA_I32_32X32X8I8:
+    case AMDGPU::V_MFMA_I32_4X4X4I8:
+    case AMDGPU::V_MOV_B32_dpp8_gfx10:
+    case AMDGPU::V_MOV_B32_dpp:
+    case AMDGPU::V_MOV_B64_DPP_PSEUDO:
+    case AMDGPU::V_PERMLANE16_B32:
+    case AMDGPU::V_PERMLANEX16_B32:
+    case AMDGPU::V_WRITELANE_B32:
+      // TODO: add more opcodes
+      // TODO: generate from .td files?
+      return true;
+    }
+    for (auto *MMO : MN->memoperands()) {
+      unsigned AS = MMO->getAddrSpace();
+      // A flat load may access private memory.
+      if (AS == AMDGPUAS::PRIVATE_ADDRESS || AS == AMDGPUAS::FLAT_ADDRESS)
+        return true;
+    }
+  }
   return false;
 }
 
 bool SITargetLowering::denormalsEnabledForType(const SelectionDAG &DAG,
                                                EVT VT) const {
   switch (VT.getScalarType().getSimpleVT().SimpleTy) {
   case MVT::f32:
     return hasFP32Denormals(DAG.getMachineFunction());