Cost model for VPMemory operations on PowerPC.

[bmahjour]

Summary

Implemented getVPMemoryOpCost for PowerPC. Depends on D109416

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

146

make this a static const member variable and include "P9" in the name.

[bmahjour]

turn the macro into a named constant and fix formatting issues.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1362

These variables are only used in the if block and should be declared there.

1363

This variable is only used once - the expression can be used directly instead.

1367

The Instruction pointer I is optional but is dereferenced unconditionally. The is likely to segfault. There is a dyn_cast but the result is not checked for nullptr so there may also be a segfault if I is something else. If a valid vp intrinsic pointer is expected for this target implementation then there should be an assert to check the dyn_cast result.

1367

Having the entire middle of the function occupied by a path that one probably never wants to execute is slightly painful. I would suggest handling the shouldDoNothing path first instead.

1402

If this is made an or condition with not pwr9 then the pwr10 case can return here so there aren't two return LT.first lines.

1405

Missing vectorCostAdjustment to account for P9.

[bmahjour]

Address all but one of Roland's comments.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1367

Actually the getVPLegalizationStrategy doesn't seem like the right interface to use (because we may not have the VP intrinsics available at the time we do cost-analysis). What we really need is something that can tell us if the opcode/type combination is considered legal on a target (ie PPC). I think I asked this question before, and someone told me to use hasActiveVectorLength(). However, I think we need to change that interface to take in more pieces of info. I'm going to look into that next.

[bmahjour]

Implement and use hasActiveVectorLength() for PPC target.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

I don't follow this reasoning. Maybe 64-bit data is aligned half the time, but how is that true for char data for instance?

1431

The scalar loads have scalar type results. I assume if the code is being vectorized and the function of a vector load is being replaced, the results have to go in a vector. I think there is some insert cost here.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

Does this sound better?

return P9PipelineFlushEstimate / ((Alignment/8) + 1);

It would also be an equivalent of saying:

return ((Alignment == 8) ? P9PipelineFlushEstimate / 2 : P9PipelineFlushEstimate);

unless we cast the values to float in that formula. Any other suggestion?

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1431

The call to getScalarizationOverhead is supposed to account for insert vs extract cost (notice we pass IsLoad vs !isLoad to indicate whether we need insert or extract cost respectively).

There is an existing function, getCommonMaskedMemoryOpCost() which computes cost in a very similar manor and is called by getMaskedMemoryOpCost(). I'm thinking of replacing lines 1412-1437 with a call to that function instead. Any thoughts @RolandF ?

[bmahjour]

Made the following changes:

• Made hasActiveVectorLength() return false for anything other than load/stores.
• Only halve the cost of pipeline flush for 8-byte aligned accesses. For smaller alignments we consider the full cost.
• Replaced the calculation of the scalarization cost with getMaskedMemoryOpCost().

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

I think that it makes sense to use float. Something like

Misaligned = (NumElements - 1) / NumElements
Result = Misaligned * P9PipelineFlushEstimate + (1 - Misaligned) * Cost

The present calculation I think implies that the aligned cost is zero.

[bmahjour]

Addressed comments.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1400

I like this approach better. But I think the AlignmentProb calculation is incorrect. This would calculate 0% aligned for align 1 and 7/8 aligned for align 8 instead of 50%. If you want to key off alignment rather than number of elements then I think you want to do size / alignment to figure out the number of possible offsets and use that to calculate probability.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1400

The accesses are consecutive and the minimum alignment of the address is specified on the load/store instruction, which is passed to this function, so I don't think the element size or the size of the vector matters.

I've changed the alignment probability calculation to simply divide the specified alignment by the desired alignment. This way we'll get 50% probability for 8-byte aligned accesses, 25% probability for 4-byte aligned accesses and so forth.

[RolandF]

LGTM

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

InstructionCost getInterleavedMemoryOpCost(
       Align Alignment, unsigned AddressSpace,
       TTI::TargetCostKind CostKind = TTI::TCK_SizeAndLatency,
       bool UseMaskForCond = false, bool UseMaskForGaps = false);
   InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
                                         TTI::TargetCostKind CostKind);
   bool areFunctionArgsABICompatible(const Function *Caller,
                                     const Function *Callee,
                                     SmallPtrSetImpl<Argument *> &Args) const;
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I = nullptr);
+// The following constant is used for estimating costs on power9.
+#define PPC_PIPELINE_FLUSH_ESTIMATE 80
   /// @}
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

case Intrinsic::ppc_vsx_stxvp: {
     return true;
   }
   default:
     break;
   }
 
   return false;
 }
+
+InstructionCost PPCTTIImpl::getVPMemoryOpCost(unsigned Opcode, Type *Src,
+                                              Align Alignment,
+                                              unsigned AddressSpace,
+                                              TTI::TargetCostKind CostKind,
+                                              const Instruction *I) {
+  InstructionCost Cost = BaseT::getVPMemoryOpCost(Opcode, Src, Alignment,
+                                                  AddressSpace, CostKind, I);
+  if (TLI->getValueType(DL, Src, true) == MVT::Other)
+    return Cost;
+  // TODO: Handle other cost kinds.
+  if (CostKind != TTI::TCK_RecipThroughput)
+    return Cost;
+
+  assert((Opcode == Instruction::Load || Opcode == Instruction::Store) &&
+         "Invalid Opcode");
+  bool IsLoad = (Opcode == Instruction::Load);
+
+  auto *SrcVTy = dyn_cast<FixedVectorType>(Src);
+  assert(SrcVTy && "Expected a vector type for VP memory operations");
+
+  // VSX masks have lanes per bit, but the predication is per halfword.
+  unsigned NumElems = SrcVTy->getNumElements();
+  auto *SrcScalarTy = SrcVTy->getScalarType();
+  auto *MaskI8Ty = Type::getInt8Ty(SrcVTy->getContext());
+  auto *MaskTy = FixedVectorType::get(MaskI8Ty, NumElems);
+
+  if (!getVPLegalizationStrategy(*dyn_cast<VPIntrinsic>(I)).shouldDoNothing()) {
+    // Currently we can only lower intrinsics with evl but no mask, on Power
+    // 9/10. Otherwise, we must scalarize. We need to extract the most/least
+    // significant byte of all halfwords aligned with vector elements, and do an
+    // access predicated on its 0th bit. We make the simplifying assumption that
+    // byte-extraction costs are stride-invariant, so we model the extraction as
+    // scalarizing a load of <NumElems x i8>.
+    InstructionCost MaskSplitCost =
+        getScalarizationOverhead(MaskTy, false, true);
+    const InstructionCost ScalarCompareCostInstrCost =
+        getCmpSelInstrCost(Instruction::ICmp, MaskI8Ty, nullptr,
+                           CmpInst::BAD_ICMP_PREDICATE, CostKind);
+
+    assert(ScalarCompareCostInstrCost.isValid() &&
+           "Expected valid instruction cost");
+    int ScalarCompareCost = *(ScalarCompareCostInstrCost.getValue());
+
+    const InstructionCost BranchInstrCost =
+        getCFInstrCost(Instruction::Br, CostKind);
+    assert(BranchInstrCost.isValid() && "Expected valid instruction cost");
+    int BranchCost = *BranchInstrCost.getValue();
+    int MaskCmpCost = NumElems * (BranchCost + ScalarCompareCost);
+
+    InstructionCost ValueSplitCost =
+        getScalarizationOverhead(SrcVTy, IsLoad, !IsLoad);
+    const InstructionCost ScalarMemOpInstrCost =
+        NumElems * BaseT::getMemoryOpCost(Opcode, SrcScalarTy, Alignment,
+                                          AddressSpace, CostKind);
+    assert(ScalarMemOpInstrCost.isValid() && "Expected valid instruction cost");
+    int ScalarMemOpCost = *(ScalarMemOpInstrCost.getValue());
+    return ScalarMemOpCost + ValueSplitCost + MaskSplitCost + MaskCmpCost;
+  }
+
+  std::pair<InstructionCost, MVT> LT = TLI->getTypeLegalizationCost(DL, SrcVTy);
+
+  if (Alignment >= 16)
+    // If the op is guaranteed to be aligned to 128 bytes,
+    // then VSX masked memops cost the same as unmasked memops.
+    return LT.first;
+  else
+      // On P9 but not on P10, if the op is misaligned
+      // then it will cause a pipeline flush.
+      // We assume the average case: that ops with alignment <= 128
+      // will flush a full pipeline about half the time.
+      // The cost when this happens is about 80 cycles.
+      if (ST->getCPUDirective() == PPC::DIR_PWR9)
+    return PPC_PIPELINE_FLUSH_ESTIMATE / 2;
+  else
+    return LT.first;
+}