Cost model for VPMemory operations on PowerPC.

[bmahjour]

Summary

Implemented getVPMemoryOpCost for PowerPC. Depends on D109416

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

144

make this a static const member variable and include "P9" in the name.

[bmahjour]

turn the macro into a named constant and fix formatting issues.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1362

These variables are only used in the if block and should be declared there.

1363

This variable is only used once - the expression can be used directly instead.

1367

The Instruction pointer I is optional but is dereferenced unconditionally. The is likely to segfault. There is a dyn_cast but the result is not checked for nullptr so there may also be a segfault if I is something else. If a valid vp intrinsic pointer is expected for this target implementation then there should be an assert to check the dyn_cast result.

1367

Having the entire middle of the function occupied by a path that one probably never wants to execute is slightly painful. I would suggest handling the shouldDoNothing path first instead.

1402

If this is made an or condition with not pwr9 then the pwr10 case can return here so there aren't two return LT.first lines.

1405

Missing vectorCostAdjustment to account for P9.

[bmahjour]

Address all but one of Roland's comments.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1367

Actually the getVPLegalizationStrategy doesn't seem like the right interface to use (because we may not have the VP intrinsics available at the time we do cost-analysis). What we really need is something that can tell us if the opcode/type combination is considered legal on a target (ie PPC). I think I asked this question before, and someone told me to use hasActiveVectorLength(). However, I think we need to change that interface to take in more pieces of info. I'm going to look into that next.

[bmahjour]

Implement and use hasActiveVectorLength() for PPC target.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

I don't follow this reasoning. Maybe 64-bit data is aligned half the time, but how is that true for char data for instance?

1431

The scalar loads have scalar type results. I assume if the code is being vectorized and the function of a vector load is being replaced, the results have to go in a vector. I think there is some insert cost here.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

Does this sound better?

return P9PipelineFlushEstimate / ((Alignment/8) + 1);

It would also be an equivalent of saying:

return ((Alignment == 8) ? P9PipelineFlushEstimate / 2 : P9PipelineFlushEstimate);

unless we cast the values to float in that formula. Any other suggestion?

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1431

The call to getScalarizationOverhead is supposed to account for insert vs extract cost (notice we pass IsLoad vs !isLoad to indicate whether we need insert or extract cost respectively).

There is an existing function, getCommonMaskedMemoryOpCost() which computes cost in a very similar manor and is called by getMaskedMemoryOpCost(). I'm thinking of replacing lines 1412-1437 with a call to that function instead. Any thoughts @RolandF ?

[bmahjour]

Made the following changes:

• Made hasActiveVectorLength() return false for anything other than load/stores.
• Only halve the cost of pipeline flush for 8-byte aligned accesses. For smaller alignments we consider the full cost.
• Replaced the calculation of the scalarization cost with getMaskedMemoryOpCost().

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1395

I think that it makes sense to use float. Something like

Misaligned = (NumElements - 1) / NumElements
Result = Misaligned * P9PipelineFlushEstimate + (1 - Misaligned) * Cost

The present calculation I think implies that the aligned cost is zero.

[bmahjour]

Addressed comments.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1400

I like this approach better. But I think the AlignmentProb calculation is incorrect. This would calculate 0% aligned for align 1 and 7/8 aligned for align 8 instead of 50%. If you want to key off alignment rather than number of elements then I think you want to do size / alignment to figure out the number of possible offsets and use that to calculate probability.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1400

The accesses are consecutive and the minimum alignment of the address is specified on the load/store instruction, which is passed to this function, so I don't think the element size or the size of the vector matters.

I've changed the alignment probability calculation to simply divide the specified alignment by the desired alignment. This way we'll get 50% probability for 8-byte aligned accesses, 25% probability for 4-byte aligned accesses and so forth.

[RolandF]

LGTM

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
 
   /// \return The cost of Load and Store instructions.
   InstructionCost
   getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                   unsigned AddressSpace,
                   TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
                   const Instruction *I = nullptr) const;
 
+  /// \return The cost of VP Load and Store instructions.
+  InstructionCost
+  getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                    unsigned AddressSpace,
+                    TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
+                    const Instruction *I = nullptr) const;
+
   /// \return The cost of masked Load and Store instructions.
   InstructionCost getMaskedMemoryOpCost(
       unsigned Opcode, Type *Src, Align Alignment, unsigned AddressSpace,
       TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput) const;
 
   /// \return The cost of Gather or Scatter operation
   /// \p Opcode - is a type of memory access Load or Store
   /// \p DataTy - a vector type of the data to be loaded or stored
virtual InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                              const Instruction *I) = 0;
   virtual InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
                                              unsigned Index) = 0;
   virtual InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,
                                           Align Alignment,
                                           unsigned AddressSpace,
                                           TTI::TargetCostKind CostKind,
                                           const Instruction *I) = 0;
+  virtual InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src,
+                                            Align Alignment,
+                                            unsigned AddressSpace,
+                                            TTI::TargetCostKind CostKind,
+                                            const Instruction *I) = 0;
   virtual InstructionCost
   getMaskedMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                         unsigned AddressSpace,
                         TTI::TargetCostKind CostKind) = 0;
   virtual InstructionCost
   getGatherScatterOpCost(unsigned Opcode, Type *DataTy, const Value *Ptr,
                          bool VariableMask, Align Alignment,
                          TTI::TargetCostKind CostKind,
public:
   }
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                                   unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I) override {
     return Impl.getMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
                                 CostKind, I);
   }
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I) override {
+    return Impl.getVPMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
+                                  CostKind, I);
+  }
   InstructionCost getMaskedMemoryOpCost(unsigned Opcode, Type *Src,
                                         Align Alignment, unsigned AddressSpace,
                                         TTI::TargetCostKind CostKind) override {
     return Impl.getMaskedMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
                                       CostKind);
   }
   InstructionCost
   getGatherScatterOpCost(unsigned Opcode, Type *DataTy, const Value *Ptr,

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                                   unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I) const {
     return 1;
   }
 
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I) const {
+    return 1;
+  }
+
   InstructionCost getMaskedMemoryOpCost(unsigned Opcode, Type *Src,
                                         Align Alignment, unsigned AddressSpace,
                                         TTI::TargetCostKind CostKind) const {
     return 1;
   }
 
   InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
                                          const Value *Ptr, bool VariableMask,

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

InstructionCost getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
       Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
       bool UseMaskForCond = false, bool UseMaskForGaps = false);
   InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
                                         TTI::TargetCostKind CostKind);
   bool areFunctionArgsABICompatible(const Function *Caller,
                                     const Function *Callee,
                                     SmallPtrSetImpl<Argument *> &Args) const;
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I = nullptr);
+
+private:
+  // The following constant is used for estimating costs on power9.
+  static const InstructionCost::CostType P9PipelineFlushEstimate = 80;
+
   /// @}
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

case Intrinsic::ppc_vsx_stxvp: {
     return true;
   }
   default:
     break;
   }
 
   return false;
 }
+
+InstructionCost PPCTTIImpl::getVPMemoryOpCost(unsigned Opcode, Type *Src,
+                                              Align Alignment,
+                                              unsigned AddressSpace,
+                                              TTI::TargetCostKind CostKind,
+                                              const Instruction *I) {
+  InstructionCost Cost = BaseT::getVPMemoryOpCost(Opcode, Src, Alignment,
+                                                  AddressSpace, CostKind, I);
+  if (TLI->getValueType(DL, Src, true) == MVT::Other)
+    return Cost;
+  // TODO: Handle other cost kinds.
+  if (CostKind != TTI::TCK_RecipThroughput)
+    return Cost;
+
+  assert((Opcode == Instruction::Load || Opcode == Instruction::Store) &&
+         "Invalid Opcode");
+  bool IsLoad = (Opcode == Instruction::Load);
+
+  auto *SrcVTy = dyn_cast<FixedVectorType>(Src);
+  assert(SrcVTy && "Expected a vector type for VP memory operations");
+
+  if (getVPLegalizationStrategy(*dyn_cast<VPIntrinsic>(I)).shouldDoNothing()) {
+    std::pair<InstructionCost, MVT> LT =
+        TLI->getTypeLegalizationCost(DL, SrcVTy);
+    InstructionCost Cost = vectorCostAdjustment(LT.first, Opcode, Src, nullptr);
+
+    // On P9 but not on P10, if the op is misaligned then it will cause a
+    // pipeline flush. Otherwise the VSX masked memops cost the same as unmasked
+    // ones.
+    if (Alignment >= 16 || ST->getCPUDirective() != PPC::DIR_PWR9)
+      return Cost;
+
+    // We assume the average case: that ops with alignment <= 128
+    // will flush a full pipeline about half the time.
+    // The cost when this happens is about 80 cycles.
+    return P9PipelineFlushEstimate / 2;
+  }
+
+  // Usually we should not get to this point, but the following is an attempt to
+  // model the cost of legalization. Currently we can only lower intrinsics with
+  // evl but no mask, on Power 9/10. Otherwise, we must scalarize. We need to
+  // extract (from the mask) the most/least significant byte of all halfwords
+  // aligned with vector elements, and do an access predicated on its 0th bit.
+  // We make the simplifying assumption that byte-extraction costs are
+  // stride-invariant, so we model the extraction as scalarizing a load of
+  // <NumElems x i8>.
+
+  // VSX masks have lanes per bit, but the predication is per halfword.
+  unsigned NumElems = SrcVTy->getNumElements();
+  auto *MaskI8Ty = Type::getInt8Ty(SrcVTy->getContext());
+  InstructionCost MaskSplitCost = getScalarizationOverhead(
+      FixedVectorType::get(MaskI8Ty, NumElems), false, true);
+  const InstructionCost ScalarCompareCostInstrCost =
+      getCmpSelInstrCost(Instruction::ICmp, MaskI8Ty, nullptr,
+                         CmpInst::BAD_ICMP_PREDICATE, CostKind);
+
+  assert(ScalarCompareCostInstrCost.isValid() &&
+         "Expected valid instruction cost");
+  int ScalarCompareCost = *(ScalarCompareCostInstrCost.getValue());
+
+  const InstructionCost BranchInstrCost =
+      getCFInstrCost(Instruction::Br, CostKind);
+  assert(BranchInstrCost.isValid() && "Expected valid instruction cost");
+  int BranchCost = *BranchInstrCost.getValue();
+  int MaskCmpCost = NumElems * (BranchCost + ScalarCompareCost);
+
+  InstructionCost ValueSplitCost =
+      getScalarizationOverhead(SrcVTy, IsLoad, !IsLoad);
+  const InstructionCost ScalarMemOpInstrCost =
+      NumElems * BaseT::getMemoryOpCost(Opcode, SrcVTy->getScalarType(),
+                                        Alignment, AddressSpace, CostKind);
+  assert(ScalarMemOpInstrCost.isValid() && "Expected valid instruction cost");
+  int ScalarMemOpCost = *(ScalarMemOpInstrCost.getValue());
+  return ScalarMemOpCost + ValueSplitCost + MaskSplitCost + MaskCmpCost;
+}