Cost model for VPMemory operations on PowerPC.

[bmahjour]

Summary

Implemented getVPMemoryOpCost for PowerPC. Depends on D109416

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

144

make this a static const member variable and include "P9" in the name.

[bmahjour]

turn the macro into a named constant and fix formatting issues.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1413

These variables are only used in the if block and should be declared there.

1414

This variable is only used once - the expression can be used directly instead.

1418

The Instruction pointer I is optional but is dereferenced unconditionally. The is likely to segfault. There is a dyn_cast but the result is not checked for nullptr so there may also be a segfault if I is something else. If a valid vp intrinsic pointer is expected for this target implementation then there should be an assert to check the dyn_cast result.

1418

Having the entire middle of the function occupied by a path that one probably never wants to execute is slightly painful. I would suggest handling the shouldDoNothing path first instead.

1453

If this is made an or condition with not pwr9 then the pwr10 case can return here so there aren't two return LT.first lines.

1456

Missing vectorCostAdjustment to account for P9.

[bmahjour]

Address all but one of Roland's comments.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1418

Actually the getVPLegalizationStrategy doesn't seem like the right interface to use (because we may not have the VP intrinsics available at the time we do cost-analysis). What we really need is something that can tell us if the opcode/type combination is considered legal on a target (ie PPC). I think I asked this question before, and someone told me to use hasActiveVectorLength(). However, I think we need to change that interface to take in more pieces of info. I'm going to look into that next.

[bmahjour]

Implement and use hasActiveVectorLength() for PPC target.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1446

I don't follow this reasoning. Maybe 64-bit data is aligned half the time, but how is that true for char data for instance?

1482

The scalar loads have scalar type results. I assume if the code is being vectorized and the function of a vector load is being replaced, the results have to go in a vector. I think there is some insert cost here.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1446

Does this sound better?

return P9PipelineFlushEstimate / ((Alignment/8) + 1);

It would also be an equivalent of saying:

return ((Alignment == 8) ? P9PipelineFlushEstimate / 2 : P9PipelineFlushEstimate);

unless we cast the values to float in that formula. Any other suggestion?

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1482

The call to getScalarizationOverhead is supposed to account for insert vs extract cost (notice we pass IsLoad vs !isLoad to indicate whether we need insert or extract cost respectively).

There is an existing function, getCommonMaskedMemoryOpCost() which computes cost in a very similar manor and is called by getMaskedMemoryOpCost(). I'm thinking of replacing lines 1412-1437 with a call to that function instead. Any thoughts @RolandF ?

[bmahjour]

Made the following changes:

• Made hasActiveVectorLength() return false for anything other than load/stores.
• Only halve the cost of pipeline flush for 8-byte aligned accesses. For smaller alignments we consider the full cost.
• Replaced the calculation of the scalarization cost with getMaskedMemoryOpCost().

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1446

I think that it makes sense to use float. Something like

Misaligned = (NumElements - 1) / NumElements
Result = Misaligned * P9PipelineFlushEstimate + (1 - Misaligned) * Cost

The present calculation I think implies that the aligned cost is zero.

[bmahjour]

Addressed comments.

[RolandF]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1451

I like this approach better. But I think the AlignmentProb calculation is incorrect. This would calculate 0% aligned for align 1 and 7/8 aligned for align 8 instead of 50%. If you want to key off alignment rather than number of elements then I think you want to do size / alignment to figure out the number of possible offsets and use that to calculate probability.

[bmahjour]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1451

The accesses are consecutive and the minimum alignment of the address is specified on the load/store instruction, which is passed to this function, so I don't think the element size or the size of the vector matters.

I've changed the alignment probability calculation to simply divide the specified alignment by the desired alignment. This way we'll get 50% probability for 8-byte aligned accesses, 25% probability for 4-byte aligned accesses and so forth.

[RolandF]

LGTM

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

InstructionCost getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
       Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
       bool UseMaskForCond = false, bool UseMaskForGaps = false);
   InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
                                         TTI::TargetCostKind CostKind);
   bool areFunctionArgsABICompatible(const Function *Caller,
                                     const Function *Callee,
                                     SmallPtrSetImpl<Argument *> &Args) const;
+  bool hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                             Align Alignment) const;
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I = nullptr);
+
+private:
+  // The following constant is used for estimating costs on power9.
+  static const InstructionCost::CostType P9PipelineFlushEstimate = 80;
+
   /// @}
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

case Intrinsic::ppc_vsx_stxvp: {
     return true;
   }
   default:
     break;
   }
 
   return false;
 }
+
+bool PPCTTIImpl::hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                                       Align Alignment) const {
+  // Only load and stores instructions can have variable vector length on Power.
+  if (Opcode != Instruction::Load && Opcode != Instruction::Store)
+    return false;
+  // Loads/stores with length instructions use bits 0-7 of the GPR operand and
+  // therefore cannot be used in 32-bit mode.
+  if ((!ST->hasP9Vector() && !ST->hasP10Vector()) || !ST->isPPC64())
+    return false;
+  if (auto *VecTy = dyn_cast<FixedVectorType>(DataType)) {
+    unsigned VecWidth = DataType->getPrimitiveSizeInBits();
+    return VecWidth == 128;
+  }
+  Type *ScalarTy = DataType->getScalarType();
+
+  if (ScalarTy->isPointerTy())
+    return true;
+
+  if (ScalarTy->isFloatTy() || ScalarTy->isDoubleTy())
+    return true;
+
+  if (!ScalarTy->isIntegerTy())
+    return false;
+
+  unsigned IntWidth = ScalarTy->getIntegerBitWidth();
+  return IntWidth == 8 || IntWidth == 16 || IntWidth == 32 || IntWidth == 64;
+}
+
+InstructionCost PPCTTIImpl::getVPMemoryOpCost(unsigned Opcode, Type *Src,
+                                              Align Alignment,
+                                              unsigned AddressSpace,
+                                              TTI::TargetCostKind CostKind,
+                                              const Instruction *I) {
+  InstructionCost Cost = BaseT::getVPMemoryOpCost(Opcode, Src, Alignment,
+                                                  AddressSpace, CostKind, I);
+  if (TLI->getValueType(DL, Src, true) == MVT::Other)
+    return Cost;
+  // TODO: Handle other cost kinds.
+  if (CostKind != TTI::TCK_RecipThroughput)
+    return Cost;
+
+  assert((Opcode == Instruction::Load || Opcode == Instruction::Store) &&
+         "Invalid Opcode");
+
+  auto *SrcVTy = dyn_cast<FixedVectorType>(Src);
+  assert(SrcVTy && "Expected a vector type for VP memory operations");
+
+  if (hasActiveVectorLength(Opcode, Src, Alignment)) {
+    std::pair<InstructionCost, MVT> LT =
+        TLI->getTypeLegalizationCost(DL, SrcVTy);
+
+    InstructionCost CostFactor =
+        vectorCostAdjustmentFactor(Opcode, Src, nullptr);
+    if (!CostFactor.isValid())
+      return InstructionCost::getMax();
+
+    InstructionCost Cost = LT.first * CostFactor;
+    assert(Cost.isValid() && "Expected valid cost");
+
+    // On P9 but not on P10, if the op is misaligned then it will cause a
+    // pipeline flush. Otherwise the VSX masked memops cost the same as unmasked
+    // ones.
+    const Align DesiredAlignment(16);
+    if (Alignment >= DesiredAlignment || ST->getCPUDirective() != PPC::DIR_PWR9)
+      return Cost;
+
+    // Since alignment may be under estimated, we try to compute the probability
+    // that the actual address is aligned to the desired boundary. For example
+    // an 8-byte aligned load is assumed to be actually 16-byte aligned half the
+    // time, while a 4-byte aligned load has a 25% chance of being 16-byte
+    // aligned.
+    float AlignmentProb = ((float)Alignment.value()) / DesiredAlignment.value();
+    float MisalignmentProb = 1.0 - AlignmentProb;
+    return (MisalignmentProb * P9PipelineFlushEstimate) +
+           (AlignmentProb * *Cost.getValue());
+  }
+
+  // Usually we should not get to this point, but the following is an attempt to
+  // model the cost of legalization. Currently we can only lower intrinsics with
+  // evl but no mask, on Power 9/10. Otherwise, we must scalarize.
+  return getMaskedMemoryOpCost(Opcode, Src, Alignment, AddressSpace, CostKind);
+}