getVPMemoryOpCost interface

[bmahjour]

Summary

Added TTI queries for the cost of a VP Memory operation, and added DataType and Alignment to the hasActiveVectorLength() interface.

[bmahjour]

Taking over from @hussainjk as he is going to grad school.

[RolandF]

Is this still needed?

[RolandF]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

I prefer the downstream name getVariableLengthMemoryOpCost. I'm not sure
what VP stands for here and it seems like VP is already used in the context of VPlan. At least explain in the comment.

[bmahjour]

In D109416#3067495, @RolandF wrote:

Is this still needed?

Yes, this patch establishes the target interface that is implemented by D109417.

[bmahjour]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

VP stands for Vector Predication and is the name that the community has used for all the predicated intrinsics. See https://llvm.org/docs/LangRef.html#vector-predication-intrinsics

[RolandF]

In D109416#3070423, @bmahjour wrote:

In D109416#3067495, @RolandF wrote:

Is this still needed?

Yes, this patch establishes the target interface that is implemented by D109417.

Yes, I realize. Are you intending to go ahead with that patch? It has been sitting for a while. I would imagine that is useful info for other reviewers.

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

Well, I followed your link. There are no "vp" memory ops. The loads and stores are named "masked" operations.

[RolandF]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

The target implementation you propose depends on having an Instruction parameter passed in. If you pass in an Instruction you could actually figure out from the dynamic cast that this is a vp intrinsic. The existing getMemoryOpCost allows passing in an Instruction, so it seems like you could figure this out without a new interface.

[simoll]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

The LLVM LangRef wording for vp.load/store/gather/scatter are missing. The intrinsics themselves are upstream though.

[bmahjour]

In D109416#3071074, @RolandF wrote:

In D109416#3070423, @bmahjour wrote:

In D109416#3067495, @RolandF wrote:

Is this still needed?

Yes, this patch establishes the target interface that is implemented by D109417.

Yes, I realize. Are you intending to go ahead with that patch? It has been sitting for a while. I would imagine that is useful info for other reviewers.

The following patches are related and still in flight and we intend to go ahead with them all. There has been a delay due to Hussain leaving, but we definitely want to resume the work soon and finish it. If you are interested in helping commandeer some of them please let me know.

https://reviews.llvm.org/D109416 - Cost model infrastructure for VPMemory ops
https://reviews.llvm.org/D109417 - Cost model for VPMemory ops
https://reviews.llvm.org/D109584 - Expansion of VP load/store to scalar loads/stores
https://reviews.llvm.org/D109377 - Type legalization for VP memory SelectionDAG nodes
https://reviews.llvm.org/D109379 - PPC lowering for VP memory SD nodes

[bmahjour]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

There is precedence for this, for example we have separate interfaces for getMaskedMemoryOpCost and getGatherScatterOpCost, so we try to follow that in this patch.
I think the instruction being passed to getMemoryOpCost is meant to be optional, and it may not even be a vp intrinsic call. For example we could be querying the cost of turning a scalar load into a vector predicated load intrinsic, in which case the instruction passed to getMemoryOpCost would be a scalar load so we cannot do a dyn_cast on it to figure out if we want a vp intrinsic cost or otherwise.

[bmahjour]

added DataType and Alignment to the hasActiveVectorLength() interface.

[RolandF]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1125

Now that the instruction pointer is truly optional, having a separate call actually provides value and I like this patch a lot better.

[RolandF]

LGTM

[simoll]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1387

This is really just a getter that tells you whether using VP intrinsics makes sense on this target at all. It shouldn't be modified this way.

Couldn't you use the cost function to tell you whether a combination of %evl and type for memory accesses makes sense?

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1341 ↗

(On Diff #381394)

This patch should only be for TTI. You are using the default impl here anyway - target-specific changes should go in a separate patch.

[bmahjour]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1387

Whether the target supports EVL in hardware may depend on opcode and data type. For example, Power9/10 only support load and stores. Other targets may have dependency on data type or alignment. For these reasons I think we should pass opcode and datatype/alignment as input to this function. I just realized I had forgotten to add opcode to this interface. I'll do that.

Having a function that can tell us whether an opcode/type combination can be efficiently vector predicated (without having to model the cost) is helpful in LoopVectorizer as well. For example when deciding how to vectorize we don't have a VPIntrinsic yet, so we cannot use getVPLegalizationStrategy. Having something similar to isLegalMaskedLoad would be useful, and I was hoping we use hasActiveVectorLength() for that purpose (per previous conversation in D99750#inline-949716). Is there another interface I should we looking at?

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

1341 ↗

(On Diff #381394)

This is just a stub, but I'll move it to D109417 for cleaner separation.

[bmahjour]

@simoll I've addressed your comments above, please let me know if you have any further comments. I plan to commit this soon. Thanks!

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
 
   /// \return The cost of Load and Store instructions.
   InstructionCost
   getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                   unsigned AddressSpace,
                   TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
                   const Instruction *I = nullptr) const;
 
+  /// \return The cost of VP Load and Store instructions.
+  InstructionCost
+  getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                    unsigned AddressSpace,
+                    TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
+                    const Instruction *I = nullptr) const;
+
   /// \return The cost of masked Load and Store instructions.
   InstructionCost getMaskedMemoryOpCost(
       unsigned Opcode, Type *Src, Align Alignment, unsigned AddressSpace,
       TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput) const;
 
   /// \return The cost of Gather or Scatter operation
   /// \p Opcode - is a type of memory access Load or Store
   /// \p DataTy - a vector type of the data to be loaded or stored

   unsigned getGISelRematGlobalCost() const;
 
   /// \returns True if the target supports scalable vectors.
   bool supportsScalableVectors() const;
 
   /// \name Vector Predication Information
   /// @{
   /// Whether the target supports the %evl parameter of VP intrinsic efficiently
-  /// in hardware. (see LLVM Language Reference - "Vector Predication
-  /// Intrinsics") Use of %evl is discouraged when that is not the case.
-  bool hasActiveVectorLength() const;
+  /// in hardware, for the given opcode and type/alignment. (see LLVM Language
+  /// Reference - "Vector Predication Intrinsics").
+  /// Use of %evl is discouraged when that is not the case.
+  bool hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                             Align Alignment) const;
 
   struct VPLegalization {
     enum VPTransform {
       // keep the predicating parameter
       Legal = 0,
       // where legal, discard the predicate parameter
       Discard = 1,
       // transform into something else that is also predicating
virtual InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                              const Instruction *I) = 0;
   virtual InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
                                              unsigned Index) = 0;
   virtual InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,
                                           Align Alignment,
                                           unsigned AddressSpace,
                                           TTI::TargetCostKind CostKind,
                                           const Instruction *I) = 0;
+  virtual InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src,
+                                            Align Alignment,
+                                            unsigned AddressSpace,
+                                            TTI::TargetCostKind CostKind,
+                                            const Instruction *I) = 0;
   virtual InstructionCost
   getMaskedMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                         unsigned AddressSpace,
                         TTI::TargetCostKind CostKind) = 0;
   virtual InstructionCost
   getGatherScatterOpCost(unsigned Opcode, Type *DataTy, const Value *Ptr,
                          bool VariableMask, Align Alignment,
                          TTI::TargetCostKind CostKind,
virtual unsigned getStoreVectorFactor(unsigned VF, unsigned StoreSize,
                                         VectorType *VecTy) const = 0;
   virtual bool preferInLoopReduction(unsigned Opcode, Type *Ty,
                                      ReductionFlags) const = 0;
   virtual bool preferPredicatedReductionSelect(unsigned Opcode, Type *Ty,
                                                ReductionFlags) const = 0;
   virtual bool shouldExpandReduction(const IntrinsicInst *II) const = 0;
   virtual unsigned getGISelRematGlobalCost() const = 0;
   virtual bool supportsScalableVectors() const = 0;
-  virtual bool hasActiveVectorLength() const = 0;
+  virtual bool hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                                     Align Alignment) const = 0;
   virtual InstructionCost getInstructionLatency(const Instruction *I) = 0;
   virtual VPLegalization
   getVPLegalizationStrategy(const VPIntrinsic &PI) const = 0;
 };
 
 template <typename T>
 class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
   T Impl;
public:
   }
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                                   unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I) override {
     return Impl.getMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
                                 CostKind, I);
   }
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I) override {
+    return Impl.getVPMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
+                                  CostKind, I);
+  }
   InstructionCost getMaskedMemoryOpCost(unsigned Opcode, Type *Src,
                                         Align Alignment, unsigned AddressSpace,
                                         TTI::TargetCostKind CostKind) override {
     return Impl.getMaskedMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
                                       CostKind);
   }
   InstructionCost
   getGatherScatterOpCost(unsigned Opcode, Type *DataTy, const Value *Ptr,
public:
   unsigned getGISelRematGlobalCost() const override {
     return Impl.getGISelRematGlobalCost();
   }
 
   bool supportsScalableVectors() const override {
     return Impl.supportsScalableVectors();
   }
 
-  bool hasActiveVectorLength() const override {
-    return Impl.hasActiveVectorLength();
+  bool hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                             Align Alignment) const override {
+    return Impl.hasActiveVectorLength(Opcode, DataType, Alignment);
   }
 
   InstructionCost getInstructionLatency(const Instruction *I) override {
     return Impl.getInstructionLatency(I);
   }
 
   VPLegalization
   getVPLegalizationStrategy(const VPIntrinsic &PI) const override {

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                                   unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I) const {
     return 1;
   }
 
+  InstructionCost getVPMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
+                                    unsigned AddressSpace,
+                                    TTI::TargetCostKind CostKind,
+                                    const Instruction *I) const {
+    return 1;
+  }
+
   InstructionCost getMaskedMemoryOpCost(unsigned Opcode, Type *Src,
                                         Align Alignment, unsigned AddressSpace,
                                         TTI::TargetCostKind CostKind) const {
     return 1;
   }
 
   InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
                                          const Value *Ptr, bool VariableMask,
public:
   }
 
   bool shouldExpandReduction(const IntrinsicInst *II) const { return true; }
 
   unsigned getGISelRematGlobalCost() const { return 1; }
 
   bool supportsScalableVectors() const { return false; }
 
-  bool hasActiveVectorLength() const { return false; }
+  bool hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                             Align Alignment) const {
+    return false;
+  }
 
   TargetTransformInfo::VPLegalization
   getVPLegalizationStrategy(const VPIntrinsic &PI) const {
     return TargetTransformInfo::VPLegalization(
         /* EVLParamStrategy */ TargetTransformInfo::VPLegalization::Discard,
         /* OperatorStrategy */ TargetTransformInfo::VPLegalization::Convert);
   }
 

llvm/lib/Analysis/TargetTransformInfo.cpp

 unsigned TargetTransformInfo::getGISelRematGlobalCost() const {
   return TTIImpl->getGISelRematGlobalCost();
 }
 
 bool TargetTransformInfo::supportsScalableVectors() const {
   return TTIImpl->supportsScalableVectors();
 }
 
-bool TargetTransformInfo::hasActiveVectorLength() const {
-  return TTIImpl->hasActiveVectorLength();
+bool TargetTransformInfo::hasActiveVectorLength(unsigned Opcode, Type *DataType,
+                                                Align Alignment) const {
+  return TTIImpl->hasActiveVectorLength(Opcode, DataType, Alignment);
 }
 
 InstructionCost
 TargetTransformInfo::getInstructionLatency(const Instruction *I) const {
   return TTIImpl->getInstructionLatency(I);
 }
 
 InstructionCost