[TTI] Use users of GEP to guess access type in getGEPCost

[luke]

Summary

Currently getGEPCost uses the target type of the GEP as a heuristic for
the type that will be accessed, to pass onto isLegalAddressingMode.
Targets use this to work out if a GEP can then be folded into the
load/store instruction that uses the GEP.
For example, on RISC-V loads and stores can have an offset added to a
base register folded into a single instruction, so the following GEP is
free:

%p = getelementptr i32, ptr %base, i32 42 ; getInstructionCost = 0
%x = load i32, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
lw t0, a0(42)

However vector loads and stores cannot have an offset folded into them,
so the following GEP is costed:

%p = getelementptr <2 x i32>, ptr %base, i32 42 ; getInstructionCost = 1
%x = load <2 x i32>, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
addi a0, 42
vle32 v8, (a0)

The issue arises whenever there is a mismatch between the target type of
the GEP and the type that is actually accessed:

%p = getelementptr i32, ptr %base, i32 42 ; getInstructionCost = 0
%x = load <2 x i32>, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
addi a0, 42
vle32 v8, (a0)

Even though this GEP will result in an add instruction, because TTI
thinks it's loading an i32, it will think it can be folded and not
charge for it.

The target type can become mismatched with the memory access during
transformations, noticeably during SLP where a scalar base pointer will
be reused to perform a vector load or store.

This patch adds an optional AccessType argument to getGEPCost which
allows the type of memory accessed by users to be passed in as a hint,
so that we can more accurately determine if the GEP can be folded into
its users.

If AccessType is not provided, getGEPCost falls back to the old
behaviour of using the PointeeType to guess the memory access type. This
can be revisited in a later patch.

Also for now, only GEPs with exactly one user use the access type hint.
Whilst we could look through all users and use all access types to
determine if we can fold the GEP, this patch avoids doing so to prevent
O(N) behaviour.

[luke]

llvm/include/llvm/Analysis/TargetTransformInfo.h

301

Not overly thrilled with introducing a struct just for this, nor with the constructors. Suggestions are very welcome here

llvm/lib/Analysis/TargetTransformInfo.cpp

229–233

I remember seeing this logic being duplicated in a few places across llvm, earlycse is the first one that comes to mind. It might be worthwhile addressing this in a follow up patch

llvm/test/Transforms/SLPVectorizer/RISCV/gep.ll

18 ↗

(On Diff #519600)

This is the main result from all the noise in the test cases.

[nikic]

You should be able to avoid the LICM changes by adjusting the code at https://github.com/llvm/llvm-project/blob/bfb7c99f3aeab09236adf1f684f7144f384c6dd7/llvm/lib/Transforms/Scalar/LICM.cpp#L1346-L1361 to only pass the users in the loop (and drop the separate user iteration that code does).

Something I'm concerned about here is that your default implementation is going to scan all users of the instruction, which means that the cost query is now O(n) instead of O(1). This is probably not acceptable for compile-time reasons. Can we get away with only inspecting a single user in that case and assume it is representative?

This change seems to cause some pretty large code size changes on x86 at least (https://llvm-compile-time-tracker.com/compare.php?from=b77265711b390a6905e8901694d75f4b3812c71a&to=113f1989fa7e469407942249b4086e3a28da2bf4&stat=size-text), so it seems like a pretty high impact change. I can't say whether those changes are good or bad though :)

[luke]

Only account for users in loop in LICM, fix inliner test

[luke]

In D149889#4319936, @nikic wrote:

You should be able to avoid the LICM changes by adjusting the code at https://github.com/llvm/llvm-project/blob/bfb7c99f3aeab09236adf1f684f7144f384c6dd7/llvm/lib/Transforms/Scalar/LICM.cpp#L1346-L1361 to only pass the users in the loop (and drop the separate user iteration that code does).

Thanks, just tried it and it seems to work.

Something I'm concerned about here is that your default implementation is going to scan all users of the instruction, which means that the cost query is now O(n) instead of O(1). This is probably not acceptable for compile-time reasons. Can we get away with only inspecting a single user in that case and assume it is representative?

Yeah I think that should still work for our SLP use case, sounds like a good compromise. Will try that and report back.

This change seems to cause some pretty large code size changes on x86 at least (https://llvm-compile-time-tracker.com/compare.php?from=b77265711b390a6905e8901694d75f4b3812c71a&to=113f1989fa7e469407942249b4086e3a28da2bf4&stat=size-text), so it seems like a pretty high impact change. I can't say whether those changes are good or bad though :)

Indeed, from a quick look it looks like it might be affecting the inliner?

[luke]

Only use the first user to avoid O(N) compile times.
Move the memory access type logic into Instruction.h in a dependent patch.

[luke]

Remove redundant empty check

[luke]

llvm/lib/Transforms/Scalar/LICM.cpp

1346–1354 ↗

(On Diff #522223)

@nikic I kept the O(N) behaviour here because it looks like we're already iterating through all the users below. But this area seems sensitive. Do we want O(1) behaviour in this block too?

[asb]

llvm/include/llvm/Analysis/TargetTransformInfo.h

286–296

Perhaps it's worth expanding the doc comment to explain how AccessTypes may be used? e.g. "Callers may pass in the access type of users of the GEP in order to allow a more accurate cost, for instance on targets where the legal addressing modes are different for different types."

Of course personal tastes on the amount of doc comments to provide can vary....

[luke]

llvm/include/llvm/Analysis/TargetTransformInfo.h

286–296

It definitely needs some documentation, thanks for pointing that out. The usage of AccessTypes can be quite subtle when it comes to users that aren't loads/stores

[luke]

Add documentation comments

[ABataev]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1018

Shall we use getArithmeticInstrCost(ADD) instead of TCC_Basic?

1080

Same question, cost of int add instead of basic?

[luke]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1080

I agree we should use getArithmeticInstrCost here, but I have some ideas about adding a new target hook to control this and was wondering if I could defer it to that patch?
On RISC-V if Scale != 1 then we will need an extra mul instruction, as opposed to aarch64 which can perform both an add and multiply in one. I think this would make sense to expose the address calculation cost as a hook since it's target dependent, and we already have all the information available here to pass onto TTI.

[ABataev]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1080

Add FIXME/TODOs here then

[luke]

Add TODO about modelling with arithmetic instr

[ABataev]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1018

return BaseGV ? TTI::TCC_Basic : TTI::TCC_Free;

[nikic]

llvm/include/llvm/Analysis/TargetTransformInfo.h

293

to determine

296

stray "which"?

309

ArrayRef should suffice?

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1018

Why is the BaseGV case not free as well?

llvm/lib/Transforms/Scalar/LICM.cpp

1346–1354 ↗

(On Diff #522223)

It's okay, but you should drop the extra loop below. It's purpose is to essentially do the check you're doing here, just in a very crude way.

[luke]

Rebase and address review comments

[luke]

@nikic @ABataev thanks for the review, and sorry for the long delay here. I've rebased this and have run some benchmarks now that SLP is enabled by default for RISC-V.
I'm still seeing a reduction in the number of small unprofitable VFs on RISC-V with this patch, and it also seems to make a few more VF=8s profitable.
As a rough overview, here's the number vsetivlis on the benchmark before:

vsetivli : 467 total
1: 26
2: 247
3: 2
4: 169
5: 1
6: 0
7: 1
8: 17

And after:

vsetivli : 454 total
1: 24
2: 235
3: 2
4: 168
5: 1
6: 0
7: 1
8: 19

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

1018

This is just to match the behaviour of the previous if (Operands.empty()) check. I'll refactor this so that they share this logic

[reames]

A macro comment on the approach here. I think you're trying to do too much in a single change, and that makes it very hard to review.

I would advise doing the following:

• Remove the array of users API. Instead, pass a single optional access type.
• In TTIImpl, get the access type of the *sole user*. If there isn't a sole user, *do not pass one*.
• As a first step, use the access type in the legal addressing mode if available, and *otherwise the pointee type*. This is slightly wrong, but is wrong in the "it's very close to what we used to do" kinda of way which will reduce test deltas significantly. You can change this behavior in a follow up change if profitable.
• Add tests *via cost model* which demonstrate the change above. You should be able to (on RISCV) demonstrate the difference between access type of vector vs scalar.
• All of the other changes in this review - i.e. everything in lib/Transform - should be independent changes structured after the above.

[luke]

• Make AccessType just a single pointer
• Isolate changes to just cost model for now: will follow up with SLP and other users later
• Preserve original behaviour if accesstype isn't passed

[luke]

In D149889#4438606, @reames wrote:

A macro comment on the approach here. I think you're trying to do too much in a single change, and that makes it very hard to review.

I would advise doing the following:

• Remove the array of users API. Instead, pass a single optional access type.
• In TTIImpl, get the access type of the *sole user*. If there isn't a sole user, *do not pass one*.
• As a first step, use the access type in the legal addressing mode if available, and *otherwise the pointee type*. This is slightly wrong, but is wrong in the "it's very close to what we used to do" kinda of way which will reduce test deltas significantly. You can change this behavior in a follow up change if profitable.
• Add tests *via cost model* which demonstrate the change above. You should be able to (on RISCV) demonstrate the difference between access type of vector vs scalar.
• All of the other changes in this review - i.e. everything in lib/Transform - should be independent changes structured after the above.

Agreed, thanks for the pointers. I've split it up and the diff is much smaller now. Will follow up with other patches to do the lib/Transform changes.

[reames]

LGTM w/required changes.

In addition to the inline code comment, make sure you update your commit message. The current review description is out of sync with the code and you must fix that before landing.

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

4972

Pass nullptr here so that you don't change the x86 behavior.

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
   /// skipped by renaming the registers in the CPU, but they still are encoded
   /// and thus wouldn't be considered 'free' here.
   enum TargetCostConstants {
     TCC_Free = 0,     ///< Expected to fold away in lowering.
     TCC_Basic = 1,    ///< The cost of a typical 'add' instruction.
     TCC_Expensive = 4 ///< The cost of a 'div' instruction on x86.
   };
 
   /// Estimate the cost of a GEP operation when lowered.
+  ///
+  /// \p PointeeType is the source element type of the GEP.
+  /// \p Ptr is the base pointer operand.
+  /// \p Operands is the list of indices following the base pointer.
+  ///
+  /// \p AccessType is a hint as to what type of memory might be accessed by
+  /// users of the GEP. getGEPCost will use it to determine if the GEP can be
+  /// folded into the addressing mode of a load/store. If AccessType is null,
+  /// then the resulting target type based off of PointeeType will be used as an
+  /// approximation.
   InstructionCost
   getGEPCost(Type *PointeeType, const Value *Ptr,
-             ArrayRef<const Value *> Operands,
+             ArrayRef<const Value *> Operands, Type *AccessType = nullptr,
              TargetCostKind CostKind = TCK_SizeAndLatency) const;
 
   /// Describe known properties for a set of pointers.
   struct PointersChainInfo {
     /// All the GEPs in a set have same base address.
     unsigned IsSameBaseAddress : 1;
     /// These properties only valid if SameBaseAddress is set.
     /// True if all pointers are separated by a unit stride.
     unsigned IsUnitStride : 1;
     /// True if distance between any two neigbouring pointers is a known value.
     unsigned IsKnownStride : 1;
     unsigned Reserved : 29;
 
     bool isSameBase() const { return IsSameBaseAddress; }
     bool isUnitStride() const { return IsSameBaseAddress && IsUnitStride; }
     bool isKnownStride() const { return IsSameBaseAddress && IsKnownStride; }
 
     static PointersChainInfo getUnitStride() {

 };
 
 class TargetTransformInfo::Concept {
 public:
   virtual ~Concept() = 0;
   virtual const DataLayout &getDataLayout() const = 0;
   virtual InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
                                      ArrayRef<const Value *> Operands,
+                                     Type *AccessType,
                                      TTI::TargetCostKind CostKind) = 0;
   virtual InstructionCost
   getPointersChainCost(ArrayRef<const Value *> Ptrs, const Value *Base,
                        const TTI::PointersChainInfo &Info, Type *AccessTy,
                        TTI::TargetCostKind CostKind) = 0;
   virtual unsigned getInliningThresholdMultiplier() const = 0;
   virtual unsigned adjustInliningThreshold(const CallBase *CB) = 0;
   virtual int getInlinerVectorBonusPercent() const = 0;
public:
   ~Model() override = default;
 
   const DataLayout &getDataLayout() const override {
     return Impl.getDataLayout();
   }
 
   InstructionCost
   getGEPCost(Type *PointeeType, const Value *Ptr,
-             ArrayRef<const Value *> Operands,
+             ArrayRef<const Value *> Operands, Type *AccessType,
              TargetTransformInfo::TargetCostKind CostKind) override {
-    return Impl.getGEPCost(PointeeType, Ptr, Operands, CostKind);
+    return Impl.getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
   }
   InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
                                        const Value *Base,
                                        const PointersChainInfo &Info,
                                        Type *AccessTy,
                                        TargetCostKind CostKind) override {
     return Impl.getPointersChainCost(Ptrs, Base, Info, AccessTy, CostKind);
   }

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

 public:
   // Provide value semantics. MSVC requires that we spell all of these out.
   TargetTransformInfoImplBase(const TargetTransformInfoImplBase &Arg) = default;
   TargetTransformInfoImplBase(TargetTransformInfoImplBase &&Arg) : DL(Arg.DL) {}
 
   const DataLayout &getDataLayout() const { return DL; }
 
   InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
-                             ArrayRef<const Value *> Operands,
+                             ArrayRef<const Value *> Operands, Type *AccessType,
                              TTI::TargetCostKind CostKind) const {
     // In the basic model, we just assume that all-constant GEPs will be folded
     // into their uses via addressing modes.
     for (const Value *Operand : Operands)
       if (!isa<Constant>(Operand))
         return TTI::TCC_Basic;
 
     return TTI::TCC_Free;

 
 protected:
   explicit TargetTransformInfoImplCRTPBase(const DataLayout &DL) : BaseT(DL) {}
 
 public:
   using BaseT::getGEPCost;
 
   InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
-                             ArrayRef<const Value *> Operands,
+                             ArrayRef<const Value *> Operands, Type *AccessType,
                              TTI::TargetCostKind CostKind) {
     assert(PointeeType && Ptr && "can't get GEPCost of nullptr");
     assert(cast<PointerType>(Ptr->getType()->getScalarType())
                ->isOpaqueOrPointeeTypeMatches(PointeeType) &&
            "explicit pointee type doesn't match operand's pointee type");
     auto *BaseGV = dyn_cast<GlobalValue>(Ptr->stripPointerCasts());
     bool HasBaseReg = (BaseGV == nullptr);
 
InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
 
     for (auto I = Operands.begin(); I != Operands.end(); ++I, ++GTI) {
       TargetType = GTI.getIndexedType();
       // We assume that the cost of Scalar GEP with constant index and the
       // cost of Vector GEP with splat constant index are the same.
       const ConstantInt *ConstIdx = dyn_cast<ConstantInt>(*I);
       if (!ConstIdx)
         if (auto Splat = getSplatValue(*I))
           ConstIdx = dyn_cast<ConstantInt>(Splat);
       if (StructType *STy = GTI.getStructTypeOrNull()) {
         // For structures the index is always splat or scalar constant
         assert(ConstIdx && "Unexpected GEP index");
         uint64_t Field = ConstIdx->getZExtValue();
         BaseOffset += DL.getStructLayout(STy)->getElementOffset(Field);
       } else {
         // If this operand is a scalable type, bail out early.
         // TODO: handle scalable vectors
for (auto I = Operands.begin(); I != Operands.end(); ++I, ++GTI) {
           if (Scale != 0)
             // No addressing mode takes two scale registers.
             return TTI::TCC_Basic;
           Scale = ElementSize;
         }
       }
     }
 
+    // If we haven't been provided a hint, use the target type for now.
+    //
+    // TODO: Take a look at potentially removing this: This is *slightly* wrong
+    // as it's possible to have a GEP with a foldable target type but a memory
+    // access that isn't foldable. For example, this load isn't foldable on
+    // RISC-V:
+    //
+    // %p = getelementptr i32, ptr %base, i32 42
+    // %x = load <2 x i32>, ptr %p
+    if (!AccessType)
+      AccessType = TargetType;
+
+    // If the final address of the GEP is a legal addressing mode for the given
+    // access type, then we can fold it into its users.
     if (static_cast<T *>(this)->isLegalAddressingMode(
-            TargetType, const_cast<GlobalValue *>(BaseGV),
+            AccessType, const_cast<GlobalValue *>(BaseGV),
             BaseOffset.sextOrTrunc(64).getSExtValue(), HasBaseReg, Scale,
             Ptr->getType()->getPointerAddressSpace()))
       return TTI::TCC_Free;
+
+    // TODO: Instead of returning TCC_Basic here, we should use
+    // getArithmeticInstrCost. Or better yet, provide a hook to let the target
+    // model it.
     return TTI::TCC_Basic;
   }
 
   InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
                                        const Value *Base,
                                        const TTI::PointersChainInfo &Info,
                                        Type *AccessTy,
                                        TTI::TargetCostKind CostKind) {
     InstructionCost Cost = TTI::TCC_Free;
     // In the basic model we take into account GEP instructions only
     // (although here can come alloca instruction, a value, constants and/or
     // constant expressions, PHIs, bitcasts ... whatever allowed to be used as a
     // pointer). Typically, if Base is a not a GEP-instruction and all the
     // pointers are relative to the same base address, all the rest are
     // either GEP instructions, PHIs, bitcasts or constants. When we have same
     // base, we just calculate cost of each non-Base GEP as an ADD operation if
     // any their index is a non-const.
     // If no known dependecies between the pointers cost is calculated as a sum
     // of costs of GEP instructions.
     for (const Value *V : Ptrs) {
       const auto *GEP = dyn_cast<GetElementPtrInst>(V);
       if (!GEP)
         continue;
       if (Info.isSameBase() && V != Base) {
         if (GEP->hasAllConstantIndices())
           continue;
         Cost += static_cast<T *>(this)->getArithmeticInstrCost(
             Instruction::Add, GEP->getType(), CostKind,
             {TTI::OK_AnyValue, TTI::OP_None}, {TTI::OK_AnyValue, TTI::OP_None},
             std::nullopt);
       } else {
         SmallVector<const Value *> Indices(GEP->indices());
         Cost += static_cast<T *>(this)->getGEPCost(GEP->getSourceElementType(),
                                                    GEP->getPointerOperand(),
-                                                   Indices, CostKind);
+                                                   Indices, nullptr, CostKind);
       }
     }
     return Cost;
   }
 
   InstructionCost getInstructionCost(const User *U,
                                      ArrayRef<const Value *> Operands,
                                      TTI::TargetCostKind CostKind) {
InstructionCost getInstructionCost(const User *U,
     case Instruction::Freeze:
       return TTI::TCC_Free;
     case Instruction::Alloca:
       if (cast<AllocaInst>(U)->isStaticAlloca())
         return TTI::TCC_Free;
       break;
     case Instruction::GetElementPtr: {
       const auto *GEP = cast<GEPOperator>(U);
+      Type *AccessType = nullptr;
+      // For now, only provide the AccessType in the simple case where the GEP
+      // only has one user.
+      if (GEP->hasOneUser() && I)
+        AccessType = I->user_back()->getAccessType();
+
       return TargetTTI->getGEPCost(GEP->getSourceElementType(),
                                    Operands.front(), Operands.drop_front(),
-                                   CostKind);
+                                   AccessType, CostKind);
     }
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
     case Instruction::UDiv:

llvm/include/llvm/CodeGen/BasicTTIImpl.h

public:
   }
 
   unsigned getRegUsageForType(Type *Ty) {
     EVT ETy = getTLI()->getValueType(DL, Ty);
     return getTLI()->getNumRegisters(Ty->getContext(), ETy);
   }
 
   InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
-                             ArrayRef<const Value *> Operands,
+                             ArrayRef<const Value *> Operands, Type *AccessType,
                              TTI::TargetCostKind CostKind) {
-    return BaseT::getGEPCost(PointeeType, Ptr, Operands, CostKind);
+    return BaseT::getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
   }
 
   unsigned getEstimatedNumberOfCaseClusters(const SwitchInst &SI,
                                             unsigned &JumpTableSize,
                                             ProfileSummaryInfo *PSI,
                                             BlockFrequencyInfo *BFI) {
     /// Try to find the estimated number of clusters. Note that the number of
     /// clusters identified in this function could be different from the actual

llvm/lib/Analysis/TargetTransformInfo.cpp

unsigned TargetTransformInfo::getCallerAllocaCost(const CallBase *CB,
                                                   const AllocaInst *AI) const {
   return TTIImpl->getCallerAllocaCost(CB, AI);
 }
 
 int TargetTransformInfo::getInlinerVectorBonusPercent() const {
   return TTIImpl->getInlinerVectorBonusPercent();
 }
 
-InstructionCost
-TargetTransformInfo::getGEPCost(Type *PointeeType, const Value *Ptr,
-                                ArrayRef<const Value *> Operands,
-                                TTI::TargetCostKind CostKind) const {
-  return TTIImpl->getGEPCost(PointeeType, Ptr, Operands, CostKind);
+InstructionCost TargetTransformInfo::getGEPCost(
+    Type *PointeeType, const Value *Ptr, ArrayRef<const Value *> Operands,
+    Type *AccessType, TTI::TargetCostKind CostKind) const {
+  return TTIImpl->getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
 }
 
 InstructionCost TargetTransformInfo::getPointersChainCost(
     ArrayRef<const Value *> Ptrs, const Value *Base,
     const TTI::PointersChainInfo &Info, Type *AccessTy,
     TTI::TargetCostKind CostKind) const {
   assert((Base || !Info.isSameBase()) &&
          "If pointers have same base address it has to be provided.");
   return TTIImpl->getPointersChainCost(Ptrs, Base, Info, AccessTy, CostKind);

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

if (Info.isSameBase() && V != Base) {
         continue;
       Cost += getArithmeticInstrCost(Instruction::Add, GEP->getType(), CostKind,
                                      {TTI::OK_AnyValue, TTI::OP_None},
                                      {TTI::OK_AnyValue, TTI::OP_None},
                                      std::nullopt);
     } else {
       SmallVector<const Value *> Indices(GEP->indices());
       Cost += getGEPCost(GEP->getSourceElementType(), GEP->getPointerOperand(),
-                         Indices, CostKind);
+                         Indices, nullptr, CostKind);
     }
   }
   return Cost;
 }
 
 void RISCVTTIImpl::getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                            TTI::UnrollingPreferences &UP,
                                            OptimizationRemarkEmitter *ORE) {

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

X86TTIImpl::getPointersChainCost(ArrayRef<const Value *> Ptrs,
                                  Type *AccessTy, TTI::TargetCostKind CostKind) {
   if (Info.isSameBase() && Info.isKnownStride()) {
     // If all the pointers have known stride all the differences are translated
     // into constants. X86 memory addressing allows encoding it into
     // displacement. So we just need to take the base GEP cost.
     if (const auto *BaseGEP = dyn_cast<GetElementPtrInst>(Base)) {
       SmallVector<const Value *> Indices(BaseGEP->indices());
       return getGEPCost(BaseGEP->getSourceElementType(),
-                        BaseGEP->getPointerOperand(), Indices, CostKind);
+                        BaseGEP->getPointerOperand(), Indices, nullptr,
+                        CostKind);
     }
     return TTI::TCC_Free;
   }
   return BaseT::getPointersChainCost(Ptrs, Base, Info, AccessTy, CostKind);
 }
 
 InstructionCost X86TTIImpl::getAddressComputationCost(Type *Ty,
                                                       ScalarEvolution *SE,

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (isa<LoadInst, StoreInst>(VL0)) {
 
       // Remark: it not quite correct to use scalar GEP cost for a vector GEP,
       // but it's not clear how to do that without having vector GEP arguments
       // ready.
       // Perhaps using just TTI::TCC_Free/TTI::TCC_Basic would be better option.
       if (const auto *Base = dyn_cast<GetElementPtrInst>(BasePtr)) {
         SmallVector<const Value *> Indices(Base->indices());
         VecCost = TTI->getGEPCost(Base->getSourceElementType(),
-                                  Base->getPointerOperand(), Indices, CostKind);
+                                  Base->getPointerOperand(), Indices, nullptr,
+                                  CostKind);
       }
     }
 
     LLVM_DEBUG(dumpTreeCosts(E, 0, VecCost, ScalarCost,
                              "Calculated GEPs cost for Tree"));
 
     return VecCost - ScalarCost;
   };

llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll

 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %gep3 = getelementptr i8, ptr %base, <16 x i32> %indsext
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 224 for instruction: %res3 = call <16 x i8> @llvm.masked.gather.v16i8.v16p0(<16 x ptr> %gep3, i32 2, <16 x i1> %mask, <16 x i8> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 224 for instruction: call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %res3, <16 x ptr> %gep3, i32 2, <16 x i1> %mask)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %gepbs = getelementptr i16, ptr %base16, <16 x i32> %indzext
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %gepbsb = bitcast <16 x ptr> %gepbs to <16 x ptr>
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 224 for instruction: %resbs = call <16 x i8> @llvm.masked.gather.v16i8.v16p0(<16 x ptr> %gepbsb, i32 2, <16 x i1> %mask, <16 x i8> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 224 for instruction: call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %resbs, <16 x ptr> %gepbsb, i32 2, <16 x i1> %mask)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 74 for instruction: %indzext4 = zext <16 x i8> %ind8 to <16 x i32>
-; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %gep4 = getelementptr i8, ptr %base, <16 x i32> %indzext
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %gep4 = getelementptr i8, ptr %base, <16 x i32> %indzext
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 32 for instruction: %indtrunc = trunc <16 x i32> %ind32 to <16 x i8>
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 32 for instruction: call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %indtrunc, <16 x ptr> %gep4, i32 2, <16 x i1> %mask)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
 ;
   ; no offset ext
   %gep1 = getelementptr i8, ptr %base, <16 x i32> %ind32
   %res1 = call <16 x i8> @llvm.masked.gather.v16i8.v16p0(<16 x ptr> %gep1, i32 1, <16 x i1> %mask, <16 x i8> undef)
   call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %res1, <16 x ptr> %gep1, i32 2, <16 x i1> %mask)

llvm/test/Analysis/CostModel/RISCV/gep.ll

 }
 
 ; Ensure that memory operations of a different type than the pointer source type
 ; use the correct type to determine if folding is possible. These operations
 ; are on vector types so there should be a cost for the GEP as the offset cannot
 ; be folded into the instruction.
 define void @non_foldable_vector_uses(ptr %base, <2 x ptr> %base.vec) {
 ; RVI-LABEL: 'non_foldable_vector_uses'
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %1 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %1 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %x1 = load volatile <2 x i8>, ptr %1, align 2
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %2 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %2 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %x2 = call <2 x i8> @llvm.masked.load.v2i8.p0(ptr %2, i32 1, <2 x i1> undef, <2 x i8> undef)
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %3 = getelementptr i8, <2 x ptr> %base.vec, <2 x i32> <i32 42, i32 43>
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %x3 = call <2 x i8> @llvm.masked.gather.v2i8.v2p0(<2 x ptr> %3, i32 1, <2 x i1> undef, <2 x i8> undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %4 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %4 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %x4 = call <2 x i8> @llvm.masked.expandload.v2i8(ptr %4, <2 x i1> undef, <2 x i8> undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %5 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %5 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %x5 = call <2 x i8> @llvm.vp.load.v2i8.p0(ptr %5, <2 x i1> undef, i32 undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %6 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %6 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %x6 = call <2 x i8> @llvm.experimental.vp.strided.load.v2i8.p0.i64(ptr %6, i64 undef, <2 x i1> undef, i32 undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %7 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %7 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: store volatile <2 x i8> undef, ptr %7, align 2
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %8 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %8 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: call void @llvm.masked.store.v2i8.p0(<2 x i8> undef, ptr %8, i32 1, <2 x i1> undef)
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %9 = getelementptr i8, <2 x ptr> %base.vec, <2 x i32> <i32 42, i32 43>
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: call void @llvm.masked.scatter.v2i8.v2p0(<2 x i8> undef, <2 x ptr> %9, i32 1, <2 x i1> undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %10 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %10 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.masked.compressstore.v2i8(<2 x i8> undef, ptr %10, <2 x i1> undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %11 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %11 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.vp.store.v2i8.p0(<2 x i8> undef, ptr %11, <2 x i1> undef, i32 undef)
-; RVI-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %12 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %12 = getelementptr i8, ptr %base, i32 42
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.experimental.vp.strided.store.v2i8.p0.i64(<2 x i8> undef, ptr %12, i64 undef, <2 x i1> undef, i32 undef)
 ; RVI-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: ret void
 ;
   %1 = getelementptr i8, ptr %base, i32 42
   %x1 = load volatile <2 x i8>, ptr %1
 
   %2 = getelementptr i8, ptr %base, i32 42
   %x2 = call <2 x i8> @llvm.masked.load.v2i8.p0(ptr %2, i32 1, <2 x i1> undef, <2 x i8> undef)