[AArch64][CostModel] Detects that {Extract,Insert}Element at lane 0 have the same cost as other lanes for real instructions that operates on integer types

[mingmingl]

Summary

Currently, {extract,insert}-element has zero cost at lane 0 [1]. However, there is a cost (by fmov instruction at lane 0 [2], or more generally ext/ins instruction) to move values from SIMD registers to GPR registers, when the element is used explicitly as integers.

See https://godbolt.org/z/faPE1nTn8, when fmov is generated for d* register -> x* register conversion.

Implementation-wise, add a private method AArch64TTIImpl::getVectorInstrCostHelper as a helper function. This way, instruction-based method could share the core logic (e.g., returning zero cost if type is legalized to scalar).

[1] https://github.com/llvm/llvm-project/blob/2cf320d41ed708679e01eeeb93f58d6c5c88ba7a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp#L1853
[2] https://github.com/llvm/llvm-project/blob/2cf320d41ed708679e01eeeb93f58d6c5c88ba7a/llvm/lib/Target/AArch64/AArch64InstrInfo.td#L8150-L8157

[nikic]

Changing the cost model (even if it impacts other places) is the right thing to do here. We cannot accept such special cases in LICM.

[mingmingl]

In D128302#3600211, @nikic wrote:

Changing the cost model (even if it impacts other places) is the right thing to do here. We cannot accept such special cases in LICM.

Acknowledged and it makes sense to me. Is it more idiomatic to create a new revision (to update the cost model) or just modify this existing revision?

[davidxl]

Keeping this revision with discussion history in one place sounds fine.

[mingmingl]

For getVectorInstrCost method, optionally pass a pointer of Instruction, and each target can retrieve the (use) context and customize cost estimation. For AArch64 implementation, fall back to use VectorInsertExtractBaseCost rather than regard extract element at index 0 as a free operation.

[davidxl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1879

missing return instruction ?

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1879

This falls through to use ST->getVectorInsertExtractBaseCost() around line 1903. Basically, this patch reduces the case when extracting element at index 0 is considered free (using truck as a baseline).

[davidxl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1879

ok -- my view of the patch hides the line :)

[nikic]

(Not really familiar with cost model APIs.)

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Yes, all users of an instruction are instructions, it's safe to use cast here.

1879

This should probably be just U.getOperandNo() == 0?

[mingmingl]

In AArch64TargetTransformInfo.cpp, change helper function isExtractedElementUsedAsInteger by

1. assert that all users of an instruction are instructions
2. simplify the store operation usage check, since integer could only be directly used as value as it is, the source of store operation (i.e., not the destination to store the value to).
3. require that uitofpinst/sitofpinst is the only use of extract-element if we think the extract-element operation is free.

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

thanks for taking a look! I looked at a few existing cpp and they also assume so.

Changing this to an assert then.

1879

I meant to check that the extract-element is used as the value, the source of store operation.

However, upon this comment, I realize that the extracted element could only be the source, since the destination should be a pointer type. Simplified the store operation check.

[mingmingl]

Change to AArch64TargetTransformInfo.cpp only:

1. Add the missing continue for store operation check.
2. Remove the OneUse restriction added for uitofpinst/sitofpinst.

[fhahn]

Hm, do we have precedence for doing a full scan of users in other cost APIs?

[RKSimon]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

92

any chance we can add a default (nullptr) arg value and avoid all the nullptr additions?

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1859

!isa_and_nonnull<ExtractElementInst>(Inst) ?

1868

Scanning the uses like this isn't going to be cheap - and getCost calls can be called a LOT.

Why did you decide this was necessary? Do existing tests change without it? You don't appear to have added any test coverage for these cases.

llvm/test/Analysis/CostModel/AArch64/kryo.ll

37

Add the other CHECKs for other extraction indices?

43

pre-commit the test with trunk codegen and rebase the patch to show the change in cost reporting

[mingmingl]

llvm/include/llvm/CodeGen/BasicTTIImpl.h

92

This is a good question.

If we want to have a default (nullptr) arg, each derived class (AArch64TargetTransformInfo, X86TargetTransformInfo, etc) needs to override the default pointer argument, since derived classes won't inherit the default argument of base class's virtual method [1] and it's suggested not to "change the default parameters of overridden inherited functions" [2].

I don't feel strong about this, but not adding a default following the existing style of Index -> Index has a default value of -1 in class TargetTransformInfo, but not in class BasicTTIImpl or derived classes.

May I know your thoughts about this? (leaving this comment open)

[1] https://stackoverflow.com/questions/3533589/can-virtual-functions-have-default-parameters
[2] http://www.gotw.ca/gotw/005.htm

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Scanning cost is a good point; one mitigation (in the patch) is that, there is a break out of the loop once an integer use case if ound.

May I measure the compile time increase with https://github.com/nikic/llvm-compile-time-tracker and report back? (leaving this open)

llvm/test/Analysis/CostModel/AArch64/kryo.ll

43

will do in another patch (got to reply first for compile-time question around scanning uses())

[mingmingl]

Changes

1. Update commit message to mirror phabricator summary.
2. Use`!isa_and_null`, and added the other CHECKs for other extraction indices, as suggested.

[mingmingl]

llvm/test/Analysis/CostModel/AArch64/kryo.ll

43

created https://reviews.llvm.org/D128945, with this test case, and another test case in LICM/AArch64/ subdirectory.

Both test cases are going to have diff, and the diff for LICM/AArch64/extract-element.ll would be like https://www.diffchecker.com/RyiE4Zn5 (code of trunk on the left, code of this patch on the right)

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Compared between HEAD~1 and this patch gives https://llvm-compile-time-tracker.com/compare.php?from=1e556f459b44dd0ca4073e932f66ecb6f40fe31a&to=63035f714c94d31698b5b4ab79b9912814c29345&stat=instructions

Across different configs and benchmarks, the compile-time increases or decreases within +/- 0.1%. I wonder if this means it's fine in terms of the compile-time cost?

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Actually I wasn't sure if AArch64 code path is executed at this point (without reading how tracker is implemented) even if the metrics (e.g., instructions) could be collected without running the code. Do you have other suggestions to measure the cost of iterating uses?

Why did you decide this was necessary?

In order to know if there is a scalar integer use (that results in additional operations to move / dup vector subregisters to general purpose registers), all uses of the extract-element instruction are enumerated here.

Do existing tests change without it?

The reason to handle store is based on this example (https://godbolt.org/z/fn4jWr5eW).

You don't appear to have added any test coverage for these cases.

I could add https://godbolt.org/z/fn4jWr5eW as a test case if it sounds good.

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Actually I wasn't sure if AArch64 code path is executed at this point (without reading how tracker is implemented) even if the metrics (e.g., instructions) could be collected without running the code. Do you have other suggestions to measure the cost of iterating uses?

Regarding compile-time change

1. The llvm-compile-time-tracker measures x86 code paths.
2. I did a similar comparison (by scraping scripts) for -DLLVM_TARGETS_TO_BUILD="AArch64" , and build llvm-test-suite with O3
   1. The diff of instruction counts are within +/- 0.1%, and the diff of cycles changes are within 2% (CTMark results uploaded
      test.csv1 KBDownload
      )

Regarding the pattern of iterating uses() and its cycle change

• I think it's not un-common for a pass to iterate uses() of instructions, and here only a subset of extract-element instructions uses are iterated.
• On the other hand, "used-as-integer" fact might be pre-computed and maintained inside class ExtractElementInst, so AArch64TTI can query / re-use the result, if we do want to avoid calculating it repeatedly.

Would like to hear some thoughts for the topics above, thanks!

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

Between iterating all uses and not looking at uses, another option for now is to look at use_empty(), since the instruction cost would ultimately be 0 (regardless of index value) if use_empty() returns true.

Together with this option, a FIXME might be added to point out if all uses are store operations that could be performed bitwise (i.e., without cross-register-bank copies), the extract-element operation is free.

[mingmingl]

Rebase on D128945.

[mingmingl]

Address concerns about scanning uses in cost API.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1868

To summarize the question and solution:

Since no full-scan of use exists due to compile-time concerns, simplify the helper function to look at 'use_empty' only, and added a FIXME for store operations.

[davidxl]

I want to suggest a different way of changing the interfaces to make the intention clearer:

1. add a new interface for vectorcost in TTI that takes an instruction (to replace the opcode parameter)
2. for places where context instruction is not available, no changes are needed;
3. in TTIImpl, by default, the instruction based interface is simply a wrapper to the opcode based interface
4. for aarch64, additional logic is added for checking the lane ...

[mingmingl]

Address the comments by an interface change.

[davidxl]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1155

Perhaps document this that this is used when the instruction is not available.

1164

Mention that this interface is used when I is available (and the interface implementation asserts it is not nullptr).

llvm/lib/Analysis/TargetTransformInfo.cpp

866

probably submit this as a NFC patch seperately?

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

176

what is the purpose of the using statement here?

[mingmingl]

Address comments.

[mingmingl]

llvm/include/llvm/Analysis/TargetTransformInfo.h

1155

Done. Mention the typical use as provision vectorization/scalarization cost in vectorizer passes.

llvm/lib/Analysis/TargetTransformInfo.cpp

866

Sure. Will leave the FIXME inside this function, and the assert the newly added 'getVectorInstrCost'.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

176

Oh good catch. This is a mistake. 'using statement' is not needed for AArch64 since there is an override for new interface.

[davidxl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

176

Is it needed for other targets? The new method is a public method in BasicTTIImplBase, which is inherited and visible in the derived class.

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

176

Is it needed for other targets?

Yes, and I learnt it by seeing the compile error without 'using statements'.

Basically, the overloaded method in derived class could hide method in the base class, even if function signature is different. There is one example, and one relevant discussion.

Also, renaming the new interface differently also compiles, since name hiding doesn't exist by renaming.

There are using BaseT::getIntImmCost statements in <Target>TargetTransformInfo.cpp (example) (not sure if 'getIntImmCost' still helps with name hiding now, since overtime many targets overload the method with the same function signature)

[davidxl]

Since the use scanning code is removed (which raised some compile time concern), and latest revisions looks good to me. Please wait a little for other reviewers to comment.

[mingmingl]

ping.

[davidxl]

Consider split this into two patches: the first NFC just introduces the new interface (as a simple wrapper to the existing opcode based one). The second one is the functional change.

[mingmingl]

In D128302#3696604, @davidxl wrote:

Consider split this into two patches: the first NFC just introduces the new interface (as a simple wrapper to the existing opcode based one). The second one is the functional change.

Sent out D131114 as NFC.

[mingmingl]

Rebase after NFC.

[mingmingl]

rebase after D131197

[RKSimon]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1887

do you need the this-> ?

[mingmingl]

resolve comments.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1887

Ah this-> is not needed, fixed it.

(Sorry for the delay; I was out in the past week)

[RKSimon]

LGTM but a aarch64 specialist should sign off as well

[fhahn]

Just a general comment: please try to include the full context when uploading a diff: https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1887

nit: LLVM coding style uses UpperCase for variables.

1993

I might be missing something here, but the comment here doesn't seem to match the code, which just checks if the result is used. I'd expect that in all interesting cases the result will be used. And if it is not used the cost is not really that interesting, as it is effectively a no-op that should be removed.

[mingmingl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1993

It makes complete sense that instructions without users are not interesting, and [1] is sufficient for this patch (will hold on updating diff for better context). Let me elaborate why this patch and it'd be great to hear your thoughts! (This is a lot of words despite efforts to simplify them.. hopefully structure / bold fonts makes it easier. Thanks for reviewing this patch and reading the replies!)

About the motivating use case

• It is shown in LICM/AArch64/extract-element.ll -> an extract-element instruction shouldn't be considered free (and thereby sinked to loop exit) by LICM when cost kind is TCK_SizeAndLatency -> extract-element has size and latency cost, unless it could be "combined" into its user instructions (e.g., user instruction can access lane directly).
• Say the loop-exit is frequently executed in a hot path, sink extract-element (along with a series of its def instructions) is inefficient.
• Redundancy elimination might be an alternative option but I haven't got a chance to look into the alternative option yet, with the thought that fixing cost model is overall a good change as well :)*

About the motivation of this patch

• When the instruction exists in IR (indicated by function parameter const Instruction& I) and extract result is integer (a stronger signal of a fmov being emitted), this patch use the same cost all lanes (not considering lane 0 free).

So why this patch wants to tell a real vector instruction from a virtual one (by asking for const Instruction&I, not just Opcode)?

• Vectorizer passes have Opcode but not real instructions in its provisioning stage [2]. Not telling real from virtual may make vectorizer result better or worse in a fragile way.
• Plus, a new interface of getVectorInstCost won't mess up the codebase (w.r.t API/code style) when improvements for vectorizer cost models are added in the future.

But why vectorizer gives better or worse result in a fragile way?

• Take BaseTTIImpl::getScalarizationOverhead as an example, it calls getVectorInstrCost element-by-element for virtual instructions. With various SIMD instructions, element-by-element estimation is not accurate for AArch64.
• AArch64TTI doesn't override this method so BaseTTIImpl is used. Just as a reference, x86 backend overrides this method for some specialized cases.
• If this patch is NOT no-op for method getScalarizationOverhead, it could exacerbates the inaccurate cost model (SLP, etc) and misleads vectorizers.
• In practice, when non-zero cost for lane 0 is applied for "provisioned" vector instructions, some IR from llvm/test/Transforms/<vectorizer-subdir>/AArch64/ gets better generated code and some gets worse generated code. I didn't inspect them all, but presumably there are going to be better or worse cases, since without this change vectorizers sometimes gives worse output (e.g., https://godbolt.org/z/c1bjhGdT5 is for llvm/test/Transforms/SLPVectorizer/AArch64/ext-trunc.ll)

A related question, I wonder if it's with a reason (or just unintended status quo) that getVectorInstrCost interface doesn't ask for a CostKind parameter?

[1]

+InstructionCost AArch64TTIImpl::getVectorInstrCost(const Instruction &I,
+                                                   Type *Val, unsigned Index) {
+  InstructionCost cost = getVectorInstrCost(I.getOpcode(), Val, Index);
+
+  // According to NEON programmer guide, other than multiply instructions,
+  // instructions that access scalars can access any element in the vector register.
+  //
+  // The cost of an extract depends on whether the user instruction.
+  // 1. If users could use scalars in vector registers directly (e.g., store), the
+  // extract-element operation is essentially free.
+  // 2. If the user instruction requires core register as operand (i.e.,
+  // cannot use scalars in vector register), an explicit move operation will
+  // be codegen'd.
+  //
+  // 'cost' might be an optimistic 0 when lane is 0.
+  // Returns the base cost (as the same with other lanes) if the scalar is an integer.
+  //
+  // FIXME:
+  // The cost of extract-element from vector is better estimated with context (e.g. users).
+  if (!isa_and_nonnull<ExtractElementInst>(&I) || !Val->getScalarType()->isIntegerTy()) 
+    return ST->getVectorInsertExtractBaseCost();
+  return cost;
+}
+

[2] Basically, vectorizer pass sees one form (scalar or vector) of "physical" instructions (that do exists in the IR), and tries to "provision" the cost of the other form; this means instructions of the other form doesn't exist yet, and only Opcode is known and used to provision the cost.

[mingmingl]

In D128302#3725318, @fhahn wrote:

Just a general comment: please try to include the full context when uploading a diff: https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

Yeah I will remember to do this from now on (already did it earlier this week upon suggestions from another reviewer, and a partial diff was purely unintentional.) Also will hold on updating the patch to keep the patch context for discussion (leave all comments open).

[davidxl]

I guess fhahn@ meant that for all useful scenarios, the instruction needs to have some uses (otherwise they are dead). For this case, the check should probably be tightened : instead of checking if the instruction has uses, it needs to examine the uses and differentiate integer uses that requires a fmov and other use contexts (which makes the extract noop).

Checking use emptiness can pessimize cases where the cost is zero, but it returns non-zero cost.

[RKSimon]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1993

A related question, I wonder if it's with a reason (or just unintended status quo) that getVectorInstrCost interface doesn't ask for a CostKind parameter?

This is on my TODO list that was waiting for the getInstructionCost refactor (D79483) - but I want to get some other CostKind changes (notably D132216) out of the way first.

[mingmingl]

Add a private method AArch64TTIImpl::getVectorInstrCostHelper as a helper function. This way, instruction-based method could share the core logic (e.g.,
returning zero cost if type is legalized to scalar).

Uploaded with full context and updated all affected test cases.

[mingmingl]

In D128302#3743563, @davidxl wrote:

I guess fhahn@ meant that for all useful scenarios, the instruction needs to have some uses (otherwise they are dead). For this case, the check should probably be tightened : instead of checking if the instruction has uses, it needs to examine the uses and differentiate integer uses that requires a fmov and other use contexts (which makes the extract noop).

Added a FIXME to estimate extract/insert cost with use-def contexts.

Checking use emptiness can pessimize cases where the cost is zero, but it returns non-zero cost.

In D128302#3731045, @mingmingl wrote:

In D128302#3725318, @fhahn wrote:

Just a general comment: please try to include the full context when uploading a diff: https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

Yeah I will remember to do this from now on (already did it earlier this week upon suggestions from another reviewer, and a partial diff was purely unintentional.) Also will hold on updating the patch to keep the patch context for discussion (leave all comments open).

Uploaded with full context this time.

I'd like to point out the current patch still lacks test coverage for "virtual" vector instructions in print<cost-model> pass, due to the fact that instructions are physical in an IR. Will upload to get preliminary feedback on the overall direction -> it's feasible to get back test coverage, by calling op-based (not instruction based) getVectorInstrCost in print<cost-model> pass (the callsite of op-based cost API could be gated by a boolean option and turned on where it's needed).

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1887

With the refactoring this variable is gone. Updated patch should use UpperCase variables already.

1993

Thanks for making the cost-model changes! (mark this as done).

1993

Resolved by a refactoring that includes code sharing, comment update and dropping the uninteresting case.

Redundancy elimination might be an alternative option but I haven't got a chance to look into the alternative option yet, with the thought that fixing cost model is overall a good change as well :)*

Just to provide an update on this, looked into how to optimize by redundancy elimination this week -> machine-cse does eliminate the redundant machine instructions (and LICM/AArch64/extract-element.ll is supposed to be a reduced test case).

[davidxl]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

68

Maybe changing 'VirtualInst' to be name 'hasNoRealUse' or flip it to 'hasRealUse'?

[mingmingl]

address comment.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

68

Change to 'HasRealUse'. An instruction is a derived class of 'User', but I think it's fine to call it a use without ambiguity here.

[davidxl]

The patch looks reasonable to me. Check if @fhahn @RKSimon have further comments.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1855–1877

VirtualInst--> HasRealUse.

[mingmingl]

fix comment after renaming variable (VirtualInst->HasRealUse).

[RKSimon]

llvm/test/Analysis/CostModel/AArch64/kryo-inseltpoison.ll

9 ↗

(On Diff #458884)

update the comment

llvm/test/Analysis/CostModel/AArch64/kryo.ll

9

update the comment

[mingmingl]

updated test comments

[mingmingl]

Thanks for reviews! Going to commit.

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
   /// is using a compare with the specified predicate as condition. When vector
   /// types are passed, \p VecPred must be used for all lanes.
   InstructionCost
   getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                      CmpInst::Predicate VecPred,
                      TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
                      const Instruction *I = nullptr) const;
 
   /// \return The expected cost of vector Insert and Extract.
   /// Use -1 to indicate that there is no information on the index value.
+  /// Caller can provide the instruction that holds Opcode in `I`, or use
+  /// nullptr to indcate there is no instruction information.
   InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index = -1) const;
+                                     unsigned Index = -1,
+                                     const Instruction *I = nullptr) const;
 
   /// \return The cost of replication shuffle of \p VF elements typed \p EltTy
   /// \p ReplicationFactor times.
   ///
   /// For example, the mask for \p ReplicationFactor=3 and \p VF=4 is:
   ///   <0,0,0,1,1,1,2,2,2,3,3,3>
   InstructionCost getReplicationShuffleCost(Type *EltTy, int ReplicationFactor,
                                             int VF,
                                             const APInt &DemandedDstElts,
                                             TTI::TargetCostKind CostKind);
 
virtual InstructionCost getCFInstrCost(unsigned Opcode,
                                          TTI::TargetCostKind CostKind,
                                          const Instruction *I = nullptr) = 0;
   virtual InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                              Type *CondTy,
                                              CmpInst::Predicate VecPred,
                                              TTI::TargetCostKind CostKind,
                                              const Instruction *I) = 0;
   virtual InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                             unsigned Index) = 0;
+                                             unsigned Index,
+                                             const Instruction *I) = 0;
 
   virtual InstructionCost
   getReplicationShuffleCost(Type *EltTy, int ReplicationFactor, int VF,
                             const APInt &DemandedDstElts,
                             TTI::TargetCostKind CostKind) = 0;
 
   virtual InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,
                                           Align Alignment,
InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
     return Impl.getCFInstrCost(Opcode, CostKind, I);
   }
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I) override {
     return Impl.getCmpSelInstrCost(Opcode, ValTy, CondTy, VecPred, CostKind, I);
   }
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index) override {
-    return Impl.getVectorInstrCost(Opcode, Val, Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I) override {
+    return Impl.getVectorInstrCost(Opcode, Val, Index, I);
   }
   InstructionCost
   getReplicationShuffleCost(Type *EltTy, int ReplicationFactor, int VF,
                             const APInt &DemandedDstElts,
                             TTI::TargetCostKind CostKind) override {
     return Impl.getReplicationShuffleCost(EltTy, ReplicationFactor, VF,
                                           DemandedDstElts, CostKind);
   }

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I) const {
     return 1;
   }
 
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index) const {
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I) const {
     return 1;
   }
 
   unsigned getReplicationShuffleCost(Type *EltTy, int ReplicationFactor, int VF,
                                      const APInt &DemandedDstElts,
                                      TTI::TargetCostKind CostKind) {
     return 1;
   }
InstructionCost getUserCost(const User *U, ArrayRef<const Value *> Operands,
     case Instruction::InsertElement: {
       auto *IE = dyn_cast<InsertElementInst>(U);
       if (!IE)
         return TTI::TCC_Basic; // FIXME
       unsigned Idx = -1;
       if (auto *CI = dyn_cast<ConstantInt>(IE->getOperand(2)))
         if (CI->getValue().getActiveBits() <= 32)
           Idx = CI->getZExtValue();
-      return TargetTTI->getVectorInstrCost(Opcode, Ty, Idx);
+      return TargetTTI->getVectorInstrCost(Opcode, Ty, Idx, I);
     }
     case Instruction::ShuffleVector: {
       auto *Shuffle = dyn_cast<ShuffleVectorInst>(U);
       if (!Shuffle)
         return TTI::TCC_Basic; // FIXME
 
       auto *VecTy = cast<VectorType>(U->getType());
       auto *VecSrcTy = cast<VectorType>(U->getOperand(0)->getType());
case Instruction::ExtractElement: {
       auto *EEI = dyn_cast<ExtractElementInst>(U);
       if (!EEI)
         return TTI::TCC_Basic; // FIXME
       unsigned Idx = -1;
       if (auto *CI = dyn_cast<ConstantInt>(EEI->getOperand(1)))
         if (CI->getValue().getActiveBits() <= 32)
           Idx = CI->getZExtValue();
       Type *DstTy = U->getOperand(0)->getType();
-      return TargetTTI->getVectorInstrCost(Opcode, DstTy, Idx);
+      return TargetTTI->getVectorInstrCost(Opcode, DstTy, Idx, I);
     }
     }
     // By default, just classify everything as 'basic'.
     return TTI::TCC_Basic;
   }
 
   InstructionCost getInstructionLatency(const Instruction *I) {
     SmallVector<const Value *, 4> Operands(I->operand_values());

llvm/include/llvm/CodeGen/BasicTTIImpl.h

private:
   T *thisT() { return static_cast<T *>(this); }
 
   /// Estimate a cost of Broadcast as an extract and sequence of insert
   /// operations.
   InstructionCost getBroadcastShuffleOverhead(FixedVectorType *VTy) {
     InstructionCost Cost = 0;
     // Broadcast cost is equal to the cost of extracting the zero'th element
     // plus the cost of inserting it into every element of the result vector.
-    Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, VTy, 0);
+    Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, VTy, 0,
+                                        nullptr);
 
     for (int i = 0, e = VTy->getNumElements(); i < e; ++i) {
-      Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, VTy, i);
+      Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, VTy, i,
+                                          nullptr);
     }
     return Cost;
   }
 
   /// Estimate a cost of shuffle as a sequence of extract and insert
   /// operations.
   InstructionCost getPermuteShuffleOverhead(FixedVectorType *VTy) {
     InstructionCost Cost = 0;
     // Shuffle cost is equal to the cost of extracting element from its argument
     // plus the cost of inserting them onto the result vector.
 
     // e.g. <4 x float> has a mask of <0,5,2,7> i.e we need to extract from
     // index 0 of first vector, index 1 of second vector,index 2 of first
     // vector and finally index 3 of second vector and insert them at index
     // <0,1,2,3> of result vector.
     for (int i = 0, e = VTy->getNumElements(); i < e; ++i) {
-      Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, VTy, i);
-      Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, VTy, i);
+      Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, VTy, i,
+                                          nullptr);
+      Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, VTy, i,
+                                          nullptr);
     }
     return Cost;
   }
 
   /// Estimate a cost of subvector extraction as a sequence of extract and
   /// insert operations.
   InstructionCost getExtractSubvectorOverhead(VectorType *VTy, int Index,
                                        FixedVectorType *SubVTy) {
     assert(VTy && SubVTy &&
            "Can only extract subvectors from vectors");
     int NumSubElts = SubVTy->getNumElements();
     assert((!isa<FixedVectorType>(VTy) ||
             (Index + NumSubElts) <=
                 (int)cast<FixedVectorType>(VTy)->getNumElements()) &&
            "SK_ExtractSubvector index out of range");
 
     InstructionCost Cost = 0;
     // Subvector extraction cost is equal to the cost of extracting element from
     // the source type plus the cost of inserting them into the result vector
     // type.
     for (int i = 0; i != NumSubElts; ++i) {
       Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, VTy,
-                                          i + Index);
-      Cost +=
-          thisT()->getVectorInstrCost(Instruction::InsertElement, SubVTy, i);
+                                          i + Index, nullptr);
+      Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, SubVTy, i,
+                                          nullptr);
     }
     return Cost;
   }
 
   /// Estimate a cost of subvector insertion as a sequence of extract and
   /// insert operations.
   InstructionCost getInsertSubvectorOverhead(VectorType *VTy, int Index,
                                       FixedVectorType *SubVTy) {
     assert(VTy && SubVTy &&
            "Can only insert subvectors into vectors");
     int NumSubElts = SubVTy->getNumElements();
     assert((!isa<FixedVectorType>(VTy) ||
             (Index + NumSubElts) <=
                 (int)cast<FixedVectorType>(VTy)->getNumElements()) &&
            "SK_InsertSubvector index out of range");
 
     InstructionCost Cost = 0;
     // Subvector insertion cost is equal to the cost of extracting element from
     // the source type plus the cost of inserting them into the result vector
     // type.
     for (int i = 0; i != NumSubElts; ++i) {
-      Cost +=
-          thisT()->getVectorInstrCost(Instruction::ExtractElement, SubVTy, i);
+      Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, SubVTy,
+                                          i, nullptr);
       Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, VTy,
-                                          i + Index);
+                                          i + Index, nullptr);
     }
     return Cost;
   }
 
   /// Local query method delegates up to T which *must* implement this!
   const TargetSubtargetInfo *getST() const {
     return static_cast<const T *>(this)->getST();
   }
InstructionCost getCommonMaskedMemoryOpCost(unsigned Opcode, Type *DataTy,
     //
     // First, compute the cost of the individual memory operations.
     InstructionCost AddrExtractCost =
         IsGatherScatter
             ? getVectorInstrCost(Instruction::ExtractElement,
                                  FixedVectorType::get(
                                      PointerType::get(VT->getElementType(), 0),
                                      VT->getNumElements()),
-                                 -1)
+                                 -1, nullptr)
             : 0;
     InstructionCost LoadCost =
         VT->getNumElements() *
         (AddrExtractCost +
          getMemoryOpCost(Opcode, VT->getElementType(), Alignment, 0, CostKind));
 
     // Next, compute the cost of packing the result in a vector.
     InstructionCost PackingCost = getScalarizationOverhead(
         VT, Opcode != Instruction::Store, Opcode == Instruction::Store);
 
     InstructionCost ConditionalCost = 0;
     if (VariableMask) {
       // Compute the cost of conditionally executing the memory operations with
       // variable masks. This includes extracting the individual conditions, a
       // branches and PHIs to combine the results.
       // NOTE: Estimating the cost of conditionally executing the memory
       // operations accurately is quite difficult and the current solution
       // provides a very rough estimate only.
       ConditionalCost =
           VT->getNumElements() *
           (getVectorInstrCost(
                Instruction::ExtractElement,
                FixedVectorType::get(Type::getInt1Ty(DataTy->getContext()),
                                     VT->getNumElements()),
-               -1) +
+               -1, nullptr) +
            getCFInstrCost(Instruction::Br, CostKind) +
            getCFInstrCost(Instruction::PHI, CostKind));
     }
 
     return LoadCost + PackingCost + ConditionalCost;
   }
 
 protected:
assert(DemandedElts.getBitWidth() == Ty->getNumElements() &&
            "Vector size mismatch");
 
     InstructionCost Cost = 0;
 
     for (int i = 0, e = Ty->getNumElements(); i < e; ++i) {
       if (!DemandedElts[i])
         continue;
       if (Insert)
-        Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, Ty, i);
+        Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, Ty, i,
+                                            nullptr);
       if (Extract)
-        Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, i);
+        Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, i,
+                                            nullptr);
     }
 
     return Cost;
   }
 
   /// Helper wrapper for the DemandedElts variant of getScalarizationOverhead.
   InstructionCost getScalarizationOverhead(VectorType *InTy, bool Insert,
                                            bool Extract) {
InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
     }
 
     llvm_unreachable("Unhandled cast");
   }
 
   InstructionCost getExtractWithExtendCost(unsigned Opcode, Type *Dst,
                                            VectorType *VecTy, unsigned Index) {
     return thisT()->getVectorInstrCost(Instruction::ExtractElement, VecTy,
-                                       Index) +
+                                       Index, nullptr) +
            thisT()->getCastInstrCost(Opcode, Dst, VecTy->getElementType(),
                                      TTI::CastContextHint::None,
                                      TTI::TCK_RecipThroughput);
   }
 
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr) {
     return BaseT::getCFInstrCost(Opcode, CostKind, I);
if (auto *ValVTy = dyn_cast<VectorType>(ValTy)) {
       // inserting and extracting the values.
       return getScalarizationOverhead(ValVTy, true, false) + Num * Cost;
     }
 
     // Unknown scalar opcode.
     return 1;
   }
 
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index) {
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I = nullptr) {
     std::pair<InstructionCost, MVT> LT =
         getTLI()->getTypeLegalizationCost(DL, Val->getScalarType());
 
     return LT.first;
   }
 
   InstructionCost getReplicationShuffleCost(Type *EltTy, int ReplicationFactor,
                                             int VF,
InstructionCost getTreeReductionCost(unsigned Opcode, VectorType *Ty,
     // architecture-dependent length.
 
     // By default reductions need one shuffle per reduction level.
     ShuffleCost += NumReduxLevels * thisT()->getShuffleCost(
                                      TTI::SK_PermuteSingleSrc, Ty, None, 0, Ty);
     ArithCost +=
         NumReduxLevels * thisT()->getArithmeticInstrCost(Opcode, Ty, CostKind);
     return ShuffleCost + ArithCost +
-           thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, 0);
+           thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, 0,
+                                       nullptr);
   }
 
   /// Try to calculate the cost of performing strict (in-order) reductions,
   /// which involves doing a sequence of floating point additions in lane
   /// order, starting with an initial value. For example, consider a scalar
   /// initial value 'InitVal' of type float and a vector of type <4 x float>:
   ///
   ///   Vector = <float %v0, float %v1, float %v2, float %v3>
MinMaxCost +=
         NumReduxLevels *
         (thisT()->getCmpSelInstrCost(CmpOpcode, Ty, CondTy,
                                      CmpInst::BAD_ICMP_PREDICATE, CostKind) +
          thisT()->getCmpSelInstrCost(Instruction::Select, Ty, CondTy,
                                      CmpInst::BAD_ICMP_PREDICATE, CostKind));
     // The last min/max should be in vector registers and we counted it above.
     // So just need a single extractelement.
     return ShuffleCost + MinMaxCost +
-           thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, 0);
+           thisT()->getVectorInstrCost(Instruction::ExtractElement, Ty, 0,
+                                       nullptr);
   }
 
   InstructionCost getExtendedAddReductionCost(bool IsMLA, bool IsUnsigned,
                                               Type *ResTy, VectorType *Ty,
                                               TTI::TargetCostKind CostKind) {
     // Without any native support, this is equivalent to the cost of
     // vecreduce.add(ext) or if IsMLA vecreduce.add(mul(ext, ext))
     VectorType *ExtTy = VectorType::get(ResTy, Ty);

llvm/lib/Analysis/TargetTransformInfo.cpp

InstructionCost TargetTransformInfo::getCmpSelInstrCost(
   assert((I == nullptr || I->getOpcode() == Opcode) &&
          "Opcode should reflect passed instruction.");
   InstructionCost Cost =
       TTIImpl->getCmpSelInstrCost(Opcode, ValTy, CondTy, VecPred, CostKind, I);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
-InstructionCost TargetTransformInfo::getVectorInstrCost(unsigned Opcode,
-                                                        Type *Val,
-                                                        unsigned Index) const {
-  InstructionCost Cost = TTIImpl->getVectorInstrCost(Opcode, Val, Index);
+InstructionCost TargetTransformInfo::getVectorInstrCost(
+    unsigned Opcode, Type *Val, unsigned Index, const Instruction *I) const {
+  InstructionCost Cost = TTIImpl->getVectorInstrCost(Opcode, Val, Index, I);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
 InstructionCost TargetTransformInfo::getReplicationShuffleCost(
     Type *EltTy, int ReplicationFactor, int VF, const APInt &DemandedDstElts,
     TTI::TargetCostKind CostKind) {
   InstructionCost Cost = TTIImpl->getReplicationShuffleCost(
       EltTy, ReplicationFactor, VF, DemandedDstElts, CostKind);

llvm/lib/CodeGen/CodeGenPrepare.cpp

if (!TLI.allowsMisalignedMemoryAccesses(
       // If this is not supported, there is no way we can combine
       // the extract with the store.
       return false;
     }
 
     // The scalar chain of computation has to pay for the transition
     // scalar to vector.
     // The vector chain has to account for the combining cost.
-    InstructionCost ScalarCost =
-        TTI.getVectorInstrCost(Transition->getOpcode(), PromotedType, Index);
+    InstructionCost ScalarCost = TTI.getVectorInstrCost(
+        Transition->getOpcode(), PromotedType, Index, Transition);
     InstructionCost VectorCost = StoreExtractCombineCost;
     enum TargetTransformInfo::TargetCostKind CostKind =
       TargetTransformInfo::TCK_RecipThroughput;
     for (const auto &Inst : InstsToBePromoted) {
       // Compute the cost.
       // By construction, all instructions being promoted are arithmetic ones.
       // Moreover, one argument is a constant that can be viewed as a splat
       // constant.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

bool isWideningInstruction(Type *Ty, unsigned Opcode,
                              ArrayRef<const Value *> Args);
 
 public:
   explicit AArch64TTIImpl(const AArch64TargetMachine *TM, const Function &F)
       : BaseT(TM, F.getParent()->getDataLayout()), ST(TM->getSubtargetImpl(F)),
         TLI(ST->getTargetLowering()) {}
 
   bool areInlineCompatible(const Function *Caller,
                            const Function *Callee) const;
 
   /// \name Scalar TTI Implementations
   /// @{
 
   using BaseT::getIntImmCost;
   InstructionCost getIntImmCost(int64_t Val);
   InstructionCost getIntImmCost(const APInt &Imm, Type *Ty,
                                 TTI::TargetCostKind CostKind);
InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    const Instruction *I = nullptr);
 
   InstructionCost getExtractWithExtendCost(unsigned Opcode, Type *Dst,
                                            VectorType *VecTy, unsigned Index);
 
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr);
 
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
 
   InstructionCost getMinMaxReductionCost(VectorType *Ty, VectorType *CondTy,
                                          bool IsUnsigned,
                                          TTI::TargetCostKind CostKind);
 
   InstructionCost getArithmeticReductionCostSVE(unsigned Opcode,
                                                 VectorType *ValTy,
                                                 TTI::TargetCostKind CostKind);

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

InstructionCost AArch64TTIImpl::getExtractWithExtendCost(unsigned Opcode,
   // of the extend is the element type of the vector.
   auto *Src = VecTy->getElementType();
 
   // Sign- and zero-extends are for integer types only.
   assert(isa<IntegerType>(Dst) && isa<IntegerType>(Src) && "Invalid type");
 
   // Get the cost for the extract. We compute the cost (if any) for the extend
   // below.
+  //
+  // FIXME: Change signature of `getExtractWithExtendCost` so caller
+  // can optionally provide the extract-element instruction as context.
   InstructionCost Cost =
-      getVectorInstrCost(Instruction::ExtractElement, VecTy, Index);
+      getVectorInstrCost(Instruction::ExtractElement, VecTy, Index, nullptr);
 
   // Legalize the types.
   auto VecLT = TLI->getTypeLegalizationCost(DL, VecTy);
   auto DstVT = TLI->getValueType(DL, Dst);
   auto SrcVT = TLI->getValueType(DL, Src);
   TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput;
 
   // If the resulting type is still a vector and the destination type is legal,
InstructionCost AArch64TTIImpl::getCFInstrCost(unsigned Opcode,
   if (CostKind != TTI::TCK_RecipThroughput)
     return Opcode == Instruction::PHI ? 0 : 1;
   assert(CostKind == TTI::TCK_RecipThroughput && "unexpected CostKind");
   // Branches are assumed to be predicted.
   return 0;
 }
 
 InstructionCost AArch64TTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
-                                                   unsigned Index) {
+                                                   unsigned Index,
+                                                   const Instruction *I) {
   assert(Val->isVectorTy() && "This must be a vector type");
 
   if (Index != -1U) {
     // Legalize the type.
     std::pair<InstructionCost, MVT> LT = TLI->getTypeLegalizationCost(DL, Val);
 
     // This type is legalized to a scalar type.
     if (!LT.second.isVector())
       return 0;
 
     // The type may be split. For fixed-width vectors we can normalize the
     // index to the new type.
     if (LT.second.isFixedLengthVector()) {
       unsigned Width = LT.second.getVectorNumElements();
       Index = Index % Width;
     }
 
-    // The element at index zero is already inside the vector.
-    if (Index == 0)
+    // FIXME: Use `IsExtractedElementUsedAsInteger` to decide cost when index is
+    // explicit (i.e., not -1, not limited to 0).
+    auto IsExtractedElementUsedAsInteger =
+        [Val](const Instruction *Inst) -> bool {
+      if (!isa_and_nonnull<ExtractElementInst>(Inst) ||
+          !Val->getScalarType()->isIntegerTy())
+        return false;
+
+      // FIXME:
+      // If the extracted element is used in a store operation, it could be
+      // stored bit-wise without an explicit conversion to integer. So if all
+      // uses are store related, the extract-element operation is essentially
+      // free.
+      // This is not done initially since doing a full scan of users in cost API
+      // might increase compile time, and no precedence of full scan exists.
+      return !Inst->use_empty();
+    };
+
+    // For extract_element instruction, the element at index zero is already
+    // inside the floating-point subregister.
+    //
+    // If element type is integer, an explicit move operation will be codegen'd.
+    if (Index == 0 && !IsExtractedElementUsedAsInteger(I))
       return 0;
   }
 
   // All other insert/extracts cost this much.
   return ST->getVectorInsertExtractBaseCost();
 }
 
 InstructionCost AArch64TTIImpl::getArithmeticInstrCost(
     unsigned Opcode, Type *Ty, TTI::TargetCostKind CostKind,
     TTI::OperandValueKind Opd1Info, TTI::OperandValueKind Opd2Info,
     TTI::OperandValueProperties Opd1PropInfo,
     TTI::OperandValueProperties Opd2PropInfo, ArrayRef<const Value *> Args,
     const Instruction *CxtI) {
   // TODO: Handle more cost kinds.
   if (CostKind != TTI::TCK_RecipThroughput)
     return BaseT::getArithmeticInstrCost(Opcode, Ty, CostKind, Opd1Info,
                                          Opd2Info, Opd1PropInfo,
                                          Opd2PropInfo, Args, CxtI);
case ISD::SHL:
     // These nodes are marked as 'custom' for combining purposes only.
     // We know that they are legal. See LowerAdd in ISelLowering.
     return LT.first;
 
   case ISD::FADD:
   case ISD::FSUB:
   case ISD::FMUL:
   case ISD::FDIV:
   case ISD::FNEG:
     // These nodes are marked as 'custom' just to lower them to SVE.
     // We know said lowering will incur no additional cost.
     if (!Ty->getScalarType()->isFP128Ty())
       return 2 * LT.first;
 
     return BaseT::getArithmeticInstrCost(Opcode, Ty, CostKind, Opd1Info,
                                          Opd2Info, Opd1PropInfo, Opd2PropInfo);
   }

llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h

public:
 
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr);
 
   bool isInlineAsmSourceOfDivergence(const CallInst *CI,
                                      ArrayRef<unsigned> Indices = {}) const;
 
   InstructionCost getVectorInstrCost(unsigned Opcode, Type *ValTy,
-                                     unsigned Index);
+                                     unsigned Index, const Instruction *I);
   bool isSourceOfDivergence(const Value *V) const;
   bool isAlwaysUniform(const Value *V) const;
 
   unsigned getFlatAddressSpace() const {
     // Don't bother running InferAddressSpaces pass on graphics shaders which
     // don't use flat addressing.
     if (IsGraphics)
       return -1;

llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp

GCNTTIImpl::getMinMaxReductionCost(VectorType *Ty, VectorType *CondTy,
   if (!ST->hasVOP3PInsts() || OrigTy.getScalarSizeInBits() != 16)
     return BaseT::getMinMaxReductionCost(Ty, CondTy, IsUnsigned, CostKind);
 
   std::pair<InstructionCost, MVT> LT = TLI->getTypeLegalizationCost(DL, Ty);
   return LT.first * getHalfRateInstrCost(CostKind);
 }
 
 InstructionCost GCNTTIImpl::getVectorInstrCost(unsigned Opcode, Type *ValTy,
-                                               unsigned Index) {
+                                               unsigned Index,
+                                               const Instruction *I) {
   switch (Opcode) {
   case Instruction::ExtractElement:
   case Instruction::InsertElement: {
     unsigned EltSize
       = DL.getTypeSizeInBits(cast<VectorType>(ValTy)->getElementType());
     if (EltSize < 32) {
       if (EltSize == 16 && Index == 0 && ST->has16BitInsts())
         return 0;
-      return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
+      return BaseT::getVectorInstrCost(Opcode, ValTy, Index, I);
     }
 
     // Extracts are just reads of a subregister, so are free. Inserts are
     // considered free because we don't want to have any cost for scalarizing
     // operations, and we don't have to copy into a different register class.
 
     // Dynamic indexing isn't free and is best avoided.
     return Index == ~0u ? 2 : 0;
   }
   default:
-    return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
+    return BaseT::getVectorInstrCost(Opcode, ValTy, Index, I);
   }
 }
 
 /// Analyze if the results of inline asm are divergent. If \p Indices is empty,
 /// this is analyzing the collective result of all output registers. Otherwise,
 /// this is only querying a specific result index if this returns multiple
 /// registers in a struct.
 bool GCNTTIImpl::isInlineAsmSourceOfDivergence(

llvm/lib/Target/AMDGPU/R600TargetTransformInfo.h

public:
   bool isLegalToVectorizeLoadChain(unsigned ChainSizeInBytes, Align Alignment,
                                    unsigned AddrSpace) const;
   bool isLegalToVectorizeStoreChain(unsigned ChainSizeInBytes, Align Alignment,
                                     unsigned AddrSpace) const;
   unsigned getMaxInterleaveFactor(unsigned VF);
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr);
   InstructionCost getVectorInstrCost(unsigned Opcode, Type *ValTy,
-                                     unsigned Index);
+                                     unsigned Index, const Instruction *I);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_AMDGPU_R600TARGETTRANSFORMINFO_H

llvm/lib/Target/AMDGPU/R600TargetTransformInfo.cpp

InstructionCost R600TTIImpl::getCFInstrCost(unsigned Opcode,
   case Instruction::Ret:
     return 10;
   default:
     return BaseT::getCFInstrCost(Opcode, CostKind, I);
   }
 }
 
 InstructionCost R600TTIImpl::getVectorInstrCost(unsigned Opcode, Type *ValTy,
-                                                unsigned Index) {
+                                                unsigned Index,
+                                                const Instruction *I) {
   switch (Opcode) {
   case Instruction::ExtractElement:
   case Instruction::InsertElement: {
     unsigned EltSize =
         DL.getTypeSizeInBits(cast<VectorType>(ValTy)->getElementType());
     if (EltSize < 32) {
-      return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
+      return BaseT::getVectorInstrCost(Opcode, ValTy, Index, I);
     }
 
     // Extracts are just reads of a subregister, so are free. Inserts are
     // considered free because we don't want to have any cost for scalarizing
     // operations, and we don't have to copy into a different register class.
 
     // Dynamic indexing isn't free and is best avoided.
     return Index == ~0u ? 2 : 0;
   }
   default:
-    return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
+    return BaseT::getVectorInstrCost(Opcode, ValTy, Index, I);
   }
 }
 
 void R600TTIImpl::getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                           TTI::UnrollingPreferences &UP,
                                           OptimizationRemarkEmitter *ORE) {
   CommonTTI.getUnrollingPreferences(L, SE, UP, ORE);
 }
 
 void R600TTIImpl::getPeelingPreferences(Loop *L, ScalarEvolution &SE,
                                         TTI::PeelingPreferences &PP) {
   CommonTTI.getPeelingPreferences(L, SE, PP);
 }

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    TTI::TargetCostKind CostKind,
                                    const Instruction *I = nullptr);
 
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I = nullptr);
 
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
 
   InstructionCost getAddressComputationCost(Type *Val, ScalarEvolution *SE,
                                             const SCEV *Ptr);
 
   InstructionCost getArithmeticInstrCost(
       unsigned Opcode, Type *Ty, TTI::TargetCostKind CostKind,
       TTI::OperandValueKind Op1Info = TTI::OK_AnyValue,
       TTI::OperandValueKind Op2Info = TTI::OK_AnyValue,

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

InstructionCost ARMTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
   int BaseCost = ST->hasMVEIntegerOps() && Src->isVectorTy()
                      ? ST->getMVEVectorCostFactor(CostKind)
                      : 1;
   return AdjustCost(
       BaseCost * BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I));
 }
 
 InstructionCost ARMTTIImpl::getVectorInstrCost(unsigned Opcode, Type *ValTy,
-                                               unsigned Index) {
+                                               unsigned Index,
+                                               const Instruction *I) {
   // Penalize inserting into an D-subregister. We end up with a three times
   // lower estimated throughput on swift.
   if (ST->hasSlowLoadDSubregister() && Opcode == Instruction::InsertElement &&
       ValTy->isVectorTy() && ValTy->getScalarSizeInBits() <= 32)
     return 3;
 
   if (ST->hasNEON() && (Opcode == Instruction::InsertElement ||
                         Opcode == Instruction::ExtractElement)) {
     // Cross-class copies are expensive on many microarchitectures,
     // so assume they are expensive by default.
     if (cast<VectorType>(ValTy)->getElementType()->isIntegerTy())
       return 3;
 
     // Even if it's not a cross class copy, this likely leads to mixing
     // of NEON and VFP code and should be therefore penalized.
     if (ValTy->isVectorTy() &&
         ValTy->getScalarSizeInBits() <= 32)
       return std::max<InstructionCost>(
-          BaseT::getVectorInstrCost(Opcode, ValTy, Index), 2U);
+          BaseT::getVectorInstrCost(Opcode, ValTy, Index, I), 2U);
   }
 
   if (ST->hasMVEIntegerOps() && (Opcode == Instruction::InsertElement ||
                                  Opcode == Instruction::ExtractElement)) {
     // Integer cross-lane moves are more expensive than float, which can
     // sometimes just be vmovs. Integer involve being passes to GPR registers,
     // causing more of a delay.
     std::pair<InstructionCost, MVT> LT =
         getTLI()->getTypeLegalizationCost(DL, ValTy->getScalarType());
     return LT.first * (ValTy->getScalarType()->isIntegerTy() ? 4 : 1);
   }
 
-  return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
+  return BaseT::getVectorInstrCost(Opcode, ValTy, Index, I);
 }
 
 InstructionCost ARMTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                                Type *CondTy,
                                                CmpInst::Predicate VecPred,
                                                TTI::TargetCostKind CostKind,
                                                const Instruction *I) {
   int ISD = TLI->InstructionOpcodeToISD(Opcode);

llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h

InstructionCost getArithmeticInstrCost(
       TTI::OperandValueProperties Opd1PropInfo = TTI::OP_None,
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),
       const Instruction *CxtI = nullptr);
   InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    TTI::CastContextHint CCH,
                                    TTI::TargetCostKind CostKind,
                                    const Instruction *I = nullptr);
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
 
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr) {
     return 1;
   }
 
   bool isLegalMaskedStore(Type *DataType, Align Alignment);
   bool isLegalMaskedLoad(Type *DataType, Align Alignment);

llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp

if (SrcTy->isFPOrFPVectorTy() || DstTy->isFPOrFPVectorTy()) {
     if (CostKind != TTI::TCK_RecipThroughput)
       return Cost == 0 ? 0 : 1;
     return Cost;
   }
   return 1;
 }
 
 InstructionCost HexagonTTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
-                                                   unsigned Index) {
+                                                   unsigned Index,
+                                                   const Instruction *I) {
   Type *ElemTy = Val->isVectorTy() ? cast<VectorType>(Val)->getElementType()
                                    : Val;
   if (Opcode == Instruction::InsertElement) {
     // Need two rotations for non-zero index.
     unsigned Cost = (Index != 0) ? 2 : 0;
     if (ElemTy->isIntegerTy(32))
       return Cost;
     // If it's not a 32-bit value, there will need to be an extract.
-    return Cost + getVectorInstrCost(Instruction::ExtractElement, Val, Index);
+    return Cost +
+           getVectorInstrCost(Instruction::ExtractElement, Val, Index, I);
   }
 
   if (Opcode == Instruction::ExtractElement)
     return 2;
 
   return 1;
 }
 

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    TTI::TargetCostKind CostKind,
                                    const Instruction *I = nullptr);
   InstructionCost getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind,
                                  const Instruction *I = nullptr);
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I = nullptr);
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,
                                   MaybeAlign Alignment, unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I = nullptr);
   InstructionCost getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
       Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
       bool UseMaskForCond = false, bool UseMaskForGaps = false);

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

InstructionCost Cost =
       BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, VecPred, CostKind, I);
   // TODO: Handle other cost kinds.
   if (CostKind != TTI::TCK_RecipThroughput)
     return Cost;
   return Cost * CostFactor;
 }
 
 InstructionCost PPCTTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
-                                               unsigned Index) {
+                                               unsigned Index,
+                                               const Instruction *I) {
   assert(Val->isVectorTy() && "This must be a vector type");
 
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   assert(ISD && "Invalid opcode");
 
   InstructionCost CostFactor = vectorCostAdjustmentFactor(Opcode, Val, nullptr);
   if (!CostFactor.isValid())
     return InstructionCost::getMax();
 
-  InstructionCost Cost = BaseT::getVectorInstrCost(Opcode, Val, Index);
+  InstructionCost Cost = BaseT::getVectorInstrCost(Opcode, Val, Index, I);
   Cost *= CostFactor;
 
   if (ST->hasVSX() && Val->getScalarType()->isDoubleTy()) {
     // Double-precision scalars are already located in index #0 (or #1 if LE).
     if (ISD == ISD::EXTRACT_VECTOR_ELT &&
         Index == (ST->isLittleEndian() ? 1 : 0))
       return 0;
 
InstructionCost PPCTTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
   return Cost;
 }
 
 InstructionCost PPCTTIImpl::getMemoryOpCost(unsigned Opcode, Type *Src,
                                             MaybeAlign Alignment,
                                             unsigned AddressSpace,
                                             TTI::TargetCostKind CostKind,
                                             const Instruction *I) {
-
   InstructionCost CostFactor = vectorCostAdjustmentFactor(Opcode, Src, nullptr);
   if (!CostFactor.isValid())
     return InstructionCost::getMax();
 
   if (TLI->getValueType(DL, Src,  true) == MVT::Other)
     return BaseT::getMemoryOpCost(Opcode, Src, Alignment, AddressSpace,
                                   CostKind);
   // Legalize the type.
InstructionCost PPCTTIImpl::getMemoryOpCost(unsigned Opcode, Type *Src,
   Cost += LT.first * ((SrcBytes / Alignment->value()) - 1);
 
   // For a vector type, there is also scalarization overhead (only for
   // stores, loads are expanded using the vector-load + permutation sequence,
   // which is much less expensive).
   if (Src->isVectorTy() && Opcode == Instruction::Store)
     for (int i = 0, e = cast<FixedVectorType>(Src)->getNumElements(); i < e;
          ++i)
-      Cost += getVectorInstrCost(Instruction::ExtractElement, Src, i);
+      Cost += getVectorInstrCost(Instruction::ExtractElement, Src, i, I);
 
   return Cost;
 }
 
 InstructionCost PPCTTIImpl::getInterleavedMemoryOpCost(
     unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
     Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
     bool UseMaskForCond, bool UseMaskForGaps) {

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h

public:
   InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    TTI::CastContextHint CCH,
                                    TTI::TargetCostKind CostKind,
                                    const Instruction *I = nullptr);
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I = nullptr);
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
   bool isFoldableLoad(const LoadInst *Ld, const Instruction *&FoldedValue);
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,
                                   MaybeAlign Alignment, unsigned AddressSpace,
                                   TTI::TargetCostKind CostKind,
                                   const Instruction *I = nullptr);
 
   InstructionCost getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp

else { // Called with a select instruction.
       return getNumVectorRegs(ValTy) /*vsel*/ + PackCost;
     }
   }
 
   return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, VecPred, CostKind);
 }
 
 InstructionCost SystemZTTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
-                                                   unsigned Index) {
+                                                   unsigned Index,
+                                                   const Instruction *I) {
   // vlvgp will insert two grs into a vector register, so only count half the
   // number of instructions.
   if (Opcode == Instruction::InsertElement && Val->isIntOrIntVectorTy(64))
     return ((Index % 2 == 0) ? 1 : 0);
 
   if (Opcode == Instruction::ExtractElement) {
     int Cost = ((getScalarSizeInBits(Val) == 1) ? 2 /*+test-under-mask*/ : 1);
 
     // Give a slight penalty for moving out of vector pipeline to FXU unit.
     if (Index == 0 && Val->isIntOrIntVectorTy())
       Cost += 1;
 
     return Cost;
   }
 
-  return BaseT::getVectorInstrCost(Opcode, Val, Index);
+  return BaseT::getVectorInstrCost(Opcode, Val, Index, I);
 }
 
 // Check if a load may be folded as a memory operand in its user.
 bool SystemZTTIImpl::
 isFoldableLoad(const LoadInst *Ld, const Instruction *&FoldedValue) {
   if (!Ld->hasOneUse())
     return false;
   FoldedValue = Ld;

llvm/lib/Target/WebAssembly/WebAssemblyTargetTransformInfo.h

public:
   InstructionCost getArithmeticInstrCost(
       unsigned Opcode, Type *Ty, TTI::TargetCostKind CostKind,
       TTI::OperandValueKind Opd1Info = TTI::OK_AnyValue,
       TTI::OperandValueKind Opd2Info = TTI::OK_AnyValue,
       TTI::OperandValueProperties Opd1PropInfo = TTI::OP_None,
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),
       const Instruction *CxtI = nullptr);
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
 
   /// @}
 
   bool areInlineCompatible(const Function *Caller,
                            const Function *Callee) const;
 
   bool supportsTailCalls() const;
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/WebAssembly/WebAssemblyTargetTransformInfo.cpp

case Instruction::Shl:
       break;
     }
   }
   return Cost;
 }
 
 InstructionCost WebAssemblyTTIImpl::getVectorInstrCost(unsigned Opcode,
                                                        Type *Val,
-                                                       unsigned Index) {
+                                                       unsigned Index,
+                                                       const Instruction *I) {
   InstructionCost Cost =
-      BasicTTIImplBase::getVectorInstrCost(Opcode, Val, Index);
+      BasicTTIImplBase::getVectorInstrCost(Opcode, Val, Index, I);
 
   // SIMD128's insert/extract currently only take constant indices.
   if (Index == -1u)
     return Cost + 25 * TargetTransformInfo::TCC_Expensive;
 
   return Cost;
 }
 

llvm/lib/Target/X86/X86TargetTransformInfo.h

public:
   InstructionCost getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                    TTI::CastContextHint CCH,
                                    TTI::TargetCostKind CostKind,
                                    const Instruction *I = nullptr);
   InstructionCost getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                      CmpInst::Predicate VecPred,
                                      TTI::TargetCostKind CostKind,
                                      const Instruction *I = nullptr);
-  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val,
-                                     unsigned Index);
+  InstructionCost getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index,
+                                     const Instruction *I);
   InstructionCost getScalarizationOverhead(VectorType *Ty,
                                            const APInt &DemandedElts,
                                            bool Insert, bool Extract);
   InstructionCost getReplicationShuffleCost(Type *EltTy, int ReplicationFactor,
                                             int VF,
                                             const APInt &DemandedDstElts,
                                             TTI::TargetCostKind CostKind);
   InstructionCost getMemoryOpCost(unsigned Opcode, Type *Src,

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

if (ISD != ISD::DELETED_NODE) {
     if (const auto *Entry = CostTableLookup(X86CostTbl, ISD, MTy))
       return LT.first * Entry->Cost;
   }
 
   return BaseT::getIntrinsicInstrCost(ICA, CostKind);
 }
 
 InstructionCost X86TTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
-                                               unsigned Index) {
+                                               unsigned Index,
+                                               const Instruction *I) {
   static const CostTblEntry SLMCostTbl[] = {
      { ISD::EXTRACT_VECTOR_ELT,       MVT::i8,      4 },
      { ISD::EXTRACT_VECTOR_ELT,       MVT::i16,     4 },
      { ISD::EXTRACT_VECTOR_ELT,       MVT::i32,     4 },
      { ISD::EXTRACT_VECTOR_ELT,       MVT::i64,     7 }
    };
 
   assert(Val->isVectorTy() && "This must be a vector type");
if (Index != -1U && (Opcode == Instruction::ExtractElement ||
     return ShuffleCost + IntOrFpCost + RegisterFileMoveCost;
   }
 
   // Add to the base cost if we know that the extracted element of a vector is
   // destined to be moved to and used in the integer register file.
   if (Opcode == Instruction::ExtractElement && ScalarType->isPointerTy())
     RegisterFileMoveCost += 1;
 
-  return BaseT::getVectorInstrCost(Opcode, Val, Index) + RegisterFileMoveCost;
+  return BaseT::getVectorInstrCost(Opcode, Val, Index, I) +
+         RegisterFileMoveCost;
 }
 
 InstructionCost X86TTIImpl::getScalarizationOverhead(VectorType *Ty,
                                                      const APInt &DemandedElts,
                                                      bool Insert,
                                                      bool Extract) {
   assert(DemandedElts.getBitWidth() ==
              cast<FixedVectorType>(Ty)->getNumElements() &&
if (LT.second.isVector()) {
             Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, None,
                                    I * Scale, Ty128);
 
         // Add all the demanded element extractions together, but adjust the
         // index to use the equivalent of the bottom 128 bit lane.
         for (unsigned I = 0; I != NumElts; ++I)
           if (WidenedDemandedElts[I]) {
             unsigned Idx = I % Scale;
-            Cost += getVectorInstrCost(Instruction::ExtractElement, Ty, Idx);
+            Cost += getVectorInstrCost(Instruction::ExtractElement, Ty, Idx,
+                                       nullptr);
           }
 
         return Cost;
       }
     }
 
     // Fallback to default extraction.
     Cost += BaseT::getScalarizationOverhead(Ty, DemandedElts, false, Extract);
if (Size > 128) {
           TargetTransformInfo::OP_None, TargetTransformInfo::OP_None);
     }
 
     // Add the arithmetic op for this level.
     ReductionCost += getArithmeticInstrCost(Opcode, Ty, CostKind);
   }
 
   // Add the final extract element to the cost.
-  return ReductionCost + getVectorInstrCost(Instruction::ExtractElement, Ty, 0);
+  return ReductionCost +
+         getVectorInstrCost(Instruction::ExtractElement, Ty, 0, nullptr);
 }
 
 InstructionCost X86TTIImpl::getMinMaxCost(Type *Ty, Type *CondTy,
                                           bool IsUnsigned) {
   std::pair<InstructionCost, MVT> LT = TLI->getTypeLegalizationCost(DL, Ty);
 
   MVT MTy = LT.second;
 
while (NumVecElts > 1) {
 
     // Add the arithmetic op for this level.
     auto *SubCondTy =
         FixedVectorType::get(CondTy->getElementType(), Ty->getNumElements());
     MinMaxCost += getMinMaxCost(Ty, SubCondTy, IsUnsigned);
   }
 
   // Add the final extract element to the cost.
-  return MinMaxCost + getVectorInstrCost(Instruction::ExtractElement, Ty, 0);
+  return MinMaxCost +
+         getVectorInstrCost(Instruction::ExtractElement, Ty, 0, nullptr);
 }
 
 /// Calculate the cost of materializing a 64-bit value. This helper
 /// method might only calculate a fraction of a larger immediate. Therefore it
 /// is valid to return a cost of ZERO.
 InstructionCost X86TTIImpl::getIntImmCost(int64_t Val) {
   if (Val == 0)
     return TTI::TCC_Free;

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

LoopVectorizationCostModel::getUniformMemOpCost(Instruction *I,
   if (isa<LoadInst>(I)) {
     return TTI.getAddressComputationCost(ValTy) +
            TTI.getMemoryOpCost(Instruction::Load, ValTy, Alignment, AS,
                                CostKind) +
            TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy);
   }
   StoreInst *SI = cast<StoreInst>(I);
 
+  // FIXME: Use a specific type rather than unknown.
   bool isLoopInvariantStoreValue = Legal->isUniform(SI->getValueOperand());
   return TTI.getAddressComputationCost(ValTy) +
          TTI.getMemoryOpCost(Instruction::Store, ValTy, Alignment, AS,
                              CostKind) +
          (isLoopInvariantStoreValue
               ? 0
               : TTI.getVectorInstrCost(Instruction::ExtractElement, VectorTy,
-                                       VF.getKnownMinValue() - 1));
+                                       VF.getKnownMinValue() - 1, nullptr));
 }
 
 InstructionCost
 LoopVectorizationCostModel::getGatherScatterCost(Instruction *I,
                                                  ElementCount VF) {
   Type *ValTy = getLoadStoreType(I);
   auto *VectorTy = cast<VectorType>(ToVectorTy(ValTy, VF));
   const Align Alignment = getLoadStoreAlignment(I);

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

for (auto *V : VL) {
           // Add back the cost of s|zext which is subtracted separately.
           Cost += TTIRef.getCastInstrCost(
               Ext->getOpcode(), Ext->getType(), EE->getType(),
               TTI::getCastContextHint(Ext), CostKind, Ext);
           continue;
         }
       }
       Cost -= TTIRef.getVectorInstrCost(Instruction::ExtractElement,
-                                        EE->getVectorOperandType(), Idx);
+                                        EE->getVectorOperandType(), Idx, EE);
     }
     // Add a cost for subvector extracts/inserts if required.
     for (const auto &Data : ExtractVectorsTys) {
       auto *EEVTy = cast<FixedVectorType>(Data.first->getType());
       unsigned NumElts = VecTy->getNumElements();
       if (Data.second % NumElts == 0)
         continue;
       if (TTIRef.getNumberOfParts(EEVTy) > TTIRef.getNumberOfParts(VecTy)) {
case Instruction::ExtractElement: {
       // the cost of shuffles, if required to resuffle the original vector.
       if (NeedToShuffleReuses) {
         unsigned Idx = 0;
         for (unsigned I : E->ReuseShuffleIndices) {
           if (ShuffleOrOp == Instruction::ExtractElement) {
             auto *EE = cast<ExtractElementInst>(VL[I]);
             CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
                                                   EE->getVectorOperandType(),
-                                                  *getExtractIndex(EE));
+                                                  *getExtractIndex(EE), EE);
           } else {
-            CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
+            CommonCost -=
+                TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, Idx,
+                                        dyn_cast<Instruction>(VL[I]));
             ++Idx;
           }
         }
         Idx = EntryVF;
         for (Value *V : VL) {
           if (ShuffleOrOp == Instruction::ExtractElement) {
             auto *EE = cast<ExtractElementInst>(V);
             CommonCost += TTI->getVectorInstrCost(Instruction::ExtractElement,
                                                   EE->getVectorOperandType(),
-                                                  *getExtractIndex(EE));
+                                                  *getExtractIndex(EE), EE);
           } else {
             --Idx;
-            CommonCost += TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
+            CommonCost +=
+                TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, Idx,
+                                        dyn_cast<Instruction>(V));
           }
         }
       }
       if (ShuffleOrOp == Instruction::ExtractValue) {
         for (unsigned I = 0, E = VL.size(); I < E; ++I) {
           auto *EI = cast<Instruction>(VL[I]);
           // Take credit for instruction that will become dead.
           if (EI->hasOneUse()) {
             Instruction *Ext = EI->user_back();
             if ((isa<SExtInst>(Ext) || isa<ZExtInst>(Ext)) &&
                 all_of(Ext->users(),
                        [](User *U) { return isa<GetElementPtrInst>(U); })) {
               // Use getExtractWithExtendCost() to calculate the cost of
               // extractelement/ext pair.
               CommonCost -= TTI->getExtractWithExtendCost(
                   Ext->getOpcode(), Ext->getType(), VecTy, I);
               // Add back the cost of s|zext which is subtracted separately.
               CommonCost += TTI->getCastInstrCost(
                   Ext->getOpcode(), Ext->getType(), EI->getType(),
                   TTI::getCastContextHint(Ext), CostKind, Ext);
               continue;
             }
           }
-          CommonCost -=
-              TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, I);
+          CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
+                                                VecTy, I, EI);
         }
       } else {
         AdjustExtractsCost(CommonCost);
       }
       return CommonCost;
     }
     case Instruction::InsertElement: {
       assert(E->ReuseShuffleIndices.empty() &&
for (ExternalUser &EU : ExternalUses) {
     if (MinBWs.count(ScalarRoot)) {
       auto *MinTy = IntegerType::get(F->getContext(), MinBWs[ScalarRoot].first);
       auto Extend =
           MinBWs[ScalarRoot].second ? Instruction::SExt : Instruction::ZExt;
       VecTy = FixedVectorType::get(MinTy, BundleWidth);
       ExtractCost += TTI->getExtractWithExtendCost(Extend, EU.Scalar->getType(),
                                                    VecTy, EU.Lane);
     } else {
-      ExtractCost +=
-          TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, EU.Lane);
+      ExtractCost += TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy,
+                                             EU.Lane, nullptr);
     }
   }
 
   InstructionCost SpillCost = getSpillCost();
   Cost += SpillCost + ExtractCost;
   auto &&ResizeToVF = [this, &Cost](const TreeEntry *TE, ArrayRef<int> Mask) {
     InstructionCost C = 0;
     unsigned VF = Mask.size();

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

ExtractElementInst *VectorCombine::getShuffleExtract(
 
   // If the extract indexes are identical, no shuffle is needed.
   if (Index0 == Index1)
     return nullptr;
 
   Type *VecTy = Ext0->getVectorOperand()->getType();
   assert(VecTy == Ext1->getVectorOperand()->getType() && "Need matching types");
   InstructionCost Cost0 =
-      TTI.getVectorInstrCost(Ext0->getOpcode(), VecTy, Index0);
+      TTI.getVectorInstrCost(Ext0->getOpcode(), VecTy, Index0, Ext0);
   InstructionCost Cost1 =
-      TTI.getVectorInstrCost(Ext1->getOpcode(), VecTy, Index1);
+      TTI.getVectorInstrCost(Ext1->getOpcode(), VecTy, Index1, Ext1);
 
   // If both costs are invalid no shuffle is needed
   if (!Cost0.isValid() && !Cost1.isValid())
     return nullptr;
 
   // We are extracting from 2 different indexes, so one operand must be shuffled
   // before performing a vector operation and/or extract. The more expensive
   // extract will be replaced by a shuffle.
VectorOpCost = TTI.getCmpSelInstrCost(
         Opcode, VecTy, CmpInst::makeCmpResultType(VecTy), Pred);
   }
 
   // Get cost estimates for the extract elements. These costs will factor into
   // both sequences.
   unsigned Ext0Index = Ext0IndexC->getZExtValue();
   unsigned Ext1Index = Ext1IndexC->getZExtValue();
 
-  InstructionCost Extract0Cost =
-      TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy, Ext0Index);
-  InstructionCost Extract1Cost =
-      TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy, Ext1Index);
+  InstructionCost Extract0Cost = TTI.getVectorInstrCost(
+      Instruction::ExtractElement, VecTy, Ext0Index, Ext0);
+  InstructionCost Extract1Cost = TTI.getVectorInstrCost(
+      Instruction::ExtractElement, VecTy, Ext1Index, Ext1);
 
   // A more expensive extract will always be replaced by a splat shuffle.
   // For example, if Ext0 is more expensive:
   // opcode (extelt V0, Ext0), (ext V1, Ext1) -->
   // extelt (opcode (splat V0, Ext0), V1), Ext1
   // TODO: Evaluate whether that always results in lowest cost. Alternatively,
   //       check the cost of creating a broadcast shuffle and shuffling both
   //       operands to element 0.
bool VectorCombine::scalarizeBinopOrCmp(Instruction &I) {
   } else {
     ScalarOpCost = TTI.getArithmeticInstrCost(Opcode, ScalarTy);
     VectorOpCost = TTI.getArithmeticInstrCost(Opcode, VecTy);
   }
 
   // Get cost estimate for the insert element. This cost will factor into
   // both sequences.
   InstructionCost InsertCost =
-      TTI.getVectorInstrCost(Instruction::InsertElement, VecTy, Index);
+      TTI.getVectorInstrCost(Instruction::InsertElement, VecTy, Index, &I);
   InstructionCost OldCost =
       (IsConst0 ? 0 : InsertCost) + (IsConst1 ? 0 : InsertCost) + VectorOpCost;
   InstructionCost NewCost = ScalarOpCost + InsertCost +
                             (IsConst0 ? 0 : !Ins0->hasOneUse() * InsertCost) +
                             (IsConst1 ? 0 : !Ins1->hasOneUse() * InsertCost);
 
   // We want to scalarize unless the vector variant actually has lower cost.
   if (OldCost < NewCost || !NewCost.isValid())
bool VectorCombine::foldExtractedCmps(Instruction &I) {
   CmpInst::Predicate Pred = P0;
   unsigned CmpOpcode = CmpInst::isFPPredicate(Pred) ? Instruction::FCmp
                                                     : Instruction::ICmp;
   auto *VecTy = dyn_cast<FixedVectorType>(X->getType());
   if (!VecTy)
     return false;
 
   InstructionCost OldCost =
-      TTI.getVectorInstrCost(Ext0->getOpcode(), VecTy, Index0);
-  OldCost += TTI.getVectorInstrCost(Ext1->getOpcode(), VecTy, Index1);
+      TTI.getVectorInstrCost(Ext0->getOpcode(), VecTy, Index0, Ext0);
+  OldCost += TTI.getVectorInstrCost(Ext1->getOpcode(), VecTy, Index1, Ext1);
   OldCost +=
       TTI.getCmpSelInstrCost(CmpOpcode, I0->getType(),
                              CmpInst::makeCmpResultType(I0->getType()), Pred) *
       2;
   OldCost += TTI.getArithmeticInstrCost(I.getOpcode(), I.getType());
 
   // The proposed vector pattern is:
   // vcmp = cmp Pred X, VecC
   // ext (binop vNi1 vcmp, (shuffle vcmp, Index1)), Index0
   int CheapIndex = ConvertToShuf == Ext0 ? Index1 : Index0;
   int ExpensiveIndex = ConvertToShuf == Ext0 ? Index0 : Index1;
   auto *CmpTy = cast<FixedVectorType>(CmpInst::makeCmpResultType(X->getType()));
   InstructionCost NewCost = TTI.getCmpSelInstrCost(
       CmpOpcode, X->getType(), CmpInst::makeCmpResultType(X->getType()), Pred);
   SmallVector<int, 32> ShufMask(VecTy->getNumElements(), UndefMaskElem);
   ShufMask[CheapIndex] = ExpensiveIndex;
   NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, CmpTy,
                                 ShufMask);
   NewCost += TTI.getArithmeticInstrCost(I.getOpcode(), CmpTy);
-  NewCost += TTI.getVectorInstrCost(Ext0->getOpcode(), CmpTy, CheapIndex);
+  NewCost += TTI.getVectorInstrCost(Ext0->getOpcode(), CmpTy, CheapIndex, Ext0);
 
   // Aggressively form vector ops if the cost is equal because the transform
   // may enable further optimization.
   // Codegen can reverse this transform (scalarize) if it was not profitable.
   if (OldCost < NewCost || !NewCost.isValid())
     return false;
 
   // Create a vector constant from the 2 scalar constants.
if (!ScalarIdx.isSafe()) {
       // TODO: Freeze index if it is safe to do so.
       ScalarIdx.discard();
       return false;
     }
 
     auto *Index = dyn_cast<ConstantInt>(UI->getOperand(1));
     OriginalCost +=
         TTI.getVectorInstrCost(Instruction::ExtractElement, LI->getType(),
-                               Index ? Index->getZExtValue() : -1);
+                               Index ? Index->getZExtValue() : -1, LI);
     ScalarizedCost +=
         TTI.getMemoryOpCost(Instruction::Load, FixedVT->getElementType(),
                             Align(1), LI->getPointerAddressSpace());
     ScalarizedCost += TTI.getAddressComputationCost(FixedVT->getElementType());
   }
 
   if (ScalarizedCost >= OriginalCost)
     return false;

llvm/test/Analysis/CostModel/AArch64/kryo.ll

 ; RUN: opt < %s -passes='print<cost-model>' 2>&1 -disable-output -mcpu=kryo | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64--linux-gnu"
 
 ; CHECK-LABEL: vectorInstrCost
 define void @vectorInstrCost() {
 
     ; Vector extracts - extracting the first element should have a zero cost;
     ; all other elements should have a cost of two.
     ;
     ; CHECK: cost of 0 {{.*}} extractelement <2 x i64> undef, i32 0
     ; CHECK: cost of 2 {{.*}} extractelement <2 x i64> undef, i32 1
     %t1 = extractelement <2 x i64> undef, i32 0
     %t2 = extractelement <2 x i64> undef, i32 1
 
     ; Vector inserts - inserting the first element should have a zero cost; all
     ; other elements should have a cost of two.
     ;
     ; CHECK: cost of 0 {{.*}} insertelement <2 x i64> undef, i64 undef, i32 0
     ; CHECK: cost of 2 {{.*}} insertelement <2 x i64> undef, i64 undef, i32 1
     %t3 = insertelement <2 x i64> undef, i64 undef, i32 0
     %t4 = insertelement <2 x i64> undef, i64 undef, i32 1
-
     ret void
 }
 
 ; CHECK-LABEL: vectorInstrExtractCost
 define i64 @vectorInstrExtractCost(<4 x i64> %vecreg) {
-    
-    ; Vector extracts - extracting each element at index 0 is considered
-    ; free in the current implementation. When extracting element at index
-    ; 2, 2 is rounded to 0, so extracting element at index 2 has cost 0 as 
-    ; well.
-    ;
     ; CHECK: cost of 2 {{.*}} extractelement <4 x i64> %vecreg, i32 1
-    ; CHECK: cost of 0 {{.*}} extractelement <4 x i64> %vecreg, i32 2
+    ; CHECK: cost of 2 {{.*}} extractelement <4 x i64> %vecreg, i32 2
     %t1 = extractelement <4 x i64> %vecreg, i32 1
     %t2 = extractelement <4 x i64> %vecreg, i32 2
     %ele = add i64 %t2, 1
     %cond = icmp eq i64 %t1, %ele
 
-    ; CHECK: cost of 0 {{.*}} extractelement <4 x i64> %vecreg, i32 0
+    ; Vector extracts - extracting each element should have a cost
+    ; if they are used as integers.
+    ;
+    ; CHECK: cost of 2 {{.*}} extractelement <4 x i64> %vecreg, i32 0
     ; CHECK: cost of 2 {{.*}} extractelement <4 x i64> %vecreg, i32 3
     %t0 = extractelement <4 x i64> %vecreg, i32 0
     %t3 = extractelement <4 x i64> %vecreg, i32 3
     %val = select i1 %cond, i64 %t0 , i64 %t3
 
     ret i64 %val
 }

llvm/test/Transforms/LICM/AArch64/extract-element.ll

 ; CHECK-NEXT:    [[TMP8:%.*]] = load <1 x i64>, ptr [[TMP7]], align 8
 ; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <1 x i64> [[TMP8]], i64 0
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[TMP9]], -1
 ; CHECK-NEXT:    br i1 [[TMP10]], label [[TMP11]], label [[DOTSPLIT_LOOP_EXIT:%.*]]
 ; CHECK:       11:
 ; CHECK-NEXT:    [[TMP12]] = add i64 [[TMP4]], 1
 ; CHECK-NEXT:    br label [[TMP3]]
 ; CHECK:       .split.loop.exit:
-; CHECK-NEXT:    [[DOTLCSSA7:%.*]] = phi <1 x i64> [ [[TMP8]], [[TMP6]] ]
+; CHECK-NEXT:    [[DOTLCSSA:%.*]] = phi i64 [ [[TMP9]], [[TMP6]] ]
 ; CHECK-NEXT:    [[DOTLCSSA6:%.*]] = phi i64 [ [[TMP4]], [[TMP6]] ]
 ; CHECK-NEXT:    [[DOTPH:%.*]] = phi i1 [ [[TMP5]], [[TMP6]] ]
-; CHECK-NEXT:    [[TMP13:%.*]] = extractelement <1 x i64> [[DOTLCSSA7]], i64 0
-; CHECK-NEXT:    [[TMP14:%.*]] = xor i64 [[TMP13]], -1
-; CHECK-NEXT:    [[TMP15:%.*]] = add i64 [[TMP14]], [[DOTLCSSA6]]
-; CHECK-NEXT:    [[TMP16:%.*]] = icmp uge i64 [[TMP15]], [[TMP1]]
-; CHECK-NEXT:    br label [[TMP17:%.*]]
+; CHECK-NEXT:    [[TMP13:%.*]] = xor i64 [[DOTLCSSA]], -1
+; CHECK-NEXT:    [[TMP14:%.*]] = add i64 [[TMP13]], [[DOTLCSSA6]]
+; CHECK-NEXT:    [[TMP15:%.*]] = icmp uge i64 [[TMP14]], [[TMP1]]
+; CHECK-NEXT:    br label [[TMP16:%.*]]
 ; CHECK:       .split.loop.exit2:
 ; CHECK-NEXT:    [[DOTPH3:%.*]] = phi i1 [ [[TMP5]], [[TMP3]] ]
 ; CHECK-NEXT:    [[DOTPH4:%.*]] = phi i1 [ undef, [[TMP3]] ]
-; CHECK-NEXT:    br label [[TMP17]]
-; CHECK:       17:
-; CHECK-NEXT:    [[TMP18:%.*]] = phi i1 [ [[DOTPH]], [[DOTSPLIT_LOOP_EXIT]] ], [ [[DOTPH3]], [[DOTSPLIT_LOOP_EXIT2]] ]
-; CHECK-NEXT:    [[TMP19:%.*]] = phi i1 [ [[TMP16]], [[DOTSPLIT_LOOP_EXIT]] ], [ [[DOTPH4]], [[DOTSPLIT_LOOP_EXIT2]] ]
-; CHECK-NEXT:    [[TMP20:%.*]] = xor i1 [[TMP18]], true
-; CHECK-NEXT:    [[TMP21:%.*]] = select i1 [[TMP20]], i1 true, i1 [[TMP19]]
-; CHECK-NEXT:    ret i1 [[TMP21]]
+; CHECK-NEXT:    br label [[TMP16]]
+; CHECK:       16:
+; CHECK-NEXT:    [[TMP17:%.*]] = phi i1 [ [[DOTPH]], [[DOTSPLIT_LOOP_EXIT]] ], [ [[DOTPH3]], [[DOTSPLIT_LOOP_EXIT2]] ]
+; CHECK-NEXT:    [[TMP18:%.*]] = phi i1 [ [[TMP15]], [[DOTSPLIT_LOOP_EXIT]] ], [ [[DOTPH4]], [[DOTSPLIT_LOOP_EXIT2]] ]
+; CHECK-NEXT:    [[TMP19:%.*]] = xor i1 [[TMP17]], true
+; CHECK-NEXT:    [[TMP20:%.*]] = select i1 [[TMP19]], i1 true, i1 [[TMP18]]
+; CHECK-NEXT:    ret i1 [[TMP20]]
 ;
   br label %3
 
 3:                                                ; preds = %14, %2
   %4 = phi i64 [ 0, %2 ], [ %15, %14 ]
   %5 = icmp ult i64 %4, %1
   br i1 %5, label %6, label %16