[VectorCombine] Scalarize vector load/extract.

[fhahn]

Summary

This patch adds a new combine that tries to scalarize chains of
extractelement (load %ptr), %idx to load (gep %ptr, %idx). This is
profitable when extracting only a few elements out of a large vector.

At the moment, , store (extractelement (load %ptr), %idx), %ptr
operations on large vectors result in huge code in the backend.

This can easily be triggered by using the matrix extension, e.g.
https://clang.godbolt.org/z/qsccPdPf4

This should complement D98240.

[lebedev.ri]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

790–810

Since we would be extracting from the original value,
why do we care if it would have been overwritten sometimes later?
Can't we just emit all loads right where the wide load was?

812

Do we expect that all the CSE have happened by now?

[fhahn]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

790–810

Can't we just emit all loads right where the wide load was?

We could, *if* the indices are all defined before the wide load. That's not the case in the cases I am most interested in.

At the moment the scalar loads are inserted at the point of the extracts, but if the index is defined before the wide load, we could create the narrow loads next to the widened loads. But that may be good as a follow up extensions?

812

Most of it yes, but not for the code that the SLP vectorizer creates. We don't run dedicated CSE later either, just instcombine.

[fhahn]

Updated to match vector loads, so it is easier to re-trigger after store scalarization from D98240.

[RKSimon]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

901

Why are you forcing the alignment ?

[fhahn]

Use commonAlignment between alignment and alignment for scalar type.

[fhahn]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

901

I think we have to update the alignment of the scalar load, because after applying an offset to the pointer it may not be aligned as specified on the original load. I think we should be able to use the common alignment between the alignment on the load and the scalar type. WDYT?

[fhahn]

rebased after recent changes, restrict to constant indexes known to be valid, for now.

[RKSimon]

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

645

The arg says its align 1m, but here we're saying align 8?

[fhahn]

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

645

Hm that's indeed odd. I *think* the align of the load would take precedence and align 1 dereferenceable(16) are probably not needed for the test. But I think we need to choose a conservative alignment for the scalarized load, that's why align 1 is used for the scalar load.

Does that answer your question?

[fhahn]

Ping :)

[RKSimon]

Anyone else got any comments?

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

876

Pull out the magic number (MaxInstChecked = 6 ?)

901

Yes, using a common alignment would make more sense

[spatel]

llvm/test/Transforms/VectorCombine/AArch64/load-extractelement-scalarization.ll

400

Let me know if I'm overlooking it, but I don't see a positive test for this pattern (multiple extracts of a single load).
Is there some vector type variant (float?) of this test that will transform for AArch64?
If not, we might get x86 or some other target to trigger on this test as-is.

[fhahn]

Updated code to use common alignment of original alignment and the scalar type alignment.

Also rebased on top of 4efb4f674cb6, which adds a new positive test where multiple extracts of the same load are scalarized

[fhahn]

Use existing MaxInstrsToScan instead of hard-coding the limit to 6.

[fhahn]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

876

I updated the code to use MaxInstrsToScan which was added by an earlier patch.

901

I updated the code to use the common alignment.

llvm/test/Transforms/VectorCombine/AArch64/load-extractelement-scalarization.ll

400

Yes I think this case was missing. I added load_multiple_extracts_with_constant_idx_profitable to cover that I think.

[spatel]

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

645

Some of these tests are intentionally bizarre -- blame me :) -- with respect to alignment because the rules are not clear.
Looking at the code in vectorizeLoadInsert(), we have:

// Use the greater of the alignment on the load or its source pointer.
Alignment = std::max(SrcPtr->getPointerAlignment(DL), Alignment);

So on the AVX2 target, this test is going through 2 transforms now instead of just 1:

1. This patch kicks in 1st and converts the short vector load to scalar load with align 2 -- presumably because that's the datalayout-ABI-specified setting. Should the original load's align 8 attribute override that?
2. The scalar load is converted to the wider vector load again in vectorizeLoadInsert(), but now the alignment was reduced, so we end up with the test diff.

[fhahn]

Use alignment from original store when scalarizing a load with index 0.

[fhahn]

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

645

Thanks for the explanation. I think once we lowered to alignment on the load, it is not really safe to raise it based on the alignment of the pointer type. But if we scalarize with index 0, we load from the original pointer and we can preserve the original alignment. I updated the patch and the AVX2 changes are now gone.

[spatel]

LGTM

[spatel]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

116–119

I didn't check the bot failure, but we dealt with a previous sanitizer failure with an additional predicate here.

[fhahn]

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

116–119

I think the issues was that scalarizeLoadExtract was called after foldSingleElementStore, which may remove instructions. Should be fixed by 4e8c28b6fbec

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

void foldExtExtCmp(ExtractElementInst *Ext0, ExtractElementInst *Ext1,
                      Instruction &I);
   void foldExtExtBinop(ExtractElementInst *Ext0, ExtractElementInst *Ext1,
                        Instruction &I);
   bool foldExtractExtract(Instruction &I);
   bool foldBitcastShuf(Instruction &I);
   bool scalarizeBinopOrCmp(Instruction &I);
   bool foldExtractedCmps(Instruction &I);
   bool foldSingleElementStore(Instruction &I);
+  bool scalarizeLoadExtract(Instruction &I);
 };
 } // namespace
 
 static void replaceValue(Value &Old, Value &New) {
   Old.replaceAllUsesWith(&New);
   New.takeName(&Old);
 }
 
 bool VectorCombine::vectorizeLoadInsert(Instruction &I) {
   // Match insert into fixed vector of scalar value.
   // TODO: Handle non-zero insert index.
   auto *Ty = dyn_cast<FixedVectorType>(I.getType());
   Value *Scalar;
   if (!Ty || !match(&I, m_InsertElt(m_Undef(), m_Value(Scalar), m_ZeroInt())) ||
       !Scalar->hasOneUse())
     return false;
 
   // Optionally match an extract from another vector.
   Value *X;
   bool HasExtract = match(Scalar, m_ExtractElt(m_Value(X), m_ZeroInt()));
   if (!HasExtract)
     X = Scalar;
 
   // Match source value as load of scalar or vector.
   // Do not vectorize scalar load (widening) if atomic/volatile or under
   // asan/hwasan/memtag/tsan. The widened load may load data from dirty regions
   // or create data races non-existent in the source.
   auto *Load = dyn_cast<LoadInst>(X);
   if (!Load || !Load->isSimple() || !Load->hasOneUse() ||
       Load->getFunction()->hasFnAttribute(Attribute::SanitizeMemTag) ||
       mustSuppressSpeculation(*Load))
     return false;
 
   const DataLayout &DL = I.getModule()->getDataLayout();
   Value *SrcPtr = Load->getPointerOperand()->stripPointerCasts();
static bool isMemModifiedBetween(BasicBlock::iterator Begin,
                                  const MemoryLocation &Loc, AAResults &AA) {
   unsigned NumScanned = 0;
   return std::any_of(Begin, End, [&](const Instruction &Instr) {
     return isModSet(AA.getModRefInfo(&Instr, Loc)) ||
            ++NumScanned > MaxInstrsToScan;
   });
 }
 
+/// Check if it is legal to scalarize a memory access to \p VecTy at index \p
+/// Idx. \p Idx must access a valid vector element.
+static bool canScalarizeAccess(FixedVectorType *VecTy, ConstantInt *Idx) {
+  return Idx->getValue().ult(VecTy->getNumElements());
+}
+
 // Combine patterns like:
 //   %0 = load <4 x i32>, <4 x i32>* %a
 //   %1 = insertelement <4 x i32> %0, i32 %b, i32 1
 //   store <4 x i32> %1, <4 x i32>* %a
 // to:
 //   %0 = bitcast <4 x i32>* %a to i32*
 //   %1 = getelementptr inbounds i32, i32* %0, i64 0, i64 1
 //   store i32 %b, i32* %1
 bool VectorCombine::foldSingleElementStore(Instruction &I) {
   StoreInst *SI = dyn_cast<StoreInst>(&I);
   if (!SI || !SI->isSimple() ||
       !isa<FixedVectorType>(SI->getValueOperand()->getType()))
     return false;
 
   // TODO: Combine more complicated patterns (multiple insert) by referencing
   // TargetTransformInfo.
   Instruction *Source;
   Value *NewElement;
   ConstantInt *Idx;
   if (!match(SI->getValueOperand(),
              m_InsertElt(m_Instruction(Source), m_Value(NewElement),
                          m_ConstantInt(Idx))))
     return false;
 
   if (auto *Load = dyn_cast<LoadInst>(Source)) {
     auto VecTy = cast<FixedVectorType>(SI->getValueOperand()->getType());
     const DataLayout &DL = I.getModule()->getDataLayout();
     Value *SrcAddr = Load->getPointerOperand()->stripPointerCasts();
     // Don't optimize for atomic/volatile load or store. Ensure memory is not
     // modified between, vector type matches store size, and index is inbounds.
     if (!Load->isSimple() || Load->getParent() != SI->getParent() ||
         !DL.typeSizeEqualsStoreSize(Load->getType()) ||
-        Idx->uge(VecTy->getNumElements()) ||
+        !canScalarizeAccess(VecTy, Idx) ||
         SrcAddr != SI->getPointerOperand()->stripPointerCasts() ||
         isMemModifiedBetween(Load->getIterator(), SI->getIterator(),
                              MemoryLocation::get(SI), AA))
       return false;
 
     Value *GEP = GetElementPtrInst::CreateInBounds(
         SI->getPointerOperand(), {ConstantInt::get(Idx->getType(), 0), Idx});
     Builder.Insert(GEP);
     StoreInst *NSI = Builder.CreateStore(NewElement, GEP);
     NSI->copyMetadata(*SI);
     if (SI->getAlign() < NSI->getAlign())
       NSI->setAlignment(SI->getAlign());
     replaceValue(I, *NSI);
     // Need erasing the store manually.
     I.eraseFromParent();
     return true;
   }
 
   return false;
 }
 
+/// Try to scalarize vector loads feeding extractelement instructions.
+bool VectorCombine::scalarizeLoadExtract(Instruction &I) {
+  Value *Ptr;
+  ConstantInt *Idx;
+  if (!match(&I, m_ExtractElt(m_Load(m_Value(Ptr)), m_ConstantInt(Idx))))
+    return false;
+
+  auto *LI = cast<LoadInst>(I.getOperand(0));
+  const DataLayout &DL = I.getModule()->getDataLayout();
+  if (LI->isVolatile() || !DL.typeSizeEqualsStoreSize(LI->getType()))
+    return false;
+
+  auto *FixedVT = dyn_cast<FixedVectorType>(LI->getType());
+  if (!FixedVT)
+    return false;
+
+  if (!canScalarizeAccess(FixedVT, Idx))
+    return false;
+
+  InstructionCost OriginalCost = TTI.getMemoryOpCost(
+      Instruction::Load, LI->getType(), Align(LI->getAlignment()),
+      LI->getPointerAddressSpace());
+  InstructionCost ScalarizedCost = 0;
+
+  Instruction *LastCheckedInst = LI;
+  unsigned NumInstChecked = 0;
+  // Check if all users of the load are extracts with no memory modifications
+  // between the load and the extract. Compute the cost of both the original
+  // code and the scalarized version.
+  for (User *U : LI->users()) {
+    auto *UI = dyn_cast<ExtractElementInst>(U);
+    if (!UI || UI->getParent() != LI->getParent())
+      return false;
+
+    // Check if any instruction between the load and the extract may modify
+    // memory.
+    if (LastCheckedInst->comesBefore(UI)) {
+      for (Instruction &I :
+           make_range(std::next(LI->getIterator()), UI->getIterator())) {
+        // Bail out if we reached the check limit or the instruction may write
+        // to memory.
+        if (NumInstChecked == MaxInstrsToScan || I.mayWriteToMemory())
+          return false;
+        NumInstChecked++;
+      }
+    }
+
+    if (!LastCheckedInst)
+      LastCheckedInst = UI;
+    else if (LastCheckedInst->comesBefore(UI))
+      LastCheckedInst = UI;
+
+    auto *Index = dyn_cast<ConstantInt>(UI->getOperand(1));
+    OriginalCost +=
+        TTI.getVectorInstrCost(Instruction::ExtractElement, LI->getType(),
+                               Index ? Index->getZExtValue() : -1);
+    ScalarizedCost +=
+        TTI.getMemoryOpCost(Instruction::Load, FixedVT->getElementType(),
+                            Align(1), LI->getPointerAddressSpace());
+    ScalarizedCost += TTI.getAddressComputationCost(FixedVT->getElementType());
+  }
+
+  if (ScalarizedCost >= OriginalCost)
+    return false;
+
+  // Replace extracts with narrow scalar loads.
+  for (User *U : LI->users()) {
+    auto *EI = cast<ExtractElementInst>(U);
+    IRBuilder<>::InsertPointGuard Guard(Builder);
+    Builder.SetInsertPoint(EI);
+    Value *GEP = Builder.CreateInBoundsGEP(
+        FixedVT, Ptr, {Builder.getInt32(0), EI->getOperand(1)});
+    auto *NewLoad = cast<LoadInst>(Builder.CreateLoad(
+        FixedVT->getElementType(), GEP, EI->getName() + ".scalar"));
+
+    // Set the alignment for the new load. For index 0, we can use the original
+    // alignment. Otherwise choose the common alignment of the load's align and
+    // the alignment for the scalar type.
+    auto *ConstIdx = dyn_cast<ConstantInt>(EI->getOperand(1));
+    if (ConstIdx && ConstIdx->isNullValue())
+      NewLoad->setAlignment(LI->getAlign());
+    else
+      NewLoad->setAlignment(commonAlignment(
+          DL.getABITypeAlign(NewLoad->getType()), LI->getAlign()));
+    replaceValue(*EI, *NewLoad);
+  }
+
+  return true;
+}
+
 /// This is the entry point for all transforms. Pass manager differences are
 /// handled in the callers of this function.
 bool VectorCombine::run() {
   if (DisableVectorCombine)
     return false;
 
   // Don't attempt vectorization if the target does not support vectors.
   if (!TTI.getNumberOfRegisters(TTI.getRegisterClassForType(/*Vector*/ true)))
for (Instruction &I : make_early_inc_range(BB)) {
         continue;
       Builder.SetInsertPoint(&I);
       MadeChange |= vectorizeLoadInsert(I);
       MadeChange |= foldExtractExtract(I);
       MadeChange |= foldBitcastShuf(I);
       MadeChange |= scalarizeBinopOrCmp(I);
       MadeChange |= foldExtractedCmps(I);
       MadeChange |= foldSingleElementStore(I);
+      MadeChange |= scalarizeLoadExtract(I);
     }
   }
 
   // We're done with transforms, so remove dead instructions.
   if (MadeChange)
     for (BasicBlock &BB : F)
       SimplifyInstructionsInBlock(&BB);
 

llvm/test/Transforms/VectorCombine/AArch64/load-extractelement-scalarization.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -vector-combine -mtriple=arm64-apple-darwinos -S %s | FileCheck %s
+; RUN: opt -vector-combine -mtriple=arm64-apple-darwinos -S %s | FileCheck --check-prefixes=CHECK,LIMIT-DEFAULT %s
+; RUN: opt -vector-combine -mtriple=arm64-apple-darwinos -vector-combine-max-scan-instrs=2 -S %s | FileCheck --check-prefixes=CHECK,LIMIT2 %s
 
 define i32 @load_extract_idx_0(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_idx_0(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 3
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 3
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 3
   ret i32 %r
 }
 
+; If the original load had a smaller alignment than the scalar type, the
+; smaller alignment should be used.
+define i32 @load_extract_idx_0_small_alignment(<4 x i32>* %x) {
+; CHECK-LABEL: @load_extract_idx_0_small_alignment(
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 3
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 2
+; CHECK-NEXT:    ret i32 [[R]]
+;
+  %lv = load <4 x i32>, <4 x i32>* %x, align 2
+  %r = extractelement <4 x i32> %lv, i32 3
+  ret i32 %r
+}
+
 define i32 @load_extract_idx_1(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_idx_1(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 1
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 1
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 1
   ret i32 %r
 }
 
 define i32 @load_extract_idx_2(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_idx_2(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 2
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 2
   ret i32 %r
 }
 
 define i32 @load_extract_idx_3(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_idx_3(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 3
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 3
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 3
   ret i32 %r
 }
 
 ; Out-of-bounds index for extractelement, should not be converted to narrow
;
   ret i32 %r
 }
 
 declare void @clobber()
 
 define i32 @load_extract_clobber_call_before(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_clobber_call_before(
 ; CHECK-NEXT:    call void @clobber()
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 2
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   call void @clobber()
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 2
   ret i32 %r
 }
 
 define i32 @load_extract_clobber_call_between(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_clobber_call_between(
 ; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
 ; CHECK-NEXT:    call void @clobber()
 ; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   call void @clobber()
   %r = extractelement <4 x i32> %lv, i32 2
   ret i32 %r
 }
 
 define i32 @load_extract_clobber_call_after(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_extract_clobber_call_after(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 2
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    call void @clobber()
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 2
   call void @clobber()
   ret i32 %r
 }
 
 define i32 @load_extract_clobber_store_before(<4 x i32>* %x, i8* %y) {
 ; CHECK-LABEL: @load_extract_clobber_store_before(
 ; CHECK-NEXT:    store i8 0, i8* [[Y:%.*]], align 1
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 2
+; CHECK-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
   store i8 0, i8* %y
   %lv = load <4 x i32>, <4 x i32>* %x
   %r = extractelement <4 x i32> %lv, i32 2
   ret i32 %r
 }
 
;
   %add.3 = add i32 %add.2, %z.4
   store i8 0, i8* %y
   %r = extractelement <4 x i32> %lv, i32 2
   %add.4 = add i32 %add.3, %r
   ret i32 %add.4
 }
 
 define i32 @load_extract_clobber_store_after_limit(<4 x i32>* %x, i8* %y, <8 x i32> %z) {
-; CHECK-LABEL: @load_extract_clobber_store_after_limit(
-; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
-; CHECK-NEXT:    [[Z_0:%.*]] = extractelement <8 x i32> [[Z:%.*]], i32 0
-; CHECK-NEXT:    [[Z_1:%.*]] = extractelement <8 x i32> [[Z]], i32 1
-; CHECK-NEXT:    [[ADD_0:%.*]] = add i32 [[Z_0]], [[Z_1]]
-; CHECK-NEXT:    [[Z_2:%.*]] = extractelement <8 x i32> [[Z]], i32 2
-; CHECK-NEXT:    [[ADD_1:%.*]] = add i32 [[ADD_0]], [[Z_2]]
-; CHECK-NEXT:    [[Z_3:%.*]] = extractelement <8 x i32> [[Z]], i32 3
-; CHECK-NEXT:    [[ADD_2:%.*]] = add i32 [[ADD_1]], [[Z_3]]
-; CHECK-NEXT:    [[Z_4:%.*]] = extractelement <8 x i32> [[Z]], i32 4
-; CHECK-NEXT:    [[ADD_3:%.*]] = add i32 [[ADD_2]], [[Z_4]]
-; CHECK-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
-; CHECK-NEXT:    store i8 0, i8* [[Y:%.*]], align 1
-; CHECK-NEXT:    [[ADD_4:%.*]] = add i32 [[ADD_3]], [[R]]
-; CHECK-NEXT:    ret i32 [[ADD_4]]
+; LIMIT-DEFAULT-LABEL: @load_extract_clobber_store_after_limit(
+; LIMIT-DEFAULT-NEXT:    [[Z_0:%.*]] = extractelement <8 x i32> [[Z:%.*]], i32 0
+; LIMIT-DEFAULT-NEXT:    [[Z_1:%.*]] = extractelement <8 x i32> [[Z]], i32 1
+; LIMIT-DEFAULT-NEXT:    [[ADD_0:%.*]] = add i32 [[Z_0]], [[Z_1]]
+; LIMIT-DEFAULT-NEXT:    [[Z_2:%.*]] = extractelement <8 x i32> [[Z]], i32 2
+; LIMIT-DEFAULT-NEXT:    [[ADD_1:%.*]] = add i32 [[ADD_0]], [[Z_2]]
+; LIMIT-DEFAULT-NEXT:    [[Z_3:%.*]] = extractelement <8 x i32> [[Z]], i32 3
+; LIMIT-DEFAULT-NEXT:    [[ADD_2:%.*]] = add i32 [[ADD_1]], [[Z_3]]
+; LIMIT-DEFAULT-NEXT:    [[Z_4:%.*]] = extractelement <8 x i32> [[Z]], i32 4
+; LIMIT-DEFAULT-NEXT:    [[ADD_3:%.*]] = add i32 [[ADD_2]], [[Z_4]]
+; LIMIT-DEFAULT-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <4 x i32>, <4 x i32>* [[X:%.*]], i32 0, i32 2
+; LIMIT-DEFAULT-NEXT:    [[R:%.*]] = load i32, i32* [[TMP1]], align 4
+; LIMIT-DEFAULT-NEXT:    store i8 0, i8* [[Y:%.*]], align 1
+; LIMIT-DEFAULT-NEXT:    [[ADD_4:%.*]] = add i32 [[ADD_3]], [[R]]
+; LIMIT-DEFAULT-NEXT:    ret i32 [[ADD_4]]
+;
+; LIMIT2-LABEL: @load_extract_clobber_store_after_limit(
+; LIMIT2-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
+; LIMIT2-NEXT:    [[Z_0:%.*]] = extractelement <8 x i32> [[Z:%.*]], i32 0
+; LIMIT2-NEXT:    [[Z_1:%.*]] = extractelement <8 x i32> [[Z]], i32 1
+; LIMIT2-NEXT:    [[ADD_0:%.*]] = add i32 [[Z_0]], [[Z_1]]
+; LIMIT2-NEXT:    [[Z_2:%.*]] = extractelement <8 x i32> [[Z]], i32 2
+; LIMIT2-NEXT:    [[ADD_1:%.*]] = add i32 [[ADD_0]], [[Z_2]]
+; LIMIT2-NEXT:    [[Z_3:%.*]] = extractelement <8 x i32> [[Z]], i32 3
+; LIMIT2-NEXT:    [[ADD_2:%.*]] = add i32 [[ADD_1]], [[Z_3]]
+; LIMIT2-NEXT:    [[Z_4:%.*]] = extractelement <8 x i32> [[Z]], i32 4
+; LIMIT2-NEXT:    [[ADD_3:%.*]] = add i32 [[ADD_2]], [[Z_4]]
+; LIMIT2-NEXT:    [[R:%.*]] = extractelement <4 x i32> [[LV]], i32 2
+; LIMIT2-NEXT:    store i8 0, i8* [[Y:%.*]], align 1
+; LIMIT2-NEXT:    [[ADD_4:%.*]] = add i32 [[ADD_3]], [[R]]
+; LIMIT2-NEXT:    ret i32 [[ADD_4]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %z.0 = extractelement <8 x i32> %z, i32 0
   %z.1 = extractelement <8 x i32> %z, i32 1
   %add.0 = add i32 %z.0, %z.1
   %z.2 = extractelement <8 x i32> %z, i32 2
   %add.1 = add i32 %add.0, %z.2
   %z.3 = extractelement <8 x i32> %z, i32 3

 ; CHECK-NEXT:    ret i31 [[R]]
 ;
   %lv = load <4 x i31>, <4 x i31>* %x
   %r = extractelement <4 x i31> %lv, i32 1
   ret i31 %r
 }
 
 ; Scalarizing the load for multiple constant indices may not be profitable.
 define i32 @load_multiple_extracts_with_constant_idx(<4 x i32>* %x) {
 ; CHECK-LABEL: @load_multiple_extracts_with_constant_idx(
 ; CHECK-NEXT:    [[LV:%.*]] = load <4 x i32>, <4 x i32>* [[X:%.*]], align 16
 ; CHECK-NEXT:    [[SHIFT:%.*]] = shufflevector <4 x i32> [[LV]], <4 x i32> poison, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[TMP1:%.*]] = add <4 x i32> [[LV]], [[SHIFT]]
 ; CHECK-NEXT:    [[RES:%.*]] = extractelement <4 x i32> [[TMP1]], i32 0
 ; CHECK-NEXT:    ret i32 [[RES]]
 ;
   %lv = load <4 x i32>, <4 x i32>* %x
   %e.0 = extractelement <4 x i32> %lv, i32 0
   %e.1 = extractelement <4 x i32> %lv, i32 1
   %res = add i32 %e.0, %e.1
   ret i32 %res
 }
 
 ; Scalarizing the load for multiple extracts is profitable in this case,
 ; because the vector large vector requires 2 vector registers.
 define i32 @load_multiple_extracts_with_constant_idx_profitable(<8 x i32>* %x) {
 ; CHECK-LABEL: @load_multiple_extracts_with_constant_idx_profitable(
-; CHECK-NEXT:    [[LV:%.*]] = load <8 x i32>, <8 x i32>* [[X:%.*]], align 32
-; CHECK-NEXT:    [[E_0:%.*]] = extractelement <8 x i32> [[LV]], i32 0
-; CHECK-NEXT:    [[E_1:%.*]] = extractelement <8 x i32> [[LV]], i32 6
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <8 x i32>, <8 x i32>* [[X:%.*]], i32 0, i32 0
+; CHECK-NEXT:    [[E_0:%.*]] = load i32, i32* [[TMP1]], align 16
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds <8 x i32>, <8 x i32>* [[X]], i32 0, i32 6
+; CHECK-NEXT:    [[E_1:%.*]] = load i32, i32* [[TMP2]], align 4
 ; CHECK-NEXT:    [[RES:%.*]] = add i32 [[E_0]], [[E_1]]
 ; CHECK-NEXT:    ret i32 [[RES]]
 ;
-  %lv = load <8 x i32>, <8 x i32>* %x
+  %lv = load <8 x i32>, <8 x i32>* %x, align 16
   %e.0 = extractelement <8 x i32> %lv, i32 0
   %e.1 = extractelement <8 x i32> %lv, i32 6
   %res = add i32 %e.0, %e.1
   ret i32 %res
 }
 
 ; Scalarizing may or may not be profitable, depending on the target.
 define i32 @load_multiple_2_with_variable_indices(<4 x i32>* %x, i64 %idx.0, i64 %idx.1) {

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

;
   ret <8 x i32> %r
 }
 
 ; Can't safely load the offset vector, but can load+shuffle if it is profitable.
 
 define <8 x i16> @gep1_load_v2i16_extract_insert_v8i16(<2 x i16>* align 1 dereferenceable(16) %p) {
 ; SSE2-LABEL: @gep1_load_v2i16_extract_insert_v8i16(
 ; SSE2-NEXT:    [[GEP:%.*]] = getelementptr inbounds <2 x i16>, <2 x i16>* [[P:%.*]], i64 1
-; SSE2-NEXT:    [[L:%.*]] = load <2 x i16>, <2 x i16>* [[GEP]], align 8
-; SSE2-NEXT:    [[S:%.*]] = extractelement <2 x i16> [[L]], i32 0
+; SSE2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <2 x i16>, <2 x i16>* [[GEP]], i32 0, i32 0
+; SSE2-NEXT:    [[S:%.*]] = load i16, i16* [[TMP1]], align 8
 ; SSE2-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[S]], i64 0
 ; SSE2-NEXT:    ret <8 x i16> [[R]]
 ;
 ; AVX2-LABEL: @gep1_load_v2i16_extract_insert_v8i16(
 ; AVX2-NEXT:    [[TMP1:%.*]] = bitcast <2 x i16>* [[P:%.*]] to <8 x i16>*
 ; AVX2-NEXT:    [[TMP2:%.*]] = load <8 x i16>, <8 x i16>* [[TMP1]], align 4
 ; AVX2-NEXT:    [[R:%.*]] = shufflevector <8 x i16> [[TMP2]], <8 x i16> poison, <8 x i32> <i32 2, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
 ; AVX2-NEXT:    ret <8 x i16> [[R]]
 ;
   %gep = getelementptr inbounds <2 x i16>, <2 x i16>* %p, i64 1
   %l = load <2 x i16>, <2 x i16>* %gep, align 8
   %s = extractelement <2 x i16> %l, i32 0
   %r = insertelement <8 x i16> poison, i16 %s, i64 0
   ret <8 x i16> %r
 }

llvm/test/Transforms/VectorCombine/X86/load.ll

;
   ret <8 x i32> %r
 }
 
 ; Can't safely load the offset vector, but can load+shuffle if it is profitable.
 
 define <8 x i16> @gep1_load_v2i16_extract_insert_v8i16(<2 x i16>* align 1 dereferenceable(16) %p) {
 ; SSE2-LABEL: @gep1_load_v2i16_extract_insert_v8i16(
 ; SSE2-NEXT:    [[GEP:%.*]] = getelementptr inbounds <2 x i16>, <2 x i16>* [[P:%.*]], i64 1
-; SSE2-NEXT:    [[L:%.*]] = load <2 x i16>, <2 x i16>* [[GEP]], align 8
-; SSE2-NEXT:    [[S:%.*]] = extractelement <2 x i16> [[L]], i32 0
+; SSE2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds <2 x i16>, <2 x i16>* [[GEP]], i32 0, i32 0
+; SSE2-NEXT:    [[S:%.*]] = load i16, i16* [[TMP1]], align 8
 ; SSE2-NEXT:    [[R:%.*]] = insertelement <8 x i16> undef, i16 [[S]], i64 0
 ; SSE2-NEXT:    ret <8 x i16> [[R]]
 ;
 ; AVX2-LABEL: @gep1_load_v2i16_extract_insert_v8i16(
 ; AVX2-NEXT:    [[TMP1:%.*]] = bitcast <2 x i16>* [[P:%.*]] to <8 x i16>*
 ; AVX2-NEXT:    [[TMP2:%.*]] = load <8 x i16>, <8 x i16>* [[TMP1]], align 4
 ; AVX2-NEXT:    [[R:%.*]] = shufflevector <8 x i16> [[TMP2]], <8 x i16> poison, <8 x i32> <i32 2, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
 ; AVX2-NEXT:    ret <8 x i16> [[R]]
 ;
   %gep = getelementptr inbounds <2 x i16>, <2 x i16>* %p, i64 1
   %l = load <2 x i16>, <2 x i16>* %gep, align 8
   %s = extractelement <2 x i16> %l, i32 0
   %r = insertelement <8 x i16> undef, i16 %s, i64 0
   ret <8 x i16> %r
 }