[SLPVectorizer] Fix PR21780 Expansion of 256 bit vector loads fails to fold into shuffles

[dtemirbulatov]

Summary

This is the second part of PR21780 fix, here is part one https://reviews.llvm.org/D37579

This change allows to restore Load, Insert, GEP instructions removed by InstCombine pass. Here I am using intrinsic to keep information in the IR as it was discussed in http://lists.llvm.org/pipermail/llvm-dev/2017-July/115730.html.

Before restoring instructions we have have to check existing instructions for belonging to our basic block, also we check the shuffle mask and all offset, ...

[hfinkel]

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

[RKSimon]

Some more style comment as with D37579

lib/Transforms/Vectorize/SLPVectorizer.cpp

4225

clang-format - braces

5754

It'd be much clearer to early out.

if (!C || C->getZExtValue() > MaxOffset)
  return nullptr;

// We found the first InsertElem in this chain.
uint64_t Offset = C->getZExtValue();
...

5807

for (unsigned i = 0, e = MaxOffset + 1; i < e; i++)

5818

if (!Use || Use->getParent() != BB)

5842

for (unsigned i = 0, e = MaxOffset + 1; i < e; i++)

5852

You're accessing GEPs[i - 1] multiple times - pull out?

[dtemirbulatov]

In D37648#865562, @hfinkel wrote:

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

Here is the thread: http://lists.llvm.org/pipermail/llvm-dev/2017-September/117381.html

[dtemirbulatov]

Fix review remarks and add an aggregate pointer as intrinsic's second parameter, remove getNextNode() usage.

[hfinkel]

In D37648#879283, @dtemirbulatov wrote:

In D37648#865562, @hfinkel wrote:

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

Here is the thread: http://lists.llvm.org/pipermail/llvm-dev/2017-September/117381.html

Yes. Are you planing on updating that thread? Last message I see says:

but I am looking now at "speculation_marker" metadata and I am still not sure how to implement it better.

are you still looking at some other kind of metadata solution? Did you mean for "metadata" here to mean the intrinsic you're proposing?

[dtemirbulatov]

are you still looking at some other kind of metadata solution? Did you mean for "metadata" here to mean the intrinsic you're proposing?

No, I am happy with intrinsic solution.

[reames]

test/Transforms/SLPVectorizer/X86/pr21780.ll

23

I think something might be missing here. You're forming a 4x wide load, but you've only proven dereferenceability for offsets 0, 1, 3. (i.e. not 2). How do we know it's safe to dereference between the two elements 1 & 3?

[dtemirbulatov]

I think something might be missing here. You're forming a 4x wide load, but you've only proven dereferenceability for offsets 0, 1, 3. (i.e. not 2). How do we know it's safe to dereference between the two elements 1 & 3?

well, it is a maximum reference that was recorded there in InstCombine, that implies 0, 1, 2 and 3. And if you are saying that 0, 1, 3 dereferenceable and 2-nd is not at the same time then this solution is not correct.

[dtemirbulatov]

Update for solution, here it uses metadata "speculation.marker" with set of offsets instead of maximum offset as before, thread for discussion http://lists.llvm.org/pipermail/llvm-dev/2017-September/117782.html.

[hfinkel]

In D37648#906216, @dtemirbulatov wrote:

Update for solution, here it uses metadata "speculation.marker" with set of offsets instead of maximum offset as before, thread for discussion http://lists.llvm.org/pipermail/llvm-dev/2017-September/117782.html.

Can you please update the thread with your new plan for this metadata (what adds it, when, and what consumes it), including the proposed definition for the LangRef, so we can all see exactly what you're proposing?

[dtemirbulatov]

Add LangRef.rst change. Small changes in SLPVectorizer.cpp.

[RKSimon]

docs/LangRef.rst

4922

"consists"

4923

Is it a set or a min/max range? The examples only ever have 2 entries. If a set do they have to be sorted?

4925

"is to keep track of dereferanceable memory locations after load operations to that memory were deleted, as it might be beneficial for future optimizations."

include/llvm/IR/LLVMContext.h

104

// alignment

include/llvm/Transforms/Vectorize/SLPVectorizer.h

154

Unnecessary whitespace change

[dtemirbulatov]

Rebase and fix remarks.

[dtemirbulatov]

Update description in LangRef.rst.

[RKSimon]

@dtemirbulatov What is the status of this patch please?

[dtemirbulatov]

In D37648#1148425, @RKSimon wrote:

@dtemirbulatov What is the status of this patch please?

@RKSimon I think it needs review. I will rebase proposed change.

[RKSimon]

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

[dtemirbulatov]

In D37648#1653860, @RKSimon wrote:

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

no, I think I can rebase the change.

[lebedev.ri]

In D37648#1654027, @dtemirbulatov wrote:

In D37648#1653860, @RKSimon wrote:

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

no, I think I can rebase the change.

I suspect @RKSimon is might be right here, it may be better to redesign the feature with nowadays best practices in mind (attributor).

[RKSimon]

Adding @jdoerfert who can probably advise regarding whether Attributor will handle all of this

[jdoerfert]

With D65402 (and later D65593), the Attributor will determine dereferenceability based on accesses that "must be executed".
Since the Attributor runs early in the pipeline (for now only then), this will "generally" solve the issue as unused loads are still present.
However, we lack the ability to annotate dereferenceable for pointers that are not arguments, function return values, call site arguments, or call site return values.
That means, we might determine dereferenceability early but fail to manifest it in the IR or loose the information during inlining. Though, I am unsure how much of
a problem that really is. If it turns out to be bad, I'd suggest to add an intrinsic to describe dereferenceability (among other things [alignment, max object size, ...]) of a pointer.

Long story short, I'd recommend investing the time in reviewing and testing the Attributor infrastructure and potentially thinking about a pointer property intrinsic.

If I misunderstood what is going on here, please let me know.

docs/LangRef.rst

 
 .. code-block:: llvm
 
     %result = call i64 %binop(i64 %x, i64 %y), !callees !0
 
     ...
     !0 = !{i64 (i64, i64)* @add, i64 (i64, i64)* @sub}
 
+'``speculation.marker``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^
+
+``speculation.marker`` metadata must be attached to a load. It consists of
+set of ``i64`` type offsets indicating that the memory from load pointer
+address is accessable for read operation with provided offsets. The intent
+of this metadata is to keep track of dereferanceable memory locations after
+load operations to that memory were deleted, as it might be beneficial
+for future optimizations. Offsets are sorted and they aren't indicating min/max
+offsets, but rather each offset proven to be dereferanceable.
+
+.. code-block:: llvm
+
+    %ld1 = load double, double* %arrayidx1, align 8, !speculation.marker !0
+
+    ...
+    !0 = !{i64 -1, i64 2}
+
 '``unpredictable``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ``unpredictable`` metadata may be attached to any branch or switch
 instruction. It can be used to express the unpredictability of control
 flow. Similar to the llvm.expect intrinsic, it may be used to alter
 optimizations related to compare and branch instructions. The metadata
 is treated as a boolean value; if it exists, it signals that the branch

include/llvm/IR/LLVMContext.h

enum {
     MD_invariant_group = 16,          // "invariant.group"
     MD_align = 17,                    // "align"
     MD_loop = 18,                     // "llvm.loop"
     MD_type = 19,                     // "type"
     MD_section_prefix = 20,           // "section_prefix"
     MD_absolute_symbol = 21,          // "absolute_symbol"
     MD_associated = 22,               // "associated"
     MD_callees = 23,                  // "callees"
     MD_irr_loop = 24,                 // "irr_loop"
+    MD_speculation_marker = 25,       // "speculation.marker"
   };
 
   /// Known operand bundle tag IDs, which always have the same value.  All
   /// operand bundle tags that LLVM has special knowledge of are listed here.
   /// Additionally, this scheme allows LLVM to efficiently check for specific
   /// operand bundle tags without comparing strings.
   enum {
     OB_deopt = 0,         // "deopt"

include/llvm/Transforms/Vectorize/SLPVectorizer.h

private:
 
   /// \brief Vectorize the store instructions collected in Stores.
   bool vectorizeStoreChains(slpvectorizer::BoUpSLP &R);
 
   /// \brief Vectorize the index computations of the getelementptr instructions
   /// collected in GEPs.
   bool vectorizeGEPIndices(BasicBlock *BB, slpvectorizer::BoUpSLP &R);
 
+  /// \brief Restore inserts out of speculation.marker metadata.
+  InsertElementInst * restoreInserts(LoadInst *LInstr);
+
   /// Try to find horizontal reduction or otherwise vectorize a chain of binary
   /// operators.
   bool vectorizeRootInstruction(PHINode *P, Value *V, BasicBlock *BB,
                                 slpvectorizer::BoUpSLP &R,
                                 TargetTransformInfo *TTI);
 
   /// Try to vectorize trees that start at insertvalue instructions.
   bool vectorizeInsertValueInst(InsertValueInst *IVI, BasicBlock *BB,
private:
 
   bool vectorizeStores(ArrayRef<StoreInst *> Stores, slpvectorizer::BoUpSLP &R);
 
   /// The store instructions in a basic block organized by base pointer.
   StoreListMap Stores;
 
   /// The getelementptr instructions in a basic block organized by base pointer.
   WeakTrackingVHListMap GEPs;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_VECTORIZE_SLPVECTORIZER_H

lib/IR/LLVMContext.cpp

std::pair<unsigned, StringRef> MDKinds[] = {
     {MD_align, "align"},
     {MD_loop, "llvm.loop"},
     {MD_type, "type"},
     {MD_section_prefix, "section_prefix"},
     {MD_absolute_symbol, "absolute_symbol"},
     {MD_associated, "associated"},
     {MD_callees, "callees"},
     {MD_irr_loop, "irr_loop"},
+    {MD_speculation_marker, "speculation.marker"},
   };
 
   for (auto &MDKind : MDKinds) {
     unsigned ID = getMDKindID(MDKind.second);
     assert(ID == MDKind.first && "metadata kind id drifted");
     (void)ID;
   }
 

lib/Transforms/Vectorize/SLPVectorizer.cpp

for (auto BB : post_order(&F.getEntryBlock())) {
     // Vectorize trees that end at stores.
     if (!Stores.empty()) {
       DEBUG(dbgs() << "SLP: Found stores for " << Stores.size()
                    << " underlying objects.\n");
       Changed |= vectorizeStoreChains(R);
     }
 
     // Vectorize trees that end at reductions.
     Changed |= vectorizeChainsInBlock(BB, R);
 
     // Vectorize the index computations of getelementptr instructions. This
     // is primarily intended to catch gather-like idioms ending at
     // non-consecutive loads.
     if (!GEPs.empty()) {
       DEBUG(dbgs() << "SLP: Found GEPs for " << GEPs.size()
                    << " underlying objects.\n");
       Changed |= vectorizeGEPIndices(BB, R);
bool SLPVectorizerPass::vectorizeInsertValueInst(InsertValueInst *IVI,
   SmallVector<Value *, 16> BuildVectorOpds;
   if (!findBuildAggregate(IVI, BuildVector, BuildVectorOpds))
     return false;
 
   DEBUG(dbgs() << "SLP: array mappable to vector: " << *IVI << "\n");
   return tryToVectorizeList(BuildVectorOpds, R, BuildVector, false);
 }
 
+InsertElementInst *SLPVectorizerPass::restoreInserts(LoadInst *LInstr) {
+  ShuffleVectorInst *Use = nullptr;
+  Value *Ptr = nullptr;
+  SmallDenseSet<uint64_t> Offsets;
+  int64_t OffsetOfLoad;
+  DenseMap<uint64_t, LoadInst *> Loads;
+  DenseMap<uint64_t, InsertElementInst *> Inserts;
+  DenseMap<uint64_t, GetElementPtrInst *> GEPs;
+  BasicBlock *BB = LInstr->getParent();
+  VectorType *VecType = nullptr;
+  unsigned MaxOffset = 0;
+
+  if (Instruction *Instr = dyn_cast<Instruction>(LInstr->getOperand(0))) {
+    GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Instr);
+    if (!GEP)
+      return nullptr;
+    ConstantInt *C = dyn_cast<ConstantInt>(GEP->getOperand(1));
+    if (!C)
+      return nullptr;
+    OffsetOfLoad = C->getSExtValue();
+    Ptr = GEP->getOperand(0);
+    GEPs[OffsetOfLoad] = GEP;
+    Loads[OffsetOfLoad] = LInstr;
+  } else {
+    OffsetOfLoad = 0;
+    Ptr = LInstr->getOperand(0);
+    Loads[0] = LInstr;
+  }
+
+  MDNode *MD = LInstr->getMetadata(LLVMContext::MD_speculation_marker);
+  assert(MD != nullptr && "Load should contain speculation.marker metadata");
+  for (int i = 0, e = MD->getNumOperands(); i < e; i++) {
+    ConstantInt *C = mdconst::dyn_extract<ConstantInt>(MD->getOperand(i));
+    Offsets.insert(OffsetOfLoad + C->getSExtValue());
+  }
+
+  for (BasicBlock::iterator it = BB->begin(), e = BB->end(); it != e; it++) {
+    if (InsertElementInst *Insert = dyn_cast<InsertElementInst>(it)) {
+      ConstantInt *C = dyn_cast<ConstantInt>(Insert->getOperand(2));
+      LoadInst *Load = dyn_cast<LoadInst>(Insert->getOperand(1));
+      if (!C || !Load)
+        continue;
+      GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Load->getOperand(0));
+      Value *Base = (GEP == nullptr) ? Load->getOperand(0) : GEP->getOperand(0);
+      if (Base == Ptr) {
+        if (!VecType)
+          VecType = Insert->getType();
+        else if (Insert->getType() != VecType)
+          return nullptr;
+        uint64_t Offset = C->getZExtValue();
+        Inserts[Offset] = Insert;
+        Loads[Offset] = Load;
+        // GEP instruction might be missing for 0 offset.
+        if (GEP)
+          GEPs[Offset] = GEP;
+        if (!Insert->use_empty() && !Use)
+          if (ShuffleVectorInst *Shuffle =
+                  dyn_cast<ShuffleVectorInst>(Insert->user_back()))
+            Use = Shuffle;
+        continue;
+      }
+    }
+    if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(it))
+      // Examining the chain of GEP, Load, Insert.
+      if (Ptr == GEP->getOperand(0)) {
+        ConstantInt *C = dyn_cast<ConstantInt>(GEP->getOperand(1));
+        if (!C || GEP->use_empty() || !GEP->hasOneUse())
+          return nullptr;
+        uint64_t Off = C->getZExtValue();
+        if (GEPs[Off] == GEP)
+          continue;
+        LoadInst *Load = dyn_cast<LoadInst>(GEP->user_back());
+        if (!Load)
+          return nullptr;
+        if (Load->getMetadata(LLVMContext::MD_speculation_marker) == nullptr) {
+          InsertElementInst *Insert =
+              dyn_cast<InsertElementInst>(Load->user_back());
+          if (!Insert)
+            return nullptr;
+          if (!VecType)
+            VecType = Insert->getType();
+          else if (Insert->getType() != VecType)
+            return nullptr;
+          ConstantInt *C1 = cast<ConstantInt>(Insert->getOperand(2));
+          uint64_t Off_Ins = C1->getZExtValue();
+          // Offset from GEP must be equal to offset from
+          // Insert otherwise we have to ignore.
+          if (Off != Off_Ins)
+            return nullptr;
+          if (!Insert->use_empty() && !Use)
+            if (ShuffleVectorInst *Shuffle =
+                    dyn_cast<ShuffleVectorInst>(Insert->user_back()))
+              Use = Shuffle;
+          Inserts[Off] = Insert;
+        } else
+          // Strip off speculation.marker for other loads
+          // in the chain.
+          Load->setMetadata(LLVMContext::MD_speculation_marker, nullptr);
+        Loads[Off] = Load;
+        GEPs[Off] = GEP;
+      }
+  }
+  MaxOffset = VecType->getVectorNumElements();
+  if (!Use || Use->getParent() != BB)
+    return nullptr;
+
+  unsigned PrevOff = 0;
+  for (auto Off : Offsets) {
+    if (Off > MaxOffset - 1)
+      return nullptr;
+    if ((Off != 0 && GEPs[Off]) || Loads[Off] || Inserts[Off])
+      return nullptr;
+    for (unsigned i = PrevOff + 1; i < Off; i++)
+      if ((i != 0 && !GEPs[i]) || !Loads[i] || !Inserts[i])
+        return nullptr;
+    PrevOff = Off;
+  }
+
+  IRBuilder<> Builder(LInstr->getParent(), ++BasicBlock::iterator(LInstr));
+  if (!Inserts[0]) {
+    Builder.SetInsertPoint(LInstr->getPrevNode());
+    LoadInst *NewLoad = Builder.CreateLoad(Ptr);
+    NewLoad->setAlignment(LInstr->getAlignment());
+    Loads[0] = NewLoad;
+    Value *NewInsert = Builder.CreateInsertElement(
+        UndefValue::get(VecType), NewLoad, Builder.getInt32(0));
+    Inserts[0] = cast<InsertElementInst>(NewInsert);
+  }
+  for (unsigned i = 1, e = MaxOffset; i < e; i++) {
+    // Building GEP, Load, Insert for
+    // this Offset.
+    GetElementPtrInst *PrevGEP = GEPs[i - 1];
+    if (!Inserts[i]) {
+      assert(Loads[i - 1] != nullptr &&
+             "Couldn't find previous load in the chain.");
+      if (!PrevGEP)
+        Builder.SetInsertPoint(BB, ++Loads[i - 1]->getIterator());
+      else
+        Builder.SetInsertPoint(BB, ++PrevGEP->getIterator());
+      Value *NewGEP =
+          Builder.CreateGEP(LInstr->getType(), Ptr, Builder.getInt64(i));
+      GEPs[i] = cast<GetElementPtrInst>(NewGEP);
+      Builder.SetInsertPoint(BB, ++Loads[i - 1]->getIterator());
+      Builder.SetCurrentDebugLocation(LInstr->getDebugLoc());
+      LoadInst *NewLoad = Builder.CreateLoad(NewGEP);
+      NewLoad->setAlignment(LInstr->getAlignment());
+      Loads[i] = NewLoad;
+      // InsertElement must be inserted after
+      // previous InsertElement.
+      InsertElementInst *PrevIns = Inserts[i - 1];
+      Builder.SetInsertPoint(BB, ++PrevIns->getIterator());
+      Value *NewInsert =
+          Builder.CreateInsertElement(PrevIns, NewLoad, Builder.getInt32(i));
+      Inserts[i] = cast<InsertElementInst>(NewInsert);
+    }
+    Inserts[i]->setOperand(0, Inserts[i - 1]);
+  }
+
+  // Update ShuffleVector with restored insert elements.
+  Use->setOperand(0, Inserts[MaxOffset - 1]);
+  return cast<InsertElementInst>(Inserts[MaxOffset - 1]);
+}
+
 bool SLPVectorizerPass::vectorizeInsertElementInst(InsertElementInst *IEI,
                                                    BasicBlock *BB, BoUpSLP &R) {
   SmallVector<Value *, 16> BuildVector;
   SmallVector<Value *, 16> BuildVectorOpds;
   if (!findBuildVector(IEI, BuildVector, BuildVectorOpds))
     return false;
 
+  if (LoadInst *LInstr = dyn_cast<LoadInst>(IEI->getOperand(1))) {
+    if (LInstr->getMetadata(LLVMContext::MD_speculation_marker) != nullptr)
+    if (InsertElementInst *Insert = restoreInserts(LInstr)) {
+      BuildVector.clear();
+      BuildVectorOpds.clear();
+      if (!findBuildVector(Insert, BuildVector, BuildVectorOpds))
+        return false;
+    }
+  }
+
   // Vectorize starting with the build vector operands ignoring the BuildVector
   // instructions for the purpose of scheduling and user extraction.
   return tryToVectorizeList(BuildVectorOpds, R, BuildVector);
 }
 
 bool SLPVectorizerPass::vectorizeCmpInst(CmpInst *CI, BasicBlock *BB,
                                          BoUpSLP &R) {
   if (tryToVectorizePair(CI->getOperand(0), CI->getOperand(1), R))

test/ThinLTO/X86/lazyload_metadata.ll

 ; Do setup work for all below tests: generate bitcode and combined index
 ; RUN: opt -module-summary %s -o %t.bc -bitcode-mdindex-threshold=0
 ; RUN: opt -module-summary %p/Inputs/lazyload_metadata.ll -o %t2.bc -bitcode-mdindex-threshold=0
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t3.bc %t.bc %t2.bc
 ; REQUIRES: asserts
 
 ; Check that importing @globalfunc1 does not trigger loading all the global
 ; metadata for @globalfunc2 and @globalfunc3
 
 ; RUN: llvm-lto -thinlto-action=import %t2.bc -thinlto-index=%t3.bc \
 ; RUN:          -o /dev/null -stats \
 ; RUN:  2>&1 | FileCheck %s -check-prefix=LAZY
-; LAZY: 55 bitcode-reader  - Number of Metadata records loaded
+; LAZY: 57 bitcode-reader  - Number of Metadata records loaded
 ; LAZY: 2 bitcode-reader  - Number of MDStrings loaded
 
 ; RUN: llvm-lto -thinlto-action=import %t2.bc -thinlto-index=%t3.bc \
 ; RUN:          -o /dev/null -disable-ondemand-mds-loading -stats \
 ; RUN:  2>&1 | FileCheck %s -check-prefix=NOTLAZY
-; NOTLAZY: 64 bitcode-reader  - Number of Metadata records loaded
+; NOTLAZY: 66 bitcode-reader  - Number of Metadata records loaded
 ; NOTLAZY: 7 bitcode-reader  - Number of MDStrings loaded
 
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 define void @globalfunc1(i32 %arg) {
   %x = call i1 @llvm.type.test(i8* undef, metadata !"typeid1")

test/Transforms/SLPVectorizer/X86/pr21780.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=bdver2 -S | FileCheck %s
+
+define <4 x double> @foo(double* %ptr) #0 {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr double, double* [[PTR]], i64 3
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr double, double* [[PTR]], i64 1
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast double* [[PTR]] to <4 x double>*
+; CHECK-NEXT:    [[TMP4:%.*]] = load <4 x double>, <4 x double>* [[TMP3]], align 8
+; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <4 x double> [[TMP4]], i32 0
+; CHECK-NEXT:    [[INS0:%.*]] = insertelement <4 x double> undef, double [[TMP5]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x double> [[TMP4]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x double> [[INS0]], double [[TMP6]], i32 1
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x double> [[TMP4]], i32 2
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[TMP7]], double [[TMP8]], i32 2
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <4 x double> [[TMP4]], i32 3
+; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x double> [[INS2]], double [[TMP9]], i32 3
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP10]], <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %ld0 = load double, double* %ptr, align 8, !speculation.marker !0
+  %ld2 = load double, double* %arrayidx2, align 8, !speculation.marker !1
+  %ins0 = insertelement <4 x double> undef, double %ld0, i32 0
+  %ins2 = insertelement <4 x double> %ins0, double %ld2, i32 2
+  %shuffle = shufflevector <4 x double> %ins2, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}
+
+define <4 x double> @bar(double* %ptr) #0 {
+; CHECK-LABEL: @bar(
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR]], i64 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr double, double* [[PTR]], i64 3
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast double* [[PTR]] to <4 x double>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <4 x double>, <4 x double>* [[TMP2]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <4 x double> [[TMP3]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> undef, double [[TMP4]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x double> [[TMP3]], i32 1
+; CHECK-NEXT:    [[INS1:%.*]] = insertelement <4 x double> [[TMP5]], double [[TMP6]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <4 x double> [[TMP3]], i32 2
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[INS1]], double [[TMP7]], i32 2
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x double> [[TMP3]], i32 3
+; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x double> [[INS2]], double [[TMP8]], i32 3
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP9]], <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  %arrayidx1 = getelementptr inbounds double, double* %ptr, i64 1
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %ld1 = load double, double* %arrayidx1, align 8, !speculation.marker !2
+  %ld2 = load double, double* %arrayidx2, align 8, !speculation.marker !3
+  %ins1 = insertelement <4 x double> undef, double %ld1, i32 1
+  %ins2 = insertelement <4 x double> %ins1, double %ld2, i32 2
+  %shuffle = shufflevector <4 x double> %ins2, <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}
+
+attributes #0 = { "target-cpu"="bdver2" }
+
+!0 = !{i64 1, i64 3}
+!1 = !{i64 -1, i64 1}
+!2 = !{i64 -1, i64 2}
+!3 = !{i64 -2, i64 1}