[SLPVectorizer] Fix PR21780 Expansion of 256 bit vector loads fails to fold into shuffles

[dtemirbulatov]

Summary

This is the second part of PR21780 fix, here is part one https://reviews.llvm.org/D37579

This change allows to restore Load, Insert, GEP instructions removed by InstCombine pass. Here I am using intrinsic to keep information in the IR as it was discussed in http://lists.llvm.org/pipermail/llvm-dev/2017-July/115730.html.

Before restoring instructions we have have to check existing instructions for belonging to our basic block, also we check the shuffle mask and all offset, ...

[hfinkel]

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

[RKSimon]

Some more style comment as with D37579

lib/Transforms/Vectorize/SLPVectorizer.cpp

4213

clang-format - braces

5370

It'd be much clearer to early out.

if (!C || C->getZExtValue() > MaxOffset)
  return nullptr;

// We found the first InsertElem in this chain.
uint64_t Offset = C->getZExtValue();
...

5423

for (unsigned i = 0, e = MaxOffset + 1; i < e; i++)

5434

if (!Use || Use->getParent() != BB)

5458

for (unsigned i = 0, e = MaxOffset + 1; i < e; i++)

5468

You're accessing GEPs[i - 1] multiple times - pull out?

[dtemirbulatov]

In D37648#865562, @hfinkel wrote:

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

Here is the thread: http://lists.llvm.org/pipermail/llvm-dev/2017-September/117381.html

[dtemirbulatov]

Fix review remarks and add an aggregate pointer as intrinsic's second parameter, remove getNextNode() usage.

[hfinkel]

In D37648#879283, @dtemirbulatov wrote:

In D37648#865562, @hfinkel wrote:

Thanks for working on this. Please post an RFC on llvm-dev about the intrinsic and how you intend to use it. You can reference this review, but we should make sure that we have design consensus before proceeding here.

Here is the thread: http://lists.llvm.org/pipermail/llvm-dev/2017-September/117381.html

Yes. Are you planing on updating that thread? Last message I see says:

but I am looking now at "speculation_marker" metadata and I am still not sure how to implement it better.

are you still looking at some other kind of metadata solution? Did you mean for "metadata" here to mean the intrinsic you're proposing?

[dtemirbulatov]

are you still looking at some other kind of metadata solution? Did you mean for "metadata" here to mean the intrinsic you're proposing?

No, I am happy with intrinsic solution.

[reames]

test/Transforms/SLPVectorizer/X86/pr21780.ll

22

I think something might be missing here. You're forming a 4x wide load, but you've only proven dereferenceability for offsets 0, 1, 3. (i.e. not 2). How do we know it's safe to dereference between the two elements 1 & 3?

[dtemirbulatov]

I think something might be missing here. You're forming a 4x wide load, but you've only proven dereferenceability for offsets 0, 1, 3. (i.e. not 2). How do we know it's safe to dereference between the two elements 1 & 3?

well, it is a maximum reference that was recorded there in InstCombine, that implies 0, 1, 2 and 3. And if you are saying that 0, 1, 3 dereferenceable and 2-nd is not at the same time then this solution is not correct.

[dtemirbulatov]

Update for solution, here it uses metadata "speculation.marker" with set of offsets instead of maximum offset as before, thread for discussion http://lists.llvm.org/pipermail/llvm-dev/2017-September/117782.html.

[hfinkel]

In D37648#906216, @dtemirbulatov wrote:

Update for solution, here it uses metadata "speculation.marker" with set of offsets instead of maximum offset as before, thread for discussion http://lists.llvm.org/pipermail/llvm-dev/2017-September/117782.html.

Can you please update the thread with your new plan for this metadata (what adds it, when, and what consumes it), including the proposed definition for the LangRef, so we can all see exactly what you're proposing?

[dtemirbulatov]

Add LangRef.rst change. Small changes in SLPVectorizer.cpp.

[RKSimon]

docs/LangRef.rst

4893 ↗

(On Diff #120330)

"consists"

4894 ↗

(On Diff #120330)

Is it a set or a min/max range? The examples only ever have 2 entries. If a set do they have to be sorted?

4896 ↗

(On Diff #120330)

"is to keep track of dereferanceable memory locations after load operations to that memory were deleted, as it might be beneficial for future optimizations."

include/llvm/IR/LLVMContext.h

104 ↗

(On Diff #120330)

// alignment

include/llvm/Transforms/Vectorize/SLPVectorizer.h

155

Unnecessary whitespace change

[dtemirbulatov]

Rebase and fix remarks.

[dtemirbulatov]

Update description in LangRef.rst.

[RKSimon]

@dtemirbulatov What is the status of this patch please?

[dtemirbulatov]

In D37648#1148425, @RKSimon wrote:

@dtemirbulatov What is the status of this patch please?

@RKSimon I think it needs review. I will rebase proposed change.

[RKSimon]

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

[dtemirbulatov]

In D37648#1653860, @RKSimon wrote:

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

no, I think I can rebase the change.

[lebedev.ri]

In D37648#1654027, @dtemirbulatov wrote:

In D37648#1653860, @RKSimon wrote:

Abandon this? I think this is being handled by the (WIP) Attributor work to support dereference

no, I think I can rebase the change.

I suspect @RKSimon is might be right here, it may be better to redesign the feature with nowadays best practices in mind (attributor).

[RKSimon]

Adding @jdoerfert who can probably advise regarding whether Attributor will handle all of this

[jdoerfert]

With D65402 (and later D65593), the Attributor will determine dereferenceability based on accesses that "must be executed".
Since the Attributor runs early in the pipeline (for now only then), this will "generally" solve the issue as unused loads are still present.
However, we lack the ability to annotate dereferenceable for pointers that are not arguments, function return values, call site arguments, or call site return values.
That means, we might determine dereferenceability early but fail to manifest it in the IR or loose the information during inlining. Though, I am unsure how much of
a problem that really is. If it turns out to be bad, I'd suggest to add an intrinsic to describe dereferenceability (among other things [alignment, max object size, ...]) of a pointer.

Long story short, I'd recommend investing the time in reviewing and testing the Attributor infrastructure and potentially thinking about a pointer property intrinsic.

If I misunderstood what is going on here, please let me know.

include/llvm/IR/Intrinsics.td

 def int_experimental_vector_reduce_fmin : Intrinsic<[llvm_anyfloat_ty],
                                                     [llvm_anyvector_ty],
                                                     [IntrNoMem]>;
 
 //===----- Intrinsics that are used to provide predicate information -----===//
 
 def int_ssa_copy : Intrinsic<[llvm_any_ty], [LLVMMatchType<0>],
                              [IntrNoMem, Returned<0>]>;
+
+def int_phantom_mem : Intrinsic<[],
+                            [llvm_anyptr_ty, llvm_anyptr_ty, llvm_i64_ty],
+                            [IntrArgMemOnly]>;
+
 //===----------------------------------------------------------------------===//
 // Target-specific intrinsics
 //===----------------------------------------------------------------------===//
 
 include "llvm/IR/IntrinsicsPowerPC.td"
 include "llvm/IR/IntrinsicsX86.td"
 include "llvm/IR/IntrinsicsARM.td"
 include "llvm/IR/IntrinsicsAArch64.td"
 include "llvm/IR/IntrinsicsXCore.td"
 include "llvm/IR/IntrinsicsHexagon.td"
 include "llvm/IR/IntrinsicsNVVM.td"
 include "llvm/IR/IntrinsicsMips.td"
 include "llvm/IR/IntrinsicsAMDGPU.td"
 include "llvm/IR/IntrinsicsBPF.td"
 include "llvm/IR/IntrinsicsSystemZ.td"
 include "llvm/IR/IntrinsicsWebAssembly.td"

include/llvm/Transforms/Vectorize/SLPVectorizer.h

private:
 
   /// \brief Vectorize the store instructions collected in Stores.
   bool vectorizeStoreChains(slpvectorizer::BoUpSLP &R);
 
   /// \brief Vectorize the index computations of the getelementptr instructions
   /// collected in GEPs.
   bool vectorizeGEPIndices(BasicBlock *BB, slpvectorizer::BoUpSLP &R);
 
+  /// \brief Restore inserts, loads out of phantom_mem intrinsic.
+  InsertElementInst * restoreInserts(InsertElementInst *FirstInsertElem,
+                                     LoadInst *LInstr, Value* Ptr);
+
   /// Try to find horizontal reduction or otherwise vectorize a chain of binary
   /// operators.
   bool vectorizeRootInstruction(PHINode *P, Value *V, BasicBlock *BB,
                                 slpvectorizer::BoUpSLP &R,
                                 TargetTransformInfo *TTI);
 
   /// Try to vectorize trees that start at insertvalue instructions.
   bool vectorizeInsertValueInst(InsertValueInst *IVI, BasicBlock *BB,
private:
 
   bool vectorizeStores(ArrayRef<StoreInst *> Stores, slpvectorizer::BoUpSLP &R);
 
   /// The store instructions in a basic block organized by base pointer.
   StoreListMap Stores;
 
   /// The getelementptr instructions in a basic block organized by base pointer.
   WeakTrackingVHListMap GEPs;
+
+  DenseMap<Value*, uint64_t> PhantomMem;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_VECTORIZE_SLPVECTORIZER_H

lib/Transforms/Vectorize/SLPVectorizer.cpp

bool SLPVectorizerPass::runImpl(Function &F, ScalarEvolution *SE_,
 
   // Use the bottom up slp vectorizer to construct chains that start with
   // store instructions.
   BoUpSLP R(&F, SE, TTI, TLI, AA, LI, DT, AC, DB, DL, ORE_);
 
   // A general note: the vectorizer must use BoUpSLP::eraseInstruction() to
   // delete instructions.
 
+  for (auto &BB : F)
+    for (BasicBlock::iterator BBI = BB.begin(), E = BB.end(); BBI != E;) {
+      Instruction *I = &*BBI++;
+
+      CallInst *CI = dyn_cast<CallInst>(I);
+      if (!CI)
+        continue;
+      if (getIntrinsicForCallSite(CI, TLI) == Intrinsic::phantom_mem) {
+        ConstantInt *MaxOffset = cast<ConstantInt>(CI->getOperand(2));
+        PhantomMem[CI->getOperand(0)] = MaxOffset->getZExtValue();
+        CI->removeFromParent();
+        CI->dropAllReferences();
+      }
+    }
+
   // Scan the blocks in the function in post order.
   for (auto BB : post_order(&F.getEntryBlock())) {
     collectSeedInstructions(BB);
 
     // Vectorize trees that end at stores.
     if (!Stores.empty()) {
       DEBUG(dbgs() << "SLP: Found stores for " << Stores.size()
                    << " underlying objects.\n");
bool SLPVectorizerPass::vectorizeInsertValueInst(InsertValueInst *IVI,
   SmallVector<Value *, 16> BuildVectorOpds;
   if (!findBuildAggregate(IVI, BuildVector, BuildVectorOpds))
     return false;
 
   DEBUG(dbgs() << "SLP: array mappable to vector: " << *IVI << "\n");
   return tryToVectorizeList(BuildVectorOpds, R, BuildVector, false);
 }
 
+InsertElementInst *
+SLPVectorizerPass::restoreInserts(InsertElementInst *FirstInsertElem,
+                                  LoadInst *LInstr, Value *Ptr) {
+  ShuffleVectorInst *Use = nullptr;
+  DenseMap<uint64_t, LoadInst *> Loads;
+  DenseMap<uint64_t, InsertElementInst *> Inserts;
+  DenseMap<uint64_t, GetElementPtrInst *> GEPs;
+  BasicBlock *BB = FirstInsertElem->getParent();
+  VectorType *VecType = FirstInsertElem->getType();
+  uint64_t MaxOffset = PhantomMem[Ptr];
+
+  ConstantInt *C = dyn_cast<ConstantInt>(FirstInsertElem->getOperand(2));
+
+  if (!C || C->getZExtValue() > MaxOffset)
+    return nullptr;
+
+  // We found the first InsertElem in
+  // this chain.
+  uint64_t Offset = C->getZExtValue();
+  Inserts[Offset] = FirstInsertElem;
+  Loads[Offset] = LInstr;
+  GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(LInstr->getOperand(0));
+  if (GEP)
+    GEPs[Offset] = GEP;
+
+  for (BasicBlock::iterator it = BB->begin(), e = BB->end(); it != e; it++) {
+    if (InsertElementInst *Insert = dyn_cast<InsertElementInst>(it)) {
+      ConstantInt *C = dyn_cast<ConstantInt>(Insert->getOperand(2));
+      LoadInst *Load = dyn_cast<LoadInst>(Insert->getOperand(1));
+      if (!Load || Insert == FirstInsertElem)
+        continue;
+      GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Load->getOperand(0));
+      Value *Base = Load->getOperand(0);
+      if (Base == Ptr) {
+        if (!C)
+          return nullptr;
+        uint64_t Offset = C->getZExtValue();
+        if (Offset > MaxOffset || !Load->hasOneUse())
+          return nullptr;
+        Inserts[Offset] = Insert;
+        Loads[Offset] = Load;
+        // GEP instruction might be missing for some offsets.
+        if (GEP)
+          GEPs[Offset] = GEP;
+        continue;
+      }
+    }
+    if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(it))
+      // Examining the chain of GEP, Load, Insert.
+      if (Ptr == GEP->getOperand(0)) {
+        ConstantInt *C = dyn_cast<ConstantInt>(GEP->getOperand(1));
+        if (!C || GEP->use_empty() || !GEP->hasOneUse())
+          return nullptr;
+        uint64_t Off = C->getZExtValue();
+        LoadInst *Load = dyn_cast<LoadInst>(GEP->user_back());
+        if (!Load || Load->use_empty() || !Load->hasOneUse())
+          return nullptr;
+        InsertElementInst *Insert =
+            dyn_cast<InsertElementInst>(Load->user_back());
+        if (!Insert)
+          return nullptr;
+        ConstantInt *C1 = cast<ConstantInt>(Insert->getOperand(2));
+        uint64_t Off_Ins = C1->getZExtValue();
+        // Offset from GEP must be equal to offset from
+        // Insert otherwise we have to ignore.
+        if (Off != Off_Ins)
+          return nullptr;
+        if (!Insert->use_empty())
+          if (ShuffleVectorInst *Shuffle =
+                  dyn_cast<ShuffleVectorInst>(Insert->user_back()))
+            Use = Shuffle;
+        Inserts[Off] = Insert;
+        Loads[Off] = Load;
+        GEPs[Off] = GEP;
+      }
+  }
+  if (!Use || Use->getParent() != BB)
+    return nullptr;
+
+  IRBuilder<> Builder(LInstr->getParent(), ++BasicBlock::iterator(LInstr));
+  if (!Inserts[0]) {
+    Builder.SetInsertPoint(LInstr->getPrevNode());
+    LoadInst *NewLoad = Builder.CreateLoad(Ptr);
+    NewLoad->setAlignment(LInstr->getAlignment());
+    Loads[0] = NewLoad;
+    Value *NewInsert = Builder.CreateInsertElement(
+        UndefValue::get(VecType), NewLoad, Builder.getInt32(0));
+    Inserts[0] = cast<InsertElementInst>(NewInsert);
+  }
+  for (unsigned i = 1, e = MaxOffset + 1; i < e; i++) {
+    // Building GEP, Load, Insert for
+    // this Offset.
+    GetElementPtrInst *PrevGEP = GEPs[i - 1];
+    if (!Inserts[i]) {
+      assert(Loads[i - 1] != nullptr &&
+             "Couldn't find previous load in the chain.");
+      if (PrevGEP == nullptr)
+        Builder.SetInsertPoint(BB, ++Loads[i - 1]->getIterator());
+      else 
+        Builder.SetInsertPoint(BB, ++PrevGEP->getIterator());
+      Value *NewGEP =
+          Builder.CreateGEP(LInstr->getType(), Ptr, Builder.getInt64(i));
+      GEPs[i] = cast<GetElementPtrInst>(NewGEP);
+      Builder.SetInsertPoint(BB, ++Loads[i - 1]->getIterator());
+      Builder.SetCurrentDebugLocation(LInstr->getDebugLoc());
+      LoadInst *NewLoad = Builder.CreateLoad(NewGEP);
+      NewLoad->setAlignment(LInstr->getAlignment());
+      Loads[i] = NewLoad;
+      // InsertElement must be inserted just after
+      // previous InsertElement.
+      InsertElementInst *PrevIns = Inserts[i - 1];
+      Builder.SetInsertPoint(BB, ++PrevIns->getIterator());
+      Value *NewInsert =
+          Builder.CreateInsertElement(PrevIns, NewLoad, Builder.getInt32(i));
+      Inserts[i] = cast<InsertElementInst>(NewInsert);
+    }
+    Inserts[i]->setOperand(0, Inserts[i - 1]);
+  }
+
+  // Update ShuffleVector with restored insert element.
+  Use->setOperand(0, Inserts[MaxOffset]);
+  return cast<InsertElementInst>(Inserts[MaxOffset]);
+}
+
 bool SLPVectorizerPass::vectorizeInsertElementInst(InsertElementInst *IEI,
                                                    BasicBlock *BB, BoUpSLP &R) {
   SmallVector<Value *, 16> BuildVector;
   SmallVector<Value *, 16> BuildVectorOpds;
   if (!findBuildVector(IEI, BuildVector, BuildVectorOpds))
     return false;
 
+  if (LoadInst *LInstr = dyn_cast<LoadInst>(IEI->getOperand(1))) {
+    Value *Ptr = nullptr;
+    if (PhantomMem.find(LInstr->getOperand(0)) != PhantomMem.end())
+      Ptr = LInstr->getOperand(0);
+    else if (GetElementPtrInst *GEP =
+                 dyn_cast<GetElementPtrInst>(LInstr->getOperand(0)))
+      if (PhantomMem.find(GEP->getOperand(0)) != PhantomMem.end())
+        Ptr = GEP->getOperand(0);
+
+    if (!Ptr)
+      return false;
+    if (InsertElementInst *Insert = restoreInserts(IEI, LInstr, Ptr)) {
+      BuildVector.clear();
+      BuildVectorOpds.clear();
+      if (!findBuildVector(Insert, BuildVector, BuildVectorOpds))
+        return false;
+    }
+  }
+
   // Vectorize starting with the build vector operands ignoring the BuildVector
   // instructions for the purpose of scheduling and user extraction.
   return tryToVectorizeList(BuildVectorOpds, R, BuildVector);
 }
 
 bool SLPVectorizerPass::vectorizeCmpInst(CmpInst *CI, BasicBlock *BB,
                                          BoUpSLP &R) {
   if (tryToVectorizePair(CI->getOperand(0), CI->getOperand(1), R))

test/Transforms/SLPVectorizer/X86/pr21780.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=bdver2 -S | FileCheck %s
+
+define <4 x double> @load_four_scalars_but_use_two(double* %ptr) #0 {
+; CHECK-LABEL: @load_four_scalars_but_use_two(
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr double, double* [[PTR]], i64 3
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr double, double* [[PTR]], i64 1
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast double* [[PTR]] to <4 x double>*
+; CHECK-NEXT:    [[TMP4:%.*]] = load <4 x double>, <4 x double>* [[TMP3]], align 8
+; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <4 x double> [[TMP4]], i32 0
+; CHECK-NEXT:    [[INS0:%.*]] = insertelement <4 x double> undef, double [[TMP5]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x double> [[TMP4]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x double> [[INS0]], double [[TMP6]], i32 1
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x double> [[TMP4]], i32 2
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[TMP7]], double [[TMP8]], i32 2
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <4 x double> [[TMP4]], i32 3
+; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x double> [[INS2]], double [[TMP9]], i32 3
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP10]], <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  call void @llvm.phantom.mem.p0f64.p0f64(double* %ptr, double* null, i64 3)
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %ld0 = load double, double* %ptr, align 8
+  %ld2 = load double, double* %arrayidx2, align 8
+  %ins0 = insertelement <4 x double> undef, double %ld0, i32 0
+  %ins2 = insertelement <4 x double> %ins0, double %ld2, i32 2
+  %shuffle = shufflevector <4 x double> %ins2, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}
+
+define <4 x double> @load_four_scalars_but_use_two1(double* %ptr) #0 {
+; CHECK-LABEL: @load_four_scalars_but_use_two1(
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR]], i64 2
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr double, double* [[PTR]], i64 3
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast double* [[PTR]] to <4 x double>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <4 x double>, <4 x double>* [[TMP2]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <4 x double> [[TMP3]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x double> undef, double [[TMP4]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <4 x double> [[TMP3]], i32 1
+; CHECK-NEXT:    [[INS1:%.*]] = insertelement <4 x double> [[TMP5]], double [[TMP6]], i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <4 x double> [[TMP3]], i32 2
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[INS1]], double [[TMP7]], i32 2
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x double> [[TMP3]], i32 3
+; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x double> [[INS2]], double [[TMP8]], i32 3
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[TMP9]], <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  call void @llvm.phantom.mem.p0f64.p0f64(double* %ptr, double* null, i64 3)
+  %arrayidx1 = getelementptr inbounds double, double* %ptr, i64 1
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %ld1 = load double, double* %arrayidx1, align 8
+  %ld2 = load double, double* %arrayidx2, align 8
+  %ins1 = insertelement <4 x double> undef, double %ld1, i32 1
+  %ins2 = insertelement <4 x double> %ins1, double %ld2, i32 2
+  %shuffle = shufflevector <4 x double> %ins2, <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.phantom.mem.p0f64.p0f64(double*, double*, i64) #1