[InstCombine] Fix PR21780 Expansion of 256 bit vector loads fails to fold into shuffles

[dtemirbulatov]

Summary

This is a first part of PR21780 fix.

This change allows to keep information about removed Load instruction by InstCombine pass in order to restore those later. Here I am using Intrinsics to keep information in the IR as it was discussed in http://lists.llvm.org/pipermail/llvm-dev/2017-July/115730.html. Before that, I used metadata for that purpose https://reviews.llvm.org/D35139.

I introduced phantom_mem intrinsic that consist of two parameters the first is pointer and the second is maximum offset for that pointer. In the Instcombine pass, while constructing such intrisics, I try to avoid case if GEP's origin is in a different basic block and GEP itself is in a loop, but it is ok if both GEP and the pointer origin are in the loop and both belong to the same basic block.

[RKSimon]

Some style comments as the poor bracing/indentation makes it difficult to grok - more thorough review needs to wait until you've addressed @hfinkel's comments on D37648 about phantom mem intrinsics

lib/Transforms/InstCombine/InstructionCombining.cpp

1521

clang-format this - your indentation is broken.

2990

clang-format - braces and indentation are really bad

[sanjoy]

Why not leave the original load in place, and tag the load metadata like !speculation_marker ? In either case, this needs a LangRef entry.

[sanjoy]

In D37579#868685, @sanjoy wrote:

Why not leave the original load in place, and tag the load metadata like !speculation_marker ? In either case, this needs a LangRef entry.

Re-reading, I messed up the phrasing there. I meant to say two things:

• You could consider leaving the original load in place, and tag it with something like !speculation_marker. This metadata would indicate to passes like DCE that said load should not be eliminated till the very end of the optimization pipeline (so that we keep the optimization information for as long as possible). This does block optimizations though (like you will not be able to sink stores across this now present load, something you may have been able to do with the intrinsic you're introducing, though not without figuring out a way to denote it as not reading memory), but it has the upside of not introducing a new intrinsic.
• Irrespective of whether you decide to go with the intrinsic or the metadata approach, please also update the language reference with an entry describing the semantics of this new thing.

[dtemirbulatov]

In D37579#865897, @RKSimon wrote:

Some style comments as the poor bracing/indentation makes it difficult to grok - more thorough review needs to wait until you've addressed @hfinkel's comments on D37648 about phantom mem intrinsics

I think I did, it is here http://lists.llvm.org/pipermail/llvm-dev/2017-September/117381.html

[dtemirbulatov]

In D37579#869260, @sanjoy wrote:

In D37579#868685, @sanjoy wrote:

Why not leave the original load in place, and tag the load metadata like !speculation_marker ? In either case, this needs a LangRef entry.

Re-reading, I messed up the phrasing there. I meant to say two things:

• You could consider leaving the original load in place, and tag it with something like !speculation_marker. This metadata would indicate to passes like DCE that said load should not be eliminated till the very end of the optimization pipeline (so that we keep the optimization information for as long as possible). This does block optimizations though (like you will not be able to sink stores across this now present load, something you may have been able to do with the intrinsic you're introducing, though not without figuring out a way to denote it as not reading memory), but it has the upside of not introducing a new intrinsic.
• Irrespective of whether you decide to go with the intrinsic or the metadata approach, please also update the language reference with an entry describing the semantics of this new thing.

Thanks, for me, implementation with intrinsic looks better since it is detached from the actual instruction, since later this instruction might end up in a loop, etc.

[dtemirbulatov]

Fix review remarks and add an aggregate pointer intrinsic's second parameter.

[dtemirbulatov]

Fix recordDeadLoad() parameter lining with clang-format.

[dtemirbulatov]

add missed test in the last change.

[dtemirbulatov]

Update for solution, here it uses metadata "speculation.marker" with set of offsets instead of maximum offset as before, thread for discussion http://lists.llvm.org/pipermail/llvm-dev/2017-September/117782.html.

[hfinkel]

See my comment in D37648.

[dtemirbulatov]

Add LangRef.rst change, update test to check metadata values.

[dtemirbulatov]

Rebase and fix remarks.

[dtemirbulatov]

Update description in LangRef.rst.

[reames]

Initial round of comments focused on the metadata semantics only. I completely skipped the instcombine changes.

You also need verifier tests.

docs/LangRef.rst

4919 ↗

(On Diff #124020)

This needs a name change.

We already have dereferenceable attribute which marks arguments and return values as being *globally* deferenceable. This is marking a memory location as being dereferenceable at a particular contextual location.

Maybe: dereferenceable_offsets?

4935 ↗

(On Diff #124020)

More information on the format is needed. Specific missing pieces:

• what do both fields mean?
• i64 or any constant?
• signed or unsigned?
• is it a list?

I'd also suggest requiring sorting for easy of access.

test/Transforms/InstCombine/speculation_marker.ll

4

Just to comment: in your test case, we can prove that the loads post dominate the entry to the function and could update the argument with the existing dereferenceability attribute. This might be an alternate approach and separately worth implementation.

[RKSimon]

Adding @jdoerfert who can probably advise regarding whether Attributor will handle all of this

[jdoerfert]

In D37579#1654899, @RKSimon wrote:

Adding @jdoerfert who can probably advise regarding whether Attributor will handle all of this

First, sorry for not reacting sooner.

---

Long story short, I would prefer a more generic solution.

The way I understand this patch is that it tries to "remember" knowledge we are about to loose due to a transformation, correct? (If not, my next paragraphs are probably way off.)

To answer the Attributor question:
The Attributor will tie dereferenceable and nonnull knowledge to arguments and call site return values (especially the latter is not properly working due to some stuff I don't want to get into).
That means, if the base pointer is an argument (like in the test cases) and the accesses are in the must-be-executed-context of the argument (like in the test cases) it will work. If either is
not the case, it won't. That is the problem I would fix.

A similar situation arose recently in D69477. While we got a temporary solution in there, I think my response to it is adequate here as well https://reviews.llvm.org/D69477#1724399 .
I am in contact with @Tyker on this but I haven't found the time to write an RFC. If more people are interested in a generic way to preserve information I'd be happy to share my thoughts so far an help on a design.

To give you an idea, I was currently thinking about something along the lines of: llvm.assume() ["derefereceable"(%p, 8), "derefereceable"(%gep, 4), "align"(%p, 16) ]

include/llvm/IR/LLVMContext.h

enum {
     MD_invariant_group = 16,          // "invariant.group"
     MD_align = 17,                    // "align"
     MD_loop = 18,                     // "llvm.loop"
     MD_type = 19,                     // "type"
     MD_section_prefix = 20,           // "section_prefix"
     MD_absolute_symbol = 21,          // "absolute_symbol"
     MD_associated = 22,               // "associated"
     MD_callees = 23,                  // "callees"
+    MD_speculation_marker = 24,        // "speculation.marker"
   };
 
   /// Known operand bundle tag IDs, which always have the same value.  All
   /// operand bundle tags that LLVM has special knowledge of are listed here.
   /// Additionally, this scheme allows LLVM to efficiently check for specific
   /// operand bundle tags without comparing strings.
   enum {
     OB_deopt = 0,         // "deopt"

include/llvm/Transforms/Vectorize/SLPVectorizer.h

private:
 
   /// \brief Vectorize the store instructions collected in Stores.
   bool vectorizeStoreChains(slpvectorizer::BoUpSLP &R);
 
   /// \brief Vectorize the index computations of the getelementptr instructions
   /// collected in GEPs.
   bool vectorizeGEPIndices(BasicBlock *BB, slpvectorizer::BoUpSLP &R);
 
+  /// \brief Restore inserts out of speculation.marker metadata.
+  InsertElementInst * restoreInserts(LoadInst *LInstr);
+
   /// Try to find horizontal reduction or otherwise vectorize a chain of binary
   /// operators.
   bool vectorizeRootInstruction(PHINode *P, Value *V, BasicBlock *BB,
                                 slpvectorizer::BoUpSLP &R,
                                 TargetTransformInfo *TTI);
 
   /// Try to vectorize trees that start at insertvalue instructions.
   bool vectorizeInsertValueInst(InsertValueInst *IVI, BasicBlock *BB,
private:
 
   bool vectorizeStores(ArrayRef<StoreInst *> Stores, slpvectorizer::BoUpSLP &R);
 
   /// The store instructions in a basic block organized by base pointer.
   StoreListMap Stores;
 
   /// The getelementptr instructions in a basic block organized by base pointer.
   WeakTrackingVHListMap GEPs;
+
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_VECTORIZE_SLPVECTORIZER_H

lib/IR/LLVMContext.cpp

std::pair<unsigned, StringRef> MDKinds[] = {
     {MD_invariant_group, "invariant.group"},
     {MD_align, "align"},
     {MD_loop, "llvm.loop"},
     {MD_type, "type"},
     {MD_section_prefix, "section_prefix"},
     {MD_absolute_symbol, "absolute_symbol"},
     {MD_associated, "associated"},
     {MD_callees, "callees"},
+    {MD_speculation_marker, "speculation.marker"},
   };
 
   for (auto &MDKind : MDKinds) {
     unsigned ID = getMDKindID(MDKind.second);
     assert(ID == MDKind.first && "metadata kind id drifted");
     (void)ID;
   }
 

lib/Transforms/InstCombine/InstCombineInternal.h

case Intrinsic::ssub_with_overflow:
     return OCF_SIGNED_SUB;
   case Intrinsic::umul_with_overflow:
     return OCF_UNSIGNED_MUL;
   case Intrinsic::smul_with_overflow:
     return OCF_SIGNED_MUL;
   }
 }
 
+typedef DenseMap<Value*, SmallDenseSet<uint64_t>> BasePtrInfoTy;
+typedef DenseMap<Value*, std::pair<Value*, uint64_t>> PtrInfoTy;
+
 /// \brief The core instruction combiner logic.
 ///
 /// This class provides both the logic to recursively visit instructions and
 /// combine them.
 class LLVM_LIBRARY_VISIBILITY InstCombiner
     : public InstVisitor<InstCombiner, Instruction *> {
   // FIXME: These members shouldn't be public.
 public:
private:
   DominatorTree &DT;
   const DataLayout &DL;
   const SimplifyQuery SQ;
   OptimizationRemarkEmitter &ORE;
 
   // Optional analyses. When non-null, these can both be used to do better
   // combining and will be updated to reflect any changes.
   LoopInfo *LI;
+  BasePtrInfoTy &Bases;
+  PtrInfoTy &Ptrs;
 
   bool MadeIRChange = false;
 
 public:
   InstCombiner(InstCombineWorklist &Worklist, BuilderTy &Builder,
                bool MinimizeSize, bool ExpensiveCombines, AliasAnalysis *AA,
                AssumptionCache &AC, TargetLibraryInfo &TLI, DominatorTree &DT,
                OptimizationRemarkEmitter &ORE, const DataLayout &DL,
-               LoopInfo *LI)
+               LoopInfo *LI, BasePtrInfoTy &Bases, PtrInfoTy &Ptrs)
       : Worklist(Worklist), Builder(Builder), MinimizeSize(MinimizeSize),
         ExpensiveCombines(ExpensiveCombines), AA(AA), AC(AC), TLI(TLI), DT(DT),
-        DL(DL), SQ(DL, &TLI, &DT, &AC), ORE(ORE), LI(LI) {}
+        DL(DL), SQ(DL, &TLI, &DT, &AC), ORE(ORE), LI(LI),
+        Bases(Bases), Ptrs(Ptrs) {}
 
   /// \brief Run the combiner over the entire worklist until it is empty.
   ///
   /// \returns true if the IR is changed.
   bool run();
 
   AssumptionCache &getAssumptionCache() const { return AC; }
 

lib/Transforms/InstCombine/InstructionCombining.cpp

 #include "llvm/IR/Operator.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Use.h"
 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/IR/ValueHandle.h"
+#include "llvm/IR/MDBuilder.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/CBindingWrapping.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugCounter.h"
 #include "llvm/Support/ErrorHandling.h"
Value *InstCombiner::SimplifyVectorOp(BinaryOperator &Inst) {
   }
 
   return nullptr;
 }
 
 Instruction *InstCombiner::visitGetElementPtrInst(GetElementPtrInst &GEP) {
   SmallVector<Value*, 8> Ops(GEP.op_begin(), GEP.op_end());
 
+  // Here we have to avoid any GEPs with vector types or with
+  // number of operands that is not equal to two or GEPs
+  // in a loop, except it's origin is in the same basic block.
+  if (!dyn_cast<VectorType>(Ops[0]->getType()) && GEP.getNumOperands() == 2 &&
+      LI != nullptr) {
+    bool ValidGEP = (LI->getLoopDepth(GEP.getParent()) == 0);
+    Value *Ptr = cast<Value>(&GEP);
+    if (!ValidGEP)
+      if (Instruction *PtrInst = dyn_cast<Instruction>(Ptr))
+        ValidGEP = PtrInst->getParent() == GEP.getParent();
+    ConstantInt *C = dyn_cast<ConstantInt>(GEP.getOperand(1));
+    if (ValidGEP && C != nullptr) {
+      Value *BasePtr = GEP.getOperand(0);
+      Ptrs[Ptr].first = BasePtr;
+      Ptrs[Ptr].second = C->getZExtValue();
+      if (Ptrs.find(BasePtr) == Ptrs.end()) {
+        Ptrs[BasePtr].first = BasePtr;
+        Ptrs[BasePtr].second = 0;
+      }
+    }
+  }
+
   if (Value *V = SimplifyGEPInst(GEP.getSourceElementType(), Ops,
                                  SQ.getWithInstruction(&GEP)))
     return replaceInstUsesWith(GEP, V);
 
   Value *PtrOp = GEP.getOperand(0);
 
   // Eliminate unneeded casts for indices, and replace indices which displace
   // by multiples of a zero size type with zero.

 bool InstCombiner::run() {
   while (!Worklist.isEmpty()) {
     Instruction *I = Worklist.RemoveOne();
     if (I == nullptr) continue;  // skip null values.
 
     // Check to see if we can DCE the instruction.
     if (isInstructionTriviallyDead(I, &TLI)) {
       DEBUG(dbgs() << "IC: DCE: " << *I << '\n');
+      if (InsertElementInst *Insert = dyn_cast<InsertElementInst>(I)) {
+        LoadInst *Load = dyn_cast<LoadInst>(Insert->getOperand(1));
+        if (Load != nullptr) {
+          Value *Ptr = Load->getOperand(0);
+          if (Ptrs.find(Ptr) != Ptrs.end()) {
+            Value *BasePtr = Ptrs[Ptr].first;
+            Bases[BasePtr].insert(Ptrs[Ptr].second);
+          }
+        }
+      }
       eraseInstFromFunction(*I);
       ++NumDeadInst;
       MadeIRChange = true;
       continue;
     }
 
     if (!DebugCounter::shouldExecute(VisitCounter))
       continue;
 
     // Instruction isn't dead, see if we can constant propagate it.
     if (!I->use_empty() &&
         (I->getNumOperands() == 0 || isa<Constant>(I->getOperand(0)))) {
       if (Constant *C = ConstantFoldInstruction(I, DL, &TLI)) {
         DEBUG(dbgs() << "IC: ConstFold to: " << *C << " from: " << *I << '\n');
 
         // Add operands to the worklist.
         replaceInstUsesWith(*I, C);
         ++NumConstProp;
         if (isInstructionTriviallyDead(I, &TLI))
           eraseInstFromFunction(*I);
         MadeIRChange = true;
         continue;
       }
     }
 
     // In general, it is possible for computeKnownBits to determine all bits in
     // a value even when the operands are not all constants.
     Type *Ty = I->getType();
     if (ExpensiveCombines && !I->use_empty() && Ty->isIntOrIntVectorTy()) {
       KnownBits Known = computeKnownBits(I, /*Depth*/0, I);

 /// This has a couple of tricks to make the code faster and more powerful.  In
 /// particular, we constant fold and DCE instructions as we go, to avoid adding
 /// them to the worklist (this significantly speeds up instcombine on code where
 /// many instructions are dead or constant).  Additionally, if we find a branch
 /// whose condition is a known constant, we only visit the reachable successors.
 static bool AddReachableCodeToWorklist(BasicBlock *BB, const DataLayout &DL,
                                        SmallPtrSetImpl<BasicBlock *> &Visited,
                                        InstCombineWorklist &ICWorklist,
-                                       const TargetLibraryInfo *TLI) {
+                                       const TargetLibraryInfo *TLI,
+                                       BasePtrInfoTy &Bases,
+                                       PtrInfoTy &Ptrs) {
   bool MadeIRChange = false;
   SmallVector<BasicBlock*, 256> Worklist;
   Worklist.push_back(BB);
 
   SmallVector<Instruction*, 128> InstrsForInstCombineWorklist;
   DenseMap<Constant *, Constant *> FoldedConstants;
 
   do {
     BB = Worklist.pop_back_val();
 
     // We have now visited this block!  If we've already been here, ignore it.
     if (!Visited.insert(BB).second)
       continue;
 
     for (BasicBlock::iterator BBI = BB->begin(), E = BB->end(); BBI != E; ) {
       Instruction *Inst = &*BBI++;
 
       // DCE instruction if trivially dead.
       if (isInstructionTriviallyDead(Inst, TLI)) {
         ++NumDeadInst;
         DEBUG(dbgs() << "IC: DCE: " << *Inst << '\n');
+        if (InsertElementInst *Insert = dyn_cast<InsertElementInst>(Inst)) {
+          LoadInst *Load = dyn_cast<LoadInst>(Insert->getOperand(1));
+          if (Load != nullptr) {
+            Value *Ptr = Load->getOperand(0);
+            if (Ptrs.find(Ptr) != Ptrs.end()) {
+              Value *BasePtr = Ptrs[Ptr].first;
+              Bases[BasePtr].insert(Ptrs[Ptr].second);
+            }
+          }
+        }
         Inst->eraseFromParent();
         MadeIRChange = true;
         continue;
       }
 
       // ConstantProp instruction if trivially constant.
       if (!Inst->use_empty() &&
           (Inst->getNumOperands() == 0 || isa<Constant>(Inst->getOperand(0))))

 
 /// \brief Populate the IC worklist from a function, and prune any dead basic
 /// blocks discovered in the process.
 ///
 /// This also does basic constant propagation and other forward fixing to make
 /// the combiner itself run much faster.
 static bool prepareICWorklistFromFunction(Function &F, const DataLayout &DL,
                                           TargetLibraryInfo *TLI,
-                                          InstCombineWorklist &ICWorklist) {
+                                          InstCombineWorklist &ICWorklist,
+                                          BasePtrInfoTy &Bases,
+                                          PtrInfoTy &Ptrs) {
   bool MadeIRChange = false;
 
   // Do a depth-first traversal of the function, populate the worklist with
   // the reachable instructions.  Ignore blocks that are not reachable.  Keep
   // track of which blocks we visit.
   SmallPtrSet<BasicBlock *, 32> Visited;
   MadeIRChange |=
-      AddReachableCodeToWorklist(&F.front(), DL, Visited, ICWorklist, TLI);
+      AddReachableCodeToWorklist(&F.front(), DL, Visited, ICWorklist, TLI,
+                                 Bases, Ptrs);
 
   // Do a quick scan over the function.  If we find any blocks that are
   // unreachable, remove any instructions inside of them.  This prevents
   // the instcombine code from having to deal with some bad special cases.
   for (BasicBlock &BB : F) {
     if (Visited.count(&BB))
       continue;
 
static bool combineInstructionsOverFunction(
   IRBuilder<TargetFolder, IRBuilderCallbackInserter> Builder(
       F.getContext(), TargetFolder(DL),
       IRBuilderCallbackInserter([&Worklist, &AC](Instruction *I) {
         Worklist.Add(I);
         if (match(I, m_Intrinsic<Intrinsic::assume>()))
           AC.registerAssumption(cast<CallInst>(I));
       }));
 
+  BasePtrInfoTy Bases;
+  PtrInfoTy Ptrs;
+
   // Lower dbg.declare intrinsics otherwise their value may be clobbered
   // by instcombiner.
   bool MadeIRChange = false;
   if (ShouldLowerDbgDeclare)
     MadeIRChange = LowerDbgDeclare(F);
 
   // Iterate while there is work to do.
   int Iteration = 0;
   while (true) {
     ++Iteration;
     DEBUG(dbgs() << "\n\nINSTCOMBINE ITERATION #" << Iteration << " on "
                  << F.getName() << "\n");
 
-    MadeIRChange |= prepareICWorklistFromFunction(F, DL, &TLI, Worklist);
+    MadeIRChange |= prepareICWorklistFromFunction(F, DL, &TLI, Worklist, Bases,
+                                                  Ptrs);
 
     InstCombiner IC(Worklist, Builder, F.optForMinSize(), ExpensiveCombines, AA,
-                    AC, TLI, DT, ORE, DL, LI);
+                    AC, TLI, DT, ORE, DL, LI, Bases, Ptrs);
     IC.MaxArraySizeForCombine = MaxArraySize;
 
     if (!IC.run())
       break;
   }
 
+  bool SpeculativeLoad = false;
+  for (auto x : Bases)
+    if (!x.second.empty())
+      SpeculativeLoad = true;
+
+  if (SpeculativeLoad)
+    for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
+      for (BasicBlock::iterator I = BB->begin(), E = BB->end(); I != E; I++)
+        if (LoadInst *Load = dyn_cast<LoadInst>(*&I)) {
+          Value *Ptr = Load->getOperand(0);
+          if (Ptrs.find(Ptr) != Ptrs.end()) {
+            SmallVector<Metadata *, 2> Vals;
+            SmallDenseSet<uint64_t> &Set = Bases[Ptrs[Ptr].first];
+            int64_t LoadOffset = Ptrs[Ptr].second;
+            MDBuilder MDB(Load->getContext());
+            MDNode *MD = Load->getMetadata(LLVMContext::MD_speculation_marker);
+            if (MD != nullptr)
+              for (int i = 0, e = MD->getNumOperands(); i < e; i++) {
+                ConstantInt *C =
+                    mdconst::dyn_extract<ConstantInt>(MD->getOperand(i));
+                int64_t Diff = LoadOffset + (int64_t)C->getSExtValue();
+                if (Set.count(Diff) == 0)
+                  Set.insert(Diff);
+              }
+            for (auto Off : Set) {
+              int64_t Offset = (int64_t)Off;
+              Constant *C = ConstantInt::get(Load->getContext(),
+                                             APInt(64, Offset - LoadOffset));
+              Vals.push_back(MDB.createConstant(C));
+            }
+            Load->setMetadata(LLVMContext::MD_speculation_marker,
+                              MDNode::get(Load->getContext(), Vals));
+          }
+        }
+
   return MadeIRChange || Iteration > 1;
 }
 
 PreservedAnalyses InstCombinePass::run(Function &F,
                                        FunctionAnalysisManager &AM) {
   auto &AC = AM.getResult<AssumptionAnalysis>(F);
   auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
   auto &TLI = AM.getResult<TargetLibraryAnalysis>(F);

test/ThinLTO/X86/lazyload_metadata.ll

 ; Do setup work for all below tests: generate bitcode and combined index
 ; RUN: opt -module-summary %s -o %t.bc -bitcode-mdindex-threshold=0
 ; RUN: opt -module-summary %p/Inputs/lazyload_metadata.ll -o %t2.bc -bitcode-mdindex-threshold=0
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t3.bc %t.bc %t2.bc
 ; REQUIRES: asserts
 
 ; Check that importing @globalfunc1 does not trigger loading all the global
 ; metadata for @globalfunc2 and @globalfunc3
 
 ; RUN: llvm-lto -thinlto-action=import %t2.bc -thinlto-index=%t3.bc \
 ; RUN:          -o /dev/null -stats \
 ; RUN:  2>&1 | FileCheck %s -check-prefix=LAZY
-; LAZY: 53 bitcode-reader  - Number of Metadata records loaded
+; LAZY: 55 bitcode-reader  - Number of Metadata records loaded
 ; LAZY: 2 bitcode-reader  - Number of MDStrings loaded
 
 ; RUN: llvm-lto -thinlto-action=import %t2.bc -thinlto-index=%t3.bc \
 ; RUN:          -o /dev/null -disable-ondemand-mds-loading -stats \
 ; RUN:  2>&1 | FileCheck %s -check-prefix=NOTLAZY
-; NOTLAZY: 62 bitcode-reader  - Number of Metadata records loaded
+; NOTLAZY: 64 bitcode-reader  - Number of Metadata records loaded
 ; NOTLAZY: 7 bitcode-reader  - Number of MDStrings loaded
 
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 define void @globalfunc1(i32 %arg) {
   %x = call i1 @llvm.type.test(i8* undef, metadata !"typeid1")

test/Transforms/InstCombine/speculation_marker.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -instcombine -mtriple=x86_64-unknown-linux-gnu -mcpu=bdver2 -S | FileCheck %s
+
+define <4 x double> @foo(double* %ptr) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 2
+; CHECK-NEXT:    [[LD0:%.*]] = load double, double* [[PTR]], align 8, !speculation.marker !0
+; CHECK-NEXT:    [[LD2:%.*]] = load double, double* [[ARRAYIDX2]], align 8, !speculation.marker !1
+; CHECK-NEXT:    [[INS0:%.*]] = insertelement <4 x double> undef, double [[LD0]], i32 0
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[INS0]], double [[LD2]], i32 2
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[INS2]], <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  %arrayidx0 = getelementptr inbounds double, double* %ptr, i64 0
+  %arrayidx1 = getelementptr inbounds double, double* %ptr, i64 1
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %arrayidx3 = getelementptr inbounds double, double* %ptr, i64 3
+
+  %ld0 = load double, double* %arrayidx0
+  %ld1 = load double, double* %arrayidx1
+  %ld2 = load double, double* %arrayidx2
+  %ld3 = load double, double* %arrayidx3
+
+  %ins0 = insertelement <4 x double> undef, double %ld0, i32 0
+  %ins1 = insertelement <4 x double> %ins0, double %ld1, i32 1
+  %ins2 = insertelement <4 x double> %ins1, double %ld2, i32 2
+  %ins3 = insertelement <4 x double> %ins2, double %ld3, i32 3
+
+  %shuffle = shufflevector <4 x double> %ins3, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}
+
+define <4 x double> @bar(double* %ptr) {
+; CHECK-LABEL: @bar(
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds double, double* [[PTR:%.*]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[PTR]], i64 2
+; CHECK-NEXT:    [[LD1:%.*]] = load double, double* [[ARRAYIDX1]], align 8, !speculation.marker !2
+; CHECK-NEXT:    [[LD2:%.*]] = load double, double* [[ARRAYIDX2]], align 8, !speculation.marker !3
+; CHECK-NEXT:    [[INS1:%.*]] = insertelement <4 x double> undef, double [[LD1]], i32 1
+; CHECK-NEXT:    [[INS2:%.*]] = insertelement <4 x double> [[INS1]], double [[LD2]], i32 2
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <4 x double> [[INS2]], <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+; CHECK-NEXT:    ret <4 x double> [[SHUFFLE]]
+;
+  %arrayidx0 = getelementptr inbounds double, double* %ptr, i64 0
+  %arrayidx1 = getelementptr inbounds double, double* %ptr, i64 1
+  %arrayidx2 = getelementptr inbounds double, double* %ptr, i64 2
+  %arrayidx3 = getelementptr inbounds double, double* %ptr, i64 3
+
+  %ld0 = load double, double* %arrayidx0
+  %ld1 = load double, double* %arrayidx1
+  %ld2 = load double, double* %arrayidx2
+  %ld3 = load double, double* %arrayidx3
+
+  %ins0 = insertelement <4 x double> undef, double %ld0, i32 0
+  %ins1 = insertelement <4 x double> %ins0, double %ld1, i32 1
+  %ins2 = insertelement <4 x double> %ins1, double %ld2, i32 2
+  %ins3 = insertelement <4 x double> %ins2, double %ld3, i32 3
+
+  %shuffle = shufflevector <4 x double> %ins3, <4 x double> undef, <4 x i32> <i32 1, i32 1, i32 2, i32 2>
+  ret <4 x double> %shuffle
+}