[opaque pointer types] [NFC] {Load,Store}Inst: get loaded/stored type from the instruction.

[dblaikie]

lib/CodeGen/AtomicExpandPass.cpp

271

I must be missing something as this doesn't look quite right - why was the original code treating the load as though it were a load of a pointer? (load's IR type should be a value type, not a pointer type - unless it is loading a pointer (ie: load could be passed a pointer to a pointer)) & why is the new code doing something different?

lib/Transforms/Vectorize/BBVectorize.cpp

993

I take it this code is still correct - but you chose to plumb the two out params into getPairPtrInfo just so this code didn't have to do all the casting and conditionals, etc? (since you did have to add JTy/ITy to the implementation of getPairPtrInfo anyway, yes?)

2317

Same here, I take it? Or was this the part where it was necessary for getPairPtrInfo to produce these two values - and since you were producing them here you might as well use them in the other call site?

lib/Transforms/Vectorize/LoopVectorize.cpp

1991

Do we need to support a null type parameter here?

[eddyb]

lib/CodeGen/AtomicExpandPass.cpp

271

getNullValue, confusingly enough, isn't about nullptr but zeroinitializer.
And I didn't change the semantics, I just replaced LI->getPointerOperand()->getPointerElementType() with LI->getType().

lib/Transforms/Vectorize/BBVectorize.cpp

2317

Correct - otherwise I would've ended up with unused variables and duplicated work.

lib/Transforms/Vectorize/LoopVectorize.cpp

1991

This might be an artifact of rebasing, i.e. it was needed but now it's not.
I'll try testing without the nullptr default and this check.

[eddyb]

lib/Transforms/Vectorize/LoopVectorize.cpp

1991

Ah, that call site doesn't show up in the commit because it hasn't changed.
See the collectLoopUniforms method.

[mjacob]

lib/Target/PowerPC/PPCLoopPreIncPrep.cpp

209

I think you changed semantics here. What if the instruction is a prefetch intrinsic with a pointer to a vector as the first operand?

lib/Transforms/IPO/GlobalOpt.cpp

2437

Do we need this variable?

lib/Transforms/Scalar/MemCpyOptimizer.cpp

236

I think it's slightly more robost to get the type with MSI->getValue()->getType().

lib/Transforms/Scalar/Scalarizer.cpp

268–270

Can you document (and possibly add an assertion for) the requirements on VecTy?

If I understand correctly, there are two possibilities:

1. V is a vector and VecTy is nullptr.
2. V is a pointer to a vector and VecTy is the pointee type.

At least we should add an assertion that V is a vector if VecTy is nullptr.

lib/Transforms/Vectorize/LoopVectorize.cpp

1991

What if isConsecutivePtr is called from collectLoopUniforms with a pointer to a struct?

[eddyb]

lib/Target/PowerPC/PPCLoopPreIncPrep.cpp

209

Prefetch is monomorphic, it only takes one pointer, AFAICT.

lib/Transforms/IPO/GlobalOpt.cpp

2437

No, although, I might put SI->getValueOperand() in a variable just to keep the line length in check.

lib/Transforms/Scalar/Scalarizer.cpp

268–270

I agree with that - I also need to extract the element type and use it, which for some reason I didn't do back when I first made this change.

lib/Transforms/Vectorize/LoopVectorize.cpp

1991

Honestly, I'm not sure. I don't think that it affects the resulting decisions, but I can try to look deeper into it.

lib/CodeGen/AtomicExpandPass.cpp

bool AtomicExpand::expandAtomicLoadToLL(LoadInst *LI) {
 
   return true;
 }
 
 bool AtomicExpand::expandAtomicLoadToCmpXchg(LoadInst *LI) {
   IRBuilder<> Builder(LI);
   AtomicOrdering Order = LI->getOrdering();
   Value *Addr = LI->getPointerOperand();
-  Type *Ty = cast<PointerType>(Addr->getType())->getElementType();
-  Constant *DummyVal = Constant::getNullValue(Ty);
+  Constant *DummyVal = Constant::getNullValue(LI->getType());
 
   Value *Pair = Builder.CreateAtomicCmpXchg(
       Addr, DummyVal, DummyVal, Order,
       AtomicCmpXchgInst::getStrongestFailureOrdering(Order));
   Value *Loaded = Builder.CreateExtractValue(Pair, 0, "loaded");
 
   LI->replaceAllUsesWith(Loaded);
   LI->eraseFromParent();

lib/CodeGen/CodeGenPrepare.cpp

static bool IsOperandAMemoryOperand(CallInst *CI, InlineAsm *IA, Value *OpVal,
   return true;
 }
 
 /// Recursively walk all the uses of I until we find a memory use.
 /// If we find an obviously non-foldable instruction, return true.
 /// Add the ultimately found memory instructions to MemoryUses.
 static bool FindAllMemoryUses(
     Instruction *I,
-    SmallVectorImpl<std::pair<Instruction *, unsigned>> &MemoryUses,
+    SmallVectorImpl<Instruction *> &MemoryUses,
     SmallPtrSetImpl<Instruction *> &ConsideredInsts, const TargetMachine &TM) {
   // If we already considered this instruction, we're done.
   if (!ConsideredInsts.insert(I).second)
     return false;
 
   // If this is an obviously unfoldable instruction, bail out.
   if (!MightBeFoldableInst(I))
     return true;
 
   // Loop over all the uses, recursively processing them.
   for (Use &U : I->uses()) {
     Instruction *UserI = cast<Instruction>(U.getUser());
 
     if (LoadInst *LI = dyn_cast<LoadInst>(UserI)) {
-      MemoryUses.push_back(std::make_pair(LI, U.getOperandNo()));
+      MemoryUses.push_back(LI);
       continue;
     }
 
     if (StoreInst *SI = dyn_cast<StoreInst>(UserI)) {
       unsigned opNo = U.getOperandNo();
       if (opNo == 0) return true; // Storing addr, not into addr.
-      MemoryUses.push_back(std::make_pair(SI, opNo));
+      MemoryUses.push_back(SI);
       continue;
     }
 
     if (CallInst *CI = dyn_cast<CallInst>(UserI)) {
       InlineAsm *IA = dyn_cast<InlineAsm>(CI->getCalledValue());
       if (!IA) return true;
 
       // If this is a memory operand, we're cool, otherwise bail out.
isProfitableToFoldIntoAddressingMode(Instruction *I, ExtAddrMode &AMBefore,
   // ranges, we're ok with it.
   if (!BaseReg && !ScaledReg)
     return true;
 
   // If all uses of this instruction are ultimately load/store/inlineasm's,
   // check to see if their addressing modes will include this instruction.  If
   // so, we can fold it into all uses, so it doesn't matter if it has multiple
   // uses.
-  SmallVector<std::pair<Instruction*,unsigned>, 16> MemoryUses;
+  SmallVector<Instruction*, 16> MemoryUses;
   SmallPtrSet<Instruction*, 16> ConsideredInsts;
   if (FindAllMemoryUses(I, MemoryUses, ConsideredInsts, TM))
     return false;  // Has a non-memory, non-foldable use!
 
   // Now that we know that all uses of this instruction are part of a chain of
   // computation involving only operations that could theoretically be folded
   // into a memory use, loop over each of these uses and see if they could
   // *actually* fold the instruction.
   SmallVector<Instruction*, 32> MatchedAddrModeInsts;
   for (unsigned i = 0, e = MemoryUses.size(); i != e; ++i) {
-    Instruction *User = MemoryUses[i].first;
-    unsigned OpNo = MemoryUses[i].second;
-
-    // Get the access type of this use.  If the use isn't a pointer, we don't
-    // know what it accesses.
-    Value *Address = User->getOperand(OpNo);
-    PointerType *AddrTy = dyn_cast<PointerType>(Address->getType());
-    if (!AddrTy)
-      return false;
-    Type *AddressAccessTy = AddrTy->getElementType();
-    unsigned AS = AddrTy->getAddressSpace();
+    Value *Address;
+    Type *AddressAccessTy;
+    unsigned AS;
+
+    Instruction *User = MemoryUses[i];
+    if (auto *LI = dyn_cast<LoadInst>(User)) {
+      Address = LI->getPointerOperand();
+      AddressAccessTy = LI->getType();
+      AS = LI->getPointerAddressSpace();
+    } else {
+      auto *SI = cast<StoreInst>(User);
+      Address = SI->getPointerOperand();
+      AddressAccessTy = SI->getValueOperand()->getType();
+      AS = SI->getPointerAddressSpace();
+    }
 
     // Do a match against the root of this address, ignoring profitability. This
     // will tell us if the addressing mode for the memory operation will
     // *actually* cover the shared instruction.
     ExtAddrMode Result;
     TypePromotionTransaction::ConstRestorationPt LastKnownGood =
         TPT.getRestorationPoint();
     AddressingModeMatcher Matcher(MatchedAddrModeInsts, TM, AddressAccessTy, AS,

lib/Target/AMDGPU/SITypeRewriter.cpp

bool SITypeRewriter::runOnFunction(Function &F) {
   visit(F);
 
   return false;
 }
 
 void SITypeRewriter::visitLoadInst(LoadInst &I) {
   Value *Ptr = I.getPointerOperand();
   Type *PtrTy = Ptr->getType();
-  Type *ElemTy = PtrTy->getPointerElementType();
   IRBuilder<> Builder(&I);
-  if (ElemTy == v16i8)  {
+  if (I.getType() == v16i8)  {
     Value *BitCast = Builder.CreateBitCast(Ptr,
         PointerType::get(v4i32,PtrTy->getPointerAddressSpace()));
     LoadInst *Load = Builder.CreateLoad(BitCast);
     SmallVector<std::pair<unsigned, MDNode *>, 8> MD;
     I.getAllMetadataOtherThanDebugLoc(MD);
     for (unsigned i = 0, e = MD.size(); i != e; ++i) {
       Load->setMetadata(MD[i].first, MD[i].second);
     }

lib/Target/PowerPC/PPCLoopPreIncPrep.cpp

bool PPCLoopPreIncPrep::runOnLoop(Loop *L) {
 
   // Collect buckets of comparable addresses used by loads and stores.
   SmallVector<Bucket, 16> Buckets;
   for (Loop::block_iterator I = L->block_begin(), IE = L->block_end();
        I != IE; ++I) {
     for (BasicBlock::iterator J = (*I)->begin(), JE = (*I)->end();
         J != JE; ++J) {
       Value *PtrValue;
+      Type *ValTy = nullptr;
       Instruction *MemI;
 
       if (LoadInst *LMemI = dyn_cast<LoadInst>(J)) {
         MemI = LMemI;
         PtrValue = LMemI->getPointerOperand();
+        ValTy = LMemI->getType();
       } else if (StoreInst *SMemI = dyn_cast<StoreInst>(J)) {
         MemI = SMemI;
         PtrValue = SMemI->getPointerOperand();
+        ValTy = SMemI->getValueOperand()->getType();
       } else if (IntrinsicInst *IMemI = dyn_cast<IntrinsicInst>(J)) {
         if (IMemI->getIntrinsicID() == Intrinsic::prefetch) {
           MemI = IMemI;
           PtrValue = IMemI->getArgOperand(0);
         } else continue;
       } else continue;
 
       unsigned PtrAddrSpace = PtrValue->getType()->getPointerAddressSpace();
       if (PtrAddrSpace)
         continue;
 
       // There are no update forms for Altivec vector load/stores.
-      if (ST && ST->hasAltivec() &&
-          PtrValue->getType()->getPointerElementType()->isVectorTy())
+      if (ST && ST->hasAltivec() && ValTy && ValTy->isVectorTy())
         continue;
 
       if (L->isLoopInvariant(PtrValue))
         continue;
 
       const SCEV *LSCEV = SE->getSCEVAtScope(PtrValue, L);
       if (const SCEVAddRecExpr *LARSCEV = dyn_cast<SCEVAddRecExpr>(LSCEV)) {
         if (LARSCEV->getLoop() != L)

lib/Target/X86/X86ISelLowering.cpp

 bool X86TargetLowering::shouldExpandAtomicStoreInIR(StoreInst *SI) const {
   return needsCmpXchgNb(SI->getValueOperand()->getType());
 }
 
 // Note: this turns large loads into lock cmpxchg8b/16b.
 // FIXME: On 32 bits x86, fild/movq might be faster than lock cmpxchg8b.
 TargetLowering::AtomicExpansionKind
 X86TargetLowering::shouldExpandAtomicLoadInIR(LoadInst *LI) const {
-  auto PTy = cast<PointerType>(LI->getPointerOperand()->getType());
-  return needsCmpXchgNb(PTy->getElementType()) ? AtomicExpansionKind::CmpXChg
-                                               : AtomicExpansionKind::None;
+  return needsCmpXchgNb(LI->getType())
+      ? AtomicExpansionKind::CmpXChg
 }
 
 TargetLowering::AtomicExpansionKind
 X86TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   unsigned NativeWidth = Subtarget->is64Bit() ? 64 : 32;
   Type *MemType = AI->getType();
 
   // If the operand is too big, we must see if cmpxchg8/16b is available

lib/Transforms/IPO/GlobalOpt.cpp

 }
 
 
 /// Return true if this constant is simple enough for us to understand.  In
 /// particular, if it is a cast to anything other than from one pointer type to
 /// another pointer type, we punt.  We basically just support direct accesses to
 /// globals and GEP's of globals.  This should be kept up to date with
 /// CommitValueTo.
-static bool isSimpleEnoughPointerToCommit(Constant *C) {
+static bool isSimpleEnoughPointerToCommit(Constant *C, Type *ValTy) {
   // Conservatively, avoid aggregate types. This is because we don't
   // want to worry about them partially overlapping other stores.
-  if (!cast<PointerType>(C->getType())->getElementType()->isSingleValueType())
+  if (!ValTy->isSingleValueType())
     return false;
 
   if (GlobalVariable *GV = dyn_cast<GlobalVariable>(C))
     // Do not allow weak/*_odr/linkonce linkage or external globals.
     return GV->hasUniqueInitializer();
 
   if (ConstantExpr *CE = dyn_cast<ConstantExpr>(C)) {
     // Handle a constantexpr gep.
while (1) {
 
     DEBUG(dbgs() << "Evaluating Instruction: " << *CurInst << "\n");
 
     if (StoreInst *SI = dyn_cast<StoreInst>(CurInst)) {
       if (!SI->isSimple()) {
         DEBUG(dbgs() << "Store is not simple! Can not evaluate.\n");
         return false;  // no volatile/atomic accesses.
       }
-      Constant *Ptr = getVal(SI->getOperand(1));
+      Constant *Ptr = getVal(SI->getPointerOperand());
       if (ConstantExpr *CE = dyn_cast<ConstantExpr>(Ptr)) {
         DEBUG(dbgs() << "Folding constant ptr expression: " << *Ptr);
         Ptr = ConstantFoldConstantExpression(CE, DL, TLI);
         DEBUG(dbgs() << "; To: " << *Ptr << "\n");
       }
-      if (!isSimpleEnoughPointerToCommit(Ptr)) {
+      Type *ValTy = SI->getValueOperand()->getType();
+      if (!isSimpleEnoughPointerToCommit(Ptr, ValTy)) {
         // If this is too complex for us to commit, reject it.
         DEBUG(dbgs() << "Pointer is too complex for us to evaluate store.");
         return false;
       }
 
-      Constant *Val = getVal(SI->getOperand(0));
+      Constant *Val = getVal(SI->getValueOperand());
 
       // If this might be too difficult for the backend to handle (e.g. the addr
       // of one global variable divided by another) then we can't commit it.
       if (!isSimpleEnoughValueToCommit(Val, SimpleConstants, DL)) {
         DEBUG(dbgs() << "Store value is too complex to evaluate store. " << *Val
               << "\n");
         return false;
       }

lib/Transforms/Instrumentation/ThreadSanitizer.cpp

private:
   void initializeCallbacks(Module &M);
   bool instrumentLoadOrStore(Instruction *I, const DataLayout &DL);
   bool instrumentAtomic(Instruction *I, const DataLayout &DL);
   bool instrumentMemIntrinsic(Instruction *I);
   void chooseInstructionsToInstrument(SmallVectorImpl<Instruction *> &Local,
                                       SmallVectorImpl<Instruction *> &All,
                                       const DataLayout &DL);
   bool addrPointsToConstantData(Value *Addr);
-  int getMemoryAccessFuncIndex(Value *Addr, const DataLayout &DL);
+  int getMemoryAccessFuncIndex(Type *OrigTy, const DataLayout &DL);
 
   Type *IntptrTy;
   IntegerType *OrdTy;
   // Callbacks to run-time library are computed in doInitialization.
   Function *TsanFuncEntry;
   Function *TsanFuncExit;
   // Accesses sizes are powers of two: 1, 2, 4, 8, 16.
   static const size_t kNumberOfAccessSizes = 5;

 
 bool ThreadSanitizer::instrumentLoadOrStore(Instruction *I,
                                             const DataLayout &DL) {
   IRBuilder<> IRB(I);
   bool IsWrite = isa<StoreInst>(*I);
   Value *Addr = IsWrite
       ? cast<StoreInst>(I)->getPointerOperand()
       : cast<LoadInst>(I)->getPointerOperand();
-  int Idx = getMemoryAccessFuncIndex(Addr, DL);
+  Type *OrigTy = IsWrite
+      ? cast<StoreInst>(I)->getValueOperand()->getType()
+      : cast<LoadInst>(I)->getType();
+  int Idx = getMemoryAccessFuncIndex(OrigTy, DL);
   if (Idx < 0)
     return false;
   if (IsWrite && isVtableAccess(I)) {
     DEBUG(dbgs() << "  VPTR : " << *I << "\n");
     Value *StoredValue = cast<StoreInst>(I)->getValueOperand();
     // StoredValue may be a vector type if we are storing several vptrs at once.
     // In this case, just take the first element of the vector since this is
     // enough to find vptr races.
if (!IsWrite && isVtableAccess(I)) {
     IRB.CreateCall(TsanVptrLoad,
                    IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()));
     NumInstrumentedVtableReads++;
     return true;
   }
   const unsigned Alignment = IsWrite
       ? cast<StoreInst>(I)->getAlignment()
       : cast<LoadInst>(I)->getAlignment();
-  Type *OrigTy = cast<PointerType>(Addr->getType())->getElementType();
   const uint32_t TypeSize = DL.getTypeStoreSizeInBits(OrigTy);
   Value *OnAccessFunc = nullptr;
   if (Alignment == 0 || Alignment >= 8 || (Alignment % (TypeSize / 8)) == 0)
     OnAccessFunc = IsWrite ? TsanWrite[Idx] : TsanRead[Idx];
   else
     OnAccessFunc = IsWrite ? TsanUnalignedWrite[Idx] : TsanUnalignedRead[Idx];
   IRB.CreateCall(OnAccessFunc, IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()));
   if (IsWrite) NumInstrumentedWrites++;

 // http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf
 // The following page contains more background information:
 // http://www.hpl.hp.com/personal/Hans_Boehm/c++mm/
 
 bool ThreadSanitizer::instrumentAtomic(Instruction *I, const DataLayout &DL) {
   IRBuilder<> IRB(I);
   if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
     Value *Addr = LI->getPointerOperand();
-    int Idx = getMemoryAccessFuncIndex(Addr, DL);
+    int Idx = getMemoryAccessFuncIndex(LI->getType(), DL);
     if (Idx < 0)
       return false;
     const unsigned ByteSize = 1U << Idx;
     const unsigned BitSize = ByteSize * 8;
     Type *Ty = Type::getIntNTy(IRB.getContext(), BitSize);
     Type *PtrTy = Ty->getPointerTo();
     Value *Args[] = {IRB.CreatePointerCast(Addr, PtrTy),
                      createOrdering(&IRB, LI->getOrdering())};
     CallInst *C = CallInst::Create(TsanAtomicLoad[Idx], Args);
     ReplaceInstWithInst(I, C);
 
   } else if (StoreInst *SI = dyn_cast<StoreInst>(I)) {
     Value *Addr = SI->getPointerOperand();
-    int Idx = getMemoryAccessFuncIndex(Addr, DL);
+    Value *Val = SI->getValueOperand();
+    int Idx = getMemoryAccessFuncIndex(Val->getType(), DL);
     if (Idx < 0)
       return false;
     const unsigned ByteSize = 1U << Idx;
     const unsigned BitSize = ByteSize * 8;
     Type *Ty = Type::getIntNTy(IRB.getContext(), BitSize);
     Type *PtrTy = Ty->getPointerTo();
     Value *Args[] = {IRB.CreatePointerCast(Addr, PtrTy),
-                     IRB.CreateIntCast(SI->getValueOperand(), Ty, false),
+                     IRB.CreateIntCast(Val, Ty, false),
                      createOrdering(&IRB, SI->getOrdering())};
     CallInst *C = CallInst::Create(TsanAtomicStore[Idx], Args);
     ReplaceInstWithInst(I, C);
   } else if (AtomicRMWInst *RMWI = dyn_cast<AtomicRMWInst>(I)) {
     Value *Addr = RMWI->getPointerOperand();
-    int Idx = getMemoryAccessFuncIndex(Addr, DL);
+    Value *Val = RMWI->getValOperand();
+    int Idx = getMemoryAccessFuncIndex(Val->getType(), DL);
     if (Idx < 0)
       return false;
     Function *F = TsanAtomicRMW[RMWI->getOperation()][Idx];
     if (!F)
       return false;
     const unsigned ByteSize = 1U << Idx;
     const unsigned BitSize = ByteSize * 8;
     Type *Ty = Type::getIntNTy(IRB.getContext(), BitSize);
     Type *PtrTy = Ty->getPointerTo();
     Value *Args[] = {IRB.CreatePointerCast(Addr, PtrTy),
-                     IRB.CreateIntCast(RMWI->getValOperand(), Ty, false),
+                     IRB.CreateIntCast(Val, Ty, false),
                      createOrdering(&IRB, RMWI->getOrdering())};
     CallInst *C = CallInst::Create(F, Args);
     ReplaceInstWithInst(I, C);
   } else if (AtomicCmpXchgInst *CASI = dyn_cast<AtomicCmpXchgInst>(I)) {
     Value *Addr = CASI->getPointerOperand();
-    int Idx = getMemoryAccessFuncIndex(Addr, DL);
+    Value *CmpVal = CASI->getCompareOperand();
+    int Idx = getMemoryAccessFuncIndex(CmpVal->getType(), DL);
     if (Idx < 0)
       return false;
     const unsigned ByteSize = 1U << Idx;
     const unsigned BitSize = ByteSize * 8;
     Type *Ty = Type::getIntNTy(IRB.getContext(), BitSize);
     Type *PtrTy = Ty->getPointerTo();
     Value *Args[] = {IRB.CreatePointerCast(Addr, PtrTy),
-                     IRB.CreateIntCast(CASI->getCompareOperand(), Ty, false),
+                     IRB.CreateIntCast(CmpVal, Ty, false),
                      IRB.CreateIntCast(CASI->getNewValOperand(), Ty, false),
                      createOrdering(&IRB, CASI->getSuccessOrdering()),
                      createOrdering(&IRB, CASI->getFailureOrdering())};
     CallInst *C = IRB.CreateCall(TsanAtomicCAS[Idx], Args);
-    Value *Success = IRB.CreateICmpEQ(C, CASI->getCompareOperand());
+    Value *Success = IRB.CreateICmpEQ(C, CmpVal);
 
     Value *Res = IRB.CreateInsertValue(UndefValue::get(CASI->getType()), C, 0);
     Res = IRB.CreateInsertValue(Res, Success, 1);
 
     I->replaceAllUsesWith(Res);
     I->eraseFromParent();
   } else if (FenceInst *FI = dyn_cast<FenceInst>(I)) {
     Value *Args[] = {createOrdering(&IRB, FI->getOrdering())};
     Function *F = FI->getSynchScope() == SingleThread ?
         TsanAtomicSignalFence : TsanAtomicThreadFence;
     CallInst *C = CallInst::Create(F, Args);
     ReplaceInstWithInst(I, C);
   }
   return true;
 }
 
-int ThreadSanitizer::getMemoryAccessFuncIndex(Value *Addr,
+int ThreadSanitizer::getMemoryAccessFuncIndex(Type *OrigTy,
                                               const DataLayout &DL) {
-  Type *OrigPtrTy = Addr->getType();
-  Type *OrigTy = cast<PointerType>(OrigPtrTy)->getElementType();
   assert(OrigTy->isSized());
   uint32_t TypeSize = DL.getTypeStoreSizeInBits(OrigTy);
   if (TypeSize != 8  && TypeSize != 16 &&
       TypeSize != 32 && TypeSize != 64 && TypeSize != 128) {
     NumAccessesWithBadSize++;
     // Ignore all unusual sizes.
     return -1;
   }
   size_t Idx = countTrailingZeros(TypeSize / 8);
   assert(Idx < kNumberOfAccessSizes);
   return Idx;
 }

lib/Transforms/Scalar/MemCpyOptimizer.cpp

struct MemsetRange {
   // Start/End - A semi range that describes the span that this range covers.
   // The range is closed at the start and open at the end: [Start, End).
   int64_t Start, End;
 
   /// StartPtr - The getelementptr instruction that points to the start of the
   /// range.
   Value *StartPtr;
 
+  /// DestTy - The type used for writing to the start of the range.
+  Type *DestTy;
+
   /// Alignment - The known alignment of the first store.
   unsigned Alignment;
 
   /// TheStores - The actual stores that make up this range.
   SmallVector<Instruction*, 16> TheStores;
 
   bool isProfitableToUseMemset(const DataLayout &DL) const;
 };
public:
   void addInst(int64_t OffsetFromFirst, Instruction *Inst) {
     if (StoreInst *SI = dyn_cast<StoreInst>(Inst))
       addStore(OffsetFromFirst, SI);
     else
       addMemSet(OffsetFromFirst, cast<MemSetInst>(Inst));
   }
 
   void addStore(int64_t OffsetFromFirst, StoreInst *SI) {
-    int64_t StoreSize = DL.getTypeStoreSize(SI->getOperand(0)->getType());
+    Type *DestTy = SI->getValueOperand()->getType();
+    int64_t StoreSize = DL.getTypeStoreSize(DestTy);
 
     addRange(OffsetFromFirst, StoreSize,
-             SI->getPointerOperand(), SI->getAlignment(), SI);
+             SI->getPointerOperand(), DestTy,
+             SI->getAlignment(), SI);
   }
 
   void addMemSet(int64_t OffsetFromFirst, MemSetInst *MSI) {
+    Type *DestTy = Type::getInt8Ty(MSI->getContext());
     int64_t Size = cast<ConstantInt>(MSI->getLength())->getZExtValue();
-    addRange(OffsetFromFirst, Size, MSI->getDest(), MSI->getAlignment(), MSI);
+
+    addRange(OffsetFromFirst, Size, MSI->getDest(),
+             DestTy, MSI->getAlignment(), MSI);
   }
 
-  void addRange(int64_t Start, int64_t Size, Value *Ptr,
+  void addRange(int64_t Start, int64_t Size,
+                Value *Ptr, Type *DestTy,
                 unsigned Alignment, Instruction *Inst);
 
 };
 
 } // end anon namespace
 
 
 /// Add a new store to the MemsetRanges data structure.  This adds a
 /// new range for the specified store at the specified offset, merging into
 /// existing ranges as appropriate.
-void MemsetRanges::addRange(int64_t Start, int64_t Size, Value *Ptr,
+void MemsetRanges::addRange(int64_t Start, int64_t Size,
+                            Value *Ptr, Type *DestTy,
                             unsigned Alignment, Instruction *Inst) {
   int64_t End = Start+Size;
 
   range_iterator I = std::lower_bound(Ranges.begin(), Ranges.end(), Start,
     [](const MemsetRange &LHS, int64_t RHS) { return LHS.End < RHS; });
 
   // We now know that I == E, in which case we didn't find anything to merge
   // with, or that Start <= I->End.  If End < I->Start or I == E, then we need
   // to insert a new range.  Handle this now.
   if (I == Ranges.end() || End < I->Start) {
     MemsetRange &R = *Ranges.insert(I, MemsetRange());
     R.Start        = Start;
     R.End          = End;
     R.StartPtr     = Ptr;
+    R.DestTy       = DestTy;
     R.Alignment    = Alignment;
     R.TheStores.push_back(Inst);
     return;
   }
 
   // This store overlaps with I, add it.
   I->TheStores.push_back(Inst);
 
   // At this point, we may have an interval that completely contains our store.
   // If so, just add it to the interval and return.
   if (I->Start <= Start && I->End >= End)
     return;
 
   // Now we know that Start <= I->End and End >= I->Start so the range overlaps
   // but is not entirely contained within the range.
 
   // See if the range extends the start of the range.  In this case, it couldn't
   // possibly cause it to join the prior range, because otherwise we would have
   // stopped on *it*.
   if (Start < I->Start) {
     I->Start = Start;
     I->StartPtr = Ptr;
+    I->DestTy = DestTy;
     I->Alignment = Alignment;
   }
 
   // Now we know that Start <= I->End and Start >= I->Start (so the startpoint
   // is in or right at the end of I), and that End >= I->Start.  Extend I out to
   // End.
   if (End > I->End) {
     I->End = End;
if (!Range.isProfitableToUseMemset(DL))
       continue;
 
     // Otherwise, we do want to transform this!  Create a new memset.
     // Get the starting pointer of the block.
     StartPtr = Range.StartPtr;
 
     // Determine alignment
     unsigned Alignment = Range.Alignment;
-    if (Alignment == 0) {
-      Type *EltType =
-        cast<PointerType>(StartPtr->getType())->getElementType();
-      Alignment = DL.getABITypeAlignment(EltType);
-    }
+    if (Alignment == 0)
+      Alignment = DL.getABITypeAlignment(Range.DestTy);
 
     AMemSet =
       Builder.CreateMemSet(StartPtr, ByteVal, Range.End-Range.Start, Alignment);
 
     DEBUG(dbgs() << "Replace stores:\n";
           for (Instruction *SI : Range.TheStores)
             dbgs() << *SI << '\n';
           dbgs() << "With: " << *AMemSet << '\n');

lib/Transforms/Scalar/SROA.cpp

if (MemIntrinsic *MI = dyn_cast<MemIntrinsic>(U->getUser())) {
     if (MI->isVolatile())
       return false;
     if (!S.isSplittable())
       return false; // Skip any unsplittable intrinsics.
   } else if (IntrinsicInst *II = dyn_cast<IntrinsicInst>(U->getUser())) {
     if (II->getIntrinsicID() != Intrinsic::lifetime_start &&
         II->getIntrinsicID() != Intrinsic::lifetime_end)
       return false;
-  } else if (U->get()->getType()->getPointerElementType()->isStructTy()) {
-    // Disable vector promotion when there are loads or stores of an FCA.
-    return false;
   } else if (LoadInst *LI = dyn_cast<LoadInst>(U->getUser())) {
     if (LI->isVolatile())
       return false;
     Type *LTy = LI->getType();
+    // Disable vector promotion when there are loads or stores of an FCA.
+    if (LTy->isStructTy())
+      return false;
     if (P.beginOffset() > S.beginOffset() || P.endOffset() < S.endOffset()) {
       assert(LTy->isIntegerTy());
       LTy = SplitIntTy;
     }
     if (!canConvertValue(DL, SliceTy, LTy))
       return false;
   } else if (StoreInst *SI = dyn_cast<StoreInst>(U->getUser())) {
     if (SI->isVolatile())
       return false;
     Type *STy = SI->getValueOperand()->getType();
+    // Disable vector promotion when there are loads or stores of an FCA.
+    if (STy->isStructTy())
+      return false;
     if (P.beginOffset() > S.beginOffset() || P.endOffset() < S.endOffset()) {
       assert(STy->isIntegerTy());
       STy = SplitIntTy;
     }
     if (!canConvertValue(DL, STy, SliceTy))
       return false;
   } else {
     return false;

lib/Transforms/Scalar/Scalarizer.cpp

 class Scatterer {
 public:
   Scatterer() {}
 
   // Scatter V into Size components.  If new instructions are needed,
   // insert them before BBI in BB.  If Cache is nonnull, use it to cache
   // the results.
   Scatterer(BasicBlock *bb, BasicBlock::iterator bbi, Value *v,
-            ValueVector *cachePtr = nullptr);
+            Type *VecTy, ValueVector *cachePtr = nullptr);
 
   // Return component I, creating a new Value for it if necessary.
   Value *operator[](unsigned I);
 
   // Return the number of components.
   unsigned size() const { return Size; }
 
 private:
static void registerOptions() {
     // reached.
     OptionRegistry::registerOption<bool, Scalarizer,
                                  &Scalarizer::ScalarizeLoadStore>(
         "scalarize-load-store",
         "Allow the scalarizer pass to scalarize loads and store", false);
   }
 
 private:
-  Scatterer scatter(Instruction *, Value *);
+  Scatterer scatter(Instruction *, Value *, Type *VecTy = nullptr);
   void gather(Instruction *, const ValueVector &);
   bool canTransferMetadata(unsigned Kind);
   void transferMetadata(Instruction *, const ValueVector &);
   bool getVectorLayout(Type *, unsigned, VectorLayout &, const DataLayout &);
   bool finish();
 
   template<typename T> bool splitBinary(Instruction &, const T &);
 
   ScatterMap Scattered;
   GatherList Gathered;
   unsigned ParallelLoopAccessMDKind;
   bool ScalarizeLoadStore;
 };
 
 char Scalarizer::ID = 0;
 } // end anonymous namespace
 
 INITIALIZE_PASS_WITH_OPTIONS(Scalarizer, "scalarizer",
                              "Scalarize vector operations", false, false)
 
 Scatterer::Scatterer(BasicBlock *bb, BasicBlock::iterator bbi, Value *v,
-                     ValueVector *cachePtr)
+                     Type *VecTy, ValueVector *cachePtr)
   : BB(bb), BBI(bbi), V(v), CachePtr(cachePtr) {
-  Type *Ty = V->getType();
-  PtrTy = dyn_cast<PointerType>(Ty);
-  if (PtrTy)
-    Ty = PtrTy->getElementType();
-  Size = Ty->getVectorNumElements();
+  PtrTy = dyn_cast<PointerType>(V->getType());
+  Size = VecTy->getVectorNumElements();
   if (!CachePtr)
     Tmp.resize(Size, nullptr);
   else if (CachePtr->empty())
     CachePtr->resize(Size, nullptr);
   else
     assert(Size == CachePtr->size() && "Inconsistent vector sizes");
 }
 
for (BasicBlock::iterator II = BB.begin(), IE = BB.end(); II != IE;) {
         I->eraseFromParent();
     }
   }
   return finish();
 }
 
 // Return a scattered form of V that can be accessed by Point.  V must be a
 // vector or a pointer to a vector.
-Scatterer Scalarizer::scatter(Instruction *Point, Value *V) {
+Scatterer Scalarizer::scatter(Instruction *Point, Value *V, Type *VecTy) {
+  if (!VecTy)
+    VecTy = V->getType();
   if (Argument *VArg = dyn_cast<Argument>(V)) {
     // Put the scattered form of arguments in the entry block,
     // so that it can be used everywhere.
     Function *F = VArg->getParent();
     BasicBlock *BB = &F->getEntryBlock();
-    return Scatterer(BB, BB->begin(), V, &Scattered[V]);
+    return Scatterer(BB, BB->begin(), V, VecTy, &Scattered[V]);
   }
   if (Instruction *VOp = dyn_cast<Instruction>(V)) {
     // Put the scattered form of an instruction directly after the
     // instruction.
     BasicBlock *BB = VOp->getParent();
     return Scatterer(BB, std::next(BasicBlock::iterator(VOp)),
-                     V, &Scattered[V]);
+                     V, VecTy, &Scattered[V]);
   }
   // In the fallback case, just put the scattered before Point and
   // keep the result local to Point.
-  return Scatterer(Point->getParent(), Point->getIterator(), V);
+  return Scatterer(Point->getParent(), Point->getIterator(), V, VecTy);
 }
 
 // Replace Op with the gathered form of the components in CV.  Defer the
 // deletion of Op and creation of the gathered form to the end of the pass,
 // so that we can avoid creating the gathered form if all uses of Op are
 // replaced with uses of CV.
 void Scalarizer::gather(Instruction *Op, const ValueVector &CV) {
   // Since we're not deleting Op yet, stub out its operands, so that it
bool Scalarizer::visitLoadInst(LoadInst &LI) {
 
   VectorLayout Layout;
   if (!getVectorLayout(LI.getType(), LI.getAlignment(), Layout,
                        LI.getModule()->getDataLayout()))
     return false;
 
   unsigned NumElems = Layout.VecTy->getNumElements();
   IRBuilder<> Builder(&LI);
-  Scatterer Ptr = scatter(&LI, LI.getPointerOperand());
+  Scatterer Ptr = scatter(&LI, LI.getPointerOperand(), LI.getType());
   ValueVector Res;
   Res.resize(NumElems);
 
   for (unsigned I = 0; I < NumElems; ++I)
     Res[I] = Builder.CreateAlignedLoad(Ptr[I], Layout.getElemAlign(I),
                                        LI.getName() + ".i" + Twine(I));
   gather(&LI, Res);
   return true;
 }
 
 bool Scalarizer::visitStoreInst(StoreInst &SI) {
   if (!ScalarizeLoadStore)
     return false;
   if (!SI.isSimple())
     return false;
 
   VectorLayout Layout;
   Value *FullValue = SI.getValueOperand();
-  if (!getVectorLayout(FullValue->getType(), SI.getAlignment(), Layout,
+  Type *ValTy = FullValue->getType();
+  if (!getVectorLayout(ValTy, SI.getAlignment(), Layout,
                        SI.getModule()->getDataLayout()))
     return false;
 
   unsigned NumElems = Layout.VecTy->getNumElements();
   IRBuilder<> Builder(&SI);
-  Scatterer Ptr = scatter(&SI, SI.getPointerOperand());
+  Scatterer Ptr = scatter(&SI, SI.getPointerOperand(), ValTy);
   Scatterer Val = scatter(&SI, FullValue);
 
   ValueVector Stores;
   Stores.resize(NumElems);
   for (unsigned I = 0; I < NumElems; ++I) {
     unsigned Align = Layout.getElemAlign(I);
     Stores[I] = Builder.CreateAlignedStore(Val[I], Ptr[I], Align);
   }

lib/Transforms/Vectorize/BBVectorize.cpp

void computePairsConnectedTo(
     }
 
     // This determines the relative offset of two loads or stores, returning
     // true if the offset could be determined to be some constant value.
     // For example, if OffsetInElmts == 1, then J accesses the memory directly
     // after I; if OffsetInElmts == -1 then I accesses the memory
     // directly after J.
     bool getPairPtrInfo(Instruction *I, Instruction *J,
-        Value *&IPtr, Value *&JPtr, unsigned &IAlignment, unsigned &JAlignment,
+        Value *&IPtr, Value *&JPtr,
+        Type *&ITy, Type *&JTy,
+        unsigned &IAlignment, unsigned &JAlignment,
         unsigned &IAddressSpace, unsigned &JAddressSpace,
         int64_t &OffsetInElmts, bool ComputeOffset = true) {
       OffsetInElmts = 0;
       if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
         LoadInst *LJ = cast<LoadInst>(J);
         IPtr = LI->getPointerOperand();
         JPtr = LJ->getPointerOperand();
+        ITy = LI->getType();
+        JTy = LJ->getType();
         IAlignment = LI->getAlignment();
         JAlignment = LJ->getAlignment();
         IAddressSpace = LI->getPointerAddressSpace();
         JAddressSpace = LJ->getPointerAddressSpace();
       } else {
         StoreInst *SI = cast<StoreInst>(I), *SJ = cast<StoreInst>(J);
         IPtr = SI->getPointerOperand();
         JPtr = SJ->getPointerOperand();
+        ITy = SI->getValueOperand()->getType();
+        JTy = SJ->getValueOperand()->getType();
         IAlignment = SI->getAlignment();
         JAlignment = SJ->getAlignment();
         IAddressSpace = SI->getPointerAddressSpace();
         JAddressSpace = SJ->getPointerAddressSpace();
       }
 
       if (!ComputeOffset)
         return true;
 
       const SCEV *IPtrSCEV = SE->getSCEV(IPtr);
       const SCEV *JPtrSCEV = SE->getSCEV(JPtr);
 
       // If this is a trivial offset, then we'll get something like
       // 1*sizeof(type). With target data, which we need anyway, this will get
       // constant folded into a number.
       const SCEV *OffsetSCEV = SE->getMinusSCEV(JPtrSCEV, IPtrSCEV);
       if (const SCEVConstant *ConstOffSCEV =
             dyn_cast<SCEVConstant>(OffsetSCEV)) {
         ConstantInt *IntOff = ConstOffSCEV->getValue();
         int64_t Offset = IntOff->getSExtValue();
         const DataLayout &DL = I->getModule()->getDataLayout();
-        Type *VTy = IPtr->getType()->getPointerElementType();
-        int64_t VTyTSS = (int64_t)DL.getTypeStoreSize(VTy);
+        int64_t VTyTSS = (int64_t)DL.getTypeStoreSize(ITy);
 
-        Type *VTy2 = JPtr->getType()->getPointerElementType();
-        if (VTy != VTy2 && Offset < 0) {
-          int64_t VTy2TSS = (int64_t)DL.getTypeStoreSize(VTy2);
+        if (ITy != JTy && Offset < 0) {
+          int64_t VTy2TSS = (int64_t)DL.getTypeStoreSize(JTy);
           OffsetInElmts = Offset/VTy2TSS;
           return (std::abs(Offset) % VTy2TSS) == 0;
         }
 
         OffsetInElmts = Offset/VTyTSS;
         return (std::abs(Offset) % VTyTSS) == 0;
       }
 
unsigned MaxTypeBits = std::max(
       IT2->getPrimitiveSizeInBits() + JT2->getPrimitiveSizeInBits());
     if (!TTI && MaxTypeBits > Config.VectorBits)
       return false;
 
     // FIXME: handle addsub-type operations!
 
     if (IsSimpleLoadStore) {
       Value *IPtr, *JPtr;
+      Type *aTypeI, *aTypeJ;
       unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
       int64_t OffsetInElmts = 0;
-      if (getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
+      if (getPairPtrInfo(I, J, IPtr, JPtr, aTypeI, aTypeJ, IAlignment, JAlignment,
                          IAddressSpace, JAddressSpace, OffsetInElmts) &&
           std::abs(OffsetInElmts) == 1) {
         FixedOrder = (int) OffsetInElmts;
         unsigned BottomAlignment = IAlignment;
         if (OffsetInElmts < 0) BottomAlignment = JAlignment;
 
-        Type *aTypeI = isa<StoreInst>(I) ?
-          cast<StoreInst>(I)->getValueOperand()->getType() : I->getType();
-        Type *aTypeJ = isa<StoreInst>(J) ?
-          cast<StoreInst>(J)->getValueOperand()->getType() : J->getType();
         Type *VType = getVecTypeForPair(aTypeI, aTypeJ);
 
         if (Config.AlignedOnly) {
           // An aligned load or store is possible only if the instruction
           // with the lower offset has an alignment suitable for the
           // vector type.
           const DataLayout &DL = I->getModule()->getDataLayout();
           unsigned VecAlignment = DL.getPrefTypeAlignment(VType);
return (I->getName() + (IsInput ? ".v.i" : ".v.r") + utostr(o) +
              (n > 0 ? "." + utostr(n) : "")).str();
   }
 
   // Returns the value that is to be used as the pointer input to the vector
   // instruction that fuses I with J.
   Value *BBVectorize::getReplacementPointerInput(LLVMContext& Context,
                      Instruction *I, Instruction *J, unsigned o) {
     Value *IPtr, *JPtr;
+    Type *ArgTypeI, *ArgTypeJ;
     unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
     int64_t OffsetInElmts;
 
     // Note: the analysis might fail here, that is why the pair order has
     // been precomputed (OffsetInElmts must be unused here).
-    (void) getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
+    (void) getPairPtrInfo(I, J, IPtr, JPtr, ArgTypeI, ArgTypeJ,
+                          IAlignment, JAlignment,
                           IAddressSpace, JAddressSpace,
                           OffsetInElmts, false);
 
     // The pointer value is taken to be the one with the lowest offset.
     Value *VPtr = IPtr;
-
-    Type *ArgTypeI = IPtr->getType()->getPointerElementType();
-    Type *ArgTypeJ = JPtr->getType()->getPointerElementType();
     Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
     Type *VArgPtrType
       = PointerType::get(VArgType,
                          IPtr->getType()->getPointerAddressSpace());
     return new BitCastInst(VPtr, VArgPtrType, getReplacementName(I, true, o),
                         /* insert before */ I);
   }
 

lib/Transforms/Vectorize/LoopVectorize.cpp

public:
   /// Check if this  pointer is consecutive when vectorizing. This happens
   /// when the last index of the GEP is the induction variable, or that the
   /// pointer itself is an induction variable.
   /// This check allows us to vectorize A[idx] into a wide load/store.
   /// Returns:
   /// 0 - Stride is unknown or non-consecutive.
   /// 1 - Address is consecutive.
   /// -1 - Address is consecutive, and decreasing.
-  int isConsecutivePtr(Value *Ptr);
+  int isConsecutivePtr(Value *Ptr, Type *DataType = nullptr);
 
   /// Returns true if the value V is uniform within the loop.
   bool isUniform(Value *V);
 
   /// Returns true if this instruction will remain scalar after vectorization.
   bool isUniformAfterVectorization(Instruction* I) { return Uniforms.count(I); }
 
   /// Returns the information that we collected about runtime memory check.
public:
   SmallPtrSet<Value *, 8>::iterator strides_begin() {
     return StrideSet.begin();
   }
   SmallPtrSet<Value *, 8>::iterator strides_end() { return StrideSet.end(); }
 
   /// Returns true if the target machine supports masked store operation
   /// for the given \p DataType and kind of access to \p Ptr.
   bool isLegalMaskedStore(Type *DataType, Value *Ptr) {
-    return isConsecutivePtr(Ptr) && TTI->isLegalMaskedStore(DataType);
+    return isConsecutivePtr(Ptr, DataType) && TTI->isLegalMaskedStore(DataType);
   }
   /// Returns true if the target machine supports masked load operation
   /// for the given \p DataType and kind of access to \p Ptr.
   bool isLegalMaskedLoad(Type *DataType, Value *Ptr) {
-    return isConsecutivePtr(Ptr) && TTI->isLegalMaskedLoad(DataType);
+    return isConsecutivePtr(Ptr, DataType) && TTI->isLegalMaskedLoad(DataType);
   }
   /// Returns true if vector representation of the instruction \p I
   /// requires mask.
   bool isMaskRequired(const Instruction* I) {
     return (MaskedOp.count(I) != 0);
   }
   unsigned getNumStores() const {
     return LAI->getNumStores();
Value *InnerLoopVectorizer::getStepVector(Value *Val, int StartIdx,
   Step = Builder.CreateVectorSplat(VLen, Step);
   assert(Step->getType() == Val->getType() && "Invalid step vec");
   // FIXME: The newly created binary instructions should contain nsw/nuw flags,
   // which can be found from the original scalar operations.
   Step = Builder.CreateMul(Cv, Step);
   return Builder.CreateAdd(Val, Step, "induction");
 }
 
-int LoopVectorizationLegality::isConsecutivePtr(Value *Ptr) {
+int LoopVectorizationLegality::isConsecutivePtr(Value *Ptr, Type *DataType) {
   assert(Ptr->getType()->isPointerTy() && "Unexpected non-ptr");
   auto *SE = PSE.getSE();
   // Make sure that the pointer does not point to structs.
-  if (Ptr->getType()->getPointerElementType()->isAggregateType())
+  if (DataType && DataType->isAggregateType())
     return 0;
 
   // If this value is a pointer induction variable we know it is consecutive.
   PHINode *Phi = dyn_cast_or_null<PHINode>(Ptr);
   if (Phi && Inductions.count(Phi)) {
     InductionDescriptor II = Inductions[Phi];
     return II.getConsecutiveDirection();
   }
 
   GetElementPtrInst *Gep = getGEPInstruction(Ptr);
   if (!Gep)
     return 0;
 
   unsigned NumOperands = Gep->getNumOperands();
   Value *GpPtr = Gep->getPointerOperand();
   // If this GEP value is a consecutive pointer induction variable and all of
   // the indices are constant then we know it is consecutive. We can
   Phi = dyn_cast<PHINode>(GpPtr);
   if (Phi && Inductions.count(Phi)) {
 
     // Make sure that the pointer does not point to structs.
-    PointerType *GepPtrType = cast<PointerType>(GpPtr->getType());
-    if (GepPtrType->getElementType()->isAggregateType())
+    if (Gep->getSourceElementType()->isAggregateType())
       return 0;
 
     // Make sure that all of the index operands are loop invariant.
     for (unsigned i = 1; i < NumOperands; ++i)
       if (!SE->isLoopInvariant(PSE.getSCEV(Gep->getOperand(i)), TheLoop))
         return 0;
 
     InductionDescriptor II = Inductions[Phi];
if (SI && Legal->blockNeedsPredication(SI->getParent()) &&
       !Legal->isMaskRequired(SI))
     return scalarizeInstruction(Instr, true);
 
   if (ScalarAllocatedSize != VectorElementSize)
     return scalarizeInstruction(Instr);
 
   // If the pointer is loop invariant or if it is non-consecutive,
   // scalarize the load.
-  int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);
+  int ConsecutiveStride = Legal->isConsecutivePtr(Ptr, ScalarDataTy);
   bool Reverse = ConsecutiveStride < 0;
   bool UniformLoad = LI && Legal->isUniform(Ptr);
   if (!ConsecutiveStride || UniformLoad)
     return scalarizeInstruction(Instr);
 
   Constant *Zero = Builder.getInt32(0);
   VectorParts &Entry = WidenMap.get(Instr);
 
void InterleavedAccessInfo::collectConstStridedAccesses(
 
   auto &DL = TheLoop->getHeader()->getModule()->getDataLayout();
   for (auto I : AccessList) {
     LoadInst *LI = dyn_cast<LoadInst>(I);
     StoreInst *SI = dyn_cast<StoreInst>(I);
 
     Value *Ptr = LI ? LI->getPointerOperand() : SI->getPointerOperand();
     int Stride = isStridedPtr(PSE, Ptr, TheLoop, Strides);
+    Type *AccessTy = LI ? LI->getType() : SI->getValueOperand()->getType();
 
     // The factor of the corresponding interleave group.
     unsigned Factor = std::abs(Stride);
 
     // Ignore the access if the factor is too small or too large.
     if (Factor < 2 || Factor > MaxInterleaveGroupFactor)
       continue;
 
     const SCEV *Scev = replaceSymbolicStrideSCEV(PSE, Strides, Ptr);
-    PointerType *PtrTy = dyn_cast<PointerType>(Ptr->getType());
-    unsigned Size = DL.getTypeAllocSize(PtrTy->getElementType());
+    unsigned Size = DL.getTypeAllocSize(AccessTy);
 
     // An alignment of 0 means target ABI alignment.
     unsigned Align = LI ? LI->getAlignment() : SI->getAlignment();
     if (!Align)
-      Align = DL.getABITypeAlignment(PtrTy->getElementType());
+      Align = DL.getABITypeAlignment(AccessTy);
 
     StrideAccesses[I] = StrideDescriptor(Stride, Scev, Size, Align);
   }
 }
 
 // Analyze interleaved accesses and collect them into interleave groups.
 //
 // Notice that the vectorization on interleaved groups will change instruction
if (Legal->isAccessInterleaved(I)) {
 
       // FIXME: The interleaved load group with a huge gap could be even more
       // expensive than scalar operations. Then we could ignore such group and
       // use scalar operations instead.
       return Cost;
     }
 
     // Scalarized loads/stores.
-    int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);
+    int ConsecutiveStride = Legal->isConsecutivePtr(Ptr, ValTy);
     bool Reverse = ConsecutiveStride < 0;
     const DataLayout &DL = I->getModule()->getDataLayout();
     unsigned ScalarAllocatedSize = DL.getTypeAllocSize(ValTy);
     unsigned VectorElementSize = DL.getTypeStoreSize(VectorTy) / VF;
     if (!ConsecutiveStride || ScalarAllocatedSize != VectorElementSize) {
       bool IsComplexComputation =
         isLikelyComplexAddressComputation(Ptr, Legal, SE, TheLoop);
       unsigned Cost = 0;

 namespace llvm {
   Pass *createLoopVectorizePass(bool NoUnrolling, bool AlwaysVectorize) {
     return new LoopVectorize(NoUnrolling, AlwaysVectorize);
   }
 }
 
 bool LoopVectorizationCostModel::isConsecutiveLoadOrStore(Instruction *Inst) {
   // Check for a store.
-  if (StoreInst *ST = dyn_cast<StoreInst>(Inst))
-    return Legal->isConsecutivePtr(ST->getPointerOperand()) != 0;
+  if (StoreInst *ST = dyn_cast<StoreInst>(Inst)) {
+    Type *Ty = ST->getValueOperand()->getType();
+    return Legal->isConsecutivePtr(ST->getPointerOperand(), Ty) != 0;
+  }
 
   // Check for a load.
   if (LoadInst *LI = dyn_cast<LoadInst>(Inst))
-    return Legal->isConsecutivePtr(LI->getPointerOperand()) != 0;
+    return Legal->isConsecutivePtr(LI->getPointerOperand(), LI->getType()) != 0;
 
   return false;
 }
 
 void LoopVectorizationCostModel::collectValuesToIgnore() {
   // Ignore ephemeral values.
   CodeMetrics::collectEphemeralValues(TheLoop, AC, ValuesToIgnore);
 

lib/Transforms/Vectorize/SLPVectorizer.cpp

private:
   /// \returns the pointer to the vectorized value if \p VL is already
   /// vectorized, or NULL. They may happen in cycles.
   Value *alreadyVectorized(ArrayRef<Value *> VL) const;
 
   /// \brief Take the pointer operand from the Load/Store instruction.
   /// \returns NULL if this is not a valid Load/Store instruction.
   static Value *getPointerOperand(Value *I);
 
+  /// \brief Take the acccessed type from the Load/Store instruction.
+  /// \returns NULL if this is not a valid Load/Store instruction.
+  static Type *getAccessType(Value *I);
+
   /// \brief Take the address space operand from the Load/Store instruction.
   /// \returns -1 if this is not a valid Load/Store instruction.
   static unsigned getAddressSpaceOperand(Value *I);
 
   /// \returns the scalarization cost for this type. Scalarization in this
   /// context means the creation of vectors from a group of scalars.
   int getGatherCost(Type *Ty);
 

 Value *BoUpSLP::getPointerOperand(Value *I) {
   if (LoadInst *LI = dyn_cast<LoadInst>(I))
     return LI->getPointerOperand();
   if (StoreInst *SI = dyn_cast<StoreInst>(I))
     return SI->getPointerOperand();
   return nullptr;
 }
 
+Type *BoUpSLP::getAccessType(Value *I) {
+  if (LoadInst *LI = dyn_cast<LoadInst>(I))
+    return LI->getType();
+  if (StoreInst *SI = dyn_cast<StoreInst>(I))
+    return SI->getValueOperand()->getType();
+  return nullptr;
+}
+
 unsigned BoUpSLP::getAddressSpaceOperand(Value *I) {
   if (LoadInst *L = dyn_cast<LoadInst>(I))
     return L->getPointerAddressSpace();
   if (StoreInst *S = dyn_cast<StoreInst>(I))
     return S->getPointerAddressSpace();
   return -1;
 }
 
 bool BoUpSLP::isConsecutiveAccess(Value *A, Value *B, const DataLayout &DL) {
   Value *PtrA = getPointerOperand(A);
   Value *PtrB = getPointerOperand(B);
   unsigned ASA = getAddressSpaceOperand(A);
   unsigned ASB = getAddressSpaceOperand(B);
 
   // Check that the address spaces match and that the pointers are valid.
   if (!PtrA || !PtrB || (ASA != ASB))
     return false;
 
   // Make sure that A and B are different pointers of the same type.
-  if (PtrA == PtrB || PtrA->getType() != PtrB->getType())
+  Type *TyA = getAccessType(A);
+  Type *TyB = getAccessType(B);
+  if (PtrA == PtrB || TyA != TyB)
     return false;
 
   unsigned PtrBitWidth = DL.getPointerSizeInBits(ASA);
-  Type *Ty = cast<PointerType>(PtrA->getType())->getElementType();
+  Type *Ty = TyA;
   APInt Size(PtrBitWidth, DL.getTypeStoreSize(Ty));
 
   APInt OffsetA(PtrBitWidth, 0), OffsetB(PtrBitWidth, 0);
   PtrA = PtrA->stripAndAccumulateInBoundsConstantOffsets(DL, OffsetA);
   PtrB = PtrB->stripAndAccumulateInBoundsConstantOffsets(DL, OffsetB);
 
   APInt OffsetDelta = OffsetB - OffsetA;