[LoopLoadElim] Add stores with matching sizes as load-store candidates

[jolanta.jensen]

Summary

We are not building up a proper list of load-store candidates because
we are throwing away stores where the type don't match the load.
This patch adds stores with matching store sizes as candidates.
Author of the original patch: David Sherwood.

[nikic]

Not all types are bitcastable. Please add tests where the load/store are pointer and integer (in which case you need inttoptr or ptrtoint -- and watch out for non-integral address spaces) and one where an aggregate is involved (cannot be bitcast at all).

[mgabka]

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

220

is it worth to make this work also for vector types? the patch you proposed does not work when there is casting between scalar and vector type of the same size for example i32 and <2 x i16>, would it be useful to extend it to cover such cases?

[david-arm]

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

220

I don't think you can get such bitcasts, but also I think it's not even safe for big endian due to the different memory layout.

[jolanta.jensen]

1. Guarding against non-valid bitcasts.
2. Load-store forwarding works with vector types as well.
3. Added a test for load-store forwarding with vector types.
4. Added tests confirming bitcasting vectors won't corrupt the values stored.

[jolanta.jensen]

In D130233#3667850, @nikic wrote:

Not all types are bitcastable. Please add tests where the load/store are pointer and integer (in which case you need inttoptr or ptrtoint -- and watch out for non-integral address spaces) and one where an aggregate is involved (cannot be bitcast at all).

@nikic , Thank you for a quick review and apologies for the delay addressing them. I'm now guarding against non-valid bitcasts. For inttoptr or ptrtoint request, f3 in type-mismatch-opaque-ptr.ll has such a case. Do you want more extensive testing than that?

[mgabka]

llvm/test/Transforms/EarlyCSE/vector_bitcasting_be.ll

1 ↗

(On Diff #450785)

I think a good idea would be to run directly only the passes you actually want to test instead of using O3

9 ↗

(On Diff #450785)

can you just pass val1 as argument to this function and remove the alloca?

same request about f2 function and the other file with tests that you added.

[nikic]

llvm/test/Transforms/EarlyCSE/vector_bitcasting_be.ll

1 ↗

(On Diff #450785)

The test should also be in the LoopLoadElim directory rather than EarlyCSE, and use update_test_checks.py.

llvm/test/Transforms/LoopLoadElim/type-mismatch.ll

144

Eh, while that happens to work out okay here, please do replace all your getPrimitiveSizeInBits() calls with getTypeSizeInBits() on DataLayout.

[david-arm]

llvm/test/Transforms/LoopLoadElim/type-mismatch.ll

147

Hi @jolanta.jensen, I think it would be good if you could pre-commit these tests separately in a different patch, then rebase your changes on top. That way we can see the change in output and it makes it easier to see what effect the patch has.

Also, in the same pre-commit patch it would be good if you could generate the CHECK lines with the update_test_checks.py script because I find it a bit hard to see what's going on with partial CHECK lines, mixed between lines of IR.

Thanks!

[jolanta.jensen]

1. Correcting errand assumption that the size of a pointer is 0.
2. Removing vector bitcasting tests as they were adding more confusion than value.
3. Regenerating CHECK lines with update_test_checks.py script.

[jolanta.jensen]

Allowing integer-sized pointers to be forwarded.

Also updating the patch to be based on D132239 and addressing review comments.

[nikic]

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

245

I believe the condition you are looking for is CastInst::isBitOrNoopPointerCastable(StoreTy, LoadTy, DL). That should ensure that one of bitcast, inttoptr or ptrtoint is legal for these types.

487

In this whole block, I believe you are looking for CastInst::CreateBitOrPointerCast().

[jolanta.jensen]

Simplified the code in accordance with review comments.

[nikic]

Looks about right, some nits

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

119

Move this DL variable before the assert and reuse.

224

Just the isBitOrNoopPointerCastable check is enough, it implies that the sizes are the same as well.

457

Move this declaration lower, before the assignment.

467

Storing DataLayout in a variable would make this more compact.

[jolanta.jensen]

Code clean up after review comments.

[fhahn]

LGTM, thanks! Some additional suggestions inline.

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

222

nit: reflow comment, also now that you use isBitOrNoopPointerCastable it would probably be easier to say that stored values are bit/pointer-castable.

458

nit: getStoredValue() would be more explicit than getOperand(0).

459

Looks like this is only used by the assert. If that's the case, you'll need (void)DL; to avoid unused variable warning in release builds.

465

nit: stray new line?

471

Nit: could change to

Value *StoreValue = Cand.Store->getOperand(0);` 
if (...)
   StoreValue = CastInst::CreateBitOrPointerCast(StoreValue, LoadType,
                                       "store_forward_cast", Cand.Store);

[jolanta.jensen]

Yet another clean up after review comments.

[jolanta.jensen]

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

458

I did not find getStoredValue() in StoreInst class. I changed to getValueOperand() instead.

471

Also replaced getOperand(0) with getValueOperand().

[nikic]

LG

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

101

Needs a (void)StoreType, or move the call into the assert.

[jolanta.jensen]

Addressing one more review comment.

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

struct StoreToLoadForwardingCandidate {
 
   /// Return true if the dependence from the store to the load has a
   /// distance of one.  E.g. A[i+1] = A[i]
   bool isDependenceDistanceOfOne(PredicatedScalarEvolution &PSE,
                                  Loop *L) const {
     Value *LoadPtr = Load->getPointerOperand();
     Value *StorePtr = Store->getPointerOperand();
     Type *LoadType = getLoadStoreType(Load);
+    Type *StoreType = getLoadStoreType(Store);
 
-    assert(LoadPtr->getType()->getPointerAddressSpace() ==
-               StorePtr->getType()->getPointerAddressSpace() &&
-           LoadType == getLoadStoreType(Store) &&
-           "Should be a known dependence");
+    assert(
+        LoadPtr->getType()->getPointerAddressSpace() ==
+        Load->getParent()->getModule()->getDataLayout().getTypeSizeInBits(
+            LoadType) ==
+            Store->getParent()->getModule()->getDataLayout().getTypeSizeInBits(
+                StoreType) &&
 
     // Currently we only support accesses with unit stride.  FIXME: we should be
     // able to handle non unit stirde as well as long as the stride is equal to
     // the dependence distance.
     if (getPtrStride(PSE, LoadType, LoadPtr, L) != 1 ||
         getPtrStride(PSE, LoadType, StorePtr, L) != 1)
       return false;
 
     auto &DL = Load->getParent()->getModule()->getDataLayout();
     unsigned TypeByteSize = DL.getTypeAllocSize(const_cast<Type *>(LoadType));
 
     auto *LoadPtrSCEV = cast<SCEVAddRecExpr>(PSE.getSCEV(LoadPtr));
     auto *StorePtrSCEV = cast<SCEVAddRecExpr>(PSE.getSCEV(StorePtr));
 
     // We don't need to check non-wrapping here because forward/backward
     // dependence wouldn't be valid if these weren't monotonic accesses.
     auto *Dist = cast<SCEVConstant>(
for (const auto &Dep : *Deps) {
 
       auto *Store = dyn_cast<StoreInst>(Source);
       if (!Store)
         continue;
       auto *Load = dyn_cast<LoadInst>(Destination);
       if (!Load)
         continue;
 
-      // Only progagate the value if they are of the same type.
-      if (Store->getPointerOperandType() != Load->getPointerOperandType() ||
-          getLoadStoreType(Store) != getLoadStoreType(Load))
+      // Only progagate the value if the pointees are the same size.
+      if (Store->getParent()->getModule()->getDataLayout().getTypeSizeInBits(
+              getLoadStoreType(Store)) !=
+          Load->getParent()->getModule()->getDataLayout().getTypeSizeInBits(
+              getLoadStoreType(Load)))
+        continue;
+
+      // Don't propagate the value if one of the pointees is a pointer but not
+      // the other one.
+      if ((getLoadStoreType(Store)->isPointerTy() &&
+           !getLoadStoreType(Load)->isPointerTy()) ||
+          (!getLoadStoreType(Store)->isPointerTy() &&
+           getLoadStoreType(Load)->isPointerTy()))
         continue;
 
       Candidates.emplace_front(Load, Store);
     }
 
     if (!LoadsWithUnknownDepedence.empty())
       Candidates.remove_if([&](const StoreToLoadForwardingCandidate &C) {
         return LoadsWithUnknownDepedence.count(C.Load);
       });
 
     return Candidates;
   }
 
   /// Return the index of the instruction according to program order.
   unsigned getInstrIndex(Instruction *Inst) {
     auto I = InstOrder.find(Inst);
     assert(I != InstOrder.end() && "No index for instruction");
     return I->second;
   }
 
   /// If a load has multiple candidates associated (i.e. different
   /// stores), it means that it could be forwarding from multiple stores
Value *InitialPtr = SEE.expandCodeFor(PtrSCEV->getStart(), Ptr->getType(),
                                           PH->getTerminator());
     Value *Initial = new LoadInst(
         Cand.Load->getType(), InitialPtr, "load_initial",
         /* isVolatile */ false, Cand.Load->getAlign(), PH->getTerminator());
 
     PHINode *PHI = PHINode::Create(Initial->getType(), 2, "store_forwarded",
                                    &L->getHeader()->front());
     PHI->addIncoming(Initial, PH);
-    PHI->addIncoming(Cand.Store->getOperand(0), L->getLoopLatch());
+
+    Value *StoreValue;
+
+    Type *LoadType = Initial->getType();
+    Type *StoreType = Cand.Store->getOperand(0)->getType();
+
+    assert(
+        Cand.Load->getParent()->getModule()->getDataLayout().getTypeSizeInBits(
+            LoadType) == Cand.Store->getParent()
+                             ->getModule()
+                             ->getDataLayout()
+                             .getTypeSizeInBits(StoreType) &&
+        "The type sizes should match!");
+
+    assert(!((StoreType->isPointerTy() && !LoadType->isPointerTy()) ||
+             (!StoreType->isPointerTy() && LoadType->isPointerTy())) &&
+           "Forwarding pointer and non-pointer type!");
+
+    if (LoadType != StoreType) {
+      // Need a bitcast to convert to the loaded type
+      assert(CastInst::castIsValid(Instruction::BitCast, StoreType, LoadType) &&
+             "Invalid bitcast!");
+      StoreValue =
+          CastInst::Create(Instruction::BitCast, Cand.Store->getOperand(0),
+                           LoadType, "store_forward_cast", Cand.Store);
+    } else
+      StoreValue = Cand.Store->getOperand(0);
+
+    PHI->addIncoming(StoreValue, L->getLoopLatch());
 
     Cand.Load->replaceAllUsesWith(PHI);
   }
 
   /// Top-level driver for each loop: find store->load forwarding
   /// candidates, add run-time checks and perform transformation.
   bool processLoop() {
     LLVM_DEBUG(dbgs() << "\nIn \"" << L->getHeader()->getParent()->getName()
                       << "\" checking " << *L << "\n");
 
     // Look for store-to-load forwarding cases across the

llvm/test/Transforms/LoopLoadElim/type-mismatch-opaque-ptr.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt --opaque-pointers -loop-load-elim -S < %s | FileCheck %s
 
-; Don't crash if the store and the load use different types.
+; If the store and the load use different types, but have the same
+; size then we should still be able to forward the value.
 ;
 ;   for (unsigned i = 0; i < 100; i++) {
 ;     A[i+1] = B[i] + 2;
 ;     C[i] = ((float*)A)[i] * 2;
 ;   }
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
-; CHECK-LABEL: @f(
 define void @f(ptr noalias %A, ptr noalias %B, ptr noalias %C, i64 %N) {
-
+; CHECK-LABEL: @f(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOAD_INITIAL:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[STORE_FORWARDED:%.*]] = phi float [ [[LOAD_INITIAL]], [[ENTRY:%.*]] ], [ [[STORE_FORWARD_CAST:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i32, ptr [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i32, ptr [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[B:%.*]] = load i32, ptr [[BIDX]], align 4
+; CHECK-NEXT:    [[A_P1:%.*]] = add i32 [[B]], 2
+; CHECK-NEXT:    [[STORE_FORWARD_CAST]] = bitcast i32 [[A_P1]] to float
+; CHECK-NEXT:    store i32 [[A_P1]], ptr [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A:%.*]] = load float, ptr [[AIDX]], align 4
+; CHECK-NEXT:    [[C:%.*]] = fmul float [[STORE_FORWARDED]], 2.000000e+00
+; CHECK-NEXT:    [[C_INT:%.*]] = fptosi float [[C]] to i32
+; CHECK-NEXT:    store i32 [[C_INT]], ptr [[CIDX]], align 4
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
 
   %Aidx_next = getelementptr inbounds i32, ptr %A, i64 %indvars.iv.next
   %Bidx = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
   %Cidx = getelementptr inbounds i32, ptr %C, i64 %indvars.iv
   %Aidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
 
   %b = load i32, ptr %Bidx, align 4
   %a_p1 = add i32 %b, 2
   store i32 %a_p1, ptr %Aidx_next, align 4
 
-; CHECK: %a = load float, ptr %Aidx, align 4
   %a = load float, ptr %Aidx, align 4
-; CHECK-NEXT: %c = fmul float %a, 2.0
   %c = fmul float %a, 2.0
   %c.int = fptosi float %c to i32
   store i32 %c.int, ptr %Cidx, align 4
 
   %exitcond = icmp eq i64 %indvars.iv.next, %N
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }
 
-; Don't crash if the store and the load use different types.
+; If the store and the load use different types, but have the same
+; size then we should still be able to forward the value.
 ;
 ;   for (unsigned i = 0; i < 100; i++) {
 ;     A[i+1] = B[i] + 2;
 ;     A[i+1] = B[i] + 3;
 ;     C[i] = ((float*)A)[i] * 2;
 ;   }
 
-; CHECK-LABEL: @f2(
 define void @f2(ptr noalias %A, ptr noalias %B, ptr noalias %C, i64 %N) {
-
+; CHECK-LABEL: @f2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOAD_INITIAL:%.*]] = load float, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[STORE_FORWARDED:%.*]] = phi float [ [[LOAD_INITIAL]], [[ENTRY:%.*]] ], [ [[STORE_FORWARD_CAST:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i32, ptr [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i32, ptr [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[B:%.*]] = load i32, ptr [[BIDX]], align 4
+; CHECK-NEXT:    [[A_P2:%.*]] = add i32 [[B]], 2
+; CHECK-NEXT:    store i32 [[A_P2]], ptr [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A_P3:%.*]] = add i32 [[B]], 3
+; CHECK-NEXT:    [[STORE_FORWARD_CAST]] = bitcast i32 [[A_P3]] to float
+; CHECK-NEXT:    store i32 [[A_P3]], ptr [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A:%.*]] = load float, ptr [[AIDX]], align 4
+; CHECK-NEXT:    [[C:%.*]] = fmul float [[STORE_FORWARDED]], 2.000000e+00
+; CHECK-NEXT:    [[C_INT:%.*]] = fptosi float [[C]] to i32
+; CHECK-NEXT:    store i32 [[C_INT]], ptr [[CIDX]], align 4
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
 
   %Aidx_next = getelementptr inbounds i32, ptr %A, i64 %indvars.iv.next
   %Bidx = getelementptr inbounds i32, ptr %B, i64 %indvars.iv
   %Cidx = getelementptr inbounds i32, ptr %C, i64 %indvars.iv
   %Aidx = getelementptr inbounds i32, ptr %A, i64 %indvars.iv
 
   %b = load i32, ptr %Bidx, align 4
   %a_p2 = add i32 %b, 2
   store i32 %a_p2, ptr %Aidx_next, align 4
 
   %a_p3 = add i32 %b, 3
   store i32 %a_p3, ptr %Aidx_next, align 4
 
-; CHECK: %a = load float, ptr %Aidx, align 4
   %a = load float, ptr %Aidx, align 4
-; CHECK-NEXT: %c = fmul float %a, 2.0
   %c = fmul float %a, 2.0
   %c.int = fptosi float %c to i32
   store i32 %c.int, ptr %Cidx, align 4
 
   %exitcond = icmp eq i64 %indvars.iv.next, %N
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }

llvm/test/Transforms/LoopLoadElim/type-mismatch.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -loop-load-elim -S < %s | FileCheck %s
 
-; Don't crash if the store and the load use different types.
+; If the store and the load use different types, but have the same
+; size then we should still be able to forward the value.
 ;
 ;   for (unsigned i = 0; i < 100; i++) {
 ;     A[i+1] = B[i] + 2;
 ;     C[i] = ((float*)A)[i] * 2;
 ;   }
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
-; CHECK-LABEL: @f(
 define void @f(i32* noalias %A, i32* noalias %B, i32* noalias %C, i64 %N) {
-
+; CHECK-LABEL: @f(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[A1:%.*]] = bitcast i32* [[A:%.*]] to float*
+; CHECK-NEXT:    [[LOAD_INITIAL:%.*]] = load float, float* [[A1]], align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[STORE_FORWARDED:%.*]] = phi float [ [[LOAD_INITIAL]], [[ENTRY:%.*]] ], [ [[STORE_FORWARD_CAST:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i32, i32* [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX_FLOAT:%.*]] = bitcast i32* [[AIDX]] to float*
+; CHECK-NEXT:    [[B:%.*]] = load i32, i32* [[BIDX]], align 4
+; CHECK-NEXT:    [[A_P1:%.*]] = add i32 [[B]], 2
+; CHECK-NEXT:    [[STORE_FORWARD_CAST]] = bitcast i32 [[A_P1]] to float
+; CHECK-NEXT:    store i32 [[A_P1]], i32* [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A:%.*]] = load float, float* [[AIDX_FLOAT]], align 4
+; CHECK-NEXT:    [[C:%.*]] = fmul float [[STORE_FORWARDED]], 2.000000e+00
+; CHECK-NEXT:    [[C_INT:%.*]] = fptosi float [[C]] to i32
+; CHECK-NEXT:    store i32 [[C_INT]], i32* [[CIDX]], align 4
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
 
   %Aidx_next = getelementptr inbounds i32, i32* %A, i64 %indvars.iv.next
   %Bidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
   %Cidx = getelementptr inbounds i32, i32* %C, i64 %indvars.iv
   %Aidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
   %Aidx.float = bitcast i32* %Aidx to float*
 
   %b = load i32, i32* %Bidx, align 4
   %a_p1 = add i32 %b, 2
   store i32 %a_p1, i32* %Aidx_next, align 4
 
-; CHECK: %a = load float, float* %Aidx.float, align 4
   %a = load float, float* %Aidx.float, align 4
-; CHECK-NEXT: %c = fmul float %a, 2.0
   %c = fmul float %a, 2.0
   %c.int = fptosi float %c to i32
   store i32 %c.int, i32* %Cidx, align 4
 
   %exitcond = icmp eq i64 %indvars.iv.next, %N
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }
 
-; Don't crash if the store and the load use different types.
+; If the store and the load use different types, but have the same
+; size then we should still be able to forward the value.
 ;
 ;   for (unsigned i = 0; i < 100; i++) {
 ;     A[i+1] = B[i] + 2;
 ;     A[i+1] = B[i] + 3;
 ;     C[i] = ((float*)A)[i] * 2;
 ;   }
 
-; CHECK-LABEL: @f2(
 define void @f2(i32* noalias %A, i32* noalias %B, i32* noalias %C, i64 %N) {
-
+; CHECK-LABEL: @f2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[A1:%.*]] = bitcast i32* [[A:%.*]] to float*
+; CHECK-NEXT:    [[LOAD_INITIAL:%.*]] = load float, float* [[A1]], align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[STORE_FORWARDED:%.*]] = phi float [ [[LOAD_INITIAL]], [[ENTRY:%.*]] ], [ [[STORE_FORWARD_CAST:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i32, i32* [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX_FLOAT:%.*]] = bitcast i32* [[AIDX]] to float*
+; CHECK-NEXT:    [[B:%.*]] = load i32, i32* [[BIDX]], align 4
+; CHECK-NEXT:    [[A_P2:%.*]] = add i32 [[B]], 2
+; CHECK-NEXT:    store i32 [[A_P2]], i32* [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A_P3:%.*]] = add i32 [[B]], 3
+; CHECK-NEXT:    [[STORE_FORWARD_CAST]] = bitcast i32 [[A_P3]] to float
+; CHECK-NEXT:    store i32 [[A_P3]], i32* [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A:%.*]] = load float, float* [[AIDX_FLOAT]], align 4
+; CHECK-NEXT:    [[C:%.*]] = fmul float [[STORE_FORWARDED]], 2.000000e+00
+; CHECK-NEXT:    [[C_INT:%.*]] = fptosi float [[C]] to i32
+; CHECK-NEXT:    store i32 [[C_INT]], i32* [[CIDX]], align 4
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
 
   %Aidx_next = getelementptr inbounds i32, i32* %A, i64 %indvars.iv.next
   %Bidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
   %Cidx = getelementptr inbounds i32, i32* %C, i64 %indvars.iv
   %Aidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
   %Aidx.float = bitcast i32* %Aidx to float*
 
   %b = load i32, i32* %Bidx, align 4
   %a_p2 = add i32 %b, 2
   store i32 %a_p2, i32* %Aidx_next, align 4
 
   %a_p3 = add i32 %b, 3
   store i32 %a_p3, i32* %Aidx_next, align 4
 
-; CHECK: %a = load float, float* %Aidx.float, align 4
   %a = load float, float* %Aidx.float, align 4
-; CHECK-NEXT: %c = fmul float %a, 2.0
   %c = fmul float %a, 2.0
   %c.int = fptosi float %c to i32
   store i32 %c.int, i32* %Cidx, align 4
 
   %exitcond = icmp eq i64 %indvars.iv.next, %N
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }
+
+; Check that we don't forward between pointer-sized integers and actual
+; pointers; We could potentially do this in future.
+
+define void @f3(i64* noalias %A, i64* noalias %B, i64* noalias %C, i64 %N) {
+; CHECK-LABEL: @f3(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i64, i64* [[A:%.*]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i64, i64* [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i64, i64* [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i64, i64* [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX_I8P:%.*]] = bitcast i64* [[AIDX]] to i8**
+; CHECK-NEXT:    [[B:%.*]] = load i64, i64* [[BIDX]], align 8
+; CHECK-NEXT:    [[A_P1:%.*]] = add i64 [[B]], 2
+; CHECK-NEXT:    store i64 [[A_P1]], i64* [[AIDX_NEXT]], align 8
+; CHECK-NEXT:    [[A:%.*]] = load i8*, i8** [[AIDX_I8P]], align 8
+; CHECK-NEXT:    [[C:%.*]] = getelementptr i8, i8* [[A]], i64 57
+; CHECK-NEXT:    [[C_I64P:%.*]] = ptrtoint i8* [[C]] to i64
+; CHECK-NEXT:    store i64 [[C_I64P]], i64* [[CIDX]], align 8
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+
+  %Aidx_next = getelementptr inbounds i64, i64* %A, i64 %indvars.iv.next
+  %Bidx = getelementptr inbounds i64, i64* %B, i64 %indvars.iv
+  %Cidx = getelementptr inbounds i64, i64* %C, i64 %indvars.iv
+  %Aidx = getelementptr inbounds i64, i64* %A, i64 %indvars.iv
+  %Aidx.i8p = bitcast i64* %Aidx to i8**
+
+  %b = load i64, i64* %Bidx, align 8
+  %a_p1 = add i64 %b, 2
+  store i64 %a_p1, i64* %Aidx_next, align 8
+
+  %a = load i8*, i8** %Aidx.i8p, align 8
+  %c = getelementptr i8, i8* %a, i64 57
+  %c.i64p = ptrtoint i8* %c to i64
+  store i64 %c.i64p, i64* %Cidx, align 8
+
+  %exitcond = icmp eq i64 %indvars.iv.next, %N
+  br i1 %exitcond, label %for.end, label %for.body
+
+for.end:                                          ; preds = %for.body
+  ret void
+}
+
+; If the store and the load use different types, but have the same
+; size then we should still be able to forward the value--also for
+; vector types.
+;
+;   for (unsigned i = 0; i < 100; i++) {
+;     A[i+1] = B[i] + 2;
+;     C[i] = ((float*)A)[i] * 2;
+;   }
+
+define void @f4(i32* noalias %A, i32* noalias %B, i32* noalias %C, i64 %N) {
+; CHECK-LABEL: @f4(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[A1:%.*]] = bitcast i32* [[A:%.*]] to <2 x half>*
+; CHECK-NEXT:    [[LOAD_INITIAL:%.*]] = load <2 x half>, <2 x half>* [[A1]], align 4
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[STORE_FORWARDED:%.*]] = phi <2 x half> [ [[LOAD_INITIAL]], [[ENTRY:%.*]] ], [ [[STORE_FORWARD_CAST:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[AIDX_NEXT:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV_NEXT]]
+; CHECK-NEXT:    [[BIDX:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[CIDX:%.*]] = getelementptr inbounds i32, i32* [[C:%.*]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX:%.*]] = getelementptr inbounds i32, i32* [[A]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[AIDX_FLOAT:%.*]] = bitcast i32* [[AIDX]] to <2 x half>*
+; CHECK-NEXT:    [[B:%.*]] = load i32, i32* [[BIDX]], align 4
+; CHECK-NEXT:    [[A_P1:%.*]] = add i32 [[B]], 2
+; CHECK-NEXT:    [[STORE_FORWARD_CAST]] = bitcast i32 [[A_P1]] to <2 x half>
+; CHECK-NEXT:    store i32 [[A_P1]], i32* [[AIDX_NEXT]], align 4
+; CHECK-NEXT:    [[A:%.*]] = load <2 x half>, <2 x half>* [[AIDX_FLOAT]], align 4
+; CHECK-NEXT:    [[C:%.*]] = fmul <2 x half> [[STORE_FORWARDED]], <half 0xH4000, half 0xH4000>
+; CHECK-NEXT:    [[C_INT:%.*]] = bitcast <2 x half> [[C]] to i32
+; CHECK-NEXT:    store i32 [[C_INT]], i32* [[CIDX]], align 4
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+
+  %Aidx_next = getelementptr inbounds i32, i32* %A, i64 %indvars.iv.next
+  %Bidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
+  %Cidx = getelementptr inbounds i32, i32* %C, i64 %indvars.iv
+  %Aidx = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
+  %Aidx.float = bitcast i32* %Aidx to <2 x half>*
+
+  %b = load i32, i32* %Bidx, align 4
+  %a_p1 = add i32 %b, 2
+  store i32 %a_p1, i32* %Aidx_next, align 4
+
+  %a = load <2 x half>, <2 x half>* %Aidx.float, align 4
+  %c = fmul <2 x half> %a, <half 2.0, half 2.0>
+  %c.int = bitcast <2 x half> %c to i32
+  store i32 %c.int, i32* %Cidx, align 4
+
+  %exitcond = icmp eq i64 %indvars.iv.next, %N
+  br i1 %exitcond, label %for.end, label %for.body
+
+for.end:                                          ; preds = %for.body
+  ret void
+}