[GVN] Fix VNCoercion for Scalable Vector.

[huihuiz]

Summary

For VNCoercion, skip scalable vector when analysis rely on fixed size,
otherwise call TypeSize::getFixedSize() explicitly.

Add unit tests to check funtionality of GVN load elimination for scalable type.

[efriedma]

llvm/lib/Analysis/BasicAliasAnalysis.cpp

543

Can you check DL.getTypeAllocSize(GEPIdxedTy).Scalable instead, since you're calling getTypeAllocSize anyway?

I don't think this is the right way to break out of this loop early; if the offset isn't recorded in "Decomposed", other code might assume it's zero?

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

1880

I think TypeSize also has isZero now, assuming that got merged.

llvm/lib/Transforms/Utils/VNCoercion.cpp

21

Fix the comment here?

Can you refactor the whole Ty->isStructTy() || LoadTy->isArrayTy() || (Ty->isVectorTy() && Ty->getVectorIsScalable()) thing into a helper function, so the overall property has a name?

[bjope]

llvm/lib/Transforms/Utils/VNCoercion.cpp

21

I agree that refactoring this check into a helper could be wise.

Just out of curiosity, shouldn't we also check DL.typeSizeEuqalsStoreSize in such a helper?

Background for that question is that I've been investigating some problems downstream (where we got some types that aren't byte sized and thus include paddning in memory).

Consider this example (using types available upstream):

define <4 x i1> @v4i8_to_v4i1() {
  %tmp = alloca <4 x i8>, align 1
  store <4 x i8> <i8 4, i8 -1, i8 -1, i8 -1>, <4 x i8>* %tmp, align 1
  %sroa_cast = bitcast <4 x i8>* %tmp to <4 x i1>*
  %dst = load <4 x i1>, <4 x i1>* %sroa_cast, align 1
  ret <4 x i1> %dst
}

And notice that the returned value will differ depending on if we have big/little endian:

> opt -gvn -S -data-layout "e" gvn-test.ll
; ModuleID = 'gvn-test.ll'
source_filename = "gvn-test.ll"
target datalayout = "e"

define <4 x i1> @v4i8_to_v4i1() {
  %tmp = alloca <4 x i8>, align 1
  store <4 x i8> <i8 4, i8 -1, i8 -1, i8 -1>, <4 x i8>* %tmp, align 1
  %sroa_cast = bitcast <4 x i8>* %tmp to <4 x i1>*
  ret <4 x i1> <i1 false, i1 false, i1 true, i1 false>
}

> opt -gvn -S -data-layout "E" gvn-test.ll

; ModuleID = 'gvn-test.ll'
source_filename = "gvn-test.ll"
target datalayout = "E"

define <4 x i1> @v4i8_to_v4i1() {
  %tmp = alloca <4 x i8>, align 1
  store <4 x i8> <i8 4, i8 -1, i8 -1, i8 -1>, <4 x i8>* %tmp, align 1
  %sroa_cast = bitcast <4 x i8>* %tmp to <4 x i1>*
  ret <4 x i1> <i1 false, i1 true, i1 false, i1 false>
}

So we might wanna avoid VNCoercion for types with DL.typeSizeEuqalsStoreSize(Ty) being false. At least for vector types. For scalars the alignTo check a few lines down offer some protection, but maybe checking DL.typeSizeEuqalsStoreSize(Ty) make sense for scalars as well. Or what do you think?

[bjope]

llvm/lib/Transforms/Utils/VNCoercion.cpp

21

Well, it is not wrong that we get different result for big/little endian in my example above. Just that it is unclear to my where the bits go when storing <4 x i1> to memory. Is it well defined where the four bits go when storing <4 x i1> to memory?

[efriedma]

llvm/lib/Transforms/Utils/VNCoercion.cpp

21

Yes, there's a consistent rule that should be used everywhere. Maybe the simplest description is just the code in foldConstVectorToAPInt in ConstantFolding.cpp.

[huihuiz]

llvm/lib/Analysis/BasicAliasAnalysis.cpp

543

Spend sometime look again, BasicAAResult::aliasGEP is using the offset/varIndices recorded in DecomposedGEP. For scalable type, we might end up inaccurate alias result.

Working on fixes for this logic.

[huihuiz]

Address review comments.

[huihuiz]

llvm/lib/Analysis/BasicAliasAnalysis.cpp

543

Fixing BasicAA in D77828.

[efriedma]

LGTM

llvm/lib/Analysis/BasicAliasAnalysis.cpp

do {
     unsigned AS = GEPOp->getPointerAddressSpace();
     // Walk the indices of the GEP, accumulating them into BaseOff/VarIndices.
     gep_type_iterator GTI = gep_type_begin(GEPOp);
     unsigned PointerSize = DL.getPointerSizeInBits(AS);
     // Assume all GEP operands are constants until proven otherwise.
     bool GepHasConstantOffset = true;
     for (User::const_op_iterator I = GEPOp->op_begin() + 1, E = GEPOp->op_end();
          I != E; ++I, ++GTI) {
+      Type *GEPIdxedTy = GTI.getIndexedType();
+      if (GEPIdxedTy->isVectorTy() && GEPIdxedTy->getVectorIsScalable()) {
+        GepHasConstantOffset = false;
+        break;
+      }
+
       const Value *Index = *I;
       // Compute the (potentially symbolic) offset in bytes for this index.
       if (StructType *STy = GTI.getStructTypeOrNull()) {
         // For a struct, add the member offset.
         unsigned FieldNo = cast<ConstantInt>(Index)->getZExtValue();
         if (FieldNo == 0)
           continue;
 
         Decomposed.StructOffset +=
           DL.getStructLayout(STy)->getElementOffset(FieldNo);
         continue;
       }
 
       // For an array/pointer, add the element offset, explicitly scaled.
       if (const ConstantInt *CIdx = dyn_cast<ConstantInt>(Index)) {
         if (CIdx->isZero())
           continue;
         Decomposed.OtherOffset +=
-          (DL.getTypeAllocSize(GTI.getIndexedType()) *
+            (DL.getTypeAllocSize(GEPIdxedTy).getFixedSize() *
              CIdx->getValue().sextOrSelf(MaxPointerSize))
                 .sextOrTrunc(MaxPointerSize);
         continue;
       }
 
       GepHasConstantOffset = false;
 
-      APInt Scale(MaxPointerSize, DL.getTypeAllocSize(GTI.getIndexedType()));
+      APInt Scale(MaxPointerSize,
+                  DL.getTypeAllocSize(GEPIdxedTy).getFixedSize());
       unsigned ZExtBits = 0, SExtBits = 0;
 
       // If the integer type is smaller than the pointer size, it is implicitly
       // sign extended to pointer size.
       unsigned Width = Index->getType()->getIntegerBitWidth();
       if (PointerSize > Width)
         SExtBits += PointerSize - Width;
 

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

     Type *NewIndexType =
         IndexTy->isVectorTy()
             ? VectorType::get(NewScalarIndexTy, IndexTy->getVectorNumElements())
             : NewScalarIndexTy;
 
     // If the element type has zero size then any index over it is equivalent
     // to an index of zero, so replace it with zero if it is not zero already.
     Type *EltTy = GTI.getIndexedType();
-    if (EltTy->isSized() && DL.getTypeAllocSize(EltTy) == 0)
+    if (EltTy->isSized() && !DL.getTypeAllocSize(EltTy).isNonZero())
       if (!isa<Constant>(*I) || !match(I->get(), m_Zero())) {
         *I = Constant::getNullValue(NewIndexType);
         MadeChange = true;
       }
 
     if (IndexTy != NewIndexType) {
       // If we are using a wider index than needed for this platform, shrink
       // it to what we need.  If narrower, sign-extend it to what we need.

llvm/lib/Transforms/Utils/VNCoercion.cpp

 /// Return true if coerceAvailableValueToLoadType will succeed.
 bool canCoerceMustAliasedValueToLoad(Value *StoredVal, Type *LoadTy,
                                      const DataLayout &DL) {
   Type *StoredTy = StoredVal->getType();
   if (StoredTy == LoadTy)
     return true;
 
   // If the loaded or stored value is an first class array or struct, don't try
   // to transform them.  We need to be able to bitcast to integer.
   if (LoadTy->isStructTy() || LoadTy->isArrayTy() || StoredTy->isStructTy() ||
-      StoredTy->isArrayTy())
+      StoredTy->isArrayTy() ||
+      (LoadTy->isVectorTy() && LoadTy->getVectorIsScalable()) ||
+      (StoredTy->isVectorTy() && StoredTy->getVectorIsScalable()))
     return false;
 
-  uint64_t StoreSize = DL.getTypeSizeInBits(StoredTy);
+  uint64_t StoreSize = DL.getTypeSizeInBits(StoredTy).getFixedSize();
 
   // The store size must be byte-aligned to support future type casts.
   if (llvm::alignTo(StoreSize, 8) != StoreSize)
     return false;
 
   // The store has to be at least as big as the load.
-  if (StoreSize < DL.getTypeSizeInBits(LoadTy))
+  if (StoreSize < DL.getTypeSizeInBits(LoadTy).getFixedSize())
     return false;
 
   // Don't coerce non-integral pointers to integers or vice versa.
   if (DL.isNonIntegralPointerType(StoredVal->getType()->getScalarType()) !=
       DL.isNonIntegralPointerType(LoadTy->getScalarType())) {
     // As a special case, allow coercion of memset used to initialize
     // an array w/null.  Despite non-integral pointers not generally having a
     // specific bit pattern, we do assume null is zero.
static T *coerceAvailableValueToLoadTypeHelper(T *StoredVal, Type *LoadedTy,
   assert(canCoerceMustAliasedValueToLoad(StoredVal, LoadedTy, DL) &&
          "precondition violation - materialization can't fail");
   if (auto *C = dyn_cast<Constant>(StoredVal))
     StoredVal = ConstantFoldConstant(C, DL);
 
   // If this is already the right type, just return it.
   Type *StoredValTy = StoredVal->getType();
 
-  uint64_t StoredValSize = DL.getTypeSizeInBits(StoredValTy);
-  uint64_t LoadedValSize = DL.getTypeSizeInBits(LoadedTy);
+  uint64_t StoredValSize = DL.getTypeSizeInBits(StoredValTy).getFixedSize();
+  uint64_t LoadedValSize = DL.getTypeSizeInBits(LoadedTy).getFixedSize();
 
   // If the store and reload are the same size, we can always reuse it.
   if (StoredValSize == LoadedValSize) {
     // Pointer to Pointer -> use bitcast.
     if (StoredValTy->isPtrOrPtrVectorTy() && LoadedTy->isPtrOrPtrVectorTy()) {
       StoredVal = Helper.CreateBitCast(StoredVal, LoadedTy);
     } else {
       // Convert source pointers to integers, which can be bitcast.
static T *coerceAvailableValueToLoadTypeHelper(T *StoredVal, Type *LoadedTy,
   if (!StoredValTy->isIntegerTy()) {
     StoredValTy = IntegerType::get(StoredValTy->getContext(), StoredValSize);
     StoredVal = Helper.CreateBitCast(StoredVal, StoredValTy);
   }
 
   // If this is a big-endian system, we need to shift the value down to the low
   // bits so that a truncate will work.
   if (DL.isBigEndian()) {
-    uint64_t ShiftAmt = DL.getTypeStoreSizeInBits(StoredValTy) -
-                        DL.getTypeStoreSizeInBits(LoadedTy);
+    uint64_t ShiftAmt = DL.getTypeStoreSizeInBits(StoredValTy).getFixedSize() -
+                        DL.getTypeStoreSizeInBits(LoadedTy).getFixedSize();
     StoredVal = Helper.CreateLShr(
         StoredVal, ConstantInt::get(StoredVal->getType(), ShiftAmt));
   }
 
   // Truncate the integer to the right size now.
   Type *NewIntTy = IntegerType::get(StoredValTy->getContext(), LoadedValSize);
   StoredVal = Helper.CreateTruncOrBitCast(StoredVal, NewIntTy);
 

 /// up.  This returns -1 if we have to give up, or a byte number in the stored
 /// value of the piece that feeds the load.
 static int analyzeLoadFromClobberingWrite(Type *LoadTy, Value *LoadPtr,
                                           Value *WritePtr,
                                           uint64_t WriteSizeInBits,
                                           const DataLayout &DL) {
   // If the loaded or stored value is a first class array or struct, don't try
   // to transform them.  We need to be able to bitcast to integer.
-  if (LoadTy->isStructTy() || LoadTy->isArrayTy())
+  if (LoadTy->isStructTy() || LoadTy->isArrayTy() ||
+      (LoadTy->isVectorTy() && LoadTy->getVectorIsScalable()))
     return -1;
 
   int64_t StoreOffset = 0, LoadOffset = 0;
   Value *StoreBase =
       GetPointerBaseWithConstantOffset(WritePtr, StoreOffset, DL);
   Value *LoadBase = GetPointerBaseWithConstantOffset(LoadPtr, LoadOffset, DL);
   if (StoreBase != LoadBase)
     return -1;
 
   // If the load and store are to the exact same address, they should have been
   // a must alias.  AA must have gotten confused.
   // FIXME: Study to see if/when this happens.  One case is forwarding a memset
   // to a load from the base of the memset.
 
   // If the load and store don't overlap at all, the store doesn't provide
   // anything to the load.  In this case, they really don't alias at all, AA
   // must have gotten confused.
-  uint64_t LoadSize = DL.getTypeSizeInBits(LoadTy);
+  uint64_t LoadSize = DL.getTypeSizeInBits(LoadTy).getFixedSize();
 
   if ((WriteSizeInBits & 7) | (LoadSize & 7))
     return -1;
   uint64_t StoreSize = WriteSizeInBits / 8; // Convert to bytes.
   LoadSize /= 8;
 
   bool isAAFailure = false;
   if (StoreOffset < LoadOffset)

 
 /// This function is called when we have a
 /// memdep query of a load that ends up being a clobbering store.
 int analyzeLoadFromClobberingStore(Type *LoadTy, Value *LoadPtr,
                                    StoreInst *DepSI, const DataLayout &DL) {
   auto *StoredVal = DepSI->getValueOperand();
   
   // Cannot handle reading from store of first-class aggregate yet.
-  if (StoredVal->getType()->isStructTy() ||
-      StoredVal->getType()->isArrayTy())
+  if (StoredVal->getType()->isStructTy() || StoredVal->getType()->isArrayTy() ||
+      (StoredVal->getType()->isVectorTy() &&
+       StoredVal->getType()->getVectorIsScalable()))
     return -1;
 
   // Don't coerce non-integral pointers to integers or vice versa.
   if (DL.isNonIntegralPointerType(StoredVal->getType()->getScalarType()) !=
       DL.isNonIntegralPointerType(LoadTy->getScalarType())) {
     // Allow casts of zero values to null as a special case
     auto *CI = dyn_cast<Constant>(StoredVal);
     if (!CI || !CI->isNullValue())
       return -1;
   }
 
   Value *StorePtr = DepSI->getPointerOperand();
   uint64_t StoreSize =
-      DL.getTypeSizeInBits(DepSI->getValueOperand()->getType());
+      DL.getTypeSizeInBits(DepSI->getValueOperand()->getType()).getFixedSize();
   return analyzeLoadFromClobberingWrite(LoadTy, LoadPtr, StorePtr, StoreSize,
                                         DL);
 }
 
 /// Looks at a memory location for a load (specified by MemLocBase, Offs, and
 /// Size) and compares it against a load.
 ///
 /// If the specified load could be safely widened to a larger integer load
if (DepLI->getType()->isStructTy() || DepLI->getType()->isArrayTy())
     return -1;
 
   // Don't coerce non-integral pointers to integers or vice versa.
   if (DL.isNonIntegralPointerType(DepLI->getType()->getScalarType()) !=
       DL.isNonIntegralPointerType(LoadTy->getScalarType()))
     return -1;
 
   Value *DepPtr = DepLI->getPointerOperand();
-  uint64_t DepSize = DL.getTypeSizeInBits(DepLI->getType());
+  uint64_t DepSize = DL.getTypeSizeInBits(DepLI->getType()).getFixedSize();
   int R = analyzeLoadFromClobberingWrite(LoadTy, LoadPtr, DepPtr, DepSize, DL);
   if (R != -1)
     return R;
 
   // If we have a load/load clobber an DepLI can be widened to cover this load,
   // then we should widen it!
   int64_t LoadOffs = 0;
   const Value *LoadBase =
       GetPointerBaseWithConstantOffset(LoadPtr, LoadOffs, DL);
-  unsigned LoadSize = DL.getTypeStoreSize(LoadTy);
+  unsigned LoadSize = DL.getTypeStoreSize(LoadTy).getFixedSize();
 
   unsigned Size =
       getLoadLoadClobberFullWidthSize(LoadBase, LoadOffs, LoadSize, DepLI);
   if (Size == 0)
     return -1;
 
   // Check non-obvious conditions enforced by MDA which we rely on for being
   // able to materialize this potentially available value
static T *getStoreValueForLoadHelper(T *SrcVal, unsigned Offset, Type *LoadTy,
   // so we don't need to do any truncation, etc. This avoids introducing
   // ptrtoint instructions for pointers that may be non-integral.
   if (SrcVal->getType()->isPointerTy() && LoadTy->isPointerTy() &&
       cast<PointerType>(SrcVal->getType())->getAddressSpace() ==
           cast<PointerType>(LoadTy)->getAddressSpace()) {
     return SrcVal;
   }
 
-  uint64_t StoreSize = (DL.getTypeSizeInBits(SrcVal->getType()) + 7) / 8;
-  uint64_t LoadSize = (DL.getTypeSizeInBits(LoadTy) + 7) / 8;
+  uint64_t StoreSize =
+      (DL.getTypeSizeInBits(SrcVal->getType()).getFixedSize() + 7) / 8;
+  uint64_t LoadSize = (DL.getTypeSizeInBits(LoadTy).getFixedSize() + 7) / 8;
   // Compute which bits of the stored value are being used by the load.  Convert
   // to an integer type to start with.
   if (SrcVal->getType()->isPtrOrPtrVectorTy())
     SrcVal = Helper.CreatePtrToInt(SrcVal, DL.getIntPtrType(SrcVal->getType()));
   if (!SrcVal->getType()->isIntegerTy())
     SrcVal = Helper.CreateBitCast(SrcVal, IntegerType::get(Ctx, StoreSize * 8));
 
   // Shift the bits to the least significant depending on endianness.

 /// being a clobbering load.  This means that the load *may* provide bits used
 /// by the load but we can't be sure because the pointers don't must-alias.
 /// Check this case to see if there is anything more we can do before we give
 /// up.
 Value *getLoadValueForLoad(LoadInst *SrcVal, unsigned Offset, Type *LoadTy,
                            Instruction *InsertPt, const DataLayout &DL) {
   // If Offset+LoadTy exceeds the size of SrcVal, then we must be wanting to
   // widen SrcVal out to a larger load.
-  unsigned SrcValStoreSize = DL.getTypeStoreSize(SrcVal->getType());
-  unsigned LoadSize = DL.getTypeStoreSize(LoadTy);
+  unsigned SrcValStoreSize =
+      DL.getTypeStoreSize(SrcVal->getType()).getFixedSize();
+  unsigned LoadSize = DL.getTypeStoreSize(LoadTy).getFixedSize();
   if (Offset + LoadSize > SrcValStoreSize) {
     assert(SrcVal->isSimple() && "Cannot widen volatile/atomic load!");
     assert(SrcVal->getType()->isIntegerTy() && "Can't widen non-integer load");
     // If we have a load/load clobber an DepLI can be widened to cover this
     // load, then we should widen it to the next power of 2 size big enough!
     unsigned NewLoadSize = Offset + LoadSize;
     if (!isPowerOf2_32(NewLoadSize))
       NewLoadSize = NextPowerOf2(NewLoadSize);
if (Offset + LoadSize > SrcValStoreSize) {
     SrcVal = NewLoad;
   }
 
   return getStoreValueForLoad(SrcVal, Offset, LoadTy, InsertPt, DL);
 }
 
 Constant *getConstantLoadValueForLoad(Constant *SrcVal, unsigned Offset,
                                       Type *LoadTy, const DataLayout &DL) {
-  unsigned SrcValStoreSize = DL.getTypeStoreSize(SrcVal->getType());
-  unsigned LoadSize = DL.getTypeStoreSize(LoadTy);
+  unsigned SrcValStoreSize =
+      DL.getTypeStoreSize(SrcVal->getType()).getFixedSize();
+  unsigned LoadSize = DL.getTypeStoreSize(LoadTy).getFixedSize();
   if (Offset + LoadSize > SrcValStoreSize)
     return nullptr;
   return getConstantStoreValueForLoad(SrcVal, Offset, LoadTy, DL);
 }
 
 template <class T, class HelperClass>
 T *getMemInstValueForLoadHelper(MemIntrinsic *SrcInst, unsigned Offset,
                                 Type *LoadTy, HelperClass &Helper,
                                 const DataLayout &DL) {
   LLVMContext &Ctx = LoadTy->getContext();
-  uint64_t LoadSize = DL.getTypeSizeInBits(LoadTy) / 8;
+  uint64_t LoadSize = DL.getTypeSizeInBits(LoadTy).getFixedSize() / 8;
 
   // We know that this method is only called when the mem transfer fully
   // provides the bits for the load.
   if (MemSetInst *MSI = dyn_cast<MemSetInst>(SrcInst)) {
     // memset(P, 'x', 1234) -> splat('x'), even if x is a variable, and
     // independently of what the offset is.
     T *Val = cast<T>(MSI->getValue());
     if (LoadSize != 1)

llvm/test/Transforms/GVN/vscale.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S < %s -basicaa -gvn -dce | FileCheck %s
+
+; Analyze Load from clobbering Load.
+
+define <vscale x 4 x i32> @load_store_clobber_load(<vscale x 4 x i32> *%p)  {
+; CHECK-LABEL: @load_store_clobber_load(
+; CHECK-NEXT:    [[LOAD1:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* undef
+; CHECK-NEXT:    [[ADD:%.*]] = add <vscale x 4 x i32> [[LOAD1]], [[LOAD1]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[ADD]]
+;
+  %load1 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* undef
+  %load2 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p ; <- load to be eliminated
+  %add = add <vscale x 4 x i32> %load1, %load2
+  ret <vscale x 4 x i32> %add
+}
+
+define <vscale x 4 x i32> @load_store_clobber_load_mayalias(<vscale x 4 x i32>* %p, <vscale x 4 x i32>* %p2) {
+; CHECK-LABEL: @load_store_clobber_load_mayalias(
+; CHECK-NEXT:    [[LOAD1:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[P2:%.*]]
+; CHECK-NEXT:    [[LOAD2:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub <vscale x 4 x i32> [[LOAD1]], [[LOAD2]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[SUB]]
+;
+  %load1 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %p2
+  %load2 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  %sub = sub <vscale x 4 x i32> %load1, %load2
+  ret <vscale x 4 x i32> %sub
+}
+
+define <vscale x 4 x i32> @load_store_clobber_load_noalias(<vscale x 4 x i32>* noalias %p, <vscale x 4 x i32>* noalias %p2) {
+; CHECK-LABEL: @load_store_clobber_load_noalias(
+; CHECK-NEXT:    [[LOAD1:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[P2:%.*]]
+; CHECK-NEXT:    [[ADD:%.*]] = add <vscale x 4 x i32> [[LOAD1]], [[LOAD1]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[ADD]]
+;
+  %load1 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %p2
+  %load2 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p ; <- load to be eliminated
+  %add = add <vscale x 4 x i32> %load1, %load2
+  ret <vscale x 4 x i32> %add
+}
+
+; TODO: %load2 could be eliminated
+define i32 @load_clobber_load_gep1(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @load_clobber_load_gep1(
+; CHECK-NEXT:    [[GEP1:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]], i64 0, i64 1
+; CHECK-NEXT:    [[LOAD1:%.*]] = load i32, i32* [[GEP1]]
+; CHECK-NEXT:    [[P2:%.*]] = bitcast <vscale x 4 x i32>* [[P]] to i32*
+; CHECK-NEXT:    [[GEP2:%.*]] = getelementptr i32, i32* [[P2]], i64 1
+; CHECK-NEXT:    [[LOAD2:%.*]] = load i32, i32* [[GEP2]]
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[LOAD1]], [[LOAD2]]
+; CHECK-NEXT:    ret i32 [[ADD]]
+;
+  %gep1 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 0, i64 1
+  %load1 = load i32, i32* %gep1
+  %p2 = bitcast <vscale x 4 x i32>* %p to i32*
+  %gep2 = getelementptr i32, i32* %p2, i64 1
+  %load2 = load i32, i32* %gep2 ; <- load could be eliminated
+  %add = add i32 %load1, %load2
+  ret i32 %add
+}
+
+define i32 @load_clobber_load_gep2(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @load_clobber_load_gep2(
+; CHECK-NEXT:    [[GEP1:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]], i64 1, i64 0
+; CHECK-NEXT:    [[LOAD1:%.*]] = load i32, i32* [[GEP1]]
+; CHECK-NEXT:    [[P2:%.*]] = bitcast <vscale x 4 x i32>* [[P]] to i32*
+; CHECK-NEXT:    [[GEP2:%.*]] = getelementptr i32, i32* [[P2]], i64 4
+; CHECK-NEXT:    [[LOAD2:%.*]] = load i32, i32* [[GEP2]]
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[LOAD1]], [[LOAD2]]
+; CHECK-NEXT:    ret i32 [[ADD]]
+;
+  %gep1 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 0
+  %load1 = load i32, i32* %gep1
+  %p2 = bitcast <vscale x 4 x i32>* %p to i32*
+  %gep2 = getelementptr i32, i32* %p2, i64 4
+  %load2 = load i32, i32* %gep2 ; <- can not determine at compile-time if %load1 and %load2 are same addr
+  %add = add i32 %load1, %load2
+  ret i32 %add
+}
+
+define i32 @load_clobber_load_gep3(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @load_clobber_load_gep3(
+; CHECK-NEXT:    [[GEP1:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]], i64 1, i64 0
+; CHECK-NEXT:    [[LOAD1:%.*]] = load i32, i32* [[GEP1]]
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[LOAD1]], [[LOAD1]]
+; CHECK-NEXT:    ret i32 [[ADD]]
+;
+  %gep1 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 0
+  %load1 = load i32, i32* %gep1
+  %p2 = bitcast <vscale x 4 x i32>* %p to <vscale x 4 x float>*
+  %gep2 = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %p2, i64 1, i64 0
+  %load2 = load float, float* %gep2 ; <- load to be eliminated
+  %cast = bitcast float %load2 to i32
+  %add = add i32 %load1, %cast
+  ret i32 %add
+}
+
+define <vscale x 4 x i32> @load_clobber_load_fence(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @load_clobber_load_fence(
+; CHECK-NEXT:    [[LOAD1:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    call void asm "", "~{memory}"()
+; CHECK-NEXT:    [[LOAD2:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub <vscale x 4 x i32> [[LOAD1]], [[LOAD2]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[SUB]]
+;
+  %load1 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  call void asm "", "~{memory}"()
+  %load2 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  %sub = sub <vscale x 4 x i32> %load1, %load2
+  ret <vscale x 4 x i32> %sub
+}
+
+define <vscale x 4 x i32> @load_clobber_load_sideeffect(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @load_clobber_load_sideeffect(
+; CHECK-NEXT:    [[LOAD1:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    call void asm sideeffect "", ""()
+; CHECK-NEXT:    [[LOAD2:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]]
+; CHECK-NEXT:    [[ADD:%.*]] = add <vscale x 4 x i32> [[LOAD1]], [[LOAD2]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[ADD]]
+;
+  %load1 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  call void asm sideeffect "", ""()
+  %load2 = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  %add = add <vscale x 4 x i32> %load1, %load2
+  ret <vscale x 4 x i32> %add
+}
+
+; Analyze Load from clobbering Store.
+
+define <vscale x 4 x i32> @store_forward_to_load(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @store_forward_to_load(
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> zeroinitializer
+;
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %p
+  %load = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  ret <vscale x 4 x i32> %load
+}
+
+define <vscale x 4 x i32> @store_forward_to_load_sideeffect(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @store_forward_to_load_sideeffect(
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    call void asm sideeffect "", ""()
+; CHECK-NEXT:    [[LOAD:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]]
+; CHECK-NEXT:    ret <vscale x 4 x i32> [[LOAD]]
+;
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %p
+  call void asm sideeffect "", ""()
+  %load = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p
+  ret <vscale x 4 x i32> %load
+}
+
+define i32 @store_clobber_load() {
+; CHECK-LABEL: @store_clobber_load(
+; CHECK-NEXT:    [[ALLOC:%.*]] = alloca <vscale x 4 x i32>
+; CHECK-NEXT:    store <vscale x 4 x i32> undef, <vscale x 4 x i32>* [[ALLOC]]
+; CHECK-NEXT:    [[PTR:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[ALLOC]], i32 0, i32 1
+; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[PTR]]
+; CHECK-NEXT:    ret i32 [[LOAD]]
+;
+  %alloc = alloca <vscale x 4 x i32>
+  store <vscale x 4 x i32> undef, <vscale x 4 x i32>* %alloc
+  %ptr = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %alloc, i32 0, i32 1
+  %load = load i32, i32* %ptr
+  ret i32 %load
+}
+
+; Analyze Load from clobbering MemInst.
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1)
+
+define i32 @memset_clobber_load(<vscale x 4 x i32> *%p) {
+; CHECK-LABEL: @memset_clobber_load(
+; CHECK-NEXT:    [[CONV:%.*]] = bitcast <vscale x 4 x i32>* [[P:%.*]] to i8*
+; CHECK-NEXT:    tail call void @llvm.memset.p0i8.i64(i8* [[CONV]], i8 1, i64 200, i1 false)
+; CHECK-NEXT:    ret i32 16843009
+;
+  %conv = bitcast <vscale x 4 x i32>* %p to i8*
+  tail call void @llvm.memset.p0i8.i64(i8* %conv, i8 1, i64 200, i1 false)
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 0, i64 5
+  %load = load i32, i32* %gep
+  ret i32 %load
+}
+
+define i32 @memset_clobber_load_vscaled_base(<vscale x 4 x i32> *%p) {
+; CHECK-LABEL: @memset_clobber_load_vscaled_base(
+; CHECK-NEXT:    [[CONV:%.*]] = bitcast <vscale x 4 x i32>* [[P:%.*]] to i8*
+; CHECK-NEXT:    tail call void @llvm.memset.p0i8.i64(i8* [[CONV]], i8 1, i64 200, i1 false)
+; CHECK-NEXT:    [[GEP:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]], i64 1, i64 1
+; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[GEP]]
+; CHECK-NEXT:    ret i32 [[LOAD]]
+;
+  %conv = bitcast <vscale x 4 x i32>* %p to i8*
+  tail call void @llvm.memset.p0i8.i64(i8* %conv, i8 1, i64 200, i1 false)
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 1
+  %load = load i32, i32* %gep
+  ret i32 %load
+}
+
+define i32 @memset_clobber_load_nonconst_index(<vscale x 4 x i32> *%p, i64 %idx1, i64 %idx2) {
+; CHECK-LABEL: @memset_clobber_load_nonconst_index(
+; CHECK-NEXT:    [[CONV:%.*]] = bitcast <vscale x 4 x i32>* [[P:%.*]] to i8*
+; CHECK-NEXT:    tail call void @llvm.memset.p0i8.i64(i8* [[CONV]], i8 1, i64 200, i1 false)
+; CHECK-NEXT:    [[GEP:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]], i64 [[IDX1:%.*]], i64 [[IDX2:%.*]]
+; CHECK-NEXT:    [[LOAD:%.*]] = load i32, i32* [[GEP]]
+; CHECK-NEXT:    ret i32 [[LOAD]]
+;
+  %conv = bitcast <vscale x 4 x i32>* %p to i8*
+  tail call void @llvm.memset.p0i8.i64(i8* %conv, i8 1, i64 200, i1 false)
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 %idx1, i64 %idx2
+  %load = load i32, i32* %gep
+  ret i32 %load
+}
+
+
+; Load elimination across BBs
+
+define <vscale x 4 x i32>* @load_from_alloc_replaced_with_undef() {
+; CHECK-LABEL: @load_from_alloc_replaced_with_undef(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[A:%.*]] = alloca <vscale x 4 x i32>
+; CHECK-NEXT:    br i1 undef, label [[IF_END:%.*]], label [[IF_THEN:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[A]]
+; CHECK-NEXT:    br label [[IF_END]]
+; CHECK:       if.end:
+; CHECK-NEXT:    ret <vscale x 4 x i32>* [[A]]
+;
+entry:
+  %a = alloca <vscale x 4 x i32>
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %a, i64 0, i64 1
+  %load = load i32, i32* %gep ; <- load to be eliminated
+  %tobool = icmp eq i32 %load, 0 ; <- icmp to be eliminated
+  br i1 %tobool, label %if.end, label %if.then
+
+if.then:
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %a
+  br label %if.end
+
+if.end:
+  ret <vscale x 4 x i32>* %a
+}
+
+define i32 @redundant_load_elimination_1(<vscale x 4 x i32>* %p) {
+; CHECK-LABEL: @redundant_load_elimination_1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[GEP:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]], i64 1, i64 1
+; CHECK-NEXT:    [[LOAD1:%.*]] = load i32, i32* [[GEP]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[LOAD1]], 0
+; CHECK-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[IF_END:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    br label [[IF_END]]
+; CHECK:       if.end:
+; CHECK-NEXT:    ret i32 [[LOAD1]]
+;
+entry:
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 1
+  %load1 = load i32, i32* %gep
+  %cmp = icmp eq i32 %load1, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:
+  %load2 = load i32, i32* %gep ; <- load to be eliminated
+  %add = add i32 %load1, %load2
+  br label %if.end
+
+if.end:
+  %result = phi i32 [ %add, %if.then ], [ %load1, %entry ]
+  ret i32 %result
+}
+
+define void @redundant_load_elimination_2(i1 %c, <vscale x 4 x i32>* %p, i32* %q, <vscale x 4 x i32> %v) {
+; CHECK-LABEL: @redundant_load_elimination_2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[GEP1:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P:%.*]], i64 1, i64 1
+; CHECK-NEXT:    store i32 0, i32* [[GEP1]]
+; CHECK-NEXT:    [[GEP2:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]], i64 1, i64 0
+; CHECK-NEXT:    store i32 1, i32* [[GEP2]]
+; CHECK-NEXT:    br i1 [[C:%.*]], label [[IF_ELSE:%.*]], label [[IF_THEN:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    store i32 0, i32* [[Q:%.*]]
+; CHECK-NEXT:    ret void
+; CHECK:       if.else:
+; CHECK-NEXT:    ret void
+;
+entry:
+  %gep1 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 1
+  store i32 0, i32* %gep1
+  %gep2 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1, i64 0
+  store i32 1, i32* %gep2
+  br i1 %c, label %if.else, label %if.then
+
+if.then:
+  %t = load i32, i32* %gep1 ; <- load to be eliminated
+  store i32 %t, i32* %q
+  ret void
+
+if.else:
+  ret void
+}
+
+; TODO: load in if.then could have been eliminated
+define void @missing_load_elimination(i1 %c, <vscale x 4 x i32>* %p, <vscale x 4 x i32>* %q, <vscale x 4 x i32> %v) {
+; CHECK-LABEL: @missing_load_elimination(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* [[P:%.*]]
+; CHECK-NEXT:    [[P1:%.*]] = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]], i64 1
+; CHECK-NEXT:    store <vscale x 4 x i32> [[V:%.*]], <vscale x 4 x i32>* [[P1]]
+; CHECK-NEXT:    br i1 [[C:%.*]], label [[IF_ELSE:%.*]], label [[IF_THEN:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    [[T:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* [[P]]
+; CHECK-NEXT:    store <vscale x 4 x i32> [[T]], <vscale x 4 x i32>* [[Q:%.*]]
+; CHECK-NEXT:    ret void
+; CHECK:       if.else:
+; CHECK-NEXT:    ret void
+;
+entry:
+  store <vscale x 4 x i32> zeroinitializer, <vscale x 4 x i32>* %p
+  %p1 = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %p, i64 1
+  store <vscale x 4 x i32> %v, <vscale x 4 x i32>* %p1
+  br i1 %c, label %if.else, label %if.then
+
+if.then:
+  %t = load <vscale x 4 x i32>, <vscale x 4 x i32>* %p ; load could be eliminated
+  store <vscale x 4 x i32> %t, <vscale x 4 x i32>* %q
+  ret void
+
+if.else:
+  ret void
+}