[ASAN] Support memory checks on vp.load/store.

[fakepaper56]

Summary

The patch adds new member MaybeEVL into InterestingMemoryOperand to represent
the effective vector length for vp intrinsics. It may be extended for some target intrinsics in the future.

[reames]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

1475

This looks wrong? What does the bitwidth have to do with the *value* of EVL?

1487

I may be missing something here, but isn't EVL just an upper bound on the loop? If so, maybe we shouldn't use for the ForEachLane variant and should instead have a variant which takes End explicitly?

[fakepaper56]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

1487

Agree. I am too conservative to change the original code.

[fakepaper56]

Create a new interface SplitBlockAndInsertForLanes, similar as
SplitBlockAndInsertForEachLane but have input argument to control end index.

[fakepaper56]

Rebase and ping.

[reames]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

1484

I believe this case should be a trunc.

1490

I don't believe there's anything preventing AVL from being 0 here right? If so, this code is wrong. The routine you're calling has a (documented!) assumption that End > 0. To reuse, you need to add the loop guard and remove that assumption.

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

1637 ↗

(On Diff #508110)

It's this call where a dynamically zero End becomes problematic.

[fakepaper56]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

1490

Thank you for pointing my mistake. I am not positive to remove End > 0 assumption from SplitBlockAndInsertSimpleForLoop because it is used in MemorySanitizerVisitor::paintOrigin and there should not a loop guard to test End.

[fakepaper56]

Fix mistakes that Philip pointed.

[fakepaper56]

Rebase and fix test fails.

[fakepaper56]

Rebase and ping.

[fakepaper56]

Rebase and ping.

[fakepaper56]

This update does,

1. Only check whether EVL is zero when EVL is not nullptr. It helps not generate redundant zero check for non-EVL InterestingMemoryOperand.
2. Clean up my patch.

I am sorry for that the quality of the patch was terrible. I guess it was due to
that I was too rushed to land the revision before my military educational
recall. It's reasonable that no one wanted to review the terrible patch.
I promise that I will improve the quality of my succeeding patches in LLVM.

[craig.topper]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

1489

Can we use IB.CreateZExtOrTrunc(EVL, IntptrTy) and remove the ifs?

1493

extacting -> extracting

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

1670 ↗

(On Diff #515260)

Why do we need to keep setting the insertion point every iteration? Will that cause the last index to be checked first?

Is there a test case for constant evl. I didn't see one.

[fakepaper56]

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

1670 ↗

(On Diff #515260)

Why do we need to keep setting the insertion point every iteration?

This is to follow the SplitBlockAndInsertForEachLane(ElementCount, Type *, Instruction *, std::function<void(IRBuilderBase&, Value*)>). And the old one is to follow the legacy code.

Will that cause the last index to be checked first?

I think it won't because the letter index will be inserted to the back of the former index.

Is there a test case for constant evl.

Fixed vector masked.load will have a constant evl here. store.v4f32.1110 in llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll is an example.

[fakepaper56]

Use CreateZExtOrTrunc and fix typos.

[craig.topper]

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

1670 ↗

(On Diff #515260)

Why do we need to keep setting the insertion point every iteration?

This is to follow the SplitBlockAndInsertForEachLane(ElementCount, Type *, Instruction *, std::function<void(IRBuilderBase&, Value*)>). And the old one is to follow the legacy code.

Will that cause the last index to be checked first?

I think it won't because the letter index will be inserted to the back of the former index.

You're right. InsertBefore is still after whatever the previous iteration inserted so it didn't really change the insertion point.

Is there a test case for constant evl.

Fixed vector masked.load will have a constant evl here. store.v4f32.1110 in llvm/test/Instrumentation/AddressSanitizer/asan-masked-load-store.ll is an example.

[kito-cheng]

llvm/test/Instrumentation/AddressSanitizer/asan-vp-load-store.ll

30

Could you take a look to see why it use __asan_storeN, not __asan_store4? I saw D145198 can just generate __asan_store4?

[fakepaper56]

llvm/test/Instrumentation/AddressSanitizer/asan-vp-load-store.ll

30

__asan_load4 needs the input address is 4-bytes aligned. I will refine the test to check that.

[fakepaper56]

Update test to test the aligment of pointer.

[kito-cheng]

LGTM

[fakepaper56]

@reames Do you have another concern about the revision? I will land the revision this weekend if the revision does not have comment needed to address. Thank you.

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizerCommon.h

 public:
   Use *PtrUse;
   bool IsWrite;
   Type *OpType;
   TypeSize TypeStoreSize = TypeSize::Fixed(0);
   MaybeAlign Alignment;
   // The mask Value, if we're looking at a masked load/store.
   Value *MaybeMask;
+  // The EVL Value, if we're looking at a vp intrinsic.
+  Value *MaybeEVL;
 
   InterestingMemoryOperand(Instruction *I, unsigned OperandNo, bool IsWrite,
                            class Type *OpType, MaybeAlign Alignment,
-                           Value *MaybeMask = nullptr)
+                           Value *MaybeMask = nullptr,
+                           Value *MaybeEVL = nullptr)
       : IsWrite(IsWrite), OpType(OpType), Alignment(Alignment),
-        MaybeMask(MaybeMask) {
+        MaybeMask(MaybeMask), MaybeEVL(MaybeEVL) {
     const DataLayout &DL = I->getModule()->getDataLayout();
     TypeStoreSize = DL.getTypeStoreSizeInBits(OpType);
     PtrUse = &I->getOperandUse(OperandNo);
   }
 
   Instruction *getInsn() { return cast<Instruction>(PtrUse->getUser()); }
 
   Value *getPtr() { return PtrUse->get(); }

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

AddressSanitizer(Module &M, const StackSafetyGlobalInfo *SSGI,
       : CompileKernel(ClEnableKasan.getNumOccurrences() > 0 ? ClEnableKasan
                                                             : CompileKernel),
         Recover(ClRecover.getNumOccurrences() > 0 ? ClRecover : Recover),
         UseAfterScope(UseAfterScope || ClUseAfterScope),
         UseAfterReturn(ClUseAfterReturn.getNumOccurrences() ? ClUseAfterReturn
                                                             : UseAfterReturn),
         SSGI(SSGI) {
     C = &(M.getContext());
+    DL = &M.getDataLayout();
     LongSize = M.getDataLayout().getPointerSizeInBits();
     IntptrTy = Type::getIntNTy(*C, LongSize);
     Int8PtrTy = Type::getInt8PtrTy(*C);
     Int32Ty = Type::getInt32Ty(*C);
     TargetTriple = Triple(M.getTargetTriple());
 
     Mapping = getShadowMapping(TargetTriple, LongSize, this->CompileKernel);
 
struct FunctionStateRAII {
 
     ~FunctionStateRAII() {
       Pass->LocalDynamicShadow = nullptr;
       Pass->ProcessedAllocas.clear();
     }
   };
 
   LLVMContext *C;
+  const DataLayout *DL;
   Triple TargetTriple;
   int LongSize;
   bool CompileKernel;
   bool Recover;
   bool UseAfterScope;
   AsanDetectStackUseAfterReturnMode UseAfterReturn;
   Type *IntptrTy;
   Type *Int8PtrTy;
Interesting.emplace_back(I, RMW->getPointerOperandIndex(), true,
                              RMW->getValOperand()->getType(), std::nullopt);
   } else if (AtomicCmpXchgInst *XCHG = dyn_cast<AtomicCmpXchgInst>(I)) {
     if (!ClInstrumentAtomics || ignoreAccess(I, XCHG->getPointerOperand()))
       return;
     Interesting.emplace_back(I, XCHG->getPointerOperandIndex(), true,
                              XCHG->getCompareOperand()->getType(),
                              std::nullopt);
   } else if (auto CI = dyn_cast<CallInst>(I)) {
-    if (CI->getIntrinsicID() == Intrinsic::masked_load ||
-        CI->getIntrinsicID() == Intrinsic::masked_store) {
+    switch (CI->getIntrinsicID()) {
+    case Intrinsic::masked_load:
+    case Intrinsic::masked_store: {
       bool IsWrite = CI->getIntrinsicID() == Intrinsic::masked_store;
       // Masked store has an initial operand for the value.
       unsigned OpOffset = IsWrite ? 1 : 0;
       if (IsWrite ? !ClInstrumentWrites : !ClInstrumentReads)
         return;
 
       auto BasePtr = CI->getOperand(OpOffset);
       if (ignoreAccess(I, BasePtr))
         return;
       Type *Ty = IsWrite ? CI->getArgOperand(0)->getType() : CI->getType();
       MaybeAlign Alignment = Align(1);
       // Otherwise no alignment guarantees. We probably got Undef.
       if (auto *Op = dyn_cast<ConstantInt>(CI->getOperand(1 + OpOffset)))
         Alignment = Op->getMaybeAlignValue();
       Value *Mask = CI->getOperand(2 + OpOffset);
       Interesting.emplace_back(I, OpOffset, IsWrite, Ty, Alignment, Mask);
-    } else {
+      break;
+    }
+    case Intrinsic::vp_load:
+    case Intrinsic::vp_store: {
+      auto *VPI = cast<VPIntrinsic>(CI);
+      unsigned IID = CI->getIntrinsicID();
+      bool IsWrite = IID == Intrinsic::vp_store;
+      if (IsWrite ? !ClInstrumentWrites : !ClInstrumentReads)
+        return;
+      unsigned PtrOpNo = *VPI->getMemoryPointerParamPos(IID);
+      Type *Ty = IsWrite ? CI->getArgOperand(0)->getType() : CI->getType();
+      MaybeAlign Alignment = VPI->getOperand(PtrOpNo)->getPointerAlignment(*DL);
+      Interesting.emplace_back(I, PtrOpNo, IsWrite, Ty, Alignment,
+                               VPI->getMaskParam(),
+                               VPI->getVectorLengthParam());
+      break;
+    }
+    default:
       for (unsigned ArgNo = 0; ArgNo < CI->arg_size(); ArgNo++) {
         if (!ClInstrumentByval || !CI->isByValArgument(ArgNo) ||
             ignoreAccess(I, CI->getArgOperand(ArgNo)))
           continue;
         Type *Ty = CI->getParamByValType(ArgNo);
         Interesting.emplace_back(I, ArgNo, false, Ty, Align(1));
       }
     }
if (!TypeStoreSize.isScalable()) {
     }
   }
   Pass->instrumentUnusualSizeOrAlignment(I, InsertBefore, Addr, TypeStoreSize,
                                          IsWrite, nullptr, UseCalls, Exp);
 }
 
 static void instrumentMaskedLoadOrStore(AddressSanitizer *Pass,
                                         const DataLayout &DL, Type *IntptrTy,
-                                        Value *Mask, Instruction *I,
+                                        Value *Mask, Value *EVL, Instruction *I,
                                         Value *Addr, MaybeAlign Alignment,
                                         unsigned Granularity, Type *OpType,
                                         bool IsWrite, Value *SizeArgument,
                                         bool UseCalls, uint32_t Exp) {
   auto *VTy = cast<VectorType>(OpType);
 
   TypeSize ElemTypeSize = DL.getTypeStoreSizeInBits(VTy->getScalarType());
   auto Zero = ConstantInt::get(IntptrTy, 0);
 
+  // Skip if EVL has wider type then IntptrTy.
+  if (EVL && IntptrTy->getPrimitiveSizeInBits() < 32)
+    return;
+
   SplitBlockAndInsertForEachLane(VTy->getElementCount(), IntptrTy, I,
                                  [&](IRBuilderBase &IRB, Value *Index) {
     Value *MaskElem = IRB.CreateExtractElement(Mask, Index);
     if (auto *MaskElemC = dyn_cast<ConstantInt>(MaskElem)) {
       if (MaskElemC->isZero())
         // No check
         return;
-      // Unconditional check
+      // Only need to check evl.
+      if (EVL) {
+        Value *ExtEVL = IRB.CreateZExt(EVL, IntptrTy);
+        Value *InEVLRange = IRB.CreateICmpULT(Index, ExtEVL);
+        Instruction *ThenTerm = SplitBlockAndInsertIfThen(
+            InEVLRange, &*IRB.GetInsertPoint(), false);
+        IRB.SetInsertPoint(ThenTerm);
+      }
     } else {
-      // Conditional check
-      Instruction *ThenTerm = SplitBlockAndInsertIfThen(MaskElem, &*IRB.GetInsertPoint(), false);
+      // Check evl and mask.
+      Value *IsActive = MaskElem;
+      if (EVL) {
+        Value *ExtEVL = IRB.CreateZExt(EVL, IntptrTy);
+        Value *InEVLRange = IRB.CreateICmpULT(Index, ExtEVL);
+        IsActive = IRB.CreateAnd(IsActive, InEVLRange);
+      }
+      Instruction *ThenTerm =
+          SplitBlockAndInsertIfThen(IsActive, &*IRB.GetInsertPoint(), false);
       IRB.SetInsertPoint(ThenTerm);
     }
 
     Value *InstrumentedAddress = IRB.CreateGEP(VTy, Addr, {Zero, Index});
     doInstrumentAddress(Pass, I, &*IRB.GetInsertPoint(), InstrumentedAddress, Alignment,
                         Granularity, ElemTypeSize, IsWrite, SizeArgument,
                         UseCalls, Exp);
   });
void AddressSanitizer::instrumentMop(ObjectSizeOffsetVisitor &ObjSizeVis,
 
   if (O.IsWrite)
     NumInstrumentedWrites++;
   else
     NumInstrumentedReads++;
 
   unsigned Granularity = 1 << Mapping.Scale;
   if (O.MaybeMask) {
-    instrumentMaskedLoadOrStore(this, DL, IntptrTy, O.MaybeMask, O.getInsn(),
-                                Addr, O.Alignment, Granularity, O.OpType,
-                                O.IsWrite, nullptr, UseCalls, Exp);
+    instrumentMaskedLoadOrStore(this, DL, IntptrTy, O.MaybeMask, O.MaybeEVL,
+                                O.getInsn(), Addr, O.Alignment, Granularity,
+                                O.OpType, O.IsWrite, nullptr, UseCalls, Exp);
   } else {
     doInstrumentAddress(this, O.getInsn(), O.getInsn(), Addr, O.Alignment,
                         Granularity, O.TypeStoreSize, O.IsWrite, nullptr, UseCalls,
                         Exp);
   }
 }
 
 Instruction *AddressSanitizer::generateCrashCode(Instruction *InsertBefore,

llvm/test/Instrumentation/AddressSanitizer/asan-vp-load-store.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=asan -asan-instrumentation-with-call-threshold=0 -S \
+; RUN:     | FileCheck %s
+; RUN: opt < %s -passes=asan -asan-instrumentation-with-call-threshold=0 -asan-instrument-reads=0 -asan-instrument-writes=0 -S \
+; RUN:     | FileCheck %s -check-prefix=DISABLED
+
+; Support ASan instrumentation for constant-mask llvm.vp.{load,store}
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+;;;;;;;;;;;;;;;; STORE
+declare void @llvm.vp.store.v4f32.p0(<4 x float>, ptr, <4 x i1>, i32) argmemonly nounwind
+declare void @llvm.vp.store.v8i32.p0(<8 x i32>, ptr, <8 x i1>, i32) argmemonly nounwind
+declare void @llvm.vp.store.v4p0.p0(<4 x ptr>, ptr, <4 x i1>, i32) argmemonly nounwind
+
+define void @store.v4f32.1110(ptr %p, <4 x float> %arg, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v4f32.1110(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 0, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp ult i64 1, [[TMP7]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[TMP9:%.*]], label [[TMP12:%.*]]
+; CHECK:       9:
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 1
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP11]], i64 4)
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[TMP13:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP14:%.*]] = icmp ult i64 2, [[TMP13]]
+; CHECK-NEXT:    br i1 [[TMP14]], label [[TMP15:%.*]], label [[TMP18:%.*]]
+; CHECK:       15:
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 2
+; CHECK-NEXT:    [[TMP17:%.*]] = ptrtoint ptr [[TMP16]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP17]], i64 4)
+; CHECK-NEXT:    br label [[TMP18]]
+; CHECK:       18:
+; CHECK-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P]], <4 x i1> <i1 true, i1 true, i1 true, i1 false>, i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v4f32.1110(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P:%.*]], <4 x i1> <i1 true, i1 true, i1 true, i1 false>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.v4f32.p0(<4 x float> %arg, ptr %p, <4 x i1> <i1 true, i1 true, i1 true, i1 false>, i32 %evl)
+  ret void
+}
+
+define void @store.v8i32.10010110(ptr %p, <8 x i32> %arg, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v8i32.10010110(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 0, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <8 x i32>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp ult i64 3, [[TMP7]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[TMP9:%.*]], label [[TMP12:%.*]]
+; CHECK:       9:
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP11]], i64 4)
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[TMP13:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP14:%.*]] = icmp ult i64 5, [[TMP13]]
+; CHECK-NEXT:    br i1 [[TMP14]], label [[TMP15:%.*]], label [[TMP18:%.*]]
+; CHECK:       15:
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 5
+; CHECK-NEXT:    [[TMP17:%.*]] = ptrtoint ptr [[TMP16]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP17]], i64 4)
+; CHECK-NEXT:    br label [[TMP18]]
+; CHECK:       18:
+; CHECK-NEXT:    [[TMP19:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP20:%.*]] = icmp ult i64 6, [[TMP19]]
+; CHECK-NEXT:    br i1 [[TMP20]], label [[TMP21:%.*]], label [[TMP24:%.*]]
+; CHECK:       21:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 6
+; CHECK-NEXT:    [[TMP23:%.*]] = ptrtoint ptr [[TMP22]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP23]], i64 4)
+; CHECK-NEXT:    br label [[TMP24]]
+; CHECK:       24:
+; CHECK-NEXT:    tail call void @llvm.vp.store.v8i32.p0(<8 x i32> [[ARG:%.*]], ptr [[P]], <8 x i1> <i1 true, i1 false, i1 false, i1 true, i1 false, i1 true, i1 true, i1 false>, i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v8i32.10010110(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v8i32.p0(<8 x i32> [[ARG:%.*]], ptr [[P:%.*]], <8 x i1> <i1 true, i1 false, i1 false, i1 true, i1 false, i1 true, i1 true, i1 false>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.v8i32.p0(<8 x i32> %arg, ptr %p, <8 x i1> <i1 true, i1 false, i1 false, i1 true, i1 false, i1 true, i1 true, i1 false>, i32 %evl)
+  ret void
+}
+
+define void @store.v4i64.0001(ptr %p, <4 x ptr> %arg, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v4i64.0001(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 3, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x ptr>, ptr [[P:%.*]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP5]], i64 8)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    tail call void @llvm.vp.store.v4p0.p0(<4 x ptr> [[ARG:%.*]], ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v4i64.0001(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v4p0.p0(<4 x ptr> [[ARG:%.*]], ptr [[P:%.*]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.v4p0.p0(<4 x ptr> %arg, ptr %p, <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 %evl)
+  ret void
+}
+
+define void @store.v4f32.variable(ptr %p, <4 x float> %arg, <4 x i1> %mask, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v4f32.variable(
+; CHECK-NEXT:    [[TMP1:%.*]] = extractelement <4 x i1> [[MASK:%.*]], i64 0
+; CHECK-NEXT:    [[TMP2:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp ult i64 0, [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = and i1 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    br i1 [[TMP4]], label [[TMP5:%.*]], label [[TMP8:%.*]]
+; CHECK:       5:
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP7]], i64 4)
+; CHECK-NEXT:    br label [[TMP8]]
+; CHECK:       8:
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <4 x i1> [[MASK]], i64 1
+; CHECK-NEXT:    [[TMP10:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP11:%.*]] = icmp ult i64 1, [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = and i1 [[TMP9]], [[TMP11]]
+; CHECK-NEXT:    br i1 [[TMP12]], label [[TMP13:%.*]], label [[TMP16:%.*]]
+; CHECK:       13:
+; CHECK-NEXT:    [[TMP14:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 1
+; CHECK-NEXT:    [[TMP15:%.*]] = ptrtoint ptr [[TMP14]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP15]], i64 4)
+; CHECK-NEXT:    br label [[TMP16]]
+; CHECK:       16:
+; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <4 x i1> [[MASK]], i64 2
+; CHECK-NEXT:    [[TMP18:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP19:%.*]] = icmp ult i64 2, [[TMP18]]
+; CHECK-NEXT:    [[TMP20:%.*]] = and i1 [[TMP17]], [[TMP19]]
+; CHECK-NEXT:    br i1 [[TMP20]], label [[TMP21:%.*]], label [[TMP24:%.*]]
+; CHECK:       21:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 2
+; CHECK-NEXT:    [[TMP23:%.*]] = ptrtoint ptr [[TMP22]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP23]], i64 4)
+; CHECK-NEXT:    br label [[TMP24]]
+; CHECK:       24:
+; CHECK-NEXT:    [[TMP25:%.*]] = extractelement <4 x i1> [[MASK]], i64 3
+; CHECK-NEXT:    [[TMP26:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP27:%.*]] = icmp ult i64 3, [[TMP26]]
+; CHECK-NEXT:    [[TMP28:%.*]] = and i1 [[TMP25]], [[TMP27]]
+; CHECK-NEXT:    br i1 [[TMP28]], label [[TMP29:%.*]], label [[TMP32:%.*]]
+; CHECK:       29:
+; CHECK-NEXT:    [[TMP30:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP31:%.*]] = ptrtoint ptr [[TMP30]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP31]], i64 4)
+; CHECK-NEXT:    br label [[TMP32]]
+; CHECK:       32:
+; CHECK-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P]], <4 x i1> [[MASK]], i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v4f32.variable(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P:%.*]], <4 x i1> [[MASK:%.*]], i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.v4f32.p0(<4 x float> %arg, ptr %p, <4 x i1> %mask, i32 %evl)
+  ret void
+}
+
+;; Store using two vp.stores, which should instrument them both.
+define void @store.v4f32.1010.split(ptr %p, <4 x float> %arg, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v4f32.1010.split(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 2, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 2
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P]], <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v4f32.1010.split(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG:%.*]], ptr [[P:%.*]], <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.v4f32.p0(<4 x float> %arg, ptr %p, <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 %evl)
+  ret void
+}
+
+;; Store using a vp.store after a full store. Shouldn't instrument the second one.
+define void @store.v4f32.0010.after.full.store(ptr %p, <4 x float> %arg, i32 %evl) sanitize_address {
+; CHECK-LABEL: @store.v4f32.0010.after.full.store(
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    call void @__asan_store16(i64 [[TMP1]])
+; CHECK-NEXT:    store <4 x float> [[ARG:%.*]], ptr [[P]], align 16
+; CHECK-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG]], ptr [[P]], <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 [[EVL:%.*]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @store.v4f32.0010.after.full.store(
+; DISABLED-NEXT:    store <4 x float> [[ARG:%.*]], ptr [[P:%.*]], align 16
+; DISABLED-NEXT:    tail call void @llvm.vp.store.v4f32.p0(<4 x float> [[ARG]], ptr [[P]], <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  store <4 x float> %arg, ptr %p
+  tail call void @llvm.vp.store.v4f32.p0(<4 x float> %arg, ptr %p, <4 x i1> <i1 false, i1 false, i1 true, i1 false>, i32 %evl)
+  ret void
+}
+
+;;;;;;;;;;;;;;;; LOAD
+declare <4 x float> @llvm.vp.load.v4f32.p0(ptr, <4 x i1>, i32) argmemonly nounwind
+declare <8 x i32> @llvm.vp.load.v8i32.p0(ptr, <8 x i1>, i32) argmemonly nounwind
+declare <4 x ptr> @llvm.vp.load.v4p0.p0(ptr, <4 x i1>, i32) argmemonly nounwind
+
+define <8 x i32> @load.v8i32.11100001(ptr %p, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v8i32.11100001(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 0, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <8 x i32>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp ult i64 1, [[TMP7]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[TMP9:%.*]], label [[TMP12:%.*]]
+; CHECK:       9:
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 1
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP11]], i64 4)
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[TMP13:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP14:%.*]] = icmp ult i64 2, [[TMP13]]
+; CHECK-NEXT:    br i1 [[TMP14]], label [[TMP15:%.*]], label [[TMP18:%.*]]
+; CHECK:       15:
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 2
+; CHECK-NEXT:    [[TMP17:%.*]] = ptrtoint ptr [[TMP16]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP17]], i64 4)
+; CHECK-NEXT:    br label [[TMP18]]
+; CHECK:       18:
+; CHECK-NEXT:    [[TMP19:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP20:%.*]] = icmp ult i64 7, [[TMP19]]
+; CHECK-NEXT:    br i1 [[TMP20]], label [[TMP21:%.*]], label [[TMP24:%.*]]
+; CHECK:       21:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr <8 x i32>, ptr [[P]], i64 0, i64 7
+; CHECK-NEXT:    [[TMP23:%.*]] = ptrtoint ptr [[TMP22]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP23]], i64 4)
+; CHECK-NEXT:    br label [[TMP24]]
+; CHECK:       24:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <8 x i32> @llvm.vp.load.v8i32.p0(ptr [[P]], <8 x i1> <i1 true, i1 true, i1 true, i1 false, i1 false, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; CHECK-NEXT:    ret <8 x i32> [[RES]]
+;
+; DISABLED-LABEL: @load.v8i32.11100001(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <8 x i32> @llvm.vp.load.v8i32.p0(ptr [[P:%.*]], <8 x i1> <i1 true, i1 true, i1 true, i1 false, i1 false, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <8 x i32> [[RES]]
+;
+  %res = tail call <8 x i32> @llvm.vp.load.v8i32.p0(ptr %p, <8 x i1> <i1 true, i1 true, i1 true, i1 false, i1 false, i1 false, i1 false, i1 true>, i32 %evl)
+  ret <8 x i32> %res
+}
+
+define <4 x float> @load.v4f32.1001(ptr %p, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v4f32.1001(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 0, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp ult i64 3, [[TMP7]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[TMP9:%.*]], label [[TMP12:%.*]]
+; CHECK:       9:
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP11]], i64 4)
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 true, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; CHECK-NEXT:    ret <4 x float> [[RES]]
+;
+; DISABLED-LABEL: @load.v4f32.1001(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P:%.*]], <4 x i1> <i1 true, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <4 x float> [[RES]]
+;
+  %res = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr %p, <4 x i1> <i1 true, i1 false, i1 false, i1 true>, i32 %evl)
+  ret <4 x float> %res
+}
+
+define <4 x ptr> @load.v4i64.0001(ptr %p, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v4i64.0001(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 3, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x ptr>, ptr [[P:%.*]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP5]], i64 8)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <4 x ptr> @llvm.vp.load.v4p0.p0(ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; CHECK-NEXT:    ret <4 x ptr> [[RES]]
+;
+; DISABLED-LABEL: @load.v4i64.0001(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <4 x ptr> @llvm.vp.load.v4p0.p0(ptr [[P:%.*]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <4 x ptr> [[RES]]
+;
+  %res = tail call <4 x ptr> @llvm.vp.load.v4p0.p0(ptr %p, <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 %evl)
+  ret <4 x ptr> %res
+}
+
+define <4 x float> @load.v4f32.variable(ptr %p, <4 x float> %arg, <4 x i1> %mask, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v4f32.variable(
+; CHECK-NEXT:    [[TMP1:%.*]] = extractelement <4 x i1> [[MASK:%.*]], i64 0
+; CHECK-NEXT:    [[TMP2:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp ult i64 0, [[TMP2]]
+; CHECK-NEXT:    [[TMP4:%.*]] = and i1 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    br i1 [[TMP4]], label [[TMP5:%.*]], label [[TMP8:%.*]]
+; CHECK:       5:
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP7:%.*]] = ptrtoint ptr [[TMP6]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP7]], i64 4)
+; CHECK-NEXT:    br label [[TMP8]]
+; CHECK:       8:
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <4 x i1> [[MASK]], i64 1
+; CHECK-NEXT:    [[TMP10:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP11:%.*]] = icmp ult i64 1, [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = and i1 [[TMP9]], [[TMP11]]
+; CHECK-NEXT:    br i1 [[TMP12]], label [[TMP13:%.*]], label [[TMP16:%.*]]
+; CHECK:       13:
+; CHECK-NEXT:    [[TMP14:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 1
+; CHECK-NEXT:    [[TMP15:%.*]] = ptrtoint ptr [[TMP14]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP15]], i64 4)
+; CHECK-NEXT:    br label [[TMP16]]
+; CHECK:       16:
+; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <4 x i1> [[MASK]], i64 2
+; CHECK-NEXT:    [[TMP18:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP19:%.*]] = icmp ult i64 2, [[TMP18]]
+; CHECK-NEXT:    [[TMP20:%.*]] = and i1 [[TMP17]], [[TMP19]]
+; CHECK-NEXT:    br i1 [[TMP20]], label [[TMP21:%.*]], label [[TMP24:%.*]]
+; CHECK:       21:
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 2
+; CHECK-NEXT:    [[TMP23:%.*]] = ptrtoint ptr [[TMP22]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP23]], i64 4)
+; CHECK-NEXT:    br label [[TMP24]]
+; CHECK:       24:
+; CHECK-NEXT:    [[TMP25:%.*]] = extractelement <4 x i1> [[MASK]], i64 3
+; CHECK-NEXT:    [[TMP26:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP27:%.*]] = icmp ult i64 3, [[TMP26]]
+; CHECK-NEXT:    [[TMP28:%.*]] = and i1 [[TMP25]], [[TMP27]]
+; CHECK-NEXT:    br i1 [[TMP28]], label [[TMP29:%.*]], label [[TMP32:%.*]]
+; CHECK:       29:
+; CHECK-NEXT:    [[TMP30:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP31:%.*]] = ptrtoint ptr [[TMP30]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP31]], i64 4)
+; CHECK-NEXT:    br label [[TMP32]]
+; CHECK:       32:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> [[MASK]], i32 [[EVL]])
+; CHECK-NEXT:    ret <4 x float> [[RES]]
+;
+; DISABLED-LABEL: @load.v4f32.variable(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P:%.*]], <4 x i1> [[MASK:%.*]], i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <4 x float> [[RES]]
+;
+  %res = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr %p, <4 x i1> %mask, i32 %evl)
+  ret <4 x float> %res
+}
+
+;; Load using two vp.loads, which should instrument them both.
+define <4 x float> @load.v4f32.1001.split(ptr %p, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v4f32.1001.split(
+; CHECK-NEXT:    [[TMP1:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult i64 0, [[TMP1]]
+; CHECK-NEXT:    br i1 [[TMP2]], label [[TMP3:%.*]], label [[TMP6:%.*]]
+; CHECK:       3:
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr <4 x float>, ptr [[P:%.*]], i64 0, i64 0
+; CHECK-NEXT:    [[TMP5:%.*]] = ptrtoint ptr [[TMP4]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP5]], i64 4)
+; CHECK-NEXT:    br label [[TMP6]]
+; CHECK:       6:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 true, i1 false, i1 false, i1 false>, i32 [[EVL]])
+; CHECK-NEXT:    [[TMP7:%.*]] = zext i32 [[EVL]] to i64
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp ult i64 3, [[TMP7]]
+; CHECK-NEXT:    br i1 [[TMP8]], label [[TMP9:%.*]], label [[TMP12:%.*]]
+; CHECK:       9:
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr <4 x float>, ptr [[P]], i64 0, i64 3
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP11]], i64 4)
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[RES2:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; CHECK-NEXT:    ret <4 x float> [[RES2]]
+;
+; DISABLED-LABEL: @load.v4f32.1001.split(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P:%.*]], <4 x i1> <i1 true, i1 false, i1 false, i1 false>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    [[RES2:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL]])
+; DISABLED-NEXT:    ret <4 x float> [[RES2]]
+;
+  %res = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr %p, <4 x i1> <i1 true, i1 false, i1 false, i1 false>, i32 %evl)
+  %res2 = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr %p, <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 %evl)
+  ret <4 x float> %res2
+}
+
+;; Load using a vp.load after a full load. Shouldn't instrument the second one.
+define <4 x float> @load.v4f32.1001.after.full.load(ptr %p, i32 %evl) sanitize_address {
+; CHECK-LABEL: @load.v4f32.1001.after.full.load(
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[P:%.*]] to i64
+; CHECK-NEXT:    call void @__asan_load16(i64 [[TMP1]])
+; CHECK-NEXT:    [[RES:%.*]] = load <4 x float>, ptr [[P]], align 16
+; CHECK-NEXT:    [[RES2:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; CHECK-NEXT:    ret <4 x float> [[RES2]]
+;
+; DISABLED-LABEL: @load.v4f32.1001.after.full.load(
+; DISABLED-NEXT:    [[RES:%.*]] = load <4 x float>, ptr [[P:%.*]], align 16
+; DISABLED-NEXT:    [[RES2:%.*]] = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr [[P]], <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <4 x float> [[RES2]]
+;
+  %res = load <4 x float>, ptr %p
+  %res2 = tail call <4 x float> @llvm.vp.load.v4f32.p0(ptr %p, <4 x i1> <i1 false, i1 false, i1 false, i1 true>, i32 %evl)
+  ret <4 x float> %res2
+}
+
+;; Scalable vector tests
+;; ---------------------------
+declare <vscale x 4 x float> @llvm.vp.load.nxv4f32.p0(ptr, <vscale x 4 x i1>, i32)
+declare void @llvm.vp.store.nxv4f32.p0(<vscale x 4 x float>, ptr, <vscale x 4 x i1>, i32)
+
+define <vscale x 4 x float> @scalable.load.nxv4f32(ptr %p, <vscale x 4 x i1> %mask, i32 %evl) sanitize_address {
+; CHECK-LABEL: @scalable.load.nxv4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 4
+; CHECK-NEXT:    br label [[DOTSPLIT:%.*]]
+; CHECK:       .split:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ [[IV_NEXT:%.*]], [[TMP10:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <vscale x 4 x i1> [[MASK:%.*]], i64 [[IV]]
+; CHECK-NEXT:    [[TMP4:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = icmp ult i64 [[IV]], [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = and i1 [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP10]]
+; CHECK:       7:
+; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr <vscale x 4 x float>, ptr [[P:%.*]], i64 0, i64 [[IV]]
+; CHECK-NEXT:    [[TMP9:%.*]] = ptrtoint ptr [[TMP8]] to i64
+; CHECK-NEXT:    call void @__asan_loadN(i64 [[TMP9]], i64 4)
+; CHECK-NEXT:    br label [[TMP10]]
+; CHECK:       10:
+; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
+; CHECK-NEXT:    [[IV_CHECK:%.*]] = icmp eq i64 [[IV_NEXT]], [[TMP2]]
+; CHECK-NEXT:    br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
+; CHECK:       .split.split:
+; CHECK-NEXT:    [[RES:%.*]] = tail call <vscale x 4 x float> @llvm.vp.load.nxv4f32.p0(ptr [[P]], <vscale x 4 x i1> [[MASK]], i32 [[EVL]])
+; CHECK-NEXT:    ret <vscale x 4 x float> [[RES]]
+;
+; DISABLED-LABEL: @scalable.load.nxv4f32(
+; DISABLED-NEXT:    [[RES:%.*]] = tail call <vscale x 4 x float> @llvm.vp.load.nxv4f32.p0(ptr [[P:%.*]], <vscale x 4 x i1> [[MASK:%.*]], i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret <vscale x 4 x float> [[RES]]
+;
+  %res = tail call <vscale x 4 x float> @llvm.vp.load.nxv4f32.p0(ptr %p, <vscale x 4 x i1> %mask, i32 %evl)
+  ret <vscale x 4 x float> %res
+}
+
+define void @scalable.store.nxv4f32(ptr %p, <vscale x 4 x float> %arg, <vscale x 4 x i1> %mask, i32 %evl) sanitize_address {
+; CHECK-LABEL: @scalable.store.nxv4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP2:%.*]] = mul i64 [[TMP1]], 4
+; CHECK-NEXT:    br label [[DOTSPLIT:%.*]]
+; CHECK:       .split:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ [[IV_NEXT:%.*]], [[TMP10:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <vscale x 4 x i1> [[MASK:%.*]], i64 [[IV]]
+; CHECK-NEXT:    [[TMP4:%.*]] = zext i32 [[EVL:%.*]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = icmp ult i64 [[IV]], [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = and i1 [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    br i1 [[TMP6]], label [[TMP7:%.*]], label [[TMP10]]
+; CHECK:       7:
+; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr <vscale x 4 x float>, ptr [[P:%.*]], i64 0, i64 [[IV]]
+; CHECK-NEXT:    [[TMP9:%.*]] = ptrtoint ptr [[TMP8]] to i64
+; CHECK-NEXT:    call void @__asan_storeN(i64 [[TMP9]], i64 4)
+; CHECK-NEXT:    br label [[TMP10]]
+; CHECK:       10:
+; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
+; CHECK-NEXT:    [[IV_CHECK:%.*]] = icmp eq i64 [[IV_NEXT]], [[TMP2]]
+; CHECK-NEXT:    br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
+; CHECK:       .split.split:
+; CHECK-NEXT:    tail call void @llvm.vp.store.nxv4f32.p0(<vscale x 4 x float> [[ARG:%.*]], ptr [[P]], <vscale x 4 x i1> [[MASK]], i32 [[EVL]])
+; CHECK-NEXT:    ret void
+;
+; DISABLED-LABEL: @scalable.store.nxv4f32(
+; DISABLED-NEXT:    tail call void @llvm.vp.store.nxv4f32.p0(<vscale x 4 x float> [[ARG:%.*]], ptr [[P:%.*]], <vscale x 4 x i1> [[MASK:%.*]], i32 [[EVL:%.*]])
+; DISABLED-NEXT:    ret void
+;
+  tail call void @llvm.vp.store.nxv4f32.p0(<vscale x 4 x float> %arg, ptr %p, <vscale x 4 x i1> %mask, i32 %evl)
+  ret void
+}