This is an archive of the discontinued LLVM Phabricator instance.

[MSAN] Support load and stores of scalable vector types
ClosedPublic

Authored by reames on Mar 15 2023, 10:55 AM.

Download Raw Diff

Details

Reviewers

craig.topper
asb
kito-cheng
• jnspaulsson
MaskRay
vitalybuka

Commits

rG5bcb4c4da99c: [MSAN] Support load and stores of scalable vector types

Summary

This adds support for scalable vector types - at least far enough to get basic load and store cases working. It turns out that load/store without origin tracking already worked; I apparently got that working with one of the pre patches to use TypeSize utilities and didn't notice. The code changes here are required to enable origin tracking.

For origin tracking, a 4 byte value - the origin - is broadcast into a shadow region whose size exactly matches the type being accessed. This origin is only written if the shadow value is non-zero. The details of how shadow is computed from the original value being stored aren't relevant for this patch.

The code changes involve two related primitives.

First, we need to be able to perform that broadcast into a scalable sized memory region. This requires the use of a loop, and appropriate bound. The fixed size case optimizes with larger stores and alignment; I did not bother with that for the scalable case for now. We can optimize this codepath later if desired.

Second, we need a way to test if the shadow is zero. The mechanism for this in the code is to convert the shadow value into a scalar, and then zero check that. There's an assumption that this scalar is zero exactly when all elements of the shadow value are zero. As a result, we use an OR reduction on the scalable vector. This is analogous to how e.g. an array is handled. I landed a bunch of cleanup changes to remove other direct uses of the scalar conversion to convince myself there were no other undocumented invariants.

Diff Detail

Repository: rG LLVM Github Monorepo

Event Timeline

reames created this revision.Mar 15 2023, 10:55 AM

Herald added a project: Restricted Project. · View Herald TranscriptMar 15 2023, 10:55 AM

Herald added subscribers: Enna1, bollu, hiraditya, mcrosier. · View Herald Transcript

reames requested review of this revision.Mar 15 2023, 10:55 AM

Herald added a project: Restricted Project. · View Herald TranscriptMar 15 2023, 10:55 AM

Herald added a subscriber: alextsao1999. · View Herald Transcript

Harbormaster completed remote builds in B219691: Diff 505563.Mar 15 2023, 1:56 PM

vitalybuka added 1 blocking reviewer(s): vitalybuka.Mar 20 2023, 5:31 PM

vitalybuka added inline comments.

llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll
51	it should be sanitize_memory can you please fix and regenerate the test separately?
530	I am confused, how passes=msan created asan calls?

vitalybuka added inline comments.Mar 20 2023, 7:15 PM

llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll
627	could you include these cases into precommited tests?

reames mentioned this in rG473e9adb84c2: [MSAN] Update vector load/store tests to use proper attribute.Mar 22 2023, 2:03 PM

reames added inline comments.Mar 22 2023, 2:10 PM

llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll
51	Fixed in 473e9adb84c29764da51e4d2a995fce9b2e2ffa5, thanks for noticing.
530	This looks like a copy paste mistake on my part. update_test_checks.py doesn't strip out check lines which don't correspond to a run line, and I'd copied these from the ASAN test. Will fix and refresh.
627	Can't. We crash on scalable input types. That's the whole point of the patch.

Address review comment

Harbormaster completed remote builds in B221129: Diff 507498.Mar 22 2023, 2:53 PM

Thanks!

This revision is now accepted and ready to land.Mar 22 2023, 9:17 PM

This revision was landed with ongoing or failed builds.Mar 23 2023, 7:38 AM

Closed by commit rG5bcb4c4da99c: [MSAN] Support load and stores of scalable vector types (authored by reames). · Explain Why

This revision was automatically updated to reflect the committed changes.

reames added a commit: rG5bcb4c4da99c: [MSAN] Support load and stores of scalable vector types.

Revision Contents

Path

Size

llvm/

lib/

Transforms/

Instrumentation/

MemorySanitizer.cpp

20 lines

test/

Instrumentation/

MemorySanitizer/

vector-load-store.ll

509 lines

Diff 507739

llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp

Show First 20 Lines • Show All 1,177 Lines • ▼ Show 20 Lines	Value originToIntptr(IRBuilder<> &IRB, Value Origin) {
assert(IntptrSize == kOriginSize * 2);		assert(IntptrSize == kOriginSize * 2);
Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);		Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));		return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
}		}

/// Fill memory range with the given origin value.		/// Fill memory range with the given origin value.
void paintOrigin(IRBuilder<> &IRB, Value Origin, Value OriginPtr,		void paintOrigin(IRBuilder<> &IRB, Value Origin, Value OriginPtr,
TypeSize TS, Align Alignment) {		TypeSize TS, Align Alignment) {
unsigned Size = TS.getFixedValue();
const DataLayout &DL = F.getParent()->getDataLayout();		const DataLayout &DL = F.getParent()->getDataLayout();
const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);		const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);		unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
assert(IntptrAlignment >= kMinOriginAlignment);		assert(IntptrAlignment >= kMinOriginAlignment);
assert(IntptrSize >= kOriginSize);		assert(IntptrSize >= kOriginSize);

		// Note: The loop based formation works for fixed length vectors too,
		// however we prefer to unroll and specialize alignment below.
		if (TS.isScalable()) {
		Value *Size = IRB.CreateTypeSize(IRB.getInt32Ty(), TS);
		Value *RoundUp = IRB.CreateAdd(Size, IRB.getInt32(kOriginSize - 1));
		Value *End = IRB.CreateUDiv(RoundUp, IRB.getInt32(kOriginSize));
		auto [InsertPt, Index] =
		SplitBlockAndInsertSimpleForLoop(End, &*IRB.GetInsertPoint());
		IRB.SetInsertPoint(InsertPt);

		Value *GEP = IRB.CreateGEP(MS.OriginTy, OriginPtr, Index);
		IRB.CreateAlignedStore(Origin, GEP, kMinOriginAlignment);
		return;
		}

		unsigned Size = TS.getFixedValue();

unsigned Ofs = 0;		unsigned Ofs = 0;
Align CurrentAlignment = Alignment;		Align CurrentAlignment = Alignment;
if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {		if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
Value *IntptrOrigin = originToIntptr(IRB, Origin);		Value *IntptrOrigin = originToIntptr(IRB, Origin);
Value *IntptrOriginPtr =		Value *IntptrOriginPtr =
IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));		IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
for (unsigned i = 0; i < Size / IntptrSize; ++i) {		for (unsigned i = 0; i < Size / IntptrSize; ++i) {
Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)		Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
▲ Show 20 Lines • Show All 369 Lines • ▼ Show 20 Lines	struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
/// shadow may not necessarily have the same bit width as the input		/// shadow may not necessarily have the same bit width as the input
/// value, but it will always be comparable to zero.		/// value, but it will always be comparable to zero.
Value convertShadowToScalar(Value V, IRBuilder<> &IRB) {		Value convertShadowToScalar(Value V, IRBuilder<> &IRB) {
if (StructType *Struct = dyn_cast<StructType>(V->getType()))		if (StructType *Struct = dyn_cast<StructType>(V->getType()))
return collapseStructShadow(Struct, V, IRB);		return collapseStructShadow(Struct, V, IRB);
if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))		if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
return collapseArrayShadow(Array, V, IRB);		return collapseArrayShadow(Array, V, IRB);
if (isa<VectorType>(V->getType())) {		if (isa<VectorType>(V->getType())) {
		if (isa<ScalableVectorType>(V->getType()))
		return convertShadowToScalar(IRB.CreateOrReduce(V), IRB);
unsigned BitWidth =		unsigned BitWidth =
V->getType()->getPrimitiveSizeInBits().getFixedValue();		V->getType()->getPrimitiveSizeInBits().getFixedValue();
return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));		return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
}		}
return V;		return V;
}		}

// Convert a scalar value to an i1 by comparing with 0		// Convert a scalar value to an i1 by comparing with 0
▲ Show 20 Lines • Show All 4,168 Lines • Show Last 20 Lines

llvm/test/Instrumentation/MemorySanitizer/vector-load-store.ll

	Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Lines
	; ORIGINS-NEXT: [[_MSLD:%.*]] = load <1 x i32>, ptr [[TMP4]], align 4			; ORIGINS-NEXT: [[_MSLD:%.*]] = load <1 x i32>, ptr [[TMP4]], align 4
	; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 4			; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 4
	; ORIGINS-NEXT: ret void			; ORIGINS-NEXT: ret void
	;			;
	load <1 x i32>, ptr %p			load <1 x i32>, ptr %p
	ret void			ret void
	}			}

	define void @load.v2i32(ptr %p) sanitize_memory {			define void @load.v2i32(ptr %p) sanitize_memory {
				vitalybukaUnsubmitted Not Done Reply Inline Actions it should be sanitize_memory can you please fix and regenerate the test separately? vitalybuka: it should be sanitize_memory can you please fix and regenerate the test separately?
				reamesAuthorUnsubmitted Done Reply Inline Actions Fixed in 473e9adb84c29764da51e4d2a995fce9b2e2ffa5, thanks for noticing. reames: Fixed in 473e9adb84c29764da51e4d2a995fce9b2e2ffa5, thanks for noticing.
	; CHECK-LABEL: @load.v2i32(			; CHECK-LABEL: @load.v2i32(
	; CHECK-NEXT: call void @llvm.donothing()			; CHECK-NEXT: call void @llvm.donothing()
	; CHECK-NEXT: [[TMP1:%.]] = load <2 x i32>, ptr [[P:%.]], align 8			; CHECK-NEXT: [[TMP1:%.]] = load <2 x i32>, ptr [[P:%.]], align 8
	; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64			; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
	; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080			; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
	; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr			; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
	; CHECK-NEXT: [[_MSLD:%.*]] = load <2 x i32>, ptr [[TMP4]], align 8			; CHECK-NEXT: [[_MSLD:%.*]] = load <2 x i32>, ptr [[TMP4]], align 8
	; CHECK-NEXT: ret void			; CHECK-NEXT: ret void
	▲ Show 20 Lines • Show All 357 Lines • ▼ Show 20 Lines
	; ORIGINS-NEXT: store <16 x i32> zeroinitializer, ptr [[TMP3]], align 64			; ORIGINS-NEXT: store <16 x i32> zeroinitializer, ptr [[TMP3]], align 64
	; ORIGINS-NEXT: store <16 x i32> zeroinitializer, ptr [[P]], align 64			; ORIGINS-NEXT: store <16 x i32> zeroinitializer, ptr [[P]], align 64
	; ORIGINS-NEXT: ret void			; ORIGINS-NEXT: ret void
	;			;
	store <16 x i32> zeroinitializer, ptr %p			store <16 x i32> zeroinitializer, ptr %p
	ret void			ret void
	}			}

				define void @load.nxv1i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @load.nxv1i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = load <vscale x 1 x i32>, ptr [[P:%.]], align 4
				; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; CHECK-NEXT: [[_MSLD:%.*]] = load <vscale x 1 x i32>, ptr [[TMP4]], align 4
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @load.nxv1i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = load <vscale x 1 x i32>, ptr [[P:%.]], align 4
				; ADDR-NEXT: [[TMP5:%.*]] = ptrtoint ptr [[P]] to i64
				; ADDR-NEXT: [[TMP6:%.*]] = xor i64 [[TMP5]], 87960930222080
				; ADDR-NEXT: [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
				; ADDR-NEXT: [[_MSLD:%.*]] = load <vscale x 1 x i32>, ptr [[TMP7]], align 4
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @load.nxv1i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = load <vscale x 1 x i32>, ptr [[P:%.]], align 4
				; ORIGINS-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; ORIGINS-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; ORIGINS-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; ORIGINS-NEXT: [[TMP5:%.*]] = add i64 [[TMP3]], 17592186044416
				; ORIGINS-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ORIGINS-NEXT: [[_MSLD:%.*]] = load <vscale x 1 x i32>, ptr [[TMP4]], align 4
				; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 4
				; ORIGINS-NEXT: ret void
				;
				load <vscale x 1 x i32>, ptr %p
				ret void
				}

				define void @load.nxv2i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @load.nxv2i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = load <vscale x 2 x i32>, ptr [[P:%.]], align 8
				; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; CHECK-NEXT: [[_MSLD:%.*]] = load <vscale x 2 x i32>, ptr [[TMP4]], align 8
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @load.nxv2i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = load <vscale x 2 x i32>, ptr [[P:%.]], align 8
				; ADDR-NEXT: [[TMP5:%.*]] = ptrtoint ptr [[P]] to i64
				; ADDR-NEXT: [[TMP6:%.*]] = xor i64 [[TMP5]], 87960930222080
				; ADDR-NEXT: [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
				; ADDR-NEXT: [[_MSLD:%.*]] = load <vscale x 2 x i32>, ptr [[TMP7]], align 8
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @load.nxv2i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = load <vscale x 2 x i32>, ptr [[P:%.]], align 8
				; ORIGINS-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; ORIGINS-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; ORIGINS-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; ORIGINS-NEXT: [[TMP5:%.*]] = add i64 [[TMP3]], 17592186044416
				; ORIGINS-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ORIGINS-NEXT: [[_MSLD:%.*]] = load <vscale x 2 x i32>, ptr [[TMP4]], align 8
				; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 8
				; ORIGINS-NEXT: ret void
				;
				load <vscale x 2 x i32>, ptr %p
				ret void
				}

				define void @load.nxv4i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @load.nxv4i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = load <vscale x 4 x i32>, ptr [[P:%.]], align 16
				; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; CHECK-NEXT: [[_MSLD:%.*]] = load <vscale x 4 x i32>, ptr [[TMP4]], align 16
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @load.nxv4i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = load <vscale x 4 x i32>, ptr [[P:%.]], align 16
				; ADDR-NEXT: [[TMP5:%.*]] = ptrtoint ptr [[P]] to i64
				; ADDR-NEXT: [[TMP6:%.*]] = xor i64 [[TMP5]], 87960930222080
				vitalybukaUnsubmitted Not Done Reply Inline Actions I am confused, how passes=msan created asan calls? vitalybuka: I am confused, how passes=msan created asan calls?
				reamesAuthorUnsubmitted Done Reply Inline Actions This looks like a copy paste mistake on my part. update_test_checks.py doesn't strip out check lines which don't correspond to a run line, and I'd copied these from the ASAN test. Will fix and refresh. reames: This looks like a copy paste mistake on my part. update_test_checks.py doesn't strip out check…
				; ADDR-NEXT: [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
				; ADDR-NEXT: [[_MSLD:%.*]] = load <vscale x 4 x i32>, ptr [[TMP7]], align 16
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @load.nxv4i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = load <vscale x 4 x i32>, ptr [[P:%.]], align 16
				; ORIGINS-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; ORIGINS-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; ORIGINS-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; ORIGINS-NEXT: [[TMP5:%.*]] = add i64 [[TMP3]], 17592186044416
				; ORIGINS-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ORIGINS-NEXT: [[_MSLD:%.*]] = load <vscale x 4 x i32>, ptr [[TMP4]], align 16
				; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 16
				; ORIGINS-NEXT: ret void
				;
				load <vscale x 4 x i32>, ptr %p
				ret void
				}

				define void @load.nxv8i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @load.nxv8i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = load <vscale x 8 x i32>, ptr [[P:%.]], align 32
				; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; CHECK-NEXT: [[_MSLD:%.*]] = load <vscale x 8 x i32>, ptr [[TMP4]], align 32
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @load.nxv8i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = load <vscale x 8 x i32>, ptr [[P:%.]], align 32
				; ADDR-NEXT: [[TMP5:%.*]] = ptrtoint ptr [[P]] to i64
				; ADDR-NEXT: [[TMP6:%.*]] = xor i64 [[TMP5]], 87960930222080
				; ADDR-NEXT: [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
				; ADDR-NEXT: [[_MSLD:%.*]] = load <vscale x 8 x i32>, ptr [[TMP7]], align 32
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @load.nxv8i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = load <vscale x 8 x i32>, ptr [[P:%.]], align 32
				; ORIGINS-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; ORIGINS-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; ORIGINS-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; ORIGINS-NEXT: [[TMP5:%.*]] = add i64 [[TMP3]], 17592186044416
				; ORIGINS-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ORIGINS-NEXT: [[_MSLD:%.*]] = load <vscale x 8 x i32>, ptr [[TMP4]], align 32
				; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 32
				; ORIGINS-NEXT: ret void
				;
				load <vscale x 8 x i32>, ptr %p
				ret void
				}

				define void @load.nxv16i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @load.nxv16i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = load <vscale x 16 x i32>, ptr [[P:%.]], align 64
				; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; CHECK-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; CHECK-NEXT: [[_MSLD:%.*]] = load <vscale x 16 x i32>, ptr [[TMP4]], align 64
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @load.nxv16i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = load <vscale x 16 x i32>, ptr [[P:%.]], align 64
				; ADDR-NEXT: [[TMP5:%.*]] = ptrtoint ptr [[P]] to i64
				; ADDR-NEXT: [[TMP6:%.*]] = xor i64 [[TMP5]], 87960930222080
				; ADDR-NEXT: [[TMP7:%.*]] = inttoptr i64 [[TMP6]] to ptr
				; ADDR-NEXT: [[_MSLD:%.*]] = load <vscale x 16 x i32>, ptr [[TMP7]], align 64
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @load.nxv16i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = load <vscale x 16 x i32>, ptr [[P:%.]], align 64
				; ORIGINS-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P]] to i64
				; ORIGINS-NEXT: [[TMP3:%.*]] = xor i64 [[TMP2]], 87960930222080
				; ORIGINS-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
				; ORIGINS-NEXT: [[TMP5:%.*]] = add i64 [[TMP3]], 17592186044416
				; ORIGINS-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ORIGINS-NEXT: [[_MSLD:%.*]] = load <vscale x 16 x i32>, ptr [[TMP4]], align 64
				vitalybukaUnsubmitted Not Done Reply Inline Actions could you include these cases into precommited tests? vitalybuka: could you include these cases into precommited tests?
				reamesAuthorUnsubmitted Done Reply Inline Actions Can't. We crash on scalable input types. That's the whole point of the patch. reames: Can't. We crash on scalable input types. That's the whole point of the patch.
				; ORIGINS-NEXT: [[TMP7:%.*]] = load i32, ptr [[TMP6]], align 64
				; ORIGINS-NEXT: ret void
				;
				load <vscale x 16 x i32>, ptr %p
				ret void
				}


				define void @store.nxv1i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @store.nxv1i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; CHECK-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; CHECK-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[TMP3]], align 4
				; CHECK-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[P]], align 4
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @store.nxv1i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ADDR-NEXT: [[TMP5:%.*]] = xor i64 [[TMP4]], 87960930222080
				; ADDR-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ADDR-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[TMP6]], align 4
				; ADDR-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[P]], align 4
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @store.nxv1i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ORIGINS-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; ORIGINS-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; ORIGINS-NEXT: [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
				; ORIGINS-NEXT: [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
				; ORIGINS-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[TMP3]], align 4
				; ORIGINS-NEXT: [[TMP6:%.*]] = call i32 @llvm.vector.reduce.or.nxv1i32(<vscale x 1 x i32> zeroinitializer)
				; ORIGINS-NEXT: [[_MSCMP:%.*]] = icmp ne i32 [[TMP6]], 0
				; ORIGINS-NEXT: br i1 [[_MSCMP]], label [[TMP7:%.]], label [[TMP13:%.]], !prof [[PROF0:![0-9]+]]
				; ORIGINS: 7:
				; ORIGINS-NEXT: [[TMP8:%.*]] = call i32 @llvm.vscale.i32()
				; ORIGINS-NEXT: [[TMP9:%.*]] = mul i32 [[TMP8]], 4
				; ORIGINS-NEXT: [[TMP10:%.*]] = add i32 [[TMP9]], 3
				; ORIGINS-NEXT: [[TMP11:%.*]] = udiv i32 [[TMP10]], 4
				; ORIGINS-NEXT: br label [[DOTSPLIT:%.*]]
				; ORIGINS: .split:
				; ORIGINS-NEXT: [[IV:%.]] = phi i32 [ 0, [[TMP7]] ], [ [[IV_NEXT:%.]], [[DOTSPLIT]] ]
				; ORIGINS-NEXT: [[TMP12:%.*]] = getelementptr i32, ptr [[TMP5]], i32 [[IV]]
				; ORIGINS-NEXT: store i32 0, ptr [[TMP12]], align 4
				; ORIGINS-NEXT: [[IV_NEXT]] = add nuw nsw i32 [[IV]], 1
				; ORIGINS-NEXT: [[IV_CHECK:%.*]] = icmp eq i32 [[IV_NEXT]], [[TMP11]]
				; ORIGINS-NEXT: br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
				; ORIGINS: .split.split:
				; ORIGINS-NEXT: br label [[TMP13]]
				; ORIGINS: 13:
				; ORIGINS-NEXT: store <vscale x 1 x i32> zeroinitializer, ptr [[P]], align 4
				; ORIGINS-NEXT: ret void
				;
				store <vscale x 1 x i32> zeroinitializer, ptr %p
				ret void
				}

				define void @store.nxv2i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @store.nxv2i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; CHECK-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; CHECK-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[TMP3]], align 8
				; CHECK-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[P]], align 8
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @store.nxv2i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ADDR-NEXT: [[TMP5:%.*]] = xor i64 [[TMP4]], 87960930222080
				; ADDR-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ADDR-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[TMP6]], align 8
				; ADDR-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[P]], align 8
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @store.nxv2i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ORIGINS-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; ORIGINS-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; ORIGINS-NEXT: [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
				; ORIGINS-NEXT: [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
				; ORIGINS-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[TMP3]], align 8
				; ORIGINS-NEXT: [[TMP6:%.*]] = call i32 @llvm.vector.reduce.or.nxv2i32(<vscale x 2 x i32> zeroinitializer)
				; ORIGINS-NEXT: [[_MSCMP:%.*]] = icmp ne i32 [[TMP6]], 0
				; ORIGINS-NEXT: br i1 [[_MSCMP]], label [[TMP7:%.]], label [[TMP13:%.]], !prof [[PROF0]]
				; ORIGINS: 7:
				; ORIGINS-NEXT: [[TMP8:%.*]] = call i32 @llvm.vscale.i32()
				; ORIGINS-NEXT: [[TMP9:%.*]] = mul i32 [[TMP8]], 8
				; ORIGINS-NEXT: [[TMP10:%.*]] = add i32 [[TMP9]], 3
				; ORIGINS-NEXT: [[TMP11:%.*]] = udiv i32 [[TMP10]], 4
				; ORIGINS-NEXT: br label [[DOTSPLIT:%.*]]
				; ORIGINS: .split:
				; ORIGINS-NEXT: [[IV:%.]] = phi i32 [ 0, [[TMP7]] ], [ [[IV_NEXT:%.]], [[DOTSPLIT]] ]
				; ORIGINS-NEXT: [[TMP12:%.*]] = getelementptr i32, ptr [[TMP5]], i32 [[IV]]
				; ORIGINS-NEXT: store i32 0, ptr [[TMP12]], align 4
				; ORIGINS-NEXT: [[IV_NEXT]] = add nuw nsw i32 [[IV]], 1
				; ORIGINS-NEXT: [[IV_CHECK:%.*]] = icmp eq i32 [[IV_NEXT]], [[TMP11]]
				; ORIGINS-NEXT: br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
				; ORIGINS: .split.split:
				; ORIGINS-NEXT: br label [[TMP13]]
				; ORIGINS: 13:
				; ORIGINS-NEXT: store <vscale x 2 x i32> zeroinitializer, ptr [[P]], align 8
				; ORIGINS-NEXT: ret void
				;
				store <vscale x 2 x i32> zeroinitializer, ptr %p
				ret void
				}

				define void @store.nxv4i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @store.nxv4i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; CHECK-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; CHECK-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[TMP3]], align 16
				; CHECK-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[P]], align 16
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @store.nxv4i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ADDR-NEXT: [[TMP5:%.*]] = xor i64 [[TMP4]], 87960930222080
				; ADDR-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ADDR-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[TMP6]], align 16
				; ADDR-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[P]], align 16
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @store.nxv4i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ORIGINS-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; ORIGINS-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; ORIGINS-NEXT: [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
				; ORIGINS-NEXT: [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
				; ORIGINS-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[TMP3]], align 16
				; ORIGINS-NEXT: [[TMP6:%.*]] = call i32 @llvm.vector.reduce.or.nxv4i32(<vscale x 4 x i32> zeroinitializer)
				; ORIGINS-NEXT: [[_MSCMP:%.*]] = icmp ne i32 [[TMP6]], 0
				; ORIGINS-NEXT: br i1 [[_MSCMP]], label [[TMP7:%.]], label [[TMP13:%.]], !prof [[PROF0]]
				; ORIGINS: 7:
				; ORIGINS-NEXT: [[TMP8:%.*]] = call i32 @llvm.vscale.i32()
				; ORIGINS-NEXT: [[TMP9:%.*]] = mul i32 [[TMP8]], 16
				; ORIGINS-NEXT: [[TMP10:%.*]] = add i32 [[TMP9]], 3
				; ORIGINS-NEXT: [[TMP11:%.*]] = udiv i32 [[TMP10]], 4
				; ORIGINS-NEXT: br label [[DOTSPLIT:%.*]]
				; ORIGINS: .split:
				; ORIGINS-NEXT: [[IV:%.]] = phi i32 [ 0, [[TMP7]] ], [ [[IV_NEXT:%.]], [[DOTSPLIT]] ]
				; ORIGINS-NEXT: [[TMP12:%.*]] = getelementptr i32, ptr [[TMP5]], i32 [[IV]]
				; ORIGINS-NEXT: store i32 0, ptr [[TMP12]], align 4
				; ORIGINS-NEXT: [[IV_NEXT]] = add nuw nsw i32 [[IV]], 1
				; ORIGINS-NEXT: [[IV_CHECK:%.*]] = icmp eq i32 [[IV_NEXT]], [[TMP11]]
				; ORIGINS-NEXT: br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
				; ORIGINS: .split.split:
				; ORIGINS-NEXT: br label [[TMP13]]
				; ORIGINS: 13:
				; ORIGINS-NEXT: store <vscale x 4 x i32> zeroinitializer, ptr [[P]], align 16
				; ORIGINS-NEXT: ret void
				;
				store <vscale x 4 x i32> zeroinitializer, ptr %p
				ret void
				}

				define void @store.nxv8i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @store.nxv8i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; CHECK-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; CHECK-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[TMP3]], align 32
				; CHECK-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[P]], align 32
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @store.nxv8i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ADDR-NEXT: [[TMP5:%.*]] = xor i64 [[TMP4]], 87960930222080
				; ADDR-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ADDR-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[TMP6]], align 32
				; ADDR-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[P]], align 32
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @store.nxv8i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ORIGINS-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; ORIGINS-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; ORIGINS-NEXT: [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
				; ORIGINS-NEXT: [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
				; ORIGINS-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[TMP3]], align 32
				; ORIGINS-NEXT: [[TMP6:%.*]] = call i32 @llvm.vector.reduce.or.nxv8i32(<vscale x 8 x i32> zeroinitializer)
				; ORIGINS-NEXT: [[_MSCMP:%.*]] = icmp ne i32 [[TMP6]], 0
				; ORIGINS-NEXT: br i1 [[_MSCMP]], label [[TMP7:%.]], label [[TMP13:%.]], !prof [[PROF0]]
				; ORIGINS: 7:
				; ORIGINS-NEXT: [[TMP8:%.*]] = call i32 @llvm.vscale.i32()
				; ORIGINS-NEXT: [[TMP9:%.*]] = mul i32 [[TMP8]], 32
				; ORIGINS-NEXT: [[TMP10:%.*]] = add i32 [[TMP9]], 3
				; ORIGINS-NEXT: [[TMP11:%.*]] = udiv i32 [[TMP10]], 4
				; ORIGINS-NEXT: br label [[DOTSPLIT:%.*]]
				; ORIGINS: .split:
				; ORIGINS-NEXT: [[IV:%.]] = phi i32 [ 0, [[TMP7]] ], [ [[IV_NEXT:%.]], [[DOTSPLIT]] ]
				; ORIGINS-NEXT: [[TMP12:%.*]] = getelementptr i32, ptr [[TMP5]], i32 [[IV]]
				; ORIGINS-NEXT: store i32 0, ptr [[TMP12]], align 4
				; ORIGINS-NEXT: [[IV_NEXT]] = add nuw nsw i32 [[IV]], 1
				; ORIGINS-NEXT: [[IV_CHECK:%.*]] = icmp eq i32 [[IV_NEXT]], [[TMP11]]
				; ORIGINS-NEXT: br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
				; ORIGINS: .split.split:
				; ORIGINS-NEXT: br label [[TMP13]]
				; ORIGINS: 13:
				; ORIGINS-NEXT: store <vscale x 8 x i32> zeroinitializer, ptr [[P]], align 32
				; ORIGINS-NEXT: ret void
				;
				store <vscale x 8 x i32> zeroinitializer, ptr %p
				ret void
				}

				define void @store.nxv16i32(ptr %p) sanitize_memory {
				; CHECK-LABEL: @store.nxv16i32(
				; CHECK-NEXT: call void @llvm.donothing()
				; CHECK-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; CHECK-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; CHECK-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[TMP3]], align 64
				; CHECK-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[P]], align 64
				; CHECK-NEXT: ret void
				;
				; ADDR-LABEL: @store.nxv16i32(
				; ADDR-NEXT: [[TMP1:%.*]] = load i64, ptr @__msan_param_tls, align 8
				; ADDR-NEXT: call void @llvm.donothing()
				; ADDR-NEXT: [[_MSCMP:%.*]] = icmp ne i64 [[TMP1]], 0
				; ADDR-NEXT: br i1 [[_MSCMP]], label [[TMP2:%.]], label [[TMP3:%.]], !prof [[PROF0]]
				; ADDR: 2:
				; ADDR-NEXT: call void @__msan_warning_noreturn() #[[ATTR3]]
				; ADDR-NEXT: unreachable
				; ADDR: 3:
				; ADDR-NEXT: [[TMP4:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ADDR-NEXT: [[TMP5:%.*]] = xor i64 [[TMP4]], 87960930222080
				; ADDR-NEXT: [[TMP6:%.*]] = inttoptr i64 [[TMP5]] to ptr
				; ADDR-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[TMP6]], align 64
				; ADDR-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[P]], align 64
				; ADDR-NEXT: ret void
				;
				; ORIGINS-LABEL: @store.nxv16i32(
				; ORIGINS-NEXT: call void @llvm.donothing()
				; ORIGINS-NEXT: [[TMP1:%.]] = ptrtoint ptr [[P:%.]] to i64
				; ORIGINS-NEXT: [[TMP2:%.*]] = xor i64 [[TMP1]], 87960930222080
				; ORIGINS-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
				; ORIGINS-NEXT: [[TMP4:%.*]] = add i64 [[TMP2]], 17592186044416
				; ORIGINS-NEXT: [[TMP5:%.*]] = inttoptr i64 [[TMP4]] to ptr
				; ORIGINS-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[TMP3]], align 64
				; ORIGINS-NEXT: [[TMP6:%.*]] = call i32 @llvm.vector.reduce.or.nxv16i32(<vscale x 16 x i32> zeroinitializer)
				; ORIGINS-NEXT: [[_MSCMP:%.*]] = icmp ne i32 [[TMP6]], 0
				; ORIGINS-NEXT: br i1 [[_MSCMP]], label [[TMP7:%.]], label [[TMP13:%.]], !prof [[PROF0]]
				; ORIGINS: 7:
				; ORIGINS-NEXT: [[TMP8:%.*]] = call i32 @llvm.vscale.i32()
				; ORIGINS-NEXT: [[TMP9:%.*]] = mul i32 [[TMP8]], 64
				; ORIGINS-NEXT: [[TMP10:%.*]] = add i32 [[TMP9]], 3
				; ORIGINS-NEXT: [[TMP11:%.*]] = udiv i32 [[TMP10]], 4
				; ORIGINS-NEXT: br label [[DOTSPLIT:%.*]]
				; ORIGINS: .split:
				; ORIGINS-NEXT: [[IV:%.]] = phi i32 [ 0, [[TMP7]] ], [ [[IV_NEXT:%.]], [[DOTSPLIT]] ]
				; ORIGINS-NEXT: [[TMP12:%.*]] = getelementptr i32, ptr [[TMP5]], i32 [[IV]]
				; ORIGINS-NEXT: store i32 0, ptr [[TMP12]], align 4
				; ORIGINS-NEXT: [[IV_NEXT]] = add nuw nsw i32 [[IV]], 1
				; ORIGINS-NEXT: [[IV_CHECK:%.*]] = icmp eq i32 [[IV_NEXT]], [[TMP11]]
				; ORIGINS-NEXT: br i1 [[IV_CHECK]], label [[DOTSPLIT_SPLIT:%.*]], label [[DOTSPLIT]]
				; ORIGINS: .split.split:
				; ORIGINS-NEXT: br label [[TMP13]]
				; ORIGINS: 13:
				; ORIGINS-NEXT: store <vscale x 16 x i32> zeroinitializer, ptr [[P]], align 64
				; ORIGINS-NEXT: ret void
				;
				store <vscale x 16 x i32> zeroinitializer, ptr %p
				ret void
				}