Use memory region declaration intrinsic when generating code for array subscripts

[simeon]

Summary

As an alternative to https://reviews.llvm.org/D150192, we consider using the memory region declaration intrinsic, introduced in https://reviews.llvm.org/D115274, as another way to propagate the semantics of C/C++ array-indexing rules to LLVM IR.

For example, given the following C source

int f(long i) {
  int A[100];
  return A[i];
}

where we would previously emit the following for the indexed array access

%0 = load i64, ptr %X.addr, align 8
%arrayidx = getelementptr inbounds [100 x i32], ptr %A, i64 0, i64 %0
%1 = load i32, ptr %arrayidx, align 4

we now emit

%0 = load i64, ptr %X.addr, align 8
%arrayidx = getelementptr inbounds [100 x i32], ptr %A, i64 0, i64 0
%arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %arrayidx, i64 0, i64 6400)
%arrayidx1 = getelementptr inbounds i32, ptr %arrayidx.bounded, i64 %0
%1 = load i32, ptr %arrayidx1, align 4

This patch is only only a proof-of-concept at this point, intended to gather feedback and bring https://reviews.llvm.org/D115274 back into focus. As was mentioned there (and as seen from test results), completing this patch would require patching a lot of optimizations. For example, InstructionCombiningPass works with GetElementPtrInsts directly and will not recognize an intrinsic call as such. Many other passes such as IndVarSimplifyPass, EarlyCSE, etc., routinely use a switch case or call isa<GetElementPtrInst>() to detect GEPs.

We'd be fine with doing that sort of work, if we get some initial thumbs up from the community that this is the way to go. Of course, we're also open to trying a different approach if it avoids the apparently unavoidable complexity of this one.

An example of alias analysis based on the intrinsic can be seen in this independent diff: https://reviews.llvm.org/differential/diff/528871/.

Depends on D115274

[efriedma]

If we are going to do this at all, I think this is roughly what it should look like. Potential issues you might run into:

• The compile-time overhead of creating a bunch of extra intrinsics might be significant. Maybe we can mitigate to some extent by avoiding emitting the intrinsic in simple cases where it doesn't actually help (constant indexes?).
• User code might not actually obey the language rules; do we have any sanitizer that checks if user code trips over this?
• Not sure how this interacts with full restrict and related proposals.

[simeon]

• The compile-time overhead of creating a bunch of extra intrinsics might be significant. Maybe we can mitigate to some extent by avoiding emitting the intrinsic in simple cases where it doesn't actually help (constant indexes?).

Yes, that would make sense.

• User code might not actually obey the language rules; do we have any sanitizer that checks if user code trips over this?

I believe AddressSanitizer should be able to detect out-of-bounds accesses.

• Not sure how this interacts with full restrict and related proposals.

So far as alias analysis goes, the two approaches should be orthogonal, but nevertheless we'd still have to go through the usual procedure of modifying the direct GetElementPtrInst checks and casts used by such patches.

[jeroen.dobbelaere]

In D152275#4418363, @efriedma wrote:

• Not sure how this interacts with full restrict and related proposals.

the full restrict PropagateAndConvertNoAlias pass will need to learn about it, but that should be trivial.

[efriedma]

In D152275#4423845, @simeon wrote:

• User code might not actually obey the language rules; do we have any sanitizer that checks if user code trips over this?

I believe AddressSanitizer should be able to detect out-of-bounds accesses.

asan will detect if you index completely outside a memory allocation, but it won't detect if you do something like struct S { int x[10]; int y; } s = {}; return s->x[10];.

[simeon]

The patch now includes the changes that need to be made to the optimization passes we observed to be most negatively affected by the introduction of the intrinsic, primarily InstCombine and SROA. However, it is not comprehensive: we also observed missed optimization opportunities in other passes such as GlobalOpt and MergedLoadStoreMotionPass.

A large number of hand-written unit tests need to be updated by hand but as they are very sensitive to the smallest change in the definition of the intrinsic, I am postponing this until we have some initial approval.

clang/lib/CodeGen/CGExpr.cpp

if (const VariableArrayType *vla =
     if (const auto *ASE = dyn_cast<ArraySubscriptExpr>(Array))
       ArrayLV = EmitArraySubscriptExpr(ASE, /*Accessed*/ true);
     else
       ArrayLV = EmitLValue(Array);
     auto *Idx = EmitIdxAfterBase(/*Promote*/true);
 
     // Propagate the alignment from the array itself to the result.
     QualType arrayType = Array->getType();
+
+    Address ArrayLVAddr = ArrayLV.getAddress(*this);
+
+    if (!getLangOpts().isSignedOverflowDefined() && 
+        // ISO/IEC 9899:TC3, 6.5.6.8
+        (getLangOpts().C99 || getLangOpts().CPlusPlus) &&
+        getContext().getAsConstantArrayType(arrayType)) {
+      auto *CAT = getContext().getAsConstantArrayType(arrayType);
+      uint64_t BoundedRegionSize = CAT->getSize().getZExtValue() *
+        getContext().getTypeSize(CAT->getElementType()) / 8;
+
+      Address BeginOff = emitArraySubscriptGEP(
+          *this, ArrayLVAddr, 
+          {CGM.getSize(CharUnits::Zero()), CGM.getSize(CharUnits::Zero())},
+          E->getType(), !getLangOpts().isSignedOverflowDefined(), SignedIndices,
+          E->getExprLoc(), &arrayType, E->getBase());
+
+      llvm::Function *F = CGM.getIntrinsic(llvm::Intrinsic::memory_region_decl, 
+          BeginOff.getPointer()->getType());
+      llvm::Value *Call = Builder.CreateCall(F,
+          {BeginOff.getPointer(), 
+          llvm::ConstantInt::get(Int64Ty, 0), 
+          llvm::ConstantInt::get(Int64Ty, BoundedRegionSize)},
+          "arrayidx.bounded");
+      Address RetAddr(Call, BeginOff.getElementType(), 
+          ArrayLVAddr.getAlignment());
+
       Addr = emitArraySubscriptGEP(
-        *this, ArrayLV.getAddress(*this), {CGM.getSize(CharUnits::Zero()), Idx},
+          *this, RetAddr, {Idx},
           E->getType(), !getLangOpts().isSignedOverflowDefined(), SignedIndices,
           E->getExprLoc(), &arrayType, E->getBase());
+    } else {
+      Addr = emitArraySubscriptGEP(
+          *this, ArrayLVAddr, {CGM.getSize(CharUnits::Zero()), Idx},
+          E->getType(), !getLangOpts().isSignedOverflowDefined(), SignedIndices,
+          E->getExprLoc(), &arrayType, E->getBase());
+    }
+        
     EltBaseInfo = ArrayLV.getBaseInfo();
     EltTBAAInfo = CGM.getTBAAInfoForSubobject(ArrayLV, E->getType());
   } else {
     // The base must be a pointer; emit it with an estimate of its alignment.
     Addr = EmitPointerWithAlignment(E->getBase(), &EltBaseInfo, &EltTBAAInfo);
     auto *Idx = EmitIdxAfterBase(/*Promote*/true);
     QualType ptrType = E->getBase()->getType();
     Addr = emitArraySubscriptGEP(*this, Addr, Idx, E->getType(),

clang/test/CodeGen/2005-01-02-ConstantInits.c

 // NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --function-signature --check-globals --global-value-regex "@.+"
 // RUN: %clang_cc1 -triple=x86_64-unknown-linux %s -emit-llvm -o - | FileCheck %s
 
 // This tests all kinds of hard cases with initializers and
 // array subscripts.  This corresponds to PR487.
 
 struct X { int a[2]; };
 extern int bar();
 
-//.
-// CHECK: @test.i23 = internal global i32 4, align 4
-// CHECK: @i = global i32 4, align 4
-// CHECK: @Arr = global [100 x i32] zeroinitializer, align 16
-// CHECK: @foo2.X = internal global ptr getelementptr (i8, ptr @Arr, i64 196), align 8
-// CHECK: @foo2.i23 = internal global i32 0, align 4
-//.
 // CHECK-LABEL: define {{[^@]+}}@test
 // CHECK-SAME: () #[[ATTR0:[0-9]+]] {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr @test.i23, align 4
 // CHECK-NEXT:    ret i32 [[TMP0]]
 //
 int test(void) {
   static int i23 = (int) &(((struct X *)0)->a[1]);
   return i23;
 }
 
 int i = (int) &( ((struct X *)0) -> a[1]);
 
 int Arr[100];
 
 // CHECK-LABEL: define {{[^@]+}}@foo
 // CHECK-SAME: (i32 noundef [[I:%.*]]) #[[ATTR0]] {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
-// CHECK-NEXT:    [[CALL:%.*]] = call i32 (ptr, ...) @bar(ptr noundef getelementptr inbounds ([100 x i32], ptr @Arr, i64 0, i64 49))
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr @Arr, i64 0, i64 400)
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[ARRAYIDX_BOUNDED]], i64 49
+// CHECK-NEXT:    [[CALL:%.*]] = call i32 (ptr, ...) @bar(ptr noundef [[ARRAYIDX]])
 // CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[I_ADDR]], align 4
 // CHECK-NEXT:    [[IDXPROM:%.*]] = sext i32 [[TMP0]] to i64
-// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [100 x i32], ptr @Arr, i64 0, i64 [[IDXPROM]]
-// CHECK-NEXT:    [[CALL1:%.*]] = call i32 (ptr, ...) @bar(ptr noundef [[ARRAYIDX]])
-// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[CALL]], [[CALL1]]
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED1:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr @Arr, i64 0, i64 400)
+// CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds i32, ptr [[ARRAYIDX_BOUNDED1]], i64 [[IDXPROM]]
+// CHECK-NEXT:    [[CALL3:%.*]] = call i32 (ptr, ...) @bar(ptr noundef [[ARRAYIDX2]])
+// CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[CALL]], [[CALL3]]
 // CHECK-NEXT:    ret i32 [[ADD]]
 //
 int foo(int i) { return bar(&Arr[49])+bar(&Arr[i]); }
 // CHECK-LABEL: define {{[^@]+}}@foo2
 // CHECK-SAME: (i32 noundef [[I:%.*]]) #[[ATTR0]] {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    [[P:%.*]] = alloca ptr, align 8

clang/test/CodeGen/X86/va-arg-sse.c

 // CHECK-NEXT:    [[K:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    [[I:%.*]] = alloca i32, align 4
 // CHECK-NEXT:    [[TMP:%.*]] = alloca [[STRUCT_S]], align 4
 // CHECK-NEXT:    store i32 [[Z:%.*]], ptr [[Z_ADDR]], align 4
 // CHECK-NEXT:    store i32 0, ptr [[J]], align 4
 // CHECK-NEXT:    store i32 0, ptr [[K]], align 4
 // CHECK-NEXT:    [[ARRAYDECAY:%.*]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr [[AP]], i64 0, i64 0
 // CHECK-NEXT:    call void @llvm.va_start(ptr [[ARRAYDECAY]])
-// CHECK-NEXT:    store ptr getelementptr inbounds ([5 x %struct.S], ptr @a, i64 0, i64 2), ptr [[P]], align 8
-// CHECK-NEXT:    [[ARRAYDECAY2:%.*]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr [[AP]], i64 0, i64 0
-// CHECK-NEXT:    [[FP_OFFSET_P:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG:%.*]], ptr [[ARRAYDECAY2]], i32 0, i32 1
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr @a, i64 0, i64 60)
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[ARRAYIDX_BOUNDED]], i64 2
+// CHECK-NEXT:    store ptr [[ARRAYIDX]], ptr [[P]], align 8
+// CHECK-NEXT:    [[ARRAYDECAY1:%.*]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr [[AP]], i64 0, i64 0
+// CHECK-NEXT:    [[FP_OFFSET_P:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG:%.*]], ptr [[ARRAYDECAY1]], i32 0, i32 1
 // CHECK-NEXT:    [[FP_OFFSET:%.*]] = load i32, ptr [[FP_OFFSET_P]], align 4
 // CHECK-NEXT:    [[FITS_IN_FP:%.*]] = icmp ule i32 [[FP_OFFSET]], 144
 // CHECK-NEXT:    br i1 [[FITS_IN_FP]], label [[VAARG_IN_REG:%.*]], label [[VAARG_IN_MEM:%.*]]
 // CHECK:       vaarg.in_reg:
-// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG]], ptr [[ARRAYDECAY2]], i32 0, i32 3
+// CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG]], ptr [[ARRAYDECAY1]], i32 0, i32 3
 // CHECK-NEXT:    [[REG_SAVE_AREA:%.*]] = load ptr, ptr [[TMP0]], align 16
 // CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i8, ptr [[REG_SAVE_AREA]], i32 [[FP_OFFSET]]
 // CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i8, ptr [[TMP1]], i64 16
-// CHECK-NEXT:    [[TMP5:%.*]] = load <2 x float>, ptr [[TMP1]], align 16
-// CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds { <2 x float>, float }, ptr [[TMP]], i32 0, i32 0
-// CHECK-NEXT:    store <2 x float> [[TMP5]], ptr [[TMP6]], align 4
-// CHECK-NEXT:    [[TMP8:%.*]] = load float, ptr [[TMP2]], align 16
-// CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds { <2 x float>, float }, ptr [[TMP]], i32 0, i32 1
-// CHECK-NEXT:    store float [[TMP8]], ptr [[TMP9]], align 4
-// CHECK-NEXT:    [[TMP11:%.*]] = add i32 [[FP_OFFSET]], 32
-// CHECK-NEXT:    store i32 [[TMP11]], ptr [[FP_OFFSET_P]], align 4
+// CHECK-NEXT:    [[TMP3:%.*]] = load <2 x float>, ptr [[TMP1]], align 16
+// CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds { <2 x float>, float }, ptr [[TMP]], i32 0, i32 0
+// CHECK-NEXT:    store <2 x float> [[TMP3]], ptr [[TMP4]], align 4
+// CHECK-NEXT:    [[TMP5:%.*]] = load float, ptr [[TMP2]], align 16
+// CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds { <2 x float>, float }, ptr [[TMP]], i32 0, i32 1
+// CHECK-NEXT:    store float [[TMP5]], ptr [[TMP6]], align 4
+// CHECK-NEXT:    [[TMP7:%.*]] = add i32 [[FP_OFFSET]], 32
+// CHECK-NEXT:    store i32 [[TMP7]], ptr [[FP_OFFSET_P]], align 4
 // CHECK-NEXT:    br label [[VAARG_END:%.*]]
 // CHECK:       vaarg.in_mem:
-// CHECK-NEXT:    [[OVERFLOW_ARG_AREA_P:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG]], ptr [[ARRAYDECAY2]], i32 0, i32 2
+// CHECK-NEXT:    [[OVERFLOW_ARG_AREA_P:%.*]] = getelementptr inbounds [[STRUCT___VA_LIST_TAG]], ptr [[ARRAYDECAY1]], i32 0, i32 2
 // CHECK-NEXT:    [[OVERFLOW_ARG_AREA:%.*]] = load ptr, ptr [[OVERFLOW_ARG_AREA_P]], align 8
 // CHECK-NEXT:    [[OVERFLOW_ARG_AREA_NEXT:%.*]] = getelementptr i8, ptr [[OVERFLOW_ARG_AREA]], i32 16
 // CHECK-NEXT:    store ptr [[OVERFLOW_ARG_AREA_NEXT]], ptr [[OVERFLOW_ARG_AREA_P]], align 8
 // CHECK-NEXT:    br label [[VAARG_END]]
 // CHECK:       vaarg.end:
 // CHECK-NEXT:    [[VAARG_ADDR:%.*]] = phi ptr [ [[TMP]], [[VAARG_IN_REG]] ], [ [[OVERFLOW_ARG_AREA]], [[VAARG_IN_MEM]] ]
 // CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[ARG]], ptr align 4 [[VAARG_ADDR]], i64 12, i1 false)
-// CHECK-NEXT:    [[ARRAYDECAY3:%.*]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr [[AP]], i64 0, i64 0
-// CHECK-NEXT:    call void @llvm.va_end(ptr [[ARRAYDECAY3]])
-// CHECK-NEXT:    [[TMP15:%.*]] = load ptr, ptr [[P]], align 8
-// CHECK-NEXT:    [[TOBOOL:%.*]] = icmp ne ptr [[TMP15]], null
+// CHECK-NEXT:    [[ARRAYDECAY2:%.*]] = getelementptr inbounds [1 x %struct.__va_list_tag], ptr [[AP]], i64 0, i64 0
+// CHECK-NEXT:    call void @llvm.va_end(ptr [[ARRAYDECAY2]])
+// CHECK-NEXT:    [[TMP8:%.*]] = load ptr, ptr [[P]], align 8
+// CHECK-NEXT:    [[TOBOOL:%.*]] = icmp ne ptr [[TMP8]], null
 // CHECK-NEXT:    br i1 [[TOBOOL]], label [[LAND_LHS_TRUE:%.*]], label [[IF_END:%.*]]
 // CHECK:       land.lhs.true:
-// CHECK-NEXT:    [[TMP16:%.*]] = load ptr, ptr [[P]], align 8
-// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[TMP16]], i32 0, i32 0
-// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [3 x float], ptr [[A]], i64 0, i64 2
-// CHECK-NEXT:    [[TMP17:%.*]] = load float, ptr [[ARRAYIDX]], align 4
-// CHECK-NEXT:    [[A5:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[ARG]], i32 0, i32 0
-// CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds [3 x float], ptr [[A5]], i64 0, i64 2
-// CHECK-NEXT:    [[TMP18:%.*]] = load float, ptr [[ARRAYIDX6]], align 4
-// CHECK-NEXT:    [[CMP:%.*]] = fcmp une float [[TMP17]], [[TMP18]]
+// CHECK-NEXT:    [[TMP9:%.*]] = load ptr, ptr [[P]], align 8
+// CHECK-NEXT:    [[A:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[TMP9]], i32 0, i32 0
+// CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [3 x float], ptr [[A]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED4:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX3]], i64 0, i64 12)
+// CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, ptr [[ARRAYIDX_BOUNDED4]], i64 2
+// CHECK-NEXT:    [[TMP10:%.*]] = load float, ptr [[ARRAYIDX5]], align 4
+// CHECK-NEXT:    [[A6:%.*]] = getelementptr inbounds [[STRUCT_S]], ptr [[ARG]], i32 0, i32 0
+// CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds [3 x float], ptr [[A6]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED8:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX7]], i64 0, i64 12)
+// CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds float, ptr [[ARRAYIDX_BOUNDED8]], i64 2
+// CHECK-NEXT:    [[TMP11:%.*]] = load float, ptr [[ARRAYIDX9]], align 4
+// CHECK-NEXT:    [[CMP:%.*]] = fcmp une float [[TMP10]], [[TMP11]]
 // CHECK-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[IF_END]]
 // CHECK:       if.then:
 // CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 // CHECK-NEXT:    br label [[RETURN:%.*]]
 // CHECK:       if.end:
 // CHECK-NEXT:    store i32 1, ptr [[RETVAL]], align 4
 // CHECK-NEXT:    br label [[RETURN]]
 // CHECK:       return:
-// CHECK-NEXT:    [[TMP19:%.*]] = load i32, ptr [[RETVAL]], align 4
-// CHECK-NEXT:    ret i32 [[TMP19]]
+// CHECK-NEXT:    [[TMP12:%.*]] = load i32, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i32 [[TMP12]]
 //
 int check (int z, ...)
 {
   struct S arg, *p;
   va_list ap;
   int j = 0, k = 0;
   int i;
   va_start (ap, z);
   p = &a[2];
   arg = va_arg (ap, struct S);
   va_end (ap);
   if (p && p->a[2] != arg.a[2])
     return 0;
   return 1;
 }

clang/test/CodeGen/builtin-align-array.c

 // NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
 /// Check that the alignment builtins handle array-to-pointer decay
 // RUN: %clang_cc1 -triple=x86_64-unknown-unknown -o - -emit-llvm %s | FileCheck %s
 
 extern int func(char *c);
 
 // CHECK-LABEL: @test_array(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[BUF:%.*]] = alloca [1024 x i8], align 16
-// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 44
-// CHECK-NEXT:    [[INTPTR:%.*]] = ptrtoint ptr [[ARRAYIDX]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX]], i64 0, i64 1024)
+// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX_BOUNDED]], i64 44
+// CHECK-NEXT:    [[INTPTR:%.*]] = ptrtoint ptr [[ARRAYIDX1]] to i64
 // CHECK-NEXT:    [[ALIGNED_INTPTR:%.*]] = and i64 [[INTPTR]], -16
 // CHECK-NEXT:    [[DIFF:%.*]] = sub i64 [[ALIGNED_INTPTR]], [[INTPTR]]
-// CHECK-NEXT:    [[ALIGNED_RESULT:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX]], i64 [[DIFF]]
+// CHECK-NEXT:    [[ALIGNED_RESULT:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX1]], i64 [[DIFF]]
 // CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT]], i64 16) ]
 // CHECK-NEXT:    [[CALL:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT]])
-// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 22
-// CHECK-NEXT:    [[INTPTR2:%.*]] = ptrtoint ptr [[ARRAYIDX1]] to i64
-// CHECK-NEXT:    [[OVER_BOUNDARY:%.*]] = add i64 [[INTPTR2]], 31
-// CHECK-NEXT:    [[ALIGNED_INTPTR4:%.*]] = and i64 [[OVER_BOUNDARY]], -32
-// CHECK-NEXT:    [[DIFF5:%.*]] = sub i64 [[ALIGNED_INTPTR4]], [[INTPTR2]]
-// CHECK-NEXT:    [[ALIGNED_RESULT6:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX1]], i64 [[DIFF5]]
-// CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT6]], i64 32) ]
-// CHECK-NEXT:    [[CALL7:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT6]])
-// CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 16
-// CHECK-NEXT:    [[SRC_ADDR:%.*]] = ptrtoint ptr [[ARRAYIDX8]] to i64
+// CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED3:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX2]], i64 0, i64 1024)
+// CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX_BOUNDED3]], i64 22
+// CHECK-NEXT:    [[INTPTR5:%.*]] = ptrtoint ptr [[ARRAYIDX4]] to i64
+// CHECK-NEXT:    [[OVER_BOUNDARY:%.*]] = add i64 [[INTPTR5]], 31
+// CHECK-NEXT:    [[ALIGNED_INTPTR7:%.*]] = and i64 [[OVER_BOUNDARY]], -32
+// CHECK-NEXT:    [[DIFF8:%.*]] = sub i64 [[ALIGNED_INTPTR7]], [[INTPTR5]]
+// CHECK-NEXT:    [[ALIGNED_RESULT9:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX4]], i64 [[DIFF8]]
+// CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT9]], i64 32) ]
+// CHECK-NEXT:    [[CALL10:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT9]])
+// CHECK-NEXT:    [[ARRAYIDX11:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED12:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX11]], i64 0, i64 1024)
+// CHECK-NEXT:    [[ARRAYIDX13:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX_BOUNDED12]], i64 16
+// CHECK-NEXT:    [[SRC_ADDR:%.*]] = ptrtoint ptr [[ARRAYIDX13]] to i64
 // CHECK-NEXT:    [[SET_BITS:%.*]] = and i64 [[SRC_ADDR]], 63
 // CHECK-NEXT:    [[IS_ALIGNED:%.*]] = icmp eq i64 [[SET_BITS]], 0
 // CHECK-NEXT:    [[CONV:%.*]] = zext i1 [[IS_ALIGNED]] to i32
 // CHECK-NEXT:    ret i32 [[CONV]]
 //
 int test_array(void) {
   char buf[1024];
   func(__builtin_align_down(&buf[44], 16));
   func(__builtin_align_up(&buf[22], 32));
   return __builtin_is_aligned(&buf[16], 64);
 }
 
 // CHECK-LABEL: @test_array_should_not_mask(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[BUF:%.*]] = alloca [1024 x i8], align 32
-// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 64
-// CHECK-NEXT:    [[INTPTR:%.*]] = ptrtoint ptr [[ARRAYIDX]] to i64
+// CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX]], i64 0, i64 1024)
+// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX_BOUNDED]], i64 64
+// CHECK-NEXT:    [[INTPTR:%.*]] = ptrtoint ptr [[ARRAYIDX1]] to i64
 // CHECK-NEXT:    [[ALIGNED_INTPTR:%.*]] = and i64 [[INTPTR]], -16
 // CHECK-NEXT:    [[DIFF:%.*]] = sub i64 [[ALIGNED_INTPTR]], [[INTPTR]]
-// CHECK-NEXT:    [[ALIGNED_RESULT:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX]], i64 [[DIFF]]
+// CHECK-NEXT:    [[ALIGNED_RESULT:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX1]], i64 [[DIFF]]
 // CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT]], i64 16) ]
 // CHECK-NEXT:    [[CALL:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT]])
-// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 32
-// CHECK-NEXT:    [[INTPTR2:%.*]] = ptrtoint ptr [[ARRAYIDX1]] to i64
-// CHECK-NEXT:    [[OVER_BOUNDARY:%.*]] = add i64 [[INTPTR2]], 31
-// CHECK-NEXT:    [[ALIGNED_INTPTR4:%.*]] = and i64 [[OVER_BOUNDARY]], -32
-// CHECK-NEXT:    [[DIFF5:%.*]] = sub i64 [[ALIGNED_INTPTR4]], [[INTPTR2]]
-// CHECK-NEXT:    [[ALIGNED_RESULT6:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX1]], i64 [[DIFF5]]
-// CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT6]], i64 32) ]
-// CHECK-NEXT:    [[CALL7:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT6]])
+// CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [1024 x i8], ptr [[BUF]], i64 0, i64 0
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED3:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX2]], i64 0, i64 1024)
+// CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX_BOUNDED3]], i64 32
+// CHECK-NEXT:    [[INTPTR5:%.*]] = ptrtoint ptr [[ARRAYIDX4]] to i64
+// CHECK-NEXT:    [[OVER_BOUNDARY:%.*]] = add i64 [[INTPTR5]], 31
+// CHECK-NEXT:    [[ALIGNED_INTPTR7:%.*]] = and i64 [[OVER_BOUNDARY]], -32
+// CHECK-NEXT:    [[DIFF8:%.*]] = sub i64 [[ALIGNED_INTPTR7]], [[INTPTR5]]
+// CHECK-NEXT:    [[ALIGNED_RESULT9:%.*]] = getelementptr inbounds i8, ptr [[ARRAYIDX4]], i64 [[DIFF8]]
+// CHECK-NEXT:    call void @llvm.assume(i1 true) [ "align"(ptr [[ALIGNED_RESULT9]], i64 32) ]
+// CHECK-NEXT:    [[CALL10:%.*]] = call i32 @func(ptr noundef [[ALIGNED_RESULT9]])
 // CHECK-NEXT:    ret i32 1
 //
 int test_array_should_not_mask(void) {
   _Alignas(32) char buf[1024];
   // TODO: The align_up and align_down calls should be folded to no-ops
   func(__builtin_align_down(&buf[64], 16));
   func(__builtin_align_up(&buf[32], 32));
   // This expression can be constant-evaluated:
   return __builtin_is_aligned(&buf[64], 32);
 }

clang/test/CodeGenCXX/amdgcn-automatic-variable.cpp

 // CHECK-NEXT:    [[LV2_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[LV2]] to ptr
 // CHECK-NEXT:    [[LA_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[LA]] to ptr
 // CHECK-NEXT:    [[LP1_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[LP1]] to ptr
 // CHECK-NEXT:    [[LP2_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[LP2]] to ptr
 // CHECK-NEXT:    [[LVC_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[LVC]] to ptr
 // CHECK-NEXT:    store i32 1, ptr [[LV1_ASCAST]], align 4
 // CHECK-NEXT:    store i32 2, ptr [[LV2_ASCAST]], align 4
 // CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [100 x i32], ptr [[LA_ASCAST]], i64 0, i64 0
-// CHECK-NEXT:    store i32 3, ptr [[ARRAYIDX]], align 4
+// CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr [[ARRAYIDX]], i64 0, i64 400)
+// CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i32, ptr [[ARRAYIDX_BOUNDED]], i64 0
+// CHECK-NEXT:    store i32 3, ptr [[ARRAYIDX1]], align 4
 // CHECK-NEXT:    store ptr [[LV1_ASCAST]], ptr [[LP1_ASCAST]], align 8
 // CHECK-NEXT:    [[ARRAYDECAY:%.*]] = getelementptr inbounds [100 x i32], ptr [[LA_ASCAST]], i64 0, i64 0
 // CHECK-NEXT:    store ptr [[ARRAYDECAY]], ptr [[LP2_ASCAST]], align 8
 // CHECK-NEXT:    call void @_Z5func1Pi(ptr noundef [[LV1_ASCAST]])
 // CHECK-NEXT:    store i32 4, ptr [[LVC_ASCAST]], align 4
 // CHECK-NEXT:    store i32 4, ptr [[LV1_ASCAST]], align 4
 // CHECK-NEXT:    ret void
 //
void func2(void) {
   lv1 = lvc;
 }
 
 void destroy(int x);
 
 class A {
 int x;
 public:
+// CHECK-LABEL: @_ZN1AC1Ev(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 8, addrspace(5)
+// CHECK-NEXT:    [[THIS_ADDR_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[THIS_ADDR]] to ptr
+// CHECK-NEXT:    store ptr [[THIS:%.*]], ptr [[THIS_ADDR_ASCAST]], align 8
+// CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR_ASCAST]], align 8
+// CHECK-NEXT:    call void @_ZN1AC2Ev(ptr noundef nonnull align 4 dereferenceable(4) [[THIS1]])
+// CHECK-NEXT:    ret void
+//
   A():x(0) {}
+// CHECK-LABEL: @_ZN1AD1Ev(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[THIS_ADDR:%.*]] = alloca ptr, align 8, addrspace(5)
+// CHECK-NEXT:    [[THIS_ADDR_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[THIS_ADDR]] to ptr
+// CHECK-NEXT:    store ptr [[THIS:%.*]], ptr [[THIS_ADDR_ASCAST]], align 8
+// CHECK-NEXT:    [[THIS1:%.*]] = load ptr, ptr [[THIS_ADDR_ASCAST]], align 8
+// CHECK-NEXT:    call void @_ZN1AD2Ev(ptr noundef nonnull align 4 dereferenceable(4) [[THIS1]]) #[[ATTR4:[0-9]+]]
+// CHECK-NEXT:    ret void
+//
   ~A() {
    destroy(x);
   }
 };
 
 // CHECK-LABEL: @_Z5func3v(
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[A:%.*]] = alloca [[CLASS_A:%.*]], align 4, addrspace(5)

llvm/include/llvm/Analysis/PtrUseVisitor.h

default:
       return Base::visitIntrinsicInst(II);
 
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
       return; // No-op intrinsics.
     }
   }
 
+  void visitMemRegDeclInst(MemRegDeclInst &I) {
+    enqueueUsers(I);
+  }
+
   // Generically, arguments to calls and invokes escape the pointer to some
   // other function. Mark that.
   void visitCallBase(CallBase &CB) {
     PI.setEscaped(&CB);
     Base::visitCallBase(CB);
   }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_ANALYSIS_PTRUSEVISITOR_H

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

InstructionCost getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
     case Intrinsic::dbg_assign:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_value:
     case Intrinsic::dbg_label:
     case Intrinsic::invariant_start:
     case Intrinsic::invariant_end:
     case Intrinsic::launder_invariant_group:
     case Intrinsic::strip_invariant_group:
+    case Intrinsic::memory_region_decl:
     case Intrinsic::is_constant:
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
     case Intrinsic::experimental_noalias_scope_decl:
     case Intrinsic::objectsize:
     case Intrinsic::ptr_annotation:
     case Intrinsic::var_annotation:
     case Intrinsic::experimental_gc_result:

llvm/include/llvm/IR/InstVisitor.h

 //===- InstVisitor.h - Instruction visitor templates ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 
 #ifndef LLVM_IR_INSTVISITOR_H
 #define LLVM_IR_INSTVISITOR_H
 
+#include "Intrinsics.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/Module.h"
 
 namespace llvm {
 
#include "llvm/IR/Instruction.def"
   RetTy visitMemCpyInst(MemCpyInst &I)            { DELEGATE(MemTransferInst); }
   RetTy visitMemCpyInlineInst(MemCpyInlineInst &I){ DELEGATE(MemCpyInst); }
   RetTy visitMemMoveInst(MemMoveInst &I)          { DELEGATE(MemTransferInst); }
   RetTy visitMemTransferInst(MemTransferInst &I)  { DELEGATE(MemIntrinsic); }
   RetTy visitMemIntrinsic(MemIntrinsic &I)        { DELEGATE(IntrinsicInst); }
   RetTy visitVAStartInst(VAStartInst &I)          { DELEGATE(IntrinsicInst); }
   RetTy visitVAEndInst(VAEndInst &I)              { DELEGATE(IntrinsicInst); }
   RetTy visitVACopyInst(VACopyInst &I)            { DELEGATE(IntrinsicInst); }
+  RetTy visitMemRegDeclInst(MemRegDeclInst &I)    { DELEGATE(IntrinsicInst); }
   RetTy visitIntrinsicInst(IntrinsicInst &I)      { DELEGATE(CallInst); }
   RetTy visitCallInst(CallInst &I)                { DELEGATE(CallBase); }
   RetTy visitInvokeInst(InvokeInst &I)            { DELEGATE(CallBase); }
   RetTy visitCallBrInst(CallBrInst &I)            { DELEGATE(CallBase); }
 
   // While terminators don't have a distinct type modeling them, we support
   // intercepting them with dedicated a visitor callback.
   RetTy visitReturnInst(ReturnInst &I) {
if (const Function *F = I.getCalledFunction()) {
         DELEGATE(MemCpyInlineInst);
       case Intrinsic::memmove:     DELEGATE(MemMoveInst);
       case Intrinsic::memset:      DELEGATE(MemSetInst);
       case Intrinsic::memset_inline:
         DELEGATE(MemSetInlineInst);
       case Intrinsic::vastart:     DELEGATE(VAStartInst);
       case Intrinsic::vaend:       DELEGATE(VAEndInst);
       case Intrinsic::vacopy:      DELEGATE(VACopyInst);
+      case Intrinsic::memory_region_decl:
+        DELEGATE(MemRegDeclInst);
       case Intrinsic::not_intrinsic: break;
       }
     }
     DELEGATE(CallInst);
   }
 
   // An overload that will never actually be called, it is used only from dead
   // code in the dispatching from opcodes to instruction subclasses.

llvm/include/llvm/IR/IntrinsicInst.h

 // has state or virtual methods, which is an important part of this gross/neat
 // hack working.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_INTRINSICINST_H
 #define LLVM_IR_INTRINSICINST_H
 
+#include "Intrinsics.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/FPEnv.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
bool isAssumeLikeIntrinsic() const {
     case Intrinsic::dbg_assign:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_value:
     case Intrinsic::dbg_label:
     case Intrinsic::invariant_start:
     case Intrinsic::invariant_end:
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
+    case Intrinsic::memory_region_decl:
     case Intrinsic::experimental_noalias_scope_decl:
     case Intrinsic::objectsize:
     case Intrinsic::ptr_annotation:
     case Intrinsic::var_annotation:
       return true;
     }
     return false;
   }
public:
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 
   Value *getDest() const { return const_cast<Value *>(getArgOperand(0)); }
   Value *getSrc() const { return const_cast<Value *>(getArgOperand(1)); }
 };
 
+class MemRegDeclInst : public IntrinsicInst {
+public:
+  static bool classof(const IntrinsicInst *I) {
+    return I->getIntrinsicID() == Intrinsic::memory_region_decl;
+  }
+  static bool classof(const Value *V) {
+    return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
+  }
+
+  Value *getPtr() const { return const_cast<Value *>(getArgOperand(0)); }
+};
+
 /// A base class for all instrprof intrinsics.
 class InstrProfInstBase : public IntrinsicInst {
 public:
   // The name of the instrumented function.
   GlobalVariable *getName() const {
     return cast<GlobalVariable>(
         const_cast<Value *>(getArgOperand(0))->stripPointerCasts());
   }

llvm/include/llvm/IR/Intrinsics.td

def int_launder_invariant_group : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                             [LLVMMatchType<0>],
                                             [IntrInaccessibleMemOnly, IntrSpeculatable, IntrWillReturn]>;
 
 
 def int_strip_invariant_group : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                           [LLVMMatchType<0>],
                                           [IntrSpeculatable, IntrNoMem, IntrWillReturn]>;
 
+// Declares that the returned pointer (the first argument),
+// and any pointer that is (transitively) def-use based on that pointer,
+// points into the memory region [ptr+begin_offset, ptr+end_offset),
+// or is poison otherwise.
+def int_memory_region_decl : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
+    [LLVMMatchType<0> /*ptr*/, llvm_i64_ty /*begin_offset*/, llvm_i64_ty /*end_offset*/],
+    [IntrNoMem, IntrSpeculatable, ReadNone<ArgIndex<0>>]>;
+
 //===------------------------ Stackmap Intrinsics -------------------------===//
 //
 def int_experimental_stackmap : DefaultAttrsIntrinsic<[],
                                   [llvm_i64_ty, llvm_i32_ty, llvm_vararg_ty],
                                   [Throws]>;
 def int_experimental_patchpoint_void : Intrinsic<[],
                                                  [llvm_i64_ty, llvm_i32_ty,
                                                   llvm_ptr_ty, llvm_i32_ty,

llvm/lib/Analysis/AliasSetTracker.cpp

if (auto *II = dyn_cast<IntrinsicInst>(Inst)) {
     switch (II->getIntrinsicID()) {
     default:
       break;
       // FIXME: Add lifetime/invariant intrinsics (See: PR30807).
     case Intrinsic::assume:
     case Intrinsic::experimental_noalias_scope_decl:
     case Intrinsic::sideeffect:
     case Intrinsic::pseudoprobe:
+    case Intrinsic::memory_region_decl:
       return;
     }
   }
   if (!Inst->mayReadOrWriteMemory())
     return; // doesn't alias anything
 
   if (AliasSet *AS = findAliasSetForUnknownInst(Inst)) {
     AS->addUnknownInst(Inst, AA);

llvm/lib/Analysis/BasicAliasAnalysis.cpp

if (O1 != O2) {
     // location if that memory location doesn't escape. Or it may pass a
     // nocapture value to other functions as long as they don't capture it.
     if (isEscapeSource(O1) &&
         AAQI.CI->isNotCapturedBeforeOrAt(O2, cast<Instruction>(O1)))
       return AliasResult::NoAlias;
     if (isEscapeSource(O2) &&
         AAQI.CI->isNotCapturedBeforeOrAt(O1, cast<Instruction>(O2)))
       return AliasResult::NoAlias;
+
+    // If an underlying value is a call to a memory region declaration 
+    // intrinsic, extract the GEP and infer upper bounds on the MemoryLocation 
+    // size using the end offset of the region
+    auto *CB1 = dyn_cast<CallBase>(O1), *CB2 = dyn_cast<CallBase>(O2);
+    bool FoundMemRegDecl = false;
+    if (CB1 && CB1->getIntrinsicID() == Intrinsic::memory_region_decl) {
+      FoundMemRegDecl = true;
+      V1 = CB1->getArgOperand(0);
+      if (auto *End1 = dyn_cast<ConstantInt>(CB1->getArgOperand(2)))
+        if (auto End1Val = End1->getZExtValue(); 
+            End1Val > 0 && V1Size.hasValue() && End1Val < V1Size.getValue())
+          V1Size = LocationSize::upperBound(End1Val);
+    }
+    if (CB2 && CB2->getIntrinsicID() == Intrinsic::memory_region_decl) {
+      FoundMemRegDecl = true;
+      V2 = CB2->getArgOperand(0);
+      if (auto *End2 = dyn_cast<ConstantInt>(CB2->getArgOperand(2)))
+        if (auto End2Val = End2->getZExtValue(); 
+            End2Val > 0 && V2Size.hasValue() && End2Val < V2Size.getValue())
+          V2Size = LocationSize::upperBound(End2Val);
+    }
+    if (FoundMemRegDecl)
+      return aliasCheck(V1, V1Size, V2, V2Size, AAQI, CtxI);
   }
 
   // If the size of one access is larger than the entire object on the other
   // side, then we know such behavior is undefined and can assume no alias.
   bool NullIsValidLocation = NullPointerIsDefined(&F);
   if ((isObjectSmallerThan(
           O2, getMinimalExtentFrom(*V1, V1Size, DL, NullIsValidLocation), DL,
           TLI, NullIsValidLocation)) ||

llvm/lib/Analysis/ConstantFolding.cpp

bool llvm::canConstantFoldCallTo(const CallBase *Call, const Function *F) {
   case Intrinsic::bswap:
   case Intrinsic::ctpop:
   case Intrinsic::ctlz:
   case Intrinsic::cttz:
   case Intrinsic::fshl:
   case Intrinsic::fshr:
   case Intrinsic::launder_invariant_group:
   case Intrinsic::strip_invariant_group:
+  case Intrinsic::memory_region_decl:
   case Intrinsic::masked_load:
   case Intrinsic::get_active_lane_mask:
   case Intrinsic::abs:
   case Intrinsic::smax:
   case Intrinsic::smin:
   case Intrinsic::umax:
   case Intrinsic::umin:
   case Intrinsic::sadd_with_overflow:

llvm/lib/Analysis/InlineCost.cpp

if (IntrinsicInst *II = dyn_cast<IntrinsicInst>(&Call)) {
     case Intrinsic::localescape:
       HasUninlineableIntrinsic = true;
       return false;
     case Intrinsic::vastart:
       InitsVargArgs = true;
       return false;
     case Intrinsic::launder_invariant_group:
     case Intrinsic::strip_invariant_group:
+    case Intrinsic::memory_region_decl:
       if (auto *SROAArg = getSROAArgForValueOrNull(II->getOperand(0)))
         SROAArgValues[II] = SROAArg;
       return true;
     case Intrinsic::is_constant:
       return simplifyIntrinsicCallIsConstant(Call);
     case Intrinsic::objectsize:
       return simplifyIntrinsicCallObjectSize(Call);
     }

llvm/lib/Analysis/MemoryDependenceAnalysis.cpp

case Intrinsic::invariant_start:
       // These intrinsics don't really modify the memory, but returning Mod
       // will allow them to be handled conservatively.
       return ModRefInfo::Mod;
     case Intrinsic::invariant_end:
       Loc = MemoryLocation::getForArgument(II, 2, TLI);
       // These intrinsics don't really modify the memory, but returning Mod
       // will allow them to be handled conservatively.
       return ModRefInfo::Mod;
+    case Intrinsic::memory_region_decl:
+      Loc = MemoryLocation::getForArgument(II, 0, TLI);
+      return ModRefInfo::Ref;
     case Intrinsic::masked_load:
       Loc = MemoryLocation::getForArgument(II, 0, TLI);
       return ModRefInfo::Ref;
     case Intrinsic::masked_store:
       Loc = MemoryLocation::getForArgument(II, 1, TLI);
       return ModRefInfo::Mod;
     default:
       break;

llvm/lib/Analysis/MemoryLocation.cpp

if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(Call)) {
     case Intrinsic::invariant_start:
       assert(ArgIdx == 1 && "Invalid argument index");
       return MemoryLocation(
           Arg,
           LocationSize::precise(
               cast<ConstantInt>(II->getArgOperand(0))->getZExtValue()),
           AATags);
 
+    case Intrinsic::memory_region_decl:
+      assert(ArgIdx == 0 && "Invalid argument index");
+      return MemoryLocation(
+          Arg,
+          LocationSize::precise(
+              cast<ConstantInt>(II->getArgOperand(2))->getZExtValue() - 
+              cast<ConstantInt>(II->getArgOperand(1))->getZExtValue()),
+          AATags);
+
     case Intrinsic::masked_load:
       assert(ArgIdx == 0 && "Invalid argument index");
       return MemoryLocation(
           Arg,
           LocationSize::upperBound(DL.getTypeStoreSize(II->getType())),
           AATags);
 
     case Intrinsic::masked_store:

llvm/lib/Analysis/MemorySSA.cpp

if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
     //
     // FIXME: We probably don't actually want MemorySSA to model these at all
     // (including creating MemoryAccesses for them): we just end up inventing
     // clobbers where they don't really exist at all. Please see D43269 for
     // context.
     switch (II->getIntrinsicID()) {
     case Intrinsic::invariant_start:
     case Intrinsic::invariant_end:
+    case Intrinsic::memory_region_decl:
     case Intrinsic::assume:
     case Intrinsic::experimental_noalias_scope_decl:
     case Intrinsic::pseudoprobe:
       return false;
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_label:
     case Intrinsic::dbg_value:
       llvm_unreachable("debuginfo shouldn't have associated defs!");

llvm/lib/Analysis/ObjCARCInstKind.cpp

static bool isInertIntrinsic(unsigned ID) {
   case Intrinsic::eh_sjlj_lsda:
   case Intrinsic::eh_sjlj_functioncontext:
   case Intrinsic::init_trampoline:
   case Intrinsic::adjust_trampoline:
   case Intrinsic::lifetime_start:
   case Intrinsic::lifetime_end:
   case Intrinsic::invariant_start:
   case Intrinsic::invariant_end:
+  case Intrinsic::memory_region_decl:
   // Don't let dbg info affect our results.
   case Intrinsic::dbg_declare:
   case Intrinsic::dbg_value:
   case Intrinsic::dbg_label:
     // Short cut: Some intrinsics obviously don't use ObjC pointers.
     return true;
   default:
     return false;

llvm/lib/Analysis/ValueTracking.cpp

llvm::getArgumentAliasingToReturnedPointer(const CallBase *Call,
   return nullptr;
 }
 
 bool llvm::isIntrinsicReturningPointerAliasingArgumentWithoutCapturing(
     const CallBase *Call, bool MustPreserveNullness) {
   switch (Call->getIntrinsicID()) {
   case Intrinsic::launder_invariant_group:
   case Intrinsic::strip_invariant_group:
+  case Intrinsic::memory_region_decl:
   case Intrinsic::aarch64_irg:
   case Intrinsic::aarch64_tagp:
   // The amdgcn_make_buffer_rsrc function does not alter the address of the
   // input pointer (and thus preserve null-ness for the purposes of escape
   // analysis, which is where the MustPreserveNullness flag comes in to play).
   // However, it will not necessarily map ptr addrspace(N) null to ptr
   // addrspace(8) null, aka the "null descriptor", which has "all loads return
   // 0, all stores are dropped" semantics. Given the context of this intrinsic

llvm/lib/CodeGen/CodeGenPrepare.cpp

case Intrinsic::aarch64_stxr: {
       ExtVal->moveBefore(CI);
       // Mark this instruction as "inserted by CGP", so that other
       // optimizations don't touch it.
       InsertedInsts.insert(ExtVal);
       return true;
     }
 
     case Intrinsic::launder_invariant_group:
-    case Intrinsic::strip_invariant_group: {
+    case Intrinsic::strip_invariant_group:
+    case Intrinsic::memory_region_decl: {
       Value *ArgVal = II->getArgOperand(0);
       auto it = LargeOffsetGEPMap.find(II);
       if (it != LargeOffsetGEPMap.end()) {
         // Merge entries in LargeOffsetGEPMap to reflect the RAUW.
         // Make sure not to have to deal with iterator invalidation
         // after possibly adding ArgVal to LargeOffsetGEPMap.
         auto GEPs = std::move(it->second);
         LargeOffsetGEPMap[ArgVal].append(GEPs.begin(), GEPs.end());

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

bool FastISel::selectIntrinsicCall(const IntrinsicInst *II) {
   case Intrinsic::objectsize:
     llvm_unreachable("llvm.objectsize.* should have been lowered already");
 
   case Intrinsic::is_constant:
     llvm_unreachable("llvm.is.constant.* should have been lowered already");
 
   case Intrinsic::launder_invariant_group:
   case Intrinsic::strip_invariant_group:
+  case Intrinsic::memory_region_decl:
   case Intrinsic::expect: {
     Register ResultReg = getRegForValue(II->getArgOperand(0));
     if (!ResultReg)
       return false;
     updateValueMap(II, ResultReg);
     return true;
   }
   case Intrinsic::experimental_stackmap:

llvm/lib/IR/Value.cpp

if (auto *GEP = dyn_cast<GEPOperator>(V)) {
         // but it can't be marked with returned attribute, that's why it needs
         // special case.
         if (StripKind == PSK_ForAliasAnalysis &&
             (Call->getIntrinsicID() == Intrinsic::launder_invariant_group ||
              Call->getIntrinsicID() == Intrinsic::strip_invariant_group)) {
           V = Call->getArgOperand(0);
           continue;
         }
+        if (Call->getIntrinsicID() == Intrinsic::memory_region_decl) {
+          V = Call->getArgOperand(0);
+          continue;
+        }
       }
       return V;
     }
     assert(V->getType()->isPointerTy() && "Unexpected operand type!");
   } while (Visited.insert(V).second);
 
   return V;
 }

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

if (auto *PN = dyn_cast<PHINode>(PtrOp)) {
     NewGEP->insertInto(GEP.getParent(), GEP.getParent()->getFirstInsertionPt());
     return replaceOperand(GEP, 0, NewGEP);
   }
 
   if (auto *Src = dyn_cast<GEPOperator>(PtrOp))
     if (Instruction *I = visitGEPOfGEP(GEP, Src))
       return I;
 
+  if (auto *SrcIntrCall = dyn_cast<CallBase>(PtrOp); SrcIntrCall &&
+      SrcIntrCall->getIntrinsicID() == Intrinsic::memory_region_decl)
+    if (auto *Src = dyn_cast<GEPOperator>(SrcIntrCall->getArgOperand(0)))
+      if (Instruction *I = visitGEPOfGEP(GEP, Src)) {
+          I->insertInto(GEP.getParent(), GEP.getParent()->getFirstInsertionPt());
+          llvm::Instruction *Call = Builder.CreateCall(
+              Intrinsic::getDeclaration(
+                GEP.getModule(), 
+                Intrinsic::memory_region_decl, 
+                { SrcIntrCall->getType() }),
+              {I, 
+               SrcIntrCall->getArgOperand(1), 
+               SrcIntrCall->getArgOperand(2)},
+              "arrayidx.bounded");
+          return replaceInstUsesWith(GEP, Call);
+      }
+
   // Skip if GEP source element type is scalable. The type alloc size is unknown
   // at compile-time.
   if (GEP.getNumIndices() == 1 && !IsGEPSrcEleScalable) {
     unsigned AS = GEP.getPointerAddressSpace();
     if (GEP.getOperand(1)->getType()->getScalarSizeInBits() ==
         DL.getIndexSizeInBits(AS)) {
       uint64_t TyAllocSize = DL.getTypeAllocSize(GEPEltType).getFixedValue();
 
for (User *U : PI->users()) {
           case Intrinsic::invariant_end:
           case Intrinsic::lifetime_start:
           case Intrinsic::lifetime_end:
           case Intrinsic::objectsize:
             Users.emplace_back(I);
             continue;
           case Intrinsic::launder_invariant_group:
           case Intrinsic::strip_invariant_group:
+          case Intrinsic::memory_region_decl:
             Users.emplace_back(I);
             Worklist.push_back(I);
             continue;
           }
         }
 
         if (isRemovableWrite(*cast<CallBase>(I), PI, TLI)) {
           Users.emplace_back(I);

llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp

 bool isNoopIntrinsic(Instruction *I) {
   if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(I)) {
     switch (II->getIntrinsicID()) {
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
     case Intrinsic::invariant_end:
     case Intrinsic::launder_invariant_group:
     case Intrinsic::assume:
+    case Intrinsic::memory_region_decl:
       return true;
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_label:
     case Intrinsic::dbg_value:
       llvm_unreachable("Intrinsic should not be modeled in MemorySSA");
     default:
       return false;
     }

llvm/lib/Transforms/Scalar/SROA.cpp

private:
 
   void visitAddrSpaceCastInst(AddrSpaceCastInst &ASC) {
     if (ASC.use_empty())
       return markAsDead(ASC);
 
     return Base::visitAddrSpaceCastInst(ASC);
   }
 
+  void visitMemRegDeclInst(MemRegDeclInst &I) {
+    if (I.use_empty())
+      return markAsDead(I);
+
+    return Base::visitMemRegDeclInst(I);
+  }
+
   void visitGetElementPtrInst(GetElementPtrInst &GEPI) {
     if (GEPI.use_empty())
       return markAsDead(GEPI);
 
     if (SROAStrictInbounds && GEPI.isInBounds()) {
       // FIXME: This is a manually un-factored variant of the basic code inside
       // of GEPs with checking of the inbounds invariant specified in the
       // langref in a very strict sense. If we ever want to enable
bool visitBitCastInst(BitCastInst &BC) {
     return false;
   }
 
   bool visitAddrSpaceCastInst(AddrSpaceCastInst &ASC) {
     enqueueUsers(ASC);
     return false;
   }
 
+  bool visitMemRegDeclInst(MemRegDeclInst &I) {
+    enqueueUsers(I);
+    return false;
+  }
+
   // Fold gep (select cond, ptr1, ptr2) => select cond, gep(ptr1), gep(ptr2)
   bool foldGEPSelect(GetElementPtrInst &GEPI) {
     if (!GEPI.hasAllConstantIndices())
       return false;
 
     SelectInst *Sel = cast<SelectInst>(GEPI.getPointerOperand());
 
     LLVM_DEBUG(dbgs() << "  Rewriting gep(select) -> select(gep):"

llvm/lib/Transforms/Utils/Local.cpp

bool llvm::wouldInstructionBeTriviallyDead(Instruction *I,
   // Special case intrinsics that "may have side effects" but can be deleted
   // when dead.
   if (IntrinsicInst *II = dyn_cast<IntrinsicInst>(I)) {
     // Safe to delete llvm.stacksave and launder.invariant.group if dead.
     if (II->getIntrinsicID() == Intrinsic::stacksave ||
         II->getIntrinsicID() == Intrinsic::launder_invariant_group)
       return true;
 
+    if (II->getIntrinsicID() == Intrinsic::memory_region_decl)
+      return true;
+
     if (II->isLifetimeStartOrEnd()) {
       auto *Arg = II->getArgOperand(1);
       // Lifetime intrinsics are dead when their right-hand is undef.
       if (isa<UndefValue>(Arg))
         return true;
       // If the right-hand is an alloc, global, or argument and the only uses
       // are lifetime intrinsics then the intrinsics are dead.
       if (isa<AllocaInst>(Arg) || isa<GlobalValue>(Arg) || isa<Argument>(Arg))

llvm/test/Transforms/InstCombine/gep-mem-reg-decl.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S < %s -passes=instcombine | FileCheck %s
+
+%struct.S = type { [1024 x i32], [1024 x i32] }
+
+declare ptr @llvm.memory.region.decl.p0(ptr readnone, i64, i64)
+
+; test that a GEP of a GEP can be combined in the presence of
+; intermediate intrinsic calls
+define i32 @test_gep_of_gep(ptr noundef %s, i64 %i) {
+; CHECK-LABEL: @test_gep_of_gep(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX21:%.*]] = getelementptr inbounds [[STRUCT_S:%.*]], ptr [[S:%.*]], i64 0, i32 1, i64 [[I:%.*]]
+; CHECK-NEXT:    [[ARRAYIDX_BOUNDED:%.*]] = call ptr @llvm.memory.region.decl.p0(ptr nonnull [[ARRAYIDX21]], i64 0, i64 4096)
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, ptr [[ARRAYIDX_BOUNDED]], align 4
+; CHECK-NEXT:    ret i32 [[TMP0]]
+;
+entry:
+  %arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %s, i64 0, i64 4096)
+  %B = getelementptr inbounds %struct.S, ptr %s, i32 0, i32 1
+  %arrayidx.bounded1 = call ptr @llvm.memory.region.decl.p0(ptr %B, i64 0, i64 4096)
+  %arrayidx2 = getelementptr inbounds [1024 x i32], ptr %arrayidx.bounded1, i64 0, i64 %i
+  %0 = load i32, ptr %arrayidx2, align 4
+  ret i32 %0
+}
+
+; ensure that InstructionCombining.cpp:isAllocSiteRemovable()
+; does not think that pointers may escape through the intrinsic
+define i32 @test_erase_alloc_site(i32 %i) {
+; CHECK-LABEL: @test_erase_alloc_site(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret i32 0
+;
+entry:
+  %arr = alloca [1000 x i32], align 8
+  %arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %arr, i64 0, i64 8000)
+  %arrayidx1 = getelementptr inbounds i32, ptr %arrayidx.bounded, i32 %i
+  store i32 1, ptr %arrayidx1, align 8
+  ret i32 0
+}
+
+; ensure that we can constant-fold a call to the intrinsic,
+; thereby allowing isAllocSiteRemovable() to properly recognize
+; a redundant alloca
+define i32 @test_constant_fold_intrinsic() {
+; CHECK-LABEL: @test_constant_fold_intrinsic(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret i32 1
+;
+entry:
+  %s = alloca { i32 }, align 8
+  store i32 1, ptr %s, align 8
+  %arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %s, i64 0, i64 8)
+  %0 = load i32, ptr %arrayidx.bounded, align 8
+  ret i32 %0
+}

llvm/test/Transforms/SROA/mem-reg-decl.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-PRESERVE-CFG
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-MODIFY-CFG
+
+declare ptr @llvm.memory.region.decl.p0(ptr readnone, i64, i64)
+
+; ensure that SROA can "see through" the intrinsic call
+define i32 @test1() {
+; CHECK-LABEL: @test1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret i32 1
+;
+entry:
+  %s = alloca { i32 }, align 8
+  store i32 1, ptr %s, align 8
+  %arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %s, i64 0, i64 8)
+  %0 = load i32, ptr %arrayidx.bounded, align 8
+  ret i32 %0
+}
+
+; variation of the above test
+define i32 @test2() {
+; CHECK-LABEL: @test2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 undef, undef
+; CHECK-NEXT:    ret i32 [[ADD]]
+;
+entry:
+  %s = alloca [1024 x i32], align 4
+  %t = alloca [1024 x i32], align 4
+  %arrayidx.bounded = call ptr @llvm.memory.region.decl.p0(ptr %s, i64 0, i64 4096)
+  %0 = load i32, ptr %arrayidx.bounded, align 4
+  %arrayidx.bounded1 = call ptr @llvm.memory.region.decl.p0(ptr %t, i64 0, i64 4096)
+  %arrayidx2 = getelementptr inbounds i32, ptr %arrayidx.bounded1, i32 1
+  %1 = load i32, ptr %arrayidx2, align 4
+  %add = add nsw i32 %0, %1
+  ret i32 %add
+}
+
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-MODIFY-CFG: {{.*}}
+; CHECK-PRESERVE-CFG: {{.*}}