[IR][RFC] Memory region declaration intrinsic

[lebedev.ri]

Summary

See https://lists.llvm.org/pipermail/llvm-dev/2021-December/154249.html for the RFC itself

[lebedev.ri]

RFC: https://lists.llvm.org/pipermail/llvm-dev/2021-December/154249.html

[lebedev.ri]

Drop nocapture as per @efriedma / https://alive2.llvm.org/ce/z/EmvTq9.

[lebedev.ri]

Some more blurb.

[courbet]

llvm/docs/LangRef.rst

20833

*a memory region

Maybe even something like:

annotates a pointer with memory region that is accessible through that pointer.

20842

Do we want to specify that end_offset > begin_offset ?

20868–20870

I have a hard time parsing that sentence, did you mean something like:

The only ``inbounds`` addresses that can be computed from pointer ``%ptr`` are those that lie within offsets ``[-8, 24]`` from that pointer.
Other offsets will result in a :ref:`poison value <poisonvalues>`.

20869

Given the semantics above, 24 should be excluded: [8,24)

[lebedev.ri]

llvm/docs/LangRef.rst

20842

end_offset can not be required to be bigger than begin_offset.
E.g. consider the following 32-byte region: [%ptr - 32, %ptr + 0],
where %ptr is the end pointer.

Likewise, i don't think we can specify that
the total region size is non-zero,
because then we can't declare empty memory regions,
where there only is a end pointer.

What i think is something that still needs to be better clarified,
is whether the begin_offset should specify the
offset from the beginning of the memory region to the pointer (i.e. be non-negative (begin_offset s>= 0)),
or the offset from the pointer to the beginning of the memory region (i.e. be non-positive (begin_offset s<= 0)),

I guess, given that end_offset specifies the offset from the pointer
to the end of the region, we want begin_offset to be symmetrical,
and specify the offset from the pointer to the beginning of the memory region
(i.e. be non-positive (begin_offset s<= 0)).

20869

Hm, no, why? Is the end pointer not inbounds?

[courbet]

llvm/docs/LangRef.rst

20842

end_offset can not be required to be bigger than begin_offset.
E.g. consider the following 32-byte region: [%ptr - 32, %ptr + 0],
where %ptr is the end pointer.

In that base, begin_offset is -32 and end_offset is 0, right ? So we still have begin_offset <= end_offset.

Likewise, i don't think we can specify that
the total region size is non-zero,
because then we can't declare empty memory regions,
where there only is a end pointer.

Yes you're right, obviously for the case of int a[0]. BTW there is an inconsistency between GNU C and C99 on this for flexible array members: in the former case we should not emit an annotation for this while we should for the latter.

What i think is something that still needs to be better clarified,
is whether the begin_offset should specify the
offset from the beginning of the memory region to the pointer (i.e. be non-negative (begin_offset s>= 0)),
or the offset from the pointer to the beginning of the memory region (i.e. be non-positive (begin_offset s<= 0)),

My vote goes to begin_offset >= 0 => memory region begins after the pointer.

[lebedev.ri]

Misc wording improvements.

[lebedev.ri]

(@courbet i believe i have addressed your comments)

[courbet]

llvm/docs/LangRef.rst

20852–20855

I'm still confused as to why we need that restriction.

For example, why do we want to disallow the following:

%p2 = getelementptr inbounds i32, i32* %p1, i64 -42
%p2_bounded = @llvm.memory.region.decl.p0i8(%p2, 42, 43)
%p3 = getelementptr inbounds i32, i32* %p2_bounded, i64 42
%v = load i32, i32* %p3, align 4

[lebedev.ri]

llvm/docs/LangRef.rst

10364–10371

I'm still confused as to why we need that restriction.

[nikic]

llvm/docs/LangRef.rst

20852–20855

At least as defined, %p2_bounded would already be poison in that case, because %p2_bounded would be before the declared memory region. The fact that %p3 would point back into the memory region doesn't matter in that case, because the pointer is already poison.

20896

Doesn't this sentence contradict the first sentence in "Semantics"? If you want to make a distinction between inbounds/non-inbounds, then I think you have to do that in terms of restricting the visible allocated object, rather than saying that any pointer based on it cannot be outside the range. That would mean that something like %p = memory.region.decl(%p0, 8, 16) would not be poison, though dereferencing it would be and doing an inbounds gep would be, while doing a non-inbounds gep by 8 and then dereferencing would be legal.

llvm/include/llvm/IR/Intrinsics.td

1197

While technically correct, annotating it Returned means that the intrinsic is simply going to be optimized away. You want to drop returned here and instead add the intrinsic to isIntrinsicReturningPointerAliasingArgumentWithoutCapturing(), which handles various intrinsics of this kind.

[lebedev.ri]

Rebased, addressed review notes.
Ping, i feel like i'd want to start making progress here.

[lebedev.ri]

llvm/docs/LangRef.rst

20896

Hmm, i do not remember why i've added that footprint here.
I believe we don't want to make a distinction between inbounds/non-inbounds.

[jdoerfert]

I still believe this should be a generic assume.passthrough intrinsic with an operand bundle for the specific use case.
I briefly described that here: https://lists.llvm.org/pipermail/llvm-dev/2021-December/154281.html

The intrinsic would also be speculatable, readnone, all the good stuff.
The resulting value has all the annotated properties *or* is poison.
This matches attributes on call sites and as such we can retain attributes from call sites:

call @foo(i8* %p, i8* align(16) nonnull %p);

will become

%arg1 = llvm.assume.passthrough(%p) ["align"(16), "nonnull"]
call @foo(i8* %p, i8* align(16) nonnull %arg1);

just before inlining. This way the information is retained properly.
Note that we should already retain it with llvm.assume if there is a noundef as well.

[lebedev.ri]

Does anybody else have any thoughts/opinions on this?

In D115274#3361333, @jdoerfert wrote:

I still believe this should be a generic assume.passthrough intrinsic with an operand bundle for the specific use case.

Is that a blocker?
I'm guessing you also argue that the noalias stuff should likewise also be designed this way?

[nlopes]

This concept of sub-objects also shows up in the 'restrict' stuff. The difference is that with 'restrict' the aliasing constraints are dynamic: the promise is that e.g. two (dynamic) accesses in the original program don't alias. Here the constrains are static.

Ideally we would have a single solution for all this stuff, but instead right now we have multiple disjoint solutions. Each one will have to go through the pain of patching all optimizers so they understand the new intrinsics so they don't throttle back. It's not ideal..
I just wish we could sit physically in some place for a few days and work on a solution once and for all.

It's also very hard to discuss designs when so much code has been written already (especially for the 'restrict' stuff).

Regarding this patch in particular, I would mention that the storage of the returned sub-object is shared with the parent object, just to make it explicit. The concept looks ok.
Again, my concern is that to be able to use this new intrinsic from clang by default will take a lot of work. You'll likely face many perf regressions before patching a bunch of optimizations.

[lebedev.ri]

Thanks for taking a look!

In D115274#3362337, @nlopes wrote:

This concept of sub-objects also shows up in the 'restrict' stuff. The difference is that with 'restrict' the aliasing constraints are dynamic: the promise is that e.g. two (dynamic) accesses in the original program don't alias. Here the constrains are static.

Ideally we would have a single solution for all this stuff, but instead right now we have multiple disjoint solutions. Each one will have to go through the pain of patching all optimizers so they understand the new intrinsics so they don't throttle back. It's not ideal..

Right.

I just wish we could sit physically in some place for a few days and work on a solution once and for all.

It's also very hard to discuss designs when so much code has been written already (especially for the 'restrict' stuff).

FWIW, currently i don't have any further code for this.

Regarding this patch in particular, I would mention that the storage of the returned sub-object is shared with the parent object, just to make it explicit. The concept looks ok.

Done.

Again, my concern is that to be able to use this new intrinsic from clang by default will take a lot of work. You'll likely face many perf regressions before patching a bunch of optimizations.

I acknowledge this reality. I'm just not sure we have a better approach than that,
it would be good to come up with a "a solution once and for all",
but i'm not sure how that would look.

[lebedev.ri]

ping

[arichardson]

In general I like this and would be very happy if we had a better way of dealing with sub-objects in LLVM.

Looking at this from the CHERI perspective, it seems like we could enforce this intrinsic at runtime by lowering it the same way as we current do for the out-of-tree`@llvm.cheri.bounds.set()`. This intrinsic also creates a memory subregion, the only difference as far as I can see it is that negative start offsets have to be encoded by doing a negative GEP first since we only have a pointer+length argument. And of course that it is enforced at runtime by narrowing the bounds that are part of the CHERI pointer.

llvm/docs/LangRef.rst

20876

Do you envision this being used for all sub-object pointer creations? If so it might need a flag to disable it since it might break some C patterns such as container_of.

According to https://godbolt.org/z/evTbejaMf the container_of macro results in an inbounds GEP, so with sufficient inlining things might break?

About three years ago I spent quite a lot of time enforcing sub-object bounds at runtime using CHERI. Almost all code works just fine but there are things such as container_of() that require opt-out annotations. I wrote about the incompatibilities that I found in Chapter 5 of https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-949.pdf. TL;DR: not many changes needed - about 50 annotations across the entire FreeBSD source tree. Almost all annotations due to container_of or emulation of C++ inheritance in C.

[lebedev.ri]

Thank you for taking a look!

llvm/docs/LangRef.rst

20876

I'm not sure what you mean by "all sub-object pointer creations".

Roughly, front-ends should emit this intrinsic on some pointer with some bounds
iff they know that it would be UB to go *from that specific pointer* (aka, as per def-use)
outside of the specified bounds.

The one case we know of is C arrays within structs.

[arichardson]

llvm/docs/LangRef.rst

20876

By sub-object pointer creations I mean something like &obj->field. You could conceivable treat that as declaring a new sub-object bounded to just field. E.g. something like this: https://godbolt.org/z/bM7j1bxqs

[lebedev.ri]

llvm/docs/LangRef.rst

20876

I think the question is slightly wrong. It's up to front-ends to decide when they can and can't emit this.

If you are asking about https://godbolt.org/z/adq5EWx17,
then as per previous conversations about this, i do believe that code to be well-defined and not UB.

IOW, i do not believe that as per the current C/C++ standards wording each data member of a struct
is it's own sub-object from which you are not allowed to get to it's neighbor objects,
But perhaps @aaron.ballman wants to correct me on this.

[arichardson]

llvm/docs/LangRef.rst

20876

Yes absolutely agreed that this is purely up to the frontend, I just assumed you had plans to update clang to emit the new intrinsic.

It has been a long time since I looked at the C standard with regards to subobjects but if I recall correctly you are right that this is not defined as being illegal. However, doesn't that also mean that you can access the member before/after an embedded array?

[lebedev.ri]

llvm/docs/LangRef.rst

20876

Right. Having a pointer to the array member of a struct isn't going to do anything.
The magic happens when you have a pointer to the *element* of the array member:
https://godbolt.org/z/cjE5bY4G4 <- manually crafted

[aaron.ballman]

llvm/docs/LangRef.rst

20876

IOW, i do not believe that as per the current C/C++ standards wording each data member of a struct is it's own sub-object from which you are not allowed to get to it's neighbor objects, But perhaps @aaron.ballman wants to correct me on this.

The rules on pointer arithmetic in C++: http://eel.is/c++draft/expr.add#4, so https://godbolt.org/z/adq5EWx17 looks like UB to me in C++. C2x 6.5.6p9 reads similarly, so it also looks like UB to me in C.

What language rules make you think otherwise?

[lebedev.ri]

llvm/docs/LangRef.rst

20876

Hm, thank you for correcting me, that does not match my recollection from the previous time we discussed this.
What about https://godbolt.org/z/8T54zxT43, is that pointer well-defined?

[aaron.ballman]

llvm/docs/LangRef.rst

20876

I believe that's also UB for the same reason -- the subtraction violates http://eel.is/c++draft/expr.add#4.3, so we can't say much about the resulting pointer value. Note, a slightly different case of using + 1 instead of - 1 is valid because any object can be treated as an array of one (https://eel.is/c++draft/basic.compound#3.sentence-12) and a one-past-the-end pointer is valid (http://eel.is/c++draft/expr.add#4.2) so long as it's not dereferenced.

[lebedev.ri]

@aaron.ballman thank you!

llvm/docs/LangRef.rst

20876

Right, +1 (end pointer) is non-dereferenceable, and is fine.

Hmmmm, that sounds too good to be true. Then i stand corrected,
i guess we could emit it even for such cases, although that will likely break code,
and i'm not sure if a sanitizer could be implemented to catch the UB.

[jdoerfert]

In D115274#3361864, @lebedev.ri wrote:

Does anybody else have any thoughts/opinions on this?

In D115274#3361333, @jdoerfert wrote:

I still believe this should be a generic assume.passthrough intrinsic with an operand bundle for the specific use case.

Is that a blocker?

No. Though, we should put a TODO somewhere if someone comes looking for a nice project to cleanup our messes.

[lebedev.ri]

In D115274#3403284, @jdoerfert wrote:

In D115274#3361864, @lebedev.ri wrote:

Does anybody else have any thoughts/opinions on this?

In D115274#3361333, @jdoerfert wrote:

I still believe this should be a generic assume.passthrough intrinsic with an operand bundle for the specific use case.

Is that a blocker?

No. Though, we should put a TODO somewhere if someone comes looking for a nice project to cleanup our messes.

Awesome! There has been a lot of approval of the proposal, but does anyone want to rubber-stamp it before i land this?

[arichardson]

I am happy with this but I feel like someone else should also approve it :)

llvm/docs/LangRef.rst

20876

Not a sanitizer for x86, but you can use CHERI LLVM and compile with -cheri-bounds=subobject-safe to detect such violations at runtime. Code can then be run on QEMU or also on Arm's Morello boards that were recently released (if you happen to be one of the recipients).
We use that flag for the FreeBSD kernel and it works very well with only a few minor adjustments.

I believe that a sanitizer for non-CHERI hardware would require complete provenance (bounds) shadow memory for every pointer so would be a big engineering effort.

[aaron.ballman]

FWIW, this sounds like a generally useful feature to me and I believe it can be used to better express some C and C++ semantics.

[dtemirbulatov]

Ping. Do you plan to commit the change? Any blockage?

[lebedev.ri]

In D115274#3432740, @dtemirbulatov wrote:

Ping. Do you plan to commit the change? Any blockage?

I don't know. I think i'm waiting for the dust to settle on the github pr debacle.

[nikic]

In D115274#3432750, @lebedev.ri wrote:

In D115274#3432740, @dtemirbulatov wrote:

Ping. Do you plan to commit the change? Any blockage?

I don't know. I think i'm waiting for the dust to settle on the github pr debacle.

Any context on that? I feel like I missed something here...

Anyway, I find the overall wording here still a bit confusing. I would put more emphasis on the fact that this effectively restricts the "allocated object" to a certain offset range, which should have the following three effects:

• For non-inbounds GEPs, this should have no effect.
• For inbounds GEPs, if the GEP goes outside the range [ptr+begin_offset, ptr+end_offset], the GEP result is poison.
• For accesses, if the access is outside the range [ptr+begin_offset, ptr+end_offset-1], the behavior is undefined.

The current wording mostly emphasizes the middle point, but not so much the first and the last. And the fact that the "one past the end of the region" address is only valid for GEP inbounds but not for accesses is probably important for optimization purposes.

llvm/docs/LangRef.rst

20841

nit: "offset to" -> "offset from"?

llvm/include/llvm/IR/Intrinsics.td

1196

nit: Is the ReadNone here actually meaningful if the whole intrinsic is already IntrNoMem?

[lebedev.ri]

In D115274#3432812, @nikic wrote:

In D115274#3432750, @lebedev.ri wrote:

In D115274#3432740, @dtemirbulatov wrote:

Ping. Do you plan to commit the change? Any blockage?

I don't know. I think i'm waiting for the dust to settle on the github pr debacle.

Any context on that? I feel like I missed something here...

The TLDR is that i don't feel like contributing to projects that continuously shit on their fellow contributors.
Migration from phab to github pull requests will likely be my tipping point.

Anyway, I find the overall wording here still a bit confusing. I would put more emphasis on the fact that this effectively restricts the "allocated object" to a certain offset range, which should have the following three effects:

• For non-inbounds GEPs, this should have no effect.
• For inbounds GEPs, if the GEP goes outside the range [ptr+begin_offset, ptr+end_offset], the GEP result is poison.
• For accesses, if the access is outside the range [ptr+begin_offset, ptr+end_offset-1], the behavior is undefined.

The current wording mostly emphasizes the middle point, but not so much the first and the last. And the fact that the "one past the end of the region" address is only valid for GEP inbounds but not for accesses is probably important for optimization purposes.

[brooks]

This comment was removed by brooks.

[kiranchandramohan]

It will be great to have this patch landed. It will help to rewrite D126533 and make progress. Alternatively, would it be OK for someone else to commandeer this patch?
Apologies for the community issues around phabricator and github pr. But I guess it will take time to solve.

llvm/docs/LangRef.rst

define i32* @foo(%struct.ST* %s) {
       %t1 = getelementptr %struct.ST, %struct.ST* %s, i32 1                        ; yields %struct.ST*:%t1
       %t2 = getelementptr %struct.ST, %struct.ST* %t1, i32 0, i32 2                ; yields %struct.RT*:%t2
       %t3 = getelementptr %struct.RT, %struct.RT* %t2, i32 0, i32 1                ; yields [10 x [20 x i32]]*:%t3
       %t4 = getelementptr [10 x [20 x i32]], [10 x [20 x i32]]* %t3, i32 0, i32 5  ; yields [20 x i32]*:%t4
       %t5 = getelementptr [20 x i32], [20 x i32]* %t4, i32 0, i32 13               ; yields i32*:%t5
       ret i32* %t5
     }
 
+.. _getelementptr_inbounds:
+
 If the ``inbounds`` keyword is present, the result value of the
 ``getelementptr`` is a :ref:`poison value <poisonvalues>` if one of the
 following rules is violated:
 
 *  The base pointer has an *in bounds* address of an allocated object, which
    means that it points into an allocated object, or to its end. The only
    *in bounds* address for a null pointer in the default address-space is the
    null pointer itself.
 *  If the type of an index is larger than the pointer index type, the
    truncation to the pointer index type preserves the signed value.
 *  The multiplication of an index by the type size does not wrap the pointer
    index type in a signed sense (``nsw``).
 *  The successive addition of offsets (without adding the base address) does
    not wrap the pointer index type in a signed sense (``nsw``).
 *  The successive addition of the current address, interpreted as an unsigned
    number, and an offset, interpreted as a signed number, does not wrap the

 Semantics:
 """"""""""
 
 Returns another pointer that aliases its argument but which has no associated
 ``invariant.group`` metadata.
 It does not read any memory and can be speculated.
 
 
+.. _int_memory_region_decl:
+
+'``llvm.memory.region.decl``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare i8* @llvm.memory.region.decl.p0i8(i8* readnone <ptr>, i64 <begin_offset>, i64 <end_offset>) nofree nosync nounwind readnone speculatable willreturn
+
+Overview:
+"""""""""
+
+The '``llvm.memory.region.decl``' intrinsic annotates a memory region.
+
+Arguments:
+""""""""""
+
+This is an overloaded intrinsic. The memory region can belong to any address
+space. The first argument is a pointer into the memory region. The returned
+pointer, which is the first argument, must belong to the same address space
+as the argument. The second argument specifies the offset to the pointer (the
+first argument) at which the memory region begins. The third argument specifies
+the offset to the pointer (the first argument) at which the memory region ends.
+
+Semantics:
+""""""""""
+
+The returned pointer, and, transitively, any pointer that is def-use based on
+that pointer, points into the memory region ``[ptr+begin_offset, ptr+end_offset]``,
+or is a :ref:`poison value <poisonvalues>` otherwise.
+
+Note that the ``begin_offset`` is non-positive (``begin_offset s<= 0``)
+and ``end_offset`` is non-negative (``end_offset s>= 0``) by definition,
+otherwise the ``ptr`` lies outside of declared memory region,
+and the returned pointer is a :ref:`poison value <poisonvalues>`.
+
+The intent of this intrinsic is to declare a memory region that is
+a (not necessarily strict) subset of an underlying allocated object,
+as far as :ref:`getelementptr <i_getelementptr>` is concerned.
+
+The storage of the returned sub-object is shared with the parent object,
+but note that the intrinsic does **not** state that the memory region
+*actually* points into the allocated object.
+
+Once a region is declared, it is **NOT** possible to define a strict superset
+of this region **in this def-use chain**, but it is legal to define nested
+(not necessarily strict) subsets.
+
+The only ``inbounds`` addresses that can be computed from pointer ``%ptr``
+are those that lie within offsets ``[begin_offset, end_offset]`` from that
+pointer. Other offsets will result in a :ref:`poison value <poisonvalues>`.
+
+This intrinsic is intended to be an optimization hint, there are no correctness
+concerns with completely ignoring and/or dropping it. The main use-case is
+to be able to annotate array bounds in C family of languages,
+which may allow alloca splitting, and better alias analysis.
+
+Example:
+""""""""
+
+::
+
+      %ptr = call i8* @llvm.memory.region.decl(i8* %baseptr, i64 -24, i64 8)
+
+declares that the ``%baseptr`` lies `+24` bytes into the memory region
+that consists of `32` bytes total.
+
+The only :ref:`inbounds <getelementptr_inbounds>` addresses that can be computed
+from pointer ``%ptr`` are those that lie within offsets ``[-24, 8]`` from
+that pointer. Other offsets will result in a :ref:`poison value <poisonvalues>`.
+(Provided that there actually is an underlying allocated object, that is
+a superset of the declared memory region.)
+
+Note that ``%ptr + 8`` is an (``inbounds``!) `end` pointer
+that can not be dereferenced.
+
 
 .. _constrainedfp:
 
 Constrained Floating-Point Intrinsics
 -------------------------------------
 
 These intrinsics are used to provide special handling of floating-point
 operations when specific rounding mode or floating-point exception behavior is

llvm/include/llvm/IR/Intrinsics.td

def int_launder_invariant_group : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                             [LLVMMatchType<0>],
                                             [IntrInaccessibleMemOnly, IntrSpeculatable, IntrWillReturn]>;
 
 
 def int_strip_invariant_group : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                           [LLVMMatchType<0>],
                                           [IntrSpeculatable, IntrNoMem, IntrWillReturn]>;
 
+// Declares that the returned pointer (the first argument),
+// and any pointer that is (transitively) def-use based on that pointer,
+// points into the memory region [ptr+begin_offset, ptr+end_offset),
+// or is poison otherwise.
+def int_memory_region_decl : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
+    [LLVMMatchType<0> /*ptr*/, llvm_i64_ty /*begin_offset*/, llvm_i64_ty /*end_offset*/],
+    [IntrNoMem, IntrSpeculatable, ReadNone<ArgIndex<0>>]>;
+
 //===------------------------ Stackmap Intrinsics -------------------------===//
 //
 def int_experimental_stackmap : DefaultAttrsIntrinsic<[],
                                   [llvm_i64_ty, llvm_i32_ty, llvm_vararg_ty],
                                   [Throws]>;
 def int_experimental_patchpoint_void : DefaultAttrsIntrinsic<[],
                                                  [llvm_i64_ty, llvm_i32_ty,
                                                   llvm_ptr_ty, llvm_i32_ty,

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

case Intrinsic::invariant_start: {
     return true;
   }
   case Intrinsic::invariant_end:
     return true;
   case Intrinsic::expect:
   case Intrinsic::annotation:
   case Intrinsic::ptr_annotation:
   case Intrinsic::launder_invariant_group:
-  case Intrinsic::strip_invariant_group: {
+  case Intrinsic::strip_invariant_group:
+  case Intrinsic::memory_region_decl: {
     // Drop the intrinsic, but forward the value.
     MIRBuilder.buildCopy(getOrCreateVReg(CI),
                          getOrCreateVReg(*CI.getArgOperand(0)));
     return true;
   }
   case Intrinsic::assume:
   case Intrinsic::experimental_noalias_scope_decl:
   case Intrinsic::var_annotation:

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

#include "llvm/IR/VPIntrinsics.def"
 
   case Intrinsic::is_constant:
     llvm_unreachable("llvm.is.constant.* should have been lowered already");
 
   case Intrinsic::annotation:
   case Intrinsic::ptr_annotation:
   case Intrinsic::launder_invariant_group:
   case Intrinsic::strip_invariant_group:
+  case Intrinsic::memory_region_decl:
     // Drop the intrinsic, but forward the value
     setValue(&I, getValue(I.getOperand(0)));
     return;
 
   case Intrinsic::assume:
   case Intrinsic::experimental_noalias_scope_decl:
   case Intrinsic::var_annotation:
   case Intrinsic::sideeffect:

llvm/test/CodeGen/AMDGPU/GlobalISel/memory_region_decl.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-unknown-amdhsa < %s | FileCheck %s
+
+declare i8* @llvm.memory.region.decl.p0i8(i8*, i64, i64)
+
+define i8* @test_i8(i8* %ptr, i64 %begin_off, i64 %end_off) {
+; CHECK-LABEL: test_i8:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+  %r = call i8* @llvm.memory.region.decl.p0i8(i8* %ptr, i64 %begin_off, i64 %end_off)
+  ret i8* %r
+}
+
+define i8* @test_i8_naive(i8* %ptr, i64 %begin_off, i64 %end_off) {
+; CHECK-LABEL: test_i8_naive:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+  ret i8* %ptr
+}

llvm/test/CodeGen/X86/memory_region_decl.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux | FileCheck %s --check-prefixes=X64
+; RUN: llc < %s -mtriple=i686-unknown-linux | FileCheck %s --check-prefixes=X86
+
+declare i8* @llvm.memory.region.decl.p0i8(i8*, i64, i64)
+
+define i8* @test_i8(i8* %ptr, i64 %begin_off, i64 %end_off) {
+; X64-LABEL: test_i8:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_i8:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    retl
+  %r = call i8* @llvm.memory.region.decl.p0i8(i8* %ptr, i64 %begin_off, i64 %end_off)
+  ret i8* %r
+}
+
+define i8* @test_i8_naive(i8* %ptr, i64 %begin_off, i64 %end_off) {
+; X64-LABEL: test_i8_naive:
+; X64:       # %bb.0:
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_i8_naive:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    retl
+  ret i8* %ptr
+}