Page MenuHomePhabricator

GC-parseable element atomic memcpy/memmove
ClosedPublic

Authored by apilipenko on Oct 5 2020, 4:39 PM.

Details

Summary

This change introduces a GC parseable lowering for element atomic memcpy/memmove intrinsics. This way runtime can provide an implementation which can take a safepoint during copy operation. See "GC-parseable element atomic memcpy/memmove" thread on llvm-dev for the background and details:
https://groups.google.com/g/llvm-dev/c/NnENHzmX-b8/m/3PyN8Y2pCAAJ

This lowering involves three things:

  1. The call is wrapped into a statepoint.
  2. The call is lowered to a different symbol __llvm_{memcpy|memmove}_element_unordered_atomic_safepoint_<element_size>.
  3. The arguments for the call are adjusted so as to make the base pointers available in the copy function.

In order to be consistent with other calls by default element atomic memcpy/memmove intrinsics are treated as non-leaf. So, by default GC parseable lowering is generated. Old GC leaf lowering will be generated if the call is explicitly marked "gc-leaf-function" attribute.

This default choice though introduces a minor transitioning issue. In some systems, e.g. ours, GC safepoints are coupled with deoptimization mechanism (this is controlled by a cl::opt rs4gc-allow-statepoint-with-no-deopt-info). In this case we can't have a statepoint without deopt information. Normally it's up to the frontend to make sure that non-leaf calls also have proper deopt state. But element atomic memcpy/memmove intrinsic calls might be generated by the optimizer, which is not aware of this coupling. If statepoints without deopt into are not allowed and we see a non-leaf memcpy/memmove without deopt state we treat it as a leaf copy and don't produce a statepoint.

Diff Detail

Event Timeline

apilipenko created this revision.Oct 5 2020, 4:39 PM
Herald added a project: Restricted Project. · View Herald Transcript
anna added a subscriber: anna.Oct 6 2020, 8:49 AM

Hi Artur, just to explicitly state out:

But element atomic memcpy/memmove intrinsic calls might be generated by the optimizer, which is not aware of this coupling. If statepoints without deopt into are not allowed and we see a non-leaf memcpy/memmove without deopt state we treat it as a leaf copy and don't produce a statepoint

If a long running loop of loads and stores are converted to atomic memcpy (through loop idiom recognize for example) and we don't produce a statepoint - this will make the memcpy non GC parseable, which is erroneous. So, it is the responsibility of such runtimes to also introduce some mechanism to prevent long running loops from being converted to memcpys (such as having safepoint requests within the loop thereby preventing long running loops from being converted to memcpy: https://llvm.org/docs/Statepoints.html#id27).

Currently, if we have a loop with a safepoint poll it is not converted into a memcpy/memmove. This is because the safepoint has read semantic and prevents LoopIdiomRecognize from performing the transform. In theory we can have a transform which recognizes loops with safepoints and converts them to non-leaf memcpy/memmove. It will be up to this new transform to figure out the legality and interactions with the runtime requirements.

Also, note that a memcpy/memmove without "gc-leaf-function" attribute is not required to have a safepoint. It's lowered in a way which *may* have a safepoint. This is why it's correct to lower to a GC leaf representation and choose not to have a safepoint.

apilipenko updated this revision to Diff 296716.Oct 7 2020, 9:41 AM

Added a note into the doc that a GC parseable copy operation is not required to take a safepoint.

skatkov added inline comments.Oct 21 2020, 3:47 AM
llvm/docs/LangRef.rst
20257

See see?

20336

The same link as for memcpy?

llvm/docs/Statepoints.rst
836

"This makes it is possible"?

llvm/lib/Transforms/Utils/Local.cpp
2679

I'm a bit confused here. RS4GC consider memcpy/memove as gc leaf if there is no deopt bundle.
Here we do not check for deopt bundle. Is it ok?

Address review comments.

apilipenko added inline comments.Oct 21 2020, 11:29 AM
llvm/docs/LangRef.rst
20257

Fixed.

20336

Fixed.

llvm/docs/Statepoints.rst
836

Fixed.

apilipenko added inline comments.Oct 21 2020, 11:42 AM
llvm/lib/Transforms/Utils/Local.cpp
2679

callsGCLeafFunction returns true if the call is guaranteed to never safepoint. If a memcpy/memmove call is marked as gc-leaf it is guaranteed to be a leaf call, i.e. it will never take a safepoint. Otherwise the call *may* take a safepoint, but doesn't have to. This is reflected in the documentation:

Note that a GC parseable copy operation is not required to take a safepoint. For example, a short copy operation may be performed without taking a safepoint.

Interactions with deopt bundles is an implementation detail of RS4GC. For a memcpy/memmove call which may safepoint (i.e. doesn't have gc-leaf attribute) RS4GC is allowed to generate a leaf call which will never safepoint. If runtime requires deopt information to be associated with every safepoint (rs4gc-allow-statepoint-with-no-deopt-info=false) and RS4GC can not satisfy this requirement for a memcpy/memmove call it will generate a leaf call.

skatkov accepted this revision.Oct 21 2020, 9:12 PM

ok, this looks good to me. Please wait 1-2 days before landing to give a last call to others.

This revision is now accepted and ready to land.Oct 21 2020, 9:12 PM
This revision was automatically updated to reflect the committed changes.