This structural constraint (gc.result being tied to a statepoint instruction) can be violated in dead code. We've recently fixed a similar issue for gc.relocates in D128904.

LGTM with a small comment to address.

Is it true that every block which has successors will necessarily fall into this infinite loop?

No. What is special about this test is the fact that bb1 and bb2 have a branch on the same condition (undef).

So, you are crippling this transformation to avoid an infinite loop with another transform in SimplifyCFG. This is probably OK as we still handle the common case of a widenable condition going to a deoptimization block.

Could you please give a brief outline of the approach?

In other instructions, we seem to just duplicate the handling in findBaseDefiningValueOfVector and findBaseDefiningValue. Doing so for freeze would be more consistent with the existing code.

The optimizer can insert new atomic memcpy/memmove calls, e.g loop-idiom-recognize would do that. Currently, when loop-idiom-recognize inserts such a call it doesn't tag it with the element type, which makes it an incorrect transform for us.

Copying arrays of pointers in environments with moving garbage collector is different from copying of 'primitive' types of the same types.
Well, I understand that upstream may not take it as a 'justification'.

Do you mind if I put this change under an on by default cl::opt flag? This way we can temporarily turn it off downstream.

This change broke the ability to discard checks in some cases. Consider this example:

define i1 @test1(i32 %a, i32* %b_ptr) {
  %b = load i32, i32* %b_ptr, align 8, !range !{i32 0, i32 2147483646}
  %c1 = icmp slt i32 %a, %b
  br i1 %c1, label %exit, label %cont.1

I agree with Philip, widening transformation doesn't rely on deoptimizations.

Add a simple test case using -inline-cost-full cl::opt.

I also notice that the new memcmp intrinsic returns i1, which is different from libc memcmp. libc memcmp returns and int indicating which of the sides was greater. Looks like, the new intrinsic matches with bcmp semantic.

Since this is a change to LangRef, please, post the proposal to llvm-dev.

There is a target-specific hook to emit code for regular (non-atomic) memcpys: EmitTargetCodeForMemcpy. Maybe we should just implement a similar hook for element-atomic copy?

I think the name of the pass is misleading. What you are doing here is you provide an inlined lowering for a memory builtin. Your current implementation has a limitation such that it only inlines gc-leaf element atomic memcpys. I don't think the fact that it is gc-leaf is critical here. This transform can be extended to handle non-GC leaf operations. It can also be extended to handle non-atomic operations.

Looks good.

FYI, I tried to take this patch for performance verification, but it crashed in smoke testing. I haven't yet looked why.

I looked at the GC part and it looks good to me.

Can we split restructuring and generalization (possibly adding a test for the generalization part)? It's hard to assess correctness when you combine the two in one change.

Accepting this change to unblock the progress.

Address review comments.

Still digging through the main logic. In the meantime see some minor comments inline.

Added a note into the doc that a GC parseable copy operation is not required to take a safepoint.

Currently, if we have a loop with a safepoint poll it is not converted into a memcpy/memmove. This is because the safepoint has read semantic and prevents LoopIdiomRecognize from performing the transform. In theory we can have a transform which recognizes loops with safepoints and converts them to non-leaf memcpy/memmove. It will be up to this new transform to figure out the legality and interactions with the runtime requirements.

This was a draft. Abandoned in favor of D88861.

@mtrofin, I think it makes sense to have these tests available in all configurations. We have a bunch of printer passes which are not conditioned by NDEBUG.

Look good.

In D81026#2069624, @mtrofin wrote:

That sounds fine, my point is that having some run on benchmarks to quantify the benefit would help understand the value of this particular omission from the cost analysis (or at minimum show there's no regression, or understand where the regressions may be). It could shed important insights into what matters for the cost analysis, basically.

Can you please elaborate what is the issue you are solving? May be give an example?

Demonstrating an improvement accuracy due to PHI translation turned out to be a bit tricky.

Finished by @ebrevnov as D65276.

I reworked the patch using PHITransAddr. Now instead of tracking the backedge flag I keep track of the address to check and translate the address when needed.

Couple of nit comments inlined, LGTM once addressed.