[MemCpyOpt] Use memcpy source directly if dest is known to be immutable from attributes

[khei4]

Summary

This patch aims to remove memcpy on immutable arguments from attributes.

We can safely remove src-to-dest memcpy dest is used as only the call parameter if following conditions are satisfied

This is called on memcpy dest pointer arguments attributed as immutable
during call. Try to use memcpy source directly if all of the following
conditions are satisfied.

1. The memcpy dst is neither modified during the call nor captured by the call. (if readonly, noalias, nocapture attributes on call-site.)
2. The memcpy dst is an alloca with known alignment & size.
   1. The memcpy length == the alloca size which ensures that the new pointer is dereferenceable for the required range
   2. The src pointer has alignment >= the alloca alignment or can be enforced so.
3. The memcpy dst and src is not modified between the memcpy and the call. (if MSSA clobber check is safe.)
4. The memcpy src is not modified during the call. (ModRef check shows no Mod.)

(from https://reviews.llvm.org/D150967#4359222 and https://reviews.llvm.org/D150967#4365466 )

precommit tests: https://reviews.llvm.org/D150967
Motivated to resolve https://github.com/rust-lang/rust/issues/107436

[khei4]

update (currently attribute checking not working correctly

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1746

This whole can can be replaced with CB->onlyReadsMemory(i). This will check readonly and readnone and will also correctly handle inheritance of attributes from the function.

1751

CB->paramHasAttr(i, Kind) is enough, it will handle attribute inheritance.

[khei4]

apply feedbacks, remove (most of) debug streams.

[khei4]

@nikic Thank you for the review! They saved me!

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1751

Thank you for saving me! I haven't found the attribute inheritance behavior and am confused about that!

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1733

This check looks unnecessary.

1734–1739

This seems simpler than the loop :)

[khei4]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1733

Thanks! That's remaining for the past diff!

1734–1739

Yeah, agreed, conditions are simpler than before.

[khei4]

attach Differential Revision on comment

[khei4]

process alignment, handle unescaped alloca and arg with noalias, clobber check on Dest of MDep(MemCpy)

[khei4]

need to change test on https://reviews.llvm.org/D150967#4365466 and refine description

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1695–1696

dest check might be redundant because the first clobber check might already show dest is not modified between call and memcpy.

[khei4]

rebase

[khei4]

apply feedback on call with @nikic

1. remove noalias argument or unescaped alloca checks
2. add ModRefCheck for memcpy source

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1618

nit: Start comments with uppercase letter.

1631

nit: auto * when using dyn_cast.

1652

see -> check that

1654

ImmutArg->stripPointerCasts() is same as AI.

1657

Outdated comment?

1658

byval -> alloca?

1669

Is the explicit MaybeAlign() here needed?

1684

"not modified by the call" maybe?

1736

nit: Unnecessary braces.

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1686

MSSA->getMemoryAccess(&CB) is CallAccess.

1688

This second writtenBetween looks redundant to me. This is already ensured by getClobberingMemoryAccess().

1689

Hm, shouldn't this be checking the source? We already know the dest is not modified via readonly noalias.

1699

You can drop the bitcast code, not needed with opaque pointers.

[khei4]

apply feedbacks

[khei4]

@nikic
Thanks for the review! Too much obsolete comments!

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1627–1633

I decided to postpone to the next Revision to collect all possible pointers and check that all ptr are alloca. (it might be a little conservative.)

1684

"not modified by the call" maybe?

Literally right,(writtenBetween guarantees so). TBH, I want to ensure the memcpy source is not modified by another alias during the call.
That part may be done by isModSet I guess.

1686

Good catch!! The same goes for ByVal.

1689

Right! Thanks, I was confused with them.

1699

Thanks! I believe ByVal has it too! Including other comments I'll fix them on ByVal side also!

[khei4]

fix typo

[khei4]

rebase

[khei4]

apply feedbacks.

• fix styles

[khei4]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1669

That was just to adjust arg types to the off-the-shelf util functions! Although it was wrong and removed!

[khei4]

fix styles and comments and postpone detailed case

[nikic]

It looks like the alignment checks are inverted.

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1615

<= should be >= here?

1630

"Even if the arg can be based on multiple allocas" or so.

1638

&& !AllocaSize->isScalable() (and a test for this)

1653

"equals to" -> "equals the" or "is equal to the"

1664

Does that work? getZextValue() may assert for wide integers (though probably not really relevant here).

1667

smaller -> larger than or equal

1670

Then you don't need to check !MemDepAlign

1672–1674

1695

stray byval

llvm/test/Transforms/MemCpyOpt/memcpy.ll

486

This is not a VLA. You need to do something like alloca i8, i64 %n.

545–573

This case is fine, the problematic case is if the %val alignment is smaller.

[khei4]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1615

Right!

1638

&& !AllocaSize->isScalable()

Sorry, I'm not sure about a scalable vectors pointer could be transformed (maybe ||, not &&?). Do you mean scalable vectors TypeSize are available, but memcpy length cannot be scalable for it, so we couldn't handle it, right?

1664

Works fine! Thanks!

1670

Thank you!

llvm/test/Transforms/MemCpyOpt/memcpy.ll

545–573

Yeah, you seem right. Thank you for the comment!
I also found the alive2 has similar issues :)

https://alive2.llvm.org/ce/z/NZivTQ

[khei4]

apply feedback and inverse alignment vs memcpy length comparison.

also rebase for

• smaller alignment failure case
• address space failure case
• scalable vector failure case

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1630

the all -> all the

1638

Yes, exactly.

1655

Add test with volatile memcpy.

1673

Can drop the extra () here.

1696

byval -> immut

llvm/test/Transforms/MemCpyOpt/memcpy.ll

508

I'd just write alloca <vscale x 2 x i32> here. You're using a "homogeneous scalable aggregate" which also tests the right thing, but is a more advanced feature...

529

We also need a test where dst is modified between call and memcpy, unless I missed it.

572–573

This comment is outdated.

596

We're missing a test where the alignment can be enforced. For that you want to copy from another alloca with lower alignment, and the alignment will be increased by the transform.

[khei4]

rebase
remove redundancy,
fix wrong comments

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1638

Understood. Thanks!

llvm/test/Transforms/MemCpyOpt/memcpy.ll

508

Thanks! Yeah as you said in this case, unnecessary complexity!

596

Sorry, I'm not sure how this could be. I now found that MemDepAlign is just memcpy's arguments alignment attributes value, so basically enforcing from parameter attributes works, but I couldn't find the way to enforce lower alignment actually!
(I might too naive, but I tried for example,

@g = global ptr null, align 4

define void @immut_param_enforced_alignment(ptr align 16 noalias %val) {
  %val1 = alloca i8, align 16
  store ptr %val, ptr @g
  %p = load ptr, ptr @g
  call void @llvm.memcpy.p0.p0.i64(ptr align 16 %val1, ptr %p, i64 1, i1 false)
  call void @f(ptr nocapture noalias readonly %val1)
  ret void
}

this can't enforce the alignment.)

[nikic]

llvm/test/Transforms/MemCpyOpt/memcpy.ll

596

I believe you need something like this:

define void @test() {
  %val = alloca i8, align 1
  store i32 42, ptr %val
  %val1 = alloca i8, align 4
  call void @llvm.memcpy.p0.p0.i64(ptr %val1, ptr %val, i64 1, i1 false)
  call void @f_full_readonly(ptr %val1)
  ret void
}

Here the align 1 on the first alloca will be changed to align 4 by the transform.

[khei4]

rebase for enforced alignment

[khei4]

llvm/test/Transforms/MemCpyOpt/memcpy.ll

596

Thank you! Now I see DataLayout specify the alignment for stored types!

[nikic]

llvm/test/Transforms/MemCpyOpt/memcpy.ll

600

Hm, so I expected that for this test case the transform would happen and change to align 4 here. Do you know why it doesn't?

[nikic]

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

1685

I believe these MSSA->getMemoryAccess(MDep) calls can be replaced by Clobber.

llvm/test/Transforms/MemCpyOpt/memcpy.ll

600

Hm, this does work when I test the patch locally, so maybe just outdated test result?

[khei4]

llvm/test/Transforms/MemCpyOpt/memcpy.ll

600

Hm, this does work when I test the patch locally, so maybe just outdated test result?

Yeah, Sorry... I have forgotten to build before the test update. I believe this works fine.

[khei4]

update tests

[nikic]

LGTM

[khei4]

rebase

llvm/include/llvm/Transforms/Scalar/MemCpyOptimizer.h

bool performCallSlotOptzn(Instruction *cpyLoad, Instruction *cpyStore,
                             std::function<CallInst *()> GetC);
   bool processMemCpyMemCpyDependence(MemCpyInst *M, MemCpyInst *MDep,
                                      BatchAAResults &BAA);
   bool processMemSetMemCpyDependence(MemCpyInst *MemCpy, MemSetInst *MemSet,
                                      BatchAAResults &BAA);
   bool performMemCpyToMemSetOptzn(MemCpyInst *MemCpy, MemSetInst *MemSet,
                                   BatchAAResults &BAA);
   bool processByValArgument(CallBase &CB, unsigned ArgNo);
+  bool processImmutArgument(CallBase &CB, unsigned ArgNo);
   Instruction *tryMergingIntoMemset(Instruction *I, Value *StartPtr,
                                     Value *ByteVal);
   bool moveUp(StoreInst *SI, Instruction *P, const LoadInst *LI);
 
   void eraseInstruction(Instruction *I);
   bool iterateOnFunction(Function &F);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_SCALAR_MEMCPYOPTIMIZER_H

llvm/lib/Transforms/Scalar/MemCpyOptimizer.cpp

LLVM_DEBUG(dbgs() << "MemCpyOptPass: Forwarding memcpy to byval:\n"
                     << "  " << CB << "\n");
 
   // Otherwise we're good!  Update the byval argument.
   CB.setArgOperand(ArgNo, MDep->getSource());
   ++NumMemCpyInstr;
   return true;
 }
 
+/// This is called on memcpy dest pointer arguments attributed as immutable
+/// during call. Try to use memcpy source directly if all of the following
+/// conditions are satisfied.
+/// 1. The memcpy dst is neither modified during the call nor captured by the
+/// call. (if readonly, noalias, nocapture attributes on call-site.)
+/// 2. The memcpy dst is an alloca with known alignment & size.
+///     2-1. The memcpy length == the alloca size which ensures that the new
+///     pointer is dereferenceable for the required range
+///     2-2. The src pointer has alignment >= the alloca alignment or can be
+///     enforced so.
+/// 3. The memcpy dst and src is not modified between the memcpy and the call.
+/// (if MSSA clobber check is safe.)
+/// 4. The memcpy src is not modified during the call. (ModRef check shows no
+/// Mod.)
+bool MemCpyOptPass::processImmutArgument(CallBase &CB, unsigned ArgNo) {
+  // 1. Ensure passed argument is immutable during call.
+  if (!(CB.paramHasAttr(ArgNo, Attribute::NoAlias) &&
+        CB.paramHasAttr(ArgNo, Attribute::NoCapture)))
+    return false;
+  const DataLayout &DL = CB.getCaller()->getParent()->getDataLayout();
+  Value *ImmutArg = CB.getArgOperand(ArgNo);
+
+  // 2. Check that arg is alloca
+  // TODO: Even if the arg gets back to branches, we can remove memcpy if all
+  // the alloca alignments can be enforced to source alignment.
+  auto *AI = dyn_cast<AllocaInst>(ImmutArg->stripPointerCasts());
+  if (!AI)
+    return false;
+
+  std::optional<TypeSize> AllocaSize = AI->getAllocationSize(DL);
+  // Can't handle unknown size alloca.
+  // (e.g. Variable Length Array, Scalable Vector)
+  if (!AllocaSize || AllocaSize->isScalable())
+    return false;
+  MemoryLocation Loc(ImmutArg, LocationSize::precise(*AllocaSize));
+  MemoryUseOrDef *CallAccess = MSSA->getMemoryAccess(&CB);
+  if (!CallAccess)
+    return false;
+
+  MemCpyInst *MDep = nullptr;
+  BatchAAResults BAA(*AA);
+  MemoryAccess *Clobber = MSSA->getWalker()->getClobberingMemoryAccess(
+      CallAccess->getDefiningAccess(), Loc, BAA);
+  if (auto *MD = dyn_cast<MemoryDef>(Clobber))
+    MDep = dyn_cast_or_null<MemCpyInst>(MD->getMemoryInst());
+
+  // If the immut argument isn't fed by a memcpy, ignore it.  If it is fed by
+  // a memcpy, check that the arg equals the memcpy dest.
+  if (!MDep || MDep->isVolatile() || AI != MDep->getDest())
+    return false;
+
+  // The address space of the memcpy source must match the immut argument
+  if (MDep->getSource()->getType()->getPointerAddressSpace() !=
+      ImmutArg->getType()->getPointerAddressSpace())
+    return false;
+
+  // 2-1. The length of the memcpy must be equal to the size of the alloca.
+  auto *MDepLen = dyn_cast<ConstantInt>(MDep->getLength());
+  if (!MDepLen || AllocaSize != MDepLen->getValue())
+    return false;
+
+  // 2-2. the memcpy source align must be larger than or equal the alloca's
+  // align. If not so, we check to see if we can force the source of the memcpy
+  // to the alignment we need. If we fail, we bail out.
+  Align MemDepAlign = MDep->getSourceAlign().valueOrOne();
+  Align AllocaAlign = AI->getAlign();
+  if (MemDepAlign < AllocaAlign &&
+      getOrEnforceKnownAlignment(MDep->getSource(), AllocaAlign, DL, &CB, AC,
+                                 DT) < AllocaAlign)
+    return false;
+
+  // 3. Verify that the source doesn't change in between the memcpy and
+  // the call.
+  //    memcpy(a <- b)
+  //    *b = 42;
+  //    foo(*a)
+  // It would be invalid to transform the second memcpy into foo(*b).
+  if (writtenBetween(MSSA, BAA, MemoryLocation::getForSource(MDep),
+                     MSSA->getMemoryAccess(MDep), CallAccess))
+    return false;
+
+  // 4. The memcpy src must not be modified during the call.
+  if (isModSet(AA->getModRefInfo(&CB, MemoryLocation::getForSource(MDep))))
+    return false;
+
+  LLVM_DEBUG(dbgs() << "MemCpyOptPass: Forwarding memcpy to Immut src:\n"
+                    << "  " << *MDep << "\n"
+                    << "  " << CB << "\n");
+
+  // Otherwise we're good!  Update the immut argument.
+  CB.setArgOperand(ArgNo, MDep->getSource());
+  ++NumMemCpyInstr;
+  return true;
+}
+
 /// Executes one iteration of MemCpyOptPass.
 bool MemCpyOptPass::iterateOnFunction(Function &F) {
   bool MadeChange = false;
 
   // Walk all instruction in the function.
   for (BasicBlock &BB : F) {
     // Skip unreachable blocks. For example processStore assumes that an
     // instruction in a BB can't be dominated by a later instruction in the
for (BasicBlock::iterator BI = BB.begin(), BE = BB.end(); BI != BE;) {
         MadeChange |= processStore(SI, BI);
       else if (auto *M = dyn_cast<MemSetInst>(I))
         RepeatInstruction = processMemSet(M, BI);
       else if (auto *M = dyn_cast<MemCpyInst>(I))
         RepeatInstruction = processMemCpy(M, BI);
       else if (auto *M = dyn_cast<MemMoveInst>(I))
         RepeatInstruction = processMemMove(M);
       else if (auto *CB = dyn_cast<CallBase>(I)) {
-        for (unsigned i = 0, e = CB->arg_size(); i != e; ++i)
+        for (unsigned i = 0, e = CB->arg_size(); i != e; ++i) {
           if (CB->isByValArgument(i))
             MadeChange |= processByValArgument(*CB, i);
+          else if (CB->onlyReadsMemory(i))
+            MadeChange |= processImmutArgument(*CB, i);
+        }
       }
 
       // Reprocess the instruction if desired.
       if (RepeatInstruction) {
         if (BI != BB.begin())
           --BI;
         MadeChange = true;
       }
     }
   }
 
   return MadeChange;
 }
 
 PreservedAnalyses MemCpyOptPass::run(Function &F, FunctionAnalysisManager &AM) {
   auto &TLI = AM.getResult<TargetLibraryAnalysis>(F);
   auto *AA = &AM.getResult<AAManager>(F);
   auto *AC = &AM.getResult<AssumptionAnalysis>(F);
   auto *DT = &AM.getResult<DominatorTreeAnalysis>(F);
   auto *MSSA = &AM.getResult<MemorySSAAnalysis>(F);
 
   bool MadeChange = runImpl(F, &TLI, AA, AC, DT, &MSSA->getMSSA());
   if (!MadeChange)
     return PreservedAnalyses::all();

llvm/test/Transforms/MemCpyOpt/memcpy.ll

 }
 
 declare void @f1(ptr nocapture sret(%struct.big))
 declare void @f2(ptr)
 
 declare void @f(ptr)
 declare void @f_full_readonly(ptr nocapture noalias readonly)
 
-; TODO: Remove memcpy, which is guaranteed to be invariant
-; before and after the call because of its attributes.
 define void @immut_param(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param(
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
+; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL:%.*]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr align 4 nocapture noalias readonly %val1)
   ret void
 }
 

 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr align 4 nocapture noalias %val1)
   ret void
 }
 
-; TODO: Remove memcpy
 define void @immut_param_readonly(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_readonly(
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f_full_readonly(ptr align 4 [[VAL1]])
+; CHECK-NEXT:    call void @f_full_readonly(ptr align 4 [[VAL:%.*]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f_full_readonly(ptr align 4 %val1)
   ret void
 }
 
-; TODO: Remove memcpy
 define void @immut_param_no_align(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_no_align(
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL1]])
+; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL:%.*]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr nocapture noalias readonly %val1)
   ret void
 }
 
 @gp = external constant [0 x i8]
 ; Can't remove memcpy because dest is not unescaped alloca, so cpying is meaningfull.
 define void @immut_param_global(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_global(
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 @gp, ptr align 4 [[VAL:%.*]], i64 1, i1 false)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 @gp)
 ; CHECK-NEXT:    ret void
 ;
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 @gp, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr nocapture align 4 noalias readonly @gp)
   ret void
 }
 
 ; Can't remove memcpy for VLA because of unknown size and alignment.
 define void @immut_param_vla(ptr align 4 noalias %val, i64 %n) {
 ; CHECK-LABEL: @immut_param_vla(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca ptr, i64 [[N:%.*]], align 4
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 false)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca ptr, i64 %n
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy for scalable vector, because of memcpy size sufficiency is unknown
 define void @immut_param_scalable_vector(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_scalable_vector(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca <vscale x 2 x i32>, align 8
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 2, i1 false)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca <vscale x 2 x i32>
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 2, i1 false)
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy because dst is modified between call and memcpy
 define void @immut_param_modified_dst(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_modified_dst(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    store i32 13, ptr [[VAL1]], align 4
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   store i32 13, ptr %val1
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy because src is modified between call and memcpy
 define void @immut_param_modified_src(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_modified_src(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 false)
 ; CHECK-NEXT:    store i32 13, ptr [[VAL]], align 4
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   store i32 13, ptr %val
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy because memcpy is volatile
 define void @immut_param_volatile(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_volatile(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL:%.*]], i64 1, i1 true)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 true)
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy because address spaces are different.
 define void @immut_param_different_addrespace(ptr addrspace(1) align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_different_addrespace(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p1.i64(ptr align 4 [[VAL1]], ptr addrspace(1) align 4 [[VAL:%.*]], i64 1, i1 false)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly align 4 [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p1.i64(ptr align 4 %val1, ptr addrspace(1) align 4 %val, i64 1, i1 false)
   call void @f(ptr nocapture align 4 noalias readonly %val1)
   ret void
 }
 
-; TODO: remove memcpy
 define void @immut_param_bigger_align(ptr align 16 noalias %val) {
 ; CHECK-LABEL: @immut_param_bigger_align(
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr [[VAL:%.*]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL1]])
+; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL:%.*]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr %val, i64 1, i1 false)
   call void @f(ptr nocapture noalias readonly %val1)
   ret void
 }
 
 ; Can't remove memcpy if we remove, the bigger alignment couldn't replaced by smaller one.
 define void @immut_param_smaller_align(ptr align 4 noalias %val) {
 ; CHECK-LABEL: @immut_param_smaller_align(
 ; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 16
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 16 [[VAL1]], ptr [[VAL:%.*]], i64 1, i1 false)
 ; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL1]])
 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 16
   call void @llvm.memcpy.p0.p0.i64(ptr align 16 %val1, ptr %val, i64 1, i1 false)
   call void @f(ptr nocapture noalias readonly %val1)
   ret void
 }
 
-; TODO: remove memcpy.
 define void @immut_param_enforced_alignment() {
 ; CHECK-LABEL: @immut_param_enforced_alignment(
-; CHECK-NEXT:    [[VAL:%.*]] = alloca i8, align 1
+; CHECK-NEXT:    [[VAL:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    store i32 42, ptr [[VAL]], align 4
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr [[VAL1]], ptr [[VAL]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL1]])
+; CHECK-NEXT:    call void @f(ptr noalias nocapture readonly [[VAL]])
 ; CHECK-NEXT:    ret void
 ;
   %val = alloca i8, align 1
   store i32 42, ptr %val
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr %val1, ptr %val, i64 1, i1 false)
   call void @f(ptr nocapture noalias readonly %val1)
   ret void

 ; CHECK-NEXT:    ret void
 ;
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f(ptr nocapture noalias readonly %val1)
   ret void
 }
 
-; TODO: remove memcpy
 define void @immut_unescaped_alloca() {
 ; CHECK-LABEL: @immut_unescaped_alloca(
 ; CHECK-NEXT:    [[VAL:%.*]] = alloca i8, align 4
 ; CHECK-NEXT:    store i32 42, ptr [[VAL]], align 4
-; CHECK-NEXT:    [[VAL1:%.*]] = alloca i8, align 4
-; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[VAL1]], ptr align 4 [[VAL]], i64 1, i1 false)
-; CHECK-NEXT:    call void @f_full_readonly(ptr [[VAL1]])
+; CHECK-NEXT:    call void @f_full_readonly(ptr [[VAL]])
 ; CHECK-NEXT:    ret void
 ;
   %val = alloca i8, align 4
   store i32 42, ptr %val
   %val1 = alloca i8, align 4
   call void @llvm.memcpy.p0.p0.i64(ptr align 4 %val1, ptr align 4 %val, i64 1, i1 false)
   call void @f_full_readonly(ptr %val1)
   ret void