Fix missing memcpy, memmove and memset tail calls

[sanwou01]

Summary

If a wrapper around one of the mem* stdlib functions bitcasts the returned
pointer value before returning it (e.g. to a wchar_t*), LLVM does not emit a
tail call.

Add a check for this scenario so that we emit a tail call.

[lebedev.ri]

Test?

[ramred01]

Updated the summary to properly reflect the issue and the solution.

Added a test case.

[wmi]

LGTM.

[sanwou01]

Hi, I'd like to take over this change to get it committed. There is a crash lurking when llvm::returnTypeIsEligibleForTailCall is called with a function that takes no arguments, so I'd like to make some changes before committing. Thanks.

[sanwou01]

Fix crash and refactor for readability

[dmgreen]

Yeah. LGTM. Thanks for carrying this forward!

llvm/lib/CodeGen/Analysis.cpp

617 ↗

(On Diff #227067)

I don't think these two asserts are required if you wanted to leave them out. It's pretty much implied from the signature that the two values will not be null.

If you do want to keep them in, run clang format and add a message to the asserts. Something like:
assert(a && b && "Expected non-null inputs!");

lib/CodeGen/Analysis.cpp

bool llvm::returnTypeIsEligibleForTailCall(const Function *F,
 
   const Value *RetVal = Ret->getOperand(0), *CallVal = I;
   // Intrinsic like llvm.memcpy has no return value, but the expanded
   // libcall may or may not have return value. On most platforms, it
   // will be expanded as memcpy in libc, which returns the first
   // argument. On other platforms like arm-none-eabi, memcpy may be
   // expanded as library call without return value, like __aeabi_memcpy.
   const CallInst *Call = cast<CallInst>(I);
+  auto *BitCastIn = dyn_cast<BitCastInst>(Call->getArgOperand(0));
   if (Function *F = Call->getCalledFunction()) {
     Intrinsic::ID IID = F->getIntrinsicID();
     if (((IID == Intrinsic::memcpy &&
           TLI.getLibcallName(RTLIB::MEMCPY) == StringRef("memcpy")) ||
          (IID == Intrinsic::memmove &&
           TLI.getLibcallName(RTLIB::MEMMOVE) == StringRef("memmove")) ||
          (IID == Intrinsic::memset &&
           TLI.getLibcallName(RTLIB::MEMSET) == StringRef("memset"))) &&
-        RetVal == Call->getArgOperand(0))
+        (RetVal == Call->getArgOperand(0) || 
+         (BitCastIn && RetVal == BitCastIn->getOperand(0) && 
+          Call->getArgOperand(0)->getType()->isPointerTy() &&
+          BitCastIn->getOperand(0)->getType()->isPointerTy())))
       return true;
   }
 
   SmallVector<unsigned, 4> RetPath, CallPath;
   SmallVector<CompositeType *, 4> RetSubTypes, CallSubTypes;
 
   bool RetEmpty = !firstRealType(RetVal->getType(), RetSubTypes, RetPath);
   bool CallEmpty = !firstRealType(CallVal->getType(), CallSubTypes, CallPath);

test/CodeGen/AArch64/memcpy.ll

+;RUN: llc %s -o - -verify-machineinstrs | FileCheck %s
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-arm-none-eabi"
+
+; Function Attrs: minsize nounwind optsize
+;CHECK-LABEL: @wmemcpy
+;CHECK: lsl
+;CHECK-NOT: bl
+;CHECK-NOT: mov
+;CHECK-NOT: ldp
+;CHECK-NEXT: b memcpy
+define dso_local i32* @wmemcpy(i32* returned, i32* nocapture readonly, i64) local_unnamed_addr #0 {
+  %4 = bitcast i32* %0 to i8*
+  %5 = bitcast i32* %1 to i8*
+  %6 = shl i64 %2, 2
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 4 %4, i8* align 4 %5, i64 %6, i1 false)
+  ret i32* %0
+}
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1) #1
+
+attributes #0 = { minsize nounwind optsize "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { argmemonly nounwind }
+
+!llvm.module.flags = !{!0}
+!llvm.ident = !{!1}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{!"clang version 9.0.0 (https://git.llvm.org/git/clang.git/ b8120e9866e32a7748b963f3f5f7759351b47f51) (https://git.llvm.org/git/llvm.git/ 18fe2c5936a77484261539c570fecd6be33c476c)"}