[CodeGen] Respect libfunc availability when lowering intrinsic memcpy

[mstorsjo]

Summary

If the memcpy libfunc is marked as not being available (e.g. via the -fno-builtin flag), we shouldn't lower intrinsic memcpy into an actual memcpy function call.

The same should probably be done for other intrinsics (that can be spontaneously generated by e.g. clang without the functions being explicitly mentioned) that can end up lowered into a function call, but this is the one I've run into in practice.

Adding a test for this that runs on AArch64, as the patch requires separate changes to all of SelectionDAG, FastISel and GlobalISel.

[mstorsjo]

llvm/test/CodeGen/AArch64/intrinsic-memcpy-nobuiltin.ll

10

Ideally, I'd have the test also test without the no-builtins attribute, to check that the test still does what it's supposed to, i.e. that without the attribute, these still are lowered to an actual memcpy call.

[aemerson]

If we can’t lower it into a libcall, what *can* we do with it beyond optimizing into an unrolled in-line sequence? Just generate a whole memcpy implementation inline?

[mstorsjo]

In D86025#2220435, @aemerson wrote:

If we can’t lower it into a libcall, what *can* we do with it beyond optimizing into an unrolled in-line sequence? Just generate a whole memcpy implementation inline?

I guess those are the options, yeah.

For the case I'm looking at, code with an explicit struct assignment without the code calling memcpy or anything equivalent, and building it with -fno-builtin, I think the expectation is to just expand it into an inline memcpy, however long that sequence might be.

[efriedma]

Conventionally, we've assumed that memcpy is available despite -fno-builtin etc., simply because generating it inline is too painful. For example, consider what happens to your testcase if +strict-align is specified. (This is following gcc, which has a similar expectation.)

If the requirement of having a symbol named "memcpy" is problematic, I guess we could look into making compiler-rt provide a symbol with equivalent semantics, but use a compiler-reserved name so the user can't disable it with -fno-builtin. For example, __aeabi_memcpy is defined on ARM.

[aemerson]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6438

So if memcpy is not available but we have a non-constant size, then we still generate a memcpy call here?

[arsenm]

The memcpy intrinsic has nothing to do with library availability, and handling it should not depend on the library function. The backend may choose to emit a libcall for it, but this is not required (e.g. AMDGPU has no libcalls whatsoever). GlobalISel should be able to handle arbitrary memcpys with ease since it's possible to emit a loop. With SelectionDAG you have to expand the arbitrary case in the IR, but it still handles constant size cases.

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

1787

This needs to be dropped. D85199 switches these to using proper legalizeable instructions. It may make more sense for the libcall action to fail if the libraryinfo does not have memcpy

[mstorsjo]

In D86025#2222132, @efriedma wrote:

Conventionally, we've assumed that memcpy is available despite -fno-builtin etc., simply because generating it inline is too painful. For example, consider what happens to your testcase if +strict-align is specified. (This is following gcc, which has a similar expectation.)

Right... The actual case I'm trying to fix is on x86_64, and there gcc doesn't seem to generate a memcpy lightly, but for aarch64 it does indeed seem to generate such a call despite -fno-builtin, just as you say. (The reason for having the testcase for aarch64 is that the fix needed separate cases for selectiondag/fastisel/globalisel, so testing it on aarch64 felt easiest.)

If the requirement of having a symbol named "memcpy" is problematic, I guess we could look into making compiler-rt provide a symbol with equivalent semantics, but use a compiler-reserved name so the user can't disable it with -fno-builtin. For example, __aeabi_memcpy is defined on ARM.

Actually, the issue isn't that memcpy is missing, but that it has a different calling convention. In wine, certain DLLs (which are built as a regular ELF .so or MachO .dylib) would only include windows mode headers, and use a memcpy function with windows calling conventions. As long as memcpy is called explicitly, it ends up fine, but for the cases when the compiler backend invents memcpy calls, they end up using the platform's default calling convention.

This is mostly an issue on x86_64, where the calling conventions differ significantly - on other architectures the calling conventions are similar enough that there's no difference in a memcpy call.

In any case, a separate builtin function to generate a call to would probably help here as well, but doing that would break cases when building with clang but linking libgcc, no?

[mstorsjo]

In D86025#2222338, @arsenm wrote:

The memcpy intrinsic has nothing to do with library availability, and handling it should not depend on the library function. The backend may choose to emit a libcall for it, but this is not required (e.g. AMDGPU has no libcalls whatsoever).

Yeah, that's kind of the behaviour I expected when building with -fno-builtin. But then again, as @efriedma pointed out, GCC also does generate similar memcpy libcalls despite -fno-builtin (it didn't in the cases I tested before, on x86_64, but I now checked more and found cases where it does as well), so I'm a bit more hesitant about actually proceeding with this now...

GlobalISel should be able to handle arbitrary memcpys with ease since it's possible to emit a loop. With SelectionDAG you have to expand the arbitrary case in the IR, but it still handles constant size cases.

Ok, so where would I start out doing that, more concretely? (I have close to no experience in either transforms that change IR or GlobalISel.)

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

1787

The reason for placing it here, was that if we within translateKnownIntrinsic decide we don't want to handle it that way, and return false from that method, we just proceed further down here, instead of making this method return false (prompting the caller to handle it differently).

But it does indeed break cases where the target would have a different, non-libcall based lowering of the intrinsic...

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6438

Yeah, that's what the current patch does - that's clearly not ideal either...

[arsenm]

In D86025#2222453, @mstorsjo wrote:

GlobalISel should be able to handle arbitrary memcpys with ease since it's possible to emit a loop. With SelectionDAG you have to expand the arbitrary case in the IR, but it still handles constant size cases.

Ok, so where would I start out doing that, more concretely? (I have close to no experience in either transforms that change IR or GlobalISel.)

The actual expansion code (at least in the non-loop case) is already implemented, it's just in the wrong place. Currently the memory intrinsics can be expanded by the code in CombinerHelper::optimizeMemcpy/optimizeMemmove/optimizeMemset, and treat this as an optimization. I think this should be treated as a legalization decision, and moved into LegalizerHelper. LegalizerHelper::lower would then dispatch to a lowerMemcpy(), which would look similar to optimizeMemcpy today. After that, more code would be needed for the loop cases. I'll probably get to this in a month or two, but if you want to take care of it that would be great.

I would then probably add a utility to LegalizeRuleSet to help targets control this. Something like lowerIfLibcallUnavailable(TLI, LibFunc)

[aemerson]

In D86025#2222500, @arsenm wrote:

In D86025#2222453, @mstorsjo wrote:

GlobalISel should be able to handle arbitrary memcpys with ease since it's possible to emit a loop. With SelectionDAG you have to expand the arbitrary case in the IR, but it still handles constant size cases.

Ok, so where would I start out doing that, more concretely? (I have close to no experience in either transforms that change IR or GlobalISel.)

The actual expansion code (at least in the non-loop case) is already implemented, it's just in the wrong place. Currently the memory intrinsics can be expanded by the code in CombinerHelper::optimizeMemcpy/optimizeMemmove/optimizeMemset, and treat this as an optimization. I think this should be treated as a legalization decision, and moved into LegalizerHelper. LegalizerHelper::lower would then dispatch to a lowerMemcpy(), which would look similar to optimizeMemcpy today. After that, more code would be needed for the loop cases. I'll probably get to this in a month or two, but if you want to take care of it that would be great.

We can discuss this in a separate GISel thread, but I don't see how memcpy inlining is any more of an instruction legality decision than an optimization one. I'm not strongly against it, but IMO it's no more correct than a combine.

I would then probably add a utility to LegalizeRuleSet to help targets control this. Something like lowerIfLibcallUnavailable(TLI, LibFunc)

[arsenm]

In D86025#2222691, @aemerson wrote:

In D86025#2222500, @arsenm wrote:

In D86025#2222453, @mstorsjo wrote:

GlobalISel should be able to handle arbitrary memcpys with ease since it's possible to emit a loop. With SelectionDAG you have to expand the arbitrary case in the IR, but it still handles constant size cases.

Ok, so where would I start out doing that, more concretely? (I have close to no experience in either transforms that change IR or GlobalISel.)

The actual expansion code (at least in the non-loop case) is already implemented, it's just in the wrong place. Currently the memory intrinsics can be expanded by the code in CombinerHelper::optimizeMemcpy/optimizeMemmove/optimizeMemset, and treat this as an optimization. I think this should be treated as a legalization decision, and moved into LegalizerHelper. LegalizerHelper::lower would then dispatch to a lowerMemcpy(), which would look similar to optimizeMemcpy today. After that, more code would be needed for the loop cases. I'll probably get to this in a month or two, but if you want to take care of it that would be great.

We can discuss this in a separate GISel thread, but I don't see how memcpy inlining is any more of an instruction legality decision than an optimization one. I'm not strongly against it, but IMO it's no more correct than a combine.

Because whether or not emitting a library call is a valid strategy depends on the target. An inlined memcpy may not be optional, in which case it is not an optimization. The legalizer code would essentially be identical to the combiner code. The combiner could also do the same thing, but these should share code.

[efriedma]

In D86025#2222371, @mstorsjo wrote:

In D86025#2222132, @efriedma wrote:

Conventionally, we've assumed that memcpy is available despite -fno-builtin etc., simply because generating it inline is too painful. For example, consider what happens to your testcase if +strict-align is specified. (This is following gcc, which has a similar expectation.)

Right... The actual case I'm trying to fix is on x86_64, and there gcc doesn't seem to generate a memcpy lightly, but for aarch64 it does indeed seem to generate such a call despite -fno-builtin, just as you say. (The reason for having the testcase for aarch64 is that the fix needed separate cases for selectiondag/fastisel/globalisel, so testing it on aarch64 felt easiest.)

gcc will generate memcpy on x86 too; the threshold is just ridiculously high for -mtune=generic. (I don't think this choice is affected by -fno-builtin.)

I really don't really want to go down this path of inlining all memcpys for CPU targets: the code for an efficient memcpy is way too big to reasonably inline, particularly on non-x86 targets.

Note that LLVM optimizations will currently avoid forming memcpy under -fno-builtin (mostly to avoid weird issues with implementing memcpy). This doesn't affect clang.

If the requirement of having a symbol named "memcpy" is problematic, I guess we could look into making compiler-rt provide a symbol with equivalent semantics, but use a compiler-reserved name so the user can't disable it with -fno-builtin. For example, __aeabi_memcpy is defined on ARM.

Actually, the issue isn't that memcpy is missing, but that it has a different calling convention. In wine, certain DLLs (which are built as a regular ELF .so or MachO .dylib) would only include windows mode headers, and use a memcpy function with windows calling conventions. As long as memcpy is called explicitly, it ends up fine, but for the cases when the compiler backend invents memcpy calls, they end up using the platform's default calling convention.

This is mostly an issue on x86_64, where the calling conventions differ significantly - on other architectures the calling conventions are similar enough that there's no difference in a memcpy call.

You mean, the header explicitly declares a function named memcpy, and uses an attribute to explicitly overrides the calling convention so it's not the same as the "C" calling convention? Messing with the backend's calling convention for memcpy isn't hard, but I'm not sure what conditions would be appropriate. We already support clang --target=x86_64-pc-win32-elf.

In any case, a separate builtin function to generate a call to would probably help here as well, but doing that would break cases when building with clang but linking libgcc, no?

Yes.

[mstorsjo]

In D86025#2222500, @arsenm wrote:

The actual expansion code (at least in the non-loop case) is already implemented, it's just in the wrong place. Currently the memory intrinsics can be expanded by the code in CombinerHelper::optimizeMemcpy/optimizeMemmove/optimizeMemset, and treat this as an optimization. I think this should be treated as a legalization decision, and moved into LegalizerHelper. LegalizerHelper::lower would then dispatch to a lowerMemcpy(), which would look similar to optimizeMemcpy today. After that, more code would be needed for the loop cases. I'll probably get to this in a month or two, but if you want to take care of it that would be great.

Thanks for the pointers! I think I might hold off of this for now though - since the target where it's needed is x86_64, I'd primarily only need it in SelectionDAG, but I tried to fix it consistently in all cases at once.

In D86025#2222728, @efriedma wrote:

gcc will generate memcpy on x86 too; the threshold is just ridiculously high for -mtune=generic. (I don't think this choice is affected by -fno-builtin.)

That does indeed seem to be the case. For -mtune=generic, it seems to generate something that boils down to rep movsq - and you're right that -fno-builtin doesn't seem to affect it.

I really don't really want to go down this path of inlining all memcpys for CPU targets: the code for an efficient memcpy is way too big to reasonably inline, particularly on non-x86 targets.

Do you mean there's a lot of code, built with -fno-builtin, where the code itself didn't mention memcpy, but where a call to it ideally should be generated? (Not questioning the assumption, just trying to understand the concern better.)

Note that LLVM optimizations will currently avoid forming memcpy under -fno-builtin (mostly to avoid weird issues with implementing memcpy). This doesn't affect clang.

Hmm, which cases do you refer to here?

If the requirement of having a symbol named "memcpy" is problematic, I guess we could look into making compiler-rt provide a symbol with equivalent semantics, but use a compiler-reserved name so the user can't disable it with -fno-builtin. For example, __aeabi_memcpy is defined on ARM.

Actually, the issue isn't that memcpy is missing, but that it has a different calling convention. In wine, certain DLLs (which are built as a regular ELF .so or MachO .dylib) would only include windows mode headers, and use a memcpy function with windows calling conventions. As long as memcpy is called explicitly, it ends up fine, but for the cases when the compiler backend invents memcpy calls, they end up using the platform's default calling convention.

This is mostly an issue on x86_64, where the calling conventions differ significantly - on other architectures the calling conventions are similar enough that there's no difference in a memcpy call.

You mean, the header explicitly declares a function named memcpy, and uses an attribute to explicitly overrides the calling convention so it's not the same as the "C" calling convention?

Exactly. I guess that's maybe technically invalid C, but still a scenario that there's a demand to handle in some way.

Messing with the backend's calling convention for memcpy isn't hard, but I'm not sure what conditions would be appropriate. We already support clang --target=x86_64-pc-win32-elf.

I fear that tucking the decision away behind a target triple can be a bit opaque. This is also a transition in progress - 12 months ago, the memcpy called within Wine builtin modules was the host libc's, but they're transitioning towards making the wine builtin modules actual freestanding DLLs. They can still either be built as ELF/MachO, or as a real PE DLL, but they don't interact with the host environment, only other DLLs, and the memcpy used within them is one with the windows calling convention regardless of the object file format.

[efriedma]

I really don't really want to go down this path of inlining all memcpys for CPU targets: the code for an efficient memcpy is way too big to reasonably inline, particularly on non-x86 targets.

Do you mean there's a lot of code, built with -fno-builtin, where the code itself didn't mention memcpy, but where a call to it ideally should be generated? (Not questioning the assumption, just trying to understand the concern better.)

Yes. There's a lot of code built with something like -ffreestanding, that's compatible with the current understanding of the rules for when the compiler can generate memcpy.

Note that LLVM optimizations will currently avoid forming memcpy under -fno-builtin (mostly to avoid weird issues with implementing memcpy). This doesn't affect clang.

Hmm, which cases do you refer to here?

Loop idiom recognition, things like that. Basically, if you write something like void *memcpy(void *dest, const void *src, size_t n) { while (n--) *dest++ = *src++; }, we don't want to turn that into a recursive memcpy call.

If the requirement of having a symbol named "memcpy" is problematic, I guess we could look into making compiler-rt provide a symbol with equivalent semantics, but use a compiler-reserved name so the user can't disable it with -fno-builtin. For example, __aeabi_memcpy is defined on ARM.

Actually, the issue isn't that memcpy is missing, but that it has a different calling convention. In wine, certain DLLs (which are built as a regular ELF .so or MachO .dylib) would only include windows mode headers, and use a memcpy function with windows calling conventions. As long as memcpy is called explicitly, it ends up fine, but for the cases when the compiler backend invents memcpy calls, they end up using the platform's default calling convention.

This is mostly an issue on x86_64, where the calling conventions differ significantly - on other architectures the calling conventions are similar enough that there's no difference in a memcpy call.

You mean, the header explicitly declares a function named memcpy, and uses an attribute to explicitly overrides the calling convention so it's not the same as the "C" calling convention?

Exactly. I guess that's maybe technically invalid C, but still a scenario that there's a demand to handle in some way.

Messing with the backend's calling convention for memcpy isn't hard, but I'm not sure what conditions would be appropriate. We already support clang --target=x86_64-pc-win32-elf.

I fear that tucking the decision away behind a target triple can be a bit opaque. This is also a transition in progress - 12 months ago, the memcpy called within Wine builtin modules was the host libc's, but they're transitioning towards making the wine builtin modules actual freestanding DLLs. They can still either be built as ELF/MachO, or as a real PE DLL, but they don't interact with the host environment, only other DLLs, and the memcpy used within them is one with the windows calling convention regardless of the object file format.

The point here would just be that if you use x86_64-pc-win32-elf, we generate code as if the target were a Windows target; this might be useful if you want to generate an ELF library that globally uses the Windows calling convention. It might not be right if you only wanted to mess with the calling convention of memcpy.

llvm/include/llvm/CodeGen/GlobalISel/IRTranslator.h

private:
   /// Current target configuration. Controls how the pass handles errors.
   const TargetPassConfig *TPC;
 
   /// Current optimization remark emitter. Used to report failures.
   std::unique_ptr<OptimizationRemarkEmitter> ORE;
 
   FunctionLoweringInfo FuncInfo;
 
+  const TargetLibraryInfo *LibInfo;
+
   // True when either the Target Machine specifies no optimizations or the
   // function has the optnone attribute.
   bool EnableOpts = false;
 
   /// True when the block contains a tail call. This allows the IRTranslator to
   /// stop translating such blocks early.
   bool HasTailCall = false;
 

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

 #include "llvm/ADT/PostOrderIterator.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/Loads.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/GlobalISel/CallLowering.h"
 #include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
 #include "llvm/CodeGen/GlobalISel/InlineAsmLowering.h"
 #include "llvm/CodeGen/LowLevelType.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"

 static cl::opt<bool>
     EnableCSEInIRTranslator("enable-cse-in-irtranslator",
                             cl::desc("Should enable CSE in irtranslator"),
                             cl::Optional, cl::init(false));
 char IRTranslator::ID = 0;
 
 INITIALIZE_PASS_BEGIN(IRTranslator, DEBUG_TYPE, "IRTranslator LLVM IR -> MI",
                 false, false)
+INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_DEPENDENCY(GISelCSEAnalysisWrapperPass)
 INITIALIZE_PASS_END(IRTranslator, DEBUG_TYPE, "IRTranslator LLVM IR -> MI",
                 false, false)
 
 static void reportTranslationError(MachineFunction &MF,
                                    const TargetPassConfig &TPC,
                                    OptimizationRemarkEmitter &ORE,
#endif
   }
 };
 } // namespace
 #endif // ifndef NDEBUG
 
 
 void IRTranslator::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<StackProtector>();
+  AU.addRequired<TargetLibraryInfoWrapperPass>();
   AU.addRequired<TargetPassConfig>();
   AU.addRequired<GISelCSEAnalysisWrapperPass>();
   getSelectionDAGFallbackAnalysisUsage(AU);
   MachineFunctionPass::getAnalysisUsage(AU);
 }
 
 IRTranslator::ValueToVRegInfo::VRegListT &
 IRTranslator::allocateVRegs(const Value &Val) {
if (TII && ID == Intrinsic::not_intrinsic)
       ID = static_cast<Intrinsic::ID>(TII->getIntrinsicID(F));
   }
 
   if (!F || !F->isIntrinsic() || ID == Intrinsic::not_intrinsic)
     return translateCallBase(CI, MIRBuilder);
 
   assert(ID != Intrinsic::not_intrinsic && "unknown intrinsic");
 
+  if (ID == Intrinsic::memcpy && !LibInfo->has(LibFunc_memcpy))
+    return false;
   if (translateKnownIntrinsic(CI, ID, MIRBuilder))
     return true;
 
   ArrayRef<Register> ResultRegs;
   if (!CI.getType()->isVoidTy())
     ResultRegs = getOrCreateVRegs(CI);
 
   // Ignore the callsite attributes. Backend code is most likely not expecting
bool IRTranslator::runOnMachineFunction(MachineFunction &CurMF) {
   ORE = std::make_unique<OptimizationRemarkEmitter>(&F);
   FuncInfo.MF = MF;
   FuncInfo.BPI = nullptr;
   const auto &TLI = *MF->getSubtarget().getTargetLowering();
   const TargetMachine &TM = MF->getTarget();
   SL = std::make_unique<GISelSwitchLowering>(this, FuncInfo);
   SL->init(TLI, TM, *DL);
 
+  LibInfo = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
+
   EnableOpts = TM.getOptLevel() != CodeGenOpt::None && !skipFunction(F);
 
   assert(PendingPHIs.empty() && "stale PHIs");
 
   if (!DL->isLittleEndian()) {
     // Currently we don't properly handle big endian code.
     OptimizationRemarkMissed R("gisel-irtranslator", "GISelFailure",
                                F.getSubprogram(), &F.getEntryBlock());

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

   case Intrinsic::experimental_patchpoint_void:
   case Intrinsic::experimental_patchpoint_i64:
     return selectPatchpoint(II);
 
   case Intrinsic::xray_customevent:
     return selectXRayCustomEvent(II);
   case Intrinsic::xray_typedevent:
     return selectXRayTypedEvent(II);
+  case Intrinsic::memcpy:
+    if (!LibInfo->has(LibFunc_memcpy))
+      return false;
+    break;
   }
 
   return fastLowerIntrinsicCall(II);
 }
 
 bool FastISel::selectCast(const User *I, unsigned Opcode) {
   EVT SrcVT = TLI.getValueType(DL, I->getOperand(0)->getType());
   EVT DstVT = TLI.getValueType(DL, I->getType());

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/MemoryLocation.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"

         *this, dl, Chain, Dst, Src, Size, Alignment, isVol, AlwaysInline,
         DstPtrInfo, SrcPtrInfo);
     if (Result.getNode())
       return Result;
   }
 
   // If we really need inline code and the target declined to provide it,
   // use a (potentially long) sequence of loads and stores.
-  if (AlwaysInline) {
+  if (AlwaysInline || (!LibInfo->has(LibFunc_memcpy) && ConstantSize)) {
     assert(ConstantSize && "AlwaysInline requires a constant size!");
     return getMemcpyLoadsAndStores(*this, dl, Chain, Dst, Src,
                                    ConstantSize->getZExtValue(), Alignment,
                                    isVol, true, DstPtrInfo, SrcPtrInfo);
   }
 
   checkAddrSpaceIsValidForLibcall(TLI, DstPtrInfo.getAddrSpace());
   checkAddrSpaceIsValidForLibcall(TLI, SrcPtrInfo.getAddrSpace());

llvm/test/CodeGen/AArch64/intrinsic-memcpy-nobuiltin.ll

+; RUN: llc -mtriple=aarch64-linux-gnu < %s -o - | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -O0 -global-isel < %s -o - | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -O0 -fast-isel   < %s -o - | FileCheck %s
+
+%struct.s = type { [512 x i8] }
+
+define void @copy(%struct.s* readonly %src, %struct.s* %dst) #0 {
+entry:
+; CHECK-LABEL: copy
+; CHECK-NOT: memcpy
+  %0 = getelementptr %struct.s, %struct.s* %dst, i64 0, i32 0, i64 0
+  %1 = getelementptr %struct.s, %struct.s* %src, i64 0, i32 0, i64 0
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 dereferenceable(512) %0, i8* nonnull align 1 dereferenceable(512) %1, i64 512, i1 false)
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg)
+
+attributes #0 = { "no-builtins" }