[X86] Lower calls with clang.arc.attachedcall bundle

[fhahn]

Summary

This patch adds support for lowering function calls with the
clang.arc.attachedcall bundle. The goal is to expand such calls to the
following sequence of instructions:

callq   @fn
movq  %rax, %rdi
callq   _objc_retainAutoreleasedReturnValue / _objc_unsafeClaimAutoreleasedReturnValue

This sequence of instructions triggers Objective-C runtime optimizations,
hence we want to ensure no instructions get moved in between them.
This patch achieves that by adding a new CALL_RVMARKER ISD node,
which gets turned into the CALL64_RVMARKER pseudo, which eventually gets
expanded into the sequence mentioned above.

The ObjC runtime function to call is determined by the
argument in the bundle, which is passed through as a
target constant to the pseudo.

@ahatanak is working on using this attribute in the front- & middle-end.

Together with the front- & middle-end changes, this should address
PR31925 for X86.

This is the X86 version of 46bc40e50246c1902a1ca7916c8286cb837643ee,
which added similar support for AArch64.

[fhahn]

I am not too familiar with how calls are handled in the X86 backend. I would really appreciate any pointers to cases that this approach might miss.

This is only relevant for Objective-C code on X86, so it would probably be OK to drop support for the CALL32* variants?

[fhahn]

After checking with @ahatanak, we need a slightly different approach on X86 compared to ARM64. I'll update the patch shortly.

[fhahn]

Updated to work with operand bundles as available upstream, emit marker ( movq %rdi, %rax) followed by call to either _objc_retainAutoreleasedReturnValue or _objc_unsafeClaimAutoreleasedReturnValue.

[fhahn]

Fix marker, remove 32bit support, as we will only enable it for x86-64 in the frontend initially.

[fhahn]

Fix issue with pcrel32 calls, properly update RAX reg state (if it is implicit-def dead on the original call, we need to change it to implicit-def, because we add a new use; also set the state on the generated call); add reg mask for new call.

[fhahn]

Update bundle to clang.arc.attachedcall, in line with committed bundle name.

[fhahn]

Rebase & ping :)

I also added a new test case to make sure implicit operands for call arguments are added correctly to the generated calls.

[ab]

Can you (or @ahatanak) explain why/how this needs to be different from aarch64? In particular I don't think I understand why you need to emit the retainARV/claimARV calls this late, that seems like quite a drastic difference in behavior for the bundle.
I read through the discussion in D92808 but it sounds like that predates the x86_64-specific behavior, right? But skimming around the various passes involved I'm not actually sure who emits the retainARV/claimARV calls on aarch64.

llvm/lib/Target/X86/X86ISelLowering.cpp

4435

clang.arc.rv -> clang.arc.attachedcall, right? same below

4443

Sel is a bit confusing, especially in an objc context ;)

[fhahn]

Thanks for taking a look!

In D94597#2632806, @ab wrote:

Can you (or @ahatanak) explain why/how this needs to be different from aarch64? In particular I don't think I understand why you need to emit the retainARV/claimARV calls this late, that seems like quite a drastic difference in behavior for the bundle.

Unfortunately the ObjC runtime matches different sequences on AArc64 and X86. This is documented here https://github.com/opensource-apple/objc4/blob/master/runtime/objc-object.h#L974

On AArch64, the callee only checks for mov x29, x29 in the caller and the position of the claim/autorelease runtime call does not matter, which makes things much simpler. On X86_64, it specifically looks for the marker instructions, immediately followed by the runtime call. So we want to make sure that it is very hard to break up this specific sequence, to avoid missing out on the runtime optimization.

I read through the discussion in D92808 but it sounds like that predates the x86_64-specific behavior, right? But skimming around the various passes involved I'm not actually sure who emits the retainARV/claimARV calls on aarch64.

On AArch64 I think that happens during the lowering of the ARC intrinsics pre-ISel, but @ahatanak would know more about the exact place.

[fhahn]

Update references to arc bundle name, change Sel to RuntimeCallType.

[ahatanak]

In D94597#2632887, @fhahn wrote:

In D94597#2632806, @ab wrote:

I read through the discussion in D92808 but it sounds like that predates the x86_64-specific behavior, right? But skimming around the various passes involved I'm not actually sure who emits the retainARV/claimARV calls on aarch64.

On AArch64 I think that happens during the lowering of the ARC intrinsics pre-ISel, but @ahatanak would know more about the exact place.

ARC contract pass inserts the retainRV/claimRV calls before transforming the function. After the transformation is done, the pass removes the inserted runtime calls on x86-64, but leaves the calls in the IR on AArch64.

On AArch64, objc_autoreleaseReturnValue, which the callee calls before returning, looks for only the marker instruction, so it's not necessary for the retainRV/claimRV instructions to immediately follow the function call and the marker instruction. On x86-64, it looks for a movq first and then a retainRV/claimRV call immediately following the movq.

See callerAcceptsOptimizedReturn in the code below:

https://opensource.apple.com/source/objc4/objc4-818.2/runtime/objc-object.h.auto.html

[fhahn]

ping :)

[ab]

In D94597#2632887, @fhahn wrote:

Unfortunately the ObjC runtime matches different sequences on AArc64 and X86. This is documented here https://github.com/opensource-apple/objc4/blob/master/runtime/objc-object.h#L974

On AArch64, the callee only checks for mov x29, x29 in the caller and the position of the claim/autorelease runtime call does not matter, which makes things much simpler. On X86_64, it specifically looks for the marker instructions, immediately followed by the runtime call. So we want to make sure that it is very hard to break up this specific sequence, to avoid missing out on the runtime optimization.

Interesting, I didn't know that! I guess the mov rax/rdi was a natural part of the sequence on x86_64, and the old original runtime check was opportunistically based on the existing codegen.

So, looking at the patch, this looks correct, but I'm still mildly uncomfortable with the sel bit surviving that late. Here's a maybe-terrible idea: would it work if you replaced the i64 0/1 in the bundle with the actual pointer to the called function? e.g.,

%r = call i8* @foo(i64 %c, i64 %b, i64 %a) [ "clang.arc.attachedcall"(i8*(i8*)* @_objc_retainAutoreleasedReturnValue) ]

on aarch64 you just do the bundle with no args (though "attachedcall" may not be the best name at that point); on x86 you make the frontend pass the actual function it wants (edit: though it sounds like the ARC passes would need more of that logic than the frontend?), and the backend doesn't care and just turns the GV into a target globaladdr, and turns it into a call later

[fhahn]

In D94597#2720350, @ab wrote:

In D94597#2632887, @fhahn wrote:

Unfortunately the ObjC runtime matches different sequences on AArc64 and X86. This is documented here https://github.com/opensource-apple/objc4/blob/master/runtime/objc-object.h#L974

On AArch64, the callee only checks for mov x29, x29 in the caller and the position of the claim/autorelease runtime call does not matter, which makes things much simpler. On X86_64, it specifically looks for the marker instructions, immediately followed by the runtime call. So we want to make sure that it is very hard to break up this specific sequence, to avoid missing out on the runtime optimization.

Interesting, I didn't know that! I guess the mov rax/rdi was a natural part of the sequence on x86_64, and the old original runtime check was opportunistically based on the existing codegen.

So, looking at the patch, this looks correct, but I'm still mildly uncomfortable with the sel bit surviving that late. Here's a maybe-terrible idea: would it work if you replaced the i64 0/1 in the bundle with the actual pointer to the called function? e.g.,

%r = call i8* @foo(i64 %c, i64 %b, i64 %a) [ "clang.arc.attachedcall"(i8*(i8*)* @_objc_retainAutoreleasedReturnValue) ]

on aarch64 you just do the bundle with no args (though "attachedcall" may not be the best name at that point); on x86 you make the frontend pass the actual function it wants (edit: though it sounds like the ARC passes would need more of that logic than the frontend?), and the backend doesn't care and just turns the GV into a target globaladdr, and turns it into a call later

Yeah I think that would be a good option. @ahatanak WDYT?

[ahatanak]

Yes, I think that's better. The verifier currently rejects that IR since taking addresses of intrinsic functions isn't allowed (see https://reviews.llvm.org/D92808#2558928), but probably that can be fixed.

[ab]

Ah sorry, I didn't see you tried that already! I don't think we want to weaken the verifier check, this is probably the only case where it makes sense to take the address. I guess we've meandered through the same decision process, thanks for the patience ;) But it might be worth explaining these decisions in the commit message. Also in the langref paragraph for the bundle, it says the call is always emitted for the bundle.
Which brings another idea: how about emitting the call when lowering the bundle on AArch64 as well? There's not much room for better regalloc or other optimizations in the sequence anyway, right?

But back to x86: there's a couple things we could do to make the behavior more obvious (split the bundle into distinct retain/claim ones? replace the sel imm MO in these pseudos with an actual GV/ES reference to the symbol?). But I'm not sure that makes a real difference, so LGTM as-is

llvm/lib/Target/X86/X86ExpandPseudo.cpp

245

why "Cache"?

llvm/test/CodeGen/X86/call-rv-marker.ll

161

Hmm, I'm curious, do you know why this doesn't trigger? Is it just missing entries in the fold tables? (does the extra operand make that trickier than it should be?)

[fhahn]

Thanks Ahmed! I'll address your comment and then I'll land it. I expect that we will probably be making a couple of improvements once this lands and gets wider test coverage

llvm/lib/Target/X86/X86ExpandPseudo.cpp

245

I'm not sure any more, I'll rename it to Fn :)

llvm/test/CodeGen/X86/call-rv-marker.ll

161

Yes I think there's an entry missing to optimize this. I'll investigate that separately.

llvm/lib/Target/X86/X86ExpandPseudo.cpp

public:
 
   StringRef getPassName() const override {
     return "X86 pseudo instruction expansion pass";
   }
 
 private:
   void ExpandICallBranchFunnel(MachineBasicBlock *MBB,
                                MachineBasicBlock::iterator MBBI);
-
+  void expandCALL_RVMARKER(MachineBasicBlock &MBB,
+                           MachineBasicBlock::iterator MBBI);
   bool ExpandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
   bool ExpandMBB(MachineBasicBlock &MBB);
 
   /// This function expands pseudos which affects control flow.
   /// It is done in separate pass to simplify blocks navigation in main
   /// pass(calling ExpandMBB).
   bool ExpandPseudosWhichAffectControlFlow(MachineFunction &MF);
 
void X86ExpandPseudo::ExpandICallBranchFunnel(
   for (auto P : TargetMBBs) {
     MF->insert(InsPt, P.first);
     BuildMI(P.first, DL, TII->get(X86::TAILJMPd64))
         .add(JTInst->getOperand(3 + 2 * P.second));
   }
   JTMBB->erase(JTInst);
 }
 
+void X86ExpandPseudo::expandCALL_RVMARKER(MachineBasicBlock &MBB,
+                                          MachineBasicBlock::iterator MBBI) {
+  // Expand CALL_RVMARKER pseudo to call instruction, followed by the special
+  //"movq %rax, %rdi" marker.
+  // TODO: Mark the sequence as bundle, to avoid passes moving other code
+  // in between.
+  MachineInstr &MI = *MBBI;
+
+  MachineInstr *OriginalCall;
+  MachineOperand &CallTarget = MI.getOperand(1);
+  assert((CallTarget.isGlobal() || CallTarget.isReg()) &&
+         "invalid operand for regular call");
+  unsigned Opc = -1;
+  if (MI.getOpcode() == X86::CALL64m_RVMARKER)
+    Opc = X86::CALL64m;
+  else if (MI.getOpcode() == X86::CALL64r_RVMARKER)
+    Opc = X86::CALL64r;
+  else if (MI.getOpcode() == X86::CALL64pcrel32_RVMARKER)
+    Opc = X86::CALL64pcrel32;
+  else
+    llvm_unreachable("unexpected opcode");
+
+  OriginalCall = BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(Opc)).getInstr();
+  unsigned OpStart = 1;
+  bool RAXImplicitDead = false;
+  for (; OpStart < MI.getNumOperands(); ++OpStart) {
+    MachineOperand &Op = MI.getOperand(OpStart);
+    // RAX may be 'implicit dead', if there are no other users of the return
+    // value. We introduce a new use, so change it to 'implicit def'.
+    if (Op.isReg() && Op.isImplicit() && Op.isDead() &&
+        TRI->regsOverlap(Op.getReg(), X86::RAX)) {
+      Op.setIsDead(false);
+      Op.setIsDef(true);
+      RAXImplicitDead = true;
+    }
+    OriginalCall->addOperand(Op);
+  }
+
+  // Emit marker "movq %rax, %rdi".  %rdi is not callee-saved, so it cannot be
+  // live across the earlier call. The call to the ObjC runtime function returns
+  // the first argument, so the value of %rax is unchanged after the ObjC
+  // runtime call.
+  auto *Marker = BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(X86::MOV64rr))
+                     .addReg(X86::RDI, RegState::Define)
+                     .addReg(X86::RAX)
+                     .getInstr();
+  if (MI.shouldUpdateCallSiteInfo())
+    MBB.getParent()->moveCallSiteInfo(&MI, Marker);
+
+  // Emit call to ObjC runtime.
+  unsigned RuntimeCallType = MI.getOperand(0).getImm();
+  assert(RuntimeCallType <= 1 && "objc runtime call type must be 0 or 1");
+  Module *M = MBB.getParent()->getFunction().getParent();
+  auto &Context = M->getContext();
+  auto *I8PtrTy = PointerType::get(IntegerType::get(Context, 8), 0);
+  FunctionCallee Fn = M->getOrInsertFunction(
+      RuntimeCallType == 0 ? "objc_retainAutoreleasedReturnValue"
+                           : "objc_unsafeClaimAutoreleasedReturnValue",
+      FunctionType::get(I8PtrTy, {I8PtrTy}, false));
+  const uint32_t *RegMask =
+      TRI->getCallPreservedMask(*MBB.getParent(), CallingConv::C);
+  BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(X86::CALL64pcrel32))
+      .addGlobalAddress(cast<GlobalValue>(Fn.getCallee()), 0, 0)
+      .addRegMask(RegMask)
+      .addReg(X86::RAX,
+              RegState::Implicit |
+                  (RAXImplicitDead ? (RegState::Dead | RegState::Define)
+                                   : RegState::Define))
+      .getInstr();
+  MI.eraseFromParent();
+}
+
 /// If \p MBBI is a pseudo instruction, this method expands
 /// it to the corresponding (sequence of) actual instruction(s).
 /// \returns true if \p MBBI has been expanded.
 bool X86ExpandPseudo::ExpandMI(MachineBasicBlock &MBB,
                                MachineBasicBlock::iterator MBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
   const DebugLoc &DL = MBBI->getDebugLoc();
case X86::PTILESTOREDV: {
     return true;
   }
   case X86::PTILEZEROV: {
     for (int i = 2; i > 0; --i) // Remove row, col
       MI.RemoveOperand(i);
     MI.setDesc(TII->get(X86::TILEZERO));
     return true;
   }
+  case X86::CALL64pcrel32_RVMARKER:
+  case X86::CALL64r_RVMARKER:
+  case X86::CALL64m_RVMARKER:
+    expandCALL_RVMARKER(MBB, MBBI);
+    return true;
   }
   llvm_unreachable("Previous switch has a fallthrough?");
 }
 
 // This function creates additional block for storing varargs guarded
 // registers. It adds check for %al into entry block, to skip
 // GuardedRegsBlk if xmm registers should not be stored.
 //

llvm/lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
     ///     #1 - The first register result value (optional)
     ///     #2 - The second register result value (optional)
     ///
     CALL,
 
     /// Same as call except it adds the NoTrack prefix.
     NT_CALL,
 
+    // Pseudo for a OBJC call that gets emitted together with a special
+    // marker instruction.
+    CALL_RVMARKER,
+
     /// X86 compare and logical compare instructions.
     CMP,
     FCMP,
     COMI,
     UCOMI,
 
     /// X86 bit-test instructions.
     BT,

llvm/lib/Target/X86/X86ISelLowering.cpp

 #include "X86TargetObjectFile.h"
 #include "llvm/ADT/SmallBitVector.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/EHPersonalities.h"
+#include "llvm/Analysis/ObjCARCUtil.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/VectorUtils.h"
 #include "llvm/CodeGen/IntrinsicLowering.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
if (isTailCall) {
     MF.getFrameInfo().setHasTailCall();
     SDValue Ret = DAG.getNode(X86ISD::TC_RETURN, dl, NodeTys, Ops);
     DAG.addCallSiteInfo(Ret.getNode(), std::move(CSInfo));
     return Ret;
   }
 
   if (HasNoCfCheck && IsCFProtectionSupported && IsIndirectCall) {
     Chain = DAG.getNode(X86ISD::NT_CALL, dl, NodeTys, Ops);
+  } else if (CLI.CB && objcarc::hasAttachedCallOpBundle(CLI.CB)) {
+    // Calls with a "clang.arc.attachedcall" bundle are special. They should be
+    // expanded to the call, directly followed by a special marker sequence and
+    // a call to a ObjC library function. Use the CALL_RVMARKER to do that.
+    assert(!isTailCall &&
+           "tail calls cannot be marked with clang.arc.attachedcall");
+    assert(Is64Bit && "clang.arc.attachedcall is only supported in 64bit mode");
+
+    // Add target constant to select ObjC runtime call just before the call
+    // target. RuntimeCallType == 0 selects objc_retainAutoreleasedReturnValue,
+    // RuntimeCallType == 0 selects objc_unsafeClaimAutoreleasedReturnValue when
+    // epxanding the pseudo.
+    unsigned RuntimeCallType =
+        objcarc::hasAttachedCallOpBundle(CLI.CB, true) ? 0 : 1;
+    Ops.insert(Ops.begin() + 1,
+               DAG.getTargetConstant(RuntimeCallType, dl, MVT::i32));
+    Chain = DAG.getNode(X86ISD::CALL_RVMARKER, dl, NodeTys, Ops);
   } else {
     Chain = DAG.getNode(X86ISD::CALL, dl, NodeTys, Ops);
   }
+
   InFlag = Chain.getValue(1);
   DAG.addNoMergeSiteInfo(Chain.getNode(), CLI.NoMerge);
   DAG.addCallSiteInfo(Chain.getNode(), std::move(CSInfo));
 
   // Save heapallocsite metadata.
   if (CLI.CB)
     if (MDNode *HeapAlloc = CLI.CB->getMetadata("heapallocsite"))
       DAG.addHeapAllocSite(Chain.getNode(), HeapAlloc);
#define NODE_NAME_CASE(NODE) case X86ISD::NODE: return "X86ISD::" #NODE;
   NODE_NAME_CASE(FOR)
   NODE_NAME_CASE(FXOR)
   NODE_NAME_CASE(FILD)
   NODE_NAME_CASE(FIST)
   NODE_NAME_CASE(FP_TO_INT_IN_MEM)
   NODE_NAME_CASE(FLD)
   NODE_NAME_CASE(FST)
   NODE_NAME_CASE(CALL)
+  NODE_NAME_CASE(CALL_RVMARKER)
   NODE_NAME_CASE(BT)
   NODE_NAME_CASE(CMP)
   NODE_NAME_CASE(FCMP)
   NODE_NAME_CASE(STRICT_FCMP)
   NODE_NAME_CASE(STRICT_FCMPS)
   NODE_NAME_CASE(COMI)
   NODE_NAME_CASE(UCOMI)
   NODE_NAME_CASE(CMPM)

llvm/lib/Target/X86/X86InstrCompiler.td

 
 // Direct PC relative function call for small code model. 32-bit displacement
 // sign extended to 64-bit.
 def : Pat<(X86call (i64 tglobaladdr:$dst)),
           (CALL64pcrel32 tglobaladdr:$dst)>;
 def : Pat<(X86call (i64 texternalsym:$dst)),
           (CALL64pcrel32 texternalsym:$dst)>;
 
+def : Pat<(X86call_rvmarker (timm:$sel), (i64 texternalsym:$dst)),
+          (CALL64pcrel32_RVMARKER timm:$sel, texternalsym:$dst)>;
+def : Pat<(X86call_rvmarker (timm:$sel), (i64 tglobaladdr:$dst)),
+          (CALL64pcrel32_RVMARKER timm:$sel, tglobaladdr:$dst)>;
+
+
 // Tailcall stuff. The TCRETURN instructions execute after the epilog, so they
 // can never use callee-saved registers. That is the purpose of the GR64_TC
 // register classes.
 //
 // The only volatile register that is never used by the calling convention is
 // %r11. This happens when calling a vararg function with 6 arguments.
 //
 // Match an X86tcret that uses less than 7 volatile registers.

llvm/lib/Target/X86/X86InstrControl.td

let isPseudo = 1, isCall = 1, isCodeGenOnly = 1,
   let isTerminator = 1, isReturn = 1, isBarrier = 1 in {
     def INDIRECT_THUNK_TCRETURN64 :
       PseudoI<(outs), (ins GR64:$dst, i32imm:$offset), []>;
     def INDIRECT_THUNK_TCRETURN32 :
       PseudoI<(outs), (ins GR32:$dst, i32imm:$offset), []>;
   }
 }
 
+let isPseudo = 1, isCall = 1, isCodeGenOnly = 1,
+    Uses = [RSP, SSP],
+    SchedRW = [WriteJump] in {
+  def CALL64m_RVMARKER :
+     PseudoI<(outs), (ins i32imm:$sel, i64mem:$dst), [(X86call_rvmarker timm:$sel, (loadi64 addr:$dst))]>,
+             Requires<[In64BitMode]>;
+
+  def CALL64r_RVMARKER :
+    PseudoI<(outs), (ins i32imm:$sel, GR64:$dst), [(X86call_rvmarker timm:$sel, GR64:$dst)]>,
+            Requires<[In64BitMode]>;
+
+  def CALL64pcrel32_RVMARKER :
+    PseudoI<(outs), (ins i32imm:$sel, i64i32imm_brtarget:$dst), []>,
+            Requires<[In64BitMode]>;
+}
+
 // Conditional tail calls are similar to the above, but they are branches
 // rather than barriers, and they use EFLAGS.
 let isCall = 1, isTerminator = 1, isReturn = 1, isBranch = 1,
     isCodeGenOnly = 1, SchedRW = [WriteJump] in
   let Uses = [RSP, EFLAGS, SSP] in {
   def TCRETURNdi64cc : PseudoI<(outs),
                            (ins i64i32imm_brtarget:$dst, i32imm:$offset,
                             i32imm:$cond), []>;
 
   // This gets substituted to a conditional jump instruction in MC lowering.
   def TAILJMPd64_CC : PseudoI<(outs),
                               (ins i64i32imm_brtarget:$dst, i32imm:$cond), []>;
 }

llvm/lib/Target/X86/X86InstrInfo.td

 def X86callseq_end :
                  SDNode<"ISD::CALLSEQ_END",   SDT_X86CallSeqEnd,
                         [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue]>;
 
 def X86call    : SDNode<"X86ISD::CALL",     SDT_X86Call,
                         [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue,
                          SDNPVariadic]>;
 
+def X86call_rvmarker  : SDNode<"X86ISD::CALL_RVMARKER",     SDT_X86Call,
+                        [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue,
+                         SDNPVariadic]>;
+
+
 def X86NoTrackCall : SDNode<"X86ISD::NT_CALL", SDT_X86Call,
                             [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue,
                              SDNPVariadic]>;
 def X86NoTrackBrind : SDNode<"X86ISD::NT_BRIND", SDT_X86NtBrind,
                              [SDNPHasChain]>;
 
 def X86rep_stos: SDNode<"X86ISD::REP_STOS", SDTX86RepStr,
                         [SDNPHasChain, SDNPInGlue, SDNPOutGlue, SDNPMayStore]>;

llvm/test/CodeGen/X86/call-rv-marker.ll

-; RUN: llc -mtriple=x86_64-apple-macosx -o - %s | FileCheck --check-prefix=CHECK %s
+; RUN: llc -mtriple=x86_64-apple-macosx -verify-machineinstrs -o - %s | FileCheck --check-prefix=CHECK %s
 
 ; TODO: support marker generation with GlobalISel
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 
 declare i8* @foo0(i32)
 declare i8* @foo1()
 
 declare void @llvm.objc.release(i8*)
 declare void @objc_object(i8*)
 
 declare void @foo2(i8*)
 
 declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture)
 
 declare %struct.S* @_ZN1SD1Ev(%struct.S* nonnull dereferenceable(1))
 
 declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture)
 
 
 %struct.S = type { i8 }
 
 @g = global i8* null, align 8
 @fptr = global i8* ()* null, align 8
 
-define i8* @rv_marker_1() {
-; CHECK-LABEL:  rv_marker_1:
+define i8* @rv_marker_1_retain() {
+; CHECK-LABEL:  rv_marker_1_retain:
 ; CHECK:         pushq %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    callq   _foo1
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_retainAutoreleasedReturnValue
 ; CHECK-NEXT:    popq    %rcx
 ; CHECK-NEXT:    retq
 ;
 entry:
-  %call = call "rv_marker" i8* @foo1()
+  %call = call i8* @foo1() [ "clang.arc.attachedcall"(i64 0) ]
+  ret i8* %call
+}
+
+define i8* @rv_marker_1_claim() {
+; CHECK-LABEL:  rv_marker_1_claim:
+; CHECK:         pushq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    callq   _foo1
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_unsafeClaimAutoreleasedReturnValue
+; CHECK-NEXT:    popq    %rcx
+; CHECK-NEXT:    retq
+;
+entry:
+  %call = call i8* @foo1() [ "clang.arc.attachedcall"(i64 1) ]
   ret i8* %call
 }
 
 define void @rv_marker_2_select(i32 %c) {
 ; CHECK-LABEL: rv_marker_2_select:
 ; CHECK:        pushq   %rax
 ; CHECK-NEXT:   .cfi_def_cfa_offset 16
 ; CHECK-NEXT:   cmpl    $1, %edi
 ; CHECK-NEXT:   movl    $1, %edi
 ; CHECK-NEXT:   adcl    $0, %edi
 ; CHECK-NEXT:   callq   _foo0
 ; CHECK-NEXT:   movq    %rax, %rdi
+; CHECK-NEXT:   callq   _objc_retainAutoreleasedReturnValue
+; CHECK-NEXT:   movq    %rax, %rdi
 ; CHECK-NEXT:   popq    %rax
 ; CHECK-NEXT:   jmp _foo2
 ;
 entry:
   %tobool.not = icmp eq i32 %c, 0
   %.sink = select i1 %tobool.not, i32 2, i32 1
-  %call1 = call "rv_marker" i8* @foo0(i32 %.sink)
+  %call1 = call i8* @foo0(i32 %.sink) [ "clang.arc.attachedcall"(i64 0) ]
   tail call void @foo2(i8* %call1)
   ret void
 }
 
 define void @rv_marker_3() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 ; CHECK-LABEL: rv_marker_3
 ; CHECK:         pushq   %r14
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    pushq   %rbx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 24
 ; CHECK-NEXT:    pushq   %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset %rbx, -24
 ; CHECK-NEXT:    .cfi_offset %r14, -16
 ; CHECK-NEXT:    callq   _foo1
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_retainAutoreleasedReturnValue
 ; CHECK-NEXT:    movq    %rax, %rbx
 ; CHECK-NEXT: Ltmp0:
 ;
 entry:
-  %call = call "rv_marker" i8* @foo1()
+  %call = call i8* @foo1() [ "clang.arc.attachedcall"(i64 0) ]
   invoke void @objc_object(i8* %call) #5
           to label %invoke.cont unwind label %lpad
 
 invoke.cont:                                      ; preds = %entry
   tail call void @llvm.objc.release(i8* %call)
   ret void
 
 lpad:                                             ; preds = %entry

 ; CHECK-NEXT:    pushq   %rbx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 24
 ; CHECK-NEXT:    pushq   %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset %rbx, -24
 ; CHECK-NEXT:    .cfi_offset %r14, -16
 ; CHECK-NEXT: Ltmp3:
 ; CHECK-NEXT:    callq   _foo1
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_retainAutoreleasedReturnValue
 ; CHECK-NEXT: Ltmp4:
 ;
 entry:
   %s = alloca %struct.S, align 1
   %0 = getelementptr inbounds %struct.S, %struct.S* %s, i64 0, i32 0
   call void @llvm.lifetime.start.p0i8(i64 1, i8* nonnull %0) #2
-  %call = invoke "rv_marker" i8* @foo1()
+  %call = invoke i8* @foo1() [ "clang.arc.attachedcall"(i64 0) ]
           to label %invoke.cont unwind label %lpad
 
 invoke.cont:                                      ; preds = %entry
   invoke void @objc_object(i8* %call) #5
           to label %invoke.cont2 unwind label %lpad1
 
 invoke.cont2:                                     ; preds = %invoke.cont
   tail call void @llvm.objc.release(i8* %call)

 
 ehcleanup:                                        ; preds = %lpad1, %lpad
   %.pn = phi { i8*, i32 } [ %2, %lpad1 ], [ %1, %lpad ]
   %call4 = call %struct.S* @_ZN1SD1Ev(%struct.S* nonnull dereferenceable(1) %s)
   call void @llvm.lifetime.end.p0i8(i64 1, i8* nonnull %0)
   resume { i8*, i32 } %.pn
 }
 
+; TODO: This should use "callq *_fptr(%rip)".
 define i8* @rv_marker_5_indirect_call() {
 ; CHECK-LABEL: rv_marker_5_indirect_call
 ; CHECK:         pushq   %rbx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset %rbx, -16
-; CHECK-NEXT:    callq   *_fptr(%rip)
+; CHECK-NEXT:    movq    _fptr(%rip), %rax
+; CHECK-NEXT:    callq   *%rax
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_retainAutoreleasedReturnValue
 ; CHECK-NEXT:    movq    %rax, %rbx
 ; CHECK-NEXT:    movq    %rax, %rdi
 ; CHECK-NEXT:    callq   _foo2
 ; CHECK-NEXT:    movq    %rbx, %rax
 ; CHECK-NEXT:    popq    %rbx
 ; CHECK-NEXT:    retq
 ;
 entry:
-  %0 = load i8* ()*, i8* ()** @fptr, align 8
-  %call = call "rv_marker" i8* %0()
+  %lv = load i8* ()*, i8* ()** @fptr, align 8
+  %call = call i8* %lv() [ "clang.arc.attachedcall"(i64 0) ]
   tail call void @foo2(i8* %call)
   ret i8* %call
 }
 
-declare void @foo(i64, i64, i64)
+declare i8* @foo(i64, i64, i64)
 
 define void @rv_marker_multiarg(i64 %a, i64 %b, i64 %c) {
 ; CHECK-LABEL: rv_marker_multiarg
 ; CHECK:         pushq   %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    movq    %rdi, %rax
 ; CHECK-NEXT:    movq    %rdx, %rdi
 ; CHECK-NEXT:    movq    %rax, %rdx
 ; CHECK-NEXT:    callq   _foo
+; CHECK-NEXT:    movq    %rax, %rdi
+; CHECK-NEXT:    callq   _objc_retainAutoreleasedReturnValue
 ; CHECK-NEXT:    popq    %rax
 ; CHECK-NEXT:    retq
 ;
-  call "rv_marker" void @foo(i64 %c, i64 %b, i64 %a)
+  %r = call i8* @foo(i64 %c, i64 %b, i64 %a) [ "clang.arc.attachedcall"(i64 0) ]
+  ret void
+}
+
+define void @test_nonlazybind() {
+; CHECK-LABEL: _test_nonlazybind:
+; CHECK:      bb.0:
+; CHECK-NEXT:  pushq   %rax
+; CHECK-NEXT:  .cfi_def_cfa_offset 16
+; CHECK-NEXT:  callq   *_foo_nonlazybind@GOTPCREL(%rip)
+; CHECK-NEXT:  movq    %rax, %rdi
+; CHECK-NEXT:  callq   _objc_retainAutoreleasedReturnValue
+;
+  %call1 = notail call i8* @foo_nonlazybind() [ "clang.arc.attachedcall"(i64 0) ]
   ret void
 }
 
+declare i8* @foo_nonlazybind()  nonlazybind
+
 declare i32 @__gxx_personality_v0(...)

llvm/test/CodeGen/X86/expand-call-rvmarker.mir

+# RUN: llc -o - -run-pass=x86-pseudo -verify-machineinstrs %s | FileCheck %s
+
+--- |
+  target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-apple-macosx11.0.0"
+
+  declare i8* @objc_retainAutoreleasedReturnValue(i8*)
+  declare i8* @objc_unsafeClaimAutoreleasedReturnValue(i8*)
+  declare i8* @fn()
+
+  define void @test_objc_retainAutoreleaseReturnedValue() {
+    ret void
+  }
+
+  define void @test_objc_unsafeClaimAutoreleasedReturnValue() {
+    ret void
+  }
+
+  define void @test_objc_unsafeClaimAutoreleasedReturnValue_2_args() {
+    ret void
+  }
+
+  define void @test_ret_void() {
+    ret void
+  }
+
+...
+---
+# CHECK-LABEL: name: test_objc_retainAutoreleaseReturnedValue
+# CHECK:      bb.0
+# CHECK-NEXT:   frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   CFI_INSTRUCTION def_cfa_offset 16
+# CHECK-NEXT:   CALL64pcrel32 @fn, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+# CHECK-NEXT:   $rdi = MOV64rr $rax
+# CHECK-NEXT:   CALL64pcrel32 @objc_retainAutoreleasedReturnValue, csr_64, implicit $rsp, implicit $ssp, implicit-def $rax
+# CHECK-NEXT:   $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   RETQ
+#
+
+name:            test_objc_retainAutoreleaseReturnedValue
+alignment:       16
+tracksRegLiveness: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    CFI_INSTRUCTION def_cfa_offset 16
+    CALL64pcrel32_RVMARKER 0, @fn, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+    RET 0, $rax
+
+...
+---
+# CHECK-LABEL: name: test_objc_unsafeClaimAutoreleasedReturnValue
+# CHECK:      bb.0
+# CHECK-NEXT:   frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   CFI_INSTRUCTION def_cfa_offset 16
+# CHECK-NEXT:   CALL64pcrel32 @fn, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+# CHECK-NEXT:   $rdi = MOV64rr $rax
+# CHECK-NEXT:   CALL64pcrel32 @objc_unsafeClaimAutoreleasedReturnValue, csr_64, implicit $rsp, implicit $ssp, implicit-def $rax
+# CHECK-NEXT:   $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   RETQ
+#
+name:            test_objc_unsafeClaimAutoreleasedReturnValue
+alignment:       16
+tracksRegLiveness: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    CFI_INSTRUCTION def_cfa_offset 16
+    CALL64pcrel32_RVMARKER 1, @fn, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+    $rcx = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+    RET 0, $rax
+
+...
+---
+# CHECK-LABEL: name: test_objc_unsafeClaimAutoreleasedReturnValue_2_args
+# CHECK:      bb.0
+# CHECK:        frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   $rax = MOV64rr $rdi
+# CHECK-NEXT:   $rdi = MOV64rr killed $rdx
+# CHECK-NEXT:   $rdx = MOV64rr killed $rax
+# CHECK-NEXT:   CALL64pcrel32 @fn, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit $rdx, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+# CHECK-NEXT:   $rdi = MOV64rr $rax
+# CHECK-NEXT:   CALL64pcrel32 @objc_retainAutoreleasedReturnValue, csr_64, implicit $rsp, implicit $ssp, implicit-def dead $rax
+# CHECK-NEXT:   $rax = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   RETQ
+#
+name:            test_objc_unsafeClaimAutoreleasedReturnValue_2_args
+alignment:       16
+tracksRegLiveness: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $rdi, $rdx, $rsi
+
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    $rax = MOV64rr $rdi
+    $rdi = MOV64rr killed $rdx
+    $rdx = MOV64rr killed $rax
+    CALL64pcrel32_RVMARKER 0, @fn, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit $rdx, implicit-def $rsp, implicit-def $ssp, implicit-def dead $rax
+    $rax = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+    RET 0, $rax
+
+...
+# CHECK-LABEL: name: test_ret_void
+# CHECK:      bb.0
+# CHECK-NEXT:   frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   CFI_INSTRUCTION def_cfa_offset 16
+# CHECK-NEXT:   CALL64pcrel32 @fn, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def $rax
+# CHECK-NEXT:   $rdi = MOV64rr $rax
+# CHECK-NEXT:   CALL64pcrel32 @objc_retainAutoreleasedReturnValue, csr_64, implicit $rsp, implicit $ssp, implicit-def dead $rax
+# CHECK-NEXT:   $rax = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+# CHECK-NEXT:   RETQ
+#
+name:            test_ret_void
+alignment:       16
+tracksRegLiveness: true
+frameInfo:
+  stackSize:       8
+  offsetAdjustment: -8
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    frame-setup PUSH64r undef $rax, implicit-def $rsp, implicit $rsp
+    CFI_INSTRUCTION def_cfa_offset 16
+    CALL64pcrel32_RVMARKER 0, @fn, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp, implicit-def dead $rax
+    $rax = frame-destroy POP64r implicit-def $rsp, implicit $rsp
+    RET 0