Add lowering support for llvm.experimental.deoptimize

[sanjoy]

Summary

Only adds support for "naked" calls to llvm.experimental.deoptimize.
Support for round-tripping through RewriteStatepointsForGC will come
as a separate patch (should be simpler than this one).

[reames]

LGTM w/comments addressed.

docs/LangRef.rst

12174 ↗

(On Diff #51515)

"is available" to "is defined."

12175 ↗

(On Diff #51515)

non-varargs is a bit vague here. "Lowered according to the calling convention specified for the deoptimize call as if they were formal arguments of the specified types not vararg." Wording could use improved, but that gets the spirit.

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

925 ↗

(On Diff #51515)

It looks like you're only handling calls here. Using CallSite is fine, but add an assertion.

940 ↗

(On Diff #51515)

Not sure the exact variable adds anything here.

949 ↗

(On Diff #51515)

You might add a comment to make it clear GC args intentionally remain empty.

test/CodeGen/X86/deopt-intrinsic.ll

2 ↗

(On Diff #51515)

Doesn't -debug-only require an asserts build?

12 ↗

(On Diff #51515)

Might be better to just check for each expected instruction.

13 ↗

(On Diff #51515)

Add a check-next for the return. The fact there's nothing in between is worth checking.

21 ↗

(On Diff #51515)

Is this how we materialize a floating point constant? That seems like it could be improved in general.

26 ↗

(On Diff #51515)

Can you add a test with a different calling convention? There's nothing here that ensures we're using the right one for the lowering as opposed to merely the default.

[sanjoy]

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

940 ↗

(On Diff #51515)

Without it I get linker errors around StatepointDirectives::DeoptBundleStatepointID. I didn't spend time figuring out the root cause (I suspect it has to do with the move constructor in getValueOr), and this looks like an acceptable workaround.

[sanjoy]

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

925 ↗

(On Diff #51515)

In retrospect, this can be easily combined with LowerCallSiteWithDeoptBundle. I'll do that in a separate change if feasible.

[jlebar]

Hi, I think this may be broken.

In particular, I tried to land rL265092, which turns on an assertion, but had to back it out because it asserts on the test added here. (The assertion is currently only run with -debug, which is doubly bad, because now your code can break for unrelated reasons while you're trying to debug it!)

Would you mind having a look? Bonus points if you reland rL265092 after fixing the issue.

[sanjoy]

Justin Lebar wrote:
> Hi, I think this may be broken.
>
> In particular, I tried to land http://reviews.llvm.org/rL265092,
> which turns on an assertion, but had to back it out because it asserts
> on the test added here. (The assertion is currently only run with
> -debug, which is doubly bad, because now your code can break for
> unrelated reasons while you're trying to debug it!)

Looks like webkit_jscc does not work with anything but an i64 return
type. The following crashes llc, for instance:

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.11.0"

declare webkit_jscc i32 @foo()

define i32 @caller() {
entry:

%v = call webkit_jscc i32 @foo()
ret i32 %v

}

CC'ing Andy and Juergen, since they might know. For now I'll switch
the test to return i64 and turn on the assertion.

> Would you mind having a look? Bonus points if you reland
> http://reviews.llvm.org/rL265092 after fixing the issue.

• Sanjoy

[sanjoy]

Hi,

I've re-enabled the assertion in rL265099, and filed PR27172 for the
webjit_jscc issue.

• Sanjoy

Sanjoy Das wrote:

sanjoy added a comment.

Justin Lebar wrote:

>  Hi, I think this may be broken.
>
>  In particular, I tried to land http://reviews.llvm.org/rL265092,
>  which turns on an assertion, but had to back it out because it asserts
>  on the test added here.  (The assertion is currently only run with
>  -debug, which is doubly bad, because now your code can break for
>  unrelated reasons while you're trying to debug it!)

Looks like webkit_jscc does not work with anything but an i64 return
type. The following crashes llc, for instance:

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.11.0"

declare webkit_jscc i32 @foo()

define i32 @caller() {
entry:

%v = call webkit_jscc i32 @foo()
ret i32 %v

}

CC'ing Andy and Juergen, since they might know. For now I'll switch
the test to return i64 and turn on the assertion.

Would you mind having a look? Bonus points if you reland

>  http://reviews.llvm.org/rL265092 after fixing the issue.

• Sanjoy

Repository:

rL LLVM

http://reviews.llvm.org/D18429

[atrick]

In D18429#389103, @sanjoy wrote:

Looks like webkit_jscc does not work with anything but an i64 return
type. The following crashes llc, for instance:

CC'ing Andy and Juergen, since they might know. For now I'll switch
the test to return i64 and turn on the assertion.

That isn't too surprising. Although I thought it worked with f64.

We don't need to keep the webkit_jscc convention. I think its only purpose is to test codegen for patchpoints. Juergen?

llvm/trunk/docs/LangRef.rst

 ``"deopt"`` continuations present in the inlinee, and also updates calls to this
 intrinsic to return directly from the frame of the function it inlined into.
 
 .. _deoptimize_lowering:
 
 Lowering:
 """""""""
 
-Lowering for ``@llvm.experimental.deoptimize`` is not yet implemented,
-and is a work in progress.
+Calls to ``@llvm.experimental.deoptimize`` are lowered to calls to the
+symbol ``__llvm_deoptimize`` (it is the frontend's responsibility to
+ensure that this symbol is defined).  The call arguments to
+``@llvm.experimental.deoptimize`` are lowered as if they were formal
+arguments of the specified types, and not as varargs.
+
 
 Stack Map Intrinsics
 --------------------
 
 LLVM provides experimental intrinsics to support runtime patching
 mechanisms commonly desired in dynamic language JITs. These intrinsics
 are described in :doc:`StackMaps`.

llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h

enum Libcall {
     SYNC_FETCH_AND_UMIN_2,
     SYNC_FETCH_AND_UMIN_4,
     SYNC_FETCH_AND_UMIN_8,
     SYNC_FETCH_AND_UMIN_16,
 
     // Stack Protector Fail.
     STACKPROTECTOR_CHECK_FAIL,
 
+    // Deoptimization.
+    DEOPTIMIZE,
+
     UNKNOWN_LIBCALL
   };
 
   /// getFPEXT - Return the FPEXT_*_* value for the given types, or
   /// UNKNOWN_LIBCALL if there is none.
   Libcall getFPEXT(EVT OpVT, EVT RetVT);
 
   /// getFPROUND - Return the FPROUND_*_* value for the given types, or

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

public:
   // This function is responsible for the whole statepoint lowering process.
   // It uniformly handles invoke and call statepoints.
   void LowerStatepoint(ImmutableStatepoint Statepoint,
                        const BasicBlock *EHPadBB = nullptr);
 
   void LowerCallSiteWithDeoptBundle(ImmutableCallSite CS, SDValue Callee,
                                     const BasicBlock *EHPadBB);
 
+  void LowerDeoptimizeCall(const CallInst *CI);
+
 private:
   // Terminator instructions.
   void visitRet(const ReturnInst &I);
   void visitBr(const BranchInst &I);
   void visitSwitch(const SwitchInst &I);
   void visitIndirectBr(const IndirectBrInst &I);
   void visitUnreachable(const UnreachableInst &I);
   void visitCleanupRet(const CleanupReturnInst &I);

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::eh_exceptioncode: {
     unsigned VReg = FuncInfo.getCatchPadExceptionPointerVReg(CPI, PtrRC);
     SDValue N =
         DAG.getCopyFromReg(DAG.getEntryNode(), getCurSDLoc(), VReg, PtrVT);
     if (Intrinsic == Intrinsic::eh_exceptioncode)
       N = DAG.getZExtOrTrunc(N, getCurSDLoc(), MVT::i32);
     setValue(&I, N);
     return nullptr;
   }
+
+  case Intrinsic::experimental_deoptimize:
+    LowerDeoptimizeCall(&I);
+    return nullptr;
   }
 }
 
 std::pair<SDValue, SDValue>
 SelectionDAGBuilder::lowerInvokable(TargetLowering::CallLoweringInfo &CLI,
                                     const BasicBlock *EHPadBB) {
   MachineModuleInfo &MMI = DAG.getMachineFunction().getMMI();
   MCSymbol *BeginLabel = nullptr;

llvm/trunk/lib/CodeGen/SelectionDAG/StatepointLowering.cpp

SDValue SpillLoad =
                   false, false, false, 0);
 
   // Again, be conservative, don't emit pending loads
   DAG.setRoot(SpillLoad.getValue(1));
 
   assert(SpillLoad.getNode());
   setValue(&Relocate, SpillLoad);
 }
+
+void SelectionDAGBuilder::LowerDeoptimizeCall(const CallInst *CI) {
+  const auto &TLI = DAG.getTargetLoweringInfo();
+
+  SDValue Callee = DAG.getExternalSymbol(TLI.getLibcallName(RTLIB::DEOPTIMIZE),
+                                         TLI.getPointerTy(DAG.getDataLayout()));
+  StatepointLoweringInfo SI(DAG);
+  unsigned ArgBeginIndex = CI->arg_begin() - CI->op_begin();
+  populateCallLoweringInfo(SI.CLI, CI, ArgBeginIndex, CI->getNumArgOperands(),
+                           Callee, CI->getType(), false);
+
+  // We don't lower calls to __llvm_deoptimize as varargs, but as a
+  // regular call.
+  assert(!SI.CLI.IsVarArg && "Expected from populateCallLoweringInfo!");
+
+  auto DeoptBundle = *CI->getOperandBundle(LLVMContext::OB_deopt);
+
+  unsigned DefaultID = StatepointDirectives::DeoptBundleStatepointID;
+
+  auto SD = parseStatepointDirectivesFromAttrs(CI->getAttributes());
+  SI.ID = SD.StatepointID.getValueOr(DefaultID);
+  SI.NumPatchBytes = SD.NumPatchBytes.getValueOr(0);
+
+  SI.DeoptState =
+      ArrayRef<const Use>(DeoptBundle.Inputs.begin(), DeoptBundle.Inputs.end());
+  SI.StatepointFlags = static_cast<uint64_t>(StatepointFlags::None);
+
+  // NB! The GC arguments are specifically left empty.
+
+  if (SDValue ReturnVal = LowerAsSTATEPOINT(SI)) {
+    ReturnVal = lowerRangeToAssertZExt(DAG, *CI, ReturnVal);
+    setValue(CI, ReturnVal);
+  }
+}

llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp

static void InitLibcallNames(const char **Names, const Triple &TT) {
 
   // For f16/f32 conversions, Darwin uses the standard naming scheme, instead
   // of the gnueabi-style __gnu_*_ieee.
   // FIXME: What about other targets?
   if (TT.isOSDarwin()) {
     Names[RTLIB::FPEXT_F16_F32] = "__extendhfsf2";
     Names[RTLIB::FPROUND_F32_F16] = "__truncsfhf2";
   }
+
+  Names[RTLIB::DEOPTIMIZE] = "__llvm_deoptimize";
 }
 
 /// InitLibcallCallingConvs - Set default libcall CallingConvs.
 ///
 static void InitLibcallCallingConvs(CallingConv::ID *CCs) {
   for (int i = 0; i < RTLIB::UNKNOWN_LIBCALL; ++i) {
     CCs[i] = CallingConv::C;
   }

llvm/trunk/test/CodeGen/X86/deopt-intrinsic.ll

+; RUN: llc < %s | FileCheck %s
+; RUN: llc -debug-only=stackmaps < %s 2>&1 | FileCheck --check-prefix=STACKMAPS %s
+; REQUIRES: asserts
+
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-apple-macosx10.11.0"
+
+declare i32 @llvm.experimental.deoptimize.i32(...)
+declare i8  @llvm.experimental.deoptimize.i8(...)
+
+define i32 @caller_0() {
+; CHECK-LABEL: _caller_0:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: ##{{.+}}
+; CHECK-NEXT: pushq   %rax
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: callq ___llvm_deoptimize
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: popq %rcx
+; CHECK-NEXT: retq
+entry:
+  %v = call i32(...) @llvm.experimental.deoptimize.i32() [ "deopt"(i32 0) ]
+  ret i32 %v
+}
+
+define i8 @caller_1() {
+; CHECK-LABEL: _caller_1:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: ##{{.+}}
+; CHECK-NEXT: pushq %rax
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: movss {{[a-zA-Z0-9_]+}}(%rip), %xmm0    ## xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT: movl  $42, %edi
+; CHECK-NEXT: callq ___llvm_deoptimize
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: popq %rcx
+; CHECK-NEXT: retq
+
+entry:
+  %v = call i8(...) @llvm.experimental.deoptimize.i8(i32 42, float 500.0) [ "deopt"(i32 1) ]
+  ret i8 %v
+}
+
+define i8 @caller_2() {
+; CHECK-LABEL: _caller_2:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: ##{{.+}}
+; CHECK-NEXT: pushq %rax
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: {{.+cfi.+}}
+; CHECK-NEXT: movl  $1140457472, (%rsp)     ## imm = 0x43FA0000
+; CHECK-NEXT: movl  $42, %eax
+; CHECK-NEXT: callq ___llvm_deoptimize
+; CHECK-NEXT: {{Ltmp[0-9]+}}:
+; CHECK-NEXT: popq %rcx
+; CHECK-NEXT: retq
+
+entry:
+  %v = call webkit_jscc i8(...) @llvm.experimental.deoptimize.i8(i32 42, float 500.0) [ "deopt"(i32 3) ]
+  ret i8 %v
+}
+
+; STACKMAPS: Stack Maps: callsites:
+; STACKMAPS-NEXT: Stack Maps: callsite 2882400015
+; STACKMAPS-NEXT: Stack Maps:   has 4 locations
+; STACKMAPS-NEXT: Stack Maps:     Loc 0: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:     Loc 1: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:     Loc 2: Constant 1 [encoding: .byte 4, .byte 8, .short 0, .int 1]
+; STACKMAPS-NEXT: Stack Maps:     Loc 3: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:   has 0 live-out registers
+; STACKMAPS-NEXT: Stack Maps: callsite 2882400015
+; STACKMAPS-NEXT: Stack Maps:   has 4 locations
+; STACKMAPS-NEXT: Stack Maps:     Loc 0: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:     Loc 1: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:     Loc 2: Constant 1 [encoding: .byte 4, .byte 8, .short 0, .int 1]
+; STACKMAPS-NEXT: Stack Maps:     Loc 3: Constant 1 [encoding: .byte 4, .byte 8, .short 0, .int 1]
+; STACKMAPS-NEXT: Stack Maps:   has 0 live-out registers
+; STACKMAPS-NEXT: Stack Maps: callsite 2882400015
+; STACKMAPS-NEXT: Stack Maps:   has 4 locations
+; STACKMAPS-NEXT: Stack Maps:     Loc 0: Constant 12  [encoding: .byte 4, .byte 8, .short 0, .int 12]
+; STACKMAPS-NEXT: Stack Maps:     Loc 1: Constant 0 [encoding: .byte 4, .byte 8, .short 0, .int 0]
+; STACKMAPS-NEXT: Stack Maps:     Loc 2: Constant 1 [encoding: .byte 4, .byte 8, .short 0, .int 1]
+; STACKMAPS-NEXT: Stack Maps:     Loc 3: Constant 3 [encoding: .byte 4, .byte 8, .short 0, .int 3]
+; STACKMAPS-NEXT: Stack Maps:   has 0 live-out registers