[RISCV] Add Stackmap/Statepoint/Patchpoint support without targets

[Zeavee]

Summary

This patch adds stackmap support for RISC-V without targets (i.e. the nop patchable forms).

[Zeavee]

This comment was removed by Zeavee.

[Zeavee]

Removes RISCVMCInstLower.h which was not used.

[Zeavee]

Adds a missing 0 IMM in a JALR instruction.

[asb]

Thanks for the contribution! I'm not very familiar with the stackmap/patchpoint/statepoint infrastructure so it may take a bit of time to review this (if anyone else on the CC list has looked at it before, please do dive in!).

One quick question: do you expect this to work on RV32 as well? If so, it would be good to have test coverage for that also.

[Zeavee]

Hello! I am happy to help! Since it is my first contribution, I am not yet familiar with everything, so it might be a little bit messy at the start, sorry in advance.

For your question, I only implemented it for RV64 for another project, and from what I can think of now, the only issue for the current implementation to work on RV32 would be the method RISCVAsmPrinter::LowerPATCHPOINT, as it materializes a 64 bits jump address. From what I see in the other architectures, the materialized jump address always seems to be 64 bits (if I did not miss anything). Thus, either it can work for 32 bits architecture too, or those architectures only implement 64 bits. I will try locally to check this.

[Zeavee]

Removes unnecessary space on empty line.

[Zeavee]

I looked a bit deeper into a RV32 implementation, but in fact, according to this https://llvm.org/docs/StackMaps.html#stack-map-format, stackmap support is currently only implemented for 64-bit platforms. Thus it might require more work in the architecture independent part of stackmap, which I am not totally familiar with for now and I also imagine it would be better to start with the more common architectures instead of RV32.

[luismarques]

Don't forget to include the full context in your patches. See https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface
It looks like you also want to use clang-format to reformat the code.

Shouldn't the lowering code check that the subtarget is RV64, and not RV32?

[Zeavee]

Sorry for the delay, it took me some time to run clang-format. I also added the context as asked. Finally, I added an assert to make sure STACKMAP, PATCHPOINT and STATEPOINT are only used with RV64.

[asb]

In D123496#3445710, @Zeavee wrote:

Sorry for the delay, it took me some time to run clang-format. I also added the context as asked. Finally, I added an assert to make sure STACKMAP, PATCHPOINT and STATEPOINT are only used with RV64.

I'm not seeing the codepaths for the case where the intrinsics are used on RV32? You'll want to use report_fatal_error rather than assert (as the assert wont' be checked on release builds).

[Zeavee]

I replaced assert by report_fatal_error to indeed stop the execution when reaching this point. I did not include any code for when the intrinsics are run on RV32 as accordingly to the LLVM documentation, Stack map is a 64 bits feature. Thus I only implemented it for RV64. Should I do it for RV32 too?

[asb]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9529

I think you mean !Subtarget.is64Bit()?

9531

Nit: maybe "are only supported on 64-bit targets"?

[Zeavee]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9529

Indeed, sorry.

[Zeavee]

Added requested changes on error when code is executed on RV32.

[luismarques]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

160

Nit: put some ' digit separators to make it easier to read the constant. E.g. 0x0xFFFF'FFFF'FFFF. (Or use some higher-level function from MathExtras.h if there is one).

166–193

Can't we do better than this for materializing the address? If the number of instructions doesn't have to be fixed you can even use generateInstSeq from RISCVMatInt.h to generate a more optimal instruction sequence.

[reames]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

139

You repeat this code a number of times. Please pull out a helper function EmitNops(N), and maybe one for EmitNop as well.

You might be able to reuse AP.emitNops here, but that returns a fixed number of nops, not a fixed number of nop bytes. Looking at the x86 parallel, I think that's what you want here.

156

CallTarget can be a global symbol as well as a constant.

I'd suggest splitting this patch into support for patchpoint/statapoint without targets (i.e. the nop patchable forms), and then adding the target resolution separately. The call resolution adds a decent amount of complexity, and is worth reviewing in detail on it's own.

277

Seeing this on the AsmPrinter is suprising. I'd have expected this code to be in RISCVMCInstLower.cpp. Is there something I'm missing here?

[Zeavee]

I removed the code related to the target resolution, to put it in another patch as suggested. I still took the comments into account and they will be applied in the other patch. I also used emitNops to avoid code duplication when emitting the Nops.

[Zeavee]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

160

Applied in next patch.

166–193

generateInstSeq is indeed better, thank you!

277

It seems to be the case for most architectures. It is true however that the lowering of those instructions should be in RISCVMCInstLower.cpp as it would be more consistent. Should I move it there?

[reames]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

277

You should ignore me here. On a quick skim, I was confusing being in the MCInstLowering with not being on the AsmPrinter class. I was wrong, and you have the right placement.

[Zeavee]

Add case for STACKMAP, STATEPOINT and PATCHPOINT in getInstSizeInBytes.

[craig.topper]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

16

Is this used? I see the discussion about using RISCVMatInt::generateInstSeq but I don't see any code.

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

998 ↗

(On Diff #444721)

Use a switch?

[Zeavee]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

16

It will be used in the next patch with targets, but I can remove this import in this one, I missed it.

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

998 ↗

(On Diff #444721)

Sure!

[Zeavee]

Use a switch instead of if else for getInstSizeInBytes and remove unused import.

[Zeavee]

Remove Opers temporary variable to avoid compilation error in switch.

[jrtc27]

Please use update_llc/mir_test_checks.py for all the tests that aren't checking for data directives

[Zeavee]

Update tests that do not check data directive using update_llc_test_checks.py.

[jrtc27]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

103

Is it still invalid with RVC? You can have 2-byte NOPs there...

llvm/test/CodeGen/RISCV/rv64-patchpoint.ll

4

RISCV doesn't have FastISel support, it is identical to the previous line

llvm/test/CodeGen/RISCV/rv64-stackmap.ll

2

Don't line up despite your attempts to make them

llvm/test/CodeGen/RISCV/stackmap-frame-setup.ll

2

You've got no CHECK lines because this is outputting MIR not assembly, hence you want update_mir_test_checks.py

17–19

Huh?

[Zeavee]

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

103

I indeed missed the C_NOP. Should we add one when NumNOPBytes % 4 == 2 then, because emitNops only emits 4 bytes NOPs?

llvm/test/CodeGen/RISCV/rv64-patchpoint.ll

4

I will remove it then.

llvm/test/CodeGen/RISCV/stackmap-frame-setup.ll

17–19

This was added by update_llc_test_checks.py.

[Zeavee]

Use 2-bytes NOPs when they are available. Fix tests that used FastISel or an incorrect checks generator.

[jrtc27]

Why do you have both stackmap.ll and rv64-stackmap.ll? And why does stackmap-frame-setup.ll not have an rv64- prefix like the rest (once that duplication is resolved)?

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

103

I don't know, it depends what you want stackmap code to do, which is something I'm not familiar with. All I know is that, without additional context, requesting 18 bytes of NOPs is a perfectly reasonable thing to do on RISC-V, just as requesting 7 bytes is reasonable on x86 due to it having 1-byte NOPs (0x90). I'll note that @reames already pointed out the discrepancy between emitNops and getNumPatchBytes in terms of bytes vs instructions below.

Also, AsmPrinter::emitNops emits whatever TII->getNop() returns, which is C_NOP on RISC-V if RVC is enabled, so you currently have a bug that this is half the number of bytes being requested in that case.

llvm/test/CodeGen/RISCV/rv64-patchpoint.ll

3

You have a bunch of spaces like this left in various files

llvm/test/CodeGen/RISCV/rv64-stackmap.ll

4

Why's this in a test? This is documentation for how to use llc...

llvm/test/CodeGen/RISCV/stackmap.ll

4

Ditto

[Zeavee]

Fixed spaces issues in tests, added correct prefixes and merged duplicates.

[Zeavee]

In D123496#3660430, @jrtc27 wrote:

Why do you have both stackmap.ll and rv64-stackmap.ll? And why does stackmap-frame-setup.ll not have an rv64- prefix like the rest (once that duplication is resolved)?

I mirrored the AArch64 stackmap tests to try to include all tests cases. But I will add rv64- prefixes if it is better.

[Zeavee]

Rebased the patch on main. For context, this patch has been used by the LLVM backend of GraalVM's Native Image project in production for around 4 months with no major issues.

[reames]

LGTM

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

283–296

You can add the new cases here to the switch above.

[Zeavee]

Remove duplicate switch in RISCVAsmPrinter::emitInstruction

[Zeavee]

Rebase patch on main

[Zeavee]

Hello, sorry for the ping, but the revision has been accepted and the requested changes applied. Would it be possible to have it merged in the main branch?

[Zeavee]

Rebased patch on main branch.

llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp

 //===-- RISCVAsmPrinter.cpp - RISCV LLVM assembly writer ------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains a printer that converts from our internal representation
 // of machine-dependent LLVM code to the RISCV assembly language.
 //
 //===----------------------------------------------------------------------===//
 
 #include "MCTargetDesc/RISCVInstPrinter.h"
 #include "MCTargetDesc/RISCVMCExpr.h"
 #include "MCTargetDesc/RISCVTargetStreamer.h"
 #include "RISCV.h"
 #include "RISCVTargetMachine.h"
 #include "TargetInfo/RISCVTargetInfo.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/StackMaps.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCInst.h"
+#include "llvm/MC/MCInstBuilder.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/raw_ostream.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "asm-printer"
 
 STATISTIC(RISCVNumInstrsCompressed,
           "Number of RISC-V Compressed instructions emitted");
 
 namespace {
 class RISCVAsmPrinter : public AsmPrinter {
   const MCSubtargetInfo *MCSTI;
+  StackMaps SM;
   const RISCVSubtarget *STI;
 
 public:
   explicit RISCVAsmPrinter(TargetMachine &TM,
                            std::unique_ptr<MCStreamer> Streamer)
-      : AsmPrinter(TM, std::move(Streamer)), MCSTI(TM.getMCSubtargetInfo()) {}
+      : AsmPrinter(TM, std::move(Streamer)), MCSTI(TM.getMCSubtargetInfo()),
+        SM(*this) {}
 
   StringRef getPassName() const override { return "RISCV Assembly Printer"; }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
 
+  void LowerSTACKMAP(MCStreamer &OutStreamer, StackMaps &SM,
+                     const MachineInstr &MI);
+
+  void LowerPATCHPOINT(MCStreamer &OutStreamer, StackMaps &SM,
+                       const MachineInstr &MI);
+
+  void LowerSTATEPOINT(MCStreamer &OutStreamer, StackMaps &SM,
+                       const MachineInstr &MI);
+
   void emitInstruction(const MachineInstr *MI) override;
 
   bool PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
                        const char *ExtraCode, raw_ostream &OS) override;
   bool PrintAsmMemoryOperand(const MachineInstr *MI, unsigned OpNo,
                              const char *ExtraCode, raw_ostream &OS) override;
 
   void EmitToStreamer(MCStreamer &S, const MCInst &Inst);
   bool emitPseudoExpansionLowering(MCStreamer &OutStreamer,
                                    const MachineInstr *MI);
 
   // Wrapper needed for tblgenned pseudo lowering.
   bool lowerOperand(const MachineOperand &MO, MCOperand &MCOp) const {
     return LowerRISCVMachineOperandToMCOperand(MO, MCOp, *this);
   }
 
   void emitStartOfAsmFile(Module &M) override;
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
 
 private:
   void emitAttributes();
+  void emitCallProlog(MCStreamer &OutStreamer);
+  void emitCallEpilog(MCStreamer &OutStreamer);
 };
 }
 
+void RISCVAsmPrinter::emitCallProlog(MCStreamer &OutStreamer) {
+  EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ADDI)
+                                  .addReg(RISCV::X2)
+                                  .addReg(RISCV::X2)
+                                  .addImm(-8));
+  EmitToStreamer(
+      OutStreamer,
+      MCInstBuilder(RISCV::SD).addReg(RISCV::X1).addReg(RISCV::X2).addImm(0));
+}
+
+void RISCVAsmPrinter::emitCallEpilog(MCStreamer &OutStreamer) {
+  EmitToStreamer(
+      OutStreamer,
+      MCInstBuilder(RISCV::LD).addReg(RISCV::X1).addReg(RISCV::X2).addImm(0));
+  EmitToStreamer(
+      OutStreamer,
+      MCInstBuilder(RISCV::ADDI).addReg(RISCV::X2).addReg(RISCV::X2).addImm(8));
+}
+
+void RISCVAsmPrinter::LowerSTACKMAP(MCStreamer &OutStreamer, StackMaps &SM,
+                                    const MachineInstr &MI) {
+  unsigned NumNOPBytes = StackMapOpers(&MI).getNumPatchBytes();
+
+  auto &Ctx = OutStreamer.getContext();
+  MCSymbol *MILabel = Ctx.createTempSymbol();
+  OutStreamer.emitLabel(MILabel);
+
+  SM.recordStackMap(*MILabel, MI);
+  assert(NumNOPBytes % 4 == 0 && "Invalid number of NOP bytes requested!");
+
+  // Scan ahead to trim the shadow.
+  const MachineBasicBlock &MBB = *MI.getParent();
+  MachineBasicBlock::const_iterator MII(MI);
+  ++MII;
+  while (NumNOPBytes > 0) {
+    if (MII == MBB.end() || MII->isCall() ||
+        MII->getOpcode() == RISCV::DBG_VALUE ||
+        MII->getOpcode() == TargetOpcode::PATCHPOINT ||
+        MII->getOpcode() == TargetOpcode::STACKMAP)
+      break;
+    ++MII;
+    NumNOPBytes -= 4;
+  }
+
+  // Emit nops.
+  for (unsigned i = 0; i < NumNOPBytes; i += 4)
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ADDI)
+                                    .addReg(RISCV::X0)
+                                    .addReg(RISCV::X0)
+                                    .addImm(0));
+}
+
+// Lower a patchpoint of the form:
+// [<def>], <id>, <numBytes>, <target>, <numArgs>
+void RISCVAsmPrinter::LowerPATCHPOINT(MCStreamer &OutStreamer, StackMaps &SM,
+                                      const MachineInstr &MI) {
+  auto &Ctx = OutStreamer.getContext();
+  MCSymbol *MILabel = Ctx.createTempSymbol();
+  OutStreamer.emitLabel(MILabel);
+  SM.recordPatchPoint(*MILabel, MI);
+
+  PatchPointOpers Opers(&MI);
+
+  int64_t CallTarget = Opers.getCallTarget().getImm();
+  unsigned EncodedBytes = 0;
+  if (CallTarget) {
+    assert((CallTarget & 0xFFFFFFFFFFFF) == CallTarget &&
+           "High 16 bits of call target should be zero.");
+    EncodedBytes = 16;
+    // Materialize the jump address:
+
+    emitCallProlog(OutStreamer);
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ORI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X0)
+                                    .addImm((CallTarget >> 36) & 0xFFF));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::SLLI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm(12));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ORI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm((CallTarget >> 24) & 0xFFF));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::SLLI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm(12));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ORI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm((CallTarget >> 12) & 0xFFF));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::SLLI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm(12));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ORI)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm(CallTarget & 0xFFF));
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::JALR)
+                                    .addReg(RISCV::X1)
+                                    .addReg(RISCV::X1)
+                                    .addImm(0));
+    emitCallEpilog(OutStreamer);
+  }
+  // Emit padding.
+  unsigned NumBytes = Opers.getNumPatchBytes();
+  assert(NumBytes >= EncodedBytes &&
+         "Patchpoint can't request size less than the length of a call.");
+  assert((NumBytes - EncodedBytes) % 4 == 0 &&
+         "Invalid number of NOP bytes requested!");
+  for (unsigned i = EncodedBytes; i < NumBytes; i += 4)
+    EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ADDI)
+                                    .addReg(RISCV::X0)
+                                    .addReg(RISCV::X0)
+                                    .addImm(0));
+}
+
+void RISCVAsmPrinter::LowerSTATEPOINT(MCStreamer &OutStreamer, StackMaps &SM,
+                                      const MachineInstr &MI) {
+  StatepointOpers SOpers(&MI);
+  if (unsigned PatchBytes = SOpers.getNumPatchBytes()) {
+    assert(PatchBytes % 4 == 0 && "Invalid number of NOP bytes requested!");
+    for (unsigned i = 0; i < PatchBytes; i += 4)
+      EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::ADDI)
+                                      .addReg(RISCV::X0)
+                                      .addReg(RISCV::X0)
+                                      .addImm(0));
+  } else {
+    // Lower call target and choose correct opcode
+    const MachineOperand &CallTarget = SOpers.getCallTarget();
+    MCOperand CallTargetMCOp;
+    emitCallProlog(OutStreamer);
+    switch (CallTarget.getType()) {
+    case MachineOperand::MO_GlobalAddress:
+    case MachineOperand::MO_ExternalSymbol:
+      llvm::LowerRISCVMachineOperandToMCOperand(CallTarget, CallTargetMCOp,
+                                                *this);
+      EmitToStreamer(
+          OutStreamer,
+          MCInstBuilder(RISCV::PseudoCALL).addOperand(CallTargetMCOp));
+      break;
+    case MachineOperand::MO_Immediate:
+      CallTargetMCOp = MCOperand::createImm(CallTarget.getImm());
+      EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::JAL)
+                                      .addReg(RISCV::X1)
+                                      .addOperand(CallTargetMCOp));
+      break;
+    case MachineOperand::MO_Register:
+      CallTargetMCOp = MCOperand::createReg(CallTarget.getReg());
+      EmitToStreamer(OutStreamer, MCInstBuilder(RISCV::JALR)
+                                      .addReg(RISCV::X1)
+                                      .addOperand(CallTargetMCOp)
+                                      .addImm(0));
+      break;
+    default:
+      llvm_unreachable("Unsupported operand type in statepoint call target");
+      break;
+    }
+    emitCallEpilog(OutStreamer);
+  }
+
+  auto &Ctx = OutStreamer.getContext();
+  MCSymbol *MILabel = Ctx.createTempSymbol();
+  OutStreamer.emitLabel(MILabel);
+  SM.recordStatepoint(*MILabel, MI);
+}
+
 #define GEN_COMPRESS_INSTR
 #include "RISCVGenCompressInstEmitter.inc"
 void RISCVAsmPrinter::EmitToStreamer(MCStreamer &S, const MCInst &Inst) {
   MCInst CInst;
   bool Res = compressInst(CInst, Inst, *STI, OutStreamer->getContext());
   if (Res)
     ++RISCVNumInstrsCompressed;
   AsmPrinter::EmitToStreamer(*OutStreamer, Res ? CInst : Inst);
 }
 
 // Simple pseudo-instructions have their lowering (with expansion to real
 // instructions) auto-generated.
 #include "RISCVGenMCPseudoLowering.inc"
 
 void RISCVAsmPrinter::emitInstruction(const MachineInstr *MI) {
   // Do any auto-generated pseudo lowerings.
   if (emitPseudoExpansionLowering(*OutStreamer, MI))
     return;
 
   MCInst TmpInst;
+  switch (MI->getOpcode()) {
+  default:
     if (!lowerRISCVMachineInstrToMCInst(MI, TmpInst, *this))
       EmitToStreamer(*OutStreamer, TmpInst);
+    return;
+  case TargetOpcode::STACKMAP:
+    return LowerSTACKMAP(*OutStreamer, SM, *MI);
+
+  case TargetOpcode::PATCHPOINT:
+    return LowerPATCHPOINT(*OutStreamer, SM, *MI);
+
+  case TargetOpcode::STATEPOINT:
+    return LowerSTATEPOINT(*OutStreamer, SM, *MI);
+  }
 }
 
 bool RISCVAsmPrinter::PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
                                       const char *ExtraCode, raw_ostream &OS) {
   // First try the generic code, which knows about modifiers like 'c' and 'n'.
   if (!AsmPrinter::PrintAsmOperand(MI, OpNo, ExtraCode, OS))
     return false;
 

 }
 
 void RISCVAsmPrinter::emitEndOfAsmFile(Module &M) {
   RISCVTargetStreamer &RTS =
       static_cast<RISCVTargetStreamer &>(*OutStreamer->getTargetStreamer());
 
   if (TM.getTargetTriple().isOSBinFormatELF())
     RTS.finishAttributeSection();
+
+  emitStackMaps(SM);
 }
 
 void RISCVAsmPrinter::emitAttributes() {
   RISCVTargetStreamer &RTS =
       static_cast<RISCVTargetStreamer &>(*OutStreamer->getTargetStreamer());
   RTS.emitTargetAttributes(*MCSTI);
 }
 

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

RISCVTargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
   case RISCV::PseudoQuietFLE_S:
     return emitQuietFCMP(MI, BB, RISCV::FLE_S, RISCV::FEQ_S, Subtarget);
   case RISCV::PseudoQuietFLT_S:
     return emitQuietFCMP(MI, BB, RISCV::FLT_S, RISCV::FEQ_S, Subtarget);
   case RISCV::PseudoQuietFLE_D:
     return emitQuietFCMP(MI, BB, RISCV::FLE_D, RISCV::FEQ_D, Subtarget);
   case RISCV::PseudoQuietFLT_D:
     return emitQuietFCMP(MI, BB, RISCV::FLT_D, RISCV::FEQ_D, Subtarget);
+  case TargetOpcode::STACKMAP:
+  case TargetOpcode::PATCHPOINT:
+  case TargetOpcode::STATEPOINT:
+    if (!Subtarget.is64Bit())
+      report_fatal_error("STACKMAP, PATCHPOINT and STATEPOINT are only "
+                         "supported on 64-bit targets");
+    return emitPatchPoint(MI, BB);
   }
 }
 
 void RISCVTargetLowering::AdjustInstrPostInstrSelection(MachineInstr &MI,
                                                         SDNode *Node) const {
   // Add FRM dependency to any instructions with dynamic rounding mode.
   unsigned Opc = MI.getOpcode();
   auto Idx = RISCV::getNamedOperandIdx(Opc, RISCV::OpName::frm);

llvm/test/CodeGen/RISCV/rv64-patchpoint.ll

+; RUN: llc -mtriple=riscv64 -debug-entry-values -enable-misched=0                             < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -debug-entry-values -enable-misched=0 -fast-isel -fast-isel-abort=1 < %s | FileCheck %s
+
+; Trivial patchpoint codegen
+;
+define i64 @trivial_patchpoint_codegen(i64 %p1, i64 %p2, i64 %p3, i64 %p4) {
+entry:
+
+; CHECK-LABEL: trivial_patchpoint_codegen:
+; CHECK:       addi sp, sp, -8
+; CHECK-NEXT:  sd ra, 0(sp)
+; CHECK-NEXT:  ori  ra, zero, 3562
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 3518
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 3836
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 2814
+; CHECK-NEXT:  jalr ra
+; CHECK-NEXT:  ld ra, 0(sp)
+; CHECK-NEXT:  addi sp, sp, 8
+
+; CHECK:       addi sp, sp, -8
+; CHECK-NEXT:  sd ra, 0(sp)
+; CHECK-NEXT:  ori  ra, zero, 3562
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 3518
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 3836
+; CHECK-NEXT:  slli ra, ra, 12
+; CHECK-NEXT:  ori  ra, ra, 2815
+; CHECK-NEXT:  jalr ra
+; CHECK-NEXT:  ld ra, 0(sp)
+; CHECK-NEXT:  addi sp, sp, 8
+; CHECK:       ret
+  %resolveCall2 = inttoptr i64 244837814094590 to i8*
+  %result = tail call i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 2, i32 20, i8* %resolveCall2, i32 4, i64 %p1, i64 %p2, i64 %p3, i64 %p4)
+  %resolveCall3 = inttoptr i64 244837814094591 to i8*
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 3, i32 20, i8* %resolveCall3, i32 2, i64 %p1, i64 %result)
+  ret i64 %result
+}
+
+; Test patchpoints reusing the same TargetConstant.
+; There is no way to verify this, since it depends on memory allocation.
+; But I think it's useful to include as a working example.
+define i64 @testLowerConstant(i64 %arg, i64 %tmp2, i64 %tmp10, i64* %tmp33, i64 %tmp79) {
+entry:
+  %tmp80 = add i64 %tmp79, -16
+  %tmp81 = inttoptr i64 %tmp80 to i64*
+  %tmp82 = load i64, i64* %tmp81, align 8
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64 14, i32 8, i64 %arg, i64 %tmp2, i64 %tmp10, i64 %tmp82)
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 15, i32 32, i8* null, i32 3, i64 %arg, i64 %tmp10, i64 %tmp82)
+  %tmp83 = load i64, i64* %tmp33, align 8
+  %tmp84 = add i64 %tmp83, -24
+  %tmp85 = inttoptr i64 %tmp84 to i64*
+  %tmp86 = load i64, i64* %tmp85, align 8
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64 17, i32 8, i64 %arg, i64 %tmp10, i64 %tmp86)
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 18, i32 32, i8* null, i32 3, i64 %arg, i64 %tmp10, i64 %tmp86)
+  ret i64 10
+}
+
+; Test small patchpoints that don't emit calls.
+define void @small_patchpoint_codegen(i64 %p1, i64 %p2, i64 %p3, i64 %p4) {
+entry:
+; CHECK-LABEL: small_patchpoint_codegen:
+; CHECK:      Ltmp
+; CHECK:      nop
+; CHECK-NEXT: nop
+; CHECK-NEXT: nop
+; CHECK-NEXT: nop
+; CHECK-NEXT: nop
+; CHECK-NEXT: ret
+  %result = tail call i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 5, i32 20, i8* null, i32 2, i64 %p1, i64 %p2)
+  ret void
+}
+
+declare void @llvm.experimental.stackmap(i64, i32, ...)
+declare void @llvm.experimental.patchpoint.void(i64, i32, i8*, i32, ...)
+declare i64 @llvm.experimental.patchpoint.i64(i64, i32, i8*, i32, ...)
+

llvm/test/CodeGen/RISCV/rv64-stackmap-nops.ll

+; RUN: llc < %s -mtriple=riscv64 | FileCheck %s
+
+define void @test_shadow_optimization() {
+entry:
+; CHECK-LABEL: test_shadow_optimization:
+; CHECK:      nop
+; CHECK-NEXT: nop
+; CHECK-NEXT: nop
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64  0, i32  16)
+  ret void
+}
+
+declare void @llvm.experimental.stackmap(i64, i32, ...)

llvm/test/CodeGen/RISCV/rv64-stackmap.ll

+; RUN: llc -mtriple=riscv64                             < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64 -fast-isel -fast-isel-abort=1 < %s | FileCheck %s
+;
+; Note: Print verbose stackmaps using -debug-only=stackmaps.
+
+; CHECK-LABEL:  .section  .llvm_stackmaps
+; CHECK-NEXT:  __LLVM_StackMaps:
+; Header
+; CHECK-NEXT:   .byte 3
+; CHECK-NEXT:   .byte 0
+; CHECK-NEXT:   .half  0
+; Num Functions
+; CHECK-NEXT:   .word 8
+; Num LargeConstants
+; CHECK-NEXT:   .word 2
+; Num Callsites
+; CHECK-NEXT:   .word 8
+
+; Functions and stack size
+; CHECK-NEXT:   .quad constantargs
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad osrinline
+; CHECK-NEXT:   .quad 32
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad osrcold
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad propertyRead
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad propertyWrite
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad jsVoidCall
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad jsIntCall
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad liveConstant
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+
+; Num LargeConstants
+; CHECK-NEXT:   .quad   4294967295
+; CHECK-NEXT:   .quad   4294967296
+
+; Constant arguments
+;
+; CHECK-NEXT:   .quad   1
+; CHECK-NEXT:   .word   .L{{.*}}-constantargs
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   4
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   65535
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   65536
+; SmallConstant
+; CHECK-NEXT:   .byte   5
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; LargeConstant at index 0
+; CHECK-NEXT:   .byte   5
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   1
+
+define void @constantargs() {
+entry:
+  %0 = inttoptr i64 244837814094590 to i8*
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 1, i32 20, i8* %0, i32 0, i64 65535, i64 65536, i64 4294967295, i64 4294967296)
+  ret void
+}
+
+; Inline OSR Exit
+;
+; CHECK:        .word   .L{{.*}}-osrinline
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+define void @osrinline(i64 %a, i64 %b) {
+entry:
+  ; Runtime void->void call.
+  call void inttoptr (i64 244837814094590 to void ()*)()
+  ; Followed by inline OSR patchpoint with 12-byte shadow and 2 live vars.
+  call void (i64, i32, ...) @llvm.experimental.stackmap(i64 3, i32 12, i64 %a, i64 %b)
+  ret void
+}
+
+; Cold OSR Exit
+;
+; 2 live variables in register.
+;
+; CHECK:        .word   .L{{.*}}-osrcold
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+define void @osrcold(i64 %a, i64 %b) {
+entry:
+  %test = icmp slt i64 %a, %b
+  br i1 %test, label %ret, label %cold
+cold:
+  ; OSR patchpoint with 12-byte nop-slide and 2 live vars.
+  %thunk = inttoptr i64 244837814094590 to i8*
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 4, i32 20, i8* %thunk, i32 0, i64 %a, i64 %b)
+  unreachable
+ret:
+  ret void
+}
+
+; Property Read
+; CHECK:        .word   .L{{.*}}-propertyRead
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+;
+; FIXME: There are currently no stackmap entries. After moving to
+; AnyRegCC, we will have entries for the object and return value.
+define i64 @propertyRead(i64* %obj) {
+entry:
+  %resolveRead = inttoptr i64 244837814094590 to i8*
+  %result = call i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 5, i32 20, i8* %resolveRead, i32 1, i64* %obj)
+  %add = add i64 %result, 3
+  ret i64 %add
+}
+
+; Property Write
+; CHECK:        .word   .L{{.*}}-propertyWrite
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+define void @propertyWrite(i64 %dummy1, i64* %obj, i64 %dummy2, i64 %a) {
+entry:
+  %resolveWrite = inttoptr i64 244837814094590 to i8*
+  call anyregcc void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 6, i32 20, i8* %resolveWrite, i32 2, i64* %obj, i64 %a)
+  ret void
+}
+
+; Void JS Call
+;
+; 2 live variables in registers.
+;
+; CHECK:        .word   .L{{.*}}-jsVoidCall
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+define void @jsVoidCall(i64 %dummy1, i64* %obj, i64 %arg, i64 %l1, i64 %l2) {
+entry:
+  %resolveCall = inttoptr i64 244837814094590 to i8*
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 7, i32 20, i8* %resolveCall, i32 2, i64* %obj, i64 %arg, i64 %l1, i64 %l2)
+  ret void
+}
+
+; i64 JS Call
+;
+; 2 live variables in registers.
+;
+; CHECK:        .word   .L{{.*}}-jsIntCall
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   {{[0-9]+}}
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   0
+define i64 @jsIntCall(i64 %dummy1, i64* %obj, i64 %arg, i64 %l1, i64 %l2) {
+entry:
+  %resolveCall = inttoptr i64 244837814094590 to i8*
+  %result = call i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 8, i32 20, i8* %resolveCall, i32 2, i64* %obj, i64 %arg, i64 %l1, i64 %l2)
+  %add = add i64 %result, 3
+  ret i64 %add
+}
+
+; Map a constant value.
+;
+; CHECK:        .word   .L{{.*}}-liveConstant
+; CHECK-NEXT:   .half   0
+; 1 location
+; CHECK-NEXT:   .half   1
+; Loc 0: SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half   8
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .half   0
+; CHECK-NEXT:   .word   33
+
+define void @liveConstant() {
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64 15, i32 8, i32 33)
+  ret void
+}
+
+declare void @llvm.experimental.stackmap(i64, i32, ...)
+declare void @llvm.experimental.patchpoint.void(i64, i32, i8*, i32, ...)
+declare i64 @llvm.experimental.patchpoint.i64(i64, i32, i8*, i32, ...)

llvm/test/CodeGen/RISCV/stackmap-frame-setup.ll

+; RUN: llc -o - -verify-machineinstrs -mtriple=riscv64 -stop-after machine-sink %s | FileCheck %s --check-prefix=ISEL
+; RUN: llc -o - -verify-machineinstrs -mtriple=riscv64 -fast-isel -fast-isel-abort=1 -stop-after machine-sink %s | FileCheck %s --check-prefix=FAST-ISEL
+
+define void @caller_meta_leaf() {
+entry:
+  %metadata = alloca i64, i32 3, align 8
+  store i64 11, i64* %metadata
+  store i64 12, i64* %metadata
+  store i64 13, i64* %metadata
+; ISEL:      ADJCALLSTACKDOWN 0, 0, implicit-def
+; ISEL-NEXT: STACKMAP
+; ISEL-NEXT: ADJCALLSTACKUP 0, 0, implicit-def
+  call void (i64, i32, ...) @llvm.experimental.stackmap(i64 4, i32 0, i64* %metadata)
+; FAST-ISEL:      ADJCALLSTACKDOWN 0, 0, implicit-def
+; FAST-ISEL-NEXT: STACKMAP
+; FAST-ISEL-NEXT: ADJCALLSTACKUP 0, 0, implicit-def
+  ret void
+}
+
+declare void @llvm.experimental.stackmap(i64, i32, ...)

llvm/test/CodeGen/RISCV/stackmap.ll

+; RUN: llc < %s -mtriple=riscv64 | FileCheck %s
+;
+; Note: Print verbose stackmaps using -debug-only=stackmaps.
+
+; CHECK-LABEL:  .section  .llvm_stackmaps
+; CHECK-NEXT:  __LLVM_StackMaps:
+; Header
+; CHECK-NEXT:   .byte 3
+; CHECK-NEXT:   .byte 0
+; CHECK-NEXT:   .half 0
+; Num Functions
+; CHECK-NEXT:   .word 12
+; Num LargeConstants
+; CHECK-NEXT:   .word 3
+; Num Callsites
+; CHECK-NEXT:   .word 16
+
+; Functions and stack size
+; CHECK-NEXT:   .quad constantargs
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad osrinline
+; CHECK-NEXT:   .quad 32
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad osrcold
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad propertyRead
+; CHECK-NEXT:   .quad 16
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad propertyWrite
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad jsVoidCall
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad jsIntCall
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad spilledValue
+; CHECK-NEXT:   .quad 144
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad liveConstant
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 1
+; CHECK-NEXT:   .quad directFrameIdx
+; CHECK-NEXT:   .quad 48
+; CHECK-NEXT:   .quad 2
+; CHECK-NEXT:   .quad longid
+; CHECK-NEXT:   .quad 0
+; CHECK-NEXT:   .quad 4
+; CHECK-NEXT:   .quad needsStackRealignment
+; CHECK-NEXT:   .quad -1
+; CHECK-NEXT:   .quad 1
+
+; Large Constants
+; CHECK-NEXT:   .quad   2147483648
+; CHECK-NEXT:   .quad   4294967295
+; CHECK-NEXT:   .quad   4294967296
+
+; Callsites
+; Constant arguments
+;
+; CHECK-NEXT:   .quad   1
+; CHECK-NEXT:   .word   .L{{.*}}-constantargs
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  12
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   -1
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   -1
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   65536
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   2000000000
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   2147483647
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   -1
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   -1
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; LargeConstant at index 0
+; CHECK-NEXT:   .byte   5
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; LargeConstant at index 1
+; CHECK-NEXT:   .byte   5
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   1
+; LargeConstant at index 2
+; CHECK-NEXT:   .byte   5
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   2
+; SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   -1
+
+define void @constantargs() {
+entry:
+  %0 = inttoptr i64 12345 to i8*
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 1, i32 16, i8* %0, i32 0, i16 65535, i16 -1, i32 65536, i32 2000000000, i32 2147483647, i32 -1, i32 4294967295, i32 4294967296, i64 2147483648, i64 4294967295, i64 4294967296, i64 -1)
+  ret void
+}
+
+; Inline OSR Exit
+;
+; CHECK-LABEL:  .word   .L{{.*}}-osrinline
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word  0
+define void @osrinline(i64 %a, i64 %b) {
+entry:
+  ; Runtime void->void call.
+  call void inttoptr (i64 -559038737 to void ()*)()
+  ; Followed by inline OSR patchpoint with 12-byte shadow and 2 live vars.
+  call void (i64, i32, ...) @llvm.experimental.stackmap(i64 3, i32 12, i64 %a, i64 %b)
+  ret void
+}
+
+; Cold OSR Exit
+;
+; 2 live variables in register.
+;
+; CHECK-LABEL:  .word   .L{{.*}}-osrcold
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+define void @osrcold(i64 %a, i64 %b) {
+entry:
+  %test = icmp slt i64 %a, %b
+  br i1 %test, label %ret, label %cold
+cold:
+  ; OSR patchpoint with 12-byte nop-slide and 2 live vars.
+  %thunk = inttoptr i64 3735928559 to i8*
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 4, i32 16, i8* %thunk, i32 0, i64 %a, i64 %b)
+  unreachable
+ret:
+  ret void
+}
+
+; Property Read
+; CHECK-LABEL:  .word   .L{{.*}}-propertyRead
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+define i64 @propertyRead(i64* %obj) {
+entry:
+  %resolveRead = inttoptr i64 3735928559 to i8*
+  %result = call anyregcc i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 5, i32 16, i8* %resolveRead, i32 1, i64* %obj)
+  %add = add i64 %result, 3
+  ret i64 %add
+}
+
+; Property Write
+; CHECK-LABEL:  .word   .L{{.*}}-propertyWrite
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+define void @propertyWrite(i64 %dummy1, i64* %obj, i64 %dummy2, i64 %a) {
+entry:
+  %resolveWrite = inttoptr i64 3735928559 to i8*
+  call anyregcc void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 6, i32 16, i8* %resolveWrite, i32 2, i64* %obj, i64 %a)
+  ret void
+}
+
+; Void JS Call
+;
+; 2 live variables in registers.
+;
+; CHECK-LABEL:  .word   .L{{.*}}-jsVoidCall
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+define void @jsVoidCall(i64 %dummy1, i64* %obj, i64 %arg, i64 %l1, i64 %l2) {
+entry:
+  %resolveCall = inttoptr i64 3735928559 to i8*
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 7, i32 16, i8* %resolveCall, i32 2, i64* %obj, i64 %arg, i64 %l1, i64 %l2)
+  ret void
+}
+
+; i64 JS Call
+;
+; 2 live variables in registers.
+;
+; CHECK-LABEL:  .word   .L{{.*}}-jsIntCall
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+; CHECK-NEXT:   .byte   1
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  {{[0-9]+}}
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   0
+define i64 @jsIntCall(i64 %dummy1, i64* %obj, i64 %arg, i64 %l1, i64 %l2) {
+entry:
+  %resolveCall = inttoptr i64 3735928559 to i8*
+  %result = call i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 8, i32 16, i8* %resolveCall, i32 2, i64* %obj, i64 %arg, i64 %l1, i64 %l2)
+  %add = add i64 %result, 3
+  ret i64 %add
+}
+
+; Spilled stack map values.
+;
+; Verify 28 stack map entries.
+;
+; CHECK-LABEL:  .word .L{{.*}}-spilledValue
+; CHECK-NEXT:   .half 0
+; CHECK-NEXT:   .half 28
+;
+; Check that at least one is a spilled entry from RBP.
+; Location: Indirect RBP + ...
+; CHECK:        .byte 3
+; CHECK-NEXT:   .byte 0
+; CHECK-NEXT:   .half 8
+; CHECK-NEXT:   .half 2
+; CHECK-NEXT:   .half 0
+; CHECK-NEXT:   .word
+define void @spilledValue(i64 %arg0, i64 %arg1, i64 %arg2, i64 %arg3, i64 %arg4, i64 %l0, i64 %l1, i64 %l2, i64 %l3, i64 %l4, i64 %l5, i64 %l6, i64 %l7, i64 %l8, i64 %l9, i64 %l10, i64 %l11, i64 %l12, i64 %l13, i64 %l14, i64 %l15, i64 %l16, i64 %l17, i64 %l18, i64 %l19, i64 %l20, i64 %l21, i64 %l22, i64 %l23, i64 %l24, i64 %l25, i64 %l26, i64 %l27) {
+entry:
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 11, i32 20, i8* null, i32 5, i64 %arg0, i64 %arg1, i64 %arg2, i64 %arg3, i64 %arg4, i64 %l0, i64 %l1, i64 %l2, i64 %l3, i64 %l4, i64 %l5, i64 %l6, i64 %l7, i64 %l8, i64 %l9, i64 %l10, i64 %l11, i64 %l12, i64 %l13, i64 %l14, i64 %l15, i64 %l16, i64 %l17, i64 %l18, i64 %l19, i64 %l20, i64 %l21, i64 %l22, i64 %l23, i64 %l24, i64 %l25, i64 %l26, i64 %l27)
+  ret void
+}
+
+; Map a constant value.
+;
+; CHECK-LABEL:  .word .L{{.*}}-liveConstant
+; CHECK-NEXT:   .half 0
+; 1 location
+; CHECK-NEXT:   .half 1
+; Loc 0: SmallConstant
+; CHECK-NEXT:   .byte   4
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word   33
+
+define void @liveConstant() {
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64 15, i32 4, i32 33)
+  ret void
+}
+
+; Directly map an alloca's address.
+;
+; Callsite 16
+; CHECK-LABEL:  .word .L{{.*}}-directFrameIdx
+; CHECK-NEXT:   .half 0
+; 1 location
+; CHECK-NEXT:   .half  1
+; Loc 0: Direct RBP - ofs
+; CHECK-NEXT:   .byte   2
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word
+
+; Callsite 17
+; CHECK-LABEL:  .word   .L{{.*}}-directFrameIdx
+; CHECK-NEXT:   .half  0
+; 2 locations
+; CHECK-NEXT:   .half  2
+; Loc 0: Direct RBP - ofs
+; CHECK-NEXT:   .byte   2
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word
+; Loc 1: Direct RBP - ofs
+; CHECK-NEXT:   .byte   2
+; CHECK-NEXT:   .byte   0
+; CHECK-NEXT:   .half  8
+; CHECK-NEXT:   .half  2
+; CHECK-NEXT:   .half  0
+; CHECK-NEXT:   .word
+define void @directFrameIdx() {
+entry:
+  %metadata1 = alloca i64, i32 3, align 8
+  store i64 11, i64* %metadata1
+  store i64 12, i64* %metadata1
+  store i64 13, i64* %metadata1
+  call void (i64, i32, ...) @llvm.experimental.stackmap(i64 16, i32 0, i64* %metadata1)
+  %metadata2 = alloca i8, i32 4, align 8
+  %metadata3 = alloca i16, i32 4, align 8
+  call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 17, i32 4, i8* null, i32 0, i8* %metadata2, i16* %metadata3)
+  ret void
+}
+
+; Test a 64-bit ID.
+;
+; CHECK:        .quad 4294967295
+; CHECK-LABEL:  .word .L{{.*}}-longid
+; CHECK:        .quad 4294967296
+; CHECK-LABEL:  .word .L{{.*}}-longid
+; CHECK:        .quad 9223372036854775807
+; CHECK-LABEL:  .word .L{{.*}}-longid
+; CHECK:        .quad -1
+; CHECK-LABEL:  .word .L{{.*}}-longid
+define void @longid() {
+entry:
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 4294967295, i32 0, i8* null, i32 0)
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 4294967296, i32 0, i8* null, i32 0)
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 9223372036854775807, i32 0, i8* null, i32 0)
+  tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 -1, i32 0, i8* null, i32 0)
+  ret void
+}
+
+; A stack frame which needs to be realigned at runtime (to meet alignment 
+; criteria for values on the stack) does not have a fixed frame size. 
+; CHECK-LABEL:  .word .L{{.*}}-needsStackRealignment
+; CHECK-NEXT:   .half 0
+; 0 locations
+; CHECK-NEXT:   .half 0
+define void @needsStackRealignment() {
+  %val = alloca i64, i32 3, align 128
+  tail call void (...) @escape_values(i64* %val)
+; Note: Adding any non-constant to the stackmap would fail because we
+; expected to be able to address off the frame pointer.  In a realigned
+; frame, we must use the stack pointer instead.  This is a separate bug.
+  tail call void (i64, i32, ...) @llvm.experimental.stackmap(i64 0, i32 0)
+  ret void
+}
+declare void @escape_values(...)
+
+declare void @llvm.experimental.stackmap(i64, i32, ...)
+declare void @llvm.experimental.patchpoint.void(i64, i32, i8*, i32, ...)
+declare i64 @llvm.experimental.patchpoint.i64(i64, i32, i8*, i32, ...)