[WIP] MIR Statepoint refactoring: Pass GC pointergs in VRegs.

[dantrushin]

Summary

In current implementation we spill Statepoint's GC arguments into
stack slots and then reload them at corresponding gc.relocate
location. That works reasonably well for small functions/GC sets,
but when register pressure gets high of GC pointer sets grows
large, this causes ugly code, because those spills are not understood
by the register allocator and we end up with reloads from spill
locations which immediately get spilled into statepoint slots.

This change insroduces new form of STATEPOINT instruction:
GC arguments are passed on VRegs and STATEPOINT itself becomes
variadic defs instruction, e.g.:

rel1,rel2,... = STATEPOINT ..., derived1<tied-def0>, derived2<tied-def1>, ...

Then register allocator can spill/fold them as it sees appropriate.
If target runtime/GC does not support GC support GC objects in
registers, FixupStatepointCallerSaves pass is extended to spill
them (optionally allowing these pointers to stay in CSRs).

Due to SDNode memory menagement, we use simple scheme to decide
which GC pointers pass in VRegs: just take first N.
For that to work, we sort GC derived pointers to have those not
needing relocation (e.g., constants) at the end of GC arguments
list.
N is further limited by maximal amount of tied registers machine
instruction can have (15 at the moment).

For more context/details about statepoints see documentation at
https://llvm.org/docs/Statepoints.html
https://llvm.org/docs/GarbageCollection.html

This review includes all changes and aimed to provide full context of
what's being done. It will be committed in smaller parts with separate
smaller reviews for every change.

[dantrushin]

Fix style and some comments.

[dantrushin]

Update to reflect current state of development

[dantrushin]

All pieces has been landed, so closing it.
Unfortunately, final result (e.g. ISEL part) diverged quite a bit from this, but idea at large has not changed

llvm/include/llvm/CodeGen/FunctionLoweringInfo.h

public:
   /// Keep track of frame indices allocated for statepoints as they could be
   /// used across basic block boundaries (e.g. for an invoke).  For each
   /// gc.statepoint instruction, maps uniqued llvm IR values to the slots they
   /// were spilled in.  If a value is mapped to None it means we visited the
   /// value but didn't spill it (because it was a constant, for instance).
   using StatepointSpillMapTy = DenseMap<const Value *, Optional<int>>;
   DenseMap<const Instruction *, StatepointSpillMapTy> StatepointSpillMaps;
 
+  /// For each statepoint keep virtual registers its result values has
+  /// been exported to. This is similar to ValueMap, but for statepoint
+  /// results which has no existing IR Value.
+  using StatepointRegMap = DenseMap<const Value *, Register>;
+  DenseMap<const Instruction *, StatepointRegMap> StatepointRegs;
+
   /// StaticAllocaMap - Keep track of frame indices for fixed sized allocas in
   /// the entry block.  This allows the allocas to be efficiently referenced
   /// anywhere in the function.
   DenseMap<const AllocaInst*, int> StaticAllocaMap;
 
   /// ByValArgFrameIndexMap - Keep track of frame indices for byval arguments.
   DenseMap<const Argument*, int> ByValArgFrameIndexMap;
 

llvm/include/llvm/CodeGen/StackMaps.h

class StatepointOpers {
   // instruction.
   enum { IDPos, NBytesPos, NCallArgsPos, CallTargetPos, MetaEnd };
 
   // These values are relative offsets from the start of the statepoint meta
   // arguments (i.e. the end of the call arguments).
   enum { CCOffset = 1, FlagsOffset = 3, NumDeoptOperandsOffset = 5 };
 
 public:
-  explicit StatepointOpers(const MachineInstr *MI) : MI(MI) {}
+  explicit StatepointOpers(const MachineInstr *MI) : MI(MI) {
+    NumDefs = MI->getNumDefs();
+  }
 
   /// Get index of statepoint ID operand.
-  unsigned getIDPos() const { return IDPos; }
+  unsigned getIDPos() const { return NumDefs + IDPos; }
 
   /// Get index of Num Patch Bytes operand.
-  unsigned getNBytesPos() const { return NBytesPos; }
+  unsigned getNBytesPos() const { return NumDefs + NBytesPos; }
 
   /// Get index of Num Call Arguments operand.
-  unsigned getNCallArgsPos() const { return NCallArgsPos; }
+  unsigned getNCallArgsPos() const { return NumDefs + NCallArgsPos; }
 
   /// Get starting index of non call related arguments
   /// (calling convention, statepoint flags, vm state and gc state).
   unsigned getVarIdx() const {
-    return MI->getOperand(NCallArgsPos).getImm() + MetaEnd;
+    return MI->getOperand(NumDefs + NCallArgsPos).getImm() + MetaEnd + NumDefs;
   }
 
   /// Get index of Calling Convention operand.
   unsigned getCCIdx() const { return getVarIdx() + CCOffset; }
 
   /// Get index of Flags operand.
   unsigned getFlagsIdx() const { return getVarIdx() + FlagsOffset; }
 
   /// Get index of Number Deopt Arguments operand.
   unsigned getNumDeoptArgsIdx() const {
     return getVarIdx() + NumDeoptOperandsOffset;
   }
 
   /// Return the ID for the given statepoint.
-  uint64_t getID() const { return MI->getOperand(IDPos).getImm(); }
+  uint64_t getID() const { return MI->getOperand(NumDefs + IDPos).getImm(); }
 
   /// Return the number of patchable bytes the given statepoint should emit.
   uint32_t getNumPatchBytes() const {
-    return MI->getOperand(NBytesPos).getImm();
+    return MI->getOperand(NumDefs + NBytesPos).getImm();
   }
 
   /// Return the target of the underlying call.
   const MachineOperand &getCallTarget() const {
-    return MI->getOperand(CallTargetPos);
+    return MI->getOperand(NumDefs + CallTargetPos);
   }
 
   /// Return the calling convention.
   CallingConv::ID getCallingConv() const {
     return MI->getOperand(getCCIdx()).getImm();
   }
 
   /// Return the statepoint flags.
   uint64_t getFlags() const { return MI->getOperand(getFlagsIdx()).getImm(); }
 
 private:
   const MachineInstr *MI;
+  unsigned NumDefs;
 };
 
 class StackMaps {
 public:
   struct Location {
     enum LocationType {
       Unprocessed,
       Register,

llvm/include/llvm/Target/Target.td

def PATCHPOINT : StandardPseudoInstruction {
   let InOperandList = (ins i64imm:$id, i32imm:$nbytes, unknown:$callee,
                        i32imm:$nargs, i32imm:$cc, variable_ops);
   let hasSideEffects = 1;
   let isCall = 1;
   let mayLoad = 1;
   let usesCustomInserter = 1;
 }
 def STATEPOINT : StandardPseudoInstruction {
-  let OutOperandList = (outs);
+  let OutOperandList = (outs variable_ops);
   let InOperandList = (ins variable_ops);
   let usesCustomInserter = 1;
   let mayLoad = 1;
   let mayStore = 1;
   let hasSideEffects = 1;
   let isCall = 1;
 }
 def LOAD_STACK_GUARD : StandardPseudoInstruction {

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

 STATISTIC(NumSpillSlotsAllocated, "Number of spill slots allocated");
 STATISTIC(NumSpillSlotsExtended, "Number of spill slots extended");
 
 static cl::opt<bool> FixupSCSExtendSlotSize(
     "fixup-scs-extend-slot-size", cl::Hidden, cl::init(false),
     cl::desc("Allow spill in spill slot of greater size than register size"),
     cl::Hidden);
 
+static cl::opt<bool> PassGCPtrInCSR(
+    "fixup-allow-gcptr-in-csr", cl::Hidden, cl::init(false),
+    cl::desc("Allow passing GC Pointer arguments in callee saved registers"));
+
+static cl::opt<bool> EnableCopyProp(
+    "fixup-scs-enable-copy-propagation", cl::Hidden, cl::init(true),
+    cl::desc("Enable simple copy propagation during register reloading"));
+
+// This is purely debugging option.
+// It may be handy for investigating statepoint spilling issues.
+static cl::opt<unsigned> MaxStatepointsWithRegs(
+    "fixup-max-csr-statepoints", cl::Hidden,
+    cl::desc("Max number of statepoints allowed to pass GC Ptrs in registers"));
+
 namespace {
 
 class FixupStatepointCallerSaved : public MachineFunctionPass {
 public:
   static char ID;
 
   FixupStatepointCallerSaved() : MachineFunctionPass(ID) {
     initializeFixupStatepointCallerSavedPass(*PassRegistry::getPassRegistry());
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     MachineFunctionPass::getAnalysisUsage(AU);
   }
 
   StringRef getPassName() const override {
     return "Fixup Statepoint Caller Saved";
   }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
+
 };
+
 } // End anonymous namespace.
 
 char FixupStatepointCallerSaved::ID = 0;
 char &llvm::FixupStatepointCallerSavedID = FixupStatepointCallerSaved::ID;
 
 INITIALIZE_PASS_BEGIN(FixupStatepointCallerSaved, DEBUG_TYPE,
                       "Fixup Statepoint Caller Saved", false, false)
 INITIALIZE_PASS_END(FixupStatepointCallerSaved, DEBUG_TYPE,
                     "Fixup Statepoint Caller Saved", false, false)
 
 // Utility function to get size of the register.
 static unsigned getRegisterSize(const TargetRegisterInfo &TRI, Register Reg) {
   const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
   return TRI.getSpillSize(*RC);
 }
 
+// Advance iterator to the next stack map entry
+static MachineInstr::const_mop_iterator
+advanceToNextStackMapElt(MachineInstr::const_mop_iterator MOI) {
+  if (MOI->isImm()) {
+    switch (MOI->getImm()) {
+    default:
+      llvm_unreachable("Unrecognized operand type.");
+    case StackMaps::DirectMemRefOp:
+      MOI += 2; // <Reg>, <Imm>
+      break;
+    case StackMaps::IndirectMemRefOp:
+      MOI += 3; // <Size>, <Reg>, <Imm>
+      break;
+    case StackMaps::ConstantOp:
+      MOI += 1;
+      break;
+    }
+  }
+  return ++MOI;
+}
+
+// Return statepoint GC args as a set
+static SmallSet<Register, 8> collectGCRegs(MachineInstr &MI) {
+  StatepointOpers SO(&MI);
+  unsigned NumDeoptIdx = SO.getNumDeoptArgsIdx();
+  unsigned NumDeoptArgs = MI.getOperand(NumDeoptIdx).getImm();
+  MachineInstr::const_mop_iterator MOI(MI.operands_begin() + NumDeoptIdx + 1),
+      MOE(MI.operands_end());
+
+  // Skip deopt args
+  for (unsigned i = 0; i < NumDeoptArgs; ++i)
+    MOI = advanceToNextStackMapElt(MOI);
+
+  SmallSet<Register, 8> Result;
+  while (MOI != MOE) {
+    if (MOI->isReg() && !MOI->isImplicit())
+      Result.insert(MOI->getReg());
+    MOI = advanceToNextStackMapElt(MOI);
+  }
+  return Result;
+}
+
+// Try to eliminate redundant copy to register which we're going to
+// spill, i.e. try to change:
+//    X = COPY Y
+//    SPILL X
+//  to
+//    SPILL Y
+//  If there are no uses of X between copy and STATEPOINT, that COPY
+//  may be eliminated.
+//  Reg - register we're about to spill
+//  RI - On entry points to statepoint.
+//       On successful copy propagation set to new spill point.
+//  IsKill - set to true if COPY is Kill (there are no uses of Y)
+//  Returns either found source copy register or original one.
+static Register performCopyPropagation(Register Reg,
+                                       MachineBasicBlock::iterator &RI,
+                                       bool &IsKill, const TargetInstrInfo &TII,
+                                       const TargetRegisterInfo &TRI) {
+  if (!EnableCopyProp)
+    return Reg;
+
+  MachineBasicBlock *MBB = RI->getParent();
+  MachineBasicBlock::reverse_iterator E = MBB->rend();
+  MachineInstr *Def = nullptr, *Use = nullptr;
+  for (auto It = ++(RI.getReverse()); It != E; ++It) {
+    if (It->readsRegister(Reg, &TRI) && !Use)
+      Use = &*It;
+    if (It->modifiesRegister(Reg, &TRI)) {
+      Def = &*It;
+      break;
+    }
+  }
+  if (!Def)
+    return Reg;
+
+  auto DestSrc = TII.isCopyInstr(*Def);
+  if (!DestSrc || DestSrc->Destination->getReg() != Reg)
+    return Reg;
+
+  Register SrcReg = DestSrc->Source->getReg();
+
+  if (getRegisterSize(TRI, Reg) != getRegisterSize(TRI, SrcReg))
+    return Reg;
+
+  LLVM_DEBUG(dbgs() << "spillRegisters: perform copy propagation "
+                    << printReg(Reg, &TRI) << " -> " << printReg(SrcReg, &TRI)
+                    << "\n");
+
+  // Insert spill immediately after Def
+  RI = ++MachineBasicBlock::iterator(Def);
+  IsKill = DestSrc->Source->isKill();
+
+  // There are no uses of original register between COPY and STATEPOINT.
+  // There can't be any after STATEPOINT, so we can eliminate Def.
+  if (!Use) {
+    LLVM_DEBUG(dbgs() << "spillRegisters: removing dead copy " << *Def);
+    Def->eraseFromParent();
+  }
+  return SrcReg;
+}
+
 namespace {
 // Cache used frame indexes during statepoint re-write to re-use them in
 // processing next statepoint instruction.
 // Two strategies. One is to preserve the size of spill slot while another one
 // extends the size of spill slots to reduce the number of them, causing
 // the less total frame size. But unspill will have "implicit" any extend.
 class FrameIndexesCache {
 private:
   struct FrameIndexesPerSize {
     // List of used frame indexes during processing previous statepoints.
     SmallVector<int, 8> Slots;
     // Current index of un-used yet frame index.
     unsigned Index = 0;
   };
   MachineFrameInfo &MFI;
   const TargetRegisterInfo &TRI;
   // Map size to list of frame indexes of this size. If the mode is
   // FixupSCSExtendSlotSize then the key 0 is used to keep all frame indexes.
   // If the size of required spill slot is greater than in a cache then the
   // size will be increased.
   DenseMap<unsigned, FrameIndexesPerSize> Cache;
 
+  // Keeps track of slots reserved for the shared landing pad processing.
+  // Initialized from GlobalIndices for the current EHPad.
+  SmallSet<int, 8> ReservedSlots;
+
+  // Landing pad can be destination of several statepoints. Every register
+  // defined by such statepoints must be spilled to the same stack slot.
+  // This map keeps that information.
+  using RegSlotPair = std::pair<Register, int>;
+  DenseMap<const MachineBasicBlock *, SmallVector<RegSlotPair, 8>> GlobalIndices;
+
+  FrameIndexesPerSize &getCacheBucket(unsigned Size) {
+    // In FixupSCSExtendSlotSize mode the bucket with 0 index is used
+    // for all sizes.
+    return Cache[FixupSCSExtendSlotSize ? 0 : Size];
+  }
+
 public:
   FrameIndexesCache(MachineFrameInfo &MFI, const TargetRegisterInfo &TRI)
       : MFI(MFI), TRI(TRI) {}
   // Reset the current state of used frame indexes. After invocation of
   // this function all frame indexes are available for allocation.
   void reset() {
     for (auto &It : Cache)
       It.second.Index = 0;
   }
+
+  // Reset the FI cache and reserve slots assigned for shared landing pads
+  // processing.
+  void resetAndPreFill(const MachineBasicBlock *EHPad) {
+    reset();
+
+    if (EHPad && GlobalIndices.count(EHPad))
+      for (auto &RSP : GlobalIndices[EHPad])
+        ReservedSlots.insert(RSP.second);
+  }
+
   // Get frame index to spill the register.
-  int getFrameIndex(Register Reg) {
+  int getFrameIndex(Register Reg, MachineBasicBlock *EHPad) {
+    // Check if slot for Reg is already reserved at EHPad.
+    auto It = GlobalIndices.find(EHPad);
+    if (It != GlobalIndices.end()) {
+      auto &Vec = It->second;
+      auto Idx = llvm::find_if(Vec, [Reg](RegSlotPair &RSP) {
+                               return Reg == RSP.first;
+                               });
+      if (Idx != Vec.end()) {
+        int FI = Idx->second;
+        LLVM_DEBUG(dbgs() << "Found global FI " << FI << " for register "
+                          << printReg(Reg, &TRI) << " at "
+                          << printMBBReference(*EHPad));
+        return FI;
+      }
+    }
+
     unsigned Size = getRegisterSize(TRI, Reg);
-    // In FixupSCSExtendSlotSize mode the bucket with 0 index is used
-    // for all sizes.
-    unsigned Bucket = FixupSCSExtendSlotSize ? 0 : Size;
-    FrameIndexesPerSize &Line = Cache[Bucket];
-    if (Line.Index < Line.Slots.size()) {
+    FrameIndexesPerSize &Line = getCacheBucket(Size);
+    while (Line.Index < Line.Slots.size()) {
       int FI = Line.Slots[Line.Index++];
+      if (ReservedSlots.count(FI))
+        continue;
       // If all sizes are kept together we probably need to extend the
       // spill slot size.
       if (MFI.getObjectSize(FI) < Size) {
         MFI.setObjectSize(FI, Size);
         MFI.setObjectAlignment(FI, Align(Size));
         NumSpillSlotsExtended++;
       }
       return FI;
     }
     int FI = MFI.CreateSpillStackObject(Size, Align(Size));
     NumSpillSlotsAllocated++;
     Line.Slots.push_back(FI);
     ++Line.Index;
+
+    // Remember assignment {Reg, FI} for EHPad
+    if (EHPad) {
+      GlobalIndices[EHPad].push_back(std::make_pair(Reg, FI));
+      LLVM_DEBUG(dbgs() << "Reserved slot " << FI << " for spilling reg"
+                        << printReg(Reg, &TRI) << " at landing pad "
+                        << printMBBReference(*EHPad) << "\n");
+    }
+
     return FI;
   }
+
   // Sort all registers to spill in descendent order. In the
   // FixupSCSExtendSlotSize mode it will minimize the total frame size.
   // In non FixupSCSExtendSlotSize mode we can skip this step.
   void sortRegisters(SmallVectorImpl<Register> &Regs) {
     if (!FixupSCSExtendSlotSize)
       return;
     llvm::sort(Regs.begin(), Regs.end(), [&](Register &A, Register &B) {
       return getRegisterSize(TRI, A) > getRegisterSize(TRI, B);
     });
   }
 };
 
+// Check if we already inserted reload of register Reg from spill slot FI
+// in basic block MBB.
+// This can happen in EH pad block which is successor of several
+// statepoints.
+static bool hasRegReload(Register Reg, int FI, MachineBasicBlock *MBB,
+                         const TargetInstrInfo *TII,
+                         const TargetRegisterInfo *TRI) {
+  auto I = MBB->SkipPHIsLabelsAndDebug(MBB->begin()), E = MBB->end();
+  int Dummy;
+  for (; I != E; ++I) {
+    if (TII->isLoadFromStackSlot(*I, Dummy) == Reg && Dummy == FI)
+      return true;
+    if (I->modifiesRegister(Reg, TRI) || I->readsRegister(Reg, TRI))
+      return false;
+  }
+  return false;
+}
+
 // Describes the state of the current processing statepoint instruction.
 class StatepointState {
 private:
   // statepoint instruction.
   MachineInstr &MI;
   MachineFunction &MF;
+  // If non-null then statepoint is invoke, and this points to the landing pad.
+  MachineBasicBlock *EHPad;
   const TargetRegisterInfo &TRI;
   const TargetInstrInfo &TII;
   MachineFrameInfo &MFI;
   // Mask with callee saved registers.
   const uint32_t *Mask;
   // Cache of frame indexes used on previous instruction processing.
   FrameIndexesCache &CacheFI;
+  bool AllowGCPtrInCSR;
   // Operands with physical registers requiring spilling.
   SmallVector<unsigned, 8> OpsToSpill;
   // Set of register to spill.
   SmallVector<Register, 8> RegsToSpill;
+  // Set of registers to reload after statepoint.
+  SmallVector<Register, 8> RegsToReload;
   // Map Register to Frame Slot index.
   DenseMap<Register, int> RegToSlotIdx;
 
 public:
   StatepointState(MachineInstr &MI, const uint32_t *Mask,
-                  FrameIndexesCache &CacheFI)
+                  FrameIndexesCache &CacheFI, bool AllowGCPtrInCSR)
       : MI(MI), MF(*MI.getMF()), TRI(*MF.getSubtarget().getRegisterInfo()),
         TII(*MF.getSubtarget().getInstrInfo()), MFI(MF.getFrameInfo()),
-        Mask(Mask), CacheFI(CacheFI) {}
+        Mask(Mask), CacheFI(CacheFI), AllowGCPtrInCSR(AllowGCPtrInCSR) {
+
+          // Find statepoint's landing pad, if any.
+          EHPad = nullptr;
+          MachineBasicBlock *MBB=MI.getParent();
+          // Invoke statepoint must be last one in block.
+          bool Last =
+            std::none_of(++MI.getIterator(), MBB->end().getInstrIterator(), [](MachineInstr &I) {
+                         return I.getOpcode() == TargetOpcode::STATEPOINT;
+                         });
+          if (Last) {
+            auto It = llvm::find_if(MBB->successors(), [](MachineBasicBlock *B) {
+                                    return B->isEHPad();
+                                    });
+            if (It != MBB->succ_end())
+              EHPad = *It;
+          }
+        }
+
+  MachineBasicBlock *getEHPad() const { return EHPad; }
+
   // Return true if register is callee saved.
   bool isCalleeSaved(Register Reg) { return (Mask[Reg / 32] >> Reg % 32) & 1; }
+
   // Iterates over statepoint meta args to find caller saver registers.
   // Also cache the size of found registers.
   // Returns true if caller save registers found.
   bool findRegistersToSpill() {
     SmallSet<Register, 8> VisitedRegs;
+    SmallSet<Register, 8> GCRegs = collectGCRegs(MI);
     for (unsigned Idx = StatepointOpers(&MI).getVarIdx(),
                   EndIdx = MI.getNumOperands();
          Idx < EndIdx; ++Idx) {
       MachineOperand &MO = MI.getOperand(Idx);
       if (!MO.isReg() || MO.isImplicit())
         continue;
       Register Reg = MO.getReg();
       assert(Reg.isPhysical() && "Only physical regs are expected");
-      if (isCalleeSaved(Reg))
+
+      if (isCalleeSaved(Reg) && (AllowGCPtrInCSR || !is_contained(GCRegs, Reg)))
         continue;
+
+      LLVM_DEBUG(dbgs() << "Will spill " << printReg(Reg, &TRI) << " at index "
+                        << Idx << "\n");
+
       if (VisitedRegs.insert(Reg).second)
         RegsToSpill.push_back(Reg);
       OpsToSpill.push_back(Idx);
     }
     CacheFI.sortRegisters(RegsToSpill);
     return !RegsToSpill.empty();
   }
+
   // Spill all caller saved registers right before statepoint instruction.
   // Remember frame index where register is spilled.
   void spillRegisters() {
     for (Register Reg : RegsToSpill) {
-      int FI = CacheFI.getFrameIndex(Reg);
+      int FI = CacheFI.getFrameIndex(Reg, EHPad);
       const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
-      TII.storeRegToStackSlot(*MI.getParent(), MI, Reg, true /*is_Kill*/, FI,
-                              RC, &TRI);
+
       NumSpilledRegisters++;
       RegToSlotIdx[Reg] = FI;
+
+      LLVM_DEBUG(dbgs() << "Spilling " << printReg(Reg, &TRI) << " to FI " << FI
+                        << "\n");
+
+      // Perform trivial copy propagation
+      bool IsKill = true;
+      MachineBasicBlock::iterator InsertBefore(MI);
+      Reg = performCopyPropagation(Reg, InsertBefore, IsKill, TII, TRI);
+
+      LLVM_DEBUG(dbgs() << "Insert spill before " << *InsertBefore);
+      TII.storeRegToStackSlot(*MI.getParent(), InsertBefore, Reg, IsKill, FI,
+                              RC, &TRI);
     }
   }
+
+  void insertReloadBefore(unsigned Reg, MachineBasicBlock::iterator It,
+                          MachineBasicBlock *MBB) {
+    const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
+    int FI = RegToSlotIdx[Reg];
+    if (It != MBB->end()) {
+      TII.loadRegFromStackSlot(*MBB, It, Reg, FI, RC, &TRI);
+      return;
+    }
+
+    // To insert reload at the end of MBB, insert it before last instruction
+    // and then swap them.
+    assert(MBB->begin() != MBB->end() && "Empty block");
+    --It;
+    TII.loadRegFromStackSlot(*MBB, It, Reg, FI, RC, &TRI);
+    MachineInstr *Reload = It->getPrevNode();
+    int Dummy = 0;
+    assert(TII.isLoadFromStackSlot(*Reload, Dummy) == Reg);
+    assert(Dummy == FI);
+    MBB->remove(Reload);
+    MBB->insertAfter(It, Reload);
+  }
+
+  // Insert reloads of (relocated) registers spilled in statepoint.
+  void insertReloads(MachineInstr *NewStatepoint) {
+    MachineBasicBlock *MBB = NewStatepoint->getParent();
+    auto InsertPoint = std::next(NewStatepoint->getIterator());
+
+    for (auto Reg : RegsToReload) {
+      insertReloadBefore(Reg, InsertPoint, MBB);
+      LLVM_DEBUG(dbgs() << "Reloading " << printReg(Reg, &TRI) << " from FI "
+                        << RegToSlotIdx[Reg] << " after statepoint\n");
+
+      if (EHPad && !hasRegReload(Reg, RegToSlotIdx[Reg], EHPad, &TII, &TRI)) {
+        auto EHPadInsertPoint = EHPad->SkipPHIsLabelsAndDebug(EHPad->begin());
+        insertReloadBefore(Reg, EHPadInsertPoint, EHPad);
+        LLVM_DEBUG(dbgs() << "...also reload at EHPad "
+                          << printMBBReference(*EHPad) << "\n");
+      }
+    }
+  }
+
   // Re-write statepoint machine instruction to replace caller saved operands
   // with indirect memory location (frame index).
-  void rewriteStatepoint() {
+  MachineInstr *rewriteStatepoint() {
     MachineInstr *NewMI =
         MF.CreateMachineInstr(TII.get(MI.getOpcode()), MI.getDebugLoc(), true);
     MachineInstrBuilder MIB(MF, NewMI);
 
+    unsigned NumOps = MI.getNumOperands();
+
+    // New indices for the remaining defs.
+    SmallVector<unsigned, 8> NewIndices;
+    unsigned NumDefs = MI.getNumDefs();
+    for (unsigned I = 0; I < NumDefs; ++I) {
+      MachineOperand &DefMO = MI.getOperand(I);
+      assert(DefMO.isReg() && DefMO.isDef() && "Expected Reg Def operand");
+      Register Reg = DefMO.getReg();
+      if (!AllowGCPtrInCSR) {
+        assert(is_contained(RegsToSpill, Reg));
+        RegsToReload.push_back(Reg);
+      } else {
+        if (isCalleeSaved(Reg)) {
+          NewIndices.push_back(NewMI->getNumOperands());
+          MIB.addReg(Reg, RegState::Define);
+        } else {
+          NewIndices.push_back(NumOps);
+          RegsToReload.push_back(Reg);
+        }
+      }
+    }
+
     // Add End marker.
     OpsToSpill.push_back(MI.getNumOperands());
     unsigned CurOpIdx = 0;
 
-    for (unsigned I = 0; I < MI.getNumOperands(); ++I) {
+    for (unsigned I = NumDefs; I < MI.getNumOperands(); ++I) {
       MachineOperand &MO = MI.getOperand(I);
       if (I == OpsToSpill[CurOpIdx]) {
         int FI = RegToSlotIdx[MO.getReg()];
         MIB.addImm(StackMaps::IndirectMemRefOp);
         MIB.addImm(getRegisterSize(TRI, MO.getReg()));
         assert(MO.isReg() && "Should be register");
         assert(MO.getReg().isPhysical() && "Should be physical register");
         MIB.addFrameIndex(FI);
         MIB.addImm(0);
         ++CurOpIdx;
-      } else
+      } else {
         MIB.add(MO);
+        unsigned OldDef;
+        if (AllowGCPtrInCSR && MI.isRegTiedToDefOperand(I, &OldDef)) {
+          assert(OldDef < NumDefs);
+          assert(NewIndices[OldDef] < NumOps);
+          MIB->tieOperands(NewIndices[OldDef], MIB->getNumOperands() - 1);
+        }
+      }
     }
     assert(CurOpIdx == (OpsToSpill.size() - 1) && "Not all operands processed");
     // Add mem operands.
     NewMI->setMemRefs(MF, MI.memoperands());
     for (auto It : RegToSlotIdx) {
       int FrameIndex = It.second;
       auto PtrInfo = MachinePointerInfo::getFixedStack(MF, FrameIndex);
       auto *MMO = MF.getMachineMemOperand(PtrInfo, MachineMemOperand::MOLoad,
                                           getRegisterSize(TRI, It.first),
                                           MFI.getObjectAlign(FrameIndex));
       NewMI->addMemOperand(MF, MMO);
     }
+
     // Insert new statepoint and erase old one.
     MI.getParent()->insert(MI, NewMI);
+
+    LLVM_DEBUG(dbgs() << "rewritten statepoint to : " << *NewMI << "\n");
     MI.eraseFromParent();
+    return NewMI;
   }
 };
 
 class StatepointProcessor {
 private:
   MachineFunction &MF;
   const TargetRegisterInfo &TRI;
   FrameIndexesCache CacheFI;
 
 public:
   StatepointProcessor(MachineFunction &MF)
       : MF(MF), TRI(*MF.getSubtarget().getRegisterInfo()),
         CacheFI(MF.getFrameInfo(), TRI) {}
 
-  bool process(MachineInstr &MI) {
+  bool process(MachineInstr &MI, bool AllowGCPtrInCSR) {
     StatepointOpers SO(&MI);
     uint64_t Flags = SO.getFlags();
     // Do nothing for LiveIn, it supports all registers.
     if (Flags & (uint64_t)StatepointFlags::DeoptLiveIn)
       return false;
+    LLVM_DEBUG(dbgs() << "\nMBB " << MI.getParent()->getNumber() << " "
+                      << MI.getParent()->getName() << " : process statepoint "
+                      << MI);
     CallingConv::ID CC = SO.getCallingConv();
     const uint32_t *Mask = TRI.getCallPreservedMask(MF, CC);
-    CacheFI.reset();
-    StatepointState SS(MI, Mask, CacheFI);
+    StatepointState SS(MI, Mask, CacheFI, AllowGCPtrInCSR);
+    CacheFI.resetAndPreFill(SS.getEHPad());
 
     if (!SS.findRegistersToSpill())
       return false;
 
     SS.spillRegisters();
-    SS.rewriteStatepoint();
+    auto *NewStatepoint = SS.rewriteStatepoint();
+    SS.insertReloads(NewStatepoint);
     return true;
   }
 };
 } // namespace
 
 bool FixupStatepointCallerSaved::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   const Function &F = MF.getFunction();
   if (!F.hasGC())
     return false;
 
   SmallVector<MachineInstr *, 16> Statepoints;
   for (MachineBasicBlock &BB : MF)
     for (MachineInstr &I : BB)
       if (I.getOpcode() == TargetOpcode::STATEPOINT)
         Statepoints.push_back(&I);
 
   if (Statepoints.empty())
     return false;
 
   bool Changed = false;
   StatepointProcessor SPP(MF);
-  for (MachineInstr *I : Statepoints)
-    Changed |= SPP.process(*I);
+  unsigned NumStatepoints = 0;
+  bool AllowGCPtrInCSR = PassGCPtrInCSR;
+  for (MachineInstr *I : Statepoints) {
+    ++NumStatepoints;
+    if (MaxStatepointsWithRegs.getNumOccurrences() &&
+        NumStatepoints >= MaxStatepointsWithRegs)
+      AllowGCPtrInCSR = false;
+    Changed |= SPP.process(*I, AllowGCPtrInCSR);
+  }
   return Changed;
 }

llvm/lib/CodeGen/InlineSpiller.cpp

foldMemoryOperand(ArrayRef<std::pair<MachineInstr *, unsigned>> Ops,
   // Don't attempt folding in bundles.
   MachineInstr *MI = Ops.front().first;
   if (Ops.back().first != MI || MI->isBundled())
     return false;
 
   bool WasCopy = MI->isCopy();
   Register ImpReg;
 
+  bool UntieRegs = MI->getOpcode() == TargetOpcode::STATEPOINT;
+
   // Spill subregs if the target allows it.
   // We always want to spill subregs for stackmap/patchpoint pseudos.
   bool SpillSubRegs = TII.isSubregFoldable() ||
                       MI->getOpcode() == TargetOpcode::STATEPOINT ||
                       MI->getOpcode() == TargetOpcode::PATCHPOINT ||
                       MI->getOpcode() == TargetOpcode::STACKMAP;
 
   // TargetInstrInfo::foldMemoryOperand only expects explicit, non-tied
   // operands.
   SmallVector<unsigned, 8> FoldOps;
   for (const auto &OpPair : Ops) {
     unsigned Idx = OpPair.second;
     assert(MI == OpPair.first && "Instruction conflict during operand folding");
     MachineOperand &MO = MI->getOperand(Idx);
     if (MO.isImplicit()) {
       ImpReg = MO.getReg();
       continue;
     }
 
+    if (UntieRegs && MO.isTied())
+      MI->untieRegOperand(Idx);
+
     if (!SpillSubRegs && MO.getSubReg())
       return false;
     // We cannot fold a load instruction into a def.
     if (LoadMI && MO.isDef())
       return false;
     // Tied use operands should not be passed to foldMemoryOperand.
     if (!MI->isRegTiedToDefOperand(Idx))
       FoldOps.push_back(Idx);

llvm/lib/CodeGen/MachineVerifier.cpp

 void
 MachineVerifier::visitMachineOperand(const MachineOperand *MO, unsigned MONum) {
   const MachineInstr *MI = MO->getParent();
   const MCInstrDesc &MCID = MI->getDesc();
   unsigned NumDefs = MCID.getNumDefs();
   if (MCID.getOpcode() == TargetOpcode::PATCHPOINT)
     NumDefs = (MONum == 0 && MO->isReg()) ? NumDefs : 0;
 
+  if (MCID.getOpcode() == TargetOpcode::STATEPOINT)
+    NumDefs = MI->getNumDefs();
+
   // The first MCID.NumDefs operands must be explicit register defines
   if (MONum < NumDefs) {
     const MCOperandInfo &MCOI = MCID.OpInfo[MONum];
     if (!MO->isReg())
       report("Explicit definition must be a register", MO, MONum);
     else if (!MO->isDef() && !MCOI.isOptionalDef())
       report("Explicit definition marked as use", MO, MONum);
     else if (MO->isImplicit())

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

if (RegisterSDNode *RN = dyn_cast<RegisterSDNode>(Node->getOperand(I - 1)))
         continue;
     NumImpUses = N - I;
     break;
   }
 
   return N;
 }
 
+/// Return starting index of GC operand list.
+// FIXME: need a better place for this. Put it in StackMaps?
+static unsigned getStatepointGCArgStartIdx(MachineInstr *MI) {
+  assert(MI->getOpcode() == TargetOpcode::STATEPOINT &&
+         "STATEPOINT node expected");
+  unsigned OperIdx = StatepointOpers(MI).getNumDeoptArgsIdx();
+  unsigned NumDeopts = MI->getOperand(OperIdx).getImm();
+  // At this point stack references has not been lowered yet, so they
+  // take single operand.
+  ++OperIdx;
+  while (NumDeopts--) {
+    MachineOperand &MO = MI->getOperand(OperIdx);
+    if (MO.isImm() && MO.getImm() == StackMaps::ConstantOp) {
+      ++OperIdx;
+      assert(MI->getOperand(OperIdx).isImm() &&
+             "Unexpected statepoint operand");
+    }
+    ++OperIdx;
+  }
+  return OperIdx;
+}
+
 /// EmitCopyFromReg - Generate machine code for an CopyFromReg node or an
 /// implicit physical register output.
 void InstrEmitter::
 EmitCopyFromReg(SDNode *Node, unsigned ResNo, bool IsClone, bool IsCloned,
                 Register SrcReg, DenseMap<SDValue, Register> &VRBaseMap) {
   Register VRBase;
   if (SrcReg.isVirtual()) {
     // Just use the input register directly!
void InstrEmitter::CreateVirtualRegisters(SDNode *Node,
                                        DenseMap<SDValue, Register> &VRBaseMap) {
   assert(Node->getMachineOpcode() != TargetOpcode::IMPLICIT_DEF &&
          "IMPLICIT_DEF should have been handled as a special case elsewhere!");
 
   unsigned NumResults = CountResults(Node);
   bool HasVRegVariadicDefs = !MF->getTarget().usesPhysRegsForValues() &&
                              II.isVariadic() && II.variadicOpsAreDefs();
   unsigned NumVRegs = HasVRegVariadicDefs ? NumResults : II.getNumDefs();
+  if (Node->getMachineOpcode() == TargetOpcode::STATEPOINT)
+    NumVRegs = NumResults;
   for (unsigned i = 0; i < NumVRegs; ++i) {
     // If the specific node value is only used by a CopyToReg and the dest reg
     // is a vreg in the same register class, use the CopyToReg'd destination
     // register instead of creating a new vreg.
     Register VRBase;
     const TargetRegisterClass *RC =
       TRI->getAllocatableClass(TII->getRegClass(II, i, TRI, *MF));
     // Always let the value type influence the used register class. The
if (Opc == TargetOpcode::STACKMAP || Opc == TargetOpcode::PATCHPOINT) {
     // convention. However, to simplify runtime support, they clobber the same
     // scratch registers as AnyRegCC.
     unsigned CC = CallingConv::AnyReg;
     if (Opc == TargetOpcode::PATCHPOINT) {
       CC = Node->getConstantOperandVal(PatchPointOpers::CCPos);
       NumDefs = NumResults;
     }
     ScratchRegs = TLI->getScratchRegisters((CallingConv::ID) CC);
+  } else if (Opc == TargetOpcode::STATEPOINT) {
+    NumDefs = NumResults;
   }
 
   unsigned NumImpUses = 0;
   unsigned NodeOperands =
     countOperands(Node, II.getNumOperands() - NumDefs, NumImpUses);
   bool HasVRegVariadicDefs = !MF->getTarget().usesPhysRegsForValues() &&
                              II.isVariadic() && II.variadicOpsAreDefs();
   bool HasPhysRegOuts = NumResults > NumDefs &&
for (SDNode *F = Node->getGluedUser(); F; F = F->getGluedUser()) {
         }
     }
   }
 
   // Finally mark unused registers as dead.
   if (!UsedRegs.empty() || II.getImplicitDefs() || II.hasOptionalDef())
     MIB->setPhysRegsDeadExcept(UsedRegs, *TRI);
 
+  // STATEPOINT is too 'dynamic' to have meaningful machine description.
+  // We have to manually tie operands.
+  if (Opc == TargetOpcode::STATEPOINT && NumDefs > 0) {
+    assert(!HasPhysRegOuts && "STATEPOINT mishandled");
+    MachineInstr *MI = MIB;
+    unsigned Def = 0;
+    unsigned Use = getStatepointGCArgStartIdx(MI) + 1;
+    while (Def < NumDefs) {
+      if (MI->getOperand(Use).isReg())
+        MI->tieOperands(Def++, Use);
+      Use += 2;
+    }
+  }
+
   // Run post-isel target hook to adjust this instruction if needed.
   if (II.hasPostISelHook())
     TLI->AdjustInstrPostInstrSelection(*MIB, Node);
 }
 
 /// EmitSpecialNode - Generate machine code for a target-independent node and
 /// needed dependencies.
 void InstrEmitter::

llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

if (Register::isVirtualRegister(Reg))
     return;
 
   unsigned ResNo = User->getOperand(2).getResNo();
   if (Def->getOpcode() == ISD::CopyFromReg &&
       cast<RegisterSDNode>(Def->getOperand(1))->getReg() == Reg) {
     PhysReg = Reg;
   } else if (Def->isMachineOpcode()) {
     const MCInstrDesc &II = TII->get(Def->getMachineOpcode());
-    if (ResNo >= II.getNumDefs() &&
-        II.ImplicitDefs[ResNo - II.getNumDefs()] == Reg)
+    if (ResNo >= II.getNumDefs() && II.hasImplicitDefOfPhysReg(Reg))
       PhysReg = Reg;
   }
 
   if (PhysReg != 0) {
     const TargetRegisterClass *RC =
         TRI->getMinimalPhysRegClass(Reg, Def->getSimpleValueType(ResNo));
     Cost = RC->getCopyCost();
   }

llvm/lib/CodeGen/SelectionDAG/StatepointLowering.h

public:
   }
 
   bool isStackSlotAllocated(int Offset) {
     assert(Offset >= 0 && Offset < (int)AllocatedStackSlots.size() &&
            "out of bounds");
     return AllocatedStackSlots.test(Offset);
   }
 
+  /// For each statepoint keep mapping from original derived pointer to
+  /// the statepoint node result defining its new value.
+  DenseMap<const Value *, SDValue> DerivedPtrMap;
+
 private:
   /// Maps pre-relocation value (gc pointer directly incoming into statepoint)
   /// into it's location (currently only stack slots)
   DenseMap<SDValue, SDValue> Locations;
 
   /// A boolean indicator for each slot listed in the FunctionInfo as to
   /// whether it has been used in the current statepoint.  Since we try to
   /// preserve stack slots across safepoints, there can be gaps in which

llvm/lib/CodeGen/SelectionDAG/StatepointLowering.cpp

 STATISTIC(NumOfStatepoints, "Number of statepoint nodes encountered");
 STATISTIC(StatepointMaxSlotsRequired,
           "Maximum number of stack slots required for a singe statepoint");
 
 cl::opt<bool> UseRegistersForDeoptValues(
     "use-registers-for-deopt-values", cl::Hidden, cl::init(false),
     cl::desc("Allow using registers for non pointer deopt args"));
 
+cl::opt<unsigned> MaxRegistersForGCPointers(
+    "max-registers-for-gc-values", cl::Hidden, cl::init(0),
+    cl::desc("Max number of VRegs allowed to pass GC pointer meta args in"));
+
 static void pushStackMapConstant(SmallVectorImpl<SDValue>& Ops,
                                  SelectionDAGBuilder &Builder, uint64_t Value) {
   SDLoc L = Builder.getCurSDLoc();
   Ops.push_back(Builder.DAG.getTargetConstant(StackMaps::ConstantOp, L,
                                               MVT::i64));
   Ops.push_back(Builder.DAG.getTargetConstant(Value, L, MVT::i64));
 }
 
 void StatepointLoweringState::startNewStatepoint(SelectionDAGBuilder &Builder) {
   // Consistency check
   assert(PendingGCRelocateCalls.empty() &&
          "Trying to visit statepoint before finished processing previous one");
   Locations.clear();
   NextSlotToAllocate = 0;
   // Need to resize this on each safepoint - we need the two to stay in sync and
   // the clear patterns of a SelectionDAGBuilder have no relation to
   // FunctionLoweringInfo.  Also need to ensure used bits get cleared.
   AllocatedStackSlots.clear();
   AllocatedStackSlots.resize(Builder.FuncInfo.StatepointStackSlots.size());
+  DerivedPtrMap.clear();
 }
 
 void StatepointLoweringState::clear() {
   Locations.clear();
   AllocatedStackSlots.clear();
+  DerivedPtrMap.clear();
   assert(PendingGCRelocateCalls.empty() &&
          "cleared before statepoint sequence completed");
 }
 
 SDValue
 StatepointLoweringState::allocateStackSlot(EVT ValueType,
                                            SelectionDAGBuilder &Builder) {
   NumSlotsAllocatedForStatepoints++;
static Optional<int> findPreviousSpillSlot(const Value *Val,
   // put handling of simple modifications in this function like it's done
   // for bitcasts we might end up reserving i's slot for 'i+1' because order in
   // which we visit values is unspecified.
 
   // Don't know any information about this instruction
   return None;
 }
 
-
 /// Return true if-and-only-if the given SDValue can be lowered as either a
 /// constant argument or a stack reference.  The key point is that the value
 /// doesn't need to be spilled or tracked as a vreg use.
 static bool willLowerDirectly(SDValue Incoming) {
   // We are making an unchecked assumption that the frame size <= 2^16 as that
   // is the largest offset which can be encoded in the stackmap format.
   if (isa<FrameIndexSDNode>(Incoming))
     return true;
 
   // The largest constant describeable in the StackMap format is 64 bits.
   // Potential Optimization:  Constants values are sign extended by consumer,
   // and thus there are many constants of static type > 64 bits whose value
   // happens to be sext(Con64) and could thus be lowered directly.
   if (Incoming.getValueType().getSizeInBits() > 64)
     return false;
 
   return (isa<ConstantSDNode>(Incoming) || isa<ConstantFPSDNode>(Incoming) ||
           Incoming.isUndef());
 }
 
-
 /// Try to find existing copies of the incoming values in stack slots used for
 /// statepoint spilling.  If we can find a spill slot for the incoming value,
 /// mark that slot as allocated, and reuse the same slot for this safepoint.
 /// This helps to avoid series of loads and stores that only serve to reshuffle
 /// values on the stack between calls.
 static void reservePreviousStackSlotForValue(const Value *IncomingValue,
                                              SelectionDAGBuilder &Builder) {
   SDValue Incoming = Builder.getValue(IncomingValue);
if (!Loc.getNode()) {
     auto PtrInfo = MachinePointerInfo::getFixedStack(MF, Index);
     auto *StoreMMO = MF.getMachineMemOperand(
         PtrInfo, MachineMemOperand::MOStore, MFI.getObjectSize(Index),
         MFI.getObjectAlign(Index));
     Chain = Builder.DAG.getStore(Chain, Builder.getCurSDLoc(), Incoming, Loc,
                                  StoreMMO);
 
     MMO = getMachineMemOperand(MF, *cast<FrameIndexSDNode>(Loc));
-    
     Builder.StatepointLowering.setLocation(Incoming, Loc);
   }
 
   assert(Loc.getNode());
   return std::make_tuple(Loc, Chain, MMO);
 }
 
 /// Lower a single value incoming to a statepoint node.  This value can be
 /// either a deopt value or a gc value, the handling is the same.  We special
 /// case constants and allocas, then fall back to spilling if required.
 static void
 lowerIncomingStatepointValue(SDValue Incoming, bool RequireSpillSlot,
                              SmallVectorImpl<SDValue> &Ops,
                              SmallVectorImpl<MachineMemOperand *> &MemRefs,
                              SelectionDAGBuilder &Builder) {
-  
   if (willLowerDirectly(Incoming)) {
     if (FrameIndexSDNode *FI = dyn_cast<FrameIndexSDNode>(Incoming)) {
       // This handles allocas as arguments to the statepoint (this is only
       // really meaningful for a deopt value.  For GC, we'd be trying to
       // relocate the address of the alloca itself?)
       assert(Incoming.getValueType() == Builder.getFrameIndexTy() &&
              "Incoming value is a frame index!");
       Ops.push_back(Builder.DAG.getTargetFrameIndex(FI->getIndex(),
                                                     Builder.getFrameIndexTy()));
 
       auto &MF = Builder.DAG.getMachineFunction();
       auto *MMO = getMachineMemOperand(MF, *FI);
       MemRefs.push_back(MMO);
       return;
     }
 
     assert(Incoming.getValueType().getSizeInBits() <= 64);
-    
     if (Incoming.isUndef()) {
       // Put an easily recognized constant that's unlikely to be a valid
       // value so that uses of undef by the consumer of the stackmap is
       // easily recognized. This is legal since the compiler is always
       // allowed to chose an arbitrary value for undef.
       pushStackMapConstant(Ops, Builder, 0xFEFEFEFE);
       return;
     }
if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(Incoming)) {
                            C->getValueAPF().bitcastToAPInt().getZExtValue());
       return;
     }
 
     llvm_unreachable("unhandled direct lowering case");
   }
 
 
-
   if (!RequireSpillSlot) {
     // If this value is live in (not live-on-return, or live-through), we can
     // treat it the same way patchpoint treats it's "live in" values.  We'll
     // end up folding some of these into stack references, but they'll be
     // handled by the register allocator.  Note that we do not have the notion
     // of a late use so these values might be placed in registers which are
     // clobbered by the call.  This is fine for live-in. For live-through
     // fix-up pass should be executed to force spilling of such registers.

 /// lowering is described in lowerIncomingStatepointValue.  This function is
 /// responsible for lowering everything in the right position and playing some
 /// tricks to avoid redundant stack manipulation where possible.  On
 /// completion, 'Ops' will contain ready to use operands for machine code
 /// statepoint. The chain nodes will have already been created and the DAG root
 /// will be set to the last value spilled (if any were).
 static void
 lowerStatepointMetaArgs(SmallVectorImpl<SDValue> &Ops,
-                        SmallVectorImpl<MachineMemOperand*> &MemRefs,                                    SelectionDAGBuilder::StatepointLoweringInfo &SI,
+                        SmallVectorImpl<MachineMemOperand *> &MemRefs,
+                        DenseMap<SDValue, int> &LowerAsVReg,
+                        SelectionDAGBuilder::StatepointLoweringInfo &SI,
                         SelectionDAGBuilder &Builder) {
   // Lower the deopt and gc arguments for this statepoint.  Layout will be:
   // deopt argument length, deopt arguments.., gc arguments...
 #ifndef NDEBUG
   if (auto *GFI = Builder.GFI) {
     // Check that each of the gc pointer and bases we've gotten out of the
     // safepoint is something the strategy thinks might be a pointer (or vector
     // of pointers) into the GC heap.  This is basically just here to help catch
#endif
   // particular, if the callee spills callee preserved registers we may not
   // be able to find a value placed in that register during the call.  This is
   // fine for live-out, but not for live-through.  If we were willing to make
   // assumptions about the code generator producing the callee, we could
   // potentially allow live-through values in callee saved registers.
   const bool LiveInDeopt =
     SI.StatepointFlags & (uint64_t)StatepointFlags::DeoptLiveIn;
 
+  // Decide which deriver pointers will go on VRegs
+  const unsigned MaxTiedRegs = 15; // Max  number of tied regs MI can have.
+  unsigned MaxVRegPtrs =
+      std::min(MaxTiedRegs, MaxRegistersForGCPointers.getValue());
+
+  LLVM_DEBUG(dbgs() << "Desiding how to lower GC Pointers:\n");
+  unsigned CurNumVRegs = 0;
+  for (const Value *P : SI.Ptrs) {
+    if (LowerAsVReg.size() == MaxVRegPtrs)
+      break;
+    SDValue PtrSD = Builder.getValue(P);
+    if (willLowerDirectly(PtrSD) || P->getType()->isVectorTy()) {
+      LLVM_DEBUG(dbgs() << "direct/spill "; PtrSD.dump(&Builder.DAG));
+      continue;
+    }
+    LLVM_DEBUG(dbgs() << "vreg "; PtrSD.dump(&Builder.DAG));
+    LowerAsVReg[PtrSD] = CurNumVRegs++;
+  }
+  LLVM_DEBUG(dbgs() << LowerAsVReg.size()
+                    << " derived pointers will go in vregs\n");
+
   auto isGCValue = [&](const Value *V) {
     auto *Ty = V->getType();
     if (!Ty->isPtrOrPtrVectorTy())
       return false;
     if (auto *GFI = Builder.GFI)
       if (auto IsManaged = GFI->getStrategy().isGCManagedPointer(Ty))
         return *IsManaged;
     return true; // conservative
   };
 
   auto requireSpillSlot = [&](const Value *V) {
-    return !(LiveInDeopt || UseRegistersForDeoptValues) || isGCValue(V);
+    if (isGCValue(V))
+      return !LowerAsVReg.count(Builder.getValue(V));
+    return !(LiveInDeopt || UseRegistersForDeoptValues);
   };
 
   // Before we actually start lowering (and allocating spill slots for values),
   // reserve any stack slots which we judge to be profitable to reuse for a
   // particular value.  This is purely an optimization over the code below and
   // doesn't change semantics at all.  It is important for performance that we
   // reserve slots for both deopt and gc values before lowering either.
   for (const Value *V : SI.DeoptState) {
     if (requireSpillSlot(V))
       reservePreviousStackSlotForValue(V, Builder);
   }
+
   for (unsigned i = 0; i < SI.Bases.size(); ++i) {
+    SDValue SDV = Builder.getValue(SI.Bases[i]);
+    if (!LowerAsVReg.count(SDV))
       reservePreviousStackSlotForValue(SI.Bases[i], Builder);
+    SDV = Builder.getValue(SI.Ptrs[i]);
+    if (!LowerAsVReg.count(SDV))
       reservePreviousStackSlotForValue(SI.Ptrs[i], Builder);
   }
 
   // First, prefix the list with the number of unique values to be
   // lowered.  Note that this is the number of *Values* not the
   // number of SDValues required to lower them.
   const int NumVMSArgs = SI.DeoptState.size();
   pushStackMapConstant(Ops, Builder, NumVMSArgs);
 
   // The vm state arguments are lowered in an opaque manner.  We do not know
   // what type of values are contained within.
+  LLVM_DEBUG(dbgs() << "Lowering deopt state\n");
   for (const Value *V : SI.DeoptState) {
     SDValue Incoming;
     // If this is a function argument at a static frame index, generate it as
     // the frame index.
     if (const Argument *Arg = dyn_cast<Argument>(V)) {
       int FI = Builder.FuncInfo.getArgumentFrameIndex(Arg);
       if (FI != INT_MAX)
         Incoming = Builder.DAG.getFrameIndex(FI, Builder.getFrameIndexTy());
     }
     if (!Incoming.getNode())
       Incoming = Builder.getValue(V);
+    LLVM_DEBUG(dbgs() << "Value " << *V
+                      << " requireSpillSlot = " << requireSpillSlot(V) << "\n");
     lowerIncomingStatepointValue(Incoming, requireSpillSlot(V), Ops, MemRefs,
                                  Builder);
   }
 
   // Finally, go ahead and lower all the gc arguments.  There's no prefixed
   // length for this one.  After lowering, we'll have the base and pointer
   // arrays interwoven with each (lowered) base pointer immediately followed by
   // it's (lowered) derived pointer.  i.e
   // (base[0], ptr[0], base[1], ptr[1], ...)
   for (unsigned i = 0; i < SI.Bases.size(); ++i) {
-    const Value *Base = SI.Bases[i];
-    lowerIncomingStatepointValue(Builder.getValue(Base),
-                                 /*RequireSpillSlot*/ true, Ops, MemRefs,
+    bool RequireSpillSlot;
+    SDValue Base = Builder.getValue(SI.Bases[i]);
+    RequireSpillSlot = !LowerAsVReg.count(Base);
+    lowerIncomingStatepointValue(Base, RequireSpillSlot, Ops, MemRefs,
                                  Builder);
 
-    const Value *Ptr = SI.Ptrs[i];
-    lowerIncomingStatepointValue(Builder.getValue(Ptr),
-                                 /*RequireSpillSlot*/ true, Ops, MemRefs,
+    SDValue Derived = Builder.getValue(SI.Ptrs[i]);
+    RequireSpillSlot = !LowerAsVReg.count(Derived);
+    lowerIncomingStatepointValue(Derived, RequireSpillSlot, Ops, MemRefs,
                                  Builder);
   }
 
   // If there are any explicit spill slots passed to the statepoint, record
   // them, but otherwise do not do anything special.  These are user provided
   // allocas and give control over placement to the consumer.  In this case,
   // it is the contents of the slot which may get updated, not the pointer to
   // the alloca
#endif
   const Instruction *StatepointInstr = SI.StatepointInstr;
   auto &SpillMap = Builder.FuncInfo.StatepointSpillMaps[StatepointInstr];
 
   for (const GCRelocateInst *Relocate : SI.GCRelocates) {
     const Value *V = Relocate->getDerivedPtr();
     SDValue SDV = Builder.getValue(V);
     SDValue Loc = Builder.StatepointLowering.getLocation(SDV);
 
-    if (Loc.getNode()) {
+    if (LowerAsVReg.count(SDV)) {
+      SpillMap[V] = None;
+    } else if (Loc.getNode()) {
       SpillMap[V] = cast<FrameIndexSDNode>(Loc)->getIndex();
     } else {
       // Record value as visited, but not spilled. This is case for allocas
       // and constants. For this values we can avoid emitting spill load while
       // visiting corresponding gc_relocate.
       // Actually we do not need to record them in this map at all.
       // We do this only to check that we are not relocating any unvisited
       // value.
SDValue SelectionDAGBuilder::LowerAsSTATEPOINT(
   // lower it, then reverse engineer the calling sequence.
 
   NumOfStatepoints++;
   // Clear state
   StatepointLowering.startNewStatepoint(*this);
   assert(SI.Bases.size() == SI.Ptrs.size() &&
          SI.Ptrs.size() <= SI.GCRelocates.size());
 
+  LLVM_DEBUG(dbgs() << "Lowering statepoint " << *SI.StatepointInstr << "\n");
 #ifndef NDEBUG
   for (auto *Reloc : SI.GCRelocates)
     if (Reloc->getParent() == SI.StatepointInstr->getParent())
       StatepointLowering.scheduleRelocCall(*Reloc);
 #endif
 
   // Lower statepoint vmstate and gcstate arguments
   SmallVector<SDValue, 10> LoweredMetaArgs;
   SmallVector<MachineMemOperand*, 16> MemRefs;
-  lowerStatepointMetaArgs(LoweredMetaArgs, MemRefs, SI, *this);
+  // Maps derived pointer SDValue to statepoint result of relocated pointer.
+  DenseMap<SDValue, int> LowerAsVReg;
+  lowerStatepointMetaArgs(LoweredMetaArgs, MemRefs, LowerAsVReg, SI, *this);
 
   // Now that we've emitted the spills, we need to update the root so that the
   // call sequence is ordered correctly.
   SI.CLI.setChain(getRoot());
 
   // Get call node, we will replace it later with statepoint
   SDValue ReturnVal;
   SDNode *CallNode;
#endif
   Ops.push_back(Chain);
 
   // Same for the glue, but we add it only if original call had it
   if (Glue.getNode())
     Ops.push_back(Glue);
 
   // Compute return values.  Provide a glue output since we consume one as
   // input.  This allows someone else to chain off us as needed.
-  SDVTList NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);
+  SmallVector<EVT, 8> NodeTys;
+  for (auto &Ptr : SI.Ptrs) {
+    SDValue SD = getValue(Ptr);
+    if (LowerAsVReg.count(SD)) {
+      NodeTys.push_back(SD.getValueType());
+    }
+  }
+  LLVM_DEBUG(dbgs() << "Statepoint has " << NodeTys.size() << " results\n");
+  assert(NodeTys.size() == LowerAsVReg.size() && "Inconsistent GC Ptr lowering");
+  NodeTys.push_back(MVT::Other);
+  NodeTys.push_back(MVT::Glue);
 
+  unsigned NumResults = NodeTys.size();
   MachineSDNode *StatepointMCNode =
     DAG.getMachineNode(TargetOpcode::STATEPOINT, getCurSDLoc(), NodeTys, Ops);
   DAG.setNodeMemRefs(StatepointMCNode, MemRefs);
 
   SDNode *SinkNode = StatepointMCNode;
 
+  // Fill mapping from derived pointer to statepoint result denoting its
+  // relocated value.
+  auto &DPtrMap = StatepointLowering.DerivedPtrMap;
+  for (const auto *Relocate : SI.GCRelocates) {
+    Value *Derived = Relocate->getDerivedPtr();
+    SDValue SD = getValue(Derived);
+    if (LowerAsVReg.count(SD))
+      DPtrMap[Derived] = SDValue(StatepointMCNode, LowerAsVReg[SD]);
+  }
+
   // Build the GC_TRANSITION_END node if necessary.
   //
   // See the comment above regarding GC_TRANSITION_START for the layout of
   // the operands to the GC_TRANSITION_END node.
   if (IsGCTransition) {
     SmallVector<SDValue, 8> TEOps;
 
     // Add chain
-    TEOps.push_back(SDValue(StatepointMCNode, 0));
+    TEOps.push_back(SDValue(StatepointMCNode, NumResults - 2));
 
     // Add GC transition arguments
     for (const Value *V : SI.GCTransitionArgs) {
       TEOps.push_back(getValue(V));
       if (V->getType()->isPointerTy())
         TEOps.push_back(DAG.getSrcValue(V));
     }
 
     // Add glue
-    TEOps.push_back(SDValue(StatepointMCNode, 1));
+    TEOps.push_back(SDValue(StatepointMCNode, NumResults - 1));
 
     SDVTList NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);
 
     SDValue GCTransitionStart =
         DAG.getNode(ISD::GC_TRANSITION_END, getCurSDLoc(), NodeTys, TEOps);
 
     SinkNode = GCTransitionStart.getNode();
   }
 
   // Replace original call
-  DAG.ReplaceAllUsesWith(CallNode, SinkNode); // This may update Root
+  // Call: ch,glue = CALL ...
+  // Statepoint: [gc relocates],ch,glue = STATEPOINT ...
+  unsigned NumSinkValues = SinkNode->getNumValues();
+  SDValue StatepointValues[2] = {SDValue(SinkNode, NumSinkValues - 2),
+                                 SDValue(SinkNode, NumSinkValues - 1)};
+  DAG.ReplaceAllUsesWith(CallNode, StatepointValues);
   // Remove original call node
   DAG.DeleteNode(CallNode);
 
   // DON'T set the root - under the assumption that it's already set past the
   // inserted node we created.
 
   // TODO: A better future implementation would be to emit a single variable
   // argument, variable return value STATEPOINT node here and then hookup the
SI.GCTransitionArgs = ArrayRef<const Use>(I.gc_transition_args_begin(),
                                             I.gc_transition_args_end());
 
   SI.StatepointFlags = I.getFlags();
   SI.NumPatchBytes = I.getNumPatchBytes();
   SI.EHPadBB = EHPadBB;
 
   SDValue ReturnValue = LowerAsSTATEPOINT(SI);
 
+  // If relocated value will be used in different basic
+  // block, we need to export them manually. Default exporting mechanism
+  // will not work here because it is based on IR Value types, and
+  // IR statepoint has different type than the actual call or relocates.
+  // It means that by default llvm will create export register of the wrong
+  // type (always i32 - TokenTy - in our case). So instead we need to create
+  // export registers manually.
+  const BasicBlock *BB = I.getParent();
+  auto &DPtrMap = StatepointLowering.DerivedPtrMap;
+  auto &StatepointRegMap = FuncInfo.StatepointRegs[SI.StatepointInstr];
+  for (const auto *Reloc : SI.GCRelocates) {
+    if (Reloc->getParent() == BB)
+      continue;
+
+    const Value *DerivedPtr = Reloc->getDerivedPtr();
+    if (!DPtrMap.count(DerivedPtr))
+      continue; // not lowered through VReg
+    if (StatepointRegMap.count(DerivedPtr))
+      continue;
+
+    SDValue Res = DPtrMap[DerivedPtr];
+    Type *Ty = DerivedPtr->getType();
+    Register Reg = FuncInfo.CreateRegs(Ty);
+    RegsForValue RFV(*DAG.getContext(), DAG.getTargetLoweringInfo(),
+                     DAG.getDataLayout(), Reg, Ty, None);
+    SDValue Chain = DAG.getEntryNode();
+
+    RFV.getCopyToRegs(Res, DAG, getCurSDLoc(), Chain, nullptr, DerivedPtr);
+    PendingExports.push_back(Chain);
+    StatepointRegMap[DerivedPtr] = Reg;
+  }
+
   // Export the result value if needed
   const GCResultInst *GCResult = I.getGCResult();
   Type *RetTy = I.getActualReturnType();
 
   if (RetTy->isVoidTy() || !GCResult) {
     // The return value is not needed, just generate a poison value. 
     setValue(&I, DAG.getIntPtrConstant(-1, getCurSDLoc()));
     return;
   }
 
   if (GCResult->getParent() == I.getParent()) {
     // Result value will be used in a same basic block. Don't export it or
     // perform any explicit register copies. The gc_result will simply grab
     // this value. 
     setValue(&I, ReturnValue);
     return;
   }
-  
   // Result value will be used in a different basic block so we need to export
   // it now.  Default exporting mechanism will not work here because statepoint
   // call has a different type than the actual call. It means that by default
   // llvm will create export register of the wrong type (always i32 in our
   // case). So instead we need to create export register with correct type
   // manually.
   // TODO: To eliminate this problem we can remove gc.result intrinsics
   //       completely and make statepoint call to return a tuple.
   unsigned Reg = FuncInfo.CreateRegs(RetTy);
   RegsForValue RFV(*DAG.getContext(), DAG.getTargetLoweringInfo(),
                    DAG.getDataLayout(), Reg, RetTy,
                    I.getCallingConv());
   SDValue Chain = DAG.getEntryNode();
-  
   RFV.getCopyToRegs(ReturnValue, DAG, getCurSDLoc(), Chain, nullptr);
   PendingExports.push_back(Chain);
   FuncInfo.ValueMap[&I] = Reg;
 }
 
 void SelectionDAGBuilder::LowerCallSiteWithDeoptBundleImpl(
     const CallBase *Call, SDValue Callee, const BasicBlock *EHPadBB,
     bool VarArgDisallowed, bool ForceVoidReturnTy) {
void SelectionDAGBuilder::visitGCResult(const GCResultInst &CI) {
   // Statepoint is in different basic block so we should have stored call
   // result in a virtual register.
   // We can not use default getValue() functionality to copy value from this
   // register because statepoint and actual call return types can be
   // different, and getValue() will use CopyFromReg of the wrong type,
   // which is always i32 in our case.
   Type *RetTy = SI->getActualReturnType();
   SDValue CopyFromReg = getCopyFromRegs(SI, RetTy);
-  
   assert(CopyFromReg.getNode());
   setValue(&CI, CopyFromReg);
 }
 
 void SelectionDAGBuilder::visitGCRelocate(const GCRelocateInst &Relocate) {
+  const BasicBlock *StatepointBB = Relocate.getStatepoint()->getParent();
 #ifndef NDEBUG
   // Consistency check
   // We skip this check for relocates not in the same basic block as their
   // statepoint. It would be too expensive to preserve validation info through
   // different basic blocks.
-  if (Relocate.getStatepoint()->getParent() == Relocate.getParent())
+  if (StatepointBB == Relocate.getParent())
     StatepointLowering.relocCallVisited(Relocate);
 
   auto *Ty = Relocate.getType()->getScalarType();
   if (auto IsManaged = GFI->getStrategy().isGCManagedPointer(Ty))
     assert(*IsManaged && "Non gc managed pointer relocated!");
 #endif
 
   const Value *DerivedPtr = Relocate.getDerivedPtr();
   SDValue SD = getValue(DerivedPtr);
 
   if (SD.isUndef() && SD.getValueType().getSizeInBits() <= 64) {
     // Lowering relocate(undef) as arbitrary constant. Current constant value
     // is chosen such that it's unlikely to be a valid pointer.
     setValue(&Relocate, DAG.getTargetConstant(0xFEFEFEFE, SDLoc(SD), MVT::i64));
     return;
   }
 
+  // Relocate is local to statepoint block and its pointer was assigned
+  // to VReg. Use corresponding statepoint result.
+  auto &DPtrMap = StatepointLowering.DerivedPtrMap;
+  auto It = DPtrMap.find(DerivedPtr);
+  if (It != DPtrMap.end()) {
+    setValue(&Relocate, It->second);
+    assert(Relocate.getParent() == StatepointBB && "unexpected DPtrMap entry");
+    return;
+  }
+
+  auto &StatepointRegs = FuncInfo.StatepointRegs[Relocate.getStatepoint()];
+  if (StatepointRegs.count(DerivedPtr)) {
+    // Statepoint is in different basic block. Default getValue() mechanism
+    // does not work here, so we need create CopyFromRegs manually.
+    // See comment in LowerStatepoint for details.
+    assert(StatepointBB != Relocate.getParent() && "local gc.relocate");
+    Register Reg = StatepointRegs[DerivedPtr];
+    assert(Reg > 0 && "invalid register");
+    RegsForValue RFV(*DAG.getContext(), DAG.getTargetLoweringInfo(),
+                     DAG.getDataLayout(), Reg, DerivedPtr->getType(),
+                     None); // This is not an ABI copy.
+    SDValue Chain = DAG.getEntryNode();
+    SDValue Result = RFV.getCopyFromRegs(DAG, FuncInfo, getCurSDLoc(), Chain,
+                                         nullptr, DerivedPtr);
+    setValue(&Relocate, Result);
+    return;
+  }
+
   auto &SpillMap = FuncInfo.StatepointSpillMaps[Relocate.getStatepoint()];
   auto SlotIt = SpillMap.find(DerivedPtr);
   assert(SlotIt != SpillMap.end() && "Relocating not lowered gc value");
   Optional<int> DerivedPtrLocation = SlotIt->second;
 
   // We didn't need to spill these special cases (constants and allocas).
   // See the handling in spillIncomingValueForStatepoint for detail.
   if (!DerivedPtrLocation) {

llvm/lib/CodeGen/TargetInstrInfo.cpp

 void TargetInstrInfo::getNoop(MCInst &NopInst) const {
   llvm_unreachable("Not implemented");
 }
 
 static MachineInstr *foldPatchpoint(MachineFunction &MF, MachineInstr &MI,
                                     ArrayRef<unsigned> Ops, int FrameIndex,
                                     const TargetInstrInfo &TII) {
   unsigned StartIdx = 0;
+  unsigned NumDefs = 0;
   switch (MI.getOpcode()) {
   case TargetOpcode::STACKMAP: {
     // StackMapLiveValues are foldable
     StartIdx = StackMapOpers(&MI).getVarIdx();
     break;
   }
   case TargetOpcode::PATCHPOINT: {
     // For PatchPoint, the call args are not foldable (even if reported in the
     // stackmap e.g. via anyregcc).
     StartIdx = PatchPointOpers(&MI).getVarIdx();
     break;
   }
   case TargetOpcode::STATEPOINT: {
     // For statepoints, fold deopt and gc arguments, but not call arguments.
     StartIdx = StatepointOpers(&MI).getVarIdx();
+    NumDefs = MI.getNumDefs();
     break;
   }
   default:
     llvm_unreachable("unexpected stackmap opcode");
   }
 
+  unsigned DefToFoldIdx = MI.getNumOperands();
+
   // Return false if any operands requested for folding are not foldable (not
   // part of the stackmap's live values).
   for (unsigned Op : Ops) {
-    if (Op < StartIdx)
+    if (Op < NumDefs)
+      DefToFoldIdx = Op;
+    else if (Op < StartIdx)
+      return nullptr;
+    // When called from regalloc (InlineSpiller), operands must be untied,
+    // and regalloc will take care of (re)loading operand from memory.
+    // But when called from other places (e.g. peephole pass),
+    // we cannot fold operand which are tied - callers are unaware they
+    // need to reload destination register.
+    if (MI.getOperand(Op).isTied())
       return nullptr;
   }
 
   MachineInstr *NewMI =
       MF.CreateMachineInstr(TII.get(MI.getOpcode()), MI.getDebugLoc(), true);
   MachineInstrBuilder MIB(MF, NewMI);
 
   // No need to fold return, the meta data, and function arguments
   for (unsigned i = 0; i < StartIdx; ++i)
+    if (i != DefToFoldIdx)
       MIB.add(MI.getOperand(i));
 
-  for (unsigned i = StartIdx; i < MI.getNumOperands(); ++i) {
+  for (unsigned i = StartIdx, e = MI.getNumOperands(); i < e; ++i) {
     MachineOperand &MO = MI.getOperand(i);
+    unsigned TiedTo = e;
+    (void)MI.isRegTiedToDefOperand(i, &TiedTo);
+
     if (is_contained(Ops, i)) {
+      assert(TiedTo == e && "Cannot fold tied operands");
       unsigned SpillSize;
       unsigned SpillOffset;
       // Compute the spill slot size and offset.
       const TargetRegisterClass *RC =
         MF.getRegInfo().getRegClass(MO.getReg());
       bool Valid =
           TII.getStackSlotRange(RC, MO.getSubReg(), SpillSize, SpillOffset, MF);
       if (!Valid)
         report_fatal_error("cannot spill patchpoint subregister operand");
       MIB.addImm(StackMaps::IndirectMemRefOp);
       MIB.addImm(SpillSize);
       MIB.addFrameIndex(FrameIndex);
       MIB.addImm(SpillOffset);
-    }
-    else
+    } else {
       MIB.add(MO);
+      if (TiedTo < e) {
+        assert(TiedTo < NumDefs && "Bad tied operand");
+        if (TiedTo > DefToFoldIdx)
+          --TiedTo;
+        NewMI->tieOperands(TiedTo, NewMI->getNumOperands() - 1);
+      }
+    }
   }
   return NewMI;
 }
 
 MachineInstr *TargetInstrInfo::foldMemoryOperand(MachineInstr &MI,
                                                  ArrayRef<unsigned> Ops, int FI,
                                                  LiveIntervals *LIS,
                                                  VirtRegMap *VRM) const {

llvm/lib/CodeGen/TargetLoweringBase.cpp

if (!llvm::any_of(MI->operands(),
                     [](MachineOperand &Operand) { return Operand.isFI(); }))
     return MBB;
 
   MachineInstrBuilder MIB = BuildMI(MF, MI->getDebugLoc(), MI->getDesc());
 
   // Inherit previous memory operands.
   MIB.cloneMemRefs(*MI);
 
-  for (auto &MO : MI->operands()) {
+  for (unsigned i = 0; i < MI->getNumOperands(); ++i) {
+    MachineOperand &MO = MI->getOperand(i);
     if (!MO.isFI()) {
+      // Index of Def operand this Use it tied to.
+      // Since Defs are coming before Uses, if Use is tied, then
+      // index of Def must be smaller that index of that Use.
+      // Also, Defs preserve their position in new MI.
+      unsigned TiedTo = i;
+      if (MO.isReg() && MO.isTied())
+        TiedTo = MI->findTiedOperandIdx(i);
       MIB.add(MO);
+      if (TiedTo < i)
+        MIB->tieOperands(TiedTo, MIB->getNumOperands() - 1);
       continue;
     }
 
     // foldMemoryOperand builds a new MI after replacing a single FI operand
     // with the canonical set of five x86 addressing-mode operands.
     int FI = MO.getIndex();
 
     // Add frame index operands recognized by stackmaps.cpp

llvm/test/CodeGen/X86/statepoint-fixup-call.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+--- |
+  ; ModuleID = 'test/CodeGen/X86/statepoint-fixup-call.ll'
+  source_filename = "test/CodeGen/X86/statepoint-fixup-call.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @foo()
+
+  define i8 addrspace(1)* @test_one(i8 addrspace(1)* %p) gc "statepoint-example" {
+  entry:
+    %token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @foo, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p) ]
+    %p2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p2
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32 immarg, i32 immarg) #0
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_one
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$rdi', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    liveins: $rdi
+
+    ; CHECK-LABEL: name: test_one
+    ; CHECK: liveins: $rdi
+    ; CHECK: ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed $rdi :: (store 8 into %stack.0)
+    ; CHECK: STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load 8 from %stack.0)
+    ; CHECK: $rdi = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+    ; CHECK: ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: $rax = COPY killed renamable $rdi
+    ; CHECK: RET 0, killed $rax
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rdi = STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, killed renamable $rdi, renamable $rdi(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    $rax = COPY killed renamable $rdi
+    RET 0, killed $rax
+
+...

llvm/test/CodeGen/X86/statepoint-fixup-invoke.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+
+--- |
+  ; ModuleID = 'test/CodeGen/X86/statepoint-fixup-invoke.mir'
+  source_filename = "test/CodeGen/X86/statepoint-fixup-invoke.mir"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @some_call(i64 addrspace(1)*)
+
+  declare i32 @personality_function()
+
+  define i64 addrspace(1)* @test_basic(i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1) gc "statepoint-example" personality i32 ()* @personality_function {
+  entry:
+    %0 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %obj, i32 0, i32 0) [ "gc-live"(i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1), "deopt"(i32 0, i32 -1, i32 0, i32 0, i32 0) ]
+            to label %invoke_safepoint_normal_dest unwind label %exceptional_return
+
+  invoke_safepoint_normal_dest:                     ; preds = %entry
+    %obj.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 0, i32 0) ; (%obj, %obj)
+    %obj1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 1, i32 1) ; (%obj1, %obj1)
+    br label %normal_return
+
+  normal_return:                                    ; preds = %invoke_safepoint_normal_dest
+    ret i64 addrspace(1)* %obj.relocated
+
+  exceptional_return:                               ; preds = %entry
+    %landing_pad = landingpad token
+            cleanup
+    %obj.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 0, i32 0) ; (%obj, %obj)
+    %obj1.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 1, i32 1) ; (%obj1, %obj1)
+    ret i64 addrspace(1)* %obj1.relocated1
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 immarg, i32 immarg, void (i64 addrspace(1)*)*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token, i32 immarg, i32 immarg) #0
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_basic
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$rdi', virtual-reg: '' }
+  - { reg: '$rsi', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: test_basic
+  ; CHECK: bb.0.entry:
+  ; CHECK:   successors: %bb.1(0x7ffff800), %bb.3(0x00000800)
+  ; CHECK:   liveins: $rdi, $rsi
+  ; CHECK-DAG:   MOV64mr %stack.0, 1, $noreg, 0, $noreg, $rsi :: (store 8 into %stack.0)
+  ; CHECK-DAG:   MOV64mr %stack.1, 1, $noreg, 0, $noreg, $rdi :: (store 8 into %stack.1)
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp0>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, 1, 8, %stack.1, 0, 1, 8, %stack.1, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load 8 from %stack.1), (load 8 from %stack.0)
+  ; CHECK-DAG:   $r14 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+  ; CHECK-DAG:   $rbx = MOV64rm %stack.1, 1, $noreg, 0, $noreg :: (load 8 from %stack.1)
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp1>
+  ; CHECK:   JMP_1 %bb.1
+  ; CHECK: bb.1.invoke_safepoint_normal_dest:
+  ; CHECK:   successors: %bb.2(0x80000000)
+  ; CHECK:   liveins: $rbx
+  ; CHECK: bb.2.normal_return:
+  ; CHECK:   liveins: $rbx
+  ; CHECK:   $rax = COPY killed renamable $rbx
+  ; CHECK:   RET 0, $rax
+  ; CHECK: bb.3.exceptional_return (landing-pad):
+  ; CHECK:   liveins: $rax, $rdx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp2>
+  ; CHECK-DAG:   $r14 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+  ; CHECK-DAG:   $rbx = MOV64rm %stack.1, 1, $noreg, 0, $noreg :: (load 8 from %stack.1)
+  ; CHECK:   $rax = COPY killed renamable $r14
+  ; CHECK:   RET 0, $rax
+  bb.0.entry:
+    successors: %bb.1(0x7ffff800), %bb.3(0x00000800)
+    liveins: $rdi, $rsi
+
+    renamable $r14 = COPY $rsi
+    renamable $rbx = COPY $rdi
+    EH_LABEL <mcsymbol .Ltmp0>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $r14, renamable $rbx = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, killed renamable $r14, renamable $r14(tied-def 0), killed renamable $rbx, renamable $rbx(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol .Ltmp1>
+    JMP_1 %bb.1
+
+  bb.1.invoke_safepoint_normal_dest:
+    successors: %bb.2(0x80000000)
+    liveins: $rbx
+
+
+  bb.2.normal_return:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+  bb.3.exceptional_return (landing-pad):
+    liveins: $rax, $rdx, $r14
+
+    EH_LABEL <mcsymbol .Ltmp2>
+    $rax = COPY killed renamable $r14
+    RET 0, $rax
+
+...

llvm/test/CodeGen/X86/statepoint-fixup-shared-ehpad.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+
+# NOTE: MIR in this test was hand edited to have two statepoints share single landing pad.
+# This is forbidden in IR, but is allowed in MIR. I just was unable to reproduce conditions
+# under which landing pad merge happens. So I did it manually.
+#
+--- |
+  ; ModuleID = 'statepoint-fixup-shared-ehpad.ll'
+  source_filename = "statepoint-fixup-shared-ehpad.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @foo()
+
+  declare void @bar()
+
+  declare i32 @personality_function()
+
+  define i8 addrspace(1)* @test_one(i32 %a, i8 addrspace(1)* %p, i8 addrspace(1)* %q) gc "statepoint-example" personality i32 ()* @personality_function {
+  entry:
+    %cmp = icmp eq i32 %a, 0
+    br i1 %cmp, label %zero, label %nonzero
+
+  zero:                                             ; preds = %entry
+    %token = invoke token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @foo, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p, i8 addrspace(1)* %q) ]
+            to label %normal_dest_a unwind label %exceptional_return_a
+
+  nonzero:                                          ; preds = %entry
+    %token2 = invoke token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @bar, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p, i8 addrspace(1)* %q) ]
+            to label %normal_dest_b unwind label %exceptional_return_b
+
+  normal_dest_a:                                    ; preds = %zero
+    %p2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p2
+
+  exceptional_return_a:                             ; preds = %zero
+    %landing_pad = landingpad token
+            cleanup
+    %q2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %landing_pad, i32 1, i32 1) ; (%q, %q)
+    ret i8 addrspace(1)* %q2
+
+  normal_dest_b:                                    ; preds = %nonzero
+    %p3 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token2, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p3
+
+  exceptional_return_b:                             ; preds = %nonzero
+    %landing_pad2 = landingpad token
+            cleanup
+    %q3 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %landing_pad2, i32 1, i32 1) ; (%q, %q)
+    ret i8 addrspace(1)* %q3
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32 immarg, i32 immarg) #0
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_one
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$edi', virtual-reg: '' }
+  - { reg: '$rsi', virtual-reg: '' }
+  - { reg: '$rdx', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: test_one
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.3(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   liveins: $rbx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp0>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   MOV64mr [[STACK0:%stack.[0-9]+]], 1, $noreg, 0, $noreg, killed $rbx :: (store 8 into [[STACK0]])
+  ; CHECK:   MOV64mr [[STACK1:%stack.[0-9]+]], 1, $noreg, 0, $noreg, killed $r14 :: (store 8 into [[STACK1]])
+  ; CHECK:   STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK1]], 0, 1, 8, [[STACK1]], 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load 8 from [[STACK0]]), (load 8 from [[STACK1]])
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp1>
+  ; CHECK:   JMP_1 %bb.3
+  ; CHECK: bb.2:
+  ; CHECK:   successors: %bb.5(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   liveins: $rbx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp2>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK-DAG:   MOV64mr [[STACK0]], 1, $noreg, 0, $noreg, killed $rbx :: (store 8 into [[STACK0]])
+  ; CHECK-DAG:   MOV64mr [[STACK1]], 1, $noreg, 0, $noreg, killed $r14 :: (store 8 into [[STACK1]])
+  ; CHECK:   STATEPOINT 0, 0, 0, @bar, 2, 0, 2, 0, 2, 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK1]], 0, 1, 8, [[STACK1]], 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load 8 from [[STACK0]]), (load 8 from [[STACK1]])
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp3>
+  ; CHECK:   JMP_1 %bb.5
+  ; CHECK: bb.4 (landing-pad):
+  ; CHECK:   liveins: $rax, $rdx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp4>
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   $rax = COPY killed renamable $r14
+  ; CHECK:   RET 0, $rax
+  bb.0:
+    successors: %bb.1, %bb.2
+    liveins: $edi, $rdx, $rsi
+
+    renamable $r14 = COPY $rdx
+    renamable $rbx = COPY $rsi
+    TEST32rr killed renamable $edi, renamable $edi, implicit-def $eflags
+    JCC_1 %bb.2, 5, implicit killed $eflags
+    JMP_1 %bb.1
+
+  bb.1:
+    successors: %bb.3, %bb.4
+    liveins: $rbx, $r14
+
+    EH_LABEL <mcsymbol Ltmp0>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rbx, renamable $r14 = STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), killed renamable $r14, renamable $r14(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol Ltmp1>
+    JMP_1 %bb.3
+
+  bb.2:
+    successors: %bb.5, %bb.4
+    liveins: $rbx, $r14
+
+    EH_LABEL <mcsymbol Ltmp2>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rbx, renamable $r14 = STATEPOINT 0, 0, 0, @bar, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), killed renamable $r14, renamable $r14(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol Ltmp3>
+    JMP_1 %bb.5
+
+  bb.3:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+  bb.4 (landing-pad):
+    liveins: $rax, $rdx, $r14
+
+    EH_LABEL <mcsymbol Ltmp4>
+    $rax = COPY killed renamable $r14
+    RET 0, $rax
+
+  bb.5:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+...

llvm/test/CodeGen/X86/statepoint-vreg-folding.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -run-pass=greedy -o - %s | FileCheck %s
+
+--- |
+  ; ModuleID = 'folding.ll'
+  source_filename = "folding.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @func()
+
+  define i32 @test_spill(i32 addrspace(1)* %arg00, i32 addrspace(1)* %arg01, i32 addrspace(1)* %arg02, i32 addrspace(1)* %arg03, i32 addrspace(1)* %arg04, i32 addrspace(1)* %arg05, i32 addrspace(1)* %arg06, i32 addrspace(1)* %arg07, i32 addrspace(1)* %arg08) gc "statepoint-example" {
+    %token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i32 addrspace(1)* %arg00, i32 addrspace(1)* %arg01, i32 addrspace(1)* %arg02, i32 addrspace(1)* %arg03, i32 addrspace(1)* %arg04, i32 addrspace(1)* %arg05, i32 addrspace(1)* %arg06, i32 addrspace(1)* %arg07, i32 addrspace(1)* %arg08) ]
+    %rel00 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 0, i32 0) ; (%arg00, %arg00)
+    %rel01 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 1, i32 1) ; (%arg01, %arg01)
+    %rel02 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 2, i32 2) ; (%arg02, %arg02)
+    %rel03 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 3, i32 3) ; (%arg03, %arg03)
+    %rel04 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 4, i32 4) ; (%arg04, %arg04)
+    %rel05 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 5, i32 5) ; (%arg05, %arg05)
+    %rel06 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 6, i32 6) ; (%arg06, %arg06)
+    %rel07 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 7, i32 7) ; (%arg07, %arg07)
+    %rel08 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 8, i32 8) ; (%arg08, %arg08)
+    %gep00 = getelementptr i32, i32 addrspace(1)* %rel00, i64 1
+    %gep01 = getelementptr i32, i32 addrspace(1)* %rel01, i64 2
+    %gep02 = getelementptr i32, i32 addrspace(1)* %rel02, i64 3
+    %gep03 = getelementptr i32, i32 addrspace(1)* %rel03, i64 4
+    %gep04 = getelementptr i32, i32 addrspace(1)* %rel04, i64 5
+    %gep05 = getelementptr i32, i32 addrspace(1)* %rel05, i64 6
+    %gep06 = getelementptr i32, i32 addrspace(1)* %rel06, i64 7
+    %gep07 = getelementptr i32, i32 addrspace(1)* %rel07, i64 8
+    %gep08 = getelementptr i32, i32 addrspace(1)* %rel08, i64 9
+    %val00 = load i32, i32 addrspace(1)* %gep00, align 4
+    %val01 = load i32, i32 addrspace(1)* %gep01, align 4
+    %sum01 = add i32 %val00, %val01
+    %val02 = load i32, i32 addrspace(1)* %gep02, align 4
+    %sum02 = add i32 %sum01, %val02
+    %val03 = load i32, i32 addrspace(1)* %gep03, align 4
+    %sum03 = add i32 %sum02, %val03
+    %val04 = load i32, i32 addrspace(1)* %gep04, align 4
+    %sum04 = add i32 %sum03, %val04
+    %val05 = load i32, i32 addrspace(1)* %gep05, align 4
+    %sum05 = add i32 %sum04, %val05
+    %val06 = load i32, i32 addrspace(1)* %gep06, align 4
+    %sum06 = add i32 %sum05, %val06
+    %val07 = load i32, i32 addrspace(1)* %gep07, align 4
+    %sum07 = add i32 %sum06, %val07
+    %val08 = load i32, i32 addrspace(1)* %gep08, align 4
+    %sum08 = add i32 %sum07, %val08
+    ret i32 %sum08
+  }
+
+  ; Function Attrs: nounwind readonly
+  declare i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token, i32 immarg, i32 immarg) #0
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_spill
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: gr64, preferred-register: '' }
+  - { id: 2, class: gr64, preferred-register: '' }
+  - { id: 3, class: gr64, preferred-register: '' }
+  - { id: 4, class: gr64, preferred-register: '' }
+  - { id: 5, class: gr64, preferred-register: '' }
+  - { id: 6, class: gr64, preferred-register: '' }
+  - { id: 7, class: gr64, preferred-register: '' }
+  - { id: 8, class: gr64, preferred-register: '' }
+  - { id: 9, class: gr64, preferred-register: '' }
+  - { id: 10, class: gr64, preferred-register: '' }
+  - { id: 11, class: gr64, preferred-register: '' }
+  - { id: 12, class: gr64, preferred-register: '' }
+  - { id: 13, class: gr64, preferred-register: '' }
+  - { id: 14, class: gr64, preferred-register: '' }
+  - { id: 15, class: gr64, preferred-register: '' }
+  - { id: 16, class: gr64, preferred-register: '' }
+  - { id: 17, class: gr64, preferred-register: '' }
+  - { id: 18, class: gr32, preferred-register: '' }
+  - { id: 19, class: gr32, preferred-register: '' }
+  - { id: 20, class: gr32, preferred-register: '' }
+  - { id: 21, class: gr32, preferred-register: '' }
+  - { id: 22, class: gr32, preferred-register: '' }
+  - { id: 23, class: gr32, preferred-register: '' }
+  - { id: 24, class: gr32, preferred-register: '' }
+  - { id: 25, class: gr32, preferred-register: '' }
+  - { id: 26, class: gr32, preferred-register: '' }
+liveins:
+  - { reg: '$rdi', virtual-reg: '%0' }
+  - { reg: '$rsi', virtual-reg: '%1' }
+  - { reg: '$rdx', virtual-reg: '%2' }
+  - { reg: '$rcx', virtual-reg: '%3' }
+  - { reg: '$r8', virtual-reg: '%4' }
+  - { reg: '$r9', virtual-reg: '%5' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    8
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:
+  - { id: 0, type: default, offset: 16, size: 8, alignment: 16, stack-id: default,
+      isImmutable: true, isAliased: false, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, type: default, offset: 8, size: 8, alignment: 8, stack-id: default,
+      isImmutable: true, isAliased: false, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 2, type: default, offset: 0, size: 8, alignment: 16, stack-id: default,
+      isImmutable: true, isAliased: false, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $rdi, $rsi, $rdx, $rcx, $r8, $r9
+
+    ; CHECK-LABEL: name: test_spill
+    ; CHECK: liveins: $rdi, $rsi, $rdx, $rcx, $r8, $r9
+    ; CHECK: MOV64mr %stack.0, 1, $noreg, 0, $noreg, $r9 :: (store 8 into %stack.0)
+    ; CHECK: MOV64mr %stack.1, 1, $noreg, 0, $noreg, $r8 :: (store 8 into %stack.1)
+    ; CHECK: MOV64mr %stack.2, 1, $noreg, 0, $noreg, $rcx :: (store 8 into %stack.2)
+    ; CHECK: [[R1:%[0-9]+]]:gr64 = COPY $rdx
+    ; CHECK: [[R2:%[0-9]+]]:gr64 = COPY $rsi
+    ; CHECK: [[R3:%[0-9]+]]:gr64 = COPY $rdi
+    ; CHECK: [[R4:%[0-9]+]]:gr64 = MOV64rm %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.0, align 16)
+    ; CHECK: [[R5:%[0-9]+]]:gr64 = MOV64rm %fixed-stack.1, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.1)
+    ; CHECK: [[R6:%[0-9]+]]:gr64 = MOV64rm %fixed-stack.2, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.2, align 16)
+    ; CHECK: ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: [[R6]]:gr64, [[R5]]:gr64, [[R4]]:gr64, [[R1]]:gr64, [[R2]]:gr64, [[R3]]:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, [[R6]], [[R6]](tied-def 0), [[R5]], [[R5]](tied-def 1), [[R4]], [[R4]](tied-def 2), 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, 1, 8, %stack.1, 0, 1, 8, %stack.1, 0, 1, 8, %stack.2, 0, 1, 8, %stack.2, 0, [[R1]], [[R1]](tied-def 3), [[R2]], [[R2]](tied-def 4), [[R3]], [[R3]](tied-def 5), csr_64, implicit-def $rsp, implicit-def $ssp :: (load store 8 on %stack.0), (load store 8 on %stack.1), (load store 8 on %stack.2)
+    ; CHECK: ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: [[RES:%[0-9]+]]:gr32 = MOV32rm [[R3]], 1, $noreg, 4, $noreg :: (load 4 from %ir.gep00, addrspace 1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[R2]], 1, $noreg, 8, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep01, addrspace 1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[R1]], 1, $noreg, 12, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep02, addrspace 1)
+    ; CHECK: [[MOV64rm:%[0-9]+]]:gr64 = MOV64rm %stack.2, 1, $noreg, 0, $noreg :: (load 8 from %stack.2)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[MOV64rm]], 1, $noreg, 16, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep03, addrspace 1)
+    ; CHECK: [[MOV64rm1:%[0-9]+]]:gr64 = MOV64rm %stack.1, 1, $noreg, 0, $noreg :: (load 8 from %stack.1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[MOV64rm1]], 1, $noreg, 20, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep04, addrspace 1)
+    ; CHECK: [[MOV64rm2:%[0-9]+]]:gr64 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[MOV64rm2]], 1, $noreg, 24, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep05, addrspace 1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[R4]], 1, $noreg, 28, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep06, addrspace 1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[R5]], 1, $noreg, 32, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep07, addrspace 1)
+    ; CHECK: [[RES]]:gr32 = ADD32rm [[RES]], [[R6]], 1, $noreg, 36, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep08, addrspace 1)
+    ; CHECK: $eax = COPY [[RES]]
+    ; CHECK: RET 0, $eax
+    %12:gr64 = COPY $r9
+    %13:gr64 = COPY $r8
+    %14:gr64 = COPY $rcx
+    %15:gr64 = COPY $rdx
+    %16:gr64 = COPY $rsi
+    %17:gr64 = COPY $rdi
+    %11:gr64 = MOV64rm %fixed-stack.2, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.2, align 16)
+    %10:gr64 = MOV64rm %fixed-stack.1, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.1)
+    %9:gr64 = MOV64rm %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.0, align 16)
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    %9:gr64, %10:gr64, %11:gr64, %12:gr64, %13:gr64, %14:gr64, %15:gr64, %16:gr64, %17:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, %9, %9(tied-def 0), %10, %10(tied-def 1), %11, %11(tied-def 2), %12, %12(tied-def 3), %13, %13(tied-def 4), %14, %14(tied-def 5), %15, %15(tied-def 6), %16, %16(tied-def 7), %17, %17(tied-def 8), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    %20:gr32 = MOV32rm %17, 1, $noreg, 4, $noreg :: (load 4 from %ir.gep00, addrspace 1)
+    %20:gr32 = ADD32rm %20, %16, 1, $noreg, 8, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep01, addrspace 1)
+    %20:gr32 = ADD32rm %20, %15, 1, $noreg, 12, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep02, addrspace 1)
+    %20:gr32 = ADD32rm %20, %14, 1, $noreg, 16, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep03, addrspace 1)
+    %20:gr32 = ADD32rm %20, %13, 1, $noreg, 20, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep04, addrspace 1)
+    %20:gr32 = ADD32rm %20, %12, 1, $noreg, 24, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep05, addrspace 1)
+    %20:gr32 = ADD32rm %20, %11, 1, $noreg, 28, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep06, addrspace 1)
+    %20:gr32 = ADD32rm %20, %10, 1, $noreg, 32, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep07, addrspace 1)
+    %20:gr32 = ADD32rm %20, %9, 1, $noreg, 36, $noreg, implicit-def dead $eflags :: (load 4 from %ir.gep08, addrspace 1)
+    $eax = COPY %20
+    RET 0, killed $eax
+
+...

llvm/test/CodeGen/X86/statepoint-vreg.ll

+; This run is to demonstrate what MIR SSA looks like.
+; RUN: llc -max-registers-for-gc-values=4 -stop-after finalize-isel < %s | FileCheck --check-prefix=CHECK-VREG %s
+; This run is to demonstrate register allocator work.
+; RUN: llc -max-registers-for-gc-values=4 -stop-after virtregrewriter < %s | FileCheck --check-prefix=CHECK-PREG %s
+; This run is to demonstrate resulting assembly/stackmaps.
+; NOTE: When D81647 is landed this run line will need to be adjusted!
+; RUN: llc -max-registers-for-gc-values=4 -fixup-allow-gcptr-in-csr=true < %s | FileCheck --check-prefix=CHECK-ASM %s
+
+target datalayout = "e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-linux-gnu"
+
+declare i1 @return_i1()
+declare void @func()
+declare void @"some_call"(i64 addrspace(1)*)
+declare void @consume(i32 addrspace(1)*)
+declare void @consume2(i32 addrspace(1)*, i32 addrspace(1)*)
+declare void @consume5(i32 addrspace(1)*, i32 addrspace(1)*, i32 addrspace(1)*, i32 addrspace(1)*, i32 addrspace(1)*)
+declare void @use1(i32 addrspace(1)*, i8 addrspace(1)*)
+declare i32 @"personality_function"()
+
+; test most simple relocate
+define i1 @test_relocate(i32 addrspace(1)* %a) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_relocate
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    %1:gr64 = STATEPOINT 0, 0, 0, @return_i1, 2, 0, 2, 0, 2, 0, %0, %0(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def $al
+; CHECK-VREG:    %2:gr8 = COPY $al
+; CHECK-VREG:    $rdi = COPY %1
+; CHECK-VREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_relocate
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @return_i1, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def $al
+; CHECK-PREG:    renamable $bpl = COPY killed $al
+; CHECK-PREG:    $rdi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_relocate:
+; CHECK-ASM:       # %bb.0:
+; CHECK-ASM-NEXT: pushq %rbp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: pushq %rax
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -24
+; CHECK-ASM-NEXT: .cfi_offset %rbp, -16
+; CHECK-ASM-NEXT: movq  %rdi, %rbx
+; CHECK-ASM-NEXT: callq return_i1
+; CHECK-ASM-NEXT:  .Ltmp0:
+; CHECK-ASM-NEXT: movl  %eax, %ebp
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: callq consume
+; CHECK-ASM-NEXT: movl  %ebp, %eax
+; CHECK-ASM-NEXT: addq  $8, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: popq  %rbp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+entry:
+  %safepoint_token = tail call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %a)]
+  %rel1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  %res1 = call zeroext i1 @llvm.experimental.gc.result.i1(token %safepoint_token)
+  call void @consume(i32 addrspace(1)* %rel1)
+  ret i1 %res1
+}
+
+; test pointer variables intermixed with pointer constants
+define void @test_mixed(i32 addrspace(1)* %a, i32 addrspace(1)* %b, i32 addrspace(1)* %c) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_mixed
+; CHECK-VREG:    %2:gr64 = COPY $rdx
+; CHECK-VREG:    %1:gr64 = COPY $rsi
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    %3:gr64, %4:gr64, %5:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, %2, %2(tied-def 0), 2, 0, 2, 0, %1, %1(tied-def 1), %0, %0(tied-def 2), csr_64
+; CHECK-VREG:    %6:gr32 = MOV32r0 implicit-def dead $eflags
+; CHECK-VREG:    %7:gr64 = SUBREG_TO_REG 0, killed %6, %subreg.sub_32bit
+; CHECK-VREG:    $rdi = COPY %5
+; CHECK-VREG:    $rsi = COPY %7
+; CHECK-VREG:    $rdx = COPY %4
+; CHECK-VREG:    $rcx = COPY %7
+; CHECK-VREG:    $r8 = COPY %3
+; CHECK-VREG:    CALL64pcrel32 @consume5, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit $rdx, implicit $rcx, implicit $r8, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_mixed
+; CHECK-PREG:    renamable $r14 = COPY $rdx
+; CHECK-PREG:    renamable $r15 = COPY $rsi
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    renamable $r14, renamable $r15, renamable $rbx = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, killed renamable $r14, renamable $r14(tied-def 0), 2, 0, 2, 0, killed renamable $r15, renamable $r15(tied-def 1), killed renamable $rbx, renamable $rbx(tied-def 2), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    $rdi = COPY killed renamable $rbx
+; CHECK-PREG:    dead $esi = MOV32r0 implicit-def dead $eflags, implicit-def $rsi
+; CHECK-PREG:    $rdx = COPY killed renamable $r15
+; CHECK-PREG:    dead $ecx = MOV32r0 implicit-def dead $eflags, implicit-def $rcx
+; CHECK-PREG:    $r8 = COPY killed renamable $r14
+; CHECK-PREG:    CALL64pcrel32 @consume5, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit $rdx, implicit killed $rcx, implicit killed $r8, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_mixed:
+; CHECK-ASM:        # %bb.0:                                # %entry
+; CHECK-ASM-NEXT: pushq %r15
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: pushq %r14
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -32
+; CHECK-ASM-NEXT: .cfi_offset %r14, -24
+; CHECK-ASM-NEXT: .cfi_offset %r15, -16
+; CHECK-ASM-NEXT: movq  %rdx, %r14
+; CHECK-ASM-NEXT: movq  %rsi, %r15
+; CHECK-ASM-NEXT: movq  %rdi, %rbx
+; CHECK-ASM-NEXT: callq func
+; CHECK-ASM-NEXT:.Ltmp1:
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: xorl  %esi, %esi
+; CHECK-ASM-NEXT: movq  %r15, %rdx
+; CHECK-ASM-NEXT: xorl  %ecx, %ecx
+; CHECK-ASM-NEXT: movq  %r14, %r8
+; CHECK-ASM-NEXT: callq consume5
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: popq  %r14
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: popq  %r15
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+entry:
+  %safepoint_token = tail call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %a, i32 addrspace(1)* null, i32 addrspace(1)* %b, i32 addrspace(1)* null, i32 addrspace(1)* %c)]
+  %rel1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  %rel2 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 1, i32 1)
+  %rel3 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 2, i32 2)
+  %rel4 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 3, i32 3)
+  %rel5 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 4, i32 4)
+  call void @consume5(i32 addrspace(1)* %rel1, i32 addrspace(1)* %rel2, i32 addrspace(1)* %rel3, i32 addrspace(1)* %rel4, i32 addrspace(1)* %rel5)
+  ret void
+}
+
+; same as above, but for alloca
+define i32 addrspace(1)* @test_alloca(i32 addrspace(1)* %ptr) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_alloca
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    MOV64mr %stack.0.alloca, 1, $noreg, 0, $noreg, %0 :: (store 8 into %ir.alloca)
+; CHECK-VREG:    %1:gr64 = STATEPOINT 0, 0, 0, @return_i1, 2, 0, 2, 0, 2, 0, %0, %0(tied-def 0), 0, %stack.0.alloca, 0, csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def $al :: (volatile load store 8 on %stack.0.alloca)
+; CHECK-VREG:    %2:gr8 = COPY $al
+; CHECK-VREG:    %3:gr64 = MOV64rm %stack.0.alloca, 1, $noreg, 0, $noreg :: (dereferenceable load 8 from %ir.alloca)
+; CHECK-VREG:    $rdi = COPY %1
+; CHECK-VREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_alloca
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    MOV64mr %stack.0.alloca, 1, $noreg, 0, $noreg, renamable $rbx :: (store 8 into %ir.alloca)
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @return_i1, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), 0, %stack.0.alloca, 0, csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def dead $al :: (volatile load store 8 on %stack.0.alloca)
+; CHECK-PREG:    renamable $r14 = MOV64rm %stack.0.alloca, 1, $noreg, 0, $noreg :: (dereferenceable load 8 from %ir.alloca)
+; CHECK-PREG:    $rdi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_alloca:
+; CHECK-ASM:       # %bb.0: # %entry
+; CHECK-ASM-NEXT: pushq %r14
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: pushq %rax
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -24
+; CHECK-ASM-NEXT: .cfi_offset %r14, -16
+; CHECK-ASM-NEXT: movq  %rdi, %rbx
+; CHECK-ASM-NEXT: movq  %rdi, (%rsp)
+; CHECK-ASM-NEXT: callq return_i1
+; CHECK-ASM-NEXT:  .Ltmp2:
+; CHECK-ASM-NEXT: movq  (%rsp), %r14
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: callq consume
+; CHECK-ASM-NEXT: movq  %r14, %rax
+; CHECK-ASM-NEXT: addq  $8, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 24
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: popq  %r14
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+entry:
+  %alloca = alloca i32 addrspace(1)*, align 8
+  store i32 addrspace(1)* %ptr, i32 addrspace(1)** %alloca
+  %safepoint_token = call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)** %alloca, i32 addrspace(1)* %ptr)]
+  %rel1 = load i32 addrspace(1)*, i32 addrspace(1)** %alloca
+  %rel2 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 1, i32 1)
+  call void @consume(i32 addrspace(1)* %rel2)
+  ret i32 addrspace(1)* %rel1
+}
+
+; test base != derived
+define void @test_base_derived(i32 addrspace(1)* %base, i32 addrspace(1)* %derived) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_base_derived
+; CHECK-VREG:    %1:gr64 = COPY $rsi
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    MOV64mr %stack.0, 1, $noreg, 0, $noreg, %0 :: (store 8 into %stack.0)
+; CHECK-VREG:    %2:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, %1(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+; CHECK-VREG:    $rdi = COPY %2
+; CHECK-VREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_base_derived
+; CHECK-PREG:    renamable $rbx = COPY $rsi
+; CHECK-PREG:    MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed renamable $rdi :: (store 8 into %stack.0)
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, killed renamable $rbx(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+; CHECK-PREG:    $rdi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_base_derived:
+; CHECK-ASM:       # %bb.0:
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: subq  $16, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -16
+; CHECK-ASM-NEXT: movq  %rsi, %rbx
+; CHECK-ASM-NEXT: movq  %rdi, 8(%rsp)
+; CHECK-ASM-NEXT: callq func
+; CHECK-ASM-NEXT:  .Ltmp3:
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: callq consume
+; CHECK-ASM-NEXT: addq  $16, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+  %safepoint_token = tail call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %base, i32 addrspace(1)* %derived)]
+  %reloc = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 1)
+  call void @consume(i32 addrspace(1)* %reloc)
+  ret void
+}
+
+; deopt GC pointer not present in GC args must be spilled
+define void @test_deopt_gcpointer(i32 addrspace(1)* %a, i32 addrspace(1)* %b) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_deopt_gcpointer
+; CHECK-VREG:    %1:gr64 = COPY $rsi
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    MOV64mr %stack.0, 1, $noreg, 0, $noreg, %0 :: (store 8 into %stack.0)
+; CHECK-VREG:    %2:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 1, 1, 8, %stack.0, 0, %1, %1(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+; CHECK-VREG:    $rdi = COPY %2
+; CHECK-VREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    RET 0
+
+; CHECK-PREG-LABEL: name:            test_deopt_gcpointer
+; CHECK-PREG:    renamable $rbx = COPY $rsi
+; CHECK-PREG:    MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed renamable $rdi :: (store 8 into %stack.0)
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 1, 1, 8, %stack.0, 0, killed renamable $rbx, renamable $rbx(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+; CHECK-PREG:    $rdi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_deopt_gcpointer:
+; CHECK-ASM:       # %bb.0:
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: subq  $16, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -16
+; CHECK-ASM-NEXT: movq  %rsi, %rbx
+; CHECK-ASM-NEXT: movq  %rdi, 8(%rsp)
+; CHECK-ASM-NEXT: callq func
+; CHECK-ASM-NEXT:  .Ltmp4:
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: callq consume
+; CHECK-ASM-NEXT: addq  $16, %rsp
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+  %safepoint_token = tail call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["deopt" (i32 addrspace(1)* %a), "gc-live" (i32 addrspace(1)* %b)]
+  %rel = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  call void @consume(i32 addrspace(1)* %rel)
+  ret void
+}
+
+;; Two gc.relocates of the same input, should require only a single spill/fill
+define void @test_gcrelocate_uniqueing(i32 addrspace(1)* %ptr) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_gcrelocate_uniqueing
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    %1:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 2, %0, 2, 4278124286, %0, %0(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    $rdi = COPY %1
+; CHECK-VREG:    $rsi = COPY %1
+; CHECK-VREG:    CALL64pcrel32 @consume2, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_gcrelocate_uniqueing
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 2, killed renamable $rbx, 2, 4278124286, renamable $rbx, renamable $rbx(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    $rdi = COPY renamable $rbx
+; CHECK-PREG:    $rsi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @consume2, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit killed $rsi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_gcrelocate_uniqueing:
+; CHECK-ASM:       # %bb.0:
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -16
+; CHECK-ASM-NEXT: movq  %rdi, %rbx
+; CHECK-ASM-NEXT: callq func
+; CHECK-ASM-NEXT: .Ltmp5:
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: movq  %rbx, %rsi
+; CHECK-ASM-NEXT: callq consume2
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+  %tok = tail call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["deopt" (i32 addrspace(1)* %ptr, i32 undef), "gc-live" (i32 addrspace(1)* %ptr, i32 addrspace(1)* %ptr)]
+  %a = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %tok, i32 0, i32 0)
+  %b = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %tok, i32 1, i32 1)
+  call void @consume2(i32 addrspace(1)* %a, i32 addrspace(1)* %b)
+  ret void
+}
+
+; Two gc.relocates of a bitcasted pointer should only require a single spill/fill
+define void @test_gcptr_uniqueing(i32 addrspace(1)* %ptr) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_gcptr_uniqueing
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+; CHECK-VREG:    %1:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 2, %0, 2, 4278124286, %0, %0(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+; CHECK-VREG:    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+; CHECK-VREG:    $rdi = COPY %1
+; CHECK-VREG:    $rsi = COPY %1
+; CHECK-VREG:    CALL64pcrel32 @use1, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-PREG-LABEL: name:            test_gcptr_uniqueing
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    renamable $rbx = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 2, killed renamable $rbx, 2, 4278124286, renamable $rbx, renamable $rbx(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    $rdi = COPY renamable $rbx
+; CHECK-PREG:    $rsi = COPY killed renamable $rbx
+; CHECK-PREG:    CALL64pcrel32 @use1, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit killed $rsi, implicit-def $rsp, implicit-def $ssp
+
+; CHECK-ASM-LABEL: test_gcptr_uniqueing:
+; CHECK-ASM:       # %bb.0:
+; CHECK-ASM-NEXT: pushq %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT: .cfi_offset %rbx, -16
+; CHECK-ASM-NEXT: movq  %rdi, %rbx
+; CHECK-ASM-NEXT: callq func
+; CHECK-ASM-NEXT:  .Ltmp6:
+; CHECK-ASM-NEXT: movq  %rbx, %rdi
+; CHECK-ASM-NEXT: movq  %rbx, %rsi
+; CHECK-ASM-NEXT: callq use1
+; CHECK-ASM-NEXT: popq  %rbx
+; CHECK-ASM-NEXT: .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT: retq
+  %ptr2 = bitcast i32 addrspace(1)* %ptr to i8 addrspace(1)*
+  %tok = tail call token (i64, i32, void ()*, i32, i32, ...)
+      @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["deopt" (i32 addrspace(1)* %ptr, i32 undef), "gc-live" (i32 addrspace(1)* %ptr, i8 addrspace(1)* %ptr2)]
+  %a = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %tok, i32 0, i32 0)
+  %b = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %tok, i32 1, i32 1)
+  call void @use1(i32 addrspace(1)* %a, i8 addrspace(1)* %b)
+  ret void
+}
+
+;
+; test non-local relocates
+define i1 @test_cross_bb(i32 addrspace(1)* %a, i1 %external_cond) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_cross_bb
+; CHECK-VREG:  bb.0.entry:
+; CHECK-VREG:    %1:gr32 = COPY $esi
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    %4:gr8 = COPY %1.sub_8bit
+; CHECK-VREG:    %2:gr64 = STATEPOINT 0, 0, 0, @return_i1, 2, 0, 2, 0, 2, 0, %0, %0(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def $al
+; CHECK-VREG:    %5:gr8 = COPY $al
+; CHECK-VREG:    %3:gr8 = COPY %5
+; CHECK-VREG:    TEST8ri killed %4, 1, implicit-def $eflags
+; CHECK-VREG:    JCC_1 %bb.2, 4, implicit $eflags
+; CHECK-VREG:    JMP_1 %bb.1
+; CHECK-VREG:  bb.1.left:
+; CHECK-VREG:    $rdi = COPY %2
+; CHECK-VREG:    CALL64pcrel32 @consume, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    $al = COPY %3
+; CHECK-VREG:    RET 0, $al
+; CHECK-VREG:  bb.2.right:
+; CHECK-VREG:    %6:gr8 = MOV8ri 1
+; CHECK-VREG:    $al = COPY %6
+; CHECK-VREG:    RET 0, $al
+
+entry:
+  %safepoint_token = tail call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %a)]
+  br i1 %external_cond, label %left, label %right
+
+left:
+  %call1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  %call2 = call zeroext i1 @llvm.experimental.gc.result.i1(token %safepoint_token)
+  call void @consume(i32 addrspace(1)* %call1)
+  ret i1 %call2
+
+right:
+  ret i1 true
+}
+
+; No need to check post-regalloc output as it is the same
+define i1 @duplicate_reloc() gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            duplicate_reloc
+; CHECK-VREG:  bb.0.entry:
+; CHECK-VREG:    STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    %0:gr8 = MOV8ri 1
+; CHECK-VREG:    $al = COPY %0
+; CHECK-VREG:    RET 0, $al
+
+; CHECK-ASM-LABEL: duplicate_reloc:
+; CHECK-ASM:       # %bb.0: # %entry
+; CHECK-ASM-NEXT:          pushq  %rax
+; CHECK-ASM-NEXT:          .cfi_def_cfa_offset 16
+; CHECK-ASM-NEXT:          callq  func
+; CHECK-ASM-NEXT:  .Ltmp8:
+; CHECK-ASM-NEXT:          callq  func
+; CHECK-ASM-NEXT:  .Ltmp9:
+; CHECK-ASM-NEXT:          movb $1, %al
+; CHECK-ASM-NEXT:          popq %rcx
+; CHECK-ASM-NEXT:          .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT:          retq
+entry:
+  %safepoint_token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* null, i32 addrspace(1)* null)]
+  %base = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  %derived = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 1)
+  %safepoint_token2 = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %base, i32 addrspace(1)* %derived)]
+  %base_reloc = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token2,  i32 0, i32 0)
+  %derived_reloc = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token2,  i32 0, i32 1)
+  %cmp1 = icmp eq i32 addrspace(1)* %base_reloc, null
+  %cmp2 = icmp eq i32 addrspace(1)* %derived_reloc, null
+  %cmp = and i1 %cmp1, %cmp2
+  ret i1 %cmp
+}
+
+; Vectors cannot go in VRegs
+; No need to check post-regalloc output as it is lowered using old scheme
+define <2 x i8 addrspace(1)*> @test_vector(<2 x i8 addrspace(1)*> %obj) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_vector
+; CHECK-VREG:    %0:vr128 = COPY $xmm0
+; CHECK-VREG:    MOVAPSmr %stack.0, 1, $noreg, 0, $noreg, %0 :: (store 16 into %stack.0)
+; CHECK-VREG:    STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 1, 16, %stack.0, 0, 1, 16, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 16 on %stack.0)
+; CHECK-VREG:    %1:vr128 = MOVAPSrm %stack.0, 1, $noreg, 0, $noreg :: (load 16 from %stack.0)
+; CHECK-VREG:    $xmm0 = COPY %1
+; CHECK-VREG:    RET 0, $xmm0
+
+; CHECK-ASM-LABEL: test_vector:
+; CHECK-ASM:       # %bb.0: # %entry
+; CHECK-ASM-NEXT:          subq $24, %rsp
+; CHECK-ASM-NEXT:          .cfi_def_cfa_offset 32
+; CHECK-ASM-NEXT:          movaps %xmm0, (%rsp)
+; CHECK-ASM-NEXT:          callq  func
+; CHECK-ASM-NEXT:  .Ltmp10:
+; CHECK-ASM-NEXT:          movaps (%rsp), %xmm0
+; CHECK-ASM-NEXT:          addq $24, %rsp
+; CHECK-ASM-NEXT:          .cfi_def_cfa_offset 8
+; CHECK-ASM-NEXT:          retq
+entry:
+  %safepoint_token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (<2 x i8 addrspace(1)*> %obj)]
+  %obj.relocated = call coldcc <2 x i8 addrspace(1)*> @llvm.experimental.gc.relocate.v2p1i8(token %safepoint_token, i32 0, i32 0) ; (%obj, %obj)
+  ret <2 x i8 addrspace(1)*> %obj.relocated
+}
+
+
+; test limit on amount of vregs
+define void @test_limit(i32 addrspace(1)* %a, i32 addrspace(1)* %b, i32 addrspace(1)* %c, i32 addrspace(1)* %d, i32 addrspace(1)*  %e) gc "statepoint-example" {
+; CHECK-VREG-LABEL: name:            test_limit
+; CHECK-VREG:    %4:gr64 = COPY $r8
+; CHECK-VREG:    %3:gr64 = COPY $rcx
+; CHECK-VREG:    %2:gr64 = COPY $rdx
+; CHECK-VREG:    %1:gr64 = COPY $rsi
+; CHECK-VREG:    %0:gr64 = COPY $rdi
+; CHECK-VREG:    MOV64mr %stack.0, 1, $noreg, 0, $noreg, %0 :: (store 8 into %stack.0)
+; CHECK-VREG:    %5:gr64, %6:gr64, %7:gr64, %8:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, %4, %4(tied-def 0), %3, %3(tied-def 1), %2, %2(tied-def 2), %1, %1(tied-def 3), 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+; CHECK-VREG:    %9:gr64 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+; CHECK-VREG:    $rdi = COPY %9
+; CHECK-VREG:    $rsi = COPY %8
+; CHECK-VREG:    $rdx = COPY %7
+; CHECK-VREG:    $rcx = COPY %6
+; CHECK-VREG:    $r8 = COPY %5
+; CHECK-VREG:    CALL64pcrel32 @consume5, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit $rsi, implicit $rdx, implicit $rcx, implicit $r8, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    RET 0
+entry:
+  %safepoint_token = tail call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @func, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %a, i32 addrspace(1)* %b, i32 addrspace(1)* %c, i32 addrspace(1)* %d, i32 addrspace(1)* %e)]
+  %rel1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 0, i32 0)
+  %rel2 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 1, i32 1)
+  %rel3 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 2, i32 2)
+  %rel4 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 3, i32 3)
+  %rel5 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 4, i32 4)
+  call void @consume5(i32 addrspace(1)* %rel1, i32 addrspace(1)* %rel2, i32 addrspace(1)* %rel3, i32 addrspace(1)* %rel4, i32 addrspace(1)* %rel5)
+  ret void
+}
+
+define i64 addrspace(1)* @test_basic_invoke(i64 addrspace(1)* %obj,
+                                            i64 addrspace(1)* %obj1)
+; CHECK-VREG-LABEL: name:            test_basic_invoke
+; CHECK-VREG:  bb.0.entry:
+; CHECK-VREG:    %2:gr64 = COPY $rsi
+; CHECK-VREG:    %1:gr64 = COPY $rdi
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    $rdi = COPY %1
+; CHECK-VREG:    %3:gr64, %4:gr64 = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, %2, %2(tied-def 0), %1, %1(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    JMP_1 %bb.1
+; CHECK-VREG:  bb.1.invoke_safepoint_normal_dest:
+; CHECK-VREG:    %0:gr64 = COPY %4
+; CHECK-VREG:  bb.2.normal_return:
+; CHECK-VREG:    $rax = COPY %0
+; CHECK-VREG:    RET 0, $rax
+; CHECK-VREG:  bb.3.exceptional_return (landing-pad):
+; CHECK-VREG:    liveins: $rax, $rdx
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    %6:gr64 = COPY killed $rdx
+; CHECK-VREG:    %5:gr64 = COPY killed $rax
+; CHECK-VREG:    $rax = COPY %3
+; CHECK-VREG:    RET 0, $rax
+
+; CHECK-PREG-LABEL: name:            test_basic_invoke
+; CHECK-PREG:  bb.0.entry:
+; CHECK-PREG:    renamable $r14 = COPY $rsi
+; CHECK-PREG:    renamable $rbx = COPY $rdi
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rdi = COPY renamable $rbx
+; CHECK-PREG:    renamable $r14, renamable $rbx = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, killed renamable $r14, renamable $r14(tied-def 0), killed renamable $rbx, renamable $rbx(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    JMP_1 %bb.1
+; CHECK-PREG:  bb.1.invoke_safepoint_normal_dest:
+; CHECK-PREG:    successors: %bb.2(0x80000000)
+; CHECK-PREG:  bb.2.normal_return:
+; CHECK-PREG:    $rax = COPY killed renamable $rbx
+; CHECK-PREG:    RET 0, $rax
+; CHECK-PREG:  bb.3.exceptional_return (landing-pad):
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rax = COPY killed renamable $r14
+; CHECK-PREG:    RET 0, $rax
+gc "statepoint-example" personality i32 ()* @"personality_function" {
+entry:
+  %0 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %obj, i32 0, i32 0) ["gc-live" (i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1), "deopt" (i32 0, i32 -1, i32 0, i32 0, i32 0)]
+          to label %invoke_safepoint_normal_dest unwind label %exceptional_return
+
+invoke_safepoint_normal_dest:
+  %obj.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 0, i32 0)
+  %obj1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 1, i32 1)
+  br label %normal_return
+
+normal_return:
+  ret i64 addrspace(1)* %obj.relocated
+
+exceptional_return:
+  %landing_pad = landingpad token
+          cleanup
+  %obj.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 0, i32 0)
+  %obj1.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 1, i32 1)
+  ret i64 addrspace(1)* %obj1.relocated1
+}
+
+define i64 addrspace(1)* @test_invoke_same_val(i1 %cond, i64 addrspace(1)* %val1, i64 addrspace(1)* %val2, i64 addrspace(1)* %val3)
+; CHECK-VREG-LABEL: name:            test_invoke_same_val
+; CHECK-VREG:  bb.0.entry:
+; CHECK-VREG:    %9:gr64 = COPY $rcx
+; CHECK-VREG:    %8:gr64 = COPY $rdx
+; CHECK-VREG:    %7:gr64 = COPY $rsi
+; CHECK-VREG:    %6:gr32 = COPY $edi
+; CHECK-VREG:    %10:gr8 = COPY %6.sub_8bit
+; CHECK-VREG:    TEST8ri %10, 1, implicit-def $eflags
+; CHECK-VREG:    JCC_1 %bb.3, 4, implicit $eflags
+; CHECK-VREG:    JMP_1 %bb.1
+; CHECK-VREG:  bb.1.left:
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    $rdi = COPY %7
+; CHECK-VREG:    %15:gr64, %16:gr64 = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 0, %8, %8(tied-def 0), %7, %7(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    JMP_1 %bb.2
+; CHECK-VREG:  bb.2.left.relocs:
+; CHECK-VREG:    %0:gr64 = COPY %16
+; CHECK-VREG:    %1:gr64 = COPY %15
+; CHECK-VREG:    JMP_1 %bb.5
+; CHECK-VREG:  bb.3.right:
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    $rdi = COPY %7
+; CHECK-VREG:    %11:gr64, %12:gr64 = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 0, %9, %9(tied-def 0), %8, %8(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    JMP_1 %bb.4
+; CHECK-VREG:  bb.4.right.relocs:
+; CHECK-VREG:    %2:gr64 = COPY %12
+; CHECK-VREG:    %3:gr64 = COPY %11
+; CHECK-VREG:  bb.5.normal_return:
+; CHECK-VREG:    %4:gr64 = PHI %3, %bb.4, %0, %bb.2
+; CHECK-VREG:    %5:gr64 = PHI %2, %bb.4, %1, %bb.2
+; CHECK-VREG:    TEST8ri %10, 1, implicit-def $eflags
+; CHECK-VREG:    %19:gr64 = CMOV64rr %5, %4, 5, implicit $eflags
+; CHECK-VREG:    $rax = COPY %19
+; CHECK-VREG:    RET 0, $rax
+; CHECK-VREG:  bb.6.exceptional_return.left (landing-pad):
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    %18:gr64 = COPY killed $rdx
+; CHECK-VREG:    %17:gr64 = COPY killed $rax
+; CHECK-VREG:    $rax = COPY %16
+; CHECK-VREG:    RET 0, $rax
+; CHECK-VREG:  bb.7.exceptional_return.right (landing-pad):
+; CHECK-VREG:    EH_LABEL <mcsymbol >
+; CHECK-VREG:    %14:gr64 = COPY killed $rdx
+; CHECK-VREG:    %13:gr64 = COPY killed $rax
+; CHECK-VREG:    $rax = COPY %12
+; CHECK-VREG:    RET 0, $rax
+
+; CHECK-PREG-LABEL: name:            test_invoke_same_val
+; CHECK-PREG:  bb.0.entry:
+; CHECK-PREG:    renamable $r15 = COPY $rcx
+; CHECK-PREG:    renamable $rbx = COPY $rdx
+; CHECK-PREG:    renamable $rbp = COPY $rsi
+; CHECK-PREG:    renamable $r14d = COPY $edi
+; CHECK-PREG:    TEST8ri renamable $r14b, 1, implicit-def $eflags
+; CHECK-PREG:    JCC_1 %bb.3, 4, implicit killed $eflags
+; CHECK-PREG:    JMP_1 %bb.1
+; CHECK-PREG:  bb.1.left:
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rdi = COPY renamable $rbp
+; CHECK-PREG:    renamable $rbx, renamable $rbp = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), killed renamable $rbp, renamable $rbp(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    JMP_1 %bb.2
+; CHECK-PREG:  bb.2.left.relocs:
+; CHECK-PREG:    JMP_1 %bb.5
+; CHECK-PREG:  bb.3.right:
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rdi = COPY killed renamable $rbp
+; CHECK-PREG:    renamable $r15, renamable $rbx = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 0, killed renamable $r15, renamable $r15(tied-def 0), killed renamable $rbx, renamable $rbx(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    JMP_1 %bb.4
+; CHECK-PREG:  bb.4.right.relocs:
+; CHECK-PREG:    renamable $rbp = COPY killed renamable $r15
+; CHECK-PREG:  bb.5.normal_return:
+; CHECK-PREG:    TEST8ri renamable $r14b, 1, implicit-def $eflags, implicit killed $r14d
+; CHECK-PREG:    renamable $rbp = CMOV64rr killed renamable $rbp, killed renamable $rbx, 4, implicit killed $eflags
+; CHECK-PREG:    $rax = COPY killed renamable $rbp
+; CHECK-PREG:    RET 0, $rax
+; CHECK-PREG:  bb.6.exceptional_return.left (landing-pad):
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rax = COPY killed renamable $rbp
+; CHECK-PREG:    RET 0, $rax
+; CHECK-PREG:  bb.7.exceptional_return.right (landing-pad):
+; CHECK-PREG:    EH_LABEL <mcsymbol >
+; CHECK-PREG:    $rax = COPY killed renamable $rbx
+; CHECK-PREG:    RET 0, $rax
+  gc "statepoint-example" personality i32 ()* @"personality_function" {
+entry:
+  br i1 %cond, label %left, label %right
+
+left:
+  %sp1 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 0) ["gc-live"(i64 addrspace(1)* %val1, i64 addrspace(1)* %val2)]
+           to label %left.relocs unwind label %exceptional_return.left
+
+left.relocs:
+  %val1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp1, i32 0, i32 0)
+  %val2.relocated_left = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp1, i32 1, i32 1)
+  br label %normal_return
+
+right:
+  %sp2 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 0) ["gc-live"(i64 addrspace(1)* %val2, i64 addrspace(1)* %val3)]
+           to label %right.relocs unwind label %exceptional_return.right
+
+right.relocs:
+  %val2.relocated_right = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp2, i32 0, i32 0)
+  %val3.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp2, i32 1, i32 1)
+  br label %normal_return
+
+normal_return:
+  %a1 = phi i64 addrspace(1)* [%val1.relocated, %left.relocs], [%val3.relocated, %right.relocs]
+  %a2 = phi i64 addrspace(1)* [%val2.relocated_left, %left.relocs], [%val2.relocated_right, %right.relocs]
+  %ret = select i1 %cond, i64 addrspace(1)* %a1, i64 addrspace(1)* %a2
+  ret i64 addrspace(1)* %ret
+
+exceptional_return.left:
+  %landing_pad = landingpad token
+          cleanup
+  %val.relocated2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 0, i32 0)
+  ret i64 addrspace(1)* %val.relocated2
+
+exceptional_return.right:
+  %landing_pad1 = landingpad token
+          cleanup
+  %val.relocated3 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad1, i32 0, i32 0)
+  ret i64 addrspace(1)* %val.relocated3
+}
+
+declare token @llvm.experimental.gc.statepoint.p0f_i1f(i64, i32, i1 ()*, i32, i32, ...)
+declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64, i32, void ()*, i32, i32, ...)
+declare token @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...)
+declare i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token, i32, i32)
+declare i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token, i32, i32)
+declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32, i32)
+declare <2 x i8 addrspace(1)*> @llvm.experimental.gc.relocate.v2p1i8(token, i32, i32)
+declare i1 @llvm.experimental.gc.result.i1(token)
+
+; CHECK-ASM-LABEL: .section .llvm_stackmaps
+; CHECK-ASM-NEXT:  __LLVM_StackMaps:
+; Entry for test_relocate
+; CHECK-ASM:          .quad 0
+; CHECK-ASM-NEXT:       .long .Ltmp0-test_relocate
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  5
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 4 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+;  Entry for test_mixed
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp1-test_mixed
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  11
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 4 Register $r14
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  14
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Register $r14
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  14
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 6 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 7 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 8 Register $r15
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  15
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 9 Register $r15
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  15
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 10 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 11 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_alloca
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp2-test_alloca
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  6
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 4 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 6 Direct $rsp + 0
+; CHECK-ASM-NEXT: .byte 2
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  7
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_base_derive
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp3-test_base_derived
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  5
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 4 Indirect $rsp + 8
+; CHECK-ASM-NEXT: .byte 3
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  7
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 8
+; Location 5 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_deopt_gcpointer
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp4-test_deopt_gcpointer
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  6
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 1
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 1
+; Location 4Indirect $rsp + 8
+; CHECK-ASM-NEXT: .byte 3
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  7
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 8
+; Location 5 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 6
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_gcrelocate_uniqueing
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp5-test_gcrelocate_uniqueing
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  7
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 2
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 2
+; Location 4 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Constant Index 0
+; CHECK-ASM-NEXT: .byte 5
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 6 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 7 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_gcptr_uniqueing
+; CHECK-ASM:      .long .Ltmp6-test_gcptr_uniqueing
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  7
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 2
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 2
+; Location 4 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Constant Index 0
+; CHECK-ASM-NEXT: .byte 5
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 6 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 7 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Entry for test_cross_bb
+; CHECK-ASM:      .quad 0
+; CHECK-ASM-NEXT: .long .Ltmp7-test_cross_bb
+; CHECK-ASM-NEXT: .short  0
+; Num locations
+; CHECK-ASM-NEXT: .short  5
+; Location 1 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 2 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 3 Constant 0
+; CHECK-ASM-NEXT: .byte 4
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 4 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0
+; Location 5 Register $rbx
+; CHECK-ASM-NEXT: .byte 1
+; CHECK-ASM-NEXT: .byte 0
+; CHECK-ASM-NEXT: .short  8
+; CHECK-ASM-NEXT: .short  3
+; CHECK-ASM-NEXT: .short  0
+; CHECK-ASM-NEXT: .long 0

llvm/test/CodeGen/X86/statepoint-vreg.mir

+# NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=true -start-after=finalize-isel | FileCheck %s
+
+--- |
+  ; ModuleID = 'test/CodeGen/X86/statepoint-vreg.ll'
+  source_filename = "test/CodeGen/X86/statepoint-vreg.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @bar()
+
+  define i32 @test_basic(i32 addrspace(1)* %obj1, i32 addrspace(1)* %obj2) gc "statepoint-example" {
+  ; CHECK-LABEL: test_basic:
+  ; CHECK:       # %bb.0:
+  ; CHECK-NEXT:    pushq %r14
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 16
+  ; CHECK-NEXT:    pushq %rbx
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 24
+  ; CHECK-NEXT:    pushq %rax
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 32
+  ; CHECK-NEXT:    .cfi_offset %rbx, -24
+  ; CHECK-NEXT:    .cfi_offset %r14, -16
+  ; CHECK-NEXT:    movq %rsi, %r14
+  ; CHECK-NEXT:    movq %rdi, %rbx
+  ; CHECK-NEXT:    callq bar
+  ; CHECK-NEXT:  .Ltmp0:
+  ; CHECK-NEXT:    movl (%rbx), %eax
+  ; CHECK-NEXT:    addl (%r14), %eax
+  ; CHECK-NEXT:    addq $8, %rsp
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 24
+  ; CHECK-NEXT:    popq %rbx
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 16
+  ; CHECK-NEXT:    popq %r14
+  ; CHECK-NEXT:    .cfi_def_cfa_offset 8
+  ; CHECK-NEXT:    retq
+    %token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 2882400000, i32 0, void ()* @bar, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i32 addrspace(1)* %obj1, i32 addrspace(1)* %obj2) ]
+    %rel1 = call coldcc i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 0, i32 0) ; (%obj1, %obj1)
+    %rel2 = call coldcc i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %token, i32 1, i32 1) ; (%obj2, %obj2)
+    %a = load i32, i32 addrspace(1)* %rel1, align 4
+    %b = load i32, i32 addrspace(1)* %rel2, align 4
+    %c = add i32 %a, %b
+    ret i32 %c
+  }
+
+  ; CHECK-LABEL:  __LLVM_StackMaps:
+  ; CHECK-NEXT:    .byte  3
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  1
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .long  1
+  ; CHECK-NEXT:    .quad  test_basic
+  ; CHECK-NEXT:    .quad  24
+  ; CHECK-NEXT:    .quad  1
+  ; CHECK-NEXT:    .quad  2882400000
+  ; CHECK-NEXT:    .long  .Ltmp0-test_basic
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .byte  4
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  4
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  4
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  1
+  ; CHECK-NEXT:    .byte  4
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  1
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 14
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  1
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 14
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  1
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 3
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .byte  1
+  ; CHECK-NEXT:    .byte  0
+  ; CHECK-NEXT:    .short 8
+  ; CHECK-NEXT:    .short 3
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .long  0
+  ; CHECK-NEXT:    .p2align 3
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .short 0
+  ; CHECK-NEXT:    .p2align 3
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token, i32 immarg, i32 immarg) #0
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_basic
+alignment:       16
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: gr64, preferred-register: '' }
+  - { id: 2, class: gr64, preferred-register: '' }
+  - { id: 3, class: gr64, preferred-register: '' }
+  - { id: 4, class: gr32, preferred-register: '' }
+  - { id: 5, class: gr32, preferred-register: '' }
+liveins:
+  - { reg: '$rdi', virtual-reg: '%0' }
+  - { reg: '$rsi', virtual-reg: '%1' }
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $rdi, $rsi
+
+    %1:gr64 = COPY $rsi
+    %0:gr64 = COPY $rdi
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    %2:gr64, %3:gr64 = STATEPOINT 2882400000, 0, 0, @bar, 2, 0, 2, 0, 2, 1, 2, 0, %1, %1(tied-def 0), %0, %0(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    %4:gr32 = MOV32rm killed %3, 1, $noreg, 0, $noreg :: (load 4 from %ir.rel1, addrspace 1)
+    %5:gr32 = ADD32rm %4, killed %2, 1, $noreg, 0, $noreg, implicit-def dead $eflags :: (load 4 from %ir.rel2, addrspace 1)
+    $eax = COPY %5
+    RET 0, $eax
+
+...