MIR Statepoint refactoring. Part 3: Spill GC Ptr regs.

[dantrushin]

Summary

Extend FixupStatepointCallerSaved pass with ability to spill
statepoint GC pointer arguments (optionally allowing them on CSRs).
Special handling is required for invoke statepoints, because at MI
level single landing pad may be shared my multiple statepoints, so
we must ensure we spill landing pad's live-ins into the same stack
slots.

Full change set is available at D81603.

[skatkov]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

53

Add a comment that it is for debugging purpose. Please mention that not specifying this option means no restrictions at all.

127

SO.getNumDeoptArgsIdx() ?

628

If I understand correctly you are processing starting operands and they are defs.
so the condition MO.isDef() is actually exit condition of the loop (we processed all defs).
What else we can see as def? In other words what MO.isReg condition means?

643

don't you want to rephrase it as

unsigned LastStatepointwithRegsIdx = MaxStatepointsWithRegs.getNumOccurrences() ? MaxStatepointsWithRegs : Statepoints.size();

for (MachineInstr *I : Statepoints)
  Changed |= SPP.process(*I, ++NumStatepoints <= LastStatepointwithRegsIdx);

[skatkov]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

131

What if deopt args contains gc pointer?

450

in a separate function?

[dantrushin]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

53

Will do.

127

Yep.

131

At this point, we can not know.
And your code handles all deopt args uniformly already ;)
Mine adds some more restrictions

450

Ok

628

MO.isDef() asserts that MO.isReg(), so one always must check later before checking former

643

For upstream, maybe.
But since I have downstream changes here, I'd like to keep it that way if you do not mind.

[dantrushin]

Move copy propagation to separate function;
Simplify statepoint defs handling a bit;

[skatkov]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

55

Still do not see that not specifying this option actually means infinity.
Because it is a bit strange that
"Max number of statepoints allowed to pass GC Ptrs in registers" and default value 0.
So it makes me thinking that by default the behavior is off.

131

This is because I'm sure that GC pointer cannot be on register until your changes.

627

Please double check, that there is not problems with subregs here.

629

If you do not find reg in statepoint you continue to search. For example you found a Reg in previous statepoint, what does it mean?

To me it means that live gc pointer passed through next statepoint and is not mentioned in that statepoint. it is a bug.
I believe you should assert this or stop on the first statepoint.

Providing def as non-first statepoint seems to be a bug,

[skatkov]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

123

Do I understand correctly that with your changes ALL GC pointers must be defs?
So why do you need these iterations instead of just taking all defs?

491

what is the reason for this magic?

576

Don't you want to separate reload loads into separate function?
So we'll have:
spill registers
rewrite statepoint
insertReloads/unspill registers

[dantrushin]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

123

Strictly speaking, no. Only derived pointers passed in registers.
Are we guaranteed that all base pointers will appear as derived ones too?
If yes, then it is good catch, taking them from defs is simpler (but taking them from operand list instead of def list sounds a bit more natural, IMHO)

131

So nothing has changed.
At this point there is no way to detect deopt pointer which is not in gc list.
ISEL determines what pointers to pass where.
If implementation cannot handle pointer deopt value, not present in gc list, it should not enable it at all or spill all deopt values

491

The reason is that TTI.loadRegFromStackSlot can insert load only before some existing instruction.

[dantrushin]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

576

insertReloads uses local vector RegsToReload and MI (statepoint instruction).
To call insertReloads outside of rewriteStatepoint I will have to make that local vector and new statepoint instruction
available to insertReloads().

I don't think that making RegsToReload member variable or something like that:

SmallVector<Register, 8> RegsToReload;
SS.spillRegisters();
MachineInstr *NewStatepoint = SS.rewriteStatepoint(RegsToReload); // out parameter
SS.insertReloads(RegsToReload, NewStatepoint);

will be much cleaner.
But I can do that if you want.

[dantrushin]

Address review comments and fix clang-format errors

[skatkov]

A pretty close to be done.

Do we need a special tests for this patch?

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

123

I'm a bit confused here. What is the difference between derived and based pointer here?
You have an alive gc pointer. It might be relocated == can be changed. So it must be defined as def independent on whether it is a derived one or based one. Do I miss anything here?

If you are doubt and write that it works under assumption I would suggest under debug assert that sets collected by different way are the same.

491

Does it make sense to add an utility function to TTI which after some existing instruction? It looks more natural then this magic.

576

I do not have strong preference here. But separation seems to me makes sense.
At least rename the function rewriteStatepoint and definitely update the comment before the function due to it does additional things except rewriting statepoints.

[dantrushin]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

123

GC pointers always occurs in pairs (base, derived).
Only derived pointers are relocated (and so can be tied to defs of statepoint).
If base pointer need to be relocated it will appear as (base, base) pair.

It is not specified if the base pointer must be relocated together with its derived pointer.
At least, this is how I interpret LLVM docs. So I originally wrote it in a way I had no doubts of.

What's the point of having two implementation and comparing them with assert?
Assert is not proof that 'doubtful' implementation is correct.

491

I will have to implement that function in all backends and then spent 6 months or so asking people
to review change which they won't ever use...
I would like to avoid that

[skatkov]

I need an answer to the question about test and at least update doc for rewriteStatepoint.

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

123

It will help you to catch easily an error in case I was wrong with defs.

491

ok

[dantrushin]

Return back old implementation of collectGCRegs as I've found case
where statepoint defs != gc args;

Refactor insertReloads() into a separate routine.

Will add some tests soon.

[dantrushin]

Add two simple MIR tests

[skatkov]

could you please explain the case due to you need to return back in collectGCRegs implementation?

[skatkov]

As I understand a lot of things here are done to workout the case that one catch block can correspond to different invoke instructions. Can you please add the test examine this case?

[dantrushin]

Add (hand crafted) test for shared landing pad;
Slightly change cache handling code to better handle shared landing pads;
Improve debug output;

[dantrushin]

Changed DerivedPtrMap and moved it to StatepointLowering.h as it only
needed for local gc.relocate processing;
Rebased on tip, rewrote lowerStatepointMetaArgs as been told;
Deleted test mods, will add separate test later;

[dantrushin]

Restore accidently destroyed review.

Change handling of shared landing pads - now we reserve spill slots
as we go, upon encountering first statepoint jumping to that landing pad.

Slightly modified tests to be less sensitive to unimportant changes
(spill ordering)

[skatkov]

Hi Denis, after you re-write the patch in this way, I guess we can do it clearly and simply in terms of reading.

Let's add set of reserved frame indices into FrameIndexesPerSize.

In reset method, you clean this new set.
In resetAndPreFill you just add all the reserved FI into this set.
In getFrameIndex you lookup FI in GlobalIndices and then take the first available FI which is not in Reserved set.

This way we can get away from this implicit assumptions that we reserved first n elements, supposing that they will processed first and so on...

What do you think?

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

308

It does not look valid to me. You'll set Line.Index to value corresponding to last processed pair from It.
While you need a max in this case...

628

remove redundant {} now.

[dantrushin]

Rework slot reservation according to comments.
Add an option to disable copy propagation.

[skatkov]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

302

assert that frame slot is reserved? (nobody used it)

353

Can we just track for each EhPad what registers were reloaded instead of this search?

412

I wonder, do we really need to collect GC regs. According to implementation, any register found in operand after some index is considered as GC reg.
So can we just compute this index and use it in this loop.

Do I miss anything?

[dantrushin]

Add tracking of reloads inserted in landing pads.

[dantrushin]

That EHPad reload tracking does not fit very well in existing design. I added it as a separate entity stored in StatepointProcessor and passed to
StatepointState::insertReloads.
Ideally, I would insert all reloads in a single sweep after all statepoints has been processed. But I did not find a clean way to do that without refactoring
(like merging StatepointStates functionality into`StatepointProcessor`).

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

412

This is to handle pointers found in both deopt state and gc pointer section uniformly.
To avoid case when same object is passed both in register and in stack slot.

[reames]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

563–564

There's an important semantic bug here. Test case and candidate fix can be found in https://reviews.llvm.org/D84964.

While writing the test case for this, I also stumbled across an unrelated functional bug. Filed as https://bugs.llvm.org/show_bug.cgi?id=46916

[skatkov]

Please fix the bug Philip mentioned and I think we are ready to land it.

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

130

In InstrEmitter I see getStatepointGCArgStartIdx which does exactly the same what you need.
It also contains:

// FIXME: need a better place for this. Put it in StackMaps?

Can you do it in a separate patch (w/o review) and just re-use this utility function here?

393

This overall algorithm is very sensitive to alone EHPad.
Looking into machine verifier, I see that in some cases we can have more than one landing pad.

 if (LandingPadSuccs.size() > 1 &&
  !(AsmInfo &&
    AsmInfo->getExceptionHandlingType() == ExceptionHandling::SjLj &&
    BB && isa<SwitchInst>(BB->getTerminator())) &&
  !isScopedEHPersonality(classifyEHPersonality(F.getPersonalityFn())))
report("MBB has more than one landing pad successor", MBB);

I would suggest to assert that here we have only one landing pad.

[dantrushin]

• Mark MMOs for gc register spill slots as load/store ones;
• Add assert that there is only one EHPad;

[dantrushin]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

130

No, it is slightly different.
During its lifetime statepoint meta args change their format few times. From bare %stack.0 form, to stackmap-encoded
version like 1, 8, %stack.0 here and down to frame register format.
This is the primary reason why I didn't put in into StackMaps.cpp - to remain generic, it'll need some kind of
parameter to tell which exact form it is parsing. Which would look ugly, IMHO

[skatkov]

Thanks!

[RKSimon]

llvm/test/CodeGen/X86/statepoint-fixup-copy-prop-neg.mir

3

@dantrushin This is failing on EXPENSIVE_CHECKS builds - please can you take a look?

[RKSimon]

llvm/test/CodeGen/X86/statepoint-fixup-copy-prop-neg.mir

3

http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-ubuntu/builds/8206

[dantrushin]

llvm/test/CodeGen/X86/statepoint-fixup-copy-prop-neg.mir

3

Thanks. I still have no email from bot, but reproduced it locally.
Will commit fix in a moment

[bjope]

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

224

Results in "unused variable" werror in noasserts build

475

Results in "unused variable" werror in noasserts build

llvm/lib/CodeGen/FixupStatepointCallerSaved.cpp

 STATISTIC(NumSpillSlotsAllocated, "Number of spill slots allocated");
 STATISTIC(NumSpillSlotsExtended, "Number of spill slots extended");
 
 static cl::opt<bool> FixupSCSExtendSlotSize(
     "fixup-scs-extend-slot-size", cl::Hidden, cl::init(false),
     cl::desc("Allow spill in spill slot of greater size than register size"),
     cl::Hidden);
 
+static cl::opt<bool> PassGCPtrInCSR(
+    "fixup-allow-gcptr-in-csr", cl::Hidden, cl::init(false),
+    cl::desc("Allow passing GC Pointer arguments in callee saved registers"));
+
+static cl::opt<bool> EnableCopyProp(
+    "fixup-scs-enable-copy-propagation", cl::Hidden, cl::init(true),
+    cl::desc("Enable simple copy propagation during register reloading"));
+
+// This is purely debugging option.
+// It may be handy for investigating statepoint spilling issues.
+static cl::opt<unsigned> MaxStatepointsWithRegs(
+    "fixup-max-csr-statepoints", cl::Hidden,
+    cl::desc("Max number of statepoints allowed to pass GC Ptrs in registers"));
+
 namespace {
 
 class FixupStatepointCallerSaved : public MachineFunctionPass {
 public:
   static char ID;
 
   FixupStatepointCallerSaved() : MachineFunctionPass(ID) {
     initializeFixupStatepointCallerSavedPass(*PassRegistry::getPassRegistry());
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     MachineFunctionPass::getAnalysisUsage(AU);
   }
 
   StringRef getPassName() const override {
     return "Fixup Statepoint Caller Saved";
   }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
 };
+
 } // End anonymous namespace.
 
 char FixupStatepointCallerSaved::ID = 0;
 char &llvm::FixupStatepointCallerSavedID = FixupStatepointCallerSaved::ID;
 
 INITIALIZE_PASS_BEGIN(FixupStatepointCallerSaved, DEBUG_TYPE,
                       "Fixup Statepoint Caller Saved", false, false)
 INITIALIZE_PASS_END(FixupStatepointCallerSaved, DEBUG_TYPE,
                     "Fixup Statepoint Caller Saved", false, false)
 
 // Utility function to get size of the register.
 static unsigned getRegisterSize(const TargetRegisterInfo &TRI, Register Reg) {
   const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
   return TRI.getSpillSize(*RC);
 }
 
+// Advance iterator to the next stack map entry
+static MachineInstr::const_mop_iterator
+advanceToNextStackMapElt(MachineInstr::const_mop_iterator MOI) {
+  if (MOI->isImm()) {
+    switch (MOI->getImm()) {
+    default:
+      llvm_unreachable("Unrecognized operand type.");
+    case StackMaps::DirectMemRefOp:
+      MOI += 2; // <Reg>, <Imm>
+      break;
+    case StackMaps::IndirectMemRefOp:
+      MOI += 3; // <Size>, <Reg>, <Imm>
+      break;
+    case StackMaps::ConstantOp:
+      MOI += 1;
+      break;
+    }
+  }
+  return ++MOI;
+}
+
+// Return statepoint GC args as a set
+static SmallSet<Register, 8> collectGCRegs(MachineInstr &MI) {
+  StatepointOpers SO(&MI);
+  unsigned NumDeoptIdx = SO.getNumDeoptArgsIdx();
+  unsigned NumDeoptArgs = MI.getOperand(NumDeoptIdx).getImm();
+  MachineInstr::const_mop_iterator MOI(MI.operands_begin() + NumDeoptIdx + 1),
+      MOE(MI.operands_end());
+
+  // Skip deopt args
+  while (NumDeoptArgs--)
+    MOI = advanceToNextStackMapElt(MOI);
+
+  SmallSet<Register, 8> Result;
+  while (MOI != MOE) {
+    if (MOI->isReg() && !MOI->isImplicit())
+      Result.insert(MOI->getReg());
+    MOI = advanceToNextStackMapElt(MOI);
+  }
+  return Result;
+}
+
+// Try to eliminate redundant copy to register which we're going to
+// spill, i.e. try to change:
+//    X = COPY Y
+//    SPILL X
+//  to
+//    SPILL Y
+//  If there are no uses of X between copy and STATEPOINT, that COPY
+//  may be eliminated.
+//  Reg - register we're about to spill
+//  RI - On entry points to statepoint.
+//       On successful copy propagation set to new spill point.
+//  IsKill - set to true if COPY is Kill (there are no uses of Y)
+//  Returns either found source copy register or original one.
+static Register performCopyPropagation(Register Reg,
+                                       MachineBasicBlock::iterator &RI,
+                                       bool &IsKill, const TargetInstrInfo &TII,
+                                       const TargetRegisterInfo &TRI) {
+  if (!EnableCopyProp)
+    return Reg;
+
+  // First check if statepoint itself uses Reg in non-meta operands.
+  int Idx = RI->findRegisterUseOperandIdx(Reg, false, &TRI);
+  if (Idx >= 0 && (unsigned)Idx < StatepointOpers(&*RI).getNumDeoptArgsIdx())
+    return Reg;
+
+  MachineBasicBlock *MBB = RI->getParent();
+  MachineBasicBlock::reverse_iterator E = MBB->rend();
+  MachineInstr *Def = nullptr, *Use = nullptr;
+  for (auto It = ++(RI.getReverse()); It != E; ++It) {
+    if (It->readsRegister(Reg, &TRI) && !Use)
+      Use = &*It;
+    if (It->modifiesRegister(Reg, &TRI)) {
+      Def = &*It;
+      break;
+    }
+  }
+
+  if (!Def)
+    return Reg;
+
+  auto DestSrc = TII.isCopyInstr(*Def);
+  if (!DestSrc || DestSrc->Destination->getReg() != Reg)
+    return Reg;
+
+  Register SrcReg = DestSrc->Source->getReg();
+
+  if (getRegisterSize(TRI, Reg) != getRegisterSize(TRI, SrcReg))
+    return Reg;
+
+  LLVM_DEBUG(dbgs() << "spillRegisters: perform copy propagation "
+                    << printReg(Reg, &TRI) << " -> " << printReg(SrcReg, &TRI)
+                    << "\n");
+
+  // Insert spill immediately after Def
+  RI = ++MachineBasicBlock::iterator(Def);
+  IsKill = DestSrc->Source->isKill();
+
+  // There are no uses of original register between COPY and STATEPOINT.
+  // There can't be any after STATEPOINT, so we can eliminate Def.
+  if (!Use) {
+    LLVM_DEBUG(dbgs() << "spillRegisters: removing dead copy " << *Def);
+    Def->eraseFromParent();
+  }
+  return SrcReg;
+}
+
 namespace {
+// Pair {Register, FrameIndex}
+using RegSlotPair = std::pair<Register, int>;
+
+// Keeps track of what reloads were inserted in MBB.
+class RegReloadCache {
+  using ReloadSet = SmallSet<RegSlotPair, 8>;
+  DenseMap<const MachineBasicBlock *, ReloadSet> Reloads;
+
+public:
+  RegReloadCache() = default;
+
+  // Record reload of Reg from FI in block MBB
+  void recordReload(Register Reg, int FI, const MachineBasicBlock *MBB) {
+    RegSlotPair RSP(Reg, FI);
+    auto Res = Reloads[MBB].insert(RSP);
+    assert(Res.second && "reload already exists");
+  }
+
+  // Does basic block MBB contains reload of Reg from FI?
+  bool hasReload(Register Reg, int FI, const MachineBasicBlock *MBB) {
+    RegSlotPair RSP(Reg, FI);
+    return Reloads.count(MBB) && Reloads[MBB].count(RSP);
+  }
+};
+
 // Cache used frame indexes during statepoint re-write to re-use them in
 // processing next statepoint instruction.
 // Two strategies. One is to preserve the size of spill slot while another one
 // extends the size of spill slots to reduce the number of them, causing
 // the less total frame size. But unspill will have "implicit" any extend.
 class FrameIndexesCache {
 private:
   struct FrameIndexesPerSize {
     // List of used frame indexes during processing previous statepoints.
     SmallVector<int, 8> Slots;
     // Current index of un-used yet frame index.
     unsigned Index = 0;
   };
   MachineFrameInfo &MFI;
   const TargetRegisterInfo &TRI;
   // Map size to list of frame indexes of this size. If the mode is
   // FixupSCSExtendSlotSize then the key 0 is used to keep all frame indexes.
   // If the size of required spill slot is greater than in a cache then the
   // size will be increased.
   DenseMap<unsigned, FrameIndexesPerSize> Cache;
 
+  // Keeps track of slots reserved for the shared landing pad processing.
+  // Initialized from GlobalIndices for the current EHPad.
+  SmallSet<int, 8> ReservedSlots;
+
+  // Landing pad can be destination of several statepoints. Every register
+  // defined by such statepoints must be spilled to the same stack slot.
+  // This map keeps that information.
+  DenseMap<const MachineBasicBlock *, SmallVector<RegSlotPair, 8>>
+      GlobalIndices;
+
+  FrameIndexesPerSize &getCacheBucket(unsigned Size) {
+    // In FixupSCSExtendSlotSize mode the bucket with 0 index is used
+    // for all sizes.
+    return Cache[FixupSCSExtendSlotSize ? 0 : Size];
+  }
+
 public:
   FrameIndexesCache(MachineFrameInfo &MFI, const TargetRegisterInfo &TRI)
       : MFI(MFI), TRI(TRI) {}
   // Reset the current state of used frame indexes. After invocation of
-  // this function all frame indexes are available for allocation.
-  void reset() {
+  // this function all frame indexes are available for allocation with
+  // the exception of slots reserved for landing pad processing (if any).
+  void reset(const MachineBasicBlock *EHPad) {
     for (auto &It : Cache)
       It.second.Index = 0;
+
+    ReservedSlots.clear();
+    if (EHPad && GlobalIndices.count(EHPad))
+      for (auto &RSP : GlobalIndices[EHPad])
+        ReservedSlots.insert(RSP.second);
   }
+
   // Get frame index to spill the register.
-  int getFrameIndex(Register Reg) {
+  int getFrameIndex(Register Reg, MachineBasicBlock *EHPad) {
+    // Check if slot for Reg is already reserved at EHPad.
+    auto It = GlobalIndices.find(EHPad);
+    if (It != GlobalIndices.end()) {
+      auto &Vec = It->second;
+      auto Idx = llvm::find_if(
+          Vec, [Reg](RegSlotPair &RSP) { return Reg == RSP.first; });
+      if (Idx != Vec.end()) {
+        int FI = Idx->second;
+        LLVM_DEBUG(dbgs() << "Found global FI " << FI << " for register "
+                          << printReg(Reg, &TRI) << " at "
+                          << printMBBReference(*EHPad) << "\n");
+        assert(ReservedSlots.count(FI) && "using unreserved slot");
+        return FI;
+      }
+    }
+
     unsigned Size = getRegisterSize(TRI, Reg);
-    // In FixupSCSExtendSlotSize mode the bucket with 0 index is used
-    // for all sizes.
-    unsigned Bucket = FixupSCSExtendSlotSize ? 0 : Size;
-    FrameIndexesPerSize &Line = Cache[Bucket];
-    if (Line.Index < Line.Slots.size()) {
+    FrameIndexesPerSize &Line = getCacheBucket(Size);
+    while (Line.Index < Line.Slots.size()) {
       int FI = Line.Slots[Line.Index++];
+      if (ReservedSlots.count(FI))
+        continue;
       // If all sizes are kept together we probably need to extend the
       // spill slot size.
       if (MFI.getObjectSize(FI) < Size) {
         MFI.setObjectSize(FI, Size);
         MFI.setObjectAlignment(FI, Align(Size));
         NumSpillSlotsExtended++;
       }
       return FI;
     }
     int FI = MFI.CreateSpillStackObject(Size, Align(Size));
     NumSpillSlotsAllocated++;
     Line.Slots.push_back(FI);
     ++Line.Index;
+
+    // Remember assignment {Reg, FI} for EHPad
+    if (EHPad) {
+      GlobalIndices[EHPad].push_back(std::make_pair(Reg, FI));
+      LLVM_DEBUG(dbgs() << "Reserved FI " << FI << " for spilling reg "
+                        << printReg(Reg, &TRI) << " at landing pad "
+                        << printMBBReference(*EHPad) << "\n");
+    }
+
     return FI;
   }
+
   // Sort all registers to spill in descendent order. In the
   // FixupSCSExtendSlotSize mode it will minimize the total frame size.
   // In non FixupSCSExtendSlotSize mode we can skip this step.
   void sortRegisters(SmallVectorImpl<Register> &Regs) {
     if (!FixupSCSExtendSlotSize)
       return;
     llvm::sort(Regs.begin(), Regs.end(), [&](Register &A, Register &B) {
       return getRegisterSize(TRI, A) > getRegisterSize(TRI, B);
     });
   }
 };
 
 // Describes the state of the current processing statepoint instruction.
 class StatepointState {
 private:
   // statepoint instruction.
   MachineInstr &MI;
   MachineFunction &MF;
+  // If non-null then statepoint is invoke, and this points to the landing pad.
+  MachineBasicBlock *EHPad;
   const TargetRegisterInfo &TRI;
   const TargetInstrInfo &TII;
   MachineFrameInfo &MFI;
   // Mask with callee saved registers.
   const uint32_t *Mask;
   // Cache of frame indexes used on previous instruction processing.
   FrameIndexesCache &CacheFI;
+  bool AllowGCPtrInCSR;
   // Operands with physical registers requiring spilling.
   SmallVector<unsigned, 8> OpsToSpill;
   // Set of register to spill.
   SmallVector<Register, 8> RegsToSpill;
+  // Set of registers to reload after statepoint.
+  SmallVector<Register, 8> RegsToReload;
   // Map Register to Frame Slot index.
   DenseMap<Register, int> RegToSlotIdx;
 
 public:
   StatepointState(MachineInstr &MI, const uint32_t *Mask,
-                  FrameIndexesCache &CacheFI)
+                  FrameIndexesCache &CacheFI, bool AllowGCPtrInCSR)
       : MI(MI), MF(*MI.getMF()), TRI(*MF.getSubtarget().getRegisterInfo()),
         TII(*MF.getSubtarget().getInstrInfo()), MFI(MF.getFrameInfo()),
-        Mask(Mask), CacheFI(CacheFI) {}
+        Mask(Mask), CacheFI(CacheFI), AllowGCPtrInCSR(AllowGCPtrInCSR) {
+
+    // Find statepoint's landing pad, if any.
+    EHPad = nullptr;
+    MachineBasicBlock *MBB = MI.getParent();
+    // Invoke statepoint must be last one in block.
+    bool Last = std::none_of(++MI.getIterator(), MBB->end().getInstrIterator(),
+                             [](MachineInstr &I) {
+                               return I.getOpcode() == TargetOpcode::STATEPOINT;
+                             });
+
+    if (!Last)
+      return;
+
+    auto IsEHPad = [](MachineBasicBlock *B) { return B->isEHPad(); };
+
+    assert(llvm::count_if(MBB->successors(), IsEHPad) < 2 && "multiple EHPads");
+
+    auto It = llvm::find_if(MBB->successors(), IsEHPad);
+    if (It != MBB->succ_end())
+      EHPad = *It;
+  }
+
+  MachineBasicBlock *getEHPad() const { return EHPad; }
+
   // Return true if register is callee saved.
   bool isCalleeSaved(Register Reg) { return (Mask[Reg / 32] >> Reg % 32) & 1; }
+
   // Iterates over statepoint meta args to find caller saver registers.
   // Also cache the size of found registers.
   // Returns true if caller save registers found.
   bool findRegistersToSpill() {
     SmallSet<Register, 8> VisitedRegs;
+    SmallSet<Register, 8> GCRegs = collectGCRegs(MI);
     for (unsigned Idx = StatepointOpers(&MI).getVarIdx(),
                   EndIdx = MI.getNumOperands();
          Idx < EndIdx; ++Idx) {
       MachineOperand &MO = MI.getOperand(Idx);
       if (!MO.isReg() || MO.isImplicit())
         continue;
       Register Reg = MO.getReg();
       assert(Reg.isPhysical() && "Only physical regs are expected");
-      if (isCalleeSaved(Reg))
+
+      if (isCalleeSaved(Reg) && (AllowGCPtrInCSR || !is_contained(GCRegs, Reg)))
         continue;
+
+      LLVM_DEBUG(dbgs() << "Will spill " << printReg(Reg, &TRI) << " at index "
+                        << Idx << "\n");
+
       if (VisitedRegs.insert(Reg).second)
         RegsToSpill.push_back(Reg);
       OpsToSpill.push_back(Idx);
     }
     CacheFI.sortRegisters(RegsToSpill);
     return !RegsToSpill.empty();
   }
+
   // Spill all caller saved registers right before statepoint instruction.
   // Remember frame index where register is spilled.
   void spillRegisters() {
     for (Register Reg : RegsToSpill) {
-      int FI = CacheFI.getFrameIndex(Reg);
+      int FI = CacheFI.getFrameIndex(Reg, EHPad);
       const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
-      TII.storeRegToStackSlot(*MI.getParent(), MI, Reg, true /*is_Kill*/, FI,
-                              RC, &TRI);
+
       NumSpilledRegisters++;
       RegToSlotIdx[Reg] = FI;
+
+      LLVM_DEBUG(dbgs() << "Spilling " << printReg(Reg, &TRI) << " to FI " << FI
+                        << "\n");
+
+      // Perform trivial copy propagation
+      bool IsKill = true;
+      MachineBasicBlock::iterator InsertBefore(MI);
+      Reg = performCopyPropagation(Reg, InsertBefore, IsKill, TII, TRI);
+
+      LLVM_DEBUG(dbgs() << "Insert spill before " << *InsertBefore);
+      TII.storeRegToStackSlot(*MI.getParent(), InsertBefore, Reg, IsKill, FI,
+                              RC, &TRI);
     }
   }
+
+  void insertReloadBefore(unsigned Reg, MachineBasicBlock::iterator It,
+                          MachineBasicBlock *MBB) {
+    const TargetRegisterClass *RC = TRI.getMinimalPhysRegClass(Reg);
+    int FI = RegToSlotIdx[Reg];
+    if (It != MBB->end()) {
+      TII.loadRegFromStackSlot(*MBB, It, Reg, FI, RC, &TRI);
+      return;
+    }
+
+    // To insert reload at the end of MBB, insert it before last instruction
+    // and then swap them.
+    assert(MBB->begin() != MBB->end() && "Empty block");
+    --It;
+    TII.loadRegFromStackSlot(*MBB, It, Reg, FI, RC, &TRI);
+    MachineInstr *Reload = It->getPrevNode();
+    int Dummy = 0;
+    assert(TII.isLoadFromStackSlot(*Reload, Dummy) == Reg);
+    assert(Dummy == FI);
+    MBB->remove(Reload);
+    MBB->insertAfter(It, Reload);
+  }
+
+  // Insert reloads of (relocated) registers spilled in statepoint.
+  void insertReloads(MachineInstr *NewStatepoint, RegReloadCache &RC) {
+    MachineBasicBlock *MBB = NewStatepoint->getParent();
+    auto InsertPoint = std::next(NewStatepoint->getIterator());
+
+    for (auto Reg : RegsToReload) {
+      insertReloadBefore(Reg, InsertPoint, MBB);
+      LLVM_DEBUG(dbgs() << "Reloading " << printReg(Reg, &TRI) << " from FI "
+                        << RegToSlotIdx[Reg] << " after statepoint\n");
+
+      if (EHPad && !RC.hasReload(Reg, RegToSlotIdx[Reg], EHPad)) {
+        RC.recordReload(Reg, RegToSlotIdx[Reg], EHPad);
+        auto EHPadInsertPoint = EHPad->SkipPHIsLabelsAndDebug(EHPad->begin());
+        insertReloadBefore(Reg, EHPadInsertPoint, EHPad);
+        LLVM_DEBUG(dbgs() << "...also reload at EHPad "
+                          << printMBBReference(*EHPad) << "\n");
+      }
+    }
+  }
+
   // Re-write statepoint machine instruction to replace caller saved operands
   // with indirect memory location (frame index).
-  void rewriteStatepoint() {
+  MachineInstr *rewriteStatepoint() {
     MachineInstr *NewMI =
         MF.CreateMachineInstr(TII.get(MI.getOpcode()), MI.getDebugLoc(), true);
     MachineInstrBuilder MIB(MF, NewMI);
 
+    unsigned NumOps = MI.getNumOperands();
+
+    // New indices for the remaining defs.
+    SmallVector<unsigned, 8> NewIndices;
+    unsigned NumDefs = MI.getNumDefs();
+    for (unsigned I = 0; I < NumDefs; ++I) {
+      MachineOperand &DefMO = MI.getOperand(I);
+      assert(DefMO.isReg() && DefMO.isDef() && "Expected Reg Def operand");
+      Register Reg = DefMO.getReg();
+      if (!AllowGCPtrInCSR) {
+        assert(is_contained(RegsToSpill, Reg));
+        RegsToReload.push_back(Reg);
+      } else {
+        if (isCalleeSaved(Reg)) {
+          NewIndices.push_back(NewMI->getNumOperands());
+          MIB.addReg(Reg, RegState::Define);
+        } else {
+          NewIndices.push_back(NumOps);
+          RegsToReload.push_back(Reg);
+        }
+      }
+    }
+
     // Add End marker.
     OpsToSpill.push_back(MI.getNumOperands());
     unsigned CurOpIdx = 0;
 
-    for (unsigned I = 0; I < MI.getNumOperands(); ++I) {
+    for (unsigned I = NumDefs; I < MI.getNumOperands(); ++I) {
       MachineOperand &MO = MI.getOperand(I);
       if (I == OpsToSpill[CurOpIdx]) {
         int FI = RegToSlotIdx[MO.getReg()];
         MIB.addImm(StackMaps::IndirectMemRefOp);
         MIB.addImm(getRegisterSize(TRI, MO.getReg()));
         assert(MO.isReg() && "Should be register");
         assert(MO.getReg().isPhysical() && "Should be physical register");
         MIB.addFrameIndex(FI);
         MIB.addImm(0);
         ++CurOpIdx;
-      } else
+      } else {
         MIB.add(MO);
+        unsigned OldDef;
+        if (AllowGCPtrInCSR && MI.isRegTiedToDefOperand(I, &OldDef)) {
+          assert(OldDef < NumDefs);
+          assert(NewIndices[OldDef] < NumOps);
+          MIB->tieOperands(NewIndices[OldDef], MIB->getNumOperands() - 1);
+        }
+      }
     }
     assert(CurOpIdx == (OpsToSpill.size() - 1) && "Not all operands processed");
     // Add mem operands.
     NewMI->setMemRefs(MF, MI.memoperands());
     for (auto It : RegToSlotIdx) {
+      Register R = It.first;
       int FrameIndex = It.second;
       auto PtrInfo = MachinePointerInfo::getFixedStack(MF, FrameIndex);
-      auto *MMO = MF.getMachineMemOperand(PtrInfo, MachineMemOperand::MOLoad,
-                                          getRegisterSize(TRI, It.first),
-                                          MFI.getObjectAlign(FrameIndex));
+      MachineMemOperand::Flags Flags = MachineMemOperand::MOLoad;
+      if (is_contained(RegsToReload, R))
+        Flags |= MachineMemOperand::MOStore;
+      auto *MMO =
+          MF.getMachineMemOperand(PtrInfo, Flags, getRegisterSize(TRI, R),
       NewMI->addMemOperand(MF, MMO);
     }
+
     // Insert new statepoint and erase old one.
     MI.getParent()->insert(MI, NewMI);
+
+    LLVM_DEBUG(dbgs() << "rewritten statepoint to : " << *NewMI << "\n");
     MI.eraseFromParent();
+    return NewMI;
   }
 };
 
 class StatepointProcessor {
 private:
   MachineFunction &MF;
   const TargetRegisterInfo &TRI;
   FrameIndexesCache CacheFI;
+  RegReloadCache ReloadCache;
 
 public:
   StatepointProcessor(MachineFunction &MF)
       : MF(MF), TRI(*MF.getSubtarget().getRegisterInfo()),
         CacheFI(MF.getFrameInfo(), TRI) {}
 
-  bool process(MachineInstr &MI) {
+  bool process(MachineInstr &MI, bool AllowGCPtrInCSR) {
     StatepointOpers SO(&MI);
     uint64_t Flags = SO.getFlags();
     // Do nothing for LiveIn, it supports all registers.
     if (Flags & (uint64_t)StatepointFlags::DeoptLiveIn)
       return false;
+    LLVM_DEBUG(dbgs() << "\nMBB " << MI.getParent()->getNumber() << " "
+                      << MI.getParent()->getName() << " : process statepoint "
+                      << MI);
     CallingConv::ID CC = SO.getCallingConv();
     const uint32_t *Mask = TRI.getCallPreservedMask(MF, CC);
-    CacheFI.reset();
-    StatepointState SS(MI, Mask, CacheFI);
+    StatepointState SS(MI, Mask, CacheFI, AllowGCPtrInCSR);
+    CacheFI.reset(SS.getEHPad());
 
     if (!SS.findRegistersToSpill())
       return false;
 
     SS.spillRegisters();
-    SS.rewriteStatepoint();
+    auto *NewStatepoint = SS.rewriteStatepoint();
+    SS.insertReloads(NewStatepoint, ReloadCache);
     return true;
   }
 };
 } // namespace
 
 bool FixupStatepointCallerSaved::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   const Function &F = MF.getFunction();
   if (!F.hasGC())
     return false;
 
   SmallVector<MachineInstr *, 16> Statepoints;
   for (MachineBasicBlock &BB : MF)
     for (MachineInstr &I : BB)
       if (I.getOpcode() == TargetOpcode::STATEPOINT)
         Statepoints.push_back(&I);
 
   if (Statepoints.empty())
     return false;
 
   bool Changed = false;
   StatepointProcessor SPP(MF);
-  for (MachineInstr *I : Statepoints)
-    Changed |= SPP.process(*I);
+  unsigned NumStatepoints = 0;
+  bool AllowGCPtrInCSR = PassGCPtrInCSR;
+  for (MachineInstr *I : Statepoints) {
+    ++NumStatepoints;
+    if (MaxStatepointsWithRegs.getNumOccurrences() &&
+        NumStatepoints >= MaxStatepointsWithRegs)
+      AllowGCPtrInCSR = false;
+    Changed |= SPP.process(*I, AllowGCPtrInCSR);
+  }
   return Changed;
 }

llvm/test/CodeGen/X86/statepoint-fixup-call.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+--- |
+  ; ModuleID = 'test/CodeGen/X86/statepoint-fixup-call.ll'
+  source_filename = "test/CodeGen/X86/statepoint-fixup-call.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @foo()
+
+  define i8 addrspace(1)* @test_one(i8 addrspace(1)* %p) gc "statepoint-example" {
+  entry:
+    %token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @foo, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p) ]
+    %p2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p2
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32 immarg, i32 immarg) #0
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_one
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$rdi', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    liveins: $rdi
+
+    ; CHECK-LABEL: name: test_one
+    ; CHECK: liveins: $rdi
+    ; CHECK: ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed $rdi :: (store 8 into %stack.0)
+    ; CHECK: STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load store 8 on %stack.0)
+    ; CHECK: $rdi = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+    ; CHECK: ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    ; CHECK: $rax = COPY killed renamable $rdi
+    ; CHECK: RET 0, killed $rax
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rdi = STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, killed renamable $rdi, renamable $rdi(tied-def 0), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    $rax = COPY killed renamable $rdi
+    RET 0, killed $rax
+
+...

llvm/test/CodeGen/X86/statepoint-fixup-copy-prop-neg.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -run-pass fixup-statepoint-caller-saved | FileCheck %s
+
+# Check that COPY is not eliminated if dest register is used in
+# non-meta arguments of statepoint.
+
+--- |
+  ; ModuleID = 'cp.ll'
+  source_filename = "cp.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @foo(i64)
+
+  define i8 addrspace(1)* @test(i64 %a, i64 %b, i64 %c, i8 addrspace(1)* %p) gc "statepoint-example" {
+  entry:
+    %token = call token (i64, i32, void (i64)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidi64f(i64 0, i32 0, void (i64)* @foo, i32 1, i32 0, i64 %b, i3 0, i32 0) [ "deopt"(i64 %b), "gc-live"(i8 addrspace(1)* %p) ]
+    %p2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p2
+  }
+
+  ; Function Attrs: nounwind readonly
+  declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32 immarg, i32 immarg) #0
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidi64f(i64 immarg, i32 immarg, void (i64)*, i32 immarg, i32 immarg, ...)
+
+  attributes #0 = { nounwind readonly }
+
+...
+---
+name:            test
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$rsi', virtual-reg: '' }
+  - { reg: '$rcx', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    8
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: default, offset: 0, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    liveins: $rcx, $rsi
+
+    ; CHECK-LABEL: name: test
+    ; CHECK: liveins: $rcx, $rsi
+    ; CHECK: renamable $rdi = COPY $rsi
+    ; CHECK: MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed renamable $rcx :: (store 8 into %stack.0)
+    ; CHECK: MOV64mr %stack.1, 1, $noreg, 0, $noreg, killed $rdi :: (store 8 into %stack.1)
+    ; CHECK: STATEPOINT 0, 0, 1, @foo, $rdi, 2, 0, 2, 0, 2, 1, 1, 8, %stack.1, 0, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0), (load 8 from %stack.1)
+    ; CHECK: renamable $rax = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+    ; CHECK: RET 0, $rax
+    renamable $rdi = COPY $rsi
+    MOV64mr %stack.0, 1, $noreg, 0, $noreg, killed renamable $rcx :: (store 8 into %stack.0)
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    STATEPOINT 0, 0, 1, @foo, $rdi, 2, 0, 2, 0, 2, 1, killed renamable $rdi, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (volatile load store 8 on %stack.0)
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rax = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+    RET 0, $rax
+
+...

llvm/test/CodeGen/X86/statepoint-fixup-invoke.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+
+--- |
+  ; ModuleID = 'test/CodeGen/X86/statepoint-fixup-invoke.mir'
+  source_filename = "test/CodeGen/X86/statepoint-fixup-invoke.mir"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @some_call(i64 addrspace(1)*)
+
+  declare i32 @personality_function()
+
+  define i64 addrspace(1)* @test_basic(i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1) gc "statepoint-example" personality i32 ()* @personality_function {
+  entry:
+    %0 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %obj, i32 0, i32 0) [ "gc-live"(i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1), "deopt"(i32 0, i32 -1, i32 0, i32 0, i32 0) ]
+            to label %invoke_safepoint_normal_dest unwind label %exceptional_return
+
+  invoke_safepoint_normal_dest:                     ; preds = %entry
+    %obj.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 0, i32 0) ; (%obj, %obj)
+    %obj1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %0, i32 1, i32 1) ; (%obj1, %obj1)
+    br label %normal_return
+
+  normal_return:                                    ; preds = %invoke_safepoint_normal_dest
+    ret i64 addrspace(1)* %obj.relocated
+
+  exceptional_return:                               ; preds = %entry
+    %landing_pad = landingpad token
+            cleanup
+    %obj.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 0, i32 0) ; (%obj, %obj)
+    %obj1.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 1, i32 1) ; (%obj1, %obj1)
+    ret i64 addrspace(1)* %obj1.relocated1
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 immarg, i32 immarg, void (i64 addrspace(1)*)*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token, i32 immarg, i32 immarg) #0
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_basic
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$rdi', virtual-reg: '' }
+  - { reg: '$rsi', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: test_basic
+  ; CHECK: bb.0.entry:
+  ; CHECK:   successors: %bb.1(0x7ffff800), %bb.3(0x00000800)
+  ; CHECK:   liveins: $rdi, $rsi
+  ; CHECK-DAG:   MOV64mr %stack.0, 1, $noreg, 0, $noreg, $rsi :: (store 8 into %stack.0)
+  ; CHECK-DAG:   MOV64mr %stack.1, 1, $noreg, 0, $noreg, $rdi :: (store 8 into %stack.1)
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp0>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, 1, 8, %stack.0, 0, 1, 8, %stack.0, 0, 1, 8, %stack.1, 0, 1, 8, %stack.1, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load store 8 on %stack.1), (load store 8 on %stack.0)
+  ; CHECK-DAG:   $r14 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+  ; CHECK-DAG:   $rbx = MOV64rm %stack.1, 1, $noreg, 0, $noreg :: (load 8 from %stack.1)
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp1>
+  ; CHECK:   JMP_1 %bb.1
+  ; CHECK: bb.1.invoke_safepoint_normal_dest:
+  ; CHECK:   successors: %bb.2(0x80000000)
+  ; CHECK:   liveins: $rbx
+  ; CHECK: bb.2.normal_return:
+  ; CHECK:   liveins: $rbx
+  ; CHECK:   $rax = COPY killed renamable $rbx
+  ; CHECK:   RET 0, $rax
+  ; CHECK: bb.3.exceptional_return (landing-pad):
+  ; CHECK:   liveins: $rax, $rdx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol .Ltmp2>
+  ; CHECK-DAG:   $r14 = MOV64rm %stack.0, 1, $noreg, 0, $noreg :: (load 8 from %stack.0)
+  ; CHECK-DAG:   $rbx = MOV64rm %stack.1, 1, $noreg, 0, $noreg :: (load 8 from %stack.1)
+  ; CHECK:   $rax = COPY killed renamable $r14
+  ; CHECK:   RET 0, $rax
+  bb.0.entry:
+    successors: %bb.1(0x7ffff800), %bb.3(0x00000800)
+    liveins: $rdi, $rsi
+
+    renamable $r14 = COPY $rsi
+    renamable $rbx = COPY $rdi
+    EH_LABEL <mcsymbol .Ltmp0>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $r14, renamable $rbx = STATEPOINT 0, 0, 1, @some_call, $rdi, 2, 0, 2, 0, 2, 5, 2, 0, 2, -1, 2, 0, 2, 0, 2, 0, killed renamable $r14, renamable $r14(tied-def 0), killed renamable $rbx, renamable $rbx(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol .Ltmp1>
+    JMP_1 %bb.1
+
+  bb.1.invoke_safepoint_normal_dest:
+    successors: %bb.2(0x80000000)
+    liveins: $rbx
+
+
+  bb.2.normal_return:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+  bb.3.exceptional_return (landing-pad):
+    liveins: $rax, $rdx, $r14
+
+    EH_LABEL <mcsymbol .Ltmp2>
+    $rax = COPY killed renamable $r14
+    RET 0, $rax
+
+...

llvm/test/CodeGen/X86/statepoint-fixup-shared-ehpad.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=false -run-pass fixup-statepoint-caller-saved | FileCheck %s
+
+# NOTE: MIR in this test was hand edited to have two statepoints share single landing pad.
+# This is forbidden in IR, but is allowed in MIR. I just was unable to reproduce conditions
+# under which landing pad merge happens. So I did it manually.
+#
+--- |
+  ; ModuleID = 'statepoint-fixup-shared-ehpad.ll'
+  source_filename = "statepoint-fixup-shared-ehpad.ll"
+  target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-pc-linux-gnu"
+
+  declare void @foo()
+
+  declare void @bar()
+
+  declare i32 @personality_function()
+
+  define i8 addrspace(1)* @test_one(i32 %a, i8 addrspace(1)* %p, i8 addrspace(1)* %q) gc "statepoint-example" personality i32 ()* @personality_function {
+  entry:
+    %cmp = icmp eq i32 %a, 0
+    br i1 %cmp, label %zero, label %nonzero
+
+  zero:                                             ; preds = %entry
+    %token = invoke token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @foo, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p, i8 addrspace(1)* %q) ]
+            to label %normal_dest_a unwind label %exceptional_return_a
+
+  nonzero:                                          ; preds = %entry
+    %token2 = invoke token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @bar, i32 0, i32 0, i32 0, i32 0) [ "gc-live"(i8 addrspace(1)* %p, i8 addrspace(1)* %q) ]
+            to label %normal_dest_b unwind label %exceptional_return_b
+
+  normal_dest_a:                                    ; preds = %zero
+    %p2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p2
+
+  exceptional_return_a:                             ; preds = %zero
+    %landing_pad = landingpad token
+            cleanup
+    %q2 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %landing_pad, i32 1, i32 1) ; (%q, %q)
+    ret i8 addrspace(1)* %q2
+
+  normal_dest_b:                                    ; preds = %nonzero
+    %p3 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %token2, i32 0, i32 0) ; (%p, %p)
+    ret i8 addrspace(1)* %p3
+
+  exceptional_return_b:                             ; preds = %nonzero
+    %landing_pad2 = landingpad token
+            cleanup
+    %q3 = call i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token %landing_pad2, i32 1, i32 1) ; (%q, %q)
+    ret i8 addrspace(1)* %q3
+  }
+
+  declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 immarg, i32 immarg, void ()*, i32 immarg, i32 immarg, ...)
+
+  ; Function Attrs: nounwind readonly
+  declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32 immarg, i32 immarg) #0
+
+  attributes #0 = { nounwind readonly }
+  attributes #1 = { nounwind }
+
+...
+---
+name:            test_one
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$edi', virtual-reg: '' }
+  - { reg: '$rsi', virtual-reg: '' }
+  - { reg: '$rdx', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: test_one
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.3(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   liveins: $rbx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp0>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   MOV64mr [[STACK0:%stack.[0-9]+]], 1, $noreg, 0, $noreg, killed $rbx :: (store 8 into [[STACK0]])
+  ; CHECK:   MOV64mr [[STACK1:%stack.[0-9]+]], 1, $noreg, 0, $noreg, killed $r14 :: (store 8 into [[STACK1]])
+  ; CHECK:   STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK1]], 0, 1, 8, [[STACK1]], 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load store 8 on [[STACK0]]), (load store 8 on [[STACK1]])
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp1>
+  ; CHECK:   JMP_1 %bb.3
+  ; CHECK: bb.2:
+  ; CHECK:   successors: %bb.5(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   liveins: $rbx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp2>
+  ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK-DAG:   MOV64mr [[STACK0]], 1, $noreg, 0, $noreg, killed $rbx :: (store 8 into [[STACK0]])
+  ; CHECK-DAG:   MOV64mr [[STACK1]], 1, $noreg, 0, $noreg, killed $r14 :: (store 8 into [[STACK1]])
+  ; CHECK:   STATEPOINT 0, 0, 0, @bar, 2, 0, 2, 0, 2, 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK0]], 0, 1, 8, [[STACK1]], 0, 1, 8, [[STACK1]], 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load store 8 on [[STACK0]]), (load store 8 on [[STACK1]])
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp3>
+  ; CHECK:   JMP_1 %bb.5
+  ; CHECK: bb.4 (landing-pad):
+  ; CHECK:   liveins: $rax, $rdx, $r14
+  ; CHECK:   EH_LABEL <mcsymbol Ltmp4>
+  ; CHECK-DAG:   $rbx = MOV64rm [[STACK0]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK0]])
+  ; CHECK-DAG:   $r14 = MOV64rm [[STACK1]], 1, $noreg, 0, $noreg :: (load 8 from [[STACK1]])
+  ; CHECK:   $rax = COPY killed renamable $r14
+  ; CHECK:   RET 0, $rax
+  bb.0:
+    successors: %bb.1, %bb.2
+    liveins: $edi, $rdx, $rsi
+
+    renamable $r14 = COPY $rdx
+    renamable $rbx = COPY $rsi
+    TEST32rr killed renamable $edi, renamable $edi, implicit-def $eflags
+    JCC_1 %bb.2, 5, implicit killed $eflags
+    JMP_1 %bb.1
+
+  bb.1:
+    successors: %bb.3, %bb.4
+    liveins: $rbx, $r14
+
+    EH_LABEL <mcsymbol Ltmp0>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rbx, renamable $r14 = STATEPOINT 0, 0, 0, @foo, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), killed renamable $r14, renamable $r14(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol Ltmp1>
+    JMP_1 %bb.3
+
+  bb.2:
+    successors: %bb.5, %bb.4
+    liveins: $rbx, $r14
+
+    EH_LABEL <mcsymbol Ltmp2>
+    ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    renamable $rbx, renamable $r14 = STATEPOINT 0, 0, 0, @bar, 2, 0, 2, 0, 2, 0, killed renamable $rbx, renamable $rbx(tied-def 0), killed renamable $r14, renamable $r14(tied-def 1), csr_64, implicit-def $rsp, implicit-def $ssp
+    ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+    EH_LABEL <mcsymbol Ltmp3>
+    JMP_1 %bb.5
+
+  bb.3:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+  bb.4 (landing-pad):
+    liveins: $rax, $rdx, $r14
+
+    EH_LABEL <mcsymbol Ltmp4>
+    $rax = COPY killed renamable $r14
+    RET 0, $rax
+
+  bb.5:
+    liveins: $rbx
+
+    $rax = COPY killed renamable $rbx
+    RET 0, $rax
+
+...

llvm/test/CodeGen/X86/statepoint-vreg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -max-registers-for-gc-values=4 < %s | FileCheck %s
+; RUN: llc -max-registers-for-gc-values=4 -fixup-allow-gcptr-in-csr=true < %s | FileCheck %s
 
 target datalayout = "e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-linux-gnu"
 
 declare i1 @return_i1()
 declare void @func()
 declare void @"some_call"(i64 addrspace(1)*)
 declare void @consume(i32 addrspace(1)*)

llvm/test/CodeGen/X86/statepoint-vreg.mir

 # NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-# RUN: llc -o - %s -start-after=finalize-isel | FileCheck %s
+# RUN: llc -o - %s -fixup-allow-gcptr-in-csr=true -start-after=finalize-isel | FileCheck %s
 
 --- |
   ; ModuleID = 'test/CodeGen/X86/statepoint-vreg.ll'
   source_filename = "test/CodeGen/X86/statepoint-vreg.ll"
   target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
   target triple = "x86_64-pc-linux-gnu"
 
   declare void @bar()