Support of the gc.relocates for invoke statepoints

[igor-laevsky]

Summary

This change builds up on http://reviews.llvm.org/D7760

It implements support for the gc.relocates tied to the invoke statepoint. It adds new 'FuncInfo::InvokeStatepointValueSlots' field to store exported gc values and uses it during visitGCRelocate.

[reames]

Is there a missing part to this patch? The current code doesn't seem complete/correct.

include/llvm/CodeGen/FunctionLoweringInfo.h

82 ↗

(On Diff #21229)

I don't think we're actually guaranteed that the same stack slot is used across all statepoints. Are we? If not, this needs to be keyed by a pair of derived pointer and statepoint instruction.

Consider:
V = ...
if (X) statepoint; else statepoint;

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

533 ↗

(On Diff #21229)

It's not all the spilled values here, just the relocated ones. Can you fix the comment?

535 ↗

(On Diff #21229)

Given we already have the derived pointers above, can't this be rewritten as a loop over Ptrs?

Actually, I would have expected to see this as part of the previous loop. If lowerIncomingStatepointValue returned the lowered value, this code could be expressed neatly within the previously loop.

542 ↗

(On Diff #21229)

See my comment above about coliding stack slots. If this isn't possible, please add an assertion.

681 ↗

(On Diff #21229)

It seems like it would be easier to just not run the debug loop above for invokes. Am I missing something?

722 ↗

(On Diff #21229)

CamelCase - er wait, you're just moving this. Remind me to rename this as soon as your patch lands. You don't need to and shouldn't change the name here.

746 ↗

(On Diff #21229)

Er, I don't think there should be such a case. Do you have an example or should this be an assert?

755 ↗

(On Diff #21229)

This really doesn't look like right. The allocation logic used inside spilling is assuming it can reuse slots. It looks like you've essentially created an alternate mechanism for recording locations in invokes, but not updated the stack slot reuse/allocation code paths. Am I missing something here?

[igor-laevsky]

Add more test cases.
Remove 'clearValidationInfo'
Updated comments.

[igor-laevsky]

include/llvm/CodeGen/FunctionLoweringInfo.h

82 ↗

(On Diff #21229)

You are right, we are not guaranteed that same stack slot is allocated for the same value in different statepoints. But we are guaranteed that when we start processing new safepoint with same value we already finished processing all gc relocates of the previous one and can safely discard previously saved location.

See comment on line 557 in StatepointLowering.cpp for more info. Test case "test_same_val" checks this case.

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

535 ↗

(On Diff #21229)

In Ptr's array we only have values with unique SDValues. But we need to store locations of the all possible relocated values. Notice that this loop is over StatepointSite.getRelocates().

746 ↗

(On Diff #21229)

There is such cases for constants, allocas and undefs. See "test_null_undef" and "test_alloca_and_const" tests.

755 ↗

(On Diff #21229)

At the beginning of each statepoint we know that all previous relocates were visited (otherwise statepoint intervals will intersect and it does not have any sense). It means that we are not going to need locations for this relocates and can reuse all allocated slots.

To be clear, by "previous relocates" I mean relocates that dominate current safepoint. For example relocates on exceptional path could have been left unvisited. This is not problematic because we are still reusing slots only at the point dominated by only visited relocates. I.e when we start reusing slots on exceptional path all exceptional relocates should be visited.

[reames]

Please make the changes suggested in previous rounds of discussion. I do not plan to respond further until at least the minor items are addressed. If you *disagree* with a proposed change, we can discuss, but anything which can be addressed without discussion should be done before replying. Reviewing a concrete piece of code is much easier than keeping track of proposed revisions.

include/llvm/CodeGen/FunctionLoweringInfo.h

82 ↗

(On Diff #21229)

I'm having trouble understanding your comment in context of the comment in the code. Are you saying we remove entries from this map after visiting a particular safepoint? That doesn't seem to agree with the comment in code.

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

535 ↗

(On Diff #21229)

Er, yes. But by definition, the unique SDValues *are* all the values that need to be relocated. That was my point.

746 ↗

(On Diff #21229)

Please clarify the comment to include the first half of your response.

Also, the fact this is the *exact* same case as the previous but for the invoke handling seems questionable. I might suggest:
if (isCall) {

above code;

} else {

new code with comment

}

755 ↗

(On Diff #21229)

I'm still confused here. Really, really basic question: What is the invariant this code is trying to uphold? Is the same Value always spilled to the same slot? Please summarize this in a block comment in the revised code and point me to it.

[sanjoy]

Some minor comments inline. I can't say I've fully understood this; I'll do a second, review tomorrow.

include/llvm/CodeGen/FunctionLoweringInfo.h

82 ↗

(On Diff #24709)

Why not use a DenseMap<const Value *, Optional<int>>?

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

554 ↗

(On Diff #24709)

LLVM conventions say variable names should be CamelCase starting with an uppercase letter.

574 ↗

(On Diff #24709)

So this case is for values like allocas and constants, right? If so, an assert will make things clearer.

579 ↗

(On Diff #24709)

So is it correct to say that we need this because gc_relocate only "indirectly" uses V, in a manner invisible to LLVM? For instance, if our representation of a gc_relocate was

TOK = safepoint
i8* V.relocated = gc_relocate(TOK, i8* V.original)

then we would not need this since V.original would get exported at its def if needed?

[igor-laevsky]

I think a lot of confusion were due to the two mechanisms for recording spilled values locations - one inside StatepointLowering, and one for cross basic block values in FunctionLoweringInfo.

I updated diff so that only one such mechanism is left. Now we record all value locations in "StatepointRelocatedValues" map in FuncitonLoweringInfo.

Also I included statepoint instruction as a key to this map to make it obvious that there could not be any collisions when we allocate different stack sots for the same values in different statepoints. This is not required for correctness, but makes reasoning about correctness a lot easier.

As a side effect "RelocLocations" map is no longer possible to maintain with reasonable effort, so I removed it completely. Anyway it added a lot of complexity while serving only purpose to not emit spill loads for values with non-unique sdvalues. (I.e preventing several spill loads from the same slots) After running some experiments I see that machineCSE is capable to handle such cases for us.

Also after this change it is clear that all structures inside StatepointLowering are used only during call to the "LowerStatepoint" (they are cleared at the beginning of this function). So it will be possible to remove StatepointLowering from SelectionDAG and create it only inside "LowerStatepoint".

[igor-laevsky]

include/llvm/CodeGen/FunctionLoweringInfo.h

82 ↗

(On Diff #21229)

This should not be the problem anymore as I added statepoint as a key to the map.

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

579 ↗

(On Diff #24709)

That's right, but only for values we are not spilling on the stack, like allocas. For spilled values even if we will export them we still need to know stack slot we spilled them to.

(I am not considering that we can not have gc_relocate using original value as it would mean use of unrelocated value after statepoint)

535 ↗

(On Diff #21229)

That is correct. We need to record in a StackMap and spill only unique SDValues. However we might visit two different gc_relocates relocating values with same SDValues. In order to handle this we are recording locations for all possible values. Simply for values with same SDValues we will have same locations.

746 ↗

(On Diff #21229)

Should not be the problem in updated code.

755 ↗

(On Diff #21229)

Basic invariant here is that statepoint ranges are not intersecting. Or in other words - on entry to each statepoint all previous gc_relocates are already visited.

This is not a new invariant, current code asserts it in startNewStatepoint by checking "PendingGCRelocateCalls.empty()". Unfortunately I have not found easy way of asserting it for invoke statepoints, but IR verification should prevent such cases anyway.

[igor-laevsky]

Very gentle ping. I believe code is easier to understand now. If it is still hard to read - please tell me about it, I will try something else.

[sanjoy]

I think this is mostly ready to go in. Let's do one more iteration if you don't mind.

Btw, some of the GCRelocateOperands accessors you use have been renamed; but those will show up as compiler errors.

include/llvm/CodeGen/FunctionLoweringInfo.h

79 ↗

(On Diff #25190)

What is "location" here? Is it the frame index?

83 ↗

(On Diff #25190)

I'd find this easier to understand if you had DenseMap<Instruction *, DenseMap<Value *, int>> instead. IOW, currently your map is of the type (Statept, Value) -> Index, but I think making it of the type Statept -> Value -> Index would be clearer (even though it is equivalent) as it would emphasize that per safepoint we have a mapping of Values to frame indices.

85 ↗

(On Diff #25190)

I'm not sure if the typedef is adding much here.

lib/CodeGen/SelectionDAG/StatepointLowering.cpp

540 ↗

(On Diff #25190)

Minor: should be embedded.

542 ↗

(On Diff #25190)

Use RelocatedOpers or RO as the variable name.

545 ↗

(On Diff #25190)

LLVM style would be SDV here.

553 ↗

(On Diff #25190)

I think LLVM style is } else {. You might want to just run this patch through clang-format before check-in.

557 ↗

(On Diff #25190)

I'd change the language slightly: "Actually we do not need to record them in this map at all." assuming that is what you meant.

730 ↗

(On Diff #25190)

Use RelocateOpers or RO.

[sanjoy]

LGTM.

llvm/trunk/include/llvm/CodeGen/FunctionLoweringInfo.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_FUNCTIONLOWERINGINFO_H
 #define LLVM_CODEGEN_FUNCTIONLOWERINGINFO_H
 
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/IndexedMap.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/Target/TargetRegisterInfo.h"
 #include <vector>
public:
   /// MBBMap - A mapping from LLVM basic blocks to their machine code entry.
   DenseMap<const BasicBlock*, MachineBasicBlock *> MBBMap;
 
   /// ValueMap - Since we emit code for the function a basic block at a time,
   /// we must remember which virtual registers hold the values for
   /// cross-basic-block values.
   DenseMap<const Value*, unsigned> ValueMap;
 
+  // Keep track of frame indices allocated for statepoints as they could be used
+  // across basic block boundaries.
+  // Key of the map is statepoint instruction, value is a map from spilled
+  // llvm Value to the optional stack stack slot index.
+  // If optional is unspecified it means that we have visited this value
+  // but didn't spill it.
+  typedef DenseMap<const Value*, Optional<int>> StatepointSpilledValueMapTy;
+  DenseMap<const Instruction*, StatepointSpilledValueMapTy>
+    StatepointRelocatedValues;
+
   /// StaticAllocaMap - Keep track of frame indices for fixed sized allocas in
   /// the entry block.  This allows the allocas to be efficiently referenced
   /// anywhere in the function.
   DenseMap<const AllocaInst*, int> StaticAllocaMap;
 
   /// ByValArgFrameIndexMap - Keep track of frame indices for byval arguments.
   DenseMap<const Argument*, int> ByValArgFrameIndexMap;
 

llvm/trunk/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

#ifndef NDEBUG
   CatchInfoFound.clear();
 #endif
   LiveOutRegInfo.clear();
   VisitedBBs.clear();
   ArgDbgValues.clear();
   ByValArgFrameIndexMap.clear();
   RegFixups.clear();
   StatepointStackSlots.clear();
+  StatepointRelocatedValues.clear();
   PreferredExtendType.clear();
 }
 
 /// CreateReg - Allocate a single virtual register for the given type.
 unsigned FunctionLoweringInfo::CreateReg(MVT VT) {
   return RegInfo->createVirtualRegister(
       MF->getSubtarget().getTargetLowering()->getRegClassFor(VT));
 }

llvm/trunk/lib/CodeGen/SelectionDAG/StatepointLowering.h

SDValue getLocation(SDValue val) {
     return Locations[val];
   }
   void setLocation(SDValue val, SDValue Location) {
     assert(!Locations.count(val) &&
            "Trying to allocate already allocated location");
     Locations[val] = Location;
   }
 
-  /// Returns the relocated value for a given input pointer. Will
-  /// return SDValue() if this value hasn't yet been reloaded from
-  /// it's stack slot after the statepoint.  Otherwise, the value
-  /// has already been reloaded and the SDValue of that reload will
-  /// be returned. Note that VMState values are spilled but not
-  /// reloaded (since they don't change at the safepoint unless
-  /// also listed in the GC pointer section) and will thus never
-  /// be in this map
-  SDValue getRelocLocation(SDValue val) {
-    if (!RelocLocations.count(val))
-      return SDValue();
-    return RelocLocations[val];
-  }
-  void setRelocLocation(SDValue val, SDValue Location) {
-    assert(!RelocLocations.count(val) &&
-           "Trying to allocate already allocated location");
-    RelocLocations[val] = Location;
-  }
-
   /// Record the fact that we expect to encounter a given gc_relocate
   /// before the next statepoint.  If we don't see it, we'll report
   /// an assertion.
   void scheduleRelocCall(const CallInst &RelocCall) {
     PendingGCRelocateCalls.push_back(&RelocCall);
   }
   /// Remove this gc_relocate from the list we're expecting to see
   /// before the next statepoint.  If we weren't expecting to see
assert(Offset >= 0 && Offset < (int)AllocatedStackSlots.size() &&
            "out of bounds");
     return AllocatedStackSlots[Offset];
   }
 
 private:
   /// Maps pre-relocation value (gc pointer directly incoming into statepoint)
   /// into it's location (currently only stack slots)
   DenseMap<SDValue, SDValue> Locations;
-  /// Map pre-relocated value into it's new relocated location
-  DenseMap<SDValue, SDValue> RelocLocations;
 
   /// A boolean indicator for each slot listed in the FunctionInfo as to
   /// whether it has been used in the current statepoint.  Since we try to
   /// preserve stack slots across safepoints, there can be gaps in which
   /// slots have been allocated.
   SmallVector<bool, 50> AllocatedStackSlots;
 
   /// Points just beyond the last slot known to have been allocated
   unsigned NextSlotToAllocate;
 
   /// Keep track of pending gcrelocate calls for consistency check
   SmallVector<const CallInst *, 10> PendingGCRelocateCalls;
 };
 } // end namespace llvm
 
 #endif // LLVM_LIB_CODEGEN_SELECTIONDAG_STATEPOINTLOWERING_H

llvm/trunk/lib/CodeGen/SelectionDAG/StatepointLowering.cpp

static void pushStackMapConstant(SmallVectorImpl<SDValue>& Ops,
   Ops.push_back(Builder.DAG.getTargetConstant(Value, L, MVT::i64));
 }
 
 void StatepointLoweringState::startNewStatepoint(SelectionDAGBuilder &Builder) {
   // Consistency check
   assert(PendingGCRelocateCalls.empty() &&
          "Trying to visit statepoint before finished processing previous one");
   Locations.clear();
-  RelocLocations.clear();
   NextSlotToAllocate = 0;
   // Need to resize this on each safepoint - we need the two to stay in
   // sync and the clear patterns of a SelectionDAGBuilder have no relation
   // to FunctionLoweringInfo.
   AllocatedStackSlots.resize(Builder.FuncInfo.StatepointStackSlots.size());
   for (size_t i = 0; i < AllocatedStackSlots.size(); i++) {
     AllocatedStackSlots[i] = false;
   }
 }
+
 void StatepointLoweringState::clear() {
   Locations.clear();
-  RelocLocations.clear();
   AllocatedStackSlots.clear();
   assert(PendingGCRelocateCalls.empty() &&
          "cleared before statepoint sequence completed");
 }
 
 SDValue
 StatepointLoweringState::allocateStackSlot(EVT ValueType,
                                            SelectionDAGBuilder &Builder) {
#endif
   for (Value *V : StatepointSite.gc_args()) {
     SDValue Incoming = Builder.getValue(V);
     if (FrameIndexSDNode *FI = dyn_cast<FrameIndexSDNode>(Incoming)) {
       // This handles allocas as arguments to the statepoint
       Ops.push_back(Builder.DAG.getTargetFrameIndex(FI->getIndex(),
                                                     Incoming.getValueType()));
     }
   }
+
+  // Record computed locations for all lowered values.
+  // This can not be embedded in lowering loops as we need to record *all*
+  // values, while previous loops account only values with unique SDValues.
+  const Instruction *StatepointInstr =
+    StatepointSite.getCallSite().getInstruction();
+  FunctionLoweringInfo::StatepointSpilledValueMapTy &SpillMap =
+    Builder.FuncInfo.StatepointRelocatedValues[StatepointInstr];
+
+  for (GCRelocateOperands RelocateOpers :
+       StatepointSite.getRelocates(StatepointSite)) {
+    const Value *V = RelocateOpers.getDerivedPtr();
+    SDValue SDV = Builder.getValue(V);
+    SDValue Loc = Builder.StatepointLowering.getLocation(SDV);
+
+    if (Loc.getNode()) {
+      SpillMap[V] = cast<FrameIndexSDNode>(Loc)->getIndex();
+    } else {
+      // Record value as visited, but not spilled. This is case for allocas
+      // and constants. For this values we can avoid emiting spill load while
+      // visiting corresponding gc_relocate.
+      // Actually we do not need to record them in this map at all.
+      // We do this only to check that we are not relocating any unvisited value.
+      SpillMap[V] = None;
+
+      // Default llvm mechanisms for exporting values which are used in
+      // different basic blocks does not work for gc relocates.
+      // Note that it would be incorrect to teach llvm that all relocates are
+      // uses of the corresponging values so that it would automatically
+      // export them. Relocates of the spilled values does not use original
+      // value.
+      if (StatepointSite.getCallSite().isInvoke())
+        Builder.ExportFromCurrentBlock(V);
+    }
+  }
 }
 
 void SelectionDAGBuilder::visitStatepoint(const CallInst &CI) {
   // Check some preconditions for sanity
   assert(isStatepoint(&CI) &&
          "function called must be the statepoint function");
 
   LowerStatepoint(ImmutableStatepoint(&CI));
 }
 
 void SelectionDAGBuilder::LowerStatepoint(
     ImmutableStatepoint ISP, MachineBasicBlock *LandingPad /*=nullptr*/) {
   // The basic scheme here is that information about both the original call and
   // the safepoint is encoded in the CallInst.  We create a temporary call and
   // lower it, then reverse engineer the calling sequence.
 
   NumOfStatepoints++;
   // Clear state
   StatepointLowering.startNewStatepoint(*this);
 
   ImmutableCallSite CS(ISP.getCallSite());
 
 #ifndef NDEBUG
-  // Consistency check
+  // Consistency check. Don't do this for invokes. It would be too
+  // expensive to preserve this information across different basic blocks
+  if (!CS.isInvoke()) {
     for (const User *U : CS->users()) {
       const CallInst *Call = cast<CallInst>(U);
       if (isGCRelocate(Call))
         StatepointLowering.scheduleRelocCall(*Call);
     }
+  }
 #endif
 
 #ifndef NDEBUG
   // If this is a malformed statepoint, report it early to simplify debugging.
   // This should catch any IR level mistake that's made when constructing or
   // transforming statepoints.
   ISP.verify();
 
if (isa<InvokeInst>(I)) {
     assert(CopyFromReg.getNode());
     setValue(&CI, CopyFromReg);
   } else {
     setValue(&CI, getValue(I));
   }
 }
 
 void SelectionDAGBuilder::visitGCRelocate(const CallInst &CI) {
+  GCRelocateOperands RelocateOpers(&CI);
+
 #ifndef NDEBUG
   // Consistency check
+  // We skip this check for invoke statepoints. It would be too expensive to
+  // preserve validation info through different basic blocks.
+  if (!RelocateOpers.isTiedToInvoke()) {
     StatepointLowering.relocCallVisited(CI);
+  }
 #endif
 
-  GCRelocateOperands relocateOpers(&CI);
-  SDValue SD = getValue(relocateOpers.getDerivedPtr());
+  const Value *DerivedPtr = RelocateOpers.getDerivedPtr();
+  SDValue SD = getValue(DerivedPtr);
+
+  FunctionLoweringInfo::StatepointSpilledValueMapTy &SpillMap =
+    FuncInfo.StatepointRelocatedValues[RelocateOpers.getStatepoint()];
+
+  // We should have recorded location for this pointer
+  assert(SpillMap.count(DerivedPtr) && "Relocating not lowered gc value");
+  Optional<int> DerivedPtrLocation = SpillMap[DerivedPtr];
 
-  if (isa<ConstantSDNode>(SD) || isa<FrameIndexSDNode>(SD)) {
-    // We didn't need to spill these special cases (constants and allocas).
-    // See the handling in spillIncomingValueForStatepoint for detail.
+  if (!DerivedPtrLocation) {
     setValue(&CI, SD);
     return;
   }
 
-  SDValue Loc = StatepointLowering.getRelocLocation(SD);
-  // Emit new load if we did not emit it before
-  if (!Loc.getNode()) {
-    SDValue SpillSlot = StatepointLowering.getLocation(SD);
-    int FI = cast<FrameIndexSDNode>(SpillSlot)->getIndex();
+  SDValue SpillSlot = DAG.getTargetFrameIndex(*DerivedPtrLocation,
+                                              SD.getValueType());
 
-    // Be conservative: flush all pending loads
-    // TODO: Probably we can be less restrictive on this,
-    // it may allow more scheduling opprtunities
-    SDValue Chain = getRoot();
 
-    Loc = DAG.getLoad(SpillSlot.getValueType(), getCurSDLoc(), Chain, SpillSlot,
-                      MachinePointerInfo::getFixedStack(FI), false, false,
-                      false, 0);
-
-    StatepointLowering.setRelocLocation(SD, Loc);
+  SDValue SpillLoad =
+    DAG.getLoad(SpillSlot.getValueType(), getCurSDLoc(), Chain, SpillSlot,
+                MachinePointerInfo::getFixedStack(*DerivedPtrLocation),
+                false, false, false, 0);
 
-    // Again, be conservative, don't emit pending loads
-    DAG.setRoot(Loc.getValue(1));
-  }
+  DAG.setRoot(SpillLoad.getValue(1));
 
-  assert(Loc.getNode());
-  setValue(&CI, Loc);
+  assert(SpillLoad.getNode());
+  setValue(&CI, SpillLoad);
 }

llvm/trunk/test/CodeGen/X86/statepoint-invoke.ll

 ; RUN: llc < %s 2>&1 | FileCheck %s
 
 target triple = "x86_64-pc-linux-gnu"
 
+declare void @"some_call"(i64 addrspace(1)*)
 declare i64 addrspace(1)* @"some_other_call"(i64 addrspace(1)*)
 
 declare i32 @"personality_function"()
 
-define i64 addrspace(1)* @test_result(i64 addrspace(1)* %obj, 
+define i64 addrspace(1)* @test_basic(i64 addrspace(1)* %obj,
+                                     i64 addrspace(1)* %obj1)
+gc "statepoint-example" {
+entry:
+  ; CHECK: Ltmp{{[0-9]+}}:
+  ; CHECK: callq some_call
+  ; CHECK: Ltmp{{[0-9]+}}:
+  %0 = invoke i32 (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %obj, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1)
+          to label %invoke_safepoint_normal_dest unwind label %exceptional_return
+
+invoke_safepoint_normal_dest:
+  ; CHECK: movq
+  %obj.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %0, i32 13, i32 13)
+  %obj1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %0, i32 14, i32 14)
+  br label %normal_return
+
+normal_return:
+  ; CHECK: retq
+  ret i64 addrspace(1)* %obj.relocated
+
+exceptional_return:
+  ; CHECK: Ltmp{{[0-9]+}}:
+  ; CHECK: movq
+  ; CHECK: retq
+  %landing_pad = landingpad { i8*, i32 } personality i32 ()* @"personality_function"
+          cleanup
+  %relocate_token = extractvalue { i8*, i32 } %landing_pad, 1
+  %obj.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 13, i32 13)
+  %obj1.relocated1 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 14, i32 14)
+  ret i64 addrspace(1)* %obj1.relocated1
+}
+; CHECK-LABEL: GCC_except_table{{[0-9]+}}:
+; CHECK: .long  .Ltmp{{[0-9]+}}-.Ltmp{{[0-9]+}}
+; CHECK: .long  .Ltmp{{[0-9]+}}-.Lfunc_begin{{[0-9]+}}
+; CHECK: .byte  0
+; CHECK: .align 4
+
+define i64 addrspace(1)* @test_result(i64 addrspace(1)* %obj,
                                       i64 addrspace(1)* %obj1)
   gc "statepoint-example" {
 entry:
   ; CHECK: .Ltmp{{[0-9]+}}:
   ; CHECK: callq some_other_call
   ; CHECK: .Ltmp{{[0-9]+}}:
   %0 = invoke i32 (i64, i32, i64 addrspace(1)* (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_p1i64p1i64f(i64 0, i32 0, i64 addrspace(1)* (i64 addrspace(1)*)* @some_other_call, i32 1, i32 0, i64 addrspace(1)* %obj, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* %obj, i64 addrspace(1)* %obj1)
           to label %normal_return unwind label %exceptional_return
 
 normal_return:
   ; CHECK: popq 
   ; CHECK: retq
   %ret_val = call i64 addrspace(1)* @llvm.experimental.gc.result.p1i64(i32 %0)
   ret i64 addrspace(1)* %ret_val
 
 exceptional_return:
   ; CHECK: .Ltmp{{[0-9]+}}:
-  ; CHECK: popq
-  ; CHECK: retq
+  ; CHECK: movq
   %landing_pad = landingpad { i8*, i32 } personality i32 ()* @personality_function
           cleanup
-  ret i64 addrspace(1)* %obj
+  %relocate_token = extractvalue { i8*, i32 } %landing_pad, 1
+  %obj.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 13, i32 13)
+  ret i64 addrspace(1)* %obj.relocated
 }
 ; CHECK-LABEL: GCC_except_table{{[0-9]+}}:
 ; CHECK: .long .Ltmp{{[0-9]+}}-.Ltmp{{[0-9]+}}
 ; CHECK: .long .Ltmp{{[0-9]+}}-.Lfunc_begin{{[0-9]+}}
 ; CHECK: .byte 0
 ; CHECK: .align 4
 
+define i64 addrspace(1)* @test_same_val(i1 %cond, i64 addrspace(1)* %val1, i64 addrspace(1)* %val2, i64 addrspace(1)* %val3)
+  gc "statepoint-example" {
+entry:
+  br i1 %cond, label %left, label %right
+
+left:
+  ; CHECK-LABEL: %left
+  ; CHECK: movq %rdx, 8(%rsp)
+  ; CHECK: movq
+  ; CHECK: callq some_call
+  %sp1 = invoke i32 (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* %val1, i64 addrspace(1)* %val2)
+           to label %left.relocs unwind label %exceptional_return.left
+
+left.relocs:
+  ; CHECK: movq (%rsp),
+  ; CHECK: movq 8(%rsp), [[REGVAL2:%[a-z]+]]
+  %val1.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp1, i32 13, i32 13)
+  %val2.relocated_left = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp1, i32 14, i32 14)
+  br label %normal_return
+
+right:
+  ; CHECK-LABEL: %right
+  ; CHECK: movq
+  ; CHECK: movq %rdx, (%rsp)
+  ; CHECK: callq some_call
+  %sp2 = invoke i32 (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* %val2, i64 addrspace(1)* %val3)
+           to label %right.relocs unwind label %exceptional_return.right
+
+right.relocs:
+  ; CHECK: movq (%rsp), [[REGVAL2]]
+  ; CHECK: movq
+  %val2.relocated_right = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp2, i32 13, i32 13)
+  %val3.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp2, i32 14, i32 14)
+  br label %normal_return
+
+normal_return:
+  ; CHECK-LABEL: %normal_return
+  ; CHECK: cmoveq {{.*}}[[REGVAL2]]{{.*}}
+  ; CHECK retq
+  %a1 = phi i64 addrspace(1)* [%val1.relocated, %left.relocs], [%val3.relocated, %right.relocs]
+  %a2 = phi i64 addrspace(1)* [%val2.relocated_left, %left.relocs], [%val2.relocated_right, %right.relocs]
+  %ret = select i1 %cond, i64 addrspace(1)* %a1, i64 addrspace(1)* %a2
+  ret i64 addrspace(1)* %ret
+
+exceptional_return.left:
+  %landing_pad = landingpad { i8*, i32 } personality i32 ()* @"personality_function"
+          cleanup
+  %relocate_token = extractvalue { i8*, i32 } %landing_pad, 1
+  %val.relocated2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 13, i32 13)
+  ret i64 addrspace(1)* %val.relocated2
+
+exceptional_return.right:
+  %landing_pad1 = landingpad { i8*, i32 } personality i32 ()* @"personality_function"
+          cleanup
+  %relocate_token1 = extractvalue { i8*, i32 } %landing_pad1, 1
+  %val.relocated3 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token1, i32 13, i32 13)
+  ret i64 addrspace(1)* %val.relocated3
+}
+
+define i64 addrspace(1)* @test_null_undef(i64 addrspace(1)* %val1)
+       gc "statepoint-example" {
+; CHECK-LABEL: test_null_undef:
+entry:
+  ; CHECK: callq some_call
+  %sp1 = invoke i32 (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* null, i64 addrspace(1)* undef)
+           to label %normal_return unwind label %exceptional_return
+
+normal_return:
+  ; CHECK-LABEL: %normal_return
+  ; CHECK: xorl %eax, %eax
+  ; CHECK-NEXT: popq
+  ; CHECK-NEXT: retq
+  %null.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp1, i32 13, i32 13)
+  %undef.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %sp1, i32 14, i32 14)
+  ret i64 addrspace(1)* %null.relocated
+
+exceptional_return:
+  %landing_pad = landingpad { i8*, i32 } personality i32 ()* @"personality_function"
+          cleanup
+  %relocate_token = extractvalue { i8*, i32 } %landing_pad, 1
+  %null.relocated2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 13, i32 13)
+  %undef.relocated2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 14, i32 14)
+  ret i64 addrspace(1)* %null.relocated2
+}
+
+define i64 addrspace(1)* @test_alloca_and_const(i64 addrspace(1)* %val1)
+       gc "statepoint-example" {
+; CHECK-LABEL: test_alloca_and_const:
+entry:
+  %a = alloca i32
+  %aa = addrspacecast i32* %a to i32 addrspace(1)*
+  %c = inttoptr i64 15 to i64 addrspace(1)*
+  ; CHECK: callq
+  %sp = invoke i32 (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i32 addrspace(1)* %aa, i64 addrspace(1)* %c)
+           to label %normal_return unwind label %exceptional_return
+
+normal_return:
+  ; CHECK: leaq
+  ; CHECK-NEXT: popq
+  ; CHECK-NEXT: retq
+  %aa.rel = call coldcc i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(i32 %sp, i32 13, i32 13)
+  %aa.converted = bitcast i32 addrspace(1)* %aa.rel to i64 addrspace(1)*
+  ret i64 addrspace(1)* %aa.converted
+
+exceptional_return:
+  ; CHECK: movl $15
+  ; CHECK-NEXT: popq
+  ; CHECK-NEXT: retq
+  %landing_pad = landingpad { i8*, i32 } personality i32 ()* @"personality_function"
+          cleanup
+  %relocate_token = extractvalue { i8*, i32 } %landing_pad, 1
+  %aa.rel2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32 %relocate_token, i32 14, i32 14)
+  ret i64 addrspace(1)* %aa.rel2
+}
+
+declare i32 @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...)
 declare i32 @llvm.experimental.gc.statepoint.p0f_p1i64p1i64f(i64, i32, i64 addrspace(1)* (i64 addrspace(1)*)*, i32, i32, ...)
-declare i64 addrspace(1)* @llvm.experimental.gc.result.p1i64(i32)
+
+declare i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(i32, i32, i32)
+declare i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(i32, i32, i32)
+declare i64 addrspace(1)* @llvm.experimental.gc.result.p1i64(i32)