[RS4GC] Rematerialize derived pointers before uses.

[dantrushin]

Summary

Introduce an option to rematerialize derived pointers immediately
before their uses instead of after every statepoint. This can be
beneficial when pointer is live across many statepoints but has
few uses.
Initial implementation is simple and rematerializes derived pointer
before every use, even if there are several uses in the same block
or rematerialization instructions can be hoisted etc.
Transformation is considered profitable if we would insert less
instructions than we would insert after every live statepoint.

Depends on D138910, D138911

[dantrushin]

Rebase on tip after D138910 has been landed.

[skatkov]

LGTM, please wait a bit for someone else to look at.

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

2473

Move above as to exit earlier?

2504

looks profitable

2511

!empty()?

[dantrushin]

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

2511

Nope.More than one - i.e., it has a sub-chain which could have changed.
If chain length is 1 we cannot spoil it.

[skatkov]

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

2511

Right

[dantrushin]

Address review comments.
Also include changes for pre-committed tests

[skatkov]

Still lgtm. Please wait one or two days for other possible reviews.

[anna]

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

2442

Nit: beneficial

2517

So, these remats will be present till the end right? Seems suboptimal (in terms of compile time) especially if we have multiple uses within a single BB.

2518

Nit: or if

[dantrushin]

Address review comments (fix typos).

[dantrushin]

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

2517

I left it as a follow up improvement because implementing it will complicate implementation.
There may be several statepoints intermixed with uses in the same block. Handling of this is not so trivial. I think we will need some refactoring
and this initial implementation would be motivating case for it :-)

Regarding compile time:

1. Duplicating chains will be optimized away by LocalCSE (hopefully).
2. Profitability heuristic accounts for every such copy and we guaranteed to have not more copies that we would have if we rematerialized after every statepoint.
3. As a job security, this would be opportunity for possible compile time improvement project (just kidding)

[anna]

This comment was removed by anna.

[anna]

okay LGTM then.

llvm/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

 static cl::opt<bool, true> ClobberNonLiveOverride("rs4gc-clobber-non-live",
                                                   cl::location(ClobberNonLive),
                                                   cl::Hidden);
 
 static cl::opt<bool>
     AllowStatepointWithNoDeoptInfo("rs4gc-allow-statepoint-with-no-deopt-info",
                                    cl::Hidden, cl::init(true));
 
+static cl::opt<bool> RematDerivedAtUses("rs4gc-remat-derived-at-uses",
+                                        cl::Hidden, cl::init(true));
+
 /// The IR fed into RewriteStatepointsForGC may have had attributes and
 /// metadata implying dereferenceability that are no longer valid/correct after
 /// RewriteStatepointsForGC has run. This is because semantically, after
 /// RewriteStatepointsForGC runs, all calls to gc.statepoint "free" the entire
 /// heap. stripNonValidData (conservatively) restores
 /// correctness by erasing all attributes in the module that externally imply
 /// dereferenceability. Similar reasoning also applies to the noalias
 /// attributes and metadata. gc.statepoint can touch the entire heap including
for (auto P2B : PointerToBase) {
     RematerizlizationCandidateRecord Record;
     Record.ChainToBase = ChainToBase;
     Record.RootOfChain = RootOfChain;
     Record.Cost = Cost;
     RematerizationCandidates.insert({ Derived, Record });
   }
 }
 
+// Try to rematerialize derived pointers immediately before their uses
+// (instead of rematerializing after every statepoint it is live through).
+// This can be beneficial when derived pointer is live across many
+// statepoints, but uses are rare.
+static void rematerializeLiveValuesAtUses(
+    RematCandTy &RematerizationCandidates,
+    MutableArrayRef<PartiallyConstructedSafepointRecord> Records,
+    PointerToBaseTy &PointerToBase) {
+  if (!RematDerivedAtUses)
+    return;
+
+  SmallVector<Instruction *, 32> LiveValuesToBeDeleted;
+
+  LLVM_DEBUG(dbgs() << "Rematerialize derived pointers at uses, "
+                    << "Num statepoints: " << Records.size() << '\n');
+
+  for (auto &It : RematerizationCandidates) {
+    Instruction *Cand = cast<Instruction>(It.first);
+    auto &Record = It.second;
+
+    if (Record.Cost >= RematerializationThreshold)
+      continue;
+
+    if (Cand->user_empty())
+      continue;
+
+    if (Cand->hasOneUse())
+      if (auto *U = dyn_cast<Instruction>(Cand->getUniqueUndroppableUser()))
+        if (U->getParent() == Cand->getParent())
+          continue;
+
+    // Rematerialization before PHI nodes is not implemented.
+    if (llvm::any_of(Cand->users(),
+                     [](const auto *U) { return isa<PHINode>(U); }))
+      continue;
+
+    LLVM_DEBUG(dbgs() << "Trying cand " << *Cand << " ... ");
+
+    // Count of rematerialization instructions we introduce is equal to number
+    // of candidate uses.
+    // Count of rematerialization instructions we eliminate is equal to number
+    // of statepoints it is live through.
+    // Consider transformation profitable if latter is greater than former
+    // (in other words, we create less than eliminate).
+    unsigned NumLiveStatepoints = llvm::count_if(
+        Records, [Cand](const auto &R) { return R.LiveSet.contains(Cand); });
+    unsigned NumUses = Cand->getNumUses();
+
+    LLVM_DEBUG(dbgs() << "Num uses: " << NumUses << " Num live statepoints: "
+                      << NumLiveStatepoints << " ");
+
+    if (NumLiveStatepoints < NumUses) {
+      LLVM_DEBUG(dbgs() << "not profitable\n");
+      continue;
+    }
+
+    // If rematerialization is 'free', then favor rematerialization at
+    // uses as it generally shortens live ranges.
+    // TODO: Short (size ==1) chains only?
+    if (NumLiveStatepoints == NumUses && Record.Cost > 0) {
+      LLVM_DEBUG(dbgs() << "not profitable\n");
+      continue;
+    }
+
+    LLVM_DEBUG(dbgs() << "looks profitable\n");
+
+    // ChainToBase may contain another remat candidate (as a sub chain) which
+    // has been rewritten by now. Need to recollect chain to have up to date
+    // value.
+    // TODO: sort records in findRematerializationCandidates() in
+    // decreasing chain size order?
+    if (Record.ChainToBase.size() > 1) {
+      Record.ChainToBase.clear();
+      findRematerializableChainToBasePointer(Record.ChainToBase, Cand);
+    }
+
+    // Current rematerialization algorithm is very simple: we rematerialize
+    // immediately before EVERY use, even if there are several uses in same
+    // block or if use is local to Cand Def. The reason is that this allows
+    // us to avoid recomputing liveness without complicated analysis:
+    // - If we did not eliminate all uses of original Candidate, we do not
+    //   know exaclty in what BBs it is still live.
+    // - If we rematerialize once per BB, we need to find proper insertion
+    //   place (first use in block, but after Def) and analyze if there is
+    //   statepoint between uses in the block.
+    while (!Cand->user_empty()) {
+      Instruction *UserI = cast<Instruction>(*Cand->user_begin());
+      Instruction *RematChain = rematerializeChain(
+          Record.ChainToBase, UserI, Record.RootOfChain, PointerToBase[Cand]);
+      UserI->replaceUsesOfWith(Cand, RematChain);
+      PointerToBase[RematChain] = PointerToBase[Cand];
+    }
+    LiveValuesToBeDeleted.push_back(Cand);
+  }
+
+  LLVM_DEBUG(dbgs() << "Rematerialized " << LiveValuesToBeDeleted.size()
+                    << " derived pointers\n");
+  for (auto *Cand : LiveValuesToBeDeleted) {
+    assert(Cand->use_empty() && "Unexpected user remain");
+    RematerizationCandidates.erase(Cand);
+    for (auto &R : Records) {
+      assert(!R.LiveSet.contains(Cand) ||
+             R.LiveSet.contains(PointerToBase[Cand]));
+      R.LiveSet.remove(Cand);
+    }
+  }
+
+  // Recollect not rematerialized chains - we might have rewritten
+  // their sub-chains.
+  if (!LiveValuesToBeDeleted.empty()) {
+    for (auto &P : RematerizationCandidates) {
+      auto &R = P.second;
+      if (R.ChainToBase.size() > 1) {
+        R.ChainToBase.clear();
+        findRematerializableChainToBasePointer(R.ChainToBase, P.first);
+      }
+    }
+  }
+}
+
 // From the statepoint live set pick values that are cheaper to recompute then
 // to relocate. Remove this values from the live set, rematerialize them after
 // statepoint and record them in "Info" structure. Note that similar to
 // relocated values we don't do any user adjustments here.
 static void rematerializeLiveValues(CallBase *Call,
                                     PartiallyConstructedSafepointRecord &Info,
                                     PointerToBaseTy &PointerToBase,
                                     RematCandTy &RematerizationCandidates,
#endif
 
   // Compute the cost of possible re-materialization of derived pointers.
   RematCandTy RematerizationCandidates;
   findRematerializationCandidates(PointerToBase, RematerizationCandidates, TTI);
 
   // In order to reduce live set of statepoint we might choose to rematerialize
   // some values instead of relocating them. This is purely an optimization and
   // does not influence correctness.
+  // First try rematerialization at uses, then after statepoints.
+  rematerializeLiveValuesAtUses(RematerizationCandidates, Records,
+                                PointerToBase);
   for (size_t i = 0; i < Records.size(); i++)
     rematerializeLiveValues(ToUpdate[i], Records[i], PointerToBase,
                             RematerizationCandidates, TTI);
 
   // We need this to safely RAUW and delete call or invoke return values that
   // may themselves be live over a statepoint.  For details, please see usage in
   // makeStatepointExplicitImpl.
   std::vector<DeferredReplacement> Replacements;

llvm/test/Transforms/RewriteStatepointsForGC/rematerialize-derived-pointers-at-uses.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes='rewrite-statepoints-for-gc,verify<safepoint-ir>' -S | FileCheck %s
 
 declare void @use_obj(ptr addrspace(1)) "gc-leaf-function"
 declare void @do_safepoint()
 
 
 ; Profitable test
 define i32 @test_remat(ptr addrspace(1) %base, i1 %cond) gc "statepoint-example" {
 ; CHECK-LABEL: @test_remat(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DERIVED:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE:%.*]], i32 16
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED]], i32 16
 ; CHECK-NEXT:    br i1 [[COND:%.*]], label [[HERE:%.*]], label [[THERE:%.*]]
 ; CHECK:       here:
-; CHECK-NEXT:    [[STATEPOINT_TOKEN3:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
-; CHECK-NEXT:    [[BASE_RELOCATED4:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN3]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT1:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED4]], i32 16
+; CHECK-NEXT:    [[STATEPOINT_TOKEN1:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
+; CHECK-NEXT:    [[BASE_RELOCATED2:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN1]], i32 0, i32 0)
 ; CHECK-NEXT:    br label [[MERGE:%.*]]
 ; CHECK:       there:
-; CHECK-NEXT:    [[STATEPOINT_TOKEN5:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
-; CHECK-NEXT:    [[BASE_RELOCATED6:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN5]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT2:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED6]], i32 16
+; CHECK-NEXT:    [[STATEPOINT_TOKEN3:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
+; CHECK-NEXT:    [[BASE_RELOCATED4:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN3]], i32 0, i32 0)
 ; CHECK-NEXT:    br label [[MERGE]]
 ; CHECK:       merge:
-; CHECK-NEXT:    [[DOT0:%.*]] = phi ptr addrspace(1) [ [[DERIVED_REMAT1]], [[HERE]] ], [ [[DERIVED_REMAT2]], [[THERE]] ]
-; CHECK-NEXT:    [[RET:%.*]] = load i32, ptr addrspace(1) [[DOT0]], align 4
+; CHECK-NEXT:    [[DOT0:%.*]] = phi ptr addrspace(1) [ [[BASE_RELOCATED2]], [[HERE]] ], [ [[BASE_RELOCATED4]], [[THERE]] ]
+; CHECK-NEXT:    [[DERIVED_REMAT:%.*]] = getelementptr i32, ptr addrspace(1) [[DOT0]], i32 16
+; CHECK-NEXT:    [[RET:%.*]] = load i32, ptr addrspace(1) [[DERIVED_REMAT]], align 4
 ; CHECK-NEXT:    ret i32 [[RET]]
 ;
 entry:
   %derived = getelementptr i32, ptr addrspace(1) %base, i32 16
   call void @do_safepoint() [ "deopt"() ]
   br i1 %cond, label %here, label %there
 
 here:

 ; Several uses per block
 ; TODO: We could rematerialize once per block for several uses if there is not statepoint between
 define i32 @test_same_block_with_statepoint(ptr addrspace(1) %base, i1 %cond) gc "statepoint-example" {
 ; CHECK-LABEL: @test_same_block_with_statepoint(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[DERIVED:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE:%.*]], i32 16
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED]], i32 16
 ; CHECK-NEXT:    br i1 [[COND:%.*]], label [[HERE:%.*]], label [[THERE:%.*]]
 ; CHECK:       here:
-; CHECK-NEXT:    call void @use_obj(ptr addrspace(1) [[DERIVED_REMAT]])
+; CHECK-NEXT:    [[DERIVED_REMAT3:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED]], i32 16
+; CHECK-NEXT:    call void @use_obj(ptr addrspace(1) [[DERIVED_REMAT3]])
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN4:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED5:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN4]], i32 0, i32 0)
+; CHECK-NEXT:    [[DERIVED_REMAT2:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED5]], i32 16
+; CHECK-NEXT:    call void @use_obj(ptr addrspace(1) [[DERIVED_REMAT2]])
 ; CHECK-NEXT:    [[DERIVED_REMAT1:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED5]], i32 16
-; CHECK-NEXT:    call void @use_obj(ptr addrspace(1) [[DERIVED_REMAT1]])
 ; CHECK-NEXT:    [[DUMMY:%.*]] = load i32, ptr addrspace(1) [[DERIVED_REMAT1]], align 4
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN6:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED5]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED7:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN6]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT2:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED7]], i32 16
 ; CHECK-NEXT:    br label [[MERGE:%.*]]
 ; CHECK:       there:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN8:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED9:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN8]], i32 0, i32 0)
-; CHECK-NEXT:    [[DERIVED_REMAT3:%.*]] = getelementptr i32, ptr addrspace(1) [[BASE_RELOCATED9]], i32 16
 ; CHECK-NEXT:    br label [[MERGE]]
 ; CHECK:       merge:
-; CHECK-NEXT:    [[DOT0:%.*]] = phi ptr addrspace(1) [ [[DERIVED_REMAT2]], [[HERE]] ], [ [[DERIVED_REMAT3]], [[THERE]] ]
-; CHECK-NEXT:    [[RET:%.*]] = load i32, ptr addrspace(1) [[DOT0]], align 4
+; CHECK-NEXT:    [[DOT0:%.*]] = phi ptr addrspace(1) [ [[BASE_RELOCATED7]], [[HERE]] ], [ [[BASE_RELOCATED9]], [[THERE]] ]
+; CHECK-NEXT:    [[DERIVED_REMAT:%.*]] = getelementptr i32, ptr addrspace(1) [[DOT0]], i32 16
+; CHECK-NEXT:    [[RET:%.*]] = load i32, ptr addrspace(1) [[DERIVED_REMAT]], align 4
 ; CHECK-NEXT:    ret i32 [[RET]]
 ;
 entry:
   %derived = getelementptr i32, ptr addrspace(1) %base, i32 16
   call void @do_safepoint() [ "deopt" () ]
   br i1 %cond, label %here, label %there
 
 here:

 ; Test long chain with sub-chain rematerialized
 ; TODO: If we rematerialized longer chain first (%v4), then shorter on (%v0) would become dead
 define void @test_chain(ptr addrspace(1) %base, i1 %cond1) gc "statepoint-example" {
 ; CHECK-LABEL: @test_chain(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[V0:%.*]] = getelementptr i8, ptr addrspace(1) [[BASE:%.*]], i64 16
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(ptr addrspace(1) [[BASE]]), "gc-live"(ptr addrspace(1) [[BASE]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN]], i32 0, i32 0)
-; CHECK-NEXT:    [[V0_REMAT:%.*]] = getelementptr i8, ptr addrspace(1) [[BASE_RELOCATED]], i64 16
 ; CHECK-NEXT:    br i1 [[COND1:%.*]], label [[BLOCK3:%.*]], label [[COMMON_RET:%.*]]
 ; CHECK:       block3:
+; CHECK-NEXT:    [[V0_REMAT:%.*]] = getelementptr i8, ptr addrspace(1) [[BASE_RELOCATED]], i64 16
 ; CHECK-NEXT:    [[V4:%.*]] = getelementptr i8, ptr addrspace(1) [[V0_REMAT]], i64 70
-; CHECK-NEXT:    [[STATEPOINT_TOKEN2:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(ptr addrspace(1) [[BASE_RELOCATED]]), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
-; CHECK-NEXT:    [[BASE_RELOCATED3:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN2]], i32 0, i32 0)
-; CHECK-NEXT:    [[V0_REMAT1:%.*]] = getelementptr i8, ptr addrspace(1) [[BASE_RELOCATED3]], i64 16
-; CHECK-NEXT:    [[V4_REMAT:%.*]] = getelementptr i8, ptr addrspace(1) [[V0_REMAT1]], i64 70
+; CHECK-NEXT:    [[STATEPOINT_TOKEN1:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(ptr addrspace(1) [[BASE_RELOCATED]]), "gc-live"(ptr addrspace(1) [[BASE_RELOCATED]]) ]
+; CHECK-NEXT:    [[BASE_RELOCATED2:%.*]] = call coldcc ptr addrspace(1) @llvm.experimental.gc.relocate.p1(token [[STATEPOINT_TOKEN1]], i32 0, i32 0)
+; CHECK-NEXT:    [[V0_REMAT_REMAT:%.*]] = getelementptr i8, ptr addrspace(1) [[BASE_RELOCATED2]], i64 16
+; CHECK-NEXT:    [[V4_REMAT:%.*]] = getelementptr i8, ptr addrspace(1) [[V0_REMAT_REMAT]], i64 70
 ; CHECK-NEXT:    [[V5:%.*]] = load atomic i8, ptr addrspace(1) [[V4_REMAT]] unordered, align 2
 ; CHECK-NEXT:    br label [[COMMON_RET]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %v0 = getelementptr i8, ptr addrspace(1) %base, i64 16
   call void @do_safepoint() [ "deopt"(ptr addrspace(1) %base) ]

llvm/test/Transforms/RewriteStatepointsForGC/rematerialize-derived-pointers.ll

 }
 
 define void @test_loop(i32 addrspace(1)* %base) gc "statepoint-example" {
 ; CHECK-LABEL: @test_loop(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[PTR_GEP:%.*]] = getelementptr i32, i32 addrspace(1)* [[BASE:%.*]], i32 15
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[DOT01:%.*]] = phi i32 addrspace(1)* [ [[PTR_GEP]], [[ENTRY:%.*]] ], [ [[PTR_GEP_REMAT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[DOT0:%.*]] = phi i32 addrspace(1)* [ [[BASE]], [[ENTRY]] ], [ [[BASE_RELOCATED_CASTED:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    call void @use_obj32(i32 addrspace(1)* [[DOT01]])
+; CHECK-NEXT:    [[DOT0:%.*]] = phi i32 addrspace(1)* [ [[BASE]], [[ENTRY:%.*]] ], [ [[BASE_RELOCATED_CASTED:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[PTR_GEP_REMAT:%.*]] = getelementptr i32, i32 addrspace(1)* [[DOT0]], i32 15
+; CHECK-NEXT:    call void @use_obj32(i32 addrspace(1)* [[PTR_GEP_REMAT]])
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 2882400000, i32 0, void ()* elementtype(void ()) @do_safepoint, i32 0, i32 0, i32 0, i32 0) [ "deopt"(), "gc-live"(i32 addrspace(1)* [[DOT0]]) ]
 ; CHECK-NEXT:    [[BASE_RELOCATED:%.*]] = call coldcc i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token [[STATEPOINT_TOKEN]], i32 0, i32 0)
 ; CHECK-NEXT:    [[BASE_RELOCATED_CASTED]] = bitcast i8 addrspace(1)* [[BASE_RELOCATED]] to i32 addrspace(1)*
-; CHECK-NEXT:    [[PTR_GEP_REMAT]] = getelementptr i32, i32 addrspace(1)* [[BASE_RELOCATED_CASTED]], i32 15
 ; CHECK-NEXT:    br label [[LOOP]]
 ;
 entry:
   %ptr.gep = getelementptr i32, i32 addrspace(1)* %base, i32 15
   br label %loop
 
 loop:                                             ; preds = %loop, %entry
   call void @use_obj32(i32 addrspace(1)* %ptr.gep)