[SROA] Maintain shadow/backing alloca when some slices are noncapturnig read-only calls to allow alloca partitioning/promotion

[lebedev.ri]

Summary

This is inspired by the original variant of D109749 by Graham Hunter,
but is a more general version.

Roughly, instead of promoting the alloca, we call it a shadow/backing alloca,
go through all it's slices, clone(!) instructions that operated on it,
but make them operate on the cloned alloca, and promote cloned alloca instead.

This keeps the shadow/backing alloca, and all the original instructions around,
which results in said shadow/backing alloca being a perfect mirror/representation
of the promoted alloca's content, so calls that take the alloca
as arguments (non-capturingly!) can be supported.

For now, we require that the calls also don't modify the alloca's content,
but that is only to simplify the initial implementation,
and that will be supported in a follow-up.

Overall, this leads to *smaller* codesize:
https://llvm-compile-time-tracker.com/compare.php?from=a8b4f5bbab62091835205f3d648902432a4a5b58&to=aeae054055b125b011c1122f82c86457e159436f&stat=size-total
and is roughly neutral compile-time wise:
https://llvm-compile-time-tracker.com/compare.php?from=a8b4f5bbab62091835205f3d648902432a4a5b58&to=aeae054055b125b011c1122f82c86457e159436f&stat=instructions

[jdoerfert]

This is really cool. There is a FIXME, are you expecting to address this? Other than that, reading through it seems sensible to me.

llvm/lib/Transforms/Scalar/CMakeLists.txt

101 ↗

(On Diff #385987)

leftover, I assume

[lebedev.ri]

Some assortment of small improvements.

In D113520#3119995, @jdoerfert wrote:

This is really cool.

Yep.

There is a FIXME, are you expecting to address this? Other than that, reading through it seems sensible to me.

I've added run line with -opaque-pointers, and everything appears to just work.
@nikic - can you confirm that this is fine?

llvm/lib/Transforms/Scalar/CMakeLists.txt

101 ↗

(On Diff #385987)

yep.

[lebedev.ri]

llvm/test/Transforms/SROA/non-capturing-call.ll

659–661

We fail to drop some dead instructions, which prevents deletion of the alloca itself.
Not yet sure how to deal with this.

[lebedev.ri]

In D113520#3120892, @lebedev.ri wrote:

Some assortment of small improvements.

In D113520#3119995, @jdoerfert wrote:

This is really cool.

Yep.

There is a FIXME, are you expecting to address this? Other than that, reading through it seems sensible to me.

I've added run line with -opaque-pointers, and everything appears to just work.
@nikic - can you confirm that this is fine?

Actually thinking about it, that is obviously right,
because what opaque pointers disallow is *extracting*
the pointee type out of the pointer type,
and here we only get/use the whole pointer type,
so it doesn't matter if it contains the pointee knowledge or not.

[lebedev.ri]

Ok, it turned out that we do need to delete no-longer-used instructions,
the added test would otherwise assert as not being promoteable.
Also, don't perform this fixup if there are no non-escaping uses of alloca.

There is at least one more problem, but i'm not sure what it is yet.

[lebedev.ri]

Introduce profitability heuristic so that the compiler doesn't take out
the whole machine while trying to build e.g. vanilla llvm test-suite :)

The current rule-of-thumb is:
For an escaped alloca to be worthy of promotion,
it must be used by loads/stores in this function (i.e., no point in promoting https://godbolt.org/z/c51Yavv9P)
and either have a small total size (32 bytes currently,
very much a guess), or most of the bytes of the alloca
(>=80%, also a guess) must be loaded/stored within this function (i.e. not much point in promoting https://godbolt.org/z/7KToqzb6z)

This results in these numbers:
https://llvm-compile-time-tracker.com/compare.php?from=8d35c054e31e5a2bee082f7a587a660eeb24bf99&to=c7226d770b1ffa1117f724dd25de7fa2881eed18&stat=instructions
... and i'm not sure what is going on with tramp3d-v4's NewPM-ReleaseThinLTO build,
i can't reproduce the horrible slowdown locally (do i need a production debug/assert-less build?).

This transform does not do anything crazy on that code,
the alloca's that happen to be promoted are very tiny (<=32b),
and we introduce only a single spill/reload per each,
so this isn't something i see as preventable with profitability checks.

[xbolva00]

tramp3d-v4 - codesize increased by 4% so more inlining(?), more IR, slower build.

[lebedev.ri]

In D113520#3129441, @lebedev.ri wrote:

Introduce profitability heuristic so that the compiler doesn't take out
the whole machine while trying to build e.g. vanilla llvm test-suite :)

Fixed the "occupancy" to only consider bytes that are both loaded and stored.
After all, we are mostly interested in allowing mem2reg,
aka, avoiding using stack instead of registers, but if not a single byte
is eligible, then we don't really care. Notably, the current check is *still*
too soft, as the last test shows.

This results in these numbers:
https://llvm-compile-time-tracker.com/compare.php?from=8d35c054e31e5a2bee082f7a587a660eeb24bf99&to=c7226d770b1ffa1117f724dd25de7fa2881eed18&stat=instructions
... and i'm not sure what is going on with tramp3d-v4's NewPM-ReleaseThinLTO build,
i can't reproduce the horrible slowdown locally (do i need a production debug/assert-less build?).

... and now we get https://llvm-compile-time-tracker.com/compare.php?from=2c91f48c48c42f9bd730d0791fcb19dbe0038d96&to=b4282b9b3cf6ebe17938a4041af21681b762b881&stat=instructions,
and the horrible tramp3d-v4 regression is gone :)

This is mostly the point where i'd want some feedback.

[lebedev.ri]

Ping.
I'm not really sure who is comfortable/okay with reviewing SROA changes nowadays, so the CC list is a bit too large.

[lebedev.ri]

Did some perf testing on RawSpeed, while the results will be heavily workload-depend,
here they seem to be mostly positively neutral with some improvements,
and no heavy losses.

rsbench.log114 KBDownload

[lebedev.ri]

ping

[reames]

Spent a decent amount of time looking at this, but haven't yet fully formed a recommendation. I'm going to summarize my findings to date in the hopes this is useful.

I think the basic idea here is valuable, but as implemented, I don't think this is a good idea. The current implementation can result in a serious pessimization in cases like the following:

a = alloca <4 x i8> ... // assume a is otherwise splittable
<init a to zero>
loop {
   if (very_rare) a[3] = 5;
   foo(a);
}

This transformation will have the effect of rewriting this as:

a = alloca <4 x i8> // all other uses of a split and ssa constructed
loop {
   *a =  very_rate ? <0,0,0,5> : <0,0,0,0> 
   foo(a);
}

The result is that we've made the store to a dramatically more expensive by moving it into the hotpath.

I think this is a fatal flaw with the approach. We could maybe patch around it with some profiling data, but in general, we don't want to be moving init logic.

However, this is where I get struck. I don't really have a suggestion on how to implement this better. Conceptually, what we want is something along the lines of D113289, but applied to SROA. That is, doing SSA formation to eliminate loads while leaving the stores in place for use by the calls which reads the allocas. However, there are two problems with that:

1. If we want to support non-readonly calls, we'd need to materialize a load after the call. This has the same "move to hot" problem as above. Maybe we can just restrict it to readonly arguments?
2. This really doesn't cleanly fit into the code structure of SROA. While SROA indirectly does SSA construction, it does so by forming another alloca, and then mem2regging that. We'd have to figure out something analogous to mem2reg which didn't remove the stores or the allocas.

Purely in terms of the code structure of the current patch, the integration of this is weird. The remat transform should be done over the AllocaSlices array after initially scanned, and should directly rebuild that data structure rather than relying on the code to handle duplicates in the array and double scanning. That doesn't address the algorithmic concern above though, so this ends up being mostly an aside.

This might be helpful to brainstorm offline. Let me know if you'd like to chat.

[lebedev.ri]

In D113520#3162304, @reames wrote:

Spent a decent amount of time looking at this, but haven't yet fully formed a recommendation. I'm going to summarize my findings to date in the hopes this is useful.

Thank you for taking a look!

I think the basic idea here is valuable, but as implemented, I don't think this is a good idea. The current implementation can result in a serious pessimization in cases like the following:

a = alloca <4 x i8> ... // assume a is otherwise splittable
<init a to zero>
loop {
   if (very_rare) a[3] = 5;
   foo(a);
}

This transformation will have the effect of rewriting this as:

a = alloca <4 x i8> // all other uses of a split and ssa constructed
loop {
   *a =  very_rate ? <0,0,0,5> : <0,0,0,0> 
   foo(a);
}

The result is that we've made the store to a dramatically more expensive by moving it into the hotpath.

I think this is a fatal flaw with the approach. We could maybe patch around it with some profiling data, but in general, we don't want to be moving init logic.

Indeed: https://godbolt.org/z/n64ozqnsT

However, this is where I get struck. I don't really have a suggestion on how to implement this better.

Conceptually, what we want is something along the lines of D113289, but applied to SROA.

Right. Not sure how much relevant potential is left after that change, to be noted.

That is, doing SSA formation to eliminate loads while leaving the stores in place for use by the calls which reads the allocas. However, there are two problems with that:

1. If we want to support non-readonly calls, we'd need to materialize a load after the call. This has the same "move to hot" problem as above.

True.

Maybe we can just restrict it to readonly arguments?

2. This really doesn't cleanly fit into the code structure of SROA. While SROA indirectly does SSA construction, it does so by forming another alloca, and then mem2regging that. We'd have to figure out something analogous to mem2reg which didn't remove the stores or the allocas.

Well, this one seems straight-forward - instead of spilling before the call, just go through all the slices,
and duplicate each store to store into our new alloca, this will essentially preserve all the original stores.

Purely in terms of the code structure of the current patch, the integration of this is weird. The remat transform should be done over the AllocaSlices array after initially scanned, and should directly rebuild that data structure rather than relying on the code to handle duplicates in the array and double scanning. That doesn't address the algorithmic concern above though, so this ends up being mostly an aside.

Note that i only scan the newly-added uses, i do not rescan everything, so i don't believe this particular concern is significant.

This might be helpful to brainstorm offline. Let me know if you'd like to chat.

[lebedev.ri]

I think the obvious solution is:

1. Keep all stores. (in terms of this patch, iterate over all slices and duplicate all store instructions to also store into the cloned alloca)
2. Keep all loads (in terms of this patch, iterate over all slices and before the load from the original alloca, load from the cloned alloca and store into the original alloca) UNLESS we can omit particular load because we can tell that there was no taint (may-write calls) on every path from the every previous store.

[reames]

@lebedev.ri Your suggested approach makes sense to me at a basic level. Forming the alloca is starting to seem more and more like a hack, but we can come back to implementing partial mem2reg as a follow on code improvement.

A couple of suggestions:

• Please leave taint tracking to later patch. Start with the requirement that the non-escaping call use must also be readonly in that argument.
• Instead of having the cloned alloca be the one kept, have it be the one which gets mem2reged. That is, duplicate all the uses (except the escaping call) and let the code handle it. The primary point here is just to keep the naming in the IR more obvious. :)

[lebedev.ri]

In D113520#3164812, @reames wrote:

@lebedev.ri Your suggested approach makes sense to me at a basic level. Forming the alloca is starting to seem more and more like a hack, but we can come back to implementing partial mem2reg as a follow on code improvement.

A couple of suggestions:

• Please leave taint tracking to later patch. Start with the requirement that the non-escaping call use must also be readonly in that argument.

That was my plan, yes.

• Instead of having the cloned alloca be the one kept, have it be the one which gets mem2reged. That is, duplicate all the uses (except the escaping call) and let the code handle it. The primary point here is just to keep the naming in the IR more obvious. :)

Yeah, i was wondering about that, but didn't do that yet.

[lebedev.ri]

Finally managed to subdue crippling fear of the potential complexity
of the alternative design, only to discover it to be rather simple :)

In D113520#3164812, @reames wrote:

@lebedev.ri Your suggested approach makes sense to me at a basic level. Forming the alloca is starting to seem more and more like a hack, but we can come back to implementing partial mem2reg as a follow on code improvement.

A couple of suggestions:

• Please leave taint tracking to later patch. Start with the requirement that the non-escaping call use must also be readonly in that argument.
• Instead of having the cloned alloca be the one kept, have it be the one which gets mem2reged. That is, duplicate all the uses (except the escaping call) and let the code handle it. The primary point here is just to keep the naming in the IR more obvious. :)

Done, PTAL :)

[lebedev.ri]

ping @reames
thanks

[djtodoro]

The idea behind this seems very interesting/valuable, thanks for this!

I am wondering if you are able to get some benchmark data (SPEC, or phoronix) for this? IIUC, the code-size will be increased, so is that size reasonable?

llvm/lib/Transforms/Scalar/SROA.cpp

3688

Should we add a TODO here for clarifying that there is a plan to support calls that do modify the alloca's content?

[lebedev.ri]

Thanks for taking a look!

In D113520#3356615, @djtodoro wrote:

The idea behind this seems very interesting/valuable, thanks for this!

I am wondering if you are able to get some benchmark data (SPEC, or phoronix) for this? IIUC, the code-size will be increased, so is that size reasonable?

I suppose the LICM change, that came after this was initially posted,
has rendered this being less crucial than it would be,
but important nonetheless.

Size-wise, this is actually a win i would say:
https://llvm-compile-time-tracker.com/compare.php?from=a8b4f5bbab62091835205f3d648902432a4a5b58&to=aeae054055b125b011c1122f82c86457e159436f&stat=size-total

[djtodoro]

LGTM for my side, but please wait a few days to see if someone has some additional comments (although it looks like all the comments are already addressed).

llvm/lib/Transforms/Scalar/SROA.cpp

297

I guess ; is unintentional here.

[lebedev.ri]

In D113520#3357128, @djtodoro wrote:

LGTM for my side, but please wait a few days to see if someone has some additional comments (although it looks like all the comments are already addressed).

Thank you!

I'm hoping @reames would find time to re-review this, but given that
we've settled on the approach, and it's rather straight-forward,
i don't know how long i should wait indeed.

[lebedev.ri]

@reames i'm intending to land this in 24 hours if there are no further comments.

[lebedev.ri]

Ok, i'm not yet sure how to deal with this, but somehow we end up trying to re-promote the alloca's here:

; Function Attrs: argmemonly nofree nounwind willreturn
declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg) #0

define fastcc void @TraceLine(i64 %tmp) {
entry:
  %LDir = alloca i64, align 8
  %NewDir1 = alloca i64, align 8
  %LDir.cast = bitcast i64* %LDir to i8*
  %NewDir1.cast = bitcast i64* %NewDir1 to i8*
  store i64 %tmp, i64* %LDir

  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %NewDir1.cast, i8* %LDir.cast, i64 8, i1 false)
  ;%reload = load i64, i64* %LDir
  ;store i64 %reload, i64* %NewDir1

  %call33 = call fastcc double @IntersectObjs(i64* %NewDir1)
  %call38 = call fastcc double @IntersectObjs(i64* %LDir)
  ret void
}

declare fastcc double @IntersectObjs(i64* nocapture readonly)

; uselistorder directives
uselistorder double (i64*)* @IntersectObjs, { 1, 0 }

attributes #0 = { argmemonly nofree nounwind willreturn }

llvm/lib/Transforms/Scalar/SROA.cpp

 STATISTIC(MaxPartitionsPerAlloca, "Maximum number of partitions per alloca");
 STATISTIC(NumAllocaPartitionUses, "Number of alloca partition uses rewritten");
 STATISTIC(MaxUsesPerAllocaPartition, "Maximum number of uses of a partition");
 STATISTIC(NumNewAllocas, "Number of new, smaller allocas introduced");
 STATISTIC(NumPromoted, "Number of allocas promoted to SSA values");
 STATISTIC(NumLoadsSpeculated, "Number of loads speculated to allow promotion");
 STATISTIC(NumDeleted, "Number of instructions deleted");
 STATISTIC(NumVectorized, "Number of vectorized aggregates");
+STATISTIC(NumAllocaSpills, "Number of alloca spills before pointer escapes");
+STATISTIC(NumAllocaReloads,
+          "Number of reloads of alloca after pointer escapes");
 
 /// Hidden option to experiment with completely strict handling of inbounds
 /// GEPs.
 static cl::opt<bool> SROAStrictInbounds("sroa-strict-inbounds", cl::init(false),
                                         cl::Hidden);
 
+/// Controls the beneficiality of promoting of alloca's that escape
+/// normal promotion by spilling/reloading their contents into helper alloca
+/// around the escape points. The thresholds are `OR`'ed, i.e. if the alloca
+/// has non-zero occupancy and
+///   EITHER the alloca size is not greater than SROAEscapingAllocaMaxSize
+///   OR the occupancy percentage is at least SROAEscapingAllocaMinOccupancy
+/// then it is promoted.
+static cl::opt<unsigned>
+    SROAEscapingAllocaMaxSize("sroa-sroa-escaping-alloca-max-size", cl::Hidden,
+                              cl::init(32));
+static cl::opt<unsigned>
+    SROAEscapingAllocaMinOccupancy("sroa-escaping-alloca-min-occupancy",
+                                   cl::Hidden, cl::init(80));
+
 namespace {
 
 /// A custom IRBuilder inserter which prefixes all names, but only in
 /// Assert builds.
 class IRBuilderPrefixedInserter final : public IRBuilderDefaultInserter {
   std::string Prefix;
 
   Twine getNameWithPrefix(const Twine &Name) const {

 /// This class represents the slices of an alloca which are formed by its
 /// various uses. If a pointer escapes, we can't fully build a representation
 /// for the slices used and we reflect that in this structure. The uses are
 /// stored, sorted by increasing beginning offset and with unsplittable slices
 /// starting at a particular offset before splittable slices.
 class llvm::sroa::AllocaSlices {
 public:
   /// Construct the slices of a particular alloca.
-  AllocaSlices(const DataLayout &DL, AllocaInst &AI);
+  AllocaSlices(const DataLayout &DL, AllocaInst &AI, bool &Changed);
 
   /// Test whether a pointer to the allocation escapes our analysis.
   ///
   /// If this is true, the slices are never fully built and should be
   /// ignored.
   bool isEscaped() const { return PointerEscapingInstr; }
 
   /// Support for iterating over the slices.
public:
   ArrayRef<Use *> getDeadUsesIfPromotable() const {
     return DeadUseIfPromotable;
   }
 
   /// Access the dead operands referring to this alloca.
   ///
   /// These are operands which have cannot actually be used to refer to the
   /// alloca as they are outside its range and the user doesn't correct for
   /// that. These mostly consist of PHI node inputs and the like which we just
   /// need to replace with undef.
   ArrayRef<Use *> getDeadOperands() const { return DeadOperands; }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   void print(raw_ostream &OS, const_iterator I, StringRef Indent = "  ") const;
   void printSlice(raw_ostream &OS, const_iterator I,
                   StringRef Indent = "  ") const;
   void printUse(raw_ostream &OS, const_iterator I,
                 StringRef Indent = "  ") const;
   void print(raw_ostream &OS) const;
   void dump(const_iterator I) const;
   void dump() const;
 #endif
 
 private:
   template <typename DerivedT, typename RetT = void> class BuilderBase;
   class SliceBuilder;
 
   friend class AllocaSlices::SliceBuilder;
 
+  Instruction *fixupRewritableEscapes(AllocaInst &AI, bool &Changed);
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   /// Handle to alloca instruction to simplify method interfaces.
   AllocaInst &AI;
 #endif
 
+  /// Certain escaping uses of an alloca (non-capturing-ones)
+  /// do not prevent promotion, but we have to rewrite them
+  /// to make promotion possible. This records all such uses.
+  SmallVector<std::pair<Use *, APInt>> RewritableEscapes;
+
   /// The instruction responsible for this alloca not having a known set
   /// of slices.
   ///
   /// When an instruction (potentially) escapes the pointer to the alloca, we
   /// store a pointer to that here and abort trying to form slices of the
   /// alloca. This will be null if the alloca slices are analyzed successfully.
   Instruction *PointerEscapingInstr;
 
void visitPHINodeOrSelectInst(Instruction &I) {
 
     insertUse(I, Offset, Size);
   }
 
   void visitPHINode(PHINode &PN) { visitPHINodeOrSelectInst(PN); }
 
   void visitSelectInst(SelectInst &SI) { visitPHINodeOrSelectInst(SI); }
 
+  void visitCallBase(CallBase &CB) {
+    if (!IsOffsetKnown || !CB.doesNotCapture(U->getOperandNo()))
+      return PI.setAborted(&CB);
+    // If we know that the callee does not retain the pointer,
+    // then it does not prevent SROA, although we have to workaround this.
+    AS.RewritableEscapes.emplace_back(U, Offset);
+  }
+
   /// Disable SROA entirely if there are unhandled users of the alloca.
   void visitInstruction(Instruction &I) { PI.setAborted(&I); }
 };
 
-AllocaSlices::AllocaSlices(const DataLayout &DL, AllocaInst &AI)
+AllocaSlices::AllocaSlices(const DataLayout &DL, AllocaInst &AI, bool &Changed)
     :
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
       AI(AI),
 #endif
       PointerEscapingInstr(nullptr) {
   SliceBuilder PB(DL, AI, *this);
   SliceBuilder::PtrInfo PtrI = PB.visitPtr(AI);
   if (PtrI.isEscaped() || PtrI.isAborted()) {
     // FIXME: We should sink the escape vs. abort info into the caller nicely,
     // possibly by just storing the PtrInfo in the AllocaSlices.
     PointerEscapingInstr = PtrI.getEscapingInst() ? PtrI.getEscapingInst()
                                                   : PtrI.getAbortingInst();
     assert(PointerEscapingInstr && "Did not track a bad instruction");
     return;
   }
 
+  // We may have found that the pointer to the AI escapes, but isn't captured.
+  if (!RewritableEscapes.empty()) {
+    LLVM_DEBUG(dbgs() << "Alloca is escaped by calls! Original slices:\n");
+    LLVM_DEBUG(print(dbgs()));
+    LLVM_DEBUG(dbgs() << "Escapes:\n"; for (const auto &E
+                                            : RewritableEscapes) dbgs()
+                                       << "  " << *E.first->getUser() << "\n";);
+
+    auto IsProfitableToTransform = [&]() {
+      unsigned AllocatedBytes = *AI.getAllocationSizeInBits(DL) / 8;
+      APInt LoadedBytes(AllocatedBytes, 0);
+      APInt StoredBytes(AllocatedBytes, 0);
+      for (const Slice &S : Slices) {
+        if (S.isDead())
+          continue;
+        auto *I = cast<Instruction>(S.getUse()->getUser());
+        switch (unsigned Opc = I->getOpcode()) {
+        default:
+          continue;
+        case Instruction::Load:
+        case Instruction::Store: {
+          APInt &Map = Opc == Instruction::Load ? LoadedBytes : StoredBytes;
+          Map.setBits(S.beginOffset(), S.endOffset());
+          continue;
+        }
+        }
+      }
+      APInt LoadedAndStoredBytes = LoadedBytes;
+      LoadedAndStoredBytes &= StoredBytes;
+      unsigned UsedBytes = LoadedAndStoredBytes.countPopulation();
+      unsigned OccupancyPct = divideCeil(100 * UsedBytes, AllocatedBytes);
+      LLVM_DEBUG(dbgs() << "Performing profitability check. ");
+      LLVM_DEBUG(dbgs() << "Alloca size: " << AllocatedBytes
+                        << ", used bytes: " << UsedBytes
+                        << ", occupancy: " << OccupancyPct << "%\n");
+      bool IsProfitable =
+          OccupancyPct > 0 && (AllocatedBytes <= SROAEscapingAllocaMaxSize ||
+                               OccupancyPct >= SROAEscapingAllocaMinOccupancy);
+      LLVM_DEBUG(dbgs() << "Rule: occupancy > 0% && (alloca size <= "
+                        << SROAEscapingAllocaMaxSize << " || occupancy >= "
+                        << SROAEscapingAllocaMinOccupancy << "%), deeming it "
+                        << (IsProfitable ? "profitable!" : "NOT profitable.")
+                        << "\n");
+      return IsProfitable;
+    };
+
+    // Are there any slices of an alloca that would benefit from the promotion?
+    // If the alloca is only used by escaping calls, and isn't loaded/stored to,
+    // then there is no point in promoting it.
+    LLVM_DEBUG(dbgs() << "Can rewrite escapes and make alloca promoteable.\n");
+    if (!IsProfitableToTransform()) {
+      // Backtrack, and pretend that we aborted at the first escape.
+      LLVM_DEBUG(dbgs() << "Profitability check failed, will not try to "
+                           "promote due to the escapes.\n");
+      PointerEscapingInstr =
+          cast<Instruction>(RewritableEscapes.front().first->getUser());
+      return;
+    }
+
+    // Rewrite these uses to not affect the promotion of the alloca.
+    Instruction *NewUsesOfAI = fixupRewritableEscapes(AI, Changed);
+    assert(NewUsesOfAI && "Returns non-null.");
+
+    // Reanalyze new uses of an alloca.
+    LLVM_DEBUG(dbgs() << "Reanalyzing new loads/stores.\n");
+    SliceBuilder::PtrInfo PtrI = PB.visitPtr(*NewUsesOfAI);
+    (void)PtrI;
+    assert(!PtrI.isEscaped() && !PtrI.isAborted() &&
+           "Failed to analyze new memory operations?");
+  }
+
   llvm::erase_if(Slices, [](const Slice &S) { return S.isDead(); });
 
   // Sort the uses. This arranges for the offsets to be in ascending order,
   // and the sizes to be in descending order.
   llvm::stable_sort(Slices);
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
private:
   bool visitSelectInst(SelectInst &SI) {
     enqueueUsers(SI);
     return false;
   }
 };
 
 } // end anonymous namespace
 
+Instruction *AllocaSlices::fixupRewritableEscapes(AllocaInst &AI,
+                                                  bool &Changed) {
+  assert(!RewritableEscapes.empty() &&
+         "Should not be called if there is nothing to rewrite.");
+
+  LLVM_DEBUG(dbgs() << "Rewriting escapes to operate on a new helper alloca\n");
+
+  Changed |= true;
+
+  Instruction *OrigAlloca = &AI;
+
+  // A cache of rebased pointers.
+  SmallDenseMap<std::pair<APInt, Type *>, Value *> RebasedPtrsCSE;
+  // A single instruction may consume multiple pointers into alloca,
+  // let's spill only once per instruction.
+  // Around which instructions have we performed spill/reload already?
+  SmallDenseSet<Instruction *> SpillForInstAlreadyPerformed;
+
+  // First, duplicate the alloca. This is fine to do,
+  // since we know that the old alloca should go away.
+  auto *CloneAlloca = cast<AllocaInst>(OrigAlloca->clone());
+  CloneAlloca->setName(OrigAlloca->getName() + ".remat");
+  CloneAlloca->insertAfter(OrigAlloca);
+
+  IRBuilderTy Builder(OrigAlloca->getContext());
+  const DataLayout &DL = AI.getModule()->getDataLayout();
+
+  // In order to simplify our life, let's base all the new uses of an AI,
+  // off of a single new use-def. This will simplify it's reanalysis.
+  OrigAlloca = GetElementPtrInst::CreateInBounds(
+      AI.getAllocatedType(), OrigAlloca, None, AI.getName() + ".new.uses",
+      CloneAlloca);
+
+  // Spill the entire original alloca into our new clone alloca.
+  auto SpillOrigAllocaBefore = [&](Instruction *InsertBefore) {
+    Builder.SetInsertPoint(InsertBefore);
+    Value *SpilledValue = Builder.CreateLoad(AI.getAllocatedType(), OrigAlloca,
+                                             AI.getName() + ".spill");
+    Builder.CreateStore(SpilledValue, CloneAlloca);
+    ++NumAllocaSpills;
+  };
+
+  // Reload the entire original alloca from our new clone alloca.
+  auto ReloadOrigAllocaBefore = [&](Instruction *InsertBefore) {
+    Builder.SetInsertPoint(InsertBefore);
+    Value *ReloadedValue = Builder.CreateLoad(
+        AI.getAllocatedType(), CloneAlloca, AI.getName() + ".reload");
+    Builder.CreateStore(ReloadedValue, OrigAlloca);
+    ++NumAllocaReloads;
+  };
+
+  // Rebase this pointer into orig alloca to be based on clone alloca.
+  auto RebaseOrigAllocaPtr =
+      [this, AllocaBB = OrigAlloca->getParent(), &Builder, &RebasedPtrsCSE,
+       CloneAlloca, DL](const std::pair<Use *, APInt> &EscapingPtrUse) {
+        const APInt &Offset = EscapingPtrUse.second;
+        Use *U = EscapingPtrUse.first;
+        Type *PtrTy = U->get()->getType();
+        Value *NewPtr;
+
+        auto It = RebasedPtrsCSE.find({Offset, PtrTy});
+        if (It != RebasedPtrsCSE.end())
+          NewPtr = It->second;
+        else {
+          BasicBlock::iterator I = CloneAlloca->getIterator();
+          while (isa<AllocaInst>(I)) {
+            ++I;
+            assert(I != AllocaBB->end() && "Block has no insertion point?");
+          }
+          Builder.SetInsertPoint(&*I);
+
+          NewPtr = getAdjustedPtr(Builder, DL, CloneAlloca, Offset, PtrTy, "");
+          RebasedPtrsCSE[{Offset, PtrTy}] = NewPtr;
+        }
+
+        auto *OldPtr = cast<Instruction>(*U);
+        U->set(NewPtr);
+        if (OldPtr->use_empty())
+          DeadUsers.emplace_back(OldPtr);
+      };
+
+  // For each escaping pointer to the orig alloca.
+  for (const std::pair<Use *, APInt> &RewritableEscape : RewritableEscapes) {
+    auto *EscapingUserInst =
+        cast<Instruction>(RewritableEscape.first->getUser());
+
+    // Rewrite this escaping pointer to be clone alloca-based.
+    RebaseOrigAllocaPtr(RewritableEscape);
+
+    // Did we already spill/reload around this instruction?
+    if (SpillForInstAlreadyPerformed.contains(EscapingUserInst))
+      continue;
+    // We did not. Let's do that now.
+
+    // Before said instruction, spill the current state
+    // of the orig alloca into the clone alloca.
+    SpillOrigAllocaBefore(EscapingUserInst);
+
+    // And after the instruction, restore the state of orig alloca.
+    // Note that if the instruction is a terminator,
+    // we have to do that on *each* path.
+    if (!EscapingUserInst->isTerminator())
+      ReloadOrigAllocaBefore(EscapingUserInst->getNextNode());
+    else {
+      for (BasicBlock *SuccBB : successors(EscapingUserInst->getParent())) {
+        BasicBlock::iterator I = SuccBB->getFirstInsertionPt();
+        assert(I != SuccBB->end() && "Successor block has no insertion point?");
+        ReloadOrigAllocaBefore(&*I);
+      }
+    }
+
+    // If we happen to revisit this instruction (perhaps it takes several
+    // pointers into this alloca), don't redo spill/reload.
+    SpillForInstAlreadyPerformed.insert(EscapingUserInst);
+  }
+
+  // Resplit any FCA load/stores we may have introduced.
+  LLVM_DEBUG(
+      dbgs() << "Done rewriting escapes, making new loads/stores analyzable\n");
+  AggLoadStoreRewriter(DL).rewrite(*OrigAlloca);
+
+  return OrigAlloca;
+}
+
 /// Strip aggregate type wrapping.
 ///
 /// This removes no-op aggregate types wrapping an underlying type. It will
 /// strip as many layers of types as it can without changing either the type
 /// size or the allocated size.
 static Type *stripAggregateTypeWrapping(const DataLayout &DL, Type *Ty) {
   if (Ty->isSingleValueType())
     return Ty;
bool SROAPass::runOnAlloca(AllocaInst &AI) {
   bool Changed = false;
 
   // First, split any FCA loads and stores touching this alloca to promote
   // better splitting and promotion opportunities.
   AggLoadStoreRewriter AggRewriter(DL);
   Changed |= AggRewriter.rewrite(AI);
 
   // Build the slices using a recursive instruction-visiting builder.
-  AllocaSlices AS(DL, AI);
+  AllocaSlices AS(DL, AI, Changed);
   LLVM_DEBUG(AS.print(dbgs()));
   if (AS.isEscaped())
     return Changed;
 
   // Delete all the dead users of this alloca before splitting and rewriting it.
   for (Instruction *DeadUser : AS.getDeadUsers()) {
     // Free up everything used by this instruction.
     for (Use &DeadOp : DeadUser->operands())

llvm/test/Transforms/PhaseOrdering/lifetime-sanitizer.ll

 ; RUN: opt < %s -O0 -S | FileCheck %s
 ; RUN: opt < %s -O1 -S | FileCheck %s
 ; RUN: opt < %s -O2 -S | FileCheck %s
 ; RUN: opt < %s -O3 -S | FileCheck %s
 ; RUN: opt < %s -passes='default<O0>' -S | FileCheck %s
 ; RUN: opt < %s -passes='default<O1>' -S | FileCheck %s
 ; RUN: opt < %s -passes='default<O2>' -S | FileCheck %s
 ; RUN: opt < %s -passes='default<O3>' -S | FileCheck %s
 
 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture)
 declare void @llvm.lifetime.end.p0i8(i64, i8* nocapture)
-declare void @foo(i8* nocapture)
+declare void @foo(i8*)
 
 define void @asan() sanitize_address {
 entry:
   ; CHECK-LABEL: @asan(
   %text = alloca i8, align 1
 
   call void @llvm.lifetime.start.p0i8(i64 1, i8* %text)
   call void @llvm.lifetime.end.p0i8(i64 1, i8* %text)

llvm/test/Transforms/SROA/non-capturing-call.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=sroa -S | FileCheck %s --check-prefix=CHECK
 ; RUN: opt < %s -passes=sroa -opaque-pointers -S | FileCheck %s --check-prefix=CHECK-OPAQUE
 
 define i32 @alloca_used_in_call(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @alloca_used_in_call(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL]])
-; CHECK-NEXT:    [[I1:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    ret i32 [[I1]]
+; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL_REMAT]])
+; CHECK-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 ; CHECK-OPAQUE-LABEL: @alloca_used_in_call(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL]])
-; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    ret i32 [[I1]]
+; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL_REMAT]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 entry:
   %retval = alloca i32, align 4
   store i32 0, i32* %retval, align 4
   br label %loop
 
 loop:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %loop ]
exit:
   %i0 = call i32 @capture_of_alloca(i32* nonnull %retval)
   %i1 = load i32, i32* %retval, align 4
   ret i32 %i1
 }
 
 define i32 @alloca_with_gep_used_in_call(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @alloca_with_gep_used_in_call(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr i32, i32* [[RETVAL]], i32 0
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[GEP]])
-; CHECK-NEXT:    [[I1:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    ret i32 [[I1]]
+; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL_REMAT]])
+; CHECK-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 ; CHECK-OPAQUE-LABEL: @alloca_with_gep_used_in_call(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[GEP:%.*]] = getelementptr i32, ptr [[RETVAL]], i32 0
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[GEP]])
-; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    ret i32 [[I1]]
+; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL_REMAT]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 entry:
   %retval = alloca i32, align 4
   store i32 0, i32* %retval, align 4
   br label %loop
 
 loop:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %loop ]
exit:
   %i0 = call i32 @capture_with_multiple_args(i32* nocapture nonnull %retval, i32* nonnull %retval)
   %i1 = load i32, i32* %retval, align 4
   ret i32 %i1
 }
 
 define i32 @alloca_used_in_maybe_throwing_call(i32* nocapture nonnull readonly %data, i64 %n) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*)  {
 ; CHECK-LABEL: @alloca_used_in_maybe_throwing_call(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL]])
+; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL_REMAT]])
 ; CHECK-NEXT:    to label [[CONT:%.*]] unwind label [[UW:%.*]]
 ; CHECK:       cont:
+; CHECK-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
 ; CHECK-NEXT:    br label [[END:%.*]]
 ; CHECK:       uw:
 ; CHECK-NEXT:    [[I1:%.*]] = landingpad { i8*, i32 }
 ; CHECK-NEXT:    catch i8* null
+; CHECK-NEXT:    [[RETVAL_RELOAD1:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
 ; CHECK-NEXT:    br label [[END]]
 ; CHECK:       end:
-; CHECK-NEXT:    [[I2:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    ret i32 [[I2]]
+; CHECK-NEXT:    [[RETVAL_1:%.*]] = phi i32 [ [[RETVAL_RELOAD]], [[CONT]] ], [ [[RETVAL_RELOAD1]], [[UW]] ]
+; CHECK-NEXT:    ret i32 [[RETVAL_1]]
 ;
 ; CHECK-OPAQUE-LABEL: @alloca_used_in_maybe_throwing_call(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL]])
+; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL_REMAT]])
 ; CHECK-OPAQUE-NEXT:    to label [[CONT:%.*]] unwind label [[UW:%.*]]
 ; CHECK-OPAQUE:       cont:
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
 ; CHECK-OPAQUE-NEXT:    br label [[END:%.*]]
 ; CHECK-OPAQUE:       uw:
 ; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = landingpad { ptr, i32 }
 ; CHECK-OPAQUE-NEXT:    catch ptr null
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD1:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
 ; CHECK-OPAQUE-NEXT:    br label [[END]]
 ; CHECK-OPAQUE:       end:
-; CHECK-OPAQUE-NEXT:    [[I2:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    ret i32 [[I2]]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_1:%.*]] = phi i32 [ [[RETVAL_RELOAD]], [[CONT]] ], [ [[RETVAL_RELOAD1]], [[UW]] ]
+; CHECK-OPAQUE-NEXT:    ret i32 [[RETVAL_1]]
 ;
 entry:
   %retval = alloca i32, align 4
   store i32 0, i32* %retval, align 4
   br label %loop
 
 loop:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %loop ]

 end:
   %i2 = load i32, i32* %retval, align 4
   ret i32 %i2
 }
 
 define i32 @alloca_used_in_maybe_throwing_call_with_same_dests(i32* nocapture nonnull readonly %data, i64 %n) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*)  {
 ; CHECK-LABEL: @alloca_used_in_maybe_throwing_call_with_same_dests(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL]])
+; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL_REMAT]])
 ; CHECK-NEXT:    to label [[END:%.*]] unwind label [[UW:%.*]]
 ; CHECK:       uw:
 ; CHECK-NEXT:    [[I1:%.*]] = landingpad { i8*, i32 }
 ; CHECK-NEXT:    catch i8* null
+; CHECK-NEXT:    [[RETVAL_RELOAD1:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
 ; CHECK-NEXT:    br label [[END]]
 ; CHECK:       end:
-; CHECK-NEXT:    [[I2:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    ret i32 [[I2]]
+; CHECK-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, i32* [[RETVAL_REMAT]], align 4
+; CHECK-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 ; CHECK-OPAQUE-LABEL: @alloca_used_in_maybe_throwing_call_with_same_dests(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_REMAT:%.*]] = alloca i32, align 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL]])
+; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = invoke i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL_REMAT]])
 ; CHECK-OPAQUE-NEXT:    to label [[END:%.*]] unwind label [[UW:%.*]]
 ; CHECK-OPAQUE:       uw:
 ; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = landingpad { ptr, i32 }
 ; CHECK-OPAQUE-NEXT:    catch ptr null
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD1:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
 ; CHECK-OPAQUE-NEXT:    br label [[END]]
 ; CHECK-OPAQUE:       end:
-; CHECK-OPAQUE-NEXT:    [[I2:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    ret i32 [[I2]]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_RELOAD:%.*]] = load i32, ptr [[RETVAL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    ret i32 [[RETVAL_RELOAD]]
 ;
 entry:
   %retval = alloca i32, align 4
   store i32 0, i32* %retval, align 4
   br label %loop
 
 loop:
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %loop ]

 end:
   %i2 = load i32, i32* %retval, align 4
   ret i32 %i2
 }
 
 define [2 x i32] @part_of_alloca_used_in_call(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @part_of_alloca_used_in_call(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_0_GEP]], align 4
-; CHECK-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_1_GEP]], align 4
-; CHECK-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 1
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[RETVAL]])
-; CHECK-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, i32* [[I1_FCA_0_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, i32* [[I1_FCA_1_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(i32* nocapture nonnull [[SROA_IDX]])
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 ; CHECK-OPAQUE-LABEL: @part_of_alloca_used_in_call(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-OPAQUE-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[RETVAL_FULL_REMAT]], i64 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[RETVAL]])
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, ptr [[I1_FCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, ptr [[I1_FCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-OPAQUE-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca(ptr nocapture nonnull [[SROA_IDX]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-OPAQUE-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 entry:
   %retval.full = alloca [2 x i32], align 4
   store [2 x i32] zeroinitializer, [2 x i32]* %retval.full, align 4
   %retval = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 1
   br label %loop
 
exit:
   %i0 = call i32 @user_of_alloca(i32* nocapture nonnull %retval)
   %i1 = load [2 x i32], [2 x i32]* %retval.full, align 4
   ret [2 x i32] %i1
 }
 
 define [2 x i32] @all_parts_of_alloca_used_in_call_with_multiple_args(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @all_parts_of_alloca_used_in_call_with_multiple_args(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_0_GEP]], align 4
-; CHECK-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_1_GEP]], align 4
-; CHECK-NEXT:    [[RETVAL_BASE:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 0
-; CHECK-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-NEXT:    [[SROA_IDX1:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 1
+; CHECK-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 0
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[RETVAL]], i32* nocapture nonnull [[RETVAL_BASE]])
-; CHECK-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, i32* [[I1_FCA_0_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, i32* [[I1_FCA_1_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[SROA_IDX1]], i32* nocapture nonnull [[SROA_IDX]])
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 ; CHECK-OPAQUE-LABEL: @all_parts_of_alloca_used_in_call_with_multiple_args(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[RETVAL_BASE:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 0
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-OPAQUE-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[RETVAL_FULL_REMAT]], i64 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[RETVAL]], ptr nocapture nonnull [[RETVAL_BASE]])
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, ptr [[I1_FCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, ptr [[I1_FCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-OPAQUE-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[SROA_IDX]], ptr nocapture nonnull [[RETVAL_FULL_REMAT]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-OPAQUE-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 entry:
   %retval.full = alloca [2 x i32], align 4
   store [2 x i32] zeroinitializer, [2 x i32]* %retval.full, align 4
   %retval.base = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 0
   %retval = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 1
   br label %loop
exit:
   %i0 = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull %retval, i32* nocapture nonnull %retval.base)
   %i1 = load [2 x i32], [2 x i32]* %retval.full, align 4
   ret [2 x i32] %i1
 }
 
 define [2 x i32] @part_of_alloca_used_in_call_with_multiple_args(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @part_of_alloca_used_in_call_with_multiple_args(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_0_GEP]], align 4
-; CHECK-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_1_GEP]], align 4
-; CHECK-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 1
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[RETVAL]], i32* nocapture nonnull [[RETVAL]])
-; CHECK-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, i32* [[I1_FCA_0_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, i32* [[I1_FCA_1_GEP]], align 4
-; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[SROA_IDX]], i32* nocapture nonnull [[SROA_IDX]])
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 ; CHECK-OPAQUE-LABEL: @part_of_alloca_used_in_call_with_multiple_args(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
+; CHECK-OPAQUE-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[RETVAL_FULL_REMAT]], i64 4
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SROA_4_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_4_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[RETVAL]], ptr nocapture nonnull [[RETVAL]])
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_LOAD:%.*]] = load i32, ptr [[I1_FCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I1_FCA_0_LOAD]], 0
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_LOAD:%.*]] = load i32, ptr [[I1_FCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[I1_FCA_1_LOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-OPAQUE-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[SROA_IDX]], ptr nocapture nonnull [[SROA_IDX]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-OPAQUE-NEXT:    [[I1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
 ; CHECK-OPAQUE-NEXT:    ret [2 x i32] [[I1_FCA_1_INSERT]]
 ;
 entry:
   %retval.full = alloca [2 x i32], align 4
   store [2 x i32] zeroinitializer, [2 x i32]* %retval.full, align 4
   %retval = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 1
   br label %loop
 
exit:
   %i0 = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull %retval, i32* nocapture nonnull %retval)
   %i1 = load [2 x i32], [2 x i32]* %retval.full, align 4
   ret [2 x i32] %i1
 }
 
 define [2 x i32] @all_parts_of_alloca_used_in_calls_with_multiple_args(i32* nocapture nonnull readonly %data, i64 %n) {
 ; CHECK-LABEL: @all_parts_of_alloca_used_in_calls_with_multiple_args(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
 ; CHECK-NEXT:    [[SOME_ANOTHER_ALLOCA_FULL:%.*]] = alloca [42 x i32], align 4
-; CHECK-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_0_GEP]], align 4
-; CHECK-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    store i32 0, i32* [[DOTFCA_1_GEP]], align 4
-; CHECK-NEXT:    [[RETVAL_BASE:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 0
-; CHECK-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-NEXT:    [[SROA_IDX3:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 1
+; CHECK-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], i64 0, i64 0
 ; CHECK-NEXT:    [[SOME_ANOTHER_ALLOCA:%.*]] = getelementptr inbounds [42 x i32], [42 x i32]* [[SOME_ANOTHER_ALLOCA_FULL]], i64 0, i64 0
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[RETVAL_FULL_SROA_6_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[RDX:%.*]] = load i32, i32* [[RETVAL]], align 4
-; CHECK-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-NEXT:    store i32 [[RDX_INC]], i32* [[RETVAL]], align 4
+; CHECK-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_6_0]], [[LD]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[RETVAL]], i32* nocapture nonnull [[RETVAL_BASE]])
-; CHECK-NEXT:    [[I1:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[RETVAL_BASE]], i32* nocapture nonnull [[RETVAL]])
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[SROA_IDX3]], i32* nocapture nonnull [[SROA_IDX]])
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_SPILL1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
+; CHECK-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL1_FCA_1_INSERT]], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[I1:%.*]] = call i32 @user_of_alloca_with_multiple_args(i32* nocapture nonnull [[SROA_IDX]], i32* nocapture nonnull [[SROA_IDX3]])
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD2:%.*]] = load [2 x i32], [2 x i32]* [[RETVAL_FULL_REMAT]], align 4
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD2_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD2]], 0
+; CHECK-NEXT:    [[RETVAL_FULL_RELOAD2_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD2]], 1
 ; CHECK-NEXT:    [[I2:%.*]] = call i32 @capture_of_alloca(i32* [[SOME_ANOTHER_ALLOCA]])
-; CHECK-NEXT:    [[I3_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-NEXT:    [[I3_FCA_0_LOAD:%.*]] = load i32, i32* [[I3_FCA_0_GEP]], align 4
-; CHECK-NEXT:    [[I3_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I3_FCA_0_LOAD]], 0
-; CHECK-NEXT:    [[I3_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], [2 x i32]* [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-NEXT:    [[I3_FCA_1_LOAD:%.*]] = load i32, i32* [[I3_FCA_1_GEP]], align 4
-; CHECK-NEXT:    [[I3_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I3_FCA_0_INSERT]], i32 [[I3_FCA_1_LOAD]], 1
+; CHECK-NEXT:    [[I3_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD2_FCA_0_EXTRACT]], 0
+; CHECK-NEXT:    [[I3_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I3_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD2_FCA_1_EXTRACT]], 1
 ; CHECK-NEXT:    ret [2 x i32] [[I3_FCA_1_INSERT]]
 ;
 ; CHECK-OPAQUE-LABEL: @all_parts_of_alloca_used_in_calls_with_multiple_args(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL:%.*]] = alloca [2 x i32], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_REMAT:%.*]] = alloca [2 x i32], align 4
 ; CHECK-OPAQUE-NEXT:    [[SOME_ANOTHER_ALLOCA_FULL:%.*]] = alloca [42 x i32], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[DOTFCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    store i32 0, ptr [[DOTFCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[RETVAL_BASE:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 0
-; CHECK-OPAQUE-NEXT:    [[RETVAL:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i64 0, i64 1
+; CHECK-OPAQUE-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[RETVAL_FULL_REMAT]], i64 4
 ; CHECK-OPAQUE-NEXT:    [[SOME_ANOTHER_ALLOCA:%.*]] = getelementptr inbounds [42 x i32], ptr [[SOME_ANOTHER_ALLOCA_FULL]], i64 0, i64 0
 ; CHECK-OPAQUE-NEXT:    br label [[LOOP:%.*]]
 ; CHECK-OPAQUE:       loop:
-; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SROA_6_0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[RDX_INC:%.*]], [[LOOP]] ]
+; CHECK-OPAQUE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-OPAQUE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV]]
 ; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX:%.*]] = load i32, ptr [[RETVAL]], align 4
-; CHECK-OPAQUE-NEXT:    [[RDX_INC:%.*]] = add nsw i32 [[RDX]], [[LD]]
-; CHECK-OPAQUE-NEXT:    store i32 [[RDX_INC]], ptr [[RETVAL]], align 4
+; CHECK-OPAQUE-NEXT:    [[RDX_INC]] = add nsw i32 [[RETVAL_FULL_SROA_6_0]], [[LD]]
 ; CHECK-OPAQUE-NEXT:    [[INDVARS_IV_NEXT]] = add nsw i64 [[INDVARS_IV]], 1
 ; CHECK-OPAQUE-NEXT:    [[EXITCOND:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT]], [[N:%.*]]
 ; CHECK-OPAQUE-NEXT:    br i1 [[EXITCOND]], label [[LOOP]], label [[EXIT:%.*]]
 ; CHECK-OPAQUE:       exit:
-; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[RETVAL]], ptr nocapture nonnull [[RETVAL_BASE]])
-; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[RETVAL_BASE]], ptr nocapture nonnull [[RETVAL]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 0, 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL_FCA_0_INSERT]], i32 [[RDX_INC]], 1
+; CHECK-OPAQUE-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL_FCA_1_INSERT]], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I0:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[SROA_IDX]], ptr nocapture nonnull [[RETVAL_FULL_REMAT]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD:%.*]] = load [2 x i32], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL1_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD_FCA_0_EXTRACT]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_SPILL1_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[RETVAL_FULL_SPILL1_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD_FCA_1_EXTRACT]], 1
+; CHECK-OPAQUE-NEXT:    store [2 x i32] [[RETVAL_FULL_SPILL1_FCA_1_INSERT]], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[I1:%.*]] = call i32 @user_of_alloca_with_multiple_args(ptr nocapture nonnull [[RETVAL_FULL_REMAT]], ptr nocapture nonnull [[SROA_IDX]])
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD2:%.*]] = load [2 x i32], ptr [[RETVAL_FULL_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD2_FCA_0_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD2]], 0
+; CHECK-OPAQUE-NEXT:    [[RETVAL_FULL_RELOAD2_FCA_1_EXTRACT:%.*]] = extractvalue [2 x i32] [[RETVAL_FULL_RELOAD2]], 1
 ; CHECK-OPAQUE-NEXT:    [[I2:%.*]] = call i32 @capture_of_alloca(ptr [[SOME_ANOTHER_ALLOCA]])
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_0_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 0
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_0_LOAD:%.*]] = load i32, ptr [[I3_FCA_0_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[I3_FCA_0_LOAD]], 0
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_1_GEP:%.*]] = getelementptr inbounds [2 x i32], ptr [[RETVAL_FULL]], i32 0, i32 1
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_1_LOAD:%.*]] = load i32, ptr [[I3_FCA_1_GEP]], align 4
-; CHECK-OPAQUE-NEXT:    [[I3_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I3_FCA_0_INSERT]], i32 [[I3_FCA_1_LOAD]], 1
+; CHECK-OPAQUE-NEXT:    [[I3_FCA_0_INSERT:%.*]] = insertvalue [2 x i32] undef, i32 [[RETVAL_FULL_RELOAD2_FCA_0_EXTRACT]], 0
+; CHECK-OPAQUE-NEXT:    [[I3_FCA_1_INSERT:%.*]] = insertvalue [2 x i32] [[I3_FCA_0_INSERT]], i32 [[RETVAL_FULL_RELOAD2_FCA_1_EXTRACT]], 1
 ; CHECK-OPAQUE-NEXT:    ret [2 x i32] [[I3_FCA_1_INSERT]]
 ;
 entry:
   %retval.full = alloca [2 x i32], align 4
   %some.another.alloca.full = alloca [42 x i32], align 4
   store [2 x i32] zeroinitializer, [2 x i32]* %retval.full, align 4
   %retval.base = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 0
   %retval = getelementptr inbounds [2 x i32], [2 x i32]* %retval.full, i64 0, i64 1
entry:
   ret i32 0
 }
 
 declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture)
 
 define i64 @do_schedule_instrs_for_dce_after_fixups() {
 ; CHECK-LABEL: @do_schedule_instrs_for_dce_after_fixups(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[C:%.*]] = alloca i64, align 2
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i64* [[C]] to i8*
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 1, i8* [[TMP0]])
-; CHECK-NEXT:    store i64 0, i64* [[C]], align 4
-; CHECK-NEXT:    [[ARRAYDECAY:%.*]] = bitcast i64* [[C]] to i32*
+; CHECK-NEXT:    [[C_REMAT:%.*]] = alloca i64, align 2
+; CHECK-NEXT:    [[SROA_RAW_CAST:%.*]] = bitcast i64* [[C_REMAT]] to i8*
+; CHECK-NEXT:    [[SROA_RAW_IDX:%.*]] = getelementptr inbounds i8, i8* [[SROA_RAW_CAST]], i64 4
+; CHECK-NEXT:    [[SROA_CAST:%.*]] = bitcast i8* [[SROA_RAW_IDX]] to i32*
 ; CHECK-NEXT:    br label [[IF_END:%.*]]
 ; CHECK:       if.end:
-; CHECK-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds i32, i32* [[ARRAYDECAY]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @user_of_alloca(i32* [[ADD_PTR]])
-; CHECK-NEXT:    [[LD:%.*]] = load i64, i64* [[C]], align 4
-; CHECK-NEXT:    ret i64 [[LD]]
+; CHECK-NEXT:    store i64 0, i64* [[C_REMAT]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = call i32 @user_of_alloca(i32* [[SROA_CAST]])
+; CHECK-NEXT:    [[C_RELOAD:%.*]] = load i64, i64* [[C_REMAT]], align 4
+; CHECK-NEXT:    ret i64 [[C_RELOAD]]
 ;
 ; CHECK-OPAQUE-LABEL: @do_schedule_instrs_for_dce_after_fixups(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[C:%.*]] = alloca i64, align 2
-; CHECK-OPAQUE-NEXT:    [[TMP0:%.*]] = bitcast ptr [[C]] to ptr
-; CHECK-OPAQUE-NEXT:    call void @llvm.lifetime.start.p0(i64 1, ptr [[TMP0]])
-; CHECK-OPAQUE-NEXT:    store i64 0, ptr [[C]], align 4
-; CHECK-OPAQUE-NEXT:    [[ARRAYDECAY:%.*]] = bitcast ptr [[C]] to ptr
+; CHECK-OPAQUE-NEXT:    [[C_REMAT:%.*]] = alloca i64, align 2
+; CHECK-OPAQUE-NEXT:    [[SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[C_REMAT]], i64 4
 ; CHECK-OPAQUE-NEXT:    br label [[IF_END:%.*]]
 ; CHECK-OPAQUE:       if.end:
-; CHECK-OPAQUE-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds i32, ptr [[ARRAYDECAY]], i64 1
-; CHECK-OPAQUE-NEXT:    [[TMP1:%.*]] = call i32 @user_of_alloca(ptr [[ADD_PTR]])
-; CHECK-OPAQUE-NEXT:    [[LD:%.*]] = load i64, ptr [[C]], align 4
-; CHECK-OPAQUE-NEXT:    ret i64 [[LD]]
+; CHECK-OPAQUE-NEXT:    store i64 0, ptr [[C_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    [[TMP0:%.*]] = call i32 @user_of_alloca(ptr [[SROA_IDX]])
+; CHECK-OPAQUE-NEXT:    [[C_RELOAD:%.*]] = load i64, ptr [[C_REMAT]], align 4
+; CHECK-OPAQUE-NEXT:    ret i64 [[C_RELOAD]]
 ;
 entry:
   %c = alloca i64, align 2
   %0 = bitcast i64* %c to i8*
   call void @llvm.lifetime.start.p0i8(i64 1, i8* %0)
   store i64 0, i64* %c
   %arraydecay = bitcast i64* %c to i32*
   br label %if.end
entry:
   %a = alloca i8
   call void @byte_user_of_alloca(i8* %a)
   %r = load i8, i8* %a
   ret i8 %r
 }
 define i8 @transform_load_and_store() {
 ; CHECK-LABEL: @transform_load_and_store(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[A:%.*]] = alloca i8, align 1
-; CHECK-NEXT:    store i8 0, i8* [[A]], align 1
-; CHECK-NEXT:    call void @byte_user_of_alloca(i8* [[A]])
-; CHECK-NEXT:    [[R:%.*]] = load i8, i8* [[A]], align 1
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    [[A_REMAT:%.*]] = alloca i8, align 1
+; CHECK-NEXT:    store i8 0, i8* [[A_REMAT]], align 1
+; CHECK-NEXT:    call void @byte_user_of_alloca(i8* [[A_REMAT]])
+; CHECK-NEXT:    [[A_RELOAD:%.*]] = load i8, i8* [[A_REMAT]], align 1
+; CHECK-NEXT:    ret i8 [[A_RELOAD]]
 ;
 ; CHECK-OPAQUE-LABEL: @transform_load_and_store(
 ; CHECK-OPAQUE-NEXT:  entry:
-; CHECK-OPAQUE-NEXT:    [[A:%.*]] = alloca i8, align 1
-; CHECK-OPAQUE-NEXT:    store i8 0, ptr [[A]], align 1
-; CHECK-OPAQUE-NEXT:    call void @byte_user_of_alloca(ptr [[A]])
-; CHECK-OPAQUE-NEXT:    [[R:%.*]] = load i8, ptr [[A]], align 1
-; CHECK-OPAQUE-NEXT:    ret i8 [[R]]
+; CHECK-OPAQUE-NEXT:    [[A_REMAT:%.*]] = alloca i8, align 1
+; CHECK-OPAQUE-NEXT:    store i8 0, ptr [[A_REMAT]], align 1
+; CHECK-OPAQUE-NEXT:    call void @byte_user_of_alloca(ptr [[A_REMAT]])
+; CHECK-OPAQUE-NEXT:    [[A_RELOAD:%.*]] = load i8, ptr [[A_REMAT]], align 1
+; CHECK-OPAQUE-NEXT:    ret i8 [[A_RELOAD]]
 ;
 entry:
   %a = alloca i8
   store i8 0, i8* %a
   call void @byte_user_of_alloca(i8* %a)
   %r = load i8, i8* %a
   ret i8 %r
 }