[SROA] For non-speculatable `load`s of `select`s -- split block, insert then/else blocks, form two-entry PHI node

[lebedev.ri]

Summary

Currently, SROA is CFG-preserving.
Not doing so does not affect any pipeline test. (???)
Internally, SROA requires Dominator Tree, and uses it solely for the final -mem2reg call.

By design, we can't really SROA alloca if their address escapes somehow,
but we have logic to deal with load of select/PHI,
where at least one of the possible addresses prevents promotion,
by speculating the loads and selecting between loaded values.

As one would expect, that requires ensuring that the speculation is actually legal.
Even ignoring complexity bailouts, that logic does not deal with everything,
e.g. isSafeToLoadUnconditionally() does not recurse into hands of select.
There can also be cases where the load is genuinely non-speculate.

So if we can't prove that the load can be speculated,
unfold the select, produce two-entry phi node, and perform predicated load.

Now, that transformation must obviously update Dominator Tree,
since we require it later on. Doing so is trivial.
Additionally, we don't want to do this for the final SROA invocation (D136806).

In the end, this ends up having negative (!) compile-time cost:
https://llvm-compile-time-tracker.com/compare.php?from=c6d7e80ec4c17a415673b1cfd25924f98ac83608&to=ddf9600365093ea50d7e278696cbfa01641c959d&stat=instructions:u

Though indeed, this only deals with selects, PHIs are still using speculation.

Should we update some more analysis?

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

Are there any cases where this should still try to speculate?

llvm/lib/Transforms/Scalar/SROA.cpp

1384

Single quote newline

[lebedev.ri]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

I'm not sure i understand the question.
As it can be seen from the rest of the diff here,
for select's, we will always form diamond control flow and two-entry phi node,
and never check if we can speculate load.
For PHI's, we still speculate.

[lebedev.ri]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

One improvement over this we can do, is to cache
all of the pointers derived from the current alloca,
and speculate the loads that belong to that set.
That might be cheap to do. Is that what you had in mind?

We also should probably report CFG as preserved unless we actually changed it.

[lebedev.ri]

I'm guessing we should keep the old approach at least for the new last SROA run.
Not quite sure about PHI handling yet.

[lebedev.ri]

Okay, only modifying CFG if we must is even better.
Please take a look :)

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

66

Move to last field?

llvm/lib/Passes/PassBuilder.cpp

840 ↗

(On Diff #479132)

I thought pass options were-spelled-like-this?

llvm/lib/Transforms/Scalar/SROA.cpp

1402–1403

I know it was already like this, but can't you just do CreateAlignedLoad to start?

1470

Missing check if CFG modify is allowed? Could also move this out to where this is called and check the return value (which it looks like is already there)

4790–4792

Why not just OS << (PreserveCFG ? "<PreserveCFG>" : "<ModifyCFG>";

llvm/test/Transforms/SROA/2009-02-20-InstCombine-SROA.ll

274–276 ↗

(On Diff #479132)

Just remove the prefixes?

[lebedev.ri]

@arsenm thank you for taking a look!
I believe, this addresses your comments.

llvm/include/llvm/Transforms/Scalar/SROA.h

66

I'm guessing you meant "to the end of this var decl group?"

llvm/lib/Passes/PassBuilder.cpp

840 ↗

(On Diff #479132)

Meh. Okay.

llvm/lib/Transforms/Scalar/SROA.cpp

1470

No, nothing is missing, this is just defensive coding.

If we can't modify cfg, then we really won't get here,
and this just confirms that invariant.

E.g. we won't pass DTU into call to rewriteSelectInstLoads()
if we are not allowed to perform CFG changes.

And earlier, see if (PreserveCFG) return {}; bailouts in isSafeSelectToSpeculate().

llvm/test/Transforms/SROA/2009-02-20-InstCombine-SROA.ll

274–276 ↗

(On Diff #479132)

Unless you really insist, i'd like to keep this as is.
If i remove the prefixes,

1. the tests become unconsistent. because currently they all have the same run lines and check prefixes
2. Should the run lines start producing different outputs, without this, the output will get silently lost, and requires manual changes to unbreak

The latter is really painful in e.g. codegen tests.
I would really really prefer to avoid creating that problem.

[lebedev.ri]

Add a few more tests (non-simple loads, select-of-select).
mem2reg bails on volatile (but not atomic!) loads,
so we shouldn't form predicated loads for them.

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

113

Use opaque pointer example?

llvm/lib/Passes/PassRegistry.def

481 ↗

(On Diff #479727)

outdated option names?

[lebedev.ri]

@arsenm thank you for taking a look!
Addressing comments.

[nikic]

llvm/include/llvm/Transforms/Scalar/SROA.h

22

nit: Wrong include style

llvm/lib/Passes/PassBuilder.cpp

849 ↗

(On Diff #480182)

Do I understand correctly that this will prevent doing a plain -passes=sroa run? If so, please pick one of these as a default -- I don't really care which one, but people who just want to clean up some IR with SROA certainly don't need to know about these details. This would also violate existing practice for all other passes.

[lebedev.ri]

@arsenm @nikic thank you for the review!

[uabelho]

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

[lebedev.ri]

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

[uabelho]

In D138238#3991889, @lebedev.ri wrote:

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

Yeah, llvm isn't clean of gcc warnings.
We compile with gcc downstream and used -Werror at first but had to give up that since there are no public bots running with it.
Now we just notice when new warnings pop up instead.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

Our out-of-tree backend is asserting in this function call (particularly at assert(cast<PointerType>(Ptr->getType())->isOpaqueOrPointeeTypeMatches(Ty)) which occurs while creating the load). I suspect this is due to the lost bitcasts, since when I add them back there's no assert.

The code I have adding the bitcasts back looks like the following:

IRB.SetInsertPoint(&LI);

// OUT OF TREE
if (SI.hasOneUse()) {
  if (BitCastInst *BC = dyn_cast<BitCastInst>(SI.user_back())) {
    // assert(BC->user_back() == &LI); <- unclear if this is guaranteed
    TV = IRB.CreateBitCast(TV, BC->getType(), TV->getName() + ".sroa.cast");
    FV = IRB.CreateBitCast(FV, BC->getType(), FV->getName() + ".sroa.cast");
  }
}
// END OUT OF TREE

LoadInst *TL = ...

To be honest, I have not entirely analyzed this patch, so I'm not confident that the way I have added the bitcasts back is legal (especially now that there's no while loop on SI's uses...). Could you advise if this is legal/correct (to unblock us)? I can try to get an anonymized test case for this, so a full fix can be made.

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

We are literally days away from stopping accepting such fixes.
That being said, let me fix this one...

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

9f27f4536e19e93349b0662338408efe6d1cb2fd, but it is probably one of last ones of it's kind.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

Thanks for the quick turn around

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

Sorry, that was too quick and it got reverted.

@nikic please can you estimate when we switch to not dealing with typed pointer bugs?

[nikic]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

Here's my proposal for a hard timeline: https://reviews.llvm.org/D140487 (tl;dr supported until release/16.x branch creation on Jan 24th).

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

2cb393590ea537b06fa66e6d847a1159c227a313, looks like it stuck this time around.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1406

Haha no worries. Glad it's finally landed 👍

[brunodf]

Hi,

In D138238#3991889, @lebedev.ri wrote:

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

This warning is still present in current builds using GCC, also using recent GCC versions, see this buildbot:

/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/include/llvm/Transforms/Scalar/SROA.h:96:7: warning: ‘llvm::SROAPass’ declared with greater visibility than the type of its field ‘llvm::SROAPass::SelectsToRewrite’ [-Wattributes]

The warning does seem to point at a genuine problem. The intent of the sroa namespace is to hide details from clients (possibly in other modules), see here.

/// A private "module" namespace for types and utilities used by SROA. These
/// are implementation details and should not be used by clients.
namespace LLVM_LIBRARY_VISIBILITY sroa {

But because the SROAPass has the field SelectsToRewrite, which contains sroa::PossiblySpeculatableLoads values, it is required to know these details to calculate its size, determine its layout, etc.

Would it be an option to put PossiblySpeculatableLoads (and its constituents) in another namespace (for example sroa_detail), with normal visibility? Or can we store the SelectsToRewrite as a unique_ptr rather than as the map vector directly (pimpl pattern)? The latter would allow to remove the details of PossiblySpeculatableLoads (and its constituents) from the SROA.h header, but would require to follow an extra indirection to access SelectsToRewrite.

Thanks!

[nikic]

@brunodf Ideally, we'd move the SROA implementation into a separate class in the cpp file and only export a thin NewPM wrapper around it in the header.

[brunodf]

In D138238#4656717, @nikic wrote:

@brunodf Ideally, we'd move the SROA implementation into a separate class in the cpp file and only export a thin NewPM wrapper around it in the header.

This is a pull request implementing the reorganization suggested by @nikic : https://github.com/llvm/llvm-project/pull/72846
It solves the GCC warning obviously, but it is also a better organization, it can do away with the llvm::sroa namespace entirely, etc.
If you can, please review it.

llvm/include/llvm/Transforms/Scalar/SROA.h

 
 #ifndef LLVM_TRANSFORMS_SCALAR_SROA_H
 #define LLVM_TRANSFORMS_SCALAR_SROA_H
 
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/ValueHandle.h"
 #include <vector>
 
 namespace llvm {
 
 class AllocaInst;
 class AssumptionCache;
 class DominatorTree;
+class DomTreeUpdater;
 class Function;
 class LLVMContext;
 class PHINode;
 class SelectInst;
 class Use;
 
 /// A private "module" namespace for types and utilities used by SROA. These
 /// are implementation details and should not be used by clients.

 ///    promotion. This can be replacing a memset with a scalar store of an
 ///    integer value, or it can involve speculating operations on a PHI or
 ///    select to be a PHI or select of the results.
 /// 3) Finally, this will try to detect a pattern of accesses which map cleanly
 ///    onto insert and extract operations on a vector value, and convert them to
 ///    this form. By doing so, it will enable promotion of vector aggregates to
 ///    SSA vector values.
 class SROAPass : public PassInfoMixin<SROAPass> {
   LLVMContext *C = nullptr;
-  DominatorTree *DT = nullptr;
+  DomTreeUpdater *DTU = nullptr;
   AssumptionCache *AC = nullptr;
 
   /// Worklist of alloca instructions to simplify.
   ///
   /// Each alloca in the function is added to this. Each new alloca formed gets
   /// added to it as well to recursively simplify unless that alloca can be
   /// directly promoted. Finally, each time we rewrite a use of an alloca other
   /// the one being actively rewritten, we add it back onto the list if not
class SROAPass : public PassInfoMixin<SROAPass> {
 
   /// A worklist of PHIs to speculate prior to promoting allocas.
   ///
   /// All of these PHIs have been checked for the safety of speculation and by
   /// being speculated will allow promoting allocas currently in the promotable
   /// queue.
   SetVector<PHINode *, SmallVector<PHINode *, 2>> SpeculatablePHIs;
 
-  /// A worklist of select instructions to speculate prior to promoting
+  /// A worklist of select instructions to rewrite prior to promoting
   /// allocas.
-  ///
-  /// All of these select instructions have been checked for the safety of
-  /// speculation and by being speculated will allow promoting allocas
-  /// currently in the promotable queue.
-  SetVector<SelectInst *, SmallVector<SelectInst *, 2>> SpeculatableSelects;
+  SetVector<SelectInst *, SmallVector<SelectInst *, 2>> SelectsToRewrite;
 
 public:
   SROAPass() = default;
 
   /// Run the pass over the function.
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 
 private:
   friend class sroa::AllocaSliceRewriter;
   friend class sroa::SROALegacyPass;
 
   /// Helper used by both the public run method and by the legacy pass.
+  PreservedAnalyses runImpl(Function &F, DomTreeUpdater &RunDTU,
+                            AssumptionCache &RunAC);
   PreservedAnalyses runImpl(Function &F, DominatorTree &RunDT,
                             AssumptionCache &RunAC);
 
   bool presplitLoadsAndStores(AllocaInst &AI, sroa::AllocaSlices &AS);
   AllocaInst *rewritePartition(AllocaInst &AI, sroa::AllocaSlices &AS,
                                sroa::Partition &P);
   bool splitAlloca(AllocaInst &AI, sroa::AllocaSlices &AS);
   bool runOnAlloca(AllocaInst &AI);
   void clobberUse(Use &U);
   bool deleteDeadInstructions(SmallPtrSetImpl<AllocaInst *> &DeletedAllocas);
   bool promoteAllocas(Function &F);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_SCALAR_SROA_H

llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h

 /// After:
 ///   Head
 ///   if (Cond)
 ///     ThenBlock
 ///   else
 ///     ElseBlock
 ///   SplitBefore
 ///   Tail
+///
+/// Updates DT if given.
 void SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
                                    Instruction **ThenTerm,
                                    Instruction **ElseTerm,
-                                   MDNode *BranchWeights = nullptr);
+                                   MDNode *BranchWeights = nullptr,
+                                   DomTreeUpdater *DTU = nullptr);
 
 /// Check whether BB is the merge point of a if-region.
 /// If so, return the branch instruction that determines which entry into
 /// BB will be taken.  Also, return by references the block that will be
 /// entered from if the condition is true, and the block that will be
 /// entered if the condition is false.
 ///
 /// This does no checking to see if the true/false blocks have large or unsavory

llvm/lib/Transforms/Scalar/SROA.cpp

 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Analysis/AssumptionCache.h"
+#include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/Loads.h"
 #include "llvm/Analysis/PtrUseVisitor.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/ConstantFolder.h"
 #include "llvm/IR/Constants.h"

 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Scalar.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Utils/PromoteMemToReg.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <cstring>
 #include <iterator>

 STATISTIC(NumAllocasAnalyzed, "Number of allocas analyzed for replacement");
 STATISTIC(NumAllocaPartitions, "Number of alloca partitions formed");
 STATISTIC(MaxPartitionsPerAlloca, "Maximum number of partitions per alloca");
 STATISTIC(NumAllocaPartitionUses, "Number of alloca partition uses rewritten");
 STATISTIC(MaxUsesPerAllocaPartition, "Maximum number of uses of a partition");
 STATISTIC(NumNewAllocas, "Number of new, smaller allocas introduced");
 STATISTIC(NumPromoted, "Number of allocas promoted to SSA values");
 STATISTIC(NumLoadsSpeculated, "Number of loads speculated to allow promotion");
+STATISTIC(NumLoadsRewritten, "Number of loads rewritten to allow promotion");
 STATISTIC(NumDeleted, "Number of instructions deleted");
 STATISTIC(NumVectorized, "Number of vectorized aggregates");
 
 /// Hidden option to experiment with completely strict handling of inbounds
 /// GEPs.
 static cl::opt<bool> SROAStrictInbounds("sroa-strict-inbounds", cl::init(false),
                                         cl::Hidden);
 

 /// to:
 ///   %V1 = load i32* %Alloca      -> will be mem2reg'd
 ///   %V2 = load i32* %Other
 ///   %V = select i1 %cond, i32 %V1, i32 %V2
 ///
 /// We can do this to a select if its only uses are loads and if the operand
 /// to the select can be loaded unconditionally. If found an intervening bitcast
 /// with a single use of the load, allow the promotion.
-static bool isSafeSelectToSpeculate(SelectInst &SI) {
-  Value *TValue = SI.getTrueValue();
-  Value *FValue = SI.getFalseValue();
-  const DataLayout &DL = SI.getModule()->getDataLayout();
-
+static bool isSafeSelectToRewrite(SelectInst &SI) {
   for (User *U : SI.users()) {
     LoadInst *LI;
     BitCastInst *BC = dyn_cast<BitCastInst>(U);
     if (BC && BC->hasOneUse())
       LI = dyn_cast<LoadInst>(*BC->user_begin());
     else
       LI = dyn_cast<LoadInst>(U);
 
-    if (!LI || !LI->isSimple())
-      return false;
-
-    // Both operands to the select need to be dereferenceable, either
-    // absolutely (e.g. allocas) or at this point because we can see other
-    // accesses to it.
-    if (!isSafeToLoadUnconditionally(TValue, LI->getType(),
-                                     LI->getAlign(), DL, LI))
-      return false;
-    if (!isSafeToLoadUnconditionally(FValue, LI->getType(),
-                                     LI->getAlign(), DL, LI))
+    if (!LI)
       return false;
   }
 
   return true;
 }
 
-static void speculateSelectInstLoads(IRBuilderTy &IRB, SelectInst &SI) {
+static void rewriteLoadOfSelect(SelectInst &SI, LoadInst &LI,
+                                DomTreeUpdater &DTU) {
+  BasicBlock *Head = LI.getParent();
+  Instruction *ThenTerm = nullptr;
+  Instruction *ElseTerm = nullptr;
+  SplitBlockAndInsertIfThenElse(SI.getCondition(), &LI, &ThenTerm, &ElseTerm,
+                                SI.getMetadata(LLVMContext::MD_prof), &DTU);
+  BasicBlock *Tail = LI.getParent();
+  Tail->setName(Head->getName() + ".cont");
+  auto *PN = PHINode::Create(LI.getType(), 2, "", &LI);
+  for (auto CondBlock : enumerate(successors(Head))) {
+    auto *BB = CondBlock.value();
+    bool IsThen = CondBlock.index() == 0;
+    BB->setName(Head->getName() + (IsThen ? ".then" : ".else"));
+    auto *CondLoad = cast<LoadInst>(LI.clone());
+    CondLoad->insertBefore(BB->getTerminator());
+    CondLoad->setOperand(0, SI.getOperand(1 + CondBlock.index()));
+    CondLoad->setName(LI.getName() + (IsThen ? ".then" : ".else") + ".val");
+    PN->addIncoming(CondLoad, BB);
+  }
+  PN->takeName(&LI);
+  NumLoadsRewritten += 2;
+  LLVM_DEBUG(dbgs() << "          to: " << *PN << "\n");
+  LI.replaceAllUsesWith(PN);
+}
+
+static void rewriteSelectInstLoads(SelectInst &SI, DomTreeUpdater &DTU) {
   LLVM_DEBUG(dbgs() << "    original: " << SI << "\n");
 
-  IRB.SetInsertPoint(&SI);
-  Value *TV = SI.getTrueValue();
-  Value *FV = SI.getFalseValue();
   // Replace the loads of the select with a select of two loads.
   while (!SI.use_empty()) {
     LoadInst *LI;
     BitCastInst *BC = dyn_cast<BitCastInst>(SI.user_back());
     if (BC) {
       assert(BC->hasOneUse() && "Bitcast should have a single use.");
       LI = cast<LoadInst>(BC->user_back());
     } else {
       LI = cast<LoadInst>(SI.user_back());
     }
 
-    assert(LI->isSimple() && "We only speculate simple loads");
-
-    IRB.SetInsertPoint(LI);
-    Value *NewTV =
-        BC ? IRB.CreateBitCast(TV, BC->getType(), TV->getName() + ".sroa.cast")
-           : TV;
-    Value *NewFV =
-        BC ? IRB.CreateBitCast(FV, BC->getType(), FV->getName() + ".sroa.cast")
-           : FV;
-    LoadInst *TL = IRB.CreateLoad(LI->getType(), NewTV,
-                                  LI->getName() + ".sroa.speculate.load.true");
-    LoadInst *FL = IRB.CreateLoad(LI->getType(), NewFV,
-                                  LI->getName() + ".sroa.speculate.load.false");
-    NumLoadsSpeculated += 2;
-
-    // Transfer alignment and AA info if present.
-    TL->setAlignment(LI->getAlign());
-    FL->setAlignment(LI->getAlign());
-
-    AAMDNodes Tags = LI->getAAMetadata();
-    if (Tags) {
-      TL->setAAMetadata(Tags);
-      FL->setAAMetadata(Tags);
-    }
-
-    Value *V = IRB.CreateSelect(SI.getCondition(), TL, FL,
-                                LI->getName() + ".sroa.speculated");
-
-    LLVM_DEBUG(dbgs() << "          speculated to: " << *V << "\n");
-    LI->replaceAllUsesWith(V);
+    rewriteLoadOfSelect(SI, *LI, DTU);
     LI->eraseFromParent();
+
     if (BC)
       BC->eraseFromParent();
   }
   SI.eraseFromParent();
 }
 
 /// Build a GEP out of a base pointer and indices.
 ///
 /// This will return the BasePtr if that is valid, or build a new GEP
 /// instruction using the IRBuilder if GEP-ing is needed.
do {
     } else if (VectorType *VectorTy = dyn_cast<VectorType>(ElementTy)) {
       ElementTy = VectorTy->getElementType();
       Indices.push_back(IRB.getInt32(0));
     } else if (StructType *STy = dyn_cast<StructType>(ElementTy)) {
       if (STy->element_begin() == STy->element_end())
         break; // Nothing left to descend into.
       ElementTy = *STy->element_begin();
       Indices.push_back(IRB.getInt32(0));
     } else {
       break;
     }
     ++NumLayers;
   } while (ElementTy != TargetTy);
   if (ElementTy != TargetTy)
     Indices.erase(Indices.end() - NumLayers, Indices.end());
 
   return buildGEP(IRB, BasePtr, Indices, NamePrefix);
for (PHINode *PHI : PHIUsers)
     if (!isSafePHIToSpeculate(*PHI)) {
       Promotable = false;
       PHIUsers.clear();
       SelectUsers.clear();
       break;
     }
 
   for (SelectInst *Sel : SelectUsers)
-    if (!isSafeSelectToSpeculate(*Sel)) {
+    if (!isSafeSelectToRewrite(*Sel)) {
       Promotable = false;
       PHIUsers.clear();
       SelectUsers.clear();
       break;
     }
 
   if (Promotable) {
     for (Use *U : AS.getDeadUsesIfPromotable()) {
       auto *OldInst = dyn_cast<Instruction>(U->get());
       Value::dropDroppableUse(*U);
       if (OldInst)
         if (isInstructionTriviallyDead(OldInst))
           DeadInsts.push_back(OldInst);
     }
     if (PHIUsers.empty() && SelectUsers.empty()) {
       // Promote the alloca.
       PromotableAllocas.push_back(NewAI);
     } else {
       // If we have either PHIs or Selects to speculate, add them to those
       // worklists and re-queue the new alloca so that we promote in on the
       // next iteration.
       for (PHINode *PHIUser : PHIUsers)
         SpeculatablePHIs.insert(PHIUser);
       for (SelectInst *SelectUser : SelectUsers)
-        SpeculatableSelects.insert(SelectUser);
+        SelectsToRewrite.insert(SelectUser);
       Worklist.insert(NewAI);
     }
   } else {
     // Drop any post-promotion work items if promotion didn't happen.
     while (PostPromotionWorklist.size() > PPWOldSize)
       PostPromotionWorklist.pop_back();
 
     // We couldn't promote and we didn't create a new partition, nothing
if (AS.begin() == AS.end())
     return Changed;
 
   Changed |= splitAlloca(AI, AS);
 
   LLVM_DEBUG(dbgs() << "  Speculating PHIs\n");
   while (!SpeculatablePHIs.empty())
     speculatePHINodeLoads(IRB, *SpeculatablePHIs.pop_back_val());
 
-  LLVM_DEBUG(dbgs() << "  Speculating Selects\n");
-  while (!SpeculatableSelects.empty())
-    speculateSelectInstLoads(IRB, *SpeculatableSelects.pop_back_val());
+  LLVM_DEBUG(dbgs() << "  Rewriting Selects\n");
+  while (!SelectsToRewrite.empty())
+    rewriteSelectInstLoads(*SelectsToRewrite.pop_back_val(), *DTU);
 
   return Changed;
 }
 
 /// Delete the dead instructions accumulated in this run.
 ///
 /// Recursively deletes the dead instructions we've accumulated. This is done
 /// at the very end to maximize locality of the recursive delete and to

 /// This function returns whether any promotion occurred.
 bool SROAPass::promoteAllocas(Function &F) {
   if (PromotableAllocas.empty())
     return false;
 
   NumPromoted += PromotableAllocas.size();
 
   LLVM_DEBUG(dbgs() << "Promoting allocas with mem2reg...\n");
-  PromoteMemToReg(PromotableAllocas, *DT, AC);
+  PromoteMemToReg(PromotableAllocas, DTU->getDomTree(), AC);
   PromotableAllocas.clear();
   return true;
 }
 
-PreservedAnalyses SROAPass::runImpl(Function &F, DominatorTree &RunDT,
+PreservedAnalyses SROAPass::runImpl(Function &F, DomTreeUpdater &RunDTU,
                                     AssumptionCache &RunAC) {
   LLVM_DEBUG(dbgs() << "SROA function: " << F.getName() << "\n");
   C = &F.getContext();
-  DT = &RunDT;
+  DTU = &RunDTU;
   AC = &RunAC;
 
   BasicBlock &EntryBB = F.getEntryBlock();
   for (BasicBlock::iterator I = EntryBB.begin(), E = std::prev(EntryBB.end());
        I != E; ++I) {
     if (AllocaInst *AI = dyn_cast<AllocaInst>(I)) {
       if (isa<ScalableVectorType>(AI->getAllocatedType())) {
         if (isAllocaPromotable(AI))
do {
     Worklist = PostPromotionWorklist;
     PostPromotionWorklist.clear();
   } while (!Worklist.empty());
 
   if (!Changed)
     return PreservedAnalyses::all();
 
   PreservedAnalyses PA;
-  PA.preserveSet<CFGAnalyses>();
+  PA.preserve<DominatorTreeAnalysis>();
   return PA;
 }
 
+PreservedAnalyses SROAPass::runImpl(Function &F, DominatorTree &RunDT,
+                                    AssumptionCache &RunAC) {
+  DomTreeUpdater DTU(RunDT, DomTreeUpdater::UpdateStrategy::Lazy);
+  return runImpl(F, DTU, RunAC);
+}
+
 PreservedAnalyses SROAPass::run(Function &F, FunctionAnalysisManager &AM) {
   return runImpl(F, AM.getResult<DominatorTreeAnalysis>(F),
                  AM.getResult<AssumptionAnalysis>(F));
 }
 
 /// A legacy pass for the legacy pass manager that wraps the \c SROA pass.
 ///
 /// This is in the llvm namespace purely to allow it to be a friend of the \c
 /// SROA pass.
 class llvm::sroa::SROALegacyPass : public FunctionPass {
   /// The SROA implementation.
   SROAPass Impl;
 
 public:
   static char ID;
 
   SROALegacyPass() : FunctionPass(ID) {
     initializeSROALegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnFunction(Function &F) override {
     if (skipFunction(F))
       return false;
 
     auto PA = Impl.runImpl(
         F, getAnalysis<DominatorTreeWrapperPass>().getDomTree(),
         getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F));
     return !PA.areAllPreserved();
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AssumptionCacheTracker>();
     AU.addRequired<DominatorTreeWrapperPass>();
     AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.setPreservesCFG();
+    AU.addPreserved<DominatorTreeWrapperPass>();
   }
 
   StringRef getPassName() const override { return "SROA"; }
 };
 
 char SROALegacyPass::ID = 0;
 
 FunctionPass *llvm::createSROAPass() { return new SROALegacyPass(); }
 
 INITIALIZE_PASS_BEGIN(SROALegacyPass, "sroa",
                       "Scalar Replacement Of Aggregates", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_END(SROALegacyPass, "sroa", "Scalar Replacement Of Aggregates",
                     false, false)

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

Instruction *llvm::SplitBlockAndInsertIfThen(Value *Cond,
   return SplitBlockAndInsertIfThenImpl(Cond, SplitBefore, Unreachable,
                                        BranchWeights, DTU, /*DT=*/nullptr, LI,
                                        ThenBlock);
 }
 
 void llvm::SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
                                          Instruction **ThenTerm,
                                          Instruction **ElseTerm,
-                                         MDNode *BranchWeights) {
+                                         MDNode *BranchWeights,
+                                         DomTreeUpdater *DTU) {
   BasicBlock *Head = SplitBefore->getParent();
+
+  SmallPtrSet<BasicBlock *, 8> UniqueOrigSuccessors;
+  if (DTU)
+    UniqueOrigSuccessors.insert(succ_begin(Head), succ_end(Head));
+
   BasicBlock *Tail = Head->splitBasicBlock(SplitBefore->getIterator());
   Instruction *HeadOldTerm = Head->getTerminator();
   LLVMContext &C = Head->getContext();
   BasicBlock *ThenBlock = BasicBlock::Create(C, "", Head->getParent(), Tail);
   BasicBlock *ElseBlock = BasicBlock::Create(C, "", Head->getParent(), Tail);
   *ThenTerm = BranchInst::Create(Tail, ThenBlock);
   (*ThenTerm)->setDebugLoc(SplitBefore->getDebugLoc());
   *ElseTerm = BranchInst::Create(Tail, ElseBlock);
   (*ElseTerm)->setDebugLoc(SplitBefore->getDebugLoc());
   BranchInst *HeadNewTerm =
     BranchInst::Create(/*ifTrue*/ThenBlock, /*ifFalse*/ElseBlock, Cond);
   HeadNewTerm->setMetadata(LLVMContext::MD_prof, BranchWeights);
   ReplaceInstWithInst(HeadOldTerm, HeadNewTerm);
+  if (DTU) {
+    SmallVector<DominatorTree::UpdateType, 8> Updates;
+    Updates.reserve(4 + 2 * UniqueOrigSuccessors.size());
+    for (BasicBlock *Succ : successors(Head)) {
+      Updates.push_back({DominatorTree::Insert, Head, Succ});
+      Updates.push_back({DominatorTree::Insert, Succ, Tail});
+    }
+    for (BasicBlock *UniqueOrigSuccessor : UniqueOrigSuccessors)
+      Updates.push_back({DominatorTree::Insert, Tail, UniqueOrigSuccessor});
+    for (BasicBlock *UniqueOrigSuccessor : UniqueOrigSuccessors)
+      Updates.push_back({DominatorTree::Delete, Head, UniqueOrigSuccessor});
+    DTU->applyUpdates(Updates);
+  }
 }
 
 BranchInst *llvm::GetIfCondition(BasicBlock *BB, BasicBlock *&IfTrue,
                                  BasicBlock *&IfFalse) {
   PHINode *SomePHI = dyn_cast<PHINode>(BB->begin());
   BasicBlock *Pred1 = nullptr;
   BasicBlock *Pred2 = nullptr;
 

llvm/test/Transforms/SROA/addrspacecast.ll

;
 
   %cond.in = select i1 %b, ptr addrspace(1) %asc, ptr addrspace(1) %asc
   %cond = load i64, ptr addrspace(1) %cond.in, align 8
   ret void
 }
 
 define void @select_addrspacecast_const_op(i1 %a, i1 %b) {
 ; CHECK-LABEL: @select_addrspacecast_const_op(
-; CHECK-NEXT:    [[C:%.*]] = alloca i64, align 8
-; CHECK-NEXT:    [[C_0_ASC_SROA_CAST:%.*]] = addrspacecast ptr [[C]] to ptr addrspace(1)
-; CHECK-NEXT:    [[COND_IN:%.*]] = select i1 [[B:%.*]], ptr addrspace(1) [[C_0_ASC_SROA_CAST]], ptr addrspace(1) null
-; CHECK-NEXT:    [[COND:%.*]] = load i64, ptr addrspace(1) [[COND_IN]], align 8
+; CHECK-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    [[COND_ELSE_VAL:%.*]] = load i64, ptr addrspace(1) null, align 8
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[DOTTHEN]] ], [ [[COND_ELSE_VAL]], [[DOTELSE]] ]
 ; CHECK-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %asc = addrspacecast ptr %p.0.c to ptr addrspace(1)
 
   %cond.in = select i1 %b, ptr addrspace(1) %asc, ptr addrspace(1) null
   %cond = load i64, ptr addrspace(1) %cond.in, align 8
   ret void
 }
 
 ;; If this was external, we wouldn't be able to prove dereferenceability
 ;; of the location.
 @gv = addrspace(1) global i64 zeroinitializer
 
 define void @select_addrspacecast_gv(i1 %a, i1 %b) {
 ; CHECK-LABEL: @select_addrspacecast_gv(
-; CHECK-NEXT:    [[COND_SROA_SPECULATE_LOAD_FALSE:%.*]] = load i64, ptr addrspace(1) @gv, align 8
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[B:%.*]], i64 undef, i64 [[COND_SROA_SPECULATE_LOAD_FALSE]]
+; CHECK-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    [[COND_ELSE_VAL:%.*]] = load i64, ptr addrspace(1) @gv, align 8
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[DOTTHEN]] ], [ [[COND_ELSE_VAL]], [[DOTELSE]] ]
 ; CHECK-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %asc = addrspacecast ptr %p.0.c to ptr addrspace(1)
 
   %cond.in = select i1 %b, ptr addrspace(1) %asc, ptr addrspace(1) @gv
   %cond = load i64, ptr addrspace(1) %cond.in, align 8
   ret void
 }
 
 define void @select_addrspacecast_gv_constexpr(i1 %a, i1 %b) {
 ; CHECK-LABEL: @select_addrspacecast_gv_constexpr(
-; CHECK-NEXT:    [[COND_SROA_SPECULATE_LOAD_FALSE:%.*]] = load i64, ptr addrspace(2) addrspacecast (ptr addrspace(1) @gv to ptr addrspace(2)), align 8
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[B:%.*]], i64 undef, i64 [[COND_SROA_SPECULATE_LOAD_FALSE]]
+; CHECK-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    [[COND_ELSE_VAL:%.*]] = load i64, ptr addrspace(2) addrspacecast (ptr addrspace(1) @gv to ptr addrspace(2)), align 8
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[DOTTHEN]] ], [ [[COND_ELSE_VAL]], [[DOTELSE]] ]
 ; CHECK-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %asc = addrspacecast ptr %p.0.c to ptr addrspace(2)
 
   %cond.in = select i1 %b, ptr addrspace(2) %asc, ptr addrspace(2) addrspacecast (ptr addrspace(1) @gv to ptr addrspace(2))
   %cond = load i64, ptr addrspace(2) %cond.in, align 8
   ret void
 }
 
 define i8 @select_addrspacecast_i8(i1 %c) {
 ; CHECK-LABEL: @select_addrspacecast_i8(
-; CHECK-NEXT:    [[RET_SROA_SPECULATED:%.*]] = select i1 [[C:%.*]], i8 undef, i8 undef
-; CHECK-NEXT:    ret i8 [[RET_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[C:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[RET:%.*]] = phi i8 [ undef, [[DOTTHEN]] ], [ undef, [[DOTELSE]] ]
+; CHECK-NEXT:    ret i8 [[RET]]
 ;
   %a = alloca i8
   %b = alloca i8
 
   %a.ptr = addrspacecast ptr %a to ptr addrspace(1)
   %b.ptr = addrspacecast ptr %b to ptr addrspace(1)
 
   %ptr = select i1 %c, ptr addrspace(1) %a.ptr, ptr addrspace(1) %b.ptr

llvm/test/Transforms/SROA/basictest.ll

 
 end:
   call void @llvm.memcpy.p0.p0.i32(ptr %data, ptr %tmp, i32 %size, i1 false)
   ret void
 }
 
 define void @PR15805(i1 %a, i1 %b) {
 ; CHECK-LABEL: @PR15805(
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[B:%.*]], i64 undef, i64 undef
+; CHECK-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[DOTTHEN]] ], [ undef, [[DOTELSE]] ]
 ; CHECK-NEXT:    ret void
 ;
 
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %cond.in = select i1 %b, ptr %p.0.c, ptr %c
   %cond = load i64, ptr %cond.in, align 8
   ret void
 }
 
 define void @PR15805.1(i1 %a, i1 %b, i1 %c2) {
 ; Same as the normal PR15805, but rigged to place the use before the def inside
 ; of looping unreachable code. This helps ensure that we aren't sensitive to the
 ; order in which the uses of the alloca are visited.
 ;
 ; CHECK-LABEL: @PR15805.1(
 ; CHECK-NEXT:    br label [[EXIT:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[A:%.*]], i64 undef, i64 undef
+; CHECK-NEXT:    br i1 [[A:%.*]], label [[LOOP_THEN:%.*]], label [[LOOP_ELSE:%.*]]
+; CHECK:       loop.then:
+; CHECK-NEXT:    [[C_0_LOAD:%.*]] = load i64, ptr poison, align 8
+; CHECK-NEXT:    br label [[LOOP_CONT:%.*]]
+; CHECK:       loop.else:
+; CHECK-NEXT:    [[C_0_LOAD1:%.*]] = load i64, ptr poison, align 8
+; CHECK-NEXT:    br label [[LOOP_CONT]]
+; CHECK:       loop.cont:
+; CHECK-NEXT:    [[COND:%.*]] = phi i64 [ [[C_0_LOAD]], [[LOOP_THEN]] ], [ [[C_0_LOAD1]], [[LOOP_ELSE]] ]
 ; CHECK-NEXT:    br i1 [[C2:%.*]], label [[LOOP:%.*]], label [[EXIT]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
 
   %c = alloca i64, align 8
   br label %exit
 

 define void @PR16651.2(<2 x float> %val, i1 %c1) {
 ; This test case caused a crash due to failing to promote given a select that
 ; can't be speculated. It shouldn't be promoted, but we missed that fact when
 ; analyzing whether we could form a vector promotion because that code didn't
 ; bail on select instructions.
 ;
 ; CHECK-LABEL: @PR16651.2(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TV1_SROA_0:%.*]] = alloca <2 x float>, align 8
-; CHECK-NEXT:    store <2 x float> [[VAL:%.*]], ptr [[TV1_SROA_0]], align 8
-; CHECK-NEXT:    [[COND105_IN_I_I:%.*]] = select i1 [[C1:%.*]], ptr null, ptr [[TV1_SROA_0]]
-; CHECK-NEXT:    [[COND105_I_I:%.*]] = load float, ptr [[COND105_IN_I_I]], align 8
+; CHECK-NEXT:    br i1 [[C1:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    [[COND105_I_I_THEN_VAL:%.*]] = load float, ptr null, align 8
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    [[TV1_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <2 x float> [[VAL:%.*]], i32 0
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[COND105_I_I:%.*]] = phi float [ [[COND105_I_I_THEN_VAL]], [[ENTRY_THEN]] ], [ [[TV1_SROA_0_0_VEC_EXTRACT]], [[ENTRY_ELSE]] ]
 ; CHECK-NEXT:    ret void
 ;
 
 entry:
   %tv1 = alloca { <2 x float>, <2 x float> }, align 8
   %0 = getelementptr { <2 x float>, <2 x float> }, ptr %tv1, i64 0, i32 1
   store <2 x float> %val, ptr %0, align 8
   %1 = getelementptr inbounds { <2 x float>, <2 x float> }, ptr %tv1, i64 0, i32 1, i64 0

 
 %struct.STest = type { %struct.SPos, %struct.SPos }
 %struct.SPos = type { float, float }
 
 define void @PR25873(ptr %outData) {
 ; CHECK-LABEL: @PR25873(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    store i32 1123418112, ptr [[OUTDATA:%.*]], align 4
-; CHECK-NEXT:    [[DOTSROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[OUTDATA:%.*]], i64 4
-; CHECK-NEXT:    store i32 1139015680, ptr [[DOTSROA_IDX]], align 4
-; CHECK-NEXT:    [[TMPDATA_SROA_6_0__SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[OUTDATA:%.*]], i64 8
+; CHECK-NEXT:    [[OUTDATA_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[OUTDATA]], i64 4
+; CHECK-NEXT:    store i32 1139015680, ptr [[OUTDATA_SROA_IDX]], align 4
+; CHECK-NEXT:    [[TMPDATA_SROA_6_0_OUTDATA_SROA_IDX:%.*]] = getelementptr inbounds i8, ptr [[OUTDATA]], i64 8
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_4_0_INSERT_EXT:%.*]] = zext i32 1139015680 to i64
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_4_0_INSERT_SHIFT:%.*]] = shl i64 [[TMPDATA_SROA_6_SROA_4_0_INSERT_EXT]], 32
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_4_0_INSERT_MASK:%.*]] = and i64 undef, 4294967295
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_4_0_INSERT_INSERT:%.*]] = or i64 [[TMPDATA_SROA_6_SROA_4_0_INSERT_MASK]], [[TMPDATA_SROA_6_SROA_4_0_INSERT_SHIFT]]
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_0_0_INSERT_EXT:%.*]] = zext i32 1123418112 to i64
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_0_0_INSERT_MASK:%.*]] = and i64 [[TMPDATA_SROA_6_SROA_4_0_INSERT_INSERT]], -4294967296
 ; CHECK-NEXT:    [[TMPDATA_SROA_6_SROA_0_0_INSERT_INSERT:%.*]] = or i64 [[TMPDATA_SROA_6_SROA_0_0_INSERT_MASK]], [[TMPDATA_SROA_6_SROA_0_0_INSERT_EXT]]
-; CHECK-NEXT:    store i64 [[TMPDATA_SROA_6_SROA_0_0_INSERT_INSERT]], ptr [[TMPDATA_SROA_6_0__SROA_IDX]], align 4
+; CHECK-NEXT:    store i64 [[TMPDATA_SROA_6_SROA_0_0_INSERT_INSERT]], ptr [[TMPDATA_SROA_6_0_OUTDATA_SROA_IDX]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %tmpData = alloca %struct.STest, align 8
   call void @llvm.lifetime.start.p0(i64 16, ptr %tmpData)
   store float 1.230000e+02, ptr %tmpData, align 8
   %y = getelementptr inbounds %struct.STest, ptr %tmpData, i64 0, i32 0, i32 1
   store float 4.560000e+02, ptr %y, align 4

llvm/test/Transforms/SROA/phi-and-select.ll

exit:
   %result = load i32, ptr %phi
   ret i32 %result
 }
 
 define i32 @test2() {
 ; CHECK-LABEL: @test2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[COND:%.*]] = icmp sle i32 0, 1
-; CHECK-NEXT:    [[RESULT_SROA_SPECULATED:%.*]] = select i1 [[COND]], i32 1, i32 0
-; CHECK-NEXT:    ret i32 [[RESULT_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[COND]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[RESULT:%.*]] = phi i32 [ 1, [[ENTRY_THEN]] ], [ 0, [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    ret i32 [[RESULT]]
 ;
 entry:
   %a = alloca [2 x i32]
 
   %a1 = getelementptr [2 x i32], ptr %a, i64 0, i32 1
   store i32 0, ptr %a
   store i32 1, ptr %a1
   %v0 = load i32, ptr %a
   %v1 = load i32, ptr %a1
 
   %cond = icmp sle i32 %v0, %v1
   %select = select i1 %cond, ptr %a1, ptr %a
 
   %result = load i32, ptr %select
   ret i32 %result
 }
 
 define float @test2_bitcast() {
 ; CHECK-LABEL: @test2_bitcast(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[COND:%.*]] = icmp sle i32 0, 1
+; CHECK-NEXT:    br i1 [[COND]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32 1 to float
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32 0 to float
-; CHECK-NEXT:    [[RESULT_SROA_SPECULATED:%.*]] = select i1 [[COND]], float [[TMP0]], float [[TMP1]]
-; CHECK-NEXT:    ret float [[RESULT_SROA_SPECULATED]]
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[RESULT:%.*]] = phi float [ [[TMP0]], [[ENTRY_THEN]] ], [ [[TMP1]], [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    ret float [[RESULT]]
 ;
 entry:
   %a = alloca [2 x i32]
   %a1 = getelementptr [2 x i32], ptr %a, i64 0, i32 1
   store i32 0, ptr %a
   store i32 1, ptr %a1
   %v0 = load i32, ptr %a
   %v1 = load i32, ptr %a1

 }
 
 define i32 @test9(i32 %b, ptr %ptr) {
 ; Same as @test8 but for a select rather than a PHI node.
 ; CHECK-LABEL: @test9(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    store i32 0, ptr [[PTR:%.*]], align 4
 ; CHECK-NEXT:    [[TEST:%.*]] = icmp ne i32 [[B:%.*]], 0
-; CHECK-NEXT:    [[LOADED_SROA_SPECULATE_LOAD_FALSE:%.*]] = load i32, ptr [[PTR]], align 4
-; CHECK-NEXT:    [[LOADED_SROA_SPECULATED:%.*]] = select i1 [[TEST]], i32 undef, i32 [[LOADED_SROA_SPECULATE_LOAD_FALSE]]
-; CHECK-NEXT:    ret i32 [[LOADED_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[TEST]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    [[LOADED_ELSE_VAL:%.*]] = load i32, ptr [[PTR]], align 4
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[LOADED:%.*]] = phi i32 [ undef, [[ENTRY_THEN]] ], [ [[LOADED_ELSE_VAL]], [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    ret i32 [[LOADED]]
 ;
 
 entry:
   %f = alloca float
   store i32 0, ptr %ptr
   %test = icmp ne i32 %b, 0
   %select = select i1 %test, ptr %f, ptr %ptr
   %loaded = load i32, ptr %select, align 4

 define float @test11(i32 %b, ptr %ptr) {
 ; Same as @test10 but for a select rather than a PHI node.
 ; CHECK-LABEL: @test11(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[F:%.*]] = alloca double, align 8
 ; CHECK-NEXT:    store double 0.000000e+00, ptr [[F]], align 8
 ; CHECK-NEXT:    store float 0.000000e+00, ptr [[PTR:%.*]], align 4
 ; CHECK-NEXT:    [[TEST:%.*]] = icmp ne i32 [[B:%.*]], 0
-; CHECK-NEXT:    [[F_0_LOADED_SROA_SPECULATE_LOAD_TRUE:%.*]] = load float, ptr [[F]], align 8
-; CHECK-NEXT:    [[LOADED_SROA_SPECULATE_LOAD_FALSE:%.*]] = load float, ptr [[PTR]], align 4
-; CHECK-NEXT:    [[LOADED_SROA_SPECULATED:%.*]] = select i1 [[TEST]], float [[F_0_LOADED_SROA_SPECULATE_LOAD_TRUE]], float [[LOADED_SROA_SPECULATE_LOAD_FALSE]]
-; CHECK-NEXT:    ret float [[LOADED_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[TEST]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    [[F_0_LOADED_THEN_VAL:%.*]] = load float, ptr [[F]], align 8
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    [[LOADED_ELSE_VAL:%.*]] = load float, ptr [[PTR]], align 4
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[LOADED:%.*]] = phi float [ [[F_0_LOADED_THEN_VAL]], [[ENTRY_THEN]] ], [ [[LOADED_ELSE_VAL]], [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    ret float [[LOADED]]
 ;
 
 entry:
   %f = alloca double
   store double 0.0, ptr %f
   store float 0.0, ptr %ptr
   %test = icmp ne i32 %b, 0
   %select = select i1 %test, ptr %f, ptr %ptr
exit:
   %phi = phi ptr [ %a1x, %then ], [ %z, %entry ]
   %result = load i32, ptr %phi, align 4
   ret i32 %result
 }
 
 ; Don't speculate a load based on an earlier volatile operation.
 define i8 @volatile_select(ptr %p, i1 %b) {
 ; CHECK-LABEL: @volatile_select(
-; CHECK-NEXT:    [[P2:%.*]] = alloca i8, align 1
-; CHECK-NEXT:    store i8 0, ptr [[P2]], align 1
 ; CHECK-NEXT:    store volatile i8 0, ptr [[P:%.*]], align 1
-; CHECK-NEXT:    [[PX:%.*]] = select i1 [[B:%.*]], ptr [[P]], ptr [[P2]]
-; CHECK-NEXT:    [[V2:%.*]] = load i8, ptr [[PX]], align 1
+; CHECK-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    [[V2_THEN_VAL:%.*]] = load i8, ptr [[P]], align 1
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[V2:%.*]] = phi i8 [ [[V2_THEN_VAL]], [[DOTTHEN]] ], [ 0, [[DOTELSE]] ]
 ; CHECK-NEXT:    ret i8 [[V2]]
 ;
   %p2 = alloca i8
   store i8 0, ptr %p2
   store volatile i8 0, ptr %p
   %px = select i1 %b, ptr %p, ptr %p2
   %v2 = load i8, ptr %px
   ret i8 %v2
 }

llvm/test/Transforms/SROA/select-gep.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -passes=sroa < %s | FileCheck %s
 
 %pair = type { i32, i32 }
 
 define i32 @test_sroa_select_gep(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_select_gep(
 ; CHECK-NEXT:  bb:
-; CHECK-NEXT:    [[LOAD_SROA_SPECULATED:%.*]] = select i1 [[COND:%.*]], i32 1, i32 2
-; CHECK-NEXT:    ret i32 [[LOAD_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_THEN:%.*]], label [[BB_ELSE:%.*]]
+; CHECK:       bb.then:
+; CHECK-NEXT:    br label [[BB_CONT:%.*]]
+; CHECK:       bb.else:
+; CHECK-NEXT:    br label [[BB_CONT]]
+; CHECK:       bb.cont:
+; CHECK-NEXT:    [[LOAD:%.*]] = phi i32 [ 1, [[BB_THEN]] ], [ 2, [[BB_ELSE]] ]
+; CHECK-NEXT:    ret i32 [[LOAD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %b = alloca %pair, align 4
   %gep_a = getelementptr inbounds %pair, ptr %a, i32 0, i32 1
   %gep_b = getelementptr inbounds %pair, ptr %b, i32 0, i32 1
   store i32 1, ptr %gep_a, align 4
   store i32 2, ptr %gep_b, align 4
   %select = select i1 %cond, ptr %a, ptr %b
   %gep = getelementptr inbounds %pair, ptr %select, i32 0, i32 1
   %load = load i32, ptr %gep, align 4
   ret i32 %load
 }
 
 define i32 @test_sroa_select_gep_non_inbound(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_select_gep_non_inbound(
 ; CHECK-NEXT:  bb:
-; CHECK-NEXT:    [[LOAD_SROA_SPECULATED:%.*]] = select i1 [[COND:%.*]], i32 1, i32 2
-; CHECK-NEXT:    ret i32 [[LOAD_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_THEN:%.*]], label [[BB_ELSE:%.*]]
+; CHECK:       bb.then:
+; CHECK-NEXT:    br label [[BB_CONT:%.*]]
+; CHECK:       bb.else:
+; CHECK-NEXT:    br label [[BB_CONT]]
+; CHECK:       bb.cont:
+; CHECK-NEXT:    [[LOAD:%.*]] = phi i32 [ 1, [[BB_THEN]] ], [ 2, [[BB_ELSE]] ]
+; CHECK-NEXT:    ret i32 [[LOAD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %b = alloca %pair, align 4
   %gep_a = getelementptr %pair, ptr %a, i32 0, i32 1
   %gep_b = getelementptr %pair, ptr %b, i32 0, i32 1
   store i32 1, ptr %gep_a, align 4
   store i32 2, ptr %gep_b, align 4

 ; CHECK-NEXT:    [[A_SROA_0:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[A_SROA_2:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[B_SROA_0:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[B_SROA_2:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    store i32 11, ptr [[A_SROA_0]], align 4
 ; CHECK-NEXT:    store i32 12, ptr [[B_SROA_0]], align 4
 ; CHECK-NEXT:    store i32 21, ptr [[A_SROA_2]], align 4
 ; CHECK-NEXT:    store i32 22, ptr [[B_SROA_2]], align 4
-; CHECK-NEXT:    [[SELECT_SROA_SEL:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr [[B_SROA_0]]
-; CHECK-NEXT:    [[LOAD1:%.*]] = load volatile i32, ptr [[SELECT_SROA_SEL]], align 4
-; CHECK-NEXT:    [[SELECT_SROA_SEL3:%.*]] = select i1 [[COND]], ptr [[A_SROA_2]], ptr [[B_SROA_2]]
-; CHECK-NEXT:    [[LOAD2:%.*]] = load volatile i32, ptr [[SELECT_SROA_SEL3]], align 4
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_THEN2:%.*]], label [[BB_ELSE3:%.*]]
+; CHECK:       bb.then2:
+; CHECK-NEXT:    [[A_SROA_0_0_A_SROA_0_0_LOAD1_THEN_VAL:%.*]] = load volatile i32, ptr [[A_SROA_0]], align 4
+; CHECK-NEXT:    br label [[BB_CONT1:%.*]]
+; CHECK:       bb.else3:
+; CHECK-NEXT:    [[B_SROA_0_0_LOAD1_ELSE_VAL:%.*]] = load volatile i32, ptr [[B_SROA_0]], align 4
+; CHECK-NEXT:    br label [[BB_CONT1]]
+; CHECK:       bb.cont1:
+; CHECK-NEXT:    [[LOAD1:%.*]] = phi i32 [ [[A_SROA_0_0_A_SROA_0_0_LOAD1_THEN_VAL]], [[BB_THEN2]] ], [ [[B_SROA_0_0_LOAD1_ELSE_VAL]], [[BB_ELSE3]] ]
+; CHECK-NEXT:    br i1 [[COND]], label [[BB_THEN:%.*]], label [[BB_ELSE:%.*]]
+; CHECK:       bb.then:
+; CHECK-NEXT:    [[A_SROA_2_0_A_SROA_2_4_LOAD2_THEN_VAL:%.*]] = load volatile i32, ptr [[A_SROA_2]], align 4
+; CHECK-NEXT:    br label [[BB_CONT:%.*]]
+; CHECK:       bb.else:
+; CHECK-NEXT:    [[B_SROA_2_0_LOAD2_ELSE_VAL:%.*]] = load volatile i32, ptr [[B_SROA_2]], align 4
+; CHECK-NEXT:    br label [[BB_CONT]]
+; CHECK:       bb.cont:
+; CHECK-NEXT:    [[LOAD2:%.*]] = phi i32 [ [[A_SROA_2_0_A_SROA_2_4_LOAD2_THEN_VAL]], [[BB_THEN]] ], [ [[B_SROA_2_0_LOAD2_ELSE_VAL]], [[BB_ELSE]] ]
 ; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[LOAD1]], [[LOAD2]]
 ; CHECK-NEXT:    ret i32 [[ADD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %b = alloca %pair, align 4
   store i32 11, ptr %a, align 4
   store i32 12, ptr %b, align 4
   %gep_a1 = getelementptr inbounds %pair, ptr %a, i32 0, i32 1
   %gep_b1 = getelementptr inbounds %pair, ptr %b, i32 0, i32 1
   store i32 21, ptr %gep_a1, align 4
   store i32 22, ptr %gep_b1, align 4
   %select = select i1 %cond, ptr %a, ptr %b
   %load1 = load volatile i32, ptr %select, align 4
   %gep2 = getelementptr inbounds %pair, ptr %select, i32 0, i32 1
   %load2 = load volatile i32, ptr %gep2, align 4
   %add = add i32 %load1, %load2
   ret i32 %add
 }
 
 define i32 @test_sroa_select_gep_poison(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_select_gep_poison(
 ; CHECK-NEXT:  bb:
-; CHECK-NEXT:    [[A_SROA_0:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[SELECT_SROA_SEL:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr poison
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32, ptr [[SELECT_SROA_SEL]], align 4
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_THEN:%.*]], label [[BB_ELSE:%.*]]
+; CHECK:       bb.then:
+; CHECK-NEXT:    br label [[BB_CONT:%.*]]
+; CHECK:       bb.else:
+; CHECK-NEXT:    [[LOAD_ELSE_VAL:%.*]] = load i32, ptr poison, align 4
+; CHECK-NEXT:    br label [[BB_CONT]]
+; CHECK:       bb.cont:
+; CHECK-NEXT:    [[LOAD:%.*]] = phi i32 [ undef, [[BB_THEN]] ], [ [[LOAD_ELSE_VAL]], [[BB_ELSE]] ]
 ; CHECK-NEXT:    ret i32 [[LOAD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %select = select i1 %cond, ptr %a, ptr poison
   %gep = getelementptr inbounds %pair, ptr %select, i32 0, i32 1
   %load = load i32, ptr %gep, align 4
   ret i32 %load
 }
 
 define i32 @test_sroa_gep_select_gep(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_gep_select_gep(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[A_SROA_0:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[B_SROA_0:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    store i32 1, ptr [[A_SROA_0]], align 4
 ; CHECK-NEXT:    store i32 2, ptr [[B_SROA_0]], align 4
-; CHECK-NEXT:    [[SELECT_SROA_SEL:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr [[B_SROA_0]]
-; CHECK-NEXT:    [[SELECT2:%.*]] = select i1 [[COND]], ptr [[SELECT_SROA_SEL]], ptr [[A_SROA_0]]
+; CHECK-NEXT:    [[SELECT:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr [[B_SROA_0]]
+; CHECK-NEXT:    [[SELECT2:%.*]] = select i1 [[COND]], ptr [[SELECT]], ptr [[A_SROA_0]]
 ; CHECK-NEXT:    [[LOAD:%.*]] = load i32, ptr [[SELECT2]], align 4
 ; CHECK-NEXT:    ret i32 [[LOAD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %b = alloca %pair, align 4
   %gep_a = getelementptr inbounds %pair, ptr %a, i32 0, i32 1
   %gep_b = getelementptr inbounds %pair, ptr %b, i32 0, i32 1

llvm/test/Transforms/SROA/select-load.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -passes=sroa < %s | FileCheck %s
 
 %st.half = type { half }
 
 ; Allow speculateSelectInstLoads to fold load and select
 ; even if there is an intervening bitcast.
 define <2 x i16> @test_load_bitcast_select(i1 %cond1, i1 %cond2) {
 ; CHECK-LABEL: @test_load_bitcast_select(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 [[COND1:%.*]], label [[ENTRY_THEN2:%.*]], label [[ENTRY_ELSE3:%.*]]
+; CHECK:       entry.then2:
 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast half 0xHFFFF to i16
+; CHECK-NEXT:    br label [[ENTRY_CONT1:%.*]]
+; CHECK:       entry.else3:
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast half 0xH0000 to i16
-; CHECK-NEXT:    [[LD1_SROA_SPECULATED:%.*]] = select i1 [[COND1:%.*]], i16 [[TMP0]], i16 [[TMP1]]
-; CHECK-NEXT:    [[V1:%.*]] = insertelement <2 x i16> poison, i16 [[LD1_SROA_SPECULATED]], i32 0
+; CHECK-NEXT:    br label [[ENTRY_CONT1]]
+; CHECK:       entry.cont1:
+; CHECK-NEXT:    [[LD1:%.*]] = phi i16 [ [[TMP0]], [[ENTRY_THEN2]] ], [ [[TMP1]], [[ENTRY_ELSE3]] ]
+; CHECK-NEXT:    [[V1:%.*]] = insertelement <2 x i16> poison, i16 [[LD1]], i32 0
+; CHECK-NEXT:    br i1 [[COND2:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
 ; CHECK-NEXT:    [[TMP2:%.*]] = bitcast half 0xHFFFF to i16
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast half 0xH0000 to i16
-; CHECK-NEXT:    [[LD2_SROA_SPECULATED:%.*]] = select i1 [[COND2:%.*]], i16 [[TMP2]], i16 [[TMP3]]
-; CHECK-NEXT:    [[V2:%.*]] = insertelement <2 x i16> [[V1]], i16 [[LD2_SROA_SPECULATED]], i32 1
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[LD2:%.*]] = phi i16 [ [[TMP2]], [[ENTRY_THEN]] ], [ [[TMP3]], [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    [[V2:%.*]] = insertelement <2 x i16> [[V1]], i16 [[LD2]], i32 1
 ; CHECK-NEXT:    ret <2 x i16> [[V2]]
 ;
 entry:
   %true = alloca half, align 2
   %false = alloca half, align 2
   store half 0xHFFFF, ptr %true, align 2
   store half 0xH0000, ptr %false, align 2
   %sel1 = select i1 %cond1, ptr %true, ptr %false
   %ld1 = load i16, ptr %sel1, align 2
   %v1 = insertelement <2 x i16> poison, i16 %ld1, i32 0
   %sel2 = select i1 %cond2, ptr %true, ptr %false
   %ld2 = load i16, ptr %sel2, align 2
   %v2 = insertelement <2 x i16> %v1, i16 %ld2, i32 1
   ret <2 x i16> %v2
 }
 
 %st.args = type { i32, ptr }
 
 ; A bitcasted load and a direct load of select.
 define void @test_multiple_loads_select(i1 %cmp){
 ; CHECK-LABEL: @test_multiple_loads_select(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[ADDR_I8_SROA_SPECULATED:%.*]] = select i1 [[CMP:%.*]], ptr undef, ptr undef
-; CHECK-NEXT:    call void @foo_i8(ptr [[ADDR_I8_SROA_SPECULATED]])
-; CHECK-NEXT:    [[ADDR_I32_SROA_SPECULATED:%.*]] = select i1 [[CMP]], ptr undef, ptr undef
-; CHECK-NEXT:    call void @foo_i32(ptr [[ADDR_I32_SROA_SPECULATED]])
+; CHECK-NEXT:    br i1 [[CMP:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[ADDR_I8:%.*]] = phi ptr [ undef, [[ENTRY_THEN]] ], [ undef, [[ENTRY_ELSE]] ]
+; CHECK-NEXT:    call void @foo_i8(ptr [[ADDR_I8]])
+; CHECK-NEXT:    br i1 [[CMP]], label [[ENTRY_CONT_THEN:%.*]], label [[ENTRY_CONT_ELSE:%.*]]
+; CHECK:       entry.cont.then:
+; CHECK-NEXT:    br label [[ENTRY_CONT_CONT:%.*]]
+; CHECK:       entry.cont.else:
+; CHECK-NEXT:    br label [[ENTRY_CONT_CONT]]
+; CHECK:       entry.cont.cont:
+; CHECK-NEXT:    [[ADDR_I32:%.*]] = phi ptr [ undef, [[ENTRY_CONT_THEN]] ], [ undef, [[ENTRY_CONT_ELSE]] ]
+; CHECK-NEXT:    call void @foo_i32(ptr [[ADDR_I32]])
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %args = alloca [2 x %st.args], align 16
   %arr1 = getelementptr inbounds [2 x %st.args], ptr %args, i64 0, i64 1
   %sel = select i1 %cmp, ptr %arr1, ptr %args
   %addr = getelementptr inbounds %st.args, ptr %sel, i64 0, i32 1
   %addr.i8 = load ptr, ptr %addr, align 8
   call void @foo_i8(ptr %addr.i8)
   %addr.i32 = load ptr, ptr %addr, align 8
   call void @foo_i32 (ptr %addr.i32)
   ret void
 }
 
 declare void @foo_i8(ptr)
 declare void @foo_i32(ptr)
 
 ; Lifetime intrinsics should not prevent dereferenceability inferrence.
 define i32 @interfering_lifetime(ptr %data, i64 %indvars.iv) {
 ; CHECK-LABEL: @interfering_lifetime(
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV:%.*]]
 ; CHECK-NEXT:    [[I1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
 ; CHECK-NEXT:    [[CMP_I_I:%.*]] = icmp slt i32 [[I1]], 0
-; CHECK-NEXT:    [[I3_SROA_SPECULATE_LOAD_FALSE:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[I3_SROA_SPECULATED:%.*]] = select i1 [[CMP_I_I]], i32 0, i32 [[I3_SROA_SPECULATE_LOAD_FALSE]]
-; CHECK-NEXT:    ret i32 [[I3_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[CMP_I_I]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    [[I3_ELSE_VAL:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[I3:%.*]] = phi i32 [ 0, [[DOTTHEN]] ], [ [[I3_ELSE_VAL]], [[DOTELSE]] ]
+; CHECK-NEXT:    ret i32 [[I3]]
 ;
   %min = alloca i32, align 4
   %arrayidx = getelementptr inbounds i32, ptr %data, i64 %indvars.iv
   %i1 = load i32, ptr %arrayidx, align 4
   call void @llvm.lifetime.start.p0(i64 4, ptr %min)
   store i32 0, ptr %min, align 4
   %cmp.i.i = icmp slt i32 %i1, 0
   %__b.__a.i.i = select i1 %cmp.i.i, ptr %min, ptr %arrayidx
   %i3 = load i32, ptr %__b.__a.i.i, align 4
   ret i32 %i3
 }
 
 ; We should recursively evaluate select's.
 define i32 @clamp_load_to_constant_range(ptr %data, i64 %indvars.iv) {
 ; CHECK-LABEL: @clamp_load_to_constant_range(
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV:%.*]]
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MIN]])
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MAX]])
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
 ; CHECK-NEXT:    [[I1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
 ; CHECK-NEXT:    [[CMP_I_I:%.*]] = icmp slt i32 [[I1]], 0
 ; CHECK-NEXT:    [[I2:%.*]] = tail call i32 @llvm.smax.i32(i32 [[I1]], i32 0)
-; CHECK-NEXT:    [[__B___A_I_I:%.*]] = select i1 [[CMP_I_I]], ptr [[MIN]], ptr [[ARRAYIDX]]
 ; CHECK-NEXT:    [[CMP_I1_I:%.*]] = icmp ugt i32 [[I2]], 4095
-; CHECK-NEXT:    [[__B___A_I2_I:%.*]] = select i1 [[CMP_I1_I]], ptr [[MAX]], ptr [[__B___A_I_I]]
-; CHECK-NEXT:    [[I3:%.*]] = load i32, ptr [[__B___A_I2_I]], align 4
+; CHECK-NEXT:    br i1 [[CMP_I1_I]], label [[DOTTHEN:%.*]], label [[DOTELSE:%.*]]
+; CHECK:       .then:
+; CHECK-NEXT:    br label [[DOTCONT:%.*]]
+; CHECK:       .else:
+; CHECK-NEXT:    br i1 [[CMP_I_I]], label [[DOTELSE_THEN:%.*]], label [[DOTELSE_ELSE:%.*]]
+; CHECK:       .else.then:
+; CHECK-NEXT:    br label [[DOTELSE_CONT:%.*]]
+; CHECK:       .else.else:
+; CHECK-NEXT:    [[I3_ELSE_VAL_ELSE_VAL:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-NEXT:    br label [[DOTELSE_CONT]]
+; CHECK:       .else.cont:
+; CHECK-NEXT:    [[I3_ELSE_VAL:%.*]] = phi i32 [ 0, [[DOTELSE_THEN]] ], [ [[I3_ELSE_VAL_ELSE_VAL]], [[DOTELSE_ELSE]] ]
+; CHECK-NEXT:    br label [[DOTCONT]]
+; CHECK:       .cont:
+; CHECK-NEXT:    [[I3:%.*]] = phi i32 [ 4095, [[DOTTHEN]] ], [ [[I3_ELSE_VAL]], [[DOTELSE_CONT]] ]
 ; CHECK-NEXT:    ret i32 [[I3]]
 ;
   %min = alloca i32, align 4
   %max = alloca i32, align 4
   %arrayidx = getelementptr inbounds i32, ptr %data, i64 %indvars.iv
   call void @llvm.lifetime.start.p0(i64 4, ptr %min)
   store i32 0, ptr %min, align 4
   call void @llvm.lifetime.start.p0(i64 4, ptr %max)
   store i32 4095, ptr %max, align 4
   %i1 = load i32, ptr %arrayidx, align 4
   %cmp.i.i = icmp slt i32 %i1, 0
   %i2 = tail call i32 @llvm.smax.i32(i32 %i1, i32 0)
   %__b.__a.i.i = select i1 %cmp.i.i, ptr %min, ptr %arrayidx
   %cmp.i1.i = icmp ugt i32 %i2, 4095
   %__b.__a.i2.i = select i1 %cmp.i1.i, ptr %max, ptr %__b.__a.i.i
   %i3 = load i32, ptr %__b.__a.i2.i, align 4
   ret i32 %i3
 }
 
 define i32 @non_speculatable_load_of_select(i1 %cond, ptr %data) {
 ; CHECK-LABEL: @non_speculatable_load_of_select(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[DATA:%.*]], !prof [[PROF0:![0-9]+]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF0:![0-9]+]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    [[R_ELSE_VAL:%.*]] = load i32, ptr [[DATA:%.*]], align 4
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[R:%.*]] = phi i32 [ 0, [[ENTRY_THEN]] ], [ [[R_ELSE_VAL]], [[ENTRY_ELSE]] ]
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr = select i1 %cond, ptr %min, ptr %data, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_load_of_select_inverted(i1 %cond, ptr %data) {
 ; CHECK-LABEL: @non_speculatable_load_of_select_inverted(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[DATA:%.*]], ptr [[MAX]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF0]]
+; CHECK:       entry.then:
+; CHECK-NEXT:    [[R_THEN_VAL:%.*]] = load i32, ptr [[DATA:%.*]], align 4
+; CHECK-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK:       entry.else:
+; CHECK-NEXT:    br label [[ENTRY_CONT]]
+; CHECK:       entry.cont:
+; CHECK-NEXT:    [[R:%.*]] = phi i32 [ [[R_THEN_VAL]], [[ENTRY_THEN]] ], [ 4095, [[ENTRY_ELSE]] ]
 ; CHECK-NEXT:    ret i32 [[R]]
 ;
 entry:
   %max = alloca i32, align 4
   store i32 4095, ptr %max, align 4
   %addr = select i1 %cond, ptr %data, ptr %max, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 
 !0  = !{!"branch_weights", i32 1,  i32 99}
 
 ; Ensure that the branch metadata is reversed to match the reversals above.
-; CHECK: !0 = {{.*}} i32 1, i32 99}
+; CHECK: [[PROF0]] = {{.*}} i32 1, i32 99}
 
 declare void @llvm.lifetime.start.p0(i64, ptr )
 declare void @llvm.lifetime.end.p0(i64, ptr)
 declare i32 @llvm.smax.i32(i32, i32)

llvm/test/Transforms/SROA/std-clamp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=sroa -S | FileCheck %s
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 
 define float @_Z8stdclampfff(float %x, float %lo, float %hi) {
 ; CHECK-LABEL: @_Z8stdclampfff(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[I4:%.*]] = alloca float, align 4
 ; CHECK-NEXT:    store float [[HI:%.*]], ptr [[I4]], align 4
 ; CHECK-NEXT:    [[I5:%.*]] = fcmp fast olt float [[X:%.*]], [[LO:%.*]]
 ; CHECK-NEXT:    [[I6:%.*]] = fcmp fast olt float [[HI]], [[X]]
-; CHECK-NEXT:    [[I9_SROA_SPECULATE_LOAD_FALSE_SROA_SPECULATE_LOAD_TRUE:%.*]] = load float, ptr [[I4]], align 4
-; CHECK-NEXT:    [[I9_SROA_SPECULATE_LOAD_FALSE_SROA_SPECULATED:%.*]] = select i1 [[I6]], float [[I9_SROA_SPECULATE_LOAD_FALSE_SROA_SPECULATE_LOAD_TRUE]], float [[X]]
-; CHECK-NEXT:    [[I9_SROA_SPECULATED:%.*]] = select i1 [[I5]], float [[LO]], float [[I9_SROA_SPECULATE_LOAD_FALSE_SROA_SPECULATED]]
-; CHECK-NEXT:    ret float [[I9_SROA_SPECULATED]]
+; CHECK-NEXT:    br i1 [[I5]], label [[BB_THEN:%.*]], label [[BB_ELSE:%.*]]
+; CHECK:       bb.then:
+; CHECK-NEXT:    br label [[BB_CONT:%.*]]
+; CHECK:       bb.else:
+; CHECK-NEXT:    br i1 [[I6]], label [[BB_ELSE_THEN:%.*]], label [[BB_ELSE_ELSE:%.*]]
+; CHECK:       bb.else.then:
+; CHECK-NEXT:    [[I9_ELSE_VAL_THEN_VAL:%.*]] = load float, ptr [[I4]], align 4
+; CHECK-NEXT:    br label [[BB_ELSE_CONT:%.*]]
+; CHECK:       bb.else.else:
+; CHECK-NEXT:    br label [[BB_ELSE_CONT]]
+; CHECK:       bb.else.cont:
+; CHECK-NEXT:    [[I9_ELSE_VAL:%.*]] = phi float [ [[I9_ELSE_VAL_THEN_VAL]], [[BB_ELSE_THEN]] ], [ [[X]], [[BB_ELSE_ELSE]] ]
+; CHECK-NEXT:    br label [[BB_CONT]]
+; CHECK:       bb.cont:
+; CHECK-NEXT:    [[I9:%.*]] = phi float [ [[LO]], [[BB_THEN]] ], [ [[I9_ELSE_VAL]], [[BB_ELSE_CONT]] ]
+; CHECK-NEXT:    ret float [[I9]]
 ;
 bb:
   %i = alloca float, align 4
   %i3 = alloca float, align 4
   %i4 = alloca float, align 4
   store float %x, ptr %i, align 4
   store float %lo, ptr %i3, align 4
   store float %hi, ptr %i4, align 4
   %i5 = fcmp fast olt float %x, %lo
   %i6 = fcmp fast olt float %hi, %x
   %i7 = select i1 %i6, ptr %i4, ptr %i
   %i8 = select i1 %i5, ptr %i3, ptr %i7
   %i9 = load float, ptr %i8, align 4
   ret float %i9
 }