[SROA] For non-speculatable `load`s of `select`s -- split block, insert then/else blocks, form two-entry PHI node

[lebedev.ri]

Summary

Currently, SROA is CFG-preserving.
Not doing so does not affect any pipeline test. (???)
Internally, SROA requires Dominator Tree, and uses it solely for the final -mem2reg call.

By design, we can't really SROA alloca if their address escapes somehow,
but we have logic to deal with load of select/PHI,
where at least one of the possible addresses prevents promotion,
by speculating the loads and selecting between loaded values.

As one would expect, that requires ensuring that the speculation is actually legal.
Even ignoring complexity bailouts, that logic does not deal with everything,
e.g. isSafeToLoadUnconditionally() does not recurse into hands of select.
There can also be cases where the load is genuinely non-speculate.

So if we can't prove that the load can be speculated,
unfold the select, produce two-entry phi node, and perform predicated load.

Now, that transformation must obviously update Dominator Tree,
since we require it later on. Doing so is trivial.
Additionally, we don't want to do this for the final SROA invocation (D136806).

In the end, this ends up having negative (!) compile-time cost:
https://llvm-compile-time-tracker.com/compare.php?from=c6d7e80ec4c17a415673b1cfd25924f98ac83608&to=ddf9600365093ea50d7e278696cbfa01641c959d&stat=instructions:u

Though indeed, this only deals with selects, PHIs are still using speculation.

Should we update some more analysis?

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

Are there any cases where this should still try to speculate?

llvm/lib/Transforms/Scalar/SROA.cpp

1417

Single quote newline

[lebedev.ri]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

I'm not sure i understand the question.
As it can be seen from the rest of the diff here,
for select's, we will always form diamond control flow and two-entry phi node,
and never check if we can speculate load.
For PHI's, we still speculate.

[lebedev.ri]

llvm/include/llvm/Transforms/Scalar/SROA.h

106–109

One improvement over this we can do, is to cache
all of the pointers derived from the current alloca,
and speculate the loads that belong to that set.
That might be cheap to do. Is that what you had in mind?

We also should probably report CFG as preserved unless we actually changed it.

[lebedev.ri]

I'm guessing we should keep the old approach at least for the new last SROA run.
Not quite sure about PHI handling yet.

[lebedev.ri]

Okay, only modifying CFG if we must is even better.
Please take a look :)

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

92

Move to last field?

llvm/lib/Passes/PassBuilder.cpp

840

I thought pass options were-spelled-like-this?

llvm/lib/Transforms/Scalar/SROA.cpp

1417–1418

I know it was already like this, but can't you just do CreateAlignedLoad to start?

1487

Missing check if CFG modify is allowed? Could also move this out to where this is called and check the return value (which it looks like is already there)

4961–4963

Why not just OS << (PreserveCFG ? "<PreserveCFG>" : "<ModifyCFG>";

llvm/test/Transforms/SROA/2009-02-20-InstCombine-SROA.ll

274–276

Just remove the prefixes?

[lebedev.ri]

@arsenm thank you for taking a look!
I believe, this addresses your comments.

llvm/include/llvm/Transforms/Scalar/SROA.h

92

I'm guessing you meant "to the end of this var decl group?"

llvm/lib/Passes/PassBuilder.cpp

840

Meh. Okay.

llvm/lib/Transforms/Scalar/SROA.cpp

1487

No, nothing is missing, this is just defensive coding.

If we can't modify cfg, then we really won't get here,
and this just confirms that invariant.

E.g. we won't pass DTU into call to rewriteSelectInstLoads()
if we are not allowed to perform CFG changes.

And earlier, see if (PreserveCFG) return {}; bailouts in isSafeSelectToSpeculate().

llvm/test/Transforms/SROA/2009-02-20-InstCombine-SROA.ll

274–276

Unless you really insist, i'd like to keep this as is.
If i remove the prefixes,

1. the tests become unconsistent. because currently they all have the same run lines and check prefixes
2. Should the run lines start producing different outputs, without this, the output will get silently lost, and requires manual changes to unbreak

The latter is really painful in e.g. codegen tests.
I would really really prefer to avoid creating that problem.

[lebedev.ri]

Add a few more tests (non-simple loads, select-of-select).
mem2reg bails on volatile (but not atomic!) loads,
so we shouldn't form predicated loads for them.

[arsenm]

llvm/include/llvm/Transforms/Scalar/SROA.h

140

Use opaque pointer example?

llvm/lib/Passes/PassRegistry.def

481

outdated option names?

[lebedev.ri]

@arsenm thank you for taking a look!
Addressing comments.

[nikic]

llvm/include/llvm/Transforms/Scalar/SROA.h

24

nit: Wrong include style

llvm/lib/Passes/PassBuilder.cpp

849

Do I understand correctly that this will prevent doing a plain -passes=sroa run? If so, please pick one of these as a default -- I don't really care which one, but people who just want to clean up some IR with SROA certainly don't need to know about these details. This would also violate existing practice for all other passes.

[lebedev.ri]

@arsenm @nikic thank you for the review!

[uabelho]

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

[lebedev.ri]

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

[uabelho]

In D138238#3991889, @lebedev.ri wrote:

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

Yeah, llvm isn't clean of gcc warnings.
We compile with gcc downstream and used -Werror at first but had to give up that since there are no public bots running with it.
Now we just notice when new warnings pop up instead.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

Our out-of-tree backend is asserting in this function call (particularly at assert(cast<PointerType>(Ptr->getType())->isOpaqueOrPointeeTypeMatches(Ty)) which occurs while creating the load). I suspect this is due to the lost bitcasts, since when I add them back there's no assert.

The code I have adding the bitcasts back looks like the following:

IRB.SetInsertPoint(&LI);

// OUT OF TREE
if (SI.hasOneUse()) {
  if (BitCastInst *BC = dyn_cast<BitCastInst>(SI.user_back())) {
    // assert(BC->user_back() == &LI); <- unclear if this is guaranteed
    TV = IRB.CreateBitCast(TV, BC->getType(), TV->getName() + ".sroa.cast");
    FV = IRB.CreateBitCast(FV, BC->getType(), FV->getName() + ".sroa.cast");
  }
}
// END OUT OF TREE

LoadInst *TL = ...

To be honest, I have not entirely analyzed this patch, so I'm not confident that the way I have added the bitcasts back is legal (especially now that there's no while loop on SI's uses...). Could you advise if this is legal/correct (to unblock us)? I can try to get an anonymized test case for this, so a full fix can be made.

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

We are literally days away from stopping accepting such fixes.
That being said, let me fix this one...

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

9f27f4536e19e93349b0662338408efe6d1cb2fd, but it is probably one of last ones of it's kind.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

Thanks for the quick turn around

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

Sorry, that was too quick and it got reverted.

@nikic please can you estimate when we switch to not dealing with typed pointer bugs?

[nikic]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

Here's my proposal for a hard timeline: https://reviews.llvm.org/D140487 (tl;dr supported until release/16.x branch creation on Jan 24th).

[lebedev.ri]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

2cb393590ea537b06fa66e6d847a1159c227a313, looks like it stuck this time around.

[mkitzan]

llvm/lib/Transforms/Scalar/SROA.cpp

1409

Haha no worries. Glad it's finally landed 👍

[brunodf]

Hi,

In D138238#3991889, @lebedev.ri wrote:

In D138238#3991171, @uabelho wrote:

Hello,

I noticed that when compiling with gcc I get the following new warning with this patch:

00:14:51 ../include/llvm/Transforms/Scalar/SROA.h:95:7: warning: 'llvm::SROAPass' declared with greater visibility than the type of its field 'llvm::SROAPass::SelectsToRewrite' [-Wattributes]
00:14:51    95 | class SROAPass : public PassInfoMixin<SROAPass> {
00:14:51       |       ^~~~~~~~

I guess it's because

using PossiblySpeculatableLoad =
    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;

are declared inside

namespace sroa LLVM_LIBRARY_VISIBILITY {

but then PossiblySpeculatableLoads is used in the SROAPass member

SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
    SelectsToRewrite;

I've no idea if this is anythinig to care about though?

Yeah, @arsenm told me about that. I'm not sure about this diagnostic.
What really worries me though, is that we don't appear to have GCC -Werror bots?

This warning is still present in current builds using GCC, also using recent GCC versions, see this buildbot:

/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/include/llvm/Transforms/Scalar/SROA.h:96:7: warning: ‘llvm::SROAPass’ declared with greater visibility than the type of its field ‘llvm::SROAPass::SelectsToRewrite’ [-Wattributes]

The warning does seem to point at a genuine problem. The intent of the sroa namespace is to hide details from clients (possibly in other modules), see here.

/// A private "module" namespace for types and utilities used by SROA. These
/// are implementation details and should not be used by clients.
namespace LLVM_LIBRARY_VISIBILITY sroa {

But because the SROAPass has the field SelectsToRewrite, which contains sroa::PossiblySpeculatableLoads values, it is required to know these details to calculate its size, determine its layout, etc.

Would it be an option to put PossiblySpeculatableLoads (and its constituents) in another namespace (for example sroa_detail), with normal visibility? Or can we store the SelectsToRewrite as a unique_ptr rather than as the map vector directly (pimpl pattern)? The latter would allow to remove the details of PossiblySpeculatableLoads (and its constituents) from the SROA.h header, but would require to follow an extra indirection to access SelectsToRewrite.

Thanks!

[nikic]

@brunodf Ideally, we'd move the SROA implementation into a separate class in the cpp file and only export a thin NewPM wrapper around it in the header.

[brunodf]

In D138238#4656717, @nikic wrote:

@brunodf Ideally, we'd move the SROA implementation into a separate class in the cpp file and only export a thin NewPM wrapper around it in the header.

This is a pull request implementing the reorganization suggested by @nikic : https://github.com/llvm/llvm-project/pull/72846
It solves the GCC warning obviously, but it is also a better organization, it can do away with the llvm::sroa namespace entirely, etc.
If you can, please review it.

llvm/include/llvm/Transforms/Scalar.h

 // remove computations of dead bits.
 //
 FunctionPass *createBitTrackingDCEPass();
 
 //===----------------------------------------------------------------------===//
 //
 // SROA - Replace aggregates or pieces of aggregates with scalar SSA values.
 //
-FunctionPass *createSROAPass();
+FunctionPass *createSROAPass(bool PreserveCFG = true);
 
 //===----------------------------------------------------------------------===//
 //
 // InductiveRangeCheckElimination - Transform loops to elide range checks on
 // linear functions of the induction variable.
 //
 Pass *createInductiveRangeCheckEliminationPass();
 

llvm/include/llvm/Transforms/Scalar/SROA.h

-//===- SROA.h - Scalar Replacement Of Aggregates ----------------*- C++ -*-===//
+//===- SROA.h - Scalar Replacement Of Aggregates ----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file provides the interface for LLVM's Scalar Replacement of
 /// Aggregates pass. This pass provides both aggregate splitting and the
 /// primary SSA formation used in the compiler.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_SCALAR_SROA_H
 #define LLVM_TRANSFORMS_SCALAR_SROA_H
 
+#include "llvm/ADT/MapVector.h"
+#include "llvm/ADT/PointerIntPair.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/ValueHandle.h"
 #include <vector>
 
 namespace llvm {
 
 class AllocaInst;
+class LoadInst;
 class AssumptionCache;
 class DominatorTree;
+class DomTreeUpdater;
 class Function;
 class LLVMContext;
 class PHINode;
 class SelectInst;
 class Use;
 
 /// A private "module" namespace for types and utilities used by SROA. These
 /// are implementation details and should not be used by clients.
 namespace sroa LLVM_LIBRARY_VISIBILITY {
 
 class AllocaSliceRewriter;
 class AllocaSlices;
 class Partition;
 class SROALegacyPass;
 
+class SelectHandSpeculativity {
+  unsigned char Storage = 0;
+  using TrueVal = Bitfield::Element<bool, 0, 1>;  // Low 0'th bit.
+  using FalseVal = Bitfield::Element<bool, 1, 1>; // Low 1'th bit.
+public:
+  SelectHandSpeculativity() = default;
+  SelectHandSpeculativity &setAsSpeculatable(bool isTrueVal);
+  bool isSpeculatable(bool isTrueVal) const;
+  bool areAllSpeculatable() const;
+  bool areAnySpeculatable() const;
+  bool areNoneSpeculatable() const;
+  // For interop as int half of PointerIntPair.
+  explicit operator intptr_t() const { return static_cast<intptr_t>(Storage); }
+  explicit SelectHandSpeculativity(intptr_t Storage_) : Storage(Storage_) {}
+};
+static_assert(sizeof(SelectHandSpeculativity) == sizeof(unsigned char));
+
+using PossiblySpeculatableLoad =
+    PointerIntPair<LoadInst *, 2, sroa::SelectHandSpeculativity>;
+using PossiblySpeculatableLoads = SmallVector<PossiblySpeculatableLoad, 2>;
+
 } // end namespace sroa
 
+enum class SROAOptions : bool { ModifyCFG, PreserveCFG };
+
 /// An optimization pass providing Scalar Replacement of Aggregates.
 ///
 /// This pass takes allocations which can be completely analyzed (that is, they
 /// don't escape) and tries to turn them into scalar SSA values. There are
 /// a few steps to this process.
 ///
 /// 1) It takes allocations of aggregates and analyzes the ways in which they
 ///    are used to try to split them into smaller allocations, ideally of
 ///    a single scalar data type. It will split up memcpy and memset accesses
 ///    as necessary and try to isolate individual scalar accesses.
 /// 2) It will transform accesses into forms which are suitable for SSA value
 ///    promotion. This can be replacing a memset with a scalar store of an
 ///    integer value, or it can involve speculating operations on a PHI or
 ///    select to be a PHI or select of the results.
 /// 3) Finally, this will try to detect a pattern of accesses which map cleanly
 ///    onto insert and extract operations on a vector value, and convert them to
 ///    this form. By doing so, it will enable promotion of vector aggregates to
 ///    SSA vector values.
 class SROAPass : public PassInfoMixin<SROAPass> {
   LLVMContext *C = nullptr;
-  DominatorTree *DT = nullptr;
+  DomTreeUpdater *DTU = nullptr;
   AssumptionCache *AC = nullptr;
+  const bool PreserveCFG;
 
   /// Worklist of alloca instructions to simplify.
   ///
   /// Each alloca in the function is added to this. Each new alloca formed gets
   /// added to it as well to recursively simplify unless that alloca can be
   /// directly promoted. Finally, each time we rewrite a use of an alloca other
   /// the one being actively rewritten, we add it back onto the list if not
   /// already present to ensure it is re-visited.
class SROAPass : public PassInfoMixin<SROAPass> {
   /// A collection of alloca instructions we can directly promote.
   std::vector<AllocaInst *> PromotableAllocas;
 
   /// A worklist of PHIs to speculate prior to promoting allocas.
   ///
   /// All of these PHIs have been checked for the safety of speculation and by
   /// being speculated will allow promoting allocas currently in the promotable
   /// queue.
-  SetVector<PHINode *, SmallVector<PHINode *, 2>> SpeculatablePHIs;
+  SetVector<PHINode *, SmallVector<PHINode *, 8>> SpeculatablePHIs;
 
-  /// A worklist of select instructions to speculate prior to promoting
+  /// A worklist of select instructions to rewrite prior to promoting
   /// allocas.
-  ///
-  /// All of these select instructions have been checked for the safety of
-  /// speculation and by being speculated will allow promoting allocas
-  /// currently in the promotable queue.
-  SetVector<SelectInst *, SmallVector<SelectInst *, 2>> SpeculatableSelects;
+  SmallMapVector<SelectInst *, sroa::PossiblySpeculatableLoads, 8>
+      SelectsToRewrite;
+
+  /// Select instructions that use an alloca and are subsequently loaded can be
+  /// rewritten to load both input pointers and then select between the result,
+  /// allowing the load of the alloca to be promoted.
+  /// From this:
+  ///   %P2 = select i1 %cond, ptr %Alloca, ptr %Other
+  ///   %V = load <type>, ptr %P2
+  /// to:
+  ///   %V1 = load <type>, ptr %Alloca      -> will be mem2reg'd
+  ///   %V2 = load <type>, ptr %Other
+  ///   %V = select i1 %cond, <type> %V1, <type> %V2
+  ///
+  /// We can do this to a select if its only uses are loads
+  /// and if either the operand to the select can be loaded unconditionally,
+  ///        or if we are allowed to perform CFG modifications.
+  /// If found an intervening bitcast with a single use of the load,
+  /// allow the promotion.
+  static std::optional<sroa::PossiblySpeculatableLoads>
+  isSafeSelectToSpeculate(SelectInst &SI, bool PreserveCFG);
 
 public:
-  SROAPass() = default;
+  /// If \p PreserveCFG is set, then the pass is not allowed to modify CFG
+  /// in any way, even if it would update CFG analyses.
+  SROAPass(SROAOptions PreserveCFG);
 
   /// Run the pass over the function.
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 
+  void printPipeline(raw_ostream &OS,
+                     function_ref<StringRef(StringRef)> MapClassName2PassName);
+
 private:
   friend class sroa::AllocaSliceRewriter;
   friend class sroa::SROALegacyPass;
 
   /// Helper used by both the public run method and by the legacy pass.
+  PreservedAnalyses runImpl(Function &F, DomTreeUpdater &RunDTU,
+                            AssumptionCache &RunAC);
   PreservedAnalyses runImpl(Function &F, DominatorTree &RunDT,
                             AssumptionCache &RunAC);
 
   bool presplitLoadsAndStores(AllocaInst &AI, sroa::AllocaSlices &AS);
   AllocaInst *rewritePartition(AllocaInst &AI, sroa::AllocaSlices &AS,
                                sroa::Partition &P);
   bool splitAlloca(AllocaInst &AI, sroa::AllocaSlices &AS);
-  bool runOnAlloca(AllocaInst &AI);
+  std::pair<bool /*Changed*/, bool /*CFGChanged*/> runOnAlloca(AllocaInst &AI);
   void clobberUse(Use &U);
   bool deleteDeadInstructions(SmallPtrSetImpl<AllocaInst *> &DeletedAllocas);
   bool promoteAllocas(Function &F);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_SCALAR_SROA_H

llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h

 /// After:
 ///   Head
 ///   if (Cond)
 ///     ThenBlock
 ///   else
 ///     ElseBlock
 ///   SplitBefore
 ///   Tail
+///
+/// Updates DT if given.
 void SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
                                    Instruction **ThenTerm,
                                    Instruction **ElseTerm,
-                                   MDNode *BranchWeights = nullptr);
+                                   MDNode *BranchWeights = nullptr,
+                                   DomTreeUpdater *DTU = nullptr);
 
 /// Check whether BB is the merge point of a if-region.
 /// If so, return the branch instruction that determines which entry into
 /// BB will be taken.  Also, return by references the block that will be
 /// entered from if the condition is true, and the block that will be
 /// entered if the condition is false.
 ///
 /// This does no checking to see if the true/false blocks have large or unsavory

llvm/lib/Passes/PassBuilder.cpp

if (ParamName == "pre") {
       return make_error<StringError>(
           formatv("invalid GVN pass parameter '{0}' ", ParamName).str(),
           inconvertibleErrorCode());
     }
   }
   return Result;
 }
 
+Expected<SROAOptions> parseSROAOptions(StringRef Params) {
+  if (Params.empty() || Params == "modify-cfg")
+    return SROAOptions::ModifyCFG;
+  if (Params == "preserve-cfg")
+    return SROAOptions::PreserveCFG;
+  return make_error<StringError>(
+      formatv("invalid SROA pass parameter '{0}' (either preserve-cfg or "
+              "modify-cfg can be specified)",
+              Params)
+          .str(),
+      inconvertibleErrorCode());
+}
+
 Expected<StackLifetime::LivenessType>
 parseStackLifetimeOptions(StringRef Params) {
   StackLifetime::LivenessType Result = StackLifetime::LivenessType::May;
   while (!Params.empty()) {
     StringRef ParamName;
     std::tie(ParamName, Params) = Params.split(';');
 
     if (ParamName == "may") {

llvm/lib/Passes/PassBuilderPipelines.cpp

 FunctionPassManager
 PassBuilder::buildO1FunctionSimplificationPipeline(OptimizationLevel Level,
                                                    ThinOrFullLTOPhase Phase) {
 
   FunctionPassManager FPM;
 
   // Form SSA out of local memory accesses after breaking apart aggregates into
   // scalars.
-  FPM.addPass(SROAPass());
+  FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
 
   // Catch trivial redundancies
   FPM.addPass(EarlyCSEPass(true /* Enable mem-ssa. */));
 
   // Hoisting of scalars and load expressions.
   FPM.addPass(
       SimplifyCFGPass(SimplifyCFGOptions().convertSwitchRangeToICmp(true)));
   FPM.addPass(InstCombinePass());
PassBuilder::buildO1FunctionSimplificationPipeline(OptimizationLevel Level,
   FPM.addPass(InstCombinePass());
   // The loop passes in LPM2 (LoopFullUnrollPass) do not preserve MemorySSA.
   // *All* loop passes must preserve it, in order to be able to use it.
   FPM.addPass(createFunctionToLoopPassAdaptor(std::move(LPM2),
                                               /*UseMemorySSA=*/false,
                                               /*UseBlockFrequencyInfo=*/false));
 
   // Delete small array after loop unroll.
-  FPM.addPass(SROAPass());
+  FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
 
   // Specially optimize memory movement as it doesn't look like dataflow in SSA.
   FPM.addPass(MemCpyOptPass());
 
   // Sparse conditional constant propagation.
   // FIXME: It isn't clear why we do this *after* loop passes rather than
   // before...
   FPM.addPass(SCCPPass());
PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
   // construction readability.
   if (Level.getSpeedupLevel() == 1)
     return buildO1FunctionSimplificationPipeline(Level, Phase);
 
   FunctionPassManager FPM;
 
   // Form SSA out of local memory accesses after breaking apart aggregates into
   // scalars.
-  FPM.addPass(SROAPass());
+  FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
 
   // Catch trivial redundancies
   FPM.addPass(EarlyCSEPass(true /* Enable mem-ssa. */));
   if (EnableKnowledgeRetention)
     FPM.addPass(AssumeSimplifyPass());
 
   // Hoisting of scalars and load expressions.
   if (EnableGVNHoist)
PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
   // The loop passes in LPM2 (LoopIdiomRecognizePass, IndVarSimplifyPass,
   // LoopDeletionPass and LoopFullUnrollPass) do not preserve MemorySSA.
   // *All* loop passes must preserve it, in order to be able to use it.
   FPM.addPass(createFunctionToLoopPassAdaptor(std::move(LPM2),
                                               /*UseMemorySSA=*/false,
                                               /*UseBlockFrequencyInfo=*/false));
 
   // Delete small array after loop unroll.
-  FPM.addPass(SROAPass());
+  FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
 
   // Try vectorization/scalarization transforms that are both improvements
   // themselves and can allow further folds with GVN and InstCombine.
   FPM.addPass(VectorCombinePass(/*TryEarlyFoldsOnly=*/true));
 
   // Eliminate redundancies.
   FPM.addPass(MergedLoadStoreMotionPass());
   if (RunNewGVN)
if (!IsCS && !DisablePreInliner) {
     // FIXME: this comment is cargo culted from the old pass manager, revisit).
     IP.HintThreshold = Level.isOptimizingForSize() ? PreInlineThreshold : 325;
     ModuleInlinerWrapperPass MIWP(
         IP, /* MandatoryFirst */ true,
         InlineContext{LTOPhase, InlinePass::EarlyInliner});
     CGSCCPassManager &CGPipeline = MIWP.getPM();
 
     FunctionPassManager FPM;
-    FPM.addPass(SROAPass());
+    FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
     FPM.addPass(EarlyCSEPass()); // Catch trivial redundancies.
     FPM.addPass(SimplifyCFGPass(SimplifyCFGOptions().convertSwitchRangeToICmp(
         true)));                    // Merge & remove basic blocks.
     FPM.addPass(InstCombinePass()); // Combine silly sequences.
     invokePeepholeEPCallbacks(FPM, Level);
 
     CGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
         std::move(FPM), PTO.EagerlyInvalidateAnalyses));
PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
 
   // Create an early function pass manager to cleanup the output of the
   // frontend.
   FunctionPassManager EarlyFPM;
   // Lower llvm.expect to metadata before attempting transforms.
   // Compare/branch metadata may alter the behavior of passes like SimplifyCFG.
   EarlyFPM.addPass(LowerExpectIntrinsicPass());
   EarlyFPM.addPass(SimplifyCFGPass());
-  EarlyFPM.addPass(SROAPass());
+  EarlyFPM.addPass(SROAPass(SROAOptions::ModifyCFG));
   EarlyFPM.addPass(EarlyCSEPass());
   if (Level == OptimizationLevel::O3)
     EarlyFPM.addPass(CallSiteSplittingPass());
 
   // In SamplePGO ThinLTO backend, we need instcombine before profile annotation
   // to convert bitcast to direct calls so that they can be inlined during the
   // profile annotation prepration step.
   // More details about SamplePGO design can be found in:
if (EnableUnrollAndJam && PTO.LoopUnrolling)
           LoopUnrollAndJamPass(Level.getSpeedupLevel())));
     FPM.addPass(LoopUnrollPass(LoopUnrollOptions(
         Level.getSpeedupLevel(), /*OnlyWhenForced=*/!PTO.LoopUnrolling,
         PTO.ForgetAllSCEVInLoopUnroll)));
     FPM.addPass(WarnMissedTransformationsPass());
     // Now that we are done with loop unrolling, be it either by LoopVectorizer,
     // or LoopUnroll passes, some variable-offset GEP's into alloca's could have
     // become constant-offset, thus enabling SROA and alloca promotion. Do so.
-    FPM.addPass(SROAPass());
+    // NOTE: we are very late in the pipeline, and we don't have any LICM
+    // or SimplifyCFG passes scheduled after us, that would cleanup
+    // the CFG mess this may created if allowed to modify CFG, so forbid that.
+    FPM.addPass(SROAPass(SROAOptions::PreserveCFG));
   }
 
   if (!IsFullLTO) {
     // Eliminate loads by forwarding stores from the previous iteration to loads
     // of the current iteration.
     FPM.addPass(LoopLoadEliminationPass());
   }
   // Cleanup after the loop optimization passes.
if (!IsFullLTO) {
     }
     FPM.addPass(LoopUnrollPass(LoopUnrollOptions(
         Level.getSpeedupLevel(), /*OnlyWhenForced=*/!PTO.LoopUnrolling,
         PTO.ForgetAllSCEVInLoopUnroll)));
     FPM.addPass(WarnMissedTransformationsPass());
     // Now that we are done with loop unrolling, be it either by LoopVectorizer,
     // or LoopUnroll passes, some variable-offset GEP's into alloca's could have
     // become constant-offset, thus enabling SROA and alloca promotion. Do so.
-    FPM.addPass(SROAPass());
+    // NOTE: we are very late in the pipeline, and we don't have any LICM
+    // or SimplifyCFG passes scheduled after us, that would cleanup
+    // the CFG mess this may created if allowed to modify CFG, so forbid that.
+    FPM.addPass(SROAPass(SROAOptions::PreserveCFG));
     FPM.addPass(InstCombinePass());
     FPM.addPass(
         RequireAnalysisPass<OptimizationRemarkEmitterAnalysis, Function>());
     FPM.addPass(createFunctionToLoopPassAdaptor(
         LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap,
                  /*AllowSpeculation=*/true),
         /*UseMemorySSA=*/true, /*UseBlockFrequencyInfo=*/true));
   }
if (PGOOpt) {
     else if (PGOOpt->CSAction == PGOOptions::CSIRUse)
       addPGOInstrPasses(MPM, Level, /* RunProfileGen */ false,
                         /* IsCS */ true, PGOOpt->ProfileFile,
                         PGOOpt->ProfileRemappingFile,
                         ThinOrFullLTOPhase::FullLTOPostLink);
   }
 
   // Break up allocas
-  FPM.addPass(SROAPass());
+  FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
 
   // LTO provides additional opportunities for tailcall elimination due to
   // link-time inlining, and visibility of nocapture attribute.
   FPM.addPass(TailCallElimPass());
 
   // Run a few AA driver optimizations here and now to cleanup the code.
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM),
                                                 PTO.EagerlyInvalidateAnalyses));

llvm/lib/Passes/PassRegistry.def

 FUNCTION_PASS("scalarize-masked-mem-intrin", ScalarizeMaskedMemIntrinPass())
 FUNCTION_PASS("scalarizer", ScalarizerPass())
 FUNCTION_PASS("separate-const-offset-from-gep", SeparateConstOffsetFromGEPPass())
 FUNCTION_PASS("sccp", SCCPPass())
 FUNCTION_PASS("sink", SinkingPass())
 FUNCTION_PASS("slp-vectorizer", SLPVectorizerPass())
 FUNCTION_PASS("slsr", StraightLineStrengthReducePass())
 FUNCTION_PASS("speculative-execution", SpeculativeExecutionPass())
-FUNCTION_PASS("sroa", SROAPass())
 FUNCTION_PASS("strip-gc-relocates", StripGCRelocates())
 FUNCTION_PASS("structurizecfg", StructurizeCFGPass())
 FUNCTION_PASS("tailcallelim", TailCallElimPass())
 FUNCTION_PASS("unify-loop-exits", UnifyLoopExitsPass())
 FUNCTION_PASS("vector-combine", VectorCombinePass())
 FUNCTION_PASS("verify", VerifierPass())
 FUNCTION_PASS("verify<domtree>", DominatorTreeVerifierPass())
 FUNCTION_PASS("verify<loops>", LoopVerifierPass())
FUNCTION_PASS_WITH_PARAMS("gvn",
                            [](GVNOptions Opts) {
                              return GVNPass(Opts);
                            },
                           parseGVNOptions,
                           "no-pre;pre;"
                           "no-load-pre;load-pre;"
                           "no-split-backedge-load-pre;split-backedge-load-pre;"
                           "no-memdep;memdep")
+FUNCTION_PASS_WITH_PARAMS("sroa",
+                          "SROAPass",
+                          [](SROAOptions PreserveCFG) {
+                            return SROAPass(PreserveCFG);
+                          },
+                          parseSROAOptions,
+                          "preserve-cfg;modify-cfg")
 FUNCTION_PASS_WITH_PARAMS("print<stack-lifetime>",
                           "StackLifetimePrinterPass",
                            [](StackLifetime::LivenessType Type) {
                              return StackLifetimePrinterPass(dbgs(), Type);
                            },
                           parseStackLifetimeOptions,
                           "may;must")
 FUNCTION_PASS_WITH_PARAMS("print<da>",

llvm/lib/Transforms/Scalar/SROA.cpp

 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Analysis/AssumptionCache.h"
+#include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/Loads.h"
 #include "llvm/Analysis/PtrUseVisitor.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/ConstantFolder.h"
 #include "llvm/IR/Constants.h"

 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Scalar.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Utils/PromoteMemToReg.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <cstring>
 #include <iterator>

 STATISTIC(NumAllocasAnalyzed, "Number of allocas analyzed for replacement");
 STATISTIC(NumAllocaPartitions, "Number of alloca partitions formed");
 STATISTIC(MaxPartitionsPerAlloca, "Maximum number of partitions per alloca");
 STATISTIC(NumAllocaPartitionUses, "Number of alloca partition uses rewritten");
 STATISTIC(MaxUsesPerAllocaPartition, "Maximum number of uses of a partition");
 STATISTIC(NumNewAllocas, "Number of new, smaller allocas introduced");
 STATISTIC(NumPromoted, "Number of allocas promoted to SSA values");
 STATISTIC(NumLoadsSpeculated, "Number of loads speculated to allow promotion");
+STATISTIC(NumLoadsPredicated,
+          "Number of loads rewritten into predicated loads to allow promotion");
 STATISTIC(NumDeleted, "Number of instructions deleted");
 STATISTIC(NumVectorized, "Number of vectorized aggregates");
 
 /// Hidden option to experiment with completely strict handling of inbounds
 /// GEPs.
 static cl::opt<bool> SROAStrictInbounds("sroa-strict-inbounds", cl::init(false),
                                         cl::Hidden);
-
 namespace {
 
 /// A custom IRBuilder inserter which prefixes all names, but only in
 /// Assert builds.
 class IRBuilderPrefixedInserter final : public IRBuilderDefaultInserter {
   std::string Prefix;
 
   Twine getNameWithPrefix(const Twine &Name) const {
for (unsigned Idx = 0, Num = PN.getNumIncomingValues(); Idx != Num; ++Idx) {
     NewPN->addIncoming(Load, Pred);
     InjectedLoads[Pred] = Load;
   }
 
   LLVM_DEBUG(dbgs() << "          speculated to: " << *NewPN << "\n");
   PN.eraseFromParent();
 }
 
-/// Select instructions that use an alloca and are subsequently loaded can be
-/// rewritten to load both input pointers and then select between the result,
-/// allowing the load of the alloca to be promoted.
-/// From this:
-///   %P2 = select i1 %cond, i32* %Alloca, i32* %Other
-///   %V = load i32* %P2
-/// to:
-///   %V1 = load i32* %Alloca      -> will be mem2reg'd
-///   %V2 = load i32* %Other
-///   %V = select i1 %cond, i32 %V1, i32 %V2
-///
-/// We can do this to a select if its only uses are loads and if the operand
-/// to the select can be loaded unconditionally. If found an intervening bitcast
-/// with a single use of the load, allow the promotion.
-static bool isSafeSelectToSpeculate(SelectInst &SI) {
-  Value *TValue = SI.getTrueValue();
-  Value *FValue = SI.getFalseValue();
+sroa::SelectHandSpeculativity &
+sroa::SelectHandSpeculativity::setAsSpeculatable(bool isTrueVal) {
+  if (isTrueVal)
+    Bitfield::set<sroa::SelectHandSpeculativity::TrueVal>(Storage, true);
+  else
+    Bitfield::set<sroa::SelectHandSpeculativity::FalseVal>(Storage, true);
+  return *this;
+}
+
+bool sroa::SelectHandSpeculativity::isSpeculatable(bool isTrueVal) const {
+  return isTrueVal
+             ? Bitfield::get<sroa::SelectHandSpeculativity::TrueVal>(Storage)
+             : Bitfield::get<sroa::SelectHandSpeculativity::FalseVal>(Storage);
+}
+
+bool sroa::SelectHandSpeculativity::areAllSpeculatable() const {
+  return isSpeculatable(/*isTrueVal=*/true) &&
+         isSpeculatable(/*isTrueVal=*/false);
+}
+
+bool sroa::SelectHandSpeculativity::areAnySpeculatable() const {
+  return isSpeculatable(/*isTrueVal=*/true) ||
+         isSpeculatable(/*isTrueVal=*/false);
+}
+bool sroa::SelectHandSpeculativity::areNoneSpeculatable() const {
+  return !areAnySpeculatable();
+}
+
+static sroa::SelectHandSpeculativity
+isSafeLoadOfSelectToSpeculate(LoadInst &LI, SelectInst &SI, bool PreserveCFG) {
+  assert(LI.isSimple() && "Only for simple loads");
+  sroa::SelectHandSpeculativity Spec;
+
   const DataLayout &DL = SI.getModule()->getDataLayout();
+  for (Value *Value : {SI.getTrueValue(), SI.getFalseValue()})
+    if (isSafeToLoadUnconditionally(Value, LI.getType(), LI.getAlign(), DL,
+                                    &LI))
+      Spec.setAsSpeculatable(/*isTrueVal=*/Value == SI.getTrueValue());
+    else if (PreserveCFG)
+      return Spec;
+
+  return Spec;
+}
+
+std::optional<sroa::PossiblySpeculatableLoads>
+SROAPass::isSafeSelectToSpeculate(SelectInst &SI, bool PreserveCFG) {
+  PossiblySpeculatableLoads Loads;
 
   for (User *U : SI.users()) {
     LoadInst *LI;
     BitCastInst *BC = dyn_cast<BitCastInst>(U);
     if (BC && BC->hasOneUse())
       LI = dyn_cast<LoadInst>(*BC->user_begin());
     else
       LI = dyn_cast<LoadInst>(U);
 
-    if (!LI || !LI->isSimple())
-      return false;
+    // Note that atomic loads can be transformed;
+    // atomic semantics do not have any meaning for a local alloca.
+    if (!LI || LI->isVolatile())
+      return {}; // Give up on this `select`.
+
+    PossiblySpeculatableLoad Load(LI);
+
+    if (!LI->isSimple()) {
+      // If the `load` is not simple, we can't speculatively execute it,
+      // but we could handle this via a CFG modification. But can we?
+      if (PreserveCFG)
+        return {}; // Give up on this `select`.
+      Loads.emplace_back(Load);
+      continue;
+    }
 
-    // Both operands to the select need to be dereferenceable, either
-    // absolutely (e.g. allocas) or at this point because we can see other
-    // accesses to it.
-    if (!isSafeToLoadUnconditionally(TValue, LI->getType(),
-                                     LI->getAlign(), DL, LI))
-      return false;
-    if (!isSafeToLoadUnconditionally(FValue, LI->getType(),
-                                     LI->getAlign(), DL, LI))
-      return false;
+    sroa::SelectHandSpeculativity Spec =
+        isSafeLoadOfSelectToSpeculate(*LI, SI, PreserveCFG);
+    if (PreserveCFG && !Spec.areAllSpeculatable())
+      return {}; // Give up on this `select`.
+
+    Load.setInt(Spec);
+    Loads.emplace_back(Load);
   }
 
-  return true;
+  return Loads;
 }
 
-static void speculateSelectInstLoads(IRBuilderTy &IRB, SelectInst &SI) {
-  LLVM_DEBUG(dbgs() << "    original: " << SI << "\n");
+static void speculateSelectInstLoads(SelectInst &SI, LoadInst &LI,
+                                     IRBuilderTy &IRB) {
+  LLVM_DEBUG(dbgs() << "    original load: " << SI << "\n");
 
   IRB.SetInsertPoint(&SI);
   Value *TV = SI.getTrueValue();
   Value *FV = SI.getFalseValue();
-  // Replace the loads of the select with a select of two loads.
-  while (!SI.use_empty()) {
-    LoadInst *LI;
-    BitCastInst *BC = dyn_cast<BitCastInst>(SI.user_back());
-    if (BC) {
-      assert(BC->hasOneUse() && "Bitcast should have a single use.");
-      LI = cast<LoadInst>(BC->user_back());
-    } else {
-      LI = cast<LoadInst>(SI.user_back());
-    }
+  // Replace the given load of the select with a select of two loads.
 
-    assert(LI->isSimple() && "We only speculate simple loads");
+  assert(LI.isSimple() && "We only speculate simple loads");
 
-    IRB.SetInsertPoint(LI);
-    Value *NewTV =
-        BC ? IRB.CreateBitCast(TV, BC->getType(), TV->getName() + ".sroa.cast")
-           : TV;
-    Value *NewFV =
-        BC ? IRB.CreateBitCast(FV, BC->getType(), FV->getName() + ".sroa.cast")
-           : FV;
-    LoadInst *TL = IRB.CreateLoad(LI->getType(), NewTV,
-                                  LI->getName() + ".sroa.speculate.load.true");
-    LoadInst *FL = IRB.CreateLoad(LI->getType(), NewFV,
-                                  LI->getName() + ".sroa.speculate.load.false");
-    NumLoadsSpeculated += 2;
+  IRB.SetInsertPoint(&LI);
+  LoadInst *TL =
+      IRB.CreateAlignedLoad(LI.getType(), TV, LI.getAlign(),
+                            LI.getName() + ".sroa.speculate.load.true");
+  LoadInst *FL =
+      IRB.CreateAlignedLoad(LI.getType(), FV, LI.getAlign(),
+                            LI.getName() + ".sroa.speculate.load.false");
 
-    // Transfer alignment and AA info if present.
-    TL->setAlignment(LI->getAlign());
-    FL->setAlignment(LI->getAlign());
+  TL->setAlignment(LI.getAlign());
+  FL->setAlignment(LI.getAlign());
 
-    AAMDNodes Tags = LI->getAAMetadata();
-    if (Tags) {
-      TL->setAAMetadata(Tags);
-      FL->setAAMetadata(Tags);
-    }
+  AAMDNodes Tags = LI.getAAMetadata();
 
-    Value *V = IRB.CreateSelect(SI.getCondition(), TL, FL,
-                                LI->getName() + ".sroa.speculated");
+                              LI.getName() + ".sroa.speculated");
 
-    LLVM_DEBUG(dbgs() << "          speculated to: " << *V << "\n");
-    LI->replaceAllUsesWith(V);
+  LI.replaceAllUsesWith(V);
+}
+
+static void rewriteLoadOfSelect(SelectInst &SI, LoadInst &LI,
+                                sroa::SelectHandSpeculativity Spec,
+                                DomTreeUpdater &DTU) {
+  LLVM_DEBUG(dbgs() << "    original load: " << SI << "\n");
+  BasicBlock *Head = LI.getParent();
+  Instruction *ThenTerm = nullptr;
+  Instruction *ElseTerm = nullptr;
+  if (Spec.areNoneSpeculatable())
+    SplitBlockAndInsertIfThenElse(SI.getCondition(), &LI, &ThenTerm, &ElseTerm,
+                                  SI.getMetadata(LLVMContext::MD_prof), &DTU);
+  else {
+    SplitBlockAndInsertIfThen(SI.getCondition(), &LI, /*Unreachable=*/false,
+                              SI.getMetadata(LLVMContext::MD_prof), &DTU,
+                              /*LI=*/nullptr, /*ThenBlock=*/nullptr);
+    if (Spec.isSpeculatable(/*isTrueVal=*/true))
+      cast<BranchInst>(Head->getTerminator())->swapSuccessors();
+  }
+  auto *HeadBI = cast<BranchInst>(Head->getTerminator());
+  Spec = {}; // Do not use `Spec` beyond this point.
+  BasicBlock *Tail = LI.getParent();
+  Tail->setName(Head->getName() + ".cont");
+  auto *PN = PHINode::Create(LI.getType(), 2, "", &LI);
+  for (BasicBlock *SuccBB : successors(Head)) {
+    bool IsThen = SuccBB == HeadBI->getSuccessor(0);
+    int SuccIdx = IsThen ? 0 : 1;
+    auto *NewLoadBB = SuccBB == Tail ? Head : SuccBB;
+    if (NewLoadBB != Head) {
+      NewLoadBB->setName(Head->getName() + (IsThen ? ".then" : ".else"));
+      ++NumLoadsPredicated;
+    } else
+      ++NumLoadsSpeculated;
+    auto *CondLoad = cast<LoadInst>(LI.clone());
+    CondLoad->insertBefore(NewLoadBB->getTerminator());
+    CondLoad->setOperand(0, SI.getOperand(1 + SuccIdx));
+    CondLoad->setName(LI.getName() + (IsThen ? ".then" : ".else") + ".val");
+    PN->addIncoming(CondLoad, NewLoadBB);
+  }
+  PN->takeName(&LI);
+  LLVM_DEBUG(dbgs() << "          to: " << *PN << "\n");
+  LI.replaceAllUsesWith(PN);
+}
+
+static bool rewriteSelectInstLoads(SelectInst &SI,
+                                   const sroa::PossiblySpeculatableLoads &Loads,
+                                   IRBuilderTy &IRB, DomTreeUpdater *DTU) {
+  bool CFGChanged = false;
+  LLVM_DEBUG(dbgs() << "    original select: " << SI << "\n");
+
+  for (const PossiblySpeculatableLoad &Load : Loads) {
+    LoadInst *LI = Load.getPointer();
+    sroa::SelectHandSpeculativity Spec = Load.getInt();
+    if (Spec.areAllSpeculatable()) {
+      speculateSelectInstLoads(SI, *LI, IRB);
+    } else {
+      assert("Should not get here when not allowed to modify the CFG!");
+      rewriteLoadOfSelect(SI, *LI, Spec, *DTU);
+      CFGChanged = true;
+    }
     LI->eraseFromParent();
-    if (BC)
-      BC->eraseFromParent();
   }
+
+  for (User *U : make_early_inc_range(SI.users()))
+    cast<BitCastInst>(U)->eraseFromParent();
   SI.eraseFromParent();
+  return CFGChanged;
 }
 
 /// Build a GEP out of a base pointer and indices.
 ///
 /// This will return the BasePtr if that is valid, or build a new GEP
 /// instruction using the IRBuilder if GEP-ing is needed.
 static Value *buildGEP(IRBuilderTy &IRB, Value *BasePtr,
                        SmallVectorImpl<Value *> &Indices,
AllocaInst *SROAPass::rewritePartition(AllocaInst &AI, AllocaSlices &AS,
   for (PHINode *PHI : PHIUsers)
     if (!isSafePHIToSpeculate(*PHI)) {
       Promotable = false;
       PHIUsers.clear();
       SelectUsers.clear();
       break;
     }
 
-  for (SelectInst *Sel : SelectUsers)
-    if (!isSafeSelectToSpeculate(*Sel)) {
+  SmallVector<std::pair<SelectInst *, PossiblySpeculatableLoads>, 2>
+      NewSelectsToRewrite;
+  NewSelectsToRewrite.reserve(SelectUsers.size());
+  for (SelectInst *Sel : SelectUsers) {
+    std::optional<PossiblySpeculatableLoads> Loads =
+        isSafeSelectToSpeculate(*Sel, PreserveCFG);
+    if (!Loads) {
       Promotable = false;
       PHIUsers.clear();
       SelectUsers.clear();
+      NewSelectsToRewrite.clear();
       break;
     }
+    NewSelectsToRewrite.emplace_back(std::make_pair(Sel, *Loads));
+  }
 
   if (Promotable) {
     for (Use *U : AS.getDeadUsesIfPromotable()) {
       auto *OldInst = dyn_cast<Instruction>(U->get());
       Value::dropDroppableUse(*U);
       if (OldInst)
         if (isInstructionTriviallyDead(OldInst))
           DeadInsts.push_back(OldInst);
     }
     if (PHIUsers.empty() && SelectUsers.empty()) {
       // Promote the alloca.
       PromotableAllocas.push_back(NewAI);
     } else {
       // If we have either PHIs or Selects to speculate, add them to those
       // worklists and re-queue the new alloca so that we promote in on the
       // next iteration.
       for (PHINode *PHIUser : PHIUsers)
         SpeculatablePHIs.insert(PHIUser);
-      for (SelectInst *SelectUser : SelectUsers)
-        SpeculatableSelects.insert(SelectUser);
+      SelectsToRewrite.reserve(SelectsToRewrite.size() +
+                               NewSelectsToRewrite.size());
+      for (auto &&KV : llvm::make_range(
+               std::make_move_iterator(NewSelectsToRewrite.begin()),
+               std::make_move_iterator(NewSelectsToRewrite.end())))
+        SelectsToRewrite.insert(std::move(KV));
       Worklist.insert(NewAI);
     }
   } else {
     // Drop any post-promotion work items if promotion didn't happen.
     while (PostPromotionWorklist.size() > PPWOldSize)
       PostPromotionWorklist.pop_back();
 
     // We couldn't promote and we didn't create a new partition, nothing
if (Instruction *OldI = dyn_cast<Instruction>(OldV))
     }
 }
 
 /// Analyze an alloca for SROA.
 ///
 /// This analyzes the alloca to ensure we can reason about it, builds
 /// the slices of the alloca, and then hands it off to be split and
 /// rewritten as needed.
-bool SROAPass::runOnAlloca(AllocaInst &AI) {
+std::pair<bool /*Changed*/, bool /*CFGChanged*/>
+SROAPass::runOnAlloca(AllocaInst &AI) {
+  bool Changed = false;
+  bool CFGChanged = false;
+
   LLVM_DEBUG(dbgs() << "SROA alloca: " << AI << "\n");
   ++NumAllocasAnalyzed;
 
   // Special case dead allocas, as they're trivial.
   if (AI.use_empty()) {
     AI.eraseFromParent();
-    return true;
+    Changed = true;
+    return {Changed, CFGChanged};
   }
   const DataLayout &DL = AI.getModule()->getDataLayout();
 
   // Skip alloca forms that this analysis can't handle.
   auto *AT = AI.getAllocatedType();
   if (AI.isArrayAllocation() || !AT->isSized() || isa<ScalableVectorType>(AT) ||
       DL.getTypeAllocSize(AT).getFixedSize() == 0)
-    return false;
-
-  bool Changed = false;
+    return {Changed, CFGChanged};
 
   // First, split any FCA loads and stores touching this alloca to promote
   // better splitting and promotion opportunities.
   IRBuilderTy IRB(&AI);
   AggLoadStoreRewriter AggRewriter(DL, IRB);
   Changed |= AggRewriter.rewrite(AI);
 
   // Build the slices using a recursive instruction-visiting builder.
   AllocaSlices AS(DL, AI);
   LLVM_DEBUG(AS.print(dbgs()));
   if (AS.isEscaped())
-    return Changed;
+    return {Changed, CFGChanged};
 
   // Delete all the dead users of this alloca before splitting and rewriting it.
   for (Instruction *DeadUser : AS.getDeadUsers()) {
     // Free up everything used by this instruction.
     for (Use &DeadOp : DeadUser->operands())
       clobberUse(DeadOp);
 
     // Now replace the uses of this instruction.
     DeadUser->replaceAllUsesWith(PoisonValue::get(DeadUser->getType()));
 
     // And mark it for deletion.
     DeadInsts.push_back(DeadUser);
     Changed = true;
   }
   for (Use *DeadOp : AS.getDeadOperands()) {
     clobberUse(*DeadOp);
     Changed = true;
   }
 
   // No slices to split. Leave the dead alloca for a later pass to clean up.
   if (AS.begin() == AS.end())
-    return Changed;
+    return {Changed, CFGChanged};
 
   Changed |= splitAlloca(AI, AS);
 
   LLVM_DEBUG(dbgs() << "  Speculating PHIs\n");
   while (!SpeculatablePHIs.empty())
     speculatePHINodeLoads(IRB, *SpeculatablePHIs.pop_back_val());
 
-  LLVM_DEBUG(dbgs() << "  Speculating Selects\n");
-  while (!SpeculatableSelects.empty())
-    speculateSelectInstLoads(IRB, *SpeculatableSelects.pop_back_val());
+  LLVM_DEBUG(dbgs() << "  Rewriting Selects\n");
+  auto RemainingSelectsToRewrite = SelectsToRewrite.takeVector();
+  while (!RemainingSelectsToRewrite.empty()) {
+    const auto [K, V] = RemainingSelectsToRewrite.pop_back_val();
+    CFGChanged |=
+        rewriteSelectInstLoads(*K, V, IRB, PreserveCFG ? nullptr : DTU);
+  }
 
-  return Changed;
+  return {Changed, CFGChanged};
 }
 
 /// Delete the dead instructions accumulated in this run.
 ///
 /// Recursively deletes the dead instructions we've accumulated. This is done
 /// at the very end to maximize locality of the recursive delete and to
 /// minimize the problems of invalidated instruction pointers as such pointers
 /// are used heavily in the intermediate stages of the algorithm.

 /// This function returns whether any promotion occurred.
 bool SROAPass::promoteAllocas(Function &F) {
   if (PromotableAllocas.empty())
     return false;
 
   NumPromoted += PromotableAllocas.size();
 
   LLVM_DEBUG(dbgs() << "Promoting allocas with mem2reg...\n");
-  PromoteMemToReg(PromotableAllocas, *DT, AC);
+  PromoteMemToReg(PromotableAllocas, DTU->getDomTree(), AC);
   PromotableAllocas.clear();
   return true;
 }
 
-PreservedAnalyses SROAPass::runImpl(Function &F, DominatorTree &RunDT,
+PreservedAnalyses SROAPass::runImpl(Function &F, DomTreeUpdater &RunDTU,
                                     AssumptionCache &RunAC) {
   LLVM_DEBUG(dbgs() << "SROA function: " << F.getName() << "\n");
   C = &F.getContext();
-  DT = &RunDT;
+  DTU = &RunDTU;
   AC = &RunAC;
 
   BasicBlock &EntryBB = F.getEntryBlock();
   for (BasicBlock::iterator I = EntryBB.begin(), E = std::prev(EntryBB.end());
        I != E; ++I) {
     if (AllocaInst *AI = dyn_cast<AllocaInst>(I)) {
       if (isa<ScalableVectorType>(AI->getAllocatedType())) {
         if (isAllocaPromotable(AI))
           PromotableAllocas.push_back(AI);
       } else {
         Worklist.insert(AI);
       }
     }
   }
 
   bool Changed = false;
+  bool CFGChanged = false;
   // A set of deleted alloca instruction pointers which should be removed from
   // the list of promotable allocas.
   SmallPtrSet<AllocaInst *, 4> DeletedAllocas;
 
   do {
     while (!Worklist.empty()) {
-      Changed |= runOnAlloca(*Worklist.pop_back_val());
+      auto [IterationChanged, IterationCFGChanged] =
+          runOnAlloca(*Worklist.pop_back_val());
+      Changed |= IterationChanged;
+      CFGChanged |= IterationCFGChanged;
+
       Changed |= deleteDeadInstructions(DeletedAllocas);
 
       // Remove the deleted allocas from various lists so that we don't try to
       // continue processing them.
       if (!DeletedAllocas.empty()) {
         auto IsInSet = [&](AllocaInst *AI) { return DeletedAllocas.count(AI); };
         Worklist.remove_if(IsInSet);
         PostPromotionWorklist.remove_if(IsInSet);
         llvm::erase_if(PromotableAllocas, IsInSet);
         DeletedAllocas.clear();
       }
     }
 
     Changed |= promoteAllocas(F);
 
     Worklist = PostPromotionWorklist;
     PostPromotionWorklist.clear();
   } while (!Worklist.empty());
 
+  assert((!CFGChanged || Changed) && "Can not only modify the CFG.");
+  assert((!CFGChanged || !PreserveCFG) &&
+         "Should not have modified the CFG when told to preserve it.");
+
   if (!Changed)
     return PreservedAnalyses::all();
 
   PreservedAnalyses PA;
+  if (!CFGChanged)
     PA.preserveSet<CFGAnalyses>();
+  PA.preserve<DominatorTreeAnalysis>();
   return PA;
 }
 
+PreservedAnalyses SROAPass::runImpl(Function &F, DominatorTree &RunDT,
+                                    AssumptionCache &RunAC) {
+  DomTreeUpdater DTU(RunDT, DomTreeUpdater::UpdateStrategy::Lazy);
+  return runImpl(F, DTU, RunAC);
+}
+
 PreservedAnalyses SROAPass::run(Function &F, FunctionAnalysisManager &AM) {
   return runImpl(F, AM.getResult<DominatorTreeAnalysis>(F),
                  AM.getResult<AssumptionAnalysis>(F));
 }
 
+void SROAPass::printPipeline(
+    raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
+  static_cast<PassInfoMixin<SROAPass> *>(this)->printPipeline(
+      OS, MapClassName2PassName);
+  OS << (PreserveCFG ? "<preserve-cfg>" : "<modify-cfg>");
+}
+
+SROAPass::SROAPass(SROAOptions PreserveCFG_)
+    : PreserveCFG(PreserveCFG_ == SROAOptions::PreserveCFG) {}
+
 /// A legacy pass for the legacy pass manager that wraps the \c SROA pass.
 ///
 /// This is in the llvm namespace purely to allow it to be a friend of the \c
 /// SROA pass.
 class llvm::sroa::SROALegacyPass : public FunctionPass {
   /// The SROA implementation.
   SROAPass Impl;
 
 public:
   static char ID;
 
-  SROALegacyPass() : FunctionPass(ID) {
+  SROALegacyPass(SROAOptions PreserveCFG = SROAOptions::PreserveCFG)
+      : FunctionPass(ID), Impl(PreserveCFG) {
     initializeSROALegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnFunction(Function &F) override {
     if (skipFunction(F))
       return false;
 
     auto PA = Impl.runImpl(
         F, getAnalysis<DominatorTreeWrapperPass>().getDomTree(),
         getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F));
     return !PA.areAllPreserved();
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AssumptionCacheTracker>();
     AU.addRequired<DominatorTreeWrapperPass>();
     AU.addPreserved<GlobalsAAWrapperPass>();
-    AU.setPreservesCFG();
+    AU.addPreserved<DominatorTreeWrapperPass>();
   }
 
   StringRef getPassName() const override { return "SROA"; }
 };
 
 char SROALegacyPass::ID = 0;
 
-FunctionPass *llvm::createSROAPass() { return new SROALegacyPass(); }
+FunctionPass *llvm::createSROAPass(bool PreserveCFG) {
+  return new SROALegacyPass(PreserveCFG ? SROAOptions::PreserveCFG
+                                        : SROAOptions::ModifyCFG);
+}
 
 INITIALIZE_PASS_BEGIN(SROALegacyPass, "sroa",
                       "Scalar Replacement Of Aggregates", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_END(SROALegacyPass, "sroa", "Scalar Replacement Of Aggregates",
                     false, false)

llvm/lib/Transforms/Utils/BasicBlockUtils.cpp

Instruction *llvm::SplitBlockAndInsertIfThen(Value *Cond,
   return SplitBlockAndInsertIfThenImpl(Cond, SplitBefore, Unreachable,
                                        BranchWeights, DTU, /*DT=*/nullptr, LI,
                                        ThenBlock);
 }
 
 void llvm::SplitBlockAndInsertIfThenElse(Value *Cond, Instruction *SplitBefore,
                                          Instruction **ThenTerm,
                                          Instruction **ElseTerm,
-                                         MDNode *BranchWeights) {
+                                         MDNode *BranchWeights,
+                                         DomTreeUpdater *DTU) {
   BasicBlock *Head = SplitBefore->getParent();
+
+  SmallPtrSet<BasicBlock *, 8> UniqueOrigSuccessors;
+  if (DTU)
+    UniqueOrigSuccessors.insert(succ_begin(Head), succ_end(Head));
+
   BasicBlock *Tail = Head->splitBasicBlock(SplitBefore->getIterator());
   Instruction *HeadOldTerm = Head->getTerminator();
   LLVMContext &C = Head->getContext();
   BasicBlock *ThenBlock = BasicBlock::Create(C, "", Head->getParent(), Tail);
   BasicBlock *ElseBlock = BasicBlock::Create(C, "", Head->getParent(), Tail);
   *ThenTerm = BranchInst::Create(Tail, ThenBlock);
   (*ThenTerm)->setDebugLoc(SplitBefore->getDebugLoc());
   *ElseTerm = BranchInst::Create(Tail, ElseBlock);
   (*ElseTerm)->setDebugLoc(SplitBefore->getDebugLoc());
   BranchInst *HeadNewTerm =
     BranchInst::Create(/*ifTrue*/ThenBlock, /*ifFalse*/ElseBlock, Cond);
   HeadNewTerm->setMetadata(LLVMContext::MD_prof, BranchWeights);
   ReplaceInstWithInst(HeadOldTerm, HeadNewTerm);
+  if (DTU) {
+    SmallVector<DominatorTree::UpdateType, 8> Updates;
+    Updates.reserve(4 + 2 * UniqueOrigSuccessors.size());
+    for (BasicBlock *Succ : successors(Head)) {
+      Updates.push_back({DominatorTree::Insert, Head, Succ});
+      Updates.push_back({DominatorTree::Insert, Succ, Tail});
+    }
+    for (BasicBlock *UniqueOrigSuccessor : UniqueOrigSuccessors)
+      Updates.push_back({DominatorTree::Insert, Tail, UniqueOrigSuccessor});
+    for (BasicBlock *UniqueOrigSuccessor : UniqueOrigSuccessors)
+      Updates.push_back({DominatorTree::Delete, Head, UniqueOrigSuccessor});
+    DTU->applyUpdates(Updates);
+  }
 }
 
 BranchInst *llvm::GetIfCondition(BasicBlock *BB, BasicBlock *&IfTrue,
                                  BasicBlock *&IfFalse) {
   PHINode *SomePHI = dyn_cast<PHINode>(BB->begin());
   BasicBlock *Pred1 = nullptr;
   BasicBlock *Pred2 = nullptr;
 

llvm/test/Transforms/SROA/2009-02-20-InstCombine-SROA.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; rdar://6417724
 
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:128:128"
 target triple = "i386-apple-darwin9.6"
 
 %"struct.std::vector<int,std::allocator<int> >" = type { %"struct.std::_Vector_base<int,std::allocator<int> >" }
 %"struct.std::_Vector_base<int,std::allocator<int> >" = type { %"struct.std::_Vector_base<int,std::allocator<int> >::_Vector_impl" }
return:                                           ; preds = %_ZSt4findIN9__gnu_cxx17__normal_iteratorIPiSt6vectorIiSaIiEEEEiET_S7_S7_RKT0_.exit
   %35 = load ptr, ptr %.in.in, align 4
   ret ptr %35
 }
 
 ; Function Attrs: argmemonly nofree nosync nounwind willreturn
 declare void @llvm.memcpy.p0.p0.i64(ptr noalias nocapture writeonly, ptr noalias nocapture readonly, i64, i1 immarg) #0
 
 attributes #0 = { argmemonly nofree nosync nounwind willreturn }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/address-spaces.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-p1:16:16:16-p3:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.memcpy.p0.p0.i32(ptr nocapture, ptr nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p1.p0.i32(ptr addrspace(1) nocapture, ptr nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p0.p1.i32(ptr nocapture, ptr addrspace(1) nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p1.p1.i32(ptr addrspace(1) nocapture, ptr addrspace(1) nocapture readonly, i32, i1)
 
 
;
   %f2 = load float, ptr %a.gep2
   %sum = fadd float %f1, %f2
   store float %sum, ptr %a
   store float %sum, ptr %a.gep2
   %v2 = load i64, ptr %a
   store i64 %v2, ptr addrspace(1) %complex2
   ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/addrspacecast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-p:64:64:64-p1:16:16:16-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.memcpy.p0.p1.i32(ptr nocapture writeonly, ptr addrspace(1) nocapture readonly, i32, i1 immarg) #0
 declare void @llvm.memcpy.p1.p0.i32(ptr addrspace(1) nocapture writeonly, ptr nocapture readonly, i32, i1 immarg) #0
 
 define i64 @alloca_addrspacecast_bitcast(i64 %X) {
 ; CHECK-LABEL: @alloca_addrspacecast_bitcast(
;
   %asc = addrspacecast ptr %p.0.c to ptr addrspace(1)
 
   %cond.in = select i1 %b, ptr addrspace(1) %asc, ptr addrspace(1) %asc
   %cond = load i64, ptr addrspace(1) %cond.in, align 8
   ret void
 }
 
 define void @select_addrspacecast_const_op(i1 %a, i1 %b) {
-; CHECK-LABEL: @select_addrspacecast_const_op(
-; CHECK-NEXT:    [[C:%.*]] = alloca i64, align 8
-; CHECK-NEXT:    [[C_0_ASC_SROA_CAST:%.*]] = addrspacecast ptr [[C]] to ptr addrspace(1)
-; CHECK-NEXT:    [[COND_IN:%.*]] = select i1 [[B:%.*]], ptr addrspace(1) [[C_0_ASC_SROA_CAST]], ptr addrspace(1) null
-; CHECK-NEXT:    [[COND:%.*]] = load i64, ptr addrspace(1) [[COND_IN]], align 8
-; CHECK-NEXT:    ret void
+; CHECK-preserve-cfg-LABEL: @select_addrspacecast_const_op(
+; CHECK-preserve-cfg-NEXT:    [[C:%.*]] = alloca i64, align 8
+; CHECK-preserve-cfg-NEXT:    [[C_0_ASC_SROA_CAST:%.*]] = addrspacecast ptr [[C]] to ptr addrspace(1)
+; CHECK-preserve-cfg-NEXT:    [[COND_IN:%.*]] = select i1 [[B:%.*]], ptr addrspace(1) [[C_0_ASC_SROA_CAST]], ptr addrspace(1) null
+; CHECK-preserve-cfg-NEXT:    [[COND:%.*]] = load i64, ptr addrspace(1) [[COND_IN]], align 8
+; CHECK-preserve-cfg-NEXT:    ret void
+;
+; CHECK-modify-cfg-LABEL: @select_addrspacecast_const_op(
+; CHECK-modify-cfg-NEXT:    br i1 [[B:%.*]], label [[DOTCONT:%.*]], label [[DOTELSE:%.*]]
+; CHECK-modify-cfg:       .else:
+; CHECK-modify-cfg-NEXT:    [[COND_ELSE_VAL:%.*]] = load i64, ptr addrspace(1) null, align 8
+; CHECK-modify-cfg-NEXT:    br label [[DOTCONT]]
+; CHECK-modify-cfg:       .cont:
+; CHECK-modify-cfg-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[TMP0:%.*]] ], [ [[COND_ELSE_VAL]], [[DOTELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %asc = addrspacecast ptr %p.0.c to ptr addrspace(1)
 
   %cond.in = select i1 %b, ptr addrspace(1) %asc, ptr addrspace(1) null
   %cond = load i64, ptr addrspace(1) %cond.in, align 8
   ret void

llvm/test/Transforms/SROA/alignment.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
-; RUN: opt -passes='debugify,function(sroa)' -S < %s | FileCheck %s -check-prefix CHECK-DEBUGLOC
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
+; RUN: opt -passes='debugify,function(sroa<preserve-cfg>)' -S < %s | FileCheck %s -check-prefix CHECK-DEBUGLOC
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.memcpy.p0.p0.i32(ptr, ptr, i32, i1)
 
 define void @test1(ptr %a, ptr %b) {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
;
   %B = alloca %struct, align 4
   %1 = load i32, ptr %A, align 4
   store atomic volatile i32 %1, ptr %B release, align 4
   %x = load atomic volatile i32, ptr %B acquire, align 4
   ret i32 %x
 }
 
 declare void @populate(ptr)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/alloca-address-space.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-p1:16:16:16-p2:32:32-p3:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64-A2"
 
 declare void @llvm.memcpy.p2.p2.i32(ptr addrspace(2) nocapture, ptr addrspace(2) nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p1.p2.i32(ptr addrspace(1) nocapture, ptr addrspace(2) nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p2.p1.i32(ptr addrspace(2) nocapture, ptr addrspace(1) nocapture readonly, i32, i1)
 declare void @llvm.memcpy.p1.p1.i32(ptr addrspace(1) nocapture, ptr addrspace(1) nocapture readonly, i32, i1)
 
 

 ;
   %alloca = alloca i8, align 8, addrspace(2)
   %cast = addrspacecast ptr addrspace(2) %alloca to ptr
   call void @llvm.lifetime.start.p0(i64 2, ptr %cast)
   ret void
 }
 
 declare void @llvm.lifetime.start.p0(i64 %size, ptr nocapture %ptr)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/assume.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 %struct.n.2 = type { i8 }
 %struct.bi = type { %class.as }
 %class.as = type { %class.ao }
 %class.ao = type { %class.ai }
entry:
   call void @llvm.assume(i1 true) [ "nonnull"(ptr %y) ]
   ret void
 }
 
 ; Function Attrs: nofree norecurse nounwind willreturn
 declare void @llvm.assume(i1) #0
 
 attributes #0 = { nofree norecurse nounwind willreturn }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/basictest.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-p:64:64:64-p1:16:16:16-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.lifetime.start.p0(i64, ptr nocapture)
 declare void @llvm.lifetime.end.p0(i64, ptr nocapture)
 
 define i32 @test0() {
 ; CHECK-LABEL: @test0(
bb1:
   br label %end
 
 end:
   call void @llvm.memcpy.p0.p0.i32(ptr %data, ptr %tmp, i32 %size, i1 false)
   ret void
 }
 
 define void @PR15805(i1 %a, i1 %b) {
-; CHECK-LABEL: @PR15805(
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[B:%.*]], i64 undef, i64 undef
-; CHECK-NEXT:    ret void
+; CHECK-preserve-cfg-LABEL: @PR15805(
+; CHECK-preserve-cfg-NEXT:    [[C:%.*]] = alloca i64, align 8
+; CHECK-preserve-cfg-NEXT:    [[COND_IN:%.*]] = select i1 [[B:%.*]], ptr [[C]], ptr [[C]]
+; CHECK-preserve-cfg-NEXT:    [[COND:%.*]] = load i64, ptr [[COND_IN]], align 8
+; CHECK-preserve-cfg-NEXT:    ret void
+;
+; CHECK-modify-cfg-LABEL: @PR15805(
+; CHECK-modify-cfg-NEXT:    br i1 [[B:%.*]], label [[DOTCONT:%.*]], label [[DOTELSE:%.*]]
+; CHECK-modify-cfg:       .else:
+; CHECK-modify-cfg-NEXT:    br label [[DOTCONT]]
+; CHECK-modify-cfg:       .cont:
+; CHECK-modify-cfg-NEXT:    [[COND:%.*]] = phi i64 [ undef, [[TMP0:%.*]] ], [ undef, [[DOTELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   %p.0.c = select i1 %a, ptr %c, ptr %c
   %cond.in = select i1 %b, ptr %p.0.c, ptr %c
   %cond = load i64, ptr %cond.in, align 8
   ret void
 }
 
 define void @PR15805.1(i1 %a, i1 %b, i1 %c2) {
 ; Same as the normal PR15805, but rigged to place the use before the def inside
 ; of looping unreachable code. This helps ensure that we aren't sensitive to the
 ; order in which the uses of the alloca are visited.
 ;
 ;
-; CHECK-LABEL: @PR15805.1(
-; CHECK-NEXT:    br label [[EXIT:%.*]]
-; CHECK:       loop:
-; CHECK-NEXT:    [[COND_SROA_SPECULATED:%.*]] = select i1 [[A:%.*]], i64 undef, i64 undef
-; CHECK-NEXT:    br i1 [[C2:%.*]], label [[LOOP:%.*]], label [[EXIT]]
-; CHECK:       exit:
-; CHECK-NEXT:    ret void
+; CHECK-preserve-cfg-LABEL: @PR15805.1(
+; CHECK-preserve-cfg-NEXT:    [[C:%.*]] = alloca i64, align 8
+; CHECK-preserve-cfg-NEXT:    br label [[EXIT:%.*]]
+; CHECK-preserve-cfg:       loop:
+; CHECK-preserve-cfg-NEXT:    [[COND_IN:%.*]] = select i1 [[A:%.*]], ptr [[C]], ptr [[C]]
+; CHECK-preserve-cfg-NEXT:    [[COND:%.*]] = load i64, ptr [[COND_IN]], align 8
+; CHECK-preserve-cfg-NEXT:    br i1 [[C2:%.*]], label [[LOOP:%.*]], label [[EXIT]]
+; CHECK-preserve-cfg:       exit:
+; CHECK-preserve-cfg-NEXT:    ret void
+;
+; CHECK-modify-cfg-LABEL: @PR15805.1(
+; CHECK-modify-cfg-NEXT:    br label [[EXIT:%.*]]
+; CHECK-modify-cfg:       loop:
+; CHECK-modify-cfg-NEXT:    [[C_0_LOAD:%.*]] = load i64, ptr poison, align 8
+; CHECK-modify-cfg-NEXT:    br i1 [[A:%.*]], label [[LOOP_CONT:%.*]], label [[LOOP_ELSE:%.*]]
+; CHECK-modify-cfg:       loop.else:
+; CHECK-modify-cfg-NEXT:    [[C_0_LOAD1:%.*]] = load i64, ptr poison, align 8
+; CHECK-modify-cfg-NEXT:    br label [[LOOP_CONT]]
+; CHECK-modify-cfg:       loop.cont:
+; CHECK-modify-cfg-NEXT:    [[COND:%.*]] = phi i64 [ [[C_0_LOAD]], [[LOOP:%.*]] ], [ [[C_0_LOAD1]], [[LOOP_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    br i1 [[C2:%.*]], label [[LOOP]], label [[EXIT]]
+; CHECK-modify-cfg:       exit:
+; CHECK-modify-cfg-NEXT:    ret void
 ;
   %c = alloca i64, align 8
   br label %exit
 
 loop:
   %cond.in = select i1 %a, ptr %c, ptr %p.0.c
   %p.0.c = select i1 %b, ptr %c, ptr %c
   %cond = load i64, ptr %cond.in, align 8

 
 define void @PR16651.2(<2 x float> %val, i1 %c1) {
 ; This test case caused a crash due to failing to promote given a select that
 ; can't be speculated. It shouldn't be promoted, but we missed that fact when
 ; analyzing whether we could form a vector promotion because that code didn't
 ; bail on select instructions.
 ;
 ;
-; CHECK-LABEL: @PR16651.2(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TV1_SROA_0:%.*]] = alloca <2 x float>, align 8
-; CHECK-NEXT:    store <2 x float> [[VAL:%.*]], ptr [[TV1_SROA_0]], align 8
-; CHECK-NEXT:    [[COND105_IN_I_I:%.*]] = select i1 [[C1:%.*]], ptr null, ptr [[TV1_SROA_0]]
-; CHECK-NEXT:    [[COND105_I_I:%.*]] = load float, ptr [[COND105_IN_I_I]], align 8
-; CHECK-NEXT:    ret void
+; CHECK-preserve-cfg-LABEL: @PR16651.2(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[TV1_SROA_0:%.*]] = alloca <2 x float>, align 8
+; CHECK-preserve-cfg-NEXT:    store <2 x float> [[VAL:%.*]], ptr [[TV1_SROA_0]], align 8
+; CHECK-preserve-cfg-NEXT:    [[COND105_IN_I_I:%.*]] = select i1 [[C1:%.*]], ptr null, ptr [[TV1_SROA_0]]
+; CHECK-preserve-cfg-NEXT:    [[COND105_I_I:%.*]] = load float, ptr [[COND105_IN_I_I]], align 8
+; CHECK-preserve-cfg-NEXT:    ret void
+;
+; CHECK-modify-cfg-LABEL: @PR16651.2(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[TV1_SROA_0_0_VEC_EXTRACT:%.*]] = extractelement <2 x float> [[VAL:%.*]], i32 0
+; CHECK-modify-cfg-NEXT:    br i1 [[C1:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_CONT:%.*]]
+; CHECK-modify-cfg:       entry.then:
+; CHECK-modify-cfg-NEXT:    [[COND105_I_I_THEN_VAL:%.*]] = load float, ptr null, align 8
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[COND105_I_I:%.*]] = phi float [ [[COND105_I_I_THEN_VAL]], [[ENTRY_THEN]] ], [ [[TV1_SROA_0_0_VEC_EXTRACT]], [[ENTRY:%.*]] ]
+; CHECK-modify-cfg-NEXT:    ret void
 ;
 entry:
   %tv1 = alloca { <2 x float>, <2 x float> }, align 8
   %0 = getelementptr { <2 x float>, <2 x float> }, ptr %tv1, i64 0, i32 1
   store <2 x float> %val, ptr %0, align 8
   %1 = getelementptr inbounds { <2 x float>, <2 x float> }, ptr %tv1, i64 0, i32 1, i64 0
   %cond105.in.i.i = select i1 %c1, ptr null, ptr %1
   %cond105.i.i = load float, ptr %cond105.in.i.i, align 8

llvm/test/Transforms/SROA/big-endian.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "E-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 define i8 @test1() {
 ; We fully promote these to the i24 load or store size, resulting in just masks
 ; and other operations that instcombine will fold, but no alloca. Note this is
 ; the same as test12 in basictest.ll, but here we assert big-endian byte
 ; ordering.
entry:
   %tmp3 = load i64, ptr %a, align 4
   %tmp5 = load i32, ptr %a, align 4
 
   call void @f(i64 %tmp3, i32 %tmp5)
   ret void
 }
 
 declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/dbg-addr-diamond.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -use-dbg-addr -passes=sroa -S < %s | FileCheck %s
+; RUN: opt -use-dbg-addr -passes='sroa<preserve-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -use-dbg-addr -passes='sroa<modify-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; ModuleID = '<stdin>'
 source_filename = "newvars.c"
 target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc19.0.24215"
 
 %struct.Pair = type { i32, i32 }
 

 !47 = distinct !DILexicalBlock(scope: !38, file: !1, line: 9, column: 10)
 !48 = !DILocation(line: 10, column: 9, scope: !47)
 !49 = !DILocation(line: 11, column: 7, scope: !47)
 !50 = !DILocation(line: 11, column: 9, scope: !47)
 !51 = !DILocation(line: 13, column: 10, scope: !8)
 !52 = !{i64 0, i64 4, !53, i64 4, i64 4, !53}
 !53 = !{!31, !31, i64 0}
 !54 = !DILocation(line: 14, column: 1, scope: !8)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/dbg-inline.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; Test that SROA can deal with allocas that have more than one
 ; dbg.declare hanging off of it.
 
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 source_filename = "/tmp/inlinesplit.cpp"
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.15.0"
 
 %struct.pair = type { i64, i64 }
 
 ; Function Attrs: noinline optnone ssp uwtable
 define i64 @_Z1g4pair(i64 %p.coerce0, i64 %p.coerce1) #0 !dbg !8 {

 !19 = !DILocation(line: 10, column: 12, scope: !8)
 !20 = !DILocation(line: 10, column: 10, scope: !8)
 !21 = !DILocalVariable(name: "p", arg: 1, scope: !22, file: !9, line: 5, type: !13)
 !22 = distinct !DISubprogram(name: "f", linkageName: "_ZL1f4pair", scope: !9, file: !9, line: 5, type: !10, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !0, retainedNodes: !2)
 !23 = !DILocation(line: 5, column: 27, scope: !22, inlinedAt: !24)
 !24 = distinct !DILocation(line: 10, column: 10, scope: !8)
 !25 = !DILocation(line: 6, column: 12, scope: !22, inlinedAt: !24)
 !26 = !DILocation(line: 10, column: 3, scope: !8)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/dbg-single-piece.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa %s -S | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' %s -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' %s -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 %foo = type { [8 x i8], [8 x i8] }
 
 declare void @llvm.dbg.declare(metadata, metadata, metadata) #0
 define void @_ZL18findInsertLocationPN4llvm17MachineBasicBlockENS_9SlotIndexERNS_13LiveIntervalsE() {
 ; CHECK-LABEL: @_ZL18findInsertLocationPN4llvm17MachineBasicBlockENS_9SlotIndexERNS_13LiveIntervalsE(
 ; CHECK-NEXT:  entry:

 !2 = distinct !DISubprogram(name: "findInsertLocation", linkageName: "_ZL18findInsertLocationPN4llvm17MachineBasicBlockENS_9SlotIndexERNS_13LiveIntervalsE", scope: !3, file: !3, line: 937, isLocal: true, isDefinition: true, scopeLine: 938, flags: DIFlagPrototyped, isOptimized: true, unit: !9)
 !3 = !DIFile(filename: "none", directory: ".")
 !4 = !DICompositeType(tag: DW_TAG_class_type, name: "bundle_iterator<llvm::MachineInstr, llvm::ilist_iterator<llvm::MachineInstr> >", scope: !5, file: !3, line: 163, size: 128, align: 64, elements: !6, templateParams: !6, identifier: "_ZTSN4llvm17MachineBasicBlock15bundle_iteratorINS_12MachineInstrENS_14ilist_iteratorIS2_EEEE")
 !5 = distinct !DICompositeType(tag: DW_TAG_class_type, name: "MachineBasicBlock", file: !3, line: 68, size: 1408, align: 64, identifier: "_ZTSN4llvm17MachineBasicBlockE")
 !6 = !{}
 !7 = !DIExpression()
 !8 = !DILocation(line: 947, column: 35, scope: !2)
 !9 = distinct !DICompileUnit(language: DW_LANG_Julia, file: !3)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/dead-inst.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes='bdce,sroa<preserve-cfg>,bdce' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='bdce,sroa<modify-cfg>,bdce' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
+
 ; SROA fails to rewrite allocs but does rewrite some phis and delete
 ; dead instructions. Ensure that this invalidates analyses required
 ; for other passes.
-; RUN: opt < %s -passes=bdce,sroa,bdce -o %t -debug-pass-manager 2>&1 | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-n32:64"
 target triple = "powerpc64le-grtev4-linux-gnu"
 
 %class.b = type { i64 }
 
 declare void @D(ptr sret(%class.b), ptr dereferenceable(32)) local_unnamed_addr
 
 ; Function Attrs: nounwind
 define void @H(ptr noalias nocapture readnone, [2 x i64], ptr %ptr, i32 signext %v, i64 %l, i64 %idx, ptr nonnull dereferenceable(32) %ptr2) {
+; CHECK-LABEL: @H(
+; CHECK-NEXT:    [[TMP3:%.*]] = alloca [[CLASS_B:%.*]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = extractvalue [2 x i64] [[TMP1:%.*]], 1
+; CHECK-NEXT:    switch i64 [[TMP4]], label [[TMP6:%.*]] [
+; CHECK-NEXT:    i64 4, label [[FOO:%.*]]
+; CHECK-NEXT:    i64 5, label [[TMP5:%.*]]
+; CHECK-NEXT:    ]
+; CHECK:       5:
+; CHECK-NEXT:    br label [[TMP12:%.*]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = icmp ugt i64 [[TMP4]], 5
+; CHECK-NEXT:    br i1 [[TMP7]], label [[TMP8:%.*]], label [[TMP12]]
+; CHECK:       8:
+; CHECK-NEXT:    [[TMP9:%.*]] = load i8, ptr inttoptr (i64 4 to ptr), align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i8 [[TMP9]], 47
+; CHECK-NEXT:    [[TMP11:%.*]] = select i1 [[TMP10]], i64 5, i64 4
+; CHECK-NEXT:    br label [[TMP12]]
+; CHECK:       12:
+; CHECK-NEXT:    [[TMP13:%.*]] = phi i64 [ 4, [[TMP5]] ], [ [[TMP11]], [[TMP8]] ], [ 4, [[TMP6]] ]
+; CHECK-NEXT:    [[TMP14:%.*]] = icmp ne i64 [[TMP4]], 0
+; CHECK-NEXT:    [[TMP15:%.*]] = icmp ugt i64 [[TMP4]], [[TMP13]]
+; CHECK-NEXT:    [[TMP16:%.*]] = and i1 [[TMP14]], [[TMP15]]
+; CHECK-NEXT:    br i1 [[TMP16]], label [[TMP17:%.*]], label [[A_EXIT:%.*]]
+; CHECK:       17:
+; CHECK-NEXT:    [[TMP18:%.*]] = tail call ptr @memchr(ptr [[PTR:%.*]], i32 signext [[V:%.*]], i64 [[L:%.*]])
+; CHECK-NEXT:    [[TMP19:%.*]] = icmp eq ptr [[TMP18]], null
+; CHECK-NEXT:    [[TMP20:%.*]] = sext i1 [[TMP19]] to i64
+; CHECK-NEXT:    br label [[A_EXIT]]
+; CHECK:       a.exit:
+; CHECK-NEXT:    [[TMP21:%.*]] = phi i64 [ -1, [[TMP12]] ], [ [[TMP20]], [[TMP17]] ]
+; CHECK-NEXT:    [[TMP22:%.*]] = inttoptr i64 0 to ptr
+; CHECK-NEXT:    [[TMP23:%.*]] = sub nsw i64 [[TMP21]], [[TMP13]]
+; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 32, ptr nonnull [[TMP3]])
+; CHECK-NEXT:    [[TMP24:%.*]] = icmp ult i64 [[TMP23]], 2
+; CHECK-NEXT:    br i1 [[TMP24]], label [[G_EXIT:%.*]], label [[TMP25:%.*]]
+; CHECK:       25:
+; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds i8, ptr [[TMP22]], i64 [[IDX:%.*]]
+; CHECK-NEXT:    [[TMP27:%.*]] = icmp eq ptr [[TMP26]], null
+; CHECK-NEXT:    br i1 [[TMP27]], label [[TMP28:%.*]], label [[TMP29:%.*]]
+; CHECK:       28:
+; CHECK-NEXT:    unreachable
+; CHECK:       29:
+; CHECK-NEXT:    call void @D(ptr nonnull sret([[CLASS_B]]) [[TMP3]], ptr nonnull dereferenceable(32) [[PTR2:%.*]])
+; CHECK-NEXT:    br label [[G_EXIT]]
+; CHECK:       G.exit:
+; CHECK-NEXT:    call void @llvm.lifetime.end.p0(i64 32, ptr nonnull [[TMP3]])
+; CHECK-NEXT:    br label [[FOO]]
+; CHECK:       foo:
+; CHECK-NEXT:    ret void
+;
   %3 = alloca %class.b, align 8
   %.sroa.0 = alloca i64, align 8
   store i64 0, ptr %.sroa.0, align 8
   %4 = extractvalue [2 x i64] %1, 1
   switch i64 %4, label %6 [
   i64 4, label %foo
   i64 5, label %5
   ]

 declare ptr @memchr(ptr, i32 signext, i64) local_unnamed_addr
 
 ; Function Attrs: argmemonly nounwind
 declare void @llvm.lifetime.start.p0(i64, ptr nocapture)
 
 ; Function Attrs: argmemonly nounwind
 declare void @llvm.lifetime.end.p0(i64, ptr nocapture)
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
-; CHECK: {{.*}}
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/fca.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
+
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 define { i32, i32 } @test0(i32 %x, i32 %y, { i32, i32 } %v) {
 ; CHECK-LABEL: @test0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[V_FCA_0_EXTRACT:%.*]] = extractvalue { i32, i32 } [[V:%.*]], 0
 ; CHECK-NEXT:    [[V_FCA_1_EXTRACT:%.*]] = extractvalue { i32, i32 } [[V]], 1
 ; CHECK-NEXT:    [[RESULT_FCA_0_INSERT:%.*]] = insertvalue { i32, i32 } poison, i32 [[X:%.*]], 0
entry:
   store i32 %x, ptr %gep1
   %gep2 = getelementptr inbounds { i32, i32 }, ptr %a, i32 0, i32 1
   store i32 %y, ptr %gep2
 
   %result = load volatile { i32, i32 }, ptr %a
   store volatile { i32, i32 } %result, ptr %b
   ret { i32, i32 } %result
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/ignore-droppable.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 declare void @llvm.assume(i1)
 declare void @llvm.lifetime.start.p0(i64 %size, ptr nocapture %ptr)
 declare void @llvm.lifetime.end.p0(i64 %size, ptr nocapture %ptr)
 
 define void @positive_assume_uses(ptr %arg) {
 ; CHECK-LABEL: @positive_assume_uses(
 ; CHECK-NEXT:    call void @llvm.assume(i1 true) [ "nonnull"(ptr [[ARG:%.*]]), "ignore"(ptr undef, i64 2) ]
;
   call void @llvm.lifetime.start.p0(i64 2, ptr %A)
   call void @llvm.assume(i1 true) ["nonnull"(ptr %A), "align"(ptr %A, i64 8), "align"(ptr %A, i64 16)]
   store i8 1, ptr %A
   call void @llvm.lifetime.end.p0(i64 2, ptr %A)
   call void @llvm.assume(i1 true) ["nonnull"(ptr %A), "align"(ptr %A, i64 2), "nonnull"(ptr %A)]
   call void @llvm.assume(i1 true) ["nonnull"(ptr %A), "align"(ptr %A, i64 2), "nonnull"(ptr %A)]
   ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/invariant-group.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa -S -o - < %s | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' -S -o - < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' -S -o - < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %t = type { i32, i32 }
 
 declare ptr @llvm.launder.invariant.group.p0(ptr %a)
 declare ptr @llvm.strip.invariant.group.p0(ptr %a)
 declare void @h(i32 %a)
 declare i32 @somevalue()
 
;
   %a1_stripped = call ptr @llvm.strip.invariant.group.p0(ptr %a)
   %a1_int = ptrtoint ptr %a1_stripped to i32
   call void @h(i32 %a1_int)
 
   ret void
 }
 
 !0 = !{}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/irregular-type.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %S = type { [4 x i8] }
 
 ; Ensure the load/store of integer types whose size is not equal to the store
 ; size are not split.
 
 define i8 @foo(i23 %0) {
 ; CHECK-LABEL: @foo(

 ;
 Entry:
   %1 = alloca %S
   store i16 %0, ptr %1
   %2 = load i17, ptr %1
   %3 = zext i17 %2 to i32
   ret i32 %3
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/lifetime-intrinsic.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %i32x2 = type { [2 x i32] }
 
 ; Note %arr is the union
 ; union {
 ;   int i[2];
 ;   short s[4];
 ; };

 }
 
 declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #1
 
 declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #1
 
 attributes #0 = { alwaysinline nounwind }
 attributes #1 = { argmemonly nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/mem-par-metadata-sroa-cast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 ;
 ; Make sure the llvm.access.group meta-data is preserved
 ; when a load/store is replaced with another load/store by sroa
 ; Ensure this is done for casting too.
 ;
 
 %CMPLX = type { float, float }
 
entry:
   %dummy = sext i16 0 to i64
   %T = getelementptr %CMPLX, ptr %PART, i64 %dummy
   %X37 = load i64, ptr %T, align 8, !llvm.access.group !0
   store i64 %X37, ptr %PREV, align 8
   ret void
 }
 
 !0 = distinct !{}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 ;
 ; Make sure the llvm.access.group meta-data is preserved
 ; when a load/store is replaced with another load/store by sroa
 ;
 ; class Complex {
 ; private:
 ;  float real_;
 ;  float imaginary_;

 
 !0 = !{!"clang version 4.0.0 (cfe/trunk 277751)"}
 !1 = distinct !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
 !2 = !{!"llvm.loop.vectorize.enable", i1 true}
 !3 = !{!4}
 !4 = distinct !{!4, !5, !"_ZNK7ComplexplERKS_: %agg.result"}
 !5 = distinct !{!5, !"_ZNK7ComplexplERKS_"}
 !11 = distinct !{}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/non-capturing-call-readonly.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s --check-prefix=CHECK
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 define i32 @alloca_used_in_call(ptr %data, i64 %n) {
 ; CHECK-LABEL: @alloca_used_in_call(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:

 declare dso_local i32 @capture_of_alloca(ptr)
 declare dso_local i32 @capture_with_multiple_args(ptr nocapture readonly, ptr)
 
 declare dso_local void @byte_user_of_alloca(ptr nocapture readonly)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 
 declare void @llvm.memcpy.p0.p0.i32(ptr, ptr, i32, i1)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/non-integral-pointers.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa -S < %s | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; This test checks that SROA does not introduce ptrtoint and inttoptr
 ; casts from and to non-integral pointers.  The "ni:4" bit in the
 ; datalayout states that pointers of address space 4 are to be
 ; considered "non-integral".
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128-ni:4"
 target triple = "x86_64-unknown-linux-gnu"

 ;
   %1 = alloca %union.anon, align 8
   store ptr addrspace(4) %p, ptr %1, align 8
   %2 = load ptr, ptr %1, align 8
   ret ptr %2
 }
 
 declare void @llvm.memset.p0.i64(ptr, i8, i64, i1)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/phi-and-select.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 define i32 @test1() {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[COND:%.*]] = icmp sle i32 0, 1
 ; CHECK-NEXT:    br i1 [[COND]], label [[THEN:%.*]], label [[EXIT:%.*]]
 ; CHECK:       then:

 exit:
   %phi = phi ptr [ %a1x, %then ], [ %z, %entry ]
   %result = load i32, ptr %phi, align 4
   ret i32 %result
 }
 
 ; Don't speculate a load based on an earlier volatile operation.
 define i8 @volatile_select(ptr %p, i1 %b) {
-; CHECK-LABEL: @volatile_select(
-; CHECK-NEXT:    [[P2:%.*]] = alloca i8, align 1
-; CHECK-NEXT:    store i8 0, ptr [[P2]], align 1
-; CHECK-NEXT:    store volatile i8 0, ptr [[P:%.*]], align 1
-; CHECK-NEXT:    [[PX:%.*]] = select i1 [[B:%.*]], ptr [[P]], ptr [[P2]]
-; CHECK-NEXT:    [[V2:%.*]] = load i8, ptr [[PX]], align 1
-; CHECK-NEXT:    ret i8 [[V2]]
+; CHECK-preserve-cfg-LABEL: @volatile_select(
+; CHECK-preserve-cfg-NEXT:    [[P2:%.*]] = alloca i8, align 1
+; CHECK-preserve-cfg-NEXT:    store i8 0, ptr [[P2]], align 1
+; CHECK-preserve-cfg-NEXT:    store volatile i8 0, ptr [[P:%.*]], align 1
+; CHECK-preserve-cfg-NEXT:    [[PX:%.*]] = select i1 [[B:%.*]], ptr [[P]], ptr [[P2]]
+; CHECK-preserve-cfg-NEXT:    [[V2:%.*]] = load i8, ptr [[PX]], align 1
+; CHECK-preserve-cfg-NEXT:    ret i8 [[V2]]
+;
+; CHECK-modify-cfg-LABEL: @volatile_select(
+; CHECK-modify-cfg-NEXT:    store volatile i8 0, ptr [[P:%.*]], align 1
+; CHECK-modify-cfg-NEXT:    br i1 [[B:%.*]], label [[DOTTHEN:%.*]], label [[DOTCONT:%.*]]
+; CHECK-modify-cfg:       .then:
+; CHECK-modify-cfg-NEXT:    [[V2_THEN_VAL:%.*]] = load i8, ptr [[P]], align 1
+; CHECK-modify-cfg-NEXT:    br label [[DOTCONT]]
+; CHECK-modify-cfg:       .cont:
+; CHECK-modify-cfg-NEXT:    [[V2:%.*]] = phi i8 [ [[V2_THEN_VAL]], [[DOTTHEN]] ], [ 0, [[TMP0:%.*]] ]
+; CHECK-modify-cfg-NEXT:    ret i8 [[V2]]
 ;
   %p2 = alloca i8
   store i8 0, ptr %p2
   store volatile i8 0, ptr %p
   %px = select i1 %b, ptr %p, ptr %p2
   %v2 = load i8, ptr %px
   ret i8 %v2
 }

llvm/test/Transforms/SROA/phi-catchswitch.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
 
 %struct.foo = type { i32 }
 declare i32 @__gxx_wasm_personality_v0(...)
 declare void @foo()
 
bb9:                                              ; preds = %bb3
   unreachable
 
 bb10:                                             ; preds = %catch.dispatch, %entry
   %tmp11 = phi ptr [ %tmp, %entry ], [ %tmp5, %catch.dispatch ]
   %tmp12 = cleanuppad within none []
   store i32 0, ptr %tmp11, align 4
   unreachable
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/phi-gep.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa < %s | FileCheck %s
+; RUN: opt -S -passes='sroa<preserve-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %pair = type { i32, i32 }
 
 define i32 @test_sroa_phi_gep(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_phi_gep(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[COND:%.*]], label [[IF_THEN:%.*]], label [[END:%.*]]
 ; CHECK:       if.then:
bb.4:                                                ; preds = %bb.1, %bb.1, %bb.3, %bb.2
   ret i32 %load
 }
 
 declare ptr @foo()
 
 declare i32 @__gxx_personality_v0(...)
 
 declare void @llvm.memcpy.p0.p0.i64(ptr noalias nocapture writeonly, ptr noalias nocapture readonly, i64, i1 immarg)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/phi-speculate-different-load-types.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa < %s -S | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' < %s -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' < %s -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 define void @f(i1 %i) {
 ; CHECK-LABEL: @f(
 ; CHECK-NEXT:    [[A1:%.*]] = alloca i64, align 8
 ; CHECK-NEXT:    [[A2:%.*]] = alloca i64, align 8
 ; CHECK-NEXT:    br i1 [[I:%.*]], label [[BB1:%.*]], label [[BB:%.*]]
 ; CHECK:       bb:
 ; CHECK-NEXT:    br label [[BB2:%.*]]
bb2:
   %tmp4 = load i64, ptr %tmp3
   call void @use32(i32 %tmp5)
   call void @use64(i64 %tmp4)
   ret void
 }
 
 declare void @use32(i32)
 declare void @use64(i64)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/phi-with-duplicate-pred.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 @a = external global i16, align 1
 
 declare void @maybe_writes()
 
 define void @f2(i1 %c1) {
 ; CHECK-LABEL: @f2(

 final:
   call void @maybe_writes()
   %0 = load i16, ptr %g.0, align 1
   unreachable
 
 cleanup7:                                         ; preds = %cleanup
   ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/pointer-offset-size.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64:32"
 
 %struct.test = type { %struct.basic, %struct.basic }
 %struct.basic = type { i16, i8 }
 
 define i16 @test(ptr %ts2.i) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[S_SROA_0:%.*]] = alloca [3 x i8], align 8
 ; CHECK-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 1 [[TS2_I:%.*]], ptr align 8 [[S_SROA_0]], i32 3, i1 false)
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i16, ptr [[TS2_I]], align 2
 ; CHECK-NEXT:    ret i16 [[TMP0]]
 ;
 entry:
   %s = alloca %struct.test
   call void @llvm.memcpy.p0.p0.i32(ptr %ts2.i, ptr %s, i32 3, i1 false)
   %0 = load i16, ptr %ts2.i
   ret i16 %0
 }
 
 declare void @llvm.memcpy.p0.p0.i32(ptr nocapture writeonly, ptr nocapture readonly, i32, i1)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/ppcf128-no-fold.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "E-m:e-i64:64-n32:64"
 target triple = "powerpc64-unknown-linux-gnu"
 
 %struct.ld2 = type { [2 x ppc_fp128] }
 declare void @bar(ptr, [2 x i128])
 
 define void @foo(ptr %v) #0 {
 ; CHECK-LABEL: @foo(
entry:
   %1 = load [2 x i128], ptr %z, align 1
   call void @bar(ptr %0, [2 x i128] %1)
   ret void
 }
 
 
 attributes #0 = { nounwind }
 
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/pr26972.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-linux"
 
 ; Make sure we properly handle allocas where the allocated
 ; size overflows a uint32_t. This specific constant results in
 ; the size in bits being 32 after truncation to a 32-bit int.
 define void @fn1() {
 ; CHECK-LABEL: @fn1(
 ; CHECK-NEXT:    ret void
 ;
   %a = alloca [1073741825 x i32], align 16
   call void @llvm.lifetime.end.p0(i64 4294967300, ptr %a)
   ret void
 }
 
 declare void @llvm.lifetime.end.p0(i64, ptr nocapture)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/pr37267.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "E-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-n32:64-S128"
 target triple = "sparcv9-sun-solaris"
 
 ; PR37267
 ; Check that we don't crash on this test.
 
 define i16 @f1() {
 ; CHECK-LABEL: @f1(
; slice 3: [8,10)
   %_tmp13 = load i16, ptr %_tmp12, align 1
 ; slice 4: [2,4)
   %_tmp15 = getelementptr inbounds [6 x i16], ptr %a.3, i16 0, i16 1
   %_tmp16 = load i16, ptr %_tmp15, align 1
 
   %rc = add i16 %_tmp13, %_tmp16
   ret i16 %rc
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/preserve-nonnull.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 ;
 ; Make sure that SROA doesn't lose nonnull metadata
 ; on loads from allocas that get optimized out.
 
 declare void @llvm.memcpy.p0.p0.i64(ptr nocapture writeonly, ptr nocapture readonly, i64, i1)
 
 ; Check that we do basic propagation of nonnull when rewriting.
 define ptr @propagate_nonnull(ptr %v) {
entry:
   %a.gep1 = getelementptr [2 x ptr], ptr %a, i32 0, i32 1
   store i64 42, ptr %a.gep1
   store i64 0, ptr %a
   %load = load ptr, ptr %a.gep1, align 8, !nonnull !0
   ret ptr %load
 }
 
 !0 = !{}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/scalable-vectors.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; This test checks that SROA runs mem2reg on scalable vectors.
 
 define <vscale x 16 x i1> @alloca_nxv16i1(<vscale x 16 x i1> %pg) {
 ; CHECK-LABEL: @alloca_nxv16i1(
 ; CHECK-NEXT:    ret <vscale x 16 x i1> [[PG:%.*]]
 ;
   %pg.addr = alloca <vscale x 16 x i1>
;
   %retval = alloca <16 x i32>
   %retval.coerce = alloca <vscale x 4 x i32>
   call void @llvm.memcpy.p0.p0.i64(ptr align 16 %retval.coerce, ptr align 16 %retval, i64 64, i1 false)
   %1 = load <vscale x 4 x i32>, ptr %retval.coerce
   ret <vscale x 4 x i32> %1
 }
 
 declare void @llvm.memcpy.p0.p0.i64(ptr nocapture, ptr nocapture, i64, i1) nounwind
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/select-gep.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa < %s | FileCheck %s
+; RUN: opt -S -passes='sroa<preserve-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %pair = type { i32, i32 }
 
 define i32 @test_sroa_select_gep(i1 %cond) {
 ; CHECK-LABEL: @test_sroa_select_gep(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[LOAD_SROA_SPECULATED:%.*]] = select i1 [[COND:%.*]], i32 1, i32 2
 ; CHECK-NEXT:    ret i32 [[LOAD_SROA_SPECULATED]]
bb:
   %load1 = load volatile i32, ptr %select, align 4
   %gep2 = getelementptr inbounds %pair, ptr %select, i32 0, i32 1
   %load2 = load volatile i32, ptr %gep2, align 4
   %add = add i32 %load1, %load2
   ret i32 %add
 }
 
 define i32 @test_sroa_select_gep_poison(i1 %cond) {
-; CHECK-LABEL: @test_sroa_select_gep_poison(
-; CHECK-NEXT:  bb:
-; CHECK-NEXT:    [[A_SROA_0:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[SELECT_SROA_SEL:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr poison
-; CHECK-NEXT:    [[LOAD:%.*]] = load i32, ptr [[SELECT_SROA_SEL]], align 4
-; CHECK-NEXT:    ret i32 [[LOAD]]
+; CHECK-preserve-cfg-LABEL: @test_sroa_select_gep_poison(
+; CHECK-preserve-cfg-NEXT:  bb:
+; CHECK-preserve-cfg-NEXT:    [[A_SROA_0:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    [[SELECT_SROA_SEL:%.*]] = select i1 [[COND:%.*]], ptr [[A_SROA_0]], ptr poison
+; CHECK-preserve-cfg-NEXT:    [[LOAD:%.*]] = load i32, ptr [[SELECT_SROA_SEL]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[LOAD]]
+;
+; CHECK-modify-cfg-LABEL: @test_sroa_select_gep_poison(
+; CHECK-modify-cfg-NEXT:  bb:
+; CHECK-modify-cfg-NEXT:    br i1 [[COND:%.*]], label [[BB_CONT:%.*]], label [[BB_ELSE:%.*]]
+; CHECK-modify-cfg:       bb.else:
+; CHECK-modify-cfg-NEXT:    [[LOAD_ELSE_VAL:%.*]] = load i32, ptr poison, align 4
+; CHECK-modify-cfg-NEXT:    br label [[BB_CONT]]
+; CHECK-modify-cfg:       bb.cont:
+; CHECK-modify-cfg-NEXT:    [[LOAD:%.*]] = phi i32 [ undef, [[BB:%.*]] ], [ [[LOAD_ELSE_VAL]], [[BB_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[LOAD]]
 ;
 bb:
   %a = alloca %pair, align 4
   %select = select i1 %cond, ptr %a, ptr poison
   %gep = getelementptr inbounds %pair, ptr %select, i32 0, i32 1
   %load = load i32, ptr %gep, align 4
   ret i32 %load
 }

llvm/test/Transforms/SROA/select-load.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa < %s | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %st.half = type { half }
 
 ; Allow speculateSelectInstLoads to fold load and select
 ; even if there is an intervening bitcast.
 define <2 x i16> @test_load_bitcast_select(i1 %cond1, i1 %cond2) {
 ; CHECK-LABEL: @test_load_bitcast_select(
 ; CHECK-NEXT:  entry:
;
   %cmp.i.i = icmp slt i32 %i1, 0
   %__b.__a.i.i = select i1 %cmp.i.i, ptr %min, ptr %arrayidx
   %i3 = load i32, ptr %__b.__a.i.i, align 4
   ret i32 %i3
 }
 
 ; We should recursively evaluate select's.
 define i32 @clamp_load_to_constant_range(ptr %data, i64 %indvars.iv) {
-; CHECK-LABEL: @clamp_load_to_constant_range(
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV:%.*]]
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MIN]])
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MAX]])
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
-; CHECK-NEXT:    [[I1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[CMP_I_I:%.*]] = icmp slt i32 [[I1]], 0
-; CHECK-NEXT:    [[I2:%.*]] = tail call i32 @llvm.smax.i32(i32 [[I1]], i32 0)
-; CHECK-NEXT:    [[__B___A_I_I:%.*]] = select i1 [[CMP_I_I]], ptr [[MIN]], ptr [[ARRAYIDX]]
-; CHECK-NEXT:    [[CMP_I1_I:%.*]] = icmp ugt i32 [[I2]], 4095
-; CHECK-NEXT:    [[__B___A_I2_I:%.*]] = select i1 [[CMP_I1_I]], ptr [[MAX]], ptr [[__B___A_I_I]]
-; CHECK-NEXT:    [[I3:%.*]] = load i32, ptr [[__B___A_I2_I]], align 4
-; CHECK-NEXT:    ret i32 [[I3]]
+; CHECK-preserve-cfg-LABEL: @clamp_load_to_constant_range(
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    [[MAX:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV:%.*]]
+; CHECK-preserve-cfg-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MIN]])
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    call void @llvm.lifetime.start.p0(i64 4, ptr [[MAX]])
+; CHECK-preserve-cfg-NEXT:    store i32 4095, ptr [[MAX]], align 4
+; CHECK-preserve-cfg-NEXT:    [[I1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-preserve-cfg-NEXT:    [[CMP_I_I:%.*]] = icmp slt i32 [[I1]], 0
+; CHECK-preserve-cfg-NEXT:    [[I2:%.*]] = tail call i32 @llvm.smax.i32(i32 [[I1]], i32 0)
+; CHECK-preserve-cfg-NEXT:    [[__B___A_I_I:%.*]] = select i1 [[CMP_I_I]], ptr [[MIN]], ptr [[ARRAYIDX]]
+; CHECK-preserve-cfg-NEXT:    [[CMP_I1_I:%.*]] = icmp ugt i32 [[I2]], 4095
+; CHECK-preserve-cfg-NEXT:    [[__B___A_I2_I:%.*]] = select i1 [[CMP_I1_I]], ptr [[MAX]], ptr [[__B___A_I_I]]
+; CHECK-preserve-cfg-NEXT:    [[I3:%.*]] = load i32, ptr [[__B___A_I2_I]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[I3]]
+;
+; CHECK-modify-cfg-LABEL: @clamp_load_to_constant_range(
+; CHECK-modify-cfg-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, ptr [[DATA:%.*]], i64 [[INDVARS_IV:%.*]]
+; CHECK-modify-cfg-NEXT:    [[I1:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-modify-cfg-NEXT:    [[CMP_I_I:%.*]] = icmp slt i32 [[I1]], 0
+; CHECK-modify-cfg-NEXT:    [[I2:%.*]] = tail call i32 @llvm.smax.i32(i32 [[I1]], i32 0)
+; CHECK-modify-cfg-NEXT:    [[CMP_I1_I:%.*]] = icmp ugt i32 [[I2]], 4095
+; CHECK-modify-cfg-NEXT:    br i1 [[CMP_I1_I]], label [[DOTCONT:%.*]], label [[DOTELSE:%.*]]
+; CHECK-modify-cfg:       .else:
+; CHECK-modify-cfg-NEXT:    br i1 [[CMP_I_I]], label [[DOTELSE_CONT:%.*]], label [[DOTELSE_ELSE:%.*]]
+; CHECK-modify-cfg:       .else.else:
+; CHECK-modify-cfg-NEXT:    [[I3_ELSE_VAL_ELSE_VAL:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
+; CHECK-modify-cfg-NEXT:    br label [[DOTELSE_CONT]]
+; CHECK-modify-cfg:       .else.cont:
+; CHECK-modify-cfg-NEXT:    [[I3_ELSE_VAL:%.*]] = phi i32 [ 0, [[DOTELSE]] ], [ [[I3_ELSE_VAL_ELSE_VAL]], [[DOTELSE_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    br label [[DOTCONT]]
+; CHECK-modify-cfg:       .cont:
+; CHECK-modify-cfg-NEXT:    [[I3:%.*]] = phi i32 [ 4095, [[TMP0:%.*]] ], [ [[I3_ELSE_VAL]], [[DOTELSE_CONT]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[I3]]
 ;
   %min = alloca i32, align 4
   %max = alloca i32, align 4
   %arrayidx = getelementptr inbounds i32, ptr %data, i64 %indvars.iv
   call void @llvm.lifetime.start.p0(i64 4, ptr %min)
   store i32 0, ptr %min, align 4
   call void @llvm.lifetime.start.p0(i64 4, ptr %max)
   store i32 4095, ptr %max, align 4
   %i1 = load i32, ptr %arrayidx, align 4
   %cmp.i.i = icmp slt i32 %i1, 0
   %i2 = tail call i32 @llvm.smax.i32(i32 %i1, i32 0)
   %__b.__a.i.i = select i1 %cmp.i.i, ptr %min, ptr %arrayidx
   %cmp.i1.i = icmp ugt i32 %i2, 4095
   %__b.__a.i2.i = select i1 %cmp.i1.i, ptr %max, ptr %__b.__a.i.i
   %i3 = load i32, ptr %__b.__a.i2.i, align 4
   ret i32 %i3
 }
 
 define i32 @non_speculatable_load_of_select(i1 %cond, ptr %else.addr) {
-; CHECK-LABEL: @non_speculatable_load_of_select(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0:![0-9]+]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0:![0-9]+]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_CONT:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF0:![0-9]+]]
+; CHECK-modify-cfg:       entry.else:
+; CHECK-modify-cfg-NEXT:    [[R_ELSE_VAL:%.*]] = load i32, ptr [[ELSE_ADDR:%.*]], align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[R_ELSE_VAL]], [[ENTRY_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr = select i1 %cond, ptr %min, ptr %else.addr, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_load_of_select_inverted(i1 %cond, ptr %then.addr) {
-; CHECK-LABEL: @non_speculatable_load_of_select_inverted(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select_inverted(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MAX:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 4095, ptr [[MAX]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select_inverted(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_CONT:%.*]], !prof [[PROF1:![0-9]+]]
+; CHECK-modify-cfg:       entry.then:
+; CHECK-modify-cfg-NEXT:    [[R_THEN_VAL:%.*]] = load i32, ptr [[THEN_ADDR:%.*]], align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ [[R_THEN_VAL]], [[ENTRY_THEN]] ], [ 4095, [[ENTRY:%.*]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %max = alloca i32, align 4
   store i32 4095, ptr %max, align 4
   %addr = select i1 %cond, ptr %then.addr, ptr %max, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 
 define i32 @non_speculatable_volatile_load_of_select(i1 %cond, ptr %else.addr) {
-; CHECK-LABEL: @non_speculatable_volatile_load_of_select(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_volatile_load_of_select(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_volatile_load_of_select(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-modify-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-modify-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr = select i1 %cond, ptr %min, ptr %else.addr, !prof !0
   %r = load volatile i32, ptr %addr, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_volatile_load_of_select_inverted(i1 %cond, ptr %then.addr) {
-; CHECK-LABEL: @non_speculatable_volatile_load_of_select_inverted(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_volatile_load_of_select_inverted(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MAX:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 4095, ptr [[MAX]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_volatile_load_of_select_inverted(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[MAX:%.*]] = alloca i32, align 4
+; CHECK-modify-cfg-NEXT:    store i32 4095, ptr [[MAX]], align 4
+; CHECK-modify-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = load volatile i32, ptr [[ADDR]], align 4
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %max = alloca i32, align 4
   store i32 4095, ptr %max, align 4
   %addr = select i1 %cond, ptr %then.addr, ptr %max, !prof !0
   %r = load volatile i32, ptr %addr, align 4
   ret i32 %r
 }
 
 define i32 @non_speculatable_atomic_unord_load_of_select(i1 %cond, ptr %else.addr) {
-; CHECK-LABEL: @non_speculatable_atomic_unord_load_of_select(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load atomic i32, ptr [[ADDR]] unordered, align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_atomic_unord_load_of_select(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[MIN]], ptr [[ELSE_ADDR:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load atomic i32, ptr [[ADDR]] unordered, align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_atomic_unord_load_of_select(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg:       entry.then:
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK-modify-cfg:       entry.else:
+; CHECK-modify-cfg-NEXT:    [[R_ELSE_VAL:%.*]] = load atomic i32, ptr [[ELSE_ADDR:%.*]] unordered, align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ 0, [[ENTRY_THEN]] ], [ [[R_ELSE_VAL]], [[ENTRY_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr = select i1 %cond, ptr %min, ptr %else.addr, !prof !0
   %r = load atomic i32, ptr %addr unordered, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_atomic_unord_load_of_select_inverted(i1 %cond, ptr %then.addr) {
-; CHECK-LABEL: @non_speculatable_atomic_unord_load_of_select_inverted(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MAX:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 4095, ptr [[MAX]], align 4
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load atomic i32, ptr [[ADDR]] unordered, align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_atomic_unord_load_of_select_inverted(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MAX:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 4095, ptr [[MAX]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND:%.*]], ptr [[THEN_ADDR:%.*]], ptr [[MAX]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load atomic i32, ptr [[ADDR]] unordered, align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_atomic_unord_load_of_select_inverted(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    br i1 [[COND:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg:       entry.then:
+; CHECK-modify-cfg-NEXT:    [[R_THEN_VAL:%.*]] = load atomic i32, ptr [[THEN_ADDR:%.*]] unordered, align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT:%.*]]
+; CHECK-modify-cfg:       entry.else:
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ [[R_THEN_VAL]], [[ENTRY_THEN]] ], [ 4095, [[ENTRY_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %max = alloca i32, align 4
   store i32 4095, ptr %max, align 4
   %addr = select i1 %cond, ptr %then.addr, ptr %max, !prof !0
   %r = load atomic i32, ptr %addr unordered, align 4
   ret i32 %r
 }
 
 define i32 @non_speculatable_load_of_select_outer(i1 %cond_inner, i1 %cond_outer, ptr %data_then, ptr %data_else) {
-; CHECK-LABEL: @non_speculatable_load_of_select_outer(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN]], ptr [[ADDR_DATA]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select_outer(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN]], ptr [[ADDR_DATA]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select_outer(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    br i1 [[COND_OUTER:%.*]], label [[ENTRY_CONT:%.*]], label [[ENTRY_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-modify-cfg:       entry.else:
+; CHECK-modify-cfg-NEXT:    [[R_ELSE_VAL:%.*]] = load i32, ptr [[ADDR_DATA]], align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[R_ELSE_VAL]], [[ENTRY_ELSE]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr.data = select i1 %cond_inner, ptr %data_then, ptr %data_else, !prof !0
   %addr = select i1 %cond_outer, ptr %min, ptr %addr.data, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_load_of_select_outer_inverted(i1 %cond_inner, i1 %cond_outer, ptr %data_then, ptr %data_else) {
-; CHECK-LABEL: @non_speculatable_load_of_select_outer_inverted(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[ADDR_DATA]], ptr [[MIN]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select_outer_inverted(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[ADDR_DATA]], ptr [[MIN]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select_outer_inverted(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[DATA_THEN:%.*]], ptr [[DATA_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    br i1 [[COND_OUTER:%.*]], label [[ENTRY_THEN:%.*]], label [[ENTRY_CONT:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg:       entry.then:
+; CHECK-modify-cfg-NEXT:    [[R_THEN_VAL:%.*]] = load i32, ptr [[ADDR_DATA]], align 4
+; CHECK-modify-cfg-NEXT:    br label [[ENTRY_CONT]]
+; CHECK-modify-cfg:       entry.cont:
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = phi i32 [ [[R_THEN_VAL]], [[ENTRY_THEN]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %addr.data = select i1 %cond_inner, ptr %data_then, ptr %data_else, !prof !0
   %addr = select i1 %cond_outer, ptr %addr.data, ptr %min, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 
 define i32 @non_speculatable_load_of_select_inner(i1 %cond_inner, i1 %cond_outer, ptr %data_else, ptr %min_else) {
-; CHECK-LABEL: @non_speculatable_load_of_select_inner(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN]], ptr [[MIN_ELSE:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select_inner(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN]], ptr [[MIN_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select_inner(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-modify-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-modify-cfg-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN]], ptr [[MIN_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %min.addr.data = select i1 %cond_inner, ptr %min, ptr %min_else, !prof !0
   %addr = select i1 %cond_outer, ptr %min.addr.data, ptr %data_else, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r
 }
 define i32 @non_speculatable_load_of_select_inner_inverted(i1 %cond_inner, i1 %cond_outer, ptr %data_else, ptr %min_then) {
-; CHECK-LABEL: @non_speculatable_load_of_select_inner_inverted(
-; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[MIN:%.*]] = alloca i32, align 4
-; CHECK-NEXT:    store i32 0, ptr [[MIN]], align 4
-; CHECK-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN_THEN:%.*]], ptr [[MIN]], !prof [[PROF0]]
-; CHECK-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
-; CHECK-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
-; CHECK-NEXT:    ret i32 [[R]]
+; CHECK-preserve-cfg-LABEL: @non_speculatable_load_of_select_inner_inverted(
+; CHECK-preserve-cfg-NEXT:  entry:
+; CHECK-preserve-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-preserve-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-preserve-cfg-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN_THEN:%.*]], ptr [[MIN]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF0]]
+; CHECK-preserve-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-preserve-cfg-NEXT:    ret i32 [[R]]
+;
+; CHECK-modify-cfg-LABEL: @non_speculatable_load_of_select_inner_inverted(
+; CHECK-modify-cfg-NEXT:  entry:
+; CHECK-modify-cfg-NEXT:    [[MIN:%.*]] = alloca i32, align 4
+; CHECK-modify-cfg-NEXT:    store i32 0, ptr [[MIN]], align 4
+; CHECK-modify-cfg-NEXT:    [[MIN_ADDR_DATA:%.*]] = select i1 [[COND_INNER:%.*]], ptr [[MIN_THEN:%.*]], ptr [[MIN]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[ADDR:%.*]] = select i1 [[COND_OUTER:%.*]], ptr [[MIN_ADDR_DATA]], ptr [[DATA_ELSE:%.*]], !prof [[PROF1]]
+; CHECK-modify-cfg-NEXT:    [[R:%.*]] = load i32, ptr [[ADDR]], align 4
+; CHECK-modify-cfg-NEXT:    ret i32 [[R]]
 ;
 entry:
   %min = alloca i32, align 4
   store i32 0, ptr %min, align 4
   %min.addr.data = select i1 %cond_inner, ptr %min_then, ptr %min, !prof !0
   %addr = select i1 %cond_outer, ptr %min.addr.data, ptr %data_else, !prof !0
   %r = load i32, ptr %addr, align 4
   ret i32 %r

llvm/test/Transforms/SROA/slice-order-independence.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-p1:16:16:16-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.memcpy.p0.p0.i32(ptr nocapture, ptr nocapture, i32, i1) nounwind
 
 ; Check that the chosen type for a split is independent from the order of
 ; slices even in case of types that are skipped because their width is not a
 ; byte width multiple
 define void @skipped_inttype_first(ptr) {

 ;
   %arg = alloca { ptr, i32 }, align 8
   call void @llvm.memcpy.p0.p0.i32(ptr align 8 %arg, ptr align 8 %0, i32 16, i1 false)
   %b = getelementptr inbounds { ptr, i32 }, ptr %arg, i64 0, i32 0
   %b1 = load ptr, ptr %b
   %b0 = load i63, ptr %b
   ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/slice-width.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-p1:16:16:16-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-f80:128-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 declare void @llvm.memcpy.p0.p0.i32(ptr nocapture, ptr nocapture, i32, i1) nounwind
 declare void @llvm.memset.p0.i32(ptr nocapture, i8, i32, i1) nounwind
 declare void @llvm.memset.p0.i64(ptr nocapture, i8, i64, i1) nounwind
 
 ; This tests that allocas are not split into slices that are not byte width multiple
 define void @no_split_on_non_byte_width(i32) {

 ; CHECK-NEXT:    [[A_SROA_0_0_A_SROA_0_0_L2:%.*]] = load i1, ptr [[A_SROA_0]], align 2
 ; CHECK-NEXT:    ret i1 [[A_SROA_0_0_A_SROA_0_0_L2]]
 ;
   %A = alloca i16
   %L1 = load i32, ptr %A
   %L2 = load i1, ptr %A
   ret i1 %L2
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa -S < %s | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 %"struct.a" = type { <8 x half> }
 %"struct.b" = type { %"struct.a" }
 %"struct.c" = type { %"struct.a", i32, i8 }
 %"struct.d" = type { [4 x i32], %"struct.a" }
 %"struct.e" = type { [2 x <8 x half>], i32, i32 }
 %"struct.f" = type { [2 x <8 x i16>], i32, i32 }
 %"array.a" = type [2 x <8 x half>]
bb:
   ret void
 }
 
 declare <4 x float> @llvm.amdgcn.raw.buffer.load.v4f32(<4 x i32>, i32, i32, i32) #0
 declare void @llvm.memset.p0.i64(ptr nocapture writeonly, i8, i64, i1) nounwind
 declare void @llvm.memset.p0.i32(ptr nocapture writeonly, i8, i32, i1) nounwind
 attributes #0 = { nounwind readonly }
 
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/std-clamp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 
 define float @_Z8stdclampfff(float %x, float %lo, float %hi) {
 ; CHECK-LABEL: @_Z8stdclampfff(
 ; CHECK-NEXT:  bb:
 ; CHECK-NEXT:    [[I4:%.*]] = alloca float, align 4
 ; CHECK-NEXT:    store float [[HI:%.*]], ptr [[I4]], align 4
bb:
   store float %hi, ptr %i4, align 4
   %i5 = fcmp fast olt float %x, %lo
   %i6 = fcmp fast olt float %hi, %x
   %i7 = select i1 %i6, ptr %i4, ptr %i
   %i8 = select i1 %i5, ptr %i3, ptr %i7
   %i9 = load float, ptr %i8, align 4
   ret float %i9
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/tbaa-struct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa %s | FileCheck %s
+; RUN: opt -S -passes='sroa<preserve-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; SROA should keep `!tbaa.struct` metadata
 
 %vector = type { float, float }
 declare void @llvm.memcpy.p0.p0.i64(ptr writeonly, ptr readonly, i64, i1 immarg)
 declare <2 x float> @foo(ptr %0)
 
 define void @bar(ptr %y2) {

 }
 
 !4 = !{!"omnipotent char", !5, i64 0}
 !5 = !{!"Simple C++ TBAA"}
 !7 = !{!"vector", !8, i64 0, !8, i64 4}
 !8 = !{!"float", !4, i64 0}
 !10 = !{i64 0, i64 4, !11, i64 4, i64 4, !11}
 !11 = !{!8, !8, i64 0}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/tbaa-struct2.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa %s | FileCheck %s
+; RUN: opt -S -passes='sroa<preserve-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; SROA should correctly offset `!tbaa.struct` metadata
 
 %struct.Wishart = type { double, i32 }
 declare void @llvm.memcpy.p0.p0.i64(ptr writeonly, ptr readonly, i64, i1 immarg)
 declare double @subcall(double %g, i32 %m)
 
 define double @bar(ptr %wishart) {

 
 !2 = !{i64 0, i64 8, !3, i64 8, i64 4, !7}
 !3 = !{!4, !4, i64 0}
 !4 = !{!"double", !5, i64 0}
 !5 = !{!"omnipotent char", !6, i64 0}
 !6 = !{!"Simple C++ TBAA"}
 !7 = !{!8, !8, i64 0}
 !8 = !{!"int", !5, i64 0}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/tbaa-subload.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -passes=sroa %s | FileCheck %s
+; RUN: opt -S -passes='sroa<preserve-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; This should not crash
 
 %class.ar = type { i8 }
 %class.anon = type <{ %class.ar, [7 x i8], { i64, i64 } }>
 
 define void @caller() {
 ; CHECK-LABEL: @caller(

 declare void @use(ptr %this)
 
 !3 = !{!"omnipotent char", !4, i64 0}
 !4 = !{!"Simple C++ TBAA"}
 
 !7 = !{!8, !3, i64 8}
 !8 = !{!"_ZTSZN2ax2baEMS_FvvE2an2arE3$_0", !9, i64 0, !3, i64 8}
 !9 = !{!"_ZTS2ar"}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/vector-conversion.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 define <4 x i64> @vector_ptrtoint({<2 x ptr>, <2 x ptr>} %x) {
 ; CHECK-LABEL: @vector_ptrtoint(
 ; CHECK-NEXT:    [[X_FCA_0_EXTRACT:%.*]] = extractvalue { <2 x ptr>, <2 x ptr> } [[X:%.*]], 0
 ; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint <2 x ptr> [[X_FCA_0_EXTRACT]] to <2 x i64>
 ; CHECK-NEXT:    [[A_SROA_0_0_VEC_EXPAND:%.*]] = shufflevector <2 x i64> [[TMP1]], <2 x i64> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[A_SROA_0_0_VECBLEND:%.*]] = select <4 x i1> <i1 true, i1 true, i1 false, i1 false>, <4 x i64> [[A_SROA_0_0_VEC_EXPAND]], <4 x i64> undef
;
   %a = alloca {<2 x ptr>, <2 x ptr>}
 
   store {<2 x ptr>, <2 x ptr>} %x, ptr %a
 
   %vec = load <16 x i8>, ptr %a
 
   ret <16 x i8> %vec
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/vector-lifetime-intrinsic.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=sroa -S < %s | FileCheck %s
+; RUN: opt -passes='sroa<preserve-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -passes='sroa<modify-cfg>' -S < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 target datalayout = "e-p:64:32-i64:32-v32:32-n32-S64"
 
 ; Function Attrs: nounwind
 declare void @llvm.lifetime.start.p0(i64, ptr nocapture) #0
 
 ; Function Attrs: nounwind
 declare void @llvm.lifetime.end.p0(i64, ptr nocapture) #0
bb:
   call void @wombat3(<3 x float> %tmp18)
   ret void
 }
 
 ; Function Attrs: nounwind
 declare void @wombat3(<3 x float>) #0
 
 attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/vector-promotion-different-size.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 define <4 x i1> @vector_bitcast() {
 ; CHECK-LABEL: @vector_bitcast(
 ; CHECK-NEXT:    [[A:%.*]] = alloca <3 x i1>, align 1
 ; CHECK-NEXT:    store <3 x i1> <i1 true, i1 false, i1 true>, ptr [[A]], align 1
 ; CHECK-NEXT:    [[A_0_VEC:%.*]] = load <4 x i1>, ptr [[A]], align 1
 ; CHECK-NEXT:    ret <4 x i1> [[A_0_VEC]]

 ; CHECK-NEXT:    [[P_0_LOAD:%.*]] = load <64 x i16>, ptr [[P]], align 64
 ; CHECK-NEXT:    ret <64 x i16> [[P_0_LOAD]]
 ;
   %p = alloca <32 x i16>
   store <32 x i16> %v, ptr %p
   %load = load <64 x i16>, ptr %p
   ret <64 x i16> %load
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/vector-promotion.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa -S | FileCheck %s
+; RUN: opt < %s -passes='sroa<preserve-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt < %s -passes='sroa<modify-cfg>' -S | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n8:16:32:64"
 
 %S1 = type { i64, [42 x float] }
 
 define i32 @test1(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[A_SROA_0_8_VEC_EXTRACT:%.*]] = extractelement <4 x i32> [[X:%.*]], i32 2
;
   call void @llvm.memcpy.p0.p0.i64(ptr %tmp, ptr %x, i64 15, i1 false)
   tail call void @llvm.memcpy.p0.p0.i64(ptr %x, ptr %y, i64 15, i1 false)
   call void @llvm.memcpy.p0.p0.i64(ptr %y, ptr %tmp, i64 15, i1 false)
   ret void
 }
 
 declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1)
 declare void @llvm.lifetime.end.p0(i64, ptr)
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}

llvm/test/Transforms/SROA/vectors-of-pointers.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=sroa
+; RUN: opt -S -passes='sroa<preserve-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-preserve-cfg
+; RUN: opt -S -passes='sroa<modify-cfg>' < %s | FileCheck %s --check-prefixes=CHECK,CHECK-modify-cfg
 
 ; Make sure we don't crash on this one.
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.8.0"
 
 define void @foo(i1 %c1, i1 %c2) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 [[C1:%.*]], label [[BB0_EXIT158:%.*]], label [[IF_THEN_I_I_I_I_I138:%.*]]
+; CHECK:       if.then.i.i.i.i.i138:
+; CHECK-NEXT:    ret void
+; CHECK:       bb0.exit158:
+; CHECK-NEXT:    br i1 [[C2:%.*]], label [[BB0_EXIT257:%.*]], label [[IF_THEN_I_I_I_I_I237:%.*]]
+; CHECK:       if.then.i.i.i.i.i237:
+; CHECK-NEXT:    unreachable
+; CHECK:       bb0.exit257:
+; CHECK-NEXT:    ret void
+;
 entry:
   %Args.i = alloca <2 x ptr>, align 16
   br i1 %c1, label %bb0.exit158, label %if.then.i.i.i.i.i138
 
 if.then.i.i.i.i.i138:
   ret void
 
 bb0.exit158:
   br i1 %c2, label %bb0.exit257, label %if.then.i.i.i.i.i237
 
 if.then.i.i.i.i.i237:
   unreachable
 
 bb0.exit257:
   %0 = load <2 x ptr>, ptr %Args.i, align 16
   ret void
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK-modify-cfg: {{.*}}
+; CHECK-preserve-cfg: {{.*}}