[Phase Ordering] Don't speculate in SimplifyCFG before PGO annotation

[tejohnson]

Summary

SimplifyCFG contains a number of transformations that perform
speculation, and will use profile data if it exists in the IR.
However, the first invocations of SimplifyCFG are before both
instrumented and sample PGO annotation. We found cases where
unprofitable speculation was performed in these early invocations, when
subsequent PGO profile data would have prevented it.

Disable the SpeculateBlocks flag before PGO annotation, in compiles
where PGO is provided.

This change resulted in noticeable and significant performance
improvements in multiple large internal codes.

Finally, remove an old test that had a difference with this change
(clang/test/CodeGen/pgo-sample-preparation.c), however, was testing
a pipeline configuration that no longer exists in the same way as when
the test was added. The code change being checked does not seem
indicative of a performance issue as the critical path remains the same,
and I could not find any description of a performance issue with the
original commit.

[aeubanks]

the pipeline change and simplifycfg change should be split into two changes

llvm/lib/Passes/PassBuilderPipelines.cpp

277

can we just drop the flag and make this change?

llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll

3

these checks won't work for a update_test_checks.py test. should add tests like the one added in https://reviews.llvm.org/D153391

llvm/test/Transforms/SimplifyCFG/pipeline-delay-speculation-pgo.ll

1

do you have a phase ordering test instead?

[tejohnson]

llvm/lib/Passes/PassBuilderPipelines.cpp

277

Ok, I will remove.

llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll

3

Can you clarify - are you saying remove the changes from this test and add a new test for this change like llvm/test/Transforms/SimplifyCFG/speculate-blocks.ll, or change the checks here?

llvm/test/Transforms/SimplifyCFG/pipeline-delay-speculation-pgo.ll

1

Can you clarify what you are referring to?

[aeubanks]

llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll

3

revert the changes here (update_test_checks.py tests shouldn't have manually added CHECK lines) and add new function(s) to llvm/test/Transforms/SimplifyCFG/speculate-blocks.ll that show the difference between speculate-blocks and no-speculate-blocks

llvm/test/Transforms/SimplifyCFG/pipeline-delay-speculation-pgo.ll

1

llvm/test/Transforms/PhaseOrdering has IR tests that get run through an entire pipeline (e.g. thinlto-pre-link<O3>) to verify some property of the entire pipeline. ideally you can get some somewhat-reduced IR that exhibits the problem you were seeing before the change and show that it improves with the pipeline change. you'll need a profile as part of that test I suppose

[tejohnson]

In D155997#4529861, @aeubanks wrote:

the pipeline change and simplifycfg change should be split into two changes

SimplifyCFG change split into D156194.

[tejohnson]

Address comments

[aeubanks]

llvm/test/Transforms/PhaseOrdering/simplifycfg-speculate-blocks.ll

1 ↗

(On Diff #544109)

hmm typically these phase ordering tests use llvm/utils/update_test_checks.py, but that doesn't support the llvm-profdata RUN line. I think a non-update_test_checks test is probably fine for this, @nikic does that make sense?

22 ↗

(On Diff #544109)

lto-pre-link

83 ↗

(On Diff #544109)

the debug info is necessary for sample profile to work I presume?

I think some of this can still be simplified, like PIC Level, etc

[nikic]

I'm not a fan of the PGO conditional behavior here. I'd prefer to disable speculation unconditionally if that is feasible. This is basically what D153392 did. While the specific test case there was addressed in a different way, if we take it in conjunction with this one, I think we have enough motivation to do the change.

Another piece of motivation would be that the EarlyFPM change would allow us to move LowerExpectIntrinsic to the end of EarlyFPM again, which would make expect intrinsics work in Rust again. The pass was moved earlier to make the first SimplifyCFG run take them into account, but that would no longer be a problem if we disable speculation.

And another piece of motivation would be that we could move SimplifyCFG after SROA, where it can be much more effective (but doing so with speculation breaks IPSCCP too much). I think doing that needs some further changes, but it's a move in the right direction.

So I think the EarlyFPM change at least seems like something we should pretty clearly do independently of PGO. The addPGOInstrPasses() change is of course PGO specific anyway. I think only the GlobalCleanupPM change may not be completely clear cut.

llvm/test/Transforms/PhaseOrdering/simplifycfg-speculate-blocks.ll

1 ↗

(On Diff #544109)

I thought UTC supports unrecognized commands (by just executing them)... If not, a non-UTC test is fine.

[tejohnson]

In D155997#4543156, @nikic wrote:

I'm not a fan of the PGO conditional behavior here. I'd prefer to disable speculation unconditionally if that is feasible. This is basically what D153392 did. While the specific test case there was addressed in a different way, if we take it in conjunction with this one, I think we have enough motivation to do the change.

It looks like D153392 does roughly the same disabling in practice, but you are right it doesn't gate on whether we will eventually feed back profile info. For the issue I was trying to solve, it was because of the PGO based profitability checks in SimplifyCFG speculation, but if it makes sense in other cases to disable it until later, then that works for me. BTW doesn't look like that patch disables speculation in addPGOInstrPasses, which we need for the issue I'm trying to solve.

Another piece of motivation would be that the EarlyFPM change would allow us to move LowerExpectIntrinsic to the end of EarlyFPM again, which would make expect intrinsics work in Rust again. The pass was moved earlier to make the first SimplifyCFG run take them into account, but that would no longer be a problem if we disable speculation.

And another piece of motivation would be that we could move SimplifyCFG after SROA, where it can be much more effective (but doing so with speculation breaks IPSCCP too much). I think doing that needs some further changes, but it's a move in the right direction.

So I think the EarlyFPM change at least seems like something we should pretty clearly do independently of PGO. The addPGOInstrPasses() change is of course PGO specific anyway. I think only the GlobalCleanupPM change may not be completely clear cut.

For the case I was looking at, with instrumentation PGO, any of the pre-feedback SimplifyCFG speculation passes would do the wrong thing without that PGO data.

[tejohnson]

llvm/test/Transforms/PhaseOrdering/simplifycfg-speculate-blocks.ll

83 ↗

(On Diff #544109)

Right, that's need for SamplePGO. Removed the other stuff that wasn't necessary.

[tejohnson]

Address test comments

[tejohnson]

ping

[aeubanks]

can we try not gating this on PGO as suggested? minimizing differences between pipelines is nice, and as mentioned it'll help with other cases

llvm/test/Transforms/PhaseOrdering/simplifycfg-speculate-blocks.ll

1 ↗

(On Diff #544109)

update_test_checks.py doesn't handle %t (maybe that's an improvement we could make to the script)

[tejohnson]

In D155997#4570562, @aeubanks wrote:

can we try not gating this on PGO as suggested? minimizing differences between pipelines is nice, and as mentioned it'll help with other cases

Sorry for the delay, was OOO for a bit and this ended up being a bit more complicated. Since I added a change to guard branch folding with the simplifyBlocks flag in D156194, there were additional tests that had differences that I have been trying to analyze and mitigate, with partial but not complete success (i.e. additional to what was seen when @aeubanks tried this in D153392):

1. llvm/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll

The Hexagon loop idiom recognition pass is a little rigid in the detected code sequences, which were different because once we did the branch folding in a later SimplifyCFG invocation, followed by InstCombine, there was no EarlyCSE before the hexagon passes to do the cleanup required to get the old sequence. I can fix this by adding an invocation of EarlyCSE after those passes in buildFunctionSimplificationPipeline. There were no test changes from that fix, other than some tests that check the order of passes in each pipeline.

2. llvm/test/Transforms/PhaseOrdering/X86/vector-reductions-logical.ll

Not all function in this test are affected, but several are affected due to the phase ordering, specifically we stopped doing some of the branch folding completely which blocked some of the vectorization handling. In one of the functions I examined I found this was due to the fact that an EarlyCSE pass before we reran SimplifyCFG with speculation enabled did some commoning that blocked branch folding due to a limitation in the handling there. Specifically, the one added in https://reviews.llvm.org/rG909cba969981032c5740774ca84a34b7f76b909b. Undoing that does make the branch folding kick in more in this test, but isn't correct in general (it sounds like that patch was submitted to address a problem that was difficult to solve more optimally). In fact, that change had some similar affects on this test to my change, but there have been quite a few changes impacting this test over the past few years.

3. clang/test/Headers/__clang_hip_math.hip

Some fairly superficial diffs that don't appear important to the test.

Thoughts on how should I proceed? I can send the simple change I did for 1) above separately, since it doesn't seem to do any harm and presumably EarlyCSE is cheap. For 2) however, I'm not sure how much effort I should invest to get the test to behave exactly the same as before. There is clearly a limitation with SimplifyCFG that has been looked at before but isn't easy to solve.

llvm/test/Transforms/PhaseOrdering/simplifycfg-speculate-blocks.ll

1 ↗

(On Diff #544109)

I'm not sure what the alternative is here to using %t - do we need make the test support update_test_checks.py?

clang/test/CodeGen/pgo-sample-preparation.c

-// Test if PGO sample use preparation passes are executed correctly.
-//
-// Ensure that instcombine is executed after simplifycfg and sroa so that
-// "a < 255" will not be converted to a * 256 < 255 * 256.
-// RUN: %clang_cc1 -O2 -fprofile-sample-use=%S/Inputs/pgo-sample.prof %s -emit-llvm -o - 2>&1 | FileCheck %s
-
-void bar(int);
-void foo(int x, int y, int z) {
-  int m;
-  for (m = 0; m < x ; m++) {
-    int a = (((y >> 8) & 0xff) * z) / 256;
-    bar(a < 255 ? a : 255);
-  }
-}
-
-// CHECK-NOT: icmp slt i32 %mul, 65280

llvm/lib/Passes/PassBuilderPipelines.cpp

cl::values(clEnumValN(AttributorRunOption::ALL, "all",
                           "enable call graph SCC attributor runs"),
                clEnumValN(AttributorRunOption::NONE, "none",
                           "disable attributor runs")));
 
 cl::opt<bool> EnableMemProfContextDisambiguation(
     "enable-memprof-context-disambiguation", cl::init(false), cl::Hidden,
     cl::ZeroOrMore, cl::desc("Enable MemProf context disambiguation"));
 
+static cl::opt<bool> AlwaysAllowSimplifyCFGSpeculation(
+    "always-allow-simplify-cfg-speculation", cl::init(false), cl::Hidden,
+    cl::desc("Enable SimplifyCFG speculation even before profile annotation"));
+
 PipelineTuningOptions::PipelineTuningOptions() {
   LoopInterleaving = true;
   LoopVectorization = true;
   SLPVectorization = false;
   LoopUnrolling = true;
   ForgetAllSCEVInLoopUnroll = ForgetSCEVInLoopUnroll;
   LicmMssaOptCap = SetLicmMssaOptCap;
   LicmMssaNoAccForPromotionCap = SetLicmMssaNoAccForPromotionCap;
if (!IsCS && !DisablePreInliner) {
     ModuleInlinerWrapperPass MIWP(
         IP, /* MandatoryFirst */ true,
         InlineContext{LTOPhase, InlinePass::EarlyInliner});
     CGSCCPassManager &CGPipeline = MIWP.getPM();
 
     FunctionPassManager FPM;
     FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
     FPM.addPass(EarlyCSEPass()); // Catch trivial redundancies.
-    FPM.addPass(SimplifyCFGPass(SimplifyCFGOptions().convertSwitchRangeToICmp(
-        true)));                    // Merge & remove basic blocks.
+    FPM.addPass(SimplifyCFGPass(
+        SimplifyCFGOptions()
+            .convertSwitchRangeToICmp(true) // Merge & remove basic blocks.
+            // Don't enable speculation before PGO annotation unless forced,
+            // since later invocations will utilize the profile for speculation
+            // decisions.
+            .speculateBlocks(AlwaysAllowSimplifyCFGSpeculation)));
     FPM.addPass(InstCombinePass()); // Combine silly sequences.
     invokePeepholeEPCallbacks(FPM, Level);
 
     CGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
         std::move(FPM), PTO.EagerlyInvalidateAnalyses));
 
     MPM.addPass(std::move(MIWP));
 
PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
 
   // In ThinLTO mode, when flattened profile is used, all the available
   // profile information will be annotated in PreLink phase so there is
   // no need to load the profile again in PostLink.
   bool LoadSampleProfile =
       HasSampleProfile &&
       !(FlattenedProfileUsed && Phase == ThinOrFullLTOPhase::ThinLTOPostLink);
 
+  bool LoadFirstSampleProfile =
+      HasSampleProfile && Phase != ThinOrFullLTOPhase::ThinLTOPostLink;
+
+  bool DoPGOInstr = PGOOpt && Phase != ThinOrFullLTOPhase::ThinLTOPostLink &&
+                    (PGOOpt->Action == PGOOptions::IRInstr ||
+                     PGOOpt->Action == PGOOptions::IRUse);
+
   // During the ThinLTO backend phase we perform early indirect call promotion
   // here, before globalopt. Otherwise imported available_externally functions
   // look unreferenced and are removed. If we are going to load the sample
   // profile then defer until later.
   // TODO: See if we can move later and consolidate with the location where
   // we perform ICP when we are loading a sample profile.
   // TODO: We pass HasSampleProfile (whether there was a sample profile file
   // passed to the compile) to the SamplePGO flag of ICP. This is used to
if (Phase != ThinOrFullLTOPhase::ThinLTOPostLink) {
     MPM.addPass(InferFunctionAttrsPass());
     MPM.addPass(CoroEarlyPass());
 
     FunctionPassManager EarlyFPM;
     // Lower llvm.expect to metadata before attempting transforms.
     // Compare/branch metadata may alter the behavior of passes like
     // SimplifyCFG.
     EarlyFPM.addPass(LowerExpectIntrinsicPass());
-    EarlyFPM.addPass(SimplifyCFGPass());
+    // Don't enable speculation before PGO annotation unless forced, since later
+    // invocations will utilize the profile for speculation decisions.
+    EarlyFPM.addPass(SimplifyCFGPass(SimplifyCFGOptions().speculateBlocks(
+        (!DoPGOInstr && !LoadFirstSampleProfile) |
+        AlwaysAllowSimplifyCFGSpeculation)));
     EarlyFPM.addPass(SROAPass(SROAOptions::ModifyCFG));
     EarlyFPM.addPass(EarlyCSEPass());
     if (Level == OptimizationLevel::O3)
       EarlyFPM.addPass(CallSiteSplittingPass());
     MPM.addPass(createModuleToFunctionPassAdaptor(
         std::move(EarlyFPM), PTO.EagerlyInvalidateAnalyses));
   }
 
PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
 
   // Create a small function pass pipeline to cleanup after all the global
   // optimizations.
   FunctionPassManager GlobalCleanupPM;
   // FIXME: Should this instead by a run of SROA?
   GlobalCleanupPM.addPass(PromotePass());
   GlobalCleanupPM.addPass(InstCombinePass());
   invokePeepholeEPCallbacks(GlobalCleanupPM, Level);
-  GlobalCleanupPM.addPass(
-      SimplifyCFGPass(SimplifyCFGOptions().convertSwitchRangeToICmp(true)));
+  // Don't enable speculation before PGO annotation unless forced, since later
+  // invocations will utilize the profile for speculation decisions. At this
+  // point SamplePGO annotation is done, so we don't need to disable it in that
+  // case.
+  GlobalCleanupPM.addPass(SimplifyCFGPass(
+      SimplifyCFGOptions().convertSwitchRangeToICmp(true).speculateBlocks(
+          !DoPGOInstr | AlwaysAllowSimplifyCFGSpeculation)));
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(GlobalCleanupPM),
                                                 PTO.EagerlyInvalidateAnalyses));
 
   // Add all the requested passes for instrumentation PGO, if requested.
   if (PGOOpt && Phase != ThinOrFullLTOPhase::ThinLTOPostLink &&
       (PGOOpt->Action == PGOOptions::IRInstr ||
        PGOOpt->Action == PGOOptions::IRUse)) {
     addPGOInstrPasses(MPM, Level,

llvm/lib/Transforms/Utils/SimplifyCFG.cpp

if (LandingPadInst *LPad = dyn_cast<LandingPadInst>(I)) {
     if (I->isTerminator() && TryToMergeLandingPad(LPad, BI, BB, DTU))
       return true;
   }
 
   // If this basic block is ONLY a compare and a branch, and if a predecessor
   // branches to us and our successor, fold the comparison into the
   // predecessor and use logical operations to update the incoming value
   // for PHI nodes in common successor.
-  if (FoldBranchToCommonDest(BI, DTU, /*MSSAU=*/nullptr, &TTI,
+  if (Options.SpeculateBlocks &&
+      FoldBranchToCommonDest(BI, DTU, /*MSSAU=*/nullptr, &TTI,
                              Options.BonusInstThreshold))
     return requestResimplify();
   return false;
 }
 
 static BasicBlock *allPredecessorsComeFromSameSource(BasicBlock *BB) {
   BasicBlock *PredPred = nullptr;
   for (auto *P : predecessors(BB)) {
if (Imp) {
     BI->setCondition(TorF);
     RecursivelyDeleteTriviallyDeadInstructions(OldCond);
     return requestResimplify();
   }
 
   // If this basic block is ONLY a compare and a branch, and if a predecessor
   // branches to us and one of our successors, fold the comparison into the
   // predecessor and use logical operations to pick the right destination.
-  if (FoldBranchToCommonDest(BI, DTU, /*MSSAU=*/nullptr, &TTI,
+  if (Options.SpeculateBlocks &&
+      FoldBranchToCommonDest(BI, DTU, /*MSSAU=*/nullptr, &TTI,
                              Options.BonusInstThreshold))
     return requestResimplify();
 
   // We have a conditional branch to two blocks that are only reachable
   // from BI.  We know that the condbr dominates the two blocks, so see if
   // there is any identical code in the "then" and "else" blocks.  If so, we
   // can hoist it up to the branching block.
   if (BI->getSuccessor(0)->getSinglePredecessor()) {

llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -S -passes=simplifycfg -simplifycfg-require-and-preserve-domtree=1 -bonus-inst-threshold=10 | FileCheck %s
+; RUN: opt < %s -S -passes='simplifycfg<no-speculate-blocks>' -simplifycfg-require-and-preserve-domtree=1 -bonus-inst-threshold=10 | FileCheck %s --check-prefix=NOSPECULATE
 
 declare void @sideeffect0()
 declare void @sideeffect1()
 declare void @sideeffect2()
 declare void @use8(i8)
 declare i1 @gen1()
 declare i32 @speculate_call(ptr) #0
 
 ; Basic cases, blocks have nothing other than the comparison itself.
 
 define void @one_pred(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred(
+; NOSPECULATE-LABEL: @one_pred(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C0]], i1 [[C1]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
 ; CHECK-NEXT:    call void @sideeffect0()
 ; CHECK-NEXT:    br label [[COMMON_RET:%.*]]
 ; CHECK:       final_right:
 ; CHECK-NEXT:    call void @sideeffect1()
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @two_preds(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds(
+; NOSPECULATE-LABEL: @two_preds(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND1:%.*]] = select i1 [[C1]], i1 true, i1 [[C3_OLD]]
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
final_right:
   call void @sideeffect1()
   ret void
 }
 
 ; More complex case, there's an extra op that is safe to execute unconditionally.
 
 define void @one_pred_with_extra_op(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C0]], i1 [[C1]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
 ; CHECK-NEXT:    call void @sideeffect0()
 ; CHECK-NEXT:    br label [[COMMON_RET:%.*]]
 ; CHECK:       final_right:
 ; CHECK-NEXT:    call void @sideeffect1()
final_right:
   ret void
 }
 
 ; When we fold the dispatch block into pred, the call is moved to pred
 ; and the attribute nonnull propagates poison paramater. However, since the
 ; function is speculatable, it can never cause UB. So, we need not technically drop it.
 define void @one_pred_with_spec_call(i8 %v0, i8 %v1, ptr %p) {
 ; CHECK-LABEL: @one_pred_with_spec_call(
+; NOSPECULATE-LABEL: @one_pred_with_spec_call(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp ne ptr [[P:%.*]], null
 ; CHECK-NEXT:    [[X:%.*]] = call i32 @speculate_call(ptr nonnull [[P]])
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C0]], i1 [[C1]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[COMMON_RET:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_right:
 ; CHECK-NEXT:    call void @sideeffect0()
 ; CHECK-NEXT:    br label [[COMMON_RET]]
 ;
 pred:

 final_right:
   call void @sideeffect0()
   ret void
 }
 
 ; Drop dereferenceable on the parameter
 define void @one_pred_with_spec_call_deref(i8 %v0, i8 %v1, ptr %p) {
 ; CHECK-LABEL: @one_pred_with_spec_call_deref(
+; NOSPECULATE-LABEL: @one_pred_with_spec_call_deref(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp ne ptr [[P:%.*]], null
 ; CHECK-NEXT:    [[X:%.*]] = call i32 @speculate_call(ptr [[P]])
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C0]], i1 [[C1]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[COMMON_RET:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_right:
 ; CHECK-NEXT:    call void @sideeffect0()
 ; CHECK-NEXT:    br label [[COMMON_RET]]
 ;
 pred:

 
 final_right:
   call void @sideeffect0()
   ret void
 }
 
 define void @two_preds_with_extra_op(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2:%.*]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3_ADJ_OLD]], 0
 ; CHECK-NEXT:    [[OR_COND1:%.*]] = select i1 [[C1]], i1 true, i1 [[C3_OLD]]
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
final_right:
   call void @sideeffect1()
   ret void
 }
 
 ; More complex case, there's an extra op that is safe to execute unconditionally, and it has multiple uses.
 
 define void @one_pred_with_extra_op_multiuse(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_multiuse(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_multiuse(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[V1_ADJ_ADJ:%.*]] = add i8 [[V1_ADJ]], [[V1_ADJ]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1_ADJ_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C0]], i1 [[C1]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
 ; CHECK-NEXT:    call void @sideeffect0()
 ; CHECK-NEXT:    br label [[COMMON_RET:%.*]]
 ; CHECK:       final_right:
 ; CHECK-NEXT:    call void @sideeffect1()
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @two_preds_with_extra_op_multiuse(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op_multiuse(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_multiuse(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2:%.*]]
 ; CHECK-NEXT:    [[V3_ADJ_ADJ_OLD:%.*]] = add i8 [[V3_ADJ_OLD]], [[V3_ADJ_OLD]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3_ADJ_ADJ_OLD]], 0
 ; CHECK-NEXT:    [[OR_COND1:%.*]] = select i1 [[C1]], i1 true, i1 [[C3_OLD]]
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[V3_ADJ_ADJ:%.*]] = add i8 [[V3_ADJ]], [[V3_ADJ]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3_ADJ_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
final_right:
   ret void
 }
 
 ; More complex case, there's an op that is safe to execute unconditionally,
 ; and said op is live-out.
 
 define void @one_pred_with_extra_op_liveout(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_liveout(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_liveout(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1_ADJ]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
final_left:
   call void @use8(i8 %v1_adj)
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 define void @one_pred_with_extra_op_liveout_multiuse(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_liveout_multiuse(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_liveout_multiuse(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1_ADJ]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @one_pred_with_extra_op_liveout_distant_phi(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_liveout_distant_phi(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_liveout_distant_phi(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED:%.*]], label [[LEFT_END:%.*]]
 ; CHECK:       pred:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V2_ADJ:%.*]] = add i8 [[V0]], [[V1]]
left_end:
   ret void
 final_right:
   call void @sideeffect2()
   ret void
 }
 
 define void @two_preds_with_extra_op_liveout(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op_liveout(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_liveout(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3_ADJ_OLD]], 0
 ; CHECK-NEXT:    br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @two_preds_with_extra_op_liveout_multiuse(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op_liveout_multiuse(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_liveout_multiuse(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3_ADJ]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3_ADJ_OLD]], 0
 ; CHECK-NEXT:    br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
final_right:
   ret void
 }
 
 ; More complex case, there's an op that is safe to execute unconditionally,
 ; and said op is live-out, and it is only used externally.
 
 define void @one_pred_with_extra_op_eexternally_used_only(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_eexternally_used_only(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_eexternally_used_only(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
final_left:
   call void @use8(i8 %v1_adj)
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 define void @one_pred_with_extra_op_externally_used_only_multiuse(i8 %v0, i8 %v1) {
 ; CHECK-LABEL: @one_pred_with_extra_op_externally_used_only_multiuse(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_externally_used_only_multiuse(
 ; CHECK-NEXT:  pred:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @two_preds_with_extra_op_externally_used_only(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op_externally_used_only(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_externally_used_only(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3]], 0
 ; CHECK-NEXT:    br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @two_preds_with_extra_op_externally_used_only_multiuse(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
 ; CHECK-LABEL: @two_preds_with_extra_op_externally_used_only_multiuse(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_externally_used_only_multiuse(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V1]], [[V2]]
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3]], 0
 ; CHECK-NEXT:    br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:

 final_right:
   call void @sideeffect1()
   ret void
 }
 
 ; The liveout instruction can be located after the branch condition.
 define void @one_pred_with_extra_op_externally_used_only_after_cond_distant_phi(i8 %v0, i8 %v1, i8 %v3, i8 %v4, i8 %v5) {
 ; CHECK-LABEL: @one_pred_with_extra_op_externally_used_only_after_cond_distant_phi(
+; NOSPECULATE-LABEL: @one_pred_with_extra_op_externally_used_only_after_cond_distant_phi(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED:%.*]], label [[LEFT_END:%.*]]
 ; CHECK:       pred:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
left_end:
   call void @use8(i8 %merge_left)
   ret void
 final_right:
   call void @sideeffect2()
   ret void
 }
 define void @two_preds_with_extra_op_externally_used_only_after_cond(i8 %v0, i8 %v1, i8 %v2, i8 %v3, i8 %v4, i8 %v5) {
 ; CHECK-LABEL: @two_preds_with_extra_op_externally_used_only_after_cond(
+; NOSPECULATE-LABEL: @two_preds_with_extra_op_externally_used_only_after_cond(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
 ; CHECK:       pred0:
 ; CHECK-NEXT:    [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
 ; CHECK:       pred1:
 ; CHECK-NEXT:    [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
 ; CHECK-NEXT:    [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
 ; CHECK-NEXT:    [[V3_ADJ:%.*]] = add i8 [[V4:%.*]], [[V5:%.*]]
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[C2]], i1 [[C3]], i1 false
+; NOSPECULATE-NOT: select
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
 ; CHECK:       dispatch:
 ; CHECK-NEXT:    [[C3_OLD:%.*]] = icmp eq i8 [[V3]], 0
 ; CHECK-NEXT:    [[V3_ADJ_OLD:%.*]] = add i8 [[V4]], [[V5]]
 ; CHECK-NEXT:    br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
 ; CHECK:       common.ret:
 ; CHECK-NEXT:    ret void
 ; CHECK:       final_left:
final_left:
   ret void
 final_right:
   call void @sideeffect1()
   ret void
 }
 
 define void @pr48450() {
 ; CHECK-LABEL: @pr48450(
+; NOSPECULATE-LABEL: @pr48450(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[COUNTDOWN:%.*]] = phi i8 [ 8, [[ENTRY:%.*]] ], [ [[DEC:%.*]], [[FOR_BODYTHREAD_PRE_SPLIT:%.*]] ]
 ; CHECK-NEXT:    [[C:%.*]] = call i1 @gen1()
 ; CHECK-NEXT:    br i1 [[C]], label [[FOR_INC:%.*]], label [[IF_THEN:%.*]]
 ; CHECK:       for.inc:
 ; CHECK-NEXT:    [[DEC]] = add i8 [[COUNTDOWN]], -1
for.bodythread-pre-split:
   br label %for.body
 
 if.end.loopexit:
   ret void
 }
 
 define void @pr48450_2(i1 %enable_loopback) {
 ; CHECK-LABEL: @pr48450_2(
+; NOSPECULATE-LABEL: @pr48450_2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[COUNTDOWN:%.*]] = phi i8 [ 8, [[ENTRY:%.*]] ], [ [[DEC:%.*]], [[FOR_BODYTHREAD_PRE_SPLIT:%.*]] ]
 ; CHECK-NEXT:    [[C:%.*]] = call i1 @gen1()
 ; CHECK-NEXT:    br i1 [[C]], label [[FOR_INC:%.*]], label [[IF_THEN:%.*]]
 ; CHECK:       for.inc:
 ; CHECK-NEXT:    [[DEC]] = add i8 [[COUNTDOWN]], -1

 
 if.end.loopexit:
   ret void
 }
 
 @f.b = external global i8, align 1
 define void @pr48450_3() {
 ; CHECK-LABEL: @pr48450_3(
+; NOSPECULATE-LABEL: @pr48450_3(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_COND1:%.*]]
 ; CHECK:       for.cond1:
 ; CHECK-NEXT:    [[V:%.*]] = load i8, ptr @f.b, align 1
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i8 [[V]], 1
 ; CHECK-NEXT:    call void @llvm.assume(i1 [[CMP]])
 ; CHECK-NEXT:    br label [[FOR_COND1]]
 ;

 cleanup:
   unreachable
 }
 
 @global_pr49510 = external global i16, align 1
 
 define void @pr49510() {
 ; CHECK-LABEL: @pr49510(
+; NOSPECULATE-LABEL: @pr49510(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_COND:%.*]]
 ; CHECK:       for.cond:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i16, ptr @global_pr49510, align 1
 ; CHECK-NEXT:    [[TOBOOL:%.*]] = icmp ne i16 [[TMP0]], 0
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i16 [[TMP0]], 0
 ; CHECK-NEXT:    [[OR_COND:%.*]] = and i1 [[TOBOOL]], [[CMP]]
 ; CHECK-NEXT:    br i1 [[OR_COND]], label [[FOR_COND]], label [[FOR_END:%.*]]

 ; FIXME:
 ; This is a miscompile if we replace a phi incoming value
 ; with an updated loaded value *after* it was stored.
 
 @global_pr51125 = global i32 1, align 4
 
 define i32 @pr51125() {
 ; CHECK-LABEL: @pr51125(
+; NOSPECULATE-LABEL: @pr51125(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[L:%.*]]
 ; CHECK:       L:
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr @global_pr51125, align 4
 ; CHECK-NEXT:    [[ISZERO:%.*]] = icmp eq i32 [[LD]], 0
 ; CHECK-NEXT:    br i1 [[ISZERO]], label [[EXIT:%.*]], label [[L2:%.*]]
 ; CHECK:       L2:
 ; CHECK-NEXT:    store i32 -1, ptr @global_pr51125, align 4

 exit:
   %r = phi i32 [ %ld, %L2 ], [ %ld, %L ]
   ret i32 %r
 }
 
 ; https://github.com/llvm/llvm-project/issues/53861
 define i32 @firewall(ptr %data) {
 ; CHECK-LABEL: @firewall(
+; NOSPECULATE-LABEL: @firewall(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[I:%.*]] = load i8, ptr [[DATA:%.*]], align 1
 ; CHECK-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds i8, ptr [[DATA]], i64 64
 ; CHECK-NEXT:    [[I2:%.*]] = load i16, ptr [[ADD_PTR]], align 2
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i8 [[I]], 17
 ; CHECK-NEXT:    [[CMP3:%.*]] = icmp eq i16 [[I2]], 1
 ; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[CMP]], i1 [[CMP3]], i1 false
 ; CHECK-NEXT:    [[CMP10:%.*]] = icmp eq i16 [[I2]], 2

 
 cleanup:
   %retval.0 = phi i32 [ 1, %entry ], [ 1, %if.end ], [ 1, %if.end13 ], [ %., %if.end22 ]
   ret i32 %retval.0
 }
 
 define i32 @test_builtin_fpclassify(float %x) {
 ; CHECK-LABEL: @test_builtin_fpclassify(
+; NOSPECULATE-LABEL: @test_builtin_fpclassify(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ISZERO:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    br i1 [[ISZERO]], label [[FPCLASSIFY_END:%.*]], label [[FPCLASSIFY_NOT_ZERO:%.*]]
 ; CHECK:       fpclassify_end:
 ; CHECK-NEXT:    [[FPCLASSIFY_RESULT:%.*]] = phi i32 [ 2, [[ENTRY:%.*]] ], [ 0, [[FPCLASSIFY_NOT_ZERO]] ], [ 1, [[FPCLASSIFY_NOT_NAN:%.*]] ], [ [[NORMAL_OR_SUBNORMAL:%.*]], [[FPCLASSIFY_NOT_INF:%.*]] ]
 ; CHECK-NEXT:    ret i32 [[FPCLASSIFY_RESULT]]
 ; CHECK:       fpclassify_not_zero:
 ; CHECK-NEXT:    [[CMP:%.*]] = fcmp uno float [[X]], 0.000000e+00

llvm/test/Transforms/SimplifyCFG/pipeline-delay-speculation-pgo.ll

+;; Test that the pipelines delay simplify CFG speculation until after
+;; pgo annotation.
+
+; RUN: touch %t.profdata
+
+;; Test the default optimization pipeline with no PGO, instrumentation PGO and sample PGO.
+;; The first should not disable simplify CFG speculation, the latter two should before
+;; profile annotation.
+; RUN: opt -passes='default<O2>' -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='default<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,INSTRPGO
+; RUN: opt -passes='default<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,SAMPLEPGO
+
+;; Test the ThinLTO pre-link pipeline with no PGO, instrumentation PGO and sample PGO.
+;; The first should not disable simplify CFG speculation, the latter two should before
+;; profile annotation.
+; RUN: opt -passes='thinlto-pre-link<O2>' -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,INSTRPGO
+; RUN: opt -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,SAMPLEPGO
+
+;; Test the LTO pre-link pipeline with no PGO, instrumentation PGO and sample PGO.
+;; The first should not disable simplify CFG speculation, the latter two should before
+;; profile annotation.
+; RUN: opt -passes='lto-pre-link<O2>' -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='lto-pre-link<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,INSTRPGO
+; RUN: opt -passes='lto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --check-prefixes=PGO,SAMPLEPGO
+
+;; The ThinLTO post-link pipeline will already have performed profile annotation,
+;; and should never disable simplifyCFG speculation.
+; RUN: opt -passes='thinlto<O2>' -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='thinlto<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='thinlto<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+
+;; The LTO post-link pipeline will already have performed profile annotation,
+;; and should never disable simplifyCFG speculation.
+; RUN: opt -passes='lto<O2>' -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='lto<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -passes='lto<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+
+;; Ensure the -always-allow-simplify-cfg-speculation option enables speculation
+;; as expected.
+; RUN: opt -always-allow-simplify-cfg-speculation -passes='default<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+; RUN: opt -always-allow-simplify-cfg-speculation -passes='default<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%t.profdata -print-pipeline-passes -S < %s | FileCheck %s --implicit-check-not=no-speculate-blocks --check-prefix=NOPGO
+
+;; With no PGO, we can always speculate (the corresponding FileCheck invocations
+;; have an --implicit-check-not=no-speculate-blocks).
+; NOPGO: simplifycfg
+; NOPGO-SAME: ;speculate-blocks
+
+;; With either type of PGO, we should disable simplifyCFG speculation before profile
+;; annotation and allow it afterwards.
+; PGO: simplifycfg
+; PGO-NOT: ;speculate-blocks
+; PGO-SAME: no-speculate-blocks
+; PGO-NOT: ;speculate-blocks
+; INSTRPGO-SAME: pgo-instr-use
+; SAMPLEPGO-SAME: sample-profile
+; PGO-NOT: no-speculate-blocks
+; PGO-SAME: ;speculate-blocks
+; PGO-NOT: no-speculate-blocks