[AMDGPUUnifyDivergentExitNodes] Add NewPM support

[gandhi21299]

Summary

Meanwhile, use UniformityAnalysis instead of LegacyDivergenceAnalysis to collect divergence info.

[gandhi21299]

• added checks for the opt command in si-annotate-nested-control-flow.ll
• applied clang-format

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

Not sure why you really need to change this, but the function should probably be a template argument. Is DA just missing a layer to present the PM independent analysis?

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

The LegacyPM version depends on LegacyDivergenceAnalysis info whereas the NewPM version of this pass depends on DivergenceInfo.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

There should be a common logic DivergenceInfo that both pass versions export

[gandhi21299]

• isUniformlyReached now accepts a template argument instead of a std::function to allow both pass managers to use their corresponding DivergenceAnalysis info.
• DivergenceAnalysis::isUniform(Value &V) -> DivergenceAnalysis::isUniform(Value *V)

[gandhi21299]

Rebased and applied clang-format

[gandhi21299]

• Rebased and initialized TTI = nullptr

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

How do these changes look @arsenm ?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

I'm still confused. LegacyDivergenceAnalysis and DivergenceAnalysis/DivergenceInfo are two *different* analyses. LegacyDivergenceAnalysis is *not* the Legacy pass manager version of DivergenceAnalysis. New pass manager shouldn't invisibly switch the analysis used. I think LegacyDivergenceAnalysis needs to be ported to new PM first

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

Ahh I guess I got confused with the names as well. Sure, I will port LegacyDivergenceAnalysis to newPM first.

[gandhi21299]

Rebased

[gandhi21299]

Added missing newline

[gandhi21299]

minor nits

[sameerds]

At this point, there are three analyses as follows:

1. LegacyDA is actually wrong, but useful in some cases involving irreducible control flow.
2. GpuDA is correct, but treats irreducible control flow very conservatively.
3. UniformityAnalysis is both correct and better at irreducible control flow.

Note that in almost every use on AMDGPU, LegacyDA simply acts as a wrapper for GpuDA.

Since this is fresh work, why not just go straight to UniformityAnalysis? I would definitely recommend that because not only does go straight to new stuff, it also avoids the unnecessary template. This would be a good time to use the new analysis. And if issues crop up that can't be fixed, then we should fall back no further than GpuDA, and unify their exposed Info objects along the way.

llvm/include/llvm/Analysis/DivergenceAnalysis.h

177

Why is this change necessary? The point of accepting a reference is to indicate that nullptr is not expected inside the function.

[sameerds]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97–98

No! Don't even try. It was considered and discarded long ago.

305

As already noted, this legacy is not related to the pass manager. Just use the same DA in both.

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

97

Can you give a more meaningful name to the template argument? For instance, DivergenceAnalysisType?

[ruiling]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

190–192

I think you just removed this accidentally.

[gandhi21299]

I will create a new patch to replace LegacyDivergenceAnalysis with Uniformity analysis, unless it's too early to do so.

[gandhi21299]

• Rebased with uniformity analysis dependency
• Removed template param
• Refactors

[sameerds]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

162–163

Either declare parameters as references, or implement checks for nullptr in the function body. Also, can any of these be const?

190–192

Was this comment addressed?

303–304

If you know that a variable cannot be nullptr, it is preferable to declare it as a reference rather than a pointer. It can be converted into a pointer at the point where some function expects a pointer. For example, the call below can be "whatever.run(&DT, &PDT, UA)".

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

190–192

yup, I moved it under AMDGPUUnifyDivergentExitNodesImpl::run(..)

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

162–163

DominatorTree may be updated but PostDominatorTree and UniformityInfo are preserved.

[gandhi21299]

• Addressed reviewer comments

[foad]

The commit message should make it clear that you're also changing it to use UniformityAnalysis instead of (Legacy?)DivergenceAnalysis.

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.h

58

It looks like only this class needs to be declared in the .h file. The others could stay in the .cpp file.

[gandhi21299]

• Move AMDGPUUnifyDivergentExitNodesImpl and AMDGPUUnifyDivergentExitNodes classes to the source file from the header file.

[gandhi21299]

• Move includes from header to source

[gandhi21299]

• do not need to include PassManager
• TTI should be private instead of protected in the Impl

[sameerds]

LGTM, but wait for a day to let other reviewers comment.

[gandhi21299]

Internal CI passed.

[vitalybuka]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

308

DT can be null where, which is UB https://lab.llvm.org/buildbot/#/builders/5/builds/32228/steps/16/logs/stdio

[gandhi21299]

Impl::run(..) now accepts DominatorTree* instead of DominatorTree& to avoid undefined behavior due to the possibility of DT == nullptr.

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

308

Great catch! How does it look now?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

162–163

When is DT null but PDT won't be?

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

162–163

This pass depends on PDT so it can never be null. DT may be null if it does not need to be preserved, which is specified by -simplifycfg-require-and-preserve-domtree.

llvm/include/llvm/Analysis/DivergenceAnalysis.h

public:
   }
 
   /// Whether \p U is divergent. Uses of a uniform value can be divergent.
   bool isDivergentUse(const Use &U) const {
     return ContainsIrreducible || DA->isDivergentUse(U);
   }
 
   /// Whether \p V is uniform/non-divergent.
-  bool isUniform(const Value &V) const { return !isDivergent(V); }
+  bool isUniform(const Value *V) const { return !isDivergent(*V); }
 
   /// Whether \p U is uniform/non-divergent. Uses of a uniform value can be
   /// divergent.
   bool isUniformUse(const Use &U) const { return !isDivergentUse(U); }
 };
 
 /// \brief Divergence analysis frontend for GPU kernels.
 class DivergenceAnalysis : public AnalysisInfoMixin<DivergenceAnalysis> {

llvm/lib/Target/AMDGPU/AMDGPU.h

 extern char &SIFormMemoryClausesID;
 
 void initializeSIPostRABundlerPass(PassRegistry&);
 extern char &SIPostRABundlerID;
 
 void initializeGCNCreateVOPDPass(PassRegistry &);
 extern char &GCNCreateVOPDID;
 
-void initializeAMDGPUUnifyDivergentExitNodesPass(PassRegistry&);
+void initializeAMDGPUUnifyDivergentExitNodesPass(PassRegistry &);
 extern char &AMDGPUUnifyDivergentExitNodesID;
 
 ImmutablePass *createAMDGPUAAWrapperPass();
 void initializeAMDGPUAAWrapperPassPass(PassRegistry&);
 ImmutablePass *createAMDGPUExternalAAWrapperPass();
 void initializeAMDGPUExternalAAWrapperPass(PassRegistry&);
 
 void initializeAMDGPUArgumentUsageInfoPass(PassRegistry &);

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

 #include "AMDGPU.h"
 #include "AMDGPUAliasAnalysis.h"
 #include "AMDGPUCtorDtorLowering.h"
 #include "AMDGPUExportClustering.h"
 #include "AMDGPUIGroupLP.h"
 #include "AMDGPUMacroFusion.h"
 #include "AMDGPUTargetObjectFile.h"
 #include "AMDGPUTargetTransformInfo.h"
+#include "AMDGPUUnifyDivergentExitNodes.h"
 #include "GCNIterativeScheduler.h"
 #include "GCNSchedStrategy.h"
 #include "GCNVOPDUtils.h"
 #include "R600.h"
 #include "R600MachineFunctionInfo.h"
 #include "R600TargetMachine.h"
 #include "SIMachineFunctionInfo.h"
 #include "SIMachineScheduler.h"
PB.registerPipelineParsingCallback(
         if (PassName == "amdgpu-propagate-attributes-early") {
           PM.addPass(AMDGPUPropagateAttributesEarlyPass(*this));
           return true;
         }
         if (PassName == "amdgpu-promote-kernel-arguments") {
           PM.addPass(AMDGPUPromoteKernelArgumentsPass());
           return true;
         }
+        if (PassName == "amdgpu-unify-divergent-exit-nodes") {
+          PM.addPass(AMDGPUUnifyDivergentExitNodesPass());
+          return true;
+        }
         return false;
       });
 
   PB.registerAnalysisRegistrationCallback([](FunctionAnalysisManager &FAM) {
     FAM.registerPass([&] { return AMDGPUAA(); });
   });
 
   PB.registerParseAACallback([](StringRef AAName, AAManager &AAM) {

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.h

+//===- AMDGPUUnifyDivergentExitNodes.h ------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This is a variant of the UnifyFunctionExitNodes pass. Rather than ensuring
+// there is at most one ret and one unreachable instruction, it ensures there is
+// at most one divergent exiting block.
+//
+// StructurizeCFG can't deal with multi-exit regions formed by branches to
+// multiple return nodes. It is not desirable to structurize regions with
+// uniform branches, so unifying those to the same return block as divergent
+// branches inhibits use of scalar branching. It still can't deal with the case
+// where one branch goes to return, and one unreachable. Replace unreachable in
+// this case with a return.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPU.h"
+#include "llvm/Analysis/DomTreeUpdater.h"
+#include "llvm/Analysis/LegacyDivergenceAnalysis.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+
+namespace llvm {
+
+class AMDGPUUnifyDivergentExitNodesImpl {
+protected:
+  const TargetTransformInfo *TTI = nullptr;
+
+public:
+  // We can preserve non-critical-edgeness when we unify function exit nodes
+  // void getAnalysisUsage(AnalysisUsage &AU) const override;
+  // bool runOnFunction(Function &F) override;
+  BasicBlock *unifyReturnBlockSet(Function &F, DomTreeUpdater &DTU,
+                                  ArrayRef<BasicBlock *> ReturningBlocks,
+                                  StringRef Name);
+  template <typename T>
+  bool run(Function &F, DominatorTree *DT, PostDominatorTree *PDT, T *DA);
+};
+
+class AMDGPUUnifyDivergentExitNodes : public FunctionPass,
+                                      public AMDGPUUnifyDivergentExitNodesImpl {
+public:
+  static char ID;
+  AMDGPUUnifyDivergentExitNodes() : FunctionPass(ID) {
+    initializeAMDGPUUnifyDivergentExitNodesPass(
+        *PassRegistry::getPassRegistry());
+  }
+  void getAnalysisUsage(AnalysisUsage &AU) const override;
+  bool runOnFunction(Function &F) override;
+};
+
+class AMDGPUUnifyDivergentExitNodesPass
+    : public PassInfoMixin<AMDGPUUnifyDivergentExitNodesPass>,
+      public AMDGPUUnifyDivergentExitNodesImpl {
+public:
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+
+} // end namespace llvm
 No newline at end of file

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

 // multiple return nodes. It is not desirable to structurize regions with
 // uniform branches, so unifying those to the same return block as divergent
 // branches inhibits use of scalar branching. It still can't deal with the case
 // where one branch goes to return, and one unreachable. Replace unreachable in
 // this case with a return.
 //
 //===----------------------------------------------------------------------===//
 
+#include "AMDGPUUnifyDivergentExitNodes.h"
 #include "AMDGPU.h"
 #include "SIDefines.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Analysis/DivergenceAnalysis.h"
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/IntrinsicsAMDGPU.h"
+#include "llvm/IR/PassManager.h"
 #include "llvm/IR/Type.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Utils.h"
 #include "llvm/Transforms/Utils/Local.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "amdgpu-unify-divergent-exit-nodes"
 
-namespace {
-
-class AMDGPUUnifyDivergentExitNodes : public FunctionPass {
-private:
-  const TargetTransformInfo *TTI = nullptr;
-
-public:
-  static char ID; // Pass identification, replacement for typeid
-
-  AMDGPUUnifyDivergentExitNodes() : FunctionPass(ID) {
-    initializeAMDGPUUnifyDivergentExitNodesPass(*PassRegistry::getPassRegistry());
-  }
-
-  // We can preserve non-critical-edgeness when we unify function exit nodes
-  void getAnalysisUsage(AnalysisUsage &AU) const override;
-  BasicBlock *unifyReturnBlockSet(Function &F, DomTreeUpdater &DTU,
-                                  ArrayRef<BasicBlock *> ReturningBlocks,
-                                  StringRef Name);
-  bool runOnFunction(Function &F) override;
-};
-
-} // end anonymous namespace
-
 char AMDGPUUnifyDivergentExitNodes::ID = 0;
 
 char &llvm::AMDGPUUnifyDivergentExitNodesID = AMDGPUUnifyDivergentExitNodes::ID;
 
 INITIALIZE_PASS_BEGIN(AMDGPUUnifyDivergentExitNodes, DEBUG_TYPE,
                       "Unify divergent function exit nodes", false, false)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
 INITIALIZE_PASS_END(AMDGPUUnifyDivergentExitNodes, DEBUG_TYPE,
                     "Unify divergent function exit nodes", false, false)
 
-void AMDGPUUnifyDivergentExitNodes::getAnalysisUsage(AnalysisUsage &AU) const{
+void AMDGPUUnifyDivergentExitNodes::getAnalysisUsage(AnalysisUsage &AU) const {
   if (RequireAndPreserveDomTree)
     AU.addRequired<DominatorTreeWrapperPass>();
 
   AU.addRequired<PostDominatorTreeWrapperPass>();
 
   AU.addRequired<LegacyDivergenceAnalysis>();
 
   if (RequireAndPreserveDomTree) {
void AMDGPUUnifyDivergentExitNodes::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addPreservedID(LowerSwitchID);
   FunctionPass::getAnalysisUsage(AU);
 
   AU.addRequired<TargetTransformInfoWrapperPass>();
 }
 
 /// \returns true if \p BB is reachable through only uniform branches.
 /// XXX - Is there a more efficient way to find this?
-static bool isUniformlyReached(const LegacyDivergenceAnalysis &DA,
-                               BasicBlock &BB) {
+template <typename T> static bool isUniformlyReached(T *DA, BasicBlock &BB) {
   SmallVector<BasicBlock *, 8> Stack(predecessors(&BB));
   SmallPtrSet<BasicBlock *, 8> Visited;
 
   while (!Stack.empty()) {
     BasicBlock *Top = Stack.pop_back_val();
-    if (!DA.isUniform(Top->getTerminator()))
+    if (!DA->isUniform(Top->getTerminator()))
       return false;
 
     for (BasicBlock *Pred : predecessors(Top)) {
       if (Visited.insert(Pred).second)
         Stack.push_back(Pred);
     }
   }
 
   return true;
 }
 
-BasicBlock *AMDGPUUnifyDivergentExitNodes::unifyReturnBlockSet(
+BasicBlock *AMDGPUUnifyDivergentExitNodesImpl::unifyReturnBlockSet(
     Function &F, DomTreeUpdater &DTU, ArrayRef<BasicBlock *> ReturningBlocks,
     StringRef Name) {
   // Otherwise, we need to insert a new basic block into the function, add a PHI
   // nodes (if the function returns values), and convert all of the return
   // instructions into unconditional branches.
   BasicBlock *NewRetBlock = BasicBlock::Create(F.getContext(), Name, &F);
   IRBuilder<> B(NewRetBlock);
 
BasicBlock *AMDGPUUnifyDivergentExitNodesImpl::unifyReturnBlockSet(
   for (BasicBlock *BB : ReturningBlocks) {
     // Cleanup possible branch to unconditional branch to the return.
     simplifyCFG(BB, *TTI, RequireAndPreserveDomTree ? &DTU : nullptr,
                 SimplifyCFGOptions().bonusInstThreshold(2));
   }
 
   return NewRetBlock;
 }
 
-bool AMDGPUUnifyDivergentExitNodes::runOnFunction(Function &F) {
-  DominatorTree *DT = nullptr;
-  if (RequireAndPreserveDomTree)
-    DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-
-  auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
-  if (PDT.root_size() == 0 ||
-      (PDT.root_size() == 1 &&
-       !isa<BranchInst>(PDT.getRoot()->getTerminator())))
-    return false;
-
-  LegacyDivergenceAnalysis &DA = getAnalysis<LegacyDivergenceAnalysis>();
-  TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-
+template <typename T>
+bool AMDGPUUnifyDivergentExitNodesImpl::run(Function &F, DominatorTree *DT,
+                                            PostDominatorTree *PDT, T *DA) {
   // Loop over all of the blocks in a function, tracking all of the blocks that
   // return.
   SmallVector<BasicBlock *, 4> ReturningBlocks;
   SmallVector<BasicBlock *, 4> UnreachableBlocks;
 
   // Dummy return block for infinite loop.
   BasicBlock *DummyReturnBB = nullptr;
 
   bool Changed = false;
   std::vector<DominatorTree::UpdateType> Updates;
 
   // TODO: For now we unify all exit blocks, even though they are uniformly
   // reachable, if there are any exits not uniformly reached. This is to
   // workaround the limitation of structurizer, which can not handle multiple
   // function exits. After structurizer is able to handle multiple function
   // exits, we should only unify UnreachableBlocks that are not uniformly
   // reachable.
   bool HasDivergentExitBlock = llvm::any_of(
-      PDT.roots(), [&](auto BB) { return !isUniformlyReached(DA, *BB); });
+      PDT->roots(), [&](auto BB) { return !isUniformlyReached(DA, *BB); });
 
-  for (BasicBlock *BB : PDT.roots()) {
+  for (BasicBlock *BB : PDT->roots()) {
     if (isa<ReturnInst>(BB->getTerminator())) {
       if (HasDivergentExitBlock)
         ReturningBlocks.push_back(BB);
     } else if (isa<UnreachableInst>(BB->getTerminator())) {
       if (HasDivergentExitBlock)
         UnreachableBlocks.push_back(BB);
     } else if (BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator())) {
 
if (ReturningBlocks.empty())
     return Changed; // No blocks return
 
   if (ReturningBlocks.size() == 1)
     return Changed; // Already has a single return block
 
   unifyReturnBlockSet(F, DTU, ReturningBlocks, "UnifiedReturnBlock");
   return true;
 }
+
+bool AMDGPUUnifyDivergentExitNodes::runOnFunction(Function &F) {
+  DominatorTree *DT = nullptr;
+  if (RequireAndPreserveDomTree)
+    DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
+
+  auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
+  LegacyDivergenceAnalysis &DA = getAnalysis<LegacyDivergenceAnalysis>();
+  TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+  return AMDGPUUnifyDivergentExitNodes::run(F, DT, &PDT, &DA);
+}
+
+PreservedAnalyses
+AMDGPUUnifyDivergentExitNodesPass::run(Function &F,
+                                       FunctionAnalysisManager &AM) {
+  DominatorTree *DT = nullptr;
+  if (RequireAndPreserveDomTree)
+    DT = &AM.getResult<DominatorTreeAnalysis>(F);
+
+  auto *PDT = &AM.getResult<PostDominatorTreeAnalysis>(F);
+  auto *DA = &AM.getResult<DivergenceAnalysis>(F);
+  TTI = &AM.getResult<TargetIRAnalysis>(F);
+  return AMDGPUUnifyDivergentExitNodesImpl::run(F, DT, PDT, DA)
+             ? PreservedAnalyses::none()
+             : PreservedAnalyses::all();
+}

llvm/test/CodeGen/AMDGPU/si-annotate-nested-control-flows.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: llc -mtriple=amdgcn-amd-amdhsa %s -o - | FileCheck %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -p simplifycfg,amdgpu-unify-divergent-exit-nodes %s -S -o - | FileCheck %s --check-prefix=OPT
+; RUN: llc -mtriple=amdgcn-amd-amdhsa %s -o - | FileCheck %s --check-prefix=ISA
 
 define void @nested_inf_loop(i1 %0, i1 %1) {
-; CHECK-LABEL: nested_inf_loop:
-; CHECK-NEXT: %bb.0:                                ; %BB
-; CHECK-NEXT:   s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:   v_and_b32_e32 v1, 1, v1
-; CHECK-NEXT:   v_and_b32_e32 v0, 1, v0
-; CHECK-NEXT:   v_cmp_eq_u32_e64 s[4:5], 1, v1
-; CHECK-NEXT:   v_cmp_eq_u32_e32 vcc, 1, v0
-; CHECK-NEXT:   s_xor_b64 s[6:7], vcc, -1
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT: .LBB0_1:                                ; %BB1
-; CHECK:        s_and_b64 s[10:11], exec, s[6:7]
-; CHECK-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
-; CHECK-NEXT:   s_andn2_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_cbranch_execnz .LBB0_1
-; CHECK-NEXT: %bb.2:                                ; %BB2
-; CHECK:        s_or_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT: .LBB0_3:                                ; %BB4
-; CHECK:        s_and_b64 s[10:11], exec, s[4:5]
-; CHECK-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
-; CHECK-NEXT:   s_andn2_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_cbranch_execnz .LBB0_3
-; CHECK-NEXT: %bb.4:                                ; %loop.exit.guard
-; CHECK:        s_or_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_mov_b64 vcc, 0
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT:   s_branch .LBB0_1
-; CHECK-NEXT: %bb.5:                                ; %DummyReturnBlock
-; CHECK-NEXT:   s_setpc_b64 s[30:31]
+; OPT-LABEL: @nested_inf_loop(
+; OPT-NEXT:  BB:
+; OPT-NEXT:    br label [[BB1:%.*]]
+; OPT:       BB1:
+; OPT-NEXT:    [[BRMERGE:%.*]] = select i1 [[TMP0:%.*]], i1 true, i1 [[TMP1:%.*]]
+; OPT-NEXT:    br i1 [[BRMERGE]], label [[BB1]], label [[INFLOOP:%.*]]
+; OPT:       infloop:
+; OPT-NEXT:    br i1 true, label [[INFLOOP]], label [[DUMMYRETURNBLOCK:%.*]]
+; OPT:       DummyReturnBlock:
+; OPT-NEXT:    ret void
+;
+; ISA-LABEL: nested_inf_loop:
+; ISA-NEXT: %bb.0:                                ; %BB
+; ISA-NEXT:   s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; ISA-NEXT:   v_and_b32_e32 v1, 1, v1
+; ISA-NEXT:   v_and_b32_e32 v0, 1, v0
+; ISA-NEXT:   v_cmp_eq_u32_e64 s[4:5], 1, v1
+; ISA-NEXT:   v_cmp_eq_u32_e32 vcc, 1, v0
+; ISA-NEXT:   s_xor_b64 s[6:7], vcc, -1
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT: .LBB0_1:                                ; %BB1
+; ISA:        s_and_b64 s[10:11], exec, s[6:7]
+; ISA-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
+; ISA-NEXT:   s_andn2_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_cbranch_execnz .LBB0_1
+; ISA-NEXT: %bb.2:                                ; %BB2
+; ISA:        s_or_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT: .LBB0_3:                                ; %BB4
+; ISA:        s_and_b64 s[10:11], exec, s[4:5]
+; ISA-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
+; ISA-NEXT:   s_andn2_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_cbranch_execnz .LBB0_3
+; ISA-NEXT: %bb.4:                                ; %loop.exit.guard
+; ISA:        s_or_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_mov_b64 vcc, 0
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT:   s_branch .LBB0_1
+; ISA-NEXT: %bb.5:                                ; %DummyReturnBlock
+; ISA-NEXT:   s_setpc_b64 s[30:31]
 BB:
   br label %BB1
 
 BB1:
   br i1 %0, label %BB3, label %BB2
 
 BB2:
   br label %BB4
 
 BB4:
   br i1 %1, label %BB3, label %BB4
 
 BB3:
   br label %BB1
 }