[AMDGPUUnifyDivergentExitNodes] Add NewPM support

[gandhi21299]

Summary

Meanwhile, use UniformityAnalysis instead of LegacyDivergenceAnalysis to collect divergence info.

[gandhi21299]

• added checks for the opt command in si-annotate-nested-control-flow.ll
• applied clang-format

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

Not sure why you really need to change this, but the function should probably be a template argument. Is DA just missing a layer to present the PM independent analysis?

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

The LegacyPM version depends on LegacyDivergenceAnalysis info whereas the NewPM version of this pass depends on DivergenceInfo.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

There should be a common logic DivergenceInfo that both pass versions export

[gandhi21299]

• isUniformlyReached now accepts a template argument instead of a std::function to allow both pass managers to use their corresponding DivergenceAnalysis info.
• DivergenceAnalysis::isUniform(Value &V) -> DivergenceAnalysis::isUniform(Value *V)

[gandhi21299]

Rebased and applied clang-format

[gandhi21299]

• Rebased and initialized TTI = nullptr

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

How do these changes look @arsenm ?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

I'm still confused. LegacyDivergenceAnalysis and DivergenceAnalysis/DivergenceInfo are two *different* analyses. LegacyDivergenceAnalysis is *not* the Legacy pass manager version of DivergenceAnalysis. New pass manager shouldn't invisibly switch the analysis used. I think LegacyDivergenceAnalysis needs to be ported to new PM first

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

Ahh I guess I got confused with the names as well. Sure, I will port LegacyDivergenceAnalysis to newPM first.

[gandhi21299]

Rebased

[gandhi21299]

Added missing newline

[gandhi21299]

minor nits

[sameerds]

At this point, there are three analyses as follows:

1. LegacyDA is actually wrong, but useful in some cases involving irreducible control flow.
2. GpuDA is correct, but treats irreducible control flow very conservatively.
3. UniformityAnalysis is both correct and better at irreducible control flow.

Note that in almost every use on AMDGPU, LegacyDA simply acts as a wrapper for GpuDA.

Since this is fresh work, why not just go straight to UniformityAnalysis? I would definitely recommend that because not only does go straight to new stuff, it also avoids the unnecessary template. This would be a good time to use the new analysis. And if issues crop up that can't be fixed, then we should fall back no further than GpuDA, and unify their exposed Info objects along the way.

llvm/include/llvm/Analysis/DivergenceAnalysis.h

177

Why is this change necessary? The point of accepting a reference is to indicate that nullptr is not expected inside the function.

[sameerds]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120–121

No! Don't even try. It was considered and discarded long ago.

337

As already noted, this legacy is not related to the pass manager. Just use the same DA in both.

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

120

Can you give a more meaningful name to the template argument? For instance, DivergenceAnalysisType?

[ruiling]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

190–192

I think you just removed this accidentally.

[gandhi21299]

I will create a new patch to replace LegacyDivergenceAnalysis with Uniformity analysis, unless it's too early to do so.

[gandhi21299]

• Rebased with uniformity analysis dependency
• Removed template param
• Refactors

[sameerds]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

187–194

Either declare parameters as references, or implement checks for nullptr in the function body. Also, can any of these be const?

190–192

Was this comment addressed?

335–336

If you know that a variable cannot be nullptr, it is preferable to declare it as a reference rather than a pointer. It can be converted into a pointer at the point where some function expects a pointer. For example, the call below can be "whatever.run(&DT, &PDT, UA)".

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

190–192

yup, I moved it under AMDGPUUnifyDivergentExitNodesImpl::run(..)

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

187–194

DominatorTree may be updated but PostDominatorTree and UniformityInfo are preserved.

[gandhi21299]

• Addressed reviewer comments

[foad]

The commit message should make it clear that you're also changing it to use UniformityAnalysis instead of (Legacy?)DivergenceAnalysis.

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.h

57 ↗

(On Diff #504953)

It looks like only this class needs to be declared in the .h file. The others could stay in the .cpp file.

[gandhi21299]

• Move AMDGPUUnifyDivergentExitNodesImpl and AMDGPUUnifyDivergentExitNodes classes to the source file from the header file.

[gandhi21299]

• Move includes from header to source

[gandhi21299]

• do not need to include PassManager
• TTI should be private instead of protected in the Impl

[sameerds]

LGTM, but wait for a day to let other reviewers comment.

[gandhi21299]

Internal CI passed.

[vitalybuka]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

340

DT can be null where, which is UB https://lab.llvm.org/buildbot/#/builders/5/builds/32228/steps/16/logs/stdio

[gandhi21299]

Impl::run(..) now accepts DominatorTree* instead of DominatorTree& to avoid undefined behavior due to the possibility of DT == nullptr.

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

340

Great catch! How does it look now?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

186–187

When is DT null but PDT won't be?

[gandhi21299]

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

186–187

This pass depends on PDT so it can never be null. DT may be null if it does not need to be preserved, which is specified by -simplifycfg-require-and-preserve-domtree.

llvm/include/llvm/Analysis/DivergenceAnalysis.h

public:
   }
 
   /// Whether \p U is divergent. Uses of a uniform value can be divergent.
   bool isDivergentUse(const Use &U) const {
     return ContainsIrreducible || DA->isDivergentUse(U);
   }
 
   /// Whether \p V is uniform/non-divergent.
-  bool isUniform(const Value &V) const { return !isDivergent(V); }
+  bool isUniform(const Value *V) const { return !isDivergent(*V); }
 
   /// Whether \p U is uniform/non-divergent. Uses of a uniform value can be
   /// divergent.
   bool isUniformUse(const Use &U) const { return !isDivergentUse(U); }
 };
 
 /// \brief Divergence analysis frontend for GPU kernels.
 class DivergenceAnalysis : public AnalysisInfoMixin<DivergenceAnalysis> {

llvm/lib/Target/AMDGPU/AMDGPU.h

 extern char &SIFormMemoryClausesID;
 
 void initializeSIPostRABundlerPass(PassRegistry&);
 extern char &SIPostRABundlerID;
 
 void initializeGCNCreateVOPDPass(PassRegistry &);
 extern char &GCNCreateVOPDID;
 
-void initializeAMDGPUUnifyDivergentExitNodesPass(PassRegistry&);
+void initializeAMDGPUUnifyDivergentExitNodesLegacyPass(PassRegistry &);
 extern char &AMDGPUUnifyDivergentExitNodesID;
 
+struct AMDGPUUnifyDivergentExitNodesPass
+    : PassInfoMixin<AMDGPUUnifyDivergentExitNodesPass> {
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+
 ImmutablePass *createAMDGPUAAWrapperPass();
 void initializeAMDGPUAAWrapperPassPass(PassRegistry&);
 ImmutablePass *createAMDGPUExternalAAWrapperPass();
 void initializeAMDGPUExternalAAWrapperPass(PassRegistry&);
 
 void initializeAMDGPUArgumentUsageInfoPass(PassRegistry &);
 
 ModulePass *createAMDGPUOpenCLEnqueuedBlockLoweringPass();

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
   initializeSIPreEmitPeepholePass(*PR);
   initializeSILateBranchLoweringPass(*PR);
   initializeSIMemoryLegalizerPass(*PR);
   initializeSIOptimizeExecMaskingPass(*PR);
   initializeSIPreAllocateWWMRegsPass(*PR);
   initializeSIFormMemoryClausesPass(*PR);
   initializeSIPostRABundlerPass(*PR);
   initializeGCNCreateVOPDPass(*PR);
-  initializeAMDGPUUnifyDivergentExitNodesPass(*PR);
+  initializeAMDGPUUnifyDivergentExitNodesLegacyPass(*PR);
   initializeAMDGPUAAWrapperPassPass(*PR);
   initializeAMDGPUExternalAAWrapperPass(*PR);
   initializeAMDGPUUseNativeCallsPass(*PR);
   initializeAMDGPUSimplifyLibCallsPass(*PR);
   initializeAMDGPUPrintfRuntimeBindingPass(*PR);
   initializeAMDGPUResourceUsageAnalysisPass(*PR);
   initializeGCNNSAReassignPass(*PR);
   initializeGCNPreRAOptimizationsPass(*PR);
PB.registerPipelineParsingCallback(
         if (PassName == "amdgpu-propagate-attributes-early") {
           PM.addPass(AMDGPUPropagateAttributesEarlyPass(*this));
           return true;
         }
         if (PassName == "amdgpu-promote-kernel-arguments") {
           PM.addPass(AMDGPUPromoteKernelArgumentsPass());
           return true;
         }
+        if (PassName == "amdgpu-unify-divergent-exit-nodes") {
+          PM.addPass(AMDGPUUnifyDivergentExitNodesPass());
+          return true;
+        }
         return false;
       });
 
   PB.registerAnalysisRegistrationCallback([](FunctionAnalysisManager &FAM) {
     FAM.registerPass([&] { return AMDGPUAA(); });
   });
 
   PB.registerParseAACallback([](StringRef AAName, AAManager &AAM) {

llvm/lib/Target/AMDGPU/AMDGPUUnifyDivergentExitNodes.cpp

 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "SIDefines.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Analysis/DivergenceAnalysis.h"
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/IntrinsicsAMDGPU.h"
+#include "llvm/IR/PassManager.h"
 #include "llvm/IR/Type.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Utils.h"
 #include "llvm/Transforms/Utils/Local.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "amdgpu-unify-divergent-exit-nodes"
 
 namespace {
 
-class AMDGPUUnifyDivergentExitNodes : public FunctionPass {
+class AMDGPUUnifyDivergentExitNodesLegacy : public FunctionPass {
 private:
-  const TargetTransformInfo *TTI = nullptr;
+  const TargetTransformInfo *TTI;
 
 public:
   static char ID; // Pass identification, replacement for typeid
 
-  AMDGPUUnifyDivergentExitNodes() : FunctionPass(ID) {
-    initializeAMDGPUUnifyDivergentExitNodesPass(*PassRegistry::getPassRegistry());
+  AMDGPUUnifyDivergentExitNodesLegacy() : FunctionPass(ID) {
+    initializeAMDGPUUnifyDivergentExitNodesLegacyPass(
+        *PassRegistry::getPassRegistry());
   }
 
   // We can preserve non-critical-edgeness when we unify function exit nodes
   void getAnalysisUsage(AnalysisUsage &AU) const override;
-  BasicBlock *unifyReturnBlockSet(Function &F, DomTreeUpdater &DTU,
-                                  ArrayRef<BasicBlock *> ReturningBlocks,
-                                  StringRef Name);
   bool runOnFunction(Function &F) override;
 };
 
 } // end anonymous namespace
 
-char AMDGPUUnifyDivergentExitNodes::ID = 0;
+char AMDGPUUnifyDivergentExitNodesLegacy::ID = 0;
 
-char &llvm::AMDGPUUnifyDivergentExitNodesID = AMDGPUUnifyDivergentExitNodes::ID;
+char &llvm::AMDGPUUnifyDivergentExitNodesID =
+    AMDGPUUnifyDivergentExitNodesLegacy::ID;
 
-INITIALIZE_PASS_BEGIN(AMDGPUUnifyDivergentExitNodes, DEBUG_TYPE,
+INITIALIZE_PASS_BEGIN(AMDGPUUnifyDivergentExitNodesLegacy, DEBUG_TYPE,
                       "Unify divergent function exit nodes", false, false)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(LegacyDivergenceAnalysis)
-INITIALIZE_PASS_END(AMDGPUUnifyDivergentExitNodes, DEBUG_TYPE,
+INITIALIZE_PASS_END(AMDGPUUnifyDivergentExitNodesLegacy, DEBUG_TYPE,
                     "Unify divergent function exit nodes", false, false)
 
-void AMDGPUUnifyDivergentExitNodes::getAnalysisUsage(AnalysisUsage &AU) const{
+void AMDGPUUnifyDivergentExitNodesLegacy::getAnalysisUsage(
+    AnalysisUsage &AU) const {
   if (RequireAndPreserveDomTree)
     AU.addRequired<DominatorTreeWrapperPass>();
 
   AU.addRequired<PostDominatorTreeWrapperPass>();
-
   AU.addRequired<LegacyDivergenceAnalysis>();
 
   if (RequireAndPreserveDomTree) {
     AU.addPreserved<DominatorTreeWrapperPass>();
     // FIXME: preserve PostDominatorTreeWrapperPass
   }
 
   // No divergent values are changed, only blocks and branch edges.
   AU.addPreserved<LegacyDivergenceAnalysis>();
 
   // We preserve the non-critical-edgeness property
   AU.addPreservedID(BreakCriticalEdgesID);
 
   // This is a cluster of orthogonal Transforms
   AU.addPreservedID(LowerSwitchID);
   FunctionPass::getAnalysisUsage(AU);
 
   AU.addRequired<TargetTransformInfoWrapperPass>();
 }
 
 /// \returns true if \p BB is reachable through only uniform branches.
 /// XXX - Is there a more efficient way to find this?
-static bool isUniformlyReached(const LegacyDivergenceAnalysis &DA,
+template<typename T>
+static bool isUniformlyReached(const T *DA,
                                BasicBlock &BB) {
   SmallVector<BasicBlock *, 8> Stack(predecessors(&BB));
   SmallPtrSet<BasicBlock *, 8> Visited;
 
   while (!Stack.empty()) {
     BasicBlock *Top = Stack.pop_back_val();
-    if (!DA.isUniform(Top->getTerminator()))
+    if (!DA->isUniform(Top->getTerminator()))
       return false;
 
     for (BasicBlock *Pred : predecessors(Top)) {
       if (Visited.insert(Pred).second)
         Stack.push_back(Pred);
     }
   }
 
   return true;
 }
 
-BasicBlock *AMDGPUUnifyDivergentExitNodes::unifyReturnBlockSet(
-    Function &F, DomTreeUpdater &DTU, ArrayRef<BasicBlock *> ReturningBlocks,
-    StringRef Name) {
+BasicBlock *unifyReturnBlockSet(Function &F, DomTreeUpdater &DTU,
+                                ArrayRef<BasicBlock *> ReturningBlocks,
+                                StringRef Name,
+                                const TargetTransformInfo *TTI) {
   // Otherwise, we need to insert a new basic block into the function, add a PHI
   // nodes (if the function returns values), and convert all of the return
   // instructions into unconditional branches.
   BasicBlock *NewRetBlock = BasicBlock::Create(F.getContext(), Name, &F);
   IRBuilder<> B(NewRetBlock);
 
   PHINode *PN = nullptr;
   if (F.getReturnType()->isVoidTy()) {
BasicBlock *unifyReturnBlockSet(Function &F, DomTreeUpdater &DTU,
   for (BasicBlock *BB : ReturningBlocks) {
     // Cleanup possible branch to unconditional branch to the return.
     simplifyCFG(BB, *TTI, RequireAndPreserveDomTree ? &DTU : nullptr,
                 SimplifyCFGOptions().bonusInstThreshold(2));
   }
 
   return NewRetBlock;
 }
 
-bool AMDGPUUnifyDivergentExitNodes::runOnFunction(Function &F) {
-  DominatorTree *DT = nullptr;
-  if (RequireAndPreserveDomTree)
-    DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-
-  auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
-  if (PDT.root_size() == 0 ||
-      (PDT.root_size() == 1 &&
-       !isa<BranchInst>(PDT.getRoot()->getTerminator())))
+template<typename T>
+bool unifyDivergentExitNodesImpl(Function &F, DominatorTree *DT,
+                                 PostDominatorTree *PDT,
+                                 T DA,
+                                 const TargetTransformInfo *TTI) {
+  if (PDT->root_size() == 0 ||
+      (PDT->root_size() == 1 &&
+       !isa<BranchInst>(PDT->getRoot()->getTerminator())))
     return false;
 
-  LegacyDivergenceAnalysis &DA = getAnalysis<LegacyDivergenceAnalysis>();
-  TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-
   // Loop over all of the blocks in a function, tracking all of the blocks that
   // return.
   SmallVector<BasicBlock *, 4> ReturningBlocks;
   SmallVector<BasicBlock *, 4> UnreachableBlocks;
 
   // Dummy return block for infinite loop.
   BasicBlock *DummyReturnBB = nullptr;
 
   bool Changed = false;
   std::vector<DominatorTree::UpdateType> Updates;
 
   // TODO: For now we unify all exit blocks, even though they are uniformly
   // reachable, if there are any exits not uniformly reached. This is to
   // workaround the limitation of structurizer, which can not handle multiple
   // function exits. After structurizer is able to handle multiple function
   // exits, we should only unify UnreachableBlocks that are not uniformly
   // reachable.
-  bool HasDivergentExitBlock = llvm::any_of(
-      PDT.roots(), [&](auto BB) { return !isUniformlyReached(DA, *BB); });
+  bool HasDivergentExitBlock = llvm::any_of(PDT->roots(), [&](auto BB) {
+    return !isUniformlyReached(DA, *BB);
+  });
 
-  for (BasicBlock *BB : PDT.roots()) {
+  for (BasicBlock *BB : PDT->roots()) {
     if (isa<ReturnInst>(BB->getTerminator())) {
       if (HasDivergentExitBlock)
         ReturningBlocks.push_back(BB);
     } else if (isa<UnreachableInst>(BB->getTerminator())) {
       if (HasDivergentExitBlock)
         UnreachableBlocks.push_back(BB);
     } else if (BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator())) {
 
bool unifyDivergentExitNodesImpl(Function &F, DominatorTree *DT,
 
   // Now handle return blocks.
   if (ReturningBlocks.empty())
     return Changed; // No blocks return
 
   if (ReturningBlocks.size() == 1)
     return Changed; // Already has a single return block
 
-  unifyReturnBlockSet(F, DTU, ReturningBlocks, "UnifiedReturnBlock");
+  unifyReturnBlockSet(F, DTU, ReturningBlocks, "UnifiedReturnBlock", TTI);
   return true;
 }
+
+bool AMDGPUUnifyDivergentExitNodesLegacy::runOnFunction(Function &F) {
+  DominatorTree *DT = nullptr;
+  if (RequireAndPreserveDomTree)
+    DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
+
+  auto &PDT = getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
+  LegacyDivergenceAnalysis &DA = getAnalysis<LegacyDivergenceAnalysis>();
+  TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+  return unifyDivergentExitNodesImpl(F, DT, &PDT, &DA, TTI);
+}
+
+PreservedAnalyses
+AMDGPUUnifyDivergentExitNodesPass::run(Function &F,
+                                       FunctionAnalysisManager &AM) {
+  DominatorTree *DT = nullptr;
+  if (RequireAndPreserveDomTree)
+    DT = &AM.getResult<DominatorTreeAnalysis>(F);
+
+  auto *PDT = &AM.getResult<PostDominatorTreeAnalysis>(F);
+  auto *DA = &AM.getResult<DivergenceAnalysis>(F);
+  auto *TTI = &AM.getResult<TargetIRAnalysis>(F);
+  return unifyDivergentExitNodesImpl(F, DT, PDT, DA, TTI)
+             ? PreservedAnalyses::none()
+             : PreservedAnalyses::all();
+}

llvm/test/CodeGen/AMDGPU/si-annotate-nested-control-flows.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: llc -mtriple=amdgcn-amd-amdhsa %s -o - | FileCheck %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -p simplifycfg,amdgpu-unify-divergent-exit-nodes %s -S -o - | FileCheck %s --check-prefix=OPT
+; RUN: llc -mtriple=amdgcn-amd-amdhsa %s -o - | FileCheck %s --check-prefix=ISA
 
 define void @nested_inf_loop(i1 %0, i1 %1) {
-; CHECK-LABEL: nested_inf_loop:
-; CHECK-NEXT: %bb.0:                                ; %BB
-; CHECK-NEXT:   s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:   v_and_b32_e32 v1, 1, v1
-; CHECK-NEXT:   v_and_b32_e32 v0, 1, v0
-; CHECK-NEXT:   v_cmp_eq_u32_e64 s[4:5], 1, v1
-; CHECK-NEXT:   v_cmp_eq_u32_e32 vcc, 1, v0
-; CHECK-NEXT:   s_xor_b64 s[6:7], vcc, -1
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT: .LBB0_1:                                ; %BB1
-; CHECK:        s_and_b64 s[10:11], exec, s[6:7]
-; CHECK-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
-; CHECK-NEXT:   s_andn2_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_cbranch_execnz .LBB0_1
-; CHECK-NEXT: %bb.2:                                ; %BB2
-; CHECK:        s_or_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT: .LBB0_3:                                ; %BB4
-; CHECK:        s_and_b64 s[10:11], exec, s[4:5]
-; CHECK-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
-; CHECK-NEXT:   s_andn2_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_cbranch_execnz .LBB0_3
-; CHECK-NEXT: %bb.4:                                ; %loop.exit.guard
-; CHECK:        s_or_b64 exec, exec, s[8:9]
-; CHECK-NEXT:   s_mov_b64 vcc, 0
-; CHECK-NEXT:   s_mov_b64 s[8:9], 0
-; CHECK-NEXT:   s_branch .LBB0_1
-; CHECK-NEXT: %bb.5:                                ; %DummyReturnBlock
-; CHECK-NEXT:   s_setpc_b64 s[30:31]
+; OPT-LABEL: @nested_inf_loop(
+; OPT-NEXT:  BB:
+; OPT-NEXT:    br label [[BB1:%.*]]
+; OPT:       BB1:
+; OPT-NEXT:    [[BRMERGE:%.*]] = select i1 [[TMP0:%.*]], i1 true, i1 [[TMP1:%.*]]
+; OPT-NEXT:    br i1 [[BRMERGE]], label [[BB1]], label [[INFLOOP:%.*]]
+; OPT:       infloop:
+; OPT-NEXT:    br i1 true, label [[INFLOOP]], label [[DUMMYRETURNBLOCK:%.*]]
+; OPT:       DummyReturnBlock:
+; OPT-NEXT:    ret void
+;
+; ISA-LABEL: nested_inf_loop:
+; ISA-NEXT: %bb.0:                                ; %BB
+; ISA-NEXT:   s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; ISA-NEXT:   v_and_b32_e32 v1, 1, v1
+; ISA-NEXT:   v_and_b32_e32 v0, 1, v0
+; ISA-NEXT:   v_cmp_eq_u32_e64 s[4:5], 1, v1
+; ISA-NEXT:   v_cmp_eq_u32_e32 vcc, 1, v0
+; ISA-NEXT:   s_xor_b64 s[6:7], vcc, -1
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT: .LBB0_1:                                ; %BB1
+; ISA:        s_and_b64 s[10:11], exec, s[6:7]
+; ISA-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
+; ISA-NEXT:   s_andn2_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_cbranch_execnz .LBB0_1
+; ISA-NEXT: %bb.2:                                ; %BB2
+; ISA:        s_or_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT: .LBB0_3:                                ; %BB4
+; ISA:        s_and_b64 s[10:11], exec, s[4:5]
+; ISA-NEXT:   s_or_b64 s[8:9], s[10:11], s[8:9]
+; ISA-NEXT:   s_andn2_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_cbranch_execnz .LBB0_3
+; ISA-NEXT: %bb.4:                                ; %loop.exit.guard
+; ISA:        s_or_b64 exec, exec, s[8:9]
+; ISA-NEXT:   s_mov_b64 vcc, 0
+; ISA-NEXT:   s_mov_b64 s[8:9], 0
+; ISA-NEXT:   s_branch .LBB0_1
+; ISA-NEXT: %bb.5:                                ; %DummyReturnBlock
+; ISA-NEXT:   s_setpc_b64 s[30:31]
 BB:
   br label %BB1
 
 BB1:
   br i1 %0, label %BB3, label %BB2
 
 BB2:
   br label %BB4
 
 BB4:
   br i1 %1, label %BB3, label %BB4
 
 BB3:
   br label %BB1
 }