This is an archive of the discontinued LLVM Phabricator instance.

Differential D149646

[LLVM][Uniformity] Propagate temporal divergence explicitly
ClosedPublic

Authored by sameerds on May 2 2023, 5:50 AM.

Details

Reviewers
Petar.Avramovic
foad
nhaehnle
Commits
rGb0f0dd2554c7: [LLVM][Uniformity] Propagate temporal divergence explicitly
Summary

At a cycle C with divergent exits, UA was using a naive traversal of the exiting
edges to locate blocks that may use values defined inside C. But this traversal
fails when it encounters a cycle. This is now replaced with a much simpler
propagation that iterates over every instruction in C and checks any uses that
are outside C. But such an iteration can be expensive when C is very large; the
original strategy may need to be reconsidered if there is a regression in
compilation times.

Also fixed lit tests that should have originally caught the missed propagation
of temporal divergence.

Diff Detail

Unit TestsFailed

TimeTest
11,280 msx64 debian > AddressSanitizer-x86_64-linux-dynamic.TestCases/Linux::uar_signals.cpp

Event Timeline

sameerds created this revision.May 2 2023, 5:50 AM
Herald added a project: Restricted Project. · View Herald TranscriptMay 2 2023, 5:50 AM
Herald added subscribers: kosarev, kerbowa, mgrang and 2 others. · View Herald Transcript
sameerds requested review of this revision.May 2 2023, 5:50 AM
Herald added a project: Restricted Project. · View Herald TranscriptMay 2 2023, 5:50 AM
Herald added a subscriber: llvm-commits. · View Herald Transcript
sameerds added reviewers: Petar.Avramovic, foad, nhaehnle.May 2 2023, 5:51 AM
sameerds added a subscriber: Restricted Project.
Herald added a subscriber: StephenFan. · View Herald TranscriptMay 2 2023, 5:51 AM
foad added inline comments.May 2 2023, 6:06 AM
llvm/include/llvm/ADT/GenericUniformityImpl.h
456–457

Comment should be updated. It doesn't mention I.

846–850

I wonder if there is a quick test you could use to avoid scanning some BBs, e.g. I think if BB does not dominate any edges that exit OuterDivCycle then instructions in BB cannot have any uses outside OuterDivCycle. This should be fairly common because it applies to the body of a simple C "for" loop.

llvm/lib/Analysis/UniformityAnalysis.cpp
79–80

Just iterate over users() instead of uses().

Harbormaster completed remote builds in B229406: Diff 518693.May 2 2023, 6:57 AM
dstuttard added a subscriber: dstuttard.May 12 2023, 8:21 AM

Confirmed that this fixes an issue we observed in an app.

sameerds updated this revision to Diff 521993.May 14 2023, 5:02 AM
  • only examine blocks that dominate an exit.
  • FIXME about narrowing this down to live-out values at each divergent exit.
Harbormaster completed remote builds in B231858: Diff 521993.May 14 2023, 5:49 AM
yassingh added a subscriber: yassingh.May 14 2023, 8:31 AM
yassingh added inline comments.
llvm/lib/CodeGen/MachineUniformityAnalysis.cpp
117–136

Is the case described in the summary also triggering for MIR? If yes maybe you can add a relevant test.

sameerds marked 3 inline comments as done.May 14 2023, 11:14 PM
sameerds added inline comments.
llvm/lib/CodeGen/MachineUniformityAnalysis.cpp
117–136

I just tried adding an identical MIR test. Unfortunately, it runs into the problem with SI_LOOP that is being fixed in D150438. I'll add the test to that review.

foad accepted this revision.May 15 2023, 2:24 AM

Looks fine to me, thanks!

llvm/include/llvm/ADT/GenericUniformityImpl.h
817–819

Conceptually you could unify these two cases by looking for blocks B that dominate an exit edge; but I have no idea whether DomTree has good support for querying dominance on CFG edges.

839

Might be cleaner to use any_of?

This revision is now accepted and ready to land.May 15 2023, 2:24 AM
foad added inline comments.May 15 2023, 3:28 AM
llvm/include/llvm/ADT/GenericUniformityImpl.h
818

Nit: say "DefCycle" instead of "C" (which you have not defined)?

This revision was landed with ongoing or failed builds.May 15 2023, 7:48 AM
Closed by commit rGb0f0dd2554c7: [LLVM][Uniformity] Propagate temporal divergence explicitly (authored by sameerds). · Explain Why
This revision was automatically updated to reflect the committed changes.
sameerds added a commit: rGb0f0dd2554c7: [LLVM][Uniformity] Propagate temporal divergence explicitly.
sameerds marked 2 inline comments as done.May 15 2023, 8:01 AM
sameerds added inline comments.
llvm/include/llvm/ADT/GenericUniformityImpl.h
817–819

I had a cursory look. The DominatorTree interface has methods to query edges dominating uses, but none to check for defs dominating edges.

Revision Contents

PathSize
llvm/
include/
llvm/
ADT/
133 lines
lib/
Analysis/
15 lines
CodeGen/
22 lines
test/
Analysis/
UniformityAnalysis/
AMDGPU/
73 lines

Diff 521993

llvm/include/llvm/ADT/GenericUniformityImpl.h

Show First 20 Lines • Show All 445 Lines • ▼ Show 20 Linesprivate:
void taintAndPushPhiNodes(const BlockT &JoinBlock); void taintAndPushPhiNodes(const BlockT &JoinBlock);
/// \brief Identify all Instructions that become divergent because \p DivExit /// \brief Identify all Instructions that become divergent because \p DivExit
/// is a divergent cycle exit of \p DivCycle. Mark those instructions as /// is a divergent cycle exit of \p DivCycle. Mark those instructions as
/// divergent and push them on the worklist. /// divergent and push them on the worklist.
void propagateCycleExitDivergence(const BlockT &DivExit, void propagateCycleExitDivergence(const BlockT &DivExit,
const CycleT &DivCycle); const CycleT &DivCycle);
/// \brief Internal implementation function for propagateCycleExitDivergence. /// Mark as divergent all external uses of values defined in \p DefCycle.
void analyzeCycleExitDivergence(const CycleT &OuterDivCycle); void analyzeCycleExitDivergence(const CycleT &DefCycle);
/// \brief Mark all instruction as divergent that use a value defined in \p /// \brief Mark as divergent all uses of \p I that are outside \p DefCycle.
foadUnsubmitted
Done
ReplyInline Actions

Comment should be updated. It doesn't mention I.

foad: Comment should be updated. It doesn't mention I.
/// OuterDivCycle. Push their users on the worklist. void propagateTemporalDivergence(const InstructionT &I,
void analyzeTemporalDivergence(const InstructionT &I, const CycleT &DefCycle);
const CycleT &OuterDivCycle);
/// \brief Push all users of \p Val (in the region) to the worklist. /// \brief Push all users of \p Val (in the region) to the worklist.
void pushUsers(const InstructionT &I); void pushUsers(const InstructionT &I);
void pushUsers(ConstValueRefT V); void pushUsers(ConstValueRefT V);
bool usesValueFromCycle(const InstructionT &I, const CycleT &DefCycle) const; bool usesValueFromCycle(const InstructionT &I, const CycleT &DefCycle) const;
/// \brief Whether \p Def is divergent when read in \p ObservingBlock. /// \brief Whether \p Def is divergent when read in \p ObservingBlock.
▲ Show 20 Lines • Show All 335 Lines • ▼ Show 20 Lines
} }
template <typename ContextT> template <typename ContextT>
void GenericUniformityAnalysisImpl<ContextT>::addUniformOverride( void GenericUniformityAnalysisImpl<ContextT>::addUniformOverride(
const InstructionT &Instr) { const InstructionT &Instr) {
UniformOverrides.insert(&Instr); UniformOverrides.insert(&Instr);
} }
template <typename ContextT> // Mark as divergent all external uses of values defined in \p DefCycle.
void GenericUniformityAnalysisImpl<ContextT>::analyzeTemporalDivergence(
const InstructionT &I, const CycleT &OuterDivCycle) {
if (isDivergent(I))
return;
LLVM_DEBUG(dbgs() << "Analyze temporal divergence: " << Context.print(&I)
<< "\n");
if (isAlwaysUniform(I))
return;
if (!usesValueFromCycle(I, OuterDivCycle))
return;
if (markDivergent(I))
Worklist.push_back(&I);
}
// Mark all external users of values defined inside \param
// OuterDivCycle as divergent.
// //
// This follows all live out edges wherever they may lead. Potential // A value V defined by a block B inside \p DefCycle may be used outside the
// users of values defined inside DivCycle could be anywhere in the // cycle only if the use is a PHI in some exit block, or B dominates some exit
// dominance region of DivCycle (including its fringes for phi nodes). // block. Thus, we check uses as follows:
// A cycle C dominates a block B iff every path from the entry block //
// to B must pass through a block contained in C. If C is a reducible // - Check all PHIs in all exit blocks for inputs defined in the cycle.
// cycle (or natural loop), C dominates B iff the header of C // - For every block B inside C that dominates at least one exit block, check
foadUnsubmitted
Done
ReplyInline Actions

Nit: say "DefCycle" instead of "C" (which you have not defined)?

foad: Nit: say "DefCycle" instead of "C" (which you have not defined)?
// dominates B. But in general, we iteratively examine cycle cycle // all uses outside C.
foadUnsubmitted
Not Done
ReplyInline Actions

Conceptually you could unify these two cases by looking for blocks B that dominate an exit edge; but I have no idea whether DomTree has good support for querying dominance on CFG edges.

foad: Conceptually you could unify these two cases by looking for blocks B that dominate an exit…
sameerdsAuthorUnsubmitted
Done
ReplyInline Actions

I had a cursory look. The DominatorTree interface has methods to query edges dominating uses, but none to check for defs dominating edges.

sameerds: I had a cursory look. The DominatorTree interface has methods to query edges dominating uses…
// exits and their successors. //
// FIXME: This function does not distinguish between divergent and uniform
// exits. For each divergent exit, only the values that are live at that exit
// need to be propagated as divergent at their use outside the cycle.
template <typename ContextT> template <typename ContextT>
void GenericUniformityAnalysisImpl<ContextT>::analyzeCycleExitDivergence( void GenericUniformityAnalysisImpl<ContextT>::analyzeCycleExitDivergence(
const CycleT &OuterDivCycle) { const CycleT &DefCycle) {
// Set of blocks that are dominated by the cycle, i.e., each is only SmallVector<BlockT *> Exits;
// reachable from paths that pass through the cycle. DefCycle.getExitBlocks(Exits);
SmallPtrSet<BlockT *, 16> DomRegion; for (auto *Exit : Exits) {
for (auto &Phi : Exit->phis()) {
// The boundary of DomRegion, formed by blocks that are not if (usesValueFromCycle(Phi, DefCycle)) {
// dominated by the cycle. if (markDivergent(Phi))
SmallVector<BlockT *> DomFrontier; Worklist.push_back(&Phi);
OuterDivCycle.getExitBlocks(DomFrontier);
// Returns true if BB is dominated by the cycle.
auto isInDomRegion = [&](BlockT *BB) {
for (auto *P : predecessors(BB)) {
if (OuterDivCycle.contains(P))
continue;
if (DomRegion.count(P))
continue;
return false;
}
return true;
};
// Keep advancing the frontier along successor edges, while
// promoting blocks to DomRegion.
while (true) {
bool Promoted = false;
SmallVector<BlockT *> Temp;
for (auto *W : DomFrontier) {
if (!isInDomRegion(W)) {
Temp.push_back(W);
continue;
} }
DomRegion.insert(W);
Promoted = true;
for (auto *Succ : successors(W)) {
if (DomRegion.contains(Succ))
continue;
Temp.push_back(Succ);
} }
} }
if (!Promoted)
break;
// Restore the set property for the temporary vector auto dominatesAnExit = [&](BlockT *BB) {
llvm::sort(Temp); for (auto *Exit : Exits) {
foadUnsubmitted
Done
ReplyInline Actions

Might be cleaner to use any_of?

foad: Might be cleaner to use `any_of`?
Temp.erase(std::unique(Temp.begin(), Temp.end()), Temp.end()); if (DT.dominates(BB, Exit))
return true;
DomFrontier = Temp;
}
// At DomFrontier, only the PHI nodes are affected by temporal
// divergence.
for (const auto *UserBlock : DomFrontier) {
LLVM_DEBUG(dbgs() << "Analyze phis after cycle exit: "
<< Context.print(UserBlock) << "\n");
for (const auto &Phi : UserBlock->phis()) {
LLVM_DEBUG(dbgs() << " " << Context.print(&Phi) << "\n");
analyzeTemporalDivergence(Phi, OuterDivCycle);
}
} }
return false;
};
// All instructions inside the dominance region are affected by for (auto *BB : DefCycle.blocks()) {
// temporal divergence. if (!dominatesAnExit(BB))
for (const auto *UserBlock : DomRegion) { continue;
LLVM_DEBUG(dbgs() << "Analyze non-phi users after cycle exit: " for (auto &II : *BB) {
<< Context.print(UserBlock) << "\n"); propagateTemporalDivergence(II, DefCycle);
foadUnsubmitted
Done
ReplyInline Actions

I wonder if there is a quick test you could use to avoid scanning some BBs, e.g. I think if BB does not dominate any edges that exit OuterDivCycle then instructions in BB cannot have any uses outside OuterDivCycle. This should be fairly common because it applies to the body of a simple C "for" loop.

foad: I wonder if there is a quick test you could use to avoid scanning some BBs, e.g. I think if BB…
for (const auto &I : *UserBlock) {
LLVM_DEBUG(dbgs() << " " << Context.print(&I) << "\n");
analyzeTemporalDivergence(I, OuterDivCycle);
} }
} }
} }
template <typename ContextT> template <typename ContextT>
void GenericUniformityAnalysisImpl<ContextT>::propagateCycleExitDivergence( void GenericUniformityAnalysisImpl<ContextT>::propagateCycleExitDivergence(
const BlockT &DivExit, const CycleT &InnerDivCycle) { const BlockT &DivExit, const CycleT &InnerDivCycle) {
LLVM_DEBUG(dbgs() << "\tpropCycleExitDiv " << Context.print(&DivExit) LLVM_DEBUG(dbgs() << "\tpropCycleExitDiv " << Context.print(&DivExit)
▲ Show 20 Lines • Show All 513 Lines • Show Last 20 Lines

llvm/lib/Analysis/UniformityAnalysis.cpp

Show First 20 Lines • Show All 70 Lines • ▼ Show 20 Linesbool llvm::GenericUniformityAnalysisImpl<SSAContext>::usesValueFromCycle(
assert(!isAlwaysUniform(I)); assert(!isAlwaysUniform(I));
for (const Use &U : I.operands()) { for (const Use &U : I.operands()) {
if (auto *I = dyn_cast<Instruction>(&U)) { if (auto *I = dyn_cast<Instruction>(&U)) {
if (DefCycle.contains(I->getParent())) if (DefCycle.contains(I->getParent()))
return true; return true;
} }
} }
return false; return false;
} }
foadUnsubmitted
Done
ReplyInline Actions

Just iterate over users() instead of uses().

foad: Just iterate over `users()` instead of `uses()`.
template <> template <>
void llvm::GenericUniformityAnalysisImpl<
SSAContext>::propagateTemporalDivergence(const Instruction &I,
const Cycle &DefCycle) {
if (isDivergent(I))
return;
for (auto *User : I.users()) {
auto *UserInstr = cast<Instruction>(User);
if (DefCycle.contains(UserInstr->getParent()))
continue;
if (markDivergent(*UserInstr))
Worklist.push_back(UserInstr);
}
}
template <>
bool llvm::GenericUniformityAnalysisImpl<SSAContext>::isDivergentUse( bool llvm::GenericUniformityAnalysisImpl<SSAContext>::isDivergentUse(
const Use &U) const { const Use &U) const {
const auto *V = U.get(); const auto *V = U.get();
if (isDivergent(V)) if (isDivergent(V))
return true; return true;
if (const auto *DefInstr = dyn_cast<Instruction>(V)) { if (const auto *DefInstr = dyn_cast<Instruction>(V)) {
const auto *UseInstr = cast<Instruction>(U.getUser()); const auto *UseInstr = cast<Instruction>(U.getUser());
return isTemporalDivergent(*UseInstr->getParent(), *DefInstr); return isTemporalDivergent(*UseInstr->getParent(), *DefInstr);
▲ Show 20 Lines • Show All 80 Lines • Show Last 20 Lines

llvm/lib/CodeGen/MachineUniformityAnalysis.cpp

Show First 20 Lines • Show All 108 Lines • ▼ Show 20 Lines for (auto &Op : I.operands()) {
auto *Def = F.getRegInfo().getVRegDef(Reg); auto *Def = F.getRegInfo().getVRegDef(Reg);
if (DefCycle.contains(Def->getParent())) if (DefCycle.contains(Def->getParent()))
return true; return true;
} }
return false; return false;
} }
template <> template <>
void llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::
propagateTemporalDivergence(const MachineInstr &I,
const MachineCycle &DefCycle) {
const auto &RegInfo = F.getRegInfo();
for (auto &Op : I.operands()) {
if (!Op.isReg() || !Op.isDef())
continue;
if (!Op.getReg().isVirtual())
continue;
auto Reg = Op.getReg();
if (isDivergent(Reg))
continue;
for (MachineInstr &UserInstr : RegInfo.use_instructions(Reg)) {
if (DefCycle.contains(UserInstr.getParent()))
continue;
if (markDivergent(UserInstr))
Worklist.push_back(&UserInstr);
}
}
}
yassinghUnsubmitted
Not Done
ReplyInline Actions

Is the case described in the summary also triggering for MIR? If yes maybe you can add a relevant test.

yassingh: Is the case described in the summary also triggering for MIR? If yes maybe you can add a…
sameerdsAuthorUnsubmitted
Done
ReplyInline Actions

I just tried adding an identical MIR test. Unfortunately, it runs into the problem with SI_LOOP that is being fixed in D150438. I'll add the test to that review.

sameerds: I just tried adding an identical MIR test. Unfortunately, it runs into the problem with SI_LOOP…
template <>
bool llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::isDivergentUse( bool llvm::GenericUniformityAnalysisImpl<MachineSSAContext>::isDivergentUse(
const MachineOperand &U) const { const MachineOperand &U) const {
if (!U.isReg()) if (!U.isReg())
return false; return false;
auto Reg = U.getReg(); auto Reg = U.getReg();
if (isDivergent(Reg)) if (isDivergent(Reg))
return true; return true;
▲ Show 20 Lines • Show All 122 Lines • Show Last 20 Lines

llvm/test/Analysis/UniformityAnalysis/AMDGPU/temporal_diverge.ll

; RUN: opt -mtriple amdgcn-unknown-amdhsa -passes='print<uniformity>' -disable-output %s 2>&1 | FileCheck %s ; RUN: opt -mtriple amdgcn-unknown-amdhsa -passes='print<uniformity>' -disable-output %s 2>&1 | FileCheck %s
; temporal-divergent use of value carried by divergent loop ; temporal-divergent use of value carried by divergent loop
define amdgpu_kernel void @temporal_diverge(i32 %n, i32 %a, i32 %b) #0 { define amdgpu_kernel void @temporal_diverge(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'temporal_diverge': ; CHECK-LABEL: for function 'temporal_diverge':
; CHECK-NOT: DIVERGENT: %uni. ; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni. ; CHECK-NOT: DIVERGENT: br i1 %uni.
entry: entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%uni.cond = icmp slt i32 %a, 0 %uni.cond = icmp slt i32 %a, 0
br label %H br label %H
H: H:
%uni.merge.h = phi i32 [ 0, %entry ], [ %uni.inc, %H ] %uni.merge.h = phi i32 [ 0, %entry ], [ %uni.inc, %H ]
%uni.inc = add i32 %uni.merge.h, 1 %uni.inc = add i32 %uni.merge.h, 1
; CHECK: DIVERGENT: %div.exitx =
%div.exitx = icmp slt i32 %tid, 0 %div.exitx = icmp slt i32 %tid, 0
; CHECK: DIVERGENT: br i1 %div.exitx,
br i1 %div.exitx, label %X, label %H ; divergent branch br i1 %div.exitx, label %X, label %H ; divergent branch
X:
; CHECK: DIVERGENT: %div.user =
%div.user = add i32 %uni.inc, 5
ret void
}
define amdgpu_kernel void @phi_at_exit(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'phi_at_exit':
; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni.
entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x()
%uni.cond = icmp slt i32 %a, 0
br i1 %uni.cond, label %H, label %X
H:
%uni.merge.h = phi i32 [ 0, %entry ], [ %uni.inc, %H ]
%uni.inc = add i32 %uni.merge.h, 1
; CHECK: DIVERGENT: %div.exitx = ; CHECK: DIVERGENT: %div.exitx =
%div.exitx = icmp slt i32 %tid, 0
; CHECK: DIVERGENT: br i1 %div.exitx, ; CHECK: DIVERGENT: br i1 %div.exitx,
br i1 %div.exitx, label %X, label %H ; divergent branch
X: X:
%div.user = add i32 %uni.inc, 5 ; CHECK: DIVERGENT: %div.phi =
%div.phi = phi i32 [ 0, %entry], [ %uni.inc, %H ]
%div.user = add i32 %div.phi, 5
ret void
}
define amdgpu_kernel void @phi_after_exit(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'phi_after_exit':
; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni.
entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x()
%uni.cond = icmp slt i32 %a, 0
br i1 %uni.cond, label %H, label %Y
H:
%uni.merge.h = phi i32 [ 0, %entry ], [ %uni.inc, %H ]
%uni.inc = add i32 %uni.merge.h, 1
; CHECK: DIVERGENT: %div.exitx =
%div.exitx = icmp slt i32 %tid, 0
; CHECK: DIVERGENT: br i1 %div.exitx,
br i1 %div.exitx, label %X, label %H ; divergent branch
X:
br label %Y
Y:
; CHECK: DIVERGENT: %div.phi =
%div.phi = phi i32 [ 0, %entry], [ %uni.inc, %X ]
%div.user = add i32 %div.phi, 5
ret void ret void
} }
; temporal-divergent use of value carried by divergent loop inside a top-level loop ; temporal-divergent use of value carried by divergent loop inside a top-level loop
define amdgpu_kernel void @temporal_diverge_inloop(i32 %n, i32 %a, i32 %b) #0 { define amdgpu_kernel void @temporal_diverge_inloop(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'temporal_diverge_inloop': ; CHECK-LABEL: for function 'temporal_diverge_inloop':
; CHECK-NOT: DIVERGENT: %uni. ; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni. ; CHECK-NOT: DIVERGENT: br i1 %uni.
Show All 19 LinesX:
br i1 %uni.cond, label %G, label %Y br i1 %uni.cond, label %G, label %Y
Y: Y:
%div.alsouser = add i32 %uni.inc, 5 %div.alsouser = add i32 %uni.inc, 5
ret void ret void
} }
; temporal-uniform use of a valud, definition and users are carried by a surrounding divergent loop ; temporal-uniform use of a value, definition and users are carried by a
; surrounding divergent loop
define amdgpu_kernel void @temporal_uniform_indivloop(i32 %n, i32 %a, i32 %b) #0 { define amdgpu_kernel void @temporal_uniform_indivloop(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'temporal_uniform_indivloop': ; CHECK-LABEL: for function 'temporal_uniform_indivloop':
; CHECK-NOT: DIVERGENT: %uni. ; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni. ; CHECK-NOT: DIVERGENT: br i1 %uni.
entry: entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%uni.cond = icmp slt i32 %a, 0 %uni.cond = icmp slt i32 %a, 0
Show All 12 LinesX:
%div.exity = icmp slt i32 %tid, 0 %div.exity = icmp slt i32 %tid, 0
; CHECK: DIVERGENT: %div.exity = ; CHECK: DIVERGENT: %div.exity =
br i1 %div.exity, label %G, label %Y br i1 %div.exity, label %G, label %Y
; CHECK: DIVERGENT: br i1 %div.exity, ; CHECK: DIVERGENT: br i1 %div.exity,
Y: Y:
%div.alsouser = add i32 %uni.inc, 5 %div.alsouser = add i32 %uni.inc, 5
ret void ret void
; CHECK: DIVERGENT: %div.alsouser =
} }
; temporal-divergent use of value carried by divergent loop, user is inside sibling loop ; temporal-divergent use of value carried by divergent loop, user is inside sibling loop
define amdgpu_kernel void @temporal_diverge_loopuser(i32 %n, i32 %a, i32 %b) #0 { define amdgpu_kernel void @temporal_diverge_loopuser(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'temporal_diverge_loopuser': ; CHECK-LABEL: for function 'temporal_diverge_loopuser':
; CHECK-NOT: DIVERGENT: %uni. ; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni. ; CHECK-NOT: DIVERGENT: br i1 %uni.
Show All 12 Lines
; CHECK: DIVERGENT: br i1 %div.exitx, ; CHECK: DIVERGENT: br i1 %div.exitx,
X: X:
br label %G br label %G
G: G:
%div.user = add i32 %uni.inc, 5 %div.user = add i32 %uni.inc, 5
br i1 %uni.cond, label %G, label %Y br i1 %uni.cond, label %G, label %Y
; CHECK: DIVERGENT: %div.user =
Y: Y:
ret void ret void
} }
; temporal-divergent use of value carried by divergent loop, user is inside sibling loop, defs and use are carried by a uniform loop ; temporal-divergent use of value carried by divergent loop, user is inside sibling loop, defs and use are carried by a uniform loop
define amdgpu_kernel void @temporal_diverge_loopuser_nested(i32 %n, i32 %a, i32 %b) #0 { define amdgpu_kernel void @temporal_diverge_loopuser_nested(i32 %n, i32 %a, i32 %b) #0 {
; CHECK-LABEL: for function 'temporal_diverge_loopuser_nested': ; CHECK-LABEL: for function 'temporal_diverge_loopuser_nested':
; CHECK-NOT: DIVERGENT: %uni. ; CHECK-NOT: DIVERGENT: %uni.
; CHECK-NOT: DIVERGENT: br i1 %uni. ; CHECK-NOT: DIVERGENT: br i1 %uni.
entry: entry:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%uni.cond = icmp slt i32 %a, 0 %uni.cond = icmp slt i32 %a, 0
br label %G
G:
br label %H br label %H
H: H:
%uni.merge.h = phi i32 [ 0, %entry ], [ %uni.inc, %H ] %uni.merge.h = phi i32 [ 0, %G ], [ %uni.inc, %H ]
%uni.inc = add i32 %uni.merge.h, 1 %uni.inc = add i32 %uni.merge.h, 1
%div.exitx = icmp slt i32 %tid, 0 %div.exitx = icmp slt i32 %tid, 0
br i1 %div.exitx, label %X, label %H ; divergent branch br i1 %div.exitx, label %X, label %H ; divergent branch
; CHECK: DIVERGENT: %div.exitx = ; CHECK: DIVERGENT: %div.exitx =
; CHECK: DIVERGENT: br i1 %div.exitx, ; CHECK: DIVERGENT: br i1 %div.exitx,
X: X:
br label %G ; CHECK: DIVERGENT: %div.user =
G:
%div.user = add i32 %uni.inc, 5 %div.user = add i32 %uni.inc, 5
br i1 %uni.cond, label %G, label %Y br i1 %uni.cond, label %X, label %G
Y: Y:
ret void ret void
} }
declare i32 @llvm.amdgcn.workitem.id.x() #0 declare i32 @llvm.amdgcn.workitem.id.x() #0
attributes #0 = { nounwind readnone } attributes #0 = { nounwind readnone }