This is an archive of the discontinued LLVM Phabricator instance.

Differential D130742

[AMDGPU] Extend SILoadStoreOptimizer to s_load instructions
ClosedPublic

Authored by critson on Jul 28 2022, 7:08 PM.

Details

Reviewers
rampitec
foad
nhaehnle
Commits
rG4c4db81630e2: [AMDGPU] Extend SILoadStoreOptimizer to s_load instructions
Summary

Apply merging to s_load as is done for s_buffer_load.

Diff Detail

Event Timeline

critson created this revision.Jul 28 2022, 7:08 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 28 2022, 7:08 PM
Herald added subscribers: kosarev, jsilvanus, kerbowa and 8 others. · View Herald Transcript
critson requested review of this revision.Jul 28 2022, 7:08 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 28 2022, 7:08 PM
Herald added subscribers: llvm-commits, wdng. · View Herald Transcript
Harbormaster completed remote builds in B178188: Diff 448499.Jul 28 2022, 8:29 PM
critson added a comment.Jul 28 2022, 11:14 PM

Compilation testing on graphics shader corpus (~10k pipelines) for Navi10.

~15% reduction in s_load_dword instructions.
~1% reduction in s_waitcnt instructions.
~1% increase in s_mov instructions.
~0.75% overall decrease in instructions.

VGPR usage increase in ~400 pipelines (mostly 1-2 VGPRs).
VGPR usage decrease in ~500 pipelines.
Mean VGPR usage change: -0.02 VGPRs.

SGPR usage increase in ~3000 pipelines.
SGPR usage decrease in ~1000 pipelines.
Mean SGPR usage change: 0.62 SGPRs.

No changes in scratch usage.

Occupancy (executable wavefronts based on VGPR count) increased in 234 pipelines, decreased in 125 pipelines.

foad accepted this revision.Jul 29 2022, 7:19 AM

LGTM, thanks! The SILoadStoreOptimizer.cpp diff is impressively small.

In future perhaps we should also handle the *_SGPR_IMM forms of various SMEM loads.

llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp
2335–2336

Maybe rename to something like mergeSMEMLoadImmPair?

This revision is now accepted and ready to land.Jul 29 2022, 7:19 AM
critson marked an inline comment as done.Jul 29 2022, 7:41 PM
This revision was landed with ongoing or failed builds.Jul 29 2022, 7:41 PM
Closed by commit rG4c4db81630e2: [AMDGPU] Extend SILoadStoreOptimizer to s_load instructions (authored by critson). · Explain Why
This revision was automatically updated to reflect the committed changes.
critson added a commit: rG4c4db81630e2: [AMDGPU] Extend SILoadStoreOptimizer to s_load instructions.

Large Diff

This large diff affects 108 files. Files without inline comments have been collapsed. Expand All Files

Revision Contents

PathSize
llvm/
lib/
Target/
AMDGPU/
43 lines
test/
CodeGen/
AMDGPU/
GlobalISel/
110 lines
7 lines
50 lines
168 lines
145 lines
152 lines
13 lines
213 lines
91 lines
16 lines
134 lines
2 lines
779 lines
2 lines
3 lines
24 lines
253 lines
18 lines
229 lines
132 lines
38 lines
491 lines
361 lines
392 lines
446 lines
383 lines
641 lines
9 lines
64 lines
12 lines
9 lines
11 lines
43 lines
20 lines
8 lines
22 lines
14 lines
124 lines
78 lines
74 lines
52 lines
122 lines
88 lines
384 lines
23 lines
95 lines
292 lines
330 lines
27 lines
194 lines
6 lines
8 lines
8 lines
8 lines
8 lines
33 lines
2 lines
82 lines
6 lines
35 lines
6 lines
6 lines
6 lines
162 lines
8 lines
30 lines
45 lines
78 lines
3720 lines
3720 lines
3720 lines
3662 lines
3472 lines
5479 lines
5584 lines
5143 lines
5584 lines
5584 lines
128 lines
128 lines
128 lines
12 lines
128 lines
128 lines
65 lines
3 lines
78 lines
133 lines
3 lines
158 lines
78 lines
5 lines
4 lines
17 lines
77 lines
146 lines
10 lines
96 lines
116 lines
5 lines
50 lines
33 lines
71 lines
115 lines
62 lines
7 lines
17 lines

Diff 448763

llvm/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp

Show First 20 Lines • Show All 68 Lines • ▼ Show 20 Lines
#define DEBUG_TYPE "si-load-store-opt" #define DEBUG_TYPE "si-load-store-opt"
namespace { namespace {
enum InstClassEnum { enum InstClassEnum {
UNKNOWN, UNKNOWN,
DS_READ, DS_READ,
DS_WRITE, DS_WRITE,
S_BUFFER_LOAD_IMM, S_BUFFER_LOAD_IMM,
S_LOAD_IMM,
BUFFER_LOAD, BUFFER_LOAD,
BUFFER_STORE, BUFFER_STORE,
MIMG, MIMG,
TBUFFER_LOAD, TBUFFER_LOAD,
TBUFFER_STORE, TBUFFER_STORE,
GLOBAL_LOAD_SADDR, GLOBAL_LOAD_SADDR,
GLOBAL_STORE_SADDR, GLOBAL_STORE_SADDR,
FLAT_LOAD, FLAT_LOAD,
▲ Show 20 Lines • Show All 142 Lines • ▼ Show 20 Linesprivate:
unsigned write2ST64Opcode(unsigned EltSize) const; unsigned write2ST64Opcode(unsigned EltSize) const;
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeWrite2Pair(CombineInfo &CI, CombineInfo &Paired, mergeWrite2Pair(CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore); MachineBasicBlock::iterator InsertBefore);
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeImagePair(CombineInfo &CI, CombineInfo &Paired, mergeImagePair(CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore); MachineBasicBlock::iterator InsertBefore);
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeSBufferLoadImmPair(CombineInfo &CI, CombineInfo &Paired, mergeSMemLoadImmPair(CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore); MachineBasicBlock::iterator InsertBefore);
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeBufferLoadPair(CombineInfo &CI, CombineInfo &Paired, mergeBufferLoadPair(CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore); MachineBasicBlock::iterator InsertBefore);
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeBufferStorePair(CombineInfo &CI, CombineInfo &Paired, mergeBufferStorePair(CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore); MachineBasicBlock::iterator InsertBefore);
MachineBasicBlock::iterator MachineBasicBlock::iterator
mergeTBufferLoadPair(CombineInfo &CI, CombineInfo &Paired, mergeTBufferLoadPair(CombineInfo &CI, CombineInfo &Paired,
▲ Show 20 Lines • Show All 75 Lines • ▼ Show 20 Lines if (TII.isMIMG(MI)) {
return countPopulation(DMaskImm); return countPopulation(DMaskImm);
} }
if (TII.isMTBUF(Opc)) { if (TII.isMTBUF(Opc)) {
return AMDGPU::getMTBUFElements(Opc); return AMDGPU::getMTBUFElements(Opc);
} }
switch (Opc) { switch (Opc) {
case AMDGPU::S_BUFFER_LOAD_DWORD_IMM: case AMDGPU::S_BUFFER_LOAD_DWORD_IMM:
case AMDGPU::S_LOAD_DWORD_IMM:
case AMDGPU::GLOBAL_LOAD_DWORD: case AMDGPU::GLOBAL_LOAD_DWORD:
case AMDGPU::GLOBAL_LOAD_DWORD_SADDR: case AMDGPU::GLOBAL_LOAD_DWORD_SADDR:
case AMDGPU::GLOBAL_STORE_DWORD: case AMDGPU::GLOBAL_STORE_DWORD:
case AMDGPU::GLOBAL_STORE_DWORD_SADDR: case AMDGPU::GLOBAL_STORE_DWORD_SADDR:
case AMDGPU::FLAT_LOAD_DWORD: case AMDGPU::FLAT_LOAD_DWORD:
case AMDGPU::FLAT_STORE_DWORD: case AMDGPU::FLAT_STORE_DWORD:
return 1; return 1;
case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM:
case AMDGPU::S_LOAD_DWORDX2_IMM:
case AMDGPU::GLOBAL_LOAD_DWORDX2: case AMDGPU::GLOBAL_LOAD_DWORDX2:
case AMDGPU::GLOBAL_LOAD_DWORDX2_SADDR: case AMDGPU::GLOBAL_LOAD_DWORDX2_SADDR:
case AMDGPU::GLOBAL_STORE_DWORDX2: case AMDGPU::GLOBAL_STORE_DWORDX2:
case AMDGPU::GLOBAL_STORE_DWORDX2_SADDR: case AMDGPU::GLOBAL_STORE_DWORDX2_SADDR:
case AMDGPU::FLAT_LOAD_DWORDX2: case AMDGPU::FLAT_LOAD_DWORDX2:
case AMDGPU::FLAT_STORE_DWORDX2: case AMDGPU::FLAT_STORE_DWORDX2:
return 2; return 2;
case AMDGPU::GLOBAL_LOAD_DWORDX3: case AMDGPU::GLOBAL_LOAD_DWORDX3:
case AMDGPU::GLOBAL_LOAD_DWORDX3_SADDR: case AMDGPU::GLOBAL_LOAD_DWORDX3_SADDR:
case AMDGPU::GLOBAL_STORE_DWORDX3: case AMDGPU::GLOBAL_STORE_DWORDX3:
case AMDGPU::GLOBAL_STORE_DWORDX3_SADDR: case AMDGPU::GLOBAL_STORE_DWORDX3_SADDR:
case AMDGPU::FLAT_LOAD_DWORDX3: case AMDGPU::FLAT_LOAD_DWORDX3:
case AMDGPU::FLAT_STORE_DWORDX3: case AMDGPU::FLAT_STORE_DWORDX3:
return 3; return 3;
case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM:
case AMDGPU::S_LOAD_DWORDX4_IMM:
case AMDGPU::GLOBAL_LOAD_DWORDX4: case AMDGPU::GLOBAL_LOAD_DWORDX4:
case AMDGPU::GLOBAL_LOAD_DWORDX4_SADDR: case AMDGPU::GLOBAL_LOAD_DWORDX4_SADDR:
case AMDGPU::GLOBAL_STORE_DWORDX4: case AMDGPU::GLOBAL_STORE_DWORDX4:
case AMDGPU::GLOBAL_STORE_DWORDX4_SADDR: case AMDGPU::GLOBAL_STORE_DWORDX4_SADDR:
case AMDGPU::FLAT_LOAD_DWORDX4: case AMDGPU::FLAT_LOAD_DWORDX4:
case AMDGPU::FLAT_STORE_DWORDX4: case AMDGPU::FLAT_STORE_DWORDX4:
return 4; return 4;
case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM:
case AMDGPU::S_LOAD_DWORDX8_IMM:
return 8; return 8;
case AMDGPU::DS_READ_B32: LLVM_FALLTHROUGH; case AMDGPU::DS_READ_B32: LLVM_FALLTHROUGH;
case AMDGPU::DS_READ_B32_gfx9: LLVM_FALLTHROUGH; case AMDGPU::DS_READ_B32_gfx9: LLVM_FALLTHROUGH;
case AMDGPU::DS_WRITE_B32: LLVM_FALLTHROUGH; case AMDGPU::DS_WRITE_B32: LLVM_FALLTHROUGH;
case AMDGPU::DS_WRITE_B32_gfx9: case AMDGPU::DS_WRITE_B32_gfx9:
return 1; return 1;
case AMDGPU::DS_READ_B64: LLVM_FALLTHROUGH; case AMDGPU::DS_READ_B64: LLVM_FALLTHROUGH;
case AMDGPU::DS_READ_B64_gfx9: LLVM_FALLTHROUGH; case AMDGPU::DS_READ_B64_gfx9: LLVM_FALLTHROUGH;
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Lines if (TII.isMTBUF(Opc)) {
} }
} }
return UNKNOWN; return UNKNOWN;
case AMDGPU::S_BUFFER_LOAD_DWORD_IMM: case AMDGPU::S_BUFFER_LOAD_DWORD_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM:
return S_BUFFER_LOAD_IMM; return S_BUFFER_LOAD_IMM;
case AMDGPU::S_LOAD_DWORD_IMM:
case AMDGPU::S_LOAD_DWORDX2_IMM:
case AMDGPU::S_LOAD_DWORDX4_IMM:
case AMDGPU::S_LOAD_DWORDX8_IMM:
return S_LOAD_IMM;
case AMDGPU::DS_READ_B32: case AMDGPU::DS_READ_B32:
case AMDGPU::DS_READ_B32_gfx9: case AMDGPU::DS_READ_B32_gfx9:
case AMDGPU::DS_READ_B64: case AMDGPU::DS_READ_B64:
case AMDGPU::DS_READ_B64_gfx9: case AMDGPU::DS_READ_B64_gfx9:
return DS_READ; return DS_READ;
case AMDGPU::DS_WRITE_B32: case AMDGPU::DS_WRITE_B32:
case AMDGPU::DS_WRITE_B32_gfx9: case AMDGPU::DS_WRITE_B32_gfx9:
case AMDGPU::DS_WRITE_B64: case AMDGPU::DS_WRITE_B64:
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Linesstatic unsigned getInstSubclass(unsigned Opc, const SIInstrInfo &TII) {
case AMDGPU::DS_WRITE_B64: case AMDGPU::DS_WRITE_B64:
case AMDGPU::DS_WRITE_B64_gfx9: case AMDGPU::DS_WRITE_B64_gfx9:
return Opc; return Opc;
case AMDGPU::S_BUFFER_LOAD_DWORD_IMM: case AMDGPU::S_BUFFER_LOAD_DWORD_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM:
return AMDGPU::S_BUFFER_LOAD_DWORD_IMM; return AMDGPU::S_BUFFER_LOAD_DWORD_IMM;
case AMDGPU::S_LOAD_DWORD_IMM:
case AMDGPU::S_LOAD_DWORDX2_IMM:
case AMDGPU::S_LOAD_DWORDX4_IMM:
case AMDGPU::S_LOAD_DWORDX8_IMM:
return AMDGPU::S_LOAD_DWORD_IMM;
case AMDGPU::GLOBAL_LOAD_DWORD: case AMDGPU::GLOBAL_LOAD_DWORD:
case AMDGPU::GLOBAL_LOAD_DWORDX2: case AMDGPU::GLOBAL_LOAD_DWORDX2:
case AMDGPU::GLOBAL_LOAD_DWORDX3: case AMDGPU::GLOBAL_LOAD_DWORDX3:
case AMDGPU::GLOBAL_LOAD_DWORDX4: case AMDGPU::GLOBAL_LOAD_DWORDX4:
case AMDGPU::FLAT_LOAD_DWORD: case AMDGPU::FLAT_LOAD_DWORD:
case AMDGPU::FLAT_LOAD_DWORDX2: case AMDGPU::FLAT_LOAD_DWORDX2:
case AMDGPU::FLAT_LOAD_DWORDX3: case AMDGPU::FLAT_LOAD_DWORDX3:
case AMDGPU::FLAT_LOAD_DWORDX4: case AMDGPU::FLAT_LOAD_DWORDX4:
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Linesstatic AddressRegs getRegs(unsigned Opc, const SIInstrInfo &TII) {
switch (Opc) { switch (Opc) {
default: default:
return Result; return Result;
case AMDGPU::S_BUFFER_LOAD_DWORD_IMM: case AMDGPU::S_BUFFER_LOAD_DWORD_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM:
case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM: case AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM:
case AMDGPU::S_LOAD_DWORD_IMM:
case AMDGPU::S_LOAD_DWORDX2_IMM:
case AMDGPU::S_LOAD_DWORDX4_IMM:
case AMDGPU::S_LOAD_DWORDX8_IMM:
Result.SBase = true; Result.SBase = true;
return Result; return Result;
case AMDGPU::DS_READ_B32: case AMDGPU::DS_READ_B32:
case AMDGPU::DS_READ_B64: case AMDGPU::DS_READ_B64:
case AMDGPU::DS_READ_B32_gfx9: case AMDGPU::DS_READ_B32_gfx9:
case AMDGPU::DS_READ_B64_gfx9: case AMDGPU::DS_READ_B64_gfx9:
case AMDGPU::DS_WRITE_B32: case AMDGPU::DS_WRITE_B32:
case AMDGPU::DS_WRITE_B64: case AMDGPU::DS_WRITE_B64:
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Lines EltSize =
: 4; : 4;
break; break;
case DS_WRITE: case DS_WRITE:
EltSize = EltSize =
(Opc == AMDGPU::DS_WRITE_B64 || Opc == AMDGPU::DS_WRITE_B64_gfx9) ? 8 (Opc == AMDGPU::DS_WRITE_B64 || Opc == AMDGPU::DS_WRITE_B64_gfx9) ? 8
: 4; : 4;
break; break;
case S_BUFFER_LOAD_IMM: case S_BUFFER_LOAD_IMM:
case S_LOAD_IMM:
EltSize = AMDGPU::convertSMRDOffsetUnits(*LSO.STM, 4); EltSize = AMDGPU::convertSMRDOffsetUnits(*LSO.STM, 4);
break; break;
default: default:
EltSize = 4; EltSize = 4;
break; break;
} }
if (InstClass == MIMG) { if (InstClass == MIMG) {
▲ Show 20 Lines • Show All 304 Lines • ▼ Show 20 Lines
bool SILoadStoreOptimizer::widthsFit(const GCNSubtarget &STM, bool SILoadStoreOptimizer::widthsFit(const GCNSubtarget &STM,
const CombineInfo &CI, const CombineInfo &CI,
const CombineInfo &Paired) { const CombineInfo &Paired) {
const unsigned Width = (CI.Width + Paired.Width); const unsigned Width = (CI.Width + Paired.Width);
switch (CI.InstClass) { switch (CI.InstClass) {
default: default:
return (Width <= 4) && (STM.hasDwordx3LoadStores() || (Width != 3)); return (Width <= 4) && (STM.hasDwordx3LoadStores() || (Width != 3));
case S_BUFFER_LOAD_IMM: case S_BUFFER_LOAD_IMM:
case S_LOAD_IMM:
switch (Width) { switch (Width) {
default: default:
return false; return false;
case 2: case 2:
case 4: case 4:
case 8: case 8:
return true; return true;
} }
▲ Show 20 Lines • Show All 296 Lines • ▼ Show 20 Lines BuildMI(*MBB, InsertBefore, DL, CopyDesc)
.add(*Dest1) .add(*Dest1)
.addReg(DestReg, RegState::Kill, SubRegIdx1); .addReg(DestReg, RegState::Kill, SubRegIdx1);
CI.I->eraseFromParent(); CI.I->eraseFromParent();
Paired.I->eraseFromParent(); Paired.I->eraseFromParent();
return New; return New;
} }
MachineBasicBlock::iterator SILoadStoreOptimizer::mergeSBufferLoadImmPair( MachineBasicBlock::iterator SILoadStoreOptimizer::mergeSMemLoadImmPair(
CombineInfo &CI, CombineInfo &Paired, CombineInfo &CI, CombineInfo &Paired,
MachineBasicBlock::iterator InsertBefore) { MachineBasicBlock::iterator InsertBefore) {
MachineBasicBlock *MBB = CI.I->getParent(); MachineBasicBlock *MBB = CI.I->getParent();
DebugLoc DL = CI.I->getDebugLoc(); DebugLoc DL = CI.I->getDebugLoc();
const unsigned Opcode = getNewOpcode(CI, Paired); const unsigned Opcode = getNewOpcode(CI, Paired);
const TargetRegisterClass *SuperRC = getTargetRegisterClass(CI, Paired); const TargetRegisterClass *SuperRC = getTargetRegisterClass(CI, Paired);
▲ Show 20 Lines • Show All 313 Lines • ▼ Show 20 Lines default:
return 0; return 0;
case 2: case 2:
return AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM; return AMDGPU::S_BUFFER_LOAD_DWORDX2_IMM;
case 4: case 4:
return AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM; return AMDGPU::S_BUFFER_LOAD_DWORDX4_IMM;
case 8: case 8:
return AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM; return AMDGPU::S_BUFFER_LOAD_DWORDX8_IMM;
} }
case S_LOAD_IMM:
switch (Width) {
default:
return 0;
case 2:
return AMDGPU::S_LOAD_DWORDX2_IMM;
case 4:
return AMDGPU::S_LOAD_DWORDX4_IMM;
case 8:
return AMDGPU::S_LOAD_DWORDX8_IMM;
}
case GLOBAL_LOAD: case GLOBAL_LOAD:
switch (Width) { switch (Width) {
default: default:
return 0; return 0;
case 2: case 2:
return AMDGPU::GLOBAL_LOAD_DWORDX2; return AMDGPU::GLOBAL_LOAD_DWORDX2;
case 3: case 3:
return AMDGPU::GLOBAL_LOAD_DWORDX3; return AMDGPU::GLOBAL_LOAD_DWORDX3;
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 LinesSILoadStoreOptimizer::getSubRegIdxs(const CombineInfo &CI,
} }
return std::make_pair(Idx0, Idx1); return std::make_pair(Idx0, Idx1);
} }
const TargetRegisterClass * const TargetRegisterClass *
SILoadStoreOptimizer::getTargetRegisterClass(const CombineInfo &CI, SILoadStoreOptimizer::getTargetRegisterClass(const CombineInfo &CI,
const CombineInfo &Paired) { const CombineInfo &Paired) {
if (CI.InstClass == S_BUFFER_LOAD_IMM) { if (CI.InstClass == S_BUFFER_LOAD_IMM || CI.InstClass == S_LOAD_IMM) {
switch (CI.Width + Paired.Width) { switch (CI.Width + Paired.Width) {
default: default:
return nullptr; return nullptr;
case 2: case 2:
return &AMDGPU::SReg_64_XEXECRegClass; return &AMDGPU::SReg_64_XEXECRegClass;
case 4: case 4:
return &AMDGPU::SGPR_128RegClass; return &AMDGPU::SGPR_128RegClass;
case 8: case 8:
▲ Show 20 Lines • Show All 552 Lines • ▼ Show 20 Lines default:
break; break;
case DS_READ: case DS_READ:
NewMI = mergeRead2Pair(CI, Paired, Where->I); NewMI = mergeRead2Pair(CI, Paired, Where->I);
break; break;
case DS_WRITE: case DS_WRITE:
NewMI = mergeWrite2Pair(CI, Paired, Where->I); NewMI = mergeWrite2Pair(CI, Paired, Where->I);
break; break;
case S_BUFFER_LOAD_IMM: case S_BUFFER_LOAD_IMM:
NewMI = mergeSBufferLoadImmPair(CI, Paired, Where->I); case S_LOAD_IMM:
NewMI = mergeSMemLoadImmPair(CI, Paired, Where->I);
foadUnsubmitted
Done
ReplyInline Actions

Maybe rename to something like mergeSMEMLoadImmPair?

foad: Maybe rename to something like `mergeSMEMLoadImmPair`?
OptimizeListAgain |= CI.Width + Paired.Width < 8; OptimizeListAgain |= CI.Width + Paired.Width < 8;
break; break;
case BUFFER_LOAD: case BUFFER_LOAD:
NewMI = mergeBufferLoadPair(CI, Paired, Where->I); NewMI = mergeBufferLoadPair(CI, Paired, Where->I);
OptimizeListAgain |= CI.Width + Paired.Width < 4; OptimizeListAgain |= CI.Width + Paired.Width < 4;
break; break;
case BUFFER_STORE: case BUFFER_STORE:
NewMI = mergeBufferStorePair(CI, Paired, Where->I); NewMI = mergeBufferStorePair(CI, Paired, Where->I);
▲ Show 20 Lines • Show All 87 Lines • Show Last 20 Lines

llvm/test/CodeGen/AMDGPU/GlobalISel/cvt_f32_ubyte.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/fp-atomics-gfx940.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/implicit-kernarg-backend-usage-global-isel.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.intersect_ray.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.sbfe.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.ubfe.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.update.dpp.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll

llvm/test/CodeGen/AMDGPU/GlobalISel/udivrem.ll

llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll

llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll

llvm/test/CodeGen/AMDGPU/always-uniform.ll

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

llvm/test/CodeGen/AMDGPU/amdgpu.private-memory.ll

llvm/test/CodeGen/AMDGPU/any_extend_vector_inreg.ll

llvm/test/CodeGen/AMDGPU/bfe-patterns.ll

llvm/test/CodeGen/AMDGPU/bfi_int.ll

llvm/test/CodeGen/AMDGPU/bfm.ll

llvm/test/CodeGen/AMDGPU/bitreverse.ll

llvm/test/CodeGen/AMDGPU/cluster_stores.ll

llvm/test/CodeGen/AMDGPU/combine-cond-add-sub.ll

llvm/test/CodeGen/AMDGPU/ctlz.ll

llvm/test/CodeGen/AMDGPU/ctlz_zero_undef.ll

llvm/test/CodeGen/AMDGPU/ctpop16.ll

llvm/test/CodeGen/AMDGPU/cttz.ll

llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll

llvm/test/CodeGen/AMDGPU/cvt_f32_ubyte.ll

llvm/test/CodeGen/AMDGPU/disable_form_clauses.ll

llvm/test/CodeGen/AMDGPU/divergence-driven-sext-inreg.ll

llvm/test/CodeGen/AMDGPU/ds_read2.ll

llvm/test/CodeGen/AMDGPU/extract_vector_elt-f16.ll

llvm/test/CodeGen/AMDGPU/extract_vector_elt-i16.ll

llvm/test/CodeGen/AMDGPU/fabs.f16.ll

llvm/test/CodeGen/AMDGPU/fabs.ll

llvm/test/CodeGen/AMDGPU/fcopysign.f32.ll

llvm/test/CodeGen/AMDGPU/fmin_legacy.ll

llvm/test/CodeGen/AMDGPU/fp-atomics-gfx940.ll

llvm/test/CodeGen/AMDGPU/fp-min-max-buffer-atomics.ll

llvm/test/CodeGen/AMDGPU/fp64-min-max-buffer-atomics.ll

llvm/test/CodeGen/AMDGPU/fp_to_sint.ll

llvm/test/CodeGen/AMDGPU/fp_to_uint.ll

llvm/test/CodeGen/AMDGPU/fshl.ll

llvm/test/CodeGen/AMDGPU/fshr.ll

llvm/test/CodeGen/AMDGPU/half.ll

llvm/test/CodeGen/AMDGPU/implicit-kernarg-backend-usage.ll

llvm/test/CodeGen/AMDGPU/insert_vector_dynelt.ll

llvm/test/CodeGen/AMDGPU/insert_vector_elt.ll

llvm/test/CodeGen/AMDGPU/kernel-args.ll

llvm/test/CodeGen/AMDGPU/kernel-argument-dag-lowering.ll

llvm/test/CodeGen/AMDGPU/lds-atomic-fmin-fmax.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.format.d16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.cvt.pk.i16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.cvt.pk.u16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.cvt.pknorm.i16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.cvt.pknorm.u16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.cvt.pkrtz.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.intersect_ray.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.store.format.d16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.sched.group.barrier.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.store.format.d16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.tbuffer.store.d16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.d16.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.ubfe.ll

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.update.dpp.ll

llvm/test/CodeGen/AMDGPU/llvm.r600.read.local.size.ll

llvm/test/CodeGen/AMDGPU/lshr.v2i16.ll

llvm/test/CodeGen/AMDGPU/mad_64_32.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-agent.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-singlethread.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-system.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-wavefront.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-workgroup.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-global-agent.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-global-singlethread.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-global-system.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-global-wavefront.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-global-workgroup.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-agent.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-singlethread.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-system.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-volatile.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-wavefront.ll

llvm/test/CodeGen/AMDGPU/memory-legalizer-local-workgroup.ll

llvm/test/CodeGen/AMDGPU/memory_clause.ll

llvm/test/CodeGen/AMDGPU/min.ll

llvm/test/CodeGen/AMDGPU/mul_int24.ll

llvm/test/CodeGen/AMDGPU/mul_uint24-amdgcn.ll

llvm/test/CodeGen/AMDGPU/s_addk_i32.ll

llvm/test/CodeGen/AMDGPU/scalar_to_vector.v8i16.ll

llvm/test/CodeGen/AMDGPU/sdiv64.ll

llvm/test/CodeGen/AMDGPU/select-constant-cttz.ll

llvm/test/CodeGen/AMDGPU/select-vectors.ll

llvm/test/CodeGen/AMDGPU/sext-divergence-driven-isel.ll

llvm/test/CodeGen/AMDGPU/shl.v2i16.ll

llvm/test/CodeGen/AMDGPU/sign_extend.ll

llvm/test/CodeGen/AMDGPU/sminmax.v2i16.ll

llvm/test/CodeGen/AMDGPU/srem64.ll

llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll

llvm/test/CodeGen/AMDGPU/sub.ll

llvm/test/CodeGen/AMDGPU/sub.v2i16.ll

llvm/test/CodeGen/AMDGPU/subreg-coalescer-undef-use.ll

llvm/test/CodeGen/AMDGPU/udiv.ll

llvm/test/CodeGen/AMDGPU/udiv64.ll

llvm/test/CodeGen/AMDGPU/use-sgpr-multiple-times.ll

llvm/test/CodeGen/AMDGPU/wait.ll

llvm/test/CodeGen/AMDGPU/zext-divergence-driven-isel.ll