AMDGPU/SI: Don't reserve FLAT_SCRATCH on non-HSA targets

[mareko]

Summary

This frees 2 scalar registers.

[• tstellarAMD]

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

397–398 ↗

(On Diff #79356)

There should be a query in SIMachineFunctionInfo to check if flat scratch will be used, because it's not even always required for hsa depending on the program.

[mareko]

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

397–398 ↗

(On Diff #79356)

How to know if flat scratch will be used by HSA? Or can that be done in a later patch?

[• tstellarAMD]

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

397–398 ↗

(On Diff #79356)

I think the query should be added for this patch, but it can just return ST.hasHsaOS(). The smarter logic can be added in a later patch.

[mareko]

Update: Add SIMachineFunctionInfo::flatUsesScratch()

[arsenm]

hasStackObjects is a good enough approximation for now. Ideally it would be enabled only if it has an alloca that is addrspacecasted or has a captured private pointer. hasStackObjects would ideally only be alloca derived stack objects, not anything inserted by lowering (which should exist by the time reserved registers are frozen)

[mareko]

In D27150#607117, @arsenm wrote:

hasStackObjects is a good enough approximation for now. Ideally it would be enabled only if it has an alloca that is addrspacecasted or has a captured private pointer. hasStackObjects would ideally only be alloca derived stack objects, not anything inserted by lowering (which should exist by the time reserved registers are frozen)

hasStackObjects doesn't say that scratch is accessed by flat instructions. The user may still want to use a buffer descriptor. Also, I wouldn't like to change the behavior for HSA in this patch (i.e. depend on hasStackObjects), because that needs lit tests adjustments and maybe even more tests.

[arsenm]

In D27150#607579, @mareko wrote:

In D27150#607117, @arsenm wrote:

hasStackObjects is a good enough approximation for now. Ideally it would be enabled only if it has an alloca that is addrspacecasted or has a captured private pointer. hasStackObjects would ideally only be alloca derived stack objects, not anything inserted by lowering (which should exist by the time reserved registers are frozen)

hasStackObjects doesn't say that scratch is accessed by flat instructions. The user may still want to use a buffer descriptor. Also, I wouldn't like to change the behavior for HSA in this patch (i.e. depend on hasStackObjects), because that needs lit tests adjustments and maybe even more tests.

That's why it's an approximation. If there are no stack objects, then obviously none are accessed through flat instructions. I don't think accessing the stack through a user buffer descriptor should be supported

[mareko]

Update: Use FlatScratchInit, also reserve FLAT_SCRATCH on CI.

[• tstellarAMD]

LGTM.

llvm/trunk/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

for (const MachineInstr &MI : MBB) {
         case AMDGPU::VCC_LO:
         case AMDGPU::VCC_HI:
           VCCUsed = true;
           continue;
 
         case AMDGPU::FLAT_SCR:
         case AMDGPU::FLAT_SCR_LO:
         case AMDGPU::FLAT_SCR_HI:
+          // Even if FLAT_SCRATCH is implicitly used, it has no effect if flat
+          // instructions aren't used to access the scratch buffer.
+          if (MFI->hasFlatScratchInit())
             FlatUsed = true;
           continue;
 
         case AMDGPU::TBA:
         case AMDGPU::TBA_LO:
         case AMDGPU::TBA_HI:
         case AMDGPU::TMA:
         case AMDGPU::TMA_LO:
         case AMDGPU::TMA_HI:

llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h

 
 #include "AMDGPURegisterInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 
 namespace llvm {
 
 class SISubtarget;
 class MachineRegisterInfo;
+class SIMachineFunctionInfo;
 
 class SIRegisterInfo final : public AMDGPURegisterInfo {
 private:
   unsigned SGPRSetID;
   unsigned VGPRSetID;
   BitVector SGPRPressureSets;
   BitVector VGPRPressureSets;
 
public:
 
   /// \returns Total number of SGPRs supported by the subtarget.
   unsigned getTotalNumSGPRs(const SISubtarget &ST) const;
 
   /// \returns Number of addressable SGPRs supported by the subtarget.
   unsigned getNumAddressableSGPRs(const SISubtarget &ST) const;
 
   /// \returns Number of reserved SGPRs supported by the subtarget.
-  unsigned getNumReservedSGPRs(const SISubtarget &ST) const;
+  unsigned getNumReservedSGPRs(const SISubtarget &ST,
+                               const SIMachineFunctionInfo &MFI) const;
 
   /// \returns Minimum number of SGPRs that meets given number of waves per
   /// execution unit requirement for given subtarget.
   unsigned getMinNumSGPRs(const SISubtarget &ST, unsigned WavesPerEU) const;
 
   /// \returns Maximum number of SGPRs that meets given number of waves per
   /// execution unit requirement for given subtarget.
   unsigned getMaxNumSGPRs(const SISubtarget &ST, unsigned WavesPerEU,

llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.cpp

 }
 
 unsigned SIRegisterInfo::getNumAddressableSGPRs(const SISubtarget &ST) const {
   if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS)
     return 102;
   return 104;
 }
 
-unsigned SIRegisterInfo::getNumReservedSGPRs(const SISubtarget &ST) const {
+unsigned SIRegisterInfo::getNumReservedSGPRs(const SISubtarget &ST,
+                                             const SIMachineFunctionInfo &MFI) const {
+  if (MFI.hasFlatScratchInit()) {
     if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS)
-    return 6; // VCC, FLAT_SCRATCH, XNACK.
-  if (ST.getGeneration() >= AMDGPUSubtarget::SEA_ISLANDS)
-    return 4; // VCC, FLAT_SCRATCH.
+      return 6; // FLAT_SCRATCH, XNACK, VCC (in that order)
+
+    if (ST.getGeneration() == AMDGPUSubtarget::SEA_ISLANDS)
+      return 4; // FLAT_SCRATCH, VCC (in that order)
+  }
+
+  if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS)
+    return 4; // XNACK, VCC (in that order)
+
   return 2; // VCC.
 }
 
 unsigned SIRegisterInfo::getMinNumSGPRs(const SISubtarget &ST,
                                         unsigned WavesPerEU) const {
   if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS) {
     switch (WavesPerEU) {
       case 0:  return 0;
unsigned SIRegisterInfo::getMaxNumSGPRs(const MachineFunction &MF) const {
 
   // Check if maximum number of SGPRs was explicitly requested using
   // "amdgpu-num-sgpr" attribute.
   if (F.hasFnAttribute("amdgpu-num-sgpr")) {
     unsigned Requested = AMDGPU::getIntegerAttribute(
       F, "amdgpu-num-sgpr", MaxNumSGPRs);
 
     // Make sure requested value does not violate subtarget's specifications.
-    if (Requested && (Requested <= getNumReservedSGPRs(ST)))
+    if (Requested && (Requested <= getNumReservedSGPRs(ST, MFI)))
       Requested = 0;
 
     // If more SGPRs are required to support the input user/system SGPRs,
     // increase to accommodate them.
     //
     // FIXME: This really ends up using the requested number of SGPRs + number
     // of reserved special registers in total. Theoretically you could re-use
     // the last input registers for these special registers, but this would
if (F.hasFnAttribute("amdgpu-num-sgpr")) {
 
     if (Requested)
       MaxNumSGPRs = Requested;
   }
 
   if (ST.hasSGPRInitBug())
     MaxNumSGPRs = SISubtarget::FIXED_SGPR_COUNT_FOR_INIT_BUG;
 
-  return std::min(MaxNumSGPRs - getNumReservedSGPRs(ST), MaxNumAddressableSGPRs);
+  return std::min(MaxNumSGPRs - getNumReservedSGPRs(ST, MFI),
+                  MaxNumAddressableSGPRs);
 }
 
 unsigned SIRegisterInfo::getNumDebuggerReservedVGPRs(
   const SISubtarget &ST) const {
   if (ST.debuggerReserveRegs())
     return 4;
   return 0;
 }

llvm/trunk/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size.ll

 ; CHECK: NumVGPRsForWavesPerEU: 1
 define void @min_128_max_128() #2 {
 entry:
   ret void
 }
 attributes #2 = {"amdgpu-flat-work-group-size"="128,128"}
 
 ; CHECK-LABEL: {{^}}min_1024_max_2048
-; CHECK: SGPRBlocks: 2
+; CHECK: SGPRBlocks: 1
 ; CHECK: VGPRBlocks: 7
-; CHECK: NumSGPRsForWavesPerEU: 19
+; CHECK: NumSGPRsForWavesPerEU: 13
 ; CHECK: NumVGPRsForWavesPerEU: 32
 @var = addrspace(1) global float 0.0
 define void @min_1024_max_2048() #3 {
   %val0 = load volatile float, float addrspace(1)* @var
   %val1 = load volatile float, float addrspace(1)* @var
   %val2 = load volatile float, float addrspace(1)* @var
   %val3 = load volatile float, float addrspace(1)* @var
   %val4 = load volatile float, float addrspace(1)* @var

llvm/trunk/test/CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll

 ; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -amdgpu-spill-sgpr-to-smem=0 -verify-machineinstrs < %s | FileCheck -check-prefix=TOSGPR -check-prefix=ALL %s
 ; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -amdgpu-spill-sgpr-to-smem=1 -verify-machineinstrs < %s | FileCheck -check-prefix=TOSMEM -check-prefix=ALL %s
 
 ; If spilling to smem, additional registers are used for the resource
 ; descriptor.
 
-; ALL-LABEL: {{^}}max_14_sgprs:
+; ALL-LABEL: {{^}}max_12_sgprs:
 
 ; FIXME: Should be ablo to skip this copying of the private segment
 ; buffer because all the SGPR spills are to VGPRs.
 
 ; ALL: s_mov_b64 s[6:7], s[2:3]
 ; ALL: s_mov_b64 s[4:5], s[0:1]
 ; ALL: SGPRBlocks: 1
-; ALL: NumSGPRsForWavesPerEU: 14
-define void @max_14_sgprs(i32 addrspace(1)* %out1,
+; ALL: NumSGPRsForWavesPerEU: 12
+define void @max_12_sgprs(i32 addrspace(1)* %out1,
 
                           i32 addrspace(1)* %out2,
                           i32 addrspace(1)* %out3,
                           i32 addrspace(1)* %out4,
                           i32 %one, i32 %two, i32 %three, i32 %four) #0 {
   store i32 %one, i32 addrspace(1)* %out1
   store i32 %two, i32 addrspace(1)* %out2
   store i32 %three, i32 addrspace(1)* %out3
   store i32 %four, i32 addrspace(1)* %out4
   ret void
 }
 
 ; private resource: 4
 ; scratch wave offset: 1
 ; workgroup ids: 3
 ; dispatch id: 2
 ; queue ptr: 2
 ; flat scratch init: 2
 ; ---------------------
 ; total: 14
 
-; + reserved vcc, xnack, flat_scratch = 20
+; + reserved vcc = 16
 
 ; Because we can't handle re-using the last few input registers as the
 ; special vcc etc. registers (as well as decide to not use the unused
 ; features when the number of registers is frozen), this ends up using
 ; more than expected.
 
 ; ALL-LABEL: {{^}}max_12_sgprs_14_input_sgprs:
-; TOSGPR: SGPRBlocks: 2
-; TOSGPR: NumSGPRsForWavesPerEU: 20
+; TOSGPR: SGPRBlocks: 1
+; TOSGPR: NumSGPRsForWavesPerEU: 16
 
 ; TOSMEM: s_mov_b64 s[6:7], s[2:3]
+; TOSMEM: s_mov_b32 s9, s13
 ; TOSMEM: s_mov_b64 s[4:5], s[0:1]
-; TOSMEM: s_mov_b32 s3, s13
 
-; TOSMEM: SGPRBlocks: 2
-; TOSMEM: NumSGPRsForWavesPerEU: 20
+; TOSMEM: SGPRBlocks: 1
+; TOSMEM: NumSGPRsForWavesPerEU: 16
 define void @max_12_sgprs_14_input_sgprs(i32 addrspace(1)* %out1,
                                         i32 addrspace(1)* %out2,
                                         i32 addrspace(1)* %out3,
                                         i32 addrspace(1)* %out4,
                                         i32 %one, i32 %two, i32 %three, i32 %four) #2 {
   store volatile i32 0, i32* undef
   %x.0 = call i32 @llvm.amdgcn.workgroup.id.x()
   store volatile i32 %x.0, i32 addrspace(1)* undef

llvm/trunk/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll

entry:
   ret void
 }
 attributes #8 = {"amdgpu-waves-per-eu"="5,10"}
 
 @var = addrspace(1) global float 0.0
 
 ; Exactly 10 waves per execution unit.
 ; CHECK-LABEL: {{^}}exactly_10:
-; CHECK: SGPRBlocks: 2
+; CHECK: SGPRBlocks: 1
 ; CHECK: VGPRBlocks: 5
-; CHECK: NumSGPRsForWavesPerEU: 19
+; CHECK: NumSGPRsForWavesPerEU: 13
 ; CHECK: NumVGPRsForWavesPerEU: 24
 define void @exactly_10() #9 {
   %val0 = load volatile float, float addrspace(1)* @var
   %val1 = load volatile float, float addrspace(1)* @var
   %val2 = load volatile float, float addrspace(1)* @var
   %val3 = load volatile float, float addrspace(1)* @var
   %val4 = load volatile float, float addrspace(1)* @var
   %val5 = load volatile float, float addrspace(1)* @var

llvm/trunk/test/CodeGen/AMDGPU/exceed-max-sgprs.ll

define void @use_too_many_sgprs_bonaire() #1 {
   call void asm sideeffect "", "~{SGPR72_SGPR73_SGPR74_SGPR75_SGPR76_SGPR77_SGPR78_SGPR79}" ()
   call void asm sideeffect "", "~{SGPR80_SGPR81_SGPR82_SGPR83_SGPR84_SGPR85_SGPR86_SGPR87}" ()
   call void asm sideeffect "", "~{SGPR88_SGPR89_SGPR90_SGPR91_SGPR92_SGPR93_SGPR94_SGPR95}" ()
   call void asm sideeffect "", "~{SGPR96_SGPR97_SGPR98_SGPR99_SGPR100_SGPR101_SGPR102_SGPR103}" ()
   call void asm sideeffect "", "~{VCC}" ()
   ret void
 }
 
-; ERROR: error: scalar registers limit of 104 exceeded (108) in use_too_many_sgprs_bonaire_flat_scr
+; ERROR: error: scalar registers limit of 104 exceeded (106) in use_too_many_sgprs_bonaire_flat_scr
 define void @use_too_many_sgprs_bonaire_flat_scr() #1 {
   call void asm sideeffect "", "~{SGPR0_SGPR1_SGPR2_SGPR3_SGPR4_SGPR5_SGPR6_SGPR7}" ()
   call void asm sideeffect "", "~{SGPR8_SGPR9_SGPR10_SGPR11_SGPR12_SGPR13_SGPR14_SGPR15}" ()
   call void asm sideeffect "", "~{SGPR16_SGPR17_SGPR18_SGPR19_SGPR20_SGPR21_SGPR22_SGPR23}" ()
   call void asm sideeffect "", "~{SGPR24_SGPR25_SGPR26_SGPR27_SGPR28_SGPR29_SGPR30_SGPR31}" ()
   call void asm sideeffect "", "~{SGPR32_SGPR33_SGPR34_SGPR35_SGPR36_SGPR37_SGPR38_SGPR39}" ()
   call void asm sideeffect "", "~{SGPR40_SGPR41_SGPR42_SGPR43_SGPR44_SGPR45_SGPR46_SGPR47}" ()
   call void asm sideeffect "", "~{SGPR48_SGPR49_SGPR50_SGPR51_SGPR52_SGPR53_SGPR54_SGPR55}" ()

llvm/trunk/test/CodeGen/AMDGPU/flat-scratch-reg.ll

-; RUN: llc -march=amdgcn -mcpu=kaveri -verify-machineinstrs < %s | FileCheck -check-prefix=NOXNACK -check-prefix=CI -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefix=NOXNACK -check-prefix=VI -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mcpu=carrizo -verify-machineinstrs < %s | FileCheck -check-prefix=XNACK -check-prefix=VI  -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mcpu=stoney -verify-machineinstrs < %s | FileCheck -check-prefix=XNACK -check-prefix=VI  -check-prefix=GCN %s
-
-; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=kaveri -verify-machineinstrs < %s | FileCheck -check-prefix=CI -check-prefix=NOXNACK -check-prefix=HSA-NOXNACK -check-prefix=HSA -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=-xnack -verify-machineinstrs < %s | FileCheck -check-prefix=VI -check-prefix=NOXNACK -check-prefix=HSA-NOXNACK -check-prefix=HSA -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=+xnack -verify-machineinstrs < %s | FileCheck -check-prefix=VI -check-prefix=XNACK -check-prefix=HSA-XNACK -check-prefix=HSA -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=kaveri -verify-machineinstrs < %s | FileCheck -check-prefix=CI -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefix=VI-NOXNACK -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=carrizo -verify-machineinstrs < %s | FileCheck -check-prefix=VI-XNACK  -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=stoney -verify-machineinstrs < %s | FileCheck -check-prefix=VI-XNACK  -check-prefix=GCN %s
+
+; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=kaveri -verify-machineinstrs < %s | FileCheck -check-prefix=HSA-CI -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=-xnack -verify-machineinstrs < %s | FileCheck -check-prefix=HSA-VI-NOXNACK -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=+xnack -verify-machineinstrs < %s | FileCheck -check-prefix=HSA-VI-XNACK -check-prefix=GCN %s
 
 ; GCN-LABEL: {{^}}no_vcc_no_flat:
-; HSA-NOXNACK: is_xnack_enabled = 0
-; HSA-XNACK: is_xnack_enabled = 1
-
-; NOXNACK: ; NumSgprs: 8
-; XNACK: ; NumSgprs: 12
+; HSA-CI: is_xnack_enabled = 0
+; HSA-VI-NOXNACK: is_xnack_enabled = 0
+; HSA-VI-XNACK: is_xnack_enabled = 1
+
+; CI: ; NumSgprs: 8
+; VI-NOXNACK: ; NumSgprs: 8
+; VI-XNACK: ; NumSgprs: 12
 define void @no_vcc_no_flat() {
 entry:
   call void asm sideeffect "", "~{SGPR7}"()
   ret void
 }
 
 ; GCN-LABEL: {{^}}vcc_no_flat:
-; HSA-NOXNACK: is_xnack_enabled = 0
-; HSA-XNACK: is_xnack_enabled = 1
-
-; NOXNACK: ; NumSgprs: 10
-; XNACK: ; NumSgprs: 12
+; HSA-CI: is_xnack_enabled = 0
+; HSA-VI-NOXNACK: is_xnack_enabled = 0
+; HSA-VI-XNACK: is_xnack_enabled = 1
+
+; CI: ; NumSgprs: 10
+; VI-NOXNACK: ; NumSgprs: 10
+; VI-XNACK: ; NumSgprs: 12
 define void @vcc_no_flat() {
 entry:
   call void asm sideeffect "", "~{SGPR7},~{VCC}"()
   ret void
 }
 
 ; GCN-LABEL: {{^}}no_vcc_flat:
-; HSA-NOXNACK: is_xnack_enabled = 0
-; HSA-XNACK: is_xnack_enabled = 1
-
-; CI: ; NumSgprs: 12
-; VI: ; NumSgprs: 14
+; HSA-CI: is_xnack_enabled = 0
+; HSA-VI-NOXNACK: is_xnack_enabled = 0
+; HSA-VI-XNACK: is_xnack_enabled = 1
+
+; CI: ; NumSgprs: 8
+; VI-NOXNACK: ; NumSgprs: 8
+; VI-XNACK: ; NumSgprs: 12
+; HSA-CI: ; NumSgprs: 8
+; HSA-VI-NOXNACK: ; NumSgprs: 8
+; HSA-VI-XNACK: ; NumSgprs: 12
 define void @no_vcc_flat() {
 entry:
   call void asm sideeffect "", "~{SGPR7},~{FLAT_SCR}"()
   ret void
 }
 
 ; GCN-LABEL: {{^}}vcc_flat:
 ; HSA-NOXNACK: is_xnack_enabled = 0
 ; HSA-XNACK: is_xnack_enabled = 1
 
-; CI: ; NumSgprs: 12
-; VI: ; NumSgprs: 14
+; CI: ; NumSgprs: 10
+; VI-NOXNACK: ; NumSgprs: 10
+; VI-XNACK: ; NumSgprs: 12
+; HSA-CI: ; NumSgprs: 10
+; HSA-VI-NOXNACK: ; NumSgprs: 10
+; HSA-VI-XNACK: ; NumSgprs: 12
 define void @vcc_flat() {
 entry:
   call void asm sideeffect "", "~{SGPR7},~{VCC},~{FLAT_SCR}"()
   ret void
 }

llvm/trunk/test/CodeGen/AMDGPU/spill-m0.ll

 ; RUN: llc -O0 -amdgpu-spill-sgpr-to-vgpr=1 -march=amdgcn -mattr=+vgpr-spilling -verify-machineinstrs < %s | FileCheck -check-prefix=TOVGPR -check-prefix=GCN %s
 ; RUN: llc -O0 -amdgpu-spill-sgpr-to-vgpr=1 -amdgpu-spill-sgpr-to-smem=0 -march=amdgcn -mcpu=tonga -mattr=+vgpr-spilling  -verify-machineinstrs < %s | FileCheck -check-prefix=TOVGPR -check-prefix=GCN %s
 ; RUN: llc -O0 -amdgpu-spill-sgpr-to-vgpr=0 -march=amdgcn -mattr=+vgpr-spilling -verify-machineinstrs < %s | FileCheck -check-prefix=TOVMEM -check-prefix=GCN %s
 ; RUN: llc -O0 -amdgpu-spill-sgpr-to-vgpr=0 -amdgpu-spill-sgpr-to-smem=0 -march=amdgcn -mcpu=tonga -mattr=+vgpr-spilling -verify-machineinstrs < %s | FileCheck -check-prefix=TOVMEM -check-prefix=GCN %s
 ; RUN: llc -O0 -amdgpu-spill-sgpr-to-vgpr=0 -amdgpu-spill-sgpr-to-smem=1 -march=amdgcn -mcpu=tonga -mattr=+vgpr-spilling -verify-machineinstrs < %s | FileCheck -check-prefix=TOSMEM -check-prefix=GCN %s
 
 ; XXX - Why does it like to use vcc?
 
 ; GCN-LABEL: {{^}}spill_m0:
-; TOSMEM: s_mov_b32 s84, SCRATCH_RSRC_DWORD0
+; TOSMEM: s_mov_b32 s[[LO:[0-9]+]], SCRATCH_RSRC_DWORD0
+; TOSMEM: s_mov_b32 s[[HI:[0-9]+]], 0xe80000
 
 ; GCN-DAG: s_cmp_lg_u32
 
 ; TOVGPR-DAG: s_mov_b32 [[M0_COPY:s[0-9]+]], m0
 ; TOVGPR: v_writelane_b32 [[SPILL_VREG:v[0-9]+]], [[M0_COPY]], 0
 
 ; TOVMEM-DAG: s_mov_b32 [[M0_COPY:s[0-9]+]], m0
 ; TOVMEM-DAG: v_mov_b32_e32 [[SPILL_VREG:v[0-9]+]], [[M0_COPY]]
 ; TOVMEM: buffer_store_dword [[SPILL_VREG]], off, s{{\[[0-9]+:[0-9]+\]}}, s{{[0-9]+}} ; 4-byte Folded Spill
 ; TOVMEM: s_waitcnt vmcnt(0)
 
 ; TOSMEM-DAG: s_mov_b32 [[M0_COPY:s[0-9]+]], m0
 ; TOSMEM: s_mov_b32 m0, s3{{$}}
 ; TOSMEM-NOT: [[M0_COPY]]
-; TOSMEM: s_buffer_store_dword [[M0_COPY]], s[84:87], m0 ; 4-byte Folded Spill
+; TOSMEM: s_buffer_store_dword [[M0_COPY]], s{{\[}}[[LO]]:[[HI]]], m0 ; 4-byte Folded Spill
 ; TOSMEM: s_waitcnt lgkmcnt(0)
 
 ; GCN: s_cbranch_scc1 [[ENDIF:BB[0-9]+_[0-9]+]]
 
 ; GCN: [[ENDIF]]:
 ; TOVGPR: v_readlane_b32 [[M0_RESTORE:s[0-9]+]], [[SPILL_VREG]], 0
 ; TOVGPR: s_mov_b32 m0, [[M0_RESTORE]]
 
 ; TOVMEM: buffer_load_dword [[RELOAD_VREG:v[0-9]+]], off, s{{\[[0-9]+:[0-9]+\]}}, s{{[0-9]+}} ; 4-byte Folded Reload
 ; TOVMEM: s_waitcnt vmcnt(0)
 ; TOVMEM: v_readfirstlane_b32 [[M0_RESTORE:s[0-9]+]], [[RELOAD_VREG]]
 ; TOVMEM: s_mov_b32 m0, [[M0_RESTORE]]
 
 ; TOSMEM: s_mov_b32 m0, s3{{$}}
-; TOSMEM: s_buffer_load_dword [[M0_RESTORE:s[0-9]+]], s[84:87], m0 ; 4-byte Folded Reload
+; TOSMEM: s_buffer_load_dword [[M0_RESTORE:s[0-9]+]], s{{\[}}[[LO]]:[[HI]]], m0 ; 4-byte Folded Reload
 ; TOSMEM-NOT: [[M0_RESTORE]]
 ; TOSMEM: s_mov_b32 m0, [[M0_RESTORE]]
 
 ; GCN: s_add_i32 s{{[0-9]+}}, m0, 1
 define void @spill_m0(i32 %cond, i32 addrspace(1)* %out) #0 {
 entry:
   %m0 = call i32 asm sideeffect "s_mov_b32 m0, 0", "={M0}"() #0
   %cmp0 = icmp eq i32 %cond, 0
endif:
   ret void
 }
 
 ; GCN-LABEL: {{^}}restore_m0_lds:
 ; TOSMEM: s_load_dwordx2 [[REG:s\[[0-9]+:[0-9]+\]]]
 ; TOSMEM: s_cmp_eq_u32
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_mov_b32 m0, s3
-; TOSMEM: s_buffer_store_dwordx2 [[REG]], s[84:87], m0 ; 8-byte Folded Spill
+; TOSMEM: s_buffer_store_dwordx2 [[REG]], s[88:91], m0 ; 8-byte Folded Spill
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_add_u32 m0, s3, 0x200
-; TOSMEM: s_buffer_store_dword s{{[0-9]+}}, s[84:87], m0 ; 4-byte Folded Spill
+; TOSMEM: s_buffer_store_dword s{{[0-9]+}}, s[88:91], m0 ; 4-byte Folded Spill
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_cbranch_scc1
 
 ; TOSMEM: s_mov_b32 m0, -1
 
 ; TOSMEM: s_mov_b32 vcc_hi, m0
 ; TOSMEM: s_mov_b32 m0, s3
-; TOSMEM: s_buffer_load_dwordx2 s{{\[[0-9]+:[0-9]+\]}}, s[84:87], m0 ; 8-byte Folded Reload
+; TOSMEM: s_buffer_load_dwordx2 s{{\[[0-9]+:[0-9]+\]}}, s[88:91], m0 ; 8-byte Folded Reload
 ; TOSMEM: s_mov_b32 m0, vcc_hi
 ; TOSMEM: s_waitcnt lgkmcnt(0)
 
 ; TOSMEM: ds_write_b64
 
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_add_u32 m0, s3, 0x200
-; TOSMEM: s_buffer_load_dword s0, s[84:87], m0 ; 4-byte Folded Reload
+; TOSMEM: s_buffer_load_dword s0, s[88:91], m0 ; 4-byte Folded Reload
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_waitcnt lgkmcnt(0)
 ; TOSMEM-NOT: m0
 ; TOSMEM: s_mov_b32 m0, s0
 ; TOSMEM: ; use m0
 
 ; TOSMEM: s_dcache_wb
 ; TOSMEM: s_endpgm