AMDGPU: Mark scalar loads as rematerializable

[arsenm]

Summary

This should be true, but this is useless as is. The rematerialization
logic only permits rematerialize with constant physical register uses,
so non-constant physregs or virtual register uses (the case that
really matters) are not rematerialized. Add the tests which shows
nothing happens, but should in the future.

Also, all loads should really be rematerializable so in the future
this should apply to all the other kinds.

[arsenm]

I don't this is generally OK. It will probably work now since we only use SMRD currently for constant memory, but if it were a volatile load it can't be rematerialized. The generic version checks the AA if the load is invariant.

[michel.daenzer]

Also, is rematerializing SMRD instructions to save SGPR spills really a win? 64 SGPRs can be spilled to a single VGPR, but the rematerialized SMRD instruction is only cheap if the loaded value is still in cache, right?

[arsenm]

lib/Target/AMDGPU/SIInstrInfo.cpp

79–81 ↗

(On Diff #30980)

We probably don't need this. It looks like if isReallyTriviallyReMaterializable fails, it will still check isReallyTriviallyReMaterializableGeneric which checks the AA for you

[mareko]

In D11621#215003, @michel.daenzer wrote:

Also, is rematerializing SMRD instructions to save SGPR spills really a win? 64 SGPRs can be spilled to a single VGPR, but the rematerialized SMRD instruction is only cheap if the loaded value is still in cache, right?

Yes, it is a win. The SMEM cache is shared by 4 CUs, which means only one out of at least 16 and at most 160 waves will get a cache miss in most cases. That leaves us with cache hits, which take 16-20 clocks (I don't remember exactly), which is the approximate cost of 4 v_read/writelane instructions. By that logic, spilling and restoring is twice as expensive as repeating the load.

[arsenm]

A test would be nice. This should assert that it isn't a store though in case we see a scalar store

[arsenm]

I think we need https://reviews.llvm.org/D106408 to make this really work, and avoid the special case in isReallyTriviallyReMaterializable

[rampitec]

I do not think this is OK. We may issue a scalar load before a first potentially aliasing store. After the store it must become a VMEM load because caches are not coherent. That said you cannot reorder SMRD with aliasing VMEM store, both because of the coherency and because memory may be indeed overwritten.

[arsenm]

In D11621#3958647, @rampitec wrote:

I do not think this is OK. We may issue a scalar load before a first potentially aliasing store. After the store it must become a VMEM load because caches are not coherent. That said you cannot reorder SMRD with aliasing VMEM store, both because of the coherency and because memory may be indeed overwritten.

This only happens for invariant MMOs. This isn't the only condition

[rampitec]

In D11621#3958649, @arsenm wrote:

In D11621#3958647, @rampitec wrote:

I do not think this is OK. We may issue a scalar load before a first potentially aliasing store. After the store it must become a VMEM load because caches are not coherent. That said you cannot reorder SMRD with aliasing VMEM store, both because of the coherency and because memory may be indeed overwritten.

This only happens for invariant MMOs. This isn't the only condition

Where is that checked? Also needs tests, both positive and negative.

[arsenm]

In D11621#3961342, @rampitec wrote:

In D11621#3958649, @arsenm wrote:

In D11621#3958647, @rampitec wrote:

I do not think this is OK. We may issue a scalar load before a first potentially aliasing store. After the store it must become a VMEM load because caches are not coherent. That said you cannot reorder SMRD with aliasing VMEM store, both because of the coherency and because memory may be indeed overwritten.

This only happens for invariant MMOs. This isn't the only condition

Where is that checked? Also needs tests, both positive and negative.

https://github.com/llvm/llvm-project/blob/442c13f9ff8a8c6739d649c5a68e065f2a727480/llvm/lib/CodeGen/TargetInstrInfo.cpp#L956

The patch as-is doesn't apply obviously. I wrote tests and found they don't work due to virtual register uses in any real case (which is I'm guessing what the special case in isReallyTriviallyReMaterializable is about)

[rampitec]

In D11621#3961345, @arsenm wrote:

In D11621#3961342, @rampitec wrote:

In D11621#3958649, @arsenm wrote:

In D11621#3958647, @rampitec wrote:

I do not think this is OK. We may issue a scalar load before a first potentially aliasing store. After the store it must become a VMEM load because caches are not coherent. That said you cannot reorder SMRD with aliasing VMEM store, both because of the coherency and because memory may be indeed overwritten.

This only happens for invariant MMOs. This isn't the only condition

Where is that checked? Also needs tests, both positive and negative.

https://github.com/llvm/llvm-project/blob/442c13f9ff8a8c6739d649c5a68e065f2a727480/llvm/lib/CodeGen/TargetInstrInfo.cpp#L956

The patch as-is doesn't apply obviously. I wrote tests and found they don't work due to virtual register uses in any real case (which is I'm guessing what the special case in isReallyTriviallyReMaterializable is about)

I thought we are overriding TargetInstrInfo::isReallyTriviallyReMaterializable completely.

[arsenm]

In D11621#3961388, @rampitec wrote:

I thought we are overriding TargetInstrInfo::isReallyTriviallyReMaterializable completely.

That doesn't bypass the virtual register use checks.

[arsenm]

Rebase and add tests, but this doesn't do anything. Either need to bring back the hack or fix virtual register handling

[arsenm]

ping

[foad]

Is this NFC?

[arsenm]

In D11621#4398353, @foad wrote:

Is this NFC?

Apparently not, @piotr reported some big improvements. So I suppose this could use more tests where it demonstrates real improvements

[arsenm]

In D11621#4399175, @arsenm wrote:

In D11621#4398353, @foad wrote:

Is this NFC?

Apparently not, @piotr reported some big improvements. So I suppose this could use more tests where it demonstrates real improvements

Apparently that was with a patch of the patch, so this is probably still NFC until something else happens

[piotr]

Yes - this is NFC, but paves the way for other changes.

I experimented with adding isSMRD() check in AMDGPU hook SIInstrInfo::isReallyTriviallyReMaterializable on top of D11621, and that resulted in some useful rematerialization.

However, it also resulted in some redundant registers being used because there is no mechanism to shrink the load to only load the part needed for current rematerialization context. So that needs to be added before we extend SIInstrInfo::isReallyTriviallyReMaterializable I think.

[arsenm]

4c0fc4841b43aa3d6efb76cf0030c46cb0cc9a3e

llvm/lib/Target/AMDGPU/SMInstructions.td

 
 class SM_Load_Pseudo <string opName, RegisterClass baseClass,
                       RegisterClass dstClass, OffsetMode offsets>
   : SM_Pseudo<opName, (outs dstClass:$sdst),
               !con((ins baseClass:$sbase), offsets.Ins, (ins CPol:$cpol)),
               " $sdst, $sbase, " # offsets.Asm # "$cpol", []> {
   RegisterClass BaseClass = baseClass;
   let mayLoad = 1;
+  let isReMaterializable = 1;
   let mayStore = 0;
   let has_glc = 1;
   let has_dlc = 1;
   let has_offset = offsets.HasOffset;
   let has_soffset = offsets.HasSOffset;
   let PseudoInstr = opName # offsets.Variant;
 }
 

llvm/test/CodeGen/AMDGPU/remat-smrd.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -verify-machineinstrs --stress-regalloc=2 -start-before=greedy -stop-after=virtregrewriter -o - %s | FileCheck -check-prefix=GCN %s
+
+# Case that should really rematerialize
+---
+name:            test_remat_s_load_dword_imm
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_imm
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 4, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 8, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 0, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 4, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 8, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_load_dword_immx2
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_immx2
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = S_LOAD_DWORDX2_IMM renamable $sgpr0_sgpr1, 0, 0 :: (dereferenceable invariant load (s64), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr2_sgpr3, %stack.1, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = S_LOAD_DWORDX2_IMM renamable $sgpr0_sgpr1, 4, 0 :: (dereferenceable invariant load (s64), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr2_sgpr3, %stack.0, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = S_LOAD_DWORDX2_IMM renamable $sgpr0_sgpr1, 8, 0 :: (dereferenceable invariant load (s64), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr2_sgpr3, %stack.2, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = SI_SPILL_S64_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr2_sgpr3
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr2_sgpr3
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = SI_SPILL_S64_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr2_sgpr3
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_64_xexec = S_LOAD_DWORDX2_IMM %0, 0, 0 :: (invariant dereferenceable load (s64), addrspace 4)
+    %2:sreg_64_xexec = S_LOAD_DWORDX2_IMM %0, 4, 0 :: (invariant dereferenceable load (s64), addrspace 4)
+    %3:sreg_64_xexec = S_LOAD_DWORDX2_IMM %0, 8, 0 :: (invariant dereferenceable load (s64), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_load_dword_immx4
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_immx4
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = S_LOAD_DWORDX4_IMM renamable $sgpr0_sgpr1, 0, 0 :: (dereferenceable invariant load (s128), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S128_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7, %stack.1, implicit $exec, implicit $sp_reg :: (store (s128) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = S_LOAD_DWORDX4_IMM renamable $sgpr0_sgpr1, 4, 0 :: (dereferenceable invariant load (s128), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S128_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7, %stack.0, implicit $exec, implicit $sp_reg :: (store (s128) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = S_LOAD_DWORDX4_IMM renamable $sgpr0_sgpr1, 8, 0 :: (dereferenceable invariant load (s128), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S128_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7, %stack.2, implicit $exec, implicit $sp_reg :: (store (s128) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = SI_SPILL_S128_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s128) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = SI_SPILL_S128_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s128) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = SI_SPILL_S128_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s128) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sgpr_128 = S_LOAD_DWORDX4_IMM %0, 0, 0 :: (invariant dereferenceable load (s128), addrspace 4)
+    %2:sgpr_128 = S_LOAD_DWORDX4_IMM %0, 4, 0 :: (invariant dereferenceable load (s128), addrspace 4)
+    %3:sgpr_128 = S_LOAD_DWORDX4_IMM %0, 8, 0 :: (invariant dereferenceable load (s128), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_load_dword_immx8
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_immx8
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = S_LOAD_DWORDX8_IMM renamable $sgpr0_sgpr1, 0, 0 :: (dereferenceable invariant load (s256), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S256_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11, %stack.1, implicit $exec, implicit $sp_reg :: (store (s256) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = S_LOAD_DWORDX8_IMM renamable $sgpr0_sgpr1, 4, 0 :: (dereferenceable invariant load (s256), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S256_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11, %stack.0, implicit $exec, implicit $sp_reg :: (store (s256) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = S_LOAD_DWORDX8_IMM renamable $sgpr0_sgpr1, 8, 0 :: (dereferenceable invariant load (s256), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S256_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11, %stack.2, implicit $exec, implicit $sp_reg :: (store (s256) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = SI_SPILL_S256_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s256) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = SI_SPILL_S256_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s256) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11 = SI_SPILL_S256_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s256) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sgpr_256 = S_LOAD_DWORDX8_IMM %0, 0, 0 :: (invariant dereferenceable load (s256), addrspace 4)
+    %2:sgpr_256 = S_LOAD_DWORDX8_IMM %0, 4, 0 :: (invariant dereferenceable load (s256), addrspace 4)
+    %3:sgpr_256 = S_LOAD_DWORDX8_IMM %0, 8, 0 :: (invariant dereferenceable load (s256), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_load_dword_immx16
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_immx16
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = S_LOAD_DWORDX16_IMM renamable $sgpr0_sgpr1, 0, 0 :: (dereferenceable invariant load (s512), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S512_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19, %stack.1, implicit $exec, implicit $sp_reg :: (store (s512) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = S_LOAD_DWORDX16_IMM renamable $sgpr0_sgpr1, 4, 0 :: (dereferenceable invariant load (s512), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S512_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19, %stack.0, implicit $exec, implicit $sp_reg :: (store (s512) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = S_LOAD_DWORDX16_IMM renamable $sgpr0_sgpr1, 8, 0 :: (dereferenceable invariant load (s512), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S512_SAVE killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19, %stack.2, implicit $exec, implicit $sp_reg :: (store (s512) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = SI_SPILL_S512_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s512) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = SI_SPILL_S512_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s512) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19 = SI_SPILL_S512_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s512) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sgpr_512 = S_LOAD_DWORDX16_IMM %0, 0, 0 :: (invariant dereferenceable load (s512), addrspace 4)
+    %2:sgpr_512 = S_LOAD_DWORDX16_IMM %0, 4, 0 :: (invariant dereferenceable load (s512), addrspace 4)
+    %3:sgpr_512 = S_LOAD_DWORDX16_IMM %0, 8, 0 :: (invariant dereferenceable load (s512), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_get_waveid_in_workgroup
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_get_waveid_in_workgroup
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_GET_WAVEID_IN_WORKGROUP
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_GET_WAVEID_IN_WORKGROUP
+    ; GCN-NEXT: renamable $sgpr0 = S_GET_WAVEID_IN_WORKGROUP
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_GET_WAVEID_IN_WORKGROUP
+    %2:sreg_32_xm0_xexec = S_GET_WAVEID_IN_WORKGROUP
+    %3:sreg_32_xm0_xexec = S_GET_WAVEID_IN_WORKGROUP
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+
+---
+name:            test_remat_s_load_dword_sgpr
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9, $sgpr10
+    ; GCN-LABEL: name: test_remat_s_load_dword_sgpr
+    ; GCN: liveins: $sgpr10, $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = COPY $sgpr10
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR renamable $sgpr2_sgpr3, renamable $sgpr0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR renamable $sgpr2_sgpr3, renamable $sgpr0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR renamable $sgpr2_sgpr3, renamable $sgpr0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.2, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.2, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.2, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3, implicit killed renamable $sgpr0
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sgpr_32 = COPY $sgpr10
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR %0, %1, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR %0, %1, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %4:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR %0, %1, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_NOP 0, implicit %4
+    S_ENDPGM 0, implicit %0, implicit %1
+...
+
+---
+name:            test_remat_s_load_dword_sgpr_imm
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9, $sgpr10
+    ; GCN-LABEL: name: test_remat_s_load_dword_sgpr_imm
+    ; GCN: liveins: $sgpr10, $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = COPY $sgpr10
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR_IMM renamable $sgpr2_sgpr3, renamable $sgpr0, 0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR_IMM renamable $sgpr2_sgpr3, renamable $sgpr0, 4, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_SGPR_IMM renamable $sgpr2_sgpr3, renamable $sgpr0, 8, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.2, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.2, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.2, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3, implicit killed renamable $sgpr0
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sgpr_32 = COPY $sgpr10
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR_IMM %0, %1, 0, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR_IMM %0, %1, 4, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %4:sreg_32_xm0_xexec = S_LOAD_DWORD_SGPR_IMM %0, %1, 8, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_NOP 0, implicit %4
+    S_ENDPGM 0, implicit %0, implicit %1
+...
+
+---
+name:            test_remat_s_load_dword_imm_ci
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_load_dword_imm_ci
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM_ci renamable $sgpr2_sgpr3, 0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_IMM_ci renamable $sgpr2_sgpr3, 4, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM_ci renamable $sgpr2_sgpr3, 8, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM_ci %0, 0, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM_ci %0, 4, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM_ci %0, 8, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_buffer_load_dword_imm
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-LABEL: name: test_remat_s_buffer_load_dword_imm
+    ; GCN: liveins: $sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr4_sgpr5_sgpr6_sgpr7 = COPY $sgpr8_sgpr9_sgpr10_sgpr11
+    ; GCN-NEXT: renamable $sgpr0 = S_BUFFER_LOAD_DWORD_IMM renamable $sgpr4_sgpr5_sgpr6_sgpr7, 0, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_BUFFER_LOAD_DWORD_IMM renamable $sgpr4_sgpr5_sgpr6_sgpr7, 4, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_BUFFER_LOAD_DWORD_IMM renamable $sgpr4_sgpr5_sgpr6_sgpr7, 8, 0 :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr4_sgpr5_sgpr6_sgpr7
+    %0:sgpr_128 = COPY $sgpr8_sgpr9_sgpr10_sgpr11
+    %1:sreg_32_xm0_xexec = S_BUFFER_LOAD_DWORD_IMM %0, 0, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_BUFFER_LOAD_DWORD_IMM %0, 4, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_BUFFER_LOAD_DWORD_IMM %0, 8, 0 :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_remat_s_scratch_load_dword_imm
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_remat_s_scratch_load_dword_imm
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_SCRATCH_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 0, 0, implicit $flat_scr :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_SCRATCH_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 4, 0, implicit $flat_scr :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_SCRATCH_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 8, 0, implicit $flat_scr :: (dereferenceable invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_SCRATCH_LOAD_DWORD_IMM %0, 0, 0, implicit $flat_scr :: (invariant dereferenceable load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_SCRATCH_LOAD_DWORD_IMM %0, 4, 0, implicit $flat_scr :: (invariant dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_SCRATCH_LOAD_DWORD_IMM %0, 8, 0, implicit $flat_scr :: (invariant dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+#---------------------------------------------------------------------
+# Negative tests
+#---------------------------------------------------------------------
+
+---
+name:            test_no_remat_s_load_dword_not_invariant
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_no_remat_s_load_dword_not_invariant
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 0, 0 :: (dereferenceable load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 4, 0 :: (dereferenceable load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 8, 0 :: (dereferenceable load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 0, 0 :: (dereferenceable load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 4, 0 :: (dereferenceable load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 8, 0 :: (dereferenceable load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_no_remat_s_load_dword_not_dereferenceable
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_no_remat_s_load_dword_not_dereferenceable
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = COPY $sgpr8_sgpr9
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 0, 0 :: (invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.0, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.0, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr1 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 4, 0 :: (invariant load (s32), addrspace 4)
+    ; GCN-NEXT: renamable $sgpr0 = S_LOAD_DWORD_IMM renamable $sgpr2_sgpr3, 8, 0 :: (invariant load (s32), addrspace 4)
+    ; GCN-NEXT: SI_SPILL_S32_SAVE killed renamable $sgpr0, %stack.1, implicit $exec, implicit $sp_reg :: (store (s32) into %stack.1, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.0, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr1
+    ; GCN-NEXT: renamable $sgpr0 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s32) from %stack.1, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr2_sgpr3
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 0, 0 :: (invariant load (s32), addrspace 4)
+    %2:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 4, 0 :: (invariant load (s32), addrspace 4)
+    %3:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM %0, 8, 0 :: (invariant load (s32), addrspace 4)
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_no_remat_s_memtime
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_no_remat_s_memtime
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: SI_SPILL_S64_SAVE $sgpr8_sgpr9, %stack.0, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = S_MEMTIME
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = S_MEMTIME
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr0_sgpr1, %stack.2, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = S_MEMTIME
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr0_sgpr1, %stack.1, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr2_sgpr3
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0_sgpr1
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0_sgpr1
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_64_xexec = S_MEMTIME
+    %2:sreg_64_xexec = S_MEMTIME
+    %3:sreg_64_xexec = S_MEMTIME
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...
+
+---
+name:            test_no_remat_s_memrealtime
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr8_sgpr9
+    ; GCN-LABEL: name: test_no_remat_s_memrealtime
+    ; GCN: liveins: $sgpr8_sgpr9
+    ; GCN-NEXT: {{  $}}
+    ; GCN-NEXT: SI_SPILL_S64_SAVE $sgpr8_sgpr9, %stack.0, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr2_sgpr3 = S_MEMREALTIME
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = S_MEMREALTIME
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr0_sgpr1, %stack.2, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = S_MEMREALTIME
+    ; GCN-NEXT: SI_SPILL_S64_SAVE killed renamable $sgpr0_sgpr1, %stack.1, implicit $exec, implicit $sp_reg :: (store (s64) into %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr2_sgpr3
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.2, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.2, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0_sgpr1
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.1, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.1, align 4, addrspace 5)
+    ; GCN-NEXT: S_NOP 0, implicit killed renamable $sgpr0_sgpr1
+    ; GCN-NEXT: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load (s64) from %stack.0, align 4, addrspace 5)
+    ; GCN-NEXT: S_ENDPGM 0, implicit killed renamable $sgpr0_sgpr1
+    %0:sreg_64_xexec = COPY $sgpr8_sgpr9
+    %1:sreg_64_xexec = S_MEMREALTIME
+    %2:sreg_64_xexec = S_MEMREALTIME
+    %3:sreg_64_xexec = S_MEMREALTIME
+    S_NOP 0, implicit %1
+    S_NOP 0, implicit %2
+    S_NOP 0, implicit %3
+    S_ENDPGM 0, implicit %0
+...