[RISCV][PrologEpilogInserter] "Float" emergency spill slots to avoid making them immediately unreachable from the stack pointer

[rogfer01]

Summary

In RISC-V there is a single addressing mode of the form imm(reg) where imm is a signed integer of 12-bit with a range of [-2048..2047] bytes from reg.

The test MultiSource/UnitTests/C++11/frame_layout of the LLVM test-suite exercises several scenarios with the stack, including function call where the stack will need to be realigned to to a local variable having a large alignment of 4096 bytes.

In situations of large stacks, the RISC-V backend (in RISCVFrameLowering) reserves an extra emergency spill slot which can be used (if no free register is found) by the register scavenger after the frame indexes have been eliminated. PrologEpilogInserter already takes care of keeping the emergency spill slots as close as possible to the stack pointer or frame pointer (depending on what the function will use). However there is a final alignment step to honour the maximum alignment of the stack that, when using the stack pointer to access the emergency spill slots, has the side effect of setting them farther from the stack pointer.

In the case of the frame_layout testcase, the net result is that we do have an emergency spill slot but it is so far from the stack pointer (more than 2048 bytes due to the extra alignment of a variable to 4096 bytes) that it becomes unreachable via any immediate offset.

During elimination of the frame index, many (regular) offsets of the stack may be immediately unreachable already. Their address needs to be computed using a register. A virtual register is created and later RegisterScavenger should be able to find an unused (physical) register. However if no register is available, RegisterScavenger will pick a physical register and spill it onto an emergency stack slot, while we compute the offset (restoring the chosen register after all this). This assumes that the emergency stack slot is easily reachable (this is, without requiring another register!).

This is the assumption we seem to break when we perform the extra alignment in PrologEpilogInserter.

My understanding is that we can "float" the emergency spill slots by increasing (in absolute value) their offsets from the incoming stack pointer. This way the emergency spill slots will remain close to the stack pointer (once the function has allocated storage for the stack, including the needed realignment). The new size computed in PrologEpilogInserter is padding so it should be OK to move the emergency spill slots there. Also because we're increasing the alignment, the new location should stay aligned for the purpose of the emergency spill slots.

Note that this change also impacts other backends as shown by the tests. So far they seem minor adjustments to the emergency stack slot offset to me, making it even closer to the stack pointer than before, so they seem OK.

A final comment: this seems an issue that could impact any target. However this problem seems only easily reproducible in RISC-V. For some reason other targets (such as ARM or AArch64) always happen to find a free register and the emergency spill slot (if any) ends going unused. Why we are so unlucky in RISC-V I have not been able to determine yet but this seems some side-effect of other unrelated changes in CodeGen.

[rogfer01]

ChangeLog:

• Forgot to add the test for RISC-V

[luismarques]

The code makes sense to me and it seems correct, but there may be assumptions about the way that mechanism operates that I'm not familiar with. Some comments:

• Thanks for the thorough work for this patch and D89237. Much better than just tweaking the RISC-V stack spill limit, sweeping the issue under the rug and calling it a day!
• It might be a good idea to add additional reviewers, to check the tests for the other targets?
• Nit: could the RISC-V test be further minimized? It still seems a bit big/complex. I'm assuming that's the minimized version of the test suite file that was failing to compile. Maybe also add a small comment in that file explaining what's being tested and describing the origin of the test.
• Could there be cases (due to future changes elsewhere) where these tests still pass but are no longer checking the conditions handled by the new code? If so, we could check that a debug run of the test prints one of the new debug strings (probably the one inside the loop). That's done in llvm/test/CodeGen/RISCV/disjoint.ll, by using REQUIRES: asserts and -debug-only=....

[efriedma]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1085

If the stack alignment is large enough that all immediate offsets from the stack pointer are stuck in padding, the performance of the resulting code is going to be terrible, period. Really, we want the gap somewhere between the allocas that require it, and the rest of the stack allocations. Changing the location of the scavenging slot, specifically, seems a bit hacky. Maybe it's good enough, though.

On a related note, I'm tempted to say we should add a mechanism to allow targets to cap stack realignment to something sane, to allow avoiding situations like this. (We currently have isStackRealignable(), but that completely disables realignment.)

[rogfer01]

Looking into the origin of this problem, git bisect points at cb82de29601 but I don't think it is the culprit. Looks like we may have been a bit unlucky during register allocation with the new sequence of instructions.

After cb82de29601, -stop-before=prologepilog shows we need a couple more of spill slots for RegAlloc. It is accessing to one of them what triggers the issue with the register scavenger and the emergency spill slot.

[rogfer01]

ChangeLog:

• Use llvm-reduce to reduce the test even further
• Use update_llc_test_checks.py to reduce maintenance burden of CHECK lines
• Add DEBUG checks so we can tell PEI is adjusting the offsets.

[rogfer01]

In D89239#2325224, @luismarques wrote:

The code makes sense to me and it seems correct, but there may be assumptions about the way that mechanism operates that I'm not familiar with. Some comments:

• It might be a good idea to add additional reviewers, to check the tests for the other targets?

Sure!

@sdesmalen @MatzeB mind to take a look at the AArch64 changes, they seem OK to me.

@arsenm is the AMDGPU change OK?

• Nit: could the RISC-V test be further minimized? It still seems a bit big/complex. I'm assuming that's the minimized version of the test suite file that was failing to compile. Maybe also add a small comment in that file explaining what's being tested and describing the origin of the test.

I was able to reduce it a bit more with llvm-reduce. Not very noticeable but it was worth updating the test.

• Could there be cases (due to future changes elsewhere) where these tests still pass but are no longer checking the conditions handled by the new code? If so, we could check that a debug run of the test prints one of the new debug strings (probably the one inside the loop). That's done in llvm/test/CodeGen/RISCV/disjoint.ll, by using REQUIRES: asserts and -debug-only=....

So I've done two things here:

• to reduce the burden of maintaning the test I used update_llc_test_checks.py this time.
• I included your suggestion of using the debug message plus a REQUIRES: asserts so we can ensure we are going through the specific new path of PEI.

[sdesmalen]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1092

Is this approach the same as setting the alignment of the last SFI to StackAlign?

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

This test seems quite substantial, is there no simpler IR to test this?

[arichardson]

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

This test can definitely be reduced a lot more. Also you could run just this pass and make it a MIR test instead.

[rogfer01]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1092

Not sure I understood you comment.

This removes the additional offset that was introduced in line 1081 for all the SFIs, not only the last one.

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

The test is already an MIR test. The YAML representation contains a copy of the IR (which it is required to create essential data structures such as llvm::Function). I can remove most of it as I understand adds none to little information here.

I will look into how to reduce the large amount of machine instructions: part of the difficulty is making sure no register is left for the register scavenger when eliminating the frame index of the problematic stack slot (the one that used to be, before this change, too far from the top of the stack).

I'll look into this again because I think it is possible to do so.

[arichardson]

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

I meant using -run-pass=prologepilog and using update_mir_checks.py to generate the CHECK lines.

I just ran creduce over your test case and it gave me this:

# RUN: llc -mtriple riscv64 -start-before=prologepilog -o - -verify-machineinstrs %s
name:            a
stack:
  - { id: 0 , size: 8, alignment: 4096 }
body:             |
  bb.0:
    $= ADD $, $x10
    SD $x12, $, $x13
    $= DIVU $, $x30
    $= DIVU $, $x29
    $= DIV $, $x15
    $= DIV $, $x14
    $= DIV $, $x1
    $= DIV $, $x7
    $= DIV $, $x6
    $= DIV $, $x31
    $= DIV $, $x28
    $= DIV $, $x5
    $= DIV $, $x17
    $= DIV $, $x16
    $= DIV $x11, $

Feeding that into /bin/llc -mtriple riscv64 -start-before=prologepilog -o - -verify-machineinstrs -stop-before=prologepilog
yields this:

--- |
  ; ModuleID = '/local/scratch/alr48/cheri/build/llvm-project-build/test-reduce.mir'
  source_filename = "/local/scratch/alr48/cheri/build/llvm-project-build/test-reduce.mir"
  target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128"
  target triple = "riscv64"

  define void @a() {
  entry:
    unreachable
  }

...
---
name:            a
alignment:       1
exposesReturnsTwice: false
legalized:       false
regBankSelected: false
selected:        false
failedISel:      false
tracksRegLiveness: false
hasWinCFI:       false
registers:       []
liveins:         []
frameInfo:
  isFrameAddressTaken: false
  isReturnAddressTaken: false
  hasStackMap:     false
  hasPatchPoint:   false
  stackSize:       0
  offsetAdjustment: 0
  maxAlignment:    4096
  adjustsStack:    false
  hasCalls:        false
  stackProtector:  ''
  maxCallFrameSize: 4294967295
  cvBytesOfCalleeSavedRegisters: 0
  hasOpaqueSPAdjustment: false
  hasVAStart:      false
  hasMustTailInVarArgFunc: false
  localFrameSize:  0
  savePoint:       ''
  restorePoint:    ''
fixedStack:      []
stack:
  - { id: 0, name: '', type: default, offset: 0, size: 8, alignment: 4096,
      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
callSites:       []
constants:       []
machineFunctionInfo: {}
body:             |
  bb.0:
    $noreg = ADD $noreg, $x10
    SD $x12, $noreg, $x13
    $noreg = DIVU $noreg, $x30
    $noreg = DIVU $noreg, $x29
    $noreg = DIV $noreg, $x15
    $noreg = DIV $noreg, $x14
    $noreg = DIV $noreg, $x1
    $noreg = DIV $noreg, $x7
    $noreg = DIV $noreg, $x6
    $noreg = DIV $noreg, $x31
    $noreg = DIV $noreg, $x28
    $noreg = DIV $noreg, $x5
    $noreg = DIV $noreg, $x17
    $noreg = DIV $noreg, $x16
    $noreg = DIV $x11, $noreg

...

[rogfer01]

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

Thanks @arichardson for the reduced testcase.

However, I'm not sure we'd be testing the right thing here. We need a frame index that when eliminated has to use the emergency spill slot. Something like the SD instruction in line 500:

SD killed renamable $x12, %stack.1, 0

Maybe I'm missing something.

[arichardson]

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

2

Yeah it probably needs something like this.

The reduced testcase I posted here was just what creduce gave me for the error message that I see when compiling the LLVM test suite right now: LLVM ERROR: Incomplete scavenging after 2nd pass

[sdesmalen]

@rogfer01 the changes to the aarch64 tests seem fine to me.

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1092

You're right, these are all allocated individually, so changing the alignment of the last SFI would indeed only affect the position of that one.

I guess I was thinking along the lines to what is done for the Local stack allocation block. i.e. allocate the SFIs together as one block (in this case aligned to the stack alignment), and then adjust the individual offsets for each of the SFIs? That would place the logic for this in one place. Now it first assigns offsets for the slots, then calculates stack alignment, and then fixes up the offsets based on the stack alignment. Is that something you have considered?

[rogfer01]

ChangeLog:

• Largely simplify the testcase by making a call that uses all the eligible registers forcing the RegisterScavenger to use the emergency spill.

[rogfer01]

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1092

Hi Sander, thanks for the suggestion. I hadn't considered it. I understand you suggest to do the adjustment earlier, in the loop in line 1053.

However I took a look and so far it does not seem an obvious thing to me. It seems I still have to take into account things like !TFI.targetHandlesStackFrameRounding() and MFI.adjustsStack() || MFI.hasVarSizedObjects(), etc. Also MaxAlign is passed by reference to AdjustStackoffset and may change during the loop. I am afraid it would make the code a messier. I could do that after the loop (before 1056) but I still have to copy some of the checks and calculations that happen in the body of 1058.

[rogfer01]

Ping.

[sdesmalen]

LGTM, thanks @rogfer01!

llvm/lib/CodeGen/PrologEpilogInserter.cpp

1083

nit: constraints

1092

My thought was more to create a new single object that comprises all SFIs, that is either allocated early or late, possibly adjusted for stack alignment. And then have one loop that allocates the individual SFIs 'within' that object, by setting offsets based on the encompassing object's start address. But thinking this through, it probably wouldn't be much of an improvement to the current mechanism, as it still requires allocating an object and adjusting it's offset.

Apologies if I have stalled the patch with this suggestion.

[sebastian-ne]

Hi,
I’m not completely sure what is going on here, but I think this change is causing a bug.
I’m seeing multiple FIs pointing to the same memory (they have the same offset), the debug output looks like this (shortened):

Reserved FI 24 for spilling FP
alloc FI(33) at SP[0]
…
alloc FI(22) at SP[128]
alloc FI(24) at SP[132]
alloc FI(34) at SP[136]
Adjusting emergency spill slots!
Adjusting offset of emergency spill slot #34 from 136 to 132

In the end, the emergency spill slot overlaps with FI #24.

This happened on the amdgpu target (it’s not visible in the test change here). I think the problem occurs if the stack grows upwards (i.e. growing towards higher addresses; yeah, weird target). The code in AdjustStackOffset behaves differently in that case.

As a sidenote about the problem solved with this change:
In amdgpu, if we cannot scavenge a register to access a stack slot (i.e. because we currently want to spill one), we fallback to modifying the stack pointer:

add sp, <offset>    ; <offset> is too large to be an immediate and we cannot scavenge a register
load r1, sp         ; Access stack slot
sub sp, <offset>    ; Restore original value

[asb]

@sebastian-ne: to ensure the right people see this, could you file a bug for your problem please? https://llvm.org/docs/HowToSubmitABug.html

[sebastian-ne]

Thanks @asb. I created 49686.

llvm/lib/CodeGen/PrologEpilogInserter.cpp

if (MFI.adjustsStack() || MFI.hasVarSizedObjects() ||
         (RegInfo->needsStackRealignment(MF) && MFI.getObjectIndexEnd() != 0))
       StackAlign = TFI.getStackAlign();
     else
       StackAlign = TFI.getTransientStackAlign();
 
     // If the frame pointer is eliminated, all frame offsets will be relative to
     // SP not FP. Align to MaxAlign so this works.
     StackAlign = std::max(StackAlign, MaxAlign);
+    int64_t OffsetBeforeAlignment = Offset;
     Offset = alignTo(Offset, StackAlign, Skew);
+
+    // If we have increased the offset to fulfill the alignment constrants,
+    // then the scavenging spill slots may become harder to reach from the
+    // stack pointer, float them so they stay close.
+    if (OffsetBeforeAlignment != Offset && RS && !EarlyScavengingSlots) {
+      SmallVector<int, 2> SFIs;
+      RS->getScavengingFrameIndices(SFIs);
+      LLVM_DEBUG(if (!SFIs.empty()) llvm::dbgs()
+                     << "Adjusting emergency spill slots!\n";);
+      int64_t Delta = Offset - OffsetBeforeAlignment;
+      for (SmallVectorImpl<int>::iterator I = SFIs.begin(), IE = SFIs.end();
+           I != IE; ++I) {
+        LLVM_DEBUG(llvm::dbgs() << "Adjusting offset of emergency spill slot #"
+                                << *I << " from " << MFI.getObjectOffset(*I););
+        MFI.setObjectOffset(*I, MFI.getObjectOffset(*I) - Delta);
+        LLVM_DEBUG(llvm::dbgs() << " to " << MFI.getObjectOffset(*I) << "\n";);
+      }
+    }
   }
 
   // Update frame info to pretend that this is part of the stack...
   int64_t StackSize = Offset - LocalAreaOffset;
   MFI.setStackSize(StackSize);
   NumBytesStackSpace += StackSize;
 }
 

llvm/test/CodeGen/AArch64/framelayout-scavengingslot.mir

 # RUN: llc -mtriple=aarch64-none-linux-gnu -run-pass=prologepilog %s -o - | FileCheck %s
 ---
 # This test verifies that the emergency scavenging slot is located near
 # the SP when the stack is realigned.
 name: LateScavengingSlotRealignment
 # CHECK-LABEL: name: LateScavengingSlotRealignment
 # CHECK: bb.0:
-# CHECK:      STRXui killed $[[SCRATCH:x[0-9]+]], $sp, 3
+# CHECK:      STRXui killed $[[SCRATCH:x[0-9]+]], $sp, 0
 # CHECK-NEXT: $[[SCRATCH]] = ADDXri $sp, 40, 0
 # CHECK-NEXT: STRXui $x0, killed $[[SCRATCH]], 4095
-# CHECK-NEXT: $[[SCRATCH]] = LDRXui $sp, 3
+# CHECK-NEXT: $[[SCRATCH]] = LDRXui $sp, 0
 # CHECK: bb.1:
 tracksRegLiveness: true
 frameInfo:
   isFrameAddressTaken: true
 stack:
   - { id: 0, size:    16, alignment: 16 }
   - { id: 1, size: 32768, alignment: 32 }
 body: |
   bb.0:
     liveins: $x0, $x8
     STRXui $x0, %stack.0, 0
     B %bb.1
   bb.1:
     liveins: $x1, $x2, $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12, $x13, $x14, $x15, $x16, $x17, $x18, $x19, $x20, $x21, $x22, $x23, $x24, $x25, $x26, $x27, $x28, $lr
     RET_ReallyLR implicit $x19, implicit $x20, implicit $x21, implicit $x22, implicit $x23, implicit $x24, implicit $x25, implicit $x26, implicit $x27, implicit $x28, implicit $lr
 ...

llvm/test/CodeGen/AArch64/framelayout-sve-scavengingslot.mir

 # RUN: llc -mtriple=aarch64-none-linux-gnu -run-pass=prologepilog -mattr=+sve %s -o - | FileCheck %s
 ---
 # This test verifies that the emergency scavenging slot is located near the SP/BP.
 name: LateScavengingSlot
 # CHECK-LABEL: name: LateScavengingSlot
 # CHECK: bb.0:
 # CHECK:      $sp = frame-setup ADDVL_XXI $sp, -1
 # CHECK-NEXT: $sp = frame-setup SUBXri $sp, 8, 12
 # CHECK-NEXT: $sp = frame-setup SUBXri $sp, 16, 0
-# CHECK:      STRXui killed $[[SCRATCH:x[0-9]+]], $sp, 1
+# CHECK:      STRXui killed $[[SCRATCH:x[0-9]+]], $sp, 0
 # CHECK-NEXT: $[[SCRATCH]] = ADDVL_XXI $fp, -1
 # CHECK-NEXT: STRXui $x0, killed $[[SCRATCH]], 0
 # CHECK: bb.1:
 tracksRegLiveness: true
 frameInfo:
   isFrameAddressTaken: true
 stack:
   - { id: 0, name: '', size: 32761, alignment: 8 }

llvm/test/CodeGen/AArch64/swiftself-scavenger.ll

 ; RUN: llc -o - %s | FileCheck %s
 ; Check that we reserve an emergency spill slot, even if we added an extra
 ; CSR spill for the values used by the swiftself parameter.
 ; CHECK-LABEL: func:
-; CHECK: str [[REG:x[0-9]+]], [sp, #8]
+; CHECK: str [[REG:x[0-9]+]], [sp]
 ; CHECK: add [[REG]], sp, #248
 ; CHECK: str xzr, [{{\s*}}[[REG]], #32760]
-; CHECK: ldr [[REG]], [sp, #8]
+; CHECK: ldr [[REG]], [sp]
 target triple = "arm64-apple-ios"
 
 @ptr8 = external global i8*
 @ptr64 = external global i64
 
 define hidden swiftcc void @func(i8* swiftself %arg) #0 {
 bb:
   %stack0 = alloca i8*, i32 5000, align 8

llvm/test/CodeGen/AMDGPU/pei-scavenge-vgpr-spill.mir

bb.0:
 
     ; GFX8-LABEL: name: pei_scavenge_vgpr_spill
     ; GFX8: liveins: $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15, $vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31, $vgpr32_vgpr33_vgpr34_vgpr35_vgpr36_vgpr37_vgpr38_vgpr39_vgpr40_vgpr41_vgpr42_vgpr43_vgpr44_vgpr45_vgpr46_vgpr47, $vgpr48_vgpr49_vgpr50_vgpr51_vgpr52_vgpr53_vgpr54_vgpr55_vgpr56_vgpr57_vgpr58_vgpr59_vgpr60_vgpr61_vgpr62_vgpr63, $vgpr64_vgpr65_vgpr66_vgpr67_vgpr68_vgpr69_vgpr70_vgpr71_vgpr72_vgpr73_vgpr74_vgpr75_vgpr76_vgpr77_vgpr78_vgpr79, $vgpr80_vgpr81_vgpr82_vgpr83_vgpr84_vgpr85_vgpr86_vgpr87_vgpr88_vgpr89_vgpr90_vgpr91_vgpr92_vgpr93_vgpr94_vgpr95, $vgpr96_vgpr97_vgpr98_vgpr99_vgpr100_vgpr101_vgpr102_vgpr103_vgpr104_vgpr105_vgpr106_vgpr107_vgpr108_vgpr109_vgpr110_vgpr111, $vgpr112_vgpr113_vgpr114_vgpr115_vgpr116_vgpr117_vgpr118_vgpr119_vgpr120_vgpr121_vgpr122_vgpr123_vgpr124_vgpr125_vgpr126_vgpr127, $vgpr128_vgpr129_vgpr130_vgpr131_vgpr132_vgpr133_vgpr134_vgpr135_vgpr136_vgpr137_vgpr138_vgpr139_vgpr140_vgpr141_vgpr142_vgpr143, $vgpr144_vgpr145_vgpr146_vgpr147_vgpr148_vgpr149_vgpr150_vgpr151_vgpr152_vgpr153_vgpr154_vgpr155_vgpr156_vgpr157_vgpr158_vgpr159, $vgpr160_vgpr161_vgpr162_vgpr163_vgpr164_vgpr165_vgpr166_vgpr167_vgpr168_vgpr169_vgpr170_vgpr171_vgpr172_vgpr173_vgpr174_vgpr175, $vgpr176_vgpr177_vgpr178_vgpr179_vgpr180_vgpr181_vgpr182_vgpr183_vgpr184_vgpr185_vgpr186_vgpr187_vgpr188_vgpr189_vgpr190_vgpr191, $vgpr192_vgpr193_vgpr194_vgpr195_vgpr196_vgpr197_vgpr198_vgpr199_vgpr200_vgpr201_vgpr202_vgpr203_vgpr204_vgpr205_vgpr206_vgpr207, $vgpr208_vgpr209_vgpr210_vgpr211_vgpr212_vgpr213_vgpr214_vgpr215_vgpr216_vgpr217_vgpr218_vgpr219_vgpr220_vgpr221_vgpr222_vgpr223, $vgpr224_vgpr225_vgpr226_vgpr227_vgpr228_vgpr229_vgpr230_vgpr231_vgpr232_vgpr233_vgpr234_vgpr235_vgpr236_vgpr237_vgpr238_vgpr239, $vgpr240_vgpr241_vgpr242_vgpr243_vgpr244_vgpr245_vgpr246_vgpr247, $vgpr248_vgpr249_vgpr250_vgpr251, $vgpr252_vgpr253_vgpr254_vgpr255, $vgpr2
     ; GFX8: $vgpr2 = V_WRITELANE_B32 $sgpr33, 0, undef $vgpr2
     ; GFX8: $sgpr4 = frame-setup S_ADD_U32 $sgpr32, 524224, implicit-def $scc
     ; GFX8: $sgpr33 = frame-setup S_AND_B32 killed $sgpr4, 4294443008, implicit-def $scc
     ; GFX8: $sgpr32 = frame-setup S_ADD_U32 $sgpr32, 1572864, implicit-def $scc
     ; GFX8: $vgpr0 = V_LSHRREV_B32_e64 6, $sgpr33, implicit $exec
-    ; GFX8: $sgpr4 = S_ADD_U32 $sgpr33, 524544, implicit-def $scc
-    ; GFX8: BUFFER_STORE_DWORD_OFFSET killed $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, killed $sgpr4, 0, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.3, addrspace 5)
+    ; GFX8: BUFFER_STORE_DWORD_OFFSET killed $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr33, 12, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.3, addrspace 5)
     ; GFX8: $vgpr3 = V_LSHRREV_B32_e64 6, $sgpr33, implicit $exec
     ; GFX8: $vcc_lo = S_MOV_B32 8192
     ; GFX8: $vgpr3, dead $vcc = V_ADD_CO_U32_e64 killed $vcc_lo, killed $vgpr3, 0, implicit $exec
     ; GFX8: $vgpr0 = V_OR_B32_e32 killed $vgpr3, $vgpr1, implicit $exec
     ; GFX8: $sgpr32 = frame-destroy S_SUB_U32 $sgpr32, 1572864, implicit-def $scc
     ; GFX8: $sgpr33 = V_READLANE_B32 $vgpr2, 0
-    ; GFX8: $sgpr4 = S_ADD_U32 $sgpr33, 524544, implicit-def $scc
-    ; GFX8: $vgpr3 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, killed $sgpr4, 0, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.3, addrspace 5)
+    ; GFX8: $vgpr3 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr33, 12, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.3, addrspace 5)
     ; GFX8: S_ENDPGM 0, csr_amdgpu_allvgprs
     ; GFX9-LABEL: name: pei_scavenge_vgpr_spill
     ; GFX9: liveins: $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15, $vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31, $vgpr32_vgpr33_vgpr34_vgpr35_vgpr36_vgpr37_vgpr38_vgpr39_vgpr40_vgpr41_vgpr42_vgpr43_vgpr44_vgpr45_vgpr46_vgpr47, $vgpr48_vgpr49_vgpr50_vgpr51_vgpr52_vgpr53_vgpr54_vgpr55_vgpr56_vgpr57_vgpr58_vgpr59_vgpr60_vgpr61_vgpr62_vgpr63, $vgpr64_vgpr65_vgpr66_vgpr67_vgpr68_vgpr69_vgpr70_vgpr71_vgpr72_vgpr73_vgpr74_vgpr75_vgpr76_vgpr77_vgpr78_vgpr79, $vgpr80_vgpr81_vgpr82_vgpr83_vgpr84_vgpr85_vgpr86_vgpr87_vgpr88_vgpr89_vgpr90_vgpr91_vgpr92_vgpr93_vgpr94_vgpr95, $vgpr96_vgpr97_vgpr98_vgpr99_vgpr100_vgpr101_vgpr102_vgpr103_vgpr104_vgpr105_vgpr106_vgpr107_vgpr108_vgpr109_vgpr110_vgpr111, $vgpr112_vgpr113_vgpr114_vgpr115_vgpr116_vgpr117_vgpr118_vgpr119_vgpr120_vgpr121_vgpr122_vgpr123_vgpr124_vgpr125_vgpr126_vgpr127, $vgpr128_vgpr129_vgpr130_vgpr131_vgpr132_vgpr133_vgpr134_vgpr135_vgpr136_vgpr137_vgpr138_vgpr139_vgpr140_vgpr141_vgpr142_vgpr143, $vgpr144_vgpr145_vgpr146_vgpr147_vgpr148_vgpr149_vgpr150_vgpr151_vgpr152_vgpr153_vgpr154_vgpr155_vgpr156_vgpr157_vgpr158_vgpr159, $vgpr160_vgpr161_vgpr162_vgpr163_vgpr164_vgpr165_vgpr166_vgpr167_vgpr168_vgpr169_vgpr170_vgpr171_vgpr172_vgpr173_vgpr174_vgpr175, $vgpr176_vgpr177_vgpr178_vgpr179_vgpr180_vgpr181_vgpr182_vgpr183_vgpr184_vgpr185_vgpr186_vgpr187_vgpr188_vgpr189_vgpr190_vgpr191, $vgpr192_vgpr193_vgpr194_vgpr195_vgpr196_vgpr197_vgpr198_vgpr199_vgpr200_vgpr201_vgpr202_vgpr203_vgpr204_vgpr205_vgpr206_vgpr207, $vgpr208_vgpr209_vgpr210_vgpr211_vgpr212_vgpr213_vgpr214_vgpr215_vgpr216_vgpr217_vgpr218_vgpr219_vgpr220_vgpr221_vgpr222_vgpr223, $vgpr224_vgpr225_vgpr226_vgpr227_vgpr228_vgpr229_vgpr230_vgpr231_vgpr232_vgpr233_vgpr234_vgpr235_vgpr236_vgpr237_vgpr238_vgpr239, $vgpr240_vgpr241_vgpr242_vgpr243_vgpr244_vgpr245_vgpr246_vgpr247, $vgpr248_vgpr249_vgpr250_vgpr251, $vgpr252_vgpr253_vgpr254_vgpr255, $vgpr2
     ; GFX9: $vgpr2 = V_WRITELANE_B32 $sgpr33, 0, undef $vgpr2
     ; GFX9: $sgpr4 = frame-setup S_ADD_U32 $sgpr32, 524224, implicit-def $scc
     ; GFX9: $sgpr33 = frame-setup S_AND_B32 killed $sgpr4, 4294443008, implicit-def $scc
     ; GFX9: $sgpr32 = frame-setup S_ADD_U32 $sgpr32, 1572864, implicit-def $scc
     ; GFX9: $vgpr0 = V_LSHRREV_B32_e64 6, $sgpr33, implicit $exec
-    ; GFX9: $sgpr4 = S_ADD_U32 $sgpr33, 524544, implicit-def $scc
-    ; GFX9: BUFFER_STORE_DWORD_OFFSET killed $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, killed $sgpr4, 0, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.3, addrspace 5)
+    ; GFX9: BUFFER_STORE_DWORD_OFFSET killed $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr33, 12, 0, 0, 0, 0, 0, implicit $exec :: (store 4 into %stack.3, addrspace 5)
     ; GFX9: $vgpr3 = V_LSHRREV_B32_e64 6, $sgpr33, implicit $exec
     ; GFX9: $vgpr3 = V_ADD_U32_e32 8192, killed $vgpr3, implicit $exec
     ; GFX9: $vgpr0 = V_OR_B32_e32 killed $vgpr3, $vgpr1, implicit $exec
     ; GFX9: $sgpr32 = frame-destroy S_SUB_U32 $sgpr32, 1572864, implicit-def $scc
     ; GFX9: $sgpr33 = V_READLANE_B32 $vgpr2, 0
-    ; GFX9: $sgpr4 = S_ADD_U32 $sgpr33, 524544, implicit-def $scc
-    ; GFX9: $vgpr3 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, killed $sgpr4, 0, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.3, addrspace 5)
+    ; GFX9: $vgpr3 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr33, 12, 0, 0, 0, 0, 0, implicit $exec :: (load 4 from %stack.3, addrspace 5)
     ; GFX9: S_ENDPGM 0, csr_amdgpu_allvgprs
     ; GFX9-FLATSCR-LABEL: name: pei_scavenge_vgpr_spill
     ; GFX9-FLATSCR: liveins: $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15, $vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31, $vgpr32_vgpr33_vgpr34_vgpr35_vgpr36_vgpr37_vgpr38_vgpr39_vgpr40_vgpr41_vgpr42_vgpr43_vgpr44_vgpr45_vgpr46_vgpr47, $vgpr48_vgpr49_vgpr50_vgpr51_vgpr52_vgpr53_vgpr54_vgpr55_vgpr56_vgpr57_vgpr58_vgpr59_vgpr60_vgpr61_vgpr62_vgpr63, $vgpr64_vgpr65_vgpr66_vgpr67_vgpr68_vgpr69_vgpr70_vgpr71_vgpr72_vgpr73_vgpr74_vgpr75_vgpr76_vgpr77_vgpr78_vgpr79, $vgpr80_vgpr81_vgpr82_vgpr83_vgpr84_vgpr85_vgpr86_vgpr87_vgpr88_vgpr89_vgpr90_vgpr91_vgpr92_vgpr93_vgpr94_vgpr95, $vgpr96_vgpr97_vgpr98_vgpr99_vgpr100_vgpr101_vgpr102_vgpr103_vgpr104_vgpr105_vgpr106_vgpr107_vgpr108_vgpr109_vgpr110_vgpr111, $vgpr112_vgpr113_vgpr114_vgpr115_vgpr116_vgpr117_vgpr118_vgpr119_vgpr120_vgpr121_vgpr122_vgpr123_vgpr124_vgpr125_vgpr126_vgpr127, $vgpr128_vgpr129_vgpr130_vgpr131_vgpr132_vgpr133_vgpr134_vgpr135_vgpr136_vgpr137_vgpr138_vgpr139_vgpr140_vgpr141_vgpr142_vgpr143, $vgpr144_vgpr145_vgpr146_vgpr147_vgpr148_vgpr149_vgpr150_vgpr151_vgpr152_vgpr153_vgpr154_vgpr155_vgpr156_vgpr157_vgpr158_vgpr159, $vgpr160_vgpr161_vgpr162_vgpr163_vgpr164_vgpr165_vgpr166_vgpr167_vgpr168_vgpr169_vgpr170_vgpr171_vgpr172_vgpr173_vgpr174_vgpr175, $vgpr176_vgpr177_vgpr178_vgpr179_vgpr180_vgpr181_vgpr182_vgpr183_vgpr184_vgpr185_vgpr186_vgpr187_vgpr188_vgpr189_vgpr190_vgpr191, $vgpr192_vgpr193_vgpr194_vgpr195_vgpr196_vgpr197_vgpr198_vgpr199_vgpr200_vgpr201_vgpr202_vgpr203_vgpr204_vgpr205_vgpr206_vgpr207, $vgpr208_vgpr209_vgpr210_vgpr211_vgpr212_vgpr213_vgpr214_vgpr215_vgpr216_vgpr217_vgpr218_vgpr219_vgpr220_vgpr221_vgpr222_vgpr223, $vgpr224_vgpr225_vgpr226_vgpr227_vgpr228_vgpr229_vgpr230_vgpr231_vgpr232_vgpr233_vgpr234_vgpr235_vgpr236_vgpr237_vgpr238_vgpr239, $vgpr240_vgpr241_vgpr242_vgpr243_vgpr244_vgpr245_vgpr246_vgpr247, $vgpr248_vgpr249_vgpr250_vgpr251, $vgpr252_vgpr253_vgpr254_vgpr255, $vgpr2
     ; GFX9-FLATSCR: $vgpr2 = V_WRITELANE_B32 $sgpr33, 0, undef $vgpr2
     ; GFX9-FLATSCR: $sgpr4 = frame-setup S_ADD_U32 $sgpr32, 8191, implicit-def $scc
     ; GFX9-FLATSCR: $sgpr33 = frame-setup S_AND_B32 killed $sgpr4, 4294959104, implicit-def $scc
     ; GFX9-FLATSCR: $sgpr32 = frame-setup S_ADD_U32 $sgpr32, 24576, implicit-def $scc
     ; GFX9-FLATSCR: $vgpr0 = V_MOV_B32_e32 $sgpr33, implicit $exec

llvm/test/CodeGen/RISCV/out-of-reach-emergency-slot.mir

+# NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+# REQUIRES: asserts
+# RUN: llc -mtriple riscv64 -start-before=prologepilog -o - \
+# RUN:     -verify-machineinstrs %s | FileCheck %s
+#
+# RUN: llc -mtriple riscv64 -start-before=prologepilog -o /dev/null \
+# RUN:     -debug-only=prologepilog -verify-machineinstrs %s 2>&1 \
+# RUN:     | FileCheck --check-prefix=DEBUG %s
+#
+# DEBUG: Adjusting emergency spill slots!
+# DEBUG: Adjusting offset of emergency spill slot #4 from -4112 to -8192
+
+--- |
+  ; ModuleID = 'reduced.ll'
+  source_filename = "frame_layout-1253b1.cpp"
+  target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128"
+  target triple = "riscv64"
+
+  ; Function Attrs: nounwind
+  define weak_odr dso_local void @foo(i8* %ay) nounwind {
+  ; CHECK-LABEL: foo:
+  ; CHECK:       # %bb.0: # %entry
+  ; CHECK-NEXT:    addi sp, sp, -2032
+  ; CHECK-NEXT:    sd ra, 2024(sp) # 8-byte Folded Spill
+  ; CHECK-NEXT:    sd s0, 2016(sp) # 8-byte Folded Spill
+  ; CHECK-NEXT:    addi s0, sp, 2032
+  ; CHECK-NEXT:    sd a1, 0(sp)
+  ; CHECK-NEXT:    lui a1, 2
+  ; CHECK-NEXT:    addiw a1, a1, -2032
+  ; CHECK-NEXT:    sub sp, sp, a1
+  ; CHECK-NEXT:    srli a1, sp, 12
+  ; CHECK-NEXT:    slli sp, a1, 12
+  ; CHECK-NEXT:    lui a1, 1
+  ; CHECK-NEXT:    addiw a1, a1, -8
+  ; CHECK-NEXT:    add a1, sp, a1
+  ; CHECK-NEXT:    sd a0, 0(a1)
+  ; CHECK-NEXT:    ld a1, 0(sp)
+  ; CHECK-NEXT:    call foo@plt
+  ; CHECK-NEXT:    lui a0, 2
+  ; CHECK-NEXT:    sub sp, s0, a0
+  ; CHECK-NEXT:    lui a0, 2
+  ; CHECK-NEXT:    addiw a0, a0, -2032
+  ; CHECK-NEXT:    add sp, sp, a0
+  ; CHECK-NEXT:    ld s0, 2016(sp) # 8-byte Folded Reload
+  ; CHECK-NEXT:    ld ra, 2024(sp) # 8-byte Folded Reload
+  ; CHECK-NEXT:    addi sp, sp, 2032
+  ; CHECK-NEXT:    ret
+  entry:
+    ret void
+  }
+
+
+...
+---
+name:            foo
+alignment:       2
+tracksRegLiveness: false
+frameInfo:
+  maxAlignment:    4096
+stack:
+  - { id: 0, size: 8, alignment: 4096 }
+  - { id: 1, type: spill-slot, size: 8, alignment: 8 }
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    liveins: $x1, $x5, $x6, $x7, $x10, $x11, $x12, $x13, $x14, $x15, $x16, $x17, $x28, $x29, $x30, $x31
+
+    ; This is to store something to the (non-emergency) spill slot %stack.1.
+    SD $x10, %stack.1, 0
+    ; This is here just to make all the eligible registers live at this point.
+    ; This way when we replace the frame index %stack.1 with its actual address
+    ; we have to allocate a virtual register to compute it.
+    ; A later run of the the register scavenger won't find an available register
+    ; either so it will have to spill one to the emergency spill slot.
+    PseudoCALL target-flags(riscv-plt) @foo, csr_ilp32_lp64, implicit-def $x1, implicit-def $x2, implicit $x1, implicit $x5, implicit $x6, implicit $x7, implicit $x10, implicit $x11, implicit $x12, implicit $x13, implicit $x14, implicit $x15, implicit $x16, implicit $x17, implicit $x28, implicit $x29, implicit $x30, implicit $x31
+    PseudoRET
+
+...

llvm/test/CodeGen/Thumb/emergency-spill-slot.ll

 ; CHECK-NEXT:    lsrs r4, r4, #4
 ; CHECK-NEXT:    lsls r4, r4, #4
 ; CHECK-NEXT:    mov sp, r4
 ; CHECK-NEXT:    add r0, sp, #16
 ; CHECK-NEXT:    adds r1, r7, #7
 ; CHECK-NEXT:    adds r1, #1
 ; CHECK-NEXT:    @APP
 ; CHECK-NEXT:    @NO_APP
-; CHECK-NEXT:    str r0, [sp, #12]
+; CHECK-NEXT:    str r0, [sp]
 ; CHECK-NEXT:    ldr r0, .LCPI5_0
 ; CHECK-NEXT:    str r5, [r0, r7]
-; CHECK-NEXT:    ldr r0, [sp, #12]
+; CHECK-NEXT:    ldr r0, [sp]
 ; CHECK-NEXT:    @APP
 ; CHECK-NEXT:    @NO_APP
 ; CHECK-NEXT:    subs r4, r7, #7
 ; CHECK-NEXT:    subs r4, #5
 ; CHECK-NEXT:    mov sp, r4
 ; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1: