Prevent register coalescing in functions whith setjmp

[dnsampaio]

Summary

In the the given example, a stack slot pointer is merged
between a setjmp and longjmp. This pointer is spilled,
so it does not get correctly restored, addinga undefined
behaviour where it shouldn't.

Change-Id: I60ec010844f2a24ce01ceccf12eb5eba5ab94abb

[efriedma]

I'm not sure I understand the problem; can you give a C testcase?

In general, stack coloring should be okay, as long as we aren't truncating the lifetime of a variable. If a variable which is live at the point setjmp is called is still live at the point of the longjmp, it shouldn't overlap with anything relevant. If the variable isn't live at the point of the longjmp, we don't have any obligation to preserve the value according to the standard (from C99: "All accessible objects have values [...]").

If we truncate the lifetime of a variable, that could be a problem: a variable which the standard requires to be live would be dead, and potentially overlap with something relevant. But I'm not sure how that would happen in practice if we still have an alloca going into isel.

[dnsampaio]

Hi @efriedma,
thanks for looking into this.

I created https://bugs.llvm.org/show_bug.cgi?id=45489 to tackle this bug, with further details.

Bare with me, but the C++ code for this is like this:

file043220.h:#include <setjmp.h>
file043220.h:struct S38 {
file043220.h:  long long M0;
file043220.h:  bool M1;
file043220.h:};
file043220.h:struct S37 {
file043220.h:  char M1;
file043220.h:  S38 M2;
file043220.h:  long long M3[4];
file043220.h:  int M0a;
file043220.h:  unsigned long long M1a;
file043220.h:  int M6;
file043220.h:  S37() : M2(), M3(), M6()  {}
file043220.h:};
file043220.h:struct S23 {
file043220.h:  long long M0[2];
file043220.h:  __fp16 M1;
file043220.h:};
file043220.h:struct S18  {
file043220.h:  int Bx;
file043220.h:  long long BM0;
file043220.h:  long long BM1;
file043220.h:  long long BM2;
file043220.h:  long M0;
file043220.h:  S23 M3;
file043220.h:  int M4;
file043220.h:  S18() : BM0(0), M0(42), M3(), M4() {}
file043220.h:};
file043220.h:void foo(jmp_buf incoming_jb, S18 P0) __attribute__((noreturn));
file043220.h:void bar(jmp_buf incoming_jb, S37 P0) __attribute__((noreturn));

nolto.cpp:#include <assert.h>
nolto.cpp:#include <stdlib.h>
nolto.cpp:#include "file043220.h"
nolto.cpp:void foo(jmp_buf incoming_jb, S18 P0)  {
nolto.cpp:  assert(P0.M0 == 42);
nolto.cpp:  exit(0);
nolto.cpp:}
nolto.cpp:void bar(jmp_buf incoming_jb, S37 P0)  {
nolto.cpp:  longjmp(incoming_jb, 1);
nolto.cpp:}

lto.cpp:#include <assert.h>
lto.cpp:#include <alloca.h>
lto.cpp:#include "file043220.h"
lto.cpp:signed int main(void) {
lto.cpp:  {
lto.cpp:    S37 P0;
lto.cpp:    jmp_buf jb1;
lto.cpp:    if (!setjmp(jb1)) {
lto.cpp:      bar(jb1, P0);
lto.cpp:    }
lto.cpp:  }
lto.cpp:  {
lto.cpp:    char *V6 = (char *)alloca(128);
lto.cpp:    __asm volatile("" : : "r"(V6));
lto.cpp:  }
lto.cpp:  {
lto.cpp:    __attribute((aligned(16))) char V7[16];
lto.cpp:    __asm volatile("" : : "r"(&V7[0]));
lto.cpp:  }
lto.cpp:  __asm volatile("" : : "r"(__builtin_return_address(0)));
lto.cpp:  S18 P1;
lto.cpp:  jmp_buf jb2;
lto.cpp:  if (!setjmp(jb2)) {
lto.cpp:    foo(jb2, P1);
lto.cpp:  }
lto.cpp:  return 0;
lto.cpp:}

From my understanding, when we join variables with disjoint aliveness, one from before and the other after the long, changing the "pointer" to this new joined variable is dangerous. If it is spilled, it won't be restored by the longjmp.
If I understand correctly, in the example, S37 P0 and S18 P1 are selected to share the same stack slot. As the function bar is marked as noreturn, when performing bar(jb1, P0);, the copy of P0 increments the pointer by 72.
bar then calls longjmp, that does not restore the pointer value of P0 as it is spilled. When doing the copy of P1 for foo(jb2, P1);, the read access of P1 is misaligned by 72.

However, the only reason why we see the bug is because the initialization of P1 writes to the correct address, just because it does not write to the start of the container, and after Local Stack Slot Allocation it uses a different "pointer". Thanks to that we hit the assert failure for P1.M0 == 42 inside foo.

[efriedma]

From my understanding, when we join variables with disjoint aliveness, one from before and the other after the long, changing the "pointer" to this new joined variable is dangerous. If it is spilled, it won't be restored by the longjmp.

You're saying the problem is the address of the local variable, not its contents? In that case, this patch is almost certainly covering up the real problem; an equivalent "merging" transform can be done at the source level. And it's probably feasible to hit related issues in other ways.

My best guess based on your description is that greedy regalloc isn't being conservative enough about its use of spill slots in functions that call setjmp.

[rnk]

I suspect Eli is right, if this issue has to do with spill slots, it has more to do with the register allocator that does the spilling. I think we need to know more about that to take action. We can't just sprinkle "don't optimize because setjmp" throughout the compiler without a more principled reason.

[dnsampaio]

Hi @rnk and @efriedma, indeed my initial thoughts were as well the register allocator, specially because using the pbqp allocator there are no spills for this example so it does not fail.
From what I see in the code it seems that none of the register allocators know anything about setjmp. There's an ancient llvm-dev post confirming it: https://lists.llvm.org/pipermail/llvm-dev/2011-October/043731.html

So the doubt I have is, what is the correct solution for it? I guess we can't block a variable from being spilled if it's life-range crosses the setjmp function. From my understanding of setjmp/longjmp, for any of such variables, when evicted, needs two spill slots. Spill slot 1 (ss1) holds the current value of the variable. Spill slot 2 (ss2) holds the variable value right before the setjmp, copied from ss1. Right after setfmp we always copy the ss2 to ss1 restoring it. The issue I see with doing it is that it will increase the register pressure at this point of the program by one, as we need it for doing the copying, unless the variable is also live in register at this point, which would avoid the second spill slot all together.

A possible optimization would be finding a lower register pressure point that post-dominates the last may-write to the variable before the setjmp function call and that dominates the function call, if doing so would avoid a second spill.
Does that sound correct to you?

[efriedma]

From what I see in the code it seems that none of the register allocators know anything about setjmp

StackSlotColoring checks MF.exposesReturnsTwice(); that's related to register allocation. But yes, the allocators themselves don't have any checks.

Going into the register allocator, distinct SSA values should be in distinct registers. We can take advantage of this to preserve our invariants. If each of those distinct SSA values gets its own spill slot, that should be enough to avoid this class of issue: after the setjmp, the value would always be the same value we computed before the setjmp. This is basically the point of the check in StackSlotColoring.

But I guess the check there isn't enough in some cases. Maybe the problem is register coalescing?

(This is basically the same conclusion as https://lists.llvm.org/pipermail/llvm-dev/2011-October/043734.html .)

[dnsampaio]

Hi @efriedma and @rnk ,

indeed following the log it is register coalescing that finally merges the two distinct stack pointers values into a single virtual register. For fixing it I see 3 options, but perhaps you can come with a better idea.
1 ) A quick fix for that would be simply disable register coalescing if the function exposesReturnsTwice. It is over-conservative, although I wouldn't consider that performance is so critical in such functions anyway due backing-up all register.

2. The most generic solution I can think of is, after performing spills, to verify every spilled value if they have their value altered between a call to setjmp and a possible call to longjmp (we need to trace the buffer argument).

If it is, then we need to emit extra spill slot and code for performing saving the input-value at the the call to setjmp and restore the value just after the call.

3. Do not allow to merge instructions if one is alive before the setjmp call and the other is alive after. As well, if one variable is alive both before and after a setjmp call and altered before any potential call to longjmp, that variable also

can't be merged. Although I think this one could be tricky, as we need to follow where the buffer argument can escape of the function.

Do you have any opinion about this, or perhaps a better idea?
Regards

[efriedma]

The simplest thing is just disabling coalescing.

The next simplest thing to do would be to specifically disable coalescing registers live across setjmp calls (not worrying about any corresponding longjmp calls). This is probably straightforward to query: register allocation already computes the necessary liveness information, I think. But really, it's probably not worth the effort; disabling coalescing altogether isn't that big of a performance hit.

[dnsampaio]

Do not perform register coalescing

[dnsampaio]

Fixed x86 test

[efriedma]

Can you fix the MIR testcase to check the register coalescing pass itself, rather than the whole pipeline after stack coloring?

[dnsampaio]

Now testing only simple-register-coalescing

[dnsampaio]

Ping

[rnk]

llvm/test/CodeGen/ARM/no-register-coalescing-in-returnsTwice.mir

18

I can't read ARM assembly well enough to tell if this is a good test or to suggest how to make it better, so I'll have to ask Eli to review.

llvm/test/CodeGen/X86/speculative-load-hardening-call-and-ret.ll

376

Seems like we don't want to lose these labels. See rGe3ea164659ff37cb4db623c33de880e91aa29ebb. Please regenerate with --no_x86_scrub_rip.

For a while, I have wanted update_llc_test checks to store this option in the comment at the beginning of the file and then read the options back out when regenerating test cases, but it has not come to pass.

[efriedma]

llvm/test/CodeGen/ARM/no-register-coalescing-in-returnsTwice.mir

18

It should be possible to construct a shorter testcase by artificially forcing up the register pressure using inline asm clobbers.

See also http://llvm.org/docs/MIRLangRef.html?highlight=filecheck#simplifying-mir-files .

[dnsampaio]

Addressed requests

llvm/test/CodeGen/ARM/no-register-coalescing-in-returnsTwice.mir

18

Indeed replaced most of the code in my original reproducer by the inline asm clobbering r0 - r14 between the two function calls still gives me the same error.
And I simplified the mir as much as possible.

[efriedma]

LGTM

llvm/lib/CodeGen/RegisterCoalescer.cpp

void RegisterCoalescer::releaseMemory() {
   ErasedInstrs.clear();
   WorkList.clear();
   DeadDefs.clear();
   InflateRegs.clear();
   LargeLIVisitCounter.clear();
 }
 
 bool RegisterCoalescer::runOnMachineFunction(MachineFunction &fn) {
+  LLVM_DEBUG(dbgs() << "********** SIMPLE REGISTER COALESCING **********\n"
+                    << "********** Function: " << fn.getName() << '\n');
+
+  // Variables changed between a setjmp and a longjump can have undefined value
+  // after the longjmp. This behaviour can be observed if such a variable is
+  // spilled, so longjmp won't restore the value in the spill slot.
+  // RegisterCoalescer should not run in functions with a setjmp to avoid
+  // merging such undefined variables with predictable ones.
+  //
+  // TODO: Could specifically disable coalescing registers live across setjmp
+  // calls
+  if (fn.exposesReturnsTwice()) {
+    LLVM_DEBUG(
+        dbgs() << "* Skipped as it exposes funcions that returns twice.\n");
+    return false;
+  }
+
   MF = &fn;
   MRI = &fn.getRegInfo();
   const TargetSubtargetInfo &STI = fn.getSubtarget();
   TRI = STI.getRegisterInfo();
   TII = STI.getInstrInfo();
   LIS = &getAnalysis<LiveIntervals>();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
   Loops = &getAnalysis<MachineLoopInfo>();
   if (EnableGlobalCopies == cl::BOU_UNSET)
     JoinGlobalCopies = STI.enableJoinGlobalCopies();
   else
     JoinGlobalCopies = (EnableGlobalCopies == cl::BOU_TRUE);
 
   // The MachineScheduler does not currently require JoinSplitEdges. This will
   // either be enabled unconditionally or replaced by a more general live range
   // splitting optimization.
   JoinSplitEdges = EnableJoinSplits;
 
-  LLVM_DEBUG(dbgs() << "********** SIMPLE REGISTER COALESCING **********\n"
-                    << "********** Function: " << MF->getName() << '\n');
-
   if (VerifyCoalescing)
     MF->verify(this, "Before register coalescing");
 
   DbgVRegToValues.clear();
   DbgMergedVRegNums.clear();
   buildVRegToDbgValueMap(fn);
 
   RegClassInfo.runOnMachineFunction(fn);

llvm/test/CodeGen/ARM/no-register-coalescing-in-returnsTwice.mir

+# RUN: llc --run-pass=simple-register-coalescing -o - %s | FileCheck %s
+# pr45489
+# Coalescing variables across a setjmp call can add a undefined
+# variable value when longjmp if such variables are spilled and
+# altered between the setjmp and longjmp.
+
+# This file tests a very particular case for
+# no coalescing stack pointers across the
+# setjmp call.
+# CHECK: %[[R1:[0-9]+]]:gpr = ADDri %stack.0.P0, 0, 14
+
+# Stack pointer
+# CHECK: %[[R2:[0-9]+]]:gpr = nuw ADDri %[[R1]], 8, 14
+# CHECK: @setjmp
+
+# Not changed between setjmp and bar(longjmp)
+# CHECK-NOT: %{{[0-9]+}}:dpr, %[[R2]]:gpr = VLD1d32wb_fixed %[[R2]], 0,
+# CHECK: BL @_Z3barPx3S37{{.*}}
+# CHECK: %[[R3:[0-9]+]]:gpr = COPY %[[R2]]
+
+# Used after bar
+# CHECK-NOT: VLD1d32wb_fixed %[[R2]]
+# CHECK: VLD1d32wb_fixed %[[R3]]
+--- |
+  target triple = "armv8-arm-none-eabi"
+
+  %"class.std::__1::ios_base::Init" = type { i8 }
+  %struct.S37 = type <{ i8, [7 x i8], %struct.S38, [2 x %"class.std::__1::complex"], float, [4 x i8], i64, i32, [4 x i8] }>
+  %struct.S38 = type { double, i8 }
+  %"class.std::__1::complex" = type { double, double }
+  %struct.S18 = type <{ i32, [4 x i8], double, double, double, i32, [4 x i8], %struct.S23, i32, [4 x i8] }>
+  %struct.S23 = type { [2 x double], half }
+  define i32 @main() {
+  entry:
+    %P0 = alloca %struct.S37, align 8
+    %0 = bitcast %struct.S37* %P0 to %struct.S18*
+    %jb1 = alloca [20 x i64], align 8
+    %P1 = alloca %struct.S18, align 8
+    %jb2 = alloca [20 x i64], align 8
+    %1 = bitcast %struct.S37* %P0 to i8*
+    %M2.i = getelementptr inbounds %struct.S37, %struct.S37* %P0, i32 0, i32 2
+    %2 = bitcast %struct.S38* %M2.i to i8*
+    call void @llvm.memset.p0i8.i64(i8* nonnull align 8 dereferenceable(48) %2, i8 0, i64 48, i1 false)
+    %M6.i = getelementptr inbounds %struct.S37, %struct.S37* %P0, i32 0, i32 7
+    store i32 0, i32* %M6.i, align 8
+    %3 = bitcast [20 x i64]* %jb1 to i8*
+    %arraydecay1 = bitcast [20 x i64]* %jb1 to i64*
+    %call1 = call i32 @setjmp(i64* nonnull %arraydecay1)
+    %tobool = icmp eq i32 %call1, 0
+    br i1 %tobool, label %if.then, label %if.end
+  if.then:                                          ; preds = %entry
+    %4 = bitcast [20 x i64]* %jb1 to i64*
+    call void (i64*, %struct.S37*, ...) @_Z3barPx3S37z(i64* nonnull %4, %struct.S37* nonnull byval(%struct.S37) align 8 %P0)
+    unreachable
+  if.end:                                           ; preds = %entry
+    %5 = bitcast [20 x i64]* %jb1 to i8*
+    %6 = bitcast %struct.S37* %P0 to i8*
+    call void asm sideeffect "", "~{r0},~{r1},~{r2},~{r3},~{r4},~{r5},~{r6},~{r7},~{r8},~{r9},~{r10},~{r11},~{r12},~{sp},~{lr}"()
+    %7 = bitcast %struct.S18* %0 to i8*
+    %BM0.i = getelementptr inbounds %struct.S18, %struct.S18* %0, i32 0, i32 2
+    store double 0.000000e+00, double* %BM0.i, align 8
+    %M0.i = getelementptr inbounds %struct.S18, %struct.S18* %0, i32 0, i32 5
+    store i32 42, i32* %M0.i, align 8
+    %M3.i = getelementptr inbounds %struct.S18, %struct.S18* %0, i32 0, i32 7
+    %8 = bitcast %struct.S23* %M3.i to i8*
+    call void @llvm.memset.p0i8.i64(i8* nonnull align 8 dereferenceable(28) %8, i8 0, i64 28, i1 false)
+    %9 = bitcast [20 x i64]* %jb1 to i8*
+    %arraydecay42 = bitcast [20 x i64]* %jb1 to i64*
+    %call5 = call i32 @setjmp(i64* nonnull %arraydecay42)
+    %tobool6 = icmp eq i32 %call5, 0
+    br i1 %tobool6, label %if.then7, label %if.end10
+  if.then7:                                         ; preds = %if.end
+    %10 = bitcast [20 x i64]* %jb1 to i64*
+    call void (i64*, %struct.S18*, ...) @_Z3fooPx3S18z(i64* nonnull %10, %struct.S18* nonnull byval(%struct.S18) align 8 %0)
+    unreachable
+  if.end10:                                         ; preds = %if.end
+    %11 = bitcast [20 x i64]* %jb1 to i8*
+    %12 = bitcast %struct.S18* %0 to i8*
+    ret i32 0
+  }
+  declare i32 @setjmp(i64*)
+  declare void @_Z3barPx3S37z(i64*, %struct.S37* byval(%struct.S37) align 8, ...)
+  declare void @_Z3fooPx3S18z(i64*, %struct.S18* byval(%struct.S18) align 8, ...)
+  declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+...
+---
+name:            main
+exposesReturnsTwice: true
+stack:
+  - { id: 0, name: P0, size: 80, alignment: 8, local-offset: -80 }
+  - { id: 1, name: jb1, size: 160, alignment: 8, local-offset: -240 }
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    successors: %bb.1(0x00000001), %bb.4(0x7fffffff)
+    %0:qpr = VMOVv4i32 0, 14, $noreg
+    %1:gpr = ADDri %stack.0.P0, 0, 14, $noreg, $noreg
+    %2:gpr = ADDri %1, 40, 14, $noreg, $noreg
+    VST1q64 %2, 0, %0, 14, $noreg
+    %3:gpr = ADDri %1, 24, 14, $noreg, $noreg
+    VST1q64 killed %3, 0, %0, 14, $noreg
+    %4:gpr = nuw ADDri %1, 8, 14, $noreg, $noreg
+    VST1q64 %4, 0, %0, 14, $noreg
+    %5:gpr = MOVi 0, 14, $noreg, $noreg
+    STRi12 %5, %stack.0.P0, 72, 14, $noreg
+    ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %6:gpr = ADDri %stack.1.jb1, 0, 14, $noreg, $noreg
+    $r0 = COPY killed %6
+    BL @setjmp, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit-def $sp, implicit-def $r0
+    ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %7:gpr = COPY killed $r0
+    CMPri killed %7, 0, 14, $noreg, implicit-def $cpsr
+    Bcc %bb.4, 1, killed $cpsr
+    B %bb.1
+  bb.1:
+    ADJCALLSTACKDOWN 72, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %24:gpr = LDRi12 %stack.0.P0, 0, 14, $noreg
+    %25:gpr = LDRi12 %stack.0.P0, 4, 14, $noreg
+    %27:gpr = COPY $sp
+    %29:gpr = MOVi16 72, 14, $noreg
+    %61:gpr = COPY killed %29
+    %62:gpr = COPY killed %4
+    %63:gpr = COPY killed %27
+  bb.2:
+    %35:gpr = COPY killed %63
+    %33:gpr = COPY killed %62
+    %31:gpr = COPY killed %61
+    %32:gpr = COPY killed %33
+    %36:dpr, %32:gpr = VLD1d32wb_fixed %32, 0, 14, $noreg
+    %34:gpr = COPY killed %35
+    %34:gpr = VST1d32wb_fixed %34, 0, killed %36, 14, $noreg
+    %30:gpr = SUBri killed %31, 8, 14, $noreg, def $cpsr
+    %61:gpr = COPY killed %30
+    %62:gpr = COPY killed %32
+    %63:gpr = COPY killed %34
+    Bcc %bb.2, 1, killed $cpsr
+  bb.3:
+    successors:
+    %28:gpr = ADDri %stack.1.jb1, 0, 14, $noreg, $noreg
+    $r0 = COPY killed %28
+    $r2 = COPY killed %24
+    $r3 = COPY killed %25
+    BL @_Z3barPx3S37z, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit killed $r2, implicit killed $r3, implicit-def $sp
+    ADJCALLSTACKUP 72, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+  bb.4:
+    successors: %bb.5(0x00000001), %bb.6(0x7fffffff)
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0, 12, implicit-def dead early-clobber $r1, 12, implicit-def dead early-clobber $r2, 12, implicit-def dead early-clobber $r3, 12, implicit-def dead early-clobber $r4, 12, implicit-def dead early-clobber $r5, 12, implicit-def dead early-clobber $r6, 12, implicit-def dead early-clobber $r7, 12, implicit-def dead early-clobber $r8, 12, implicit-def dead early-clobber $r9, 12, implicit-def dead early-clobber $r10, 12, implicit-def dead early-clobber $r11, 12, implicit-def dead early-clobber $r12, 12, implicit-def early-clobber $sp, 12, implicit-def dead early-clobber $lr
+    VST1q64 killed %2, 0, %0, 14, $noreg
+    %11:gpr = ADDri killed %1, 52, 14, $noreg, $noreg
+    VST1q32 killed %11, 0, killed %0, 14, $noreg
+    STRi12 %5, %stack.0.P0, 12, 14, $noreg
+    STRi12 killed %5, %stack.0.P0, 8, 14, $noreg
+    %13:gpr = MOVi 42, 14, $noreg, $noreg
+    STRi12 killed %13, %stack.0.P0, 32, 14, $noreg
+    ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %14:gpr = ADDri %stack.1.jb1, 0, 14, $noreg, $noreg
+    $r0 = COPY killed %14
+    BL @setjmp, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit-def $sp, implicit-def $r0
+    ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %15:gpr = COPY killed $r0
+    CMPri killed %15, 0, 14, $noreg, implicit-def $cpsr
+    Bcc %bb.6, 1, killed $cpsr
+    B %bb.5
+  bb.5:
+    successors:
+    ADJCALLSTACKDOWN 64, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+    %18:gpr = LDRi12 %stack.0.P0, 0, 14, $noreg
+    %19:gpr = LDRi12 %stack.0.P0, 4, 14, $noreg
+    %21:gpr = COPY $sp
+    %37:gpr = COPY killed %4
+    %39:dpr, %37:gpr = VLD1d32wb_fixed %37, 0, 14, $noreg
+    %38:gpr = COPY killed %21
+    %38:gpr = VST1d32wb_fixed %38, 0, killed %39, 14, $noreg
+    %40:gpr = COPY killed %37
+    %42:dpr, %40:gpr = VLD1d32wb_fixed %40, 0, 14, $noreg
+    %41:gpr = COPY killed %38
+    %41:gpr = VST1d32wb_fixed %41, 0, killed %42, 14, $noreg
+    %43:gpr = COPY killed %40
+    %45:dpr, %43:gpr = VLD1d32wb_fixed %43, 0, 14, $noreg
+    %44:gpr = COPY killed %41
+    %44:gpr = VST1d32wb_fixed %44, 0, killed %45, 14, $noreg
+    %46:gpr = COPY killed %43
+    %48:dpr, %46:gpr = VLD1d32wb_fixed %46, 0, 14, $noreg
+    %47:gpr = COPY killed %44
+    %47:gpr = VST1d32wb_fixed %47, 0, killed %48, 14, $noreg
+    %49:gpr = COPY killed %46
+    %51:dpr, %49:gpr = VLD1d32wb_fixed %49, 0, 14, $noreg
+    %50:gpr = COPY killed %47
+    %50:gpr = VST1d32wb_fixed %50, 0, killed %51, 14, $noreg
+    %52:gpr = COPY killed %49
+    %54:dpr, %52:gpr = VLD1d32wb_fixed %52, 0, 14, $noreg
+    %53:gpr = COPY killed %50
+    %53:gpr = VST1d32wb_fixed %53, 0, killed %54, 14, $noreg
+    %55:gpr = COPY killed %52
+    %57:dpr, %55:gpr = VLD1d32wb_fixed %55, 0, 14, $noreg
+    %56:gpr = COPY killed %53
+    %56:gpr = VST1d32wb_fixed %56, 0, killed %57, 14, $noreg
+    %58:gpr = COPY killed %55
+    %60:dpr, dead %58:gpr = VLD1d32wb_fixed %58, 0, 14, $noreg
+    %59:gpr = COPY killed %56
+    dead %59:gpr = VST1d32wb_fixed %59, 0, killed %60, 14, $noreg
+    %22:gpr = ADDri %stack.1.jb1, 0, 14, $noreg, $noreg
+    $r0 = COPY killed %22
+    $r2 = COPY killed %18
+    $r3 = COPY killed %19
+    BL @_Z3fooPx3S18z, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit killed $r2, implicit killed $r3, implicit-def $sp
+    ADJCALLSTACKUP 64, 0, 14, $noreg, implicit-def dead $sp, implicit $sp
+  bb.6:
+    %16:gpr = MOVi 0, 14, $noreg, $noreg
+    $r0 = COPY killed %16
+    BX_RET 14, $noreg, implicit killed $r0
+...

llvm/test/CodeGen/X86/speculative-load-hardening-call-and-ret.ll

 declare i32 @__sigsetjmp(i8* %foo, i8* %bar, i32 %baz) returns_twice
 
 define i32 @test_call_setjmp(i32 *%ptr) nounwind {
 ; X64-NOPIC-LABEL: test_call_setjmp:
 ; X64-NOPIC:       # %bb.0: # %entry
 ; X64-NOPIC-NEXT:    pushq %rbp
 ; X64-NOPIC-NEXT:    pushq %r15
 ; X64-NOPIC-NEXT:    pushq %r14
+; X64-NOPIC-NEXT:    pushq %r13
 ; X64-NOPIC-NEXT:    pushq %r12
 ; X64-NOPIC-NEXT:    pushq %rbx
-; X64-NOPIC-NEXT:    subq $16, %rsp
+; X64-NOPIC-NEXT:    subq $24, %rsp
 ; X64-NOPIC-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-NEXT:    movq %rdi, %rbx
 ; X64-NOPIC-NEXT:    movq $-1, %r15
 ; X64-NOPIC-NEXT:    sarq $63, %rax
-; X64-NOPIC-NEXT:    movq %rsp, %r14
+; X64-NOPIC-NEXT:    leaq {{[0-9]+}}(%rsp), %r14
 ; X64-NOPIC-NEXT:    shlq $47, %rax
 ; X64-NOPIC-NEXT:    movq %r14, %rdi
 ; X64-NOPIC-NEXT:    orq %rax, %rsp
 ; X64-NOPIC-NEXT:    movq $.Lslh_ret_addr4, %rbp
 ; X64-NOPIC-NEXT:    callq setjmp
 ; X64-NOPIC-NEXT:  .Lslh_ret_addr4:
 ; X64-NOPIC-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-NEXT:    sarq $63, %rax
 ; X64-NOPIC-NEXT:    cmpq $.Lslh_ret_addr4, %rbp
 ; X64-NOPIC-NEXT:    cmovneq %r15, %rax
 ; X64-NOPIC-NEXT:    movl (%rbx), %ebp
+; X64-NOPIC-NEXT:    movl $42, %r12d
 ; X64-NOPIC-NEXT:    shlq $47, %rax
 ; X64-NOPIC-NEXT:    movq %r14, %rdi
-; X64-NOPIC-NEXT:    movl $42, %esi
+; X64-NOPIC-NEXT:    movl %r12d, %esi
 ; X64-NOPIC-NEXT:    orq %rax, %rsp
-; X64-NOPIC-NEXT:    movq $.Lslh_ret_addr5, %r12
+; X64-NOPIC-NEXT:    movq $.Lslh_ret_addr5, %r13
 ; X64-NOPIC-NEXT:    callq sigsetjmp
 ; X64-NOPIC-NEXT:  .Lslh_ret_addr5:
 ; X64-NOPIC-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-NEXT:    sarq $63, %rax
-; X64-NOPIC-NEXT:    cmpq $.Lslh_ret_addr5, %r12
+; X64-NOPIC-NEXT:    cmpq $.Lslh_ret_addr5, %r13
 ; X64-NOPIC-NEXT:    cmovneq %r15, %rax
 ; X64-NOPIC-NEXT:    addl (%rbx), %ebp
 ; X64-NOPIC-NEXT:    shlq $47, %rax
 ; X64-NOPIC-NEXT:    movq %r14, %rdi
 ; X64-NOPIC-NEXT:    movq %r14, %rsi
-; X64-NOPIC-NEXT:    movl $42, %edx
+; X64-NOPIC-NEXT:    movl %r12d, %edx
 ; X64-NOPIC-NEXT:    orq %rax, %rsp
 ; X64-NOPIC-NEXT:    movq $.Lslh_ret_addr6, %r14
 ; X64-NOPIC-NEXT:    callq __sigsetjmp
 ; X64-NOPIC-NEXT:  .Lslh_ret_addr6:
-; X64-NOPIC-NEXT:    movq %rsp, %rcx
-; X64-NOPIC-NEXT:    sarq $63, %rcx
+; X64-NOPIC-NEXT:    movq %rsp, %rax
+; X64-NOPIC-NEXT:    sarq $63, %rax
 ; X64-NOPIC-NEXT:    cmpq $.Lslh_ret_addr6, %r14
+; X64-NOPIC-NEXT:    movq %rax, %rcx
 ; X64-NOPIC-NEXT:    cmovneq %r15, %rcx
 ; X64-NOPIC-NEXT:    addl (%rbx), %ebp
-; X64-NOPIC-NEXT:    orl %ecx, %ebp
-; X64-NOPIC-NEXT:    shlq $47, %rcx
 ; X64-NOPIC-NEXT:    movl %ebp, %eax
+; X64-NOPIC-NEXT:    orl %ecx, %eax
+; X64-NOPIC-NEXT:    shlq $47, %rcx
 ; X64-NOPIC-NEXT:    orq %rcx, %rsp
-; X64-NOPIC-NEXT:    addq $16, %rsp
+; X64-NOPIC-NEXT:    addq $24, %rsp
 ; X64-NOPIC-NEXT:    popq %rbx
 ; X64-NOPIC-NEXT:    popq %r12
+; X64-NOPIC-NEXT:    popq %r13
 ; X64-NOPIC-NEXT:    popq %r14
 ; X64-NOPIC-NEXT:    popq %r15
 ; X64-NOPIC-NEXT:    popq %rbp
 ; X64-NOPIC-NEXT:    retq
 ;
 ; X64-NOPIC-MCM-LABEL: test_call_setjmp:
 ; X64-NOPIC-MCM:       # %bb.0: # %entry
 ; X64-NOPIC-MCM-NEXT:    pushq %rbp
 ; X64-NOPIC-MCM-NEXT:    pushq %r15
 ; X64-NOPIC-MCM-NEXT:    pushq %r14
+; X64-NOPIC-MCM-NEXT:    pushq %r13
 ; X64-NOPIC-MCM-NEXT:    pushq %r12
 ; X64-NOPIC-MCM-NEXT:    pushq %rbx
-; X64-NOPIC-MCM-NEXT:    subq $16, %rsp
+; X64-NOPIC-MCM-NEXT:    subq $24, %rsp
 ; X64-NOPIC-MCM-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-MCM-NEXT:    movq %rdi, %rbx
 ; X64-NOPIC-MCM-NEXT:    movq $-1, %r15
 ; X64-NOPIC-MCM-NEXT:    sarq $63, %rax
-; X64-NOPIC-MCM-NEXT:    movq %rsp, %r14
+; X64-NOPIC-MCM-NEXT:    leaq {{[0-9]+}}(%rsp), %r14
 ; X64-NOPIC-MCM-NEXT:    shlq $47, %rax
 ; X64-NOPIC-MCM-NEXT:    movq %r14, %rdi
 ; X64-NOPIC-MCM-NEXT:    orq %rax, %rsp
 ; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr4(%rip), %rbp
 ; X64-NOPIC-MCM-NEXT:    callq setjmp
 ; X64-NOPIC-MCM-NEXT:  .Lslh_ret_addr4:
 ; X64-NOPIC-MCM-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-MCM-NEXT:    sarq $63, %rax
 ; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr4(%rip), %rcx
 ; X64-NOPIC-MCM-NEXT:    cmpq %rcx, %rbp
 ; X64-NOPIC-MCM-NEXT:    cmovneq %r15, %rax
 ; X64-NOPIC-MCM-NEXT:    movl (%rbx), %ebp
+; X64-NOPIC-MCM-NEXT:    movl $42, %r12d
 ; X64-NOPIC-MCM-NEXT:    shlq $47, %rax
 ; X64-NOPIC-MCM-NEXT:    movq %r14, %rdi
-; X64-NOPIC-MCM-NEXT:    movl $42, %esi
+; X64-NOPIC-MCM-NEXT:    movl %r12d, %esi
 ; X64-NOPIC-MCM-NEXT:    orq %rax, %rsp
-; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr5(%rip), %r12
+; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr5(%rip), %r13
 ; X64-NOPIC-MCM-NEXT:    callq sigsetjmp
 ; X64-NOPIC-MCM-NEXT:  .Lslh_ret_addr5:
 ; X64-NOPIC-MCM-NEXT:    movq %rsp, %rax
 ; X64-NOPIC-MCM-NEXT:    sarq $63, %rax
 ; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr5(%rip), %rcx
-; X64-NOPIC-MCM-NEXT:    cmpq %rcx, %r12
+; X64-NOPIC-MCM-NEXT:    cmpq %rcx, %r13
 ; X64-NOPIC-MCM-NEXT:    cmovneq %r15, %rax
 ; X64-NOPIC-MCM-NEXT:    addl (%rbx), %ebp
 ; X64-NOPIC-MCM-NEXT:    shlq $47, %rax
 ; X64-NOPIC-MCM-NEXT:    movq %r14, %rdi
 ; X64-NOPIC-MCM-NEXT:    movq %r14, %rsi
-; X64-NOPIC-MCM-NEXT:    movl $42, %edx
+; X64-NOPIC-MCM-NEXT:    movl %r12d, %edx
 ; X64-NOPIC-MCM-NEXT:    orq %rax, %rsp
 ; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr6(%rip), %r14
 ; X64-NOPIC-MCM-NEXT:    callq __sigsetjmp
 ; X64-NOPIC-MCM-NEXT:  .Lslh_ret_addr6:
-; X64-NOPIC-MCM-NEXT:    movq %rsp, %rcx
-; X64-NOPIC-MCM-NEXT:    sarq $63, %rcx
-; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr6(%rip), %rax
-; X64-NOPIC-MCM-NEXT:    cmpq %rax, %r14
+; X64-NOPIC-MCM-NEXT:    movq %rsp, %rax
+; X64-NOPIC-MCM-NEXT:    sarq $63, %rax
+; X64-NOPIC-MCM-NEXT:    leaq .Lslh_ret_addr6(%rip), %rcx
+; X64-NOPIC-MCM-NEXT:    cmpq %rcx, %r14
+; X64-NOPIC-MCM-NEXT:    movq %rax, %rcx
 ; X64-NOPIC-MCM-NEXT:    cmovneq %r15, %rcx
 ; X64-NOPIC-MCM-NEXT:    addl (%rbx), %ebp
-; X64-NOPIC-MCM-NEXT:    orl %ecx, %ebp
-; X64-NOPIC-MCM-NEXT:    shlq $47, %rcx
 ; X64-NOPIC-MCM-NEXT:    movl %ebp, %eax
+; X64-NOPIC-MCM-NEXT:    orl %ecx, %eax
+; X64-NOPIC-MCM-NEXT:    shlq $47, %rcx
 ; X64-NOPIC-MCM-NEXT:    orq %rcx, %rsp
-; X64-NOPIC-MCM-NEXT:    addq $16, %rsp
+; X64-NOPIC-MCM-NEXT:    addq $24, %rsp
 ; X64-NOPIC-MCM-NEXT:    popq %rbx
 ; X64-NOPIC-MCM-NEXT:    popq %r12
+; X64-NOPIC-MCM-NEXT:    popq %r13
 ; X64-NOPIC-MCM-NEXT:    popq %r14
 ; X64-NOPIC-MCM-NEXT:    popq %r15
 ; X64-NOPIC-MCM-NEXT:    popq %rbp
 ; X64-NOPIC-MCM-NEXT:    retq
 ;
 ; X64-PIC-LABEL: test_call_setjmp:
 ; X64-PIC:       # %bb.0: # %entry
 ; X64-PIC-NEXT:    pushq %rbp
 ; X64-PIC-NEXT:    pushq %r15
 ; X64-PIC-NEXT:    pushq %r14
+; X64-PIC-NEXT:    pushq %r13
 ; X64-PIC-NEXT:    pushq %r12
 ; X64-PIC-NEXT:    pushq %rbx
-; X64-PIC-NEXT:    subq $16, %rsp
+; X64-PIC-NEXT:    subq $24, %rsp
 ; X64-PIC-NEXT:    movq %rsp, %rax
 ; X64-PIC-NEXT:    movq %rdi, %rbx
 ; X64-PIC-NEXT:    movq $-1, %r15
 ; X64-PIC-NEXT:    sarq $63, %rax
-; X64-PIC-NEXT:    movq %rsp, %r14
+; X64-PIC-NEXT:    leaq {{[0-9]+}}(%rsp), %r14
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    movq %r14, %rdi
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    leaq .Lslh_ret_addr4(%rip), %rbp
 ; X64-PIC-NEXT:    callq setjmp@PLT
 ; X64-PIC-NEXT:  .Lslh_ret_addr4:
 ; X64-PIC-NEXT:    movq %rsp, %rax
 ; X64-PIC-NEXT:    sarq $63, %rax
 ; X64-PIC-NEXT:    leaq .Lslh_ret_addr4(%rip), %rcx
 ; X64-PIC-NEXT:    cmpq %rcx, %rbp
 ; X64-PIC-NEXT:    cmovneq %r15, %rax
 ; X64-PIC-NEXT:    movl (%rbx), %ebp
+; X64-PIC-NEXT:    movl $42, %r12d
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    movq %r14, %rdi
-; X64-PIC-NEXT:    movl $42, %esi
+; X64-PIC-NEXT:    movl %r12d, %esi
 ; X64-PIC-NEXT:    orq %rax, %rsp
-; X64-PIC-NEXT:    leaq .Lslh_ret_addr5(%rip), %r12
+; X64-PIC-NEXT:    leaq .Lslh_ret_addr5(%rip), %r13
 ; X64-PIC-NEXT:    callq sigsetjmp@PLT
 ; X64-PIC-NEXT:  .Lslh_ret_addr5:
 ; X64-PIC-NEXT:    movq %rsp, %rax
 ; X64-PIC-NEXT:    sarq $63, %rax
 ; X64-PIC-NEXT:    leaq .Lslh_ret_addr5(%rip), %rcx
-; X64-PIC-NEXT:    cmpq %rcx, %r12
+; X64-PIC-NEXT:    cmpq %rcx, %r13
 ; X64-PIC-NEXT:    cmovneq %r15, %rax
 ; X64-PIC-NEXT:    addl (%rbx), %ebp
 ; X64-PIC-NEXT:    shlq $47, %rax
 ; X64-PIC-NEXT:    movq %r14, %rdi
 ; X64-PIC-NEXT:    movq %r14, %rsi
-; X64-PIC-NEXT:    movl $42, %edx
+; X64-PIC-NEXT:    movl %r12d, %edx
 ; X64-PIC-NEXT:    orq %rax, %rsp
 ; X64-PIC-NEXT:    leaq .Lslh_ret_addr6(%rip), %r14
 ; X64-PIC-NEXT:    callq __sigsetjmp@PLT
 ; X64-PIC-NEXT:  .Lslh_ret_addr6:
-; X64-PIC-NEXT:    movq %rsp, %rcx
-; X64-PIC-NEXT:    sarq $63, %rcx
-; X64-PIC-NEXT:    leaq .Lslh_ret_addr6(%rip), %rax
-; X64-PIC-NEXT:    cmpq %rax, %r14
+; X64-PIC-NEXT:    movq %rsp, %rax
+; X64-PIC-NEXT:    sarq $63, %rax
+; X64-PIC-NEXT:    leaq .Lslh_ret_addr6(%rip), %rcx
+; X64-PIC-NEXT:    cmpq %rcx, %r14
+; X64-PIC-NEXT:    movq %rax, %rcx
 ; X64-PIC-NEXT:    cmovneq %r15, %rcx
 ; X64-PIC-NEXT:    addl (%rbx), %ebp
-; X64-PIC-NEXT:    orl %ecx, %ebp
-; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    movl %ebp, %eax
+; X64-PIC-NEXT:    orl %ecx, %eax
+; X64-PIC-NEXT:    shlq $47, %rcx
 ; X64-PIC-NEXT:    orq %rcx, %rsp
-; X64-PIC-NEXT:    addq $16, %rsp
+; X64-PIC-NEXT:    addq $24, %rsp
 ; X64-PIC-NEXT:    popq %rbx
 ; X64-PIC-NEXT:    popq %r12
+; X64-PIC-NEXT:    popq %r13
 ; X64-PIC-NEXT:    popq %r14
 ; X64-PIC-NEXT:    popq %r15
 ; X64-PIC-NEXT:    popq %rbp
 ; X64-PIC-NEXT:    retq
 entry:
   %env = alloca i8, i32 16
   ; Call a normal setjmp function.
   call i32 @setjmp(i8* %env)