[TSAN, PowerPC] Setjmp/longjmp handling for PowerPC

[wschmidt]

Summary

This patch builds on Simone Atzeni's patches (D12840, D12841) to enable TSAN for Power. Simone wasn't comfortable attempting to write the assembly code for intercepting setjmp and sigsetjmp calls, so I've done that here.

The main part of the patch is the new tsan_rtl_ppc64.S file, based on what's done in tsan_rtl_amd64.S. Our version looks much more complicated because we can't do a tail call optimization on an indirect call on Power, since we need to restore the TOC pointer (r2) following the call. Therefore we have to stack a frame, leading to problems with correct manipulation of the jmpbuf. The solution used here (idea due to Ulrich Weigand) is to replace the tail call to the libc setjmp() with a hand-inlined version of the setjmp logic. Since kernels and libc must have the same view of a jmpbuf, it is safe to assume this will not change. We can then modify the code slightly to save the original stack pointer of our caller, and its mangled version, instead of the current stack pointer.

A header file, ppc-regs.h, is included to allow us to write more legible assembly code, distinguishing more easily between register numbers and constants.

The LongJmp interceptor in tsan_interceptors.cc is modified to look for the mangled stack pointer in the architected jmpbuf location for Power.

One test case, race_on_mutex.c, is modified to be sufficiently general for the test to pass on Power. There are two issues:

(1) The atomic read as part of pthread_mutex_init and pthread_mutex_lock is of size 8 on Power, as opposed to size 1 for existing targets.  
(2) The libc implementation of pthread_mutex_init on Power includes a memset, which is on top of the stack trace for the "previous write."  So the position of pthread_mutex_init will be at #1 rather than #0 for Power.  I've changed the test to verify that pthread_mutex_init and Thread1 are on the stack, but not necessarily right at the top.

These changes fix 6 of the 7 test case failures that remained after applying Simone's patches. The remaining java-related patch needs more investigation, but we might want to XFAIL that test for Power for now and go ahead with enabling TSAN when Simone is finished with his remaining investigations.

This patch will not be applied until Simone's work is complete.

[wschmidt]

Ping...

[samsonov]

Looks reasonable, but I'll let Dmitry sign this off.

lib/tsan/rtl/tsan_interceptors.cc

458

remove comment

[wschmidt]

lib/tsan/rtl/tsan_rtl_ppc64.S

24

It turns out that I won't be able to use this form, as older binutils versions do not recognize .TOC. as a true symbol. There is another way to do this: see https://sourceware.org/ml/binutils/2012-11/msg00055.html. I will have to use that method instead.

[dvyukov]

I did not look at the assembly because I don't understand it. But I am fine with it as long as tests pass. This code does not change frequently.

lib/tsan/rtl/ppc-regs.h

1 ↗

(On Diff #37354)

all tsan source file names currently start with tsan_, please keep this convention

test/tsan/race_on_mutex.c

36 ↗

(On Diff #37354)

How does it happen that the access is not 1 byte?
This memory access is emulated in __tsan::MutexLock, and it explicitly coded as 1-byte.

This test change is unrelated to setjmp/longjmp and should go into a separate patch.

[wschmidt]

lib/tsan/rtl/ppc-regs.h

1 ↗

(On Diff #37354)

OK, will do.

test/tsan/race_on_mutex.c

36 ↗

(On Diff #37354)

Hm, I need to look into this again. We are seeing an 8-byte access, which I thought was due to not having a single-byte lock instruction implemented. But now that I look at it, the lbarx instruction is in our machine description now. So it's a good question why we're getting a different size. I'll drop this from this patch for now, and at some point we'll have to sort this out.

[dvyukov]

Bill, you did not upload a new version. Waiting.

[wschmidt]

In D13729#279937, @dvyukov wrote:

Bill, you did not upload a new version. Waiting.

Right, I don't have one ready yet. I need to fix some of the assembly code as well, and haven't yet done so. I should have time to work on this later today.

[wschmidt]

All comments should be addressed here. I have dropped the test case that still needs a little investigation. Again, this patch requires Simone's patches as a prerequisite. I had to do a little massaging of his compiler-rt patch to bring it up to current trunk.

All setjmp/longjmp-related TSAN tests now pass. We still have failures in race_on_mutex.c and java_race_pc.cc. I believe the former is still a test case that is too "tight," but I need to prove that at some point.

[wschmidt]

I forgot to mention that I now have separate ways of materializing the TOC pointer for BE and LE. For BE, the TOC can be materialized from an offset of 8 from the beginning of the OPD, where the OPD is equivalent to the function address. For LE, all supported binutils define the .TOC. symbol, so that method can be used instead.

Simone, can you please test this assembly code on the machine where you were having difficulty with the previous patch?

[dvyukov]

I am fine with landing this as long as it works.
I did not read the assembly.

[simoatze]

Simone, can you please test this assembly code on the machine where you were having difficulty with the previous patch?

I am trying to test it on the big endian machine with older kernel, but since I updated to the to the current trunk
and applied the new patch with my patch, all tests are failing and just before running the tests I get this message:

lit.py: discovery.py:190: warning: test suite 'ThreadSanitizer-Unit' contained no tests

did something change on Tsan that can affect the execution with older kernel?
It’s working on little endian newer kernel though.

Any idea?

Best,
Simone

Repository:
rL LLVM

http://reviews.llvm.org/D13729

[dvyukov]

In D13729#283296, @simoatze wrote:

Simone, can you please test this assembly code on the machine where you were having difficulty with the previous patch?

I am trying to test it on the big endian machine with older kernel, but since I updated to the to the current trunk
and applied the new patch with my patch, all tests are failing and just before running the tests I get this message:

lit.py: discovery.py:190: warning: test suite 'ThreadSanitizer-Unit' contained no tests

did something change on Tsan that can affect the execution with older kernel?
It’s working on little endian newer kernel though.

Any idea?

Best,
Simone

Repository:
rL LLVM

http://reviews.llvm.org/D13729

Looking at the error message, it seems that the tests are just not enabled on your machine inside of cmake files. Probably cmake queries identity of your machine in some way (e.g. uname -a), and then fails to recognize it as power.

+Alexey for cmake magic, where does recognition of target machine happens inside of cmake files?

[wschmidt]

This comment was removed by wschmidt.

[simoatze]

Hi Bill,

looks like that the older Kernel on PPC64/BE still does not like the assembly part.
If I comment out the tsan_rtl_ppc64.S from the CMakeList.txt it compiles
and when I run the check-tsan it fails all the setjump tests plus some other few tests (which I am already trying to fix).
However, if I compile also the assembly code it fails all the tests.

I’ll try to look at into it with GDB and see if I can understand what is going on.

Best,
Simone

[wschmidt]

OK, thank you. I am wondering if this is a glibc "feature" when initializing a thread for Power. For LE, I noticed that it was not setting up the TOC for a call to setjmp from init_thread, which is technically within its rights only if setjmp cannot be overridden. Hence I added code to materialize the TOC from the .TOC. symbol. For BE, the TOC must be initialized by placing it in the OPD. If the OPD TOC slot contains garbage, which would be true if init_thread didn't set it up, then we have no way to materialize the TOC on BE systems. If this turns out to be the case, we would either need to disable TSAN for BE (not ideal), or try to figure out when setjmp is being called in this heinous way.

I should emphasize that I'm speculating, so looking at things with GDB is the way to go. But this may give you something to look for.

[simoatze]

Bill,

I guess you might be right.
The only info that I can gather from GDB are the following:

---

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffb7196430 (LWP 13646)]
0x7c0802a6f821ffd0 in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.7.ppc64
(gdb) bt
#0 0x7c0802a6f821ffd0 in ?? ()
#1 0x00000fffb7f6e590 in .start_thread () from /lib64/libpthread.so.0

#2 0x00000fffb7d448ec in .__clone () from /lib64/libc.so.6

Looks like there the problem comes from libthread or glibc. So, could be the init_thread problem you are talking about.
I think some debuginfo are missing, but I don’t have any root access to this machine so I can’t install anything.

Do you have any other idea how to investigate this problem?

Best,
Simone

[wschmidt]

Simone, thanks for confirming. I've discussed this with one of our ABI experts. For BE, we are going to have to do something pretty ugly, I'm afraid. We'll likely need a trampoline to a separate function so that the linker will create an OPD from which we can load the TOC. But we'll still need to keep the link register clean along that path, which could be a little tricky.

The question is whether we want to go to all this trouble, or whether we should just enable TSAN for little endian. LE is the path going forward. Do you know whether LLNL has a need to support TSAN on big-endian systems? If not, I might advocate to just drop big-endian support for TSAN.

[wschmidt]

Actually, I managed to dig up a BE Power8 machine and run the tests, and although I have failures due to the setjmp handling, most of the tests pass for me. What sort of BE machine are you using? Is it a Power7?

[wschmidt]

In D13729#287039, @wschmidt wrote:

Actually, I managed to dig up a BE Power8 machine and run the tests, and although I have failures due to the setjmp handling, most of the tests pass for me. What sort of BE machine are you using? Is it a Power7?

Well, scratch that. I accidentally rebuilt on the LE machine after applying patches to the BE one.

At the moment I'm having trouble building on the BE machine due to recent bugs introduced in Clang. Will let you know when I have actual results.

[uweigand]

I've now had a look at the current patch, and it seems the problem is simply that the assembly does not actually provide any .opd section in the ELFv1 case.

So instead of:

        .globl _setjmp
        .type _setjmp, @function
        .align 4
_setjmp:

you need something along the lines of:

        .globl _setjmp
        .type   _setjmp, @function
       ​ .align 4
        .section        ".opd","aw"
        .align 3
_setjmp:
        .quad   .L._setjmp,.TOC.@tocbase,0
        .previous
.L._setjmp:

so that the _setjmp symbol actually points to the OPD! (Otherwise, calling the function will load the first 8 bytes of function text and treat it as code address ....)

For a "normal" function, you could support both ELFv1 and ELFv2 by a prolog like this:

        .globl _setjmp
        .type _setjmp, @function
        .align 2
#if _CALL_ELF == 2
_setjmp:
         addis 2,12,(.TOC.-_setjmp)@ha;
         addi 2,2,(.TOC.-_setjmp)@l;
         .localentry _setjmp,.-_setjmp
#else
         .section        ".opd","aw"
         .align 3
_setjmp:
         .quad   .L._setjmp,.TOC.@tocbase,0
         .previous
.L._setjmp:
#endif

But if you interpose a function from a different shared library, this will not work correctly since caller and callee may get confused as to who is setting up the TOC. To handle those cases, you'll have to explicitly set up the TOC just like your patch did:

        .globl _setjmp
        .type _setjmp, @function
        .align 2
#if _CALL_ELF == 2
_setjmp:
#else
         .section        ".opd","aw"
         .align 3
_setjmp:
         .quad   .L._setjmp,.TOC.@tocbase,0
         .previous
.L._setjmp:
#endif

[...]
        // Materialize a TOC in case we were called from libc.
        // For ELFv1, we load the TOC from the OPD.  For
        // ELFv2 we use the .TOC. symbol to find it.
        nop
        bcl     20,31,0f
0:
        mflr    r2
#if _CALL_ELF == 2
        addis   r2,r2,.TOC.-0b@ha
        addi    r2,r2,.TOC.-0b@l
#else
        addis   r2,r2,_setjmp-0b@ha 
        addi    r2,r2,_setjmp-0b@l
        ld      r2,8(r2)
#endif

(Note that the _setjmp symbol now refers to the OPD entry in the ELFv1 case, not the code address!)

[wschmidt]

Oh, of course! Thanks, Uli. I totally forgot about having to declare the OPD. And it would be better if I were using _CALL_ELF rather than LITTLE_ENDIAN for the ABI distinction. I'll update the patch and try to solve my build problems to test it.

Thanks again!

[wschmidt]

This version adds the correct OPD implementation per Uli's suggestions. The only change beyond that is to make .L._setjmp and .L.__sigsetjmp unconditional symbols, so they can be used in the .size calculation for their respective functions.

I tested this as best I could on a BE system and verified that the setjmp/longjmp logic is now working correctly. There are other problems because I don't have Simone's BE fixes yet, but the remaining failures don't appear to have anything to do with this code.

[wschmidt]

Committed as r255059. Thanks!

lib/tsan/CMakeLists.txt

if(arch STREQUAL "x86_64")
       set(BUILDGO_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/go/buildgo.sh)
       add_custom_target(GotsanRuntimeCheck
         COMMAND env "CC=${CMAKE_C_COMPILER} ${CMAKE_C_COMPILER_ARG1}"
                 IN_TMPDIR=1 SILENT=1 ${BUILDGO_SCRIPT}
         DEPENDS clang_rt.tsan-${arch} ${BUILDGO_SCRIPT}
         WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/go
         COMMENT "Checking TSan Go runtime..."
         VERBATIM)
+    elseif(arch MATCHES "powerpc64|powerpc64le")
+      set(TSAN_ASM_SOURCES rtl/tsan_rtl_ppc64.S)
+      # Pass ASM file directly to the C++ compiler.
+      set_source_files_properties(${TSAN_ASM_SOURCES} PROPERTIES
+        LANGUAGE C)
     else()
       set(TSAN_ASM_SOURCES)
     endif()
     add_compiler_rt_runtime(clang_rt.tsan
       STATIC
       ARCHS ${arch}
       SOURCES ${TSAN_SOURCES} ${TSAN_ASM_SOURCES}
               $<TARGET_OBJECTS:RTInterception.${arch}>

lib/tsan/rtl/tsan_interceptors.cc

static void SetJmp(ThreadState *thr, uptr sp, uptr mangled_sp) {
   buf->in_blocking_func = sctx ?
       atomic_load(&sctx->in_blocking_func, memory_order_relaxed) :
       false;
   buf->in_signal_handler = atomic_load(&thr->in_signal_handler,
       memory_order_relaxed);
 }
 
 static void LongJmp(ThreadState *thr, uptr *env) {
-#if SANITIZER_FREEBSD
+#ifdef __powerpc__
+  uptr mangled_sp = env[0];
+#elif SANITIZER_FREEBSD
   uptr mangled_sp = env[2];
 #else
   uptr mangled_sp = env[6];
-#endif  // SANITIZER_FREEBSD
+#endif
   // Find the saved buf by mangled_sp.
   for (uptr i = 0; i < thr->jmp_bufs.Size(); i++) {
     JmpBuf *buf = &thr->jmp_bufs[i];
     if (buf->mangled_sp == mangled_sp) {
       CHECK_GE(thr->shadow_stack_pos, buf->shadow_stack_pos);
       // Unwind the stack.
       while (thr->shadow_stack_pos > buf->shadow_stack_pos)
         FuncExit(thr);

lib/tsan/rtl/tsan_ppc_regs.h

+#define r0 0
+#define r1 1
+#define r2 2
+#define r3 3
+#define r4 4
+#define r5 5
+#define r6 6
+#define r7 7
+#define r8 8
+#define r9 9
+#define r10 10
+#define r11 11
+#define r12 12
+#define r13 13
+#define r14 14
+#define r15 15
+#define r16 16
+#define r17 17
+#define r18 18
+#define r19 19
+#define r20 20
+#define r21 21
+#define r22 22
+#define r23 23
+#define r24 24
+#define r25 25
+#define r26 26
+#define r27 27
+#define r28 28
+#define r29 29
+#define r30 30
+#define r31 31
+#define f0 0
+#define f1 1
+#define f2 2
+#define f3 3
+#define f4 4
+#define f5 5
+#define f6 6
+#define f7 7
+#define f8 8
+#define f9 9
+#define f10 10
+#define f11 11
+#define f12 12
+#define f13 13
+#define f14 14
+#define f15 15
+#define f16 16
+#define f17 17
+#define f18 18
+#define f19 19
+#define f20 20
+#define f21 21
+#define f22 22
+#define f23 23
+#define f24 24
+#define f25 25
+#define f26 26
+#define f27 27
+#define f28 28
+#define f29 29
+#define f30 30
+#define f31 31
+#define v0 0
+#define v1 1
+#define v2 2
+#define v3 3
+#define v4 4
+#define v5 5
+#define v6 6
+#define v7 7
+#define v8 8
+#define v9 9
+#define v10 10
+#define v11 11
+#define v12 12
+#define v13 13
+#define v14 14
+#define v15 15
+#define v16 16
+#define v17 17
+#define v18 18
+#define v19 19
+#define v20 20
+#define v21 21
+#define v22 22
+#define v23 23
+#define v24 24
+#define v25 25
+#define v26 26
+#define v27 27
+#define v28 28
+#define v29 29
+#define v30 30
+#define v31 31

lib/tsan/rtl/tsan_rtl_ppc64.S

+#include "tsan_ppc_regs.h"
+
+        .section .text
+        .hidden __tsan_setjmp
+        .globl _setjmp
+        .type _setjmp, @function
+        .align 4
+#if _CALL_ELF == 2
+_setjmp:
+#else
+  .section ".opd","aw"
+  .align 3
+_setjmp:
+  .quad   .L._setjmp,.TOC.@tocbase,0
+  .previous
+#endif
+.L._setjmp:
+        mflr    r0
+        stdu    r1,-48(r1)
+        std     r2,24(r1)
+        std     r3,32(r1)
+        std     r0,40(r1)
+        // r3 is the original stack pointer.
+        addi    r3,r1,48
+        // r4 is the mangled stack pointer (see glibc)
+        ld      r4,-28696(r13)
+        xor     r4,r3,r4
+        // Materialize a TOC in case we were called from libc.
+        // For big-endian, we load the TOC from the OPD.  For little-
+        // endian, we use the .TOC. symbol to find it.
+        nop
+        bcl     20,31,0f
+0:
+        mflr    r2
+#if _CALL_ELF == 2
+        addis   r2,r2,.TOC.-0b@ha
+        addi    r2,r2,.TOC.-0b@l
+#else
+        addis   r2,r2,_setjmp-0b@ha
+        addi    r2,r2,_setjmp-0b@l
+        ld      r2,8(r2)
+#endif
+        // Call the interceptor.
+        bl      __tsan_setjmp
+        nop
+        // Restore regs needed for setjmp.
+        ld      r3,32(r1)
+        ld      r0,40(r1)
+        // Emulate the real setjmp function.  We do this because we can't
+        // perform a sibcall:  The real setjmp function trashes the TOC
+        // pointer, and with a sibcall we have no way to restore it.
+        // This way we can make sure our caller's stack pointer and
+        // link register are saved correctly in the jmpbuf.
+        ld      r6,-28696(r13)
+        addi    r5,r1,48  // original stack ptr of caller
+        xor     r5,r6,r5
+        std     r5,0(r3)  // mangled stack ptr of caller
+        ld      r5,24(r1)
+        std     r5,8(r3)  // caller's saved TOC pointer
+        xor     r0,r6,r0
+        std     r0,16(r3) // caller's mangled return address
+        mfcr    r0
+        // Nonvolatiles.
+        std     r14,24(r3)
+        stfd    f14,176(r3)
+        stw     r0,172(r3) // CR
+        std     r15,32(r3)
+        stfd    f15,184(r3)
+        std     r16,40(r3)
+        stfd    f16,192(r3)
+        std     r17,48(r3)
+        stfd    f17,200(r3)
+        std     r18,56(r3)
+        stfd    f18,208(r3)
+        std     r19,64(r3)
+        stfd    f19,216(r3)
+        std     r20,72(r3)
+        stfd    f20,224(r3)
+        std     r21,80(r3)
+        stfd    f21,232(r3)
+        std     r22,88(r3)
+        stfd    f22,240(r3)
+        std     r23,96(r3)
+        stfd    f23,248(r3)
+        std     r24,104(r3)
+        stfd    f24,256(r3)
+        std     r25,112(r3)
+        stfd    f25,264(r3)
+        std     r26,120(r3)
+        stfd    f26,272(r3)
+        std     r27,128(r3)
+        stfd    f27,280(r3)
+        std     r28,136(r3)
+        stfd    f28,288(r3)
+        std     r29,144(r3)
+        stfd    f29,296(r3)
+        std     r30,152(r3)
+        stfd    f30,304(r3)
+        std     r31,160(r3)
+        stfd    f31,312(r3)
+        addi    r5,r3,320
+        mfvrsave r0
+        stw     r0,168(r3)  // VRSAVE
+        addi    r6,r5,16
+        stvx    v20,0,r5
+        addi    r5,r5,32
+        stvx    v21,0,r6
+        addi    r6,r6,32
+        stvx    v22,0,r5
+        addi    r5,r5,32
+        stvx    v23,0,r6
+        addi    r6,r6,32
+        stvx    v24,0,r5
+        addi    r5,r5,32
+        stvx    v25,0,r6
+        addi    r6,r6,32
+        stvx    v26,0,r5
+        addi    r5,r5,32
+        stvx    v27,0,r6
+        addi    r6,r6,32
+        stvx    v28,0,r5
+        addi    r5,r5,32
+        stvx    v29,0,r6
+        addi    r6,r6,32
+        stvx    v30,0,r5
+        stvx    v31,0,r6
+        // Clear the "mask-saved" slot.
+        li      r4,0
+        stw     r4,512(r3)
+        // Restore TOC, LR, and stack and return to caller.
+        ld      r2,24(r1)
+        ld      r0,40(r1)
+        addi    r1,r1,48
+        li      r3,0  // This is the setjmp return path
+        mtlr    r0
+        blr
+        .size _setjmp, .-.L._setjmp
+
+        .globl setjmp
+        .type setjmp, @function
+        .align 4
+setjmp:
+        b       _setjmp
+        .size setjmp, .-setjmp
+
+        // sigsetjmp is like setjmp, except that the mask in r4 needs
+        // to be saved at offset 512 of the jump buffer.
+        .globl __sigsetjmp
+        .type __sigsetjmp, @function
+        .align 4
+#if _CALL_ELF == 2
+__sigsetjmp:
+#else
+  .section ".opd","aw"
+  .align 3
+__sigsetjmp:
+  .quad   .L.__sigsetjmp,.TOC.@tocbase,0
+  .previous
+#endif
+.L.__sigsetjmp:
+        mflr    r0
+        stdu    r1,-64(r1)
+        std     r2,24(r1)
+        std     r3,32(r1)
+        std     r4,40(r1)
+        std     r0,48(r1)
+        // r3 is the original stack pointer.
+        addi    r3,r1,64
+        // r4 is the mangled stack pointer (see glibc)
+        ld      r4,-28696(r13)
+        xor     r4,r3,r4
+        // Materialize a TOC in case we were called from libc.
+        // For big-endian, we load the TOC from the OPD.  For little-
+        // endian, we use the .TOC. symbol to find it.
+        nop
+        bcl     20,31,1f
+1:
+        mflr    r2
+#if _CALL_ELF == 2
+        addis   r2,r2,.TOC.-1b@ha
+        addi    r2,r2,.TOC.-1b@l
+#else
+        addis   r2,r2,_setjmp-1b@ha
+        addi    r2,r2,_setjmp-1b@l
+        ld      r2,8(r2)
+#endif
+        // Call the interceptor.
+        bl      __tsan_setjmp
+        nop
+        // Restore regs needed for __sigsetjmp.
+        ld      r3,32(r1)
+        ld      r4,40(r1)
+        ld      r0,48(r1)
+        // Emulate the real sigsetjmp function.  We do this because we can't
+        // perform a sibcall:  The real sigsetjmp function trashes the TOC
+        // pointer, and with a sibcall we have no way to restore it.
+        // This way we can make sure our caller's stack pointer and
+        // link register are saved correctly in the jmpbuf.
+        ld      r6,-28696(r13)
+        addi    r5,r1,64  // original stack ptr of caller
+        xor     r5,r6,r5
+        std     r5,0(r3)  // mangled stack ptr of caller
+        ld      r5,24(r1)
+        std     r5,8(r3)  // caller's saved TOC pointer
+        xor     r0,r6,r0
+        std     r0,16(r3) // caller's mangled return address
+        mfcr    r0
+        // Nonvolatiles.
+        std     r14,24(r3)
+        stfd    f14,176(r3)
+        stw     r0,172(r3) // CR
+        std     r15,32(r3)
+        stfd    f15,184(r3)
+        std     r16,40(r3)
+        stfd    f16,192(r3)
+        std     r17,48(r3)
+        stfd    f17,200(r3)
+        std     r18,56(r3)
+        stfd    f18,208(r3)
+        std     r19,64(r3)
+        stfd    f19,216(r3)
+        std     r20,72(r3)
+        stfd    f20,224(r3)
+        std     r21,80(r3)
+        stfd    f21,232(r3)
+        std     r22,88(r3)
+        stfd    f22,240(r3)
+        std     r23,96(r3)
+        stfd    f23,248(r3)
+        std     r24,104(r3)
+        stfd    f24,256(r3)
+        std     r25,112(r3)
+        stfd    f25,264(r3)
+        std     r26,120(r3)
+        stfd    f26,272(r3)
+        std     r27,128(r3)
+        stfd    f27,280(r3)
+        std     r28,136(r3)
+        stfd    f28,288(r3)
+        std     r29,144(r3)
+        stfd    f29,296(r3)
+        std     r30,152(r3)
+        stfd    f30,304(r3)
+        std     r31,160(r3)
+        stfd    f31,312(r3)
+        addi    r5,r3,320
+        mfvrsave r0
+        stw     r0,168(r3) // VRSAVE
+        addi    r6,r5,16
+        stvx    v20,0,r5
+        addi    r5,r5,32
+        stvx    v21,0,r6
+        addi    r6,r6,32
+        stvx    v22,0,r5
+        addi    r5,r5,32
+        stvx    v23,0,r6
+        addi    r6,r6,32
+        stvx    v24,0,r5
+        addi    r5,r5,32
+        stvx    v25,0,r6
+        addi    r6,r6,32
+        stvx    v26,0,r5
+        addi    r5,r5,32
+        stvx    v27,0,r6
+        addi    r6,r6,32
+        stvx    v28,0,r5
+        addi    r5,r5,32
+        stvx    v29,0,r6
+        addi    r6,r6,32
+        stvx    v30,0,r5
+        stvx    v31,0,r6
+        // Save into the "mask-saved" slot.
+        stw     r4,512(r3)
+        // Restore TOC, LR, and stack and return to caller.
+        ld      r2,24(r1)
+        ld      r0,48(r1)
+        addi    r1,r1,64
+        li      r3,0  // This is the sigsetjmp return path
+        mtlr    r0
+        blr
+        .size __sigsetjmp, .-.L.__sigsetjmp
+
+        .globl sigsetjmp
+        .type sigsetjmp, @function
+        .align 4
+sigsetjmp:
+        b       __sigsetjmp
+        .size sigsetjmp, .-sigsetjmp