[SystemZ][z/OS] Modify cxxabi to be compatible with existing z/OS runtime

[muiez]

Summary

This patch is to enable exception handling on the z/OS platform that is compatible with the existing z/OS runtime. No functionality of libcxxabi has been changed for other platforms. With this patch the hope is we can add z/OS as a platform to perform testing on any C++ ABI changes.

There is a primary difference for the z/OS implementation. On z/OS the thrown object is added to a linked list of caught and uncaught exceptions. The unwinder uses the top one as the current exception it is trying to find the landing pad for. We have to pop the top exception after we get it’s landing pad for our unwinder to correctly get any subsequent rethrows or nested exception calls.

[Jonathan.Crowther]

Ping

[abhina.sreeskantharajan]

libcxxabi/src/cxa_personality.cpp

888

minor nit: This can be changed to #elif

[zibi]

libcxxabi/include/zos/unwind.h

2

Please add the copyright statement.

49

I see you use context as parameter name for some functions and ctx` for others. Perhaps you can make this consistent for readability.

54

[muiez]

libcxxabi/src/cxa_personality.cpp

1087

minor nit: would be nice to fix the spacing

[Jonathan.Crowther]

Addressed comments

[MaskRay]

libcxxabi/include/zos/unwind.h

50

The LLVM coding standard doesn't add a space after the function name.

libcxxabi/src/cxa_personality.cpp

605

This is a regression on other platforms.
results.languageSpecificData is now nullptr.

[MaskRay]

libcxxabi/include/zos/unwind.h

64

If you are going to implement these, I highly recommend that you split the implementation from the main Itanium implementaion.

An ARM contributor has signed up the work to extract the ARM EH impl since it is so different.
I am feeling similarly for z/OS. Coupling z/OS specific logic makes the generic Itanium implementation much more difficult.

[Jonathan.Crowther]

If you are going to implement these, I highly recommend that you split the implementation from the main Itanium implementaion.

An ARM contributor has signed up the work to extract the ARM EH impl since it is so different.
I am feeling similarly for z/OS. Coupling z/OS specific logic makes the generic Itanium implementation much more difficult.

We would like as best as possible to use the generic itanium implementation. To do so we will be revising our z/OS runtime to match the interfaces of Itanium . Once this is done I will be able to remove a large portion of the z/OS specific changes. I'll leave this patch inactive while we handle that on our end and I will ping once I have made the revisions.

libcxxabi/include/zos/unwind.h

64

If you are going to implement these, I highly recommend that you split the implementation from the main Itanium implementaion.

An ARM contributor has signed up the work to extract the ARM EH impl since it is so different.
I am feeling similarly for z/OS. Coupling z/OS specific logic makes the generic Itanium implementation much more difficult.

[muiez]

Address comments. Use common API by getting LSDA from the unwinder using _Unwind_GetLanguageSpecificdata.

[muiez]

Removed whitespace

[muiez]

Use external unwind.h header.

[muiez]

Ping :)

[SeanP]

LGTM

[muiez]

kickoff CI

[muiez]

Ping @libc++abi

[muiez]

Ping, all comments addressed and green CI.

[ldionne]

libcxxabi/src/CMakeLists.txt

51

Where is LLVM_EXTERNAL_UNWIND_SOURCE_DIR defined? I just want to make sure we're not introducing yet another way of building libc++/libc++abi using external magically provided headers, since that roughly means it's going to be not supported officially from the start.

Is there a reason why we aren't using LLVM's libunwind in the tree as we normally do?

[SeanP]

Can you update the summary to remove the part about including the unwind header.

[muiez]

definition of LLVM_EXTERNAL_UNWIND_SOURCE_DIR added

[muiez]

libcxxabi/src/CMakeLists.txt

51

I updated the revision to add the definition of LLVM_EXTERNAL_UNWIND_SOURCE_DIR to a zOS.cmake file.

On z/OS, the unwind library is unique from the one provided in LLVM. It needs to interact with undocumented parts of LE. For the purposes of libc++abi, the unwind library is an external library that is not part of the standard C runtime. The build needs to add the -I to find the header and -L/-l to find the library.

A subsequent PR would be made to support the unwinder as an external project.

[zibi]

libcxx/cmake/caches/zOS.cmake

1 ↗

(On Diff #389610)

nit: I would rename this file to be zos.cmake to match what we have for libcxx downstream yet to be upstreamed.

[muiez]

renamed zos cache file name

[muiez]

formalize LLVM_EXTERNAL_UNWIND_SOURCE_DIR definition in libcxxabi/CMakeLists.txt

[muiez]

Added comment

[ldionne]

Is there a reason why we're not adding _Unwind_PopException to LLVM's libunwind and then using that libunwind instead?

If we really want to allow using an arbitrary "external" unwinding library, then I would like to see the following path explored instead: Define an interface target in CMake that represents the unwinding library, and have libcxx and libcxxabi use it. That interface target could represent either the LLVM libunwind or something selected by the user. In your case, you'd use your own interface target (and the definition of that wouldn't even have to be upstreamed). Instead of adding yet another ad-hoc way to tweak the build, it would add the general capability to pick a different unwinder, and would clean up the existing mess around LIBCXXABI_USE_LLVM_UNWINDER & friends by generalizing the setting.

[ldionne]

We can chat on Discord if you want more info on what I'm thinking about.

[MaskRay]

(In a trip; having limited Internet connection)
Drive-by comment:
AIUI _Unwind_ is the https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html namespace but also used by some GCC extensions.
Defining a _Unwind_ identifier may have potential for a conflict with libgcc if they aren't informed.

[ldionne]

In D99913#3174140, @MaskRay wrote:

(In a trip; having limited Internet connection)
Drive-by comment:
AIUI _Unwind_ is the https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html namespace but also used by some GCC extensions.
Defining a _Unwind_ identifier may have potential for a conflict with libgcc if they aren't informed.

I guess they could add _UnwindZOS_PopException when building LLVM's libunwind on z/OS.

Nonetheless, I think the "building with an external libunwind" approach is fine, but we should do it properly, not with ad-hoc include_directories and introducing a new CMake variable that only works on z/OS.

[SeanP]

In D99913#3173537, @ldionne wrote:

We can chat on Discord if you want more info on what I'm thinking about.

Thanks Louis. We'll join you on Discord shortly. We just need to some internal approvals. Could you point us (either here or elsewhere) to what we need to do in Discord to join the chats, that would be great.

[ldionne]

In D99913#3174157, @SeanP wrote:

In D99913#3173537, @ldionne wrote:

We can chat on Discord if you want more info on what I'm thinking about.

Thanks Louis. We'll join you on Discord shortly. We just need to some internal approvals. Could you point us (either here or elsewhere) to what we need to do in Discord to join the chats, that would be great.

Sure thing! This is the link: https://discord.gg/jzUbyP26tQ (documented on https://libcxx.llvm.org/Contributing.html). You'll want to join the #libcxx channel. It'll be easier to have design discussions there.

[muiez]

libcxxabi/src/CMakeLists.txt

51

In D99913#3173529, @ldionne wrote:

Is there a reason why we're not adding _Unwind_PopException to LLVM's libunwind and then using that libunwind instead?

If we really want to allow using an arbitrary "external" unwinding library, then I would like to see the following path explored instead: Define an interface target in CMake that represents the unwinding library, and have libcxx and libcxxabi use it. That interface target could represent either the LLVM libunwind or something selected by the user. In your case, you'd use your own interface target (and the definition of that wouldn't even have to be upstreamed). Instead of adding yet another ad-hoc way to tweak the build, it would add the general capability to pick a different unwinder, and would clean up the existing mess around LIBCXXABI_USE_LLVM_UNWINDER & friends by generalizing the setting.

@ldionne I'm not sure how we can get away with not including the unwind.h header from the external unwinder when building libc++abi. We'd get the following error, since the declaration is in the z/OS unwind.h:

libcxxabi/src/cxa_exception.cpp:444:5: error: use of undeclared identifier '_UnwindZOS_PopException'; did you mean '_Unwind_Exception'?
    _UnwindZOS_PopException();
    ^

We do have a similar approach (in CMake, as mentioned above) to build the external unwinder and use it within libc++abi, which will be a separate patch. For now, we need to include the external unwind.h in order to build the abi library.

[muiez]

Ping :)

[muiez]

@libc++abi Can I get a review please? Thanks

[muiez]

Removed logic for using an external unwinder. The following patch will add support for building/linking an external unwinder in a non-ad hoc way: https://reviews.llvm.org/D128348.

[muiez]

Ping :) @ldionne @libcxxabi

[ldionne]

LGTM, this isn't much now. IIUC this will require D128348 to do anything useful.

libcxxabi/include/zos/unwind.h

+//===--------------------------- zos/unwind.h -----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _UNWIND_H
+#define _UNWIND_H
+
+typedef enum {
+  _URC_NO_REASON = 0,
+  _URC_OK = 0,
+  _URC_FOREIGN_EXCEPTION_CAUGHT = 1,
+  _URC_FATAL_PHASE2_ERROR = 2,
+  _URC_FATAL_PHASE1_ERROR = 3,
+  _URC_NORMAL_STOP = 4,
+  _URC_END_OF_STACK = 5,
+  _URC_HANDLER_FOUND = 6,
+  _URC_INSTALL_CONTEXT = 7,
+  _URC_CONTINUE_UNWIND = 8,
+} _Unwind_Reason_Code;
+
+typedef enum {
+  _UA_SEARCH_PHASE = 1,
+  _UA_CLEANUP_PHASE = 2,
+  _UA_HANDLER_FRAME = 4,
+  _UA_FORCE_UNWIND = 8,
+  _UA_END_OF_STACK = 16 // gcc extension to C++ ABI
+} _Unwind_Action;
+
+struct _Unwind_Exception;
+struct _Unwind_Context;
+
+typedef void (*_Unwind_Exception_Cleanup_Fn)
+    (_Unwind_Reason_Code reason, struct _Unwind_Exception *exc);
+
+struct _Unwind_Exception {
+  char reserved[160] __attribute__ ((aligned(8)));
+  uint64_t exception_class;
+  _Unwind_Exception_Cleanup_Fn exception_cleanup;
+};
+
+#if __cplusplus
+extern "C"
+{
+#endif
+_Unwind_Reason_Code _Unwind_RaiseException( struct _Unwind_Exception *exception_object );
+void _Unwind_Resume (struct _Unwind_Exception *exception_object);
+void _Unwind_DeleteException (struct _Unwind_Exception *exception_object);
+uintptr_t  _Unwind_GetIP(void *context);
+void _Unwind_SetIP (void *context, uint64_t new_value);
+uintptr_t _Unwind_GetRegionStart(void *context);
+
+void _Unwind_SetGR (struct _Unwind_Context *context, int index, uint64_t new_value);
+uintptr_t _Unwind_GetGR (void *context, int index);
+
+void *_Unwind_StackPtr(void *context);
+void* _Unwind_GetLandingPad(void *context);
+void _Unwind_SetLandingPad(void *context, void *landingPad);
+void *_Unwind_GetExceptionAddress(void *context);
+void _Unwind_SetExceptionAddress(void *context, void *exception);
+unsigned int _Unwind_GetTypeSelector(void *context);
+void _Unwind_SetTypeSelector(void *context, unsigned int ttypeIndex);
+
+struct _Unwind_Exception *_UnwindZOS_PopException();
+#if __cplusplus
+}
+#endif
+#endif

libcxxabi/src/CMakeLists.txt

list(APPEND LIBCXXABI_SOURCES
     cxa_thread_atexit.cpp
   )
 endif()
 
 set(LIBCXXABI_HEADERS
   ../include/cxxabi.h
 )
 
+if (ZOS)
+  list(APPEND LIBCXXABI_HEADERS
+    ../include/zos/unwind.h
+  )
+  include_directories("../include/zos/")
+endif()
+
 # Add all the headers to the project for IDEs.
 if (MSVC_IDE OR XCODE)
   # Force them all into the headers dir on MSVC, otherwise they end up at
   # project scope because they don't have extensions.
   if (MSVC_IDE)
     source_group("Header Files" FILES ${LIBCXXABI_HEADERS})
   endif()
 endif()

libcxxabi/src/cxa_exception.cpp

__cxa_begin_catch(void* unwind_arg) throw()
     // exception_header is a hackish offset from a foreign exception, but it
     //   works as long as we're careful not to try to access any __cxa_exception
     //   parts.
     __cxa_exception* exception_header =
             cxa_exception_from_exception_unwind_exception
             (
                 static_cast<_Unwind_Exception*>(unwind_exception)
             );
+
+#if defined(__MVS__)
+    _UnwindZOS_PopException();
+#endif
+
     if (native_exception)
     {
         // Increment the handler count, removing the flag about being rethrown
         exception_header->handlerCount = exception_header->handlerCount < 0 ?
             -exception_header->handlerCount + 1 : exception_header->handlerCount + 1;
         //  place the exception on the top of the stack if it's not already
         //    there by a previous rethrow
         if (exception_header != globals->caughtExceptions)

libcxxabi/src/cxa_personality.cpp

3.  Scan for cleanups.  If a handler is found and this isn't forced unwind,
         May also report an error on invalid input.
         May terminate for invalid exception table.
         _UA_CLEANUP_PHASE && !_UA_HANDLER_FRAME
 */
 
 static void scan_eh_tab(scan_results &results, _Unwind_Action actions,
                         bool native_exception,
                         _Unwind_Exception *unwind_exception,
+                        const uint8_t *lsda,
                         _Unwind_Context *context) {
     // Initialize results to found nothing but an error
     results.ttypeIndex = 0;
     results.actionRecord = 0;
     results.languageSpecificData = 0;
     results.landingPad = 0;
     results.adjustedPtr = 0;
     results.reason = _URC_FATAL_PHASE1_ERROR;
static void scan_eh_tab(scan_results &results, _Unwind_Action actions,
     }
     else // Neither _UA_SEARCH_PHASE nor _UA_CLEANUP_PHASE is set
     {
         // One of these should be set.
         //   Client error
         results.reason = _URC_FATAL_PHASE1_ERROR;
         return;
     }
-    // Start scan by getting exception table address
-    const uint8_t *lsda = (const uint8_t *)_Unwind_GetLanguageSpecificData(context);
     if (lsda == 0)
     {
         // There is no exception table
         results.reason = _URC_CONTINUE_UNWIND;
         return;
     }
     results.languageSpecificData = lsda;
     // Get the current instruction pointer and offset it before next
     // instruction in the current frame which threw the exception.

 
 #if !defined(_LIBCXXABI_ARM_EHABI)
 #if defined(__SEH__) && !defined(__USING_SJLJ_EXCEPTIONS__)
 static _Unwind_Reason_Code __gxx_personality_imp
 #else
 _LIBCXXABI_FUNC_VIS _Unwind_Reason_Code
 #ifdef __USING_SJLJ_EXCEPTIONS__
 __gxx_personality_sj0
+#elif defined(__MVS__)
+__zos_cxx_personality_v2
 #else
 __gxx_personality_v0
 #endif
 #endif
-                    (int version, _Unwind_Action actions, uint64_t exceptionClass,
-                     _Unwind_Exception* unwind_exception, _Unwind_Context* context)
+                    (int version, _Unwind_Action actions,
+#if defined(__MVS__)
+                    const uint8_t* lsda,
+#else
+                    uint64_t exceptionClass,
+#endif
+                    _Unwind_Exception* unwind_exception,
+                    _Unwind_Context* context)
 {
     if (version != 1 || unwind_exception == 0 || context == 0)
         return _URC_FATAL_PHASE1_ERROR;
 
+#if defined(__MVS__)
+    uint64_t exceptionClass = unwind_exception->exception_class;
+#endif
+
     bool native_exception = (exceptionClass     & get_vendor_and_language) ==
                             (kOurExceptionClass & get_vendor_and_language);
     scan_results results;
     // Process a catch handler for a native exception first.
     if (actions == (_UA_CLEANUP_PHASE | _UA_HANDLER_FRAME) &&
         native_exception) {
         // Reload the results from the phase 1 cache.
         __cxa_exception* exception_header =
             (__cxa_exception*)(unwind_exception + 1) - 1;
         results.ttypeIndex = exception_header->handlerSwitchValue;
         results.actionRecord = exception_header->actionRecord;
         results.languageSpecificData = exception_header->languageSpecificData;
         results.landingPad =
             reinterpret_cast<uintptr_t>(exception_header->catchTemp);
         results.adjustedPtr = exception_header->adjustedPtr;
 
         // Jump to the handler.
         set_registers(unwind_exception, context, results);
         return _URC_INSTALL_CONTEXT;
     }
 
     // In other cases we need to scan LSDA.
-    scan_eh_tab(results, actions, native_exception, unwind_exception, context);
+#if !defined(__MVS__)
+    const uint8_t *lsda = (const uint8_t *)_Unwind_GetLanguageSpecificData(context);
+#endif
+    scan_eh_tab(results, actions, native_exception, unwind_exception, lsda, context);
+
     if (results.reason == _URC_CONTINUE_UNWIND ||
         results.reason == _URC_FATAL_PHASE1_ERROR)
         return results.reason;
 
     if (actions & _UA_SEARCH_PHASE)
     {
         // Phase 1 search:  All we're looking for in phase 1 is a handler that
         //   halts unwinding
#if !defined(LIBCXXABI_USE_LLVM_UNWINDER)
     _Unwind_SetGR(context, REG_UCB, reinterpret_cast<uint32_t>(unwind_exception));
 #endif
 
     // Check the undocumented force unwinding behavior
     bool is_force_unwinding = state & _US_FORCE_UNWIND;
     state &= ~_US_FORCE_UNWIND;
 
     scan_results results;
+    const uint8_t *lsda = (const uint8_t *)_Unwind_GetLanguageSpecificData(context);
     switch (state) {
     case _US_VIRTUAL_UNWIND_FRAME:
         if (is_force_unwinding)
             return continue_unwind(unwind_exception, context);
 
         // Phase 1 search:  All we're looking for in phase 1 is a handler that halts unwinding
-        scan_eh_tab(results, _UA_SEARCH_PHASE, native_exception, unwind_exception, context);
+        scan_eh_tab(results, _UA_SEARCH_PHASE, native_exception, unwind_exception, lsda, context);
         if (results.reason == _URC_HANDLER_FOUND)
         {
             unwind_exception->barrier_cache.sp = _Unwind_GetGR(context, REG_SP);
             if (native_exception)
                 save_results_to_barrier_cache(unwind_exception, results);
             return _URC_HANDLER_FOUND;
         }
         // Did not find the catch handler
case _US_UNWIND_FRAME_STARTING:
                 // Load the result from the native exception barrier cache.
                 load_results_from_barrier_cache(results, unwind_exception);
                 results.reason = _URC_HANDLER_FOUND;
             }
             else
             {
                 // Search for the catching handler again for the foreign exception.
                 scan_eh_tab(results, static_cast<_Unwind_Action>(_UA_CLEANUP_PHASE | _UA_HANDLER_FRAME),
-                            native_exception, unwind_exception, context);
+                            native_exception, unwind_exception, lsda, context);
                 if (results.reason != _URC_HANDLER_FOUND)  // phase1 search should guarantee to find one
                     call_terminate(native_exception, unwind_exception);
             }
 
             // Install the context for the catching handler
             set_registers(unwind_exception, context, results);
             return _URC_INSTALL_CONTEXT;
         }
 
         // Either we didn't do a phase 1 search (due to forced unwinding), or
-        //  phase 1 reported no catching-handlers.
+        // phase 1 reported no catching-handlers.
         // Search for a (non-catching) cleanup
-        scan_eh_tab(results, _UA_CLEANUP_PHASE, native_exception, unwind_exception, context);
+        scan_eh_tab(results, _UA_CLEANUP_PHASE, native_exception, unwind_exception, lsda, context);
         if (results.reason == _URC_HANDLER_FOUND)
         {
             // Found a non-catching handler
 
             // ARM EHABI 8.4.2: Before we can jump to the cleanup handler, we have to setup some
             // internal data structures, so that __cxa_end_cleanup() can get unwind_exception from
             // __cxa_get_globals().
             __cxa_begin_cleanup(unwind_exception);