[OpenMP] Changes to enable MSVC ARM64 build of libomp

[pulidocr]

Summary

This is the first in a series of changes to the OpenMP runtime that have been done internally by Microsoft. This patch makes the necessary changes to enable libomp.dll to build with the MSVC compiler targeting ARM64.

[jdoerfert]

@AndreyChurbanov is someone you want to have as reviewer on all these patches, you can also include others I added here.

[JonChesterfield]

Exciting! Thank you.

If I understand correctly, we presently have neither windows nor aarch64 working. Doing both of those at the same time sounds hazardous. Mixes a variety of failure modes.

We could enable windows on x86-64, which will hammer out things like uses of dlopen. That can probably be tested via a virtual machine running on a linux host, or maybe on Azure.

We could enable linux on aarch64, which may be almost a no-op. That can probably be tested via qemu for people who don't have an aarch64 box, or maybe on AWS.

If we can get linux/aarch64 passing tests, and windows/x64 passing tests, there's a much reduced cross section for bugs when trying windows/aarch64.

edit: there's also the host compiler split, building this with clang / msvc / other, which probably induces a different axis of bugs

[jdoerfert]

We talked about this in our OpenMP in LLVM meeting (https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing) today and we are looking for people and ways to test and review this.
It might be good if you could join us next Wednesday and give a short talk about the overall plan, what patches we can expect, buildbots, things like that.

[natgla]

@jdoerfert : yes, it's a great idea, I can join next Wednesday. From the agenda I didn't know you'd like to discuss this today. Hopefully Chris will be able to come too, but I'll let him tell for himself

[pulidocr]

@jdoerfert @natgla Sounds great, I'd also like to attend next meeting.

In D101173#2722833, @JonChesterfield wrote:

If I understand correctly, we presently have neither windows nor aarch64 working.

I'm confused about this statement. Does this mean that libomp is not available for Windows at all, for any architecture? I'm confused because I see libomp.dll as a part of the pre-built Windows binaries that users can download, for at least for x64 which I have installed.

[JonChesterfield]

In D101173#2724204, @pulidocr wrote:

@jdoerfert @natgla Sounds great, I'd also like to attend next meeting.

In D101173#2722833, @JonChesterfield wrote:

If I understand correctly, we presently have neither windows nor aarch64 working.

I'm confused about this statement. Does this mean that libomp is not available for Windows at all, for any architecture? I'm confused because I see libomp.dll as a part of the pre-built Windows binaries that users can download, for at least for x64 which I have installed.

That was my understanding, but not surprised to be wrong. I'm almost certain libomptarget doesn't work on windows. Libomp.dll suggests that part does build for windows, so given that information I'd then assume it passes tests too.

This patch does however read like two orthogonal sets of fixes, some for windows (which might mean msvc? Code says windows, description says msvc), some for aarch64.

[Meinersbur]

In D101173#2724204, @pulidocr wrote:

I'm confused about this statement. Does this mean that libomp is not available for Windows at all, for any architecture? I'm confused because I see libomp.dll as a part of the pre-built Windows binaries that users can download, for at least for x64 which I have installed.

I think this applies to the libomptarget library.

libomp work on Windows, I used it successfully myself. When Intel donated the library, Windows was supported from the beginning. The only problem I had is that the clang driver incorrectly wants to link to libomp.lib, but only a libomp.dll.lib is built. I don't know the status of AARCH64 though.

[AndreyChurbanov]

LGTM, though I cannot test Aarch64 functionality.

[pulidocr]

Thanks. Do I need to worry about that one clang-tidy message, or do anything else before committing?

[AndreyChurbanov]

In D101173#2745309, @pulidocr wrote:

Thanks. Do I need to worry about that one clang-tidy message, or do anything else before committing?

I think clang-tidy fails to cope with tricky macros, the symbol is declared and defined for x86 and x86_64.
Looks like we can safely skip the warning.

[pulidocr]

Apologies, I don't have commit access. Can someone commit on my behalf? Thanks in advance.

openmp/runtime/src/CMakeLists.txt

set(LIBOMP_CXXFILES
     kmp_dispatch.cpp
     kmp_lock.cpp
     kmp_sched.cpp
   )
   if(WIN32)
     # Windows specific files
     libomp_append(LIBOMP_CXXFILES z_Windows_NT_util.cpp)
     libomp_append(LIBOMP_CXXFILES z_Windows_NT-586_util.cpp)
+    if(${LIBOMP_ARCH} STREQUAL "i386" OR ${LIBOMP_ARCH} STREQUAL "x86_64")
       libomp_append(LIBOMP_ASMFILES z_Windows_NT-586_asm.asm) # Windows assembly file
+    endif()
   else()
     # Unix specific files
     libomp_append(LIBOMP_CXXFILES z_Linux_util.cpp)
     libomp_append(LIBOMP_CXXFILES kmp_gsupport.cpp)
     libomp_append(LIBOMP_ASMFILES z_Linux_asm.S) # Unix assembly file
   endif()
   libomp_append(LIBOMP_CXXFILES thirdparty/ittnotify/ittnotify_static.cpp LIBOMP_USE_ITT_NOTIFY)
   libomp_append(LIBOMP_CXXFILES kmp_debugger.cpp LIBOMP_USE_DEBUGGER)

openmp/runtime/src/dllexports

%ifdef arch_32
         __kmpc_atomic_cmplx16_add_a16      2261
         __kmpc_atomic_cmplx16_sub_a16      2262
         __kmpc_atomic_cmplx16_mul_a16      2263
         __kmpc_atomic_cmplx16_div_a16      2264
         %endif
 
     %endif
 
+    # These are specific to x86 and x64
     %ifndef arch_64
+    %ifndef arch_aarch64
 
         # ATOMIC extensions for OpenMP 3.1 spec (x86 and x64 only)
 
         __kmpc_atomic_fixed1_rd                2265
         __kmpc_atomic_fixed2_rd                2266
         __kmpc_atomic_fixed4_rd                2267
         __kmpc_atomic_fixed8_rd                2268
         __kmpc_atomic_float4_rd                2269
%ifndef arch_aarch64
             __kmpc_atomic_float4_sub_cpt_rev_fp
             __kmpc_atomic_float4_div_cpt_rev_fp
             __kmpc_atomic_float8_sub_cpt_rev_fp
             __kmpc_atomic_float8_div_cpt_rev_fp
             __kmpc_atomic_float10_sub_cpt_rev_fp
             __kmpc_atomic_float10_div_cpt_rev_fp
             %endif
 
+    %endif   # arch_aarch64
     %endif   # arch_64
 
     %ifdef HAVE_QUAD
     __kmpc_atomic_fixed1u_add_fp
     __kmpc_atomic_fixed1u_sub_fp
     __kmpc_atomic_fixed1u_mul_fp
     __kmpc_atomic_fixed2u_add_fp
     __kmpc_atomic_fixed2u_sub_fp

openmp/runtime/src/kmp.h

 extern void __kmp_clear_system_time(void);
 extern void __kmp_read_system_time(double *delta);
 
 extern void __kmp_check_stack_overlap(kmp_info_t *thr);
 
 extern void __kmp_expand_host_name(char *buffer, size_t size);
 extern void __kmp_expand_file_name(char *result, size_t rlen, char *pattern);
 
-#if KMP_ARCH_X86 || KMP_ARCH_X86_64
+#if KMP_ARCH_X86 || KMP_ARCH_X86_64 || (KMP_OS_WINDOWS && KMP_ARCH_AARCH64)
 extern void
 __kmp_initialize_system_tick(void); /* Initialize timer tick value */
 #endif
 
 extern void
 __kmp_runtime_initialize(void); /* machine specific initialization */
 extern void __kmp_runtime_destroy(void);
 

openmp/runtime/src/kmp_atomic.cpp

while (!KMP_COMPARE_AND_STORE_ACQ##BITS(                                   \
                                                                                \
       *old_value.vvv = *(volatile kmp_int##BITS *)lhs;                         \
       new_value.cmp = (TYPE)(old_value.cmp OP rhs);                            \
     }                                                                          \
   }
 // end of the first part of the workaround for C78287
 #endif // USE_CMPXCHG_FIX
 
+#if KMP_OS_WINDOWS && KMP_ARCH_AARCH64
+// Undo explicit type casts to get MSVC ARM64 to build. Uses
+// OP_CMPXCHG_WORKAROUND definition for OP_CMPXCHG
+#undef OP_CMPXCHG
+#define OP_CMPXCHG(TYPE, BITS, OP)                                             \
+  {                                                                            \
+    struct _sss {                                                              \
+      TYPE cmp;                                                                \
+      kmp_int##BITS *vvv;                                                      \
+    };                                                                         \
+    struct _sss old_value, new_value;                                          \
+    old_value.vvv = (kmp_int##BITS *)&old_value.cmp;                           \
+    new_value.vvv = (kmp_int##BITS *)&new_value.cmp;                           \
+    *old_value.vvv = *(volatile kmp_int##BITS *)lhs;                           \
+    new_value.cmp = old_value.cmp OP rhs;                                      \
+    while (!KMP_COMPARE_AND_STORE_ACQ##BITS(                                   \
+        (kmp_int##BITS *)lhs, *VOLATILE_CAST(kmp_int##BITS *) old_value.vvv,   \
+        *VOLATILE_CAST(kmp_int##BITS *) new_value.vvv)) {                      \
+      KMP_DO_PAUSE;                                                            \
+                                                                               \
+      *old_value.vvv = *(volatile kmp_int##BITS *)lhs;                         \
+      new_value.cmp = old_value.cmp OP rhs;                                    \
+    }                                                                          \
+  }
+
+#undef OP_UPDATE_CRITICAL
+#define OP_UPDATE_CRITICAL(TYPE, OP, LCK_ID)                                   \
+  __kmp_acquire_atomic_lock(&ATOMIC_LOCK##LCK_ID, gtid);                       \
+  (*lhs) = (*lhs)OP rhs;                                                       \
+  __kmp_release_atomic_lock(&ATOMIC_LOCK##LCK_ID, gtid);
+
+#endif // KMP_OS_WINDOWS && KMP_ARCH_AARCH64
+
 #if KMP_ARCH_X86 || KMP_ARCH_X86_64
 
 // ------------------------------------------------------------------------
 // X86 or X86_64: no alignment problems ====================================
 #define ATOMIC_FIXED_ADD(TYPE_ID, OP_ID, TYPE, BITS, OP, LCK_ID, MASK,         \
                          GOMP_FLAG)                                            \
   ATOMIC_BEGIN(TYPE_ID, OP_ID, TYPE, void)                                     \
   OP_UPDATE_GOMP_CRITICAL(TYPE, OP, GOMP_FLAG)                                 \

openmp/runtime/src/kmp_os.h

 typedef struct kmp_struct64 kmp_int64;
 typedef struct kmp_struct64 kmp_uint64;
 /* Not sure what to use for KMP_[U]INT64_SPEC here */
 #endif
 #if KMP_ARCH_X86 && KMP_MSVC_COMPAT
 #undef KMP_USE_X87CONTROL
 #define KMP_USE_X87CONTROL 1
 #endif
-#if KMP_ARCH_X86_64
+#if KMP_ARCH_X86_64 || KMP_ARCH_AARCH64
 #define KMP_INTPTR 1
 typedef __int64 kmp_intptr_t;
 typedef unsigned __int64 kmp_uintptr_t;
 #define KMP_INTPTR_SPEC "I64d"
 #define KMP_UINTPTR_SPEC "I64u"
 #endif
 #endif /* KMP_OS_WINDOWS */
 

 #define KMP_XCHG_FIXED64(p, v)                                                 \
   InterlockedExchange64((volatile kmp_int64 *)(p), (kmp_int64)(v))
 
 inline kmp_real32 KMP_XCHG_REAL32(volatile kmp_real32 *p, kmp_real32 v) {
   kmp_int32 tmp = InterlockedExchange((volatile long *)p, *(long *)&v);
   return *(kmp_real32 *)&tmp;
 }
 
-// Routines that we still need to implement in assembly.
-extern kmp_int8 __kmp_test_then_add8(volatile kmp_int8 *p, kmp_int8 v);
+#define KMP_TEST_THEN_OR8(p, v) __kmp_test_then_or8((p), (v))
+#define KMP_TEST_THEN_AND8(p, v) __kmp_test_then_and8((p), (v))
+#define KMP_TEST_THEN_OR32(p, v) __kmp_test_then_or32((p), (v))
+#define KMP_TEST_THEN_AND32(p, v) __kmp_test_then_and32((p), (v))
+#define KMP_TEST_THEN_OR64(p, v) __kmp_test_then_or64((p), (v))
+#define KMP_TEST_THEN_AND64(p, v) __kmp_test_then_and64((p), (v))
+
 extern kmp_int8 __kmp_test_then_or8(volatile kmp_int8 *p, kmp_int8 v);
 extern kmp_int8 __kmp_test_then_and8(volatile kmp_int8 *p, kmp_int8 v);
 extern kmp_int32 __kmp_test_then_add32(volatile kmp_int32 *p, kmp_int32 v);
 extern kmp_uint32 __kmp_test_then_or32(volatile kmp_uint32 *p, kmp_uint32 v);
 extern kmp_uint32 __kmp_test_then_and32(volatile kmp_uint32 *p, kmp_uint32 v);
 extern kmp_int64 __kmp_test_then_add64(volatile kmp_int64 *p, kmp_int64 v);
 extern kmp_uint64 __kmp_test_then_or64(volatile kmp_uint64 *p, kmp_uint64 v);
 extern kmp_uint64 __kmp_test_then_and64(volatile kmp_uint64 *p, kmp_uint64 v);
 
+#if KMP_ARCH_AARCH64 && KMP_COMPILER_MSVC && !KMP_COMPILER_CLANG
+#define KMP_TEST_THEN_INC64(p) _InterlockedExchangeAdd64((p), 1LL)
+#define KMP_TEST_THEN_INC_ACQ64(p) _InterlockedExchangeAdd64_acq((p), 1LL)
+#define KMP_TEST_THEN_ADD4_64(p) _InterlockedExchangeAdd64((p), 4LL)
+// #define KMP_TEST_THEN_ADD4_ACQ64(p) _InterlockedExchangeAdd64_acq((p), 4LL)
+// #define KMP_TEST_THEN_DEC64(p) _InterlockedExchangeAdd64((p), -1LL)
+// #define KMP_TEST_THEN_DEC_ACQ64(p) _InterlockedExchangeAdd64_acq((p), -1LL)
+// #define KMP_TEST_THEN_ADD8(p, v) _InterlockedExchangeAdd8((p), (v))
+#define KMP_TEST_THEN_ADD64(p, v) _InterlockedExchangeAdd64((p), (v))
+
+#define KMP_COMPARE_AND_STORE_ACQ8(p, cv, sv)                                  \
+  __kmp_compare_and_store_acq8((p), (cv), (sv))
+#define KMP_COMPARE_AND_STORE_REL8(p, cv, sv)                                  \
+  __kmp_compare_and_store_rel8((p), (cv), (sv))
+#define KMP_COMPARE_AND_STORE_ACQ16(p, cv, sv)                                 \
+  __kmp_compare_and_store_acq16((p), (cv), (sv))
+// #define KMP_COMPARE_AND_STORE_REL16(p, cv, sv)                                 \
+//   __kmp_compare_and_store_rel16((p), (cv), (sv))
+#define KMP_COMPARE_AND_STORE_ACQ32(p, cv, sv)                                 \
+  __kmp_compare_and_store_acq32((volatile kmp_int32 *)(p), (kmp_int32)(cv),    \
+                                (kmp_int32)(sv))
+#define KMP_COMPARE_AND_STORE_REL32(p, cv, sv)                                 \
+  __kmp_compare_and_store_rel32((volatile kmp_int32 *)(p), (kmp_int32)(cv),    \
+                                (kmp_int32)(sv))
+#define KMP_COMPARE_AND_STORE_ACQ64(p, cv, sv)                                 \
+  __kmp_compare_and_store_acq64((volatile kmp_int64 *)(p), (kmp_int64)(cv),    \
+                                (kmp_int64)(sv))
+#define KMP_COMPARE_AND_STORE_REL64(p, cv, sv)                                 \
+  __kmp_compare_and_store_rel64((volatile kmp_int64 *)(p), (kmp_int64)(cv),    \
+                                (kmp_int64)(sv))
+#define KMP_COMPARE_AND_STORE_PTR(p, cv, sv)                                   \
+  __kmp_compare_and_store_ptr((void *volatile *)(p), (void *)(cv), (void *)(sv))
+
+//  KMP_COMPARE_AND_STORE expects this order:       pointer, compare, exchange
+// _InterlockedCompareExchange expects this order:  pointer, exchange, compare
+// KMP_COMPARE_AND_STORE also returns a bool indicating a successful write. A
+// write is successful if the return value of _InterlockedCompareExchange is the
+// same as the compare value.
+inline kmp_int8 __kmp_compare_and_store_acq8(volatile kmp_int8 *p, kmp_int8 cv,
+                                             kmp_int8 sv) {
+  return _InterlockedCompareExchange8_acq(p, sv, cv) == cv;
+}
+
+inline kmp_int8 __kmp_compare_and_store_rel8(volatile kmp_int8 *p, kmp_int8 cv,
+                                             kmp_int8 sv) {
+  return _InterlockedCompareExchange8_rel(p, sv, cv) == cv;
+}
+
+inline kmp_int16 __kmp_compare_and_store_acq16(volatile kmp_int16 *p,
+                                               kmp_int16 cv, kmp_int16 sv) {
+  return _InterlockedCompareExchange16_acq(p, sv, cv) == cv;
+}
+
+inline kmp_int16 __kmp_compare_and_store_rel16(volatile kmp_int16 *p,
+                                               kmp_int16 cv, kmp_int16 sv) {
+  return _InterlockedCompareExchange16_rel(p, sv, cv) == cv;
+}
+
+inline kmp_int32 __kmp_compare_and_store_acq32(volatile kmp_int32 *p,
+                                               kmp_int32 cv, kmp_int32 sv) {
+  return _InterlockedCompareExchange_acq((volatile long *)p, sv, cv) == cv;
+}
+
+inline kmp_int32 __kmp_compare_and_store_rel32(volatile kmp_int32 *p,
+                                               kmp_int32 cv, kmp_int32 sv) {
+  return _InterlockedCompareExchange_rel((volatile long *)p, sv, cv) == cv;
+}
+
+inline kmp_int32 __kmp_compare_and_store_acq64(volatile kmp_int64 *p,
+                                               kmp_int64 cv, kmp_int64 sv) {
+  return _InterlockedCompareExchange64_acq(p, sv, cv) == cv;
+}
+
+inline kmp_int32 __kmp_compare_and_store_rel64(volatile kmp_int64 *p,
+                                               kmp_int64 cv, kmp_int64 sv) {
+  return _InterlockedCompareExchange64_rel(p, sv, cv) == cv;
+}
+
+inline kmp_int32 __kmp_compare_and_store_ptr(void *volatile *p, void *cv,
+                                             void *sv) {
+  return _InterlockedCompareExchangePointer(p, sv, cv) == cv;
+}
+
+// The _RET versions return the value instead of a bool
+// #define KMP_COMPARE_AND_STORE_RET8(p, cv, sv) \
+//   _InterlockedCompareExchange8((p), (sv), (cv))
+// #define KMP_COMPARE_AND_STORE_RET16(p, cv, sv) \
+//   _InterlockedCompareExchange16((p), (sv), (cv))
+#define KMP_COMPARE_AND_STORE_RET64(p, cv, sv)                                 \
+  _InterlockedCompareExchange64((volatile kmp_int64 *)(p), (kmp_int64)(sv),    \
+                                (kmp_int64)(cv))
+
+// #define KMP_XCHG_FIXED8(p, v)                                                  \
+//   _InterlockedExchange8((volatile kmp_int8 *)(p), (kmp_int8)(v));
+// #define KMP_XCHG_FIXED16(p, v) _InterlockedExchange16((p), (v));
+// #define KMP_XCHG_REAL64(p, v) __kmp_xchg_real64((p), (v)));
+
+// inline kmp_real64 __kmp_xchg_real64(volatile kmp_real64 *p, kmp_real64 v) {
+//   kmp_int64 tmp = _InterlockedExchange64((volatile kmp_int64 *)p, *(kmp_int64
+//   *)&v); return *(kmp_real64 *)&tmp;
+// }
+
+#else // !KMP_ARCH_AARCH64
+
+// Routines that we still need to implement in assembly.
+extern kmp_int8 __kmp_test_then_add8(volatile kmp_int8 *p, kmp_int8 v);
+
 extern kmp_int8 __kmp_compare_and_store8(volatile kmp_int8 *p, kmp_int8 cv,
                                          kmp_int8 sv);
 extern kmp_int16 __kmp_compare_and_store16(volatile kmp_int16 *p, kmp_int16 cv,
                                            kmp_int16 sv);
 extern kmp_int32 __kmp_compare_and_store32(volatile kmp_int32 *p, kmp_int32 cv,
                                            kmp_int32 sv);
 extern kmp_int32 __kmp_compare_and_store64(volatile kmp_int64 *p, kmp_int64 cv,
                                            kmp_int64 sv);

 //#define KMP_TEST_THEN_DEC32(p) __kmp_test_then_add32((p), -1)
 //#define KMP_TEST_THEN_DEC_ACQ32(p) __kmp_test_then_add32((p), -1)
 #define KMP_TEST_THEN_DEC64(p) __kmp_test_then_add64((p), -1LL)
 #define KMP_TEST_THEN_DEC_ACQ64(p) __kmp_test_then_add64((p), -1LL)
 //#define KMP_TEST_THEN_ADD32(p, v) __kmp_test_then_add32((p), (v))
 #define KMP_TEST_THEN_ADD8(p, v) __kmp_test_then_add8((p), (v))
 #define KMP_TEST_THEN_ADD64(p, v) __kmp_test_then_add64((p), (v))
 
-#define KMP_TEST_THEN_OR8(p, v) __kmp_test_then_or8((p), (v))
-#define KMP_TEST_THEN_AND8(p, v) __kmp_test_then_and8((p), (v))
-#define KMP_TEST_THEN_OR32(p, v) __kmp_test_then_or32((p), (v))
-#define KMP_TEST_THEN_AND32(p, v) __kmp_test_then_and32((p), (v))
-#define KMP_TEST_THEN_OR64(p, v) __kmp_test_then_or64((p), (v))
-#define KMP_TEST_THEN_AND64(p, v) __kmp_test_then_and64((p), (v))
 
 #define KMP_COMPARE_AND_STORE_ACQ8(p, cv, sv)                                  \
   __kmp_compare_and_store8((p), (cv), (sv))
 #define KMP_COMPARE_AND_STORE_REL8(p, cv, sv)                                  \
   __kmp_compare_and_store8((p), (cv), (sv))
 #define KMP_COMPARE_AND_STORE_ACQ16(p, cv, sv)                                 \
   __kmp_compare_and_store16((p), (cv), (sv))
 #define KMP_COMPARE_AND_STORE_REL16(p, cv, sv)                                 \

 
 #define KMP_XCHG_FIXED8(p, v)                                                  \
   __kmp_xchg_fixed8((volatile kmp_int8 *)(p), (kmp_int8)(v));
 #define KMP_XCHG_FIXED16(p, v) __kmp_xchg_fixed16((p), (v));
 //#define KMP_XCHG_FIXED32(p, v) __kmp_xchg_fixed32((p), (v));
 //#define KMP_XCHG_FIXED64(p, v) __kmp_xchg_fixed64((p), (v));
 //#define KMP_XCHG_REAL32(p, v) __kmp_xchg_real32((p), (v));
 #define KMP_XCHG_REAL64(p, v) __kmp_xchg_real64((p), (v));
+#endif
 
 #elif (KMP_ASM_INTRINS && KMP_OS_UNIX) || !(KMP_ARCH_X86 || KMP_ARCH_X86_64)
 
 /* cast p to correct type so that proper intrinsic will be used */
 #define KMP_TEST_THEN_INC32(p)                                                 \
   __sync_fetch_and_add((volatile kmp_int32 *)(p), 1)
 #define KMP_TEST_THEN_INC_ACQ32(p)                                             \
   __sync_fetch_and_add((volatile kmp_int32 *)(p), 1)

 #else
 #define KMP_MB() /* _asm{ nop } */
 #define KMP_IMB() /* _asm{ nop } */
 #endif
 #endif /* KMP_OS_WINDOWS */
 
 #if KMP_ARCH_PPC64 || KMP_ARCH_ARM || KMP_ARCH_AARCH64 || KMP_ARCH_MIPS ||     \
     KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64
+#if KMP_OS_WINDOWS
+#undef KMP_MB
+#define KMP_MB() std::atomic_thread_fence(std::memory_order_seq_cst)
+#else /* !KMP_OS_WINDOWS */
 #define KMP_MB() __sync_synchronize()
 #endif
+#endif
 
 #ifndef KMP_MB
 #define KMP_MB() /* nothing to do */
 #endif
 
 #ifndef KMP_IMB
 #define KMP_IMB() /* nothing to do */
 #endif

openmp/runtime/src/kmp_platform.h

 #define KMP_ARCH_MIPS 0
 #define KMP_ARCH_MIPS64 0
 #define KMP_ARCH_RISCV64 0
 
 #if KMP_OS_WINDOWS
 #if defined(_M_AMD64) || defined(__x86_64)
 #undef KMP_ARCH_X86_64
 #define KMP_ARCH_X86_64 1
+#elif defined(__aarch64__) || defined(_M_ARM64)
+#undef KMP_ARCH_AARCH64
+#define KMP_ARCH_AARCH64 1
 #else
 #undef KMP_ARCH_X86
 #define KMP_ARCH_X86 1
 #endif
 #endif
 
 #if KMP_OS_UNIX
 #if defined __x86_64

openmp/runtime/src/z_Windows_NT-586_util.cpp

 /*
  * z_Windows_NT-586_util.cpp -- platform specific routines.
  */
 
 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "kmp.h"
 
-#if (KMP_ARCH_X86 || KMP_ARCH_X86_64)
+#if (KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_ARCH_AARCH64)
 /* Only 32-bit "add-exchange" instruction on IA-32 architecture causes us to
    use compare_and_store for these routines */
 
 kmp_int8 __kmp_test_then_or8(volatile kmp_int8 *p, kmp_int8 d) {
   kmp_int8 old_value, new_value;
 
   old_value = TCR_1(*p);
   new_value = old_value | d;
 
-  while (!__kmp_compare_and_store8(p, old_value, new_value)) {
+  while (!KMP_COMPARE_AND_STORE_REL8(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_1(*p);
     new_value = old_value | d;
   }
   return old_value;
 }
 
 kmp_int8 __kmp_test_then_and8(volatile kmp_int8 *p, kmp_int8 d) {
   kmp_int8 old_value, new_value;
 
   old_value = TCR_1(*p);
   new_value = old_value & d;
 
-  while (!__kmp_compare_and_store8(p, old_value, new_value)) {
+  while (!KMP_COMPARE_AND_STORE_REL8(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_1(*p);
     new_value = old_value & d;
   }
   return old_value;
 }
 
 kmp_uint32 __kmp_test_then_or32(volatile kmp_uint32 *p, kmp_uint32 d) {
   kmp_uint32 old_value, new_value;
 
   old_value = TCR_4(*p);
   new_value = old_value | d;
 
-  while (!__kmp_compare_and_store32((volatile kmp_int32 *)p, old_value,
+  while (!KMP_COMPARE_AND_STORE_REL32((volatile kmp_int32 *)p, old_value,
                                       new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_4(*p);
     new_value = old_value | d;
   }
   return old_value;
 }
 
 kmp_uint32 __kmp_test_then_and32(volatile kmp_uint32 *p, kmp_uint32 d) {
   kmp_uint32 old_value, new_value;
 
   old_value = TCR_4(*p);
   new_value = old_value & d;
 
-  while (!__kmp_compare_and_store32((volatile kmp_int32 *)p, old_value,
+  while (!KMP_COMPARE_AND_STORE_REL32((volatile kmp_int32 *)p, old_value,
                                       new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_4(*p);
     new_value = old_value & d;
   }
   return old_value;
 }
 
+#if KMP_ARCH_X86 || KMP_ARCH_X86_64
 kmp_int8 __kmp_test_then_add8(volatile kmp_int8 *p, kmp_int8 d) {
   kmp_int64 old_value, new_value;
 
   old_value = TCR_1(*p);
   new_value = old_value + d;
   while (!__kmp_compare_and_store8(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_1(*p);
kmp_int64 __kmp_test_then_add64(volatile kmp_int64 *p, kmp_int64 d) {
   while (!__kmp_compare_and_store64(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_8(*p);
     new_value = old_value + d;
   }
   return old_value;
 }
 #endif /* KMP_ARCH_X86 */
+#endif /* KMP_ARCH_X86 || KMP_ARCH_X86_64 */
 
 kmp_uint64 __kmp_test_then_or64(volatile kmp_uint64 *p, kmp_uint64 d) {
   kmp_uint64 old_value, new_value;
 
   old_value = TCR_8(*p);
   new_value = old_value | d;
-  while (!__kmp_compare_and_store64((volatile kmp_int64 *)p, old_value,
+  while (!KMP_COMPARE_AND_STORE_REL64((volatile kmp_int64 *)p, old_value,
                                       new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_8(*p);
     new_value = old_value | d;
   }
 
   return old_value;
 }
 
 kmp_uint64 __kmp_test_then_and64(volatile kmp_uint64 *p, kmp_uint64 d) {
   kmp_uint64 old_value, new_value;
 
   old_value = TCR_8(*p);
   new_value = old_value & d;
-  while (!__kmp_compare_and_store64((volatile kmp_int64 *)p, old_value,
+  while (!KMP_COMPARE_AND_STORE_REL64((volatile kmp_int64 *)p, old_value,
                                       new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_8(*p);
     new_value = old_value & d;
   }
 
   return old_value;
 }
 
-#endif /* KMP_ARCH_X86 || KMP_ARCH_X86_64 */
+#if KMP_ARCH_AARCH64
+int __kmp_invoke_microtask(microtask_t pkfn, int gtid, int tid, int argc,
+                           void *p_argv[]
+#if OMPT_SUPPORT
+                           ,
+                           void **exit_frame_ptr
+#endif
+) {
+#if OMPT_SUPPORT
+  *exit_frame_ptr = OMPT_GET_FRAME_ADDRESS(0);
+#endif
+
+  switch (argc) {
+  case 0:
+    (*pkfn)(&gtid, &tid);
+    break;
+  case 1:
+    (*pkfn)(&gtid, &tid, p_argv[0]);
+    break;
+  case 2:
+    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1]);
+    break;
+  case 3:
+    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2]);
+    break;
+  case 4:
+    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3]);
+    break;
+  case 5:
+    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4]);
+    break;
+  default: {
+    // p_argv[6] and onwards must be passed on the stack since 8 registers are
+    // already used.
+    size_t len = (argc - 6) * sizeof(void *);
+    void *argbuf = alloca(len);
+    memcpy(argbuf, &p_argv[6], len);
+  }
+    [[fallthrough]];
+  case 6:
+    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5]);
+    break;
+  }
+
+#if OMPT_SUPPORT
+  *exit_frame_ptr = 0;
+#endif
+
+  return 1;
+}
+#endif
+
+#endif /* KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_ARCH_AARCH64 */