Add __atomic_is_lock_free to compiler-rt

[oontvoo]

Summary

This was one of the last required functions missing from Clang's libatomic.

Changes in behaviours:

• On x86-64, it'll now be able to detect and use cmpxchg16b
• The rest should be unchanged, given it'd already been able to figure it out statically.

[oontvoo]

Handled the remaining sizes

[jfb]

This is technically a behavior change, someone like @ldionne ought to chime in, and we probably want to synchronize with libstdc++ @jwakely.

[ldionne]

Can we please get a description of this change in the commit message and the Phab review?

[oontvoo]

updated

[oontvoo]

Removed use of GNU case range extension
(Triggered a bunch of clang-tidys)

[oontvoo]

• Remove the checking of non-power-of-2 sizes from is_lock_free for now (because the atomic_* ops don't handle them yet)
• Rename function to _c and add pragma redefine

[jfb]

Does this x86 change actually match what the runtime does in the same file?

[oontvoo]

Updated diff

[oontvoo]

In D85044#2213896, @jfb wrote:

Does this x86 change actually match what the runtime does in the same file?

Yes

[jfb]

Can you please add tests for this?

compiler-rt/lib/builtins/atomic.c

36

These are OS specific and won't work everywhere.

163

Please document what you're looking up here. It's definitely not "have_cas".

[oontvoo]

Updated diff

[oontvoo]

In D85044#2216023, @jfb wrote:

Can you please add tests for this?

I don't see this file being built anywhere. Can you point me to where to add the test?

[oontvoo]

Simple test

[jyknight]

There is going to be a bunch more complexity required here, I'm afraid.

Not only do we need to detect the CPU capabilities in order to implement atomic_is_lock_free, but we also need to ensure that the implementations of atomic_load_c / etc can actually use those atomic instructions as appropriate.

This means we need to compile the function with different flags, e.g. -mcx16 for x86-64, in order for the compiler to be able to emit cmpxchg16b, or -march=i586 on 32bit x86. But -- we can't compile the entire file that way. Only the functions called after the runtime detection has assured that we're running on the appropriate hardware can be built with those flags.

GCC handles this by compiling the source files multiple times, with different flags, adding some additional #defines which change the function names for each of those compiles. Then, it arranges to dispatch to the correct function depending on the runtime CPU detection. Ideally, we'd also be able to use the "target" attribute to do this on an individual function within the file, but it looks like that may not actually work properly.

compiler-rt/lib/builtins/atomic.c

34

Same check used twice? Also, I think we don't need a configure check for this, #if __has_include(<asm/hwcap.h>) would be ok.

145–147

This isn't threadsafe as written. It would be okay to use a relaxed atomic to load and store __have_atomic_cas (Relaxed is okay, because it's okay to potentially call check_x86_atomic_cas multiple times if this is reached simultaneously on multiple threads, and it will result in the same answer each time).

153–157

It's confusing to have "have_atomic_cas" sometimes mean "have CMPXCHG8b" and sometimes "have CMPXCHG16b".

And that confusion has led to a bug here -- on x86-64, cmpxchg8b is always available, but this code returns false for 8-byte atomics, unless CMPXCHG16b is also available.

168

AArch64 always supports atomics, the HWCAP_ATOMICS is needed only for the newer optimized instructions instead of the older LL/SC loop. (We may want to use those instructions for performance in the future, but it's not necessary for correctness, so we can leave out that complexity for now.

188

Let's leave aside ARM32 support for the first draft and just concentrate on x86 to get the overall framework in place.

193–194

At the end, it'll be better to fail to compile on platforms we haven't implemented, than to do the wrong thing.

But in order to allow implementing platforms piecemeal at the beginning, I'm ok to leave this fallback for now.

220

In C code (compiled with Clang), this can be spelled __attribute__((fallthrough)).

compiler-rt/test/builtins/Unit/atomic_lock_free_test.cc

4

Given that the contract for __atomic_is_lock_free doesn't require actual pointers, I'd leave out the construction of real objects in this test, and just pass in constructed values e.g. (void*)~7.

Also, all these assertions will need to be platform-specific.

[oontvoo]

[WIP] mov stuff to target-specific dirs

[oontvoo]

@jyknight: friendly ping ...

[oontvoo]

rebase.

Note: This has effectively undone D86510 because the patch was trying to avoid _atomic_is_lock_free, which hadn't existed then.

[ldionne]

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

[oontvoo]

In D85044#2426999, @ldionne wrote:

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

The implementation there provides the *right* answers for the cases that can be determined statically.
There are cases where it can only be determined at runtime.
For eg., unaligned non-power of 2 sizes, or more importantly 16-byte atomic ops can have lock-free implementations on some but not all x86-64 .

Most of the gunk in this patch is to set up for these checks to be extensible.
Eg., for x86/x86-64, we wants to look for CMPXCHG8B/CMPXCHG16B

Then there's the actual making use of CMPXCHG8B/CMPXCHG16B when we know it's available in *_atomic_load/store.

[ldionne]

In D85044#2427077, @oontvoo wrote:

In D85044#2426999, @ldionne wrote:

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

The implementation there provides the *right* answers for the cases that can be determined statically.

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it? I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

[oontvoo]

In D85044#2431334, @ldionne wrote:

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

Yes, D92302 provides an implementation of is_lock_free that is consistent with the current _atomic_load_* implementations.
(it's a no change in behaviour)

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it?

Yes.

I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

The two TODOs in this patch was the attempt to simply this so that we have the basic framework in place before filling in more meaty details (ARMs and odd sizes support) . I suspect we'd do those in subsequent patches.

[ldionne]

In D85044#2432266, @oontvoo wrote:

In D85044#2431334, @ldionne wrote:

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

Yes, D92302 provides an implementation of is_lock_free that is consistent with the current _atomic_load_* implementations.
(it's a no change in behaviour)

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it?

Yes.

I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

The two TODOs in this patch was the attempt to simply this so that we have the basic framework in place before filling in more meaty details (ARMs and odd sizes support) . I suspect we'd do those in subsequent patches.

Got it, thanks a lot for the clarifications. Unless we're confident that we can move forward with this patch in the near future, I would start by doing D92302 since it solves real problems for libc++ (and Apple platforms to the extent that we're currently shipping a broken compiler-rt). @arichardson There are some tests in this patch, perhaps you could grab them? @oontvoo would you be OK with us trying to land D92302 and then rebasing your patch on top?

[oontvoo]

Yes, sounds good. I don't see why it should wait.

[ldionne]

In D85044#2432326, @oontvoo wrote:

Yes, sounds good. I don't see why it should wait.

Excellent, thank you!

[oontvoo]

updated tests

[oontvoo]

rename aux file to .c

[oontvoo]

(typos)

[arichardson]

compiler-rt/test/builtins/Unit/atomic_lock_free_test.cc

58

This will fail on many 32-bit architectures (e.g. RISC-V 32)

[oontvoo]

addressed comment: restrict size-8 testing to x86-64

[ldionne]

I don't think I'll be able to contribute to this discussion meaningfully without a significant time investment since I don't touch compiler-rt a lot. Since I have a huge review queue in libc++, I'd rather resign from the review than give the impression that it's blocked on me.

compiler-rt/cmake/config-ix.cmake

 
 # Symbols.
 check_symbol_exists(__func__ "" COMPILER_RT_HAS_FUNC_SYMBOL)
 
 # Includes.
 check_cxx_compiler_flag(-nostdinc++ COMPILER_RT_HAS_NOSTDINCXX_FLAG)
 check_cxx_compiler_flag(-nostdlib++ COMPILER_RT_HAS_NOSTDLIBXX_FLAG)
 check_include_files("sys/auxv.h"    COMPILER_RT_HAS_AUXV)
+check_include_files("asm/hwcap"     COMPILER_RT_HAS_HWCAP)
 
 # Libraries.
 check_library_exists(dl dlopen "" COMPILER_RT_HAS_LIBDL)
 check_library_exists(rt shm_open "" COMPILER_RT_HAS_LIBRT)
 check_library_exists(m pow "" COMPILER_RT_HAS_LIBM)
 check_library_exists(pthread pthread_create "" COMPILER_RT_HAS_LIBPTHREAD)
 check_library_exists(execinfo backtrace "" COMPILER_RT_HAS_LIBEXECINFO)
 

compiler-rt/lib/builtins/CMakeLists.txt

set(GENERIC_SOURCES
     enable_execute_stack.c
     eprintf.c
   )
 endif()
 
 if(COMPILER_RT_HAS_ATOMIC_KEYWORD AND NOT COMPILER_RT_EXCLUDE_ATOMIC_BUILTIN)
   set(GENERIC_SOURCES
     ${GENERIC_SOURCES}
+    atomic_alt.h
+    atomic_alt.c
     atomic.c
   )
+  SET(X86_64_ATOMIC_ALT_SOURCES x86_64/atomic_alt_cx16.c)
+  SET(X86_64_ATOMIC_ALT_FLAGS "-mcx16")
 endif()
 
+
 if(APPLE)
   set(GENERIC_SOURCES
     ${GENERIC_SOURCES}
     atomic_flag_clear.c
     atomic_flag_clear_explicit.c
     atomic_flag_test_and_set.c
     atomic_flag_test_and_set_explicit.c
     atomic_signal_fence.c
if (CAN_TARGET_${arch})
 
       add_compiler_rt_runtime(clang_rt.builtins
                               STATIC
                               ARCHS ${arch}
                               SOURCES ${${arch}_SOURCES}
                               DEFS ${BUILTIN_DEFS}
                               CFLAGS ${BUILTIN_CFLAGS}
                               PARENT_TARGET builtins)
+
+      if (DEFINED ${${arch}_ATOMIC_ALT_SOURCES})
+        add_compiler_rt_runtime(clang_rt.builtins.atomic_alt
+                                STATIC
+                                SOURCES ${ATOMIC_${arch}_ALT_SOURCES}
+                                CLAGS ${${arch}_ATOMIC_ALT_FLAGS}
+                                PARENT_TARGET clang_rt.builtins)
+        add_dependencies(clang_rt.builtins, clang_rt.builtins.atomic_alt)
+      endif()
     endif ()
   endforeach ()
 endif ()
 
 add_dependencies(compiler-rt builtins)

compiler-rt/lib/builtins/atomic.c

 //
 //===----------------------------------------------------------------------===//
 
 #include <stdbool.h>
 #include <stdint.h>
 #include <string.h>
 
 #include "assembly.h"
+#include "atomic_alt.h"
 
 // Clang objects if you redefine a builtin.  This little hack allows us to
 // define a function with the same name as an intrinsic.
 #pragma redefine_extname __atomic_load_c SYMBOL_NAME(__atomic_load)
 #pragma redefine_extname __atomic_store_c SYMBOL_NAME(__atomic_store)
 #pragma redefine_extname __atomic_exchange_c SYMBOL_NAME(__atomic_exchange)
 #pragma redefine_extname __atomic_compare_exchange_c SYMBOL_NAME(              \
     __atomic_compare_exchange)
+#pragma redefine_extname __atomic_is_lock_free_c SYMBOL_NAME(                  \
+    __atomic_is_lock_free)
 
 /// Number of locks.  This allocates one page on 32-bit platforms, two on
 /// 64-bit.  This can be specified externally if a different trade between
 /// memory usage and contention probability is required for a given platform.
 #ifndef SPINLOCK_COUNT
 #define SPINLOCK_COUNT (1 << 10)
 #endif
 static const long SPINLOCK_MASK = SPINLOCK_COUNT - 1;

 /// Returns a lock to use for a given pointer.
 static __inline Lock *lock_for_pointer(void *ptr) {
   intptr_t hash = (intptr_t)ptr;
   // Disregard the lowest 4 bits.  We want all values that may be part of the
   // same memory operation to hash to the same value and therefore use the same
   // lock.
   hash >>= 4;
   // Use the next bits as the basis for the hash
   intptr_t low = hash & SPINLOCK_MASK;
   // Now use the high(er) set of bits to perturb the hash, so that we don't
   // get collisions from atomic fields in a single object
   hash >>= 16;
   hash ^= low;
   // Return a pointer to the word to use
   return locks + (hash & SPINLOCK_MASK);
 }
 
 /// Macros for determining whether a size is lock free.
-#define ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(size, p)                  \
-  (__atomic_always_lock_free(size, p) ||                                       \
-   (__atomic_always_lock_free(size, 0) && ((uintptr_t)p % size) == 0))
-#define IS_LOCK_FREE_1(p) ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(1, p)
-#define IS_LOCK_FREE_2(p) ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(2, p)
-#define IS_LOCK_FREE_4(p) ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(4, p)
-#define IS_LOCK_FREE_8(p) ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(8, p)
-#define IS_LOCK_FREE_16(p) ATOMIC_ALWAYS_LOCK_FREE_OR_ALIGNED_LOCK_FREE(16, p)
+#define IS_LOCK_FREE_1 __c11_atomic_is_lock_free(1)
+#define IS_LOCK_FREE_2 __c11_atomic_is_lock_free(2)
+#define IS_LOCK_FREE_4 __c11_atomic_is_lock_free(4)
+#define IS_LOCK_FREE_8 __c11_atomic_is_lock_free(8)
 
-/// Macro that calls the compiler-generated lock-free versions of functions
-/// when they exist.
-#define TRY_LOCK_FREE_CASE(n, type, ptr)                                       \
-  case n:                                                                      \
-    if (IS_LOCK_FREE_##n(ptr)) {                                               \
-      LOCK_FREE_ACTION(type);                                                  \
-    }                                                                          \
-    break;
 #ifdef __SIZEOF_INT128__
-#define TRY_LOCK_FREE_CASE_16(p) TRY_LOCK_FREE_CASE(16, __uint128_t, p)
+#define IS_LOCK_FREE_16 __c11_atomic_is_lock_free(16)
+#define HANDLE_CASE_16(LOCK_FREE_ACTION_ALT)                                   \
+  if (IS_LOCK_FREE_16) {                                                       \
+    LOCK_FREE_ACTION_ALT(__uint128_t);                                         \
+  }
 #else
-#define TRY_LOCK_FREE_CASE_16(p) /* __uint128_t not available */
-#endif
+#define IS_LOCK_FREE_16 0
+#define HANDLE_CASE_16(LOCK_FREE_ACTION_ALT)
+#endif // __SIZEOF_INT128__
 
-#define LOCK_FREE_CASES(ptr)                                                   \
+/// Macro that calls the compiler-generated lock-free versions of functions
+/// when they exist.
+/// The argument is the alternative action to use for case-8 and case-16
+#define LOCK_FREE_CASES(LOCK_FREE_ACTION_ALT)                                  \
   do {                                                                         \
     switch (size) {                                                            \
-      TRY_LOCK_FREE_CASE(1, uint8_t, ptr)                                      \
-      TRY_LOCK_FREE_CASE(2, uint16_t, ptr)                                     \
-      TRY_LOCK_FREE_CASE(4, uint32_t, ptr)                                     \
-      TRY_LOCK_FREE_CASE(8, uint64_t, ptr)                                     \
-      TRY_LOCK_FREE_CASE_16(ptr) /* __uint128_t may not be supported */        \
-    default:                                                                   \
+    case 1:                                                                    \
+      if (IS_LOCK_FREE_1) {                                                    \
+        LOCK_FREE_ACTION(uint8_t);                                             \
+      }                                                                        \
+      break;                                                                   \
+    case 2:                                                                    \
+      if (IS_LOCK_FREE_2) {                                                    \
+        LOCK_FREE_ACTION(uint16_t);                                            \
+      }                                                                        \
+      break;                                                                   \
+    case 4:                                                                    \
+      if (IS_LOCK_FREE_4) {                                                    \
+        LOCK_FREE_ACTION(uint32_t);                                            \
+      }                                                                        \
+      break;                                                                   \
+    case 8:                                                                    \
+      if (IS_LOCK_FREE_8) {                                                    \
+        LOCK_FREE_ACTION_ALT(uint64_t);                                        \
+      }                                                                        \
+      break;                                                                   \
+    case 16:                                                                   \
+      /* Special handling because not all platforms have uint_128*/            \
+      HANDLE_CASE_16(LOCK_FREE_ACTION_ALT)                                     \
       break;                                                                   \
     }                                                                          \
   } while (0)
 
 /// An atomic load operation.  This is atomic with respect to the source
 /// pointer only.
 void __atomic_load_c(int size, void *src, void *dest, int model) {
 #define LOCK_FREE_ACTION(type)                                                 \
   *((type *)dest) = __c11_atomic_load((_Atomic(type) *)src, model);            \
   return;
-  LOCK_FREE_CASES(src);
+#define LOCK_FREE_ACTION_ALT(type)                                             \
+  *((type *)dest) = __c11_atomic_load_ifunc((_Atomic(type) *)src, model);      \
+  return;
+  LOCK_FREE_CASES(LOCK_FREE_ACTION_ALT);
 #undef LOCK_FREE_ACTION
+#undef LOCK_FREE_ACTION_ALT
   Lock *l = lock_for_pointer(src);
   lock(l);
   memcpy(dest, src, size);
   unlock(l);
 }
 
 /// An atomic store operation.  This is atomic with respect to the destination
 /// pointer only.
 void __atomic_store_c(int size, void *dest, void *src, int model) {
 #define LOCK_FREE_ACTION(type)                                                 \
   __c11_atomic_store((_Atomic(type) *)dest, *(type *)src, model);              \
   return;
-  LOCK_FREE_CASES(dest);
+  LOCK_FREE_CASES(LOCK_FREE_ACTION);
 #undef LOCK_FREE_ACTION
+#undef LOCK_FREE_ACTION_ALT
   Lock *l = lock_for_pointer(dest);
   lock(l);
   memcpy(dest, src, size);
   unlock(l);
 }
 
 /// Atomic compare and exchange operation.  If the value at *ptr is identical
 /// to the value at *expected, then this copies value at *desired to *ptr.  If
 /// they  are not, then this stores the current value from *ptr in *expected.
 ///
 /// This function returns 1 if the exchange takes place or 0 if it fails.
 int __atomic_compare_exchange_c(int size, void *ptr, void *expected,
                                 void *desired, int success, int failure) {
 #define LOCK_FREE_ACTION(type)                                                 \
   return __c11_atomic_compare_exchange_strong(                                 \
       (_Atomic(type) *)ptr, (type *)expected, *(type *)desired, success,       \
       failure)
-  LOCK_FREE_CASES(ptr);
+#define LOCK_FREE_ACTION_ALT(type)                                             \
+  return __c11_atomic_compare_exchange_strong_ifunc(                           \
+      (_Atomic(type) *)ptr, (type *)expected, *(type *)desired, success,       \
+      failure)
+  LOCK_FREE_CASES(LOCK_FREE_ACTION_ALT);
 #undef LOCK_FREE_ACTION
+#undef LOCK_FREE_ACTION_ALT
   Lock *l = lock_for_pointer(ptr);
   lock(l);
   if (memcmp(ptr, expected, size) == 0) {
     memcpy(ptr, desired, size);
     unlock(l);
     return 1;
   }
   memcpy(expected, ptr, size);
   unlock(l);
   return 0;
 }
 
 /// Performs an atomic exchange operation between two pointers.  This is atomic
 /// with respect to the target address.
 void __atomic_exchange_c(int size, void *ptr, void *val, void *old, int model) {
 #define LOCK_FREE_ACTION(type)                                                 \
   *(type *)old =                                                               \
       __c11_atomic_exchange((_Atomic(type) *)ptr, *(type *)val, model);        \
   return;
-  LOCK_FREE_CASES(ptr);
+  LOCK_FREE_CASES(LOCK_FREE_ACTION);
 #undef LOCK_FREE_ACTION
   Lock *l = lock_for_pointer(ptr);
   lock(l);
   memcpy(old, ptr, size);
   memcpy(ptr, val, size);
   unlock(l);
 }
 
#define OPTIMISED_CASES                                                        \
   OPTIMISED_CASE(1, IS_LOCK_FREE_1, uint8_t)                                   \
   OPTIMISED_CASE(2, IS_LOCK_FREE_2, uint16_t)                                  \
   OPTIMISED_CASE(4, IS_LOCK_FREE_4, uint32_t)                                  \
   OPTIMISED_CASE(8, IS_LOCK_FREE_8, uint64_t)
 #endif
 
 #define OPTIMISED_CASE(n, lockfree, type)                                      \
   type __atomic_load_##n(type *src, int model) {                               \
-    if (lockfree(src))                                                         \
-      return __c11_atomic_load((_Atomic(type) *)src, model);                   \
+    if (lockfree)                                                              \
+      return __c11_atomic_load_ifunc((_Atomic(type) *)src, model);             \
     Lock *l = lock_for_pointer(src);                                           \
     lock(l);                                                                   \
     type val = *src;                                                           \
     unlock(l);                                                                 \
     return val;                                                                \
   }
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 
 #define OPTIMISED_CASE(n, lockfree, type)                                      \
   void __atomic_store_##n(type *dest, type val, int model) {                   \
-    if (lockfree(dest)) {                                                      \
+    if (lockfree) {                                                            \
       __c11_atomic_store((_Atomic(type) *)dest, val, model);                   \
       return;                                                                  \
     }                                                                          \
     Lock *l = lock_for_pointer(dest);                                          \
     lock(l);                                                                   \
     *dest = val;                                                               \
     unlock(l);                                                                 \
     return;                                                                    \
   }
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 
 #define OPTIMISED_CASE(n, lockfree, type)                                      \
   type __atomic_exchange_##n(type *dest, type val, int model) {                \
-    if (lockfree(dest))                                                        \
+    if (lockfree)                                                              \
       return __c11_atomic_exchange((_Atomic(type) *)dest, val, model);         \
     Lock *l = lock_for_pointer(dest);                                          \
     lock(l);                                                                   \
     type tmp = *dest;                                                          \
     *dest = val;                                                               \
     unlock(l);                                                                 \
     return tmp;                                                                \
   }
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 
 #define OPTIMISED_CASE(n, lockfree, type)                                      \
   bool __atomic_compare_exchange_##n(type *ptr, type *expected, type desired,  \
                                      int success, int failure) {               \
-    if (lockfree(ptr))                                                         \
+    if (lockfree)                                                              \
       return __c11_atomic_compare_exchange_strong(                             \
           (_Atomic(type) *)ptr, expected, desired, success, failure);          \
     Lock *l = lock_for_pointer(ptr);                                           \
     lock(l);                                                                   \
     if (*ptr == *expected) {                                                   \
       *ptr = desired;                                                          \
       unlock(l);                                                               \
       return true;                                                             \
     }                                                                          \
     *expected = *ptr;                                                          \
     unlock(l);                                                                 \
     return false;                                                              \
   }
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 
 ////////////////////////////////////////////////////////////////////////////////
 // Atomic read-modify-write operations for integers of various sizes.
 ////////////////////////////////////////////////////////////////////////////////
 #define ATOMIC_RMW(n, lockfree, type, opname, op)                              \
   type __atomic_fetch_##opname##_##n(type *ptr, type val, int model) {         \
-    if (lockfree(ptr))                                                         \
+    if (lockfree)                                                              \
       return __c11_atomic_fetch_##opname((_Atomic(type) *)ptr, val, model);    \
     Lock *l = lock_for_pointer(ptr);                                           \
     lock(l);                                                                   \
     type tmp = *ptr;                                                           \
     *ptr = tmp op val;                                                         \
     unlock(l);                                                                 \
     return tmp;                                                                \
   }

compiler-rt/lib/builtins/atomic_alt.h

+#ifndef COMPILERRT_ATOMIC_ALT_H
+#define COMPILERRT_ATOMIC_ALT_H
+
+#include "int_lib.h"
+#include <stdbool.h>
+
+#ifdef __SIZEOF_INT128__
+__uint128_t __c11_atomic_load_ifunc(__uint128_t *src, int model);
+int __c11_atomic_compare_exchange_strong_ifunc(__uint128_t *ptr,
+                                               __uint128_t *expected,
+                                               __uint128_t desired, int success,
+                                               int failure);
+#else
+uint64_t __c11_atomic_load_ifunc(uint64_t *src, int model);
+int __c11_atomic_compare_exchange_strong_ifunc(uint64_t *ptr,
+                                               uint64_t *expected,
+                                               uint64_t desired, int success,
+                                               int failure);
+
+#endif
+
+// This will be renamed in atomic.c
+bool __atomic_is_lock_free_c(unsigned long size, const volatile void *ptr);
+
+#endif // COMPILERRT_ATOMIC_ALT_H

compiler-rt/lib/builtins/atomic_alt.c

+#include "atomic_alt.h"
+
+#include <stdint.h>
+
+#if defined(__x86_64__) || defined(__x86__)
+#include <cpuid.h>
+#endif
+
+#if __has_include(<asm/hwcap.h>) && __has_include(<sys/auxv.h>)
+#include <asm/hwcap.h>
+#include <sys/auxv.h>
+#define HAS_ARM_HWCAP 1
+#else
+#define HAS_ARM_HWCAP 0
+#endif
+
+#if defined(__x86_64__) || defined(__x86__)
+
+#ifdef __x86_64__
+#define FEAT_REG ecx
+#define MASK bit_CMPXCHG16B
+#else
+#define FEAT_REG edx
+#define MASK bit_CMPXCHG8B
+#endif
+
+static inline bool check_x86_atomic_cas(void) {
+  unsigned int eax, ebx, ecx = 0, edx = 0;
+  __get_cpuid(1, &eax, &ebx, &ecx, &edx);
+  return (FEAT_REG & MASK) != 0;
+}
+
+bool have_atomic_cap(int N) {
+  static int __have_atomic_cas = -1;
+  if (__have_atomic_cas == -1) {
+    __have_atomic_cas = check_x86_atomic_cas() != 0 ? 1 : 0;
+  }
+  switch (N) {
+  case 1:
+  case 2:
+  case 4:
+    return true;
+  case 8:
+#ifdef __x86_64__
+    return true;
+  case 16:
+#endif
+    return __have_atomic_cas;
+  }
+  return false;
+}
+#elif defined(__aarch64__) && HAS_ARM_HWCAP
+
+bool have_atomic_cap(int N) {
+  static int __has_atomic_cap = -1;
+  if (__have_atomic_cap == -1) {
+    __have_atomic_cap = (getauxval(AT_HWCAP) & HWCAP_ATOMICS) != 0 ? 1 : 0;
+  }
+  switch (N) {
+  case 1:
+  case 2:
+  case 4:
+  case 8:
+    return __have_atomic_cap;
+  }
+  return false;
+}
+
+#else
+static inline bool have_atomic_cap(int N) { return false; }
+#endif
+
+// Return true if it could  positively be determined to be lock free.
+// Otherwise, fall through to the next bucket (next power-of-2).
+#define CHECK_LOCK_FREE_POW2(N)                                                \
+  do {                                                                         \
+    uintptr_t r = (uintptr_t)ptr & (N - 1);                                    \
+    if (r != 0)                                                                \
+      break;                                                                   \
+    if (__atomic_always_lock_free(N, 0))                                       \
+      return true;                                                             \
+    if (have_atomic_cap(N))                                                    \
+      return true;                                                             \
+  } while (0)
+
+bool __atomic_is_lock_free_c(unsigned long size, const volatile void *ptr) {
+  // FIXME: We don't support non-power-of-2 sizes now. They could be handled
+  // by rounding up to the next power-of-2 bucket. But all the __atomic_*
+  // operations will need to do the same thing as well.
+  switch (size) {
+  case 0:
+    return true;
+  case 2:
+    CHECK_LOCK_FREE_POW2(2);
+    __attribute__((fallthrough));
+  case 4:
+    CHECK_LOCK_FREE_POW2(4);
+    __attribute__((fallthrough));
+  case 8:
+    CHECK_LOCK_FREE_POW2(8);
+    __attribute__((fallthrough));
+  case 16:
+    CHECK_LOCK_FREE_POW2(16);
+    break;
+  }
+  return false;
+}
+
+#ifdef __X86_64__
+
+// A wrapper is needed because the resolver must return a
+// (TU-)local function and not an exernally defined function.
+extern uint64_t __c11_atomic_load_cx16(uint64_t *src, int model);
+uint64_t __c11_atomic_load_cx16_wrapper(uint64_t *src, int model) {
+  return __c11_atomic_load_cx16(src, model);
+}
+
+extern int __c11_atomic_compare_exchange_strong_cx16(uint64_t *ptr,
+                                                     uint64_t *expected,
+                                                     uint64_t desired,
+                                                     int success, int failure);
+int __c11_atomic_compare_exchange_strong_cx16_wrapper(uint64_t *ptr,
+                                                      uint64_t *expected,
+                                                      uint64_t desired,
+                                                      int success,
+                                                      int failure) {
+  return __c11_atomic_compare_exchange_strong_cx16(ptr, expected, desired,
+                                                   success, failure);
+}
+
+#ifdef __SIZEOF_INT128__
+extern __uint128_t __c11_atomic_load_cx16(__uint128_t *src, int model);
+__uint128_t __c11_atomic_load_cx16_wrapper(__uint128_t *src, int model) {
+  return __c11_atomic_load_cx16(src, model);
+}
+
+extern int __c11_atomic_compare_exchange_strong_cx16(__uint128_t *ptr,
+                                                     __uint128_t *expected,
+                                                     __uint128_t desired,
+                                                     int success, int failure);
+int __c11_atomic_compare_exchange_strong_cx16_wrapper(__uint128_t *ptr,
+                                                      __uint128_t *expected,
+                                                      __uint128_t desired,
+                                                      int success,
+                                                      int failure) {
+  return __c11_atomic_compare_exchange_strong_cx16(ptr, expected, desired,
+                                                   success, failure);
+}
+#endif // __SIZEOF_INT128__
+#endif // __X86_64__
+
+#ifdef __SIZEOF_INT128__
+__attribute__((ifunc("atomic_load_resolver"))) __uint128_t
+__c11_atomic_load_ifunc(__uint128_t *src, int model);
+
+extern __uint128_t __c11_atomic_load(__uint128_t *src, int model);
+__uint128_t __c11_atomic_load_wrapper(__uint128_t *src, int model) {
+  return __c11_atomic_load(src, model);
+}
+
+__attribute__((ifunc("atomic_compare_exchange_resolver"))) int
+__c11_atomic_compare_exchange_strong_ifunc(__uint128_t *ptr,
+                                           __uint128_t *expected,
+                                           __uint128_t desired, int success,
+                                           int failure);
+
+extern int __c11_atomic_compare_exchange_strong(__uint128_t *ptr,
+                                                __uint128_t *expected,
+                                                __uint128_t desired,
+                                                int success, int failure);
+extern int __c11_atomic_compare_exchange_strong_wrapper(__uint128_t *ptr,
+                                                        __uint128_t *expected,
+                                                        __uint128_t desired,
+                                                        int success,
+                                                        int failure) {
+  return __c11_atomic_compare_exchange_strong(ptr, expected, desired, success,
+                                              failure);
+}
+#else
+__attribute__((ifunc("atomic_load_resolver"))) uint64_t
+__c11_atomic_load_ifunc(uint64_t *src, int model);
+
+extern uint64_t __c11_atomic_load(uint64_t *src, int model);
+uint64_t __c11_atomic_load_wrapper(uint64_t *src, int model) {
+  return __c11_atomic_load(src, model);
+}
+
+__attribute__((ifunc("atomic_compare_exchange_resolver"))) int
+__c11_atomic_compare_exchange_strong_ifunc(uint64_t *ptr, uint64_t *expected,
+                                           uint64_t desired, int success,
+                                           int failure);
+
+extern int __c11_atomic_compare_exchange_strong(uint64_t *ptr,
+                                                uint64_t *expected,
+                                                uint64_t desired, int success,
+                                                int failure);
+
+int __c11_atomic_compare_exchange_strong_wrapper(uint64_t *ptr,
+                                                 uint64_t *expected,
+                                                 uint64_t desired, int success,
+                                                 int failure) {
+  return __c11_atomic_compare_exchange_strong(ptr, expected, desired, success,
+                                              failure);
+}
+
+#endif // __SIZEOF_INT128__
+
+void *atomic_load_resolver(void) {
+#ifdef __X86_64__
+  return has_atomic_cap() ? __c11_atomic_load_cx16_wrapper : __c11_atomic_load;
+#else
+  // FIXME: fill in the rest of the archs.
+  return __c11_atomic_load_wrapper;
+#endif
+}
+
+void *atomic_compare_exchange_resolver(void) {
+#ifdef __X86_64__
+  return has_atomic_cap() ? __c11_compare_exchange_strong_cx16_wrapper
+                          : __c11_atomic_compare_exchange_strong_wrapper;
+#else
+  // FIXME: fill in the rest of the archs.
+  return __c11_atomic_compare_exchange_strong_wrapper;
+#endif
+}

compiler-rt/lib/builtins/x86_64/atomic_alt_cx16.c

+// This file is compiled with -mcx16
+
+#include "../int_lib.h"
+
+COMPILER_RT_ALIAS(__c11_atomic_load, __c11_atomic_load_cx16)
+
+COMPILER_RT_ALIAS(__c11_atomic_compare_exchange_strong,
+                  __c11_atomic_compare_exchange_strong_cx16)

compiler-rt/test/builtins/Unit/atomic_lock_free_test.cc

+// RUN: %clang_builtins %s %librt -o %t && %run %t
+// REQUIRES: librt_has_atomic
+//===-- atomic_lock_free_test.c - Test is_lock_free function         ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <assert.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+bool __atomic_is_lock_free_c(unsigned long size, const volatile void *ptr);
+
+void test_is_lock_free_0(void) {
+  // size = 0. Doesn't matter what the aligment is, it should be lock-free.
+  assert(__atomic_is_lock_free_c(0, (void *)~7) && "size=0 is not lock free");
+  fprintf(stdout, "test_is_lock_free_0 PASSED\n");
+}
+
+void test_is_lock_free_aligned(void) {
+  // 1
+  char ch = 'a';
+  assert(__atomic_is_lock_free_c(1, &ch) && "size=1 is not lock free");
+
+  // 2
+  short sh __attribute__((aligned(2))) = 2;
+  assert(__atomic_is_lock_free_c(2, &sh) && "size=2 is not lock free");
+
+  struct S2 {
+    char a;
+    char b;
+  } __attribute__((aligned(2)));
+  assert(2 == sizeof(struct S2));
+  struct S2 s2;
+  assert(__atomic_is_lock_free_c(2, &s2) && "size=2 struct is not lock free");
+
+  // 4
+  int i __attribute__((aligned(4))) = 4;
+  assert(__atomic_is_lock_free_c(4, &i) && "size=4 is not lock free");
+
+  struct S4 {
+    short a;
+    short b;
+  } __attribute__((aligned(4)));
+  assert(4 == sizeof(struct S4));
+  struct S4 s4;
+  assert(__atomic_is_lock_free_c(4, &s4) && "size=4 struct is not lock free");
+
+  // 8
+#ifdef __x86_64__
+  double d __attribute__((aligned(8))) = 8.0;
+  assert(__atomic_is_lock_free_c(8, &d) && "size=8 is not lock free");
+
+  struct S8 {
+    int a;
+    int b;
+  } __attribute__((aligned(8)));
+  assert(8 == sizeof(struct S8));
+  struct S8 s8;
+  assert(__atomic_is_lock_free_c(8, &s8) && "size=8 struct is not lock free");
+
+#ifdef __SIZEOF_INT128__
+  // 16
+  struct S16 {
+    double a;
+    double b;
+
+  } __attribute__((aligned(16)));
+
+  assert(16 == sizeof(struct S16));
+  struct S16 s16;
+  assert(__atomic_is_lock_free_c(16, &s16) && "size=16 is not lock free");
+#endif // __SIZEOF_INT128__
+#endif // __X86_64__
+  fprintf(stdout, "test_is_lock_free_aligned PASSED\n");
+}
+
+void test_is_lock_free_unaligned(void) {
+  struct S3 {
+    char a;
+    char b;
+    char c;
+  } __attribute__((aligned(1)));
+  assert(3 == sizeof(struct S3));
+  struct S3 s3;
+  assert(!__atomic_is_lock_free_c(3, &s3) && "size=3 struct is lock free");
+
+  struct S34 {
+    char a;
+    char b;
+    char c;
+  } __attribute__((aligned(4)));
+  assert(4 == sizeof(struct S34));
+  struct S34 s34;
+  assert(__atomic_is_lock_free_c(4, &s34) && "size=3, aligned 4 struct is not lock free");
+
+  struct S5 {
+    char a;
+    char b;
+    char c;
+    char d;
+    char e;
+  } __attribute__((aligned(1)));
+  assert(5 == sizeof(struct S5));
+  struct S5 s5;
+  assert(!__atomic_is_lock_free_c(5, &s5) && "size=5 struct is lock free");
+
+  struct S58 {
+    char a;
+    char b;
+    char c;
+    char d;
+    char e;
+  } __attribute__((aligned(8)));
+  assert(8 == sizeof(struct S58));
+  struct S58 s58;
+  assert(__atomic_is_lock_free_c(sizeof(s58), &s58) && "size=5, aligned 8 struct is not lock free");
+
+  struct S9 {
+    char one;
+    char two;
+    char three;
+    char four;
+    char five;
+    char six;
+    char seven;
+    char eight;
+    char nine;
+  } __attribute__((aligned(1)));
+
+  assert(9 == sizeof(struct S9));
+  struct S9 s9;
+  assert(!__atomic_is_lock_free_c(9, &s9) && "size=9 is lock free");
+
+#ifdef __x86_64__
+#ifdef __SIZEOF_INT128__
+  struct S9_16 {
+    char one;
+    char two;
+    char three;
+    char four;
+    char five;
+    char six;
+    char seven;
+    char eight;
+    char nine;
+  } __attribute__((aligned(16)));
+
+  assert(16 == sizeof(struct S9_16));
+  struct S9_16 s9_16;
+  assert(__atomic_is_lock_free_c(sizeof(s9_16), &s9) && "size=9, aligned 16 struct  is not lock free");
+#endif
+#endif
+  fprintf(stdout, "test_is_lock_free_unaligned PASSED\n");
+}
+
+int main() {
+  test_is_lock_free_0();
+  test_is_lock_free_aligned();
+  test_is_lock_free_unaligned();
+}

llvm/include/llvm/Config/config.h.cmake

 #ifndef CONFIG_H
 #define CONFIG_H
 
 // Include this header only under the llvm source tree.
 // This is a private header.
 
 /* Exported configuration */
 #include "llvm/Config/llvm-config.h"
 
 /* Bug report URL. */
 #define BUG_REPORT_URL "${BUG_REPORT_URL}"
 
+#cmakedefine COMPILER_RT_HAS_AUXV ${COMPILER_RT_HAS_AUXV}
+#cmakedefine COMPILER_RT_HAS_HWCAP ${COMPILER_RT_HAS_HWCAP}
+
 /* Define to 1 to enable backtraces, and to 0 otherwise. */
 #cmakedefine01 ENABLE_BACKTRACES
 
 /* Define to 1 to enable crash overrides, and to 0 otherwise. */
 #cmakedefine01 ENABLE_CRASH_OVERRIDES
 
 /* Define to 1 to enable crash memory dumps, and to 0 otherwise. */
 #cmakedefine01 LLVM_ENABLE_CRASH_DUMPS