Add __atomic_is_lock_free to compiler-rt

[oontvoo]

Summary

This was one of the last required functions missing from Clang's libatomic.

Changes in behaviours:

• On x86-64, it'll now be able to detect and use cmpxchg16b
• The rest should be unchanged, given it'd already been able to figure it out statically.

[oontvoo]

Handled the remaining sizes

[jfb]

This is technically a behavior change, someone like @ldionne ought to chime in, and we probably want to synchronize with libstdc++ @jwakely.

[ldionne]

Can we please get a description of this change in the commit message and the Phab review?

[oontvoo]

updated

[oontvoo]

Removed use of GNU case range extension
(Triggered a bunch of clang-tidys)

[oontvoo]

• Remove the checking of non-power-of-2 sizes from is_lock_free for now (because the atomic_* ops don't handle them yet)
• Rename function to _c and add pragma redefine

[jfb]

Does this x86 change actually match what the runtime does in the same file?

[oontvoo]

Updated diff

[oontvoo]

In D85044#2213896, @jfb wrote:

Does this x86 change actually match what the runtime does in the same file?

Yes

[jfb]

Can you please add tests for this?

compiler-rt/lib/builtins/atomic.c

36

These are OS specific and won't work everywhere.

172

Please document what you're looking up here. It's definitely not "have_cas".

[oontvoo]

Updated diff

[oontvoo]

In D85044#2216023, @jfb wrote:

Can you please add tests for this?

I don't see this file being built anywhere. Can you point me to where to add the test?

[oontvoo]

Simple test

[jyknight]

There is going to be a bunch more complexity required here, I'm afraid.

Not only do we need to detect the CPU capabilities in order to implement atomic_is_lock_free, but we also need to ensure that the implementations of atomic_load_c / etc can actually use those atomic instructions as appropriate.

This means we need to compile the function with different flags, e.g. -mcx16 for x86-64, in order for the compiler to be able to emit cmpxchg16b, or -march=i586 on 32bit x86. But -- we can't compile the entire file that way. Only the functions called after the runtime detection has assured that we're running on the appropriate hardware can be built with those flags.

GCC handles this by compiling the source files multiple times, with different flags, adding some additional #defines which change the function names for each of those compiles. Then, it arranges to dispatch to the correct function depending on the runtime CPU detection. Ideally, we'd also be able to use the "target" attribute to do this on an individual function within the file, but it looks like that may not actually work properly.

compiler-rt/lib/builtins/atomic.c

34

Same check used twice? Also, I think we don't need a configure check for this, #if __has_include(<asm/hwcap.h>) would be ok.

154–156

This isn't threadsafe as written. It would be okay to use a relaxed atomic to load and store __have_atomic_cas (Relaxed is okay, because it's okay to potentially call check_x86_atomic_cas multiple times if this is reached simultaneously on multiple threads, and it will result in the same answer each time).

162–166

It's confusing to have "have_atomic_cas" sometimes mean "have CMPXCHG8b" and sometimes "have CMPXCHG16b".

And that confusion has led to a bug here -- on x86-64, cmpxchg8b is always available, but this code returns false for 8-byte atomics, unless CMPXCHG16b is also available.

177

AArch64 always supports atomics, the HWCAP_ATOMICS is needed only for the newer optimized instructions instead of the older LL/SC loop. (We may want to use those instructions for performance in the future, but it's not necessary for correctness, so we can leave out that complexity for now.

197

Let's leave aside ARM32 support for the first draft and just concentrate on x86 to get the overall framework in place.

202–203

At the end, it'll be better to fail to compile on platforms we haven't implemented, than to do the wrong thing.

But in order to allow implementing platforms piecemeal at the beginning, I'm ok to leave this fallback for now.

229

In C code (compiled with Clang), this can be spelled __attribute__((fallthrough)).

compiler-rt/test/builtins/Unit/atomic_lock_free_test.cc

3 ↗

(On Diff #286408)

Given that the contract for __atomic_is_lock_free doesn't require actual pointers, I'd leave out the construction of real objects in this test, and just pass in constructed values e.g. (void*)~7.

Also, all these assertions will need to be platform-specific.

[oontvoo]

[WIP] mov stuff to target-specific dirs

[oontvoo]

@jyknight: friendly ping ...

[oontvoo]

rebase.

Note: This has effectively undone D86510 because the patch was trying to avoid _atomic_is_lock_free, which hadn't existed then.

[ldionne]

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

[oontvoo]

In D85044#2426999, @ldionne wrote:

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

The implementation there provides the *right* answers for the cases that can be determined statically.
There are cases where it can only be determined at runtime.
For eg., unaligned non-power of 2 sizes, or more importantly 16-byte atomic ops can have lock-free implementations on some but not all x86-64 .

Most of the gunk in this patch is to set up for these checks to be extensible.
Eg., for x86/x86-64, we wants to look for CMPXCHG8B/CMPXCHG16B

Then there's the actual making use of CMPXCHG8B/CMPXCHG16B when we know it's available in *_atomic_load/store.

[ldionne]

In D85044#2427077, @oontvoo wrote:

In D85044#2426999, @ldionne wrote:

I'm definitely naive w.r..t compiler-rt, but why is a simple approach like D92302 not sufficient?

The implementation there provides the *right* answers for the cases that can be determined statically.

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it? I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

[oontvoo]

In D85044#2431334, @ldionne wrote:

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

Yes, D92302 provides an implementation of is_lock_free that is consistent with the current _atomic_load_* implementations.
(it's a no change in behaviour)

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it?

Yes.

I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

The two TODOs in this patch was the attempt to simply this so that we have the basic framework in place before filling in more meaty details (ARMs and odd sizes support) . I suspect we'd do those in subsequent patches.

[ldionne]

In D85044#2432266, @oontvoo wrote:

In D85044#2431334, @ldionne wrote:

Doesn't D92302 also check for the alignment of the pointer? My understanding is that with D92302, we're returning true from atomic_is_lock_free whenever the implementation in compiler-rt, as of D92302, would be lockfree. Whether that's the very best we can do on the hardware is a different story -- but D92302's atomic_is_lockfree will be consistent with the actual implementation. Am i mistaken?

Yes, D92302 provides an implementation of is_lock_free that is consistent with the current _atomic_load_* implementations.
(it's a no change in behaviour)

So, basically, this patch does the above, but also allows using a lockfree implementation in more cases, is that it?

Yes.

I'm trying to figure out whether it makes sense to land D92302, and then this patch, since this one seems a lot more involved (and still has TODOs). Please let me know if I'm not getting this right, I'm a bit out of my depth trying to read this patch TBH.

The two TODOs in this patch was the attempt to simply this so that we have the basic framework in place before filling in more meaty details (ARMs and odd sizes support) . I suspect we'd do those in subsequent patches.

Got it, thanks a lot for the clarifications. Unless we're confident that we can move forward with this patch in the near future, I would start by doing D92302 since it solves real problems for libc++ (and Apple platforms to the extent that we're currently shipping a broken compiler-rt). @arichardson There are some tests in this patch, perhaps you could grab them? @oontvoo would you be OK with us trying to land D92302 and then rebasing your patch on top?

[oontvoo]

Yes, sounds good. I don't see why it should wait.

[ldionne]

In D85044#2432326, @oontvoo wrote:

Yes, sounds good. I don't see why it should wait.

Excellent, thank you!

[oontvoo]

updated tests

[oontvoo]

rename aux file to .c

[oontvoo]

(typos)

[arichardson]

compiler-rt/test/builtins/Unit/atomic_lock_free_test.cc

57 ↗

(On Diff #309982)

This will fail on many 32-bit architectures (e.g. RISC-V 32)

[oontvoo]

addressed comment: restrict size-8 testing to x86-64

[ldionne]

I don't think I'll be able to contribute to this discussion meaningfully without a significant time investment since I don't touch compiler-rt a lot. Since I have a huge review queue in libc++, I'd rather resign from the review than give the impression that it's blocked on me.

compiler-rt/lib/builtins/atomic.c

 //  always acquired first, to avoid deadlock.
 //
 //===----------------------------------------------------------------------===//
 
 #include <stdbool.h>
 #include <stdint.h>
 #include <string.h>
 
+#if defined(__x86_64__) || defined(__i386__)
+#include <cpuid.h>
+#endif
+
+#if defined(__aarch64__)
+#include <asm/hwcap.h>
+#include <sys/auxv.h>
+#endif
+
 #include "assembly.h"
 
 // Clang objects if you redefine a builtin.  This little hack allows us to
 // define a function with the same name as an intrinsic.
 #pragma redefine_extname __atomic_load_c SYMBOL_NAME(__atomic_load)
 #pragma redefine_extname __atomic_store_c SYMBOL_NAME(__atomic_store)
 #pragma redefine_extname __atomic_exchange_c SYMBOL_NAME(__atomic_exchange)
 #pragma redefine_extname __atomic_compare_exchange_c SYMBOL_NAME(              \
     __atomic_compare_exchange)
+#pragma redefine_extname __atomic_is_lock_free_c SYMBOL_NAME(                  \
+    __atomic_is_lock_free)
 
 /// Number of locks.  This allocates one page on 32-bit platforms, two on
 /// 64-bit.  This can be specified externally if a different trade between
 /// memory usage and contention probability is required for a given platform.
 #ifndef SPINLOCK_COUNT
 #define SPINLOCK_COUNT (1 << 10)
 #endif
 static const long SPINLOCK_MASK = SPINLOCK_COUNT - 1;
static __inline Lock *lock_for_pointer(void *ptr) {
   // Now use the high(er) set of bits to perturb the hash, so that we don't
   // get collisions from atomic fields in a single object
   hash >>= 16;
   hash ^= low;
   // Return a pointer to the word to use
   return locks + (hash & SPINLOCK_MASK);
 }
 
-/// Macros for determining whether a size is lock free.  Clang can not yet
-/// codegen __atomic_is_lock_free(16), so for now we assume 16-byte values are
-/// not lock free.
+#if defined(__x86_64__) || defined(__i386__)
+
+#ifdef __x86_64__
+#define FEAT_REG ecx
+#define MASK bit_CMPXCHG16B
+#else
+#define FEAT_REG edx
+#define MASK bit_CMPXCHG8B
+#endif
+
+static inline bool check_x86_atomic_cas(void) {
+  unsigned int eax, ebx, ecx = 0, edx = 0;
+  __get_cpuid(1, &eax, &ebx, &ecx, &edx);
+  return (FEAT_REG & MASK) != 0;
+}
+
+static inline bool have_cas(int N) {
+  static int __have_atomic_cas = -1;
+  if (__have_atomic_cas == -1) {
+    __have_atomic_cas = check_x86_atomic_cas() != 0 ? 1 : 0;
+  }
+  switch (N) {
+  case 1:
+  case 2:
+  case 4:
+    return true;
+  case 8:
+#ifdef __x86_64__
+  case 16:
+#endif
+    return __have_atomic_cas;
+  }
+  return false;
+}
+#elif defined(__aarch64__)
+static inline bool have_cas(int N) {
+  static int __has_atomic_cap = -1;
+  if (__have_atomic_cap == -1) {
+    __have_atomic_cap = (getauxval(AT_HWCAP) & HWCAP_ATOMICS) != 0 ? 1 : 0;
+  }
+  switch (N) {
+  case 1:
+  case 2:
+  case 4:
+  case 8:
+    return __have_atomic_cap;
+  }
+  return false;
+}
+#elif defined(__arm__)
+static inline bool have_cas(int N) {
+  switch (N) {
+  case 1:
+  case 2:
+  case 4:
+  case 8:
+    return false; // FIXME: not sure the check similar to aarch64 works
+  }
+  return false;
+}
+#else
+static inline bool have_cas(int) { return false; }
+#endif
+
+// Return true if it could  positively be determined to be lock free.
+// Otherwise, fall through to the next bucket (next power-of-2).
+#define CHECK_LOCK_FREE_POW2(N)                                                \
+  do {                                                                         \
+    uintptr_t r = (uintptr_t)ptr & (N - 1);                                    \
+    if (r != 0)                                                                \
+      break;                                                                   \
+    if (__atomic_always_lock_free(N, 0))                                       \
+      return true;                                                             \
+    if (have_cas(N))                                                           \
+      return true;                                                             \
+  } while (0)
+
+bool __atomic_is_lock_free_c(unsigned long size, const volatile void *ptr) {
+  // FIXME: We don't support non-power-of-2 sizes now. They could be handled
+  // by rounding up to the next power-of-2 bucket. But all the __atomic_*
+  // operations will need to do the same thing as well.
+  switch (size) {
+  case 0:
+    return true;
+  case 2:
+    CHECK_LOCK_FREE_POW2(2);
+    [[clang::fallthrough]];
+  case 4:
+    CHECK_LOCK_FREE_POW2(4);
+    [[clang::fallthrough]];
+  case 8:
+    CHECK_LOCK_FREE_POW2(8);
+    [[clang::fallthrough]];
+  case 16:
+    CHECK_LOCK_FREE_POW2(16);
+    break;
+  }
+  return false;
+}
+
+/// Macros for determining whether a size is lock free.
 #define IS_LOCK_FREE_1 __c11_atomic_is_lock_free(1)
 #define IS_LOCK_FREE_2 __c11_atomic_is_lock_free(2)
 #define IS_LOCK_FREE_4 __c11_atomic_is_lock_free(4)
 #define IS_LOCK_FREE_8 __c11_atomic_is_lock_free(8)
+#ifdef __SIZEOF_INT128__
+#define IS_LOCK_FREE_16 __c11_atomic_is_lock_free(16)
+#define HANDLE_CASE_16                                                         \
+  if (IS_LOCK_FREE_16) {                                                       \
+    LOCK_FREE_ACTION(__uint128_t);                                             \
+  }                                                                            
+#else
 #define IS_LOCK_FREE_16 0
-
+#define HANDLE_CASE_16
+#endif
 /// Macro that calls the compiler-generated lock-free versions of functions
 /// when they exist.
 #define LOCK_FREE_CASES()                                                      \
   do {                                                                         \
     switch (size) {                                                            \
     case 1:                                                                    \
       if (IS_LOCK_FREE_1) {                                                    \
         LOCK_FREE_ACTION(uint8_t);                                             \
case 4:                                                                    \
       }                                                                        \
       break;                                                                   \
     case 8:                                                                    \
       if (IS_LOCK_FREE_8) {                                                    \
         LOCK_FREE_ACTION(uint64_t);                                            \
       }                                                                        \
       break;                                                                   \
     case 16:                                                                   \
-      if (IS_LOCK_FREE_16) {                                                   \
-        /* FIXME: __uint128_t isn't available on 32 bit platforms.             \
-        LOCK_FREE_ACTION(__uint128_t);*/                                       \
-      }                                                                        \
+      /* Special handling because not all platforms have uint_128*/            \
+      HANDLE_CASE_16                                                           \
       break;                                                                   \
     }                                                                          \
   } while (0)
 
 /// An atomic load operation.  This is atomic with respect to the source
 /// pointer only.
 void __atomic_load_c(int size, void *src, void *dest, int model) {
 #define LOCK_FREE_ACTION(type)                                                 \