[clang][compiler-rt][atomics] Add `__c11_atomic_fetch_nand` builtin and support `__atomic_fetch_nand` libcall

[lkail]

Summary

Add __c11_atomic_fetch_nand builtin to language extensions and support __atomic_fetch_nand libcall in compiler-rt.

[theraven]

Looks good to me, thanks!

I missed the clang patches, but it looks as if they didn't update clang's LanguageExtensions.html to document the new builtins - please can you add them?

[lkail]

I missed the clang patches, but it looks as if they didn't update clang's LanguageExtensions.html to document the new builtins - please can you add them?

Looks __atomic_fetch_nand is a GNU compatible builtin and it has been mentioned roughly since https://reviews.llvm.org/D10847#C226804NL1718. Do we have to document these GNU compatible __atomic_* builtins in another section?

[theraven]

Ah, it looks as if you didn't add the __c11_ variant of the builtin? Please can you add this?

The GCC builtins paint the ABI into a corner. They accept non-_Atomic-qualified types (the C11 spec guarantees only that these operations work on _Atomic types). The goal of the original C++ specification was to allow implementations to use atomic operations for register-sized chunks and fall back to an implementation with an inline lock for larger types, so std::atomic<T> would either have a single field of T or a std::atomic_flag field and a T field, depending on the size of T. The goal of the C11 import was to allow _Atomic(T) to be ABI-compatible with std::atomic<T>. Implementing this requires that _Atomic(T) be allowed to have a different representation to T. GCC messed this up and defined builtins that took a T*, not an _Atomic(T*), which forced all GCC-compatible ABIs to have the same representation for T and _Atomic(T). This, in turn, meant that the atomics support library couldn't use inline locks, and had to maintain a pool of locks to use for different types. This, in turn, means that _Atomic(T) silently fails in surprising ways in shared memory, some of the time, depending on the target CPU. This is a horrible mess and I would like to ensure that we always provide builtins that allow target ABIs to do the right thing, even if Linux and *BSD are trapped in a broken ABI by GCC and legacy compatibility.

[lkail]

Add __c11_atomic_fetch_nand in clang; Use __c11_atomic_fetch_nand to implement lock-free __atomic_fetch_nand libcall; Updated LanguageExtensions.rst.

[lkail]

The GCC builtins paint the ABI into a corner. They accept non-_Atomic-qualified types (the C11 spec guarantees only that these operations work on _Atomic types). The goal of the original C++ specification was to allow implementations to use atomic operations for register-sized chunks and fall back to an implementation with an inline lock for larger types, so std::atomic<T> would either have a single field of T or a std::atomic_flag field and a T field, depending on the size of T. The goal of the C11 import was to allow _Atomic(T) to be ABI-compatible with std::atomic<T>. Implementing this requires that _Atomic(T) be allowed to have a different representation to T. GCC messed this up and defined builtins that took a T*, not an _Atomic(T*), which forced all GCC-compatible ABIs to have the same representation for T and _Atomic(T). This, in turn, meant that the atomics support library couldn't use inline locks, and had to maintain a pool of locks to use for different types. This, in turn, means that _Atomic(T) silently fails in surprising ways in shared memory, some of the time, depending on the target CPU. This is a horrible mess and I would like to ensure that we always provide builtins that allow target ABIs to do the right thing, even if Linux and *BSD are trapped in a broken ABI by GCC and legacy compatibility.

Good to know, thanks for your detailed explanation!

[theraven]

Thanks!

[jyknight]

compiler-rt/lib/builtins/atomic.c

339

Same as ATOMIC_RMW now, isn't it?

[lkail]

compiler-rt/lib/builtins/atomic.c

339

Not totally. The ATOMIC_RMW macro also accept binary op sign as its argument, i.e., in the form a op b. However, nand is ~(a & b).

clang/docs/LanguageExtensions.rst

 * ``__c11_atomic_exchange``
 * ``__c11_atomic_compare_exchange_strong``
 * ``__c11_atomic_compare_exchange_weak``
 * ``__c11_atomic_fetch_add``
 * ``__c11_atomic_fetch_sub``
 * ``__c11_atomic_fetch_and``
 * ``__c11_atomic_fetch_or``
 * ``__c11_atomic_fetch_xor``
+* ``__c11_atomic_fetch_nand`` (Nand is not presented in <stdatomic.h>)
 * ``__c11_atomic_fetch_max``
 * ``__c11_atomic_fetch_min``
 
 The macros ``__ATOMIC_RELAXED``, ``__ATOMIC_CONSUME``, ``__ATOMIC_ACQUIRE``,
 ``__ATOMIC_RELEASE``, ``__ATOMIC_ACQ_REL``, and ``__ATOMIC_SEQ_CST`` are
 provided, with values corresponding to the enumerators of C11's
 ``memory_order`` enumeration.
 

clang/include/clang/Basic/Builtins.def

 ATOMIC_BUILTIN(__c11_atomic_exchange, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_compare_exchange_strong, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_compare_exchange_weak, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_add, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_sub, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_and, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_or, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_xor, "v.", "t")
+ATOMIC_BUILTIN(__c11_atomic_fetch_nand, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_max, "v.", "t")
 ATOMIC_BUILTIN(__c11_atomic_fetch_min, "v.", "t")
 BUILTIN(__c11_atomic_thread_fence, "vi", "n")
 BUILTIN(__c11_atomic_signal_fence, "vi", "n")
 BUILTIN(__c11_atomic_is_lock_free, "bz", "n")
 
 // GNU atomic builtins.
 ATOMIC_BUILTIN(__atomic_load, "v.", "t")

clang/lib/AST/Expr.cpp

unsigned AtomicExpr::getNumSubExprs(AtomicOp Op) {
   case AO__atomic_store:
   case AO__atomic_store_n:
   case AO__atomic_exchange_n:
   case AO__c11_atomic_fetch_add:
   case AO__c11_atomic_fetch_sub:
   case AO__c11_atomic_fetch_and:
   case AO__c11_atomic_fetch_or:
   case AO__c11_atomic_fetch_xor:
+  case AO__c11_atomic_fetch_nand:
   case AO__c11_atomic_fetch_max:
   case AO__c11_atomic_fetch_min:
   case AO__atomic_fetch_add:
   case AO__atomic_fetch_sub:
   case AO__atomic_fetch_and:
   case AO__atomic_fetch_or:
   case AO__atomic_fetch_xor:
   case AO__atomic_fetch_nand:

clang/lib/CodeGen/CGAtomic.cpp

static void EmitAtomicOp(CodeGenFunction &CGF, AtomicExpr *E, Address Dest,
   case AtomicExpr::AO__opencl_atomic_fetch_xor:
   case AtomicExpr::AO__atomic_fetch_xor:
     Op = llvm::AtomicRMWInst::Xor;
     break;
 
   case AtomicExpr::AO__atomic_nand_fetch:
     PostOp = llvm::Instruction::And; // the NOT is special cased below
     LLVM_FALLTHROUGH;
+  case AtomicExpr::AO__c11_atomic_fetch_nand:
   case AtomicExpr::AO__atomic_fetch_nand:
     Op = llvm::AtomicRMWInst::Nand;
     break;
   }
 
   llvm::Value *LoadVal1 = CGF.Builder.CreateLoad(Val1);
   llvm::AtomicRMWInst *RMWI =
       CGF.Builder.CreateAtomicRMW(Op, Ptr.getPointer(), LoadVal1, Order, Scope);
RValue CodeGenFunction::EmitAtomicExpr(AtomicExpr *E) {
   case AtomicExpr::AO__c11_atomic_exchange:
   case AtomicExpr::AO__opencl_atomic_store:
   case AtomicExpr::AO__opencl_atomic_exchange:
   case AtomicExpr::AO__atomic_store_n:
   case AtomicExpr::AO__atomic_exchange_n:
   case AtomicExpr::AO__c11_atomic_fetch_and:
   case AtomicExpr::AO__c11_atomic_fetch_or:
   case AtomicExpr::AO__c11_atomic_fetch_xor:
+  case AtomicExpr::AO__c11_atomic_fetch_nand:
   case AtomicExpr::AO__c11_atomic_fetch_max:
   case AtomicExpr::AO__c11_atomic_fetch_min:
   case AtomicExpr::AO__opencl_atomic_fetch_and:
   case AtomicExpr::AO__opencl_atomic_fetch_or:
   case AtomicExpr::AO__opencl_atomic_fetch_xor:
   case AtomicExpr::AO__opencl_atomic_fetch_min:
   case AtomicExpr::AO__opencl_atomic_fetch_max:
   case AtomicExpr::AO__atomic_fetch_and:
if (UseLibcall) {
     case AtomicExpr::AO__opencl_atomic_fetch_add:
     case AtomicExpr::AO__atomic_fetch_add:
     case AtomicExpr::AO__c11_atomic_fetch_and:
     case AtomicExpr::AO__opencl_atomic_fetch_and:
     case AtomicExpr::AO__atomic_fetch_and:
     case AtomicExpr::AO__c11_atomic_fetch_or:
     case AtomicExpr::AO__opencl_atomic_fetch_or:
     case AtomicExpr::AO__atomic_fetch_or:
+    case AtomicExpr::AO__c11_atomic_fetch_nand:
     case AtomicExpr::AO__atomic_fetch_nand:
     case AtomicExpr::AO__c11_atomic_fetch_sub:
     case AtomicExpr::AO__opencl_atomic_fetch_sub:
     case AtomicExpr::AO__atomic_fetch_sub:
     case AtomicExpr::AO__c11_atomic_fetch_xor:
     case AtomicExpr::AO__opencl_atomic_fetch_xor:
     case AtomicExpr::AO__opencl_atomic_fetch_min:
     case AtomicExpr::AO__opencl_atomic_fetch_max:
case AtomicExpr::AO__opencl_atomic_fetch_max:
       AddDirectArgument(*this, Args, UseOptimizedLibcall, Val1.getPointer(),
                         LoweredMemTy, E->getExprLoc(), TInfo.Width);
       break;
     // T __atomic_nand_fetch_N(T *mem, T val, int order)
     // T __atomic_fetch_nand_N(T *mem, T val, int order)
     case AtomicExpr::AO__atomic_nand_fetch:
       PostOp = llvm::Instruction::And; // the NOT is special cased below
       LLVM_FALLTHROUGH;
+    case AtomicExpr::AO__c11_atomic_fetch_nand:
     case AtomicExpr::AO__atomic_fetch_nand:
       LibCallName = "__atomic_fetch_nand";
       AddDirectArgument(*this, Args, UseOptimizedLibcall, Val1.getPointer(),
                         MemTy, E->getExprLoc(), TInfo.Width);
       break;
     }
 
     if (E->isOpenCL()) {

clang/lib/Sema/SemaChecking.cpp

ExprResult Sema::BuildAtomicExpr(SourceRange CallRange, SourceRange ExprRange,
   case AtomicExpr::AO__atomic_add_fetch:
   case AtomicExpr::AO__atomic_sub_fetch:
     IsAddSub = true;
     Form = Arithmetic;
     break;
   case AtomicExpr::AO__c11_atomic_fetch_and:
   case AtomicExpr::AO__c11_atomic_fetch_or:
   case AtomicExpr::AO__c11_atomic_fetch_xor:
+  case AtomicExpr::AO__c11_atomic_fetch_nand:
   case AtomicExpr::AO__opencl_atomic_fetch_and:
   case AtomicExpr::AO__opencl_atomic_fetch_or:
   case AtomicExpr::AO__opencl_atomic_fetch_xor:
   case AtomicExpr::AO__atomic_fetch_and:
   case AtomicExpr::AO__atomic_fetch_or:
   case AtomicExpr::AO__atomic_fetch_xor:
   case AtomicExpr::AO__atomic_fetch_nand:
   case AtomicExpr::AO__atomic_and_fetch:

clang/test/Sema/atomic-implicit-seq_cst.c

 int bad_bitor_1(int i) {
   return atom | i; // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
 }
 
 int bad_bitor_2(int i) {
   return i | atom; // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
 }
 
+int bad_bitnand_1(int i) {
+  return ~(atom & i); // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
+}
+
+int bad_bitnand_2(int i) {
+  return ~(i & atom); // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
+}
+
 int bad_and_1(int i) {
   return atom && i; // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
 }
 
 int bad_and_2(int i) {
   return i && atom; // expected-warning {{implicit use of sequentially-consistent atomic may incur stronger memory barriers than necessary}}
 }
 

 int good_c11_atomic_exchange(int i) { return __c11_atomic_exchange(&atom, i, __ATOMIC_RELAXED); }
 int good_c11_atomic_compare_exchange_strong(int *e, int i) { return __c11_atomic_compare_exchange_strong(&atom, e, i, __ATOMIC_RELAXED, __ATOMIC_RELAXED); }
 int good_c11_atomic_compare_exchange_weak(int *e, int i) { return __c11_atomic_compare_exchange_weak(&atom, e, i, __ATOMIC_RELAXED, __ATOMIC_RELAXED); }
 int good_c11_atomic_fetch_add(int i) { return __c11_atomic_fetch_add(&atom, i, __ATOMIC_RELAXED); }
 int good_c11_atomic_fetch_sub(int i) { return __c11_atomic_fetch_sub(&atom, i, __ATOMIC_RELAXED); }
 int good_c11_atomic_fetch_and(int i) { return __c11_atomic_fetch_and(&atom, i, __ATOMIC_RELAXED); }
 int good_c11_atomic_fetch_or(int i) { return __c11_atomic_fetch_or(&atom, i, __ATOMIC_RELAXED); }
 int good_c11_atomic_fetch_xor(int i) { return __c11_atomic_fetch_xor(&atom, i, __ATOMIC_RELAXED); }
+int good_c11_atomic_fetch_nand(int i) { return __c11_atomic_fetch_nand(&atom, i, __ATOMIC_RELAXED); }
 
 void good_cast_to_void(void) { (void)atom; }
 _Atomic(int) * good_address_of(void) { return &atom; }
 int good_sizeof(void) { return sizeof(atom); }
 _Atomic(int) * good_pointer_arith(_Atomic(int) * p) { return p + 10; }
 _Bool good_pointer_to_bool(_Atomic(int) * p) { return p; }
 void good_no_init(void) { _Atomic(int) no_init; }
 void good_init(void) { _Atomic(int) init = 42; }

clang/test/Sema/atomic-ops.c

void memory_checks(_Atomic(int) *Ap, int *p, int val) {
 
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_relaxed);
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_acquire);
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_consume);
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_release);
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_acq_rel);
   (void)__c11_atomic_fetch_xor(Ap, val, memory_order_seq_cst);
 
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_relaxed);
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_acquire);
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_consume);
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_release);
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_acq_rel);
+  (void)__c11_atomic_fetch_nand(Ap, val, memory_order_seq_cst);
+
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_relaxed);
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_acquire);
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_consume);
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_release);
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_acq_rel);
   (void)__c11_atomic_fetch_min(Ap, val, memory_order_seq_cst);
 
   (void)__c11_atomic_fetch_max(Ap, val, memory_order_relaxed);
void nullPointerWarning() {
   (void)__c11_atomic_fetch_sub((volatile _Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_sub((_Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_and((volatile _Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_and((_Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_or((volatile _Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_or((_Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_xor((volatile _Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__c11_atomic_fetch_xor((_Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
+  (void)__c11_atomic_fetch_nand((volatile _Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
+  (void)__c11_atomic_fetch_nand((_Atomic(int)*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
 
   __atomic_store_n((volatile int*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   __atomic_store_n((int*)0, 42, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   __atomic_store((volatile int*)0, &i, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   __atomic_store((int*)0, &i, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   __atomic_store(&vi, (int*)0, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   __atomic_store(&i, (int*)0, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
   (void)__atomic_load_n((volatile int*)0, memory_order_relaxed); // expected-warning {{null passed to a callee that requires a non-null argument}}
void nullPointerWarning() {
   (void)__c11_atomic_fetch_sub(&vai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_sub(&ai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_and(&vai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_and(&ai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_or(&vai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_or(&ai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_xor(&vai, 0, memory_order_relaxed);
   (void)__c11_atomic_fetch_xor(&ai, 0, memory_order_relaxed);
+  (void)__c11_atomic_fetch_nand(&vai, 0, memory_order_relaxed);
+  (void)__c11_atomic_fetch_nand(&ai, 0, memory_order_relaxed);
 
   // Ditto.
   __atomic_store_n(&vi, 0, memory_order_relaxed);
   __atomic_store_n(&i, 0, memory_order_relaxed);
   __atomic_store_n(&vp, NULL, memory_order_relaxed);
   __atomic_store_n(&p, NULL, memory_order_relaxed);
   (void)__atomic_exchange_n(&vi, 0, memory_order_relaxed);
   (void)__atomic_exchange_n(&i, 0, memory_order_relaxed);

compiler-rt/lib/builtins/atomic.c

type __atomic_fetch_##opname##_##n(type *ptr, type val, int model) {         \
     Lock *l = lock_for_pointer(ptr);                                           \
     lock(l);                                                                   \
     type tmp = *ptr;                                                           \
     *ptr = tmp op val;                                                         \
     unlock(l);                                                                 \
     return tmp;                                                                \
   }
 
+#define ATOMIC_RMW_NAND(n, lockfree, type)                                     \
+  type __atomic_fetch_nand_##n(type *ptr, type val, int model) {               \
+    if (lockfree(ptr))                                                         \
+      return __c11_atomic_fetch_nand((_Atomic(type) *)ptr, val, model);        \
+    Lock *l = lock_for_pointer(ptr);                                           \
+    lock(l);                                                                   \
+    type tmp = *ptr;                                                           \
+    *ptr = ~(tmp & val);                                                       \
+    unlock(l);                                                                 \
+    return tmp;                                                                \
+  }
+
 #define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW(n, lockfree, type, add, +)
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 #define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW(n, lockfree, type, sub, -)
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 #define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW(n, lockfree, type, and, &)
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 #define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW(n, lockfree, type, or, |)
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
 #define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW(n, lockfree, type, xor, ^)
 OPTIMISED_CASES
 #undef OPTIMISED_CASE
+#define OPTIMISED_CASE(n, lockfree, type) ATOMIC_RMW_NAND(n, lockfree, type)
+OPTIMISED_CASES
+#undef OPTIMISED_CASE

compiler-rt/test/builtins/Unit/atomic_test.c

 uint32_t __atomic_fetch_or_4(uint32_t *ptr, uint32_t val, int model);
 uint64_t __atomic_fetch_or_8(uint64_t *ptr, uint64_t val, int model);
 
 uint8_t __atomic_fetch_xor_1(uint8_t *ptr, uint8_t val, int model);
 uint16_t __atomic_fetch_xor_2(uint16_t *ptr, uint16_t val, int model);
 uint32_t __atomic_fetch_xor_4(uint32_t *ptr, uint32_t val, int model);
 uint64_t __atomic_fetch_xor_8(uint64_t *ptr, uint64_t val, int model);
 
+uint8_t __atomic_fetch_nand_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_nand_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_nand_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_nand_8(uint64_t *ptr, uint64_t val, int model);
+
 // We conditionally test the *_16 atomic function variants based on the same
 // condition that compiler_rt (atomic.c) uses to conditionally generate them.
 // Currently atomic.c tests if __SIZEOF_INT128__ is defined (which can be the
 // case on 32-bit platforms, by using -fforce-enable-int128), instead of using
 // CRT_HAS_128BIT.
 
 #ifdef __SIZEOF_INT128__
 #define TEST_16
 #endif
 
 #ifdef TEST_16
 typedef __uint128_t uint128_t;
 typedef uint128_t maxuint_t;
 uint128_t __atomic_load_16(uint128_t *src, int model);
 void __atomic_store_16(uint128_t *dest, uint128_t val, int model);
 uint128_t __atomic_exchange_16(uint128_t *dest, uint128_t val, int model);
 bool __atomic_compare_exchange_16(uint128_t *ptr, uint128_t *expected,
                                   uint128_t desired, int success, int failure);
 uint128_t __atomic_fetch_add_16(uint128_t *ptr, uint128_t val, int model);
 uint128_t __atomic_fetch_sub_16(uint128_t *ptr, uint128_t val, int model);
 uint128_t __atomic_fetch_and_16(uint128_t *ptr, uint128_t val, int model);
 uint128_t __atomic_fetch_or_16(uint128_t *ptr, uint128_t val, int model);
 uint128_t __atomic_fetch_xor_16(uint128_t *ptr, uint128_t val, int model);
+uint128_t __atomic_fetch_nand_16(uint128_t *ptr, uint128_t val, int model);
 #else
 typedef uint64_t maxuint_t;
 #endif
 
 #define U8(value) ((uint8_t)(value))
 #define U16(value) ((uint16_t)(value))
 #define U32(value) ((uint32_t)(value))
 #define U64(value) ((uint64_t)(value))
#endif
     if (b64 != U64(V + m) || a64 != U64((V + m) ^ ONES))
       abort();
 #ifdef TEST_16
     b128 = __atomic_fetch_xor_16(&a128, ONES, model);
     if (b128 != (V + m) || a128 != ((V + m) ^ ONES))
       abort();
 #endif
 
+    // Fetch nand.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_nand_1(&a8, U8(ONES), model);
+    if (b8 != U8(V + m) || a8 != U8(~((V + m) & ONES)))
+      abort();
+    b16 = __atomic_fetch_nand_2(&a16, U16(ONES), model);
+    if (b16 != U16(V + m) || a16 != U16(~((V + m) & ONES)))
+      abort();
+    b32 = __atomic_fetch_nand_4(&a32, U32(ONES), model);
+    if (b32 != U32(V + m) || a32 != U32(~((V + m) & ONES)))
+      abort();
+    b64 = __atomic_fetch_nand_8(&a64, U64(ONES), model);
+    if (b64 != U64(V + m) || a64 != U64(~((V + m) & ONES)))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_nand_16(&a128, ONES, model);
+    if (b128 != (V + m) || a128 != ~((V + m) & ONES))
+      abort();
+#endif
+
     // Check signed integer overflow behavior
 
     set_a_values(V + m);
     __atomic_fetch_add_1(&a8, U8(V), model);
     if (a8 != U8(V * 2 + m))
       abort();
     __atomic_fetch_sub_1(&a8, U8(V), model);
     if (a8 != U8(V + m))