[compiler-rt][builtins] Add tests for atomic builtins support functions

[luismarques]

Summary

Adds some simple sanity checks that the support functions for the atomic
builtins do the right thing. This doesn't test concurrency and memory model
issues.

[asb]

I think there's value in adding these tests (I note libatomic in GCC has a similar set of simple functionality tests), and this seems a sensible way to do it.

I added one inline comment regarding handling the existing builtin. Could you please confirm if the pre-commit test failures are real or spurious build infra related issues?

It might be useful to add some comments alongside the blocks for the atomic fetch_op tests to make it a bit clearer what is being compared.

It would be good if someone who works more actively with atomics could review or at least comment. @jfb @jyknight - any thoughts?

compiler-rt/test/builtins/Unit/atomic_test.c

27

Did you consider using the same #pragma redefine_extname hack as in lib/builtins/atomic.c?

[luismarques]

compiler-rt/test/builtins/Unit/atomic_test.c

27

I discarded that option because the documentation for the #pragma redefine_extname indicates that the asm labels are a better tool all around. See this document for details: https://gcc.gnu.org/onlinedocs/gcc/Symbol-Renaming-Pragmas.html

[luismarques]

Add comment about not resetting the test variables between the tests of different operations in the section that tests __atomic_fetch_<op>_* stuff.

[luismarques]

Correctly gate all instances of 128-bit variables and tests.

[luismarques]

In D81348#2081781, @asb wrote:

Could you please confirm if the pre-commit test failures are real or spurious build infra related issues?

I'm only seeing seemingly unrelated errors. These have persisted even after updating the patch, so it seems it's not something very transient. I think that a buildbot maintainer will have to manually fix some build configuration parameters for these to go away, as these will probably affect other people.

[jfb]

You should probably also test overflow behavior for the RMW variants. Specifically, this clause:

Remarks: For signed integer types, the result is as if the object value and parameters were converted to their corresponding unsigned types, the computation performed on those types, and the result converted back to the signed type. [ Note: There are no undefined results arising from the computation. — end note ]

compiler-rt/test/builtins/Unit/atomic_test.c

126

Do you mean to test alignment properties of the functions? Here you're getting any alignment, and it might be worth checking what the behavior is for aligned as well as unaligned variants of the functions?

154

Same comment on alignment.

197

Here and other places, do you want to test return values?

281

The only requirements here are: The failure argument shall not be memory_order_release nor memory_order_acq_rel.

See: http://wg21.link/p0418

[jyknight]

This isn't sufficient to verify that atomic.c actually works properly (which, BTW, it does not). But at least verifying that the basic single-threaded functionality semantics hold is a good start.

compiler-rt/test/builtins/Unit/atomic_test.c

120

Reading the code below, I was wondering where the "v" variable came from. It looked like it should be a local variable name like "m" so I was confused to find it up here in a #define.

141

I would suggest splitting this into 5 different test functions (for load, store, exchange, compare_exchange, fetch_op) for clarity.

153

All of the size-generic functions (__atomic_load, __atomic_store, __atomic_exchange, __atomic_compare_exchange) are required to handle arbitrary alignments.

It would be good to test that. Maybe add an __attribute__((aligned(16))) to the "data" array to make sure it's aligned. And then, do this loop twice -- once to test aligned, with "data" as the address, and once to test misaligned, with data+1 as the address. (However, atomic.c doesn't currently handle the misaligned case properly, so it may fail, depending on the CPU architecture.)

156

I was going to suggest using gtest and EXPECT_EQ(x, y) instead of if(x != y) abort(). However, given that this code is following the existing style of other tests in this directory (none of which use gtest), I won't push it.

[luismarques]

In D81348#2082596, @jfb wrote:

You should probably also test overflow behavior for the RMW variants. Specifically, this clause:

Remarks: For signed integer types, the result is as if the object value and parameters were converted to their corresponding unsigned types, the computation performed on those types, and the result converted back to the signed type. [ Note: There are no undefined results arising from the computation. — end note ]

Thanks, that's useful to know. I'll update the patch with that, the __atomic_compare_exchange_* tests (which I forgot), the void return types for the atomic store functions, and a conservative alignment for data.
If I misinterpreted your comment about the alignment please clarify (you said "unaligned variants of the *functions*").

compiler-rt/test/builtins/Unit/atomic_test.c

126

Good point about the data being "unaligned". Is it even valid to try to call those atomic functions on unaligned values? I'd normally assume that you need things to be naturally aligned, although the documentation of __atomic_is_lock_free and __atomic_always_lock_free do make me question that a bit. Assuming you need to align things properly, I'll force data to be aligned to N.

197

Oh, that should return void, that was a copy-paste error. Well spotted!

281

Great! My searches missed that report, which nicely clarifies the vague wording on the documentation.

[luismarques]

compiler-rt/test/builtins/Unit/atomic_test.c

153

Thanks, that does clarify jfb's comments. The suggested approach makes sense to me. BTW, where are the requirements about handling unaligned data documented? Not that I'm doubting you, but I had been mostly looking at https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html, which seems omissive about that, so it would be good to know where to look instead.

156

I have no problem with updating the patch to use the gtest macros, but I would like some guidance on whether that is wise to do, given that the other compiler_rt tests don't use it (e.g. are we introducing problematic dependencies, etc.).

[luismarques]

Address review concerns. Various minor fixes and improvements.

[luismarques]

Minor fixes.

[jfb]

compiler-rt/test/builtins/Unit/atomic_test.c

29

Mark src as const.

37

Mark src as const.

132

This should be const.

222

Here you want to do the same alignment thing as above.

230

Ditto on alignment.

[jyknight]

compiler-rt/test/builtins/Unit/atomic_test.c

153

https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary
and also https://llvm.org/docs/Atomics.html#libcalls-atomic

156

I don't actually know if there's an active reason the other code in this directory doesn't use it, or if it's just history. So, I'm afraid I don't know if switching this test to using gtest would cause a problem. Sorry. So maybe best to just keep it like you have. =)

It is used for the sanitizer tests in compiler-rt, *shrug*.

[luismarques]

Add more consts and alignments.

[luismarques]

Some final tweaks to the unaligned tests. I changed my mind about this when I was about to mark some review comments as done. Sorry about the churn!

[luismarques]

compiler-rt/test/builtins/Unit/atomic_test.c

132

Thanks for the suggestion of adding const here. That has already revealed an issue with the GNU libatomic implementation! The following crashes:

struct S {
    int x;
    int y;
    int z;
};

int main() {
    __attribute__((aligned(16))) static const _Atomic struct S a;
    struct S b;
    b = a;
}

The alignment attribute is not needed when compiling with Clang, which already uses that alignment by default, so it's easier to run into this issue when compiling with Clang than with GCC.

GNU bug report: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95722

[jfb]

compiler-rt/test/builtins/Unit/atomic_test.c

132

We've also had issues in LLVM w.r.t. emitting cmpxchg when it's the only way to get a single-copy atomic load for a wide type. I can't find the bugs I have in mind, maybe @t.p.northover remembers? We don't always use FP operations, or non-tearing paired loads, when we see such atomic loads. Sometimes we can't use FP, and sometimes we don't know if paired loads tear or not.

On the GCC side, I know @jwakely was looking at this type of issue earlier this week.

Generally C and C++ aren't very good at this specific issue (whether large const atomic can use cmpxchg, and therefore fault when you load from them).

So I guess for load tests, don't make data const. Only do so for tests where we're not accessing data atomically...

[jyknight]

compiler-rt/test/builtins/Unit/atomic_test.c

132

GCC switched away from directly emitting 16-byte atomics on x86 even with -mcx16, exactly because there's no atomic load instruction. (Clang should really do this too.)

Not only does that break if used on read-only memory, emitting a write when the user expected only a read access ruins the performance of many lock-free algorithms.

Unfortunately, although they stopped emitting the instructions inline, for compatibility with code produced by older GCC versions (and clang), libatomic _must_ continue implementing 16byte atomic loads with cmpxchg16b, rather than using an external mutex.

So....yeah. Sigh.

[jwakely]

compiler-rt/test/builtins/Unit/atomic_test.c

132

What I've been looking at is https://gcc.gnu.org/PR95378 and the related question of why Clang allows the 3rd argument to __atomic_exchange to be volatile T* iff the first argument is volatile T*. It doesn't seem useful (and possibly not even correct) to use a pointer to volatile for that argument.

I don't have anything useful to add here though, sorry.

[lenary]

compiler-rt/test/builtins/Unit/atomic_test.c

189

I think you need a +1 in the declared array length here.

[efriedma]

llvm-commits is not CC'ed on this patch; please resubmit.

[luismarques]

compiler-rt/test/builtins/Unit/atomic_test.c

189

In the unaligned test the loop condition is n < LEN(data) (without the equals, unlike the aligned test), so the largest access we do is LEN(data) - 1. For instance, if data has size 16, we only copy 15 bytes, so we have margin to start a byte further in. Am I missing something?

[luismarques]

This is superseded by D86278, due to issues with subscribing llvm-commits.

[lenary]

compiler-rt/test/builtins/Unit/atomic_test.c

189

No, I missed the comparison was < not <=. Sorry

compiler-rt/test/builtins/Unit/atomic_test.c

+// RUN: %clang_builtins %s %librt -o %t && %run %t
+// REQUIRES: librt_has_atomic
+//===-- atomic_test.c - Test support functions for atomic operations ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file performs some simple testing of the support functions for the
+// atomic builtins. All tests are single-threaded, so this is only a sanity
+// check.
+//
+//===----------------------------------------------------------------------===//
+
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+// We directly test the library atomic functions, not using the C builtins. This
+// should avoid confounding factors, ensuring that we actually test the
+// functions themselves, regardless of how the builtins are lowered. We need to
+// use asm labels because we can't redeclare the builtins.
+
+void __atomic_load_c(int size, void *src, void *dest,
+                     int model) asm("__atomic_load");
+
+uint8_t __atomic_load_1(uint8_t *src, int model);
+uint16_t __atomic_load_2(uint16_t *src, int model);
+uint32_t __atomic_load_4(uint32_t *src, int model);
+uint64_t __atomic_load_8(uint64_t *src, int model);
+
+void __atomic_store_c(int size, void *dest, void *src,
+                      int model) asm("__atomic_store");
+
+void __atomic_store_1(uint8_t *dest, uint8_t val, int model);
+void __atomic_store_2(uint16_t *dest, uint16_t val, int model);
+void __atomic_store_4(uint32_t *dest, uint32_t val, int model);
+void __atomic_store_8(uint64_t *dest, uint64_t val, int model);
+
+void __atomic_exchange_c(int size, void *ptr, void *val, void *old, int model)
+                         asm("__atomic_exchange");
+
+uint8_t __atomic_exchange_1(uint8_t *dest, uint8_t val, int model);
+uint16_t __atomic_exchange_2(uint16_t *dest, uint16_t val, int model);
+uint32_t __atomic_exchange_4(uint32_t *dest, uint32_t val, int model);
+uint64_t __atomic_exchange_8(uint64_t *dest, uint64_t val, int model);
+
+int __atomic_compare_exchange_c(int size, void *ptr, void *expected,
+                                void *desired, int success, int failure)
+                                asm("__atomic_compare_exchange");
+
+bool __atomic_compare_exchange_1(uint8_t *ptr, uint8_t *expected,
+                                 uint8_t desired, int success, int failure);
+bool __atomic_compare_exchange_2(uint16_t *ptr, uint16_t *expected,
+                                 uint16_t desired, int success, int failure);
+bool __atomic_compare_exchange_4(uint32_t *ptr, uint32_t *expected,
+                                 uint32_t desired, int success, int failure);
+bool __atomic_compare_exchange_8(uint64_t *ptr, uint64_t *expected,
+                                 uint64_t desired, int success, int failure);
+
+uint8_t __atomic_fetch_add_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_add_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_add_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_add_8(uint64_t *ptr, uint64_t val, int model);
+
+uint8_t __atomic_fetch_sub_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_sub_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_sub_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_sub_8(uint64_t *ptr, uint64_t val, int model);
+
+uint8_t __atomic_fetch_and_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_and_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_and_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_and_8(uint64_t *ptr, uint64_t val, int model);
+
+uint8_t __atomic_fetch_or_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_or_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_or_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_or_8(uint64_t *ptr, uint64_t val, int model);
+
+uint8_t __atomic_fetch_xor_1(uint8_t *ptr, uint8_t val, int model);
+uint16_t __atomic_fetch_xor_2(uint16_t *ptr, uint16_t val, int model);
+uint32_t __atomic_fetch_xor_4(uint32_t *ptr, uint32_t val, int model);
+uint64_t __atomic_fetch_xor_8(uint64_t *ptr, uint64_t val, int model);
+
+// We conditionally test the *_16 atomic function variants based on the same
+// condition that compiler_rt (atomic.c) uses to conditionally generate them.
+// Currently atomic.c tests if __SIZEOF_INT128__ is defined (which can be the
+// case on 32-bit platforms, by using -fforce-enable-int128), instead of using
+// CRT_HAS_128BIT.
+
+#ifdef __SIZEOF_INT128__
+#define TEST_16
+#endif
+
+#ifdef TEST_16
+typedef __uint128_t uint128_t;
+typedef uint128_t maxuint_t;
+uint128_t __atomic_load_16(uint128_t *src, int model);
+void __atomic_store_16(uint128_t *dest, uint128_t val, int model);
+uint128_t __atomic_exchange_16(uint128_t *dest, uint128_t val, int model);
+bool __atomic_compare_exchange_16(uint128_t *ptr, uint128_t *expected,
+                                  uint128_t desired, int success, int failure);
+uint128_t __atomic_fetch_add_16(uint128_t *ptr, uint128_t val, int model);
+uint128_t __atomic_fetch_sub_16(uint128_t *ptr, uint128_t val, int model);
+uint128_t __atomic_fetch_and_16(uint128_t *ptr, uint128_t val, int model);
+uint128_t __atomic_fetch_or_16(uint128_t *ptr, uint128_t val, int model);
+uint128_t __atomic_fetch_xor_16(uint128_t *ptr, uint128_t val, int model);
+#else
+typedef uint64_t maxuint_t;
+#endif
+
+#define U8(value) ((uint8_t)(value))
+#define U16(value) ((uint16_t)(value))
+#define U32(value) ((uint32_t)(value))
+#define U64(value) ((uint64_t)(value))
+
+#ifdef TEST_16
+#define V ((((uint128_t)0x4243444546474849) << 64) | 0x4a4b4c4d4e4f5051)
+#define ONES ((((uint128_t)0x0101010101010101) << 64) | 0x0101010101010101)
+#else
+#define V 0x4243444546474849
+#define ONES 0x0101010101010101
+#endif
+
+#define LEN(array) (sizeof(array) / sizeof(array[0]))
+
+__attribute__((aligned(16))) static char data[] = {
+    0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
+    0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
+    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27,
+    0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,
+};
+
+uint8_t a8, b8;
+uint16_t a16, b16;
+uint32_t a32, b32;
+uint64_t a64, b64;
+#ifdef TEST_16
+uint128_t a128, b128;
+#endif
+
+void set_a_values(maxuint_t value) {
+  a8 = U8(value);
+  a16 = U16(value);
+  a32 = U32(value);
+  a64 = U64(value);
+#ifdef TEST_16
+  a128 = value;
+#endif
+}
+
+void set_b_values(maxuint_t value) {
+  b8 = U8(value);
+  b16 = U16(value);
+  b32 = U32(value);
+  b64 = U64(value);
+#ifdef TEST_16
+  b128 = value;
+#endif
+}
+
+void test_loads(void) {
+  static int atomic_load_models[] = {
+      __ATOMIC_RELAXED,
+      __ATOMIC_CONSUME,
+      __ATOMIC_ACQUIRE,
+      __ATOMIC_SEQ_CST,
+  };
+
+  for (int m = 0; m < LEN(atomic_load_models); m++) {
+    int model = atomic_load_models[m];
+
+    // Test with aligned data.
+    for (int n = 1; n <= LEN(data); n++) {
+      char dst[LEN(data)] = {0};
+      __atomic_load_c(n, data, dst, model);
+      if (memcmp(dst, data, n) != 0)
+        abort();
+    }
+
+    // Test with unaligned data.
+    for (int n = 1; n < LEN(data); n++) {
+      char dst[LEN(data) - 1] = {0};
+      __atomic_load_c(n, data + 1, dst, model);
+      if (memcmp(dst, data + 1, n) != 0)
+        abort();
+    }
+
+    set_a_values(V + m);
+    if (__atomic_load_1(&a8, model) != U8(V + m))
+      abort();
+    if (__atomic_load_2(&a16, model) != U16(V + m))
+      abort();
+    if (__atomic_load_4(&a32, model) != U32(V + m))
+      abort();
+    if (__atomic_load_8(&a64, model) != U64(V + m))
+      abort();
+#ifdef TEST_16
+    if (__atomic_load_16(&a128, model) != V + m)
+      abort();
+#endif
+  }
+}
+
+void test_stores(void) {
+  static int atomic_store_models[] = {
+      __ATOMIC_RELAXED,
+      __ATOMIC_RELEASE,
+      __ATOMIC_SEQ_CST,
+  };
+
+  for (int m = 0; m < LEN(atomic_store_models); m++) {
+    int model = atomic_store_models[m];
+
+    // Test with aligned data.
+    for (int n = 1; n <= LEN(data); n++) {
+      char dst[LEN(data)];
+      __atomic_store_c(n, dst, data, model);
+      if (memcmp(data, dst, n) != 0)
+        abort();
+    }
+
+    // Test with unaligned data.
+    for (int n = 1; n < LEN(data); n++) {
+      char dst[LEN(data) - 1];
+      __atomic_store_c(n, dst, data + 1, model);
+      if (memcmp(data + 1, dst, n) != 0)
+        abort();
+    }
+
+    __atomic_store_1(&a8, U8(V + m), model);
+    if (a8 != U8(V + m))
+      abort();
+    __atomic_store_2(&a16, U16(V + m), model);
+    if (a16 != U16(V + m))
+      abort();
+    __atomic_store_4(&a32, U32(V + m), model);
+    if (a32 != U32(V + m))
+      abort();
+    __atomic_store_8(&a64, U64(V + m), model);
+    if (a64 != U64(V + m))
+      abort();
+#ifdef TEST_16
+    __atomic_store_16(&a128, V + m, model);
+    if (a128 != V + m)
+      abort();
+#endif
+  }
+}
+
+void test_exchanges() {
+  static int atomic_exchange_models[] = {
+      __ATOMIC_RELAXED,
+      __ATOMIC_ACQUIRE,
+      __ATOMIC_RELEASE,
+      __ATOMIC_ACQ_REL,
+      __ATOMIC_SEQ_CST,
+  };
+
+  set_a_values(V);
+
+  for (int m = 0; m < LEN(atomic_exchange_models); m++) {
+    int model = atomic_exchange_models[m];
+
+    // Test with aligned data.
+    for (int n = 1; n <= LEN(data); n++) {
+      char dst[LEN(data)];
+      char old[LEN(data)];
+      for (int i = 0; i < LEN(dst); i++)
+        dst[i] = i + m;
+      __atomic_exchange_c(n, dst, data, old, model);
+      for (int i = 0; i < n; i++) {
+        if (dst[i] != 0x10 + i || old[i] != i + m)
+          abort();
+      }
+    }
+
+    // Test with unaligned data.
+    for (int n = 1; n < LEN(data); n++) {
+      char dst[LEN(data) - 1];
+      char old[LEN(data) - 1];
+      for (int i = 0; i < LEN(dst); i++)
+        dst[i] = i + m;
+      __atomic_exchange_c(n, dst, data + 1, old, model);
+      for (int i = 0; i < n; i++) {
+        if (dst[i] != 0x10 + i + 1 || old[i] != i + m)
+          abort();
+      }
+    }
+
+    if (__atomic_exchange_1(&a8, U8(V + m + 1), model) != U8(V + m))
+      abort();
+    if (__atomic_exchange_2(&a16, U16(V + m + 1), model) != U16(V + m))
+      abort();
+    if (__atomic_exchange_4(&a32, U32(V + m + 1), model) != U32(V + m))
+      abort();
+    if (__atomic_exchange_8(&a64, U64(V + m + 1), model) != U64(V + m))
+      abort();
+#ifdef TEST_16
+    if (__atomic_exchange_16(&a128, V + m + 1, model) != V + m)
+      abort();
+#endif
+  }
+}
+
+void test_compare_exchanges(void) {
+  static int atomic_compare_exchange_models[] = {
+      __ATOMIC_RELAXED,
+      __ATOMIC_CONSUME,
+      __ATOMIC_ACQUIRE,
+      __ATOMIC_SEQ_CST,
+      __ATOMIC_RELEASE,
+      __ATOMIC_ACQ_REL,
+  };
+
+  for (int m1 = 0; m1 < LEN(atomic_compare_exchange_models); m1++) {
+    // Skip the last two: __ATOMIC_RELEASE and __ATOMIC_ACQ_REL.
+    // See <http://wg21.link/p0418> for details.
+    for (int m2 = 0; m2 < LEN(atomic_compare_exchange_models) - 2; m2++) {
+      int m_succ = atomic_compare_exchange_models[m1];
+      int m_fail = atomic_compare_exchange_models[m2];
+
+      // Test with aligned data.
+      for (int n = 1; n <= LEN(data); n++) {
+        char dst[LEN(data)] = {0};
+        char exp[LEN(data)] = {0};
+        if (!__atomic_compare_exchange_c(n, dst, exp, data, m_succ, m_fail))
+          abort();
+        if (memcmp(dst, data, n) != 0)
+          abort();
+        if (__atomic_compare_exchange_c(n, dst, exp, data, m_succ, m_fail))
+          abort();
+        if (memcmp(exp, data, n) != 0)
+          abort();
+      }
+
+      // Test with unaligned data.
+      for (int n = 1; n < LEN(data); n++) {
+        char dst[LEN(data) - 1] = {0};
+        char exp[LEN(data) - 1] = {0};
+        if (!__atomic_compare_exchange_c(n, dst, exp, data + 1, m_succ, m_fail))
+          abort();
+        if (memcmp(dst, data + 1, n) != 0)
+          abort();
+        if (__atomic_compare_exchange_c(n, dst, exp, data + 1, m_succ, m_fail))
+          abort();
+        if (memcmp(exp, data + 1, n) != 0)
+          abort();
+      }
+
+      set_a_values(ONES);
+      set_b_values(ONES * 2);
+
+      if (__atomic_compare_exchange_1(&a8, &b8, U8(V + m1), m_succ, m_fail))
+        abort();
+      if (a8 != U8(ONES) || b8 != U8(ONES))
+        abort();
+      if (!__atomic_compare_exchange_1(&a8, &b8, U8(V + m1), m_succ, m_fail))
+        abort();
+      if (a8 != U8(V + m1) || b8 != U8(ONES))
+        abort();
+
+      if (__atomic_compare_exchange_2(&a16, &b16, U16(V + m1), m_succ, m_fail))
+        abort();
+      if (a16 != U16(ONES) || b16 != U16(ONES))
+        abort();
+      if (!__atomic_compare_exchange_2(&a16, &b16, U16(V + m1), m_succ, m_fail))
+        abort();
+      if (a16 != U16(V + m1) || b16 != U16(ONES))
+        abort();
+
+      if (__atomic_compare_exchange_4(&a32, &b32, U32(V + m1), m_succ, m_fail))
+        abort();
+      if (a32 != U32(ONES) || b32 != U32(ONES))
+        abort();
+      if (!__atomic_compare_exchange_4(&a32, &b32, U32(V + m1), m_succ, m_fail))
+        abort();
+      if (a32 != U32(V + m1) || b32 != U32(ONES))
+        abort();
+
+      if (__atomic_compare_exchange_8(&a64, &b64, U64(V + m1), m_succ, m_fail))
+        abort();
+      if (a64 != U64(ONES) || b64 != U64(ONES))
+        abort();
+      if (!__atomic_compare_exchange_8(&a64, &b64, U64(V + m1), m_succ, m_fail))
+        abort();
+      if (a64 != U64(V + m1) || b64 != U64(ONES))
+        abort();
+
+#ifdef TEST_16
+      if (__atomic_compare_exchange_16(&a128, &b128, V + m1, m_succ, m_fail))
+        abort();
+      if (a128 != ONES || b128 != ONES)
+        abort();
+      if (!__atomic_compare_exchange_16(&a128, &b128, V + m1, m_succ, m_fail))
+        abort();
+      if (a128 != V + m1 || b128 != ONES)
+        abort();
+#endif
+    }
+  }
+}
+
+void test_fetch_op(void) {
+  static int atomic_fetch_models[] = {
+      __ATOMIC_RELAXED,
+      __ATOMIC_CONSUME,
+      __ATOMIC_ACQUIRE,
+      __ATOMIC_RELEASE,
+      __ATOMIC_ACQ_REL,
+      __ATOMIC_SEQ_CST,
+  };
+
+  for (int m = 0; m < LEN(atomic_fetch_models); m++) {
+    int model = atomic_fetch_models[m];
+
+    // Fetch add.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_add_1(&a8, U8(ONES), model);
+    if (b8 != U8(V + m) || a8 != U8(V + m + ONES))
+      abort();
+    b16 = __atomic_fetch_add_2(&a16, U16(ONES), model);
+    if (b16 != U16(V + m) || a16 != U16(V + m + ONES))
+      abort();
+    b32 = __atomic_fetch_add_4(&a32, U32(ONES), model);
+    if (b32 != U32(V + m) || a32 != U32(V + m + ONES))
+      abort();
+    b64 = __atomic_fetch_add_8(&a64, U64(ONES), model);
+    if (b64 != U64(V + m) || a64 != U64(V + m + ONES))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_add_16(&a128, ONES, model);
+    if (b128 != V + m || a128 != V + m + ONES)
+      abort();
+#endif
+
+    // Fetch sub.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_sub_1(&a8, U8(ONES), model);
+    if (b8 != U8(V + m) || a8 != U8(V + m - ONES))
+      abort();
+    b16 = __atomic_fetch_sub_2(&a16, U16(ONES), model);
+    if (b16 != U16(V + m) || a16 != U16(V + m - ONES))
+      abort();
+    b32 = __atomic_fetch_sub_4(&a32, U32(ONES), model);
+    if (b32 != U32(V + m) || a32 != U32(V + m - ONES))
+      abort();
+    b64 = __atomic_fetch_sub_8(&a64, U64(ONES), model);
+    if (b64 != U64(V + m) || a64 != U64(V + m - ONES))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_sub_16(&a128, ONES, model);
+    if (b128 != V + m || a128 != V + m - ONES)
+      abort();
+#endif
+
+    // Fetch and.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_and_1(&a8, U8(V + m), model);
+    if (b8 != U8(V + m) || a8 != U8(V + m))
+      abort();
+    b16 = __atomic_fetch_and_2(&a16, U16(V + m), model);
+    if (b16 != U16(V + m) || a16 != U16(V + m))
+      abort();
+    b32 = __atomic_fetch_and_4(&a32, U32(V + m), model);
+    if (b32 != U32(V + m) || a32 != U32(V + m))
+      abort();
+    b64 = __atomic_fetch_and_8(&a64, U64(V + m), model);
+    if (b64 != U64(V + m) || a64 != U64(V + m))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_and_16(&a128, V + m, model);
+    if (b128 != V + m || a128 != V + m)
+      abort();
+#endif
+
+    // Fetch or.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_or_1(&a8, U8(ONES), model);
+    if (b8 != U8(V + m) || a8 != U8((V + m) | ONES))
+      abort();
+    b16 = __atomic_fetch_or_2(&a16, U16(ONES), model);
+    if (b16 != U16(V + m) || a16 != U16((V + m) | ONES))
+      abort();
+    b32 = __atomic_fetch_or_4(&a32, U32(ONES), model);
+    if (b32 != U32(V + m) || a32 != U32((V + m) | ONES))
+      abort();
+    b64 = __atomic_fetch_or_8(&a64, U64(ONES), model);
+    if (b64 != U64(V + m) || a64 != U64((V + m) | ONES))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_or_16(&a128, ONES, model);
+    if (b128 != V + m || a128 != ((V + m) | ONES))
+      abort();
+#endif
+
+    // Fetch xor.
+
+    set_a_values(V + m);
+    set_b_values(0);
+    b8 = __atomic_fetch_xor_1(&a8, U8(ONES), model);
+    if (b8 != U8(V + m) || a8 != U8((V + m) ^ ONES))
+      abort();
+    b16 = __atomic_fetch_xor_2(&a16, U16(ONES), model);
+    if (b16 != U16(V + m) || a16 != U16((V + m) ^ ONES))
+      abort();
+    b32 = __atomic_fetch_xor_4(&a32, U32(ONES), model);
+    if (b32 != U32(V + m) || a32 != U32((V + m) ^ ONES))
+      abort();
+    b64 = __atomic_fetch_xor_8(&a64, U64(ONES), model);
+    if (b64 != U64(V + m) || a64 != U64((V + m) ^ ONES))
+      abort();
+#ifdef TEST_16
+    b128 = __atomic_fetch_xor_16(&a128, ONES, model);
+    if (b128 != (V + m) || a128 != ((V + m) ^ ONES))
+      abort();
+#endif
+
+    // Check signed integer overflow behavior
+
+    set_a_values(V + m);
+    __atomic_fetch_add_1(&a8, U8(V), model);
+    if (a8 != U8(V * 2 + m))
+      abort();
+    __atomic_fetch_sub_1(&a8, U8(V), model);
+    if (a8 != U8(V + m))
+      abort();
+    __atomic_fetch_add_2(&a16, U16(V), model);
+    if (a16 != U16(V * 2 + m))
+      abort();
+    __atomic_fetch_sub_2(&a16, U16(V), model);
+    if (a16 != U16(V + m))
+      abort();
+    __atomic_fetch_add_4(&a32, U32(V), model);
+    if (a32 != U32(V * 2 + m))
+      abort();
+    __atomic_fetch_sub_4(&a32, U32(V), model);
+    if (a32 != U32(V + m))
+      abort();
+    __atomic_fetch_add_8(&a64, U64(V), model);
+    if (a64 != U64(V * 2 + m))
+      abort();
+    __atomic_fetch_sub_8(&a64, U64(V), model);
+    if (a64 != U64(V + m))
+      abort();
+#ifdef TEST_16
+    __atomic_fetch_add_16(&a128, V, model);
+    if (a128 != V * 2 + m)
+      abort();
+    __atomic_fetch_sub_16(&a128, V, model);
+    if (a128 != V + m)
+      abort();
+#endif
+  }
+}
+
+int main() {
+  test_loads();
+  test_stores();
+  test_exchanges();
+  test_compare_exchanges();
+  test_fetch_op();
+}