Implement C++20 std::atomic_ref and test

[BenjaminTrapani]

Summary

Implement the atomic ref class by reusing the atomic_base_impl class and friends. Add overloads as required for reference types. The entire spec is tested and working under clang and g++. I could not figure out how to test the implementation falling back to locks (only used in freestanding builds). The additional code there needs to be tested. Should I investigate adding a regression test for freestanding atomic builds?

[__simt__]

That's a creative use of __cxx_atomic_impl that I hadn't foreseen. Seems to work. Ok.

It would be good if we could de-duplicate code here, because we're on a trajectory of having 4+ versions of every function that differ only on the type they take. Sounds like a perfect opportunity to sprinkle some enable_if here with a helper trait to determine if the thing is a cxx_atomic_impl, volatile cxx_atomic_impl, cxx_atomic_impl of reference, or volatile cxx_atomic_impl of reference.

As for the freestanding mode (or to be more accurate, the builtin-only mode) the only option you have for atomic_ref is to static_assert that it is always lock-free. There is presently 1 platform that builds with this mode (CUDA 10.2+) and this is what we planned to do anyway.

[BenjaminTrapani]

Get rid of duplication across __cxx_atomic_* functions by using a traits class and SFINAE to check types and select overrides when required. Static assert that template type is always lock free when built using builtin only mode.

[BenjaminTrapani]

Remove redundant namespace qualifiers

[BenjaminTrapani]

Provide only two versions of __cxx_atomic_pointer_to_data that perform different cast operations. Get rid of type-specific versions by extracting original type qualifiers and re-annoting atomic type.

[BenjaminTrapani]

Any update on this?

[BenjaminTrapani]

For those that are interested in testing out this change, here is a release build of clang 10 with these changes: https://skytopsoftware.ddns.net/binaries/compilers/llvm/llvm_10_release_linux_amd64.tar.gz

Changes are built as of d32170dbd5b0d54436537b6b75beaf44324e0c28 with the following:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/benjamintrapani/libs/llvm-10-release -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;compiler-rt;debuginfo-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;mlir;openmp;parallel-libs;polly;pstl'

[ldionne]

Thanks for working on that! I left some general comments, however I believe the best person to give you an in-depth review about this is @__simt__, as a co-author of the paper and as being the de-facto owner of <atomic>.

Misc note: Please update libcxx/www/cxx2a_status.html.

libcxx/include/atomic

112

// since C++20 (applies below too)

130

It is customary to put these members closer to the top.

847

Reserved identifiers are of the form _Foo, not Foo_. You'll need to make that (simple) change throughout.

1464

What is preventing you from adding this check on Clang today?

1810

You should guard this with #if _LIBCPP_STD_VER > 17 (applies below too).

libcxx/test/std/atomics/atomics.types.generic/address.pass.cpp

138

I believe it would be better to add new test files just for atomic_ref.

libcxx/test/std/atomics/atomics.types.generic/trivially_copyable_ref.fail.cpp

55

How do you expect it to fail? I would strongly suggest using a .verify.cpp test instead and pinning down the error you're expecting to see.

[__simt__]

We're looking at this patch in the context of the libcudacxx fork. I don't have more feedback right now but we probably will get there soon.

libcxx/include/atomic

1810

We will want to allow using this from C++11 onwards. If libcxx doesn't do that out of the box, then we'll be doing that in our private patchset.

libcxx/test/std/atomics/atomics.types.generic/address.pass.cpp

138

Actually, I would prefer if we found a way to make the atomics tests cover atomic_ref, at breadth.

There is absolutely no way that this patch has sufficient test coverage right now, and duplicating all the tests is an inefficient use of our time (now and in the future).

For the most part, this could be tested by doing sufficiently clever refactoring in atomic_helpers.

[wmaxey]

[ldionne]

libcxx/include/atomic

1810

Ok. I guess we've been down that road already anyway.

libcxx/test/std/atomics/atomics.types.generic/address.pass.cpp

138

As you prefer. I imagine it might look like some generic test that works with atomic *and* atomic_ref. If so, it's easy to have the generic test in a header file and then instantiate it once from a atomic.pass.cpp file and once from a atomic_ref.pass.cpp. This would keep the test structure consistent to what it is today.

But I am not strongly attached to a particular approach -- whatever's more convenient works, I was just trying to retain some consistency with the way we've been doing things.

[rsmith]

libcxx/include/atomic

990

I find this cast concerning. There's no guarantee that _Atomic(T) is compatible with plain T in general (they might have different representations), and even if the layout is identical, I don't think Clang's strict aliasing rules provide a guarantee that you can access a T via an _Atomic(T). (We happen to treat atomic types as can-alias-anything right now, but that's only because we've not implemented strict aliasing for atomics yet.)

Can we use the implementation in terms of __atomic_* for atomic_ref instead? Those builtins are designed to be used on objects that are declared with type T rather than type _Atomic(T).

[__simt__]

libcxx/include/atomic

990

I agree with you there. It looks like atomic_ref<> is not implementable (at least not at high QoI) on top of C11 the way that atomic<> was.

I do wish the _Atomic(T) path simply wasn't there in libcxx, and we just always used the __atomic_* path but the Mac build chooses the _Atomic(T) path. At least it does today.

Not sure how to reconcile that.

[BenjaminTrapani]

libcxx/include/atomic

990

Sorry for the delay in responding to all these comments. I will fix per these comments today. For the purposes of this change, is it ok to leave atomic_ref unimplemented for the C11 path, and look into removing the C11 path later?

[BenjaminTrapani]

Remove C11 atomic path from atomic header. Update tests to check for specific errors. Other style fixes.

[BenjaminTrapani]

Pull in original portion of patch

[BenjaminTrapani]

libcxx/include/atomic

1464

Should be fine to enable, although probably best done in another patch so we can revert if it breaks many things for a large clang user.

libcxx/test/std/atomics/atomics.types.generic/address.pass.cpp

138

I thought about this a bit and the interfaces under test are substantially different. We could share the common parts by always constructing from lvalues.

[pkasting]

It looks like this went silent. I'm interested in atomic_ref as a downstream user; @BenjaminTrapani are you still interested in pushing this forward? If not I might be able to find someone to look into it...

libcxx/include/__config

 // Thread-unsafe functions such as strtok() and localtime()
 // are not available.
 #ifdef __CloudABI__
 #define _LIBCPP_HAS_NO_THREAD_UNSAFE_C_FUNCTIONS
 #endif
 
 #if __has_feature(cxx_atomic) || __has_extension(c_atomic) || __has_keyword(_Atomic)
 #  define _LIBCPP_HAS_C_ATOMIC_IMP
-#elif defined(_LIBCPP_COMPILER_GCC)
+#endif
+
+#if defined(_LIBCPP_COMPILER_GCC) || (__has_builtin(__atomic_load) && __has_builtin(__atomic_store) && __has_builtin(__atomic_exchange) && __has_builtin(__atomic_compare_exchange))
 #  define _LIBCPP_HAS_GCC_ATOMIC_IMP
 #endif
 
 #if (!defined(_LIBCPP_HAS_C_ATOMIC_IMP) && \
      !defined(_LIBCPP_HAS_GCC_ATOMIC_IMP) && \
      !defined(_LIBCPP_HAS_EXTERNAL_ATOMIC_IMP)) \
      || defined(_LIBCPP_HAS_NO_THREADS)
 #  define _LIBCPP_HAS_NO_ATOMIC_HEADER

libcxx/include/atomic

struct atomic
     void wait(T, memory_order = memory_order::seq_cst) const volatile noexcept;
     void wait(T, memory_order = memory_order::seq_cst) const noexcept;
     void notify_one() volatile noexcept;
     void notify_one() noexcept;
     void notify_all() volatile noexcept;
     void notify_all() noexcept;
 };
 
+template <class T>
+struct atomic_ref // since C++20
+{
+    static constexpr bool is_always_lock_free;
+    static constexpr size_t required_alignment;
+
+    explicit atomic_ref(T& obj) noexcept;
+    atomic_ref(const atomic_ref&) noexcept;
+    T operator=(T desired) const noexcept;
+    atomic_ref& operator=(const atomic_ref&) = delete;
+
+    bool is_lock_free() const noexcept;
+    void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+    T load(memory_order m = memory_order_seq_cst) const noexcept;
+    operator T() const noexcept;
+    T exchange(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_weak(T& expc, T desr, memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_strong(T& expc, T desr,
+                                 memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_weak(T& expc, T desr,
+                               memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_strong(T& expc, T desr,
+                                 memory_order m = memory_order_seq_cst) const noexcept;
+};
+
 template <>
 struct atomic<integral>
 {
     using value_type = integral;
     using difference_type = value_type;
 
     static constexpr bool is_always_lock_free;
+
     bool is_lock_free() const volatile noexcept;
     bool is_lock_free() const noexcept;
 
-    atomic() noexcept = default;
-    constexpr atomic(integral desr) noexcept;
-    atomic(const atomic&) = delete;
-    atomic& operator=(const atomic&) = delete;
-    atomic& operator=(const atomic&) volatile = delete;
-
     integral load(memory_order m = memory_order_seq_cst) const volatile noexcept;
     integral load(memory_order m = memory_order_seq_cst) const noexcept;
     operator integral() const volatile noexcept;
     operator integral() const noexcept;
     void store(integral desr, memory_order m = memory_order_seq_cst) volatile noexcept;
     void store(integral desr, memory_order m = memory_order_seq_cst) noexcept;
     integral operator=(integral desr) volatile noexcept;
     integral operator=(integral desr) noexcept;
struct atomic<integral>
     integral fetch_sub(integral op, memory_order m = memory_order_seq_cst) noexcept;
     integral fetch_and(integral op, memory_order m = memory_order_seq_cst) volatile noexcept;
     integral fetch_and(integral op, memory_order m = memory_order_seq_cst) noexcept;
     integral fetch_or(integral op, memory_order m = memory_order_seq_cst) volatile noexcept;
     integral fetch_or(integral op, memory_order m = memory_order_seq_cst) noexcept;
     integral fetch_xor(integral op, memory_order m = memory_order_seq_cst) volatile noexcept;
     integral fetch_xor(integral op, memory_order m = memory_order_seq_cst) noexcept;
 
+    atomic() noexcept = default;
+    constexpr atomic(integral desr) noexcept;
+    atomic(const atomic&) = delete;
+    atomic& operator=(const atomic&) = delete;
+    atomic& operator=(const atomic&) volatile = delete;
+    integral operator=(integral desr) volatile noexcept;
+    integral operator=(integral desr) noexcept;
+
     integral operator++(int) volatile noexcept;
     integral operator++(int) noexcept;
     integral operator--(int) volatile noexcept;
     integral operator--(int) noexcept;
     integral operator++() volatile noexcept;
     integral operator++() noexcept;
     integral operator--() volatile noexcept;
     integral operator--() noexcept;
struct atomic<integral>
     void wait(integral, memory_order = memory_order::seq_cst) const volatile noexcept;
     void wait(integral, memory_order = memory_order::seq_cst) const noexcept;
     void notify_one() volatile noexcept;
     void notify_one() noexcept;
     void notify_all() volatile noexcept;
     void notify_all() noexcept;
 };
 
+template <>
+struct atomic_ref<integral> // since C++20
+{
+    static constexpr bool is_always_lock_free;
+    static constexpr size_t required_alignment;
+
+    explicit atomic_ref(integral& obj) noexcept;
+    atomic_ref(const atomic_ref&) noexcept;
+    integral operator=(integral desired) const noexcept;
+    atomic_ref& operator=(const atomic_ref&) = delete;
+
+    bool is_lock_free() const noexcept;
+    void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+    integral load(memory_order m = memory_order_seq_cst) const noexcept;
+    operator integral() const noexcept;
+    integral exchange(integral desr, memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_weak(integral& expc, integral desr, memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_strong(integral& expc, integral desr,
+                                 memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_weak(integral& expc, integral desr,
+                               memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_strong(integral& expc, integral desr,
+                                 memory_order m = memory_order_seq_cst) const noexcept;
+
+    integral fetch_add(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_sub(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_and(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_or(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_xor(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+
+    integral operator++(int) const noexcept;
+    integral operator--(int) const noexcept;
+    integral operator++() const noexcept;
+    integral operator--() const noexcept;
+    integral operator+=(integral op) const noexcept;
+    integral operator-=(integral op) const noexcept;
+    integral operator&=(integral op) const noexcept;
+    integral operator|=(integral op) const noexcept;
+    integral operator^=(integral op) const noexcept;
+};
+
 template <class T>
 struct atomic<T*>
 {
     using value_type = T*;
     using difference_type = ptrdiff_t;
 
     static constexpr bool is_always_lock_free;
     bool is_lock_free() const volatile noexcept;
struct atomic<T*>
     void wait(T*, memory_order = memory_order::seq_cst) const volatile noexcept;
     void wait(T*, memory_order = memory_order::seq_cst) const noexcept;
     void notify_one() volatile noexcept;
     void notify_one() noexcept;
     void notify_all() volatile noexcept;
     void notify_all() noexcept;
 };
 
+template <class T>
+struct atomic_ref<T*> // since C++20
+{
+    static constexpr bool is_always_lock_free;
+    static constexpr size_t required_alignment;
+
+    explicit atomic_ref(T* obj) noexcept;
+    atomic_ref(const atomic_ref&) noexcept;
+    T* operator=(T* desired) const noexcept;
+    atomic_ref& operator=(const atomic_ref&) = delete;
+
+    bool is_lock_free() const noexcept;
+    void store(T* desr, memory_order m = memory_order_seq_cst) const noexcept;
+    T* load(memory_order m = memory_order_seq_cst) const noexcept;
+    operator T*() const noexcept;
+    T* exchange(T* desr, memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_weak(T*& expc, T* desr, memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_strong(T*& expc, T* desr,
+                                 memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_weak(T*& expc, T* desr,
+                               memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_strong(T*& expc, T* desr,
+                                 memory_order m = memory_order_seq_cst) const noexcept;
+
+    T* fetch_add(ptrdiff_t op, memory_order m = memory_order_seq_cst) noexcept;
+    T* fetch_sub(ptrdiff_t op, memory_order m = memory_order_seq_cst) noexcept;
+
+    T* operator++(int) const noexcept;
+    T* operator--(int) const noexcept;
+    T* operator++() const noexcept;
+    T* operator--() const noexcept;
+    T* operator+=(ptrdiff_t op) const noexcept;
+    T* operator-=(ptrdiff_t op) const noexcept;
+};
 
 template <class T>
   bool atomic_is_lock_free(const volatile atomic<T>* obj) noexcept;
 
 template <class T>
   bool atomic_is_lock_free(const atomic<T>* obj) noexcept;
 
 template <class T>
struct __cxx_atomic_base_impl {
   _LIBCPP_INLINE_VISIBILITY
 #ifndef _LIBCPP_CXX03_LANG
     __cxx_atomic_base_impl() _NOEXCEPT = default;
 #else
     __cxx_atomic_base_impl() _NOEXCEPT : __a_value() {}
 #endif // _LIBCPP_CXX03_LANG
   _LIBCPP_CONSTEXPR explicit __cxx_atomic_base_impl(_Tp value) _NOEXCEPT
     : __a_value(value) {}
+  using __contained_t = _Tp;
   _Tp __a_value;
 };
 
+template<typename _Tp, template<typename> class _TemplateTp>
+struct __is_instantiation_of : false_type { };
+
+template<typename _Tp, template<typename> class _TemplateTp>
+struct __is_instantiation_of<_TemplateTp<_Tp>, _TemplateTp> : true_type {};
+
+template <typename _Tp, typename = typename enable_if<__is_instantiation_of<typename remove_volatile<typename _Tp::__base>::type, __cxx_atomic_base_impl>::value, bool>::type>
+struct __cxx_atomic_base_impl_traits {
+  static constexpr bool __is_value_volatile = is_volatile<_Tp>::value;
+  static constexpr bool __is_value_ref = is_reference<typename _Tp::__contained_t>::value;
+  using __underlying_t = typename remove_volatile<typename remove_reference<typename _Tp::__contained_t>::type>::type;
+  static constexpr bool __is_value_pointer = is_pointer<__underlying_t>::value;
+};
+
 _LIBCPP_INLINE_VISIBILITY inline _LIBCPP_CONSTEXPR int __to_gcc_order(memory_order __order) {
   // Avoid switch statement to make this a constexpr.
   return __order == memory_order_relaxed ? __ATOMIC_RELAXED:
          (__order == memory_order_acquire ? __ATOMIC_ACQUIRE:
           (__order == memory_order_release ? __ATOMIC_RELEASE:
            (__order == memory_order_seq_cst ? __ATOMIC_SEQ_CST:
             (__order == memory_order_acq_rel ? __ATOMIC_ACQ_REL:
               __ATOMIC_CONSUME))));
 }
 
 _LIBCPP_INLINE_VISIBILITY inline _LIBCPP_CONSTEXPR int __to_gcc_failure_order(memory_order __order) {
   // Avoid switch statement to make this a constexpr.
   return __order == memory_order_relaxed ? __ATOMIC_RELAXED:
          (__order == memory_order_acquire ? __ATOMIC_ACQUIRE:
           (__order == memory_order_release ? __ATOMIC_RELAXED:
            (__order == memory_order_seq_cst ? __ATOMIC_SEQ_CST:
             (__order == memory_order_acq_rel ? __ATOMIC_ACQUIRE:
               __ATOMIC_CONSUME))));
 }
 
-template <typename _Tp>
+template <typename _Tp, typename enable_if<__cxx_atomic_base_impl_traits<_Tp>::__is_value_volatile, bool>::type = 0>
 _LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_init(volatile __cxx_atomic_base_impl<_Tp>* __a,  _Tp __val) {
+void __cxx_atomic_init(_Tp* __a,  typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __val) {
   __cxx_atomic_assign_volatile(__a->__a_value, __val);
 }
 
-template <typename _Tp>
+template <typename _Tp, typename enable_if<!__cxx_atomic_base_impl_traits<_Tp>::__is_value_volatile, bool>::type = 0>
 _LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp>* __a,  _Tp __val) {
+void __cxx_atomic_init(_Tp* __a,  typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __val) {
   __a->__a_value = __val;
 }
 
 _LIBCPP_INLINE_VISIBILITY inline
 void __cxx_atomic_thread_fence(memory_order __order) {
   __atomic_thread_fence(__to_gcc_order(__order));
 }
 
 _LIBCPP_INLINE_VISIBILITY inline
 void __cxx_atomic_signal_fence(memory_order __order) {
   __atomic_signal_fence(__to_gcc_order(__order));
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_store(volatile __cxx_atomic_base_impl<_Tp>* __a,  _Tp __val,
-                        memory_order __order) {
-  __atomic_store(&__a->__a_value, &__val,
-                 __to_gcc_order(__order));
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp>* __a,  _Tp __val,
-                        memory_order __order) {
+void __cxx_atomic_store(_Tp* __a,  typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __val, memory_order __order) {
   __atomic_store(&__a->__a_value, &__val,
                  __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_load(const volatile __cxx_atomic_base_impl<_Tp>* __a,
+_LIBCPP_INLINE_VISIBILITY 
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_load(const _Tp* __a,
                       memory_order __order) {
-  _Tp __ret;
-  __atomic_load(&__a->__a_value, &__ret,
-                __to_gcc_order(__order));
-  return __ret;
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_load(const __cxx_atomic_base_impl<_Tp>* __a, memory_order __order) {
-  _Tp __ret;
+  typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __ret;
   __atomic_load(&__a->__a_value, &__ret,
                 __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_exchange(volatile __cxx_atomic_base_impl<_Tp>* __a,
-                          _Tp __value, memory_order __order) {
-  _Tp __ret;
-  __atomic_exchange(&__a->__a_value, &__value, &__ret,
-                    __to_gcc_order(__order));
-  return __ret;
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp>* __a, _Tp __value,
-                          memory_order __order) {
-  _Tp __ret;
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_exchange(_Tp* __a,
+                      typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __value,
+          memory_order __order) {
+  typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __ret;
   __atomic_exchange(&__a->__a_value, &__value, &__ret,
                     __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
 bool __cxx_atomic_compare_exchange_strong(
-    volatile __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value,
+    _Tp* __a, typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t* __expected,
+    typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __value,
     memory_order __success, memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    false,
                                    __to_gcc_order(__success),
                                    __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_strong(
-    __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value, memory_order __success,
-    memory_order __failure) {
-  return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
-                                   false,
-                                   __to_gcc_order(__success),
-                                   __to_gcc_failure_order(__failure));
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
 bool __cxx_atomic_compare_exchange_weak(
-    volatile __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value,
+    _Tp* __a, typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t* __expected,
+    typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __value,
     memory_order __success, memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    true,
                                    __to_gcc_order(__success),
                                    __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_weak(
-    __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value, memory_order __success,
-    memory_order __failure) {
-  return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
-                                   true,
-                                   __to_gcc_order(__success),
-                                   __to_gcc_failure_order(__failure));
-}
-
-template <typename _Tp>
 struct __skip_amt { enum {value = 1}; };
 
 template <typename _Tp>
 struct __skip_amt<_Tp*> { enum {value = sizeof(_Tp)}; };
 
 // FIXME: Haven't figured out what the spec says about using arrays with
 // atomic_fetch_add. Force a failure rather than creating bad behavior.
 template <typename _Tp>
 struct __skip_amt<_Tp[]> { };
 template <typename _Tp, int n>
 struct __skip_amt<_Tp[n]> { };
 
 template <typename _Tp, typename _Td>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_add(volatile __cxx_atomic_base_impl<_Tp>* __a,
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_fetch_add(_Tp* __a,
                            _Td __delta, memory_order __order) {
-  return __atomic_fetch_add(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
+  return __atomic_fetch_add(&__a->__a_value, __delta * __skip_amt<typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t>::value,
                             __to_gcc_order(__order));
 }
 
 template <typename _Tp, typename _Td>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp>* __a, _Td __delta,
-                           memory_order __order) {
-  return __atomic_fetch_add(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __to_gcc_order(__order));
-}
-
-template <typename _Tp, typename _Td>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_sub(volatile __cxx_atomic_base_impl<_Tp>* __a,
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_fetch_sub(_Tp* __a,
                            _Td __delta, memory_order __order) {
-  return __atomic_fetch_sub(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __to_gcc_order(__order));
-}
-
-template <typename _Tp, typename _Td>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp>* __a, _Td __delta,
-                           memory_order __order) {
-  return __atomic_fetch_sub(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __to_gcc_order(__order));
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_and(volatile __cxx_atomic_base_impl<_Tp>* __a,
-                           _Tp __pattern, memory_order __order) {
-  return __atomic_fetch_and(&__a->__a_value, __pattern,
+  return __atomic_fetch_sub(&__a->__a_value, __delta * __skip_amt<typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t>::value,
                             __to_gcc_order(__order));
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp>* __a,
-                           _Tp __pattern, memory_order __order) {
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_fetch_and(_Tp* __a,
+                       typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __pattern,
+           memory_order __order) {
   return __atomic_fetch_and(&__a->__a_value, __pattern,
                             __to_gcc_order(__order));
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_or(volatile __cxx_atomic_base_impl<_Tp>* __a,
-                          _Tp __pattern, memory_order __order) {
-  return __atomic_fetch_or(&__a->__a_value, __pattern,
-                           __to_gcc_order(__order));
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp>* __a, _Tp __pattern,
-                          memory_order __order) {
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_fetch_or(_Tp* __a,
+                      typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __pattern,
+          memory_order __order) {
   return __atomic_fetch_or(&__a->__a_value, __pattern,
                            __to_gcc_order(__order));
 }
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_xor(volatile __cxx_atomic_base_impl<_Tp>* __a,
-                           _Tp __pattern, memory_order __order) {
-  return __atomic_fetch_xor(&__a->__a_value, __pattern,
-                            __to_gcc_order(__order));
-}
-
-template <typename _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp>* __a, _Tp __pattern,
-                           memory_order __order) {
+typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t
+__cxx_atomic_fetch_xor(_Tp* __a,
+                       typename __cxx_atomic_base_impl_traits<_Tp>::__underlying_t __pattern,
+           memory_order __order) {
   return __atomic_fetch_xor(&__a->__a_value, __pattern,
                             __to_gcc_order(__order));
 }
 
 #define __cxx_atomic_is_lock_free(__s) __atomic_is_lock_free(__s, 0)
 
-#elif defined(_LIBCPP_HAS_C_ATOMIC_IMP)
-
-template <typename _Tp>
-struct __cxx_atomic_base_impl {
-
-  _LIBCPP_INLINE_VISIBILITY
-#ifndef _LIBCPP_CXX03_LANG
-    __cxx_atomic_base_impl() _NOEXCEPT = default;
-#else
-    __cxx_atomic_base_impl() _NOEXCEPT : __a_value() {}
-#endif // _LIBCPP_CXX03_LANG
-  _LIBCPP_CONSTEXPR explicit __cxx_atomic_base_impl(_Tp value) _NOEXCEPT
-    : __a_value(value) {}
-  _LIBCPP_DISABLE_EXTENSION_WARNING _Atomic(_Tp) __a_value;
-};
-
-#define __cxx_atomic_is_lock_free(__s) __c11_atomic_is_lock_free(__s)
-
-_LIBCPP_INLINE_VISIBILITY inline
-void __cxx_atomic_thread_fence(memory_order __order) _NOEXCEPT {
-    __c11_atomic_thread_fence(static_cast<__memory_order_underlying_t>(__order));
-}
-
-_LIBCPP_INLINE_VISIBILITY inline
-void __cxx_atomic_signal_fence(memory_order __order) _NOEXCEPT {
-    __c11_atomic_signal_fence(static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __val) _NOEXCEPT {
-    __c11_atomic_init(&__a->__a_value, __val);
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp> * __a, _Tp __val) _NOEXCEPT {
-    __c11_atomic_init(&__a->__a_value, __val);
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __val, memory_order __order) _NOEXCEPT {
-    __c11_atomic_store(&__a->__a_value, __val, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp> * __a, _Tp __val, memory_order __order) _NOEXCEPT {
-    __c11_atomic_store(&__a->__a_value, __val, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_load(__cxx_atomic_base_impl<_Tp> const volatile* __a, memory_order __order) _NOEXCEPT {
-    using __ptr_type = typename remove_const<decltype(__a->__a_value)>::type*;
-    return __c11_atomic_load(const_cast<__ptr_type>(&__a->__a_value), static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_load(__cxx_atomic_base_impl<_Tp> const* __a, memory_order __order) _NOEXCEPT {
-    using __ptr_type = typename remove_const<decltype(__a->__a_value)>::type*;
-    return __c11_atomic_load(const_cast<__ptr_type>(&__a->__a_value), static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __value, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_exchange(&__a->__a_value, __value, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp> * __a, _Tp __value, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_exchange(&__a->__a_value, __value, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_strong(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp* __expected, _Tp __value, memory_order __success, memory_order __failure) _NOEXCEPT {
-    return __c11_atomic_compare_exchange_strong(&__a->__a_value, __expected, __value, static_cast<__memory_order_underlying_t>(__success), static_cast<__memory_order_underlying_t>(__failure));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_strong(__cxx_atomic_base_impl<_Tp> * __a, _Tp* __expected, _Tp __value, memory_order __success, memory_order __failure) _NOEXCEPT {
-    return __c11_atomic_compare_exchange_strong(&__a->__a_value, __expected, __value, static_cast<__memory_order_underlying_t>(__success), static_cast<__memory_order_underlying_t>(__failure));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_weak(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp* __expected, _Tp __value, memory_order __success, memory_order __failure) _NOEXCEPT {
-    return __c11_atomic_compare_exchange_weak(&__a->__a_value, __expected, __value, static_cast<__memory_order_underlying_t>(__success), static_cast<__memory_order_underlying_t>(__failure));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-bool __cxx_atomic_compare_exchange_weak(__cxx_atomic_base_impl<_Tp> * __a, _Tp* __expected, _Tp __value, memory_order __success, memory_order __failure) _NOEXCEPT {
-    return __c11_atomic_compare_exchange_weak(&__a->__a_value, __expected, __value,  static_cast<__memory_order_underlying_t>(__success), static_cast<__memory_order_underlying_t>(__failure));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_add(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp> * __a, _Tp __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_add(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp* __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_add(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp* __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> * __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_add(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp> * __a, _Tp __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp* __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp* __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp*> * __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_and(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_and(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_or(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_or(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_xor(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-template<class _Tp>
-_LIBCPP_INLINE_VISIBILITY
-_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, memory_order __order) _NOEXCEPT {
-    return __c11_atomic_fetch_xor(&__a->__a_value, __pattern, static_cast<__memory_order_underlying_t>(__order));
-}
-
-#endif // _LIBCPP_HAS_GCC_ATOMIC_IMP, _LIBCPP_HAS_C_ATOMIC_IMP
+#endif // _LIBCPP_HAS_GCC_ATOMIC_IMP
 
 template <class _Tp>
 _LIBCPP_INLINE_VISIBILITY
 _Tp kill_dependency(_Tp __y) _NOEXCEPT
 {
     return __y;
 }
 
 #if defined(__CLANG_ATOMIC_BOOL_LOCK_FREE)
 # define ATOMIC_BOOL_LOCK_FREE      __CLANG_ATOMIC_BOOL_LOCK_FREE
 # define ATOMIC_CHAR_LOCK_FREE      __CLANG_ATOMIC_CHAR_LOCK_FREE

 _Tp __cxx_atomic_fetch_sub(__cxx_atomic_lock_impl<_Tp>* __a,
                            _Td __delta, memory_order) {
   __a->__lock();
   _Tp __old = __a->__a_value;
   __a->__a_value -= __delta;
   __a->__unlock();
   return __old;
 }
+// TODO specialize for pointer fetch_sub
 
 template <typename _Tp>
 _LIBCPP_INLINE_VISIBILITY
 _Tp __cxx_atomic_fetch_and(volatile __cxx_atomic_lock_impl<_Tp>* __a,
                            _Tp __pattern, memory_order) {
   __a->__lock();
   _Tp __old;
   __cxx_atomic_assign_volatile(__old, __a->__a_value);
template <typename _Tp,
           typename _Base = typename conditional<__cxx_is_always_lock_free<_Tp>::__value,
                                                 __cxx_atomic_base_impl<_Tp>,
                                                 __cxx_atomic_lock_impl<_Tp> >::type>
 #else
 template <typename _Tp,
           typename _Base = __cxx_atomic_base_impl<_Tp> >
 #endif //_LIBCPP_ATOMIC_ONLY_USE_BUILTINS
 struct __cxx_atomic_impl : public _Base {
-
-#if _GNUC_VER >= 501
-    static_assert(is_trivially_copyable<_Tp>::value,
-      "std::atomic<Tp> requires that 'Tp' be a trivially copyable type");
-#endif
-
+   using __base = _Base;
   _LIBCPP_INLINE_VISIBILITY __cxx_atomic_impl() _NOEXCEPT _LIBCPP_DEFAULT
   _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR explicit __cxx_atomic_impl(_Tp value) _NOEXCEPT
     : _Base(value) {}
 };
 
 #ifdef __linux__
     using __cxx_contention_t = int32_t;
 #else

 struct __atomic_base  // false
 {
     mutable __cxx_atomic_impl<_Tp> __a_;
 
 #if defined(__cpp_lib_atomic_is_always_lock_free)
   static _LIBCPP_CONSTEXPR bool is_always_lock_free = __atomic_always_lock_free(sizeof(__a_), 0);
 #endif
 
+// TODO perform this check when compiled with clang too.
+#if _GNUC_VER >= 501
+    static_assert(is_trivially_copyable<_Tp>::value,
+      "std::atomic<Tp> requires that 'Tp' be a trivially copyable type");
+#endif
+
     _LIBCPP_INLINE_VISIBILITY
     bool is_lock_free() const volatile _NOEXCEPT
         {return __cxx_atomic_is_lock_free(sizeof(_Tp));}
     _LIBCPP_INLINE_VISIBILITY
     bool is_lock_free() const _NOEXCEPT
         {return static_cast<__atomic_base const volatile*>(this)->is_lock_free();}
     _LIBCPP_INLINE_VISIBILITY
     void store(_Tp __d, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
private:
     __atomic_base(const __atomic_base&);
     _LIBCPP_INLINE_VISIBILITY
     __atomic_base& operator=(const __atomic_base&);
     _LIBCPP_INLINE_VISIBILITY
     __atomic_base& operator=(const __atomic_base&) volatile;
 #endif
 };
 
+// general atomic_ref<T>
+
+template <class _Tp, bool = is_integral<_Tp>::value && !is_same<_Tp, bool>::value>
+struct __atomic_ref_base  // false
+{
+    mutable __cxx_atomic_impl<_Tp&> __a_;
+
+#if defined(__cpp_lib_atomic_is_always_lock_free)
+  static _LIBCPP_CONSTEXPR bool is_always_lock_free = __atomic_always_lock_free(sizeof(__a_.__a_value), 0);
+#endif
+
+#if defined(_LIBCPP_ATOMIC_ONLY_USE_BUILTINS)
+  static_assert(__cxx_is_always_lock_free<_Tp>::__value, "std::atomic_ref<_Tp> requires _Tp to always be lock free when built using builtin only mode");
+#endif
+
+    static_assert(is_trivially_copyable<_Tp>::value, "std::atomic_ref<_Tp> requires that '_Tp' be a trivially copyable type");
+    
+    static _LIBCPP_CONSTEXPR size_t required_alignment = alignment_of<decltype(__cxx_atomic_impl<_Tp>::__a_value)>::value;
+  
+    _LIBCPP_INLINE_VISIBILITY
+    bool is_lock_free() const _NOEXCEPT
+        {return __cxx_atomic_is_lock_free(sizeof(_Tp));}
+    _LIBCPP_INLINE_VISIBILITY
+    void store(_Tp __d, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+      _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m)
+        {__cxx_atomic_store(&__a_, __d, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp load(memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+      _LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
+        {return __cxx_atomic_load(&__a_, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    operator _Tp() const _NOEXCEPT          {return load();}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp exchange(_Tp __d, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_exchange(&__a_, __d, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    bool compare_exchange_weak(_Tp& __e, _Tp __d,
+                               memory_order __s, memory_order __f) const _NOEXCEPT
+      _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __s, __f);}
+    _LIBCPP_INLINE_VISIBILITY
+    bool compare_exchange_strong(_Tp& __e, _Tp __d,
+                                 memory_order __s, memory_order __f) const _NOEXCEPT
+      _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __s, __f);}
+    _LIBCPP_INLINE_VISIBILITY
+    bool compare_exchange_weak(_Tp& __e, _Tp __d,
+                               memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __m, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    bool compare_exchange_strong(_Tp& __e, _Tp __d,
+                                 memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __m, __m);}
+
+    _LIBCPP_INLINE_VISIBILITY _LIBCPP_EXPLICIT
+    __atomic_ref_base(_Tp& __d) _NOEXCEPT : __a_(__d) {}
+    _LIBCPP_INLINE_VISIBILITY
+    __atomic_ref_base(const __atomic_ref_base& other) _NOEXCEPT
+        : __a_(other.__a_) {}
+
+    _Tp operator=(const _Tp __d) const _NOEXCEPT {
+      store(__d);
+      return __d;
+    }
+#ifndef _LIBCPP_CXX03_LANG
+    __atomic_ref_base& operator=(const __atomic_ref_base&) = delete;
+#else
+private:
+    __atomic_ref_base& operator=(const __atomic_ref_base&);
+#endif
+};
+
 #if defined(__cpp_lib_atomic_is_always_lock_free)
 template <class _Tp, bool __b>
 _LIBCPP_CONSTEXPR bool __atomic_base<_Tp, __b>::is_always_lock_free;
+template <class _Tp, bool __b>
+_LIBCPP_CONSTEXPR bool __atomic_ref_base<_Tp, __b>::is_always_lock_free;
 #endif
 
 // atomic<Integral>
 
 template <class _Tp>
 struct __atomic_base<_Tp, true>
     : public __atomic_base<_Tp, false>
 {
struct __atomic_base<_Tp, true>
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator|=(_Tp __op) _NOEXCEPT          {return fetch_or(__op) | __op;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator^=(_Tp __op) volatile _NOEXCEPT {return fetch_xor(__op) ^ __op;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator^=(_Tp __op) _NOEXCEPT          {return fetch_xor(__op) ^ __op;}
 };
 
+// atomic_ref<Integral>
+
+template <class _Tp>
+struct __atomic_ref_base<_Tp, true>
+    : public __atomic_ref_base<_Tp, false>
+{
+    typedef __atomic_ref_base<_Tp, false> __base;
+
+    _LIBCPP_INLINE_VISIBILITY _LIBCPP_EXPLICIT
+    __atomic_ref_base(_Tp& __d) _NOEXCEPT : __base(__d) {}
+    _LIBCPP_INLINE_VISIBILITY
+    __atomic_ref_base(const __atomic_ref_base& other) _NOEXCEPT
+        : __base(other) {}
+
+    _Tp operator=(const _Tp __d) const _NOEXCEPT {
+      return __base::operator=(__d);
+    }
+
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp fetch_add(_Tp __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp fetch_sub(_Tp __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp fetch_and(_Tp __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_and(&this->__a_, __op, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp fetch_or(_Tp __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_or(&this->__a_, __op, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp fetch_xor(_Tp __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_xor(&this->__a_, __op, __m);}
+
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator++(int) const _NOEXCEPT               {return fetch_add(_Tp(1));}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator--(int) const _NOEXCEPT               {return fetch_sub(_Tp(1));}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator++() const _NOEXCEPT                  {return fetch_add(_Tp(1)) + _Tp(1);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator--() const _NOEXCEPT                  {return fetch_sub(_Tp(1)) - _Tp(1);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator+=(_Tp __op) const _NOEXCEPT          {return fetch_add(__op) + __op;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator-=(_Tp __op) const _NOEXCEPT          {return fetch_sub(__op) - __op;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator&=(_Tp __op) const _NOEXCEPT          {return fetch_and(__op) & __op;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator|=(_Tp __op) const _NOEXCEPT          {return fetch_or(__op) | __op;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp operator^=(_Tp __op) const _NOEXCEPT          {return fetch_xor(__op) ^ __op;}
+};
+
 // atomic<T>
 
 template <class _Tp>
 struct atomic
     : public __atomic_base<_Tp>
 {
     typedef __atomic_base<_Tp> __base;
     typedef _Tp value_type;
     typedef value_type difference_type;
     _LIBCPP_INLINE_VISIBILITY
     atomic() _NOEXCEPT _LIBCPP_DEFAULT
     _LIBCPP_INLINE_VISIBILITY
     _LIBCPP_CONSTEXPR atomic(_Tp __d) _NOEXCEPT : __base(__d) {}
 
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator=(_Tp __d) volatile _NOEXCEPT
         {__base::store(__d); return __d;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator=(_Tp __d) _NOEXCEPT
         {__base::store(__d); return __d;}
 };
 
+// atomic_ref<T>
+template <class _Tp>
+struct atomic_ref
+  : public __atomic_ref_base<_Tp>
+{
+  typedef __atomic_ref_base<_Tp> __base;
+  _LIBCPP_INLINE_VISIBILITY
+  _LIBCPP_EXPLICIT atomic_ref(_Tp& obj) : __base(obj) {}
+  _LIBCPP_INLINE_VISIBILITY
+  atomic_ref(const atomic_ref& ref) _NOEXCEPT : __base(ref) {}
+
+  _LIBCPP_INLINE_VISIBILITY
+  _Tp operator=(const _Tp __d) const _NOEXCEPT {
+    return __base::operator=(__d);
+  }
+};
+
 // atomic<T*>
 
 template <class _Tp>
 struct atomic<_Tp*>
     : public __atomic_base<_Tp*>
 {
     typedef __atomic_base<_Tp*> __base;
     typedef _Tp* value_type;
struct atomic<_Tp*>
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator+=(ptrdiff_t __op) _NOEXCEPT          {return fetch_add(__op) + __op;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator-=(ptrdiff_t __op) volatile _NOEXCEPT {return fetch_sub(__op) - __op;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator-=(ptrdiff_t __op) _NOEXCEPT          {return fetch_sub(__op) - __op;}
 };
 
+// atomic_ref<T*>
+
+template <class _Tp>
+struct atomic_ref<_Tp*>
+    : public __atomic_ref_base<_Tp*>
+{
+  typedef __atomic_ref_base<_Tp*> __base;
+  _LIBCPP_INLINE_VISIBILITY
+  _LIBCPP_EXPLICIT atomic_ref(_Tp*& obj) : __base(obj) {}
+  _LIBCPP_INLINE_VISIBILITY
+  atomic_ref(const atomic_ref& ref) _NOEXCEPT : __base(ref) {}
+  
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator=(_Tp* __d) const _NOEXCEPT {
+      return __base::operator=(__d);
+    }
+
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* fetch_add(ptrdiff_t __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* fetch_sub(ptrdiff_t __op, memory_order __m = memory_order_seq_cst) const _NOEXCEPT
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
+
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator++(int) const _NOEXCEPT                     {return fetch_add(1);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator--(int) const _NOEXCEPT                     {return fetch_sub(1);}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator++() const _NOEXCEPT                        {return fetch_add(1) + 1;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator--() const _NOEXCEPT                        {return fetch_sub(1) - 1;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator+=(ptrdiff_t __op) const _NOEXCEPT          {return fetch_add(__op) + __op;}
+    _LIBCPP_INLINE_VISIBILITY
+    _Tp* operator-=(ptrdiff_t __op) const _NOEXCEPT          {return fetch_sub(__op) - __op;}
+};
+
 // atomic_is_lock_free
 
 template <class _Tp>
 _LIBCPP_INLINE_VISIBILITY
 bool
 atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT
 {
     return __o->is_lock_free();

libcxx/test/std/atomics/atomics.lockfree/isalwayslockfree.pass.cpp

 
 #include "test_macros.h"
 
 #if !defined(__cpp_lib_atomic_is_always_lock_free)
 # error Feature test macro missing.
 #endif
 
 template <typename T> void checkAlwaysLockFree() {
-  if (std::atomic<T>::is_always_lock_free)
+  assert(std::atomic<T>::is_always_lock_free ==
+         std::atomic_ref<T>::is_always_lock_free);
+  if (std::atomic<T>::is_always_lock_free) {
     assert(std::atomic<T>().is_lock_free());
+    T val;
+    assert(std::atomic_ref<T>(val).is_lock_free());
+  }
 }
 
 // FIXME: This separate test is needed to work around llvm.org/PR31864
 // which causes ATOMIC_LLONG_LOCK_FREE to be defined as '1' in 32-bit builds
 // even though __atomic_always_lock_free returns true for the same type.
 constexpr bool NeedWorkaroundForPR31864 =
 #if defined(__clang__)
 (sizeof(void*) == 4); // Needed on 32 bit builds

libcxx/test/std/atomics/atomics.types.generic/address.pass.cpp

 #include <atomic>
 #include <new>
 #include <type_traits>
 #include <cassert>
 
 #include <cmpxchg_loop.h>
 
 #include "test_macros.h"
+#include "test_compare_exchange_helpers.h"
 
 template <class A, class T>
 void
 do_test()
 {
     typedef typename std::remove_pointer<T>::type X;
     A obj(T(0));
     bool b0 = obj.is_lock_free();
assert(obj == T(2*sizeof(X)));
         TEST_ALIGNAS_TYPE(A) char storage[sizeof(A)] = {23};
         A& zero = *new (storage) A();
         assert(zero == T(0));
         zero.~A();
     }
 }
 
 template <class A, class T>
-void test()
-{
-    do_test<A, T>();
-    do_test<volatile A, T>();
+void test() {
 }
 
+/*
+template <class T>
+struct atomic_ref<T*>
+{
+    static constexpr bool is_always_lock_free;
+    static constexpr size_t required_alignment;
+
+    explicit atomic_ref(T* obj) noexcept;
+    atomic_ref(const atomic_ref&) noexcept;
+    T* operator=(T* desired) const noexcept;
+    atomic_ref& operator=(const atomic_ref&) = delete;
+
+    bool is_lock_free() const noexcept;
+    void store(T* desr, memory_order m = memory_order_seq_cst) const noexcept;
+    T* load(memory_order m = memory_order_seq_cst) const noexcept;
+    operator T*() const noexcept;
+    T* exchange(T* desr, memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_weak(T*& expc, T* desr, memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_strong(T*& expc, T* desr,
+                                 memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_weak(T*& expc, T* desr,
+                               memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_strong(T*& expc, T* desr,
+                                 memory_order m = memory_order_seq_cst) const noexcept;
+
+    T* fetch_add(ptrdiff_t op, memory_order m = memory_order_seq_cst) noexcept;
+    T* fetch_sub(ptrdiff_t op, memory_order m = memory_order_seq_cst) noexcept;
+
+    T* operator++(int) const noexcept;
+    T* operator--(int) const noexcept;
+    T* operator++() const noexcept;
+    T* operator--() const noexcept;
+    T* operator+=(ptrdiff_t op) const noexcept;
+    T* operator-=(ptrdiff_t op) const noexcept;
+}; */
+
+template <class T>
+void test_atomic_ref() {
+  using atomic_ref_t = std::atomic_ref<T>;
+  assert(atomic_ref_t::is_always_lock_free);
+  assert(atomic_ref_t::required_alignment >= sizeof(T));
+  assert(atomic_ref_t::required_alignment % sizeof(T) == 0);
+  constexpr T base_pointer = nullptr;
+  T underlying = base_pointer;
+  const atomic_ref_t test_ref(underlying);
+  assert(test_ref.is_lock_free());
+
+  test_ref.store(base_pointer + 1);
+  assert(test_ref.load() == base_pointer + 1);
+  test_ref.store(base_pointer + 2, std::memory_order_relaxed);
+  assert(test_ref.load(std::memory_order_relaxed) == base_pointer + 2);
+
+  test_ref.store(base_pointer + 3);
+  assert(static_cast<T>(test_ref) == base_pointer + 3);
+
+  assert(test_ref.exchange(base_pointer + 4) == base_pointer + 3);
+  assert(test_ref.exchange(base_pointer + 5, std::memory_order_relaxed) ==
+         base_pointer + 4);
+  assert(test_ref.load() == base_pointer + 5);
+
+  test_compare_exchange_weak<T>(test_ref, base_pointer);
+  test_compare_exchange_weak<T, std::memory_order_relaxed>(test_ref,
+                                                           base_pointer);
+  test_compare_exchange_weak<T, std::memory_order_relaxed,
+                             std::memory_order_relaxed>(test_ref, base_pointer);
+
+  test_compare_exchange_strong<T>(test_ref, base_pointer);
+  test_compare_exchange_strong<T, std::memory_order_relaxed>(test_ref,
+                                                             base_pointer);
+  test_compare_exchange_strong<T, std::memory_order_relaxed,
+                               std::memory_order_relaxed>(test_ref,
+                                                          base_pointer);
+
+  test_ref.store(base_pointer);
+  assert(test_ref.fetch_add(std::ptrdiff_t(1)) == base_pointer);
+  assert(test_ref.fetch_add(std::ptrdiff_t(2), std::memory_order_relaxed) ==
+         base_pointer + 1);
+  assert(test_ref.fetch_sub(std::ptrdiff_t(2)) == base_pointer + 3);
+  assert(test_ref.fetch_sub(std::ptrdiff_t(1), std::memory_order_relaxed) ==
+         base_pointer + 1);
+  assert(test_ref.load() == base_pointer);
+
+  test_ref.store(base_pointer + 2);
+  assert(test_ref.load() == base_pointer + 2);
+  assert(underlying == base_pointer + 2);
+
+  const atomic_ref_t test_ref_2(test_ref);
+  assert(test_ref_2.load() == base_pointer + 2);
+  assert((test_ref_2 = base_pointer + 3) == base_pointer + 3);
+  assert(test_ref.load() == base_pointer + 3);
+  assert(underlying == base_pointer + 3);
+
+  assert(test_ref_2++ == base_pointer + 3);
+  assert(test_ref_2.load() == base_pointer + 4);
+  assert(++test_ref_2 == base_pointer + 5);
+  assert(test_ref_2.load() == base_pointer + 5);
+
+  assert(test_ref_2-- == base_pointer + 5);
+  assert(test_ref_2.load() == base_pointer + 4);
+  assert(--test_ref_2 == base_pointer + 3);
+  assert(test_ref_2.load() == base_pointer + 3);
+
+  assert((test_ref_2 += std::ptrdiff_t(2)) == base_pointer + 5);
+  assert((test_ref_2 -= std::ptrdiff_t(1)) == base_pointer + 4);
+
+  assert(test_ref_2.load() == base_pointer + 4);
+  assert(test_ref.load() == base_pointer + 4);
+  assert(underlying == base_pointer + 4);
+}
+
 int main(int, char**)
 {
     test<std::atomic<int*>, int*>();
+    test_atomic_ref<int*>();
 
     return 0;
 }

libcxx/test/std/atomics/atomics.types.generic/assign_to_atomic_ref_deleted.fail.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// <atomic>
+
+// template <class T>
+// struct atomic_ref
+// {
+//     static constexpr bool is_always_lock_free;
+//     static constexpr size_t required_alignment;
+//
+//     explicit atomic_ref(T& obj) noexcept;
+//     atomic_ref(const atomic_ref&) noexcept;
+//     T operator=(T desired) const noexcept;
+//     atomic_ref& operator=(const atomic_ref&) = delete;
+//
+//     bool is_lock_free() const noexcept;
+//     void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     T load(memory_order m = memory_order_seq_cst) const noexcept;
+//     operator T() const noexcept;
+//     T exchange(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr, memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr,
+//                                memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order m = memory_order_seq_cst) const noexcept;
+// };
+
+#include <atomic>
+#include <new>
+#include <cassert>
+
+int main(int, char**) {
+  int val = 1;
+  std::atomic_ref<int> t0(val);
+  std::atomic_ref<int> t1(val);
+  t0 =
+      t1; // expected-error {{object of type 'std::atomic_ref<int>' cannot be assigned because its copy assignment operator is implicitly deleted}}
+
+  return 0;
+}

libcxx/test/std/atomics/atomics.types.generic/integral.pass.cpp

 
 #include <atomic>
 #include <new>
 #include <cassert>
 
 #include <cmpxchg_loop.h>
 
 #include "test_macros.h"
+#include "test_compare_exchange_helpers.h"
 
 template <class A, class T>
 void
 do_test()
 {
     A obj(T(0));
     assert(obj == T(0));
     std::atomic_init(&obj, T(1));

 
 template <class A, class T>
 void test()
 {
     do_test<A, T>();
     do_test<volatile A, T>();
 }
 
+/*
+template <>
+struct atomic_ref<integral>
+{
+    static constexpr bool is_always_lock_free;
+    static constexpr size_t required_alignment;
+
+    bool is_lock_free() const noexcept;
+    void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+    integral load(memory_order m = memory_order_seq_cst) const noexcept;
+    operator integral() const noexcept;
+    integral exchange(integral desr, memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_weak(integral& expc, integral desr, memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_strong(integral& expc, integral desr,
+                                 memory_order s, memory_order f) const noexcept;
+    bool compare_exchange_weak(integral& expc, integral desr,
+                               memory_order m = memory_order_seq_cst) const noexcept;
+    bool compare_exchange_strong(integral& expc, integral desr,
+                                 memory_order m = memory_order_seq_cst) const noexcept;
+
+    integral fetch_add(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_sub(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_and(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_or(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+    integral fetch_xor(integral op, memory_order m = memory_order_seq_cst) const noexcept;
+
+    explicit atomic_ref(integral& obj) noexcept;
+    atomic_ref(const atomic_ref&) noexcept;
+    integral operator=(integral desired) const noexcept;
+    atomic_ref& operator=(const atomic_ref&) = delete;
+
+    integral operator++(int) const noexcept;
+    integral operator--(int) const noexcept;
+    integral operator++() const noexcept;
+    integral operator--() const noexcept;
+    integral operator+=(integral op) const noexcept;
+    integral operator-=(integral op) const noexcept;
+    integral operator&=(integral op) const noexcept;
+    integral operator|=(integral op) const noexcept;
+    integral operator^=(integral op) const noexcept;
+};
+*/
+
+template <class T>
+void test_atomic_ref() {
+  constexpr T initial_value = 42;
+  T underlying(42);
+  using atomic_ref_t = std::atomic_ref<T>;
+  const atomic_ref_t test_ref(underlying);
+
+  const auto is_lock_free = test_ref.is_lock_free();
+  (void)is_lock_free;
+  constexpr auto is_always_lock_free = atomic_ref_t::is_always_lock_free;
+  (void)is_always_lock_free;
+  constexpr auto required_alignment = atomic_ref_t::required_alignment;
+  assert(required_alignment >= sizeof(T));
+  assert(required_alignment % sizeof(T) == 0);
+  assert(test_ref.load(std::memory_order_relaxed) == initial_value);
+
+  assert(test_ref.load() == initial_value);
+  test_ref.store(initial_value + T(1), std::memory_order_relaxed);
+  assert(test_ref.load(std::memory_order_relaxed) == initial_value + T(1));
+  test_ref.store(initial_value + T(2));
+  assert(test_ref.load() == initial_value + T(2));
+  assert(static_cast<T>(test_ref) == initial_value + T(2));
+
+  assert(test_ref.exchange(initial_value) == initial_value + T(2));
+  assert(test_ref.exchange(initial_value - T(1), std::memory_order_relaxed) ==
+         initial_value);
+  assert(test_ref.load() == initial_value - T(1));
+
+  test_compare_exchange_weak<T>(test_ref, initial_value);
+  test_compare_exchange_weak<T, std::memory_order_relaxed>(test_ref,
+                                                           initial_value);
+  test_compare_exchange_weak<T, std::memory_order_relaxed,
+                             std::memory_order_relaxed>(test_ref,
+                                                        initial_value);
+
+  test_compare_exchange_strong<T>(test_ref, initial_value);
+  test_compare_exchange_strong<T, std::memory_order_relaxed>(test_ref,
+                                                             initial_value);
+  test_compare_exchange_strong<T, std::memory_order_relaxed,
+                               std::memory_order_relaxed>(test_ref,
+                                                          initial_value);
+
+  test_ref.store(initial_value + T(7));
+  assert(test_ref.fetch_add(T(3)) == initial_value + T(7));
+  assert(test_ref.load() == initial_value + T(10));
+  assert(test_ref.fetch_add(T(2), std::memory_order_relaxed) ==
+         initial_value + T(10));
+  assert(test_ref.load() == initial_value + T(12));
+
+  assert(test_ref.fetch_sub(T(2)) == initial_value + T(12));
+  assert(test_ref.load() == initial_value + T(10));
+  assert(test_ref.fetch_sub(T(3), std::memory_order_relaxed) ==
+         initial_value + T(10));
+  assert(test_ref.load() == initial_value + T(7));
+
+  test_ref.store(T(7));
+  assert(test_ref.fetch_and(T(3)) == T(7));
+  assert(test_ref.load() == T(3));
+  assert(test_ref.fetch_and(T(1), std::memory_order_relaxed) == T(3));
+  assert(test_ref.load() == T(1));
+
+  test_ref.store(T(0));
+  assert(test_ref.fetch_or(T(1)) == T(0));
+  assert(test_ref.load() == T(1));
+  assert(test_ref.fetch_or(T(2), std::memory_order_relaxed) == T(1));
+  assert(test_ref.load() == T(3));
+
+  test_ref.store(T(128) | T(32) | T(8));
+  assert(test_ref.fetch_xor(T(64) | T(16) | T(8)) == (T(128) | T(32) | T(8)));
+  assert(test_ref.load() == (T(128) | T(64) | T(32) | T(16)));
+  assert(
+      test_ref.fetch_xor(T(128) | T(64) | T(16), std::memory_order_relaxed) ==
+      (T(128) | T(64) | T(32) | T(16)));
+  assert(test_ref.load() == T(32));
+
+  assert(underlying == T(32));
+  const atomic_ref_t test_ref_2(test_ref);
+  assert(test_ref_2.load() == T(32));
+  assert((test_ref_2 = T(64)) == T(64));
+  assert(test_ref.load() == T(64));
+  assert(test_ref_2.load() == T(64));
+
+  assert(test_ref_2++ == T(64));
+  assert(static_cast<T>(test_ref_2) == T(65));
+  assert(++test_ref_2 == T(66));
+  assert(test_ref_2.load() == T(66));
+  assert(test_ref_2-- == T(66));
+  assert(test_ref_2.load() == T(65));
+  assert(--test_ref_2 == T(64));
+  assert(test_ref_2.load() == T(64));
+
+  assert((test_ref_2 += T(16)) == T(80));
+  assert((test_ref_2 -= T(32)) == T(48));
+  assert(test_ref_2.load() == T(48));
+  assert((test_ref_2 = T(31)) == T(31));
+  assert((test_ref_2 &= T(15)) == T(15));
+  assert(test_ref_2.load() == T(15));
+  assert((test_ref_2 |= T(16)) == T(31));
+  assert(test_ref_2.load() == T(31));
+  constexpr T expected_final_value = T(1) | T(2) | T(4) | T(8) | T(64);
+  assert((test_ref_2 ^= (T(16) | T(64))) == expected_final_value);
+  assert(test_ref_2.load() == expected_final_value);
+  assert(test_ref.load() == expected_final_value);
+  assert(underlying == expected_final_value);
+}
 
 int main(int, char**)
 {
     test<std::atomic_char, char>();
     test<std::atomic_schar, signed char>();
     test<std::atomic_uchar, unsigned char>();
     test<std::atomic_short, short>();
     test<std::atomic_ushort, unsigned short>();
#endif  // _LIBCPP_HAS_NO_UNICODE_CHARS
     test<volatile std::atomic_uint8_t,  uint8_t>();
     test<volatile std::atomic_int16_t,   int16_t>();
     test<volatile std::atomic_uint16_t, uint16_t>();
     test<volatile std::atomic_int32_t,   int32_t>();
     test<volatile std::atomic_uint32_t, uint32_t>();
     test<volatile std::atomic_int64_t,   int64_t>();
     test<volatile std::atomic_uint64_t, uint64_t>();
 
+    test_atomic_ref<char>();
+    test_atomic_ref<signed char>();
+    test_atomic_ref<unsigned char>();
+    test_atomic_ref<short>();
+    test_atomic_ref<unsigned short>();
+    test_atomic_ref<int>();
+    test_atomic_ref<unsigned int>();
+    test_atomic_ref<long>();
+    test_atomic_ref<unsigned long>();
+    test_atomic_ref<long long>();
+    test_atomic_ref<unsigned long long>();
+#ifndef _LIBCPP_HAS_NO_UNICODE_CHARS
+    test_atomic_ref<char16_t>();
+    test_atomic_ref<char32_t>();
+#endif // _LIBCPP_HAS_NO_UNICODE_CHARS
+    test_atomic_ref<wchar_t>();
+
+    test_atomic_ref<int8_t>();
+    test_atomic_ref<uint8_t>();
+    test_atomic_ref<int16_t>();
+    test_atomic_ref<uint16_t>();
+    test_atomic_ref<int32_t>();
+    test_atomic_ref<uint32_t>();
+    test_atomic_ref<int64_t>();
+    test_atomic_ref<uint64_t>();
+
     return 0;
 }

libcxx/test/std/atomics/atomics.types.generic/test_compare_exchange_helpers.h

+#pragma once
+#include <atomic>
+
+template <class T, std::memory_order... memory_orders>
+void test_compare_exchange_weak(const std::atomic_ref<T> test_ref,
+                                const T initial_value) {
+  test_ref.store(initial_value);
+  const T desired_value = T(initial_value + 2);
+  T expected = T(initial_value + 3);
+  assert(!test_ref.compare_exchange_weak(expected, desired_value,
+                                         memory_orders...));
+  assert(expected == initial_value);
+  assert(test_ref.load() == initial_value);
+  while (!test_ref.compare_exchange_weak(expected, desired_value,
+                                         memory_orders...)) {
+    assert(expected == initial_value);
+    assert(test_ref.load() == initial_value);
+  }
+  assert(test_ref.load() == desired_value);
+}
+
+template <class T, std::memory_order... memory_orders>
+void test_compare_exchange_strong(const std::atomic_ref<T> test_ref,
+                                  const T initial_value) {
+  test_ref.store(initial_value);
+  const T desired_value = T(initial_value + 2);
+  T expected = T(initial_value + 3);
+  assert(!test_ref.compare_exchange_strong(expected, desired_value,
+                                           memory_orders...));
+  assert(expected == initial_value);
+  assert(test_ref.load() == initial_value);
+  assert(test_ref.compare_exchange_strong(expected, desired_value,
+                                          memory_orders...));
+  assert(test_ref.load() == desired_value);
+}

libcxx/test/std/atomics/atomics.types.generic/trivially_copyable.pass.cpp

 //     constexpr atomic(T desr) noexcept;
 //     atomic(const atomic&) = delete;
 //     atomic& operator=(const atomic&) = delete;
 //     atomic& operator=(const atomic&) volatile = delete;
 //     T operator=(T) volatile noexcept;
 //     T operator=(T) noexcept;
 // };
 
+// template <class T>
+// struct atomic_ref
+// {
+//     static constexpr bool is_always_lock_free;
+//     static constexpr size_t required_alignment;
+//
+//     explicit atomic_ref(T& obj) noexcept;
+//     atomic_ref(const atomic_ref&) noexcept;
+//     T operator=(T desired) const noexcept;
+//     atomic_ref& operator=(const atomic_ref&) = delete;
+//
+//     bool is_lock_free() const noexcept;
+//     void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     T load(memory_order m = memory_order_seq_cst) const noexcept;
+//     operator T() const noexcept;
+//     T exchange(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr, memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr,
+//                                memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order m = memory_order_seq_cst) const noexcept;
+// };
+
 #include <atomic>
 #include <new>
 #include <cassert>
 #include <thread> // for thread_id
 #include <chrono> // for nanoseconds
 
 #include "test_macros.h"
 
 struct TriviallyCopyable {
     TriviallyCopyable ( int i ) : i_(i) {}
     int i_;
     };
 
 template <class T>
 void test ( T t ) {
     std::atomic<T> t0(t);
+    std::atomic_ref<T> ref_t0(t);
     }
 
 int main(int, char**)
 {
     test(TriviallyCopyable(42));
     test(std::this_thread::get_id());
     test(std::chrono::nanoseconds(2));
 
   return 0;
 }

libcxx/test/std/atomics/atomics.types.generic/trivially_copyable_ref.fail.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// <atomic>
+
+// template <class T>
+// struct atomic_ref
+// {
+//     static constexpr bool is_always_lock_free;
+//     static constexpr size_t required_alignment;
+
+//     bool is_lock_free() const noexcept;
+//     void store(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     T load(memory_order m = memory_order_seq_cst) const noexcept;
+//     operator T() const noexcept;
+//     T exchange(T desr, memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr, memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order s, memory_order f) const noexcept;
+//     bool compare_exchange_weak(T& expc, T desr,
+//                                memory_order m = memory_order_seq_cst) const noexcept;
+//     bool compare_exchange_strong(T& expc, T desr,
+//                                  memory_order m = memory_order_seq_cst) const noexcept;
+
+//     explicit atomic_ref(T& obj) noexcept;
+//     atomic_ref(const atomic_ref&) noexcept;
+//     T operator=(T desired) const noexcept;
+//     atomic_ref& operator=(const atomic_ref&) = delete;
+// };
+
+#include <atomic>
+#include <new>
+#include <cassert>
+#include <thread> // for thread_id
+#include <chrono> // for nanoseconds
+
+struct NotTriviallyCopyable {
+  NotTriviallyCopyable(int i) : i_(i) {}
+  NotTriviallyCopyable(const NotTriviallyCopyable& rhs) : i_(rhs.i_) {}
+  int i_;
+};
+
+template <class T>
+void test(T t) {
+  std::atomic_ref<T> t0(
+      t); // expected-error@atomic:* {{static_assert failed due to requirement 'is_trivially_copyable<NotTriviallyCopyable>::value' "std::atomic_ref<_Tp> requires that '_Tp' be a trivially copyable type"}}
+}
+
+int main(int, char**) {
+  test(NotTriviallyCopyable(42));
+
+  return 0;
+}