decoupling Freestanding atomic<T> from libatomic.a

[__simt__]

Summary

Hello,

I volunteered to write a first patch to make use of a Freestanding configuration in libcxx. The goal of this patch is to decouple non-lock-free atomic<T> from platform libraries, like libatomic.a, when a Freestanding implementation is selected. This plays a big role in the accelerator-integrated library that we are working on.

This patch is based on the GitHub monorepo as of today.

Copy/pasting how this works from my email thread to -dev:

1. Creates a single cxx_atomic_* layer which is an adaptation of either c11_atomic_*, preferentially, or else atomic_*. (For atomic_*, it’s a rename of the layer that’s there now. For __c11_atomic_*, it’s a new but boring wrapper that just forwards calls.)

2. Creates a __cxx_atomic_type<> template that is an alias to what should wrap the _Tp inside of an atomic<T> / atomic_flag. Defaults to the same _Atomic(_Tp) as today, either from C11 or from the adaptation layer.

3. Under #ifdef LIBCXX_FREESTANDING, introduce cxx_locked_atomic_type<> with an implementation of cxx_atomic_*, and redirect cxx_atomic_type<> based on always-lock-freedom. a. In C++17 mode, it uses atomic_is_always_lock_free. b. Otherwise, it uses the C lock-free macros as shown in wg21.link/p0152.

Thanks for looking this over,

Olivier

[__simt__]

Just a clarification - please evaluate the design aspects first. There are nits that I know are wrong and am still working on.

[jfb]

One downside to forwarding layers is that if you don't have an always inline attribute, some build might not inline. This leads to code bloat and crappy codegen because the memory_order attribute is no longer a known constexpr value. Maybe we should have always inline?

You don't define LIBCXX_FREESTANDING yet, right?

I think at a high level this looks fine.

libcxx/include/atomic

651 ↗

(On Diff #182520)

IIRC this __c11_atomic_* code is only there under #if defined(_LIBCPP_HAS_GCC_ATOMIC_IMP), which works around limitations when you don't have the compiler builtins? clang otherwise defines them in include/clang/Basic/Builtins.def. Have you tried this out on GCC to make sure it works as expected?

FWIW the docs are here: https://clang.llvm.org/docs/LanguageExtensions.html#c11-atomic-builtins

I think the forwarding layer looks fine, just want to make sure we're on the same page.

857 ↗

(On Diff #182520)

Can you add a comment on what #if the #else corresponds to?

1062 ↗

(On Diff #182520)

Fine code indeed 😉

1074 ↗

(On Diff #182520)

Does libcxx have a workaround for this? I know LLVM and Chrome do...

[__simt__]

Updated the patch with a bit higher-quality and better-tested code than what I originally showed.

[__simt__]

In D56913#1363801, @jfb wrote:

One downside to forwarding layers is that if you don't have an always inline attribute, some build might not inline. This leads to code bloat and crappy codegen because the memory_order attribute is no longer a known constexpr value. Maybe we should have always inline?

You don't define LIBCXX_FREESTANDING yet, right?

I think at a high level this looks fine.

This reminded me that I needed the inline visibility macro on this.

I define LIBCXX_FREESTANDING in my tree. I've tested libcxx with it both ON and OFF.

I addressed some of your code comments as well.

The path that calls down to c11_atomic is the main path. The path that calls down to atomic is the fallback for when the C11 primitives aren't available -- this happens to be the case on my platform, I fake out the __atomic path.

[__simt__]

Quick update before a longer update: I have a simpler patch on the way.

[__simt__]

Simplified the changes significantly. By switching my back-end to slide under the C11 side instead of the GCC/Clang side, I can live without the new interposer layer.

Note: I'll describe this in more detail next, but in this patch I also added support for passing through scope information, for scoped memory models.

[__simt__]

Fixed some spurious whitespace changes I didn't intend.

[__simt__]

Would be better if it passed the libcxx tests with the feature turned on. Like now.

[__simt__]

Removed an inadvertent #define left in there for testing.

[jfb]

Should all these macros be in __config?

libcxx/include/atomic

911 ↗

(On Diff #184517)

Can we make the lock uint8_t or something small for platforms where we have operations for it? Or could it rely on something like wg21.link/p1135r3 and use atomic_flag?

1265 ↗

(On Diff #184517)

Do we support atomic in C++03 mode?

[__simt__]

In D56913#1379046, @jfb wrote:

Should all these macros be in __config?

There are 3 macros that I introduced: _LIBCPP_FREESTANDING, _LIBCPP_ATOMIC_FLAG_TYPE, _LIBCPP_ATOMIC_SCOPE_DEFAULT.

They could live in __config, yes.

EDIT: I also added _Atomic_s(T,S). I think you want me to use another name for this because there's no such thing as WG14 _Atomic_s. I'm not married to the name.

libcxx/include/atomic

911 ↗

(On Diff #184517)

It can't be atomic_flag literally, no.

Later in the file I introduce _LIBCPP_ATOMIC_FLAG_TYPE. I could use this here as well.

1265 ↗

(On Diff #184517)

I couldn't get this to build fully successfully without -std=c++11.

Here my change is for consistency, this is what's done on other objects in this file.

[ldionne]

libcxx/include/atomic

896 ↗

(On Diff #184517)

Why is this not in std::?

929 ↗

(On Diff #184517)

It seems wrong to have those be named __c11_atomic_xxx since they have nothing to do with C11. Instead, the code in atomic and __atomic_base could make calls to functions called __atomic_xxx_impl(...), and that could be implemented as C11 atomics in one case and as your __cxx_locked_atomic_type in another case.

So basically, instead of introducing this new version of __c11_atomic_init, rename the existing one to __atomic_init_impl, and then add your version under the name __atomic_init_impl too. What do you think? If you agree this would yield more clarity, we could apply this rename in a separate patch prior to this one.

I don't think the rename would be an ABI break because all those functions are static inline today, so they have internal linkage anyway (and hence they exist in each TU where they're used).

1045 ↗

(On Diff #184517)

Not as cute, but you could use a traditional metafunction instead of an alias to make this potentially work in C++03. Not that I'm attached to supporting atomics in C++03, but it seems like we do (looking at the tests).

1045 ↗

(On Diff #184517)

Also, I think this layer is interesting because it marks the boundary between implementation and interface. Basically, atomic and __atomic_base are both used to provide the interface that's mandated by the standard. __atomic_type is where the actual implementation begins, be it through _Atomic or the __cxx_locked_atomic_type you're introducing.

I believe that making this explicit in the name of the type might add clarity, for example __atomic_impl. WDYT?

1059 ↗

(On Diff #184517)

Can you explain what this scope is?

1063 ↗

(On Diff #184517)

I don't think adding the _Sco template parameter here is an issue from an ABI stand point, because I don't think __atomic_base ever gets exposed at an ABI boundary. At the very least, we don't export any symbol in libc++ that contains this type. But I could be wrong so I'd like someone else to double-check this.

1265 ↗

(On Diff #184517)

I'm confused as to why this is needed. It seems like just atomic() _NOEXCEPT _LIBCPP_DEFAULT should still work fine, as even in the case where _LIBCPP_DEFAULT expands to {}, we get:

atomic() _NOEXCEPT {}

But that should default-construct __base as expected. Do you mean that you tried building in C++03 mode without this change and the tests didn't pass? That sure is surprising. If that's the case, then leave as-is, but otherwise if it's just for consistency then I would stick with what's there.

[__simt__]

Thanks for the comments, Louis, responses below.

I will get working on a new patch with the __cxx_atomic_..._impl layer back in. Shouldn't take long, I can mostly revive it from history.

libcxx/include/atomic

896 ↗

(On Diff #184517)

In the current version it introduces overloads to some symbols that are in the global namespace, and need to be selected against. This being said this version works least-well on Clang and that's probably not good to go. I think the way to address this is the same as your next comment -- reintroduce the interposer layer I had in the first version of the patch.

929 ↗

(On Diff #184517)

The code I have here is exactly how this file already works today, it defines __c11_atomic... when it's not defined. In my first patch version, I added an interposer layer that laundered the N different backend types through the same __cxx_atomic... interface. I will come back with a patch that works that way.

1045 ↗

(On Diff #184517)

SGTM. I'll use impl in the name of the resurrected interposer layer.

1059 ↗

(On Diff #184517)

When it's not short-circuited (like here) its a tag type to dispatch to the right atomic backend for a scoped memory model. i.e. "acquire @device-scope" or "release @system-scope".

This part of the patch is insufficiently-justified, I understand, feel free to ask me more questions.

Here I scoped out (pun intended) the amount of plumbing one would need and it's very little (this line above, and each occurrence of _Sco in the file). You can tell me to rip this out and come back separately but it will affect the non-lock-free adapter in this patch, more significantly than the rest (as you can see).

1265 ↗

(On Diff #184517)

I haven't actually built this in C++03 mode, though I did get some mileage by piggybacking on this path in CUDA, it's not a good enough reason to keep this in the patch if you don't believe the consistency argument. I can just as well drop this part.

[ldionne]

libcxx/include/atomic

896 ↗

(On Diff #184517)

Which symbols in the global namespace are you thinking about?

929 ↗

(On Diff #184517)

Well, I was mostly suggesting this approach to see what you (and others) would think of it. It's really the same thing except we change the name from __c11_atomic_xxx to __atomic_xxx_impl or similar. From your reply I gather that you like this better too.

934 ↗

(On Diff #184517)

You don't need those to be static inline. They are templates, so they are implicitly inline, and the linkage / visibility is controlled by _LIBCPP_INLINE_VISIBILITY (so no need for static).

1059 ↗

(On Diff #184517)

I think I would do that in a separate patch, this way we can have a longer discussion about whether we want it and whether it's the right way of doing it.

1265 ↗

(On Diff #184517)

It's just that I don't understand why it's needed. If I did, maybe the consistency argument would begin to make more sense. I'll add other reviewers who might know better.

1905 ↗

(On Diff #184517)

What's the benefit of macro-izing this? I guess it's because you want to override it under some circumstances?

[__simt__]

In this version I've restored the __cxx_atomic_... layer to which both the GCC and C11 backends map. This addresses the comment about introducing more functions named __c11_atomic... which are not part of the C11 builtin set. I do not introduce any new versions of _Atomic anymore.

There are four new configuration macros used by this patch:

• _LIBCPP_FREESTANDING, requests the mode decoupled from libatomic.a by introducing the non-lock-free specialization. I need to set this in the CUDA configuration.
• _LIBCPP_HAS_EXTERNAL_ATOMIC_IMP, indicates that there is an implementation of __cxx_atomic_... functions outside this file. For the CUDA configuration, I supply a scoped implementation.
• _LIBCPP_ATOMIC_FLAG_TYPE, allows the configuration to override the default type of atomic_flag and the non-lock-free specialization's type (the default is bool). I need to override this to be a 32-bit type, I don't have native 8-bit atomics.
• _LIBCPP_ATOMIC_SCOPE_DEFAULT, is there to plug the plumbing parameter that passes statically typed scope information through __atomic_base to the __cxx_atomic_... layer. It now defaults to a vacuous tag type __cxx_atomic_scope.

There are no new symbols being introduced outside of std::v1::, as there is no longer any attempt to grow the overload set of pre-existing functions.

Sorry if this once-again patch update scrambles the comments a bit.

[ldionne]

I like where this is going generally speaking, but there's a couple of things I'd like to understand. It would also be great if other maintainers with more experience could chime in regarding the rename of __c11_atomic_xxx to __cxx_atomic_xxx -- is this removing functionality that we're providing (but we shouldn't be providing :-)?

There are several things in this patch what will not work in C++03. I guess we have to either:

• Have a discussion about whether <atomic> should support C++03. I vote NO, but I wasn't there when that choice was made and I don't know off the top of my head what the repercussions might be.
• Rewrite all of this patch in a way that works in C++03. It's possible everywhere (sometimes we might need to use an extension like __typeof), but it's going to be a pain in the neck.

libcxx/include/atomic

557 ↗

(On Diff #185202)

IIUC, it doesn't make sense for libc++ to have _LIBCPP_HAS_EXTERNAL_ATOMIC_IMP since it's a purely non-libc++ thing. Am I wrong? If not, please remove this knob for now and we can re-add it later on when we have a reason to (e.g. when/if we introduce scoped atomics into libc++).

588 ↗

(On Diff #185202)

Please get rid of scoped atomics for the moment since those are orthogonal to the rest of the patch.

658 ↗

(On Diff #185202)

This rename of __gcc_atomic::__can_assign to __cxx_atomic_type::__can_assign looks wrong -- __cxx_atomic_type is not a namespace. I don't think this will compile. I think that just __can_assign is what you need? Same comment applies below.

1049 ↗

(On Diff #185202)

I think the first parameter should be __cxx_atomic_type<_Tp*>, but then that function already exists above.

1054 ↗

(On Diff #185202)

Ditto.

1092 ↗

(On Diff #185202)

Please implement _LIBCPP_FREESTANDING in __config as

#ifndef __STDC_HOSTED__
#  define _LIBCPP_FREESTANDING
#endif

This way, _LIBCPP_FREESTANDING will be defined whenever -ffreestanding is passed to Clang, which is the goal IIUC.

1109 ↗

(On Diff #185202)

Indentation looks weird here.

1135 ↗

(On Diff #185202)

None of those functions is going to work in C++03.

651 ↗

(On Diff #182520)

I have the same concern -- I like where this patch is going and TBH I don't think that defining _Atomic and __c11_atomic_xxx operations in a user-facing way in libc++ is a good idea. But I want to understand what we're breaking by removing that.

[__simt__]

I will come back with another patch that addresses these comments.

Also will start a thread to debate C++03 support.

libcxx/include/atomic

557 ↗

(On Diff #185202)

Actually this has been good for me.

First, there is _LIBCPP_HAS_THREAD_API_EXTERNAL, which I've had to set for CUDA in RTC mode. This is analogous to that, in that you don't know what the underlying support is, but something has vouched that it conforms to the contract with the header.

Second, this allows a back-end that isn't GCC or C11 to slide underneath <atomic>. In that case, <atomic> does nothing to help, it assumes outright that the __cxx_atomic... symbols exist. That's the contract.

I would like to keep this one, but like anything else, I can keep it in the downstream.

588 ↗

(On Diff #185202)

Ok. Next patch won't have them.

658 ↗

(On Diff #185202)

Whoops. Sorry. Will fix.

1049 ↗

(On Diff #185202)

It exists in another #ifdef, right? I could probably use the using name here, if it mattered. This path is the C11 path, though, so _Atomic is what we're looking to match.

FWIW, template type deduction involving _Atomic(_Tp) is terrible.

1092 ↗

(On Diff #185202)

Ok. Will do.

896 ↗

(On Diff #184517)

It was __c11_... but now it's moot.

929 ↗

(On Diff #184517)

On Clang it's the only approach that works, AFAICT. GCC was more lenient.

934 ↗

(On Diff #184517)

Ok. I'll clean that up. I don't think this header is particularly good about minimizing the decorations.

1045 ↗

(On Diff #184517)

We should have a top-level discussion about the need for C++03 support.

651 ↗

(On Diff #182520)

Agreed. My code used to live under this backend side, I understand what it does.

But, as you found out just below, I didn't test this version on GCC w/o my primary path active.

[ldionne]

In D56913#1386593, @__simt__ wrote:

I will come back with another patch that addresses these comments.

Great! I'll be away until next Monday, so don't expect replies from me until then.

Also will start a thread to debate C++03 support.

Good, thanks!

Also, I'm still looking for other maintainers to take a look, especially at the part that removes __c11_atomic_xxx functions from the global namespace and removes the #define _Atomic(_Tp). I like those changes, just want to make sure it works for everybody.

libcxx/include/atomic

557 ↗

(On Diff #185202)

Seeing the precedent for _LIBCPP_HAS_THREAD_API_EXTERNAL, I am comfortable with keeping this. Ideally we'd have a design document for <atomic> like we do for <thread> (docs/DesignDocs/ThreadingSupportAPI.rst), but I'm not putting that on you.

1049 ↗

(On Diff #185202)

I don't understand this. You (rightly) removed all functions of the form __cxx_atomic_xxx(_Atomic(_Tp), ...) in favour of __cxx_atomic(__cxx_atomic_type<_Tp*>, ...) but you only left this one. I don't understand why this one is still here.

[__simt__]

In this version:

1. Macros moved to __config.
2. Redundant static/_LIBCPP_INLINE_VISIBILITY simplified.
3. Redundant inline/template simplified.
4. Applied _LIBCPP_DEFAULT where it is sufficient for c++03.
5. Scopes removed.
6. A few cut-n-paste mistakes fixed.

Potentially the last major question I see is whether the new code needs to support c++03 or not.

[__simt__]

I need to test the GCC path better, it still has some bugs. Be right back.

[__simt__]

This version passes libcxx tests with each combination of path that can be used (force GCC, force C11, force freestanding+non-lock-free). There were quite a few problems around volatile that I had not addressed yet, apologies.

[ldionne]

Here's a summary of what I understand the current state of the patch is (mostly for my own reference as a bird's eye view):

#if defined(_LIBCPP_HAS_GCC_ATOMIC_IMP)

    template <typename _Tp>
    struct __cxx_atomic_type {
        // Implementation using a raw _Tp and GCC's __atomic_xxx operations
    };

    template<typename _Tp>
    using __cxx_atomic_base_impl = __cxx_atomic_type<_Tp>;

#elif defined(_LIBCPP_HAS_C_ATOMIC_IMP)

    template <typename _Tp>
    struct __cxx_atomic_type {
        // Implementation using an _Atomic(_Tp) and the __c11_atomic_xxx operations
    };

    template<typename _Tp>
    using __cxx_atomic_base_impl = __cxx_atomic_type<_Tp>;

#endif


#if defined(_LIBCPP_FREESTANDING) && defined(__cpp_lib_atomic_is_always_lock_free)

    template<typename _Tp>
    struct __cxx_atomic_lock_impl {
        // Implementation using a raw _Tp and a lock implemented as a __cxx_atomic_base_impl
    };

    template <class _Tp>
    using __cxx_atomic_impl = _Tp is always lock free ? __cxx_atomic_base_impl<_Tp> : __cxx_atomic_lock_impl<_Tp>;

#else

    template <class _Tp>
    using __cxx_atomic_impl = __cxx_atomic_base_impl<_Tp>;

#endif

template <class _Tp>
struct __atomic_base {
    // Implements the interface of <atomic> and uses a __cxx_atomic_impl<_Tp> under the hood
};

Would it make sense to decide whether we want to use GCC's non-lockfree atomics or not based on a configuration macro that's not _LIBCPP_FREESTANDING?

libcxx/include/atomic

555 ↗

(On Diff #186053)

Wouldn't it be simpler to say:

#if defined(_LIBCPP_HAS_NO_ATOMIC_HEADER)
#  error <atomic> is not implemented
#endif

592 ↗

(On Diff #186053)

I think uses of this can be replaced by declval<_Tp const&>()?

602 ↗

(On Diff #186053)

Would std::is_assignable<_Td&, _Tp const&> work too instead? Those traits are available even in C++03 mode.

641–642 ↗

(On Diff #186053)

Can you use _VSTD::copy(to, end, from) instead? Also, you should mangle the names to __to, __end and __from.

I admit that having to include <algorithm> here sucks, but at least please lift this definition into a function you can reuse below. In the future we should simply put copy and other basic algorithms in a header like <__algorithm_base> that can be used in places where we don't need all of <algorithm>.

EDIT: Nevermind this: I just saw this was a pre-existing condition in the code. Keeping the comment here to point out this is not great, but you don't have to fix it in this patch.

718 ↗

(On Diff #186053)

I'm not seeing the value of this type not being called __cxx_atomic_base_impl directly. Is there any reason for this?

973 ↗

(On Diff #186053)

Likewise, this type could be called __cxx_atomic_base_impl directly?

1146 ↗

(On Diff #186053)

This #endif doesn't look right anymore.

1199 ↗

(On Diff #186053)

__reader & friends should be mangled.

1215 ↗

(On Diff #186053)

Can you please put the semi-colon on its own line, like so:

while (1 == __cxx_atomic_exchange(...))
    /* spin */;

or something like that. Just to make it obvious we're not calling __f in the loop. This applies elsewhere too.

[__simt__]

Would it make sense to decide whether we want to use GCC's non-lockfree atomics or not based on a configuration macro that's not _LIBCPP_FREESTANDING?

Sure it can make sense. That bridge is somewhere in my future, even. When I could do now is create another _LIBCPP... macro that gets set in the __config portion for atomics when _LIBCPP_FREESTANDING is set.

libcxx/include/atomic

555 ↗

(On Diff #186053)

Yes. I'll do that for the next one.

592 ↗

(On Diff #186053)

These are there for c++03 support, I think. Much of this section is borrowed from the previous version, but expanded to suit the additional uses in the file now.

602 ↗

(On Diff #186053)

Are they? I can probably use them then. When I test in c++03 mode next, I'll experiment with the substitution.

641–642 ↗

(On Diff #186053)

Thanks for the EDIT... because the patch is trending towards fixing-all-the-things.

Also, introducing a dependency on <algorithm> is a problem for me right now. I'd like to avoid it for a while longer, or else create the <__algorithm_base> at that time.

718 ↗

(On Diff #186053)

This makes more sense in the scoped version. In this version I can do what you requested here, that's what I'll come back with.

973 ↗

(On Diff #186053)

Likewise.

1146 ↗

(On Diff #186053)

Right. Will fix.

1199 ↗

(On Diff #186053)

Ok. But if we retain c++03 support then this is going to have to change. Or at least be guarded by another #if condition if we temporarily force c++11 support.

1215 ↗

(On Diff #186053)

Will do.

[__simt__]

This version addresses the preceding comments and passes libcxx tests across c++03, 11, 14 and 17 modes.

[ldionne]

LGTM, with some de-duplication suggested. I'm not an atomic wrangler, though, so I'd feel more comfortable if someone like @jfb could quickly double-check the implementation of the atomic operations.

libcxx/include/atomic

1198 ↗

(On Diff #186508)

__cxx_atomic_compare_exchange_weak and __cxx_atomic_compare_exchange_strong seem the same. Should one just forward to the other?

[__simt__]

Dear other reviewers, what else needs to happen here?

[__simt__]

Hey guys, can I get additional feedback or else proceed with this patch?

[ldionne]

Given that I and JF have reviewed the patch and we're OK with it, I'll merge this at the end of the week unless other reviewers chime in. Changes can always be made as a followup, but I'd like to avoid delaying this too much (especially since this is a large change that will quickly become hard to rebase).

[ldionne]

libcxx/include/atomic

1337 ↗

(On Diff #186508)

It just occurred to me that you define this function (and others) that are only used for the atomic<Integral> specialization. Is there a case in which an integral type would be a non-lockfree atomic? Currently, we do not test non-lockfree integral types in the test suite (at least not on my platform, where I don't think there are non-lockfree atomics for integral types).

[__simt__]

libcxx/include/atomic

1337 ↗

(On Diff #186508)

I'm not aware of any non-lock-free integral or pointer types on a platform that libcxx supports at this moment, but it's conceivable. The minimum HW requirement for C++ implementations is to have at least one of: byte test-and-set, or byte exchange. There exist real platforms that come close to this minimum, and would have non-lock-free atomic_int.

Also, I tested this code with an override to force almost everything to take the non-lock-free path, even when the lock-free path could have been used. So while it's not exercised via normal testing, I have lightly tested it. Maybe twice. :)

You could keep or delete them, either is OK with me, but personally I'd keep them.

[davide]

This commit broke the atomic lldb data formatter.

http://lab.llvm.org:8080/green/view/LLDB/job/lldb-cmake/21037/

@shafik @jingham

[__simt__]

In D56913#1417172, @davide wrote:

This commit broke the atomic lldb data formatter.

http://lab.llvm.org:8080/green/view/LLDB/job/lldb-cmake/21037/

@shafik @jingham

It looks like LLDB is naming internal names here. Unfortunately, it now needs to learn the new names. On the plus side, there is now a single stable internal name for all compilation paths for this header, whereas it used to differ. So the final result ought to be seen as an improvement.

[davide]

In D56913#1417184, @__simt__ wrote:

In D56913#1417172, @davide wrote:

This commit broke the atomic lldb data formatter.

http://lab.llvm.org:8080/green/view/LLDB/job/lldb-cmake/21037/

@shafik @jingham

It looks like LLDB is naming internal names here. Unfortunately, it now needs to learn the new names. On the plus side, there is now a single stable internal name for all compilation paths for this header, whereas it used to differ. So the final result ought to be seen as an improvement.

Nobody disagrees but the formatter has to be fixed, hence the two people cc:ed.

[ldionne]

In D56913#1417198, @davide wrote:

In D56913#1417184, @__simt__ wrote:

In D56913#1417172, @davide wrote:

This commit broke the atomic lldb data formatter.

http://lab.llvm.org:8080/green/view/LLDB/job/lldb-cmake/21037/

@shafik @jingham

It looks like LLDB is naming internal names here. Unfortunately, it now needs to learn the new names. On the plus side, there is now a single stable internal name for all compilation paths for this header, whereas it used to differ. So the final result ought to be seen as an improvement.

Nobody disagrees but the formatter has to be fixed, hence the two people cc:ed.

Sorry -- I should have run the tests for the LLDB formatters. This is not the first time it happens.

However, seeing how error-prone it is to change something in libc++ without running the LLDB tests, would it make sense to test the data formatters as part of libc++ itself? One way to see the data formatters is as something that libc++ provides to work nicely with LLDB instead of as something LLDB tries to provide for libc++. When seen that way, it makes more sense that the data formatters use private parts of libc++, and the tests should be part of libc++'s test suite. I don't know how big of a change this is, but I'd encourage LLDB folks to consider this as this would reduce a cause of semi-frequent mistakes.

[shafik]

In D56913#1417291, @ldionne wrote:

In D56913#1417198, @davide wrote:

In D56913#1417184, @__simt__ wrote:

In D56913#1417172, @davide wrote:

This commit broke the atomic lldb data formatter.

http://lab.llvm.org:8080/green/view/LLDB/job/lldb-cmake/21037/

@shafik @jingham

It looks like LLDB is naming internal names here. Unfortunately, it now needs to learn the new names. On the plus side, there is now a single stable internal name for all compilation paths for this header, whereas it used to differ. So the final result ought to be seen as an improvement.

Nobody disagrees but the formatter has to be fixed, hence the two people cc:ed.

Sorry -- I should have run the tests for the LLDB formatters. This is not the first time it happens.

However, seeing how error-prone it is to change something in libc++ without running the LLDB tests, would it make sense to test the data formatters as part of libc++ itself? One way to see the data formatters is as something that libc++ provides to work nicely with LLDB instead of as something LLDB tries to provide for libc++. When seen that way, it makes more sense that the data formatters use private parts of libc++, and the tests should be part of libc++'s test suite. I don't know how big of a change this is, but I'd encourage LLDB folks to consider this as this would reduce a cause of semi-frequent mistakes.

Some of us had a brief discussion about this and it would probably require us to move them to python as opposed to C++ and that would not be a trivial amount of work.

I have a fix that I am testing now, I would be happy to explain it to anyone interested in understanding how to work with the formatters once I commit it.

libcxx/trunk/include/__config

 #endif
 
 #define _LIBCPP_VERSION 9000
 
 #ifndef _LIBCPP_ABI_VERSION
 #  define _LIBCPP_ABI_VERSION 1
 #endif
 
+#ifndef __STDC_HOSTED__
+#  define _LIBCPP_FREESTANDING
+#endif
+
 #ifndef _LIBCPP_STD_VER
 #  if  __cplusplus <= 201103L
 #    define _LIBCPP_STD_VER 11
 #  elif __cplusplus <= 201402L
 #    define _LIBCPP_STD_VER 14
 #  elif __cplusplus <= 201703L
 #    define _LIBCPP_STD_VER 17
 #  else

 #endif
 
 #if __has_feature(cxx_atomic) || __has_extension(c_atomic) || __has_keyword(_Atomic)
 #  define _LIBCPP_HAS_C_ATOMIC_IMP
 #elif _GNUC_VER > 407
 #  define _LIBCPP_HAS_GCC_ATOMIC_IMP
 #endif
 
-#if (!defined(_LIBCPP_HAS_C_ATOMIC_IMP) && !defined(_LIBCPP_HAS_GCC_ATOMIC_IMP)) \
+#if (!defined(_LIBCPP_HAS_C_ATOMIC_IMP) && \
+     !defined(_LIBCPP_HAS_GCC_ATOMIC_IMP) && \
+     !defined(_LIBCPP_HAS_EXTERNAL_ATOMIC_IMP)) \
      || defined(_LIBCPP_HAS_NO_THREADS)
-#define _LIBCPP_HAS_NO_ATOMIC_HEADER
+#  define _LIBCPP_HAS_NO_ATOMIC_HEADER
+#else
+#  ifndef _LIBCPP_ATOMIC_FLAG_TYPE
+#    define _LIBCPP_ATOMIC_FLAG_TYPE bool
+#  endif
+#  ifdef _LIBCPP_FREESTANDING
+#    define _LIBCPP_ATOMIC_ONLY_USE_BUILTINS
+#  endif
 #endif
 
 #ifndef _LIBCPP_DISABLE_UBSAN_UNSIGNED_INTEGER_CHECK
 #define _LIBCPP_DISABLE_UBSAN_UNSIGNED_INTEGER_CHECK
 #endif
 
 #if defined(_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS)
 #  if defined(__clang__) && __has_attribute(acquire_capability)

libcxx/trunk/include/atomic

 #include <type_traits>
 #include <version>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 #ifdef _LIBCPP_HAS_NO_THREADS
-#error <atomic> is not supported on this single threaded system
+# error <atomic> is not supported on this single threaded system
 #endif
-#if !defined(_LIBCPP_HAS_C_ATOMIC_IMP) && !defined(_LIBCPP_HAS_GCC_ATOMIC_IMP)
-#error <atomic> is not implemented
+#ifdef _LIBCPP_HAS_NO_ATOMIC_HEADER
+# error <atomic> is not implemented
 #endif
 #ifdef kill_dependency
-#error C++ standard library is incompatible with <stdatomic.h>
+# error C++ standard library is incompatible with <stdatomic.h>
 #endif
 
 #define _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m) \
   _LIBCPP_DIAGNOSE_WARNING(__m == memory_order_consume || \
                            __m == memory_order_acquire || \
                            __m == memory_order_acq_rel,   \
                         "memory order argument to atomic operation is invalid")
 

 _LIBCPP_BEGIN_NAMESPACE_STD
 
 typedef enum memory_order
 {
     memory_order_relaxed, memory_order_consume, memory_order_acquire,
     memory_order_release, memory_order_acq_rel, memory_order_seq_cst
 } memory_order;
 
-#if defined(_LIBCPP_HAS_GCC_ATOMIC_IMP)
-namespace __gcc_atomic {
-template <typename _Tp>
-struct __gcc_atomic_t {
+#if defined(_LIBCPP_HAS_GCC_ATOMIC_IMP) || \
+    defined(_LIBCPP_ATOMIC_ONLY_USE_BUILTINS)
+
+// [atomics.types.generic]p1 guarantees _Tp is trivially copyable. Because
+// the default operator= in an object is not volatile, a byte-by-byte copy
+// is required.
+template <typename _Tp, typename _Tv> _LIBCPP_INLINE_VISIBILITY
+typename enable_if<is_assignable<_Tp&, _Tv>::value>::type
+__cxx_atomic_assign_volatile(_Tp& __a_value, _Tv const& __val) {
+  __a_value = __val;
+}
+template <typename _Tp, typename _Tv> _LIBCPP_INLINE_VISIBILITY
+typename enable_if<is_assignable<_Tp&, _Tv>::value>::type
+__cxx_atomic_assign_volatile(_Tp volatile& __a_value, _Tv volatile const& __val) {
+  volatile char* __to = reinterpret_cast<volatile char*>(&__a_value);
+  volatile char* __end = __to + sizeof(_Tp);
+  volatile const char* __from = reinterpret_cast<volatile const char*>(&__val);
+  while (__to != __end)
+    *__to++ = *__from++;
+}
 
-#if _GNUC_VER >= 501
-    static_assert(is_trivially_copyable<_Tp>::value,
-      "std::atomic<Tp> requires that 'Tp' be a trivially copyable type");
 #endif
 
+#if defined(_LIBCPP_HAS_GCC_ATOMIC_IMP)
+
+template <typename _Tp>
+struct __cxx_atomic_base_impl {
+
   _LIBCPP_INLINE_VISIBILITY
 #ifndef _LIBCPP_CXX03_LANG
-    __gcc_atomic_t() _NOEXCEPT = default;
+    __cxx_atomic_base_impl() _NOEXCEPT = default;
 #else
-    __gcc_atomic_t() _NOEXCEPT : __a_value() {}
+    __cxx_atomic_base_impl() _NOEXCEPT : __a_value() {}
 #endif // _LIBCPP_CXX03_LANG
-  _LIBCPP_CONSTEXPR explicit __gcc_atomic_t(_Tp value) _NOEXCEPT
+  _LIBCPP_CONSTEXPR explicit __cxx_atomic_base_impl(_Tp value) _NOEXCEPT
     : __a_value(value) {}
   _Tp __a_value;
 };
-#define _Atomic(x) __gcc_atomic::__gcc_atomic_t<x>
-
-template <typename _Tp> _Tp __create();
-
-template <typename _Tp, typename _Td>
-typename enable_if<sizeof(_Tp()->__a_value = __create<_Td>()), char>::type
-    __test_atomic_assignable(int);
-template <typename _Tp, typename _Up>
-__two __test_atomic_assignable(...);
-
-template <typename _Tp, typename _Td>
-struct __can_assign {
-  static const bool value =
-      sizeof(__test_atomic_assignable<_Tp, _Td>(1)) == sizeof(char);
-};
 
-static inline _LIBCPP_CONSTEXPR int __to_gcc_order(memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY inline _LIBCPP_CONSTEXPR int __to_gcc_order(memory_order __order) {
   // Avoid switch statement to make this a constexpr.
   return __order == memory_order_relaxed ? __ATOMIC_RELAXED:
          (__order == memory_order_acquire ? __ATOMIC_ACQUIRE:
           (__order == memory_order_release ? __ATOMIC_RELEASE:
            (__order == memory_order_seq_cst ? __ATOMIC_SEQ_CST:
             (__order == memory_order_acq_rel ? __ATOMIC_ACQ_REL:
               __ATOMIC_CONSUME))));
 }
 
-static inline _LIBCPP_CONSTEXPR int __to_gcc_failure_order(memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY inline _LIBCPP_CONSTEXPR int __to_gcc_failure_order(memory_order __order) {
   // Avoid switch statement to make this a constexpr.
   return __order == memory_order_relaxed ? __ATOMIC_RELAXED:
          (__order == memory_order_acquire ? __ATOMIC_ACQUIRE:
           (__order == memory_order_release ? __ATOMIC_RELAXED:
            (__order == memory_order_seq_cst ? __ATOMIC_SEQ_CST:
             (__order == memory_order_acq_rel ? __ATOMIC_ACQUIRE:
               __ATOMIC_CONSUME))));
 }
 
-} // namespace __gcc_atomic
-
-template <typename _Tp>
-static inline
-typename enable_if<
-    __gcc_atomic::__can_assign<volatile _Atomic(_Tp)*, _Tp>::value>::type
-__c11_atomic_init(volatile _Atomic(_Tp)* __a,  _Tp __val) {
-  __a->__a_value = __val;
-}
-
 template <typename _Tp>
-static inline
-typename enable_if<
-    !__gcc_atomic::__can_assign<volatile _Atomic(_Tp)*, _Tp>::value &&
-     __gcc_atomic::__can_assign<         _Atomic(_Tp)*, _Tp>::value>::type
-__c11_atomic_init(volatile _Atomic(_Tp)* __a,  _Tp __val) {
-  // [atomics.types.generic]p1 guarantees _Tp is trivially copyable. Because
-  // the default operator= in an object is not volatile, a byte-by-byte copy
-  // is required.
-  volatile char* to = reinterpret_cast<volatile char*>(&__a->__a_value);
-  volatile char* end = to + sizeof(_Tp);
-  char* from = reinterpret_cast<char*>(&__val);
-  while (to != end) {
-    *to++ = *from++;
-  }
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(volatile __cxx_atomic_base_impl<_Tp>* __a,  _Tp __val) {
+  __cxx_atomic_assign_volatile(__a->__a_value, __val);
 }
 
 template <typename _Tp>
-static inline void __c11_atomic_init(_Atomic(_Tp)* __a,  _Tp __val) {
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp>* __a,  _Tp __val) {
   __a->__a_value = __val;
 }
 
-static inline void __c11_atomic_thread_fence(memory_order __order) {
-  __atomic_thread_fence(__gcc_atomic::__to_gcc_order(__order));
+_LIBCPP_INLINE_VISIBILITY inline
+void __cxx_atomic_thread_fence(memory_order __order) {
+  __atomic_thread_fence(__to_gcc_order(__order));
 }
 
-static inline void __c11_atomic_signal_fence(memory_order __order) {
-  __atomic_signal_fence(__gcc_atomic::__to_gcc_order(__order));
+_LIBCPP_INLINE_VISIBILITY inline
+void __cxx_atomic_signal_fence(memory_order __order) {
+  __atomic_signal_fence(__to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline void __c11_atomic_store(volatile _Atomic(_Tp)* __a,  _Tp __val,
-                                      memory_order __order) {
-  return __atomic_store(&__a->__a_value, &__val,
-                        __gcc_atomic::__to_gcc_order(__order));
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(volatile __cxx_atomic_base_impl<_Tp>* __a,  _Tp __val,
+  __atomic_store(&__a->__a_value, &__val,
+                 __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline void __c11_atomic_store(_Atomic(_Tp)* __a,  _Tp __val,
-                                      memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp>* __a,  _Tp __val,
   __atomic_store(&__a->__a_value, &__val,
-                 __gcc_atomic::__to_gcc_order(__order));
+                 __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_load(const volatile _Atomic(_Tp)* __a,
-                                    memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(const volatile __cxx_atomic_base_impl<_Tp>* __a,
   _Tp __ret;
   __atomic_load(&__a->__a_value, &__ret,
-                __gcc_atomic::__to_gcc_order(__order));
+                __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_load(const _Atomic(_Tp)* __a, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(const __cxx_atomic_base_impl<_Tp>* __a, memory_order __order) {
   _Tp __ret;
   __atomic_load(&__a->__a_value, &__ret,
-                __gcc_atomic::__to_gcc_order(__order));
+                __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_exchange(volatile _Atomic(_Tp)* __a,
-                                        _Tp __value, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(volatile __cxx_atomic_base_impl<_Tp>* __a,
   _Tp __ret;
   __atomic_exchange(&__a->__a_value, &__value, &__ret,
-                    __gcc_atomic::__to_gcc_order(__order));
+                    __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_exchange(_Atomic(_Tp)* __a, _Tp __value,
-                                        memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp>* __a, _Tp __value,
   _Tp __ret;
   __atomic_exchange(&__a->__a_value, &__value, &__ret,
-                    __gcc_atomic::__to_gcc_order(__order));
+                    __to_gcc_order(__order));
   return __ret;
 }
 
 template <typename _Tp>
-static inline bool __c11_atomic_compare_exchange_strong(
-    volatile _Atomic(_Tp)* __a, _Tp* __expected, _Tp __value,
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(
+    volatile __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value,
     memory_order __success, memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    false,
-                                   __gcc_atomic::__to_gcc_order(__success),
-                                   __gcc_atomic::__to_gcc_failure_order(__failure));
+                                   __to_gcc_order(__success),
+                                   __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
-static inline bool __c11_atomic_compare_exchange_strong(
-    _Atomic(_Tp)* __a, _Tp* __expected, _Tp __value, memory_order __success,
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(
+    __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value, memory_order __success,
     memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    false,
-                                   __gcc_atomic::__to_gcc_order(__success),
-                                   __gcc_atomic::__to_gcc_failure_order(__failure));
+                                   __to_gcc_order(__success),
+                                   __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
-static inline bool __c11_atomic_compare_exchange_weak(
-    volatile _Atomic(_Tp)* __a, _Tp* __expected, _Tp __value,
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(
+    volatile __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value,
     memory_order __success, memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    true,
-                                   __gcc_atomic::__to_gcc_order(__success),
-                                   __gcc_atomic::__to_gcc_failure_order(__failure));
+                                   __to_gcc_order(__success),
+                                   __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
-static inline bool __c11_atomic_compare_exchange_weak(
-    _Atomic(_Tp)* __a, _Tp* __expected, _Tp __value, memory_order __success,
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(
+    __cxx_atomic_base_impl<_Tp>* __a, _Tp* __expected, _Tp __value, memory_order __success,
     memory_order __failure) {
   return __atomic_compare_exchange(&__a->__a_value, __expected, &__value,
                                    true,
-                                   __gcc_atomic::__to_gcc_order(__success),
-                                   __gcc_atomic::__to_gcc_failure_order(__failure));
+                                   __to_gcc_order(__success),
+                                   __to_gcc_failure_order(__failure));
 }
 
 template <typename _Tp>
 struct __skip_amt { enum {value = 1}; };
 
 template <typename _Tp>
 struct __skip_amt<_Tp*> { enum {value = sizeof(_Tp)}; };
 
 // FIXME: Haven't figured out what the spec says about using arrays with
 // atomic_fetch_add. Force a failure rather than creating bad behavior.
 template <typename _Tp>
 struct __skip_amt<_Tp[]> { };
 template <typename _Tp, int n>
 struct __skip_amt<_Tp[n]> { };
 
 template <typename _Tp, typename _Td>
-static inline _Tp __c11_atomic_fetch_add(volatile _Atomic(_Tp)* __a,
-                                         _Td __delta, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(volatile __cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_add(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp, typename _Td>
-static inline _Tp __c11_atomic_fetch_add(_Atomic(_Tp)* __a, _Td __delta,
-                                         memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp>* __a, _Td __delta,
   return __atomic_fetch_add(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp, typename _Td>
-static inline _Tp __c11_atomic_fetch_sub(volatile _Atomic(_Tp)* __a,
-                                         _Td __delta, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(volatile __cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_sub(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp, typename _Td>
-static inline _Tp __c11_atomic_fetch_sub(_Atomic(_Tp)* __a, _Td __delta,
-                                         memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp>* __a, _Td __delta,
   return __atomic_fetch_sub(&__a->__a_value, __delta * __skip_amt<_Tp>::value,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_and(volatile _Atomic(_Tp)* __a,
-                                         _Tp __pattern, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(volatile __cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_and(&__a->__a_value, __pattern,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_and(_Atomic(_Tp)* __a,
-                                         _Tp __pattern, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_and(&__a->__a_value, __pattern,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_or(volatile _Atomic(_Tp)* __a,
-                                        _Tp __pattern, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(volatile __cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_or(&__a->__a_value, __pattern,
-                           __gcc_atomic::__to_gcc_order(__order));
+                           __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_or(_Atomic(_Tp)* __a, _Tp __pattern,
-                                        memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp>* __a, _Tp __pattern,
   return __atomic_fetch_or(&__a->__a_value, __pattern,
-                           __gcc_atomic::__to_gcc_order(__order));
+                           __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_xor(volatile _Atomic(_Tp)* __a,
-                                         _Tp __pattern, memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(volatile __cxx_atomic_base_impl<_Tp>* __a,
   return __atomic_fetch_xor(&__a->__a_value, __pattern,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
 }
 
 template <typename _Tp>
-static inline _Tp __c11_atomic_fetch_xor(_Atomic(_Tp)* __a, _Tp __pattern,
-                                         memory_order __order) {
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp>* __a, _Tp __pattern,
   return __atomic_fetch_xor(&__a->__a_value, __pattern,
-                            __gcc_atomic::__to_gcc_order(__order));
+                            __to_gcc_order(__order));
+}
+
+#define __cxx_atomic_is_lock_free(__s) __atomic_is_lock_free(__s, 0)
+
+#elif defined(_LIBCPP_HAS_C_ATOMIC_IMP)
+
+template <typename _Tp>
+struct __cxx_atomic_base_impl {
+
+  _LIBCPP_INLINE_VISIBILITY
+#ifndef _LIBCPP_CXX03_LANG
+    __cxx_atomic_base_impl() _NOEXCEPT = default;
+#else
+    __cxx_atomic_base_impl() _NOEXCEPT : __a_value() {}
+#endif // _LIBCPP_CXX03_LANG
+  _LIBCPP_CONSTEXPR explicit __cxx_atomic_base_impl(_Tp value) _NOEXCEPT
+    : __a_value(value) {}
+  _Atomic(_Tp) __a_value;
+};
+
+#define __cxx_atomic_is_lock_free(__s) __c11_atomic_is_lock_free(__s)
+
+_LIBCPP_INLINE_VISIBILITY inline
+void __cxx_atomic_thread_fence(int __order) {
+    __c11_atomic_thread_fence(__order);
+}
+
+_LIBCPP_INLINE_VISIBILITY inline
+void __cxx_atomic_signal_fence(int __order) {
+    __c11_atomic_signal_fence(__order);
 }
-#endif // _LIBCPP_HAS_GCC_ATOMIC_IMP
 
-template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
-_Tp
-kill_dependency(_Tp __y) _NOEXCEPT
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __val) {
+    __c11_atomic_init(&__a->__a_value, __val);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(__cxx_atomic_base_impl<_Tp> * __a, _Tp __val) {
+    __c11_atomic_init(&__a->__a_value, __val);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __val, int __order) {
+    __c11_atomic_store(&__a->__a_value, __val, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(__cxx_atomic_base_impl<_Tp> * __a, _Tp __val, int __order) {
+    __c11_atomic_store(&__a->__a_value, __val, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(__cxx_atomic_base_impl<_Tp> const volatile* __a, int __order) {
+    using __ptr_type = typename remove_const<decltype(__a->__a_value)>::type*;
+    return __c11_atomic_load(const_cast<__ptr_type>(&__a->__a_value), __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(__cxx_atomic_base_impl<_Tp> const* __a, int __order) {
+    using __ptr_type = typename remove_const<decltype(__a->__a_value)>::type*;
+    return __c11_atomic_load(const_cast<__ptr_type>(&__a->__a_value), __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __value, int __order) {
+    return __c11_atomic_exchange(&__a->__a_value, __value, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(__cxx_atomic_base_impl<_Tp> * __a, _Tp __value, int __order) {
+    return __c11_atomic_exchange(&__a->__a_value, __value, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp* __expected, _Tp __value, int __success, int __failure) {
+    return __c11_atomic_compare_exchange_strong(&__a->__a_value, __expected, __value, __success, __failure);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(__cxx_atomic_base_impl<_Tp> * __a, _Tp* __expected, _Tp __value, int __success, int __failure) {
+    return __c11_atomic_compare_exchange_strong(&__a->__a_value, __expected, __value, __success, __failure);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp* __expected, _Tp __value, int __success, int __failure) {
+    return __c11_atomic_compare_exchange_weak(&__a->__a_value, __expected, __value, __success, __failure);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(__cxx_atomic_base_impl<_Tp> * __a, _Tp* __expected, _Tp __value, int __success, int __failure) {
+    return __c11_atomic_compare_exchange_weak(&__a->__a_value, __expected, __value,  __success, __failure);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __delta, int __order) {
+    return __c11_atomic_fetch_add(&__a->__a_value, __delta, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp> * __a, _Tp __delta, int __order) {
+    return __c11_atomic_fetch_add(&__a->__a_value, __delta, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, int __order) {
+    return __c11_atomic_fetch_add(&__a->__a_value, __delta, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> * __a, ptrdiff_t __delta, int __order) {
+    return __c11_atomic_fetch_add(&__a->__a_value, __delta, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __delta, int __order) {
+    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp> * __a, _Tp __delta, int __order) {
+    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, int __order) {
+    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp*> * __a, ptrdiff_t __delta, int __order) {
+    return __c11_atomic_fetch_sub(&__a->__a_value, __delta, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_and(&__a->__a_value, __pattern, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_and(&__a->__a_value, __pattern, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_or(&__a->__a_value, __pattern, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_or(&__a->__a_value, __pattern, __order);
+}
+
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp> volatile* __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_xor(&__a->__a_value, __pattern, __order);
+}
+template<class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(__cxx_atomic_base_impl<_Tp> * __a, _Tp __pattern, int __order) {
+    return __c11_atomic_fetch_xor(&__a->__a_value, __pattern, __order);
+}
+
+#endif // _LIBCPP_HAS_GCC_ATOMIC_IMP, _LIBCPP_HAS_C_ATOMIC_IMP
+
+template <class _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp kill_dependency(_Tp __y) _NOEXCEPT
 {
     return __y;
 }
 
 #if defined(__CLANG_ATOMIC_BOOL_LOCK_FREE)
 # define ATOMIC_BOOL_LOCK_FREE      __CLANG_ATOMIC_BOOL_LOCK_FREE
 # define ATOMIC_CHAR_LOCK_FREE      __CLANG_ATOMIC_CHAR_LOCK_FREE
 # define ATOMIC_CHAR16_T_LOCK_FREE  __CLANG_ATOMIC_CHAR16_T_LOCK_FREE
 # define ATOMIC_CHAR32_T_LOCK_FREE  __CLANG_ATOMIC_CHAR32_T_LOCK_FREE
 # define ATOMIC_WCHAR_T_LOCK_FREE   __CLANG_ATOMIC_WCHAR_T_LOCK_FREE
 # define ATOMIC_SHORT_LOCK_FREE     __CLANG_ATOMIC_SHORT_LOCK_FREE
 # define ATOMIC_INT_LOCK_FREE       __CLANG_ATOMIC_INT_LOCK_FREE
 # define ATOMIC_LONG_LOCK_FREE      __CLANG_ATOMIC_LONG_LOCK_FREE
 # define ATOMIC_LLONG_LOCK_FREE     __CLANG_ATOMIC_LLONG_LOCK_FREE
 # define ATOMIC_POINTER_LOCK_FREE   __CLANG_ATOMIC_POINTER_LOCK_FREE
-#else
+#elif defined(__GCC_ATOMIC_BOOL_LOCK_FREE)
 # define ATOMIC_BOOL_LOCK_FREE      __GCC_ATOMIC_BOOL_LOCK_FREE
 # define ATOMIC_CHAR_LOCK_FREE      __GCC_ATOMIC_CHAR_LOCK_FREE
 # define ATOMIC_CHAR16_T_LOCK_FREE  __GCC_ATOMIC_CHAR16_T_LOCK_FREE
 # define ATOMIC_CHAR32_T_LOCK_FREE  __GCC_ATOMIC_CHAR32_T_LOCK_FREE
 # define ATOMIC_WCHAR_T_LOCK_FREE   __GCC_ATOMIC_WCHAR_T_LOCK_FREE
 # define ATOMIC_SHORT_LOCK_FREE     __GCC_ATOMIC_SHORT_LOCK_FREE
 # define ATOMIC_INT_LOCK_FREE       __GCC_ATOMIC_INT_LOCK_FREE
 # define ATOMIC_LONG_LOCK_FREE      __GCC_ATOMIC_LONG_LOCK_FREE
 # define ATOMIC_LLONG_LOCK_FREE     __GCC_ATOMIC_LLONG_LOCK_FREE
 # define ATOMIC_POINTER_LOCK_FREE   __GCC_ATOMIC_POINTER_LOCK_FREE
 #endif
 
+#ifdef _LIBCPP_ATOMIC_ONLY_USE_BUILTINS
+
+template<typename _Tp>
+struct __cxx_atomic_lock_impl {
+
+  _LIBCPP_INLINE_VISIBILITY
+  __cxx_atomic_lock_impl() _NOEXCEPT
+    : __a_value(), __a_lock(0) {}
+  _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR explicit
+  __cxx_atomic_lock_impl(_Tp value) _NOEXCEPT
+    : __a_value(value), __a_lock(0) {}
+
+  _Tp __a_value;
+  mutable __cxx_atomic_base_impl<_LIBCPP_ATOMIC_FLAG_TYPE> __a_lock;
+
+  _LIBCPP_INLINE_VISIBILITY void __lock() const volatile {
+    while(1 == __cxx_atomic_exchange(&__a_lock, _LIBCPP_ATOMIC_FLAG_TYPE(true), memory_order_acquire))
+        /*spin*/;
+  }
+  _LIBCPP_INLINE_VISIBILITY void __lock() const {
+    while(1 == __cxx_atomic_exchange(&__a_lock, _LIBCPP_ATOMIC_FLAG_TYPE(true), memory_order_acquire))
+        /*spin*/;
+  }
+  _LIBCPP_INLINE_VISIBILITY void __unlock() const volatile {
+    __cxx_atomic_store(&__a_lock, _LIBCPP_ATOMIC_FLAG_TYPE(false), memory_order_release);
+  }
+  _LIBCPP_INLINE_VISIBILITY void __unlock() const {
+    __cxx_atomic_store(&__a_lock, _LIBCPP_ATOMIC_FLAG_TYPE(false), memory_order_release);
+  }
+  _LIBCPP_INLINE_VISIBILITY _Tp __read() const volatile {
+    __lock();
+    _Tp __old;
+    __cxx_atomic_assign_volatile(__old, __a_value);
+    __unlock();
+    return __old;
+  }
+  _LIBCPP_INLINE_VISIBILITY _Tp __read() const {
+    __lock();
+    _Tp __old = __a_value;
+    __unlock();
+    return __old;
+  }
+};
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(volatile __cxx_atomic_lock_impl<_Tp>* __a,  _Tp __val) {
+  __cxx_atomic_assign_volatile(__a->__a_value, __val);
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_init(__cxx_atomic_lock_impl<_Tp>* __a,  _Tp __val) {
+  __a->__a_value = __val;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(volatile __cxx_atomic_lock_impl<_Tp>* __a,  _Tp __val, memory_order) {
+  __a->__lock();
+  __cxx_atomic_assign_volatile(__a->__a_value, __val);
+  __a->__unlock();
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+void __cxx_atomic_store(__cxx_atomic_lock_impl<_Tp>* __a,  _Tp __val, memory_order) {
+  __a->__lock();
+  __a->__a_value = __val;
+  __a->__unlock();
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(const volatile __cxx_atomic_lock_impl<_Tp>* __a, memory_order) {
+  return __a->__read();
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_load(const __cxx_atomic_lock_impl<_Tp>* __a, memory_order) {
+  return __a->__read();
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(volatile __cxx_atomic_lock_impl<_Tp>* __a, _Tp __value, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, __value);
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_exchange(__cxx_atomic_lock_impl<_Tp>* __a, _Tp __value, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value = __value;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                                          _Tp* __expected, _Tp __value, memory_order, memory_order) {
+  __a->__lock();
+  _Tp temp;
+  __cxx_atomic_assign_volatile(temp, __a->__a_value);
+  bool __ret = temp == *__expected;
+  if(__ret)
+    __cxx_atomic_assign_volatile(__a->__a_value, __value);
+  else
+    __cxx_atomic_assign_volatile(*__expected, __a->__a_value);
+  __a->__unlock();
+  return __ret;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_strong(__cxx_atomic_lock_impl<_Tp>* __a,
+                                          _Tp* __expected, _Tp __value, memory_order, memory_order) {
+  __a->__lock();
+  bool __ret = __a->__a_value == *__expected;
+  if(__ret)
+    __a->__a_value = __value;
+  else
+    *__expected = __a->__a_value;
+  __a->__unlock();
+  return __ret;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                                        _Tp* __expected, _Tp __value, memory_order, memory_order) {
+  __a->__lock();
+  _Tp temp;
+  __cxx_atomic_assign_volatile(temp, __a->__a_value);
+  bool __ret = temp == *__expected;
+  if(__ret)
+    __cxx_atomic_assign_volatile(__a->__a_value, __value);
+  else
+    __cxx_atomic_assign_volatile(*__expected, __a->__a_value);
+  __a->__unlock();
+  return __ret;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+bool __cxx_atomic_compare_exchange_weak(__cxx_atomic_lock_impl<_Tp>* __a,
+                                        _Tp* __expected, _Tp __value, memory_order, memory_order) {
+  __a->__lock();
+  bool __ret = __a->__a_value == *__expected;
+  if(__ret)
+    __a->__a_value = __value;
+  else
+    *__expected = __a->__a_value;
+  __a->__unlock();
+  return __ret;
+}
+
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                           _Td __delta, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, _Tp(__old + __delta));
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_add(__cxx_atomic_lock_impl<_Tp>* __a,
+                           _Td __delta, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value += __delta;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_add(volatile __cxx_atomic_lock_impl<_Tp*>* __a,
+                           ptrdiff_t __delta, memory_order) {
+  __a->__lock();
+  _Tp* __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, __old + __delta);
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp* __cxx_atomic_fetch_add(__cxx_atomic_lock_impl<_Tp*>* __a,
+                           ptrdiff_t __delta, memory_order) {
+  __a->__lock();
+  _Tp* __old = __a->__a_value;
+  __a->__a_value += __delta;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                           _Td __delta, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, _Tp(__old - __delta));
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp, typename _Td>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_sub(__cxx_atomic_lock_impl<_Tp>* __a,
+                           _Td __delta, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value -= __delta;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                           _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, _Tp(__old & __pattern));
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_and(__cxx_atomic_lock_impl<_Tp>* __a,
+                           _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value &= __pattern;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                          _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, _Tp(__old | __pattern));
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_or(__cxx_atomic_lock_impl<_Tp>* __a,
+                          _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value |= __pattern;
+  __a->__unlock();
+  return __old;
+}
+
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(volatile __cxx_atomic_lock_impl<_Tp>* __a,
+                           _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old;
+  __cxx_atomic_assign_volatile(__old, __a->__a_value);
+  __cxx_atomic_assign_volatile(__a->__a_value, _Tp(__old ^ __pattern));
+  __a->__unlock();
+  return __old;
+}
+template <typename _Tp>
+_LIBCPP_INLINE_VISIBILITY
+_Tp __cxx_atomic_fetch_xor(__cxx_atomic_lock_impl<_Tp>* __a,
+                           _Tp __pattern, memory_order) {
+  __a->__lock();
+  _Tp __old = __a->__a_value;
+  __a->__a_value ^= __pattern;
+  __a->__unlock();
+  return __old;
+}
+
+#ifdef __cpp_lib_atomic_is_always_lock_free
+
+template<typename _Tp> struct __cxx_is_always_lock_free {
+    enum { __value = __atomic_always_lock_free(sizeof(_Tp), 0) }; };
+
+#else
+
+template<typename _Tp> struct __cxx_is_always_lock_free { enum { __value = false }; };
+// Implementations must match the C ATOMIC_*_LOCK_FREE macro values.
+template<> struct __cxx_is_always_lock_free<bool> { enum { __value = 2 == ATOMIC_BOOL_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<char> { enum { __value = 2 == ATOMIC_CHAR_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<signed char> { enum { __value = 2 == ATOMIC_CHAR_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<unsigned char> { enum { __value = 2 == ATOMIC_CHAR_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<char16_t> { enum { __value = 2 == ATOMIC_CHAR16_T_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<char32_t> { enum { __value = 2 == ATOMIC_CHAR32_T_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<wchar_t> { enum { __value = 2 == ATOMIC_WCHAR_T_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<short> { enum { __value = 2 == ATOMIC_SHORT_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<unsigned short> { enum { __value = 2 == ATOMIC_SHORT_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<int> { enum { __value = 2 == ATOMIC_INT_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<unsigned int> { enum { __value = 2 == ATOMIC_INT_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<long> { enum { __value = 2 == ATOMIC_LONG_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<unsigned long> { enum { __value = 2 == ATOMIC_LONG_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<long long> { enum { __value = 2 == ATOMIC_LLONG_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<unsigned long long> { enum { __value = 2 == ATOMIC_LLONG_LOCK_FREE }; };
+template<typename _Tp> struct __cxx_is_always_lock_free<_Tp*> { enum { __value = 2 == ATOMIC_POINTER_LOCK_FREE }; };
+template<> struct __cxx_is_always_lock_free<std::nullptr_t> { enum { __value = 2 == ATOMIC_POINTER_LOCK_FREE }; };
+
+#endif //__cpp_lib_atomic_is_always_lock_free
+
+template <typename _Tp,
+          typename _Base = typename conditional<__cxx_is_always_lock_free<_Tp>::__value,
+                                                __cxx_atomic_base_impl<_Tp>,
+                                                __cxx_atomic_lock_impl<_Tp> >::type>
+#else
+template <typename _Tp,
+          typename _Base = __cxx_atomic_base_impl<_Tp> >
+#endif //_LIBCPP_ATOMIC_ONLY_USE_BUILTINS
+struct __cxx_atomic_impl : public _Base {
+
+#if _GNUC_VER >= 501
+    static_assert(is_trivially_copyable<_Tp>::value,
+      "std::atomic<Tp> requires that 'Tp' be a trivially copyable type");
+#endif
+
+  _LIBCPP_INLINE_VISIBILITY __cxx_atomic_impl() _NOEXCEPT _LIBCPP_DEFAULT
+  _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR explicit __cxx_atomic_impl(_Tp value) _NOEXCEPT
+    : _Base(value) {}
+};
+
 // general atomic<T>
 
 template <class _Tp, bool = is_integral<_Tp>::value && !is_same<_Tp, bool>::value>
 struct __atomic_base  // false
 {
-    mutable _Atomic(_Tp) __a_;
+    mutable __cxx_atomic_impl<_Tp> __a_;
 
 #if defined(__cpp_lib_atomic_is_always_lock_free)
   static _LIBCPP_CONSTEXPR bool is_always_lock_free = __atomic_always_lock_free(sizeof(__a_), 0);
 #endif
 
     _LIBCPP_INLINE_VISIBILITY
     bool is_lock_free() const volatile _NOEXCEPT
-    {
-#if defined(_LIBCPP_HAS_C_ATOMIC_IMP)
-    return __c11_atomic_is_lock_free(sizeof(_Tp));
-#else
-    return __atomic_is_lock_free(sizeof(_Tp), 0);
-#endif
-    }
+        {return __cxx_atomic_is_lock_free(sizeof(_Tp));}
     _LIBCPP_INLINE_VISIBILITY
     bool is_lock_free() const _NOEXCEPT
         {return static_cast<__atomic_base const volatile*>(this)->is_lock_free();}
     _LIBCPP_INLINE_VISIBILITY
     void store(_Tp __d, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
       _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m)
-        {__c11_atomic_store(&__a_, __d, __m);}
+        {__cxx_atomic_store(&__a_, __d, __m);}
     _LIBCPP_INLINE_VISIBILITY
     void store(_Tp __d, memory_order __m = memory_order_seq_cst) _NOEXCEPT
       _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m)
-        {__c11_atomic_store(&__a_, __d, __m);}
+        {__cxx_atomic_store(&__a_, __d, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp load(memory_order __m = memory_order_seq_cst) const volatile _NOEXCEPT
       _LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
-        {return __c11_atomic_load(&__a_, __m);}
+        {return __cxx_atomic_load(&__a_, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp load(memory_order __m = memory_order_seq_cst) const _NOEXCEPT
       _LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
-        {return __c11_atomic_load(&__a_, __m);}
+        {return __cxx_atomic_load(&__a_, __m);}
     _LIBCPP_INLINE_VISIBILITY
     operator _Tp() const volatile _NOEXCEPT {return load();}
     _LIBCPP_INLINE_VISIBILITY
     operator _Tp() const _NOEXCEPT          {return load();}
     _LIBCPP_INLINE_VISIBILITY
     _Tp exchange(_Tp __d, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_exchange(&__a_, __d, __m);}
+        {return __cxx_atomic_exchange(&__a_, __d, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp exchange(_Tp __d, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_exchange(&__a_, __d, __m);}
+        {return __cxx_atomic_exchange(&__a_, __d, __m);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_weak(_Tp& __e, _Tp __d,
                                memory_order __s, memory_order __f) volatile _NOEXCEPT
       _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
-        {return __c11_atomic_compare_exchange_weak(&__a_, &__e, __d, __s, __f);}
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __s, __f);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_weak(_Tp& __e, _Tp __d,
                                memory_order __s, memory_order __f) _NOEXCEPT
       _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
-        {return __c11_atomic_compare_exchange_weak(&__a_, &__e, __d, __s, __f);}
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __s, __f);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_strong(_Tp& __e, _Tp __d,
                                  memory_order __s, memory_order __f) volatile _NOEXCEPT
       _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
-        {return __c11_atomic_compare_exchange_strong(&__a_, &__e, __d, __s, __f);}
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __s, __f);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_strong(_Tp& __e, _Tp __d,
                                  memory_order __s, memory_order __f) _NOEXCEPT
       _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
-        {return __c11_atomic_compare_exchange_strong(&__a_, &__e, __d, __s, __f);}
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __s, __f);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_weak(_Tp& __e, _Tp __d,
                               memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_compare_exchange_weak(&__a_, &__e, __d, __m, __m);}
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __m, __m);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_weak(_Tp& __e, _Tp __d,
                                memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_compare_exchange_weak(&__a_, &__e, __d, __m, __m);}
+        {return __cxx_atomic_compare_exchange_weak(&__a_, &__e, __d, __m, __m);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_strong(_Tp& __e, _Tp __d,
                               memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_compare_exchange_strong(&__a_, &__e, __d, __m, __m);}
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __m, __m);}
     _LIBCPP_INLINE_VISIBILITY
     bool compare_exchange_strong(_Tp& __e, _Tp __d,
                                  memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_compare_exchange_strong(&__a_, &__e, __d, __m, __m);}
+        {return __cxx_atomic_compare_exchange_strong(&__a_, &__e, __d, __m, __m);}
 
     _LIBCPP_INLINE_VISIBILITY
-#ifndef _LIBCPP_CXX03_LANG
-    __atomic_base() _NOEXCEPT = default;
-#else
-    __atomic_base() _NOEXCEPT : __a_() {}
-#endif // _LIBCPP_CXX03_LANG
+    __atomic_base() _NOEXCEPT _LIBCPP_DEFAULT
+
+    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR
+    __atomic_base(_Tp __d) _NOEXCEPT : __a_(__d) {}
 
-    _LIBCPP_INLINE_VISIBILITY
-    _LIBCPP_CONSTEXPR __atomic_base(_Tp __d) _NOEXCEPT : __a_(__d) {}
 #ifndef _LIBCPP_CXX03_LANG
     __atomic_base(const __atomic_base&) = delete;
     __atomic_base& operator=(const __atomic_base&) = delete;
     __atomic_base& operator=(const __atomic_base&) volatile = delete;
 #else
 private:
     __atomic_base(const __atomic_base&);
     __atomic_base& operator=(const __atomic_base&);
struct __atomic_base<_Tp, true>
     typedef __atomic_base<_Tp, false> __base;
     _LIBCPP_INLINE_VISIBILITY
     __atomic_base() _NOEXCEPT _LIBCPP_DEFAULT
     _LIBCPP_INLINE_VISIBILITY
     _LIBCPP_CONSTEXPR __atomic_base(_Tp __d) _NOEXCEPT : __base(__d) {}
 
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_add(_Tp __op, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_fetch_add(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_add(_Tp __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_add(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_sub(_Tp __op, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_fetch_sub(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_sub(_Tp __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_sub(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_and(_Tp __op, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_fetch_and(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_and(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_and(_Tp __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_and(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_and(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_or(_Tp __op, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_fetch_or(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_or(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_or(_Tp __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_or(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_or(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_xor(_Tp __op, memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_fetch_xor(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_xor(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp fetch_xor(_Tp __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_xor(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_xor(&this->__a_, __op, __m);}
 
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator++(int) volatile _NOEXCEPT      {return fetch_add(_Tp(1));}
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator++(int) _NOEXCEPT               {return fetch_add(_Tp(1));}
     _LIBCPP_INLINE_VISIBILITY
     _Tp operator--(int) volatile _NOEXCEPT      {return fetch_sub(_Tp(1));}
     _LIBCPP_INLINE_VISIBILITY
_Tp* operator=(_Tp* __d) volatile _NOEXCEPT
         {__base::store(__d); return __d;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator=(_Tp* __d) _NOEXCEPT
         {__base::store(__d); return __d;}
 
     _LIBCPP_INLINE_VISIBILITY
     _Tp* fetch_add(ptrdiff_t __op, memory_order __m = memory_order_seq_cst)
                                                                         volatile _NOEXCEPT
-        {return __c11_atomic_fetch_add(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* fetch_add(ptrdiff_t __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_add(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_add(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* fetch_sub(ptrdiff_t __op, memory_order __m = memory_order_seq_cst)
                                                                         volatile _NOEXCEPT
-        {return __c11_atomic_fetch_sub(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* fetch_sub(ptrdiff_t __op, memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_fetch_sub(&this->__a_, __op, __m);}
+        {return __cxx_atomic_fetch_sub(&this->__a_, __op, __m);}
 
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator++(int) volatile _NOEXCEPT            {return fetch_add(1);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator++(int) _NOEXCEPT                     {return fetch_add(1);}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator--(int) volatile _NOEXCEPT            {return fetch_sub(1);}
     _LIBCPP_INLINE_VISIBILITY
struct atomic<_Tp*>
     _Tp* operator-=(ptrdiff_t __op) volatile _NOEXCEPT {return fetch_sub(__op) - __op;}
     _LIBCPP_INLINE_VISIBILITY
     _Tp* operator-=(ptrdiff_t __op) _NOEXCEPT          {return fetch_sub(__op) - __op;}
 };
 
 // atomic_is_lock_free
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT
 {
     return __o->is_lock_free();
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT
 {
     return __o->is_lock_free();
 }
 
 // atomic_init
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_init(volatile atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
-    __c11_atomic_init(&__o->__a_, __d);
+    __cxx_atomic_init(&__o->__a_, __d);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_init(atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
-    __c11_atomic_init(&__o->__a_, __d);
+    __cxx_atomic_init(&__o->__a_, __d);
 }
 
 // atomic_store
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_store(volatile atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
     __o->store(__d);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_store(atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
     __o->store(__d);
 }
 
 // atomic_store_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_store_explicit(volatile atomic<_Tp>* __o, _Tp __d, memory_order __m) _NOEXCEPT
   _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m)
 {
     __o->store(__d, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 void
 atomic_store_explicit(atomic<_Tp>* __o, _Tp __d, memory_order __m) _NOEXCEPT
   _LIBCPP_CHECK_STORE_MEMORY_ORDER(__m)
 {
     __o->store(__d, __m);
 }
 
 // atomic_load
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_load(const volatile atomic<_Tp>* __o) _NOEXCEPT
 {
     return __o->load();
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_load(const atomic<_Tp>* __o) _NOEXCEPT
 {
     return __o->load();
 }
 
 // atomic_load_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_load_explicit(const volatile atomic<_Tp>* __o, memory_order __m) _NOEXCEPT
   _LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
 {
     return __o->load(__m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_load_explicit(const atomic<_Tp>* __o, memory_order __m) _NOEXCEPT
   _LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
 {
     return __o->load(__m);
 }
 
 // atomic_exchange
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_exchange(volatile atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
     return __o->exchange(__d);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_exchange(atomic<_Tp>* __o, _Tp __d) _NOEXCEPT
 {
     return __o->exchange(__d);
 }
 
 // atomic_exchange_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_exchange_explicit(volatile atomic<_Tp>* __o, _Tp __d, memory_order __m) _NOEXCEPT
 {
     return __o->exchange(__d, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp
 atomic_exchange_explicit(atomic<_Tp>* __o, _Tp __d, memory_order __m) _NOEXCEPT
 {
     return __o->exchange(__d, __m);
 }
 
 // atomic_compare_exchange_weak
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_weak(volatile atomic<_Tp>* __o, _Tp* __e, _Tp __d) _NOEXCEPT
 {
     return __o->compare_exchange_weak(*__e, __d);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_weak(atomic<_Tp>* __o, _Tp* __e, _Tp __d) _NOEXCEPT
 {
     return __o->compare_exchange_weak(*__e, __d);
 }
 
 // atomic_compare_exchange_strong
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_strong(volatile atomic<_Tp>* __o, _Tp* __e, _Tp __d) _NOEXCEPT
 {
     return __o->compare_exchange_strong(*__e, __d);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_strong(atomic<_Tp>* __o, _Tp* __e, _Tp __d) _NOEXCEPT
 {
     return __o->compare_exchange_strong(*__e, __d);
 }
 
 // atomic_compare_exchange_weak_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_weak_explicit(volatile atomic<_Tp>* __o, _Tp* __e,
                                       _Tp __d,
                                       memory_order __s, memory_order __f) _NOEXCEPT
   _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
 {
     return __o->compare_exchange_weak(*__e, __d, __s, __f);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_weak_explicit(atomic<_Tp>* __o, _Tp* __e, _Tp __d,
                                       memory_order __s, memory_order __f) _NOEXCEPT
   _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
 {
     return __o->compare_exchange_weak(*__e, __d, __s, __f);
 }
 
 // atomic_compare_exchange_strong_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_strong_explicit(volatile atomic<_Tp>* __o,
                                         _Tp* __e, _Tp __d,
                                         memory_order __s, memory_order __f) _NOEXCEPT
   _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
 {
     return __o->compare_exchange_strong(*__e, __d, __s, __f);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 bool
 atomic_compare_exchange_strong_explicit(atomic<_Tp>* __o, _Tp* __e,
                                         _Tp __d,
                                         memory_order __s, memory_order __f) _NOEXCEPT
   _LIBCPP_CHECK_EXCHANGE_MEMORY_ORDER(__s, __f)
 {
     return __o->compare_exchange_strong(*__e, __d, __s, __f);
 }
 
 // atomic_fetch_add
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_add(volatile atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_add(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_add(atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_add(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_add(volatile atomic<_Tp*>* __o, ptrdiff_t __op) _NOEXCEPT
 {
     return __o->fetch_add(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_add(atomic<_Tp*>* __o, ptrdiff_t __op) _NOEXCEPT
 {
     return __o->fetch_add(__op);
 }
 
 // atomic_fetch_add_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_add_explicit(volatile atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_add(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_add_explicit(atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_add(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_add_explicit(volatile atomic<_Tp*>* __o, ptrdiff_t __op,
                           memory_order __m) _NOEXCEPT
 {
     return __o->fetch_add(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_add_explicit(atomic<_Tp*>* __o, ptrdiff_t __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_add(__op, __m);
 }
 
 // atomic_fetch_sub
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_sub(volatile atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_sub(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_sub(atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_sub(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_sub(volatile atomic<_Tp*>* __o, ptrdiff_t __op) _NOEXCEPT
 {
     return __o->fetch_sub(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_sub(atomic<_Tp*>* __o, ptrdiff_t __op) _NOEXCEPT
 {
     return __o->fetch_sub(__op);
 }
 
 // atomic_fetch_sub_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_sub_explicit(volatile atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_sub(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_sub_explicit(atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_sub(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_sub_explicit(volatile atomic<_Tp*>* __o, ptrdiff_t __op,
                           memory_order __m) _NOEXCEPT
 {
     return __o->fetch_sub(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 _Tp*
 atomic_fetch_sub_explicit(atomic<_Tp*>* __o, ptrdiff_t __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_sub(__op, __m);
 }
 
 // atomic_fetch_and
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_and(volatile atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_and(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_and(atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_and(__op);
 }
 
 // atomic_fetch_and_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_and_explicit(volatile atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_and(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_and_explicit(atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_and(__op, __m);
 }
 
 // atomic_fetch_or
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_or(volatile atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_or(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_or(atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_or(__op);
 }
 
 // atomic_fetch_or_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_or_explicit(volatile atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_or(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_or_explicit(atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_or(__op, __m);
 }
 
 // atomic_fetch_xor
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_xor(volatile atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_xor(__op);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_xor(atomic<_Tp>* __o, _Tp __op) _NOEXCEPT
 {
     return __o->fetch_xor(__op);
 }
 
 // atomic_fetch_xor_explicit
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_xor_explicit(volatile atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_xor(__op, __m);
 }
 
 template <class _Tp>
-inline _LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY
 typename enable_if
 <
     is_integral<_Tp>::value && !is_same<_Tp, bool>::value,
     _Tp
 >::type
 atomic_fetch_xor_explicit(atomic<_Tp>* __o, _Tp __op, memory_order __m) _NOEXCEPT
 {
     return __o->fetch_xor(__op, __m);
 }
 
 // flag type and operations
 
 typedef struct atomic_flag
 {
-    _Atomic(bool) __a_;
+    __cxx_atomic_impl<_LIBCPP_ATOMIC_FLAG_TYPE> __a_;
 
     _LIBCPP_INLINE_VISIBILITY
     bool test_and_set(memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {return __c11_atomic_exchange(&__a_, true, __m);}
+        {return __cxx_atomic_exchange(&__a_, _LIBCPP_ATOMIC_FLAG_TYPE(true), __m);}
     _LIBCPP_INLINE_VISIBILITY
     bool test_and_set(memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {return __c11_atomic_exchange(&__a_, true, __m);}
+        {return __cxx_atomic_exchange(&__a_, _LIBCPP_ATOMIC_FLAG_TYPE(true), __m);}
     _LIBCPP_INLINE_VISIBILITY
     void clear(memory_order __m = memory_order_seq_cst) volatile _NOEXCEPT
-        {__c11_atomic_store(&__a_, false, __m);}
+        {__cxx_atomic_store(&__a_, _LIBCPP_ATOMIC_FLAG_TYPE(false), __m);}
     _LIBCPP_INLINE_VISIBILITY
     void clear(memory_order __m = memory_order_seq_cst) _NOEXCEPT
-        {__c11_atomic_store(&__a_, false, __m);}
+        {__cxx_atomic_store(&__a_, _LIBCPP_ATOMIC_FLAG_TYPE(false), __m);}
 
     _LIBCPP_INLINE_VISIBILITY
-#ifndef _LIBCPP_CXX03_LANG
-    atomic_flag() _NOEXCEPT = default;
-#else
-    atomic_flag() _NOEXCEPT : __a_() {}
-#endif // _LIBCPP_CXX03_LANG
+    atomic_flag() _NOEXCEPT _LIBCPP_DEFAULT
 
     _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR
     atomic_flag(bool __b) _NOEXCEPT : __a_(__b) {} // EXTENSION
 
 #ifndef _LIBCPP_CXX03_LANG
     atomic_flag(const atomic_flag&) = delete;
     atomic_flag& operator=(const atomic_flag&) = delete;
     atomic_flag& operator=(const atomic_flag&) volatile = delete;

 }
 
 // fences
 
 inline _LIBCPP_INLINE_VISIBILITY
 void
 atomic_thread_fence(memory_order __m) _NOEXCEPT
 {
-    __c11_atomic_thread_fence(__m);
+    __cxx_atomic_thread_fence(__m);
 }
 
 inline _LIBCPP_INLINE_VISIBILITY
 void
 atomic_signal_fence(memory_order __m) _NOEXCEPT
 {
-    __c11_atomic_signal_fence(__m);
+    __cxx_atomic_signal_fence(__m);
 }
 
 // Atomics for standard typedef types
 
 typedef atomic<bool>               atomic_bool;
 typedef atomic<char>               atomic_char;
 typedef atomic<signed char>        atomic_schar;
 typedef atomic<unsigned char>      atomic_uchar;