[libc++][ABI BREAK] Do not use the C++03 emulation for std::nullptr_t by default

[ldionne]

Summary

We only support Clangs that implement nullptr as an extension in C++03 mode,
and we don't support GCC in C++03 mode. Hence, this patch disables the
use of the std::nullptr_t emulation in C++03 mode by default. Doing that
is technically an ABI break since it changes the mangling for std::nullptr_t.
However:

(1) The only affected users are those compiling in C++03 mode that have

std::nullptr_t as part of their ABI, which should be reasonably rare.

(2) Those users already have a lingering problem in that their code will

be incompatible in C++03 and C++11 modes because of that very ABI break.
Hence, the only users that could really be inconvenienced about this
change is those that planned on compiling in C++03 mode forever - for
other users, we're just breaking them now instead of letting them break
themselves later on when they try to upgrade to C++11.

(3) The ABI break will cause a linker error since the mangling changed,

and will not result in an obscure runtime error.

Furthermore, if anyone is broken by this, they can define the
_LIBCPP_ABI_USE_CXX03_NULLPTR_EMULATION macro to return to the
previous behavior. We will then remove that macro after shipping
this for one release if we haven't seen widespread issues.

Concretely, the motivation for making this change is to make our own ABI
consistent in C++03 and C++11 modes and to remove complexity around the
definition of nullptr.

Furthermore, we could investigate making nullptr a keyword in C++03 mode
as a Clang extension -- I don't think that would break anyone, since
libc++ already defines nullptr as a macro to something else. Only users
that do not use libc++ and compile in C++03 mode could potentially be
broken by that.

[ldionne]

I'm putting this change up both for discussion and also to give people the chance to run test builds on internal code bases using this change. I think this change hinges quite a bit on the assumption that people almost never rely on std::nullptr_t in their ABI in C++03 mode. If that assumption turns out to be false, we may need to revisit the feasibility of this change.

Pinging some vendors to see what they think about this. @danalbert @phosek @dim @emaste

Perhaps @cjdb could run a build at Google? I'll do it at Apple.

[• Quuxplusone]

FWLIW, I'm generally in favor of this. Although I think it's ironic that we already had an ABI flag for this, such that we'd have gotten this change for free if we ever managed to reach ABIv2. Feels like this PR is the most public admission yet that we'll never actually be able to reach std::__2.

libcxx/include/cstddef

46

Surely this should just unconditionally say typedef decltype(nullptr) nullptr_t; — why would we ever not want std::nullptr_t to exist?
(Vice versa, personally I don't understand why we define a type named ::nullptr_t; but I assume there's some good reason not to remove it now.)

[ldionne]

In D109459#2990186, @Quuxplusone wrote:

FWLIW, I'm generally in favor of this. Although I think it's ironic that we already had an ABI flag for this, such that we'd have gotten this change for free if we ever managed to reach ABIv2. Feels like this PR is the most public admission yet that we'll never actually be able to reach std::__2.

I don't think that's accurate. IMO, there are things we need to guard behind an ABI flag because they would actually break people and cause problems if we did not. Then there are things (like this nullptr_t emulation) that wouldn't break anyone anyways if we turned it on, so we might as well do it in the ABI v1 (proof is needed regarding whether I'm right about that, and I'm running builds to check that out right now).

IMO, the idea of an ABI v2 is a bit funky, I think we need to define it better. I think something more useful would be to formalize how different platforms get to select different ABI flags, since "breaking" the ABI is easiest done when one introduces a new platform. That doesn't happen super often, but it's better than nothing. Anyway, this is a discussion for a different forum.

libcxx/include/cstddef

46

Sure, I can drop _LIBCPP_USING_IF_EXISTS here since it's always defined (by us).

Regarding ::nullptr_t, I was surprised to read that <stddef.h> is supposed to provide such a type 🤷🏼‍♂️.

[ldionne]

Try to fix build on GCC and address Arthur's comment about using_if_exists

[ldionne]

Fix the modules build.

[danalbert]

I think this is probably fine from the NDK perspective. Even before Clang defaulted to C++11 or newer, our build system did that when using libc++. Other build systems may not have, but unfortunately I'm not currently able to get at the data to get a concrete answer.

[ldionne]

I did an internal Apple build and didn't see any issues. So this is OK from Apple's internal code base perspective.

However, it doesn't mean that this will be okay for users of libc++, whose ABIs could conceivably leak the mangling of std::nullptr_t. I'd still like to get input from other vendors before making this move. Accepting as myself only to record that this is OK for Apple.

[ldionne]

@dexonsmith has had useful views on ABI changes in the past, I'm curious to have his opinion on this change, since I'm still ambivalent.

TL;DR: Today, std::nullptr_t is a different type in C++03 and C++11, which means that the ABI breaks between C++03 and C++11. This patch breaks the ABI in C++03 mode once, right now, to reconcile the C++03 and C++11 ABIs and remove some convolution from libc++. My hunch is that this will not affect too many people and hence it's reasonable. Remember that nullptr is a C++11 construct, so it would have to be someone compiling their code in C++03 mode *and* using our nullptr extension, *and* leaking nullptr_t in their ABI. Do you have an opinion?

[ldionne]

Rebase onto main

[ldionne]

Rebase onto main. I'm still on the fence about this.

[dexonsmith]

In D109459#3043589, @ldionne wrote:

@dexonsmith has had useful views on ABI changes in the past, I'm curious to have his opinion on this change, since I'm still ambivalent.

I missed this before!

TL;DR: Today, std::nullptr_t is a different type in C++03 and C++11, which means that the ABI breaks between C++03 and C++11. This patch breaks the ABI in C++03 mode once, right now, to reconcile the C++03 and C++11 ABIs and remove some convolution from libc++. My hunch is that this will not affect too many people and hence it's reasonable. Remember that nullptr is a C++11 construct, so it would have to be someone compiling their code in C++03 mode *and* using our nullptr extension, *and* leaking nullptr_t in their ABI. Do you have an opinion?

This feels relatively safe to me for the reasons given, but hard to be sure. One option would be to change behaviour behind a vendor+user macro combo for a couple of releases to test (vendors and/or users could revert to behaviour as-is today if/when necessary), then drop the configurations once there’s more confidence. I’m not sure it’s worth the trouble though.

@rjmccall, any thoughts from you?

Re clang/libc++ behaviour, if you wanted to continue providing the extension but take an ABI break, I think you could have clang expose a __nullptr_type__ in C++03 that it mangled like the C++11 nullptr_t and then add a typedef for it in libc++ to expose it.

[ldionne]

In D109459#3159309, @dexonsmith wrote:

In D109459#3043589, @ldionne wrote:

@dexonsmith has had useful views on ABI changes in the past, I'm curious to have his opinion on this change, since I'm still ambivalent.

I missed this before!

TL;DR: Today, std::nullptr_t is a different type in C++03 and C++11, which means that the ABI breaks between C++03 and C++11. This patch breaks the ABI in C++03 mode once, right now, to reconcile the C++03 and C++11 ABIs and remove some convolution from libc++. My hunch is that this will not affect too many people and hence it's reasonable. Remember that nullptr is a C++11 construct, so it would have to be someone compiling their code in C++03 mode *and* using our nullptr extension, *and* leaking nullptr_t in their ABI. Do you have an opinion?

This feels relatively safe to me for the reasons given, but hard to be sure. One option would be to change behaviour behind a vendor+user macro combo for a couple of releases to test (vendors and/or users could revert to behaviour as-is today if/when necessary), then drop the configurations once there’s more confidence. I’m not sure it’s worth the trouble though.

Thanks for chiming in. I like this a lot. I will repurpose this patch to enable the ABI break by default while still letting users return to the previous behavior by defining a macro. After shipping that for one release, we can rip it out entirely if there aren't widespread issues reported.

[ldionne]

Turn on ABI break by default, but let users turn it off.

[• Quuxplusone]

FWIW, I like this idea.

libcxx/docs/ReleaseNotes.rst

102–111

s/favour/favor/
s/Standard modes/standard modes/ ?
s/these user's/these users'/
s/linking/linker/ ?

[ldionne]

Address wording issues in the release note. Thanks Arthur!

I'm going to ship this now since I think this is pretty uncontroversial.

[ldionne]

Thanks a lot @dexonsmith, you had the magic suggestion that is simple, elegant and lets us move forward.

libcxx/docs/ReleaseNotes.rst

- The declarations of functions ``declare_reachable``, ``undeclare_reachable``, ``declare_no_pointers``,
   existing compiled code. All of these functions have always behaved as no-ops.
 
 ABI Changes
 -----------
 
 - The C++17 variable templates ``is_error_code_enum_v`` and
   ``is_error_condition_enum_v`` are now of type ``bool`` instead of ``size_t``.
 
+- The C++03 emulation type for ``std::nullptr_t`` has been removed in favour of
+  using ``decltype(nullptr)`` in all Standard modes. This is an ABI break for
+  anyone compiling in C++03 mode and who has ``std::nullptr_t`` as part of their
+  ABI. However, previously, these user's ABI would be incompatible with any other
+  binary or static archive compiled with C++11 or later.
+
 Build System Changes
 --------------------
 
 - Building the libc++ shared or static library requires a C++ 20 capable compiler.
   Consider using a Bootstrapping build to build libc++ with a fresh Clang if you
   can't use the system compiler to build libc++ anymore.
 
 - Historically, there has been numerous ways of building libc++ and libc++abi. This has
   culminated in over 5 different ways to build the runtimes, which made it impossible to
   maintain with a good level of support. Starting with this release, the runtimes support
   exactly two ways of being built, which should cater to all use-cases. Furthermore,
   these builds are as lightweight as possible and will work consistently even when targetting

libcxx/include/CMakeLists.txt

set(files
   __memory/raw_storage_iterator.h
   __memory/shared_ptr.h
   __memory/temporary_buffer.h
   __memory/uninitialized_algorithms.h
   __memory/unique_ptr.h
   __memory/uses_allocator.h
   __mutex_base
   __node_handle
-  __nullptr
   __random/bernoulli_distribution.h
   __random/binomial_distribution.h
   __random/cauchy_distribution.h
   __random/chi_squared_distribution.h
   __random/default_random_engine.h
   __random/discard_block_engine.h
   __random/discrete_distribution.h
   __random/exponential_distribution.h

libcxx/include/__config

 // Fix undefined behavior in how std::list stores its linked nodes.
 #  define _LIBCPP_ABI_LIST_REMOVE_NODE_POINTER_UB
 // Fix undefined behavior in  how __tree stores its end and parent nodes.
 #  define _LIBCPP_ABI_TREE_REMOVE_NODE_POINTER_UB
 // Fix undefined behavior in how __hash_table stores its pointer types.
 #  define _LIBCPP_ABI_FIX_UNORDERED_NODE_POINTER_UB
 #  define _LIBCPP_ABI_FORWARD_LIST_REMOVE_NODE_POINTER_UB
 #  define _LIBCPP_ABI_FIX_UNORDERED_CONTAINER_SIZE_TYPE
-// Don't use a nullptr_t simulation type in C++03 instead using C++11 nullptr
-// provided under the alternate keyword __nullptr, which changes the mangling
-// of nullptr_t. This option is ABI incompatible with GCC in C++03 mode.
-#  define _LIBCPP_ABI_ALWAYS_USE_CXX11_NULLPTR
 // Define a key function for `bad_function_call` in the library, to centralize
 // its vtable and typeinfo to libc++ rather than having all other libraries
 // using that class define their own copies.
 #  define _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION
 // Override the default return value of exception::what() for
 // bad_function_call::what() with a string that is specific to
 // bad_function_call (see http://wg21.link/LWG2233). This is an ABI break
 // because it changes the vtable layout of bad_function_call.

 #endif
 
 #if __has_feature(cxx_attributes)
 #  define _LIBCPP_NORETURN [[noreturn]]
 #else
 #  define _LIBCPP_NORETURN __attribute__ ((noreturn))
 #endif
 
-#if !(__has_feature(cxx_nullptr))
-#  if (__has_extension(cxx_nullptr) || __has_keyword(__nullptr)) && defined(_LIBCPP_ABI_ALWAYS_USE_CXX11_NULLPTR)
-#    define nullptr __nullptr
-#  else
-#    define _LIBCPP_HAS_NO_NULLPTR
-#  endif
+#ifdef _LIBCPP_CXX03_LANG
+# define nullptr __nullptr
 #endif
 
 // Objective-C++ features (opt-in)
 #if __has_feature(objc_arc)
 #define _LIBCPP_HAS_OBJC_ARC
 #endif
 
 #if __has_feature(objc_arc_weak)

libcxx/include/__debug

 
 #include <__config>
 #include <iosfwd>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
-#if defined(_LIBCPP_HAS_NO_NULLPTR)
-# include <cstddef>
-#endif
-
 #if _LIBCPP_DEBUG_LEVEL >= 1 || defined(_LIBCPP_BUILDING_LIBRARY)
 #   include <cstdlib>
 #   include <cstdio>
 #   include <cstddef>
 #endif
 
 #if _LIBCPP_DEBUG_LEVEL == 0
 #   define _LIBCPP_DEBUG_ASSERT(x, m) ((void)0)

libcxx/include/__memory/auto_ptr.h

 // -*- C++ -*-
 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _LIBCPP___MEMORY_AUTO_PTR_H
 #define _LIBCPP___MEMORY_AUTO_PTR_H
 
 #include <__config>
-#include <__nullptr>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
 template <class _Tp>

libcxx/include/__nullptr

-// -*- C++ -*-
-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef _LIBCPP_NULLPTR
-#define _LIBCPP_NULLPTR
-
-#include <__config>
-
-#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
-#pragma GCC system_header
-#endif
-
-#ifdef _LIBCPP_HAS_NO_NULLPTR
-
-_LIBCPP_BEGIN_NAMESPACE_STD
-
-struct _LIBCPP_TEMPLATE_VIS nullptr_t
-{
-    void* __lx;
-
-    struct __nat {int __for_bool_;};
-
-    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR nullptr_t() : __lx(0) {}
-    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR nullptr_t(int __nat::*) : __lx(0) {}
-
-    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR operator int __nat::*() const {return 0;}
-
-    template <class _Tp>
-        _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR
-        operator _Tp* () const {return 0;}
-
-    template <class _Tp, class _Up>
-        _LIBCPP_INLINE_VISIBILITY
-        operator _Tp _Up::* () const {return 0;}
-
-    friend _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR bool operator==(nullptr_t, nullptr_t) {return true;}
-    friend _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR bool operator!=(nullptr_t, nullptr_t) {return false;}
-};
-
-inline _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR nullptr_t __get_nullptr_t() {return nullptr_t(0);}
-
-#define nullptr _VSTD::__get_nullptr_t()
-
-_LIBCPP_END_NAMESPACE_STD
-
-#else  // _LIBCPP_HAS_NO_NULLPTR
-
-namespace std
-{
-    typedef decltype(nullptr) nullptr_t;
-}
-
-#endif // _LIBCPP_HAS_NO_NULLPTR
-
-#endif // _LIBCPP_NULLPTR

libcxx/include/__support/win32/locale_win32.h

 // -*- C++ -*-
 //===-----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _LIBCPP_SUPPORT_WIN32_LOCALE_WIN32_H
 #define _LIBCPP_SUPPORT_WIN32_LOCALE_WIN32_H
 
 #include <__config>
-#include <__nullptr>
+#include <cstddef>
 #include <locale.h> // _locale_t
 #include <stdio.h>
 
 #define _X_ALL LC_ALL
 #define _X_COLLATE LC_COLLATE
 #define _X_CTYPE LC_CTYPE
 #define _X_MONETARY LC_MONETARY
 #define _X_NUMERIC LC_NUMERIC

libcxx/include/cstddef

Types:
     nullptr_t
     byte // C++17
 
 }  // std
 
 */
 
 #include <__config>
+#include <stddef.h>
 #include <version>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
-// Don't include our own <stddef.h>; we don't want to declare ::nullptr_t.
-#include_next <stddef.h>
-#include <__nullptr>
-
 _LIBCPP_BEGIN_NAMESPACE_STD
 
+using ::nullptr_t;
 using ::ptrdiff_t _LIBCPP_USING_IF_EXISTS;
 using ::size_t _LIBCPP_USING_IF_EXISTS;
 
 #if !defined(_LIBCPP_CXX03_LANG)
 using ::max_align_t _LIBCPP_USING_IF_EXISTS;
 #endif
 
 template <class _Tp> struct __libcpp_is_integral                     { enum { value = 0 }; };

libcxx/include/module.modulemap

module std [system] {
   module __debug             { header "__debug"             export * }
   module __errc              { private header "__errc"              export * }
   module __function_like     { private header "__function_like.h"   export * }
   module __hash_table        { header "__hash_table"        export * }
   module __locale            { private header "__locale"            export * }
   module __mbstate           { private header "__mbstate_t.h"       export * }
   module __mutex_base        { private header "__mutex_base"        export * }
   module __node_handle       { private header "__node_handle"       export * }
-  module __nullptr           { header "__nullptr"           export * }
   module __split_buffer      { private header "__split_buffer"      export * }
   module __std_stream        { private header "__std_stream"        export * }
   module __string            { private header "__string"            export * }
   module __threading_support { header "__threading_support" export * }
   module __tree              { header "__tree"              export * }
   module __tuple             { private header "__tuple"             export * }
   module __undef_macros      { header "__undef_macros"      export * }
 

libcxx/include/stddef.h

 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 #include_next <stddef.h>
 
 #ifdef __cplusplus
-
-extern "C++" {
-#include <__nullptr>
-using std::nullptr_t;
-}
-
+    typedef decltype(nullptr) nullptr_t;
 #endif
 
 #endif // _LIBCPP_STDDEF_H

libcxx/test/std/language.support/support.types/nullptr_t.pass.cpp

static_assert(sizeof(std::nullptr_t) == sizeof(void*),
         test_conversions<std::nullptr_t>();
         test_conversions<void*>();
         test_conversions<A*>();
         test_conversions<void(*)()>();
         test_conversions<void(A::*)()>();
         test_conversions<int A::*>();
     }
     {
-#ifdef _LIBCPP_HAS_NO_NULLPTR
+        // TODO: Enable this assertion when GCC compilers implements http://wg21.link/CWG583.
+#if !defined(TEST_COMPILER_GCC)
         static_assert(!has_less<std::nullptr_t>::value, "");
-        // FIXME: our C++03 nullptr emulation still allows for comparisons
-        // with other pointer types by way of the conversion operator.
-        //static_assert(!has_less<void*>::value, "");
-#else
-        // TODO Enable this assertion when all compilers implement core DR 583.
-        // static_assert(!has_less<std::nullptr_t>::value, "");
 #endif
         test_comparisons<std::nullptr_t>();
         test_comparisons<void*>();
         test_comparisons<A*>();
         test_comparisons<void(*)()>();
     }
     test_nullptr_conversions();
 
   return 0;
 }

libcxx/utils/generate_private_header_tests.py

def relative_path(path):
     return path.as_posix()[len(include_path + '/'):]
 
 
 def is_still_public(path):
     rp = relative_path(path)
     return not rp.startswith('__support') and rp not in [
         "__bsd_locale_defaults.h", "__bsd_locale_fallbacks.h", "__config",
         "__config_site.in", "__debug", "__hash_table", "__functional_base",
-        "__libcpp_version", "__nullptr", "__threading_support", "__tree",
-        "__undef_macros"
+        "__libcpp_version", "__threading_support", "__tree", "__undef_macros"
     ]
 
 
 def main():
     paths = [
         relative_path(p) for p in Path(include_path).rglob('*')
         if relative_path(p).startswith('__') and not p.is_dir()
         and is_still_public(p)