[libc++] cuchar redeclares ::mbstate_t when it's in its own clang module

[iana]

Summary

When cuchar is compiled independently on platforms that don't have <uchar.h>, it doesn't get a declaration of mbstate_t, and so makes an empty declaration for ::mbstate_t. That conflicts with the declarations in __mbstate_t.h and cwchar since both of those headers do get mbstate_t before making ::mbstate_t.

Change __mbstate_t.h to just get the underlying declaration for mbstate_t and not make a declaration for ::mbstate_t. Include __mbstate_t.h in uchar.h and wchar.h when their next headers aren't available so that at least mbstate_t gets defined.

Add __std_mbstate_t.h to declare ::mbstate_t for headers that need that when cuchar or cwchar aren't available (because either _LIBCPP_HAS_NO_WIDE_CHARACTERS or _LIBCPP_CXX03_LANG).

[philnik]

Why do we want to change this? The current version looks just fine.

[iana]

In D148542#4274570, @philnik wrote:

Why do we want to change this? The current version looks just fine.

The redeclaration becomes an error with clang modules when __mbstate.h and cuchar are in different modules.

[Mordante]

LGTM modulo one comment. I leave the final approval to @philnik.

libcxx/include/cuchar

53 ↗

(On Diff #514279)

Can you update this comment why this is needed, this is useful for future readers of the code.

[philnik]

In D148542#4274624, @iana wrote:

In D148542#4274570, @philnik wrote:

Why do we want to change this? The current version looks just fine.

The redeclaration becomes an error with clang modules when __mbstate.h and cuchar are in different modules.

Please add that information to the commit message.

LGTM with comments addressed.

libcxx/include/cuchar

52 ↗

(On Diff #514279)

I don't think this check makes sense. It seems like right now we might have this symbol in C++03 in some headers, but not in cuchar. If we only want this in C++11 we should guard it in __mbstate_t.h.

[iana]

In D148542#4274624, @iana wrote:

In D148542#4274570, @philnik wrote:

Why do we want to change this? The current version looks just fine.

The redeclaration becomes an error with clang modules when __mbstate_t.h and cuchar are in different modules.

I dug into it a little more. When <cuchar> is compiled all by itself on Apple platforms, mbstate_t isn't defined because Apple doesn't have a <uchar.h>, and <stddef.h> doesn't declare mbstate_t either. But when <__mbstate_t.h> (and <cwchar>) are compiled, they both get the OS declaration of mbstate_t and their using declaration gets a type. I think it works out with the current std mega module because declarations leak between the submodules, but when those three headers are in different top level modules, that's when things get mad at <cuchar>.

So it looks like it's fine to leave the using ::mbstate_t _LIBCPP_USING_IF_EXISTS; line in <cuchar>, but we need to include something to get mbstate_t defined. Do we want to unconditionally include __mbstate_t.h? That will have the side effect of often including <wchar.h>, and also declaring ::mbstate_t which maybe we don't want. We could leave that include conditional on _LIBCPP_CXX03_LANG? Or we could instead include <bits/types/mbstate_t.h> or <sys/_types/_mbstate_t.h> to get just the C declaration?

[iana]

Redo the fix to make the libc++ uchar.h try harder to find mbstate_t.

[iana]

The sanitizer and aix test failures don't look related to this change, I get the same ones in an unrelated review too.

libcxx/include/uchar.h

51

This should maybe be

# define __need_size_t
# include_next <stddef.h>

Or maybe that's a bit too stringent.

[ldionne]

libcxx/include/uchar.h

51

Shouldn't we include __mbstate_t.h instead? Then we can tweak what that header does as needed to avoid including wchar.h from it unconditionally, perhaps?

[iana]

libcxx/include/uchar.h

51

The problem with including __mbstate_t.h here is it adds ::mbstate_t which I don't think we want in the C header. I was originally doing that in cuchar, but then that was kind of weird because it would declare ::mbstate_t all the time when the rest of the file is gated on _LIBCPP_CXX03_LANG. People thought it was weird to conditionally include __mbstate_t.h in cuchar too. Also __mbstate_t.h starts off trying to include all of wchar.h which seems counter to the comment at its top?

[iana]

Edits with feedback from @ldionne

[ldionne]

So hard for such a small patch! Thanks for working through this with me!

libcxx/include/__mbstate_t.h

18–25

34–38

libcxx/include/wchar.h

126

[iana]

Review changes, fix failed test.

[iana]

__locale uses mbstate_t without including __mbstate_t.h

[iana]

std::mbstate_t is still needed even when uchar/wchar aren't available. Make a __std_mbstate_t.h header for that.

[iana]

Revert ignore_format.txt

[ldionne]

This LGTM to merge, the CI failures are unrelated, I just checked.

[sylvestre.ledru]

i wonder if this isn't caused this regression:
https://github.com/llvm/llvm-project/issues/62523

[sylvestre.ledru]

I confirmed that a local revert fixed the issue for me

libcxx/include/CMakeLists.txt

set(files
   __ranges/subrange.h
   __ranges/take_view.h
   __ranges/take_while_view.h
   __ranges/transform_view.h
   __ranges/view_interface.h
   __ranges/views.h
   __ranges/zip_view.h
   __split_buffer
+  __std_mbstate_t.h
   __string/char_traits.h
   __string/constexpr_c_functions.h
   __string/extern_template_lists.h
   __support/android/locale_bionic.h
   __support/fuchsia/xlocale.h
   __support/ibm/gettod_zos.h
   __support/ibm/locale_mgmt_zos.h
   __support/ibm/nanosleep.h

libcxx/include/__locale

 #include <string>
 
 // Some platforms require more includes than others. Keep the includes on all plaforms for now.
 #include <cstddef>
 #include <cstring>
 
 #ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
 #  include <cwchar>
+#else
+#  include <__std_mbstate_t.h>
 #endif
 
 #if defined(_LIBCPP_MSVCRT_LIKE)
 # include <__support/win32/locale_win32.h>
 #elif defined(_AIX) || defined(__MVS__)
 # include <__support/ibm/xlocale.h>
 #elif defined(__ANDROID__)
 # include <__support/android/locale_bionic.h>

libcxx/include/__mbstate_t.h

 #ifndef _LIBCPP___MBSTATE_T_H
 #define _LIBCPP___MBSTATE_T_H
 
 #include <__config>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
-// TODO(ldionne):
-// The goal of this header is to provide mbstate_t without having to pull in
-// <wchar.h> or <uchar.h>. This is necessary because we need that type even
-// when we don't have (or try to provide) support for wchar_t, because several
-// types like std::fpos are defined in terms of mbstate_t.
+// The goal of this header is to provide mbstate_t without requiring all of
+// <uchar.h> or <wchar.h>. It's also used by the libc++ versions of <uchar.h>
+// and <wchar.h> to get mbstate_t when the C library doesn't provide <uchar.h>
+// or <wchar.h>, hence the #include_next of those headers instead of #include.
+// (e.g. if <wchar.h> isn't present in the C library, the libc++ <wchar.h>
+// will include this header. This header needs to not turn around and cyclically
+// include <wchar.h>, but fall through to <uchar.h>.)
 //
-// This is a gruesome hack, but I don't know how to make it cleaner for
-// the time being.
+// This does not define std::mbstate_t -- this only brings in the declaration
+// in the global namespace.
 
-#ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
-#   include <wchar.h> // for mbstate_t
-#elif __has_include(<bits/types/mbstate_t.h>)
+#if __has_include(<bits/types/mbstate_t.h>)
 #   include <bits/types/mbstate_t.h> // works on most Unixes
 #elif __has_include(<sys/_types/_mbstate_t.h>)
 #   include <sys/_types/_mbstate_t.h> // works on Darwin
+#elif !defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS) && __has_include_next(<wchar.h>)
+#   include_next <wchar.h> // fall back to the C standard provider of mbstate_t
+#elif __has_include_next(<uchar.h>)
+#   include_next <uchar.h> // <uchar.h> is also required to make mbstate_t visible
 #else
-#   error "The library was configured without support for wide-characters, but we don't know how to get the definition of mbstate_t without <wchar.h> on your platform."
+#   error "We don't know how to get the definition of mbstate_t without <wchar.h> on your platform."
 #endif
 
-_LIBCPP_BEGIN_NAMESPACE_STD
-
-using ::mbstate_t _LIBCPP_USING_IF_EXISTS;
-
-_LIBCPP_END_NAMESPACE_STD
-
 #endif // _LIBCPP___MBSTATE_T_H

libcxx/include/__std_mbstate_t.h

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP___STD_MBSTATE_T_H
+#define _LIBCPP___STD_MBSTATE_T_H
+
+#include <__config>
+#include <__mbstate_t.h>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+// The goal of this header is to provide std::mbstate_t without requiring all
+// of <cuchar> or <cwchar>.
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+using ::mbstate_t _LIBCPP_USING_IF_EXISTS;
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP___STD_MBSTATE_T_H

libcxx/include/iosfwd

 #include <__config>
 #include <__fwd/fstream.h>
 #include <__fwd/ios.h>
 #include <__fwd/istream.h>
 #include <__fwd/ostream.h>
 #include <__fwd/sstream.h>
 #include <__fwd/streambuf.h>
 #include <__fwd/string.h>
-#include <__mbstate_t.h>
+#include <__std_mbstate_t.h>
 #include <version>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 

libcxx/include/module.modulemap.in

   module __hash_table        {         header "__hash_table"        export * }
   module __locale            {
     @requires_LIBCXX_ENABLE_LOCALIZATION@
     private header "__locale" export *
   }
   module __mbstate_t         { private header "__mbstate_t.h"       export * }
   module __node_handle       { private header "__node_handle"       export * }
   module __split_buffer      { private header "__split_buffer"      export * }
+  module __std_mbstate_t     { private header "__std_mbstate_t.h"   export * }
   module __threading_support {         header "__threading_support" export * }
   module __tree              {         header "__tree"              export * }
   module __undef_macros      {         header "__undef_macros"      export * }
   module __verbose_abort     {         header "__verbose_abort"     export * }
 
   module experimental {
     requires cplusplus11
 

libcxx/include/uchar.h

 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
 #if !defined(_LIBCPP_CXX03_LANG)
 
 // Some platforms don't implement <uchar.h> and we don't want to give a hard
 // error on those platforms. When the platform doesn't provide <uchar.h>, at
-// least include <stddef.h> so we get the declaration for size_t.
+// least include <stddef.h> so we get the declaration for size_t, and try to
+// get the declaration of mbstate_t too.
 #if __has_include_next(<uchar.h>)
 # include_next <uchar.h>
 #else
+# include <__mbstate_t.h>
 # include <stddef.h>
 #endif
 
 #endif // _LIBCPP_CXX03_LANG
 
 #endif // _LIBCPP_UCHAR_H

libcxx/include/wchar.h

 #endif
 
 #ifdef __cplusplus
 #define __CORRECT_ISO_CPP_WCHAR_H_PROTO
 #endif
 
 #  if __has_include_next(<wchar.h>)
 #    include_next <wchar.h>
+#  else
+#    include <__mbstate_t.h> // make sure we have mbstate_t regardless of the existence of <wchar.h>
 #  endif
 
 // Determine whether we have const-correct overloads for wcschr and friends.
 #if defined(_WCHAR_H_CPLUSPLUS_98_CONFORMANCE_)
 #  define _LIBCPP_WCHAR_H_HAS_CONST_OVERLOADS 1
 #elif defined(__GLIBC_PREREQ)
 #  if __GLIBC_PREREQ(2, 10)
 #    define _LIBCPP_WCHAR_H_HAS_CONST_OVERLOADS 1

libcxx/test/libcxx/private_headers.verify.cpp

 #include <__ranges/subrange.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/subrange.h'}}
 #include <__ranges/take_view.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/take_view.h'}}
 #include <__ranges/take_while_view.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/take_while_view.h'}}
 #include <__ranges/transform_view.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/transform_view.h'}}
 #include <__ranges/view_interface.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/view_interface.h'}}
 #include <__ranges/views.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/views.h'}}
 #include <__ranges/zip_view.h> // expected-error@*:* {{use of private header from outside its module: '__ranges/zip_view.h'}}
 #include <__split_buffer> // expected-error@*:* {{use of private header from outside its module: '__split_buffer'}}
+#include <__std_mbstate_t.h> // expected-error@*:* {{use of private header from outside its module: '__std_mbstate_t.h'}}
 #include <__string/char_traits.h> // expected-error@*:* {{use of private header from outside its module: '__string/char_traits.h'}}
 #include <__string/constexpr_c_functions.h> // expected-error@*:* {{use of private header from outside its module: '__string/constexpr_c_functions.h'}}
 #include <__string/extern_template_lists.h> // expected-error@*:* {{use of private header from outside its module: '__string/extern_template_lists.h'}}
 #include <__system_error/errc.h> // expected-error@*:* {{use of private header from outside its module: '__system_error/errc.h'}}
 #include <__system_error/error_category.h> // expected-error@*:* {{use of private header from outside its module: '__system_error/error_category.h'}}
 #include <__system_error/error_code.h> // expected-error@*:* {{use of private header from outside its module: '__system_error/error_code.h'}}
 #include <__system_error/error_condition.h> // expected-error@*:* {{use of private header from outside its module: '__system_error/error_condition.h'}}
 #include <__system_error/system_error.h> // expected-error@*:* {{use of private header from outside its module: '__system_error/system_error.h'}}

libcxx/utils/generate_iwyu_mapping.py

for i in detail_files:
         elif i == '__mutex_base': continue
         elif i == '__node_handle': public = ['map', 'set', 'unordered_map', 'unordered_set']
         elif i == '__pstl_algorithm': continue
         elif i == '__pstl_config_site.in': continue
         elif i == '__pstl_execution': continue
         elif i == '__pstl_memory': continue
         elif i == '__pstl_numeric': continue
         elif i == '__split_buffer': public = ['deque', 'vector']
+        elif i == '__std_mbstate_t.h': continue
         elif i == '__threading_support': public = ['atomic', 'mutex', 'semaphore', 'thread']
         elif i == '__tree': public = ['map', 'set']
         elif i == '__undef_macros': continue
         elif i == '__verbose_abort': continue
-        else: panic()
+        else: panic(i)
 
         for p in public:
             result.append(f'{generate(f"<{i}>", p)},')
 
     result.sort()
     return result
 
 def main():