Page MenuHomePhabricator

[libc++][format] Implement Unicode support.
Needs RevisionPublic

Authored by Mordante on Mon, May 31, 9:05 AM.

Details

Reviewers
curdeius
ldionne
miscco
vitaut
Group Reviewers
Restricted Project
Summary

This adds the width estimation functions to the std-format-spec.

Implements parts of:

  • P0645 Text Formatting
  • P1868 width: clarifying units of width and precision in std::format

Diff Detail

Unit TestsFailed

TimeTest
620 mslibcxx CI GCC-next/C++20 > libc++.std/atomics/atomics_types_operations/atomics_types_operations_req::atomic_is_lock_free.pass.cpp
Script: -- : 'COMPILED WITH'; /usr/bin/g++-11 /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.req/atomic_is_lock_free.pass.cpp -v -include /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support/nasty_macros.h -nostdinc++ -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/include/c++/v1 -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/include/c++build -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-user-defined-literals -Wno-noexcept-type -Wno-aligned-allocation-unavailable -Wno-atomic-alignment -Wno-sized-deallocation -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_DISABLE_AVAILABILITY -Wno-macro-redefined -D_LIBCPP_HAS_THREAD_API_PTHREAD -Wno-macro-redefined -D_LIBCPP_ABI_VERSION=1 -lc++experimental -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -nodefaultlibs -lc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -latomic -o /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/test/std/atomics/atomics.types.
1,870 mslibcxx CI GCC-next/C++20 > libc++.std/iterators/iterator_primitives/range_iter_ops/range_iter_ops_advance::advance.pass.cpp
Script: -- : 'COMPILED WITH'; /usr/bin/g++-11 /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/std/iterators/iterator.primitives/range.iter.ops/range.iter.ops.advance/advance.pass.cpp -v -include /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support/nasty_macros.h -nostdinc++ -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/include/c++/v1 -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/include/c++build -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-user-defined-literals -Wno-noexcept-type -Wno-aligned-allocation-unavailable -Wno-atomic-alignment -Wno-sized-deallocation -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_DISABLE_AVAILABILITY -Wno-macro-redefined -D_LIBCPP_HAS_THREAD_API_PTHREAD -Wno-macro-redefined -D_LIBCPP_ABI_VERSION=1 -lc++experimental -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -nodefaultlibs -lc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -latomic -o /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/test/std/iterators/iterator.
1,370 mslibcxx CI GCC-next/C++20 > libc++.std/iterators/iterator_primitives/range_iter_ops/range_iter_ops_next::iterator_count.pass.cpp
Script: -- : 'COMPILED WITH'; /usr/bin/g++-11 /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/std/iterators/iterator.primitives/range.iter.ops/range.iter.ops.next/iterator_count.pass.cpp -v -include /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support/nasty_macros.h -nostdinc++ -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/include/c++/v1 -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/include/c++build -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-user-defined-literals -Wno-noexcept-type -Wno-aligned-allocation-unavailable -Wno-atomic-alignment -Wno-sized-deallocation -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_DISABLE_AVAILABILITY -Wno-macro-redefined -D_LIBCPP_HAS_THREAD_API_PTHREAD -Wno-macro-redefined -D_LIBCPP_ABI_VERSION=1 -lc++experimental -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -nodefaultlibs -lc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -latomic -o /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/test/std/iterators/iterator.
1,380 mslibcxx CI GCC-next/C++20 > libc++.std/iterators/iterator_primitives/range_iter_ops/range_iter_ops_next::iterator_count_sentinel.pass.cpp
Script: -- : 'COMPILED WITH'; /usr/bin/g++-11 /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/std/iterators/iterator.primitives/range.iter.ops/range.iter.ops.next/iterator_count_sentinel.pass.cpp -v -include /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support/nasty_macros.h -nostdinc++ -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/include/c++/v1 -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/include/c++build -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-user-defined-literals -Wno-noexcept-type -Wno-aligned-allocation-unavailable -Wno-atomic-alignment -Wno-sized-deallocation -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_DISABLE_AVAILABILITY -Wno-macro-redefined -D_LIBCPP_HAS_THREAD_API_PTHREAD -Wno-macro-redefined -D_LIBCPP_ABI_VERSION=1 -lc++experimental -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -nodefaultlibs -lc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -latomic -o /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/test/std/iterators/iterator.
1,640 mslibcxx CI GCC-next/C++20 > libc++.std/iterators/iterator_primitives/range_iter_ops/range_iter_ops_next::iterator_sentinel.pass.cpp
Script: -- : 'COMPILED WITH'; /usr/bin/g++-11 /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/std/iterators/iterator.primitives/range.iter.ops/range.iter.ops.next/iterator_sentinel.pass.cpp -v -include /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support/nasty_macros.h -nostdinc++ -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/include/c++/v1 -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/include/c++build -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -I/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-user-defined-literals -Wno-noexcept-type -Wno-aligned-allocation-unavailable -Wno-atomic-alignment -Wno-sized-deallocation -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_DISABLE_AVAILABILITY -Wno-macro-redefined -D_LIBCPP_HAS_THREAD_API_PTHREAD -Wno-macro-redefined -D_LIBCPP_ABI_VERSION=1 -lc++experimental -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -L/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -Wl,-rpath,/home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/./lib -nodefaultlibs -lc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -latomic -o /home/libcxx-builder/.buildkite-agent/builds/e19784d8162c-1/llvm-project/libcxx-ci/build/generic-gcc-next/projects/libcxx/test/std/iterators/iterator.
View Full Test Results (12 Failed)

Event Timeline

Mordante created this revision.Mon, May 31, 9:05 AM
Mordante requested review of this revision.Mon, May 31, 9:05 AM
Herald added a project: Restricted Project. · View Herald TranscriptMon, May 31, 9:05 AM
Herald added a reviewer: Restricted Project. · View Herald Transcript
Mordante updated this revision to Diff 348827.Mon, May 31, 9:40 AM

Remove an accidental edit, which caused the code to become invalid.

Mordante updated this revision to Diff 349346.Wed, Jun 2, 12:17 PM

Rebase.
Improve comments.

Mordante updated this revision to Diff 349456.Wed, Jun 2, 10:47 PM

Temporary disable a failing test on Windows.

Mordante edited the summary of this revision. (Show Details)Sat, Jun 5, 4:12 AM
Mordante added reviewers: curdeius, ldionne, miscco, vitaut.
vitaut requested changes to this revision.Fri, Jun 18, 7:56 AM
vitaut added inline comments.
libcxx/include/__format/parser_std_format_spec.h
669

I think you mean code units and not characters. Same throughout the diff.

679

16 ->32

687–688

I think a more robust error handling would be to count each invalid code unit as contributing 1 to the width. Terminals normally use replacement characters so it will give the correct result.

744–767

A sequence of ifs would likely be more readable.

828–830

Not sure what you mean by "a full blown Unicode parser" but I don't think it's ever a good idea to throw exception on width estimation.

832

Why not set __threshold to the maximum value to disable it instead of introducing another flag?

851

I recommend returning a struct with meaningful field names.

994

What does the check __precision != __format::__number_max do here?

Note that __number_max is a valid precision.

This revision now requires changes to proceed.Fri, Jun 18, 7:56 AM