[libc++] Run picolibc tests with qemu

[philnik]

libcxx/CMakeLists.txt

480 ↗

(On Diff #541257)

I'm not super happy with this. Could this instead be set through an environment variable when running the tests with qemu-system-arm as the default?

[DavidSpickett]

Surprised to not see bootcode here, but it is qemu and one can get away without it a lot of the time. Is that the case?

I will add qemu-system-arm to our containers. Focal's apt version is 4.2.1 and I checked it has the board and cpu options needed, so I expect it'll work fine.

[michaelplatings]

Move code to find qemu-system-arm program

[michaelplatings]

In D155521#4509345, @DavidSpickett wrote:

Surprised to not see bootcode here, but it is qemu and one can get away without it a lot of the time. Is that the case?

I will add qemu-system-arm to our containers. Focal's apt version is 4.2.1 and I checked it has the board and cpu options needed, so I expect it'll work fine.

Thanks very much David.

I think bootcode is coming from the "crt0-semihost" library which is part of picolibc. Source is here.

libcxx/CMakeLists.txt

480 ↗

(On Diff #541257)

I agree its not ideal, it has the inappropriate intimacy code smell.

I want to keep find_program because it's good for developer experience. With environment variables you have to remember to keep them set in your environment whereas with CMake cache variables you can set them once at configure time. There's precedent for using find_program in a cache file in clang/cmake/caches/3-stage-base.cmake so I've moved the find_program to Armv7M-picolibc.cmake where it sits more comfortably.

[michaelplatings]

Use 16MB area of RAM to enable running bigger tests

[DavidSpickett]

QEMU has been installed.

buildkite-linaro-aarch64-libcxx-02:/# qemu-system-arm --version
QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.27)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

And it's on PATH, no /usr/local... stuff needed this time.

[michaelplatings]

In D155521#4510398, @DavidSpickett wrote:

QEMU has been installed.

buildkite-linaro-aarch64-libcxx-02:/# qemu-system-arm --version
QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.27)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

And it's on PATH, no /usr/local... stuff needed this time.

Great, thanks David. If it's on PATH then this should Just Work...

[arichardson]

libcxx/test/libcxx/selftest/convenience_substitutions/build_run.sh.cpp

9 ↗

(On Diff #541466)

Can be handled by the qemu script but that can always be a follow up change.

libcxx/utils/run-qemu.sh

16 ↗

(On Diff #541466)

The arg= semihosting flag can be used to create argv.

[michaelplatings]

Handle command line arguments

libcxx/utils/run-qemu.sh

16 ↗

(On Diff #541466)

Thanks, I didn't know qemu had that option.

[michaelplatings]

Rebase after -Wcast-qual error was fixed

[ldionne]

While you are iterating on this, please consider removing all the other jobs from the buildkite-pipeline.yml file to avoid spamming the CI. We're currently crunching for the LLVM 17 release and our CI resources are under heavy stress.

BTW, I'll be happy to review this once the dust settles for LLVM 17. I'm silent on this so far for lack of time, not interest.

[michaelplatings]

OK if I get any more review comments I'll do that. The new "Armv7-M picolibc" configuration is passing in BuildKite now and the failures in other configurations look unrelated so I have no changes planned at this point.

[michaelplatings]

Rebase

[michaelplatings]

Ping

[michaelplatings]

Rebase

[arichardson]

libcxx/utils/run-qemu.sh

19 ↗

(On Diff #557479)

I wonder if this should instead be a full executor such as qemu_system.py (similar to ssh.py) that accepts all the same flags and just requires something like --qemu=qemu-system-arm --cpu=.. --machine=...

[arichardson]

libcxx/utils/run-qemu.sh

19 ↗

(On Diff #557479)

Done as part of https://github.com/llvm/llvm-project/pull/68643, let me know if that works for you?

[michaelplatings]

libcxx/utils/run-qemu.sh

19 ↗

(On Diff #557479)

I'm not sure whether the Python script is an improvement but I'd go with whatever gets these patches landed sooner. I think that to be used in place of run.py the Python script would need to support the full interface of run.py i.e. --codesign_identity et al.

[arichardson]

libcxx/utils/run-qemu.sh

19 ↗

(On Diff #557479)

That's a good point, so far I've tested it with libunwind, will try libc++ as well.

[arichardson]

libcxx/utils/run-qemu.sh

19 ↗

(On Diff #557479)

I was able to use the new script for libc++ since the other arguments do not appear to be used.

[domin144]

Use qemu_baremetal.py script from https://github.com/llvm/llvm-project/pull/68643

[domin144]

Rebase
Fix paths to qemu_baremetal.py

[domin144]

Xfail remaining libunwind test on clang-16

[DavidSpickett]

For reviewer context, Linaro's builders are on clang 16.0.5 at the moment and we prefer to keep that the same across the buildbots. An upgrade comes with more risk than this one test, so I advised Dominik to go with xfailing it. Linaro will of course move to 17 eventually and this test will start running.

(plus the xfail is a useful note for users other than the buildbots themselves)

[domin144]

reupload to retrigger the buildkite job

[ldionne]

libcxx/test/configs/armv7m-libc++.cfg.in

26 ↗

(On Diff #558140)

libcxx/test/libcxx/selftest/pass.mm/run-error.pass.mm and probably others are now using the libcxx-fake-executor Lit feature, but it's not defined anywhere after this patch.

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

9

Remove this XFAIL?

[domin144]

Applied changes from comments.

[domin144]

Re-add xfail for libunwind_02.pass.cpp tests.
It somehow got missed in the last update.

[ldionne]

Thanks! LGTM assuming the CI passes with my comment applied.

libcxx/test/configs/armv7m-picolibc-libc++.cfg.in

39–40

Instead, please do this in your CMake cache:

set(LIBCXX_TEST_PARAMS "long_tests=False" CACHE STRING "")

[domin144]

Applied comment:

• moved "long_tests=False" to the cmake cache file.

libcxx/cmake/caches/Armv7M-picolibc.cmake

 set(LIBCXX_ENABLE_THREADS OFF CACHE BOOL "")
 set(LIBCXX_ENABLE_WIDE_CHARACTERS OFF CACHE BOOL "")
 set(LIBCXX_INCLUDE_BENCHMARKS OFF CACHE BOOL "")
+# Long tests are prohibitively slow when run via emulation.
+set(LIBCXX_TEST_PARAMS "long_tests=False" CACHE STRING "")
 set(LIBCXX_USE_COMPILER_RT ON CACHE BOOL "")
 set(LIBUNWIND_ENABLE_SHARED OFF CACHE BOOL "")
 set(LIBUNWIND_ENABLE_STATIC ON CACHE BOOL "")

 set(LIBUNWIND_IS_BAREMETAL ON CACHE BOOL "")
 set(LIBUNWIND_REMEMBER_HEAP_ALLOC ON CACHE BOOL "")
 set(LIBUNWIND_USE_COMPILER_RT ON CACHE BOOL "")
+find_program(QEMU_SYSTEM_ARM qemu-system-arm)

libcxx/docs/index.rst

 Android 5.0+          i386, x86_64, arm, arm64
 Windows               i386, x86_64              Both MSVC and MinGW style environments, ABI in MSVC environments is :doc:`unstable <DesignDocs/ABIVersioning>`
 AIX 7.2TL5+           powerpc, powerpc64
-Embedded (picolibc)   arm                       Support for building with picolibc is currently work-in-progress
+Embedded (picolibc)   arm
 ===================== ========================= ============================
 
 Generally speaking, libc++ should work on any platform that provides a fairly complete

libcxx/test/configs/armv7m-picolibc-libc++.cfg.in

     ' -Wl,--defsym=__ram_size=0x1000000'
     ' -Wl,--defsym=__stack_size=0x1000'
 ))
+
+config.executor = (
+    '@LIBCXX_SOURCE_DIR@/utils/qemu_baremetal.py'
+    ' --qemu @QEMU_SYSTEM_ARM@'
+    ' --machine mps2-an385'
+    ' --cpu cortex-m3')
 config.substitutions.append(('%{exec}',
-    'true' # TODO use qemu-system-arm
+    '%{executor}'
+    ' --execdir %T'
 ))
-config.available_features.add('libcxx-fake-executor')
 
 import os, site
 site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

 #
 # ===----------------------------------------------------------------------===##
 
-# XFAIL: libcxx-fake-executor
+# With picolibc, test_program_stderr_is_not_conflated_with_stdout fails
+# because stdout & stderr are treated as the same.
+# XFAIL: LIBCXX-PICOLIBC-FIXME
 
 # Note: We prepend arguments with 'x' to avoid thinking there are too few
 #       arguments in case an argument is an empty string.

libcxx/test/libcxx/selftest/pass.cpp/run-error.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 
-// UNSUPPORTED: libcxx-fake-executor
 // XFAIL: *
 
 // Make sure the test DOES NOT pass if it fails at runtime.

libcxx/test/libcxx/selftest/pass.mm/run-error.pass.mm

 //===----------------------------------------------------------------------===//
 
 // REQUIRES: objective-c++
-// UNSUPPORTED: libcxx-fake-executor
 
 // XFAIL: *
 

libcxx/test/libcxx/selftest/stdin-is-piped.sh.cpp

 
 // Make sure that the executor pipes standard input to the test-executable being run.
 
+// XFAIL: LIBCXX-PICOLIBC-FIXME
+
 // RUN: %{build}
 // RUN: echo "abc" | %{exec} %t.exe
 

libcxx/test/std/algorithms/alg.sorting/alg.sort/sort/sort.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 
+// This test appears to hang with picolibc & qemu.
+// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
+
 // <algorithm>
 
 // template<RandomAccessIterator Iter>

libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp

 //===----------------------------------------------------------------------===//
 
 // XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12}}
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 // <system_error>
 

libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp

 // const error_category& system_category();
 
 // XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12}}
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 #include <system_error>
 #include <cassert>

libcxx/test/std/input.output/iostream.objects/narrow.stream.objects/cerr.sh.cpp

 //
 //===----------------------------------------------------------------------===//
 
-// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 // <iostream>
 

libcxx/test/std/input.output/iostream.objects/narrow.stream.objects/cin.sh.cpp

 
 // TODO: Investigate
 // UNSUPPORTED: LIBCXX-AIX-FIXME
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 // This test hangs on Android devices that lack shell_v2, which was added in
 // Android N (API 24).

libcxx/test/std/input.output/iostream.objects/narrow.stream.objects/clog.sh.cpp

 //
 //===----------------------------------------------------------------------===//
 
-// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 // <iostream>
 

libcxx/test/std/input.output/iostream.objects/narrow.stream.objects/cout.sh.cpp

 //
 //===----------------------------------------------------------------------===//
 
-// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
-
 // <iostream>
 
 // ostream cout;

libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/sized_delete_array14.pass.cpp

 // test sized operator delete[] replacement.
 
 // UNSUPPORTED: sanitizer-new-delete, c++03, c++11
-// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
 
 // NOTE: Clang does not enable sized-deallocation in C++14 and beyond by
 // default. It is only enabled when -fsized-deallocation is given.

libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/sized_delete14.pass.cpp

 // test sized operator delete replacement.
 
 // UNSUPPORTED: sanitizer-new-delete, c++03, c++11
-// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
 
 // NOTE: Clang does not enable sized-deallocation in C++14 and beyond by
 // default. It is only enabled when -fsized-deallocation is given.

libcxx/test/std/localization/locale.categories/category.ctype/facet.ctype.special/facet.ctype.char.statics/classic_table.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 
+// XFAIL: LIBCXX-PICOLIBC-FIXME
+
 // <locale>
 
 // template <> class ctype<char>

libcxx/test/std/localization/locale.categories/category.numeric/locale.nm.put/facet.num.put.members/put_long_double.pass.cpp

 // iter_type put(iter_type s, ios_base& iob, char_type fill, long double v) const;
 
 // XFAIL: win32-broken-printf-g-precision
+// XFAIL: LIBCXX-PICOLIBC-FIXME
 
 // Needs more investigation, but this is probably failing on Android M (API 23)
 // and up because the printf formatting of NAN changed.

libcxx/test/std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.bin/eval.PR44847.pass.cpp

 // Serializing/deserializing the state of the RNG requires iostreams
 // UNSUPPORTED: no-localization
 
+// This test appears to hang with picolibc & qemu.
+// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
+
 #include <random>
 #include <numeric>
 #include <vector>

libcxx/test/std/time/time.clock/time.clock.file/now.pass.cpp

 
 // UNSUPPORTED: availability-filesystem-missing
 
+// qemu: Unsupported SemiHosting SWI 0x30
+// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
+
 // <chrono>
 
 // file_clock

libcxxabi/test/configs/armv7m-picolibc-libc++abi.cfg.in

     ' -Wl,--defsym=__ram_size=0x1000000'
     ' -Wl,--defsym=__stack_size=0x1000'
 ))
+
+config.executor = (
+    '@LIBCXXABI_LIBCXX_PATH@/utils/qemu_baremetal.py'
+    ' --qemu @QEMU_SYSTEM_ARM@'
+    ' --machine mps2-an385'
+    ' --cpu cortex-m3')
 config.substitutions.append(('%{exec}',
-    'true' # TODO use qemu-system-arm
+    '%{executor}'
+    ' --execdir %T'
 ))
-config.available_features.add('libcxx-fake-executor')
 
 import os, site
 site.addsitedir(os.path.join('@LIBCXXABI_LIBCXX_PATH@', 'utils'))

libunwind/test/configs/armv7m-picolibc-libunwind.cfg.in

     ' -Wl,--defsym=__ram_size=0x1000000'
     ' -Wl,--defsym=__stack_size=0x1000'
 ))
+
+config.executor = (
+    '@LIBUNWIND_LIBCXX_PATH@/utils/qemu_baremetal.py'
+    ' --qemu @QEMU_SYSTEM_ARM@'
+    ' --machine mps2-an385'
+    ' --cpu cortex-m3')
 config.substitutions.append(('%{exec}',
-    'true' # TODO use qemu-system-arm
+    '%{executor}'
+    ' --execdir %T'
 ))
-config.available_features.add('libcxx-fake-executor')
 
 import os, site
 site.addsitedir(os.path.join('@LIBUNWIND_LIBCXX_PATH@', 'utils'))

libunwind/test/libunwind_02.pass.cpp

 // TODO: Figure out why this fails with Memory Sanitizer.
 // XFAIL: msan
 
+// This test fails on older llvm, when built with picolibc.
+// XFAIL: clang-16 && LIBCXX-PICOLIBC-FIXME
+
 #undef NDEBUG
 #include <assert.h>
 #include <stdlib.h>