This is an archive of the discontinued LLVM Phabricator instance.

[libc++] Add CI job testing on FreeBSD
ClosedPublic

Authored by emaste on Jun 17 2022, 11:48 AM.

Details

Reviewers
infrandomness
dim
ldionne
Mordante
Group Reviewers
Restricted Project
Restricted Project
Commits
rG720e6297d1f5: [libc++] Add CI job testing on FreeBSD

Diff Detail

Event Timeline

ldionne created this revision.Jun 17 2022, 11:48 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 17 2022, 11:48 AM
Herald added a subscriber: arichardson. · View Herald Transcript
ldionne requested review of this revision.Jun 17 2022, 11:48 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 17 2022, 11:48 AM
Herald added a reviewer: Restricted Project. · View Herald Transcript

@infrandomness Nicely suggested using one of their machines running FreeBSD. We've set up a BuildKite agent and this patch should allow targeting the agent in our CI.

We are aware that some tests are failing on FreeBSD, and it would be nice if the FreeBSD maintainers could take a look. Also, it would be nice to add at least one other FreeBSD runner that is not backed by an individual to ensure that our CI setup is robust. If @infrandomness can't provide the machine anymore in the future, or even if they simply have a power outage, it would be nice for our CI setup to still work. But this is an excellent start!

@infrandomness thanks a lot for providing this CI runner!

I had a look at the failed tests and it seems the number isn't too large and most of them seem to be filesystem and locales.
Two areas where I'm not surprised that there are issues.

@ldionne Do we want to add XFAIL to the failing tests or add a soft_fail entry?

I have some WIP patches for the locale tests that I should update . I'm not sure how long it will take to fix all the tests so maybe xfail is better so that regressions are caught?

I have some WIP patches for the locale tests that I should update . I'm not sure how long it will take to fix all the tests so maybe xfail is better so that regressions are caught?

It would be great when you can fix these tests! We can indeed make them XFAIL now and remove that in patches that fixes the tests.
We do the same for AIX, they have some people submitting fixes for their failing locale tests.

ldionne updated this revision to Diff 439491.Jun 23 2022, 12:12 PM

Add a bunch of XFAILs for tests failing on FreeBSD. Let's try to keep those temporary.

ldionne updated this revision to Diff 439515.Jun 23 2022, 1:24 PM

Rebase onto main.

ldionne updated this revision to Diff 440185.Jun 27 2022, 5:49 AM

Add XFAIL for libc++abi test.

Herald added a project: Restricted Project. · View Herald TranscriptJun 27 2022, 5:49 AM
Herald added a reviewer: Restricted Project. · View Herald Transcript

@dim @emaste Can someone on FreeBSD please pick this up?

@dim @emaste Can someone on FreeBSD please pick this up?

Ping, LLVM 15 is branching soon and it would be awesome to have some confidence on our FreeBSD support.

@dim @emaste what can we do to get this landed? I really would like to see this CI integrated in the pre-commit CI.

Ping.

This situation is exactly why we have the support policy that we have: we are interested in supporting platforms that want to be supported, i.e. that want to invest the time required to make things work. People have volunteered machines to test on FreeBSD, and have volunteered time to get this patch up. All it needs is someone (probably someone who cares first hand about FreeBSD support) to finish it up and start fixing the LIBCXX-FREEBSD-FIXMEs added here.

Ping.

This situation is exactly why we have the support policy that we have: we are interested in supporting platforms that want to be supported, i.e. that want to invest the time required to make things work. People have volunteered machines to test on FreeBSD, and have volunteered time to get this patch up. All it needs is someone (probably someone who cares first hand about FreeBSD support) to finish it up and start fixing the LIBCXX-FREEBSD-FIXMEs added here.

I would be happy to address these FIXMEs, but I would have to do so outside working hours so my time is somewhat limited. I recently ran the testsuite on my Arm Morello system running CheriBSD and it seems like a lot of the filesystem ones are fixed on this FreeBSD 14 snapshot.

Which version of FreeBSD is the builder running? This would allow me to reproduce and attach a debugger on a similar system.

emaste added inline comments.Oct 17 2022, 1:35 PM
libcxx/include/__config
134 ↗(On Diff #440185)

should this part be committed independently?

philnik added a subscriber: philnik.Dec 6 2022, 4:43 AM

@dim @emaste Could you please pick this up? If you don't pick this up until mid of January IMO we have to revive and land D127657. You had more than enough time to get pre-commit CI going.

buildkite reports

$ trap 'kill -- $$' INT TERM QUIT; libcxx/utils/ci/generate-buildkite-pipeline | buildkite-agent pipeline upload
2022-12-14 21:08:55 INFO   Reading pipeline config from STDIN
2022-12-14 21:08:56 INFO   Updating BUILDKITE_COMMIT to "421692721d91d2db7d2d24f299239eaecbfc1a89"
2022-12-14 21:08:56 FATAL  Pipeline parsing of "(stdin)" failed (Failed to parse pipeline: line 850: did not find expected key)
🚨 Error: The command exited with status 1
user command error: exit status 1
libcxx/include/__config
134 ↗(On Diff #440185)

Can we commit this separately first?

libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
9 ↗(On Diff #440185)

fails to apply, // XFAIL: LIBCXX-AIX-FIXME has been added here (and neg_format.pass.cpp, pos_format.pass.cpp, thousands_sep.pass.cpp, get_one_wide.pass.cpp, get_one.pass.cpp)

will deal with this once buildkite is sorted out

philnik added inline comments.Dec 14 2022, 1:37 PM
libcxx/include/__config
134 ↗(On Diff #440185)

I'm sure we can. Just create a patch; though I don't understand why this change is even needed.

libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
9 ↗(On Diff #440185)

You should be able to apply the patch with arc patch D128084 and then rebase.

Tried applying this change via D140050 but the status reports "Waiting for agent 1h 13m"
https://buildkite.com/llvm-project/libcxx-ci/builds/16012#01851280-1015-418c-a381-c88ae2623955

Buildkite is running on my test worker configured with the token I've been provided. Looking to see what else needs to be configured

Tried applying this change via D140050 but the status reports "Waiting for agent 1h 13m"
https://buildkite.com/llvm-project/libcxx-ci/builds/16012#01851280-1015-418c-a381-c88ae2623955

Buildkite is running on my test worker configured with the token I've been provided. Looking to see what else needs to be configured

With what invocation are you running the buildkite agent? Looking at our Dockerfile you probably have to do something like echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=freebsd\"" >> "/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg". (FYI for these kinds of questions the libc++ Discord channel might be easier)

emaste added inline comments.Dec 14 2022, 3:21 PM
libcxx/include/__config
134 ↗(On Diff #440185)

Related to D126462 I suppose, @ldionne?

libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
9 ↗(On Diff #440185)

Indeed, but it doesn't matter too much until buildkite is sorted out anyhow.

emaste added a comment.EditedDec 14 2022, 3:38 PM

tags were not set correctly; I'll see about submitting a patch to update AddingNewCIJobs.rst with a sample buildkite-agent.cfg snippet.

Next up, it seems that I need to iterate on installing the right packages. Is there an easy way to rerun a failed job?

https://buildkite.com/llvm-project/libcxx-ci/builds/16012#01851280-1015-418c-a381-c88ae2623955

emaste added inline comments.Dec 15 2022, 6:55 AM
libcxx/utils/libcxx/test/features.py
257 ↗(On Diff #440185)

The intent of this one is to tag specifically those XFAILs added as part of enabling CI? There is one case of XFAIL: freebsd in the tree now (trivial_copy_move_ABI.pass.cpp).

philnik added inline comments.Dec 15 2022, 7:04 AM
libcxx/utils/libcxx/test/features.py
257 ↗(On Diff #440185)

Yeah, this is just that the test fails for some reason. IOW they should be worked through after this patch is landed and checked whether to convert the XFAIL: LIBCXX-FREEBSD-FIXME to an XFAIL: freebsd if the failure is expected/wanted, or fix the test/library that is broken. Looking at the list, std::filesystem probably just isn't implemented on FreeBSD and the locale tests will have to be modified because it's the least portable portable interface ever.

With the XFAILS from this review added I'm seeing Failed Tests (494)

Many of these are compile failures while building the test code, e.g.

******************** TEST 'llvm-libc++-shared.cfg.in :: libcxx/utilities/format/version.compile.pass.cpp' FAILED ********************^M
Script:^M
--^M
: 'COMPILED WITH';  /usr/bin/c++ /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp  --target=x86_64-unknown-freebsd13.1 -nostdinc++ -I /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1 -I /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1 -I /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/support -std=c++2b -Werror -Wall -Wctad-maybe-unsupported -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-noexcept-type -Wno-atomic-alignment -Wno-user-defined-literals -Wno-tautological-compare -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER -D_LIBCPP_ENABLE_EXPERIMENTAL -D_LIBCPP_DISABLE_AVAILABILITY -fcoroutines-ts -Werror=thread-safety -Wuser-defined-warnings  -fsyntax-only^M
--^M
Exit Code: 1^M
^M
Command Output (stdout):^M
--^M
$ ":" "COMPILED WITH"^M
note: command had no output on stdout or stderr^M
$ "/usr/bin/c++" "/usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp" "--target=x86_64-unknown-freebsd13.1" "-nostdinc++" "-I" "/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1" "-I" "/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1" "-I" "/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/support" "-std=c++2b" "-Werror" "-Wall" "-Wctad-maybe-unsupported" "-Wextra" "-Wshadow" "-Wundef" "-Wno-unused-command-line-argument" "-Wno-attributes" "-Wno-pessimizing-move" "-Wno-c++11-extensions" "-Wno-noexcept-type" "-Wno-atomic-alignment" "-Wno-user-defined-literals" "-Wno-tautological-compare" "-Wsign-compare" "-Wunused-variable" "-Wunused-parameter" "-Wunreachable-code" "-Wno-unused-local-typedef" "-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER" "-D_LIBCPP_ENABLE_EXPERIMENTAL" "-D_LIBCPP_DISABLE_AVAILABILITY" "-fcoroutines-ts" "-Werror=thread-safety" "-Wuser-defined-warnings" "-fsyntax-only"^M
# command stderr:^M
In file included from /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp:13:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/format:182:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/format_functions.h:33:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_bool.h:21:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_integral.h:18:^M
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_output.h:376:1: error: anonymous structs and classes must be class members^M
struct __null_sentinel {};^M
^^M
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_output.h:376:1: error: declaration does not declare anything [-Werror,-Wmissing-declarations]^M
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_output.h:379:61: error: expected parameter declarator^M
_LIBCPP_HIDE_FROM_ABI bool operator==(const _CharT* __cstr, __null_sentinel) {^M
                                                            ^^M
/usr/include/sys/cdefs.h:454:25: note: expanded from macro '__null_sentinel'^M
#define __null_sentinel __attribute__((__sentinel__))^M
                        ^^M
In file included from /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/libcxx/utilities/format/version.compile.pass.cpp:13:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/format:182:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/format_functions.h:33:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_bool.h:21:^M
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_integral.h:18:^M
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_output.h:379:28: error: overloaded 'operator==' must be a binary operator (has 1 parameter)^M
_LIBCPP_HIDE_FROM_ABI bool operator==(const _CharT* __cstr, __null_sentinel) {^M
                           ^^M
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__format/formatter_output.h:387:31: error: expected expression^M
  std::ranges::copy(__prefix, __null_sentinel{}, __out_it);^M
                              ^^M
/usr/include/sys/cdefs.h:454:25: note: expanded from macro '__null_sentinel'^M
#define __null_sentinel __attribute__((__sentinel__))^M
                        ^^M
5 errors generated.^M
^M
error: command failed with exit status: 1^M
^M
--^M
^M
********************^M

D140117 should fix the clash of the macro and the struct.

Quick status update, with D140117 applied I still have the following failures:

Failed Tests (31):
  llvm-libc++-shared.cfg.in :: libcxx/atomics/bit-int.verify.cpp
  llvm-libc++-shared.cfg.in :: libcxx/experimental/fexperimental-library.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/algorithms/alg.modifying.operations/alg.transform/ranges.transform.pass.cpp
  llvm-libc++-shared.cfg.in :: std/input.output/stream.buffers/streambuf/streambuf.protected/streambuf.put.area/pbump2gig.pass.cpp
  llvm-libc++-shared.cfg.in :: std/thread/futures/futures.task/futures.task.members/ctad.static.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.cal/time.cal.month/time.cal.month.nonmembers/ostream.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.day.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.duration.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.month.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.weekday.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.year.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.U.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.inplace_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/member.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/ctad.static.compile.pass.cpp

I'll iterate on adjusting XFAILs before submitting a final version of this change

Quick status update, with D140117 applied I still have the following failures:

Failed Tests (31):
  llvm-libc++-shared.cfg.in :: libcxx/atomics/bit-int.verify.cpp
  llvm-libc++-shared.cfg.in :: libcxx/experimental/fexperimental-library.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/algorithms/alg.modifying.operations/alg.transform/ranges.transform.pass.cpp
  llvm-libc++-shared.cfg.in :: std/input.output/stream.buffers/streambuf/streambuf.protected/streambuf.put.area/pbump2gig.pass.cpp
  llvm-libc++-shared.cfg.in :: std/thread/futures/futures.task/futures.task.members/ctad.static.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.cal/time.cal.month/time.cal.month.nonmembers/ostream.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.day.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.duration.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.month.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.weekday.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.year.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.U.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.inplace_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/member.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/ctad.static.compile.pass.cpp

I'll iterate on adjusting XFAILs before submitting a final version of this change

@emaste which compiler are you using? If it's /usr/bin/clang++ that is too old I believe and would explain those test failures.

Quick status update, with D140117 applied I still have the following failures:

Failed Tests (31):
  llvm-libc++-shared.cfg.in :: libcxx/atomics/bit-int.verify.cpp
  llvm-libc++-shared.cfg.in :: libcxx/experimental/fexperimental-library.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/algorithms/alg.modifying.operations/alg.transform/ranges.transform.pass.cpp
  llvm-libc++-shared.cfg.in :: std/input.output/stream.buffers/streambuf/streambuf.protected/streambuf.put.area/pbump2gig.pass.cpp
  llvm-libc++-shared.cfg.in :: std/thread/futures/futures.task/futures.task.members/ctad.static.compile.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.cal/time.cal.month/time.cal.month.nonmembers/ostream.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.day.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.duration.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.month.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.weekday.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.year.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.U.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.inplace_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.expected/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.copy.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/assign.unexpected.move.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/assign/emplace.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/ctor/ctor.unexpect_init_list.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/dtor.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/free.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/expected/expected.void/swap/member.swap.pass.cpp
  llvm-libc++-shared.cfg.in :: std/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/ctad.static.compile.pass.cpp

I'll iterate on adjusting XFAILs before submitting a final version of this change

@emaste which compiler are you using? If it's /usr/bin/clang++ that is too old I believe and would explain those test failures.

Looking at the CI logs, clang 13 seems to be used, which is indeed too old. You have to upgrade to at least 14, 15 recommended for better test coverage.

jrtc27 added a subscriber: jrtc27.Dec 16 2022, 6:05 AM
jrtc27 added inline comments.
libcxx/utils/ci/buildkite-pipeline.yml
1033

You'll need CC/CXX to be clang(++)-14 or higher here like the other generic-cxx2b jobs now that the default moved out of run-buildbot as of ca04b49597ccf137da9dfcbb1844d2e8b6a83c07

Forcing CI to use clang15 for now and have the following failures:

Failed Tests (9):
  llvm-libc++-shared.cfg.in :: std/localization/locale.stdcvt/depr.verify.cpp
  llvm-libc++-shared.cfg.in :: std/localization/locales/locale.convenience/conversions/conversions.buffer/depr.verify.cpp
  llvm-libc++-shared.cfg.in :: std/localization/locales/locale.convenience/conversions/conversions.string/depr.verify.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.cal/time.cal.month/time.cal.month.nonmembers/ostream.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.day.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.duration.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.month.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.weekday.pass.cpp
  llvm-libc++-shared.cfg.in :: std/time/time.syn/formatter.year.pass.cpp

https://reviews.llvm.org/harbormaster/unit/203630/

Looking much better, one or two more iterations on XFAILs and should be good to go.

With remaining failing tests marked with XFAIL we report no failures, although the test still returns error:

...
Testing Time: 2034.53s
  Unsupported        :  261
  Passed             : 7558
  Expectedly Failed  :   64
  Unexpectedly Passed:   26
FAILED: libcxx/test/CMakeFiles/check-cxx /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/libcxx/test/CMakeFiles/check-cxx
cd /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/libcxx/test && /usr/local/bin/python3 /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/bin/llvm-lit -sv --show-unsupported --xunit-xml-output test-results.xml --timeout=1500 --time-tests /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/libcxx/test
ninja: build stopped: subcommand failed.
🚨 Error: The command exited with status 1
user command error: exit status 1

Do the unexpected passes cause the run to fail?

With that addressed failure is now

+ check-abi-list
+ echo '+++ Running the libc++ ABI list test'
Running the libc++ ABI list test
+ ninja -vC /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b check-cxx-abilist
ninja: Entering directory `/usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b'
ninja: error: unknown target 'check-cxx-abilist'

With that addressed failure is now

+ check-abi-list
+ echo '+++ Running the libc++ ABI list test'
Running the libc++ ABI list test
+ ninja -vC /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b check-cxx-abilist
ninja: Entering directory `/usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b'
ninja: error: unknown target 'check-cxx-abilist'

I had a look and there are two issues

  1. check-cxx-abilist requires and ABI list to be available. This can be generated with building the target generate-cxx-abilist. This should generate an abilist that should be included in libc++.
  2. the target generate-cxx-abilist fails
Traceback (most recent call last):

  File "/root/.buildkite-agent/builds/Lezard-Runner-2/llvm-project/libcxx-ci/libcxx/utils/generate_abi_list.py", line 40, in <module>

    main(sys.argv[1:])

  File "/root/.buildkite-agent/builds/Lezard-Runner-2/llvm-project/libcxx-ci/libcxx/utils/generate_abi_list.py", line 33, in main

    symbols = libcxx.sym_check.extract.extract_symbols(args.library)

  File "/root/.buildkite-agent/builds/Lezard-Runner-2/llvm-project/libcxx-ci/libcxx/utils/libcxx/sym_check/extract.py", line 294, in extract_symbols

    return extractor.extract(lib_file)

  File "/root/.buildkite-agent/builds/Lezard-Runner-2/llvm-project/libcxx-ci/libcxx/utils/libcxx/sym_check/extract.py", line 145, in extract

    dyn_syms = self.get_dynsym_table(out)

  File "/root/.buildkite-agent/builds/Lezard-Runner-2/llvm-project/libcxx-ci/libcxx/utils/libcxx/sym_check/extract.py", line 182, in get_dynsym_table

    assert start != -1

This needs modifications for FreeBSD in the file utils/libcxx/sym_check/extract.py. I think it makes sense to do that change in a separate review. Which modifications are required is outside my area of expertise.

Hrmph, GNU readelf produces:

Symbol table '.dynsym' contains 2256 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
...

llvm-readelf agrees:

Symbol table '.dynsym' contains 2256 entries:
   Num:    Value          Size Type    Bind   Vis       Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   UND

while ELF Tool Chain readelf has slightly different output:

Symbol table (.dynsym) contains 2256 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND

Proposed fix in D140313

abi list added in D140431

Thanks for picking this up, it's great to see progress being made.

libcxx/include/__config
134 ↗(On Diff #440185)

I don't understand the purpose of this change either TBH. If FreeBSD doesn't support the old ABI for pair, I think it should just be removed, so this macro can go away entirely?

With (quite) a few hacks I finally managed to get a green FreeBSD run
https://buildkite.com/llvm-project/libcxx-ci/builds/17075#01859d7d-2444-4152-a2e6-a4e41c3912fc

There were a couple of tests that produced inconsistent results, so to make progress on the CI infrastructure I just skipped them for now.

As my next step I'll curate the patches and hacks and submit individual reviews for those changes.

emaste added inline comments.Jan 12 2023, 1:51 PM
libcxx/include/__config
134 ↗(On Diff #440185)

Without this, test fails:

# command stderr:
In file included from /usr/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/test/libcxx/utilities/utility/pairs/pairs.pair/non_trivial_copy_move_ABI.pass.cpp:21:
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/utility:240:
In file included from /home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__assert:13:
/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/__config:156:15: error: '_LIBCPP_DEPRECATED_ABI_DISABLE_PAIR_TRIVIAL_COPY_CTOR' macro redefined [-Werror,-Wmacro-redefined]
#      define _LIBCPP_DEPRECATED_ABI_DISABLE_PAIR_TRIVIAL_COPY_CTOR
              ^
<command line>:4:9: note: previous definition is here
#define _LIBCPP_DEPRECATED_ABI_DISABLE_PAIR_TRIVIAL_COPY_CTOR 1
        ^
1 error generated.

error: command failed with exit status: 1
philnik added inline comments.Jan 12 2023, 1:55 PM
libcxx/include/__config
134 ↗(On Diff #440185)

Let's add //ADDITIONAL_COMPILE_FLAGS: -Wno-macro-redefined to non_trivial_copy_move_ABI.pass.cpp then. This shouldn't be defined by a user normally.

emaste commandeered this revision.Jan 16 2023, 8:23 AM
emaste updated this revision to Diff 489570.
emaste edited reviewers, added: ldionne; removed: emaste.

Update & rebase after committing a number of related changes, and limit this review to the CI portion only

  • XFAIL adjustments are in D141542
  • I've temporarily removed the clang-tidy bits in my local tree as there is an issue that needs to be investigated/resolved
  • A number of tests either had inconsistent results or timeouts and I have excluded them temporarily
    • atomic_wait.pass.cpp
    • atomic_wait_explicit.pass.cpp
    • atomic_notify_one.pass.cpp
    • thread.lock.shared.locking/lock.pass.cpp
emaste marked an inline comment as done.Jan 20 2023, 8:47 AM

Everything should be ready now, test run in progress at https://buildkite.com/llvm-project/libcxx-ci/builds/17740

Mordante accepted this revision.Jan 22 2023, 6:57 AM

Thanks a lot for working on this, LGTM!
I don't have approval rights, for the ABI group, @ldionne, can you approve that?

Before you land this, can you make sure you rebase and give the CI another test run?

I've been using D140050 as a scratch-pad to iterate on testing (in an attempt to avoid spamming everyone on the subscriber list here) and will keep it updated regularly until this lands. I did submit many runs with an additional change that removed the other, non-FreeBSD CI targets to speed iteration and avoid unnecessary CI cycles.

Once this is in I will have to keep an eye on the load on the CI runner to make sure it's sufficient for the volume of CI runs.

I've been using D140050 as a scratch-pad to iterate on testing (in an attempt to avoid spamming everyone on the subscriber list here) and will keep it updated regularly until this lands. I did submit many runs with an additional change that removed the other, non-FreeBSD CI targets to speed iteration and avoid unnecessary CI cycles.

Thanks. I just wanted to make sure the test was up-to-date. I noticed some of the std::format tests fail and I added several new parts of std::format last week.

Once this is in I will have to keep an eye on the load on the CI runner to make sure it's sufficient for the volume of CI runs.

Great! I already had a look and the last run took 39 minutes which is slightly more than Apple at 33 minutes. However there are more Apple jobs than runners. So I expect FreeBSD not to be the slowest runner.

ldionne accepted this revision.Jan 24 2023, 9:16 AM

Can you rebase onto main? I don't think you had D141294 in your tree.

LGTM once this is green. Thanks a bunch for the work, I am really glad to see this finally working and will be happy to advertise official support for FreeBSD.

This revision is now accepted and ready to land.Jan 24 2023, 9:16 AM
emaste added a comment.EditedJan 24 2023, 9:41 AM

Can you rebase onto main? I don't think you had D141294 in your tree.

I ran with a stripped-down version of D141294 before it landed, and my more recent D140050 runs were done after it landed. (Recent runs were done with only this change.)

One more FreeBSD libc++ change is in D142422

emaste added a comment.Mar 1 2023, 1:57 PM

I was away for a bit, and will get back to this. I've pushed a rebased test version to D140050 to check for any new regressions, and if it looks good I will push this.

emaste added a comment.Mar 2 2023, 6:02 AM

It's currently failing in c++2b before getting to my job:

/home/libcxx-builder/.buildkite-agent/builds/0306c2d9b628-1/llvm-project/libcxx-ci/build/generic-cxx2b/include/c++/v1/bitset:396:63: error: shift count >= width of type [-Werror,-Wshift-count-overflow]
        __r |= static_cast<unsigned long long>(__first_[__i]) << (sizeof(__storage_type) * CHAR_BIT);
                                                              ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I have one new XFAIL to add, in D145120, and one XFAIL to be removed from libcxx/test/libcxx/modules_include.sh.cpp.

emaste added a comment.Mar 2 2023, 9:47 AM

Seeing an occasional timeout in libcxx/test/std/atomics/atomics.types.operations/atomics.types.operations.wait/atomic_wait_explicit.pass.cpp or other atomic tests

emaste updated this revision to Diff 503471.Mar 8 2023, 12:18 PM

rebase

XFAIL removal in libcxx/test/libcxx/modules_include.sh.cpp is a separate commit

This revision was landed with ongoing or failed builds.Mar 9 2023, 10:03 AM
This revision was automatically updated to reflect the committed changes.

Thanks, really nice to see we now have a FreeBSD runner!