This is an archive of the discontinued LLVM Phabricator instance.

[libc++] Option to specify a different compiler for testing
AbandonedPublic

Authored by Jake-Egan on Oct 31 2022, 1:32 PM.

Details

Reviewers
hubert.reinterpretcast
daltenty
francii
ldionne
Group Reviewers
Restricted Project
Restricted Project
Restricted Project
Summary

This patch gives the option to change the Lit test compiler by setting LIBCXX_TEST_COMPILER. Defaults to CMAKE_CXX_COMPILER if not specified.

Diff Detail

Event Timeline

Jake-Egan created this revision.Oct 31 2022, 1:32 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptOct 31 2022, 1:32 PM
Herald added a reviewer: Restricted Project. · View Herald Transcript
Jake-Egan requested review of this revision.Oct 31 2022, 1:32 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptOct 31 2022, 1:32 PM
Herald added a reviewer: Restricted Project. · View Herald Transcript
Herald added a reviewer: Restricted Project. · View Herald Transcript
Jake-Egan edited the summary of this revision. (Show Details)Oct 31 2022, 1:33 PM
ldionne requested changes to this revision.Nov 1 2022, 9:35 AM
ldionne added a subscriber: ldionne.

There are a few options here (including re-adding the old cxx_under_test lit param (perhaps under another name)). But first, I'd like to understand why you're not setting CMAKE_CXX_COMPILER to the compiler you're trying to run the tests with?

Requesting changes for review queue status.

This revision now requires changes to proceed.Nov 1 2022, 9:35 AM

But first, I'd like to understand why you're not setting CMAKE_CXX_COMPILER to the compiler you're trying to run the tests with?

Is my understanding correct that changing CMAKE_CXX_COMPILER also changes the build compiler for the shared libraries?
The intention is to build using a "stable" compiler, but still allow testing with a compiler under development.

phosek added a subscriber: phosek.Nov 1 2022, 10:49 AM

But first, I'd like to understand why you're not setting CMAKE_CXX_COMPILER to the compiler you're trying to run the tests with?

Is my understanding correct that changing CMAKE_CXX_COMPILER also changes the build compiler for the shared libraries?
The intention is to build using a "stable" compiler, but still allow testing with a compiler under development.

What's the use case? I find this potentially problematic. What if the compiler specified for running tests with doesn't support the same set of flags as the one used for building?

What's the use case? I find this potentially problematic. What if the compiler specified for running tests with doesn't support the same set of flags as the one used for building?

The use case is to test a newer version of the compiler that is used to build. So the compiler specified for running tests should support at least the flags as the one used for building.

It looks like cxx_under_test would work for our use case. Is this option preferred?

If we'd like to go that route, I could update this patch to re-add cxx_under_test (or other name) as a lit parameter.

philnik added a subscriber: philnik.Nov 8 2022, 3:16 PM

But first, I'd like to understand why you're not setting CMAKE_CXX_COMPILER to the compiler you're trying to run the tests with?

Is my understanding correct that changing CMAKE_CXX_COMPILER also changes the build compiler for the shared libraries?
The intention is to build using a "stable" compiler, but still allow testing with a compiler under development.

Why can't you compile the library with the experimental compiler? If it can't properly compile that, it surely can't run the test suite.

I think the preferred way forward here would be to make the libc++ test suite usable on its own, outside of LLVM. And then, you could use CMAKE_CXX_COMPILER to configure it, but you'd only be configuring the test suite, not the build of libc++ itself.

I think that would be extremely useful to other implementations as well, like libstdc++ and MSVC who want to run our test suite but don't care about building our runtimes.

But first, I'd like to understand why you're not setting CMAKE_CXX_COMPILER to the compiler you're trying to run the tests with?

Is my understanding correct that changing CMAKE_CXX_COMPILER also changes the build compiler for the shared libraries?
The intention is to build using a "stable" compiler, but still allow testing with a compiler under development.

Why can't you compile the library with the experimental compiler? If it can't properly compile that, it surely can't run the test suite.

That's simply not the same test though. In practice the same built shared library + headers can be consumed by multiple different compilers (which may encounter unique issues) when deployed. We'd like a hook to test such scenarios.

I think the preferred way forward here would be to make the libc++ test suite usable on its own, outside of LLVM. And then, you could use CMAKE_CXX_COMPILER to configure it, but you'd only be configuring the test suite, not the build of libc++ itself.

I think that would be extremely useful to other implementations as well, like libstdc++ and MSVC who want to run our test suite but don't care about building our runtimes.

Would we accomplish that by means of some flags we pass to the CMake config then? Essentially some options to say, skip building the library, just stand up the test hardness (maybe with an option to control whether the libc++ specific or just standard test are included). Very roughly, that would make configuring and invoking it something like this (assuming the library isn't the compiler search paths) IIUC:

$ cmake  -DCMAKE_CXX_COMPILER=/some/c++  -DLLVM_ENABLE_RUNTIMES=libcxx -DLIBCXX_TEST_ONLY=ON  -DLIBCXX_LIBRARY_DIR=/path/to/build/lib -DLIBCXX_HEADER_DIR=/path/to/build/include/c++/v1 -DLIBCXX_TEST_CONFIG="appropriate.cfg.in"
$ make check-cxx

I'm not entirely sure what the best way would be to set things up to run the testsuite on its own, but I've been doing that manually, for running individual tests with third party C++ library implementations.

So far, I've made it by first taking a regular build+test setup, then making a copy of the lit site configs and llvm-lit scripts and modifying them to suit my use case. In my case, I test with the default C++ library shipped with a compiler, so I can mostly rip out most of what's set in %{compile_flags} and %{link_flags} (which serves to make the compiler use the just-built libc++ instead of the default one shipped with the compiler), but if you want to test specifically the just-built libc++ with a different compiler, you probably need fewer changes.

Anyway, my setup, for running libc++ tests against MSVC STL, looks like this:
A frontend shell script:

#!/bin/sh
# export PATH=/path/to/my/external/compiler:$PATH
export LIBCXX_SITE_CONFIG=$(cd $(dirname $0) && pwd)/msvc-lit.site.cfg
python3 $(dirname $0)/custom-lit "$@"

A custom copy of the llvm-lit script, named custom-lit: https://martin.st/temp/custom-lit
A handwritten site config, msvc-lit.site.cfg: https://martin.st/temp/msvc-lit.site.cfg
(The latter is mostly a hand-merged/set up version of cmake-bridge.cfg.in and one of the regular config files.)

Similarly I've got a set up for running tests against libstdc++.

Jake-Egan abandoned this revision.Dec 30 2022, 6:09 AM