Add support to --gcc-toolchain flag for GCC compiled with --enable-version-specific-runtime-libs.

[rpenacob]

Summary

GCC's compiled with --enable-version-specific-runtime-libs. change the paths where includes and libs are found.
This patch adds support for these cases

[MaskRay]

Can you dump the gcc search paths (include search paths and -L library search paths) with and without --enable-version-specific-runtime-libs, like the following?

# This is Debian multiarch gcc. See https://maskray.me/blog/2021-03-28-compiler-driver-and-cross-compilation
% gcc a.c -v -xc /dev/null |& sed -E 's/ "?-[iIL]/\n&/
...
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/11/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
...
 -L/usr/lib/gcc/x86_64-linux-gnu/11
 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu
 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib
 -L/lib/x86_64-linux-gnu
 -L/lib/../lib
 -L/usr/lib/x86_64-linux-gnu
 -L/usr/lib/../lib
 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. /tmp/cc7IN3uz.o ...

For driver tests, it is recommended to test -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DCLANG_DEFAULT_UNWINDLIB=libunwind -DCLANG_DEFAULT_RTLIB=compiler-rt beside the default build to ensure --stdlib= and --rtlib are correctly set.

clang/test/Driver/gcc-toolchain-rt-libs-multi.cpp

2

No need for ( ... ) | .... Just use plain ... 2>&1 | FileCheck ...

7

Consider the form in linux-cross.cpp.

Avoid testing clang. The extra information is not needed and causes trouble to Windows and some build environments where %clang resolves to a file not named clang.

18

no trailing blank line

[rpenacob]

Sure!

GCC
With the --enable-version-specific-runtime-libs

#include "..." search starts here:
#include <...> search starts here:
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed
 /usr/local/include
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/../../include
 /usr/include
...
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc
-L/lib/../lib64
-L/usr/lib/../lib64
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../..

Without it

#include "..." search starts here:
#include <...> search starts here:
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed
 /usr/local/include
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/../../include
 /usr/include
...
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../lib64
-L/lib/../lib64
-L/usr/lib/../lib64
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../..

G++
With the --enable-version-specific-runtime-libs

#include "..." search starts here:
#include <...> search starts here:
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/c++
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/c++/x86_64-pc-linux-gnu
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/c++/backward
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed
 /usr/local/include
 /home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/../../include
 /usr/include
...
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc
-L/lib/../lib64
-L/usr/lib/../lib64
-L/home/rpenacob/.../gcc-rt-libs/install-rt/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../..

Without it

#include "..." search starts here:
#include <...> search starts here:
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../include/c++/11.2.0
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../include/c++/11.2.0/x86_64-pc-linux-gnu
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../include/c++/11.2.0/backward
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed
 /usr/local/include
 /home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/../../include
 /usr/include
...
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../lib64
-L/lib/../lib64
-L/usr/lib/../lib64
-L/home/rpenacob/.../gcc-rt-libs/install/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../..

[MaskRay]

Thanks for the output. Seems like addLibStdCXXIncludePaths is needed, but the addPathIfExists library path seems unnecessary?

[rpenacob]

That is an interesting case. It looks like we're hitting https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32415 , so gcc doesn't seem to search for its own libgcc_s.so when configured with --enable-version-specific-runtime-libs

Most users don't notice this issue because a system-wide libgcc_s.so is usually found instead.

The gcc bug manifests like this:

With --enable-version-specific-runtime-libs
$ <GCC_TOOLCHAIN>/bin/g++ -print-file-name=libgcc_s.so
/lib/../lib64/libgcc_s.so

Without --enable-version-specific-runtime-libs
$ <GCC_TOOLCHAIN>/bin/g++ -print-file-name=libgcc_s.so
<GCC_TOOLCHAIN>/bin/../lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../lib64/libgcc_s.so

However clang doesn't get to find the system wide library, so this is why we need an extra library path.

[MaskRay]

The library path (lib/gcc/$triple/$libdir) still looks weird and I think I don't understand the GCC rationale well.
If you still think it's the right thing the do, let's do it.

For driver tests involving search paths, it's usually worth checking whether the test passes with -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_UNWINDLIB=libunwind -DCLANG_DEFAULT_CXX_STDLIB=libc++.
Some groups use this configuration and expect the tests to pass in this mode.

[rpenacob]

To test that this change does not break anything we built clang trunk with the patch. Our stage one looks like this.

-DCMAKE_INSTALL_PREFIX=/path/to/stage-one
-DCMAKE_BUILD_TYPE=Release
-DLLVM_ENABLE_PROJECTS=clang
-DLLVM_ENABLE_RUNTIMES="libcxx;compiler-rt;libcxxabi;libunwind"
-DCLANG_DEFAULT_RTLIB=compiler-rt
-DCLANG_DEFAULT_UNWINDLIB=libunwind
-DCLANG_DEFAULT_CXX_STDLIB=libc++
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON # We don't have multilib for i386 in this test system
-DLLVM_USE_LINKER=lld

Then we built a stage two using the stage one (-DCMAKE_C_COMPILER and -DCMAKE_CXX_COMPILER pointing to stage-one's clang and clang++) with only -DLLVM_ENABLE_PROJECTS=clang. No test failed.

(We had to set LD_LIBRARY_PATH to /path/to/stage-one/lib/x86_64-unknown-linux-gnu so it finds runtime libs like libc++ and such.)

Do you think this is enough testing? Perhaps there is something else we could try.

clang/lib/Driver/ToolChains/Gnu.cpp

if (const auto &PathsCallback = Multilibs.filePathsCallback())
       for (const auto &Path : PathsCallback(SelectedMultilib))
         addPathIfExists(D, GCCInstallation.getInstallPath() + Path, Paths);
 
     // Add lib/gcc/$triple/$version, with an optional /multilib suffix.
     addPathIfExists(
         D, GCCInstallation.getInstallPath() + SelectedMultilib.gccSuffix(),
         Paths);
 
+    // Add lib/gcc/$triple/$libdir
+    // For GCC built with --enable-version-specific-runtime-libs.
+    addPathIfExists(D, GCCInstallation.getInstallPath() + "/../" + OSLibDir,
+                    Paths);
+
     // GCC cross compiling toolchains will install target libraries which ship
     // as part of the toolchain under <prefix>/<triple>/<libdir> rather than as
     // any part of the GCC installation in
     // <prefix>/<libdir>/gcc/<triple>/<version>. This decision is somewhat
     // debatable, but is the reality today. We need to search this tree even
     // when we have a sysroot somewhere else. It is the responsibility of
     // whomever is doing the cross build targeting a sysroot using a GCC
     // installation that is *not* within the system root to ensure two things:
bool Generic_GCC::addGCCLibStdCxxIncludePaths(
   const GCCVersion &Version = GCCInstallation.getVersion();
 
   // Try /../$triple/include/c++/$version (gcc --print-multiarch is not empty).
   if (addLibStdCXXIncludePaths(
           LibDir.str() + "/../" + TripleStr + "/include/c++/" + Version.Text,
           TripleStr, Multilib.includeSuffix(), DriverArgs, CC1Args))
     return true;
 
+  // Try /gcc/$triple/$version/include/c++/ (gcc --print-multiarch is not
+  // empty). Like above but for GCC built with
+  // --enable-version-specific-runtime-libs.
+  if (addLibStdCXXIncludePaths(LibDir.str() + "/gcc/" + TripleStr + "/" +
+                                   Version.Text + "/include/c++/",
+                               TripleStr, Multilib.includeSuffix(), DriverArgs,
+                               CC1Args))
+    return true;
+
   // Detect Debian g++-multiarch-incdir.diff.
   if (addLibStdCXXIncludePaths(LibDir.str() + "/../include/c++/" + Version.Text,
                                DebianMultiarch, Multilib.includeSuffix(),
                                DriverArgs, CC1Args, /*Debian=*/true))
     return true;
 
   // Try /../include/c++/$version (gcc --print-multiarch is empty).
   if (addLibStdCXXIncludePaths(LibDir.str() + "/../include/c++/" + Version.Text,

clang/test/Driver/gcc-toolchain-rt-libs-multi.cpp

+// RUN: %clangxx %s -### -stdlib=libstdc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs_multilib --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=X64-STDCPLUS
+// RUN: %clangxx %s -### -stdlib=libc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs_multilib --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=X64-LIBCPLUS
+// RUN: %clangxx %s -m32 -### -stdlib=libstdc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs_multilib --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=X32-STDCPLUS
+// RUN: %clangxx %s -m32 -### -stdlib=libc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs_multilib --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=X32-LIBCPLUS
+
+int main() {}
+
+// X64-STDCPLUS: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// X64-STDCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0"
+// X64-STDCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../lib64"
+
+// X64-LIBCPLUS-NOT: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// X64-LIBCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0"
+// X64-LIBCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../lib64"
+
+// X32-STDCPLUS: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// X32-STDCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/32"
+// X32-STDCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../lib32"
+
+// X32-LIBCPLUS-NOT: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// X32-LIBCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/32"
+// X32-LIBCPLUS: "-L{{[^ ]*}}gcc_version_parsing_rt_libs_multilib/lib/gcc/x86_64-redhat-linux/10.2.0/../lib32"

clang/test/Driver/gcc-toolchain-rt-libs.cpp

+// RUN: %clangxx %s -### -stdlib=libstdc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=STDCPLUS
+// RUN: %clangxx %s -### -stdlib=libc++ --gcc-toolchain=%S/Inputs/gcc_version_parsing_rt_libs --target=x86_64-redhat-linux 2>&1 | FileCheck %s -check-prefix=LIBCPLUS
+
+int main() {}
+
+// STDCPLUS: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// STDCPLUS: "-L{{.*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0"
+// STDCPLUS: "-L{{.*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0/../lib64"
+
+// LIBCPLUS-NOT: "-internal-isystem" "{{[^ ]*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0/../../../gcc/x86_64-redhat-linux/10.2.0/include/c++/"
+// LIBCPLUS: "-L{{.*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0"
+// LIBCPLUS: "-L{{.*}}gcc_version_parsing_rt_libs/lib/gcc/x86_64-redhat-linux/10.2.0/../lib64"