cmake: Add CLANG_GOLD_LIBDIR_SUFFIX to specify loc of LLVMgold.so

[mgorny]

Summary

Add CLANG_GOLD_LIBDIR_SUFFIX that defaults to ${LLVM_LIBDIR_SUFFIX}
and can be overriden if LLVMgold.so is installed elsewhere. The use case
are multilib systems where binutils is 64-bit and clang is 32-bit,
therefore the gold plugin is installed in 64-bit libdir while clang
is not.

Bug: https://llvm.org/bugs/show_bug.cgi?id=23793
Depends on: https://reviews.llvm.org/D23752

[mgorny]

@rafael, ping. Could you review this, please? This is the solution you suggested on the bug.

[mgorny]

@beanz, could you also look at this one? I'd like to replace CLANG_LIBDIR_SUFFIX with the runtimes suffix, and for this I'd have to get rid of this CLANG_LIBDIR_SUFFIX occurrence as well. However, I don't think LLVMgold.so really counts as a 'runtime', so I guess a default of LLVM_LIBDIR_SUFFIX with possible explicit override would work here.

[beanz]

@mgorny, I don't think LLVMgold qualifies as a runtime in the traditional sense. It more closely aligns with the tools vended by LLVM even though it is a shared library not an executable.

Runtime libraries are specifically libraries that products of clang are linked against (builtins, sanitizers, libcxx, libunwind...).

That bit of semantics aside, I don't think there is any situation where LLVMgold's libdir suffix would be different from LLVM's libdir suffix. Since LLVMgold is built and installed as part of LLVM, there is no mechanism (nor do I think there should be) to cause such a differentiation.

@rafael may disagree, and I will defer to his judgment if he does. My take is that we should just use LLVM_LIBDIR_SUFFIX for LLVMgold, and not have a separate variable.

[mgorny]

In D23754#580268, @beanz wrote:

@mgorny, I don't think LLVMgold qualifies as a runtime in the traditional sense. It more closely aligns with the tools vended by LLVM even though it is a shared library not an executable.

Runtime libraries are specifically libraries that products of clang are linked against (builtins, sanitizers, libcxx, libunwind...).

That bit of semantics aside, I don't think there is any situation where LLVMgold's libdir suffix would be different from LLVM's libdir suffix. Since LLVMgold is built and installed as part of LLVM, there is no mechanism (nor do I think there should be) to cause such a differentiation.

@rafael may disagree, and I will defer to his judgment if he does. My take is that we should just use LLVM_LIBDIR_SUFFIX for LLVMgold, and not have a separate variable.

The difference is that LLVMgold.so is not used by LLVM or clang directly but by the system binutils, and so it must match the ABI of the linker. If you have 32-bit libclang*, LLVM_LIBDIR_SUFFIX is 32 but ld is still a 64-bit executable that needs 64-bit LLVMgold.so.

[chandlerc]

In D23754#580277, @mgorny wrote:

In D23754#580268, @beanz wrote:

@mgorny, I don't think LLVMgold qualifies as a runtime in the traditional sense. It more closely aligns with the tools vended by LLVM even though it is a shared library not an executable.

Runtime libraries are specifically libraries that products of clang are linked against (builtins, sanitizers, libcxx, libunwind...).

That bit of semantics aside, I don't think there is any situation where LLVMgold's libdir suffix would be different from LLVM's libdir suffix. Since LLVMgold is built and installed as part of LLVM, there is no mechanism (nor do I think there should be) to cause such a differentiation.

@rafael may disagree, and I will defer to his judgment if he does. My take is that we should just use LLVM_LIBDIR_SUFFIX for LLVMgold, and not have a separate variable.

The difference is that LLVMgold.so is not used by LLVM or clang directly but by the system binutils, and so it must match the ABI of the linker. If you have 32-bit libclang*, LLVM_LIBDIR_SUFFIX is 32 but ld is still a 64-bit executable that needs 64-bit LLVMgold.so.

But how does that work? The LLVM libraries linked into LLVMgold.so can't be 32-bit if it is loaded into a 64-bit binutils.

[mgorny]

In D23754#580441, @chandlerc wrote:

In D23754#580277, @mgorny wrote:

The difference is that LLVMgold.so is not used by LLVM or clang directly but by the system binutils, and so it must match the ABI of the linker. If you have 32-bit libclang*, LLVM_LIBDIR_SUFFIX is 32 but ld is still a 64-bit executable that needs 64-bit LLVMgold.so.

But how does that work? The LLVM libraries linked into LLVMgold.so can't be 32-bit if it is loaded into a 64-bit binutils.

Just to be clear, the suffix I'm talking about is used only in Driver to specify location of LLVMgold.so. The file is a part of a separate 64-bit install of LLVM (+ clang) which obviously has all the necessary 64-bit libraries.

[chandlerc]

In D23754#580612, @mgorny wrote:

In D23754#580441, @chandlerc wrote:

In D23754#580277, @mgorny wrote:

The difference is that LLVMgold.so is not used by LLVM or clang directly but by the system binutils, and so it must match the ABI of the linker. If you have 32-bit libclang*, LLVM_LIBDIR_SUFFIX is 32 but ld is still a 64-bit executable that needs 64-bit LLVMgold.so.

But how does that work? The LLVM libraries linked into LLVMgold.so can't be 32-bit if it is loaded into a 64-bit binutils.

Just to be clear, the suffix I'm talking about is used only in Driver to specify location of LLVMgold.so. The file is a part of a separate 64-bit install of LLVM (+ clang) which obviously has all the necessary 64-bit libraries.

I'm not sure it really makes sense for the Clang driver to go hunting for an LLVMgold.so from an unrelated build of Clang and LLVM.

The Clang driver is going to run a particular CC1 invocation, ask it to produce bitcode, and then hand that to a linker invocation and give it a plugin to read that bitcode. I think it is reasonable for the only plugin it looks for to be the one installed alongside its lib/clang/... and not one from some other installation in some other directory. How would it even know that this other LLVMgold.so can read the bitcode that CC1 is producing? What if they're different versions? Or support different targets?

[mgorny]

In D23754#580620, @chandlerc wrote:

I'm not sure it really makes sense for the Clang driver to go hunting for an LLVMgold.so from an unrelated build of Clang and LLVM.

The Clang driver is going to run a particular CC1 invocation, ask it to produce bitcode, and then hand that to a linker invocation and give it a plugin to read that bitcode. I think it is reasonable for the only plugin it looks for to be the one installed alongside its lib/clang/... and not one from some other installation in some other directory. How would it even know that this other LLVMgold.so can read the bitcode that CC1 is producing? What if they're different versions? Or support different targets?

Well, this is the builder/distributor's responsibility. If they override this specific suffix, they need to ensure that the value is correct.

That said, I don't mind using the generic runtimes directory approach, that is having LLVMgold.so relative to lib/clang. However, in this case it would probably be reasonable to adjust the install path in LLVM as well (the move of runtimes path to LLVM was requested separately).

Another option is to just pass --plugin=LLVMgold.so and rely on the linker search instead of providing a full path. However, I guess that might break some uncommon installations.

[chandlerc]

In D23754#580631, @mgorny wrote:

In D23754#580620, @chandlerc wrote:

I'm not sure it really makes sense for the Clang driver to go hunting for an LLVMgold.so from an unrelated build of Clang and LLVM.

The Clang driver is going to run a particular CC1 invocation, ask it to produce bitcode, and then hand that to a linker invocation and give it a plugin to read that bitcode. I think it is reasonable for the only plugin it looks for to be the one installed alongside its lib/clang/... and not one from some other installation in some other directory. How would it even know that this other LLVMgold.so can read the bitcode that CC1 is producing? What if they're different versions? Or support different targets?

Well, this is the builder/distributor's responsibility. If they override this specific suffix, they need to ensure that the value is correct.

That said, I don't mind using the generic runtimes directory approach, that is having LLVMgold.so relative to lib/clang. However, in this case it would probably be reasonable to adjust the install path in LLVM as well (the move of runtimes path to LLVM was requested separately).

I think all of these options are just way too much magic to support a very strange use case (IMO). Mixing and matching a 32-bit Clang build with a 64-bit LLVMgold build and expecting magic flags like '-flto' to Just Work seems like too much.

It isn't like this prevents users from directly specifying things. The syntax isn't even that crazy. And that way they'll know what they're getting.

Another option is to just pass --plugin=LLVMgold.so and rely on the linker search instead of providing a full path. However, I guess that might break some uncommon installations.

Sure. Or we could tell users to directly do this and keep the '-flto' behavior maximally simple and predictable.

[mgorny]

Works for me. I don't know if this even has a valid failure case, just felt uneasy about having the obviously-wrong path in code.

CMakeLists.txt

 
 set(CLANG_VENDOR_UTI "org.llvm.clang" CACHE STRING
   "Vendor-specific uti.")
 
 set(CLANG_LIBDIR_SUFFIX "${LLVM_LIBDIR_SUFFIX}" CACHE
   STRING "Define suffix of library directory name for clang runtime (32/64)")
 set(CLANG_LIBRARY_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib${CLANG_LIBDIR_SUFFIX})
 
+set(CLANG_GOLD_LIBDIR_SUFFIX "${LLVM_LIBDIR_SUFFIX}" CACHE
+  STRING "Define suffix of library directory name that contains LLVMgold.so (32/64)")
+
 set(CLANG_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
 set(CLANG_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
 
 if( CMAKE_SOURCE_DIR STREQUAL CMAKE_BINARY_DIR AND NOT MSVC_IDE )
   message(FATAL_ERROR "In-source builds are not allowed. CMake would overwrite "
 "the makefiles distributed with LLVM. Please create a directory and run cmake "
 "from there, passing the path to this source directory as the last argument. "
 "This process created the file `CMakeCache.txt' and the directory "

include/clang/Config/config.h.cmake

 #define CLANG_DEFAULT_RTLIB "${CLANG_DEFAULT_RTLIB}"
 
 /* Default OpenMP runtime used by -fopenmp. */
 #define CLANG_DEFAULT_OPENMP_RUNTIME "${CLANG_DEFAULT_OPENMP_RUNTIME}"
 
 /* Multilib suffix for libdir. */
 #define CLANG_LIBDIR_SUFFIX "${CLANG_LIBDIR_SUFFIX}"
 
+/* Multilib suffix for libdir containing LLVMgold.so. */
+#define CLANG_GOLD_LIBDIR_SUFFIX "${CLANG_GOLD_LIBDIR_SUFFIX}"
+
 /* Relative directory for resource files */
 #define CLANG_RESOURCE_DIR "${CLANG_RESOURCE_DIR}"
 
 /* Directories clang will search for headers */
 #define C_INCLUDE_DIRS "${C_INCLUDE_DIRS}"
 
 /* Default <path> to all compiler invocations for --sysroot=<path>. */
 #define DEFAULT_SYSROOT "${DEFAULT_SYSROOT}"

lib/Driver/Tools.cpp

 
 static void AddGoldPlugin(const ToolChain &ToolChain, const ArgList &Args,
                           ArgStringList &CmdArgs, bool IsThinLTO) {
   // Tell the linker to load the plugin. This has to come before AddLinkerInputs
   // as gold requires -plugin to come before any -plugin-opt that -Wl might
   // forward.
   CmdArgs.push_back("-plugin");
   std::string Plugin =
-      ToolChain.getDriver().Dir + "/../lib" CLANG_LIBDIR_SUFFIX "/LLVMgold.so";
+      ToolChain.getDriver().Dir + "/../lib" CLANG_GOLD_LIBDIR_SUFFIX "/LLVMgold.so";
   CmdArgs.push_back(Args.MakeArgString(Plugin));
 
   // Try to pass driver level flags relevant to LTO code generation down to
   // the plugin.
 
   // Handle flags for selecting CPU variants.
   std::string CPU = getCPUName(Args, ToolChain.getTriple());
   if (!CPU.empty())