[LLD] Allow usage of LLD as a library

[aganea]

Summary

As discussed in https://github.com/llvm/llvm-project/issues/53475 this patch allows using LLD-as-a-lib. It also lets clients link-in only the drivers that they want (see tests).

This also adds the unittests infra as in the other LLVM projects. Among the test coverage, I've added the original issue from @krzysz00, see this.

Important note: this doesn't allow (yet) linking in parallel. This will come a bit later, in subsequent patches, for COFF at last.

[mstorsjo]

lld/tools/lld/lld.cpp

127

This is the MSVC specific C++ mangling, and doesn't work as such for the non-MSVC version of LLVM_WEAK_SYMBOL here.

[jerryyin]

May I ask what the state of this CL is @aganea? I bumped into this and think it is still quite useful so we should still shoot for landing it.

[aganea]

In D119049#3830581, @jerryyin wrote:

May I ask what the state of this CL is @aganea? I bumped into this and think it is still quite useful so we should still shoot for landing it.

This was just exploratory. The next thing users would probably want (cargo/rust for ex.) is multi-threaded calls. However the drivers aren't thread-safe currently. Ideally, I wanted to finish that first (and iron our all the perf. regressions due to thread safety that @MaskRay mentioned in the past). But if others feel that we should have library-fication beforehand, I can clean this & add tests and make it ready for review.

[MaskRay]

(Note: the ELF port is very very far away from supporting multiple concurrent links)

[jerryyin]

But if others feel that we should have library-fication beforehand, I can clean this & add tests and make it ready for review.

I would prefer that allowing LLD using as lib to come before multi-threaded support. Appreciate the effort If it isn't too much trouble on your end. The use case I'm encountering is to populate a self sustained MLIR static library that incorporate lld (rather than dependent on a system installed lld). It is completely okay to require only single threaded access as adding a mutex in library call is fairly straightforward.

[jerryyin]

@aganea If you'd be willing to pick this CL again within one/two month-ish timeline, that's great. I can help with testing since I'm a developer from ROCm side. Hopefully we can wrap this whole thing up by the end of this quarter. On the other hand, if this CL may not be your top priority to work on, and just want to let you know that I'd be willing to take over and finish the rest of work of it. But I have close to zero knowledge of the internals of LLD so may need your pointers and code reviews to make sure I'm on the right track.

[aganea]

In D119049#3833491, @jerryyin wrote:

@aganea If you'd be willing to pick this CL again within one/two month-ish timeline, that's great. I can help with testing since I'm a developer from ROCm side. Hopefully we can wrap this whole thing up by the end of this quarter. On the other hand, if this CL may not be your top priority to work on, and just want to let you know that I'd be willing to take over and finish the rest of work of it. But I have close to zero knowledge of the internals of LLD so may need your pointers and code reviews to make sure I'm on the right track.

Sorry for the delay, please see the updated patch.

[aganea]

Fix casing in CMakeLists.txt

lld/tools/lld/lld.cpp

127

Fixed. Using the default OBJ-in-LIB override mechanism now, as described well by Raymond Chen here.

[sbc100]

lld/Common/CMakeLists.txt

30

Can we come up with a better name for this file maybe? That name makes me think its specifically for the library use case but that isn't true.

Also, we don't capitalize lld elsewhere, with in the filenames or directory names.

How about lldMain.cpp, or perhaps we can come up with a name that doesn't start with lld at all?

lld/include/lld/Common/Driver.h

43–44

Why not llvm::ArrayRef<const char *>?

[aganea]

As suggested by @sbc100

lld/Common/CMakeLists.txt

30

DriversMain.cpp ?

lld/include/lld/Common/Driver.h

43–44

Please see updated diff. We had std::vector just because of how the -flavor flag is treated and then stripped from args. I changed to llvm::ArrayRef as you suggested and added -flavor in all drivers as a shallow flag. If we want to go that route, I could send a separate patch for that change.

[jerryyin]

Thanks for your work @aganea. I abstain from giving review from lld's implementation side but thing looks good on the ROCm.cpp unit test. (BTW, do you aim to create that unit test as a demo and then to remove it later? Or else i'd expect to see the "Inputs" folder mocked up in certain way.) Although I did notice a fair amount of test breakages on windows. If you are still actively working on it, I'm willing to approve it once you fix the test failures. Also I plan to give this a test by pulling this diff locally assuming the base of this diff wouldn't cause too much conflicts with our tip of clone.

[aganea]

Fix (hopefully) CI build.

[aganea]

@jerryyin The build errors were due to the fact that I only used LLD as stage-0 linker in my previous tests, assuming that its behavior is conformant with other system linkers. It is not, as described in this paragraph. The same holds for both ELF and COFF drivers.

It seems that there's no universal solution for overriding a symbol in a LIB by a symbol in another LIB. MSVC has /alternatename which works, but there's no such thing on ELF side -- weak symbols cannot be overriden by another LIB once they are visible in the module, or at least I don't know how. There's also the --include=symbol for MSVC and --u=symbol for Unix that can solve the issue by pushing the symbol early in the symbol table, but then we have to deal with the mangling and that doesn't sounds fun. In the end, if users want to use a partial set of the LLD drivers, I resorted to simply implementing the missing drivers with the LLD_IMPLEMENT_SHALLOW_DRIVER macro (see ROCm test). If there's a better/simpler solution for a user perspective, let me know and I'll try it.

[aganea]

Add binaries into diff to fix the preflight build.

[aganea]

hmm the buildkite preflight is still broken, it seems Phabricator is stripping out the binary files from the diff.
My local diff is fine:

but after downloading from this page, they are empty:

[jerryyin]

It seems that there's no universal solution for overriding a symbol in a LIB by a symbol in another LIB...simply implementing the missing drivers with the LLD_IMPLEMENT_SHALLOW_DRIVER macro

Thanks for letting me know, sounds like a fair solution.

lld/Common/DriversMain.cpp

163–172 ↗

(On Diff #491742)

I have a naïve question to ask:

• In my limited knowledge, previously lldELF is dependent on lldCommon target. lld.cpp would compile just fine because its target is an executable.
• Now, since DriversMain.cpp become part of lldCommon, and in the implementation of DriversMain,cpp, we start to invoke those specific link target, it implicitly requires, for example, lld::elf::link to be available before lldCommon is compiled. In other words, lldCommon is dependent on lldELF.

Did we just have a cyclic dependency here? And if it is, do you need to move the lldMain() and safeLldMain() into their own independent target such that we can avoid the existence of such a cycle in the compilation?

[aganea]

lld/Common/DriversMain.cpp

163–172 ↗

(On Diff #491742)

The trickery is that we push lldCommon twice on the command-line, once in lldELF (same in all the other drivers), in another time in the final module. I retained the same mechanic for the unittests.

We could have another lib for LLD-as-lib usage, but I thought from the users' perspective it's simpler if we provided a single "common" lib + the specific drivers' libs.

[arsenm]

lld/CMakeLists.txt

194–199

Feels like this isn't something lld should have to roll its own handling for

lld/Common/DriversMain.cpp

162–172 ↗

(On Diff #491742)

Should move to a separate function with a switch/return

[aganea]

As suggested by @arsenm

[jerryyin]

LGTM with my limited knowledge of lld. Approving it as this is an extremely useful feature to have, and I want it to make progress after almost a year of staging.

lld/Common/DriversMain.cpp

163–172 ↗

(On Diff #491742)

Just want to let you know that I made an (very risky attempt) trying to test this patch in our fork, but this errors out unable to build lld. Based on Harbormaster build result, I don't believe this is due to issues from your patch, but rather our lld fork has been a couple of months old. (Refer to https://github.com/ROCmSoftwarePlatform/rocMLIR/commits/test-lld-update, I pulled the last 3 commits in between this period hoping they are enough narrow the gap.) However, ninja lld would yield a number of linker error as below:

ld.lld: error: undefined symbol: lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool)
referenced by DriversMain.cpp:0 (/root/rocMLIR/external/llvm-project/lld/Common/DriversMain.cpp:0)

external/llvm-project/llvm/tools/lld/Common/CMakeFiles/lldCommon.dir/DriversMain.cpp.o:(lld::lldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool))

It would take a while for our fork to completely catch up with the tip of upstream, and I'll give this another test after then. In the mean time I will not hold this review up because of it.

[aganea]

Thanks for reviewing @jerryyin !
Before going further, I'd like an review/approval from the regular lld contributors as well, whenever you got the chance @mstorsjo @MaskRay @hans @thieta

[MaskRay]

There is a layering issue. lld/Common/DriversMain.cpp references lld::elf::link which is defined in the library lldELF. lldELF depends on lldCommon so lldCommon cannot depend on lldELF.
You shall see a linker error with -DBUILD_SHARED_LIBS=on due to -Wl,-z,defs

[aganea]

As suggested by @MaskRay. This changes a few things, but it's for the best I think:

• Fix circular dependency between lldCommon and the lld drivers.
• Change safeLldMain -> lldMain and make it the only entry point.
• Pass down drivers symbols from the caller to lldCommon. This also makes it easier to discriminate between what drivers are linked in the PE, and what it isn't available.
• Remove all drivers' declarations from Drivers.h, the only usage allowed is to just use lldMain().
• Tested on Linux with -DBUILD_SHARED_LIBS=ON.

[MaskRay]

lld/Common/CMakeLists.txt

30

Keep the file list sorted

lld/Common/DriversMain.cpp

1 ↗

(On Diff #494949)

DriverMain instead of DriversMain? I wonder whether Dispatcher in the filename can make it clearer. We have many functions/files with main or Main in the names now...

Also, the header needs a fix.

lld/tools/lld/lld.cpp

92

This can call unsafeLldMain instead, then the parameter bool libUsage can be removed. The downside is that unsafeLldMain will be in an include header but that's probably fine (unsure whether moving it into a nested namespace to discourage its use is useful).

[MaskRay]

Thanks for the patch! I have gone through the code and it looks in a good shape. Left some suggestions.

lld/Common/DriversMain.cpp

60 ↗

(On Diff #494949)

Hopefully share with other template instantiations to decrease the executable size.

152 ↗

(On Diff #494949)

178 ↗

(On Diff #494949)

lld::lldMain (see https://llvm.org/docs/CodingStandards.html#use-namespace-qualifiers-to-implement-previously-declared-functions)

Fix other places as well.

lld/ELF/Driver.cpp

112

https://llvm.org/docs/CodingStandards.html#use-namespace-qualifiers-to-implement-previously-declared-functions

lld/ELF/Options.td

739 ↗

(On Diff #494949)

We should avoid ignoring -flavor in all drivers. It's probably not difficult to strip -flavor from the driver dispatcher.

lld/MachO/Driver.cpp

1372

https://llvm.org/docs/CodingStandards.html#use-namespace-qualifiers-to-implement-previously-declared-functions

lld/unittests/AsLibAll/AllDrivers.cpp

19

static

lld/unittests/AsLibAll/CMakeLists.txt

2

Add a file-level comment explaining what AsLibAll means.

lld/unittests/AsLibELF/CMakeLists.txt

2

Add a file-level comment explaining what AsLibELF means.

lld/unittests/AsLibELF/ROCm.cpp

9

Add a comment about the purpose.

22

remove braces

29

static

lld/unittests/AsLibELF/SomeDrivers.cpp

13

static

[MaskRay]

lld/ELF/Options.td

739 ↗

(On Diff #494949)

Note: we can assert that -flavor is a very first option, i.e. lld -xxx -yyy -flavor gnu isn't allowed.
This may simplify striping the -flavor option.

[MaskRay]

lld/tools/lld/lld.cpp

71

Latest clang gives warnings in a -DLLD_DEFAULT_LD_LLD_IS_MINGW=on build.

/home/maskray/llvm/lld/tools/lld/lld.cpp:64:21: warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]
LLD_HAS_DRIVER(coff);
                    ^

[MaskRay]

The comment around lld/docs/index.rst:35 needs an update. It still suggests lld::elf::link.

[aganea]

As suggested by @MaskRay

[aganea]

Thanks for the in-depth review @MaskRay, I appreciate!

lld/Common/DriversMain.cpp

60 ↗

(On Diff #494949)

I checked with CruncherSharp, there are already ,256 instances in the PE image.

lld/ELF/Driver.cpp

112

This pattern works only if the symbol is "public", as in, declared in a header that has been seen already by the current TU. Which is not the case here, I've removed declarations for direct drivers calls, only lld::lldMain remains in Drivers.h. Same comment for the other places where this pattern isn't applicable.

lld/MachO/Driver.cpp

1372

It isn't applicable here, since I remove the declarations from Driver.h

lld/tools/lld/lld.cpp

71

Hmm I don't see this with latest patch, would you have a chance to check again if it's still there?

[MaskRay]

lld/Common/Args.cpp

95 ↗

(On Diff #498530)

This code introduces a new function. We can strip -flavor near whichDriver or its caller to move all -flavor handling to a single place and avoid polluting the common code.

lld/Common/DriverDispatcher.cpp

160

delete blank line

163

d(...) returns a bool, not an int. exitLld(r) passes the bool to CrashRecoveryContext::throwIfCrash (isCrash expects an exit code) which seems incorrect.

lld/unittests/AsLibAll/AllDrivers.cpp

17

warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]

lld/unittests/AsLibELF/ROCm.cpp

37

warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]

lld/unittests/AsLibELF/SomeDrivers.cpp

18

warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]

[aganea]

As sugested by @MaskRay

lld/Common/DriverDispatcher.cpp

163

Actually the bool to int conversion is correct. The drivers and thus d returns false on error, true on success. We invert the result and that makes r equal 1 on error, and 0 on success, which is the value that will be returned by the application. throwIfCrash will be really useful when getting back from the CrashRecoveryHandler below in lld::lldMain, but on this particular codepath in unsafeLldMain that will never be hit.
I've added an extra comment to clarify. Please let me know if you think more should be done.

[MaskRay]

Looks great!

lld/tools/lld/lld.cpp

71

The warning goes away! Thanks.

[aganea]

Thank you for the thorough review @MaskRay !

[MaskRay]

This will now need a rebase to fix merge conflicts. This is a very desired improvement:)

A reminder that don't forget to test LLD_DEFAULT_LD_LLD_IS_MINGW

[aganea]

In D119049#4418851, @MaskRay wrote:

This will now need a rebase to fix merge conflicts. This is a very desired improvement:)

A reminder that don't forget to test LLD_DEFAULT_LD_LLD_IS_MINGW

Ah yes, I did test it in the past, but I'll give it a try again. Thanks for the reminder.

[thakis]

lld/test/Unit/lit.site.cfg.py.in

10

Is this read by anything? clang/test/Unit/lit.site.cfg.py doesn't have it for example.

If not: What about the other vars in here? Are they copy-pasta, or are they needed?

[paulkirth]

Hi, I think we're seeing some test failures due to this patch. Would you mind taking a look, and reverting for now if it will be hard to fix?

Failing Test: LLD-Unit :: AsLibELF/./LLDAsLibELFTests/AsLib/ROCm

bot: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8778372807208184913/overview

[srj]

Hi, I'm trying to update Halide's usage of this API to work properly, there are a few things that aren't clear to me:

• It appears that lldMain is now the preferred entry points; I presume that unsafeLldMain and safeLldMain are intended for internal use only?
• It's not clear to me whether use of CrashRecoveryContext with this API is required or optional. (We previously didn't use it, though we did wrap calls to the old entry point with std::signal() usage to save and restore signal handlers.)
• It's not clear to me whether we should use exitLld() or not -- in the event of failure, we always previously exited anyway, but it appears that exitLld() always exits with SIGABRT, which would break the existing contract of our tool (which would exit via exit(1)). Does exitLld() do special cleanup (eg, temp file cleanup) that would be missed otherwise?
  
  Thanks in advance for any guidance you can give here!

[leonardchan]

Hi. This fails for me locally with:

FAIL: LLD-Unit :: AsLibELF/./LLDAsLibELFTests/0/2 (3 of 2771) 
******************** TEST 'LLD-Unit :: AsLibELF/./LLDAsLibELFTests/0/2' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/usr/local/google/home/leonardchan/llvm-monorepo/llvm-build-3-master-fuchsia-toolchain/tools/lld/unittests/AsLibELF/./LLDAsLibELFTests-LLD-Unit-2079095-0-2.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=2 GTEST_SHARD_INDEX=0 /usr/loca
l/google/home/leonardchan/llvm-monorepo/llvm-build-3-master-fuchsia-toolchain/tools/lld/unittests/AsLibELF/./LLDAsLibELFTests
--

Script:
--
/usr/local/google/home/leonardchan/llvm-monorepo/llvm-build-3-master-fuchsia-toolchain/tools/lld/unittests/AsLibELF/./LLDAsLibELFTests --gtest_filter=AsLib.ROCm
--
ld.lld: error: cannot open lld/unittests/AsLibELF/Inputs/kernel1.o: No such file or directory
Failed to link: lld/unittests/AsLibELF/Inputs/kernel1.o
lld/unittests/AsLibELF/ROCm.cpp:64: Failure
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
ld.lld: error: cannot open lld/unittests/AsLibELF/Inputs/kernel2.o: No such file or directory
Failed to link: lld/unittests/AsLibELF/Inputs/kernel2.o
lld/unittests/AsLibELF/ROCm.cpp:65: Failure
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false
Expected: true
ld.lld: error: cannot open lld/unittests/AsLibELF/Inputs/kernel1.o: No such file or directory
Failed to link: lld/unittests/AsLibELF/Inputs/kernel1.o
lld/unittests/AsLibELF/ROCm.cpp:66: Failure
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
ld.lld: error: cannot open lld/unittests/AsLibELF/Inputs/kernel2.o: No such file or directory
Failed to link: lld/unittests/AsLibELF/Inputs/kernel2.o
lld/unittests/AsLibELF/ROCm.cpp:67: Failure
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false
Expected: true

lld/unittests/AsLibELF/ROCm.cpp:64
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
lld/unittests/AsLibELF/ROCm.cpp:65
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false
Expected: true
lld/unittests/AsLibELF/ROCm.cpp:66
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
lld/unittests/AsLibELF/ROCm.cpp:67
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false
Expected: true


********************
********************
Failed Tests (1):
  LLD-Unit :: AsLibELF/./LLDAsLibELFTests/AsLib/ROCm

Looks like lld can't find the object files? Could you send out a fix or revert?

[aganea]

Hello, I’m out this afternoon, is anybody able to disable this test until I can take a look at it please? Otherwise I’ll do it tonight.

[aganea]

I've reproduced (on Linux) the issue mentioned above. This doesn't happen on Windows/MSVC because __FILE__ (used in the ROCm.cpp test) always expands to absolute paths when building LLVM, since -DLLVM_USE_RELATIVE_PATHS_IN_FILES requires -ffile-prefix-map which is unsupported on MSVC-cl/clang-cl. Instead, I've switched to using an environment variable LLD_SRC_DIR (which stores relative paths where applicable), see changes in ROCm.cpp and lld/test/Unit/lit.cfg.py.

I've also incorporated @thakis 's changes (if you don't mind) to make the lld unittest paths relocatable. You would still need D152885 to make lld/test relocatable.

@MaskRay @thakis Would you mind taking another look please?

lld/test/Unit/lit.site.cfg.py.in

10

Removed. It was remnants from a copy-paste (indeed) from MLIR.

[MaskRay]

arc patch D119049
ninja -C /tmp/Rel check-lld-unit

gives me failures.

% grep LLVM_USE_RELATIVE_PATHS_IN_FILES /tmp/Rel/CMakeCache.txt
LLVM_USE_RELATIVE_PATHS_IN_FILES:BOOL=OFF

--
/tmp/Rel/tools/lld/unittests/AsLibELF/./LLDAsLibELFTests --gtest_filter=AsLib.ROCm
--
ld.lld: error: target emulation unknown: -m or at least one .o file required
Failed to link: /home/maskray/llvm/lld/unittests/AsLibELF/Inputs/kernel1.o
/home/maskray/llvm/lld/unittests/AsLibELF/ROCm.cpp:69: Failure
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
ld.lld: error: target emulation unknown: -m or at least one .o file required
Failed to link: /home/maskray/llvm/lld/unittests/AsLibELF/Inputs/kernel2.o
/home/maskray/llvm/lld/unittests/AsLibELF/ROCm.cpp:70: Failure
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false

[aganea]

In D119049#4429502, @MaskRay wrote:

arc patch D119049
ninja -C /tmp/Rel check-lld-unit

gives me failures.

% grep LLVM_USE_RELATIVE_PATHS_IN_FILES /tmp/Rel/CMakeCache.txt
LLVM_USE_RELATIVE_PATHS_IN_FILES:BOOL=OFF

--
/tmp/Rel/tools/lld/unittests/AsLibELF/./LLDAsLibELFTests --gtest_filter=AsLib.ROCm
--
ld.lld: error: target emulation unknown: -m or at least one .o file required
Failed to link: /home/maskray/llvm/lld/unittests/AsLibELF/Inputs/kernel1.o
/home/maskray/llvm/lld/unittests/AsLibELF/ROCm.cpp:69: Failure
Value of: runLinker("%S/Inputs/kernel1.o")
  Actual: false
Expected: true
ld.lld: error: target emulation unknown: -m or at least one .o file required
Failed to link: /home/maskray/llvm/lld/unittests/AsLibELF/Inputs/kernel2.o
/home/maskray/llvm/lld/unittests/AsLibELF/ROCm.cpp:70: Failure
Value of: runLinker("%S/Inputs/kernel2.o")
  Actual: false

The two binary files (.o) are missing from the patch. Hold on, I will post it again.

[aganea]

With binary files.

[MaskRay]

In D119049#4429521, @aganea wrote:

With binary files.

I see that

config.environment["LLD_SRC_DIR"] = config.lld_src_dir

sets the correct path, but ninja check-lld-unit still fails due to thisPath.append(getenv("LLD_SRC_DIR")); getting an empty string...

lld/test/Unit/lit.cfg.py

26

Double quotes to be consistent with https://reviews.llvm.org/D150545

[MaskRay]

Perhaps move the unittests part to a separate patch and land the rest?

[aganea]

With double quotes in lit.cfg.py

[aganea]

In D119049#4429553, @MaskRay wrote:

In D119049#4429521, @aganea wrote:

With binary files.

I see that

config.environment["LLD_SRC_DIR"] = config.lld_src_dir

sets the correct path, but ninja check-lld-unit still fails due to thisPath.append(getenv("LLD_SRC_DIR")); getting an empty string...

Can you post your entire cmake command?

[MaskRay]

In D119049#4429615, @aganea wrote:

In D119049#4429553, @MaskRay wrote:

In D119049#4429521, @aganea wrote:

With binary files.

I see that

config.environment["LLD_SRC_DIR"] = config.lld_src_dir

sets the correct path, but ninja check-lld-unit still fails due to thisPath.append(getenv("LLD_SRC_DIR")); getting an empty string...

Can you post your entire cmake command?

A bit complex and unrelated to the issue...

https://github.com/MaskRay/Config/wiki/LLVM#configuration
configure-llvm preset

I confirm that /tmp/Debug/tools/lld/unittests/AsLibELF/LLDAsLibELFTests does receive LLD_SRC_DIR and kernel1.o can be found.
The problem is that curl -L 'https://reviews.llvm.org/D119049?download=1' doesn't contain kernel1.o or kernel2.o, so others cannot verify.

You may land the patch and disable the unittest if needed...

[aganea]

Hello @srj ! Sorry for the late answer.

In D119049#4421785, @srj wrote:

Hi, I'm trying to update Halide's usage of this API to work properly, there are a few things that aren't clear to me:

• It appears that lldMain is now the preferred entry points; I presume that unsafeLldMain and safeLldMain are intended for internal use only?

Yes.

• It's not clear to me whether use of CrashRecoveryContext with this API is required or optional. (We previously didn't use it, though we did wrap calls to the old entry point with std::signal() usage to save and restore signal handlers.)

The CrashRecoveryContext is mandatory in this context since lldMain goes through safeLldMain which also happens to clean the context before the function exists (which is not what regular lld.exe does, it simply leaks all buffers for faster exit). My thinking is that lldMain will be used in long-running applications like a daemon. If you have suggestions/improvements for the API, let me know.

• It's not clear to me whether we should use exitLld() or not -- in the event of failure, we always previously exited anyway, but it appears that exitLld() always exits with SIGABRT, which would break the existing contract of our tool (which would exit via exit(1)). Does exitLld() do special cleanup (eg, temp file cleanup) that would be missed otherwise?

The SIGABRT you're seeing is simply because exitLld() forwards the crash code through this: https://github.com/llvm/llvm-project/blob/main/lld/Common/ErrorHandler.cpp#L100 - you can set r.retCode = 1; in your code before calling exitLld() to avoid this situation.

[thakis]

lld/unittests/AsLibELF/ROCm.cpp

34

From what I can tell, this is a reinvention of add_llvm_unittest_with_input_files() which is discouraged because it entangle build and source data. The recommendation is to have a lit test for things that need to read in-tree things. Unit tests shouldn't affect the source directory.

lld/CMakeLists.txt

install(DIRECTORY include/
     PATTERN "*.h"
     )
 endif()
 
 add_subdirectory(Common)
 add_subdirectory(tools/lld)
 
 if (LLVM_INCLUDE_TESTS)
+  add_custom_target(LLDUnitTests)
+  llvm_add_unittests(LLD_UNITTESTS_ADDED)
   add_subdirectory(test)
 endif()
 
 add_subdirectory(docs)
 add_subdirectory(COFF)
 add_subdirectory(ELF)
 add_subdirectory(MachO)
 add_subdirectory(MinGW)
 add_subdirectory(wasm)
 
 add_subdirectory(cmake/modules)

lld/COFF/Driver.cpp

 using namespace llvm::object;
 using namespace llvm::COFF;
 using namespace llvm::sys;
 
 namespace lld::coff {
 
 bool link(ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
           llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput) {
-  // This driver-specific context will be freed later by lldMain().
+  // This driver-specific context will be freed later by unsafeLldMain().
   auto *ctx = new COFFLinkerContext;
 
   ctx->e.initialize(stdoutOS, stderrOS, exitEarly, disableOutput);
   ctx->e.logName = args::getFilenameWithoutExe(args[0]);
   ctx->e.errorLimitExceededMsg = "too many errors emitted, stopping now"
                                  " (use /errorlimit:0 to see all errors)";
 
   ctx->driver.linkerMain(args);

lld/Common/CMakeLists.txt

 # Mark the generated header as being generated.
 set_source_files_properties("${version_inc}"
   PROPERTIES GENERATED TRUE
   HEADER_FILE_ONLY TRUE)
 
 add_lld_library(lldCommon
   Args.cpp
   CommonLinkerContext.cpp
+  DriverDispatcher.cpp
   DWARF.cpp
   ErrorHandler.cpp
   Filesystem.cpp
   Memory.cpp
   Reproduce.cpp
   Strings.cpp
   TargetOptionsCommandFlags.cpp
   Timer.cpp
   VCSVersion.inc
   Version.cpp
 
   ADDITIONAL_HEADER_DIRS

lld/Common/DriverDispatcher.cpp

+//===- DriverDispatcher.cpp - Support using LLD as a library --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lld/Common/CommonLinkerContext.h"
+#include "lld/Common/Driver.h"
+#include "lld/Common/ErrorHandler.h"
+#include "lld/Common/Memory.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringSwitch.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
+#include "llvm/TargetParser/Host.h"
+#include "llvm/TargetParser/Triple.h"
+#include <cstdlib>
+
+using namespace lld;
+using namespace llvm;
+using namespace llvm::sys;
+
+static void err(const Twine &s) { llvm::errs() << s << "\n"; }
+
+static Flavor getFlavor(StringRef s) {
+  return StringSwitch<Flavor>(s)
+      .CasesLower("ld", "ld.lld", "gnu", Gnu)
+      .CasesLower("wasm", "ld-wasm", Wasm)
+      .CaseLower("link", WinLink)
+      .CasesLower("ld64", "ld64.lld", "darwin", Darwin)
+      .Default(Invalid);
+}
+
+static cl::TokenizerCallback getDefaultQuotingStyle() {
+  if (Triple(sys::getProcessTriple()).getOS() == Triple::Win32)
+    return cl::TokenizeWindowsCommandLine;
+  return cl::TokenizeGNUCommandLine;
+}
+
+static bool isPETargetName(StringRef s) {
+  return s == "i386pe" || s == "i386pep" || s == "thumb2pe" || s == "arm64pe";
+}
+
+static std::optional<bool> isPETarget(llvm::ArrayRef<const char *> args) {
+  for (auto it = args.begin(); it + 1 != args.end(); ++it) {
+    if (StringRef(*it) != "-m")
+      continue;
+    return isPETargetName(*(it + 1));
+  }
+
+  // Expand response files (arguments in the form of @<filename>)
+  // to allow detecting the -m argument from arguments in them.
+  SmallVector<const char *, 256> expandedArgs(args.data(),
+                                              args.data() + args.size());
+  BumpPtrAllocator a;
+  StringSaver saver(a);
+  cl::ExpansionContext ectx(saver.getAllocator(), getDefaultQuotingStyle());
+  if (Error e = ectx.expandResponseFiles(expandedArgs)) {
+    err(toString(std::move(e)));
+    return std::nullopt;
+  }
+
+  for (auto it = expandedArgs.begin(); it + 1 != expandedArgs.end(); ++it) {
+    if (StringRef(*it) != "-m")
+      continue;
+    return isPETargetName(*(it + 1));
+  }
+
+#ifdef LLD_DEFAULT_LD_LLD_IS_MINGW
+  return true;
+#else
+  return false;
+#endif
+}
+
+static Flavor parseProgname(StringRef progname) {
+  // Use GNU driver for "ld" by default.
+  if (progname == "ld")
+    return Gnu;
+
+  // Progname may be something like "lld-gnu". Parse it.
+  SmallVector<StringRef, 3> v;
+  progname.split(v, "-");
+  for (StringRef s : v)
+    if (Flavor f = getFlavor(s))
+      return f;
+  return Invalid;
+}
+
+static Flavor
+parseFlavorWithoutMinGW(llvm::SmallVectorImpl<const char *> &argsV) {
+  // Parse -flavor option.
+  if (argsV.size() > 1 && argsV[1] == StringRef("-flavor")) {
+    if (argsV.size() <= 2) {
+      err("missing arg value for '-flavor'");
+      return Invalid;
+    }
+    Flavor f = getFlavor(argsV[2]);
+    if (f == Invalid) {
+      err("Unknown flavor: " + StringRef(argsV[2]));
+      return Invalid;
+    }
+    argsV.erase(argsV.begin() + 1, argsV.begin() + 3);
+    return f;
+  }
+
+  // Deduct the flavor from argv[0].
+  StringRef arg0 = path::filename(argsV[0]);
+  if (arg0.ends_with_insensitive(".exe"))
+    arg0 = arg0.drop_back(4);
+  Flavor f = parseProgname(arg0);
+  if (f == Invalid) {
+    err("lld is a generic driver.\n"
+        "Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld"
+        " (WebAssembly) instead");
+    return Invalid;
+  }
+  return f;
+}
+
+static Flavor parseFlavor(llvm::SmallVectorImpl<const char *> &argsV) {
+  Flavor f = parseFlavorWithoutMinGW(argsV);
+  if (f == Gnu) {
+    auto isPE = isPETarget(argsV);
+    if (!isPE)
+      return Invalid;
+    if (*isPE)
+      return MinGW;
+  }
+  return f;
+}
+
+static Driver whichDriver(llvm::SmallVectorImpl<const char *> &argsV,
+                          llvm::ArrayRef<DriverDef> drivers) {
+  Flavor f = parseFlavor(argsV);
+  auto it =
+      llvm::find_if(drivers, [=](auto &driverdef) { return driverdef.f == f; });
+  if (it == drivers.end()) {
+    // Driver is invalid or not available in this build.
+    return [](llvm::ArrayRef<const char *>, llvm::raw_ostream &,
+              llvm::raw_ostream &, bool, bool) { return false; };
+  }
+  return it->d;
+}
+
+namespace lld {
+bool inTestOutputDisabled = false;
+
+/// Universal linker main(). This linker emulates the gnu, darwin, or
+/// windows linker based on the argv[0] or -flavor option.
+int unsafeLldMain(llvm::ArrayRef<const char *> args,
+                  llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS,
+                  llvm::ArrayRef<DriverDef> drivers, bool exitEarly) {
+  SmallVector<const char *, 256> argsV(args);
+  Driver d = whichDriver(argsV, drivers);
+  // Run the driver. If an error occurs, false will be returned.
+  int r = !d(argsV, stdoutOS, stderrOS, exitEarly, inTestOutputDisabled);
+  // At this point 'r' is either 1 for error, and 0 for no error.
+
+  // Call exit() if we can to avoid calling destructors.
+  if (exitEarly)
+    exitLld(r);
+
+  // Delete the global context and clear the global context pointer, so that it
+  // cannot be accessed anymore.
+  CommonLinkerContext::destroy();
+
+  return r;
+}
+} // namespace lld
+
+Result lld::lldMain(llvm::ArrayRef<const char *> args,
+                    llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS,
+                    llvm::ArrayRef<DriverDef> drivers) {
+  int r = 0;
+  {
+    // The crash recovery is here only to be able to recover from arbitrary
+    // control flow when fatal() is called (through setjmp/longjmp or
+    // __try/__except).
+    llvm::CrashRecoveryContext crc;
+    if (!crc.RunSafely([&]() {
+          r = unsafeLldMain(args, stdoutOS, stderrOS, drivers,
+                            /*exitEarly=*/false);
+        }))
+      return {crc.RetCode, /*canRunAgain=*/false};
+  }
+
+  // Cleanup memory and reset everything back in pristine condition. This path
+  // is only taken when LLD is in test, or when it is used as a library.
+  llvm::CrashRecoveryContext crc;
+  if (!crc.RunSafely([&]() { CommonLinkerContext::destroy(); })) {
+    // The memory is corrupted beyond any possible recovery.
+    return {r, /*canRunAgain=*/false};
+  }
+  return {r, /*canRunAgain=*/true};
+}

lld/ELF/Driver.cpp

void Ctx::reset() {
   duplicates.clear();
   nonPrevailingSyms.clear();
   whyExtractRecords.clear();
   backwardReferences.clear();
   hasSympart.store(false, std::memory_order_relaxed);
   needsTlsLd.store(false, std::memory_order_relaxed);
 }
 
-bool elf::link(ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-               llvm::raw_ostream &stderrOS, bool exitEarly,
-               bool disableOutput) {
-  // This driver-specific context will be freed later by lldMain().
+namespace lld {
+namespace elf {
+bool link(ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
+          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput) {
+  // This driver-specific context will be freed later by unsafeLldMain().
   auto *ctx = new CommonLinkerContext;
 
   ctx->e.initialize(stdoutOS, stderrOS, exitEarly, disableOutput);
   ctx->e.cleanupCallback = []() {
     elf::ctx.reset();
     symtab = SymbolTable();
 
     outputSections.clear();
bool link(ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
   partitions.emplace_back();
 
   config->progName = args[0];
 
   elf::ctx.driver.linkerMain(args);
 
   return errorCount() == 0;
 }
+} // namespace elf
+} // namespace lld
 
 // Parses a linker -m option.
 static std::tuple<ELFKind, uint16_t, uint8_t> parseEmulation(StringRef emul) {
   uint8_t osabi = 0;
   StringRef s = emul;
   if (s.ends_with("_fbsd")) {
     s = s.drop_back(5);
     osabi = ELFOSABI_FREEBSD;

lld/MachO/Driver.cpp

if (config->hasExplicitExports) {
     parallelForEach(symtab->getSymbols(), [](Symbol *sym) {
       if (auto *defined = dyn_cast<Defined>(sym))
         if (config->unexportedSymbols.match(defined->getName()))
           defined->privateExtern = true;
     });
   }
 }
 
-bool macho::link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
-                 llvm::raw_ostream &stderrOS, bool exitEarly,
-                 bool disableOutput) {
+namespace lld {
+namespace macho {
+bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
+          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput) {
   // This driver-specific context will be freed later by lldMain().
   auto *ctx = new CommonLinkerContext;
 
   ctx->e.initialize(stdoutOS, stderrOS, exitEarly, disableOutput);
   ctx->e.cleanupCallback = []() {
     resolvedFrameworks.clear();
     resolvedLibraries.clear();
     cachedReads.clear();
bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
   }
 
   if (errorCount() != 0 || config->strictAutoLink)
     for (const auto &warning : missingAutolinkWarnings)
       warn(warning);
 
   return errorCount() == 0;
 }
+} // namespace macho
+} // namespace lld

lld/MinGW/Driver.cpp

if (!bStatic) {
       if (std::optional<std::string> s = findFile(dir, name + ".dll"))
         return *s;
     }
   }
   error("unable to find library -l" + name);
   return "";
 }
 
+namespace lld {
+namespace coff {
+bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
+          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
+}
+
+namespace mingw {
 // Convert Unix-ish command line arguments to Windows-ish ones and
 // then call coff::link.
-bool mingw::link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
-                 llvm::raw_ostream &stderrOS, bool exitEarly,
-                 bool disableOutput) {
+bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
+          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput) {
   auto *ctx = new CommonLinkerContext;
   ctx->e.initialize(stdoutOS, stderrOS, exitEarly, disableOutput);
 
   MinGWOptTable parser;
   opt::InputArgList args = parser.parse(argsArr.slice(1));
 
   if (errorCount())
     return false;
bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
   // the right prefix.
   vec[0] = argsArr[0];
 
   // The context will be re-created in the COFF driver.
   lld::CommonLinkerContext::destroy();
 
   return coff::link(vec, stdoutOS, stderrOS, exitEarly, disableOutput);
 }
+} // namespace mingw
+} // namespace lld

lld/docs/NewLLD.rst

 The ELF, COFF and Wasm Linkers
 ==============================
 
 The ELF Linker as a Library
 ---------------------------
 
 You can embed LLD to your program by linking against it and calling the linker's
-entry point function lld::elf::link.
+entry point function ``lld::lldMain``.
 
 The current policy is that it is your responsibility to give trustworthy object
 files. The function is guaranteed to return as long as you do not pass corrupted
 or malicious object files. A corrupted file could cause a fatal error or SEGV.
 That being said, you don't need to worry too much about it if you create object
 files in the usual way and give them to the linker. It is naturally expected to
 work, or otherwise it's a linker's bug.
 

lld/docs/index.rst

- It is always a cross-linker, meaning that it always supports all the
   above targets however it was built. In fact, we don't provide a
   build-time option to enable/disable each target. This should make it
   easy to use our linker as part of a cross-compile toolchain.
 
 - You can embed LLD in your program to eliminate dependencies on
   external linkers. All you have to do is to construct object files
   and command line arguments just like you would do to invoke an
   external linker and then call the linker's main function,
-  ``lld::elf::link``, from your code.
+  ``lld::lldMain``, from your code.
 
 - It is small. We are using LLVM libObject library to read from object
   files, so it is not a completely fair comparison, but as of February
   2017, LLD/ELF consists only of 21k lines of C++ code while GNU gold
   consists of 198k lines of C++ code.
 
 - Link-time optimization (LTO) is supported by default. Essentially,
   all you have to do to do LTO is to pass the ``-flto`` option to clang.

lld/include/lld/Common/Driver.h

 //===- lld/Common/Driver.h - Linker Driver Emulator -----------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_COMMON_DRIVER_H
 #define LLD_COMMON_DRIVER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace lld {
-struct SafeReturn {
-  int ret;
+enum Flavor {
+  Invalid,
+  Gnu,     // -flavor gnu
+  MinGW,   // -flavor gnu MinGW
+  WinLink, // -flavor link
+  Darwin,  // -flavor darwin
+  Wasm,    // -flavor wasm
+};
+
+using Driver = bool (*)(llvm::ArrayRef<const char *>, llvm::raw_ostream &,
+                        llvm::raw_ostream &, bool, bool);
+
+struct DriverDef {
+  Flavor f;
+  Driver d;
+};
+
+struct Result {
+  int retCode;
   bool canRunAgain;
 };
 
 // Generic entry point when using LLD as a library, safe for re-entry, supports
 // crash recovery. Returns a general completion code and a boolean telling
 // whether it can be called again. In some cases, a crash could corrupt memory
 // and re-entry would not be possible anymore. Use exitLld() in that case to
 // properly exit your application and avoid intermittent crashes on exit caused
 // by cleanup.
-SafeReturn safeLldMain(int argc, const char **argv, llvm::raw_ostream &stdoutOS,
-                       llvm::raw_ostream &stderrOS);
-
-namespace coff {
-bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
-}
-
-namespace mingw {
-bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
-}
-
-namespace elf {
-bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
-}
+Result lldMain(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
+               llvm::raw_ostream &stderrOS, llvm::ArrayRef<DriverDef> drivers);
+} // namespace lld
 
-namespace macho {
-bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
+// With this macro, library users must specify which drivers they use, provide
+// that information to lldMain() in the `drivers` param, and link the
+// corresponding driver library in their executable.
+#define LLD_HAS_DRIVER(name)                                                   \
+  namespace lld {                                                              \
+  namespace name {                                                             \
+  bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,    \
+            llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);  \
+  }                                                                            \
   }
 
-namespace wasm {
-bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
-          llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
+// An array which declares that all LLD drivers are linked in your executable.
+// Must be used along with LLD_HAS_DRIVERS. See examples in LLD unittests.
+#define LLD_ALL_DRIVERS                                                        \
+  {                                                                            \
+    {lld::WinLink, &lld::coff::link}, {lld::Gnu, &lld::elf::link},             \
+        {lld::MinGW, &lld::mingw::link}, {lld::Darwin, &lld::macho::link}, {   \
+      lld::Wasm, &lld::wasm::link                                              \
+    }                                                                          \
   }
-} // namespace lld
 
 #endif

lld/test/CMakeLists.txt

 llvm_canonicalize_cmake_booleans(
   ENABLE_BACKTRACES
   LLVM_ENABLE_ZLIB
   LLVM_ENABLE_ZSTD
   LLVM_ENABLE_LIBXML2
   LLD_DEFAULT_LD_LLD_IS_MINGW
   LLVM_BUILD_EXAMPLES
   LLVM_ENABLE_PLUGINS
   LLVM_BYE_LINK_INTO_TOOLS
   )
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
   MAIN_CONFIG
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py
-  )
+  PATHS
+  "LLVM_SOURCE_DIR"
+  "LLVM_BINARY_DIR"
+  "LLVM_TOOLS_DIR"
+  "LLVM_LIBS_DIR"
+  "SHLIBDIR"
+  "LLVM_LIT_TOOLS_DIR"
+  "LLD_BINARY_DIR"
+  "CURRENT_LIBS_DIR"
+  "CURRENT_TOOLS_DIR"
+  "LLD_SOURCE_DIR"
+)
+configure_lit_site_cfg(
+  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.py.in
+  ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg.py
+  MAIN_CONFIG
+  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.cfg.py
+  PATHS
+  "LLVM_SOURCE_DIR"
+  "LLVM_BINARY_DIR"
+  "LLVM_TOOLS_DIR"
+  "SHLIBDIR"
+  "LLD_BINARY_DIR"
+  "CURRENT_LIBS_DIR"
+  "CURRENT_TOOLS_DIR"
+  "LLD_SOURCE_DIR"
 
 set(LLD_TEST_DEPS lld)
 if (NOT LLD_BUILT_STANDALONE)
   list(APPEND LLD_TEST_DEPS
     FileCheck
+    LLDUnitTests
     count
     dsymutil
     llc
     llvm-ar
     llvm-as
     llvm-bcanalyzer
     llvm-config
     llvm-cvtres
if (NOT WIN32 AND LLVM_BUILD_EXAMPLES)
         )
     endif()
 endif()
 
 add_lit_testsuite(check-lld "Running lld test suite"
   ${CMAKE_CURRENT_BINARY_DIR}
   DEPENDS ${LLD_TEST_DEPS}
   )
+set_target_properties(check-lld PROPERTIES FOLDER "lld tests")
 
 add_custom_target(lld-test-depends DEPENDS ${LLD_TEST_DEPS})
 set_target_properties(lld-test-depends PROPERTIES FOLDER "lld tests")
 
 add_lit_testsuites(LLD ${CMAKE_CURRENT_SOURCE_DIR}
   DEPENDS ${LLD_TEST_DEPS}
   )
 
-set_target_properties(check-lld PROPERTIES FOLDER "lld tests")
-
 # Add a legacy target spelling: lld-test
 add_custom_target(lld-test)
 add_dependencies(lld-test check-lld)
 set_target_properties(lld-test PROPERTIES FOLDER "lld tests")

lld/test/Unit/lit.cfg.py

+# -*- Python -*-
+
+# Configuration file for the 'lit' test runner.
+
+import os
+import subprocess
+
+import lit.formats
+
+# name: The name of this test suite.
+config.name = 'LLD-Unit'
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = []
+
+# test_source_root: The root path where tests are located.
+# test_exec_root: The root path where tests should be run.
+config.test_exec_root = os.path.join(config.lld_obj_root, 'unittests')
+config.test_source_root = config.test_exec_root
+
+# testFormat: The test format to use to interpret tests.
+config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, 'Tests')
+
+# Propagate the temp directory. Windows requires this because it uses \Windows\
+# if none of these are present.
+if 'TMP' in os.environ:
+    config.environment['TMP'] = os.environ['TMP']
+if 'TEMP' in os.environ:
+    config.environment['TEMP'] = os.environ['TEMP']
+
+# Propagate HOME as it can be used to override incorrect homedir in passwd
+# that causes the tests to fail.
+if 'HOME' in os.environ:
+    config.environment['HOME'] = os.environ['HOME']
+
+# Win32 seeks DLLs along %PATH%.
+if sys.platform in ['win32', 'cygwin'] and os.path.isdir(config.shlibdir):
+    config.environment['PATH'] = os.path.pathsep.join((
+            config.shlibdir, config.environment['PATH']))
+
+# Win32 may use %SYSTEMDRIVE% during file system shell operations, so propogate.
+if sys.platform == 'win32' and 'SYSTEMDRIVE' in os.environ:
+    config.environment['SYSTEMDRIVE'] = os.environ['SYSTEMDRIVE']
+
+# Expand the LLD source path so that unittests can use associated input files.
+# (see AsLibELF/ROCm.cpp test)
+config.environment["LLD_SRC_DIR"] = config.lld_src_dir

lld/test/Unit/lit.site.cfg.py.in

+@LIT_SITE_CFG_IN_HEADER@
+
+import sys
+
+config.llvm_src_root = path(r"@LLVM_SOURCE_DIR@")
+config.llvm_obj_root = path(r"@LLVM_BINARY_DIR@")
+config.llvm_tools_dir = lit_config.substitute(path(r"@LLVM_TOOLS_DIR@"))
+config.llvm_build_mode = lit_config.substitute("@LLVM_BUILD_MODE@")
+config.shlibdir = lit_config.substitute(path(r"@SHLIBDIR@"))
+config.lld_obj_root = path(r"@LLD_BINARY_DIR@")
+config.lld_src_dir = path(r"@LLD_SOURCE_DIR@")
+config.lld_libs_dir = lit_config.substitute(path(r"@CURRENT_LIBS_DIR@"))
+config.lld_tools_dir = lit_config.substitute(path(r"@CURRENT_TOOLS_DIR@"))
+
+# Let the main config do the real work.
+lit_config.load_config(
+    config, os.path.join(config.lld_src_dir, "test/Unit/lit.cfg.py"))

lld/tools/lld/lld.cpp

 //  - lld-link:  COFF (Windows)
 //  - ld-wasm:   WebAssembly
 //
 // lld can be invoked as "lld" along with "-flavor" option. This is for
 // backward compatibility and not recommended.
 //
 //===----------------------------------------------------------------------===//
 
-#include "lld/Common/CommonLinkerContext.h"
 #include "lld/Common/Driver.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/Memory.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/LLVMDriver.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PluginLoader.h"
 #include "llvm/Support/Process.h"
 #include "llvm/TargetParser/Host.h"
 #include "llvm/TargetParser/Triple.h"
 #include <cstdlib>
 #include <optional>
 
 using namespace lld;
 using namespace llvm;
 using namespace llvm::sys;
 
-enum Flavor {
-  Invalid,
-  Gnu,     // -flavor gnu
-  WinLink, // -flavor link
-  Darwin,  // -flavor darwin
-  Wasm,    // -flavor wasm
-};
-
-[[noreturn]] static void die(const Twine &s) {
-  llvm::errs() << s << "\n";
-  exit(1);
-}
-
-static Flavor getFlavor(StringRef s) {
-  return StringSwitch<Flavor>(s)
-      .CasesLower("ld", "ld.lld", "gnu", Gnu)
-      .CasesLower("wasm", "ld-wasm", Wasm)
-      .CaseLower("link", WinLink)
-      .CasesLower("ld64", "ld64.lld", "darwin", Darwin)
-      .Default(Invalid);
-}
-
-static cl::TokenizerCallback getDefaultQuotingStyle() {
-  if (Triple(sys::getProcessTriple()).getOS() == Triple::Win32)
-    return cl::TokenizeWindowsCommandLine;
-  return cl::TokenizeGNUCommandLine;
-}
-
-static bool isPETargetName(StringRef s) {
-  return s == "i386pe" || s == "i386pep" || s == "thumb2pe" || s == "arm64pe";
-}
-
-static bool isPETarget(std::vector<const char *> &v) {
-  for (auto it = v.begin(); it + 1 != v.end(); ++it) {
-    if (StringRef(*it) != "-m")
-      continue;
-    return isPETargetName(*(it + 1));
-  }
-  // Expand response files (arguments in the form of @<filename>)
-  // to allow detecting the -m argument from arguments in them.
-  SmallVector<const char *, 256> expandedArgs(v.data(), v.data() + v.size());
-  BumpPtrAllocator a;
-  StringSaver saver(a);
-  cl::ExpansionContext ECtx(saver.getAllocator(), getDefaultQuotingStyle());
-  if (Error Err = ECtx.expandResponseFiles(expandedArgs))
-    die(toString(std::move(Err)));
-  for (auto it = expandedArgs.begin(); it + 1 != expandedArgs.end(); ++it) {
-    if (StringRef(*it) != "-m")
-      continue;
-    return isPETargetName(*(it + 1));
-  }
-
-#ifdef LLD_DEFAULT_LD_LLD_IS_MINGW
-  return true;
-#else
-  return false;
-#endif
-}
-
-static Flavor parseProgname(StringRef progname) {
-  // Use GNU driver for "ld" by default.
-  if (progname == "ld")
-    return Gnu;
-
-  // Progname may be something like "lld-gnu". Parse it.
-  SmallVector<StringRef, 3> v;
-  progname.split(v, "-");
-  for (StringRef s : v)
-    if (Flavor f = getFlavor(s))
-      return f;
-  return Invalid;
-}
-
-static Flavor parseFlavor(std::vector<const char *> &v) {
-  // Parse -flavor option.
-  if (v.size() > 1 && v[1] == StringRef("-flavor")) {
-    if (v.size() <= 2)
-      die("missing arg value for '-flavor'");
-    Flavor f = getFlavor(v[2]);
-    if (f == Invalid)
-      die("Unknown flavor: " + StringRef(v[2]));
-    v.erase(v.begin() + 1, v.begin() + 3);
-    return f;
-  }
-
-  // Deduct the flavor from argv[0].
-  StringRef arg0 = path::filename(v[0]);
-  if (arg0.ends_with_insensitive(".exe"))
-    arg0 = arg0.drop_back(4);
-  return parseProgname(arg0);
-}
-
-bool inTestOutputDisabled = false;
-
-/// Universal linker main(). This linker emulates the gnu, darwin, or
-/// windows linker based on the argv[0] or -flavor option.
-static int lldMain(int argc, const char **argv, llvm::raw_ostream &stdoutOS,
-                   llvm::raw_ostream &stderrOS, bool exitEarly = true) {
-  std::vector<const char *> args(argv, argv + argc);
-  auto link = [&args]() {
-    Flavor f = parseFlavor(args);
-    if (f == Gnu && isPETarget(args))
-      return mingw::link;
-    else if (f == Gnu)
-      return elf::link;
-    else if (f == WinLink)
-      return coff::link;
-    else if (f == Darwin)
-      return macho::link;
-    else if (f == Wasm)
-      return lld::wasm::link;
-    else
-      die("lld is a generic driver.\n"
-          "Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld"
-          " (WebAssembly) instead");
-  }();
-  // Run the driver. If an error occurs, false will be returned.
-  bool r = link(args, stdoutOS, stderrOS, exitEarly, inTestOutputDisabled);
-
-  // Call exit() if we can to avoid calling destructors.
-  if (exitEarly)
-    exitLld(!r ? 1 : 0);
-
-  // Delete the global context and clear the global context pointer, so that it
-  // cannot be accessed anymore.
-  CommonLinkerContext::destroy();
-
-  return !r ? 1 : 0;
-}
-
-// Similar to lldMain except that exceptions are caught.
-SafeReturn lld::safeLldMain(int argc, const char **argv,
-                            llvm::raw_ostream &stdoutOS,
-                            llvm::raw_ostream &stderrOS) {
-  int r = 0;
-  {
-    // The crash recovery is here only to be able to recover from arbitrary
-    // control flow when fatal() is called (through setjmp/longjmp or
-    // __try/__except).
-    llvm::CrashRecoveryContext crc;
-    if (!crc.RunSafely([&]() {
-          r = lldMain(argc, argv, stdoutOS, stderrOS, /*exitEarly=*/false);
-        }))
-      return {crc.RetCode, /*canRunAgain=*/false};
-  }
-
-  // Cleanup memory and reset everything back in pristine condition. This path
-  // is only taken when LLD is in test, or when it is used as a library.
-  llvm::CrashRecoveryContext crc;
-  if (!crc.RunSafely([&]() { CommonLinkerContext::destroy(); })) {
-    // The memory is corrupted beyond any possible recovery.
-    return {r, /*canRunAgain=*/false};
-  }
-  return {r, /*canRunAgain=*/true};
-}
+namespace lld {
+extern bool inTestOutputDisabled;
+
+// Bypass the crash recovery handler, which is only meant to be used in
+// LLD-as-lib scenarios.
+int unsafeLldMain(llvm::ArrayRef<const char *> args,
+                  llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS,
+                  llvm::ArrayRef<DriverDef> drivers, bool exitEarly);
+} // namespace lld
 
 // When in lit tests, tells how many times the LLD tool should re-execute the
 // main loop with the same inputs. When not in test, returns a value of 0 which
 // signifies that LLD shall not release any memory after execution, to speed up
 // process destruction.
 static unsigned inTestVerbosity() {
   unsigned v = 0;
   StringRef(getenv("LLD_IN_TEST")).getAsInteger(10, v);
   return v;
 }
 
+LLD_HAS_DRIVER(coff)
+LLD_HAS_DRIVER(elf)
+LLD_HAS_DRIVER(mingw)
+LLD_HAS_DRIVER(macho)
+LLD_HAS_DRIVER(wasm)
+
 int lld_main(int argc, char **argv, const llvm::ToolContext &) {
   InitLLVM x(argc, argv);
   sys::Process::UseANSIEscapeCodes(true);
 
   if (::getenv("FORCE_LLD_DIAGNOSTICS_CRASH")) {
     llvm::errs()
         << "crashing due to environment variable FORCE_LLD_DIAGNOSTICS_CRASH\n";
     LLVM_BUILTIN_TRAP;
   }
 
+  ArrayRef<const char *> args(argv, argv + argc);
+
   // Not running in lit tests, just take the shortest codepath with global
   // exception handling and no memory cleanup on exit.
-  if (!inTestVerbosity())
-    return lldMain(argc, const_cast<const char **>(argv), llvm::outs(),
-                   llvm::errs());
+  if (!inTestVerbosity()) {
+    int r =
+        lld::unsafeLldMain(args, llvm::outs(), llvm::errs(), LLD_ALL_DRIVERS,
+                           /*exitEarly=*/true);
+    return r;
+  }
 
   std::optional<int> mainRet;
   CrashRecoveryContext::Enable();
 
   for (unsigned i = inTestVerbosity(); i > 0; --i) {
     // Disable stdout/stderr for all iterations but the last one.
     inTestOutputDisabled = (i != 1);
 
     // Execute one iteration.
-    auto r = safeLldMain(argc, const_cast<const char **>(argv), llvm::outs(),
-                         llvm::errs());
+    auto r = lldMain(args, llvm::outs(), llvm::errs(), LLD_ALL_DRIVERS);
     if (!r.canRunAgain)
-      exitLld(r.ret); // Exit now, can't re-execute again.
+      exitLld(r.retCode); // Exit now, can't re-execute again.
 
     if (!mainRet) {
-      mainRet = r.ret;
-    } else if (r.ret != *mainRet) {
+      mainRet = r.retCode;
+    } else if (r.retCode != *mainRet) {
       // Exit now, to fail the tests if the result is different between runs.
-      return r.ret;
+      return r.retCode;
     }
   }
   return *mainRet;
 }

lld/unittests/AsLibAll/AllDrivers.cpp

+//===- AllDrivers.cpp -------------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// This test shows a typical case where all LLD drivers are linked into the
+// application binary. This is very similar to how lld.exe binary is linked,
+// except that here we cleanup the internal LLD memory context after each call.
+//===----------------------------------------------------------------------===//
+
+#include "lld/Common/Driver.h"
+#include "gmock/gmock.h"
+
+LLD_HAS_DRIVER(coff)
+LLD_HAS_DRIVER(elf)
+LLD_HAS_DRIVER(mingw)
+LLD_HAS_DRIVER(macho)
+LLD_HAS_DRIVER(wasm)
+
+static bool lldInvoke(std::vector<const char *> args) {
+  args.push_back("--version");
+  lld::Result r =
+      lld::lldMain(args, llvm::outs(), llvm::errs(), LLD_ALL_DRIVERS);
+  return !r.retCode && r.canRunAgain;
+}
+
+TEST(AsLib, AllDrivers) {
+  EXPECT_TRUE(lldInvoke({"ld.lld"}));
+  EXPECT_TRUE(lldInvoke({"ld64.lld"}));
+  EXPECT_TRUE(lldInvoke({"ld", "-m", "i386pe"})); // MinGW
+  EXPECT_TRUE(lldInvoke({"lld-link"}));
+  EXPECT_TRUE(lldInvoke({"wasm-ld"}));
+}

lld/unittests/AsLibAll/CMakeLists.txt

+# Test usage of LLD as a library.
+# This test covers a typical case where all the LLD drivers are pulled into the
+# application executable.
+
+add_lld_unittests(LLDAsLibAllTests
+  AllDrivers.cpp
+)
+
+target_link_libraries(LLDAsLibAllTests
+  PRIVATE
+  lldCommon
+  lldCOFF
+  lldELF
+  lldMachO
+  lldMinGW
+  lldWasm
+)

lld/unittests/AsLibELF/CMakeLists.txt

+# Test usage of LLD as a library.
+# This test covers a more singular case where only one LLD driver is used in the
+# target application executable.
+
+add_lld_unittests(LLDAsLibELFTests
+  ROCm.cpp
+  SomeDrivers.cpp
+)
+
+target_link_libraries(LLDAsLibELFTests
+  PRIVATE
+  lldCommon
+  lldELF
+)

lld/unittests/AsLibELF/ROCm.cpp

+//===- ROCm.cpp -------------------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// The purpose of this test is to showcase a more singular usage of LLD as a
+// library, where only one LLD driver is being used (and linked in the target
+// application). We also expect that linking twice the same object files
+// would yield a successfull result. When used as library, LLD always cleans its
+// internal memory context after each linker call.
+//===----------------------------------------------------------------------===//
+
+// When this flag is on, we actually need the MinGW driver library, not the
+// ELF one. Here our test only covers the case where the ELF driver is linked
+// into the unit test binary.
+#ifndef LLD_DEFAULT_LD_LLD_IS_MINGW
+
+#include "lld/Common/Driver.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/Path.h"
+#include "gmock/gmock.h"
+#include <algorithm>
+
+static std::string expand(const char *path) {
+  if (!llvm::StringRef(path).contains("%"))
+    return std::string(path);
+
+  llvm::SmallString<256> thisPath;
+  thisPath.append(getenv("LLD_SRC_DIR"));
+  llvm::sys::path::append(thisPath, "unittests", "AsLibELF");
+
+  std::string expanded(path);
+  expanded.replace(expanded.find("%S"), 2, thisPath.data(), thisPath.size());
+  return expanded;
+}
+
+LLD_HAS_DRIVER(elf)
+
+static bool lldInvoke(const char *inPath, const char *outPath) {
+  std::vector<const char *> args{"ld.lld", "-shared", inPath, "-o", outPath};
+  lld::Result s = lld::lldMain(args, llvm::outs(), llvm::errs(),
+                               {{lld::Gnu, &lld::elf::link}});
+  return !s.retCode && s.canRunAgain;
+}
+
+static bool runLinker(const char *path) {
+  // Create a temp file for HSA code object.
+  int tempHsacoFD = -1;
+  llvm::SmallString<128> tempHsacoFilename;
+  if (llvm::sys::fs::createTemporaryFile("kernel", "hsaco", tempHsacoFD,
+                                         tempHsacoFilename)) {
+    return false;
+  }
+  llvm::FileRemover cleanupHsaco(tempHsacoFilename);
+  // Invoke lld. Expect a true return value from lld.
+  std::string expandedPath = expand(path);
+  if (!lldInvoke(expandedPath.data(), tempHsacoFilename.c_str())) {
+    llvm::errs() << "Failed to link: " << expandedPath << "\n";
+    return false;
+  }
+  return true;
+}
+
+TEST(AsLib, ROCm) {
+  EXPECT_TRUE(runLinker("%S/Inputs/kernel1.o"));
+  EXPECT_TRUE(runLinker("%S/Inputs/kernel2.o"));
+  EXPECT_TRUE(runLinker("%S/Inputs/kernel1.o"));
+  EXPECT_TRUE(runLinker("%S/Inputs/kernel2.o"));
+}
+#endif

lld/unittests/AsLibELF/SomeDrivers.cpp

+//===- SomeDrivers.cpp ------------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// In this test we showcase the fact that only one LLD driver can be invoked -
+// the ELF driver that was linked in the test binary. Calling other drivers
+// would return a failure. When using LLD as a library, any driver can be
+// linked into your application.
+//===----------------------------------------------------------------------===//
+
+#include "lld/Common/Driver.h"
+#include "gmock/gmock.h"
+
+LLD_HAS_DRIVER(elf)
+
+static bool lldInvoke(const char *lldExe) {
+  std::vector<const char *> args{lldExe, "--version"};
+  lld::Result s = lld::lldMain(args, llvm::outs(), llvm::errs(),
+                               {{lld::Gnu, &lld::elf::link}});
+  return !s.retCode && s.canRunAgain;
+}
+
+TEST(AsLib, SomeDrivers) {
+  // When this flag is on, we actually need the MinGW driver library, not the
+  // ELF one. Here our test only covers the case where the ELF driver is linked
+  // into the unit test binary.
+#ifndef LLD_DEFAULT_LD_LLD_IS_MINGW
+  EXPECT_TRUE(lldInvoke("ld.lld")); // ELF
+#endif
+  // These drivers are not linked in this unit test.
+  EXPECT_FALSE(lldInvoke("ld64.lld")); // Mach-O
+  EXPECT_FALSE(lldInvoke("lld-link")); // COFF
+  EXPECT_FALSE(lldInvoke("wasm-ld"));  // Wasm
+}

lld/unittests/CMakeLists.txt

+set_target_properties(LLDUnitTests PROPERTIES FOLDER "lld tests")
+
+function(add_lld_unittests test_dirname)
+  add_unittest(LLDUnitTests ${test_dirname} ${ARGN})
+endfunction()
+
+add_subdirectory(AsLibAll)
+add_subdirectory(AsLibELF)

lld/wasm/Driver.cpp

private:
   bool inWholeArchive = false;
 
   std::vector<InputFile *> files;
 };
 } // anonymous namespace
 
 bool link(ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
           llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput) {
-  // This driver-specific context will be freed later by lldMain().
+  // This driver-specific context will be freed later by unsafeLldMain().
   auto *ctx = new CommonLinkerContext;
 
   ctx->e.initialize(stdoutOS, stderrOS, exitEarly, disableOutput);
   ctx->e.logName = args::getFilenameWithoutExe(args[0]);
   ctx->e.errorLimitExceededMsg = "too many errors emitted, stopping now (use "
                                  "-error-limit=0 to see all errors)";
 
   config = make<Configuration>();

llvm/cmake/modules/AddLLVM.cmake

 function(setup_host_tool tool_name setting_name exe_var_name target_var_name)
   get_host_tool_path(${tool_name} ${setting_name} ${exe_var_name} ${target_var_name})
   # Set up a native tool build if necessary
   if(LLVM_USE_HOST_TOOLS AND NOT ${setting_name})
     build_native_tool(${tool_name} exe_name DEPENDS ${tool_name})
     add_custom_target(${target_var_name} DEPENDS ${exe_name})
   endif()
 endfunction()
+
+# Adds the unittests folder if gtest is available.
+function(llvm_add_unittests tests_added)
+  if (EXISTS ${LLVM_THIRD_PARTY_DIR}/unittest/googletest/include/gtest/gtest.h)
+    add_subdirectory(unittests)
+    set(${tests_added} ON PARENT_SCOPE)
+  else()
+    message(WARNING "gtest not found, unittests will not be available")
+  endif()
+endfunction()