This is an archive of the discontinued LLVM Phabricator instance.

[lld-macho] Handle TAPI and regular re-exports uniformly
ClosedPublic

Authored by int3 on Aug 6 2020, 12:12 AM.

Details

Reviewers
smeenai
Group Reviewers
Restricted Project
Commits
rG7394460d8759: [lld-macho] Handle TAPI and regular re-exports uniformly
Summary

The re-exports list in a TAPI document can either refer to other inlined
TAPI documents, or to on-disk files (which may themselves be TBD or
regular files.) Similarly, the re-exports of a regular dylib can refer
to a TBD file.

Diff Detail

Unit TestsFailed

Event Timeline

int3 created this revision.Aug 6 2020, 12:12 AM
Herald added a project: Restricted Project. · View Herald TranscriptAug 6 2020, 12:12 AM
int3 requested review of this revision.Aug 6 2020, 12:12 AM
int3 added inline comments.Aug 6 2020, 10:09 AM
lld/test/MachO/Inputs/iPhoneSimulator.sdk/usr/lib/libSystem.tbd
24

is not re-exported*

smeenai added inline comments.
lld/MachO/Driver.cpp
133

IIUC, the previous version would prefer the tbd to the dylib, and resolveDylibPath will prefer the dylib to the TBD.

IIRC, if ld64 finds both a TBD and a dylib for the same file, it verifies the TBD against the dylib and falls back to the dylib in case of a mismatch. I don't think we need to implement that (and I don't think LLVM's version of TextAPI has the equivalence checking logic yet anyway), so perhaps always preferring the dylib in that case (as you're doing now) is reasonable.

182

This matches ld64's behavior as far as I can tell, so it looks good, but can you add a test? Both for this and for the standard search paths only being searched for under the syslibroots?

(This and the tests should also probably be their own diff.)

387–389

Can we get a test for this fix as well? (It should also probably be its own diff.)

lld/MachO/InputFiles.cpp
375

If I'm understanding correctly, this will only be able to find any inline documents we've seen in our input files thus far. Does this match ld64, or is it able to find inline documents from later files?

448–449

Nit: we're inconsistent about reexport vs reExport

lld/test/MachO/Inputs/MacOSX.sdk/usr/lib/libc++abi.tbd
5

Should this be libc++abi?

lld/test/MachO/reexport-stub.s
16

I thought that for sub-libraries, the bind information reflected the actual sub-library the symbol was coming from, and that seems to be the case in a basic test I did with ld64. Can you confirm how this is supposed to behave?

(IIRC there's options to re-export specific symbols, and those bind to the re-exporting library.)

int3 updated this revision to Diff 285741.Aug 14 2020, 12:47 PM
int3 marked 4 inline comments as done.

address some comments

lld/MachO/Driver.cpp
133

yeah, I did inadvertently change things here. From a quick glance at the code, I think your description of ld64's behavior is correct. ld64 can also take an environment variable that makes it always prefer the TAPI file. Do you know if having both the TAPI and the dylib has a real use case? Given the amount of logic ld64 has devoted to it, I guess there might be, but it seems odd. Maybe I could just raise an error for now if both are present?

182
387–389

whoops, I did not mean to leave this change in at all. It isn't a fix, it's wrong :p

lld/MachO/InputFiles.cpp
375

ld64 is indeed able to find documents from later files. It seems like pretty bizarre behavior to me, though -- what would be the use case? (In my initial implementation, I actually tried to limit re-exports in tbds to reference just those in the same file because I thought that made the most sense, but it was more awkward to implement that...)

lld/test/MachO/reexport-stub.s
16

Swapping ld64 for lld in this particular test gives the same result. (sub-library.s tests something similar, and ld64 binds to the parent library there too.) Can you share the test you did that showed a binding to the sub-library?

smeenai added inline comments.Aug 14 2020, 2:26 PM
lld/MachO/Driver.cpp
133

The Xcode 12 betas have both dylibs and TBDs present (for at least some of the SDKs), so erroring out in that case wouldn't be ideal. (I don't think the final Xcode releases every have dylibs.) I think it's fine to just use the dylib if you have both, and perhaps add a TODO to consider adding verification like ld64 has.

lld/MachO/InputFiles.cpp
375

Yeah, I'd also find it weird to reference sub-documents from other files, and I don't know if it's actually used in the SDKs.

lld/test/MachO/reexport-stub.s
16

I'm using ld64 530 from Xcode 11.3.1. If I try to run this test as-is (albeit with clang instead of llvm-mc), when I try to link test, I get an error:

ld: malformed mach-o, symbol table not in __LINKEDIT file './libreexporter.dylib'

I can just add a dummy symbol to work around that, and then

$ cat reexporter.s
.globl foo
foo:
	ret

$ cat test.s
.globl _main
_main:
	ret
.data
	.quad	___gxx_personality_v0

$ clang -c reexporter.s test.s
$ ld -dylib -lc++ -sub_library libc++ reexporter.o -o libreexporter.dylib
$ ld -o test -lSystem -L. -lreexporter test.o
$ objdump --bind test
Bind table:
segment  section            address    type       addend dylib            symbol
__DATA   __data             0x100001000 pointer         0 libc++           ___gxx_personality_v0
int3 added inline comments.Aug 14 2020, 10:48 PM
lld/test/MachO/reexport-stub.s
16

hm, I must have messed up my first attempt, because I was able to see the binding to libc++ on my second try. But I figured out why sub-library.s doesn't repro the issue: bindings only get routed directly to re-exports if those re-exports have "public install names", i.e. if they live under /usr/lib/ or if they are a top-level framework under /System/Library/Frameworks/. I couldn't find any code comments explaining why, but I'm guessing it's an optimization -- I suppose the libraries with "public install names" form a stable ABI, and the only reason they get re-exported by other libraries is to provide some convenience to the user (who doesn't need to pass in half a dozen linker flags). Avoiding unnecessary re-export processing at runtime is probably a performance win.

Addressing this seems somewhat orthogonal to this diff, so a TODO comment should suffice for now?

smeenai added inline comments.Aug 15 2020, 11:35 AM
lld/test/MachO/reexport-stub.s
16

Ah, interesting. And yup, a TODO sounds good.

int3 updated this revision to Diff 286128.Aug 17 2020, 1:29 PM
int3 marked 3 inline comments as done.

add TODOs

lld/MachO/InputFiles.cpp
375

For a quick sanity check, I got lld to load all the non-Swift TBDs under the iPhone simulator's /usr/lib folder, and it worked fine. I did it twice, the second time with all the command-line -l arguments in reverse, and that worked too. So it seems like we don't have any cross-file references to inline documents.

int3 added inline comments.Aug 17 2020, 1:31 PM
lld/test/MachO/reexport-stub.s
16

I've added the TODO to the DylibFile ctor.

int3 added inline comments.Aug 18 2020, 5:29 PM
lld/MachO/Driver.cpp
387–389

my bad, a fix for this is actually necessary. Initial attempt was wrong though x:

real fix here: D86180

int3 added inline comments.Aug 18 2020, 7:09 PM
lld/test/MachO/reexport-stub.s
16

The manpage entry for -no_implicit_dylibs confirms my guess:

-no_implicit_dylibs
            When creating a two-level namespace final linked image, normally the
            linker will hoist up public dylibs that are implicitly linked to make the
            two-level namespace encoding more efficient for dyld.  For example, Cocoa
            re-exports AppKit and AppKit re-exports Foundation.  If you link with
            -framework Cocoa and use a symbol from Foundation, the linker will
            implicitly add a load command to load Foundation and encode the symbol as
            coming from Foundation.  If you use this option, the linker will not add
            a load command for Foundation and encode the symbol as coming from Cocoa.
            Then at runtime dyld will have to search Cocoa and AppKit before finding
            the symbol in Foundation.
int3 updated this revision to Diff 287225.Aug 22 2020, 3:54 PM

restrict TAPI re-exports from referring to documents in other TBD files

smeenai accepted this revision.Aug 26 2020, 2:05 PM

LGTM. Thanks for limiting re-exports to sub-documents in the same TBD!

lld/MachO/InputFiles.cpp
370

Nit: make this static.

lld/test/MachO/reexport-stub.s
16

Ah, good find.

This revision is now accepted and ready to land.Aug 26 2020, 2:05 PM
This revision was landed with ongoing or failed builds.Aug 26 2020, 7:27 PM
This revision was automatically updated to reflect the committed changes.

Hi! I believe our mac builders are failing after this change with

FAIL: lld :: MachO/invalid/stub-link.s (2106 of 2478)
******************** TEST 'lld :: MachO/invalid/stub-link.s' FAILED ********************
Script:
--
: 'RUN: at line 3';   mkdir -p /b/s/w/ir/k/staging/llvm_build/tools/lld/test/MachO/invalid/Output/stub-link.s.tmp
: 'RUN: at line 5';   /b/s/w/ir/k/staging/llvm_build/bin/llvm-mc -filetype obj -triple x86_64-apple-ios /b/s/w/ir/k/llvm-project/lld/test/MachO/invalid/stub-link.s -o /b/s/w/ir/k/staging/llvm_build/tools/lld/test/MachO/invalid/Output/stub-link.s.tmp/test.o
: 'RUN: at line 6';   not lld -flavor darwinnew -o /b/s/w/ir/k/staging/llvm_build/tools/lld/test/MachO/invalid/Output/stub-link.s.tmp/test -Z -L/b/s/w/ir/k/llvm-project/lld/test/MachO/invalid/../Inputs/iPhoneSimulator.sdk/usr/lib -lSystem /b/s/w/ir/k/staging/llvm_build/tools/lld/test/MachO/invalid/Output/stub-link.s.tmp/test.o 2>&1 | /b/s/w/ir/k/staging/llvm_build/bin/FileCheck /b/s/w/ir/k/llvm-project/lld/test/MachO/invalid/stub-link.s
--
Exit Code: 1

Command Output (stderr):
--
/b/s/w/ir/k/llvm-project/lld/test/MachO/invalid/stub-link.s:8:14: error: CHECK-DAG: expected string not found in input
# CHECK-DAG: error: undefined symbol __cache_handle_memory_pressure_event
             ^
<stdin>:1:1: note: scanning from here
lld: error: undefined symbol _from_non_reexported_tapi_dylib, referenced from test.o
^
<stdin>:1:6: note: possible intended match here
lld: error: undefined symbol _from_non_reexported_tapi_dylib, referenced from test.o
     ^

Input file: <stdin>
Check file: /b/s/w/ir/k/llvm-project/lld/test/MachO/invalid/stub-link.s

-dump-input=help explains the following input dump.

Input was:
<<<<<<
         1: lld: error: undefined symbol _from_non_reexported_tapi_dylib, referenced from test.o
dag:8'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
dag:8'1          ?                                                                               possible intended match
>>>>>>

--

I see that you sent out a fix for the objc.s test. Would you mind taking a look at this one? Thanks.

Builder: https://luci-milo.appspot.com/p/fuchsia/builders/ci/clang-mac-x64/b8870848238102897488?

int3 added a subscriber: thakis.Aug 27 2020, 10:42 AM

Hmm, oddly enough the LLVM buildbots seem ok with it. But @thakis also mentioned the build failure on D86180: [lld-macho] Make it possible to re-export .tbd files...

I'm kicking off a build on another machine now. If I still can't figure it out in a bit I'll disable the test.