Thanks for explanation. Then the current approach in this patch LGTM. Thanks!

In D59709#1440113, @tejohnson wrote:

In D59709#1440081, @steven_wu wrote:

In D59709#1440033, @tejohnson wrote:

In D59709#1440017, @steven_wu wrote:

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

It is in fact ThinLTO that is changing the visibility (D43130). See the change I made to FunctionImport.cpp.

Interesting, I made that change! But I somehow remembered I got pushed back because of ELF semantics and end up implementing that in ld64 instead. Let me dig up some context and get back to you.

Are you thinking of https://reviews.llvm.org/D43361?

In D59709#1440033, @tejohnson wrote:

In D59709#1440017, @steven_wu wrote:

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

It is in fact ThinLTO that is changing the visibility (D43130). See the change I made to FunctionImport.cpp.

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

In D44373#1436308, @jyknight wrote:

In D44373#1036189, @steven_wu wrote:

Thanks for fixing this. LGTM.

Looks like this never actually got committed?

LGTM

LGTM.

I talked to Erik offline. I think the upgrade tests should run on .bc file rather than text format.

LGTM with a suggestion to make code cleaner.

Sorry for following up late on the patch. Removing the reachability testing for the exit block causes false positive for infinite loop cases like this:

Not sure about gn part and if that has any impact. Otherwise, LGTM.

Fix the comment

I was planning to add a test but I am not sure how to check the file type of temporary files.

In D53945#1347562, @thakis wrote:

In D53945#1343312, @thakis wrote:

Out of interest, why aren't most of the tests for this lit tests? The usual llvm way of testing stuff is to write a small llvm tool using your library (which will be useful outside of a testing context too) and then use that to write lit tests.

Looks like this is now happening (rL350341). Are most of the unit tests going to move to lit tests now?

Thanks for the clarifications. Would it be possible to utilize ThinBackendProc for this instead of a new CodeGenManager class? I.e. make a new derived version that does the index file write and spawns the local processes? The advantage is that it would start converging the implementations. And I think this could aid in refactoring suggested below to avoid duplication. Another advantage is that both LTO API's would have access to all backend implementations (in process, write indexes and exit, write indexes and use local processes, etc).

Other than a small design choice commented inline, LGTM.

See comments inline.

I think I missed something before. LGTM now!

In D55612#1328937, @arphaman wrote:

In D55612#1328773, @steven_wu wrote:

Don't we need IR support for this as well? sdk version is not in the triple so it is going to get lost when building from IR. Maybe use a metadata node?

I provide a way to set and retrieve the SDK version from the IR using the "SDK Version" metadata. Do I need something more than that? Clang can set it and we are already getting it in this patch.

Don't we need IR support for this as well? sdk version is not in the triple so it is going to get lost when building from IR. Maybe use a metadata node?

Thanks for taking a look. This patch is adding the customization points for thinLTO legacy API, which the code generator constructs clang invocations to do code generation. There is no dependency on any build system here and it only has a prove of concept codegen manager which invokes clang directly and collect the result back. You can replace this codegen manager with any protocol that is needed to talk to build system to run clang codegen.
XPC is the way to send information between process on Darwin, which is probably what we are going to use to talk to build system. If interested, I can post a patch which have example how to construct XPC communications, but there isn’t a build system you can use to listen on the other side to run the job yet.
When I say there are no code change for linker, I really mean there is no need to change a single line of code (maybe we need to add an API to select codegen manager in the future). ld64 really has a different approach using C API, which it tries to map the object file output back to the bitcode it gets as input. Terminating and relaunching the linker might has unexpected semantic changes for LTO. In the long run, maybe ld64 needs to design a new set of APIs to use the new C++ APIs but this is out of scope of this patch.

Ping. Is there any comments of implementing C API in this approach?

In D55525#1327742, @compnerd wrote:

This really feels odd. Why not expect that the developer will add the content themselves? I'm not sure I understand the motivation for this change.

This is upstreamed by Saleem already

I did some experiment and put some more thought into this patch. Not hashing PreservedSymbol passed to the C API is fine because the reason you mentioned but freestanding/no-builtin bits stills need to be handled separately in current status. That should be a different discussion so this patch is LGTM.

I wonder freestanding is still an issue after r316819. There is a function attribute for that now so it should be respected if the module is built with -ffreestanding and it should create different hash. I will do some digging to see if I can find the original report and how they setup the build. @mehdi_amini, do you remember the original problem?

Thanks!

Steven - you had a suggestion around callers to getMDNodeFwdRefOrNull, but after this change there will be no more callers to that. So should I go ahead and submit this one, then remove that interface completely in a follow up? Or do it in this patch?

My thought is that this patch should make bitcode reader more robust when lazy loading, which is always a good thing. If this is a performance regression, the regression is coming from how debug information is generated. If there are no such forward-ref in the metadata, there is no slowdown with this patch. We can investigate performance regression post commit if needed.

I reverted this in r346768.

This commit causes our internal bots failed to bootstrap clang. The error we are getting is:

"gCrashRecoveryEnabled (.llvm.1401930837577591816)", referenced from:
    clang::ParseAST(clang::Sema&, bool, bool) in 286.thinlto.o

I have a bit trouble to reproduce the problem but it is 100% reproducible on our bots. I suspect the issue is ThinLTO cache reuse because if I clear the cache then the link will succeed. I will update if I can reproduce and pinpoint the issue.

Oh, I see. This is not really related to the debug information quality. It is a pure metadata lazy loading problem. LGTM.

In D53596#1272986, @vsk wrote:

Thanks for the patch! Your explanation makes sense to me, but I'm not familiar enough with this code to give a +1. @steven_wu, any thoughts on this?

You need to do a release build without assertion. Local names like parameters are dropped from IR to save space/memory.
I think you just need to check the parameter name or the implementation in your test case.

Thanks for posting this. I will start with some high level comments.

Thanks for doing this!

Update comments

Update patch

Also a bit clarification why ModuleToDefinedGVSummaries can have missing entries, because there can be an empty module, or a module has no GV in it. And this happens during thinLTO WebKit.
For example, in WebCore MacOS build, UnifiedSource26-mm.o has empty text and data section. Its object file only has debug info.

Fix up my previous patch.

I can't find a super clean solution for this. Here is one possible solution, which is to iterate and insert the entry before entering thread pool

LGTM

LGTM!

I do prefer the current approach especially on Darwin. Some concerns of switching to use "-L + -l" are:

1. clang and compiler-rt are rev-locked. Inferring the compiler-rt from resource-dir and passing to linker with the full path can prevent mistakes of mixing them up.
2. This change does change linker semantics on Darwin. ld64 treats the inputs from command line with highest priority. Currently ld64 will use compiler-rt symbols before any other libraries passed in with -l flag. If clang decide to pass compiler-rt with -l flag, any other libraries (static or dynamic) that passed with -l flag will takes the priority over compiler-rt. This can lead to unexpected behavior.

LGTM. Thanks!

Remove the declaration in ThinLTOCodeGenerator.h?

I dont mind removing codege(). This interface exists since the first prototype and no longer exposed through C API. It seems to have the same model of optimize(), which is also not thread safe. optimize() is still used by llvm-lto but not in thread context.

I don't think we need another instance of TMBuilder. Can we just call codegenModule directly from run()? TMBuilder should be initialized already from addModule.

I feel like this is a much tricky situation than just new and init. Following example is the same situation.

__attribute__((objc_root_class))
@interface NSObject
- (void) foo;
- (void) bar;
@end

Can you add a bit more explanation other than just saying fixing PR38372?

In D49434#1170267, @pcc wrote:

I would imagine that folks who care about code size would be linking with --gc-sections (or -dead_strip in ld64) so the library functions would end up being dropped if they aren't actually needed.

For the old API, I don't think it is reasonable to expect user or linker to pass in the correct preserve list. There is likely no knowledge outside the LTO modules that these lib funcs are needed.

In D49434#1165761, @tejohnson wrote:

In D49434#1165612, @pcc wrote:

You wouldn't be making changes to the IR, only the summary. I was thinking that you might add code to http://llvm-cs.pcc.me.uk/lib/Analysis/ModuleSummaryAnalysis.cpp#534 that would effectively do:

if (ValueInfo VI = Index.getValueInfo(GlobalValue::getGUID("memcpy")))
    for (auto &Summary : VI.getSummaryList())
      Summary->setLive(true);
if (ValueInfo VI = Index.getValueInfo(GlobalValue::getGUID("memmove")))
    for (auto &Summary : VI.getSummaryList())
      Summary->setLive(true);
// etc.

If I recall correctly it wasn't the LTO dead code elimination that was removing them. Caroline, I forget what was happening in the original case - were they being internalized by LTO and then eliminated by GlobalDCE? If so, then Peter's approach should fix the issue by preventing them from being internalized (at least by ThinLTO, not sure about regular LTO).

In D49362#1163775, @tejohnson wrote:

In D49362#1163768, @steven_wu wrote:

When importing a variable definition when the ref has this bit set, it can be imported as a local copy, without promoting the name/linkage type. That will avoid the need to do any modification to the optimization passes. On the exporting side, we could do prevent the promotion if we knew that all modules importing a reference also imported the referenced variable definition. It does however look like we should be importing all non-interposable linkage constant variables that are referenced, which should be all we care about for this anyway.

What if the constant is really big? Looks like the importing decision has to be made by the thin linker directly. You can not import the "constant" global as something like "available_externally constant" because the original copy might not be visible from the original module (just like the example), unless you want to modify the original copy to be "external hidden".

Not sure I understand your question, can you clarify the concern? Right now, computeImportForReferencedGlobals during the thin link we import any referenced variable, unless it has interposable linkage or references other summaries (in which case it is not a constant). The imported variables are imported as available externally definitions. The original definition is currently promoted to external hidden.

When importing a variable definition when the ref has this bit set, it can be imported as a local copy, without promoting the name/linkage type. That will avoid the need to do any modification to the optimization passes. On the exporting side, we could do prevent the promotion if we knew that all modules importing a reference also imported the referenced variable definition. It does however look like we should be importing all non-interposable linkage constant variables that are referenced, which should be all we care about for this anyway.

LGTM. Thanks!

Add testcase using 6.0 bitcode.

Address review feedback

oops, accidentally opened a new review. closing.

In that case, we can enable verification on checked-in bitcode files as a follow-up. How about testing this by round-tripping textual IR through llvm-as and llvm-dis? You just need a function, alias, and global var with local linkage and without dso_local set, right?

In D49039#1154801, @vsk wrote:

Thanks for doing this!

Could you add a test? Adding "opt -verify <old-bitcode-file>" run lines to the compatibility tests would be fine.

It is easier and cleaner if I just fold everything into getOSVersion.

handle *-apple-macosx target option