Thanks for doing this. I think module flag is a good idea. Some comments inline.

In D61627#1497919, @wanders wrote:

Why are you interested in expending this list?

I have a (kernel) that is compiled with -mno-red-zone and -mcmodel=large which I want to compile with -fembed-bitcode for a debugging tool that needs the bitcode.

But I can carry these patches locally for now.

Like I said, I am not worried that -mno-red-zone itself changes the ABI. As long as LLVM still respect the attribute the same way, it is fine. I want to consult Tim to make sure we can support re-targeting no red zone from armv7k to arm64_32.

In D61627#1493674, @compnerd wrote:

@steven_wu - yeah, -mred-zone, -mno-red-zone does impact the ABI, at least on x86_64. It changes the way that the arguments are spilled and the stack layout.

The main concern for adding this blacklist was because of long term maintainability since the option is going to be embedded into bitcode. Looking at this specific option, I have no reason against because it doesn't affect embedded compiler flags.

When you are manually configuring the bootstrap, I still don't think trying to bypass libtool is the correct fix because you still need to specify the correct llvm-ar to use. The default CMAKE_AR inferred by cmake will not work. Maybe it is an option to add the control to force llvm-ar if wished, then you have two options when you manually configuring bootstrap:

Rebase the patch and did some fixup:

If you use cmake variable CLANG_ENABLE_BOOTSTRAP, it should take care of DYLD_LIBRARY_PATH so libtool is using the just built libLTO to create static library. The setup of the DYLD_LIBRARY_PATH is just below.

In D61255#1484855, @evgeny777 wrote:

I agree this doesn't look like the correct fix, especially the check !Sym.isExecutable() which doesn't look correct to me

The problem is that linkonce_odr functions and data should be IMO treated differently. The linkonce_odr linkage implies linker leaves only one copy of the symbol in the final image.
However it's not really a problem having multiple internal copies of linkonce_odr function because all copies follow one definition rule. This fact allows ThinLTO to efficiently optimize such function calls.
Situation however is different with linkonce_odr objects, because it's wrong to have multiple copies of linkonce_odr object because reads and writes supposed to access same memory will actually become
inconsistent. However there is still a nuance: if object is a compile time constant there can't be any write access to it, so we can safely internalize it.

I don't see this happening in legacy thinLTO code generator but I need to do more digging to see what is the difference. For legacy API, the static variable is not internalized or turned into available_externally.

ping

Add tests for the change. There is acutally an easy to test with llvm-lto

In D60924#1473166, @dexonsmith wrote:

In D60924#1473161, @steven_wu wrote:

In D60924#1473135, @dexonsmith wrote:

Is there a good way to test this?

Not currently in LLVM. It might be possible to add a test in clang repo which requires DARWIN and ld64.

Maybe we can split out the function to generate the filename, and add a unit test for that?

In D60924#1473135, @dexonsmith wrote:

Is there a good way to test this?

forgot to save the inline comments.

LGTM with one additional small comments

Thanks Teresa and Ben for the feedback! I am landing this change.

In D60421#1464989, @bd1976llvm wrote:

Sorry for the delay in replying. I was trying to understand if the upgrade path was important. Comments inline:

In D60421#1461442, @steven_wu wrote:

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

My understanding is that the on disk bitcode contains a binary representation of the IRSymtab and this is read into memory when loading the file (which is why the upgrade path is more expensive than the no-upgrade path). I am saying that currently the legacy api does not require the IRSymtab to be written to disk. Writing it may cost more time and make the input files bigger.

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

Add one file that was missing from previous patch

In D60421#1458858, @tejohnson wrote:

What part of the patch caused the need to change the internalize action to do promote+internalize in one go?

I reverted this because the old thinLTO test cases need to be updated to have data layout as a requirement to use IRSymtab. I will start a separate review to address all those issues.

I reverted this because the old thinLTO test cases need to be updated to have data layout as a requirement to use IRSymtab. I will start a separate review to address all those issues.

Review feedback.

I talked with Akira offline and we think this is probably the best approach to fix this LTO issue. I will leave others to comment if they think otherwise.

Address review feedback

LGTM once the clang patch is good to land.

In D60226#1455688, @steven_wu wrote:

In D60226#1455485, @pcc wrote:

In D60226#1455477, @tejohnson wrote:

In D60226#1453856, @steven_wu wrote:

Also do note, I did a quick benchmark on thin-link time for libLTO.dylib. thin-link was 2.6s before the change vs. 4.0s after the change. Building IRSymtab during thin-link does introduce non-negligible overhead. It might be possible to build them async in threads to reduce the overhead.

@pcc is this surprising? Typically we should just be loading it out of the bitcode file. There are cases where it has to be rebuilt, but theoretically those should be exceptions.

Yes, we normally just load it out of the bitcode file. The irsymtab only needs to be rebuilt when the version of the producer and the consumer do not match. You might be hitting the "upgrade" path here: http://llvm-cs.pcc.me.uk/lib/Object/IRSymtab.cpp#330

This is likely. I didn't bootstrap the build. I will try again with a matching version.

In D60226#1455485, @pcc wrote:

In D60226#1455477, @tejohnson wrote:

In D60226#1453856, @steven_wu wrote:

Also do note, I did a quick benchmark on thin-link time for libLTO.dylib. thin-link was 2.6s before the change vs. 4.0s after the change. Building IRSymtab during thin-link does introduce non-negligible overhead. It might be possible to build them async in threads to reduce the overhead.

@pcc is this surprising? Typically we should just be loading it out of the bitcode file. There are cases where it has to be rebuilt, but theoretically those should be exceptions.

Yes, we normally just load it out of the bitcode file. The irsymtab only needs to be rebuilt when the version of the producer and the consumer do not match. You might be hitting the "upgrade" path here: http://llvm-cs.pcc.me.uk/lib/Object/IRSymtab.cpp#330

Also do note, I did a quick benchmark on thin-link time for libLTO.dylib. thin-link was 2.6s before the change vs. 4.0s after the change. Building IRSymtab during thin-link does introduce non-negligible overhead. It might be possible to build them async in threads to reduce the overhead.

In D59945#1449696, @theraven wrote:

I am not convinced that there is sufficient abstraction here to handle multiple runtimes without significant code duplication. The Mach-O and LLVM IR parsers both appear to duplicate all knowledge of the runtime's data structures, so if you wanted to add a second runtime you'd then need four classes. I would expect to see two abstraction layers:

1. Something allowing you to specify a data structure to examine that would load it from some binary (IR, Mach-O, ELF, whatever).
2. Something built on top of this that then queries specific runtime metadata structures.
   
   In clang, we support two broad families of Objective-C runtime: Apple (Legacy and Modern, with version-specific things for macOS, iOS, and so on) and GNU (including GCC, GNUstep / WinObjC, and ObjFW). These are largely orthogonal to object file formats. For example, the GNUstep runtime is widely used with both ELF and PE/COFF binaries and can be used with Mach-O (though typically only for testing). If we wanted to add support for all of these, I suspect that we'd have a code explosion.

Thanks for explanation. Then the current approach in this patch LGTM. Thanks!

In D59709#1440113, @tejohnson wrote:

In D59709#1440081, @steven_wu wrote:

In D59709#1440033, @tejohnson wrote:

In D59709#1440017, @steven_wu wrote:

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

It is in fact ThinLTO that is changing the visibility (D43130). See the change I made to FunctionImport.cpp.

Interesting, I made that change! But I somehow remembered I got pushed back because of ELF semantics and end up implementing that in ld64 instead. Let me dig up some context and get back to you.

Are you thinking of https://reviews.llvm.org/D43361?

In D59709#1440033, @tejohnson wrote:

In D59709#1440017, @steven_wu wrote:

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

It is in fact ThinLTO that is changing the visibility (D43130). See the change I made to FunctionImport.cpp.

I am not quite sure what is the semantics for ELF linker. When thinLTO picks the linkonce_odr version and promote it to weak, it is not changing the visibility. It is a later linker optimization that auto hide the linkonce_odr symbol, correct?

In D44373#1436308, @jyknight wrote:

In D44373#1036189, @steven_wu wrote:

Thanks for fixing this. LGTM.

Looks like this never actually got committed?

LGTM

LGTM.

I talked to Erik offline. I think the upgrade tests should run on .bc file rather than text format.

LGTM with a suggestion to make code cleaner.

Sorry for following up late on the patch. Removing the reachability testing for the exit block causes false positive for infinite loop cases like this:

Not sure about gn part and if that has any impact. Otherwise, LGTM.

Fix the comment

I was planning to add a test but I am not sure how to check the file type of temporary files.

In D53945#1347562, @thakis wrote:

In D53945#1343312, @thakis wrote:

Out of interest, why aren't most of the tests for this lit tests? The usual llvm way of testing stuff is to write a small llvm tool using your library (which will be useful outside of a testing context too) and then use that to write lit tests.

Looks like this is now happening (rL350341). Are most of the unit tests going to move to lit tests now?

Thanks for the clarifications. Would it be possible to utilize ThinBackendProc for this instead of a new CodeGenManager class? I.e. make a new derived version that does the index file write and spawns the local processes? The advantage is that it would start converging the implementations. And I think this could aid in refactoring suggested below to avoid duplication. Another advantage is that both LTO API's would have access to all backend implementations (in process, write indexes and exit, write indexes and use local processes, etc).

Other than a small design choice commented inline, LGTM.

See comments inline.

I think I missed something before. LGTM now!

In D55612#1328937, @arphaman wrote:

In D55612#1328773, @steven_wu wrote:

Don't we need IR support for this as well? sdk version is not in the triple so it is going to get lost when building from IR. Maybe use a metadata node?

I provide a way to set and retrieve the SDK version from the IR using the "SDK Version" metadata. Do I need something more than that? Clang can set it and we are already getting it in this patch.

Don't we need IR support for this as well? sdk version is not in the triple so it is going to get lost when building from IR. Maybe use a metadata node?

Thanks for taking a look. This patch is adding the customization points for thinLTO legacy API, which the code generator constructs clang invocations to do code generation. There is no dependency on any build system here and it only has a prove of concept codegen manager which invokes clang directly and collect the result back. You can replace this codegen manager with any protocol that is needed to talk to build system to run clang codegen.
XPC is the way to send information between process on Darwin, which is probably what we are going to use to talk to build system. If interested, I can post a patch which have example how to construct XPC communications, but there isn’t a build system you can use to listen on the other side to run the job yet.
When I say there are no code change for linker, I really mean there is no need to change a single line of code (maybe we need to add an API to select codegen manager in the future). ld64 really has a different approach using C API, which it tries to map the object file output back to the bitcode it gets as input. Terminating and relaunching the linker might has unexpected semantic changes for LTO. In the long run, maybe ld64 needs to design a new set of APIs to use the new C++ APIs but this is out of scope of this patch.

Ping. Is there any comments of implementing C API in this approach?

In D55525#1327742, @compnerd wrote:

This really feels odd. Why not expect that the developer will add the content themselves? I'm not sure I understand the motivation for this change.

This is upstreamed by Saleem already

I did some experiment and put some more thought into this patch. Not hashing PreservedSymbol passed to the C API is fine because the reason you mentioned but freestanding/no-builtin bits stills need to be handled separately in current status. That should be a different discussion so this patch is LGTM.

I wonder freestanding is still an issue after r316819. There is a function attribute for that now so it should be respected if the module is built with -ffreestanding and it should create different hash. I will do some digging to see if I can find the original report and how they setup the build. @mehdi_amini, do you remember the original problem?

Thanks!

Steven - you had a suggestion around callers to getMDNodeFwdRefOrNull, but after this change there will be no more callers to that. So should I go ahead and submit this one, then remove that interface completely in a follow up? Or do it in this patch?

My thought is that this patch should make bitcode reader more robust when lazy loading, which is always a good thing. If this is a performance regression, the regression is coming from how debug information is generated. If there are no such forward-ref in the metadata, there is no slowdown with this patch. We can investigate performance regression post commit if needed.

I reverted this in r346768.