I break this into two commits. r372021 and r372022.

• [LTO][Legacy] Add new C inferface to query libcall functions
• [lld] Update lld to use the new LTO API to add libcall symbols

In D67568#1670049, @kledzik wrote:

How many symbols are there today? Do we want this overhead of a function call for each? Could you instead return a count and array pointer (like argc and argv) or have a "foreach" function that takes a block and calls it back for each symbol?

lto.cpp is not really tested. Maybe we can test this by move this into lib/LTO then integrate the new version into lld?

• [LTO][Legacy] Add new C inferface to query libcall functions

In D67565#1669913, @tejohnson wrote:

In D67565#1669823, @steven_wu wrote:

Sorry, should be clear that this is for doxygen documents. See @defgroup @ingroup from the comments. Those functions where in the Caching APIs group.

I don't see LLVM website is generating documents based on groups but still good to get it correct.

Just want to clean this up because I am planning to add some new APIs.

I get why they don't belong in the caching section, but now they are in the ThinLTO-specific section (LLVMCTLTO). Should they be in the earlier LLVMCLTO section which seems to include more of the generic LTO interfaces?

Sorry, should be clear that this is for doxygen documents. See @defgroup @ingroup from the comments. Those functions where in the Caching APIs group.

The diagnostics message looks good to me.

Hi Teresa, I am seeing some regressions because of this commit. ThinLTO is now producing more weak symbols than before and I am looking for a good fix. See inline comments for details.

We can discuss what is a good class name for TapiFile and TapiUniversal. It should be consistent with the other patch as well.

few small comments. Otherwise, LGTM.

LGTM

LGTM

From my previous change, we dropped clang.arc.used from the old bitcode. Should we upgrade clang.arc.used instead of dropping them?

Interesting. I did some digging through bug report as well because I am curious why this is only reported now but didn't find anything.

Implementation specific to lto.cpp is really hard to write a test. The TargetOptions are cl::opt bring in through "llvm/CodeGen/CommandFlags.inc" so we need a client for libLTO to test this (which we never had). Let me know if you have any suggestions.

LGTM

Can you add a test case? You can potentially use llc to generate a .o file from .ll and run disassembler on the object file.

ping.

In D63735#1588025, @smeenai wrote:

In D63735#1587999, @steven_wu wrote:

This adds an unnecessary static initializer to the MachOObject.cpp. Can you move the array of validArchs into filescope? You can declare it in getValidArchs() and use getValidArchs() in isValidArchs.

I was hoping that std::array wouldn't necessitate a static initializer, since it's just supposed to be a zero-cost wrapper around a C array.

This adds an unnecessary static initializer to the MachOObject.cpp. Can you move the array of validArchs into filescope? You can declare it in getValidArchs() and use getValidArchs() in isValidArchs.

In D29044#1576319, @thakis wrote:

You need to add a llvm/test/tools/llvm-objdump/lit.local.cfg that contains config.suffixes = ['.yaml'], else the .yaml files added in this change won't be run by lit.

This added an extra dependency from Analysis -> TransformUtils. I don't think this dependency is a good idea, so is adding preheader in analysis pass.

LGTM.

Sounds like an interesting optimization. Is the global storage (from the other module) also eliminated in this case? If so, can you add a testcase for that?

SGTM

We should look at either merging the implementations or providing an api function that provides access to the "llvm.linker.options" named metadata via the IRSymtab. It may not be possible to merge the implementations because in COFF and MACHO the representation of dependent libraries is a string of command line options; whereas, the ELF representation is an array of library strings. This is because for COFF and MACHO there is basically only one linker for each file format so the command line format is fixed. For ELF there are a greater variety of linkers, so I just pass the library strings through to the linker, without processing them into command line arguments, and it is deferred to the linker to interpret the library strings.

This is ELF specific and miss the initial RFC so there is quite some reading to catch up. Should we merge the MachO implementation together with the ELF one? Once you commit this, you might want to file a bug report on me to catch up on the macho side.

LGTM.

Rebase the patch to TOT. Ping again.

Thanks for doing this. I think module flag is a good idea. Some comments inline.

In D61627#1497919, @wanders wrote:

Why are you interested in expending this list?

I have a (kernel) that is compiled with -mno-red-zone and -mcmodel=large which I want to compile with -fembed-bitcode for a debugging tool that needs the bitcode.

But I can carry these patches locally for now.

Like I said, I am not worried that -mno-red-zone itself changes the ABI. As long as LLVM still respect the attribute the same way, it is fine. I want to consult Tim to make sure we can support re-targeting no red zone from armv7k to arm64_32.

In D61627#1493674, @compnerd wrote:

@steven_wu - yeah, -mred-zone, -mno-red-zone does impact the ABI, at least on x86_64. It changes the way that the arguments are spilled and the stack layout.

The main concern for adding this blacklist was because of long term maintainability since the option is going to be embedded into bitcode. Looking at this specific option, I have no reason against because it doesn't affect embedded compiler flags.

When you are manually configuring the bootstrap, I still don't think trying to bypass libtool is the correct fix because you still need to specify the correct llvm-ar to use. The default CMAKE_AR inferred by cmake will not work. Maybe it is an option to add the control to force llvm-ar if wished, then you have two options when you manually configuring bootstrap:

Rebase the patch and did some fixup:

If you use cmake variable CLANG_ENABLE_BOOTSTRAP, it should take care of DYLD_LIBRARY_PATH so libtool is using the just built libLTO to create static library. The setup of the DYLD_LIBRARY_PATH is just below.

In D61255#1484855, @evgeny777 wrote:

I agree this doesn't look like the correct fix, especially the check !Sym.isExecutable() which doesn't look correct to me

The problem is that linkonce_odr functions and data should be IMO treated differently. The linkonce_odr linkage implies linker leaves only one copy of the symbol in the final image.
However it's not really a problem having multiple internal copies of linkonce_odr function because all copies follow one definition rule. This fact allows ThinLTO to efficiently optimize such function calls.
Situation however is different with linkonce_odr objects, because it's wrong to have multiple copies of linkonce_odr object because reads and writes supposed to access same memory will actually become
inconsistent. However there is still a nuance: if object is a compile time constant there can't be any write access to it, so we can safely internalize it.

I don't see this happening in legacy thinLTO code generator but I need to do more digging to see what is the difference. For legacy API, the static variable is not internalized or turned into available_externally.

ping

Add tests for the change. There is acutally an easy to test with llvm-lto

In D60924#1473166, @dexonsmith wrote:

In D60924#1473161, @steven_wu wrote:

In D60924#1473135, @dexonsmith wrote:

Is there a good way to test this?

Not currently in LLVM. It might be possible to add a test in clang repo which requires DARWIN and ld64.

Maybe we can split out the function to generate the filename, and add a unit test for that?

In D60924#1473135, @dexonsmith wrote:

Is there a good way to test this?

forgot to save the inline comments.

LGTM with one additional small comments

Thanks Teresa and Ben for the feedback! I am landing this change.

In D60421#1464989, @bd1976llvm wrote:

Sorry for the delay in replying. I was trying to understand if the upgrade path was important. Comments inline:

In D60421#1461442, @steven_wu wrote:

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

My understanding is that the on disk bitcode contains a binary representation of the IRSymtab and this is read into memory when loading the file (which is why the upgrade path is more expensive than the no-upgrade path). I am saying that currently the legacy api does not require the IRSymtab to be written to disk. Writing it may cost more time and make the input files bigger.

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

Add one file that was missing from previous patch

In D60421#1458858, @tejohnson wrote:

What part of the patch caused the need to change the internalize action to do promote+internalize in one go?

I reverted this because the old thinLTO test cases need to be updated to have data layout as a requirement to use IRSymtab. I will start a separate review to address all those issues.

I reverted this because the old thinLTO test cases need to be updated to have data layout as a requirement to use IRSymtab. I will start a separate review to address all those issues.

Review feedback.