LGTM

LGTM

Can you create a new test for clang Driver instead of rewrite the cc1 test?

Just saw this. Thanks for taking care of the fallout.

In D102726#2771933, @dexonsmith wrote:

In D102726#2769856, @steven_wu wrote:

In D102726#2769821, @dexonsmith wrote:

Can you include the revert to attributes-3.3.ll.bc as part of this?

I don't think that matters too much

I think it matters. With the original file in place, it's clear that we actually upgrade correctly all the attributes that were in the 3.3 test. Without it, it's unclear.

I am not sure if I revert a binary file will add a duplication to git history.

Every object in Git is referenced by the hash of its content. Older commits (before @jdoerfert's patch) will point at the same hash for attributes-3.3.ll.bc as the new commits after it gets reverted back. There's no duplication here.

The case in 3.3 version is kind of covered by newly added test case except created by different llvm version.

Yeah, kind of. But not entirely.

LGTM

In D102726#2769935, @jdoerfert wrote:

Nit: Commit subject (not message) is old, maybe change prior to push locally

In D102726#2769821, @dexonsmith wrote:

Can you include the revert to attributes-3.3.ll.bc as part of this?

Remove all incompatible attributes.

Address review feedback. Now remove all incompatible attributes from non pointer type.

I don't think current implementation is the best idea. I think is better to make the gcc compatible LTO value to be a clang Driver only option, give them an option name, and handle them separately in driver to invoke correct fullLTO cc1 command, so that:

• The value remain invalid option for cc1 (CompilerInvocation should just error on that, just like now).
• The ignored value should not appear in the help or autocompletion.

The reason I only fix return type in D102201 is that I identify the source of such incompatibility, which is coming from an optimization pass, so that can be wild spread incompatibility. The optimization pass explicitly happens when it is return type and check to remove all attributes that is incompatibly returned from typeIncompatible.

That is easier than I think. Is there still any missing function in terms of NewPM after this patch?

I don’t understand code size increase as well and have no guess about what might happen. This should be noop on bitcode that can pass the verifier.

Address review feedback.

Thanks @dexonsmith and @tejohnson . I will just abandon this one.

A bit more information is that the GV that triggers the assertion now is being materialized from both Global and Local (Indirect?) Materializer because of the complexity of the GV and Alias in the module.

Ping. I still couldn't figure the algorithm used here and what semantics does the comdat have. Need some help here.

LGTM

Ok, fine with that.

In D92611#2659603, @w2yehia wrote:

In D92611#2659437, @steven_wu wrote:

In D92611#2659079, @w2yehia wrote:

@steven_wu thanks for your review. Is this good to be merged then?

Can you just revert to use parsedOptions as boolean? The state you used right now doesn't add anything. You can just leave lto_set_debug_options not guarded right?

I consider the two APIs lto_set_debug_options and lto_codegen_debug_options to be mutually exclusive, so I need to keep track whether lto_set_debug_options was called.
Before creating the enum state, I had two booleans parsedOptions and parsedOptionsEarly where parsedOptionsEarly is a boolean I added to tell that early options has been set (i.e. lto_set_debug_options called).
However, I cannot set parsedOptions when my new API function is called because I want maybeParseOptions to still do it's thing. So I needed that second boolean.

So I then created the state variable in place of the two booleans (cleaner in my opinion and avoids having to check combinations of the two booleans) to tell which state we're in:

NotParsed, // Initial state. before either lto_set_debug_options or maybeParseOptions is called.
Early,     // After lto_set_debug_options is called. This state will be skipped if user calls lto_codegen_debug_options.
Done       // After maybeParseOptions is called (i.e. unwrap(cg)->parseCodeGenDebugOptions() and lto_add_attrs(cg) have executed)

The state can either change NotParsed -> Done or NotParsed -> Early -> Done depending on which API was used to set debug options.

FYI, for the broken comdat, see the unit tests breaks

In D92611#2659079, @w2yehia wrote:

@steven_wu thanks for your review. Is this good to be merged then?

Update testcase.

@pcc @tejohnson I tried to fix the assertion failure shown in the test case (reduced from actual code generated from swift compiler) by simplifying the ValueMapper but I couldn't quite get the comdat tests to pass. I think comdat is relying on some of the indirect symbol that doesn't appear in the ValueMap for the normal context but I couldn't quite figure out how that works. Any suggestion?

I still don't like this implementation, but fine if you have to do this.

Like replied in email. The non-scaleable option is very intentional and debug_option is by its name for compiler developer only and should not be used in production. The preferred option is to create API to set every single attributes you want to set.

Can you add a test case? Otherwise, LGTM.

LTOCodegen can achieve most of the temp file by going through write_merged_module function. While I don't see this save-temps gains much advantage comparing the existing one except more granularity, it is also fine to have this new debug option. It would be good to switch the save-temp interface to using this flag in the future since it will simplify the code in libLTO and ld.

This one might be better just fix the FIXME in linkCombinedIndex since assert won't do anything in the release build.

My main concern is just that weak alias for libc++abi that never worked for Darwin :)

Looks good. with a small comment inline.

In D93003#2566906, @rprichard wrote:

Maybe this is blocked on someone from Apple reviewing the Mach-O parts?

Good in general. Suggestions in line.

This looks clean. LGTM. ThinLTO change in a separate commit?

LGTM with only one comment.

LGTM. @pcc does this look fine to you?

Offline comment to Florian is that:

MergedModule is needed to support legacy API lto_codegen_write_merged_modules. You don't technical get the module back in memory, you just need to write to a file.

Looks good now. Thanks.

LGTM. Did you figure out the answer of your inline comment?

Sounds like a good idea to me!

LGTM!

LGTM with a small comment.

I think this is good to be a first step. The approach is correct and left some small suggestion above. The linter warnings need to be fixed as well.

Feedback has been provided on the mailing list. Using function attributes is preferred and changing legacy C API is not allowed.

In D88309#2295403, @w2yehia wrote:

@steven_wu what about the alternative solution where we make libLTO query the --filetype command line option. This will not require adding a new function to the API, but it does change the behavior of existing functions (lto_codegen_write_merged_modules, lto_codegen_compile, lto_codegen_optimize, lto_codegen_compile_optimized, lto_codegen_compile_to_file)

diff --git a/llvm/tools/lto/lto.cpp b/llvm/tools/lto/lto.cpp
index b680714..8251f7b 100644
--- a/llvm/tools/lto/lto.cpp
+++ b/llvm/tools/lto/lto.cpp
@@ -157,20 +157,23 @@
 
 // Convert the subtarget features into a string to pass to LTOCodeGenerator.
 static void lto_add_attrs(lto_code_gen_t cg) {
   LTOCodeGenerator *CG = unwrap(cg);
   auto MAttrs = codegen::getMAttrs();
   if (!MAttrs.empty()) {
     std::string attrs = join(MAttrs, ",");
     CG->setAttr(attrs);
   }
 
+  if (auto FT = codegen::getExplicitFileType())
+    CG->setFileType(FT.getValue());
+
   if (OptLevel < '0' || OptLevel > '3')
     report_fatal_error("Optimization level must be between 0 and 3");
   CG->setOptLevel(OptLevel - '0');
   CG->setFreestanding(EnableFreestanding);
 }

In that case, please increment LTO_API_VERSION. If you think the API needs more documentation, please add to docs/LinkTimeOptimization.rst

In D88309#2295281, @hubert.reinterpretcast wrote:

In D88309#2295142, @steven_wu wrote:

• If this is just a temporary workaround, I would resist make this change to bake in this API that will not be used in the future.

This seems to be conflating the immediate reason with the more general value of having such an interface. The interface that @tejohnson's earlier comment mentions presumably exists because there is value in having such an interface.

Note libLTO API needs to be stable so:

• If you need to add API, you need to bump API version, etc. Please refer to other commits that add APIs to see what needs to be done.
• If this is just a temporary workaround, I would resist make this change to bake in this API that will not be used in the future.

Can you clarify your motivation for this? libLTO is used as an interface for linker and linkers, in general, do not understand assembly files.

LGTM

Ok, I guess we are on the same page. The idea sounds fine to me.

In D88114#2288737, @mtrofin wrote:

In D88114#2288732, @steven_wu wrote:

I am not sure what exactly is expected here. What is your definition for pre-optimized bitcode and how your test case ensures that? Can you explain a bit more for context?

Pre-optimized meaning before the llvm optimization pipeline is called. That's the current implementation, and the test explicitly checks that the inlining of bar into foo doesn't happen.

I could add an "alwaysinline" to bar, to further stress that.

I am not sure what exactly is expected here. What is your definition for pre-optimized bitcode and how your test case ensures that? Can you explain a bit more for context?

LGTM. Thanks.

SGTM.

seems fine. The only worry is that if you have a stack that is close to the stack limit and the unwinder might just blow the stack away so it might have bincompat concerns.

In D84803#2245819, @dmajor wrote:

Our builds are back to green with that change. I agree that the root cause is likely elsewhere, but getting to the bottom of it would unfortunately take more time than I am able to commit.

In D86767#2245775, @rprichard wrote:

In D86767#2244767, @steven_wu wrote:

Sorry for the inconvenience that libunwind is not really buildable for armv7 from iOS SDK. You can build a single threaded version without referring to those interfaces. If this is really troublesome for you, feel free to file a bug report and we can see what can be done here.

Thanks. I think I'm OK as-is.

This patch is technically not correct from a bigger context. I think, for example, watch armv7k doesn't default to sjlj exception model but it needs to build sjlj model to support some clients.

That makes sense. It looks like passing -arch armv7k to clang is enough to switch it to using DWARF and compact unwinding rather than sjlj. I also see a comment in ToolChains/Darwin.cpp:
// Only watchOS uses the new DWARF/Compact unwinding method.

It seems that Darwin/armv7k libunwind doesn't build because config.h enables _LIBUNWIND_SUPPORT_COMPACT_UNWIND, but there is no 32-bit ARM compact unwinder. It looks like setting the flag is the right thing to do, though -- maybe the rest of the code is internal to Apple.

I'll revise or abandon this patch. My main goal was to ensure that if _LIBUNWIND_USE_DL_ITERATE_PHDR was defined, then either of _LIBUNWIND_ARM_EHABI or _LIBUNWIND_SUPPORT_DWARF_UNWIND would also be defined.

In D86847#2248354, @MaskRay wrote:

In D86847#2248353, @steven_wu wrote:

I am a bit worry that linker might concatenate bitcode file with padding to achieve alignment requirement, etc. I guess you can create a termination block to mark the end but it is hard to seek the next start.

We can set the section alignment to 1 to avoid padding: (I rushed a bit, sorry: 6ae7b403c3e1aebcb825d3dd4777d3c1149d6d67)

I am a bit worry that linker might concatenate bitcode file with padding to achieve alignment requirement, etc. I guess you can create a termination block to mark the end but it is hard to seek the next start.

Thanks. LGTM.

In D84803#2245283, @dmajor wrote:

Yeah, I wish I could give a nice test case, but it's a huge build and the affected configuration needs a lot of setup so I can't even replicate it locally at the moment.

If you have proposed patches I could send them off for a dry run relatively easily though.

In D84803#2244918, @dmajor wrote:

Hi, we're seeing an assertion failure in one of the Firefox LTO bots:

char llvm::StringRef::operator[](size_t) const: Assertion `Index < Length && "Invalid index!"' failed.

I'm still working on finishing the bisection but I have a hunch that it may be this change. Unfortunately I can't seem to get a backtrace out of this particular bot.

As a guess, could it be the operator[] in getGlobalIdentifier at https://github.com/llvm/llvm-project/blob/4cb016cd2d8467c572b2e5c5d34f376ee79e4ac1/llvm/lib/IR/Globals.cpp#L140 ? It looks like there was previously a check for Name.size() > 0 that isn't checked anymore.

Sorry for the inconvenience that libunwind is not really buildable for armv7 from iOS SDK. You can build a single threaded version without referring to those interfaces. If this is really troublesome for you, feel free to file a bug report and we can see what can be done here.

ping