In D82490#2132087, @oontvoo wrote:

Does this only happen on Fuchsia? How do I repro it?

In D82490#2131523, @oontvoo wrote:

Rolling forward + fix typos and unused variables in tests

LGTM

This is an updated version of D65430 which matches the feature implemented by ld and gold in https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=f37b21b481a7804a13c5806098c19b6119288ba4. I'm happy to abandon this change in favor of D65430 if @ruiu wants to go ahead with his change.

In D55682#2101819, @MaskRay wrote:

(It is also my personal preference: we can use # as the comment marker. It is simpler. I'll not enforce that, though. When you write comments which are not RUN or CHECK lines, you can write ## . It makes comments stand out and avoids caveats for while some FileCheck developers want to prevent)

In D82276#2106260, @davezarzycki wrote:

Unless I'm missing something, consolidating this logic into LLVM's CMakeLists.txt will break stand alone builds of sub-projects and external projects that depend on this logic.

Was there a policy change? Are stand alone builds no longer supported? Are external projects no longer allowed to use LLVM's GenerateVersionFromVCS.cmake?

In D82276#2105872, @mehdi_amini wrote:

OK I see now, the description wasn't clear to me earlier. So it is about the fact that find_package failure is multiple-lines long, and it is called separately from LLVM, clang, lld, and lldb. Instead we will still display 4 times the warning but it'll be one line instead, right?

Seems like we could also call this once only from the LLVM CMakeLists.txt and make the variables available to the subprojects?

LGTM

In D55682#2099972, @phosek wrote:

In D55682#2099923, @MaskRay wrote:

LGTM. @pcc and @ruiu objected to this patch. Please give some time for them to make comments. Friday is a no meeting day for some of us, so waiting til next week will be nice.

SGTM thanks! I'm also fine waiting until the GNU change lands.

In D55682#2099923, @MaskRay wrote:

LGTM. @pcc and @ruiu objected to this patch. Please give some time for them to make comments. Friday is a no meeting day for some of us, so waiting til next week will be nice.

In D55682#2094089, @MaskRay wrote:

Please add a test, startstop-visibility.s (I would like start-stop-visibility.s, but some existing tests use the startstop- prefix).

Use -triple=x86_64 and use llvm-readelf -s to check visibility. Check both __start_foo and __stop_foo. Don't add another __start_bar.

Please mention that GNU ld and gold from binutils 2.35 onwards will support the option (
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=cae64165f47b64898c4f1982d294862cfae89a47 )

LGTM, I like this, thanks! I think we should use this approach for other flags as well, e.g. RUNTIMES_LLVM_ENABLE_RUNTIMES.

LGTM

In D65430#2089961, @MaskRay wrote:

I have mentioned this several times. You can just process -t output like { echo -n 'a: '; clang a.o -o a -Wl,-t | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} > a.d

The requirement for this feature came up again recently in our build so I'd like to revive this discussion, is there any way we could get this change landed?

@davidxl Do you have any preference between the current solution which uses the artificial padding in __llvm_prf_data section to workaround the BFD ld bug, and a backend feature flag that could be set to enable this feature?

In D76802#2089052, @phosek wrote:

@davidxl Do you have any preference between the current solution which uses the artificial padding in __llvm_prf_data section to workaround the BFD ld bug, and a backend feature flag that could be set to enable this feature?

In D76802#2086875, @phosek wrote:

In D76802#2086757, @vitalybuka wrote:

I still can reproduce with the latest patch on my workstation

Exit Code: 1

Command Output (stderr):
--
/usr/bin/ld: __llvm_prf_data has both ordered and unordered sections
/usr/bin/ld: final link failed: bad value
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

This is the other issue I mentioned which also affects Chromium bots. The root cause is the lack of support for the extended semantics of SHF_LINK_ORDER that !associated metadata rely on in bfd.ld. I'm trying to figure out what's the best way forward. We could just disable all affected profile tests when bfd.ld is used as the linker, but that's pretty drastic. Another option would be to gate this feature on a backend flag; targets that use gold or lld as their linker could turn this on by default.

In D76802#2086757, @vitalybuka wrote:

In D76802#2086742, @phosek wrote:

In D76802#2086725, @vitalybuka wrote:

In D76802#2086650, @phosek wrote:

I looked into these issue:

• http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/27806 failure was addressed by rG69c5ff4668cd4d1bfb6b45d71e15c12e3a23ae05

69c5ff4668cd4d1bfb6b45d71e15c12e3a23ae05 still fails "check-profile" for me and fails here http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/27808

I've updated the patch to handle the _IO_stdin_used symbol in the test. I couldn't reproduce this locally, but it's possible that this is due to different libc version on the bot.

I still can reproduce with the latest patch on my workstation

Exit Code: 1

Command Output (stderr):
--
/usr/bin/ld: __llvm_prf_data has both ordered and unordered sections
/usr/bin/ld: final link failed: bad value
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

In D76802#2086725, @vitalybuka wrote:

In D76802#2086650, @phosek wrote:

I looked into these issue:

• http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/27806 failure was addressed by rG69c5ff4668cd4d1bfb6b45d71e15c12e3a23ae05

69c5ff4668cd4d1bfb6b45d71e15c12e3a23ae05 still fails "check-profile" for me and fails here http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/27808

In D72904#2071861, @pcc wrote:

Should we drop this and just let users set sh_link=self if they want these semantics?

I looked into these issue:

LGTM

I apologize about the late response, I didn't get any email from Phabricator for some reason :/

Superseded by D76802.

In D76802#2077786, @davidxl wrote:

Looks good but can you do help with a clang instrumentation build (IR PGO), and see the binary and profile size change?

@davidxl ping?

I've also tested this change in Fuchsia and saw similar improvements: fuchsia.zbi (the boot image) size reduced from 90M to 42M, fvm.sparse.blk (disk image) size reduced from 3.9G to 2.8G.

This is no longer needed for the profile instrumentation.

Do you have commit permissions or do you need us to land this for you?

In D76802#2061718, @davidxl wrote:

What is the instrumented size improvement for Clang? How about profile size reduction?

In D80186#2069110, @pcc wrote:

In D80186#2068055, @phosek wrote:

In D80186#2067298, @pcc wrote:

Yeah. Thinking about it more, you may consider a different !associated scheme for the __prof*_ globals.

• __profc_* and __profn_* are associated with a dummy global (to allow the sections to be GC'd).
• __profd_* is associated with the corresponding __profc_* section.
  
  Now no special rules are required for inlining. __profc_* is kept alive via the references in the functions, __profd_ is kept alive via !associated and __profn_* is kept alive by direct reference.

I tried that, but now we get an error ld.lld: error: /tmp/instrprof-gc-sections-5f007f.o:(__llvm_prf_cnts): sh_link points to discarded section /tmp/instrprof-gc-sections-5f007f.o:(__llvm_prf_dummy) because the dummy global was discarded which creates a broken sh_link.

Hmm. Maybe instead of making the sections sh_link to a dummy global, make them sh_link to themselves. There was a proposal for sh_link=0 to mean what we want here (D72904) but I believe that sh_link=self should have the same semantics.

In D55682#1336709, @MaskRay wrote:

Does that mean there is a visibility mismatch between source code and linker-defined symbol? e.g.

//extern char __start_f __attribute__((visibility("hidden")));
extern char __start_f; // Fuchsia wants to write code like this, but its visibility is set to hidden due to -z hidden-start-stop-symbols

__attribute__((section("f"))) char *foo() {
  return &__start_f;
}

Please correct me but if my understanding is correct, I think the explicit visibility attribute in source code would be clearer.

In D80186#2067298, @pcc wrote:

Yeah. Thinking about it more, you may consider a different !associated scheme for the __prof*_ globals.

• __profc_* and __profn_* are associated with a dummy global (to allow the sections to be GC'd).
• __profd_* is associated with the corresponding __profc_* section.
  
  Now no special rules are required for inlining. __profc_* is kept alive via the references in the functions, __profd_ is kept alive via !associated and __profn_* is kept alive by direct reference.

In D80186#2067107, @pcc wrote:

In this case you might consider replacing the global reference with null and implementing my proposal in https://bugs.llvm.org/show_bug.cgi?id=41734#c2 to make the section SHF_LINK_ORDER with a zero-length .init_array section.

One option we've discussed internally within our team would be to use ELF section groups (not COMDAT which is a special case of section groups) instead of SHF_LINK_ORDER, but those aren't supported in LLVM IR at the moment, and it's also counter to what was discussed in https://groups.google.com/g/generic-abi/c/_CbBM6T6WeM/m/LZnqx1KZAQAJ.

So after testing this change a bit more, I found a fundamental issue which is the handling of cases when a function is inlined into multiple functions.

LGTM

LGTM

@davidxl would it be possible to take a look?

This needs be done on a per-target basis, not globally (e.g. you may want to use COMPILER_RT_BUILD_SANITIZERS=OFF for target1 but not target2). Have you tried setting RUNTIMES_${target}_COMPILER_RT_BUILD_SANITIZERS=OFF? This should be already supported, see for example https://github.com/llvm/llvm-project/blob/master/clang/cmake/caches/Fuchsia-stage2.cmake#L100

I've added a test which is analogous to instrprof-darwin-dead-strip.c which tests the same functionality on Darwin. I've also updated the inliner to properly handle !associated metadata in D80186 which is necessary here since instr-profiling pass runs before the inliner.

This is a prerequisite for D76802 because InstrProfiling pass runs before inliner which can invalidate the metadata.

LGTM

LGTM

LGTM

LGTM

@scott.linder is actually correct, the reason we write the file is precisely as he described in https://reviews.llvm.org/D79400#2021255. When you use tools like repo which branchless checkout, .git/logs/HEAD won't exist on initial checkout, but we want to detect changes to it later. It's not true that .git/logs/HEAD isn't used, see for example in clang/lib/Basic/CMakeLists.txt on line 24, while the header is being generated by the GenerateVersionFromVCS.cmake script, whether that script is executed is controlled by a dependency on .git/logs/HEAD, if .git/logs/HEAD doesn't change we won't rerun that script. The reason we do this rather than calling .git/logs/HEAD directly is to avoid running git rev-parse HEAD on every build invocation. The reason we depend on `.git/logs/HEAD and not .git/HEAD is because the latter may only contain symbolic information and so won't change e.g. on rebase whereas the former will, in which case we want to regenerate the header. I agree that the Subversion part of find_first_existing_vc_file could be removed now. We also don't need to check Clang or LLD source tree separately from LLVM now that everything is in one repo, but the logic for determining Git revision should be still correct.