LGTM, thanks Paul!

I can do that. Let me test the patch on my end and I'll commit.

Thank you for your time! :)

As suggested by Adrian.

Simplify.

In D94267#2491787, @rnk wrote:

In D94267#2491643, @aganea wrote:

This is mostly caused by contention when page faulting on either mmap'ed files or zero-pages. KeYieldProcessorEx is spinning while waiting for a lock for bringing pages into the 'working set'.

Interesting, I've heard similar things about LLD ELF. I wonder how much of this is IO and how much of this is locks around modifying the process page directory.

In my case at least, it's exclusively due to virtual page management. There's no disk IO, everything was already in cache. Only the two spikes at the end, which is the deferred System write of the PDB & the EXE.

FWIW, it looks like we're spending more than half of the CPU time in the NT kernel (running on Windows 10 version 2004).

I've checked several of our binaries, compared the PDB symbol stream, and except for extra S_SKIP records, it's all the same.

What are the exact conditions to repro Nick's failure? On Fedora 32, I ran py llvm-lit sigpipe-handling.c before this patch and before rGbf401256edd00e921a5d3a0bf4cf6ee66ae51cd6 , the test suceeds, it should not, should it?

Do you think it would be possible to add test coverage in Clang by using Nick's test?

Seems good to me too! Thank you!

In D86694#2463429, @russell.gallop wrote:

• With scudo all lit tests (generally) pass, but there are some intermittent failures (Access Violation) on the following tests:

ExecutionEngine/MCJIT/test-global-init-nonzero-sm-pic.ll
ExecutionEngine/MCJIT/test-ptr-reloc-sm-pic.ll
ExecutionEngine/MCJIT/multi-module-sm-pic-a.ll

This happens with VS2019 or clang-cl as the toolchain. I've run multiple times without scudo and don't see these. Unfortunately I haven't been able to get this to fail in a debugger.

This is really cool. I'll take a look at this tomorrow!

Thanks for looking at this!

Doh! @zturner I forgot to do the git commit --amend thing to credit you before pushing. Really sorry about that :-(

I've re-landed @zturner's patch and included this fix into rG0f1f13fcb17fbc8c93d505da989a04ab5cbd9ed3.

Ping!

In D92705#2437806, @rnk wrote:

We should raise the default _MSC_VER, but IMO that's separable, and I wouldn't rely on it. For cross-compilation, I would recommend setting an explicit version. Otherwise, the very host-specific auto-detection logic could run, and it might produce strange results on a Linux system.

It would be nice if this was covered by a test. I essentially did what you did, but for Windows and added a test in D92419. @dim do you think that test makes sense for Unix as well? (it would need the 'AffinityMask' to be implemented in llvm/lib/Support/Unix/Program.inc)

In D91418#2431461, @sbc100 wrote:

In D91418#2431400, @rksharma wrote:

In D91418#2431314, @MaskRay wrote:

I think changing from main to to avoids an issue and adds a new issue. We should fix the issues all together, and this requires consensus from others ports to proceed.

Is the new issue, the clash with lld::*::link and if so, then ideally we should replace link from all the ports to linkerMain?

lldMain?

This is consistent with the other drivers, LGTM.
Do you have commit access? Please ask for it if you're planning more changes in the future. Otherwise I can land it for you.

Ping.

In D91418#2394527, @sbc100 wrote:

However I was under the impression that it was a non-goal to make the current version of lld usable as a library. I mean it make very heavy use of globals as well as things like early process exiting doesn't it? Have the goals changed recently?

In D83025#2389486, @hokein wrote:

In D83025#2389140, @aganea wrote:

We're seeing the same issue mentionned in PR45498, but with the repro below. git bisect points to this patch. Could anyone please possibly confirm? Simply build clang with -DLLVM_ENABLE_ASSERTIONS=ON.

// compile with: clang-cl /c a.cpp
template < a > struct b;
template < bool a, class > using c = b< a >;
template < class > using d = void ;
template < class, class, class = void >
bool e template < class f, class g >
bool e< f, g, d< decltype(h(g())) > > template < class f, class g >
void i(f, g ) {
  e< f, g >
}
template < class _BidIt2, c< e< _BidIt, _BidIt2 >, int > = 0 >
void h(_BidIt2) short do_tolower__Last {
  i(do_tolower__First, do_tolower__Last)
}

thanks for the testcase. I'm not sure the patch is the root cause -- from the Bugzilla thread, the issue was reported in April which is prior to this patch.

btw, the issue seems to be gone on the trunk, I don't reproduce them with clang building from the trunk, I guess it might be fixed by some patches after llvm-11.

We're seeing the same issue mentionned in PR45498, but with the repro below. git bisect points to this patch. Could anyone please possibly confirm? Simply build clang with -DLLVM_ENABLE_ASSERTIONS=ON.

// compile with: clang-cl /c a.cpp
template < a > struct b;
template < bool a, class > using c = b< a >;
template < class > using d = void ;
template < class, class, class = void >
bool e template < class f, class g >
bool e< f, g, d< decltype(h(g())) > > template < class f, class g >
void i(f, g ) {
  e< f, g >
}
template < class _BidIt2, c< e< _BidIt, _BidIt2 >, int > = 0 >
void h(_BidIt2) short do_tolower__Last {
  i(do_tolower__First, do_tolower__Last)
}

In D88348#2388543, @bd1976llvm wrote:

Will this land soon? We are eagerly anticipating its arrival :)

Simplify.
Added coverage when using clang-cl since this is a MSVC-specific feature.
When -fdebug-compilation-dir=. is used, relative paths are now emitted in the S_OBJNAME record.

LGTM, thanks!

I'd like to fix the lld.ELF/invalid::symtab-sh-info.s that fails for a while, and this patch has the fix.

In D84380#2330715, @amccarth wrote:

I ask because I'm still trying to figure out why 899 LLDB tests started failing for me on Friday. (Actually, it was 936 tests, but I fixed 37 of them with a fix to a line that was introduced almost a year ago. I have not idea why the problems just started appearing on the 9th.)

In D87966#2328818, @tejohnson wrote:

In D87966#2328736, @aganea wrote:

@tejohnson I added the test in LLD, since the gold tests only run on Linux, which is harder for me to test & debug. The test fails when the following block is removed: if (BackendProc->getThreadCount() == 1) { ... }.

Great, thanks! LGTM with request for comment as described below.

This test also implictly covers the "InProcessThinBackend" codepath with /opt:lldltojobs=1 which I don't see how to cover explicitly otherwise.

I don't see how this test covers that code path since it isn't using in process backends, but I'm not really sure how to test this case effectively - like the original patch, it is just a performance change on the in process backends case.

@tejohnson I added the test in LLD, since the gold tests only run on Linux, which is harder for me to test & debug. The test fails when the following block is removed: if (BackendProc->getThreadCount() == 1) { ... }.
This test also implictly covers the "InProcessThinBackend" codepath with /opt:lldltojobs=1 which I don't see how to cover explicitly otherwise.

Simplify & address comments.

In D88348#2328425, @rnk wrote:

The code changes seem fine, but I haven't figured out why we need to sprinkle LLD_IN_TEST=1 for every LLD invocation we expect to error. What goes wrong exactly? I thought the point of your change is that, we *won't* re-run LLD when LLD_IN_TEST=2 after it reports an error.

In D84380#2315686, @aganea wrote:

Also, worth nothing that C:\Program Files\Git\usr\bin is not enough to run the lldb tests, which need make and maybe other things.
I'm not sure if that was part of your original goal, but since clang changes can affect lldb behavior, that means developers would still need some additionnal tools along Git for Windows.

Check for WriteIndexesThinBackend before re-ordering modules.

As suggested by @hans.

Sorry for the very late answer. The crash above was induced by something else commited at that time, not related to this patch.
I've re-landed again in https://reviews.llvm.org/rG66face6aa0f5