[LLD][COFF] When using LLD-as-a-library, always prevent re-entrance on failures

[aganea]

Summary

This is a follow-up for D70378 (Cover usage of LLD as a library).

I'm increasingly convinced that it is impossible to support re-entrance once a failure occurs in LLD, even if that currently works on the surface.

While debugging an intermittent failure on a bot, I recalled this scenario which causes the issue:

1. When executing lld/test/ELF/invalid/symtab-sh-info.s L45, we reach lld::elf::ObjFile::ObjFile() which goes straight into its base ELFFileBase(), then ELFFileBase::init().
2. At that point fatal() is thrown in lld/ELF/InputFiles.cpp L381, leaving a half-initialized ObjFile instance.
3. We end up in lld::exitLld() and since we are running with LLD_IN_TEST, we hapily restore the control flow to CrashRecoveryContext::RunSafely() then back in lld::safeLldMain().
4. Before this patch, we called errorHandler().reset() just after, and this attempted to reset the associated SpecificAlloc<ObjFile<ELF64LE>>. That tried to free the half-initialized ObjFile instance, and more precisely its ObjFile::dwarf member.

Sometimes that worked, sometimes it failed and was catched by the CrashRecoveryContext. This scenario was the reason we called errorHandler().reset() through a CrashRecoveryContext.

But in some rare cases, the above repro somehow corrupted the heap, creating a stack overflow. When the CrashRecoveryContext's filter (that is, __except (ExceptionFilter(GetExceptionInformation()))) tried to handle the exception, it crashed again since the stack was exhausted -- and that took the whole application down. That is the issue seen on the bot. Locally it happens about 1 times out of 15.

Now this situation can happen anywhere in LLD. Since catching stack overflows is not a reliable scenario when using CrashRecoveryContext, we're now preventing further re-entrance when such failures occur, by signaling lld::SafeReturn::canRunAgain=false. When running with LLD_IN_TEST=2, only one iteration will be executed in this case.

[aganea]

Diff with context.

[rnk]

I think I'm OK with the functional change, but I think I would prefer not to keep the code that buffers stdout/stderr for the lit test. I think the complexity/size cost outweighs the test benefit.

[MaskRay]

Since catching stack overflows is not a reliable scenario when using CrashRecoveryContext

I have seen this in a language server. It seems that CrashRecoveryContext does not (correctly) use sigaltstack(). With sigaltstack, we can recover after a stack overflow.
For the stack overflow you observed, how did that happen?

In D88348#2299551, @rnk wrote:

I think I'm OK with the functional change, but I think I would prefer not to keep the code that buffers stdout/stderr for the lit test. I think the complexity/size cost outweighs the test benefit.

I share the same feeling. Hope bufferedStdout can be avoided

llvm/lib/Support/CrashRecoveryContext.cpp

447

This deserves a comment

llvm/lib/Support/Process.cpp

24

_Exit does not require unistd.h

[aganea]

In D88348#2299620, @MaskRay wrote:

I have seen this in a language server. It seems that CrashRecoveryContext does not (correctly) use sigaltstack(). With sigaltstack, we can recover after a stack overflow.

Good to know at least it could work on Linux. I wanted to activate coverage for this at some point in D74229 but then it wasn't working on Debug builds on Windows. I'll need to get back to it.

For the stack overflow you observed, how did that happen?

The stack overflow happens in the first place somewhere in the Windows CRT while releasing heap blocks. This is rare, as it depends on the random pattern that was in memory at the location of ObjFile::dwarf when fatal() was called by ELFFileBase::init().

In D88348#2299551, @rnk wrote:

I think I'm OK with the functional change, but I think I would prefer not to keep the code that buffers stdout/stderr for the lit test. I think the complexity/size cost outweighs the test benefit.

I share the same feeling. Hope bufferedStdout can be avoided

I'll try to find another way. The issue is that we're blocking stdout/stderr on all iterations, but the last one. Which doesn't work if fatal() occurs in the first iteration (as expected, in a test). Perhaps I could simply set LLD_IN_TEST=1 for tests that expect fatal() or a crash.

[aganea]

Address comments. Removed stdout/stderr buffering, and replaced with env LLD_IN_TEST=1 set appropriately in tests, where needed.

[aganea]

Ping! Any further comments?

[rnk]

The code changes seem fine, but I haven't figured out why we need to sprinkle LLD_IN_TEST=1 for every LLD invocation we expect to error. What goes wrong exactly? I thought the point of your change is that, we *won't* re-run LLD when LLD_IN_TEST=2 after it reports an error.

[aganea]

In D88348#2328425, @rnk wrote:

The code changes seem fine, but I haven't figured out why we need to sprinkle LLD_IN_TEST=1 for every LLD invocation we expect to error. What goes wrong exactly? I thought the point of your change is that, we *won't* re-run LLD when LLD_IN_TEST=2 after it reports an error.

When LLD_IN_TEST>0 we only emit stdout/stderr on the last iteration. But if we crash, FileCheck requires for stout & stderr to be emitted before the crash. That was the point for buffering the streams in the previous version of this patch. Either that, or LLD_IN_TEST=1 below. If you think of other ideas, I would be willing to try!

[aganea]

Ping.

I'd like to fix the lld.ELF/invalid::symtab-sh-info.s that fails for a while, and this patch has the fix.

If you think this whole LLD-as-a-library should go into a different direction please let me know.

[MaskRay]

Have the stdout/stderr changes be removed? The updated diff looks good to me. I agree that in the absence of exceptions (proper stack unwinding), we cannot safely recover from fatal(). For library use cases, if they can ensure no fatal() is called, they are probably still safe.

Marking LLD_IN_TEST=1 seems fine. It serves as a hint that abort() is called. LG with some nits, though I still hope @rnk or @amccarth can take a look.

llvm/lib/Support/CrashRecoveryContext.cpp

445

Nit: newer functions should stick with the naming rule. For Exit, you can keep it capitalized to reduce disruption since you just add a default argument which does not require all call sites to change.

llvm/lib/Support/Process.cpp

101

This needs stdlib.h

[bd1976llvm]

Will this land soon? We are eagerly anticipating its arrival :)

[aganea]

In D88348#2388543, @bd1976llvm wrote:

Will this land soon? We are eagerly anticipating its arrival :)

Yes, today! :) Let me finish a few things, I'll land it around noon EST.

lld/Common/ErrorHandler.cpp

raw_ostream &lld::errs() {
   return stderrOS ? *stderrOS : llvm::errs();
 }
 
 void lld::exitLld(int val) {
   // Delete any temporary file, while keeping the memory mapping open.
   if (errorHandler().outputBuffer)
     errorHandler().outputBuffer->discard();
 
+  // Re-throw a possible signal or exception once/if it was catched by
+  // safeLldMain().
+  CrashRecoveryContext::throwIfCrash(val);
+
   // Dealloc/destroy ManagedStatic variables before calling _exit().
   // In an LTO build, allows us to get the output of -time-passes.
   // Ensures that the thread pool for the parallel algorithms is stopped to
   // avoid intermittent crashes on Windows when exiting.
   if (!CrashRecoveryContext::GetCurrent())
     llvm_shutdown();
 
   {
     std::lock_guard<std::mutex> lock(mu);
     lld::outs().flush();
     lld::errs().flush();
   }
-  llvm::sys::Process::Exit(val);
+  // When running inside safeLldMain(), restore the control flow back to the
+  // CrashRecoveryContext. Otherwise simply use _exit(), meanning no cleanup,
+  // since we want to avoid further crashes on shutdown.
+  llvm::sys::Process::Exit(val, /*NoCleanup=*/true);
 }
 
 void lld::diagnosticHandler(const DiagnosticInfo &di) {
   SmallString<128> s;
   raw_svector_ostream os(s);
   DiagnosticPrinterRawOStream dp(os);
   di.print(dp);
   switch (di.getSeverity()) {

lld/include/lld/Common/Driver.h

 struct SafeReturn {
   int ret;
   bool canRunAgain;
 };
 
 // Generic entry point when using LLD as a library, safe for re-entry, supports
 // crash recovery. Returns a general completion code and a boolean telling
 // whether it can be called again. In some cases, a crash could corrupt memory
-// and re-entry would not be possible anymore.
+// and re-entry would not be possible anymore. Use exitLld() in that case to
+// properly exit your application and avoid intermittent crashes on exit caused
+// by cleanup.
 SafeReturn safeLldMain(int argc, const char **argv, llvm::raw_ostream &stdoutOS,
                        llvm::raw_ostream &stderrOS);
 
 namespace coff {
 bool link(llvm::ArrayRef<const char *> args, bool canExitEarly,
           llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS);
 }
 

lld/test/COFF/arm-thumb-branch20-error.s

 // REQUIRES: arm
 // RUN: llvm-mc -filetype=obj -triple=thumbv7a-windows-gnu %s -o %t.obj
-// RUN: not lld-link -entry:_start -subsystem:console %t.obj -out:%t.exe 2>&1 | FileCheck %s
+// RUN: env LLD_IN_TEST=1 not lld-link -entry:_start -subsystem:console %t.obj -out:%t.exe 2>&1 | FileCheck %s
  .syntax unified
  .globl _start
 _start:
  bne too_far20
  .space 0x100000
  .section .text$a, "xr"
 too_far20:
  bx lr
 
 // When trying to add a thunk at the end of the section, the thunk itself
 // will be too far away, so this won't converge.
 
 // CHECK: adding thunks hasn't converged

lld/test/COFF/comdat-selection.s

 # link.exe (and hence lld-link) doesn't understand.
 
 # RUN: cp %t.discard.obj %t.obj && lld-link /dll /noentry /nodefaultlib %t.discard.obj %t.obj
 # RUN: cp %t.one_only.obj %t.obj && not lld-link /dll /noentry /nodefaultlib %t.one_only.obj %t.obj 2>&1 | FileCheck --check-prefix=ONEONE %s
 # ONEONE: lld-link: error: duplicate symbol: symbol
 # RUN: cp %t.same_size.obj %t.obj && lld-link /dll /noentry /nodefaultlib %t.same_size.obj %t.obj
 # RUN: cp %t.same_contents.obj %t.obj && lld-link /dll /noentry /nodefaultlib %t.same_contents.obj %t.obj
 # RUN: cp %t.largest.obj %t.obj && lld-link /dll /noentry /nodefaultlib %t.largest.obj %t.obj
-# RUN: cp %t.newest.obj %t.obj && not lld-link /dll /noentry /nodefaultlib %t.newest.obj %t.obj 2>&1 | FileCheck --check-prefix=NEWNEW %s
+# RUN: cp %t.newest.obj %t.obj && env LLD_IN_TEST=1 not lld-link /dll /noentry /nodefaultlib %t.newest.obj %t.obj 2>&1 | FileCheck --check-prefix=NEWNEW %s
 # NEWNEW: lld-link: error: unknown comdat type 7 for symbol
 
 # /force doesn't affect errors about unknown comdat types.
-# RUN: cp %t.newest.obj %t.obj && not lld-link /force /dll /noentry /nodefaultlib %t.newest.obj %t.obj 2>&1 | FileCheck --check-prefix=NEWNEWFORCE %s
+# RUN: cp %t.newest.obj %t.obj && env LLD_IN_TEST=1 not lld-link /force /dll /noentry /nodefaultlib %t.newest.obj %t.obj 2>&1 | FileCheck --check-prefix=NEWNEWFORCE %s
 # NEWNEWFORCE: lld-link: error: unknown comdat type 7 for symbol
 
 # Check that same_size, same_contents, largest do what they're supposed to.
 
 # Check that the "same_size" selection produces an error if passed two symbols
 # with different size.
 # RUN: not lld-link /dll /noentry /nodefaultlib %t.same_size.obj %t.same_size.short.obj 2>&1 | FileCheck --check-prefix=SAMESIZEDUPE %s
 # SAMESIZEDUPE: lld-link: error: duplicate symbol: symbol

lld/test/COFF/delayimports-error.test

 # RUN: mkdir -p %t.dir
 # RUN: yaml2obj %p/Inputs/delayimports-error.yaml -o %t1.obj
 # RUN: lld-link /out:%t.dir/foo.dll /dll %t1.obj /export:datasym,DATA /noentry
 
 # RUN: yaml2obj %s -o %t2.obj
-# RUN: not lld-link /out:%t.exe /entry:main %t2.obj %t.dir/foo.lib /delayload:foo.dll \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main %t2.obj \
+# RUN:   %t.dir/foo.lib /delayload:foo.dll \
 # RUN:   /alternatename:__delayLoadHelper2=main /opt:noref >& %t.log
 # RUN: FileCheck %s < %t.log
 
 # CHECK: cannot delay-load foo.dll due to import of data: __declspec(dllimport) datasym
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_AMD64

lld/test/COFF/driver-windows.test

 # REQUIRES: system-windows
-# RUN: not LLD-LINK 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not LLD-LINK 2>&1 | FileCheck %s
 CHECK: no input files

lld/test/COFF/driver.test

 # RUN: not lld-link nosuchfile.obj >& %t.log
 # RUN: FileCheck -check-prefix=MISSING %s < %t.log
 MISSING: 'nosuchfile.obj': {{[Nn]}}o such file or directory
 
 # RUN: lld-link --version | FileCheck -check-prefix=VERSION %s
 VERSION: {{LLD [0-9]+\.[0-9]+}}
 
 # RUN: yaml2obj %p/Inputs/export.yaml -o %t.obj
 # RUN: lld-link /out:%t.dll /dll %t.obj
 # RUN: not lld-link /out:%t.exe %t.dll 2>&1 | FileCheck -check-prefix=BADFILE %s
 BADFILE: bad file type. Did you specify a DLL instead of an import library?
 
 # RUN: lld-link /lib /help | FileCheck -check-prefix=LIBHELP %s
 LIBHELP: OVERVIEW: LLVM Lib
 
-# RUN: not lld-link /WX /lib 2>&1 | FileCheck -check-prefix=LIBBAD %s
+# RUN: env LLD_IN_TEST=1 not lld-link /WX /lib 2>&1 | FileCheck -check-prefix=LIBBAD %s
 LIBBAD: ignoring /lib since it's not the first argument
 
 # RUN: yaml2obj %p/Inputs/hello32.yaml -o %t.obj
 # RUN: not lld-link /out:/ %t.obj 2>&1 | FileCheck -check-prefix=DIR %s
 DIR: cannot open output file
 
-# RUN: not lld-link -version 2>&1 | FileCheck -check-prefix=SPELLVERSION %s
+# RUN: env LLD_IN_TEST=1 not lld-link -version 2>&1 | FileCheck -check-prefix=SPELLVERSION %s
 SPELLVERSION: ignoring unknown argument '-version', did you mean '--version'
 SPELLVERSION: no input files
 
-# RUN: not lld-link -nodefaultlibs 2>&1 | FileCheck -check-prefix=SPELLNODEFAULTLIB %s
+# RUN: env LLD_IN_TEST=1 not lld-link -nodefaultlibs 2>&1 | FileCheck -check-prefix=SPELLNODEFAULTLIB %s
 SPELLNODEFAULTLIB: ignoring unknown argument '-nodefaultlibs', did you mean '-nodefaultlib'
 SPELLNODEFAULTLIB: no input files
 
 # RUN: not lld-link /nodefaultlibs 2>&1 | FileCheck -check-prefix=SPELLNODEFAULTLIB_SLASH %s
 SPELLNODEFAULTLIB_SLASH: could not open '/nodefaultlibs': {{.*}}; did you mean '/nodefaultlib'
 SPELLNODEFAULTLIB_SLASH-NOT: no input files
 
 # Getting flags as typo corrections for normal input files is a side effect
 # of how spell checking for /-style flags is implemented.
 # RUN: not lld-link force 2>&1 | FileCheck -check-prefix=SPELLFORCE %s
 SPELLFORCE: could not open 'force': {{.*}}; did you mean '/force'
 SPELLFORCE-NOT: no input files

lld/test/COFF/export-limit.s

 # REQUIRES: x86
 # RUN: %python %p/Inputs/def-many.py 65535 > %t-65535.def
 # RUN: %python %p/Inputs/def-many.py 65536 > %t-65536.def
 # RUN: llvm-mc -triple x86_64-win32 %s -filetype=obj -o %t.obj
 # RUN: lld-link -dll -noentry %t.obj -out:%t.dll -def:%t-65535.def
-# RUN: not lld-link -dll -noentry %t.obj -out:%t.dll -def:%t-65536.def 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not lld-link -dll -noentry %t.obj -out:%t.dll -def:%t-65536.def 2>&1 | FileCheck %s
 
 # CHECK: error: too many exported symbols
 
         .text
         .globl f
 f:
         ret

lld/test/COFF/failifmismatch.test

 REQUIRES: x86
 
 RUN: lld-link /entry:main /subsystem:console /out:%t.exe \
 RUN:   %p/Inputs/ret42.obj
 
 RUN: lld-link /entry:main /subsystem:console /out:%t.exe \
 RUN:   %p/Inputs/ret42.obj /failifmismatch:k1=v1 /failifmismatch:k2=v1
 
 RUN: lld-link /entry:main /subsystem:console /out:%t.exe \
 RUN:   %p/Inputs/ret42.obj /failifmismatch:k1=v1 /failifmismatch:k1=v1
 
-RUN: not lld-link /entry:main /subsystem:console /out:%t.exe \
+RUN: env LLD_IN_TEST=1 not lld-link /entry:main /subsystem:console /out:%t.exe \
 RUN:   %p/Inputs/ret42.obj /failifmismatch:k1=v1 /failifmismatch:k1=v2 2>&1 | FileCheck %s 
 
 RUN: llc < %p/Inputs/failmismatch1.ll -mtriple x86_64-windows-msvc -filetype obj -o %t1.obj
 RUN: llc < %p/Inputs/failmismatch2.ll -mtriple x86_64-windows-msvc -filetype obj -o %t2.obj
-RUN: not lld-link %t1.obj %t2.obj 2>&1 | FileCheck %s -check-prefix OBJ
+RUN: env LLD_IN_TEST=1 not lld-link %t1.obj %t2.obj 2>&1 | FileCheck %s -check-prefix OBJ
 
 RUN: llvm-lib %t1.obj /out:%t.lib
-RUN: not lld-link %t.lib %t2.obj 2>&1 | FileCheck %s -check-prefix LIB
+RUN: env LLD_IN_TEST=1 not lld-link %t.lib %t2.obj 2>&1 | FileCheck %s -check-prefix LIB
 
 CHECK: lld-link: error: /failifmismatch: mismatch detected for 'k1':
 CHECK-NEXT: >>> cmd-line has value v1
 CHECK-NEXT: >>> cmd-line has value v2
 
 OBJ: lld-link: error: /failifmismatch: mismatch detected for 'TEST':
 OBJ-NEXT: >>> {{.*}}failifmismatch.test.tmp1.obj has value 1
 OBJ-NEXT: >>> {{.*}}failifmismatch.test.tmp2.obj has value 2
 
 LIB: lld-link: error: /failifmismatch: mismatch detected for 'TEST':
 LIB-NEXT: >>> {{.*}}failifmismatch.test.tmp2.obj has value 2
 LIB-NEXT: >>> failifmismatch.test.tmp.lib(failifmismatch.test.tmp1.obj) has value 1

lld/test/COFF/invalid-obj.test

 # RUN: yaml2obj %s -o %t.obj
-# RUN: not lld-link %t.obj 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not lld-link %t.obj 2>&1 | FileCheck %s
 
 # CHECK: getSectionName failed: #1:
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_AMD64
   Characteristics: []
 sections:
   - Name:            '/1'
     Characteristics: []
     SectionData:     00
 symbols:

lld/test/COFF/invalid-section-number.test

 # RUN: yaml2obj %s -o %t.obj
-# RUN: not lld-link %t.obj 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not lld-link %t.obj 2>&1 | FileCheck %s
 
 # CHECK: foo should not refer to special section -10
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_I386
   Characteristics: []
 sections:

lld/test/COFF/linkenv.test

 # RUN: env LINK=-help lld-link | FileCheck %s
 # RUN: env _LINK_=-help lld-link | FileCheck %s
 
 CHECK: OVERVIEW: LLVM Linker
 
-# RUN: env LINK=-help not lld-link /lldignoreenv 2>&1 | \
+# RUN: env LLD_IN_TEST=1 LINK=-help not lld-link /lldignoreenv 2>&1 | \
 # RUN:     FileCheck --check-prefix=ERR %s
 
 ERR: error: no input files

lld/test/COFF/manifestinput-error.test

 # UNSUPPORTED: manifest_tool
 # UNSUPPORTED: libxml2
 
 # RUN: yaml2obj %p/Inputs/ret42.yaml -o %t.obj
-# RUN: not lld-link /out:%t.exe /entry:main \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main \
 # RUN:   /manifest:embed \
 # RUN:   /manifestuac:"level='requireAdministrator'" \
 # RUN:   /manifestinput:%p/Inputs/manifestinput.test %t.obj 2>&1 | FileCheck %s
 
 # CHECK: error: unable to find mt.exe in PATH: {{[Nn]}}o such file or directory

lld/test/COFF/merge.test

 # RUN: yaml2obj %s -o %t.obj
 # RUN: lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.abc /merge:.bar=.def %t.obj /debug
 # RUN: llvm-readobj --sections %t.exe | FileCheck %s
 
 # RUN: lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.bar /merge:.bar=.abc %t.obj /debug
 # RUN: llvm-readobj --sections %t.exe | FileCheck --check-prefix=CHECK2 %s
 
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.rsrc=.foo %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-RSRC %s
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.rsrc %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-RSRC %s
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.reloc=.foo %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-RELOC %s
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.reloc %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-RELOC %s
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.foo1 /merge:.foo1=.foo %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-CYCLE %s
-# RUN: not lld-link /out:%t.exe /entry:main /subsystem:console /force \
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main /subsystem:console /force \
 # RUN:   /merge:.foo=.foo1 /merge:.foo1=.foo2 /merge:.foo2=.foo1 %t.obj /debug 2>&1 | FileCheck --check-prefix=NO-CYCLE %s
 
 # CHECK: Name: .def
 # CHECK: Name: .abc
 
 # CHECK2-NOT: Name: .bar
 # CHECK2: Name: .abc
 # CHECK2-NOT: Name: .bar

lld/test/COFF/pdata-arm64-bad.yaml

 # RUN: yaml2obj %s -o %t.obj
-# RUN: not lld-link /out:%t.exe /entry:func1 /subsystem:console %t.obj 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:func1 /subsystem:console %t.obj 2>&1 | FileCheck %s
 
 # This file is like pdata-arm64.yaml, except that .pdata has been extended with
 # 4 bytes. This can happen due to for example bad assembler input. Check that
 # lld errors gracefully instead of crashing.
 
 # CHECK: unexpected .pdata size: 20 is not a multiple of 8
 
 --- !COFF

lld/test/COFF/precomp-link.test

ObjNameSym:
       ObjectName:      'F:\svn\lld\test\COFF\precomp\precomp.obj'
 
 RUN: obj2yaml %S/Inputs/precomp.obj | grep -v 'SectionData: *04000000' > %t.precomp.yaml
 RUN: sed '/S_OBJNAME/,/ObjectName:/d' < %t.precomp.yaml > precomp-no-objname.yaml
 RUN: sed 's/Signature: *545589255/Signature: 0/' < %t.precomp.yaml > precomp-zero-sig.yaml
 RUN: yaml2obj precomp-no-objname.yaml -o %t.precomp-no-objname.obj
 RUN: yaml2obj precomp-zero-sig.yaml -o %t.precomp-zero-sig.obj
 
-RUN: not lld-link %t.precomp-no-objname.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-NO-SIGNATURE
+RUN: env LLD_IN_TEST=1 not lld-link %t.precomp-no-objname.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-NO-SIGNATURE
 
-RUN: not lld-link %t.precomp-zero-sig.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-NO-SIGNATURE
+RUN: env LLD_IN_TEST=1 not lld-link %t.precomp-zero-sig.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-NO-SIGNATURE
 
 FAILURE-NO-SIGNATURE: error: {{.*}}.obj claims to be a PCH object, but does not have a valid signature
 
 Check that two PCH objs with duplicate signatures are an error.
 
 RUN: cp %S/Inputs/precomp.obj %t.precomp-dup.obj
 
-RUN: not lld-link %S/Inputs/precomp.obj %t.precomp-dup.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-DUP-SIGNATURE
+RUN: env LLD_IN_TEST=1 not lld-link %S/Inputs/precomp.obj %t.precomp-dup.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib /entry:main /debug /pdb:%t.pdb /out:%t.exe 2>&1 | FileCheck %s -check-prefix FAILURE-DUP-SIGNATURE
 
 FAILURE-DUP-SIGNATURE: error: a PCH object with the same signature has already been provided ({{.*precomp.obj and .*precomp-dup.obj.*}})
 
 
 CHECK: Types (TPI Stream)
 CHECK-NOT: LF_PRECOMP
 CHECK-NOT: LF_ENDPRECOMP
 

lld/test/COFF/thin-archive.s

 # RUN: lld-link /entry:main %t.main.obj %t_thin.lib /out:%t.exe 2>&1 | \
 # RUN:     FileCheck --allow-empty %s
 # RUN: lld-link /entry:main %t.main.obj /wholearchive:%t_thin.lib /out:%t.exe 2>&1 | \
 # RUN:     FileCheck --allow-empty %s
 
 # RUN: rm %t.lib.obj
 # RUN: lld-link /entry:main %t.main.obj %t.lib /out:%t.exe 2>&1 | \
 # RUN:     FileCheck --allow-empty %s
-# RUN: not lld-link /entry:main %t.main.obj %t_thin.lib /out:%t.exe 2>&1 | \
-# RUN:     FileCheck --check-prefix=NOOBJ %s
-# RUN: not lld-link /entry:main %t.main.obj %t_thin.lib /out:%t.exe \
+# RUN: env LLD_IN_TEST=1 not lld-link /entry:main %t.main.obj %t_thin.lib \
+# RUN:     /out:%t.exe 2>&1 | FileCheck --check-prefix=NOOBJ %s
+# RUN: env LLD_IN_TEST=1 not lld-link /entry:main %t.main.obj %t_thin.lib /out:%t.exe \
 # RUN:     /demangle:no 2>&1 | FileCheck --check-prefix=NOOBJNODEMANGLE %s
 
 # CHECK-NOT: error: could not get the buffer for the member defining
 # NOOBJ: error: could not get the buffer for the member defining symbol int __cdecl f(void): {{.*}}.lib({{.*}}.lib.obj):
 # NOOBJNODEMANGLE: error: could not get the buffer for the member defining symbol ?f@@YAHXZ: {{.*}}.lib({{.*}}.lib.obj):
 
   .text
 
   .def main
     .scl 2
     .type 32
   .endef
   .global main
 main:
   call "?f@@YAHXZ"
   retq $0

lld/test/COFF/thunk-replace.s

 # REQUIRES: x86
 
 # RUN: llvm-mc -triple=x86_64-win32 %s -filetype=obj -o %t.main.obj
 # RUN: llvm-mc -triple=x86_64-win32 %p/Inputs/otherFunc.s -filetype=obj -o %t.other.obj
 # RUN: llvm-ar rcs %t.other.lib %t.other.obj
-# RUN: not lld-link -out:%t.exe -entry:main %t.main.obj %p/Inputs/std64.lib %t.other.lib -opt:noref 2>&1 | FileCheck %s
+# RUN: env LLD_IN_TEST=1 not lld-link -out:%t.exe -entry:main %t.main.obj \
+# RUN:     %p/Inputs/std64.lib %t.other.lib -opt:noref 2>&1 | FileCheck %s
 # CHECK: MessageBoxA was replaced
 
 .global main
 .text
 main:
   callq MessageBoxA
   callq ExitProcess
   callq otherFunc
   ret

lld/tools/lld/lld.cpp

SafeReturn lld::safeLldMain(int argc, const char **argv,
   {
     // The crash recovery is here only to be able to recover from arbitrary
     // control flow when fatal() is called (through setjmp/longjmp or
     // __try/__except).
     llvm::CrashRecoveryContext crc;
     if (!crc.RunSafely([&]() {
           r = lldMain(argc, argv, stdoutOS, stderrOS, /*exitEarly=*/false);
         }))
-      r = crc.RetCode;
+      return {crc.RetCode, /*canRunAgain=*/false};
   }
 
   // Cleanup memory and reset everything back in pristine condition. This path
   // is only taken when LLD is in test, or when it is used as a library.
   llvm::CrashRecoveryContext crc;
   if (!crc.RunSafely([&]() { errorHandler().reset(); })) {
     // The memory is corrupted beyond any possible recovery.
     return {r, /*canRunAgain=*/false};
int main(int argc, const char **argv) {
   for (unsigned i = inTestVerbosity(); i > 0; --i) {
     // Disable stdout/stderr for all iterations but the last one.
     if (i != 1)
       errorHandler().disableOutput = true;
 
     // Execute one iteration.
     auto r = safeLldMain(argc, argv, llvm::outs(), llvm::errs());
     if (!r.canRunAgain)
-      _exit(r.ret); // Exit now, can't re-execute again.
+      exitLld(r.ret); // Exit now, can't re-execute again.
 
     if (!mainRet) {
       mainRet = r.ret;
     } else if (r.ret != *mainRet) {
       // Exit now, to fail the tests if the result is different between runs.
       return r.ret;
     }
   }
-#if LLVM_ON_UNIX
-  // Re-throw the signal so it can be caught by WIFSIGNALED in
-  // llvm/lib/Support/Unix/Program.inc. This is required to correctly handle
-  // usages of `not --crash`.
-  if (*mainRet > 128) {
-    llvm::sys::unregisterHandlers();
-    raise(*mainRet - 128);
-  }
-#endif
   return *mainRet;
 }

llvm/include/llvm/Support/CrashRecoveryContext.h

bool RunSafelyOnThread(void (*Fn)(void*), void *UserData,
     return RunSafelyOnThread([&]() { Fn(UserData); }, RequestedStackSize);
   }
 
   /// Explicitly trigger a crash recovery in the current process, and
   /// return failure from RunSafely(). This function does not return.
   LLVM_ATTRIBUTE_NORETURN
   void HandleExit(int RetCode);
 
+  /// Throw again a signal or an exception, after it was catched once by a
+  /// CrashRecoveryContext.
+  static bool throwIfCrash(int RetCode);
+
   /// In case of a crash, this is the crash identifier.
   int RetCode = 0;
 
   /// Selects whether handling of failures should be done in the same way as
   /// for regular crashes. When this is active, a crash would print the
   /// callstack, clean-up any temporary files and create a coredump/minidump.
   bool DumpStackAndCleanupOnFailure = false;
 };

llvm/include/llvm/Support/Process.h

public:
 
   /// Get the result of a process wide random number generator. The
   /// generator will be automatically seeded in non-deterministic fashion.
   static unsigned GetRandomNumber();
 
   /// Equivalent to ::exit(), except when running inside a CrashRecoveryContext.
   /// In that case, the control flow will resume after RunSafely(), like for a
   /// crash, rather than exiting the current process.
+  /// Use \arg NoCleanup for calling _exit() instead of exit().
   LLVM_ATTRIBUTE_NORETURN
-  static void Exit(int RetCode);
+  static void Exit(int RetCode, bool NoCleanup = false);
 };
 
 }
 }
 
 #endif

llvm/lib/Support/CrashRecoveryContext.cpp

#else
   // HandleCrash(), then longjmp will unwind the stack for us.
   CrashRecoveryContextImpl *CRCI = (CrashRecoveryContextImpl *)Impl;
   assert(CRCI && "Crash recovery context never initialized!");
   CRCI->HandleCrash(RetCode, 0 /*no sig num*/);
 #endif
   llvm_unreachable("Most likely setjmp wasn't called!");
 }
 
+bool CrashRecoveryContext::throwIfCrash(int RetCode) {
+#if defined(_WIN32)
+  // On Windows, the high bits are reserved for kernel return codes. Values
+  // starting with 0x80000000 are reserved for "warnings"; values of 0xC0000000
+  // and up are for "errors". In practice, both are interpreted as a
+  // non-continuable signal.
+  unsigned Code = ((unsigned)RetCode & 0xF0000000) >> 28;
+  if (Code != 0xC && Code != 8)
+    return false;
+  ::RaiseException(RetCode, 0, 0, NULL);
+#else
+  // On Unix, signals are represented by return codes of 128 or higher.
+  if (RetCode <= 128)
+    return false;
+  llvm::sys::unregisterHandlers();
+  raise(RetCode - 128);
+#endif
+  return true;
+}
+
 // FIXME: Portability.
 static void setThreadBackgroundPriority() {
 #ifdef __APPLE__
   setpriority(PRIO_DARWIN_THREAD, 0, PRIO_DARWIN_BG);
 #endif
 }
 
 static bool hasThreadBackgroundPriority() {

llvm/lib/Support/Process.cpp

 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Config/config.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 
+#include <stddef.h> // for _Exit
+
 using namespace llvm;
 using namespace sys;
 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only TRULY operating system
 //===          independent code.
 //===----------------------------------------------------------------------===//
 

 
 // A CMake option controls wheter we emit core dumps by default. An application
 // may disable core dumps by calling Process::PreventCoreFiles().
 static bool coreFilesPrevented = !LLVM_ENABLE_CRASH_DUMPS;
 
 bool Process::AreCoreFilesPrevented() { return coreFilesPrevented; }
 
 LLVM_ATTRIBUTE_NORETURN
-void Process::Exit(int RetCode) {
+void Process::Exit(int RetCode, bool NoCleanup) {
   if (CrashRecoveryContext *CRC = CrashRecoveryContext::GetCurrent())
     CRC->HandleExit(RetCode);
+
+  if (NoCleanup)
+    _Exit(RetCode);
+  else
     ::exit(RetCode);
 }
 
 // Include the platform-specific parts of this class.
 #ifdef LLVM_ON_UNIX
 #include "Unix/Process.inc"
 #endif
 #ifdef _WIN32
 #include "Windows/Process.inc"
 #endif