Disable llvm-symbolizer on some of the driver tests that are timing out

[ahatanak]

Summary

These tests don't require invoking llvm-symbolizer.

[ahatanak]

These tests are all timing out after 600 seconds. https://green.lab.llvm.org/green/job/clang-stage1-RA/34005/#showFailuresLink

[shafik]

Do you know why these started timing out? I saw this locally the other day but could not figure out the root cause.

[MaskRay]

In D148851#4285583, @shafik wrote:

Do you know why these started timing out? I saw this locally the other day but could not figure out the root cause.

D86170 provides some information about llvm-symbolizer performance with an unoptimized debug build (slow llvm-symbolizer + huge debug info => slowness in crashes).
I think we probably should add LLVM_DISABLE_SYMBOLIZATION=1 to the lit level, not in individual tests.

[dblaikie]

I'm OK with this, though I wouldn't mind a more robust/general solution to this - especially all gunit death tests have this problem too - they crash and spend significant time symbolizing, I think? So it'd be great if we could find some way to run those with symbolization disabled too, not sure if there would be a more general place that'd cover both of these cases (but it might be interesting to you & perhaps you can find a fix for gunit too, even if it's a distinct solution, if the problem in general is of interest)

[dblaikie]

I think we probably should add LLVM_DISABLE_SYMBOLIZATION=1 to the lit level, not in individual tests.

Though I'm not sure how to do that in a way that it doesn't apply to test that are genuinely failing, where a symbolized backtrace is helpful/important/exactly what want?

[ahatanak]

Disable llvm-symbolizer when running lit tests.

[ahatanak]

Enable llvm-symbolizer when running disable-crash-reports.test.

[dblaikie]

In D148851#4311266, @ahatanak wrote:

Disable llvm-symbolizer when running lit tests.

This seems problematic though - when lit tests fail it's quite helpful to get a symbolized stack trace.

[shafik]

In D148851#4328084, @dblaikie wrote:

In D148851#4311266, @ahatanak wrote:

Disable llvm-symbolizer when running lit tests.

This seems problematic though - when lit tests fail it's quite helpful to get a symbolized stack trace.

While I have seen this again just today with the driver tests and it is annoying, I would also prefer a more targeted solution if possible.

[ahatanak]

In D148851#4328345, @shafik wrote:

In D148851#4328084, @dblaikie wrote:

In D148851#4311266, @ahatanak wrote:

Disable llvm-symbolizer when running lit tests.

This seems problematic though - when lit tests fail it's quite helpful to get a symbolized stack trace.

While I have seen this again just today with the driver tests and it is annoying, I would also prefer a more targeted solution if possible.

OK, I can go back to the original solution if it's important to keep the symbolizer enabled.

[ahatanak]

Disable llvm-symbolizer individually.

[ahatanak]

ping

[aeubanks]

not will set LLVM_DISABLE_SYMBOLIZATION if we use not --crash

are these tests supposed to crash or gracefully fail? if they're supposed to crash then we should use not --crash

[arsenm]

I also think we need to revert or disable https://github.com/llvm/llvm-project/commit/cead4eceb01b935fae07bf4a7e91911b344d2fec

The symbolizer is unusably slow with it

[dblaikie]

In D148851#4285705, @dblaikie wrote:

I think we probably should add LLVM_DISABLE_SYMBOLIZATION=1 to the lit level, not in individual tests.

Though I'm not sure how to do that in a way that it doesn't apply to test that are genuinely failing, where a symbolized backtrace is helpful/important/exactly what want?

Ping @MaskRay thoughts on the tension between disabling it entirely, and it being useful for developers investigating real failures?

[dblaikie]

In D148851#4357360, @arsenm wrote:

I also think we need to revert or disable https://github.com/llvm/llvm-project/commit/cead4eceb01b935fae07bf4a7e91911b344d2fec

The symbolizer is unusably slow with it

I believe that issue's been addressed by 02e8eb1a438bdb1dc9a97aea75a8c9c748048039 / D145009 ?

[delcypher]

@ahatanak Thanks for working on this. I think that annotating these particular tests so that's clear that they are supposed to crash (and therefore symbolication is of no use) is a good change. If we want to make a more global change for tests I think that should be handled separately.

However, if not --crash does the job of disabling symbolication I think we should use that because it also documents that we expect the binary being executed to crash. If you make that change then I'm happy to approve.

clang/test/Driver/crash-diagnostics-dir-3.c

4

Should we use not --crash instead? It's more explicit that we are expecting a crash and IIRC should disable symbolization as well.

clang/test/Driver/crash-diagnostics-dir-3.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: not env CLANG_CRASH_DIAGNOSTICS_DIR=%t %clang -c %s -o - 2>&1 | FileCheck %s
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
 // CHECK: diagnostic msg: {{.*}}{{/|\\}}crash-diagnostics-dir-3.c.tmp{{(/|\\).*}}.c

clang/test/Driver/crash-diagnostics-dir.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: not %clang -fcrash-diagnostics-dir=%t -c %s -o - 2>&1 | FileCheck %s
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
 // CHECK: diagnostic msg: {{.*}}{{/|\\}}crash-diagnostics-dir.c.tmp{{(/|\\).*}}.c

clang/test/Driver/crash-report-clang-cl.cpp

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: not %clang_cl -fsyntax-only /Brepro /source-charset:utf-8 \
 // RUN:     -fcrash-diagnostics-dir=%t -- %s 2>&1 | FileCheck %s
 // RUN: cat %t/crash-report-clang-cl-*.cpp | FileCheck --check-prefix=CHECKSRC %s
 // RUN: cat %t/crash-report-clang-cl-*.sh | FileCheck --check-prefix=CHECKSH %s
 

clang/test/Driver/crash-report-header.h

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTIONS=1 not %clang -fsyntax-only %s 2>&1 | FileCheck %s
 // RUN: cat %t/crash-report-header-*.h | FileCheck --check-prefix=CHECKSRC "%s"
 // RUN: cat %t/crash-report-header-*.sh | FileCheck --check-prefix=CHECKSH "%s"
 // REQUIRES: crash-recovery
 
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
 // CHECK-NEXT: note: diagnostic msg: {{.*}}.h
 FOO
 // CHECKSRC: FOO
 // CHECKSH: "-cc1"
 // CHECKSH: "-main-file-name" "crash-report-header.h"
 // CHECKSH: "crash-report-header-{{[^ ]*}}.h"

clang/test/Driver/crash-report-spaces.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf "%t"
 // RUN: mkdir "%t"
 // RUN: cp "%s" "%t/crash report spaces.c"
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTIONS=1 not %clang -fsyntax-only "%t/crash report spaces.c" 2>&1 | FileCheck "%s"
 // RUN: cat "%t/crash report spaces"-*.c | FileCheck --check-prefix=CHECKSRC "%s"
 // RUN: cat "%t/crash report spaces"-*.sh | FileCheck --check-prefix=CHECKSH "%s"
 // REQUIRES: crash-recovery
 
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
 // CHECK-NEXT: note: diagnostic msg: {{.*}}.c
 FOO
 // CHECKSRC: FOO
 // CHECKSH: "-cc1"
 // CHECKSH: "-main-file-name" "crash report spaces.c"
 // CHECKSH: "crash report spaces-{{[^ ]*}}.c"

clang/test/Driver/crash-report-with-asserts.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: echo '-fsyntax-only                                              \
 // RUN:  -F/tmp/ -I /tmp/ -idirafter /tmp/ -iquote /tmp/ -isystem /tmp/  \
 // RUN:  -iprefix /the/prefix -iwithprefix /tmp -iwithprefixbefore /tmp/ \
 // RUN:  -Xclang -internal-isystem -Xclang /tmp/                         \
 // RUN:  -Xclang -internal-externc-isystem -Xclang /tmp/                 \

clang/test/Driver/crash-report.cpp

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: echo '-fsyntax-only                                              \
 // RUN:  -F/tmp/ -I /tmp/ -idirafter /tmp/ -iquote /tmp/ -isystem /tmp/  \
 // RUN:  -iprefix /the/prefix -iwithprefix /tmp -iwithprefixbefore /tmp/ \
 // RUN:  -Xclang -internal-isystem -Xclang /tmp/                         \
 // RUN:  -Xclang -internal-externc-isystem -Xclang /tmp/                 \

clang/test/Driver/emit-reproducer.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf %t && mkdir %t
 
 // RUN: echo "%s -fcrash-diagnostics-dir=%t -fsyntax-only" | sed -e 's/\\/\\\\/g' > %t.rsp
 
 // RUN: not %clang -DFATAL @%t.rsp -gen-reproducer=off    2>&1 | FileCheck %s --check-prefix=NOT
 // RUN: not %clang -DFATAL @%t.rsp -fno-crash-diagnostics 2>&1 | FileCheck %s --check-prefix=NOT
 // RUN: not %clang -DFATAL @%t.rsp                        2>&1 | FileCheck %s
 // RUN: not %clang -DFATAL @%t.rsp -gen-reproducer=crash  2>&1 | FileCheck %s

clang/test/Driver/output-file-cleanup.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -f "%t.d" "%t1.s" "%t2.s" "%t3.s" "%t4.s" "%t5.s"
 //
 // RUN: touch %t.s
 // RUN: not %clang -S -DCRASH -o %t.s -MMD -MF %t.d %s
 // RUN: test ! -f %t.s
 // RUN: test ! -f %t.d
 
 // RUN: touch %t.s

clang/test/Driver/rewrite-map-in-diagnostics.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
+// RUN: export LLVM_DISABLE_SYMBOLIZATION=1
 // RUN: rm -rf "%t"
 // RUN: mkdir -p "%t"
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTION=1 \
 // RUN:     not %clang -fsyntax-only -frewrite-map-file=%p/Inputs/rewrite.map %s 2>&1 \
 // RUN:   | FileCheck %s
 
 #pragma clang __debug parser_crash
 
 // CHECK: note: diagnostic msg: {{.*}}rewrite.map
 
 // REQUIRES: crash-recovery