Clean up large copies of binaries copied into temp directories in tests

[tejohnson]

Summary

In looking at the disk space used by a ninja check-all, I found that a
few of the largest files were copies of clang and lld made into temp
directories by a couple of tests. These tests were added in D53021 and
D74811. Clean up these copies after usage.

[thakis]

lg

[thakis]

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

[tejohnson]

In D110276#3016295, @thakis wrote:

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

The failures I am seeing there are in a few other tests, not in the ones I modified. Can you confirm?

[tejohnson]

In D110276#3016312, @tejohnson wrote:

In D110276#3016295, @thakis wrote:

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

The failures I am seeing there are in a few other tests, not in the ones I modified. Can you confirm?

@thakis ping - I don't see any failures related to the tests I changed - can you confirm or let me know if I missed something?

[scott.linder]

Is it possible to use soft links instead of copies? It would still be good to clean up afterwards, but if the host won't allow that cleanup in some cases at least we aren't leaving a big file around.

[tejohnson]

In D110276#3027955, @scott.linder wrote:

Is it possible to use soft links instead of copies? It would still be good to clean up afterwards, but if the host won't allow that cleanup in some cases at least we aren't leaving a big file around.

Possible, but I didn't want to affect the operation of the tests. I don't believe there is any issue with being able to clean up, but I forgot about this patch while waiting for @thakis to confirm that the test failures he saw weren't actually due to this change. I can go ahead and commit this later today if he doesn't get back, and just watch the bots.

[thakis]

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

[tejohnson]

In D110276#3029952, @thakis wrote:

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

I can try submitting a change to make it use rm -f. Is this failure on a bot? I didn't get any bot failure notifications.

[tejohnson]

In D110276#3030390, @tejohnson wrote:

In D110276#3029952, @thakis wrote:

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

I can try submitting a change to make it use rm -f. Is this failure on a bot? I didn't get any bot failure notifications.

Made this change in 2f1b99ca67da18d858a4b070716790a8f53891d6. I see lots of "rm -f" in the tests, including elsewhere in clang_f_opts.c, so hopefully this should work.

[thakis]

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

[tejohnson]

In D110276#3031133, @thakis wrote:

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

Added that in b55a964197bdc651533377bbd0b46fa58edf9196. Let me know if that fixes the issue.

[thakis]

In D110276#3032448, @tejohnson wrote:

In D110276#3031133, @thakis wrote:

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

Added that in b55a964197bdc651533377bbd0b46fa58edf9196. Let me know if that fixes the issue.

I saw this failure every ~4 hours yesterday, but I haven't seen it once yet today. Fingers crossed, but it looks like that might have done the trick :)

clang/test/Driver/clang_f_opts.c

 // is cached at the new clang's inode, then clang is changed, tests run again, old signature
 // is still cached with old clang's inode, so it won't execute this time. Removing the dir
 // first guarantees a new inode without old cache entries.)
 // RUN: rm -rf "%t.r/with spaces"
 // RUN: mkdir -p "%t.r/with spaces"
 // RUN: cp %clang "%t.r/with spaces/clang"
 // RUN: "%t.r/with spaces/clang" -### -S -target x86_64-unknown-linux -frecord-gcc-switches %s 2>&1 | FileCheck -check-prefix=CHECK-RECORD-GCC-SWITCHES-ESCAPED %s
 // CHECK-RECORD-GCC-SWITCHES-ESCAPED: "-record-command-line" "{{.+}}with\\ spaces{{.+}}"
+// Clean up copy of large binary copied into temp directory to avoid bloat.
+// RUN: rm "%t.r/with spaces/clang"
 
 // RUN: %clang -### -S -ftrivial-auto-var-init=uninitialized %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-UNINIT %s
 // RUN: %clang -### -S -ftrivial-auto-var-init=pattern %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-PATTERN %s
 // RUN: %clang -### -S -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-ZERO-GOOD %s
 // RUN: %clang -### -S -ftrivial-auto-var-init=zero %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-ZERO-BAD %s
 // CHECK-TRIVIAL-UNINIT-NOT: hasn't been enabled
 // CHECK-TRIVIAL-PATTERN-NOT: hasn't been enabled
 // CHECK-TRIVIAL-ZERO-GOOD-NOT: hasn't been enabled

lld/test/COFF/pdb-relative-source-lines.test

 RUN: llvm-pdbutil pdb2yaml -modules -module-files -module-syms -subsections=lines,fc %t/out.pdb | FileCheck %s
 RUN: ./lld-link -debug "-pdbsourcepath:/usr/src" -entry:main -nodefaultlib -out:out.exe -pdb:out.pdb pdb_lines_1_relative.obj pdb_lines_2_relative.obj
 RUN: llvm-pdbutil pdb2yaml -modules -module-files -module-syms -subsections=lines,fc %t/out.pdb | FileCheck --check-prefix=POSIX %s
 
 Also check without -pdbsourcepath
 RUN: ./lld-link -debug -entry:main -nodefaultlib -out:out.exe -pdb:out.pdb pdb_lines_1_relative.obj pdb_lines_2_relative.obj
 RUN: llvm-pdbutil pdb2yaml -modules -module-files -module-syms -subsections=lines,fc %t/out.pdb | FileCheck --check-prefix=ABSOLUTE %s
 
+Clean up copy of large binary copied into temp directory to avoid bloat.
+RUN: rm ./lld-link
+
 CHECK-LABEL:  - Module:          'c:\src\pdb_lines_1_relative.obj'
 CHECK-NEXT:     ObjFile:         'c:\src\pdb_lines_1_relative.obj'
 CHECK:          SourceFiles:
 CHECK-NEXT:       - 'c:\src\pdb_lines_1.c'
 CHECK-NEXT:       - 'c:\src\foo.h'
 CHECK:          Subsections:
 CHECK:                - FileName:        'c:\src\pdb_lines_1.c'
 CHECK:                - FileName:        'c:\src\foo.h'