Page MenuHomePhabricator

Clean up large copies of binaries copied into temp directories in tests
ClosedPublic

Authored by tejohnson on Sep 22 2021, 11:45 AM.

Details

Summary

In looking at the disk space used by a ninja check-all, I found that a
few of the largest files were copies of clang and lld made into temp
directories by a couple of tests. These tests were added in D53021 and
D74811. Clean up these copies after usage.

Diff Detail

Event Timeline

tejohnson requested review of this revision.Sep 22 2021, 11:45 AM
tejohnson created this revision.
Herald added a project: Restricted Project. · View Herald TranscriptSep 22 2021, 11:45 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
This revision is now accepted and ready to land.Sep 22 2021, 1:01 PM

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

The failures I am seeing there are in a few other tests, not in the ones I modified. Can you confirm?

Oh, the win failure (https://buildkite.com/llvm-project/premerge-checks/builds/57480#f7da3275-775a-4cf4-9624-d3539dd1f709) might actually be real. Windows doesn't allow deleting files that are still in use, and due to AV or similar, the file might still be in use for a bit even after the program completes.

The failures I am seeing there are in a few other tests, not in the ones I modified. Can you confirm?

@thakis ping - I don't see any failures related to the tests I changed - can you confirm or let me know if I missed something?

Is it possible to use soft links instead of copies? It would still be good to clean up afterwards, but if the host won't allow that cleanup in some cases at least we aren't leaving a big file around.

Is it possible to use soft links instead of copies? It would still be good to clean up afterwards, but if the host won't allow that cleanup in some cases at least we aren't leaving a big file around.

Possible, but I didn't want to affect the operation of the tests. I don't believe there is any issue with being able to clean up, but I forgot about this patch while waiting for @thakis to confirm that the test failures he saw weren't actually due to this change. I can go ahead and commit this later today if he doesn't get back, and just watch the bots.

This revision was landed with ongoing or failed builds.Sep 28 2021, 5:04 PM
This revision was automatically updated to reflect the committed changes.

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

I can try submitting a change to make it use rm -f. Is this failure on a bot? I didn't get any bot failure notifications.

Softlinks don't work on Windows, that's why these tests use copies.

clang_f_opts.c sadly started flaking almost immediately after this went in: http://45.33.8.238/win/46115/step_7.txt

So I fear the Windows file system can't handle this patch as-is.

Maybe we can use rm -f and maybe that way the error isn't fatal even when deleting the file doesn't work, not sure.

I can try submitting a change to make it use rm -f. Is this failure on a bot? I didn't get any bot failure notifications.

Made this change in 2f1b99ca67da18d858a4b070716790a8f53891d6. I see lots of "rm -f" in the tests, including elsewhere in clang_f_opts.c, so hopefully this should work.

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

Added that in b55a964197bdc651533377bbd0b46fa58edf9196. Let me know if that fixes the issue.

It's my personal bot. It doesn't send email. IMHO the problem of how to make bots send emails hasn't been solved (they either send too much or too little), so I'm not pretending that it has been ;)

Sadly the test is still failing after the -f change: http://45.33.8.238/win/46139/step_7.txt

Maybe tacking on an || true helps?

Added that in b55a964197bdc651533377bbd0b46fa58edf9196. Let me know if that fixes the issue.

I saw this failure every ~4 hours yesterday, but I haven't seen it once yet today. Fingers crossed, but it looks like that might have done the trick :)