This is an archive of the discontinued LLVM Phabricator instance.

[Sanitizer] Allow setting the report path to create directory
ClosedPublic

Authored by tejohnson on Sep 14 2021, 4:39 PM.

Details

Summary

When setting the report path, recursively create the directory as
needed. This brings the profile path support for memprof on par with
normal PGO. The code was largely cloned from __llvm_profile_recursive_mkdir
in compiler-rt/lib/profile/InstrProfilingUtil.c.

Diff Detail

Event Timeline

tejohnson created this revision.Sep 14 2021, 4:39 PM
tejohnson requested review of this revision.Sep 14 2021, 4:39 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 14 2021, 4:39 PM
Herald added a subscriber: Restricted Project. · View Herald Transcript
vitalybuka added inline comments.Sep 14 2021, 6:00 PM
compiler-rt/lib/sanitizer_common/sanitizer_file.cpp
87

to avoid #ifs here can you please create CreateDir() and provide two implementations:
in sanitizer_win and sanitizer_posix

93

please clang-format as it asks

Move mkdir calls to OS specific helper.

Undo formatting change in unrelated existing code

tejohnson marked 2 inline comments as done.Sep 14 2021, 10:33 PM
vitalybuka added inline comments.Sep 15 2021, 5:23 PM
compiler-rt/lib/sanitizer_common/sanitizer_common.h
263

can we avoid #if?
we don't have one e.g. for GetProcessName

compiler-rt/lib/sanitizer_common/sanitizer_file.cpp
78

Maybe we can add ReportFile::SetReportPath unittest here:
sanitizer_common/tests/sanitizer_common_test.cpp

The function is little bit unconventional and does not create path for the last component. Maybe rename into something RecursiveCreateParentDirs(char* path)

82

We don't need two vars outside the loop.
Also if we start with 1 we can overflow on empty string.

84

if (!IsPathSeparator) ?

compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
154

int -> bool, to avoid exposing platform specific error errors.

tejohnson marked 4 inline comments as done.Sep 16 2021, 4:05 PM
tejohnson added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_common.h
263

GetProcessName has a single def in sanitizer_common.cpp. A closer analogy would be Abort(), which has implementations in all of these 4 places:
sanitizer_posix_libcdep.cpp
sanitizer_common_nolibc.cpp
sanitizer_win.cpp
sanitizer_fuchsia.cpp

Since CreateDir calls mkdir/_mkdir it also needs to be defined in similar places, and I have defs in the first 3 of the above. I believe for things to link properly on Fuchsia I would need to provide one there too. However, sanitizer_file.cpp which uses this new function has its code entirely guarded by #if !SANITIZER_FUCHSIA, so I wasn't sure it made sense. WDYT?

compiler-rt/lib/sanitizer_common/sanitizer_file.cpp
78

I ended up adding a new unittest to sanitizer_common/tests/sanitizer_libc_test.cpp instead, since it is already testing some sanitizer_file interfaces, and it probably only makes sense to test this under libc where we can mkdir anyway.

82

We start at 1 because if the path starts with "/" it doesn't make sense to try to mkdir an empty string. I added a check for the path being empty above this loop.

84

Woops, yes, I introduced this during some cleanup before sending the patch.

tejohnson updated this revision to Diff 373098.Sep 16 2021, 4:06 PM
tejohnson marked 3 inline comments as done.

Address comments

vitalybuka accepted this revision.Sep 20 2021, 8:58 AM
vitalybuka added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_common.h
263

I guess you can just move CreateDir next to IsPathSeparator into sanitizer_file.h without any defines. That header is better place for the definition anyway.

This revision is now accepted and ready to land.Sep 20 2021, 8:58 AM
vitalybuka added inline comments.Sep 20 2021, 8:59 AM
compiler-rt/lib/sanitizer_common/sanitizer_common.h
263

"without any defines" -> "without any ifdefs"

tejohnson marked an inline comment as done.Sep 21 2021, 1:51 PM
tejohnson updated this revision to Diff 374025.Sep 21 2021, 1:52 PM

Address comments

vitalybuka accepted this revision.Sep 21 2021, 2:50 PM
This revision was landed with ongoing or failed builds.Sep 21 2021, 4:43 PM
This revision was automatically updated to reflect the committed changes.
morehouse added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_win.cpp
568

This line caused a build breakage on the sanitizer bot: https://lab.llvm.org/buildbot/#/builders/127/builds/17222

C:\...\sanitizer_win.cpp(568): error C3861: '_mkdir': identifier not found

Could you please take a look?

tejohnson added inline comments.Sep 22 2021, 8:07 AM
compiler-rt/lib/sanitizer_common/sanitizer_win.cpp
568

Sorry, missed that one. Hopefully 1864976c967de36146eb5a5b86e6312d466e1031 that I just committed should take care of it.

denik added a subscriber: denik.Feb 9 2022, 6:39 PM

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

If the latter, we can probably avoid running the unit test in that environment - there are other tests that create temp files so there must be an existing mechanism to skip them or do something different.

denik added a comment.Feb 9 2022, 10:13 PM

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

Correct, I meant Chrome OS tests. Chrome OS uses Gentoo sandboxing for its unit tests.

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

log_path is set in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/profiles/base/profile.bashrc#512
Note that the directory was created in the same function. And it's called before entering a sandbox.

Can we skip CreateDir in RecursiveCreateParentDirs if the path exists? So it will be no-op in our case?

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

Correct, I meant Chrome OS tests. Chrome OS uses Gentoo sandboxing for its unit tests.

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

log_path is set in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/profiles/base/profile.bashrc#512
Note that the directory was created in the same function. And it's called before entering a sandbox.

Can we skip CreateDir in RecursiveCreateParentDirs if the path exists? So it will be no-op in our case?

Oh I see, we normally assume it will error (but we ignore the error) if the directory already exists. But in the sandboxed case it doesn't like the creation attempt, even though the dir actually exists.

Yeah, we can presumably use "stat" to check before trying to create. Actually, there is already a FileExists() in sanitizer_file.h that has implementations in the various OS specific sanitizer files. Most (except windows) call stat (but look for a regular file). I can just add something like a DirExists() interface that has the equivalent dir check using the same interfaces in each OS implementation. Will work on that tomorrow.

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

Correct, I meant Chrome OS tests. Chrome OS uses Gentoo sandboxing for its unit tests.

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

log_path is set in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/profiles/base/profile.bashrc#512
Note that the directory was created in the same function. And it's called before entering a sandbox.

Can we skip CreateDir in RecursiveCreateParentDirs if the path exists? So it will be no-op in our case?

Oh I see, we normally assume it will error (but we ignore the error) if the directory already exists. But in the sandboxed case it doesn't like the creation attempt, even though the dir actually exists.

Yeah, we can presumably use "stat" to check before trying to create. Actually, there is already a FileExists() in sanitizer_file.h that has implementations in the various OS specific sanitizer files. Most (except windows) call stat (but look for a regular file). I can just add something like a DirExists() interface that has the equivalent dir check using the same interfaces in each OS implementation. Will work on that tomorrow.

Are you able to patch in D119495 and see if that fixes the issue?

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

Correct, I meant Chrome OS tests. Chrome OS uses Gentoo sandboxing for its unit tests.

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

log_path is set in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/profiles/base/profile.bashrc#512
Note that the directory was created in the same function. And it's called before entering a sandbox.

Can we skip CreateDir in RecursiveCreateParentDirs if the path exists? So it will be no-op in our case?

Oh I see, we normally assume it will error (but we ignore the error) if the directory already exists. But in the sandboxed case it doesn't like the creation attempt, even though the dir actually exists.

Yeah, we can presumably use "stat" to check before trying to create. Actually, there is already a FileExists() in sanitizer_file.h that has implementations in the various OS specific sanitizer files. Most (except windows) call stat (but look for a regular file). I can just add something like a DirExists() interface that has the equivalent dir check using the same interfaces in each OS implementation. Will work on that tomorrow.

Are you able to patch in D119495 and see if that fixes the issue?

Yes. D119495 fixed the problem. Thanks for the quick solution!

Just curious, did __llvm_profile_recursive_mkdir also create directories in a similar way (ignoring EEXIST status)? I'm asking because the same test was creating a PGO profile in a sandbox and it did not fail.
While debugging the sandbox (before your fix), I noticed that the test crashed when the sandbox was checking lstat of the path from __sanitizer::CreateDir(). Can it be that the sanitizer intercepted the call but since it was __asan::AsanInitInternal() the handler was not set up yet? This would explain why it worked for PGO.

This change breaks tests with sanitizers running in a sandbox.
If /usr/local/tmp/asan is in the allow-list for writing, the sandbox will kill the test when it tries to create /usr.

This failure is caught on Chrome OS, https://issuetracker.google.com/209296420.

It sounds like the issue is showing up with some Chrome OS tests built with -fsanitize=address, as opposed to in the unit test added here - is that correct?

Correct, I meant Chrome OS tests. Chrome OS uses Gentoo sandboxing for its unit tests.

If the former, by default the report_file is stderr - do you know where/how the report file path is being set to something else? Note the change only creates a directory for the given report_file path, the sanitizer change itself doesn't change the path. Before the report file creation would fail if the directory didn't exist.

log_path is set in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/profiles/base/profile.bashrc#512
Note that the directory was created in the same function. And it's called before entering a sandbox.

Can we skip CreateDir in RecursiveCreateParentDirs if the path exists? So it will be no-op in our case?

Oh I see, we normally assume it will error (but we ignore the error) if the directory already exists. But in the sandboxed case it doesn't like the creation attempt, even though the dir actually exists.

Yeah, we can presumably use "stat" to check before trying to create. Actually, there is already a FileExists() in sanitizer_file.h that has implementations in the various OS specific sanitizer files. Most (except windows) call stat (but look for a regular file). I can just add something like a DirExists() interface that has the equivalent dir check using the same interfaces in each OS implementation. Will work on that tomorrow.

Are you able to patch in D119495 and see if that fixes the issue?

Yes. D119495 fixed the problem. Thanks for the quick solution!

Great, will try to get this submitted today, once I address the comments.

Just curious, did __llvm_profile_recursive_mkdir also create directories in a similar way (ignoring EEXIST status)? I'm asking because the same test was creating a PGO profile in a sandbox and it did not fail.

Interesting - I actually modeled the implementation here exactly on that function. It has the same unconditional mkdir with the same comment about ignoring failures due to existing directories.

While debugging the sandbox (before your fix), I noticed that the test crashed when the sandbox was checking lstat of the path from __sanitizer::CreateDir(). Can it be that the sanitizer intercepted the call but since it was __asan::AsanInitInternal() the handler was not set up yet? This would explain why it worked for PGO.

Not totally following - which handler?

denik added a comment.Feb 11 2022, 4:05 PM

Are you able to patch in D119495 and see if that fixes the issue?

Yes. D119495 fixed the problem. Thanks for the quick solution!

Great, will try to get this submitted today, once I address the comments.

Just curious, did __llvm_profile_recursive_mkdir also create directories in a similar way (ignoring EEXIST status)? I'm asking because the same test was creating a PGO profile in a sandbox and it did not fail.

Interesting - I actually modeled the implementation here exactly on that function. It has the same unconditional mkdir with the same comment about ignoring failures due to existing directories.

While debugging the sandbox (before your fix), I noticed that the test crashed when the sandbox was checking lstat of the path from __sanitizer::CreateDir(). Can it be that the sanitizer intercepted the call but since it was __asan::AsanInitInternal() the handler was not set up yet? This would explain why it worked for PGO.

Not totally following - which handler?

I meant the sanitizer interceptor: https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/sanitizer_common_interceptors.inc#L6510
With llvm debug info enabled:

(gdb) bt
#0  __interceptor_lstat ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6878
#1  0x00007ffff7f9e297 in sb_mkdirat_pre_check (func=0x7ffff7f963e0 "mkdir", pathname=0x31bfdc <__sanitizer::report_file+12> "/build",
    dirfd=-100) at /var/tmp/portage/sys-apps/sandbox-2.29/work/sandbox-2.29/libsandbox/pre_check_mkdirat.c:40
#2  0x00007ffff7fa167b in mkdir_DEFAULT (pathname=0x31bfdc <__sanitizer::report_file+12> "/build", mode=493)
    at /var/tmp/portage/sys-apps/sandbox-2.29/work/sandbox-2.29/libsandbox/wrapper-funcs/__wrapper_simple.c:51
#3  0x00000000002faaab in __sanitizer::CreateDir(char const*) ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:154
#4  0x00000000002eaab6 in RecursiveCreateParentDirs ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/sanitizer_common/sanitizer_file.cpp:87
#5  SetReportPath ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/sanitizer_common/sanitizer_file.cpp:112
#6  0x00000000002e03b5 in AsanInitInternal ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/asan/asan_rtl.cpp:426
(gdb) frame 0
#0  __interceptor_lstat ()
    at /var/tmp/portage/sys-devel/llvm-14.0_pre437112_p20211208-r7/work/llvm-14.0_pre437112_p20211208/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6878
6878      COMMON_INTERCEPTOR_ENTER(ctx, lstat, path, buf);
(gdb) s

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

It crashes on the COMMON_INTERCEPTOR_ENTER() call.