Page MenuHomePhabricator

Support: Add vfs::OutputBackend and OutputFile to virtualize compiler outputs
Needs ReviewPublic

Authored by steven_wu on Sep 8 2022, 9:36 AM.

Details

Summary

Add OutputBackend and OutputFile to the llvm::vfs namespace for
virtualizing compiler outputs. This is intended for use in Clang,

The headers are:

  • llvm/Support/VirtualOutputConfig.h
  • llvm/Support/VirtualOutputError.h
  • llvm/Support/VirtualOutputFile.h
  • llvm/Support/VirtualOutputBackend.h

OutputFile is moveable and owns an OutputFileImpl, which is provided by
the derived OutputBackend.

  • OutputFileImpl::keep() and OutputFileImpl::discard() should keep or discard the output. OutputFile guarantees that exactly one of these will be called before destruction.
  • OutputFile::keep() and OutputFile::discard() wrap OutputFileImpl and catch usage errors such as double-close.
  • OutputFile::discardOnDestroy() installs an error handler for the destructor to use if the file is still open. The handler will be called if discard() fails.
  • OutputFile::~OutputFile() calls report_fatal_error() if none of keep(), discard(), or discardOnDestroy() has been called. It still calls OutputFileImpl::discard().
  • getOS() returns the wrapped raw_pwrite_stream. For convenience, OutputFile has an implicit conversion to raw_ostream and raw_ostream &operator<<(OutputFile&, T&&).

OutputBackend can be stored in IntrusiveRefCntPtr.

  • Most operations are thread-safe.
  • clone() returns a backend that targets the same destination. All operations are thread-safe when done on different clones.
  • createFile() takes a path and an OutputConfig (see below) and returns an OutputFile. Backends implement createFileImpl().

OutputConfig has flags to configure the output. Backends may ignore or
override flags that aren't relevant or implementable.

  • The initial flags are:
    • AtomicWrite: whether the output should appear atomically (e.g., by using a temporary file and renaming it).
    • CrashCleanup: whether the output should be cleaned up if there's a crash (e.g., with RemoveFileOnSignal).
    • ImplyCreateDirectories: whether to implicitly create missing directories in the path to the file.
    • Text: matches sys::fs::OF_Text.
    • CRLF: matches sys::fs::OF_CRLF.
  • Each "Flag" has setFlag(bool) and bool getFlag() and shortcuts setFlag() and setNoFlag(). The setters are constexpr and return OutputConfig& to make it easy to declare a default value for a filed in a class or struct.
  • Setters and getters for Binary and TextWithCRLF are derived from Text and CRLF. For convenience, sys::fs::OpenFlags can be passed directly to setOpenFlags().

This patch intentionally lacks a number of important features that have
been left for follow-ups:

  • Set a (virtual) current working directory.
  • Create a directory.
  • Create a file or directory with a unique name (avoiding collisions with existing filenames).

Patch by dexonsmith

Diff Detail

Unit TestsFailed

TimeTest
3,840 msx64 debian > HWAddressSanitizer-x86_64.TestCases::heap-buffer-overflow.c
Script: -- : 'RUN: at line 1'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -m64 -gline-tables-only -fsanitize=hwaddress -fuse-ld=lld -fsanitize-hwaddress-experimental-aliasing -mllvm -hwasan-generate-tags-with-calls=1 -mllvm -hwasan-globals -mllvm -hwasan-use-short-granules -mllvm -hwasan-instrument-landing-pads=0 -mllvm -hwasan-instrument-personality-functions /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/hwasan/TestCases/heap-buffer-overflow.c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/heap-buffer-overflow.c.tmp

Event Timeline

steven_wu created this revision.Sep 8 2022, 9:36 AM
Herald added a project: Restricted Project. · View Herald TranscriptSep 8 2022, 9:36 AM
steven_wu requested review of this revision.Sep 8 2022, 9:36 AM
Herald added a project: Restricted Project. · View Herald TranscriptSep 8 2022, 9:36 AM

This is an updated version of https://reviews.llvm.org/D95501 addressing the review feedback. Also see parent review for the change related to raw_ostream_proxy.

tschuett added inline comments.Sep 8 2022, 11:24 AM
llvm/include/llvm/Support/VirtualOutputBackend.h
18

LLVM moved to C++17. You may use nested namespaces.

steven_wu updated this revision to Diff 458845.Sep 8 2022, 1:40 PM

Use nested namespace

steven_wu marked an inline comment as done.Sep 8 2022, 1:41 PM
n-omer added a subscriber: n-omer.Sep 11 2022, 3:27 AM

ping. Any feedback from this iteration of the OutputBackend?

grekd added a subscriber: grekd.Oct 22 2022, 12:57 PM