[This is supporting an RFC (https://lists.llvm.org/pipermail/cfe-dev/2021-January/067576.html / https://lists.llvm.org/pipermail/llvm-dev/2021-January/148124.html); a follow up patch has initial adoption in clang::CompilerInstance; marked WIP for now.]
WIP: Support: Add vfs::OutputBackend to virtualize compiler outputs
Add OutputBackend to the llvm::vfs namespace to virtualize compiler
outputs. The primary interface includes the following API:
- getDirectory() returns an OutputDirectory, which is a proxy backend that uses the requested directory as a working directory.
- createFile() creates a file and returns OutputFile for writing to it.
- createDirectory() creates a directory and then calls getDirectory(). The default implementation assumes creating a directory is a no-op, to support backends that don't have a concept of a directories without files in them.
- createUniqueFile() and createUniqueDirectory(), versions of the above that avoid creating entitity names that already exist. (StableUniqueEntityAdaptor<> can be used by backends that don't have a way to check what exists.)
This patch contains a number of backends, including:
- NullOutputBackend, for silently ignoring all outputs.
- OnDiskOutputBackend, for writing to disk.
- InMemoryOutputBackend, for writing to an InMemoryFileSystem.
- MirroringOutputBackend, for writing to multiple backends.
- FilteringOutputBackend, for filtering which outputs get written to the underlying backend.
There are also a few helpers for creating other backends:
- ProxyOutputBackend forwards all calls to an underlying backend.
- StableUniqueEntityAdaptor<> adds stable implementations of createUniqueFileImpl() and createUniqueDirectoryImpl().
The primary interface for OutputFile includes the following API:
- takeOS() returns a std::unique_ptr<raw_pwrite_stream> for writing (getOS() gives a raw pointer to the stream if it hasn't been taken).
- close() commits the file (to disk, in-memory, etc.).
- getPath() gets the path to the output.
- ~Output erases the output if it hasn't been opened.
OutputFile has support for buffering content. One use case is for
concrete subclasses that don't support streams. Another is to support
passing the same content to multiple underlying OutputFile instances
(such as for MirroringOutputBackend). OutputFile::close() calls
storeContentBuffer() with a ContentBuffer if this support was used;
otherwise, it calls storeStreamedContent().
An OutputConfig is passed when creating a file or directory to
communicate semantics about the output. There are a few flags:
- Text, for outputs that should be opened in text mode.
- Volatile, for outputs that other processes may modify after write. (OnDiskOutputBackend has support for writing its bytes to an mmap'ed region and passing a file-backed buffer on to other backends. This optimization is turned off for Volatile outputs.)
- NoCrashCleanup, for outputs that should not be removed on crash.
- NoAtomicWrite, for outputs that should be written directly, instead of moved atomically into place.
- NoImplyCreateDirectories, for outputs that require the parent path to already exist.
- NoOverwrite, for outputs that are not allowed to clobber any existing content.
Most of these flags are ignorable by backends that aren't associated
with a filesystem.
- Fix all the header docs. They've mostly bitrotted.
- Document and enforce threading guarantees. Tentatively, all OutputBackend APIs safe to call concurrently. An OutputFile cannot be used concurrently, but two files from the same backend can be.
- Test OutputDirectory.
- Test createDirectory().
- Test createDirectories().
- Test createUnique*().