[clang] Emit crash reproduction as a single tar file

[abrachet]

Summary

When clang crashes it emits many temp files, this changes clang's behavior to aggregate all temp files into a single tar file.

[hubert.reinterpretcast]

For users on Windows, would this cause extra trouble if they wanted to see what was included?

[abrachet]

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

[aaron.ballman]

In D122335#3403283, @abrachet wrote:

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

It's available on Windows 10 and later by default, but otherwise .zip is the native archive format on Windows.

[erichkeane]

In D122335#3403305, @aaron.ballman wrote:

In D122335#3403283, @abrachet wrote:

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

It's available on Windows 10 and later by default, but otherwise .zip is the native archive format on Windows.

We DO support older versions of Windows (Windows Server 2003 is still quite popular!), so presumably this requirement wouldn't be particularly acceptable.

[aaron.ballman]

In D122335#3403357, @erichkeane wrote:

In D122335#3403305, @aaron.ballman wrote:

In D122335#3403283, @abrachet wrote:

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

It's available on Windows 10 and later by default, but otherwise .zip is the native archive format on Windows.

We DO support older versions of Windows (Windows Server 2003 is still quite popular!), so presumably this requirement wouldn't be particularly acceptable.

My feeling is: we have a TAR file writer, we don't have a ZIP file writer. While it'd be nice for us to produce zip files as those are more widely supported on all the platforms Clang runs on, it's a pretty heavy lift for this feature, especially given that newer versions of Windows come with a sufficiently useful tar program to handle them. So I'm fine with us producing TAR files -- if we decide to add support for ZIP in the future, we can allow the user to pick via a command line flag (or we could pick based on the host platform) and we can set the default behavior of Clang with a CMake flag.

[erichkeane]

In D122335#3403453, @aaron.ballman wrote:

In D122335#3403357, @erichkeane wrote:

In D122335#3403305, @aaron.ballman wrote:

In D122335#3403283, @abrachet wrote:

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

It's available on Windows 10 and later by default, but otherwise .zip is the native archive format on Windows.

We DO support older versions of Windows (Windows Server 2003 is still quite popular!), so presumably this requirement wouldn't be particularly acceptable.

My feeling is: we have a TAR file writer, we don't have a ZIP file writer. While it'd be nice for us to produce zip files as those are more widely supported on all the platforms Clang runs on, it's a pretty heavy lift for this feature, especially given that newer versions of Windows come with a sufficiently useful tar program to handle them. So I'm fine with us producing TAR files -- if we decide to add support for ZIP in the future, we can allow the user to pick via a command line flag (or we could pick based on the host platform) and we can set the default behavior of Clang with a CMake flag.

Ah! I took the tests using 'tar' directly to mean we were offloading this work to 'tar', not that we had our own implementation of 'tar'. In that case, I am ok with that as well; the burden is only on compiler-engineers to have a copy of 'tar' on their windows machine which is a requirement I'm OK with making.

[aaron.ballman]

In D122335#3403459, @erichkeane wrote:

In D122335#3403453, @aaron.ballman wrote:

In D122335#3403357, @erichkeane wrote:

In D122335#3403305, @aaron.ballman wrote:

In D122335#3403283, @abrachet wrote:

In D122335#3403281, @hubert.reinterpretcast wrote:

For users on Windows, would this cause extra trouble if they wanted to see what was included?

Is tar(1) not available on Windows?

It's available on Windows 10 and later by default, but otherwise .zip is the native archive format on Windows.

We DO support older versions of Windows (Windows Server 2003 is still quite popular!), so presumably this requirement wouldn't be particularly acceptable.

My feeling is: we have a TAR file writer, we don't have a ZIP file writer. While it'd be nice for us to produce zip files as those are more widely supported on all the platforms Clang runs on, it's a pretty heavy lift for this feature, especially given that newer versions of Windows come with a sufficiently useful tar program to handle them. So I'm fine with us producing TAR files -- if we decide to add support for ZIP in the future, we can allow the user to pick via a command line flag (or we could pick based on the host platform) and we can set the default behavior of Clang with a CMake flag.

Ah! I took the tests using 'tar' directly to mean we were offloading this work to 'tar', not that we had our own implementation of 'tar'. In that case, I am ok with that as well; the burden is only on compiler-engineers to have a copy of 'tar' on their windows machine which is a requirement I'm OK with making.

Yup, I'm okay with that compiler engineer requirement; I think our existing requirements on Windows already pull in a useful tar program even though it doesn't seem like we have any tests relying on it being installed yet.

[abrachet]

Fix test failures outside of Driver/

[abrachet]

In D122335#3403474, @aaron.ballman wrote:

Yup, I'm okay with that compiler engineer requirement; I think our existing requirements on Windows already pull in a useful tar program even though it doesn't seem like we have any tests relying on it being installed yet.

There are actually a couple tests in lld that use tar https://github.com/llvm/llvm-project/blob/main/lld/test/ELF/reproduce-backslash.s#L8

Looks like there are some test failures on Windows my guess is their tar doesn't support the --wildcards flag. I'd rather not make these UNSUPORTED: windows, but am unsure how to move forward otherwise.

[abrachet]

Rebase

[jrtc27]

As a developer who often deals with crashes locally this is more annoying; currently I can just point tools at the shell script and C file in /tmp and let them go to work reducing, but now I have to also extract the files

[arichardson]

In D122335#3404358, @jrtc27 wrote:

As a developer who often deals with crashes locally this is more annoying; currently I can just point tools at the shell script and C file in /tmp and let them go to work reducing, but now I have to also extract the files

+1.

I can see this being helpful for end-users who now just have to provide developers with a single file, but it does make things more awkward for me as a developer. How about an environment variable and/or a CMake option to skip archiving that can be used locally.

[aaron.ballman]

In D122335#3404754, @arichardson wrote:

In D122335#3404358, @jrtc27 wrote:

As a developer who often deals with crashes locally this is more annoying; currently I can just point tools at the shell script and C file in /tmp and let them go to work reducing, but now I have to also extract the files

+1.

I can see this being helpful for end-users who now just have to provide developers with a single file, but it does make things more awkward for me as a developer. How about an environment variable and/or a CMake option to skip archiving that can be used locally.

I think the idea of a cmake option makes a lot of sense. It may even be worth disabling the feature by default when configuring a debug build under the assumption those are primarily for compiler developers.

[abrachet]

• Add new build option CLANG_CRASH_SAVE_TEMPS which defaults to Off, or On if CMAKE_BUILD_TYPE is Debug. It retains the old behavior alongside the tar file.
• Mark tests which use tar(1) as UNSUPPORTED: windows, it seems either the Windows bots don't have tar or it doesn't support --wildcards

[arichardson]

clang/CMakeLists.txt

520

Could you add this to the config.h header instead?

clang/lib/Driver/Driver.cpp

1434

This could be simplified with #cmakedefine01 in the config header

clang/test/Driver/crash-report-clang-cl.cpp

1

I think it would be cleaner to add a new feature if tar is available in $PATH instead of ignoring all windows bots. Some might have tar installed.

8

Alternatively, if we had an env var/command line option to keep the cpp/sh, we could use that to avoid the dependency on tar in this test.

clang/test/Driver/crash-report-save-temps.c

14

Is this line intentional?

[abrachet]

• Define CLANG_CRASH_SAVE_TEMPS in config.h and not with add_definitions
• Add CLANG_CRASH_SAVE_TEMPS env var to force emitting individual preprocessed sources

[abrachet]

clang/lib/Driver/Driver.cpp

1434

Thanks I like that a lot better.

clang/test/Driver/crash-report-clang-cl.cpp

1

It's not clear to me what the Windows builders don't like. tar(1) is used in some lld tests. My guess is that the tar shipped on Windows doesn't support --wildcards which is seemingly necessary here because the tar file will have a random name, and it's member files are prefixed with that name. I don't have easy access to a Windows machine to test that theory though.

I've gone with your suggestion to have an env var. Do you think I should remove UNSUPPORTED: windows on all tests or just this one?

8

Sure.

clang/test/Driver/crash-report-save-temps.c

14

Nope, thanks.

[abrachet]

reabase

[abrachet]

Fixing failing tests in Modules/

[abrachet]

Ping :)

[abrachet]

Ping :)

clang/CMakeLists.txt

set(LIBCLANG_LIBRARY_VERSION
     "${CLANG_VERSION_MAJOR}" CACHE STRING
     "Major version number that will be appended to the libclang library")
 mark_as_advanced(CLANG_EXECUTABLE_VERSION LIBCLANG_LIBRARY_VERSION)
 
 option(CLANG_INCLUDE_TESTS
        "Generate build targets for the Clang unit tests."
        ${LLVM_INCLUDE_TESTS})
 
+include(CMakeDependentOption)
+
+cmake_dependent_option(CLANG_CRASH_SAVE_TEMPS
+                       "In addition to tar reproducer, save all temp files that go in the tar"
+                       ON "${CMAKE_BUILD_TYPE} MATCHES Debug" OFF)
+if(CLANG_CRASH_SAVE_TEMPS)
+  add_definitions(-DCLANG_CRASH_SAVE_TEMPS)
+endif()
+
 add_subdirectory(utils/TableGen)
 
 add_subdirectory(include)
 
 # All targets below may depend on all tablegen'd files.
 get_property(CLANG_TABLEGEN_TARGETS GLOBAL PROPERTY CLANG_TABLEGEN_TARGETS)
 add_custom_target(clang-tablegen-targets
   DEPENDS

clang/docs/ReleaseNotes.rst

 - ``-Winline-namespace-reopened-noninline`` now takes into account that the
   ``inline`` keyword must appear on the original but not necessarily all
   extension definitions of an inline namespace and therefore points its note
   at the original definition. This fixes `Issue 50794 (PR51452)
   <https://github.com/llvm/llvm-project/issues/50794>`_.
 - ``-Wunused-but-set-variable`` now also warns if the variable is only used
   by unary operators.
 
+- Crash diagnostics are now aggregated into a single tar file instead of
+  being split across many temp files. Moreover, input files are written into
+  the output tar file as well.
+
 Non-comprehensive list of changes in this release
 -------------------------------------------------
 - The builtin function __builtin_dump_struct would crash clang when the target 
   struct have bitfield. Now it fixed, and __builtin_dump_struct support dump
   the bitwidth of bitfields.
   This fixes `Issue 54462 <https://github.com/llvm/llvm-project/issues/54462>`_.
 
 New Compiler Flags

clang/include/clang/Driver/Compilation.h

public:
   /// failing commands and their associated result code.
   void ExecuteJobs(
       const JobList &Jobs,
       SmallVectorImpl<std::pair<int, const Command *>> &FailingCommands) const;
 
   /// initCompilationForDiagnostics - Remove stale state and suppress output
   /// so compilation can be reexecuted to generate additional diagnostic
   /// information (e.g., preprocessed source(s)).
-  void initCompilationForDiagnostics();
+  void initCompilationForDiagnostics(bool SaveTemps);
 
   /// Return true if we're compiling for diagnostics.
   bool isForDiagnostics() const { return ForDiagnostics; }
 
   /// Return whether an error during the parsing of the input args.
   bool containsError() const { return ContainsError; }
 
   /// Force driver to fail before toolchain is created. This is necessary when

clang/lib/Driver/Compilation.cpp

if (int Res = ExecuteCommand(Job, FailingCommand)) {
       FailingCommands.push_back(std::make_pair(Res, FailingCommand));
       // Bail as soon as one command fails in cl driver mode.
       if (TheDriver.IsCLMode())
         return;
     }
   }
 }
 
-void Compilation::initCompilationForDiagnostics() {
+void Compilation::initCompilationForDiagnostics(bool SaveTemps) {
   ForDiagnostics = true;
 
   // Free actions and jobs.
   Actions.clear();
   AllActions.clear();
   Jobs.clear();
 
   // Remove temporary files.
for (auto Arg : TCArgs)
     if (Arg.second != TranslatedArgs)
       delete Arg.second;
   TCArgs.clear();
 
   // Redirect stdout/stderr to /dev/null.
   Redirects = {None, {""}, {""}};
 
   // Temporary files added by diagnostics should be kept.
-  ForceKeepTempFiles = true;
+  ForceKeepTempFiles = SaveTemps;
 }
 
 StringRef Compilation::getSysRoot() const {
   return getDriver().SysRoot;
 }
 
 void Compilation::Redirect(ArrayRef<Optional<StringRef>> Redirects) {
   this->Redirects = Redirects;
 }

clang/lib/Driver/Driver.cpp

 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/MD5.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/StringSaver.h"
+#include "llvm/Support/TarWriter.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/Support/raw_ostream.h"
+#include <functional>
 #include <map>
 #include <memory>
 #include <utility>
 #if LLVM_ON_UNIX
 #include <unistd.h> // getpid
 #endif
 
 using namespace clang::driver;
if (!CrashFilePath.empty()) {
     if (EC)
       return false;
     return true;
   }
 
   return false;
 }
 
+static std::pair<std::unique_ptr<llvm::TarWriter>, std::string>
+openReproTarFile(Driver &D, Compilation &C) {
+  SmallString<64> TmpName;
+  if (Arg *A = C.getArgs().getLastArg(options::OPT_fcrash_diagnostics_dir))
+    TmpName = A->getValue();
+  else
+    llvm::sys::path::system_temp_directory(/*ErasedOnReboot*/ true, TmpName);
+
+  if (!D.getVFS().exists(TmpName))
+    llvm::sys::fs::create_directories(TmpName);
+
+  llvm::sys::path::append(TmpName, "%%%%%%.tar");
+  std::error_code EC = llvm::sys::fs::createUniqueFile(TmpName, TmpName);
+  if (EC) {
+    D.Diag(clang::diag::err_unable_to_make_temp) << EC.message();
+    return {};
+  }
+
+  auto TarWriterOrErr =
+      llvm::TarWriter::create(TmpName, llvm::sys::path::stem(TmpName));
+  if (!TarWriterOrErr) {
+    D.Diag(clang::diag::err_unable_to_make_temp) << TmpName;
+    return {};
+  }
+
+  return {std::move(*TarWriterOrErr), TmpName.c_str()};
+}
+
 // When clang crashes, produce diagnostic information including the fully
 // preprocessed source file(s).  Request that the developer attach the
 // diagnostic information to a bug report.
 void Driver::generateCompilationDiagnostics(
     Compilation &C, const Command &FailingCommand,
     StringRef AdditionalInformation, CompilationDiagnosticReport *Report) {
+#ifdef CLANG_CRASH_SAVE_TEMPS
+  constexpr bool SaveReproTemps = true;
+#else
+  constexpr bool SaveReproTemps = false;
+#endif
+
   if (C.getArgs().hasArg(options::OPT_fno_crash_diagnostics))
     return;
 
   // Don't try to generate diagnostics for link or dsymutil jobs.
   if (FailingCommand.getCreator().isLinkJob() ||
       FailingCommand.getCreator().isDsymutilJob())
     return;
 
   // Print the version of the compiler.
   PrintVersion(C, llvm::errs());
 
   // Suppress driver output and emit preprocessor output to temp file.
   CCGenDiagnostics = true;
 
   // Save the original job command(s).
   Command Cmd = FailingCommand;
 
   // Keep track of whether we produce any errors while trying to produce
   // preprocessed sources.
   DiagnosticErrorTrap Trap(Diags);
 
   // Suppress tool output.
-  C.initCompilationForDiagnostics();
+  C.initCompilationForDiagnostics(SaveReproTemps);
 
   // Construct the list of inputs.
   InputList Inputs;
   BuildInputs(C.getDefaultToolChain(), C.getArgs(), Inputs);
 
   for (InputList::iterator it = Inputs.begin(), ie = Inputs.end(); it != ie;) {
     bool IgnoreInput = false;
 
#endif
 
   const ArgStringList &TempFiles = C.getTempFiles();
   if (TempFiles.empty()) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating preprocessed source(s).";
     return;
   }
 
+  std::unique_ptr<llvm::TarWriter> TarWriter;
+  std::string ReproFileName;
+  std::tie(TarWriter, ReproFileName) = openReproTarFile(*this, C);
+  if (!TarWriter) {
+    Diag(clang::diag::note_drv_command_failed_diag_msg)
+        << "Error generating preprocessed source(s). - cannot open tar file";
+    return;
+  }
+
+  std::function<void(StringRef, std::string)> WriteFileToTar =
+      [&WriteFileToTar, &TarWriter](StringRef FSPath, std::string TarPath) {
+        if (llvm::sys::fs::is_directory(FSPath)) {
+          using llvm::sys::fs::recursive_directory_iterator;
+          std::error_code EC;
+          for (recursive_directory_iterator I{FSPath, EC}, E; I != E && !EC;
+               I.increment(EC)) {
+            // Skip over the base path of the dir. i.e., if
+            // WriteFileToTar("a/b/dir", "tmp/"), where dir has "c" and "d"
+            // "a/b/dir/c" -> "dir/", so this will be put into the tar file at
+            // "tmp/dir/c" and "tmp/dir/d".
+            std::string Path = I->path();
+            Path = Path.data() + FSPath.size() + 1;
+            llvm::SmallString<32> Prefix{Path.begin(), Path.end()};
+            llvm::sys::path::remove_filename(Prefix);
+            if (I->type() == llvm::sys::fs::file_type::regular_file)
+              WriteFileToTar(I->path(), (TarPath + Prefix + "/").str());
+          }
+        }
+        auto MBOrErr = llvm::MemoryBuffer::getFile(FSPath);
+        if (!MBOrErr)
+          return;
+
+        TarPath += llvm::sys::path::filename(FSPath);
+        TarWriter->append(TarPath, MBOrErr->get()->getBuffer());
+      };
+
   Diag(clang::diag::note_drv_command_failed_diag_msg)
       << "\n********************\n\n"
          "PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:\n"
          "Preprocessed source(s) and associated run script(s) are located at:";
 
   SmallString<128> VFS;
   SmallString<128> ReproCrashFilename;
   for (const char *TempFile : TempFiles) {
+    if (SaveReproTemps)
       Diag(clang::diag::note_drv_command_failed_diag_msg) << TempFile;
-    if (Report)
-      Report->TemporaryFiles.push_back(TempFile);
+    WriteFileToTar(TempFile, "tmp/");
     if (ReproCrashFilename.empty()) {
       ReproCrashFilename = TempFile;
       llvm::sys::path::replace_extension(ReproCrashFilename, ".crash");
     }
     if (StringRef(TempFile).endswith(".cache")) {
       // In some cases (modules) we'll dump extra data to help with reproducing
       // the crash into a directory next to the output.
       VFS = llvm::sys::path::filename(TempFile);
       llvm::sys::path::append(VFS, "vfs", "vfs.yaml");
     }
   }
 
+  for (const auto &A : C.getArgs().filtered(options::OPT_frewrite_map_file_EQ,
+                                            options::OPT_INPUT))
+    WriteFileToTar(A->getValue(), "input/");
+
   // Assume associated files are based off of the first temporary file.
   CrashReportInfo CrashInfo(TempFiles[0], VFS);
 
+  std::string Storage;
+  llvm::raw_string_ostream ScriptOS{Storage};
+  ScriptOS << "# Crash reproducer for " << getClangFullVersion() << "\n"
+           << "# Driver args: ";
+  printArgList(ScriptOS, C.getInputArgs());
+  ScriptOS << "# Original command: ";
+  Cmd.Print(ScriptOS, "\n", /*Quote=*/true);
+  Cmd.Print(ScriptOS, "\n", /*Quote=*/true, &CrashInfo);
+  if (!AdditionalInformation.empty())
+    ScriptOS << "\n# Additional information: " << AdditionalInformation << "\n";
+  TarWriter->append("repro.sh", Storage);
+
+  if (SaveReproTemps) {
     llvm::SmallString<128> Script(CrashInfo.Filename);
     llvm::sys::path::replace_extension(Script, "sh");
     std::error_code EC;
-  llvm::raw_fd_ostream ScriptOS(Script, EC, llvm::sys::fs::CD_CreateNew,
+    llvm::raw_fd_ostream ScriptFileOS(Script, EC, llvm::sys::fs::CD_CreateNew,
                                       llvm::sys::fs::FA_Write,
                                       llvm::sys::fs::OF_Text);
     if (EC) {
       Diag(clang::diag::note_drv_command_failed_diag_msg)
           << "Error generating run script: " << Script << " " << EC.message();
     } else {
-    ScriptOS << "# Crash reproducer for " << getClangFullVersion() << "\n"
-             << "# Driver args: ";
-    printArgList(ScriptOS, C.getInputArgs());
-    ScriptOS << "# Original command: ";
-    Cmd.Print(ScriptOS, "\n", /*Quote=*/true);
-    Cmd.Print(ScriptOS, "\n", /*Quote=*/true, &CrashInfo);
-    if (!AdditionalInformation.empty())
-      ScriptOS << "\n# Additional information: " << AdditionalInformation
-               << "\n";
-    if (Report)
-      Report->TemporaryFiles.push_back(std::string(Script.str()));
+      ScriptFileOS << Storage;
       Diag(clang::diag::note_drv_command_failed_diag_msg) << Script;
     }
+  }
+
+  Diag(clang::diag::note_drv_command_failed_diag_msg) << ReproFileName;
+  if (Report)
+    Report->TemporaryFiles.push_back(ReproFileName);
 
   // On darwin, provide information about the .crash diagnostic report.
   if (llvm::Triple(llvm::sys::getProcessTriple()).isOSDarwin()) {
     SmallString<128> CrashDiagDir;
     if (getCrashDiagnosticFile(ReproCrashFilename, CrashDiagDir)) {
-      Diag(clang::diag::note_drv_command_failed_diag_msg)
-          << ReproCrashFilename.str();
+      WriteFileToTar(ReproCrashFilename.str(), "");
     } else { // Suggest a directory for the user to look for .crash files.
       llvm::sys::path::append(CrashDiagDir, Name);
       CrashDiagDir += "_<YYYY-MM-DD-HHMMSS>_<hostname>.crash";
       Diag(clang::diag::note_drv_command_failed_diag_msg)
           << "Crash backtrace is located in";
       Diag(clang::diag::note_drv_command_failed_diag_msg)
           << CrashDiagDir.str();
       Diag(clang::diag::note_drv_command_failed_diag_msg)
           << "(choose the .crash file that corresponds to your crash)";
     }
   }
 
-  for (const auto &A : C.getArgs().filtered(options::OPT_frewrite_map_file_EQ))
-    Diag(clang::diag::note_drv_command_failed_diag_msg) << A->getValue();
-
   Diag(clang::diag::note_drv_command_failed_diag_msg)
       << "\n\n********************";
 }
 
 void Driver::setUpResponseFiles(Compilation &C, Command &Cmd) {
   // Since commandLineFitsWithinSystemLimits() may underestimate system's
   // capacity if the tool does not support response files, there is a chance/
   // that things will just work without a response file, so we silently just

clang/test/CMakeLists.txt

 # Test runner infrastructure for Clang. This configures the Clang test trees
 # for use by Lit, and delegates to LLVM's lit test handlers.
 
 llvm_canonicalize_cmake_booleans(
   CLANG_BUILD_EXAMPLES
   CLANG_DEFAULT_PIE_ON_LINUX
   CLANG_ENABLE_ARCMT
   CLANG_ENABLE_STATIC_ANALYZER
   CLANG_PLUGIN_SUPPORT
   CLANG_SPAWN_CC1
   ENABLE_BACKTRACES
   LLVM_ENABLE_NEW_PASS_MANAGER
   LLVM_ENABLE_ZLIB
   LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
   LLVM_ENABLE_THREADS
   LLVM_WITH_Z3
+  CLANG_CRASH_SAVE_TEMPS
   )
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
   MAIN_CONFIG
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py
   PATHS

clang/test/Driver/crash-diagnostics-dir.c

 // RUN: rm -rf %t
 // RUN: not %clang -fcrash-diagnostics-dir=%t -c %s -o - 2>&1 | FileCheck %s
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK: diagnostic msg: {{.*}}{{/|\\}}crash-diagnostics-dir.c.tmp{{(/|\\).*}}.c
+// CHECK: diagnostic msg: {{.*}}{{/|\\}}crash-diagnostics-dir.c.tmp{{(/|\\).*}}.tar

clang/test/Driver/crash-report-clang-cl.cpp

+// UNSUPPORTED: windows
+
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: not %clang_cl -fsyntax-only /Brepro /source-charset:utf-8 \
 // RUN:     -fcrash-diagnostics-dir=%t -- %s 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-clang-cl-*.cpp | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-clang-cl-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-clang-cl-*.cpp" \
+// RUN:     | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh" \
+// RUN:     | FileCheck --check-prefix=CHECKSH %s
+// RN: cat %t/crash-report-clang-cl-*.cpp | FileCheck --check-prefix=CHECKSRC %s
+// RN: cat %t/crash-report-clang-cl-*.sh | FileCheck --check-prefix=CHECKSH %s
 
 // REQUIRES: crash-recovery
 
 #pragma clang __debug crash
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
+// CHECK: note: diagnostic msg: {{.*}}.tar
 
 // __has_feature(cxx_exceptions) is default-off in the cl-compatible driver.
 FOO
 #if __has_feature(cxx_exceptions)
 int a = 1;
 #else
 int a = 0;
 #endif
 // CHECKSRC:      {{^}}FOO
 // CHECKSRC-NEXT: {{^}}#if 0 /* disabled by -frewrite-includes */
 // CHECKSRC-NEXT: {{^}}#if __has_feature(cxx_exceptions)
 // CHECKSRC-NEXT: {{^}}#endif
 // CHECKSRC-NEXT: {{^}}#endif /* disabled by -frewrite-includes */
 // CHECKSRC-NEXT: {{^}}#if 0 /* evaluated by -frewrite-includes */
 // CHECKSRC-NEXT: {{^}}#
 // CHECKSRC-NEXT: {{^}}int a = 1;
 // CHECKSRC-NEXT: {{^}}#else
 // CHECKSRC-NEXT: {{^}}#
 // CHECKSRC-NEXT: {{^}}int a = 0;
 // CHECKSRC-NEXT: {{^}}#endif
 
-// CHECK-NEXT: note: diagnostic msg: {{.*}}crash-report-clang-cl-{{.*}}.cpp
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: {{.*}}"-fsyntax-only"
 // CHECKSH-SAME: /Brepro
 // CHECKSH-SAME: /source-charset:utf-8
 // CHECKSH-NOT: -mno-incremental-linker-compatible
 // CHECKSH-NOT: -finput-charset=utf-8
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-main-file-name" "crash-report-clang-cl.cpp"
 // CHECKSH: "crash-report-{{[^ ]*}}.cpp"

clang/test/Driver/crash-report-header.h

+// UNSUPPORTED: windows
+
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTIONS=1 not %clang -fsyntax-only %s 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-header-*.h | FileCheck --check-prefix=CHECKSRC "%s"
-// RUN: cat %t/crash-report-header-*.sh | FileCheck --check-prefix=CHECKSH "%s"
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-header-*.h" | FileCheck --check-prefix=CHECKSRC "%s"
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh" | FileCheck --check-prefix=CHECKSH "%s"
 // REQUIRES: crash-recovery
 
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.h
+// CHECK: note: diagnostic msg: {{.*}}.tar
 FOO
 // CHECKSRC: FOO
 // CHECKSH: "-cc1"
 // CHECKSH: "-main-file-name" "crash-report-header.h"
 // CHECKSH: "crash-report-header-{{[^ ]*}}.h"

clang/test/Driver/crash-report-modules.m

 // FIXME: Instead of %T/crmdir, it would be nice to just use %t, but the
 // filename ran into path length limits for the rm command on some Windows
 // bots.
 // RUN: rm -rf %T/crmdir
 // RUN: mkdir -p %T/crmdir/i %T/crmdir/m
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%T/crmdir TEMP=%T/crmdir TMP=%T/crmdir \
 // RUN: not %clang -fsyntax-only %s -I %S/Inputs/module -isysroot %/t/i/                 \
 // RUN: -fmodules -fmodules-cache-path=%T/crmdir/m/ -DFOO=BAR 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %T/crmdir/crash-report-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %T/crmdir/crash-report-*.sh
+// RUN: tar tf %T/crmdir/*.tar | sort | FileCheck %s --check-prefix=CHECKTAR
+
+// RUN: tar xOf %T/crmdir/*.tar --wildcards "*/tmp/crash-report-modules-*.m" \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %T/crmdir/*.tar --wildcards "*/repro.sh" \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
 // REQUIRES: crash-recovery
 
 // FIXME: This test creates excessively deep directory hierarchies that cause
 // problems on Windows.
 // UNSUPPORTED: system-windows
 
 @import simple;
 const int x = MODULE_MACRO;
 
 // CHECK: PLEASE submit a bug report to {{.*}} and include the crash backtrace, preprocessed source, and associated run script.
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK: note: diagnostic msg: {{.*}}.tar
+
+// CHECKTAR: {{.*}}/repro.sh
+// CHECKTAR-NEXT: {{.*}}/tmp/crash-report-modules-{{.*}}.m
+// CHECKTAR-NEXT: {{.*}}/tmp/modules/{{.*}}/modules.idx
+// CHECKTAR-NEXT: {{.*}}/tmp/modules/{{.*}}/simple-{{.*}}.pcm
+// CHECKTAR-NEXT: {{.*}}/tmp/vfs/{{.*}}/module/module.modulemap
+// CHECKTAR-NEXT: {{.*}}/tmp/vfs/{{.*}}/module/simple.h
+// CHECKTAR-NEXT: {{.*}}/tmp/vfs/vfs.yaml
 
 // CHECKSRC: @import simple;
 // CHECKSRC: const int x = 10;
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-SAME: "-D" "FOO=BAR"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-isysroot" "{{[^"]*}}/i/"
 // CHECKSH: "-D" "FOO=BAR"
 // CHECKSH-NOT: "-fmodules-cache-path="
 // CHECKSH: "crash-report-modules-{{[^ ]*}}.m"
 // CHECKSH: "-ivfsoverlay" "crash-report-modules-{{[^ ]*}}.cache{{(/|\\\\)}}vfs{{(/|\\\\)}}vfs.yaml"

clang/test/Driver/crash-report-save-temps.c

+// REQUIRES: save-crash-temps
+
+// RUN: rm -rf %t
+// RUN: mkdir %t
+
+// RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH=1 TMPDIR=%t TEMP=%t TMP=%t \
+// RUN:  not %clang -fsyntax-only %s 2>&1 | FileCheck %s
+
+// CHECK: Preprocessed source(s) and associated run script(s) are located at:
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.c
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.sh
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
+
+// RUN: false

clang/test/Driver/crash-report-spaces.c

+// UNSUPPORTED: windows
+
 // RUN: rm -rf "%t"
 // RUN: mkdir "%t"
 // RUN: cp "%s" "%t/crash report spaces.c"
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTIONS=1 not %clang -fsyntax-only "%t/crash report spaces.c" 2>&1 | FileCheck "%s"
-// RUN: cat "%t/crash report spaces"-*.c | FileCheck --check-prefix=CHECKSRC "%s"
-// RUN: cat "%t/crash report spaces"-*.sh | FileCheck --check-prefix=CHECKSH "%s"
+// RUN: tar xOf %t/*.tar --wildcards "*/crash report spaces-*.c" | FileCheck --check-prefix=CHECKSRC "%s"
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh" | FileCheck --check-prefix=CHECKSH "%s"
 // REQUIRES: crash-recovery
 
 #pragma clang __debug parser_crash
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.c
+// CHECK: note: diagnostic msg: {{.*}}.tar
 FOO
 // CHECKSRC: FOO
 // CHECKSH: "-cc1"
 // CHECKSH: "-main-file-name" "crash report spaces.c"
 // CHECKSH: "crash report spaces-{{[^ ]*}}.c"

clang/test/Driver/crash-report-with-asserts.c

+// UNSUPPORTED: windows
+
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: echo '-fsyntax-only                                              \
 // RUN:  -F/tmp/ -I /tmp/ -idirafter /tmp/ -iquote /tmp/ -isystem /tmp/  \
 // RUN:  -iprefix /the/prefix -iwithprefix /tmp -iwithprefixbefore /tmp/ \
 // RUN:  -Xclang -internal-isystem -Xclang /tmp/                         \
 // RUN:  -Xclang -internal-externc-isystem -Xclang /tmp/                 \
 // RUN:  -Xclang -main-file-name -Xclang foo.c                           \
 // RUN:  -DFOO=BAR -DBAR="BAZ QUX"' > %t.rsp
 
 // RUN: env TMPDIR=%t TEMP=%t TMP=%t RC_DEBUG_OPTIONS=1                  \
 // RUN:  CC_PRINT_HEADERS=1 CC_LOG_DIAGNOSTICS=1                         \
 // RUN:  not %clang %s @%t.rsp -DASSERT 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-*.c | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-*.c"            \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"                        \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+
+// RUN: rm -f %t/*.tar
 
 // RUN: env TMPDIR=%t TEMP=%t TMP=%t RC_DEBUG_OPTIONS=1                  \
 // RUN:  CC_PRINT_HEADERS=1 CC_LOG_DIAGNOSTICS=1                         \
 // RUN:  not %clang %s @%t.rsp -DUNREACHABLE 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-with-asserts-*.c | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-with-asserts-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-*.c"            \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"                        \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
 
 // REQUIRES: crash-recovery, asserts
 
 #ifdef ASSERT
 #pragma clang __debug assert
 #elif UNREACHABLE
 #pragma clang __debug llvm_unreachable
 #endif
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}crash-report-with-asserts-{{.*}}.c
+// CHECK: note: diagnostic msg: {{.*}}.tar
 FOO
 // CHECKSRC: FOO
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: {{.*}}"-fsyntax-only"
 // CHECKSH-SAME: "-D" "FOO=BAR"
 // CHECKSH-SAME: "-D" "BAR=BAZ QUX"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"

clang/test/Driver/crash-report.cpp

+// UNSUPPORTED: windows
+
 // RUN: rm -rf %t
 // RUN: mkdir %t
 
 // RUN: echo '-fsyntax-only                                              \
 // RUN:  -F/tmp/ -I /tmp/ -idirafter /tmp/ -iquote /tmp/ -isystem /tmp/  \
 // RUN:  -iprefix /the/prefix -iwithprefix /tmp -iwithprefixbefore /tmp/ \
 // RUN:  -Xclang -internal-isystem -Xclang /tmp/                         \
 // RUN:  -Xclang -internal-externc-isystem -Xclang /tmp/                 \
 // RUN:  -Xclang -main-file-name -Xclang foo.cpp                         \
 // RUN:  -DFOO=BAR -DBAR="BAZ QUX"' --target=x86_64-linux-gnu > %t.rsp
 
 // RUN: env TMPDIR=%t TEMP=%t TMP=%t RC_DEBUG_OPTIONS=1                  \
 // RUN:  CC_PRINT_HEADERS=1 CC_LOG_DIAGNOSTICS=1                         \
 // RUN:  not %clang %s @%t.rsp -DPARSER 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-*.cpp | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-*.cpp"          \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"                        \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+
+// RUN: rm -f %t/*.tar
 
 // RUN: env TMPDIR=%t TEMP=%t TMP=%t RC_DEBUG_OPTIONS=1                  \
 // RUN:  CC_PRINT_HEADERS=1 CC_LOG_DIAGNOSTICS=1                         \
 // RUN:  not %clang %s @%t.rsp -DCRASH 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-*.cpp | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-*.cpp"          \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"                        \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+
+// RUN: rm -f %t/*.tar
 
 // RUN: env TMPDIR=%t TEMP=%t TMP=%t RC_DEBUG_OPTIONS=1                  \
 // RUN:  CC_PRINT_HEADERS=1 CC_LOG_DIAGNOSTICS=1                         \
 // RUN:  not %clang %s @%t.rsp -DFATAL 2>&1 | FileCheck %s
-// RUN: cat %t/crash-report-*.cpp | FileCheck --check-prefix=CHECKSRC %s
-// RUN: cat %t/crash-report-*.sh | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-report-*.cpp"          \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"                        \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
 
 // REQUIRES: crash-recovery
 
 #ifdef PARSER
 #pragma clang __debug parser_crash
 #elif CRASH
 #pragma clang __debug crash
 #elif FATAL
 #pragma clang __debug llvm_fatal_error
 #endif
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}crash-report-{{.*}}.cpp
+// CHECK: note: diagnostic msg: {{.*}}.tar
 
 // __has_feature(cxx_exceptions) is default-on in the gcc-compatible driver.
 FOO
 #if __has_feature(cxx_exceptions)
 int a = 1;
 #else
 int a = 0;
 #endif

clang/test/Driver/rewrite-map-in-diagnostics.c

+// UNSUPPORTED: windows
+
 // RUN: rm -rf "%t"
 // RUN: mkdir -p "%t"
 // RUN: env TMPDIR="%t" TEMP="%t" TMP="%t" RC_DEBUG_OPTION=1 \
 // RUN:     not %clang -fsyntax-only -frewrite-map-file=%p/Inputs/rewrite.map %s 2>&1 \
 // RUN:   | FileCheck %s
+// RUN: tar tf %t/*.tar | sort | FileCheck --check-prefix=CHECKTAR %s
 
 #pragma clang __debug parser_crash
 
-// CHECK: note: diagnostic msg: {{.*}}rewrite.map
+// CHECK: note: diagnostic msg: {{.*}}.tar
+
+// CHECKTAR: {{.*}}/input/rewrite-map-in-diagnostics.c
+// CHECKTAR-NEXT: {{.*}}/input/rewrite.map
+// CHECKTAR-NEXT: {{.*}}/repro.sh
+// CHECKTAR-NEXT: {{.*}}/tmp/rewrite-map-in-diagnostics-{{.*}}.c
 
 // REQUIRES: crash-recovery

clang/test/Index/create-libclang-completion-reproducer.c

 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
 // RUN: %clang -cc1gen-reproducer %t/libclang-* -v | FileCheck %s
 
 // Invocation file must be removed by clang:
 // RUN: ls %t | count 0
 
 // CHECK: REPRODUCER METAINFO: {"libclang.operation": "complete", "libclang.opts": "1", "invocation-args": ["-code-completion-at={{.*}}create-libclang-completion-reproducer.c:10:1"]}
 
 // CHECK: REPRODUCER:
 // CHECK-NEXT: {
-// CHECK-NEXT: "files":["{{.*}}.c","{{.*}}.sh"]
+// CHECK-NEXT: "files":["{{.*}}.tar"]
 // CHECK-NEXT: }

clang/test/Index/create-libclang-parsing-reproducer.c

 // RUN: %clang -cc1gen-reproducer %t/libclang-* -v | FileCheck %s
 
 #pragma clang __debug parser_crash
 
 // CHECK: REPRODUCER METAINFO: {"libclang.operation": "parse", "libclang.opts": "1"}
 
 // CHECK: REPRODUCER:
 // CHECK-NEXT: {
-// CHECK-NEXT: "files":["{{.*}}.c","{{.*}}.sh"]
+// CHECK-NEXT: "files":["{{.*}}.tar"]
 // CHECK-NEXT: }

clang/test/Modules/crash-vfs-path-emptydir-entries.m

 // REQUIRES: crash-recovery, shell
 
 // FIXME: This XFAIL is cargo-culted from crash-report.c. Do we need it?
 // XFAIL: windows-gnu
 
 // Test clang can collect symbolic link headers used in modules.
 // crash reproducer if there's a symbolic link header file used in a module.
 
 // RUN: rm -rf %t
 // RUN: mkdir -p %t/i %t/m %t %t/sysroot
 // RUN: cp -R %S/Inputs/crash-recovery/usr %t/i/
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -fsyntax-only %s -I %/t/i -isysroot %/t/sysroot/ \
 // RUN:     -fmodules -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %t/crash-vfs-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %t/crash-vfs-*.sh
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN: %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "usr/include/stdio.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-vfs-*.m" \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"          \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml"  \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 #include "usr/include/stdio.h"
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKSRC: #pragma clang module import cstd.stdio
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-isysroot" "{{[^"]*}}/sysroot/"
 // CHECKSH-NOT: "-fmodules-cache-path="
 // CHECKSH: "crash-vfs-{{[^ ]*}}.m"
 // CHECKSH: "-ivfsoverlay" "crash-vfs-{{[^ ]*}}.cache/vfs/vfs.yaml"
 // CHECKSH: "-fmodules-cache-path=crash-vfs-{{[^ ]*}}.cache/repro-modules"
 
 // CHECKYAML: 'type': 'directory',
 // CHECKYAML: 'name': "",
 // CHECKYAML-NEXT: 'contents': [
 // CHECKYAML-NEXT:   {
 // CHECKYAML-NEXT:     'type': 'file',
 // CHECKYAML-NEXT:     'name': "pthread_impl.h",
 // CHECKYAML-NEXT:     'external-contents': "/{{.*}}/i/usr/include/pthread_impl.h"
 // CHECKYAML-NEXT:   },
+
+// CHECKTAR: usr/include/stdio.h

clang/test/Modules/crash-vfs-path-symlink-component.m

 // RUN: mkdir -p %t/i %t/m %t %t/sysroot
 // RUN: cp -R %S/Inputs/crash-recovery/usr %t/i/
 // RUN: ln -s include/tcl-private %t/i/usr/x
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -fsyntax-only %s -I %/t/i -isysroot %/t/sysroot/ \
 // RUN:     -fmodules -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %t/crash-vfs-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %t/crash-vfs-*.sh
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN: %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "usr/include/stdio.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-vfs-*.m" \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"          \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml" \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 #include "usr/x/../stdio.h"
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKSRC: #pragma clang module import cstd.stdio
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-isysroot" "{{[^"]*}}/sysroot/"

 // CHECKYAML: 'name': "/[[PATH:.*]]/i/usr",
 // CHECKYAML-NEXT: 'contents': [
 // CHECKYAML-NEXT:   {
 // CHECKYAML-NEXT:     'type': 'file',
 // CHECKYAML-NEXT:     'name': "module.map",
 // CHECKYAML-NEXT:     'external-contents': "/[[PATH]]/i/usr/include/module.map"
 // CHECKYAML-NEXT:   },
 
+// CHECKTAR: usr/include/stdio.h
+
 // Test that by using the previous generated YAML file clang is able to find the
 // right files inside the overlay and map the virtual request for a path that
 // previously contained a symlink to work. To make sure of this, wipe out the
 // %/t/i directory containing the symlink component.
 
 // RUN: rm -rf %/t/i
 // RUN: unset FORCE_CLANG_DIAGNOSTICS_CRASH
 // RUN: %clang -E %s -I %/t/i -isysroot %/t/sysroot/ \
 // RUN:     -ivfsoverlay %t/crash-vfs-*.cache/vfs/vfs.yaml -fmodules \
 // RUN:     -fmodules-cache-path=%t/m/ 2>&1 \
 // RUN:     | FileCheck %s --check-prefix=CHECKOVERLAY
 
 // CHECKOVERLAY: #pragma clang module import cstd.stdio /* clang -E: implicit import

clang/test/Modules/crash-vfs-path-symlink-topheader.m

 // RUN: cp -R %S/Inputs/crash-recovery/usr %t/i/
 // RUN: rm -f %t/i/usr/include/pthread_impl.h
 // RUN: ln -s pthread/pthread_impl.h %t/i/usr/include/pthread_impl.h
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -fsyntax-only %s -I %/t/i -isysroot %/t/sysroot/ \
 // RUN:     -fmodules -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %t/crash-vfs-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %t/crash-vfs-*.sh
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN: %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "usr/include/pthread_impl.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-vfs-*.m" \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"          \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml" \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 #include "usr/include/stdio.h"
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKSRC: #pragma clang module import cstd.stdio
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-isysroot" "{{[^"]*}}/sysroot/"
 // CHECKSH-NOT: "-fmodules-cache-path="
 // CHECKSH: "crash-vfs-{{[^ ]*}}.m"
 // CHECKSH: "-ivfsoverlay" "crash-vfs-{{[^ ]*}}.cache/vfs/vfs.yaml"
 // CHECKSH: "-fmodules-cache-path=crash-vfs-{{[^ ]*}}.cache/repro-modules"
 
 // CHECKYAML: 'type': 'directory',
 // CHECKYAML: 'name': "",
 // CHECKYAML-NEXT: 'contents': [
 // CHECKYAML-NEXT:   {
 // CHECKYAML-NEXT:     'type': 'file',
 // CHECKYAML-NEXT:     'name': "pthread_impl.h",
 // CHECKYAML-NEXT:     'external-contents': "/{{.*}}/i/usr/include/pthread_impl.h"
 // CHECKYAML-NEXT:   },
+
+// CHECKTAR: usr/include/pthread_impl.h

clang/test/Modules/crash-vfs-path-traversal.m

 // REQUIRES: crash-recovery, shell
 // UNSUPPORTED: ms-sdk, ps4
 
 // FIXME: Canonicalizing paths to remove relative traversal components
 // currenty fails a unittest on windows and is disable by default.
 // FIXME: This XFAIL is cargo-culted from crash-report.c. Do we need it?
 // XFAIL: windows-gnu
 
 // RUN: rm -rf %t
 // RUN: mkdir -p %t/i %t/m %t
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -fsyntax-only %s -I %S/Inputs/crash-recovery -isysroot %/t/i/ \
 // RUN: -fmodules -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %t/crash-vfs-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %t/crash-vfs-*.sh
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN: %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "Inputs/crash-recovery/usr/include/stdio.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-vfs-*.m" \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"          \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml" \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 #include "usr/././//////include/../include/./././../include/stdio.h"
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKSRC: #pragma clang module import cstd.stdio
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-isysroot" "{{[^"]*}}/i/"

 // CHECKYAML:     'name': "/[[PATH:.*]]/Inputs/crash-recovery/usr/include",
 // CHECKYAML-NEXT: 'contents': [
 // CHECKYAML-NEXT:   {
 // CHECKYAML-NEXT:     'type': 'file',
 // CHECKYAML-NEXT:     'name': "module.map",
 // CHECKYAML-NEXT:     'external-contents': "/[[PATH]]/Inputs/crash-recovery/usr/include/module.map"
 // CHECKYAML-NEXT:   },
 
+// CHECKTAR: Inputs/crash-recovery/usr/include/stdio.h
+
 // Replace the paths in the YAML files with relative ".." traversals
 // and fed into clang to test whether we're correctly representing them
 // in the VFS overlay.
 
 // RUN: sed -e "s@usr/include@usr/include/../include@g" \
 // RUN:     %t/crash-vfs-*.cache/vfs/vfs.yaml > %t/vfs.yaml
 // RUN: cp %t/vfs.yaml %t/crash-vfs-*.cache/vfs/vfs.yaml
 // RUN: unset FORCE_CLANG_DIAGNOSTICS_CRASH
 // RUN: %clang -E %s -I %S/Inputs/crash-recovery -isysroot %/t/i/ \
 // RUN:     -ivfsoverlay %t/crash-vfs-*.cache/vfs/vfs.yaml -fmodules \
 // RUN:     -fmodules-cache-path=%t/m/ 2>&1 \
 // RUN:     | FileCheck %s --check-prefix=CHECKOVERLAY
 
 // CHECKOVERLAY: #pragma clang module import cstd.stdio /* clang -E: implicit import

clang/test/Modules/crash-vfs-relative-overlay.m

 // REQUIRES: crash-recovery, shell
 
 // FIXME: This XFAIL is cargo-culted from crash-report.c. Do we need it?
 // XFAIL: windows-gnu
 
 // RUN: rm -rf %t
 // RUN: mkdir -p %t/i %t/m %t
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -fsyntax-only -nostdinc %s \
 // RUN:     -I %S/Inputs/crash-recovery/usr/include -isysroot %/t/i/ \
 // RUN:     -fmodules -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKSRC %s -input-file %t/crash-vfs-*.m
-// RUN: FileCheck --check-prefix=CHECKSH %s -input-file %t/crash-vfs-*.sh
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN: %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "Inputs/crash-recovery/usr/include/stdio.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/crash-vfs-*.m"      \
+// RUN:  | FileCheck --check-prefix=CHECKSRC %s
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh"               \
+// RUN:  | FileCheck --check-prefix=CHECKSH %s
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml"       \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 #include <stdio.h>
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKSRC: #pragma clang module import cstd.stdio
 
 // CHECKSH: # Crash reproducer
 // CHECKSH-NEXT: # Driver args: "-fsyntax-only"
 // CHECKSH-NEXT: # Original command: {{.*$}}
 // CHECKSH-NEXT: "-cc1"
 // CHECKSH: "-resource-dir"

 // CHECKYAML-NEXT: 'contents': [
 // CHECKYAML-NEXT:   {
 // CHECKYAML-NEXT:     'type': 'file',
 // CHECKYAML-NEXT:     'name': "module.map",
 // CHECKYAML-NOT:      'external-contents': "{{[^ ]*}}.cache
 // CHECKYAML-NEXT:     'external-contents': "/[[PATH]]/Inputs/crash-recovery/usr/include/module.map"
 // CHECKYAML-NEXT:   },
 
+// CHECKTAR: Inputs/crash-recovery/usr/include/stdio.h
+
 // Test that reading the YAML file will yield the correct path after
 // the overlay dir is prefixed to access headers in .cache/vfs directory.
 
 // RUN: unset FORCE_CLANG_DIAGNOSTICS_CRASH
 // RUN: %clang -E %s -I %S/Inputs/crash-recovery/usr/include -isysroot %/t/i/ \
 // RUN:     -ivfsoverlay %t/crash-vfs-*.cache/vfs/vfs.yaml -fmodules \
 // RUN:     -fmodules-cache-path=%t/m/ 2>&1 \
 // RUN:     | FileCheck %s --check-prefix=CHECKOVERLAY
 
 // CHECKOVERLAY: #pragma clang module import cstd.stdio /* clang -E: implicit import

clang/test/Modules/crash-vfs-umbrella-frameworks.m

 // REQUIRES: crash-recovery, shell
 
 // FIXME: This XFAIL is cargo-culted from crash-report.c. Do we need it?
 // XFAIL: windows-gnu
 
 // RUN: rm -rf %t
 // RUN: mkdir -p %t/i %t/m %t
 // RUN: cp -R %S/Inputs/crash-recovery/Frameworks %t/i/
 // RUN: mkdir -p %t/i/Frameworks/A.framework/Frameworks
 // RUN: ln -s ../../B.framework %t/i/Frameworks/A.framework/Frameworks/B.framework
 
 // RUN: env FORCE_CLANG_DIAGNOSTICS_CRASH= TMPDIR=%t TEMP=%t TMP=%t \
 // RUN: not %clang -nostdinc -fsyntax-only %s \
 // RUN:     -F %/t/i/Frameworks -fmodules \
 // RUN:     -fmodules-cache-path=%t/m/ 2>&1 | FileCheck %s
 
-// RUN: FileCheck --check-prefix=CHECKYAML %s -input-file \
-// RUN:         %t/crash-vfs-*.cache/vfs/vfs.yaml
-// RUN: find %t/crash-vfs-*.cache/vfs | \
-// RUN:   grep "B.framework/Headers/B.h" | count 1
+// RUN: tar xOf %t/*.tar --wildcards "*/tmp/vfs/vfs.yaml" \
+// RUN:  | FileCheck --check-prefix=CHECKYAML %s
+// RUN: tar tf %t/*.tar | FileCheck --check-prefix=CHECKTAR %s
 
 // CHECK: Preprocessed source(s) and associated run script(s) are located at:
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.m
-// CHECK-NEXT: note: diagnostic msg: {{.*}}.cache
+// CHECK-NEXT: note: diagnostic msg: {{.*}}.tar
 
 // CHECKYAML:      'type': 'directory',
 // CHECKYAML:      'name': "/[[PATH:.*]]/i/Frameworks/A.framework/Frameworks/B.framework/Headers",
 // CHECKYAML-NEXT:      'contents': [
 // CHECKYAML-NEXT:        {
 // CHECKYAML-NEXT:          'type': 'file',
 // CHECKYAML-NEXT:          'name': "B.h",
 // CHECKYAML-NEXT:          'external-contents': "/[[PATH]]/i/Frameworks/B.framework/Headers/B.h"
 
 // CHECKYAML:      'type': 'directory',
 // CHECKYAML:      'name': "/[[PATH]]/i/Frameworks/B.framework/Headers",
 // CHECKYAML-NEXT:      'contents': [
 // CHECKYAML-NEXT:        {
 // CHECKYAML-NEXT:          'type': 'file',
 // CHECKYAML-NEXT:          'name': "B.h",
 // CHECKYAML-NEXT:          'external-contents': "/[[PATH]]/i/Frameworks/B.framework/Headers/B.h"
 
+// CHECKTAR: B.framework/Headers/B.h
+
 @import I;
 
 // Run the reproducer script - regular exit code is enough to test it works. The
 // intent here is to guarantee that the collect umbrella headers into the VFS
 // can be used, testing that vfs::recursive_directory_iterator is used correctly
 // Make sure to erase the include paths used to build the modules to guarantee
 // that the VFS overlay won't fallback to use it. Also wipe out the module cache
 // to force header search.
 //
 // RUN: cd %t
 // RUN: rm -rf i
 // RUN: rm -rf crash-vfs-umbrella-*.cache/modules/*
-// RUN: chmod 755 crash-vfs-*.sh
-// RUN: ./crash-vfs-*.sh
+// RUN: tar xOf %t/*.tar --wildcards "*/repro.sh" > repro.sh
+// RUN: chmod 755 repro.sh
+// RUN: ./repro.sh

clang/test/lit.cfg.py

 if platform.system() not in ['Windows']:
     config.available_features.add('utf8-capable-terminal')
 
 # Support for libgcc runtime. Used to rule out tests that require
 # clang to run with -rtlib=libgcc.
 if platform.system() not in ['Darwin', 'Fuchsia']:
     config.available_features.add('libgcc')
 
+if config.save_crash_temps:
+    print("\n\n\n\n\nsave_crash_temps is true\n\n\n\n\n")
+    config.available_features.add('save-crash-temps')
+
 # Case-insensitive file system
 
 
 def is_filesystem_case_insensitive():
     handle, path = tempfile.mkstemp(
         prefix='case-test', dir=config.test_exec_root)
     isInsensitive = os.path.exists(
         os.path.join(

clang/test/lit.site.cfg.py.in

 config.enable_experimental_new_pass_manager = @LLVM_ENABLE_NEW_PASS_MANAGER@
 config.enable_threads = @LLVM_ENABLE_THREADS@
 config.host_arch = "@HOST_ARCH@"
 config.python_executable = "@Python3_EXECUTABLE@"
 config.use_z3_solver = lit_config.params.get('USE_Z3_SOLVER', "@USE_Z3_SOLVER@")
 config.has_plugins = @CLANG_PLUGIN_SUPPORT@
 config.clang_vendor_uti = "@CLANG_VENDOR_UTI@"
 config.llvm_external_lit = path(r"@LLVM_EXTERNAL_LIT@")
+config.save_crash_temps = @CLANG_CRASH_SAVE_TEMPS@
 
 import lit.llvm
 lit.llvm.initialize(lit_config, config)
 
 # Let the main config do the real work.
 lit_config.load_config(
     config, os.path.join(config.clang_src_dir, "test/lit.cfg.py"))