[clang-scan-deps] Introduce the DependencyScanning library with the thread worker code and better error handling

[arphaman]

Summary

This patch extracts out the code that will powers the fast scanning worker into a new file in a new DependencyScanning library. The error and output handling is improved so that the clients can gather errors/results from the worker directly.

[aganea]

LGTM!

Some quick stats on our end (running Windows 10 on a Intel W-2135, 6-core, 3.7 GHz, NVMe SSD): on a large .SLN compiling approx. 16,000 .CPP files through 600 unity .CPPs and 23,000 .H files, out of 86 secs spent in ClangScanDeps, about 32 secs are spent in DirectoryLookup::LookUpFile.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h

29 ↗

(On Diff #206100)

How do you actually select between the fast mode or the regular preprocess?

[aganea]

A bit more detail on what we're seeing on our end (specs in the post above). The 'Count' column represents the number of 1ms samples taken in that function. The 'Weight' column is cumulated times for all cores, for a given process, in ms.

Image below shows most of the time is spent in clang-scan-deps.dll and indirectly, in ntoskrnl.exe. The graph at the bottom also shows that preprocessing the ~600 unity .CPP files in the project issues 16 million IOPS. The graphs on the right show no bottlenecks on the disk I/O on the OS I/O-side.

Top functions in ntoskrnl.exe, most callstacks end up in clang-scan-deps.dll:

A good chunk of all this is caused by llvm::sys::status(), even though is goes through the FileSystemStatCache, like I was suggesting previously (ping @rnk)
(beware, the callstacks are reversed in the image below)

I can take a look at llvm::sys::status() after the vacations (somewhere in August)

All this was implemented & investigated by @SamChaps

[arphaman]

clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h

29 ↗

(On Diff #206100)

You can't select it yet, as it's not yet implemented :)
A couple more follow-up patches, and it will be usable.

[arphaman]

@aganea The FS status and most of file reads will be cached with the shared FS. Hopefully I can put up a patch for it this week.

cfe/trunk/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h

+//===- DependencyScanningWorker.h - clang-scan-deps worker ===---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_WORKER_H
+#define LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_WORKER_H
+
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/LLVM.h"
+#include "clang/Frontend/PCHContainerOperations.h"
+#include "clang/Tooling/CompilationDatabase.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileSystem.h"
+#include <string>
+
+namespace clang {
+namespace tooling {
+namespace dependencies {
+
+/// An individual dependency scanning worker that is able to run on its own
+/// thread.
+///
+/// The worker computes the dependencies for the input files by preprocessing
+/// sources either using a fast mode where the source files are minimized, or
+/// using the regular processing run.
+class DependencyScanningWorker {
+public:
+  DependencyScanningWorker();
+
+  /// Print out the dependency information into a string using the dependency
+  /// file format that is specified in the options (-MD is the default) and
+  /// return it.
+  ///
+  /// \returns A \c StringError with the diagnostic output if clang errors
+  /// occurred, dependency file contents otherwise.
+  llvm::Expected<std::string> getDependencyFile(const std::string &Input,
+                                                StringRef WorkingDirectory,
+                                                const CompilationDatabase &CDB);
+
+private:
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts;
+  std::shared_ptr<PCHContainerOperations> PCHContainerOps;
+
+  /// The file system that is used by each worker when scanning for
+  /// dependencies. This filesystem persists accross multiple compiler
+  /// invocations.
+  llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> WorkerFS;
+};
+
+} // end namespace dependencies
+} // end namespace tooling
+} // end namespace clang
+
+#endif // LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_WORKER_H

cfe/trunk/include/clang/Tooling/Tooling.h

public:
   /// append them to ASTs.
   int buildASTs(std::vector<std::unique_ptr<ASTUnit>> &ASTs);
 
   /// Sets whether working directory should be restored after calling run(). By
   /// default, working directory is restored. However, it could be useful to
   /// turn this off when running on multiple threads to avoid the raciness.
   void setRestoreWorkingDir(bool RestoreCWD);
 
+  /// Sets whether an error message should be printed out if an action fails. By
+  /// default, if an action fails, a message is printed out to stderr.
+  void setPrintErrorMessage(bool PrintErrorMessage);
+
   /// Returns the file manager used in the tool.
   ///
   /// The file manager is shared between all translation units.
   FileManager &getFiles() { return *Files; }
 
   llvm::ArrayRef<std::string> getSourcePaths() const { return SourcePaths; }
 
 private:
private:
 
   llvm::StringSet<> SeenWorkingDirectories;
 
   ArgumentsAdjuster ArgsAdjuster;
 
   DiagnosticConsumer *DiagConsumer = nullptr;
 
   bool RestoreCWD = true;
+  bool PrintErrorMessage = true;
 };
 
 template <typename T>
 std::unique_ptr<FrontendActionFactory> newFrontendActionFactory() {
   class SimpleFrontendActionFactory : public FrontendActionFactory {
   public:
     FrontendAction *create() override { return new T; }
   };

cfe/trunk/lib/Tooling/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   Support
   )
 
 add_subdirectory(Core)
 add_subdirectory(Inclusions)
 add_subdirectory(Refactoring)
 add_subdirectory(ASTDiff)
 add_subdirectory(Syntax)
+add_subdirectory(DependencyScanning)
 
 add_clang_library(clangTooling
   AllTUsExecution.cpp
   ArgumentsAdjusters.cpp
   CommonOptionsParser.cpp
   CompilationDatabase.cpp
   Execution.cpp
   FileMatchTrie.cpp

cfe/trunk/lib/Tooling/DependencyScanning/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Core
+  Support
+  )
+
+add_clang_library(clangDependencyScanning
+  DependencyScanningWorker.cpp
+
+  DEPENDS
+  ClangDriverOptions
+
+  LINK_LIBS
+  clangAST
+  clangBasic
+  clangDriver
+  clangFrontend
+  clangFrontendTool
+  clangLex
+  clangParse
+  clangSerialization
+  clangTooling
+)

cfe/trunk/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp

+//===- DependencyScanningWorker.cpp - clang-scan-deps worker --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Tooling/DependencyScanning/DependencyScanningWorker.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "clang/Frontend/Utils.h"
+#include "clang/Tooling/Tooling.h"
+
+using namespace clang;
+using namespace tooling;
+using namespace dependencies;
+
+namespace {
+
+/// Prints out all of the gathered dependencies into a string.
+class DependencyPrinter : public DependencyFileGenerator {
+public:
+  DependencyPrinter(std::unique_ptr<DependencyOutputOptions> Opts,
+                    std::string &S)
+      : DependencyFileGenerator(*Opts), Opts(std::move(Opts)), S(S) {}
+
+  void finishedMainFile(DiagnosticsEngine &Diags) override {
+    llvm::raw_string_ostream OS(S);
+    outputDependencyFile(OS);
+  }
+
+private:
+  std::unique_ptr<DependencyOutputOptions> Opts;
+  std::string &S;
+};
+
+/// A proxy file system that doesn't call `chdir` when changing the working
+/// directory of a clang tool.
+class ProxyFileSystemWithoutChdir : public llvm::vfs::ProxyFileSystem {
+public:
+  ProxyFileSystemWithoutChdir(
+      llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS)
+      : ProxyFileSystem(std::move(FS)) {}
+
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {
+    assert(!CWD.empty() && "empty CWD");
+    return CWD;
+  }
+
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
+    CWD = Path.str();
+    return {};
+  }
+
+private:
+  std::string CWD;
+};
+
+/// A clang tool that runs the preprocessor in a mode that's optimized for
+/// dependency scanning for the given compiler invocation.
+class DependencyScanningAction : public tooling::ToolAction {
+public:
+  DependencyScanningAction(StringRef WorkingDirectory,
+                           std::string &DependencyFileContents)
+      : WorkingDirectory(WorkingDirectory),
+        DependencyFileContents(DependencyFileContents) {}
+
+  bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
+                     FileManager *FileMgr,
+                     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
+                     DiagnosticConsumer *DiagConsumer) override {
+    // Create a compiler instance to handle the actual work.
+    CompilerInstance Compiler(std::move(PCHContainerOps));
+    Compiler.setInvocation(std::move(Invocation));
+    FileMgr->getFileSystemOpts().WorkingDir = WorkingDirectory;
+    Compiler.setFileManager(FileMgr);
+
+    // Don't print 'X warnings and Y errors generated'.
+    Compiler.getDiagnosticOpts().ShowCarets = false;
+    // Create the compiler's actual diagnostics engine.
+    Compiler.createDiagnostics(DiagConsumer, /*ShouldOwnClient=*/false);
+    if (!Compiler.hasDiagnostics())
+      return false;
+
+    Compiler.createSourceManager(*FileMgr);
+
+    // Create the dependency collector that will collect the produced
+    // dependencies.
+    //
+    // This also moves the existing dependency output options from the
+    // invocation to the collector. The options in the invocation are reset,
+    // which ensures that the compiler won't create new dependency collectors,
+    // and thus won't write out the extra '.d' files to disk.
+    auto Opts = llvm::make_unique<DependencyOutputOptions>(
+        std::move(Compiler.getInvocation().getDependencyOutputOpts()));
+    // We need at least one -MT equivalent for the generator to work.
+    if (Opts->Targets.empty())
+      Opts->Targets = {"clang-scan-deps dependency"};
+    Compiler.addDependencyCollector(std::make_shared<DependencyPrinter>(
+        std::move(Opts), DependencyFileContents));
+
+    auto Action = llvm::make_unique<PreprocessOnlyAction>();
+    const bool Result = Compiler.ExecuteAction(*Action);
+    FileMgr->clearStatCache();
+    return Result;
+  }
+
+private:
+  StringRef WorkingDirectory;
+  /// The dependency file will be written to this string.
+  std::string &DependencyFileContents;
+};
+
+} // end anonymous namespace
+
+DependencyScanningWorker::DependencyScanningWorker() {
+  DiagOpts = new DiagnosticOptions();
+  PCHContainerOps = std::make_shared<PCHContainerOperations>();
+  /// FIXME: Use the shared file system from the service for fast scanning
+  /// mode.
+  WorkerFS = new ProxyFileSystemWithoutChdir(llvm::vfs::getRealFileSystem());
+}
+
+llvm::Expected<std::string>
+DependencyScanningWorker::getDependencyFile(const std::string &Input,
+                                            StringRef WorkingDirectory,
+                                            const CompilationDatabase &CDB) {
+  // Capture the emitted diagnostics and report them to the client
+  // in the case of a failure.
+  std::string DiagnosticOutput;
+  llvm::raw_string_ostream DiagnosticsOS(DiagnosticOutput);
+  TextDiagnosticPrinter DiagPrinter(DiagnosticsOS, DiagOpts.get());
+
+  WorkerFS->setCurrentWorkingDirectory(WorkingDirectory);
+  tooling::ClangTool Tool(CDB, Input, PCHContainerOps, WorkerFS);
+  Tool.clearArgumentsAdjusters();
+  Tool.setRestoreWorkingDir(false);
+  Tool.setPrintErrorMessage(false);
+  Tool.setDiagnosticConsumer(&DiagPrinter);
+  std::string Output;
+  DependencyScanningAction Action(WorkingDirectory, Output);
+  if (Tool.run(&Action)) {
+    return llvm::make_error<llvm::StringError>(DiagnosticsOS.str(),
+                                               llvm::inconvertibleErrorCode());
+  }
+  return Output;
+}

cfe/trunk/lib/Tooling/Tooling.cpp

for (CompileCommand &CompileCommand : CompileCommandsForFile) {
       // customized message for each file.
       LLVM_DEBUG({ llvm::dbgs() << "Processing: " << File << ".\n"; });
       ToolInvocation Invocation(std::move(CommandLine), Action, Files.get(),
                                 PCHContainerOps);
       Invocation.setDiagnosticConsumer(DiagConsumer);
 
       if (!Invocation.run()) {
         // FIXME: Diagnostics should be used instead.
+        if (PrintErrorMessage)
           llvm::errs() << "Error while processing " << File << ".\n";
         ProcessingFailed = true;
       }
     }
   }
 
   if (!InitialWorkingDir.empty()) {
     if (auto EC =
             OverlayFileSystem->setCurrentWorkingDirectory(InitialWorkingDir))
int ClangTool::buildASTs(std::vector<std::unique_ptr<ASTUnit>> &ASTs) {
   ASTBuilderAction Action(ASTs);
   return run(&Action);
 }
 
 void ClangTool::setRestoreWorkingDir(bool RestoreCWD) {
   this->RestoreCWD = RestoreCWD;
 }
 
+void ClangTool::setPrintErrorMessage(bool PrintErrorMessage) {
+  this->PrintErrorMessage = PrintErrorMessage;
+}
+
 namespace clang {
 namespace tooling {
 
 std::unique_ptr<ASTUnit>
 buildASTFromCode(StringRef Code, StringRef FileName,
                  std::shared_ptr<PCHContainerOperations> PCHContainerOps) {
   return buildASTFromCodeWithArgs(Code, std::vector<std::string>(), FileName,
                                   "clang-tool", std::move(PCHContainerOps));

cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json

 [
 {
   "directory": "DIR",
   "command": "clang -E -fsyntax-only DIR/regular_cdb2.cpp -IInputs -D INCLUDE_HEADER2 -MD -MF DIR/regular_cdb2.d",
   "file": "DIR/regular_cdb2.cpp"
 },
 {
   "directory": "DIR",
-  "command": "clang -E -fsyntax-only DIR/regular_cdb.cpp -IInputs",
+  "command": "clang -E DIR/regular_cdb.cpp -IInputs",
   "file": "DIR/regular_cdb.cpp"
 }
 ]

cfe/trunk/test/ClangScanDeps/error.cpp

+// RUN: rm -rf %t.dir
+// RUN: rm -rf %t.cdb
+// RUN: mkdir -p %t.dir
+// RUN: cp %s %t.dir/regular_cdb.cpp
+// RUN: sed -e "s|DIR|%/t.dir|g" %S/Inputs/regular_cdb.json > %t.cdb
+//
+// RUN: not clang-scan-deps -compilation-database %t.cdb -j 1 2>%t.dir/errs
+// RUN: echo EOF >> %t.dir/errs
+// RUN: FileCheck %s --input-file %t.dir/errs
+
+#include "missing.h"
+
+// CHECK: Error while scanning dependencies
+// CHECK-NEXT: error: no such file or directory:
+// CHECK-NEXT: error: no input files
+// CHECK-NEXT: error:
+// CHECK-NEXT: Error while scanning dependencies
+// CHECK-NEXT: fatal error: 'missing.h' file not found
+// CHECK-NEXT: "missing.h"
+// CHECK-NEXT: ^
+// CHECK-NEXT: EOF

cfe/trunk/tools/clang-scan-deps/CMakeLists.txt

set(CLANG_SCAN_DEPS_LIB_DEPS
   clangCodeGen
   clangDriver
   clangFrontend
   clangFrontendTool
   clangLex
   clangParse
   clangSerialization
   clangTooling
+  clangDependencyScanning
   )
 
 target_link_libraries(clang-scan-deps
   PRIVATE
   ${CLANG_SCAN_DEPS_LIB_DEPS}
   )
 

cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp

 //===- ClangScanDeps.cpp - Implementation of clang-scan-deps --------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Frontend/CompilerInstance.h"
-#include "clang/Frontend/CompilerInvocation.h"
-#include "clang/Frontend/FrontendActions.h"
-#include "clang/Frontend/PCHContainerOperations.h"
-#include "clang/FrontendTool/Utils.h"
 #include "clang/Tooling/CommonOptionsParser.h"
+#include "clang/Tooling/DependencyScanning/DependencyScanningWorker.h"
 #include "clang/Tooling/JSONCompilationDatabase.h"
-#include "clang/Tooling/Tooling.h"
-#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/InitLLVM.h"
-#include "llvm/Support/JSON.h"
 #include "llvm/Support/Options.h"
-#include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/Threading.h"
 #include <mutex>
 #include <thread>
 
 using namespace clang;
+using namespace tooling::dependencies;
 
 namespace {
 
 class SharedStream {
 public:
   SharedStream(raw_ostream &OS) : OS(OS) {}
   void applyLocked(llvm::function_ref<void(raw_ostream &OS)> Fn) {
     std::unique_lock<std::mutex> LockGuard(Lock);
     Fn(OS);
     OS.flush();
   }
 
 private:
   std::mutex Lock;
   raw_ostream &OS;
 };
 
-/// Prints out all of the gathered dependencies into one output stream instead
-/// of using the output dependency file.
-class DependencyPrinter : public DependencyFileGenerator {
-public:
-  DependencyPrinter(std::unique_ptr<DependencyOutputOptions> Opts,
-                    SharedStream &OS)
-      : DependencyFileGenerator(*Opts), Opts(std::move(Opts)), OS(OS) {}
-
-  void finishedMainFile(DiagnosticsEngine &Diags) override {
-    OS.applyLocked([this](raw_ostream &OS) { outputDependencyFile(OS); });
-  }
-
-private:
-  std::unique_ptr<DependencyOutputOptions> Opts;
-  SharedStream &OS;
-};
-
-/// A clang tool that runs the preprocessor only for the given compiler
-/// invocation.
-class PreprocessorOnlyTool : public tooling::ToolAction {
-public:
-  PreprocessorOnlyTool(StringRef WorkingDirectory, SharedStream &OS)
-      : WorkingDirectory(WorkingDirectory), OS(OS) {}
-
-  bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
-                     FileManager *FileMgr,
-                     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-                     DiagnosticConsumer *DiagConsumer) override {
-    // Create a compiler instance to handle the actual work.
-    CompilerInstance Compiler(std::move(PCHContainerOps));
-    Compiler.setInvocation(std::move(Invocation));
-    FileMgr->getFileSystemOpts().WorkingDir = WorkingDirectory;
-    Compiler.setFileManager(FileMgr);
-
-    // Create the compiler's actual diagnostics engine.
-    Compiler.createDiagnostics(DiagConsumer, /*ShouldOwnClient=*/false);
-    if (!Compiler.hasDiagnostics())
-      return false;
-
-    Compiler.createSourceManager(*FileMgr);
-
-    // Create the dependency collector that will collect the produced
-    // dependencies.
-    //
-    // This also moves the existing dependency output options from the
-    // invocation to the collector. The options in the invocation are reset,
-    // which ensures that the compiler won't create new dependency collectors,
-    // and thus won't write out the extra '.d' files to disk.
-    auto Opts = llvm::make_unique<DependencyOutputOptions>(
-        std::move(Compiler.getInvocation().getDependencyOutputOpts()));
-    // We need at least one -MT equivalent for the generator to work.
-    if (Opts->Targets.empty())
-      Opts->Targets = {"clang-scan-deps dependency"};
-    Compiler.addDependencyCollector(
-        std::make_shared<DependencyPrinter>(std::move(Opts), OS));
-
-    auto Action = llvm::make_unique<PreprocessOnlyAction>();
-    const bool Result = Compiler.ExecuteAction(*Action);
-    FileMgr->clearStatCache();
-    return Result;
-  }
-
-private:
-  StringRef WorkingDirectory;
-  SharedStream &OS;
-};
-
-/// A proxy file system that doesn't call `chdir` when changing the working
-/// directory of a clang tool.
-class ProxyFileSystemWithoutChdir : public llvm::vfs::ProxyFileSystem {
-public:
-  ProxyFileSystemWithoutChdir(
-      llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS)
-      : ProxyFileSystem(std::move(FS)) {}
-
-  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {
-    assert(!CWD.empty() && "empty CWD");
-    return CWD;
-  }
-
-  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
-    CWD = Path.str();
-    return {};
-  }
-
-private:
-  std::string CWD;
-};
-
 /// The high-level implementation of the dependency discovery tool that runs on
 /// an individual worker thread.
 class DependencyScanningTool {
 public:
   /// Construct a dependency scanning tool.
   ///
   /// \param Compilations     The reference to the compilation database that's
   /// used by the clang tool.
   DependencyScanningTool(const tooling::CompilationDatabase &Compilations,
-                         SharedStream &OS)
-      : Compilations(Compilations), OS(OS) {
-    PCHContainerOps = std::make_shared<PCHContainerOperations>();
-    BaseFS = new ProxyFileSystemWithoutChdir(llvm::vfs::getRealFileSystem());
-  }
+                         SharedStream &OS, SharedStream &Errs)
+      : Compilations(Compilations), OS(OS), Errs(Errs) {}
 
-  /// Computes the dependencies for the given file.
+  /// Computes the dependencies for the given file and prints them out.
   ///
   /// \returns True on error.
   bool runOnFile(const std::string &Input, StringRef CWD) {
-    BaseFS->setCurrentWorkingDirectory(CWD);
-    tooling::ClangTool Tool(Compilations, Input, PCHContainerOps, BaseFS);
-    Tool.clearArgumentsAdjusters();
-    Tool.setRestoreWorkingDir(false);
-    PreprocessorOnlyTool Action(CWD, OS);
-    return Tool.run(&Action);
+    auto MaybeFile = Worker.getDependencyFile(Input, CWD, Compilations);
+    if (!MaybeFile) {
+      llvm::handleAllErrors(
+          MaybeFile.takeError(), [this, &Input](llvm::StringError &Err) {
+            Errs.applyLocked([&](raw_ostream &OS) {
+              OS << "Error while scanning dependencies for " << Input << ":\n";
+              OS << Err.getMessage();
+            });
+          });
+      return true;
+    }
+    OS.applyLocked([&](raw_ostream &OS) { OS << *MaybeFile; });
+    return false;
   }
 
 private:
+  DependencyScanningWorker Worker;
   const tooling::CompilationDatabase &Compilations;
   SharedStream &OS;
-  std::shared_ptr<PCHContainerOperations> PCHContainerOps;
-  /// The real filesystem used as a base for all the operations performed by the
-  /// tool.
-  llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS;
+  SharedStream &Errs;
 };
 
 llvm::cl::opt<bool> Help("h", llvm::cl::desc("Alias for -help"),
                          llvm::cl::Hidden);
 
 llvm::cl::OptionCategory DependencyScannerCategory("Tool options");
 
 llvm::cl::opt<unsigned>
AdjustingCompilations->appendArgumentsAdjuster(
         AdjustedArgs.push_back("/dev/null");
         AdjustedArgs.push_back("-Xclang");
         AdjustedArgs.push_back("-Eonly");
         AdjustedArgs.push_back("-Xclang");
         AdjustedArgs.push_back("-sys-header-deps");
         return AdjustedArgs;
       });
 
+  SharedStream Errs(llvm::errs());
   // Print out the dependency results to STDOUT by default.
   SharedStream DependencyOS(llvm::outs());
   unsigned NumWorkers =
       NumThreads == 0 ? llvm::hardware_concurrency() : NumThreads;
   std::vector<std::unique_ptr<DependencyScanningTool>> WorkerTools;
   for (unsigned I = 0; I < NumWorkers; ++I)
     WorkerTools.push_back(llvm::make_unique<DependencyScanningTool>(
-        *AdjustingCompilations, DependencyOS));
+        *AdjustingCompilations, DependencyOS, Errs));
 
   std::vector<std::thread> WorkerThreads;
   std::atomic<bool> HadErrors(false);
   std::mutex Lock;
   size_t Index = 0;
 
   llvm::outs() << "Running clang-scan-deps on " << Inputs.size()
                << " files using " << NumWorkers << " workers\n";