[clang][deps] Prune unused header search paths

[jansvoboda11]

Summary

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.

In this patch, the lookup cache in HeaderSearch is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.

We could also prune the header search paths when serializing HeaderSearchOptions into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.

There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.

Depends on D102923.

[jansvoboda11]

Undo changes to modules-full.cpp test

[dexonsmith]

The code looks mostly good; some inline comments.

I think it'd be useful to have direct tests for "detecting used search paths". That might be a nice thing to separate out and test independently. E.g., you could add a remark, -Rused-search-path, which printed a diagnostic the first time any given search path is used in a TU.

It'd also be really nice to be able to dump this info from the .pcm file. I guess it'd be outside the scope of this patch, but I really think there should be a clang-pcm-analyzer where we can put functionality like that. Somewhat useful for more fine-grained testing; super useful as a tool for debugging.

clang/include/clang/Lex/HeaderSearch.h

295 ↗

(On Diff #345412)

I think the word "get" here is a bit misleading, since this isn't a simple accessor, it's a potentially expensive computation. I suggest "compute", unless it's changed to be computed on-the-fly.

I wonder, would llvm::BitVector be more appropriate here, since we expect it to count from 0 and be relatively dense?

clang/lib/Serialization/ASTReader.cpp

4763–4771

If this were stored as a blob it could be lazily parsed; WDYT?

Alternatively, it might be more compact if stored as a bit vector, storing 64-bits positions per element of the record.

[jansvoboda11]

Use, serialize and deserialize llvm::BitVector instead of std::set.

[jansvoboda11]

In D102488#2760406, @dexonsmith wrote:

I think it'd be useful to have direct tests for "detecting used search paths". That might be a nice thing to separate out and test independently. E.g., you could add a remark, -Rused-search-path, which printed a diagnostic the first time any given search path is used in a TU.

Good idea, I'll look into that.

It'd also be really nice to be able to dump this info from the .pcm file. I guess it'd be outside the scope of this patch, but I really think there should be a clang-pcm-analyzer where we can put functionality like that. Somewhat useful for more fine-grained testing; super useful as a tool for debugging.

Agreed, it's in my todo list.

clang/lib/Serialization/ASTReader.cpp

4763–4771

I'll look into making this lazy.

[jansvoboda11]

Make PCM parsing lazier

[jansvoboda11]

Fix test on Windows

[jansvoboda11]

Split out extra patch

[jansvoboda11]

This is now ready for review.

clang/test/ClangScanDeps/Inputs/header-search-pruning/cdb.json

5

The extra numerical include directories here force AST{Writer,Reader} to work with multi-byte blob, providing extra test coverage.

clang/test/ClangScanDeps/header-search-pruning.cpp

28

The being and end include directories here are sentinels that make it easier to make positive/negative occurrence checks with -NEXT.

[jansvoboda11]

clang/tools/clang-scan-deps/ClangScanDeps.cpp

361

This change will be reverted in favor of D103516.

[jansvoboda11]

After speaking with @dexonsmith, the laziness is probably not necessary when dealing with a short bit vector. I'll also explore using llvm::BitVector instead of std::vector<bool>.

[jansvoboda11]

Rebase, switch from std::vector<bool> to llvm::BitVector, load the bit vector eagerly.

[dexonsmith]

LGTM, thanks!

clang/include/clang/Serialization/ASTBitCodes.h

enum UnhashedControlBlockRecordTypes {
   /// Record code for the content hash of the AST block.
   AST_BLOCK_HASH,
 
   /// Record code for the diagnostic options table.
   DIAGNOSTIC_OPTIONS,
 
   /// Record code for \#pragma diagnostic mappings.
   DIAG_PRAGMA_MAPPINGS,
+
+  /// Record code for the indices of used header search entries.
+  HEADER_SEARCH_ENTRY_USAGE,
 };
 
 /// Record code for extension blocks.
 enum ExtensionBlockRecordTypes {
   /// Metadata describing this particular extension.
   EXTENSION_METADATA = 1,
 
   /// The first record ID allocated to the extensions themselves.

clang/include/clang/Serialization/ModuleFile.h

 #define LLVM_CLANG_SERIALIZATION_MODULEFILE_H
 
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/Module.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Serialization/ASTBitCodes.h"
 #include "clang/Serialization/ContinuousRangeMap.h"
 #include "clang/Serialization/ModuleFileExtension.h"
+#include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/PointerIntPair.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Bitstream/BitstreamReader.h"
 #include "llvm/Support/Endian.h"
 #include <cassert>
public:
   /// The signature of the module file, which may be used instead of the size
   /// and modification time to identify this particular file.
   ASTFileSignature Signature;
 
   /// The signature of the AST block of the module file, this can be used to
   /// unique module files based on AST contents.
   ASTFileSignature ASTBlockHash;
 
+  /// The bit vector denoting usage of each header search entry (true = used).
+  llvm::BitVector SearchPathUsage;
+
   /// Whether this module has been directly imported by the
   /// user.
   bool DirectlyImported = false;
 
   /// The generation of which this module file is a part.
   unsigned Generation;
 
   /// The memory buffer that stores the data associated with

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

 };
 
 /// The dependency scanning service contains the shared state that is used by
 /// the invidual dependency scanning workers.
 class DependencyScanningService {
 public:
   DependencyScanningService(ScanningMode Mode, ScanningOutputFormat Format,
                             bool ReuseFileManager = true,
-                            bool SkipExcludedPPRanges = true);
+                            bool SkipExcludedPPRanges = true,
+                            bool OptimizeArgs = false);
 
   ScanningMode getMode() const { return Mode; }
 
   ScanningOutputFormat getFormat() const { return Format; }
 
   bool canReuseFileManager() const { return ReuseFileManager; }
 
   bool canSkipExcludedPPRanges() const { return SkipExcludedPPRanges; }
 
+  bool canOptimizeArgs() const { return OptimizeArgs; }
+
   DependencyScanningFilesystemSharedCache &getSharedCache() {
     return SharedCache;
   }
 
 private:
   const ScanningMode Mode;
   const ScanningOutputFormat Format;
   const bool ReuseFileManager;
   /// Set to true to use the preprocessor optimization that skips excluded PP
   /// ranges by bumping the buffer pointer in the lexer instead of lexing the
   /// tokens in the range until reaching the corresponding directive.
   const bool SkipExcludedPPRanges;
+  /// Whether to optimize the modules' command-line arguments.
+  const bool OptimizeArgs;
   /// The global file system cache.
   DependencyScanningFilesystemSharedCache SharedCache;
 };
 
 } // end namespace dependencies
 } // end namespace tooling
 } // end namespace clang
 
 #endif // LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_SERVICE_H

clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h

private:
   /// The file system that is used by each worker when scanning for
   /// dependencies. This filesystem persists across multiple compiler
   /// invocations.
   llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS;
   /// The file manager that is reused across multiple invocations by this
   /// worker. If null, the file manager will not be reused.
   llvm::IntrusiveRefCntPtr<FileManager> Files;
   ScanningOutputFormat Format;
+  /// Whether to optimize the modules' command-line arguments.
+  bool OptimizeArgs;
 };
 
 } // end namespace dependencies
 } // end namespace tooling
 } // end namespace clang
 
 #endif // LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_WORKER_H

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

 };
 
 /// Collects modular and non-modular dependencies of the main file by attaching
 /// \c ModuleDepCollectorPP to the preprocessor.
 class ModuleDepCollector final : public DependencyCollector {
 public:
   ModuleDepCollector(std::unique_ptr<DependencyOutputOptions> Opts,
                      CompilerInstance &I, DependencyConsumer &C,
-                     CompilerInvocation &&OriginalCI);
+                     CompilerInvocation &&OriginalCI, bool OptimizeArgs);
 
   void attachToPreprocessor(Preprocessor &PP) override;
   void attachToASTReader(ASTReader &R) override;
 
 private:
   friend ModuleDepCollectorPP;
 
   /// The compiler instance for the current translation unit.
   CompilerInstance &Instance;
   /// The consumer of collected dependency information.
   DependencyConsumer &Consumer;
   /// Path to the main source file.
   std::string MainFile;
   /// Hash identifying the compilation conditions of the current TU.
   std::string ContextHash;
   /// Non-modular file dependencies. This includes the main source file and
   /// textually included header files.
   std::vector<std::string> FileDeps;
   /// Direct and transitive modular dependencies of the main source file.
   std::unordered_map<const Module *, ModuleDeps> ModularDeps;
   /// Options that control the dependency output generation.
   std::unique_ptr<DependencyOutputOptions> Opts;
   /// The original Clang invocation passed to dependency scanner.
   CompilerInvocation OriginalInvocation;
+  /// Whether to optimize the modules' command-line arguments.
+  bool OptimizeArgs;
 
   /// Checks whether the module is known as being prebuilt.
   bool isPrebuiltModule(const Module *M);
 
   /// Constructs a CompilerInvocation that can be used to build the given
   /// module, excluding paths to discovered modular dependencies that are yet to
   /// be built.
-  CompilerInvocation
-  makeInvocationForModuleBuildWithoutPaths(const ModuleDeps &Deps) const;
+  CompilerInvocation makeInvocationForModuleBuildWithoutPaths(
+      const ModuleDeps &Deps,
+      llvm::function_ref<void(CompilerInvocation &)> Optimize) const;
 };
 
 } // end namespace dependencies
 } // end namespace tooling
 } // end namespace clang
 
 #endif // LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_MODULE_DEP_COLLECTOR_H

clang/lib/Serialization/ASTReader.cpp

while (true) {
 
     case llvm::BitstreamEntry::Record:
       // The interesting case.
       break;
     }
 
     // Read and process a record.
     Record.clear();
-    Expected<unsigned> MaybeRecordType = Stream.readRecord(Entry.ID, Record);
+    StringRef Blob;
+    Expected<unsigned> MaybeRecordType =
+        Stream.readRecord(Entry.ID, Record, &Blob);
     if (!MaybeRecordType) {
       // FIXME this drops the error.
       return Failure;
     }
     switch ((UnhashedControlBlockRecordTypes)MaybeRecordType.get()) {
     case SIGNATURE:
       if (F)
         F->Signature = ASTFileSignature::create(Record.begin(), Record.end());
case DIAG_PRAGMA_MAPPINGS:
       if (!F)
         break;
       if (F->PragmaDiagMappings.empty())
         F->PragmaDiagMappings.swap(Record);
       else
         F->PragmaDiagMappings.insert(F->PragmaDiagMappings.end(),
                                      Record.begin(), Record.end());
       break;
+    case HEADER_SEARCH_ENTRY_USAGE:
+      if (!F)
+        break;
+      unsigned Count = Record[0];
+      const char *Byte = Blob.data();
+      F->SearchPathUsage = llvm::BitVector(Count, 0);
+      for (unsigned I = 0; I < Count; ++Byte)
+        for (unsigned Bit = 0; Bit < 8 && I < Count; ++Bit, ++I)
+          if (*Byte & (1 << Bit))
+            F->SearchPathUsage[I] = 1;
+      break;
     }
   }
 }
 
 /// Parse a record and blob containing module file extension metadata.
 static bool parseModuleFileExtensionMetadata(
               const SmallVectorImpl<uint64_t> &Record,
               StringRef Blob,

clang/lib/Serialization/ASTWriter.cpp

 }
 
 template <typename T>
 static StringRef bytes(const SmallVectorImpl<T> &v) {
   return StringRef(reinterpret_cast<const char*>(v.data()),
                          sizeof(T) * v.size());
 }
 
+static std::string bytes(const std::vector<bool> &V) {
+  std::string Str;
+  Str.reserve(V.size() / 8);
+  for (unsigned I = 0, E = V.size(); I < E;) {
+    char Byte = 0;
+    for (unsigned Bit = 0; Bit < 8 && I < E; ++Bit, ++I)
+      Byte |= V[I] << Bit;
+    Str += Byte;
+  }
+  return Str;
+}
+
 //===----------------------------------------------------------------------===//
 // Type serialization
 //===----------------------------------------------------------------------===//
 
 static TypeCode getTypeCodeForTypeClass(Type::TypeClass id) {
   switch (id) {
 #define TYPE_BIT_CODE(CLASS_ID, CODE_ID, CODE_VALUE) \
   case Type::CLASS_ID: return TYPE_##CODE_ID;
ASTWriter::createSignature(StringRef AllBytes, StringRef ASTBlockBytes) {
   Hash = Hasher.result();
   ASTFileSignature Signature = ASTFileSignature::create(Hash);
 
   return std::make_pair(ASTBlockHash, Signature);
 }
 
 ASTFileSignature ASTWriter::writeUnhashedControlBlock(Preprocessor &PP,
                                                       ASTContext &Context) {
+  using namespace llvm;
+
   // Flush first to prepare the PCM hash (signature).
   Stream.FlushToWord();
   auto StartOfUnhashedControl = Stream.GetCurrentBitNo() >> 3;
 
   // Enter the block and prepare to write records.
   RecordData Record;
   Stream.EnterSubblock(UNHASHED_CONTROL_BLOCK_ID, 5);
 
#include "clang/Basic/DiagnosticOptions.def"
   for (unsigned I = 0, N = DiagOpts.Warnings.size(); I != N; ++I)
     AddString(DiagOpts.Warnings[I], Record);
   Record.push_back(DiagOpts.Remarks.size());
   for (unsigned I = 0, N = DiagOpts.Remarks.size(); I != N; ++I)
     AddString(DiagOpts.Remarks[I], Record);
   // Note: we don't serialize the log or serialization file names, because they
   // are generally transient files and will almost always be overridden.
   Stream.EmitRecord(DIAGNOSTIC_OPTIONS, Record);
+  Record.clear();
 
   // Write out the diagnostic/pragma mappings.
   WritePragmaDiagnosticMappings(Diags, /* isModule = */ WritingModule);
 
+  // Header search entry usage.
+  auto HSEntryUsage = PP.getHeaderSearchInfo().computeUserEntryUsage();
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(HEADER_SEARCH_ENTRY_USAGE));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Number of bits.
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob));      // Bit vector.
+  unsigned HSUsageAbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+  {
+    RecordData::value_type Record[] = {HEADER_SEARCH_ENTRY_USAGE,
+                                       HSEntryUsage.size()};
+    Stream.EmitRecordWithBlob(HSUsageAbbrevCode, Record, bytes(HSEntryUsage));
+  }
+
   // Leave the options block.
   Stream.ExitBlock();
   return Signature;
 }
 
 /// Write the control block.
 void ASTWriter::WriteControlBlock(Preprocessor &PP, ASTContext &Context,
                                   StringRef isysroot,

clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp

 #include "llvm/Support/TargetSelect.h"
 
 using namespace clang;
 using namespace tooling;
 using namespace dependencies;
 
 DependencyScanningService::DependencyScanningService(
     ScanningMode Mode, ScanningOutputFormat Format, bool ReuseFileManager,
-    bool SkipExcludedPPRanges)
+    bool SkipExcludedPPRanges, bool OptimizeArgs)
     : Mode(Mode), Format(Format), ReuseFileManager(ReuseFileManager),
-      SkipExcludedPPRanges(SkipExcludedPPRanges) {
+      SkipExcludedPPRanges(SkipExcludedPPRanges), OptimizeArgs(OptimizeArgs) {
   // Initialize targets for object file support.
   llvm::InitializeAllTargets();
   llvm::InitializeAllTargetMCs();
   llvm::InitializeAllAsmPrinters();
   llvm::InitializeAllAsmParsers();
 }

clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp

 /// A clang tool that runs the preprocessor in a mode that's optimized for
 /// dependency scanning for the given compiler invocation.
 class DependencyScanningAction : public tooling::ToolAction {
 public:
   DependencyScanningAction(
       StringRef WorkingDirectory, DependencyConsumer &Consumer,
       llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS,
       ExcludedPreprocessorDirectiveSkipMapping *PPSkipMappings,
-      ScanningOutputFormat Format, llvm::Optional<StringRef> ModuleName = None)
+      ScanningOutputFormat Format, bool OptimizeArgs,
+      llvm::Optional<StringRef> ModuleName = None)
       : WorkingDirectory(WorkingDirectory), Consumer(Consumer),
         DepFS(std::move(DepFS)), PPSkipMappings(PPSkipMappings), Format(Format),
-        ModuleName(ModuleName) {}
+        OptimizeArgs(OptimizeArgs), ModuleName(ModuleName) {}
 
   bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
                      FileManager *FileMgr,
                      std::shared_ptr<PCHContainerOperations> PCHContainerOps,
                      DiagnosticConsumer *DiagConsumer) override {
     // Make a deep copy of the original Clang invocation.
     CompilerInvocation OriginalInvocation(*Invocation);
 
bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
     switch (Format) {
     case ScanningOutputFormat::Make:
       Compiler.addDependencyCollector(
           std::make_shared<DependencyConsumerForwarder>(std::move(Opts),
                                                         Consumer));
       break;
     case ScanningOutputFormat::Full:
       Compiler.addDependencyCollector(std::make_shared<ModuleDepCollector>(
-          std::move(Opts), Compiler, Consumer, std::move(OriginalInvocation)));
+          std::move(Opts), Compiler, Consumer, std::move(OriginalInvocation),
+          OptimizeArgs));
       break;
     }
 
     // Consider different header search and diagnostic options to create
     // different modules. This avoids the unsound aliasing of module PCMs.
     //
-    // TODO: Implement diagnostic bucketing and header search pruning to reduce
-    // the impact of strict context hashing.
+    // TODO: Implement diagnostic bucketing to reduce the impact of strict
+    // context hashing.
     Compiler.getHeaderSearchOpts().ModulesStrictContextHash = true;
 
     std::unique_ptr<FrontendAction> Action;
 
     if (ModuleName.hasValue())
       Action = std::make_unique<GetDependenciesByModuleNameAction>(*ModuleName);
     else
       Action = std::make_unique<ReadPCHAndPreprocessAction>();
 
     const bool Result = Compiler.ExecuteAction(*Action);
     if (!DepFS)
       FileMgr->clearStatCache();
     return Result;
   }
 
 private:
   StringRef WorkingDirectory;
   DependencyConsumer &Consumer;
   llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS;
   ExcludedPreprocessorDirectiveSkipMapping *PPSkipMappings;
   ScanningOutputFormat Format;
+  bool OptimizeArgs;
   llvm::Optional<StringRef> ModuleName;
 };
 
 } // end anonymous namespace
 
 DependencyScanningWorker::DependencyScanningWorker(
     DependencyScanningService &Service)
-    : Format(Service.getFormat()) {
+    : Format(Service.getFormat()), OptimizeArgs(Service.canOptimizeArgs()) {
   PCHContainerOps = std::make_shared<PCHContainerOperations>();
   PCHContainerOps->registerReader(
       std::make_unique<ObjectFilePCHContainerReader>());
   // We don't need to write object files, but the current PCH implementation
   // requires the writer to be registered as well.
   PCHContainerOps->registerWriter(
       std::make_unique<ObjectFilePCHContainerWriter>());
 
llvm::Error DependencyScanningWorker::computeDependencies(
   std::vector<const char *> FinalCCommandLine(CommandLine.size(), nullptr);
   llvm::transform(CommandLine, FinalCCommandLine.begin(),
                   [](const std::string &Str) { return Str.c_str(); });
 
   return runWithDiags(CreateAndPopulateDiagOpts(FinalCCommandLine).release(),
                       [&](DiagnosticConsumer &DC, DiagnosticOptions &DiagOpts) {
                         DependencyScanningAction Action(
                             WorkingDirectory, Consumer, DepFS,
-                            PPSkipMappings.get(), Format, ModuleName);
+                            PPSkipMappings.get(), Format, OptimizeArgs,
+                            ModuleName);
                         // Create an invocation that uses the underlying file
                         // system to ensure that any file system requests that
                         // are made by the driver do not go through the
                         // dependency scanning filesystem.
                         ToolInvocation Invocation(FinalCommandLine, &Action,
                                                   CurrentFiles.get(),
                                                   PCHContainerOps);
                         Invocation.setDiagnosticConsumer(&DC);
                         Invocation.setDiagnosticOptions(&DiagOpts);
                         return Invocation.run();
                       });
 }

clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp

 #include "clang/Lex/Preprocessor.h"
 #include "clang/Tooling/DependencyScanning/DependencyScanningWorker.h"
 #include "llvm/Support/StringSaver.h"
 
 using namespace clang;
 using namespace tooling;
 using namespace dependencies;
 
+static void optimizeHeaderSearchOpts(HeaderSearchOptions &Opts,
+                                     ASTReader &Reader,
+                                     const serialization::ModuleFile &MF) {
+  // Only preserve search paths that were used during the dependency scan.
+  std::vector<HeaderSearchOptions::Entry> Entries = Opts.UserEntries;
+  Opts.UserEntries.clear();
+  for (unsigned I = 0; I < Entries.size(); ++I)
+    if (MF.SearchPathUsage[I])
+      Opts.UserEntries.push_back(Entries[I]);
+}
+
 CompilerInvocation ModuleDepCollector::makeInvocationForModuleBuildWithoutPaths(
-    const ModuleDeps &Deps) const {
+    const ModuleDeps &Deps,
+    llvm::function_ref<void(CompilerInvocation &)> Optimize) const {
   // Make a deep copy of the original Clang invocation.
   CompilerInvocation CI(OriginalInvocation);
 
   CI.getLangOpts()->resetNonModularOptions();
   CI.getPreprocessorOpts().resetNonModularOptions();
 
   // Remove options incompatible with explicit module build.
   CI.getFrontendOpts().Inputs.clear();
   CI.getFrontendOpts().OutputFile.clear();
 
   CI.getFrontendOpts().ProgramAction = frontend::GenerateModule;
   CI.getLangOpts()->ModuleName = Deps.ID.ModuleName;
   CI.getFrontendOpts().IsSystemModule = Deps.IsSystem;
 
   CI.getLangOpts()->ImplicitModules = false;
 
   // Report the prebuilt modules this module uses.
   for (const auto &PrebuiltModule : Deps.PrebuiltModuleDeps) {
     CI.getFrontendOpts().ModuleFiles.push_back(PrebuiltModule.PCMFile);
     CI.getFrontendOpts().ModuleMapFiles.push_back(PrebuiltModule.ModuleMapFile);
   }
 
+  Optimize(CI);
+
   return CI;
 }
 
 static std::vector<std::string>
 serializeCompilerInvocation(const CompilerInvocation &CI) {
   // Set up string allocator.
   llvm::BumpPtrAllocator Alloc;
   llvm::StringSaver Strings(Alloc);
MDC.Instance.getASTReader()->visitInputFiles(
       });
 
   // Add direct prebuilt module dependencies now, so that we can use them when
   // creating a CompilerInvocation and computing context hash for this
   // ModuleDeps instance.
   llvm::DenseSet<const Module *> SeenModules;
   addAllSubmodulePrebuiltDeps(M, MD, SeenModules);
 
-  MD.Invocation = MDC.makeInvocationForModuleBuildWithoutPaths(MD);
+  MD.Invocation = MDC.makeInvocationForModuleBuildWithoutPaths(
+      MD, [&](CompilerInvocation &CI) {
+        if (MDC.OptimizeArgs)
+          optimizeHeaderSearchOpts(CI.getHeaderSearchOpts(),
+                                   *MDC.Instance.getASTReader(), *MF);
+      });
   MD.ID.ContextHash = MD.Invocation.getModuleHash();
 
   llvm::DenseSet<const Module *> AddedModules;
   addAllSubmoduleDeps(M, MD, AddedModules);
 
   return MD.ID;
 }
 
if (Import->getTopLevelModule() != M->getTopLevelModule() &&
       if (AddedModules.insert(Import->getTopLevelModule()).second)
         MD.ClangModuleDeps.push_back(ImportID);
     }
   }
 }
 
 ModuleDepCollector::ModuleDepCollector(
     std::unique_ptr<DependencyOutputOptions> Opts, CompilerInstance &I,
-    DependencyConsumer &C, CompilerInvocation &&OriginalCI)
+    DependencyConsumer &C, CompilerInvocation &&OriginalCI, bool OptimizeArgs)
     : Instance(I), Consumer(C), Opts(std::move(Opts)),
-      OriginalInvocation(std::move(OriginalCI)) {}
+      OriginalInvocation(std::move(OriginalCI)), OptimizeArgs(OptimizeArgs) {}
 
 void ModuleDepCollector::attachToPreprocessor(Preprocessor &PP) {
   PP.addPPCallbacks(std::make_unique<ModuleDepCollectorPP>(Instance, *this));
 }
 
 void ModuleDepCollector::attachToASTReader(ASTReader &R) {}
 
 bool ModuleDepCollector::isPrebuiltModule(const Module *M) {

clang/test/ClangScanDeps/Inputs/header-search-pruning/cdb.json

+[
+  {
+    "directory": "DIR",
+    "command": "clang -E DIR/header-search-pruning.cpp -Ibegin -I1 -Ia -I3 -I4 -I5 -I6 -Ib -I8 -Iend DEFINES -fmodules -fcxx-modules -fmodules-cache-path=DIR/module-cache -fimplicit-modules -fmodule-map-file=DIR/module.modulemap",
+    "file": "DIR/header-search-pruning.cpp"
+  }
+]

clang/test/ClangScanDeps/Inputs/header-search-pruning/mod.h

+#include "begin.h"
+
+#ifdef INCLUDE_A
+#include "a.h"
+#endif
+
+#ifdef INCLUDE_B
+#include "b.h"
+#endif
+
+#include "end.h"

clang/test/ClangScanDeps/Inputs/header-search-pruning/module.modulemap

+module mod {
+    header "mod.h"
+    export *
+}

clang/test/ClangScanDeps/header-search-pruning.cpp

+// RUN: rm -rf %t && mkdir -p %t
+// RUN: cp -r %S/Inputs/header-search-pruning/* %t
+// RUN: cp %S/header-search-pruning.cpp %t/header-search-pruning.cpp
+// RUN: sed -e "s|DIR|%/t|g" -e "s|DEFINES|-DINCLUDE_A|g"             %S/Inputs/header-search-pruning/cdb.json > %t/cdb_a.json
+// RUN: sed -e "s|DIR|%/t|g" -e "s|DEFINES|-DINCLUDE_B|g"             %S/Inputs/header-search-pruning/cdb.json > %t/cdb_b.json
+// RUN: sed -e "s|DIR|%/t|g" -e "s|DEFINES|-DINCLUDE_A -DINCLUDE_B|g" %S/Inputs/header-search-pruning/cdb.json > %t/cdb_ab.json
+//
+// RUN: clang-scan-deps -compilation-database %t/cdb_a.json -format experimental-full -optimize-args >> %t/result_a.json
+// RUN: cat %t/result_a.json | sed 's/\\/\//g' | FileCheck --check-prefixes=CHECK_A %s
+//
+// RUN: clang-scan-deps -compilation-database %t/cdb_b.json -format experimental-full -optimize-args >> %t/result_b.json
+// RUN: cat %t/result_b.json | sed 's/\\/\//g' | FileCheck --check-prefixes=CHECK_B %s
+//
+// RUN: clang-scan-deps -compilation-database %t/cdb_ab.json -format experimental-full -optimize-args >> %t/result_ab.json
+// RUN: cat %t/result_ab.json | sed 's/\\/\//g' | FileCheck --check-prefixes=CHECK_AB %s
+
+#include "mod.h"
+
+// CHECK_A:        {
+// CHECK_A-NEXT:     "modules": [
+// CHECK_A-NEXT:       {
+// CHECK_A-NEXT:         "clang-module-deps": [],
+// CHECK_A-NEXT:         "clang-modulemap-file": "{{.*}}",
+// CHECK_A-NEXT:         "command-line": [
+// CHECK_A-NEXT:           "-cc1"
+// CHECK_A:                "-I",
+// CHECK_A-NEXT:           "begin",
+// CHECK_A-NEXT:           "-I",
+// CHECK_A-NEXT:           "a",
+// CHECK_A-NEXT:           "-I",
+// CHECK_A-NEXT:           "end"
+// CHECK_A:              ],
+// CHECK_A-NEXT:         "context-hash": "{{.*}}",
+// CHECK_A-NEXT:         "file-deps": [
+// CHECK_A:              ],
+// CHECK_A-NEXT:         "name": "mod"
+// CHECK_A-NEXT:       }
+// CHECK_A-NEXT:     ]
+// CHECK_A:        }
+
+// CHECK_B:        {
+// CHECK_B-NEXT:     "modules": [
+// CHECK_B-NEXT:       {
+// CHECK_B-NEXT:         "clang-module-deps": [],
+// CHECK_B-NEXT:         "clang-modulemap-file": "{{.*}}",
+// CHECK_B-NEXT:         "command-line": [
+// CHECK_B-NEXT:           "-cc1"
+// CHECK_B:                "-I",
+// CHECK_B-NEXT:           "begin",
+// CHECK_B-NEXT:           "-I",
+// CHECK_B-NEXT:           "b",
+// CHECK_B-NEXT:           "-I",
+// CHECK_B-NEXT:           "end"
+// CHECK_B:              ],
+// CHECK_B-NEXT:         "context-hash": "{{.*}}",
+// CHECK_B-NEXT:         "file-deps": [
+// CHECK_B:              ],
+// CHECK_B-NEXT:         "name": "mod"
+// CHECK_B-NEXT:       }
+// CHECK_B-NEXT:     ]
+// CHECK_B:        }
+
+// CHECK_AB:       {
+// CHECK_AB-NEXT:    "modules": [
+// CHECK_AB-NEXT:      {
+// CHECK_AB-NEXT:        "clang-module-deps": [],
+// CHECK_AB-NEXT:        "clang-modulemap-file": "{{.*}}",
+// CHECK_AB-NEXT:        "command-line": [
+// CHECK_AB-NEXT:          "-cc1"
+// CHECK_AB:               "-I",
+// CHECK_AB-NEXT:          "begin",
+// CHECK_AB-NEXT:          "-I",
+// CHECK_AB-NEXT:          "a",
+// CHECK_AB-NEXT:          "-I",
+// CHECK_AB-NEXT:          "b",
+// CHECK_AB-NEXT:          "-I",
+// CHECK_AB-NEXT:          "end"
+// CHECK_AB:             ],
+// CHECK_AB-NEXT:        "context-hash": "{{.*}}",
+// CHECK_AB-NEXT:        "file-deps": [
+// CHECK_AB:             ],
+// CHECK_AB-NEXT:        "name": "mod"
+// CHECK_AB-NEXT:      }
+// CHECK_AB-NEXT:    ]
+// CHECK_AB:       }

clang/tools/clang-scan-deps/ClangScanDeps.cpp

 
 static llvm::cl::opt<std::string> ModuleFilesDir(
     "module-files-dir",
     llvm::cl::desc("With '-generate-modules-path-args', paths to module files "
                    "in the generated command lines will begin with the "
                    "specified directory instead the module cache directory."),
     llvm::cl::cat(DependencyScannerCategory));
 
+static llvm::cl::opt<bool> OptimizeArgs(
+    "optimize-args",
+    llvm::cl::desc("Whether to optimize command-line arguments of modules."),
+    llvm::cl::init(false), llvm::cl::cat(DependencyScannerCategory));
+
 llvm::cl::opt<unsigned>
     NumThreads("j", llvm::cl::Optional,
                llvm::cl::desc("Number of worker threads to use (default: use "
                               "all concurrent threads)"),
                llvm::cl::init(0), llvm::cl::cat(DependencyScannerCategory));
 
 llvm::cl::opt<std::string>
     CompilationDB("compilation-database",
StringRef lookupPCMPath(ModuleID MID) {
     if (PCMPath.second)
       PCMPath.first->second = constructPCMPath(lookupModuleDeps(MID));
     return PCMPath.first->second;
   }
 
   /// Construct a path for the explicitly built PCM.
   std::string constructPCMPath(const ModuleDeps &MD) const {
     StringRef Filename = llvm::sys::path::filename(MD.ImplicitModulePCMPath);
 
     SmallString<256> ExplicitPCMPath(
         !ModuleFilesDir.empty()
             ? ModuleFilesDir
             : MD.Invocation.getHeaderSearchOpts().ModuleCachePath);
     llvm::sys::path::append(ExplicitPCMPath, MD.ID.ContextHash, Filename);
     return std::string(ExplicitPCMPath);
   }
 
AdjustingCompilations->appendArgumentsAdjuster(
         return AdjustedArgs;
       });
 
   SharedStream Errs(llvm::errs());
   // Print out the dependency results to STDOUT by default.
   SharedStream DependencyOS(llvm::outs());
 
   DependencyScanningService Service(ScanMode, Format, ReuseFileManager,
-                                    SkipExcludedPPRanges);
+                                    SkipExcludedPPRanges, OptimizeArgs);
   llvm::ThreadPool Pool(llvm::hardware_concurrency(NumThreads));
   std::vector<std::unique_ptr<DependencyScanningTool>> WorkerTools;
   for (unsigned I = 0; I < Pool.getThreadCount(); ++I)
     WorkerTools.push_back(std::make_unique<DependencyScanningTool>(Service));
 
   std::vector<tooling::CompileCommand> Inputs =
       AdjustingCompilations->getAllCompileCommands();