[clang][deps] Support inferred modules

[jansvoboda11]

Summary

This patch adds support for inferred modules to the dependency scanner.

Effectively a cherry-pick of https://github.com/apple/llvm-project/pull/699 authored by @Bigcheese with libclang and other changes omitted.

Contains following changes:

1. [Clang][ScanDeps] Ignore __inferred_module.map dependency.
   • This shows up with inferred modules, but it doesn't exist on disk, so don't report it as a dependency.

2. [Clang][ScanDeps] Use the module map a module was inferred from for inferred modules.

Also includes a smoke test that uses clang-scan-deps output to perform an explicit build. There's no intention to duplicate whatever test/Modules contains, just to verify the produced command-line does "work" (with very loose definition of work).

Split from D100934.

[dexonsmith]

LGTM, thanks for splitting out and moving the tool to clang/utils.

[RKSimon]

clang/utils/module-deps-to-rsp.py

1

@Bigcheese - https://lab.llvm.org/buildbot/#/builders/139/builds/4147 doesn't like this - should this be python3 ?

[thakis]

Looks like this breaks tests on windows: http://45.33.8.238/win/38908/step_7.txt

[jansvoboda11]

clang/utils/module-deps-to-rsp.py

1

Thanks, that seems to do the trick: e827d74a97dcf6623470baf3bda86b1c9544e3df, https://lab.llvm.org/buildbot/#/builders/139/builds/4154.

[jansvoboda11]

In D102495#2763080, @thakis wrote:

Looks like this breaks tests on windows: http://45.33.8.238/win/38908/step_7.txt

Thanks for the heads-up. This might be hitting behavior introduced here: https://reviews.llvm.org/D102491#C2576775NL1045

Reverting this patch for now.

[thakis]

Did you see my message about the windows breakage?

[jansvoboda11]

In D102495#2763115, @thakis wrote:

Did you see my message about the windows breakage?

Yes, reverted in ccd485e55424b78c07c2f9520faf00b805f78c9b.

[jansvoboda11]

Re-landed this in c98833cdaad01787ea70ecdfabb05a7e142a6671 after reproducing and fixing the Windows failure.

[frgossen]

clang/test/ClangScanDeps/modules-inferred-explicit-build.m

13–15

These three commands rely on file creations in the current working directory.

[jansvoboda11]

clang/test/ClangScanDeps/modules-inferred-explicit-build.m

13–15

Thanks for the heads-up and for the revert. Fixed in 7defab082070.

clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp

makeInvocationForModuleBuildWithoutPaths(const ModuleDeps &Deps,
   CI.getFrontendOpts().Inputs.clear();
   CI.getFrontendOpts().OutputFile.clear();
 
   CI.getFrontendOpts().ProgramAction = frontend::GenerateModule;
   CI.getLangOpts()->ModuleName = Deps.ID.ModuleName;
   CI.getFrontendOpts().IsSystemModule = Deps.IsSystem;
 
   CI.getLangOpts()->ImplicitModules = false;
-  CI.getHeaderSearchOpts().ImplicitModuleMaps = false;
 
   return CI;
 }
 
 static std::vector<std::string>
 serializeCompilerInvocation(const CompilerInvocation &CI) {
   // Set up string allocator.
   llvm::BumpPtrAllocator Alloc;
ModuleID ModuleDepCollectorPP::handleTopLevelModule(const Module *M) {
   MD.ID.ModuleName = M->getFullModuleName();
   MD.ImportedByMainFile = DirectModularDeps.contains(M);
   MD.ImplicitModulePCMPath = std::string(M->getASTFile()->getName());
   MD.IsSystem = M->IsSystem;
 
   const FileEntry *ModuleMap = Instance.getPreprocessor()
                                    .getHeaderSearchInfo()
                                    .getModuleMap()
-                                   .getContainingModuleMapFile(M);
+                                   .getModuleMapFileForUniquing(M);
   MD.ClangModuleMapFile = std::string(ModuleMap ? ModuleMap->getName() : "");
 
   serialization::ModuleFile *MF =
       MDC.Instance.getASTReader()->getModuleManager().lookup(M->getASTFile());
   MDC.Instance.getASTReader()->visitInputFiles(
       *MF, true, true, [&](const serialization::InputFile &IF, bool isSystem) {
+        // __inferred_module.map is the result of the way in which an implicit
+        // module build handles inferred modules. It adds an overlay VFS with
+        // this file in the proper directory and relies on the rest of Clang to
+        // handle it like normal. With explicitly built modules we don't need
+        // to play VFS tricks, so replace it with the correct module map.
+        if (IF.getFile()->getName().endswith("__inferred_module.map")) {
+          MD.FileDeps.insert(ModuleMap->getName());
+          return;
+        }
         MD.FileDeps.insert(IF.getFile()->getName());
       });
 
   MD.Invocation =
       makeInvocationForModuleBuildWithoutPaths(MD, Instance.getInvocation());
   MD.ID.ContextHash = MD.Invocation.getModuleHash();
 
   llvm::DenseSet<const Module *> AddedModules;

clang/test/ClangScanDeps/Inputs/frameworks/Inferred.framework/Headers/Inferred.h

+typedef int inferred;

clang/test/ClangScanDeps/Inputs/frameworks/System.framework/Headers/System.h

+enum { bigger_than_int = 0x80000000 };

clang/test/ClangScanDeps/Inputs/frameworks/System.framework/Modules/module.modulemap

+framework module System [system] {
+  umbrella header "System.h"
+}

clang/test/ClangScanDeps/Inputs/frameworks/module.modulemap

+framework module * {}

clang/test/ClangScanDeps/Inputs/modules_inferred_cdb.json

+[
+{
+  "directory": "DIR",
+  "command": "clang -E DIR/modules_cdb_input.cpp -FFRAMEWORKS -fmodules -fmodules-cache-path=DIR/module-cache -fimplicit-modules -fimplicit-module-maps -pedantic -Werror",
+  "file": "DIR/modules_cdb_input.cpp"
+}
+]

clang/test/ClangScanDeps/modules-full.cpp

 // CHECK-NEXT:       "clang-modulemap-file": "[[PREFIX]]/Inputs/module.modulemap",
 // CHECK-NEXT:       "command-line": [
 // CHECK-NEXT:         "-cc1"
 // CHECK-NO-ABS-NOT:   "-fmodule-map-file={{.*}}"
 // CHECK-ABS:          "-fmodule-map-file=[[PREFIX]]/Inputs/module.modulemap"
 // CHECK:              "-emit-module"
 // CHECK-NO-ABS-NOT:   "-fmodule-file={{.*}}"
 // CHECK-ABS:          "-fmodule-file=[[PREFIX]]/module-cache{{(_clangcl)?}}/[[CONTEXT_HASH_H1]]/header2-{{[A-Z0-9]+}}.pcm"
-// CHECK-NOT:          "-fimplicit-module-maps"
 // CHECK:              "-fmodule-name=header1"
 // CHECK:              "-fno-implicit-modules"
 // CHECK:            ],
 // CHECK-NEXT:       "context-hash": "[[CONTEXT_HASH_H1]]",
 // CHECK-NEXT:       "file-deps": [
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/header.h",
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/module.modulemap"
 // CHECK-NEXT:       ],
 // CHECK-NEXT:       "name": "header1"
 // CHECK-NEXT:     },
 // CHECK-NEXT:     {
 // CHECK-NEXT:       "clang-module-deps": [],
 // CHECK-NEXT:       "clang-modulemap-file": "[[PREFIX]]/Inputs/module.modulemap",
 // CHECK-NEXT:       "command-line": [
 // CHECK-NEXT:         "-cc1",
 // CHECK:              "-emit-module",
-// CHECK-NOT:          "-fimplicit-module-maps",
 // CHECK:              "-fmodule-name=header1",
 // CHECK:              "-fno-implicit-modules",
 // CHECK:            ],
 // CHECK-NEXT:       "context-hash": "[[CONTEXT_HASH_H2:[A-Z0-9]+]]",
 // CHECK-NEXT:       "file-deps": [
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/header.h",
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/module.modulemap"
 // CHECK-NEXT:       ],
 // CHECK-NEXT:       "name": "header1"
 // CHECK-NEXT:     },
 // CHECK-NEXT:     {
 // CHECK-NEXT:       "clang-module-deps": [],
 // CHECK-NEXT:       "clang-modulemap-file": "[[PREFIX]]/Inputs/module.modulemap",
 // CHECK-NEXT:       "command-line": [
 // CHECK-NEXT:         "-cc1",
 // CHECK:              "-emit-module",
-// CHECK-NOT:          "-fimplicit-module-maps",
 // CHECK:              "-fmodule-name=header2",
 // CHECK:              "-fno-implicit-modules",
 // CHECK:            ],
 // CHECK-NEXT:       "context-hash": "[[CONTEXT_HASH_H1]]",
 // CHECK-NEXT:       "file-deps": [
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/header2.h",
 // CHECK-NEXT:         "[[PREFIX]]/Inputs/module.modulemap"
 // CHECK-NEXT:       ],

clang/test/ClangScanDeps/modules-inferred-explicit-build.m

+// RUN: rm -rf %t.dir
+// RUN: rm -rf %t.cdb
+// RUN: mkdir -p %t.dir
+// RUN: cp %s %t.dir/modules_cdb_input.cpp
+// RUN: sed -e "s|DIR|%/t.dir|g" -e "s|FRAMEWORKS|%/S/Inputs/frameworks|g" -e "s|-E|-x objective-c -E|g" \
+// RUN:   %S/Inputs/modules_inferred_cdb.json > %t.cdb
+//
+// RUN: clang-scan-deps -compilation-database %t.cdb -j 1 -format experimental-full \
+// RUN:   -mode preprocess-minimized-sources > %t.db
+// RUN: %S/../../utils/module-deps-to-rsp.py %t.db --module-name=Inferred > %t.inferred.cc1.rsp
+// RUN: %S/../../utils/module-deps-to-rsp.py %t.db --module-name=System > %t.system.cc1.rsp
+// RUN: %S/../../utils/module-deps-to-rsp.py %t.db --tu-index=0 > %t.tu.rsp
+// RUN: %clang @%t.inferred.cc1.rsp -pedantic -Werror
+// RUN: %clang @%t.system.cc1.rsp -pedantic -Werror
+// RUN: %clang -x objective-c -fsyntax-only %t.dir/modules_cdb_input.cpp \
+// RUN:   -F%S/Inputs/frameworks -fmodules -fimplicit-module-maps \
+// RUN:   -pedantic -Werror @%t.tu.rsp
+
+#include <Inferred/Inferred.h>
+#include <System/System.h>
+
+inferred a = bigger_than_int;

clang/test/ClangScanDeps/modules-inferred.m

+// RUN: rm -rf %t.dir
+// RUN: rm -rf %t.cdb
+// RUN: mkdir -p %t.dir
+// RUN: cp %s %t.dir/modules_cdb_input.cpp
+// RUN: sed -e "s|DIR|%/t.dir|g" -e "s|FRAMEWORKS|%S/Inputs/frameworks|g" \
+// RUN:   %S/Inputs/modules_inferred_cdb.json > %t.cdb
+//
+// RUN: echo -%t.dir > %t.result
+// RUN: echo -%S >> %t.result
+// RUN: clang-scan-deps -compilation-database %t.cdb -j 1 -format experimental-full \
+// RUN:   -generate-modules-path-args -mode preprocess-minimized-sources >> %t.result
+// RUN: cat %t.result | sed 's/\\/\//g' | FileCheck --check-prefixes=CHECK %s
+
+#include <Inferred/Inferred.h>
+
+inferred a = 0;
+
+// CHECK: -[[PREFIX:.*]]
+// CHECK-NEXT: -[[SOURCEDIR:.*]]
+// CHECK-NEXT: {
+// CHECK-NEXT:   "modules": [
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "clang-module-deps": [],
+// CHECK-NEXT:       "clang-modulemap-file": "[[SOURCEDIR]]/Inputs/frameworks/module.modulemap",
+// CHECK-NEXT:       "command-line": [
+// CHECK-NEXT:         "-cc1",
+// CHECK:              "-emit-module",
+// CHECK:              "-fmodule-name=Inferred",
+// CHECK:              "-fno-implicit-modules",
+// CHECK:            ],
+// CHECK-NEXT:       "context-hash": "[[CONTEXT_HASH_H1:[A-Z0-9]+]]",
+// CHECK-NEXT:       "file-deps": [
+// CHECK-NEXT:         "[[SOURCEDIR]]/Inputs/frameworks/Inferred.framework/Frameworks/Sub.framework/Headers/Sub.h",
+// CHECK-NEXT:         "[[SOURCEDIR]]/Inputs/frameworks/Inferred.framework/Headers/Inferred.h",
+// CHECK-NEXT:         "[[SOURCEDIR]]/Inputs/frameworks/module.modulemap"
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "name": "Inferred"
+// CHECK-NEXT:     }
+// CHECK-NEXT:   ],
+// CHECK-NEXT:   "translation-units": [
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "clang-context-hash": "[[CONTEXT_HASH_H1]]",
+// CHECK-NEXT:       "clang-module-deps": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "context-hash": "[[CONTEXT_HASH_H1]]",
+// CHECK-NEXT:           "module-name": "Inferred"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "command-line": [
+// CHECK-NEXT:         "-fno-implicit-modules",
+// CHECK-NEXT:         "-fno-implicit-module-maps",
+// CHECK-NEXT:         "-fmodule-file=[[PREFIX]]/module-cache/[[CONTEXT_HASH_H1]]/Inferred-{{[A-Z0-9]+}}.pcm",
+// CHECK-NEXT:         "-fmodule-map-file=[[SOURCEDIR]]/Inputs/frameworks/module.modulemap"
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "file-deps": [
+// CHECK-NEXT:         "[[PREFIX]]/modules_cdb_input.cpp"
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "input-file": "[[PREFIX]]/modules_cdb_input.cpp"
+// CHECK-NEXT:     }
+// CHECK-NEXT:   ]
+// CHECK-NEXT: }

clang/utils/module-deps-to-rsp.py

+#!/usr/bin/env python
+
+# Converts clang-scan-deps output into response files.
+#   * For modules, arguments in the resulting response file are enough to build a PCM.
+#   * For translation units, the response file needs to be added to the original Clang invocation from compilation
+#     database.
+#
+# Usage:
+#
+#   clang-scan-deps -compilation-database compile_commands.json ... > deps.json
+#   module-deps-to-rsp.py deps.json --module-name=ModuleName > module_name.cc1.rsp
+#   module-deps-to-rsp.py deps.json --tu-index=0 > tu.rsp
+#   clang @module_name.cc1.rsp
+#   clang ... @tu.rsp
+
+import argparse
+import json
+import sys
+
+class ModuleNotFoundError(Exception):
+  def __init__(self, module_name):
+    self.module_name = module_name
+
+class FullDeps:
+  def __init__(self):
+    self.modules = dict()
+    self.translation_units = str()
+    
+def getModulePathArgs(modules, full_deps):
+  cmd = []
+  for md in modules:
+    m = full_deps.modules[md['module-name'] + '-' + md['context-hash']]
+    cmd += [u'-fmodule-map-file=' + m['clang-modulemap-file']]
+    cmd += [u'-fmodule-file=' + md['module-name'] + '-' + md['context-hash'] + '.pcm']
+  return cmd
+
+def getCommandLineForModule(module_name, full_deps):
+  for m in full_deps.modules.values():
+    if m['name'] == module_name:
+      module = m
+      break
+  else:
+    raise ModuleNotFoundError(module_name)
+
+  cmd = m['command-line']
+  cmd += getModulePathArgs(m['clang-module-deps'], full_deps)
+  cmd += [u'-o', m['name'] + '-' + m['context-hash'] + '.pcm']
+  cmd += [m['clang-modulemap-file']]
+  
+  return cmd
+  
+def getCommandLineForTU(tu, full_deps):
+  cmd = tu['command-line']
+  cmd += getModulePathArgs(tu['clang-module-deps'], full_deps)
+  return cmd
+
+def parseFullDeps(json):
+  ret = FullDeps()
+  for m in json['modules']:
+    ret.modules[m['name'] + '-' + m['context-hash']] = m
+  ret.translation_units = json['translation-units']
+  return ret
+
+def main():
+  parser = argparse.ArgumentParser()
+  parser.add_argument("full_deps_file", help="Path to the full dependencies json file",
+                      type=str)
+  action = parser.add_mutually_exclusive_group(required=True)
+  action.add_argument("--module-name", help="The name of the module to get arguments for",
+                      type=str)
+  action.add_argument("--tu-index", help="The index of the translation unit to get arguments for",
+                      type=int)
+  args = parser.parse_args()
+  
+  full_deps = parseFullDeps(json.load(open(args.full_deps_file, 'r')))
+  
+  try:
+    if args.module_name:
+      print(" ".join(getCommandLineForModule(args.module_name, full_deps)))
+    
+    elif args.tu_index != None:
+      print(" ".join(getCommandLineForTU(full_deps.translation_units[args.tu_index], full_deps)))
+  except:
+    print("Unexpected error:", sys.exc_info()[0])
+    raise
+
+if __name__ == '__main__':
+  main()