Modules: add command line option to support loading prebuilt modules on demand, without parsing any module map

[manmanren]

Summary

We add -fmodules-use-prebuilt-modules to support using prebuilt modules. In this mode, there is no need to load any module map and the programmer can simply use "@import" syntax to load the module directly from module cache without parsing any module map. We don't support rebuilding of the modules and ignore configuration mismatches.

[benlangmuir]

How about -fmodules-use-prebuilt-module-cache for the flag name? Saying "prebuilt-modules" is confusing to me, since -fmodule-file can also be used to load a prebuilt module, but doesn't use a cache.

lib/Driver/Tools.cpp

5416 ↗

(On Diff #66677)

if (IsPrebuilt) {

lib/Frontend/CompilerInstance.cpp

1491 ↗

(On Diff #66677)

Why do we assume this?

[manmanren]

lib/Driver/Tools.cpp

5416 ↗

(On Diff #66677)

Will do.

lib/Frontend/CompilerInstance.cpp

1491 ↗

(On Diff #66677)

Thanks for reviewing so quickly!

I was not sure about what value we should set IsSystem. Since we are mostly prebuilding system modules, I assumed "true". Do you have any suggestion here?

[benlangmuir]

Sorry for the delay, I apparently forgot to hit submit. Replied inline.

lib/Frontend/CompilerInstance.cpp

1491 ↗

(On Diff #66677)

I guess treating it as system is the more forgiving option, so I'm okay with keeping it. It's probably worth adding a comment that this is not always the correct choice, since it can affect diagnostics, etc.

[manmanren]

Add comments for setting IsSystem to true.

[benlangmuir]

Still missing the change for if (IsPrebuilt) {, but otherwise LGTM.

[manmanren]

In D23125#504942, @benlangmuir wrote:

How about -fmodules-use-prebuilt-module-cache for the flag name? Saying "prebuilt-modules" is confusing to me, since -fmodule-file can also be used to load a prebuilt module, but doesn't use a cache.

Oops, forgot to address your other comments.
Yes, I will change the flag name as you suggested and use if (IsPrebuilt).

Cheers,
Manman

[rsmith]

This doesn't seem like quite the right user interface for this feature. The module cache is volatile, and it's not really reasonable for people to assume they know what is and isn't within it. Instead, it seems like what we want to provide is zero or more paths to directories containing prebuilt .pcm files whose file names are the same as their top-level module names, completely separate from the module cache.

Specifically, rather than spelling this as -fmodules-use-prebuilt-modules -fmodules-cache-path=X, I'd suggest spelling it as -fprebuilt-module-path=X or similar (which can be repeated to provide multiple search paths). That would then combine orthogonally with -fno-implicit-modules and -fno-implicit-module-maps, where implicit modules would still use the module cache. Does that make sense for your use case?

You're also missing updates to the modules documentation.

[rsmith]

lib/Frontend/CompilerInstance.cpp

1483 ↗

(On Diff #67431)

This looks wrong: you're asking ReadAST to not diagnose a configuration mismatch, and you also don't diagnose it yourself.

1502–1510 ↗

(On Diff #67431)

You requested that ReadAST produces diagnostics for the OutOfDate and Missing cases when using prebuilt modules, so this diagnostic is redundant.

lib/Serialization/ASTReader.cpp

495–498 ↗

(On Diff #67431)

This seems unnecessarily complicated. Perhaps you could make CompilerInstance::loadModule treat all prebuilt modules as MK_ExplicitModule, or add a third ModuleKind for them?

[manmanren]

In D23125#510632, @rsmith wrote:

This doesn't seem like quite the right user interface for this feature. The module cache is volatile, and it's not really reasonable for people to assume they know what is and isn't within it. Instead, it seems like what we want to provide is zero or more paths to directories containing prebuilt .pcm files whose file names are the same as their top-level module names, completely separate from the module cache.

Specifically, rather than spelling this as -fmodules-use-prebuilt-modules -fmodules-cache-path=X, I'd suggest spelling it as -fprebuilt-module-path=X or similar (which can be repeated to provide multiple search paths). That would then combine orthogonally with -fno-implicit-modules and -fno-implicit-module-maps, where implicit modules would still use the module cache. Does that make sense for your use case?

What we want is to achieve performance by prebuilding the modules and using @import to load the corresponding pcm without parsing the module maps. In ASTReader, we want to treat these modules in the same way as the explicit modules.

-fprebuilt-module-path sounds reasonable. Let me see how the implementation goes ...

You're also missing updates to the modules documentation.

Yes, I will update the document.

Cheers,
Manman

lib/Frontend/CompilerInstance.cpp

1502–1510 ↗

(On Diff #67431)

Yes, I can remove this diagnostic.

lib/Serialization/ASTReader.cpp

495–498 ↗

(On Diff #67431)

I tried adding a ModuleKind at some point, but somehow it didn't work out. I will take another look at that.

[manmanren]

Addressing Richard's comments.

[rsmith]

include/clang/Lex/HeaderSearchOptions.h

96–97 ↗

(On Diff #67621)

It would seem preferable to allow multiple of these, to support mixing (for instance) a path to prebuilt system modules and a path to some per-project modules. I'd be fine handling that as a separate patch / discussion if you prefer.

lib/Frontend/CompilerInstance.cpp

1438–1450 ↗

(On Diff #67621)

It looks like the logic here is:

• if we found the module in a module map, then look for it in the module cache
• otherwise, if we have a prebuilt module path, then look for it there
• otherwise, error

I think that may be surprising; if I explicitly load a module map, perhaps via -fmodule-map-file=, and supply a prebuilt form of that module in the prebuilt modules path, I'd expect my prebuilt form to still be used.

Perhaps instead we could first search the prebuilt module path for an existing .pcm file, and if that fails and we have a Module then look for it in the module cache?

1497–1505 ↗

(On Diff #67621)

The module file will have provided a Module instance, if it is in fact a module file for the requested module. Instead of inventing a Module here, can you check that one exists and produce an error if not?

lib/Serialization/ASTReader.cpp

2273 ↗

(On Diff #67621)

You're allowing configuration mismatches that we don't consider to be "compatible configuration mismatches" here. That seems really scary -- I believe this allows using a C++ module from C and similar unreasonable things, that will cause clang to crash. Did you really mean to do that?

[manmanren]

Thanks,
Manman

include/clang/Lex/HeaderSearchOptions.h

96–97 ↗

(On Diff #67621)

It sounds reasonable to provide prebuilt system modules and prebuilt project modules. I assume via the same command-line option -fprebuilt-module-path.

lib/Frontend/CompilerInstance.cpp

1438–1450 ↗

(On Diff #67621)

I was wondering about the priority of loading from prebuilt module path vs. loading from module cache. Your suggestion makes sense.

1497–1505 ↗

(On Diff #67621)

This is nice to know. I didn't realize we construct a Module when calling ReadAST. Can you be more specific on how to check if a Module exists?

lib/Serialization/ASTReader.cpp

2273 ↗

(On Diff #67621)

I originally had this in my testing patch to be really relaxing on prebuilt modules. I will remove this :]

[bruno]

lib/Frontend/CompilerInstance.cpp

1438–1450 ↗

(On Diff #67621)

+1 to Richard's suggestion here.

I wonder about the workflow on generating such modules. Suppose I want to build some modules to re-use later with -fprebuilt-module-path=. RIght now, we get a module cache with hash in the names unless we disable hash computation via cc1 option -fdisable-module-hash. Should we assume that the user will have to copy out the cache dir and rename the modules OR that it will use -fdisable-module-hash instead? Should this option also try to read out modules with a hash in the name? Should we promote -fdisable-module-hash to a driver option as well?

@Manman, do you have any workflow in mind? @Richard, do you have any feedback in this respect?

In the crash reproducer we often have to rebuild the modules when running the script in a different machine because of the paths used to hash the modules, it would be nice (depending on the resulting design) if the crash reproducer could use -fprebuilt-module-path in order to try to re-use the pre-built modules resulting from the crash.

[manmanren]

lib/Frontend/CompilerInstance.cpp

1438–1450 ↗

(On Diff #67621)

If the modules are built implicitly with module hash, to load the module files as prebuilt modules, the user needs to copy out the cache dir and rename the modules. The user can choose to emit the modules explicitly or disable module hash when building the modules.

1497–1505 ↗

(On Diff #67621)

I assumed ReadAST only constructed Module instances for submodules, because it is inside ReadSubmoduleBlock. It turns out it also constructed Module instance for the top level module. I will update the patch.

[manmanren]

Addressing Richard's comments.

[manmanren]

Richard,

Are you okay with the patch now?

Thanks,
Manman

[rsmith]

Just a couple of things, but this basically LGTM.

docs/Modules.rst

217 ↗

(On Diff #67727)

Please document that the option can be given multiple times.

lib/Frontend/CompilerInstance.cpp

1502 ↗

(On Diff #67727)

This should have a real diagnostic rather than an assertion. This would happen if the pcm file in the prebuilt module path had the wrong filename, for instance.

It'd also be good to check that Module->ASTFile actually refers to the file that we found in the prebuilt module path, and not (say) to one of its transitive dependencies that ReadAST also loaded.

[manmanren]

Thanks,

Manman

lib/Frontend/CompilerInstance.cpp

1502 ↗

(On Diff #68442)

Updated the patch to use error diagnostics instead of assertion.

<<<
It'd also be good to check that Module->ASTFile actually refers to the file that we found in the prebuilt module path, and not (say) to one of its transitive dependencies that ReadAST also loaded.

Did we already check this when creating the module in ASTReader?

if (!ParentModule) {
  if (const FileEntry *CurFile = CurrentModule->getASTFile()) {
    if (CurFile != F.File) {
      ...
    }
  }

  CurrentModule->setASTFile(F.File);
}

[rsmith]

lib/Frontend/CompilerInstance.cpp

1502 ↗

(On Diff #68442)

The AST reader has no idea which module we're trying to load a module file for, so it can't check this. The check you quoted is ensuring that we don't have two different module files providing the same module.

A testcase for this would be:

module A is in prebuilt-module-cache/x.pcm
module B is in prebuilt-module-cache/A.pcm and depends on x.pcm

When trying to load module A, ReadAST will load A.pcm, and we'll find that we now have a definition for module A, but we still didn't get the module from the expected file.

[manmanren]

Cheers,

Manman

lib/Frontend/CompilerInstance.cpp

1502 ↗

(On Diff #68456)

Thanks for the explanation, it makes sense.

Updated the patch accordingly.

[rsmith]

lib/Frontend/CompilerInstance.cpp

1503 ↗

(On Diff #68456)

It'd be safer to check that FileMgr.getFile(ModuleFileName) == Module->getASTFile() in case the filename gets canonicalized by the file manager in some way.

[manmanren]

lib/Frontend/CompilerInstance.cpp

1503 ↗

(On Diff #68456)

Yes you are right. I will update the patch and commit the change.

Thanks for all the useful information!

cfe/trunk/docs/Modules.rst

``-fmodules-search-all``
   If a symbol is not found, search modules referenced in the current module maps but not imported for symbols, so the error message can reference the module by name.  Note that if the global module index has not been built before, this might take some time as it needs to build all the modules.  Note that this option doesn't apply in module builds, to avoid the recursion.
 
 ``-fno-implicit-modules``
   All modules used by the build must be specified with ``-fmodule-file``.
 
 ``-fmodule-file=<file>``
   Load the given precompiled module file.
 
+``-fprebuilt-module-path=<directory>``
+  Specify the path to the prebuilt modules. If specified, we will look for modules in this directory for a given top-level module name. We don't need a module map for loading prebuilt modules in this directory and the compiler will not try to rebuild these modules. This can be specified multiple times.
+
 Module Semantics
 ================
 
 Modules are modeled as if each submodule were a separate translation unit, and a module import makes names from the other translation unit visible. Each submodule starts with a new preprocessor state and an empty translation unit.
 
 .. note::
 
   This behavior is currently only approximated when building a module with submodules. Entities within a submodule that has already been built are visible when building later submodules in that module. This can lead to fragile modules that depend on the build order used for the submodules of the module, and should not be relied upon. This behavior is subject to change.

cfe/trunk/include/clang/Basic/DiagnosticCommonKinds.td

 def err_module_header_missing : Error<
   "%select{|umbrella }0header '%1' not found">;
 def err_module_lock_failure : Error<
   "could not acquire lock file for module '%0': %1">, DefaultFatal;
 def err_module_lock_timeout : Error<
   "timed out waiting to acquire lock file for module '%0'">, DefaultFatal;
 def err_module_cycle : Error<"cyclic dependency in module '%0': %1">, 
   DefaultFatal;
+def err_module_prebuilt : Error<
+  "error in loading module '%0' from prebuilt module path">, DefaultFatal;
 def note_pragma_entered_here : Note<"#pragma entered here">;  
 def note_decl_hiding_tag_type : Note<
   "%1 %0 is hidden by a non-type declaration of %0 here">;
 def err_attribute_not_type_attr : Error<
   "%0 attribute cannot be applied to types">;
 def err_enum_template : Error<"enumeration cannot be a template">;
 
 }

cfe/trunk/include/clang/Driver/Options.td

HelpText<"Parse templated function definitions at the end of the "
            "translation unit">,  Flags<[CC1Option]>;
 def fms_memptr_rep_EQ : Joined<["-"], "fms-memptr-rep=">, Group<f_Group>, Flags<[CC1Option]>;
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, Group<i_Group>,
   Flags<[DriverOption, CC1Option]>, MetaVarName<"<directory>">,
   HelpText<"Specify the module cache path">;
 def fmodules_user_build_path : Separate<["-"], "fmodules-user-build-path">, Group<i_Group>,
   Flags<[DriverOption, CC1Option]>, MetaVarName<"<directory>">,
   HelpText<"Specify the module user build path">;
+def fprebuilt_module_path : Joined<["-"], "fprebuilt-module-path=">, Group<i_Group>,
+  Flags<[DriverOption, CC1Option]>, MetaVarName<"<directory>">,
+  HelpText<"Specify the prebuilt module path">;
 def fmodules_prune_interval : Joined<["-"], "fmodules-prune-interval=">, Group<i_Group>,
   Flags<[CC1Option]>, MetaVarName<"<seconds>">,
   HelpText<"Specify the interval (in seconds) between attempts to prune the module cache">;
 def fmodules_prune_after : Joined<["-"], "fmodules-prune-after=">, Group<i_Group>,
   Flags<[CC1Option]>, MetaVarName<"<seconds>">,
   HelpText<"Specify the interval (in seconds) after which a module file will be considered unused">;
 def fmodules_search_all : Flag <["-"], "fmodules-search-all">, Group<f_Group>,
   Flags<[DriverOption, CC1Option]>,

cfe/trunk/include/clang/Lex/HeaderSearch.h

public:
   /// \brief Retrieve the name of the module file that should be used to 
   /// load a module with the given name.
   ///
   /// \param ModuleName The module whose module file name will be returned.
   ///
   /// \param ModuleMapPath A path that when combined with \c ModuleName
   /// uniquely identifies this module. See Module::ModuleMap.
   ///
+  /// \param UsePrebuiltPath Whether we should use the prebuilt module path.
+  ///
   /// \returns The name of the module file that corresponds to this module,
   /// or an empty string if this module does not correspond to any module file.
-  std::string getModuleFileName(StringRef ModuleName, StringRef ModuleMapPath);
+  std::string getModuleFileName(StringRef ModuleName, StringRef ModuleMapPath,
+                                bool UsePrebuiltPath);
 
   /// \brief Lookup a module Search for a module with the given name.
   ///
   /// \param ModuleName The name of the module we're looking for.
   ///
   /// \param AllowSearch Whether we are allowed to search in the various
   /// search directories to produce a module definition. If not, this lookup
   /// will only return an already-known module.

cfe/trunk/include/clang/Lex/HeaderSearchOptions.h

public:
   std::string ResourceDir;
 
   /// \brief The directory used for the module cache.
   std::string ModuleCachePath;
 
   /// \brief The directory used for a user build.
   std::string ModuleUserBuildPath;
 
+  /// \brief The directories used to load prebuilt module files.
+  std::vector<std::string> PrebuiltModulePaths;
+
   /// The module/pch container format.
   std::string ModuleFormat;
 
   /// \brief Whether we should disable the use of the hash string within the
   /// module cache.
   ///
   /// Note: Only used for testing!
   unsigned DisableModuleHash : 1;
public:
   /// header.
   void AddSystemHeaderPrefix(StringRef Prefix, bool IsSystemHeader) {
     SystemHeaderPrefixes.emplace_back(Prefix, IsSystemHeader);
   }
 
   void AddVFSOverlayFile(StringRef Name) {
     VFSOverlayFiles.push_back(Name);
   }
+
+  void AddPrebuiltModulePath(StringRef Name) {
+    PrebuiltModulePaths.push_back(Name);
+  }
 };
 
 } // end namespace clang
 
 #endif

cfe/trunk/include/clang/Serialization/Module.h

 }
 
 /// \brief Specifies the kind of module that has been loaded.
 enum ModuleKind {
   MK_ImplicitModule, ///< File is an implicitly-loaded module.
   MK_ExplicitModule, ///< File is an explicitly-loaded module.
   MK_PCH,            ///< File is a PCH file treated as such.
   MK_Preamble,       ///< File is a PCH file treated as the preamble.
-  MK_MainFile        ///< File is a PCH file treated as the actual main file.
+  MK_MainFile,       ///< File is a PCH file treated as the actual main file.
+  MK_PrebuiltModule  ///< File is from a prebuilt module path.
 };
 
 /// \brief The input file that has been loaded from this AST file, along with
 /// bools indicating whether this was an overridden buffer or if it was
 /// out-of-date or not-found.
 class InputFile {
   enum {
     Overridden = 1,
public:
   llvm::SetVector<ModuleFile *> Imports;
 
   /// \brief Determine whether this module was directly imported at
   /// any point during translation.
   bool isDirectlyImported() const { return DirectlyImported; }
 
   /// \brief Is this a module file for a module (rather than a PCH or similar).
   bool isModule() const {
-    return Kind == MK_ImplicitModule || Kind == MK_ExplicitModule;
+    return Kind == MK_ImplicitModule || Kind == MK_ExplicitModule ||
+           Kind == MK_PrebuiltModule;
   }
 
   /// \brief Dump debugging output for this module.
   void dump();
 };
 
 } // end namespace serialization
 
 } // end namespace clang
 
 #endif

cfe/trunk/lib/Driver/Tools.cpp

if (C.isForDiagnostics()) {
       appendUserToPath(Path);
       llvm::sys::path::append(Path, "ModuleCache");
     }
     const char Arg[] = "-fmodules-cache-path=";
     Path.insert(Path.begin(), Arg, Arg + strlen(Arg));
     CmdArgs.push_back(Args.MakeArgString(Path));
   }
 
+  if (HaveModules) {
+    // -fprebuilt-module-path specifies where to load the prebuilt module files.
+    for (const Arg *A : Args.filtered(options::OPT_fprebuilt_module_path))
+      CmdArgs.push_back(Args.MakeArgString(
+          std::string("-fprebuilt-module-path=") + A->getValue()));
+  }
+      
   // -fmodule-name specifies the module that is currently being built (or
   // used for header checking by -fmodule-maps).
   Args.AddLastArg(CmdArgs, options::OPT_fmodule_name_EQ);
 
   // -fmodule-map-file can be used to specify files containing module
   // definitions.
   Args.AddAllArgs(CmdArgs, options::OPT_fmodule_map_file);
 

cfe/trunk/lib/Frontend/ASTUnit.cpp

const FileEntry *ASTUnit::getPCHFile() {
   if (!Reader)
     return nullptr;
 
   serialization::ModuleFile *Mod = nullptr;
   Reader->getModuleManager().visit([&Mod](serialization::ModuleFile &M) {
     switch (M.Kind) {
     case serialization::MK_ImplicitModule:
     case serialization::MK_ExplicitModule:
+    case serialization::MK_PrebuiltModule:
       return true; // skip dependencies.
     case serialization::MK_PCH:
       Mod = &M;
       return true; // found it.
     case serialization::MK_Preamble:
       return false; // look in dependencies.
     case serialization::MK_MainFile:
       return false; // look in dependencies.

cfe/trunk/lib/Frontend/CompilerInstance.cpp

if (Known != KnownModules.end()) {
     Module = Known->second;    
   } else if (ModuleName == getLangOpts().CurrentModule) {
     // This is the module we're building. 
     Module = PP->getHeaderSearchInfo().lookupModule(ModuleName);
     Known = KnownModules.insert(std::make_pair(Path[0].first, Module)).first;
   } else {
     // Search for a module with the given name.
     Module = PP->getHeaderSearchInfo().lookupModule(ModuleName);
-    if (!Module) {
+    HeaderSearchOptions &HSOpts =
+        PP->getHeaderSearchInfo().getHeaderSearchOpts();
+
+    std::string ModuleFileName;
+    bool LoadFromPrebuiltModulePath = false;
+    // We try to load the module from the prebuilt module paths. If not
+    // successful, we then try to find it in the module cache.
+    if (!HSOpts.PrebuiltModulePaths.empty()) {
+      // Load the module from the prebuilt module path.
+      ModuleFileName = PP->getHeaderSearchInfo().getModuleFileName(
+          ModuleName, "", /*UsePrebuiltPath*/ true);
+      if (!ModuleFileName.empty())
+        LoadFromPrebuiltModulePath = true;
+    }
+    if (!LoadFromPrebuiltModulePath && Module) {
+      // Load the module from the module cache.
+      ModuleFileName = PP->getHeaderSearchInfo().getModuleFileName(Module);
+    } else if (!LoadFromPrebuiltModulePath) {
+      // We can't find a module, error out here.
       getDiagnostics().Report(ModuleNameLoc, diag::err_module_not_found)
       << ModuleName
       << SourceRange(ImportLoc, ModuleNameLoc);
       ModuleBuildFailed = true;
       return ModuleLoadResult();
     }
 
-    std::string ModuleFileName =
-        PP->getHeaderSearchInfo().getModuleFileName(Module);
     if (ModuleFileName.empty()) {
-      if (Module->HasIncompatibleModuleFile) {
+      if (Module && Module->HasIncompatibleModuleFile) {
         // We tried and failed to load a module file for this module. Fall
         // back to textual inclusion for its headers.
         return ModuleLoadResult(nullptr, /*missingExpected*/true);
       }
 
       getDiagnostics().Report(ModuleNameLoc, diag::err_module_build_disabled)
           << ModuleName;
       ModuleBuildFailed = true;
       return ModuleLoadResult();
     }
 
     // If we don't already have an ASTReader, create one now.
     if (!ModuleManager)
       createModuleManager();
 
     llvm::Timer Timer;
     if (FrontendTimerGroup)
       Timer.init("Loading " + ModuleFileName, *FrontendTimerGroup);
     llvm::TimeRegion TimeLoading(FrontendTimerGroup ? &Timer : nullptr);
 
-    // Try to load the module file.
-    unsigned ARRFlags = ASTReader::ARR_OutOfDate | ASTReader::ARR_Missing;
+    // Try to load the module file. If we are trying to load from the prebuilt
+    // module path, we don't have the module map files and don't know how to
+    // rebuild modules.
+    unsigned ARRFlags = LoadFromPrebuiltModulePath ?
+                        ASTReader::ARR_ConfigurationMismatch :
+                        ASTReader::ARR_OutOfDate | ASTReader::ARR_Missing;
     switch (ModuleManager->ReadAST(ModuleFileName,
+                                   LoadFromPrebuiltModulePath ?
+                                   serialization::MK_PrebuiltModule :
                                    serialization::MK_ImplicitModule,
-                                   ImportLoc, ARRFlags)) {
-    case ASTReader::Success:
+                                   ImportLoc,
+                                   ARRFlags)) {
+    case ASTReader::Success: {
+      if (LoadFromPrebuiltModulePath && !Module) {
+        Module = PP->getHeaderSearchInfo().lookupModule(ModuleName);
+        if (!Module || !Module->getASTFile() ||
+            FileMgr->getFile(ModuleFileName) != Module->getASTFile()) {
+          // Error out if Module does not refer to the file in the prebuilt
+          // module path.
+          getDiagnostics().Report(ModuleNameLoc, diag::err_module_prebuilt)
+              << ModuleName;
+          ModuleBuildFailed = true;
+          KnownModules[Path[0].first] = nullptr;
+          return ModuleLoadResult();
+        }
+      }
       break;
+    }
 
     case ASTReader::OutOfDate:
     case ASTReader::Missing: {
+      if (LoadFromPrebuiltModulePath) {
+        // We can't rebuild the module without a module map. Since ReadAST
+        // already produces diagnostics for these two cases, we simply
+        // error out here.
+        ModuleBuildFailed = true;
+        KnownModules[Path[0].first] = nullptr;
+        return ModuleLoadResult();
+      }
+
       // The module file is missing or out-of-date. Build it.
       assert(Module && "missing module file");
       // Check whether there is a cycle in the module graph.
       ModuleBuildStack ModPath = getSourceManager().getModuleBuildStack();
       ModuleBuildStack::iterator Pos = ModPath.begin(), PosEnd = ModPath.end();
       for (; Pos != PosEnd; ++Pos) {
         if (Pos->first == ModuleName)
           break;
case ASTReader::Missing: {
         ModuleBuildFailed = true;
         return ModuleLoadResult();
       }
 
       // Okay, we've rebuilt and now loaded the module.
       break;
     }
 
-    case ASTReader::VersionMismatch:
     case ASTReader::ConfigurationMismatch:
+      if (LoadFromPrebuiltModulePath)
+        getDiagnostics().Report(SourceLocation(),
+                                diag::warn_module_config_mismatch)
+            << ModuleFileName;
+      // Fall through to error out.
+    case ASTReader::VersionMismatch:
     case ASTReader::HadErrors:
       ModuleLoader::HadFatalFailure = true;
       // FIXME: The ASTReader will already have complained, but can we shoehorn
       // that diagnostic information into a more useful form?
       KnownModules[Path[0].first] = nullptr;
       return ModuleLoadResult();
 
     case ASTReader::Failure:

cfe/trunk/lib/Frontend/CompilerInvocation.cpp

static void ParseHeaderSearchArgs(HeaderSearchOptions &Opts, ArgList &Args) {
   Opts.UseBuiltinIncludes = !Args.hasArg(OPT_nobuiltininc);
   Opts.UseStandardSystemIncludes = !Args.hasArg(OPT_nostdsysteminc);
   Opts.UseStandardCXXIncludes = !Args.hasArg(OPT_nostdincxx);
   if (const Arg *A = Args.getLastArg(OPT_stdlib_EQ))
     Opts.UseLibcxx = (strcmp(A->getValue(), "libc++") == 0);
   Opts.ResourceDir = Args.getLastArgValue(OPT_resource_dir);
   Opts.ModuleCachePath = Args.getLastArgValue(OPT_fmodules_cache_path);
   Opts.ModuleUserBuildPath = Args.getLastArgValue(OPT_fmodules_user_build_path);
+  for (const Arg *A : Args.filtered(OPT_fprebuilt_module_path))
+    Opts.AddPrebuiltModulePath(A->getValue());
   Opts.DisableModuleHash = Args.hasArg(OPT_fdisable_module_hash);
   Opts.ModulesValidateDiagnosticOptions =
       !Args.hasArg(OPT_fmodules_disable_diagnostic_validation);
   Opts.ImplicitModuleMaps = Args.hasArg(OPT_fimplicit_module_maps);
   Opts.ModuleMapFileHomeIsCwd = Args.hasArg(OPT_fmodule_map_file_home_is_cwd);
   Opts.ModuleCachePruneInterval =
       getLastArgIntValue(Args, OPT_fmodules_prune_interval, 7 * 24 * 60 * 60);
   Opts.ModuleCachePruneAfter =

cfe/trunk/lib/Frontend/FrontendActions.cpp

GenerateModuleAction::ComputeASTConsumerArguments(CompilerInstance &CI,
                                                   std::string &Sysroot,
                                                   std::string &OutputFile) {
   // If no output file was provided, figure out where this module would go
   // in the module cache.
   if (CI.getFrontendOpts().OutputFile.empty()) {
     HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
     CI.getFrontendOpts().OutputFile =
         HS.getModuleFileName(CI.getLangOpts().CurrentModule,
-                             ModuleMapForUniquing->getName());
+                             ModuleMapForUniquing->getName(),
+                             /*UsePrebuiltPath=*/false);
   }
 
   // We use createOutputFile here because this is exposed via libclang, and we
   // must disable the RemoveFileOnSignal behavior.
   // We use a temporary to avoid race conditions.
   std::unique_ptr<raw_pwrite_stream> OS =
       CI.createOutputFile(CI.getFrontendOpts().OutputFile, /*Binary=*/true,
                           /*RemoveFileOnSignal=*/false, InFile,

cfe/trunk/lib/Lex/HeaderSearch.cpp

const HeaderMap *HeaderSearch::CreateHeaderMap(const FileEntry *FE) {
   }
 
   return nullptr;
 }
 
 std::string HeaderSearch::getModuleFileName(Module *Module) {
   const FileEntry *ModuleMap =
       getModuleMap().getModuleMapFileForUniquing(Module);
-  return getModuleFileName(Module->Name, ModuleMap->getName());
+  return getModuleFileName(Module->Name, ModuleMap->getName(),
+                           /*UsePrebuiltPath*/false);
 }
 
 std::string HeaderSearch::getModuleFileName(StringRef ModuleName,
-                                            StringRef ModuleMapPath) {
+                                            StringRef ModuleMapPath,
+                                            bool UsePrebuiltPath) {
+  if (UsePrebuiltPath) {
+    if (HSOpts->PrebuiltModulePaths.empty())
+      return std::string();
+
+    // Go though each prebuilt module path and try to find the pcm file.
+    for (const std::string &Dir : HSOpts->PrebuiltModulePaths) {
+      SmallString<256> Result(Dir);
+      llvm::sys::fs::make_absolute(Result);
+
+      llvm::sys::path::append(Result, ModuleName + ".pcm");
+      if (getFileMgr().getFile(Result.str()))
+        return Result.str().str();
+    }
+    return std::string();
+  }
+
   // If we don't have a module cache path or aren't supposed to use one, we
   // can't do anything.
   if (getModuleCachePath().empty())
     return std::string();
 
   SmallString<256> Result(getModuleCachePath());
   llvm::sys::fs::make_absolute(Result);
 

cfe/trunk/lib/Serialization/ASTReader.cpp

bool ASTReader::ReadSLocEntry(int ID) {
   }
 
   case SM_SLOC_BUFFER_ENTRY: {
     const char *Name = Blob.data();
     unsigned Offset = Record[0];
     SrcMgr::CharacteristicKind
       FileCharacter = (SrcMgr::CharacteristicKind)Record[2];
     SourceLocation IncludeLoc = ReadSourceLocation(*F, Record[1]);
-    if (IncludeLoc.isInvalid() &&
-        (F->Kind == MK_ImplicitModule || F->Kind == MK_ExplicitModule)) {
+    if (IncludeLoc.isInvalid() && F->isModule()) {
       IncludeLoc = getImportLocation(F);
     }
 
     auto Buffer = ReadBuffer(SLocEntryCursor, Name);
     if (!Buffer)
       return true;
     SourceMgr.createFileID(std::move(Buffer), FileCharacter, ID,
                            BaseOffset + Offset, IncludeLoc);
std::pair<SourceLocation, StringRef> ASTReader::getModuleImportLoc(int ID) {
 
   if (unsigned(-ID) - 2 >= getTotalNumSLocs() || ID > 0) {
     Error("source location entry ID out-of-range for AST file");
     return std::make_pair(SourceLocation(), "");
   }
 
   // Find which module file this entry lands in.
   ModuleFile *M = GlobalSLocEntryMap.find(-ID)->second;
-  if (M->Kind != MK_ImplicitModule && M->Kind != MK_ExplicitModule)
+  if (!M->isModule())
     return std::make_pair(SourceLocation(), "");
 
   // FIXME: Can we map this down to a particular submodule? That would be
   // ideal.
   return std::make_pair(M->ImportLoc, StringRef(M->ModuleName));
 }
 
 /// \brief Find the location where the module F is imported.
for (auto &MMR : ModuleMacros) {
       bool Inserted = false;
       Module *Owner = getSubmodule(MMR.SubModID);
       PP.addModuleMacro(Owner, II, MMR.MI, Overrides, Inserted);
     }
   }
 
   // Don't read the directive history for a module; we don't have anywhere
   // to put it.
-  if (M.Kind == MK_ImplicitModule || M.Kind == MK_ExplicitModule)
+  if (M.isModule())
     return;
 
   // Deserialize the macro directives history in reverse source-order.
   MacroDirective *Latest = nullptr, *Earliest = nullptr;
   unsigned Idx = 0, N = Record.size();
   while (Idx < N) {
     MacroDirective *MD = nullptr;
     SourceLocation Loc = ReadSourceLocation(M, Record, Idx);
while (1) {
     case llvm::BitstreamEntry::EndBlock: {
       // Validate input files.
       const HeaderSearchOptions &HSOpts =
           PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
       // All user input files reside at the index range [0, NumUserInputs), and
       // system input files reside at [NumUserInputs, NumInputs). For explicitly
       // loaded module files, ignore missing inputs.
-      if (!DisableValidation && F.Kind != MK_ExplicitModule) {
+      if (!DisableValidation && F.Kind != MK_ExplicitModule &&
+          F.Kind != MK_PrebuiltModule) {
         bool Complain = (ClientLoadCapabilities & ARR_OutOfDate) == 0;
 
         // If we are reading a module, we will create a verification timestamp,
         // so we verify all input files.  Otherwise, verify only user input
         // files.
 
         unsigned N = NumUserInputs;
         if (ValidateSystemInputs ||
case llvm::BitstreamEntry::EndBlock: {
 
       if (Listener && Listener->needsInputFileVisitation()) {
         unsigned N = Listener->needsSystemInputFileVisitation() ? NumInputs
                                                                 : NumUserInputs;
         for (unsigned I = 0; I < N; ++I) {
           bool IsSystem = I >= NumUserInputs;
           InputFileInfo FI = readInputFileInfo(F, I+1);
           Listener->visitInputFile(FI.Filename, IsSystem, FI.Overridden,
-                                   F.Kind == MK_ExplicitModule);
+                                   F.Kind == MK_ExplicitModule ||
+                                   F.Kind == MK_PrebuiltModule);
         }
       }
 
       return Result;
     }
 
     case llvm::BitstreamEntry::SubBlock:
       switch (Entry.ID) {
case llvm::BitstreamEntry::SubBlock:
         // is required, because we checked them when we built it.
         if (Listener && !ImportedBy) {
           // Should we allow the configuration of the module file to differ from
           // the configuration of the current translation unit in a compatible
           // way?
           //
           // FIXME: Allow this for files explicitly specified with -include-pch.
           bool AllowCompatibleConfigurationMismatch =
-              F.Kind == MK_ExplicitModule;
+              F.Kind == MK_ExplicitModule || F.Kind == MK_PrebuiltModule;
           const HeaderSearchOptions &HSOpts =
               PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
           Result = ReadOptionsBlock(Stream, ClientLoadCapabilities,
                                     AllowCompatibleConfigurationMismatch,
                                     *Listener, SuggestedPredefines,
                                     HSOpts.ModulesValidateDiagnosticOptions);
           if (Result == Failure) {
case MODULE_DIRECTORY: {
       assert(!F.ModuleName.empty() &&
              "MODULE_DIRECTORY found before MODULE_NAME");
       // If we've already loaded a module map file covering this module, we may
       // have a better path for it (relative to the current build).
       Module *M = PP.getHeaderSearchInfo().lookupModule(F.ModuleName);
       if (M && M->Directory) {
         // If we're implicitly loading a module, the base directory can't
         // change between the build and use.
-        if (F.Kind != MK_ExplicitModule) {
+        if (F.Kind != MK_ExplicitModule && F.Kind != MK_PrebuiltModule) {
           const DirectoryEntry *BuildDir =
               PP.getFileManager().getDirectory(Blob);
           if (!BuildDir || BuildDir != M->Directory) {
             if ((ClientLoadCapabilities & ARR_OutOfDate) == 0)
               Diag(diag::err_imported_module_relocated)
                   << F.ModuleName << Blob << M->Directory->getName();
             return OutOfDate;
           }
case DELETE_EXPRS_TO_ANALYZE:
           DelayedDeleteExprs.push_back(ReadSourceLocation(F, Record, I).getRawEncoding());
           bool IsArrayForm = Record[I++] == 1;
           DelayedDeleteExprs.push_back(IsArrayForm);
         }
       }
       break;
 
     case IMPORTED_MODULES: {
-      if (F.Kind != MK_ImplicitModule && F.Kind != MK_ExplicitModule) {
+      if (!F.isModule()) {
         // If we aren't loading a module (which has its own exports), make
         // all of the imported modules visible.
         // FIXME: Deal with macros-only imports.
         for (unsigned I = 0, N = Record.size(); I != N; /**/) {
           unsigned GlobalID = getGlobalSubmoduleID(F, Record[I++]);
           SourceLocation Loc = ReadSourceLocation(F, Record, I);
           if (GlobalID)
             ImportedModules.push_back(ImportedSubmodule(GlobalID, Loc));

 
 ASTReader::ASTReadResult
 ASTReader::ReadModuleMapFileBlock(RecordData &Record, ModuleFile &F,
                                   const ModuleFile *ImportedBy,
                                   unsigned ClientLoadCapabilities) {
   unsigned Idx = 0;
   F.ModuleMapPath = ReadPath(F, Record, Idx);
 
-  if (F.Kind == MK_ExplicitModule) {
+  if (F.Kind == MK_ExplicitModule || F.Kind == MK_PrebuiltModule) {
     // For an explicitly-loaded module, we don't care whether the original
     // module map file exists or matches.
     return Success;
   }
 
   // Try to resolve ModuleName in the current header search context and
   // verify that it is found in the same module map file as we saved. If the
   // top-level AST file is a main file, skip this check because there is no
for (SmallVectorImpl<ImportedModule>::iterator M = Loaded.begin(),
     // any translation.
     if (!M->ImportedBy)
       F.ImportLoc = M->ImportLoc;
     else
       F.ImportLoc = TranslateSourceLocation(*M->ImportedBy, M->ImportLoc);
   }
 
   if (!Context.getLangOpts().CPlusPlus ||
-      (Type != MK_ImplicitModule && Type != MK_ExplicitModule)) {
+      (Type != MK_ImplicitModule && Type != MK_ExplicitModule &&
+       Type != MK_PrebuiltModule)) {
     // Mark all of the identifiers in the identifier table as being out of date,
     // so that various accessors know to check the loaded modules when the
     // identifier is used.
     //
     // For C++ modules, we don't need information on many identifiers (just
     // those that provide macros or are poisoned), so we mark all of
     // the interesting ones via PreloadIdentifierOffsets.
     for (IdentifierTable::iterator Id = PP.getIdentifierTable().begin(),

 }
 
 static unsigned moduleKindForDiagnostic(ModuleKind Kind) {
   switch (Kind) {
   case MK_PCH:
     return 0; // PCH
   case MK_ImplicitModule:
   case MK_ExplicitModule:
+  case MK_PrebuiltModule:
     return 1; // module
   case MK_MainFile:
   case MK_Preamble:
     return 2; // main source file
   }
   llvm_unreachable("unknown module kind");
 }
 
while (1) {
     case CONTROL_BLOCK_ID:
       HaveReadControlBlock = true;
       switch (ReadControlBlock(F, Loaded, ImportedBy, ClientLoadCapabilities)) {
       case Success:
         // Check that we didn't try to load a non-module AST file as a module.
         //
         // FIXME: Should we also perform the converse check? Loading a module as
         // a PCH file sort of works, but it's a bit wonky.
-        if ((Type == MK_ImplicitModule || Type == MK_ExplicitModule) &&
+        if ((Type == MK_ImplicitModule || Type == MK_ExplicitModule ||
+             Type == MK_PrebuiltModule) &&
             F.ModuleName.empty()) {
           auto Result = (Type == MK_ImplicitModule) ? OutOfDate : Failure;
           if (Result != OutOfDate ||
               (ClientLoadCapabilities & ARR_OutOfDate) == 0)
             Diag(diag::err_module_file_not_module) << FileName;
           return Result;
         }
         break;
while (!PendingIdentifierInfos.empty() ||
     for (unsigned I = 0; I != PendingMacroIDs.size(); ++I) {
       IdentifierInfo *II = PendingMacroIDs.begin()[I].first;
       SmallVector<PendingMacroInfo, 2> GlobalIDs;
       GlobalIDs.swap(PendingMacroIDs.begin()[I].second);
       // Initialize the macro history from chained-PCHs ahead of module imports.
       for (unsigned IDIdx = 0, NumIDs = GlobalIDs.size(); IDIdx != NumIDs;
            ++IDIdx) {
         const PendingMacroInfo &Info = GlobalIDs[IDIdx];
-        if (Info.M->Kind != MK_ImplicitModule &&
-            Info.M->Kind != MK_ExplicitModule)
+        if (!Info.M->isModule())
           resolvePendingMacro(II, Info);
       }
       // Handle module imports.
       for (unsigned IDIdx = 0, NumIDs = GlobalIDs.size(); IDIdx != NumIDs;
            ++IDIdx) {
         const PendingMacroInfo &Info = GlobalIDs[IDIdx];
-        if (Info.M->Kind == MK_ImplicitModule ||
-            Info.M->Kind == MK_ExplicitModule)
+        if (Info.M->isModule())
           resolvePendingMacro(II, Info);
       }
     }
     PendingMacroIDs.clear();
 
     // Wire up the DeclContexts for Decls that we delayed setting until
     // recursive loading is completed.
     while (!PendingDeclContextInfos.empty()) {

cfe/trunk/lib/Serialization/ASTReaderDecl.cpp

void ASTDeclReader::VisitNamespaceDecl(NamespaceDecl *D) {
 
   mergeRedeclarable(D, Redecl);
 
   if (AnonNamespace) {
     // Each module has its own anonymous namespace, which is disjoint from
     // any other module's anonymous namespaces, so don't attach the anonymous
     // namespace at all.
     NamespaceDecl *Anon = cast<NamespaceDecl>(Reader.GetDecl(AnonNamespace));
-    if (F.Kind != MK_ImplicitModule && F.Kind != MK_ExplicitModule)
+    if (!F.isModule())
       D->setAnonymousNamespace(Anon);
   }
 }
 
 void ASTDeclReader::VisitNamespaceAliasDecl(NamespaceAliasDecl *D) {
   RedeclarableResult Redecl = VisitRedeclarable(D);
   VisitNamedDecl(D);
   D->NamespaceLoc = ReadSourceLocation(Record, Idx);
while (Idx < Record.size()) {
 
     case UPD_CXX_ADDED_ANONYMOUS_NAMESPACE: {
       NamespaceDecl *Anon
         = Reader.ReadDeclAs<NamespaceDecl>(ModuleFile, Record, Idx);
       
       // Each module has its own anonymous namespace, which is disjoint from
       // any other module's anonymous namespaces, so don't attach the anonymous
       // namespace at all.
-      if (ModuleFile.Kind != MK_ImplicitModule &&
-          ModuleFile.Kind != MK_ExplicitModule) {
+      if (!ModuleFile.isModule()) {
         if (TranslationUnitDecl *TU = dyn_cast<TranslationUnitDecl>(D))
           TU->setAnonymousNamespace(Anon);
         else
           cast<NamespaceDecl>(D)->setAnonymousNamespace(Anon);
       }
       break;
     }
 

cfe/trunk/lib/Serialization/ModuleManager.cpp

ModuleManager::addModule(StringRef FileName, ModuleKind Type,
                          ASTFileSignatureReader ReadSignature,
                          ModuleFile *&Module,
                          std::string &ErrorStr) {
   Module = nullptr;
 
   // Look for the file entry. This only fails if the expected size or
   // modification time differ.
   const FileEntry *Entry;
-  if (Type == MK_ExplicitModule) {
+  if (Type == MK_ExplicitModule || Type == MK_PrebuiltModule) {
     // If we're not expecting to pull this file out of the module cache, it
     // might have a different mtime due to being moved across filesystems in
     // a distributed build. The size must still match, though. (As must the
     // contents, but we can't check that.)
     ExpectedModTime = 0;
   }
   if (lookupModuleFile(FileName, ExpectedSize, ExpectedModTime, Entry)) {
     ErrorStr = "module file out of date";

cfe/trunk/test/Driver/modules.m

 // MODULES_VALIDATE_SYSTEM_HEADERS: -fmodules-validate-system-headers
 
 // RUN: %clang -### %s 2>&1 | FileCheck -check-prefix=MODULES_DISABLE_DIAGNOSTIC_VALIDATION_DEFAULT %s
 // MODULES_DISABLE_DIAGNOSTIC_VALIDATION_DEFAULT-NOT: -fmodules-disable-diagnostic-validation
 
 // RUN: %clang -fmodules-disable-diagnostic-validation -### %s 2>&1 | FileCheck -check-prefix=MODULES_DISABLE_DIAGNOSTIC_VALIDATION %s
 // MODULES_DISABLE_DIAGNOSTIC_VALIDATION: -fmodules-disable-diagnostic-validation
 
+// RUN: %clang -fmodules -### %s 2>&1 | FileCheck -check-prefix=MODULES_PREBUILT_PATH_DEFAULT %s
+// MODULES_PREBUILT_PATH_DEFAULT-NOT: -fprebuilt-module-path
+
+// RUN: %clang -fmodules -fprebuilt-module-path=foo -fprebuilt-module-path=bar -### %s 2>&1 | FileCheck -check-prefix=MODULES_PREBUILT_PATH %s
+// MODULES_PREBUILT_PATH: "-fprebuilt-module-path=foo"
+// MODULES_PREBUILT_PATH: "-fprebuilt-module-path=bar"
+
 // RUN: %clang -fmodules -fmodule-map-file=foo.map -fmodule-map-file=bar.map -### %s 2>&1 | FileCheck -check-prefix=CHECK-MODULE-MAP-FILES %s
 // CHECK-MODULE-MAP-FILES: "-fmodules"
 // CHECK-MODULE-MAP-FILES: "-fmodule-map-file=foo.map"
 // CHECK-MODULE-MAP-FILES: "-fmodule-map-file=bar.map"
 
 // RUN: %clang -fmodules -fmodule-file=foo.pcm -fmodule-file=bar.pcm -### %s 2>&1 | FileCheck -check-prefix=CHECK-MODULE-FILES %s
 // CHECK-MODULE-FILES: "-fmodules"
 // CHECK-MODULE-FILES: "-fmodule-file=foo.pcm"
 // CHECK-MODULE-FILES: "-fmodule-file=bar.pcm"
 
 // RUN: %clang -fno-modules -fmodule-file=foo.pcm -fmodule-file=bar.pcm -### %s 2>&1 | FileCheck -check-prefix=CHECK-NO-MODULE-FILES %s
 // CHECK-NO-MODULE-FILES-NOT: "-fmodules"
 // CHECK-NO-MODULE-FILES-NOT: "-fmodule-file=foo.pcm"
 // CHECK-NO-MODULE-FILES-NOT: "-fmodule-file=bar.pcm"

cfe/trunk/test/Modules/Inputs/prebuilt-module/a.h

+const int a = 1;

cfe/trunk/test/Modules/Inputs/prebuilt-module/module.modulemap

+module prebuilt { header "a.h" }

cfe/trunk/test/Modules/prebuilt-module.m

+// RUN: rm -rf %t
+//
+// RUN: %clang_cc1 -fmodules -x objective-c -I %S/Inputs/prebuilt-module -triple %itanium_abi_triple -emit-module %S/Inputs/prebuilt-module/module.modulemap -fmodule-name=prebuilt -o %t/prebuilt.pcm
+// RUN: %clang_cc1 -fmodules -fprebuilt-module-path=%t/ -fdisable-module-hash %s -verify
+
+// expected-no-diagnostics
+@import prebuilt;
+int test() {
+  return a;
+}