Module: hash the pcm content and use it as SIGNATURE for implicit modules.

[dexonsmith]

Summary

We change ASTFileSignature from a random 32-bit number to the actual SHA hash of the pcm content.

1> Definition of ASTFileSignature is moved to Basic/Module.h so Module and ASTSourceDescriptor can use it. Before this patch both are using uint64_t as the type of the signature.
2> Add DIAGNOSTIC_OPTIONS_BLOCK and make sure it is the last block of the pcm file. This block contains DIAGNOSTIC_OPTIONS and SIGNATURE.
We are using this patch to solve a PCH+Modules issue: PCH are handled by the build system and modules are handled by the compiler, when the compiler overwrites a pcm file with the same content except the diagnostic differences, we can still treat the PCH as valid and continue using it. This requires us not hashing the diagnostic differences. Moving the diagnostic differences to the end of the pcm makes the task easy:

In a few records, we save bit positions in the pcm file, if the size of the diagnostic options is different, the bit positions can be off.
We only need to hash the content till the start of the DIAGNOSTIC_OPTIONS_BLOCK.

3> When we hash the pcm content, there is no point in saving the file size and modification time for the imported module file. The patch will use 0 for both. And the validation will not check the size differences or the modification time differences.
4> In GlobalModuleIndexBuilder, when the signature is non-zero, we check the signature instead of file size/modification time for imported modules.
5> Debug info has a 64-bit slot for dwo id, this patch uses the lower 64 bits of the SHA hash.

I want to gather some feedback on the general direction of this. Given that this is a big patch, I am open to separate it into smaller patches.

Cheers,
Manman

[aprantl]

include/clang/Serialization/ASTBitCodes.h

322 ↗

(On Diff #81145)

FYI, could be fixed in a separate commit for the entire file: These uses of \brief are redundant and we don't need them any more. We use the autobrief option in our Doxygen configuration file.

lib/CodeGen/CGDebugInfo.cpp

1978 ↗

(On Diff #81145)

Just wanted to comment that this change LGTM.

lib/CodeGen/ObjectFilePCHContainerOperations.cpp

247 ↗

(On Diff #81145)

Maybe factor this out into a getDwoId() helper function?

[rsmith]

Generally this seems reasonable to me. I don't see any particular need to split this patch up into smaller pieces.

include/clang/Serialization/ASTBitCodes.h

256 ↗

(On Diff #81145)

Add a comment describing this block.

Please also give this a name that describes its purpose (to hold records that should not be part of the signature). The signature is not a diagnostic option.

323 ↗

(On Diff #81145)

Move this and DIAGNOSTIC_OPTIONS to a separate enum holding record types for the new block.

include/clang/Serialization/ASTWriter.h

432 ↗

(On Diff #81145)

Fix indentation

435 ↗

(On Diff #81145)

Give the parameter a name

501–502 ↗

(On Diff #81145)

Fix indentation

534–535 ↗

(On Diff #81145)

Fix indentation

include/clang/Serialization/Module.h

19 ↗

(On Diff #81145)

Redundant include?

lib/Serialization/ASTWriter.cpp

1356 ↗

(On Diff #81145)

I don't think it makes sense to use ImplicitModules to control this. Doing that for the old signature computation was at best a hack.

If there is no measurable performance difference from the hashing, we should just do it unconditionally in all modes. Otherwise, there should be a separate flag to control it; we should probably force that flag to be enabled when the frontend implicitly builds a module and inserts it into the module cache, but otherwise let the user control it as they see fit.

[dexonsmith]

include/clang/Serialization/ASTBitCodes.h

256 ↗

(On Diff #81145)

Also add a comma to the end of the line to avoid future additions to the enum from touching this line.

[manmanren]

Addressing review comments!

I still need to measure the performance impact of calculating the hash then decide how to enable this.

[manmanren]

Rebase on top of r291686.

[manmanren]

Bruno helped on collecting performance data for this patch:
$ cat cocoa-test.h
#import <Cocoa/Cocoa.h>

$ rm -rf tmp; clang -x objective-c -isysroot xcrun --sdk macosx --show-sdk-path -fblocks -fmodules -fmodules-cache-path=tmp cocoa-test.h -fsyntax-only

clang-release (10 runs)

name avg min med max SD SD% total
user 2.0023 1.9804 1.9994 2.0408 0.0149 0.74 20.0225
sys 0.3601 0.3547 0.3505 0.4454 0.0287 7.98 3.6007
wall 2.5432 2.4291 2.4264 3.3604 0.2737 10.76 25.4324

clang-modhash-release (10 runs)

name avg min med max SD SD% total
user 2.1101 2.0935 2.1095 2.1274 0.0112 0.53 21.1012
sys 0.3485 0.3462 0.3556 0.3476 0.0052 1.50 3.4854
wall 2.5614 2.5163 2.5605 2.6657 0.0419 1.63 25.6138

This ~5% slowdown for the total time of compiling all the modules (30+ modules).

A similar test for one object file in selfhost clang:

[TemplateName.cpp.o] clang-release (10 runs)

name avg min med max SD SD% total
user 7.4332 7.3716 7.4369 7.4976 0.0444 0.60 74.3318
sys 0.5646 0.5512 0.5688 0.5570 0.0161 2.86 5.6464
wall 8.3357 8.1675 8.2991 8.2570 0.3240 3.89 83.3568

[TemplateName.cpp.o] clang-modhash-release (10 runs)

name avg min med max SD SD% total
user 7.6911 7.6202 7.6979 7.7779 0.0418 0.54 76.9114
sys 0.5531 0.5437 0.5520 0.5622 0.0105 1.89 5.5315
wall 8.4598 8.3868 8.4839 8.5497 0.0455 0.54 84.5977

~3.5% slowdown for total of modules:

$ ls module.cache/1UKP65OUKXURH/
Clang_AST-1XMT5DOVQWGLJ.pcm LLVM_C-2WKTHJHA6SDF0.pcm LLVM_Pass-HNKP05GYNHFQ.pcm _Builtin_stddef_max_align_t-2T552E4NSAIDI.pcm
Clang_Basic-1XMT5DOVQWGLJ.pcm LLVM_Config_ABI_Breaking-2FT1AFJTZ51QZ.pcm LLVM_Support_DataTypes-2FT1AFJTZ51QZ.pcm modules.idx
Darwin-399Z2LWDSMSTV.pcm LLVM_IR-HNKP05GYNHFQ.pcm LLVM_Utils-HNKP05GYNHFQ.pcm std-1337JMP4UGPIP.pcm

Note that the cache was wiped out right before every tests instance.

Thanks Bruno. I am going to guard the hashing with a CC1 option (something like fmodules-hash-content) and force that flag to be enabled when the frontend implicitly builds a module. We will try to improve the hashing performance later (either improving SHA1 or using MD5 instead of SHA1).

Manman

[manmanren]

Add CC1 option to control hashing of the module file content.

[manmanren]

include/clang/Serialization/Module.h

19 ↗

(On Diff #81145)

Definition of ASTFileSignature is moved from Serialization/Module.h to Basic/Module.h so we need to have this include to get the definition.

lib/Serialization/ASTWriter.cpp

1356 ↗

(On Diff #81145)

Use HSOpts.ModulesHashContent to guard the hashing of module file content. This flag is set to true when we implicitly build a module.

[manmanren]

Ping :]
I am hoping to wrap this up this week. Thanks,
Manman

[dexonsmith]

Taking this over to rebase.

[dexonsmith]

I've rebased the patch on ToT and cleaned a few things up.

Sadly, the clang/test/Modules/diagnostic-options-out-of-date.m test fails because r293123 makes the command-line diagnostic options affect the AST, and thus makes -Wconversion vs -Wno-conversion change ASTFileSignature.

I'm not sure how to move forward.

[dexonsmith]

Rebased past r293123, which filled in diagnostic pragma records for modules. I've moved the DIAG_PRAGMA_MAPPINGS to the unhashed block, as suggested by Richard, so the tests now pass.

Richard, can you take a look?

[dexonsmith]

Ping.

[rsmith]

clang/include/clang/Serialization/ASTBitCodes.h

258

This comment is out of date. Maybe just point to the UnhashedControlBlockRecordTypes for the definitive list of records within this block?

clang/lib/Serialization/ASTReader.cpp

2240–2241

I guess the reason to do this is because reading that block depends on certain things in this block having been already read, and reading other things in this block depends on that block having been read? A comment explaining that would be useful.

4012–4013

We shouldn't return Failure without producing an error message.

[dexonsmith]

Thanks for the review Richard! Committed in r297655 with those changes.

Sorry for the long delay on this. Somehow I missed until today the review after my ultimate ping. I'd like to blame Phab, but it's more likely that I just failed at email :/.

clang/include/clang/AST/ExternalASTSource.h

 //  construction of AST nodes from some external source.
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 #define LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 
 #include "clang/AST/CharUnits.h"
 #include "clang/AST/DeclBase.h"
+#include "clang/Basic/Module.h"
 #include "llvm/ADT/DenseMap.h"
 
 namespace clang {
 
 class ASTConsumer;
 class CXXBaseSpecifier;
 class CXXCtorInitializer;
 class DeclarationName;
public:
 
   /// Abstracts clang modules and precompiled header files and holds
   /// everything needed to generate debug info for an imported module
   /// or PCH.
   class ASTSourceDescriptor {
     StringRef PCHModuleName;
     StringRef Path;
     StringRef ASTFile;
-    uint64_t Signature = 0;
+    ASTFileSignature Signature;
     const Module *ClangModule = nullptr;
 
   public:
     ASTSourceDescriptor(){};
     ASTSourceDescriptor(StringRef Name, StringRef Path, StringRef ASTFile,
-                        uint64_t Signature)
+                        ASTFileSignature Signature)
         : PCHModuleName(std::move(Name)), Path(std::move(Path)),
           ASTFile(std::move(ASTFile)), Signature(Signature){};
     ASTSourceDescriptor(const Module &M);
     std::string getModuleName() const;
     StringRef getPath() const { return Path; }
     StringRef getASTFile() const { return ASTFile; }
-    uint64_t getSignature() const { return Signature; }
+    ASTFileSignature getSignature() const { return Signature; }
     const Module *getModuleOrNull() const { return ClangModule; }
   };
 
   /// Return a descriptor for the corresponding module, if one exists.
   virtual llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID);
 
   /// \brief Finds all declarations lexically contained within the given
   /// DeclContext, after applying an optional filter predicate.

clang/include/clang/Basic/Module.h

 namespace clang {
   
 class LangOptions;
 class TargetInfo;
 class IdentifierInfo;
   
 /// \brief Describes the name of a module.
 typedef SmallVector<std::pair<std::string, SourceLocation>, 2> ModuleId;
-  
+/// The signature of a module, which is a hash of the AST content.
+struct ASTFileSignature : std::array<uint32_t, 5> {
+  ASTFileSignature(std::array<uint32_t, 5> S = {{0}})
+      : std::array<uint32_t, 5>(std::move(S)) {}
+
+  explicit operator bool() const {
+    return *this != std::array<uint32_t, 5>({{0}});
+  }
+};
+
 /// \brief Describes a module or submodule.
 class Module {
 public:
   /// \brief The name of this module.
   std::string Name;
   
   /// \brief The location of the module definition.
   SourceLocation DefinitionLoc;
 
   /// \brief The parent of this module. This will be NULL for the top-level
   /// module.
   Module *Parent;
 
   /// \brief The build directory of this module. This is the directory in
   /// which the module is notionally built, and relative to which its headers
   /// are found.
   const DirectoryEntry *Directory;
 
   /// \brief The umbrella header or directory.
   llvm::PointerUnion<const DirectoryEntry *, const FileEntry *> Umbrella;
 
   /// \brief The module signature.
-  uint64_t Signature;
+  ASTFileSignature Signature;
 
   /// \brief The name of the umbrella entry, as written in the module map.
   std::string UmbrellaAsWritten;
   
 private:
   /// \brief The submodules of this module, indexed by name.
   std::vector<Module *> SubModules;
   

clang/include/clang/Driver/CC1Options.td

 //===----------------------------------------------------------------------===//
 // Header Search Options
 //===----------------------------------------------------------------------===//
 
 def nostdsysteminc : Flag<["-"], "nostdsysteminc">,
   HelpText<"Disable standard system #include directories">;
 def fdisable_module_hash : Flag<["-"], "fdisable-module-hash">,
   HelpText<"Disable the module hash">;
+def fmodules_hash_content : Flag<["-"], "fmodules-hash-content">,
+  HelpText<"Enable hashing the content of a module file">;
 def c_isystem : JoinedOrSeparate<["-"], "c-isystem">, MetaVarName<"<directory>">,
   HelpText<"Add directory to the C SYSTEM include search path">;
 def objc_isystem : JoinedOrSeparate<["-"], "objc-isystem">,
   MetaVarName<"<directory>">,
   HelpText<"Add directory to the ObjC SYSTEM include search path">;
 def objcxx_isystem : JoinedOrSeparate<["-"], "objcxx-isystem">,
   MetaVarName<"<directory>">,
   HelpText<"Add directory to the ObjC++ SYSTEM include search path">;

clang/include/clang/Frontend/PCHContainerOperations.h

 //===--- Frontend/PCHContainerOperations.h - PCH Containers -----*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_PCH_CONTAINER_OPERATIONS_H
 #define LLVM_CLANG_PCH_CONTAINER_OPERATIONS_H
 
+#include "clang/Basic/Module.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include <memory>
 
 namespace llvm {
 class raw_pwrite_stream;
 }
 
 using llvm::StringRef;
 
 namespace clang {
 
 class ASTConsumer;
 class CodeGenOptions;
 class DiagnosticsEngine;
 class CompilerInstance;
 
 struct PCHBuffer {
-  uint64_t Signature;
+  ASTFileSignature Signature;
   llvm::SmallVector<char, 0> Data;
   bool IsComplete;
 };
   
 /// This abstract interface provides operations for creating
 /// containers for serialized ASTs (precompiled headers and clang
 /// modules).
 class PCHContainerWriter {

clang/include/clang/Lex/HeaderSearchOptions.h

public:
   /// \brief Whether to validate system input files when a module is loaded.
   unsigned ModulesValidateSystemHeaders : 1;
 
   /// Whether the module includes debug information (-gmodules).
   unsigned UseDebugInfo : 1;
 
   unsigned ModulesValidateDiagnosticOptions : 1;
 
+  unsigned ModulesHashContent : 1;
+
   HeaderSearchOptions(StringRef _Sysroot = "/")
       : Sysroot(_Sysroot), ModuleFormat("raw"), DisableModuleHash(0),
         ImplicitModuleMaps(0), ModuleMapFileHomeIsCwd(0),
         ModuleCachePruneInterval(7 * 24 * 60 * 60),
         ModuleCachePruneAfter(31 * 24 * 60 * 60), BuildSessionTimestamp(0),
         UseBuiltinIncludes(true), UseStandardSystemIncludes(true),
         UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false),
         ModulesValidateOncePerBuildSession(false),
-        ModulesValidateSystemHeaders(false),
-        UseDebugInfo(false), ModulesValidateDiagnosticOptions(true) {}
+        ModulesValidateSystemHeaders(false), UseDebugInfo(false),
+        ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) {}
 
   /// AddPath - Add the \p Path path to the specified \p Group list.
   void AddPath(StringRef Path, frontend::IncludeDirGroup Group,
                bool IsFramework, bool IgnoreSysRoot) {
     UserEntries.emplace_back(Path, Group, IsFramework, IgnoreSysRoot);
   }
 
   /// AddSystemHeaderPrefix - Override whether \#include directives naming a

clang/include/clang/Serialization/ASTBitCodes.h

enum BlockIDs {
       PREPROCESSOR_BLOCK_ID,
 
       /// \brief The block containing the definitions of all of the
       /// types and decls used within the AST file.
       DECLTYPES_BLOCK_ID,
 
       /// \brief The block containing the detailed preprocessing record.
       PREPROCESSOR_DETAIL_BLOCK_ID,
-      
       /// \brief The block containing the submodule structure.
       SUBMODULE_BLOCK_ID,
 
       /// \brief The block containing comments.
       COMMENTS_BLOCK_ID,
 
       /// \brief The control block, which contains all of the
       /// information that needs to be validated prior to committing
enum BlockIDs {
       /// a module is being used in a configuration compatible with the
       /// configuration in which it was built.
       ///
       /// This block is part of the control block.
       OPTIONS_BLOCK_ID,
 
       /// \brief A block containing a module file extension.
       EXTENSION_BLOCK_ID,
+
+      /// A block containing unhashed contents. It currently holds Diagnostic
+      /// Options and Signature.
+      UNHASHED_CONTROL_BLOCK_ID,
     };
 
     /// \brief Record types that occur within the control block.
     enum ControlRecordTypes {
       /// \brief AST file metadata, including the AST file version number
       /// and information about the compiler used to build this AST file.
       METADATA = 1,
 
enum ControlRecordTypes {
 
       /// \brief Record code for the module name.
       MODULE_NAME,
 
       /// \brief Record code for the module map file that was used to build this
       /// AST file.
       MODULE_MAP_FILE,
 
-      /// \brief Record code for the signature that identifiers this AST file.
-      SIGNATURE,
-
       /// \brief Record code for the module build directory.
       MODULE_DIRECTORY,
     };
 
     /// \brief Record types that occur within the options block inside
     /// the control block.
     enum OptionsRecordTypes {
       /// \brief Record code for the language options table.
       ///
       /// The record with this code contains the contents of the
       /// LangOptions structure. We serialize the entire contents of
       /// the structure, and let the reader decide which options are
       /// actually important to check.
       LANGUAGE_OPTIONS = 1,
 
       /// \brief Record code for the target options table.
       TARGET_OPTIONS,
 
-      /// \brief Record code for the diagnostic options table.
-      DIAGNOSTIC_OPTIONS,
-
       /// \brief Record code for the filesystem options table.
       FILE_SYSTEM_OPTIONS,
 
       /// \brief Record code for the headers search options table.
       HEADER_SEARCH_OPTIONS,
 
       /// \brief Record code for the preprocessor options table.
       PREPROCESSOR_OPTIONS,
     };
 
+    /// Record codes for the unhashed control block.
+    enum UnhashedControlBlockRecordTypes {
+      /// Record code for the signature that identifiers this AST file.
+      SIGNATURE = 1,
+
+      /// Record code for the diagnostic options table.
+      DIAGNOSTIC_OPTIONS,
+    };
+
     /// \brief Record code for extension blocks.
     enum ExtensionBlockRecordTypes {
       /// Metadata describing this particular extension.
       EXTENSION_METADATA = 1,
 
       /// The first record ID allocated to the extensions themselves.
       FIRST_EXTENSION_RECORD_ID = 4
     };

clang/include/clang/Serialization/ASTReader.h

ImportedModule(ModuleFile *Mod,
                    SourceLocation ImportLoc)
       : Mod(Mod), ImportedBy(ImportedBy), ImportLoc(ImportLoc) { }
   };
 
   ASTReadResult ReadASTCore(StringRef FileName, ModuleKind Type,
                             SourceLocation ImportLoc, ModuleFile *ImportedBy,
                             SmallVectorImpl<ImportedModule> &Loaded,
                             off_t ExpectedSize, time_t ExpectedModTime,
-                            serialization::ASTFileSignature ExpectedSignature,
+                            ASTFileSignature ExpectedSignature,
                             unsigned ClientLoadCapabilities);
   ASTReadResult ReadControlBlock(ModuleFile &F,
                                  SmallVectorImpl<ImportedModule> &Loaded,
                                  const ModuleFile *ImportedBy,
                                  unsigned ClientLoadCapabilities);
   static ASTReadResult ReadOptionsBlock(
       llvm::BitstreamCursor &Stream, unsigned ClientLoadCapabilities,
       bool AllowCompatibleConfigurationMismatch, ASTReaderListener &Listener,
-      std::string &SuggestedPredefines, bool ValidateDiagnosticOptions);
+      std::string &SuggestedPredefines);
+
+  /// Read the unhashed control block.
+  ///
+  /// This has no effect on \c F.Stream, instead creating a fresh cursor from
+  /// \c F.Data and reading ahead.
+  ASTReadResult readUnhashedControlBlock(ModuleFile &F, bool WasImportedBy,
+                                         unsigned ClientLoadCapabilities);
+
+  static ASTReadResult
+  readUnhashedControlBlockImpl(ModuleFile *F, llvm::StringRef StreamData,
+                               unsigned ClientLoadCapabilities,
+                               bool AllowCompatibleConfigurationMismatch,
+                               ASTReaderListener *Listener,
+                               bool ValidateDiagnosticOptions);
+
   ASTReadResult ReadASTBlock(ModuleFile &F, unsigned ClientLoadCapabilities);
   ASTReadResult ReadExtensionBlock(ModuleFile &F);
   bool ParseLineTable(ModuleFile &F, const RecordData &Record);
   bool ReadSourceManagerBlock(ModuleFile &F);
   llvm::BitstreamCursor &SLocCursorForID(int ID);
   SourceLocation getImportLocation(ModuleFile *F);
   ASTReadResult ReadModuleMapFileBlock(RecordData &Record, ModuleFile &F,
                                        const ModuleFile *ImportedBy,

clang/include/clang/Serialization/ASTWriter.h

private:
   /// Keys in the map never have const/volatile qualifiers.
   typedef llvm::DenseMap<QualType, serialization::TypeIdx,
                          serialization::UnsafeQualTypeDenseMapInfo>
     TypeIdxMap;
 
   /// \brief The bitstream writer used to emit this precompiled header.
   llvm::BitstreamWriter &Stream;
 
+  /// The buffer associated with the bitstream.
+  const SmallVectorImpl<char> &Buffer;
+
   /// \brief The ASTContext we're writing.
   ASTContext *Context = nullptr;
 
   /// \brief The preprocessor we're writing.
   Preprocessor *PP = nullptr;
 
   /// \brief The reader of existing AST files, if we're chaining.
   ASTReader *Chain = nullptr;
private:
 
   /// \brief Retrieve or create a submodule ID for this module.
   unsigned getSubmoduleID(Module *Mod);
 
   /// \brief Write the given subexpression to the bitstream.
   void WriteSubStmt(Stmt *S);
 
   void WriteBlockInfoBlock();
-  uint64_t WriteControlBlock(Preprocessor &PP, ASTContext &Context,
-                             StringRef isysroot, const std::string &OutputFile);
+  void WriteControlBlock(Preprocessor &PP, ASTContext &Context,
+
+  /// Write out the signature and diagnostic options, and return the signature.
+  ASTFileSignature writeUnhashedControlBlock(Preprocessor &PP,
+                                             ASTContext &Context);
+
+  /// Calculate hash of the pcm content.
+  static ASTFileSignature createSignature(StringRef Bytes);
+
   void WriteInputFiles(SourceManager &SourceMgr, HeaderSearchOptions &HSOpts,
                        bool Modules);
   void WriteSourceManagerBlock(SourceManager &SourceMgr,
                                const Preprocessor &PP);
   void WritePreprocessor(const Preprocessor &PP, bool IsModule);
   void WriteHeaderSearch(const HeaderSearch &HS);
   void WritePreprocessorDetail(PreprocessingRecord &PPRec);
   void WriteSubmodules(Module *WritingModule);
private:
   unsigned DeclRefExprAbbrev = 0;
   unsigned CharacterLiteralAbbrev = 0;
   unsigned IntegerLiteralAbbrev = 0;
   unsigned ExprImplicitCastAbbrev = 0;
 
   void WriteDeclAbbrevs();
   void WriteDecl(ASTContext &Context, Decl *D);
 
-  uint64_t WriteASTCore(Sema &SemaRef,
-                        StringRef isysroot, const std::string &OutputFile,
+  ASTFileSignature WriteASTCore(Sema &SemaRef, StringRef isysroot,
+                                const std::string &OutputFile,
                                 Module *WritingModule);
 
 public:
   /// \brief Create a new precompiled header writer that outputs to
   /// the given bitstream.
-  ASTWriter(llvm::BitstreamWriter &Stream,
+  ASTWriter(llvm::BitstreamWriter &Stream, SmallVectorImpl<char> &Buffer,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             bool IncludeTimestamps = true);
   ~ASTWriter() override;
 
   const LangOptions &getLangOpts() const;
 
   /// \brief Get a timestamp for output into the AST file. The actual timestamp
   /// of the specified file may be ignored if we have been instructed to not
public:
   /// writing a precompiled header.
   ///
   /// \param isysroot if non-empty, write a relocatable file whose headers
   /// are relative to the given system root. If we're writing a module, its
   /// build directory will be used in preference to this if both are available.
   ///
   /// \return the module signature, which eventually will be a hash of
   /// the module but currently is merely a random 32-bit number.
-  uint64_t WriteAST(Sema &SemaRef, const std::string &OutputFile,
+  ASTFileSignature WriteAST(Sema &SemaRef, const std::string &OutputFile,
                             Module *WritingModule, StringRef isysroot,
                             bool hasErrors = false);
 
   /// \brief Emit a token.
   void AddToken(const Token &Tok, RecordDataImpl &Record);
 
   /// \brief Emit a source location.
   void AddSourceLocation(SourceLocation Loc, RecordDataImpl &Record);
 
   /// \brief Emit a source range.

clang/include/clang/Serialization/Module.h

 //  been loaded from an AST file.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_SERIALIZATION_MODULE_H
 #define LLVM_CLANG_SERIALIZATION_MODULE_H
 
 #include "clang/Basic/FileManager.h"
+#include "clang/Basic/Module.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Serialization/ASTBitCodes.h"
 #include "clang/Serialization/ContinuousRangeMap.h"
 #include "clang/Serialization/ModuleFileExtension.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/Bitcode/BitstreamReader.h"
 #include "llvm/Support/Endian.h"
 #include <memory>
public:
   }
 
   const FileEntry *getFile() const { return Val.getPointer(); }
   bool isOverridden() const { return Val.getInt() == Overridden; }
   bool isOutOfDate() const { return Val.getInt() == OutOfDate; }
   bool isNotFound() const { return Val.getInt() == NotFound; }
 };
 
-typedef unsigned ASTFileSignature;
-
 /// \brief Information about a module that has been loaded by the ASTReader.
 ///
 /// Each instance of the Module class corresponds to a single AST file, which
 /// may be a precompiled header, precompiled preamble, a module, or an AST file
 /// of some sort loaded as the main file, all of which are specific formulations
 /// of the general notion of a "module". A module may depend on any number of
 /// other modules.
 class ModuleFile {
public:
   bool RelocatablePCH = false;
 
   /// \brief Whether timestamps are included in this module file.
   bool HasTimestamps = false;
 
   /// \brief The file entry for the module file.
   const FileEntry *File = nullptr;
 
-  /// \brief The signature of the module file, which may be used along with size
+  /// The signature of the module file, which may be used instead of the size
   /// and modification time to identify this particular file.
-  ASTFileSignature Signature = 0;
+  ASTFileSignature Signature;
 
   /// \brief Whether this module has been directly imported by the
   /// user.
   bool DirectlyImported = false;
 
   /// \brief The generation of which this module file is a part.
   unsigned Generation;
   

clang/lib/Basic/Module.cpp

 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace clang;
 
 Module::Module(StringRef Name, SourceLocation DefinitionLoc, Module *Parent,
                bool IsFramework, bool IsExplicit, unsigned VisibilityID)
     : Name(Name), DefinitionLoc(DefinitionLoc), Parent(Parent), Directory(),
-      Umbrella(), Signature(0), ASTFile(nullptr), VisibilityID(VisibilityID),
+      Umbrella(), ASTFile(nullptr), VisibilityID(VisibilityID),
       IsMissingRequirement(false), HasIncompatibleModuleFile(false),
       IsAvailable(true), IsFromModuleFile(false), IsFramework(IsFramework),
       IsExplicit(IsExplicit), IsSystem(false), IsExternC(false),
       IsInferred(false), InferSubmodules(false), InferExplicitSubmodules(false),
       InferExportWildcard(false), ConfigMacrosExhaustive(false),
       NoUndeclaredIncludes(false), NameVisibility(Hidden) {
   if (Parent) {
     if (!Parent->isAvailable())

clang/lib/CodeGen/CGDebugInfo.cpp

for (auto &M : PPOpts.Macros) {
       OS << '\"';
     }
   }
 
   bool IsRootModule = M ? !M->Parent : true;
   if (CreateSkeletonCU && IsRootModule) {
     // PCH files don't have a signature field in the control block,
     // but LLVM detects skeleton CUs by looking for a non-zero DWO id.
-    uint64_t Signature = Mod.getSignature() ? Mod.getSignature() : ~1ULL;
+    // We use the lower 64 bits for debug info.
+    uint64_t Signature =
+        Mod.getSignature()
+            ? (uint64_t)Mod.getSignature()[1] << 32 | Mod.getSignature()[0]
+            : ~1ULL;
     llvm::DIBuilder DIB(CGM.getModule());
     DIB.createCompileUnit(TheCU->getSourceLanguage(),
                           DIB.createFile(Mod.getModuleName(), Mod.getPath()),
                           TheCU->getProducer(), true, StringRef(), 0,
                           Mod.getASTFile(), llvm::DICompileUnit::FullDebug,
                           Signature);
     DIB.finalize();
   }

clang/lib/CodeGen/ObjectFilePCHContainerOperations.cpp

void Initialize(ASTContext &Context) override {
     M.reset(new llvm::Module(MainFileName, *VMContext));
     M->setDataLayout(Ctx->getTargetInfo().getDataLayout());
     Builder.reset(new CodeGen::CodeGenModule(
         *Ctx, HeaderSearchOpts, PreprocessorOpts, CodeGenOpts, *M, Diags));
 
     // Prepare CGDebugInfo to emit debug info for a clang module.
     auto *DI = Builder->getModuleDebugInfo();
     StringRef ModuleName = llvm::sys::path::filename(MainFileName);
-    DI->setPCHDescriptor({ModuleName, "", OutputFileName, ~1ULL});
+    DI->setPCHDescriptor({ModuleName, "", OutputFileName,
+                          ASTFileSignature{{{~0U, ~0U, ~0U, ~0U, ~1U}}}});
     DI->setModuleMap(MMap);
   }
 
   bool HandleTopLevelDecl(DeclGroupRef D) override {
     if (Diags.hasErrorOccurred())
       return true;
 
     // Collect debug info for all decls in this group.
void HandleTranslationUnit(ASTContext &Ctx) override {
     if (Diags.hasErrorOccurred())
       return;
 
     M->setTargetTriple(Ctx.getTargetInfo().getTriple().getTriple());
     M->setDataLayout(Ctx.getTargetInfo().getDataLayout());
 
     // PCH files don't have a signature field in the control block,
     // but LLVM detects DWO CUs by looking for a non-zero DWO id.
-    uint64_t Signature = Buffer->Signature ? Buffer->Signature : ~1ULL;
+    // We use the lower 64 bits for debug info.
+    uint64_t Signature =
+        Buffer->Signature
+            ? (uint64_t)Buffer->Signature[1] << 32 | Buffer->Signature[0]
+            : ~1ULL;
     Builder->getModuleDebugInfo()->setDwoId(Signature);
 
     // Finalize the Builder.
     if (Builder)
       Builder->Release();
 
     // Ensure the target exists.
     std::string Error;

clang/lib/Frontend/ASTUnit.cpp

void OnDiskData::Cleanup() {
   CleanPreambleFile();
 }
 
 struct ASTUnit::ASTWriterData {
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream;
   ASTWriter Writer;
 
-  ASTWriterData() : Stream(Buffer), Writer(Stream, { }) { }
+  ASTWriterData() : Stream(Buffer), Writer(Stream, Buffer, {}) {}
 };
 
 void ASTUnit::clearFileLevelDecls() {
   llvm::DeleteContainerSeconds(FileDecls);
 }
 
 void ASTUnit::CleanTemporaryFiles() {
   getOnDiskData(this).CleanTemporaryFiles();
bool ASTUnit::serialize(raw_ostream &OS) {
   bool hasErrors = getDiagnostics().hasUncompilableErrorOccurred();
 
   if (WriterData)
     return serializeUnit(WriterData->Writer, WriterData->Buffer,
                          getSema(), hasErrors, OS);
 
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream(Buffer);
-  ASTWriter Writer(Stream, { });
+  ASTWriter Writer(Stream, Buffer, {});
   return serializeUnit(Writer, Buffer, getSema(), hasErrors, OS);
 }
 
 typedef ContinuousRangeMap<unsigned, int, 2> SLocRemap;
 
 void ASTUnit::TranslateStoredDiagnostics(
                           FileManager &FileMgr,
                           SourceManager &SrcMgr,

clang/lib/Frontend/CompilerInstance.cpp

static bool compileModuleImpl(CompilerInstance &ImportingInstance,
   
   // For any options that aren't intended to affect how a module is built,
   // reset them to their default values.
   Invocation->getLangOpts()->resetNonModularOptions();
   PPOpts.resetNonModularOptions();
 
   // Remove any macro definitions that are explicitly ignored by the module.
   // They aren't supposed to affect how the module is built anyway.
-  const HeaderSearchOptions &HSOpts = Invocation->getHeaderSearchOpts();
+  HeaderSearchOptions &HSOpts = Invocation->getHeaderSearchOpts();
   PPOpts.Macros.erase(
       std::remove_if(PPOpts.Macros.begin(), PPOpts.Macros.end(),
                      [&HSOpts](const std::pair<std::string, bool> &def) {
         StringRef MacroDef = def.first;
         return HSOpts.ModulesIgnoreMacros.count(
                    llvm::CachedHashString(MacroDef.split('=').first)) > 0;
       }),
       PPOpts.Macros.end());
static bool compileModuleImpl(CompilerInstance &ImportingInstance,
   // If there is a module map file, build the module using the module map.
   // Set up the inputs/outputs so that we build the module from its umbrella
   // header.
   FrontendOptions &FrontendOpts = Invocation->getFrontendOpts();
   FrontendOpts.OutputFile = ModuleFileName.str();
   FrontendOpts.DisableFree = false;
   FrontendOpts.GenerateGlobalModuleIndex = false;
   FrontendOpts.BuildingImplicitModule = true;
+  // Force implicitly-built modules to hash the content of the module file.
+  HSOpts.ModulesHashContent = true;
   FrontendOpts.Inputs.clear();
   InputKind IK = getSourceInputKindFromOptions(*Invocation->getLangOpts());
 
   // Don't free the remapped file buffers; they are owned by our caller.
   PPOpts.RetainRemappedFileBuffers = true;
     
   Invocation->getDiagnosticOpts().VerifyDiagnostics = 0;
   assert(ImportingInstance.getInvocation().getModuleHash() ==

clang/lib/Frontend/CompilerInvocation.cpp

static void ParseHeaderSearchArgs(HeaderSearchOptions &Opts, ArgList &Args) {
   if (const Arg *A = Args.getLastArg(OPT_stdlib_EQ))
     Opts.UseLibcxx = (strcmp(A->getValue(), "libc++") == 0);
   Opts.ResourceDir = Args.getLastArgValue(OPT_resource_dir);
   Opts.ModuleCachePath = Args.getLastArgValue(OPT_fmodules_cache_path);
   Opts.ModuleUserBuildPath = Args.getLastArgValue(OPT_fmodules_user_build_path);
   for (const Arg *A : Args.filtered(OPT_fprebuilt_module_path))
     Opts.AddPrebuiltModulePath(A->getValue());
   Opts.DisableModuleHash = Args.hasArg(OPT_fdisable_module_hash);
+  Opts.ModulesHashContent = Args.hasArg(OPT_fmodules_hash_content);
   Opts.ModulesValidateDiagnosticOptions =
       !Args.hasArg(OPT_fmodules_disable_diagnostic_validation);
   Opts.ImplicitModuleMaps = Args.hasArg(OPT_fimplicit_module_maps);
   Opts.ModuleMapFileHomeIsCwd = Args.hasArg(OPT_fmodule_map_file_home_is_cwd);
   Opts.ModuleCachePruneInterval =
       getLastArgIntValue(Args, OPT_fmodules_prune_interval, 7 * 24 * 60 * 60);
   Opts.ModuleCachePruneAfter =
       getLastArgIntValue(Args, OPT_fmodules_prune_after, 31 * 24 * 60 * 60);

clang/lib/Serialization/ASTReader.cpp

static bool isDiagnosedResult(ASTReader::ASTReadResult ARR, unsigned Caps) {
   }
 
   llvm_unreachable("unknown ASTReadResult");
 }
 
 ASTReader::ASTReadResult ASTReader::ReadOptionsBlock(
     BitstreamCursor &Stream, unsigned ClientLoadCapabilities,
     bool AllowCompatibleConfigurationMismatch, ASTReaderListener &Listener,
-    std::string &SuggestedPredefines, bool ValidateDiagnosticOptions) {
+    std::string &SuggestedPredefines) {
   if (Stream.EnterSubBlock(OPTIONS_BLOCK_ID))
     return Failure;
 
   // Read all of the records in the options block.
   RecordData Record;
   ASTReadResult Result = Success;
   while (true) {
     llvm::BitstreamEntry Entry = Stream.advance();
while (true) {
     case TARGET_OPTIONS: {
       bool Complain = (ClientLoadCapabilities & ARR_ConfigurationMismatch) == 0;
       if (ParseTargetOptions(Record, Complain, Listener,
                              AllowCompatibleConfigurationMismatch))
         Result = ConfigurationMismatch;
       break;
     }
 
-    case DIAGNOSTIC_OPTIONS: {
-      bool Complain = (ClientLoadCapabilities & ARR_OutOfDate) == 0;
-      if (ValidateDiagnosticOptions &&
-          !AllowCompatibleConfigurationMismatch &&
-          ParseDiagnosticOptions(Record, Complain, Listener))
-        return OutOfDate;
-      break;
-    }
-
     case FILE_SYSTEM_OPTIONS: {
       bool Complain = (ClientLoadCapabilities & ARR_ConfigurationMismatch) == 0;
       if (!AllowCompatibleConfigurationMismatch &&
           ParseFileSystemOptions(Record, Complain, Listener))
         Result = ConfigurationMismatch;
       break;
     }
 
ASTReader::ReadControlBlock(ModuleFile &F,
   BitstreamCursor &Stream = F.Stream;
   ASTReadResult Result = Success;
 
   if (Stream.EnterSubBlock(CONTROL_BLOCK_ID)) {
     Error("malformed block record in AST file");
     return Failure;
   }
 
+  // Lambda to read the unhashed control block the first time it's called.
+  bool HasReadUnhashedControlBlock = false;
+  auto readUnhashedControlBlockOnce = [&]() {
+    if (!HasReadUnhashedControlBlock) {
+      HasReadUnhashedControlBlock = true;
+      if (ASTReadResult Result =
+              readUnhashedControlBlock(F, ImportedBy, ClientLoadCapabilities))
+        return Result;
+    }
+    return Success;
+  };
+
   // Read all of the records and blocks in the control block.
   RecordData Record;
   unsigned NumInputs = 0;
   unsigned NumUserInputs = 0;
   while (true) {
     llvm::BitstreamEntry Entry = Stream.advance();
 
     switch (Entry.Kind) {
     case llvm::BitstreamEntry::Error:
       Error("malformed block record in AST file");
       return Failure;
     case llvm::BitstreamEntry::EndBlock: {
+      // Validate the module before returning.  This call catches an AST with
+      // no module name and no imports.
+      if (ASTReadResult Result = readUnhashedControlBlockOnce())
+        return Result;
+
       // Validate input files.
       const HeaderSearchOptions &HSOpts =
           PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
       // All user input files reside at the index range [0, NumUserInputs), and
       // system input files reside at [NumUserInputs, NumInputs). For explicitly
       // loaded module files, ignore missing inputs.
       if (!DisableValidation && F.Kind != MK_ExplicitModule &&
case llvm::BitstreamEntry::SubBlock:
         if (Listener && !ImportedBy) {
           // Should we allow the configuration of the module file to differ from
           // the configuration of the current translation unit in a compatible
           // way?
           //
           // FIXME: Allow this for files explicitly specified with -include-pch.
           bool AllowCompatibleConfigurationMismatch =
               F.Kind == MK_ExplicitModule || F.Kind == MK_PrebuiltModule;
-          const HeaderSearchOptions &HSOpts =
-              PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
           Result = ReadOptionsBlock(Stream, ClientLoadCapabilities,
                                     AllowCompatibleConfigurationMismatch,
-                                    *Listener, SuggestedPredefines,
-                                    HSOpts.ModulesValidateDiagnosticOptions);
+                                    *Listener, SuggestedPredefines);
           if (Result == Failure) {
             Error("malformed block record in AST file");
             return Result;
           }
 
           if (DisableValidation ||
               (AllowConfigurationMismatch && Result == ConfigurationMismatch))
             Result = Success;
case METADATA: {
       if (StringRef(CurBranch) != ASTBranch && !DisableValidation) {
         if ((ClientLoadCapabilities & ARR_VersionMismatch) == 0)
           Diag(diag::err_pch_different_branch) << ASTBranch << CurBranch;
         return VersionMismatch;
       }
       break;
     }
 
-    case SIGNATURE:
-      assert((!F.Signature || F.Signature == Record[0]) && "signature changed");
-      F.Signature = Record[0];
-      break;
-
     case IMPORTS: {
+      // Validate the AST before processing any imports (otherwise, untangling
+      // them can be error-prone and expensive).  A module will have a name and
+      // will already have been validated, but this catches the PCH case.
+      if (ASTReadResult Result = readUnhashedControlBlockOnce())
+        return Result;
+
       // Load each of the imported PCH files.
       unsigned Idx = 0, N = Record.size();
       while (Idx < N) {
         // Read information about the AST file.
         ModuleKind ImportedKind = (ModuleKind)Record[Idx++];
         // The import location will be the local one for now; we will adjust
         // all import locations of module imports after the global source
         // location info are setup, in ReadAST.
         SourceLocation ImportLoc =
             ReadUntranslatedSourceLocation(Record[Idx++]);
         off_t StoredSize = (off_t)Record[Idx++];
         time_t StoredModTime = (time_t)Record[Idx++];
-        ASTFileSignature StoredSignature = Record[Idx++];
+        ASTFileSignature StoredSignature = {
+            {{(uint32_t)Record[Idx++], (uint32_t)Record[Idx++],
+              (uint32_t)Record[Idx++], (uint32_t)Record[Idx++],
+              (uint32_t)Record[Idx++]}}};
         auto ImportedFile = ReadPath(F, Record, Idx);
 
         // If our client can't cope with us being out of date, we can't cope with
         // our dependency being missing.
         unsigned Capabilities = ClientLoadCapabilities;
         if ((ClientLoadCapabilities & ARR_OutOfDate) == 0)
           Capabilities &= ~ARR_Missing;
 
while (true) {
     case ORIGINAL_PCH_DIR:
       F.OriginalDir = Blob;
       break;
 
     case MODULE_NAME:
       F.ModuleName = Blob;
       if (Listener)
         Listener->ReadModuleName(F.ModuleName);
+
+      // Validate the AST as soon as we have a name so we can exit early on
+      // failure.
+      if (ASTReadResult Result = readUnhashedControlBlockOnce())
+        return Result;
+
       break;
 
     case MODULE_DIRECTORY: {
       assert(!F.ModuleName.empty() &&
              "MODULE_DIRECTORY found before MODULE_NAME");
       // If we've already loaded a module map file covering this module, we may
       // have a better path for it (relative to the current build).
       Module *M = PP.getHeaderSearchInfo().lookupModule(F.ModuleName);
ASTReader::ASTReadResult ASTReader::ReadAST(StringRef FileName,
   // Defer any pending actions until we get to the end of reading the AST file.
   Deserializing AnASTFile(this);
 
   // Bump the generation number.
   unsigned PreviousGeneration = incrementGeneration(Context);
 
   unsigned NumModules = ModuleMgr.size();
   SmallVector<ImportedModule, 4> Loaded;
-  switch(ASTReadResult ReadResult = ReadASTCore(FileName, Type, ImportLoc,
-                                                /*ImportedBy=*/nullptr, Loaded,
-                                                0, 0, 0,
-                                                ClientLoadCapabilities)) {
+  switch (ASTReadResult ReadResult =
+              ReadASTCore(FileName, Type, ImportLoc,
+                          /*ImportedBy=*/nullptr, Loaded, 0, 0,
+                          ASTFileSignature(), ClientLoadCapabilities)) {
   case Failure:
   case Missing:
   case OutOfDate:
   case VersionMismatch:
   case ConfigurationMismatch:
   case HadErrors: {
     llvm::SmallPtrSet<ModuleFile *, 4> LoadedSet;
     for (const ImportedModule &IM : Loaded)
case AST_BLOCK_ID:
           Diag(diag::err_pch_version_too_old);
         return VersionMismatch;
       }
 
       // Record that we've loaded this module.
       Loaded.push_back(ImportedModule(M, ImportedBy, ImportLoc));
       return Success;
 
+    case UNHASHED_CONTROL_BLOCK_ID:
+      // This block is handled using look-ahead during ReadControlBlock.  We
+      // shouldn't get here!
+      return Failure;
+
     default:
       if (Stream.SkipBlock()) {
         Error("malformed block record in AST file");
         return Failure;
       }
       break;
     }
   }
 
   return Success;
 }
 
+ASTReader::ASTReadResult
+ASTReader::readUnhashedControlBlock(ModuleFile &F, bool WasImportedBy,
+                                    unsigned ClientLoadCapabilities) {
+  const HeaderSearchOptions &HSOpts =
+      PP.getHeaderSearchInfo().getHeaderSearchOpts();
+  bool AllowCompatibleConfigurationMismatch =
+      F.Kind == MK_ExplicitModule || F.Kind == MK_PrebuiltModule;
+
+  ASTReadResult Result = readUnhashedControlBlockImpl(
+      &F, F.Data, ClientLoadCapabilities, AllowCompatibleConfigurationMismatch,
+      Listener.get(),
+      WasImportedBy ? false : HSOpts.ModulesValidateDiagnosticOptions);
+
+  if (DisableValidation || WasImportedBy ||
+      (AllowConfigurationMismatch && Result == ConfigurationMismatch))
+    return Success;
+
+  if (Result == Failure)
+    Error("malformed block record in AST file");
+
+  return Result;
+}
+
+ASTReader::ASTReadResult ASTReader::readUnhashedControlBlockImpl(
+    ModuleFile *F, llvm::StringRef StreamData, unsigned ClientLoadCapabilities,
+    bool AllowCompatibleConfigurationMismatch, ASTReaderListener *Listener,
+    bool ValidateDiagnosticOptions) {
+  // Initialize a stream.
+  BitstreamCursor Stream(StreamData);
+
+  // Sniff for the signature.
+  if (!startsWithASTFileMagic(Stream))
+    return Failure;
+
+  // Scan for the UNHASHED_CONTROL_BLOCK_ID block.
+  if (SkipCursorToBlock(Stream, UNHASHED_CONTROL_BLOCK_ID))
+    return Failure;
+
+  // Read all of the records in the options block.
+  RecordData Record;
+  ASTReadResult Result = Success;
+  while (1) {
+    llvm::BitstreamEntry Entry = Stream.advance();
+
+    switch (Entry.Kind) {
+    case llvm::BitstreamEntry::Error:
+    case llvm::BitstreamEntry::SubBlock:
+      return Failure;
+
+    case llvm::BitstreamEntry::EndBlock:
+      return Result;
+
+    case llvm::BitstreamEntry::Record:
+      // The interesting case.
+      break;
+    }
+
+    // Read and process a record.
+    Record.clear();
+    switch (
+        (UnhashedControlBlockRecordTypes)Stream.readRecord(Entry.ID, Record)) {
+    case SIGNATURE: {
+      if (F)
+        std::copy(Record.begin(), Record.end(), F->Signature.data());
+      break;
+    }
+    case DIAGNOSTIC_OPTIONS: {
+      bool Complain = (ClientLoadCapabilities & ARR_OutOfDate) == 0;
+      if (Listener && ValidateDiagnosticOptions &&
+          !AllowCompatibleConfigurationMismatch &&
+          ParseDiagnosticOptions(Record, Complain, *Listener))
+        return OutOfDate;
+      break;
+    }
+    }
+  }
+}
+
 /// Parse a record and blob containing module file extension metadata.
 static bool parseModuleFileExtensionMetadata(
               const SmallVectorImpl<uint64_t> &Record,
               StringRef Blob,
               ModuleFileExtensionMetadata &Metadata) {
   if (Record.size() < 4) return true;
 
   Metadata.MajorVersion = Record[0];
void ASTReader::finalizeForWriting() {
   // Nothing to do for now.
 }
 
 /// \brief Reads and return the signature record from \p PCH's control block, or
 /// else returns 0.
 static ASTFileSignature readASTFileSignature(StringRef PCH) {
   BitstreamCursor Stream(PCH);
   if (!startsWithASTFileMagic(Stream))
-    return 0;
+    return ASTFileSignature();
 
-  // Scan for the CONTROL_BLOCK_ID block.
-  if (SkipCursorToBlock(Stream, CONTROL_BLOCK_ID))
-    return 0;
+  // Scan for the UNHASHED_CONTROL_BLOCK_ID block.
+  if (SkipCursorToBlock(Stream, UNHASHED_CONTROL_BLOCK_ID))
+    return ASTFileSignature();
 
-  // Scan for SIGNATURE inside the control block.
+  // Scan for SIGNATURE inside the diagnostic options block.
   ASTReader::RecordData Record;
   while (true) {
     llvm::BitstreamEntry Entry = Stream.advanceSkippingSubblocks();
     if (Entry.Kind != llvm::BitstreamEntry::Record)
-      return 0;
+      return ASTFileSignature();
 
     Record.clear();
     StringRef Blob;
     if (SIGNATURE == Stream.readRecord(Entry.ID, Record, &Blob))
-      return Record[0];
+      return {{{(uint32_t)Record[0], (uint32_t)Record[1], (uint32_t)Record[2],
+                (uint32_t)Record[3], (uint32_t)Record[4]}}};
   }
 }
 
 /// \brief Retrieve the name of the original source file name
 /// directly from the AST file, without actually loading the AST
 /// file.
 std::string ASTReader::getOriginalSourceFile(
     const std::string &ASTFileName, FileManager &FileMgr,
bool ASTReader::readASTFileControlBlock(
   // FIXME: This allows use of the VFS; we do not allow use of the
   // VFS when actually loading a module.
   auto Buffer = FileMgr.getBufferForFile(Filename);
   if (!Buffer) {
     return true;
   }
 
   // Initialize the stream
-  BitstreamCursor Stream(PCHContainerRdr.ExtractPCH(**Buffer));
+  StringRef Bytes = PCHContainerRdr.ExtractPCH(**Buffer);
+  BitstreamCursor Stream(Bytes);
 
   // Sniff for the signature.
   if (!startsWithASTFileMagic(Stream))
     return true;
 
   // Scan for the CONTROL_BLOCK_ID block.
   if (SkipCursorToBlock(Stream, CONTROL_BLOCK_ID))
     return true;
while (!DoneWithControlBlock) {
 
     switch (Entry.Kind) {
     case llvm::BitstreamEntry::SubBlock: {
       switch (Entry.ID) {
       case OPTIONS_BLOCK_ID: {
         std::string IgnoredSuggestedPredefines;
         if (ReadOptionsBlock(Stream, ARR_ConfigurationMismatch | ARR_OutOfDate,
                              /*AllowCompatibleConfigurationMismatch*/ false,
-                             Listener, IgnoredSuggestedPredefines,
-                             ValidateDiagnosticOptions) != Success)
+                             Listener, IgnoredSuggestedPredefines) != Success)
           return true;
         break;
       }
 
       case INPUT_FILES_BLOCK_ID:
         InputFilesCursor = Stream;
         if (Stream.SkipBlock() ||
             (NeedsInputFiles &&
while (!DoneWithControlBlock) {
     default:
       // No other validation to perform.
       break;
     }
   }
 
   // Look for module file extension blocks, if requested.
   if (FindModuleFileExtensions) {
+    BitstreamCursor SavedStream = Stream;
     while (!SkipCursorToBlock(Stream, EXTENSION_BLOCK_ID)) {
       bool DoneWithExtensionBlock = false;
       while (!DoneWithExtensionBlock) {
        llvm::BitstreamEntry Entry = Stream.advance();
 
        switch (Entry.Kind) {
        case llvm::BitstreamEntry::SubBlock:
          if (Stream.SkipBlock())
while (!SkipCursorToBlock(Stream, EXTENSION_BLOCK_ID)) {
            return true;
 
          Listener.readModuleFileExtension(Metadata);
          break;
        }
        }
       }
     }
+    Stream = SavedStream;
   }
 
+  // Scan for the UNHASHED_CONTROL_BLOCK_ID block.
+  if (readUnhashedControlBlockImpl(
+          nullptr, Bytes, ARR_ConfigurationMismatch | ARR_OutOfDate,
+          /*AllowCompatibleConfigurationMismatch*/ false, &Listener,
+          ValidateDiagnosticOptions) != Success)
+    return true;
+
   return false;
 }
 
 bool ASTReader::isAcceptableASTFile(
     StringRef Filename, FileManager &FileMgr,
     const PCHContainerReader &PCHContainerRdr, const LangOptions &LangOpts,
     const TargetOptions &TargetOpts, const PreprocessorOptions &PPOpts,
     std::string ExistingModuleCachePath) {

clang/lib/Serialization/ASTWriter.cpp

 
 #include "clang/Serialization/ASTWriter.h"
 #include "ASTCommon.h"
 #include "ASTReaderInternals.h"
 #include "MultiOnDiskHashTable.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/ASTUnresolvedSet.h"
 #include "clang/AST/Decl.h"
-#include "clang/AST/DeclContextInternals.h"
 #include "clang/AST/DeclCXX.h"
+#include "clang/AST/DeclContextInternals.h"
 #include "clang/AST/DeclFriend.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/Expr.h"
 #include "clang/AST/ExprCXX.h"
 #include "clang/AST/LambdaCapture.h"
 #include "clang/AST/NestedNameSpecifier.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/AST/TemplateName.h"
 #include "clang/AST/Type.h"
 #include "clang/AST/TypeLocVisitor.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/FileSystemOptions.h"
-#include "clang/Basic/LangOptions.h"
 #include "clang/Basic/LLVM.h"
+#include "clang/Basic/LangOptions.h"
 #include "clang/Basic/Module.h"
 #include "clang/Basic/ObjCRuntime.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/SourceManagerInternals.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Basic/Version.h"
 #include "clang/Basic/VersionTuple.h"

 #include "clang/Serialization/Module.h"
 #include "clang/Serialization/ModuleFileExtension.h"
 #include "clang/Serialization/SerializationDiagnostic.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallString.h"
-#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Bitcode/BitCodes.h"
 #include "llvm/Bitcode/BitstreamWriter.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/EndianStream.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
+#include "llvm/Support/SHA1.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <cstdlib>
 #include <cstring>
 #include <deque>
 #include <limits>
void ASTWriter::WriteBlockInfoBlock() {
   Stream.EnterBlockInfoBlock();
 
 #define BLOCK(X) EmitBlockID(X ## _ID, #X, Stream, Record)
 #define RECORD(X) EmitRecordID(X, #X, Stream, Record)
 
   // Control Block.
   BLOCK(CONTROL_BLOCK);
   RECORD(METADATA);
-  RECORD(SIGNATURE);
   RECORD(MODULE_NAME);
   RECORD(MODULE_DIRECTORY);
   RECORD(MODULE_MAP_FILE);
   RECORD(IMPORTS);
   RECORD(ORIGINAL_FILE);
   RECORD(ORIGINAL_PCH_DIR);
   RECORD(ORIGINAL_FILE_ID);
   RECORD(INPUT_FILE_OFFSETS);
 
   BLOCK(OPTIONS_BLOCK);
   RECORD(LANGUAGE_OPTIONS);
   RECORD(TARGET_OPTIONS);
-  RECORD(DIAGNOSTIC_OPTIONS);
   RECORD(FILE_SYSTEM_OPTIONS);
   RECORD(HEADER_SEARCH_OPTIONS);
   RECORD(PREPROCESSOR_OPTIONS);
 
   BLOCK(INPUT_FILES_BLOCK);
   RECORD(INPUT_FILE);
 
   // AST Top-Level Block.
#define RECORD(X) EmitRecordID(X, #X, Stream, Record)
   RECORD(PPD_MACRO_EXPANSION);
   RECORD(PPD_MACRO_DEFINITION);
   RECORD(PPD_INCLUSION_DIRECTIVE);
 
   // Decls and Types block.
   BLOCK(EXTENSION_BLOCK);
   RECORD(EXTENSION_METADATA);
 
+  BLOCK(UNHASHED_CONTROL_BLOCK);
+  RECORD(SIGNATURE);
+  RECORD(DIAGNOSTIC_OPTIONS);
+
 #undef RECORD
 #undef BLOCK
   Stream.ExitBlock();
 }
 
 /// \brief Prepares a path for being written to an AST file by converting it
 /// to an absolute path and removing nested './'s.
 ///
if (!llvm::sys::path::is_separator(Filename[Pos])) {
     // in a row. Use a better mechanism to indicate whether we have emitted an
     // absolute or relative path.
     ++Pos;
   }
 
   return Filename + Pos;
 }
 
-static ASTFileSignature getSignature() {
-  while (true) {
-    if (ASTFileSignature S = llvm::sys::Process::GetRandomNumber())
-      return S;
-    // Rely on GetRandomNumber to eventually return non-zero...
-  }
+ASTFileSignature ASTWriter::createSignature(StringRef Bytes) {
+  // Calculate the hash till start of UNHASHED_CONTROL_BLOCK.
+  llvm::SHA1 Hasher;
+  Hasher.update(ArrayRef<uint8_t>(Bytes.bytes_begin(), Bytes.size()));
+  auto Hash = Hasher.result();
+
+  // Convert to an array [5*i32].
+  ASTFileSignature Signature;
+  auto LShift = [&](unsigned char Val, unsigned Shift) {
+    return (uint32_t)Val << Shift;
+  };
+  for (int I = 0; I != 5; ++I)
+    Signature[I] = LShift(Hash[I * 4 + 0], 24) | LShift(Hash[I * 4 + 1], 16) |
+                   LShift(Hash[I * 4 + 2], 8) | LShift(Hash[I * 4 + 3], 0);
+
+  return Signature;
+
+ASTFileSignature ASTWriter::writeUnhashedControlBlock(Preprocessor &PP,
+                                                      ASTContext &Context) {
+  // Flush first to prepare the PCM hash (signature).
+  Stream.FlushToWord();
+  auto StartOfUnhashedControl = Stream.GetCurrentBitNo() >> 3;
+
+  // Enter the block and prepare to write records.
+  RecordData Record;
+  Stream.EnterSubblock(UNHASHED_CONTROL_BLOCK_ID, 5);
+
+  // For implicit modules, write the hash of the PCM as its signature.
+  ASTFileSignature Signature;
+  if (WritingModule &&
+      PP.getHeaderSearchInfo().getHeaderSearchOpts().ModulesHashContent) {
+    Signature = createSignature(StringRef(Buffer.begin(), StartOfUnhashedControl));
+    Record.append(Signature.begin(), Signature.end());
+    Stream.EmitRecord(SIGNATURE, Record);
+    Record.clear();
+  }
+
+  // Diagnostic options.
+  const DiagnosticOptions &DiagOpts =
+      Context.getDiagnostics().getDiagnosticOptions();
+#define DIAGOPT(Name, Bits, Default) Record.push_back(DiagOpts.Name);
+#define ENUM_DIAGOPT(Name, Type, Bits, Default)                                \
+  Record.push_back(static_cast<unsigned>(DiagOpts.get##Name()));
+#include "clang/Basic/DiagnosticOptions.def"
+  Record.push_back(DiagOpts.Warnings.size());
+  for (unsigned I = 0, N = DiagOpts.Warnings.size(); I != N; ++I)
+    AddString(DiagOpts.Warnings[I], Record);
+  Record.push_back(DiagOpts.Remarks.size());
+  for (unsigned I = 0, N = DiagOpts.Remarks.size(); I != N; ++I)
+    AddString(DiagOpts.Remarks[I], Record);
+  // Note: we don't serialize the log or serialization file names, because they
+  // are generally transient files and will almost always be overridden.
+  Stream.EmitRecord(DIAGNOSTIC_OPTIONS, Record);
+
+  // Leave the options block.
+  Stream.ExitBlock();
+  return Signature;
 }
 
 /// \brief Write the control block.
-uint64_t ASTWriter::WriteControlBlock(Preprocessor &PP,
-                                      ASTContext &Context,
-                                      StringRef isysroot,
-                                      const std::string &OutputFile) {
-  ASTFileSignature Signature = 0;
-
+void ASTWriter::WriteControlBlock(Preprocessor &PP, ASTContext &Context,
   using namespace llvm;
   Stream.EnterSubblock(CONTROL_BLOCK_ID, 5);
   RecordData Record;
   
   // Metadata
   auto MetadataAbbrev = std::make_shared<BitCodeAbbrev>();
   MetadataAbbrev->Add(BitCodeAbbrevOp(METADATA));
   MetadataAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 16)); // Major
assert((!WritingModule || isysroot.empty()) &&
     RecordData::value_type Record[] = {METADATA, VERSION_MAJOR, VERSION_MINOR,
                                        CLANG_VERSION_MAJOR, CLANG_VERSION_MINOR,
                                        !isysroot.empty(), IncludeTimestamps,
                                        ASTHasCompilerErrors};
     Stream.EmitRecordWithBlob(MetadataAbbrevCode, Record,
                               getClangFullRepositoryVersion());
   }
   if (WritingModule) {
-    // For implicit modules we output a signature that we can use to ensure
-    // duplicate module builds don't collide in the cache as their output order
-    // is non-deterministic.
-    // FIXME: Remove this when output is deterministic.
-    if (Context.getLangOpts().ImplicitModules) {
-      Signature = getSignature();
-      RecordData::value_type Record[] = {Signature};
-      Stream.EmitRecord(SIGNATURE, Record);
-    }
 
     // Module name
     auto Abbrev = std::make_shared<BitCodeAbbrev>();
     Abbrev->Add(BitCodeAbbrevOp(MODULE_NAME));
     Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
     unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
     RecordData::value_type Record[] = {MODULE_NAME};
     Stream.EmitRecordWithBlob(AbbrevCode, Record, WritingModule->Name);
if (Chain) {
 
     for (ModuleFile &M : Mgr) {
       // Skip modules that weren't directly imported.
       if (!M.isDirectlyImported())
         continue;
 
       Record.push_back((unsigned)M.Kind); // FIXME: Stable encoding
       AddSourceLocation(M.ImportLoc, Record);
-      Record.push_back(M.File->getSize());
-      Record.push_back(getTimestampForOutput(M.File));
-      Record.push_back(M.Signature);
+
+      // If we have calculated signature, there is no need to store
+      // the size or timestamp.
+      Record.push_back(M.Signature ? 0 : M.File->getSize());
+      Record.push_back(M.Signature ? 0 : getTimestampForOutput(M.File));
+
+      for (auto I : M.Signature)
+        Record.push_back(I);
+
       AddPath(M.FileName, Record);
     }
     Stream.EmitRecord(IMPORTS, Record);
   }
 
   // Write the options block.
   Stream.EnterSubblock(OPTIONS_BLOCK_ID, 4);
 
for (unsigned I = 0, N = TargetOpts.FeaturesAsWritten.size(); I != N; ++I) {
     AddString(TargetOpts.FeaturesAsWritten[I], Record);
   }
   Record.push_back(TargetOpts.Features.size());
   for (unsigned I = 0, N = TargetOpts.Features.size(); I != N; ++I) {
     AddString(TargetOpts.Features[I], Record);
   }
   Stream.EmitRecord(TARGET_OPTIONS, Record);
 
-  // Diagnostic options.
-  Record.clear();
-  const DiagnosticOptions &DiagOpts
-    = Context.getDiagnostics().getDiagnosticOptions();
-#define DIAGOPT(Name, Bits, Default) Record.push_back(DiagOpts.Name);
-#define ENUM_DIAGOPT(Name, Type, Bits, Default) \
-  Record.push_back(static_cast<unsigned>(DiagOpts.get##Name()));
-#include "clang/Basic/DiagnosticOptions.def"
-  Record.push_back(DiagOpts.Warnings.size());
-  for (unsigned I = 0, N = DiagOpts.Warnings.size(); I != N; ++I)
-    AddString(DiagOpts.Warnings[I], Record);
-  Record.push_back(DiagOpts.Remarks.size());
-  for (unsigned I = 0, N = DiagOpts.Remarks.size(); I != N; ++I)
-    AddString(DiagOpts.Remarks[I], Record);
-  // Note: we don't serialize the log or serialization file names, because they
-  // are generally transient files and will almost always be overridden.
-  Stream.EmitRecord(DIAGNOSTIC_OPTIONS, Record);
-
   // File system options.
   Record.clear();
   const FileSystemOptions &FSOpts =
       Context.getSourceManager().getFileManager().getFileSystemOpts();
   AddString(FSOpts.WorkingDir, Record);
   Stream.EmitRecord(FILE_SYSTEM_OPTIONS, Record);
 
   // Header search options.
if (!OutputFile.empty() && OutputFile != "-") {
     RecordData::value_type Record[] = {ORIGINAL_PCH_DIR};
     Stream.EmitRecordWithBlob(AbbrevCode, Record, origDir);
   }
 
   WriteInputFiles(Context.SourceMgr,
                   PP.getHeaderSearchInfo().getHeaderSearchOpts(),
                   PP.getLangOpts().Modules);
   Stream.ExitBlock();
-  return Signature;
 }
 
 namespace  {
 
   /// \brief An input file.
   struct InputFileEntry {
     const FileEntry *File;
     bool IsSystemFile;
void ASTWriter::SetSelectorOffset(Selector Sel, uint32_t Offset) {
   // Don't record offsets for selectors that are also available in a different
   // file.
   if (ID < FirstSelectorID)
     return;
   SelectorOffsets[ID - FirstSelectorID] = Offset;
 }
 
 ASTWriter::ASTWriter(llvm::BitstreamWriter &Stream,
+                     SmallVectorImpl<char> &Buffer,
                      ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
                      bool IncludeTimestamps)
-    : Stream(Stream), IncludeTimestamps(IncludeTimestamps) {
+    : Stream(Stream), Buffer(Buffer), IncludeTimestamps(IncludeTimestamps) {
   for (const auto &Ext : Extensions) {
     if (auto Writer = Ext->createExtensionWriter(*this))
       ModuleFileExtensionWriters.push_back(std::move(Writer));
   }
 }
 
 ASTWriter::~ASTWriter() {
   llvm::DeleteContainerSeconds(FileDeclIDs);
 }
 
 const LangOptions &ASTWriter::getLangOpts() const {
   assert(WritingAST && "can't determine lang opts when not writing AST");
   return Context->getLangOpts();
 }
 
 time_t ASTWriter::getTimestampForOutput(const FileEntry *E) const {
   return IncludeTimestamps ? E->getModificationTime() : 0;
 }
 
-uint64_t ASTWriter::WriteAST(Sema &SemaRef, const std::string &OutputFile,
+ASTFileSignature ASTWriter::WriteAST(Sema &SemaRef,
+                                     const std::string &OutputFile,
                                      Module *WritingModule, StringRef isysroot,
                                      bool hasErrors) {
   WritingAST = true;
 
   ASTHasCompilerErrors = hasErrors;
   
   // Emit the file header.
   Stream.Emit((unsigned)'C', 8);
   Stream.Emit((unsigned)'P', 8);
   Stream.Emit((unsigned)'C', 8);

 static void AddLazyVectorDecls(ASTWriter &Writer, Vector &Vec,
                                ASTWriter::RecordData &Record) {
   for (typename Vector::iterator I = Vec.begin(nullptr, true), E = Vec.end();
        I != E; ++I) {
     Writer.AddDeclRef(*I, Record);
   }
 }
 
-uint64_t ASTWriter::WriteASTCore(Sema &SemaRef, StringRef isysroot,
+ASTFileSignature ASTWriter::WriteASTCore(Sema &SemaRef, StringRef isysroot,
                                          const std::string &OutputFile,
                                          Module *WritingModule) {
   using namespace llvm;
 
   bool isModule = WritingModule != nullptr;
 
   // Make sure that the AST reader knows to finalize itself.
   if (Chain)
     Chain->finalizeForWriting();
   
for (const auto &DeleteExprsInfo :
     DeleteExprsToAnalyze.push_back(DeleteExprsInfo.second.size());
     for (const auto &DeleteLoc : DeleteExprsInfo.second) {
       AddSourceLocation(DeleteLoc.first, DeleteExprsToAnalyze);
       DeleteExprsToAnalyze.push_back(DeleteLoc.second);
     }
   }
 
   // Write the control block
-  uint64_t Signature = WriteControlBlock(PP, Context, isysroot, OutputFile);
+  WriteControlBlock(PP, Context, isysroot, OutputFile);
 
   // Write the remaining AST contents.
   Stream.EnterSubblock(AST_BLOCK_ID, 5);
 
   // This is so that older clang versions, before the introduction
   // of the control block, can read and reject the newer PCH format.
   {
     RecordData Record = {VERSION_MAJOR};
RecordData::value_type Record[] = {
       NumStatements, NumMacros, NumLexicalDeclContexts, NumVisibleDeclContexts};
   Stream.EmitRecord(STATISTICS, Record);
   Stream.ExitBlock();
 
   // Write the module file extension blocks.
   for (const auto &ExtWriter : ModuleFileExtensionWriters)
     WriteModuleFileExtension(SemaRef, *ExtWriter);
 
-  return Signature;
+  return writeUnhashedControlBlock(PP, Context);
 }
 
 void ASTWriter::WriteDeclUpdatesBlocks(RecordDataImpl &OffsetsRecord) {
   if (DeclUpdates.empty())
     return;
 
   DeclUpdateMap LocalUpdates;
   LocalUpdates.swap(DeclUpdates);

clang/lib/Serialization/GeneratePCH.cpp

 
 PCHGenerator::PCHGenerator(
     const Preprocessor &PP, StringRef OutputFile, StringRef isysroot,
     std::shared_ptr<PCHBuffer> Buffer,
     ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
     bool AllowASTWithErrors, bool IncludeTimestamps)
     : PP(PP), OutputFile(OutputFile), isysroot(isysroot.str()),
       SemaPtr(nullptr), Buffer(Buffer), Stream(Buffer->Data),
-      Writer(Stream, Extensions, IncludeTimestamps),
+      Writer(Stream, Buffer->Data, Extensions, IncludeTimestamps),
       AllowASTWithErrors(AllowASTWithErrors) {
   Buffer->IsComplete = false;
 }
 
 PCHGenerator::~PCHGenerator() {
 }
 
 void PCHGenerator::HandleTranslationUnit(ASTContext &Ctx) {

clang/lib/Serialization/GlobalModuleIndex.cpp

namespace {
   /// \brief Provides information about a specific module file.
   struct ModuleFileInfo {
     /// \brief The numberic ID for this module file.
     unsigned ID;
 
     /// \brief The set of modules on which this module depends. Each entry is
     /// a module ID.
     SmallVector<unsigned, 4> Dependencies;
+    ASTFileSignature Signature;
+  };
+
+  struct ImportedModuleFileInfo {
+    off_t StoredSize;
+    time_t StoredModTime;
+    ASTFileSignature StoredSignature;
+    ImportedModuleFileInfo(off_t Size, time_t ModTime, ASTFileSignature Sig)
+        : StoredSize(Size), StoredModTime(ModTime), StoredSignature(Sig) {}
   };
 
   /// \brief Builder that generates the global module index file.
   class GlobalModuleIndexBuilder {
     FileManager &FileMgr;
     const PCHContainerReader &PCHContainerRdr;
 
-    /// \brief Mapping from files to module file information.
+    /// Mapping from files to module file information.
     typedef llvm::MapVector<const FileEntry *, ModuleFileInfo> ModuleFilesMap;
 
-    /// \brief Information about each of the known module files.
+    /// Information about each of the known module files.
     ModuleFilesMap ModuleFiles;
 
+    /// \brief Mapping from the imported module file to the imported
+    /// information.
+    typedef std::multimap<const FileEntry *, ImportedModuleFileInfo>
+        ImportedModuleFilesMap;
+
+    /// \brief Information about each importing of a module file.
+    ImportedModuleFilesMap ImportedModuleFiles;
+
     /// \brief Mapping from identifiers to the list of module file IDs that
     /// consider this identifier to be interesting.
     typedef llvm::StringMap<SmallVector<unsigned, 2> > InterestingIdentifierMap;
 
     /// \brief A mapping from all interesting identifiers to the set of module
     /// files in which those identifiers are considered interesting.
     InterestingIdentifierMap InterestingIdentifiers;
     
explicit GlobalModuleIndexBuilder(
         : FileMgr(FileMgr), PCHContainerRdr(PCHContainerRdr) {}
 
     /// \brief Load the contents of the given module file into the builder.
     ///
     /// \returns true if an error occurred, false otherwise.
     bool loadModuleFile(const FileEntry *File);
 
     /// \brief Write the index to the given bitstream.
-    void writeIndex(llvm::BitstreamWriter &Stream);
+    /// \returns true if an error occurred, false otherwise.
+    bool writeIndex(llvm::BitstreamWriter &Stream);
   };
 }
 
 static void emitBlockID(unsigned ID, const char *Name,
                         llvm::BitstreamWriter &Stream,
                         SmallVectorImpl<uint64_t> &Record) {
   Record.clear();
   Record.push_back(ID);
if (InStream.Read(8) != 'C' ||
     return true;
   }
 
   // Record this module file and assign it a unique ID (if it doesn't have
   // one already).
   unsigned ID = getModuleFileInfo(File).ID;
 
   // Search for the blocks and records we care about.
-  enum { Other, ControlBlock, ASTBlock } State = Other;
+  enum { Other, ControlBlock, ASTBlock, DiagnosticOptionsBlock } State = Other;
   bool Done = false;
   while (!Done) {
     llvm::BitstreamEntry Entry = InStream.advance();
     switch (Entry.Kind) {
     case llvm::BitstreamEntry::Error:
       Done = true;
       continue;
 
case llvm::BitstreamEntry::SubBlock:
         if (InStream.EnterSubBlock(AST_BLOCK_ID))
           return true;
 
         // Found the AST block.
         State = ASTBlock;
         continue;
       }
 
+      if (Entry.ID == UNHASHED_CONTROL_BLOCK_ID) {
+        if (InStream.EnterSubBlock(UNHASHED_CONTROL_BLOCK_ID))
+          return true;
+
+        // Found the Diagnostic Options block.
+        State = DiagnosticOptionsBlock;
+        continue;
+      }
+
       if (InStream.SkipBlock())
         return true;
 
       continue;
 
     case llvm::BitstreamEntry::EndBlock:
       State = Other;
       continue;
if (State == ControlBlock && Code == IMPORTS) {
         ++Idx;
 
         // Load stored size/modification time. 
         off_t StoredSize = (off_t)Record[Idx++];
         time_t StoredModTime = (time_t)Record[Idx++];
 
         // Skip the stored signature.
         // FIXME: we could read the signature out of the import and validate it.
-        Idx++;
+        ASTFileSignature StoredSignature = {
+            {{(uint32_t)Record[Idx++], (uint32_t)Record[Idx++],
+              (uint32_t)Record[Idx++], (uint32_t)Record[Idx++],
+              (uint32_t)Record[Idx++]}}};
 
         // Retrieve the imported file name.
         unsigned Length = Record[Idx++];
         SmallString<128> ImportedFile(Record.begin() + Idx,
                                       Record.begin() + Idx + Length);
         Idx += Length;
 
         // Find the imported module file.
         const FileEntry *DependsOnFile
           = FileMgr.getFile(ImportedFile, /*openFile=*/false,
                             /*cacheFailure=*/false);
-        if (!DependsOnFile ||
-            (StoredSize != DependsOnFile->getSize()) ||
-            (StoredModTime != DependsOnFile->getModificationTime()))
+
+        if (!DependsOnFile)
           return true;
 
+        // Save the information in ImportedModuleFileInfo so we can verify after
+        // loading all pcms.
+        ImportedModuleFiles.insert(std::make_pair(
+            DependsOnFile, ImportedModuleFileInfo(StoredSize, StoredModTime,
+                                                  StoredSignature)));
+
         // Record the dependency.
         unsigned DependsOnID = getModuleFileInfo(DependsOnFile).ID;
         getModuleFileInfo(File).Dependencies.push_back(DependsOnID);
       }
 
       continue;
     }
 
if (State == ASTBlock && Code == IDENTIFIER_TABLE && Record[0] > 0) {
         std::pair<StringRef, bool> Ident = *D;
         if (Ident.second)
           InterestingIdentifiers[Ident.first].push_back(ID);
         else
           (void)InterestingIdentifiers[Ident.first];
       }
     }
 
+    // Get Signature.
+    if (State == DiagnosticOptionsBlock && Code == SIGNATURE)
+      getModuleFileInfo(File).Signature = {
+          {{(uint32_t)Record[0], (uint32_t)Record[1], (uint32_t)Record[2],
+            (uint32_t)Record[3], (uint32_t)Record[4]}}};
+
     // We don't care about this record.
   }
 
   return false;
 }
 
 namespace {
 
void EmitData(raw_ostream& Out, key_type_ref Key, data_type_ref Data,
     using namespace llvm::support;
     for (unsigned I = 0, N = Data.size(); I != N; ++I)
       endian::Writer<little>(Out).write<uint32_t>(Data[I]);
   }
 };
 
 }
 
-void GlobalModuleIndexBuilder::writeIndex(llvm::BitstreamWriter &Stream) {
+bool GlobalModuleIndexBuilder::writeIndex(llvm::BitstreamWriter &Stream) {
+  for (auto MapEntry : ImportedModuleFiles) {
+    auto *File = MapEntry.first;
+    ImportedModuleFileInfo &Info = MapEntry.second;
+    if (getModuleFileInfo(File).Signature) {
+      if (getModuleFileInfo(File).Signature != Info.StoredSignature)
+        // Verify Signature.
+        return true;
+    } else if (Info.StoredSize != File->getSize() ||
+               Info.StoredModTime != File->getModificationTime())
+      // Verify Size and ModTime.
+      return true;
+  }
+
   using namespace llvm;
   
   // Emit the file header.
   Stream.Emit((unsigned)'B', 8);
   Stream.Emit((unsigned)'C', 8);
   Stream.Emit((unsigned)'G', 8);
   Stream.Emit((unsigned)'I', 8);
 
// Write the identifier -> module file mapping.
     unsigned IDTableAbbrev = Stream.EmitAbbrev(std::move(Abbrev));
 
     // Write the identifier table
     uint64_t Record[] = {IDENTIFIER_INDEX, BucketOffset};
     Stream.EmitRecordWithBlob(IDTableAbbrev, Record, IdentifierTable);
   }
 
   Stream.ExitBlock();
+  return false;
 }
 
 GlobalModuleIndex::ErrorCode
 GlobalModuleIndex::writeIndex(FileManager &FileMgr,
                               const PCHContainerReader &PCHContainerRdr,
                               StringRef Path) {
   llvm::SmallString<128> IndexPath;
   IndexPath += Path;
for (llvm::sys::fs::directory_iterator D(Path, EC), DEnd;
     if (Builder.loadModuleFile(ModuleFile))
       return EC_IOError;
   }
 
   // The output buffer, into which the global index will be written.
   SmallVector<char, 16> OutputBuffer;
   {
     llvm::BitstreamWriter OutputStream(OutputBuffer);
-    Builder.writeIndex(OutputStream);
+    if (Builder.writeIndex(OutputStream))
+      return EC_IOError;
   }
 
   // Write the global index file to a temporary file.
   llvm::SmallString<128> IndexTmpPath;
   int TmpFD;
   if (llvm::sys::fs::createUniqueFile(IndexPath + "-%%%%%%%%", TmpFD,
                                       IndexTmpPath))
     return EC_IOError;

clang/test/Modules/diagnostic-options-out-of-date.m

 // RUN: rm -rf %t
 // Build A.pcm
 // RUN: %clang_cc1 -Werror -Wno-conversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs %s -fmodules-disable-diagnostic-validation
 // Build pch that imports A.pcm
 // RUN: %clang_cc1 -Werror -Wno-conversion -emit-pch -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -o %t.pch -I %S/Inputs -x objective-c-header %S/Inputs/pch-import-module-out-of-date.pch -fmodules-disable-diagnostic-validation
 // Make sure that we don't rebuild A.pcm and overwrite the original A.pcm that the pch imports
 // RUN: %clang_cc1 -Werror -Wconversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs %s -fmodules-disable-diagnostic-validation
 // Make sure we don't error out when using the pch
 // RUN: %clang_cc1 -Werror -Wno-conversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -fsyntax-only -I %S/Inputs -include-pch %t.pch %s -verify -fmodules-disable-diagnostic-validation
+
+// Build A.pcm
+// RUN: %clang_cc1 -Werror -Wno-conversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs %s
+// Build pch that imports A.pcm
+// RUN: %clang_cc1 -Werror -Wno-conversion -emit-pch -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -o %t.pch -I %S/Inputs -x objective-c-header %S/Inputs/pch-import-module-out-of-date.pch
+// We will rebuild A.pcm and overwrite the original A.pcm that the pch imports, but the two versions have the same hash.
+// RUN: %clang_cc1 -Werror -Wconversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -I %S/Inputs %s
+// Make sure we don't error out when using the pch
+// RUN: %clang_cc1 -Werror -Wno-conversion -fmodules-cache-path=%t -fmodules -fimplicit-module-maps -fsyntax-only -I %S/Inputs -include-pch %t.pch %s -verify
+
 // expected-no-diagnostics
 
 @import DiagOutOfDate;

clang/test/Modules/module_file_info.m

 // CHECK:   Module features:
 // CHECK:     myfeature
 
 // CHECK: Target options:
 // CHECK:     Triple:
 // CHECK:     CPU:
 // CHECK:     ABI:
 
-// CHECK: Diagnostic options:
-// CHECK:   IgnoreWarnings: Yes
-// CHECK:   Diagnostic flags:
-// CHECK:     -Wunused
-
 // CHECK: Header search options:
 // CHECK:   System root [-isysroot=]: '/'
 // CHECK:   Use builtin include directories [-nobuiltininc]: Yes
 // CHECK:   Use standard system include directories [-nostdinc]: No
 // CHECK:   Use standard C++ include directories [-nostdinc++]: Yes
 // CHECK:   Use libc++ (rather than libstdc++) [-stdlib=]:
 
 // CHECK: Preprocessor options:
 // CHECK:   Uses compiler/target-specific predefines [-undef]: Yes
 // CHECK:   Uses detailed preprocessing record (for indexing): No
 // CHECK:   Predefined macros:
 // CHECK:     -DBLARG
 // CHECK:     -DWIBBLE=WOBBLE
+
+// CHECK: Diagnostic options:
+// CHECK:   IgnoreWarnings: Yes
+// CHECK:   Diagnostic flags:
+// CHECK:     -Wunused

clang/test/Modules/rebuild.m

 // Rebuild Module, reset its timestamp, and verify its size hasn't changed
 // RUN: rm %t/Module.pcm
 // RUN: echo '@import Module;' | %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -fdisable-module-hash -fsyntax-only -F %S/Inputs -x objective-c -
 // RUN: touch -m -a -t 201101010000 %t/Module.pcm
 // RUN: wc -c %t/Module.pcm > %t/Module.size
 // RUN: diff %t/Module.size %t/Module.size.saved
 // RUN: cp %t/Module.pcm %t/Module.pcm.saved.2
 
-// But the signature at least is expected to change, so we rebuild DependsOnModule.
-// NOTE: if we change how the signature is created, this test may need updating.
+// The signature is the hash of the PCM content, we will not rebuild rebuild DependsOnModule.
 // RUN: %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -fdisable-module-hash -fsyntax-only -F %S/Inputs %s
 // RUN: diff %t/Module.pcm %t/Module.pcm.saved.2
-// RUN: not diff %t/DependsOnModule.pcm %t/DependsOnModule.pcm.saved
+// RUN: diff %t/DependsOnModule.pcm %t/DependsOnModule.pcm.saved
 
 // Rebuild Module, reset its timestamp, and verify its size hasn't changed
 // RUN: rm %t/Module.pcm
 // RUN: echo '@import Module;' | %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -fdisable-module-hash -fsyntax-only -F %S/Inputs -x objective-c -
 // RUN: touch -m -a -t 201101010000 %t/Module.pcm
 // RUN: wc -c %t/Module.pcm > %t/Module.size
 // RUN: diff %t/Module.size %t/Module.size.saved
 // RUN: cp %t/Module.pcm %t/Module.pcm.saved.2
 
 // Verify again with Module pre-imported.
-// NOTE: if we change how the signature is created, this test may need updating.
 // RUN: %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -fdisable-module-hash -fsyntax-only -F %S/Inputs %s
 // RUN: diff %t/Module.pcm %t/Module.pcm.saved.2
-// RUN: not diff %t/DependsOnModule.pcm %t/DependsOnModule.pcm.saved
+// RUN: diff %t/DependsOnModule.pcm %t/DependsOnModule.pcm.saved
 
 #ifdef PREIMPORT
 @import Module;
 #endif
 @import DependsOnModule;