Add a module Hash in the bitcode and the combined index.

[mehdi_amini]

Summary

This is the equivalent of a "build-id". The module that is written is hashed on the fly, and the last record will be the hash itself.
It is intended to be used for ThinLTO incremental build, so it is included along with each module entry in the combined index. The function importer will compute the set of imports and where they are coming from. Then before the actual import, the hashes of all modules that we are about to import from are combined together to produce a single hash. This hash identifies uniquely the module we're about to create with the importing and thus can be used as a key for a cache supporting incremental build.

The SHA1 implementation is not very optimized, on the performance part WriteModule() takes 1.9s when disabled and 2.1s when enabled (writing LTO llvm-tblgen with full dbg-info).
Note that is it only performed when we emit the summary information, i.e. only with ThinLTO and only by the frontend. The link stage is just consuming this information.

[• rafael]

lib/Bitcode/Writer/BitcodeWriter.cpp

3175

This produces different results in 32 and 64 bit machines, no?

[mehdi_amini]

Disable by default...

[mehdi_amini]

lib/Bitcode/Writer/BitcodeWriter.cpp

3175

Uh, dunno, is it a property of hash_combine_range?

[mehdi_amini]

Minor bug fix (forgot to nullptr check)

[mehdi_amini]

Insert the full 160bits of the SHA1 in the bitcode. Also add a unittests.

[vsk]

Have you considered adding a textual-IR representation of the module hash?

I can think of two uses for it: 1) we can add -module-summary to the run line in compatibility.ll and get additional coverage, 2) we can debug any future hash-related issues by looking at textual IR.

[mehdi_amini]

I would be quite tricky to add the hash at the IR level, because it requires to generate bitcode to get a hash any time you want to print textual IR.

[mehdi_amini]

Add missing Inputs/ for the lit test.

[tejohnson]

Can we make generation of the hashes conditional? They aren't needed if a module cache is not being used for incremental rebuilds. I anticipate we will use a different mechanism with our distributed build system, for example. It is simple to suppress generation of the new MODULE_CODE_HASH record, but would be good to also suppress the larger module string table as well. Maybe have a separate set of record types for when we want to include the hashes?

include/llvm/IR/ModuleSummaryIndex.h

232

Why not use uint32_t? That seems to be what is used in SHA1.h.

307–308

No longer returning an iterator. Ditto for non-const method below.

352

Comment needs update: "with the given Hash". Also, is it possible to combine this with the above using a default argument value of {{0}}?

lib/Bitcode/Reader/BitcodeReader.cpp

5653

5*i32? Assert below checking for 5 values. And 32*5==160, so that seems like the right number. And update the description of this record accordingly in LLVMBitCodes.h where it just says [unsigned].

5922

Why is this one 6*i32 whereas the MODULE_CODE_HASH is only expected to have 5 i32 (according to implementation, comment above incorrectly says it is 6*i32)?

lib/Bitcode/Writer/BitcodeWriter.cpp

2831

What is the added VBR8 here and in the 6-bit case below? I don't see anything else being written. And it isn't in the 8-bit case above.

3173

5*i32?

3187

Please make this conditional so it isn't generating a hash if we aren't using a module cache.

3245

ditto.

test/Bitcode/Inputs/module_hash.ll

5 ↗

(On Diff #51257)

Fix?

test/Bitcode/module_hash.ll

6 ↗

(On Diff #51257)

Won't these values change frequently as the compiler changes and generates slightly different bitcode? How about just checking if the per module values match the combined index values via FileCheck variables/pattern matching?

20 ↗

(On Diff #51257)

Fix?

[llvm-commits]

We can make it conditional (with a clang flag?), but the desired behavior is to be able to decide on enabling/disabling the caching at link time, so any bitcode static archive distributed would need to have a hash embedded. The user friendly choice would be to enabled it by *default* when -flto=thing, and an explicit option to disable it.

[tejohnson]

In D18213#380319, @llvm-commits wrote:

We can make it conditional (with a clang flag?), but the desired behavior is to be able to decide on enabling/disabling the caching at link time, so any bitcode static archive distributed would need to have a hash embedded. The user friendly choice would be to enabled it by *default* when -flto=thing, and an explicit option to disable it.

I'm less concerned about the default than just having a way to disable it.

[mehdi_amini]

Update after Teresa's comments

[mehdi_amini]

Thanks for the review Teresa, I updated accordingly (but for the FileCheck, see comment below).

test/Bitcode/module_hash.ll

6 ↗

(On Diff #51257)

The hope is that a "void foo()" won't change much, but yeah it's absolutely not nice.
What you suggest is fine for testing that we correctly read and propagate to the combined index.
HoweverI'm not comfortable not checking a value, because we lose coverage for the production of the hash. I.e. writing only zero in the first place would make the test pass :(
Would it be enough to add a comment here saying that is fine updating the value is someone change the IR/bitcode?
I'm gonna CC Duncan for an opinion on this.

[mehdi_amini]

Typo

[mehdi_amini]

Forgot to remote the SHA1 for the MST_ENTRY abbrev.

[tejohnson]

include/llvm/IR/ModuleSummaryIndex.h

308–314

s/Table of module/Table of modules/ (here and below)

350

Now returns iterator not StringRef.

lib/Bitcode/Reader/BitcodeReader.cpp

5935

In order for this check to work after the first module id + hash is encountered, LastSeenModulePath needs to be set to the end() after it is used in the for loop below. Otherwise it will silently overwrite the earlier module path's values if not preceded by a new module path.

lib/Bitcode/Writer/BitcodeWriter.cpp

2846

"Optionally emitted after..."

[mehdi_amini]

Taking comments into account.

[tejohnson]

Needs test with new llvm-as option then fed through llvm-bcanalyzer to trigger new matching check.

lib/Bitcode/Reader/BitcodeReader.cpp

5943

s/unexpectingly/unexpectedly/

[mehdi_amini]

The test are checking for "match" in the output of llvm-bcanalyzer

[tejohnson]

In D18213#386848, @joker.eph wrote:

The test are checking for "match" in the output of llvm-bcanalyzer

Which test? Is there a new test missing from the patch? The hash generation is off by default so I assume you need to add a new one (or modify an existing) to pass the new -module-hash option to llvm-as. Otherwise I don't think the checking in llvm-bcanalyzer will ever kick in.

tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

539

Is the MODULE_CODE_HASH ever expected outside a module block? If not, can the first part of the if be turned into an assert inside the if body?

[mehdi_amini]

Add back the test, I lost it from the commit at some point

[mehdi_amini]

test/Bitcode/module_hash.ll

7 ↗

(On Diff #51672)

Came up with a different approach!

1. llvm-bcanalyzer will re-hash and check for mismatch
2. using a temporary file to collect the hash, I check that whatever is in the module is correctly propagated into the combined index.

tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

539

I think the code a reused across Blocks, so you need to know that you are in a module block to interpret the code as being the hash. Did I misunderstand your point?

[mehdi_amini]

s/unexpectingly/unexpectedly/

[tejohnson]

test/Bitcode/module_hash.ll

7 ↗

(On Diff #51672)

Weird, I still don't see the test when I go to http://reviews.llvm.org/D18213, only when I click on this comment which opens up http://reviews.llvm.org/D18213?id=51672#inline-155131.

In any case, this sounds good. The version of the test I am seeing doesn't implement it yet (assume that is WIP).

tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

539

Ok, got it, checking both does make sense.

[mehdi_amini]

Try to add the test, if it still does not work I'll make some coffee first...

[mehdi_amini]

I'm confident this one will work...

[tejohnson]

In D18213#386957, @joker.eph wrote:

I'm confident this one will work...

Yep, it is there now!

LGTM, but as Duncan suggested, do the NFC commit of the clang-formatted LLVMBitCodes.h first.

[mehdi_amini]

r265111

include/llvm/Bitcode/BitstreamWriter.h

 #define LLVM_BITCODE_BITSTREAMWRITER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Bitcode/BitCodes.h"
 #include "llvm/Support/Endian.h"
+#include "llvm/Support/SHA1.h"
 #include <vector>
 
 namespace llvm {
 
 class BitstreamWriter {
   SmallVectorImpl<char> &Out;
 
   /// CurBit - Always between 0 and 31 inclusive, specifies the next bit to use.
   unsigned CurBit;
 
   /// CurValue - The current value.  Only bits < CurBit are valid.
   uint32_t CurValue;
 
   /// CurCodeSize - This is the declared size of code values used for the
   /// current block, in bits.
   unsigned CurCodeSize;
 
   /// BlockInfoCurBID - When emitting a BLOCKINFO_BLOCK, this is the currently
   /// selected BLOCK ID.
   unsigned BlockInfoCurBID;
 
+  /// The writer has the ability to compute the hash of the stream on the fly.
+  SHA1 Hash;
+
+  /// Flag to mark the hashing enabled/disabled.
+  bool HashEnabled = false;
+
   /// CurAbbrevs - Abbrevs installed at in this block.
   std::vector<IntrusiveRefCntPtr<BitCodeAbbrev>> CurAbbrevs;
 
   struct Block {
     unsigned PrevCodeSize;
     size_t StartSizeWord;
     std::vector<IntrusiveRefCntPtr<BitCodeAbbrev>> PrevAbbrevs;
     Block(unsigned PCS, size_t SSW) : PrevCodeSize(PCS), StartSizeWord(SSW) {}
class BitstreamWriter {
   std::vector<BlockInfo> BlockInfoRecords;
 
   void WriteByte(unsigned char Value) {
     Out.push_back(Value);
   }
 
   void WriteWord(unsigned Value) {
     Value = support::endian::byte_swap<uint32_t, support::little>(Value);
+    if (HashEnabled)
+      Hash.write(reinterpret_cast<const char *>(&Value), sizeof(Value));
     Out.append(reinterpret_cast<const char *>(&Value),
                reinterpret_cast<const char *>(&Value + 1));
   }
 
   size_t GetBufferOffset() const { return Out.size(); }
 
   size_t GetWordIndex() const {
     size_t Offset = GetBufferOffset();
public:
   }
 
   /// \brief Retrieve the current position in the stream, in bits.
   uint64_t GetCurrentBitNo() const { return GetBufferOffset() * 8 + CurBit; }
 
   /// \brief Retrieve the number of bits currently used to encode an abbrev ID.
   unsigned GetAbbrevIDWidth() const { return CurCodeSize; }
 
+  /// Enable (and reset) the hashing
+  void enableHash(bool Enabled) {
+    HashEnabled = Enabled;
+    Hash.init();
+  }
+
+  StringRef getCurrentHash() { return Hash.result(); }
+
   //===--------------------------------------------------------------------===//
   // Basic Primitives for emitting bits to the stream.
   //===--------------------------------------------------------------------===//
 
   /// Backpatch a 32-bit word in the output at the given bit offset
   /// with the specified value.
   void BackpatchWord(uint64_t BitNo, unsigned NewWord) {
     using namespace llvm::support;

include/llvm/Bitcode/LLVMBitCodes.h

enum ModuleCodes {
     // ALIAS: [alias value type, addrspace, aliasee val#, linkage, visibility]
     MODULE_CODE_ALIAS       = 14,
 
     // METADATA_VALUES: [numvals]
     MODULE_CODE_METADATA_VALUES = 15,
 
     // SOURCE_FILENAME: [namechar x N]
     MODULE_CODE_SOURCE_FILENAME = 16,
+
+    // HASH: [unsigned]
+    MODULE_CODE_HASH = 17,
   };
 
   /// PARAMATTR blocks have code for defining a parameter attribute set.
   enum AttributeCodes {
     // FIXME: Remove `PARAMATTR_CODE_ENTRY_OLD' in 4.0
     PARAMATTR_CODE_ENTRY_OLD  = 1, // ENTRY: [paramidx0, attr0,
                                    //         paramidx1, attr1...]
     PARAMATTR_CODE_ENTRY      = 2, // ENTRY: [paramidx0, attrgrp0,
enum ValueSymtabCodes {
     // VST_COMBINED_GVDEFENTRY: [valueid, sumoffset, guid]
     VST_CODE_COMBINED_GVDEFENTRY = 4,
     // VST_COMBINED_ENTRY: [valueid, refguid]
     VST_CODE_COMBINED_ENTRY = 5
   };
 
   // The module path symbol table only has one code (MST_CODE_ENTRY).
   enum ModulePathSymtabCodes {
-    MST_CODE_ENTRY   = 1,  // MST_ENTRY: [modid, namechar x N]
+    MST_CODE_ENTRY = 1, // MST_ENTRY: [modid, hash, namechar x N]
   };
 
   // The summary section uses different codes in the per-module
   // and combined index cases.
   enum GlobalValueSummarySymtabCodes {
     // PERMODULE: [valueid, linkage, instcount, numrefs, numrefs x valueid,
     //             n x (valueid, callsitecount)]
     FS_PERMODULE = 1,

include/llvm/IR/ModuleSummaryIndex.h

public:
   /// Get the bitcode index recorded for this value symbol table entry.
   uint64_t bitcodeIndex() const { return BitcodeIndex; }
 
   /// Set the bitcode index recorded for this value symbol table entry.
   void setBitcodeIndex(uint64_t Offset) { BitcodeIndex = Offset; }
 };
 
 /// List of global value info structures for a particular value held
 /// in the GlobalValueMap. Requires a vector in the case of multiple
 /// COMDAT values of the same name.
 typedef std::vector<std::unique_ptr<GlobalValueInfo>> GlobalValueInfoList;
 
 /// Map from global value GUID to corresponding info structures.
 /// Use a std::map rather than a DenseMap since it will likely incur
 /// less overhead, as the value type is not very small and the size
 /// of the map is unknown, resulting in inefficiencies due to repeated
 /// insertions and resizing.
 typedef std::map<uint64_t, GlobalValueInfoList> GlobalValueInfoMapTy;
 
 /// Type used for iterating through the global value info map.
 typedef GlobalValueInfoMapTy::const_iterator const_globalvalueinfo_iterator;
 typedef GlobalValueInfoMapTy::iterator globalvalueinfo_iterator;
 
 /// String table to hold/own module path strings, which additionally holds the
-/// module ID assigned to each module during the plugin step. The StringMap
-/// makes a copy of and owns inserted strings.
-typedef StringMap<uint64_t> ModulePathStringTableTy;
+/// module ID assigned to each module during the plugin step, as well as a hash
+/// of the module. The StringMap makes a copy of and owns inserted strings.
+typedef StringMap<std::pair<uint64_t, unsigned>> ModulePathStringTableTy;
 
 /// Class to hold module path string table and global value map,
 /// and encapsulate methods for operating on them.
 class ModuleSummaryIndex {
 private:
   /// Map from value name to list of information instances for values of that
   /// name (may be duplicates in the COMDAT case, e.g.).
   GlobalValueInfoMapTy GlobalValueMap;
public:
   }
 
   /// Add a global value info for a value of the given GUID.
   void addGlobalValueInfo(uint64_t ValueGUID,
                           std::unique_ptr<GlobalValueInfo> Info) {
     GlobalValueMap[ValueGUID].push_back(std::move(Info));
   }
 
   /// Iterator to allow writer to walk through table during emission.
-  iterator_range<StringMap<uint64_t>::const_iterator>
-  modPathStringEntries() const {
-    return llvm::make_range(ModulePathStringTable.begin(),
-                            ModulePathStringTable.end());
+  const StringMap<std::pair<uint64_t, unsigned>> &modulePaths() const {
+    return ModulePathStringTable;
+  }
+
+  /// Iterator
+  StringMap<std::pair<uint64_t, unsigned>> &modulePaths() {
+    return ModulePathStringTable;
   }
 
   /// Get the module ID recorded for the given module path.
   uint64_t getModuleId(const StringRef ModPath) const {
-    return ModulePathStringTable.lookup(ModPath);
+    return ModulePathStringTable.lookup(ModPath).first;
+  }
+
+  /// Get the module ID recorded for the given module path.
+  unsigned getModuleHash(const StringRef ModPath) const {
+    return ModulePathStringTable.lookup(ModPath).second;
   }
 
   /// Add the given per-module index into this module index/summary,
   /// assigning it the given module ID. Each module merged in should have
   /// a unique ID, necessary for consistent renaming of promoted
   /// static (local) variables.
   void mergeFrom(std::unique_ptr<ModuleSummaryIndex> Other,
                  uint64_t NextModuleId);
 
   /// Convenience method for creating a promoted global name
   /// for the given value name of a local, and its original module's ID.
   static std::string getGlobalNameForLocal(StringRef Name, uint64_t ModId) {
     SmallString<256> NewName(Name);
     NewName += ".llvm.";
     raw_svector_ostream(NewName) << ModId;
     return NewName.str();
   }
 
   /// Add a new module path, mapped to the given module Id, and return StringRef
   /// owned by string table map.
-  StringRef addModulePath(StringRef ModPath, uint64_t ModId) {
-    return ModulePathStringTable.insert(std::make_pair(ModPath, ModId))
+  StringRef addModulePath(StringRef ModPath, uint64_t ModId,
+                          unsigned Hash = 0) {
+    return ModulePathStringTable.insert(std::make_pair(
+                                            ModPath, std::make_pair(ModId, 0)))
         .first->first();
   }
 
   /// Check if the given Module has any functions available for exporting
   /// in the index. We consider any module present in the ModulePathStringTable
   /// to have exported functions.
   bool hasExportedFunctions(const Module &M) const {
     return ModulePathStringTable.count(M.getModuleIdentifier());
   }
 
   /// Remove entries in the GlobalValueMap that have empty summaries due to the
   /// eager nature of map entry creation during VST parsing. These would

lib/Bitcode/Reader/BitcodeReader.cpp

case BitstreamEntry::SubBlock:
       case bitc::MODULE_STRTAB_BLOCK_ID:
         if (std::error_code EC = parseModuleStringTable())
           return EC;
         break;
       }
       continue;
 
     case BitstreamEntry::Record:
-      // Once we find the last record of interest, skip the rest.
-      if (VSTOffset > 0)
-        Stream.skipRecord(Entry.ID);
-      else {
+      {
         Record.clear();
         auto BitCode = Stream.readRecord(Entry.ID, Record);
         switch (BitCode) {
         default:
           break; // Default behavior, ignore unknown content.
         /// MODULE_CODE_SOURCE_FILENAME: [namechar x N]
         case bitc::MODULE_CODE_SOURCE_FILENAME: {
           SmallString<128> ValueName;
           if (convertToString(Record, 0, ValueName))
             return error("Invalid record");
           SourceFileName = ValueName.c_str();
           break;
         }
+        /// MODULE_CODE_HASH: [unsigned]
+        case bitc::MODULE_CODE_HASH: {
+          if (Record.size() != 1)
+            return error("Invalid hash length");
+          if (TheIndex->modulePaths().empty())
+            return error("Didn't populate module info?");
+          if (TheIndex->modulePaths().size() != 1)
+            return error("Don't expect multiple module defined?");
+          TheIndex->modulePaths().begin()->second.second = Record[0];
+          break;
+        }
         /// MODULE_CODE_VSTOFFSET: [offset]
         case bitc::MODULE_CODE_VSTOFFSET:
           if (Record.size() < 1)
             return error("Invalid record");
           VSTOffset = Record[0];
           break;
         // GLOBALVAR: [pointer type, isconst, initid,
         //             linkage, alignment, section, visibility, threadlocal,
case BitstreamEntry::Record:
       break;
     }
 
     Record.clear();
     switch (Stream.readRecord(Entry.ID, Record)) {
     default: // Default behavior: ignore.
       break;
     case bitc::MST_CODE_ENTRY: {
-      // MST_ENTRY: [modid, namechar x N]
-      if (convertToString(Record, 1, ModulePath))
+      // MST_ENTRY: [modid, hash, namechar x N]
+      if (convertToString(Record, 2, ModulePath))
         return error("Invalid record");
       uint64_t ModuleId = Record[0];
-      StringRef ModulePathInMap = TheIndex->addModulePath(ModulePath, ModuleId);
+      unsigned ModuleHash = Record[1];
+      StringRef ModulePathInMap =
+          TheIndex->addModulePath(ModulePath, ModuleId, ModuleHash);
       ModuleIdMap[ModuleId] = ModulePathInMap;
       ModulePath.clear();
       break;
     }
     }
   }
   llvm_unreachable("Exit infinite loop");
 }
 
 // Parse the function info index from the bitcode streamer into the given index.
 std::error_code ModuleSummaryIndexBitcodeReader::parseSummaryIndexInto(
     std::unique_ptr<DataStreamer> Streamer, ModuleSummaryIndex *I) {
   TheIndex = I;
 
   if (std::error_code EC = initStream(std::move(Streamer)))
     return EC;
 
   // Sniff for the signature.
   if (!hasValidBitcodeHeader(Stream))
     return error("Invalid bitcode signature");
 
   // We expect a number of well-defined blocks, though we don't necessarily
   // need to understand them all.
   while (1) {
     if (Stream.AtEndOfStream()) {

lib/Bitcode/Writer/BitcodeWriter.cpp

static void WriteModStrings(const ModuleSummaryIndex &I,
   Stream.EnterSubblock(bitc::MODULE_STRTAB_BLOCK_ID, 3);
 
   // TODO: See which abbrev sizes we actually need to emit
 
   // 8-bit fixed-width MST_ENTRY strings.
   BitCodeAbbrev *Abbv = new BitCodeAbbrev();
   Abbv->Add(BitCodeAbbrevOp(bitc::MST_CODE_ENTRY));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
+  Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Array));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 8));
   unsigned Abbrev8Bit = Stream.EmitAbbrev(Abbv);
 
   // 7-bit fixed width MST_ENTRY strings.
   Abbv = new BitCodeAbbrev();
   Abbv->Add(BitCodeAbbrevOp(bitc::MST_CODE_ENTRY));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
+  Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Array));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 7));
   unsigned Abbrev7Bit = Stream.EmitAbbrev(Abbv);
 
   // 6-bit char6 MST_ENTRY strings.
   Abbv = new BitCodeAbbrev();
   Abbv->Add(BitCodeAbbrevOp(bitc::MST_CODE_ENTRY));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
+  Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Array));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Char6));
   unsigned Abbrev6Bit = Stream.EmitAbbrev(Abbv);
 
   SmallVector<unsigned, 64> NameVals;
-  for (const StringMapEntry<uint64_t> &MPSE : I.modPathStringEntries()) {
+  for (const auto &MPSE : I.modulePaths()) {
     StringEncoding Bits =
         getStringEncoding(MPSE.getKey().data(), MPSE.getKey().size());
     unsigned AbbrevToUse = Abbrev8Bit;
     if (Bits == SE_Char6)
       AbbrevToUse = Abbrev6Bit;
     else if (Bits == SE_Fixed7)
       AbbrevToUse = Abbrev7Bit;
 
-    NameVals.push_back(MPSE.getValue());
+    NameVals.push_back(MPSE.getValue().first);
+    NameVals.push_back(MPSE.getValue().second);
 
     for (const auto P : MPSE.getKey())
       NameVals.push_back((unsigned char)P);
 
     // Emit the finished record.
     Stream.EmitRecord(bitc::MST_CODE_ENTRY, NameVals, AbbrevToUse);
     NameVals.clear();
   }
static void WriteIdentificationBlock(const Module *M, BitstreamWriter &Stream) {
   Abbv->Add(BitCodeAbbrevOp(bitc::IDENTIFICATION_CODE_EPOCH));
   Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6));
   auto EpochAbbrev = Stream.EmitAbbrev(Abbv);
   SmallVector<unsigned, 1> Vals = {bitc::BITCODE_CURRENT_EPOCH};
   Stream.EmitRecord(bitc::IDENTIFICATION_CODE_EPOCH, Vals, EpochAbbrev);
   Stream.ExitBlock();
 }
 
+static void writeModuleHash(BitstreamWriter &Stream) {
+  // Emit the module's hash.
+  // MODULE_CODE_HASH: [unsigned]
+  auto SHA1 = Stream.getCurrentHash();
+  unsigned Hash = hash_combine_range(&*SHA1.begin(), &*SHA1.end());
+
+  // Emit the finished record.
+  Stream.EmitRecord(bitc::MODULE_CODE_HASH, makeArrayRef(Hash));
+}
+
 /// WriteModule - Emit the specified module to the bitstream.
 static void WriteModule(const Module *M, BitstreamWriter &Stream,
                         bool ShouldPreserveUseListOrder,
                         uint64_t BitcodeStartBit, bool EmitSummaryIndex) {
   Stream.EnterSubblock(bitc::MODULE_BLOCK_ID, 3);
 
+  if (EmitSummaryIndex)
+    Stream.enableHash(true);
+
   SmallVector<unsigned, 1> Vals;
   unsigned CurVersion = 1;
   Vals.push_back(CurVersion);
   Stream.EmitRecord(bitc::MODULE_CODE_VERSION, Vals);
 
   // Analyze the module, enumerating globals, functions, etc.
   ValueEnumerator VE(*M, ShouldPreserveUseListOrder);
 
static void WriteModule(const Module *M, BitstreamWriter &Stream,
   // Need to write after the above call to WriteFunction which populates
   // the summary information in the index.
   if (EmitSummaryIndex)
     WritePerModuleGlobalValueSummary(FunctionIndex, M, VE, Stream);
 
   WriteValueSymbolTable(M->getValueSymbolTable(), VE, Stream,
                         VSTOffsetPlaceholder, BitcodeStartBit, &FunctionIndex);
 
+  if (EmitSummaryIndex) {
+    writeModuleHash(Stream);
+    Stream.enableHash(false);
+  }
+
   Stream.ExitBlock();
 }
 
 /// EmitDarwinBCHeader - If generating a bc file on darwin, we have to emit a
 /// header and trailer to make it compatible with the system archiver.  To do
 /// this we emit the following header, and then emit a trailer that pads the
 /// file out to be a multiple of 16 bytes.
 ///

tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

default: return nullptr;
       STRINGIFY_CODE(MODULE_CODE, GLOBALVAR)
       STRINGIFY_CODE(MODULE_CODE, FUNCTION)
       STRINGIFY_CODE(MODULE_CODE, ALIAS)
       STRINGIFY_CODE(MODULE_CODE, PURGEVALS)
       STRINGIFY_CODE(MODULE_CODE, GCNAME)
       STRINGIFY_CODE(MODULE_CODE, VSTOFFSET)
       STRINGIFY_CODE(MODULE_CODE, METADATA_VALUES)
       STRINGIFY_CODE(MODULE_CODE, SOURCE_FILENAME)
+      STRINGIFY_CODE(MODULE_CODE, HASH)
     }
   case bitc::IDENTIFICATION_BLOCK_ID:
     switch (CodeID) {
     default:
       return nullptr;
       STRINGIFY_CODE(IDENTIFICATION_CODE, STRING)
       STRINGIFY_CODE(IDENTIFICATION_CODE, EPOCH)
     }
if (DumpRecords) {
         Abbv = Stream.getAbbrev(Entry.ID);
         outs() << " abbrevid=" << Entry.ID;
       }
 
       for (unsigned i = 0, e = Record.size(); i != e; ++i)
         outs() << " op" << i << "=" << (int64_t)Record[i];
 
       outs() << "/>";
 
       if (Abbv) {
         for (unsigned i = 1, e = Abbv->getNumOperandInfos(); i != e; ++i) {
           const BitCodeAbbrevOp &Op = Abbv->getOperandInfo(i);
           if (!Op.isEncoding() || Op.getEncoding() != BitCodeAbbrevOp::Array)
             continue;
           assert(i + 2 == e && "Array op not second to last");
           std::string Str;
           bool ArrayIsPrintable = true;