[LLD][COFF] Reduce the maximum size of the GHASH table

[aganea]

Summary

Before this patch, the maximum size of the GHASH table was 2^31 buckets. However we were storing the bucket index into a TypeIndex which has an input limit of (2^31)-4095 indices, see this link. Any value above that limit will improperly set the TypeIndex's high bit, which is interpreted as DecoratedItemIdMask. This used to cause bad indices on extraction when calling TypeIndex::toArrayIndex().

[rnk]

lgtm

lld/COFF/DebugTypes.cpp

void TypeMerger::mergeTypesWithGHash() {
   // small compared to total memory usage, at eight bytes per input type record,
   // and most input type records are larger than eight bytes.
   size_t tableSize = 0;
   for (TpiSource *source : TpiSource::instances)
     tableSize += source->ghashes.size();
 
   // Cap the table size so that we can use 32-bit cell indices. Type indices are
   // also 32-bit, so this is an inherent PDB file format limit anyway.
-  tableSize = std::min(size_t(INT32_MAX), tableSize);
+  tableSize =
+      std::min(size_t(INT32_MAX) - TypeIndex::FirstNonSimpleIndex, tableSize);
   ghashState.table.init(static_cast<uint32_t>(tableSize));
 
   // Insert ghashes in parallel. During concurrent insertion, we cannot observe
   // the contents of the hash table cell, but we can remember the insertion
   // position. Because the table does not rehash, the position will not change
   // under insertion. After insertion is done, the value of the cell can be read
   // to retrieve the final PDB type index.
   parallelForEachN(0, TpiSource::instances.size(), [&](size_t tpiSrcIdx) {