[DWARFLibrary] Add support to re-construct cu-index

[ayermolo]

Summary

According to DWARF5 specification and gnu specification for DWARF4 the offset
entry in the CU/TU Index is 32 bits. This presents a problem when
.debug_info.dwo in DWP file grows beyond 4GB. The CU Index becomes partially
corrupted.

This diff adds manual parsing of .debug_info.dwo/.debug_abbrev.dwo to
reconstruct CU index in general, and TU index for DWARF5. This is a work around
until DWARF6 spec is finalized.

Next patch will change internal CU/TU struct to 64 bit, and change uses as
necessary. The plan is to land all the patches in one go after all are approved.

This patch originates from the discussion in: https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902

[dblaikie]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

I'm a bit more worried about implementing this for DWARFv4 due to needing to rebuild .debug_abbrev sections together, which is less reliable/guaranteed (there's no guarantee that the abbrev contributions are written in the same order as the .debug_info sections - though it's the case in reality I guess) than .debug_info parsing.

Any chance this workaround can be restricted to only DWARFv5?

llvm/lib/DebugInfo/DWARF/DWARFUnitIndex.cpp

188–199

Could we put this after the old table reading code (& then the old code wouldn't need to be modified - yeah, it'd read the corrupted values, but they'd be overwritten afterwards anyway)? Might be a smaller change?

202

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

I would prefer to keep it more generic and include DWARF4. Part of the reason is that we still heavily rely on DWARF4.
One thing we can do is that if DWOID is "garbage". As in when we populate CU Index and can't find the signature in a map we stop.
With the current implementation we can just clear the map and parse CU/TU index as before. If we move implementation of updating contributions entires after parsing we just stop updating further. I think with this approach we can handle the common case of how things are now in reality, and on of chance the assumption fails we are no worse than we were before. Although it does assume that with wrong abbrev. getDWOID doesn't crash.... What do you think?

[dblaikie]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

maybe a way to make this more robust for DWARFv4 would be to parse the index as-written first, and using the index entries to find the suitable .debug_abbrev contribution for the CU (found via the manual walk, associated via the DWO ID)

As for the fallback behavior in case of "garbage" DWO IDs - given how tenuous all this already is, perhaps we should stop/error if that happens, rather than producing corrupted data?

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

Sorry, not sure I follow. When we manually walk there is implicit assumption that .debug_info.dwo section and corresponding .debug_abbrev.dwo are in the same order. Thus we can use abbrev to parse unit die to get DWO ID (since this is DWARF4 and it's one of the attributes). Then use DWO ID to figure out which contribution we need to modify.

When we parse Index CUs are in the order in which DWOIDs got hashed into the table. Since .debug_info offsets column is corrupt (at least for some), we still can't map the abbrev section to the corresponding debug section.

The sentiment from users is better to have some/most of debug information than none. Which is why we have dwp with over 4GB of debug info even thought in trunk it's an error in llvm-dwp....

[dblaikie]

(could you link in the patch description to the thread where we were discussing this so I can page the context back in, etc? I'm probably considering/rehashing some of that conversation in my head - like introducing a new/custom index version number to handle these cases, etc)

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

Ah, right, right - circular problem. Sorry.

I'm still a bit worried about the complexity of supporting v4 here... not sure it's the right thing to do.

819–820

I guess we could at least validate that the abbrev offset was probably correct - once we have the DWO ID we could look up the index entry, find the abbrev section offset, and confirm it was the one we used?

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

We could. For the case where we use wrong abbrev, interpret some random 8 bytes as DWO ID, and they happen to be same ones as a valid signature?

[clayborg]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

797–802

We shouldn't need to do anything for .debug_abbrev stuff right? That section will never exceed 4GB, so no need to do anything when manually parsing.

814–815

This will be wrong for DWARF64. Just use the DWARFUnitHeader::getNextUnitOffset() function as it correctly figures out the length.

819–820

Both solutions are a bit messy. We must get the abbrev offset correct somehow. If we parse the CU index first, you can most probably identify the right .debug_abbrev offset. It will be very doubtful that you will have two 32 bit CU offsets that truncate to the same offset. It could happen, but very unlikely. So I would rather we don't assume anything about .debug_abbrev layout and parse the standard 32 bit CU index first, and then to find the .debug_abbrev offset, we find the truncated 32 bit CU offset in the list.

llvm/lib/DebugInfo/DWARF/DWARFUnitIndex.cpp

188–199

I agree here. Just let it parse normally, and then run some code to fix up the wrong values only if needed.

[ayermolo]

@dblaikie How would you like to proceed?

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

Not sure I follow. Only algo I could come up with that would do it is:

Parse headers and have a map from truncated and normal ones under 4GB to 64bit real offsets. In case of collision
Parse the CU Index, and iterate over it.
As we do it, we find a cu offset in a map 
   get its real 64 bit offsets. 
   Get abbrev offset from cu index
   Until DWO ID matches signature iterate over 64 bit real offsets
       Use it to parse UnitDIE and check
       update cu index with real 64bit offset if check passes

*Assumption is that parsing CU with wrong abbrev won't just crash somewhere

or did you had something else in mind?

[clayborg]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

yeah, my flow would be:

• parse CU index using previous code and let 32 bit CU offsets get truncated
• if (sizeof(.debug_info|types) > 4GB) then call this function
• parse all CU headers
  • find the current CU's .debug_abbrev offset by searching truncated map from first step using a truncated 32 bit offset
  • parse the CU's first DIE using the correct abbreviation table

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

To confirm the last step so we can verify DWO ID in a UnitDIE is the same as a signature in contribution?

[clayborg]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

You would only need to do this if you have multiple CUs whose truncated offset is in the parsed CU index. If there is only one match, we are good to go. So maybe a better flow is:

• parse CU index using previous code and let 32 bit CU offsets get truncated
• if (sizeof(.debug_info|types) <= 4GB) then we are done
• parse all CU headers
  • search the CU index for a truncated offset that matches the current CU's truncated 64 bit offset
    • If the there is only one match, just fix the truncated CU offset and continue, no need to read the CU DIE at all for the DWO ID
    • if there are multiple matches, then you need to try and get the the right abbreviation so you can verify the DWO ID. If we have the wrong abbrev table, then we might get decoding errors, or we might not, so this will be the tricky part to get right. You could rely on ordering etc to help try and get the right abbrev table as you were doing, but you only need to do this _if_ you have multiple matches.

[clayborg]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

If we have DWARF 5, then the DWO ID is in the DWARFUnitHeader, so no need for .debug_abbrev parsing at all in that case. So we only need to do heroics for DWARF 4 and earlier.

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

OK, sounds like we are on same page on what you proposed.
@dblaikie what do you think?

[dblaikie]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

Nice not to have to read the abbrev more than necessary (ie: only when there's an ambiguity in the truncated offset) - but the code is still a bit iffy to have to figure out the abbrev section to be able to read the DWO ID to use...

Would it be acceptable to skip that, and still fail in those cases & rely only on the unambiguous truncation logic?

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

Hard to tell since we don't know how often collision will happen in real world, but probably as a first attempt good enough?
If we start seeing issues we can revisit?

Although TBH personally I don't see a big deal with current approach in this diff. Yes realistically speaking we are relying on implicit behavior of debug sections and abbrev sections being in the same order, but tool that is being used is llvm-dwp and it's in the same llvm toolchain. I believe it's more robust than relying on assumption that collisions will be rare.

[clayborg]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

I'd doubt you will have many if any cases where truncated compile units match. Can we rely on the ordering of the CU Offsets at all in the CU index? Can we assume they are ever increasing? That might make it easier to match up things to CUs. Or is the CU index ordered by the DWO ID or is there no order to it at all? I can't remember.

We have rock solid ways to do this with no collisions. With collisions, which will be really rare, we can make a best effort. Don't care how it is done. Might be worth checking out a binary from the other dwp tool? Is there one from GCC?

[ayermolo]

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

819–820

For CU Index we cannot rely on ordering. They are ordered on based on hash of DWO ID in the table.

I tried DWP GNU dwp version 2.30-119.el8 with bzip2 build with clang-15 and it coredumped.
It also coredumped with bzip2 build with gcc (GCC) 11.x 20221024

[ayermolo]

Wasn't sure whether to override this diff or create a new one. For clarity went with a new one. Based on Gregs suggestion.
https://reviews.llvm.org/D137882

llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h

class DWARFContext : public DIContext {
   /// section.
   enum MacroSecType {
     MacinfoSection,
     MacinfoDwoSection,
     MacroSection,
     MacroDwoSection
   };
 
+  // When set parses debug_info.dwo/debug_abbrev.dwo manually and populates CU
+  // Index, and TU Index for DWARF5.
+  bool ParseCUTUIndexManually;
+
 public:
   DWARFContext(std::unique_ptr<const DWARFObject> DObj,
                std::string DWPName = "",
                std::function<void(Error)> RecoverableErrorHandler =
                    WithColor::defaultErrorHandler,
                std::function<void(Error)> WarningHandler =
                    WithColor::defaultWarningHandler);
   ~DWARFContext() override;
public:
   }
 
   /// Return the compile unit which contains instruction with provided
   /// address.
   /// TODO: change input parameter from "uint64_t Address"
   ///       into "SectionedAddress Address"
   DWARFCompileUnit *getCompileUnitForAddress(uint64_t Address);
 
+  /// Returns whether CU/TU should be populated manually. TU Index populated
+  /// manually only for DWARF5.
+  bool getParseCUTUIndexManually() const { return ParseCUTUIndexManually; }
+
+  /// Sets whether CU/TU should be populated manually. TU Index populated
+  /// manually only for DWARF5.
+  void setParseCUTUIndexManually(bool PCUTU) { ParseCUTUIndexManually = PCUTU; }
+
 private:
   /// Parse a macro[.dwo] or macinfo[.dwo] section.
   std::unique_ptr<DWARFDebugMacro>
   parseMacroOrMacinfo(MacroSecType SectionType);
 
   void addLocalsForDie(DWARFCompileUnit *CU, DWARFDie Subprogram, DWARFDie Die,
                        std::vector<DILocal> &Result);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_DEBUGINFO_DWARF_DWARFCONTEXT_H

llvm/include/llvm/DebugInfo/DWARF/DWARFUnit.h

public:
   dwarf::DwarfFormat getFormat() const { return FormParams.Format; }
   uint8_t getAddressByteSize() const { return FormParams.AddrSize; }
   uint8_t getRefAddrByteSize() const { return FormParams.getRefAddrByteSize(); }
   uint8_t getDwarfOffsetByteSize() const {
     return FormParams.getDwarfOffsetByteSize();
   }
   uint64_t getLength() const { return Length; }
   uint64_t getAbbrOffset() const { return AbbrOffset; }
+  void setAbbrOffset(uint64_t Offset) { AbbrOffset = Offset; }
   Optional<uint64_t> getDWOId() const { return DWOId; }
   void setDWOId(uint64_t Id) {
     assert((!DWOId || *DWOId == Id) && "setting DWOId to a different value");
     DWOId = Id;
   }
   const DWARFUnitIndex::Entry *getIndexEntry() const { return IndexEntry; }
   uint64_t getTypeHash() const { return TypeHash; }
   uint64_t getTypeOffset() const { return TypeOffset; }

llvm/include/llvm/DebugInfo/DWARF/DWARFUnitIndex.h

 //===- DWARFUnitIndex.h -----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_DWARF_DWARFUNITINDEX_H
 #define LLVM_DEBUGINFO_DWARF_DWARFUNITINDEX_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include <cstdint>
 #include <memory>
+#include <unordered_map>
 
 namespace llvm {
 
 class raw_ostream;
 class DataExtractor;
 
 /// The enum of section identifiers to be used in internal interfaces.
 ///
public:
     const SectionContribution *getContribution() const;
 
     const SectionContribution *getContributions() const {
       return Contributions.get();
     }
 
     uint64_t getSignature() const { return Signature; }
   };
+  using EntryType = DWARFUnitIndex::Entry::SectionContribution;
+  using EntryMap = std::unordered_map<uint64_t, EntryType>;
 
 private:
   struct Header Header;
 
   DWARFSectionKind InfoColumnKind;
   int InfoColumn = -1;
   std::unique_ptr<DWARFSectionKind[]> ColumnKinds;
   // This is a parallel array of section identifiers as they read from the input
   // file. The mapping from raw values to DWARFSectionKind is not revertable in
   // case of unknown identifiers, so we keep them here.
   std::unique_ptr<uint32_t[]> RawSectionIds;
   std::unique_ptr<Entry[]> Rows;
   mutable std::vector<Entry *> OffsetLookup;
 
   static StringRef getColumnHeader(DWARFSectionKind DS);
 
-  bool parseImpl(DataExtractor IndexData);
+  bool parseImpl(DataExtractor IndexData, EntryMap &SignatureMap);
 
 public:
   DWARFUnitIndex(DWARFSectionKind InfoColumnKind)
       : InfoColumnKind(InfoColumnKind) {}
 
   explicit operator bool() const { return Header.NumBuckets; }
 
+  bool parse(DataExtractor IndexData, EntryMap &SignatureMap);
   bool parse(DataExtractor IndexData);
   void dump(raw_ostream &OS) const;
 
   uint32_t getVersion() const { return Header.Version; }
 
   const Entry *getFromOffset(uint32_t Offset) const;
   const Entry *getFromHash(uint64_t Offset) const;
 

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

bool DWARFContext::verify(raw_ostream &OS, DIDumpOptions DumpOpts) {
   if (DumpOpts.DumpType & DIDT_DebugInfo)
     Success &= verifier.handleDebugInfo();
   if (DumpOpts.DumpType & DIDT_DebugLine)
     Success &= verifier.handleDebugLine();
   Success &= verifier.handleAccelTables();
   return Success;
 }
 
+enum class IndexType { CUIndex, TUIndex };
+static DWARFUnitIndex::EntryMap
+parseDebugInfo(const DWARFObject &DObj, DWARFContext &C, IndexType type) {
+  DWARFUnitIndex::EntryMap SignatureMap;
+  if (DObj.getCUIndexSection().empty())
+    return SignatureMap;
+
+  DWARFUnitVector UnitVector;
+  uint64_t Offset = 0;
+  DObj.forEachInfoDWOSections([&](const DWARFSection &S) {
+    if (!(C.getParseCUTUIndexManually() ||
+          S.Data.size() >= std::numeric_limits<uint32_t>::max()))
+      return;
+    DWARFDebugAbbrev AbbrevDWO;
+    DataExtractor abbrData(DObj.getAbbrevDWOSection(), C.isLittleEndian(), 0);
+    AbbrevDWO.extract(abbrData);
+    SmallVector<uint32_t, 10> AbbrOffsets;
+    for (auto &I : AbbrevDWO)
+      AbbrOffsets.push_back(I.second.getOffset());
+
+    DWARFDataExtractor Data(DObj, S, C.isLittleEndian(), 0);
+    int I = 0;
+    while (Data.isValidOffset(Offset)) {
+      DWARFUnitHeader Header;
+      if (!Header.extract(C, Data, &Offset, DWARFSectionKind::DW_SECT_INFO)) {
+        llvm::errs() << "Failed to parse CU header in DWP file\n";
+        SignatureMap.clear();
+        break;
+      }
+
+      Offset =
+          Header.getOffset() + Header.getLength() + /* Size of legth field */ 4;
+
+      // If we are parsing TU-index and for .debug_types section we don't need
+      // to do anything.
+      if (Header.getVersion() == 4 && type == IndexType::TUIndex)
+        break;
+
+      // Parsing CU Index, can skip all the TUs
+      if (Header.isTypeUnit() && type == IndexType::CUIndex)
+        continue;
+
+      Header.setAbbrOffset(AbbrOffsets[I]);
+      std::unique_ptr<DWARFUnit> U;
+
+      if (Header.isTypeUnit()) {
+        // If we are hitting type unit, we are parsing TU Index.
+        U = std::make_unique<DWARFTypeUnit>(
+            C, S, Header, C.getDebugAbbrevDWO(), &DObj.getRangesDWOSection(),
+            &DObj.getLocDWOSection(), DObj.getStrDWOSection(),
+            DObj.getStrOffsetsDWOSection(), &DObj.getAddrSection(),
+            DObj.getLineDWOSection(), C.isLittleEndian(), true, UnitVector);
+      } else {
+        // TUs share same abbrev as CU. So we move to next abbrev when we
+        // encounter a CU.
+        ++I;
+        if (type == IndexType::TUIndex)
+          continue;
+        U = std::make_unique<DWARFCompileUnit>(
+            C, S, Header, C.getDebugAbbrevDWO(), &DObj.getRangesDWOSection(),
+            &DObj.getLocDWOSection(), DObj.getStrDWOSection(),
+            DObj.getStrOffsetsDWOSection(), &DObj.getAddrSection(),
+            DObj.getLineDWOSection(), C.isLittleEndian(), true, UnitVector);
+      }
+
+      uint64_t ID = Header.isTypeUnit() ? Header.getTypeHash() : *U->getDWOId();
+      SignatureMap.insert(
+          {ID,
+           {(uint32_t)U->getOffset(),
+            (uint32_t)(U->getNextUnitOffset() - U->getOffset())}});
+    }
+  });
+
+  return SignatureMap;
+}
+
 const DWARFUnitIndex &DWARFContext::getCUIndex() {
   if (CUIndex)
     return *CUIndex;
 
+  DWARFUnitIndex::EntryMap SignatureMap =
+      parseDebugInfo(*DObj, *this, IndexType::CUIndex);
   DataExtractor CUIndexData(DObj->getCUIndexSection(), isLittleEndian(), 0);
 
   CUIndex = std::make_unique<DWARFUnitIndex>(DW_SECT_INFO);
-  CUIndex->parse(CUIndexData);
+  CUIndex->parse(CUIndexData, SignatureMap);
   return *CUIndex;
 }
 
 const DWARFUnitIndex &DWARFContext::getTUIndex() {
   if (TUIndex)
     return *TUIndex;
 
+  DWARFUnitIndex::EntryMap SignatureMap =
+      parseDebugInfo(*DObj, *this, IndexType::TUIndex);
   DataExtractor TUIndexData(DObj->getTUIndexSection(), isLittleEndian(), 0);
 
   TUIndex = std::make_unique<DWARFUnitIndex>(DW_SECT_EXT_TYPES);
-  TUIndex->parse(TUIndexData);
+  TUIndex->parse(TUIndexData, SignatureMap);
   return *TUIndex;
 }
 
 DWARFGdbIndex &DWARFContext::getGdbIndex() {
   if (GdbIndex)
     return *GdbIndex;
 
   DataExtractor GdbIndexData(DObj->getGdbIndexSection(), true /*LE*/, 0);

llvm/lib/DebugInfo/DWARF/DWARFUnitIndex.cpp

bool DWARFUnitIndex::Header::parse(DataExtractor IndexData,
   NumBuckets = IndexData.getU32(OffsetPtr);
   return true;
 }
 
 void DWARFUnitIndex::Header::dump(raw_ostream &OS) const {
   OS << format("version = %u, units = %u, slots = %u\n\n", Version, NumUnits, NumBuckets);
 }
 
-bool DWARFUnitIndex::parse(DataExtractor IndexData) {
-  bool b = parseImpl(IndexData);
+bool DWARFUnitIndex::parse(DataExtractor IndexData, EntryMap &SignatureMap) {
+  bool b = parseImpl(IndexData, SignatureMap);
   if (!b) {
     // Make sure we don't try to dump anything
     Header.NumBuckets = 0;
     // Release any partially initialized data.
     ColumnKinds.reset();
     Rows.reset();
   }
   return b;
 }
 
-bool DWARFUnitIndex::parseImpl(DataExtractor IndexData) {
+bool DWARFUnitIndex::parse(DataExtractor IndexData) {
+  EntryMap SignatureMap;
+  return parse(IndexData, SignatureMap);
+}
+
+bool DWARFUnitIndex::parseImpl(DataExtractor IndexData,
+                               EntryMap &SignatureMap) {
   uint64_t Offset = 0;
   if (!Header.parse(IndexData, &Offset))
     return false;
 
   // Fix InfoColumnKind: in DWARFv5, type units are in .debug_info.dwo.
   if (Header.Version == 5)
     InfoColumnKind = DW_SECT_INFO;
 
   if (!IndexData.isValidOffsetForDataOfSize(
           Offset, Header.NumBuckets * (8 + 4) +
                       (2 * Header.NumUnits + 1) * 4 * Header.NumColumns))
     return false;
 
   Rows = std::make_unique<Entry[]>(Header.NumBuckets);
   auto Contribs =
       std::make_unique<Entry::SectionContribution *[]>(Header.NumUnits);
   ColumnKinds = std::make_unique<DWARFSectionKind[]>(Header.NumColumns);
   RawSectionIds = std::make_unique<uint32_t[]>(Header.NumColumns);
 
   // Read Hash Table of Signatures
   for (unsigned i = 0; i != Header.NumBuckets; ++i)
     Rows[i].Signature = IndexData.getU64(&Offset);
 
+  auto Signatures = std::make_unique<uint64_t[]>(Header.NumUnits);
   // Read Parallel Table of Indexes
   for (unsigned i = 0; i != Header.NumBuckets; ++i) {
     auto Index = IndexData.getU32(&Offset);
     if (!Index)
       continue;
     Rows[i].Index = this;
     Rows[i].Contributions =
         std::make_unique<Entry::SectionContribution[]>(Header.NumColumns);
     Contribs[Index - 1] = Rows[i].Contributions.get();
+    Signatures[Index - 1] = Rows[i].Signature;
   }
 
   // Read the Column Headers
   for (unsigned i = 0; i != Header.NumColumns; ++i) {
     RawSectionIds[i] = IndexData.getU32(&Offset);
     ColumnKinds[i] = deserializeSectionKind(RawSectionIds[i], Header.Version);
     if (ColumnKinds[i] == InfoColumnKind) {
       if (InfoColumn != -1)
         return false;
       InfoColumn = i;
     }
   }
 
   if (InfoColumn == -1)
     return false;
 
-  // Read Table of Section Offsets
-  for (unsigned i = 0; i != Header.NumUnits; ++i) {
+  // Populating offsets and length for .debug-info.dwo section if we parsed it
+  // manually.
+  for (unsigned i = 0; !SignatureMap.empty() && i != Header.NumUnits; ++i) {
     auto *Contrib = Contribs[i];
-    for (unsigned i = 0; i != Header.NumColumns; ++i)
-      Contrib[i].Offset = IndexData.getU32(&Offset);
+    auto Iter = SignatureMap.find(Signatures[i]);
+    if (Iter == SignatureMap.end()) {
+      llvm::errs() << "Signature not found\n";
+      continue;
+    }
+    Contrib[InfoColumn].Offset = Iter->second.Offset;
+    Contrib[InfoColumn].Length = Iter->second.Length;
   }
 
-  // Read Table of Section Sizes
+  auto PopulateField =
+      [&](uint32_t DWARFUnitIndex::Entry::SectionContribution::*Field) -> void {
     for (unsigned i = 0; i != Header.NumUnits; ++i) {
       auto *Contrib = Contribs[i];
-    for (unsigned i = 0; i != Header.NumColumns; ++i)
-      Contrib[i].Length = IndexData.getU32(&Offset);
+      for (unsigned i = 0; i != Header.NumColumns; ++i) {
+        if (i == (unsigned)InfoColumn && !SignatureMap.empty()) {
+          Offset += 4;
+          continue;
+        }
+        Contrib[i].*Field = IndexData.getU32(&Offset);
+      }
     }
+  };
+  // Read Table of Section Offsets
+  PopulateField(&DWARFUnitIndex::Entry::SectionContribution::Offset);
+
+  // Read Table of Section Sizes
+  PopulateField(&DWARFUnitIndex::Entry::SectionContribution::Length);
 
   return true;
 }
 
 StringRef DWARFUnitIndex::getColumnHeader(DWARFSectionKind DS) {
   switch (DS) {
 #define HANDLE_DW_SECT(ID, NAME)                                               \
   case DW_SECT_##NAME:                                                         \

llvm/test/tools/llvm-dwp/X86/debug_macro_v5.s

 # This test checks the support for writing macro sections and their index (v5).
 
 # RUN: llvm-mc -triple x86_64-unknown-linux --filetype=obj --split-dwarf-file=%t.dwo -dwarf-version=5 %s -o %t.o
 # RUN: llvm-dwp %t.dwo -o %t.dwp 2>&1
-# RUN: llvm-dwarfdump -debug-macro -debug-cu-index %t.dwp | FileCheck %s
+# RUN: llvm-dwarfdump -debug-macro -debug-cu-index %t.dwp | FileCheck -check-prefix=CHECK %s
+# RUN: llvm-dwarfdump -debug-macro -debug-cu-index -manaully-generate-cu-tu-index %t.dwp | FileCheck -check-prefix=CHECK2 %s
 
 # CHECK-DAG: .debug_macro.dwo contents:
 # CHECK: macro header: version = 0x0005, flags = 0x00, format = DWARF32
 # CHECK-NEXT: DW_MACRO_start_file - lineno: 0 filenum: 0
 # CHECK-NEXT: DW_MACRO_define_strx - lineno: 1 macro: x 5
 # CHECK-NEXT: DW_MACRO_end_file
 
 # CHECK-DAG: .debug_cu_index contents:
 # CHECK-NEXT: version = 5, units = 1, slots = 2
 # CHECK: Index Signature          INFO                     ABBREV                   STR_OFFSETS              MACRO
 # CHECK:     1 0x0000000000000000 [0x00000000, 0x00000019) [0x00000000, 0x00000008) [0x00000000, 0x0000000c) [0x00000000, 0x0000000b)
 
+# CHECK2: Index Signature          INFO                     ABBREV                   STR_OFFSETS              MACRO
+# CHECK2:     1 0x0000000000000000 [0x00000000, 0x00000019) [0x00000000, 0x00000008) [0x00000000, 0x0000000c) [0x00000000, 0x0000000b)
+
     .section  .debug_info.dwo,"e",@progbits
     .long .Ldebug_info_dwo_end0-.Ldebug_info_dwo_start0 # Length of Unit
 .Ldebug_info_dwo_start0:
     .short  5                               # DWARF version number
     .byte 5                               # DWARF Unit Type (DW_UT_split_compile)
     .byte 8                               # Address Size (in bytes)
     .long 0                               # Offset Into Abbrev. Section
     .quad 0

llvm/test/tools/llvm-dwp/X86/info-v5.s

 # this checks llvm-dwp handling of DWARFv5 Info section header.
 
 # RUN: llvm-mc --triple=x86_64-unknown-linux --filetype=obj --split-dwarf-file=%t.dwo -dwarf-version=5 %s -o %t.o
 
 # RUN: llvm-dwp %t.dwo -o %t.dwp
-# RUN: llvm-dwarfdump -v %t.dwp | FileCheck %s
+# RUN: llvm-dwarfdump -v %t.dwp | FileCheck -check-prefix=CHECK %s
+# RUN: llvm-dwarfdump -manaully-generate-cu-tu-index -v %t.dwp | FileCheck -check-prefix=CHECK2 %s
 
 #CHECK-DAG: .debug_info.dwo contents:
 #CHECK: 0x00000000: Compile Unit: length = 0x00000050, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_compile, abbr_offset = 0x0000, addr_size = 0x08, DWO_id = [[DWOID:.*]] (next unit at 0x00000054)
 
 # CHECK-DAG: .debug_cu_index contents:
 # CHECK: version = 5, units = 1, slots = 2
 # CHECK: Index Signature          INFO                     ABBREV
 # CHECK: 1 [[DWOID]] [0x00000000, 0x00000054) [0x00000000, 0x0000002a)
 
+#CHECK2-DAG: .debug_info.dwo contents:
+#CHECK2: 0x00000000: Compile Unit: length = 0x00000050, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_compile, abbr_offset = 0x0000, addr_size = 0x08, DWO_id = [[DWOID2:.*]] (next unit at 0x00000054)
+
+# CHECK2-DAG: .debug_cu_index contents:
+# CHECK2: version = 5, units = 1, slots = 2
+# CHECK2: Index Signature          INFO
+# CHECK21: 1 [[DWOID2]] [0x00000000, 0x00000054) [0x00000000, 0x0000002a)
+
   .section  .debug_info.dwo,"e",@progbits
   .long .Ldebug_info_dwo_end0-.Ldebug_info_dwo_start0 # Length of Unit
 .Ldebug_info_dwo_start0:
   .short  5                      # DWARF version number
   .byte 5                       # DWARF Unit Type
   .byte 8                       # Address Size (in bytes)
   .long 0                       # Offset Into Abbrev. Section
   .quad -1173350285159172090

llvm/test/tools/llvm-dwp/X86/loclists.s

 # This test checks if llvm-dwp outputs .debug_loclists.
 
 # RUN: llvm-mc -triple x86_64-unknown-linux %s -filetype=obj -o %t.o \
 # RUN:         -split-dwarf-file=%t.dwo -dwarf-version=5
 # RUN: llvm-dwp %t.dwo -o %t.dwp
-# RUN: llvm-dwarfdump -debug-loclists -debug-cu-index -debug-tu-index %t.dwp | FileCheck %s
+# RUN: llvm-dwarfdump -debug-loclists -debug-cu-index -debug-tu-index %t.dwp | FileCheck -check-prefix=CHECK %s
+# RUN: llvm-dwarfdump -debug-cu-index -debug-tu-index -manaully-generate-cu-tu-index %t.dwp | FileCheck -check-prefix=CHECK2 %s
 
 # CHECK-DAG: .debug_loclists.dwo contents:
 # CHECK: locations list header: length = 0x00000019, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000001
 # CHECK-NEXT: offsets: [
 # CHECK-NEXT: 0x00000004
 # CHECK-NEXT: ]
 # CHECK: DW_LLE_base_addressx   (0x0000000000000000)
 # CHECK-NEXT: DW_LLE_offset_pair     (0x0000000000000000, 0x0000000000000004): DW_OP_reg5 RDI
 # CHECK-NEXT: DW_LLE_offset_pair     (0x0000000000000004, 0x0000000000000008): DW_OP_reg3 RBX
 
 # CHECK-DAG: .debug_cu_index contents:
 # CHECK: Index Signature          INFO                     ABBREV                   LOCLISTS
 # CHECK:     1 {{.*}} [0x00000018, 0x0000002d) [0x00000000, 0x00000004) [0x00000000, 0x0000001d)
 
 # CHECK-DAG: .debug_tu_index contents:
 # CHECK: Index Signature          INFO                     ABBREV                   LOCLISTS
 # CHECK:     2 {{.*}} [0x00000000, 0x00000018) [0x00000000, 0x00000004) [0x00000000, 0x0000001d)
 
+# CHECK2-DAG: .debug_cu_index contents:
+# CHECK2: Index Signature          INFO                     ABBREV                   LOCLISTS
+# CHECK2:     1 {{.*}} [0x00000018, 0x0000002d) [0x00000000, 0x00000004) [0x00000000, 0x0000001d)
+
+# CHECK2-DAG: .debug_tu_index contents:
+# CHECK2: Index Signature          INFO                     ABBREV                   LOCLISTS
+# CHECK2:     2 {{.*}} [0x00000000, 0x00000018) [0x00000000, 0x00000004) [0x00000000, 0x0000001d)
+
   .section  .debug_info.dwo,"e",@progbits
   .long .Ldebug_info_dwo_end0-.Ldebug_info_dwo_start0 # Length of Unit
 .Ldebug_info_dwo_start0:
   .short  5                               # DWARF version number
   .byte 6                               # DWARF Unit Type
   .byte 8                               # Address Size (in bytes)
   .long 0                               # Offset Into Abbrev. Section
   .quad -4287463584810542331            # Type Signature

llvm/test/tools/llvm-dwp/X86/rnglists.s

 # This test checks if llvm-dwp outputs .debug_rnglists.
 
 # RUN: llvm-mc -triple x86_64-unknown-linux %s -filetype=obj -o %t.o \
 # RUN:         -split-dwarf-file=%t.dwo -dwarf-version=5
 # RUN: llvm-dwp %t.dwo -o %t.dwp
-# RUN: llvm-dwarfdump -debug-rnglists -debug-cu-index -debug-tu-index %t.dwp | FileCheck %s
+# RUN: llvm-dwarfdump -debug-rnglists -debug-cu-index -debug-tu-index %t.dwp | FileCheck -check-prefix=CHECK %s
+# RUN: llvm-dwarfdump -debug-cu-index -debug-tu-index -manaully-generate-cu-tu-index %t.dwp | FileCheck -check-prefix=CHECK2 %s
 
 # CHECK-DAG: .debug_cu_index contents:
 # CHECK: Index Signature          INFO                     ABBREV                   RNGLISTS
 # CHECK:     1 {{.*}} [0x00000018, 0x0000002d) [0x00000000, 0x00000004) [0x00000000, 0x00000017)
 
 # CHECK-DAG: .debug_tu_index contents:
 # CHECK: Index Signature          INFO                     ABBREV                   RNGLISTS
 # CHECK:     2 {{.*}} [0x00000000, 0x00000018) [0x00000000, 0x00000004) [0x00000000, 0x00000017)
 
 # CHECK-DAG: .debug_rnglists.dwo contents:
 # range list header: length = 0x00000013, format = DWARF32, version = 0x0005, addr_size = 0x08, seg_size = 0x00, offset_entry_count = 0x00000001
 # CHECK: offsets: [
 # CHECK-NEXT: 0x00000004
 # CHECK-NEXT: ]
 # CHECK-NEXT: ranges:
 # CHECK-NEXT: [0x0000000000000004, 0x0000000000000008)
 # CHECK-NEXT: [0x000000000000000c, 0x0000000000000010)
 
+# CHECK2-DAG: .debug_cu_index contents:
+# CHECK2: Index Signature          INFO                     ABBREV                   RNGLISTS
+# CHECK2:     1 {{.*}} [0x00000018, 0x0000002d) [0x00000000, 0x00000004) [0x00000000, 0x00000017)
+
+# CHECK2-DAG: .debug_tu_index contents:
+# CHECK2: Index Signature          INFO                     ABBREV                   RNGLISTS
+# CHECK2:     2 {{.*}} [0x00000000, 0x00000018) [0x00000000, 0x00000004) [0x00000000, 0x00000017)
+
   .section  .debug_info.dwo,"e",@progbits
   .long .Ldebug_info_dwo_end0-.Ldebug_info_dwo_start0 # Length of Unit
 .Ldebug_info_dwo_start0:
   .short  5                               # DWARF version number
   .byte 6                               # DWARF Unit Type
   .byte 8                               # Address Size (in bytes)
   .long 0                               # Offset Into Abbrev. Section
   .quad -4287463584810542331            # Type Signature

llvm/test/tools/llvm-dwp/X86/tu_units_v5.s

 # This test checks if llvm-dwp can correctly generate the tu index section (v5).
 
 # RUN: llvm-mc -triple x86_64-unknown-linux %s -filetype=obj -o %t.o \
 # RUN:         -split-dwarf-file=%t.dwo -dwarf-version=5
 # RUN: llvm-dwp %t.dwo -o %t.dwp
-# RUN: llvm-dwarfdump -debug-info -debug-tu-index %t.dwp | FileCheck %s
+# RUN: llvm-dwarfdump -debug-info -debug-tu-index %t.dwp | FileCheck -check-prefix=CHECK %s
+# RUN2: llvm-dwarfdump -debug-info -debug-tu-index -manaully-generate-cu-tu-index %t.dwp | FileCheck -check-prefix=CHECK2 %s
 
 ## Note: In order to check whether the type unit index is generated
 ## there is no need to add the missing DIEs for the structure type of the type unit.
 
 # CHECK-DAG: .debug_info.dwo contents:
 # CHECK: 0x00000000: Type Unit: length = 0x00000017, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_type, abbr_offset = 0x0000, addr_size = 0x08, name = '', type_signature = [[TUID1:.*]], type_offset = 0x0019 (next unit at 0x0000001b)
 # CHECK: 0x0000001b: Type Unit: length = 0x00000017, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_type, abbr_offset = 0x0000, addr_size = 0x08, name = '', type_signature = [[TUID2:.*]], type_offset = 0x0019 (next unit at 0x00000036)
 # CHECK-DAG: .debug_tu_index contents:
 # CHECK: version = 5, units = 2, slots = 4
 # CHECK: Index Signature          INFO                     ABBREV
 # CHECK:     1 [[TUID1]]          [0x00000000, 0x0000001b) [0x00000000, 0x00000010)
 # CHECK:     4 [[TUID2]]          [0x0000001b, 0x00000036) [0x00000000, 0x00000010)
 
+# CHECK2-DAG: .debug_info.dwo contents:
+# CHECK2: 0x00000000: Type Unit: length = 0x00000017, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_type, abbr_offset = 0x0000, addr_size = 0x08, name = '', type_signature = [[TUID1:.*]], type_offset = 0x0019 (next unit at 0x0000001b)
+# CHECK2: 0x0000001b: Type Unit: length = 0x00000017, format = DWARF32, version = 0x0005, unit_type = DW_UT_split_type, abbr_offset = 0x0000, addr_size = 0x08, name = '', type_signature = [[TUID2:.*]], type_offset = 0x0019 (next unit at 0x00000036)
+# CHECK2-DAG: .debug_tu_index contents:
+# CHECK2: version = 5, units = 2, slots = 4
+# CHECK2: Index Signature          INFO                     ABBREV
+# CHECK2:     1 [[TUID1]]          [0x00000000, 0x0000001b) [0x00000000, 0x00000010)
+# CHECK2:     4 [[TUID2]]          [0x0000001b, 0x00000036) [0x00000000, 0x00000010)
+
     .section  .debug_info.dwo,"e",@progbits
     .long .Ldebug_info_dwo_end0-.Ldebug_info_dwo_start0 # Length of Unit
 .Ldebug_info_dwo_start0:
     .short  5                               # DWARF version number
     .byte 6                               # DWARF Unit Type (DW_UT_split_type)
     .byte 8                               # Address Size (in bytes)
     .long 0                               # Offset Into Abbrev. Section
     .quad 5657452045627120676             # Type Signature

llvm/test/tools/llvm-dwp/X86/type_dedup.test

 RUN: llvm-dwp %p/../Inputs/type_dedup/a.dwo %p/../Inputs/type_dedup/b.dwo -o %t
-RUN: llvm-dwarfdump -v %t | FileCheck %s
+RUN: llvm-dwarfdump -v %t | FileCheck -check-prefix=CHECK %s
+RUN: llvm-dwarfdump -v -manaully-generate-cu-tu-index %t | FileCheck -check-prefix=CHECK2 %s
 RUN: llvm-dwp %p/../Inputs/type_dedup/b.dwo -o %tb.dwp
 RUN: llvm-dwp %p/../Inputs/type_dedup/a.dwo %tb.dwp -o %t
-RUN: llvm-dwarfdump -v %t | FileCheck %s
+RUN: llvm-dwarfdump -v %t | FileCheck -check-prefix=CHECK %s
+RUN: llvm-dwarfdump -v -manaully-generate-cu-tu-index %t | FileCheck -check-prefix=CHECK2 %s
 
 a.cpp:
   struct common { };
   common a1;
   struct adistinct { };
   adistinct a2;
 
 b.cpp:

 CHECK:                 DW_AT_name {{.*}} "adistinct"
 CHECK: [[BUOFF]]:
 CHECK-LABEL: Type Unit: length = 0x00000020, format = DWARF32, version = 0x0004, abbr_offset =
 CHECK:         0x{{.*}}, addr_size = 0x08, name = 'bdistinct', type_signature = [[BSIG:0x[0-9a-f]*]], type_offset = 0x[[BOFF:.*]] (next unit at [[XUOFF:.*]])
 CHECK:             DW_TAG_type_unit
 CHECK: 0x00000066:   DW_TAG_structure_type
 CHECK:                 DW_AT_name {{.*}} "bdistinct"
 CHECK-NOT: Type Unit
+
+CHECK2-LABEL: .debug_types.dwo contents:
+CHECK2: [[COMMONUOFF:0x[0-9a-f]*]]:
+CHECK2-LABEL: Type Unit: length = 0x00000020, format = DWARF32, version = 0x0004, abbr_offset =
+CHECK2:         0x0000, addr_size = 0x08, name = 'common', type_signature = [[COMMONSIG:0x[0-9a-f]*]], type_offset = 0x[[COMMONOFF:.*]] (next unit at [[AUOFF:.*]])
+CHECK2:                DW_TAG_type_unit
+CHECK2: [[COMMONOFF]]:   DW_TAG_structure_type
+CHECK2:                    DW_AT_name {{.*}} "common"
+CHECK2: [[AUOFF]]:
+CHECK2-LABEL: Type Unit: length = 0x00000020, format = DWARF32, version = 0x0004, abbr_offset =
+CHECK2:         0x0000, addr_size = 0x08, name = 'adistinct', type_signature = [[ASIG:0x[0-9a-f]*]], type_offset = 0x[[AOFF:.*]] (next unit at [[BUOFF:.*]])
+CHECK2:             DW_TAG_type_unit
+CHECK2: 0x00000042:   DW_TAG_structure_type
+CHECK2:                 DW_AT_name {{.*}} "adistinct"
+CHECK2: [[BUOFF]]:
+CHECK2-LABEL: Type Unit: length = 0x00000020, format = DWARF32, version = 0x0004, abbr_offset =
+CHECK2:         0x{{.*}}, addr_size = 0x08, name = 'bdistinct', type_signature = [[BSIG:0x[0-9a-f]*]], type_offset = 0x[[BOFF:.*]] (next unit at [[XUOFF:.*]])
+CHECK2:             DW_TAG_type_unit
+CHECK2: 0x00000066:   DW_TAG_structure_type
+CHECK2:                 DW_AT_name {{.*}} "bdistinct"
+CHECK2-NOT: Type Unit

llvm/tools/llvm-dwarfdump/llvm-dwarfdump.cpp

static cl::opt<bool>
     ShowSectionSizes("show-section-sizes",
                      cl::desc("Show the sizes of all debug sections, "
                               "expressed in bytes."),
                      cat(DwarfDumpCategory));
 static cl::opt<bool>
     ShowSources("show-sources",
                 cl::desc("Show the sources across all compilation units."),
                 cat(DwarfDumpCategory));
+static cl::opt<bool>
+    ManuallyGenerateCUIndex("manaully-generate-cu-tu-index",
+                            cl::desc("if the input is dwp file, parse debug "
+                                     "info section and use it to populate "
+                                     "DEBUG_INFO contributions in cu-index. "
+                                     "For DWARF5 it also populated TU Index."),
+                            cl::init(false), cl::Hidden,
+                            cl::cat(DwarfDumpCategory));
 // facebook begin D13311561
 static cl::opt<bool>
     Analyze("analyze",
             cl::desc("Analyze the DWARF and report detailed information."),
             cl::cat(AnalyzeCategory));
 // facebook end
 static opt<bool> Verify("verify", desc("Verify the DWARF debug info."),
                         cat(DwarfDumpCategory));
auto RecoverableErrorHandler = [&](Error E) {
     Result = false;
     WithColor::defaultErrorHandler(std::move(E));
   };
   if (auto *Obj = dyn_cast<ObjectFile>(BinOrErr->get())) {
     if (filterArch(*Obj)) {
       std::unique_ptr<DWARFContext> DICtx = DWARFContext::create(
           *Obj, DWARFContext::ProcessDebugRelocations::Process, nullptr, "",
           RecoverableErrorHandler);
+      DICtx->setParseCUTUIndexManually(ManuallyGenerateCUIndex);
       if (!HandleObj(*Obj, *DICtx, Filename, OS))
         Result = false;
     }
   } else if (auto *Fat = dyn_cast<MachOUniversalBinary>(BinOrErr->get()))
     for (auto &ObjForArch : Fat->objects()) {
       std::string ObjName =
           (Filename + "(" + ObjForArch.getArchFlagName() + ")").str();
       if (auto MachOOrErr = ObjForArch.getAsObjectFile()) {