[DebugInfo] Make offsets of dwarf units 64-bit (19/19).

[ikudrin]

Summary

In the case of LTO, several DWARF units can be emitted in one section. For an extremely large application, they may exceed the limit of 4GiB for 32-bit offsets. As it is now possible to emit 64-bit debugging info, the patch enables storing the larger offsets.

[dblaikie]

Looks OK - open to discussion on the assert/report_fatal_error issue.

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

These might be report_fatal_error-worthy? (if there's ways to fail the build more nicely (without fatally stopping the program) that'd be nice - not sure how other attempts to try to use the AsmPrinter when the use is invalid fail/whether they can fail gracefully)

[jhenderson]

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

My personal thoughts:

1. If this code can actually be hit, it should be converted to return Error/Expected up to whatever point does the error handling.
2. report_fatal_error could be a temporary fallback if the above point is too big a piece of work for now, but shouldn't be the final state of the code (since report_fatal_error results essentially in a crash).
3. assert is most appropriate if this situation can't actually be hit (i.e. something else somewhere is protecting the limit).
4. It might make sense to add a check earlier in the code to allow the assert here if this assert could otherwise be triggered.

[ikudrin]

• Added reporting a fatal error if the generated debug info exceeds the limits supported by the format.

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

In the update, I am adding reporting a fatal error in DwarfFile::computeSizeAndOffsets(), where the offsets for DWARF units are calculated and set. As DWARF64 is not supported for Apple targets, these asserts are now not expected to be hit with the normal usage.

[jhenderson]

LGTM. Might want to give @dblaikie another chance to comment.

[dblaikie]

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

By "not supported" you mean "has no guarantees if you ask for it" or do you mean "will produce an error early telling the user this isn't valid/supported" & thus this assert should (assuming no other programmer bugs - but importantly /not/ assuming some well intentioned user - a user can ask for an invalid thing and should get an error, not an assert failure/crash) ?

[ikudrin]

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

I mean that DWARF64 is not enabled for anything except ELF (see D87011), while computeAccelTableKind() returns AccelTableKind::Apple only if TT.isOSBinFormatMachO(), or if an internal tool is called with -accel-tables=Apple switch. The Apple accelerator tables are emitted only if getAccelTableKind() returns AccelTableKind::Apple, thus I cannot see a way for an end-user to trigger emitting those tables when DWARF64 is on. And if DWARF64 is off, and the debug data is extremely big, an error will be reported in DwarfFile::computeSizeAndOffsets().

[dblaikie]

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

594–601

Ah, I see (just stating it here for the record: D87011 makes the DWARF64 flag a no-op/silently does nothing except on ELF targets). Yep, assert seems like the right thing here then. Thanks for walking me through it.

llvm/include/llvm/CodeGen/DIE.h

public:
   /// eventually used to call \a setAbbrevNumber().
   DIEAbbrev generateAbbrev() const;
 
   /// Set the abbreviation number for this DIE.
   void setAbbrevNumber(unsigned I) { AbbrevNumber = I; }
 
   /// Get the absolute offset within the .debug_info or .debug_types section
   /// for this DIE.
-  unsigned getDebugSectionOffset() const;
+  uint64_t getDebugSectionOffset() const;
 
   /// Compute the offset of this DIE and all its children.
   ///
   /// This function gets called just before we are going to generate the debug
   /// information and gives each DIE a chance to figure out its CU relative DIE
   /// offset, unique its abbreviation and fill in the abbreviation code, and
   /// return the unit offset that points to where the next DIE will be emitted
   /// within the debug unit section. After this function has been called for all
public:
   virtual const MCSymbol *getCrossSectionRelativeBaseAddress() const {
     return nullptr;
   }
 
   /// Return the section that this DIEUnit will be emitted into.
   ///
   /// \returns Section pointer which can be NULL.
   MCSection *getSection() const { return Section; }
-  void setDebugSectionOffset(unsigned O) { Offset = O; }
-  unsigned getDebugSectionOffset() const { return Offset; }
+  void setDebugSectionOffset(uint64_t O) { Offset = O; }
+  uint64_t getDebugSectionOffset() const { return Offset; }
   DIE &getUnitDie() { return Die; }
   const DIE &getUnitDie() const { return Die; }
 };
 
 struct BasicDIEUnit final : DIEUnit {
   explicit BasicDIEUnit(dwarf::Tag UnitTag) : DIEUnit(UnitTag) {}
 };
 

llvm/lib/CodeGen/AsmPrinter/AccelTable.cpp

llvm::function_ref<unsigned(const DWARF5AccelTableStaticData &)>
         getCUIndexForEntry) {
   Contents.finalize(Asm, "names");
   Dwarf5AccelTableWriter<DWARF5AccelTableStaticData>(Asm, Contents, CUs,
                                                      getCUIndexForEntry)
       .emit();
 }
 
 void AppleAccelTableOffsetData::emit(AsmPrinter *Asm) const {
+  assert(Die.getDebugSectionOffset() <= UINT32_MAX &&
+         "The section offset exceeds the limit.");
   Asm->emitInt32(Die.getDebugSectionOffset());
 }
 
 void AppleAccelTableTypeData::emit(AsmPrinter *Asm) const {
+  assert(Die.getDebugSectionOffset() <= UINT32_MAX &&
+         "The section offset exceeds the limit.");
   Asm->emitInt32(Die.getDebugSectionOffset());
   Asm->emitInt16(Die.getTag());
   Asm->emitInt8(0);
 }
 
 void AppleAccelTableStaticOffsetData::emit(AsmPrinter *Asm) const {
   Asm->emitInt32(Offset);
 }

llvm/lib/CodeGen/AsmPrinter/DIE.cpp

for (const DIEValue &V : values())
     if (V.getForm() == dwarf::DW_FORM_implicit_const)
       Abbrev.AddImplicitConstAttribute(V.getAttribute(),
                                        V.getDIEInteger().getValue());
     else
       Abbrev.AddAttribute(V.getAttribute(), V.getForm());
   return Abbrev;
 }
 
-unsigned DIE::getDebugSectionOffset() const {
+uint64_t DIE::getDebugSectionOffset() const {
   const DIEUnit *Unit = getUnit();
   assert(Unit && "DIE must be owned by a DIEUnit to get its absolute offset");
   return Unit->getDebugSectionOffset() + getOffset();
 }
 
 const DIE *DIE::getUnitDie() const {
   const DIE *p = this;
   while (p) {
case dwarf::DW_FORM_ref8:
     return;
 
   case dwarf::DW_FORM_ref_udata:
     AP->emitULEB128(Entry->getOffset());
     return;
 
   case dwarf::DW_FORM_ref_addr: {
     // Get the absolute offset for this DIE within the debug info/types section.
-    unsigned Addr = Entry->getDebugSectionOffset();
+    uint64_t Addr = Entry->getDebugSectionOffset();
     if (const MCSymbol *SectionSym =
             Entry->getUnit()->getCrossSectionRelativeBaseAddress()) {
       AP->emitLabelPlusOffset(SectionSym, Addr, SizeOf(AP, Form), true);
       return;
     }
 
     AP->OutStreamer->emitIntValue(Addr, SizeOf(AP, Form));
     return;

llvm/lib/CodeGen/AsmPrinter/DwarfFile.cpp

void DwarfFile::emitUnit(DwarfUnit *TheU, bool UseOffsets) {
 
   if (MCSymbol *EndLabel = TheU->getEndLabel())
     Asm->OutStreamer->emitLabel(EndLabel);
 }
 
 // Compute the size and offset for each DIE.
 void DwarfFile::computeSizeAndOffsets() {
   // Offset from the first CU in the debug info section is 0 initially.
-  unsigned SecOffset = 0;
+  uint64_t SecOffset = 0;
 
   // Iterate over each compile unit and set the size and offsets for each
   // DIE within each compile unit. All offsets are CU relative.
   for (const auto &TheU : CUs) {
     if (TheU->getCUNode()->isDebugDirectivesOnly())
       continue;
 
     // Skip CUs that ended up not being needed (split CUs that were abandoned
     // because they added no information beyond the non-split CU)
     if (llvm::empty(TheU->getUnitDie().values()))
       return;
 
     TheU->setDebugSectionOffset(SecOffset);
     SecOffset += computeSizeAndOffsetsForUnit(TheU.get());
   }
+  if (SecOffset > UINT32_MAX && !Asm->isDwarf64())
+    report_fatal_error("The generated debug information is too large "
+                       "for the 32-bit DWARF format.");
 }
 
 unsigned DwarfFile::computeSizeAndOffsetsForUnit(DwarfUnit *TheU) {
   // CU-relative offset is reset to 0 here.
   unsigned Offset = Asm->getUnitLengthFieldByteSize() + // Length of Unit Info
                     TheU->getHeaderSize();              // Unit-specific headers
 
   // The return value here is CU-relative, after laying out

llvm/unittests/DebugInfo/DWARF/DwarfGenerator.cpp

llvm::Error dwarfgen::Generator::init(Triple TheTriple, uint16_t V) {
   StringPool = std::make_unique<DwarfStringPool>(Allocator, *Asm, StringRef());
   StringOffsetsStartSym = Asm->createTempSymbol("str_offsets_base");
 
   return Error::success();
 }
 
 StringRef dwarfgen::Generator::generate() {
   // Offset from the first CU in the debug info section is 0 initially.
-  unsigned SecOffset = 0;
+  uint64_t SecOffset = 0;
 
   // Iterate over each compile unit and set the size and offsets for each
   // DIE within each compile unit. All offsets are CU relative.
   for (auto &CU : CompileUnits) {
     // Set the absolute .debug_info offset for this compile unit.
     CU->setOffset(SecOffset);
     // The DIEs contain compile unit relative offsets.
     unsigned CUOffset = 11;