Initial DWARF64 support for .debug_line

[emaste]

Summary

Other DWARF sections still need to be done, as well as 64-bit offsets in DataExtractor also needed (as described DebugFrame).

http://llvm.org/bugs/show_bug.cgi?id=17411

[dblaikie]

Could you include some test cases?

Since LLVM cannot produce DWARF64 yet (so far as I know) you may need to
generate them with GCC (you'll find other binary tests in
test/DebugInfo/Inputs/dwarfdump-* (with corresponding tests like
test/DebugInfo/dwarfdump-test.test)). Please include comments about which
version of GCC and what flags were used to generate the binaries.

[emaste]

I compiled the existing dwarfdump-inl-test.cc for FreeBSD mips64:

% g++ --version
g++ (GCC) 4.2.1 20070831 patched [FreeBSD]
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
% g++ -g dwarfdump-inl-test.cc -o dwarfdump-inl-test.freebsd-elf-mips64

current state:

% bin/llvm-dwarfdump --debug-dump=line dwarfdump-inl-test.freebsd-elf-mips64
dwarfdump-inl-test.freebsd-elf-mips64:  file format ELF64-unknown


.debug_line contents:
%

with this patch:

% bin/llvm-dwarfdump --debug-dump=line dwarfdump-inl-test.freebsd-elf-mips64
dwarfdump-inl-test.freebsd-elf-mips64:  file format ELF64-unknown


.debug_line contents:
Line table prologue:
   total_length: 0x00000212
        version: 2
prologue_length: 0x000001ab
min_inst_length: 1
default_is_stmt: 1
      line_base: -5
     line_range: 14
    opcode_base: 13
...

this only outputs the first CompileUnit though, perhaps because the parsing fails in another section in DWARFContext::parseCompileUnits(). Is it reasonable to add this dwarfdump-inl-test.freebsd-elf-mips64 binary and a basic test, fleshing it out when DWARF64 parsing for other sections is added?

(the test file is 15638 bytes -- larger than the clang-compiled version, due to the lack of -gline-tables-only)

[echristo]

Is this actually using 64-bit dwarf or is it dwarf with 64-bit
pointers? If it's the former ... then how big is the testcase? It
really shouldn't be creating dwarf64 unless the references in the
object are large (not the size of a pointer) unless this is dwarf2.

-eric

[emaste]

This patch addresses the output of GCC and gas on FreeBSD mips64

# gcc --version
gcc (GCC) 4.2.1 20070831 patched [FreeBSD]
# as --version
GNU assembler 2.17.50 [FreeBSD] 2007-07-03

This toolchain generally outputs dwarf2 but uses the 64-bit format from dwarf3. This is the format that uses 0xffffffff in the initial length field, followed by a 64-bit length, and 64-bit offsets throughout. Equivalent support already exists in DWARFDebugFrame.cpp.

My testcase binary is tiny; 64-bit is not necessary. The (older) version of GCC and gas in FreeBSD output this format for some reason; I do not know if more recent versions still do.

for whatever reason the toolchain insists on

[echristo]

That's really a shame. For dwarf3 it shouldn't generate 64-bit dwarf
unless the offsets in the file require 64-bits to represent. For
dwarf2 it's based on pointer size. Seems like a bug. Which version of
gcc is this?

-eric

[emaste]

The version was in the Phabricator review or email :-)
gcc 4.2.1, gas "2.17.50"

I agree that this is a shame, but it's what we've got. I haven't investigated in detail, but quick Googling suggests that the GNU toolchain switched n64 Linux to DWARF32 in 2006; I did not find evidence that the same change was made for FreeBSD.

[emaste]

It appears the GNU toolchain was changed to default to dwarf32 on all MIPS platforms sometime later, and I will make a similar change to the FreeBSD in-tree toolchain. That said, we still have MIPS binaries and libraries built with dwarf64 data today.

[echristo]

Excellent. Thanks.

[emaste]

FreeBSD in-tree toolchain updated in svn r256859:
http://svnweb.freebsd.org/base?view=revision&revision=256859

Shall I still proceed with this patch? We (a) have objects with 64-bit dwarf .debug_line already due to the silly default in older toolchains, and (b) may eventually encounter an object that actually requires this support.

[echristo]

The patch itself is ok, though it needs a few things:

a) comments on what is and is not handled for dwarf64. The compile unit header is probably the best place to document that there's at least some partial support for reading line tables.

b) testcase. I'm ok with adding the testcase and a comment on it that dwarf64 parsing isn't very good at this point and that the only thing that's expected to work are the line tables. You can, for now, just dump the line table in the testcase.

[emaste]

Added comments and a basic test

(Comments are in DWARFContext.h as that seemed to be a more clear reference to the individual classes.)

[echristo]

One comment, otherwise it looks ok.

lib/DebugInfo/DWARFDebugLine.cpp

220

Not sure I understand the error here.

[echristo]

(Sorry for the delay)

[emaste]

lib/DebugInfo/DWARFDebugLine.cpp

220

gas has a bug that produces an incorrect prologue_length for 64-bit DWARF. As it turns out we can handle that condition, so we can still return true.

I fixed the underlying bug in FreeBSD here:
http://svnweb.freebsd.org/base?view=revision&revision=256692
and your note reminded me that I never forwarded this to the binutils upstream, so I've now posted to their mailing list as well.

Maybe just drop the message?

[echristo]

Sounds good to me.

Thanks!

[echristo]

Was this ever committed? Any reason why not?

[emaste]

Rebase

[emaste]

It just fell off my radar during the previous iterations. I subsequently committed an improved version of the change to LLDB - that's in http://llvm-reviews.chandlerc.com/D2007 .

We'll want the equivalent change here too, but I think it makes sense to bring in this by itself since it's already done. It wasn't a trivial rebase against head so I'll commit if I can get a review of the new diff.

[samsonov]

I'm ok with this patch.

test/DebugInfo/dwarfdump-64-bit-dwarf.test

1 ↗

(On Diff #10244)

Is "inl-test" necessary in binary name? If not, please rename it.

3 ↗

(On Diff #10244)

Please add a comment about limited DWARF-64 parsing capabilities.

[emaste]

test/DebugInfo/dwarfdump-64-bit-dwarf.test

1 ↗

(On Diff #10244)

I named it this only because it was built from dwarfdump-inl-test.cc

The actual source isn't really relevant though, so I can just call it dwarfdump.elf-mips64-64-bit-dwarf if that's preferable.

[emaste]

Add comment and rename input binary file

[samsonov]

LGTM

[emaste]

correct arg types for printf

[emaste]

Rebase

[emaste]

Sorry this slipped through the cracks for so long. I'll commit shortly, but have uploaded a rebased version of the change for now.

lib/DebugInfo/DWARFDebugLine.h

struct FileNameEntry {
     const char *Name;
     uint64_t DirIdx;
     uint64_t ModTime;
     uint64_t Length;
   };
 
   struct Prologue {
     Prologue()
-      : TotalLength(0), Version(0), PrologueLength(0), MinInstLength(0),
-        DefaultIsStmt(0), LineBase(0), LineRange(0), OpcodeBase(0) {}
+      : IsDWARF64(0), TotalLength(0), Version(0), PrologueLength(0),
+        MinInstLength(0), DefaultIsStmt(0), LineBase(0), LineRange(0),
+        OpcodeBase(0) {}
 
+    // Prologue represents 64-bit DWARF
+    bool IsDWARF64;
     // The size in bytes of the statement information for this compilation unit
     // (not including the total_length field itself).
-    uint32_t TotalLength;
+    uint64_t TotalLength;
     // Version identifier for the statement information format.
     uint16_t Version;
     // The number of bytes following the prologue_length field to the beginning
     // of the first byte of the statement program itself.
-    uint32_t PrologueLength;
+    uint64_t PrologueLength;
     // The size in bytes of the smallest target machine instruction. Statement
     // program opcodes that alter the address register first multiply their
     // operands by this value.
     uint8_t MinInstLength;
     // The initial value of theis_stmtregister.
     uint8_t DefaultIsStmt;
     // This parameter affects the meaning of the special opcodes. See below.
     int8_t LineBase;
     // This parameter affects the meaning of the special opcodes. See below.
     uint8_t LineRange;
     // The number assigned to the first special opcode.
     uint8_t OpcodeBase;
     std::vector<uint8_t> StandardOpcodeLengths;
     std::vector<const char*> IncludeDirectories;
     std::vector<FileNameEntry> FileNames;
 
+    uint32_t sizeofTotalLength() const {
+      return IsDWARF64 ? 12 : 4;
+    }
+    uint32_t sizeofPrologueLength() const {
+      return IsDWARF64 ? 8 : 4;
+    }
+
     // Length of the prologue in bytes.
     uint32_t getLength() const {
-      return PrologueLength + sizeof(TotalLength) + sizeof(Version) +
-             sizeof(PrologueLength);
+      return PrologueLength + sizeofTotalLength() + sizeof(Version) +
+             sizeofPrologueLength();
     }
     // Length of the line table data in bytes (not including the prologue).
     uint32_t getStatementTableLength() const {
-      return TotalLength + sizeof(TotalLength) - getLength();
+      return TotalLength + sizeofTotalLength() - getLength();
     }
     int32_t getMaxLineIncrementForSpecialOpcode() const {
       return LineBase + (int8_t)LineRange - 1;
     }
     void dump(raw_ostream &OS) const;
     void clear() {
+      IsDWARF64 = false;
       TotalLength = Version = PrologueLength = 0;
       MinInstLength = LineBase = LineRange = OpcodeBase = 0;
       StandardOpcodeLengths.clear();
       IncludeDirectories.clear();
       FileNames.clear();
     }
   };
 

lib/DebugInfo/DWARFDebugLine.cpp

 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 using namespace llvm;
 using namespace dwarf;
 
 void DWARFDebugLine::Prologue::dump(raw_ostream &OS) const {
   OS << "Line table prologue:\n"
-     << format("   total_length: 0x%8.8x\n", TotalLength)
+     << format("   total_length: 0x%8.8" PRIx64 "\n", TotalLength)
      << format("        version: %u\n", Version)
-     << format("prologue_length: 0x%8.8x\n", PrologueLength)
+     << format("prologue_length: 0x%8.8" PRIx64 "\n", PrologueLength)
      << format("min_inst_length: %u\n", MinInstLength)
      << format("default_is_stmt: %u\n", DefaultIsStmt)
      << format("      line_base: %i\n", LineBase)
      << format("     line_range: %u\n", LineRange)
      << format("    opcode_base: %u\n", OpcodeBase);
 
   for (uint32_t i = 0; i < StandardOpcodeLengths.size(); ++i)
     OS << format("standard_opcode_lengths[%s] = %u\n", LNStandardString(i+1),
if (pos.second) {
     pos.first->second = state;
   }
   return &pos.first->second;
 }
 
 bool
 DWARFDebugLine::parsePrologue(DataExtractor debug_line_data,
                               uint32_t *offset_ptr, Prologue *prologue) {
-  const uint32_t prologue_offset = *offset_ptr;
+  const uint64_t prologue_offset = *offset_ptr;
 
   prologue->clear();
   prologue->TotalLength = debug_line_data.getU32(offset_ptr);
+  if (prologue->TotalLength == UINT32_MAX) {
+    prologue->IsDWARF64 = true;
+    prologue->TotalLength = debug_line_data.getU64(offset_ptr);
+  } else if (prologue->TotalLength >= 0xffffff00) {
+    return false;
+  }
   prologue->Version = debug_line_data.getU16(offset_ptr);
   if (prologue->Version != 2)
     return false;
 
-  prologue->PrologueLength = debug_line_data.getU32(offset_ptr);
-  const uint32_t end_prologue_offset = prologue->PrologueLength + *offset_ptr;
+  prologue->PrologueLength = debug_line_data.getUnsigned(offset_ptr,
+                                              prologue->sizeofPrologueLength());
+  const uint64_t end_prologue_offset = prologue->PrologueLength + *offset_ptr;
   prologue->MinInstLength = debug_line_data.getU8(offset_ptr);
   prologue->DefaultIsStmt = debug_line_data.getU8(offset_ptr);
   prologue->LineBase = debug_line_data.getU8(offset_ptr);
   prologue->LineRange = debug_line_data.getU8(offset_ptr);
   prologue->OpcodeBase = debug_line_data.getU8(offset_ptr);
 
   prologue->StandardOpcodeLengths.reserve(prologue->OpcodeBase-1);
   for (uint32_t i = 1; i < prologue->OpcodeBase; ++i) {
if (name && name[0]) {
       fileEntry.ModTime = debug_line_data.getULEB128(offset_ptr);
       fileEntry.Length = debug_line_data.getULEB128(offset_ptr);
       prologue->FileNames.push_back(fileEntry);
     } else {
       break;
     }
   }
 
-  if (*offset_ptr != end_prologue_offset) {
-    fprintf(stderr, "warning: parsing line table prologue at 0x%8.8x should"
-                    " have ended at 0x%8.8x but it ended at 0x%8.8x\n",
-            prologue_offset, end_prologue_offset, *offset_ptr);
+  if (prologue->IsDWARF64 && end_prologue_offset - *offset_ptr == 12) {
+    fprintf(stderr, "warning: apparent 64-bit DWARF GNU as prologue length bug"
+                    " detected (prologue length 12 bytes too long)\n");
+    // Could correct the field: prologue->PrologueLength -= 12;
+  } else if (*offset_ptr != end_prologue_offset) {
+    fprintf(stderr, "warning: parsing line table prologue at 0x%8.8" PRIx64
+                    " should have ended at 0x%8.8" PRIx64
+                    " but it ended at 0x%8.8" PRIx64 "\n",
+            prologue_offset, end_prologue_offset, (uint64_t)*offset_ptr);
     return false;
   }
   return true;
 }
 
 bool
 DWARFDebugLine::parseStatementTable(DataExtractor debug_line_data, 
                                     const RelocAddrMap *RMap,
                                     uint32_t *offset_ptr, State &state) {
   const uint32_t debug_line_offset = *offset_ptr;
 
   Prologue *prologue = &state.Prologue;
 
   if (!parsePrologue(debug_line_data, offset_ptr, prologue)) {
     // Restore our offset and return false to indicate failure!
     *offset_ptr = debug_line_offset;
     return false;
   }
 
   const uint32_t end_offset = debug_line_offset + prologue->TotalLength +
-                              sizeof(prologue->TotalLength);
+                              prologue->sizeofTotalLength();
 
   state.reset();
 
   while (*offset_ptr < end_offset) {
     uint8_t opcode = debug_line_data.getU8(offset_ptr);
 
     if (opcode == 0) {
       // Extended Opcodes always start with a zero opcode followed by