[llvm-dwarfdump] - Teach dwarfdump to decompress debug sections in zlib style.

[grimar]

Summary

Before this llvm-dwarfdump only recognized zlib-gnu compression style of headers,
this patch adds support for zlib style.
It looks reasonable to support both styles for dumping,
even if we are not going to suport generating of deprecated gnu one.

[llvm-commits]

Commit message typo: "depricated" should be "deprecated". Probably also
missing a "we" before [or use "it is" instead of] "are" in the last line.

Mats

• msg-12333-130.txt162 BDownload

[dblaikie]

Generally looks good. I'm not sure about renaming the zdebug section to debug in the output, but I'm not set against it either.

lib/DebugInfo/DWARF/DWARFContext.cpp

626 ↗

(On Diff #57931)

You could just increment Offset (by sizeof(Elf64_Word)) here, I think?

681–685 ↗

(On Diff #57931)

This involves making extra copies of the data. Is there a reason you went with this over the original code that decompressed straight into the UncompressedSections?

[grimar]

• Addressed review comments.

lib/DebugInfo/DWARF/DWARFContext.cpp

626 ↗

(On Diff #57931)

Right.

681–685 ↗

(On Diff #57931)

My point was to avoid possible calls of pop_back right after resize if decompression fails.
That should be rare case (I think decompression fail is infrequent, except cases of disabled zlib (now check for that is inside tryDecompress)), but code itself looks bit overcomplicated.

I just forgot to use std::move() to avoid extras when rewrote. Fixed. There should be no more extra heap allocations here.

[grimar]

Except this defeats/comes at a cost of the small vector optimization - std::move on a small vector in small mode is the same as a copy.

Sorry, perhaps I should've commented the original code to explain/justify it better.

• David

Small mode for small vector in this place is when it's size less than 32 bytes, right ?
Should we really care of this for the sake of simplicity of code ? I mean that for debug sections 32 bytes
is probably almost never a option, they usually should be greater anyways I think.

Just in case - I plan to revert to initial logic of this soon and update the patch, as it does not affect the main logic
and we can land patch separatelly and return to this place later if we want to.

George.

[grimar]

• Restored original way to work with UncompressedSections vector, it is unrelated to this patch.

[dblaikie]

Looks good - thanks!

test/DebugInfo/Inputs/dwarfdump-test-zlib.cc

24 ↗

(On Diff #58084)

Might be worth including in the comment some detail about how you verified that the output did actually contain compressed sections (just that the size of the output is smaller, for example) in both cases? (I realize this was missing originally as well, but some paper trail seems helpful to add)

llvm/trunk/include/llvm/Object/COFF.h

protected:
   void moveSectionNext(DataRefImpl &Sec) const override;
   std::error_code getSectionName(DataRefImpl Sec,
                                  StringRef &Res) const override;
   uint64_t getSectionAddress(DataRefImpl Sec) const override;
   uint64_t getSectionSize(DataRefImpl Sec) const override;
   std::error_code getSectionContents(DataRefImpl Sec,
                                      StringRef &Res) const override;
   uint64_t getSectionAlignment(DataRefImpl Sec) const override;
+  bool isSectionCompressed(DataRefImpl Sec) const override;
   bool isSectionText(DataRefImpl Sec) const override;
   bool isSectionData(DataRefImpl Sec) const override;
   bool isSectionBSS(DataRefImpl Sec) const override;
   bool isSectionVirtual(DataRefImpl Sec) const override;
   relocation_iterator section_rel_begin(DataRefImpl Sec) const override;
   relocation_iterator section_rel_end(DataRefImpl Sec) const override;
 
   void moveRelocationNext(DataRefImpl &Rel) const override;

llvm/trunk/include/llvm/Object/ELFObjectFile.h

protected:
   void moveSectionNext(DataRefImpl &Sec) const override;
   std::error_code getSectionName(DataRefImpl Sec,
                                  StringRef &Res) const override;
   uint64_t getSectionAddress(DataRefImpl Sec) const override;
   uint64_t getSectionSize(DataRefImpl Sec) const override;
   std::error_code getSectionContents(DataRefImpl Sec,
                                      StringRef &Res) const override;
   uint64_t getSectionAlignment(DataRefImpl Sec) const override;
+  bool isSectionCompressed(DataRefImpl Sec) const override;
   bool isSectionText(DataRefImpl Sec) const override;
   bool isSectionData(DataRefImpl Sec) const override;
   bool isSectionBSS(DataRefImpl Sec) const override;
   bool isSectionVirtual(DataRefImpl Sec) const override;
   relocation_iterator section_rel_begin(DataRefImpl Sec) const override;
   relocation_iterator section_rel_end(DataRefImpl Sec) const override;
   section_iterator getRelocatedSection(DataRefImpl Sec) const override;
 

 }
 
 template <class ELFT>
 uint64_t ELFObjectFile<ELFT>::getSectionAlignment(DataRefImpl Sec) const {
   return getSection(Sec)->sh_addralign;
 }
 
 template <class ELFT>
+bool ELFObjectFile<ELFT>::isSectionCompressed(DataRefImpl Sec) const {
+  return getSection(Sec)->sh_flags & ELF::SHF_COMPRESSED;
+}
+
+template <class ELFT>
 bool ELFObjectFile<ELFT>::isSectionText(DataRefImpl Sec) const {
   return getSection(Sec)->sh_flags & ELF::SHF_EXECINSTR;
 }
 
 template <class ELFT>
 bool ELFObjectFile<ELFT>::isSectionData(DataRefImpl Sec) const {
   const Elf_Shdr *EShdr = getSection(Sec);
   return EShdr->sh_flags & (ELF::SHF_ALLOC | ELF::SHF_WRITE) &&

llvm/trunk/include/llvm/Object/MachO.h

public:
   void moveSectionNext(DataRefImpl &Sec) const override;
   std::error_code getSectionName(DataRefImpl Sec,
                                  StringRef &Res) const override;
   uint64_t getSectionAddress(DataRefImpl Sec) const override;
   uint64_t getSectionSize(DataRefImpl Sec) const override;
   std::error_code getSectionContents(DataRefImpl Sec,
                                      StringRef &Res) const override;
   uint64_t getSectionAlignment(DataRefImpl Sec) const override;
+  bool isSectionCompressed(DataRefImpl Sec) const override;
   bool isSectionText(DataRefImpl Sec) const override;
   bool isSectionData(DataRefImpl Sec) const override;
   bool isSectionBSS(DataRefImpl Sec) const override;
   bool isSectionVirtual(DataRefImpl Sec) const override;
   bool isSectionBitcode(DataRefImpl Sec) const override;
   relocation_iterator section_rel_begin(DataRefImpl Sec) const override;
   relocation_iterator section_rel_end(DataRefImpl Sec) const override;
 

llvm/trunk/include/llvm/Object/ObjectFile.h

public:
   std::error_code getName(StringRef &Result) const;
   uint64_t getAddress() const;
   uint64_t getSize() const;
   std::error_code getContents(StringRef &Result) const;
 
   /// @brief Get the alignment of this section as the actual value (not log 2).
   uint64_t getAlignment() const;
 
+  bool isCompressed() const;
   bool isText() const;
   bool isData() const;
   bool isBSS() const;
   bool isVirtual() const;
   bool isBitcode() const;
 
   bool containsSymbol(SymbolRef S) const;
 
protected:
   virtual void moveSectionNext(DataRefImpl &Sec) const = 0;
   virtual std::error_code getSectionName(DataRefImpl Sec,
                                          StringRef &Res) const = 0;
   virtual uint64_t getSectionAddress(DataRefImpl Sec) const = 0;
   virtual uint64_t getSectionSize(DataRefImpl Sec) const = 0;
   virtual std::error_code getSectionContents(DataRefImpl Sec,
                                              StringRef &Res) const = 0;
   virtual uint64_t getSectionAlignment(DataRefImpl Sec) const = 0;
+  virtual bool isSectionCompressed(DataRefImpl Sec) const = 0;
   virtual bool isSectionText(DataRefImpl Sec) const = 0;
   virtual bool isSectionData(DataRefImpl Sec) const = 0;
   virtual bool isSectionBSS(DataRefImpl Sec) const = 0;
   // A section is 'virtual' if its contents aren't present in the object image.
   virtual bool isSectionVirtual(DataRefImpl Sec) const = 0;
   virtual bool isSectionBitcode(DataRefImpl Sec) const;
   virtual relocation_iterator section_rel_begin(DataRefImpl Sec) const = 0;
   virtual relocation_iterator section_rel_end(DataRefImpl Sec) const = 0;

 inline std::error_code SectionRef::getContents(StringRef &Result) const {
   return OwningObject->getSectionContents(SectionPimpl, Result);
 }
 
 inline uint64_t SectionRef::getAlignment() const {
   return OwningObject->getSectionAlignment(SectionPimpl);
 }
 
+inline bool SectionRef::isCompressed() const {
+  return OwningObject->isSectionCompressed(SectionPimpl);
+}
+
 inline bool SectionRef::isText() const {
   return OwningObject->isSectionText(SectionPimpl);
 }
 
 inline bool SectionRef::isData() const {
   return OwningObject->isSectionData(SectionPimpl);
 }
 

llvm/trunk/lib/DebugInfo/DWARF/DWARFContext.cpp

 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/DebugInfo/DWARF/DWARFAcceleratorTable.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugArangeSet.h"
 #include "llvm/DebugInfo/DWARF/DWARFUnitIndex.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/Dwarf.h"
+#include "llvm/Support/ELF.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 using namespace llvm;
 using namespace dwarf;
 using namespace object;
 
if (Spec.FLIKind != FileLineInfoKind::None) {
                                    CallColumn);
       }
     }
     InliningInfo.addFrame(Frame);
   }
   return InliningInfo;
 }
 
-static bool consumeCompressedDebugSectionHeader(StringRef &data,
-                                                uint64_t &OriginalSize) {
+static bool consumeCompressedGnuHeader(StringRef &data,
   // Consume "ZLIB" prefix.
   if (!data.startswith("ZLIB"))
     return false;
   data = data.substr(4);
   // Consume uncompressed section size (big-endian 8 bytes).
   DataExtractor extractor(data, false, 8);
   uint32_t Offset = 0;
   OriginalSize = extractor.getU64(&Offset);
   if (Offset == 0)
     return false;
   data = data.substr(Offset);
   return true;
 }
 
+static bool consumeCompressedZLibHeader(StringRef &Data, uint64_t &OriginalSize,
+                                        bool IsLE, bool Is64Bit) {
+  using namespace ELF;
+  uint64_t HdrSize = Is64Bit ? sizeof(Elf64_Chdr) : sizeof(Elf32_Chdr);
+  if (Data.size() < HdrSize)
+    return false;
+
+  DataExtractor Extractor(Data, IsLE, 0);
+  uint32_t Offset = 0;
+  if (Extractor.getUnsigned(&Offset, Is64Bit ? sizeof(Elf64_Word)
+                                             : sizeof(Elf32_Word)) !=
+      ELFCOMPRESS_ZLIB)
+    return false;
+
+  // Skip Elf64_Chdr::ch_reserved field.
+  if (Is64Bit)
+    Offset += sizeof(Elf64_Word);
+
+  OriginalSize = Extractor.getUnsigned(&Offset, Is64Bit ? sizeof(Elf64_Xword)
+                                                        : sizeof(Elf32_Word));
+  Data = Data.substr(HdrSize);
+  return true;
+}
+
+static bool tryDecompress(StringRef &Name, StringRef &Data,
+                          SmallString<32> &Out, bool ZLibStyle, bool IsLE,
+                          bool Is64Bit) {
+  if (!zlib::isAvailable())
+    return false;
+
+  uint64_t OriginalSize;
+  bool Result =
+      ZLibStyle ? consumeCompressedZLibHeader(Data, OriginalSize, IsLE, Is64Bit)
+                : consumeCompressedGnuHeader(Data, OriginalSize);
+
+  if (!Result || zlib::uncompress(Data, Out, OriginalSize) != zlib::StatusOK)
+    return false;
+
+  // gnu-style names are started from "z", consume that.
+  if (!ZLibStyle)
+    Name = Name.substr(1);
+  return true;
+}
+
 DWARFContextInMemory::DWARFContextInMemory(const object::ObjectFile &Obj,
     const LoadedObjectInfo *L)
     : IsLittleEndian(Obj.isLittleEndian()),
       AddressSize(Obj.getBytesInAddress()) {
   for (const SectionRef &Section : Obj.sections()) {
     StringRef name;
     Section.getName(name);
     // Skip BSS and Virtual sections, they aren't interesting.
for (const SectionRef &Section : Obj.sections()) {
     // Try to obtain an already relocated version of this section.
     // Else use the unrelocated section from the object file. We'll have to
     // apply relocations ourselves later.
     if (!L || !L->getLoadedSectionContents(*RelocatedSection,data))
       Section.getContents(data);
 
     name = name.substr(name.find_first_not_of("._")); // Skip . and _ prefixes.
 
-    // Check if debug info section is compressed with zlib.
-    if (name.startswith("zdebug_")) {
-      uint64_t OriginalSize;
-      if (!zlib::isAvailable() ||
-          !consumeCompressedDebugSectionHeader(data, OriginalSize))
+    bool ZLibStyleCompressed = Section.isCompressed();
+    if (ZLibStyleCompressed || name.startswith("zdebug_")) {
+      SmallString<32> Out;
+      if (!tryDecompress(name, data, Out, ZLibStyleCompressed, IsLittleEndian,
+                         AddressSize == 8))
         continue;
-      UncompressedSections.resize(UncompressedSections.size() + 1);
-      if (zlib::uncompress(data, UncompressedSections.back(), OriginalSize) !=
-          zlib::StatusOK) {
-        UncompressedSections.pop_back();
-        continue;
-      }
-      // Make data point to uncompressed section contents and save its contents.
-      name = name.substr(1);
+      UncompressedSections.emplace_back(std::move(Out));
       data = UncompressedSections.back();
     }
 
     StringRef *SectionData =
         StringSwitch<StringRef *>(name)
             .Case("debug_info", &InfoSection.Data)
             .Case("debug_abbrev", &AbbrevSection)
             .Case("debug_loc", &LocSection.Data)

llvm/trunk/lib/Object/COFFObjectFile.cpp

std::error_code COFFObjectFile::getSectionContents(DataRefImpl Ref,
   return EC;
 }
 
 uint64_t COFFObjectFile::getSectionAlignment(DataRefImpl Ref) const {
   const coff_section *Sec = toSec(Ref);
   return Sec->getAlignment();
 }
 
+bool COFFObjectFile::isSectionCompressed(DataRefImpl Sec) const {
+  return false;
+}
+
 bool COFFObjectFile::isSectionText(DataRefImpl Ref) const {
   const coff_section *Sec = toSec(Ref);
   return Sec->Characteristics & COFF::IMAGE_SCN_CNT_CODE;
 }
 
 bool COFFObjectFile::isSectionData(DataRefImpl Ref) const {
   const coff_section *Sec = toSec(Ref);
   return Sec->Characteristics & COFF::IMAGE_SCN_CNT_INITIALIZED_DATA;

llvm/trunk/lib/Object/MachOObjectFile.cpp

uint64_t MachOObjectFile::getSectionAlignment(DataRefImpl Sec) const {
   } else {
     MachO::section Sect = getSection(Sec);
     Align = Sect.align;
   }
 
   return uint64_t(1) << Align;
 }
 
+bool MachOObjectFile::isSectionCompressed(DataRefImpl Sec) const {
+  return false;
+}
+
 bool MachOObjectFile::isSectionText(DataRefImpl Sec) const {
   uint32_t Flags = getSectionFlags(this, Sec);
   return Flags & MachO::S_ATTR_PURE_INSTRUCTIONS;
 }
 
 bool MachOObjectFile::isSectionData(DataRefImpl Sec) const {
   uint32_t Flags = getSectionFlags(this, Sec);
   unsigned SectionType = Flags & MachO::SECTION_TYPE;

llvm/trunk/test/DebugInfo/Inputs/dwarfdump-test-zlib.cc

int f(int a, int b) {
   DummyClass c(a);
   return c.add(b);
 }
 
 int main() {
   return f(2, 3);
 }
 
-// Built with Clang 3.2 and ld.gold linker:
+// Built with Clang 3.9 and GNU gold (GNU Binutils for Ubuntu 2.26) 1.11:
 // $ mkdir -p /tmp/dbginfo
 // $ cp dwarfdump-test-zlib.cc /tmp/dbginfo
 // $ cd /tmp/dbginfo
-// $ clang++ -g dwarfdump-test-zlib.cc -Wl,--compress-debug-sections=zlib \
-//   -o <output>
+// $ clang++ -g dwarfdump-test-zlib.cc -Wl,--compress-debug-sections=zlib -o dwarfdump-test-zlib.elf-x86-64
+// $ clang++ -g dwarfdump-test-zlib.cc -Wl,--compress-debug-sections=zlib-gnu -o dwarfdump-test-zlibgnu.elf-x86-64
+// llvm-readobj -sections can be used to see that outputs really contain the compressed sections, also output in both
+//   cases is slightly smaller, that is because of compression.
 No newline at end of file

llvm/trunk/test/DebugInfo/dwarfdump-zlib.test

 REQUIRES: zlib
 
 RUN: llvm-dwarfdump %p/Inputs/dwarfdump-test-zlib.elf-x86-64 | FileCheck %s
+RUN: llvm-dwarfdump %p/Inputs/dwarfdump-test-zlibgnu.elf-x86-64 | FileCheck %s
 
 CHECK: .debug_abbrev contents
+
+// Dump content of a little section to check that both zlib and zlib gnu styles do
+// the decompression correctly and result is the same. This and above also checks
+// that sections names are properly shown in zlib-gnu style (without additional 'z' prefix).
 CHECK: .debug_info contents
+CHECK: 0x00000000: Compile Unit: length = 0x00000144 version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x00000148)