[lib/Object] - Introduce Decompressor class.

[grimar]

Summary

Decompressor intention is to reduce duplication of code.
Currently LLD has own implementation of decompressor
for compressed debug sections.

This class helps to avoid it and share the code.
LLD patch for reusing it is D28106

[davide]

Overall decent, first round of comments inline.

include/llvm/DebugInfo/DWARF/DWARFCompression.h

37 ↗

(On Diff #82471)

Use an uniform style for comments. Also, doxygen.

40 ↗

(On Diff #82471)

Returns what? Probably you mean returns true?

lib/DebugInfo/DWARF/DWARFContext.cpp

608 ↗

(On Diff #82471)

decompress instead of uncompress maybe?

614 ↗

(On Diff #82471)

this deserves a comment or needs to be hidden behind and API

[dblaikie]

include/llvm/DebugInfo/DWARF/DWARFCompression.h

21 ↗

(On Diff #82471)

Slightly odd/non-descript name (DWARFDecompressor? it seems specifically for decompressing)

31–32 ↗

(On Diff #82471)

Could uncompress return Error instead of bool, and then the error message could be included there?

34–38 ↗

(On Diff #82471)

As an aside/follow-up: once we have this abstraction, it might be worth considering adding a streaming API (eg: pass in a lambda that gets called with uncompressed bytes repeatedly until done). That way linkers (& dwp) could stream uncompressed bytes out and write them out to the output file (especially since they can know the uncompressed size from the header without actually uncompressing the bytes - so they can allocate the necessary size in the output (even update at least some of the gdb-index tables without inspecting the contents of the comperssed bytes too) and then just wait to decompress, in parallel, while streaming/writing the output)

43–44 ↗

(On Diff #82471)

strange to have a mutable public member - should this be an accessor?

[grimar]

David, Davide, thanks for review !
I'll address comments for this in early 2017 (have a vacation + holidays here).

[aprantl]

include/llvm/DebugInfo/DWARF/DWARFCompression.h

23 ↗

(On Diff #82471)

Please comment the parameters here.

29 ↗

(On Diff #82471)

MutableArrayRef?

[ruiu]

I don't think DWARFCompression is a good name because you can compress not only debug sections but any types of sections. Likewise, it shouldn't be in lib/DebugInfo directory.

[davide]

In D28105#634636, @ruiu wrote:

I don't think DWARFCompression is a good name because you can compress not only debug sections but any types of sections. Likewise, it shouldn't be in lib/DebugInfo directory.

Probably. lib/Object or lib/Support would be a better fit? (I vote for the former).

[grimar]

• Addressed review comments.
• Moved to lib/Object

include/llvm/DebugInfo/DWARF/DWARFCompression.h

21 ↗

(On Diff #82471)

Renamed to Decompressor, since it is lib/Object/Decompressor now

23 ↗

(On Diff #82471)

Done.

29 ↗

(On Diff #82471)

Done.

31–32 ↗

(On Diff #82471)

Done.

37 ↗

(On Diff #82471)

Done.

40 ↗

(On Diff #82471)

Done.

43–44 ↗

(On Diff #82471)

Made an accessor.

lib/DebugInfo/DWARF/DWARFContext.cpp

608 ↗

(On Diff #82471)

Done.

614 ↗

(On Diff #82471)

Added comment for ".z".

Note that line comes from original code, I just moved it below and added "z" handling.
Not sure why originally "_" was also dropped,
I never saw sections with name like "._xxx".
So I commented only my new logic for dropping "z".

[grimar]

• Addressed Rafael's comments.

[grimar]

• Addressed Rafael's review comments.

llvm/trunk/include/llvm/Object/Decompressor.h

+//===-- Decompressor.h ------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===/
+
+#ifndef LLVM_OBJECT_DECOMPRESSOR_H
+#define LLVM_OBJECT_DECOMPRESSOR_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Object/ObjectFile.h"
+
+namespace llvm {
+namespace object {
+
+/// @brief Decompressor helps to handle decompression of compressed sections.
+class Decompressor {
+public:
+  /// @brief Create decompressor object.
+  /// @param Name        Section name.
+  /// @param Data        Section content.
+  /// @param IsLE        Flag determines if Data is in little endian form.
+  /// @param Name        Flag determines if object is 64 bit.
+  static Expected<Decompressor> create(StringRef Name, StringRef Data,
+                                       bool IsLE, bool Is64Bit);
+
+  /// @brief Resize the buffer and uncompress section data into it.
+  /// @param Out         Destination buffer.
+  Error decompress(SmallString<32> &Out);
+
+  /// @brief Uncompress section data to raw buffer provided.
+  /// @param Buffer      Destination buffer.
+  Error decompress(MutableArrayRef<char> Buffer);
+
+  /// @brief Return memory buffer size required for decompression.
+  uint64_t getDecompressedSize() { return DecompressedSize; }
+
+  /// @brief Return true if section is compressed, including gnu-styled case.
+  static bool isCompressed(const object::SectionRef &Section);
+
+  /// @brief Return true if section is a ELF compressed one.
+  static bool isCompressedELFSection(uint64_t Flags, StringRef Name);
+
+  /// @brief Return true if section name matches gnu style compressed one.
+  static bool isGnuStyle(StringRef Name);
+
+private:
+  Decompressor(StringRef Data);
+
+  Error consumeCompressedGnuHeader();
+  Error consumeCompressedZLibHeader(bool Is64Bit, bool IsLittleEndian);
+
+  StringRef SectionData;
+  uint64_t DecompressedSize;
+};
+
+} // end namespace object
+} // end namespace llvm
+
+#endif // LLVM_OBJECT_DECOMPRESSOR_H

llvm/trunk/lib/DebugInfo/DWARF/DWARFContext.cpp

 //===-- DWARFContext.cpp --------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/DebugInfo/DWARF/DWARFAcceleratorTable.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugArangeSet.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugPubTable.h"
 #include "llvm/DebugInfo/DWARF/DWARFUnitIndex.h"
+#include "llvm/Object/Decompressor.h"
 #include "llvm/Object/MachO.h"
 #include "llvm/Object/RelocVisitor.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/Dwarf.h"
 #include "llvm/Support/ELF.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
if (Spec.FLIKind != FileLineInfoKind::None) {
         FunctionDIE.getCallerFrame(CallFile, CallLine, CallColumn);
       }
     }
     InliningInfo.addFrame(Frame);
   }
   return InliningInfo;
 }
 
-static bool consumeCompressedGnuHeader(StringRef &data,
-                                       uint64_t &OriginalSize) {
-  // Consume "ZLIB" prefix.
-  if (!data.startswith("ZLIB"))
-    return false;
-  data = data.substr(4);
-  // Consume uncompressed section size (big-endian 8 bytes).
-  DataExtractor extractor(data, false, 8);
-  uint32_t Offset = 0;
-  OriginalSize = extractor.getU64(&Offset);
-  if (Offset == 0)
-    return false;
-  data = data.substr(Offset);
-  return true;
-}
-
-static bool consumeCompressedZLibHeader(StringRef &Data, uint64_t &OriginalSize,
-                                        bool IsLE, bool Is64Bit) {
-  using namespace ELF;
-  uint64_t HdrSize = Is64Bit ? sizeof(Elf64_Chdr) : sizeof(Elf32_Chdr);
-  if (Data.size() < HdrSize)
-    return false;
-
-  DataExtractor Extractor(Data, IsLE, 0);
-  uint32_t Offset = 0;
-  if (Extractor.getUnsigned(&Offset, Is64Bit ? sizeof(Elf64_Word)
-                                             : sizeof(Elf32_Word)) !=
-      ELFCOMPRESS_ZLIB)
-    return false;
-
-  // Skip Elf64_Chdr::ch_reserved field.
-  if (Is64Bit)
-    Offset += sizeof(Elf64_Word);
-
-  OriginalSize = Extractor.getUnsigned(&Offset, Is64Bit ? sizeof(Elf64_Xword)
-                                                        : sizeof(Elf32_Word));
-  Data = Data.substr(HdrSize);
-  return true;
-}
-
-static bool tryDecompress(StringRef &Name, StringRef &Data,
-                          SmallString<32> &Out, bool ZLibStyle, bool IsLE,
-                          bool Is64Bit) {
-  if (!zlib::isAvailable())
-    return false;
-
-  uint64_t OriginalSize;
-  bool Result =
-      ZLibStyle ? consumeCompressedZLibHeader(Data, OriginalSize, IsLE, Is64Bit)
-                : consumeCompressedGnuHeader(Data, OriginalSize);
-
-  if (!Result || zlib::uncompress(Data, Out, OriginalSize) != zlib::StatusOK)
-    return false;
-
-  // gnu-style names are started from "z", consume that.
-  if (!ZLibStyle)
-    Name = Name.substr(1);
-  return true;
-}
-
 DWARFContextInMemory::DWARFContextInMemory(const object::ObjectFile &Obj,
     const LoadedObjectInfo *L)
     : IsLittleEndian(Obj.isLittleEndian()),
       AddressSize(Obj.getBytesInAddress()) {
   for (const SectionRef &Section : Obj.sections()) {
     StringRef name;
     Section.getName(name);
     // Skip BSS and Virtual sections, they aren't interesting.
     bool IsBSS = Section.isBSS();
     if (IsBSS)
       continue;
     bool IsVirtual = Section.isVirtual();
     if (IsVirtual)
       continue;
     StringRef data;
 
     section_iterator RelocatedSection = Section.getRelocatedSection();
     // Try to obtain an already relocated version of this section.
     // Else use the unrelocated section from the object file. We'll have to
     // apply relocations ourselves later.
     if (!L || !L->getLoadedSectionContents(*RelocatedSection,data))
       Section.getContents(data);
 
-    name = name.substr(name.find_first_not_of("._")); // Skip . and _ prefixes.
-
-    bool ZLibStyleCompressed = Section.isCompressed();
-    if (ZLibStyleCompressed || name.startswith("zdebug_")) {
+    if (Decompressor::isCompressed(Section)) {
+      Expected<Decompressor> Decompressor =
+          Decompressor::create(name, data, IsLittleEndian, AddressSize == 8);
+      if (!Decompressor)
+        continue;
       SmallString<32> Out;
-      if (!tryDecompress(name, data, Out, ZLibStyleCompressed, IsLittleEndian,
-                         AddressSize == 8))
+      if (auto Err = Decompressor->decompress(Out))
         continue;
       UncompressedSections.emplace_back(std::move(Out));
       data = UncompressedSections.back();
     }
 
+    // Compressed sections names in GNU style starts from ".z",
+    // at this point section is decompressed and we drop compression prefix.
+    name = name.substr(
+        name.find_first_not_of("._z")); // Skip ".", "z" and "_" prefixes.
+
     StringRef *SectionData =
         StringSwitch<StringRef *>(name)
             .Case("debug_info", &InfoSection.Data)
             .Case("debug_abbrev", &AbbrevSection)
             .Case("debug_loc", &LocSection.Data)
             .Case("debug_line", &LineSection.Data)
             .Case("debug_aranges", &ARangeSection)
             .Case("debug_frame", &DebugFrameSection)

llvm/trunk/lib/Object/CMakeLists.txt

 add_llvm_library(LLVMObject
   Archive.cpp
   ArchiveWriter.cpp
   Binary.cpp
   COFFObjectFile.cpp
+  Decompressor.cpp
   ELF.cpp
   ELFObjectFile.cpp
   Error.cpp
   IRObjectFile.cpp
   MachOObjectFile.cpp
   MachOUniversal.cpp
   ModuleSummaryIndexObjectFile.cpp
   ModuleSymbolTable.cpp

llvm/trunk/lib/Object/Decompressor.cpp

+//===-- Decompressor.cpp --------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Object/Decompressor.h"
+#include "llvm/Object/ELFObjectFile.h"
+#include "llvm/Support/Compression.h"
+#include "llvm/Support/DataExtractor.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/ELF.h"
+
+using namespace llvm;
+using namespace llvm::support::endian;
+using namespace object;
+
+Expected<Decompressor> Decompressor::create(StringRef Name, StringRef Data,
+                                            bool IsLE, bool Is64Bit) {
+  if (!zlib::isAvailable())
+    return createError("zlib is not available");
+
+  Decompressor D(Data);
+  Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()
+                               : D.consumeCompressedZLibHeader(Is64Bit, IsLE);
+  if (Err)
+    return std::move(Err);
+  return D;
+}
+
+Decompressor::Decompressor(StringRef Data)
+    : SectionData(Data), DecompressedSize(0) {}
+
+Error Decompressor::consumeCompressedGnuHeader() {
+  if (!SectionData.startswith("ZLIB"))
+    return createError("corrupted compressed section header");
+
+  SectionData = SectionData.substr(4);
+
+  // Consume uncompressed section size (big-endian 8 bytes).
+  if (SectionData.size() < 8)
+    return createError("corrupted uncompressed section size");
+  DecompressedSize = read64be(SectionData.data());
+  SectionData = SectionData.substr(8);
+
+  return Error::success();
+}
+
+Error Decompressor::consumeCompressedZLibHeader(bool Is64Bit,
+                                                bool IsLittleEndian) {
+  using namespace ELF;
+  uint64_t HdrSize = Is64Bit ? sizeof(Elf64_Chdr) : sizeof(Elf32_Chdr);
+  if (SectionData.size() < HdrSize)
+    return createError("corrupted compressed section header");
+
+  DataExtractor Extractor(SectionData, IsLittleEndian, 0);
+  uint32_t Offset = 0;
+  if (Extractor.getUnsigned(&Offset, Is64Bit ? sizeof(Elf64_Word)
+                                             : sizeof(Elf32_Word)) !=
+      ELFCOMPRESS_ZLIB)
+    return createError("unsupported compression type");
+
+  // Skip Elf64_Chdr::ch_reserved field.
+  if (Is64Bit)
+    Offset += sizeof(Elf64_Word);
+
+  DecompressedSize = Extractor.getUnsigned(
+      &Offset, Is64Bit ? sizeof(Elf64_Xword) : sizeof(Elf32_Word));
+  SectionData = SectionData.substr(HdrSize);
+  return Error::success();
+}
+
+bool Decompressor::isGnuStyle(StringRef Name) {
+  return Name.startswith(".zdebug");
+}
+
+bool Decompressor::isCompressed(const object::SectionRef &Section) {
+  StringRef Name;
+  if (std::error_code E = Section.getName(Name))
+    return false;
+  return Section.isCompressed() || isGnuStyle(Name);
+}
+
+bool Decompressor::isCompressedELFSection(uint64_t Flags, StringRef Name) {
+  return (Flags & ELF::SHF_COMPRESSED) || isGnuStyle(Name);
+}
+
+Error Decompressor::decompress(SmallString<32> &Out) {
+  Out.resize(DecompressedSize);
+  return decompress({Out.data(), (size_t)DecompressedSize});
+}
+
+Error Decompressor::decompress(MutableArrayRef<char> Buffer) {
+  size_t Size = Buffer.size();
+  zlib::Status Status = zlib::uncompress(SectionData, Buffer.data(), Size);
+  if (Status != zlib::StatusOK)
+    return createError("decompression failed");
+  return Error::success();
+}