[llvm-mc] - Teach llvm-mc to generate zlib styled compression sections.

[grimar]

Summary

This patch is strongly based on previously reverted D20331.
(because of gnuutils < 2.26 does not support compressed debug sections in non zlib-gnu style)

Difference that this patch supports both zlib and zlib-gnu styles.

-compress-debug-sections option now supports next values:

1. -compress-debug-sections=zlib-gnu
2. -compress-debug-sections=zlib
3. -compress-debug-sections=none

Previously specifying -compress-debug-sections enabled zlib-gnu compression,
so anyone can put "-compress-debug-sections=zlib-gnu" to restore the behavior
that was before this patch for case when compression was enabled.

[grimar]

tools/llvm-mc/llvm-mc.cpp

67

At first I tried:

static cl::opt<DebugCompressionType>
CompressDebugSections("compress-debug-sections", cl::ValueOptional,
  cl::init(DebugCompressionType::DCT_None),
  cl::desc("Choose DWARF debug sections compression:"),
  cl::values(
    clEnumValN(DebugCompressionType::DCT_None, "none",
      "No compression"),
    clEnumValN(DebugCompressionType::DCT_Zlib, "zlib",
      "Use zlib compression"),
    clEnumValN(DebugCompressionType::DCT_Zlib, "zlib-gabi",
      "The same as zlib"),
    clEnumValN(DebugCompressionType::DCT_ZlibGnu, "zlib-gnu",
      "Use zlib-gnu compression (depricated)"),
    clEnumValEnd));

But this does not allow just to write "-compress-debug-sections", without
specifying the value for some reason. Approach that uses std::string works well.

[• rafael]

include/llvm/MC/MCAsmInfo.h

57

You shouldn't need the Unknown bit. See the comment about the option in llvm-mc.

lib/CodeGen/LLVMTargetMachine.cpp

74

I think in the first patch you will have to default to zlib-gnu. The reason is that right now there would be no way for someone using clang to set which one to use.

Once this patch is in with zlib-gnu as the default, we can teach TargetMachine and clang about the two types and change the default.

tools/llvm-mc/llvm-mc.cpp

419

llvm-mc doesn't need to be command line compatible with as, so please simplify this.
The option can be a cl::opt over an enum with just 3 values: none, zlib, zlib-gnu.

See FileType for a similar option.

[grimar]

lib/CodeGen/LLVMTargetMachine.cpp

74

Ok. I started with it, but switched to the zlib here at last second. Just because previous patch has compression on/off only and had zlib as default.
Will switch back.

tools/llvm-mc/llvm-mc.cpp

419

Yep, that what I did at first (see my comment above), but I thought command line compatibility is important, so had to do that.
I am happy to simplify this to enum parse way.

[grimar]

• Addressed review comments.

[• rafael]

lib/MC/ELFObjectWriter.cpp

987

This function does different things depending on the format

• for zlib it writes the header
• for gnu it prepends the header to CompressedContents

I would be cleaner to rename this to maybeWriteCompression header and have it write the header in both cases.

[grimar]

• Addressed review comments

[grimar]

Oops, let me update this again.

[grimar]

• Now it do what I wanted to write.

[grimar]

• Removed unrelated change.

[• rafael]

LGTM. Thanks!

include/llvm/MC/MCAsmInfo.h

enum class ExceptionHandling {
   ARM,      /// ARM EHABI
   WinEH,    /// Windows Exception Handling
 };
 
 namespace LCOMM {
 enum LCOMMType { NoAlignment, ByteAlignment, Log2Alignment };
 }
 
+enum class DebugCompressionType {
+  DCT_None,    // no compression
+  DCT_Zlib,    // zlib style complession
+  DCT_ZlibGnu  // zlib-gnu style compression
+};
+
 /// This class is intended to be used as a base class for asm
 /// properties and features specific to the target.
 class MCAsmInfo {
 protected:
   //===------------------------------------------------------------------===//
   // Properties to be set by the target writer, used to configure asm printer.
   //
 
protected:
 
   /// Should we use the integrated assembler?
   /// The integrated assembler should be enabled by default (by the
   /// constructors) when failing to parse a valid piece of assembly (inline
   /// or otherwise) is considered a bug. It may then be overridden after
   /// construction (see LLVMTargetMachine::initAsmInfo()).
   bool UseIntegratedAssembler;
 
-  /// Compress DWARF debug sections. Defaults to false.
-  bool CompressDebugSections;
+  /// Compress DWARF debug sections. Defaults to no compression.
+  DebugCompressionType CompressDebugSections;
 
   /// True if the integrated assembler should interpret 'a >> b' constant
   /// expressions as logical rather than arithmetic.
   bool UseLogicalShr;
 
   // If true, emit GOTPCRELX/REX_GOTPCRELX instead of GOTPCREL, on
   // X86_64 ELF.
   bool RelaxELFRelocations;
public:
   /// Return true if assembly (inline or otherwise) should be parsed.
   bool useIntegratedAssembler() const { return UseIntegratedAssembler; }
 
   /// Set whether assembly (inline or otherwise) should be parsed.
   virtual void setUseIntegratedAssembler(bool Value) {
     UseIntegratedAssembler = Value;
   }
 
-  bool compressDebugSections() const { return CompressDebugSections; }
+  DebugCompressionType compressDebugSections() const {
+    return CompressDebugSections;
+  }
 
-  void setCompressDebugSections(bool CompressDebugSections) {
+  void setCompressDebugSections(DebugCompressionType CompressDebugSections) {
     this->CompressDebugSections = CompressDebugSections;
   }
 
   bool shouldUseLogicalShr() const { return UseLogicalShr; }
 
   bool canRelaxRelocations() const { return RelaxELFRelocations; }
 };
 }
 
 #endif

include/llvm/MC/MCSectionELF.h

public:
   /// Decides whether a '.section' directive should be printed before the
   /// section name
   bool ShouldOmitSectionDirective(StringRef Name, const MCAsmInfo &MAI) const;
 
   StringRef getSectionName() const { return SectionName; }
   unsigned getType() const { return Type; }
   unsigned getFlags() const { return Flags; }
   unsigned getEntrySize() const { return EntrySize; }
+  void setFlags(unsigned F) { Flags = F; }
   const MCSymbolELF *getGroup() const { return Group; }
 
   void PrintSwitchToSection(const MCAsmInfo &MAI, raw_ostream &OS,
                             const MCExpr *Subsection) const override;
   bool UseCodeAlign() const override;
   bool isVirtualSection() const override;
 
   bool isUnique() const { return UniqueID != ~0U; }

lib/CodeGen/LLVMTargetMachine.cpp

void LLVMTargetMachine::initAsmInfo() {
   assert(TmpAsmInfo && "MCAsmInfo not initialized. "
          "Make sure you include the correct TargetSelect.h"
          "and that InitializeAllTargetMCs() is being invoked!");
 
   if (Options.DisableIntegratedAS)
     TmpAsmInfo->setUseIntegratedAssembler(false);
 
   if (Options.CompressDebugSections)
-    TmpAsmInfo->setCompressDebugSections(true);
+    TmpAsmInfo->setCompressDebugSections(DebugCompressionType::DCT_ZlibGnu);
 
   AsmInfo = TmpAsmInfo;
 }
 
 LLVMTargetMachine::LLVMTargetMachine(const Target &T,
                                      StringRef DataLayoutString,
                                      const Triple &TT, StringRef CPU,
                                      StringRef FS, TargetOptions Options,

lib/MC/ELFObjectWriter.cpp

class ELFObjectWriter : public MCObjectWriter {
   }
   unsigned getRelocType(MCContext &Ctx, const MCValue &Target,
                         const MCFixup &Fixup, bool IsPCRel) const {
     return TargetObjectWriter->getRelocType(Ctx, Target, Fixup, IsPCRel);
   }
 
   void align(unsigned Alignment);
 
+  bool maybeWriteCompression(uint64_t Size,
+                             SmallVectorImpl<char> &CompressedContents,
+                             bool ZLibStyle, unsigned Alignment);
+
 public:
   ELFObjectWriter(MCELFObjectTargetWriter *MOTW, raw_pwrite_stream &OS,
                   bool IsLittleEndian)
       : MCObjectWriter(OS, IsLittleEndian), TargetObjectWriter(MOTW) {}
 
   void reset() override {
     Renames.clear();
     Relocations.clear();
ELFObjectWriter::createRelocationSection(MCContext &Ctx,
 
   MCSectionELF *RelaSection = Ctx.createELFRelSection(
       RelaSectionName, hasRelocationAddend() ? ELF::SHT_RELA : ELF::SHT_REL,
       Flags, EntrySize, Sec.getGroup(), &Sec);
   RelaSection->setAlignment(is64Bit() ? 8 : 4);
   return RelaSection;
 }
 
-// Include the debug info compression header:
+// Include the debug info compression header.
+bool ELFObjectWriter::maybeWriteCompression(
+    uint64_t Size, SmallVectorImpl<char> &CompressedContents, bool ZLibStyle,
+    unsigned Alignment) {
+  if (ZLibStyle) {
+    uint64_t HdrSize =
+        is64Bit() ? sizeof(ELF::Elf32_Chdr) : sizeof(ELF::Elf64_Chdr);
+    if (Size <= HdrSize + CompressedContents.size())
+      return false;
+    // Platform specific header is followed by compressed data.
+    if (is64Bit()) {
+      // Write Elf64_Chdr header.
+      write(static_cast<ELF::Elf64_Word>(ELF::ELFCOMPRESS_ZLIB));
+      write(static_cast<ELF::Elf64_Word>(0)); // ch_reserved field.
+      write(static_cast<ELF::Elf64_Xword>(Size));
+      write(static_cast<ELF::Elf64_Xword>(Alignment));
+    } else {
+      // Write Elf32_Chdr header otherwise.
+      write(static_cast<ELF::Elf32_Word>(ELF::ELFCOMPRESS_ZLIB));
+      write(static_cast<ELF::Elf32_Word>(Size));
+      write(static_cast<ELF::Elf32_Word>(Alignment));
+    }
+    return true;
+  }
+
   // "ZLIB" followed by 8 bytes representing the uncompressed size of the section,
   // useful for consumers to preallocate a buffer to decompress into.
-static bool
-prependCompressionHeader(uint64_t Size,
-                         SmallVectorImpl<char> &CompressedContents) {
   const StringRef Magic = "ZLIB";
   if (Size <= Magic.size() + sizeof(Size) + CompressedContents.size())
     return false;
-  if (sys::IsLittleEndianHost)
-    sys::swapByteOrder(Size);
-  CompressedContents.insert(CompressedContents.begin(),
-                            Magic.size() + sizeof(Size), 0);
-  std::copy(Magic.begin(), Magic.end(), CompressedContents.begin());
-  std::copy(reinterpret_cast<char *>(&Size),
-            reinterpret_cast<char *>(&Size + 1),
-            CompressedContents.begin() + Magic.size());
+  write(ArrayRef<char>(Magic.begin(), Magic.size()));
+  writeBE64(Size);
   return true;
 }
 
 void ELFObjectWriter::writeSectionData(const MCAssembler &Asm, MCSection &Sec,
                                        const MCAsmLayout &Layout) {
   MCSectionELF &Section = static_cast<MCSectionELF &>(Sec);
   StringRef SectionName = Section.getSectionName();
 
   // Compressing debug_frame requires handling alignment fragments which is
   // more work (possibly generalizing MCAssembler.cpp:writeFragment to allow
   // for writing to arbitrary buffers) for little benefit.
-  if (!Asm.getContext().getAsmInfo()->compressDebugSections() ||
-      !SectionName.startswith(".debug_") || SectionName == ".debug_frame") {
+  bool CompressionEnabled =
+      Asm.getContext().getAsmInfo()->compressDebugSections() !=
+      DebugCompressionType::DCT_None;
+  if (!CompressionEnabled || !SectionName.startswith(".debug_") ||
+      SectionName == ".debug_frame") {
     Asm.writeSectionData(&Section, Layout);
     return;
   }
 
   SmallVector<char, 128> UncompressedData;
   raw_svector_ostream VecOS(UncompressedData);
   raw_pwrite_stream &OldStream = getStream();
   setStream(VecOS);
   Asm.writeSectionData(&Section, Layout);
   setStream(OldStream);
 
   SmallVector<char, 128> CompressedContents;
   zlib::Status Success = zlib::compress(
       StringRef(UncompressedData.data(), UncompressedData.size()),
       CompressedContents);
   if (Success != zlib::StatusOK) {
     getStream() << UncompressedData;
     return;
   }
 
-  if (!prependCompressionHeader(UncompressedData.size(), CompressedContents)) {
+  bool ZlibStyle = Asm.getContext().getAsmInfo()->compressDebugSections() ==
+                   DebugCompressionType::DCT_Zlib;
+  if (!maybeWriteCompression(UncompressedData.size(), CompressedContents,
+                             ZlibStyle, Sec.getAlignment())) {
     getStream() << UncompressedData;
     return;
   }
+
+  if (ZlibStyle)
+    // Set the compressed flag. That is zlib style.
+    Section.setFlags(Section.getFlags() | ELF::SHF_COMPRESSED);
+  else
+    // Add "z" prefix to section name. This is zlib-gnu style.
     Asm.getContext().renameELFSection(&Section,
                                       (".z" + SectionName.drop_front(1)).str());
   getStream() << CompressedContents;
 }
 
 void ELFObjectWriter::WriteSecHdrEntry(uint32_t Name, uint32_t Type,
                                        uint64_t Flags, uint64_t Address,
                                        uint64_t Offset, uint64_t Size,
                                        uint32_t Link, uint32_t Info,
                                        uint64_t Alignment,

lib/MC/MCAsmInfo.cpp

MCAsmInfo::MCAsmInfo() {
   //   - MCAsmInfoCOFF is handling this case, should it be MCAsmInfoMicrosoft?
   // - MachO targets always enables the integrated assembler by default
   //   - MCAsmInfoDarwin is handling this case
   // - Generic_GCC toolchains enable the integrated assembler on a per
   //   architecture basis.
   //   - The target subclasses for AArch64, ARM, and X86 handle these cases
   UseIntegratedAssembler = false;
 
-  CompressDebugSections = false;
+  CompressDebugSections = DebugCompressionType::DCT_None;
 }
 
 MCAsmInfo::~MCAsmInfo() {
 }
 
 bool MCAsmInfo::isSectionAtomizableBySymbols(const MCSection &Section) const {
   return false;
 }

test/MC/ELF/compression.s

-// RUN: llvm-mc -filetype=obj -compress-debug-sections -triple x86_64-pc-linux-gnu < %s -o %t
-// RUN: llvm-objdump -s %t | FileCheck %s
+// Check zlib-gnu style
+// RUN: llvm-mc -filetype=obj -compress-debug-sections=zlib-gnu -triple x86_64-pc-linux-gnu < %s -o %t
+// RUN: llvm-objdump -s %t | FileCheck --check-prefix=CHECK-GNU-STYLE %s
 // RUN: llvm-dwarfdump -debug-dump=str %t | FileCheck --check-prefix=STR %s
-// RUN: llvm-mc -filetype=obj -compress-debug-sections -triple i386-pc-linux-gnu < %s \
-// RUN:     | llvm-readobj -symbols - | FileCheck --check-prefix=386-SYMBOLS %s
+// RUN: llvm-mc -filetype=obj -compress-debug-sections=zlib-gnu -triple i386-pc-linux-gnu < %s \
+// RUN:     | llvm-readobj -symbols - | FileCheck --check-prefix=386-SYMBOLS-GNU %s
+
+// Check zlib style
+// RUN: llvm-mc -filetype=obj -compress-debug-sections=zlib -triple x86_64-pc-linux-gnu < %s -o %t
+// RUN: llvm-objdump -s %t | FileCheck --check-prefix=CHECK-ZLIB-STYLE %s
+// RUN: llvm-dwarfdump -debug-dump=str %t | FileCheck --check-prefix=STR %s
+// RUN: llvm-mc -filetype=obj -compress-debug-sections=zlib -triple i386-pc-linux-gnu < %s \
+// RUN:     | llvm-readobj -symbols - | FileCheck --check-prefix=386-SYMBOLS-ZLIB %s
+// RUN: llvm-readobj -sections %t | FileCheck --check-prefix=ZLIB-STYLE-FLAGS %s
 
 // REQUIRES: zlib
 
 // Don't compress small sections, such as this simple debug_abbrev example
-// CHECK: Contents of section .debug_abbrev:
-// CHECK-NOT: ZLIB
-// CHECK-NOT: Contents of
+// CHECK-GNU-STYLE: Contents of section .debug_abbrev:
+// CHECK-GNU-STYLE-NOT: ZLIB
+// CHECK-GNU-STYLE-NOT: Contents of
 
-// CHECK: Contents of section .debug_info:
+// CHECK-GNU-STYLE: Contents of section .debug_info:
 
-// CHECK: Contents of section .zdebug_str:
+// CHECK-GNU-STYLE: Contents of section .zdebug_str:
 // Check for the 'ZLIB' file magic at the start of the section only
-// CHECK-NEXT: ZLIB
-// CHECK-NOT: ZLIB
-
+// CHECK-GNU-STYLE-NEXT: ZLIB
+// CHECK-GNU-STYLE-NOT: ZLIB
 // FIXME: Handle compressing alignment fragments to support compressing debug_frame
-// CHECK: Contents of section .debug_frame:
-// CHECK-NOT: ZLIB
-// CHECK: Contents of
+// CHECK-GNU-STYLE: Contents of section .debug_frame:
+// CHECK-GNU-STYLE-NOT: ZLIB
+// CHECK-GNU-STYLE: Contents of
 
 // Decompress one valid dwarf section just to check that this roundtrips,
 // we use .zdebug_str section for that
 // STR: perfectly compressable data sample *****************************************
 
 // In x86 32 bit named symbols are used for temporary symbols in merge
 // sections, so make sure we handle symbols inside compressed sections
-// 386-SYMBOLS: Name: .Linfo_string0
-// 386-SYMBOLS-NOT: }
-// 386-SYMBOLS: Section: .zdebug_str
+// 386-SYMBOLS-GNU: Name: .Linfo_string0
+// 386-SYMBOLS-GNU-NOT: }
+// 386-SYMBOLS-GNU: Section: .zdebug_str
+
+// Now check the zlib style output:
+
+// Don't compress small sections, such as this simple debug_abbrev example
+// CHECK-ZLIB-STYLE: Contents of section .debug_abbrev:
+// CHECK-ZLIB-STYLE-NOT: ZLIB
+// CHECK-ZLIB-STYLE-NOT: Contents of
+// CHECK-ZLIB-STYLE: Contents of section .debug_info:
+// FIXME: Handle compressing alignment fragments to support compressing debug_frame
+// CHECK-ZLIB-STYLE: Contents of section .debug_frame:
+// CHECK-ZLIB-STYLE-NOT: ZLIB
+// CHECK-ZLIB-STYLE: Contents of
+
+// Check that debug_line section was not renamed, so it is
+// zlib-style, not zlib-gnu one. Check that SHF_COMPRESSED was set.
+// ZLIB-STYLE-FLAGS:      Section {
+// ZLIB-STYLE-FLAGS:        Index:
+// ZLIB-STYLE-FLAGS:        Name: .debug_str
+// ZLIB-STYLE-FLAGS-NEXT:   Type: SHT_PROGBITS
+// ZLIB-STYLE-FLAGS-NEXT:   Flags [
+// ZLIB-STYLE-FLAGS-NEXT:     SHF_COMPRESSED
+
+// 386-SYMBOLS-ZLIB: Name: .Linfo_string0
+// 386-SYMBOLS-ZLIB-NOT: }
+// 386-SYMBOLS-ZLIB: Section: .debug_str
 
   .section  .debug_line,"",@progbits
 
   .section  .debug_abbrev,"",@progbits
 .Lsection_abbrev:
   .byte 1                       # Abbreviation Code
   .byte 17                      # DW_TAG_compile_unit
   .byte 0                       # DW_CHILDREN_no

tools/llvm-mc/llvm-mc.cpp

 
 static cl::opt<std::string>
 OutputFilename("o", cl::desc("Output filename"),
                cl::value_desc("filename"));
 
 static cl::opt<bool>
 ShowEncoding("show-encoding", cl::desc("Show instruction encodings"));
 
-static cl::opt<bool>
-CompressDebugSections("compress-debug-sections",
-                      cl::desc("Compress DWARF debug sections"));
+static cl::opt<DebugCompressionType>
+CompressDebugSections("compress-debug-sections", cl::ValueOptional,
+  cl::init(DebugCompressionType::DCT_None),
+  cl::desc("Choose DWARF debug sections compression:"),
+  cl::values(
+    clEnumValN(DebugCompressionType::DCT_None, "none",
+      "No compression"),
+    clEnumValN(DebugCompressionType::DCT_Zlib, "zlib",
+      "Use zlib compression"),
+    clEnumValN(DebugCompressionType::DCT_ZlibGnu, "zlib-gnu",
+      "Use zlib-gnu compression (depricated)"),
+    clEnumValEnd));
 
 static cl::opt<bool>
 ShowInst("show-inst", cl::desc("Show internal instruction representation"));
 
 static cl::opt<bool>
 ShowInstOperands("show-inst-operands",
                  cl::desc("Show instructions operands as parsed"));
 
 static cl::opt<unsigned>
int main(int argc, char **argv) {
   SrcMgr.setIncludeDirs(IncludeDirs);
 
   std::unique_ptr<MCRegisterInfo> MRI(TheTarget->createMCRegInfo(TripleName));
   assert(MRI && "Unable to create target register info!");
 
   std::unique_ptr<MCAsmInfo> MAI(TheTarget->createMCAsmInfo(*MRI, TripleName));
   assert(MAI && "Unable to create target asm info!");
 
-  if (CompressDebugSections) {
+  if (CompressDebugSections != DebugCompressionType::DCT_None) {
     if (!zlib::isAvailable()) {
       errs() << ProgName
              << ": build tools with zlib to enable -compress-debug-sections";
       return 1;
     }
-    MAI->setCompressDebugSections(true);
+    MAI->setCompressDebugSections(CompressDebugSections);
   }
 
   // FIXME: This is not pretty. MCContext has a ptr to MCObjectFileInfo and
   // MCObjectFileInfo needs a MCContext reference in order to initialize itself.
   MCObjectFileInfo MOFI;
   MCContext Ctx(MAI.get(), MRI.get(), &MOFI, &SrcMgr);
   MOFI.InitMCObjectFileInfo(TheTriple, PIC, CMModel, Ctx);