[ELF] Support GNU compressed sections

[phosek]

Summary

These are still being used by some tools, most notable Go, which
triggers an error in llgo tests anytime lld is being used as the
default linker for Clang.

This has been raised in https://github.com/golang/go/issues/27110

[ruiu]

I personally think that they should update their toolchain so that they can handle the standardized compressed debug section instead of a GNU-extension, or they should disable tests that depends on the GNU extension by examining if -compressed-debug-section=gnu is supported before running the tests. I particularly dislike the fact that the GNU extension uses .z as a prefix for a compressed section, which honestly seems ugly.

[pcc]

The reason why Go tries to use this flag is because of a bug in Go which has now been fixed. I'd prefer that rather than implementing this feature just for the sake of a bug that has been fixed, affected developers should upgrade Go. To prevent the test from failing, we could arrange to disable the Go bindings test if lld is being used and the version of Go is too old.

[phosek]

In D58205#1397063, @pcc wrote:

The reason why Go tries to use this flag is because of a bug in Go which has now been fixed. I'd prefer that rather than implementing this feature just for the sake of a bug that has been fixed, affected developers should upgrade Go. To prevent the test from failing, we could arrange to disable the Go bindings test if lld is being used and the version of Go is too old.

Do you know when was it fixed? I'm using go1.11.4 and I'm still seeing the failure in Bindings/Go/go.test when running check-llvm.

[pcc]

In D58205#1397161, @phosek wrote:

In D58205#1397063, @pcc wrote:

The reason why Go tries to use this flag is because of a bug in Go which has now been fixed. I'd prefer that rather than implementing this feature just for the sake of a bug that has been fixed, affected developers should upgrade Go. To prevent the test from failing, we could arrange to disable the Go bindings test if lld is being used and the version of Go is too old.

Do you know when was it fixed? I'm using go1.11.4 and I'm still seeing the failure in Bindings/Go/go.test when running check-llvm.

I think you might need go1.12rc1.

lld/ELF/Config.h

 enum class SortSectionPolicy { Default, None, Alignment, Name, Priority };
 
 // For --target2
 enum class Target2Policy { Abs, Rel, GotRel };
 
 // For tracking ARM Float Argument PCS
 enum class ARMVFPArgKind { Default, Base, VFP, ToolChain };
 
+// For --compress-debug-sections.
+enum class CompressDebugSectionsPolicy { None, Zlib, ZlibGnu };
+
 struct SymbolVersion {
   llvm::StringRef Name;
   bool IsExternCpp;
   bool HasWildcard;
 };
 
 // This struct contains symbols version definition that
 // can be found in version script if it is used for link.
struct Configuration {
   bool ARMHasBlx = false;
   bool ARMHasMovtMovw = false;
   bool ARMJ1J2BranchEncoding = false;
   bool AsNeeded = false;
   bool Bsymbolic;
   bool BsymbolicFunctions;
   bool CallGraphProfileSort;
   bool CheckSections;
-  bool CompressDebugSections;
+  CompressDebugSectionsPolicy CompressDebugSections;
   bool Cref;
   bool DefineCommon;
   bool Demangle = true;
   bool DisableVerify;
   bool EhFrameHdr;
   bool EmitLLVM;
   bool EmitRelocs;
   bool EnableNewDtags;

lld/ELF/Driver.cpp

for (const Elf_CGProfile_Impl<ELFT> &CGPE : Obj->CGProfile) {
       auto *From = dyn_cast_or_null<InputSectionBase>(FromSym->Section);
       auto *To = dyn_cast_or_null<InputSectionBase>(ToSym->Section);
       if (From && To)
         Config->CallGraphProfile[{From, To}] += CGPE.cgp_weight;
     }
   }
 }
 
-static bool getCompressDebugSections(opt::InputArgList &Args) {
+static CompressDebugSectionsPolicy getCompressDebugSections(opt::InputArgList &Args) {
   StringRef S = Args.getLastArgValue(OPT_compress_debug_sections, "none");
-  if (S == "none")
-    return false;
-  if (S != "zlib")
-    error("unknown --compress-debug-sections value: " + S);
+  if (S == "zlib" || S == "zlib-gnu") {
     if (!zlib::isAvailable())
       error("--compress-debug-sections: zlib is not available");
-  return true;
+    if (S == "zlib")
+      return CompressDebugSectionsPolicy::Zlib;
+    if (S == "zlib-gnu")
+      return CompressDebugSectionsPolicy::ZlibGnu;
+  } else if (S != "none") {
+    error("unknown --compress-debug-sections value: " + S);
+  }
+  return CompressDebugSectionsPolicy::None;
 }
 
 static std::pair<StringRef, StringRef> getOldNewOptions(opt::InputArgList &Args,
                                                         unsigned Id) {
   auto *Arg = Args.getLastArg(Id);
   if (!Arg)
     return {"", ""};
 

lld/ELF/OutputSections.h

public:
   bool Noload = false;
   bool ExpressionsUseSymbols = false;
   bool InOverlay = false;
 
   template <class ELFT> void finalize();
   template <class ELFT> void writeTo(uint8_t *Buf);
   template <class ELFT> void maybeCompress();
 
+  StringRef finalName();
+
   void sort(llvm::function_ref<int(InputSectionBase *S)> Order);
   void sortInitFini();
   void sortCtorsDtors();
 
 private:
   // Used for implementation of --compress-debug-sections option.
   std::vector<uint8_t> ZDebugHeader;
   llvm::SmallVector<char, 1> CompressedData;

lld/ELF/OutputSections.cpp

static void fill(uint8_t *Buf, size_t Size,
   memcpy(Buf + I, Filler.data(), Size - I);
 }
 
 // Compress section contents if this section contains debug info.
 template <class ELFT> void OutputSection::maybeCompress() {
   typedef typename ELFT::Chdr Elf_Chdr;
 
   // Compress only DWARF debug sections.
-  if (!Config->CompressDebugSections || (Flags & SHF_ALLOC) ||
-      !Name.startswith(".debug_"))
+  if (Config->CompressDebugSections == CompressDebugSectionsPolicy::None ||
+      (Flags & SHF_ALLOC) || !Name.startswith(".debug_"))
     return;
 
   // Create a section header.
   ZDebugHeader.resize(sizeof(Elf_Chdr));
   auto *Hdr = reinterpret_cast<Elf_Chdr *>(ZDebugHeader.data());
   Hdr->ch_type = ELFCOMPRESS_ZLIB;
   Hdr->ch_size = Size;
   Hdr->ch_addralign = Alignment;
 
   // Write section contents to a temporary buffer and compress it.
   std::vector<uint8_t> Buf(Size);
   writeTo<ELFT>(Buf.data());
   if (Error E = zlib::compress(toStringRef(Buf), CompressedData))
     fatal("compress failed: " + llvm::toString(std::move(E)));
 
   // Update section headers.
   Size = sizeof(Elf_Chdr) + CompressedData.size();
   Flags |= SHF_COMPRESSED;
 }
 
+StringRef OutputSection::finalName() {
+  if (Config->CompressDebugSections == CompressDebugSectionsPolicy::ZlibGnu &&
+      !(Flags & SHF_ALLOC) && Name.startswith(".debug_"))
+    return Saver.save(".z" + Name.substr(1));
+  return Name;
+}
+
 static void writeInt(uint8_t *Buf, uint64_t Data, uint64_t Size) {
   if (Size == 1)
     *Buf = Data;
   else if (Size == 2)
     write16(Buf, Data);
   else if (Size == 4)
     write32(Buf, Data);
   else if (Size == 8)

lld/ELF/Writer.cpp

template <class ELFT> void Writer<ELFT>::finalizeSections() {
   // This is a bit of a hack. A value of 0 means undef, so we set it
   // to 1 to make __ehdr_start defined. The section number is not
   // particularly relevant.
   Out::ElfHeader->SectionIndex = 1;
 
   for (size_t I = 0, E = OutputSections.size(); I != E; ++I) {
     OutputSection *Sec = OutputSections[I];
     Sec->SectionIndex = I + 1;
-    Sec->ShName = In.ShStrTab->addString(Sec->Name);
+    Sec->ShName = In.ShStrTab->addString(Sec->finalName());
   }
 
   // Binary and relocatable output does not have PHDRS.
   // The headers have to be created before finalize as that can influence the
   // image base and the dynamic section on mips includes the image base.
   if (!Config->Relocatable && !Config->OFormatBinary) {
     Phdrs = Script->hasPhdrsCommands() ? Script->createPhdrs() : createPhdrs();
     addPtArmExid(Phdrs);

lld/test/ELF/compress-debug-sections.s

 # RUN: ld.lld %t.o -o %t2 --compress-debug-sections zlib
 # RUN: llvm-objdump -s %t2 | FileCheck %s --check-prefix=ZLIBCONTENT
 # RUN: llvm-readobj -s %t2 | FileCheck %s --check-prefix=ZLIBFLAGS
 
 # RUN: not ld.lld %t.o -o %t1 --compress-debug-sections=zlib-gabi 2>&1 | \
 # RUN:   FileCheck -check-prefix=ERR %s
 # ERR: unknown --compress-debug-sections value: zlib-gabi
 
+## Test GNU version.
+# RUN: ld.lld %t.o -o %t3 --compress-debug-sections=zlib-gnu
+
+# RUN: llvm-objdump -s %t3 | FileCheck %s --check-prefix=ZLIBGNUCONTENT
+# ZLIBGNUCONTENT:     Contents of section .zdebug_str:
+# ZLIBGNUCONTENT-NOT: AAAAAAAAA
+
+# RUN: llvm-readobj -s %t3 | FileCheck %s --check-prefix=ZLIBGNUFLAGS
+# ZLIBGNUFLAGS:       Section {
+# ZLIBGNUFLAGS:         Index:
+# ZLIBGNUFLAGS:         Name: .zdebug_str
+# ZLIBGNUFLAGS-NEXT:    Type: SHT_PROGBITS
+# ZLIBGNUFLAGS-NEXT:    Flags [
+# ZLIBGNUFLAGS-NEXT:      SHF_COMPRESSED
+
 .section .debug_str,"MS",@progbits,1
 .Linfo_string0:
   .asciz "AAAAAAAAAAAAAAAAAAAAAAAAAAA"
 .Linfo_string1:
   .asciz "BBBBBBBBBBBBBBBBBBBBBBBBBBB"