Add -gnu-map option to emit a map file in the GNU-tsyle format.

[ruiu]

Summary

lld supports -Map option to write the layout information to a specified
file in a human-readable format. GNU ld and gold support that option as
well, but their format is different from others.

If the output were consumed only by humans, that's not a problem. However,
in practice, there are a lot of post-link tools that parses map files, and
they almost always support the GNU-style map file because of a historical
reason. It is not too hard to update a program if a post-link tool is
written by a scripting language, but if it is a proprietary tool, it's not
easy to update. In reality, it is sometimes very hard to update a program
so that the tool can read the lld-style map file.

So, in this patch, I added -gnu-map option to print out a map file in
the GNU-style format. I'm not super happy to do this, as this is basically
a duplicate feature, but given that the amount of code to implement the
feature is very small, I think doing this makes sense. It should help
users who has tools that consume map files.

[MaskRay]

That being said, -Map output is oftentimes processed by post-link tools

Just curious, do you have some concrete examples? Since ld.bfd and gold have different -map formats. These tools may have to learn the format of gold -Map when they migrate from ld.bfd to gold, no?

[MaskRay]

lld/ELF/Options.td

192

Without looking at the documentation, I think people would mostly likely think -gnu-map means a GNU ld compatible format.

ld.bfd is GNU ld :) (I have to say -gold-map sounds like a weird option name)

[ruiu]

Yes, we have an internal user who has a proprietary tool to process a -Map output. As to the difference between bfd and gold, they don't look too big, so perhaps most tools can consume both. In this output, I modeled gold.

[peter.smith]

In D63190#1539544, @ruiu wrote:

Yes, we have an internal user who has a proprietary tool to process a -Map output. As to the difference between bfd and gold, they don't look too big, so perhaps most tools can consume both. In this output, I modeled gold.

I think it isn't likely that tools will accept both gold and bfd map file formats; from a conversation with an employee of a Linux Distro, gold was used about 3% of the time although I expect its use to be significantly higher in Google. I've seen map file post processing most used in embedded systems as this is where placement of sections and size information is most crucial. I think most of those users are with ld.bfd at the moment though.

At the Euro LLVM binutils (LLD as honorary member) it was suggested that a machine readable format option for outputs such as the map file would be ideal. There was some belief that binutils could be persuaded to adopt a similar format. The TI proprietary linker does this http://downloads.ti.com/docs/esd/SLAU131K/Content/SLAU131K_HTML/xml_link_information_file_description.html#STDZ0820750 although they use XML which is probably overkill. Something like JSON would be quite simple for tools to parse.

Would it be worth asking if the customer would be willing to rewrite their parser to consume something like JSON and go down that route? Having said all that I've no objections to adding support for other linker map file formats, and adding gold doesn't prevent adding others later.

lld/ELF/Options.td

192

I agree with MaskRay here. Perhaps another command line option --map-file-format=<format> where <format> defaults to lld. We could then add gold, bfd etc.

[peter.smith]

Not had a chance to go through the gold source yet to see if there is anything obviously missing. From a quick experiment using a linker-script derived from ld.bfd it seems like gold's map file remains much the same, whereas ld.bfd will give information about symbol resolution etc. Will take a look at the gold source this afternoon.

lld/test/ELF/gnu-map.s

65

The path names will be different, will need to use some kind of regex here to skip the first part.

[peter.smith]

I've been through to see if I could spot any significant differences. I've recorded what I found although I'm not sure how significant they are to the person attempting to parse the map file though.

lld/ELF/MapFile.cpp

257

gold right justifies the size, but I doubt that is significant.

259

gold will print out the LMA if it is different at this point. Followed by (before compression) if the output section has been subject to debug compression.

268

Gold has some custom name printing for its linker generated content (do_print_to_mapfile) . For example the .plt comes out as

**PLT

.rela.dyn comes out as

** dynamic relocs

273

gold size is right justified (not significant).

274

Gold does not print the path and object for linker generated content like the .plt, LLD prints out <internal>:(.plt)

277

gold only prints "ordinary symbols", for example it does not print sharedFoo, sharedBar, sharedFunc1, sharedFunc2

[MaskRay]

lld/ELF/MapFile.cpp

245

gold reports ** file header and ** segment headers.

257

gold essentially uses 0x%08llx for 32-bit targets.

(the actual code is fprintf(this->map_file_, "0x%0*llx %10s", parameters->target().get_size() / 4, static_cast<unsigned long long>(os->address()), sizebuf);)

259

LMA is printed if either AT(LMA) or AT>LMA_REGION is specified.

This can be tested with if (OSec->LMAExpr || OSec->LMARegion) but we don't seem to store the LMA address as gold does. So I don't know how we can dump this piece of information.

268

** gdb_index
** GOT PLT
** PLT
** .LA25.stubs
** section headers
** .reginfo
** string table
** relocs for static relocation sections.

[MaskRay]

lld/test/ELF/gnu-map.s

65

This can be tested with:

FileCheck -DOBJ=%t.o %s

// CHECK-NEXT: .... [[OBJ]]:(.text)

[phosek]

In D63190#1539564, @peter.smith wrote:

In D63190#1539544, @ruiu wrote:

Yes, we have an internal user who has a proprietary tool to process a -Map output. As to the difference between bfd and gold, they don't look too big, so perhaps most tools can consume both. In this output, I modeled gold.

I think it isn't likely that tools will accept both gold and bfd map file formats; from a conversation with an employee of a Linux Distro, gold was used about 3% of the time although I expect its use to be significantly higher in Google. I've seen map file post processing most used in embedded systems as this is where placement of sections and size information is most crucial. I think most of those users are with ld.bfd at the moment though.

At the Euro LLVM binutils (LLD as honorary member) it was suggested that a machine readable format option for outputs such as the map file would be ideal. There was some belief that binutils could be persuaded to adopt a similar format. The TI proprietary linker does this http://downloads.ti.com/docs/esd/SLAU131K/Content/SLAU131K_HTML/xml_link_information_file_description.html#STDZ0820750 although they use XML which is probably overkill. Something like JSON would be quite simple for tools to parse.

Would it be worth asking if the customer would be willing to rewrite their parser to consume something like JSON and go down that route? Having said all that I've no objections to adding support for other linker map file formats, and adding gold doesn't prevent adding others later.

I just had a discussion with one of our internal teams that's looking at lld and this exact request came out (and it wasn't the first time). There are several examples of people parsing .map (or asking for help with) files:

• Windows GUI tool for analyzing .map files
• Another Windows tool
• fpv-gcc
• Hackaday tutorial
• Forum post asking for this
• ... and many more

There are also many internal examples from Google that I cannot show publicly.

Given how often we're being asked for this, especially from teams working in the embedded space that heavily relies on -Map output, JSON output is something I'd like to look into soon.

When that's supported, we could provide a script that would allow converting the JSON output to GNU-tstyle format. That would obviate the need for supporting every possible output format directly in lld.

[mierle]

I am one of the engineers at Google wanting to have machine readable .map file output from lld and ld. Regarding Peter's suggestion to eventually only offer a .json format and helper scripts, then removing the other .map output formats, I'm less excited. In practice maintaining scripts (and their associated runtimes and dependencies) can be a big burden. The map export code isn't enormous and so I would suggest we just keep it.

lld/ELF/Options.td

192

As someone who wants to get a JSON-based map format added to LLD, I like Peter's suggestion. With JSON added, the flags could be:

• --map-file-format=lld
• --map-file-format=gold
• --map-file-format=json
• --map-file-format=csv

which feels clean, readable, and extendable.

lld/ELF/Config.h

struct Configuration {
   uint32_t AndFeatures = 0;
   llvm::CachePruningPolicy ThinLTOCachePolicy;
   llvm::StringMap<uint64_t> SectionStartMap;
   llvm::StringRef Chroot;
   llvm::StringRef DynamicLinker;
   llvm::StringRef DwoDir;
   llvm::StringRef Entry;
   llvm::StringRef Emulation;
+  llvm::StringRef GnuMap;
   llvm::StringRef Fini;
   llvm::StringRef Init;
   llvm::StringRef LTOAAPipeline;
   llvm::StringRef LTOCSProfileFile;
   llvm::StringRef LTONewPmPasses;
   llvm::StringRef LTOObjPath;
   llvm::StringRef LTOSampleProfile;
   llvm::StringRef MapFile;

lld/ELF/Driver.cpp

static void readConfigs(opt::InputArgList &Args) {
   Config->ExportDynamic =
       Args.hasFlag(OPT_export_dynamic, OPT_no_export_dynamic, false);
   Config->FilterList = args::getStrings(Args, OPT_filter);
   Config->Fini = Args.getLastArgValue(OPT_fini, "_fini");
   Config->FixCortexA53Errata843419 = Args.hasArg(OPT_fix_cortex_a53_843419);
   Config->ForceBTI = Args.hasArg(OPT_force_bti);
   Config->RequireCET = Args.hasArg(OPT_require_cet);
   Config->GcSections = Args.hasFlag(OPT_gc_sections, OPT_no_gc_sections, false);
-  Config->GnuUnique = Args.hasFlag(OPT_gnu_unique, OPT_no_gnu_unique, true);
   Config->GdbIndex = Args.hasFlag(OPT_gdb_index, OPT_no_gdb_index, false);
+  Config->GnuMap = Args.getLastArgValue(OPT_gnu_map);
+  Config->GnuUnique = Args.hasFlag(OPT_gnu_unique, OPT_no_gnu_unique, true);
   Config->ICF = getICF(Args);
   Config->IgnoreDataAddressEquality =
       Args.hasArg(OPT_ignore_data_address_equality);
   Config->IgnoreFunctionAddressEquality =
       Args.hasArg(OPT_ignore_function_address_equality);
   Config->Init = Args.getLastArgValue(OPT_init, "_init");
   Config->LTOAAPipeline = Args.getLastArgValue(OPT_lto_aa_pipeline);
   Config->LTOCSProfileGenerate = Args.hasArg(OPT_lto_cs_profile_generate);

lld/ELF/MapFile.h

 //===- MapFile.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_MAPFILE_H
 #define LLD_ELF_MAPFILE_H
 
 namespace lld {
 namespace elf {
 void writeMapFile();
+void writeGnuStyleMapFile();
 void writeCrossReferenceTable();
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/MapFile.cpp

for (BaseCommand *Base : OSec->SectionCommands) {
                     Cmd->Size, 1);
         OS << Indent8 << Cmd->CommandString << '\n';
         continue;
       }
     }
   }
 }
 
+// Print out a map file in the same format as GNU gold. This function
+// is called when you pass `-gnu-map`.
+//
+// Technically, `-gnu-map` is a duplicate feature as `-Map`. lld's map
+// file format is (in our opinion) better than GNU's, as our map file
+// is arguably easier to read and also contains LMAs that the GNU
+// format lacks.
+//
+// That being said, `-Map` output is oftentimes processed by post-link
+// tools, and sometimes updating the tools to be able to understand
+// the lld-style map file is hard. Therefore, we support both formats.
+void elf::writeGnuStyleMapFile() {
+  if (Config->GnuMap.empty())
+    return;
+
+  // Open a map file for writing.
+  std::error_code EC;
+  raw_fd_ostream OS(Config->GnuMap, EC, sys::fs::F_None);
+  if (EC) {
+    error("cannot open " + Config->GnuMap + ": " + EC.message());
+    return;
+  }
+
+  // Collect symbol info that we want to print out.
+  std::vector<Defined *> Syms = getSymbols();
+  SymbolMapTy SectionSyms = getSectionSyms(Syms);
+
+  // Print out the header line.
+  OS << "Memory map\n";
+
+  for (BaseCommand *Base : Script->SectionCommands) {
+    OutputSection *OSec = dyn_cast<OutputSection>(Base);
+    if (!OSec)
+      continue;
+
+    OS << "\n";
+    if (OSec->Name.size() <= 15)
+      OS << left_justify(OSec->Name, 15);
+    else
+      OS << OSec->Name << "\n               ";
+
+    OS << format(" 0x%016llx", OSec->Addr) << " "
+       << left_justify(("0x" + Twine::utohexstr(OSec->Size)).str(), 11) << "\n";
+
+    // Dump symbols for each input section.
+    for (BaseCommand *Base : OSec->SectionCommands) {
+      auto *Desc = dyn_cast<InputSectionDescription>(Base);
+      if (!Desc)
+        continue;
+
+      for (InputSection *IS : Desc->Sections) {
+        if (IS->Name.size() <= 14)
+          OS << " " << left_justify(IS->Name, 14);
+        else
+          OS << " " << IS->Name << "\n               ";
+
+        OS << format(" 0x%016llx", IS->getVA(0)) << " "
+           << left_justify(("0x" + Twine::utohexstr(IS->getSize())).str(), 11)
+           << " " << toString(IS) << "\n";
+
+        for (Symbol *Sym : SectionSyms[IS]) {
+          OS << "                " << format("0x%016llx", Sym->getVA())
+             << "                " << toString(*Sym) << "\n";
+        }
+      }
+    }
+  }
+}
+
 static void print(StringRef A, StringRef B) {
   outs() << left_justify(A, 49) << " " << B << "\n";
 }
 
 // Output a cross reference table to stdout. This is for --cref.
 //
 // For each global symbol, we print out a file that defines the symbol
 // followed by files that uses that symbol. Here is an example.

lld/ELF/Options.td

 defm gc_sections: B<"gc-sections",
     "Enable garbage collection of unused sections",
     "Disable garbage collection of unused sections (default)">;
 
 defm gdb_index: B<"gdb-index",
     "Generate .gdb_index section",
     "Do not generate .gdb_index section (default)">;
 
+defm gnu_map: Eq<"gnu-map",
+    "Print a link map to the specified file in the same format as GNU gold">;
+
 defm gnu_unique: B<"gnu-unique",
   "Enable STB_GNU_UNIQUE symbol binding (default)",
   "Disable STB_GNU_UNIQUE symbol binding">;
 
 defm hash_style: Eq<"hash-style", "Specify hash style (sysv, gnu or both)">;
 
 def help: F<"help">, HelpText<"Print option help">;
 

lld/ELF/Writer.cpp

template <class ELFT> void Writer<ELFT>::run() {
   // Backfill .note.gnu.build-id section content. This is done at last
   // because the content is usually a hash value of the entire output file.
   writeBuildId();
   if (errorCount())
     return;
 
   // Handle -Map and -cref options.
   writeMapFile();
+  writeGnuStyleMapFile();
   writeCrossReferenceTable();
   if (errorCount())
     return;
 
   if (auto E = Buffer->commit())
     error("failed to write to the output file: " + toString(std::move(E)));
 }
 

lld/test/ELF/gnu-map.s

+// REQUIRES: x86
+
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t1.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %p/Inputs/map-file2.s -o %t2.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %p/Inputs/map-file3.s -o %t3.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %p/Inputs/map-file4.s -o %t4.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %p/Inputs/map-file5.s -o %t5.o
+// RUN: ld.lld -shared %t5.o -o %t5.so -soname dso
+// RUN: rm -f %t4.a
+// RUN: llvm-ar rc %t4.a %t4.o
+// RUN: ld.lld %t1.o %t2.o %t3.o %t4.a %t5.so -o %t -gnu-map=- | \
+// RUN:   FileCheck --match-full-lines %s
+
+.global _start
+_start:
+.cfi_startproc
+.cfi_endproc
+ .quad sharedFoo
+ .quad sharedBar
+ .byte 0xe8
+ .long sharedFunc1 - .
+ .byte 0xe8
+ .long sharedFunc2 - .
+ .byte 0xe8
+ .long baz - .
+.global _Z1fi
+_Z1fi:
+.cfi_startproc
+nop
+.cfi_endproc
+.weak bar
+bar:
+.long bar - .
+.long zed - .
+local:
+.comm   common,4,16
+.global abs
+abs = 0xAB5
+labs = 0x1AB5
+
+//       CHECK: Memory map
+// CHECK-EMPTY:
+//  CHECK-NEXT: .dynsym         0x0000000000200200 0x78
+//  CHECK-NEXT:  .dynsym        0x0000000000200200 0x78        <internal>:(.dynsym)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .gnu.hash       0x0000000000200278 0x2c
+//  CHECK-NEXT:  .gnu.hash      0x0000000000200278 0x2c        <internal>:(.gnu.hash)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .hash           0x00000000002002a4 0x30
+//  CHECK-NEXT:  .hash          0x00000000002002a4 0x30        <internal>:(.hash)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .dynstr         0x00000000002002d4 0x31
+//  CHECK-NEXT:  .dynstr        0x00000000002002d4 0x31        <internal>:(.dynstr)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .rela.dyn       0x0000000000200308 0x30
+//  CHECK-NEXT:  .rela.dyn      0x0000000000200308 0x30        <internal>:(.rela.dyn)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .rela.plt       0x0000000000200338 0x30
+//  CHECK-NEXT:  .rela.plt      0x0000000000200338 0x30        <internal>:(.rela.plt)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .eh_frame       0x0000000000200368 0x64
+//  CHECK-NEXT:  .eh_frame      0x0000000000200368 0x64        <internal>:(.eh_frame)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .text           0x0000000000201000 0x2d
+//  CHECK-NEXT:  .text          0x0000000000201000 0x28        /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp1.o:(.text)
+//  CHECK-NEXT:                 0x0000000000201000                _start
+//  CHECK-NEXT:                 0x000000000020101f                f(int)
+//  CHECK-NEXT:                 0x0000000000201028                local
+//  CHECK-NEXT:  .text          0x0000000000201028 0x2         /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp2.o:(.text)
+//  CHECK-NEXT:                 0x0000000000201028                foo
+//  CHECK-NEXT:                 0x0000000000201029                bar
+//  CHECK-NEXT:  .text.zed      0x000000000020102a 0x0         /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp2.o:(.text.zed)
+//  CHECK-NEXT:                 0x000000000020102a                zed
+//  CHECK-NEXT:  .text          0x000000000020102c 0x0         /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp3.o:(.text)
+//  CHECK-NEXT:                 0x000000000020102c                bah
+//  CHECK-NEXT:  .text          0x000000000020102c 0x1         /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp4.a(gnu-map.s.tmp4.o):(.text)
+//  CHECK-NEXT:                 0x000000000020102c                baz
+// CHECK-EMPTY:
+//  CHECK-NEXT: .plt            0x0000000000201030 0x30
+//  CHECK-NEXT:  .plt           0x0000000000201030 0x30        <internal>:(.plt)
+//  CHECK-NEXT:                 0x0000000000201040                sharedFunc1
+//  CHECK-NEXT:                 0x0000000000201050                sharedFunc2
+// CHECK-EMPTY:
+//  CHECK-NEXT: .dynamic        0x0000000000202000 0x100
+//  CHECK-NEXT:  .dynamic       0x0000000000202000 0x100       <internal>:(.dynamic)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .got.plt        0x0000000000203000 0x28
+//  CHECK-NEXT:  .got.plt       0x0000000000203000 0x28        <internal>:(.got.plt)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .bss            0x0000000000203030 0x10
+//  CHECK-NEXT:  COMMON         0x0000000000203030 0x4         /home/ruiu/llvm/b/tools/lld/test/ELF/Output/gnu-map.s.tmp1.o:(COMMON)
+//  CHECK-NEXT:                 0x0000000000203030                common
+//  CHECK-NEXT:  .bss           0x0000000000203034 0x4         <internal>:(.bss)
+//  CHECK-NEXT:                 0x0000000000203034                sharedFoo
+//  CHECK-NEXT:  .bss           0x0000000000203038 0x8         <internal>:(.bss)
+//  CHECK-NEXT:                 0x0000000000203038                sharedBar
+// CHECK-EMPTY:
+//  CHECK-NEXT: .comment        0x0000000000000000 0x8
+//  CHECK-NEXT:  .comment       0x0000000000000000 0x8         <internal>:(.comment)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .symtab         0x0000000000000000 0x198
+//  CHECK-NEXT:  .symtab        0x0000000000000000 0x198       <internal>:(.symtab)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .shstrtab       0x0000000000000000 0x84
+//  CHECK-NEXT:  .shstrtab      0x0000000000000000 0x84        <internal>:(.shstrtab)
+// CHECK-EMPTY:
+//  CHECK-NEXT: .strtab         0x0000000000000000 0x6d
+//  CHECK-NEXT:  .strtab        0x0000000000000000 0x6d        <internal>:(.strtab)