[ELF] Add --print-archive-stats=

[MaskRay]

Summary

gold has an option --print-symbol-counts= which prints:

// For each archive
archive $archive $members $fetched_members
// For each object file
symbols $object $defined_symbols $used_defined_symbols

In most cases, $defined_symbols = $used_defined_symbols unless weak
symbols are present. Strangely $used_defined_symbols includes symbols defined relative to --gc-sections discarded sections.
The symbols lines do not appear to be useful.

archive lines are useful: $fetched_members=0 lines correspond to
unused archives. The information can be used to trim dependencies.

This patch implements --print-archive-stats= which prints the number of
members and the number of fetched members for each archive.

[alexander-shaposhnikov]

lld/ELF/InputFiles.cpp

1171

1. what about std::distance(file->child_begin(err), file->child_end()); ?
2. just curious why is the error consumed ? e.g. alternatively this could return Expected<...>

[alexander-shaposhnikov]

lld/ELF/Driver.h

33

if this vector is not modified outside of this class then probably it would be good to keep it private and provide a read-only accessor instead.

[MaskRay]

lld/ELF/InputFiles.cpp

1171

1. difference_type for these iterator types is not defined so std::distance can't be used:

/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_iterator_base_funcs.h:138:5: note: candidate template ignored: substitution failure
 [with _InputIterator = llvm::fallible_iterator<llvm::object::Archive::ChildFallibleIterator>]: no type named 'difference_type' in 'std::iterator_trai
ts<llvm::fallible_iterator<llvm::object::Archive::ChildFallibleIterator> >'

2. just curious why is the error consumed ? e.g. alternatively this could return Expected<...>

--whole-archive and a special case call getArchiveMembers. When the control flow reaches here, err will guaranteed to be success. For the rest the archive may be corrupted but we just don't care about the error here.

[MaskRay]

lld/ELF/Driver.h

33

The code style in lld may be a bit special. In many places public member variables are used in place of public member functions (especially getters and setters)...

[grimar]

lld/ELF/Driver.h

33

I think a bit more consistent way would be to add a archiveFiles vector and initialize it from doParseFile:

template <class ELFT> static void doParseFile(InputFile *file) {
  if (!isCompatible(file))
    return;

  // Binary file
  if (auto *f = dyn_cast<BinaryFile>(file)) {
    binaryFiles.push_back(f);
    f->parse();
    return;
  }

  // .a file
  if (auto *f = dyn_cast<ArchiveFile>(file)) {
    f->parse(); 
    /// <- HERE
    return;
  }
....

This would be consistent with other vectors we have:

std::vector<BinaryFile *> binaryFiles;
std::vector<BitcodeFile *> bitcodeFiles;
std::vector<LazyObjFile *> lazyObjFiles;
std::vector<InputFile *> objectFiles;
std::vector<SharedFile *> sharedFiles;

lld/ELF/InputFiles.cpp

1171

What about this?

size_t ArchiveFile::getMemberCount() const {
  Error err = Error::success();
  auto range = file->children(err);
  // A comment about why `consumeError` is OK.
  consumeError(std::move(err));

  size_t count = 0;
  for (auto i = range.begin(), e = range.end(); i != e; ++i)
    ++count;
  return count;
}

lld/ELF/MapFile.cpp

263

Should this code belong to MapFile.cpp? (I am not sure. The file header says it is for -Map, though we also handle -cref here)

270

I'd suggest to add the option name. We do it sometimes:

\lld\ELF\Driver.cpp(1408):    error("-image-base: number expected, but got " + s);
\lld\ELF\Driver.cpp(1479):    error("--undefined-glob: " + toString(pat.takeError()));

lld/ELF/Options.td

301

speicified -> specified

Perhaps, make it more generic? Something like "Write archives usage statistic to the .."

lld/ELF/Writer.cpp

604

The comment needs an update.

[MaskRay]

Address comments

lld/ELF/Driver.h

33

This will add yet another global variable, but since it is grouped with other similar variables, I think it is fine. In return, we can avoid private->public visibility change and expose Driver.h to MapFile.cpp, it is probably slightly better.

This exposes another issue that D46034 does not clear lazyObjFiles when relinking. I will clear it as well.

lld/ELF/InputFiles.cpp

1171

llvm::object::Archive::child_iterator keeps a reference to the Error object. consumeError(std::move(err)) has to be called after the iteration is done.

Using for (auto i = range.begin(), e = range.end(); i != e; ++i) can avoid an unused loop variable but it seems to become more complex. In the end I still prefer a range-based for loop, even if there is an unused loop variable.

lld/ELF/MapFile.cpp

263

It looks like we are using MapFile.cpp for ad-hoc statistical features.. This file hasn't become very complex. I think it is probably fine to keep it here (writeArchiveStats takes less than 20 lines). When it becomes more complex, we can consider a file rename.

[grimar]

lld/ELF/InputFiles.cpp

1176

What do you think if instead of adding a size_t getMemberCount() const; to ArchiveFile we add:

Archive* getArchiveFile() const;

And then just inline the logic of getMemberCount() to writeArchiveStats()?

(The logic is not used anywhere else and I can't say that such loops look nice,
perhaps hiding it away from the general code might look better).

[grimar]

LGTM.

lld/ELF/InputFiles.cpp

1171

but it seems to become more complex.

It shows the intention better though (IMO).

1176

Thinking about it again, I am not sure it will look much better as we already have
size_t getFetchedMemberCount() const { return seen.size(); } anyways.

lld/ELF/InputFiles.h

330

Members?

[grimar]

lld/ELF/InputFiles.h

330

"member count" seems correct, so nevermind.

lld/ELF/Config.h

struct Configuration {
   llvm::StringRef ltoObjPath;
   llvm::StringRef ltoSampleProfile;
   llvm::StringRef mapFile;
   llvm::StringRef outputFile;
   llvm::StringRef optRemarksFilename;
   llvm::StringRef optRemarksPasses;
   llvm::StringRef optRemarksFormat;
   llvm::StringRef progName;
+  llvm::StringRef printArchiveStats;
   llvm::StringRef printSymbolOrder;
   llvm::StringRef soName;
   llvm::StringRef sysroot;
   llvm::StringRef thinLTOCacheDir;
   llvm::StringRef thinLTOIndexOnlyArg;
   llvm::StringRef ltoBasicBlockSections;
   std::pair<llvm::StringRef, llvm::StringRef> thinLTOObjectSuffixReplace;
   std::pair<llvm::StringRef, llvm::StringRef> thinLTOPrefixReplace;

lld/ELF/Driver.h

 extern class LinkerDriver *driver;
 
 class LinkerDriver {
 public:
   void main(ArrayRef<const char *> args);
   void addFile(StringRef path, bool withLOption);
   void addLibrary(StringRef name);
 
+  std::vector<InputFile *> files;
+
 private:
   void createFiles(llvm::opt::InputArgList &args);
   void inferMachineType();
   template <class ELFT> void link(llvm::opt::InputArgList &args);
   template <class ELFT> void compileBitcodeFiles();
 
   // True if we are in --whole-archive and --no-whole-archive.
   bool inWholeArchive = false;
 
   // True if we are in --start-lib and --end-lib.
   bool inLib = false;
 
   // For LTO.
   std::unique_ptr<BitcodeCompiler> lto;
-
-  std::vector<InputFile *> files;
 };
 
 // Parses command line options.
 class ELFOptTable : public llvm::opt::OptTable {
 public:
   ELFOptTable();
   llvm::opt::InputArgList parse(ArrayRef<const char *> argv);
 };

lld/ELF/Driver.cpp

static void readConfigs(opt::InputArgList &args) {
   config->optimize = args::getInteger(args, OPT_O, 1);
   config->orphanHandling = getOrphanHandling(args);
   config->outputFile = args.getLastArgValue(OPT_o);
   config->pie = args.hasFlag(OPT_pie, OPT_no_pie, false);
   config->printIcfSections =
       args.hasFlag(OPT_print_icf_sections, OPT_no_print_icf_sections, false);
   config->printGcSections =
       args.hasFlag(OPT_print_gc_sections, OPT_no_print_gc_sections, false);
+  config->printArchiveStats = args.getLastArgValue(OPT_print_archive_stats);
   config->printSymbolOrder =
       args.getLastArgValue(OPT_print_symbol_order);
   config->rpath = getRpath(args);
   config->relocatable = args.hasArg(OPT_relocatable);
   config->saveTemps = args.hasArg(OPT_save_temps);
   if (args.hasArg(OPT_shuffle_sections))
     config->shuffleSectionSeed = args::getInteger(args, OPT_shuffle_sections, 0);
   config->searchPaths = args::getStrings(args, OPT_library_path);

lld/ELF/InputFiles.h

public:
   void parse();
 
   // Pulls out an object file that contains a definition for Sym and
   // returns it. If the same file was instantiated before, this
   // function does nothing (so we don't instantiate the same file
   // more than once.)
   void fetch(const Archive::Symbol &sym);
 
+  size_t getMemberCount() const;
+  size_t getFetchedMemberCount() const { return seen.size(); }
+
 private:
   std::unique_ptr<Archive> file;
   llvm::DenseSet<uint64_t> seen;
 };
 
 class BitcodeFile : public InputFile {
 public:
   BitcodeFile(MemoryBufferRef m, StringRef archiveName,

lld/ELF/InputFiles.cpp

if (tar && c.getParent()->isThin())
     tar->append(relativeToRoot(CHECK(c.getFullName(), this)), mb.getBuffer());
 
   InputFile *file = createObjectFile(
       mb, getName(), c.getParent()->isThin() ? 0 : c.getChildOffset());
   file->groupId = groupId;
   parseFile(file);
 }
 
+size_t ArchiveFile::getMemberCount() const {
+  size_t count = 0;
+  Error err = Error::success();
+  for (const Archive::Child &c : file->children(err)) {
+    (void)c;
+    ++count;
+  }
+  consumeError(std::move(err));
+  return count;
+}
+
 unsigned SharedFile::vernauxNum;
 
 // Parse the version definitions in the object file if present, and return a
 // vector whose nth element contains a pointer to the Elf_Verdef for version
 // identifier n. Version identifiers that are not definitions map to nullptr.
 template <typename ELFT>
 static std::vector<const void *> parseVerdefs(const uint8_t *base,
                                               const typename ELFT::Shdr *sec) {

lld/ELF/MapFile.h

 //===- MapFile.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_MAPFILE_H
 #define LLD_ELF_MAPFILE_H
 
 namespace lld {
 namespace elf {
 void writeMapFile();
 void writeCrossReferenceTable();
+void writeArchiveStats();
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/MapFile.cpp

 //   00201000 00000015     4 .text
 //   00201000 0000000e     4         test.o:(.text)
 //   0020100e 00000000     0                 local
 //   00201005 00000000     0                 f(int)
 //
 //===----------------------------------------------------------------------===//
 
 #include "MapFile.h"
+#include "Driver.h"
 #include "InputFiles.h"
 #include "LinkerScript.h"
 #include "OutputSections.h"
 #include "SymbolTable.h"
 #include "Symbols.h"
 #include "SyntheticSections.h"
 #include "lld/Common/Strings.h"
 #include "lld/Common/Threads.h"
for (auto kv : map) {
 
     print(toString(*sym), toString(sym->file));
     for (InputFile *file : files)
       if (file != sym->file)
         print("", toString(file));
   }
 }
 
+void writeArchiveStats() {
+  if (config->printArchiveStats.empty())
+    return;
+
+  std::error_code ec;
+  raw_fd_ostream os(config->printArchiveStats, ec, sys::fs::OF_None);
+  if (ec) {
+    error("cannot open " + config->printArchiveStats + ": " + ec.message());
+    return;
+  }
+
+  os << "members\tfetched\tarchive\n";
+  for (const elf::InputFile *file : driver->files)
+    if (const auto *f = dyn_cast<ArchiveFile>(file))
+      os << f->getMemberCount() << '\t' << f->getFetchedMemberCount() << '\t'
+         << f->getName() << '\n';
+}
+
 } // namespace elf
 } // namespace lld

lld/ELF/Options.td

 defm print_gc_sections: B<"print-gc-sections",
     "List removed unused sections",
     "Do not list removed unused sections (default)">;
 
 defm print_icf_sections: B<"print-icf-sections",
     "List identical folded sections",
     "Do not list identical folded sections (default)">;
 
+def print_archive_stats: J<"print-archive-stats=">,
+  HelpText<"Print the numbers of members and fetched members for each archive into the speicified file">;
+
 defm print_symbol_order: Eq<"print-symbol-order",
   "Print a symbol order specified by --call-graph-ordering-file into the specified file">;
 
 def pop_state: F<"pop-state">,
   HelpText<"Undo the effect of -push-state">;
 
 def push_state: F<"push-state">,
   HelpText<"Save the current state of -as-needed, -static and -whole-archive">;

lld/ELF/Writer.cpp

template <class ELFT> void Writer<ELFT>::run() {
 
   for (Partition &part : partitions)
     setPhdrs(part);
 
   if (config->relocatable)
     for (OutputSection *sec : outputSections)
       sec->addr = 0;
 
   // Handle --print-map(-M)/--Map and --cref. Dump them before checkSections()
   // because the files may be useful in case checkSections() or openFile()
   // fails, for example, due to an erroneous file size.
   writeMapFile();
   writeCrossReferenceTable();
+  writeArchiveStats();
 
   if (config->checkSections)
     checkSections();
 
   // It does not make sense try to open the file if we have error already.
   if (errorCount())
     return;
   // Write the result down to a file.

lld/docs/ld.lld.1

 .It Fl -pie , Fl -pic-executable
 Create a position independent executable.
 .It Fl -print-gc-sections
 List removed unused sections.
 .It Fl -print-icf-sections
 List identical folded sections.
 .It Fl -print-map
 Print a link map to the standard output.
+.It Fl -print-archive-stats Ns = Ns Ar file
+Print the numbers of members and fetched members for each archive into the speicified file.
 .It Fl -push-state
 Save the current state of
 .Fl -as-needed ,
 .Fl -static ,
 and
 .Fl -whole-archive.
 .It Fl -pop-state
 Undo the effect of

lld/test/ELF/print-archive-stats.s

+# REQUIRES: x86
+
+# RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o %t.o
+# RUN: echo '.globl weak; weak:' | llvm-mc -filetype=obj -triple=x86_64 - -o %tweak.o
+# RUN: echo '.global foo; foo:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t1.o
+# RUN: echo '.global bar; bar:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t2.o
+# RUN: echo '.global baz; baz:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t3.o
+# RUN: rm -f %tweak.a && llvm-ar rc %tweak.a %tweak.o
+# RUN: rm -f %t1.a && llvm-ar rc %t1.a %t1.o %t2.o %t3.o
+
+# RUN: ld.lld %t.o %tweak.a %t1.a --print-archive-stats=%t.txt -o /dev/null
+# RUN: FileCheck --input-file=%t.txt -DT=%t %s --match-full-lines --strict-whitespace
+
+## Fetches 0 member from %tweak.a and 2 members from %t1.a
+#      CHECK:members  fetched archive
+# CHECK-NEXT:1  0 [[T]]weak.a
+# CHECK-NEXT:3  2 [[T]]1.a
+
+# RUN: ld.lld %t.o %tweak.a %t1.a --print-archive-stats=/dev/stdout -o /dev/null | diff %t.txt -
+
+## The second %t1.a has 0 fetched member.
+# RUN: ld.lld %t.o %tweak.a %t1.a %t1.a --print-archive-stats=/dev/stdout -o /dev/null | \
+# RUN:   FileCheck --check-prefix=CHECK2 %s
+# CHECK2:      members  fetched archive
+# CHECK2-NEXT: 1  0 {{.*}}weak.a
+# CHECK2-NEXT: 3  2 {{.*}}1.a
+# CHECK2-NEXT: 3  0 {{.*}}1.a
+
+# RUN: not ld.lld -shared %t.o --print-archive-stats=/ -o /dev/null 2>&1 | FileCheck --check-prefix=ERR %s
+# ERR: error: cannot open /: {{.*}}
+
+.globl _start
+.weak weak
+_start:
+  call foo
+  call bar
+  call weak