[ELF] Add --print-archive-stats=

[MaskRay]

Summary

gold has an option --print-symbol-counts= which prints:

// For each archive
archive $archive $members $fetched_members
// For each object file
symbols $object $defined_symbols $used_defined_symbols

In most cases, $defined_symbols = $used_defined_symbols unless weak
symbols are present. Strangely $used_defined_symbols includes symbols defined relative to --gc-sections discarded sections.
The symbols lines do not appear to be useful.

archive lines are useful: $fetched_members=0 lines correspond to
unused archives. The information can be used to trim dependencies.

This patch implements --print-archive-stats= which prints the number of
members and the number of fetched members for each archive.

[alexander-shaposhnikov]

lld/ELF/InputFiles.cpp

1173

1. what about std::distance(file->child_begin(err), file->child_end()); ?
2. just curious why is the error consumed ? e.g. alternatively this could return Expected<...>

[alexander-shaposhnikov]

lld/ELF/Driver.h

33 ↗

(On Diff #260548)

if this vector is not modified outside of this class then probably it would be good to keep it private and provide a read-only accessor instead.

[MaskRay]

lld/ELF/InputFiles.cpp

1173

1. difference_type for these iterator types is not defined so std::distance can't be used:

/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_iterator_base_funcs.h:138:5: note: candidate template ignored: substitution failure
 [with _InputIterator = llvm::fallible_iterator<llvm::object::Archive::ChildFallibleIterator>]: no type named 'difference_type' in 'std::iterator_trai
ts<llvm::fallible_iterator<llvm::object::Archive::ChildFallibleIterator> >'

2. just curious why is the error consumed ? e.g. alternatively this could return Expected<...>

--whole-archive and a special case call getArchiveMembers. When the control flow reaches here, err will guaranteed to be success. For the rest the archive may be corrupted but we just don't care about the error here.

[MaskRay]

lld/ELF/Driver.h

33 ↗

(On Diff #260548)

The code style in lld may be a bit special. In many places public member variables are used in place of public member functions (especially getters and setters)...

[grimar]

lld/ELF/Driver.h

33 ↗

(On Diff #260548)

I think a bit more consistent way would be to add a archiveFiles vector and initialize it from doParseFile:

template <class ELFT> static void doParseFile(InputFile *file) {
  if (!isCompatible(file))
    return;

  // Binary file
  if (auto *f = dyn_cast<BinaryFile>(file)) {
    binaryFiles.push_back(f);
    f->parse();
    return;
  }

  // .a file
  if (auto *f = dyn_cast<ArchiveFile>(file)) {
    f->parse(); 
    /// <- HERE
    return;
  }
....

This would be consistent with other vectors we have:

std::vector<BinaryFile *> binaryFiles;
std::vector<BitcodeFile *> bitcodeFiles;
std::vector<LazyObjFile *> lazyObjFiles;
std::vector<InputFile *> objectFiles;
std::vector<SharedFile *> sharedFiles;

lld/ELF/InputFiles.cpp

1173

What about this?

size_t ArchiveFile::getMemberCount() const {
  Error err = Error::success();
  auto range = file->children(err);
  // A comment about why `consumeError` is OK.
  consumeError(std::move(err));

  size_t count = 0;
  for (auto i = range.begin(), e = range.end(); i != e; ++i)
    ++count;
  return count;
}

lld/ELF/MapFile.cpp

262

Should this code belong to MapFile.cpp? (I am not sure. The file header says it is for -Map, though we also handle -cref here)

269

I'd suggest to add the option name. We do it sometimes:

\lld\ELF\Driver.cpp(1408):    error("-image-base: number expected, but got " + s);
\lld\ELF\Driver.cpp(1479):    error("--undefined-glob: " + toString(pat.takeError()));

lld/ELF/Options.td

298

speicified -> specified

Perhaps, make it more generic? Something like "Write archives usage statistic to the .."

lld/ELF/Writer.cpp

603–604

The comment needs an update.

[MaskRay]

Address comments

lld/ELF/Driver.h

33 ↗

(On Diff #260548)

This will add yet another global variable, but since it is grouped with other similar variables, I think it is fine. In return, we can avoid private->public visibility change and expose Driver.h to MapFile.cpp, it is probably slightly better.

This exposes another issue that D46034 does not clear lazyObjFiles when relinking. I will clear it as well.

lld/ELF/InputFiles.cpp

1173

llvm::object::Archive::child_iterator keeps a reference to the Error object. consumeError(std::move(err)) has to be called after the iteration is done.

Using for (auto i = range.begin(), e = range.end(); i != e; ++i) can avoid an unused loop variable but it seems to become more complex. In the end I still prefer a range-based for loop, even if there is an unused loop variable.

lld/ELF/MapFile.cpp

262

It looks like we are using MapFile.cpp for ad-hoc statistical features.. This file hasn't become very complex. I think it is probably fine to keep it here (writeArchiveStats takes less than 20 lines). When it becomes more complex, we can consider a file rename.

[grimar]

lld/ELF/InputFiles.cpp

1178

What do you think if instead of adding a size_t getMemberCount() const; to ArchiveFile we add:

Archive* getArchiveFile() const;

And then just inline the logic of getMemberCount() to writeArchiveStats()?

(The logic is not used anywhere else and I can't say that such loops look nice,
perhaps hiding it away from the general code might look better).

[grimar]

LGTM.

lld/ELF/InputFiles.cpp

1173

but it seems to become more complex.

It shows the intention better though (IMO).

1178

Thinking about it again, I am not sure it will look much better as we already have
size_t getFetchedMemberCount() const { return seen.size(); } anyways.

lld/ELF/InputFiles.h

330

Members?

[grimar]

lld/ELF/InputFiles.h

330

"member count" seems correct, so nevermind.

lld/ELF/Config.h

struct Configuration {
   llvm::StringRef ltoObjPath;
   llvm::StringRef ltoSampleProfile;
   llvm::StringRef mapFile;
   llvm::StringRef outputFile;
   llvm::StringRef optRemarksFilename;
   llvm::StringRef optRemarksPasses;
   llvm::StringRef optRemarksFormat;
   llvm::StringRef progName;
+  llvm::StringRef printArchiveStats;
   llvm::StringRef printSymbolOrder;
   llvm::StringRef soName;
   llvm::StringRef sysroot;
   llvm::StringRef thinLTOCacheDir;
   llvm::StringRef thinLTOIndexOnlyArg;
   llvm::StringRef ltoBasicBlockSections;
   std::pair<llvm::StringRef, llvm::StringRef> thinLTOObjectSuffixReplace;
   std::pair<llvm::StringRef, llvm::StringRef> thinLTOPrefixReplace;

lld/ELF/Driver.cpp

bool link(ArrayRef<const char *> args, bool canExitEarly, raw_ostream &stdoutOS,
   errorHandler().errorLimitExceededMsg =
       "too many errors emitted, stopping now (use "
       "-error-limit=0 to see all errors)";
   errorHandler().exitEarly = canExitEarly;
   stderrOS.enable_colors(stderrOS.has_colors());
 
   inputSections.clear();
   outputSections.clear();
+  archiveFiles.clear();
   binaryFiles.clear();
   bitcodeFiles.clear();
   lazyObjFiles.clear();
   objectFiles.clear();
   sharedFiles.clear();
   backwardReferences.clear();
 
   config = make<Configuration>();
static void readConfigs(opt::InputArgList &args) {
   config->optimize = args::getInteger(args, OPT_O, 1);
   config->orphanHandling = getOrphanHandling(args);
   config->outputFile = args.getLastArgValue(OPT_o);
   config->pie = args.hasFlag(OPT_pie, OPT_no_pie, false);
   config->printIcfSections =
       args.hasFlag(OPT_print_icf_sections, OPT_no_print_icf_sections, false);
   config->printGcSections =
       args.hasFlag(OPT_print_gc_sections, OPT_no_print_gc_sections, false);
+  config->printArchiveStats = args.getLastArgValue(OPT_print_archive_stats);
   config->printSymbolOrder =
       args.getLastArgValue(OPT_print_symbol_order);
   config->rpath = getRpath(args);
   config->relocatable = args.hasArg(OPT_relocatable);
   config->saveTemps = args.hasArg(OPT_save_temps);
   if (args.hasArg(OPT_shuffle_sections))
     config->shuffleSectionSeed = args::getInteger(args, OPT_shuffle_sections, 0);
   config->searchPaths = args::getStrings(args, OPT_library_path);

lld/ELF/InputFiles.h

public:
   void parse();
 
   // Pulls out an object file that contains a definition for Sym and
   // returns it. If the same file was instantiated before, this
   // function does nothing (so we don't instantiate the same file
   // more than once.)
   void fetch(const Archive::Symbol &sym);
 
+  size_t getMemberCount() const;
+  size_t getFetchedMemberCount() const { return seen.size(); }
+
 private:
   std::unique_ptr<Archive> file;
   llvm::DenseSet<uint64_t> seen;
 };
 
 class BitcodeFile : public InputFile {
 public:
   BitcodeFile(MemoryBufferRef m, StringRef archiveName,
InputFile *createObjectFile(MemoryBufferRef mb, StringRef archiveName = "",
                             uint64_t offsetInArchive = 0);
 
 inline bool isBitcode(MemoryBufferRef mb) {
   return identify_magic(mb.getBuffer()) == llvm::file_magic::bitcode;
 }
 
 std::string replaceThinLTOSuffix(StringRef path);
 
+extern std::vector<ArchiveFile *> archiveFiles;
 extern std::vector<BinaryFile *> binaryFiles;
 extern std::vector<BitcodeFile *> bitcodeFiles;
 extern std::vector<LazyObjFile *> lazyObjFiles;
 extern std::vector<InputFile *> objectFiles;
 extern std::vector<SharedFile *> sharedFiles;
 
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/InputFiles.cpp

else
       f->toStringCache = (f->archiveName + "(" + f->getName() + ")").str();
   }
   return f->toStringCache;
 }
 
 namespace elf {
 bool InputFile::isInGroup;
 uint32_t InputFile::nextGroupId;
+std::vector<ArchiveFile *> archiveFiles;
 std::vector<BinaryFile *> binaryFiles;
 std::vector<BitcodeFile *> bitcodeFiles;
 std::vector<LazyObjFile *> lazyObjFiles;
 std::vector<InputFile *> objectFiles;
 std::vector<SharedFile *> sharedFiles;
 
 std::unique_ptr<TarWriter> tar;
 
template <class ELFT> static void doParseFile(InputFile *file) {
   if (auto *f = dyn_cast<BinaryFile>(file)) {
     binaryFiles.push_back(f);
     f->parse();
     return;
   }
 
   // .a file
   if (auto *f = dyn_cast<ArchiveFile>(file)) {
+    archiveFiles.push_back(f);
     f->parse();
     return;
   }
 
   // Lazy object file
   if (auto *f = dyn_cast<LazyObjFile>(file)) {
     lazyObjFiles.push_back(f);
     f->parse<ELFT>();
if (tar && c.getParent()->isThin())
     tar->append(relativeToRoot(CHECK(c.getFullName(), this)), mb.getBuffer());
 
   InputFile *file = createObjectFile(
       mb, getName(), c.getParent()->isThin() ? 0 : c.getChildOffset());
   file->groupId = groupId;
   parseFile(file);
 }
 
+size_t ArchiveFile::getMemberCount() const {
+  size_t count = 0;
+  Error err = Error::success();
+  for (const Archive::Child &c : file->children(err)) {
+    (void)c;
+    ++count;
+  }
+  // This function is used by --print-archive-stats=, where an error does not
+  // really matter.
+  consumeError(std::move(err));
+  return count;
+}
+
 unsigned SharedFile::vernauxNum;
 
 // Parse the version definitions in the object file if present, and return a
 // vector whose nth element contains a pointer to the Elf_Verdef for version
 // identifier n. Version identifiers that are not definitions map to nullptr.
 template <typename ELFT>
 static std::vector<const void *> parseVerdefs(const uint8_t *base,
                                               const typename ELFT::Shdr *sec) {

lld/ELF/MapFile.h

 //===- MapFile.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_MAPFILE_H
 #define LLD_ELF_MAPFILE_H
 
 namespace lld {
 namespace elf {
 void writeMapFile();
 void writeCrossReferenceTable();
+void writeArchiveStats();
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/MapFile.cpp

for (auto kv : map) {
 
     print(toString(*sym), toString(sym->file));
     for (InputFile *file : files)
       if (file != sym->file)
         print("", toString(file));
   }
 }
 
+void writeArchiveStats() {
+  if (config->printArchiveStats.empty())
+    return;
+
+  std::error_code ec;
+  raw_fd_ostream os(config->printArchiveStats, ec, sys::fs::OF_None);
+  if (ec) {
+    error("--print-archive-stats=: cannot open " + config->printArchiveStats +
+          ": " + ec.message());
+    return;
+  }
+
+  os << "members\tfetched\tarchive\n";
+  for (const ArchiveFile *f : archiveFiles)
+    os << f->getMemberCount() << '\t' << f->getFetchedMemberCount() << '\t'
+       << f->getName() << '\n';
+}
+
 } // namespace elf
 } // namespace lld

lld/ELF/Options.td

 defm print_gc_sections: B<"print-gc-sections",
     "List removed unused sections",
     "Do not list removed unused sections (default)">;
 
 defm print_icf_sections: B<"print-icf-sections",
     "List identical folded sections",
     "Do not list identical folded sections (default)">;
 
+def print_archive_stats: J<"print-archive-stats=">,
+  HelpText<"Write archive usage statistics to the specified file. "
+           "Print the numbers of members and fetched members for each archive">;
+
 defm print_symbol_order: Eq<"print-symbol-order",
   "Print a symbol order specified by --call-graph-ordering-file into the specified file">;
 
 def pop_state: F<"pop-state">,
   HelpText<"Undo the effect of -push-state">;
 
 def push_state: F<"push-state">,
   HelpText<"Save the current state of -as-needed, -static and -whole-archive">;

lld/ELF/Writer.cpp

else
     assignFileOffsetsBinary();
 
   for (Partition &part : partitions)
     setPhdrs(part);
 
   if (config->relocatable)
     for (OutputSection *sec : outputSections)
       sec->addr = 0;
 
-  // Handle --print-map(-M)/--Map and --cref. Dump them before checkSections()
-  // because the files may be useful in case checkSections() or openFile()
-  // fails, for example, due to an erroneous file size.
+  // Handle --print-map(-M)/--Map, --cref and --print-archive-stats=. Dump them
+  // before checkSections() because the files may be useful in case
+  // checkSections() or openFile() fails, for example, due to an erroneous file
+  // size.
   writeMapFile();
   writeCrossReferenceTable();
+  writeArchiveStats();
 
   if (config->checkSections)
     checkSections();
 
   // It does not make sense try to open the file if we have error already.
   if (errorCount())
     return;
   // Write the result down to a file.

lld/docs/ld.lld.1

 .It Fl -pie , Fl -pic-executable
 Create a position independent executable.
 .It Fl -print-gc-sections
 List removed unused sections.
 .It Fl -print-icf-sections
 List identical folded sections.
 .It Fl -print-map
 Print a link map to the standard output.
+.It Fl -print-archive-stats Ns = Ns Ar file
+Write archive usage statistics to the specified file.
+Print the numbers of members and fetched members for each archive.
 .It Fl -push-state
 Save the current state of
 .Fl -as-needed ,
 .Fl -static ,
 and
 .Fl -whole-archive.
 .It Fl -pop-state
 Undo the effect of

lld/test/ELF/print-archive-stats.s

+# REQUIRES: x86
+
+# RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o %t.o
+# RUN: echo '.globl weak; weak:' | llvm-mc -filetype=obj -triple=x86_64 - -o %tweak.o
+# RUN: echo '.global foo; foo:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t1.o
+# RUN: echo '.global bar; bar:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t2.o
+# RUN: echo '.global baz; baz:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t3.o
+# RUN: rm -f %tweak.a && llvm-ar rc %tweak.a %tweak.o
+# RUN: rm -f %t1.a && llvm-ar rc %t1.a %t1.o %t2.o %t3.o
+
+# RUN: ld.lld %t.o %tweak.a %t1.a --print-archive-stats=%t.txt -o /dev/null
+# RUN: FileCheck --input-file=%t.txt -DT=%t %s --match-full-lines --strict-whitespace
+
+## Fetches 0 member from %tweak.a and 2 members from %t1.a
+#      CHECK:members  fetched archive
+# CHECK-NEXT:1  0 [[T]]weak.a
+# CHECK-NEXT:3  2 [[T]]1.a
+
+## - means stdout.
+# RUN: ld.lld %t.o %tweak.a %t1.a --print-archive-stats=- -o /dev/null | diff %t.txt -
+
+## The second %t1.a has 0 fetched member.
+# RUN: ld.lld %t.o %tweak.a %t1.a %t1.a --print-archive-stats=- -o /dev/null | \
+# RUN:   FileCheck --check-prefix=CHECK2 %s
+# CHECK2:      members  fetched archive
+# CHECK2-NEXT: 1  0 {{.*}}weak.a
+# CHECK2-NEXT: 3  2 {{.*}}1.a
+# CHECK2-NEXT: 3  0 {{.*}}1.a
+
+# RUN: not ld.lld -shared %t.o --print-archive-stats=/ -o /dev/null 2>&1 | FileCheck --check-prefix=ERR %s
+# ERR: error: --print-archive-stats=: cannot open /: {{.*}}
+
+.globl _start
+.weak weak
+_start:
+  call foo
+  call bar
+  call weak