[dsymutil] Add reproducers to dsymutil

[JDevlieghere]

Summary

Add support for generating a dsymutil reproducer, which is a folder containing all the object files for linking. When --gen-reproducer is passed, dsymutil uses a FileCollectorFileSystem which keeps track of all the files used by dsymutil. These files are copied into a temporary directory when dsymutil exists. When this path is passed to --use-reproducer, dsymutil uses a RedirectingFileSystem that will use the files from the reproducer directory instead of the actual paths. This means you don't need to mess with the OSO path prefix.

[JDevlieghere]

This needs tests and some polish but please let me know what you think of the general approach.

[avl]

generally it looks good for me, I only have a couple of inline comments.

llvm/tools/dsymutil/Reproducer.cpp

42

It looks like it is not an error, but just logging. Thus it should probably be put under Verbose flag?

if (Options.Verbose)
      outs() << "Reproducer written"

llvm/tools/dsymutil/dsymutil.cpp

101

it would be good to initialize this:
ReproMode = ReproducerMode::Off;

[aprantl]

llvm/tools/dsymutil/Options.td

166

is there a reason why this isn't called --generate-reproducer? People are not going to call this very often (I hope!) so we should optimize for legibility?

170

What about --record-reproducer and --replay-reproducer?

[JDevlieghere]

llvm/tools/dsymutil/Options.td

166

I did it for consistently with clang, which has the same flag.

170

It's funny you say this because that's what I had originally. It would certainly be more consistent with lldb. I changed it because I don't think "replay" is accurate for dsymutil. I don't plan on capturing anything beyond the input files, so really there's nothing being "replayed". The functionality is more closely related to what clang does, which is why I went with "generate" instead of "capture".

llvm/tools/dsymutil/Reproducer.cpp

42

It's more than logging, it's the only way to find out where the reproducer has been written. I'll change it to outs(), but I don't think it should be behind the Verbose flag.

[avl]

llvm/tools/dsymutil/Reproducer.cpp

42

I see. Let it be just outs() then.

[JDevlieghere]

• Address code review feedback
• Add tests

[aprantl]

llvm/test/tools/dsymutil/X86/reproducer.test

27

most of these should be CHECK-NEXT?

llvm/test/tools/dsymutil/cmdline.test

3

What do you think about adding

RUN: cat dsymutil.rst | FileCheck --check-prefix=HELP

here to ensure all options are documented?

llvm/tools/dsymutil/Options.td

170

Can you add these to the man page as well?

[JDevlieghere]

• Use CHECK-NEXT

llvm/test/tools/dsymutil/cmdline.test

3

https://reviews.llvm.org/rGb7924d6525be103b6ffd93c7b9787d4339b3ad61

[friss]

I don't know enough about the VFS to do a thorough review, but the high level logic SGTM

[thakis]

My bots fail with:

../../llvm/tools/dsymutil/Reproducer.cpp:10:10: error: 'Reproducer.h' file not found with <angled> include; use "quotes" instead
#include <Reproducer.h>
         ^~~~~~~~~~~~~~
         "Reproducer.h"

Any reason for the <> include style?

[JDevlieghere]

In D79398#2049526, @thakis wrote:

My bots fail with:

../../llvm/tools/dsymutil/Reproducer.cpp:10:10: error: 'Reproducer.h' file not found with <angled> include; use "quotes" instead
#include <Reproducer.h>
         ^~~~~~~~~~~~~~
         "Reproducer.h"

Any reason for the <> include style?

No, must have been unconscious/unintentional. Fixed in d395eacca579858972fd4f79f4c9637db7666392.

llvm/test/tools/dsymutil/X86/reproducer.test

+# Recreate the folder structure in a temp directory we can remove later.
+RUN: rm -rf %t
+RUN: mkdir -p %t/Inputs
+RUN: cp %p/../Inputs/basic.macho.x86_64 %t/Inputs
+RUN: cp %p/../Inputs/basic1.macho.x86_64.o %t/Inputs
+RUN: cp %p/../Inputs/basic2.macho.x86_64.o %t/Inputs
+RUN: cp %p/../Inputs/basic3.macho.x86_64.o %t/Inputs
+
+# Sanity check all the files are present.
+RUN: dsymutil -f -o - -oso-prepend-path=%t %t/Inputs/basic.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s
+
+# Create a reproducer.
+RUN: env DSYMUTIL_REPRODUCER_PATH=%t.repro dsymutil -gen-reproducer -f -o %t.generate -oso-prepend-path=%t %t/Inputs/basic.macho.x86_64 | FileCheck %s --check-prefixes=REPRODUCER
+RUN: llvm-dwarfdump -a %t.generate | FileCheck %s
+
+# Remove the input files and sanity check that was successful.
+RUN: rm -rf %t
+RUN: not dsymutil -f -o %t.error -oso-prepend-path=%t %t/Inputs/basic.macho.x86_64 2>&1 | FileCheck %s --check-prefix=ERROR
+
+# Use the reproducer.
+RUN: dsymutil -use-reproducer %t.repro -f -o - -oso-prepend-path=%t %t/Inputs/basic.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s
+
+# Conflicting options.
+RUN: not dsymutil -gen-reproducer -use-reproducer %t.repro -f -o %t.error -oso-prepend-path=%t %t/Inputs/basic.macho.x86_64 2>&1 |  FileCheck %s --check-prefix=CONFLICT
+
+CHECK:  DW_TAG_compile_unit
+CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
+CHECK:    DW_AT_language (DW_LANG_C99)
+CHECK:    DW_AT_name ("basic1.c")
+CHECK:    DW_AT_stmt_list (0x00000000)
+CHECK:    DW_AT_comp_dir ("/Inputs")
+CHECK:    DW_AT_low_pc (0x0000000100000ea0)
+CHECK:    DW_TAG_subprogram
+CHECK:    DW_AT_name ("main")
+CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
+CHECK:      DW_AT_decl_line (23)
+CHECK:      DW_AT_prototyped (0x01)
+CHECK:      DW_AT_type (0x00000063
+CHECK:      DW_AT_external (0x01)
+CHECK:      DW_AT_accessibility (DW_ACCESS_public)
+CHECK:      DW_AT_low_pc (0x0000000100000ea0)
+CHECK:      DW_AT_high_pc (0x0000000100000ec4)
+CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
+CHECK:      DW_TAG_formal_parameter
+CHECK:        DW_AT_name ("argc")
+CHECK:        DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
+CHECK:        DW_AT_decl_line (23)
+CHECK:        DW_AT_type (0x00000063
+CHECK:        DW_AT_location (DW_OP_fbreg -8)
+CHECK:      DW_TAG_formal_parameter
+CHECK:        DW_AT_name ("argv")
+CHECK:        DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
+CHECK:        DW_AT_decl_line (23)
+CHECK:        DW_AT_type (0x0000006a
+CHECK:        DW_AT_location (DW_OP_fbreg -16)
+CHECK:      NULL
+CHECK:    DW_TAG_base_type
+CHECK:      DW_AT_name ("int")
+CHECK:      DW_AT_encoding (DW_ATE_signed)
+CHECK:      DW_AT_byte_size (0x04)
+CHECK:    DW_TAG_pointer_type
+CHECK:      DW_AT_type (0x0000006f
+CHECK:    DW_TAG_pointer_type
+CHECK:      DW_AT_type (0x00000074
+CHECK:    DW_TAG_const_type
+CHECK:      DW_AT_type (0x00000079
+CHECK:    DW_TAG_base_type
+CHECK:      DW_AT_name ("char")
+CHECK:      DW_AT_encoding (DW_ATE_signed_char)
+CHECK:      DW_AT_byte_size (0x01)
+CHECK:    NULL
+
+REPRODUCER: reproducer written
+ERROR: error: cannot parse the debug map
+CONFLICT: cannot combine --gen-reproducer and --use-reproducer

llvm/test/tools/dsymutil/cmdline.test

 RUN: dsymutil -help 2>&1 | FileCheck --check-prefix=HELP %s
 HELP: OVERVIEW: manipulate archived DWARF debug symbol files.
 HELP: USAGE: {{.*}}dsymutil{{[^ ]*}} [options] <input files>
 HELP-NOT: -reverse-iterate
 HELP: Dsymutil Options:
 HELP: -accelerator
 HELP: -arch <arch>
 HELP: -dump-debug-map
 HELP: -flat
+HELP: -gen-reproducer
 HELP: -minimize
 HELP: -no-odr
 HELP: -no-output
 HELP: -no-swiftmodule-timestamp
 HELP: -num-threads <threads>
 HELP: -object-prefix-map <prefix=remapped>
 HELP: -oso-prepend-path <path>
 HELP: {{ -o <filename> }}
 HELP: -papertrail
 HELP: -remarks-output-format <format>
 HELP: -remarks-prepend-path <path>
 HELP: -statistics
 HELP: -symbol-map
 HELP: -symtab
 HELP: {{ -S }}
 HELP: -toolchain
 HELP: -update
+HELP: -use-reproducer <path>
 HELP: -verbose
 HELP: -verify
 HELP: {{ -y }}
 HELP-NOT: -reverse-iterate
 
 RUN: dsymutil --version 2>&1 | FileCheck --check-prefix=VERSION %s
 VERSION: {{ version }}
 
 RUN: not dsymutil 2>&1 | FileCheck --check-prefix=NOINPUT %s
 NOINPUT: error: no input files specified
 
 RUN: dsymutil -bogus -help 2>&1 | FileCheck --check-prefix=BOGUS %s
 BOGUS: warning: ignoring unknown option: -bogus

llvm/tools/dsymutil/CMakeLists.txt

 add_llvm_tool(dsymutil
   dsymutil.cpp
   BinaryHolder.cpp
   CFBundle.cpp
   DebugMap.cpp
   DwarfLinkerForBinary.cpp
   MachODebugMapParser.cpp
   MachOUtils.cpp
+  Reproducer.cpp
   SymbolMap.cpp
 
   DEPENDS
   intrinsics_gen
   ${tablegen_deps}
   )
 
 if(APPLE)
   target_link_libraries(dsymutil PRIVATE "-framework CoreFoundation")
 endif(APPLE)
 
 if(HAVE_CXX_ATOMICS_WITH_LIB OR HAVE_CXX_ATOMICS64_WITH_LIB)
   target_link_libraries(dsymutil PRIVATE atomic)
 endif()

llvm/tools/dsymutil/Options.td

def threads: Separate<["--", "-"], "num-threads">,
   MetaVarName<"<threads>">,
   HelpText<"Specifies the maximum number of simultaneous threads to use when linking multiple architectures.">,
   Group<grp_general>;
 def: Separate<["-"], "j">,
   Alias<threads>,
   HelpText<"Alias for --num-threads">,
   Group<grp_general>;
 
+def gen_reproducer: F<"gen-reproducer">,
+  HelpText<"Generate a reproducer consisting of the input object files.">,
+  Group<grp_general>;
+
+def use_reproducer: Separate<["--", "-"], "use-reproducer">,
+  MetaVarName<"<path>">,
+  HelpText<"Use the object files from the given reproducer path.">,
+  Group<grp_general>;
+def: Joined<["--", "-"], "use-reproducer=">, Alias<use_reproducer>;
+
 def remarks_prepend_path: Separate<["--", "-"], "remarks-prepend-path">,
   MetaVarName<"<path>">,
   HelpText<"Specify a directory to prepend to the paths of the external remark files.">,
   Group<grp_general>;
 def: Joined<["--", "-"], "remarks-prepend-path=">, Alias<remarks_prepend_path>;
 
 def remarks_output_format: Separate<["--", "-"], "remarks-output-format">,
   MetaVarName<"<format>">,
   HelpText<"Specify the format to be used when serializing the linked remarks.">,
   Group<grp_general>;
 def: Joined<["--", "-"], "remarks-output-format=">, Alias<remarks_output_format>;

llvm/tools/dsymutil/Reproducer.h

+//===- tools/dsymutil/Reproducer.h ------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_DSYMUTIL_REPRODUCER_H
+#define LLVM_TOOLS_DSYMUTIL_REPRODUCER_H
+
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileCollector.h"
+#include "llvm/Support/VirtualFileSystem.h"
+
+namespace llvm {
+namespace dsymutil {
+
+enum class ReproducerMode {
+  Generate,
+  Use,
+  Off,
+};
+
+class Reproducer {
+public:
+  Reproducer();
+  virtual ~Reproducer();
+  IntrusiveRefCntPtr<vfs::FileSystem> getVFS() const { return VFS; }
+
+  static llvm::Expected<std::unique_ptr<Reproducer>>
+  createReproducer(ReproducerMode Mode, StringRef Root);
+
+protected:
+  IntrusiveRefCntPtr<vfs::FileSystem> VFS;
+};
+
+class ReproducerGenerate : public Reproducer {
+public:
+  ReproducerGenerate(std::error_code &EC);
+  ~ReproducerGenerate() override;
+
+private:
+  std::string Root;
+  std::shared_ptr<FileCollector> FC;
+};
+
+class ReproducerUse : public Reproducer {
+public:
+  ReproducerUse(StringRef Root, std::error_code &EC);
+  ~ReproducerUse() override;
+
+private:
+  std::string Root;
+};
+
+} // end namespace dsymutil
+} // end namespace llvm
+
+#endif // LLVM_TOOLS_DSYMUTIL_REPRODUCER_H

llvm/tools/dsymutil/Reproducer.cpp

+//===- Reproducer.cpp -----------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/Path.h"
+#include <Reproducer.h>
+
+using namespace llvm;
+using namespace llvm::dsymutil;
+
+static std::string createReproducerDir(std::error_code &EC) {
+  SmallString<128> Root;
+  if (const char *Path = getenv("DSYMUTIL_REPRODUCER_PATH")) {
+    Root.assign(Path);
+    EC = sys::fs::create_directory(Root);
+  } else {
+    EC = sys::fs::createUniqueDirectory("dsymutil", Root);
+  }
+  return EC ? "" : std::string(Root);
+}
+
+Reproducer::Reproducer() : VFS(vfs::getRealFileSystem()) {}
+Reproducer::~Reproducer() = default;
+
+ReproducerGenerate::ReproducerGenerate(std::error_code &EC)
+    : Root(createReproducerDir(EC)), FC() {
+  if (!Root.empty())
+    FC = std::make_shared<FileCollector>(Root, Root);
+  VFS = FileCollector::createCollectorVFS(vfs::getRealFileSystem(), FC);
+}
+
+ReproducerGenerate::~ReproducerGenerate() {
+  if (!FC)
+    return;
+  FC->copyFiles(false);
+  SmallString<128> Mapping(Root);
+  sys::path::append(Mapping, "mapping.yaml");
+  FC->writeMapping(Mapping.str());
+  outs() << "reproducer written to " << Root << '\n';
+}
+
+ReproducerUse::~ReproducerUse() = default;
+
+ReproducerUse::ReproducerUse(StringRef Root, std::error_code &EC) {
+  SmallString<128> Mapping(Root);
+  sys::path::append(Mapping, "mapping.yaml");
+  ErrorOr<std::unique_ptr<MemoryBuffer>> Buffer =
+      vfs::getRealFileSystem()->getBufferForFile(Mapping.str());
+
+  if (!Buffer) {
+    EC = Buffer.getError();
+    return;
+  }
+
+  VFS = llvm::vfs::getVFSFromYAML(std::move(Buffer.get()), nullptr, Mapping);
+}
+
+llvm::Expected<std::unique_ptr<Reproducer>>
+Reproducer::createReproducer(ReproducerMode Mode, StringRef Root) {
+  switch (Mode) {
+  case ReproducerMode::Generate: {
+    std::error_code EC;
+    auto Repro = std::make_unique<ReproducerGenerate>(EC);
+    if (EC)
+      return errorCodeToError(EC);
+    return Repro;
+  }
+  case ReproducerMode::Use: {
+    std::error_code EC;
+    auto Repro = std::make_unique<ReproducerUse>(Root, EC);
+    if (EC)
+      return errorCodeToError(EC);
+    return Repro;
+  }
+  case ReproducerMode::Off:
+    return std::make_unique<Reproducer>();
+  }
+}

llvm/tools/dsymutil/dsymutil.cpp

 //===----------------------------------------------------------------------===//
 
 #include "dsymutil.h"
 #include "BinaryHolder.h"
 #include "CFBundle.h"
 #include "DebugMap.h"
 #include "LinkUtils.h"
 #include "MachOUtils.h"
+#include "Reproducer.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFVerifier.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/MachO.h"
 #include "llvm/Option/Arg.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileCollector.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/WithColor.h"
 #include "llvm/Support/raw_ostream.h"
struct DsymutilOptions {
   bool DumpStab = false;
   bool Flat = false;
   bool InputIsYAMLDebugMap = false;
   bool PaperTrailWarnings = false;
   bool Verify = false;
   std::string SymbolMap;
   std::string OutputFile;
   std::string Toolchain;
+  std::string ReproducerPath;
   std::vector<std::string> Archs;
   std::vector<std::string> InputFiles;
   unsigned NumThreads;
+  ReproducerMode ReproMode = ReproducerMode::Off;
   dsymutil::LinkOptions LinkOpts;
 };
 
 /// Return a list of input files. This function has logic for dealing with the
 /// special case where we might have dSYM bundles as input. The function
 /// returns an error when the directory structure doesn't match that of a dSYM
 /// bundle.
 static Expected<std::vector<std::string>> getInputs(opt::InputArgList &Args,
return make_error<StringError>(
         "cannot use -o with multiple inputs in flat mode.",
         errc::invalid_argument);
 
   if (Options.PaperTrailWarnings && Options.InputIsYAMLDebugMap)
     return make_error<StringError>(
         "paper trail warnings are not supported for YAML input.",
         errc::invalid_argument);
 
+  if (!Options.ReproducerPath.empty() &&
+      Options.ReproMode != ReproducerMode::Use)
+    return make_error<StringError>(
+        "cannot combine --gen-reproducer and --use-reproducer.",
+        errc::invalid_argument);
+
   return Error::success();
 }
 
 static Expected<AccelTableKind> getAccelTableKind(opt::InputArgList &Args) {
   if (opt::Arg *Accelerator = Args.getLastArg(OPT_accelerator)) {
     StringRef S = Accelerator->getValue();
     if (S == "Apple")
       return AccelTableKind::Apple;
static Expected<DsymutilOptions> getOptions(opt::InputArgList &Args) {
   Options.LinkOpts.Minimize = Args.hasArg(OPT_minimize);
   Options.LinkOpts.NoODR = Args.hasArg(OPT_no_odr);
   Options.LinkOpts.NoOutput = Args.hasArg(OPT_no_output);
   Options.LinkOpts.NoTimestamp = Args.hasArg(OPT_no_swiftmodule_timestamp);
   Options.LinkOpts.Update = Args.hasArg(OPT_update);
   Options.LinkOpts.Verbose = Args.hasArg(OPT_verbose);
   Options.LinkOpts.Statistics = Args.hasArg(OPT_statistics);
 
+  if (opt::Arg *ReproducerPath = Args.getLastArg(OPT_use_reproducer)) {
+    Options.ReproMode = ReproducerMode::Use;
+    Options.ReproducerPath = ReproducerPath->getValue();
+  }
+
+  if (Args.hasArg(OPT_gen_reproducer))
+    Options.ReproMode = ReproducerMode::Generate;
+
   if (Expected<AccelTableKind> AccelKind = getAccelTableKind(Args)) {
     Options.LinkOpts.TheAccelTableKind = *AccelKind;
   } else {
     return AccelKind.takeError();
   }
 
   if (opt::Arg *SymbolMap = Args.getLastArg(OPT_symbolmap))
     Options.SymbolMap = SymbolMap->getValue();
int main(int argc, char **argv) {
 
   auto &Options = *OptionsOrErr;
 
   InitializeAllTargetInfos();
   InitializeAllTargetMCs();
   InitializeAllTargets();
   InitializeAllAsmPrinters();
 
+  auto Repro =
+      Reproducer::createReproducer(Options.ReproMode, Options.ReproducerPath);
+  if (!Repro) {
+    WithColor::error() << toString(Repro.takeError());
+    return 1;
+  }
+
+  Options.LinkOpts.VFS = (*Repro)->getVFS();
+
   for (const auto &Arch : Options.Archs)
     if (Arch != "*" && Arch != "all" &&
         !object::MachOObjectFile::isValidArch(Arch)) {
       WithColor::error() << "unsupported cpu architecture: '" << Arch << "'\n";
       return 1;
     }
 
   SymbolMapLoader SymMapLoader(Options.SymbolMap);