[lld-macho] Implement -object_path_lto

[int3]

Summary

This makes it possible for STABS entries to reference the debug info
contained in the LTO-compiled output.

I'm not sure how to test the file mtime within llvm-lit -- GNU and BSD
stat take different command-line arguments. I've omitted the check for
now.

[int3]

error handling

[int3]

@clayborg was hoping to get your opinion on my approach to modtime in this diff

[clayborg]

In D92537#2438256, @int3 wrote:

@clayborg was hoping to get your opinion on my approach to modtime in this diff

What is the main reason for this? Is this so you can run the linker on the same set of input .o files and end up with a binary whose MD5 checksum doesn't change? The danger of this approach is that we will end up allowing any lto.o file to be loaded and linked with dsymutil or lldb. If the lto.o file is modified, we will have no way of knowing if we are loading the right file. Can we add an option for this if this is solely needed for testing? I think people in the real world are going to want the modification file set correctly. If we fix https://reviews.llvm.org/D92736 to generate the UUID with the MD5 of all of the sections that don't change with where or when the binary was built, you can end up using the UUID for verification that things are the same or correct.

lld/MachO/LTO.cpp

61

Remove this.

If you just emit the N_OSO STABS entry for the lto.o file with zero a the value (mod time) then LLDB will not try to match the mod time of the binary and won't reject it if it doesn't match. Same goes for dsymutil.

110

Remove this (see above inline comment) and maybe special case it when emitting the N_OSO?

[int3]

What is the main reason for this? Is this so you can run the linker on the same set of input .o files and end up with a binary whose MD5 checksum doesn't change?

Yeah pretty much... but you raise good points. I'll change it and have the test just skip checking for the exact timestamp

[int3]

emit real modTime in STABS

[thakis]

FWIW we're very interested in having deterministic build outputs (in the senses of https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html), and putting file mtimes in the linker output defeats that. Could we put content hashes in the mtime instead? (We could make the compiler produce a content hash of its .o output at compile time so that we could just read the content off out the .o file instead of computing it if computing it is a perf issue.)

[int3]

That sounds like it'd need a corresponding change in LLDB. (And possibly gdb too?)

Seems out of scope for now...

[int3]

If I understood @clayborg's earlier comments, LLDB will match nonzero STABS values with the mtime, so a content hash would result in a mismatch. But a zero STABS value will match any file (at the given path), so I guess we could have a flag to enable that behavior in order to get reproducible builds?

[clayborg]

In D92537#2443306, @thakis wrote:

FWIW we're very interested in having deterministic build outputs (in the senses of https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html), and putting file mtimes in the linker output defeats that. Could we put content hashes in the mtime instead? (We could make the compiler produce a content hash of its .o output at compile time so that we could just read the content off out the .o file instead of computing it if computing it is a perf issue.)

We can't change the value of the N_OSO symbol without a change to LLDB and GDB. The debuggers really want to have this timestamp because someone can recompile a .o file and then try to debug a program that refers to this .o file which now has completely different contents. If we use a hash, then each time you debug, the debugger must fully hash each .o file to figure out if the hash matches which we really do not want to do as that will force it to touch every page in each .o file each time you debug and that will kill performance.

If we can come up with a solution that doesn't adversely affect performance, the N_OSO currently must have the "n_desc" field of each symbol set to 1. We could set this to 2 and know that the value in the symbol would be different. But that again requires an update to the debugger and anyone else that tries to load a binary with this change into an older debugger will get no debug info with no explanation of what is going wrong.

[clayborg]

In D92537#2444027, @int3 wrote:

If I understood @clayborg's earlier comments, LLDB will match nonzero STABS values with the mtime, so a content hash would result in a mismatch. But a zero STABS value will match any file (at the given path), so I guess we could have a flag to enable that behavior in order to get reproducible builds?

zeroing one the value of the N_OSO can cause the debugger to just load any version of the .o file even if it is out of date.

BTW: dsymutil does support loading the debug map from a yaml file (or maybe JSON?). They use this for testing. So there is nothing that states that the debug map must be in the binary itself. We could teach dsymutil to check for N_OSO entries in the binary it is trying to link and if there are none, check for a JSON or YAML file right next to the binary and load the debug map from there automatically. Another option is to add a new STAB entry that specifies the path to the YAML file, and this could be made relative for repro builds.

[thakis]

Ok, then let's not worry about that for now, thanks.

lld/MachO/Config.h

struct Configuration {
   bool staticLink = false;
   bool isPic = false;
   bool headerPadMaxInstallNames = false;
   bool searchDylibsFirst = false;
   bool saveTemps = false;
   uint32_t headerPad;
   llvm::StringRef installName;
   llvm::StringRef outputFile;
+  llvm::StringRef ltoObjPath;
   bool demangle = false;
   llvm::MachO::Architecture arch;
   PlatformInfo platform;
   llvm::MachO::HeaderFileType outputType;
   std::vector<llvm::StringRef> systemLibraryRoots;
   std::vector<llvm::StringRef> librarySearchPaths;
   std::vector<llvm::StringRef> frameworkSearchPaths;
   std::vector<llvm::StringRef> runtimePaths;

lld/MachO/Driver.cpp

bool macho::link(llvm::ArrayRef<const char *> argsArr, bool canExitEarly,
 
   config->entry = symtab->addUndefined(args.getLastArgValue(OPT_e, "_main"));
   config->outputFile = args.getLastArgValue(OPT_o, "a.out");
   config->installName =
       args.getLastArgValue(OPT_install_name, config->outputFile);
   config->headerPad = args::getHex(args, OPT_headerpad, /*Default=*/32);
   config->headerPadMaxInstallNames =
       args.hasArg(OPT_headerpad_max_install_names);
+  config->ltoObjPath = args.getLastArgValue(OPT_object_path_lto);
   config->outputType = getOutputType(args);
   config->runtimePaths = args::getStrings(args, OPT_rpath);
   config->allLoad = args.hasArg(OPT_all_load);
   config->forceLoadObjC = args.hasArg(OPT_ObjC);
   config->demangle = args.hasArg(OPT_demangle);
 
   if (const opt::Arg *arg = args.getLastArg(OPT_static, OPT_dynamic))
     config->staticLink = (arg->getOption().getID() == OPT_static);

lld/MachO/LTO.cpp

 //===- LTO.cpp ------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "LTO.h"
 #include "Config.h"
 #include "InputFiles.h"
 
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/Strings.h"
 #include "lld/Common/TargetOptionsCommandFlags.h"
 #include "llvm/LTO/LTO.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace lld;
 using namespace lld::macho;
 using namespace llvm;
+using namespace llvm::sys;
 
 static lto::Config createConfig() {
   lto::Config c;
   c.Options = initTargetOptionsFromCodeGenFlags();
   return c;
 }
 
 BitcodeCompiler::BitcodeCompiler() {
for (const lto::InputFile::Symbol &objSym : objSyms) {
     r.Prevailing = !objSym.isUndefined();
 
     // TODO: set the other resolution configs properly
     r.VisibleToRegularObj = true;
   }
   checkError(ltoObj->add(std::move(f.obj), resols));
 }
 
+static void setModTime(StringRef path, std::time_t t) {
+  int fd;
+  std::error_code ec = fs::openFileForRead(path, fd);
+  if (ec)
+    fatal("failed to open " + path);
+  ec = fs::setLastAccessAndModificationTime(fd, toTimePoint(t));
+  if (ec)
+    fatal("failed to set modification time for " + path);
+  ec = fs::closeFile(fd);
+  if (ec)
+    fatal("failed to close " + path);
+}
+
 // Merge all the bitcode files we have seen, codegen the result
 // and return the resulting ObjectFile(s).
 std::vector<ObjFile *> BitcodeCompiler::compile() {
   unsigned maxTasks = ltoObj->getMaxTasks();
   buf.resize(maxTasks);
 
   checkError(ltoObj->run([&](size_t task) {
     return std::make_unique<lto::NativeObjectStream>(
         std::make_unique<raw_svector_ostream>(buf[task]));
   }));
 
   if (config->saveTemps) {
     if (!buf[0].empty())
       saveBuffer(buf[0], config->outputFile + ".lto.o");
     for (unsigned i = 1; i != maxTasks; ++i)
       saveBuffer(buf[i], config->outputFile + Twine(i) + ".lto.o");
   }
 
-  // TODO: set modTime properly
+  if (!config->ltoObjPath.empty())
+    fs::create_directories(config->ltoObjPath);
+
   std::vector<ObjFile *> ret;
-  for (unsigned i = 0; i != maxTasks; ++i)
-    if (!buf[i].empty())
-      ret.push_back(
-          make<ObjFile>(MemoryBufferRef(buf[i], "lto.tmp"), /*modTime=*/0, ""));
+  for (unsigned i = 0; i != maxTasks; ++i) {
+    if (buf[i].empty()) {
+      continue;
+    }
+    SmallString<261> filePath("/tmp/lto.tmp");
+    if (!config->ltoObjPath.empty()) {
+      filePath = config->ltoObjPath;
+      path::append(filePath, Twine(i) + "." +
+                                 getArchitectureName(config->arch) + ".lto.o");
+      saveBuffer(buf[i], filePath);
+
+      // Instead of retrieving the real modtime, we instead zero it out --
+      // this ensures that the output binary is reproducible, instead of
+      // containing a STABS timestamp entry that keeps changing.
+      setModTime(filePath, 0);
+    }
+        make<ObjFile>(MemoryBufferRef(buf[i], saver.save(filePath.str())),
+                      /*modTime=*/0, ""));
+  }
 
   return ret;
 }

lld/MachO/Options.td

def no_function_starts : Flag<["-"], "no_function_starts">,
      Group<grp_rare>;
 def no_objc_category_merging : Flag<["-"], "no_objc_category_merging">,
      HelpText<"Do not merge Objective-C categories into their classes">,
      Flags<[HelpHidden]>,
      Group<grp_rare>;
 def object_path_lto : Separate<["-"], "object_path_lto">,
      MetaVarName<"<path>">,
      HelpText<"Retain any temporary mach-o file in <path> that would otherwise be deleted during LTO">,
-     Flags<[HelpHidden]>,
      Group<grp_rare>;
 def lto_library : Separate<["-"], "lto_library">,
      MetaVarName<"<path>">,
      HelpText<"Override the default ../lib/libLTO.dylib as <path>">,
      Flags<[HelpHidden]>,
      Group<grp_rare>;
 def cache_path_lto : Separate<["-"], "cache_path_lto">,
      MetaVarName<"<path>">,

lld/test/MachO/lto-object-path.ll

+; REQUIRES: x86
+
+; RUN: rm -rf %t; mkdir %t
+; RUN: llvm-as %s -o %t/test.o
+
+; RUN: %lld %t/test.o -o %t/test
+; RUN: llvm-nm -pa %t/test | FileCheck %s -DOSO=/tmp/lto.tmp
+
+; RUN: %lld %t/test.o -o %t/test -object_path_lto %t/lto-temps
+; RUN: llvm-nm -pa %t/test | FileCheck %s -DOSO=%t/lto-temps/0.x86_64.lto.o
+
+; CHECK:      0000000000000000 - 00 0000    SO /tmp/test.cpp
+; CHECK-NEXT: 0000000000000000 - 03 0001   OSO [[OSO]]
+; CHECK-NEXT: {{[0-9a-f]+}}    - 01 0000   FUN _main
+; CHECK-NEXT: 0000000000000001 - 00 0000   FUN
+; CHECK-NEXT: 0000000000000000 - 01 0000    SO
+; CHECK-NEXT: {{[0-9a-f]+}}    T _main
+
+target triple = "x86_64-apple-macosx10.15.0"
+target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+
+define void @main() #0 !dbg !4 {
+  ret void
+}
+
+!llvm.module.flags = !{ !0, !1 }
+!llvm.dbg.cu = !{!2}
+
+!0 = !{i32 7, !"Dwarf Version", i32 4}
+!1 = !{i32 2, !"Debug Info Version", i32 3}
+!2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, emissionKind: FullDebug)
+!3 = !DIFile(filename: "test.cpp", directory: "/tmp")
+!4 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, unit: !2)
+!5 = !DISubroutineType(types: !6)
+!6 = !{}