[COFF] Add vfsoverlay flag

[abrachet]

Summary

This patch adds a new flag vfsoverlay similar to clang’s ivfsoverlay flag. This is helpful when compiling on case sensitive file systems when cross compiling to Windows. Particularly when compiling third party code containing #pragma comment(“linker”, “/defaultlib:...”) which can’t be easily changed.

[smeenai]

In the LLVM Windows toolchain, we work around this by creating a directory of symlinks: https://github.com/llvm/llvm-project/blob/8527f32f0a16a77edb0b53a5ab8074e518eeff54/llvm/cmake/platforms/WinMsvc.cmake#L133-L157. A VFS overlay would be a nicer solution, and I'm excited to have this supported. I'll leave the review to @rnk and @mstorsjo though.

[rnk]

This seems reasonable to me, but surely this is not the complete list of places where we must use the VFS?

I added folks with cross-compilation & case (in)sensitivity experience to the review. +@ayzhao as well.

[thakis]

Looks alright to me.

I'll mention that you can get this working without an overlay with ciopfs (https://source.chromium.org/chromium/chromium/src/+/main:build/vs_toolchain.py;l=494?q=ciopfs), but this is an ok approach as well.

[abrachet]

Fix test on windows

[abrachet]

In D125800#3528136, @rnk wrote:

This seems reasonable to me, but surely this is not the complete list of places where we must use the VFS?

Not sure where else it would be useful. Specifically I thought this would be useful because /defaultlib can be specified in source code through pragma comment which might be harder to change than build flags. If folks have any other ideas of where it would be useful to look in the vfs I would be happy to add them.

[abrachet]

Ping :)

[phosek]

@rnk @hans @thakis Is it OK with you if we land this?

[hans]

Seems good to me in principle, I just have a few nits.

Have you checked whether this affects the performance when linking something large-ish without the vfs overlay?

lld/COFF/Driver.cpp

453

Is the explicit SmallString constructor call needed, or would just path = getFilename(...) work?

458

Ditto.

1367

nit: I think return nullptr would be more common llvm style.

1371

Shouldn't we give the user an error message?

1374

Ditto if this fails.

lld/test/COFF/vfsoverlay.test

7

Can we have tests cases for when the /vfsoverlay file doesn't exist or is invalid?

[abrachet]

Better error handling

[abrachet]

In D125800#3615601, @hans wrote:

Have you checked whether this affects the performance when linking something large-ish without the vfs overlay?

I did not, but can if you believe it could reasonably affect performance. My thinking is that this only adds an if statement to each findFile invocation, which for COFF seems to be very rare, and even for ld.lld which would end up searching for files much more frequently, the cost that one check per findFile doesn't seem high.

lld/COFF/Driver.cpp

453

That makes the SmallString ctor fail with an assertion "Attempting to reference an element of the vector in an operation that invalidates it"

[hans]

lgtm

lld/COFF/Config.h

 #define LLD_COFF_CONFIG_H
 
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Object/COFF.h"
 #include "llvm/Support/CachePruning.h"
+#include "llvm/Support/VirtualFileSystem.h"
 #include <cstdint>
 #include <map>
 #include <set>
 #include <string>
 
 namespace lld {
 namespace coff {
 
struct Configuration {
   llvm::MapVector<std::pair<const SectionChunk *, const SectionChunk *>,
                   uint64_t>
       callGraphProfile;
   bool callGraphProfileSort = false;
 
   // Used for /print-symbol-order:
   StringRef printSymbolOrder;
 
+  // Used for /vfsoverlay:
+  std::unique_ptr<llvm::vfs::FileSystem> vfs;
+
   uint64_t align = 4096;
   uint64_t imageBase = -1;
   uint64_t fileAlign = 512;
   uint64_t stackReserve = 1024 * 1024;
   uint64_t stackCommit = 4096;
   uint64_t heapReserve = 1024 * 1024;
   uint64_t heapCommit = 4096;
   uint32_t majorImageVersion = 0;

lld/COFF/Driver.cpp

default:
       error(arg->getSpelling() + " is not allowed in .drectve");
     }
   }
 }
 
 // Find file from search paths. You can omit ".obj", this function takes
 // care of that. Note that the returned path is not guaranteed to exist.
 StringRef LinkerDriver::doFindFile(StringRef filename) {
+  auto getFilename = [](StringRef filename) -> StringRef {
+    if (config->vfs)
+      if (auto statOrErr = config->vfs->status(filename))
+        return saver().save(statOrErr->getName());
+    return filename;
+  };
+
   bool hasPathSep = (filename.find_first_of("/\\") != StringRef::npos);
   if (hasPathSep)
-    return filename;
+    return getFilename(filename);
   bool hasExt = filename.contains('.');
   for (StringRef dir : searchPaths) {
     SmallString<128> path = dir;
     sys::path::append(path, filename);
+    path = SmallString<128>{getFilename(path.str())};
     if (sys::fs::exists(path.str()))
       return saver().save(path.str());
     if (!hasExt) {
       path.append(".obj");
+      path = SmallString<128>{getFilename(path.str())};
       if (sys::fs::exists(path.str()))
         return saver().save(path.str());
     }
   }
   return filename;
 }
 
 static Optional<sys::fs::UniqueID> getUniqueID(StringRef path) {
Optional<std::string> getReproduceFile(const opt::InputArgList &args) {
   // This is intentionally not guarded by OPT_lldignoreenv since writing
   // a repro tar file doesn't affect the main output.
   if (auto *path = getenv("LLD_REPRODUCE"))
     return std::string(path);
 
   return None;
 }
 
+static std::unique_ptr<llvm::vfs::FileSystem>
+getVFS(const opt::InputArgList &args) {
+  using namespace llvm::vfs;
+
+  const opt::Arg *arg = args.getLastArg(OPT_vfsoverlay);
+  if (!arg)
+    return nullptr;
+
+  auto bufOrErr = llvm::MemoryBuffer::getFile(arg->getValue());
+  if (!bufOrErr) {
+    checkError(errorCodeToError(bufOrErr.getError()));
+    return nullptr;
+  }
+
+  if (auto ret = vfs::getVFSFromYAML(std::move(*bufOrErr), /*DiagHandler*/ nullptr,
+                             arg->getValue()))
+    return ret;
+
+  error("Invalid vfs overlay");
+  return nullptr;
+}
+
 void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   ScopedTimer rootTimer(ctx.rootTimer);
 
   // Needed for LTO.
   InitializeAllTargetInfos();
   InitializeAllTargets();
   InitializeAllTargetMCs();
   InitializeAllAsmParsers();
void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   if (auto *arg = args.getLastArg(OPT_errorlimit)) {
     int n = 20;
     StringRef s = arg->getValue();
     if (s.getAsInteger(10, n))
       error(arg->getSpelling() + " number expected, but got " + s);
     errorHandler().errorLimit = n;
   }
 
+  config->vfs = getVFS(args);
+
   // Handle /help
   if (args.hasArg(OPT_help)) {
     printHelp(argsArr[0]);
     return;
   }
 
   // /threads: takes a positive integer and provides the default value for
   // /opt:lldltojobs=.

lld/COFF/Options.td

defm call_graph_profile_sort: B<
     "Reorder sections with call graph profile (default)",
     "Do not reorder sections with call graph profile">;
 def print_symbol_order: P<
     "print-symbol-order",
     "Print a symbol order specified by /call-graph-ordering-file and "
     "/call-graph-profile-sort into the specified file">;
 def wrap : P_priv<"wrap">;
 
+def vfsoverlay : P<"vfsoverlay", "Path to a vfsoverlay yaml file to optionally look for /defaultlib's in">;
+
 // Flags for debugging
 def lldmap : F<"lldmap">;
 def lldmap_file : P_priv<"lldmap">;
 def map : F<"map">;
 def map_file : P_priv<"map">;
 def show_timing : F<"time">;
 def summary : F<"summary">;
 

lld/test/COFF/vfsoverlay.test

+# RUN: rm -rf %t
+# RUN: split-file %s %t
+# RUN: cp %p/Inputs/std64.lib %t/std64.lib
+# RUN: sed -e "s|REPLACE|%/t/std64.lib|g" %t/overlay.yaml.in > %t/overlay.yaml
+
+# RUN: lld-link %S/Inputs/hello64.obj /libpath:/noexist /out:%t.exe /entry:main /defaultlib:notstd64 /vfsoverlay:%t/overlay.yaml
+
+# RUN: not lld-link %S/Inputs/hello64.obj /libpath:/noexist /out:%t.exe /entry:main /defaultlib:notstd64 /vfsoverlay:noexist 2>&1 \
+# RUN:   | FileCheck %s
+# CHECK: error: No such file or directory
+
+# RUN: sed -e "s|{|bad|g" %t/overlay.yaml > %t/badoverlay.yaml
+# RUN: not lld-link %S/Inputs/hello64.obj /libpath:/noexist /out:%t.exe /entry:main /defaultlib:notstd64 /vfsoverlay:%t/badoverlay.yaml 2>&1 \
+# RUN:   | FileCheck %s --check-prefix=BAD-OVERLAY
+# BAD-OVERLAY: error: Invalid vfs overlay
+
+#--- overlay.yaml.in
+{
+  'version': 0,
+  'roots' : [
+    {
+      'name': '/noexist',
+      'type': 'directory',
+      'contents': [
+        {
+          'name': 'notstd64.lib',
+          'type': 'file',
+          'external-contents': 'REPLACE'
+        }
+      ]
+    }
+  ]
+}