[LLD] [COFF] Add support for the -wholearchive option

[mstorsjo]

Summary

This fixes PR31824.

[ruiu]

COFF/Options.td

101–102

Since they are lld-specific flags, I'd name -lld-whole-archive and -lld-no-whole-archive.

[pcc]

COFF/Options.td

101–102

I don't think you need to add lld-specific flags to support --whole-archive. Instead, you could implement support for the equivalent link.exe flag, /wholearchive. See also PR31824.

[mstorsjo]

COFF/Options.td

101–102

@pcc - Oh, that probably makes sense - I hadn't seen that option before. Then the MinGW driver just needs to inject -wholearchive: before each library between --whole-archive and --no-whole-archive. Since the driver doesn't do file magic detection, it probably needs to only match this for *.a, *.lib and -lfoo entries, but that's probably fine for most sensible use cases.

[pcc]

COFF/Options.td

101–102

Right. I think you wouldn't even need to do the matching; MSVC link.exe allows /wholearchive to be used with object files, and we can do the same.

[mstorsjo]

Adjusted the COFF driver to implement the link.exe option -wholearchive instead.

[ruiu]

Generally looking good.

Could you split this patch into two, one for COFF and one for MinGW?

COFF/Driver.cpp

135

It's not new code, but can you write it in two lines?

Symtab->addFile(make<ArchiveFile>(MBRef));
return;

Because addFile returns void and this function returns void too, it is legal to write return Symtab->addFile(...), but this is still an odd feature of C++, and I believe we generally want to avoid this unless it's generated as instantiated templates.

137

nit: add a blank line befor ethis line.

138

Likewise, separate into two lines.

975

Replacing WholeArchive with Args.hasArg(OPT_wholearchive_flag) might improve readability.

MinGW/Driver.cpp

175

I think I'd name this just Prefix as we generally prefer shorter variable names in lld.

183

80 cols (please run clang-format if you have installed it.)

[mstorsjo]

In D37709#868490, @ruiu wrote:

Could you split this patch into two, one for COFF and one for MinGW?

Sure, will repost with it split, and with your suggestions applied.

COFF/Driver.cpp

135

Sure, I can fix that at the same time.

[mstorsjo]

Split the patch, moving the MinGW specific parts to a separate patch. Applied Rui's style suggestions.

[ruiu]

LGTM

COFF/Driver.cpp

144

nit: I'd rename Member M.

145

Instead of "*", let's pass something like "<whole-archive>" instead.

[pcc]

COFF/Driver.cpp

229

Are you sure that /wholearchive has no effect on archives specified by linker directives?

975

Wouldn't that create O(N^2) complexity in the size of the argument list though?

[mstorsjo]

COFF/Driver.cpp

229

No, I can try to construct a test to figure out how it's supposed to behave.

[mstorsjo]

COFF/Driver.cpp

229

I checked - link.exe doesn't apply the -wholearchive option to -defaultlib options, neither on the command line nor in directives. So I'll update the handling of the command line version of -defaultlib accordingly as well.

975

So it seems, based on reading ArgList.h. Will push with this changed back to a bool WholeArchiveFlag variable, to avoid the O(N^2) complexity.

COFF/Driver.h

private:
   // This function looks up the symbol table and resolve corresponding
   // entry point name.
   StringRef findDefaultEntry();
   WindowsSubsystem inferSubsystem();
 
   void invokeMSVC(llvm::opt::InputArgList &Args);
 
   MemoryBufferRef takeBuffer(std::unique_ptr<MemoryBuffer> MB);
-  void addBuffer(std::unique_ptr<MemoryBuffer> MB);
+  void addBuffer(std::unique_ptr<MemoryBuffer> MB, bool WholeArchive);
   void addArchiveBuffer(MemoryBufferRef MBRef, StringRef SymName,
                         StringRef ParentName);
 
-  void enqueuePath(StringRef Path);
+  void enqueuePath(StringRef Path, bool WholeArchive);
 
   void enqueueTask(std::function<void()> Task);
   bool run();
 
   std::list<std::function<void()>> TaskQueue;
   std::vector<StringRef> FilePaths;
   std::vector<MemoryBufferRef> Resources;
 };

COFF/Driver.cpp

MemoryBufferRef LinkerDriver::takeBuffer(std::unique_ptr<MemoryBuffer> MB) {
   make<std::unique_ptr<MemoryBuffer>>(std::move(MB)); // take ownership
 
   if (Driver->Tar)
     Driver->Tar->append(relativeToRoot(MBRef.getBufferIdentifier()),
                         MBRef.getBuffer());
   return MBRef;
 }
 
-void LinkerDriver::addBuffer(std::unique_ptr<MemoryBuffer> MB) {
+void LinkerDriver::addBuffer(std::unique_ptr<MemoryBuffer> MB,
+                             bool WholeArchive) {
   MemoryBufferRef MBRef = takeBuffer(std::move(MB));
 
   // File type is detected by contents, not by file extension.
   file_magic Magic = identify_magic(MBRef.getBuffer());
   if (Magic == file_magic::windows_resource) {
     Resources.push_back(MBRef);
     return;
   }
 
   FilePaths.push_back(MBRef.getBufferIdentifier());
-  if (Magic == file_magic::archive)
+  if (Magic == file_magic::archive) {
+    if (WholeArchive) {
+      std::unique_ptr<Archive> File =
+          check(Archive::create(MBRef),
+                MBRef.getBufferIdentifier() + ": failed to parse archive");
+
+      for (MemoryBufferRef Member : getArchiveMembers(File.get()))
+        addArchiveBuffer(Member, "*", MBRef.getBufferIdentifier());
+      return;
+    }
     return Symtab->addFile(make<ArchiveFile>(MBRef));
+  }
   if (Magic == file_magic::bitcode)
     return Symtab->addFile(make<BitcodeFile>(MBRef));
 
   if (Magic == file_magic::coff_cl_gl_object)
     error(MBRef.getBufferIdentifier() + ": is not a native COFF file. "
           "Recompile without /GL");
   else
     Symtab->addFile(make<ObjFile>(MBRef));
 }
 
-void LinkerDriver::enqueuePath(StringRef Path) {
+void LinkerDriver::enqueuePath(StringRef Path, bool WholeArchive) {
   auto Future =
       std::make_shared<std::future<MBErrPair>>(createFutureForFile(Path));
   std::string PathStr = Path;
   enqueueTask([=]() {
     auto MBOrErr = Future->get();
     if (MBOrErr.second)
       error("could not open " + PathStr + ": " + MBOrErr.second.message());
     else
-      Driver->addBuffer(std::move(MBOrErr.first));
+      Driver->addBuffer(std::move(MBOrErr.first), WholeArchive);
   });
 }
 
 void LinkerDriver::addArchiveBuffer(MemoryBufferRef MB, StringRef SymName,
                                     StringRef ParentName) {
   file_magic Magic = identify_magic(MB.getBuffer());
   if (Magic == file_magic::coff_import_library) {
     Symtab->addFile(make<ImportFile>(MB));
for (auto *Arg : Args) {
     case OPT_aligncomm:
       parseAligncomm(Arg->getValue());
       break;
     case OPT_alternatename:
       parseAlternateName(Arg->getValue());
       break;
     case OPT_defaultlib:
       if (Optional<StringRef> Path = findLib(Arg->getValue()))
-        enqueuePath(*Path);
+        enqueuePath(*Path, false);
       break;
     case OPT_export: {
       Export E = parseExport(Arg->getValue());
       E.Directives = true;
       Config->Exports.push_back(E);
       break;
     }
     case OPT_failifmismatch:
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
 
   Config->MapFile = getMapFile(Args);
 
   if (ErrorCount)
     return;
 
   // Create a list of input files. Files can be given as arguments
   // for /defaultlib option.
+  bool WholeArchive = Args.hasArg(OPT_wholearchive_flag);
   std::vector<MemoryBufferRef> MBs;
-  for (auto *Arg : Args.filtered(OPT_INPUT))
+  for (auto *Arg : Args.filtered(OPT_INPUT, OPT_wholearchive_file)) {
+    switch (Arg->getOption().getID()) {
+    case OPT_INPUT:
+      if (Optional<StringRef> Path = findFile(Arg->getValue()))
+        enqueuePath(*Path, WholeArchive);
+      break;
+    case OPT_wholearchive_file:
       if (Optional<StringRef> Path = findFile(Arg->getValue()))
-      enqueuePath(*Path);
+        enqueuePath(*Path, true);
+      break;
+    }
+  }
   for (auto *Arg : Args.filtered(OPT_defaultlib))
     if (Optional<StringRef> Path = findLib(Arg->getValue()))
-      enqueuePath(*Path);
+      enqueuePath(*Path, WholeArchive);
 
   // Windows specific -- Create a resource file containing a manifest file.
   if (Config->Manifest == Configuration::Embed)
-    addBuffer(createManifestRes());
+    addBuffer(createManifestRes(), false);
 
   // Read all input files given via the command line.
   run();
 
   // We should have inferred a machine type by now from the input files, but if
   // not we assume x64.
   if (Config->Machine == IMAGE_FILE_MACHINE_UNKNOWN) {
     warn("/machine is not specified. x64 is assumed");
     Config->Machine = AMD64;
   }
 
   // Input files can be Windows resource files (.res files). We use
   // WindowsResource to convert resource files to a regular COFF file,
   // then link the resulting file normally.
   if (!Resources.empty())
-    addBuffer(convertResToCOFF(Resources));
+    addBuffer(convertResToCOFF(Resources), false);
 
   if (Tar)
     Tar->append("response.txt",
                 createResponseFile(Args, FilePaths,
                                    ArrayRef<StringRef>(SearchPaths).slice(1)));
 
   // Handle /largeaddressaware
   if (Config->is64() || Args.hasArg(OPT_largeaddressaware))

COFF/Options.td

 def opt     : P<"opt", "Control optimizations">;
 def out     : P<"out", "Path to file to write output">;
 def pdb : P<"pdb", "PDB file path">;
 def section : P<"section", "Specify section attributes">;
 def stack   : P<"stack", "Size of the stack">;
 def stub    : P<"stub", "Specify DOS stub file">;
 def subsystem : P<"subsystem", "Specify subsystem">;
 def version : P<"version", "Specify a version number in the PE header">;
+def wholearchive_file : P<"wholearchive", "Include all object files from this archive">;
 
 def disallowlib : Joined<["/", "-", "-?"], "disallowlib:">, Alias<nodefaultlib>;
 
 def manifest : F<"manifest">;
 def manifest_colon : P<"manifest", "Create manifest file">;
 def manifestuac : P<"manifestuac", "User access control">;
 def manifestfile : P<"manifestfile", "Manifest file path">;
 def manifestdependency : P<"manifestdependency",

 def dll : F<"dll">, HelpText<"Create a DLL">;
 def driver : P<"driver", "Generate a Windows NT Kernel Mode Driver">;
 def nodefaultlib_all : F<"nodefaultlib">;
 def noentry : F<"noentry">;
 def profile : F<"profile">;
 def swaprun_cd : F<"swaprun:cd">;
 def swaprun_net : F<"swaprun:net">;
 def verbose : F<"verbose">;
+def wholearchive_flag : F<"wholearchive">;
 
 def force : F<"force">,
     HelpText<"Allow undefined symbols when creating executables">;
 def force_unresolved : F<"force:unresolved">;
 
 defm allowbind: B<"allowbind", "Disable DLL binding">;
 defm allowisolation : B<"allowisolation", "Set NO_ISOLATION bit">;
 defm appcontainer : B<"appcontainer",
                       "Image can only be run in an app container">;
 defm dynamicbase : B<"dynamicbase",
                      "Disable address space layout randomization">;
 defm fixed    : B<"fixed", "Enable base relocations">;
 defm highentropyva : B<"highentropyva", "Set HIGH_ENTROPY_VA bit">;
 defm largeaddressaware : B<"largeaddressaware", "Disable large addresses">;
 defm nxcompat : B<"nxcompat", "Disable data execution provention">;
 defm safeseh : B<"safeseh", "Produce an image with Safe Exception Handler">;
 defm tsaware  : B<"tsaware", "Create non-Terminal Server aware executable">;
 
 def help : F<"help">;
 def help_q : Flag<["/?", "-?"], "">, Alias<help>;
 
 // LLD extensions
 def nopdb : F<"nopdb">, HelpText<"Disable PDB generation for DWARF users">;
 def nosymtab : F<"nosymtab">;
 def msvclto : F<"msvclto">;
 def rsp_quoting : Joined<["--"], "rsp-quoting=">,
   HelpText<"Quoting style for response files, 'windows' (default) or 'posix'">;
 

MinGW/Driver.cpp

if (Args.getLastArgValue(OPT_m) == "i386pe")
     Add("-alternatename:__image_base__=___ImageBase");
   else
     Add("-alternatename:__image_base__=__ImageBase");
 
   std::vector<StringRef> SearchPaths;
   for (auto *A : Args.filtered(OPT_L))
     SearchPaths.push_back(A->getValue());
 
-  for (auto *A : Args.filtered(OPT_INPUT, OPT_l)) {
-    if (A->getOption().getUnaliasedOption().getID() == OPT_INPUT)
-      Add(A->getValue());
-    else
-      Add(searchLibrary(A->getValue(), SearchPaths, Args.hasArg(OPT_Bstatic)));
+  StringRef WholeArchivePrefix = "";
+  for (auto *A : Args.filtered(OPT_INPUT, OPT_l, OPT_whole_archive,
+                               OPT_no_whole_archive)) {
+    switch (A->getOption().getID()) {
+    case OPT_INPUT:
+      Add(WholeArchivePrefix + StringRef(A->getValue()));
+      break;
+    case OPT_l:
+      Add(WholeArchivePrefix + searchLibrary(A->getValue(), SearchPaths, Args.hasArg(OPT_Bstatic)));
+      break;
+    case OPT_whole_archive:
+      WholeArchivePrefix = "-wholearchive:";
+      break;
+    case OPT_no_whole_archive:
+      WholeArchivePrefix = "";
+      break;
+    }
   }
 
   if (Args.hasArg(OPT_verbose))
     Add("-verbose");
 
   if (Args.hasArg(OPT_verbose) || Args.hasArg(OPT__HASH_HASH_HASH))
     outs() << llvm::join(LinkArgs, " ") << "\n";
 

MinGW/Options.td

 include "llvm/Option/OptParser.td"
 
 class F<string name>: Flag<["--", "-"], name>;
 class J<string name>: Joined<["--", "-"], name>;
 class S<string name>: Separate<["--", "-"], name>;
 
 def L: JoinedOrSeparate<["-"], "L">, MetaVarName<"<dir>">,
   HelpText<"Add a directory to the library search path">;
 def entry: S<"entry">, MetaVarName<"<entry>">,
   HelpText<"Name of entry point symbol">;
 def l: JoinedOrSeparate<["-"], "l">, MetaVarName<"<libName>">,
   HelpText<"Root name of library to use">;
 def m: JoinedOrSeparate<["-"], "m">, HelpText<"Set target emulation">;
 def mllvm: S<"mllvm">;
+def no_whole_archive: F<"no-whole-archive">,
+    HelpText<"No longer include all object files for following archives">;
 def o: JoinedOrSeparate<["-"], "o">, MetaVarName<"<path>">,
   HelpText<"Path to file to write output">;
 def out_implib: Separate<["--"], "out-implib">, HelpText<"Import library name">;
 def shared: F<"shared">, HelpText<"Build a shared object">;
 def subs: Separate<["--"], "subsystem">, HelpText<"Specify subsystem">;
 def stack: Separate<["--"], "stack">;
+def whole_archive: F<"whole-archive">,
+    HelpText<"Include all object files for following archives">;
 def verbose: F<"verbose">, HelpText<"Verbose mode">;
 
 // LLD specific options
 def _HASH_HASH_HASH : Flag<["-"], "###">,
     HelpText<"Print (but do not run) the commands to run for this compilation">;
 
 // Currently stubs to avoid errors
 def Bdynamic: F<"Bdynamic">, HelpText<"Link against shared libraries">;

test/COFF/wholearchive.s

+# REQEUIRES: x86
+
+# RUN: yaml2obj < %p/Inputs/export.yaml > %t.archive.obj
+# RUN: llvm-ar rcs %t.archive.lib %t.archive.obj
+# RUN: llvm-mc -triple=x86_64-windows-msvc %s -filetype=obj -o %t.main.obj
+
+# RUN: lld-link -dll -out:%t.dll -entry:main %t.main.obj -wholearchive:%t.archive.lib -implib:%t.lib
+# RUN: llvm-readobj %t.lib | FileCheck %s -check-prefix CHECK-IMPLIB
+
+# RUN: lld-link -dll -out:%t.dll -entry:main %t.main.obj -wholearchive %t.archive.lib -implib:%t.lib
+# RUN: llvm-readobj %t.lib | FileCheck %s -check-prefix CHECK-IMPLIB
+
+# CHECK-IMPLIB: Symbol: __imp_exportfn3
+# CHECK-IMPLIB: Symbol: exportfn3
+
+.global main
+.text
+main:
+  ret

test/MinGW/driver.test

 RUN: ld.lld -### foo.o -m i386pep --stack 4194304,8192 | FileCheck -check-prefix=STACK %s
 STACK: -stack:4194304,8192
 
 RUN: ld.lld -### foo.o -m i386pep --verbose | FileCheck -check-prefix=VERBOSE %s
 VERBOSE: -verbose
 
 RUN: ld.lld -### -shared -m i386pe -e _DllMainCRTStartup@12 foo.o | FileCheck -check-prefix I386-ENTRY %s
 I386-ENTRY: -entry:DllMainCRTStartup@12
+
+RUN: ld.lld -### -shared -m i386pep foo.o --whole-archive bar.a --no-whole-archive baz.a | FileCheck -check-prefix WHOLE-ARCHIVE %s
+WHOLE-ARCHIVE: foo.o -wholearchive:bar.a baz.a