[LLD] [COFF] Add support for the -wholearchive option

[mstorsjo]

Summary

This fixes PR31824.

[ruiu]

COFF/Options.td

99–100

Since they are lld-specific flags, I'd name -lld-whole-archive and -lld-no-whole-archive.

[pcc]

COFF/Options.td

99–100

I don't think you need to add lld-specific flags to support --whole-archive. Instead, you could implement support for the equivalent link.exe flag, /wholearchive. See also PR31824.

[mstorsjo]

COFF/Options.td

99–100

@pcc - Oh, that probably makes sense - I hadn't seen that option before. Then the MinGW driver just needs to inject -wholearchive: before each library between --whole-archive and --no-whole-archive. Since the driver doesn't do file magic detection, it probably needs to only match this for *.a, *.lib and -lfoo entries, but that's probably fine for most sensible use cases.

[pcc]

COFF/Options.td

99–100

Right. I think you wouldn't even need to do the matching; MSVC link.exe allows /wholearchive to be used with object files, and we can do the same.

[mstorsjo]

Adjusted the COFF driver to implement the link.exe option -wholearchive instead.

[ruiu]

Generally looking good.

Could you split this patch into two, one for COFF and one for MinGW?

COFF/Driver.cpp

135–136

It's not new code, but can you write it in two lines?

Symtab->addFile(make<ArchiveFile>(MBRef));
return;

Because addFile returns void and this function returns void too, it is legal to write return Symtab->addFile(...), but this is still an odd feature of C++, and I believe we generally want to avoid this unless it's generated as instantiated templates.

137

nit: add a blank line befor ethis line.

138

Likewise, separate into two lines.

973

Replacing WholeArchive with Args.hasArg(OPT_wholearchive_flag) might improve readability.

MinGW/Driver.cpp

179–187

I think I'd name this just Prefix as we generally prefer shorter variable names in lld.

187

80 cols (please run clang-format if you have installed it.)

[mstorsjo]

In D37709#868490, @ruiu wrote:

Could you split this patch into two, one for COFF and one for MinGW?

Sure, will repost with it split, and with your suggestions applied.

COFF/Driver.cpp

135–136

Sure, I can fix that at the same time.

[mstorsjo]

Split the patch, moving the MinGW specific parts to a separate patch. Applied Rui's style suggestions.

[ruiu]

LGTM

COFF/Driver.cpp

144

nit: I'd rename Member M.

145

Instead of "*", let's pass something like "<whole-archive>" instead.

[pcc]

COFF/Driver.cpp

229

Are you sure that /wholearchive has no effect on archives specified by linker directives?

973

Wouldn't that create O(N^2) complexity in the size of the argument list though?

[mstorsjo]

COFF/Driver.cpp

229

No, I can try to construct a test to figure out how it's supposed to behave.

[mstorsjo]

COFF/Driver.cpp

229

I checked - link.exe doesn't apply the -wholearchive option to -defaultlib options, neither on the command line nor in directives. So I'll update the handling of the command line version of -defaultlib accordingly as well.

973

So it seems, based on reading ArgList.h. Will push with this changed back to a bool WholeArchiveFlag variable, to avoid the O(N^2) complexity.

COFF/Driver.h

private:
   // This function looks up the symbol table and resolve corresponding
   // entry point name.
   StringRef findDefaultEntry();
   WindowsSubsystem inferSubsystem();
 
   void invokeMSVC(llvm::opt::InputArgList &Args);
 
   MemoryBufferRef takeBuffer(std::unique_ptr<MemoryBuffer> MB);
-  void addBuffer(std::unique_ptr<MemoryBuffer> MB);
+  void addBuffer(std::unique_ptr<MemoryBuffer> MB, bool WholeArchive);
   void addArchiveBuffer(MemoryBufferRef MBRef, StringRef SymName,
                         StringRef ParentName);
 
-  void enqueuePath(StringRef Path);
+  void enqueuePath(StringRef Path, bool WholeArchive);
 
   void enqueueTask(std::function<void()> Task);
   bool run();
 
   std::list<std::function<void()>> TaskQueue;
   std::vector<StringRef> FilePaths;
   std::vector<MemoryBufferRef> Resources;
 };

COFF/Driver.cpp

MemoryBufferRef LinkerDriver::takeBuffer(std::unique_ptr<MemoryBuffer> MB) {
   make<std::unique_ptr<MemoryBuffer>>(std::move(MB)); // take ownership
 
   if (Driver->Tar)
     Driver->Tar->append(relativeToRoot(MBRef.getBufferIdentifier()),
                         MBRef.getBuffer());
   return MBRef;
 }
 
-void LinkerDriver::addBuffer(std::unique_ptr<MemoryBuffer> MB) {
+void LinkerDriver::addBuffer(std::unique_ptr<MemoryBuffer> MB,
+                             bool WholeArchive) {
   MemoryBufferRef MBRef = takeBuffer(std::move(MB));
 
   // File type is detected by contents, not by file extension.
   file_magic Magic = identify_magic(MBRef.getBuffer());
   if (Magic == file_magic::windows_resource) {
     Resources.push_back(MBRef);
     return;
   }
 
   FilePaths.push_back(MBRef.getBufferIdentifier());
-  if (Magic == file_magic::archive)
+  if (Magic == file_magic::archive) {
+    if (WholeArchive) {
+      std::unique_ptr<Archive> File =
+          check(Archive::create(MBRef),
+                MBRef.getBufferIdentifier() + ": failed to parse archive");
+
+      for (MemoryBufferRef Member : getArchiveMembers(File.get()))
+        addArchiveBuffer(Member, "*", MBRef.getBufferIdentifier());
+      return;
+    }
     return Symtab->addFile(make<ArchiveFile>(MBRef));
+  }
   if (Magic == file_magic::bitcode)
     return Symtab->addFile(make<BitcodeFile>(MBRef));
 
   if (Magic == file_magic::coff_cl_gl_object)
     error(MBRef.getBufferIdentifier() + ": is not a native COFF file. "
           "Recompile without /GL");
   else
     Symtab->addFile(make<ObjFile>(MBRef));
 }
 
-void LinkerDriver::enqueuePath(StringRef Path) {
+void LinkerDriver::enqueuePath(StringRef Path, bool WholeArchive) {
   auto Future =
       std::make_shared<std::future<MBErrPair>>(createFutureForFile(Path));
   std::string PathStr = Path;
   enqueueTask([=]() {
     auto MBOrErr = Future->get();
     if (MBOrErr.second)
       error("could not open " + PathStr + ": " + MBOrErr.second.message());
     else
-      Driver->addBuffer(std::move(MBOrErr.first));
+      Driver->addBuffer(std::move(MBOrErr.first), WholeArchive);
   });
 }
 
 void LinkerDriver::addArchiveBuffer(MemoryBufferRef MB, StringRef SymName,
                                     StringRef ParentName) {
   file_magic Magic = identify_magic(MB.getBuffer());
   if (Magic == file_magic::coff_import_library) {
     Symtab->addFile(make<ImportFile>(MB));
for (auto *Arg : Args) {
     case OPT_aligncomm:
       parseAligncomm(Arg->getValue());
       break;
     case OPT_alternatename:
       parseAlternateName(Arg->getValue());
       break;
     case OPT_defaultlib:
       if (Optional<StringRef> Path = findLib(Arg->getValue()))
-        enqueuePath(*Path);
+        enqueuePath(*Path, false);
       break;
     case OPT_export: {
       Export E = parseExport(Arg->getValue());
       E.Directives = true;
       Config->Exports.push_back(E);
       break;
     }
     case OPT_failifmismatch:
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   Config->MapFile = getMapFile(Args);
 
   if (ErrorCount)
     return;
 
   // Create a list of input files. Files can be given as arguments
   // for /defaultlib option.
   std::vector<MemoryBufferRef> MBs;
-  for (auto *Arg : Args.filtered(OPT_INPUT))
+  bool WholeArchive = false;
+  for (auto *Arg :
+       Args.filtered(OPT_INPUT, OPT_whole_archive, OPT_no_whole_archive)) {
+    switch (Arg->getOption().getID()) {
+    case OPT_INPUT:
       if (Optional<StringRef> Path = findFile(Arg->getValue()))
-      enqueuePath(*Path);
+        enqueuePath(*Path, WholeArchive);
+      break;
+    case OPT_whole_archive:
+      WholeArchive = true;
+      break;
+    case OPT_no_whole_archive:
+      WholeArchive = false;
+      break;
+    }
+  }
   for (auto *Arg : Args.filtered(OPT_defaultlib))
     if (Optional<StringRef> Path = findLib(Arg->getValue()))
-      enqueuePath(*Path);
+      enqueuePath(*Path, false);
 
   // Windows specific -- Create a resource file containing a manifest file.
   if (Config->Manifest == Configuration::Embed)
-    addBuffer(createManifestRes());
+    addBuffer(createManifestRes(), false);
 
   // Read all input files given via the command line.
   run();
 
   // We should have inferred a machine type by now from the input files, but if
   // not we assume x64.
   if (Config->Machine == IMAGE_FILE_MACHINE_UNKNOWN) {
     warn("/machine is not specified. x64 is assumed");
     Config->Machine = AMD64;
   }
 
   // Input files can be Windows resource files (.res files). We use
   // WindowsResource to convert resource files to a regular COFF file,
   // then link the resulting file normally.
   if (!Resources.empty())
-    addBuffer(convertResToCOFF(Resources));
+    addBuffer(convertResToCOFF(Resources), false);
 
   if (Tar)
     Tar->append("response.txt",
                 createResponseFile(Args, FilePaths,
                                    ArrayRef<StringRef>(SearchPaths).slice(1)));
 
   // Handle /largeaddressaware
   if (Config->is64() || Args.hasArg(OPT_largeaddressaware))

COFF/Options.td

defm dynamicbase : B<"dynamicbase",
                      "Disable address space layout randomization">;
 defm fixed    : B<"fixed", "Enable base relocations">;
 defm highentropyva : B<"highentropyva", "Set HIGH_ENTROPY_VA bit">;
 defm largeaddressaware : B<"largeaddressaware", "Disable large addresses">;
 defm nxcompat : B<"nxcompat", "Disable data execution provention">;
 defm safeseh : B<"safeseh", "Produce an image with Safe Exception Handler">;
 defm tsaware  : B<"tsaware", "Create non-Terminal Server aware executable">;
 
+def whole_archive : F<"whole-archive">;
+def no_whole_archive : F<"no-whole-archive">;
+
 def help : F<"help">;
 def help_q : Flag<["/?", "-?"], "">, Alias<help>;
 
 // LLD extensions
 def nopdb : F<"nopdb">, HelpText<"Disable PDB generation for DWARF users">;
 def nosymtab : F<"nosymtab">;
 def msvclto : F<"msvclto">;
 def rsp_quoting : Joined<["--"], "rsp-quoting=">,

MinGW/Driver.cpp

if (Args.getLastArgValue(OPT_m) == "i386pe")
     Add("-alternatename:__image_base__=___ImageBase");
   else
     Add("-alternatename:__image_base__=__ImageBase");
 
   std::vector<StringRef> SearchPaths;
   for (auto *A : Args.filtered(OPT_L))
     SearchPaths.push_back(A->getValue());
 
-  for (auto *A : Args.filtered(OPT_INPUT, OPT_l)) {
-    if (A->getOption().getUnaliasedOption().getID() == OPT_INPUT)
+  for (auto *A : Args.filtered(OPT_INPUT, OPT_l, OPT_whole_archive,
+                               OPT_no_whole_archive)) {
+    switch (A->getOption().getID()) {
+    case OPT_INPUT:
       Add(A->getValue());
-    else
+      break;
+    case OPT_l:
       Add(searchLibrary(A->getValue(), SearchPaths, Args.hasArg(OPT_Bstatic)));
+      break;
+    case OPT_whole_archive:
+      Add("-whole-archive");
+      break;
+    case OPT_no_whole_archive:
+      Add("-no-whole-archive");
+      break;
+    }
   }
 
   if (Args.hasArg(OPT_verbose))
     Add("-verbose");
 
   if (Args.hasArg(OPT_verbose) || Args.hasArg(OPT__HASH_HASH_HASH))
     outs() << llvm::join(LinkArgs, " ") << "\n";
 

MinGW/Options.td

 include "llvm/Option/OptParser.td"
 
 class F<string name>: Flag<["--", "-"], name>;
 class J<string name>: Joined<["--", "-"], name>;
 class S<string name>: Separate<["--", "-"], name>;
 
 def L: JoinedOrSeparate<["-"], "L">, MetaVarName<"<dir>">,
   HelpText<"Add a directory to the library search path">;
 def entry: S<"entry">, MetaVarName<"<entry>">,
   HelpText<"Name of entry point symbol">;
 def m: JoinedOrSeparate<["-"], "m">, HelpText<"Set target emulation">;
+def no_whole_archive: F<"no-whole-archive">,
+    HelpText<"No longer include all object files for following archives">;
 def o: JoinedOrSeparate<["-"], "o">, MetaVarName<"<path>">,
   HelpText<"Path to file to write output">;
 def l: JoinedOrSeparate<["-"], "l">, MetaVarName<"<libName>">,
   HelpText<"Root name of library to use">;
 def shared: F<"shared">, HelpText<"Build a shared object">;
 def subs: Separate<["--"], "subsystem">, HelpText<"Specify subsystem">;
 def stack: Separate<["--"], "stack">;
 def outlib: Separate<["--"], "out-implib">, HelpText<"Import library name">;
+def whole_archive: F<"whole-archive">,
+    HelpText<"Include all object files for following archives">;
 def verbose: F<"verbose">, HelpText<"Verbose mode">;
 
 // LLD specific options
 def _HASH_HASH_HASH : Flag<["-"], "###">,
     HelpText<"Print (but do not run) the commands to run for this compilation">;
 
 // Currently stubs to avoid errors
 def Bdynamic: F<"Bdynamic">, HelpText<"Link against shared libraries">;

test/MinGW/Inputs/export.yaml

+--- !COFF
+header:
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: []
+sections:
+  - Name:            .text
+    Characteristics: [ IMAGE_SCN_CNT_CODE, IMAGE_SCN_MEM_EXECUTE, IMAGE_SCN_MEM_READ ]
+    Alignment:       4
+    SectionData:     B800000000506800000000680000000050E80000000050E800000000
+  - Name:            .drectve
+    Characteristics: [ IMAGE_SCN_LNK_INFO, IMAGE_SCN_LNK_REMOVE ]
+    Alignment:       1
+    SectionData:     2f6578706f72743a6578706f7274666e3300  # /export:exportfn3
+symbols:
+  - Name:            .text
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition:
+      Length:          28
+      NumberOfRelocations: 4
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            _DllMainCRTStartup
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            exportfn1
+    Value:           8
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            exportfn2
+    Value:           16
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            exportfn3
+    Value:           16
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            '?mangled@@YAHXZ'
+    Value:           16
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+...

test/MinGW/whole-archive.test

+# REQEUIRES: x86
+
+# RUN: yaml2obj < %p/Inputs/export.yaml > %t.archive.obj
+# RUN: llvm-ar rcs %t.archive.lib %t.archive.obj
+# RUN: llvm-mc -triple=x86_64-windows-msvc %s -filetype=obj -o %t.main.obj
+# RUN: ld.lld -m i386pep --shared --entry main %t.main.obj --whole-archive %t.archive.lib --no-whole-archive -o %t.dll --out-implib %t.lib
+# RUN: llvm-readobj %t.lib | FileCheck %s -check-prefix CHECK-IMPLIB
+
+# CHECK-IMPLIB: Symbol: __imp_exportfn3
+# CHECK-IMPLIB: Symbol: exportfn3
+
+.global main
+.text
+main:
+  ret