lld-link: Take /SUBSYSTEM into account for automatic /ENTRY detection.

[thakis]

Summary

If /subsystem:windows is passed, link.exe only looks for WinMain and wWinMain, and if /subsystem:console is passed it only looks for main and wmain. lld-link used to look for all 4 in both cases. This patch makes lld-link match link.exe's behavior.

This requires that the subsystem is known by the time findDefaultEntry() gets called. findDefaultEntry() is called before the main link loop, so that the loop can mark the entry point as undefined. That means inferSubsystem() has to be called above the main loop as well. This in turn means /subsystem: from .drectve sections only has an effect on entry point inference for obj files passed to lld-link directly (and not in obj files found later in .lib files). link.exe seems to ignore /subsystem: for obj files from lib files completely (while in lld it's ignored only for entry point detection but it still overrides /subsystem: flags passed on the command line for the value that gets written in the output file).

Also, if the subsytem isn't needed (e.g. when only writing a /def: lib file and not writing a coff file), link.exe doesn't complain if the subsystem isn't known, so both subsystem and entry point handling should be below the early return lld has for that case.

Fixes PR36523.

[ruiu]

lld/COFF/Driver.cpp

428–429

This function seems a bit tricky to me especially where a boolean value is used as 0 or 1 to access an array. Maybe something like this would work:

auto Find = [](StringRef X, StringRef Y) {
  return findUnderscoreMangle(Config->NoDefaultLibAll ? Y : X);
};

if (Config->Subsystem == IMAGE_SUBSYSTEM_WINDOWS_CUI) {
  if (find("main", "mainCRTStartup"))
    return mangle("main");
  if (find("wmain", "wmainCRTStartup"))
    return mangle("wmain");
}

if (find("WinMain", "WinMainCRTStartup"))
  return mangle("WinMain");
if (find("wWinMain", "wWinMainCRTStartup"))
  return mangle("wWinMain");
return "";

[thakis]

lld/COFF/Driver.cpp

428–429

I agree the current function is overly clever. Thank you for your suggestion, it much easier tor read. On the other hand it looks a bit repetitive (the entry points each repeat once, and the overall structure is repeated a few times). Since link.exe prints a warning if both main and wmain or both WinMain and wWinMain are present and that's nicer to implement in your version, I went with that (without the lambda, which makes things a bit clever again). Please take another look.

[ruiu]

LGTM

[thakis]

339165, thanks!

lld/COFF/Driver.cpp

Symbol *LinkerDriver::addUndefined(StringRef Name) {
   return B;
 }
 
 // Windows specific -- find default entry point name.
 //
 // There are four different entry point functions for Windows executables,
 // each of which corresponds to a user-defined "main" function. This function
 // infers an entry point from a user-defined "main" function.
 StringRef LinkerDriver::findDefaultEntry() {
-  // User-defined main functions and their corresponding entry points.
-  static const char *Entries[][2] = {
-      {"main", "mainCRTStartup"},
-      {"wmain", "wmainCRTStartup"},
-      {"WinMain", "WinMainCRTStartup"},
-      {"wWinMain", "wWinMainCRTStartup"},
-  };
-  for (auto E : Entries) {
-    // As a special case, if /nodefaultlib is given, we directly look for an
-    // entry point. This is because, if no default library is linked, users
-    // need to define an entry point instead of a "main".
-    if (findUnderscoreMangle(E[Config->NoDefaultLibAll]))
-      return mangle(E[1]);
-  }
+  assert(Config->Subsystem != IMAGE_SUBSYSTEM_UNKNOWN &&
+         "must handle /subsystem before calling this");
+
+  bool FindMain = !Config->NoDefaultLibAll;
+  if (Config->Subsystem == IMAGE_SUBSYSTEM_WINDOWS_GUI) {
+    if (findUnderscoreMangle(FindMain ? "WinMain" : "WinMainCRTStartup"))
+      return mangle("WinMainCRTStartup");
+    if (findUnderscoreMangle(FindMain ? "wWinMain" : "wWinMainCRTStartup"))
+      return mangle("wWinMainCRTStartup");
+  }
+  if (findUnderscoreMangle(FindMain ? "main" : "mainCRTStartup"))
+    return mangle("mainCRTStartup");
+  if (findUnderscoreMangle(FindMain ? "wmain" : "wmainCRTStartup"))
+    return mangle("wmainCRTStartup");
   return "";
 }
 
 WindowsSubsystem LinkerDriver::inferSubsystem() {
   if (Config->DLL)
     return IMAGE_SUBSYSTEM_WINDOWS_GUI;
   if (findUnderscoreMangle("main") || findUnderscoreMangle("wmain"))
     return IMAGE_SUBSYSTEM_WINDOWS_CUI;
Config->HighEntropyVA =
       Config->is64() &&
       Args.hasFlag(OPT_highentropyva, OPT_highentropyva_no, true);
 
   if (!Config->DynamicBase &&
       (Config->Machine == ARMNT || Config->Machine == ARM64))
     error("/dynamicbase:no is not compatible with " +
           machineToStr(Config->Machine));
 
-  // Handle /entry and /dll
-  if (auto *Arg = Args.getLastArg(OPT_entry)) {
-    Config->Entry = addUndefined(mangle(Arg->getValue()));
-  } else if (!Config->Entry && !Config->NoEntry) {
-    if (Args.hasArg(OPT_dll)) {
-      StringRef S = (Config->Machine == I386) ? "__DllMainCRTStartup@12"
-                                              : "_DllMainCRTStartup";
-      Config->Entry = addUndefined(S);
-    } else {
-      // Windows specific -- If entry point name is not given, we need to
-      // infer that from user-defined entry name.
-      StringRef S = findDefaultEntry();
-      if (S.empty())
-        fatal("entry point must be defined");
-      Config->Entry = addUndefined(S);
-      log("Entry name inferred: " + S);
-    }
-  }
-
   // Handle /export
   for (auto *Arg : Args.filtered(OPT_export)) {
     Export E = parseExport(Arg->getValue());
     if (Config->Machine == I386) {
       if (!isDecorated(E.Name))
         E.Name = Saver.save("_" + E.Name);
       if (!E.ExtName.empty() && !isDecorated(E.ExtName))
         E.ExtName = Saver.save("_" + E.ExtName);
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
 
   // Handle generation of import library from a def file.
   if (!Args.hasArg(OPT_INPUT)) {
     fixupExports();
     createImportLibrary(/*AsLib=*/true);
     return;
   }
 
+  // Windows specific -- if no /subsystem is given, we need to infer
+  // that from entry point name.  Must happen before /entry handling,
+  // and after the early return when just writing an import library.
+  if (Config->Subsystem == IMAGE_SUBSYSTEM_UNKNOWN) {
+    Config->Subsystem = inferSubsystem();
+    if (Config->Subsystem == IMAGE_SUBSYSTEM_UNKNOWN)
+      fatal("subsystem must be defined");
+  }
+
+  // Handle /entry and /dll
+  if (auto *Arg = Args.getLastArg(OPT_entry)) {
+    Config->Entry = addUndefined(mangle(Arg->getValue()));
+  } else if (!Config->Entry && !Config->NoEntry) {
+    if (Args.hasArg(OPT_dll)) {
+      StringRef S = (Config->Machine == I386) ? "__DllMainCRTStartup@12"
+                                              : "_DllMainCRTStartup";
+      Config->Entry = addUndefined(S);
+    } else {
+      // Windows specific -- If entry point name is not given, we need to
+      // infer that from user-defined entry name.
+      StringRef S = findDefaultEntry();
+      if (S.empty())
+        fatal("entry point must be defined");
+      Config->Entry = addUndefined(S);
+      log("Entry name inferred: " + S);
+    }
+  }
+
   // Handle /delayload
   for (auto *Arg : Args.filtered(OPT_delayload)) {
     Config->DelayLoads.insert(StringRef(Arg->getValue()).lower());
     if (Config->Machine == I386) {
       Config->DelayLoadHelper = addUndefined("___delayLoadHelper2@8");
     } else {
       Config->DelayLoadHelper = addUndefined("__delayLoadHelper2");
     }
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   Symtab->addCombinedLTOObjects();
   run();
 
   // Make sure we have resolved all symbols.
   Symtab->reportRemainingUndefines();
   if (errorCount())
     return;
 
-  // Windows specific -- if no /subsystem is given, we need to infer
-  // that from entry point name.
-  if (Config->Subsystem == IMAGE_SUBSYSTEM_UNKNOWN) {
-    Config->Subsystem = inferSubsystem();
-    if (Config->Subsystem == IMAGE_SUBSYSTEM_UNKNOWN)
-      fatal("subsystem must be defined");
-  }
-
   // Handle /safeseh.
   if (Args.hasFlag(OPT_safeseh, OPT_safeseh_no, false)) {
     for (ObjFile *File : ObjFile::Instances)
       if (!File->hasSafeSEH())
         error("/safeseh: " + File->getName() + " is not compatible with SEH");
     if (errorCount())
       return;
   }

lld/test/COFF/entry-inference3.test

 # RUN: sed -e s/ENTRYNAME/mainCRTStartup/ %s | yaml2obj > %t.obj
-# RUN: not lld-link /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
+# RUN: lld-link /subsystem:console /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
 # RUN: FileCheck %s < %t.log
 
 # RUN: sed -e s/ENTRYNAME/?mainCRTStartup@@YAHXZ/ %s | yaml2obj > %t.obj
-# RUN: not lld-link /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
+# RUN: lld-link /subsystem:console /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
 # RUN: FileCheck %s < %t.log
 
 # CHECK: Entry name inferred: mainCRTStartup
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_AMD64
   Characteristics: []

lld/test/COFF/entry-inference332.test

 # RUN: sed -e s/ENTRYNAME/_mainCRTStartup/ %s | yaml2obj > %t.obj
-# RUN: not lld-link /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
+# RUN: lld-link /subsystem:console /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
 # RUN: FileCheck %s < %t.log
 
 # RUN: sed -e s/ENTRYNAME/?mainCRTStartup@@YAHXZ/ %s | yaml2obj > %t.obj
-# RUN: not lld-link /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
+# RUN: lld-link /subsystem:console /out:%t.exe %t.obj /verbose /nodefaultlib > %t.log 2>&1
 # RUN: FileCheck %s < %t.log
 
 # CHECK: Entry name inferred: _mainCRTStartup
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_I386
   Characteristics: []

lld/test/COFF/entry-inference4.test

+# RUN: sed 's/ENTRY1/WinMain/;s/ENTRY2/main/' %s | yaml2obj > %t.obj
+# RUN: not lld-link /subsystem:windows /out:%t.exe %t.obj > %t.log 2>&1
+# RUN: FileCheck -check-prefix=WINMAIN %s < %t.log
+
+# RUN: sed 's/ENTRY1/wWinMain/;s/ENTRY2/main/' %s | yaml2obj > %t.obj
+# RUN: not lld-link /subsystem:windows /out:%t.exe %t.obj > %t.log 2>&1
+# RUN: FileCheck -check-prefix=WWINMAIN %s < %t.log
+
+# RUN: sed 's/ENTRY1/WinMain/;s/ENTRY2/main/' %s | yaml2obj > %t.obj
+# RUN: not lld-link /subsystem:console /out:%t.exe %t.obj > %t.log 2>&1
+# RUN: FileCheck -check-prefix=MAIN %s < %t.log
+
+# RUN: sed 's/ENTRY1/WinMain/;s/ENTRY2/wmain/' %s | yaml2obj > %t.obj
+# RUN: not lld-link /subsystem:console /out:%t.exe %t.obj > %t.log 2>&1
+# RUN: FileCheck -check-prefix=WMAIN %s < %t.log
+
+# MAIN:     error: <root>: undefined symbol: mainCRTStartup
+# WMAIN:    error: <root>: undefined symbol: wmainCRTStartup
+# WINMAIN:  error: <root>: undefined symbol: WinMainCRTStartup
+# WWINMAIN: error: <root>: undefined symbol: wWinMainCRTStartup
+
+--- !COFF
+header:
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: []
+sections:
+  - Name:            .text
+    Characteristics: [ IMAGE_SCN_CNT_CODE, IMAGE_SCN_MEM_EXECUTE, IMAGE_SCN_MEM_READ ]
+    Alignment:       4
+    SectionData:     B82A000000C3
+symbols:
+  - Name:            .text
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition:
+      Length:          6
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            ENTRY1
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            ENTRY2
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+...