[LLD] [COFF] Do MinGW specific entry/subsystem inference

[mstorsjo]

Summary

ld.bfd doesn't do any inference of subsystem; unless the windows subsystem is specified, the console subsystem is used.

For the console subsystem, the entry point is called mainCRTStartup, regardless of whether the the user code entry point is main or wmain. The same goes for the windows subsystem, where the entry point always is WinMainCRTStartup, for both WinMain and wWinMain in user code.

One detail that we don't emulate, is that if the inferred entry point is undefined, ld.bfd silently just sets the entry point to the start of the image. And if an explicit entry point is set, but it is undefined, the link still succeeds but the linker warns about the entry point not being found.

[ruiu]

This function is called only when you did not pass /entry:.

If MinGW uses only mainCRTStartup or WinMainCRTStartup, can you determine which is the case just by looking at a command line? If so, I'd always explicitly pass /entry: to the lld-link driver from the mingw driver.

[mstorsjo]

In D52931#1256687, @ruiu wrote:

This function is called only when you did not pass /entry:.

If MinGW uses only mainCRTStartup or WinMainCRTStartup, can you determine which is the case just by looking at a command line? If so, I'd always explicitly pass /entry: to the lld-link driver from the mingw driver.

We could do that, but that departs a bit from what the gcc driver passes to ld.bfd. However I took a closer look at what ld.bfd actually does wrt this, and we can emulate it much better.

[mstorsjo]

Emulating the ld.bfd behaviour much more closely.

[ruiu]

LGTM

COFF/Driver.cpp

 //
 // There are four different entry point functions for Windows executables,
 // each of which corresponds to a user-defined "main" function. This function
 // infers an entry point from a user-defined "main" function.
 StringRef LinkerDriver::findDefaultEntry() {
   assert(Config->Subsystem != IMAGE_SUBSYSTEM_UNKNOWN &&
          "must handle /subsystem before calling this");
 
+  if (Config->MinGW)
+    return mangle(Config->Subsystem == IMAGE_SUBSYSTEM_WINDOWS_GUI
+                      ? "WinMainCRTStartup"
+                      : "mainCRTStartup");
+
   if (Config->Subsystem == IMAGE_SUBSYSTEM_WINDOWS_GUI) {
     if (findUnderscoreMangle("wWinMain")) {
       if (!findUnderscoreMangle("WinMain"))
         return mangle("wWinMainCRTStartup");
       warn("found both wWinMain and WinMain; using latter");
     }
     return mangle("WinMainCRTStartup");
   }
   if (findUnderscoreMangle("wmain")) {
     if (!findUnderscoreMangle("main"))
       return mangle("wmainCRTStartup");
     warn("found both wmain and main; using latter");
   }
   return mangle("mainCRTStartup");
 }
 
 WindowsSubsystem LinkerDriver::inferSubsystem() {
   if (Config->DLL)
     return IMAGE_SUBSYSTEM_WINDOWS_GUI;
+  if (Config->MinGW)
+    return IMAGE_SUBSYSTEM_WINDOWS_CUI;
   // Note that link.exe infers the subsystem from the presence of these
   // functions even if /entry: or /nodefaultlib are passed which causes them
   // to not be called.
   bool HaveMain = findUnderscoreMangle("main");
   bool HaveWMain = findUnderscoreMangle("wmain");
   bool HaveWinMain = findUnderscoreMangle("WinMain");
   bool HaveWWinMain = findUnderscoreMangle("wWinMain");
   if (HaveMain || HaveWMain) {

test/COFF/entry-inference-mingw.s

+# REQUIRES: x86
+# RUN: llvm-mc -triple=x86_64-windows-gnu %s -filetype=obj -o %t.o
+
+# RUN: lld-link -lldmingw %t.o -out:%t-default.exe 2>&1 | FileCheck -allow-empty -check-prefix=LINK %s
+# RUN: lld-link -lldmingw %t.o -out:%t-cui.exe -subsystem:console 2>&1 | FileCheck -allow-empty -check-prefix=LINK %s
+# RUN: lld-link -lldmingw %t.o -out:%t-gui.exe -subsystem:windows 2>&1 | FileCheck -allow-empty -check-prefix=LINK %s
+
+# RUN: llvm-readobj -file-headers %t-default.exe | FileCheck -check-prefix=CUI %s
+# RUN: llvm-readobj -file-headers %t-cui.exe | FileCheck -check-prefix=CUI %s
+# RUN: llvm-readobj -file-headers %t-gui.exe | FileCheck -check-prefix=GUI %s
+
+# Check that this doesn't print any warnings.
+# LINK-NOT: found both wmain and main
+
+# CUI: AddressOfEntryPoint: 0x1001
+# CUI: Subsystem: IMAGE_SUBSYSTEM_WINDOWS_CUI (0x3)
+
+# GUI: AddressOfEntryPoint: 0x1002
+# GUI: Subsystem: IMAGE_SUBSYSTEM_WINDOWS_GUI (0x2)
+
+
+        .text
+        .globl mainCRTStartup
+        .globl WinMainCRTStartup
+# MinGW only uses the entry points above, these other ones aren't
+# used as entry.
+        .globl main
+        .globl wmain
+        .globl wmainCRTStartup
+        .globl wWinMainCRTStartup
+foo:
+        ret
+mainCRTStartup:
+        ret
+WinMainCRTStartup:
+        ret
+main:
+        ret
+wmain:
+        ret
+wmainCRTStartup:
+        ret
+wWinMainCRTStartup:
+        ret

test/COFF/subsystem-inference-mingw.s

+# REQUIRES: x86
+# RUN: llvm-mc -triple=x86_64-windows-gnu %s -filetype=obj -o %t.o
+
+# RUN: lld-link -lldmingw %t.o -out:%t.exe
+# RUN: llvm-readobj -file-headers %t.exe | FileCheck %s
+
+# CHECK: AddressOfEntryPoint: 0x1001
+# CHECK: Subsystem: IMAGE_SUBSYSTEM_WINDOWS_CUI (0x3)
+
+        .text
+        .globl foo
+        .globl mainCRTStartup
+foo:
+        ret
+mainCRTStartup:
+        call foo
+        ret