[LLD] [COFF] Do MinGW specific entry/subsystem inference

[mstorsjo]

Summary

ld.bfd doesn't do any inference of subsystem; unless the windows subsystem is specified, the console subsystem is used.

For the console subsystem, the entry point is called mainCRTStartup, regardless of whether the the user code entry point is main or wmain. The same goes for the windows subsystem, where the entry point always is WinMainCRTStartup, for both WinMain and wWinMain in user code.

One detail that we don't emulate, is that if the inferred entry point is undefined, ld.bfd silently just sets the entry point to the start of the image. And if an explicit entry point is set, but it is undefined, the link still succeeds but the linker warns about the entry point not being found.

[ruiu]

This function is called only when you did not pass /entry:.

If MinGW uses only mainCRTStartup or WinMainCRTStartup, can you determine which is the case just by looking at a command line? If so, I'd always explicitly pass /entry: to the lld-link driver from the mingw driver.

[mstorsjo]

In D52931#1256687, @ruiu wrote:

This function is called only when you did not pass /entry:.

If MinGW uses only mainCRTStartup or WinMainCRTStartup, can you determine which is the case just by looking at a command line? If so, I'd always explicitly pass /entry: to the lld-link driver from the mingw driver.

We could do that, but that departs a bit from what the gcc driver passes to ld.bfd. However I took a closer look at what ld.bfd actually does wrt this, and we can emulate it much better.

[mstorsjo]

Emulating the ld.bfd behaviour much more closely.

[ruiu]

LGTM

COFF/Driver.cpp

 StringRef LinkerDriver::findDefaultEntry() {
   assert(Config->Subsystem != IMAGE_SUBSYSTEM_UNKNOWN &&
          "must handle /subsystem before calling this");
 
   if (Config->Subsystem == IMAGE_SUBSYSTEM_WINDOWS_GUI) {
     if (findUnderscoreMangle("wWinMain")) {
       if (!findUnderscoreMangle("WinMain"))
         return mangle("wWinMainCRTStartup");
+      if (!Config->MinGW)
         warn("found both wWinMain and WinMain; using latter");
     }
     return mangle("WinMainCRTStartup");
   }
   if (findUnderscoreMangle("wmain")) {
     if (!findUnderscoreMangle("main"))
       return mangle("wmainCRTStartup");
+    if (!Config->MinGW)
       warn("found both wmain and main; using latter");
   }
   return mangle("mainCRTStartup");
 }
 
 WindowsSubsystem LinkerDriver::inferSubsystem() {
   if (Config->DLL)
     return IMAGE_SUBSYSTEM_WINDOWS_GUI;
   // Note that link.exe infers the subsystem from the presence of these

test/COFF/entry-inference5.s

+# REQUIRES: x86
+# RUN: echo -e ".text\n.globl otherfunc\notherfunc:\nret" > %t-lib1.s
+# RUN: echo -e ".text\n.globl wmain\nwmain:\nret" > %t-lib2.s
+# RUN: llvm-mc -triple=x86_64-windows-gnu %s -filetype=obj -o %t.o
+# RUN: llvm-mc -triple=x86_64-windows-gnu %t-lib1.s -filetype=obj -o %t-lib1.o
+# RUN: llvm-mc -triple=x86_64-windows-gnu %t-lib2.s -filetype=obj -o %t-lib2.o
+# RUN: llvm-ar crs %t-lib.a %t-lib1.o %t-lib2.o
+
+# RUN: lld-link -lldmingw %t.o %t-lib.a -out:%t.exe 2>&1 | FileCheck -allow-empty %s
+
+# Check that this doesn't print any warnings.
+# CHECK-NOT: found both wmain and main
+
+        .text
+        .globl main
+        .globl mainCRTStartup
+main:
+        call otherfunc
+        ret
+mainCRTStartup:
+        call main
+        ret