[Mangler] Mangle aliases to fastcall/vectorcall functions correctly

[Amanieu]

Summary

These aliases are produced by MergeFunctions and need to be mangled according to the calling convention of the function they are pointing to instead of defaulting to the C calling convention.

[Amanieu]

Fixed to use dyn_cast_or_null.

[rnk]

I have some misgivings about this, but I think this is probably the best way forward for now.

I think it was always a mistake to do IR mangling based on the calling convention and prototype. However, that's what we do, and removing that is a long process. FWIW clang never relies on the IR mangler for functions like this, it does the mangling itself.

The other way to handle this is to teach merge functions to do IR mangling. Merge functions can create a Mangler object with the DataLayout from the Module. Then it can create escaped aliases with names like "\01@foo@20". After thinking about it, I think I prefer your approach, despite the surprising behavior.

In the long run, I would like to remove this calling convention mangling altogether, at which point this change will be deleted.

[rnk]

lgtm

[Amanieu]

For context, I encountered a linker error caused by this in Rust which does use the IR mangler.

llvm/lib/IR/Mangler.cpp

if (!GV->hasName()) {
     return;
   }
 
   StringRef Name = GV->getName();
   char Prefix = DL.getGlobalPrefix();
 
   // Mangle functions with Microsoft calling conventions specially.  Only do
   // this mangling for x86_64 vectorcall and 32-bit x86.
-  const Function *MSFunc = dyn_cast<Function>(GV);
+  const Function *MSFunc = dyn_cast_or_null<Function>(GV->getAliaseeObject());
 
   // Don't add byte count suffixes when '\01' or '?' are in the first
   // character.
   if (Name.startswith("\01") ||
       (DL.doNotMangleLeadingQuestionMark() && Name.startswith("?")))
     MSFunc = nullptr;
 
   CallingConv::ID CC =

llvm/test/CodeGen/X86/fastcall-correct-mangling.ll

 }
 
 define private x86_fastcallcc void @dontCrash() {
 ; The name is fairly arbitrary since it is private. Just don't crash.
 ; CHECK32-LABEL: {{^}}L@dontCrash@0:
 ; CHECK64-LABEL: {{^}}.LdontCrash:
   ret void
 }
+
+@alias = alias void(i64, i8, i8, i16), void(i64, i8, i8, i16)* @func
+; CHECK32-LABEL: {{^}}.set @alias@20, @func@20
+; CHECK64-LABEL: {{^}}.set alias, func