Microsoft mangling of return type deducing functions/lambdas that return local types

[majnemer]

Summary

The following code causes clang to enter into an infinite recursion on windows 7 while attempting to microsoft-mangle the following lambda's operator function type:

auto L = [] () {
   struct M {};
   return M{};
};  

auto X = L();

I have kludged together a quick and dirty patch to identify where the problem lies, so that those who understand the microsoft-mangling process better can either help me refine it - or if they prefer - modify it to make it commit-worthy and commit it.

[faisalv]

lib/AST/MicrosoftMangle.cpp

1557

FYI - I suspect local enums might cause such a problem too, right (what else?)

[majnemer]

Ugh, the MS ABI is not very amenable to this sort of thing. Current versions of MSVC cannot reason about the following source let alone mangle it:

template <typename T>
auto fun(T) {
  auto L = []() {
    struct M {};
    return M{};
  };
  return L();
}
auto doit() { return fun(0); }

Heck, even GCC does strange things here:
_ZZ3funIiEDaT_ENUlvE_4_FUNEv which demangles as auto fun<int>(int)::'lambda'()::_FUN()

I'd prefer if we mangled L's type as: 'R<lambda'_{mangling_number}>::operator()() const::M

[majnemer]

A different fix was landed in rL205282.

lib/AST/MicrosoftMangle.cpp

 
 #include "clang/AST/Mangle.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/ASTLambda.h"
 #include "clang/AST/Attr.h"
 #include "clang/AST/CXXInheritance.h"
 #include "clang/AST/CharUnits.h"

     Out << '@';
   } else {
     QualType ResultType = Proto->getReturnType();
+    bool AlreadyMangledReturnType = false;
     if (ResultType->isVoidType())
       ResultType = ResultType.getUnqualifiedType();
-    mangleType(ResultType, Range, QMM_Result);
+    else if (ResultType->getContainedAutoType()) {
+      if (ResultType->isRecordType()) {
+        CXXRecordDecl *RD = ResultType->getAsCXXRecordDecl();
+        // Is this type defined within the function/lambda itself and 
+        // deduced as the return type? i.e
+        //   auto fun() { struct M{}; return M{}; } 
+        // If so, prevent infinite recursion, by mangling this here as a temporary fix me.
+        if (FunctionDecl *FD = dyn_cast<FunctionDecl>(RD->getDeclContext())) {
+          while (FD) {
+            FD = FD->getFirstDecl();
+            if (FD == D) {
+              Out << '?';
+              mangleQualifiers(ResultType.getLocalQualifiers(), false);
+              mangleSourceName("<fixme-nested-local-tag>");
+              AlreadyMangledReturnType = true;
+              break;
+            }
+            FD = dyn_cast<FunctionDecl>(getLambdaAwareParentOfDeclContext(FD));
+          }
+        }
+      } else if (ResultType->isUndeducedType()) {
+        Out << '?';
+        mangleQualifiers(ResultType.getLocalQualifiers(), false);
+        mangleSourceName("<fixme-undeduced-auto-return-type>");
+        AlreadyMangledReturnType = true;
+      }
+    }
+    if (!AlreadyMangledReturnType)  
+      mangleType(ResultType, Range, QMM_Result);
   }
 
   // <argument-list> ::= X # void