[MinGW] Don't let template instantiation declarations cover nested classes

[mstorsjo]

Summary

An explicit template instantiation declaration used to let callers assume both outer and nested classes instantiations were defined in a different translation unit.

If the instantiation is marked dllexport, only the outer class is exported, but the caller will try to reference the instantiation of both outer and inner classes.

This makes MinGW mode match both MSVC and Windows Itanium, by having instantations only cover the outer class, and locally emitting definitions of the nested classes. Windows Itanium was changed to use this behavious in SVN r300804.

This deviates from what GCC does, but should be safe (and only inflate the object file size a bit, but MSVC and Windows Itanium modes do the same), and fixes cases where inner classes aren't dllexported.

This fixes missing references in combination with dllexported/imported template intantiations.

GCC suffers from the same issue, reported at [1], but the issue is still unresolved there. The issue can probably be solved either by making dllexport cover all nested classes as well, or this way (matching MSVC).

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89087

[hans]

Seems okay to me.

[rnk]

lib/Sema/SemaTemplateInstantiate.cpp

2687–2689

I think this can be simplified to "if Windows" at this point. But, I'm confused why we need this change to the general template instantiation machinery... Anyway, I'll send the simplification as a code review.

[mstorsjo]

lib/Sema/SemaTemplateInstantiate.cpp

2687–2689

Yes, I believe so. Not sure about why this is needed though.

In case GCC actually does make a move on this matter, and chooses a different path for fixing it, we might want to follow that so in that case we might need to reintroduce some condition like this. But until that hypothetical case, let's simplify it.

lib/Sema/SemaTemplateInstantiate.cpp

if (auto *Function = dyn_cast<FunctionDecl>(D)) {
 
       MemberSpecializationInfo *MSInfo = Record->getMemberSpecializationInfo();
       assert(MSInfo && "No member specialization information?");
 
       if (MSInfo->getTemplateSpecializationKind()
                                                 == TSK_ExplicitSpecialization)
         continue;
 
       if ((Context.getTargetInfo().getCXXABI().isMicrosoft() ||
-           Context.getTargetInfo().getTriple().isWindowsItaniumEnvironment()) &&
+           Context.getTargetInfo().getTriple().isWindowsItaniumEnvironment() ||
+           Context.getTargetInfo().getTriple().isWindowsGNUEnvironment()) &&
           TSK == TSK_ExplicitInstantiationDeclaration) {
         // In MSVC and Windows Itanium mode, explicit instantiation decl of the
         // outer class doesn't affect the inner class.
+        // In GNU mode, inner classes aren't dllexported. Don't let the
+        // instantiation cover the inner class, to avoid undefined references
+        // to inner classes that weren't exported.
         continue;
       }
 
       if (CheckSpecializationInstantiationRedecl(PointOfInstantiation, TSK,
                                                  Record,
                                         MSInfo->getTemplateSpecializationKind(),
                                               MSInfo->getPointOfInstantiation(),
                                                  SuppressNew) ||

test/CodeGenCXX/mingw-template-dllexport.cpp

 // RUN: %clang_cc1 -emit-llvm -triple i686-mingw32 %s -o - | FileCheck %s
 
+#define JOIN2(x, y) x##y
+#define JOIN(x, y) JOIN2(x, y)
+#define UNIQ(name) JOIN(name, __LINE__)
+#define USEMEMFUNC(class, func) void (class::*UNIQ(use)())() { return &class::func; }
+
 template <class T>
 class c {
   void f();
 };
 
 template <class T> void c<T>::f() {}
 
 template class __declspec(dllexport) c<int>;

 
 template <class T> void outer<T>::f() {}
 template <class T> void outer<T>::inner::f() {}
 
 template class __declspec(dllexport) outer<int>;
 
 // CHECK: define {{.*}} dllexport {{.*}} @_ZN5outerIiE1fEv
 // CHECK-NOT: define {{.*}} dllexport {{.*}} @_ZN5outerIiE5inner1fEv
+
+extern template class __declspec(dllimport) outer<char>;
+USEMEMFUNC(outer<char>, f)
+USEMEMFUNC(outer<char>::inner, f)
+
+// CHECK: declare dllimport {{.*}} @_ZN5outerIcE1fEv
+// CHECK: define {{.*}} @_ZN5outerIcE5inner1fEv