[clang][ASTImporter] Add import of type-related nodes

[danix800]

Summary

Add import of type-related nodes:

• bitIntType
• constantMatrixType
• dependentAddressSpaceType
• dependentBitIntType
• dependentSizedMatrixType
• dependentVectorType
• objcTypeParamDecl
• objcTypeParamType
• pipeType
• vectorType

[balazske]

clang/lib/AST/ASTImporter.cpp

1758

At cases like this (many imported values) importChecked can be used, like at VisitDependentSizedArrayType.

clang/unittests/AST/ASTImporterObjCTest.cpp

100

This single assert can be sufficient for this test because the ToTU was empty before import, it is not expected that more than one instance will be created. Better is to check for example if getTypeForDecl is correctly imported.

[danix800]

1. Simplify error handling
2. Remove unnecessary tests

[danix800]

Changes planned for https://reviews.llvm.org/D158872 is still under discussion.

[danix800]

Pull in local matchers for testcases, since https://reviews.llvm.org/D158872 is abandoned.

[haowei]

Hi I am working on updating GoogleTest in LLVM to v1.14.0 and during my local testing, I noticed that this patch added 2 new tests ImportMatrixType and ImportOpenCLPipe through TEST_P calls, however, both tests were not instantiated by INSTANTIATE_TEST_SUITE_P so they were not ran during check-clang. This will causes test failures when using GoogleTest v1.14.0 since it now explicitly fail when an INSTANTIATE_TEST_SUITE_P is not paired with an TEST_P I manually added

INSTANTIATE_TEST_SUITE_P(ParameterizedTests, ImportMatrixType,
                         DefaultTestValuesForRunOptions);

INSTANTIATE_TEST_SUITE_P(ParameterizedTests, ImportOpenCLPipe,
                         DefaultTestValuesForRunOptions);

to ASTImporterTest.cpp and reran the clang test and it failed with error message:

[ RUN      ] ParameterizedTests/ImportOpenCLPipe.ImportPipeType/0
Not implemented yet!
UNREACHABLE executed at clang/lib/Testing/CommandLineArgs.cpp:47!
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  ASTTests 0x00005565e9e71d18

--
exit: -6
--

Looking at https://github.com/llvm/llvm-project/blob/37a20cc68f545647e614c5ba4ae311dc3fd277e9/clang/lib/Testing/CommandLineArgs.cpp#L47, this is were the unreachable code was hit. Is it the intended behavior?

Shall I enable ImportMatrixType and disable ImportOpenCLPipe tests so we can update the GoogleTest in LLVM?

[danix800]

In D158948#4642422, @haowei wrote:

Looking at https://github.com/llvm/llvm-project/blob/37a20cc68f545647e614c5ba4ae311dc3fd277e9/clang/lib/Testing/CommandLineArgs.cpp#L47, this is were the unreachable code was hit. Is it the intended behavior?

Shall I enable ImportMatrixType and disable ImportOpenCLPipe tests so we can update the GoogleTest in LLVM?

I'm not familiar with GoogleTest so I wasn't expecting this to happen.

To enabling ImportOpenCLPipe we might use "-x", "cl", "-cl-no-stdinc", "-cl-std=CL2.0" here.

See also https://reviews.llvm.org/D158872 for some extra context of both testing with ImportMatrixType and ImportOpenCLPipe.

[danix800]

@haowei I noticed https://github.com/llvm/llvm-project/pull/65823, is it related to your work?

[haowei]

In D158948#4642475, @danix800 wrote:

In D158948#4642422, @haowei wrote:

Looking at https://github.com/llvm/llvm-project/blob/37a20cc68f545647e614c5ba4ae311dc3fd277e9/clang/lib/Testing/CommandLineArgs.cpp#L47, this is were the unreachable code was hit. Is it the intended behavior?

Shall I enable ImportMatrixType and disable ImportOpenCLPipe tests so we can update the GoogleTest in LLVM?

I'm not familiar with GoogleTest so I wasn't expecting this to happen.

To enabling ImportOpenCLPipe we might use "-x", "cl", "-cl-no-stdinc", "-cl-std=CL2.0" here.

See also https://reviews.llvm.org/D158872 for some extra context of both testing with ImportMatrixType and ImportOpenCLPipe.

In a nutshell, ImportOpenCLPipe wasn't enabled after your patch, that is why the test failure was unnoticed. You have to add

INSTANTIATE_TEST_SUITE_P(ParameterizedTests, ImportOpenCLPipe,
                         DefaultTestValuesForRunOptions);

in the same namespace like other tests in this file to let it run. I am very unfamiliar with Clang AST codebase so sorry I don't really know what changes you need to make this test pass. At current state, this test will lead to unreachable code error.

https://github.com/llvm/llvm-project/pull/65823 is my attempt to roll GoogleTest 1.14.0 to bring some upstream bug fixes and improvements to LLVM. Patch https://github.com/llvm/llvm-project/pull/65823/commits/ff67955ecbafcf847e11188c44e8dd070899b0b6 attempts to enable ImportMatrixType and explicitly disable ImportOpenCLPipe so the check-clang step can pass under GoogleTest v1.14.0

clang/lib/AST/ASTImporter.cpp

public:
     explicit ASTNodeImporter(ASTImporter &Importer) : Importer(Importer) {}
 
     using TypeVisitor<ASTNodeImporter, ExpectedType>::Visit;
     using DeclVisitor<ASTNodeImporter, ExpectedDecl>::Visit;
     using StmtVisitor<ASTNodeImporter, ExpectedStmt>::Visit;
 
     // Importing types
     ExpectedType VisitType(const Type *T);
-    ExpectedType VisitAtomicType(const AtomicType *T);
-    ExpectedType VisitBuiltinType(const BuiltinType *T);
-    ExpectedType VisitDecayedType(const DecayedType *T);
-    ExpectedType VisitComplexType(const ComplexType *T);
-    ExpectedType VisitPointerType(const PointerType *T);
-    ExpectedType VisitBlockPointerType(const BlockPointerType *T);
-    ExpectedType VisitLValueReferenceType(const LValueReferenceType *T);
-    ExpectedType VisitRValueReferenceType(const RValueReferenceType *T);
-    ExpectedType VisitMemberPointerType(const MemberPointerType *T);
-    ExpectedType VisitConstantArrayType(const ConstantArrayType *T);
-    ExpectedType VisitIncompleteArrayType(const IncompleteArrayType *T);
-    ExpectedType VisitVariableArrayType(const VariableArrayType *T);
-    ExpectedType VisitDependentSizedArrayType(const DependentSizedArrayType *T);
-    ExpectedType
-    VisitDependentSizedExtVectorType(const DependentSizedExtVectorType *T);
-    ExpectedType VisitVectorType(const VectorType *T);
-    ExpectedType VisitExtVectorType(const ExtVectorType *T);
-    ExpectedType VisitFunctionNoProtoType(const FunctionNoProtoType *T);
-    ExpectedType VisitFunctionProtoType(const FunctionProtoType *T);
-    ExpectedType VisitUnresolvedUsingType(const UnresolvedUsingType *T);
-    ExpectedType VisitParenType(const ParenType *T);
-    ExpectedType VisitTypedefType(const TypedefType *T);
-    ExpectedType VisitTypeOfExprType(const TypeOfExprType *T);
-    // FIXME: DependentTypeOfExprType
-    ExpectedType VisitTypeOfType(const TypeOfType *T);
-    ExpectedType VisitUsingType(const UsingType *T);
-    ExpectedType VisitDecltypeType(const DecltypeType *T);
-    ExpectedType VisitUnaryTransformType(const UnaryTransformType *T);
-    ExpectedType VisitAutoType(const AutoType *T);
-    ExpectedType VisitDeducedTemplateSpecializationType(
-        const DeducedTemplateSpecializationType *T);
-    ExpectedType VisitInjectedClassNameType(const InjectedClassNameType *T);
-    // FIXME: DependentDecltypeType
-    ExpectedType VisitRecordType(const RecordType *T);
-    ExpectedType VisitEnumType(const EnumType *T);
-    ExpectedType VisitAttributedType(const AttributedType *T);
-    ExpectedType VisitTemplateTypeParmType(const TemplateTypeParmType *T);
-    ExpectedType VisitSubstTemplateTypeParmType(
-        const SubstTemplateTypeParmType *T);
-    ExpectedType
-    VisitSubstTemplateTypeParmPackType(const SubstTemplateTypeParmPackType *T);
-    ExpectedType VisitTemplateSpecializationType(
-        const TemplateSpecializationType *T);
-    ExpectedType VisitElaboratedType(const ElaboratedType *T);
-    ExpectedType VisitDependentNameType(const DependentNameType *T);
-    ExpectedType VisitPackExpansionType(const PackExpansionType *T);
-    ExpectedType VisitDependentTemplateSpecializationType(
-        const DependentTemplateSpecializationType *T);
-    ExpectedType VisitObjCInterfaceType(const ObjCInterfaceType *T);
-    ExpectedType VisitObjCObjectType(const ObjCObjectType *T);
-    ExpectedType VisitObjCObjectPointerType(const ObjCObjectPointerType *T);
-    ExpectedType VisitMacroQualifiedType(const MacroQualifiedType *T);
+#define TYPE(Class, Base)                                                    \
+    ExpectedType Visit##Class##Type(const Class##Type *T);
+#include "clang/AST/TypeNodes.inc"
 
     // Importing declarations
     Error ImportDeclParts(NamedDecl *D, DeclarationName &Name, NamedDecl *&ToD,
                           SourceLocation &Loc);
     Error ImportDeclParts(
         NamedDecl *D, DeclContext *&DC, DeclContext *&LexicalDC,
         DeclarationName &Name, NamedDecl *&ToD, SourceLocation &Loc);
     Error ImportDefinitionIfNeeded(Decl *FromD, Decl *ToD = nullptr);
ASTNodeImporter::VisitMacroQualifiedType(const MacroQualifiedType *T) {
   if (!ToUnderlyingTypeOrErr)
     return ToUnderlyingTypeOrErr.takeError();
 
   IdentifierInfo *ToIdentifier = Importer.Import(T->getMacroIdentifier());
   return Importer.getToContext().getMacroQualifiedType(*ToUnderlyingTypeOrErr,
                                                        ToIdentifier);
 }
 
+ExpectedType clang::ASTNodeImporter::VisitAdjustedType(const AdjustedType *T) {
+  Error Err = Error::success();
+  QualType ToOriginalType = importChecked(Err, T->getOriginalType());
+  QualType ToAdjustedType = importChecked(Err, T->getAdjustedType());
+  if (Err)
+    return std::move(Err);
+
+  return Importer.getToContext().getAdjustedType(ToOriginalType,
+                                                 ToAdjustedType);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitBitIntType(const BitIntType *T) {
+  return Importer.getToContext().getBitIntType(T->isUnsigned(),
+                                               T->getNumBits());
+}
+
+ExpectedType clang::ASTNodeImporter::VisitBTFTagAttributedType(
+    const clang::BTFTagAttributedType *T) {
+  Error Err = Error::success();
+  const BTFTypeTagAttr *ToBTFAttr = importChecked(Err, T->getAttr());
+  QualType ToWrappedType = importChecked(Err, T->getWrappedType());
+  if (Err)
+    return std::move(Err);
+
+  return Importer.getToContext().getBTFTagAttributedType(ToBTFAttr,
+                                                         ToWrappedType);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitConstantMatrixType(
+    const clang::ConstantMatrixType *T) {
+  ExpectedType ToElementTypeOrErr = import(T->getElementType());
+  if (!ToElementTypeOrErr)
+    return ToElementTypeOrErr.takeError();
+
+  return Importer.getToContext().getConstantMatrixType(
+      *ToElementTypeOrErr, T->getNumRows(), T->getNumColumns());
+}
+
+ExpectedType clang::ASTNodeImporter::VisitDependentAddressSpaceType(
+    const clang::DependentAddressSpaceType *T) {
+  Error Err = Error::success();
+  QualType ToPointeeType = importChecked(Err, T->getPointeeType());
+  Expr *ToAddrSpaceExpr = importChecked(Err, T->getAddrSpaceExpr());
+  SourceLocation ToAttrLoc = importChecked(Err, T->getAttributeLoc());
+  if (Err)
+    return std::move(Err);
+
+  return Importer.getToContext().getDependentAddressSpaceType(
+      ToPointeeType, ToAddrSpaceExpr, ToAttrLoc);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitDependentBitIntType(
+    const clang::DependentBitIntType *T) {
+  ExpectedExpr ToNumBitsExprOrErr = import(T->getNumBitsExpr());
+  if (!ToNumBitsExprOrErr)
+    return ToNumBitsExprOrErr.takeError();
+  return Importer.getToContext().getDependentBitIntType(T->isUnsigned(),
+                                                        *ToNumBitsExprOrErr);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitDependentSizedMatrixType(
+    const clang::DependentSizedMatrixType *T) {
+  Error Err = Error::success();
+  QualType ToElementType = importChecked(Err, T->getElementType());
+  Expr *ToRowExpr = importChecked(Err, T->getRowExpr());
+  Expr *ToColumnExpr = importChecked(Err, T->getColumnExpr());
+  SourceLocation ToAttrLoc = importChecked(Err, T->getAttributeLoc());
+  if (Err)
+    return std::move(Err);
+
+  return Importer.getToContext().getDependentSizedMatrixType(
+      ToElementType, ToRowExpr, ToColumnExpr, ToAttrLoc);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitDependentVectorType(
+    const clang::DependentVectorType *T) {
+  Error Err = Error::success();
+  QualType ToElementType = importChecked(Err, T->getElementType());
+  Expr *ToSizeExpr = importChecked(Err, T->getSizeExpr());
+  SourceLocation ToAttrLoc = importChecked(Err, T->getAttributeLoc());
+  if (Err)
+    return std::move(Err);
+
+  return Importer.getToContext().getDependentVectorType(
+      ToElementType, ToSizeExpr, ToAttrLoc, T->getVectorKind());
+}
+
+ExpectedType clang::ASTNodeImporter::VisitObjCTypeParamType(
+    const clang::ObjCTypeParamType *T) {
+  Expected<ObjCTypeParamDecl *> ToDeclOrErr = import(T->getDecl());
+  if (!ToDeclOrErr)
+    return ToDeclOrErr.takeError();
+
+  SmallVector<ObjCProtocolDecl *, 4> ToProtocols;
+  for (ObjCProtocolDecl *FromProtocol : T->getProtocols()) {
+    Expected<ObjCProtocolDecl *> ToProtocolOrErr = import(FromProtocol);
+    if (!ToProtocolOrErr)
+      return ToProtocolOrErr.takeError();
+    ToProtocols.push_back(*ToProtocolOrErr);
+  }
+
+  return Importer.getToContext().getObjCTypeParamType(*ToDeclOrErr,
+                                                      ToProtocols);
+}
+
+ExpectedType clang::ASTNodeImporter::VisitPipeType(const clang::PipeType *T) {
+  ExpectedType ToElementTypeOrErr = import(T->getElementType());
+  if (!ToElementTypeOrErr)
+    return ToElementTypeOrErr.takeError();
+
+  ASTContext &ToCtx = Importer.getToContext();
+  if (T->isReadOnly())
+    return ToCtx.getReadPipeType(*ToElementTypeOrErr);
+  else
+    return ToCtx.getWritePipeType(*ToElementTypeOrErr);
+}
+
 //----------------------------------------------------------------------------
 // Import Declarations
 //----------------------------------------------------------------------------
 Error ASTNodeImporter::ImportDeclParts(
     NamedDecl *D, DeclContext *&DC, DeclContext *&LexicalDC,
     DeclarationName &Name, NamedDecl *&ToD, SourceLocation &Loc) {
   // Check if RecordDecl is in FunctionDecl parameters to avoid infinite loop.
   // example: int struct_in_proto(struct data_t{int a;int b;} *d);
ExpectedDecl ASTNodeImporter::VisitObjCTypeParamDecl(ObjCTypeParamDecl *D) {
   ObjCTypeParamDecl *Result;
   if (GetImportedOrCreateDecl(
           Result, D, Importer.getToContext(), DC, D->getVariance(),
           ToVarianceLoc, D->getIndex(),
           ToLocation, Name.getAsIdentifierInfo(),
           ToColonLoc, ToTypeSourceInfo))
     return Result;
 
+  // Only import 'ObjCTypeParamType' after the decl is created.
+  auto ToTypeForDecl = importChecked(Err, D->getTypeForDecl());
+  if (Err)
+    return std::move(Err);
+  Result->setTypeForDecl(ToTypeForDecl);
   Result->setLexicalDeclContext(LexicalDC);
   return Result;
 }
 
 ExpectedDecl ASTNodeImporter::VisitObjCCategoryDecl(ObjCCategoryDecl *D) {
   // Import the major distinguishing characteristics of a category.
   DeclContext *DC, *LexicalDC;
   DeclarationName Name;

clang/unittests/AST/ASTImporterObjCTest.cpp

for (clang::ObjCPropertyDecl *prop : ToClass->properties()) {
     ASSERT_TRUE(prop->getGetterMethodDecl());
     ASSERT_TRUE(prop->getSetterMethodDecl());
     // The getters/setters should be able to find the right associated property.
     ASSERT_EQ(prop->getGetterMethodDecl()->findPropertyDecl(), prop);
     ASSERT_EQ(prop->getSetterMethodDecl()->findPropertyDecl(), prop);
   }
 }
 
+TEST_P(ImportObjCDecl, ImportObjCTypeParamDecl) {
+  Decl *FromTU = getTuDecl(
+      R"(
+        @interface X <FirstParam: id>
+        @end
+      )",
+      Lang_OBJCXX, "input.mm");
+  auto *FromInterfaceDecl = FirstDeclMatcher<ObjCInterfaceDecl>().match(
+      FromTU, namedDecl(hasName("X")));
+  auto *FromTypeParamDecl =
+      FromInterfaceDecl->getTypeParamListAsWritten()->front();
+
+  auto *ToTypeParamDeclImported = Import(FromTypeParamDecl, Lang_OBJCXX);
+  ASSERT_TRUE(ToTypeParamDeclImported);
+}
+
 static const auto ObjCTestArrayForRunOptions =
     std::array<std::vector<std::string>, 2>{
         {std::vector<std::string>{"-fno-objc-arc"},
          std::vector<std::string>{"-fobjc-arc"}}};
 
 const auto ObjCTestValuesForRunOptions =
     ::testing::ValuesIn(ObjCTestArrayForRunOptions);
 
 INSTANTIATE_TEST_SUITE_P(ParameterizedTests, ImportObjCDecl,
                          ObjCTestValuesForRunOptions);

clang/unittests/AST/ASTImporterTest.cpp

 TEST_P(ImportType, ImportAtomicType) {
   MatchVerifier<Decl> Verifier;
   testImport(
       "void declToImport() { typedef _Atomic(int) a_int; }",
       Lang_CXX11, "", Lang_CXX11, Verifier,
       functionDecl(hasDescendant(typedefDecl(has(atomicType())))));
 }
 
+TEST_P(ImportType, ImportBitIntType) {
+  const AstTypeMatcher<BitIntType> bitIntType;
+  MatchVerifier<Decl> Verifier;
+  testImport("_BitInt(10) declToImport;", Lang_CXX11, "", Lang_CXX11, Verifier,
+             varDecl(hasType(bitIntType())));
+}
+
+TEST_P(ImportType, ImportDependentBitIntType) {
+  const AstTypeMatcher<DependentBitIntType> dependentBitIntType;
+  MatchVerifier<Decl> Verifier;
+  testImport("template<int Width> using declToImport = _BitInt(Width);",
+             Lang_CXX11, "", Lang_CXX11, Verifier,
+             typeAliasTemplateDecl(
+                 has(typeAliasDecl(hasType(dependentBitIntType())))));
+}
+
+TEST_P(ImportType, ImportDependentAddressSpaceType) {
+  const AstTypeMatcher<DependentAddressSpaceType> dependentAddressSpaceType;
+  MatchVerifier<Decl> Verifier;
+  testImport(
+      R"(
+        template<typename T, int AddrSpace>
+        using declToImport = T __attribute__((address_space(AddrSpace)));
+      )",
+      Lang_CXX11, "", Lang_CXX11, Verifier,
+      typeAliasTemplateDecl(
+          has(typeAliasDecl(hasType(dependentAddressSpaceType())))));
+}
+
+TEST_P(ImportType, ImportVectorType) {
+  const AstTypeMatcher<VectorType> vectorType;
+  MatchVerifier<Decl> Verifier;
+  testImport("typedef int __attribute__((vector_size(12))) declToImport;",
+             Lang_CXX11, "", Lang_CXX11, Verifier,
+             typedefDecl(hasType(vectorType())));
+}
+
+TEST_P(ImportType, ImportDependentVectorType) {
+  const AstTypeMatcher<DependentVectorType> dependentVectorType;
+  MatchVerifier<Decl> Verifier;
+  testImport(
+      R"(
+        template<typename T, int Size>
+        using declToImport = T __attribute__((vector_size(Size)));
+      )",
+      Lang_CXX11, "", Lang_CXX11, Verifier,
+      typeAliasTemplateDecl(
+          has(typeAliasDecl(hasType(dependentVectorType())))));
+}
+
+struct ImportOpenCLPipe : ImportType {
+  std::vector<std::string> getExtraArgs() const override {
+    return {"-x", "cl", "-cl-no-stdinc", "-cl-std=CL2.0"};
+  }
+};
+
+TEST_P(ImportOpenCLPipe, ImportPipeType) {
+  const AstTypeMatcher<PipeType> pipeType;
+  MatchVerifier<Decl> Verifier;
+  testImport("typedef pipe int declToImport;", Lang_OpenCL, "", Lang_OpenCL,
+             Verifier, typedefDecl(hasType(pipeType())));
+}
+
+struct ImportMatrixType : ImportType {
+  std::vector<std::string> getExtraArgs() const override {
+    return {"-fenable-matrix"};
+  }
+};
+
+TEST_P(ImportMatrixType, ImportConstantMatrixType) {
+  const AstTypeMatcher<ConstantMatrixType> constantMatrixType;
+  MatchVerifier<Decl> Verifier;
+  testImport("typedef int __attribute__((matrix_type(5, 5))) declToImport;",
+             Lang_CXX11, "", Lang_CXX11, Verifier,
+             typedefDecl(hasType(constantMatrixType())));
+}
+
+TEST_P(ImportMatrixType, ImportDependentSizedMatrixType) {
+  const AstTypeMatcher<DependentSizedMatrixType> dependentSizedMatrixType;
+  MatchVerifier<Decl> Verifier;
+  testImport(
+      R"(
+        template<typename T, int Rows, int Cols>
+        using declToImport = T __attribute__((matrix_type(Rows, Cols)));
+      )",
+      Lang_CXX11, "", Lang_CXX11, Verifier,
+      typeAliasTemplateDecl(
+          has(typeAliasDecl(hasType(dependentSizedMatrixType())))));
+}
+
 TEST_P(ImportType, ImportUsingType) {
   MatchVerifier<Decl> Verifier;
   testImport("struct C {};"
              "void declToImport() { using ::C; new C{}; }",
              Lang_CXX11, "", Lang_CXX11, Verifier,
              functionDecl(hasDescendant(cxxNewExpr(hasType(pointerType(
                  pointee(elaboratedType(namesType(usingType())))))))));
 }