[AST] Add RParen loc for decltype AutoTypeloc.

[thakis]

Looks like this breaks tests: http://45.33.8.238/linux/64726/step_8.txt

[hokein]

In D116919#3231246, @thakis wrote:

Looks like this breaks tests: http://45.33.8.238/linux/64726/step_8.txt

sorry, looking.

[mstorsjo]

The relanded version in rG41fbdfa4d5601cccbcdc0ded8ef35190d502f7f3 seems to be breaking some builds with PCH for me, failing asserts like this:

clang: ../tools/clang/include/clang/Basic/SourceLocation.h:135: clang::SourceLocation clang::SourceLocation::getLocWithOffset(clang::SourceLocation::IntTy) const: Assertion `((getOffset()+Offset) & MacroIDBit) == 0 && "offset overflow"' failed.

[fhahn]

In D116919#3236899, @mstorsjo wrote:

The relanded version in rG41fbdfa4d5601cccbcdc0ded8ef35190d502f7f3 seems to be breaking some builds with PCH for me, failing asserts like this:

clang: ../tools/clang/include/clang/Basic/SourceLocation.h:135: clang::SourceLocation clang::SourceLocation::getLocWithOffset(clang::SourceLocation::IntTy) const: Assertion `((getOffset()+Offset) & MacroIDBit) == 0 && "offset overflow"' failed.

This also breaks stage2 builds with debug info, e.g. https://green.lab.llvm.org/green/job/clang-stage2-Rthinlto/5088/console

Given that the bot has been red a while now, I reverted the commit eadb4cfeeff5 for now.

The crash should be reproducible when doing a stage2 build with debug info. It crashes, e.g. when building llvm-project/llvm/lib/Support/Timer.cpp

 /Users/buildslave/jenkins/workspace/clang-stage2-Rthinlto/host-compiler/bin/clang++  -DGTEST_HAS_RTTI=0 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/Users/buildslave/jenkins/workspace/clang-stage2-Rthinlto/llvm-project/llvm/lib/Support -Iinclude -I/Users/buildslave/jenkins/workspace/clang-stage2-Rthinlto/llvm-project/llvm/include -fno-stack-protector -fno-common -Wno-profile-instr-unprofiled -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -fmodules -fmodules-cache-path=/Users/buildslave/jenkins/workspace/clang-stage2-Rthinlto/clang-build/Build/module.cache -fcxx-modules -Xclang -fmodules-local-submodule-visibility -gmodules -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -flto=thin  -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk   -std=c++14  -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/Timer.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/Timer.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/Timer.cpp.o -c /Users/buildslave/jenkins/workspace/clang-stage2-Rthinlto/llvm-project/llvm/lib/Support/Timer.cpp
Assertion failed: (((getOffset()+Offset) & MacroIDBit) == 0 && "offset overflow"), function getLocWithOffset, file /Users/buildslave/jenkins/workspace/clang-stage1-RA/llvm-project/clang/include/clang/Basic/SourceLocation.h, line 135.

[hokein]

In D116919#3236899, @mstorsjo wrote:

The relanded version in rG41fbdfa4d5601cccbcdc0ded8ef35190d502f7f3 seems to be breaking some builds with PCH for me, failing asserts like this:

clang: ../tools/clang/include/clang/Basic/SourceLocation.h:135: clang::SourceLocation clang::SourceLocation::getLocWithOffset(clang::SourceLocation::IntTy) const: Assertion `((getOffset()+Offset) & MacroIDBit) == 0 && "offset overflow"' failed.

Thanks for raising this. It would be great if you can provide a reproduce case and a more detailed log. A failed PCH build usually indicates something wrong around the ASTReader/ASTWriter code, but I didn't spot out any issue there. At the mean time, I will try to investigate it.

[mstorsjo]

Ok, now I managed to make a standalone reproducer: https://martin.st/temp/pch-preproc.hxx and https://martin.st/temp/main-preproc.cpp

$ bin/clang -target x86_64-w64-mingw32 -w -x c++-header -c pch-preproc.hxx -o pch-preproc.hxx.pch -std=c++17 -O3
$ bin/clang -target x86_64-w64-mingw32 -w -Xclang -include-pch -Xclang pch-preproc.hxx.pch -c main-preproc.cpp -std=c++17 -O3

[thakis]

Since this is changing serialization format you might have to do something like https://reviews.llvm.org/rGb8b7a9dcdcbc as well (see https://reviews.llvm.org/D73202 for background). I doubt that's the cause of mstorsjo's repro given that LLVM_APPEND_VC_REV is on by default, but it seems like a good thing to do in the reland regardless.

(We also saw this crash on chromium's bots, https://crbug.com/1286675. Thanks for the revert, and for the reduced repro!)

[hokein]

Thanks, everyone!

I manage to figure out the cause, the crash was caused by an arbitrary RPareLoc -- we missed to set the RPareLoc in TreeTransform::TransformAutoType. I reland the patch in ab3f100bec03d72ecee947a323c51698d4b95207.

[mstorsjo]

In D116919#3247887, @hokein wrote:

Thanks, everyone!

I manage to figure out the cause, the crash was caused by an arbitrary RPareLoc -- we missed to set the RPareLoc in TreeTransform::TransformAutoType. I reland the patch in ab3f100bec03d72ecee947a323c51698d4b95207.

I rebuilt my full, non-reduced testcase now, and it seems like it built fine this time. Thanks!

[hokein]

In D116919#3248113, @mstorsjo wrote:

In D116919#3247887, @hokein wrote:

Thanks, everyone!

I manage to figure out the cause, the crash was caused by an arbitrary RPareLoc -- we missed to set the RPareLoc in TreeTransform::TransformAutoType. I reland the patch in ab3f100bec03d72ecee947a323c51698d4b95207.

I rebuilt my full, non-reduced testcase now, and it seems like it built fine this time. Thanks!

Glad to hear, and thanks for the verification!

clang-tools-extra/clangd/unittests/SelectionTests.cpp

Case Cases[] = {
 
       {
           R"cpp(
         int a;
         decltype([[^a]] + a) b;
         )cpp",
           "DeclRefExpr"},
       {"[[decltype^(1)]] b;", "DecltypeTypeLoc"}, // Not the VarDecl.
+      // decltype(auto) is an AutoTypeLoc!
+      {"[[de^cltype(a^uto)]] a = 1;", "AutoTypeLoc"},
 
       // Objective-C nullability attributes.
       {
           R"cpp(
             @interface I{}
             @property(nullable) [[^I]] *x;
             @end
           )cpp",

clang/include/clang/AST/TypeLoc.h

 
 struct AutoTypeLocInfo : TypeSpecLocInfo {
   NestedNameSpecifierLoc NestedNameSpec;
   SourceLocation TemplateKWLoc;
   SourceLocation ConceptNameLoc;
   NamedDecl *FoundDecl;
   SourceLocation LAngleLoc;
   SourceLocation RAngleLoc;
+
+  // For decltype(auto).
+  SourceLocation RParenLoc;
 };
 
 class AutoTypeLoc
     : public ConcreteTypeLoc<DeducedTypeLoc,
                              AutoTypeLoc,
                              AutoType,
                              AutoTypeLocInfo> {
 public:
   AutoTypeKeyword getAutoKeyword() const {
     return getTypePtr()->getKeyword();
   }
 
+  bool isDecltypeAuto() const { return getTypePtr()->isDecltypeAuto(); }
+  SourceLocation getRParenLoc() const { return getLocalData()->RParenLoc; }
+  void setRParenLoc(SourceLocation Loc) { getLocalData()->RParenLoc = Loc; }
+
   bool isConstrained() const {
     return getTypePtr()->isConstrained();
   }
 
   const NestedNameSpecifierLoc &getNestedNameSpecifierLoc() const {
     return getLocalData()->NestedNameSpec;
   }
 
public:
   }
 
   TemplateArgumentLoc getArgLoc(unsigned i) const {
     return TemplateArgumentLoc(getTypePtr()->getTypeConstraintArguments()[i],
                                getArgLocInfo(i));
   }
 
   SourceRange getLocalSourceRange() const {
-    return{
-        isConstrained()
+    return {isConstrained()
                 ? (getNestedNameSpecifierLoc()
                        ? getNestedNameSpecifierLoc().getBeginLoc()
-               : (getTemplateKWLoc().isValid()
-                  ? getTemplateKWLoc()
+                       : (getTemplateKWLoc().isValid() ? getTemplateKWLoc()
                                                        : getConceptNameLoc()))
                 : getNameLoc(),
-        getNameLoc()
-    };
+            isDecltypeAuto() ? getRParenLoc() : getNameLoc()};
   }
 
   void copy(AutoTypeLoc Loc) {
     unsigned size = getFullDataSize();
     assert(size == Loc.getFullDataSize());
     memcpy(Data, Loc.Data, size);
   }
 

clang/lib/AST/TypeLoc.cpp

 
 void AutoTypeLoc::initializeLocal(ASTContext &Context, SourceLocation Loc) {
   setNestedNameSpecifierLoc(NestedNameSpecifierLoc());
   setTemplateKWLoc(Loc);
   setConceptNameLoc(Loc);
   setFoundDecl(nullptr);
   setRAngleLoc(Loc);
   setLAngleLoc(Loc);
+  setRParenLoc(Loc);
   TemplateSpecializationTypeLoc::initializeArgLocs(Context, getNumArgs(),
                                                    getTypePtr()->getArgs(),
                                                    getArgInfos(), Loc);
   setNameLoc(Loc);
 }
 
 
 namespace {

clang/lib/Sema/SemaType.cpp

 #include "clang/AST/ASTStructuralEquivalence.h"
 #include "clang/AST/CXXInheritance.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/Expr.h"
 #include "clang/AST/TypeLoc.h"
 #include "clang/AST/TypeLocVisitor.h"
 #include "clang/Basic/PartialDiagnostic.h"
+#include "clang/Basic/Specifiers.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Sema/DeclSpec.h"
 #include "clang/Sema/DelayedDiagnostic.h"
 #include "clang/Sema/Lookup.h"
 #include "clang/Sema/ParsedTemplate.h"
 #include "clang/Sema/ScopeInfo.h"
 #include "clang/Sema/SemaInternal.h"
void VisitDependentTemplateSpecializationTypeLoc(
           TInfo->getTypeLoc().castAs<DependentTemplateSpecializationTypeLoc>());
     }
     void VisitAutoTypeLoc(AutoTypeLoc TL) {
       assert(DS.getTypeSpecType() == TST_auto ||
              DS.getTypeSpecType() == TST_decltype_auto ||
              DS.getTypeSpecType() == TST_auto_type ||
              DS.getTypeSpecType() == TST_unspecified);
       TL.setNameLoc(DS.getTypeSpecTypeLoc());
+      if (DS.getTypeSpecType() == TST_decltype_auto)
+        TL.setRParenLoc(DS.getTypeofParensRange().getEnd());
       if (!DS.isConstrainedAuto())
         return;
       TemplateIdAnnotation *TemplateId = DS.getRepAsTemplateId();
       if (!TemplateId)
         return;
       if (DS.getTypeSpecScope().isNotEmpty())
         TL.setNestedNameSpecifierLoc(
             DS.getTypeSpecScope().getWithLocInContext(Context));

clang/lib/Serialization/ASTReader.cpp

if (Reader.readBool()) {
     TL.setConceptNameLoc(readSourceLocation());
     TL.setFoundDecl(Reader.readDeclAs<NamedDecl>());
     TL.setLAngleLoc(readSourceLocation());
     TL.setRAngleLoc(readSourceLocation());
     for (unsigned i = 0, e = TL.getNumArgs(); i != e; ++i)
       TL.setArgLocInfo(i, Reader.readTemplateArgumentLocInfo(
                               TL.getTypePtr()->getArg(i).getKind()));
   }
+  if (Reader.readBool())
+    TL.setRParenLoc(readSourceLocation());
 }
 
 void TypeLocReader::VisitDeducedTemplateSpecializationTypeLoc(
     DeducedTemplateSpecializationTypeLoc TL) {
   TL.setTemplateNameLoc(readSourceLocation());
 }
 
 void TypeLocReader::VisitRecordTypeLoc(RecordTypeLoc TL) {

clang/lib/Serialization/ASTWriter.cpp

if (TL.isConstrained()) {
     Record.AddSourceLocation(TL.getConceptNameLoc());
     Record.AddDeclRef(TL.getFoundDecl());
     Record.AddSourceLocation(TL.getLAngleLoc());
     Record.AddSourceLocation(TL.getRAngleLoc());
     for (unsigned I = 0; I < TL.getNumArgs(); ++I)
       Record.AddTemplateArgumentLocInfo(TL.getTypePtr()->getArg(I).getKind(),
                                         TL.getArgLocInfo(I));
   }
+  Record.push_back(TL.isDecltypeAuto());
+  if (TL.isDecltypeAuto())
+    Record.AddSourceLocation(TL.getRParenLoc());
 }
 
 void TypeLocWriter::VisitDeducedTemplateSpecializationTypeLoc(
     DeducedTemplateSpecializationTypeLoc TL) {
   Record.AddSourceLocation(TL.getTemplateNameLoc());
 }
 
 void TypeLocWriter::VisitRecordTypeLoc(RecordTypeLoc TL) {

clang/test/AST/ast-dump-template-decls-json.cpp

 // CHECK-NEXT:      },
 // CHECK-NEXT:      "range": {
 // CHECK-NEXT:       "begin": {
 // CHECK-NEXT:        "offset": 705,
 // CHECK-NEXT:        "col": 11,
 // CHECK-NEXT:        "tokLen": 8
 // CHECK-NEXT:       },
 // CHECK-NEXT:       "end": {
-// CHECK-NEXT:        "offset": 705,
-// CHECK-NEXT:        "col": 11,
-// CHECK-NEXT:        "tokLen": 8
+// CHECK-NEXT:        "offset": 718,
+// CHECK-NEXT:        "col": 24,
+// CHECK-NEXT:        "tokLen": 1
 // CHECK-NEXT:       }
 // CHECK-NEXT:      },
 // CHECK-NEXT:      "type": {
 // CHECK-NEXT:       "qualType": "decltype(auto)"
 // CHECK-NEXT:      },
 // CHECK-NEXT:      "depth": 0,
 // CHECK-NEXT:      "index": 0
 // CHECK-NEXT:     },

clang/test/AST/ast-dump-template-decls.cpp

 
 template <auto>
 // CHECK: ClassTemplateDecl 0x{{[^ ]*}} <line:[[@LINE-1]]:1, line:[[@LINE+2]]:11> col:8 T
 // CHECK-NEXT: NonTypeTemplateParmDecl 0x{{[^ ]*}} <line:[[@LINE-2]]:11> col:15 'auto' depth 0 index 0
 struct T {};
 
 template <decltype(auto)>
 // CHECK: ClassTemplateDecl 0x{{[^ ]*}} <line:[[@LINE-1]]:1, line:[[@LINE+2]]:11> col:8 U
-// CHECK-NEXT: NonTypeTemplateParmDecl 0x{{[^ ]*}} <line:[[@LINE-2]]:11> col:25 'decltype(auto)' depth 0 index 0
+// CHECK-NEXT: NonTypeTemplateParmDecl 0x{{[^ ]*}} <line:[[@LINE-2]]:11, col:24> col:25 'decltype(auto)' depth 0 index 0
 struct U {};
 
 template <typename Ty>
 // CHECK: ClassTemplateDecl 0x{{[^ ]*}} <line:[[@LINE-1]]:1, line:[[@LINE+7]]:1> line:[[@LINE+2]]:8 V
 // CHECK-NEXT: TemplateTypeParmDecl 0x{{[^ ]*}} <line:[[@LINE-2]]:11, col:20> col:20 typename depth 0 index 0 Ty
 struct V {
   template <typename Uy>
   // CHECK: FunctionTemplateDecl 0x{{[^ ]*}} <line:[[@LINE-1]]:3, line:[[@LINE+2]]:10> col:8 f

clang/unittests/AST/SourceLocationTest.cpp

TEST(TypeLoc, DecltypeTypeLocRange) {
   };
   const auto *Target1 = MatchedLocs[0].getNodeAs<DecltypeTypeLoc>("target");
   verify(Target1->getSourceRange(), Code.range("full1"));
 
   const auto *Target2 = MatchedLocs[1].getNodeAs<DecltypeTypeLoc>("target");
   verify(Target2->getSourceRange(), Code.range("full2"));
 }
 
+TEST(TypeLoc, AutoTypeLocRange) {
+  RangeVerifier<TypeLoc> Verifier;
+  Verifier.expectRange(1, 1, 1, 14);
+  EXPECT_TRUE(Verifier.match("decltype(auto) a = 1;", typeLoc(loc(autoType())),
+                             Lang_CXX11));
+}
+
 TEST(TypeLoc, LongDoubleRange) {
   RangeVerifier<TypeLoc> Verifier;
   Verifier.expectRange(1, 1, 1, 6);
   EXPECT_TRUE(Verifier.match("long double a;", typeLoc()));
 }
 
 TEST(TypeLoc, DoubleLongRange) {
   RangeVerifier<TypeLoc> Verifier;