[clangd] Implement hover for "auto" and "decltype"

[malaperle]

Summary

This allows hovering on keywords that refer to deduced types.
This should cover most useful cases. Not covered:

• auto template parameters: Since this can be instantiated with many types,

it would not be practical to show the types.

• Structured binding: This could be done later to show multiple deduced types

in the hover.

• auto:: (part of concepts): Outside the scope of this patch.

Signed-off-by: Marc-Andre Laperle <[e-mail]>

[malaperle]

Rebased.

[sammccall]

This is so great, thank you!

clangd/XRefs.cpp

544 ↗

(On Diff #152958)

maybe a bit more context: "we use this to display the actual type when hovering over an auto keyword or decltype() expression"

546 ↗

(On Diff #152958)

nit: SourceLocation is 32 bits, just copy it?
(Reference seems fine here but less obvious than it could be)

559 ↗

(On Diff #152958)

out of curiosity, why not implement VisitTypeLoc and handle all the cases where it turns out to be auto etc?
Even for auto& I'd expect the inner auto to have a TypeLoc you could visit, saving the trouble of unwrapping.

(I'm probably wrong about all this, I don't know the AST well. But I'd like to learn!)

640 ↗

(On Diff #152958)

This is a really nice standalone piece of logic.
It might be slightly cleaner to put it in AST.h with fine-grained unit-tests there, and just smoke test it here. That way this file is more focused on *what* the features do, and the lower layer has details about how they work.
(And if we find another use for this, like a "replace with deduced type" refactoring, we could easily reuse it).

That said, the current stuff in this file doesn't exhibit that layering/separation today. Happy if you prefer to land it here, and I/someone may do such a refactoring in the future.

656 ↗

(On Diff #152958)

This will end up deserializing the whole preamble I think, which is slow.
The fix is just to traverse from each of the top-level *non-preamble* decls instead, i.e. AST.getLocalTopLevelDecls().

(we've fixed this bug many times so far, it would be nice if there's a systematic way to fix/catch this but I can't think of one)

unittests/clangd/TestTU.h

47 ↗

(On Diff #152958)

headerSymbols() and index() also depend on these flags - can you make these extra args a member instead? (can be public, just like Code)

[malaperle]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

From what I saw, there are actually two different AutoType* for each textual "auto". The AutoType* containing the deduced type does not get visited via a typeloc. It's not entirely clear to me why since I don't know the AST well either. I was thinking maybe the first is created when the type is not deduced yet and later on, then the rest of the function or expression is parsed, a second one with the actual type deduced is created. If I look at the code paths where they are created, it seems like this is roughly what's happening. The first one is created when the declarator is parsed (no deduced type yet) and the second is created when the expression of the initializer (or return statement) is evaluated and the type is then deduced. The visitor only visits the first one's typeloc. I don't think I'm knowledgeable enough to say whether or not that's a bug but it seems on purpose that it is modelled this way. Although it would be much nicer to only have to visit typelocs...

640 ↗

(On Diff #152958)

Good idea! The refactoring would be a neat feature too. I think I'd prefer to leave this refactoring for a bit later since this patch looks like it's close to ready to go.

[malaperle]

Address comments.

[sammccall]

All sounds good to me.

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

The AutoType* containing the deduced type does not get visited via a typeloc

Ah, OK.
Could you add a high level comment (maybe on the class) saying this is the reason for the implementation? Otherwise as a reader I'll think "this seems unneccesarily complicated" but not understand why.

@klimek Can you shed any light on this?

[klimek]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

Can't you go from AutoTypeLoc -> AutoType -> getDeducedType()?

[malaperle]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

The visitor doesn't visit the AutoTypeLoc that has the deduced type. In fact, there are two AutoType* instances. I'm not sure that's is a bug that there are two AutoType*, or if not visiting both AutoTypeLoc is a bug...or neither.

[klimek]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

+Richard Smith:

This is weird. If I just create a minimal example:

int f() {
  auto i = f();
  return i;
}

I only get the undeduced auto type - Richard, in which cases are auto-typed being deduced? The AST dump doens't give an indication that there was an auto involved at all. Is this the famous syntactic vs. smenatic form problem? Do we have a backlink between the AutoTypeLoc and the deduced type somewhere?

[malaperle]

Add comment about AutoTypeLoc work-around.

[klimek]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

Given that Richard is known to have ~1 month ping times now and then I think it's fine to land this with a FIXME above to figure out how to represent this better in the AST. I'd still say it's a missing feature in the AST :)

[malaperle]

Tweak comment with FIXME.

[malaperle]

clangd/XRefs.cpp

559 ↗

(On Diff #152958)

Thanks! I'm looking forward to simplifying this code when the AST is improved.

clang-tools-extra/trunk/clangd/XRefs.cpp

 //===--- XRefs.cpp ----------------------------------------------*- C++-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===---------------------------------------------------------------------===//
 #include "XRefs.h"
 #include "AST.h"
 #include "Logger.h"
 #include "SourceCode.h"
 #include "URI.h"
 #include "clang/AST/DeclTemplate.h"
+#include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/Index/IndexDataConsumer.h"
 #include "clang/Index/IndexingAction.h"
 #include "clang/Index/USRGeneration.h"
 #include "llvm/Support/Path.h"
 namespace clang {
 namespace clangd {
 using namespace llvm;
 namespace {
static Hover getHoverContents(const Decl *D) {
   D->print(OS, Policy);
 
   OS.flush();
 
   H.contents.value += DeclText;
   return H;
 }
 
+/// Generate a \p Hover object given the type \p T.
+static Hover getHoverContents(QualType T, ASTContext &ASTCtx) {
+  Hover H;
+  std::string TypeText;
+  llvm::raw_string_ostream OS(TypeText);
+  PrintingPolicy Policy = PrintingPolicyForDecls(ASTCtx.getPrintingPolicy());
+  T.print(OS, Policy);
+  OS.flush();
+  H.contents.value += TypeText;
+  return H;
+}
+
 /// Generate a \p Hover object given the macro \p MacroInf.
 static Hover getHoverContents(StringRef MacroName) {
   Hover H;
 
   H.contents.value = "#define ";
   H.contents.value += MacroName;
 
   return H;
 }
 
+namespace {
+/// Computes the deduced type at a given location by visiting the relevant
+/// nodes. We use this to display the actual type when hovering over an "auto"
+/// keyword or "decltype()" expression.
+/// FIXME: This could have been a lot simpler by visiting AutoTypeLocs but it
+/// seems that the AutoTypeLocs that can be visited along with their AutoType do
+/// not have the deduced type set. Instead, we have to go to the appropriate
+/// DeclaratorDecl/FunctionDecl and work our back to the AutoType that does have
+/// a deduced type set. The AST should be improved to simplify this scenario.
+class DeducedTypeVisitor : public RecursiveASTVisitor<DeducedTypeVisitor> {
+  SourceLocation SearchedLocation;
+  llvm::Optional<QualType> DeducedType;
+
+public:
+  DeducedTypeVisitor(SourceLocation SearchedLocation)
+      : SearchedLocation(SearchedLocation) {}
+
+  llvm::Optional<QualType> getDeducedType() { return DeducedType; }
+
+  // Handle auto initializers:
+  //- auto i = 1;
+  //- decltype(auto) i = 1;
+  //- auto& i = 1;
+  bool VisitDeclaratorDecl(DeclaratorDecl *D) {
+    if (!D->getTypeSourceInfo() ||
+        D->getTypeSourceInfo()->getTypeLoc().getLocStart() != SearchedLocation)
+      return true;
+
+    auto DeclT = D->getType();
+    // "auto &" is represented as a ReferenceType containing an AutoType
+    if (const ReferenceType *RT = dyn_cast<ReferenceType>(DeclT.getTypePtr()))
+      DeclT = RT->getPointeeType();
+
+    const AutoType *AT = dyn_cast<AutoType>(DeclT.getTypePtr());
+    if (AT && !AT->getDeducedType().isNull()) {
+      // For auto, use the underlying type because the const& would be
+      // represented twice: written in the code and in the hover.
+      // Example: "const auto I = 1", we only want "int" when hovering on auto,
+      // not "const int".
+      //
+      // For decltype(auto), take the type as is because it cannot be written
+      // with qualifiers or references but its decuded type can be const-ref.
+      DeducedType = AT->isDecltypeAuto() ? DeclT : DeclT.getUnqualifiedType();
+    }
+    return true;
+  }
+
+  // Handle auto return types:
+  //- auto foo() {}
+  //- auto& foo() {}
+  //- auto foo() -> decltype(1+1) {}
+  //- operator auto() const { return 10; }
+  bool VisitFunctionDecl(FunctionDecl *D) {
+    if (!D->getTypeSourceInfo())
+      return true;
+    // Loc of auto in return type (c++14).
+    auto CurLoc = D->getReturnTypeSourceRange().getBegin();
+    // Loc of "auto" in operator auto()
+    if (CurLoc.isInvalid() && dyn_cast<CXXConversionDecl>(D))
+      CurLoc = D->getTypeSourceInfo()->getTypeLoc().getBeginLoc();
+    // Loc of "auto" in function with traling return type (c++11).
+    if (CurLoc.isInvalid())
+      CurLoc = D->getSourceRange().getBegin();
+    if (CurLoc != SearchedLocation)
+      return true;
+
+    auto T = D->getReturnType();
+    // "auto &" is represented as a ReferenceType containing an AutoType.
+    if (const ReferenceType *RT = dyn_cast<ReferenceType>(T.getTypePtr()))
+      T = RT->getPointeeType();
+
+    const AutoType *AT = dyn_cast<AutoType>(T.getTypePtr());
+    if (AT && !AT->getDeducedType().isNull()) {
+      DeducedType = T.getUnqualifiedType();
+    } else { // auto in a trailing return type just points to a DecltypeType.
+      const DecltypeType *DT = dyn_cast<DecltypeType>(T.getTypePtr());
+      if (!DT->getUnderlyingType().isNull())
+        DeducedType = DT->getUnderlyingType();
+    }
+    return true;
+  }
+
+  // Handle non-auto decltype, e.g.:
+  // - auto foo() -> decltype(expr) {}
+  // - decltype(expr);
+  bool VisitDecltypeTypeLoc(DecltypeTypeLoc TL) {
+    if (TL.getBeginLoc() != SearchedLocation)
+      return true;
+
+    // A DecltypeType's underlying type can be another DecltypeType! E.g.
+    //  int I = 0;
+    //  decltype(I) J = I;
+    //  decltype(J) K = J;
+    const DecltypeType *DT = dyn_cast<DecltypeType>(TL.getTypePtr());
+    while (DT && !DT->getUnderlyingType().isNull()) {
+      DeducedType = DT->getUnderlyingType();
+      DT = dyn_cast<DecltypeType>(DeducedType->getTypePtr());
+    }
+    return true;
+  }
+};
+} // namespace
+
+/// Retrieves the deduced type at a given location (auto, decltype).
+llvm::Optional<QualType> getDeducedType(ParsedAST &AST,
+                                        SourceLocation SourceLocationBeg) {
+  Token Tok;
+  auto &ASTCtx = AST.getASTContext();
+  // Only try to find a deduced type if the token is auto or decltype.
+  if (!SourceLocationBeg.isValid() ||
+      Lexer::getRawToken(SourceLocationBeg, Tok, ASTCtx.getSourceManager(),
+                         ASTCtx.getLangOpts(), false) ||
+      !Tok.is(tok::raw_identifier)) {
+    return {};
+  }
+  AST.getPreprocessor().LookUpIdentifierInfo(Tok);
+  if (!(Tok.is(tok::kw_auto) || Tok.is(tok::kw_decltype)))
+    return {};
+
+  DeducedTypeVisitor V(SourceLocationBeg);
+  for (Decl *D : AST.getLocalTopLevelDecls())
+    V.TraverseDecl(D);
+  return V.getDeducedType();
+}
+
 Optional<Hover> getHover(ParsedAST &AST, Position Pos) {
   const SourceManager &SourceMgr = AST.getASTContext().getSourceManager();
   SourceLocation SourceLocationBeg =
       getBeginningOfIdentifier(AST, Pos, SourceMgr.getMainFileID());
   // Identified symbols at a specific position.
   auto Symbols = getSymbolAtPosition(AST, SourceLocationBeg);
 
   if (!Symbols.Macros.empty())
     return getHoverContents(Symbols.Macros[0].Name);
 
   if (!Symbols.Decls.empty())
     return getHoverContents(Symbols.Decls[0]);
 
+  auto DeducedType = getDeducedType(AST, SourceLocationBeg);
+  if (DeducedType && !DeducedType->isNull())
+    return getHoverContents(*DeducedType, AST.getASTContext());
+
   return None;
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/trunk/unittests/clangd/TestTU.h

struct TestTU {
   // The code to be compiled.
   std::string Code;
   std::string Filename = "TestTU.cpp";
 
   // Define contents of a header which will be implicitly included by Code.
   std::string HeaderCode;
   std::string HeaderFilename = "TestTU.h";
 
+  // Extra arguments for the compiler invocation.
+  std::vector<const char *> ExtraArgs;
+
   ParsedAST build() const;
   SymbolSlab headerSymbols() const;
   std::unique_ptr<SymbolIndex> index() const;
 };
 
 // Look up an index symbol by qualified name, which must be unique.
 const Symbol &findSymbol(const SymbolSlab &, llvm::StringRef QName);
 // Look up an AST symbol by qualified name, which must be unique and top-level.
 const NamedDecl &findDecl(ParsedAST &AST, llvm::StringRef QName);
 // Look up a main-file AST symbol by unqualified name, which must be unique.
 const NamedDecl &findAnyDecl(ParsedAST &AST, llvm::StringRef Name);
 
 } // namespace clangd
 } // namespace clang
 #endif

clang-tools-extra/trunk/unittests/clangd/TestTU.cpp

std::string FullFilename = testPath(Filename),
               FullHeaderName = testPath(HeaderFilename);
   std::vector<const char *> Cmd = {"clang", FullFilename.c_str()};
   // FIXME: this shouldn't need to be conditional, but it breaks a
   // GoToDefinition test for some reason (getMacroArgExpandedLocation fails).
   if (!HeaderCode.empty()) {
     Cmd.push_back("-include");
     Cmd.push_back(FullHeaderName.c_str());
   }
+  Cmd.insert(Cmd.end(), ExtraArgs.begin(), ExtraArgs.end());
   auto AST = ParsedAST::Build(
       createInvocationFromCommandLine(Cmd), nullptr,
       MemoryBuffer::getMemBufferCopy(Code),
       std::make_shared<PCHContainerOperations>(),
       buildTestFS({{FullFilename, Code}, {FullHeaderName, HeaderCode}}));
   if (!AST.hasValue()) {
     ADD_FAILURE() << "Failed to build code:\n" << Code;
     llvm_unreachable("Failed to build TestTU!");

clang-tools-extra/trunk/unittests/clangd/XRefsTests.cpp

 TEST(Hover, All) {
   struct OneTest {
     StringRef Input;
     StringRef ExpectedHover;
   };
 
   OneTest Tests[] = {
       {
+          R"cpp(// No hover
+            ^int main() {
+            }
+          )cpp",
+          "",
+      },
+      {
           R"cpp(// Local variable
             int main() {
               int bonjour;
               ^bonjour = 2;
               int test1 = bonjour;
             }
           )cpp",
           "Declared in function main\n\nint bonjour",
OneTest Tests[] = {
       {
           R"cpp(// Nothing
             void foo() {
               ^
             }
           )cpp",
           "",
       },
+      {
+          R"cpp(// Simple initialization with auto
+            void foo() {
+              ^auto i = 1;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Simple initialization with const auto
+            void foo() {
+              const ^auto i = 1;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Simple initialization with const auto&
+            void foo() {
+              const ^auto& i = 1;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Simple initialization with auto&
+            void foo() {
+              ^auto& i = 1;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Auto with initializer list.
+            namespace std
+            {
+              template<class _E>
+              class initializer_list {};
+            }
+            void foo() {
+              ^auto i = {1,2};
+            }
+          )cpp",
+          "class std::initializer_list<int>",
+      },
+      {
+          R"cpp(// User defined conversion to auto
+            struct Bar {
+              operator ^auto() const { return 10; }
+            };
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Simple initialization with decltype(auto)
+            void foo() {
+              ^decltype(auto) i = 1;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// Simple initialization with const decltype(auto)
+            void foo() {
+              const int j = 0;
+              ^decltype(auto) i = j;
+            }
+          )cpp",
+          "const int",
+      },
+      {
+          R"cpp(// Simple initialization with const& decltype(auto)
+            void foo() {
+              int k = 0;
+              const int& j = k;
+              ^decltype(auto) i = j;
+            }
+          )cpp",
+          "const int &",
+      },
+      {
+          R"cpp(// Simple initialization with & decltype(auto)
+            void foo() {
+              int k = 0;
+              int& j = k;
+              ^decltype(auto) i = j;
+            }
+          )cpp",
+          "int &",
+      },
+      {
+          R"cpp(// decltype with initializer list: nothing
+            namespace std
+            {
+              template<class _E>
+              class initializer_list {};
+            }
+            void foo() {
+              ^decltype(auto) i = {1,2};
+            }
+          )cpp",
+          "",
+      },
+      {
+          R"cpp(// auto function return with trailing type
+            struct Bar {};
+            ^auto test() -> decltype(Bar()) {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// trailing return type
+            struct Bar {};
+            auto test() -> ^decltype(Bar()) {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// auto in function return
+            struct Bar {};
+            ^auto test() {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// auto& in function return
+            struct Bar {};
+            ^auto& test() {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// const auto& in function return
+            struct Bar {};
+            const ^auto& test() {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// decltype(auto) in function return
+            struct Bar {};
+            ^decltype(auto) test() {
+              return Bar();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// decltype(auto) reference in function return
+            struct Bar {};
+            ^decltype(auto) test() {
+              int a;
+              return (a);
+            }
+          )cpp",
+          "int &",
+      },
+      {
+          R"cpp(// decltype lvalue reference
+            void foo() {
+              int I = 0;
+              ^decltype(I) J = I;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// decltype lvalue reference
+            void foo() {
+              int I= 0;
+              int &K = I;
+              ^decltype(K) J = I;
+            }
+          )cpp",
+          "int &",
+      },
+      {
+          R"cpp(// decltype lvalue reference parenthesis
+            void foo() {
+              int I = 0;
+              ^decltype((I)) J = I;
+            }
+          )cpp",
+          "int &",
+      },
+      {
+          R"cpp(// decltype rvalue reference
+            void foo() {
+              int I = 0;
+              ^decltype(static_cast<int&&>(I)) J = static_cast<int&&>(I);
+            }
+          )cpp",
+          "int &&",
+      },
+      {
+          R"cpp(// decltype rvalue reference function call
+            int && bar();
+            void foo() {
+              int I = 0;
+              ^decltype(bar()) J = bar();
+            }
+          )cpp",
+          "int &&",
+      },
+      {
+          R"cpp(// decltype of function with trailing return type.
+            struct Bar {};
+            auto test() -> decltype(Bar()) {
+              return Bar();
+            }
+            void foo() {
+              ^decltype(test()) i = test();
+            }
+          )cpp",
+          "struct Bar",
+      },
+      {
+          R"cpp(// decltype of var with decltype.
+            void foo() {
+              int I = 0;
+              decltype(I) J = I;
+              ^decltype(J) K = J;
+            }
+          )cpp",
+          "int",
+      },
+      {
+          R"cpp(// structured binding. Not supported yet
+            struct Bar {};
+            void foo() {
+              Bar a[2];
+              ^auto [x,y] = a;
+            }
+          )cpp",
+          "",
+      },
+      {
+          R"cpp(// Template auto parameter. Nothing (Not useful).
+            template<^auto T>
+            void func() {
+            }
+            void foo() {
+               func<1>();
+            }
+          )cpp",
+          "",
+      },
   };
 
   for (const OneTest &Test : Tests) {
     Annotations T(Test.Input);
-    auto AST = TestTU::withCode(T.code()).build();
+    TestTU TU = TestTU::withCode(T.code());
+    TU.ExtraArgs.push_back("-std=c++17");
+    auto AST = TU.build();
     if (auto H = getHover(AST, T.point())) {
       EXPECT_NE("", Test.ExpectedHover) << Test.Input;
-      EXPECT_EQ(H->contents.value, Test.ExpectedHover) << Test.Input;
+      EXPECT_EQ(H->contents.value, Test.ExpectedHover.str()) << Test.Input;
     } else
-      EXPECT_EQ("", Test.ExpectedHover) << Test.Input;
+      EXPECT_EQ("", Test.ExpectedHover.str()) << Test.Input;
   }
 }
 
 TEST(GoToInclude, All) {
   MockFSProvider FS;
   IgnoreDiagnostics DiagConsumer;
   MockCompilationDatabase CDB;
   ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());