This is an archive of the discontinued LLVM Phabricator instance.

[clangd] Code hover for Clangd
ClosedPublic

Authored by Nebiroth on Jul 26 2017, 8:24 AM.

Diff Detail

Repository
rL LLVM

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
malaperle added inline comments.Nov 28 2017, 8:47 PM
clangd/ClangdUnit.cpp
1117 ↗(On Diff #124116)

I think we need to simplify this part a bit. I'll try to find a way. Feel free to wait until more comments before updating.

This revision now requires changes to proceed.Nov 28 2017, 8:47 PM
malaperle added inline comments.Nov 28 2017, 8:49 PM
clangd/ClangdUnit.cpp
1173 ↗(On Diff #124116)

I think it has to do with macro expansions, the plot thickens..

malaperle added inline comments.Nov 29 2017, 9:40 AM
clangd/ClangdUnit.cpp
1173 ↗(On Diff #124116)

ObjCMethodDecl is forward-declared in astfwd.h as part of a macro expansion. We have to either use the expansion loc or spelling loc to get the appropriate file entry. Probably expansion loc makes more sense but I'll test a bit.

malaperle added inline comments.Nov 29 2017, 10:23 AM
clangd/Protocol.h
453 ↗(On Diff #124116)

The doc should use /// like the others

malaperle added inline comments.Nov 29 2017, 12:39 PM
clangd/ClangdUnit.cpp
1135 ↗(On Diff #124116)

Not sure why but I don't get the comments when I hover on "push_back" but I do get it on many other methods. This can be investigated separately I think. It could be a bug in getRawCommentForDeclNoCache

malaperle added inline comments.Nov 29 2017, 1:08 PM
clangd/ClangdUnit.h
320 ↗(On Diff #124116)

I think this can be only in the source file, in a an anonymous namespace.

malaperle added inline comments.Nov 29 2017, 1:53 PM
clangd/ClangdUnit.cpp
1029 ↗(On Diff #124116)
const FileEntry *F =
        SourceMgr.getFileEntryForID(SourceMgr.getFileID(LocStart));
if (!F)
  return llvm::errc::no_such_file_or_directory;
1039 ↗(On Diff #124116)

FE can be null. How about returning a llvm::ErrorOr ?

1242 ↗(On Diff #124116)

FE can be null. How about returning a llvm::ErrorOr ?

Nebiroth marked 20 inline comments as done.Nov 29 2017, 2:54 PM
Nebiroth updated this revision to Diff 124833.Nov 29 2017, 3:14 PM
Nebiroth edited edge metadata.

Invalid FileEntries now return llvm:ErrorOr

malaperle added inline comments.Nov 29 2017, 3:27 PM
clangd/ClangdUnit.cpp
1117 ↗(On Diff #124116)

Here's the version in which I tried to simplify this *a bit*. With the new ErrorOr checks as well.

Hover clangd::getHover(ParsedAST &AST, Position Pos, clangd::Logger &Logger) {
  const SourceManager &SourceMgr = AST.getASTContext().getSourceManager();
  const LangOptions &LangOpts = AST.getASTContext().getLangOpts();
  const FileEntry *FE = SourceMgr.getFileEntryForID(SourceMgr.getMainFileID());
  if (!FE)
    return Hover();

  SourceLocation SourceLocationBeg = getBeginningOfIdentifier(AST, Pos, FE);

  auto DeclMacrosFinder = std::make_shared<DeclarationAndMacrosFinder>(
      llvm::errs(), SourceLocationBeg, AST.getASTContext(),
      AST.getPreprocessor());
  index::IndexingOptions IndexOpts;
  IndexOpts.SystemSymbolFilter =
      index::IndexingOptions::SystemSymbolFilterKind::All;
  IndexOpts.IndexFunctionLocals = true;
  indexTopLevelDecls(AST.getASTContext(), AST.getTopLevelDecls(),
                     DeclMacrosFinder, IndexOpts);

  auto Decls = DeclMacrosFinder->takeDecls();
  if (!Decls.empty() && Decls[0]) {
    const Decl *LocationDecl = Decls[0];
    std::vector<MarkedString> HoverContents;

    // Compute scope as the first "section" of the hover.
    if (const NamedDecl *NamedD = dyn_cast<NamedDecl>(LocationDecl)) {
      std::string Scope = getScope(NamedD, AST.getASTContext().getLangOpts());
      if (!Scope.empty()) {
        MarkedString Info = {"", "C++", "In " + Scope};
        HoverContents.push_back(Info);
      }
    }

    // Adjust beginning of hover text depending on the presence of templates and comments.
    TemplateDecl * TDec = LocationDecl->getDescribedTemplate();
    const Decl * BeginDecl = TDec ? TDec : LocationDecl;
    SourceLocation BeginLocation = BeginDecl->getSourceRange().getBegin();
    RawComment *RC = AST.getASTContext().getRawCommentForDeclNoCache(
        BeginDecl);
    if (RC)
      BeginLocation = RC->getLocStart();

    // Adjust end of hover text for things that have braces/bodies. We don't
    // want to show the whole definition of a function, class, etc.
    SourceLocation EndLocation = LocationDecl->getSourceRange().getEnd();
    if (auto FuncDecl = dyn_cast<FunctionDecl>(LocationDecl)) {
      if (FuncDecl->isThisDeclarationADefinition() && FuncDecl->getBody())
        EndLocation = FuncDecl->getBody()->getLocStart();
    } else if (auto TagDeclaration = dyn_cast<TagDecl>(LocationDecl)) {
      if (TagDeclaration->isThisDeclarationADefinition())
        EndLocation = TagDeclaration->getBraceRange().getBegin();
    } else if (auto NameDecl = dyn_cast<NamespaceDecl>(LocationDecl)) {
      SourceLocation BeforeLBraceLoc = Lexer::getLocForEndOfToken(
          LocationDecl->getLocation(), 0, SourceMgr, LangOpts);
      if (BeforeLBraceLoc.isValid())
        EndLocation = BeforeLBraceLoc;
    }

    SourceRange SR(BeginLocation, EndLocation);
    if (SR.isValid()) {
      auto L = getDeclarationLocation(AST, SR);
      if (L) {
        auto Ref = getBufferDataFromSourceRange(AST, *L);
        if (Ref) {
          Hover H;
          if (SourceMgr.isInMainFile(BeginLocation))
            H.range = L->range;
          MarkedString MS = {"", "C++", *Ref};
          HoverContents.push_back(MS);
          H.contents = std::move(HoverContents);
          return H;
        }
      }
    }
  }

  auto MacroInfos = DeclMacrosFinder->takeMacroInfos();
  if (!MacroInfos.empty() && MacroInfos[0]) {
    const MacroInfo *MacroInf = MacroInfos[0];
    SourceRange SR(MacroInf->getDefinitionLoc(),
                                 MacroInf->getDefinitionEndLoc());
    if (SR.isValid()) {
      auto L = getDeclarationLocation(AST, SR);
      if (L) {
        auto Ref = getBufferDataFromSourceRange(AST, *L);
        if (Ref) {
          Hover H;
          if (SourceMgr.isInMainFile(SR.getBegin()))
            H.range = L->range;
          MarkedString MS = {"", "C++", "#define " + Ref->str()};
          H.contents.push_back(MS);
          return H;
        }
      }
    }
  }

  return Hover();
}
1172 ↗(On Diff #124116)

The range could be in another file but we can only highlight things in the file that the user current has open. For example, if I'm foo.cpp and I hover on something declared in foo.h, it will change the background color in foo.cpp but using the line numbers of foo.h! The protocol should be more clear about this but since there are no Uri in the Hover struct, we can assume this range should only apply to the open file, i.e. the main file. So I suggest this check:

if (SourceMgr.isInMainFile(BeginLocation))
  H.range = L->range;
malaperle requested changes to this revision.Nov 29 2017, 3:29 PM
malaperle added inline comments.
clangd/Protocol.cpp
498 ↗(On Diff #124833)

remove, we have to return the contents of the H that was passed as parameter, not a new one. I hit this bug while testing with no range (hover text in another file)

This revision now requires changes to proceed.Nov 29 2017, 3:29 PM

I copy/pasted the comments again so that you don't miss them.

clangd/ClangdUnit.cpp
1117 ↗(On Diff #124116)

Here's the version in which I tried to simplify this *a bit*. With the new ErrorOr checks as well.

Hover clangd::getHover(ParsedAST &AST, Position Pos, clangd::Logger &Logger) {
  const SourceManager &SourceMgr = AST.getASTContext().getSourceManager();
  const LangOptions &LangOpts = AST.getASTContext().getLangOpts();
  const FileEntry *FE = SourceMgr.getFileEntryForID(SourceMgr.getMainFileID());
  if (!FE)
    return Hover();

  SourceLocation SourceLocationBeg = getBeginningOfIdentifier(AST, Pos, FE);

  auto DeclMacrosFinder = std::make_shared<DeclarationAndMacrosFinder>(
      llvm::errs(), SourceLocationBeg, AST.getASTContext(),
      AST.getPreprocessor());
  index::IndexingOptions IndexOpts;
  IndexOpts.SystemSymbolFilter =
      index::IndexingOptions::SystemSymbolFilterKind::All;
  IndexOpts.IndexFunctionLocals = true;
  indexTopLevelDecls(AST.getASTContext(), AST.getTopLevelDecls(),
                     DeclMacrosFinder, IndexOpts);

  auto Decls = DeclMacrosFinder->takeDecls();
  if (!Decls.empty() && Decls[0]) {
    const Decl *LocationDecl = Decls[0];
    std::vector<MarkedString> HoverContents;

    // Compute scope as the first "section" of the hover.
    if (const NamedDecl *NamedD = dyn_cast<NamedDecl>(LocationDecl)) {
      std::string Scope = getScope(NamedD, AST.getASTContext().getLangOpts());
      if (!Scope.empty()) {
        MarkedString Info = {"", "C++", "In " + Scope};
        HoverContents.push_back(Info);
      }
    }

    // Adjust beginning of hover text depending on the presence of templates and comments.
    TemplateDecl * TDec = LocationDecl->getDescribedTemplate();
    const Decl * BeginDecl = TDec ? TDec : LocationDecl;
    SourceLocation BeginLocation = BeginDecl->getSourceRange().getBegin();
    RawComment *RC = AST.getASTContext().getRawCommentForDeclNoCache(
        BeginDecl);
    if (RC)
      BeginLocation = RC->getLocStart();

    // Adjust end of hover text for things that have braces/bodies. We don't
    // want to show the whole definition of a function, class, etc.
    SourceLocation EndLocation = LocationDecl->getSourceRange().getEnd();
    if (auto FuncDecl = dyn_cast<FunctionDecl>(LocationDecl)) {
      if (FuncDecl->isThisDeclarationADefinition() && FuncDecl->getBody())
        EndLocation = FuncDecl->getBody()->getLocStart();
    } else if (auto TagDeclaration = dyn_cast<TagDecl>(LocationDecl)) {
      if (TagDeclaration->isThisDeclarationADefinition())
        EndLocation = TagDeclaration->getBraceRange().getBegin();
    } else if (auto NameDecl = dyn_cast<NamespaceDecl>(LocationDecl)) {
      SourceLocation BeforeLBraceLoc = Lexer::getLocForEndOfToken(
          LocationDecl->getLocation(), 0, SourceMgr, LangOpts);
      if (BeforeLBraceLoc.isValid())
        EndLocation = BeforeLBraceLoc;
    }

    SourceRange SR(BeginLocation, EndLocation);
    if (SR.isValid()) {
      auto L = getDeclarationLocation(AST, SR);
      if (L) {
        auto Ref = getBufferDataFromSourceRange(AST, *L);
        if (Ref) {
          Hover H;
          if (SourceMgr.isInMainFile(BeginLocation))
            H.range = L->range;
          MarkedString MS = {"", "C++", *Ref};
          HoverContents.push_back(MS);
          H.contents = std::move(HoverContents);
          return H;
        }
      }
    }
  }

  auto MacroInfos = DeclMacrosFinder->takeMacroInfos();
  if (!MacroInfos.empty() && MacroInfos[0]) {
    const MacroInfo *MacroInf = MacroInfos[0];
    SourceRange SR(MacroInf->getDefinitionLoc(),
                                 MacroInf->getDefinitionEndLoc());
    if (SR.isValid()) {
      auto L = getDeclarationLocation(AST, SR);
      if (L) {
        auto Ref = getBufferDataFromSourceRange(AST, *L);
        if (Ref) {
          Hover H;
          if (SourceMgr.isInMainFile(SR.getBegin()))
            H.range = L->range;
          MarkedString MS = {"", "C++", "#define " + Ref->str()};
          H.contents.push_back(MS);
          return H;
        }
      }
    }
  }

  return Hover();
}
1172 ↗(On Diff #124116)

The range could be in another file but we can only highlight things in the file that the user current has open. For example, if I'm foo.cpp and I hover on something declared in foo.h, it will change the background color in foo.cpp but using the line numbers of foo.h! The protocol should be more clear about this but since there are no Uri in the Hover struct, we can assume this range should only apply to the open file, i.e. the main file. So I suggest this check:

if (SourceMgr.isInMainFile(BeginLocation))
  H.range = L->range;
malaperle added inline comments.Nov 29 2017, 3:37 PM
clangd/ClangdUnit.cpp
1071 ↗(On Diff #124833)

llvm::ErrorOr<Location>

1075 ↗(On Diff #124833)

To fix the macro expansion crash:

SourceLocation LocStart = SourceMgr.getExpansionLoc(ValSourceRange.getBegin());
const FileEntry *F =
        SourceMgr.getFileEntryForID(SourceMgr.getFileID(LocStart));

and simple early return

if (!F)
  return llvm::errc::no_such_file_or_directory;
1079 ↗(On Diff #124833)

getExpansionLineNumber

1080 ↗(On Diff #124833)

getExpansionColumnNumber

1082 ↗(On Diff #124833)

getExpansionLineNumber

1083 ↗(On Diff #124833)

getExpansionColumnNumber

1087 ↗(On Diff #124833)

You can do this earlier and simpler, see comment above.

malaperle added inline comments.Nov 29 2017, 9:54 PM
clangd/ClangdUnit.cpp
1117 ↗(On Diff #124116)
if (SourceMgr.isInMainFile(SR.getBegin()))
  H.range = L->range;

This is not quite right in case you have a macro definition in the preamble but still in the same "document" (they are different FileIDs for the SourceManager!). For example:

#include "foo.h"
#define MY_MACRO 1

void bar() {
   MY_MACRO;
}

The #define is *also* part of the preamble so isMainFile will return false. So in that case, I'm not sure what's the best way to make sure the macro expansion and the definition are in the same file but comparing FileEntry should work although a bit verbose:

FileID FID = SourceMgr.getFileID(SourceMgr.getExpansionLoc(SR.getBegin()));
bool IsInMainFile = FID.isValid() && SourceMgr.getMainFileID() == FID;
if (!IsInMainFile) {
  // Special case when a macro is defined in a preamble, it could still be in the "main file", below the inclusions.
  // The SourceManager considers the preamble and the main file as two different FileIDs but the FileEntries should match.
  bool IsInPreamble = FID == SourceMgr.getPreambleFileID();
  if (IsInPreamble)
    IsInMainFile = SourceMgr.getFileEntryForID(SourceMgr.getMainFileID()) == SourceMgr.getFileEntryForID(FID);
}
if (IsInMainFile)
  H.range = L->range;
Nebiroth updated this revision to Diff 124945.Nov 30 2017, 8:07 AM
Nebiroth edited edge metadata.
Nebiroth marked an inline comment as done.
Simplified getHover() function
Proper usage of ErrorOr to return errors
Given range for Hover struct now only applies to the open file
Fixed crash on MacroExpansion
malaperle requested changes to this revision.Nov 30 2017, 12:44 PM
malaperle added inline comments.
clangd/ClangdLSPServer.cpp
228 ↗(On Diff #124945)

remove extra line

clangd/ClangdUnit.cpp
1054 ↗(On Diff #124945)

else is not needed because it returns in the if

1080 ↗(On Diff #124945)

else is not needed since you return in the if

1209 ↗(On Diff #124945)
else
           H.range = Range();

Remove this, once the bug in Protocol.cpp is fixed, the Range will be truly optional.

1247 ↗(On Diff #124945)
else
           H.range = Range();

Remove this, once the bug in Protocol.cpp is fixed, the Range will be truly optional.

1505 ↗(On Diff #124945)

revert this change

clangd/Protocol.cpp
498 ↗(On Diff #124945)

remove, we have to return the contents of the H that was passed as parameter, not a new one. I hit this bug while testing with no range (hover text in another file)

So this should be

if (H.range.hasValue()) {
    return json::obj{
        {"contents", json::ary(H.contents)},
        {"range", H.range.getValue()},
    };
  }

  return json::obj{
    {"contents", json::ary(H.contents)},
};
clangd/Protocol.h
26 ↗(On Diff #124945)

revert this change?

463 ↗(On Diff #124945)

Documentation should use /// like the others

468 ↗(On Diff #124945)

Documentation should use /// like the others

This revision now requires changes to proceed.Nov 30 2017, 12:44 PM
Nebiroth marked 9 inline comments as done.Dec 1 2017, 1:19 PM
Nebiroth added inline comments.
clangd/Protocol.h
26 ↗(On Diff #124945)

#include <string> is not needed.

malaperle added inline comments.Dec 1 2017, 1:24 PM
clangd/Protocol.h
26 ↗(On Diff #124945)

I meant removing YAMLParser.h

Nebiroth added inline comments.Dec 1 2017, 1:42 PM
clangd/Protocol.h
26 ↗(On Diff #124945)

That's what I figured. I'll remove that.

Nebiroth updated this revision to Diff 125224.Dec 1 2017, 2:19 PM
Nebiroth edited edge metadata.

Minor code cleanup
Merge with master

malaperle accepted this revision.Dec 3 2017, 8:06 PM

This looks good to me. @ilya-biryukov Any objections?
I think we can do further iterations later to handle macros and other specific cases (multiple Decls at the position, etc) . It's already very useful functionality.

malaperle requested changes to this revision.Dec 14 2017, 6:14 AM

Needs to be rebased.

This revision now requires changes to proceed.Dec 14 2017, 6:14 AM
Nebiroth updated this revision to Diff 127131.Dec 15 2017, 7:24 AM

Rebase on master

ilya-biryukov added inline comments.Dec 18 2017, 3:56 AM
clangd/ClangdLSPServer.cpp
284 ↗(On Diff #127131)

NIT: remove empty line

clangd/ClangdServer.cpp
428 ↗(On Diff #127131)

We don't need to capture this, remove it from the capture list.

550 ↗(On Diff #127131)

Return an error instead of asserting to follow how other functions do that:

if (!Resources)
  return llvm::make_error<llvm::StringError>(
      "findHover called on non-added file",
      llvm::errc::invalid_argument);
clangd/ClangdServer.h
303 ↗(On Diff #127131)

Please put this right after findDocumentHighlights. It is really weird to see findHover in the middle of formatting functions.

306 ↗(On Diff #127131)

These functions are duplicates of existing ones, they should not be here.
Bad merge?

clangd/ClangdUnit.cpp
222 ↗(On Diff #127131)

NIT: remove the empty line

288 ↗(On Diff #127131)

NIT: remove the empty line

303 ↗(On Diff #127131)

Bad merge? This comment should not be changed.

308 ↗(On Diff #127131)

std::move(Decls) is better. Bad merge?

313 ↗(On Diff #127131)

This comment is not upstream now and we're removed it in the other patch. Remove it here too.

438 ↗(On Diff #127131)

We should avoid doing the string-matching when we have the AST.
Use something like this instead (haven't tested, but you should get the idea):

/// Returns a NamedDecl that could be used to print the qualifier.
Decl* getScopeOfDecl(Decl* D) {
  // Code from NamedDecl::printQualifiedName, we should probably move it into 
  const DeclContext *Ctx = D->getDeclContext();

  // For ObjC methods, look through categories and use the interface as context.
  if (auto *MD = dyn_cast<ObjCMethodDecl>(D))
    if (auto *ID = MD->getClassInterface())
      Ctx = ID;

  if (Ctx->isFunctionOrMethod()) {
    /// This is a function-local entity.
    return nullptr;
  }

  return Ctx;
}


std::string printScopeOfDecl(Decl* Scope) {
  if (isa<TranslationUnitDecl>(Scope))
    return "global namespace";

  if (isa<NamedDecl>(Scope))
    return cast<NamedDecl>(Scope)->printQualifiedName();

  return "";
}


// Later use like this:
auto ScopeText = printScopeOfDecl(getScopeOfDecl(D));
462 ↗(On Diff #127131)

NIT: llvm::Expected instead of llvm::ErrorOr

485 ↗(On Diff #127131)

NIT: llvm::Expected instead of llvm::ErrorOr

498 ↗(On Diff #127131)

Why are we changing the semantics here? Why should we use expansion locations instead of spelling locations here?

591 ↗(On Diff #127131)

This code seems pretty involved. Don't RawComments handle this case?

658 ↗(On Diff #127131)

Body of this if statement is a great candidate to be extracted into a separate function. Same for macros. The code is really hard to read as it is right now.

std::vector<MarkedString> getHoverContents(Decl *D);
std::vector<MarkedString> getHoverContents(MacroDef *M);
`
670 ↗(On Diff #127131)

the indent is off here. Use clang-format.

968 ↗(On Diff #127131)

This should clearly go before the comment!
Also, we're currently setting this flag when building the preamble, but not when parsing the AST. We should definitely do that in both cases.

clangd/Protocol.h
401 ↗(On Diff #127131)

NIT: remove empty line

427 ↗(On Diff #127131)

NIT: remove empty line

430 ↗(On Diff #127131)

NIT: remove empty comment line

436 ↗(On Diff #127131)

NIT: remove empty comment line

test/clangd/initialize-params-invalid.test
34 ↗(On Diff #127131)

indent is off by one

test/clangd/initialize-params.test
34 ↗(On Diff #127131)

indent is off by one

ilya-biryukov requested changes to this revision.Dec 18 2017, 4:42 AM
This revision now requires changes to proceed.Dec 18 2017, 4:42 AM
Nebiroth marked 21 inline comments as done.Dec 18 2017, 1:54 PM
Nebiroth added inline comments.
clangd/ClangdServer.h
306 ↗(On Diff #127131)

Yes, this is a bad merge.

clangd/ClangdUnit.cpp
438 ↗(On Diff #127131)

I'm not sure I understand. The intention was to have a separate MarkedString display the scope for whatever we are hovering on. The way I understand the above code, this would display a scope that isn't as precise as I would like which defeats the purpose of having this feature in the first place.

498 ↗(On Diff #127131)

Bad merge.

591 ↗(On Diff #127131)

Bad merge.

968 ↗(On Diff #127131)

Do you mean also adding this line somewhere in ASTUnit.cpp?

Nebiroth updated this revision to Diff 127413.Dec 18 2017, 2:13 PM
Nebiroth marked 3 inline comments as done.
findHover properly returns an error
Removed many cases of bad merging
Separated getHover in two functions
Minor indenting and NIT fixes
sammccall added inline comments.
clangd/ClangdUnit.cpp
941 ↗(On Diff #127413)

Doesn't this slow down parsing by a lot? I thought @ilya-biryukov had seen that it was responsible for big performance losses in code complete.

ISTM we might prefer to look up the comment from the index by USR, and just live without it if we can't find it.

(Apologies if this has been discussed)

clangd/ClangdUnit.h
1 ↗(On Diff #127413)

Please don't cram more stuff into clangdunit that's not directly related to building ASTs and managing their lifetimes. Instead, can we pull as much as possible out into Documentation.h or similar?

Note not Hover.h as there are other use cases, like we should use this logic for the code completion item documentation too.

test/clangd/hover.test
1 ↗(On Diff #127413)

Could I ask you to do primary/exhaustive testing of this with unit tests, and just a basic smoke test with lit?
Then it's possible to read the tests, understand whether they're correct, and understand the failure messages.

I'm trying to pull more of the features in this direction - see CodeCompleteTests.cpp for an example of how this can work.

We've had quite a few cases of bugs that had tests, but nobody noticed that the tests were wrong because it's just not possible to read them.

simark updated this revision to Diff 133367.Feb 7 2018, 8:59 PM
simark added a subscriber: simark.

New version

Here's a new version of the patch, quite reworked. I scaled down the scope a
little bit to make it simpler (see commit log). We'll add more features later
on.

Thanks for picking this up!
I have just one major comment related to how we generate the hover text.

Current implementation tries to get the actual source code for every declaration and then patch it up to look nice on in the output.
It is quite a large piece of code (spanning most of the patch) and tries to mix and match source code and output from the AST when there's not enough information in the AST (e.g. tryFixupIndentation and stripOpeningBracket). This approach is really complex as we'll inevitably try to parse "parts" of C++ and figure out how to deal with macros, etc.
Worse than that, I would expect this code to grow uncontrollably with time and be very hard to maintain.
Moreover, in presence of macros it arguably gives non-useful results. For example, consider this:

#define DEFINITIONS int foo(); int bar(); int baz();

// somewhere else
DEFINITIONS

// somewhere else
int test() {
  foo(); // <-- hover currently doesn't work here. And even if it did, showing a line with just DEFINITIONS is not that useful.
}

I suggest we move to a different approach of pretty-printing the relevant parts of the AST. It is already implemented in clang, handles all the cases in the AST (and will be updated along when AST is changed), shows useful information in presence of macros and is much easier to implement.
The version of getHoverContents using this is just a few lines of code:

static std::string getHoverContents(const Decl *D) {
  if (TemplateDecl *TD = D->getDescribedTemplate())
    D = TD; // we want to see the template bits.

  std::string Result;
  llvm::raw_string_ostream OS(Result);

  PrintingPolicy Policy(D->getASTContext().getLangOpts());
  Policy.TerseOutput = true;
  D->print(OS, Policy);

  OS.flush();
  return Result;
}

It doesn't add the "Defined in ..." piece, but illustrates the APIs we can use. We should use the same APIs for the scope as well, avoiding fallbacks to manual printing if we can.
If there's something missing/wrong in the upstream pretty printers, it's fine and we can fix them (e.g., the thing that I've immediately noticed is that namespaces are always printed with curly braces).

clangd/ClangdLSPServer.cpp
325 ↗(On Diff #133367)

NIT: remove the empty line at the start of function

clangd/Protocol.cpp
324 ↗(On Diff #133367)

NIT: use nullptr instead of NULL
(irrelevant if we use StringRef, see other comment below)

331 ↗(On Diff #133367)
  • StringRef is a nicer abstraction, maybe use it instead of const char*?
  • Maybe move declaration of KindStr closer to its usage?
  • Maybe extract a helper function to mark the code path for invalid kinds as unreachable? I.e.
static StringRef toTextKind(MarkupKind Kind) {
  switch (Kind) {
    case MarkupKind::PlainText:
      return "plaintext";
    case MarkupKind::Markdown:
      return "markdown";
  }
  llvm_unreachable("Invalid MarkupKind");
}
clangd/XRefs.cpp
326 ↗(On Diff #133367)

Same suggestion as before. Could we extract a helper function to mark invalid enum values unreachable?

552 ↗(On Diff #133367)

This assert seems rather unlikely and just makes the code more complex. Let's assume it's true and remove it (along with the subsequent if statement)

560 ↗(On Diff #133367)

This assert seems rather unlikely and just makes the code more complex. Let's assume it's true and remove it (along with the subsequent if statement)

test/clangd/hover.test
2 ↗(On Diff #133367)

We have a more readable format for the lit-tests now. Could we update this test to use it?
(see other tests in test/clangd for example)

unittests/clangd/XRefsTests.cpp
231 ↗(On Diff #133367)

NIT: remove empty line at the start of the function

233 ↗(On Diff #133367)

NIT: Use StringRef instead of const char*

simark added a comment.Feb 8 2018, 1:04 PM

Thanks for picking this up!
I have just one major comment related to how we generate the hover text.

Current implementation tries to get the actual source code for every declaration and then patch it up to look nice on in the output.
It is quite a large piece of code (spanning most of the patch) and tries to mix and match source code and output from the AST when there's not enough information in the AST (e.g. tryFixupIndentation and stripOpeningBracket). This approach is really complex as we'll inevitably try to parse "parts" of C++ and figure out how to deal with macros, etc.
Worse than that, I would expect this code to grow uncontrollably with time and be very hard to maintain.
Moreover, in presence of macros it arguably gives non-useful results. For example, consider this:

#define DEFINITIONS int foo(); int bar(); int baz();

// somewhere else
DEFINITIONS

// somewhere else
int test() {
  foo(); // <-- hover currently doesn't work here. And even if it did, showing a line with just DEFINITIONS is not that useful.
}

I suggest we move to a different approach of pretty-printing the relevant parts of the AST. It is already implemented in clang, handles all the cases in the AST (and will be updated along when AST is changed), shows useful information in presence of macros and is much easier to implement.
The version of getHoverContents using this is just a few lines of code:

static std::string getHoverContents(const Decl *D) {
  if (TemplateDecl *TD = D->getDescribedTemplate())
    D = TD; // we want to see the template bits.

  std::string Result;
  llvm::raw_string_ostream OS(Result);

  PrintingPolicy Policy(D->getASTContext().getLangOpts());
  Policy.TerseOutput = true;
  D->print(OS, Policy);

  OS.flush();
  return Result;
}

It doesn't add the "Defined in ..." piece, but illustrates the APIs we can use. We should use the same APIs for the scope as well, avoiding fallbacks to manual printing if we can.
If there's something missing/wrong in the upstream pretty printers, it's fine and we can fix them (e.g., the thing that I've immediately noticed is that namespaces are always printed with curly braces).

I thought it would be nice to show in the hover the declaration formatted as it is written in the source file, because that's how the developer is used to read it. But on the other hand, I completely agree that trying to do string formatting ourselves is opening a can of worms. I'll try the approach you suggest.

simark added inline comments.Feb 8 2018, 2:04 PM
clangd/ClangdLSPServer.cpp
325 ↗(On Diff #133367)

Done.

clangd/Protocol.cpp
324 ↗(On Diff #133367)

Ack.

331 ↗(On Diff #133367)

Done.

clangd/XRefs.cpp
326 ↗(On Diff #133367)

Done.

552 ↗(On Diff #133367)

Done...

560 ↗(On Diff #133367)

... and done.

test/clangd/hover.test
2 ↗(On Diff #133367)

Indeed the new format looks much more readable. I'll modify this one to look like completion.test.

unittests/clangd/XRefsTests.cpp
231 ↗(On Diff #133367)

Done.

233 ↗(On Diff #133367)

Done.

simark added a comment.EditedFeb 9 2018, 6:39 AM

Another example where pretty-printing the AST gives better results:

int x, y = 5, z = 6;

Hover the z will now show int z = 6, before it would have shown int x, y = 5, z = 6. I think new version is better because it only shows the variable we care about.

Is there something similar we can do when printing macros, or for them the approach of getting the text from the source file is still the best thing we have?

simark added a comment.EditedFeb 9 2018, 7:08 AM

The only problem I see is that when hovering a function/struct name, it now prints the whole function/struct definition. When talking with @malaperle, he told me that you had discussed it before and we should not have the definition in the hover, just the prototype for a function and the name (e.g. "struct foo") for a struct. Glancing quickly at the PrintingPolicy, I don't see a way to do that. Do you have any idea?

simark added a comment.Feb 9 2018, 7:32 AM

The only problem I see is that when hovering a function/struct name, it now prints the whole function/struct definition. When talking with @malaperle, he told me that you had discussed it before and we should not have the definition in the hover, just the prototype for a function and the name (e.g. "struct foo") for a struct. Glancing quickly at the PrintingPolicy, I don't see a way to do that. Do you have any idea?

Arrg, Sorry, I just re-read your comment and saw that you used TerseOutput which does that. I did not catch that when transcribing the code (I did not want to copy paste to understand the code better, but got bitten).

The only problem I see is that when hovering a function/struct name, it now prints the whole function/struct definition. When talking with @malaperle, he told me that you had discussed it before and we should not have the definition in the hover, just the prototype for a function and the name (e.g. "struct foo") for a struct. Glancing quickly at the PrintingPolicy, I don't see a way to do that. Do you have any idea?

Arrg, Sorry, I just re-read your comment and saw that you used TerseOutput which does that. I did not catch that when transcribing the code (I did not want to copy paste to understand the code better, but got bitten).

Yeah, the TerseOutput bit was important :-)

The only problem I see is that when hovering a function/struct name, it now prints the whole function/struct definition. When talking with @malaperle, he told me that you had discussed it before and we should not have the definition in the hover, just the prototype for a function and the name (e.g. "struct foo") for a struct. Glancing quickly at the PrintingPolicy, I don't see a way to do that. Do you have any idea?

I'm not aware of the code that pretty-prints macros. There's a code that dumps debug output, but it's not gonna help in that case.
Alternatively, we could simply print the macro name and not print the definition. That's super-simple and is in line with hovers for the AST decls. Let's do that in the initial version.

For macros using the source code is also less of a problem, so we may try to do it. C++ doesn't allow to generate a macro inside another macro, so we don't run into the same problems we may hit with the AST. And the overall syntax is much simpler than for AST, so we can hope to get string manipulation right with reasonable effort. (Simply reindenting should do the trick).

I'm not aware of the code that pretty-prints macros. There's a code that dumps debug output, but it's not gonna help in that case.
Alternatively, we could simply print the macro name and not print the definition. That's super-simple and is in line with hovers for the AST decls. Let's do that in the initial version.

Ok, I'm considering printing "#define MACRO". Just printing the name would not be very useful, at least printing "#define" gives the information that the hovered entity is defined as a macro (and not a function or variable).

Is there a way to get the macro name from the MacroInfo object? I couldn't find any, so the solution I'm considering is to make DeclarationAndMacrosFinder::takeMacroInfos return an std::vector<std::pair<StringRef, const MacroInfo *>>, where the first member of the pair is the macro name. It would come from IdentifierInfo->getName() in DeclarationAndMacrosFinder::finish. Does that make sense, or is there a simpler way?

Is there a way to get the macro name from the MacroInfo object? I couldn't find any, so the solution I'm considering is to make DeclarationAndMacrosFinder::takeMacroInfos return an std::vector<std::pair<StringRef, const MacroInfo *>>, where the first member of the pair is the macro name. It would come from IdentifierInfo->getName() in DeclarationAndMacrosFinder::finish. Does that make sense, or is there a simpler way?

I don't think there's a way to get macro name from MacroInfo. pair<Name, MacroInfo*> sounds good to me, I'd probably even use a named struct here: struct MacroDecl { StringRef Name; const MacroInfo &Info; }

simark updated this revision to Diff 134306.Feb 14 2018, 1:45 PM

Generate Hover by pretty-printing the AST

This new version of the patch generates the hover by pretty-printing the AST.
The unit tests are updated accordingly. There are some inconsistencies in how
things are printed. For example, I find it a bit annoying that we print empty
curly braces after class names (e.g. "class Foo {}"). Also, namespace and
enums are printed with a newline between the two braces. These are things that
could get fixed by doing changes to the clang AST printer.

The hover.test test now uses a more readable format.

simark added inline comments.Feb 14 2018, 2:36 PM
unittests/clangd/XRefsTests.cpp
561 ↗(On Diff #134306)

Note that I used .str() here to make the output of failing tests readable and useful. By default, gtest tries to print StringRef as if it was a container. I tried to make add a PrintTo function to specify how it should be printed, but couldn't get it to work (to have it called by the compiler), so I settled for this.

Just a few last remarks and this is good to go.
Should I land it for you after the last comments are fixed?

clangd/XRefs.cpp
354 ↗(On Diff #134306)

We should call flush() before returning Name here. The raw_string_ostream is buffered.

373 ↗(On Diff #134306)

NIT: use llvm::None here instead of {}

394 ↗(On Diff #134306)

Accidental leftover from previous code?

unittests/clangd/XRefsTests.cpp
262 ↗(On Diff #134306)

NIT: LLVM uses UpperCamelCase for field names.

561 ↗(On Diff #134306)

Thanks for spotting that.
We have a fix for that in LLVM's gtest extensions (D43330).
str() can now be removed.

simark added inline comments.Feb 15 2018, 8:29 AM
clangd/XRefs.cpp
354 ↗(On Diff #134306)

I replaced it with Stream.str().

373 ↗(On Diff #134306)

Done.

394 ↗(On Diff #134306)

Oops, thanks.

unittests/clangd/XRefsTests.cpp
262 ↗(On Diff #134306)

Ok, I fixed all the structs this patch adds.

561 ↗(On Diff #134306)

Removed. Even if this patch is merged before D43330 it's not a big deal, since it only affects debug output of failing test cases.

simark updated this revision to Diff 134437.Feb 15 2018, 8:33 AM

New version

  • Rebase on master, change findHover to have a callback-based interface
  • Fix nits from previous review comments

Just a few last remarks and this is good to go.
Should I land it for you after the last comments are fixed?

I just received my commit access, so if you are ok with it, I would try to push it once I get your final OK.

Only the naming of fields left.

Also, please make sure to run git-clang-format on the code before submitting to make sure it's formatted properly.

clangd/ClangdLSPServer.cpp
331 ↗(On Diff #134437)

NIT: use return replyError to be consistent with other methods.

clangd/XRefs.cpp
354 ↗(On Diff #134306)

Also works, albeit less efficient (will probably do a copy where move is enough).

unittests/clangd/XRefsTests.cpp
262 ↗(On Diff #134306)

Argh, sorry that I didn't mention it before, but we have an exception for structs in Procotol.h, they follow the spelling in the LSP spec (i.e. should be lowerCamelCase).

Only the naming of fields left.

Also, please make sure to run git-clang-format on the code before submitting to make sure it's formatted properly.

Ahh, I was running clang-format by hand, didn't know about git-clang-format (and obviously forgot some files). Thanks.

clangd/ClangdLSPServer.cpp
331 ↗(On Diff #134437)

Ok.

clangd/XRefs.cpp
354 ↗(On Diff #134306)

Ah, didn't think about that. I'll change it to flush + return Name.

unittests/clangd/XRefsTests.cpp
262 ↗(On Diff #134306)

Haha, no problem, changing them back is one keystroke away.

simark updated this revision to Diff 134440.Feb 15 2018, 8:54 AM

Fix some nits

  • Revert case change of struct fields in Protocol.h
  • Change return formatting in onHover
  • Use flush() + return Name in NamedDeclQualifiedName and TypeDeclToString
ilya-biryukov accepted this revision.Feb 15 2018, 9:18 AM

Thanks for fixing all the NITs!

simark updated this revision to Diff 134452.Feb 15 2018, 10:17 AM

Fix field names in XRefsTests.cpp

I realized I forgot to add some fixups in the previous version, the field names
in XRefsTests.cpp were not updated and therefore it did not build.

malaperle updated this revision to Diff 134708.Feb 16 2018, 1:19 PM

Add a missing std::move, fix some formatting.

malaperle accepted this revision.Feb 16 2018, 1:39 PM

Works well and code looks good. There were only minor tweaks since the last approval. I'll land this since there are some issues with Simon's svn account.

This revision is now accepted and ready to land.Feb 16 2018, 1:39 PM
This revision was automatically updated to reflect the committed changes.
This revision was automatically updated to reflect the committed changes.