Page MenuHomePhabricator

[clangd] Emit publishSemanticHighlighting in LSP if enabled

Authored by jvikstrom on Jun 28 2019, 1:07 AM.

Diff Detail


Event Timeline

jvikstrom created this revision.Jun 28 2019, 1:07 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 28 2019, 1:07 AM
hokein added inline comments.Jun 28 2019, 1:59 AM
31 ↗(On Diff #207010)

The structures defined here are similar to the structures proposed in LSP SemanticHighlightingParams and SemanticHighlightingInformation, we could define these structures in the Protocol.h (you can find similar pattern there).

jvikstrom updated this revision to Diff 207031.Jun 28 2019, 3:50 AM
jvikstrom marked an inline comment as done.

Added SemanticHighlightingInformation and SemantigHighlightingParams.

hokein added inline comments.Jun 28 2019, 6:05 AM
332 ↗(On Diff #207031)

I'm a bit nervous -- we will turn on this feature when the client states that it supports semantic highlighting, client may disable its reg-based highlighting, and relies on clangd's, but the feature in clangd is in pretty early stage...

I think we'd need to add a new command-line flag to clangd (disabled by default).

1179 ↗(On Diff #207031)

please use /// which is doxygen comment.

And /// Represents a semantic highlighting information that has to be applied on a specific line of the text document.

1182 ↗(On Diff #207031)

nit: using int is fine here.

1189 ↗(On Diff #207031)

Add a comment: /// Parameters for the semantic highlighting (server-side) push notification.

120 ↗(On Diff #207031)

add unittests for this function?

132 ↗(On Diff #207031)

the implementation requires Tokens sorted, maybe consider using a map to the group-by?

152 ↗(On Diff #207031)

thinking more about this, the .cpp suffix is language-specific, clangd also supports other C-family languages (e.g. C, ObjC), so we may need corresponding language suffix in the scopes (based on the language mode).

No need to do the change here, but please add a FIXME.

8 ↗(On Diff #207031)

The semantic highlighting is not in LSP yet, I think we need some documentations here to explain some more details about this feature in clangd, like the implementations are based on the proposal (

19 ↗(On Diff #207031)

as this kind is used as the lookup table index, let's add Variable = 0

37 ↗(On Diff #207031)

I'd name it textMateScopeLookupTable

40 ↗(On Diff #207031)

the assumption seems too strict here. IIUC, LSP doesn't require the order of the SemanticHighlightingInformation[], would be nice to return an ordered one (but that's implementation detail).

46 ↗(On Diff #207031)

just toSemanticHighlightingInformation.

jvikstrom updated this revision to Diff 207241.Jul 1 2019, 1:48 AM
jvikstrom marked 13 inline comments as done.

Addressed comments.

hokein added inline comments.Jul 2 2019, 3:14 AM
333 ↗(On Diff #207241)

I'd put the logic here rather than in ClangdServer.cpp, just ClangdServerOpts.SemanticHighlighting = ClangdServerOpts.HiddenFeatures && Params.capabilities.SemanticHighlighting;.

131 ↗(On Diff #207241)

here we lookup the map twice, we could save one cost of lookup.

TokenLines[LindIdx].push_back(Token) should work.

143 ↗(On Diff #207241)

could we have some comments explaining the details about encoding Token here?

144 ↗(On Diff #207241)

if the token is across multiple line, we will emit an ill-formed results.

44 ↗(On Diff #207241)

just: convert to LSP's semantic highlighting information.

103 ↗(On Diff #207241)

we could drop the explicit HighlightingToken, just

... Tokens = {
   {HighlightingKind::Variable, Range {...}},
104 ↗(On Diff #207241)

Range{ /*start*/{3, 8}, /*end*/{3, 12} } should be compilable.

112 ↗(On Diff #207241)

here as well, just { {1, "...." } }.

jvikstrom updated this revision to Diff 207537.Jul 2 2019, 6:16 AM
jvikstrom marked 8 inline comments as done.

Address comments.

jvikstrom added inline comments.Jul 2 2019, 6:17 AM
144 ↗(On Diff #207241)

There's a FIXME above (which is where it should probably be handled). A bit unsure how to solve though. If a token is a block comment spanning multiple lines we would need to know the length of every line in the block comment. Probably something that can be solved with the ASTContext or SourceManager but that can't be accessed in this function.

104 ↗(On Diff #207241)

Doesn't compile because of the default initialization of line and character in Position.

jvikstrom updated this revision to Diff 207563.Jul 2 2019, 8:01 AM

Removed from hiding under hidden flag .

jvikstrom updated this revision to Diff 207714.Jul 3 2019, 12:32 AM

Removed unused headers

hokein added inline comments.Jul 3 2019, 3:46 AM
409 ↗(On Diff #207563)

The proposal says If the client declares its capabilities with respect to the semantic highlighting feature, and if the server supports this feature too, the server should set all the available TextMate scopes as a "lookup table" during the initialize request..

now it seems that clangd always emit this lookup table to the client.

74 ↗(On Diff #207563)

The name U doesn't provide much information, maybe Bytes?

138 ↗(On Diff #207563)

The code is a bit tricky here, if I understand the code correctly, LineHighlights is the binary data of tokens (each char represents a byte).
Maybe LineEncodingTokens?

141 ↗(On Diff #207563)

don't repeat what the code does in the comment. Here we are encoding a Token in a form specified in the proposal, I think we should have high-level comment.

144 ↗(On Diff #207241)

oh, i missed that FIXME, the FIXME is a bit far away, maybe move it here (now we assume the token is always on the same line).

109 ↗(On Diff #207563)

nit: ActualResults?

111 ↗(On Diff #207563)

nit: ExpectedResults?

114 ↗(On Diff #207563)

nit: we should use EXPECT_* when comparing the expected results and actual results.

jvikstrom updated this revision to Diff 207758.Jul 3 2019, 4:55 AM
jvikstrom marked 8 inline comments as done.

Not sending TextMate scopes over LSP if semantic highlighting is not enabled.

jvikstrom marked an inline comment as done.Jul 3 2019, 4:57 AM
jvikstrom added inline comments.
138 ↗(On Diff #207563)

They aren't "fully" encoded yet though, they get encoded after the inner loop is done. How about LineByteTokens?

144 ↗(On Diff #207241)

I think that this should probably be handled above (so if a token covers N different lines it would be separated into N tokens in the TokenLines map.
I don't really see how it would be possible to handle that in this loop (without basically putting all the smallvectors for every line in another vector and writing to Lines in another loop after the loop is done. But it feels like that would be way more complex)

8 ↗(On Diff #207031)

Changed the initialize-params test to have semantic highlighting enabled as well.

hokein added a comment.Jul 3 2019, 6:44 AM

mostly good, a few more nits.

1187 ↗(On Diff #207758)

nit: remove the blank line to be consistent with the existing style in this file.

1199 ↗(On Diff #207758)

nit: remove this blank line.

138 ↗(On Diff #207563)

sounds good.

121 ↗(On Diff #207758)

nit: remove the "{}"

138 ↗(On Diff #207758)


|<--- 4 bytes  --->|<-2 bytes->|<-2 bytes->|
|    character      |  length       |    index       |

to the comment, it helps reader to understand how the encode like.

156 ↗(On Diff #207758)

nit: remove the {} for one-body statement.

7 ↗(On Diff #207758)

nit: this should in a new section:

//==-- SemanticHighlighting.h - Generating highlights from the AST-- C++ -*-==//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//  An implementation ....
3 ↗(On Diff #207758)

How about moving this to the new semantic-highlighting.test?

jvikstrom updated this revision to Diff 207783.Jul 3 2019, 6:59 AM
jvikstrom marked 7 inline comments as done.

Addressed comments.

hokein accepted this revision.Jul 4 2019, 12:37 AM

thanks, looks good.

59 ↗(On Diff #207783)

nit: clang-format.

36 ↗(On Diff #207783)

nit: we could just verify this field like:

# CHECK:  "id": 0,
# CHECK:  " "semanticHighlighting": {
# CHECK-NEXT:        "scopes": [
# CHECK-NEXT:          [
# CHECK-NEXT:            "variable.cpp"
# CHECK-NEXT:          ],
# CHECK-NEXT:          [
# CHECK-NEXT:            ""
# CHECK-NEXT:          ]
# CHECK-NEXT:        ]
# CHECK-NEXT:      },
This revision is now accepted and ready to land.Jul 4 2019, 12:37 AM
This revision was automatically updated to reflect the committed changes.
jvikstrom marked 2 inline comments as done.
Herald added a project: Restricted Project. · View Herald TranscriptJul 4 2019, 12:54 AM