[clangd] Inactive regions support as an extension to semantic highlighting

[nridge]

This patch adds server-side support for greying out code in inactive preprocessor branches (issue #132).

I didn't write test yet. I wanted to post the patch for feedback first, to see if the general approach is ok.

[nridge]

For reference, I also posted a proof-of-concept client impl in D67537, which I used to test/validate this patch.

[sammccall]

Rather than a separate method with parallel implementation, this seems very closely related to the syntax highlighting feature.

The minimal way to model this (no new protocol) would be for each disabled line, to add one token spanning the whole line with a TextMate scope like comment.disabled or meta.disabled.
There's no technical reason tokens can't overlap in the protocol, though it's possible editors will choose to turn off local heuristic highlighting on anything that's touched by an LSP-provided token.

Failing that, I'd suggest encoding a list of line-styles on SemanticHighlightingInformation, that should be combined with any tokens on that line.

[nridge]

Ok, I can try formulating this as part of / an extension to semantic highlighting.

[nridge]

Reformulate as an extension to semantic highlighting

[nridge]

The updated patch formulates this as a new semantic highlighting kind.

The tokens created for inactive regions are one per line, with the character offset and length being zero; the idea is that the client will handle this highlighting specially (see D67537) and highlight the entire line.

Note, highlighting the entire line is different than highlighting just the portion of the line that contains text. We want the grey background highlight to extend to the entire row in the editor, even over columns beyond the end of the text line (as accomplished with vscode's isWholeLine=true option in DecorationRenderOptions).

I don't love special-casing the scope in the client, but this does avoid having to extend the protocol itself which would be considerably more work.

[hokein]

Could you add tests for this?

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

I'm not sure how does this flag impact the size of Preamble/AST, @ilya-biryukov any thoughts?

clang-tools-extra/clangd/SemanticHighlighting.cpp

164

I think the current implementation only collects the skipped preprocessing ranges after preamble region in the main file. We probably need to store these ranges in PreambleData to make the ranges in the preamble region work, no need to do it in this patch, but we'd better have a test reflecting this fact.

#ifdef ActiveCOde
// inactive code here
#endif

#include "foo.h"
// preamble ends here

namespace ns {
// ..
}

173

This seems too couple with VSCode client, I would prefer to calculate the range of the line and return to the client.

Is there any big differences in VSCode between highlighting with the isWholeLine and highlighting with the range of the line?

clang-tools-extra/clangd/SemanticHighlighting.h

45

This is a different kind group, I would put it after the Macro, we'd need to update the LastKind.

The name seems too specific, how about "UnreachableCode"?

[nridge]

In D67536#1691107, @hokein wrote:

Could you add tests for this?

Certainly, I just wanted to discuss the general approach first, as it will affect what the tests look like.

clang-tools-extra/clangd/SemanticHighlighting.cpp

173

I took some screenshots to illustrate to difference.

Highlighting only to the end of the line of text:

Highlighting the whole line:

I think the first one looks pretty bad, and is inconsistent with existing practice.

Note also that the suggestion is not to special-case the VSCode client specifically; it's to special-case one particular highlighting, which any client can implement.

If this special-casing is really unpalatable, we could instead try this suggestion by @sammccall:

Failing that, I'd suggest encoding a list of line-styles on SemanticHighlightingInformation, that should be combined with any tokens on that line.

I guess one consideration when evaluating these options is, do we expect to use that "list of line-styles" for anything else in the future? I can't think of anything at the moment, but perhaps there are other uses for it.

If not, we could do something slightly simpler, and add a single isInactive flag to SemanticHighlightingInformation.

[MaskRay]

Why "inactive region", not "skipped ranges"?

[ilya-biryukov]

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

Have no idea, but why do we need this in the first place?
PPCallbacks::SourceRangeSkipped should allow to record all skipped ranges in the main file. Can we use it?

[nridge]

In D67536#1694098, @MaskRay wrote:

Why "inactive region", not "skipped ranges"?

I got the name from "$cquery/publishInactiveRegions" :) But I don't particularly care what we call it.

[nridge]

Update to use PPCallbacks::SourceRangeSkipped

[sammccall]

clang-tools-extra/clangd/SemanticHighlighting.cpp

161

I think this comment could be clearer, e.g. // Add tokens indicating lines skipped by the preprocessor.

173

Three approaches seem feasible here:

1. clients that know about the specific scope can extend it to the whole line.
2. [0,0) or so indicates "highlight the whole line"
3. use a dedicated property for line styles (vs token styles)

3 is clearly better than 2 I think, it's more explicit. I don't have a strong opinion of 1 vs 3, but if going with 1 I think it's a good idea to measure the line as Haojian says, so we at least get a basic version of the feature if the client doesn't know about line styles.

I guess one consideration when evaluating these options is, do we expect to use that "list of line-styles" for anything else in the future? I can't think of anything at the moment

Preprocessor directives maybe? (Though these are easy enough for clients to highlight with regex)

[nridge]

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

Yes, PPCallbacks::SourceRangeSkipped works in place of using DetailedRecord. Thank you for the suggestion.

clang-tools-extra/clangd/SemanticHighlighting.h

45

I changed it to "InactiveCode". ("Unreachable" seemed like the wrong word, it brings to mind control flow.)

[nridge]

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

[sammccall]

In D67536#1697533, @nridge wrote:

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

If you look at functions like offsetToPosition in SourceCode.h, basically you get the buffer as a string (contains utf-8), find the offset you're interested in, and then zoom around looking for \n. Once you have a substring, calling lspLength() on it will give you the length in UTF-16 or whatever LSP is speaking at the moment.

[hokein]

In D67536#1697533, @nridge wrote:

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

Yes, there is no existing API for that, I think you'd need to get the source code from the SM at the specific offset, and do the \n counting.

clang-tools-extra/clangd/ParsedAST.h

100

Instead of adding new member and methods in ParsedAST, I think we can do it in CollectMainFileMacros (adding a new field SkippRanges in MainFileMacros), then we can get the skipped ranges for preamble region within the main file for free.

clang-tools-extra/clangd/SemanticHighlighting.cpp

173

I can't say whether highlighting the line is better than highlighting the range of the line text, but below is the how the inactive TS code is highlighted in VSCode (only the range of text), I personally prefer this style.

[nridge]

One thing that may be worth considering as well, is that if the client prefers to highlight the text of the line only, it can calculate the length of the line itself. In VSCode for instance, the line lengths are readily available; I imagine other editors are similar since they need that information for many purposes.

[sammccall]

In D67536#1697696, @nridge wrote:

One thing that may be worth considering as well, is that if the client prefers to highlight the text of the line only, it can calculate the length of the line itself. In VSCode for instance, the line lengths are readily available; I imagine other editors are similar since they need that information for many purposes.

So I don't think clients will/should prefer that - for best rendering they should know this is a line highlight.

I think this comes down to how line highlights are represented in the protocol:

• by a separate field: no need to send line length
• by a special token bounds (e.g. [0,0)): no need to send line length
• by a special scope: sending line length is a nice-to-have as it provides graceful degradation for clients that don't understand this extension

clang-tools-extra/clangd/SemanticHighlighting.cpp

173

I think that's an argument for making sure clients clearly distinguish between regular tokens and marking lines: overlapping tokens don't compose well, but we can easily say lines and token styles should compose.

(That particular style is not for me, but it doesn't matter)

[ilya-biryukov]

In D67536#1697825, @sammccall wrote:

So I don't think clients will/should prefer that - for best rendering they should know this is a line highlight.

I have actually seen clients that just make the text gray and it looks pretty nice (ReSharper for Visual Studio and IntelliJ IDEA definitely do that).
It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

So I wouldn't be so sure, at least some clients might prefer to do exclusively range (as opposed to line-based) highlightings.

PS Sorry if this was discussed before and I'm just missing context.

[nridge]

clang-tools-extra/clangd/SemanticHighlighting.cpp

173

I find this a convincing argument for using line styles, thanks.

Especially since, at some point in the future, I would like us to be able to produce regular token highlightings for inactive code, much like in that TS code screenshot.

[nridge]

In D67536#1699221, @ilya-biryukov wrote:

I have actually seen clients that just make the text gray and it looks pretty nice (ReSharper for Visual Studio and IntelliJ IDEA definitely do that).

I don't think there is a conflict between this style, and using line highlightings: if the style being applied is a foreground color (gray text) as opposed to a background color, then it will look the same regardless of whether you apply it as a line style, or as a character range style.

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

If we later introduce highlightings for dead expressions or statements, we can of course use character ranges and not lines for them.

[ilya-biryukov]

In D67536#1700872, @nridge wrote:

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

Note that the opposite is true - inactive preprocessor branches can be expressed as range-based highlightings.

If we later introduce highlightings for dead expressions or statements, we can of course use character ranges and not lines for them.

Then we'll have two modes of operation - line-based and range-based highlightings. Clients will have to support both, we'll have code dealing with both.
The alternative is having only range-based highlightings and using them everywhere. That's the main reason I would advocate for range-based.

[ilya-biryukov]

Just noticed the next version of LSP added diagnostic tags for things like "unused field" or "dead code":
https://microsoft.github.io/language-server-protocol/specifications/specification-3-15, search for DiagnosticTag.

So I guess we won't need ever add range-based highlightings for dead code (and similar), we'll use diagnostics instead.

Concerns about having two kinds of highlightings to handle on the clients are still there, though.

[nridge]

In D67536#1701038, @ilya-biryukov wrote:

In D67536#1700872, @nridge wrote:

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

Note that the opposite is true - inactive preprocessor branches can be expressed as range-based highlightings.

Except that if we want to allow background styling, the client would need to special-case the highlighting scope to know to apply the background style to the entire line.

[nridge]

Updated to use as isInactive flag on SemanticHighlightingInformation

[nridge]

Ok, I updated the patch to convey the line highlight separately from the token highlights.

I did use the simplified approach I mentioned, where I use a single isInactive flag. If you'd prefer the more general approach where we send an array of line styles, I can do that, though my personal preference would be to wait until we have an actual use case before generalizing.

There are still no tests yet, as I'm still not sure whether we have agreement on the approach. If you like the approach, please let me know, and I'll follow up with tests.

[nridge]

Review / feedback ping :)

[ilya-biryukov]

Just a few NITs from my side. I'll let the other review the actual implementation

clang-tools-extra/clangd/ParsedAST.cpp

217

Maybe keep only ranges from the main file?

clang-tools-extra/clangd/ParsedAST.h

100

NIT: please return llvm::ArrayRef<SourceRange> instead

[nridge]

Addressed nits

[hokein]

Thanks, I'm fine with the current approach, feel free to add unittests.

clang-tools-extra/clangd/Protocol.h

1212

could you add some documentation describing this is a clangd extension?

clang-tools-extra/clangd/SemanticHighlighting.cpp

480

it took me a while to understand this code,

If the NewLine is IsInactive, it just contains a single token whose range is [0, 0), can't we just?

if (NewLine.back().Tokens.empty()) continue;

bool InactiveLine = NewLine.back().Tokens.front().Kind == InactiveCode;
assert(InactiveLine && NewLine.back().Tokens.size() == 1 && "IncativeCode must have a single token");
DiffedLines.back().IsInactive = true;

clang-tools-extra/clangd/SemanticHighlighting.h

46

we should document this Kind as it is different than other kinds, it is for line-style I believe?

[nridge]

Addressed some nits and got existing tests to pass

Will follow up with new tests in the next update

clang-tools-extra/clangd/SemanticHighlighting.cpp

480

An inactive line can contain token highlightings as well. For example, we highlight macro references in the condition of an #ifdef, and that line is also inactive if the condition is false. Clients can merge the line style (which is typically a background color) with the token styles (typically a foreground color).

I did expand the comment to explain what the loop is doing more clearly.

[nridge]

Add unit tests

[nridge]

Address one minor remaining comment

clang-tools-extra/clangd/SemanticHighlighting.cpp

164

Your observation is correct: the current implementation only highlights inactive lines after the preamble. For now, I added a test case with a FIXME for this.

[hokein]

clang-tools-extra/clangd/ParsedAST.h

100

This comment is undone.

clang-tools-extra/clangd/SemanticHighlighting.cpp

164

I'd prefer to fix it in this patch, it should not require large effort, and probably simplify the patch, you don't need to implement a new PPCallbacks. See my another comment.

[nridge]

clang-tools-extra/clangd/ParsedAST.h

100

My apologies, I overlooked this.

[nridge]

Support preamble as well

[hokein]

thanks, mostly good, a few more nits.

clang-tools-extra/clangd/ParsedAST.h

137

nit: it is not used anymore, remove it.

clang-tools-extra/clangd/Protocol.h

1212

could you document that the line-style highlightings can be combined with any token-style highlightings?

clang-tools-extra/clangd/SemanticHighlighting.cpp

480

thanks, I see.

I think we can still simplify the code like below, this could improve the code readability, and avoid the comment explaining it.

llvm::erase_if(AddedLine.Tokens, [](const Token& T) { return T.Kind == InactiveCode;});

[nridge]

Address nits

clang-tools-extra/clangd/SemanticHighlighting.cpp

480

Done. Note that we still need to set AddedLine.IsInactive appropriately. I did that in the lambda, which feels like a strange thing for an erase_if predicate to do. The alternative would be doing a count_if first (but then we're looping over the tokens twice).

[hokein]

looks good, thanks!

clang-tools-extra/clangd/ParsedAST.h

public:
 
   /// Gets all macro references (definition, expansions) present in the main
   /// file, including those in the preamble region.
   const MainFileMacros &getMacros() const;
   /// Tokens recorded while parsing the main file.
   /// (!) does not have tokens from the preamble.
   const syntax::TokenBuffer &getTokens() const { return Tokens; }
 
+  llvm::ArrayRef<SourceRange> getSkippedRanges() const { return SkippedRanges; }
+
 private:
   ParsedAST(std::shared_ptr<const PreambleData> Preamble,
             std::unique_ptr<CompilerInstance> Clang,
             std::unique_ptr<FrontendAction> Action, syntax::TokenBuffer Tokens,
             MainFileMacros Macros, std::vector<Decl *> LocalTopLevelDecls,
             std::vector<Diag> Diags, IncludeStructure Includes,
-            CanonicalIncludes CanonIncludes);
+            CanonicalIncludes CanonIncludes,
+            std::vector<SourceRange> SkippedRanges);
 
   // In-memory preambles must outlive the AST, it is important that this member
   // goes before Clang and Action.
   std::shared_ptr<const PreambleData> Preamble;
   // We store an "incomplete" FrontendAction (i.e. no EndSourceFile was called
   // on it) and CompilerInstance used to run it. That way we don't have to do
   // complex memory management of all Clang structures on our own. (They are
   // stored in CompilerInstance and cleaned up by
private:
   MainFileMacros Macros;
   // Data, stored after parsing.
   std::vector<Diag> Diags;
   // Top-level decls inside the current file. Not that this does not include
   // top-level decls from the preamble.
   std::vector<Decl *> LocalTopLevelDecls;
   IncludeStructure Includes;
   CanonicalIncludes CanonIncludes;
+  // Ranges skipped during preprocessing.
+  std::vector<SourceRange> SkippedRanges;
 };
 
 /// Build an AST from provided user inputs. This function does not check if
 /// preamble can be reused, as this function expects that \p Preamble is the
 /// result of calling buildPreamble.
 llvm::Optional<ParsedAST>
 buildAST(PathRef FileName, std::unique_ptr<CompilerInvocation> Invocation,
          llvm::ArrayRef<Diag> CompilerInvocationDiags,

clang-tools-extra/clangd/ParsedAST.cpp

private:
 
   const IncludeStructure &Includes;
   PPCallbacks *Delegate;
   const SourceManager &SM;
   Preprocessor &PP;
   const LangOptions &LangOpts;
 };
 
+class CollectSkippedRanges : public PPCallbacks {
+public:
+  explicit CollectSkippedRanges(SourceManager &SM,
+                                std::vector<SourceRange> &SkippedRanges)
+      : SM(SM), SkippedRanges(SkippedRanges) {}
+
+  void SourceRangeSkipped(SourceRange Range, SourceLocation EndifLoc) override {
+    if (isInsideMainFile(Range.getBegin(), SM)) {
+      SkippedRanges.push_back(Range);
+    }
+  }
+
+private:
+  SourceManager &SM;
+  std::vector<SourceRange> &SkippedRanges;
+};
+
 } // namespace
 
 void dumpAST(ParsedAST &AST, llvm::raw_ostream &OS) {
   AST.getASTContext().getTranslationUnitDecl()->dump(OS, true);
 }
 
 llvm::Optional<ParsedAST>
 ParsedAST::build(std::unique_ptr<clang::CompilerInvocation> CI,
ParsedAST::build(std::unique_ptr<clang::CompilerInvocation> CI,
   // Copy over the macros in the preamble region of the main file, and combine
   // with non-preamble macros below.
   MainFileMacros Macros;
   if (Preamble)
     Macros = Preamble->Macros;
   Clang->getPreprocessor().addPPCallbacks(
       std::make_unique<CollectMainFileMacros>(Clang->getSourceManager(),
                                               Clang->getLangOpts(), Macros));
+  std::vector<SourceRange> SkippedRanges;
+  Clang->getPreprocessor().addPPCallbacks(
+      std::make_unique<CollectSkippedRanges>(Clang->getSourceManager(),
+                                             SkippedRanges));
 
   // Copy over the includes from the preamble, then combine with the
   // non-preamble includes below.
   CanonicalIncludes CanonIncludes;
   if (Preamble)
     CanonIncludes = Preamble->CanonIncludes;
   else
     CanonIncludes.addSystemHeadersMapping(Clang->getLangOpts());
ParsedAST::build(std::unique_ptr<clang::CompilerInvocation> CI,
   // Finally, add diagnostics coming from the AST.
   {
     std::vector<Diag> D = ASTDiags.take(CTContext.getPointer());
     Diags.insert(Diags.end(), D.begin(), D.end());
   }
   return ParsedAST(std::move(Preamble), std::move(Clang), std::move(Action),
                    std::move(Tokens), std::move(Macros), std::move(ParsedDecls),
                    std::move(Diags), std::move(Includes),
-                   std::move(CanonIncludes));
+                   std::move(CanonIncludes), std::move(SkippedRanges));
 }
 
 ParsedAST::ParsedAST(ParsedAST &&Other) = default;
 
 ParsedAST &ParsedAST::operator=(ParsedAST &&Other) = default;
 
 ParsedAST::~ParsedAST() {
   if (Action) {

 }
 
 ParsedAST::ParsedAST(std::shared_ptr<const PreambleData> Preamble,
                      std::unique_ptr<CompilerInstance> Clang,
                      std::unique_ptr<FrontendAction> Action,
                      syntax::TokenBuffer Tokens, MainFileMacros Macros,
                      std::vector<Decl *> LocalTopLevelDecls,
                      std::vector<Diag> Diags, IncludeStructure Includes,
-                     CanonicalIncludes CanonIncludes)
+                     CanonicalIncludes CanonIncludes,
+                     std::vector<SourceRange> SkippedRanges)
     : Preamble(std::move(Preamble)), Clang(std::move(Clang)),
       Action(std::move(Action)), Tokens(std::move(Tokens)),
       Macros(std::move(Macros)), Diags(std::move(Diags)),
       LocalTopLevelDecls(std::move(LocalTopLevelDecls)),
-      Includes(std::move(Includes)), CanonIncludes(std::move(CanonIncludes)) {
+      Includes(std::move(Includes)), CanonIncludes(std::move(CanonIncludes)),
+      SkippedRanges(std::move(SkippedRanges)) {
   assert(this->Clang);
   assert(this->Action);
 }
 
 llvm::Optional<ParsedAST>
 buildAST(PathRef FileName, std::unique_ptr<CompilerInvocation> Invocation,
          llvm::ArrayRef<Diag> CompilerInvocationDiags,
          const ParseInputs &Inputs,

clang-tools-extra/clangd/Protocol.h

 
 /// Represents a semantic highlighting information that has to be applied on a
 /// specific line of the text document.
 struct SemanticHighlightingInformation {
   /// The line these highlightings belong to.
   int Line = 0;
   /// The base64 encoded string of highlighting tokens.
   std::string Tokens;
+  /// Is the line in an inactive preprocessor branch?
+  /// This is a clangd extension.
+  bool IsInactive = false;
 };
 bool operator==(const SemanticHighlightingInformation &Lhs,
                 const SemanticHighlightingInformation &Rhs);
 llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting);
 
 /// Parameters for the semantic highlighting (server-side) push notification.
 struct SemanticHighlightingParams {
   /// The textdocument these highlightings belong to.

clang-tools-extra/clangd/Protocol.cpp

 
 bool operator==(const SemanticHighlightingInformation &Lhs,
                 const SemanticHighlightingInformation &Rhs) {
   return Lhs.Line == Rhs.Line && Lhs.Tokens == Rhs.Tokens;
 }
 
 llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting) {
   return llvm::json::Object{{"line", Highlighting.Line},
-                            {"tokens", Highlighting.Tokens}};
+                            {"tokens", Highlighting.Tokens},
+                            {"isInactive", Highlighting.IsInactive}};
 }
 
 llvm::json::Value toJSON(const SemanticHighlightingParams &Highlighting) {
   return llvm::json::Object{
       {"textDocument", Highlighting.TextDocument},
       {"lines", std::move(Highlighting.Lines)},
   };
 }

clang-tools-extra/clangd/SemanticHighlighting.h

enum class HighlightingKind {
   Enum,
   EnumConstant,
   Typedef,
   DependentType,
   DependentName,
   Namespace,
   TemplateParameter,
   Primitive,
   Macro,
 
-  LastKind = Macro
+  // This one is different from the other kinds as it's a line style
+  // rather than a token style.
+  InactiveCode,
+
+  LastKind = InactiveCode
 };
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K);
 
 // Contains all information needed for the highlighting a token.
 struct HighlightingToken {
   HighlightingKind Kind;
   Range R;
 };
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R);
 bool operator<(const HighlightingToken &L, const HighlightingToken &R);
 
 /// Contains all information about highlightings on a single line.
 struct LineHighlightings {
   int Line;
   std::vector<HighlightingToken> Tokens;
+  bool IsInactive;
 };
 
 bool operator==(const LineHighlightings &L, const LineHighlightings &R);
 
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST);
 

clang-tools-extra/clangd/SemanticHighlighting.cpp

for (ArrayRef<HighlightingToken> TokRef = Tokens; !TokRef.empty();) {
       // If there is exactly one token with this range it's non conflicting and
       // should be in the highlightings.
       if (Conflicting.size() == 1)
         NonConflicting.push_back(TokRef.front());
       // TokRef[Conflicting.size()] is the next token with a different range (or
       // the end of the Tokens).
       TokRef = TokRef.drop_front(Conflicting.size());
     }
+    // Add inactive highlighting tokens.
+    const SourceManager &SM = AST.getSourceManager();
+    for (const SourceRange &R : AST.getSkippedRanges()) {
+      if (isInsideMainFile(R.getBegin(), SM)) {
+        // Create one token for each line in the skipped range, so it works
+        // with line-based diffing.
+        Position Begin = sourceLocToPosition(SM, R.getBegin());
+        Position End = sourceLocToPosition(SM, R.getEnd());
+        assert(Begin.line <= End.line);
+        for (int Line = Begin.line; Line < End.line; ++Line) {
+          // Don't bother computing the offset for the end of the line, just use
+          // zero. The client will treat this highlighting kind specially, and
+          // highlight the entire line visually (i.e. not just to where the text
+          // on the line ends, but to the end of the screen).
+          NonConflicting.push_back({HighlightingKind::InactiveCode,
+                                    {Position{Line, 0}, Position{Line, 0}}});
+        }
+      }
+    }
+    // Re-sort the tokens because that's what the diffing expects.
+    llvm::sort(NonConflicting);
     return NonConflicting;
   }
 
   bool VisitNamespaceAliasDecl(NamespaceAliasDecl *NAD) {
     // The target namespace of an alias can not be found in any other way.
     addToken(NAD->getTargetNameLoc(), NAD->getAliasedNamespace());
     return true;
   }
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K) {
   case HighlightingKind::Namespace:
     return OS << "Namespace";
   case HighlightingKind::TemplateParameter:
     return OS << "TemplateParameter";
   case HighlightingKind::Primitive:
     return OS << "Primitive";
   case HighlightingKind::Macro:
     return OS << "Macro";
+  case HighlightingKind::InactiveCode:
+    return OS << "InactiveCode";
   }
   llvm_unreachable("invalid HighlightingKind");
 }
 
 std::vector<LineHighlightings>
 diffHighlightings(ArrayRef<HighlightingToken> New,
                   ArrayRef<HighlightingToken> Old) {
   assert(std::is_sorted(New.begin(), New.end()) &&
int NextOld = OldLine.end() != OldEnd ? OldLine.end()->R.start.line
                                           : std::numeric_limits<int>::max();
     return std::min(NextNew, NextOld);
   };
 
   for (int LineNumber = 0; NewLine.end() < NewEnd || OldLine.end() < OldEnd;
        LineNumber = NextLineNumber()) {
     NewLine = takeLine(New, NewLine.end(), LineNumber);
     OldLine = takeLine(Old, OldLine.end(), LineNumber);
-    if (NewLine != OldLine)
-      DiffedLines.push_back({LineNumber, NewLine});
+    if (NewLine != OldLine) {
+      DiffedLines.push_back({LineNumber, NewLine, /*IsInactive=*/false});
+
+      // Turn a HighlightingKind::InactiveCode token into the IsInactive flag.
+      // We do this by iterating over the tokens in the line, and if we find
+      // one with kind InactiveCode, removing that token and setting the
+      // IsInactive flag on the line instead.
+      auto &AddedLine = DiffedLines.back();
+      for (auto Iter = AddedLine.Tokens.begin();
+           Iter != AddedLine.Tokens.end();) {
+        if (Iter->Kind == HighlightingKind::InactiveCode) {
+          Iter = AddedLine.Tokens.erase(Iter);
+          AddedLine.IsInactive = true;
+        } else {
+          ++Iter;
+        }
+      }
+    }
   }
 
   return DiffedLines;
 }
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind) == std::tie(R.R, R.Kind);
 }
for (const auto &Token : Line.Tokens) {
       // |<---- 4 bytes ---->|<-- 2 bytes -->|<--- 2 bytes -->|
       // |    character      |  length       |    index       |
 
       write32be(Token.R.start.character, OS);
       write16be(Token.R.end.character - Token.R.start.character, OS);
       write16be(static_cast<int>(Token.Kind), OS);
     }
 
-    Lines.push_back({Line.Line, encodeBase64(LineByteTokens)});
+    Lines.push_back({Line.Line, encodeBase64(LineByteTokens), Line.IsInactive});
   }
 
   return Lines;
 }
 
 llvm::StringRef toTextMateScope(HighlightingKind Kind) {
   // FIXME: Add scopes for C and Objective C.
   switch (Kind) {
llvm::StringRef toTextMateScope(HighlightingKind Kind) {
   case HighlightingKind::Namespace:
     return "entity.name.namespace.cpp";
   case HighlightingKind::TemplateParameter:
     return "entity.name.type.template.cpp";
   case HighlightingKind::Primitive:
     return "storage.type.primitive.cpp";
   case HighlightingKind::Macro:
     return "entity.name.function.preprocessor.cpp";
+  case HighlightingKind::InactiveCode:
+    return "meta.disabled";
   }
   llvm_unreachable("unhandled HighlightingKind");
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/test/semantic-highlighting.test

 # CHECK-NEXT:          [
 # CHECK-NEXT:            "entity.name.type.template.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "storage.type.primitive.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "entity.name.function.preprocessor.cpp"
+# CHECK-NEXT:          ],
+# CHECK-NEXT:          [
+# CHECK-NEXT:            "meta.disabled"
 # CHECK-NEXT:          ]
 # CHECK-NEXT:        ]
 # CHECK-NEXT:      },
 ---
 {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","version":1,"text":"int x = 2;"}}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 0,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:    ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo2.cpp","languageId":"cpp","version":1,"text":"int x = 2;\nint y = 2;"}}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 0,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:    ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo2.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp","version":2},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 0,"character": 10}},"rangeLength": 0,"text": "\nint y = 2;"}]}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:   ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp","version":2},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 1,"character": 10}},"rangeLength": 11,"text": ""}]}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": ""
 # CHECK-NEXT:      }
 # CHECK-NEXT:   ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","id":3,"method":"shutdown"}
 ---
 {"jsonrpc":"2.0","method":"exit"}

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

void checkDiffedHighlights(llvm::StringRef OldCode, llvm::StringRef NewCode) {
   std::vector<LineHighlightings> ExpectedLinePairHighlighting;
   for (const HighlightingToken &Token : NewTokens) {
     auto It = ExpectedLines.find(Token.R.start.line);
     if (It != ExpectedLines.end())
       It->second.push_back(Token);
   }
   for (auto &LineTokens : ExpectedLines)
     ExpectedLinePairHighlighting.push_back(
-        {LineTokens.first, LineTokens.second});
+        {LineTokens.first, LineTokens.second, /*IsInactive = */ false});
 
   std::vector<LineHighlightings> ActualDiffed =
       diffHighlightings(NewTokens, OldTokens);
   EXPECT_THAT(ActualDiffed,
               testing::UnorderedElementsAreArray(ExpectedLinePairHighlighting))
       << OldCode;
 }
 
)cpp",
           return $TemplateParameter[[T]]::$DependentName[[Field]];
         }
       };
     )cpp",
       // Highlighting the using decl as the underlying using shadow decl.
       R"cpp(
       void $Function[[foo]]();
       using ::$Function[[foo]];
+    )cpp",
+      // Inactive code highlighting
+      R"cpp(
+      // FIXME: In the preamble, no inactive code highlightings are produced.
+      #ifdef $Macro[[test]]
+      #endif
+
+      // A declaration to cause the preamble to end.
+      int $Variable[[EndPreamble]];
+
+      // Code after the preamble.
+      // Inactive lines get an empty InactiveCode token at the beginning.
+      // Code inside inactive blocks does not get regular highlightings
+      // because it's not part of the AST.
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      int Inactive1;
+$InactiveCode[[]]      #endif
+
+      #ifndef $Macro[[test]]
+      int $Variable[[Active1]];
+      #endif
+
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      int Inactive2;
+$InactiveCode[[]]      #else
+      int $Variable[[Active2]];
+      #endif
     )cpp"};
   for (const auto &TestCase : TestCases) {
     checkHighlightings(TestCase);
   }
 
   checkHighlightings(R"cpp(
     class $Class[[A]] {
       #include "imp.h"
auto CreatePosition = [](int Line, int Character) -> Position {
     return Pos;
   };
 
   std::vector<LineHighlightings> Tokens{
       {3,
        {{HighlightingKind::Variable,
          Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
         {HighlightingKind::Function,
-         Range{CreatePosition(3, 4), CreatePosition(3, 7)}}}},
+         Range{CreatePosition(3, 4), CreatePosition(3, 7)}}},
+       /* IsInactive = */ false},
       {1,
        {{HighlightingKind::Variable,
-         Range{CreatePosition(1, 1), CreatePosition(1, 5)}}}}};
+         Range{CreatePosition(1, 1), CreatePosition(1, 5)}}},
+       /* IsInactive = */ true}};
   std::vector<SemanticHighlightingInformation> ActualResults =
       toSemanticHighlightingInformation(Tokens);
   std::vector<SemanticHighlightingInformation> ExpectedResults = {
       {3, "AAAACAAEAAAAAAAEAAMAAw=="}, {1, "AAAAAQAEAAA="}};
   EXPECT_EQ(ActualResults, ExpectedResults);
 }
 
 TEST(SemanticHighlighting, HighlightingDiffer) {