[clangd] Inactive regions support as an extension to semantic highlighting

[nridge]

This patch adds server-side support for greying out code in inactive preprocessor branches (issue #132).

I didn't write test yet. I wanted to post the patch for feedback first, to see if the general approach is ok.

[nridge]

For reference, I also posted a proof-of-concept client impl in D67537, which I used to test/validate this patch.

[sammccall]

Rather than a separate method with parallel implementation, this seems very closely related to the syntax highlighting feature.

The minimal way to model this (no new protocol) would be for each disabled line, to add one token spanning the whole line with a TextMate scope like comment.disabled or meta.disabled.
There's no technical reason tokens can't overlap in the protocol, though it's possible editors will choose to turn off local heuristic highlighting on anything that's touched by an LSP-provided token.

Failing that, I'd suggest encoding a list of line-styles on SemanticHighlightingInformation, that should be combined with any tokens on that line.

[nridge]

Ok, I can try formulating this as part of / an extension to semantic highlighting.

[nridge]

Reformulate as an extension to semantic highlighting

[nridge]

The updated patch formulates this as a new semantic highlighting kind.

The tokens created for inactive regions are one per line, with the character offset and length being zero; the idea is that the client will handle this highlighting specially (see D67537) and highlight the entire line.

Note, highlighting the entire line is different than highlighting just the portion of the line that contains text. We want the grey background highlight to extend to the entire row in the editor, even over columns beyond the end of the text line (as accomplished with vscode's isWholeLine=true option in DecorationRenderOptions).

I don't love special-casing the scope in the client, but this does avoid having to extend the protocol itself which would be considerably more work.

[hokein]

Could you add tests for this?

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

I'm not sure how does this flag impact the size of Preamble/AST, @ilya-biryukov any thoughts?

clang-tools-extra/clangd/SemanticHighlighting.cpp

194

I think the current implementation only collects the skipped preprocessing ranges after preamble region in the main file. We probably need to store these ranges in PreambleData to make the ranges in the preamble region work, no need to do it in this patch, but we'd better have a test reflecting this fact.

#ifdef ActiveCOde
// inactive code here
#endif

#include "foo.h"
// preamble ends here

namespace ns {
// ..
}

203

This seems too couple with VSCode client, I would prefer to calculate the range of the line and return to the client.

Is there any big differences in VSCode between highlighting with the isWholeLine and highlighting with the range of the line?

clang-tools-extra/clangd/SemanticHighlighting.h

45

This is a different kind group, I would put it after the Macro, we'd need to update the LastKind.

The name seems too specific, how about "UnreachableCode"?

[nridge]

In D67536#1691107, @hokein wrote:

Could you add tests for this?

Certainly, I just wanted to discuss the general approach first, as it will affect what the tests look like.

clang-tools-extra/clangd/SemanticHighlighting.cpp

203

I took some screenshots to illustrate to difference.

Highlighting only to the end of the line of text:

Highlighting the whole line:

I think the first one looks pretty bad, and is inconsistent with existing practice.

Note also that the suggestion is not to special-case the VSCode client specifically; it's to special-case one particular highlighting, which any client can implement.

If this special-casing is really unpalatable, we could instead try this suggestion by @sammccall:

Failing that, I'd suggest encoding a list of line-styles on SemanticHighlightingInformation, that should be combined with any tokens on that line.

I guess one consideration when evaluating these options is, do we expect to use that "list of line-styles" for anything else in the future? I can't think of anything at the moment, but perhaps there are other uses for it.

If not, we could do something slightly simpler, and add a single isInactive flag to SemanticHighlightingInformation.

[MaskRay]

Why "inactive region", not "skipped ranges"?

[ilya-biryukov]

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

Have no idea, but why do we need this in the first place?
PPCallbacks::SourceRangeSkipped should allow to record all skipped ranges in the main file. Can we use it?

[nridge]

In D67536#1694098, @MaskRay wrote:

Why "inactive region", not "skipped ranges"?

I got the name from "$cquery/publishInactiveRegions" :) But I don't particularly care what we call it.

[nridge]

Update to use PPCallbacks::SourceRangeSkipped

[sammccall]

clang-tools-extra/clangd/SemanticHighlighting.cpp

191

I think this comment could be clearer, e.g. // Add tokens indicating lines skipped by the preprocessor.

203

Three approaches seem feasible here:

1. clients that know about the specific scope can extend it to the whole line.
2. [0,0) or so indicates "highlight the whole line"
3. use a dedicated property for line styles (vs token styles)

3 is clearly better than 2 I think, it's more explicit. I don't have a strong opinion of 1 vs 3, but if going with 1 I think it's a good idea to measure the line as Haojian says, so we at least get a basic version of the feature if the client doesn't know about line styles.

I guess one consideration when evaluating these options is, do we expect to use that "list of line-styles" for anything else in the future? I can't think of anything at the moment

Preprocessor directives maybe? (Though these are easy enough for clients to highlight with regex)

[nridge]

clang-tools-extra/clangd/Compiler.cpp

66 ↗

(On Diff #222336)

Yes, PPCallbacks::SourceRangeSkipped works in place of using DetailedRecord. Thank you for the suggestion.

clang-tools-extra/clangd/SemanticHighlighting.h

45

I changed it to "InactiveCode". ("Unreachable" seemed like the wrong word, it brings to mind control flow.)

[nridge]

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

[sammccall]

In D67536#1697533, @nridge wrote:

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

If you look at functions like offsetToPosition in SourceCode.h, basically you get the buffer as a string (contains utf-8), find the offset you're interested in, and then zoom around looking for \n. Once you have a substring, calling lspLength() on it will give you the length in UTF-16 or whatever LSP is speaking at the moment.

[hokein]

In D67536#1697533, @nridge wrote:

How would one even measure the line length? SourceManager doesn't sem to have a method like getLineLength() or similar.

Yes, there is no existing API for that, I think you'd need to get the source code from the SM at the specific offset, and do the \n counting.

clang-tools-extra/clangd/ParsedAST.h

100 ↗

(On Diff #223577)

Instead of adding new member and methods in ParsedAST, I think we can do it in CollectMainFileMacros (adding a new field SkippRanges in MainFileMacros), then we can get the skipped ranges for preamble region within the main file for free.

clang-tools-extra/clangd/SemanticHighlighting.cpp

203

I can't say whether highlighting the line is better than highlighting the range of the line text, but below is the how the inactive TS code is highlighted in VSCode (only the range of text), I personally prefer this style.

[nridge]

One thing that may be worth considering as well, is that if the client prefers to highlight the text of the line only, it can calculate the length of the line itself. In VSCode for instance, the line lengths are readily available; I imagine other editors are similar since they need that information for many purposes.

[sammccall]

In D67536#1697696, @nridge wrote:

One thing that may be worth considering as well, is that if the client prefers to highlight the text of the line only, it can calculate the length of the line itself. In VSCode for instance, the line lengths are readily available; I imagine other editors are similar since they need that information for many purposes.

So I don't think clients will/should prefer that - for best rendering they should know this is a line highlight.

I think this comes down to how line highlights are represented in the protocol:

• by a separate field: no need to send line length
• by a special token bounds (e.g. [0,0)): no need to send line length
• by a special scope: sending line length is a nice-to-have as it provides graceful degradation for clients that don't understand this extension

clang-tools-extra/clangd/SemanticHighlighting.cpp

203

I think that's an argument for making sure clients clearly distinguish between regular tokens and marking lines: overlapping tokens don't compose well, but we can easily say lines and token styles should compose.

(That particular style is not for me, but it doesn't matter)

[ilya-biryukov]

In D67536#1697825, @sammccall wrote:

So I don't think clients will/should prefer that - for best rendering they should know this is a line highlight.

I have actually seen clients that just make the text gray and it looks pretty nice (ReSharper for Visual Studio and IntelliJ IDEA definitely do that).
It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

So I wouldn't be so sure, at least some clients might prefer to do exclusively range (as opposed to line-based) highlightings.

PS Sorry if this was discussed before and I'm just missing context.

[nridge]

clang-tools-extra/clangd/SemanticHighlighting.cpp

203

I find this a convincing argument for using line styles, thanks.

Especially since, at some point in the future, I would like us to be able to produce regular token highlightings for inactive code, much like in that TS code screenshot.

[nridge]

In D67536#1699221, @ilya-biryukov wrote:

I have actually seen clients that just make the text gray and it looks pretty nice (ReSharper for Visual Studio and IntelliJ IDEA definitely do that).

I don't think there is a conflict between this style, and using line highlightings: if the style being applied is a foreground color (gray text) as opposed to a background color, then it will look the same regardless of whether you apply it as a line style, or as a character range style.

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

If we later introduce highlightings for dead expressions or statements, we can of course use character ranges and not lines for them.

[ilya-biryukov]

In D67536#1700872, @nridge wrote:

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

Note that the opposite is true - inactive preprocessor branches can be expressed as range-based highlightings.

If we later introduce highlightings for dead expressions or statements, we can of course use character ranges and not lines for them.

Then we'll have two modes of operation - line-based and range-based highlightings. Clients will have to support both, we'll have code dealing with both.
The alternative is having only range-based highlightings and using them everywhere. That's the main reason I would advocate for range-based.

[ilya-biryukov]

Just noticed the next version of LSP added diagnostic tags for things like "unused field" or "dead code":
https://microsoft.github.io/language-server-protocol/specifications/specification-3-15, search for DiagnosticTag.

So I guess we won't need ever add range-based highlightings for dead code (and similar), we'll use diagnostics instead.

Concerns about having two kinds of highlightings to handle on the clients are still there, though.

[nridge]

In D67536#1701038, @ilya-biryukov wrote:

In D67536#1700872, @nridge wrote:

It also lets them consistently highlight part of the line (e.g. dead expressions or statements can be marked in gray even if they are on the same line).

Highlighting part of a line is not applicable to inactive preprocessor branches in C++.

Note that the opposite is true - inactive preprocessor branches can be expressed as range-based highlightings.

Except that if we want to allow background styling, the client would need to special-case the highlighting scope to know to apply the background style to the entire line.

[nridge]

Updated to use as isInactive flag on SemanticHighlightingInformation

[nridge]

Ok, I updated the patch to convey the line highlight separately from the token highlights.

I did use the simplified approach I mentioned, where I use a single isInactive flag. If you'd prefer the more general approach where we send an array of line styles, I can do that, though my personal preference would be to wait until we have an actual use case before generalizing.

There are still no tests yet, as I'm still not sure whether we have agreement on the approach. If you like the approach, please let me know, and I'll follow up with tests.

[nridge]

Review / feedback ping :)

[ilya-biryukov]

Just a few NITs from my side. I'll let the other review the actual implementation

clang-tools-extra/clangd/ParsedAST.cpp

217 ↗

(On Diff #224480)

Maybe keep only ranges from the main file?

clang-tools-extra/clangd/ParsedAST.h

100 ↗

(On Diff #224480)

NIT: please return llvm::ArrayRef<SourceRange> instead

[nridge]

Addressed nits

[hokein]

Thanks, I'm fine with the current approach, feel free to add unittests.

clang-tools-extra/clangd/Protocol.h

1212

could you add some documentation describing this is a clangd extension?

clang-tools-extra/clangd/SemanticHighlighting.cpp

431

it took me a while to understand this code,

If the NewLine is IsInactive, it just contains a single token whose range is [0, 0), can't we just?

if (NewLine.back().Tokens.empty()) continue;

bool InactiveLine = NewLine.back().Tokens.front().Kind == InactiveCode;
assert(InactiveLine && NewLine.back().Tokens.size() == 1 && "IncativeCode must have a single token");
DiffedLines.back().IsInactive = true;

clang-tools-extra/clangd/SemanticHighlighting.h

46

we should document this Kind as it is different than other kinds, it is for line-style I believe?

[nridge]

Addressed some nits and got existing tests to pass

Will follow up with new tests in the next update

clang-tools-extra/clangd/SemanticHighlighting.cpp

431

An inactive line can contain token highlightings as well. For example, we highlight macro references in the condition of an #ifdef, and that line is also inactive if the condition is false. Clients can merge the line style (which is typically a background color) with the token styles (typically a foreground color).

I did expand the comment to explain what the loop is doing more clearly.

[nridge]

Add unit tests

[nridge]

Address one minor remaining comment

clang-tools-extra/clangd/SemanticHighlighting.cpp

194

Your observation is correct: the current implementation only highlights inactive lines after the preamble. For now, I added a test case with a FIXME for this.

[hokein]

clang-tools-extra/clangd/ParsedAST.h

100 ↗

(On Diff #223577)

This comment is undone.

clang-tools-extra/clangd/SemanticHighlighting.cpp

194

I'd prefer to fix it in this patch, it should not require large effort, and probably simplify the patch, you don't need to implement a new PPCallbacks. See my another comment.

[nridge]

clang-tools-extra/clangd/ParsedAST.h

100 ↗

(On Diff #223577)

My apologies, I overlooked this.

[nridge]

Support preamble as well

[hokein]

thanks, mostly good, a few more nits.

clang-tools-extra/clangd/ParsedAST.h

134 ↗

(On Diff #229707)

nit: it is not used anymore, remove it.

clang-tools-extra/clangd/Protocol.h

1212

could you document that the line-style highlightings can be combined with any token-style highlightings?

clang-tools-extra/clangd/SemanticHighlighting.cpp

431

thanks, I see.

I think we can still simplify the code like below, this could improve the code readability, and avoid the comment explaining it.

llvm::erase_if(AddedLine.Tokens, [](const Token& T) { return T.Kind == InactiveCode;});

[nridge]

Address nits

clang-tools-extra/clangd/SemanticHighlighting.cpp

431

Done. Note that we still need to set AddedLine.IsInactive appropriately. I did that in the lambda, which feels like a strange thing for an erase_if predicate to do. The alternative would be doing a count_if first (but then we're looping over the tokens twice).

[hokein]

looks good, thanks!

clang-tools-extra/clangd/CollectMacros.h

struct MainFileMacros {
   llvm::StringSet<> Names;
   // Instead of storing SourceLocation, we have to store the token range because
   // SourceManager from preamble is not available when we build the AST.
   llvm::DenseMap<SymbolID, std::vector<Range>> MacroRefs;
   // Somtimes it is not possible to compute the SymbolID for the Macro, e.g. a
   // reference to an undefined macro. Store them separately, e.g. for semantic
   // highlighting.
   std::vector<Range> UnknownMacros;
+  // Ranges skipped by the preprocessor due to being inactive.
+  std::vector<Range> SkippedRanges;
 };
 
 /// Collects macro references (e.g. definitions, expansions) in the main file.
 /// It is used to:
 ///  - collect macros in the preamble section of the main file (in Preamble.cpp)
 ///  - collect macros after the preamble of the main file (in ParsedAST.cpp)
 class CollectMainFileMacros : public PPCallbacks {
 public:
void Ifndef(SourceLocation Loc, const Token &MacroName,
     add(MacroName, MD.getMacroInfo());
   }
 
   void Defined(const Token &MacroName, const MacroDefinition &MD,
                SourceRange Range) override {
     add(MacroName, MD.getMacroInfo());
   }
 
+  void SourceRangeSkipped(SourceRange R, SourceLocation EndifLoc) override {
+    if (!InMainFile)
+      return;
+    Position Begin = sourceLocToPosition(SM, R.getBegin());
+    Position End = sourceLocToPosition(SM, R.getEnd());
+    Out.SkippedRanges.push_back(Range{Begin, End});
+  }
+
 private:
   void add(const Token &MacroNameTok, const MacroInfo *MI) {
     if (!InMainFile)
       return;
     auto Loc = MacroNameTok.getLocation();
     if (Loc.isMacroID())
       return;
 

clang-tools-extra/clangd/Protocol.h

 
 /// Represents a semantic highlighting information that has to be applied on a
 /// specific line of the text document.
 struct SemanticHighlightingInformation {
   /// The line these highlightings belong to.
   int Line = 0;
   /// The base64 encoded string of highlighting tokens.
   std::string Tokens;
+  /// Is the line in an inactive preprocessor branch?
+  /// This is a clangd extension.
+  /// An inactive line can still contain highlighting tokens as well;
+  /// clients should combine line style and token style if possible.
+  bool IsInactive = false;
 };
 bool operator==(const SemanticHighlightingInformation &Lhs,
                 const SemanticHighlightingInformation &Rhs);
 llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting);
 
 /// Parameters for the semantic highlighting (server-side) push notification.
 struct SemanticHighlightingParams {
   /// The textdocument these highlightings belong to.

clang-tools-extra/clangd/Protocol.cpp

 
 bool operator==(const SemanticHighlightingInformation &Lhs,
                 const SemanticHighlightingInformation &Rhs) {
   return Lhs.Line == Rhs.Line && Lhs.Tokens == Rhs.Tokens;
 }
 
 llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting) {
   return llvm::json::Object{{"line", Highlighting.Line},
-                            {"tokens", Highlighting.Tokens}};
+                            {"tokens", Highlighting.Tokens},
+                            {"isInactive", Highlighting.IsInactive}};
 }
 
 llvm::json::Value toJSON(const SemanticHighlightingParams &Highlighting) {
   return llvm::json::Object{
       {"textDocument", Highlighting.TextDocument},
       {"lines", std::move(Highlighting.Lines)},
   };
 }

clang-tools-extra/clangd/SemanticHighlighting.h

enum class HighlightingKind {
   Enum,
   EnumConstant,
   Typedef,
   DependentType,
   DependentName,
   Namespace,
   TemplateParameter,
   Primitive,
   Macro,
 
-  LastKind = Macro
+  // This one is different from the other kinds as it's a line style
+  // rather than a token style.
+  InactiveCode,
+
+  LastKind = InactiveCode
 };
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K);
 
 // Contains all information needed for the highlighting a token.
 struct HighlightingToken {
   HighlightingKind Kind;
   Range R;
 };
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R);
 bool operator<(const HighlightingToken &L, const HighlightingToken &R);
 
 /// Contains all information about highlightings on a single line.
 struct LineHighlightings {
   int Line;
   std::vector<HighlightingToken> Tokens;
+  bool IsInactive;
 };
 
 bool operator==(const LineHighlightings &L, const LineHighlightings &R);
 
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST);
 

clang-tools-extra/clangd/SemanticHighlighting.cpp

 #include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/AST/Type.h"
 #include "clang/AST/TypeLoc.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/Casting.h"
 #include <algorithm>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 /// Some names are not written in the source code and cannot be highlighted,
void addToken(SourceLocation Loc, HighlightingKind Kind) {
     if (!Range) {
       // R should always have a value, if it doesn't something is very wrong.
       elog("Tried to add semantic token with an invalid range");
       return;
     }
     Tokens.push_back(HighlightingToken{Kind, *Range});
   }
 
-  std::vector<HighlightingToken> collect() && {
+  std::vector<HighlightingToken> collect(ParsedAST &AST) && {
     // Initializer lists can give duplicates of tokens, therefore all tokens
     // must be deduplicated.
     llvm::sort(Tokens);
     auto Last = std::unique(Tokens.begin(), Tokens.end());
     Tokens.erase(Last, Tokens.end());
 
     // Macros can give tokens that have the same source range but conflicting
     // kinds. In this case all tokens sharing this source range should be
for (ArrayRef<HighlightingToken> TokRef = Tokens; !TokRef.empty();) {
       // If there is exactly one token with this range it's non conflicting and
       // should be in the highlightings.
       if (Conflicting.size() == 1)
         NonConflicting.push_back(TokRef.front());
       // TokRef[Conflicting.size()] is the next token with a different range (or
       // the end of the Tokens).
       TokRef = TokRef.drop_front(Conflicting.size());
     }
+    // Add tokens indicating lines skipped by the preprocessor.
+    for (const Range &R : AST.getMacros().SkippedRanges) {
+      // Create one token for each line in the skipped range, so it works
+      // with line-based diffing.
+      assert(R.start.line <= R.end.line);
+      for (int Line = R.start.line; Line < R.end.line; ++Line) {
+        // Don't bother computing the offset for the end of the line, just use
+        // zero. The client will treat this highlighting kind specially, and
+        // highlight the entire line visually (i.e. not just to where the text
+        // on the line ends, but to the end of the screen).
+        NonConflicting.push_back({HighlightingKind::InactiveCode,
+                                  {Position{Line, 0}, Position{Line, 0}}});
+      }
+    }
+    // Re-sort the tokens because that's what the diffing expects.
+    llvm::sort(NonConflicting);
     return NonConflicting;
   }
 
 private:
   const SourceManager &SourceMgr;
   const LangOptions &LangOpts;
   std::vector<HighlightingToken> Tokens;
 };
std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST) {
   // Add highlightings for macro references.
   for (const auto &SIDToRefs : AST.getMacros().MacroRefs) {
     for (const auto &M : SIDToRefs.second)
       Builder.addToken({HighlightingKind::Macro, M});
   }
   for (const auto &M : AST.getMacros().UnknownMacros)
     Builder.addToken({HighlightingKind::Macro, M});
 
-  return std::move(Builder).collect();
+  return std::move(Builder).collect(AST);
 }
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K) {
   switch (K) {
   case HighlightingKind::Variable:
     return OS << "Variable";
   case HighlightingKind::LocalVariable:
     return OS << "LocalVariable";
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K) {
   case HighlightingKind::Namespace:
     return OS << "Namespace";
   case HighlightingKind::TemplateParameter:
     return OS << "TemplateParameter";
   case HighlightingKind::Primitive:
     return OS << "Primitive";
   case HighlightingKind::Macro:
     return OS << "Macro";
+  case HighlightingKind::InactiveCode:
+    return OS << "InactiveCode";
   }
   llvm_unreachable("invalid HighlightingKind");
 }
 
 std::vector<LineHighlightings>
 diffHighlightings(ArrayRef<HighlightingToken> New,
                   ArrayRef<HighlightingToken> Old) {
   assert(std::is_sorted(New.begin(), New.end()) &&
int NextOld = OldLine.end() != OldEnd ? OldLine.end()->R.start.line
                                           : std::numeric_limits<int>::max();
     return std::min(NextNew, NextOld);
   };
 
   for (int LineNumber = 0; NewLine.end() < NewEnd || OldLine.end() < OldEnd;
        LineNumber = NextLineNumber()) {
     NewLine = takeLine(New, NewLine.end(), LineNumber);
     OldLine = takeLine(Old, OldLine.end(), LineNumber);
-    if (NewLine != OldLine)
-      DiffedLines.push_back({LineNumber, NewLine});
+    if (NewLine != OldLine) {
+      DiffedLines.push_back({LineNumber, NewLine, /*IsInactive=*/false});
+
+      // Turn a HighlightingKind::InactiveCode token into the IsInactive flag.
+      auto &AddedLine = DiffedLines.back();
+      llvm::erase_if(AddedLine.Tokens, [&](const HighlightingToken &T) {
+        if (T.Kind == HighlightingKind::InactiveCode) {
+          AddedLine.IsInactive = true;
+          return true;
+        }
+        return false;
+      });
+    }
   }
 
   return DiffedLines;
 }
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind) == std::tie(R.R, R.Kind);
 }
for (const auto &Token : Line.Tokens) {
       // |<---- 4 bytes ---->|<-- 2 bytes -->|<--- 2 bytes -->|
       // |    character      |  length       |    index       |
 
       write32be(Token.R.start.character, OS);
       write16be(Token.R.end.character - Token.R.start.character, OS);
       write16be(static_cast<int>(Token.Kind), OS);
     }
 
-    Lines.push_back({Line.Line, encodeBase64(LineByteTokens)});
+    Lines.push_back({Line.Line, encodeBase64(LineByteTokens), Line.IsInactive});
   }
 
   return Lines;
 }
 
 llvm::StringRef toTextMateScope(HighlightingKind Kind) {
   // FIXME: Add scopes for C and Objective C.
   switch (Kind) {
llvm::StringRef toTextMateScope(HighlightingKind Kind) {
   case HighlightingKind::Namespace:
     return "entity.name.namespace.cpp";
   case HighlightingKind::TemplateParameter:
     return "entity.name.type.template.cpp";
   case HighlightingKind::Primitive:
     return "storage.type.primitive.cpp";
   case HighlightingKind::Macro:
     return "entity.name.function.preprocessor.cpp";
+  case HighlightingKind::InactiveCode:
+    return "meta.disabled";
   }
   llvm_unreachable("unhandled HighlightingKind");
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/test/semantic-highlighting.test

 # CHECK-NEXT:          [
 # CHECK-NEXT:            "entity.name.type.template.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "storage.type.primitive.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "entity.name.function.preprocessor.cpp"
+# CHECK-NEXT:          ],
+# CHECK-NEXT:          [
+# CHECK-NEXT:            "meta.disabled"
 # CHECK-NEXT:          ]
 # CHECK-NEXT:        ]
 # CHECK-NEXT:      },
 ---
 {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","version":1,"text":"int x = 2;"}}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 0,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:    ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo2.cpp","languageId":"cpp","version":1,"text":"int x = 2;\nint y = 2;"}}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 0,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:    ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo2.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp","version":2},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 0,"character": 10}},"rangeLength": 0,"text": "\nint y = 2;"}]}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": "AAAABAABAAA="
 # CHECK-NEXT:      }
 # CHECK-NEXT:   ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp","version":2},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 1,"character": 10}},"rangeLength": 11,"text": ""}]}}
 #      CHECK:  "method": "textDocument/semanticHighlighting",
 # CHECK-NEXT:  "params": {
 # CHECK-NEXT:    "lines": [
 # CHECK-NEXT:      {
+# CHECK-NEXT:        "isInactive": false,
 # CHECK-NEXT:        "line": 1,
 # CHECK-NEXT:        "tokens": ""
 # CHECK-NEXT:      }
 # CHECK-NEXT:   ],
 # CHECK-NEXT:    "textDocument": {
 # CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp"
 # CHECK-NEXT:    }
 # CHECK-NEXT:  }
 # CHECK-NEXT:}
 ---
 {"jsonrpc":"2.0","id":3,"method":"shutdown"}
 ---
 {"jsonrpc":"2.0","method":"exit"}

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

void checkDiffedHighlights(llvm::StringRef OldCode, llvm::StringRef NewCode) {
   std::vector<LineHighlightings> ExpectedLinePairHighlighting;
   for (const HighlightingToken &Token : NewTokens) {
     auto It = ExpectedLines.find(Token.R.start.line);
     if (It != ExpectedLines.end())
       It->second.push_back(Token);
   }
   for (auto &LineTokens : ExpectedLines)
     ExpectedLinePairHighlighting.push_back(
-        {LineTokens.first, LineTokens.second});
+        {LineTokens.first, LineTokens.second, /*IsInactive = */ false});
 
   std::vector<LineHighlightings> ActualDiffed =
       diffHighlightings(NewTokens, OldTokens);
   EXPECT_THAT(ActualDiffed,
               testing::UnorderedElementsAreArray(ExpectedLinePairHighlighting))
       << OldCode;
 }
 
)cpp",
       // highlighting all macro references
       R"cpp(
       #ifndef $Macro[[name]]
       #define $Macro[[name]]
       #endif
 
       #define $Macro[[test]]
       #undef $Macro[[test]]
-      #ifdef $Macro[[test]]
-      #endif
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      #endif
 
-      #if defined($Macro[[test]])
-      #endif
+$InactiveCode[[]]      #if defined($Macro[[test]])
+$InactiveCode[[]]      #endif
     )cpp",
       R"cpp(
       struct $Class[[S]] {
         float $Field[[Value]];
         $Class[[S]] *$Field[[Next]];
       };
       $Class[[S]] $Variable[[Global]][2] = {$Class[[S]](), $Class[[S]]()};
       void $Function[[f]]($Class[[S]] $Parameter[[P]]) {
)cpp",
       // Highlighting of template template arguments.
       R"cpp(
       template <template <class> class $TemplateParameter[[TT]],
                 template <class> class ...$TemplateParameter[[TTs]]>
       struct $Class[[Foo]] {
         $Class[[Foo]]<$TemplateParameter[[TT]], $TemplateParameter[[TTs]]...>
           *$Field[[t]];
       }
+    )cpp",
+      // Inactive code highlighting
+      R"cpp(
+      // Code in the preamble.
+      // Inactive lines get an empty InactiveCode token at the beginning.
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      #endif
+
+      // A declaration to cause the preamble to end.
+      int $Variable[[EndPreamble]];
+
+      // Code after the preamble.
+      // Code inside inactive blocks does not get regular highlightings
+      // because it's not part of the AST.
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      int Inactive2;
+$InactiveCode[[]]      #endif
+
+      #ifndef $Macro[[test]]
+      int $Variable[[Active1]];
+      #endif
+
+$InactiveCode[[]]      #ifdef $Macro[[test]]
+$InactiveCode[[]]      int Inactive3;
+$InactiveCode[[]]      #else
+      int $Variable[[Active2]];
+      #endif
     )cpp"};
   for (const auto &TestCase : TestCases) {
     checkHighlightings(TestCase);
   }
 
   checkHighlightings(R"cpp(
     class $Class[[A]] {
       #include "imp.h"
auto CreatePosition = [](int Line, int Character) -> Position {
     return Pos;
   };
 
   std::vector<LineHighlightings> Tokens{
       {3,
        {{HighlightingKind::Variable,
          Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
         {HighlightingKind::Function,
-         Range{CreatePosition(3, 4), CreatePosition(3, 7)}}}},
+         Range{CreatePosition(3, 4), CreatePosition(3, 7)}}},
+       /* IsInactive = */ false},
       {1,
        {{HighlightingKind::Variable,
-         Range{CreatePosition(1, 1), CreatePosition(1, 5)}}}}};
+         Range{CreatePosition(1, 1), CreatePosition(1, 5)}}},
+       /* IsInactive = */ true}};
   std::vector<SemanticHighlightingInformation> ActualResults =
       toSemanticHighlightingInformation(Tokens);
   std::vector<SemanticHighlightingInformation> ExpectedResults = {
       {3, "AAAACAAEAAAAAAAEAAMAAw=="}, {1, "AAAAAQAEAAA="}};
   EXPECT_EQ(ActualResults, ExpectedResults);
 }
 
 TEST(SemanticHighlighting, HighlightingDiffer) {