[clangd] Inactive regions support via dedicated protocol

[nridge]

Summary

This implements the server side of the approach discussed at
https://github.com/clangd/vscode-clangd/pull/193#issuecomment-1044315732

[nridge]

Corresponding client-side support is implemented for vscode in the latest version of https://github.com/clangd/vscode-clangd/pull/193.

[nridge]

The issue addressed by this patch is attracting a steady stream of user interest (two recent examples are https://github.com/clangd/clangd/issues/1545 and https://github.com/clangd/vscode-clangd/issues/469).

A review would be appreciated :)

[hokein]

Thanks, and sorry for late response.

The patch looks good to me in general.

clang-tools-extra/clangd/ClangdLSPServer.cpp

591

nit: add trailing // clangd extension, probably better to move to below line 585 (since we group all extension there)

clang-tools-extra/clangd/ClangdServer.cpp

120

nit: just AST.getMacros().SkppedRanges()?

974

As discussed in the GitHub thread (the experiment of highlighting the inactive regions as comment is considered as a failure), we should always not include the inactive region in the semantic highlighting, this will also simplify the existing code in semantic highlighting (e.g. https://github.com/llvm/llvm-project/blob/main/clang-tools-extra/clangd/SemanticHighlighting.cpp#L464-L513). I think we can do it in a separated patch.

clang-tools-extra/clangd/ClangdServer.h

91

it would be nice to have a unittest for this, I think it should not be to hard to add one in ClangdTests.cpp (with a customize Callbacks passing to the ClangdServer).

clang-tools-extra/clangd/Protocol.h

521

nit: notificatin => notification

[nridge]

Address review comments

[nridge]

clang-tools-extra/clangd/ClangdServer.cpp

974

I think it might make sense to keep this support around for a while, so that users whose clients do not yet support this extension still have some indication of which preprocessor branches are inactive.

However, we could make it optional, since some users (for example those who have to work on large sections of code in inactive preprocessor branches) may prefer to see the client-side colors over having it all highlighted as one color.

clang-tools-extra/clangd/ClangdServer.h

91

Good idea -- added!

[hokein]

Thanks, the implementation looks good.

clang-tools-extra/clangd/ClangdServer.cpp

974

Rendering the inactive code as comment is the best we can perform per the standard LSP spec, but it provides an suboptimal (possibly confusing) UX experience, we have received some user complains about that.

Since we are doing it via an extension, I think we should just remove the old one in the next release (17).

clang-tools-extra/clangd/Protocol.h

1743

nit: mention this is a clangd extension as well.

[nridge]

address nit

clang-tools-extra/clangd/ClangdServer.cpp

974

I think we should just remove the old one in the next release (17).

To give other clients a bit more time to implement the extension, perhaps we could:

• in v17: disable the comment tokens by default, but still have an option to turn them on
• in v18: remove it altogether?

This way, if someone using a non-vscode client has upgraded to clangd 17, but their client hasn't implemented the protocol extension yet, and they prefer to see the comment tokens (to know which branches are inactive), they can turn them on.

[nridge]

The patch is causing initialize-params.test to fail, fix coming right up.

[nridge]

In D143974#4267320, @nridge wrote:

The patch is causing initialize-params.test to fail, fix coming right up.

Fixed in https://github.com/llvm/llvm-project/commit/28575f41cd7df98012fb15f18434411a941ec228.

clang-tools-extra/clangd/ClangdLSPServer.h

 
 private:
   // Implement ClangdServer::Callbacks.
   void onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                           std::vector<Diag> Diagnostics) override;
   void onFileUpdated(PathRef File, const TUStatus &Status) override;
   void onBackgroundIndexProgress(const BackgroundQueue::Stats &Stats) override;
   void onSemanticsMaybeChanged(PathRef File) override;
+  void onInactiveRegionsReady(PathRef File,
+                              std::vector<Range> InactiveRegions) override;
 
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onInitialized(const InitializedParams &);
   void onShutdown(const NoParams &, Callback<std::nullptr_t>);
   void onSync(const NoParams &, Callback<std::nullptr_t>);
   void onDocumentDidOpen(const DidOpenTextDocumentParams &);
private:
 
   /// Outgoing LSP calls.
   LSPBinder::OutgoingMethod<ApplyWorkspaceEditParams,
                             ApplyWorkspaceEditResponse>
       ApplyWorkspaceEdit;
   LSPBinder::OutgoingNotification<ShowMessageParams> ShowMessage;
   LSPBinder::OutgoingNotification<PublishDiagnosticsParams> PublishDiagnostics;
   LSPBinder::OutgoingNotification<FileStatus> NotifyFileStatus;
+  LSPBinder::OutgoingNotification<InactiveRegionsParams> PublishInactiveRegions;
   LSPBinder::OutgoingMethod<WorkDoneProgressCreateParams, std::nullptr_t>
       CreateWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressBegin>>
       BeginWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressReport>>
       ReportWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressEnd>>
       EndWorkDoneProgress;

clang-tools-extra/clangd/ClangdLSPServer.cpp

void ClangdLSPServer::onInitialize(const InitializeParams &Params,
   SupportFileStatus = Params.initializationOptions.FileStatus;
   HoverContentFormat = Params.capabilities.HoverContentFormat;
   Opts.LineFoldingOnly = Params.capabilities.LineFoldingOnly;
   SupportsOffsetsInSignatureHelp = Params.capabilities.OffsetsInSignatureHelp;
   if (Params.capabilities.WorkDoneProgress)
     BackgroundIndexProgressState = BackgroundIndexProgress::Empty;
   BackgroundIndexSkipCreate = Params.capabilities.ImplicitProgressCreation;
   Opts.ImplicitCancellation = !Params.capabilities.CancelsStaleRequests;
+  Opts.PublishInactiveRegions = Params.capabilities.InactiveRegions;
 
   if (Opts.UseDirBasedCDB) {
     DirectoryBasedGlobalCompilationDatabase::Options CDBOpts(TFS);
     if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
       CDBOpts.CompileCommandsDir = Dir;
     CDBOpts.ContextProvider = Opts.ContextProvider;
     BaseCDB =
         std::make_unique<DirectoryBasedGlobalCompilationDatabase>(CDBOpts);
llvm::json::Object ServerCaps{
       {"typeHierarchyProvider", true},
       // Unfortunately our extension made use of the same capability name as the
       // standard. Advertise this capability to tell clients that implement our
       // extension we really have support for the standardized one as well.
       {"standardTypeHierarchyProvider", true}, // clangd extension
       {"memoryUsageProvider", true},           // clangd extension
       {"compilationDatabase",                  // clangd extension
        llvm::json::Object{{"automaticReload", true}}},
+      {"inactiveRegionsProvider", true}, // clangd extension
       {"callHierarchyProvider", true},
       {"clangdInlayHintsProvider", true},
       {"inlayHintProvider", true},
       {"foldingRangeProvider", true},
   };
 
   {
     LSPBinder Binder(Handlers, *this);
     bindMethods(Binder, Params.capabilities);
     if (Opts.FeatureModules)
       for (auto &Mod : *Opts.FeatureModules)
         Mod.initializeLSP(Binder, Params.rawCapabilities, ServerCaps);
   }
void ClangdLSPServer::bindMethods(LSPBinder &Bind,
   Bind.method("textDocument/inlayHint", this, &ClangdLSPServer::onInlayHint);
   Bind.method("$/memoryUsage", this, &ClangdLSPServer::onMemoryUsage);
   Bind.method("textDocument/foldingRange", this, &ClangdLSPServer::onFoldingRange);
   Bind.command(ApplyFixCommand, this, &ClangdLSPServer::onCommandApplyEdit);
   Bind.command(ApplyTweakCommand, this, &ClangdLSPServer::onCommandApplyTweak);
 
   ApplyWorkspaceEdit = Bind.outgoingMethod("workspace/applyEdit");
   PublishDiagnostics = Bind.outgoingNotification("textDocument/publishDiagnostics");
+  if (Caps.InactiveRegions)
+    PublishInactiveRegions = Bind.outgoingNotification("textDocument/inactiveRegions");
   ShowMessage = Bind.outgoingNotification("window/showMessage");
   NotifyFileStatus = Bind.outgoingNotification("textDocument/clangd.fileStatus");
   CreateWorkDoneProgress = Bind.outgoingMethod("window/workDoneProgress/create");
   BeginWorkDoneProgress = Bind.outgoingNotification("$/progress");
   ReportWorkDoneProgress = Bind.outgoingNotification("$/progress");
   EndWorkDoneProgress = Bind.outgoingNotification("$/progress");
   if(Caps.SemanticTokenRefreshSupport)
     SemanticTokensRefresh = Bind.outgoingMethod("workspace/semanticTokens/refresh");
// Cache FixIts
     std::lock_guard<std::mutex> Lock(FixItsMutex);
     FixItsMap[File] = LocalFixIts;
   }
 
   // Send a notification to the LSP client.
   PublishDiagnostics(Notification);
 }
 
+void ClangdLSPServer::onInactiveRegionsReady(
+    PathRef File, std::vector<Range> InactiveRegions) {
+  InactiveRegionsParams Notification;
+  Notification.TextDocument = {URIForFile::canonicalize(File, /*TUPath=*/File)};
+  Notification.InactiveRegions = std::move(InactiveRegions);
+
+  PublishInactiveRegions(Notification);
+}
+
 void ClangdLSPServer::onBackgroundIndexProgress(
     const BackgroundQueue::Stats &Stats) {
   static const char ProgressToken[] = "backgroundIndexProgress";
 
   // The background index did some work, maybe we need to cleanup
   maybeCleanupMemory();
 
   std::lock_guard<std::mutex> Lock(BackgroundIndexProgressMutex);

clang-tools-extra/clangd/ClangdServer.h

public:
 
     /// Called when the meaning of a source code may have changed without an
     /// edit. Usually clients assume that responses to requests are valid until
     /// they next edit the file. If they're invalidated at other times, we
     /// should tell the client. In particular, when an asynchronous preamble
     /// build finishes, we can provide more accurate semantic tokens, so we
     /// should tell the client to refresh.
     virtual void onSemanticsMaybeChanged(PathRef File) {}
+
+    /// Called by ClangdServer when some \p InactiveRegions for \p File are
+    /// ready.
+    virtual void onInactiveRegionsReady(PathRef File,
+                                        std::vector<Range> InactiveRegions) {}
   };
   /// Creates a context provider that loads and installs config.
   /// Errors in loading config are reported as diagnostics via Callbacks.
   /// (This is typically used as ClangdServer::Options::ContextProvider).
   static std::function<Context(PathRef)>
   createConfiguredContextProvider(const config::Provider *Provider,
                                   ClangdServer::Callbacks *);
 
struct Options {
 
     // If true, parse emplace-like functions in the preamble.
     bool PreambleParseForwardingFunctions = false;
 
     /// Whether include fixer insertions for Objective-C code should use #import
     /// instead of #include.
     bool ImportInsertions = false;
 
+    /// Whether to collect and publish information about inactive preprocessor
+    /// regions in the document.
+    bool PublishInactiveRegions = false;
+
     explicit operator TUScheduler::Options() const;
   };
   // Sensible default options for use in tests.
   // Features like indexing must be enabled if desired.
   static Options optsForTest();
 
   /// Creates a new ClangdServer instance.
   ///
private:
 
   // Whether the client supports folding only complete lines.
   bool LineFoldingOnly = false;
 
   bool PreambleParseForwardingFunctions = false;
 
   bool ImportInsertions = false;
 
+  bool PublishInactiveRegions = false;
+
   // GUARDED_BY(CachedCompletionFuzzyFindRequestMutex)
   llvm::StringMap<std::optional<FuzzyFindRequest>>
       CachedCompletionFuzzyFindRequestByFile;
   mutable std::mutex CachedCompletionFuzzyFindRequestMutex;
 
   std::optional<std::string> WorkspaceRoot;
   std::optional<AsyncTaskRunner> IndexTasks; // for stdlib indexing.
   std::optional<TUScheduler> WorkScheduler;

clang-tools-extra/clangd/ClangdServer.cpp

 namespace clang {
 namespace clangd {
 namespace {
 
 // Update the FileIndex with new ASTs and plumb the diagnostics responses.
 struct UpdateIndexCallbacks : public ParsingCallbacks {
   UpdateIndexCallbacks(FileIndex *FIndex,
                        ClangdServer::Callbacks *ServerCallbacks,
-                       const ThreadsafeFS &TFS, AsyncTaskRunner *Tasks)
+                       const ThreadsafeFS &TFS, AsyncTaskRunner *Tasks,
+                       bool CollectInactiveRegions)
       : FIndex(FIndex), ServerCallbacks(ServerCallbacks), TFS(TFS),
-        Stdlib{std::make_shared<StdLibSet>()}, Tasks(Tasks) {}
+        Stdlib{std::make_shared<StdLibSet>()}, Tasks(Tasks),
+        CollectInactiveRegions(CollectInactiveRegions) {}
 
   void onPreambleAST(PathRef Path, llvm::StringRef Version,
                      const CompilerInvocation &CI, ASTContext &Ctx,
                      Preprocessor &PP,
                      const CanonicalIncludes &CanonIncludes) override {
     // If this preamble uses a standard library we haven't seen yet, index it.
     if (FIndex)
       if (auto Loc = Stdlib->add(*CI.getLangOpts(), PP.getHeaderSearchInfo()))
void onMainAST(PathRef Path, ParsedAST &AST, PublishFn Publish) override {
 
     assert(AST.getDiagnostics() &&
            "We issue callback only with fresh preambles");
     std::vector<Diag> Diagnostics = *AST.getDiagnostics();
     if (ServerCallbacks)
       Publish([&]() {
         ServerCallbacks->onDiagnosticsReady(Path, AST.version(),
                                             std::move(Diagnostics));
+        if (CollectInactiveRegions) {
+          ServerCallbacks->onInactiveRegionsReady(
+              Path, std::move(AST.getMacros().SkippedRanges));
+        }
       });
   }
 
   void onFailedAST(PathRef Path, llvm::StringRef Version,
                    std::vector<Diag> Diags, PublishFn Publish) override {
     if (ServerCallbacks)
       Publish(
           [&]() { ServerCallbacks->onDiagnosticsReady(Path, Version, Diags); });
struct UpdateIndexCallbacks : public ParsingCallbacks {
   }
 
 private:
   FileIndex *FIndex;
   ClangdServer::Callbacks *ServerCallbacks;
   const ThreadsafeFS &TFS;
   std::shared_ptr<StdLibSet> Stdlib;
   AsyncTaskRunner *Tasks;
+  bool CollectInactiveRegions;
 };
 
 class DraftStoreFS : public ThreadsafeFS {
 public:
   DraftStoreFS(const ThreadsafeFS &Base, const DraftStore &Drafts)
       : Base(Base), DirtyFiles(Drafts) {}
 
 private:
ClangdServer::ClangdServer(const GlobalCompilationDatabase &CDB,
                            Callbacks *Callbacks)
     : FeatureModules(Opts.FeatureModules), CDB(CDB), TFS(TFS),
       DynamicIdx(Opts.BuildDynamicSymbolIndex ? new FileIndex() : nullptr),
       ClangTidyProvider(Opts.ClangTidyProvider),
       UseDirtyHeaders(Opts.UseDirtyHeaders),
       LineFoldingOnly(Opts.LineFoldingOnly),
       PreambleParseForwardingFunctions(Opts.PreambleParseForwardingFunctions),
       ImportInsertions(Opts.ImportInsertions),
+      PublishInactiveRegions(Opts.PublishInactiveRegions),
       WorkspaceRoot(Opts.WorkspaceRoot),
       Transient(Opts.ImplicitCancellation ? TUScheduler::InvalidateOnUpdate
                                           : TUScheduler::NoInvalidation),
       DirtyFS(std::make_unique<DraftStoreFS>(TFS, DraftMgr)) {
   if (Opts.AsyncThreadsCount != 0)
     IndexTasks.emplace();
   // Pass a callback into `WorkScheduler` to extract symbols from a newly
   // parsed file and rebuild the file index synchronously each time an AST
   // is parsed.
   WorkScheduler.emplace(CDB, TUScheduler::Options(Opts),
                         std::make_unique<UpdateIndexCallbacks>(
                             DynamicIdx.get(), Callbacks, TFS,
-                            IndexTasks ? &*IndexTasks : nullptr));
+                            IndexTasks ? &*IndexTasks : nullptr,
+                            PublishInactiveRegions));
   // Adds an index to the stack, at higher priority than existing indexes.
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(std::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;
     }
auto Action =
         CB(clangd::getDocumentLinks(InpAST->AST));
       };
   WorkScheduler->runWithAST("DocumentLinks", File, std::move(Action),
                             Transient);
 }
 
 void ClangdServer::semanticHighlights(
     PathRef File, Callback<std::vector<HighlightingToken>> CB) {
-  auto Action =
-      [CB = std::move(CB)](llvm::Expected<InputsAndAST> InpAST) mutable {
-        if (!InpAST)
-          return CB(InpAST.takeError());
-        CB(clangd::getSemanticHighlightings(InpAST->AST));
-      };
+
+  auto Action = [CB = std::move(CB),
+                 PublishInactiveRegions = PublishInactiveRegions](
+                    llvm::Expected<InputsAndAST> InpAST) mutable {
+    // Include inactive regions in semantic highlighting tokens only if the
+    // client doesn't support a dedicated protocol for being informed about
+    // them.
+    CB(clangd::getSemanticHighlightings(InpAST->AST, !PublishInactiveRegions));
   WorkScheduler->runWithAST("SemanticHighlights", File, std::move(Action),
                             Transient);
 }
 
 void ClangdServer::getAST(PathRef File, std::optional<Range> R,
                           Callback<std::optional<ASTNode>> CB) {
   auto Action =
       [R, CB(std::move(CB))](llvm::Expected<InputsAndAST> Inputs) mutable {

clang-tools-extra/clangd/Protocol.h

struct ClientCapabilities {
 
   /// Whether the client claims to cancel stale requests.
   /// general.staleRequestSupport.cancel
   bool CancelsStaleRequests = false;
 
   /// Whether the client implementation supports a refresh request sent from the
   /// server to the client.
   bool SemanticTokenRefreshSupport = false;
+
+  /// Whether the client supports the textDocument/inactiveRegions
+  /// notification. This is a clangd extension.
+  bool InactiveRegions = false;
 };
 bool fromJSON(const llvm::json::Value &, ClientCapabilities &,
               llvm::json::Path);
 
 /// Clangd extension that's used in the 'compilationDatabaseChanges' in
 /// workspace/didChangeConfiguration to record updates to the in-memory
 /// compilation database.
 struct ClangdCompileCommand {
struct SemanticTokensOrDelta {
   std::string resultId;
   /// Set if we computed edits relative to a previous set of tokens.
   std::optional<std::vector<SemanticTokensEdit>> edits;
   /// Set if we computed a fresh set of tokens.
   std::optional<std::vector<SemanticToken>> tokens; // encoded as integer array
 };
 llvm::json::Value toJSON(const SemanticTokensOrDelta &);
 
+/// Parameters for the inactive regions (server-side) push notification.
+/// This is a clangd extension.
+struct InactiveRegionsParams {
+  /// The textdocument these inactive regions belong to.
+  TextDocumentIdentifier TextDocument;
+  /// The inactive regions that should be sent.
+  std::vector<Range> InactiveRegions;
+};
+llvm::json::Value toJSON(const InactiveRegionsParams &InactiveRegions);
+
 struct SelectionRangeParams {
   /// The text document.
   TextDocumentIdentifier textDocument;
 
   /// The positions inside the text document.
   std::vector<Position> positions;
 };
 bool fromJSON(const llvm::json::Value &, SelectionRangeParams &,

clang-tools-extra/clangd/Protocol.cpp

bool fromJSON(const llvm::json::Value &Params, ClientCapabilities &R,
   }
   if (auto *TextDocument = O->getObject("textDocument")) {
     if (auto *SemanticHighlighting =
             TextDocument->getObject("semanticHighlightingCapabilities")) {
       if (auto SemanticHighlightingSupport =
               SemanticHighlighting->getBoolean("semanticHighlighting"))
         R.TheiaSemanticHighlighting = *SemanticHighlightingSupport;
     }
+    if (auto *InactiveRegions =
+            TextDocument->getObject("inactiveRegionsCapabilities")) {
+      if (auto InactiveRegionsSupport =
+              InactiveRegions->getBoolean("inactiveRegions")) {
+        R.InactiveRegions = *InactiveRegionsSupport;
+      }
+    }
     if (TextDocument->getObject("semanticTokens"))
       R.SemanticTokens = true;
     if (auto *Diagnostics = TextDocument->getObject("publishDiagnostics")) {
       if (auto CategorySupport = Diagnostics->getBoolean("categorySupport"))
         R.DiagnosticCategory = *CategorySupport;
       if (auto CodeActions = Diagnostics->getBoolean("codeActionsInline"))
         R.DiagnosticFixes = *CodeActions;
       if (auto RelatedInfo = Diagnostics->getBoolean("relatedInformation"))

 
 bool fromJSON(const llvm::json::Value &Params, SemanticTokensDeltaParams &R,
               llvm::json::Path P) {
   llvm::json::ObjectMapper O(Params, P);
   return O && O.map("textDocument", R.textDocument) &&
          O.map("previousResultId", R.previousResultId);
 }
 
+llvm::json::Value toJSON(const InactiveRegionsParams &InactiveRegions) {
+  return llvm::json::Object{
+      {"textDocument", InactiveRegions.TextDocument},
+      {"regions", std::move(InactiveRegions.InactiveRegions)}};
+}
+
 llvm::raw_ostream &operator<<(llvm::raw_ostream &O,
                               const DocumentHighlight &V) {
   O << V.range;
   if (V.kind == DocumentHighlightKind::Read)
     O << "(r)";
   if (V.kind == DocumentHighlightKind::Write)
     O << "(w)";
   return O;

clang-tools-extra/clangd/SemanticHighlighting.h

struct HighlightingToken {
   }
 };
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R);
 bool operator<(const HighlightingToken &L, const HighlightingToken &R);
 
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
-std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST);
+std::vector<HighlightingToken>
+getSemanticHighlightings(ParsedAST &AST, bool IncludeInactiveRegionTokens);
 
 std::vector<SemanticToken> toSemanticTokens(llvm::ArrayRef<HighlightingToken>,
                                             llvm::StringRef Code);
 llvm::StringRef toSemanticTokenType(HighlightingKind Kind);
 llvm::StringRef toSemanticTokenModifier(HighlightingModifier Modifier);
 std::vector<SemanticTokensEdit> diffTokens(llvm::ArrayRef<SemanticToken> Before,
                                            llvm::ArrayRef<SemanticToken> After);
 
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/SemanticHighlighting.cpp

resolveConflict(ArrayRef<HighlightingToken> Tokens) {
   for (size_t I = 2; Winner && I < Tokens.size(); ++I)
     Winner = resolveConflict(*Winner, Tokens[I]);
   return Winner;
 }
 
 /// Consumes source locations and maps them to text ranges for highlightings.
 class HighlightingsBuilder {
 public:
-  HighlightingsBuilder(const ParsedAST &AST)
+  HighlightingsBuilder(const ParsedAST &AST, bool IncludeInactiveRegionTokens)
       : TB(AST.getTokens()), SourceMgr(AST.getSourceManager()),
-        LangOpts(AST.getLangOpts()) {}
+        LangOpts(AST.getLangOpts()),
+        IncludeInactiveRegionTokens(IncludeInactiveRegionTokens) {}
 
   HighlightingToken &addToken(SourceLocation Loc, HighlightingKind Kind) {
     auto Range = getRangeForSourceLocation(Loc);
     if (!Range)
       return InvalidHighlightingToken;
 
     return addToken(*Range, Kind);
   }
for (ArrayRef<HighlightingToken> TokRef = Tokens; !TokRef.empty();) {
 
         NonConflicting.push_back(*Resolved);
       }
       // TokRef[Conflicting.size()] is the next token with a different range (or
       // the end of the Tokens).
       TokRef = TokRef.drop_front(Conflicting.size());
     }
 
+    if (!IncludeInactiveRegionTokens)
+      return NonConflicting;
+
     const auto &SM = AST.getSourceManager();
     StringRef MainCode = SM.getBufferOrFake(SM.getMainFileID()).getBuffer();
 
     // Merge token stream with "inactive line" markers.
     std::vector<HighlightingToken> WithInactiveLines;
     auto SortedSkippedRanges = AST.getMacros().SkippedRanges;
     llvm::sort(SortedSkippedRanges);
     auto It = NonConflicting.begin();
if (!Tok)
       return std::nullopt;
     return halfOpenToRange(SourceMgr,
                            Tok->range(SourceMgr).toCharRange(SourceMgr));
   }
 
   const syntax::TokenBuffer &TB;
   const SourceManager &SourceMgr;
   const LangOptions &LangOpts;
+  bool IncludeInactiveRegionTokens;
   std::vector<HighlightingToken> Tokens;
   std::map<Range, llvm::SmallVector<HighlightingModifier, 1>> ExtraModifiers;
   const HeuristicResolver *Resolver = nullptr;
   // returned from addToken(InvalidLoc)
   HighlightingToken InvalidHighlightingToken;
 };
 
 std::optional<HighlightingModifier> scopeModifier(const NamedDecl *D) {
bool TraverseNestedNameSpecifierLoc(NestedNameSpecifierLoc Q) {
     return RecursiveASTVisitor::TraverseNestedNameSpecifierLoc(Q);
   }
 
 private:
   HighlightingsBuilder &H;
 };
 } // namespace
 
-std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST) {
+std::vector<HighlightingToken>
+getSemanticHighlightings(ParsedAST &AST, bool IncludeInactiveRegionTokens) {
   auto &C = AST.getASTContext();
   // Add highlightings for AST nodes.
-  HighlightingsBuilder Builder(AST);
+  HighlightingsBuilder Builder(AST, IncludeInactiveRegionTokens);
   // Highlight 'decltype' and 'auto' as their underlying types.
   CollectExtraHighlightings(Builder).TraverseAST(C);
   // Highlight all decls and references coming from the AST.
   findExplicitReferences(
       C,
       [&](ReferenceLoc R) {
         for (const NamedDecl *Decl : R.Targets) {
           if (!canHighlightName(Decl->getDeclName()))

clang-tools-extra/clangd/refactor/tweaks/AnnotateHighlightings.cpp

for (auto *N = Inputs.ASTSelection.commonAncestor(); N && !CommonDecl;
        N = N->Parent)
     CommonDecl = N->ASTNode.get<Decl>();
 
   std::vector<HighlightingToken> HighlightingTokens;
   if (!CommonDecl) {
     // Now we hit the TUDecl case where commonAncestor() returns null
     // intendedly. We only annotate tokens in the main file, so use the default
     // traversal scope (which is the top level decls of the main file).
-    HighlightingTokens = getSemanticHighlightings(*Inputs.AST);
+    HighlightingTokens = getSemanticHighlightings(
+        *Inputs.AST, /*IncludeInactiveRegionTokens=*/true);
   } else {
     // Store the existing scopes.
     const auto &BackupScopes = Inputs.AST->getASTContext().getTraversalScope();
     // Narrow the traversal scope to the selected node.
     Inputs.AST->getASTContext().setTraversalScope(
         {const_cast<Decl *>(CommonDecl)});
-    HighlightingTokens = getSemanticHighlightings(*Inputs.AST);
+    HighlightingTokens = getSemanticHighlightings(
+        *Inputs.AST, /*IncludeInactiveRegionTokens=*/true);
     // Restore the traversal scope.
     Inputs.AST->getASTContext().setTraversalScope(BackupScopes);
   }
   auto &SM = Inputs.AST->getSourceManager();
   tooling::Replacements Result;
   llvm::StringRef FilePath = SM.getFilename(Inputs.Cursor);
   for (const auto &Token : HighlightingTokens) {
     assert(Token.R.start.line == Token.R.end.line &&

clang-tools-extra/clangd/tool/Check.cpp

void buildInlayHints(std::optional<Range> LineRange) {
 
     for (const auto &Hint : Hints) {
       vlog("  {0} {1} {2}", Hint.kind, Hint.position, Hint.label);
     }
   }
 
   void buildSemanticHighlighting(std::optional<Range> LineRange) {
     log("Building semantic highlighting");
-    auto Highlights = getSemanticHighlightings(*AST);
+    auto Highlights =
+        getSemanticHighlightings(*AST, /*IncludeInactiveRegionTokens=*/true);
     for (const auto HL : Highlights)
       if (!LineRange || LineRange->contains(HL.R))
         vlog(" {0} {1} {2}", HL.R, HL.Kind, HL.Modifiers);
   }
 
   // Run AST-based features at each token in the file.
   void testLocationFeatures(std::optional<Range> LineRange) {
     trace::Span Trace("testLocationFeatures");

clang-tools-extra/clangd/unittests/ClangdTests.cpp

TEST(ClangdServer, RespectsTweakFormatting) {
   Server.applyTweak(FooCpp, {}, TweakID, [&](llvm::Expected<Tweak::Effect> E) {
     ASSERT_TRUE(static_cast<bool>(E));
     EXPECT_THAT(llvm::cantFail(E->ApplyEdits.lookup(FooCpp).apply()),
                 NewContents);
     N.notify();
   });
   N.wait();
 }
+
+TEST(ClangdServer, InactiveRegions) {
+  struct InactiveRegionsCallback : ClangdServer::Callbacks {
+    std::vector<std::vector<Range>> FoundInactiveRegions;
+
+    void onInactiveRegionsReady(PathRef FIle,
+                                std::vector<Range> InactiveRegions) override {
+      FoundInactiveRegions.push_back(std::move(InactiveRegions));
+    }
+  };
+
+  MockFS FS;
+  MockCompilationDatabase CDB;
+  CDB.ExtraClangFlags.push_back("-DCMDMACRO");
+  auto Opts = ClangdServer::optsForTest();
+  Opts.PublishInactiveRegions = true;
+  InactiveRegionsCallback Callback;
+  ClangdServer Server(CDB, FS, Opts, &Callback);
+  Annotations Source(R"cpp(
+#define PREAMBLEMACRO 42
+#if PREAMBLEMACRO > 40
+  #define ACTIVE
+$inactive1[[#else
+  #define INACTIVE
+#endif]]
+int endPreamble;
+$inactive2[[#ifndef CMDMACRO
+    int inactiveInt;
+#endif]]
+#undef CMDMACRO
+$inactive3[[#ifdef CMDMACRO
+  int inactiveInt2;
+#else]]
+  int activeInt;
+#endif
+  )cpp");
+  Server.addDocument(testPath("foo.cpp"), Source.code());
+  ASSERT_TRUE(Server.blockUntilIdleForTest());
+  EXPECT_THAT(Callback.FoundInactiveRegions,
+              ElementsAre(ElementsAre(Source.range("inactive1"),
+                                      Source.range("inactive2"),
+                                      Source.range("inactive3"))));
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

void checkHighlightings(llvm::StringRef Code,
   TU.ExtraArgs.push_back("-std=c++20");
   TU.ExtraArgs.push_back("-xobjective-c++");
   TU.ExtraArgs.insert(std::end(TU.ExtraArgs), std::begin(AdditionalArgs),
                       std::end(AdditionalArgs));
 
   for (auto File : AdditionalFiles)
     TU.AdditionalFiles.insert({File.first, std::string(File.second)});
   auto AST = TU.build();
-  auto Actual = getSemanticHighlightings(AST);
+  auto Actual =
+      getSemanticHighlightings(AST, /*IncludeInactiveRegionTokens=*/true);
   for (auto &Token : Actual)
     Token.Modifiers &= ModifierMask;
 
   EXPECT_EQ(Code, annotate(Test.code(), Actual));
 }
 
 constexpr static uint32_t ScopeModifierMask =
     1 << unsigned(HighlightingModifier::FunctionScope) |