[clangd] [WIP] Semantic highlighting prototype for the vscode extension.

[jvikstrom]

Summary

Contains all the functionality for the highlighting, is going to get cleaned up a lot though. Might end up wanting to create a color theme as well, none of the default ones are that great with this...

[jvikstrom]

Readded guard against rules without scopes that somehow disappeared.

[jvikstrom]

Fixed bug that made it impossible to find colors that were a perfect match to the scope we are looking for (ex: fixes function being color as functions and not as entity.name)

Explanation for the bug: When adding the function highlighting definition the fully qualified name "entity.name.function" was added to a separate list that we did not try to match partial scope matches to.

[jvikstrom]

clang-tools-extra/clangd/clients/clangd-vscode/src/TextMate.ts

21

I'm pretty sure this matching logic is wrong. An example of the scopes for variables in the Light+ theme:

variable.css
variable.scss
variable.other.less
variable.language.wildcard.java
variable.language
storage.type.variable.cs
variable
meta.definition.variable.name
support.variable
entity.name.variable

With this kind of matching we have now this means that variable.other.less will match variable.other and variable.other.less.
As there is no perfect match for variable.other.field it will try to match variable.other and find the color for less.

What should probably be done is remove the divided map. The query function should still be the same but query on this.colors instead of this.divided.

[jvikstrom]

Fix matching logic on TM scopes.

[hokein]

clang-tools-extra/clangd/clients/clangd-vscode/src/TextMate.ts

21

The suffix of textmate indicates the language it belongs to (e.g. java), I think we can ignore non-C++ textmates, and merely find the longest-prefix match in all c++ textmates?

[jvikstrom]

clang-tools-extra/clangd/clients/clangd-vscode/src/TextMate.ts

21

But we have no real way of knowing if a suffix is a language or a more "precise" scope.
I don't think it's really possible to differentiate between variable.other and variable.css and say that variable.css is not relevant to c++ while variable.other is. (other than having an exhaustive list of language suffixes, which isn't feasible)

The implementation I changed to just saves the full scopes and than when we try to find the color of a scope we try to find it in the map, otherwise remove a suffix and try to find that scope (over and over)

Ex: Find scope for variable.other.field.cpp ->

First check for `variable.other.field.cpp` (finds nothing)
Check `variable.other.field` (finds nothing)
Check `variable.other` (still finds nothing
Check `variable` (finds the variable scope and returns that color)

This seems to work, but maybe there's some case I miss. Anyway, there is probably a defined order on how you are supposed to find these scopes, I think I read a blog post on how VSCode actually does this matching internally. I'll try to dig that up and make sure our implementation matches before it's time to put up the actual CLs for this.

[nridge]

Do you plan to support text decoration options other than color, e.g. bold / underline / italic?

Are users going to be able to change the color (and other decoration options) for specific highlightings in the VSCode Settings?

[hokein]

In D65637#1613692, @nridge wrote:

Do you plan to support text decoration options other than color, e.g. bold / underline / italic?

I think we'd just support color, and we don't have further plan to support richer renderings at the moment -- ideally, vscode should have built-in support for semantic highlighting if the proposal is accepted in LSP, this means this part of extension code will be removed in the end.

Are users going to be able to change the color (and other decoration options) for specific highlightings in the VSCode Settings?

vscode seems to provide a mechanism to override the theme settings, like

"editor.tokenColorCustomizations": {
        "[Visual Studio Dark]": {
            "textMateRules": [
                {
                    "scope": "entity.name.type",
                    "settings": {
                        "foreground": "#FF0000",
                    }
                }
            ]    
        }

I think our extension should respect it.

clang-tools-extra/clangd/clients/clangd-vscode/.gitignore

 out
 node_modules
+.vscode-test

clang-tools-extra/clangd/clients/clangd-vscode/src/SemanticHighlighting.ts

+import * as vscode from 'vscode';
+import * as vscodelc from 'vscode-languageclient';
+import * as TM from './TextMate';
+import { timingSafeEqual } from 'crypto';
+
+export namespace SemanticHighlighting {
+    interface HighlightingInformation {
+        textDocument: {
+            uri: string,
+        },
+        lines: [{
+            line: number,
+            tokens: string,
+        }],
+    }
+    interface HighlightingToken {
+        character: number,
+        scope: number,
+        length: number,
+    }
+    interface HighlightingLine {
+        line: number,
+        tokens: HighlightingToken[],
+    }
+
+    export const NotificationType = new vscodelc.NotificationType<{}, void>('textDocument/semanticHighlighting');
+
+    // The feature that should be registered in the vscode lsp for enabling experimental semantic highlighting.
+    export class Feature implements vscodelc.StaticFeature {
+        scopes: string[];
+        tokenCache: HighlightingCache = new HighlightingCache([]);
+        ready: boolean = false
+        fillClientCapabilities(capabilities: vscodelc.ClientCapabilities) {
+            // Must use an experimental capability as the other's are strongly typed.
+            const textDocumentCapabilities: vscodelc.TextDocumentClientCapabilities & { semanticHighlightingCapabilities?: { semanticHighlighting: boolean } } = capabilities.textDocument;
+            textDocumentCapabilities.semanticHighlightingCapabilities = { semanticHighlighting: true };
+        }
+
+        initialize(capabilities: vscodelc.ServerCapabilities, documentSelector: vscodelc.DocumentSelector | undefined) {
+            this.tokenCache.scopes = this.scopes;
+            // Get the scopes from an experimental capability for the same reason that it is registered as experimental.
+            const serverCapabilities: vscodelc.ServerCapabilities & { semanticHighlighting?: { scopes: string[] } } = capabilities;
+            if(!serverCapabilities.semanticHighlighting)
+                return;
+            this.scopes = serverCapabilities.semanticHighlighting.scopes;
+            TM.SemanticHighlighting.setupTMScopes(() => {
+                this.tokenCache.rerenderAll(); // Colors changed.
+            }).then(() => this.ready = true);
+        }
+
+        handleNotification(params: HighlightingInformation) {
+            this.tokenCache.scopes = this.scopes;
+            console.log('Params: ', params);
+            const tokenLines = params.lines.map((line): HighlightingLine => {
+                return {
+                    line: line.line,
+                    tokens: decodeTokens(line.tokens),
+                }
+            });
+            
+            if(this.ready)
+            this.tokenCache.setFileLines(vscode.Uri.parse(params.textDocument.uri), tokenLines);
+            else
+            console.log('Not ready to go yet...'); // TOOD: This is going to get fixed in a better way...
+        }
+    }
+
+    function decodeTokens(tokens: string): HighlightingToken[] {
+        const scopeMask = 0xFFFF;
+        const lenShift = 0x10;
+        const buf = Buffer.from(tokens, 'base64');
+        const retTokens = [];
+        for (let i = 0, end = buf.length / 4; i < end; i += 2) {
+            const start = buf.readUInt32BE(i * 4);
+            const lenKind = buf.readUInt32BE((i + 1) * 4);
+            const scope = lenKind & scopeMask;
+            const len = lenKind >>> lenShift;
+            retTokens.push({ character: start, scope: scope, length: len });
+        }
+
+        return retTokens;
+    }
+
+    export class HighlightingCache {
+        files: Map<string, Map<number, HighlightingToken[]>> = new Map();
+        renderers: Map<string, HighlightingRenderer> = new Map();
+        dispose: vscode.Disposable
+        scopes: string[]
+        constructor(scopes: string[]) {
+            this.scopes = scopes;
+            vscode.workspace.onDidCloseTextDocument((e) => this.onCloseFile(e.uri));
+            vscode.window.onDidChangeVisibleTextEditors((e) => {
+                // Let's just rerender everything
+                this.rerenderAll();
+                // And remove any texteditors that shouldn't be here
+                const actualVisible = e.map((e) => e.document.uri.toString());
+                const keys = Array.from(this.renderers.keys());
+                keys.forEach((key) => {
+                    if(actualVisible.indexOf(key) == -1)
+                        this.renderers.delete(key);
+                });
+            });
+            
+        }
+
+        setFileLines(uri: vscode.Uri, tokens: HighlightingLine[]) {
+            const uriString = uri.toString();
+            if(!this.files.has(uriString)) {
+                const m = new Map<number, HighlightingToken[]>();
+                tokens.forEach((line) => m.set(line.line, line.tokens));
+                this.files.set(uriString, m);
+            }
+            if(!this.renderers.has(uriString)) {
+                const rend = new HighlightingRenderer(uri, this.scopes);
+                this.renderers.set(uriString, rend);
+                rend.render(tokens);
+                return;
+            }
+            const m = this.files.get(uriString);
+            tokens.forEach((line) => m.set(line.line, line.tokens));
+            this.renderers.get(uriString).render(tokens);
+        }
+
+        rerenderAll() {
+            vscode.window.visibleTextEditors.forEach((e) => {
+                const uriString = e.document.uri.toString();
+                const cached: HighlightingLine[] = [];
+                if(!this.files.has(uriString)) {
+                    console.error('File does not exit?');
+                    return;
+                }
+                this.files.get(uriString).forEach((toks, line) => cached.push({line: line, tokens: toks}));
+                if(!this.renderers.has(uriString)) {
+                    this.renderers.set(uriString, new HighlightingRenderer(e.document.uri, this.scopes));
+                }
+                this.renderers.get(uriString).render(cached);
+            });
+        }
+        onCloseFile(uri: vscode.Uri) {
+            this.renderers.delete(uri.toString());
+            this.files.delete(uri.toString());
+        }
+    }
+
+    // Renders highlightings.
+    class HighlightingRenderer {
+        decorations: Map<number,vscode.TextEditorDecorationType[]> = new Map();
+        uri: vscode.Uri
+        tmColors: TM.SemanticHighlighting.TextMateColorProvider = TM.SemanticHighlighting.TextMateColorProvider.get()
+        scopes: string[]
+        constructor(uri: vscode.Uri, scopes: string[]) {
+            this.uri = uri;
+            this.scopes = scopes;
+        }
+        render(lines: HighlightingLine[]) {
+            const editor = vscode.window.visibleTextEditors.find(e => e.document.uri.toString() == this.uri.toString());
+            if(!editor)
+                return; // The TextEditor this renderer belongs to is not currently visible.
+            lines.forEach(line => {
+                const old = this.decorations.get(line.line);
+                const newLineDecorations = line.tokens.map(tok => {
+                    const options: vscode.DecorationRenderOptions = {
+                        // FIXME: This is a race condition.
+                        color: this.tmColors.getColors().getColor(this.scopes[tok.scope][0]) ,//'#ff0000',
+                        rangeBehavior: vscode.DecorationRangeBehavior.ClosedClosed,
+                    };
+                    const decoration = vscode.window.createTextEditorDecorationType(options);
+                    editor.setDecorations(decoration, [new vscode.Range(new vscode.Position(line.line, tok.character), new vscode.Position(line.line, tok.character + tok.length))]);
+                    return decoration;
+                });
+                // Must clear any previous highlightings on this line or vscode will keep stale highlightings. Does this after the new highlightings have been added so there aren't weird gaps of time where there aren't any highlightings.
+                if(old)
+                    old.forEach(o => o.dispose());
+                this.decorations.set(line.line, newLineDecorations);
+            });
+        }
+    }
+
+};

clang-tools-extra/clangd/clients/clangd-vscode/src/TextMate.ts

+import * as vscode from 'vscode';
+import * as fs from 'fs';
+import * as path from 'path';
+import * as jsonc from "jsonc-parser";
+import { SSL_OP_NETSCAPE_CA_DN_BUG } from 'constants';
+
+export namespace SemanticHighlighting {
+    class TMColors {
+        private colors: Map<string, string> = new Map();
+        setColor(scope: string | Array<string>, color: string) {
+            if (scope instanceof Array) {
+                scope.forEach((s: string) => this.setColor(s, color));
+                return;
+            }
+            this.colors.set(scope, color);
+        }
+
+        getColor(scope: string): string {
+            if(this.colors.has(scope))
+                return this.colors.get(scope);
+
+                // Here we should find the TM scope (should also probably preprocess indexes -> colors)
+            const split = scope.split('.');
+            while(split.length > 0) {
+                split.splice(-1);
+                const joined = split.join('.');
+                if(this.colors.has(joined)) {
+                    return this.colors.get(joined);
+                }
+                    
+            }
+            return '#ff0000';
+        }
+    }
+
+    // Singleton for reading/writing TM scopes/colors. Emits an event with new TMColors whenever the colors changes.
+    export class TextMateColorProvider extends vscode.EventEmitter<TMColors> {
+        // Provides a mapping from TM scopes -> colors
+        private static instance: TextMateColorProvider = new TextMateColorProvider();
+        private constructor() {
+            super();
+        }
+        private colors: TMColors = undefined;
+        static get() {
+            return TextMateColorProvider.instance;
+        }
+
+        setColors(colors: TMColors) {
+            this.colors = colors;
+            this.fire(colors);
+        }
+        getColors(): TMColors {
+            return this.colors;
+        }
+    }
+
+    // Returns a dispose function to a listener that listens for changes to the TM scopes.
+    export async function setupTMScopes(cb: Function): Promise<vscode.Disposable> {
+        async function setTMColors() {
+            const name = vscode.workspace.getConfiguration('workbench').get('colorTheme');
+            if(typeof name != 'string') {
+                console.warn('The current theme name is not a string, is: ' + (typeof name) + ', value: ', name);
+                return;
+            }
+
+            const tm = await getTextMateColors(name as string);
+            TextMateColorProvider.get().setColors(tm);
+        }
+
+        // Initialize the TM scopes and colors.
+        await setTMColors();
+
+        // Listen for changes to the theme and reload the TM scopes and colors if it changes.
+        return vscode.workspace.onDidChangeConfiguration((conf: vscode.ConfigurationChangeEvent) => {
+            if (conf.affectsConfiguration('workbench'))
+                // Configuration affected the workbench meaning the current theme might have changed.
+                setTMColors().then(() => cb());
+        });
+    }
+
+    // Gets a TM theme with themeName and returns class with a mapping from TM scopes to colors.
+    async function getTextMateColors(themeName: string): Promise<TMColors> {
+        const fileContents = await getFullNamedTheme(themeName);
+        const tmColors = new TMColors();
+        fileContents.forEach((content) => {
+            if (!content.tokenColors)
+                return;
+            content.tokenColors.forEach((rule: any) => {
+                if(!rule.scope || !rule.settings || !rule.settings.foreground)
+                    return;
+                    
+                tmColors.setColor(rule.scope, rule.settings.foreground);
+            });
+        });
+
+        return tmColors;
+    }
+
+    // Gets a TextMate theme and all its included themes.
+    async function getFullNamedTheme(themeName: string): Promise<any[]> {
+        const extension = vscode.extensions.all.find((extension: vscode.Extension<any>) => {
+            const contribs = extension.packageJSON.contributes;
+            if (!contribs || !contribs.themes)
+                return false;
+            return contribs.themes.some((theme: any) => theme.id === themeName || theme.label === themeName);
+        });
+
+        if (!extension) {
+            // Could not find an extension with the specified themeName.
+            return Promise.reject('Could not find a theme with name: ' + themeName);
+        }
+
+        const extensionInfo = extension.packageJSON.contributes.themes.find((theme: any) => theme.id === themeName || theme.label === themeName);
+        return recursiveGetTextMateGrammarPath(path.join(extension.extensionPath, extensionInfo.path));
+    }
+
+    // TM grammars can include other TM grammars, this function recursively gets all of them.
+    async function recursiveGetTextMateGrammarPath(fullPath: string): Promise<any[]> {
+        // If there is an error opening a file the TM files that were correctly found further up the chain should be returned. 
+        try {
+            const contents = await readFileText(fullPath);
+            const parsed = jsonc.parse(contents);
+            if (parsed.include)
+                // Get all includes and merge into a flat list of parsed json.
+                return [...(await recursiveGetTextMateGrammarPath(path.join(path.dirname(fullPath), parsed.include))), parsed];
+            return [parsed];
+        } catch (err) {
+            console.warn('Could not open file: ' + fullPath + ', error: ', err);
+        }
+
+        return [];
+    }
+
+    function readFileText(path: string): Promise<string> {
+        return new Promise((res, rej) => {
+            fs.readFile(path, 'utf8', (err, data) => {
+                if (err) {
+                    rej(err);
+                }
+                res(data);
+            });
+        });
+    }
+}

clang-tools-extra/clangd/clients/clangd-vscode/src/extension.ts

 import * as vscode from 'vscode';
 import * as vscodelc from 'vscode-languageclient';
+import { SemanticHighlighting } from './SemanticHighlighting';
 
 /**
  * Method to get workspace configuration option
  * @param option name of the option (e.g. for clangd.path should be path)
  * @param defaultValue default value to return if option is not set
  */
 function getConfig<T>(option: string, defaultValue?: any): T {
     const config = vscode.workspace.getConfiguration('clangd');
     return config.get<T>(option, defaultValue);
 }
 
 namespace SwitchSourceHeaderRequest {
     export const type =
-    new vscodelc.RequestType<vscodelc.TextDocumentIdentifier, string|undefined,
-                             void, void>('textDocument/switchSourceHeader');
+        new vscodelc.RequestType<vscodelc.TextDocumentIdentifier, string | undefined,
 }
 
 class FileStatus {
     private statuses = new Map<string, any>();
     private readonly statusBarItem =
         vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Left, 10);
 
     onFileUpdated(fileStatus: any) {
         const filePath = vscode.Uri.parse(fileStatus.uri);
         this.statuses.set(filePath.fsPath, fileStatus);
         this.updateStatus();
     }
 
     updateStatus() {
         const path = vscode.window.activeTextEditor.document.fileName;
         const status = this.statuses.get(path);
         if (!status) {
             this.statusBarItem.hide();
             return;
         }
         this.statusBarItem.text = `clangd: ` + status.state;
         this.statusBarItem.show();
     }
 
     clear() {
         this.statuses.clear();
         this.statusBarItem.hide();
export function activate(context: vscode.ExtensionContext) {
     const serverOptions: vscodelc.ServerOptions = clangd;
 
     // Note that CUDA ('.cu') files are special. When opening files of all other
     // extensions, VSCode would load clangd automatically. This is achieved by
     // having a corresponding 'onLanguage:...' activation event in package.json.
     // However, VSCode does not have CUDA as a supported language yet, so we
     // cannot add a corresponding activationEvent for CUDA files and clangd will
     // *not* load itself automatically on '.cu' files.
-    const cudaFilePattern: string = '**/*.{' +['cu'].join()+ '}';
+    const cudaFilePattern: string = '**/*.{' + ['cu'].join() + '}';
     const clientOptions: vscodelc.LanguageClientOptions = {
         // Register the server for c-family and cuda files.
         documentSelector: [
             { scheme: 'file', language: 'c' },
             { scheme: 'file', language: 'cpp' },
-            { scheme: 'file', language: 'objective-c'},
-            { scheme: 'file', language: 'objective-cpp'},
+            { scheme: 'file', language: 'objective-c' },
+            { scheme: 'file', language: 'objective-cpp' },
             { scheme: 'file', pattern: cudaFilePattern },
         ],
         synchronize: !syncFileEvents ? undefined : {
             // FIXME: send sync file events when clangd provides implemenatations.
         },
         initializationOptions: { clangdFileStatus: true },
         // Do not switch to output window when clangd returns output
         revealOutputChannelOn: vscodelc.RevealOutputChannelOn.Never
     };
 
-    const clangdClient = new vscodelc.LanguageClient('Clang Language Server',serverOptions, clientOptions);
+    const clangdClient = new vscodelc.LanguageClient('Clang Language Server', serverOptions, clientOptions);
+    const semanticHighlightingFeature = new SemanticHighlighting.Feature();
+    clangdClient.registerFeature(semanticHighlightingFeature);
+    // The notification handler must be registered after the client is ready or the client will crash.
+    clangdClient.onReady().then(() => clangdClient.onNotification(SemanticHighlighting.NotificationType, semanticHighlightingFeature.handleNotification.bind(semanticHighlightingFeature)));
+
     console.log('Clang Language Server is now active!');
     context.subscriptions.push(clangdClient.start());
     context.subscriptions.push(vscode.commands.registerCommand(
         'clangd-vscode.switchheadersource', async () => {
             const uri =
                 vscode.Uri.file(vscode.window.activeTextEditor.document.fileName);
             if (!uri) {
                 return;
clangdClient.onDidChangeState(
             } else if (newState == vscodelc.State.Stopped) {
                 // Clear all cached statuses when clangd crashes.
                 status.clear();
             }
         })
     // An empty place holder for the activate command, otherwise we'll get an
     // "command is not registered" error.
     context.subscriptions.push(vscode.commands.registerCommand(
-            'clangd-vscode.activate', async () => {}));
+        'clangd-vscode.activate', async () => { }));
 }

clang-tools-extra/clangd/clients/clangd-vscode/test/extension.test.ts

 /** The module 'assert' provides assertion methods from node */
 import * as assert from 'assert';
 
 import * as vscode from 'vscode';
-import * as myExtension from '../src/extension';
+import {SemanticHighlighting} from '../src/SemanticHighlighting';
 
-// TODO: add tests
 suite("Extension Tests", () => {
+    test('HighlightingCache caches incrementaly', () => {
+        const feature = new SemanticHighlighting.Feature();
+        const uri = 'file://text';
+        feature.handleNotification({
+            textDocument: {uri: uri},
+            lines: [{
+                line: 1,
+                tokens: 'AAAABAABAAA=',
+            }],
+        });
+        feature.handleNotification({
+            textDocument: {uri: uri},
+            lines: [{
+                line: 2,
+                tokens: 'AAAABAABAAA=',
+            }],
+        });
+//        const tokens = feature.tokenCache.files.get(vscode.Uri.parse(uri).toString()).tokens;
+//        assert.deepEqual(tokens.get(1), [{character: 4, length: 1, scope: 0}]);
+//        assert.deepEqual(tokens.get(2), [{character: 4, length: 1, scope: 0}]);
+        
         
-    // Defines a Mocha unit test
-    test("Something 1", () => {
-        assert.equal(-1, [1, 2, 3].indexOf(5));
-        assert.equal(-1, [1, 2, 3].indexOf(0));
     });
 });
 No newline at end of file