[LLDB} Added syntax highlighting support

[teemperor]

Summary

This patch adds syntax highlighting support to LLDB. When enabled (and lldb is allowed
to use colors), printed source code is annotated with the ANSI color escape sequences.

So far we have only one highlighter which is based on Clang and is responsible for all
languages that are supported by Clang. It essentially just runs the raw lexer over the input
and then surrounds the specific tokens with the configured escape sequences.

[teemperor]

Also: This patch doesn't answer the question how the user will be able to configure the specific colors used for each token (i.e. there is only one style available at the moment). The reason for this is that there seems to be a lot of opinions about whether we want to flood the settings with highlighting-specific fields or if we just provide a fixed set of common styles. So I would prefer if we could defer the discussion about this to another patch that will address this (Unless we somehow magically achieve consensus on this one).

[zturner]

Ansi escape codes do not work on Windows. LLVM’s stream output classes have
color support that does work in a cross platform manner. Can we use those
instead?

[labath]

We're also trying to avoid adding new clang-specific code to the debugger core. I think it would make more sense if the (clang-based) c++ highlighter was provided by some plugin. I see a couple of options:

• the c++ language plugin: I think this is the most low-level plugin that is still language specific. However, it is specific to c(++), whereas here you format other languages too.
• the clang expression parser plugin: it's a bit of a stretch, but syntax higlighting is a kind of expression parsing
• a completely new plugin

[teemperor]

@zturner We can migrate the existing AnsiTerminal.h to reuse the LLVM coloring backend. This way we fix also the code that already uses this convenient interface.

@labath I think we can add to the Language class the option to add its related syntax highlighting support. I'll check if/how that would work.

(Thanks for the reviews!)

[teemperor]

[Updated patch to address Pavel's comments, thanks!]

@zturner So I looked into the Windows support:

Windows requires us to directly flush/signal/write/flush to the console output stream. However lldb's output is buffered by design into a StreamString first to prevent overlapping with the process output. So we can't just add Windows support to the coloring backend as we don't have direct access to the output stream.

If we want to fix this in lldb, then we could write an ANSI color code interpreter and let that run over our final output while we print it. That wouldn't be too complex and would fix all existing coloring output in lldb. It would also fix that lldb configs that enable color support on Windows are broken (because people just add color codes there).

However, it seems Windows starting with W10 anyway supports ANSI color codes (or at least you can enable them in the settings). So that interpreter is only really necessary until everyone moved to a version that supports color codes: https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#span-idtextformattingspanspan-idtextformattingspanspan-idtextformattingspantext-formatting

So I would suggest we maybe just hack in the color interpreter and drop it when W7 reaches EoL (?). I can make another patch for that if it sounds good.

[teemperor]

Actually, we also have to tear down the plugins, so let me fix that first.

[teemperor]

• Removed some leftover code from the refactoring.
• Correctly SetUp/TearDown the test case.

[labath]

The patch is bigger than ideal for a single change, but I like the way it is structured. Thank you for abstracting the clang specifics.

The one high-level question that came to mind when looking this over is whether we really need to do all this file name matching to get the language type. I would expect we normally display the source code we have debug info for. Is that ever not the case? If it is, then we should be able to just get the language from the debug info and avoid relying on the file names. Have you looked at whether that is possible?

include/lldb/Core/Highlighter.h

50

given stream

150

s/hat/that/

source/Core/Highlighter.cpp

17–18

Are these clang includes used here? If they are, this entire abstraction exercise will have been for nothing.

30

This call isn't exactly cheap. Maybe you could just call the function once during initialization and just store the result?

35

This isn't correct, as you're not writing m_prefix, but it's transmogrified version. Btw, do you really need this return value anyway?

source/Plugins/Language/ClangCommon/ClangHighlighter.cpp

22

Is this used?

156

Do you need to make a copy of the data here? It looks like both of these objects get destroyed at the end of this function anyway.

unittests/Language/Highlighting/HighlighterTest.cpp

133–134

You can just do EXPECT_EQ("", highlightC("", s)). EXPECT_EQ uses operator==, which does the right thing when one of the args is a std::string.

[teemperor]

• Addressed Pavel's comments. Also cleaned up some more includes.

Yes, knowing the language we currently display would be nice, but I didn't see a good way to reliably get this information. The File class doesn't know anything about the meaning of its contents. And the SourceManager can be given a SymbolContextList, but looking at the test coverage that list is often just a nullptr.

So I went with the file name matching as it doesn't have any surprising side effects (like starting to demangle symbols) or any special requirements to the caller (like already knowing the language type/symbol context of what we will display).

I think it's a reasonable fallback that should always work and we can later add functionality to pass a language to the SourceManager/File/etc., if the caller has that information available.

(Also thanks for the review).

[teemperor]

source/Core/Highlighter.cpp

35

Good catch. And the return value is just to make the SourceManager happy which always returns the total amount of bytes written. I'm working on a patch that will move all the 'written byte counting' in lldb into the Stream class, but as of now that's how it works.

[labath]

Thanks for the explanation. This looks fine to me, though I would feel better if someone else gave it a look too.

source/Core/Highlighter.cpp

30

extra semicolons

35

That sounds like a good idea. When you do that, could you please refer to the llvm raw_ostream classes to see how they handle this. Long term, it would be great if we could replace lldb's classes with those, so I'd like to avoid diverging from them if possible.

[teemperor]

• Removed extra semicolons that slipped in somehow.

[teemperor]

source/Core/Highlighter.cpp

35

raw_ostream has the same functionality with tell(), so that is actually in line with the idea of moving to LLVM's stream classes.

include/lldb/Core/Debugger.h

public:
   bool GetUseExternalEditor() const;
 
   bool SetUseExternalEditor(bool use_external_editor_p);
 
   bool GetUseColor() const;
 
   bool SetUseColor(bool use_color);
 
+  bool GetHighlightSource() const;
+
   lldb::StopShowColumn GetStopShowColumn() const;
 
   const FormatEntity::Entry *GetStopShowColumnAnsiPrefix() const;
 
   const FormatEntity::Entry *GetStopShowColumnAnsiSuffix() const;
 
   uint32_t GetStopSourceLineCount(bool before) const;
 

include/lldb/Core/Highlighter.h

+//===-- Highlighter.h -------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef liblldb_Highlighter_h_
+#define liblldb_Highlighter_h_
+
+#include <utility>
+#include <vector>
+
+#include "lldb/Utility/Stream.h"
+#include "lldb/lldb-enumerations.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace lldb_private {
+
+//----------------------------------------------------------------------
+/// Represents style that the highlighter should apply to the given source code.
+/// Stores information about how every kind of token should be annotated.
+//----------------------------------------------------------------------
+struct HighlightStyle {
+
+  //----------------------------------------------------------------------
+  /// A pair of strings that should be placed around a certain token. Usually
+  /// stores color codes in these strings (the suffix string is often used for
+  /// resetting the terminal attributes back to normal).
+  //----------------------------------------------------------------------
+  class ColorStyle {
+    std::string m_prefix;
+    std::string m_suffix;
+
+  public:
+    ColorStyle() = default;
+    ColorStyle(llvm::StringRef prefix, llvm::StringRef suffix) {
+      Set(prefix, suffix);
+    }
+
+    /// Applies this style to the given value.
+    /// \param s
+    ///     The stream to which the result should be appended.
+    /// \param value
+    ///     The value that we should place our strings around.
+    /// \return
+    ///     The number of bytes that have been written to the given string.
+    std::size_t Apply(Stream &s, llvm::StringRef value) const;
+
+    /// Sets the prefix and suffix strings.
+    /// @param prefix
+    /// @param suffix
+    void Set(llvm::StringRef prefix, llvm::StringRef suffix) {
+      m_prefix = prefix;
+      m_suffix = suffix;
+    }
+  };
+
+  /// Matches identifiers to variable or functions.
+  ColorStyle identifier;
+  /// Matches any string or character literals in the language: "foo" or 'f'
+  ColorStyle string_literal;
+  /// Matches scalar value literals like '42' or '0.1'.
+  ColorStyle scalar_literal;
+  /// Matches all reserved keywords in the language.
+  ColorStyle keyword;
+  /// Matches any comments in the language.
+  ColorStyle comment;
+  /// Matches commas: ','
+  ColorStyle comma;
+  /// Matches one colon: ':'
+  ColorStyle colon;
+  /// Matches any semicolon: ';'
+  ColorStyle semicolons;
+  /// Matches operators like '+', '-', '%', '&', '='
+  ColorStyle operators;
+
+  /// Matches '{' or '}'
+  ColorStyle braces;
+  /// Matches '[' or ']'
+  ColorStyle square_brackets;
+  /// Matches '(' or ')'
+  ColorStyle parentheses;
+
+  //-----------------------------------------------------------------------
+  // C language specific options
+  //-----------------------------------------------------------------------
+
+  /// Matches directives to a preprocessor (if the language has any).
+  ColorStyle pp_directive;
+
+  /// Returns a HighlightStyle that is based on vim's default highlight style.
+  static HighlightStyle MakeVimStyle();
+};
+
+//----------------------------------------------------------------------
+/// Annotates source code with color attributes.
+//----------------------------------------------------------------------
+class Highlighter {
+public:
+  Highlighter() = default;
+  virtual ~Highlighter() = default;
+  DISALLOW_COPY_AND_ASSIGN(Highlighter);
+
+  /// Returns a human readable name for the selected highlighter.
+  virtual llvm::StringRef GetName() const = 0;
+
+  /// Highlights the given line
+  /// \param options
+  /// \param line
+  ///     The user supplied line that needs to be highlighted.
+  /// \param previous_lines
+  ///     Any previous lines the user has written which we should only use
+  ///     for getting the context of the Highlighting right.
+  /// \param s
+  ///     The stream to which the highlighted version of the user string should
+  ///     be written.
+  /// \return
+  ///     The number of bytes that have been written to the stream.
+  virtual std::size_t Highlight(const HighlightStyle &options,
+                                llvm::StringRef line,
+                                llvm::StringRef previous_lines,
+                                Stream &s) const = 0;
+
+  /// Utility method for calling Highlight without a stream.
+  std::string Highlight(const HighlightStyle &options, llvm::StringRef line,
+                        llvm::StringRef previous_lines = "") const;
+};
+
+/// A default highlighter that does nothing. Used as a fallback.
+class NoHighlighter : public Highlighter {
+public:
+  llvm::StringRef GetName() const override { return "none"; }
+
+  std::size_t Highlight(const HighlightStyle &options, llvm::StringRef line,
+                        llvm::StringRef previous_lines,
+                        Stream &s) const override;
+};
+
+/// Manages the available highlighters.
+class HighlighterManager {
+  /// All available highlighters in the list of priority.
+  std::vector<std::unique_ptr<Highlighter>> m_highlighters;
+  NoHighlighter m_no_highlighter;
+
+public:
+  HighlighterManager();
+  /// Queries all known highlighter for one that can highlight some source code.
+  /// \param language_type
+  ///     The language type hat the caller thinks the source code was given in.
+  /// \param path
+  ///     The path to the file the source code is from. Used as a fallback when
+  ///     the user can't provide a language.
+  /// \return
+  ///     The highlighter that wants to highlight the source code. Could be an
+  ///     empty highlighter that does nothing.
+  const Highlighter &getHighlighterFor(lldb::LanguageType language_type,
+                                       llvm::StringRef path) const;
+};
+
+} // namespace lldb_private
+
+#endif // liblldb_Highlighter_h_

include/lldb/Target/Language.h

 // C++ Includes
 #include <functional>
 #include <memory>
 #include <set>
 #include <vector>
 
 // Other libraries and framework includes
 // Project includes
+#include "lldb/Core/Highlighter.h"
 #include "lldb/Core/PluginInterface.h"
 #include "lldb/DataFormatters/DumpValueObjectOptions.h"
 #include "lldb/DataFormatters/FormatClasses.h"
 #include "lldb/DataFormatters/StringPrinter.h"
 #include "lldb/lldb-private.h"
 #include "lldb/lldb-public.h"
 
 namespace lldb_private {
enum class FunctionNameRepresentation {
     eNameWithArgs,
     eNameWithNoArgs
   };
 
   ~Language() override;
 
   static Language *FindPlugin(lldb::LanguageType language);
 
+  /// Returns the Language associated with the given file path or a nullptr
+  /// if there is no known language.
+  static Language *FindPlugin(llvm::StringRef file_path);
+
+  static Language *FindPlugin(lldb::LanguageType language,
+                              llvm::StringRef file_path);
+
   // return false from callback to stop iterating
   static void ForEach(std::function<bool(Language *)> callback);
 
   virtual lldb::LanguageType GetLanguageType() const = 0;
 
   virtual bool IsTopLevelFunction(Function &function);
 
+  virtual bool IsSourceFile(llvm::StringRef file_path) const = 0;
+
+  virtual const Highlighter *GetHighlighter() const { return nullptr; }
+
   virtual lldb::TypeCategoryImplSP GetFormatters();
 
   virtual HardcodedFormatters::HardcodedFormatFinder GetHardcodedFormats();
 
   virtual HardcodedFormatters::HardcodedSummaryFinder GetHardcodedSummaries();
 
   virtual HardcodedFormatters::HardcodedSyntheticFinder
   GetHardcodedSynthetics();

packages/Python/lldbsuite/test/source-manager/TestSourceManager.py

 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 def ansi_underline_surround_regex(inner_regex_text):
     # return re.compile(r"\[4m%s\[0m" % inner_regex_text)
     return "4.+\033\\[4m%s\033\\[0m" % inner_regex_text
 
+def ansi_color_surround_regex(inner_regex_text):
+    return "\033\\[3[0-7]m%s\033\\[0m" % inner_regex_text
 
 class SourceManagerTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     NO_DEBUG_INFO_TESTCASE = True
 
     def setUp(self):
def get_expected_stop_column_number(self):
         stop_line = get_line(self.file, self.line)
         # The number of spaces that must be skipped to get to the first non-
         # whitespace character --- where we expect the debugger breakpoint
         # column to be --- is equal to the number of characters that get
         # stripped off the front when we lstrip it, plus one to specify
         # the character column after the initial whitespace.
         return len(stop_line) - len(stop_line.lstrip()) + 1
 
-    def do_display_source_python_api(self, use_color, column_marker_regex):
+    def do_display_source_python_api(self, use_color, needle_regex, highlight_source=False):
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         target = self.dbg.CreateTarget(exe)
         self.assertTrue(target, VALID_TARGET)
 
         # Launch the process, and do not stop at the entry point.
         args = None
         envp = None
         process = target.LaunchSimple(
             args, envp, self.get_process_working_directory())
         self.assertIsNotNone(process)
 
         #
         # Exercise Python APIs to display source lines.
         #
 
         # Setup whether we should use ansi escape sequences, including color
         # and styles such as underline.
         self.dbg.SetUseColor(use_color)
+        # Disable syntax highlighting if needed.
+
+        self.runCmd("settings set highlight-source " + str(highlight_source).lower())
 
         filespec = lldb.SBFileSpec(self.file, False)
         source_mgr = self.dbg.GetSourceManager()
         # Use a string stream as the destination.
         stream = lldb.SBStream()
         column = self.get_expected_stop_column_number()
         context_before = 2
         context_after = 2
         current_line_prefix = "=>"
         source_mgr.DisplaySourceLinesWithLineNumbersAndColumn(
             filespec, self.line, column, context_before, context_after,
             current_line_prefix, stream)
 
         #    2
         #    3    int main(int argc, char const *argv[]) {
         # => 4        printf("Hello world.\n"); // Set break point at this line.
         #    5        return 0;
         #    6    }
-        self.expect(stream.GetData(), "Source code displayed correctly",
+        self.expect(stream.GetData(), "Source code displayed correctly:\n" + stream.GetData(),
                     exe=False,
                     patterns=['=> %d.*Hello world' % self.line,
-                              column_marker_regex])
+                              needle_regex])
 
         # Boundary condition testings for SBStream().  LLDB should not crash!
         stream.Print(None)
         stream.RedirectToFile(None, True)
 
     @add_test_categories(['pyapi'])
     def test_display_source_python_dumb_terminal(self):
         """Test display of source using the SBSourceManager API, using a
         dumb terminal and thus no color support (the default)."""
         use_color = False
         self.do_display_source_python_api(use_color, r"\s+\^")
 
     @add_test_categories(['pyapi'])
     def test_display_source_python_ansi_terminal(self):
         """Test display of source using the SBSourceManager API, using a
         dumb terminal and thus no color support (the default)."""
         use_color = True
         underline_regex = ansi_underline_surround_regex(r".")
         self.do_display_source_python_api(use_color, underline_regex)
 
+    @add_test_categories(['pyapi'])
+    def test_display_source_python_ansi_terminal_syntax_highlighting(self):
+        """Test display of source using the SBSourceManager API and check for
+        the syntax highlighted output"""
+        use_color = True
+        syntax_highlighting = True;
+
+        # Just pick 'int' as something that should be colored.
+        color_regex = ansi_color_surround_regex("int")
+        self.do_display_source_python_api(use_color, color_regex, syntax_highlighting)
+
+        # Same for 'char'.
+        color_regex = ansi_color_surround_regex("char")
+        self.do_display_source_python_api(use_color, color_regex, syntax_highlighting)
+
+        # Test that we didn't color unrelated identifiers.
+        self.do_display_source_python_api(use_color, r" printf\(", syntax_highlighting)
+
     def test_move_and_then_display_source(self):
         """Test that target.source-map settings work by moving main.c to hidden/main.c."""
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Move main.c to hidden/main.c.
         hidden = self.getBuildArtifact("hidden")

source/Core/CMakeLists.txt

add_lldb_library(lldbCore
   DumpDataExtractor.cpp
   DumpRegisterValue.cpp
   DynamicLoader.cpp
   EmulateInstruction.cpp
   Event.cpp
   FileLineResolver.cpp
   FileSpecList.cpp
   FormatEntity.cpp
+  Highlighter.cpp
   IOHandler.cpp
   Listener.cpp
   Mangled.cpp
   Module.cpp
   ModuleChild.cpp
   ModuleList.cpp
   Opcode.cpp
   PluginManager.cpp

source/Core/Debugger.cpp

static PropertyDefinition g_properties[] = {
     {"stop-line-count-after", OptionValue::eTypeSInt64, true, 3, nullptr,
      nullptr,
      "The number of sources lines to display that come after the "
      "current source line when displaying a stopped context."},
     {"stop-line-count-before", OptionValue::eTypeSInt64, true, 3, nullptr,
      nullptr,
      "The number of sources lines to display that come before the "
      "current source line when displaying a stopped context."},
+    {"highlight-source", OptionValue::eTypeBoolean, true, true, nullptr,
+     nullptr, "If true, LLDB will highlight the displayed source code."},
     {"stop-show-column", OptionValue::eTypeEnum, false,
      eStopShowColumnAnsiOrCaret, nullptr, s_stop_show_column_values,
      "If true, LLDB will use the column information from the debug info to "
      "mark the current position when displaying a stopped context."},
     {"stop-show-column-ansi-prefix", OptionValue::eTypeFormatEntity, true, 0,
      DEFAULT_STOP_SHOW_COLUMN_ANSI_PREFIX, nullptr,
      "When displaying the column marker in a color-enabled (i.e. ANSI) "
      "terminal, use the ANSI terminal code specified in this format at the "
enum {
   ePropertyFrameFormat,
   ePropertyNotiftVoid,
   ePropertyPrompt,
   ePropertyScriptLanguage,
   ePropertyStopDisassemblyCount,
   ePropertyStopDisassemblyDisplay,
   ePropertyStopLineCountAfter,
   ePropertyStopLineCountBefore,
+  ePropertyHighlightSource,
   ePropertyStopShowColumn,
   ePropertyStopShowColumnAnsiPrefix,
   ePropertyStopShowColumnAnsiSuffix,
   ePropertyTerminalWidth,
   ePropertyThreadFormat,
   ePropertyThreadStopFormat,
   ePropertyUseExternalEditor,
   ePropertyUseColor,

 
 bool Debugger::SetUseColor(bool b) {
   const uint32_t idx = ePropertyUseColor;
   bool ret = m_collection_sp->SetPropertyAtIndexAsBoolean(nullptr, idx, b);
   SetPrompt(GetPrompt());
   return ret;
 }
 
+bool Debugger::GetHighlightSource() const {
+  const uint32_t idx = ePropertyHighlightSource;
+  return m_collection_sp->GetPropertyAtIndexAsBoolean(
+      nullptr, idx, g_properties[idx].default_uint_value);
+}
+
 StopShowColumn Debugger::GetStopShowColumn() const {
   const uint32_t idx = ePropertyStopShowColumn;
   return (lldb::StopShowColumn)m_collection_sp->GetPropertyAtIndexAsEnumeration(
       nullptr, idx, g_properties[idx].default_uint_value);
 }
 
 const FormatEntity::Entry *Debugger::GetStopShowColumnAnsiPrefix() const {
   const uint32_t idx = ePropertyStopShowColumnAnsiPrefix;

source/Core/Highlighter.cpp

+//===-- Highlighter.cpp -----------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Core/Highlighter.h"
+
+#include "lldb/Target/Language.h"
+#include "lldb/Utility/AnsiTerminal.h"
+#include "lldb/Utility/StreamString.h"
+
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/ADT/StringSet.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+#include <unordered_set>
+
+using namespace lldb_private;
+
+std::size_t HighlightStyle::ColorStyle::Apply(Stream &s,
+                                              llvm::StringRef value) const {
+  // If we have no prefix, skip the expensive call below.
+  if (!m_prefix.empty())
+    s << lldb_utility::ansi::FormatAnsiTerminalCodes(m_prefix);
+  s << value;
+  if (!m_suffix.empty())
+    s << lldb_utility::ansi::FormatAnsiTerminalCodes(m_suffix);
+  // Calculate how many bytes we have written.
+  return m_prefix.size() + value.size() + m_suffix.size();
+}
+
+std::size_t NoHighlighter::Highlight(const HighlightStyle &options,
+                                     llvm::StringRef line,
+                                     llvm::StringRef previous_lines,
+                                     Stream &s) const {
+  // We just forward the input to the output and do no highlighting.
+  s << line;
+  return line.size();
+}
+
+static HighlightStyle::ColorStyle GetColor(const char *c) {
+  return HighlightStyle::ColorStyle(c, "${ansi.normal}");
+}
+
+HighlightStyle HighlightStyle::MakeVimStyle() {
+  HighlightStyle result;
+  result.comment = GetColor("${ansi.fg.purple}");
+  result.scalar_literal = GetColor("${ansi.fg.red}");
+  result.keyword = GetColor("${ansi.fg.green}");
+  return result;
+}
+
+class ClangHighlighter : public Highlighter {
+  llvm::StringSet<> keywords;
+  /// Returns true if the given string represents a keywords in any Clang
+  /// supported language.
+  bool isKeyword(llvm::StringRef token) const;
+
+  /// Determines which style should be applied to the given token.
+  /// \param token
+  ///     The current token.
+  /// \param tok_str
+  ///     The string in the source code the token represents.
+  /// \param options
+  ///     The style we use for coloring the source code.
+  /// \param in_pp_directive
+  ///     If we are currently in a preprocessor directive. NOTE: This is
+  ///     passed by reference and will be updated if the current token starts
+  ///     or ends a preprocessor directive.
+  /// \return
+  ///     The ColorStyle that should be applied to the token.
+  HighlightStyle::ColorStyle determineClangStyle(const clang::Token &token,
+                                                 llvm::StringRef tok_str,
+                                                 const HighlightStyle &options,
+                                                 bool &in_pp_directive) const;
+
+public:
+  ClangHighlighter();
+  llvm::StringRef GetName() const override { return "clang"; }
+
+  std::size_t Highlight(const HighlightStyle &options, llvm::StringRef line,
+                        llvm::StringRef previous_lines,
+                        Stream &s) const override;
+  bool ShouldHighlightFile(lldb::LanguageType language,
+                           llvm::StringRef path) const override;
+};
+
+bool ClangHighlighter::isKeyword(llvm::StringRef token) const {
+  return keywords.find(token) != keywords.end();
+}
+
+ClangHighlighter::ClangHighlighter() {
+#define KEYWORD(X, N) keywords.insert(#X);
+#include "clang/Basic/TokenKinds.def"
+}
+
+HighlightStyle::ColorStyle ClangHighlighter::determineClangStyle(
+    const clang::Token &token, llvm::StringRef tok_str,
+    const HighlightStyle &options, bool &in_pp_directive) const {
+  using namespace clang;
+
+  if (token.is(tok::comment)) {
+    // If we were in a preprocessor directive before, we now left it.
+    in_pp_directive = false;
+    return options.comment;
+  } else if (in_pp_directive || token.getKind() == tok::hash) {
+    // Let's assume that the rest of the line is a PP directive.
+    in_pp_directive = true;
+    // Preprocessor directives are hard to match, so we have to hack this in.
+    return options.pp_directive;
+  } else if (tok::isStringLiteral(token.getKind()))
+    return options.string_literal;
+  else if (tok::isLiteral(token.getKind()))
+    return options.scalar_literal;
+  else if (isKeyword(tok_str))
+    return options.keyword;
+  else
+    switch (token.getKind()) {
+    case tok::raw_identifier:
+    case tok::identifier:
+      return options.identifier;
+    case tok::l_brace:
+    case tok::r_brace:
+      return options.braces;
+    case tok::l_square:
+    case tok::r_square:
+      return options.square_brackets;
+    case tok::l_paren:
+    case tok::r_paren:
+      return options.parentheses;
+    case tok::comma:
+      return options.comma;
+    case tok::coloncolon:
+    case tok::colon:
+      return options.colon;
+
+    case tok::amp:
+    case tok::ampamp:
+    case tok::ampequal:
+    case tok::star:
+    case tok::starequal:
+    case tok::plus:
+    case tok::plusplus:
+    case tok::plusequal:
+    case tok::minus:
+    case tok::arrow:
+    case tok::minusminus:
+    case tok::minusequal:
+    case tok::tilde:
+    case tok::exclaim:
+    case tok::exclaimequal:
+    case tok::slash:
+    case tok::slashequal:
+    case tok::percent:
+    case tok::percentequal:
+    case tok::less:
+    case tok::lessless:
+    case tok::lessequal:
+    case tok::lesslessequal:
+    case tok::spaceship:
+    case tok::greater:
+    case tok::greatergreater:
+    case tok::greaterequal:
+    case tok::greatergreaterequal:
+    case tok::caret:
+    case tok::caretequal:
+    case tok::pipe:
+    case tok::pipepipe:
+    case tok::pipeequal:
+    case tok::question:
+    case tok::equal:
+    case tok::equalequal:
+      return options.operators;
+    default:
+      break;
+    }
+  return HighlightStyle::ColorStyle();
+}
+
+std::size_t ClangHighlighter::Highlight(const HighlightStyle &options,
+                                        llvm::StringRef line,
+                                        llvm::StringRef previous_lines,
+                                        Stream &result) const {
+  using namespace clang;
+
+  std::size_t written_bytes = 0;
+
+  FileSystemOptions file_opts;
+  FileManager file_mgr(file_opts);
+
+  unsigned line_number = previous_lines.count('\n') + 1U;
+
+  // Let's build the actual source code Clang needs and setup some utility
+  // objects.
+  std::string full_source = previous_lines.str() + line.str();
+  llvm::IntrusiveRefCntPtr<DiagnosticIDs> diag_ids(new DiagnosticIDs());
+  llvm::IntrusiveRefCntPtr<DiagnosticOptions> diags_opts(
+      new DiagnosticOptions());
+  DiagnosticsEngine diags(diag_ids, diags_opts);
+  clang::SourceManager SM(diags, file_mgr);
+  std::unique_ptr<llvm::MemoryBuffer> buf =
+      llvm::MemoryBuffer::getMemBufferCopy(full_source);
+
+  FileID FID = SM.createFileID(clang::SourceManager::Unowned, buf.get());
+
+  // Let's just enable the latest ObjC and C++ which should get most tokens
+  // right.
+  LangOptions Opts;
+  Opts.ObjC2 = true;
+  Opts.CPlusPlus17 = true;
+  Opts.LineComment = true;
+
+  Lexer lex(FID, buf.get(), SM, Opts);
+  // The lexer should keep whitespace around.
+  lex.SetKeepWhitespaceMode(true);
+
+  // Keeps track if we have entered a PP directive.
+  bool in_pp_directive = false;
+
+  // True once we actually lexed the user provided line.
+  bool found_user_line = false;
+
+  Token token;
+  bool exit = false;
+  while (!exit) {
+    // Returns true if this is the last token we get from the lexer.
+    exit = lex.LexFromRawLexer(token);
+
+    bool invalid = false;
+    unsigned current_line_number =
+        SM.getSpellingLineNumber(token.getLocation(), &invalid);
+    if (current_line_number != line_number)
+      continue;
+    found_user_line = true;
+
+    // We don't need to print any tokens without a spelling line number.
+    if (invalid)
+      continue;
+
+    // Same as above but with the column number.
+    invalid = false;
+    unsigned start = SM.getSpellingColumnNumber(token.getLocation(), &invalid);
+    if (invalid)
+      continue;
+    // Column numbers start at 1, but indexes in our string start at 0.
+    --start;
+
+    // Annotations don't have a length, so let's skip them.
+    if (token.isAnnotation())
+      continue;
+
+    // Extract the token string from our source code.
+    llvm::StringRef tok_str = line.substr(start, token.getLength());
+
+    // If the token is just an empty string, we can skip all the work below.
+    if (tok_str.empty())
+      continue;
+
+    // See how we are supposed to highlight this token.
+    HighlightStyle::ColorStyle color =
+        determineClangStyle(token, tok_str, options, in_pp_directive);
+
+    written_bytes += color.Apply(result, tok_str);
+  }
+
+  // If we went over the whole file but couldn't find our own file, then
+  // somehow our setup was wrong. When we're in release mode we just give the
+  // user the normal line and pretend we don't know how to highlight it. In
+  // debug mode we bail out with an assert as this should never happen.
+  if (!found_user_line) {
+    result << line;
+    written_bytes += line.size();
+    assert(false && "We couldn't find the user line in the input file?");
+  }
+
+  return written_bytes;
+}
+
+bool ClangHighlighter::ShouldHighlightFile(lldb::LanguageType language,
+                                           llvm::StringRef path) const {
+  switch (language) {
+  case lldb::LanguageType::eLanguageTypeC:
+  case lldb::LanguageType::eLanguageTypeObjC:
+  case lldb::LanguageType::eLanguageTypeObjC_plus_plus:
+  case lldb::LanguageType::eLanguageTypeC89:
+  case lldb::LanguageType::eLanguageTypeC99:
+  case lldb::LanguageType::eLanguageTypeC11:
+  case lldb::LanguageType::eLanguageTypeC_plus_plus:
+  case lldb::LanguageType::eLanguageTypeC_plus_plus_03:
+  case lldb::LanguageType::eLanguageTypeC_plus_plus_11:
+  case lldb::LanguageType::eLanguageTypeC_plus_plus_14:
+  case lldb::LanguageType::eLanguageTypeOpenCL:
+    return true;
+  default:
+    break;
+  }
+
+  // User didn't provide any language, so we have to guess based on the file
+  // path.
+  const auto suffixes = {".cpp", ".cxx", ".c++", ".cc",  ".c",
+                         ".h",   ".hh",  ".hpp", ".hxx", ".h++"};
+  for (auto suffix : suffixes) {
+    if (path.endswith_lower(suffix))
+      return true;
+  }
+
+  // One final effort to check if we're in the STL path and should highlight.
+  return path.contains("/c++/");
+}
+
+HighlighterManager::HighlighterManager() {
+  m_highlighters.push_back(llvm::make_unique<ClangHighlighter>());
+
+  // Our final highlighter will always match and just do nothing.
+  m_highlighters.push_back(llvm::make_unique<NoHighlighter>());
+}
+
+const Highlighter &
+HighlighterManager::getHighlighterFor(lldb::LanguageType language_type,
+                                      llvm::StringRef path) const {
+  Language *language = lldb_private::Language::FindPlugin(language_type, path);
+  if (language && language->GetHighlighter())
+    return *language->GetHighlighter();
+  return m_no_highlighter;
+}
+
+std::string Highlighter::Highlight(const HighlightStyle &options,
+                                   llvm::StringRef line,
+                                   llvm::StringRef previous_lines) const {
+  StreamString s;
+  Highlight(options, line, previous_lines, s);
+  s.Flush();
+  return s.GetString().str();
+}

source/Core/SourceManager.cpp

 //===-- SourceManager.cpp ---------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Core/SourceManager.h"
 
 #include "lldb/Core/Address.h"      // for Address
 #include "lldb/Core/AddressRange.h" // for AddressRange
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/FormatEntity.h" // for FormatEntity
+#include "lldb/Core/Highlighter.h"
 #include "lldb/Core/Module.h"
 #include "lldb/Core/ModuleList.h" // for ModuleList
 #include "lldb/Host/FileSystem.h"
 #include "lldb/Symbol/CompileUnit.h"
 #include "lldb/Symbol/Function.h"
 #include "lldb/Symbol/LineEntry.h" // for LineEntry
 #include "lldb/Symbol/SymbolContext.h"
 #include "lldb/Target/PathMappingList.h" // for PathMappingList
 #include "lldb/Target/Target.h"
 #include "lldb/Utility/ConstString.h" // for ConstString
 #include "lldb/Utility/DataBuffer.h"
 #include "lldb/Utility/DataBufferLLVM.h"
 #include "lldb/Utility/RegularExpression.h"
 #include "lldb/Utility/Stream.h"
+#include "lldb/Utility/StreamString.h"
 #include "lldb/lldb-enumerations.h" // for StopShowColumn::eStopSho...
 
 #include "llvm/ADT/Twine.h" // for Twine
 
 #include <memory>
 #include <utility> // for pair
 
 #include <assert.h> // for assert
else
       file_sp = std::make_shared<File>(file_spec, debugger_sp);
 
     if (debugger_sp)
       debugger_sp->GetSourceFileCache().AddSourceFile(file_sp);
   }
   return file_sp;
 }
 
+static bool should_highlight_source(DebuggerSP debugger_sp) {
+  if (!debugger_sp)
+    return false;
+
+  // We don't use ANSI stop column formatting if the debugger doesn't think it
+  // should be using color.
+  if (!debugger_sp->GetUseColor())
+    return false;
+
+  return debugger_sp->GetHighlightSource();
+}
+
 static bool should_show_stop_column_with_ansi(DebuggerSP debugger_sp) {
   // We don't use ANSI stop column formatting if we can't lookup values from
   // the debugger.
   if (!debugger_sp)
     return false;
 
   // We don't use ANSI stop column formatting if the debugger doesn't think it
   // should be using color.
   if (!debugger_sp->GetUseColor())
     return false;
 
+  // Don't use terminal attributes when we have highlighting enabled. This
+  // can mess up the command line.
+  if (debugger_sp->GetHighlightSource())
+    return false;
+
   // We only use ANSI stop column formatting if we're either supposed to show
   // ANSI where available (which we know we have when we get to this point), or
   // if we're only supposed to use ANSI.
   const auto value = debugger_sp->GetStopShowColumn();
   return ((value == eStopShowColumnAnsiOrCaret) ||
           (value == eStopShowColumnAnsi));
 }
 
size_t SourceManager::File::DisplaySourceLines(uint32_t line, uint32_t column,
   // Nothing to write if there's no stream.
   if (!s)
     return 0;
 
   // Sanity check m_data_sp before proceeding.
   if (!m_data_sp)
     return 0;
 
+  std::string previous_content;
+
+  HighlightStyle style = HighlightStyle::MakeVimStyle();
+  HighlighterManager mgr;
+  std::string path = GetFileSpec().GetPath(/*denormalize*/ false);
+  auto &highlighter =
+      mgr.getHighlighterFor(lldb::LanguageType::eLanguageTypeUnknown, path);
+
   const uint32_t start_line =
       line <= context_before ? 1 : line - context_before;
   const uint32_t start_line_offset = GetLineOffset(start_line);
   if (start_line_offset != UINT32_MAX) {
     const uint32_t end_line = line + context_after;
     uint32_t end_line_offset = GetLineOffset(end_line + 1);
     if (end_line_offset == UINT32_MAX)
       end_line_offset = m_data_sp->GetByteSize();
 
     assert(start_line_offset <= end_line_offset);
     size_t bytes_written = 0;
     if (start_line_offset < end_line_offset) {
       size_t count = end_line_offset - start_line_offset;
       const uint8_t *cstr = m_data_sp->GetBytes() + start_line_offset;
 
+      auto ref = llvm::StringRef(reinterpret_cast<const char *>(cstr), count);
       bool displayed_line = false;
 
-      if (column && (column < count)) {
-        auto debugger_sp = m_debugger_wp.lock();
+      if (should_highlight_source(debugger_sp)) {
+        bytes_written +=
+            highlighter.Highlight(style, ref, previous_content, *s);
+        displayed_line = true;
+        // Add the new line to the previous lines.
+        previous_content += ref.str();
+      }
+
+      if (!displayed_line && column && (column < count)) {
         if (should_show_stop_column_with_ansi(debugger_sp) && debugger_sp) {
           // Check if we have any ANSI codes with which to mark this column. If
           // not, no need to do this work.
           auto ansi_prefix_entry = debugger_sp->GetStopShowColumnAnsiPrefix();
           auto ansi_suffix_entry = debugger_sp->GetStopShowColumnAnsiSuffix();
 
           // We only bother breaking up the line to format the marked column if
           // there is any marking specified on both sides of the marked column.
if (start_line_offset < end_line_offset) {
             displayed_line = true;
           }
         }
       }
 
       // If we didn't end up displaying the line with ANSI codes for whatever
       // reason, display it now sans codes.
       if (!displayed_line)
-        bytes_written = s->Write(cstr, count);
+        bytes_written = s->PutCString(ref);
 
       // Ensure we get an end of line character one way or another.
-      if (!is_newline_char(cstr[count - 1]))
+      if (!is_newline_char(ref.back()))
         bytes_written += s->EOL();
     }
     return bytes_written;
   }
   return 0;
 }
 
 void SourceManager::File::FindLinesMatchingRegex(

source/Plugins/Language/CMakeLists.txt

+add_subdirectory(ClangCommon)
 add_subdirectory(CPlusPlus)
 add_subdirectory(Go)
 add_subdirectory(Java)
 add_subdirectory(ObjC)
 add_subdirectory(ObjCPlusPlus)
 add_subdirectory(OCaml)

source/Plugins/Language/CPlusPlus/CMakeLists.txt

add_lldb_library(lldbPluginCPlusPlusLanguage PLUGIN
 
   LINK_LIBS
     lldbCore
     lldbDataFormatters
     lldbHost
     lldbSymbol
     lldbTarget
     lldbUtility
+    lldbPluginClangCommon
+
   LINK_COMPONENTS
     Support
 )

source/Plugins/Language/CPlusPlus/CPlusPlusLanguage.h

 // C++ Includes
 #include <set>
 #include <vector>
 
 // Other libraries and framework includes
 #include "llvm/ADT/StringRef.h"
 
 // Project includes
+#include "Plugins/Language/ClangCommon/ClangHighlighter.h"
 #include "lldb/Target/Language.h"
 #include "lldb/Utility/ConstString.h"
 #include "lldb/lldb-private.h"
 
 namespace lldb_private {
 
 class CPlusPlusLanguage : public Language {
+  ClangHighlighter m_highlighter;
+
 public:
   class MethodName {
   public:
     MethodName()
         : m_full(), m_basename(), m_context(), m_arguments(), m_qualifiers(),
           m_parsed(false), m_parse_error(false) {}
 
     MethodName(const ConstString &s)
public:
   std::unique_ptr<TypeScavenger> GetTypeScavenger() override;
   lldb::TypeCategoryImplSP GetFormatters() override;
 
   HardcodedFormatters::HardcodedSummaryFinder GetHardcodedSummaries() override;
 
   HardcodedFormatters::HardcodedSyntheticFinder
   GetHardcodedSynthetics() override;
 
+  bool IsSourceFile(llvm::StringRef file_path) const override;
+
+  const Highlighter *GetHighlighter() const override { return &m_highlighter; }
+
   //------------------------------------------------------------------
   // Static Functions
   //------------------------------------------------------------------
   static void Initialize();
 
   static void Terminate();
 
   static lldb_private::Language *CreateInstance(lldb::LanguageType language);

source/Plugins/Language/CPlusPlus/CPlusPlusLanguage.cpp

g_formatters.push_back([](lldb_private::ValueObject &valobj,
       }
       return nullptr;
     });
 
   });
 
   return g_formatters;
 }
+
+bool CPlusPlusLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  const auto suffixes = {".cpp", ".cxx", ".c++", ".cc",  ".c",
+                         ".h",   ".hh",  ".hpp", ".hxx", ".h++"};
+  for (auto suffix : suffixes) {
+    if (file_path.endswith_lower(suffix))
+      return true;
+  }
+
+  // Check if we're in a STL path (where the files usually have no extension
+  // that we could check for.
+  return file_path.contains("/usr/include/c++/");
+}

source/Plugins/Language/ClangCommon/CMakeLists.txt

+add_lldb_library(lldbPluginClangCommon PLUGIN
+  ClangHighlighter.cpp
+
+  LINK_LIBS
+    lldbCore
+    lldbUtility
+  LINK_COMPONENTS
+    Support
+)

source/Plugins/Language/ClangCommon/ClangHighlighter.h

+//===-- ClangHighlighter.h --------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef liblldb_ClangHighlighter_h_
+#define liblldb_ClangHighlighter_h_
+
+// C Includes
+// C++ Includes
+// Other libraries and framework includes
+#include "lldb/Utility/StreamString.h"
+#include "llvm/ADT/StringSet.h"
+
+// Project includes
+#include "lldb/Core/Highlighter.h"
+
+namespace lldb_private {
+
+class ClangHighlighter : public Highlighter {
+  llvm::StringSet<> keywords;
+
+public:
+  ClangHighlighter();
+  llvm::StringRef GetName() const override { return "clang"; }
+
+  std::size_t Highlight(const HighlightStyle &options, llvm::StringRef line,
+                        llvm::StringRef previous_lines,
+                        Stream &s) const override;
+
+  /// Returns true if the given string represents a keywords in any Clang
+  /// supported language.
+  bool isKeyword(llvm::StringRef token) const;
+};
+
+} // namespace lldb_private
+
+#endif // liblldb_ClangHighlighter_h_

source/Plugins/Language/ClangCommon/ClangHighlighter.cpp

+//===-- ClangHighlighter.cpp ------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "ClangHighlighter.h"
+
+#include "lldb/Target/Language.h"
+#include "lldb/Utility/AnsiTerminal.h"
+#include "lldb/Utility/StreamString.h"
+
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/ADT/StringSet.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+#include <unordered_set>
+
+using namespace lldb_private;
+
+bool ClangHighlighter::isKeyword(llvm::StringRef token) const {
+  return keywords.find(token) != keywords.end();
+}
+
+ClangHighlighter::ClangHighlighter() {
+#define KEYWORD(X, N) keywords.insert(#X);
+#include "clang/Basic/TokenKinds.def"
+}
+
+/// Determines which style should be applied to the given token.
+/// \param highlighter
+///     The current highlighter that should use the style.
+/// \param token
+///     The current token.
+/// \param tok_str
+///     The string in the source code the token represents.
+/// \param options
+///     The style we use for coloring the source code.
+/// \param in_pp_directive
+///     If we are currently in a preprocessor directive. NOTE: This is
+///     passed by reference and will be updated if the current token starts
+///     or ends a preprocessor directive.
+/// \return
+///     The ColorStyle that should be applied to the token.
+static HighlightStyle::ColorStyle
+determineClangStyle(const ClangHighlighter &highlighter,
+                    const clang::Token &token, llvm::StringRef tok_str,
+                    const HighlightStyle &options, bool &in_pp_directive) {
+  using namespace clang;
+
+  if (token.is(tok::comment)) {
+    // If we were in a preprocessor directive before, we now left it.
+    in_pp_directive = false;
+    return options.comment;
+  } else if (in_pp_directive || token.getKind() == tok::hash) {
+    // Let's assume that the rest of the line is a PP directive.
+    in_pp_directive = true;
+    // Preprocessor directives are hard to match, so we have to hack this in.
+    return options.pp_directive;
+  } else if (tok::isStringLiteral(token.getKind()))
+    return options.string_literal;
+  else if (tok::isLiteral(token.getKind()))
+    return options.scalar_literal;
+  else if (highlighter.isKeyword(tok_str))
+    return options.keyword;
+  else
+    switch (token.getKind()) {
+    case tok::raw_identifier:
+    case tok::identifier:
+      return options.identifier;
+    case tok::l_brace:
+    case tok::r_brace:
+      return options.braces;
+    case tok::l_square:
+    case tok::r_square:
+      return options.square_brackets;
+    case tok::l_paren:
+    case tok::r_paren:
+      return options.parentheses;
+    case tok::comma:
+      return options.comma;
+    case tok::coloncolon:
+    case tok::colon:
+      return options.colon;
+
+    case tok::amp:
+    case tok::ampamp:
+    case tok::ampequal:
+    case tok::star:
+    case tok::starequal:
+    case tok::plus:
+    case tok::plusplus:
+    case tok::plusequal:
+    case tok::minus:
+    case tok::arrow:
+    case tok::minusminus:
+    case tok::minusequal:
+    case tok::tilde:
+    case tok::exclaim:
+    case tok::exclaimequal:
+    case tok::slash:
+    case tok::slashequal:
+    case tok::percent:
+    case tok::percentequal:
+    case tok::less:
+    case tok::lessless:
+    case tok::lessequal:
+    case tok::lesslessequal:
+    case tok::spaceship:
+    case tok::greater:
+    case tok::greatergreater:
+    case tok::greaterequal:
+    case tok::greatergreaterequal:
+    case tok::caret:
+    case tok::caretequal:
+    case tok::pipe:
+    case tok::pipepipe:
+    case tok::pipeequal:
+    case tok::question:
+    case tok::equal:
+    case tok::equalequal:
+      return options.operators;
+    default:
+      break;
+    }
+  return HighlightStyle::ColorStyle();
+}
+
+std::size_t ClangHighlighter::Highlight(const HighlightStyle &options,
+                                        llvm::StringRef line,
+                                        llvm::StringRef previous_lines,
+                                        Stream &result) const {
+  using namespace clang;
+
+  std::size_t written_bytes = 0;
+
+  FileSystemOptions file_opts;
+  FileManager file_mgr(file_opts);
+
+  unsigned line_number = previous_lines.count('\n') + 1U;
+
+  // Let's build the actual source code Clang needs and setup some utility
+  // objects.
+  std::string full_source = previous_lines.str() + line.str();
+  llvm::IntrusiveRefCntPtr<DiagnosticIDs> diag_ids(new DiagnosticIDs());
+  llvm::IntrusiveRefCntPtr<DiagnosticOptions> diags_opts(
+      new DiagnosticOptions());
+  DiagnosticsEngine diags(diag_ids, diags_opts);
+  clang::SourceManager SM(diags, file_mgr);
+  std::unique_ptr<llvm::MemoryBuffer> buf =
+      llvm::MemoryBuffer::getMemBufferCopy(full_source);
+
+  FileID FID = SM.createFileID(clang::SourceManager::Unowned, buf.get());
+
+  // Let's just enable the latest ObjC and C++ which should get most tokens
+  // right.
+  LangOptions Opts;
+  Opts.ObjC2 = true;
+  Opts.CPlusPlus17 = true;
+  Opts.LineComment = true;
+
+  Lexer lex(FID, buf.get(), SM, Opts);
+  // The lexer should keep whitespace around.
+  lex.SetKeepWhitespaceMode(true);
+
+  // Keeps track if we have entered a PP directive.
+  bool in_pp_directive = false;
+
+  // True once we actually lexed the user provided line.
+  bool found_user_line = false;
+
+  Token token;
+  bool exit = false;
+  while (!exit) {
+    // Returns true if this is the last token we get from the lexer.
+    exit = lex.LexFromRawLexer(token);
+
+    bool invalid = false;
+    unsigned current_line_number =
+        SM.getSpellingLineNumber(token.getLocation(), &invalid);
+    if (current_line_number != line_number)
+      continue;
+    found_user_line = true;
+
+    // We don't need to print any tokens without a spelling line number.
+    if (invalid)
+      continue;
+
+    // Same as above but with the column number.
+    invalid = false;
+    unsigned start = SM.getSpellingColumnNumber(token.getLocation(), &invalid);
+    if (invalid)
+      continue;
+    // Column numbers start at 1, but indexes in our string start at 0.
+    --start;
+
+    // Annotations don't have a length, so let's skip them.
+    if (token.isAnnotation())
+      continue;
+
+    // Extract the token string from our source code.
+    llvm::StringRef tok_str = line.substr(start, token.getLength());
+
+    // If the token is just an empty string, we can skip all the work below.
+    if (tok_str.empty())
+      continue;
+
+    // See how we are supposed to highlight this token.
+    HighlightStyle::ColorStyle color =
+        determineClangStyle(*this, token, tok_str, options, in_pp_directive);
+
+    written_bytes += color.Apply(result, tok_str);
+  }
+
+  // If we went over the whole file but couldn't find our own file, then
+  // somehow our setup was wrong. When we're in release mode we just give the
+  // user the normal line and pretend we don't know how to highlight it. In
+  // debug mode we bail out with an assert as this should never happen.
+  if (!found_user_line) {
+    result << line;
+    written_bytes += line.size();
+    assert(false && "We couldn't find the user line in the input file?");
+  }
+
+  return written_bytes;
+}

source/Plugins/Language/Go/GoLanguage.h

lldb::LanguageType GetLanguageType() const override {
     return lldb::eLanguageTypeGo;
   }
 
   HardcodedFormatters::HardcodedSummaryFinder GetHardcodedSummaries() override;
 
   HardcodedFormatters::HardcodedSyntheticFinder
   GetHardcodedSynthetics() override;
 
+  bool IsSourceFile(llvm::StringRef file_path) const override;
+
   //------------------------------------------------------------------
   // Static Functions
   //------------------------------------------------------------------
   static void Initialize();
 
   static void Terminate();
 
   static lldb_private::Language *CreateInstance(lldb::LanguageType language);

source/Plugins/Language/Go/GoLanguage.cpp

g_formatters.push_back(
             return formatter_sp;
           }
           return nullptr;
         });
   });
 
   return g_formatters;
 }
+
+bool GoLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  return file_path.endswith(".go");
+}

source/Plugins/Language/Java/JavaLanguage.h

public:
 
   ConstString GetPluginName() override;
 
   uint32_t GetPluginVersion() override;
 
   bool IsNilReference(ValueObject &valobj) override;
 
   lldb::TypeCategoryImplSP GetFormatters() override;
+
+  bool IsSourceFile(llvm::StringRef file_path) const override;
 };
 
 } // namespace lldb_private
 
 #endif // liblldb_JavaLanguage_h_

source/Plugins/Language/Java/JavaLanguage.cpp

#ifndef LLDB_DISABLE_PYTHON
           lldb_private::formatters::JavaArraySyntheticFrontEndCreator,
           "Java array synthetic children", ConstString(array_regexp),
           SyntheticChildren::Flags().SetCascades(true), true);
 #endif
     }
   });
   return g_category;
 }
+
+bool JavaLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  return file_path.endswith(".java");
+}

source/Plugins/Language/OCaml/OCamlLanguage.h

 namespace lldb_private {
 
 class OCamlLanguage : public Language {
 public:
   lldb::LanguageType GetLanguageType() const override {
     return lldb::eLanguageTypeOCaml;
   }
 
+  bool IsSourceFile(llvm::StringRef file_path) const override;
+
   static void Initialize();
 
   static void Terminate();
 
   static lldb_private::Language *CreateInstance(lldb::LanguageType language);
 
   static lldb_private::ConstString GetPluginNameStatic();
 

source/Plugins/Language/OCaml/OCamlLanguage.cpp

 #include "lldb/DataFormatters/DataVisualization.h"
 #include "lldb/DataFormatters/FormattersHelpers.h"
 #include "lldb/Symbol/OCamlASTContext.h"
 #include "lldb/Utility/ConstString.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
+bool OCamlLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  const auto suffixes = {".ml", ".mli"};
+  for (auto suffix : suffixes) {
+    if (file_path.endswith_lower(suffix))
+      return true;
+  }
+  return false;
+}
+
 void OCamlLanguage::Initialize() {
   PluginManager::RegisterPlugin(GetPluginNameStatic(), "OCaml Language",
                                 CreateInstance);
 }
 
 void OCamlLanguage::Terminate() {
   PluginManager::UnregisterPlugin(CreateInstance);
 }

source/Plugins/Language/ObjC/CMakeLists.txt

LINK_LIBS
     lldbCore
     lldbDataFormatters
     lldbExpression
     lldbHost
     lldbSymbol
     lldbTarget
     lldbUtility
     lldbPluginAppleObjCRuntime
+    lldbPluginClangCommon
 
   EXTRA_CXXFLAGS ${EXTRA_CXXFLAGS}
 )

source/Plugins/Language/ObjC/ObjCLanguage.h

 
 // C Includes
 // C++ Includes
 #include <cstring>
 #include <vector>
 
 // Other libraries and framework includes
 // Project includes
+#include "Plugins/Language/ClangCommon/ClangHighlighter.h"
 #include "lldb/Target/Language.h"
 #include "lldb/Utility/ConstString.h"
 #include "lldb/lldb-private.h"
 
 namespace lldb_private {
 
 class ObjCLanguage : public Language {
+  ClangHighlighter m_highlighter;
+
 public:
   class MethodName {
   public:
     enum Type { eTypeUnspecified, eTypeClassMethod, eTypeInstanceMethod };
 
     MethodName()
         : m_full(), m_class(), m_category(), m_selector(),
           m_type(eTypeUnspecified), m_category_is_valid(false) {}
public:
   std::unique_ptr<TypeScavenger> GetTypeScavenger() override;
 
   bool GetFormatterPrefixSuffix(ValueObject &valobj, ConstString type_hint,
                                 std::string &prefix,
                                 std::string &suffix) override;
 
   bool IsNilReference(ValueObject &valobj) override;
 
+  bool IsSourceFile(llvm::StringRef file_path) const override;
+
+  const Highlighter *GetHighlighter() const override { return &m_highlighter; }
+
   //------------------------------------------------------------------
   // Static Functions
   //------------------------------------------------------------------
   static void Initialize();
 
   static void Terminate();
 
   static lldb_private::Language *CreateInstance(lldb::LanguageType language);

source/Plugins/Language/ObjC/ObjCLanguage.cpp

bool ObjCLanguage::IsNilReference(ValueObject &valobj) {
   bool isObjCpointer =
       (((valobj.GetCompilerType().GetTypeInfo(nullptr)) & mask) == mask);
   if (!isObjCpointer)
     return false;
   bool canReadValue = true;
   bool isZero = valobj.GetValueAsUnsigned(0, &canReadValue) == 0;
   return canReadValue && isZero;
 }
+
+bool ObjCLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  const auto suffixes = {".h", ".m", ".M"};
+  for (auto suffix : suffixes) {
+    if (file_path.endswith_lower(suffix))
+      return true;
+  }
+  return false;
+}

source/Plugins/Language/ObjCPlusPlus/CMakeLists.txt

 add_lldb_library(lldbPluginObjCPlusPlusLanguage PLUGIN
   ObjCPlusPlusLanguage.cpp
-  
   LINK_LIBS
     lldbCore
     lldbTarget
+    lldbPluginClangCommon
 )

source/Plugins/Language/ObjCPlusPlus/ObjCPlusPlusLanguage.h

 //===-- ObjCPlusPlusLanguage.h ----------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_ObjCPlusPlusLanguage_h_
 #define liblldb_ObjCPlusPlusLanguage_h_
 
 // C Includes
 // C++ Includes
 // Other libraries and framework includes
 // Project includes
+#include "Plugins/Language/ClangCommon/ClangHighlighter.h"
 #include "lldb/Target/Language.h"
 #include "lldb/lldb-private.h"
 
 namespace lldb_private {
 
 class ObjCPlusPlusLanguage : public Language {
+  ClangHighlighter m_highlighter;
+
 public:
   ObjCPlusPlusLanguage() = default;
 
   ~ObjCPlusPlusLanguage() override = default;
 
   lldb::LanguageType GetLanguageType() const override {
     return lldb::eLanguageTypeObjC_plus_plus;
   }
 
+  bool IsSourceFile(llvm::StringRef file_path) const override;
+
+  const Highlighter *GetHighlighter() const override { return &m_highlighter; }
+
   //------------------------------------------------------------------
   // Static Functions
   //------------------------------------------------------------------
   static void Initialize();
 
   static void Terminate();
 
   static lldb_private::Language *CreateInstance(lldb::LanguageType language);

source/Plugins/Language/ObjCPlusPlus/ObjCPlusPlusLanguage.cpp

 #include "ObjCPlusPlusLanguage.h"
 
 #include "lldb/Core/PluginManager.h"
 #include "lldb/Utility/ConstString.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
+bool ObjCPlusPlusLanguage::IsSourceFile(llvm::StringRef file_path) const {
+  const auto suffixes = {".h", ".mm"};
+  for (auto suffix : suffixes) {
+    if (file_path.endswith_lower(suffix))
+      return true;
+  }
+  return false;
+}
+
 void ObjCPlusPlusLanguage::Initialize() {
   PluginManager::RegisterPlugin(GetPluginNameStatic(), "Objective-C++ Language",
                                 CreateInstance);
 }
 
 void ObjCPlusPlusLanguage::Terminate() {
   PluginManager::UnregisterPlugin(CreateInstance);
 }

source/Target/Language.cpp

if (language_ptr) {
       map[language] = std::unique_ptr<Language>(language_ptr);
       return language_ptr;
     }
   }
 
   return nullptr;
 }
 
+Language *Language::FindPlugin(llvm::StringRef file_path) {
+  Language *result = nullptr;
+  ForEach([&result, file_path](Language *language) {
+    if (language->IsSourceFile(file_path)) {
+      result = language;
+      return false;
+    }
+    return true;
+  });
+  return result;
+}
+
+Language *Language::FindPlugin(LanguageType language,
+                               llvm::StringRef file_path) {
+  Language *result = FindPlugin(language);
+  // Finding a language by file path is slower, we so we use this as the
+  // fallback.
+  if (!result)
+    result = FindPlugin(file_path);
+  return result;
+}
+
 void Language::ForEach(std::function<bool(Language *)> callback) {
+  // If we want to iterate over all languages, we first have to complete the
+  // LanguagesMap.
+  static llvm::once_flag g_initialize;
+  llvm::call_once(g_initialize, [] {
+    for (unsigned lang = eLanguageTypeUnknown; lang < eNumLanguageTypes;
+         ++lang) {
+      FindPlugin(static_cast<lldb::LanguageType>(lang));
+    }
+  });
+
   std::lock_guard<std::mutex> guard(GetLanguagesMutex());
   LanguagesMap &map(GetLanguagesMap());
   for (const auto &entry : map) {
     if (!callback(entry.second.get()))
       break;
   }
 }
 

unittests/Language/CMakeLists.txt

 add_subdirectory(CPlusPlus)
+add_subdirectory(Highlighting)

unittests/Language/Highlighting/CMakeLists.txt

+add_lldb_unittest(HighlighterTests
+  HighlighterTest.cpp
+
+  LINK_LIBS
+    lldbPluginCPlusPlusLanguage
+    lldbPluginObjCLanguage
+    lldbPluginObjCPlusPlusLanguage
+    lldbPluginJavaLanguage
+    lldbPluginOCamlLanguage
+    lldbPluginGoLanguage
+  )

unittests/Language/Highlighting/HighlighterTest.cpp

+//===-- HighlighterTest.cpp -------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+
+#include "lldb/Core/Highlighter.h"
+
+#include "llvm/Support/Threading.h"
+
+#include "Plugins/Language/CPlusPlus/CPlusPlusLanguage.h"
+#include "Plugins/Language/Go/GoLanguage.h"
+#include "Plugins/Language/Java/JavaLanguage.h"
+#include "Plugins/Language/OCaml/OCamlLanguage.h"
+#include "Plugins/Language/ObjC/ObjCLanguage.h"
+#include "Plugins/Language/ObjCPlusPlus/ObjCPlusPlusLanguage.h"
+
+using namespace lldb_private;
+
+static void InitLanguages() {
+  static llvm::once_flag g_once_flag;
+  llvm::call_once(g_once_flag, []() {
+    // The HighlighterManager uses the language plugins under the hood, so we
+    // have to initialize them here for our test process.
+    CPlusPlusLanguage::Initialize();
+    GoLanguage::Initialize();
+    JavaLanguage::Initialize();
+    ObjCLanguage::Initialize();
+    ObjCPlusPlusLanguage::Initialize();
+    OCamlLanguage::Initialize();
+  });
+}
+
+static std::string getName(lldb::LanguageType type) {
+  HighlighterManager mgr;
+  return mgr.getHighlighterFor(type, "").GetName().str();
+}
+
+static std::string getName(llvm::StringRef path) {
+  HighlighterManager mgr;
+  return mgr.getHighlighterFor(lldb::eLanguageTypeUnknown, path)
+      .GetName()
+      .str();
+}
+
+TEST(HighlighterTest, HighlighterSelectionType) {
+  InitLanguages();
+  EXPECT_EQ(getName(lldb::eLanguageTypeC_plus_plus), "clang");
+  EXPECT_EQ(getName(lldb::eLanguageTypeC_plus_plus_03), "clang");
+  EXPECT_EQ(getName(lldb::eLanguageTypeC_plus_plus_11), "clang");
+  EXPECT_EQ(getName(lldb::eLanguageTypeC_plus_plus_14), "clang");
+  EXPECT_EQ(getName(lldb::eLanguageTypeObjC), "clang");
+  EXPECT_EQ(getName(lldb::eLanguageTypeObjC_plus_plus), "clang");
+
+  EXPECT_EQ(getName(lldb::eLanguageTypeUnknown), "none");
+  EXPECT_EQ(getName(lldb::eLanguageTypeJulia), "none");
+  EXPECT_EQ(getName(lldb::eLanguageTypeJava), "none");
+  EXPECT_EQ(getName(lldb::eLanguageTypeHaskell), "none");
+}
+
+TEST(HighlighterTest, HighlighterSelectionPath) {
+  InitLanguages();
+  EXPECT_EQ(getName("myfile.cc"), "clang");
+  EXPECT_EQ(getName("moo.cpp"), "clang");
+  EXPECT_EQ(getName("mar.cxx"), "clang");
+  EXPECT_EQ(getName("foo.C"), "clang");
+  EXPECT_EQ(getName("bar.CC"), "clang");
+  EXPECT_EQ(getName("a/dir.CC"), "clang");
+  EXPECT_EQ(getName("/a/dir.hpp"), "clang");
+  EXPECT_EQ(getName("header.h"), "clang");
+
+  EXPECT_EQ(getName(""), "none");
+  EXPECT_EQ(getName("/dev/null"), "none");
+  EXPECT_EQ(getName("Factory.java"), "none");
+  EXPECT_EQ(getName("poll.py"), "none");
+  EXPECT_EQ(getName("reducer.hs"), "none");
+}
+
+TEST(HighlighterTest, FallbackHighlighter) {
+  InitLanguages();
+  HighlighterManager mgr;
+  const Highlighter &h =
+      mgr.getHighlighterFor(lldb::eLanguageTypePascal83, "foo.pas");
+
+  HighlightStyle style;
+  style.identifier.Set("[", "]");
+  style.semicolons.Set("<", ">");
+
+  const char *code = "program Hello;";
+  std::string output = h.Highlight(style, code);
+
+  EXPECT_STREQ(output.c_str(), code);
+}
+
+TEST(HighlighterTest, DefaultHighlighter) {
+  InitLanguages();
+  HighlighterManager mgr;
+  const Highlighter &h = mgr.getHighlighterFor(lldb::eLanguageTypeC, "main.c");
+
+  HighlightStyle style;
+
+  const char *code = "int my_main() { return 22; } \n";
+  std::string output = h.Highlight(style, code);
+
+  EXPECT_STREQ(output.c_str(), code);
+}
+
+//------------------------------------------------------------------------------
+// Tests highlighting with the Clang highlighter.
+//------------------------------------------------------------------------------
+
+static std::string highlightC(llvm::StringRef code, HighlightStyle style) {
+  InitLanguages();
+  HighlighterManager mgr;
+  const Highlighter &h = mgr.getHighlighterFor(lldb::eLanguageTypeC, "main.c");
+  return h.Highlight(style, code);
+}
+
+TEST(HighlighterTest, ClangEmptyInput) {
+  HighlightStyle s;
+
+  std::string output = highlightC("", s);
+  EXPECT_STREQ("", output.c_str());
+}
+
+TEST(HighlighterTest, ClangScalarLiterals) {
+  HighlightStyle s;
+  s.scalar_literal.Set("<scalar>", "</scalar>");
+
+  std::string output = highlightC(" int i = 22;", s);
+  EXPECT_STREQ(" int i = <scalar>22</scalar>;", output.c_str());
+}
+
+TEST(HighlighterTest, ClangStringLiterals) {
+  HighlightStyle s;
+  s.string_literal.Set("<str>", "</str>");
+
+  std::string output = highlightC("const char *f = 22 + \"foo\";", s);
+  EXPECT_STREQ("const char *f = 22 + <str>\"foo\"</str>;", output.c_str());
+}
+
+TEST(HighlighterTest, ClangUnterminatedString) {
+  HighlightStyle s;
+  s.string_literal.Set("<str>", "</str>");
+
+  std::string output = highlightC(" f = \"", s);
+  EXPECT_STREQ(" f = \"", output.c_str());
+}
+
+TEST(HighlighterTest, Keywords) {
+  HighlightStyle s;
+  s.keyword.Set("<k>", "</k>");
+
+  std::string output = highlightC(" return 1; ", s);
+  EXPECT_STREQ(" <k>return</k> 1; ", output.c_str());
+}
+
+TEST(HighlighterTest, Colons) {
+  HighlightStyle s;
+  s.colon.Set("<c>", "</c>");
+
+  std::string output = highlightC("foo::bar:", s);
+  EXPECT_STREQ("foo<c>:</c><c>:</c>bar<c>:</c>", output.c_str());
+}
+
+TEST(HighlighterTest, ClangBraces) {
+  HighlightStyle s;
+  s.braces.Set("<b>", "</b>");
+
+  std::string output = highlightC("a{}", s);
+  EXPECT_STREQ("a<b>{</b><b>}</b>", output.c_str());
+}
+
+TEST(HighlighterTest, ClangSquareBrackets) {
+  HighlightStyle s;
+  s.square_brackets.Set("<sb>", "</sb>");
+
+  std::string output = highlightC("a[]", s);
+  EXPECT_STREQ("a<sb>[</sb><sb>]</sb>", output.c_str());
+}
+
+TEST(HighlighterTest, ClangCommas) {
+  HighlightStyle s;
+  s.comma.Set("<comma>", "</comma>");
+
+  std::string output = highlightC(" bool f = foo(), 1;", s);
+  EXPECT_STREQ(" bool f = foo()<comma>,</comma> 1;", output.c_str());
+}
+
+TEST(HighlighterTest, ClangPPDirectives) {
+  HighlightStyle s;
+  s.pp_directive.Set("<pp>", "</pp>");
+
+  std::string output = highlightC(" #include \"foo\" // comment\n", s);
+  EXPECT_STREQ(" <pp>#</pp><pp>include</pp><pp> </pp><pp>\"foo\"</pp><pp> </pp>"
+               "// comment\n",
+               output.c_str());
+}
+
+TEST(HighlighterTest, ClangComments) {
+  HighlightStyle s;
+  s.comment.Set("<cc>", "</cc>");
+
+  std::string output = highlightC(" /*com */ // com /*n*/", s);
+  EXPECT_STREQ(" <cc>/*com */</cc> <cc>// com /*n*/</cc>", output.c_str());
+}
+
+TEST(HighlighterTest, ClangOperators) {
+  HighlightStyle s;
+  s.operators.Set("[", "]");
+
+  std::string output = highlightC(" 1+2/a*f&x|~l", s);
+  EXPECT_STREQ(" 1[+]2[/]a[*]f[&]x[|][~]l", output.c_str());
+}
+
+TEST(HighlighterTest, ClangIdentifiers) {
+  HighlightStyle s;
+  s.identifier.Set("<id>", "</id>");
+
+  std::string output = highlightC(" foo c = bar(); return 1;", s);
+  EXPECT_STREQ(" <id>foo</id> <id>c</id> = <id>bar</id>(); return 1;",
+               output.c_str());
+}