[lldb] Add a C language REPL to test LLDB's REPL infrastructure

[teemperor]

Summary

LLDB has a bunch of code that implements REPL support, but all that code is unreachable as no language
in master currently has an implemented REPL backend. The only REPL that exists is in the downstream
Swift fork. All patches for this generic REPL code therefore also only have tests downstream which is
clearly not a good situation.

This patch implements a basic C language REPL on top of LLDB's REPL framework. Beside implementing
the REPL interface and hooking it up into the plugin manager, the only other small part of this patch
is making the --language flag of the expression command compatible with the --repl flag. The --repl
flag uses the value of --language to see which REPL should be started, but right now the --language
flag is only available in OptionGroups 1 and 2, but not in OptionGroup 3 where the --repl flag is
declared.

The REPL currently can currently only start if a running target exists. I'll add the 'create and run a dummy executable'
logic from Swift (which is requires when doing lldb --repl) when I have time to translate all this logic
to something that will work with Clang.

I should point out that the REPL currently uses the C expression parser's approach to persistent variables
where only result variables and the ones starting with a '$' are transferred between expressions. I'll fix
that in a follow up patch. Also the REPL currently doesn't work in a non-interactive terminal. This seems to
be fixed in the Swift fork, so I assume one of our many REPL downstream changes addresses the issue.

[teemperor]

(And I'm aware there are a lot more tests needed to test all the REPL code that had no tests so far. I'll see what's missing once this his the coverage bot and then I'll make put reviews up for that).

[mib]

Some nitpicking but this looks fine to me.

lldb/source/Plugins/REPL/Clang/ClangREPL.cpp

79

I guess this is not the final implementation, so may be add a // FIXME

[mib]

lldb/source/Plugins/REPL/Clang/ClangREPL.cpp

65

$ swift
Welcome to Apple Swift version 5.3 (swiftlang-1200.0.29.2 clang-1200.0.30.1).
Type :help for assistance.
  1> "\(#file)" 
$R0: String = "repl.swift"

To keep consistency with the SwiftREPL, maybe the REPL source file should be named repl.c

[teemperor]

• Added comments to unimplemented functions.
• Changed repl source file name.

[teemperor]

• Retry uploading diff.

[aprantl]

This looks mostly good! I just have a few minor nitpicks.

lldb/source/Plugins/REPL/Clang/ClangREPL.cpp

29

We recently added a isFortran() function to Dwarf.h in llvm... We might want to add a isCplusplus() function, too?
Hmm.. but looping over all constants and asking isCpluPlus() seems dumb, too. Otoh — I'm convinced that we'll forget to add C++17 here in the future....

lldb/source/Plugins/REPL/Clang/ClangREPL.h

16

Doxygen comment(s)?

31

This is obviously not important at all — I found some plugins that seem to just return a char * or a StringRef here. If that is possible I guess that would be micro-optimization we could do here, too?

lldb/test/API/repl/clang/TestClangREPL.py

5

Wheee! PExpect ...

[teemperor]

• Addressed comments.

[JDevlieghere]

Should we ship this?

[JDevlieghere]

LGTM (accepting on my own behalf)

lldb/source/Commands/Options.td

def expression_options_unwind_on_error : Option<"unwind-on-error", "u">,
     Groups<[1,2]>, Arg<"Boolean">,
     Desc<"Clean up program state if the expression causes a crash, or raises a "
     "signal. Note, unlike gdb hitting a breakpoint is controlled by another "
     "option (-i).">;
   def expression_options_debug : Option<"debug", "g">, Groups<[1,2]>,
     Desc<"When specified, debug the JIT code by setting a breakpoint on the "
     "first instruction and forcing breakpoints to not be ignored (-i0) and no "
     "unwinding to happen on error (-u0).">;
-  def expression_options_language : Option<"language", "l">, Groups<[1,2]>,
+  def expression_options_language : Option<"language", "l">, Groups<[1,2,3]>,
     Arg<"Language">, Desc<"Specifies the Language to use when parsing the "
     "expression.  If not set the target.language setting is used.">;
   def expression_options_apply_fixits : Option<"apply-fixits", "X">,
     Groups<[1,2]>, Arg<"Language">, Desc<"If true, simple fix-it hints will be "
     "automatically applied to the expression.">;
   def expression_options_description_verbosity :
     Option<"description-verbosity", "v">, Group<1>,
     OptionalEnumArg<"DescriptionVerbosity", "DescriptionVerbosityTypes()">,

lldb/source/Plugins/CMakeLists.txt

 add_subdirectory(ABI)
 add_subdirectory(Architecture)
 add_subdirectory(Disassembler)
 add_subdirectory(DynamicLoader)
 add_subdirectory(ExpressionParser)
 add_subdirectory(Instruction)
 add_subdirectory(InstrumentationRuntime)
 add_subdirectory(JITLoader)
 add_subdirectory(Language)
 add_subdirectory(LanguageRuntime)
 add_subdirectory(MemoryHistory)
 add_subdirectory(ObjectContainer)
 add_subdirectory(ObjectFile)
 add_subdirectory(OperatingSystem)
 add_subdirectory(Platform)
 add_subdirectory(Process)
+add_subdirectory(REPL)
 add_subdirectory(ScriptInterpreter)
 add_subdirectory(StructuredData)
 add_subdirectory(SymbolFile)
 add_subdirectory(SystemRuntime)
 add_subdirectory(SymbolVendor)
 add_subdirectory(Trace)
 add_subdirectory(TraceExporter)
 add_subdirectory(TypeSystem)

lldb/source/Plugins/REPL/CMakeLists.txt

+add_subdirectory(Clang)

lldb/source/Plugins/REPL/Clang/CMakeLists.txt

+add_lldb_library(lldbPluginClangREPL PLUGIN
+  ClangREPL.cpp
+
+  LINK_LIBS
+    lldbCore
+    lldbDataFormatters
+    lldbHost
+    lldbSymbol
+    lldbTarget
+    lldbUtility
+    lldbPluginClangCommon
+    lldbPluginCPPRuntime
+    lldbPluginTypeSystemClang
+
+  LINK_COMPONENTS
+    Support
+)

lldb/source/Plugins/REPL/Clang/ClangREPL.h

+//===-- ClangREPL.h ---------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_SOURCE_PLUGINS_REPL_CLANG_CLANGREPL_H
+#define LLDB_SOURCE_PLUGINS_REPL_CLANG_CLANGREPL_H
+
+#include "lldb/Expression/REPL.h"
+
+namespace lldb_private {
+/// Implements a Clang-based REPL for C languages on top of LLDB's REPL
+/// framework.
+class ClangREPL : public REPL {
+public:
+  ClangREPL(lldb::LanguageType language, Target &target);
+
+  ~ClangREPL() override;
+
+  static void Initialize();
+
+  static void Terminate();
+
+  static lldb::REPLSP CreateInstance(Status &error, lldb::LanguageType language,
+                                     Debugger *debugger, Target *target,
+                                     const char *repl_options);
+
+  static lldb_private::ConstString GetPluginNameStatic() {
+    return ConstString("ClangREPL");
+  }
+
+protected:
+  Status DoInitialization() override;
+
+  ConstString GetSourceFileBasename() override;
+
+  const char *GetAutoIndentCharacters() override;
+
+  bool SourceIsComplete(const std::string &source) override;
+
+  lldb::offset_t GetDesiredIndentation(const StringList &lines,
+                                       int cursor_position,
+                                       int tab_size) override;
+
+  lldb::LanguageType GetLanguage() override;
+
+  bool PrintOneVariable(Debugger &debugger, lldb::StreamFileSP &output_sp,
+                        lldb::ValueObjectSP &valobj_sp,
+                        ExpressionVariable *var = nullptr) override;
+
+  void CompleteCode(const std::string &current_code,
+                    CompletionRequest &request) override;
+
+private:
+  /// The specific C language of this REPL.
+  lldb::LanguageType m_language;
+  /// A regex matching the implicitly created LLDB result variables.
+  lldb_private::RegularExpression m_implicit_expr_result_regex;
+};
+} // namespace lldb_private
+
+#endif // LLDB_SOURCE_PLUGINS_REPL_CLANG_CLANGREPL_H

lldb/source/Plugins/REPL/Clang/ClangREPL.cpp

+//===-- ClangREPL.cpp -----------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "ClangREPL.h"
+#include "lldb/Core/Debugger.h"
+#include "lldb/Core/PluginManager.h"
+#include "lldb/Expression/ExpressionVariable.h"
+
+using namespace lldb_private;
+
+LLDB_PLUGIN_DEFINE(ClangREPL)
+
+ClangREPL::ClangREPL(lldb::LanguageType language, Target &target)
+    : REPL(eKindClang, target), m_language(language),
+      m_implicit_expr_result_regex("\\$[0-9]+") {}
+
+ClangREPL::~ClangREPL() {}
+
+void ClangREPL::Initialize() {
+  LanguageSet languages;
+  // FIXME: There isn't a way to ask CPlusPlusLanguage and ObjCLanguage for
+  // a list of languages they support.
+  languages.Insert(lldb::LanguageType::eLanguageTypeC);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC89);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC99);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC11);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC_plus_plus);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC_plus_plus_03);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC_plus_plus_11);
+  languages.Insert(lldb::LanguageType::eLanguageTypeC_plus_plus_14);
+  languages.Insert(lldb::LanguageType::eLanguageTypeObjC);
+  languages.Insert(lldb::LanguageType::eLanguageTypeObjC_plus_plus);
+  PluginManager::RegisterPlugin(GetPluginNameStatic(), "C language REPL",
+                                &CreateInstance, languages);
+}
+
+void ClangREPL::Terminate() {
+  PluginManager::UnregisterPlugin(&CreateInstance);
+}
+
+lldb::REPLSP ClangREPL::CreateInstance(Status &error,
+                                       lldb::LanguageType language,
+                                       Debugger *debugger, Target *target,
+                                       const char *repl_options) {
+  // Creating a dummy target if only a debugger is given isn't implemented yet.
+  if (!target) {
+    error.SetErrorString("must have a target to create a REPL");
+    return nullptr;
+  }
+  lldb::REPLSP result = std::make_shared<ClangREPL>(language, *target);
+  target->SetREPL(language, result);
+  error = Status();
+  return result;
+}
+
+Status ClangREPL::DoInitialization() { return Status(); }
+
+ConstString ClangREPL::GetSourceFileBasename() {
+  return ConstString("repl.c");
+}
+
+const char *ClangREPL::GetAutoIndentCharacters() { return "  "; }
+
+bool ClangREPL::SourceIsComplete(const std::string &source) {
+  // FIXME: There isn't a good way to know if the input source is complete or
+  // not, so just say that every single REPL line is ready to be parsed.
+  return !source.empty();
+}
+
+lldb::offset_t ClangREPL::GetDesiredIndentation(const StringList &lines,
+                                                int cursor_position,
+                                                int tab_size) {
+  // FIXME: Not implemented.
+  return LLDB_INVALID_OFFSET;
+}
+
+lldb::LanguageType ClangREPL::GetLanguage() { return m_language; }
+
+bool ClangREPL::PrintOneVariable(Debugger &debugger,
+                                 lldb::StreamFileSP &output_sp,
+                                 lldb::ValueObjectSP &valobj_sp,
+                                 ExpressionVariable *var) {
+  // If a ExpressionVariable was passed, check first if that variable is just
+  // an automatically created expression result. These variables are already
+  // printed by the REPL so this is done to prevent printing the variable twice.
+  if (var) {
+    if (m_implicit_expr_result_regex.Execute(var->GetName().GetStringRef()))
+      return true;
+  }
+  valobj_sp->Dump(*output_sp);
+  return true;
+}
+
+void ClangREPL::CompleteCode(const std::string &current_code,
+                             CompletionRequest &request) {
+  // Not implemented.
+}

lldb/test/API/repl/clang/Makefile

+C_SOURCES := main.c
+include Makefile.rules

lldb/test/API/repl/clang/TestClangREPL.py

+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test.lldbpexpect import PExpectTest
+
+class TestCase(PExpectTest):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    def expect_repl(self, expr, substrs=[]):
+        """ Evaluates the expression in the REPL and verifies that the list
+        of substrs is in the REPL output."""
+        # Only single line expressions supported.
+        self.assertNotIn("\n", expr)
+        self.child.send(expr + "\n")
+        for substr in substrs:
+            self.child.expect_exact(substr)
+        # Look for the start of the next REPL input line.
+        self.current_repl_line_number += 1
+        self.child.expect_exact(str(self.current_repl_line_number) + ">")
+
+    # PExpect uses many timeouts internally and doesn't play well
+    # under ASAN on a loaded machine..
+    @skipIfAsan
+    @skipIfEditlineSupportMissing
+    def test_basic_completion(self):
+        """Test that we can complete a simple multiline expression"""
+        self.build()
+        self.current_repl_line_number = 1
+
+        self.launch(executable=self.getBuildArtifact("a.out"), dimensions=(100,500))
+        # Try launching the REPL before we have a running target.
+        self.expect("expression --repl -l c --", substrs=["REPL requires a running target process."])
+
+        self.expect("b main", substrs=["Breakpoint 1", "address ="])
+        self.expect("run", substrs=["stop reason = breakpoint 1"])
+
+        # Start the REPL.
+        self.child.send("expression --repl -l c --\n")
+        self.child.expect_exact("1>")
+
+        # Try evaluating a simple expression.
+        self.expect_repl("3 + 3", substrs=["(int) $0 = 6"])
+
+        # Try declaring a persistent variable.
+        self.expect_repl("long $persistent = 7; 5",
+                         substrs=["(int) $1 = 5",
+                                  "(long) $persistent = 7"])
+
+        # Try using the persistent variable from before.
+        self.expect_repl("$persistent + 10",
+                         substrs=["(long) $2 = 17"])
+
+        self.quit()

lldb/test/API/repl/clang/main.c

+int main(int argc, char **argv) {
+  return 0;
+}