[lldb] Support debugging utility functions

[JDevlieghere]

Summary

LLDB uses utility functions to run code in the inferior for its own internal purposes, such as reading classes from the Objective-C runtime for example. Because these expressions should be transparent to the user, we ignore breakpoints and unwind the stack on errors, which makes them hard to debug.

This patch adds a new setting target.debug-utility-expression that, when enabled, changes these options to facilitate debugging. It enables breakpoints, disables unwinding and writes out the utility function source code to disk.

Here's what this looks like in action. I added a nullptr dereference to __lldb_apple_objc_v2_get_shared_cache_class_info and used TestDataFormatterObjCNSContainer as an example:

$ lldb a.out 
(lldb) target create "a.out"
Current executable set to 'a.out' (x86_64).
(lldb) b main.m:782
Breakpoint 1: where = a.out`main + 11752 at main.m:783:4, address = 0x0000000100006838
(lldb) setting set target.debug-utility-expression true
(lldb) r
Process 77039 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100006838 a.out`main(argc=1, argv=0x00007ffeefbff500) at main.m:783:4
   780               forKeyPath:@"atoms"
   781                  options:0
   782                  context:NULL]; // Set break point at this line.
-> 783    [newMutableDictionary addObserver:[My_KVO_Observer new]
   784                           forKeyPath:@"weirdKeyToKVO"
   785                              options:NSKeyValueObservingOptionNew
   786                              context:NULL];

Process 77039 launched: 'a.out' (x86_64)
(lldb) frame variable newArray nsDictionary newDictionary nscfDictionary cfDictionaryRef newMutableDictionary newMutableDictionaryRef cfarray_ref mutable_array_ref
warning: could not execute support code to read Objective-C class data in the process. This may reduce the quality of type information available.
Process 77039 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001001df2de __lldb_apple_objc_v2_get_shared_cache_class_info`__lldb_apple_objc_v2_get_shared_cache_class_info(objc_opt_ro_ptr=0x00007fff2031bc08, class_infos_ptr=0x000000010063b000, class_infos_byte_size=1572864, should_log=0) at lldb-9f30fe.expr:68
   65                                                     uint32_t class_infos_byte_size,
   66                                                     uint32_t should_log)
   67   {
-> 68       int *i = 0;
   69       *i = 10;
   70       uint32_t idx = 0;
   71       DEBUG_PRINTF ("objc_opt_ro_ptr = %p\n", objc_opt_ro_ptr);

[jingham]

This will be really handy! The code for UserExpressions ends up doing this same thing (over in ClangExpressionParser::ParseInternal. But given how light-weight creating the file is I'm pretty sure it isn't worth trying to activate that code starting from the UtilityExpression. LGTM.

[shafik]

lldb/source/Expression/FunctionCaller.cpp

325–326

It feels a little weird you are using the name debug but using it in the call to SetDebug(...) but are using it in the call to SetGenerateDebugInfo(...)

lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp

39

Why additional #line 1.

[JDevlieghere]

lldb/source/Expression/FunctionCaller.cpp

325–326

Do you have a suggestion? Change the variable to debug_utility_function?

lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp

39

Good catch, this is a remnant of me playing around with the line directive, it shouldn't have made it into the patch.

[teemperor]

lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp

39

I assume this is to get the proper file name in the final source code. This would in theory also be required for top-level expressions, but as they are not curiously not using ClangExpressionSourceCode they never get this prefix (and are missing all our fancy type definitions here).

Anyway, this change will prompt the ClangModulesDeclVendor (which usually loads Obj-C modules) to also start loading the imported C++ modules (which I believe won't break anything but just cause a bunch of warnings about failing to load modules). See the only other use of g_prefix_file_name where we filter out imported modules from the wrapper file.

I think the idea is that the prefix starts the wrapper and then ClangExpressionSourceCode adds some generated prefixes and then we do a #line to terminate the wrapper. Not sure that's the right place for this use case, but just putting it into the ClangUtilityFunction where we concat the source code should work (even though then this logic is stuck in the initalizer, but oh well...)

lldb/source/Plugins/ExpressionParser/Clang/ClangUtilityFunction.cpp

39

Can you document debug (and maybe we could call it allow_debugging or generate_debuginfo which is how we call that in normal expressions)?

lldb/source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCRuntimeV2.cpp

1595 ↗

(On Diff #325658)

Unrelated?

lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp

9715

Not sure if that's needed. ScratchTypeSystem is bound to a specific target, so this seems like a good central place to read/use the value of the setting. I wouldn't mind the current way, but this is extending the generic TypeSystem interface and that's just always a bit of a pain to change in the future (when we hopefully have more TypeSystems for different langauges).

[JDevlieghere]

Address code review feedback

[teemperor]

Some small nitpicks about comments, otherwise LGTM

I was kinda thinking how we could test this, but all our utility functions are anyway only on macOS and very Obj-C runtime related, so that sounds like a pain to do right...

(PS: Also I would appreciate if you could shoehorn tablegen into this patch somehow. The getters/setters for the properties seems like a good candidate...)

lldb/source/Expression/FunctionCaller.cpp

103

unrelated change

lldb/source/Plugins/ExpressionParser/Clang/ClangUtilityFunction.cpp

45

// Write the source code to a file so that LLDB's source manager can display it when debugging the code

lldb/source/Plugins/ExpressionParser/Clang/ClangUtilityFunction.h

55

I think this should match the parameter name in the source (and have a doxygen comment).

[jingham]

I am pretty sure other Unix-es share the code to dlopen images that backs "Process::LoadImage". It is trivial compared to the ObjC ones, but is does have a couple of lines of code, 'cause it tries the dlopen and then if that fails, calls dlerror to fetch the error. That is done with a UtilityFunction because it turns out that's performance sensitive for swift.

lldb/include/lldb/Expression/UtilityFunction.h

public:
 
   /// Constructor
   ///
   /// \param[in] text
   ///     The text of the function.  Must be a full translation unit.
   ///
   /// \param[in] name
   ///     The name of the function, as used in the text.
+  ///
+  /// \param[in] enable_debugging
+  ///     Enable debugging of this function.
   UtilityFunction(ExecutionContextScope &exe_scope, std::string text,
-                  std::string name);
+                  std::string name, bool enable_debugging);
 
   ~UtilityFunction() override;
 
   /// Install the utility function into a process
   ///
   /// \param[in] diagnostic_manager
   ///     A diagnostic manager to print parse errors and warnings to.
   ///

lldb/include/lldb/Target/Target.h

public:
   void SetRequireHardwareBreakpoints(bool b);
 
   bool GetRequireHardwareBreakpoints() const;
 
   bool GetAutoInstallMainExecutable() const;
 
   void UpdateLaunchInfoFromProperties();
 
+  void SetDebugUtilityExpression(bool debug);
+
+  bool GetDebugUtilityExpression() const;
+
 private:
   // Callbacks for m_launch_info.
   void Arg0ValueChangedCallback();
   void RunArgsValueChangedCallback();
   void EnvVarsValueChangedCallback();
   void InputPathValueChangedCallback();
   void OutputPathValueChangedCallback();
   void ErrorPathValueChangedCallback();

lldb/source/Expression/FunctionCaller.cpp

if (m_parser->GetGenerateDebugInfo()) {
     lldb::ModuleSP jit_module_sp(m_execution_unit_sp->GetJITModule());
 
     if (jit_module_sp) {
       ConstString const_func_name(FunctionName());
       FileSpec jit_file;
       jit_file.GetFilename() = const_func_name;
       jit_module_sp->SetFileSpecAndObjectName(jit_file, ConstString());
       m_jit_module_wp = jit_module_sp;
       process->GetTarget().GetImages().Append(jit_module_sp, 
                                               true /* notify */);
     }
   }
   if (process && m_jit_start_addr)
     m_jit_process_wp = process->shared_from_this();
 
   m_JITted = true;
 

 
 lldb::ExpressionResults FunctionCaller::ExecuteFunction(
     ExecutionContext &exe_ctx, lldb::addr_t *args_addr_ptr,
     const EvaluateExpressionOptions &options,
     DiagnosticManager &diagnostic_manager, Value &results) {
   lldb::ExpressionResults return_value = lldb::eExpressionSetupError;
 
   // FunctionCaller::ExecuteFunction execution is always just to get the
-  // result. Do make sure we ignore breakpoints, unwind on error, and don't try
-  // to debug it.
+  // result. Unless explicitly asked for, ignore breakpoints and unwind on
+  // error.
+  const bool enable_debugging =
+      exe_ctx.GetTargetPtr() &&
+      exe_ctx.GetTargetPtr()->GetDebugUtilityExpression();
   EvaluateExpressionOptions real_options = options;
-  real_options.SetDebug(false);
-  real_options.SetUnwindOnError(true);
-  real_options.SetIgnoreBreakpoints(true);
+  real_options.SetDebug(false); // This halts the expression for debugging.
+  real_options.SetGenerateDebugInfo(enable_debugging);
+  real_options.SetUnwindOnError(!enable_debugging);
+  real_options.SetIgnoreBreakpoints(!enable_debugging);
 
   lldb::addr_t args_addr;
 
   if (args_addr_ptr != nullptr)
     args_addr = *args_addr_ptr;
   else
     args_addr = LLDB_INVALID_ADDRESS;
 

lldb/source/Expression/UtilityFunction.cpp

 /// Constructor
 ///
 /// \param[in] text
 ///     The text of the function.  Must be a full translation unit.
 ///
 /// \param[in] name
 ///     The name of the function, as used in the text.
 UtilityFunction::UtilityFunction(ExecutionContextScope &exe_scope,
-                                 std::string text, std::string name)
+                                 std::string text, std::string name,
+                                 bool enable_debugging)
     : Expression(exe_scope), m_execution_unit_sp(), m_jit_module_wp(),
       m_function_text(std::move(text)), m_function_name(std::move(name)) {}
 
 UtilityFunction::~UtilityFunction() {
   lldb::ProcessSP process_sp(m_jit_process_wp.lock());
   if (process_sp) {
     lldb::ModuleSP jit_module_sp(m_jit_module_wp.lock());
     if (jit_module_sp)

lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.h

 class ExecutionContext;
 
 class ClangExpressionSourceCode : public ExpressionSourceCode {
 public:
   /// The file name we use for the wrapper code that we inject before
   /// the user expression.
   static const llvm::StringRef g_prefix_file_name;
   static const char *g_expression_prefix;
+  static const char *g_expression_suffix;
 
   /// The possible ways an expression can be wrapped.
   enum class WrapKind {
     /// Wrapped in a non-static member function of a C++ class.
     CppMemberFunction,
     /// Wrapped in an instance Objective-C method.
     ObjCInstanceMethod,
     /// Wrapped in a static Objective-C method.

lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionSourceCode.cpp

 #include "lldb/Target/Platform.h"
 #include "lldb/Target/StackFrame.h"
 #include "lldb/Target/Target.h"
 #include "lldb/Utility/StreamString.h"
 
 using namespace lldb_private;
 
 #define PREFIX_NAME "<lldb wrapper prefix>"
+#define SUFFIX_NAME "<lldb wrapper suffix>"
 
 const llvm::StringRef ClangExpressionSourceCode::g_prefix_file_name = PREFIX_NAME;
 
 const char *ClangExpressionSourceCode::g_expression_prefix =
 "#line 1 \"" PREFIX_NAME R"("
 #ifndef offsetof
 #define offsetof(t, d) __builtin_offsetof(t, d)
 #endif
 #ifndef NULL
 #define NULL (__null)
 #endif
 #ifndef Nil
 #define Nil (__null)

 typedef __PTRDIFF_TYPE__ ptrdiff_t;
 typedef unsigned short unichar;
 extern "C"
 {
     int printf(const char * __restrict, ...);
 }
 )";
 
+const char *ClangExpressionSourceCode::g_expression_suffix =
+    "\n;\n#line 1 \"" SUFFIX_NAME "\"\n";
+
 namespace {
 
 class AddMacroState {
   enum State {
     CURRENT_FILE_NOT_YET_PUSHED,
     CURRENT_FILE_PUSHED,
     CURRENT_FILE_POPPED
   };

 lldb_private::ClangExpressionSourceCode::ClangExpressionSourceCode(
     llvm::StringRef filename, llvm::StringRef name, llvm::StringRef prefix,
     llvm::StringRef body, Wrapping wrap, WrapKind wrap_kind)
     : ExpressionSourceCode(name, prefix, body, wrap), m_wrap_kind(wrap_kind) {
   // Use #line markers to pretend that we have a single-line source file
   // containing only the user expression. This will hide our wrapper code
   // from the user when we render diagnostics with Clang.
   m_start_marker = "#line 1 \"" + filename.str() + "\"\n";
-  m_end_marker = "\n;\n#line 1 \"<lldb wrapper suffix>\"\n";
+  m_end_marker = g_expression_suffix;
 }
 
 namespace {
 /// Allows checking if a token is contained in a given expression.
 class TokenVerifier {
   /// The tokens we found in the expression.
   llvm::StringSet<> m_tokens;
 

lldb/source/Plugins/ExpressionParser/Clang/ClangUtilityFunction.h

public:
 
   /// Constructor
   ///
   /// \param[in] text
   ///     The text of the function.  Must be a full translation unit.
   ///
   /// \param[in] name
   ///     The name of the function, as used in the text.
+  ///
+  /// \param[in] enable_debugging
+  ///     Enable debugging of this function.
   ClangUtilityFunction(ExecutionContextScope &exe_scope, std::string text,
-                       std::string name);
+                       std::string name, bool enable_debugging);
 
   ~ClangUtilityFunction() override;
 
   ExpressionTypeSystemHelper *GetTypeSystemHelper() override {
     return &m_type_system_helper;
   }
 
   ClangExpressionDeclMap *DeclMap() { return m_type_system_helper.DeclMap(); }

lldb/source/Plugins/ExpressionParser/Clang/ClangUtilityFunction.cpp

 #include "lldb/Utility/ConstString.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/Stream.h"
 
 using namespace lldb_private;
 
 char ClangUtilityFunction::ID;
 
-/// Constructor
-///
-/// \param[in] text
-///     The text of the function.  Must be a full translation unit.
-///
-/// \param[in] name
-///     The name of the function, as used in the text.
 ClangUtilityFunction::ClangUtilityFunction(ExecutionContextScope &exe_scope,
-                                           std::string text, std::string name)
+                                           std::string text, std::string name,
+                                           bool enable_debugging)
     : UtilityFunction(
           exe_scope,
-          std::string(ClangExpressionSourceCode::g_expression_prefix) + text,
-          std::move(name)) {}
+          std::string(ClangExpressionSourceCode::g_expression_prefix) + text +
+              std::string(ClangExpressionSourceCode::g_expression_suffix),
+          std::move(name), enable_debugging) {
+  // Write the source code to a file so that LLDB's source manager can display
+  // it when debugging the code.
+  if (enable_debugging) {
+    int temp_fd = -1;
+    llvm::SmallString<128> result_path;
+    llvm::sys::fs::createTemporaryFile("lldb", "expr", temp_fd, result_path);
+    if (temp_fd != -1) {
+      lldb_private::NativeFile file(temp_fd, File::eOpenOptionWrite, true);
+      text = "#line 1 \"" + std::string(result_path) + "\"\n" + text;
+      size_t bytes_written = text.size();
+      file.Write(text.c_str(), bytes_written);
+      if (bytes_written == text.size()) {
+        // If we successfully wrote the source to a temporary file, replace the
+        // function text with the next text containing the line directive.
+        m_function_text =
+            std::string(ClangExpressionSourceCode::g_expression_prefix) + text +
+            std::string(ClangExpressionSourceCode::g_expression_suffix);
+      }
+      file.Close();
+    }
+  }
+}
 
 ClangUtilityFunction::~ClangUtilityFunction() {}
 
 /// Install the utility function into a process
 ///
 /// \param[in] diagnostic_manager
 ///     A diagnostic manager to report errors and warnings to.
 ///

lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp

if (!process)
     return nullptr;
 
   return new ClangFunctionCaller(*process, return_type, function_address,
                                  arg_value_list, name);
 }
 
 std::unique_ptr<UtilityFunction>
 ScratchTypeSystemClang::CreateUtilityFunction(std::string text,
                                               std::string name) {
   TargetSP target_sp = m_target_wp.lock();
   if (!target_sp)
     return {};
 
   return std::make_unique<ClangUtilityFunction>(
-      *target_sp.get(), std::move(text), std::move(name));
+      *target_sp.get(), std::move(text), std::move(name),
+      target_sp->GetDebugUtilityExpression());
 }
 
 PersistentExpressionState *
 ScratchTypeSystemClang::GetPersistentExpressionState() {
   return m_persistent_variables.get();
 }
 
 void ScratchTypeSystemClang::ForgetSource(ASTContext *src_ctx,

lldb/source/Target/Target.cpp

 
 void TargetProperties::DisableSTDIOValueChangedCallback() {
   if (GetDisableSTDIO())
     m_launch_info.GetFlags().Set(lldb::eLaunchFlagDisableSTDIO);
   else
     m_launch_info.GetFlags().Clear(lldb::eLaunchFlagDisableSTDIO);
 }
 
+bool TargetProperties::GetDebugUtilityExpression() const {
+  const uint32_t idx = ePropertyDebugUtilityExpression;
+  return m_collection_sp->GetPropertyAtIndexAsBoolean(
+      nullptr, idx, g_target_properties[idx].default_uint_value != 0);
+}
+
+void TargetProperties::SetDebugUtilityExpression(bool debug) {
+  const uint32_t idx = ePropertyDebugUtilityExpression;
+  m_collection_sp->SetPropertyAtIndexAsBoolean(nullptr, idx, debug);
+}
+
 // Target::TargetEventData
 
 Target::TargetEventData::TargetEventData(const lldb::TargetSP &target_sp)
     : EventData(), m_target_sp(target_sp), m_module_list() {}
 
 Target::TargetEventData::TargetEventData(const lldb::TargetSP &target_sp,
                                          const ModuleList &module_list)
     : EventData(), m_target_sp(target_sp), m_module_list(module_list) {}

lldb/source/Target/TargetProperties.td

def NonStopModeEnabled: Property<"non-stop-mode", "Boolean">,
     DefaultFalse,
     Desc<"Disable lock-step debugging, instead control threads independently.">;
   def RequireHardwareBreakpoints: Property<"require-hardware-breakpoint", "Boolean">,
     DefaultFalse,
     Desc<"Require all breakpoints to be hardware breakpoints.">;
   def AutoInstallMainExecutable: Property<"auto-install-main-executable", "Boolean">,
     DefaultTrue,
     Desc<"Always install the main executable when connected to a remote platform.">;
+  def DebugUtilityExpression: Property<"debug-utility-expression", "Boolean">,
+    DefaultFalse,
+    Desc<"Enable debugging of LLDB-internal utility expressions.">;
 }
 
 let Definition = "process_experimental" in {
   def OSPluginReportsAllThreads: Property<"os-plugin-reports-all-threads", "Boolean">,
     Global,
     DefaultTrue,
     Desc<"Set to False if your OS Plugins doesn't report all threads on each stop.">;
 }