Support demangling for D symbols via dlopen

[timotheecour]

Summary

moved from: https://github.com/llvm-mirror/lldb/pull/3

addresses issue https://issues.dlang.org/show_bug.cgi?id=817 OSX

symbols mangled on gdb,ggdb,cgdb,lldb but not on ubuntu; no line numbers on stacktraces
also, a follow-up to https://reviews.llvm.org/D24794 (Use Clang for D language support until there is a proper language plugin for it.)
related discussion prior to this: http://lists.llvm.org/pipermail/lldb-dev/2016-September/011351.html [lldb-dev] Adding D language demangling support

to try it out, you need lldbdplugin

the shared library that is dlopen'd is defined separately in https://github.com/timotheecour/dtools/blob/master/dtools/lldbdplugin.d (but could be any other implementation with same entry point API)
add this to .lldbinit:
settings set plugin.language.D.pluginfile "path_to/liblldbdplugin.dylib"

before:

(lldb) bt
error: need to add support for DW_TAG_base_type 'immutable(char)' encoded with DW_ATE = 0x10, bit_size = 8
error: need to add support for DW_TAG_base_type 'char' encoded with DW_ATE = 0x10, bit_size = 8
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000100000cd0 test TIMBREAK at foo/test2.d:3
    frame #1: 0x0000000100000e60 test _D4test3funFiZv(a=11) + 128 at test.d:11
    frame #2: 0x0000000100000ec7 test _D4test__T4fun1TiZQiFiZv(a=11) + 23 at test.d:16
    frame #3: 0x0000000100000e8c test _Dmain(args=string[] @ 0x00007ffeefbfcec8) + 28 at test.d:20
    frame #4: 0x000000010014d80f test _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv + 15
    frame #5: 0x000000010014d74e test _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv + 14
    frame #6: 0x000000010014d652 test _d_run_main + 466
    frame #7: 0x00000001000019b5 test main(argc=1, argv=0x00007ffeefbfd028) + 37 at __entrypoint.d:8
    frame #8: 0x00007fff7279f115 libdyld.dylib start + 1
    frame #9: 0x00007fff7279f115 libdyld.dylib start + 1
(lldb)

after:

(lldb) bt
error: need to add support for DW_TAG_base_type 'immutable(char)' encoded with DW_ATE = 0x10, bit_size = 8
error: need to add support for DW_TAG_base_type 'char' encoded with DW_ATE = 0x10, bit_size = 8
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000100000cd0 test TIMBREAK at foo/test2.d:3
    frame #1: 0x0000000100000e60 test void test.fun(a=11) + 128 at test.d:11
    frame #2: 0x0000000100000ec7 test void test.fun1!(a=11).fun1(int) + 23 at test.d:16
    frame #3: 0x0000000100000e8c test _Dmain(args=string[] @ 0x00007ffeefbfce68) + 28 at test.d:20
    frame #4: 0x000000010014d80f test void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll().__lambda1() + 15
    frame #5: 0x000000010014d74e test void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) + 14
    frame #6: 0x000000010014d652 test _d_run_main + 466
    frame #7: 0x00000001000019b5 test main(argc=1, argv=0x00007ffeefbfcfc8) + 37 at __entrypoint.d:8
    frame #8: 0x00007fff7279f115 libdyld.dylib start + 1
    frame #9: 0x00007fff7279f115 libdyld.dylib start + 1

• 1. limitations:
• breakpoint on fully qualified names work, eg: b test.fun but not on templates, eg: b test.fun1

• 1. note
• building lldb from source (even without this patch), bt FOO can be up to 5X slower than when using lldb (version <=5.0.1), which I reported in http://lists.llvm.org/pipermail/lldb-dev/2018-March/013416.html ; this is not related to this PR.

• why is LLDB tag marked as archived?

• I couldn't find lldb as an option in Repository

[johanengelen]

source/Plugins/Language/D/DLanguage.cpp

1

fix header, and also need to clang-format the file

20

add comment about the purpose of these fwd decls.

24

Did you look into using llvm/Support/DynamicLibrary.h ?
That would make the code crossplatform (notably: Windows).

35

typo: "if"

108

Would it help to initialize druntime using a static module constructor in the lldbdplugin dll?
(then you can also do de-init using a static module destructor)

source/Plugins/Language/D/DLanguage.h

1

fix header line

[davide]

This patch has no testcase. It shouldn't be particularly hard to write one, you can take inspiration from the one in lit/.

Thanks!

[timotheecour]

• format
• format
• fixup

[timotheecour]

source/Plugins/Language/D/DLanguage.cpp

108

I don't really like static module constructor because it adds cyclic dependencies, see for vibe.d moving away from it: https://forum.dlang.org/post/[e-mail]
explicit calling d_initialize is simple enough.

[timotheecour]

• use llvm::sys::DynamicLibrary
• format

[timotheecour]

source/Plugins/Language/D/DLanguage.cpp

24

thx for tip! done, used llvm::sys::DynamicLibrary

[timotheecour]

In D44321#1033325, @davide wrote:

This patch has no testcase. It shouldn't be particularly hard to write one, you can take inspiration from the one in lit/.

Thanks!

5d29d3f0aa2f540c277a67eac5873feca8c62d51 (Support for OCaml native debugging) introduced a much larger change and had no test case either. If you really insist on a test case please provide more concrete guidance.

[davide]

In D44321#1034043, @timotheecour wrote:

In D44321#1033325, @davide wrote:

This patch has no testcase. It shouldn't be particularly hard to write one, you can take inspiration from the one in lit/.

Thanks!

5d29d3f0aa2f540c277a67eac5873feca8c62d51 (Support for OCaml native debugging) introduced a much larger change and had no test case either. If you really insist on a test case please provide more concrete guidance.

That was a while ago. In fact, the OCaml support is going to be removed. New languages need to meet an high(er) bar to be included. One of the requirements is, e.g. being tested.
It's a little more complicated for D because it's an out-of-tree compiler so it poses interesting challenges.
Let me think about it a little more and I'll come back to you.

Thanks,

Davide

[timotheecour]

• format
• use llvm::sys::DynamicLibrary
• added DLanguageProperties

[timotheecour]

update: we can now set lldbdplugin in .lldbinit via settings set plugin.language.D.pluginfile so this avoids hardcoding that file

[timotheecour]

• added doc

[timotheecour]

It's a little more complicated for D because it's an out-of-tree compiler so it poses interesting challenges.

the demangling itself is thoroughly tested in https://github.com/dlang/druntime/blob/master/src/core/demangle.d

[timotheecour]

Note that the D plugin is not loaded by default (so default behavior is not affected) ; it's only loaded if user adds to his .lldbinit:
settings set plugin.language.D.pluginfile "path_to/liblldbdplugin.dylib"

[johanengelen]

source/Plugins/Language/D/DLanguage.cpp

108

Module ctors don't add cyclic dependencies by themselves. There is also no danger of that here.
How do you de-initialize druntime? (without de-init, there is a big mem leak)

[johanengelen]

In D44321#1034141, @timotheecour wrote:

It's a little more complicated for D because it's an out-of-tree compiler so it poses interesting challenges.

the demangling itself is thoroughly tested in https://github.com/dlang/druntime/blob/master/src/core/demangle.d

It's not the correct demangling of strings that needs to be tested. What needs testing:

• recognising D language and D mangled name
• loading of dynamic lib (plus testing that when loading fails lldb doesn't crash/exit; note: errmsg can be improved to e.g. "Failed to load D language plugin...")
• test that indeed the demangling function of the dyn lib is called

To test this, I think you can make a mock dynamic lib that implements the required interface, but in C++. The mock demangler could be something like this or even simpler:

extern "C" char* lldbd_demangle(size_t length, const char* mangled) {
   if (mangled == "_D3fooFZv") // pseudo code
       return "void foo()";
   else
       return mangled;
}

Then build that library in the test, and test loading and use on a fabricated piece of code (probably easiest to use C code with this in it: extern "C" _D3fooFZv() { trap(); }.

[johanengelen]

In D44321#1034168, @johanengelen wrote:

extern "C" char* lldbd_demangle(size_t length, const char* mangled) {
   if (mangled == "_D3fooFZv") // pseudo code
       return "void foo()";
   else
       return mangled;
}

Actually, this is not the correct interface is it? The returned pointer should point to newly allocated memory using malloc, right?
Good to document that somewhere.

[timotheecour]

• document C interface

[timotheecour]

Actually, this is not the correct interface is it? The returned pointer should point to newly allocated memory using malloc, right?
Good to document that somewhere.

not 100% sure what you mean in that comment but I just pushed a commit that clarifies that point: 'The returned pointer should be allocated via malloc and owned by the caller' ; the reference implementation of liblldbplugin does the right thing.

[timotheecour]

• added d_terminate
• format

[timotheecour]

This comment was removed by timotheecour.

[johanengelen]

In D44321#1038160, @timotheecour wrote:

How do you de-initialize druntime? (without de-init, there is a big mem leak)

There is no memory leak because d_initialize once (using c++11 static initialization pattern) and is intended to last for duration of application; so druntime will be initialized only once, upon 1st use.

When druntime is initialized, a number of resources are allocated (e.g. memory and mutex). Yes you initialize druntime once, I can see that. You don't deinitialize druntime at all: that's the resource leak.

[timotheecour]

source/Plugins/Language/D/DLanguage.cpp

108

Module ctors don't add cyclic dependencies by themselves. There is also no danger of that here.

• We can have cyclic dependencies but not when Module ctors are involved; see previous argument I made with vibe.d, or other threads (eg https://forum.dlang.org/post/[e-mail]).

• Also, current code allows for liblldbplugin.d to be imported from another module whereas it would cause complications if it contained a module ctor that would initialize druntime.

calling d_initialize as I'm doing works fine.

108

How do you de-initialize druntime? (without de-init, there is a big mem leak)

There is no memory leak because d_initialize once (using c++11 static initialization pattern) and is intended to last for duration of application; so druntime will be initialized only once, upon 1st use.

I've however added d_terminate in case a future PR wants to de-initialize druntime (eg in DLanguage::Terminate)

[timotheecour]

When druntime is initialized, a number of resources are allocated (e.g. memory and mutex). Yes you initialize druntime once, I can see that. You don't deinitialize druntime at all: that's the resource leak.

Where would you want me to deinit? inside DLanguage::Terminate ?

[timotheecour]

• fix assert fail when plugin.language.D.pluginfile empty

[johanengelen]

In D44321#1038241, @timotheecour wrote:

When druntime is initialized, a number of resources are allocated (e.g. memory and mutex). Yes you initialize druntime once, I can see that. You don't deinitialize druntime at all: that's the resource leak.

Where would you want me to deinit? inside DLanguage::Terminate ?

I don't know enough of LLDB to know where to put the deinitialization. Sorry, can't help you there, except for making it easy and doing the deinitialization using a dtor in the library...

[labath]

The Terminate function is a good place to do this kind of cleanup.

source/Plugins/Language/D/DLanguage.cpp

35–37

Instead of using decltype everywhere, you could just define the type of these functions with typedef and use that.

[jblachly]

I sat down to write my own plugin today and found this.

@timotheecour , thanks for writing this -- can I help with anything to get this across the finish line?

include/lldb/Core/Mangled.h

enum NamePreference {
     ePreferMangled,
     ePreferDemangled,
     ePreferDemangledWithoutArguments
   };
 
   enum ManglingScheme {
     eManglingSchemeNone = 0,
     eManglingSchemeMSVC,
-    eManglingSchemeItanium
+    eManglingSchemeItanium,
+    eManglingSchemeD
   };
 
   //----------------------------------------------------------------------
   /// Default constructor.
   ///
   /// Initialize with both mangled and demangled names empty.
   //----------------------------------------------------------------------
   Mangled();

source/Core/CMakeLists.txt

LINK_LIBS
     lldbHost
     lldbInterpreter
     lldbSymbol
     lldbTarget
     lldbUtility
     lldbPluginProcessUtility
     lldbPluginCPlusPlusLanguage
     lldbPluginObjCLanguage
+    lldbPluginDLanguage
     lldbPluginObjectFileJIT
     ${LLDB_CURSES_LIBS}
 
   LINK_COMPONENTS
     BinaryFormat
     Support
     Demangle
   )
 
 # Needed to properly resolve references in a debug build.
 # TODO: Remove once we have better layering
 set_target_properties(lldbCore PROPERTIES LINK_INTERFACE_MULTIPLICITY 4)

source/Core/Mangled.cpp

 #include "lldb/Utility/Logging.h"
 #include "lldb/Utility/RegularExpression.h"
 #include "lldb/Utility/Stream.h"
 #include "lldb/Utility/Timer.h"
 #include "lldb/lldb-enumerations.h" // for LanguageType
 
 #include "Plugins/Language/CPlusPlus/CPlusPlusLanguage.h"
 #include "Plugins/Language/ObjC/ObjCLanguage.h"
+#include "Plugins/Language/D/DLanguage.h"
 
 #include "llvm/ADT/StringRef.h"    // for StringRef
 #include "llvm/Support/Compiler.h" // for LLVM_PRETT...
 
 #include <mutex>   // for mutex, loc...
 #include <string>  // for string
 #include <utility> // for pair
 

 #endif
 
 static inline Mangled::ManglingScheme cstring_mangling_scheme(const char *s) {
   if (s) {
     if (s[0] == '?')
       return Mangled::eManglingSchemeMSVC;
     if (s[0] == '_' && s[1] == 'Z')
       return Mangled::eManglingSchemeItanium;
+    if (DLanguage::IsDMangledName(s))
+      return Mangled::eManglingSchemeD;
   }
   return Mangled::eManglingSchemeNone;
 }
 
 static inline bool cstring_is_mangled(const char *s) {
   return cstring_mangling_scheme(s) != Mangled::eManglingSchemeNone;
 }
 
#endif
             log->Printf("demangled itanium: %s -> \"%s\"", mangled_name,
                         demangled_name);
           else
             log->Printf("demangled itanium: %s -> error: failed to demangle",
                         mangled_name);
         }
         break;
       }
+      case eManglingSchemeD: {
+        demangled_name = DLanguage::demangle(m_mangled);
+        break;
+      }
       case eManglingSchemeNone:
         break;
       }
       if (demangled_name) {
         m_demangled.SetCStringWithMangledCounterpart(demangled_name, m_mangled);
         free(demangled_name);
       }
     }
lldb::LanguageType Mangled::GuessLanguage() const {
   ConstString mangled = GetMangledName();
   if (mangled) {
     if (GetDemangledName(lldb::eLanguageTypeUnknown)) {
       const char *mangled_name = mangled.GetCString();
       if (CPlusPlusLanguage::IsCPPMangledName(mangled_name))
         return lldb::eLanguageTypeC_plus_plus;
       else if (ObjCLanguage::IsPossibleObjCMethodName(mangled_name))
         return lldb::eLanguageTypeObjC;
+      else if (DLanguage::IsDMangledName(mangled_name))
+        return lldb::eLanguageTypeD;
     }
   } else {
     // ObjC names aren't really mangled, so they won't necessarily be in the 
     // mangled name slot.
     ConstString demangled_name = GetDemangledName(lldb::eLanguageTypeUnknown);
     if (demangled_name 
         && ObjCLanguage::IsPossibleObjCMethodName(demangled_name.GetCString()))
       return lldb::eLanguageTypeObjC;

source/Plugins/Language/CMakeLists.txt

 add_subdirectory(CPlusPlus)
 add_subdirectory(Go)
 add_subdirectory(Java)
 add_subdirectory(ObjC)
 add_subdirectory(ObjCPlusPlus)
 add_subdirectory(OCaml)
+add_subdirectory(D)

source/Plugins/Language/D/CMakeLists.txt

+add_lldb_library(lldbPluginDLanguage PLUGIN
+  DLanguage.cpp
+  
+  LINK_LIBS
+    lldbCore
+    lldbTarget
+)

source/Plugins/Language/D/DLanguage.h

+//===-- DLanguage.h ----------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef liblldb_DLanguage_h_
+#define liblldb_DLanguage_h_
+
+// C Includes
+// C++ Includes
+// Other libraries and framework includes
+// Project includes
+#include "lldb/Target/Language.h"
+#include "lldb/lldb-private.h"
+
+namespace lldb_private {
+
+class DLanguage : public Language {
+public:
+  DLanguage() = default;
+
+  ~DLanguage() override = default;
+
+  lldb::LanguageType GetLanguageType() const override {
+    return lldb::eLanguageTypeD;
+  }
+
+  //------------------------------------------------------------------
+  // Static Functions
+  //------------------------------------------------------------------
+  static void Initialize();
+
+  static void Terminate();
+
+  static lldb_private::Language *CreateInstance(lldb::LanguageType language);
+
+  static lldb_private::ConstString GetPluginNameStatic();
+
+  static bool IsDMangledName(const char *name);
+  // TODO: not static?
+  static char* demangle(const ConstString &mangled);
+
+  //------------------------------------------------------------------
+  // PluginInterface protocol
+  //------------------------------------------------------------------
+  ConstString GetPluginName() override;
+
+  uint32_t GetPluginVersion() override;
+};
+
+} // namespace lldb_private
+
+#endif // liblldb_DLanguage_h_

source/Plugins/Language/D/DLanguage.cpp

+//===-- DLanguage.cpp
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "DLanguage.h"
+
+#include "lldb/Core/PluginManager.h"
+#include "lldb/Utility/ConstString.h"
+
+#include <dlfcn.h>
+
+using namespace lldb;
+using namespace lldb_private;
+
+char* lldbd_demangle(size_t length, const char* mangled);
+void d_initialize();
+
+// TODO:MOVE
+struct SharedLib{
+  void* handle;
+  const char* libraryFile;
+  int flagDefault = RTLD_LOCAL | RTLD_LAZY;
+  SharedLib(){
+  }
+
+  ~SharedLib(){
+    release();
+  }
+
+  // Return true of `dlopen` succeeded
+  bool open(const char* libraryFile, int flag)
+  {
+      release();
+      this->libraryFile = libraryFile;
+      handle = dlopen(libraryFile, flag);
+      if (handle)
+          return true;
+      return false;
+  }
+
+  void release()
+  {
+      // if(handle) seems needed: https://stackoverflow.com/questions/11412943is-it-safe-to-call-dlclosenull
+      if (handle)
+          dlclose(handle);
+  }
+
+  template<typename Fun>
+  Fun* getFun(const char*symbol)
+  {
+      assert(handle);
+      return reinterpret_cast<Fun*>(dlsym(handle, symbol));
+  }
+};
+
+void DLanguage::Initialize() {
+  PluginManager::RegisterPlugin(GetPluginNameStatic(), "D Language",
+                                CreateInstance);
+}
+
+void DLanguage::Terminate() {
+  PluginManager::UnregisterPlugin(CreateInstance);
+}
+
+lldb_private::ConstString DLanguage::GetPluginNameStatic() {
+  static ConstString g_name("D");
+  return g_name;
+}
+
+//------------------------------------------------------------------
+// PluginInterface protocol
+//------------------------------------------------------------------
+lldb_private::ConstString DLanguage::GetPluginName() {
+  return GetPluginNameStatic();
+}
+
+uint32_t DLanguage::GetPluginVersion() { return 1; }
+
+//------------------------------------------------------------------
+// Static Functions
+//------------------------------------------------------------------
+Language *DLanguage::CreateInstance(lldb::LanguageType language) {
+  switch (language) {
+  case lldb::eLanguageTypeD:
+    return new DLanguage();
+  default:
+    return nullptr;
+  }
+}
+
+char* DLanguage::demangle(const ConstString &mangled) {
+  auto len=mangled.GetLength();
+  auto s=mangled.GetCString();
+  // IMPROVE
+  static auto fun=[]() -> decltype(lldbd_demangle)*{
+    auto lib2=new SharedLib();
+    // TODO: so vs dylib
+    auto file="liblldbdplugin.dylib";
+    if(!lib2->open(file, lib2->flagDefault)){
+      return nullptr;
+    }
+
+    auto fun0=lib2->getFun<decltype(d_initialize)>("d_initialize");
+    (*fun0)();
+
+    auto fun=lib2->getFun<decltype(lldbd_demangle)>("lldbd_demangle");
+    assert(fun);
+    return fun;
+  }();
+  if(!fun) return nullptr;
+  return (*fun)(len, s);
+}
+
+bool DLanguage::IsDMangledName(const char *name) {
+  if (name == nullptr)
+    return false;
+  return (name[0] != '\0' && name[0] == '_' && name[1] == 'D');
+}