[lli] Add --jit-linker command line argument

[sgraenitz]

Summary

The argument value determines the dynamic linker to use (default, rtdyld or jitlink). The JITLink implementation only supports in-process JITing for now.

[sgraenitz]

The major issue I faced with this patch is the clash of the GDB JIT interface symbols __jit_debug_register_code and __jit_debug_descriptor. These are defined in both libraries, ExecutionEngine and OrcTargetProcess, and I didn't find a "nice" way to deduplicate them. My current approach turns the definitions in ExecutionEngine into externals and adds OrcTargetProcess as link component to ExecutionEngine. Maybe there is another way to fix this?

[lhames]

This patch adds a --dyld flag

Could this be changed to -jit-linker? It's verbose, but I think -dyld is too likely to cause confusion, especially when we already have -dlopen (which really does call the native dynamic linker). I'm open to shorter names too, it's -dyld that I really want to avoid.

In D97339#2583283, @sgraenitz wrote:

The major issue I faced with this patch is the clash of the GDB JIT interface symbols __jit_debug_register_code and __jit_debug_descriptor. These are defined in both libraries, ExecutionEngine and OrcTargetProcess, and I didn't find a "nice" way to deduplicate them. My current approach turns the definitions in ExecutionEngine into externals and adds OrcTargetProcess as link component to ExecutionEngine. Maybe there is another way to fix this?

We could create an entirely new library for this, but that feels like overkill. I think OrcTargetProcess seems reasonable. I think that the new dependence diagram would look like:

ORC -> { JITLink, OrcShared }
OrcTargetProcess -> { OrcShared }
ExecutionEngine -> { RuntimeDyld, OrcTargetProcess }

[sgraenitz]

In D97339#2584160, @lhames wrote:

This patch adds a --dyld flag

Could this be changed to -jit-linker? It's verbose, but I think -dyld is too likely to cause confusion

I don't really like the duplication in -jit-linker=jitlink, but well it's acceptable.

We could create an entirely new library for this, but that feels like overkill. I think OrcTargetProcess seems reasonable. I think that the new dependence diagram would look like:

ORC -> { JITLink, OrcShared }
OrcTargetProcess -> { OrcShared }
ExecutionEngine -> { RuntimeDyld, OrcTargetProcess }

Yes, OrcTargetProcess is so small that the overhead in ExecutionEngine will hardly be noticeable.
So let's keep it like that, including the FIXME. I guess we can move the symbol definitions from OrcTargetProcess the the Orc runtime one day.

[lhames]

In D97339#2584418, @sgraenitz wrote:

I guess we can move the symbol definitions from OrcTargetProcess the the Orc runtime one day.

That was my first thought too, but then I realized that for the debugger to see these symbols in the runtime (which is linked in via the JIT) we'd need to have a different solution for registering debug info for JIT'd code, at which point this would be redundant. :)

[sgraenitz]

Rename command line flag: dyld -> jit-linker

[sgraenitz]

Remove the debug support plugin for the moment

[thakis]

My bots are failing a few tests on non-mac: http://45.33.8.238/linux/40624/step_12.txt

I think that's due to a build config problem in my end but it's not obvious to me what's wrong. In case you immediately see it, I'm grateful for hints. If not, I'll figure it out eventually :)

Looks like it's not findig the mac mangled gdb symbol name -- what's supposed to provide that?

[thakis]

In D97339#2597610, @thakis wrote:

My bots are failing a few tests on non-mac: http://45.33.8.238/linux/40624/step_12.txt

I think that's due to a build config problem in my end but it's not obvious to me what's wrong. In case you immediately see it, I'm grateful for hints. If not, I'll figure it out eventually :)

Looks like it's not findig the mac mangled gdb symbol name -- what's supposed to provide that?

(Sorry, this question would've been a better fit on D97335 :) Still happy for pointers, though.)

[thakis]

(Sorry, this question would've been a better fit on D97335 :) Still happy for pointers, though.)

I think I figured out the linux bit. SelfTargetProcessControl::lookupSymbols strips GlobalManglingPrefix and the test actually looks up llvm_orc_registerJITLoaderGDBWrapper without underscore. For dlsym() to find this in dlopen(NULL, ...), llvm-jitlink needs to be linked with -rdynamic. I hope it's similarly straightforward on Windows...

[thakis]

I think I figured out the linux bit. SelfTargetProcessControl::lookupSymbols strips GlobalManglingPrefix and the test actually looks up llvm_orc_registerJITLoaderGDBWrapper without underscore. For dlsym() to find this in dlopen(NULL, ...), llvm-jitlink needs to be linked with -rdynamic. I hope it's similarly straightforward on Windows...

And I found a Windows box too. Looks like the test fails on Windows with a normal build config too. Windows doesn't support dlsym()ing (well, GetProcAddress()ing) symbols from the main executable, only from dynamic libraries. So the whole approach seems Windows-incompatible. I'll try to revert some of the bits here to unbreak stuff.

(It'd be nice to land changes spread out over time, so that if a change breaks something it's easier to revert....)

[klausler]

I'm getting a Release mode shared library build failure on x86-64 Linux w/ gcc 9.3.0:

/opt/binutils/2.32/bin/ld: CMakeFiles/lli.dir/lli.cpp.o: undefined reference to symbol '_ZN4llvm7jitlink20JITLinkMemoryManagerD2Ev'
/opt/binutils/2.32/bin/ld: /home/pklausler/llvm-project/build/x86/gcc/9.3.0/Release/shared/./lib/libLLVMJITLink.so.13git: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
tools/lli/CMakeFiles/lli.dir/build.make:139: recipe for target 'bin/lli' failed

Not sure yet whether it's due to this commit or something else.

UPDATE: Reverting this commit doesn't fix the build; will look elsewhere.

[klausler]

I still have not found the specific commit that has broken the shared library Release build on x86-64 Linux with the error that I quoted above. (That build remains broken for me, though.)

[sgraenitz]

@klausler Thanks for reporting this. lli was missing JITLink as a link component in CMake. I fixed it with https://reviews.llvm.org/rG295ea050ad594b8868b6dd944824ba9a16273f91. Sorry for the inconvenience.

llvm/tools/lli/lli.cpp

 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/CodeGen/CommandFlags.h"
 #include "llvm/CodeGen/LinkAllCodegenComponents.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/ExecutionEngine/GenericValue.h"
 #include "llvm/ExecutionEngine/Interpreter.h"
+#include "llvm/ExecutionEngine/JITSymbol.h"
 #include "llvm/ExecutionEngine/JITEventListener.h"
 #include "llvm/ExecutionEngine/MCJIT.h"
 #include "llvm/ExecutionEngine/ObjectCache.h"
 #include "llvm/ExecutionEngine/Orc/DebugUtils.h"
 #include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
 #include "llvm/ExecutionEngine/Orc/JITTargetMachineBuilder.h"
 #include "llvm/ExecutionEngine/Orc/LLJIT.h"
 #include "llvm/ExecutionEngine/Orc/MachOPlatform.h"
 #include "llvm/ExecutionEngine/Orc/OrcRemoteTargetClient.h"
 #include "llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h"
+#include "llvm/ExecutionEngine/Orc/SymbolStringPool.h"
+#include "llvm/ExecutionEngine/Orc/TPCEHFrameRegistrar.h"
+#include "llvm/ExecutionEngine/Orc/TargetProcess/RegisterEHFrames.h"
 #include "llvm/ExecutionEngine/Orc/TargetProcess/TargetExecutionUtils.h"
 #include "llvm/ExecutionEngine/SectionMemoryManager.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/IRReader/IRReader.h"

 
 static codegen::RegisterCodeGenFlags CGF;
 
 #define DEBUG_TYPE "lli"
 
 namespace {
 
   enum class JITKind { MCJIT, OrcLazy };
+  enum class JITLinkerKind { Default, RuntimeDyld, JITLink };
 
   cl::opt<std::string>
   InputFile(cl::desc("<input bitcode>"), cl::Positional, cl::init("-"));
 
   cl::list<std::string>
   InputArgv(cl::ConsumeAfter, cl::desc("<program arguments>..."));
 
   cl::opt<bool> ForceInterpreter("force-interpreter",
                                  cl::desc("Force interpretation: disable JIT"),
                                  cl::init(false));
 
   cl::opt<JITKind> UseJITKind(
       "jit-kind", cl::desc("Choose underlying JIT kind."),
       cl::init(JITKind::MCJIT),
       cl::values(clEnumValN(JITKind::MCJIT, "mcjit", "MCJIT"),
                  clEnumValN(JITKind::OrcLazy, "orc-lazy",
                             "Orc-based lazy JIT.")));
 
+  cl::opt<JITLinkerKind>
+      JITLinker("jit-linker", cl::desc("Choose the dynamic linker/loader."),
+                cl::init(JITLinkerKind::Default),
+                cl::values(clEnumValN(JITLinkerKind::Default, "default",
+                                      "Default for platform and JIT-kind"),
+                           clEnumValN(JITLinkerKind::RuntimeDyld, "rtdyld",
+                                      "RuntimeDyld"),
+                           clEnumValN(JITLinkerKind::JITLink, "jitlink",
+                                      "Orc-specific linker")));
+
   cl::opt<unsigned>
   LazyJITCompileThreads("compile-threads",
                         cl::desc("Choose the number of compile threads "
                                  "(jit-kind=orc-lazy only)"),
                         cl::init(0));
 
   cl::list<std::string>
   ThreadEntryPoints("thread-entry",
cl::list<BuiltinFunctionKind> GenerateBuiltinFunctions(
                             "__dump_jit_debug_objects",
                             "Dump __jit_debug_descriptor in-memory debug "
                             "objects as tool output")),
       cl::Hidden);
 
   ExitOnError ExitOnErr;
 }
 
+LLVM_ATTRIBUTE_USED void linkComponents() {
+  errs() << (void *)&llvm_orc_registerEHFrameSectionWrapper
+         << (void *)&llvm_orc_deregisterEHFrameSectionWrapper;
+}
+
 //===----------------------------------------------------------------------===//
 // Object cache
 //
 // This object cache implementation writes cached objects to disk to the
 // directory specified by CacheDir, using a filename provided in the module
 // descriptor. The cache tries to load a saved object using that path if the
 // file exists. CacheDir defaults to "", in which case objects are cached
 // alongside their originating bitcodes.
case LLJITPlatform::MachO:
       Builder.setPlatformSetUp(orc::setUpMachOPlatform);
       ExitOnErr(orc::enableObjCRegistration("libobjc.dylib"));
       break;
     default:
       llvm_unreachable("Unrecognized platform value");
     }
   }
 
+  std::unique_ptr<orc::TargetProcessControl> TPC = nullptr;
+  if (JITLinker == JITLinkerKind::JITLink) {
+    TPC = ExitOnErr(orc::SelfTargetProcessControl::Create(
+        std::make_shared<orc::SymbolStringPool>()));
+
+    Builder.setObjectLinkingLayerCreator([&TPC](orc::ExecutionSession &ES,
+                                                const Triple &) {
+      auto L = std::make_unique<orc::ObjectLinkingLayer>(ES, TPC->getMemMgr());
+      L->addPlugin(std::make_unique<orc::EHFrameRegistrationPlugin>(
+          ES, ExitOnErr(orc::TPCEHFrameRegistrar::Create(*TPC))));
+      return L;
+    });
+  }
+
   auto J = ExitOnErr(Builder.create());
 
   auto *ObjLayer = &J->getObjLinkingLayer();
   if (auto *RTDyldObjLayer = dyn_cast<orc::RTDyldObjectLinkingLayer>(ObjLayer))
     RTDyldObjLayer->registerJITEventListener(
         *JITEventListener::createGDBRegistrationListener());
 
   if (PerModuleLazy)
int runOrcLazyJIT(const char *ProgName) {
   for (auto &ThreadEntryPoint : ThreadEntryPoints) {
     auto EntryPointSym = ExitOnErr(J->lookup(ThreadEntryPoint));
     typedef void (*EntryPointPtr)();
     auto EntryPoint =
       reinterpret_cast<EntryPointPtr>(static_cast<uintptr_t>(EntryPointSym.getAddress()));
     AltEntryThreads.push_back(std::thread([EntryPoint]() { EntryPoint(); }));
   }
 
-  // Run main.
-  auto MainSym = ExitOnErr(J->lookup("main"));
-
-  typedef int (*MainFnPtr)(int, char *[]);
-  auto Result = orc::runAsMain(
-      jitTargetAddressToFunction<MainFnPtr>(MainSym.getAddress()), InputArgv,
-      StringRef(InputFile));
+  // Resolve and run the main function.
+  JITEvaluatedSymbol MainSym = ExitOnErr(J->lookup("main"));
+  int Result;
+
+  if (TPC) {
+    // TargetProcessControl-based execution with JITLink.
+    Result = ExitOnErr(TPC->runAsMain(MainSym.getAddress(), InputArgv));
+  } else {
+    // Manual in-process execution with RuntimeDyld.
+    using MainFnTy = int(int, char *[]);
+    auto MainFn = jitTargetAddressToFunction<MainFnTy *>(MainSym.getAddress());
+    Result = orc::runAsMain(MainFn, InputArgv, StringRef(InputFile));
+  }
 
   // Wait for -entry-point threads.
   for (auto &AltEntryThread : AltEntryThreads)
     AltEntryThread.join();
 
   // Run destructors.
   ExitOnErr(J->deinitialize(J->getMainJITDylib()));