[lli] Test debug support in RuntimeDyld with built-in functions

[sgraenitz]

Summary

When lli runs the below IR, it emits in-memory debug objects and registers them with the GDB JIT interface. The tests dump and check the registered information. IR has limited ability to produce complex output in a portable way. Instead the tests rely on built-in functions implemented in lli. They use a new command line flag -generate=function-name to instruct the ORC JIT to expose the built-in function with the given name to the JITed program.

debug-descriptor-elf-minimal.ll calls __dump_jit_debug_descriptor() to reflect the list of debug entries issued for itself after emitting the main module. The output is textual and can be checked straight away.

debug-objects-elf-minimal.ll calls __dump_jit_debug_objects(), which instructs lli to walk through the list of debug entries and append the encountered in-memory objects to the program output. We feed this output into llvm-dwarfdump to parse the DWARF in each file and dump their structures.

We can do the same for JITLink once D97335 has landed.

[sgraenitz]

I can imagine tests like these will be handy for testing advanced JIT features inside LLVM in the future. Adding the -generate flag with a hard-coded set of options was not my preferred solution, but all other attempts failed for portability reasons. I am happy to discuss ideas how to improve it.

llvm/tools/lli/ExecutionUtils.cpp

93

This global static pointer is working for the moment, but of course it's not exactly ideal. I need a better way to expose a member function callback. TPCIndirectionUtils offers writeResolverBlock(), but it only works with TPC. LocalTrampolinePool also uses resolver blocks, but I don't need the landing address resolution.

@lhames Can you think of a solution that also works for RuntimeDyld?

[lhames]

I can imagine tests like these will be handy for testing advanced JIT features inside LLVM in the future. Adding the -generate flag with a hard-coded set of options was not my preferred solution, but all other attempts failed for portability reasons. I am happy to discuss ideas how to improve it.

What other options did you consider?
How about a list of strings, rather than hard-coded options?
Or writing __dump_jit_debug_descriptor and __dump_jit_debug_objects as extern "C" functions and dispensing with the generator entirely (they'd be picked up as process symbols)?

I think you can land this as-is and we can revise the design later if we come up with a better approach.

[sgraenitz]

In D97694#2596188, @lhames wrote:

I can imagine tests like these will be handy for testing advanced JIT features inside LLVM in the future. Adding the -generate flag with a hard-coded set of options was not my preferred solution, but all other attempts failed for portability reasons. I am happy to discuss ideas how to improve it.

What other options did you consider?

In my first approach in D97337 I put all the code in IR, but I didn't get the dumping via printf to work cross platform. I considered platform-specific versions, but they appear a lot less attractive, because they add much extra work and they would be restricted to test only their native object formats. In a second approach, I moved the dumping code to an external C file and compiled it to the LLVM build-tree with the native toolchain. The idea was to add it to the execution session with the -extra-objects command line option. Unfortunately, it shows the same limitation as the "IR in platform-specific versions" approach. Due to mangling differences it's restricted to its native object format: leading underscore with MachO, MSVC mangling with COFF (and in lookups ORC assumes mangling to be identical across all translation units). As a result, the lookup of printf failed (for ELF on MachO e.g. it was looking for _printf).

How about a list of strings, rather than hard-coded options?
Or writing __dump_jit_debug_descriptor and __dump_jit_debug_objects as extern "C" functions and dispensing with the generator entirely (they'd be picked up as process symbols)?

I considered that, but as they are built-ins, they would still be hard-coded eventually. For the reader it would obfuscate the connection between the command line argument and the implementation of a built-in. The user couldn't lookup valid argument values anymore (via lli --help-hidden). Also, __dump_jit_debug_objects has an extra requirement: it needs to create a tool output stream to append the debug objects. With a static function it required a global static pointer in the same way as I use it now for the generator.

I think you can land this as-is and we can revise the design later if we come up with a better approach.

Yes let's do that. It's all in the lli tool and not in libraries, so there is no risk to break client code.

[mgorny]

This test is causing a segfault on 32-bit x86: https://bugs.llvm.org/show_bug.cgi?id=51292

Could you take a look, please?

[sgraenitz]

Yes, I can make some time for it next week. Do we have a bot that builds the relevant configuration? Otherwise, is it straightforward for you to reproduce on Linux with LLVM_BUILD_32_BITS=ON? Or does it take any extra steps/settings?

[sgraenitz]

For the moment D107640 limits the debug object tests to 64-bit processes on x86_64 hosts. This is what the debug plugin and tests were made for.

llvm/test/ExecutionEngine/OrcLazy/debug-descriptor-elf-minimal.ll

+; RUN: lli --jit-kind=orc-lazy --per-module-lazy \
+; RUN:     --generate=__dump_jit_debug_descriptor %s | FileCheck %s
+;
+; CHECK: Reading __jit_debug_descriptor at 0x{{.*}}
+; CHECK: Version: 1
+; CHECK: Action: JIT_REGISTER_FN
+; CHECK:       Entry               Symbol File             Size  Previous Entry
+; CHECK: [ 0]  0x{{.*}}            0x{{.*}}              {{.*}}  0x0000000000000000
+
+target triple = "x86_64-unknown-unknown-elf"
+
+; Built-in symbol provided by the JIT
+declare void @__dump_jit_debug_descriptor(i8*)
+
+; Host-process symbol from the GDB JIT interface
+@__jit_debug_descriptor = external global i8, align 1
+
+define i32 @main() !dbg !9 {
+  %1 = alloca i32, align 4
+  store i32 0, i32* %1, align 4
+  call void @__dump_jit_debug_descriptor(i8* @__jit_debug_descriptor), !dbg !13
+  ret i32 0, !dbg !14
+}
+
+!llvm.module.flags = !{!0, !1, !2, !3, !4}
+!llvm.dbg.cu = !{!5}
+!llvm.ident = !{!8}
+
+!0 = !{i32 2, !"SDK Version", [3 x i32] [i32 10, i32 15, i32 6]}
+!1 = !{i32 7, !"Dwarf Version", i32 4}
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = !{i32 1, !"wchar_size", i32 4}
+!4 = !{i32 7, !"PIC Level", i32 2}
+!5 = distinct !DICompileUnit(language: DW_LANG_C99, file: !6, producer: "compiler version", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, nameTableKind: None)
+!6 = !DIFile(filename: "source-file.c", directory: "/workspace")
+!7 = !{}
+!8 = !{!"compiler version"}
+!9 = distinct !DISubprogram(name: "main", scope: !6, file: !6, line: 4, type: !10, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !5, retainedNodes: !7)
+!10 = !DISubroutineType(types: !11)
+!11 = !{!12}
+!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!13 = !DILocation(line: 5, column: 3, scope: !9)
+!14 = !DILocation(line: 6, column: 3, scope: !9)

llvm/test/ExecutionEngine/OrcLazy/debug-objects-elf-minimal.ll

+; RUN: lli --jit-kind=orc-lazy --per-module-lazy \
+; RUN:     --generate=__dump_jit_debug_objects %s | llvm-dwarfdump --diff - | FileCheck %s
+;
+; CHECK: -: file format elf64-x86-64
+; CHECK: .debug_info contents:
+; CHECK: 0x00000000: Compile Unit: length = 0x00000047, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x0000004b)
+; CHECK: DW_TAG_compile_unit
+; CHECK:               DW_AT_producer ("compiler version")
+; CHECK:               DW_AT_language (DW_LANG_C99)
+; CHECK:               DW_AT_name ("source-file.c")
+; CHECK:               DW_AT_stmt_list  ()
+; CHECK:               DW_AT_comp_dir ("/workspace")
+; CHECK:               DW_AT_low_pc ()
+; CHECK:               DW_AT_high_pc  ()
+; CHECK:   DW_TAG_subprogram
+; CHECK:                 DW_AT_low_pc ()
+; CHECK:                 DW_AT_high_pc  ()
+; CHECK:                 DW_AT_frame_base (DW_OP_reg7 RSP)
+; CHECK:                 DW_AT_name ("main")
+; CHECK:                 DW_AT_decl_file  ("/workspace/source-file.c")
+; CHECK:                 DW_AT_decl_line  (4)
+; CHECK:                 DW_AT_type ("int")
+; CHECK:                 DW_AT_external (true)
+; CHECK:   DW_TAG_base_type
+; CHECK:                 DW_AT_name ("int")
+; CHECK:                 DW_AT_encoding (DW_ATE_signed)
+; CHECK:                 DW_AT_byte_size  (0x04)
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-unknown-elf"
+
+; Built-in symbol provided by the JIT
+declare void @__dump_jit_debug_objects(i8*)
+
+; Host-process symbol from the GDB JIT interface
+@__jit_debug_descriptor = external global i8, align 1
+
+define i32 @main() !dbg !9 {
+  %1 = alloca i32, align 4
+  store i32 0, i32* %1, align 4
+  call void @__dump_jit_debug_objects(i8* @__jit_debug_descriptor), !dbg !13
+  ret i32 0, !dbg !14
+}
+
+!llvm.module.flags = !{!0, !1, !2, !3, !4}
+!llvm.dbg.cu = !{!5}
+!llvm.ident = !{!8}
+
+!0 = !{i32 2, !"SDK Version", [3 x i32] [i32 10, i32 15, i32 6]}
+!1 = !{i32 7, !"Dwarf Version", i32 4}
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = !{i32 1, !"wchar_size", i32 4}
+!4 = !{i32 7, !"PIC Level", i32 2}
+!5 = distinct !DICompileUnit(language: DW_LANG_C99, file: !6, producer: "compiler version", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, nameTableKind: None)
+!6 = !DIFile(filename: "source-file.c", directory: "/workspace")
+!7 = !{}
+!8 = !{!"compiler version"}
+!9 = distinct !DISubprogram(name: "main", scope: !6, file: !6, line: 4, type: !10, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !5, retainedNodes: !7)
+!10 = !DISubroutineType(types: !11)
+!11 = !{!12}
+!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!13 = !DILocation(line: 5, column: 3, scope: !9)
+!14 = !DILocation(line: 6, column: 3, scope: !9)

llvm/tools/lli/CMakeLists.txt

set(LLVM_LINK_COMPONENTS
     DebugInfoDWARF
     PerfJITEvents
     Object
     )
 endif( LLVM_USE_PERF )
 
 add_llvm_tool(lli
   lli.cpp
+  ExecutionUtils.cpp
 
   DEPENDS
   intrinsics_gen
   )
 
 export_executable_symbols(lli)

llvm/tools/lli/ExecutionUtils.h

+//===- ExecutionUtils.h - Utilities for executing code in lli ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Contains utilities for executing code in lli.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLI_EXECUTIONUTILS_H
+#define LLVM_TOOLS_LLI_EXECUTIONUTILS_H
+
+#include "llvm/ExecutionEngine/JITSymbol.h"
+#include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/Mangling.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ToolOutputFile.h"
+
+#include <memory>
+#include <utility>
+
+namespace llvm {
+
+enum class BuiltinFunctionKind {
+  DumpDebugDescriptor,
+  DumpDebugObjects,
+};
+
+// Utility class to expose symbols for special-purpose functions to the JIT.
+class LLIBuiltinFunctionGenerator : public orc::DefinitionGenerator {
+public:
+  LLIBuiltinFunctionGenerator(std::vector<BuiltinFunctionKind> Enabled,
+                              orc::MangleAndInterner &Mangle);
+
+  Error tryToGenerate(orc::LookupState &LS, orc::LookupKind K,
+                      orc::JITDylib &JD, orc::JITDylibLookupFlags JDLookupFlags,
+                      const orc::SymbolLookupSet &Symbols) override;
+
+  void appendDebugObject(const char *Addr, size_t Size) {
+    TestOut->os().write(Addr, Size);
+  }
+
+private:
+  orc::SymbolMap BuiltinFunctions;
+  std::unique_ptr<ToolOutputFile> TestOut;
+
+  template <typename T> void expose(orc::SymbolStringPtr Name, T *Handler) {
+    BuiltinFunctions[Name] = JITEvaluatedSymbol(
+        pointerToJITTargetAddress(Handler), JITSymbolFlags::Exported);
+  }
+
+  static std::unique_ptr<ToolOutputFile> createToolOutput();
+};
+
+} // end namespace llvm
+
+#endif // LLVM_TOOLS_LLI_EXECUTIONUTILS_H

llvm/tools/lli/ExecutionUtils.cpp

+//===---- ExecutionUtils.cpp - Utilities for executing functions in lli ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "ExecutionUtils.h"
+
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <cstdint>
+#include <vector>
+
+// Declarations follow the GDB JIT interface (version 1, 2009) and must match
+// those of the DYLD used for testing. See:
+//
+//   llvm/lib/ExecutionEngine/Orc/TargetProcess/JITLoaderGDB.cpp
+//   llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp
+//
+typedef enum {
+  JIT_NOACTION = 0,
+  JIT_REGISTER_FN,
+  JIT_UNREGISTER_FN
+} jit_actions_t;
+
+struct jit_code_entry {
+  struct jit_code_entry *next_entry;
+  struct jit_code_entry *prev_entry;
+  const char *symfile_addr;
+  uint64_t symfile_size;
+};
+
+struct jit_descriptor {
+  uint32_t version;
+  // This should be jit_actions_t, but we want to be specific about the
+  // bit-width.
+  uint32_t action_flag;
+  struct jit_code_entry *relevant_entry;
+  struct jit_code_entry *first_entry;
+};
+
+namespace llvm {
+
+template <typename... Ts> static void outsv(const char *Fmt, Ts &&...Vals) {
+  outs() << formatv(Fmt, Vals...);
+}
+
+static const char *actionFlagToStr(uint32_t ActionFlag) {
+  switch (ActionFlag) {
+  case JIT_NOACTION:
+    return "JIT_NOACTION";
+  case JIT_REGISTER_FN:
+    return "JIT_REGISTER_FN";
+  case JIT_UNREGISTER_FN:
+    return "JIT_UNREGISTER_FN";
+  }
+  return "<invalid action_flag>";
+}
+
+// Sample output:
+//
+//   Reading __jit_debug_descriptor at 0x0000000000404048
+//
+//   Version: 0
+//   Action: JIT_REGISTER_FN
+//
+//         Entry               Symbol File         Size  Previous Entry
+//   [ 0]  0x0000000000451290  0x0000000000002000   200  0x0000000000000000
+//   [ 1]  0x0000000000451260  0x0000000000001000   100  0x0000000000451290
+//   ...
+//
+static void dumpDebugDescriptor(void *Addr) {
+  outsv("Reading __jit_debug_descriptor at {0}\n\n", Addr);
+
+  jit_descriptor *Descriptor = reinterpret_cast<jit_descriptor *>(Addr);
+  outsv("Version: {0}\n", Descriptor->version);
+  outsv("Action: {0}\n\n", actionFlagToStr(Descriptor->action_flag));
+  outsv("{0,11}  {1,24}  {2,15}  {3,14}\n", "Entry", "Symbol File", "Size",
+        "Previous Entry");
+
+  unsigned Idx = 0;
+  for (auto *Entry = Descriptor->first_entry; Entry; Entry = Entry->next_entry)
+    outsv("[{0,2}]  {1:X16}  {2:X16}  {3,8:D}  {4}\n", Idx++, Entry,
+          reinterpret_cast<const void *>(Entry->symfile_addr),
+          Entry->symfile_size, Entry->prev_entry);
+}
+
+static LLIBuiltinFunctionGenerator *Generator = nullptr;
+
+static void dumpDebugObjects(void *Addr) {
+  jit_descriptor *Descriptor = reinterpret_cast<jit_descriptor *>(Addr);
+  for (auto *Entry = Descriptor->first_entry; Entry; Entry = Entry->next_entry)
+    Generator->appendDebugObject(Entry->symfile_addr, Entry->symfile_size);
+}
+
+LLIBuiltinFunctionGenerator::LLIBuiltinFunctionGenerator(
+    std::vector<BuiltinFunctionKind> Enabled, orc::MangleAndInterner &Mangle)
+    : TestOut(nullptr) {
+  Generator = this;
+  for (BuiltinFunctionKind F : Enabled) {
+    switch (F) {
+    case BuiltinFunctionKind::DumpDebugDescriptor:
+      expose(Mangle("__dump_jit_debug_descriptor"), &dumpDebugDescriptor);
+      break;
+    case BuiltinFunctionKind::DumpDebugObjects:
+      expose(Mangle("__dump_jit_debug_objects"), &dumpDebugObjects);
+      TestOut = createToolOutput();
+      break;
+    }
+  }
+}
+
+Error LLIBuiltinFunctionGenerator::tryToGenerate(
+    orc::LookupState &LS, orc::LookupKind K, orc::JITDylib &JD,
+    orc::JITDylibLookupFlags JDLookupFlags,
+    const orc::SymbolLookupSet &Symbols) {
+  orc::SymbolMap NewSymbols;
+  for (const auto &NameFlags : Symbols) {
+    auto It = BuiltinFunctions.find(NameFlags.first);
+    if (It != BuiltinFunctions.end())
+      NewSymbols.insert(*It);
+  }
+
+  if (NewSymbols.empty())
+    return Error::success();
+
+  return JD.define(absoluteSymbols(std::move(NewSymbols)));
+}
+
+// static
+std::unique_ptr<ToolOutputFile>
+LLIBuiltinFunctionGenerator::createToolOutput() {
+  std::error_code EC;
+  auto TestOut = std::make_unique<ToolOutputFile>("-", EC, sys::fs::OF_None);
+  if (EC) {
+    errs() << "Error creating tool output file: " << EC.message() << '\n';
+    exit(1);
+  }
+  return TestOut;
+}
+
+} // namespace llvm

llvm/tools/lli/lli.cpp

 //===- lli.cpp - LLVM Interpreter / Dynamic compiler ----------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This utility provides a simple wrapper around the LLVM Execution Engines,
 // which allow the direct execution of LLVM programs through a Just-In-Time
 // compiler, or through an interpreter if no JIT is available for this platform.
 //
 //===----------------------------------------------------------------------===//
 
+#include "ExecutionUtils.h"
 #include "RemoteJITUtils.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/CodeGen/CommandFlags.h"
 #include "llvm/CodeGen/LinkAllCodegenComponents.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/ExecutionEngine/GenericValue.h"
cl::opt<DumpKind> OrcDumpKind(
                  clEnumValN(DumpKind::DumpModsToStdOut, "mods-to-stdout",
                             "Dump modules to stdout."),
                  clEnumValN(DumpKind::DumpModsToDisk, "mods-to-disk",
                             "Dump modules to the current "
                             "working directory. (WARNING: "
                             "will overwrite existing files).")),
       cl::Hidden);
 
+  cl::list<BuiltinFunctionKind> GenerateBuiltinFunctions(
+      "generate",
+      cl::desc("Provide built-in functions for access by JITed code "
+               "(jit-kind=orc-lazy only)"),
+      cl::values(clEnumValN(BuiltinFunctionKind::DumpDebugDescriptor,
+                            "__dump_jit_debug_descriptor",
+                            "Dump __jit_debug_descriptor contents to stdout"),
+                 clEnumValN(BuiltinFunctionKind::DumpDebugObjects,
+                            "__dump_jit_debug_objects",
+                            "Dump __jit_debug_descriptor in-memory debug "
+                            "objects as tool output")),
+      cl::Hidden);
+
   ExitOnError ExitOnErr;
 }
 
 //===----------------------------------------------------------------------===//
 // Object cache
 //
 // This object cache implementation writes cached objects to disk to the
 // directory specified by CacheDir, using a filename provided in the module
int runOrcLazyJIT(const char *ProgName) {
   if (!NoProcessSymbols)
     J->getMainJITDylib().addGenerator(
         ExitOnErr(orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
             J->getDataLayout().getGlobalPrefix(),
             [MainName = Mangle("main")](const orc::SymbolStringPtr &Name) {
               return Name != MainName;
             })));
 
+  if (GenerateBuiltinFunctions.size() > 0)
+    J->getMainJITDylib().addGenerator(
+        std::make_unique<LLIBuiltinFunctionGenerator>(GenerateBuiltinFunctions,
+                                                      Mangle));
+
   // Add the main module.
   ExitOnErr(J->addLazyIRModule(std::move(MainModule)));
 
   // Create JITDylibs and add any extra modules.
   {
     // Create JITDylibs, keep a map from argument index to dylib. We will use
     // -extra-module argument indexes to determine what dylib to use for each
     // -extra-module.