Add DWARFContext generation from LinkGraphs, use in perf support

[pchintalapudi]

Summary

This patch adds line numbers to perf jitdump records emitted by the PerfSupportPlugin, by parsing and using a DWARFContext from preserved debug sections.

Depends on D146169

[lhames]

Thanks very much @pchintalapudi!

Is it possible to write a testcase for this?

llvm/lib/ExecutionEngine/Orc/DebugInfoSupport.cpp

1

Needs LLVM licensing header.

llvm/lib/ExecutionEngine/Orc/PerfSupportPlugin.cpp

1

Needs LLVM licensing header.

151–153

I probably missed this in the previous review, but any debug lines should be wrapped in an LLVM_DEBUG macro prior to commit.

249

This is a good question.

Right now errors can be sent to ExecutionSession::reportError and they won't automatically take down the session. If you give this error a discernible type ("BadDWARF"?) then clients could choose to ignore completely or log based on that.

[pchintalapudi]

• Address review comments

[pchintalapudi]

For testing, there are a couple of options that I think should be possible.

Right now D146169's test is only that the jitdump file exists, not on the accuracy of the content. I've been verifying the content of that file using my own custom python script, so one option is to upload that script (to where?), output to json, and check that json for accuracy (with what tools?).

Another option is to take advantage of the cross-process support and replace the default called addresses in the plugin with calls to verification functions. This would probably take the form of a new cpp unit test (don't know how to do this) and has the advantage of being able to test the state on structured data rather than reading a binary file that's sensitive to bugs in the writer.

[pchintalapudi]

Diff against parent revision

[pchintalapudi]

Propagate stacked change

[lhames]

In D146391#4205531, @pchintalapudi wrote:

For testing, there are a couple of options that I think should be possible.

Right now D146169's test is only that the jitdump file exists, not on the accuracy of the content. I've been verifying the content of that file using my own custom python script, so one option is to upload that script (to where?), output to json, and check that json for accuracy (with what tools?).

If you were going to go this way then I think the right thing would be to add your test tool to llvm/utils and then register it along with the other known tools in llvm/test/lit.cfg.py. It's probably worth workshopping that idea on discord / discourse though.

Another option is to take advantage of the cross-process support and replace the default called addresses in the plugin with calls to verification functions. This would probably take the form of a new cpp unit test (don't know how to do this) and has the advantage of being able to test the state on structured data rather than reading a binary file that's sensitive to bugs in the writer.

Yeah -- there are a few ways to approach this, but they're all a bit awkward with the tools we have available. I'll have a think about what we can do to improve this.

I don't think the testcase discussion should prevent this landing. LGTM. Thanks again Prem!

[pchintalapudi]

Rebase on D146169

llvm/include/llvm/ExecutionEngine/Orc/DebugInfoSupport.h

+//===--- DebugInfoSupport.h -- Utils for debug info support ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Utilities to preserve and parse debug info from LinkGraphs.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_ORC_DEBUGINFOSUPPORT_H
+#define LLVM_EXECUTIONENGINE_ORC_DEBUGINFOSUPPORT_H
+
+#include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
+
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+
+namespace llvm {
+
+namespace orc {
+
+Error preserveDebugSections(jitlink::LinkGraph &G);
+// The backing stringmap is also returned, for memory liftime management.
+Expected<std::pair<std::unique_ptr<DWARFContext>,
+                   StringMap<std::unique_ptr<MemoryBuffer>>>>
+createDWARFContext(jitlink::LinkGraph &G);
+
+// Thin wrapper around preserveDebugSections to be used as a standalone plugin.
+class DebugInfoPreservationPlugin : public ObjectLinkingLayer::Plugin {
+public:
+  void modifyPassConfig(MaterializationResponsibility &MR,
+                        jitlink::LinkGraph &LG,
+                        jitlink::PassConfiguration &PassConfig) override {
+    PassConfig.PrePrunePasses.push_back(preserveDebugSections);
+  }
+
+  Error notifyRemovingResources(JITDylib &JD, ResourceKey K) override {
+    // Do nothing.
+    return Error::success();
+  }
+  Error notifyFailed(MaterializationResponsibility &MR) override {
+    // Do nothing.
+    return Error::success();
+  }
+  void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
+                                   ResourceKey SrcKey) override {
+    // Do nothing.
+  }
+
+  static Expected<std::unique_ptr<DebugInfoPreservationPlugin>> Create() {
+    return std::make_unique<DebugInfoPreservationPlugin>();
+  }
+};
+
+} // namespace orc
+
+} // namespace llvm
+
+#endif
 No newline at end of file

llvm/include/llvm/ExecutionEngine/Orc/PerfSupportPlugin.h

 namespace llvm {
 namespace orc {
 
 class PerfSupportPlugin : public ObjectLinkingLayer::Plugin {
 public:
   PerfSupportPlugin(ExecutorProcessControl &EPC,
                     ExecutorAddr RegisterPerfStartAddr,
                     ExecutorAddr RegisterPerfEndAddr,
-                    ExecutorAddr RegisterPerfImplAddr, bool EmitUnwindInfo);
+                    ExecutorAddr RegisterPerfImplAddr, bool EmitDebugInfo,
+                    bool EmitUnwindInfo);
   ~PerfSupportPlugin();
 
   void modifyPassConfig(MaterializationResponsibility &MR,
                         jitlink::LinkGraph &G,
                         jitlink::PassConfiguration &Config) override;
 
   Error notifyFailed(MaterializationResponsibility &MR) override {
     return Error::success();
   }
 
   Error notifyRemovingResources(JITDylib &JD, ResourceKey K) override {
     return Error::success();
   }
 
   void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
                                    ResourceKey SrcKey) override {}
 
   static Expected<std::unique_ptr<PerfSupportPlugin>>
-  Create(ExecutorProcessControl &EPC, JITDylib &JD, bool EmitUnwindInfo);
+  Create(ExecutorProcessControl &EPC, JITDylib &JD, bool EmitDebugInfo,
+         bool EmitUnwindInfo);
 
 private:
   ExecutorProcessControl &EPC;
   ExecutorAddr RegisterPerfStartAddr;
   ExecutorAddr RegisterPerfEndAddr;
   ExecutorAddr RegisterPerfImplAddr;
   std::atomic<uint64_t> CodeIndex;
+  bool EmitDebugInfo;
   bool EmitUnwindInfo;
 };
 
 // The following are POD struct definitions from the perf jit specification
 
 enum class PerfJITRecordType {
   JIT_CODE_LOAD = 0,
   JIT_CODE_MOVE = 1, // not emitted, code isn't moved

 };
 
 } // namespace shared
 
 } // namespace orc
 } // namespace llvm
 
 #endif // LLVM_EXECUTIONENGINE_ORC_PERFSUPPORTPLUGIN_H
 No newline at end of file

llvm/lib/ExecutionEngine/Orc/CMakeLists.txt

 if( CMAKE_HOST_UNIX AND HAVE_LIBRT )
   set(rt_lib rt)
 endif()
 
 add_llvm_component_library(LLVMOrcJIT
   COFFVCRuntimeSupport.cpp
   COFFPlatform.cpp
   CompileOnDemandLayer.cpp
   CompileUtils.cpp
   Core.cpp
   DebugObjectManagerPlugin.cpp
   DebuggerSupportPlugin.cpp
+  DebugInfoSupport.cpp
   DebugUtils.cpp
   EPCDynamicLibrarySearchGenerator.cpp
   EPCDebugObjectRegistrar.cpp
   EPCEHFrameRegistrar.cpp
   EPCGenericDylibManager.cpp
   EPCGenericJITLinkMemoryManager.cpp
   EPCGenericRTDyldMemoryManager.cpp
   EPCIndirectionUtils.cpp

llvm/lib/ExecutionEngine/Orc/DebugInfoSupport.cpp

+#include "llvm/ExecutionEngine/Orc/DebugInfoSupport.h"
+
+#include "llvm/Support/SmallVectorMemoryBuffer.h"
+
+#define DEBUG_TYPE "orc"
+
+using namespace llvm;
+using namespace llvm::orc;
+using namespace llvm::jitlink;
+
+namespace {
+static DenseSet<StringRef> DWARFSectionNames = {
+#define HANDLE_DWARF_SECTION(ENUM_NAME, ELF_NAME, CMDLINE_NAME, OPTION)        \
+  StringRef(ELF_NAME),
+#include "llvm/BinaryFormat/Dwarf.def"
+#undef HANDLE_DWARF_SECTION
+};
+
+// We might be able to drop relocations to symbols that do end up
+// being pruned by the linker, but for now we just preserve all
+static void preserveDWARFSection(LinkGraph &G, Section &Sec) {
+  DenseMap<Block *, Symbol *> Preserved;
+  for (auto Sym : Sec.symbols()) {
+    if (Sym->isLive()) {
+      Preserved[&Sym->getBlock()] = Sym;
+    } else if (!Preserved.count(&Sym->getBlock())) {
+      Preserved[&Sym->getBlock()] = Sym;
+    }
+  }
+  for (auto Block : Sec.blocks()) {
+    auto &PSym = Preserved[Block];
+    if (!PSym) {
+      PSym = &G.addAnonymousSymbol(*Block, 0, 0, false, true);
+    } else if (!PSym->isLive()) {
+      PSym->setLive(true);
+    }
+  }
+}
+
+static SmallVector<char, 0> getSectionData(Section &Sec) {
+  SmallVector<char, 0> SecData;
+  SmallVector<Block *, 8> SecBlocks(Sec.blocks().begin(), Sec.blocks().end());
+  std::sort(SecBlocks.begin(), SecBlocks.end(), [](Block *LHS, Block *RHS) {
+    return LHS->getAddress() < RHS->getAddress();
+  });
+  // Convert back to what object file would have, one blob of section content
+  // Assumes all zerofill
+  // TODO handle alignment?
+  // TODO handle alignment offset?
+  for (auto *Block : SecBlocks) {
+    if (Block->isZeroFill()) {
+      SecData.resize(SecData.size() + Block->getSize(), 0);
+    } else {
+      SecData.append(Block->getContent().begin(), Block->getContent().end());
+    }
+  }
+  return SecData;
+}
+
+static void dumpDWARFContext(DWARFContext &DC) {
+  auto options = llvm::DIDumpOptions();
+  options.DumpType &= ~DIDT_UUID;
+  options.DumpType &= ~(1 << DIDT_ID_DebugFrame);
+  DC.dump(dbgs(), options);
+}
+
+} // namespace
+
+Error llvm::orc::preserveDebugSections(LinkGraph &G) {
+  if (!G.getTargetTriple().isOSBinFormatELF()) {
+    return make_error<StringError>(
+        "preserveDebugSections only supports ELF LinkGraphs!",
+        inconvertibleErrorCode());
+  }
+  for (auto &Sec : G.sections()) {
+    if (DWARFSectionNames.count(Sec.getName())) {
+      dbgs() << "Preserving DWARF section " << Sec.getName() << "\n";
+      preserveDWARFSection(G, Sec);
+    } else {
+      dbgs() << "Not preserving section " << Sec.getName() << "\n";
+    }
+  }
+  return Error::success();
+}
+
+Expected<std::pair<std::unique_ptr<DWARFContext>,
+                   StringMap<std::unique_ptr<MemoryBuffer>>>>
+llvm::orc::createDWARFContext(LinkGraph &G) {
+  if (!G.getTargetTriple().isOSBinFormatELF()) {
+    return make_error<StringError>(
+        "createDWARFContext only supports ELF LinkGraphs!",
+        inconvertibleErrorCode());
+  }
+  StringMap<std::unique_ptr<MemoryBuffer>> DWARFSectionData;
+  for (auto &Sec : G.sections()) {
+    if (DWARFSectionNames.count(Sec.getName())) {
+      auto SecData = getSectionData(Sec);
+      auto Name = Sec.getName();
+      // DWARFContext expects the section name to not start with a dot
+      if (Name.startswith(".")) {
+        Name = Name.drop_front();
+      }
+      dbgs() << "Creating DWARFContext section " << Name << " with size "
+             << SecData.size() << "\n";
+      DWARFSectionData[Name] =
+          std::make_unique<SmallVectorMemoryBuffer>(std::move(SecData));
+    }
+  }
+  auto Ctx = DWARFContext::create(DWARFSectionData, G.getPointerSize(),
+                                  G.getEndianness() == support::little);
+  dumpDWARFContext(*Ctx);
+  return std::make_pair(std::move(Ctx), std::move(DWARFSectionData));
+}

llvm/lib/ExecutionEngine/Orc/PerfSupportPlugin.cpp

 #include "llvm/ExecutionEngine/Orc/PerfSupportPlugin.h"
 
 #include "llvm/ExecutionEngine/Orc/TargetProcess/JITLoaderPerf.h"
 
-#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/ExecutionEngine/Orc/DebugInfoSupport.h"
+
 #include "llvm/ExecutionEngine/JITLink/x86_64.h"
 #include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
 #include "llvm/ExecutionEngine/Orc/Shared/SimplePackedSerialization.h"
 
 #ifdef __linux__
 #include <time.h>     // clock_gettime(), time(), localtime_r() */
 #endif
 
Record.Prefix.TotalSize =
        + 4 * sizeof(uint64_t) // vma, code_addr, code_size, code_index
        + Name.size() + 1      // symbol name
        + Record.CodeSize      // code
       );
   return Record;
 }
 
 static std::optional<PerfJITDebugInfoRecord>
-getDebugInfoRecord(const Symbol &Sym, DWARFContext *DC) {
-  // dbgs() << "Getting debug info for symbol " << Sym.getName() << "\n";
-  if (!DC) {
-    // No debug info available
-    dbgs() << "No debug info available\n";
-    return std::nullopt;
-  }
+getDebugInfoRecord(const Symbol &Sym, DWARFContext &DC) {
   auto &Section = Sym.getBlock().getSection();
   auto Addr = Sym.getAddress();
   auto Size = Sym.getSize();
   auto SAddr = object::SectionedAddress{Addr.getValue(), Section.getOrdinal()};
   dbgs() << "Getting debug info for symbol " << Sym.getName() << " at address "
-         << Addr.getValue() << " with size " << Size << "\n";
+         << (void *)Addr.getValue() << " with size " << Size << "\n";
   dbgs() << "Section ordinal: " << Section.getOrdinal() << "\n";
-  auto LInfo = DC->getLineInfoForAddressRange(
+  auto LInfo = DC.getLineInfoForAddressRange(
       SAddr, Size, DILineInfoSpecifier::FileLineInfoKind::AbsoluteFilePath);
   if (LInfo.empty()) {
     // No line info available
     dbgs() << "No line info available\n";
     return std::nullopt;
   }
   PerfJITDebugInfoRecord Record;
   Record.Prefix.Id = PerfJITRecordType::JIT_CODE_DEBUG_INFO;
static PerfJITCodeUnwindingInfoRecord getUnwindingRecord(LinkGraph &G) {
   if (!G.getTargetTriple().isOSBinFormatELF()) {
     dbgs() << "Not an ELF file, will not emit unwinding info\n";
     return Record;
   }
   auto SR = SectionRange(*Eh_frame);
   auto EHFrameSize = SR.getSize();
   auto Eh_frame_hdr = G.findSectionByName(".eh_frame_hdr");
   if (!Eh_frame_hdr) {
-    if (G.getTargetTriple().getArch() != Triple::x86_64) {
+    if (G.getTargetTriple().getArch() == Triple::x86_64) {
       Record.EHFrameHdr =
           createX64EHFrameHeader(*Eh_frame, G.getEndianness(), true);
     } else {
       dbgs() << "No .eh_frame_hdr section found\n";
       return Record;
     }
     Record.EHFrameHdrAddr = 0;
     Record.EHFrameHdrSize = Record.EHFrameHdr.size();
static PerfJITCodeUnwindingInfoRecord getUnwindingRecord(LinkGraph &G) {
   dbgs() << "Created unwind record\n";
   dbgs() << "Total size: " << Record.Prefix.TotalSize << "\n";
   dbgs() << "Unwind size: " << Record.UnwindDataSize << "\n";
   dbgs() << "EHFrame size: " << EHFrameSize << "\n";
   dbgs() << "EHFrameHdr size: " << Record.EHFrameHdrSize << "\n";
   return Record;
 }
 
-static PerfJITRecordBatch getRecords(LinkGraph &G, DWARFContext *DC,
+static PerfJITRecordBatch getRecords(LinkGraph &G,
                                      std::atomic<uint64_t> &CodeIndex,
-                                     bool EmitUnwindInfo) {
+                                     bool EmitDebugInfo, bool EmitUnwindInfo) {
+  std::unique_ptr<DWARFContext> DC;
+  StringMap<std::unique_ptr<MemoryBuffer>> DCBacking;
+  if (EmitDebugInfo) {
+    auto EDC = createDWARFContext(G);
+    if (!EDC) {
+      // TODO fatal errors when debug info fails? Some other way of reporting?
+      consumeError(EDC.takeError());
+      EmitDebugInfo = false;
+    } else {
+      DC = std::move(EDC->first);
+      DCBacking = std::move(EDC->second);
+    }
+  }
   PerfJITRecordBatch Batch;
   for (auto Sym : G.defined_symbols()) {
     if (!Sym->hasName())
       continue;
-    auto DebugInfo = getDebugInfoRecord(*Sym, DC);
+    if (EmitDebugInfo) {
+      auto DebugInfo = getDebugInfoRecord(*Sym, *DC);
       if (DebugInfo)
         Batch.DebugInfoRecords.push_back(std::move(*DebugInfo));
+    }
     Batch.CodeLoadRecords.push_back(getCodeLoadRecord(*Sym, CodeIndex));
   }
   if (EmitUnwindInfo) {
     Batch.UnwindingRecord = getUnwindingRecord(G);
   } else {
     Batch.UnwindingRecord.Prefix.TotalSize = 0;
   }
   return Batch;
 }
 } // namespace
 
 PerfSupportPlugin::PerfSupportPlugin(ExecutorProcessControl &EPC,
                                      ExecutorAddr RegisterPerfStartAddr,
                                      ExecutorAddr RegisterPerfEndAddr,
                                      ExecutorAddr RegisterPerfImplAddr,
-                                     bool EmitUnwindInfo)
+                                     bool EmitDebugInfo, bool EmitUnwindInfo)
     : EPC(EPC), RegisterPerfStartAddr(RegisterPerfStartAddr),
       RegisterPerfEndAddr(RegisterPerfEndAddr),
       RegisterPerfImplAddr(RegisterPerfImplAddr), CodeIndex(0),
-      EmitUnwindInfo(EmitUnwindInfo) {
+      EmitDebugInfo(EmitDebugInfo), EmitUnwindInfo(EmitUnwindInfo) {
   cantFail(EPC.callSPSWrapper<void()>(RegisterPerfStartAddr));
   dbgs() << "Created perf support plugin\n";
 }
 PerfSupportPlugin::~PerfSupportPlugin() {
   dbgs() << "Destroying perf support plugin\n";
   cantFail(EPC.callSPSWrapper<void()>(RegisterPerfEndAddr));
 }
 
 void PerfSupportPlugin::modifyPassConfig(MaterializationResponsibility &MR,
                                          LinkGraph &G,
                                          PassConfiguration &Config) {
   dbgs() << "Modifying perf support plugin pass config\n";
   Config.PostFixupPasses.push_back([this, &MR](LinkGraph &G) {
     dbgs() << "Running perf support plugin post-fixup pass\n";
     dbgs() << serialize(G) << "\n";
-    // TODO get an actual DWARFContext for line info
-    DWARFContext *DWC = nullptr;
-    auto Batch = getRecords(G, DWC, CodeIndex, EmitUnwindInfo);
+    auto Batch = getRecords(G, CodeIndex, EmitDebugInfo, EmitUnwindInfo);
     G.allocActions().push_back(
         {cantFail(shared::WrapperFunctionCall::Create<
                   shared::SPSArgList<shared::SPSPerfJITRecordBatch>>(
              RegisterPerfImplAddr, Batch)),
          {}});
     dbgs() << "Registered alloc action\n";
     dbgs() << "Impl address: " << RegisterPerfImplAddr << "\n";
     return Error::success();
   });
 }
 
 Expected<std::unique_ptr<PerfSupportPlugin>>
 PerfSupportPlugin::Create(ExecutorProcessControl &EPC, JITDylib &JD,
-                          bool EmitUnwindInfo) {
+                          bool EmitDebugInfo, bool EmitUnwindInfo) {
   auto &ES = EPC.getExecutionSession();
   auto StartName = ES.intern(RegisterPerfStartSymbolName);
   auto EndName = ES.intern(RegisterPerfEndSymbolName);
   auto ImplName = ES.intern(RegisterPerfImplSymbolName);
   SymbolLookupSet SLS({StartName, EndName, ImplName});
   auto Res = ES.lookup(makeJITDylibSearchOrder({&JD}), std::move(SLS));
   if (!Res)
     return Res.takeError();
   auto StartAddr = ExecutorAddr(Res->find(StartName)->second.getAddress());
   auto EndAddr = ExecutorAddr(Res->find(EndName)->second.getAddress());
   auto ImplAddr = ExecutorAddr(Res->find(ImplName)->second.getAddress());
   return std::make_unique<PerfSupportPlugin>(EPC, StartAddr, EndAddr, ImplAddr,
-                                             EmitUnwindInfo);
+                                             EmitDebugInfo, EmitUnwindInfo);
 }

llvm/tools/llvm-jitlink/llvm-jitlink.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm-jitlink.h"
 
 #include "llvm/BinaryFormat/Magic.h"
 #include "llvm/ExecutionEngine/Orc/COFFPlatform.h"
 #include "llvm/ExecutionEngine/Orc/COFFVCRuntimeSupport.h"
+#include "llvm/ExecutionEngine/Orc/DebugInfoSupport.h"
 #include "llvm/ExecutionEngine/Orc/DebugObjectManagerPlugin.h"
 #include "llvm/ExecutionEngine/Orc/DebuggerSupportPlugin.h"
 #include "llvm/ExecutionEngine/Orc/ELFNixPlatform.h"
 #include "llvm/ExecutionEngine/Orc/EPCDebugObjectRegistrar.h"
 #include "llvm/ExecutionEngine/Orc/EPCDynamicLibrarySearchGenerator.h"
 #include "llvm/ExecutionEngine/Orc/EPCEHFrameRegistrar.h"
 #include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
 #include "llvm/ExecutionEngine/Orc/IndirectionUtils.h"
Session::Session(std::unique_ptr<ExecutorProcessControl> EPC, Error &Err)
   ExitOnErr(loadDylibs(*this));
 
   auto &TT = ES.getTargetTriple();
 
   if (DebuggerSupport && TT.isOSBinFormatMachO())
     ObjLayer.addPlugin(ExitOnErr(
         GDBJITDebugInfoRegistrationPlugin::Create(this->ES, *MainJD, TT)));
 
-  if (PerfSupport && TT.isOSBinFormatELF())
+  if (PerfSupport && TT.isOSBinFormatELF()) {
+    ObjLayer.addPlugin(ExitOnErr(DebugInfoPreservationPlugin::Create()));
     ObjLayer.addPlugin(ExitOnErr(PerfSupportPlugin::Create(
-        this->ES.getExecutorProcessControl(), *MainJD, true)));
+        this->ES.getExecutorProcessControl(), *MainJD, true, true)));
+  }
 
   // Set up the platform.
   if (TT.isOSBinFormatMachO() && !OrcRuntime.empty()) {
     if (auto P =
             MachOPlatform::Create(ES, ObjLayer, *MainJD, OrcRuntime.c_str()))
       ES.setPlatform(std::move(*P));
     else {
       Err = P.takeError();