[ORC][ORC_RT] Handle ELF .init_array with non-default priority

[housel]

Summary

ELF-based platforms currently support defining multiple static initializer table sections with differing priorities, for example .init_array.0 or .init_array.100; the default .init_array corresponds to a priority of 65535. When building a shared library or executable, the system linker normally sorts these sections and combines them into a single .init_array section. This change adds the capability to recognize ELF static initializers with priorities other than the default, and to properly sort them by priority, to Orc and the Orc runtime.

[lhames]

Very nice. LGTM. Thanks @housel!

[MaskRay]

compiler-rt/lib/orc/elfnix_platform.cpp

379

65536 is better.

See lld/ELF/OutputSections.cpp::576

391

stable_sort

[MaskRay]

llvm/lib/ExecutionEngine/Orc/ELFNixPlatform.cpp

814

https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

[lhames]

compiler-rt/lib/orc/elfnix_platform.cpp

391

Does ELF make any guarantees about order-of-initialization beyond priority ordering?

We don't currently number MaterializationUnits, so initialization order is already at the mercy of the dependence graph (likely opaque to clients), and the scheduler (if concurrent compilation is enabled). I'm not opposed to stable_sort (I like that it should make run-to-run behavior of llvm-jitlink more stable), but it's probably worth a comment that order of initialization is still not guaranteed in general.

[MaskRay]

compiler-rt/lib/orc/elfnix_platform.cpp

391

For determinism (using different standard library implementations should give the same result), stable_sort should be used. Relying on the order is user code issue, but that's unrelated to the general determinism guarantee provided by the toolchain (https://maskray.me/blog/2021-11-07-init-ctors-init-array)

[housel]

compiler-rt/lib/orc/elfnix_platform.cpp

379

I saw that, but it wasn't clear what advantage that would have (as opposed to reversing the mapping in llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp:1060)

391

At the point where this occurs, ELFNixPlatform has already collected all initializers of each priority level into a single entry, so this sort by priority level is already going to be completely deterministic. I can change this to stable_sort but it will never have any effect on the result.

[lhames]

compiler-rt/lib/orc/elfnix_platform.cpp

391

I'd prefer to stick with std::sort. stable_sort seems like it's promising something it wouldn't actually deliver here.

[housel]

Updated brace usage

compiler-rt/lib/orc/elfnix_platform.h

struct ELFNixJITDylibInitializers {
 
   ELFNixJITDylibInitializers() = default;
   ELFNixJITDylibInitializers(std::string Name, ExecutorAddr DSOHandleAddress)
       : Name(std::move(Name)), DSOHandleAddress(std::move(DSOHandleAddress)) {}
 
   std::string Name;
   ExecutorAddr DSOHandleAddress;
 
-  std::unordered_map<std::string, SectionList> InitSections;
+  std::vector<std::pair<std::string, SectionList>> InitSections;
 };
 
 class ELFNixJITDylibDeinitializers {};
 
 using ELFNixJITDylibInitializerSequence =
     std::vector<ELFNixJITDylibInitializers>;
 
 using ELFNixJITDylibDeinitializerSequence =

compiler-rt/lib/orc/elfnix_platform.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "elfnix_platform.h"
 #include "common.h"
 #include "error.h"
 #include "wrapper_function_utils.h"
 
+#include <algorithm>
 #include <map>
 #include <mutex>
 #include <sstream>
 #include <unordered_map>
 #include <vector>
 
 using namespace __orc_rt;
 using namespace __orc_rt::elfnix;
private:
 
   Expected<ELFNixJITDylibInitializerSequence>
   getJITDylibInitializersByName(string_view Path);
   Expected<void *> dlopenInitialize(string_view Path, int Mode);
   Error initializeJITDylib(ELFNixJITDylibInitializers &MOJDIs);
 
   static ELFNixPlatformRuntimeState *MOPS;
 
-  using InitSectionHandler =
-      Error (*)(const std::vector<ExecutorAddrRange> &Sections,
-                const ELFNixJITDylibInitializers &MOJDIs);
-  const std::vector<std::pair<const char *, InitSectionHandler>> InitSections =
-      {{".init_array", runInitArray}};
-
   void *PlatformJDDSOHandle;
 
   // FIXME: Move to thread-state.
   std::string DLFcnError;
 
   std::recursive_mutex JDStatesMutex;
   std::unordered_map<void *, PerJITDylibState> JDStates;
   std::unordered_map<std::string, void *> JDNameToHeader;
Expected<void *> ELFNixPlatformRuntimeState::dlopenInitialize(string_view Path,
 
   // Return the header for the last item in the list.
   auto *JDS = getJITDylibStateByHeaderAddr(
       InitSeq->back().DSOHandleAddress.toPtr<void *>());
   assert(JDS && "Missing state entry for JD");
   return JDS->Header;
 }
 
+long getPriority(const std::string &name) {
+  auto pos = name.find_last_not_of("0123456789");
+  if (pos == name.size() - 1)
+    return 65535;
+  else
+    return std::strtol(name.c_str() + pos + 1, nullptr, 10);
+}
+
 Error ELFNixPlatformRuntimeState::initializeJITDylib(
     ELFNixJITDylibInitializers &MOJDIs) {
 
   auto &JDS = getOrCreateJITDylibState(MOJDIs);
   ++JDS.RefCount;
 
-  for (auto &KV : InitSections) {
-    const auto &Name = KV.first;
-    const auto &Handler = KV.second;
-    auto I = MOJDIs.InitSections.find(Name);
-    if (I != MOJDIs.InitSections.end()) {
-      if (auto Err = Handler(I->second, MOJDIs))
-        return Err;
-    }
-  }
+  using SectionList = std::vector<ExecutorAddrRange>;
+  std::sort(MOJDIs.InitSections.begin(), MOJDIs.InitSections.end(),
+            [](const std::pair<std::string, SectionList> &LHS,
+               const std::pair<std::string, SectionList> &RHS) -> bool {
+              return getPriority(LHS.first) < getPriority(RHS.first);
+            });
+  for (auto &Entry : MOJDIs.InitSections)
+    if (auto Err = runInitArray(Entry.second, MOJDIs))
 
   return Error::success();
 }
 class ELFNixPlatformRuntimeTLVManager {
 public:
   void *getInstance(const char *ThreadData);
 
 private:

compiler-rt/test/orc/TestCases/FreeBSD/x86-64/priority-static-initializer.S

+// Test that ELF static initializers with different constructor priorities work
+// and are executed in the proper order.
+//
+// RUN: %clang -c -o %t %s
+// RUN: %llvm_jitlink %t | FileCheck %s
+
+// CHECK: constructor 100
+// CHECK-NEXT: constructor 200
+// CHECK-NEXT: constructor 65535
+// CHECK-NEXT: main
+// CHECK-NEXT: destructor
+
+  .text
+
+  .globl  destructor
+  .p2align  4, 0x90
+  .type destructor,@function
+destructor:
+.Ldestructor$local:
+
+  leaq  .L.str.d(%rip), %rdi
+  jmp puts@PLT
+
+  .globl  main
+  .p2align  4, 0x90
+  .type main,@function
+main:
+.Lmain$local:
+
+  pushq %rax
+  leaq  .L.str(%rip), %rdi
+  callq puts@PLT
+  xorl  %eax, %eax
+  popq  %rcx
+  retq
+
+  .p2align  4, 0x90
+  .type constructor.65535,@function
+constructor.65535:
+
+  pushq %rax
+  leaq  .L.str.65535(%rip), %rdi
+  callq puts@PLT
+  leaq  .Ldestructor$local(%rip), %rdi
+  leaq  __dso_handle(%rip), %rdx
+  xorl  %esi, %esi
+  popq  %rax
+  jmp __cxa_atexit@PLT
+
+  .p2align  4, 0x90
+  .type constructor.200,@function
+constructor.200:
+
+  leaq  .L.str.200(%rip), %rdi
+  jmp puts@PLT
+
+  .p2align  4, 0x90
+  .type constructor.100,@function
+constructor.100:
+
+  leaq  .L.str.100(%rip), %rdi
+  jmp puts@PLT
+
+  .hidden __dso_handle
+  .type .L.str,@object
+  .section  .rodata.str1.1,"aMS",@progbits,1
+.L.str:
+  .asciz  "main"
+  .size .L.str, 5
+
+  .type .L.str.100,@object
+.L.str.100:
+  .asciz  "constructor 100"
+  .size .L.str.100, 16
+
+  .type .L.str.200,@object
+.L.str.200:
+  .asciz  "constructor 200"
+  .size .L.str.200, 16
+
+  .type .L.str.65535,@object
+.L.str.65535:
+  .asciz  "constructor 65535"
+  .size .L.str.65535, 18
+
+  .type .L.str.d,@object
+.L.str.d:
+  .asciz  "destructor"
+  .size .L.str.d, 11
+
+  .section  .init_array.100,"aw",@init_array
+  .p2align  3
+  .quad constructor.100
+  .section  .init_array.200,"aw",@init_array
+  .p2align  3
+  .quad constructor.200
+  .section  .init_array,"aw",@init_array
+  .p2align  3
+  .quad constructor.65535

compiler-rt/test/orc/TestCases/Linux/x86-64/priority-static-initializer.S

+// Test that ELF static initializers with different constructor priorities work
+// and are executed in the proper order.
+//
+// RUN: %clang -c -o %t %s
+// RUN: %llvm_jitlink %t | FileCheck %s
+
+// CHECK: constructor 100
+// CHECK-NEXT: constructor 200
+// CHECK-NEXT: constructor 65535
+// CHECK-NEXT: main
+// CHECK-NEXT: destructor
+
+  .text
+
+  .globl  destructor
+  .p2align  4, 0x90
+  .type destructor,@function
+destructor:
+.Ldestructor$local:
+
+  leaq  .L.str.d(%rip), %rdi
+  jmp puts@PLT
+
+  .globl  main
+  .p2align  4, 0x90
+  .type main,@function
+main:
+.Lmain$local:
+
+  pushq %rax
+  leaq  .L.str(%rip), %rdi
+  callq puts@PLT
+  xorl  %eax, %eax
+  popq  %rcx
+  retq
+
+  .p2align  4, 0x90
+  .type constructor.65535,@function
+constructor.65535:
+
+  pushq %rax
+  leaq  .L.str.65535(%rip), %rdi
+  callq puts@PLT
+  leaq  .Ldestructor$local(%rip), %rdi
+  leaq  __dso_handle(%rip), %rdx
+  xorl  %esi, %esi
+  popq  %rax
+  jmp __cxa_atexit@PLT
+
+  .p2align  4, 0x90
+  .type constructor.200,@function
+constructor.200:
+
+  leaq  .L.str.200(%rip), %rdi
+  jmp puts@PLT
+
+  .p2align  4, 0x90
+  .type constructor.100,@function
+constructor.100:
+
+  leaq  .L.str.100(%rip), %rdi
+  jmp puts@PLT
+
+  .hidden __dso_handle
+  .type .L.str,@object
+  .section  .rodata.str1.1,"aMS",@progbits,1
+.L.str:
+  .asciz  "main"
+  .size .L.str, 5
+
+  .type .L.str.100,@object
+.L.str.100:
+  .asciz  "constructor 100"
+  .size .L.str.100, 16
+
+  .type .L.str.200,@object
+.L.str.200:
+  .asciz  "constructor 200"
+  .size .L.str.200, 16
+
+  .type .L.str.65535,@object
+.L.str.65535:
+  .asciz  "constructor 65535"
+  .size .L.str.65535, 18
+
+  .type .L.str.d,@object
+.L.str.d:
+  .asciz  "destructor"
+  .size .L.str.d, 11
+
+  .section  .init_array.100,"aw",@init_array
+  .p2align  3
+  .quad constructor.100
+  .section  .init_array.200,"aw",@init_array
+  .p2align  3
+  .quad constructor.200
+  .section  .init_array,"aw",@init_array
+  .p2align  3
+  .quad constructor.65535

llvm/lib/ExecutionEngine/Orc/ELFNixPlatform.cpp

 };
 
 StringRef EHFrameSectionName = ".eh_frame";
 StringRef InitArrayFuncSectionName = ".init_array";
 
 StringRef ThreadBSSSectionName = ".tbss";
 StringRef ThreadDataSectionName = ".tdata";
 
-StringRef InitSectionNames[] = {InitArrayFuncSectionName};
-
 } // end anonymous namespace
 
 namespace llvm {
 namespace orc {
 
 Expected<std::unique_ptr<ELFNixPlatform>>
 ELFNixPlatform::Create(ExecutionSession &ES,
                        ObjectLinkingLayer &ObjLinkingLayer,
static const std::pair<const char *, const char *>
           {"__orc_rt_jit_dlsym", "__orc_rt_elfnix_jit_dlsym"},
           {"__orc_rt_log_error", "__orc_rt_log_error_to_stderr"}};
 
   return ArrayRef<std::pair<const char *, const char *>>(
       StandardRuntimeUtilityAliases);
 }
 
 bool ELFNixPlatform::isInitializerSection(StringRef SecName) {
-  for (auto &Name : InitSectionNames) {
-    if (Name.equals(SecName))
-      return true;
-  }
+  if (SecName.consume_front(InitArrayFuncSectionName) &&
+      (SecName.empty() || SecName[0] == '.'))
   return false;
 }
 
 bool ELFNixPlatform::supportedTarget(const Triple &TT) {
   switch (TT.getArch()) {
   case Triple::x86_64:
   case Triple::aarch64:
     return true;
Config.PostFixupPasses.push_back([this](jitlink::LinkGraph &G) -> Error {
     return Error::success();
   });
 }
 
 Error ELFNixPlatform::ELFNixPlatformPlugin::preserveInitSections(
     jitlink::LinkGraph &G, MaterializationResponsibility &MR) {
 
   JITLinkSymbolSet InitSectionSymbols;
-  for (auto &InitSectionName : InitSectionNames) {
+  for (auto &InitSection : G.sections()) {
     // Skip non-init sections.
-    auto *InitSection = G.findSectionByName(InitSectionName);
-    if (!InitSection)
+    if (!isInitializerSection(InitSection.getName()))
       continue;
 
     // Make a pass over live symbols in the section: those blocks are already
     // preserved.
     DenseSet<jitlink::Block *> AlreadyLiveBlocks;
-    for (auto &Sym : InitSection->symbols()) {
+    for (auto &Sym : InitSection.symbols()) {
       auto &B = Sym->getBlock();
       if (Sym->isLive() && Sym->getOffset() == 0 &&
           Sym->getSize() == B.getSize() && !AlreadyLiveBlocks.count(&B)) {
         InitSectionSymbols.insert(Sym);
         AlreadyLiveBlocks.insert(&B);
       }
     }
 
     // Add anonymous symbols to preserve any not-already-preserved blocks.
-    for (auto *B : InitSection->blocks())
+    for (auto *B : InitSection.blocks())
       if (!AlreadyLiveBlocks.count(B))
         InitSectionSymbols.insert(
             &G.addAnonymousSymbol(*B, 0, B->getSize(), false, true));
   }
 
   if (!InitSectionSymbols.empty()) {
     std::lock_guard<std::mutex> Lock(PluginMutex);
     InitSymbolDeps[&MR] = std::move(InitSectionSymbols);
   }
 
   return Error::success();
 }
 
 Error ELFNixPlatform::ELFNixPlatformPlugin::registerInitSections(
     jitlink::LinkGraph &G, JITDylib &JD) {
 
   SmallVector<jitlink::Section *> InitSections;
 
   LLVM_DEBUG({ dbgs() << "ELFNixPlatform::registerInitSections\n"; });
 
-  for (auto InitSectionName : InitSectionNames) {
-    if (auto *Sec = G.findSectionByName(InitSectionName)) {
-      InitSections.push_back(Sec);
+  for (auto &Sec : G.sections()) {
+    if (isInitializerSection(Sec.getName())) {
+      InitSections.push_back(&Sec);
     }
   }
 
   // Dump the scraped inits.
   LLVM_DEBUG({
     dbgs() << "ELFNixPlatform: Scraped " << G.getName() << " init sections:\n";
     for (auto *Sec : InitSections) {
       jitlink::SectionRange R(*Sec);

llvm/lib/ExecutionEngine/Orc/ObjectLinkingLayer.cpp

for (auto &Sec : G.sections())
           Sec.getName() == "__TEXT,__swift5_proto" ||
           Sec.getName() == "__TEXT,__swift5_types" ||
           Sec.getName() == "__DATA,__mod_init_func")
         return true;
     return false;
   }
 
   static bool hasELFInitSection(LinkGraph &G) {
-    for (auto &Sec : G.sections())
-      if (Sec.getName() == ".init_array")
+    for (auto &Sec : G.sections()) {
+      auto SecName = Sec.getName();
+      if (SecName.consume_front(".init_array") &&
+          (SecName.empty() || SecName[0] == '.'))
         return true;
+    }
     return false;
   }
 
   static SymbolStringPtr makeInitSymbol(ExecutionSession &ES, LinkGraph &G) {
     std::string InitSymString;
     raw_string_ostream(InitSymString)
         << "$." << G.getName() << ".__inits" << Counter++;
     return ES.intern(InitSymString);