[ORC] Fix weak hidden symbols failure on PPC with runtimedyld

[sunho]

Summary

Fix "JIT session error: Symbols not found: [ DW.ref.__gxx_personality_v0 ] error" which happens when trying to use exceptions on ppc linux. To do this, it expands AutoClaimSymbols option in RTDyldObjectLinkingLayer to also claim weak symbols before they are tried to be resovled. In ppc linux, DW.ref symbols is emitted as weak hidden symbols in the later stage of MC pipeline. This means when using IRLayer (i.e. LLJIT), IRLayer will not claim responsibility for such symbols and RuntimeDyld will skip defining this symbol even though it couldn't resolve corresponding external symbol.

[sunho]

Expand existing option instead.

[v.g.vassilev]

This LGTM but I'd wait for @lhames.

[sgraenitz]

Thanks for the patch! I think this is the right way forward.

llvm/lib/ExecutionEngine/Orc/LLJIT.cpp

752 ↗

(On Diff #442728)

What about further conditions like isOSLinux() or isOSBinFormatELF()? The issue seems quite specific and AutoClaimResponsibility should be used with care. Intuitively, it seems we should tightening it to the exact platform as much as we can. It would also reduce the risk of breaking legacy use-cases downstream (which is the majority of RuntimeDyld users I think).

Are there good reasons why we wouldn't?

llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp

154

SymbolStringPool::intern() involves a mutex lock and a string-map traversal. Might be worth doing it only once where ever we can.

[sunho]

Address comments.

[sunho]

llvm/lib/ExecutionEngine/Orc/LLJIT.cpp

752 ↗

(On Diff #442728)

thanks for review! Yes, there was no paticular reason for not restricting the target.

[sgraenitz]

Sorry for the delay, LGTM!

llvm/include/llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h

public:
   /// symbols may not be detected until late, and detection may depend on
   /// the flow of control through JIT'd code. Use with care.
   RTDyldObjectLinkingLayer &
   setAutoClaimResponsibilityForObjectSymbols(bool AutoClaimObjectSymbols) {
     this->AutoClaimObjectSymbols = AutoClaimObjectSymbols;
     return *this;
   }
 
+  /// If set, this RTDyldObjectLinkingLayer instance will claim responsibility
+  /// for any weak hidden symbols provided by a given object file that were not
+  /// already in the MaterializationResponsibility instance. Setting this flag
+  /// allows to expose certain weak hidden symbols inserted later in the MC
+  /// pipeline. A notalbe example is DW.ref. symbols which are emitted as weak
+  /// hidden in ppc linux enviornment. (i.e. DW.ref.__gxx_personality_v0)
+  RTDyldObjectLinkingLayer &setAutoClaimResponsibilityForWeakHiddenSymbols(
+      bool AutoClaimWeakHiddenSymbols) {
+    this->AutoClaimWeakHiddenSymbols = AutoClaimWeakHiddenSymbols;
+    return *this;
+  }
+
   /// Register a JITEventListener.
   void registerJITEventListener(JITEventListener &L);
 
   /// Unregister a JITEventListener.
   void unregisterJITEventListener(JITEventListener &L);
 
 private:
   using MemoryManagerUP = std::unique_ptr<RuntimeDyld::MemoryManager>;
private:
 
   mutable std::mutex RTDyldLayerMutex;
   GetMemoryManagerFunction GetMemoryManager;
   NotifyLoadedFunction NotifyLoaded;
   NotifyEmittedFunction NotifyEmitted;
   bool ProcessAllSections = false;
   bool OverrideObjectFlags = false;
   bool AutoClaimObjectSymbols = false;
+  bool AutoClaimWeakHiddenSymbols = false;
   DenseMap<ResourceKey, std::vector<MemoryManagerUP>> MemMgrs;
   std::vector<JITEventListener *> EventListeners;
 };
 
 } // end namespace orc
 } // end namespace llvm
 
 #endif // LLVM_EXECUTIONENGINE_ORC_RTDYLDOBJECTLINKINGLAYER_H

llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp

if (!Obj) {
     R->failMaterialization();
     return;
   }
 
   // Collect the internal symbols from the object file: We will need to
   // filter these later.
   auto InternalSymbols = std::make_shared<std::set<StringRef>>();
   {
+    SymbolFlagsMap ExtraSymbolsToClaim;
     for (auto &Sym : (*Obj)->symbols()) {
 
       // Skip file symbols.
       if (auto SymType = Sym.getType()) {
         if (*SymType == object::SymbolRef::ST_File)
           continue;
       } else {
         ES.reportError(SymType.takeError());
         R->failMaterialization();
         return;
       }
 
       Expected<uint32_t> SymFlagsOrErr = Sym.getFlags();
       if (!SymFlagsOrErr) {
         // TODO: Test this error.
         ES.reportError(SymFlagsOrErr.takeError());
         R->failMaterialization();
         return;
       }
 
+      // Try to claim responsibility of weak hidden symbols
+      // if AutoClaimWeakHiddenSymbols flag is set.
+      if (AutoClaimWeakHiddenSymbols &&
+          (*SymFlagsOrErr & object::BasicSymbolRef::SF_Weak) &&
+          (*SymFlagsOrErr & object::BasicSymbolRef::SF_Hidden)) {
+        auto SymName = Sym.getName();
+        if (!SymName) {
+          ES.reportError(SymName.takeError());
+          R->failMaterialization();
+          return;
+        }
+
+        // Already included in responsibility set, skip it
+        if (R->getSymbols().count(ES.intern(*SymName))) {
+          continue;
+        }
+
+        auto SymFlags = JITSymbolFlags::fromObjectSymbol(Sym);
+        if (!SymFlags) {
+          ES.reportError(SymFlags.takeError());
+          R->failMaterialization();
+          return;
+        }
+
+        ExtraSymbolsToClaim[ES.intern(*SymName)] = *SymFlags;
+        continue;
+      }
+
       // Don't include symbols that aren't global.
       if (!(*SymFlagsOrErr & object::BasicSymbolRef::SF_Global)) {
         if (auto SymName = Sym.getName())
           InternalSymbols->insert(*SymName);
         else {
           ES.reportError(SymName.takeError());
           R->failMaterialization();
           return;
         }
       }
     }
+
+    if (!ExtraSymbolsToClaim.empty()) {
+      if (auto Err = R->defineMaterializing(ExtraSymbolsToClaim)) {
+        ES.reportError(std::move(Err));
+        R->failMaterialization();
+      }
+    }
   }
 
   auto MemMgr = GetMemoryManager();
   auto &MemMgrRef = *MemMgr;
 
   // Switch to shared ownership of MR so that it can be captured by both
   // lambdas below.
   std::shared_ptr<MaterializationResponsibility> SharedR(std::move(R));