[JITLink][Orc] Add MemoryMapper interface with POSIX shared memory implementation

[argentite]

Summary

MemoryMapper interface provides 3 functions:
+ reserve() reserves address space in executor process and working

memory in the controller process

+ finalize() ensures working memory is transferred to executor process

with appropriate memory protections

+ release() deallocates memory on both side

SharedMemoryMapper implementation uses POSIX shared memory APIs to map
shared memory pages in both processes at reserve(). finalize()
only sets the memory protections and release() unmaps the memory.
Executor side is implemented with ExecutorSharedMemoryManager.

[lhames]

I think it's worth adding an InProcessMapper too. That can be defined in ORC alongside the Mapper interface, and would give us a good way to test the API design separate from the shared memory details.

llvm/include/llvm/ExecutionEngine/Orc/MemoryMapper.h

29

I'd leave out the implicit cast here -- only people writing new mappers should need to worry about this, and it's fine (and safer) to make them spell out .RemoteAddr explicitly.

llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h

28

I lazily re-used the MemoryManager suffix for SimpleExecutorMemoryManager when I wrote it, but I think that was a mistake: it makes it harder to understand the difference between this "memory manager" and the JITLinkMemoryManager.

Maybe ExecutorSharedMemoryMapperService would be a better name for this. What do you think?

32

This operation should be renamed to reserve. We want to be clear that there will no longer be a 1-1 mapping between the alloc calls on our JITLinkMemoryManager and calls to ExecutorSharedMemoryManager::reserve.

We can start off that way (depending on whether you want to land this first patch earlier or later), but the eventual aim is that our new JITLinkMemoryManager's alloc method should look like this:

void allocate(const jitlink::JITLinkDylib *JD, jitlink::LinkGraph &G,
  OnAllocatedFunction OnAllocated) override {
  if (!haveSpaceForGraph(G))
    if (auto Err = reserveMoreSpace()) // RPC to ExecutorSharedMemoryManager::reserve.
      return Err;
  // Space is guaranteed to be available once we get here.
  return allocFromReservedSpace(JD, G, std::move(OnAllocated));
}

llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.cpp

34

I think we want to #ifdef out this entire class if LLVM_ON_UNIX is not set.

For now I'd literally wrap all the methods, and have a second set of implementations with llvm_unreachable for Windows. If the file gets unwieldy we can move to conditional includes like in https://github.com/llvm/llvm-project/blob/eb68cbb405512b448a2f3336e102982c144ad4e9/llvm/lib/Support/Memory.cpp#L21.

72

This looks like it was derived from SimpleExecutorMemoryManager -- I think we can actually shoot for something simpler in this new scheme where we separate the slab from the individual allocations within the slab.

Within ExecutorSharedMemoryManager you'll want to track two things: the slabs (which we'll just treat as pure memory ranges to be freed later), and allocations (finalized ranges within a slab with associated deallocation actions).

In this new scheme the finalize method just needs to apply protections, run finalization actions, and (on success) record the deallocation actions. We don't need to deallocate anything on failure, so we no longer need the complex bailout logic.

I think that this can simplify to:

Error ExecutorSharedMemoryManager::finalize(
  tpctypes::SharedMemoryFinalizeRequest &FR) {

  ExecutorAddr Base(~0ULL);
  for (auto &Seg : FR.Segments)
    Base = std::min(Base, Seg.Addr);

  // Future TODO -- scope exit to restore permissions (and ideally disassociate
  //   physical pages) on any subsequent errors.
  // --- Apply segment memory protections here ---

  auto DeallocActions = runFinalizeActions(FR.Actions);
  if (!DeallocActions)
    return DeallocActions.takeError();

  // Record allocation:
  Allocation[Base] = std::move(*DeallocActions);

  return Error::success();
}

[argentite]

Remove implicit ExecutorAddr cast from Mapping
Renamed allocate to reserve and deallocate to release

llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h

28

How about just ExecutorSharedMemoryService? Besides mapping pages, it also needs to keep track of allocation and their actions as well.

[lhames]

llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h

28

Hrm. I thought ExecutorSharedMemoryMapperService to make clear that it's the executor-side implementation of the ShardedMemoryMapper class.

Maybe we need a better name than Mapper?

I think go with whichever name suits you best for now and we can revisit the question later.

[argentite]

This code has been merged as part of https://reviews.llvm.org/D127491 and https://reviews.llvm.org/D128544

llvm/include/llvm/ExecutionEngine/Orc/MemoryMapper.h

+//===- MemoryMapper.h - Cross-process memory mapper -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// TODO
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_ORC_MEMORYMAPPER_H
+#define LLVM_EXECUTIONENGINE_ORC_MEMORYMAPPER_H
+
+#include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/Support/Error.h"
+
+namespace llvm {
+namespace orc {
+
+class MemoryMapper {
+public:
+  struct Mapping {
+    ExecutorAddr RemoteAddr;
+    void *LocalAddr;
+    size_t Size;
+
+    operator ExecutorAddr() const { return RemoteAddr; }
+  };
+
+  using ReserveResult = Expected<Mapping>;
+  using OnReservedFunction = unique_function<void(ReserveResult)>;
+  virtual void reserve(size_t NumBytes, OnReservedFunction OnReserved) = 0;
+  virtual ReserveResult reserve(size_t NumBytes);
+
+  using OnFinalizedFunction = unique_function<void(Error)>;
+  virtual void finalize(Mapping &M,
+                        tpctypes::SharedMemoryFinalizeRequest &Actions,
+                        OnFinalizedFunction OnReserved) = 0;
+  virtual Error finalize(Mapping &M,
+                         tpctypes::SharedMemoryFinalizeRequest &Actions);
+
+  using OnDeallocatedFunction = unique_function<void(Error)>;
+  virtual Error release(std::vector<Mapping> &Allocs,
+                        OnDeallocatedFunction OnDeallocation) = 0;
+  virtual Error release(std::vector<Mapping> &Allocs);
+
+  virtual ~MemoryMapper();
+};
+
+class SharedMemoryMapper final : public MemoryMapper {
+public:
+  struct SymbolAddrs {
+    ExecutorAddr Manager;
+    ExecutorAddr Reserve;
+    ExecutorAddr Finalize;
+    ExecutorAddr Deallocate;
+  };
+
+  SharedMemoryMapper(ExecutorProcessControl &EPC, SymbolAddrs SAs)
+      : EPC(EPC), SAs(SAs) {}
+
+  void reserve(size_t NumBytes, OnReservedFunction OnReserved) override;
+
+  void finalize(Mapping &M, tpctypes::SharedMemoryFinalizeRequest &Actions,
+                OnFinalizedFunction OnReserved) override;
+
+  Error release(std::vector<Mapping> &Mappings,
+                OnDeallocatedFunction OnDeallocated) override;
+
+  ~SharedMemoryMapper() override;
+
+private:
+  ExecutorProcessControl &EPC;
+  SymbolAddrs SAs;
+};
+
+} // namespace orc
+} // end namespace llvm
+
+#endif // LLVM_EXECUTIONENGINE_ORC_MEMORYMAPPER_H

llvm/include/llvm/ExecutionEngine/Orc/Shared/OrcRTBridge.h

 extern const char *SimpleExecutorDylibManagerOpenWrapperName;
 extern const char *SimpleExecutorDylibManagerLookupWrapperName;
 
 extern const char *SimpleExecutorMemoryManagerInstanceName;
 extern const char *SimpleExecutorMemoryManagerReserveWrapperName;
 extern const char *SimpleExecutorMemoryManagerFinalizeWrapperName;
 extern const char *SimpleExecutorMemoryManagerDeallocateWrapperName;
 
+extern const char *ExecutorSharedMemoryManagerInstanceName;
+extern const char *ExecutorSharedMemoryManagerReserveWrapperName;
+extern const char *ExecutorSharedMemoryManagerFinalizeWrapperName;
+extern const char *ExecutorSharedMemoryManagerDeallocateWrapperName;
+
 extern const char *MemoryWriteUInt8sWrapperName;
 extern const char *MemoryWriteUInt16sWrapperName;
 extern const char *MemoryWriteUInt32sWrapperName;
 extern const char *MemoryWriteUInt64sWrapperName;
 extern const char *MemoryWriteBuffersWrapperName;
 
 extern const char *RegisterEHFrameSectionWrapperName;
 extern const char *DeregisterEHFrameSectionWrapperName;

 using SPSSimpleExecutorMemoryManagerReserveSignature =
     shared::SPSExpected<shared::SPSExecutorAddr>(shared::SPSExecutorAddr,
                                                  uint64_t);
 using SPSSimpleExecutorMemoryManagerFinalizeSignature =
     shared::SPSError(shared::SPSExecutorAddr, shared::SPSFinalizeRequest);
 using SPSSimpleExecutorMemoryManagerDeallocateSignature = shared::SPSError(
     shared::SPSExecutorAddr, shared::SPSSequence<shared::SPSExecutorAddr>);
 
+using SPSExecutorSharedMemoryManagerReserveSignature = shared::SPSExpected<
+    shared::SPSTuple<shared::SPSExecutorAddr, shared::SPSString>>(
+    shared::SPSExecutorAddr, uint64_t);
+using SPSExecutorSharedMemoryManagerFinalizeSignature = shared::SPSError(
+    shared::SPSExecutorAddr, shared::SPSSharedMemoryFinalizeRequest);
+using SPSExecutorSharedMemoryManagerDeallocateSignature = shared::SPSError(
+    shared::SPSExecutorAddr, shared::SPSSequence<shared::SPSExecutorAddr>);
+
 using SPSRunAsMainSignature = int64_t(shared::SPSExecutorAddr,
                                       shared::SPSSequence<shared::SPSString>);
 
 } // end namespace rt
 } // end namespace orc
 } // end namespace llvm
 
 #endif // LLVM_EXECUTIONENGINE_ORC_SHARED_ORCRTBRIDGE_H

llvm/include/llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h

struct SegFinalizeRequest {
   ArrayRef<char> Content;
 };
 
 struct FinalizeRequest {
   std::vector<SegFinalizeRequest> Segments;
   shared::AllocActions Actions;
 };
 
+struct SharedMemorySegFinalizeRequest {
+  WireProtectionFlags Prot;
+  ExecutorAddr Addr;
+  uint64_t Size;
+};
+
+struct SharedMemoryFinalizeRequest {
+  std::vector<SharedMemorySegFinalizeRequest> Segments;
+  shared::AllocActions Actions;
+};
+
 template <typename T> struct UIntWrite {
   UIntWrite() = default;
   UIntWrite(ExecutorAddr Addr, T Value) : Addr(Addr), Value(Value) {}
 
   ExecutorAddr Addr;
   T Value = 0;
 };
 

 
 using SPSSegFinalizeRequest =
     SPSTuple<SPSMemoryProtectionFlags, SPSExecutorAddr, uint64_t,
              SPSSequence<char>>;
 
 using SPSFinalizeRequest = SPSTuple<SPSSequence<SPSSegFinalizeRequest>,
                                     SPSSequence<SPSAllocActionCallPair>>;
 
+using SPSSharedMemorySegFinalizeRequest =
+    SPSTuple<SPSMemoryProtectionFlags, SPSExecutorAddr, uint64_t>;
+
+using SPSSharedMemoryFinalizeRequest =
+    SPSTuple<SPSSequence<SPSSharedMemorySegFinalizeRequest>,
+             SPSSequence<SPSAllocActionCallPair>>;
+
 template <typename T>
 using SPSMemoryAccessUIntWrite = SPSTuple<SPSExecutorAddr, T>;
 
 using SPSMemoryAccessUInt8Write = SPSMemoryAccessUIntWrite<uint8_t>;
 using SPSMemoryAccessUInt16Write = SPSMemoryAccessUIntWrite<uint16_t>;
 using SPSMemoryAccessUInt32Write = SPSMemoryAccessUIntWrite<uint32_t>;
 using SPSMemoryAccessUInt64Write = SPSMemoryAccessUIntWrite<uint64_t>;
 
static bool serialize(SPSOutputBuffer &OB,
     return FRAL::serialize(OB, FR.Segments, FR.Actions);
   }
 
   static bool deserialize(SPSInputBuffer &IB, tpctypes::FinalizeRequest &FR) {
     return FRAL::deserialize(IB, FR.Segments, FR.Actions);
   }
 };
 
+template <>
+class SPSSerializationTraits<SPSSharedMemorySegFinalizeRequest,
+                             tpctypes::SharedMemorySegFinalizeRequest> {
+  using SFRAL = SPSSharedMemorySegFinalizeRequest::AsArgList;
+
+public:
+  static size_t size(const tpctypes::SharedMemorySegFinalizeRequest &SFR) {
+    return SFRAL::size(SFR.Prot, SFR.Addr, SFR.Size);
+  }
+
+  static bool serialize(SPSOutputBuffer &OB,
+                        const tpctypes::SharedMemorySegFinalizeRequest &SFR) {
+
+    return SFRAL::serialize(OB, SFR.Prot, SFR.Addr, SFR.Size);
+  }
+
+  static bool deserialize(SPSInputBuffer &IB,
+                          tpctypes::SharedMemorySegFinalizeRequest &SFR) {
+    return SFRAL::deserialize(IB, SFR.Prot, SFR.Addr, SFR.Size);
+  }
+};
+
+template <>
+class SPSSerializationTraits<SPSSharedMemoryFinalizeRequest,
+                             tpctypes::SharedMemoryFinalizeRequest> {
+  using FRAL = SPSSharedMemoryFinalizeRequest::AsArgList;
+
+public:
+  static size_t size(const tpctypes::SharedMemoryFinalizeRequest &FR) {
+    return FRAL::size(FR.Segments, FR.Actions);
+  }
+
+  static bool serialize(SPSOutputBuffer &OB,
+                        const tpctypes::SharedMemoryFinalizeRequest &FR) {
+    return FRAL::serialize(OB, FR.Segments, FR.Actions);
+  }
+
+  static bool deserialize(SPSInputBuffer &IB,
+                          tpctypes::SharedMemoryFinalizeRequest &FR) {
+    return FRAL::deserialize(IB, FR.Segments, FR.Actions);
+  }
+};
+
 template <typename T>
 class SPSSerializationTraits<SPSMemoryAccessUIntWrite<T>,
                              tpctypes::UIntWrite<T>> {
 public:
   static size_t size(const tpctypes::UIntWrite<T> &W) {
     return SPSTuple<SPSExecutorAddr, T>::AsArgList::size(W.Addr, W.Value);
   }
 
public:
 
   static bool deserialize(SPSInputBuffer &IB, tpctypes::BufferWrite &W) {
     return SPSTuple<SPSExecutorAddr,
                     SPSSequence<char>>::AsArgList ::deserialize(IB, W.Addr,
                                                                 W.Buffer);
   }
 };
 
-
 } // end namespace shared
 } // end namespace orc
 } // end namespace llvm
 
 #endif // LLVM_EXECUTIONENGINE_ORC_SHARED_TARGETPROCESSCONTROLTYPES_H

llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h

+//===---------------- ExecutorSharedMemoryManager.h -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// TODO
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_ORC_TARGETPROCESS_EXECUTORSHAREDMEMORYMANAGER_H
+#define LLVM_EXECUTIONENGINE_ORC_TARGETPROCESS_EXECUTORSHAREDMEMORYMANAGER_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.h"
+#include "llvm/ExecutionEngine/Orc/TargetProcess/ExecutorBootstrapService.h"
+#include "llvm/Support/Error.h"
+
+#include <atomic>
+#include <mutex>
+
+namespace llvm {
+namespace orc {
+namespace rt_bootstrap {
+
+class ExecutorSharedMemoryManager : public ExecutorBootstrapService {
+public:
+  virtual ~ExecutorSharedMemoryManager();
+
+  Expected<std::pair<ExecutorAddr, std::string>> allocate(uint64_t Size);
+  Error finalize(tpctypes::SharedMemoryFinalizeRequest &FR);
+  Error deallocate(const std::vector<ExecutorAddr> &Bases);
+
+  Error shutdown() override;
+  void addBootstrapSymbols(StringMap<ExecutorAddr> &M) override;
+
+private:
+  struct Allocation {
+    size_t Size = 0;
+    std::vector<shared::WrapperFunctionCall> DeallocationActions;
+  };
+
+  using AllocationsMap = DenseMap<void *, Allocation>;
+
+  Error deallocateImpl(void *Base, Allocation &A);
+
+  static llvm::orc::shared::CWrapperFunctionResult
+  reserveWrapper(const char *ArgData, size_t ArgSize);
+
+  static llvm::orc::shared::CWrapperFunctionResult
+  finalizeWrapper(const char *ArgData, size_t ArgSize);
+
+  static llvm::orc::shared::CWrapperFunctionResult
+  deallocateWrapper(const char *ArgData, size_t ArgSize);
+
+  std::atomic<int> SharedMemoryCount{0};
+  std::mutex M;
+  AllocationsMap Allocations;
+};
+
+} // end namespace rt_bootstrap
+} // end namespace orc
+} // end namespace llvm
+
+#endif // LLVM_EXECUTIONENGINE_ORC_TARGETPROCESS_EXECUTORSHAREDMEMORYMANAGER_H

llvm/lib/ExecutionEngine/Orc/CMakeLists.txt

add_llvm_component_library(LLVMOrcJIT
   IRCompileLayer.cpp
   IRTransformLayer.cpp
   JITTargetMachineBuilder.cpp
   LazyReexports.cpp
   Layer.cpp
   LookupAndRecordAddrs.cpp
   LLJIT.cpp
   MachOPlatform.cpp
+  MemoryMapper.cpp
   ELFNixPlatform.cpp
   Mangling.cpp
   ObjectLinkingLayer.cpp
   ObjectTransformLayer.cpp
   OrcABISupport.cpp
   OrcV2CBindings.cpp
   RTDyldObjectLinkingLayer.cpp
   SimpleRemoteEPC.cpp

llvm/lib/ExecutionEngine/Orc/MemoryMapper.cpp

+//===- MemoryMapper.cpp - Cross-process memory mapper ------------*- C++ -*-==//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// TODO
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/Orc/MemoryMapper.h"
+#include "llvm/ExecutionEngine/Orc/Shared/OrcRTBridge.h"
+
+#include <string>
+
+#ifdef LLVM_ON_UNIX
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
+
+namespace llvm {
+namespace orc {
+
+MemoryMapper::ReserveResult MemoryMapper::reserve(size_t NumBytes) {
+  std::promise<ReserveResult> P;
+  auto F = P.get_future();
+  reserve(NumBytes,
+          [&](MemoryMapper::ReserveResult R) { P.set_value(std::move(R)); });
+  return F.get();
+}
+
+Error MemoryMapper::finalize(MemoryMapper::Mapping &M,
+                             tpctypes::SharedMemoryFinalizeRequest &Actions) {
+  std::promise<Error> P;
+  auto F = P.get_future();
+  finalize(M, Actions, [&](Error R) { P.set_value(std::move(R)); });
+  return F.get();
+}
+
+Error MemoryMapper::release(std::vector<MemoryMapper::Mapping> &Allocs) {
+  std::promise<Error> P;
+  auto F = P.get_future();
+  Error ErrLocal = release(Allocs, [&](Error R) { P.set_value(std::move(R)); });
+  return joinErrors(std::move(ErrLocal), F.get());
+}
+
+MemoryMapper::~MemoryMapper() {}
+
+void SharedMemoryMapper::reserve(size_t Size, OnReservedFunction OnReserved) {
+  EPC.callSPSWrapperAsync<rt::SPSExecutorSharedMemoryManagerReserveSignature>(
+      SAs.Reserve,
+      [Size, OnReserved = std::move(OnReserved)](
+          Error SerializationErr,
+          Expected<std::pair<ExecutorAddr, std::string>> Result) mutable {
+        if (SerializationErr) {
+          cantFail(Result.takeError());
+          return OnReserved(std::move(SerializationErr));
+        }
+        if (!Result)
+          return OnReserved(Result.takeError());
+
+        ExecutorAddr RemoteAddr;
+        std::string SharedMemoryName;
+        std::tie(RemoteAddr, SharedMemoryName) = std::move(*Result);
+
+#ifdef LLVM_ON_UNIX
+        int SharedMemoryFile = shm_open(SharedMemoryName.c_str(), O_RDWR, 0700);
+        if (SharedMemoryFile < 0) {
+          return OnReserved(errorCodeToError(
+              std::error_code(errno, std::generic_category())));
+        }
+
+        // this prevents other programs from accessing it by name
+        shm_unlink(SharedMemoryName.c_str());
+
+        void *LocalAddr = mmap(nullptr, Size, PROT_READ | PROT_WRITE,
+                               MAP_SHARED, SharedMemoryFile, 0);
+        if (LocalAddr == MAP_FAILED) {
+          return OnReserved(errorCodeToError(
+              std::error_code(errno, std::generic_category())));
+        }
+        close(SharedMemoryFile);
+#endif
+        OnReserved(Mapping{RemoteAddr, LocalAddr, Size});
+      },
+      SAs.Manager, Size);
+}
+
+void SharedMemoryMapper::finalize(Mapping &M,
+                                  tpctypes::SharedMemoryFinalizeRequest &FR,
+                                  OnFinalizedFunction OnFinalize) {
+  EPC.callSPSWrapperAsync<rt::SPSExecutorSharedMemoryManagerFinalizeSignature>(
+      SAs.Finalize,
+      [OnFinalize = std::move(OnFinalize)](Error SerializationErr,
+                                           Error FinalizeErr) mutable {
+        if (SerializationErr) {
+          cantFail(std::move(FinalizeErr));
+          OnFinalize(std::move(SerializationErr));
+        } else if (FinalizeErr)
+          OnFinalize(std::move(FinalizeErr));
+        else
+          OnFinalize(Error::success());
+      },
+      SAs.Manager, std::move(FR));
+}
+
+Error SharedMemoryMapper::release(std::vector<Mapping> &Mappings,
+                                  OnDeallocatedFunction OnDeallocated) {
+  std::vector<ExecutorAddr> Bases(Mappings.begin(), Mappings.end());
+  EPC.callSPSWrapperAsync<
+      rt::SPSExecutorSharedMemoryManagerDeallocateSignature>(
+      SAs.Deallocate,
+      [OnDeallocated = std::move(OnDeallocated)](Error SerErr,
+                                                 Error DeallocErr) mutable {
+        if (SerErr) {
+          cantFail(std::move(DeallocErr));
+          OnDeallocated(std::move(SerErr));
+        } else
+          OnDeallocated(std::move(DeallocErr));
+      },
+      SAs.Manager, Bases);
+
+  Error Err = Error::success();
+
+  for (auto M : Mappings) {
+#ifdef LLVM_ON_UNIX
+    if (munmap(M.LocalAddr, M.Size) != 0) {
+      Err = joinErrors(std::move(Err), errorCodeToError(std::error_code(
+                                           errno, std::generic_category())));
+    }
+#endif
+  }
+
+  return Err;
+}
+
+SharedMemoryMapper::~SharedMemoryMapper() {}
+
+} // namespace orc
+
+} // namespace llvm

llvm/lib/ExecutionEngine/Orc/Shared/OrcRTBridge.cpp

 const char *SimpleExecutorMemoryManagerInstanceName =
     "__llvm_orc_SimpleExecutorMemoryManager_Instance";
 const char *SimpleExecutorMemoryManagerReserveWrapperName =
     "__llvm_orc_SimpleExecutorMemoryManager_reserve_wrapper";
 const char *SimpleExecutorMemoryManagerFinalizeWrapperName =
     "__llvm_orc_SimpleExecutorMemoryManager_finalize_wrapper";
 const char *SimpleExecutorMemoryManagerDeallocateWrapperName =
     "__llvm_orc_SimpleExecutorMemoryManager_deallocate_wrapper";
+const char *ExecutorSharedMemoryManagerInstanceName =
+    "__llvm_orc_ExecutorSharedMemoryManager_Instance";
+const char *ExecutorSharedMemoryManagerReserveWrapperName =
+    "__llvm_orc_ExecutorSharedMemoryManager_reserve_wrapper";
+const char *ExecutorSharedMemoryManagerFinalizeWrapperName =
+    "__llvm_orc_ExecutorSharedMemoryManager_finalize_wrapper";
+const char *ExecutorSharedMemoryManagerDeallocateWrapperName =
+    "__llvm_orc_ExecutorSharedMemoryManager_deallocate_wrapper";
 const char *MemoryWriteUInt8sWrapperName =
     "__llvm_orc_bootstrap_mem_write_uint8s_wrapper";
 const char *MemoryWriteUInt16sWrapperName =
     "__llvm_orc_bootstrap_mem_write_uint16s_wrapper";
 const char *MemoryWriteUInt32sWrapperName =
     "__llvm_orc_bootstrap_mem_write_uint32s_wrapper";
 const char *MemoryWriteUInt64sWrapperName =
     "__llvm_orc_bootstrap_mem_write_uint64s_wrapper";

llvm/lib/ExecutionEngine/Orc/TargetProcess/CMakeLists.txt

 add_llvm_component_library(LLVMOrcTargetProcess
+  ExecutorSharedMemoryManager.cpp
   JITLoaderGDB.cpp
   OrcRTBootstrap.cpp
   RegisterEHFrames.cpp
   SimpleExecutorDylibManager.cpp
   SimpleExecutorMemoryManager.cpp
   SimpleRemoteEPCServer.cpp
   TargetExecutionUtils.cpp
 

llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.cpp

+//===- ExecutorShardMemoryManagare.cpp - Executor-side shared memory mgmt -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h"
+
+#include "llvm/ExecutionEngine/Orc/Shared/OrcRTBridge.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/Process.h"
+
+#include <sstream>
+
+#ifdef LLVM_ON_UNIX
+#include <errno.h>
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
+
+namespace llvm {
+namespace orc {
+namespace rt_bootstrap {
+
+ExecutorSharedMemoryManager::~ExecutorSharedMemoryManager() {
+  assert(Allocations.empty() && "shutdown not called?");
+}
+
+Expected<std::pair<ExecutorAddr, std::string>>
+ExecutorSharedMemoryManager::allocate(uint64_t Size) {
+#ifdef LLVM_ON_UNIX
+  std::string SharedMemoryName;
+  {
+    std::stringstream SharedMemoryNameStream;
+    SharedMemoryNameStream << "/jitlink_" << sys::Process::getProcessId() << '_'
+                           << (++SharedMemoryCount);
+    SharedMemoryName = SharedMemoryNameStream.str();
+  }
+
+  int SharedMemoryFile =
+      shm_open(SharedMemoryName.c_str(), O_RDWR | O_CREAT | O_EXCL, 0700);
+  if (SharedMemoryFile < 0) {
+    return errorCodeToError(std::error_code(errno, std::generic_category()));
+  }
+
+  // by default size is 0
+  if (ftruncate(SharedMemoryFile, Size) < 0) {
+    return errorCodeToError(std::error_code(errno, std::generic_category()));
+  }
+
+  void *Addr = mmap(nullptr, Size, PROT_NONE, MAP_SHARED, SharedMemoryFile, 0);
+  if (Addr == MAP_FAILED) {
+    return errorCodeToError(std::error_code(errno, std::generic_category()));
+  }
+
+  close(SharedMemoryFile);
+
+  {
+    std::lock_guard<std::mutex> Lock(M);
+    assert(!Allocations.count(Addr) && "Duplicate allocation addr");
+    Allocations[Addr].Size = Size;
+  }
+
+  return std::make_pair(ExecutorAddr::fromPtr(Addr),
+                        std::move(SharedMemoryName));
+#endif
+}
+
+Error ExecutorSharedMemoryManager::finalize(
+    tpctypes::SharedMemoryFinalizeRequest &FR) {
+  ExecutorAddr Base(~0ULL);
+  std::vector<shared::WrapperFunctionCall> DeallocationActions;
+  size_t SuccessfulFinalizationActions = 0;
+
+  if (FR.Segments.empty()) {
+    if (FR.Actions.empty())
+      return Error::success();
+    return make_error<StringError>(
+        "Finalization actions attached to empty finalization request",
+        inconvertibleErrorCode());
+  }
+
+  for (auto &Seg : FR.Segments)
+    Base = std::min(Base, Seg.Addr);
+
+  for (auto &ActPair : FR.Actions)
+    if (ActPair.Dealloc)
+      DeallocationActions.push_back(ActPair.Dealloc);
+
+  // Get the Allocation for this finalization.
+  size_t AllocSize = 0;
+  {
+    std::lock_guard<std::mutex> Lock(M);
+    auto I = Allocations.find(Base.toPtr<void *>());
+    if (I == Allocations.end())
+      return make_error<StringError>(
+          formatv("Attempt to finalize unrecognized allocation {0:x}",
+                  Base.getValue()),
+          inconvertibleErrorCode());
+    AllocSize = I->second.Size;
+    I->second.DeallocationActions = std::move(DeallocationActions);
+  }
+  ExecutorAddr AllocEnd = Base + ExecutorAddrDiff(AllocSize);
+
+  // Bail-out function: this will run deallocation actions corresponding to any
+  // completed finalization actions, then deallocate memory.
+  auto BailOut = [&](Error Err) {
+    std::pair<void *, Allocation> AllocToDestroy;
+
+    // Get allocation to destory.
+    {
+      std::lock_guard<std::mutex> Lock(M);
+      auto I = Allocations.find(Base.toPtr<void *>());
+
+      // Check for missing allocation (effective a double free).
+      if (I == Allocations.end())
+        return joinErrors(
+            std::move(Err),
+            make_error<StringError>(
+                formatv("No allocation entry found for {0:x}", Base.getValue()),
+                inconvertibleErrorCode()));
+      AllocToDestroy = std::move(*I);
+      Allocations.erase(I);
+    }
+
+    // Run deallocation actions for all completed finalization actions.
+    while (SuccessfulFinalizationActions)
+      Err =
+          joinErrors(std::move(Err), FR.Actions[--SuccessfulFinalizationActions]
+                                         .Dealloc.runWithSPSRetErrorMerged());
+
+    // Deallocate memory.
+    // TODO
+    sys::MemoryBlock MB(AllocToDestroy.first, AllocToDestroy.second.Size);
+    if (auto EC = sys::Memory::releaseMappedMemory(MB))
+      Err = joinErrors(std::move(Err), errorCodeToError(EC));
+
+    return Err;
+  };
+
+  // Copy content and apply permissions.
+  for (auto &Seg : FR.Segments) {
+
+    ExecutorAddr SegEnd = Seg.Addr + ExecutorAddrDiff(Seg.Size);
+    if (LLVM_UNLIKELY(Seg.Addr < Base || SegEnd > AllocEnd))
+      return BailOut(make_error<StringError>(
+          formatv("Segment {0:x} -- {1:x} crosses boundary of "
+                  "allocation {2:x} -- {3:x}",
+                  Seg.Addr.getValue(), SegEnd.getValue(), Base.getValue(),
+                  AllocEnd.getValue()),
+          inconvertibleErrorCode()));
+
+    char *Mem = Seg.Addr.toPtr<char *>();
+    assert(Seg.Size <= std::numeric_limits<size_t>::max());
+
+#ifdef LLVM_ON_UNIX
+
+    int MmapProt = 0;
+    if (Seg.Prot & tpctypes::WPF_Read)
+      MmapProt |= PROT_READ;
+    if (Seg.Prot & tpctypes::WPF_Write)
+      MmapProt |= PROT_WRITE;
+    if (Seg.Prot & tpctypes::WPF_Exec)
+      MmapProt |= PROT_EXEC;
+    mprotect(Mem, Seg.Size, MmapProt);
+
+    if (Seg.Prot & tpctypes::WPF_Exec)
+      sys::Memory::InvalidateInstructionCache(Mem, Seg.Size);
+  }
+#endif
+
+  // Run finalization actions.
+  for (auto &ActPair : FR.Actions) {
+    if (auto Err = ActPair.Finalize.runWithSPSRetErrorMerged())
+      return BailOut(std::move(Err));
+    ++SuccessfulFinalizationActions;
+  }
+
+  return Error::success();
+}
+
+Error ExecutorSharedMemoryManager::deallocate(
+    const std::vector<ExecutorAddr> &Bases) {
+  std::vector<std::pair<void *, Allocation>> AllocPairs;
+  AllocPairs.reserve(Bases.size());
+
+  // Get allocation to destory.
+  Error Err = Error::success();
+  {
+    std::lock_guard<std::mutex> Lock(M);
+    for (auto &Base : Bases) {
+      auto I = Allocations.find(Base.toPtr<void *>());
+
+      // Check for missing allocation (effective a double free).
+      if (I != Allocations.end()) {
+        AllocPairs.push_back(std::move(*I));
+        Allocations.erase(I);
+      } else
+        Err = joinErrors(
+            std::move(Err),
+            make_error<StringError>(
+                formatv("No allocation entry found for {0:x}", Base.getValue()),
+                inconvertibleErrorCode()));
+    }
+  }
+
+  while (!AllocPairs.empty()) {
+    auto &P = AllocPairs.back();
+    Err = joinErrors(std::move(Err), deallocateImpl(P.first, P.second));
+    AllocPairs.pop_back();
+  }
+
+  return Err;
+}
+
+Error ExecutorSharedMemoryManager::shutdown() {
+
+  AllocationsMap AM;
+  {
+    std::lock_guard<std::mutex> Lock(M);
+    AM = std::move(Allocations);
+  }
+
+  Error Err = Error::success();
+  for (auto &KV : AM)
+    Err = joinErrors(std::move(Err), deallocateImpl(KV.first, KV.second));
+  return Err;
+}
+
+void ExecutorSharedMemoryManager::addBootstrapSymbols(
+    StringMap<ExecutorAddr> &M) {
+  M[rt::ExecutorSharedMemoryManagerInstanceName] = ExecutorAddr::fromPtr(this);
+  M[rt::ExecutorSharedMemoryManagerReserveWrapperName] =
+      ExecutorAddr::fromPtr(&reserveWrapper);
+  M[rt::ExecutorSharedMemoryManagerFinalizeWrapperName] =
+      ExecutorAddr::fromPtr(&finalizeWrapper);
+  M[rt::ExecutorSharedMemoryManagerDeallocateWrapperName] =
+      ExecutorAddr::fromPtr(&deallocateWrapper);
+}
+
+Error ExecutorSharedMemoryManager::deallocateImpl(void *Base, Allocation &A) {
+  Error Err = Error::success();
+
+  while (!A.DeallocationActions.empty()) {
+    Err = joinErrors(std::move(Err),
+                     A.DeallocationActions.back().runWithSPSRetErrorMerged());
+    A.DeallocationActions.pop_back();
+  }
+
+#ifdef LLVM_ON_UNIX
+  if (munmap(Base, A.Size) != 0)
+    Err = joinErrors(std::move(Err), errorCodeToError(std::error_code(
+                                         errno, std::generic_category())));
+#endif
+
+  return Err;
+}
+
+llvm::orc::shared::CWrapperFunctionResult
+ExecutorSharedMemoryManager::reserveWrapper(const char *ArgData,
+                                            size_t ArgSize) {
+  return shared::WrapperFunction<
+             rt::SPSExecutorSharedMemoryManagerReserveSignature>::
+      handle(ArgData, ArgSize,
+             shared::makeMethodWrapperHandler(
+                 &ExecutorSharedMemoryManager::allocate))
+          .release();
+}
+
+llvm::orc::shared::CWrapperFunctionResult
+ExecutorSharedMemoryManager::finalizeWrapper(const char *ArgData,
+                                             size_t ArgSize) {
+  return shared::WrapperFunction<
+             rt::SPSExecutorSharedMemoryManagerFinalizeSignature>::
+      handle(ArgData, ArgSize,
+             shared::makeMethodWrapperHandler(
+                 &ExecutorSharedMemoryManager::finalize))
+          .release();
+}
+
+llvm::orc::shared::CWrapperFunctionResult
+ExecutorSharedMemoryManager::deallocateWrapper(const char *ArgData,
+                                               size_t ArgSize) {
+  return shared::WrapperFunction<
+             rt::SPSExecutorSharedMemoryManagerDeallocateSignature>::
+      handle(ArgData, ArgSize,
+             shared::makeMethodWrapperHandler(
+                 &ExecutorSharedMemoryManager::deallocate))
+          .release();
+}
+
+} // namespace rt_bootstrap
+} // end namespace orc
+} // end namespace llvm

llvm/unittests/ExecutionEngine/Orc/CMakeLists.txt

set(LLVM_LINK_COMPONENTS
   Support
   native
   )
 
 add_llvm_unittest(OrcJITTests
   CoreAPIsTest.cpp
   ExecutorAddressTest.cpp
   ExecutionSessionWrapperFunctionCallsTest.cpp
+  ExecutorSharedMemoryManagerTest.cpp
   EPCGenericJITLinkMemoryManagerTest.cpp
   EPCGenericMemoryAccessTest.cpp
   IndirectionUtilsTest.cpp
   JITTargetMachineBuilderTest.cpp
   LazyCallThroughAndReexportsTest.cpp
   LookupAndRecordAddrsTest.cpp
+  SharedMemoryMapperTest.cpp
   ObjectLinkingLayerTest.cpp
   OrcCAPITest.cpp
   OrcTestCommon.cpp
   ResourceTrackerTest.cpp
   RTDyldObjectLinkingLayerTest.cpp
   SimpleExecutorMemoryManagerTest.cpp
   SimplePackedSerializationTest.cpp
   SymbolStringPoolTest.cpp

llvm/unittests/ExecutionEngine/Orc/ExecutorSharedMemoryManagerTest.cpp

+//===---------------- ExecutorSharedMemoryManagerTest.cpp -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gtest/gtest.h"
+
+#include <cstring>
+#include <limits>
+#include <vector>
+
+#ifdef LLVM_ON_UNIX
+#include <fcntl.h>
+#include <sys/mman.h>
+#endif
+
+using namespace llvm;
+using namespace llvm::orc;
+using namespace llvm::orc::shared;
+using namespace llvm::orc::rt_bootstrap;
+
+namespace {
+
+#ifdef LLVM_ON_UNIX
+orc::shared::CWrapperFunctionResult incrementWrapper(const char *ArgData,
+                                                     size_t ArgSize) {
+  return WrapperFunction<SPSError(SPSExecutorAddr)>::handle(
+             ArgData, ArgSize,
+             [](ExecutorAddr A) -> Error {
+               *A.toPtr<int *>() += 1;
+               return Error::success();
+             })
+      .release();
+}
+
+TEST(ExecutorSharedMemoryManagerTest, AllocFinalizeFree) {
+  ExecutorSharedMemoryManager MemMgr;
+
+  constexpr unsigned AllocSize = 16384;
+
+  auto Mem = MemMgr.allocate(AllocSize);
+  EXPECT_THAT_ERROR(Mem.takeError(), Succeeded());
+
+  ExecutorAddr ExecAddr;
+  std::string Name;
+  std::tie(ExecAddr, Name) = *Mem;
+
+  int File = shm_open(Name.c_str(), O_RDWR, 0700);
+  EXPECT_GT(File, 0);
+
+  void *WorkingAddr =
+      mmap(nullptr, AllocSize, PROT_READ | PROT_WRITE, MAP_SHARED, File, 0);
+  EXPECT_NE(WorkingAddr, MAP_FAILED);
+
+  shm_unlink(Name.c_str());
+
+  std::string HW = "Hello, world!";
+
+  std::strcpy((char *)WorkingAddr, HW.c_str());
+
+  int FinalizeCounter = 0;
+  int DeallocateCounter = 0;
+
+  tpctypes::SharedMemoryFinalizeRequest FR;
+  FR.Segments.push_back(tpctypes::SharedMemorySegFinalizeRequest{
+      tpctypes::WPF_Read | tpctypes::WPF_Write,
+      ExecAddr,
+      AllocSize,
+  });
+  FR.Actions.push_back(
+      {/* Finalize: */
+       cantFail(WrapperFunctionCall::Create<SPSArgList<SPSExecutorAddr>>(
+           ExecutorAddr::fromPtr(incrementWrapper),
+           ExecutorAddr::fromPtr(&FinalizeCounter))),
+       /*  Deallocate: */
+       cantFail(WrapperFunctionCall::Create<SPSArgList<SPSExecutorAddr>>(
+           ExecutorAddr::fromPtr(incrementWrapper),
+           ExecutorAddr::fromPtr(&DeallocateCounter)))});
+
+  EXPECT_EQ(FinalizeCounter, 0);
+  EXPECT_EQ(DeallocateCounter, 0);
+
+  auto FinalizeErr = MemMgr.finalize(FR);
+  EXPECT_THAT_ERROR(std::move(FinalizeErr), Succeeded());
+
+  EXPECT_EQ(FinalizeCounter, 1);
+  EXPECT_EQ(DeallocateCounter, 0);
+
+  EXPECT_EQ(HW, std::string(ExecAddr.toPtr<const char *>()));
+
+  auto DeallocateErr = MemMgr.deallocate({ExecAddr});
+  EXPECT_THAT_ERROR(std::move(DeallocateErr), Succeeded());
+
+  EXPECT_EQ(FinalizeCounter, 1);
+  EXPECT_EQ(DeallocateCounter, 1);
+}
+#endif
+
+} // namespace

llvm/unittests/ExecutionEngine/Orc/SharedMemoryMapperTest.cpp

+//===- SharedMemoryMapperTest.cpp -- Tests for SharedMemoryMapper ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "OrcTestCommon.h"
+
+#include "llvm/ExecutionEngine/Orc/MemoryMapper.h"
+#include "llvm/ExecutionEngine/Orc/Shared/OrcRTBridge.h"
+#include "llvm/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryManager.h"
+#include "llvm/Testing/Support/Error.h"
+
+using namespace llvm;
+using namespace llvm::orc;
+using namespace llvm::orc::shared;
+using namespace llvm::orc::rt_bootstrap;
+
+TEST(SharedMemoryMapperTest, MemReserveFinalizeDealloc) {
+  auto SelfEPC = cantFail(SelfExecutorProcessControl::Create());
+
+  ExecutorSharedMemoryManager MemMgr;
+
+  SharedMemoryMapper::SymbolAddrs SAs;
+  {
+    StringMap<ExecutorAddr> Map;
+    MemMgr.addBootstrapSymbols(Map);
+    SAs.Manager = Map[rt::ExecutorSharedMemoryManagerInstanceName];
+    SAs.Reserve = Map[rt::ExecutorSharedMemoryManagerReserveWrapperName];
+    SAs.Finalize = Map[rt::ExecutorSharedMemoryManagerFinalizeWrapperName];
+    SAs.Deallocate = Map[rt::ExecutorSharedMemoryManagerDeallocateWrapperName];
+  }
+
+  // barrier
+  std::promise<void> RP;
+  auto RF = RP.get_future();
+
+  {
+    size_t ReqSize = 4096 * 4;
+    auto Mapper = std::make_unique<SharedMemoryMapper>(*SelfEPC, SAs);
+    Mapper->reserve(ReqSize, [Mapper = std::move(Mapper), RP = std::move(RP),
+                              ReqSize](
+                                 Expected<MemoryMapper::Mapping> M) mutable {
+      EXPECT_THAT_ERROR(M.takeError(), Succeeded());
+      EXPECT_GE(M->Size, ReqSize);
+
+      tpctypes::SharedMemoryFinalizeRequest FR;
+      FR.Segments.push_back(tpctypes::SharedMemorySegFinalizeRequest{
+          tpctypes::WPF_Read | tpctypes::WPF_Write,
+          M->RemoteAddr,
+          ReqSize,
+      });
+
+      Mapper->finalize(*M, FR,
+                       [Mapper = std::move(Mapper), RP = std::move(RP),
+                        M = std::move(*M)](Error Err) mutable {
+                         EXPECT_THAT_ERROR(std::move(Err), Succeeded());
+
+                         std::vector<MemoryMapper::Mapping> Blocks = {M};
+                         auto Res = Mapper->release(
+                             Blocks, [RP = std::move(RP)](Error Err) mutable {
+                               EXPECT_THAT_ERROR(std::move(Err), Succeeded());
+                               RP.set_value();
+                             });
+
+                         // Unmapping from controller process succeeded
+                         EXPECT_THAT_ERROR(std::move(Res), Succeeded());
+                       });
+    });
+  }
+
+  RF.wait();
+
+  EXPECT_THAT_ERROR(MemMgr.shutdown(), Succeeded());
+
+  cantFail(SelfEPC->disconnect());
+}