[OpenMP][Libomptarget] Introduce Remote Offloading Plugin

[atmnpatel]

Summary

This introduces a remote offloading plugin for libomptarget. This
implementation relies on gRPC and protobuf, so this library will only
build if both libraries are available on the system. The corresponding
server is compiled to openmp-offloading-server.

This is a large change, but the only way to split this up is into RTL/server
but I fear that could introduce an inconsistency amongst them.

Ideally, tests for this should be added to the current ones that but that is
problematic for at least one reason. Given that libomptarget registers plugin
on a first-come-first-serve basis, if we wanted to offload onto a local x86
through a different process, then we'd have to either re-order the plugin list
in rtl.cpp (which is what I did locally for testing) or find a better
solution for runtime plugin registration in libomptarget.

[tschuett]

clangd has also machinery for detecting grpc:
https://reviews.llvm.org/D77794

[jdoerfert]

openmp/libomptarget/plugins/remote/src/Client.h

183

Remove these /// things all over the place.

openmp/libomptarget/plugins/remote/src/rtl.cpp

24

I think you can just include common/elf_common/elf_common.h now

[tianshilei1992]

openmp/libomptarget/plugins/remote/src/rtl.cpp

24

#include "elf_common.h" instead, and then link to elf_common in CMake.

[JonChesterfield]

As a proof of concept, I like this. It shows roughly the plumbing needed to launch a target region on a different machine to the one that is running the main process which clearly has serious potential as a programming model.

Review gets rapidly less detailed after the first few comments. Overall this is a bit weird stylistically but the broad pattern of how the client and server fit together looks as expected.

I don't think any of this is thread safe, which is bad as the plugin can definitely be called from multiple threads on the host.

There are no tests which is unfortunate as there's a lot of subtle control flow in the patch and a lot of ways RPC can go wrong. The documentation should probably mention that 'printf' will output on the server.

Given the lack of testing, and some general scepticism about how robustly this handles things going wrong, I think this should be guarded by a cmake variable. LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN or similar.

It looks from the valid binary test like this will claim to be a working plugin for most binaries if the server is running. The libomptarget logic for choosing a plugin picks the first plugin that claims to work. I think that will need to be refined to take multiple valid plugins into account, and perhaps surface a user interface for it.

openmp/libomptarget/plugins/remote/include/openmp.proto

42

Would probably go with I32 and I64 (or Int32, Int64) instead of Int and Long, as some platforms expect long to be 32 bits

openmp/libomptarget/plugins/remote/lib/Utils.cpp

69

std::map on a void* is a little strange - it's an ordered tree structure, so will probably be sorting by address. unordered_map?

72

malloc is a bit of a hazard in c++. This looks like it's missing a placement new.

(*Desc). to Desc-> ?

83

Not sure about this loop, seems to be at least one too many iterator. Maybe clearer as a for (size_t i = 0; ...) style one.

openmp/libomptarget/plugins/remote/src/Client.cpp

58

This is a bit sketchy. Try again on failure, when the failure cause is unknown, is not very robust. Bounding the number of events by an environment variable seems reasonable though.

In particular, it means the functions passed must all be safe to run multiple times, as the server might have run it and then fallen over, and that's not clear from the interface.

openmp/libomptarget/plugins/remote/src/rtl.cpp

31

This setup looks similar to some of the code above, perhaps worth putting in a common header

[atmnpatel]

Updates.

[atmnpatel]

In D95314#2521637, @JonChesterfield wrote:

I don't think any of this is thread safe, which is bad as the plugin can definitely be called from multiple threads on the host.

Yep, it is not designed for that and I added a warning in the documentation.

The documentation should probably mention that 'printf' will output on the server.

I wrapped it so its only visible during debug/info.

Given the lack of testing, and some general scepticism about how robustly this handles things going wrong, I think this should be guarded by a cmake variable. LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN or similar.

Agreed and Done.

It looks from the valid binary test like this will claim to be a working plugin for most binaries if the server is running. The libomptarget logic for choosing a plugin picks the first plugin that claims to work. I think that will need to be refined to take multiple valid plugins into account, and perhaps surface a user interface for it.

Yessir, ideally we get a config file at runtime that has priorities/a more explicit fallback mechanism.

@tschuett Thanks for pointing that out! I forgot to check for existing gRPC/protobuf cmake machinery. At this point however, it seems like we'd have to include clang's cmake infrastructure into openmp which seems like a decision that should be made carefully, so I'm going to leave it this way for now.

openmp/libomptarget/plugins/remote/src/Client.cpp

58

I removed it for now, it's not necessary for a MVP.

[jdoerfert]

A few nits but given the opt-in semantics now I don't see a reason not to include this plugin (as we included other plugins before).

openmp/docs/SupportAndFAQ.rst

119 ↗

(On Diff #319206)

I think we can use a generated link here. :ref:... maybe with a custom anchor in front of the remote offloading section .. _remote_offloading:

openmp/docs/design/Runtimes.rst

321 ↗

(On Diff #319206)

In the beginning, say that any devices of the remote host will be exposed to the application as if they were local devices. That is you can offload to the host or GPUs of the remote machine using the appropriate device number. If the remote host is the host, you basically see every device twice, once native and through the remote interface.

openmp/libomptarget/plugins/remote/include/Utils.h

15

Address the clang tidy warning plz

48

Avoid a return type like this and prefer a struct with named members. Also consider passing such a struct as reference to avoid dynamic allocations.

98

Can we avoid the global namespace here? What if we put everything in the remoteoffloading namespace?

openmp/libomptarget/plugins/remote/lib/Utils.cpp

276

This is magic, add a comment at the top.

[JonChesterfield]

I'm happy with this. It's a work in progress, but clearly marked experimental. My hope is it'll facilitate experimentation in this area.

It's plausible that this will spawn various uses. I'd like to run a test suite across multiple different architectures from a central point. Some people may prefer it to openmpi. If it gains traction, we can implement a robust RFC subsystem and appropriate application side config.

[jdoerfert]

LG, with the nits I mentioned before.

[atmnpatel]

Updates.

[atmnpatel]

Most of these clang-tidy errors are from a mismatch between gRPC naming conventions and ours, I'm going to keep it as the gRPC ones because that's what we do elsewhere (in clangd for instance) unless there are any objections.

openmp/libomptarget/plugins/CMakeLists.txt

 endmacro()
 
 add_subdirectory(aarch64)
 add_subdirectory(cuda)
 add_subdirectory(ppc64)
 add_subdirectory(ppc64le)
 add_subdirectory(ve)
 add_subdirectory(x86_64)
+add_subdirectory(remote)
 
 # Make sure the parent scope can see the plugins that will be created.
 set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS}" PARENT_SCOPE)
 set(LIBOMPTARGET_TESTED_PLUGINS "${LIBOMPTARGET_TESTED_PLUGINS}" PARENT_SCOPE)
 

openmp/libomptarget/plugins/exports

global:
     __tgt_rtl_data_exchange;
     __tgt_rtl_data_exchange_async;
     __tgt_rtl_data_delete;
     __tgt_rtl_run_target_team_region;
     __tgt_rtl_run_target_team_region_async;
     __tgt_rtl_run_target_region;
     __tgt_rtl_run_target_region_async;
     __tgt_rtl_synchronize;
+    __tgt_rtl_register_lib;
+    __tgt_rtl_unregister_lib;
   local:
     *;
 };

openmp/libomptarget/plugins/remote/CMakeLists.txt

+##===----------------------------------------------------------------------===##
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+##===----------------------------------------------------------------------===##
+#
+# Build a plugin (client) and server for remote offloading.
+#
+##===----------------------------------------------------------------------===#
+
+cmake_minimum_required(VERSION 3.15)
+
+if (NOT(CMAKE_SYSTEM_NAME MATCHES "Linux"))
+  libomptarget_say("Not building remote offloading plugin: only support Linux hosts.")
+  return()
+endif()
+
+find_package(Protobuf)
+find_package(gRPC CONFIG)
+
+find_program(PROTOC protoc)
+find_program(GRPC_CPP_PLUGIN grpc_cpp_plugin)
+
+if (Protobuf_FOUND AND gRPC_FOUND AND PROTOC AND GRPC_CPP_PLUGIN)
+  libomptarget_say("Building remote offloading plugin.")
+  set(directory "${CMAKE_BINARY_DIR}/include/openmp/libomptarget/plugins/remote/")
+  file(MAKE_DIRECTORY ${directory})
+  execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory ${directory})
+  execute_process(
+          COMMAND protoc --cpp_out=${directory} -I ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/openmp.proto
+          COMMAND protoc --grpc_out=${directory} -I ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/openmp.proto --plugin=protoc-gen-grpc=${GRPC_CPP_PLUGIN}
+  )
+
+  set(GRPC_SRC_FILES
+      ${directory}/openmp.grpc.pb.cc
+      ${directory}/openmp.pb.cc
+  )
+
+  set(GRPC_INCLUDE_DIR
+      ${directory}
+  )
+else()
+  libomptarget_say("Not building remote offloading plugin: required libraries were not found.")
+endif()
+
+set(RPC_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/include/)
+set(RPC_SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR}/lib/)
+
+add_subdirectory(src)
+add_subdirectory(server)

openmp/libomptarget/plugins/remote/include/Utils.h

+//===----------------- Utils.h - Utilities for Remote RTL -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Utilities for data transfer through protobuf and debugging.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef INCLUDE_UTILS_H_
+#define INCLUDE_UTILS_H_
+
+#include "Debug.h"
+#include "omptarget.h"
+#include "openmp.grpc.pb.h"
+#include "openmp.pb.h"
+#include "rtl.h"
+#include <string>
+
+#define CLIENT_DBG(...)                                                        \
+  {                                                                            \
+    if (DebugLevel > 0) {                                                      \
+      fprintf(stderr, "[[Client]] --> ");                                      \
+      fprintf(stderr, __VA_ARGS__);                                            \
+      fprintf(stderr, "\n");                                                   \
+    }                                                                          \
+  }
+
+#define SERVER_DBG(...)                                                        \
+  {                                                                            \
+    if (DebugLevel > 0) {                                                      \
+      fprintf(stderr, "[[Server]] --> ");                                      \
+      fprintf(stderr, __VA_ARGS__);                                            \
+      fprintf(stderr, "\n");                                                   \
+    }                                                                          \
+  }
+
+using remoteoffloading::DeviceOffloadEntry;
+using remoteoffloading::TargetBinaryDescription;
+using remoteoffloading::TargetOffloadEntry;
+using remoteoffloading::TargetTable;
+
+/// Loads a target binary description into protobuf.
+void loadTargetBinaryDescription(const __tgt_bin_desc *Desc,
+                                 TargetBinaryDescription &Request);
+
+/// Unload a target binary description from protobuf. The map is used to keep
+/// track of already copied device images.
+void unloadTargetBinaryDescription(
+    const TargetBinaryDescription *Request, __tgt_bin_desc *Desc,
+    std::map<const void *, __tgt_device_image *> &HostToRemoteDeviceImage);
+
+/// Frees argument as constructed by loadTargetBinaryDescription
+void freeTargetBinaryDescription(__tgt_bin_desc *Desc);
+
+/// Copies from TargetOffloadEntry protobuf to a tgt_bin_desc during unloading.
+void copyOffloadEntry(const TargetOffloadEntry &EntryResponse,
+                      __tgt_offload_entry *Entry);
+
+/// Copies from tgt_bin_desc into TargetOffloadEntry protobuf during loading.
+void copyOffloadEntry(const __tgt_offload_entry *Entry,
+                      TargetOffloadEntry *EntryResponse);
+
+/// Shallow copy of offload entry from tgt_bin_desc to TargetOffloadEntry
+/// during loading.
+void shallowCopyOffloadEntry(const __tgt_offload_entry *Entry,
+                             TargetOffloadEntry *EntryResponse);
+
+/// Copies DeviceOffloadEntries into table during unloading.
+void copyOffloadEntry(const DeviceOffloadEntry &EntryResponse,
+                      __tgt_offload_entry *Entry);
+
+/// Loads tgt_target_table into a TargetTable protobuf message.
+void loadTargetTable(__tgt_target_table *Table, TargetTable &TableResponse,
+                     __tgt_device_image *Image);
+
+/// Unloads from a target_table from protobuf.
+void unloadTargetTable(TargetTable &TableResponse, __tgt_target_table *Table,
+                       std::map<void *, void *> &HostToRemoteTargetTableMap);
+
+/// Frees argument as constructed by unloadTargetTable
+void freeTargetTable(__tgt_target_table *Table);
+
+void dump(const void *Start, const void *End);
+void dump(__tgt_offload_entry *Entry);
+void dump(TargetOffloadEntry Entry);
+void dump(__tgt_target_table *Table);
+void dump(__tgt_device_image *Image);
+
+#endif

openmp/libomptarget/plugins/remote/include/openmp.proto

+syntax = "proto3";
+
+package remoteoffloading;
+option cc_enable_arenas = true;
+
+service RemoteOffload {
+  rpc Shutdown(Null) returns (Int) {}
+
+  rpc RegisterLib(TargetBinaryDescription) returns (Int) {}
+  rpc UnregisterLib(Pointer) returns (Int) {}
+
+  rpc IsValidBinary(TargetDeviceImagePtr) returns (Int) {}
+  rpc GetNumberOfDevices(Null) returns (Int) {}
+
+  rpc InitDevice(Int) returns (Int) {}
+  rpc InitRequires(Long) returns (Int) {}
+
+  rpc LoadBinary(Binary) returns (TargetTable) {}
+  rpc Synchronize(SynchronizeDevice) returns (Int) {}
+
+  rpc DataAlloc(AllocData) returns (Pointer) {}
+  rpc DataDelete(DeleteData) returns (Int) {}
+
+  rpc DataSubmitAsync(stream SubmitDataAsync) returns (Int) {}
+  rpc DataRetrieveAsync(RetrieveDataAsync) returns (stream Data) {}
+
+  rpc IsDataExchangeable(DevicePair) returns (Int) {}
+  rpc DataExchangeAsync(ExchangeDataAsync) returns (Int) {}
+
+  rpc RunTargetRegionAsync(TargetRegionAsync) returns (Int) {}
+  rpc RunTargetTeamRegionAsync(TargetTeamRegionAsync) returns (Int) {}
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+message Null {}
+
+message Pointer { uint64 number = 1; }
+
+message Int { int32 number = 1; }
+
+message Long { int64 number = 1; }
+
+message DevicePair {
+  int32 src_dev_id = 1;
+  int32 dst_dev_id = 2;
+}
+
+message Binary {
+  uint64 image_ptr = 1;
+  int32 device_id = 2;
+}
+
+message TargetOffloadEntry {
+  bytes data = 1;
+  string name = 2;
+  int32 flags = 3;
+  int32 reserved = 4;
+}
+
+message DeviceOffloadEntry {
+  string name = 1;
+  uint64 addr = 2;
+  int32 flags = 3;
+  int32 reserved = 4;
+  int32 size = 5;
+}
+
+message TargetTable {
+  repeated DeviceOffloadEntry entries = 1;
+  repeated uint64 entry_ptrs = 2;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+message TargetDeviceImagePtr {
+  uint64 image_ptr = 1;
+  repeated uint64 entry_ptrs = 2;
+}
+
+message TargetDeviceImage {
+  bytes binary = 1;
+  repeated TargetOffloadEntry entries = 2;
+}
+
+message ImagePtrs {
+  uint64 img_ptr = 1;
+  repeated uint64 entry_ptrs = 2;
+}
+
+message TargetBinaryDescription {
+  repeated ImagePtrs image_ptrs = 1;
+  repeated TargetOffloadEntry entries = 2;
+  repeated TargetDeviceImage images = 3;
+  repeated uint64 entry_ptrs = 4;
+  uint64 bin_ptr = 5;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+message SynchronizeDevice {
+  uint64 queue_ptr = 1;
+  int32 device_id = 2;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+message AllocData {
+  uint64 size = 1;
+  uint64 hst_ptr = 2;
+  int32 device_id = 3;
+}
+
+message SubmitDataAsync {
+  bytes data = 1;
+  uint64 hst_ptr = 2;
+  uint64 tgt_ptr = 3;
+  uint64 queue_ptr = 4;
+  uint64 start = 5;
+  uint64 size = 6;
+  int32 device_id = 7;
+}
+
+message RetrieveDataAsync {
+  uint64 hst_ptr = 1;
+  uint64 tgt_ptr = 2;
+  uint64 size = 3;
+  uint64 queue_ptr = 4;
+  int32 device_id = 5;
+}
+
+message Data {
+  bytes data = 1;
+  uint64 start = 2;
+  uint64 size = 3;
+  int32 ret = 4;
+}
+
+message ExchangeDataAsync {
+  uint64 src_dev_id = 1;
+  uint64 src_ptr = 2;
+  uint64 dst_dev_id = 3;
+  uint64 dst_ptr = 4;
+  uint64 queue_ptr = 5;
+  uint64 size = 6;
+}
+
+message DeleteData {
+  uint64 tgt_ptr = 1;
+  int32 device_id = 2;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+message TargetRegionAsync {
+  repeated uint64 tgt_args = 1;
+  repeated int64 tgt_offsets = 2;
+  uint64 tgt_entry_ptr = 3;
+  uint64 queue_ptr = 4;
+  int32 device_id = 5;
+  int32 arg_num = 6;
+}
+
+message TargetTeamRegionAsync {
+  repeated uint64 tgt_args = 1;
+  repeated int64 tgt_offsets = 2;
+  uint64 tgt_entry_ptr = 3;
+  uint64 loop_tripcount = 4;
+  uint64 queue_ptr = 5;
+  int32 device_id = 6;
+  int32 arg_num = 7;
+  int32 team_num = 8;
+  int32 thread_limit = 9;
+}

openmp/libomptarget/plugins/remote/lib/Utils.cpp

+//===---------------- Utils.cpp - Utilities for Remote RTL ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Utilities for data movement and debugging.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cstdlib>
+#include <string>
+
+#include "Utils.h"
+#include "omptarget.h"
+
+void loadTargetBinaryDescription(const __tgt_bin_desc *Desc,
+                                 TargetBinaryDescription &Request) {
+  // Keeps track of entries which have already been deep copied.
+  std::vector<void *> DeepCopiedEntryAddrs;
+
+  // Copy Global Offload Entries
+  for (auto *CurEntry = Desc->HostEntriesBegin;
+       CurEntry != Desc->HostEntriesEnd; CurEntry++) {
+    auto *NewEntry = Request.add_entries();
+    copyOffloadEntry(CurEntry, NewEntry);
+
+    // Copy the pointer of the offload entry of the image into the Request
+    Request.add_entry_ptrs((uint64_t)CurEntry);
+    DeepCopiedEntryAddrs.push_back(CurEntry);
+  }
+
+  // Copy Device Images and Device Offload Entries
+  __tgt_device_image *CurImage = Desc->DeviceImages;
+  for (auto I = 0; I < Desc->NumDeviceImages; I++, CurImage++) {
+    auto *Image = Request.add_images();
+    auto Size = (char *)CurImage->ImageEnd - (char *)CurImage->ImageStart;
+    Image->set_binary(CurImage->ImageStart, Size);
+
+    // Copy the pointer of the image into the Request
+    auto *NewImagePtr = Request.add_image_ptrs();
+    NewImagePtr->set_img_ptr((uint64_t)CurImage->ImageStart);
+
+    // Copy Device Offload Entries
+    for (auto *CurEntry = CurImage->EntriesBegin;
+         CurEntry != CurImage->EntriesEnd; CurEntry++) {
+      auto *NewEntry = Image->add_entries();
+
+      auto Entry = std::find(DeepCopiedEntryAddrs.begin(),
+                             DeepCopiedEntryAddrs.end(), CurEntry);
+      if (Entry != DeepCopiedEntryAddrs.end()) {
+        // Offload entry has already been loaded
+        shallowCopyOffloadEntry(CurEntry, NewEntry);
+      } else { // Offload Entry has not been loaded into the Request
+        copyOffloadEntry(CurEntry, NewEntry);
+        DeepCopiedEntryAddrs.push_back(CurEntry);
+      }
+
+      // Copy the pointer of the offload entry of the image into the Request
+      NewImagePtr->add_entry_ptrs((uint64_t)CurEntry);
+    }
+  }
+}
+
+void unloadTargetBinaryDescription(
+    const TargetBinaryDescription *Request, __tgt_bin_desc *Desc,
+    std::map<const void *, __tgt_device_image *> &HostToRemoteDeviceImage) {
+  std::map<const void *, __tgt_offload_entry *> CopiedOffloadEntries;
+  (*Desc).NumDeviceImages = Request->images_size();
+  (*Desc).DeviceImages = (__tgt_device_image *)malloc(
+      sizeof(__tgt_device_image) * (*Desc).NumDeviceImages);
+
+  if (Request->entries_size())
+    (*Desc).HostEntriesBegin = (__tgt_offload_entry *)malloc(
+        sizeof(__tgt_offload_entry) * Request->entries_size());
+  else {
+    Desc->HostEntriesBegin = nullptr;
+    Desc->HostEntriesEnd = nullptr;
+  }
+
+  // Copy Global Offload Entries
+  __tgt_offload_entry *CurEntry = (*Desc).HostEntriesBegin;
+  const auto *EntryPtrItr = Request->entry_ptrs().begin();
+  for (auto Entry : Request->entries()) {
+    copyOffloadEntry(Entry, CurEntry);
+    CopiedOffloadEntries[(void *)*EntryPtrItr] = CurEntry;
+    CurEntry++;
+    EntryPtrItr++;
+  }
+  (*Desc).HostEntriesEnd = CurEntry;
+
+  // Copy Device Images and Device Offload Entries
+  __tgt_device_image *CurImage = (*Desc).DeviceImages;
+  auto ImageItr = Request->image_ptrs().begin();
+  for (auto Image : Request->images()) {
+    // Copy Device Offload Entries
+    auto *CurEntry = (*Desc).HostEntriesBegin;
+    const auto *EntryPtrItr = ImageItr->entry_ptrs().begin();
+    bool Found = false;
+
+    if (!Desc->HostEntriesBegin) {
+      CurImage->EntriesBegin = nullptr;
+      CurImage->EntriesEnd = nullptr;
+    }
+
+    for (auto Entry : Image.entries()) {
+      auto TgtEntry = CopiedOffloadEntries.find((void *)*EntryPtrItr);
+      if (TgtEntry != CopiedOffloadEntries.end()) {
+        if (!Found)
+          (*CurImage).EntriesBegin = CurEntry;
+
+        Found = true;
+        if (Found) {
+          (*CurImage).EntriesEnd = CurEntry + 1;
+        }
+      } else {
+        Found = false;
+        copyOffloadEntry(Entry, CurEntry);
+        CopiedOffloadEntries[static_cast<const void *>(EntryPtrItr)] = CurEntry;
+      }
+      CurEntry++;
+      EntryPtrItr++;
+    }
+
+    // Copy Device Image
+    CurImage->ImageStart = malloc(Image.binary().size());
+    memcpy(CurImage->ImageStart,
+           static_cast<const void *>(Image.binary().data()),
+           Image.binary().size());
+    CurImage->ImageEnd =
+        (void *)((char *)CurImage->ImageStart + Image.binary().size());
+
+    HostToRemoteDeviceImage[(void *)ImageItr->img_ptr()] = CurImage;
+    CurImage++;
+    ImageItr++;
+  }
+}
+
+void freeTargetBinaryDescription(__tgt_bin_desc *Desc) {
+  __tgt_device_image *CurImage = Desc->DeviceImages;
+  for (auto I = 0; I < Desc->NumDeviceImages; I++, CurImage++) {
+    free(CurImage);
+  }
+
+  for (auto *Entry = Desc->HostEntriesBegin; Entry != Desc->HostEntriesEnd;
+       Entry++) {
+    free(Entry->name);
+    free(Entry->addr);
+  }
+
+  free(Desc->HostEntriesBegin);
+  free(Desc);
+}
+
+void freeTargetTable(__tgt_target_table *Table) {
+  for (auto *Entry = Table->EntriesBegin; Entry != Table->EntriesEnd; Entry++) {
+    free(Entry->name);
+    free(Entry->addr);
+  }
+
+  free(Table->EntriesBegin);
+}
+
+void loadTargetTable(__tgt_target_table *Table, TargetTable &TableResponse,
+                     __tgt_device_image *Image) {
+  auto *ImageEntry = Image->EntriesBegin;
+  for (__tgt_offload_entry *CurEntry = Table->EntriesBegin;
+       CurEntry != Table->EntriesEnd; CurEntry++, ImageEntry++) {
+    // TODO: This can probably be trimmed substantially.
+    auto *NewEntry = TableResponse.add_entries();
+    NewEntry->set_name(CurEntry->name);
+    NewEntry->set_addr((uint64_t)CurEntry->addr);
+    NewEntry->set_flags(CurEntry->flags);
+    NewEntry->set_reserved(CurEntry->reserved);
+    NewEntry->set_size(CurEntry->size);
+    TableResponse.add_entry_ptrs((int64_t)CurEntry);
+  }
+}
+
+void unloadTargetTable(TargetTable &TableResponse, __tgt_target_table *Table,
+                       std::map<void *, void *> &HostToRemoteTargetTableMap) {
+  (*Table).EntriesBegin = (__tgt_offload_entry *)malloc(
+      sizeof(__tgt_offload_entry) * TableResponse.entries_size());
+
+  auto *CurEntry = (*Table).EntriesBegin;
+  const auto *EntryPtrItr = TableResponse.entry_ptrs().begin();
+  for (auto Entry : TableResponse.entries()) {
+    copyOffloadEntry(Entry, CurEntry);
+    HostToRemoteTargetTableMap[CurEntry->addr] = (void *)*EntryPtrItr;
+    CurEntry++;
+    EntryPtrItr++;
+  }
+  (*Table).EntriesEnd = CurEntry;
+}
+
+void copyOffloadEntry(const TargetOffloadEntry &EntryResponse,
+                      __tgt_offload_entry *Entry) {
+  Entry->name = strdup(EntryResponse.name().c_str());
+  Entry->reserved = EntryResponse.reserved();
+  Entry->flags = EntryResponse.flags();
+  Entry->addr = strdup(EntryResponse.data().c_str());
+  Entry->size = EntryResponse.data().size();
+}
+
+void copyOffloadEntry(const DeviceOffloadEntry &EntryResponse,
+                      __tgt_offload_entry *Entry) {
+  Entry->name = strdup(EntryResponse.name().c_str());
+  Entry->reserved = EntryResponse.reserved();
+  Entry->flags = EntryResponse.flags();
+  Entry->addr = (void *)EntryResponse.addr();
+  Entry->size = EntryResponse.size();
+}
+
+/// We shallow copy with just the name because it is a convenient identifier, we
+/// do actually just match off of the address.
+void shallowCopyOffloadEntry(const __tgt_offload_entry *Entry,
+                             TargetOffloadEntry *EntryResponse) {
+  EntryResponse->set_name(Entry->name);
+}
+
+void copyOffloadEntry(const __tgt_offload_entry *Entry,
+                      TargetOffloadEntry *EntryResponse) {
+  shallowCopyOffloadEntry(Entry, EntryResponse);
+  EntryResponse->set_reserved(Entry->reserved);
+  EntryResponse->set_flags(Entry->flags);
+  EntryResponse->set_data(Entry->addr, Entry->size);
+}
+
+void dump(const void *Start, const void *End) {
+  unsigned char Line[17];
+  const unsigned char *PrintCharacter = (const unsigned char *)Start;
+
+  unsigned int I = 0;
+  for (; I < ((const int *)End - (const int *)Start); I++) {
+    if ((I % 16) == 0) {
+      if (I != 0)
+        printf("  %s\n", Line);
+
+      printf("  %04x ", I);
+    }
+
+    printf(" %02x", PrintCharacter[I]);
+
+    if ((PrintCharacter[I] < 0x20) || (PrintCharacter[I] > 0x7e))
+      Line[I % 16] = '.';
+    else
+      Line[I % 16] = PrintCharacter[I];
+
+    Line[(I % 16) + 1] = '\0';
+  }
+
+  while ((I % 16) != 0) {
+    printf("   ");
+    I++;
+  }
+
+  printf("  %s\n", Line);
+}
+
+void dump(__tgt_offload_entry *Entry) {
+  fprintf(stderr, "Entry (%p):\n", (void *)Entry);
+  fprintf(stderr, "  Name: %s (%p)\n", Entry->name, (void *)&Entry->name);
+  fprintf(stderr, "  Reserved: %d (%p)\n", Entry->reserved,
+          (void *)&Entry->reserved);
+  fprintf(stderr, "  Flags: %d (%p)\n", Entry->flags, (void *)&Entry->flags);
+  fprintf(stderr, "  Addr: %p\n", Entry->addr);
+  fprintf(stderr, "  Size: %lu\n", Entry->size);
+}
+
+void dump(__tgt_target_table *Table) {
+  for (auto *CurEntry = Table->EntriesBegin; CurEntry != Table->EntriesEnd;
+       CurEntry++)
+    dump(CurEntry);
+}
+
+void dump(TargetOffloadEntry Entry) {
+  fprintf(stderr, "Entry: ");
+  fprintf(stderr, "    %s\n", Entry.name().c_str());
+  fprintf(stderr, "    %d\n", Entry.reserved());
+  fprintf(stderr, "    %d\n", Entry.flags());
+  fprintf(stderr, "    %ld\n", Entry.data().size());
+  dump(static_cast<const void *>(Entry.data().data()),
+       static_cast<const void *>((Entry.data().c_str() + Entry.data().size())));
+}
+
+void dump(__tgt_device_image *Image) {
+  dump(Image->ImageStart, Image->ImageEnd);
+  __tgt_offload_entry *EntryItr = Image->EntriesBegin;
+  for (; EntryItr != Image->EntriesEnd; EntryItr++)
+    dump(EntryItr);
+}
+
+void dump(std::map<void *, __tgt_offload_entry *> &Map) {
+  fprintf(stderr, "Host to Remote Entry Map:\n");
+  for (auto Entry : Map)
+    fprintf(stderr, "  Host (%p) -> Tgt (%p): Addr((%p))\n", Entry.first,
+            (void *)Entry.second, (void *)Entry.second->addr);
+}

openmp/libomptarget/plugins/remote/server/CMakeLists.txt

+##===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+##===----------------------------------------------------------------------===##
+#
+# Build server for remote offloading.
+#
+##===----------------------------------------------------------------------===##
+
+
+libomptarget_say("Building RPC offloading plugin - server")
+
+include_directories(${LIBOMPTARGET_DEP_LIBELF_INCLUDE_DIRS})
+include_directories(${LIBOMPTARGET_SRC_DIR})
+include_directories(${LIBOMPTARGET_INCLUDE_DIR})
+include_directories(${GRPC_INCLUDE_DIR})
+include_directories(${RPC_INCLUDE_DIR})
+
+add_executable(openmp-offloading-server
+        ${LIBOMPTARGET_SRC_FILES}
+        ${GRPC_SRC_FILES}
+        ${RPC_SRC_DIR}/Utils.cpp
+        Server.cpp
+        OffloadingServer.cpp
+)
+
+target_link_libraries(openmp-offloading-server
+        grpc++
+        protobuf
+        "-ldl" "-lomp" "-fopenmp" "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../../exports" ${LIBOMPTARGET_DEP_LIBELF_LIBRARIES})

openmp/libomptarget/plugins/remote/server/OffloadingServer.cpp

+//===------------- OffloadingServer.cpp - Server Application --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Offloading server for remote host.
+//
+//===----------------------------------------------------------------------===//
+
+#include <future>
+#include <grpcpp/server.h>
+#include <grpcpp/server_builder.h>
+#include <iostream>
+#include <thread>
+
+#include "Server.h"
+
+using grpc::Server;
+using grpc::ServerBuilder;
+
+std::promise<void> ShutdownPromise;
+
+int main() {
+  std::string ServerAddress = "0.0.0.0:50051";
+  if (const char *Env = std::getenv("RPC_ADDRESS"))
+    ServerAddress = Env;
+  uint64_t MaxSize = 1 << 30;
+  if (const char *Env = std::getenv("RPC_ALLOCATOR_MAX"))
+    MaxSize = std::stoi(Env);
+  uint64_t BlockSize = 1 << 20;
+  if (const char *Env = std::getenv("RPC_BLOCK_SIZE"))
+    BlockSize = std::stoi(Env);
+
+  RemoteOffloadImpl Service(MaxSize, BlockSize);
+
+  ServerBuilder Builder;
+  Builder.AddListeningPort(ServerAddress, grpc::InsecureServerCredentials());
+  Builder.RegisterService(&Service);
+  Builder.SetMaxMessageSize(INT_MAX);
+  std::unique_ptr<Server> Server(Builder.BuildAndStart());
+  std::cout << "Server listening on " << ServerAddress << std::endl;
+
+  auto WaitForServer = [&]() { Server->Wait(); };
+
+  std::thread ServerThread(WaitForServer);
+
+  auto ShutdownFuture = ShutdownPromise.get_future();
+  ShutdownFuture.wait();
+  Server->Shutdown();
+  ServerThread.join();
+
+  return 0;
+}

openmp/libomptarget/plugins/remote/server/Server.h

+//===-------------------------- Server.h - Server -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Offloading gRPC server for remote host.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SRC_SERVER_H_
+#define SRC_SERVER_H_
+
+#include <grpcpp/server_context.h>
+#include <map>
+
+#include "Utils.h"
+#include "device.h"
+#include "omptarget.h"
+#include "openmp.grpc.pb.h"
+#include "openmp.pb.h"
+#include "rtl.h"
+
+using grpc::ServerContext;
+using grpc::ServerReader;
+using grpc::ServerWriter;
+using grpc::Status;
+
+using remoteoffloading::RemoteOffload;
+
+using remoteoffloading::AllocData;
+using remoteoffloading::Binary;
+using remoteoffloading::Data;
+using remoteoffloading::DeleteData;
+using remoteoffloading::DevicePair;
+using remoteoffloading::ExchangeDataAsync;
+using remoteoffloading::Int;
+using remoteoffloading::Long;
+using remoteoffloading::Null;
+using remoteoffloading::Pointer;
+using remoteoffloading::RetrieveDataAsync;
+using remoteoffloading::SubmitDataAsync;
+using remoteoffloading::SynchronizeDevice;
+using remoteoffloading::TargetBinaryDescription;
+using remoteoffloading::TargetDeviceImagePtr;
+using remoteoffloading::TargetRegionAsync;
+using remoteoffloading::TargetTable;
+using remoteoffloading::TargetTeamRegionAsync;
+
+using namespace google;
+
+extern PluginManager *PM;
+
+class RemoteOffloadImpl final : public RemoteOffload::Service {
+private:
+  int32_t mapHostRTLDeviceId(int32_t RTLDeviceID);
+
+  std::map<const void *, __tgt_device_image *> HostToRemoteDeviceImage;
+  std::map<const void *, __tgt_offload_entry *> HostToRemoteOffloadEntry;
+  std::vector<std::pair<void *, __tgt_bin_desc *>> Descriptions;
+  __tgt_bin_desc *Desc = nullptr;
+  __tgt_target_table *Table = nullptr;
+
+  int DebugLevel;
+  uint64_t MaxSize;
+  uint64_t BlockSize;
+  std::unique_ptr<protobuf::Arena> Arena;
+
+public:
+  RemoteOffloadImpl(uint64_t MaxSize, uint64_t BlockSize)
+      : MaxSize(MaxSize), BlockSize(BlockSize) {
+    DebugLevel = getDebugLevel();
+    Arena = std::make_unique<protobuf::Arena>();
+  }
+
+  Status Shutdown(ServerContext *Context, const Null *Request,
+                  Int *Reply) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+  // Remote Registrations.
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status RegisterLib(ServerContext *Context,
+                     const TargetBinaryDescription *Description,
+                     Int *Reply) override;
+  Status UnregisterLib(ServerContext *Context, const Pointer *Request,
+                       Int *Reply) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status IsValidBinary(ServerContext *Context,
+                       const TargetDeviceImagePtr *Image,
+                       Int *IsValid) override;
+  Status GetNumberOfDevices(ServerContext *Context, const Null *Null,
+                            Int *NumberOfDevices) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status InitDevice(ServerContext *Context, const Int *DeviceNum,
+                    Int *Reply) override;
+  Status InitRequires(ServerContext *Context, const Long *RequiresFlag,
+                      Int *Reply) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status LoadBinary(ServerContext *Context, const Binary *Binary,
+                    TargetTable *Reply) override;
+  Status Synchronize(ServerContext *Context, const SynchronizeDevice *Info,
+                     Int *Reply) override;
+  Status IsDataExchangeable(ServerContext *Context, const DevicePair *Request,
+                            Int *Reply) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status DataAlloc(ServerContext *Context, const AllocData *Request,
+                   Pointer *Reply) override;
+
+  Status DataSubmitAsync(ServerContext *Context,
+                         ServerReader<SubmitDataAsync> *Reader,
+                         Int *Reply) override;
+  Status DataRetrieveAsync(ServerContext *Context,
+                           const RetrieveDataAsync *Request,
+                           ServerWriter<Data> *Writer) override;
+
+  Status DataExchangeAsync(ServerContext *Context,
+                           const ExchangeDataAsync *Request,
+                           Int *Reply) override;
+
+  Status DataDelete(ServerContext *Context, const DeleteData *Request,
+                    Int *Reply) override;
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  Status RunTargetRegionAsync(ServerContext *Context,
+                              const TargetRegionAsync *Request,
+                              Int *Reply) override;
+
+  Status RunTargetTeamRegionAsync(ServerContext *Context,
+                                  const TargetTeamRegionAsync *Request,
+                                  Int *Reply) override;
+};
+
+#endif

openmp/libomptarget/plugins/remote/server/Server.cpp

+//===----------------- Server.cpp - Server Implementation -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Offloading gRPC server for remote host.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cmath>
+#include <future>
+
+#include "Server.h"
+#include "omptarget.h"
+#include "openmp.grpc.pb.h"
+#include "openmp.pb.h"
+
+using grpc::WriteOptions;
+
+extern std::promise<void> ShutdownPromise;
+
+Status RemoteOffloadImpl::Shutdown(ServerContext *Context, const Null *Request,
+                                   Int *Reply) {
+  SERVER_DBG("Shutting down the server");
+
+  for (auto Desc : Descriptions)
+    freeTargetBinaryDescription(Desc.second);
+  Descriptions.clear();
+
+  Reply->set_number(0);
+  ShutdownPromise.set_value();
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Remote Registrations.
+///////////////////////////////////////////////////////////////////////////////
+
+Status
+RemoteOffloadImpl::RegisterLib(ServerContext *Context,
+                               const TargetBinaryDescription *Description,
+                               Int *Reply) {
+  SERVER_DBG("Registering library");
+
+  Desc = static_cast<__tgt_bin_desc *>(malloc(sizeof(__tgt_bin_desc)));
+
+  unloadTargetBinaryDescription(Description, Desc, HostToRemoteDeviceImage);
+  PM->RTLs.RegisterLib(Desc);
+
+  Descriptions.emplace_back((void *)Description->bin_ptr(), Desc);
+
+  SERVER_DBG("Registered library");
+  Reply->set_number(0);
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::UnregisterLib(ServerContext *Context,
+                                        const Pointer *Request, Int *Reply) {
+  SERVER_DBG("Unregistering library");
+
+  auto DescIt = std::find_if(Descriptions.begin(), Descriptions.end(),
+                             [Request](auto DescIt) {
+                               return DescIt.first == (void *)Request->number();
+                             });
+
+  if (DescIt == Descriptions.end()) {
+    Reply->set_number(1);
+    return Status::OK;
+  }
+
+  PM->RTLs.UnregisterLib(DescIt->second);
+
+  SERVER_DBG("Unregistered library");
+  Reply->set_number(0);
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+Status RemoteOffloadImpl::IsValidBinary(ServerContext *Context,
+                                        const TargetDeviceImagePtr *DeviceImage,
+                                        Int *IsValid) {
+  SERVER_DBG("Checking if binary (%p) is valid",
+             (void *)(DeviceImage->image_ptr()));
+
+  __tgt_device_image *Image =
+      HostToRemoteDeviceImage[(void *)DeviceImage->image_ptr()];
+
+  IsValid->set_number(0);
+
+  for (auto &RTL : PM->RTLs.AllRTLs)
+    if (auto Ret = RTL.is_valid_binary(Image)) {
+      IsValid->set_number(Ret);
+      break;
+    }
+
+  SERVER_DBG("Checked if binary (%p) is valid",
+             (void *)(DeviceImage->image_ptr()));
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::GetNumberOfDevices(ServerContext *Context,
+                                             const Null *Null,
+                                             Int *NumberOfDevices) {
+  SERVER_DBG("Getting number of devices");
+  std::call_once(PM->RTLs.initFlag, &RTLsTy::LoadRTLs, &PM->RTLs);
+
+  int32_t Devices = 0;
+  PM->RTLsMtx.lock();
+  for (auto &RTL : PM->RTLs.AllRTLs)
+    Devices += RTL.NumberOfDevices;
+  PM->RTLsMtx.unlock();
+
+  NumberOfDevices->set_number(Devices);
+
+  SERVER_DBG("Got number of devices");
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+Status RemoteOffloadImpl::InitDevice(ServerContext *Context,
+                                     const Int *DeviceNum, Int *Reply) {
+  SERVER_DBG("Initializing device %d", DeviceNum->number());
+
+  Reply->set_number(PM->Devices[DeviceNum->number()].RTL->init_device(
+      mapHostRTLDeviceId(DeviceNum->number())));
+
+  SERVER_DBG("Initialized device %d", DeviceNum->number());
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::InitRequires(ServerContext *Context,
+                                       const Long *RequiresFlag, Int *Reply) {
+  SERVER_DBG("Initializing requires for devices");
+
+  for (auto &Device : PM->Devices)
+    if (Device.RTL->init_requires)
+      Device.RTL->init_requires(RequiresFlag->number());
+  Reply->set_number(RequiresFlag->number());
+
+  SERVER_DBG("Initialized requires for devices");
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+Status RemoteOffloadImpl::LoadBinary(ServerContext *Context,
+                                     const Binary *Binary, TargetTable *Reply) {
+  SERVER_DBG("Loading binary (%p) to device %d", (void *)Binary->image_ptr(),
+             Binary->device_id());
+
+  __tgt_device_image *Image =
+      HostToRemoteDeviceImage[(void *)Binary->image_ptr()];
+
+  Table = PM->Devices[Binary->device_id()].RTL->load_binary(
+      mapHostRTLDeviceId(Binary->device_id()), Image);
+  if (Table)
+    loadTargetTable(Table, *Reply, Image);
+
+  SERVER_DBG("Loaded binary (%p) to device %d", (void *)Binary->image_ptr(),
+             Binary->device_id());
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::Synchronize(ServerContext *Context,
+                                      const SynchronizeDevice *Info,
+                                      Int *Reply) {
+  SERVER_DBG("Synchronizing device %d (probably won't work)",
+             Info->device_id());
+
+  void *AsyncInfoPtr = (void *)Info->queue_ptr();
+  Reply->set_number(0);
+  if (PM->Devices[Info->device_id()].RTL->synchronize)
+    Reply->set_number(PM->Devices[Info->device_id()].synchronize(
+        (__tgt_async_info *)AsyncInfoPtr));
+
+  SERVER_DBG("Synchronized device %d", Info->device_id());
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::IsDataExchangeable(ServerContext *Context,
+                                             const DevicePair *Request,
+                                             Int *Reply) {
+  SERVER_DBG("Checking if data exchangable between device %d and device %d",
+             Request->src_dev_id(), Request->dst_dev_id());
+
+  Reply->set_number(-1);
+  if (PM->Devices[mapHostRTLDeviceId(Request->src_dev_id())]
+          .RTL->is_data_exchangable)
+    Reply->set_number(PM->Devices[mapHostRTLDeviceId(Request->src_dev_id())]
+                          .RTL->is_data_exchangable(Request->src_dev_id(),
+                                                    Request->dst_dev_id()));
+
+  SERVER_DBG("Checked if data exchangable between device %d and device %d",
+             Request->src_dev_id(), Request->dst_dev_id());
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+Status RemoteOffloadImpl::DataAlloc(ServerContext *Context,
+                                    const AllocData *Request, Pointer *Reply) {
+  SERVER_DBG("Allocating %ld bytes on sevice %d", Request->size(),
+             Request->device_id());
+
+  uint64_t TgtPtr = (uint64_t)PM->Devices[Request->device_id()].RTL->data_alloc(
+      mapHostRTLDeviceId(Request->device_id()), Request->size(),
+      (void *)Request->hst_ptr());
+  Reply->set_number(TgtPtr);
+
+  SERVER_DBG("Allocated at " DPxMOD "", DPxPTR((void *)TgtPtr));
+
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::DataSubmitAsync(ServerContext *Context,
+                                          ServerReader<SubmitDataAsync> *Reader,
+                                          Int *Reply) {
+  SubmitDataAsync Request;
+  char *HostCopy = nullptr;
+  while (Reader->Read(&Request)) {
+    if (Request.start() == 0 && Request.size() == Request.data().size()) {
+      SERVER_DBG("Submitting %lu bytes async to (%p) on device %d",
+                 Request.data().size(), (void *)Request.tgt_ptr(),
+                 Request.device_id());
+
+      SERVER_DBG("  Host Pointer Info: %p, %p", (void *)Request.hst_ptr(),
+                 static_cast<const void *>(Request.data().data()));
+
+      Reader->SendInitialMetadata();
+
+      Reply->set_number(PM->Devices[Request.device_id()].RTL->data_submit(
+          mapHostRTLDeviceId(Request.device_id()), (void *)Request.tgt_ptr(),
+          (void *)Request.data().data(), Request.data().size()));
+
+      SERVER_DBG("Submitted %lu bytes async to (%p) on device %d",
+                 Request.data().size(), (void *)Request.tgt_ptr(),
+                 Request.device_id());
+
+      return Status::OK;
+    }
+    if (!HostCopy) {
+      HostCopy = (char *)malloc(sizeof(char) * Request.size());
+      Reader->SendInitialMetadata();
+    }
+
+    SERVER_DBG("Submitting %lu-%lu/%lu bytes async to (%p) on device %d",
+               Request.start(), Request.start() + Request.data().size(),
+               Request.size(), (void *)Request.tgt_ptr(), Request.device_id());
+
+    memcpy((void *)((char *)HostCopy + Request.start()), Request.data().data(),
+           Request.data().size());
+  }
+  SERVER_DBG("  Host Pointer Info: %p, %p", (void *)Request.hst_ptr(),
+             static_cast<const void *>(Request.data().data()));
+
+  Reply->set_number(PM->Devices[Request.device_id()].RTL->data_submit(
+      mapHostRTLDeviceId(Request.device_id()), (void *)Request.tgt_ptr(),
+      HostCopy, Request.size()));
+
+  free(HostCopy);
+
+  SERVER_DBG("Submitted %lu bytes to (%p) on device %d", Request.data().size(),
+             (void *)Request.tgt_ptr(), Request.device_id());
+
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::DataRetrieveAsync(ServerContext *Context,
+                                            const RetrieveDataAsync *Request,
+                                            ServerWriter<Data> *Writer) {
+  auto *HstPtr = malloc(sizeof(char) * Request->size());
+  auto Ret = PM->Devices[Request->device_id()].RTL->data_retrieve(
+      mapHostRTLDeviceId(Request->device_id()), HstPtr,
+      (void *)Request->tgt_ptr(), Request->size());
+
+  if (Arena->SpaceUsed() >= MaxSize)
+    Arena->Reset();
+
+  if (Request->size() > BlockSize) {
+    uint64_t Start = 0, End = BlockSize;
+    for (auto I = 0; I < ceil((float)Request->size() / BlockSize); I++) {
+      auto *Reply = protobuf::Arena::CreateMessage<Data>(Arena.get());
+
+      Reply->set_start(Start);
+      Reply->set_size(Request->size());
+      Reply->set_data((char *)HstPtr + Start, End - Start);
+      Reply->set_ret(Ret);
+
+      SERVER_DBG("Retrieving %lu-%lu/%lu bytes from (%p) on device %d", Start,
+                 End, Request->size(), (void *)Request->tgt_ptr(),
+                 mapHostRTLDeviceId(Request->device_id()));
+
+      if (!Writer->Write(*Reply)) {
+        CLIENT_DBG("Broken stream when submitting data");
+      }
+
+      SERVER_DBG("Retrieved %lu-%lu/%lu bytes from (%p) on device %d", Start,
+                 End, Request->size(), (void *)Request->tgt_ptr(),
+                 mapHostRTLDeviceId(Request->device_id()));
+
+      Start += BlockSize;
+      End += BlockSize;
+      if (End >= Request->size())
+        End = Request->size();
+    }
+  } else {
+    auto *Reply = protobuf::Arena::CreateMessage<Data>(Arena.get());
+
+    SERVER_DBG("Retrieve %lu bytes from (%p) on device %d", Request->size(),
+               (void *)Request->tgt_ptr(),
+               mapHostRTLDeviceId(Request->device_id()));
+
+    Reply->set_start(0);
+    Reply->set_size(Request->size());
+    Reply->set_data((char *)HstPtr, Request->size());
+    Reply->set_ret(Ret);
+
+    SERVER_DBG("Retrieved %lu bytes from (%p) on device %d", Request->size(),
+               (void *)Request->tgt_ptr(),
+               mapHostRTLDeviceId(Request->device_id()));
+
+    Writer->WriteLast(*Reply, WriteOptions());
+  }
+
+  free(HstPtr);
+
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::DataExchangeAsync(ServerContext *Context,
+                                            const ExchangeDataAsync *Request,
+                                            Int *Reply) {
+  SERVER_DBG(
+      "Exchanging data asynchronously from device %d (%p) to device %d (%p) of "
+      "size %lu",
+      mapHostRTLDeviceId(Request->src_dev_id()), (void *)Request->src_ptr(),
+      mapHostRTLDeviceId(Request->dst_dev_id()), (void *)Request->dst_ptr(),
+      Request->size());
+
+  if (PM->Devices[Request->src_dev_id()].RTL->data_exchange) {
+    int32_t Ret = PM->Devices[Request->src_dev_id()].RTL->data_exchange(
+        mapHostRTLDeviceId(Request->src_dev_id()), (void *)Request->src_ptr(),
+        mapHostRTLDeviceId(Request->dst_dev_id()), (void *)Request->dst_ptr(),
+        Request->size());
+    Reply->set_number(Ret);
+  } else
+    Reply->set_number(-1);
+
+  SERVER_DBG(
+      "Exchanged data asynchronously from device %d (%p) to device %d (%p) of "
+      "size %lu",
+      mapHostRTLDeviceId(Request->src_dev_id()), (void *)Request->src_ptr(),
+      mapHostRTLDeviceId(Request->dst_dev_id()), (void *)Request->dst_ptr(),
+      Request->size());
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::DataDelete(ServerContext *Context,
+                                     const DeleteData *Request, Int *Reply) {
+  SERVER_DBG("Deleting data from (%p) on device %d", (void *)Request->tgt_ptr(),
+             mapHostRTLDeviceId(Request->device_id()));
+
+  auto Ret = PM->Devices[Request->device_id()].RTL->data_delete(
+      mapHostRTLDeviceId(Request->device_id()), (void *)Request->tgt_ptr());
+  Reply->set_number(Ret);
+
+  SERVER_DBG("Deleted data from (%p) on device %d", (void *)Request->tgt_ptr(),
+             mapHostRTLDeviceId(Request->device_id()));
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+Status RemoteOffloadImpl::RunTargetRegionAsync(ServerContext *Context,
+                                               const TargetRegionAsync *Request,
+                                               Int *Reply) {
+  SERVER_DBG("Running TargetRegionAsync on device %d with %d args",
+             mapHostRTLDeviceId(Request->device_id()), Request->arg_num());
+
+  char **TgtArgs = (char **)malloc(sizeof(char *) * Request->arg_num());
+  for (auto I = 0; I < Request->arg_num(); I++) {
+    TgtArgs[I] = (char *)malloc(sizeof(void *));
+    TgtArgs[I] = (char *)Request->tgt_args()[I];
+  }
+
+  ptrdiff_t *TgtOffsets =
+      (ptrdiff_t *)malloc(sizeof(ptrdiff_t) * Request->arg_num());
+  const auto *TgtOffsetItr = Request->tgt_offsets().begin();
+  for (auto I = 0; I < Request->arg_num(); I++, TgtOffsetItr++)
+    TgtOffsets[I] = (ptrdiff_t)*TgtOffsetItr;
+
+  void *TgtEntryPtr = ((__tgt_offload_entry *)Request->tgt_entry_ptr())->addr;
+
+  int32_t Ret = PM->Devices[Request->device_id()].RTL->run_region(
+      mapHostRTLDeviceId(Request->device_id()), TgtEntryPtr, (void **)TgtArgs,
+      TgtOffsets, Request->arg_num());
+
+  Reply->set_number(Ret);
+
+  free(TgtArgs);
+  free(TgtOffsets);
+
+  SERVER_DBG("Ran TargetRegionAsync on device %d with %d args",
+             mapHostRTLDeviceId(Request->device_id()), Request->arg_num());
+  return Status::OK;
+}
+
+Status RemoteOffloadImpl::RunTargetTeamRegionAsync(
+    ServerContext *Context, const TargetTeamRegionAsync *Request, Int *Reply) {
+  SERVER_DBG("Running TargetTeamRegionAsync on device %d with %d args",
+             mapHostRTLDeviceId(Request->device_id()), Request->arg_num());
+
+  char **TgtArgs = (char **)malloc(sizeof(char *) * Request->arg_num());
+  for (auto I = 0; I < Request->arg_num(); I++) {
+    TgtArgs[I] = (char *)malloc(sizeof(void *));
+    TgtArgs[I] = (char *)Request->tgt_args()[I];
+  }
+
+  ptrdiff_t *TgtOffsets =
+      (ptrdiff_t *)malloc(sizeof(ptrdiff_t) * Request->arg_num());
+  const auto *TgtOffsetItr = Request->tgt_offsets().begin();
+  for (auto I = 0; I < Request->arg_num(); I++, TgtOffsetItr++)
+    TgtOffsets[I] = (ptrdiff_t)*TgtOffsetItr;
+
+  void *TgtEntryPtr = ((__tgt_offload_entry *)Request->tgt_entry_ptr())->addr;
+  int32_t Ret = PM->Devices[Request->device_id()].RTL->run_team_region(
+      mapHostRTLDeviceId(Request->device_id()), TgtEntryPtr, (void **)TgtArgs,
+      TgtOffsets, Request->arg_num(), Request->team_num(),
+      Request->thread_limit(), Request->loop_tripcount());
+
+  Reply->set_number(Ret);
+
+  free(TgtArgs);
+  free(TgtOffsets);
+
+  SERVER_DBG("Ran TargetTeamRegionAsync on device %d with %d args",
+             mapHostRTLDeviceId(Request->device_id()), Request->arg_num());
+  return Status::OK;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteOffloadImpl::mapHostRTLDeviceId(int32_t RTLDeviceID) {
+  for (auto &RTL : PM->RTLs.UsedRTLs) {
+    if (RTLDeviceID - RTL->NumberOfDevices >= 0)
+      RTLDeviceID -= RTL->NumberOfDevices;
+    else
+      break;
+  }
+  return RTLDeviceID;
+}

openmp/libomptarget/plugins/remote/src/CMakeLists.txt

+##===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+##===----------------------------------------------------------------------===##
+#
+# Build a plugin for remote offloading.
+#
+##===----------------------------------------------------------------------===##
+
+cmake_minimum_required(VERSION 3.15)
+
+libomptarget_say("Building RPC offloading plugin - client.")
+
+# Define the suffix for the runtime messaging dumps.
+add_definitions(-DTARGET_NAME=RPC)
+
+include_directories(${LIBOMPTARGET_DEP_LIBELF_INCLUDE_DIRS})
+include_directories(${LIBOMPTARGET_SRC_DIR})
+include_directories(${LIBOMPTARGET_INCLUDE_DIR})
+include_directories(${GRPC_INCLUDE_DIR})
+include_directories(${RPC_INCLUDE_DIR})
+
+add_library(omptarget.rtl.rpc SHARED
+        ${LIBOMPTARGET_SRC_FILES}
+        ${GRPC_SRC_FILES}
+        ${RPC_SRC_DIR}/Utils.cpp
+        Client.cpp
+        rtl.cpp
+)
+
+# Install plugin under the lib destination folder.
+install(TARGETS omptarget.rtl.rpc LIBRARY DESTINATION "${OPENMP_INSTALL_LIBDIR}")
+
+target_link_libraries(omptarget.rtl.rpc
+  ${LIBOMPTARGET_DEP_LIBELF_LIBRARIES}
+  grpc++
+  protobuf
+  "-ldl"
+  "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../../exports")
+
+# Report to the parent scope that we are building a plugin for RPC.
+set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS} rpc" PARENT_SCOPE)

openmp/libomptarget/plugins/remote/src/Client.h

+//===------------------ Client.h - Client Implementation ------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// gRPC Client for the remote plugin.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SRC_CLIENT_H_
+#define SRC_CLIENT_H_
+
+#include "Debug.h"
+#include "Utils.h"
+#include "omptarget.h"
+#include <google/protobuf/arena.h>
+#include <grpcpp/grpcpp.h>
+#include <grpcpp/security/credentials.h>
+#include <grpcpp/support/channel_arguments.h>
+#include <memory>
+#include <numeric>
+
+using grpc::Channel;
+using remoteoffloading::RemoteOffload;
+
+using namespace google;
+
+class RemoteOffloadClient {
+  const int Timeout;
+  const int Shots;
+
+  int DebugLevel;
+  uint64_t MaxSize;
+  int64_t BlockSize;
+
+  std::unique_ptr<RemoteOffload::Stub> Stub;
+  std::unique_ptr<protobuf::Arena> Arena;
+
+  std::map<int32_t, std::map<void *, void *>> RemoteEntries;
+  std::map<int32_t, __tgt_target_table *> DevicesToTables;
+
+  template <typename Fn1, typename Fn2, typename TReturn>
+  auto remoteCall(Fn1 Preprocess, Fn2 Postprocess, TReturn ErrorValue,
+                  bool Retry = true, bool Timeout = true);
+
+public:
+  RemoteOffloadClient(std::shared_ptr<Channel> Channel, int Timeout, int Shots,
+                      uint64_t MaxSize, int64_t BlockSize)
+      : Timeout(Timeout), Shots(Shots), MaxSize(MaxSize), BlockSize(BlockSize),
+        Stub(RemoteOffload::NewStub(Channel)) {
+    DebugLevel = getDebugLevel();
+    Arena = std::make_unique<protobuf::Arena>();
+  }
+
+  RemoteOffloadClient(RemoteOffloadClient &&C) = default;
+
+  ~RemoteOffloadClient() {
+    for (auto TableIt : DevicesToTables)
+      freeTargetTable(TableIt.second);
+  }
+
+  int32_t shutdown(void);
+
+  /////////////////////////////////////////////////////////////////////////////
+  // Remote Registrations.
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t registerLib(__tgt_bin_desc *Desc);
+  int32_t unregisterLib(__tgt_bin_desc *Desc);
+
+  /////////////////////////////////////////////////////////////////////////////
+  // Target Runtime Library Functions.
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t isValidBinary(__tgt_device_image *Image);
+  int32_t getNumberOfDevices();
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t initDevice(int32_t DeviceId);
+  int32_t initRequires(int64_t RequiresFlags);
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_image *Image);
+  int64_t synchronize(int32_t DeviceId, __tgt_async_info *AsyncInfoPtr);
+  int32_t isDataExchangeable(int32_t SrcDevId, int32_t DstDevId);
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  void *dataAlloc(int32_t DeviceId, int64_t Size, void *HstPtr);
+  int32_t dataDelete(int32_t DeviceId, void *TgtPtr);
+
+  int32_t dataSubmitAsync(int32_t DeviceId, void *TgtPtr, void *HstPtr,
+                          int64_t Size, __tgt_async_info *AsyncInfoPtr);
+  int32_t dataRetrieveAsync(int32_t DeviceId, void *HstPtr, void *TgtPtr,
+                            int64_t Size, __tgt_async_info *AsyncInfoPtr);
+
+  int32_t dataExchangeAsync(int32_t SrcDevId, void *SrcPtr, int32_t DstDevId,
+                            void *DstPtr, int64_t Size,
+                            __tgt_async_info *AsyncInfoPtr);
+
+  ///////////////////////////////////////////////////////////////////////////////
+
+  int32_t runTargetRegionAsync(int32_t DeviceId, void *TgtEntryPtr,
+                               void **TgtArgs, ptrdiff_t *TgtOffsets,
+                               int32_t ArgNum, __tgt_async_info *AsyncInfoPtr);
+
+  int32_t runTargetTeamRegionAsync(int32_t DeviceId, void *TgtEntryPtr,
+                                   void **TgtArgs, ptrdiff_t *TgtOffsets,
+                                   int32_t ArgNum, int32_t TeamNum,
+                                   int32_t ThreadLimit, uint64_t LoopTripCount,
+                                   __tgt_async_info *AsyncInfoPtr);
+};
+
+class RemoteClientManager {
+private:
+  std::vector<std::string> Addresses;
+  std::vector<RemoteOffloadClient> Clients;
+  std::vector<int> Devices;
+
+  std::pair<int32_t, int32_t> mapDeviceId(int32_t DeviceId);
+  int DebugLevel;
+
+public:
+  RemoteClientManager(std::vector<std::string> Addresses, int Timeout,
+                      int Shots, uint64_t MaxSize, int64_t BlockSize)
+      : Addresses(Addresses) {
+    grpc::ChannelArguments ChArgs;
+    ChArgs.SetMaxReceiveMessageSize(-1);
+    DebugLevel = getDebugLevel();
+    for (auto Address : Addresses) {
+      Clients.push_back(RemoteOffloadClient(
+          grpc::CreateChannel(Address, grpc::InsecureChannelCredentials()),
+          Timeout, Shots, MaxSize, BlockSize));
+    }
+  }
+
+  int32_t shutdown(void);
+
+  /////////////////////////////////////////////////////////////////////////////
+  // Remote Registrations.
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t registerLib(__tgt_bin_desc *Desc);
+  int32_t unregisterLib(__tgt_bin_desc *Desc);
+
+  /////////////////////////////////////////////////////////////////////////////
+  // Target Runtime Library Functions.
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t isValidBinary(__tgt_device_image *Image);
+  int32_t getNumberOfDevices();
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  int32_t initDevice(int32_t DeviceId);
+  int32_t initRequires(int64_t RequiresFlags);
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_image *Image);
+  int64_t synchronize(int32_t DeviceId, __tgt_async_info *AsyncInfoPtr);
+  int32_t isDataExchangeable(int32_t SrcDevId, int32_t DstDevId);
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  void *dataAlloc(int32_t DeviceId, int64_t Size, void *HstPtr);
+  int32_t dataDelete(int32_t DeviceId, void *TgtPtr);
+
+  int32_t dataSubmitAsync(int32_t DeviceId, void *TgtPtr, void *HstPtr,
+                          int64_t Size, __tgt_async_info *AsyncInfoPtr);
+  int32_t dataRetrieveAsync(int32_t DeviceId, void *HstPtr, void *TgtPtr,
+                            int64_t Size, __tgt_async_info *AsyncInfoPtr);
+
+  int32_t dataExchangeAsync(int32_t SrcDevId, void *SrcPtr, int32_t DstDevId,
+                            void *DstPtr, int64_t Size,
+                            __tgt_async_info *AsyncInfoPtr);
+
+  ///////////////////////////////////////////////////////////////////////////////
+
+  int32_t runTargetRegionAsync(int32_t DeviceId, void *TgtEntryPtr,
+                               void **TgtArgs, ptrdiff_t *TgtOffsets,
+                               int32_t ArgNum, __tgt_async_info *AsyncInfoPtr);
+
+  int32_t runTargetTeamRegionAsync(int32_t DeviceId, void *TgtEntryPtr,
+                                   void **TgtArgs, ptrdiff_t *TgtOffsets,
+                                   int32_t ArgNum, int32_t TeamNum,
+                                   int32_t ThreadLimit, uint64_t LoopTripCount,
+                                   __tgt_async_info *AsyncInfoPtr);
+};
+
+#endif

openmp/libomptarget/plugins/remote/src/Client.cpp

+//===----------------- Client.cpp - Client Implementation -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// gRPC (Client) for the remote plugin.
+//
+//===----------------------------------------------------------------------===//
+
+#include <chrono>
+#include <cmath>
+#include <tuple>
+
+#include "Client.h"
+#include "omptarget.h"
+#include "openmp.pb.h"
+
+using remoteoffloading::AllocData;
+using remoteoffloading::Binary;
+using remoteoffloading::Data;
+using remoteoffloading::DeleteData;
+using remoteoffloading::DevicePair;
+using remoteoffloading::ExchangeDataAsync;
+using remoteoffloading::Int;
+using remoteoffloading::Long;
+using remoteoffloading::Null;
+using remoteoffloading::Pointer;
+using remoteoffloading::RetrieveDataAsync;
+using remoteoffloading::SubmitDataAsync;
+using remoteoffloading::SynchronizeDevice;
+using remoteoffloading::TargetBinaryDescription;
+using remoteoffloading::TargetDeviceImagePtr;
+using remoteoffloading::TargetRegionAsync;
+using remoteoffloading::TargetTable;
+using remoteoffloading::TargetTeamRegionAsync;
+
+using namespace std::chrono;
+
+using grpc::ClientContext;
+using grpc::ClientReader;
+using grpc::ClientWriter;
+using grpc::Status;
+
+///////////////////////////////////////////////////////////////////////////////
+// Registrations with the Remote.
+///////////////////////////////////////////////////////////////////////////////
+
+template <typename Fn1, typename Fn2, typename TReturn>
+auto RemoteOffloadClient::remoteCall(Fn1 Preprocess, Fn2 Postprocess,
+                                     TReturn ErrorValue, bool Retry,
+                                     bool Timeout) {
+  if (Arena->SpaceAllocated() >= MaxSize)
+    Arena->Reset();
+
+  int Attempts = Shots;
+  while (Attempts--) {
+    ClientContext Context;
+    if (Timeout) {
+      auto Deadline =
+          std::chrono::system_clock::now() + std::chrono::seconds(Timeout);
+      Context.set_deadline(Deadline);
+    }
+
+    Status RPCStatus;
+    auto Reply = Preprocess(RPCStatus, Context);
+
+    // TODO: Error handle more appropriately
+    if (!RPCStatus.ok()) {
+      CLIENT_DBG("%s", RPCStatus.error_message().c_str());
+      if (!Retry)
+        break;
+      if (Attempts)
+        CLIENT_DBG("Trying again");
+    } else {
+      return Postprocess(Reply);
+      break;
+    }
+  }
+
+  CLIENT_DBG("Failed");
+  return ErrorValue;
+}
+
+int32_t RemoteOffloadClient::shutdown(void) {
+  ClientContext Context;
+  Null Request;
+  Int Reply;
+  CLIENT_DBG("Shutting down server.");
+  auto Status = Stub->Shutdown(&Context, Request, &Reply);
+  if (Status.ok())
+    return Reply.number();
+  return 1;
+}
+
+int32_t RemoteOffloadClient::registerLib(__tgt_bin_desc *Desc) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<TargetBinaryDescription>(
+            Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        loadTargetBinaryDescription(Desc, *Request);
+        Request->set_bin_ptr((uint64_t)Desc);
+
+        CLIENT_DBG("Registering library");
+        RPCStatus = Stub->RegisterLib(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (Reply->number() == 0) {
+          CLIENT_DBG("Registered library");
+          return 0;
+        }
+        return 1;
+      },
+      /* Error Value */ 1);
+}
+
+int32_t RemoteOffloadClient::unregisterLib(__tgt_bin_desc *Desc) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<Pointer>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+
+        Request->set_number((uint64_t)Desc);
+
+        CLIENT_DBG("Unregistering library");
+        RPCStatus = Stub->UnregisterLib(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (Reply->number() == 0) {
+          CLIENT_DBG("Unregistered library");
+          return 0;
+        }
+        CLIENT_DBG("Failed to unregister library");
+        return 1;
+      },
+      /* Error Value */ 1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Target Runtime Library Functions.
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteOffloadClient::isValidBinary(__tgt_device_image *Image) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request =
+            protobuf::Arena::CreateMessage<TargetDeviceImagePtr>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+
+        Request->set_image_ptr((uint64_t)Image->ImageStart);
+
+        auto *EntryItr = Image->EntriesBegin;
+        while (EntryItr != Image->EntriesEnd)
+          Request->add_entry_ptrs((uint64_t)EntryItr++);
+
+        CLIENT_DBG("Validating binary");
+        RPCStatus = Stub->IsValidBinary(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Validated binary");
+        } else {
+          CLIENT_DBG("Could not validate binary");
+        }
+        return Reply->number();
+      },
+      /* Error Value */ 0);
+}
+
+int32_t RemoteOffloadClient::getNumberOfDevices() {
+  return remoteCall(
+      /* Preprocess */
+      [&](Status &RPCStatus, ClientContext &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<Null>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+
+        CLIENT_DBG("Getting number of devices");
+        RPCStatus = Stub->GetNumberOfDevices(&Context, *Request, Reply);
+
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Found %d devices", Reply->number());
+        } else {
+          CLIENT_DBG("Could not get the number of devices");
+        }
+        return Reply->number();
+      },
+      /*Error Value*/ -1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteOffloadClient::initDevice(int32_t DeviceId) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+
+        Request->set_number(DeviceId);
+
+        CLIENT_DBG("Initializing device %d", DeviceId);
+        RPCStatus = Stub->InitDevice(&Context, *Request, Reply);
+
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (!Reply->number()) {
+          CLIENT_DBG("Initialized device %d", DeviceId);
+        } else {
+          CLIENT_DBG("Could not initialize device %d", DeviceId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+int32_t RemoteOffloadClient::initRequires(int64_t RequiresFlags) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<Long>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        Request->set_number(RequiresFlags);
+        CLIENT_DBG("Initializing requires");
+        RPCStatus = Stub->InitRequires(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](const auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Initialized requires");
+        } else {
+          CLIENT_DBG("Could not initialize requires");
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+__tgt_target_table *RemoteOffloadClient::loadBinary(int32_t DeviceId,
+                                                    __tgt_device_image *Image) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *ImageMessage =
+            protobuf::Arena::CreateMessage<Binary>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<TargetTable>(Arena.get());
+        ImageMessage->set_image_ptr((uint64_t)Image->ImageStart);
+        ImageMessage->set_device_id(DeviceId);
+
+        CLIENT_DBG("Loading Image %p to device %d", Image, DeviceId);
+        RPCStatus = Stub->LoadBinary(&Context, *ImageMessage, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (Reply->entries_size() == 0) {
+          CLIENT_DBG("Could not load image %p onto device %d", Image, DeviceId);
+          return (__tgt_target_table *)nullptr;
+        }
+        DevicesToTables[DeviceId] =
+            (__tgt_target_table *)malloc(sizeof(__tgt_target_table));
+        unloadTargetTable(*Reply, DevicesToTables[DeviceId],
+                          RemoteEntries[DeviceId]);
+
+        CLIENT_DBG("Loaded Image %p to device %d with %d entries", Image,
+                   DeviceId, Reply->entries_size());
+
+        return DevicesToTables[DeviceId];
+      },
+      /* Error Value */ (__tgt_target_table *)nullptr,
+      /* Retry */ true,
+      /* Timeout */ false);
+}
+
+int64_t RemoteOffloadClient::synchronize(int32_t DeviceId,
+                                         __tgt_async_info *AsyncInfoPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Info =
+            protobuf::Arena::CreateMessage<SynchronizeDevice>(Arena.get());
+
+        Info->set_device_id(DeviceId);
+        Info->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+        CLIENT_DBG("Synchronizing device %d", DeviceId);
+        RPCStatus = Stub->Synchronize(&Context, *Info, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Synchronized device %d", DeviceId);
+        } else {
+          CLIENT_DBG("Could not synchronize device %d", DeviceId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+int32_t RemoteOffloadClient::isDataExchangeable(int32_t SrcDevId,
+                                                int32_t DstDevId) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request = protobuf::Arena::CreateMessage<DevicePair>(Arena.get());
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+
+        Request->set_src_dev_id(SrcDevId);
+        Request->set_dst_dev_id(DstDevId);
+
+        CLIENT_DBG("Asking if data is exchangeable between %d, %d", SrcDevId,
+                   DstDevId);
+        RPCStatus = Stub->IsDataExchangeable(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Data is exchangeable between %d, %d", SrcDevId, DstDevId);
+        } else {
+          CLIENT_DBG("Data is not exchangeable between %d, %d", SrcDevId,
+                     DstDevId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+void *RemoteOffloadClient::dataAlloc(int32_t DeviceId, int64_t Size,
+                                     void *HstPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Pointer>(Arena.get());
+        auto *Request = protobuf::Arena::CreateMessage<AllocData>(Arena.get());
+
+        Request->set_device_id(DeviceId);
+        Request->set_size(Size);
+        Request->set_hst_ptr((uint64_t)HstPtr);
+
+        CLIENT_DBG("Allocating %ld bytes on device %d", Size, DeviceId);
+        RPCStatus = Stub->DataAlloc(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG("Allocated %ld bytes on device %d at %p", Size, DeviceId,
+                     (void *)Reply->number());
+        } else {
+          CLIENT_DBG("Could not allocate %ld bytes on device %d at %p", Size,
+                     DeviceId, (void *)Reply->number());
+        }
+        return (void *)Reply->number();
+      },
+      /* Error Value */ (void *)nullptr);
+}
+
+int32_t RemoteOffloadClient::dataSubmitAsync(int32_t DeviceId, void *TgtPtr,
+                                             void *HstPtr, int64_t Size,
+                                             __tgt_async_info *AsyncInfoPtr) {
+
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        std::unique_ptr<ClientWriter<SubmitDataAsync>> Writer(
+            Stub->DataSubmitAsync(&Context, Reply));
+
+        if (Size > BlockSize) {
+          int64_t Start = 0, End = BlockSize;
+          for (auto I = 0; I < ceil((float)Size / BlockSize); I++) {
+            auto *Request =
+                protobuf::Arena::CreateMessage<SubmitDataAsync>(Arena.get());
+
+            Request->set_device_id(DeviceId);
+            Request->set_data((char *)HstPtr + Start, End - Start);
+            Request->set_hst_ptr((uint64_t)HstPtr);
+            Request->set_tgt_ptr((uint64_t)TgtPtr);
+            Request->set_start(Start);
+            Request->set_size(Size);
+            Request->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+            CLIENT_DBG("Submitting %ld-%ld/%ld bytes async on device %d at %p",
+                       Start, End, Size, DeviceId, TgtPtr)
+
+            if (!Writer->Write(*Request)) {
+              CLIENT_DBG("Broken stream when submitting data");
+              Reply->set_number(0);
+              return Reply;
+            }
+
+            Start += BlockSize;
+            End += BlockSize;
+            if (End >= Size)
+              End = Size;
+          }
+        } else {
+          auto *Request =
+              protobuf::Arena::CreateMessage<SubmitDataAsync>(Arena.get());
+
+          Request->set_device_id(DeviceId);
+          Request->set_data(HstPtr, Size);
+          Request->set_hst_ptr((uint64_t)HstPtr);
+          Request->set_tgt_ptr((uint64_t)TgtPtr);
+          Request->set_start(0);
+          Request->set_size(Size);
+
+          CLIENT_DBG("Submitting %ld bytes async on device %d at %p", Size,
+                     DeviceId, TgtPtr)
+          if (!Writer->Write(*Request)) {
+            CLIENT_DBG("Broken stream when submitting data");
+            Reply->set_number(0);
+            return Reply;
+          }
+        }
+
+        Writer->WritesDone();
+        RPCStatus = Writer->Finish();
+
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (!Reply->number()) {
+          CLIENT_DBG("Async submitted %ld bytes on device %d at %p", Size,
+                     DeviceId, TgtPtr)
+        } else {
+          CLIENT_DBG("Could not async submit %ld bytes on device %d at %p",
+                     Size, DeviceId, TgtPtr)
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1,
+      /* Retry */ true,
+      /* Timeout */ false);
+}
+
+int32_t RemoteOffloadClient::dataRetrieveAsync(int32_t DeviceId, void *HstPtr,
+                                               void *TgtPtr, int64_t Size,
+                                               __tgt_async_info *AsyncInfoPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Request =
+            protobuf::Arena::CreateMessage<RetrieveDataAsync>(Arena.get());
+
+        Request->set_device_id(DeviceId);
+        Request->set_size(Size);
+        Request->set_hst_ptr((int64_t)HstPtr);
+        Request->set_tgt_ptr((int64_t)TgtPtr);
+        Request->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+        auto *Reply = protobuf::Arena::CreateMessage<Data>(Arena.get());
+        std::unique_ptr<ClientReader<Data>> Reader(
+            Stub->DataRetrieveAsync(&Context, *Request));
+        Reader->WaitForInitialMetadata();
+        while (Reader->Read(Reply)) {
+          if (Reply->ret()) {
+            CLIENT_DBG("Could not async retrieve %ld bytes on device %d at %p "
+                       "for %p",
+                       Size, DeviceId, TgtPtr, HstPtr)
+            return Reply;
+          }
+
+          if (Reply->start() == 0 && Reply->size() == Reply->data().size()) {
+            CLIENT_DBG("Async retrieving %ld bytes on device %d at %p for %p",
+                       Size, DeviceId, TgtPtr, HstPtr)
+
+            memcpy(HstPtr, Reply->data().data(), Reply->data().size());
+
+            return Reply;
+          }
+          CLIENT_DBG("Retrieving %lu-%lu/%lu bytes async from (%p) to (%p) "
+                     "on Device %d",
+                     Reply->start(), Reply->start() + Reply->data().size(),
+                     Reply->size(), (void *)Request->tgt_ptr(), HstPtr,
+                     Request->device_id());
+
+          memcpy((void *)((char *)HstPtr + Reply->start()),
+                 Reply->data().data(), Reply->data().size());
+        }
+        RPCStatus = Reader->Finish();
+
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (!Reply->ret()) {
+          CLIENT_DBG("Async retrieve %ld bytes on Device %d", Size, DeviceId);
+        } else {
+          CLIENT_DBG("Could not async retrieve %ld bytes on Device %d", Size,
+                     DeviceId);
+        }
+        return Reply->ret();
+      },
+      /* Error Value */ -1,
+      /* Retry */ true,
+      /* Timeout */ false);
+}
+
+int32_t RemoteOffloadClient::dataExchangeAsync(int32_t SrcDevId, void *SrcPtr,
+                                               int32_t DstDevId, void *DstPtr,
+                                               int64_t Size,
+                                               __tgt_async_info *AsyncInfoPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Request =
+            protobuf::Arena::CreateMessage<ExchangeDataAsync>(Arena.get());
+
+        Request->set_src_dev_id(SrcDevId);
+        Request->set_src_ptr((uint64_t)SrcPtr);
+        Request->set_dst_dev_id(DstDevId);
+        Request->set_dst_ptr((uint64_t)DstPtr);
+        Request->set_size(Size);
+        Request->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+        CLIENT_DBG(
+            "Exchanging %ld bytes on device %d at %p for %p on device %d", Size,
+            SrcDevId, SrcPtr, DstPtr, DstDevId);
+        RPCStatus = Stub->DataExchangeAsync(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (Reply->number()) {
+          CLIENT_DBG(
+              "Exchanged %ld bytes on device %d at %p for %p on device %d",
+              Size, SrcDevId, SrcPtr, DstPtr, DstDevId);
+        } else {
+          CLIENT_DBG("Could not exchange %ld bytes on device %d at %p for %p "
+                     "on device %d",
+                     Size, SrcDevId, SrcPtr, DstPtr, DstDevId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+int32_t RemoteOffloadClient::dataDelete(int32_t DeviceId, void *TgtPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Request = protobuf::Arena::CreateMessage<DeleteData>(Arena.get());
+
+        Request->set_device_id(DeviceId);
+        Request->set_tgt_ptr((uint64_t)TgtPtr);
+
+        CLIENT_DBG("Deleting data at %p on device %d", TgtPtr, DeviceId)
+        RPCStatus = Stub->DataDelete(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (!Reply->number()) {
+          CLIENT_DBG("Deleted data at %p on device %d", TgtPtr, DeviceId)
+        } else {
+          CLIENT_DBG("Could not deleted data at %p on device %d", TgtPtr,
+                     DeviceId)
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteOffloadClient::runTargetRegionAsync(
+    int32_t DeviceId, void *TgtEntryPtr, void **TgtArgs, ptrdiff_t *TgtOffsets,
+    int32_t ArgNum, __tgt_async_info *AsyncInfoPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Request =
+            protobuf::Arena::CreateMessage<TargetRegionAsync>(Arena.get());
+
+        Request->set_device_id(DeviceId);
+        Request->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+        Request->set_tgt_entry_ptr(
+            (uint64_t)RemoteEntries[DeviceId][TgtEntryPtr]);
+
+        char **ArgPtr = (char **)TgtArgs;
+        for (auto I = 0; I < ArgNum; I++, ArgPtr++)
+          Request->add_tgt_args((uint64_t)*ArgPtr);
+
+        char *OffsetPtr = (char *)TgtOffsets;
+        for (auto I = 0; I < ArgNum; I++, OffsetPtr++)
+          Request->add_tgt_offsets((uint64_t)*OffsetPtr);
+
+        Request->set_arg_num(ArgNum);
+
+        CLIENT_DBG("Running target region async on device %d", DeviceId);
+        RPCStatus = Stub->RunTargetRegionAsync(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (!Reply->number()) {
+          CLIENT_DBG("Ran target region async on device %d", DeviceId);
+        } else {
+          CLIENT_DBG("Could not run target region async on device %d",
+                     DeviceId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1,
+      /* Retry */ false,
+      /* Timeout */ false);
+}
+
+int32_t RemoteOffloadClient::runTargetTeamRegionAsync(
+    int32_t DeviceId, void *TgtEntryPtr, void **TgtArgs, ptrdiff_t *TgtOffsets,
+    int32_t ArgNum, int32_t TeamNum, int32_t ThreadLimit,
+    uint64_t LoopTripcount, __tgt_async_info *AsyncInfoPtr) {
+  return remoteCall(
+      /* Preprocess */
+      [&](auto &RPCStatus, auto &Context) {
+        auto *Reply = protobuf::Arena::CreateMessage<Int>(Arena.get());
+        auto *Request =
+            protobuf::Arena::CreateMessage<TargetTeamRegionAsync>(Arena.get());
+
+        Request->set_device_id(DeviceId);
+        Request->set_queue_ptr((uint64_t)AsyncInfoPtr);
+
+        Request->set_tgt_entry_ptr(
+            (uint64_t)RemoteEntries[DeviceId][TgtEntryPtr]);
+
+        char **ArgPtr = (char **)TgtArgs;
+        for (auto I = 0; I < ArgNum; I++, ArgPtr++) {
+          Request->add_tgt_args((uint64_t)*ArgPtr);
+        }
+
+        char *OffsetPtr = (char *)TgtOffsets;
+        for (auto I = 0; I < ArgNum; I++, OffsetPtr++)
+          Request->add_tgt_offsets((uint64_t)*OffsetPtr);
+
+        Request->set_arg_num(ArgNum);
+        Request->set_team_num(TeamNum);
+        Request->set_thread_limit(ThreadLimit);
+        Request->set_loop_tripcount(LoopTripcount);
+
+        CLIENT_DBG("Running target team region async on device %d", DeviceId);
+        RPCStatus = Stub->RunTargetTeamRegionAsync(&Context, *Request, Reply);
+        return Reply;
+      },
+      /* Postprocess */
+      [&](auto &Reply) {
+        if (!Reply->number()) {
+          CLIENT_DBG("Ran target team region async on device %d", DeviceId);
+        } else {
+          CLIENT_DBG("Could not run target team region async on device %d",
+                     DeviceId);
+        }
+        return Reply->number();
+      },
+      /* Error Value */ -1,
+      /* Retry */ false,
+      /* Timeout */ false);
+}
+
+/////////////////////////////////////////////////////////////////////////////
+
+// TODO: Better error handling for these three cases
+int32_t RemoteClientManager::shutdown(void) {
+  int32_t Ret = 0;
+  for (auto &Client : Clients)
+    Ret &= Client.shutdown();
+  return Ret;
+}
+
+int32_t RemoteClientManager::registerLib(__tgt_bin_desc *Desc) {
+  int32_t Ret = 0;
+  for (auto &Client : Clients)
+    Ret &= Client.registerLib(Desc);
+  return Ret;
+}
+
+int32_t RemoteClientManager::unregisterLib(__tgt_bin_desc *Desc) {
+  int32_t Ret = 0;
+  for (auto &Client : Clients)
+    Ret &= Client.unregisterLib(Desc);
+  return Ret;
+}
+
+int32_t RemoteClientManager::isValidBinary(__tgt_device_image *Image) {
+  int32_t ClientIdx = 0;
+  for (auto &Client : Clients) {
+    if (auto Ret = Client.isValidBinary(Image))
+      return Ret;
+    ClientIdx++;
+  }
+  return 0;
+}
+
+int32_t RemoteClientManager::getNumberOfDevices() {
+  auto ClientIdx = 0;
+  for (auto &Client : Clients) {
+    if (auto NumDevices = Client.getNumberOfDevices()) {
+      Devices.push_back(NumDevices);
+    }
+    ClientIdx++;
+  }
+
+  return std::accumulate(Devices.begin(), Devices.end(), 0);
+}
+
+std::pair<int32_t, int32_t> RemoteClientManager::mapDeviceId(int32_t DeviceId) {
+  for (size_t ClientIdx = 0; ClientIdx < Devices.size(); ClientIdx++) {
+    if (!(DeviceId >= Devices[ClientIdx]))
+      return {ClientIdx, DeviceId};
+    DeviceId -= Devices[ClientIdx];
+  }
+  return {-1, -1};
+}
+
+/////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteClientManager::initDevice(int32_t DeviceId) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].initDevice(DeviceIdx);
+}
+
+int32_t RemoteClientManager::initRequires(int64_t RequiresFlags) {
+  for (auto &Client : Clients)
+    Client.initRequires(RequiresFlags);
+
+  return RequiresFlags;
+}
+
+/////////////////////////////////////////////////////////////////////////////
+
+__tgt_target_table *RemoteClientManager::loadBinary(int32_t DeviceId,
+                                                    __tgt_device_image *Image) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].loadBinary(DeviceIdx, Image);
+}
+
+int64_t RemoteClientManager::synchronize(int32_t DeviceId,
+                                         __tgt_async_info *AsyncInfoPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].synchronize(DeviceIdx, AsyncInfoPtr);
+}
+
+int32_t RemoteClientManager::isDataExchangeable(int32_t SrcDevId,
+                                                int32_t DstDevId) {
+  int32_t SrcClientIdx, SrcDeviceIdx, DstClientIdx, DstDeviceIdx;
+  std::tie(SrcClientIdx, SrcDeviceIdx) = mapDeviceId(SrcDevId);
+  std::tie(DstClientIdx, DstDeviceIdx) = mapDeviceId(DstDevId);
+  return Clients[SrcClientIdx].isDataExchangeable(SrcDeviceIdx, DstDeviceIdx);
+}
+
+/////////////////////////////////////////////////////////////////////////////
+
+void *RemoteClientManager::dataAlloc(int32_t DeviceId, int64_t Size,
+                                     void *HstPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].dataAlloc(DeviceIdx, Size, HstPtr);
+}
+
+int32_t RemoteClientManager::dataDelete(int32_t DeviceId, void *TgtPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].dataDelete(DeviceIdx, TgtPtr);
+}
+
+int32_t RemoteClientManager::dataSubmitAsync(int32_t DeviceId, void *TgtPtr,
+                                             void *HstPtr, int64_t Size,
+                                             __tgt_async_info *AsyncInfoPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].dataSubmitAsync(DeviceIdx, TgtPtr, HstPtr, Size,
+                                            AsyncInfoPtr);
+}
+
+int32_t RemoteClientManager::dataRetrieveAsync(int32_t DeviceId, void *HstPtr,
+                                               void *TgtPtr, int64_t Size,
+                                               __tgt_async_info *AsyncInfoPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].dataRetrieveAsync(DeviceIdx, HstPtr, TgtPtr, Size,
+                                              AsyncInfoPtr);
+}
+
+int32_t RemoteClientManager::dataExchangeAsync(int32_t SrcDevId, void *SrcPtr,
+                                               int32_t DstDevId, void *DstPtr,
+                                               int64_t Size,
+                                               __tgt_async_info *AsyncInfoPtr) {
+  int32_t SrcClientIdx, SrcDeviceIdx, DstClientIdx, DstDeviceIdx;
+  std::tie(SrcClientIdx, SrcDeviceIdx) = mapDeviceId(SrcDevId);
+  std::tie(DstClientIdx, DstDeviceIdx) = mapDeviceId(DstDevId);
+  return Clients[SrcClientIdx].dataExchangeAsync(
+      SrcDeviceIdx, SrcPtr, DstDeviceIdx, DstPtr, Size, AsyncInfoPtr);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t RemoteClientManager::runTargetRegionAsync(
+    int32_t DeviceId, void *TgtEntryPtr, void **TgtArgs, ptrdiff_t *TgtOffsets,
+    int32_t ArgNum, __tgt_async_info *AsyncInfoPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].runTargetRegionAsync(
+      DeviceIdx, TgtEntryPtr, TgtArgs, TgtOffsets, ArgNum, AsyncInfoPtr);
+}
+
+int32_t RemoteClientManager::runTargetTeamRegionAsync(
+    int32_t DeviceId, void *TgtEntryPtr, void **TgtArgs, ptrdiff_t *TgtOffsets,
+    int32_t ArgNum, int32_t TeamNum, int32_t ThreadLimit,
+    uint64_t LoopTripCount, __tgt_async_info *AsyncInfoPtr) {
+  int32_t ClientIdx, DeviceIdx;
+  std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
+  return Clients[ClientIdx].runTargetTeamRegionAsync(
+      DeviceIdx, TgtEntryPtr, TgtArgs, TgtOffsets, ArgNum, TeamNum, ThreadLimit,
+      LoopTripCount, AsyncInfoPtr);
+}

openmp/libomptarget/plugins/remote/src/rtl.cpp

+//===--------------------- rtl.cpp - Remote RTL Plugin --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// RTL for Host.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cstddef>
+#include <memory>
+#include <string>
+
+#include "Client.h"
+#include "omptarget.h"
+#include "omptargetplugin.h"
+
+#define TARGET_NAME RPC
+#define DEBUG_PREFIX "Target " GETNAME(TARGET_NAME) " RTL"
+
+#include "../../common/elf_common/elf_common.h"
+
+RemoteClientManager *Manager;
+
+__attribute__((constructor(101))) void initRPC() {
+  DP("Init RPC library!\n");
+
+  std::vector<std::string> ServerAddresses = {"0.0.0.0:50051"};
+  if (const char *Env = std::getenv("LIBOMPTARGET_RPC_ADDRESS")) {
+    ServerAddresses.clear();
+    std::string AddressString = Env;
+    std::string Delimiter = ",";
+
+    size_t Pos = 0;
+    std::string Token;
+    while ((Pos = AddressString.find(Delimiter)) != std::string::npos) {
+      Token = AddressString.substr(0, Pos);
+      ServerAddresses.push_back(Token);
+      AddressString.erase(0, Pos + Delimiter.length());
+    }
+    ServerAddresses.push_back(AddressString);
+  }
+
+  int Timeout = 5;
+  if (const char *Env1 = std::getenv("LIBOMPTARGET_RPC_LATENCY"))
+    Timeout = std::stoi(Env1);
+
+  int Shots = 2;
+  if (const char *Env = std::getenv("LIBOMPTARGET_RPC_TRY"))
+    Shots = std::stoi(Env);
+
+  uint64_t MaxSize = 1 << 30;
+  if (const char *Env = std::getenv("LIBOMPTARGET_RPC_ALLOCATOR_MAX"))
+    MaxSize = std::stoi(Env);
+
+  int BlockSize = 1 << 20;
+  if (const char *Env = std::getenv("LIBOMPTARGET_RPC_BLOCK_SIZE"))
+    BlockSize = std::stoi(Env);
+
+  Manager = new RemoteClientManager(ServerAddresses, Timeout, Shots, MaxSize,
+                                    BlockSize);
+}
+
+__attribute__((destructor(101))) void deinitRPC() {
+  Manager->shutdown(); // Error handle
+  DP("Deinit RPC library!\n");
+  delete Manager;
+}
+
+// Exposed library API function
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+///////////////////////////////////////////////////////////////////////////////
+// Registrations with the Remote.
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t __tgt_rtl_register_lib(__tgt_bin_desc *Desc) {
+  return Manager->registerLib(Desc);
+}
+
+int32_t __tgt_rtl_unregister_lib(__tgt_bin_desc *Desc) {
+  return Manager->unregisterLib(Desc);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Target Runtime Library Functions.
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *Image) {
+  return Manager->isValidBinary(Image);
+}
+
+int32_t __tgt_rtl_number_of_devices() { return Manager->getNumberOfDevices(); }
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t __tgt_rtl_init_device(int32_t DeviceId) {
+  return Manager->initDevice(DeviceId);
+}
+
+int64_t __tgt_rtl_init_requires(int64_t RequiresFlags) {
+  return Manager->initRequires(RequiresFlags);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+__tgt_target_table *__tgt_rtl_load_binary(int32_t DeviceId,
+                                          __tgt_device_image *Image) {
+  return Manager->loadBinary(DeviceId, (__tgt_device_image *)Image);
+}
+
+int32_t __tgt_rtl_synchronize(int32_t DeviceId,
+                              __tgt_async_info *AsyncInfoPtr) {
+  return Manager->synchronize(DeviceId, AsyncInfoPtr);
+}
+
+int32_t __tgt_rtl_is_data_exchangable(int32_t SrcDevId, int32_t DstDevId) {
+  return Manager->isDataExchangeable(SrcDevId, DstDevId);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+void *__tgt_rtl_data_alloc(int32_t DeviceId, int64_t Size, void *HstPtr) {
+  return Manager->dataAlloc(DeviceId, Size, HstPtr);
+}
+
+int32_t __tgt_rtl_data_submit(int32_t DeviceId, void *TgtPtr, void *HstPtr,
+                              int64_t Size) {
+  return Manager->dataSubmitAsync(DeviceId, TgtPtr, HstPtr, Size, nullptr);
+}
+
+int32_t __tgt_rtl_data_submit_async(int32_t DeviceId, void *TgtPtr,
+                                    void *HstPtr, int64_t Size,
+                                    __tgt_async_info *AsyncInfoPtr) {
+  return Manager->dataSubmitAsync(DeviceId, TgtPtr, HstPtr, Size, AsyncInfoPtr);
+}
+
+int32_t __tgt_rtl_data_retrieve(int32_t DeviceId, void *HstPtr, void *TgtPtr,
+                                int64_t Size) {
+  return Manager->dataRetrieveAsync(DeviceId, HstPtr, TgtPtr, Size, nullptr);
+}
+
+int32_t __tgt_rtl_data_retrieve_async(int32_t DeviceId, void *HstPtr,
+                                      void *TgtPtr, int64_t Size,
+                                      __tgt_async_info *AsyncInfoPtr) {
+  return Manager->dataRetrieveAsync(DeviceId, HstPtr, TgtPtr, Size,
+                                    AsyncInfoPtr);
+}
+
+int32_t __tgt_rtl_data_delete(int32_t DeviceId, void *TgtPtr) {
+  return Manager->dataDelete(DeviceId, TgtPtr);
+}
+
+int32_t __tgt_rtl_data_exchange(int32_t SrcDevId, void *SrcPtr,
+                                int32_t DstDevId, void *DstPtr, int64_t Size) {
+  return Manager->dataExchangeAsync(SrcDevId, SrcPtr, DstDevId, DstPtr, Size,
+                                    nullptr);
+}
+
+int32_t __tgt_rtl_data_exchange_async(int32_t SrcDevId, void *SrcPtr,
+                                      int32_t DstDevId, void *DstPtr,
+                                      int64_t Size,
+                                      __tgt_async_info *AsyncInfoPtr) {
+  return Manager->dataExchangeAsync(SrcDevId, SrcPtr, DstDevId, DstPtr, Size,
+                                    AsyncInfoPtr);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+int32_t __tgt_rtl_run_target_region(int32_t DeviceId, void *TgtEntryPtr,
+                                    void **TgtArgs, ptrdiff_t *TgtOffsets,
+                                    int32_t ArgNum) {
+  return Manager->runTargetRegionAsync(DeviceId, TgtEntryPtr, TgtArgs,
+                                       TgtOffsets, ArgNum, nullptr);
+}
+
+int32_t __tgt_rtl_run_target_region_async(int32_t DeviceId, void *TgtEntryPtr,
+                                          void **TgtArgs, ptrdiff_t *TgtOffsets,
+                                          int32_t ArgNum,
+                                          __tgt_async_info *AsyncInfoPtr) {
+  return Manager->runTargetRegionAsync(DeviceId, TgtEntryPtr, TgtArgs,
+                                       TgtOffsets, ArgNum, AsyncInfoPtr);
+}
+
+int32_t __tgt_rtl_run_target_team_region(int32_t DeviceId, void *TgtEntryPtr,
+                                         void **TgtArgs, ptrdiff_t *TgtOffsets,
+                                         int32_t ArgNum, int32_t TeamNum,
+                                         int32_t ThreadLimit,
+                                         uint64_t LoopTripCount) {
+  return Manager->runTargetTeamRegionAsync(DeviceId, TgtEntryPtr, TgtArgs,
+                                           TgtOffsets, ArgNum, TeamNum,
+                                           ThreadLimit, LoopTripCount, nullptr);
+}
+
+int32_t __tgt_rtl_run_target_team_region_async(
+    int32_t DeviceId, void *TgtEntryPtr, void **TgtArgs, ptrdiff_t *TgtOffsets,
+    int32_t ArgNum, int32_t TeamNum, int32_t ThreadLimit,
+    uint64_t LoopTripCount, __tgt_async_info *AsyncInfoPtr) {
+  return Manager->runTargetTeamRegionAsync(
+      DeviceId, TgtEntryPtr, TgtArgs, TgtOffsets, ArgNum, TeamNum, ThreadLimit,
+      LoopTripCount, AsyncInfoPtr);
+}
+
+// Exposed library API function
+#ifdef __cplusplus
+}
+#endif

openmp/libomptarget/src/CMakeLists.txt

 set(LIBOMPTARGET_SRC_FILES
   ${CMAKE_CURRENT_SOURCE_DIR}/api.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/device.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/interface.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/rtl.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/omptarget.cpp
 )
 
+set(LIBOMPTARGET_SRC_FILES ${LIBOMPTARGET_SRC_FILES} PARENT_SCOPE)
+
 include_directories(${LIBOMPTARGET_LLVM_INCLUDE_DIRS})
 
 # Build libomptarget library with libdl dependency. Add LLVMSupport
 # dependency if building in-tree with profiling enabled.
 if(OPENMP_STANDALONE_BUILD OR (NOT OPENMP_ENABLE_LIBOMPTARGET_PROFILING))
   add_library(omptarget SHARED ${LIBOMPTARGET_SRC_FILES})
   target_link_libraries(omptarget
     ${CMAKE_DL_LIBS}