[Libomptarget] Add support for offloading binaries in libomptarget

[jhuber6]

Summary

The previous path changed the linker wrapper to embed the offloading
binary format inside the target image instead. This will allow us to
more generically bundle metadata with these images, such as requires
clauses or the target architecture it was compiled for.

I wasn't sure how to handle this best, so I introduced a new type that
replaces the old __tgt_device_image struct that we can expand inside
the runtime library. I made the new __tgt_device_binary struct pretty
much the same for now. In the future we could change this struct to
pretty much be the OffloadBinary class in the future.

[jhuber6]

Fixing strcmp.

[jhuber6]

Fix problem with using pointers to thing in a vector, need to reserve the space beforehand so we don't realloc.

[jhuber6]

Typo.

[jhuber6]

Instead of changing the __tgt_device_image, do you think it would be better to just to make a new type that only contains the extracted metadata? We could just pass a pointer to the into struct to the is_valid_binary, might be much cleaner. Then we would make a new __tgt_device_image that points to the actual ELF and use that as we do currently.

[tianshilei1992]

openmp/libomptarget/include/omptarget.h

143

Is it better to define it as follows?

struct __tgt_device_binary {
  __tgt_device_image Image;
  __tgt_image_info Info;
}

[jhuber6]

openmp/libomptarget/include/omptarget.h

143

I did it that way to somewhat reduce noise when changing to the new format. But now that I'm thinking about it, where else will we need this metadata? It might be fine to just parse it out and create new images so we don't change them, e.g.

__tgt_image_info info = getInfoFromImage(image); // Gets the strings out from the offload binary type in the __tgt_device_image`.
__tgt_device_image image = getELFFromImage(image); //Gets the ELF out and makes a new image with begin / end set correctly (Need to make a new one, the pointers are in RO memory).
is_binary_valid(image, info); // Plugin uses info struct if necessary, every other interface is the same.

WDYT?

[tianshilei1992]

openmp/libomptarget/include/omptarget.h

143

That looks neater.
In addition, we also want to have a clear definition for front end as well. I'm not saying the front end is gonna include the header. At least we have a clear definition with proper documentation, instead of nested definition in a function.

[jhuber6]

Chaning to use an additional struct instead. Also implemented the class to parse these.

Let me know if there are runtime considerations here, with the unique pointer allocations and string conversions for the map. The usage of __tgt_is_valid_binary is supposed to be mostly lightweight without invoking the runtime, so let me know if this is too expensive.

[tianshilei1992]

Is it necessary to create __tgt_image_info in libomptarget? If the front end emits a binary compatible with OffloadBinary, all the information a plugin needs is in it right? Can we just include the header and read those information in plugin instead of in libomptarget such that we don't need to change the interface?

openmp/libomptarget/include/OffloadBinary.h

71 ↗

(On Diff #436009)

you may want to make it private

72 ↗

(On Diff #436009)

Does it make more sense to return nullptr instead of "" here?

75 ↗

(On Diff #436009)

you may want to make it private

openmp/libomptarget/src/rtl.cpp

13–15

I'd do the following.

286–287

plus clang-format of course.

[jhuber6]

In D127432#3576385, @tianshilei1992 wrote:

Is it necessary to create __tgt_image_info in libomptarget? If the front end emits a binary compatible with OffloadBinary, all the information a plugin needs is in it right? Can we just include the header and read those information in plugin instead of in libomptarget such that we don't need to change the interface?

I was originally thinking about doing that, but it's much noisier since we'd need to change a lot of code that actually loads the binaries. It would free us from reallocating these structs however. Otherwise we'd need to change each usage of the Image, this just extracts the image at the start and everything else stays the same. I don't know which is the superior approach. I find this is a "cleaner" way to augment the existing functionality while maintaining backwards compatibility, but maybe there's a need for everything else to have access to the metadata in the plugin.

openmp/libomptarget/include/OffloadBinary.h

71 ↗

(On Diff #436009)

It doesn't modify any content so

72 ↗

(On Diff #436009)

It's supposed to be a string, so an empty string is more appropriate than a null pointer I feel.

75 ↗

(On Diff #436009)

It should be private, I'll see if I can do that considering the static create functions calls it.

[jhuber6]

Making suggested changes.

[jdoerfert]

openmp/libomptarget/include/omptargetplugin.h

33

I am unsure if we should/need to create a new api here.

[jhuber6]

Changing the is_valid_binary function to be a new interface. This allows the plguins to optionally define the interface that uses the additional information struct, and we will use it if availible.

[jhuber6]

Changing the vector to a list so we always have stable pointers. Reserving a vector doesn't work because of shared libraries, also storing the extract information in the list as well.

[jhuber6]

ping.

[jhuber6]

A previous revision proposes making libomptarget link with the LLVM libraries. This allows us to use LLVMObject directly without duplicating code. Update this revision with that in mind.

[JonChesterfield]

Looks ok to me. Does propagating const into __tgt_device_image (and possibly further) work to remove the const cast? Iirc the libelf interfaces take void* and libllvm take const, so it's possible we'll be able to gradually move const all the way through the stack.

Strictly I suppose const fields in the type are an abi change, we could static_assert that making them const doesn't change offsetof if feeling paranoid.

[jhuber6]

In D127432#3656370, @JonChesterfield wrote:

Looks ok to me. Does propagating const into __tgt_device_image (and possibly further) work to remove the const cast? Iirc the libelf interfaces take void* and libllvm take const, so it's possible we'll be able to gradually move const all the way through the stack.

Strictly I suppose const fields in the type are an abi change, we could static_assert that making them const doesn't change offsetof if feeling paranoid.

Right now that would require const_cast somewhere else. I believe there are calls to libelf in the x86_64 version of the plugin which requires a char * input. We will probably be able to transition away from those if we are allowed to use LLVM libraries. The data that these images point to are global constants, so writing to them will result in whatever happens when you write to RO memory in an ELF.

[JonChesterfield]

openmp/libomptarget/src/rtl.cpp

491

This turns out to be a bad thing. unregisterLib can be called by the host application exit, at which point the plugin may have been unloaded. Is_valid_binary happens to work ok anyway, though I think that's sketchy, but is_valid_binary_info will call into cuda / hsa after they've been torn down and blow up.

[jhuber6]

openmp/libomptarget/src/rtl.cpp

491

Is there a reason to even call this? We already determined the valid binaries before. We should only need to traverse a list of the valid ones here withotu checking it again.

[JonChesterfield]

openmp/libomptarget/src/rtl.cpp

562

I'm doubtful that calling into the plugin to run destructors is any safer than is_valid_binary_info. I think code is working because libomptarget happens to be torn down before the plugins but nothing particularly ensures that will be the case.

This is however a good place to call a destructor function that we add to the plugins.

So we can add tgt_rtl_init_plugin() and tgt_rtl_dtor_plugin() functions, replace the global state constructors in the plugins with those, call init shortly before init_device and call the dtor from here. That removes this undefined behaviour and fixes the segfault Joseph just noticed.

openmp/libomptarget/include/device.h

 /// Struct for the data required to handle plugins
 struct PluginManager {
   PluginManager(bool UseEventsForAtomicTransfers)
       : UseEventsForAtomicTransfers(UseEventsForAtomicTransfers) {}
 
   /// RTLs identified on the host
   RTLsTy RTLs;
 
+  /// Device images retrieved from the offload binary.
+  std::vector<__tgt_device_binary> Binaries;
+
   /// Devices associated with RTLs
   std::vector<std::unique_ptr<DeviceTy>> Devices;
   std::mutex RTLsMtx; ///< For RTLs and Devices
 
   /// Translation table retreived from the binary
   HostEntriesBeginToTransTableTy HostEntriesBeginToTransTable;
   std::mutex TrlTblMtx; ///< For Translation Table
   /// Host offload entries in order of image registration

openmp/libomptarget/include/omptarget.h

 /// This struct is a record of the device image information
 struct __tgt_device_image {
   void *ImageStart;                  // Pointer to the target code start
   void *ImageEnd;                    // Pointer to the target code end
   __tgt_offload_entry *EntriesBegin; // Begin of table with all target entries
   __tgt_offload_entry *EntriesEnd;   // End of table (non inclusive)
 };
 
+/// This struct represents per-target information passed to the plugins.
+struct __tgt_image_info {
+  const char *Arch;
+};
+
+/// This struct is a record of the device image binary information parsed from
+/// device image.
+struct __tgt_device_binary {
+  void *ImageStart;                  // Pointer to the target code start
+  void *ImageEnd;                    // Pointer to the target code end
+  __tgt_offload_entry *EntriesBegin; // Begin of table with all target entries
+  __tgt_offload_entry *EntriesEnd;   // End of table (non inclusive)
+  __tgt_image_info Info;             // Struct containing metadata
+};
+
 /// This struct is a record of all the host code that may be offloaded to a
 /// target.
 struct __tgt_bin_desc {
   int32_t NumDeviceImages;           // Number of device types supported
   __tgt_device_image *DeviceImages;  // Array of device images (1 per dev. type)
   __tgt_offload_entry *HostEntriesBegin; // Begin of table with all host entries
   __tgt_offload_entry *HostEntriesEnd;   // End of table (non inclusive)
 };

openmp/libomptarget/include/omptargetplugin.h

 // target RTL.
 int32_t __tgt_rtl_number_of_devices(void);
 
 // Return an integer different from zero if the provided device image can be
 // supported by the runtime. The functionality is similar to comparing the
 // result of __tgt__rtl__load__binary to NULL. However, this is meant to be a
 // lightweight query to determine if the RTL is suitable for an image without
 // having to load the library, which can be expensive.
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *Image);
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *Image);
 
 // Return an integer other than zero if the data can be exchaned from SrcDevId
 // to DstDevId. If it is data exchangable, the device plugin should provide
 // function to move data from source device to destination device directly.
 int32_t __tgt_rtl_is_data_exchangable(int32_t SrcDevId, int32_t DstDevId);
 
 // Return an integer other than zero if the plugin can handle images which do
 // not contain target regions and global variables (but can contain other
 // functions)

 int32_t __tgt_rtl_deinit_device(int32_t ID);
 
 // Pass an executable image section described by image to the specified
 // device and prepare an address table of target entities. In case of error,
 // return NULL. Otherwise, return a pointer to the built address table.
 // Individual entries in the table may also be NULL, when the corresponding
 // offload region is not supported on the target device.
 __tgt_target_table *__tgt_rtl_load_binary(int32_t ID,
-                                          __tgt_device_image *Image);
+                                          __tgt_device_binary *Image);
 
 // Allocate data on the particular target device, of the specified size.
 // HostPtr is a address of the host data the allocated target data
 // will be associated with (HostPtr may be NULL if it is not known at
 // allocation time, like for example it would be for target data that
 // is allocated by omp_target_alloc() API). Return address of the
 // allocated data on the target that will be used by libomptarget.so to
 // initialize the target data mapping structures. These addresses are

openmp/libomptarget/include/rtl.h

 /// Map between the host entry begin and the translation table. Each
 /// registered library gets one TranslationTable. Use the map from
 /// __tgt_offload_entry so that we may quickly determine whether we
 /// are trying to (re)register an existing lib or really have a new one.
 struct TranslationTable {
   __tgt_target_table HostTable;
 
   // Image assigned to a given device.
-  std::vector<__tgt_device_image *> TargetsImages; // One image per device ID.
+  std::vector<__tgt_device_binary *> TargetsBinaries; // One per device ID.
 
   // Table of entry points or NULL if it was not already computed.
   std::vector<__tgt_target_table *> TargetsTable; // One table per device ID.
 };
 typedef std::map<__tgt_offload_entry *, TranslationTable>
     HostEntriesBeginToTransTableTy;
 
 /// Map between the host ptr and a table index

openmp/libomptarget/plugins/amdgpu/src/rtl.cpp

// Run on the device.
     ArgPool->deallocate(kernarg);
     DeviceInfo.FreeSignalPool.push(s);
   }
 
   DP("Kernel completed\n");
   return OFFLOAD_SUCCESS;
 }
 
-bool elf_machine_id_is_amdgcn(__tgt_device_image *image) {
+bool elf_machine_id_is_amdgcn(__tgt_device_binary *image) {
   const uint16_t amdgcnMachineID = 224; // EM_AMDGPU may not be in system elf.h
   int32_t r = elf_check_machine(image, amdgcnMachineID);
   if (!r) {
     DP("Supported machine ID not found\n");
   }
   return r;
 }
 
-uint32_t elf_e_flags(__tgt_device_image *image) {
+uint32_t elf_e_flags(__tgt_device_binary *image) {
   char *img_begin = (char *)image->ImageStart;
   size_t img_size = (char *)image->ImageEnd - img_begin;
 
   Elf *e = elf_memory(img_begin, img_size);
   if (!e) {
     DP("Unable to get ELF handle: %s!\n", elf_errmsg(-1));
     return 0;
   }
struct device_environment {
   // If it is in .bss, we must wait for it to be allocated space on the
   // gpu (trunk) and initialize after loading.
   const char *sym() { return "omptarget_device_environment"; }
 
   DeviceEnvironmentTy host_device_env;
   symbol_info si;
   bool valid = false;
 
-  __tgt_device_image *image;
+  __tgt_device_binary *image;
   const size_t img_size;
 
   device_environment(int device_id, int number_devices, int dynamic_mem_size,
-                     __tgt_device_image *image, const size_t img_size)
+                     __tgt_device_binary *image, const size_t img_size)
       : image(image), img_size(img_size) {
 
     host_device_env.NumDevices = number_devices;
     host_device_env.DeviceNum = device_id;
     host_device_env.DebugKind = 0;
     host_device_env.DynamicMemSize = dynamic_mem_size;
     if (char *envStr = getenv("LIBOMPTARGET_DEVICE_RTL_DEBUG")) {
       host_device_env.DebugKind = std::stoi(envStr);

 namespace core {
 hsa_status_t allow_access_to_all_gpu_agents(void *ptr) {
   return hsa_amd_agents_allow_access(DeviceInfo.HSAAgents.size(),
                                      &DeviceInfo.HSAAgents[0], NULL, ptr);
 }
 } // namespace core
 
 extern "C" {
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *image) {
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *image) {
   return elf_machine_id_is_amdgcn(image);
 }
 
 int __tgt_rtl_number_of_devices() {
   // If the construction failed, no methods are safe to call
   if (DeviceInfo.ConstructionSucceeded) {
     return DeviceInfo.NumberOfDevices;
   } else {
DP("Device %d: wavefront size %d, total threads %d x %d = %d\n", device_id,
      DeviceInfo.GroupsPerDevice[device_id],
      DeviceInfo.GroupsPerDevice[device_id] *
          DeviceInfo.ThreadsPerGroup[device_id]);
 
   return OFFLOAD_SUCCESS;
 }
 
 static __tgt_target_table *
-__tgt_rtl_load_binary_locked(int32_t device_id, __tgt_device_image *image);
+__tgt_rtl_load_binary_locked(int32_t device_id, __tgt_device_binary *image);
 
 __tgt_target_table *__tgt_rtl_load_binary(int32_t device_id,
-                                          __tgt_device_image *image) {
+                                          __tgt_device_binary *image) {
   DeviceInfo.load_run_lock.lock();
   __tgt_target_table *res = __tgt_rtl_load_binary_locked(device_id, image);
   DeviceInfo.load_run_lock.unlock();
   return res;
 }
 
 __tgt_target_table *__tgt_rtl_load_binary_locked(int32_t device_id,
-                                                 __tgt_device_image *image) {
+                                                 __tgt_device_binary *image) {
   // This function loads the device image onto gpu[device_id] and does other
   // per-image initialization work. Specifically:
   //
   // - Initialize an DeviceEnvironmentTy instance embedded in the
   //   image at the symbol "omptarget_device_environment"
   //   Fields DebugKind, DeviceNum, NumDevices. Used by the deviceRTL.
   //
   // - Allocate a large array per-gpu (could be moved to init_device)

openmp/libomptarget/plugins/common/elf_common/elf_common.h

 #ifndef LLVM_OPENMP_LIBOMPTARGET_PLUGINS_COMMON_ELF_COMMON_ELF_COMMON_H
 #define LLVM_OPENMP_LIBOMPTARGET_PLUGINS_COMMON_ELF_COMMON_ELF_COMMON_H
 
 #include "omptargetplugin.h"
 #include <cstdint>
 
 /// Return non-zero, if the given \p image is an ELF object, which
 /// e_machine matches \p target_id; return zero otherwise.
-EXTERN int32_t elf_check_machine(__tgt_device_image *image, uint16_t target_id);
+EXTERN int32_t elf_check_machine(__tgt_device_binary *image,
+                                 uint16_t target_id);
 
 /// Return non-zero, if the given \p image is an ET_DYN ELF object;
 /// return zero otherwise.
-EXTERN int32_t elf_is_dynamic(__tgt_device_image *image);
+EXTERN int32_t elf_is_dynamic(__tgt_device_binary *image);
 
 #endif // LLVM_OPENMP_LIBOMPTARGET_PLUGINS_COMMON_ELF_COMMON_ELF_COMMON_H

openmp/libomptarget/plugins/common/elf_common/elf_common.cpp

 using namespace llvm::ELF;
 using namespace llvm::object;
 
 /// If the given range of bytes [\p BytesBegin, \p BytesEnd) represents
 /// a valid ELF, then invoke \p Callback on the ELFObjectFileBase
 /// created from this range, otherwise, return 0.
 /// If \p Callback is invoked, then return whatever value \p Callback returns.
 template <typename F>
-static int32_t withBytesAsElf(char *BytesBegin, char *BytesEnd, F Callback) {
+static int32_t withBytesAsElf(const char *BytesBegin, const char *BytesEnd,
+                              F Callback) {
   size_t Size = BytesEnd - BytesBegin;
   StringRef StrBuf(BytesBegin, Size);
 
   auto Magic = identify_magic(StrBuf);
   if (Magic != file_magic::elf && Magic != file_magic::elf_relocatable &&
       Magic != file_magic::elf_executable &&
       Magic != file_magic::elf_shared_object && Magic != file_magic::elf_core) {
     DP("Not an ELF image!\n");
if (!Object) {
     DP("Unknown ELF format!\n");
     return 0;
   }
 
   return Callback(Object);
 }
 
 // Check whether an image is valid for execution on target_id
-int32_t elf_check_machine(__tgt_device_image *image, uint16_t target_id) {
+int32_t elf_check_machine(__tgt_device_binary *image, uint16_t target_id) {
   auto CheckMachine = [target_id](const ELFObjectFileBase *Object) {
     return target_id == Object->getEMachine();
   };
-  return withBytesAsElf(reinterpret_cast<char *>(image->ImageStart),
-                        reinterpret_cast<char *>(image->ImageEnd),
+  return withBytesAsElf(reinterpret_cast<const char *>(image->ImageStart),
+                        reinterpret_cast<const char *>(image->ImageEnd),
                         CheckMachine);
 }
 
-int32_t elf_is_dynamic(__tgt_device_image *image) {
+int32_t elf_is_dynamic(__tgt_device_binary *image) {
   auto CheckDynType = [](const ELFObjectFileBase *Object) {
     uint16_t Type = Object->getEType();
     DP("ELF Type: %d\n", Type);
     return Type == ET_DYN;
   };
-  return withBytesAsElf(reinterpret_cast<char *>(image->ImageStart),
-                        reinterpret_cast<char *>(image->ImageEnd),
+  return withBytesAsElf(reinterpret_cast<const char *>(image->ImageStart),
+                        reinterpret_cast<const char *>(image->ImageEnd),
                         CheckDynType);
 }

openmp/libomptarget/plugins/cuda/src/rtl.cpp

int deinitDevice(const int DeviceId) {
     if (!checkResult(cuDevicePrimaryCtxRelease(Device),
                      "Error returned from cuDevicePrimaryCtxRelease\n"))
       return OFFLOAD_FAIL;
 
     return OFFLOAD_SUCCESS;
   }
 
   __tgt_target_table *loadBinary(const int DeviceId,
-                                 const __tgt_device_image *Image) {
+                                 const __tgt_device_binary *Image) {
     // Clear the offload table as we are going to create a new one.
     clearOffloadEntriesTable(DeviceId);
 
     // Create the module and extract the function pointers.
     CUmodule Module;
     DP("Load data from image " DPxMOD "\n", DPxPTR(Image->ImageStart));
     CUresult Err =
         cuModuleLoadDataEx(&Module, Image->ImageStart, 0, nullptr, nullptr);

 DeviceRTLTy DeviceRTL;
 } // namespace
 
 // Exposed library API function
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *image) {
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *image) {
   return elf_check_machine(image, /* EM_CUDA */ 190);
 }
 
 int32_t __tgt_rtl_number_of_devices() { return DeviceRTL.getNumOfDevices(); }
 
 int64_t __tgt_rtl_init_requires(int64_t RequiresFlags) {
   DP("Init requires flags to %" PRId64 "\n", RequiresFlags);
   DeviceRTL.setRequiresFlag(RequiresFlags);

 int32_t __tgt_rtl_deinit_device(int32_t device_id) {
   assert(DeviceRTL.isValidDeviceId(device_id) && "device_id is invalid");
   // Context is set when deinit the device.
 
   return DeviceRTL.deinitDevice(device_id);
 }
 
 __tgt_target_table *__tgt_rtl_load_binary(int32_t device_id,
-                                          __tgt_device_image *image) {
+                                          __tgt_device_binary *image) {
   assert(DeviceRTL.isValidDeviceId(device_id) && "device_id is invalid");
 
   if (DeviceRTL.setContext(device_id) != OFFLOAD_SUCCESS)
     return nullptr;
 
   return DeviceRTL.loadBinary(device_id, image);
 }
 

openmp/libomptarget/plugins/generic-elf-64bit/src/rtl.cpp

 };
 
 static RTLDeviceInfoTy DeviceInfo(NUMBER_OF_DEVICES);
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *image) {
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *image) {
 // If we don't have a valid ELF ID we can just fail.
 #if TARGET_ELF_ID < 1
   return 0;
 #else
   return elf_check_machine(image, TARGET_ELF_ID);
 #endif
 }
 
 int32_t __tgt_rtl_number_of_devices() { return NUMBER_OF_DEVICES; }
 
 int32_t __tgt_rtl_init_device(int32_t device_id) { return OFFLOAD_SUCCESS; }
 
 __tgt_target_table *__tgt_rtl_load_binary(int32_t device_id,
-                                          __tgt_device_image *image) {
+                                          __tgt_device_binary *image) {
 
   DP("Dev %d: load binary from " DPxMOD " image\n", device_id,
      DPxPTR(image->ImageStart));
 
   assert(device_id >= 0 && device_id < NUMBER_OF_DEVICES && "bad dev id");
 
   size_t ImageSize = (size_t)image->ImageEnd - (size_t)image->ImageStart;
   size_t NumEntries = (size_t)(image->EntriesEnd - image->EntriesBegin);

openmp/libomptarget/plugins/remote/include/Utils.h

 /// Loads a target binary description into protobuf.
 void loadTargetBinaryDescription(const __tgt_bin_desc *Desc,
                                  TargetBinaryDescription &Request);
 
 /// Unload a target binary description from protobuf. The map is used to keep
 /// track of already copied device images.
 void unloadTargetBinaryDescription(
     const TargetBinaryDescription *Request, __tgt_bin_desc *Desc,
-    std::unordered_map<const void *, __tgt_device_image *>
+    std::unordered_map<const void *, __tgt_device_binary *>
         &HostToRemoteDeviceImage);
 
 /// Frees argument as constructed by loadTargetBinaryDescription
 void freeTargetBinaryDescription(__tgt_bin_desc *Desc);
 
 /// Copies from TargetOffloadEntry protobuf to a tgt_bin_desc during unloading.
 void copyOffloadEntry(const TargetOffloadEntry &EntryResponse,
                       __tgt_offload_entry *Entry);
 
 /// Copies from tgt_bin_desc into TargetOffloadEntry protobuf during loading.
 void copyOffloadEntry(const __tgt_offload_entry *Entry,
                       TargetOffloadEntry *EntryResponse);
 
 /// Shallow copy of offload entry from tgt_bin_desc to TargetOffloadEntry
 /// during loading.
 void shallowCopyOffloadEntry(const __tgt_offload_entry *Entry,
                              TargetOffloadEntry *EntryResponse);
 
 /// Copies DeviceOffloadEntries into table during unloading.
 void copyOffloadEntry(const DeviceOffloadEntry &EntryResponse,
                       __tgt_offload_entry *Entry);
 
 /// Loads tgt_target_table into a TargetTable protobuf message.
 void loadTargetTable(__tgt_target_table *Table, TargetTable &TableResponse,
-                     __tgt_device_image *Image);
+                     __tgt_device_binary *Image);
 
 /// Unloads from a target_table from protobuf.
 void unloadTargetTable(
     TargetTable &TableResponse, __tgt_target_table *Table,
     std::unordered_map<void *, void *> &HostToRemoteTargetTableMap);
 
 /// Frees argument as constructed by unloadTargetTable
 void freeTargetTable(__tgt_target_table *Table);
 
 void dump(const void *Start, const void *End);
 void dump(__tgt_offload_entry *Entry);
 void dump(TargetOffloadEntry Entry);
 void dump(__tgt_target_table *Table);
-void dump(__tgt_device_image *Image);
+void dump(__tgt_device_binary *Image);
 } // namespace RemoteOffloading
 
 #endif

openmp/libomptarget/plugins/remote/lib/Utils.cpp

for (auto *CurEntry = Desc->HostEntriesBegin;
     copyOffloadEntry(CurEntry, NewEntry);
 
     // Copy the pointer of the offload entry of the image into the Request
     Request.add_entry_ptrs((uint64_t)CurEntry);
     DeepCopiedEntryAddrs.push_back(CurEntry);
   }
 
   // Copy Device Images and Device Offload Entries
-  __tgt_device_image *CurImage = Desc->DeviceImages;
+  __tgt_device_binary *CurImage = Desc->DeviceImages;
   for (auto I = 0; I < Desc->NumDeviceImages; I++, CurImage++) {
     auto *Image = Request.add_images();
     auto Size = (char *)CurImage->ImageEnd - (char *)CurImage->ImageStart;
     Image->set_binary(CurImage->ImageStart, Size);
 
     // Copy the pointer of the image into the Request
     auto *NewImagePtr = Request.add_image_ptrs();
     NewImagePtr->set_img_ptr((uint64_t)CurImage->ImageStart);
for (auto *CurEntry = CurImage->EntriesBegin;
       // Copy the pointer of the offload entry of the image into the Request
       NewImagePtr->add_entry_ptrs((uint64_t)CurEntry);
     }
   }
 }
 
 void unloadTargetBinaryDescription(
     const TargetBinaryDescription *Request, __tgt_bin_desc *Desc,
-    std::unordered_map<const void *, __tgt_device_image *>
+    std::unordered_map<const void *, __tgt_device_binary *>
         &HostToRemoteDeviceImage) {
   std::unordered_map<const void *, __tgt_offload_entry *> CopiedOffloadEntries;
   Desc->NumDeviceImages = Request->images_size();
-  Desc->DeviceImages = new __tgt_device_image[Desc->NumDeviceImages];
+  Desc->DeviceImages = new __tgt_device_binary[Desc->NumDeviceImages];
 
   if (Request->entries_size())
     Desc->HostEntriesBegin = new __tgt_offload_entry[Request->entries_size()];
   else {
     Desc->HostEntriesBegin = nullptr;
     Desc->HostEntriesEnd = nullptr;
   }
 
   // Copy Global Offload Entries
   __tgt_offload_entry *CurEntry = Desc->HostEntriesBegin;
   size_t I = 0;
   for (auto &Entry : Request->entries()) {
     copyOffloadEntry(Entry, CurEntry);
     CopiedOffloadEntries[(void *)Request->entry_ptrs()[I]] = CurEntry;
     CurEntry++;
     I++;
   }
   Desc->HostEntriesEnd = CurEntry;
 
   // Copy Device Images and Device Offload Entries
-  __tgt_device_image *CurImage = Desc->DeviceImages;
+  __tgt_device_binary *CurImage = Desc->DeviceImages;
   auto ImageItr = Request->image_ptrs().begin();
   for (auto Image : Request->images()) {
     // Copy Device Offload Entries
     CurEntry = Desc->HostEntriesBegin;
     bool Found = false;
 
     if (!Desc->HostEntriesBegin) {
       CurImage->EntriesBegin = nullptr;
for (auto Image : Request->images()) {
 
     HostToRemoteDeviceImage[(void *)ImageItr->img_ptr()] = CurImage;
     CurImage++;
     ImageItr++;
   }
 }
 
 void freeTargetBinaryDescription(__tgt_bin_desc *Desc) {
-  __tgt_device_image *CurImage = Desc->DeviceImages;
+  __tgt_device_binary *CurImage = Desc->DeviceImages;
   for (auto I = 0; I < Desc->NumDeviceImages; I++, CurImage++)
     delete[](uint64_t *) CurImage->ImageStart;
 
   delete[] Desc->DeviceImages;
 
   for (auto *Entry = Desc->HostEntriesBegin; Entry != Desc->HostEntriesEnd;
        Entry++) {
     free(Entry->name);
     free(Entry->addr);
   }
 
   delete[] Desc->HostEntriesBegin;
 }
 
 void freeTargetTable(__tgt_target_table *Table) {
   for (auto *Entry = Table->EntriesBegin; Entry != Table->EntriesEnd; Entry++)
     free(Entry->name);
 
   delete[] Table->EntriesBegin;
 }
 
 void loadTargetTable(__tgt_target_table *Table, TargetTable &TableResponse,
-                     __tgt_device_image *Image) {
+                     __tgt_device_binary *Image) {
   auto *ImageEntry = Image->EntriesBegin;
   for (__tgt_offload_entry *CurEntry = Table->EntriesBegin;
        CurEntry != Table->EntriesEnd; CurEntry++, ImageEntry++) {
     // TODO: This can probably be trimmed substantially.
     auto *NewEntry = TableResponse.add_entries();
     NewEntry->set_name(CurEntry->name);
     NewEntry->set_addr((uint64_t)CurEntry->addr);
     NewEntry->set_flags(CurEntry->flags);
void dump(TargetOffloadEntry Entry) {
   fprintf(stderr, "  Name: %s\n", Entry.name().c_str());
   fprintf(stderr, "  Reserved: %d\n", Entry.reserved());
   fprintf(stderr, "  Flags: %d\n", Entry.flags());
   fprintf(stderr, "  Size:  %ld\n", Entry.data().size());
   dump(static_cast<const void *>(Entry.data().data()),
        static_cast<const void *>((Entry.data().c_str() + Entry.data().size())));
 }
 
-void dump(__tgt_device_image *Image) {
+void dump(__tgt_device_binary *Image) {
   dump(Image->ImageStart, Image->ImageEnd);
   __tgt_offload_entry *EntryItr = Image->EntriesBegin;
   for (; EntryItr != Image->EntriesEnd; EntryItr++)
     dump(EntryItr);
 }
 
 void dump(std::unordered_map<void *, __tgt_offload_entry *> &Map) {
   fprintf(stderr, "Host to Remote Entry Map:\n");
   for (auto Entry : Map)
     fprintf(stderr, "  Host (%p) -> Tgt (%p): Addr((%p))\n", Entry.first,
             (void *)Entry.second, (void *)Entry.second->addr);
 }
 } // namespace RemoteOffloading

openmp/libomptarget/plugins/remote/server/Server.h

 using namespace google;
 
 extern PluginManager *PM;
 
 class RemoteOffloadImpl final : public RemoteOffload::Service {
 private:
   int32_t mapHostRTLDeviceId(int32_t RTLDeviceID);
 
-  std::unordered_map<const void *, __tgt_device_image *>
+  std::unordered_map<const void *, __tgt_device_binary *>
       HostToRemoteDeviceImage;
   std::unordered_map<const void *, std::unique_ptr<__tgt_bin_desc>>
       Descriptions;
   __tgt_target_table *Table = nullptr;
 
   int DebugLevel;
   uint64_t MaxSize;
   uint64_t BlockSize;

openmp/libomptarget/plugins/remote/server/Server.cpp

Status RemoteOffloadImpl::UnregisterLib(ServerContext *Context,
   SERVER_DBG("Unregistered library")
   Reply->set_number(0);
   return Status::OK;
 }
 
 Status RemoteOffloadImpl::IsValidBinary(ServerContext *Context,
                                         const TargetDeviceImagePtr *DeviceImage,
                                         I32 *IsValid) {
-  __tgt_device_image *Image =
+  __tgt_device_binary *Image =
       HostToRemoteDeviceImage[(void *)DeviceImage->image_ptr()];
 
   IsValid->set_number(0);
 
   for (auto &RTL : PM->RTLs.AllRTLs)
     if (auto Ret = RTL.is_valid_binary(Image)) {
       IsValid->set_number(Ret);
       break;
Status RemoteOffloadImpl::InitRequires(ServerContext *Context,
   Reply->set_number(RequiresFlag->number());
 
   SERVER_DBG("Initialized requires for devices")
   return Status::OK;
 }
 
 Status RemoteOffloadImpl::LoadBinary(ServerContext *Context,
                                      const Binary *Binary, TargetTable *Reply) {
-  __tgt_device_image *Image =
+  __tgt_device_binary *Image =
       HostToRemoteDeviceImage[(void *)Binary->image_ptr()];
 
   Table = PM->Devices[Binary->device_id()]->RTL->load_binary(
       mapHostRTLDeviceId(Binary->device_id()), Image);
   if (Table)
     loadTargetTable(Table, *Reply, Image);
 
   SERVER_DBG("Loaded binary (%p) to device %d", (void *)Binary->image_ptr(),

openmp/libomptarget/plugins/remote/src/Client.h

for (auto &TableIt : DevicesToTables)
       freeTargetTable(TableIt.second.get());
   }
 
   int32_t shutdown(void);
 
   int32_t registerLib(__tgt_bin_desc *Desc);
   int32_t unregisterLib(__tgt_bin_desc *Desc);
 
-  int32_t isValidBinary(__tgt_device_image *Image);
+  int32_t isValidBinary(__tgt_device_binary *Image);
   int32_t getNumberOfDevices();
 
   int32_t initDevice(int32_t DeviceId);
   int32_t initRequires(int64_t RequiresFlags);
 
-  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_image *Image);
+  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_binary *Image);
 
   void *dataAlloc(int32_t DeviceId, int64_t Size, void *HstPtr);
   int32_t dataDelete(int32_t DeviceId, void *TgtPtr);
 
   int32_t dataSubmit(int32_t DeviceId, void *TgtPtr, void *HstPtr,
                      int64_t Size);
   int32_t dataRetrieve(int32_t DeviceId, void *HstPtr, void *TgtPtr,
                        int64_t Size);
RemoteClientManager() {
     }
   }
 
   int32_t shutdown(void);
 
   int32_t registerLib(__tgt_bin_desc *Desc);
   int32_t unregisterLib(__tgt_bin_desc *Desc);
 
-  int32_t isValidBinary(__tgt_device_image *Image);
+  int32_t isValidBinary(__tgt_device_binary *Image);
   int32_t getNumberOfDevices();
 
   int32_t initDevice(int32_t DeviceId);
   int32_t initRequires(int64_t RequiresFlags);
 
-  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_image *Image);
+  __tgt_target_table *loadBinary(int32_t DeviceId, __tgt_device_binary *Image);
 
   void *dataAlloc(int32_t DeviceId, int64_t Size, void *HstPtr);
   int32_t dataDelete(int32_t DeviceId, void *TgtPtr);
 
   int32_t dataSubmit(int32_t DeviceId, void *TgtPtr, void *HstPtr,
                      int64_t Size);
   int32_t dataRetrieve(int32_t DeviceId, void *HstPtr, void *TgtPtr,
                        int64_t Size);

openmp/libomptarget/plugins/remote/src/Client.cpp

return remoteCall(
           return 0;
         }
         CLIENT_DBG("Failed to unregister library")
         return 1;
       },
       /* Error Value */ 1);
 }
 
-int32_t RemoteOffloadClient::isValidBinary(__tgt_device_image *Image) {
+int32_t RemoteOffloadClient::isValidBinary(__tgt_device_binary *Image) {
   return remoteCall(
       /* Preprocessor */
       [&](auto &RPCStatus, auto &Context) {
         auto *Request =
             protobuf::Arena::CreateMessage<TargetDeviceImagePtr>(Arena.get());
         auto *Reply = protobuf::Arena::CreateMessage<I32>(Arena.get());
 
         Request->set_image_ptr((uint64_t)Image->ImageStart);
return remoteCall(
         } else {
           CLIENT_DBG("Could not initialize requires")
         }
         return Reply->number();
       },
       /* Error Value */ -1);
 }
 
-__tgt_target_table *RemoteOffloadClient::loadBinary(int32_t DeviceId,
-                                                    __tgt_device_image *Image) {
+__tgt_target_table *
+RemoteOffloadClient::loadBinary(int32_t DeviceId, __tgt_device_binary *Image) {
   return remoteCall(
       /* Preprocessor */
       [&](auto &RPCStatus, auto &Context) {
         auto *ImageMessage =
             protobuf::Arena::CreateMessage<Binary>(Arena.get());
         auto *Reply = protobuf::Arena::CreateMessage<TargetTable>(Arena.get());
         ImageMessage->set_image_ptr((uint64_t)Image->ImageStart);
         ImageMessage->set_device_id(DeviceId);

 
 int32_t RemoteClientManager::unregisterLib(__tgt_bin_desc *Desc) {
   int32_t Ret = 0;
   for (auto &Client : Clients)
     Ret &= Client.unregisterLib(Desc);
   return Ret;
 }
 
-int32_t RemoteClientManager::isValidBinary(__tgt_device_image *Image) {
+int32_t RemoteClientManager::isValidBinary(__tgt_device_binary *Image) {
   int32_t ClientIdx = 0;
   for (auto &Client : Clients) {
     if (auto Ret = Client.isValidBinary(Image))
       return Ret;
     ClientIdx++;
   }
   return 0;
 }

 
 int32_t RemoteClientManager::initRequires(int64_t RequiresFlags) {
   for (auto &Client : Clients)
     Client.initRequires(RequiresFlags);
 
   return RequiresFlags;
 }
 
-__tgt_target_table *RemoteClientManager::loadBinary(int32_t DeviceId,
-                                                    __tgt_device_image *Image) {
+__tgt_target_table *
+RemoteClientManager::loadBinary(int32_t DeviceId, __tgt_device_binary *Image) {
   int32_t ClientIdx, DeviceIdx;
   std::tie(ClientIdx, DeviceIdx) = mapDeviceId(DeviceId);
   return Clients[ClientIdx].loadBinary(DeviceIdx, Image);
 }
 
 int32_t RemoteClientManager::isDataExchangeable(int32_t SrcDevId,
                                                 int32_t DstDevId) {
   int32_t SrcClientIdx, SrcDeviceIdx, DstClientIdx, DstDeviceIdx;

openmp/libomptarget/plugins/remote/src/rtl.cpp

 int32_t __tgt_rtl_register_lib(__tgt_bin_desc *Desc) {
   return Manager->registerLib(Desc);
 }
 
 int32_t __tgt_rtl_unregister_lib(__tgt_bin_desc *Desc) {
   return Manager->unregisterLib(Desc);
 }
 
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *Image) {
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *Image) {
   return Manager->isValidBinary(Image);
 }
 
 int32_t __tgt_rtl_number_of_devices() { return Manager->getNumberOfDevices(); }
 
 int32_t __tgt_rtl_init_device(int32_t DeviceId) {
   return Manager->initDevice(DeviceId);
 }
 
 int64_t __tgt_rtl_init_requires(int64_t RequiresFlags) {
   return Manager->initRequires(RequiresFlags);
 }
 
 __tgt_target_table *__tgt_rtl_load_binary(int32_t DeviceId,
-                                          __tgt_device_image *Image) {
-  return Manager->loadBinary(DeviceId, (__tgt_device_image *)Image);
+                                          __tgt_device_binary *Image) {
+  return Manager->loadBinary(DeviceId, (__tgt_device_binary *)Image);
 }
 
 int32_t __tgt_rtl_is_data_exchangable(int32_t SrcDevId, int32_t DstDevId) {
   return Manager->isDataExchangeable(SrcDevId, DstDevId);
 }
 
 void *__tgt_rtl_data_alloc(int32_t DeviceId, int64_t Size, void *HstPtr,
                            int32_t Kind) {

openmp/libomptarget/plugins/ve/src/rtl.cpp

 // target RTL.
 int32_t __tgt_rtl_number_of_devices(void) { return DeviceInfo.NodeIds.size(); }
 
 // Return an integer different from zero if the provided device image can be
 // supported by the runtime. The functionality is similar to comparing the
 // result of __tgt__rtl__load__binary to NULL. However, this is meant to be a
 // lightweight query to determine if the RTL is suitable for an image without
 // having to load the library, which can be expensive.
-int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *Image) {
+int32_t __tgt_rtl_is_valid_binary(__tgt_device_binary *Image) {
 #if TARGET_ELF_ID < 1
   return 0;
 #else
   return elf_check_machine(Image, TARGET_ELF_ID);
 #endif
 }
 
 // Initialize the specified device. In case of success return 0; otherwise

 }
 
 // Pass an executable image section described by image to the specified
 // device and prepare an address table of target entities. In case of error,
 // return NULL. Otherwise, return a pointer to the built address table.
 // Individual entries in the table may also be NULL, when the corresponding
 // offload region is not supported on the target device.
 __tgt_target_table *__tgt_rtl_load_binary(int32_t ID,
-                                          __tgt_device_image *Image) {
+                                          __tgt_device_binary *Image) {
   DP("Dev %d: load binary from " DPxMOD " image\n", ID,
      DPxPTR(Image->ImageStart));
 
   assert(ID >= 0 && "bad dev id");
 
   size_t ImageSize = (size_t)Image->ImageEnd - (size_t)Image->ImageStart;
   size_t NumEntries = (size_t)(Image->EntriesEnd - Image->EntriesBegin);
   DP("Expecting to have %zd entries defined.\n", NumEntries);

openmp/libomptarget/src/omptarget.cpp

for (auto *HostEntriesBegin : PM->HostEntriesBeginRegistrationOrder) {
       }
 
       if (TransTable->TargetsTable[device_id] != 0) {
         // Library entries have already been processed
         continue;
       }
 
       // 1) get image.
-      assert(TransTable->TargetsImages.size() > (size_t)device_id &&
+      assert(TransTable->TargetsBinaries.size() > (size_t)device_id &&
              "Not expecting a device ID outside the table's bounds!");
-      __tgt_device_image *img = TransTable->TargetsImages[device_id];
+      __tgt_device_binary *img = TransTable->TargetsBinaries[device_id];
       if (!img) {
         REPORT("No image loaded for device id %d.\n", device_id);
         rc = OFFLOAD_FAIL;
         break;
       }
       // 2) load image into the target table.
       __tgt_target_table *TargetTable = TransTable->TargetsTable[device_id] =
           Device.load_binary(img);
       // Unable to get table for this image: invalidate image and fail.
       if (!TargetTable) {
         REPORT("Unable to generate entries table for device id %d.\n",
                device_id);
-        TransTable->TargetsImages[device_id] = 0;
+        TransTable->TargetsBinaries[device_id] = 0;
         rc = OFFLOAD_FAIL;
         break;
       }
 
       // Verify whether the two table sizes match.
       size_t hsize =
           TransTable->HostTable.EntriesEnd - TransTable->HostTable.EntriesBegin;
       size_t tsize = TargetTable->EntriesEnd - TargetTable->EntriesBegin;
 
       // Invalid image for these host entries!
       if (hsize != tsize) {
         REPORT(
             "Host and Target tables mismatch for device id %d [%zx != %zx].\n",
             device_id, hsize, tsize);
-        TransTable->TargetsImages[device_id] = 0;
+        TransTable->TargetsBinaries[device_id] = 0;
         TransTable->TargetsTable[device_id] = 0;
         rc = OFFLOAD_FAIL;
         break;
       }
 
       DeviceTy::HDTTMapAccessorTy HDTTMap =
           Device.HostDataToTargetMap.getExclusiveAccessor();
 

openmp/libomptarget/src/rtl.cpp

 //===----------- rtl.cpp - Target independent OpenMP target RTL -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Functionality for handling RTL plugins.
 //
 //===----------------------------------------------------------------------===//
 
 #include "rtl.h"
 #include "device.h"
 #include "private.h"
 
 #include <cassert>
 #include <cstdlib>
 #include <cstring>
 #include <dlfcn.h>
 #include <mutex>
 #include <string>
 
#endif
   return;
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // Functionality for registering libs
 
 static void RegisterImageIntoTranslationTable(TranslationTable &TT,
                                               RTLInfoTy &RTL,
-                                              __tgt_device_image *image) {
+                                              __tgt_device_binary *img) {
 
   // same size, as when we increase one, we also increase the other.
-  assert(TT.TargetsTable.size() == TT.TargetsImages.size() &&
+  assert(TT.TargetsTable.size() == TT.TargetsBinaries.size() &&
          "We should have as many images as we have tables!");
 
   // Resize the Targets Table and Images to accommodate the new targets if
   // required
   unsigned TargetsTableMinimumSize = RTL.Idx + RTL.NumberOfDevices;
 
   if (TT.TargetsTable.size() < TargetsTableMinimumSize) {
-    TT.TargetsImages.resize(TargetsTableMinimumSize, 0);
+    TT.TargetsBinaries.resize(TargetsTableMinimumSize, 0);
     TT.TargetsTable.resize(TargetsTableMinimumSize, 0);
   }
 
   // Register the image in all devices for this target type.
   for (int32_t i = 0; i < RTL.NumberOfDevices; ++i) {
     // If we are changing the image we are also invalidating the target table.
-    if (TT.TargetsImages[RTL.Idx + i] != image) {
-      TT.TargetsImages[RTL.Idx + i] = image;
+    if (TT.TargetsBinaries[RTL.Idx + i] != img) {
+      TT.TargetsBinaries[RTL.Idx + i] = img;
       TT.TargetsTable[RTL.Idx + i] = 0; // lazy initialization of target table.
     }
   }
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // Functionality for registering Ctors/Dtors
 
 static void RegisterGlobalCtorsDtorsForImage(__tgt_bin_desc *desc,
-                                             __tgt_device_image *img,
+                                             __tgt_device_binary *img,
                                              RTLInfoTy *RTL) {
 
   for (int32_t i = 0; i < RTL->NumberOfDevices; ++i) {
     DeviceTy &Device = *PM->Devices[RTL->Idx + i];
     Device.PendingGlobalsMtx.lock();
     Device.HasPendingGlobals = true;
     for (__tgt_offload_entry *entry = img->EntriesBegin;
          entry != img->EntriesEnd; ++entry) {
for (__tgt_offload_entry *entry = img->EntriesBegin;
       if (entry->flags & OMP_DECLARE_TARGET_LINK) {
         DP("The \"link\" attribute is not yet supported!\n");
       }
     }
     Device.PendingGlobalsMtx.unlock();
   }
 }
 
+static __tgt_device_binary parseDeviceImage(__tgt_device_image *Image) {
+  struct Header {
+    uint8_t Magic[4] = {0x10, 0xFF, 0x10, 0xAD}; // 0x10FF10AD magic bytes.
+    uint32_t Version = 1;                        // Version identifier.
+    uint64_t Size;        // Size in bytes of this entire binary.
+    uint64_t EntryOffset; // Offset of the metadata entry in bytes.
+    uint64_t EntrySize;   // Size of the metadata entry in bytes.
+  };
+
+  struct Entry {
+    uint16_t TheImageKind;   // The kind of the image stored.
+    uint16_t TheOffloadKind; // The producer of this image.
+    uint32_t Flags;          // Additional flags associated with the image.
+    uint64_t StringOffset;   // Offset in bytes to the string map.
+    uint64_t NumStrings;     // Number of entries in the string map.
+    uint64_t ImageOffset;    // Offset in bytes of the actual binary image.
+    uint64_t ImageSize;      // Size in bytes of the binary image.
+  };
+
+  struct StringEntry {
+    uint64_t KeyOffset;   // Offset of the key in the string table.
+    uint64_t ValueOffset; // Offset of the value in the string table.
+  };
+
+  char *Buffer = reinterpret_cast<char *>(Image->ImageStart);
+  // If this is not a bundled offloading binary just return the Image values.
+  if (strncmp(Buffer, "\x10\xFF\x10\xAD", sizeof(Header::Magic))) {
+    __tgt_device_binary Binary{};
+    Binary.ImageStart = Image->ImageStart;
+    Binary.ImageEnd = Image->ImageEnd;
+    Binary.EntriesBegin = Image->EntriesBegin;
+    Binary.EntriesEnd = Image->EntriesEnd;
+    return Binary;
+  }
+
+  const Header *TheHeader = reinterpret_cast<const Header *>(Buffer);
+  const Entry *TheEntry =
+      reinterpret_cast<const Entry *>(&Buffer[TheHeader->EntryOffset]);
+
+  assert(TheHeader->Version == 1 && "Incompatible version");
+
+  __tgt_device_binary Binary{};
+  Binary.ImageStart = &Buffer[TheEntry->ImageOffset];
+  Binary.ImageEnd = &Buffer[TheEntry->ImageOffset] + TheEntry->ImageSize;
+  Binary.EntriesBegin = Image->EntriesBegin;
+  Binary.EntriesEnd = Image->EntriesEnd;
+
+  const StringEntry *StringMapBegin =
+      reinterpret_cast<const StringEntry *>(&Buffer[TheEntry->StringOffset]);
+  for (uint64_t I = 0, E = TheEntry->NumStrings; I != E; ++I) {
+    const char *Key = &Buffer[StringMapBegin[I].KeyOffset];
+    const char *Value = &Buffer[StringMapBegin[I].ValueOffset];
+    if (strcmp(Key, "arch") == 0)
+      Binary.Info.Arch = Value;
+  }
+
+  return Binary;
+}
+
 void RTLsTy::RegisterRequires(int64_t flags) {
   // TODO: add more elaborate check.
   // Minimal check: only set requires flags if previous value
   // is undefined. This ensures that only the first call to this
   // function will set the requires flags. All subsequent calls
   // will be checked for compatibility.
   assert(flags != OMP_REQ_UNDEFINED &&
          "illegal undefined flag for requires directive!");
void RTLsTy::initAllRTLs() {
   for (auto &R : AllRTLs)
     initRTLonce(R);
 }
 
 void RTLsTy::RegisterLib(__tgt_bin_desc *desc) {
   PM->RTLsMtx.lock();
   // Register the images with the RTLs that understand them, if any.
   for (int32_t i = 0; i < desc->NumDeviceImages; ++i) {
-    // Obtain the image.
-    __tgt_device_image *img = &desc->DeviceImages[i];
+    PM->Binaries.emplace_back(parseDeviceImage(&desc->DeviceImages[i]));
+    __tgt_device_binary *img = &PM->Binaries.back();
 
     RTLInfoTy *FoundRTL = nullptr;
 
     // Scan the RTLs that have associated images until we find one that supports
     // the current image.
     for (auto &R : AllRTLs) {
       if (!R.is_valid_binary(img)) {
         DP("Image " DPxMOD " is NOT compatible with RTL %s!\n",
void RTLsTy::RegisterLib(__tgt_bin_desc *desc) {
   DP("Done registering entries!\n");
 }
 
 void RTLsTy::UnregisterLib(__tgt_bin_desc *desc) {
   DP("Unloading target library!\n");
 
   PM->RTLsMtx.lock();
   // Find which RTL understands each image, if any.
-  for (int32_t i = 0; i < desc->NumDeviceImages; ++i) {
+  for (int32_t i = 0; i < PM->Binaries.size(); ++i) {
     // Obtain the image.
-    __tgt_device_image *img = &desc->DeviceImages[i];
+    __tgt_device_binary *img = &PM->Binaries[i];
 
     RTLInfoTy *FoundRTL = NULL;
 
     // Scan the RTLs that have associated images until we find one that supports
     // the current image. We only need to scan RTLs that are already being used.
     for (auto *R : UsedRTLs) {
 
       assert(R->isUsed && "Expecting used RTLs.");
 
       if (!R->is_valid_binary(img)) {
         DP("Image " DPxMOD " is NOT compatible with RTL " DPxMOD "!\n",
            DPxPTR(img->ImageStart), DPxPTR(R->LibraryHandler));
         continue;
       }
 
       DP("Image " DPxMOD " is compatible with RTL " DPxMOD "!\n",
          DPxPTR(img->ImageStart), DPxPTR(R->LibraryHandler));
 
       FoundRTL = R;
 
if (TransTable != PM->HostEntriesBeginToTransTable.end()) {
     DP("Translation table for descriptor " DPxMOD " cannot be found, probably "
        "it has been already removed.\n",
        DPxPTR(desc->HostEntriesBegin));
   }
 
   PM->TblMapMtx.unlock();
 
   // TODO: Remove RTL and the devices it manages if it's not used anymore?
   // TODO: Write some RTL->unload_image(...) function?
 
   DP("Done unregistering library!\n");
 }