[OpenMP][WIP] Pass auxiliary YAML along with device image for OpenMP offload.

[vzakhari]

Summary

This is a RFC for bundling a YAML file along with the device image in clang-offload-wrapper, so that auxiliary information is passed to the offload plugins.

Currently, many plugins expect that the device image is an ELF image, so they look for the ELF magic word and go from there. This change-set puts extra data before the actual image like this (see code in ClangOffloadWrapper.cpp):

// Insert 1OMP header magic word.
// The expected image structure is like this:
//   struct {
//     char MagicWord[ONEOMPMAGLEN]; // "1OMP"
//     uint32_t HeaderSize;          // In little-endian.
//     char Header[HeaderSize];      // Uncompressed YAML with auxiliary
//                                   // information.
//     char ActualImage[];           // ELF or another image.
//   }

For backward compatibility, plugins have to continue working with plain ELF images. If 1OMP magic word is detected, then the corresponding Header has to be read as an YAML file with some mandatory keys.

As long as unification between multiple offload targets is not practical, vendors/plugins may use optional keys (such as CompileOptions for cuModuleLoadDataEx()) to convey auxiliary information from the source code or compiler options to the device specific plugins.

Main concerns about this change-set are the dependency on LLVMSupport (requiring pre-build llvm project) and the final size of the plugins (due to LLVMSupport linking).

Just for an example, the plugins will see this auxiliary data like this:

TARGET x86_64 RTL --> 1OMP header: HeaderSize(133), HeaderBegin(0x00000000004009f8), ImageBegin(0x0000000000400a7d), ImageEnd(0x0000000000402ba5)
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <CompileOptions: [
TARGET x86_64 RTL --> ,
TARGET x86_64 RTL --> ]>
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <LinkOptions: [
TARGET x86_64 RTL --> ,
TARGET x86_64 RTL --> ]>
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <Producer: [
TARGET x86_64 RTL --> unknown,
TARGET x86_64 RTL --> ]>
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <TargetID: [
TARGET x86_64 RTL --> ,
TARGET x86_64 RTL --> ]>
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <VendorId: [
TARGET x86_64 RTL --> 9,
TARGET x86_64 RTL --> ]>
TARGET x86_64 RTL --> Read OpenMPAttr:
TARGET x86_64 RTL --> <VendorName: [
TARGET x86_64 RTL --> llvm,
TARGET x86_64 RTL --> ]>

[jdoerfert]

vzakhari removed a reviewer: jdoerfert.

My herald rules are very annoying sometimes, apologies. Can we discuss this tomorrow? I put it on the agenda https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing

[grokos]

In D95483#2524232, @jdoerfert wrote:

My herald rules are very annoying sometimes, apologies. Can we discuss this tomorrow? I put it on the agenda https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing

The patch was uploaded to serve as a base for our discussions :)

[JonChesterfield]

There are a bunch of side channels for communicating between the compiler and the runtime already. Compiler emitted magic symbols, a msgpack blob for amdgpu, writing extra stuff into the elf in general etc.

What information do you intend to pass along through this new channel, why yaml, and why for all plugins?

[RaviNarayanaswamy]

In D95483#2525182, @JonChesterfield wrote:

There are a bunch of side channels for communicating between the compiler and the runtime already. Compiler emitted magic symbols, a msgpack blob for amdgpu, writing extra stuff into the elf in general etc.

What information do you intend to pass along through this new channel, why yaml, and why for all plugins?

Yes there may be many ways to communicate information from compiler to runtime. The proposal is one way and we others wanted to communicate information to runtime and come up with a standard scheme.
We added this to other plugins to show how it can be done. Plugins can opt in,, just dont add the unique signature and the header.

[vzakhari]

In D95483#2525899, @RaviNarayanaswamy wrote:

In D95483#2525182, @JonChesterfield wrote:

There are a bunch of side channels for communicating between the compiler and the runtime already. Compiler emitted magic symbols, a msgpack blob for amdgpu, writing extra stuff into the elf in general etc.

What information do you intend to pass along through this new channel, why yaml, and why for all plugins?

Yes there may be many ways to communicate information from compiler to runtime. The proposal is one way and we others wanted to communicate information to runtime and come up with a standard scheme.
We added this to other plugins to show how it can be done. Plugins can opt in,, just dont add the unique signature and the header.

Right, I am wondering if we can replace all the existing side channels with a unified one. But there is always an option for a plugin/vendor to use the existing methods. Code in clang-offload-wrapper is not supposed to be enabled by default, so there will be some mechanism to switch between the current and the new wrapping. A vendor-specific toolchain will be able to invoke clang-offload-wrapper as it wants.

So to answer "what information" question: all the existing information (if possible; if not possible - I would like to understand why) and more along the road, e.g. compilation/linking options for OpenCL/CUDA backends, TargetIDs for alternative images targeting different versions of a target device, ProducerID for identifying not backward compatible changes in plugins, etc.

I chose YAML just because it is human readable and easily editable format with existing support in LLVMSupport library.

[vzakhari]

I looked through the AMDGCN offload toolchain, and I think we should be able to use ELF for embedding the auxiliary information just the same way msgpack is used for AMDGCN offload. We can also use some special PT_NOTE section for common things like vendorid/vendorname/producer. I believe the targetid may be represented with e_machine and probably additional auxiliary information in vendor specific PT_NOTE section(s) (if needed at all).

I suggest adding a PT_NOTE section with new type (e.g. NT_OPENMP_METADATA) with the name matching one of the vendor-name values from table 1.2 OpenMP Additional Definitions document. The desc of this section contains a contiguous "list" of null-terminated strings. First string specifies the OpenMP metadata version (the structure of the desc), i.e. 1.0 for the initial version. Version 1.0 has only one additional string in the "list", which is the producer information (interpreted in a vendor-speific way, e.g. for llvm we could embed LLVM version and a "version" of OpenMP toolchain, which is clang-offload-bundler, clang-offload-wrapper, libomptarget, etc.).

Maybe using just LLVM version is enough: I suppose the minor version of LLVM will change if any changes to the OpenMP toolchain are made, so we can use LLVM version, if needed, to check for compatibility issues. Alternatively, we can use an OMP toolchain version and make it independent of LLVM releases.

namesz	#
descsz	#
type	NT_OPENMP_METADATA
name	llvm\0<pad>
desc	1.0\0LLVM12.0.0/OMP1.0.0\0<pad>

For the rest of the auxiliary information (whatever it is) vendor specific toolchains and implementations may use additional PT_NOTE sections (just as AMDGCN).

I have not found anything like PT_NOTE in COFF and supporting two formats in the toolchain is not very convenient, so I think it makes sense to use ELF for Windows as well. This seems to require getting rid of libelf dependency in all plugins. We can probably use LLVMObject implementation.

[vzakhari]

Hi @gregrodgers, I am trying to comprehend the example that you showed on the meeting, where you had two offload images compiled for different archs. I wonder how this should work when we execute on a system where the actual GPU can support/execute both images. Currently, an offload image is registered with an RTL whenever is_valid_binary returns true. So it looks like both such images will be registered with your RTL, but they are basically for the same OpenMP program, so it is not clear how they should interact. Obviously, they cannot be linked in any way in runtime, since they are for the same OpenMP program. This is offtopic, though.

What do you think about using PT_NOTE/SHT_NOTE for keeping the additional information with the offload image?

[vzakhari]

New implementation is here: https://reviews.llvm.org/D99360

[grokos]

In D95483#2651316, @vzakhari wrote:

New implementation is here: https://reviews.llvm.org/D99360

Are we abandoning this patch then? Can you close it?

clang/tools/clang-offload-wrapper/ClangOffloadWrapper.cpp

 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Support/WithColor.h"
+#include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 #include <cassert>
 #include <cstdint>
 
 using namespace llvm;
 
 static cl::opt<bool> Help("h", cl::desc("Alias for -help"), cl::Hidden);
static cl::list<std::string> Inputs(cl::Positional, cl::OneOrMore,
                                     cl::desc("<input files>"),
                                     cl::cat(ClangOffloadWrapperCategory));
 
 static cl::opt<std::string>
     Target("target", cl::Required,
            cl::desc("Target triple for the output module"),
            cl::value_desc("triple"), cl::cat(ClangOffloadWrapperCategory));
 
+// FIXME: set version in CMake.
+#define getOpenMPOffloadVersionString() std::string("unknown")
+
+LLVM_YAML_STRONG_TYPEDEF(std::string, OpenMPAttrTy)
+LLVM_YAML_IS_SEQUENCE_VECTOR(OpenMPAttrTy)
+
+namespace llvm {
+namespace yaml {
+template <> struct ScalarTraits<OpenMPAttrTy> {
+  static void output(const OpenMPAttrTy &Val, void *, raw_ostream &Out) {
+    Out << Val;
+  }
+  static StringRef input(StringRef Scalar, void *, OpenMPAttrTy &Val) {
+    Val = Scalar.str();
+    return StringRef();
+  }
+  static QuotingType mustQuote(StringRef) { return QuotingType::Double; }
+};
+} // end namespace yaml
+} // end namespace llvm
+
+typedef SmallVector<OpenMPAttrTy, 1> OpenMPAttrListTy;
+LLVM_YAML_IS_STRING_MAP(OpenMPAttrListTy)
+
+typedef std::map<std::string, OpenMPAttrListTy> OpenMPAttrDocTy;
+LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(OpenMPAttrDocTy)
+
 namespace {
 
 class BinaryWrapper {
   LLVMContext C;
   Module M;
 
   StructType *EntryTy = nullptr;
   StructType *ImageTy = nullptr;
GlobalVariable *createBinDesc(ArrayRef<ArrayRef<char>> Bufs) {
 
     auto *Zero = ConstantInt::get(getSizeTTy(), 0u);
     Constant *ZeroZero[] = {Zero, Zero};
 
     // Create initializer for the images array.
     SmallVector<Constant *, 4u> ImagesInits;
     ImagesInits.reserve(Bufs.size());
     for (ArrayRef<char> Buf : Bufs) {
-      auto *Data = ConstantDataArray::get(C, Buf);
+      SmallVector<char, 256> ImageBuf;
+
+      // Insert 1OMP header magic word.
+      // The expected image structure is like this:
+      //   struct {
+      //     char MagicWord[ONEOMPMAGLEN]; // "1OMP"
+      //     uint32_t HeaderSize;          // In little-endian.
+      //     char Header[HeaderSize];      // Uncompressed YAML with auxiliary
+      //                                   // information.
+      //     char ActualImage[];           // ELF or another image.
+      //   }
+      StringRef Header("1OMP");
+      ImageBuf.append(Header.begin(), Header.end());
+      // Reserve 4 bytes for the header size.
+      ImageBuf.append(4, '\0');
+
+      size_t HeaderBegin = ImageBuf.size();
+      raw_svector_ostream YamlOS(ImageBuf);
+      yaml::Output YamlOut(YamlOS, NULL, 0);
+
+      SmallVector<OpenMPAttrDocTy, 1> Attrs;
+      Attrs.push_back(OpenMPAttrDocTy());
+      OpenMPAttrDocTy &Doc = Attrs[0];
+      // TODO: different vendors will use different names
+      //       from OpenMP API Additional Definitions 2.0.
+      Doc["VendorName"].push_back(std::string("llvm"));
+      Doc["VendorId"].push_back(std::string("9"));
+      Doc["Producer"].push_back(getOpenMPOffloadVersionString());
+      // TODO: support clang-offload-wrapper option that specifies
+      //       target triple per the offload image.
+      Doc["TargetID"].push_back(std::string(""));
+      // TODO: support clang-offload-wrapper options that specify
+      //       CompileOptions/LinkOptions per the offload image.
+      //       Alternatively, driver may pass a YAML file specifying
+      //       the whole context for the wrapping that needs to be done.
+      Doc["CompileOptions"].push_back(std::string(""));
+      Doc["LinkOptions"].push_back(std::string(""));
+
+      YamlOut << Attrs;
+
+      size_t HeaderEnd = ImageBuf.size();
+      size_t HeaderSize = HeaderEnd - HeaderBegin;
+      assert(HeaderSize <= (std::numeric_limits<uint32_t>::max)() &&
+             "1OMP header is too big.");
+      // Set proper header size, which is in little-endian format.
+      ImageBuf[HeaderBegin - 4] = (HeaderSize & 0xFF);
+      ImageBuf[HeaderBegin - 3] = ((HeaderSize >> 8) & 0xFF);
+      ImageBuf[HeaderBegin - 2] = ((HeaderSize >> 16) & 0xFF);
+      ImageBuf[HeaderBegin - 1] = ((HeaderSize >> 24) & 0xFF);
+
+      const char *BufStart = Buf.data();
+      size_t BufSize = Buf.size();
+      if (ImageBuf.size() > 0) {
+        // Concatenate data.
+        //
+        // FIXME: we have to find a way to avoid copying the image
+        //        twice.
+        ImageBuf.append(Buf.begin(), Buf.end());
+        BufStart = ImageBuf.data();
+        BufSize = ImageBuf.size();
+      }
+
+      auto *Data = ConstantDataArray::get(C, makeArrayRef(BufStart, BufSize));
       auto *Image = new GlobalVariable(M, Data->getType(), /*isConstant*/ true,
                                        GlobalVariable::InternalLinkage, Data,
                                        ".omp_offloading.device_image");
       Image->setUnnamedAddr(GlobalValue::UnnamedAddr::Global);
 
-      auto *Size = ConstantInt::get(getSizeTTy(), Buf.size());
+      auto *Size = ConstantInt::get(getSizeTTy(), BufSize);
       Constant *ZeroSize[] = {Zero, Size};
 
       auto *ImageB = ConstantExpr::getGetElementPtr(Image->getValueType(),
                                                     Image, ZeroZero);
       auto *ImageE = ConstantExpr::getGetElementPtr(Image->getValueType(),
                                                     Image, ZeroSize);
 
       ImagesInits.push_back(ConstantStruct::get(getDeviceImageTy(), ImageB,

openmp/libomptarget/plugins/CMakeLists.txt

if(LIBOMPTARGET_DEP_LIBFFI_FOUND)
       target_link_libraries(
         "omptarget.rtl.${tmachine_libname}"
         elf_common
         ${LIBOMPTARGET_DEP_LIBFFI_LIBRARIES}
         ${LIBOMPTARGET_DEP_LIBELF_LIBRARIES}
         dl
         "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../exports")
 
+      if (NOT OPENMP_STANDALONE_BUILD)
+        # FIXME: we should probably require pre-built LLVM project
+        #        to be available, otherwise, in-tree and out-of-tree
+        #        builds of libomptarget will behave quite differently.
+        #        Alternatively, we can link to some YAML parsing library
+        #        statically, and require it to be available during
+        #        libomptarget build (e.g. LibYAML is under MIT license).
+        target_link_libraries(
+          "omptarget.rtl.${tmachine_libname}"
+          LLVMSupport
+          )
+      endif()
+
       list(APPEND LIBOMPTARGET_TESTED_PLUGINS
         "omptarget.rtl.${tmachine_libname}")
 
       # Report to the parent scope that we are building a plugin.
       set(LIBOMPTARGET_SYSTEM_TARGETS
         "${LIBOMPTARGET_SYSTEM_TARGETS} ${tmachine_triple}" PARENT_SCOPE)
       set(LIBOMPTARGET_TESTED_PLUGINS
         "${LIBOMPTARGET_TESTED_PLUGINS}" PARENT_SCOPE)
 
     else(LIBOMPTARGET_DEP_LIBFFI_FOUND)
       libomptarget_say("Not building ${tmachine_name} offloading plugin: libffi dependency not found.")
     endif(LIBOMPTARGET_DEP_LIBFFI_FOUND)
   else(LIBOMPTARGET_DEP_LIBELF_FOUND)
     libomptarget_say("Not building ${tmachine_name} offloading plugin: libelf dependency not found.")
   endif(LIBOMPTARGET_DEP_LIBELF_FOUND)
 else()
   libomptarget_say("Not building ${tmachine_name} offloading plugin: machine not found in the system.")
 endif()
 endmacro()
 
+if (NOT OPENMP_STANDALONE_BUILD)
+  add_definitions(-DOPENMP_MAY_USE_LLVM_SUPPORT)
+endif()
+
 add_subdirectory(aarch64)
 add_subdirectory(cuda)
 add_subdirectory(ppc64)
 add_subdirectory(ppc64le)
 add_subdirectory(ve)
 add_subdirectory(x86_64)
 add_subdirectory(remote)
 
 # Make sure the parent scope can see the plugins that will be created.
 set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS}" PARENT_SCOPE)
 set(LIBOMPTARGET_TESTED_PLUGINS "${LIBOMPTARGET_TESTED_PLUGINS}" PARENT_SCOPE)
 

openmp/libomptarget/plugins/amdgpu/src/rtl.cpp

 }
 void finiAsyncInfoPtr(__tgt_async_info *async_info_ptr) {
   assert(async_info_ptr);
   assert(async_info_ptr->Queue);
   async_info_ptr->Queue = 0;
 }
 
 bool elf_machine_id_is_amdgcn(__tgt_device_image *image) {
+  // TODO: this AuxInfo is just being discarded,
+  //       and we need to do something better.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+  if (!getAuxInfo(image, AuxInfo))
+    return 0;
+  dumpAuxInfo(AuxInfo);
+  // TODO: VendorName, VendorID, Producer and TargetID checks may be done here
+  //       by particular plugins (e.g. via calling specific API).
   const uint16_t amdgcnMachineID = 224; // EM_AMDGPU may not be in system elf.h
   int32_t r = elf_check_machine(image, amdgcnMachineID);
   if (!r) {
     DP("Supported machine ID not found\n");
   }
   return r;
 }
 
 uint32_t elf_e_flags(__tgt_device_image *image) {
-  char *img_begin = (char *)image->ImageStart;
+  char *img_begin = getImageStart(image);
   size_t img_size = (char *)image->ImageEnd - img_begin;
 
   Elf *e = elf_memory(img_begin, img_size);
   if (!e) {
     DP("Unable to get ELF handle: %s!\n", elf_errmsg(-1));
     return 0;
   }
 
device_environment(int device_id, int number_devices,
     host_device_env.device_num = device_id;
     host_device_env.debug_level = 0;
 #ifdef OMPTARGET_DEBUG
     if (char *envStr = getenv("LIBOMPTARGET_DEVICE_RTL_DEBUG")) {
       host_device_env.debug_level = std::stoi(envStr);
     }
 #endif
 
-    int rc = get_symbol_info_without_loading((char *)image->ImageStart,
+    int rc = get_symbol_info_without_loading(getImageStart(image),
                                              img_size, sym(), &si);
     if (rc != 0) {
       DP("Finding global device environment '%s' - symbol missing.\n", sym());
       return;
     }
 
     if (si.size > sizeof(host_device_env)) {
       DP("Symbol '%s' has size %u, expected at most %zu.\n", sym(), si.size,
          sizeof(host_device_env));
       return;
     }
 
     valid = true;
   }
 
   bool in_image() { return si.sh_type != SHT_NOBITS; }
 
   atmi_status_t before_loading(void *data, size_t size) {
     assert(valid);
     if (in_image()) {
       DP("Setting global device environment before load (%u bytes)\n", si.size);
-      uint64_t offset = (char *)si.addr - (char *)image->ImageStart;
+      uint64_t offset = (char *)si.addr - getImageStart(image);
       void *pos = (char *)data + offset;
       memcpy(pos, &host_device_env, si.size);
     }
     return ATMI_STATUS_SUCCESS;
   }
 
   atmi_status_t after_loading() {
     assert(valid);
__tgt_target_table *__tgt_rtl_load_binary_locked(int32_t device_id,
   //   records it in the KernelsList for quicker access later
   //
   // The initialization can be done before or after loading the image onto the
   // gpu. This function presently does a mixture. Using the hsa api to get/set
   // the information is simpler to implement, in exchange for more complicated
   // runtime behaviour. E.g. launching a kernel or using dma to get eight bytes
   // back from the gpu vs a hashtable lookup on the host.
 
-  const size_t img_size = (char *)image->ImageEnd - (char *)image->ImageStart;
+  if (!getAuxInfo(image, DeviceInfo.AuxInfo)) {
+    DP("Failed to read auxiliary information!\n");
+    return NULL;
+  }
+  // TODO: use the auxiliary information for the image processing.
+  //       For example, may the whole "omptarget_device_environment"
+  //       be embedded into the auxiliary info?
+  char *img_begin = getImageStart(image);
+  const size_t img_size = (char *)image->ImageEnd - img_begin;
 
   DeviceInfo.clearOffloadEntriesTable(device_id);
 
   // We do not need to set the ELF version because the caller of this function
   // had to do that to decide the right runtime to use
 
   if (!elf_machine_id_is_amdgcn(image)) {
     return NULL;
   }
 
   {
     auto env = device_environment(device_id, DeviceInfo.NumberOfDevices, image,
                                   img_size);
     if (!env.valid) {
       return NULL;
     }
 
     atmi_status_t err = module_register_from_memory_to_place(
-        (void *)image->ImageStart, img_size, get_gpu_place(device_id),
+        (void *)img_begin, img_size, get_gpu_place(device_id),
         [&](void *data, size_t size) {
           return env.before_loading(data, size);
         });
 
     check("Module registering", err);
     if (err != ATMI_STATUS_SUCCESS) {
       fprintf(stderr,
               "Possible gpu arch mismatch: device:%s, image:%s please check"
if (state_ptr_size < sizeof(void *)) {
       return NULL;
     }
 
     // if it's larger than a void*, assume it's a bss array and no further
     // initialization is required. Only try to set up a pointer for
     // sizeof(void*)
     if (state_ptr_size == sizeof(void *)) {
       uint64_t device_State_bytes =
-          get_device_State_bytes((char *)image->ImageStart, img_size);
+          get_device_State_bytes(img_begin, img_size);
       if (device_State_bytes == 0) {
         return NULL;
       }
 
       auto &dss = DeviceInfo.deviceStateStore[device_id];
       if (dss.first.get() == nullptr) {
         assert(dss.second == 0);
         void *ptr = NULL;
for (__tgt_offload_entry *e = HostBegin; e != HostEnd; ++e) {
     struct KernDescValType KernDescVal;
     std::string KernDescNameStr(e->name);
     KernDescNameStr += "_kern_desc";
     const char *KernDescName = KernDescNameStr.c_str();
 
     void *KernDescPtr;
     uint32_t KernDescSize;
     void *CallStackAddr = nullptr;
-    err = interop_get_symbol_info((char *)image->ImageStart, img_size,
+    err = interop_get_symbol_info(img_begin, img_size,
                                   KernDescName, &KernDescPtr, &KernDescSize);
 
     if (err == ATMI_STATUS_SUCCESS) {
       if ((size_t)KernDescSize != sizeof(KernDescVal))
         DP("Loading global computation properties '%s' - size mismatch (%u != "
            "%lu)\n",
            KernDescName, KernDescSize, sizeof(KernDescVal));
 
if (err == ATMI_STATUS_SUCCESS) {
 
       // Generic
       std::string ExecModeNameStr(e->name);
       ExecModeNameStr += "_exec_mode";
       const char *ExecModeName = ExecModeNameStr.c_str();
 
       void *ExecModePtr;
       uint32_t varsize;
-      err = interop_get_symbol_info((char *)image->ImageStart, img_size,
+      err = interop_get_symbol_info(img_begin, img_size,
                                     ExecModeName, &ExecModePtr, &varsize);
 
       if (err == ATMI_STATUS_SUCCESS) {
         if ((size_t)varsize != sizeof(int8_t)) {
           DP("Loading global computation properties '%s' - size mismatch(%u != "
              "%lu)\n",
              ExecModeName, varsize, sizeof(int8_t));
           return NULL;
if (err == ATMI_STATUS_SUCCESS) {
 
       // Flat group size
       std::string WGSizeNameStr(e->name);
       WGSizeNameStr += "_wg_size";
       const char *WGSizeName = WGSizeNameStr.c_str();
 
       void *WGSizePtr;
       uint32_t WGSize;
-      err = interop_get_symbol_info((char *)image->ImageStart, img_size,
+      err = interop_get_symbol_info(img_begin, img_size,
                                     WGSizeName, &WGSizePtr, &WGSize);
 
       if (err == ATMI_STATUS_SUCCESS) {
         if ((size_t)WGSize != sizeof(int16_t)) {
           DP("Loading global computation properties '%s' - size mismatch (%u "
              "!= "
              "%lu)\n",
              WGSizeName, WGSize, sizeof(int16_t));

openmp/libomptarget/plugins/common/CMakeLists.txt

 ##===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
 # Common parts which can be used by all plugins
 #
 ##===----------------------------------------------------------------------===##
 
+add_subdirectory(tgt_image)
 add_subdirectory(elf_common)
 add_subdirectory(MemoryManager)

openmp/libomptarget/plugins/common/elf_common/CMakeLists.txt

 ##===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
 # Common ELF functionality for target plugins
 #
 ##===----------------------------------------------------------------------===##
 
 add_library(elf_common INTERFACE)
+target_link_libraries(elf_common INTERFACE tgt_image_common)
 
 target_include_directories(elf_common INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})

openmp/libomptarget/plugins/common/elf_common/elf_common.h

 
 #if !(defined(_OMPTARGET_DEBUG_H))
 #error Include elf_common.h in the plugin source AFTER Debug.h has\
  been included.
 #endif
 
 #include <elf.h>
 #include <libelf.h>
+#include "tgt_image_common.h"
 
 // Check whether an image is valid for execution on target_id
 static inline int32_t elf_check_machine(__tgt_device_image *image,
                                         uint16_t target_id) {
 
   // Is the library version incompatible with the header file?
   if (elf_version(EV_CURRENT) == EV_NONE) {
     DP("Incompatible ELF library!\n");
     return 0;
   }
 
-  char *img_begin = (char *)image->ImageStart;
+  char *img_begin = getImageStart(image);
   char *img_end = (char *)image->ImageEnd;
   size_t img_size = img_end - img_begin;
 
   // Obtain elf handler
   Elf *e = elf_memory(img_begin, img_size);
   if (!e) {
     DP("Unable to get ELF handle: %s!\n", elf_errmsg(-1));
     return 0;
static inline int32_t elf_check_machine(__tgt_device_image *image,
   }
 
   elf_end(e);
   return MachineID == target_id;
 }
 
 static inline int32_t elf_is_dynamic(__tgt_device_image *image) {
 
-  char *img_begin = (char *)image->ImageStart;
+  char *img_begin = getImageStart(image);
   char *img_end = (char *)image->ImageEnd;
   size_t img_size = img_end - img_begin;
 
   // Obtain elf handler
   Elf *e = elf_memory(img_begin, img_size);
   if (!e) {
     DP("Unable to get ELF handle: %s!\n", elf_errmsg(-1));
     return 0;

openmp/libomptarget/plugins/common/tgt_image/CMakeLists.txt

+##===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+##===----------------------------------------------------------------------===##
+#
+# Common parts which can be used by all plugins
+#
+##===----------------------------------------------------------------------===##
+
+add_library(tgt_image_common INTERFACE)
+target_include_directories(tgt_image_common INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})

openmp/libomptarget/plugins/common/tgt_image/tgt_image_common.h

+//===-- tgt_image_common.h - Common target image functionality --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Common functionality for processing target images in target plugins.
+// Must be included in the plugin source file AFTER omptarget.h has been
+// included and macro DP(...) has been defined.
+// .
+//
+//===----------------------------------------------------------------------===//
+
+#if !(defined(_OMPTARGET_DEBUG_H))
+#error Include common.cpp in the plugin source AFTER Debug.h has been included.
+#endif
+
+#include <cstring>
+#include <map>
+#include <vector>
+#if OPENMP_MAY_USE_LLVM_SUPPORT
+#include "llvm/Support/YAMLTraits.h"
+#endif // OPENMP_MAY_USE_LLVM_SUPPORT
+
+#define ONEOMPMAGLEN 4
+#define ONEOMPMAG "1OMP"
+
+struct OpenMPHeaderDesc {
+  char *Begin;
+  uint32_t Size;
+};
+
+static inline char *getImageStart(
+    const __tgt_device_image *Image,
+    OpenMPHeaderDesc *HeaderDesc = nullptr) {
+  char *ImageStart = (char *)Image->ImageStart;
+  char *ImageEnd = (char *)Image->ImageEnd;
+
+  if (HeaderDesc) {
+    HeaderDesc->Begin = nullptr;
+    HeaderDesc->Size = 0;
+  }
+
+  if (ImageEnd - ImageStart < ONEOMPMAGLEN)
+    return ImageStart;
+
+  if (std::memcmp((void *)ImageStart, ONEOMPMAG, ONEOMPMAGLEN) != 0)
+    return ImageStart;
+
+  // "1OMP" magic word.
+  // The expected image structure is like this:
+  //   struct {
+  //     char MagicWord[ONEOMPMAGLEN]; // "1OMP"
+  //     uint32_t HeaderSize;          // In little-endian.
+  //     char Header[HeaderSize];      // Uncompressed YAML with auxiliary
+  //                                   // information.
+  //     char ActualImage[];           // ELF or another image.
+  //   }
+  uint32_t HeaderSize = *(uint32_t *)((char *)ImageStart + ONEOMPMAGLEN);
+  char *HeaderBegin = ImageStart + ONEOMPMAGLEN + sizeof(uint32_t);
+  ImageStart = HeaderBegin + HeaderSize;
+   DP("1OMP header: HeaderSize(%" PRIu32 "), HeaderBegin("
+     DPxMOD "), ImageBegin(" DPxMOD "), ImageEnd(" DPxMOD ")\n",
+     HeaderSize, DPxPTR(HeaderBegin), DPxPTR(ImageStart), DPxPTR(ImageEnd));
+
+  if (ImageStart > ImageEnd) {
+    FAILURE_MESSAGE("HeaderSize (%" PRIu32 ") is too big\n", HeaderSize);
+    FATAL_MESSAGE0(0, "invalid 1OMP image");
+  }
+
+  if (HeaderDesc) {
+    HeaderDesc->Begin = HeaderBegin;
+    HeaderDesc->Size = HeaderSize;
+  }
+
+  return ImageStart;
+}
+
+#if OPENMP_MAY_USE_LLVM_SUPPORT
+using namespace llvm;
+
+LLVM_YAML_STRONG_TYPEDEF(std::string, OpenMPAttrTy)
+LLVM_YAML_IS_SEQUENCE_VECTOR(OpenMPAttrTy)
+
+namespace llvm {
+namespace yaml {
+template <> struct ScalarTraits<OpenMPAttrTy> {
+  static void output(const OpenMPAttrTy &Val, void *, raw_ostream &Out) {
+    Out << Val;
+  }
+  static StringRef input(StringRef Scalar, void *, OpenMPAttrTy &Val) {
+    Val = Scalar.str();
+    return StringRef();
+  }
+  static QuotingType mustQuote(StringRef) { return QuotingType::Double; }
+};
+} // end namespace yaml
+} // end namespace llvm
+#else // !OPENMP_MAY_USE_LLVM_SUPPORT
+typedef std::string OpenMPAttrTy;
+#endif // !OPENMP_MAY_USE_LLVM_SUPPORT
+
+typedef std::vector<OpenMPAttrTy> OpenMPAttrListTy;
+typedef std::map<std::string, OpenMPAttrListTy> OpenMPAttrDocTy;
+
+#if OPENMP_MAY_USE_LLVM_SUPPORT
+LLVM_YAML_IS_STRING_MAP(OpenMPAttrListTy)
+LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(OpenMPAttrDocTy)
+#endif // OPENMP_MAY_USE_LLVM_SUPPORT
+
+static inline bool getAuxInfo(
+    const __tgt_device_image *Image,
+    std::vector<OpenMPAttrDocTy> &Out) {
+  OpenMPHeaderDesc HeaderDesc;
+  getImageStart(Image, &HeaderDesc);
+  if (!HeaderDesc.Size) {
+    if (!HeaderDesc.Begin) {
+      // If there is no 1OMP header, then we may assume we successfully
+      // read the auxiliary information, which is none.
+      //
+      // This may happen with older binaries, which are supported
+      // by the legacy ELF path.
+      return true;
+    } else {
+      // There is 1OMP header with 0 size.
+      // This is unexpected, since there are mandatory members
+      // like VendorName, VendorID and Producer.
+      return false;
+    }
+  }
+
+  // There 1OMP header, so we have to either read it successfully
+  // or return false.
+#if !OPENMP_MAY_USE_LLVM_SUPPORT
+  return false;
+#else // OPENMP_MAY_USE_LLVM_SUPPORT
+  StringRef HeaderBuf(HeaderDesc.Begin, HeaderDesc.Size);
+  yaml::Input YamlIn(HeaderBuf);
+  std::vector<OpenMPAttrDocTy> Docs;
+  YamlIn >> Out;
+
+  if (std::error_code Err = YamlIn.error()) {
+    FAILURE_MESSAGE("1OMP YAML reading failed\n");
+    return false;
+  }
+
+  return true;
+#endif // OPENMP_MAY_USE_LLVM_SUPPORT
+}
+
+static inline void dumpAuxInfo(const std::vector<OpenMPAttrDocTy> &Info) {
+#if OMPTARGET_DEBUG
+  if (!Info.empty()) {
+    for (auto &Doc : Info) {
+      for (auto &Attr : Doc) {
+        DP("Read OpenMPAttr:\n");
+        DP("<%s: [\n", Attr.first.c_str());
+        for (auto &Val : Attr.second) {
+          const std::string &Str = Val;
+          DP("\t%s,\n", Str.c_str());
+        }
+        DP("]>\n");
+      }
+    }
+  }
+#endif // OMPTARGET_DEBUG
+}

openmp/libomptarget/plugins/cuda/CMakeLists.txt

 target_link_libraries(omptarget.rtl.cuda
   elf_common
   MemoryManager
   ${LIBOMPTARGET_DEP_LIBRARIES}
   ${LIBOMPTARGET_DEP_LIBELF_LIBRARIES}
   "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../exports"
   "-Wl,-z,defs")
 
+if (NOT OPENMP_STANDALONE_BUILD)
+  target_link_libraries(
+    omptarget.rtl.cuda
+    LLVMSupport
+    )
+endif()
+
 # Report to the parent scope that we are building a plugin for CUDA.
 # This controls whether tests are run for the nvptx offloading target
 # Run them if libcuda is available, or if the user explicitly asked for dlopen
 # Otherwise this plugin is being built speculatively and there may be no cuda available
 if (LIBOMPTARGET_CAN_LINK_LIBCUDA OR LIBOMPTARGET_FORCE_DLOPEN_LIBCUDA)
   libomptarget_say("Enable tests using CUDA plugin")
   set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS} nvptx64-nvidia-cuda" PARENT_SCOPE)
 else()
   libomptarget_say("Disabling tests using CUDA plugin as cuda may not be available")
 endif()

openmp/libomptarget/plugins/cuda/src/rtl.cpp

class DeviceRTLTy {
 
   /// A vector of memory managers. Since the memory manager is non-copyable and
   // non-removable, we wrap them into std::unique_ptr.
   std::vector<std::unique_ptr<MemoryManagerTy>> MemoryManagers;
 
   /// Whether use memory manager
   bool UseMemoryManager = true;
 
+  /// Auxiliary information associated with the device image.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+
   // Record entry point associated with device
   void addOffloadEntry(const int DeviceId, const __tgt_offload_entry entry) {
     FuncOrGblEntryTy &E = DeviceData[DeviceId].FuncGblEntries.back();
     E.Entries.push_back(entry);
   }
 
   // Return a pointer to the entry associated with the pointer
   const __tgt_offload_entry *getOffloadEntry(const int DeviceId,
if (DeviceData[DeviceId].NumThreads >
       DeviceData[DeviceId].NumTeams = DeviceData[DeviceId].ThreadsPerBlock;
     }
 
     return OFFLOAD_SUCCESS;
   }
 
   __tgt_target_table *loadBinary(const int DeviceId,
                                  const __tgt_device_image *Image) {
+    if (!getAuxInfo(Image, AuxInfo)) {
+      DP("Failed to read auxiliary information!\n");
+      return nullptr;
+    }
+    // TODO: use the auxiliary information for the image processing.
+    //       For example, pass CompileOptions/LinkOptions
+    //       to cuModuleLoadDataEx below.
+    void *ImageStart = getImageStart(Image);
     // Set the context we are using
     CUresult Err = cuCtxSetCurrent(DeviceData[DeviceId].Context);
     if (!checkResult(Err, "Error returned from cuCtxSetCurrent\n"))
       return nullptr;
 
     // Clear the offload table as we are going to create a new one.
     clearOffloadEntriesTable(DeviceId);
 
     // Create the module and extract the function pointers.
     CUmodule Module;
-    DP("Load data from image " DPxMOD "\n", DPxPTR(Image->ImageStart));
-    Err = cuModuleLoadDataEx(&Module, Image->ImageStart, 0, nullptr, nullptr);
+    DP("Load data from image " DPxMOD "\n", DPxPTR(ImageStart));
+    Err = cuModuleLoadDataEx(&Module, ImageStart, 0, nullptr, nullptr);
     if (!checkResult(Err, "Error returned from cuModuleLoadDataEx\n"))
       return nullptr;
 
     DP("CUDA module successfully loaded!\n");
 
     Modules.push_back(Module);
 
     // Find the symbols in the module by name.

 } // namespace
 
 // Exposed library API function
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *image) {
+  // TODO: this AuxInfo is just being discarded,
+  //       and we need to do something better.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+  if (!getAuxInfo(image, AuxInfo))
+    return 0;
+  dumpAuxInfo(AuxInfo);
+  // TODO: VendorName, VendorID, Producer and TargetID checks may be done here
+  //       by particular plugins (e.g. via calling specific API).
   return elf_check_machine(image, /* EM_CUDA */ 190);
 }
 
 int32_t __tgt_rtl_number_of_devices() { return DeviceRTL.getNumOfDevices(); }
 
 int64_t __tgt_rtl_init_requires(int64_t RequiresFlags) {
   DP("Init requires flags to %" PRId64 "\n", RequiresFlags);
   DeviceRTL.setRequiresFlag(RequiresFlags);

openmp/libomptarget/plugins/generic-elf-64bit/src/rtl.cpp

 
 /// Class containing all the device information.
 class RTLDeviceInfoTy {
   std::vector<std::list<FuncOrGblEntryTy>> FuncGblEntries;
 
 public:
   std::list<DynLibTy> DynLibs;
 
+  /// Auxiliary information associated with the device image.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+
   // Record entry point associated with device.
   void createOffloadTable(int32_t device_id, __tgt_offload_entry *begin,
                           __tgt_offload_entry *end) {
     assert(device_id < (int32_t)FuncGblEntries.size() &&
            "Unexpected device id!");
     FuncGblEntries[device_id].emplace_back();
     FuncOrGblEntryTy &E = FuncGblEntries[device_id].back();
 

 extern "C" {
 #endif
 
 int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *image) {
 // If we don't have a valid ELF ID we can just fail.
 #if TARGET_ELF_ID < 1
   return 0;
 #else
+  // TODO: this AuxInfo is just being discarded,
+  //       and we need to do something better.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+  if (!getAuxInfo(image, AuxInfo))
+    return 0;
+  dumpAuxInfo(AuxInfo);
+  // TODO: VendorName, VendorID, Producer and TargetID checks may be done here
+  //       by particular plugins (e.g. via calling specific API).
   return elf_check_machine(image, TARGET_ELF_ID);
 #endif
 }
 
 int32_t __tgt_rtl_number_of_devices() { return NUMBER_OF_DEVICES; }
 
 int32_t __tgt_rtl_init_device(int32_t device_id) { return OFFLOAD_SUCCESS; }
 
 __tgt_target_table *__tgt_rtl_load_binary(int32_t device_id,
                                           __tgt_device_image *image) {
 
+  if (!getAuxInfo(image, DeviceInfo.AuxInfo)) {
+    DP("Failed to read auxiliary information!\n");
+    return NULL;
+  }
+  // TODO: use the auxiliary information for the image processing.
+  char *img_begin = getImageStart(image);
   DP("Dev %d: load binary from " DPxMOD " image\n", device_id,
-     DPxPTR(image->ImageStart));
+     DPxPTR(img_begin));
 
   assert(device_id >= 0 && device_id < NUMBER_OF_DEVICES && "bad dev id");
 
-  size_t ImageSize = (size_t)image->ImageEnd - (size_t)image->ImageStart;
+  size_t ImageSize = (size_t)image->ImageEnd - (size_t)img_begin;
   size_t NumEntries = (size_t)(image->EntriesEnd - image->EntriesBegin);
   DP("Expecting to have %zd entries defined.\n", NumEntries);
 
   // Is the library version incompatible with the header file?
   if (elf_version(EV_CURRENT) == EV_NONE) {
     DP("Incompatible ELF library!\n");
     return NULL;
   }
 
   // Obtain elf handler
-  Elf *e = elf_memory((char *)image->ImageStart, ImageSize);
+  Elf *e = elf_memory(img_begin, ImageSize);
   if (!e) {
     DP("Unable to get ELF handle: %s!\n", elf_errmsg(-1));
     return NULL;
   }
 
   if (elf_kind(e) != ELF_K_ELF) {
     DP("Invalid Elf kind!\n");
     elf_end(e);
__tgt_target_table *__tgt_rtl_load_binary(int32_t device_id,
 
   FILE *ftmp = fdopen(tmp_fd, "wb");
 
   if (!ftmp) {
     elf_end(e);
     return NULL;
   }
 
-  fwrite(image->ImageStart, ImageSize, 1, ftmp);
+  fwrite(img_begin, ImageSize, 1, ftmp);
   fclose(ftmp);
 
   DynLibTy Lib = {tmp_name, dlopen(tmp_name, RTLD_LAZY)};
 
   if (!Lib.Handle) {
     DP("Target library loading error: %s\n", dlerror());
     elf_end(e);
     return NULL;

openmp/libomptarget/plugins/ve/src/rtl.cpp

 public:
   std::vector<struct veo_proc_handle *> ProcHandles;
   std::vector<struct veo_thr_ctxt *> Contexts;
   std::vector<uint64_t> LibraryHandles;
   std::list<DynLibTy> DynLibs;
   // Maps OpenMP device Ids to Ve nodeids
   std::vector<int> NodeIds;
 
+  /// Auxiliary information associated with the device image.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+
   void buildOffloadTableFromHost(int32_t device_id, uint64_t VeoLibHandle,
                                  __tgt_offload_entry *HostBegin,
                                  __tgt_offload_entry *HostEnd) {
     FuncOrGblEntry[device_id].emplace_back();
     std::vector<__tgt_offload_entry> &T =
         FuncOrGblEntry[device_id].back().Entries;
     T.clear();
     for (__tgt_offload_entry *i = HostBegin; i != HostEnd; ++i) {

 // supported by the runtime. The functionality is similar to comparing the
 // result of __tgt__rtl__load__binary to NULL. However, this is meant to be a
 // lightweight query to determine if the RTL is suitable for an image without
 // having to load the library, which can be expensive.
 int32_t __tgt_rtl_is_valid_binary(__tgt_device_image *Image) {
 #if TARGET_ELF_ID < 1
   return 0;
 #else
+  // TODO: this AuxInfo is just being discarded,
+  //       and we need to do something better.
+  std::vector<OpenMPAttrDocTy> AuxInfo;
+  if (!getAuxInfo(image, AuxInfo))
+    return 0;
+  dumpAuxInfo(AuxInfo);
+  // TODO: VendorName, VendorID, Producer and TargetID checks may be done here
+  //       by particular plugins (e.g. via calling specific API).
   return elf_check_machine(Image, TARGET_ELF_ID);
 #endif
 }
 
 // Initialize the specified device. In case of success return 0; otherwise
 // return an error code.
 int32_t __tgt_rtl_init_device(int32_t ID) {
   DP("Available VEO version: %i\n", veo_api_version());

 
 // Pass an executable image section described by image to the specified
 // device and prepare an address table of target entities. In case of error,
 // return NULL. Otherwise, return a pointer to the built address table.
 // Individual entries in the table may also be NULL, when the corresponding
 // offload region is not supported on the target device.
 __tgt_target_table *__tgt_rtl_load_binary(int32_t ID,
                                           __tgt_device_image *Image) {
+  if (!getAuxInfo(image, DeviceInfo.AuxInfo)) {
+    DP("Failed to read auxiliary information!\n");
+    return NULL;
+  }
+  // TODO: use the auxiliary information for the image processing.
+  char *ImageStart = getImageStart(image);
+
   DP("Dev %d: load binary from " DPxMOD " image\n", ID,
-     DPxPTR(Image->ImageStart));
+     DPxPTR(ImageStart));
 
   assert(ID >= 0 && "bad dev id");
 
-  size_t ImageSize = (size_t)Image->ImageEnd - (size_t)Image->ImageStart;
+  size_t ImageSize = (size_t)Image->ImageEnd - (size_t)ImageStart;
   size_t NumEntries = (size_t)(Image->EntriesEnd - Image->EntriesBegin);
   DP("Expecting to have %zd entries defined.\n", NumEntries);
 
   // load dynamic library and get the entry points. We use the dl library
   // to do the loading of the library, but we could do it directly to avoid the
   // dump to the temporary file.
   //
   // 1) Create tmp file with the library contents.
   // 2) Use dlopen to load the file and dlsym to retrieve the symbols.
   char tmp_name[] = "/tmp/tmpfile_XXXXXX";
   int tmp_fd = mkstemp(tmp_name);
 
   if (tmp_fd == -1) {
     return NULL;
   }
 
   FILE *ftmp = fdopen(tmp_fd, "wb");
 
   if (!ftmp) {
     DP("fdopen() for %s failed. Could not write target image\n", tmp_name);
     return NULL;
   }
 
-  fwrite(Image->ImageStart, ImageSize, 1, ftmp);
+  fwrite(ImageStart, ImageSize, 1, ftmp);
 
   // at least for the static case we need to change the permissions
   chmod(tmp_name, 0700);
 
   DP("Wrote target image to %s. ImageSize=%zu\n", tmp_name, ImageSize);
 
   fclose(ftmp);