[Clang][Bundler] Use llvm-objcopy for creating fat object files

[sdmitriev]

Summary

clang-offload-bundler currently uses partial linking for creating fat object files, but such technique cannot be used on Windows due to the absence of partial linking support in the linker. This patch changes implementation to use llvm-objcopy for merging device and host objects instead of doing partial linking. This is one step forward towards enabling OpenMP offload on Windows.

[sdmitriev]

Any feedback?

[ABataev]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

485

As I understand, the resulting object file has the same structure as before? And we still can get an access to the host object file using standard programs, like readelf, objdump, etc.?

[sdmitriev]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

485

Yes, the structure is the same. The only difference is in the section flags that contain device objects - such sections used to have ALLOC attribute, but they do not have it now which I believe is a good think.

That is what we had earlier:

bash-4.2$ cat y.c
#pragma omp declare target
void foo() {}
#pragma omp end declare target
bash-4.2$ clang -fopenmp -fopenmp-targets=x86_64-pc-linux-gnu -c y.c
bash-4.2$ objdump -h y.o 

y.o:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         00000008  0000000000000000  0000000000000000  00000040  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .text.startup 00000010  0000000000000000  0000000000000000  00000050  2**4
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 __CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu 000003f8  0000000000000000  0000000000000000  00000060  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 __CLANG_OFFLOAD_BUNDLE__host-x86_64-unknown-linux-gnu 00000630  0000000000000000  0000000000000000  00000460  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .eh_frame     00000058  0000000000000000  0000000000000000  00000a90  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  5 .init_array.0 00000008  0000000000000000  0000000000000000  00000ae8  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
  6 .comment      0000006a  0000000000000000  0000000000000000  00000af0  2**0
                  CONTENTS, READONLY
  7 .note.GNU-stack 00000000  0000000000000000  0000000000000000  00000b5a  2**0
                  CONTENTS, READONLY
  8 .llvm_addrsig 00000002  0000000000000000  0000000000000000  00000b5a  2**0
                  CONTENTS, READONLY, EXCLUDE
bash-4.2$

And here is what we have with this patch:

bash-4.2$ clang -fopenmp -fopenmp-targets=x86_64-pc-linux-gnu -c y.c
bash-4.2$ objdump -h y.o 

y.o:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         00000006  0000000000000000  0000000000000000  00000040  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .text.startup 00000010  0000000000000000  0000000000000000  00000050  2**4
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .init_array.0 00000008  0000000000000000  0000000000000000  00000060  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
  3 .comment      0000006a  0000000000000000  0000000000000000  00000068  2**0
                  CONTENTS, READONLY
  4 .note.GNU-stack 00000000  0000000000000000  0000000000000000  000000d2  2**0
                  CONTENTS, READONLY
  5 .eh_frame     00000058  0000000000000000  0000000000000000  000000d8  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  6 .llvm_addrsig 00000002  0000000000000000  0000000000000000  00000238  2**0
                  CONTENTS, READONLY, EXCLUDE
  7 __CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu 000003f8  0000000000000000  0000000000000000  00000353  2**0
                  CONTENTS, READONLY
  8 __CLANG_OFFLOAD_BUNDLE__host-x86_64-unknown-linux-gnu 00000630  0000000000000000  0000000000000000  0000074b  2**0
                  CONTENTS, READONLY
bash-4.2$

[ABataev]

clang/test/Driver/clang-offload-bundler.c

233

Why does FileCheck have all these strange options?

[sdmitriev]

clang/test/Driver/clang-offload-bundler.c

233

Well, -DVAR=VALUE is a standard FileCheck option which allows to define variable that can be used in the check pattern (please see http://www.llvm.org/docs/CommandGuide/FileCheck.html)

-D<VAR=VALUE>
Sets a filecheck pattern variable VAR with value VALUE that can be used in CHECK: lines.

I am using these options for defining variables which contain input and output object names as well as the host triple.

[ABataev]

LG

[mgorny]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

467

This is broken for running tests in clang standalone builds. llvm-copy is installed in the system, unlike clang-offload-bundler that is present in build directory. Not to mention it's simply wrong to assume both executables must be in the same directory.

clang/test/CMakeLists.txt

list(APPEND CLANG_TEST_DEPS
     llvm-bcanalyzer
     llvm-cat
     llvm-cxxfilt
     llvm-dis
     llvm-dwarfdump
     llvm-lto2
     llvm-modextract
     llvm-nm
+    llvm-objcopy
     llvm-objdump
     llvm-profdata
     llvm-readelf
     llvm-readobj
     llvm-symbolizer
     opt
     )
 

clang/test/Driver/clang-offload-bundler.c

 //
 // Check object bundle/unbundle. The content should be bundled into an ELF
 // section (we are using a PowerPC little-endian host which uses ELF). We
 // have an already bundled file to check the unbundle and do a dry run on the
 // bundling as it cannot be tested in all host platforms that will run these
 // tests.
 //
 
-// RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.o,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.o -### -dump-temporary-files 2>&1 \
-// RUN: | FileCheck %s --check-prefix CK-OBJ-CMD
-// CK-OBJ-CMD: private constant [{{[0-9]+}} x i8] c"{{.+}}", section "__CLANG_OFFLOAD_BUNDLE__host-[[HOST:.+]]"
-// CK-OBJ-CMD: private constant [{{[0-9]+}} x i8] c"Content of device file 1{{.+}}", section "__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu"
-// CK-OBJ-CMD: private constant [{{[0-9]+}} x i8] c"Content of device file 2{{.+}}", section "__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu"
-// CK-OBJ-CMD: clang{{(.exe)?}}" "-r" "-target" "[[HOST]]" "-o" "{{.+}}.o" "{{.+}}.o" "{{.+}}.bc" "-nostdlib"
+// RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.o,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.o -### 2>&1 \
+// RUN: | FileCheck %s -DHOST=%itanium_abi_triple -DINOBJ1=%t.o -DINOBJ2=%t.tgt1 -DINOBJ3=%t.tgt2 -DOUTOBJ=%t.bundle3.o --check-prefix CK-OBJ-CMD
+// CK-OBJ-CMD: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=[[INOBJ1]]" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=[[INOBJ2]]" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=[[INOBJ3]]" "[[INOBJ1]]" "[[OUTOBJ]]"
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.o,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.o
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.o,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.o -unbundle
 // RUN: diff %t.o %t.res.o
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=o -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.o,%t.res.tgt2 -inputs=%t.bundle3.o -unbundle
 // RUN: diff %t.o %t.res.o

clang/tools/clang-offload-bundler/CMakeLists.txt

-set(LLVM_LINK_COMPONENTS BitWriter Core Object Support)
+set(LLVM_LINK_COMPONENTS Object Support)
 
 if(NOT CLANG_BUILT_STANDALONE)
   set(tablegen_deps intrinsics_gen)
 endif()
 
 add_clang_executable(clang-offload-bundler
   ClangOffloadBundler.cpp
   

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

 
 #include "clang/Basic/Version.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Triple.h"
-#include "llvm/Bitcode/BitcodeWriter.h"
-#include "llvm/IR/Constant.h"
-#include "llvm/IR/Constants.h"
-#include "llvm/IR/GlobalVariable.h"
-#include "llvm/IR/LLVMContext.h"
-#include "llvm/IR/Module.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Support/Signals.h"
+#include "llvm/Support/StringSaver.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <memory>
 #include <string>
 #include <system_error>
 #include <vector>
Unbundle("unbundle",
              cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
 static cl::opt<bool> PrintExternalCommands(
     "###",
     cl::desc("Print any external commands that are to be executed "
              "instead of actually executing them - for testing purposes.\n"),
     cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
-static cl::opt<bool> DumpTemporaryFiles(
-    "dump-temporary-files",
-    cl::desc("Dumps any temporary files created - for testing purposes.\n"),
-    cl::init(false), cl::cat(ClangOffloadBundlerCategory));
-
 /// Magic string that marks the existence of offloading data.
 #define OFFLOAD_BUNDLER_MAGIC_STR "__CLANG_OFFLOAD_BUNDLE__"
 
 /// The index of the host input in the list of inputs.
 static unsigned HostInputIndex = ~0u;
 
 /// Path to the current binary.
 static std::string BundlerExecutable;
 
 /// Obtain the offload kind and real machine triple out of the target
 /// information specified by the user.
 static void getOffloadKindAndTriple(StringRef Target, StringRef &OffloadKind,
                                     StringRef &Triple) {
   auto KindTriplePair = Target.split('-');
   OffloadKind = KindTriplePair.first;
   Triple = KindTriplePair.second;
 }
-static StringRef getTriple(StringRef Target) {
-  StringRef OffloadKind;
-  StringRef Triple;
-  getOffloadKindAndTriple(Target, OffloadKind, Triple);
-  return Triple;
-}
 static bool hasHostKind(StringRef Target) {
   StringRef OffloadKind;
   StringRef Triple;
   getOffloadKindAndTriple(Target, OffloadKind, Triple);
   return OffloadKind == "host";
 }
 
 /// Generic file handler interface.
class ObjectFileHandler final : public FileHandler {
 
   /// Total number of inputs.
   unsigned NumberOfInputs = 0;
 
   /// Total number of processed inputs, i.e, inputs that were already
   /// read from the buffers.
   unsigned NumberOfProcessedInputs = 0;
 
-  /// LLVM context used to create the auxiliary modules.
-  LLVMContext VMContext;
-
-  /// LLVM module used to create an object with all the bundle
-  /// components.
-  std::unique_ptr<Module> AuxModule;
-
-  /// The current triple we are working with.
-  StringRef CurrentTriple;
-
-  /// The name of the main input file.
-  StringRef MainInputFileName;
-
   /// Iterator of the current and next section.
   section_iterator CurrentSection;
   section_iterator NextSection;
 
 public:
   ObjectFileHandler(std::unique_ptr<ObjectFile> ObjIn)
       : FileHandler(), Obj(std::move(ObjIn)),
         CurrentSection(Obj->section_begin()),
public:
   }
 
   void WriteHeader(raw_fd_ostream &OS,
                    ArrayRef<std::unique_ptr<MemoryBuffer>> Inputs) final {
     assert(HostInputIndex != ~0u && "Host input index not defined.");
 
     // Record number of inputs.
     NumberOfInputs = Inputs.size();
-
-    // Create an LLVM module to have the content we need to bundle.
-    auto *M = new Module("clang-offload-bundle", VMContext);
-    M->setTargetTriple(getTriple(TargetNames[HostInputIndex]));
-    AuxModule.reset(M);
   }
 
   void WriteBundleStart(raw_fd_ostream &OS, StringRef TargetTriple) final {
     ++NumberOfProcessedInputs;
-
-    // Record the triple we are using, that will be used to name the section we
-    // will create.
-    CurrentTriple = TargetTriple;
   }
 
   bool WriteBundleEnd(raw_fd_ostream &OS, StringRef TargetTriple) final {
     assert(NumberOfProcessedInputs <= NumberOfInputs &&
            "Processing more inputs that actually exist!");
     assert(HostInputIndex != ~0u && "Host input index not defined.");
 
     // If this is not the last output, we don't have to do anything.
     if (NumberOfProcessedInputs != NumberOfInputs)
       return false;
 
-    // Create the bitcode file name to write the resulting code to. Keep it if
-    // save-temps is active.
-    SmallString<128> BitcodeFileName;
-    if (sys::fs::createTemporaryFile("clang-offload-bundler", "bc",
-                                     BitcodeFileName)) {
-      errs() << "error: unable to create temporary file.\n";
+    // Find llvm-objcopy in order to create the bundle binary.
+    ErrorOr<std::string> Objcopy = sys::findProgramByName(
+        "llvm-objcopy", sys::path::parent_path(BundlerExecutable));
+    if (!Objcopy) {
+      errs() << "error: unable to find 'llvm-objcopy' in path.\n";
       return true;
     }
 
-    // Dump the contents of the temporary file if that was requested.
-    if (DumpTemporaryFiles) {
-      errs() << ";\n; Object file bundler IR file.\n;\n";
-      AuxModule.get()->print(errs(), nullptr,
-                             /*ShouldPreserveUseListOrder=*/false,
-                             /*IsForDebug=*/true);
-      errs() << '\n';
-    }
-
-    // Find clang in order to create the bundle binary.
-    StringRef Dir = sys::path::parent_path(BundlerExecutable);
-
-    auto ClangBinary = sys::findProgramByName("clang", Dir);
-    if (ClangBinary.getError()) {
-      // Remove bitcode file.
-      sys::fs::remove(BitcodeFileName);
-
-      errs() << "error: unable to find 'clang' in path.\n";
-      return true;
-    }
-
-    // Do the incremental linking. We write to the output file directly. So, we
-    // close it and use the name to pass down to clang.
+    // We write to the output file directly. So, we close it and use the name
+    // to pass down to llvm-objcopy.
     OS.close();
-    SmallString<128> TargetName = getTriple(TargetNames[HostInputIndex]);
-    std::vector<StringRef> ClangArgs = {"clang",
-                                        "-r",
-                                        "-target",
-                                        TargetName.c_str(),
-                                        "-o",
-                                        OutputFileNames.front().c_str(),
-                                        InputFileNames[HostInputIndex].c_str(),
-                                        BitcodeFileName.c_str(),
-                                        "-nostdlib"};
+
+    // Compose command line for the objcopy tool.
+    BumpPtrAllocator Alloc;
+    StringSaver SS{Alloc};
+    SmallVector<StringRef, 8u> ObjcopyArgs{"llvm-objcopy"};
+    for (unsigned I = 0; I < NumberOfInputs; ++I)
+      ObjcopyArgs.push_back(SS.save(Twine("--add-section=") +
+                                    OFFLOAD_BUNDLER_MAGIC_STR + TargetNames[I] +
+                                    "=" + InputFileNames[I]));
+    ObjcopyArgs.push_back(InputFileNames[HostInputIndex]);
+    ObjcopyArgs.push_back(OutputFileNames.front());
 
     // If the user asked for the commands to be printed out, we do that instead
     // of executing it.
     if (PrintExternalCommands) {
-      errs() << "\"" << ClangBinary.get() << "\"";
-      for (StringRef Arg : ClangArgs)
+      errs() << "\"" << Objcopy.get() << "\"";
+      for (StringRef Arg : drop_begin(ObjcopyArgs, 1))
         errs() << " \"" << Arg << "\"";
       errs() << "\n";
     } else {
-      // Write the bitcode contents to the temporary file.
-      {
-        std::error_code EC;
-        raw_fd_ostream BitcodeFile(BitcodeFileName, EC, sys::fs::OF_None);
-        if (EC) {
-          errs() << "error: unable to open temporary file.\n";
-          return true;
-        }
-        WriteBitcodeToFile(*AuxModule, BitcodeFile);
-      }
-
-      bool Failed = sys::ExecuteAndWait(ClangBinary.get(), ClangArgs);
-
-      // Remove bitcode file.
-      sys::fs::remove(BitcodeFileName);
-
-      if (Failed) {
-        errs() << "error: incremental linking by external tool failed.\n";
+      if (sys::ExecuteAndWait(Objcopy.get(), ObjcopyArgs)) {
+        errs() << "error: llvm-objcopy tool failed.\n";
         return true;
       }
     }
 
     return false;
   }
 
-  void WriteBundle(raw_fd_ostream &OS, MemoryBuffer &Input) final {
-    Module *M = AuxModule.get();
-
-    // Create the new section name, it will consist of the reserved prefix
-    // concatenated with the triple.
-    std::string SectionName = OFFLOAD_BUNDLER_MAGIC_STR;
-    SectionName += CurrentTriple;
-
-    // Create the constant with the content of the section.
-    auto *Content = ConstantDataArray::get(
-        VMContext, ArrayRef<uint8_t>(reinterpret_cast<const uint8_t *>(
-                                         Input.getBufferStart()),
-                                     Input.getBufferSize()));
-
-    // Create the global in the desired section. We don't want these globals
-    // in the symbol table, so we mark them private.
-    auto *GV = new GlobalVariable(*M, Content->getType(), /*IsConstant=*/true,
-                                  GlobalVariable::PrivateLinkage, Content);
-    GV->setSection(SectionName);
-  }
+  void WriteBundle(raw_fd_ostream &OS, MemoryBuffer &Input) final {}
 };
 
 /// Handler for text files. The bundled file will have the following format.
 ///
 /// "Comment OFFLOAD_BUNDLER_MAGIC_STR__START__ triple"
 /// Bundle 1
 /// "Comment OFFLOAD_BUNDLER_MAGIC_STR__END__ triple"
 /// ...