[clang-offload-bundler] Enable handling of partially-linked fat objects

[grokos]

Summary

This is the bundler-side patch for enabling static library support in clang. The scheme has been discussed extensively in the past and is described in this document prepared by @sdmitriev:

offload from static libs.pdf508 KBDownload

.

Patch was developed in collaboration with @kbobrovs and a similar version has been merged with Intel's SYCL compiler (https://github.com/intel/llvm/tree/sycl).

When a fat object is created, for each bundle the bundler also creates a corresponding "size" section consisting of a single 64-bit integer storing the size of the bundle. When linking from static objects, the host linker will fetch all dependencies and do a partial linking on them; this action concatenates all sections with the same name across fetched dependencies into a new aggregate section, so for each target there will be an aggregate section containing the concatenated bundles and another aggregate section containing the concatenated sizes. By visiting the aggregate sizes section the unbundler can then split the aggregate bundle into separate output device objects.

The patch introduces a new type "oo" which is used when unbundling partially-linked fat objects. When "oo" is specified, the output file is not an object file itself; instead it is a text file containing the paths to the actual outputs (because we may have multiple device objects as outputs - one for each dependency that was fetched).

Invocation of the host linker (to do partial-linking) and cleanup of temporary files will be done by the Driver. Once the bundler patch lands, the Driver patch will follow.

[ABataev]

Partial linking may lead to some incorrect results with global constructors. How are you going to handle this?

clang/test/Driver/clang-offload-bundler-missing-size-section.cpp

1–44

Very strange test. It should not contain standard includes. Also, it should not be an executable test, you have to check for the driver output or something similar

clang/test/Driver/clang-offload-bundler-oo.cpp

1–18

Same about this test

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

84

Hmm, are you going to introduce a new kind of output? It really requires RFC.

[jdoerfert]

(I'll try to actually review this later but I left a comment below)

clang/test/Driver/clang-offload-bundler-missing-size-section.cpp

1–44

We have executable tests in the OpenMP target offloading part, if it cannot live with clang it can live there.

[jdoerfert]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

84

This is the offload-bundler tool, right? Who is using that except OpenMP (and SYCL)?

Is there a reason for oo? uo (=unboundled object), or do (=device object)?

160

I don't understand the comment. If \p FileName is a list of outputs, how does this work?

[grokos]

In D74262#1867245, @ABataev wrote:

Partial linking may lead to some incorrect results with global constructors. How are you going to handle this?

Can you give me an example of what can break? I remember reading a conversation about some linker patch some time ago but I cannot recall the details.

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

84

No one else (at least for now). But I can send out an RFC regarding the new output anyway.

oo is related to the fact that under this scheme we can have multiple .o files as output (many o's). But if you think some of the other abbreviations makes more sense, I'm happy to change it.

160

The scheme is described in the attached pdf. In short, when the host liner fetches dependencies from a static library, alongside the host bundle it also fetches the device bundle. Now, if we have multiple dependencies from multiple objects inside a static library (or multiple static libraries) the host linker will perform a partial linking between all fetched bundles for the targets we are interested in. The result is a fat object in which each target bundle is the result of concatenating the individual bundles for that target we fetched from each static library. We also keep track of the size of each fetched bundle (we use a new sizes section per target inside the fat object for this purpose) so that the unbundler can separate the partially-linked bundle into the original object files it was assembled from. Usually, we don't know a priori how many dependencies will be brought in, so we don't know how many objects we're going to have at outputs. Therefore, in oo unbundling mode, the user specifies a single output file per target (just like in any other unbundling mode) which the unbundler populates with the paths to the actual output device objects. Then the driver reads those paths and passes them on to the device linker.

[ABataev]

In D74262#1876835, @grokos wrote:

In D74262#1867245, @ABataev wrote:

Partial linking may lead to some incorrect results with global constructors. How are you going to handle this?

Can you give me an example of what can break? I remember reading a conversation about some linker patch some time ago but I cannot recall the details.

See the discussion here https://reviews.llvm.org/D65819

[grokos]

The partial linking scheme has been found to not work correctly in all cases (it fails when we have libraries with device code only). A new patch will be uploaded which will be based on archive extraction.

clang/test/Driver/clang-offload-bundler-missing-size-section.cpp

+// REQUIRES: x86-registered-target
+// RUN: %clangxx -c %s -o %t_fat.o
+// RUN: %clangxx %t_fat.o -o %t.exe
+// RUN: clang-offload-bundler -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t_host.o,%t_device.o -inputs=%t_fat.o -unbundle
+// RUN: %t.exe %t_device.o | FileCheck %s
+// CHECK:11
+
+#include <iostream>
+#include <fstream>
+#include <string>
+using namespace std;
+
+#define BUNDLE_SECTION_PREFIX "__CLANG_OFFLOAD_BUNDLE__"
+
+#define TARGET0 "host-x86_64-unknown-linux-gnu"
+#define TARGET1 "openmp-x86_64-pc-linux-gnu"
+
+// Populate section with special names recognized by the bundler;
+// this emulates fat object with one host and one device section.
+// The size sections are missing, emulating the old fat object format.
+char str0[] __attribute__((section(BUNDLE_SECTION_PREFIX TARGET0))) = { 0 };
+char str1[] __attribute__((section(BUNDLE_SECTION_PREFIX TARGET1))) = { "11\n" };
+
+// main is invoked with the bundler output file as argument -
+// read this file and print their contents to stdout.
+int main(int argc, char **argv) {
+  string DeviceObj(argv[1]);
+  string Line;
+  ifstream F(DeviceObj);
+
+  if (F.is_open()) {
+    while (getline(F, Line)) {
+      cout << Line;
+    }
+    F.close();
+  }
+  else {
+    cout << "Unable to open file " << DeviceObj;
+    return 1;
+  }
+
+  return 0;
+}
+

clang/test/Driver/clang-offload-bundler-oo.cpp

+// REQUIRES: x86-registered-target
+// RUN: %clangxx -c %s -o %t_fat.o
+// RUN: %clangxx %t_fat.o -o %t.exe
+// RUN: clang-offload-bundler -type=oo -targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.o,%t_list.txt -inputs=%t_fat.o -unbundle
+// RUN: %t.exe %t_list.txt | FileCheck %s
+// CHECK:11
+// CHECK:222
+// CHECK:3333
+
+#include <cstdint>
+#include <iostream>
+#include <fstream>
+#include <string>
+#include <vector>
+using namespace std;
+
+#define BUNDLE_SECTION_PREFIX "__CLANG_OFFLOAD_BUNDLE__"
+#define BUNDLE_SIZE_SECTION_PREFIX "__CLANG_OFFLOAD_BUNDLE_SIZE__"
+
+#define TARGET0 "host-x86_64-unknown-linux-gnu"
+#define TARGET1 "openmp-x86_64-pc-linux-gnu"
+
+// Populate section with special names recognized by the bundler;
+// this emulates fat object partially linked from 3 other fat objects.
+// The test uses the bundler to split the bundle into 3 objects and then prints
+// their contents to stdout.
+char str0[] __attribute__((section(BUNDLE_SECTION_PREFIX TARGET0))) = { 0, 0, 0 };
+int64_t size0[] __attribute__((section(BUNDLE_SIZE_SECTION_PREFIX TARGET0))) = { 1, 1, 1 };
+
+char str1[] __attribute__((section(BUNDLE_SECTION_PREFIX TARGET1))) = { "11\n222\n3333\n" };
+int64_t size1[] __attribute__((section(BUNDLE_SIZE_SECTION_PREFIX TARGET1))) = { 3, 4, 5 };
+
+void cat(const string& File) {
+  string Line;
+  ifstream F(File);
+  if (F.is_open()) {
+    while (getline(F, Line)) {
+      cout << Line << '\n';
+    }
+    F.close();
+  }
+  else cout << "Unable to open file " << File;
+}
+
+// main is invoked with the bundler output file as argument -
+// read this file and print their contents to stdout.
+int main(int argc, char **argv) {
+  string ListFile(argv[1]);
+  string Line;
+  ifstream F(ListFile);
+  vector<string> OutFiles;
+
+  if (F.is_open()) {
+    while (getline(F, Line)) {
+      OutFiles.push_back(Line);
+    }
+    F.close();
+  }
+  else {
+    cout << "Unable to open file " << ListFile;
+    return 1;
+  }
+
+  for (const auto &File : OutFiles) {
+    cat(File);
+  }
+  return 0;
+}
+

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Endian.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/Signals.h"
FilesType("type", cl::Required,
                        "  i   - cpp-output\n"
                        "  ii  - c++-cpp-output\n"
                        "  cui - cuda/hip-output\n"
                        "  d   - dependency\n"
                        "  ll  - llvm\n"
                        "  bc  - llvm-bc\n"
                        "  s   - assembler\n"
                        "  o   - object\n"
+                       "  oo  - object; output file is a list of unbundled objects\n"
                        "  gch - precompiled-header\n"
                        "  ast - clang AST file"),
               cl::cat(ClangOffloadBundlerCategory));
 static cl::opt<bool>
     Unbundle("unbundle",
              cl::desc("Unbundle bundled file into several output files.\n"),
              cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
 static cl::opt<bool> PrintExternalCommands(
     "###",
     cl::desc("Print any external commands that are to be executed "
              "instead of actually executing them - for testing purposes.\n"),
     cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
 /// Magic string that marks the existence of offloading data.
 #define OFFLOAD_BUNDLER_MAGIC_STR "__CLANG_OFFLOAD_BUNDLE__"
 
+/// Prefix of an added section name with bundle size.
+#define SIZE_SECTION_PREFIX "__CLANG_OFFLOAD_BUNDLE_SIZE__"
+
 /// The index of the host input in the list of inputs.
 static unsigned HostInputIndex = ~0u;
 
 /// Path to the current binary.
 static std::string BundlerExecutable;
 
 /// Obtain the offload kind and real machine triple out of the target
 /// information specified by the user.
public:
   ReadBundleStart(MemoryBuffer &Input) = 0;
 
   /// Read the marker that closes the current bundle.
   virtual Error ReadBundleEnd(MemoryBuffer &Input) = 0;
 
   /// Read the current bundle and write the result into the stream \a OS.
   virtual Error ReadBundle(raw_fd_ostream &OS, MemoryBuffer &Input) = 0;
 
+  /// Read the current bundle and write the result into the file \a FileName.
+  /// The meaning of \a FileName depends on unbundling type - in some
+  /// cases (type="oo") it will contain a list of actual outputs.
+  virtual Error ReadBundle(StringRef FileName, MemoryBuffer &Input) {
+    std::error_code EC;
+    raw_fd_ostream OS(FileName, EC, sys::fs::OF_None);
+
+    if (EC)
+      return createFileError(FileName, EC);
+    return ReadBundle(OS, Input);
+  }
+
   /// Write the header of the bundled file to \a OS based on the information
   /// gathered from \a Inputs.
   virtual Error WriteHeader(raw_fd_ostream &OS,
                             ArrayRef<std::unique_ptr<MemoryBuffer>> Inputs) = 0;
 
   /// Write the marker that initiates a bundle for the triple \a TargetTriple to
   /// \a OS.
   virtual Error WriteBundleStart(raw_fd_ostream &OS,
                                  StringRef TargetTriple) = 0;
 
   /// Write the marker that closes a bundle for the triple \a TargetTriple to \a
   /// OS.
   virtual Error WriteBundleEnd(raw_fd_ostream &OS, StringRef TargetTriple) = 0;
 
   /// Write the bundle from \a Input into \a OS.
   virtual Error WriteBundle(raw_fd_ostream &OS, MemoryBuffer &Input) = 0;
+
+  /// Sets a base name for temporary filename generation.
+  void SetTempFileNameBase(StringRef Base) { TempFileNameBase = Base.data(); }
+
+protected:
+  /// Serves as a base name for temporary filename generation.
+  std::string TempFileNameBase;
 };
 
 /// Handler for binary files. The bundled file will have the following format
 /// (all integers are stored in little-endian format):
 ///
 /// "OFFLOAD_BUNDLER_MAGIC_STR" (ASCII encoding of the string)
 ///
 /// NumberOfOffloadBundles (8-byte integer)
Expected<Optional<StringRef>> ReadBundleStart(MemoryBuffer &Input) final {
     return CurBundleInfo->first();
   }
 
   Error ReadBundleEnd(MemoryBuffer &Input) final {
     assert(CurBundleInfo != BundlesInfo.end() && "Invalid reader info!");
     return Error::success();
   }
 
+  using FileHandler::ReadBundle; // to avoid hiding via the overload below
+
   Error ReadBundle(raw_fd_ostream &OS, MemoryBuffer &Input) final {
     assert(CurBundleInfo != BundlesInfo.end() && "Invalid reader info!");
     StringRef FC = Input.getBuffer();
     OS.write(FC.data() + CurBundleInfo->second.Offset,
              CurBundleInfo->second.Size);
     return Error::success();
   }
 

 };
 
 } // end anonymous namespace
 
 /// Handler for object files. The bundles are organized by sections with a
 /// designated name.
 ///
 /// To unbundle, we just copy the contents of the designated section.
+///
+/// The bundler produces object file in host target native format (e.g. ELF for
+/// Linux). The sections it creates are:
+///
+/// <OFFLOAD_BUNDLER_MAGIC_STR><target triple 1>
+/// |
+/// | binary data for the <target 1>'s bundle
+/// |
+/// <SIZE_SECTION_PREFIX><target triple 1>
+/// | size of the <target1>'s bundle (8 bytes)
+/// ...
+/// <OFFLOAD_BUNDLER_MAGIC_STR><target triple N>
+/// |
+/// | binary data for the <target N>'s bundle
+/// |
+/// <SIZE_SECTION_PREFIX><target triple N>
+/// | size of the <target N>'s bundle (8 bytes)
+/// ...
+/// <OFFLOAD_BUNDLER_MAGIC_STR><host target>
+/// | 0 (1 byte long)
+/// <SIZE_SECTION_PREFIX><host target>
+/// | 1 (8 bytes)
+/// ...
+///
+/// Further, these fat objects can be "partially" linked by a platform linker:
+/// 1) ld -r a_fat.o b_fat.o c_fat.o -o abc_fat.o
+/// 2) ld -r a_fat.o -L. -lbc -o abc_fat.o
+///   where libbc.a is a static library created from b_fat.o and c_fat.o.
+/// This will still result in a fat object. But this object will have bundle and
+/// size sections for the same triple concatenated:
+/// ...
+/// <OFFLOAD_BUNDLER_MAGIC_STR><target triple 1>
+/// | binary data for the <target 1>'s bundle (from a_fat.o)
+/// | binary data for the <target 1>'s bundle (from b_fat.o)
+/// | binary data for the <target 1>'s bundle (from c_fat.o)
+/// <SIZE_SECTION_PREFIX><target triple 1>
+/// | size of the <target1>'s bundle (8 bytes) (from a_fat.o)
+/// | size of the <target1>'s bundle (8 bytes) (from b_fat.o)
+/// | size of the <target1>'s bundle (8 bytes) (from c_fat.o)
+/// ...
+///
+/// The alignment of all the added sections is set to one to avoid padding
+/// between concatenated parts.
+///
+/// The unbundler is able to unbundle both kinds of fat objects. The first one
+/// (non-partially linked) can be handled either with -type=o or -type=oo,
+/// whereas the second one with -type=oo option only. In the latter case
+/// unbundling may result in multiple output files per target and the output
+/// file specified in the command line contains a list with the names of the
+/// actual outputs.
+///
 class ObjectFileHandler final : public FileHandler {
 
+  /// Keeps infomation about a bundle for a particular target.
+  struct BundleInfo final {
+    /// The section that contains bundle data, can be a concatenation of a
+    /// number of individual bundles if produced via partial linkage of multiple
+    /// fat objects.
+    section_iterator BundleSection;
+    /// The sizes (in correct order) of the individual bundles constituting
+    /// bundle data.
+    SmallVector<uint64_t, 4> ObjectSizes;
+
+    BundleInfo(section_iterator S) : BundleSection(S) {}
+  };
+
   /// The object file we are currently dealing with.
   std::unique_ptr<ObjectFile> Obj;
 
+  /// Maps triple string to its bundle information
+  StringMap<std::unique_ptr<BundleInfo>> TripleToBundleInfo;
+  /// The two iterators below are to support the
+  /// ReadBundleStart/ReadBundle/ReadBundleEnd iteration mechanism
+  StringMap<std::unique_ptr<BundleInfo>>::iterator CurBundle;
+  StringMap<std::unique_ptr<BundleInfo>>::iterator NextBundle;
+
   /// Return the input file contents.
   StringRef getInputFileContents() const { return Obj->getData(); }
 
   /// Return bundle name (<kind>-<triple>) if the provided section is an offload
   /// section.
-  static Expected<Optional<StringRef>> IsOffloadSection(SectionRef CurSection) {
+  static Expected<Optional<StringRef>> IsOffloadSection(SectionRef CurSection,
+                                                        StringRef NamePrefix) {
     Expected<StringRef> NameOrErr = CurSection.getName();
     if (!NameOrErr)
       return NameOrErr.takeError();
 
-    // If it does not start with the reserved suffix, just skip this section.
-    if (!NameOrErr->startswith(OFFLOAD_BUNDLER_MAGIC_STR))
+    // If it does not start with given prefix, just skip this section.
+    if (!NameOrErr->startswith(NamePrefix))
       return None;
 
-    // Return the triple that is right after the reserved prefix.
-    return NameOrErr->substr(sizeof(OFFLOAD_BUNDLER_MAGIC_STR) - 1);
+    // Return the suffix (the triple that is right after the reserved prefix).
+    return NameOrErr->substr(NamePrefix.size());
   }
 
   /// Total number of inputs.
   unsigned NumberOfInputs = 0;
 
   /// Total number of processed inputs, i.e, inputs that were already
   /// read from the buffers.
   unsigned NumberOfProcessedInputs = 0;
 
-  /// Iterator of the current and next section.
-  section_iterator CurrentSection;
-  section_iterator NextSection;
+  /// Input sizes.
+  SmallVector<uint64_t, 16u> InputSizes;
 
 public:
   ObjectFileHandler(std::unique_ptr<ObjectFile> ObjIn)
       : FileHandler(), Obj(std::move(ObjIn)),
-        CurrentSection(Obj->section_begin()),
-        NextSection(Obj->section_begin()) {}
+        CurBundle(TripleToBundleInfo.end()),
+        NextBundle(TripleToBundleInfo.end()) {}
 
   ~ObjectFileHandler() final {}
 
-  Error ReadHeader(MemoryBuffer &Input) final { return Error::success(); }
+  // Iterate through sections and create a map from triple to relevant bundle
+  // information.
+  Error ReadHeader(MemoryBuffer &Input) final {
+    for (section_iterator Sec = Obj->section_begin(); Sec != Obj->section_end();
+         ++Sec) {
+      // Test if current section is an offload bundle section
+      Expected<Optional<StringRef>> BundleOrErr =
+          IsOffloadSection(*Sec, OFFLOAD_BUNDLER_MAGIC_STR);
+      if (!BundleOrErr)
+        return BundleOrErr.takeError();
+      if (*BundleOrErr) {
+        StringRef OffloadTriple = **BundleOrErr;
+        std::unique_ptr<BundleInfo> &BI = TripleToBundleInfo[OffloadTriple];
+        // A BundleInfo entry for this bundle should either:
+        // 1) not exist yet in which case we allocate it and initialize the
+        //    BundleSection iterator to Sec, while leaving ObjectSizes empty.
+        // 2) have been created if we previously encountered the size section
+        //    for this target in which case the BundleSection iterator should
+        //    point to Obj->section_end() and we initialize it now to Sec.
+        assert(!BI.get() || BI->BundleSection == Obj->section_end());
 
-  Expected<Optional<StringRef>> ReadBundleStart(MemoryBuffer &Input) final {
-    while (NextSection != Obj->section_end()) {
-      CurrentSection = NextSection;
-      ++NextSection;
-
-      // Check if the current section name starts with the reserved prefix. If
-      // so, return the triple.
-      Expected<Optional<StringRef>> TripleOrErr =
-          IsOffloadSection(*CurrentSection);
-      if (!TripleOrErr)
-        return TripleOrErr.takeError();
-      if (*TripleOrErr)
-        return **TripleOrErr;
+        if (!BI.get()) {
+          BI.reset(new BundleInfo(Sec));
+        } else {
+          BI->BundleSection = Sec;
         }
+        continue;
+      }
+      // Test if current section is an offload bundle size section
+      BundleOrErr = IsOffloadSection(*Sec, SIZE_SECTION_PREFIX);
+      if (!BundleOrErr)
+        return BundleOrErr.takeError();
+      if (*BundleOrErr) {
+        StringRef OffloadTriple = **BundleOrErr;
+
+        // yes, it is - parse object sizes
+        Expected<StringRef> Content = Sec->getContents();
+        if (!Content)
+          return Content.takeError();
+        unsigned int ElemSize = sizeof(uint64_t);
+
+        // the size of the size section must be a multiple of ElemSize
+        if (Content->size() % ElemSize != 0)
+          return createStringError(
+              errc::invalid_argument,
+              "invalid size of the bundle size section for triple " +
+                  OffloadTriple + ": " + Twine(Content->size()));
+        // read sizes
+        llvm::support::endianness E = Obj->isLittleEndian()
+                                          ? llvm::support::endianness::little
+                                          : llvm::support::endianness::big;
+        std::unique_ptr<BundleInfo> &BI = TripleToBundleInfo[OffloadTriple];
+        // A BundleInfo entry for this bundle should either:
+        // 1) not exist yet in which case we allocate it and initialize the
+        //    BundleSection iterator to Obj->section_end().
+        // 2) have been created if we previously encountered the data section
+        //    for this target in which case ObjectSizes should be empty.
+        assert(!BI.get() || BI->ObjectSizes.size() == 0);
+
+        if (!BI.get()) {
+          BI.reset(new BundleInfo(Obj->section_end()));
+        }
+        for (const char *Ptr = Content->data();
+             Ptr < Content->data() + Content->size(); Ptr += ElemSize) {
+          uint64_t Size = support::endian::read64(Ptr, E);
+          BI->ObjectSizes.push_back(Size);
+        }
+      }
+    }
+    NextBundle = TripleToBundleInfo.begin();
+    return Error::success();
+  }
+
+  Expected<Optional<StringRef>> ReadBundleStart(MemoryBuffer &Input) final {
+    if (NextBundle == TripleToBundleInfo.end())
       return None;
+    CurBundle = NextBundle;
+    NextBundle++;
+    return CurBundle->getKey();
   }
 
   Error ReadBundleEnd(MemoryBuffer &Input) final { return Error::success(); }
 
   Error ReadBundle(raw_fd_ostream &OS, MemoryBuffer &Input) final {
-    Expected<StringRef> ContentOrErr = CurrentSection->getContents();
-    if (!ContentOrErr)
-      return ContentOrErr.takeError();
-    StringRef Content = *ContentOrErr;
-
-    // Copy fat object contents to the output when extracting host bundle.
-    if (Content.size() == 1u && Content.front() == 0)
-      Content = StringRef(Input.getBufferStart(), Input.getBufferSize());
+    llvm_unreachable("must not be called for the ObjectFileHandler");
+  }
+
+  Error ReadBundle(StringRef OutName, MemoryBuffer &Input) final {
+    assert(CurBundle != TripleToBundleInfo.end() &&
+           "all bundles have been read already");
+
+    // TODO: temporary workaround to copy the entire fat object to the host
+    // output until driver is fixed to correctly handle list file for the host
+    // bundle in 'oo' mode.
+    if (FilesType == "oo" && hasHostKind(CurBundle->getKey())) {
+      std::error_code EC;
+      raw_fd_ostream OS(OutName, EC);
 
-    OS.write(Content.data(), Content.size());
+      if (EC)
+        return createFileError(OutName, EC);
+      OS.write(Input.getBufferStart(), Input.getBufferSize());
+      return Error::success();
+    }
+
+    // Read content of the section representing the bundle
+    Expected<StringRef> Content =
+        CurBundle->second->BundleSection->getContents();
+    if (!Content)
+      return Content.takeError();
+
+    // Backwards compatibility adjustment: object files created with older
+    // versions of clang-offload-bundler (before support for partially-linked
+    // objects) do not contain a sizes section and rightfully so because such a
+    // section was not necessary. However, the current version of the bundler
+    // expects the object to have a sizes section, even if it is a
+    // non-partially-linked one. Emulate the existence of a sizes section by
+    // adding the related number into ObjectSizes manually.
+    if (CurBundle->second->ObjectSizes.empty() && FilesType == "o") {
+      CurBundle->second->ObjectSizes.push_back(Content->size());
+    }
+
+    const char *ObjData = Content->data();
+    // Determine the number of "device objects" (or individual bundles
+    // concatenated by partial linkage) in the bundle:
+    const auto &SizeVec = CurBundle->second->ObjectSizes;
+    auto NumObjects = SizeVec.size();
+    bool FileListMode = FilesType == "oo";
+
+    if (NumObjects > 1 && !FileListMode)
+      return createStringError(
+          errc::invalid_argument,
+          "'o' file type is requested, but the fat object contains multiple "
+          "device objects; use 'oo' instead");
+    std::string FileList;
+
+    // Iterate through individual objects and extract them
+    for (size_t I = 0; I < NumObjects; ++I) {
+      uint64_t ObjSize = SizeVec[I];
+      StringRef ObjFileName = OutName;
+      SmallString<128> Path;
+
+      // If not in file list mode there is no need for a temporary file - output
+      // goes directly to what was specified in -outputs. The same is true for
+      // the host triple.
+      if (FileListMode) {
+        std::error_code EC =
+            sys::fs::createTemporaryFile(TempFileNameBase, "devo", Path);
+        ObjFileName = Path.data();
+
+        if (EC)
+          return createFileError(ObjFileName, EC);
+      }
+      std::error_code EC;
+      raw_fd_ostream OS(ObjFileName, EC);
+
+      if (EC)
+        return createFileError(ObjFileName, EC);
+      OS.write(ObjData, ObjSize);
+
+      if (FileListMode) {
+        // add the written file name to the output list of files
+        FileList = (Twine(FileList) + Twine(ObjFileName) + Twine("\n")).str();
+      }
+      // Move "object data" pointer to the next object within the concatenated
+      // bundle.
+      ObjData += ObjSize;
+    }
+    if (FileListMode) {
+      // dump the list of files into the file list specified in -outputs for the
+      // current target
+      std::error_code EC;
+      raw_fd_ostream OS1(OutName, EC);
+
+      if (EC)
+        return createFileError(OutName, EC);
+      OS1.write(FileList.data(), FileList.size());
+    }
     return Error::success();
   }
 
   Error WriteHeader(raw_fd_ostream &OS,
                     ArrayRef<std::unique_ptr<MemoryBuffer>> Inputs) final {
     assert(HostInputIndex != ~0u && "Host input index not defined.");
 
     // Record number of inputs.
     NumberOfInputs = Inputs.size();
+
+    // And input sizes.
+    for (unsigned I = 0; I < NumberOfInputs; ++I)
+      InputSizes.push_back(Inputs[I]->getBufferSize());
     return Error::success();
   }
 
   Error WriteBundleStart(raw_fd_ostream &OS, StringRef TargetTriple) final {
     ++NumberOfProcessedInputs;
     return Error::success();
   }
 
for (unsigned I = 0; I < NumberOfInputs; ++I) {
         if (!TempFileOrErr)
           return TempFileOrErr.takeError();
         InputFile = *TempFileOrErr;
       }
 
       ObjcopyArgs.push_back(SS.save(Twine("--add-section=") +
                                     OFFLOAD_BUNDLER_MAGIC_STR + TargetNames[I] +
                                     "=" + InputFile));
+
+      // Create temporary file with the section size contents.
+      Expected<StringRef> SizeFileOrErr = TempFiles.Create(makeArrayRef(
+          reinterpret_cast<char *>(&InputSizes[I]), sizeof(InputSizes[I])));
+      if (!SizeFileOrErr)
+        return SizeFileOrErr.takeError();
+
+      // And add one more section with target object size.
+      ObjcopyArgs.push_back(SS.save(Twine("--add-section=") +
+                                    SIZE_SECTION_PREFIX + TargetNames[I] + "=" +
+                                    *SizeFileOrErr));
     }
     ObjcopyArgs.push_back(InputFileNames[HostInputIndex]);
     ObjcopyArgs.push_back(IntermediateObj);
 
     if (Error Err = executeObjcopy(*Objcopy, ObjcopyArgs))
       return Err;
 
     // And run llvm-objcopy for the second time to update section flags.
     ObjcopyArgs.resize(1);
-    for (unsigned I = 0; I < NumberOfInputs; ++I)
+    for (unsigned I = 0; I < NumberOfInputs; ++I) {
       ObjcopyArgs.push_back(SS.save(Twine("--set-section-flags=") +
                                     OFFLOAD_BUNDLER_MAGIC_STR + TargetNames[I] +
                                     "=readonly,exclude"));
+      ObjcopyArgs.push_back(SS.save(Twine("--set-section-flags=") +
+                                    SIZE_SECTION_PREFIX + TargetNames[I] +
+                                    "=readonly,exclude"));
+    }
     ObjcopyArgs.push_back(IntermediateObj);
     ObjcopyArgs.push_back(OutputFileNames.front());
 
     if (Error Err = executeObjcopy(*Objcopy, ObjcopyArgs))
       return Err;
 
     return Error::success();
   }
Error ReadBundleEnd(MemoryBuffer &Input) final {
     size_t TripleEnd = ReadChars = FC.find("\n", ReadChars + 1);
     if (TripleEnd != FC.npos)
       // Next time we read after the new line.
       ++ReadChars;
 
     return Error::success();
   }
 
+  using FileHandler::ReadBundle; // to avoid hiding via the overload below
+
   Error ReadBundle(raw_fd_ostream &OS, MemoryBuffer &Input) final {
     StringRef FC = Input.getBuffer();
     size_t BundleStart = ReadChars;
 
     // Find end of the bundle.
     size_t BundleEnd = ReadChars = FC.find(BundleEndString, ReadChars);
 
     StringRef Bundle(&FC.data()[BundleStart], BundleEnd - BundleStart);
CreateFileHandler(MemoryBuffer &FirstInput) {
   if (FilesType == "d")
     return std::make_unique<TextFileHandler>(/*Comment=*/"#");
   if (FilesType == "ll")
     return std::make_unique<TextFileHandler>(/*Comment=*/";");
   if (FilesType == "bc")
     return std::make_unique<BinaryFileHandler>();
   if (FilesType == "s")
     return std::make_unique<TextFileHandler>(/*Comment=*/"#");
-  if (FilesType == "o")
+  if (FilesType == "o" || FilesType == "oo")
     return CreateObjectFileHandler(FirstInput);
   if (FilesType == "gch")
     return std::make_unique<BinaryFileHandler>();
   if (FilesType == "ast")
     return std::make_unique<BinaryFileHandler>();
 
   return createStringError(errc::invalid_argument,
                            "'" + FilesType + "': invalid file type specified");
static Error UnbundleFiles() {
   Expected<std::unique_ptr<FileHandler>> FileHandlerOrErr =
       CreateFileHandler(Input);
   if (!FileHandlerOrErr)
     return FileHandlerOrErr.takeError();
 
   std::unique_ptr<FileHandler> &FH = *FileHandlerOrErr;
   assert(FH);
 
+  // Seed temporary filename generation with the stem of the input file.
+  FH->SetTempFileNameBase(llvm::sys::path::stem(InputFileNames.front()));
+
   // Read the header of the bundled file.
   if (Error Err = FH->ReadHeader(Input))
     return Err;
 
   // Create a work list that consist of the map triple/output file.
   StringMap<StringRef> Worklist;
   auto Output = OutputFileNames.begin();
   for (auto &Triple : TargetNames) {
while (!Worklist.empty()) {
 
     auto Output = Worklist.find(CurTriple);
     // The file may have more bundles for other targets, that we don't care
     // about. Therefore, move on to the next triple
     if (Output == Worklist.end())
       continue;
 
     // Check if the output file can be opened and copy the bundle to it.
-    std::error_code EC;
-    raw_fd_ostream OutputFile(Output->second, EC, sys::fs::OF_None);
-    if (EC)
-      return createFileError(Output->second, EC);
-    if (Error Err = FH->ReadBundle(OutputFile, Input))
+    if (Error Err = FH->ReadBundle(Output->second, Input))
       return Err;
     if (Error Err = FH->ReadBundleEnd(Input))
       return Err;
     Worklist.erase(Output);
 
     // Record if we found the host bundle.
     if (hasHostKind(CurTriple))
       FoundHostBundle = true;
if (InputFileNames.size() != 1) {
           errc::invalid_argument,
           "only one input file supported in unbundling mode"));
     }
     if (OutputFileNames.size() != TargetNames.size()) {
       Error = true;
       reportError(createStringError(errc::invalid_argument,
                                     "number of output files and targets should "
                                     "match in unbundling mode"));
+      if (FilesType == "oo") {
+        Error = true;
+        errs() << "error: type \"oo\" cannot be used in bundling mode\n.";
+      }
     }
   } else {
     if (OutputFileNames.size() != 1) {
       Error = true;
       reportError(createStringError(
           errc::invalid_argument,
           "only one output file supported in bundling mode"));
     }