[clang-offload-wrapper] Disabled ELF offload notes embedding by default.

[vzakhari]

Summary

This change-set puts 93d08acaacec951dbb302f77eeae51974985b6b2 functionality
under -add-omp-offload-notes switch that is OFF by default.
CUDA toolchain is not able to handle ELF images with LLVMOMPOFFLOAD
notes for unknown reason (see https://reviews.llvm.org/D99551#2950272).
I disable the ELF notes embedding until the CUDA issue is triaged and resolved.

[jhuber6]

LGTM, applications compile and run now.

[jdoerfert]

I would prefer to revert 93d08acaacec951dbb302f77eeae51974985b6b2 now and then fix the CUDA toolchain rather than making it someone else's problem.

[vzakhari]

In D108246#2950650, @jdoerfert wrote:

I would prefer to revert 93d08acaacec951dbb302f77eeae51974985b6b2 now and then fix the CUDA toolchain rather than making it someone else's problem.

@jdoerfert, can we please keep the code upstream? I've spent some time already to resolve the conflicts downstream, and I do not really want to do it once again. I am taking the AR to triage the CUDA issue.

[jdoerfert]

As discussed, LG, we will look into the NVIDIA GPU problem now to get rid of this again.

[JonChesterfield]

Amusingly similar to D108303. LGTM.

[JonChesterfield]

clang/tools/clang-offload-wrapper/ClangOffloadWrapper.cpp

79

I'd have probably gone with an explicit false here but it doesn't make much difference.

clang/test/Driver/clang-offload-wrapper.c

 //
 // Generate a file to wrap.
 //
 // RUN: echo 'Content of device file' > %t.tgt
 
 //
 // Check bitcode produced by the wrapper tool.
 //
-// RUN: clang-offload-wrapper -target=x86_64-pc-linux-gnu -o %t.wrapper.bc %t.tgt 2>&1 | FileCheck %s --check-prefix ELF-WARNING
+// RUN: clang-offload-wrapper -add-omp-offload-notes -target=x86_64-pc-linux-gnu -o %t.wrapper.bc %t.tgt 2>&1 | FileCheck %s --check-prefix ELF-WARNING
 // RUN: llvm-dis %t.wrapper.bc -o - | FileCheck %s --check-prefix CHECK-IR
 
 // ELF-WARNING: is not an ELF image, so notes cannot be added to it.
 // CHECK-IR: target triple = "x86_64-pc-linux-gnu"
 
 // CHECK-IR-DAG: [[ENTTY:%.+]] = type { i8*, i8*, i{{32|64}}, i32, i32 }
 // CHECK-IR-DAG: [[IMAGETY:%.+]] = type { i8*, i8*, [[ENTTY]]*, [[ENTTY]]* }
 // CHECK-IR-DAG: [[DESCTY:%.+]] = type { i32, [[IMAGETY]]*, [[ENTTY]]*, [[ENTTY]]* }

 // CHECK-IR:   call void @__tgt_unregister_lib([[DESCTY]]* [[DESC]])
 // CHECK-IR:   ret void
 
 // CHECK-IR: declare void @__tgt_unregister_lib([[DESCTY]]*)
 
 // Check that clang-offload-wrapper adds LLVMOMPOFFLOAD notes
 // into the ELF offload images:
 // RUN: yaml2obj %S/Inputs/empty-elf-template.yaml -o %t.64le -DBITS=64 -DENCODING=LSB
-// RUN: clang-offload-wrapper -target=x86_64-pc-linux-gnu -o %t.wrapper.elf64le.bc %t.64le
+// RUN: clang-offload-wrapper -add-omp-offload-notes -target=x86_64-pc-linux-gnu -o %t.wrapper.elf64le.bc %t.64le
 // RUN: llvm-dis %t.wrapper.elf64le.bc -o - | FileCheck %s --check-prefix OMPNOTES
 // RUN: yaml2obj %S/Inputs/empty-elf-template.yaml -o %t.64be -DBITS=64 -DENCODING=MSB
-// RUN: clang-offload-wrapper -target=x86_64-pc-linux-gnu -o %t.wrapper.elf64be.bc %t.64be
+// RUN: clang-offload-wrapper -add-omp-offload-notes -target=x86_64-pc-linux-gnu -o %t.wrapper.elf64be.bc %t.64be
 // RUN: llvm-dis %t.wrapper.elf64be.bc -o - | FileCheck %s --check-prefix OMPNOTES
 // RUN: yaml2obj %S/Inputs/empty-elf-template.yaml -o %t.32le -DBITS=32 -DENCODING=LSB
-// RUN: clang-offload-wrapper -target=x86_64-pc-linux-gnu -o %t.wrapper.elf32le.bc %t.32le
+// RUN: clang-offload-wrapper -add-omp-offload-notes -target=x86_64-pc-linux-gnu -o %t.wrapper.elf32le.bc %t.32le
 // RUN: llvm-dis %t.wrapper.elf32le.bc -o - | FileCheck %s --check-prefix OMPNOTES
 // RUN: yaml2obj %S/Inputs/empty-elf-template.yaml -o %t.32be -DBITS=32 -DENCODING=MSB
-// RUN: clang-offload-wrapper -target=x86_64-pc-linux-gnu -o %t.wrapper.elf32be.bc %t.32be
+// RUN: clang-offload-wrapper -add-omp-offload-notes -target=x86_64-pc-linux-gnu -o %t.wrapper.elf32be.bc %t.32be
 // RUN: llvm-dis %t.wrapper.elf32be.bc -o - | FileCheck %s --check-prefix OMPNOTES
 
 // There is no clean way for extracting the offload image
 // from the object file currently, so try to find
 // the inserted ELF notes in the device image variable's
 // initializer:
 // OMPNOTES: @{{.+}} = internal unnamed_addr constant [{{[0-9]+}} x i8] c"{{.*}}LLVMOMPOFFLOAD{{.*}}LLVMOMPOFFLOAD{{.*}}LLVMOMPOFFLOAD{{.*}}"

clang/tools/clang-offload-wrapper/ClangOffloadWrapper.cpp

Target("target", cl::Required,
            cl::value_desc("triple"), cl::cat(ClangOffloadWrapperCategory));
 
 static cl::opt<bool> SaveTemps(
     "save-temps",
     cl::desc("Save temporary files that may be produced by the tool. "
              "This option forces print-out of the temporary files' names."),
     cl::Hidden);
 
+static cl::opt<bool> AddOpenMPOffloadNotes(
+    "add-omp-offload-notes",
+    cl::desc("Add LLVMOMPOFFLOAD ELF notes to ELF device images."), cl::Hidden);
+
 namespace {
 
 class BinaryWrapper {
   LLVMContext C;
   Module M;
 
   StructType *EntryTy = nullptr;
   StructType *ImageTy = nullptr;
int main(int argc, const char **argv) {
   for (const std::string &File : Inputs) {
     ErrorOr<std::unique_ptr<MemoryBuffer>> BufOrErr =
         MemoryBuffer::getFileOrSTDIN(File);
     if (!BufOrErr) {
       reportError(createFileError(File, BufOrErr.getError()));
       return 1;
     }
     std::unique_ptr<MemoryBuffer> Buffer(std::move(*BufOrErr));
-    if (File != "-") {
+    if (File != "-" && AddOpenMPOffloadNotes) {
       // Adding ELF notes for STDIN is not supported yet.
       Buffer = Wrapper.addELFNotes(std::move(Buffer), File);
     }
     const std::unique_ptr<MemoryBuffer> &Buf =
         Buffers.emplace_back(std::move(Buffer));
     Images.emplace_back(Buf->getBufferStart(), Buf->getBufferSize());
   }