[IRLinker] Suppress linker warnings when linking with CUDA libdevice.

[tra]

Summary

libdevice bitcode provided by NVIDIA is linked with clang/LLVM-generated IR
which uses nvptx*-nvidia-cuda triple. We need to mark them as compatible.

[jdoerfert]

FWIW, with OpenMP we now see 3 warnings:

[ 90%] Building CXX object ...
warning: linking module '/usr/local/cuda-11.0/nvvm/libdevice/libdevice.10.bc': Linking two modules of different data layouts: '/usr/local/cuda-11.0/nvvm/libdevice/libdevice.10.bc' is '' whereas '...' is 'e-i64:64-i128:128-v16:16-v32:32-n16:32:64'
 [-Wlinker-warnings]
warning: linking module '/usr/local/cuda-11.0/nvvm/libdevice/libdevice.10.bc': Linking two modules of different target triples: '/usr/local/cuda-11.0/nvvm/libdevice/libdevice.10.bc' is 'nvptx64-nvidia-gpulibs' whereas '...' is 'nvptx64-nvidia-cuda'
 [-Wlinker-warnings]
warning: linking module '/soft/llvm/main-20210824/lib/libomptarget-nvptx-sm_80.bc': Linking two modules of different target triples: '/soft/llvm/main-20210824/lib/libomptarget-nvptx-sm_80.bc' is 'nvptx64' whereas '...' is 'nvptx64-nvidia-cuda'
 [-Wlinker-warnings]

This patch should fix the middle one. The last one we can fix ourselves by always embedding canonical/full triples. The first one I am not sure about yet.

Long story short, I think this makes sense. LGTM.

[tra]

I've checked libdevice in different CUDA versions and the results are rather inconsistent:

For triples, CUDA-10+ uses nvptx64-nvidia-gpulibs, older versions use nvptx-unknown-unknown
Data layout is absent until CUDA-11.1.

Instead of chasing all the variations, I wonder if we should just disable linker warning for libdevice somewhere in clang.

[jdoerfert]

In D108835#2969791, @tra wrote:

I've checked libdevice in different CUDA versions and the results are rather inconsistent:

For triples, CUDA-10+ uses nvptx64-nvidia-gpulibs, older versions use nvptx-unknown-unknown
Data layout is absent until CUDA-11.1.

Instead of chasing all the variations, I wonder if we should just disable linker warning for libdevice somewhere in clang.

I was initially thinking to have a flag to do so. With this patch I though we could make it work without. If this patch doesn't work,
I agree, linking libdevice should set a "don't look too closely" flag.

[jdoerfert]

The warnings we see are most likely caused by D108603, just to have a connection to the discussion.

[tra]

How is libomptarget-nvptx-sm_80.bc built? Perhaps it should be changed to use a more detailed triple. Perhaps match libdevice's nvptx64-nvidia-gpulibs or have a distinct openmp-specific one?

Do we have to deal with the libomptarget-nvptx-sm_80.bc shipped with the current nvptx64 triple or is it shipped with clang and we don't need to worry about the older versions in the wild?

[jdoerfert]

In D108835#2972612, @tra wrote:

How is libomptarget-nvptx-sm_80.bc built? Perhaps it should be changed to use a more detailed triple. Perhaps match libdevice's nvptx64-nvidia-gpulibs or have a distinct openmp-specific one?

Do we have to deal with the libomptarget-nvptx-sm_80.bc shipped with the current nvptx64 triple or is it shipped with clang and we don't need to worry about the older versions in the wild?

We do not have to care about older versions and we can change the triple there. I'm not sure changing it to gpulibs is the right thing though.
My plan was to normalize nvptx64 to nvptx-nvidia-cuda in the frontend such that user can use any subset of that normalized triple w/o warning.
Unsure what we gain by having "gpulibs" and special handling there. That said we could certainly do it.

I'm however unsure how that helps us with libdevice, libomptarget-nvptx is in our control so certainly less of a problem.

[tra]

My plan was to normalize nvptx64 to nvptx-nvidia-cuda in the frontend such that user can use any subset of that normalized triple w/o warning.

SGTM. So, for now I'll just deal with the libdevice-related warnings.

[jdoerfert]

In D108835#2973021, @tra wrote:

My plan was to normalize nvptx64 to nvptx-nvidia-cuda in the frontend such that user can use any subset of that normalized triple w/o warning.

SGTM. So, for now I'll just deal with the libdevice-related warnings.

Yes, libdevice we need to somehow make work for all the version incl. mismatching/missing target information.

[tra]

Suppress warnings about triple and DataLayout mismatches related to CUDA's
libdevice. Tested on CUDA versions 8.0-11.3

[tra]

llvm/lib/Support/Triple.cpp

1643 ↗

(On Diff #369594)

I think this could be incorporated into the SuppressDLWarning calculation above. This way we'll keep this libdevice quirk handling to one place. I'll update the patch tomorrow.

[tra]

Consolidated the checks into IRMover.

[tra]

OK, this patch no longer produces warnings about Triple and DL when we're linking in libbdevice from CUDA-8.0 through 11.4.
I've tried to make it as specific as I could.

[jdoerfert]

We can actually test this, right?
Commit subject and message needs to be updated too.

[tra]

Added tests.

[jdoerfert]

LG :)

llvm/lib/Linker/IRMover.cpp

 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/GVMaterializer.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/PseudoProbe.h"
 #include "llvm/IR/TypeFinder.h"
 #include "llvm/Object/ModuleSymbolTable.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include <utility>
 using namespace llvm;
 
 //===----------------------------------------------------------------------===//
 // TypeMap implementation.
 //===----------------------------------------------------------------------===//
 
if (SrcM->getMaterializer())
     if (Error Err = SrcM->getMaterializer()->materializeMetadata())
       return Err;
 
   // Inherit the target data from the source module if the destination module
   // doesn't have one already.
   if (DstM.getDataLayout().isDefault())
     DstM.setDataLayout(SrcM->getDataLayout());
 
-  if (SrcM->getDataLayout() != DstM.getDataLayout()) {
+  // Copy the target triple from the source to dest if the dest's is empty.
+  if (DstM.getTargetTriple().empty() && !SrcM->getTargetTriple().empty())
+    DstM.setTargetTriple(SrcM->getTargetTriple());
+
+  Triple SrcTriple(SrcM->getTargetTriple()), DstTriple(DstM.getTargetTriple());
+
+  // During CUDA compilation we have to link with the bitcode supplied with
+  // CUDA. libdevice bitcode either has no data layout set (pre-CUDA-11), or has
+  // the layout that is different from the one used by LLVM/clang (it does not
+  // include i128). Issuing a warning is not very helpful as there's not much
+  // the user can do about it.
+  bool EnableDLWarning = true;
+  bool EnableTripleWarning = true;
+  if (SrcTriple.isNVPTX() && DstTriple.isNVPTX()) {
+    std::string ModuleId = SrcM->getModuleIdentifier();
+    StringRef FileName = llvm::sys::path::filename(ModuleId);
+    bool SrcIsLibDevice =
+        FileName.startswith("libdevice") && FileName.endswith(".10.bc");
+    bool SrcHasLibDeviceDL =
+        (SrcM->getDataLayoutStr().empty() ||
+         SrcM->getDataLayoutStr() == "e-i64:64-v16:16-v32:32-n16:32:64");
+    // libdevice bitcode uses nvptx64-nvidia-gpulibs or just
+    // 'nvptx-unknown-unknown' triple (before CUDA-10.x) and is compatible with
+    // all NVPTX variants.
+    bool SrcHasLibDeviceTriple = (SrcTriple.getVendor() == Triple::NVIDIA &&
+                                  SrcTriple.getOSName() == "gpulibs") ||
+                                 (SrcTriple.getVendorName() == "unknown" &&
+                                  SrcTriple.getOSName() == "unknown");
+    EnableTripleWarning = !(SrcIsLibDevice && SrcHasLibDeviceTriple);
+    EnableDLWarning = !(SrcIsLibDevice && SrcHasLibDeviceDL);
+  }
+
+  if (EnableDLWarning && (SrcM->getDataLayout() != DstM.getDataLayout())) {
     emitWarning("Linking two modules of different data layouts: '" +
                 SrcM->getModuleIdentifier() + "' is '" +
                 SrcM->getDataLayoutStr() + "' whereas '" +
                 DstM.getModuleIdentifier() + "' is '" +
                 DstM.getDataLayoutStr() + "'\n");
   }
 
-  // Copy the target triple from the source to dest if the dest's is empty.
-  if (DstM.getTargetTriple().empty() && !SrcM->getTargetTriple().empty())
-    DstM.setTargetTriple(SrcM->getTargetTriple());
-
-  Triple SrcTriple(SrcM->getTargetTriple()), DstTriple(DstM.getTargetTriple());
-
-  if (!SrcM->getTargetTriple().empty()&&
+  if (EnableTripleWarning && !SrcM->getTargetTriple().empty() &&
       !SrcTriple.isCompatibleWith(DstTriple))
     emitWarning("Linking two modules of different target triples: '" +
                 SrcM->getModuleIdentifier() + "' is '" +
                 SrcM->getTargetTriple() + "' whereas '" +
                 DstM.getModuleIdentifier() + "' is '" + DstM.getTargetTriple() +
                 "'\n");
 
   DstM.setTargetTriple(SrcTriple.merge(DstTriple));

llvm/test/Linker/Inputs/libdevice-cuda-10.ll

+target triple = "nvptx64-nvidia-gpulibs"
+

llvm/test/Linker/Inputs/libdevice-cuda-11.ll

+target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64"
+target triple = "nvptx64-nvidia-gpulibs"
+

llvm/test/Linker/Inputs/libdevice-cuda-9.ll

+target triple = "nvptx-unknown-unknown"
+

llvm/test/Linker/Inputs/not-a-libdevice.ll

+target triple = "nvptx64-nvidia-nosuchthing"
+target datalayout = "e-i64:64-i128:128-v32:32-n16:32:64"

llvm/test/Linker/cuda-libdevice.ll

+; Prepare bitcode files.
+; RUN: rm -rf %t && mkdir -p %t
+; RUN: llvm-as %s -o %t/main.bc
+; RUN: llvm-as %p/Inputs/libdevice-cuda-9.ll -o %t/libdevice.compute_35.10.bc
+; RUN: llvm-as %p/Inputs/libdevice-cuda-10.ll -o %t/libdevice.10.bc
+; RUN: llvm-as %p/Inputs/libdevice-cuda-11.ll -o %t/libdevice.11.10.bc
+; RUN: llvm-as %p/Inputs/libdevice-cuda-9.ll -o %t/correct-libdevice-wrong-filename.bc
+; RUN: llvm-as %p/Inputs/not-a-libdevice.ll -o %t/libdevice-with-wrong-info.bc
+
+; No warnings expected when we link with libdevice variants
+; RUN: llvm-link %t/main.bc %t/libdevice.compute_35.10.bc -S 2>&1 \
+; RUN:  | FileCheck --check-prefixes COMMON,NOWARN %s
+; RUN: llvm-link %t/main.bc %t/libdevice.10.bc -S 2>&1 \
+; RUN:  | FileCheck --check-prefixes COMMON,NOWARN %s
+; RUN: llvm-link %t/main.bc %t/libdevice.11.10.bc -S 2>&1 \
+; RUN:  | FileCheck --check-prefixes COMMON,NOWARN %s
+
+; But make sure we still issue warnings if we see unexpected filename, or
+; unexpected triple or datalayout within a libdevice filename.
+; RUN: llvm-link %t/main.bc %t/correct-libdevice-wrong-filename.bc -S 2>&1 \
+; RUN:  | FileCheck --check-prefixes COMMON,WARN-TRIPLE %s
+; RUN: llvm-link %t/main.bc %t/libdevice-with-wrong-info.bc -S 2>&1 \
+; RUN:  | FileCheck --check-prefixes COMMON,WARN-TRIPLE,WARN-DL %s
+
+
+target triple = "nvptx64-nvidia-cuda"
+target datalayout = "e-i64:64-i128:128-v16:16-v32:32-n16:32:64"
+
+; WARN-TRIPLE-DAG: warning: Linking two modules of different target triples:
+; WARN-DL-DAG: warning: Linking two modules of different data layouts:
+
+; NOWARN-NOT: warning:
+; COMMON-DAG: target triple = "nvptx64-nvidia-cuda"
+; NOWARN-NOT: warning: