[OpenMP] Emit offloading entries for indirect target variables

[jhuber6]

Summary

OpenMP 5.1 allows emission of the indirect clause on declare target
functions, see https://www.openmp.org/spec-html/5.1/openmpsu70.html#x98-1080002.14.7.
The intended use of this is to permit calling device functions via their
associated host pointer. In order to do this the first step will be
building a map associating these variables. Doing this will require the
same offloading entry handling we use for other kernels and globals.

We intentionally emit a new global on the device side. Although it's
possible to look up the device function's address directly, this would
require changing the visibility and would prevent us from making static
functions indirect. Also, the CUDA toolchain will optimize out unused
functions and using a global prevents that. The downside is that the
runtime will need to read the global and copy its value, but there
shouldn't be any other costs.

Note that this patch just performs the codegen, currently this new
offloading entry type is unused and will be ignored by the runtime.

[JonChesterfield]

calling device functions via their associated host pointer

What does this mean? Defining a function foo such that the host and each individual target each have their own machine code for it, such that &foo on the host can be copied over to the target and then invoked to mean call the function on the local target with the same name?

If so, calling through the pointer &foo on the GPU doing a logarithmic search through a table to choose a function address to branch to sounds like something that will codegen into very slow code. Does it do that search on every call?

Is there an ambition to have &foo on the host and &foo on the (each) target return the same value, in the pointer equality sense?

Searching the linked spec for indirect finds the following

If the indirect clause is present and invoked-by-fptr evaluates to true, any procedures that appear in a to clause on the directive may be called with an indirect device invocation. If the indirect clause is present and invoked-by-fptr does not evaluate to true, any procedures that appear in a to clause on the directive may not be called with an indirect device invocation. Unless otherwise specified by an indirect clause, procedures may not be called with an indirect device invocation.

Which tells me that the indirect clause means procedures can be called with an indirect device invocation. Searching for the expression "indirect device invocation" finds that paragraph and nothing else. So... where does the spec say what this thing is?

[jhuber6]

In D157738#4586465, @JonChesterfield wrote:

calling device functions via their associated host pointer

What does this mean? Defining a function foo such that the host and each individual target each have their own machine code for it, such that &foo on the host can be copied over to the target and then invoked to mean call the function on the local target with the same name?

If so, calling through the pointer &foo on the GPU doing a logarithmic search through a table to choose a function address to branch to sounds like something that will codegen into very slow code. Does it do that search on every call?

Is there an ambition to have &foo on the host and &foo on the target return the same value, in the pointer equality sense?

That's exactly what it means, the mapping is only done for targets with indirect declared on them. The indirect calls themselves, I think @jdoerfert is implementing some specialization? He just asked me to implement this since it's related to copying of virtual classes to the device.

[RaviNarayanaswamy]

In D157738#4586465, @JonChesterfield wrote:

calling device functions via their associated host pointer

What does this mean? Defining a function foo such that the host and each individual target each have their own machine code for it, such that &foo on the host can be copied over to the target and then invoked to mean call the function on the local target with the same name?

If so, calling through the pointer &foo on the GPU doing a logarithmic search through a table to choose a function address to branch to sounds like something that will codegen into very slow code. Does it do that search on every call?

Is there an ambition to have &foo on the host and &foo on the (each) target return the same value, in the pointer equality sense?

Searching the linked spec for indirect finds the following

If the indirect clause is present and invoked-by-fptr evaluates to true, any procedures that appear in a to clause on the directive may be called with an indirect device invocation. If the indirect clause is present and invoked-by-fptr does not evaluate to true, any procedures that appear in a to clause on the directive may not be called with an indirect device invocation. Unless otherwise specified by an indirect clause, procedures may not be called with an indirect device invocation.

Which tells me that the indirect clause means procedures can be called with an indirect device invocation. Searching for the expression "indirect device invocation" finds that paragraph and nothing else. So... where does the spec say what this thing is?

Only indirect calls on the device will do a search the table. The spec does not say how it should be implemented. One could do the translation at the target region when it is mapped on the host but this will not handle all the cases.

[JonChesterfield]

If calling an indirect function pointer on the GPU requires a table lookup (keyed by host function addresses, which I didn't think we knew at GPU compile time), and we cannot distinguish indirect function pointers from function pointers, then this feature must send _every_ indirect call on the GPU through the table search in case it hits in the table and then branch on the value if it doesn't.

So if we have a few indirect openmp functions and then we call a function which makes indirect calls of a function pointer which happened to be unrelated to this feature, it's going to search that table anyway. Say qsort on a non-inlined comparison function.

This feature, as I understand it so far, therefore induces a global slowdown on every call to an unknown function. Slow function calls are a bad thing.

What am I missing?

[arsenm]

clang/lib/CodeGen/CGOpenMPRuntime.cpp

1900–1901

not a huge fan of std::optional<pointer>

1921

target global address space

1926

isn't there a store size?

[JonChesterfield]

Ok, I'm really sure this needs to be reflected in the type system. &foo for some target indirect foo gets to be larger than 8 bytes and we use a different calling convention for it. Otherwise however we carve this the type erasure is going to make unrelated calls acquire the dynamic search overhead.

e.g. store an integer in [0 N) in the 'function pointer' and use that as the index into a jump table. Host and various targets each get their own jump table. O(1) overhead, zero cost for existing/normal indirect calls.

[jhuber6]

clang/lib/CodeGen/CGOpenMPRuntime.cpp

1900–1901

This is pretty far entrenched in the Clang handling for this attribute so I don't intend to change it here.

1926

Yeah I can use that instead.

[jhuber6]

Address comments

clang/lib/CodeGen/CGOpenMPRuntime.h

public:
   /// Emit a code for initialization of declare target variable.
   /// \param VD Declare target variable.
   /// \param Addr Address of the global variable \a VD.
   /// \param PerformInit true if initialization expression is not constant.
   virtual bool emitDeclareTargetVarDefinition(const VarDecl *VD,
                                               llvm::GlobalVariable *Addr,
                                               bool PerformInit);
 
+  /// Emit code for handling declare target functions in the runtime.
+  /// \param FD Declare target function.
+  /// \param Addr Address of the global \a FD.
+  /// \param PerformInit true if initialization expression is not constant.
+  virtual void emitDeclareTargetFunction(const FunctionDecl *FD,
+                                         llvm::GlobalValue *GV);
+
   /// Creates artificial threadprivate variable with name \p Name and type \p
   /// VarType.
   /// \param VarType Type of the artificial threadprivate variable.
   /// \param Name Name of the artificial threadprivate variable.
   virtual Address getAddrOfArtificialThreadPrivate(CodeGenFunction &CGF,
                                                    QualType VarType,
                                                    StringRef Name);
 

clang/lib/CodeGen/CGOpenMPRuntime.cpp

if (VD->getType().isDestructedType() != QualType::DK_none) {
     DtorEntryInfo.ParentName = Twine(Buffer, "_dtor").toStringRef(Out);
     OMPBuilder.OffloadInfoManager.registerTargetRegionEntryInfo(
         DtorEntryInfo, Dtor, ID,
         llvm::OffloadEntriesInfoManager::OMPTargetRegionEntryDtor);
   }
   return CGM.getLangOpts().OpenMPIsTargetDevice;
 }
 
+void CGOpenMPRuntime::emitDeclareTargetFunction(const FunctionDecl *FD,
+                                                llvm::GlobalValue *GV) {
+  std::optional<OMPDeclareTargetDeclAttr *> ActiveAttr =
+      OMPDeclareTargetDeclAttr::getActiveAttr(FD);
+
+  // We only need to handle active 'indirect' declare target functions.
+  if (!ActiveAttr || !(*ActiveAttr)->getIndirect())
+    return;
+
+  // Get a mangled name to store the new device global in.
+  llvm::TargetRegionEntryInfo EntryInfo = getEntryInfoFromPresumedLoc(
+      CGM, OMPBuilder, FD->getCanonicalDecl()->getBeginLoc(), FD->getName());
+  SmallString<128> Name;
+  OMPBuilder.OffloadInfoManager.getTargetRegionEntryFnName(Name, EntryInfo);
+
+  // We need to generate a new global to hold the address of the indirectly
+  // called device function. Doing this allows us to keep the visibility and
+  // linkage of the associated function unchanged while allowing the runtime to
+  // access its value.
+  llvm::GlobalValue *Addr = GV;
+  if (CGM.getLangOpts().OpenMPIsTargetDevice) {
+    Addr = new llvm::GlobalVariable(
+        CGM.getModule(), CGM.VoidPtrTy,
+        /*isConstant=*/true, llvm::GlobalValue::ExternalLinkage, GV, Name,
+        nullptr, llvm::GlobalValue::NotThreadLocal,
+        CGM.getModule().getDataLayout().getDefaultGlobalsAddressSpace());
+    Addr->setVisibility(llvm::GlobalValue::ProtectedVisibility);
+  }
+
+  OMPBuilder.OffloadInfoManager.registerDeviceGlobalVarEntryInfo(
+      Name, Addr, CGM.GetTargetTypeStoreSize(CGM.VoidPtrTy).getQuantity(),
+      llvm::OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect,
+      llvm::GlobalValue::WeakODRLinkage);
+}
+
 Address CGOpenMPRuntime::getAddrOfArtificialThreadPrivate(CodeGenFunction &CGF,
                                                           QualType VarType,
                                                           StringRef Name) {
   std::string Suffix = getName({"artificial", ""});
   llvm::Type *VarLVType = CGF.ConvertTypeForMem(VarType);
   llvm::GlobalVariable *GAddr = OMPBuilder.getOrCreateInternalVariable(
       VarLVType, Twine(Name).concat(Suffix).str());
   if (CGM.getLangOpts().OpenMP && CGM.getLangOpts().OpenMPUseTLS &&

clang/lib/CodeGen/CodeGenModule.cpp

void CodeGenModule::EmitGlobalFunctionDefinition(GlobalDecl GD,
   SetLLVMFunctionAttributesForDefinition(D, Fn);
 
   if (const ConstructorAttr *CA = D->getAttr<ConstructorAttr>())
     AddGlobalCtor(Fn, CA->getPriority());
   if (const DestructorAttr *DA = D->getAttr<DestructorAttr>())
     AddGlobalDtor(Fn, DA->getPriority(), true);
   if (D->hasAttr<AnnotateAttr>())
     AddGlobalAnnotations(D, Fn);
+  if (getLangOpts().OpenMP && D->hasAttr<OMPDeclareTargetDeclAttr>())
+    getOpenMPRuntime().emitDeclareTargetFunction(D, GV);
 }
 
 void CodeGenModule::EmitAliasDefinition(GlobalDecl GD) {
   const auto *D = cast<ValueDecl>(GD.getDecl());
   const AliasAttr *AA = D->getAttr<AliasAttr>();
   assert(AA && "Not an alias?");
 
   StringRef MangledName = getMangledName(GD);

clang/test/OpenMP/target_indirect_codegen.cpp

+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -x c++ -triple powerpc64le-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -x c++ -triple powerpc64le-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fvisibility=protected -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefix=DEVICE
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fvisibility=protected -fopenmp-host-ir-file-path %t-host.bc -emit-pch -o %t
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fvisibility=protected -fopenmp-host-ir-file-path %t-host.bc -include-pch %t -o - | FileCheck %s --check-prefix=DEVICE
+
+// expected-no-diagnostics
+#ifndef HEADER
+#define HEADER
+
+//.
+// HOST: @[[VAR:.+]] = global i8 0, align 1
+// HOST: @[[FOO_ENTRY_NAME:.+]] = internal unnamed_addr constant [{{[0-9]+}} x i8] c"[[FOO_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_foo_l[0-9]+]]\00"
+// HOST: @.omp_offloading.entry.[[FOO_NAME]] = weak constant %struct.__tgt_offload_entry { ptr @_Z3foov, ptr @[[FOO_ENTRY_NAME]], i64 8, i32 4, i32 0 }, section "omp_offloading_entries", align 1
+// HOST: @[[BAZ_ENTRY_NAME:.+]] = internal unnamed_addr constant [{{[0-9]+}} x i8] c"[[BAZ_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_baz_l[0-9]+]]\00"
+// HOST: @.omp_offloading.entry.[[BAZ_NAME]] = weak constant %struct.__tgt_offload_entry { ptr @_Z3bazv, ptr @[[BAZ_ENTRY_NAME]], i64 8, i32 4, i32 0 }, section "omp_offloading_entries", align 1
+// HOST: @[[VAR_ENTRY_NAME:.+]] = internal unnamed_addr constant [4 x i8] c"var\00"
+// HOST: @.omp_offloading.entry.var = weak constant %struct.__tgt_offload_entry { ptr @[[VAR]], ptr @[[VAR_ENTRY_NAME]], i64 1, i32 0, i32 0 }, section "omp_offloading_entries", align 1
+// HOST: @[[BAR_ENTRY_NAME:.+]] = internal unnamed_addr constant [{{[0-9]+}} x i8] c"[[BAR_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_bar_l[0-9]+]]\00"
+// HOST: @.omp_offloading.entry.[[BAR_NAME]] = weak constant %struct.__tgt_offload_entry { ptr @_ZL3barv, ptr @[[BAR_ENTRY_NAME]], i64 8, i32 4, i32 0 }, section "omp_offloading_entries", align 1
+//.
+// DEVICE: @[[FOO_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_foo_l[0-9]+]] = protected addrspace(1) constant ptr @_Z3foov
+// DEVICE: @[[BAZ_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_baz_l[0-9]+]] = protected addrspace(1) constant ptr @_Z3bazv
+// DEVICE: @var = protected addrspace(1) global i8 0, align 1
+// DEVICE: @[[BAR_NAME:__omp_offloading_[0-9a-z]+_[0-9a-z]+_bar_l[0-9]+]] = protected addrspace(1) constant ptr @_ZL3barv
+//.
+void foo() { }
+#pragma omp declare target to(foo) indirect
+
+static void bar() { }
+#pragma omp declare target to(bar) indirect
+
+[[gnu::visibility("hidden")]] void baz() { bar(); }
+#pragma omp declare target to(baz) indirect
+
+static void unused() { };
+#pragma omp declare target to(unused) indirect
+
+void disabled() { };
+#pragma omp declare target to(disabled) indirect(false)
+
+char var = 0;
+#pragma omp declare target to(var) indirect
+
+#endif
+//.
+// HOST-DAG: !{{[0-9]+}} = !{i32 1, !"[[FOO_NAME]]", i32 4, i32 0}
+// HOST-DAG: !{{[0-9]+}} = !{i32 1, !"[[BAZ_NAME]]", i32 4, i32 1}
+// HOST-DAG: !{{[0-9]+}} = !{i32 1, !"var", i32 0, i32 2}
+// HOST-DAG: !{{[0-9]+}} = !{i32 1, !"[[BAR_NAME]]", i32 4, i32 3}
+//.

llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h

enum OMPTargetGlobalVarEntryKind : uint32_t {
     /// Mark the entry as a to declare target.
     OMPTargetGlobalVarEntryTo = 0x0,
     /// Mark the entry as a to declare target link.
     OMPTargetGlobalVarEntryLink = 0x1,
     /// Mark the entry as a declare target enter.
     OMPTargetGlobalVarEntryEnter = 0x2,
     /// Mark the entry as having no declare target entry kind.
     OMPTargetGlobalVarEntryNone = 0x3,
+    /// Mark the entry as a declare target indirect global.
+    OMPTargetGlobalVarEntryIndirect = 0x4,
   };
 
   /// Kind of device clause for declare target variables
   /// and functions
   /// NOTE: Currently not used as a part of a variable entry
   /// used for Flang and Clang to interface with the variable
   /// related registration functions
   enum OMPTargetDeviceClauseKind : uint32_t {
     /// The target is marked for all devices
     OMPTargetDeviceClauseAny = 0x0,
     /// The target is marked for non-host devices
     OMPTargetDeviceClauseNoHost = 0x1,
     /// The target is marked for host devices
     OMPTargetDeviceClauseHost = 0x2,
     /// The target is marked as having no clause
     OMPTargetDeviceClauseNone = 0x3
   };
 
   /// Device global variable entries info.
   class OffloadEntryInfoDeviceGlobalVar final : public OffloadEntryInfo {
     /// Type of the global variable.
     int64_t VarSize;
     GlobalValue::LinkageTypes Linkage;
+    const std::string VarName;
 
   public:
     OffloadEntryInfoDeviceGlobalVar()
         : OffloadEntryInfo(OffloadingEntryInfoDeviceGlobalVar) {}
     explicit OffloadEntryInfoDeviceGlobalVar(unsigned Order,
                                              OMPTargetGlobalVarEntryKind Flags)
         : OffloadEntryInfo(OffloadingEntryInfoDeviceGlobalVar, Order, Flags) {}
     explicit OffloadEntryInfoDeviceGlobalVar(unsigned Order, Constant *Addr,
                                              int64_t VarSize,
                                              OMPTargetGlobalVarEntryKind Flags,
-                                             GlobalValue::LinkageTypes Linkage)
+                                             GlobalValue::LinkageTypes Linkage,
+                                             const std::string &VarName)
         : OffloadEntryInfo(OffloadingEntryInfoDeviceGlobalVar, Order, Flags),
-          VarSize(VarSize), Linkage(Linkage) {
+          VarSize(VarSize), Linkage(Linkage), VarName(VarName) {
       setAddress(Addr);
     }
 
     int64_t getVarSize() const { return VarSize; }
+    StringRef getVarName() const { return VarName; }
     void setVarSize(int64_t Size) { VarSize = Size; }
     GlobalValue::LinkageTypes getLinkage() const { return Linkage; }
     void setLinkage(GlobalValue::LinkageTypes LT) { Linkage = LT; }
     static bool classof(const OffloadEntryInfo *Info) {
       return Info->getKind() == OffloadingEntryInfoDeviceGlobalVar;
     }
   };
 
void emitOffloadingArrays(
       InsertPointTy AllocaIP, InsertPointTy CodeGenIP, MapInfosTy &CombinedInfo,
       TargetDataInfo &Info, bool IsNonContiguous = false,
       function_ref<void(unsigned int, Value *)> DeviceAddrCB = nullptr,
       function_ref<Value *(unsigned int)> CustomMapperCB = nullptr);
 
   /// Creates offloading entry for the provided entry ID \a ID, address \a
   /// Addr, size \a Size, and flags \a Flags.
   void createOffloadEntry(Constant *ID, Constant *Addr, uint64_t Size,
-                          int32_t Flags, GlobalValue::LinkageTypes);
+                          int32_t Flags, GlobalValue::LinkageTypes,
+                          StringRef Name = "");
 
   /// The kind of errors that can occur when emitting the offload entries and
   /// metadata.
   enum EmitMetadataErrorKind {
     EMIT_MD_TARGET_REGION_ERROR,
     EMIT_MD_DECLARE_TARGET_ERROR,
     EMIT_MD_GLOBAL_VAR_LINK_ERROR
   };

llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp

while (!Worklist.empty()) {
     for (BasicBlock *SuccBB : successors(BB))
       if (BlockSet.insert(SuccBB).second)
         Worklist.push_back(SuccBB);
   }
 }
 
 void OpenMPIRBuilder::createOffloadEntry(Constant *ID, Constant *Addr,
                                          uint64_t Size, int32_t Flags,
-                                         GlobalValue::LinkageTypes) {
+                                         GlobalValue::LinkageTypes,
+                                         StringRef Name) {
   if (!Config.isGPU()) {
-    emitOffloadingEntry(ID, Addr->getName(), Size, Flags);
+    emitOffloadingEntry(ID, Name.empty() ? Addr->getName() : Name, Size, Flags);
     return;
   }
   // TODO: Add support for global variables on the device after declare target
   // support.
   Function *Fn = dyn_cast<Function>(Addr);
   if (!Fn)
     return;
 
if (const auto *CE =
         }
         break;
       default:
         break;
       }
 
       // Hidden or internal symbols on the device are not externally visible.
       // We should not attempt to register them by creating an offloading
-      // entry.
+      // entry. Indirect variables are handled separately on the device.
       if (auto *GV = dyn_cast<GlobalValue>(CE->getAddress()))
-        if (GV->hasLocalLinkage() || GV->hasHiddenVisibility())
+        if ((GV->hasLocalLinkage() || GV->hasHiddenVisibility()) &&
+            Flags != OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect)
           continue;
 
+      // Indirect globals need to use a special name that doesn't match the name
+      // of the associated host global.
+      if (Flags == OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect)
+        createOffloadEntry(CE->getAddress(), CE->getAddress(), CE->getVarSize(),
+                           Flags, CE->getLinkage(), CE->getVarName());
+      else
         createOffloadEntry(CE->getAddress(), CE->getAddress(), CE->getVarSize(),
                            Flags, CE->getLinkage());
 
     } else {
       llvm_unreachable("Unsupported entry kind.");
     }
   }
 }
 
 void TargetRegionEntryInfo::getTargetRegionEntryFnName(
if (hasDeviceGlobalVarEntryInfo(VarName)) {
       assert(Entry.isValid() && Entry.getFlags() == Flags &&
              "Entry not initialized!");
       if (Entry.getVarSize() == 0) {
         Entry.setVarSize(VarSize);
         Entry.setLinkage(Linkage);
       }
       return;
     }
+    if (Flags == OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect)
       OffloadEntriesDeviceGlobalVar.try_emplace(VarName, OffloadingEntriesNum,
-                                              Addr, VarSize, Flags, Linkage);
+                                                Addr, VarSize, Flags, Linkage,
+                                                VarName.str());
+    else
+      OffloadEntriesDeviceGlobalVar.try_emplace(
+          VarName, OffloadingEntriesNum, Addr, VarSize, Flags, Linkage, "");
     ++OffloadingEntriesNum;
   }
 }
 
 void OffloadEntriesInfoManager::actOnDeviceGlobalVarEntriesInfo(
     const OffloadDeviceGlobalVarEntryInfoActTy &Action) {
   // Scan all target region entries and perform the provided action.
   for (const auto &E : OffloadEntriesDeviceGlobalVar)