[AMDGPU] Place global constructors in .init_array and .fini_array

[jhuber6]

Summary

For the GPU, we emit external kernels that call the initializers and
constructors, however if we had a persistent kernel like in the _start
kernel for the libc project, we could initialize the standard way of
calling constructors. This patch adds new global variables containing
pointers to the constructors to be called. If these are placed in the
.init_array and .fini_array sections, then the backend will handle
them specially. The linker will then provide the __init_array_ and
__fini_array_ sections to traverse them. An implementation would look
like this.

extern uintptr_t __init_array_start[];
extern uintptr_t __init_array_end[];
extern uintptr_t __fini_array_start[];
extern uintptr_t __fini_array_end[];

using InitCallback = void(int, char **, char **);
using FiniCallback = void(void);

extern "C" [[gnu::visibility("protected"), clang::amdgpu_kernel]] void
_start(int argc, char **argv, char **envp) {
  uint64_t init_array_size = __init_array_end - __init_array_start;
  for (uint64_t i = 0; i < init_array_size; ++i)
    reinterpret_cast<InitCallback *>(__init_array_start[i])(argc, argv, env);
  uint64_t fini_array_size = __fini_array_end - __fini_array_start;
  for (uint64_t i = 0; i < fini_array_size; ++i)
    reinterpret_cast<FiniCallback *>(__fini_array_start[i])();
}

[jhuber6]

Add section check in test.

[yaxunl]

I am curious how other targets handle @llvm.global_ctors. Is there some generic LLVM pass to change them to .init_array ?

[jhuber6]

In D149340#4301953, @yaxunl wrote:

I am curious how other targets handle @llvm.global_ctors. Is there some generic LLVM pass to change them to .init_array ?

I don't think we could fold this AMDGPU lowering pass since it need to do its own ctor / dtor handling for making the kernels. I actually don't know where the lowering happens for generic targets, I could look if it's important.

[jhuber6]

Add priority

[jhuber6]

Rebase

[jhuber6]

Remove leftoever debug prints.

[jdoerfert]

Does this work for non-AMD hardware?

Is this just adding the globals and not actually using this mechanism?

[jhuber6]

In D149340#4302728, @jdoerfert wrote:

Does this work for non-AMD hardware?

Is this just adding the globals and not actually using this mechanism?

This is all that's required on a platform with a functioning linker, Nvidia need not apply.

[jhuber6]

In D149340#4302728, @jdoerfert wrote:

Does this work for non-AMD hardware?

Is this just adding the globals and not actually using this mechanism?

If you're talking about the expected usage, it's in the commit header. I'm planning on adding that to libc in a separate patch.

[jdoerfert]

So we have different schemes for AMD and NVIDIA? That does not sound good.

[jhuber6]

In D149340#4303525, @jdoerfert wrote:

So we have different schemes for AMD and NVIDIA? That does not sound good.

There's no choice, Nvidia offers no way to export variables to sections and an incomplete ELF linker. I'd rather have AMDGPU do what every other target does and have Nvidia be the black sheep.

[MaskRay]

I know little about GPU. The generic code emittings constructors to assembly is AsmPrinter::emitSpecialLLVMGlobal and AsmPrinter::emitXXStructorList. It will use .init_array.

The priority works this way. Note that in the absence of a .N suffix, .init_array has the highest priority. https://maskray.me/blog/2021-11-07-init-ctors-init-array

a.o:(.init_array.1) b.o:(.init_array.1)
a.o:(.init_array.2) b.o:(.init_array.2)
...
a.o:(.init_array.65533) b.o:(.init_array.65533)
a.o:(.init_array.65534) b.o:(.init_array.65534)
a.o:(.init_array) b.o:(.init_array)

[jhuber6]

In D149340#4303678, @MaskRay wrote:

I know little about GPU. The generic code emittings constructors to assembly is AsmPrinter::emitSpecialLLVMGlobal and AsmPrinter::emitXXStructorList. It will use .init_array.

This pass currently deletes the global list before making it to the assembly printer, maybe we will get some of this behaviour if we don't do that?

The priority works this way. Note that in the absence of a .N suffix, .init_array has the highest priority. https://maskray.me/blog/2021-11-07-init-ctors-init-array

a.o:(.init_array.1) b.o:(.init_array.1)
a.o:(.init_array.2) b.o:(.init_array.2)
...
a.o:(.init_array.65533) b.o:(.init_array.65533)
a.o:(.init_array.65534) b.o:(.init_array.65534)
a.o:(.init_array) b.o:(.init_array)

AMD's linker is lld and targets ELF, so it should follow the exact same rules as you are familiar with.

[jhuber6]

Update, turns out that if we don't delete the ctor / dtor here it does the
default approach which is exactly what we want. The only changes required are
making sure we don't create duplicate kernels.

[yaxunl]

LGTM. Thanks.

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

I noticed these functions are called not following the priority.

However, I guess that is out of the scope of this patch.

[jhuber6]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

True, it was like that when I got here. Do you know who the current user is for this feature?

[yaxunl]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

Currently, all HIP programs use this feature when -fsanitize=addr is used.

But they do not care about priority yet.

[jhuber6]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

FWIW we could change this to be a single kernel that calls an ASAN library function, which implements the method in the commit header to traverse the list in priority order.

[yaxunl]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

We want to keep this feature as a generic approach to support dynamic initialization.

[jhuber6]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

This will also cause a duplicate symbol if someone compiles without monolithic LTO. I'm assuming that's the expected behavior.

[yaxunl]

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

76

That is fine. HIP runtime will launch all of them as long as they have init kernel metadata. It is not based on the kernel name.

llvm/lib/Target/AMDGPU/AMDGPUCtorDtorLowering.cpp

 #define DEBUG_TYPE "amdgpu-lower-ctor-dtor"
 
 namespace {
 
 static Function *createInitOrFiniKernelFunction(Module &M, bool IsCtor) {
   StringRef InitOrFiniKernelName = "amdgcn.device.init";
   if (!IsCtor)
     InitOrFiniKernelName = "amdgcn.device.fini";
+  if (Function *F = M.getFunction(InitOrFiniKernelName))
+    return F;
 
   Function *InitOrFiniKernel = Function::createWithDefaultAttr(
       FunctionType::get(Type::getVoidTy(M.getContext()), false),
       GlobalValue::ExternalLinkage, 0, InitOrFiniKernelName, &M);
   BasicBlock *InitOrFiniKernelBB =
       BasicBlock::Create(M.getContext(), "", InitOrFiniKernel);
   ReturnInst::Create(M.getContext(), InitOrFiniKernelBB);
 
static bool createInitOrFiniKernel(Module &M, StringRef GlobalName,
 
   Function *InitOrFiniKernel = createInitOrFiniKernelFunction(M, IsCtor);
   IRBuilder<> IRB(InitOrFiniKernel->getEntryBlock().getTerminator());
 
   FunctionType *ConstructorTy = InitOrFiniKernel->getFunctionType();
 
   for (Value *V : GA->operands()) {
     auto *CS = cast<ConstantStruct>(V);
+    bool AlreadyRegistered =
+        llvm::any_of(CS->getOperand(1)->uses(), [=](Use &U) {
+          if (auto *CB = dyn_cast<CallBase>(U.getUser()))
+            if (CB->getCaller() == InitOrFiniKernel)
+              return true;
+          return false;
+        });
+    if (!AlreadyRegistered)
       IRB.CreateCall(ConstructorTy, CS->getOperand(1));
   }
 
   appendToUsed(M, {InitOrFiniKernel});
-
-  GV->eraseFromParent();
   return true;
 }
 
 static bool lowerCtorsAndDtors(Module &M) {
   bool Modified = false;
   Modified |= createInitOrFiniKernel(M, "llvm.global_ctors", /*IsCtor =*/true);
   Modified |= createInitOrFiniKernel(M, "llvm.global_dtors", /*IsCtor =*/false);
   return Modified;
 }
 
 class AMDGPUCtorDtorLoweringLegacy final : public ModulePass {
 public:
   static char ID;
   AMDGPUCtorDtorLoweringLegacy() : ModulePass(ID) {}
-  bool runOnModule(Module &M) override {
-    return lowerCtorsAndDtors(M);
-  }
+  bool runOnModule(Module &M) override { return lowerCtorsAndDtors(M); }
 };
 
 } // End anonymous namespace
 
 PreservedAnalyses AMDGPUCtorDtorLoweringPass::run(Module &M,
                                                   ModuleAnalysisManager &AM) {
   return lowerCtorsAndDtors(M) ? PreservedAnalyses::none()
                                : PreservedAnalyses::all();

llvm/test/CodeGen/AMDGPU/lower-ctor-dtor-constexpr-alias.ll

 ; Check a constantexpr addrspacecast
 @llvm.global_dtors = appending addrspace(1) global [1 x { i32, ptr, ptr }] [
   { i32, ptr, ptr } { i32 1, ptr addrspacecast (ptr addrspace(1) @bar to ptr), i8* null }
 ]
 
 @foo.alias = hidden alias void (), ptr @foo
 
 ;.
-; CHECK-NOT: @llvm.global_ctors
-; CHECK-NOT: @llvm.global_dtors
 ; CHECK: @llvm.used = appending global [2 x ptr] [ptr @amdgcn.device.init, ptr @amdgcn.device.fini], section "llvm.metadata"
 ; CHECK: @foo.alias = hidden alias void (), ptr @foo
 ;.
 define void @foo() {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:    ret void
 ;
   ret void

llvm/test/CodeGen/AMDGPU/lower-ctor-dtor-existing.ll

 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-ctor-dtor < %s | FileCheck %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx700 -filetype=obj -o - < %s | llvm-readelf -s - 2>&1 | FileCheck %s -check-prefix=CHECK-VIS
 
 ; Make sure there's no crash or error if amdgcn.device.init or
 ; amdgcn.device.fini already exist.
 
 @llvm.global_ctors = appending addrspace(1) global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1, ptr @foo, ptr null }]
 @llvm.global_dtors = appending addrspace(1) global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1, ptr @bar, ptr null }]
 
-; CHECK-NOT: @llvm.global_ctors
-; CHECK-NOT: @llvm.global_dtors
-
-; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.init() #0
-; CHECK-NEXT: store
-; CHECK-NEXT: ret void
-
-; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.fini() #1
-; CHECK-NEXT: store
-; CHECK-NEXT: ret void
-
-
-; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.init.1() #0
-; CHECK-NEXT: call void @foo
-; CHECK-NEXT: ret void
+; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.init() #0 {
+; CHECK-NEXT:   store volatile i32 1, ptr addrspace(1) null
+; CHECK-NEXT:   call void @foo()
+; CHECK-NEXT:   ret void
+; CHECK-NEXT: }
 
-; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.fini.2() #1
-; CHECK-NEXT: call void @bar
-; CHECK-NEXT: ret void
+; CHECK-LABEL: define amdgpu_kernel void @amdgcn.device.fini() #1 {
+; CHECK-NEXT:    store volatile i32 0, ptr addrspace(1) null
+; CHECK-NEXT:    call void @bar()
+; CHECK-NEXT:    ret void
+; CHECK-NEXT:  }
 
 ; CHECK-NOT: amdgcn.device.
 
 ; CHECK-VIS: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.init{{$}}
 ; CHECK-VIS: OBJECT GLOBAL DEFAULT {{.*}} amdgcn.device.init.kd{{$}}
 ; CHECK-VIS: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.fini{{$}}
 ; CHECK-VIS: OBJECT   GLOBAL DEFAULT {{.*}} amdgcn.device.fini.kd{{$}}
 
-; CHECK-VIS: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.init.1{{$}}
-; CHECK-VIS: OBJECT GLOBAL DEFAULT {{.*}} amdgcn.device.init.1.kd{{$}}
-; CHECK-VIS: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.fini.2{{$}}
-; CHECK-VIS: OBJECT   GLOBAL DEFAULT {{.*}} amdgcn.device.fini.2.kd{{$}}
-
-
 define internal void @foo() {
   ret void
 }
 
 define internal void @bar() {
   ret void
 }
 

llvm/test/CodeGen/AMDGPU/lower-ctor-dtor.ll

 ; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-ctor-dtor < %s | FileCheck %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-ctor-dtor < %s | FileCheck %s
 
 ; Make sure we get the same result if we run multiple times
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-ctor-dtor,amdgpu-lower-ctor-dtor < %s | FileCheck %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx700 -filetype=obj -o - < %s | llvm-readelf -s - 2>&1 | FileCheck %s -check-prefix=VISIBILITY
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx700 -filetype=obj -o - < %s | llvm-readelf -S - 2>&1 | FileCheck %s -check-prefix=SECTION
 
 @llvm.global_ctors = appending addrspace(1) global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1, ptr @foo, ptr null }]
 @llvm.global_dtors = appending addrspace(1) global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1, ptr @bar, ptr null }]
 
-; CHECK-NOT: @llvm.global_ctors
-; CHECK-NOT: @llvm.global_dtors
+; CHECK: @llvm.used = appending global [2 x ptr] [ptr @amdgcn.device.init, ptr @amdgcn.device.fini]
 
 ; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.init() #0
 ; CHECK-NEXT: call void @foo
 ; CHECK-NEXT: ret void
 
 ; CHECK-LABEL: amdgpu_kernel void @amdgcn.device.fini() #1
 ; CHECK-NEXT: call void @bar
 ; CHECK-NEXT: ret void
 
 ; CHECK-NOT: amdgcn.device.
 
 ; VISIBILITY: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.init
 ; VISIBILITY: OBJECT GLOBAL DEFAULT {{.*}} amdgcn.device.init.kd
 ; VISIBILITY: FUNC   GLOBAL PROTECTED {{.*}} amdgcn.device.fini
 ; VISIBILITY: OBJECT   GLOBAL DEFAULT {{.*}} amdgcn.device.fini.kd
+; SECTION: .init_array.1     INIT_ARRAY      {{.*}} {{.*}} 000008 00  WA  0   0  8
+; SECTION: .fini_array.1     FINI_ARRAY      {{.*}} {{.*}} 000008 00  WA  0   0  8
 
 define internal void @foo() {
   ret void
 }
 
 define internal void @bar() {
   ret void
 }
 
 ; CHECK: attributes #0 = { "device-init" }
 ; CHECK: attributes #1 = { "device-fini" }