[amdgpu][nfc] Allocate kernel-specific LDS struct deterministically

[JonChesterfield]

Summary

A kernel may have an associated struct for laying out LDS variables.
This patch puts that instance, if present, at a deterministic address by
allocating it at the same time as the module scope instance.

This is relatively likely to be where the instance was allocated anyway (~NFC)
but will allow later patches to calculate where a given field can be found,
which means a function which is only reachable from a single kernel will be
able to access a LDS variable with zero overhead. That will be particularly
helpful for applications that instantiate a function template containing LDS
variables once per kernel.

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

131

this is both a negligible optimisation and decouples the layout from whether dynamic lds alignment is specified, which is useful for calculating where the variable is going to be from a context that doesn't know whether dynamic lds is in play

[JonChesterfield]

Hoping this is uncontroversial - got more patches to follow up with if we can land this,

[rampitec]

Any tests?

[rampitec]

In D127052#3561174, @rampitec wrote:

Any tests?

It is not immediately obvious it is NFC, especially around alignment.

[JonChesterfield]

I think the layout this chooses is always one of the layouts that could have been picked by the iteration order before. Dynamic LDS alignment gets set when encountering the dynamic variable which could be after the other two were allocated.

I'll write some IR to asm cases that would previously fail non-deterministically but will now pass reliably.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

Need to be careful about multiple anonymous functions

[scchan]

In D127052#3560815, @JonChesterfield wrote:

Hoping this is uncontroversial - got more patches to follow up with if we can land this,

This patch essentially turns the module scope LDS layout struct into creating a function scope thing for every kernel using global LDS?

[JonChesterfield]

In D127052#3563622, @scchan wrote:

In D127052#3560815, @JonChesterfield wrote:

Hoping this is uncontroversial - got more patches to follow up with if we can land this,

This patch essentially turns the module scope LDS layout struct into creating a function scope thing for every kernel using global LDS?

No. A patch by Stas some months ago adapted the module scope LDS lowering to stash remaining kernel variables in a per-kernel struct.

All this patch does, if that per-kernel struct is present, is put it at a predictable address instead of leaving it to the whims of block iteration order.

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

How so? By the time they're functions in IR they have unique names as far as I know. If they don't, the current lowering is already broken for them as it assumes a struct can be uniquely associated with a function by deriving the variable name from the function name

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

They don't have to have a name at all. If you have kernels "@0" and "@1" they'll get the same thing here

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

That's exciting. Broken at present, AMDGPULowerModuleLDSPass creates the kernel-specific variable by passing Twine("llvm.amdgcn.kernel.") + F->getName() + ".lds" to StructType::Create() with a suffix ".t".

I'd be inclined to say the right fix for that is to hard error on them on the basis that anonymous kernels are difficult to invoke by name, but failing that we could assign them names in LowerModuleLDS.

Or we could stop using string equivalence to bind a function and a struct together and do something along the lines of metadata instead which should be more robust to things like renaming the kernel.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

You could do what the final emission does, and use the name out of Mangler::getNameWithPrefix

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

Can I postpone fixing that until after this patch, given it's already broken independent of this patch?

[JonChesterfield]

• rebase

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

91

Mangler::getNameWithPrefix is a local state thing - different instances of Mangler will assign different values. We could value::setName the anonymous function but that'll be a change visible outside of LDS and feels inherently wrong.

[JonChesterfield]

Moved the anonymous function problem to D134741, this patch neither creates nor addresses it.

[JonChesterfield]

• Wrote test case, discovered control flow can be simpler, updated implementation

[arsenm]

llvm/test/CodeGen/AMDGPU/lds-frame-extern.ll

3

Should use -mtriple=amdgcn-amd-amdhsa, the checks are using the old style mesa relocations for stack

4–6

Is there a real reason to test this with all the targets?

[JonChesterfield]

llvm/test/CodeGen/AMDGPU/lds-frame-extern.ll

3

Nice, thanks. Dropping to gfx10 only as well, makes the test easier to read.

[JonChesterfield]

• triple to hsa, drop to single target

llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp

bool AMDGPUCallLowering::lowerFormalArgumentsKernel(
   MachineFunction &MF = B.getMF();
   const GCNSubtarget *Subtarget = &MF.getSubtarget<GCNSubtarget>();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>();
   const SIRegisterInfo *TRI = Subtarget->getRegisterInfo();
   const SITargetLowering &TLI = *getTLI<SITargetLowering>();
   const DataLayout &DL = F.getParent()->getDataLayout();
 
-  Info->allocateModuleLDSGlobal(F);
+  Info->allocateKnownAddressLDSGlobal(F);
 
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(F.getCallingConv(), F.isVarArg(), MF, ArgLocs, F.getContext());
 
   allocateHSAUserSGPRs(CCInfo, B, MF, *TRI, *Info);
 
   unsigned i = 0;
   const Align KernArgBaseAlign(16);
bool AMDGPUCallLowering::lowerFormalArguments(
   MachineFunction &MF = B.getMF();
   MachineBasicBlock &MBB = B.getMBB();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>();
   const GCNSubtarget &Subtarget = MF.getSubtarget<GCNSubtarget>();
   const SIRegisterInfo *TRI = Subtarget.getRegisterInfo();
   const DataLayout &DL = F.getParent()->getDataLayout();
 
-  Info->allocateModuleLDSGlobal(F);
+  Info->allocateKnownAddressLDSGlobal(F);
 
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CC, F.isVarArg(), MF, ArgLocs, F.getContext());
 
   if (Info->hasImplicitBufferPtr()) {
     Register ImplicitBufferPtrReg = Info->addImplicitBufferPtr(*TRI);
     MF.addLiveIn(ImplicitBufferPtrReg, &AMDGPU::SGPR_64RegClass);
     CCInfo.AllocateReg(ImplicitBufferPtrReg);

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h

public:
   bool isMemoryBound() const {
     return MemoryBound;
   }
 
   bool needsWaveLimiter() const {
     return WaveLimiter;
   }
 
-  unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV);
-  void allocateModuleLDSGlobal(const Function &F);
+  unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV) {
+    return allocateLDSGlobal(DL, GV, DynLDSAlign);
+  }
+  unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV,
+                             Align Trailing);
+
+  void allocateKnownAddressLDSGlobal(const Function &F);
+
+  // A kernel function may have an associated LDS allocation, and a kernel-scope
+  // LDS allocation must have an associated kernel function
+  static const GlobalVariable *
+  getKernelLDSGlobalFromFunction(const Function &F);
 
   static Optional<uint32_t> getLDSKernelIdMetadata(const Function &F);
 
   Align getDynLDSAlign() const { return DynLDSAlign; }
 
   void setDynLDSAlign(const DataLayout &DL, const GlobalVariable &GV);
 };
 
 }
 #endif

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

AMDGPUMachineFunction::AMDGPUMachineFunction(const MachineFunction &MF)
   StaticGDSSize = GDSSize;
 
   CallingConv::ID CC = F.getCallingConv();
   if (CC == CallingConv::AMDGPU_KERNEL || CC == CallingConv::SPIR_KERNEL)
     ExplicitKernArgSize = ST.getExplicitKernArgSize(F, MaxKernArgAlign);
 }
 
 unsigned AMDGPUMachineFunction::allocateLDSGlobal(const DataLayout &DL,
-                                                  const GlobalVariable &GV) {
+                                                  const GlobalVariable &GV,
+                                                  Align Trailing) {
   auto Entry = LocalMemoryObjects.insert(std::make_pair(&GV, 0));
   if (!Entry.second)
     return Entry.first->second;
 
   Align Alignment =
       DL.getValueOrABITypeAlignment(GV.getAlign(), GV.getValueType());
 
   unsigned Offset;
   if (GV.getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
     /// TODO: We should sort these to minimize wasted space due to alignment
     /// padding. Currently the padding is decided by the first encountered use
     /// during lowering.
     Offset = StaticLDSSize = alignTo(StaticLDSSize, Alignment);
 
     StaticLDSSize += DL.getTypeAllocSize(GV.getValueType());
 
-    // Update the LDS size considering the padding to align the dynamic shared
-    // memory.
-    LDSSize = alignTo(StaticLDSSize, DynLDSAlign);
+    // Align LDS size to trailing, e.g. for aligning dynamic shared memory
+    LDSSize = alignTo(StaticLDSSize, Trailing);
   } else {
     assert(GV.getAddressSpace() == AMDGPUAS::REGION_ADDRESS &&
            "expected region address space");
 
     Offset = StaticGDSSize = alignTo(StaticGDSSize, Alignment);
     StaticGDSSize += DL.getTypeAllocSize(GV.getValueType());
 
     // FIXME: Apply alignment of dynamic GDS
     GDSSize = StaticGDSSize;
   }
 
   Entry.first->second = Offset;
   return Offset;
 }
 
+const GlobalVariable *
+AMDGPUMachineFunction::getKernelLDSGlobalFromFunction(const Function &F) {
+  const Module *M = F.getParent();
+  std::string KernelLDSName = "llvm.amdgcn.kernel.";
+  KernelLDSName += F.getName();
+  KernelLDSName += ".lds";
+  return M->getNamedGlobal(KernelLDSName);
+}
+
 // This kernel calls no functions that require the module lds struct
 static bool canElideModuleLDS(const Function &F) {
   return F.hasFnAttribute("amdgpu-elide-module-lds");
 }
 
-void AMDGPUMachineFunction::allocateModuleLDSGlobal(const Function &F) {
+void AMDGPUMachineFunction::allocateKnownAddressLDSGlobal(const Function &F) {
   const Module *M = F.getParent();
+
+  // This function is called before allocating any other LDS so that it can
+  // reliably put values at known addresses. Consequently, dynamic LDS, if
+  // present, will not yet have been allocated
+
+  assert(getDynLDSAlign() == Align() && "dynamic LDS not yet allocated");
+
   if (isModuleEntryFunction()) {
+
+    // Pointer values start from zero, memory allocated per-kernel-launch
+    // Variables can be grouped into a module level struct and a struct per
+    // kernel function by AMDGPULowerModuleLDSPass. If that is done, they
+    // are allocated at statically computable addresses here.
+    //
+    // Address 0
+    // {
+    //   llvm.amdgcn.module.lds
+    // }
+    // alignment padding
+    // {
+    //   llvm.amdgcn.kernel.some-name.lds
+    // }
+    // other variables, e.g. dynamic lds, allocated after this call
+
     const GlobalVariable *GV = M->getNamedGlobal("llvm.amdgcn.module.lds");
+    const GlobalVariable *KV = getKernelLDSGlobalFromFunction(F);
+
     if (GV && !canElideModuleLDS(F)) {
-      unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *GV);
+      unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *GV, Align());
       (void)Offset;
       assert(Offset == 0 &&
              "Module LDS expected to be allocated before other LDS");
     }
+
+    if (KV) {
+      // The per-kernel offset is deterministic because it is allocated
+      // before any other non-module LDS variables.
+      unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *KV, Align());
+      (void)Offset;
+    }
   }
 }
 
 Optional<uint32_t>
 AMDGPUMachineFunction::getLDSKernelIdMetadata(const Function &F) {
   auto MD = F.getMetadata("llvm.amdgcn.lds.kernel.id");
   if (MD && MD->getNumOperands() == 1) {
     ConstantInt *KnownSize = mdconst::extract<ConstantInt>(MD->getOperand(0));

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

SDValue SITargetLowering::LowerFormalArguments(
 
   if (Subtarget->isAmdHsaOS() && AMDGPU::isGraphics(CallConv)) {
     DiagnosticInfoUnsupported NoGraphicsHSA(
         Fn, "unsupported non-compute shaders with HSA", DL.getDebugLoc());
     DAG.getContext()->diagnose(NoGraphicsHSA);
     return DAG.getEntryNode();
   }
 
-  Info->allocateModuleLDSGlobal(Fn);
+  Info->allocateKnownAddressLDSGlobal(Fn);
 
   SmallVector<ISD::InputArg, 16> Splits;
   SmallVector<CCValAssign, 16> ArgLocs;
   BitVector Skipped(Ins.size());
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
 
   bool IsGraphics = AMDGPU::isGraphics(CallConv);

llvm/test/CodeGen/AMDGPU/lds-frame-extern.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck %s
+
+; LDS is allocated per-kernel. Module scope variables are gathered into a struct which is
+; allocated at address zero, if used by the kernel. Kernel scope variables are gathered into
+; a per-kernel struct and allocated immediately after the module scope.
+; This test checks that the module and kernel scope variables are allocated in deterministic
+; order without spurious alignment padding between the two
+
+; External LDS is checked because it influences LDS padding in general and because it will
+; not be moved into either module or kernel struct
+
+@module_variable = addrspace(3) global i16 undef
+
+; Variables are allocated into module scope block when used by a non-kernel function
+define void @use_module() #0 {
+; CHECK-LABEL: use_module:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; CHECK-NEXT:    s_waitcnt_vscnt null, 0x0
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    ds_write_b16 v0, v0
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_setpc_b64 s[30:31]
+  store i16 0, i16 addrspace(3)* @module_variable
+  ret void
+}
+
+; Variables only used by kernels are specialised and allocated per-kernel
+@kernel_normal = addrspace(3) global i16 undef
+@kernel_overalign = addrspace(3) global i16 undef, align 4
+
+; External LDS shall not introduce padding between module and kernel scope variables
+@extern_normal = external addrspace(3) global [0 x float]
+@extern_overalign = external addrspace(3) global [0 x float], align 8
+
+; 2^3 cases encoded into function names
+
+define amdgpu_kernel void @module_0_kernel_normal_extern_normal(i32 %idx) #1 {
+; CHECK-LABEL: module_0_kernel_normal_extern_normal:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_load_dword s0, s[4:5], 0x0
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 2
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_lshl_b32 s0, s0, 2
+; CHECK-NEXT:    s_add_i32 s0, s0, 4
+; CHECK-NEXT:    v_mov_b32_e32 v2, s0
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b32 v2, v0
+; CHECK-NEXT:    s_endpgm
+  store i16 2, i16 addrspace(3)* @kernel_normal
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_normal, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_1_kernel_normal_extern_normal(i32 %idx) {
+; CHECK-LABEL: module_1_kernel_normal_extern_normal:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_add_u32 s8, s8, s11
+; CHECK-NEXT:    s_mov_b32 s32, 0
+; CHECK-NEXT:    s_addc_u32 s9, s9, 0
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s8
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s9
+; CHECK-NEXT:    s_add_u32 s0, s0, s11
+; CHECK-NEXT:    s_addc_u32 s1, s1, 0
+; CHECK-NEXT:    s_getpc_b64 s[8:9]
+; CHECK-NEXT:    s_add_u32 s8, s8, use_module@gotpcrel32@lo+4
+; CHECK-NEXT:    s_addc_u32 s9, s9, use_module@gotpcrel32@hi+12
+; CHECK-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x0
+; CHECK-NEXT:    s_load_dword s12, s[6:7], 0x0
+; CHECK-NEXT:    s_mov_b64 s[6:7], s[4:5]
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_swappc_b64 s[30:31], s[10:11]
+; CHECK-NEXT:    s_lshl_b32 s4, s12, 2
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 1
+; CHECK-NEXT:    s_add_i32 s4, s4, 4
+; CHECK-NEXT:    v_mov_b32_e32 v2, 2
+; CHECK-NEXT:    v_mov_b32_e32 v3, s4
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b16 v0, v2 offset:2
+; CHECK-NEXT:    ds_write_b32 v3, v0
+; CHECK-NEXT:    s_endpgm
+  call void @use_module()
+  store i16 1, i16 addrspace(3)* @module_variable
+
+  store i16 2, i16 addrspace(3)* @kernel_normal
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_normal, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_0_kernel_overalign_extern_normal(i32 %idx) #1 {
+; CHECK-LABEL: module_0_kernel_overalign_extern_normal:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_load_dword s0, s[4:5], 0x0
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 2
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_lshl_b32 s0, s0, 2
+; CHECK-NEXT:    s_add_i32 s0, s0, 4
+; CHECK-NEXT:    v_mov_b32_e32 v2, s0
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b32 v2, v0
+; CHECK-NEXT:    s_endpgm
+  store i16 2, i16 addrspace(3)* @kernel_overalign
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_normal, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_1_kernel_overalign_extern_normal(i32 %idx) {
+; CHECK-LABEL: module_1_kernel_overalign_extern_normal:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_add_u32 s8, s8, s11
+; CHECK-NEXT:    s_mov_b32 s32, 0
+; CHECK-NEXT:    s_addc_u32 s9, s9, 0
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s8
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s9
+; CHECK-NEXT:    s_add_u32 s0, s0, s11
+; CHECK-NEXT:    s_addc_u32 s1, s1, 0
+; CHECK-NEXT:    s_getpc_b64 s[8:9]
+; CHECK-NEXT:    s_add_u32 s8, s8, use_module@gotpcrel32@lo+4
+; CHECK-NEXT:    s_addc_u32 s9, s9, use_module@gotpcrel32@hi+12
+; CHECK-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x0
+; CHECK-NEXT:    s_load_dword s12, s[6:7], 0x0
+; CHECK-NEXT:    s_mov_b64 s[6:7], s[4:5]
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_swappc_b64 s[30:31], s[10:11]
+; CHECK-NEXT:    s_lshl_b32 s4, s12, 2
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 1
+; CHECK-NEXT:    s_add_i32 s4, s4, 8
+; CHECK-NEXT:    v_mov_b32_e32 v2, 2
+; CHECK-NEXT:    v_mov_b32_e32 v3, s4
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b16 v0, v2 offset:4
+; CHECK-NEXT:    ds_write_b32 v3, v0
+; CHECK-NEXT:    s_endpgm
+  call void @use_module()
+  store i16 1, i16 addrspace(3)* @module_variable
+
+  store i16 2, i16 addrspace(3)* @kernel_overalign
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_normal, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_0_kernel_normal_extern_overalign(i32 %idx) #1 {
+; CHECK-LABEL: module_0_kernel_normal_extern_overalign:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_load_dword s0, s[4:5], 0x0
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 2
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_lshl_b32 s0, s0, 2
+; CHECK-NEXT:    s_add_i32 s0, s0, 8
+; CHECK-NEXT:    v_mov_b32_e32 v2, s0
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b32 v2, v0
+; CHECK-NEXT:    s_endpgm
+  store i16 2, i16 addrspace(3)* @kernel_normal
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_overalign, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_1_kernel_normal_extern_overalign(i32 %idx) {
+; CHECK-LABEL: module_1_kernel_normal_extern_overalign:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_add_u32 s8, s8, s11
+; CHECK-NEXT:    s_mov_b32 s32, 0
+; CHECK-NEXT:    s_addc_u32 s9, s9, 0
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s8
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s9
+; CHECK-NEXT:    s_add_u32 s0, s0, s11
+; CHECK-NEXT:    s_addc_u32 s1, s1, 0
+; CHECK-NEXT:    s_getpc_b64 s[8:9]
+; CHECK-NEXT:    s_add_u32 s8, s8, use_module@gotpcrel32@lo+4
+; CHECK-NEXT:    s_addc_u32 s9, s9, use_module@gotpcrel32@hi+12
+; CHECK-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x0
+; CHECK-NEXT:    s_load_dword s12, s[6:7], 0x0
+; CHECK-NEXT:    s_mov_b64 s[6:7], s[4:5]
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_swappc_b64 s[30:31], s[10:11]
+; CHECK-NEXT:    s_lshl_b32 s4, s12, 2
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 1
+; CHECK-NEXT:    s_add_i32 s4, s4, 8
+; CHECK-NEXT:    v_mov_b32_e32 v2, 2
+; CHECK-NEXT:    v_mov_b32_e32 v3, s4
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b16 v0, v2 offset:2
+; CHECK-NEXT:    ds_write_b32 v3, v0
+; CHECK-NEXT:    s_endpgm
+  call void @use_module()
+  store i16 1, i16 addrspace(3)* @module_variable
+
+  store i16 2, i16 addrspace(3)* @kernel_normal
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_overalign, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_0_kernel_overalign_extern_overalign(i32 %idx) #1 {
+; CHECK-LABEL: module_0_kernel_overalign_extern_overalign:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_load_dword s0, s[4:5], 0x0
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 2
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_lshl_b32 s0, s0, 2
+; CHECK-NEXT:    s_add_i32 s0, s0, 8
+; CHECK-NEXT:    v_mov_b32_e32 v2, s0
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b32 v2, v0
+; CHECK-NEXT:    s_endpgm
+  store i16 2, i16 addrspace(3)* @kernel_overalign
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_overalign, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+define amdgpu_kernel void @module_1_kernel_overalign_extern_overalign(i32 %idx) {
+; CHECK-LABEL: module_1_kernel_overalign_extern_overalign:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_add_u32 s8, s8, s11
+; CHECK-NEXT:    s_mov_b32 s32, 0
+; CHECK-NEXT:    s_addc_u32 s9, s9, 0
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s8
+; CHECK-NEXT:    s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s9
+; CHECK-NEXT:    s_add_u32 s0, s0, s11
+; CHECK-NEXT:    s_addc_u32 s1, s1, 0
+; CHECK-NEXT:    s_getpc_b64 s[8:9]
+; CHECK-NEXT:    s_add_u32 s8, s8, use_module@gotpcrel32@lo+4
+; CHECK-NEXT:    s_addc_u32 s9, s9, use_module@gotpcrel32@hi+12
+; CHECK-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x0
+; CHECK-NEXT:    s_load_dword s12, s[6:7], 0x0
+; CHECK-NEXT:    s_mov_b64 s[6:7], s[4:5]
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+; CHECK-NEXT:    s_swappc_b64 s[30:31], s[10:11]
+; CHECK-NEXT:    s_lshl_b32 s4, s12, 2
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    v_mov_b32_e32 v1, 1
+; CHECK-NEXT:    s_add_i32 s4, s4, 8
+; CHECK-NEXT:    v_mov_b32_e32 v2, 2
+; CHECK-NEXT:    v_mov_b32_e32 v3, s4
+; CHECK-NEXT:    ds_write_b16 v0, v1
+; CHECK-NEXT:    ds_write_b16 v0, v2 offset:4
+; CHECK-NEXT:    ds_write_b32 v3, v0
+; CHECK-NEXT:    s_endpgm
+  call void @use_module()
+  store i16 1, i16 addrspace(3)* @module_variable
+
+  store i16 2, i16 addrspace(3)* @kernel_overalign
+
+  %arrayidx1 = getelementptr inbounds [0 x float], [0 x float] addrspace(3)* @extern_overalign, i32 0, i32 %idx
+  store float 0.0, float addrspace(3)* %arrayidx1
+  ret void
+}
+
+attributes #0 = { noinline }
+attributes #1 = { "amdgpu-elide-module-lds" }