[amdgpu][lds] Remove recalculation of LDS frame from backend

[JonChesterfield]

Summary

Do the LDS frame calculation once, in the IR pass, instead of repeating the work in the backend.

Prior to this patch:
The IR lowering pass sets up a per-kernel LDS frame and annotates the variables with absolute_symbol
metadata so that the assembler can build lookup tables out of it. There is a fragile association between
kernel functions and named structs which is used to recompute the frame layout in the backend, with
fatal_errors catching inconsistencies in the second calculation.

After this patch:
The IR lowering pass additionally sets a frame size attribute on kernels. The backend uses the same
absolute_symbol metadata that the assembler uses to place objects within that frame size.

Deleted the now dead allocation code from the backend. Left for a later cleanup:

• enabling lowering for anonymous functions
• removing the elide-module-lds attribute (test churn, it's not used by llc any more)
• adjusting the dynamic alignment check to not use symbol names

[JonChesterfield]

• reformat

[JonChesterfield]

This is D155125 with the recalculation deleted, instead of set up to catch regressions under CI.

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

94

Deleting this association gives lds lowering more data layout freedom and drops the limitation on anonymous functions

103

this association will be removed in a later patch

116

this is the recomputation of the LDS frame being deleted

[JonChesterfield]

• docs

[arsenm]

llvm/docs/AMDGPUUsage.rst

1095

Note frontends shouldn't be setting it?

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

1109

Typo lnger

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

87

Probably should go through DiagnosticInfo. I don't know why we don't have sensible predefined types for this sort of thing. DK_Lowering looks promising but is buried in one specific user

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

87

Maybe, it's not user facing. Currently the only way to reach this is via a bug in LowerModuleLDS. Could be an assert instead, or just delete the checks.

Equivalent to the fatal errors in the allocateKnownAddressLDSGlobal which this patch deletes.

[JonChesterfield]

• review comments

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

48

Not sure exactly what two variables or "profitable" this is talking about

76

If we did the same thing, could probably fix the bug with amdgpu-gds-size not interacting well with globals

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

48

The LDSSize and StaticLDSSize immediately below the comment. I'm pretty sure we don't need both of them. The distinction is meant to elide some padding in some edge cases, but I don't think that edge case exists any more. I'm having some trouble working out exactly how graphics use LDS.

76

Seems likely that gds lowering would work well with absolute addresses, yes.

llvm/docs/AMDGPUUsage.rst

.. table:: AMDGPU LLVM IR Attributes
      "amdgpu-no-default-queue"               Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
                                              kernel argument that holds the default queue pointer. If this
                                              attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
 
      "amdgpu-no-completion-action"           Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
                                              kernel argument that holds the completion action pointer. If this
                                              attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
 
+     "amdgpu-lds-size"                       The number of bytes that will be allocated in the Local Data Store at
+                                             address zero. Variables are allocated within this frame using absolute
+                                             symbol metadata, primarily by the AMDGPULowerModuleLDS pass. Internal
+                                             detail of how LDS variables are lowered, language front ends should not
+                                             set this.
+
      ======================================= ==========================================================
 
 Calling Conventions
 -------------------
 
 The AMDGPU backend supports the following calling conventions:
 
   .. table:: AMDGPU Calling Conventions

llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp

bool AMDGPUCallLowering::lowerFormalArgumentsKernel(
   MachineFunction &MF = B.getMF();
   const GCNSubtarget *Subtarget = &MF.getSubtarget<GCNSubtarget>();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>();
   const SIRegisterInfo *TRI = Subtarget->getRegisterInfo();
   const SITargetLowering &TLI = *getTLI<SITargetLowering>();
   const DataLayout &DL = F.getParent()->getDataLayout();
 
-  Info->allocateKnownAddressLDSGlobal(F);
-
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(F.getCallingConv(), F.isVarArg(), MF, ArgLocs, F.getContext());
 
   allocateHSAUserSGPRs(CCInfo, B, MF, *TRI, *Info);
 
   unsigned i = 0;
   const Align KernArgBaseAlign(16);
   const unsigned BaseOffset = Subtarget->getExplicitKernelArgOffset();
bool AMDGPUCallLowering::lowerFormalArguments(
   MachineFunction &MF = B.getMF();
   MachineBasicBlock &MBB = B.getMBB();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SIMachineFunctionInfo *Info = MF.getInfo<SIMachineFunctionInfo>();
   const GCNSubtarget &Subtarget = MF.getSubtarget<GCNSubtarget>();
   const SIRegisterInfo *TRI = Subtarget.getRegisterInfo();
   const DataLayout &DL = F.getParent()->getDataLayout();
 
-  Info->allocateKnownAddressLDSGlobal(F);
-
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CC, F.isVarArg(), MF, ArgLocs, F.getContext());
 
   if (Info->hasImplicitBufferPtr()) {
     Register ImplicitBufferPtrReg = Info->addImplicitBufferPtr(*TRI);
     MF.addLiveIn(ImplicitBufferPtrReg, &AMDGPU::SGPR_64RegClass);
     CCInfo.AllocateReg(ImplicitBufferPtrReg);
   }

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

if (!KernelsThatIndirectlyAllocateDynamicLDS.empty()) {
 
           replaceUseWithTableLookup(M, Builder, table, GV, U, nullptr);
         }
       }
     }
     return KernelToCreatedDynamicLDS;
   }
 
+  // This attribute is no longer used by the backend. TODO: Delete it in favour
+  // of pass-local state and update the tests to remove the string.
   static bool canElideModuleLDS(const Function &F) {
     return F.hasFnAttribute("amdgpu-elide-module-lds");
   }
 
   static void markElideModuleLDS(Function &F) {
     F.addFnAttr("amdgpu-elide-module-lds");
   }
 
bool runOnModule(Module &M) override {
 
     DenseMap<Function *, GlobalVariable *> KernelToCreatedDynamicLDS =
         lowerDynamicLDSVariables(M, LDSUsesInfo,
                                  KernelsThatIndirectlyAllocateDynamicLDS,
                                  DynamicVariables, OrderedKernels);
 
     // All kernel frames have been allocated. Calculate and record the
     // addresses.
-
     {
       const DataLayout &DL = M.getDataLayout();
 
       for (Function &Func : M.functions()) {
         if (Func.isDeclaration() || !isKernelLDS(&Func))
           continue;
 
         // All three of these are optional. The first variable is allocated at
-        // zero. They are allocated by allocateKnownAddressLDSGlobal in the
-        // following order:
+        // zero. They are allocated by AMDGPUMachineFunction as one block.
+        // Layout:
         //{
         //  module.lds
         //  alignment padding
         //  kernel instance
         //  alignment padding
         //  dynamic lds variables
         //}
 
// addresses.
         if (AllocateModuleScopeStruct) {
           // Allocated at zero, recorded once on construction, not once per
           // kernel
           Offset += DL.getTypeAllocSize(MaybeModuleScopeStruct->getValueType());
         }
 
         if (AllocateKernelScopeStruct) {
           GlobalVariable *KernelStruct = Replacement->second.SGV;
-
           Offset = alignTo(Offset, AMDGPU::getAlign(DL, KernelStruct));
-
           recordLDSAbsoluteAddress(&M, KernelStruct, Offset);
-
           Offset += DL.getTypeAllocSize(KernelStruct->getValueType());
-
         }
 
+        // If there is dynamic allocation, the alignment needed is included in
+        // the static frame size. There may be no reference to the dynamic
+        // variable in the kernel itself, so without including it here, that
+        // alignment padding could be missed.
         if (AllocateDynamicVariable) {
           GlobalVariable *DynamicVariable = KernelToCreatedDynamicLDS[&Func];
-
           Offset = alignTo(Offset, AMDGPU::getAlign(DL, DynamicVariable));
-
           recordLDSAbsoluteAddress(&M, DynamicVariable, Offset);
         }
+
+        if (Offset != 0)
+          Func.addFnAttr("amdgpu-lds-size", std::to_string(Offset));
       }
     }
 
     for (auto &GV : make_early_inc_range(M.globals()))
       if (AMDGPU::isLDSVariableToLower(GV)) {
         // probably want to remove from used lists
         GV.removeDeadConstantUsers();
         if (GV.use_empty())

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h

public:
 
   unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV) {
     return allocateLDSGlobal(DL, GV, DynLDSAlign);
   }
 
   unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV,
                              Align Trailing);
 
-  void allocateKnownAddressLDSGlobal(const Function &F);
-
   static std::optional<uint32_t> getLDSKernelIdMetadata(const Function &F);
   static std::optional<uint32_t> getLDSAbsoluteAddress(const GlobalValue &GV);
 
   Align getDynLDSAlign() const { return DynLDSAlign; }
 
   void setDynLDSAlign(const Function &F, const GlobalVariable &GV);
 };
 
 }
 #endif

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

AMDGPUMachineFunction::AMDGPUMachineFunction(const Function &F,
   // global sizes?
   StringRef S = F.getFnAttribute("amdgpu-gds-size").getValueAsString();
   if (!S.empty())
     S.consumeInteger(0, GDSSize);
 
   // Assume the attribute allocates before any known GDS globals.
   StaticGDSSize = GDSSize;
 
+  // The two separate variables are only profitable when the LDS module lowering
+  // pass is disabled. If graphics does not use dynamic LDS, this is never
+  // profitable. Leaving cleanup for a later change.
+  LDSSize = F.getFnAttributeAsParsedInteger("amdgpu-lds-size", 0);
+  StaticLDSSize = LDSSize;
+
   CallingConv::ID CC = F.getCallingConv();
   if (CC == CallingConv::AMDGPU_KERNEL || CC == CallingConv::SPIR_KERNEL)
     ExplicitKernArgSize = ST.getExplicitKernArgSize(F, MaxKernArgAlign);
 
   // FIXME: Shouldn't be target specific
   Attribute NSZAttr = F.getFnAttribute("no-signed-zeros-fp-math");
   NoSignedZerosFPMath =
       NSZAttr.isStringAttribute() && NSZAttr.getValueAsString() == "true";
 }
 
 unsigned AMDGPUMachineFunction::allocateLDSGlobal(const DataLayout &DL,
                                                   const GlobalVariable &GV,
                                                   Align Trailing) {
   auto Entry = LocalMemoryObjects.insert(std::pair(&GV, 0));
   if (!Entry.second)
     return Entry.first->second;
 
   Align Alignment =
       DL.getValueOrABITypeAlignment(GV.getAlign(), GV.getValueType());
 
   unsigned Offset;
   if (GV.getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
+
+    std::optional<uint32_t> MaybeAbs = getLDSAbsoluteAddress(GV);
+    if (MaybeAbs) {
+      // Absolute address LDS variables that exist prior to the LDS lowering
+      // pass raise a fatal error in that pass. These failure modes are only
+      // reachable if that lowering pass is disabled or broken. If/when adding
+      // support for absolute addresses on user specified variables, the
+      // alignment check moves to the lowering pass and the frame calculation
+      // needs to take the user variables into consideration.
+
+      uint32_t ObjectStart = *MaybeAbs;
+
+      if (ObjectStart != alignTo(ObjectStart, Alignment)) {
+        report_fatal_error("Absolute address LDS variable inconsistent with "
+                           "variable alignment");
+      }
+
+      if (isModuleEntryFunction()) {
+        // If this is a module entry function, we can also sanity check against
+        // the static frame. Strictly it would be better to check against the
+        // attribute, i.e. that the variable is within the always-allocated
+        // section, and not within some other non-absolute-address object
+        // allocated here, but the extra error detection is minimal and we would
+        // have to pass the Function around or cache the attribute value.
+        uint32_t ObjectEnd =
+            ObjectStart + DL.getTypeAllocSize(GV.getValueType());
+        if (ObjectEnd > StaticLDSSize) {
+          report_fatal_error(
+              "Absolute address LDS variable outside of static frame");
+        }
+      }
+
+      Entry.first->second = ObjectStart;
+      return ObjectStart;
+    }
+
     /// TODO: We should sort these to minimize wasted space due to alignment
     /// padding. Currently the padding is decided by the first encountered use
     /// during lowering.
     Offset = StaticLDSSize = alignTo(StaticLDSSize, Alignment);
 
     StaticLDSSize += DL.getTypeAllocSize(GV.getValueType());
 
     // Align LDS size to trailing, e.g. for aligning dynamic shared memory
     LDSSize = alignTo(StaticLDSSize, Trailing);
   } else {
     assert(GV.getAddressSpace() == AMDGPUAS::REGION_ADDRESS &&
            "expected region address space");
 
     Offset = StaticGDSSize = alignTo(StaticGDSSize, Alignment);
     StaticGDSSize += DL.getTypeAllocSize(GV.getValueType());
 
     // FIXME: Apply alignment of dynamic GDS
     GDSSize = StaticGDSSize;
   }
 
   Entry.first->second = Offset;
   return Offset;
 }
 
-static constexpr StringLiteral ModuleLDSName = "llvm.amdgcn.module.lds";
-
-static const GlobalVariable *getKernelLDSGlobalFromFunction(const Function &F) {
-  const Module *M = F.getParent();
-  std::string KernelLDSName = "llvm.amdgcn.kernel.";
-  KernelLDSName += F.getName();
-  KernelLDSName += ".lds";
-  return M->getNamedGlobal(KernelLDSName);
-}
-
 static const GlobalVariable *
 getKernelDynLDSGlobalFromFunction(const Function &F) {
   const Module *M = F.getParent();
   std::string KernelDynLDSName = "llvm.amdgcn.";
   KernelDynLDSName += F.getName();
   KernelDynLDSName += ".dynlds";
   return M->getNamedGlobal(KernelDynLDSName);
 }
 
-// This kernel calls no functions that require the module lds struct
-static bool canElideModuleLDS(const Function &F) {
-  return F.hasFnAttribute("amdgpu-elide-module-lds");
-}
-
-void AMDGPUMachineFunction::allocateKnownAddressLDSGlobal(const Function &F) {
-  const Module *M = F.getParent();
-  // This function is called before allocating any other LDS so that it can
-  // reliably put values at known addresses. Consequently, dynamic LDS, if
-  // present, will not yet have been allocated
-
-  assert(getDynLDSAlign() == Align() && "dynamic LDS not yet allocated");
-
-  if (isModuleEntryFunction()) {
-
-    // Pointer values start from zero, memory allocated per-kernel-launch
-    // Variables can be grouped into a module level struct and a struct per
-    // kernel function by AMDGPULowerModuleLDSPass. If that is done, they
-    // are allocated at statically computable addresses here.
-    //
-    // Address 0
-    // {
-    //   llvm.amdgcn.module.lds
-    // }
-    // alignment padding
-    // {
-    //   llvm.amdgcn.kernel.some-name.lds
-    // }
-    // other variables, e.g. dynamic lds, allocated after this call
-
-    const GlobalVariable *GV = M->getNamedGlobal(ModuleLDSName);
-    const GlobalVariable *KV = getKernelLDSGlobalFromFunction(F);
-    const GlobalVariable *Dyn = getKernelDynLDSGlobalFromFunction(F);
-
-    if (GV && !canElideModuleLDS(F)) {
-      unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *GV, Align());
-      std::optional<uint32_t> Expect = getLDSAbsoluteAddress(*GV);
-      if (!Expect || (Offset != *Expect)) {
-        report_fatal_error("Inconsistent metadata on module LDS variable");
-      }
-    }
-
-    if (KV) {
-      // The per-kernel offset is deterministic because it is allocated
-      // before any other non-module LDS variables.
-      unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *KV, Align());
-      std::optional<uint32_t> Expect = getLDSAbsoluteAddress(*KV);
-      if (!Expect || (Offset != *Expect)) {
-        report_fatal_error("Inconsistent metadata on kernel LDS variable");
-      }
-    }
-
-    if (Dyn) {
-      // The dynamic LDS is deterministic because the per-kernel one has the
-      // maximum alignment of any reachable and all remaining LDS variables,
-      // if this is present, are themselves dynamic LDS and will be allocated
-      // at the same address.
-      setDynLDSAlign(F, *Dyn);
-      unsigned Offset = LDSSize;
-      std::optional<uint32_t> Expect = getLDSAbsoluteAddress(*Dyn);
-      if (!Expect || (Offset != *Expect)) {
-        report_fatal_error("Inconsistent metadata on dynamic LDS variable");
-      }
-    }
-  }
-}
-
 std::optional<uint32_t>
 AMDGPUMachineFunction::getLDSKernelIdMetadata(const Function &F) {
   // TODO: Would be more consistent with the abs symbols to use a range
   MDNode *MD = F.getMetadata("llvm.amdgcn.lds.kernel.id");
   if (MD && MD->getNumOperands() == 1) {
     if (ConstantInt *KnownSize =
             mdconst::extract<ConstantInt>(MD->getOperand(0))) {
       uint64_t ZExt = KnownSize->getZExtValue();

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

SDValue SITargetLowering::LowerFormalArguments(
 
   if (Subtarget->isAmdHsaOS() && AMDGPU::isGraphics(CallConv)) {
     DiagnosticInfoUnsupported NoGraphicsHSA(
         Fn, "unsupported non-compute shaders with HSA", DL.getDebugLoc());
     DAG.getContext()->diagnose(NoGraphicsHSA);
     return DAG.getEntryNode();
   }
 
-  Info->allocateKnownAddressLDSGlobal(Fn);
-
   SmallVector<ISD::InputArg, 16> Splits;
   SmallVector<CCValAssign, 16> ArgLocs;
   BitVector Skipped(Ins.size());
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
 
   bool IsGraphics = AMDGPU::isGraphics(CallConv);
   bool IsKernel = AMDGPU::isKernel(CallConv);

llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll

 ; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8, !absolute_symbol !0
 ; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr addrspacecast (ptr addrspace(3) @llvm.amdgcn.module.lds to ptr)], section "llvm.metadata"
 ; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16, !absolute_symbol !0
 ; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16, !absolute_symbol !0
 ; CHECK: @llvm.amdgcn.kernel.k2.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k2.lds.t undef, align 2, !absolute_symbol !0
 ; CHECK: @llvm.amdgcn.kernel.k3.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k3.lds.t undef, align 4, !absolute_symbol !0
 ;.
 define amdgpu_kernel void @k0() #0 {
-; CHECK-LABEL: @k0(
+; CHECK-LABEL: @k0() #0
 ; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3), align 2, !alias.scope !1, !noalias !4
 ; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2), align 4, !alias.scope !8, !noalias !9
 ; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1), align 16, !alias.scope !10, !noalias !11
 ; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 16, !alias.scope !12, !noalias !13
 ; CHECK-NEXT:    ret void
   store i8 1, ptr addrspace(3) @lds.size.1.align.1, align 1
 
   store i8 2, ptr addrspace(3) @lds.size.2.align.2, align 2
 
   store i8 4, ptr addrspace(3) @lds.size.4.align.4, align 4
 
   store i8 16, ptr addrspace(3) @lds.size.16.align.16, align 16
 
   ret void
 }
 
 define amdgpu_kernel void @k1() #0 {
-; CHECK-LABEL: @k1(
+; CHECK-LABEL: @k1() #1
 ; CHECK-NEXT: store i8 2, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 2), align 4, !alias.scope !14, !noalias !17
 ; CHECK-NEXT: store i8 4, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.k1.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, i32 0, i32 1), align 16, !alias.scope !20, !noalias !21
 ; CHECK-NEXT: store i8 16, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 16, !alias.scope !22, !noalias !23
 ; CHECK-NEXT:    ret void
 ;
   store i8 2, ptr addrspace(3) @lds.size.2.align.2, align 2
 
   store i8 4, ptr addrspace(3) @lds.size.4.align.4, align 4
 
   store i8 16, ptr addrspace(3) @lds.size.16.align.16, align 16
 
   ret void
 }
 
 define amdgpu_kernel void @k2() #0 {
-; CHECK-LABEL: @k2(
+; CHECK-LABEL: @k2() #2
 ; CHECK-NEXT:    store i8 2, ptr addrspace(3) @llvm.amdgcn.kernel.k2.lds, align 2
 ; CHECK-NEXT:    ret void
 ;
   store i8 2, ptr addrspace(3) @lds.size.2.align.2, align 2
 
   ret void
 }
 
 define amdgpu_kernel void @k3() #0 {
-; CHECK-LABEL: @k3(
+; CHECK-LABEL: @k3() #3
 ; CHECK-NEXT:    store i8 4, ptr addrspace(3) @llvm.amdgcn.kernel.k3.lds, align 4
 ; CHECK-NEXT:    ret void
 ;
   store i8 4, ptr addrspace(3) @lds.size.4.align.4, align 4
 
   ret void
 }
 
-
+; CHECK-LABEL: @calls_f0() #4
 define amdgpu_kernel void @calls_f0() {
   call void @f0()
   ret void
 }
 
 define void @f0() {
-; CHECK-LABEL: define void @f0(
+; CHECK-LABEL: define void @f0()
 ; CHECK-NEXT: store i8 1, ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.module.lds.t, ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 8, !noalias !24
 ; CHECK-NEXT: store i8 8, ptr addrspace(3) @llvm.amdgcn.module.lds, align 8, !noalias !24
 ; CHECK-NEXT: ret void
   store i8 1, ptr addrspace(3) @lds.size.1.align.1, align 1
 
   store i8 8, ptr addrspace(3) @lds.size.8.align.8, align 4
 
   ret void
 }
 
-attributes #0 = { "amdgpu-elide-module-lds" }
-; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }
+; CHECK: attributes #0 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="23" }
+; CHECK: attributes #1 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="22" }
+; CHECK: attributes #2 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="2" }
+; CHECK: attributes #3 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="4" }
+; CHECK: attributes #4 = { "amdgpu-lds-size"="9" }
 
 ; CHECK: !0 = !{i64 0, i64 1}

llvm/test/CodeGen/AMDGPU/lower-module-lds-all-indirect-accesses.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=hybrid < %s | FileCheck %s
 
 ;; Reduced from a larger test case. Checks that functions and kernels that use only dynamic lds
 ;; are lowered successfully. Previously they only worked if the kernel happened to also use static lds
 ;; variables. Artefact of implementing dynamic variables by adapting existing code for static.
 
 @A = external addrspace(3) global [8 x ptr]
 @B = external addrspace(3) global [0 x i32]
 
 define amdgpu_kernel void @kernel_0() {
-; CHECK-LABEL: define amdgpu_kernel void @kernel_0() !llvm.amdgcn.lds.kernel.id !1 {
+; CHECK-LABEL: define amdgpu_kernel void @kernel_0() #0 !llvm.amdgcn.lds.kernel.id !1 {
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.kernel_0.lds) ]
 ; CHECK-NEXT:    call void @call_store_A()
 ; CHECK-NEXT:    ret void
 ;
   call void @call_store_A()
   ret void
 }
 
 define amdgpu_kernel void @kernel_1() {
 ; CHECK-LABEL: define amdgpu_kernel void @kernel_1() !llvm.amdgcn.lds.kernel.id !2 {
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel_1.dynlds) ]
 ; CHECK-NEXT:    [[PTR:%.*]] = call ptr @get_B_ptr()
 ; CHECK-NEXT:    ret void
 ;
   %ptr = call ptr @get_B_ptr()
   ret void
 }
 
 define amdgpu_kernel void @kernel_2() {
-; CHECK-LABEL: define amdgpu_kernel void @kernel_2() !llvm.amdgcn.lds.kernel.id !3 {
+; CHECK-LABEL: define amdgpu_kernel void @kernel_2() #0 !llvm.amdgcn.lds.kernel.id !3 {
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.kernel_2.lds) ]
 ; CHECK-NEXT:    call void @store_A()
 ; CHECK-NEXT:    ret void
 ;
   call void @store_A()
   ret void
 }
 

 ; CHECK-NEXT:    [[B:%.*]] = getelementptr inbounds [4 x i32], ptr addrspace(4) @llvm.amdgcn.dynlds.offset.table, i32 0, i32 [[TMP1]]
 ; CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr addrspace(4) [[B]], align 4
 ; CHECK-NEXT:    [[B1:%.*]] = inttoptr i32 [[TMP2]] to ptr addrspace(3)
 ; CHECK-NEXT:    [[TMP3:%.*]] = addrspacecast ptr addrspace(3) [[B1]] to ptr
 ; CHECK-NEXT:    ret ptr [[TMP3]]
 ;
   ret ptr addrspacecast (ptr addrspace(3) @B to ptr)
 }
+
+; CHECK: attributes #0 = { "amdgpu-lds-size"="64" }

llvm/test/CodeGen/AMDGPU/lower-module-lds-constantexpr.ll

 ; CHECK:      store i32 %x, ptr %5, align 4
 ; CHECK:      ret void
 define void @set_func(i32 %x) local_unnamed_addr #1 {
 entry:
   store i32 %x, ptr inttoptr (i64 add (i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @b_both to ptr) to i64), i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @b_both to ptr) to i64)) to ptr), align 4
   ret void
 }
 
-; CHECK-LABEL: @timestwo() #0
+; CHECK-LABEL: @timestwo() #1
 ; CHECK-NOT: call void @llvm.donothing()
 
 ; CHECK:      %1 = addrspacecast ptr addrspace(3) @llvm.amdgcn.kernel.timestwo.lds to ptr
 ; CHECK:      %2 = ptrtoint ptr %1 to i64
 ; CHECK:      %3 = addrspacecast ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 1) to ptr
 ; CHECK:      %4 = ptrtoint ptr %3 to i64
 ; CHECK:      %5 = add i64 %2, %4
 ; CHECK:      %6 = inttoptr i64 %5 to ptr
 ; CHECK:      %ld = load i32, ptr %6, align 4
 ; CHECK:      %mul = mul i32 %ld, 2
 ; CHECK:      %7 = addrspacecast ptr addrspace(3) getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, ptr addrspace(3) @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 1) to ptr
 ; CHECK:      %8 = ptrtoint ptr %7 to i64
 ; CHECK:      %9 = addrspacecast ptr addrspace(3) @llvm.amdgcn.kernel.timestwo.lds to ptr
 ; CHECK:      %10 = ptrtoint ptr %9 to i64
 ; CHECK:      %11 = add i64 %8, %10
 ; CHECK:      %12 = inttoptr i64 %11 to ptr
 ; CHECK:      store i32 %mul, ptr %12, align 4
 ; CHECK:      ret void
-define amdgpu_kernel void @timestwo() {
+define amdgpu_kernel void @timestwo() #1 {
   %ld = load i32, ptr inttoptr (i64 add (i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @b_both to ptr) to i64), i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @kern to ptr) to i64)) to ptr), align 4
   %mul = mul i32 %ld, 2
   store i32 %mul, ptr inttoptr (i64 add (i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @kern to ptr) to i64), i64 ptrtoint (ptr addrspacecast (ptr addrspace(3) @b_both to ptr) to i64)) to ptr), align 4
   ret void
 }
 
-; CHECK-LABEL: @through_functions()
+; CHECK-LABEL: @through_functions() #2
 define amdgpu_kernel void @through_functions() {
   %ld = call i32 @get_func()
   %mul = mul i32 %ld, 4
   call void @set_func(i32 %mul)
   ret void
 }
 
 attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }
+; CHECK: attributes #1 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="8" }
+; CHECK: attributes #2 = { "amdgpu-lds-size"="8" }

llvm/test/CodeGen/AMDGPU/lower-module-lds-indirect-extern-uses-max-reachable-alignment.ll

 ; CHECK-NEXT:    call void @use_shared2()
 ; CHECK-NEXT:    ret void
 ;
   call void @use_shared2()
   ret void
 }
 
 define amdgpu_kernel void @expect_align4() {
-; CHECK-LABEL: @expect_align4() !llvm.amdgcn.lds.kernel.id !4 {
+; CHECK-LABEL: @expect_align4() #2 !llvm.amdgcn.lds.kernel.id !4 {
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.expect_align4.dynlds) ]
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
 ; CHECK-NEXT:    call void @use_shared4()
 ; CHECK-NEXT:    ret void
 ;
   call void @use_shared4()
   ret void
 }
;
   %arrayidx = getelementptr inbounds [0 x i64], ptr addrspace(3) @dynamic_shared8, i32 0, i32 9
   store i64 3, ptr addrspace(3) %arrayidx
   call void @use_shared8()
   ret void
 }
 
 ; Note: use_shared4 uses module.lds so this will allocate at offset 4
 define amdgpu_kernel void @expect_max_of_2_and_4() {
-; CHECK-LABEL: @expect_max_of_2_and_4() !llvm.amdgcn.lds.kernel.id !6 {
+; CHECK-LABEL: @expect_max_of_2_and_4() #2 !llvm.amdgcn.lds.kernel.id !6 {
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.expect_max_of_2_and_4.dynlds) ]
 ; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
 ; CHECK-NEXT:    call void @use_shared2()
 ; CHECK-NEXT:    call void @use_shared4()
 ; CHECK-NEXT:    ret void
 ;
   call void @use_shared2()
   call void @use_shared4()
   ret void
 }
 
 
 attributes #0 = { noinline }
 
 ; Function Attrs: nocallback nofree nosync nounwind willreturn memory(none)
-; CHECK: declare void @llvm.donothing() #2
+; CHECK: declare void @llvm.donothing() #3
 
 ; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none)
-; CHECK: declare i32 @llvm.amdgcn.lds.kernel.id() #3
+; CHECK: declare i32 @llvm.amdgcn.lds.kernel.id() #4
 
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #1 = { noinline }
-; CHECK: attributes #2 = { nocallback nofree nosync nounwind willreturn memory(none) }
-; CHECK: attributes #3 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #2 = { "amdgpu-lds-size"="4" }
+; CHECK: attributes #3 = { nocallback nofree nosync nounwind willreturn memory(none) }
+; CHECK: attributes #4 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
 
 ; CHECK: !0 = !{i64 0, i64 1}
 ; CHECK: !1 = !{i64 4, i64 5}
 ; CHECK: !2 = !{i32 0}
 ; CHECK: !3 = !{i32 1}
 ; CHECK: !4 = !{i32 2}
 ; CHECK: !5 = !{i32 3}
 ; CHECK: !6 = !{i32 4}

llvm/test/CodeGen/AMDGPU/lower-module-lds-via-hybrid.ll

 
 ; OPT: declare i32 @llvm.amdgcn.lds.kernel.id()
 
 !0 = !{i32 0}
 !1 = !{i32 2}
 !2 = !{i32 1}
 
 
-; OPT: attributes #0 = { "amdgpu-elide-module-lds" }
-; OPT: attributes #1 = { nocallback nofree nosync nounwind willreturn memory(none) }
-; OPT: attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+; OPT: attributes #0 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="8" }
+; OPT: attributes #1 = { "amdgpu-lds-size"="8" }
+; OPT: attributes #2 = { "amdgpu-elide-module-lds" "amdgpu-lds-size"="12" }
+; OPT: attributes #3 = { "amdgpu-lds-size"="20" }
+; OPT: attributes #4 = { nocallback nofree nosync nounwind willreturn memory(none) }
+; OPT: attributes #5 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
 
 ; OPT: !0 = !{i64 0, i64 1}
 ; OPT: !1 = !{i64 4, i64 5}
 ; OPT: !2 = !{i64 8, i64 9}
 ; OPT: !3 = !{i32 1}
 ; OPT: !4 = !{!5}
 ; OPT: !5 = distinct !{!5, !6}
 ; OPT: !6 = distinct !{!6}

llvm/test/CodeGen/AMDGPU/lower-module-lds-via-table.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   %ld = load i8, ptr addrspace(3) @v3
   %mul = mul i8 %ld, 5
   store i8 %mul, ptr  addrspace(3) @v3
   ret void
 }
 
 ; Doesn't access any via a function, won't be in the lookup table
 define amdgpu_kernel void @kernel_no_table() {
-; OPT-LABEL: @kernel_no_table() {
+; OPT-LABEL: @kernel_no_table() #0 {
 ; OPT-NEXT:    [[LD:%.*]] = load i64, ptr addrspace(3) @llvm.amdgcn.kernel.kernel_no_table.lds, align 8
 ; OPT-NEXT:    [[MUL:%.*]] = mul i64 [[LD]], 8
 ; OPT-NEXT:    store i64 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.kernel_no_table.lds, align 8
 ; OPT-NEXT:    ret void
 ;
 ; GCN-LABEL: kernel_no_table:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    s_mov_b32 m0, -1
 ; GCN-NEXT:    ds_read_b64 v[0:1], v2
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    v_lshl_b64 v[0:1], v[0:1], 3
 ; GCN-NEXT:    ds_write_b64 v2, v[0:1]
 ; GCN-NEXT:    s_endpgm
   %ld = load i64, ptr addrspace(3) @v2
   %mul = mul i64 %ld, 8
   store i64 %mul, ptr  addrspace(3) @v2
   ret void
 }
 
 ; Access two variables, will allocate those two
 define amdgpu_kernel void @k01() {
-; OPT-LABEL: @k01() !llvm.amdgcn.lds.kernel.id !1 {
+; OPT-LABEL: @k01() #0 !llvm.amdgcn.lds.kernel.id !1 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k01.lds) ]
 ; OPT-NEXT:    call void @f0()
 ; OPT-NEXT:    call void @f1()
 ; OPT-NEXT:    ret void
 ;
 ; GCN-LABEL: k01:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_mov_b32 s32, 0

 ; GCN-NEXT:    s_endpgm
 ; GCN:         .amdhsa_group_segment_fixed_size 8
   call void @f0()
   call void @f1()
   ret void
 }
 
 define amdgpu_kernel void @k23() {
-; OPT-LABEL: @k23() !llvm.amdgcn.lds.kernel.id !7 {
+; OPT-LABEL: @k23() #1 !llvm.amdgcn.lds.kernel.id !7 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k23.lds) ]
 ; OPT-NEXT:    call void @f2()
 ; OPT-NEXT:    call void @f3()
 ; OPT-NEXT:    ret void
 ;
 ; GCN-LABEL: k23:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_mov_b32 s32, 0

 ; GCN:         .amdhsa_group_segment_fixed_size 16
   call void @f2()
   call void @f3()
   ret void
 }
 
 ; Access and allocate three variables
 define amdgpu_kernel void @k123() {
-; OPT-LABEL: @k123() !llvm.amdgcn.lds.kernel.id !13 {
+; OPT-LABEL: @k123() #2 !llvm.amdgcn.lds.kernel.id !13 {
 ; OPT-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds) ]
 ; OPT-NEXT:    call void @f1()
 ; OPT-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !20, !noalias !21
 ; OPT-NEXT:    [[MUL:%.*]] = mul i8 [[LD]], 8
 ; OPT-NEXT:    store i8 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_KERNEL_K123_LDS_T]], ptr addrspace(3) @llvm.amdgcn.kernel.k123.lds, i32 0, i32 1), align 2, !alias.scope !20, !noalias !21
 ; OPT-NEXT:    call void @f2()
 ; OPT-NEXT:    ret void
 ;
; GCN:         .amdhsa_group_segment_fixed_size 16
   store i8 %mul, ptr  addrspace(3) @v3
   call void @f2()
   ret void
 }
 
 
 ; OPT: declare i32 @llvm.amdgcn.lds.kernel.id()
 
+; OPT: attributes #0 = { "amdgpu-lds-size"="8" }
+; OPT: attributes #1 = { "amdgpu-lds-size"="12" }
+; OPT: attributes #2 = { "amdgpu-lds-size"="16" }
+
 !0 = !{i64 0, i64 1}
 !1 = !{i32 0}
 !2 = !{i32 2}
 !3 = !{i32 1}
 
 
 ; Table size length number-kernels * number-variables * sizeof(uint16_t)
 ; GCN:      .type llvm.amdgcn.lds.offset.table,@object

llvm/test/CodeGen/AMDGPU/lower-module-lds.ll

define void @func() {
   %val0 = load i32, ptr addrspace(3) @var1, align 4
   %val1 = add i32 %val0, 4
   store i32 %val1, ptr addrspace(3) @var1, align 4
   %unused0 = atomicrmw add ptr addrspace(3) @with_init, i64 1 monotonic
   ret void
 }
 
 ; This kernel calls a function that uses LDS so needs the block
-; CHECK-LABEL: @kern_call()
+; CHECK-LABEL: @kern_call() #0
 ; CHECK: call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
 ; CHECK: call void @func()
 ; CHECK: %dec = atomicrmw fsub ptr addrspace(3) @llvm.amdgcn.module.lds, float 2.000000e+00 monotonic, align 8
 define amdgpu_kernel void @kern_call() {
   call void @func()
   %dec = atomicrmw fsub ptr addrspace(3) @var0, float 2.0 monotonic
   ret void
 }
 
 ; This kernel does alloc the LDS block as it makes no calls
-; CHECK-LABEL: @kern_empty()
+; CHECK-LABEL: @kern_empty() #1
 ; CHECK-NOT: call void @llvm.donothing()
 define spir_kernel void @kern_empty() #0{
   ret void
 }
 
 ; Make sure we don't crash trying to insert code into a kernel
 ; declaration.
 declare amdgpu_kernel void @kernel_declaration()
 
 attributes #0 = { "amdgpu-elide-module-lds" }
-; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }
+
+; CHECK: attributes #0 = { "amdgpu-lds-size"="12" }
+; CHECK: attributes #1 = { "amdgpu-elide-module-lds" }