[amdgpu][lds] Fix missing markUsedByKernel calls and undef lookup table elements

[JonChesterfield]

Summary

More robust association between the kernels and lds struct.

Use poison instead of value() for lookup table elements introduced by dynamic lds lowering.

Extracted from D154946, new test from there verbatim. Segv fixed.

Fixes issues/63338

Fixes SWDEV-404491

[JonChesterfield]

• start tests

[JonChesterfield]

Not pleased with the churn in ds_write2.ll here, trying to address that in a separate patch. There are a few more tests that need updating still to go.

[JonChesterfield]

• [amdgpu][lds] Fix missing markUsedByKernel calls
• ds_write
• update most tests

[arsenm]

Don't see which one is the new asserting test?

llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll

590 ↗

(On Diff #539308)

Most of the test churn is from update_test_checks learning to check metadata

[JonChesterfield]

• test updates, more conservative annotation

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

973

The insert-on-not-found behavior of operator[] is evil, avoid using it?

[JonChesterfield]

Reasonably interesting. Introducing a llvm.donothing with an explicit use of a global changes codegen. The checks in noclobber-barrier fail for one of the atomic optimizer strategy options, looks like a regression to me.

@arsenm the segv test case in D154946 is half-fixed by this. That test crashes on use of an accidentally default initialised value, and if that is fixed, crashes at this point. Splitting the two failures out because the change is mostly noise in test cases.

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

1178

This ^ crashes if the ith kernel doesn't have an associated struct, which happens if the ith kernel was only given an index for a dynamic LDS variable. This patch fixes that and the above todo by annotating the struct close to where it was created.

[arsenm]

Does llvm.donothing not have enough memory attributes?

[JonChesterfield]

• avoid operator[]

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

973

It's a map<function*, set<globalvariable*>> so putting a default initialised value doesn't matter much. There are a lot of calls to operator[] in this file, I think I'm convinced they should all be purged as part of adding const to places. I'll avoid introducing this one.

[JonChesterfield]

• Fix the crash as well

[arsenm]

Add the fixes # thing to the commit message so it gets auto closed

llvm/test/CodeGen/AMDGPU/lower-module-lds-all-indirect-accesses.ll

3 ↗

(On Diff #539328)

Add comment explaining the test and link to the issue?

[JonChesterfield]

• comment in regression test

[JonChesterfield]

In D154972#4491717, @arsenm wrote:

Add the fixes # thing to the commit message so it gets auto closed

Good call, thanks!

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

515

One segv was here. This is assigning an array element for each kernel in OrderedKernels, where some might not have a corresponding kernel struct

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

static GlobalVariable *buildLookupTable(
     const size_t NumberVariables = Variables.size();
     const size_t NumberKernels = kernels.size();
 
     ArrayType *KernelOffsetsType =
         ArrayType::get(Type::getInt32Ty(Ctx), NumberVariables);
 
     ArrayType *AllKernelsOffsetsType =
         ArrayType::get(KernelOffsetsType, NumberKernels);
 
     std::vector<Constant *> overallConstantExprElts(NumberKernels);
     for (size_t i = 0; i < NumberKernels; i++) {
       LDSVariableReplacement Replacement = KernelToReplacement[kernels[i]];
       overallConstantExprElts[i] = getAddressesOfVariablesInKernel(
           Ctx, Variables, Replacement.LDSVarsToConstantGEP);
     }
 
     Constant *init =
public:
   lowerKernelScopeStructVariables(
       Module &M, LDSUsesInfoTy &LDSUsesInfo,
       DenseSet<GlobalVariable *> const &ModuleScopeVariables,
       DenseSet<Function *> const &KernelsThatAllocateModuleLDS,
       GlobalVariable *MaybeModuleScopeStruct) {
 
     // Create a struct for each kernel for the non-module-scope variables.
 
+    IRBuilder<> Builder(M.getContext());
     DenseMap<Function *, LDSVariableReplacement> KernelToReplacement;
     for (Function &Func : M.functions()) {
       if (Func.isDeclaration() || !isKernelLDS(&Func))
         continue;
 
       DenseSet<GlobalVariable *> KernelUsedVariables;
       // Allocating variables that are used directly in this struct to get
       // alignment aware allocation and predictable frame size.
for (Function &Func : M.functions()) {
         report_fatal_error("Anonymous kernels cannot use LDS variables");
       }
 
       std::string VarName =
           (Twine("llvm.amdgcn.kernel.") + Func.getName() + ".lds").str();
 
       auto Replacement =
           createLDSVariableReplacement(M, VarName, KernelUsedVariables);
       
+      // In case all uses are from called functions
+      markUsedByKernel(Builder, &Func, Replacement.SGV);
+
       // remove preserves existing codegen
       removeLocalVarsFromUsedLists(M, KernelUsedVariables);
       KernelToReplacement[&Func] = Replacement;
 
       // Rewrite uses within kernel to the new struct
       replaceLDSVariablesWithStruct(
           M, KernelUsedVariables, Replacement, [&Func](Use &U) {
             Instruction *I = dyn_cast<Instruction>(U.getUser());
             return I && I->getFunction() == &Func;
           });
     }
     return KernelToReplacement;
for (auto &GV : KernelAccessVariables) {
       auto &funcs = LDSToKernelsThatNeedToAccessItIndirectly[GV];
       assert(funcs.size() == 1); // Only one kernel can access it
       LDSVariableReplacement Replacement =
           KernelToReplacement[*(funcs.begin())];
 
       DenseSet<GlobalVariable *> Vec;
       Vec.insert(GV);
 
-      // TODO: Looks like a latent bug, Replacement may not be marked
-      // UsedByKernel here
       replaceLDSVariablesWithStruct(M, Vec, Replacement, [](Use &U) {
         return isa<Instruction>(U.getUser());
       });
     }
 
     // The ith element of this vector is kernel id i
     std::vector<Function *> OrderedKernels =
         assignLDSKernelIDToEachKernel(&M, KernelsThatAllocateTableLDS,
                                       KernelsThatIndirectlyAllocateDynamicLDS);
 
     if (!KernelsThatAllocateTableLDS.empty()) {
       LLVMContext &Ctx = M.getContext();
       IRBuilder<> Builder(Ctx);
 
-      for (size_t i = 0; i < OrderedKernels.size(); i++) {
-        markUsedByKernel(Builder, OrderedKernels[i],
-                         KernelToReplacement[OrderedKernels[i]].SGV);
-      }
-
       // The order must be consistent between lookup table and accesses to
       // lookup table
       std::vector<GlobalVariable *> TableLookupVariablesOrdered(
           TableLookupVariables.begin(), TableLookupVariables.end());
       llvm::sort(TableLookupVariablesOrdered.begin(),
                  TableLookupVariablesOrdered.end(),
                  [](const GlobalVariable *lhs, const GlobalVariable *rhs) {
                    return lhs->getName() < rhs->getName();

llvm/test/CodeGen/AMDGPU/ds_write2.ll

 }
 
 @sgemm.lA = internal unnamed_addr addrspace(3) global [264 x float] undef, align 4
 @sgemm.lB = internal unnamed_addr addrspace(3) global [776 x float] undef, align 4
 
 define amdgpu_kernel void @write2_sgemm_sequence(ptr addrspace(1) %C, i32 %lda, i32 %ldb, ptr addrspace(1) %in) #0 {
 ; CI-LABEL: write2_sgemm_sequence:
 ; CI:       ; %bb.0:
-; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x4
-; CI-NEXT:    s_mov_b32 m0, -1
-; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    s_load_dword s0, s[0:1], 0x0
-; CI-NEXT:    s_lshl_b32 s1, s2, 2
-; CI-NEXT:    s_add_i32 s2, s1, 0xc20
-; CI-NEXT:    s_addk_i32 s1, 0xc60
-; CI-NEXT:    v_mov_b32_e32 v0, s2
+; CI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x4
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_lshl_b32 s0, s2, 2
+; CI-NEXT:    s_add_i32 s1, s0, 0xc20
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    v_mov_b32_e32 v2, s0
+; CI-NEXT:    buffer_load_dword v0, off, s[4:7], 0
+; CI-NEXT:    s_addk_i32 s0, 0xc60
+; CI-NEXT:    v_mov_b32_e32 v2, s1
+; CI-NEXT:    s_mov_b32 m0, -1
+; CI-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; CI-NEXT:    v_mov_b32_e32 v3, s0
-; CI-NEXT:    ds_write2_b32 v0, v2, v3 offset1:1
-; CI-NEXT:    v_mov_b32_e32 v0, s1
-; CI-NEXT:    ds_write2_b32 v0, v2, v3 offset1:1
-; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v1
-; CI-NEXT:    ds_write2_b32 v0, v2, v3 offset1:1
-; CI-NEXT:    ds_write2_b32 v0, v2, v3 offset0:32 offset1:33
-; CI-NEXT:    ds_write2_b32 v0, v2, v3 offset0:64 offset1:65
+; CI-NEXT:    s_waitcnt vmcnt(0)
+; CI-NEXT:    ds_write2_b32 v2, v0, v0 offset1:1
+; CI-NEXT:    ds_write2_b32 v3, v0, v0 offset1:1
+; CI-NEXT:    ds_write2_b32 v1, v0, v0 offset1:1
+; CI-NEXT:    ds_write2_b32 v1, v0, v0 offset0:32 offset1:33
+; CI-NEXT:    ds_write2_b32 v1, v0, v0 offset0:64 offset1:65
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: write2_sgemm_sequence:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x10
-; GFX9-NEXT:    s_lshl_b32 s2, s2, 2
-; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_load_dword s0, s[0:1], 0x0
-; GFX9-NEXT:    s_add_i32 s1, s2, 0xc20
-; GFX9-NEXT:    s_addk_i32 s2, 0xc60
-; GFX9-NEXT:    v_mov_b32_e32 v0, s1
-; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 2, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    global_load_dword v0, v0, s[0:1]
+; GFX9-NEXT:    s_lshl_b32 s0, s2, 2
+; GFX9-NEXT:    s_add_i32 s1, s0, 0xc20
+; GFX9-NEXT:    s_addk_i32 s0, 0xc60
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-NEXT:    v_mov_b32_e32 v4, s0
-; GFX9-NEXT:    ds_write2_b32 v0, v3, v4 offset1:1
-; GFX9-NEXT:    ds_write2_b32 v2, v3, v4 offset1:1
-; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v1
-; GFX9-NEXT:    ds_write2_b32 v0, v3, v4 offset1:1
-; GFX9-NEXT:    ds_write2_b32 v0, v3, v4 offset0:32 offset1:33
-; GFX9-NEXT:    ds_write2_b32 v0, v3, v4 offset0:64 offset1:65
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    ds_write2_b32 v2, v0, v0 offset1:1
+; GFX9-NEXT:    ds_write2_b32 v3, v0, v0 offset1:1
+; GFX9-NEXT:    ds_write2_b32 v1, v0, v0 offset1:1
+; GFX9-NEXT:    ds_write2_b32 v1, v0, v0 offset0:32 offset1:33
+; GFX9-NEXT:    ds_write2_b32 v1, v0, v0 offset0:64 offset1:65
 ; GFX9-NEXT:    s_endpgm
   %x.i = tail call i32 @llvm.amdgcn.workgroup.id.x() #1
   %y.i = tail call i32 @llvm.amdgcn.workitem.id.y() #1
   %val = load float, ptr addrspace(1) %in
   %arrayidx44 = getelementptr inbounds [264 x float], ptr addrspace(3) @sgemm.lA, i32 0, i32 %x.i
   store float %val, ptr addrspace(3) %arrayidx44, align 4
   %add47 = add nsw i32 %x.i, 1
   %arrayidx48 = getelementptr inbounds [264 x float], ptr addrspace(3) @sgemm.lA, i32 0, i32 %add47

llvm/test/CodeGen/AMDGPU/lower-module-lds-constantexpr-phi.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @var = addrspace(3) global i32 undef, align 4
 
 ; Regression test. Duplicate constantexpr in phi nodes shall not emit broken IR
 define amdgpu_kernel void @func(i32 %c) {
 ; CHECK-LABEL: @func(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.func.lds) ]
 ; CHECK-NEXT:    switch i32 [[C:%.*]], label [[RETURN:%.*]] [
 ; CHECK-NEXT:    i32 0, label [[BB0:%.*]]
 ; CHECK-NEXT:    i32 1, label [[BB1:%.*]]
 ; CHECK-NEXT:    ]
 ; CHECK:       bb0:
 ; CHECK-NEXT:    [[TMP0:%.*]] = addrspacecast ptr addrspace(3) @llvm.amdgcn.kernel.func.lds to ptr
 ; CHECK-NEXT:    br label [[BB2:%.*]]
 ; CHECK:       bb1:

llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-ambiguous.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -check-prefixes=CHECK,M_OR_HY %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=table | FileCheck -check-prefixes=CHECK,TABLE %s
 ; RUN: not --crash opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=kernel 2>&1 | FileCheck -check-prefixes=KERNEL %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=hybrid | FileCheck -check-prefixes=CHECK,M_OR_HY %s
 
 ;; Two kernels access the same variable, specialisation gives them each their own copy of it
 
 @kernel.lds = addrspace(3) global i8 undef
 define amdgpu_kernel void @k0() {
 ; CHECK-LABEL: @k0(
+; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds) ]
 ; CHECK-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 1
 ; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[LD]], 2
 ; CHECK-NEXT:    store i8 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 1
 ; CHECK-NEXT:    ret void
 ;
   %ld = load i8, ptr addrspace(3) @kernel.lds
   %mul = mul i8 %ld, 2
   store i8 %mul, ptr  addrspace(3) @kernel.lds
   ret void
 }
 
 define amdgpu_kernel void @k1() {
 ; CHECK-LABEL: @k1(
+; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds) ]
 ; CHECK-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 1
 ; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[LD]], 3
 ; CHECK-NEXT:    store i8 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.k1.lds, align 1
 ; CHECK-NEXT:    ret void
 ;
   %ld = load i8, ptr addrspace(3) @kernel.lds
   %mul = mul i8 %ld, 3
   store i8 %mul, ptr  addrspace(3) @kernel.lds

llvm/test/CodeGen/AMDGPU/lower-module-lds-single-var-unambiguous.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -check-prefixes=CHECK,MODULE %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=table | FileCheck -check-prefixes=CHECK,TABLE %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=kernel | FileCheck -check-prefixes=CHECK,K_OR_HY %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=hybrid | FileCheck -check-prefixes=CHECK,K_OR_HY %s
 
 ;; Same checks for kernel and for hybrid as an unambiguous reference to a variable - one where exactly one kernel
 ;; can reach it - is the case where hybrid lowering can always prefer the direct access.
 
 ;; Single kernel is sole user of single variable, all options codegen as direct access to kernel struct
 
 @k0.lds = addrspace(3) global i8 undef
 define amdgpu_kernel void @k0() {
 ; CHECK-LABEL: @k0(
+; CHECK-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds) ]
 ; CHECK-NEXT:    [[LD:%.*]] = load i8, ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 1
 ; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[LD]], 2
 ; CHECK-NEXT:    store i8 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.k0.lds, align 1
 ; CHECK-NEXT:    ret void
 ;
   %ld = load i8, ptr addrspace(3) @k0.lds
   %mul = mul i8 %ld, 2
   store i8 %mul, ptr  addrspace(3) @k0.lds
   ret void
 }
 
 ;; Function is reachable from one kernel. Variable goes in module lds or the kernel struct, but never both.
 
 @f0.lds = addrspace(3) global i16 undef
 define void @f0() {
 ; MODULE-LABEL: @f0(
-; MODULE-NEXT:    [[LD:%.*]] = load i16, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !1, !noalias !4
+; MODULE-NEXT:    [[LD:%.*]] = load i16, ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T:%.*]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope [[META1:![0-9]+]], !noalias [[META4:![0-9]+]]
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i16 [[LD]], 3
-; MODULE-NEXT:    store i16 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope !1, !noalias !4
+; MODULE-NEXT:    store i16 [[MUL]], ptr addrspace(3) getelementptr inbounds ([[LLVM_AMDGCN_MODULE_LDS_T]], ptr addrspace(3) @llvm.amdgcn.module.lds, i32 0, i32 1), align 4, !alias.scope [[META1]], !noalias [[META4]]
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @f0(
 ; TABLE-NEXT:    [[TMP1:%.*]] = call i32 @llvm.amdgcn.lds.kernel.id()
 ; TABLE-NEXT:    [[F0_LDS2:%.*]] = getelementptr inbounds [2 x [2 x i32]], ptr addrspace(4) @llvm.amdgcn.lds.offset.table, i32 0, i32 [[TMP1]], i32 1
 ; TABLE-NEXT:    [[TMP2:%.*]] = load i32, ptr addrspace(4) [[F0_LDS2]], align 4
 ; TABLE-NEXT:    [[F0_LDS3:%.*]] = inttoptr i32 [[TMP2]] to ptr addrspace(3)
 ; TABLE-NEXT:    [[LD:%.*]] = load i16, ptr addrspace(3) [[F0_LDS3]], align 2
;
   %ld = load i16, ptr addrspace(3) @f0.lds
   %mul = mul i16 %ld, 3
   store i16 %mul, ptr  addrspace(3) @f0.lds
   ret void
 }
 
 define amdgpu_kernel void @k_f0() {
 ; MODULE-LABEL: @k_f0(
-; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ], !alias.scope !5, !noalias !1
+; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ], !alias.scope [[META5:![0-9]+]], !noalias [[META1]]
 ; MODULE-NEXT:    call void @f0()
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @k_f0(
 ; TABLE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k_f0.lds) ]
 ; TABLE-NEXT:    call void @f0()
 ; TABLE-NEXT:    ret void
 ;
 ; K_OR_HY-LABEL: @k_f0(
+; K_OR_HY-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k_f0.lds) ]
 ; K_OR_HY-NEXT:    call void @f0()
 ; K_OR_HY-NEXT:    ret void
 ;
   call void @f0()
   ret void
 }
 
 ;; As above, but with the kernel also uing the variable.
 
 @both.lds = addrspace(3) global i32 undef
 define void @f_both() {
 ; MODULE-LABEL: @f_both(
-; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !4
+; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope [[META5]], !noalias [[META4]]
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 4
-; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !4
+; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope [[META5]], !noalias [[META4]]
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @f_both(
 ; TABLE-NEXT:    [[TMP1:%.*]] = call i32 @llvm.amdgcn.lds.kernel.id()
 ; TABLE-NEXT:    [[BOTH_LDS2:%.*]] = getelementptr inbounds [2 x [2 x i32]], ptr addrspace(4) @llvm.amdgcn.lds.offset.table, i32 0, i32 [[TMP1]], i32 0
 ; TABLE-NEXT:    [[TMP2:%.*]] = load i32, ptr addrspace(4) [[BOTH_LDS2]], align 4
 ; TABLE-NEXT:    [[BOTH_LDS3:%.*]] = inttoptr i32 [[TMP2]] to ptr addrspace(3)
 ; TABLE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) [[BOTH_LDS3]], align 4
;
   %mul = mul i32 %ld, 4
   store i32 %mul, ptr  addrspace(3) @both.lds
   ret void
 }
 
 define amdgpu_kernel void @k0_both() {
 ; MODULE-LABEL: @k0_both(
 ; MODULE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.module.lds) ]
-; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !1
+; MODULE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope [[META5]], !noalias [[META1]]
 ; MODULE-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 5
-; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope !5, !noalias !1
+; MODULE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.module.lds, align 4, !alias.scope [[META5]], !noalias [[META1]]
 ; MODULE-NEXT:    call void @f_both()
 ; MODULE-NEXT:    ret void
 ;
 ; TABLE-LABEL: @k0_both(
 ; TABLE-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds) ]
 ; TABLE-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds, align 4
 ; TABLE-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 5
 ; TABLE-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds, align 4
 ; TABLE-NEXT:    call void @f_both()
 ; TABLE-NEXT:    ret void
 ;
 ; K_OR_HY-LABEL: @k0_both(
+; K_OR_HY-NEXT:    call void @llvm.donothing() [ "ExplicitUse"(ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds) ]
 ; K_OR_HY-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds, align 4
 ; K_OR_HY-NEXT:    [[MUL:%.*]] = mul i32 [[LD]], 5
 ; K_OR_HY-NEXT:    store i32 [[MUL]], ptr addrspace(3) @llvm.amdgcn.kernel.k0_both.lds, align 4
 ; K_OR_HY-NEXT:    call void @f_both()
 ; K_OR_HY-NEXT:    ret void
 ;
   %ld = load i32, ptr addrspace(3) @both.lds
   %mul = mul i32 %ld, 5
   store i32 %mul, ptr  addrspace(3) @both.lds
   call void @f_both()
   ret void
 }