[amdgpu] Expand all ConstantExpr users of LDS variables in instructions

[JonChesterfield]

Summary

Bug noted in D112717 can be sidestepped with this change.

Expanding all ConstantExpr involved with LDS up front makes the variable specialisation simpler. Excludes ConstantExpr that don't access LDS to avoid disturbing codegen elsewhere.

[JonChesterfield]

Haven't landed this previously because of the test churn. check-openmp is fine, check-llvm needs manual patch up.

Failed Tests (9):
  LLVM :: CodeGen/AMDGPU/ds_read2.ll
  LLVM :: CodeGen/AMDGPU/ds_write2.ll
  LLVM :: CodeGen/AMDGPU/loop_break.ll
  LLVM :: CodeGen/AMDGPU/lower-kernel-lds-constexpr.ll
  LLVM :: CodeGen/AMDGPU/lower-kernel-lds-super-align.ll
  LLVM :: CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll
  LLVM :: CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll
  LLVM :: CodeGen/AMDGPU/lower-lds-struct-aa.ll
  LLVM :: CodeGen/AMDGPU/lower-module-lds-constantexpr.ll

[JonChesterfield]

• update three tests

[JonChesterfield]

Requires https://reviews.llvm.org/D133028 to update the test cases.

[JonChesterfield]

• update some more tests

[JonChesterfield]

• rebase, two more tests. Codegen improvement is coincidental

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

327

Replacing the ConstantExpr with Instructions is not necessary since the replacement is itself a ConstantExpr. However, if a nexted ConstantExpr is used from multiple functions, the control flow to trace back to the corresponding instruction in a given function is awkward. It's probably better to unconditionally replace every constexpr in every instruction with more instructions, at least as far as LDS lowering is concerned.

[JonChesterfield]

• Change strategy, always expand constantexpr on lds

[JonChesterfield]

• cleanup

[JonChesterfield]

The current revision will fix the O0 codegen bug @carlo.bertolli is looking at.

It's possible, though awkward, to keep the original idea of deleting the call to replaceConstantUsesInFunction if the lambda passed to replaceLDSVariablesWithStruct works much harder to identify the function corresponding to a Use. That's approximately as complicated as expanding the ConstantExpr up front. Choosing this approach because it's convenient for the other WIP on LDS which replaces some expressions with function calls and thus also needs to expand ConstantExpr.

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

208

This needs a test case, though the implementation that was called into by replaceConstantUsesInFunction unconditionally dereferences it which suggests empty blocks don't get here in practice.

[JonChesterfield]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

208

Oh, right. Basic blocks can't be empty - closest one can get is a branch, and in that case we have an instruction to insert before.

[JonChesterfield]

• blocks aren't empty, contain at least a br

[rampitec]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

165

Should this be a SetVector?

[foad]

Bug noted in D112717 can be sidestepped with this change.

Can you add a regression test for that bug?

[JonChesterfield]

Passed internal epsdb in 733447

[JonChesterfield]

In D133422#3785882, @foad wrote:

Bug noted in D112717 can be sidestepped with this change.

Can you add a regression test for that bug?

I could - would have to check the exact failure mode and write a fairly complicated test case. This code can't fall victim to the same bug, as it emits instructions for each phi constantexpr, and the currently broken code is best deleted if/after this patch lands. I'm not sure the test would be value adding but will write it if necessary.

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

165

I don't think so. That would avoid pushing constantexpr that use N distinct global variables N times, but I can't think of one that looks like that. The LDS variables tend to be used by addrspacecast or bitcast nodes at the root. So I think we'd gain nothing from the set behaviour.

If I've missed a case, multiple copies of the same constantexpr here don't affect correctness anyway as there's set semantics further down.

[rampitec]

LGTM

[JonChesterfield]

• Add a regression test for duplicate constantexpr in phi arguments

[JonChesterfield]

In D133422#3786748, @JonChesterfield wrote:

In D133422#3785882, @foad wrote:

Bug noted in D112717 can be sidestepped with this change.

Can you add a regression test for that bug?

I could - would have to check the exact failure mode and write a fairly complicated test case

Test case was easier than anticipated - the constantexpr didn't have to be nested and block order doesn't make a difference. Added before commit. Fails the verifier before this patch, correct after.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

166

Needs a bunch of braces throughout

[foad]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

160

What is replaceConstantExprInFunction?

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "Utils/AMDGPUBaseInfo.h"
 #include "Utils/AMDGPUMemoryUtils.h"
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/MDBuilder.h"
static void markUsedByKernel(IRBuilder<> &Builder, Function *Func,
     Value *UseInstance[1] = {Builder.CreateInBoundsGEP(
         SGV->getValueType(), SGV, ConstantInt::get(Type::getInt32Ty(Ctx), 0))};
 
     Builder.CreateCall(FTy, Decl, {},
                        {OperandBundleDefT<Value *>("ExplicitUse", UseInstance)},
                        "");
   }
 
+  static bool eliminateConstantExprUsesOfLDSFromAllInstructions(Module &M)
+  {
+    // Constants are uniqued within LLVM. A ConstantExpr referring to a LDS
+    // global may have uses from multiple different functions as a result.
+    // This pass specialises LDS variables with respect to the kernel that
+    // allocates them.
+
+    // This is semantically equivalent to:
+    // for (auto &F : M.functions())
+    //   for (auto &BB : F)
+    //     for (auto &I : BB)
+    //       for (Use &Op : I.operands())
+    //         if (constantExprUsesLDS(Op))
+    //           replaceConstantExprInFunction(I, Op);
+
+    bool Changed = false;
+
+    // Find all ConstantExpr that are direct users of an LDS global
+    SmallVector<ConstantExpr*> Stack;
+    for (auto &GV : M.globals())
+      if (AMDGPU::isLDSVariableToLower(GV))
+        for (User *U : GV.users())
+          if (ConstantExpr *C = dyn_cast<ConstantExpr>(U))
+            Stack.push_back(C);
+
+    // Expand to include constexpr users of direct users
+    SetVector<ConstantExpr *> ConstExprUsersOfLDS;
+    while (!Stack.empty()) {
+      ConstantExpr *V = Stack.pop_back_val();
+      if (ConstExprUsersOfLDS.contains(V))
+        continue;
+
+      ConstExprUsersOfLDS.insert(V);
+
+      for (auto *Nested : V->users())
+        if (ConstantExpr *CE = dyn_cast<ConstantExpr>(Nested))
+          Stack.push_back(CE);
+      }
+
+    // Find all instructions that use any of the ConstExpr users of LDS
+    SetVector<Instruction *> InstructionWorklist;
+    for (ConstantExpr *CE : ConstExprUsersOfLDS)
+      for (User *U : CE->users())
+        if (auto *I = dyn_cast<Instruction>(U))
+          InstructionWorklist.insert(I);
+
+    // Replace those ConstExpr operands with instructions
+    while (!InstructionWorklist.empty()) {
+      Instruction *I = InstructionWorklist.pop_back_val();
+      for (Use &U : I->operands()) {
+
+        auto *BI = I;
+        if (auto *Phi = dyn_cast<PHINode>(I)) {
+          BasicBlock *BB = Phi->getIncomingBlock(U);
+          BasicBlock::iterator It = BB->getFirstInsertionPt();
+          if (It != BB->end()) {
+            BI = &(*(It));
+          } else {
+            // getAsInstruction inserts before the instruction passed,
+            // so if the appropriate basic block has no instruction, create
+            // a no-op one to use
+            LLVMContext &Ctx = M.getContext();
+            IRBuilder<> Builder(Ctx);
+            Builder.SetInsertPoint(BB);
+            FunctionType *FTy = FunctionType::get(Type::getVoidTy(Ctx), {});
+            Function *Decl =
+              Intrinsic::getDeclaration(&M, Intrinsic::donothing);
+            Instruction *Call = Builder.CreateCall(FTy, Decl, {});
+            BI = Call;
+          }
+        }
+
+        if (ConstantExpr *C = dyn_cast<ConstantExpr>(U.get())) {
+          if (ConstExprUsersOfLDS.contains(C)) {
+            Changed = true;
+            Instruction *NI = C->getAsInstruction(BI);
+            InstructionWorklist.insert(NI);
+            U.set(NI);
+            C->removeDeadConstantUsers();
+          }
+        }
+      }
+    }
+
+    return Changed;
+  }
+
 public:
   static char ID;
 
   AMDGPULowerModuleLDS() : ModulePass(ID) {
     initializeAMDGPULowerModuleLDSPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnModule(Module &M) override {
     LLVMContext &Ctx = M.getContext();
     CallGraph CG = CallGraph(M);
     bool Changed = superAlignLDSGlobals(M);
 
+    Changed |= eliminateConstantExprUsesOfLDSFromAllInstructions(M);
+    
     // Move variables used by functions into amdgcn.module.lds
     std::vector<GlobalVariable *> ModuleScopeVariables =
         AMDGPU::findLDSVariablesToLower(M, nullptr);
     if (!ModuleScopeVariables.empty()) {
       std::string VarName = "llvm.amdgcn.module.lds";
 
       GlobalVariable *SGV;
       DenseMap<GlobalVariable *, Constant *> LDSVarsToConstantGEP;
bool runOnModule(Module &M) override {
     for (Function &F : M.functions()) {
       if (F.isDeclaration())
         continue;
 
       // Only lower compute kernels' LDS.
       if (!AMDGPU::isKernel(F.getCallingConv()))
         continue;
 
+      fprintf(stderr, "Looking to lower variables in kernel %s\n",
+              F.getName().str().c_str());
+      
       std::vector<GlobalVariable *> KernelUsedVariables =
           AMDGPU::findLDSVariablesToLower(M, &F);
 
-      // Replace all constant uses with instructions if they belong to the
-      // current kernel. Unnecessary, removing will cause test churn.
-      for (GlobalVariable *GV : KernelUsedVariables) {
-        for (User *U : make_early_inc_range(GV->users())) {
-          if (ConstantExpr *C = dyn_cast<ConstantExpr>(U))
-            AMDGPU::replaceConstantUsesInFunction(C, &F);
-        }
-        GV->removeDeadConstantUsers();
-      }
-
       if (!KernelUsedVariables.empty()) {
         std::string VarName =
             (Twine("llvm.amdgcn.kernel.") + F.getName() + ".lds").str();
         GlobalVariable *SGV;
         DenseMap<GlobalVariable *, Constant *> LDSVarsToConstantGEP;
         std::tie(SGV, LDSVarsToConstantGEP) =
             createLDSVariableReplacement(M, VarName, KernelUsedVariables);
 
+        fprintf(stderr, "Created variable %s\n", SGV->getName().str().c_str());
+        
         removeFromUsedLists(M, KernelUsedVariables);
         replaceLDSVariablesWithStruct(
             M, KernelUsedVariables, SGV, LDSVarsToConstantGEP, [&F](Use &U) {
               Instruction *I = dyn_cast<Instruction>(U.getUser());
               return I && I->getFunction() == &F;
             });
         Changed = true;
       }
     }
 
     for (auto &GV : make_early_inc_range(M.globals()))
-      if (AMDGPU::isLDSVariableToLower(GV) && GV.use_empty())
+      if (AMDGPU::isLDSVariableToLower(GV)) {
+        GV.removeDeadConstantUsers();
+        if (GV.use_empty())
           GV.eraseFromParent();
-    
+      }
     return Changed;
   }
 
 private:
   // Increase the alignment of LDS globals if necessary to maximise the chance
   // that we can use aligned LDS instructions to access them.
   static bool superAlignLDSGlobals(Module &M) {
     const DataLayout &DL = M.getDataLayout();
private:
   void replaceLDSVariablesWithStruct(
       Module &M, std::vector<GlobalVariable *> const &LDSVarsToTransform,
       GlobalVariable *SGV,
       DenseMap<GlobalVariable *, Constant *> &LDSVarsToConstantGEP,
       PredicateTy Predicate) {
     LLVMContext &Ctx = M.getContext();
     const DataLayout &DL = M.getDataLayout();
 
+    fprintf(stderr, "replaceLDSVariablesWithStruct on variables:\n");
+    for (auto & GV : LDSVarsToTransform)
+      {
+        GV->dump();
+      }
+    
     // Create alias.scope and their lists. Each field in the new structure
     // does not alias with all other fields.
     SmallVector<MDNode *> AliasScopes;
     SmallVector<Metadata *> NoAliasList;
     const size_t NumberVars = LDSVarsToTransform.size();
     if (NumberVars > 1) {
       MDBuilder MDB(Ctx);
       AliasScopes.reserve(NumberVars);

llvm/lib/Target/AMDGPU/Utils/AMDGPUMemoryUtils.h

 namespace AMDGPU {
 
 Align getAlign(DataLayout const &DL, const GlobalVariable *GV);
 
 bool isLDSVariableToLower(const GlobalVariable &GV);
 std::vector<GlobalVariable *> findLDSVariablesToLower(Module &M,
                                                       const Function *F);
 
-/// Replace all uses of constant \p C with instructions in \p F.
-void replaceConstantUsesInFunction(ConstantExpr *C, const Function *F);
-
 /// Given a \p Def clobbering a load from \p Ptr according to the MSSA check
 /// if this is actually a memory update or an artificial clobber to facilitate
 /// ordering constraints.
 bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA);
 
 /// Check is a \p Load is clobbered in its function.
 bool isClobberedInFunction(const LoadInst *Load, MemorySSA *MSSA,
                            AAResults *AA);
 
 } // end namespace AMDGPU
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_AMDGPU_UTILS_AMDGPUMEMORYUTILS_H

llvm/lib/Target/AMDGPU/Utils/AMDGPUMemoryUtils.cpp

 
 namespace AMDGPU {
 
 Align getAlign(DataLayout const &DL, const GlobalVariable *GV) {
   return DL.getValueOrABITypeAlignment(GV->getPointerAlignment(DL),
                                        GV->getValueType());
 }
 
-static void collectFunctionUses(User *U, const Function *F,
-                                SetVector<Instruction *> &InstUsers) {
-  SmallVector<User *> Stack{U};
-
-  while (!Stack.empty()) {
-    U = Stack.pop_back_val();
-
-    if (auto *I = dyn_cast<Instruction>(U)) {
-      if (I->getFunction() == F)
-        InstUsers.insert(I);
-      continue;
-    }
-
-    if (!isa<ConstantExpr>(U))
-      continue;
-
-    append_range(Stack, U->users());
-  }
-}
-
-void replaceConstantUsesInFunction(ConstantExpr *C, const Function *F) {
-  SetVector<Instruction *> InstUsers;
-
-  collectFunctionUses(C, F, InstUsers);
-  for (Instruction *I : InstUsers) {
-    convertConstantExprsToInstructions(I, C);
-  }
-}
-
 static bool shouldLowerLDSToStruct(const GlobalVariable &GV,
                                    const Function *F) {
   // We are not interested in kernel LDS lowering for module LDS itself.
   if (F && GV.getName() == "llvm.amdgcn.module.lds")
     return false;
 
   bool Ret = false;
   SmallPtrSet<const User *, 8> Visited;

llvm/test/CodeGen/AMDGPU/lower-kernel-lds-constexpr.ll

;
   ret void
 }
 
 @lds.4 = internal unnamed_addr addrspace(3) global [505 x i32] undef, align 4
 
 ; Multiple constexpr use in a same instruction.
 define amdgpu_kernel void @k5() {
 ; CHECK-LABEL: @k5(
-; CHECK-NEXT: %1 = addrspacecast [505 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k5.lds.t, %llvm.amdgcn.kernel.k5.lds.t addrspace(3)* @llvm.amdgcn.kernel.k5.lds, i32 0, i32 0) to [505 x i32]*
-; CHECK-NEXT: %2 = getelementptr inbounds [505 x i32], [505 x i32]* %1, i64 0, i64 0
-; CHECK-NEXT: call void undef(i32* %2, i32* %2)
+; CHECK-NEXT:  %1 = addrspacecast [505 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k5.lds.t, %llvm.amdgcn.kernel.k5.lds.t addrspace(3)* @llvm.amdgcn.kernel.k5.lds, i32 0, i32 0) to [505 x i32]*
+; CHECK-NEXT:  %2 = getelementptr inbounds [505 x i32], [505 x i32]* %1, i64 0, i64 0
+; CHECK-NEXT:  %3 = addrspacecast [505 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k5.lds.t, %llvm.amdgcn.kernel.k5.lds.t addrspace(3)* @llvm.amdgcn.kernel.k5.lds, i32 0, i32 0) to [505 x i32]*
+; CHECK-NEXT:  %4 = getelementptr inbounds [505 x i32], [505 x i32]* %3, i64 0, i64 0
+; CHECK-NEXT:  call void undef(i32* %2, i32* %4)
 ;
   call void undef(i32* getelementptr inbounds ([505 x i32], [505 x i32]* addrspacecast ([505 x i32] addrspace(3)* @lds.4 to [505 x i32]*), i64 0, i64 0), i32* getelementptr inbounds ([505 x i32], [505 x i32]* addrspacecast ([505 x i32] addrspace(3)* @lds.4 to [505 x i32]*), i64 0, i64 0))
   ret void
 }
 
 @lds.5 = internal addrspace(3) global [4 x i32] undef, align 4
 
 ; Both the *value* and *pointer* operands of store instruction are constant expressions, and
 ; both of these constant expression paths use same lds - @lds.5. Hence both of these constant
-; expression operands of store should be replaced by corresponding instruction sequence.
+; expression operands of store should be replaced by equivalent instruction sequences.
 define amdgpu_kernel void @k6() {
 ; CHECK-LABEL: @k6(
-; CHECK-NEXT: %1 = getelementptr inbounds [4 x i32], [4 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k6.lds.t, %llvm.amdgcn.kernel.k6.lds.t addrspace(3)* @llvm.amdgcn.kernel.k6.lds, i32 0, i32 0), i32 0, i32 2
-; CHECK-NEXT: %2 = ptrtoint i32 addrspace(3)* %1 to i32
-; CHECK-NEXT: store i32 %2, i32 addrspace(3)* %1, align 8
-; CHECK-NEXT: ret void
+
+; CHECK-NEXT:  %1 = getelementptr inbounds [4 x i32], [4 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k6.lds.t, %llvm.amdgcn.kernel.k6.lds.t addrspace(3)* @llvm.amdgcn.kernel.k6.lds, i32 0, i32 0), i32 0, i32 2
+; CHECK-NEXT:  %2 = ptrtoint i32 addrspace(3)* %1 to i32
+; CHECK-NEXT:  %3 = getelementptr inbounds [4 x i32], [4 x i32] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k6.lds.t, %llvm.amdgcn.kernel.k6.lds.t addrspace(3)* @llvm.amdgcn.kernel.k6.lds, i32 0, i32 0), i32 0, i32 2
+; CHECK-NEXT:  store i32 %2, i32 addrspace(3)* %3, align 8
+; CHECK-NEXT:  ret void
 ;
   store i32 ptrtoint (i32 addrspace(3)* getelementptr inbounds ([4 x i32], [4 x i32] addrspace(3)* @lds.5, i32 0, i32 2) to i32), i32 addrspace(3)* getelementptr inbounds ([4 x i32], [4 x i32] addrspace(3)* @lds.5, i32 0, i32 2)
   ret void
 }

llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll

 @_f2 = linkonce_odr hidden local_unnamed_addr addrspace(3) global %vec_type undef, comdat, align 1
 
 ; GCN-LABEL: @test
 ; GCN: v_mov_b32_e32 [[REG:v[0-9]+]], 1
 ; GCN-NEXT: global_store_byte v{{[0-9]+}}, [[REG]]
 
 ; CHECK-LABEL: @test
 ; CHECK: store i8 3, i8 addrspace(3)* %0, align 4, !alias.scope !0, !noalias !3
-; CHECK: tail call void @llvm.memcpy.p3i8.p3i8.i64(i8 addrspace(3)* noundef align 1 dereferenceable(3) %2, i8 addrspace(3)* noundef align 1 dereferenceable(3) %1, i64 3, i1 false), !alias.scope !5, !noalias !6
+; CHECK: %1 = getelementptr
+; CHECK: %2 = getelementptr
+; CHECK: tail call void @llvm.memcpy.p3i8.p3i8.i64(i8 addrspace(3)* noundef align 1 dereferenceable(3) %1, i8 addrspace(3)* noundef align 1 dereferenceable(3) %2, i64 3, i1 false), !alias.scope !5, !noalias !6
 ; CHECK: %4 = load i8, i8 addrspace(3)* %3, align 4, !alias.scope !3, !noalias !0
-; CHECK: tail call void @llvm.memcpy.p3i8.p3i8.i64(i8 addrspace(3)* noundef align 1 dereferenceable(3) %7, i8 addrspace(3)* noundef align 1 dereferenceable(3) %6, i64 3, i1 false), !alias.scope !5, !noalias !6
+; CHECK: %5 = getelementptr
+; CHECK: %6 = getelementptr
+; CHECK: %7 = getelementptr
+; CHECK: tail call void @llvm.memcpy.p3i8.p3i8.i64(i8 addrspace(3)* noundef align 1 dereferenceable(3) %6, i8 addrspace(3)* noundef align 1 dereferenceable(3) %7, i64 3, i1 false), !alias.scope !5, !noalias !6
+; CHECK: %8 = getelementptr
 ; CHECK: %9 = load i8, i8 addrspace(3)* %8, align 4, !alias.scope !3, !noalias !0
 
 define protected amdgpu_kernel void @test(i8 addrspace(1)* nocapture %ptr.coerce) local_unnamed_addr #0 {
 entry:
   store i8 3, i8 addrspace(3)* getelementptr inbounds (%vec_type, %vec_type addrspace(3)* @_f1, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0), align 1
   tail call void @llvm.memcpy.p3i8.p3i8.i64(i8 addrspace(3)* noundef align 1 dereferenceable(3) getelementptr inbounds (%vec_type, %vec_type addrspace(3)* @_f2, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0), i8 addrspace(3)* noundef align 1 dereferenceable(3) getelementptr inbounds (%vec_type, %vec_type addrspace(3)* @_f1, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0), i64 3, i1 false)
   %0 = load i8, i8 addrspace(3)* getelementptr inbounds (%vec_type, %vec_type addrspace(3)* @_f2, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0), align 1
   %cmp.i.i = icmp eq i8 %0, 3

llvm/test/CodeGen/AMDGPU/lower-module-lds-constantexpr.ll

 ; CHECK-NOT: @both
 ; sorted both < func, so @both at null and @func at 4
 @both = addrspace(3) global float undef, align 4
 
 ; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 4
 ; CHECK: @llvm.amdgcn.kernel.timestwo.lds = internal addrspace(3) global %llvm.amdgcn.kernel.timestwo.lds.t undef, align 4
 
 ; CHECK-LABEL: @get_func()
-; CHECK: %0 = load i32, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (%llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (%llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
+; CHECK:       %0 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:       %1 = addrspacecast i32 addrspace(3)* %0 to i32*
+; CHECK:       %2 = ptrtoint i32* %1 to i64
+; CHECK:       %3 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:       %4 = addrspacecast i32 addrspace(3)* %3 to i32*
+; CHECK:       %5 = ptrtoint i32* %4 to i64
+; CHECK:       %6 = add i64 %2, %5
+; CHECK:       %7 = inttoptr i64 %6 to i32*
+; CHECK:       %8 = load i32, i32* %7, align 4
+; CHECK:       ret i32 %8
 define i32 @get_func() local_unnamed_addr #0 {
 entry:
   %0 = load i32, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @func to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @func to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   ret i32 %0
 }
 
 ; CHECK-LABEL: @set_func(i32 %x)
-; CHECK: store i32 %x, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
+; CHECK:      %0 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %1 = addrspacecast i32 addrspace(3)* %0 to i32*
+; CHECK:      %2 = ptrtoint i32* %1 to i64
+; CHECK:      %3 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %4 = addrspacecast i32 addrspace(3)* %3 to i32*
+; CHECK:      %5 = ptrtoint i32* %4 to i64
+; CHECK:      %6 = add i64 %2, %5
+; CHECK:      %7 = inttoptr i64 %6 to i32*
+; CHECK:      store i32 %x, i32* %7, align 4
+; CHECK:      ret void
 define void @set_func(i32 %x) local_unnamed_addr #1 {
 entry:
   store i32 %x, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @both to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @both to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   ret void
 }
 
 ; CHECK-LABEL: @timestwo() #0
 ; CHECK-NOT: call void @llvm.donothing()
-; CHECK: %1 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
-; CHECK: %2 = addrspacecast i32 addrspace(3)* %1 to i32*
-; CHECK: %3 = ptrtoint i32* %2 to i64
-; CHECK: %4 = add i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*) to i32*) to i64), %3
-; CHECK: %5 = inttoptr i64 %4 to i32*
-; CHECK: %ld = load i32, i32* %5, align 4
-; CHECK: %mul = mul i32 %ld, 2
-; CHECK: %6 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
-; CHECK: %7 = addrspacecast i32 addrspace(3)* %6 to i32*
-; CHECK: %8 = ptrtoint i32* %7 to i64
-; CHECK: %9 = add i64 %8, ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*) to i32*) to i64)
-; CHECK: %10 = inttoptr i64 %9 to i32*
-; CHECK: store i32 %mul, i32* %10, align 4
+
+
+; CHECK:      %1 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %2 = addrspacecast i32 addrspace(3)* %1 to i32*
+; CHECK:      %3 = ptrtoint i32* %2 to i64
+; CHECK:      %4 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:      %5 = addrspacecast i32 addrspace(3)* %4 to i32*
+; CHECK:      %6 = ptrtoint i32* %5 to i64
+; CHECK:      %7 = add i64 %3, %6
+; CHECK:      %8 = inttoptr i64 %7 to i32*
+; CHECK:      %ld = load i32, i32* %8, align 4
+; CHECK:      %mul = mul i32 %ld, 2
+; CHECK:      %9 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:      %10 = addrspacecast i32 addrspace(3)* %9 to i32*
+; CHECK:      %11 = ptrtoint i32* %10 to i64
+; CHECK:      %12 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %13 = addrspacecast i32 addrspace(3)* %12 to i32*
+; CHECK:      %14 = ptrtoint i32* %13 to i64
+; CHECK:      %15 = add i64 %11, %14
+; CHECK:      %16 = inttoptr i64 %15 to i32*
+; CHECK:      store i32 %mul, i32* %16, align 4
+; CHECK:      ret void
 define amdgpu_kernel void @timestwo() {
   %ld = load i32, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @both to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @kern to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   %mul = mul i32 %ld, 2
   store i32 %mul, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @kern to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @both to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   ret void
 }
 
 attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }