[AMDGPU] Improve indirect call support in closed modules

[jdoerfert]

Summary

If we see all functions that can be called, thus in a "closed module",
we can perform better reasoning in the presence of unknown callees of
indirect calls. We now collect all indirectly callable functions, and
match them against an indirect call site with an unknown callee. Every
function that is a candidate is potentially called.

The AMDGPU backend is the only user for now. We should enable this for
AMDGPU (and NVIDIA GPUs in certain cases) also when we run the
Attributor (or OpenMP-opt) earlier in the pipeline.

TODOs

• We should improve the "is candidate" logic.

[jdoerfert]

llvm/test/CodeGen/AMDGPU/attributor-loop-issue-58639.ll

64

We should have realized this is UB from the beginning. I think we didn't because AAPotentialValues (and maybe more AAs) is not allowed in the AMDGPUAttributor run.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

953

This should probably feed from a pass option like other end-of-LTO things

llvm/lib/Transforms/IPO/AttributorAttributes.cpp

10411

Move to a separate real function?

10416

The float and double cases are covered by T1 == T2?

[jdoerfert]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

953

Those do not feed into the backend passes, as far as I can tell. I am tempted to add the Module flag back.

llvm/lib/Transforms/IPO/AttributorAttributes.cpp

10416

Right.

[arsenm]

llvm/lib/Transforms/IPO/AttributorAttributes.cpp

10444

Need tests for all this type pun stuff

[jdoerfert]

Split out TTI stuff

[jdoerfert]

update tests

[scott.linder]

llvm/lib/Transforms/IPO/Attributor.cpp

3250

Comment should describe the new case

3254–3268

I don't understand the context fully, but is the intention to have &F pushed into IndirectlyCallableFunctions repeatedly if there are repeated calls to it? And that earlier-ordered calls are pushed to it, but none are considered once any must-tail-call are reached? I would have expected the push to be pulled out of the loop, as those constraints don't sound meaningful to me. The continue case is also confusing to me, unless there is some side-effect in the loop I'm missing? The same question applies to the original code, where I would have expected a break after CalledViaMustTail is set to true, so I assume I am just missing something?

In addition to all the above confusion the condition on the outermost if also seems strange, as there are two separable cases, one of which depends on the other, but they are combined in a non-obvious way, namely around the starting value of FI.CalledViaMustTail.

I added a suggestion of what makes sense to me, but there are far too many things I'm confused about to have any confidence my suggestions are useful. Any help in understanding would be appreciated!

[jdoerfert]

llvm/lib/Transforms/IPO/Attributor.cpp

3254–3268

I don't understand the context fully, but is the intention to have &F pushed into IndirectlyCallableFunctions repeatedly if there are repeated calls to it?

No. It always ends up in there once. I might have placed it in the wrong scope. I'll check your code.

The continue case is also confusing to me, unless there is some side-effect in the loop I'm missing? The same question applies to the original code, where I would have expected a break after CalledViaMustTail is set to true, so I assume I am just missing something?

The original code seems to be missing a break.

In addition to all the above confusion the condition on the outermost if also seems strange, as there are two separable cases, one of which depends on the other, but they are combined in a non-obvious way, namely around the starting value of FI.CalledViaMustTail.

The outer if has two conditions separated with a ||, no?
If either is true, we want to look at users.
First is isClosedWorldModule(), second is (!isModulePass() && !FI.CalledViaMustTail).
Why do they depend on each other? Maybe the confusion is that we might search for must tail if we don't need to, which is a fair criticism. If we set closed world in the backend (for now), I'll clean this up.

[scott.linder]

llvm/lib/Transforms/IPO/Attributor.cpp

3254–3268

The outer if has two conditions separated with a ||, no?
If either is true, we want to look at users.
First is isClosedWorldModule(), second is (!isModulePass() && !FI.CalledViaMustTail).
Why do they depend on each other? Maybe the confusion is that we might search for must tail if we don't need to, which is a fair criticism. If we set closed world in the backend (for now), I'll clean this up.

Ah, maybe my confusion is just with the relationship between CalledViaMustTail and IndirectlyCallableFunctions, or possibly the lack of relationship between them? In the current version my reading was !CalledViaMustTail && !F.hasLocalLinkage() implied we don't want to update IndirectlyCallableFunctions. The fact that something might just be in the wrong scope might explain the whole thing, and the alternative condition I suggested might make no sense if there is no actual connection between the two.

Anyway, if the two things can be separated entirely that would make it much easier to read. As it stands there are too many possible dynamic execution paths to keep track of easily, IMO.

[arsenm]

llvm/include/llvm/Transforms/IPO/Attributor.h

1318

Should this return ArrayRef?

[jdoerfert]

rebase on top of the changes in main

[jdoerfert]

Remove AA code paths and focus on AMDGPU

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

953

Not sure what you mean, there are pass parameters like any other

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

971–973

I don't really understand what optional with null is supposed to mean. Just drop the optional?

975

• instead of .value()?

llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll

21

poison

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

962

Can you move this to a real function with a name?

978–980

AMDGPU::isArgPassedInSGPR. Alternatively we could add some kind of variant of TTI::isAlwaysUniform which covers arguments

[jdoerfert]

All but the pass arg addressed in https://github.com/llvm/llvm-project/pull/66488

llvm/include/llvm/Transforms/IPO/Attributor.h

struct InformationCache {
   /// Return true if the stack (llvm::Alloca) can be accessed by other threads.
   bool stackIsAccessibleByOtherThreads() { return !targetIsGPU(); }
 
   /// Return true if the target is a GPU.
   bool targetIsGPU() {
     return TargetTriple.isAMDGPU() || TargetTriple.isNVPTX();
   }
 
+  const SmallVectorImpl<Function *> &getIndirectlyCallableFunctions() const {
+    return IndirectlyCallableFunctions;
+  }
+
 private:
   struct FunctionInfo {
     ~FunctionInfo();
 
     /// A nested map that remembers all instructions in a function with a
     /// certain instruction opcode (Instruction::getOpcode()).
     OpcodeInstMapTy OpcodeInstMap;
 
FunctionInfo &getFunctionInfo(const Function &F) {
     FunctionInfo *&FI = FuncInfoMap[&F];
     if (!FI) {
       FI = new (Allocator) FunctionInfo();
       initializeInformationCache(F, *FI);
     }
     return *FI;
   }
 
+  /// Vector of functions that might be callable indirectly, i.a., via a
+  /// function pointer.
+  SmallVector<Function *> IndirectlyCallableFunctions;
+
   /// Initialize the function information cache \p FI for the function \p F.
   ///
   /// This method needs to be called for all function that might be looked at
   /// through the information cache interface *prior* to looking at them.
   void initializeInformationCache(const Function &F, FunctionInfo &FI);
 
   /// The datalayout used in the module.
   const DataLayout &DL;
struct AttributorConfig {
 
   /// Flag to determine if we want to initialize all default AAs for an internal
   /// function marked live. See also: InitializationCallback>
   bool DefaultInitializeLiveInternals = true;
 
   /// Flag to determine if we should skip all liveness checks early on.
   bool UseLiveness = true;
 
+  /// Flag to indicate if the entire world is contained in this module, that
+  /// is, no outside functions exist.
+  bool IsClosedWorldModule = false;
+
   /// Callback function to be invoked on internal functions marked live.
   std::function<void(Attributor &A, const Function &F)> InitializationCallback =
       nullptr;
 
   /// Helper to update an underlying call graph and to delete functions.
   CallGraphUpdater &CGUpdater;
 
   /// If not null, a set limiting the attribute opportunities.
struct Attributor {
   }
 
   /// Return the internal information cache.
   InformationCache &getInfoCache() { return InfoCache; }
 
   /// Return true if this is a module pass, false otherwise.
   bool isModulePass() const { return Configuration.IsModulePass; }
 
+  /// Return true if the module contains the whole world, thus, no outside
+  /// functions exist.
+  bool isClosedWorldModule() const { return Configuration.IsClosedWorldModule; }
+
   /// Return true if we derive attributes for \p Fn
   bool isRunOn(Function &Fn) const { return isRunOn(&Fn); }
   bool isRunOn(Function *Fn) const {
     return Functions.empty() || Functions.count(Fn);
   }
 
   template <typename AAType> bool shouldUpdateAA(const IRPosition &IRP) {
     // If this is queried in the manifest stage, we force the AA to indicate

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

DenseSet<const char *> Allowed(
          &AAPotentialValues::ID, &AAAMDFlatWorkGroupSize::ID,
          &AAAMDWavesPerEU::ID, &AACallEdges::ID, &AAPointerInfo::ID,
          &AAPotentialConstantValues::ID, &AAUnderlyingObjects::ID});
 
     AttributorConfig AC(CGUpdater);
     AC.Allowed = &Allowed;
     AC.IsModulePass = true;
     AC.DefaultInitializeLiveInternals = false;
+    AC.IsClosedWorldModule = true;
     AC.IPOAmendableCB = [](const Function &F) {
       return F.getCallingConv() == CallingConv::AMDGPU_KERNEL;
     };
 
     Attributor A(Functions, InfoCache, AC);
 
     for (Function &F : M) {
       if (!F.isIntrinsic()) {
         A.getOrCreateAAFor<AAAMDAttributes>(IRPosition::function(F));
         A.getOrCreateAAFor<AAUniformWorkGroupSize>(IRPosition::function(F));
         if (!AMDGPU::isEntryFunctionCC(F.getCallingConv())) {
           A.getOrCreateAAFor<AAAMDFlatWorkGroupSize>(IRPosition::function(F));
           A.getOrCreateAAFor<AAAMDWavesPerEU>(IRPosition::function(F));
         }
       }
     }
 
     ChangeStatus Change = A.run();
     return Change == ChangeStatus::CHANGED;
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<CycleInfoWrapperPass>();
   }
 
   StringRef getPassName() const override { return "AMDGPU Attributor"; }
   TargetMachine *TM;
   static char ID;
 };
 } // namespace
 
 char AMDGPUAttributor::ID = 0;
 
 Pass *llvm::createAMDGPUAttributorPass() { return new AMDGPUAttributor(); }
 INITIALIZE_PASS_BEGIN(AMDGPUAttributor, DEBUG_TYPE, "AMDGPU Attributor", false,
                       false)
 INITIALIZE_PASS_DEPENDENCY(CycleInfoWrapperPass);
 INITIALIZE_PASS_END(AMDGPUAttributor, DEBUG_TYPE, "AMDGPU Attributor", false,
                     false)

llvm/lib/Transforms/IPO/Attributor.cpp

 }
 
 void Attributor::identifyDefaultAbstractAttributes(Function &F) {
   if (!VisitedFunctions.insert(&F).second)
     return;
   if (F.isDeclaration())
     return;
 
   // In non-module runs we need to look at the call sites of a function to
   // determine if it is part of a must-tail call edge. This will influence what
   // attributes we can derive.
   InformationCache::FunctionInfo &FI = InfoCache.getFunctionInfo(F);
-  if (!isModulePass() && !FI.CalledViaMustTail) {
-    for (const Use &U : F.uses())
+  if (isClosedWorldModule() || (!isModulePass() && !FI.CalledViaMustTail)) {
+    bool IsIndirectlyCallable = !isClosedWorldModule() || !F.hasLocalLinkage();
+    for (const Use &U : F.uses()) {
       if (const auto *CB = dyn_cast<CallBase>(U.getUser()))
-        if (CB->isCallee(&U) && CB->isMustTailCall())
+        if (CB->isCallee(&U))
+          if (CB->isMustTailCall()) {
             FI.CalledViaMustTail = true;
+            if (IsIndirectlyCallable)
+              break;
+            continue;
+          }
+      if (isClosedWorldModule() && IsIndirectlyCallable)
+        InfoCache.IndirectlyCallableFunctions.push_back(&F);
+    }
   }
 
   IRPosition FPos = IRPosition::function(F);
   bool IsIPOAmendable = isFunctionIPOAmendable(F);
   auto Attrs = F.getAttributes();
   auto FnAttrs = Attrs.getFnAttrs();
 
   // Check for dead BasicBlocks in every function.
   // We need dead instruction detection because we do not want to deal with

llvm/lib/Transforms/IPO/AttributorAttributes.cpp

private:
 
   /// Is there any call with a unknown callee, excluding any inline asm.
   bool HasUnknownCalleeNonAsm = false;
 };
 
 struct AACallEdgesCallSite : public AACallEdgesImpl {
   AACallEdgesCallSite(const IRPosition &IRP, Attributor &A)
       : AACallEdgesImpl(IRP, A) {}
+
   /// See AbstractAttribute::updateImpl(...).
   ChangeStatus updateImpl(Attributor &A) override {
     ChangeStatus Change = ChangeStatus::UNCHANGED;
 
+    CallBase *CB = cast<CallBase>(getCtxI());
+
+    auto IsValidTypePun = [](Type &T1, Type &T2) {
+      if (&T1 == &T2)
+        return true;
+      if (T1.isIntOrPtrTy() && T2.isIntOrIntVectorTy())
+        return true;
+      if (T1.isFloatTy() && T2.isFloatTy())
+        return true;
+      if (T1.isDoubleTy() && T2.isDoubleTy())
+        return true;
+      return false;
+    };
+
     auto VisitValue = [&](Value &V, const Instruction *CtxI) -> bool {
       if (Function *Fn = dyn_cast<Function>(&V)) {
         addCalledFunction(Fn, Change);
-      } else {
-        LLVM_DEBUG(dbgs() << "[AACallEdges] Unrecognized value: " << V << "\n");
+        // Explore all values.
+        return true;
+      }
+      if (!A.isClosedWorldModule()) {
+        LLVM_DEBUG(if (!hasUnknownCallee()) dbgs()
+                   << "[AACallEdges] Assume unknown callee due to: " << V
+                   << "\n");
         setHasUnknownCallee(true, Change);
+        // Explore all values.
+        return true;
       }
 
+      unsigned NumArgs = CB->arg_size();
+      LLVM_DEBUG(dbgs() << "[AACallEdges] Unrecognized value: " << V
+                        << ", checking indirect callable functions:\n");
+      for (auto *Fn : A.getInfoCache().getIndirectlyCallableFunctions()) {
+        if (Fn->arg_size() != NumArgs)
+          continue;
+        if (!IsValidTypePun(*Fn->getReturnType(), *CB->getType()))
+          continue;
+        bool Valid = true;
+        for (unsigned ArgNo = 0; Valid && ArgNo < NumArgs; ++ArgNo)
+          Valid &= IsValidTypePun(*Fn->getArg(ArgNo)->getType(),
+                                  *CB->getArgOperand(ArgNo)->getType());
+        if (!Valid)
+          continue;
+        LLVM_DEBUG(dbgs() << "[AACallEdges] Add compatible callable function: "
+                          << Fn->getName() << "\n");
+        addCalledFunction(Fn, Change);
+      }
       // Explore all values.
       return true;
     };
 
     SmallVector<AA::ValueAndContext> Values;
     // Process any value that we might call.
     auto ProcessCalledOperand = [&](Value *V, Instruction *CtxI) {
       if (isa<Constant>(V)) {
         VisitValue(*V, CtxI);
         return;
       }
 
       bool UsedAssumedInformation = false;
       Values.clear();
       if (!A.getAssumedSimplifiedValues(IRPosition::value(*V), *this, Values,
                                         AA::AnyScope, UsedAssumedInformation)) {
         Values.push_back({*V, CtxI});
       }
       for (auto &VAC : Values)
         VisitValue(*VAC.getValue(), VAC.getCtxI());
     };
 
-    CallBase *CB = cast<CallBase>(getCtxI());
 
     if (auto *IA = dyn_cast<InlineAsm>(CB->getCalledOperand())) {
       if (IA->hasSideEffects() &&
           !hasAssumption(*CB->getCaller(), "ompx_no_call_asm") &&
           !hasAssumption(*CB, "ompx_no_call_asm")) {
         setHasUnknownCallee(false, Change);
       }
       return Change;

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-indirect-call.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -global-isel -stop-after=irtranslator -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx900 -verify-machineinstrs -o - %s | FileCheck -enable-var-scope %s
 
 define amdgpu_kernel void @test_indirect_call_sgpr_ptr(ptr %fptr) {
   ; CHECK-LABEL: name: test_indirect_call_sgpr_ptr
   ; CHECK: bb.1 (%ir-block.0):
-  ; CHECK-NEXT:   liveins: $sgpr14, $sgpr15, $sgpr16, $vgpr0, $vgpr1, $vgpr2, $sgpr4_sgpr5, $sgpr6_sgpr7, $sgpr8_sgpr9, $sgpr10_sgpr11
+  ; CHECK-NEXT:   liveins: $sgpr8, $vgpr0, $sgpr4_sgpr5
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:vgpr_32(s32) = COPY $vgpr2
-  ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32(s32) = COPY $vgpr1
-  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:vgpr_32(s32) = COPY $vgpr0
-  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr_32 = COPY $sgpr16
-  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr_32 = COPY $sgpr15
-  ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:sgpr_32 = COPY $sgpr14
-  ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:sgpr_64 = COPY $sgpr10_sgpr11
-  ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:sgpr_64 = COPY $sgpr6_sgpr7
-  ; CHECK-NEXT:   [[COPY8:%[0-9]+]]:sgpr_64 = COPY $sgpr4_sgpr5
-  ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
+  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:vgpr_32(s32) = COPY $vgpr0
+  ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr_32 = COPY $sgpr8
+  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:_(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
   ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(p0) = G_LOAD [[INT]](p4) :: (dereferenceable invariant load (p0) from %ir.fptr.kernarg.offset1, align 16, addrspace 4)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $scc
-  ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
-  ; CHECK-NEXT:   [[COPY11:%[0-9]+]]:_(p4) = COPY [[COPY7]]
-  ; CHECK-NEXT:   [[COPY12:%[0-9]+]]:_(p4) = COPY [[COPY9]](p4)
+  ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p4) = G_IMPLICIT_DEF
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:_(p4) = COPY [[DEF]](p4)
+  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:_(p4) = COPY [[COPY2]](p4)
   ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
-  ; CHECK-NEXT:   [[PTR_ADD:%[0-9]+]]:_(p4) = G_PTR_ADD [[COPY12]], [[C]](s64)
-  ; CHECK-NEXT:   [[COPY13:%[0-9]+]]:_(s64) = COPY [[COPY6]]
-  ; CHECK-NEXT:   [[COPY14:%[0-9]+]]:_(s32) = COPY [[COPY5]]
-  ; CHECK-NEXT:   [[COPY15:%[0-9]+]]:_(s32) = COPY [[COPY4]]
-  ; CHECK-NEXT:   [[COPY16:%[0-9]+]]:_(s32) = COPY [[COPY3]]
-  ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
-  ; CHECK-NEXT:   [[COPY17:%[0-9]+]]:_(s32) = COPY [[COPY2]](s32)
-  ; CHECK-NEXT:   [[COPY18:%[0-9]+]]:_(s32) = COPY [[COPY1]](s32)
-  ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 10
-  ; CHECK-NEXT:   [[SHL:%[0-9]+]]:_(s32) = G_SHL [[COPY18]], [[C1]](s32)
-  ; CHECK-NEXT:   [[OR:%[0-9]+]]:_(s32) = G_OR [[COPY17]], [[SHL]]
-  ; CHECK-NEXT:   [[COPY19:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
-  ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 20
-  ; CHECK-NEXT:   [[SHL1:%[0-9]+]]:_(s32) = G_SHL [[COPY19]], [[C2]](s32)
-  ; CHECK-NEXT:   [[OR1:%[0-9]+]]:_(s32) = G_OR [[OR]], [[SHL1]]
-  ; CHECK-NEXT:   [[COPY20:%[0-9]+]]:_(<4 x s32>) = COPY $private_rsrc_reg
-  ; CHECK-NEXT:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY20]](<4 x s32>)
-  ; CHECK-NEXT:   $sgpr4_sgpr5 = COPY [[COPY10]](p4)
-  ; CHECK-NEXT:   $sgpr6_sgpr7 = COPY [[COPY11]](p4)
+  ; CHECK-NEXT:   [[PTR_ADD:%[0-9]+]]:_(p4) = G_PTR_ADD [[COPY4]], [[C]](s64)
+  ; CHECK-NEXT:   [[DEF1:%[0-9]+]]:_(s64) = G_IMPLICIT_DEF
+  ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:_(s32) = COPY [[COPY1]]
+  ; CHECK-NEXT:   [[DEF2:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+  ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:_(s32) = COPY [[DEF2]](s32)
+  ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:_(s32) = COPY [[DEF2]](s32)
+  ; CHECK-NEXT:   [[COPY8:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
+  ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(<4 x s32>) = COPY $private_rsrc_reg
+  ; CHECK-NEXT:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY9]](<4 x s32>)
+  ; CHECK-NEXT:   $sgpr4_sgpr5 = COPY [[DEF]](p4)
+  ; CHECK-NEXT:   $sgpr6_sgpr7 = COPY [[COPY3]](p4)
   ; CHECK-NEXT:   $sgpr8_sgpr9 = COPY [[PTR_ADD]](p4)
-  ; CHECK-NEXT:   $sgpr10_sgpr11 = COPY [[COPY13]](s64)
-  ; CHECK-NEXT:   $sgpr12 = COPY [[COPY14]](s32)
-  ; CHECK-NEXT:   $sgpr13 = COPY [[COPY15]](s32)
-  ; CHECK-NEXT:   $sgpr14 = COPY [[COPY16]](s32)
-  ; CHECK-NEXT:   $sgpr15 = COPY [[DEF]](s32)
-  ; CHECK-NEXT:   $vgpr31 = COPY [[OR1]](s32)
+  ; CHECK-NEXT:   $sgpr10_sgpr11 = COPY [[DEF1]](s64)
+  ; CHECK-NEXT:   $sgpr12 = COPY [[COPY5]](s32)
+  ; CHECK-NEXT:   $sgpr13 = COPY [[DEF2]](s32)
+  ; CHECK-NEXT:   $sgpr14 = COPY [[COPY6]](s32)
+  ; CHECK-NEXT:   $sgpr15 = COPY [[COPY7]](s32)
+  ; CHECK-NEXT:   $vgpr31 = COPY [[COPY8]](s32)
   ; CHECK-NEXT:   $sgpr30_sgpr31 = G_SI_CALL [[LOAD]](p0), 0, csr_amdgpu, implicit $sgpr0_sgpr1_sgpr2_sgpr3, implicit $sgpr4_sgpr5, implicit $sgpr6_sgpr7, implicit $sgpr8_sgpr9, implicit $sgpr10_sgpr11, implicit $sgpr12, implicit $sgpr13, implicit $sgpr14, implicit $sgpr15, implicit $vgpr31
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $scc
   ; CHECK-NEXT:   S_ENDPGM 0
   call void %fptr()
   ret void
 }
 
 define amdgpu_gfx void @test_gfx_indirect_call_sgpr_ptr(ptr %fptr) {

llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll

 declare i32 @llvm.amdgcn.workitem.id.z() #0
 
 declare ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() #0
 declare ptr addrspace(4) @llvm.amdgcn.queue.ptr() #0
 declare ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr() #0
 declare ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() #0
 declare i64 @llvm.amdgcn.dispatch.id() #0
 
 define void @use_workitem_id_x() #1 {
 ; AKF_HSA-LABEL: define {{[^@]+}}@use_workitem_id_x
 ; AKF_HSA-SAME: () #[[ATTR1:[0-9]+]] {
 ; AKF_HSA-NEXT:    [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.x()
 ; AKF_HSA-NEXT:    store volatile i32 [[VAL]], ptr addrspace(1) undef, align 4
 ; AKF_HSA-NEXT:    ret void
 ;
 ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workitem_id_x
 ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1:[0-9]+]] {

 define float @func_indirect_call(ptr %fptr) #3 {
 ; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_call
 ; AKF_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR3]] {
 ; AKF_HSA-NEXT:    [[F:%.*]] = call float [[FPTR]]()
 ; AKF_HSA-NEXT:    [[FADD:%.*]] = fadd float [[F]], 1.000000e+00
 ; AKF_HSA-NEXT:    ret float [[FADD]]
 ;
 ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_call
-; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR16]] {
+; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR17]] {
 ; ATTRIBUTOR_HSA-NEXT:    [[F:%.*]] = call float [[FPTR]]()
 ; ATTRIBUTOR_HSA-NEXT:    [[FADD:%.*]] = fadd float [[F]], 1.000000e+00
 ; ATTRIBUTOR_HSA-NEXT:    ret float [[FADD]]
 ;
   %f = call float %fptr()
   %fadd = fadd float %f, 1.0
   ret float %fadd
 }

 define float @func_null_call(ptr %fptr) #3 {
 ; AKF_HSA-LABEL: define {{[^@]+}}@func_null_call
 ; AKF_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR3]] {
 ; AKF_HSA-NEXT:    [[F:%.*]] = call float null()
 ; AKF_HSA-NEXT:    [[FADD:%.*]] = fadd float [[F]], 1.000000e+00
 ; AKF_HSA-NEXT:    ret float [[FADD]]
 ;
 ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_null_call
-; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR16]] {
+; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR17]] {
 ; ATTRIBUTOR_HSA-NEXT:    [[F:%.*]] = call float null()
 ; ATTRIBUTOR_HSA-NEXT:    [[FADD:%.*]] = fadd float [[F]], 1.000000e+00
 ; ATTRIBUTOR_HSA-NEXT:    ret float [[FADD]]
 ;
   %f = call float null()
   %fadd = fadd float %f, 1.0
   ret float %fadd
 }

llvm/test/CodeGen/AMDGPU/attributor-loop-issue-58639.ll

 bb5:                                              ; preds = %bb5, %bb3
   %tmp6 = load ptr, ptr %tmp4, align 8
   %tmp7 = call fastcc i1 @widget(ptr %tmp6)
   br label %bb5
 }
 
 define amdgpu_kernel void @entry() {
 ; CHECK-LABEL: define {{[^@]+}}@entry
-; CHECK-SAME: () #[[ATTR0]] {
+; CHECK-SAME: () #[[ATTR1:[0-9]+]] {
 ; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca [[TMP0:%.*]], align 8, addrspace(5)
 ; CHECK-NEXT:    [[CAST:%.*]] = addrspacecast ptr addrspace(5) [[ALLOCA]] to ptr
 ; CHECK-NEXT:    [[ARST:%.*]] = call double @baz(ptr [[CAST]])
 ; CHECK-NEXT:    ret void
 ;
   %alloca = alloca %0, align 8, addrspace(5)
   %cast = addrspacecast ptr addrspace(5) %alloca to ptr
   %arst = call double @baz(ptr %cast)
   ret void
 }
 ;.
-; CHECK: attributes #[[ATTR0]] = { "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR1]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
 ;.

llvm/test/CodeGen/AMDGPU/direct-indirect-call.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals
 ; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -amdgpu-attributor < %s | FileCheck %s
 
 define internal void @indirect() {
 ; CHECK-LABEL: define {{[^@]+}}@indirect
 ; CHECK-SAME: () #[[ATTR0:[0-9]+]] {
 ; CHECK-NEXT:    ret void
 ;
   ret void
 }
 
 define internal void @direct() {
 ; CHECK-LABEL: define {{[^@]+}}@direct
-; CHECK-SAME: () #[[ATTR1:[0-9]+]] {
+; CHECK-SAME: () #[[ATTR0]] {
 ; CHECK-NEXT:    [[FPTR:%.*]] = alloca ptr, align 8, addrspace(5)
 ; CHECK-NEXT:    store ptr @indirect, ptr addrspace(5) [[FPTR]], align 8
 ; CHECK-NEXT:    [[FP:%.*]] = load ptr, ptr addrspace(5) [[FPTR]], align 8
 ; CHECK-NEXT:    call void [[FP]]()
 ; CHECK-NEXT:    ret void
 ;
   %fptr = alloca ptr, addrspace(5)
   store ptr @indirect, ptr addrspace(5) %fptr
   %fp = load ptr, ptr addrspace(5) %fptr
   call void %fp()
   ret void
 }
 
 define amdgpu_kernel void @test_direct_indirect_call() {
 ; CHECK-LABEL: define {{[^@]+}}@test_direct_indirect_call
-; CHECK-SAME: () #[[ATTR1]] {
+; CHECK-SAME: () #[[ATTR0]] {
 ; CHECK-NEXT:    call void @direct()
 ; CHECK-NEXT:    ret void
 ;
   call void @direct()
   ret void
 }
 ;.
 ; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR1]] = { "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
 ;.

llvm/test/CodeGen/AMDGPU/duplicate-attribute-indirect.ll

 }
 
 attributes #0 = { "amdgpu-no-dispatch-id" }
 
 ;.
 ; AKF_GCN: attributes #[[ATTR0]] = { "amdgpu-calls" "amdgpu-no-dispatch-id" "amdgpu-stack-objects" }
 ;.
 ; ATTRIBUTOR_GCN: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
-; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "amdgpu-no-dispatch-id" "uniform-work-group-size"="false" }
+; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" }
 ;.

llvm/test/CodeGen/AMDGPU/indirect-call.ll

 ; GCN-NEXT:     priv = 0
 ; GCN-NEXT:     enable_dx10_clamp = 1
 ; GCN-NEXT:     debug_mode = 0
 ; GCN-NEXT:     enable_ieee_mode = 1
 ; GCN-NEXT:     enable_wgp_mode = 0
 ; GCN-NEXT:     enable_mem_ordered = 0
 ; GCN-NEXT:     enable_fwd_progress = 0
 ; GCN-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
-; GCN-NEXT:     user_sgpr_count = 14
+; GCN-NEXT:     user_sgpr_count = 8
 ; GCN-NEXT:     enable_trap_handler = 0
 ; GCN-NEXT:     enable_sgpr_workgroup_id_x = 1
-; GCN-NEXT:     enable_sgpr_workgroup_id_y = 1
-; GCN-NEXT:     enable_sgpr_workgroup_id_z = 1
+; GCN-NEXT:     enable_sgpr_workgroup_id_y = 0
+; GCN-NEXT:     enable_sgpr_workgroup_id_z = 0
 ; GCN-NEXT:     enable_sgpr_workgroup_info = 0
-; GCN-NEXT:     enable_vgpr_workitem_id = 2
+; GCN-NEXT:     enable_vgpr_workitem_id = 0
 ; GCN-NEXT:     enable_exception_msb = 0
 ; GCN-NEXT:     granulated_lds_size = 0
 ; GCN-NEXT:     enable_exception = 0
 ; GCN-NEXT:     enable_sgpr_private_segment_buffer = 1
-; GCN-NEXT:     enable_sgpr_dispatch_ptr = 1
-; GCN-NEXT:     enable_sgpr_queue_ptr = 1
+; GCN-NEXT:     enable_sgpr_dispatch_ptr = 0
+; GCN-NEXT:     enable_sgpr_queue_ptr = 0
 ; GCN-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
-; GCN-NEXT:     enable_sgpr_dispatch_id = 1
+; GCN-NEXT:     enable_sgpr_dispatch_id = 0
 ; GCN-NEXT:     enable_sgpr_flat_scratch_init = 1
 ; GCN-NEXT:     enable_sgpr_private_segment_size = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
 ; GCN-NEXT:     enable_wavefront_size32 = 0
 ; GCN-NEXT:     enable_ordered_append_gds = 0
 ; GCN-NEXT:     private_element_size = 1
 ; GCN-NEXT:     is_ptr64 = 1
 ; GCN-NEXT:     is_dynamic_callstack = 1
 ; GCN-NEXT:     is_debug_enabled = 0
 ; GCN-NEXT:     is_xnack_enabled = 0
 ; GCN-NEXT:     workitem_private_segment_byte_size = 16384
 ; GCN-NEXT:     workgroup_group_segment_byte_size = 0
 ; GCN-NEXT:     gds_segment_byte_size = 0
-; GCN-NEXT:     kernarg_segment_byte_size = 64
+; GCN-NEXT:     kernarg_segment_byte_size = 4
 ; GCN-NEXT:     workgroup_fbarrier_count = 0
 ; GCN-NEXT:     wavefront_sgpr_count = 68
 ; GCN-NEXT:     workitem_vgpr_count = 42
 ; GCN-NEXT:     reserved_vgpr_first = 0
 ; GCN-NEXT:     reserved_vgpr_count = 0
 ; GCN-NEXT:     reserved_sgpr_first = 0
 ; GCN-NEXT:     reserved_sgpr_count = 0
 ; GCN-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GCN-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GCN-NEXT:     kernarg_segment_alignment = 4
 ; GCN-NEXT:     group_segment_alignment = 4
 ; GCN-NEXT:     private_segment_alignment = 4
 ; GCN-NEXT:     wavefront_size = 6
 ; GCN-NEXT:     call_convention = -1
 ; GCN-NEXT:     runtime_loader_kernel_symbol = 0
 ; GCN-NEXT:    .end_amd_kernel_code_t
 ; GCN-NEXT:  ; %bb.0:
 ; GCN-NEXT:    s_mov_b32 s32, 0
-; GCN-NEXT:    s_mov_b32 flat_scratch_lo, s13
-; GCN-NEXT:    s_add_i32 s12, s12, s17
-; GCN-NEXT:    s_lshr_b32 flat_scratch_hi, s12, 8
-; GCN-NEXT:    s_add_u32 s0, s0, s17
+; GCN-NEXT:    s_mov_b32 flat_scratch_lo, s7
+; GCN-NEXT:    s_add_i32 s6, s6, s9
+; GCN-NEXT:    s_lshr_b32 flat_scratch_hi, s6, 8
+; GCN-NEXT:    s_add_u32 s0, s0, s9
 ; GCN-NEXT:    s_addc_u32 s1, s1, 0
-; GCN-NEXT:    s_mov_b32 s13, s15
-; GCN-NEXT:    s_mov_b32 s12, s14
-; GCN-NEXT:    s_getpc_b64 s[14:15]
-; GCN-NEXT:    s_add_u32 s14, s14, gv.fptr0@rel32@lo+4
-; GCN-NEXT:    s_addc_u32 s15, s15, gv.fptr0@rel32@hi+12
-; GCN-NEXT:    v_lshlrev_b32_e32 v2, 20, v2
-; GCN-NEXT:    s_load_dwordx2 s[18:19], s[14:15], 0x0
-; GCN-NEXT:    s_add_u32 s8, s8, 8
-; GCN-NEXT:    s_addc_u32 s9, s9, 0
-; GCN-NEXT:    v_lshlrev_b32_e32 v1, 10, v1
-; GCN-NEXT:    v_or_b32_e32 v0, v0, v1
-; GCN-NEXT:    v_or_b32_e32 v31, v0, v2
-; GCN-NEXT:    s_mov_b32 s14, s16
+; GCN-NEXT:    s_mov_b32 s12, s8
+; GCN-NEXT:    s_getpc_b64 s[6:7]
+; GCN-NEXT:    s_add_u32 s6, s6, gv.fptr0@rel32@lo+4
+; GCN-NEXT:    s_addc_u32 s7, s7, gv.fptr0@rel32@hi+12
+; GCN-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
+; GCN-NEXT:    s_add_u32 s8, s4, 8
+; GCN-NEXT:    s_addc_u32 s9, s5, 0
+; GCN-NEXT:    v_mov_b32_e32 v31, v0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_swappc_b64 s[30:31], s[18:19]
+; GCN-NEXT:    s_swappc_b64 s[30:31], s[6:7]
 ; GCN-NEXT:    s_endpgm
 ;
 ; GISEL-LABEL: test_indirect_call_sgpr_ptr:
 ; GISEL:         .amd_kernel_code_t
 ; GISEL-NEXT:     amd_code_version_major = 1
 ; GISEL-NEXT:     amd_code_version_minor = 2
 ; GISEL-NEXT:     amd_machine_kind = 1
 ; GISEL-NEXT:     amd_machine_version_major = 7
 ; GISEL-NEXT:     amd_machine_version_minor = 0
 ; GISEL-NEXT:     amd_machine_version_stepping = 0
 ; GISEL-NEXT:     kernel_code_entry_byte_offset = 256
 ; GISEL-NEXT:     kernel_code_prefetch_byte_size = 0
 ; GISEL-NEXT:     granulated_workitem_vgpr_count = 10
 ; GISEL-NEXT:     granulated_wavefront_sgpr_count = 8
 ; GISEL-NEXT:     priority = 0
 ; GISEL-NEXT:     float_mode = 240
 ; GISEL-NEXT:     priv = 0
 ; GISEL-NEXT:     enable_dx10_clamp = 1
 ; GISEL-NEXT:     debug_mode = 0
 ; GISEL-NEXT:     enable_ieee_mode = 1
 ; GISEL-NEXT:     enable_wgp_mode = 0
 ; GISEL-NEXT:     enable_mem_ordered = 0
 ; GISEL-NEXT:     enable_fwd_progress = 0
 ; GISEL-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
-; GISEL-NEXT:     user_sgpr_count = 14
+; GISEL-NEXT:     user_sgpr_count = 8
 ; GISEL-NEXT:     enable_trap_handler = 0
 ; GISEL-NEXT:     enable_sgpr_workgroup_id_x = 1
-; GISEL-NEXT:     enable_sgpr_workgroup_id_y = 1
-; GISEL-NEXT:     enable_sgpr_workgroup_id_z = 1
+; GISEL-NEXT:     enable_sgpr_workgroup_id_y = 0
+; GISEL-NEXT:     enable_sgpr_workgroup_id_z = 0
 ; GISEL-NEXT:     enable_sgpr_workgroup_info = 0
-; GISEL-NEXT:     enable_vgpr_workitem_id = 2
+; GISEL-NEXT:     enable_vgpr_workitem_id = 0
 ; GISEL-NEXT:     enable_exception_msb = 0
 ; GISEL-NEXT:     granulated_lds_size = 0
 ; GISEL-NEXT:     enable_exception = 0
 ; GISEL-NEXT:     enable_sgpr_private_segment_buffer = 1
-; GISEL-NEXT:     enable_sgpr_dispatch_ptr = 1
-; GISEL-NEXT:     enable_sgpr_queue_ptr = 1
+; GISEL-NEXT:     enable_sgpr_dispatch_ptr = 0
+; GISEL-NEXT:     enable_sgpr_queue_ptr = 0
 ; GISEL-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
-; GISEL-NEXT:     enable_sgpr_dispatch_id = 1
+; GISEL-NEXT:     enable_sgpr_dispatch_id = 0
 ; GISEL-NEXT:     enable_sgpr_flat_scratch_init = 1
 ; GISEL-NEXT:     enable_sgpr_private_segment_size = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
 ; GISEL-NEXT:     enable_wavefront_size32 = 0
 ; GISEL-NEXT:     enable_ordered_append_gds = 0
 ; GISEL-NEXT:     private_element_size = 1
 ; GISEL-NEXT:     is_ptr64 = 1
 ; GISEL-NEXT:     is_dynamic_callstack = 1
 ; GISEL-NEXT:     is_debug_enabled = 0
 ; GISEL-NEXT:     is_xnack_enabled = 0
 ; GISEL-NEXT:     workitem_private_segment_byte_size = 16384
 ; GISEL-NEXT:     workgroup_group_segment_byte_size = 0
 ; GISEL-NEXT:     gds_segment_byte_size = 0
-; GISEL-NEXT:     kernarg_segment_byte_size = 64
+; GISEL-NEXT:     kernarg_segment_byte_size = 4
 ; GISEL-NEXT:     workgroup_fbarrier_count = 0
 ; GISEL-NEXT:     wavefront_sgpr_count = 68
 ; GISEL-NEXT:     workitem_vgpr_count = 42
 ; GISEL-NEXT:     reserved_vgpr_first = 0
 ; GISEL-NEXT:     reserved_vgpr_count = 0
 ; GISEL-NEXT:     reserved_sgpr_first = 0
 ; GISEL-NEXT:     reserved_sgpr_count = 0
 ; GISEL-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GISEL-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GISEL-NEXT:     kernarg_segment_alignment = 4
 ; GISEL-NEXT:     group_segment_alignment = 4
 ; GISEL-NEXT:     private_segment_alignment = 4
 ; GISEL-NEXT:     wavefront_size = 6
 ; GISEL-NEXT:     call_convention = -1
 ; GISEL-NEXT:     runtime_loader_kernel_symbol = 0
 ; GISEL-NEXT:    .end_amd_kernel_code_t
 ; GISEL-NEXT:  ; %bb.0:
 ; GISEL-NEXT:    s_mov_b32 s32, 0
-; GISEL-NEXT:    s_mov_b32 flat_scratch_lo, s13
-; GISEL-NEXT:    s_add_i32 s12, s12, s17
-; GISEL-NEXT:    s_lshr_b32 flat_scratch_hi, s12, 8
-; GISEL-NEXT:    s_add_u32 s0, s0, s17
+; GISEL-NEXT:    s_mov_b32 flat_scratch_lo, s7
+; GISEL-NEXT:    s_add_i32 s6, s6, s9
+; GISEL-NEXT:    s_lshr_b32 flat_scratch_hi, s6, 8
+; GISEL-NEXT:    s_add_u32 s0, s0, s9
 ; GISEL-NEXT:    s_addc_u32 s1, s1, 0
-; GISEL-NEXT:    s_mov_b32 s13, s15
-; GISEL-NEXT:    s_mov_b32 s12, s14
-; GISEL-NEXT:    s_getpc_b64 s[14:15]
-; GISEL-NEXT:    s_add_u32 s14, s14, gv.fptr0@rel32@lo+4
-; GISEL-NEXT:    s_addc_u32 s15, s15, gv.fptr0@rel32@hi+12
-; GISEL-NEXT:    v_lshlrev_b32_e32 v1, 10, v1
-; GISEL-NEXT:    s_load_dwordx2 s[18:19], s[14:15], 0x0
-; GISEL-NEXT:    v_or_b32_e32 v0, v0, v1
-; GISEL-NEXT:    s_add_u32 s8, s8, 8
-; GISEL-NEXT:    s_addc_u32 s9, s9, 0
-; GISEL-NEXT:    v_lshlrev_b32_e32 v1, 20, v2
-; GISEL-NEXT:    v_or_b32_e32 v31, v0, v1
-; GISEL-NEXT:    s_mov_b32 s14, s16
+; GISEL-NEXT:    s_getpc_b64 s[6:7]
+; GISEL-NEXT:    s_add_u32 s6, s6, gv.fptr0@rel32@lo+4
+; GISEL-NEXT:    s_addc_u32 s7, s7, gv.fptr0@rel32@hi+12
+; GISEL-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
+; GISEL-NEXT:    s_add_u32 s4, s4, 8
+; GISEL-NEXT:    s_addc_u32 s5, s5, 0
+; GISEL-NEXT:    s_mov_b32 s12, s8
+; GISEL-NEXT:    s_mov_b64 s[8:9], s[4:5]
+; GISEL-NEXT:    v_mov_b32_e32 v31, v0
 ; GISEL-NEXT:    s_waitcnt lgkmcnt(0)
-; GISEL-NEXT:    s_swappc_b64 s[30:31], s[18:19]
+; GISEL-NEXT:    s_swappc_b64 s[30:31], s[6:7]
 ; GISEL-NEXT:    s_endpgm
   %fptr = load ptr, ptr addrspace(4) @gv.fptr0
   call void %fptr()
   ret void
 }
 
 define amdgpu_kernel void @test_indirect_call_sgpr_ptr_arg(i8) {
 ; GCN-LABEL: test_indirect_call_sgpr_ptr_arg:

 ; GCN-NEXT:     priv = 0
 ; GCN-NEXT:     enable_dx10_clamp = 1
 ; GCN-NEXT:     debug_mode = 0
 ; GCN-NEXT:     enable_ieee_mode = 1
 ; GCN-NEXT:     enable_wgp_mode = 0
 ; GCN-NEXT:     enable_mem_ordered = 0
 ; GCN-NEXT:     enable_fwd_progress = 0
 ; GCN-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
-; GCN-NEXT:     user_sgpr_count = 14
+; GCN-NEXT:     user_sgpr_count = 8
 ; GCN-NEXT:     enable_trap_handler = 0
 ; GCN-NEXT:     enable_sgpr_workgroup_id_x = 1
-; GCN-NEXT:     enable_sgpr_workgroup_id_y = 1
-; GCN-NEXT:     enable_sgpr_workgroup_id_z = 1
+; GCN-NEXT:     enable_sgpr_workgroup_id_y = 0
+; GCN-NEXT:     enable_sgpr_workgroup_id_z = 0
 ; GCN-NEXT:     enable_sgpr_workgroup_info = 0
-; GCN-NEXT:     enable_vgpr_workitem_id = 2
+; GCN-NEXT:     enable_vgpr_workitem_id = 0
 ; GCN-NEXT:     enable_exception_msb = 0
 ; GCN-NEXT:     granulated_lds_size = 0
 ; GCN-NEXT:     enable_exception = 0
 ; GCN-NEXT:     enable_sgpr_private_segment_buffer = 1
-; GCN-NEXT:     enable_sgpr_dispatch_ptr = 1
-; GCN-NEXT:     enable_sgpr_queue_ptr = 1
+; GCN-NEXT:     enable_sgpr_dispatch_ptr = 0
+; GCN-NEXT:     enable_sgpr_queue_ptr = 0
 ; GCN-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
-; GCN-NEXT:     enable_sgpr_dispatch_id = 1
+; GCN-NEXT:     enable_sgpr_dispatch_id = 0
 ; GCN-NEXT:     enable_sgpr_flat_scratch_init = 1
 ; GCN-NEXT:     enable_sgpr_private_segment_size = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
 ; GCN-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
 ; GCN-NEXT:     enable_wavefront_size32 = 0
 ; GCN-NEXT:     enable_ordered_append_gds = 0
 ; GCN-NEXT:     private_element_size = 1
 ; GCN-NEXT:     is_ptr64 = 1
 ; GCN-NEXT:     is_dynamic_callstack = 1
 ; GCN-NEXT:     is_debug_enabled = 0
 ; GCN-NEXT:     is_xnack_enabled = 0
 ; GCN-NEXT:     workitem_private_segment_byte_size = 16384
 ; GCN-NEXT:     workgroup_group_segment_byte_size = 0
 ; GCN-NEXT:     gds_segment_byte_size = 0
-; GCN-NEXT:     kernarg_segment_byte_size = 64
+; GCN-NEXT:     kernarg_segment_byte_size = 4
 ; GCN-NEXT:     workgroup_fbarrier_count = 0
 ; GCN-NEXT:     wavefront_sgpr_count = 68
 ; GCN-NEXT:     workitem_vgpr_count = 42
 ; GCN-NEXT:     reserved_vgpr_first = 0
 ; GCN-NEXT:     reserved_vgpr_count = 0
 ; GCN-NEXT:     reserved_sgpr_first = 0
 ; GCN-NEXT:     reserved_sgpr_count = 0
 ; GCN-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GCN-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GCN-NEXT:     kernarg_segment_alignment = 4
 ; GCN-NEXT:     group_segment_alignment = 4
 ; GCN-NEXT:     private_segment_alignment = 4
 ; GCN-NEXT:     wavefront_size = 6
 ; GCN-NEXT:     call_convention = -1
 ; GCN-NEXT:     runtime_loader_kernel_symbol = 0
 ; GCN-NEXT:    .end_amd_kernel_code_t
 ; GCN-NEXT:  ; %bb.0:
 ; GCN-NEXT:    s_mov_b32 s32, 0
-; GCN-NEXT:    s_mov_b32 flat_scratch_lo, s13
-; GCN-NEXT:    s_add_i32 s12, s12, s17
-; GCN-NEXT:    s_lshr_b32 flat_scratch_hi, s12, 8
-; GCN-NEXT:    s_add_u32 s0, s0, s17
+; GCN-NEXT:    s_mov_b32 flat_scratch_lo, s7
+; GCN-NEXT:    s_add_i32 s6, s6, s9
+; GCN-NEXT:    s_lshr_b32 flat_scratch_hi, s6, 8
+; GCN-NEXT:    s_add_u32 s0, s0, s9
 ; GCN-NEXT:    s_addc_u32 s1, s1, 0
-; GCN-NEXT:    s_mov_b32 s13, s15
-; GCN-NEXT:    s_mov_b32 s12, s14
-; GCN-NEXT:    s_getpc_b64 s[14:15]
-; GCN-NEXT:    s_add_u32 s14, s14, gv.fptr1@rel32@lo+4
-; GCN-NEXT:    s_addc_u32 s15, s15, gv.fptr1@rel32@hi+12
-; GCN-NEXT:    v_lshlrev_b32_e32 v2, 20, v2
-; GCN-NEXT:    v_lshlrev_b32_e32 v1, 10, v1
-; GCN-NEXT:    s_load_dwordx2 s[18:19], s[14:15], 0x0
-; GCN-NEXT:    s_add_u32 s8, s8, 8
-; GCN-NEXT:    s_addc_u32 s9, s9, 0
-; GCN-NEXT:    v_or_b32_e32 v0, v0, v1
-; GCN-NEXT:    v_or_b32_e32 v31, v0, v2
+; GCN-NEXT:    s_mov_b32 s12, s8
+; GCN-NEXT:    v_mov_b32_e32 v31, v0
+; GCN-NEXT:    s_getpc_b64 s[6:7]
+; GCN-NEXT:    s_add_u32 s6, s6, gv.fptr1@rel32@lo+4
+; GCN-NEXT:    s_addc_u32 s7, s7, gv.fptr1@rel32@hi+12
+; GCN-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
+; GCN-NEXT:    s_add_u32 s8, s4, 8
+; GCN-NEXT:    s_addc_u32 s9, s5, 0
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0x7b
-; GCN-NEXT:    s_mov_b32 s14, s16
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_swappc_b64 s[30:31], s[18:19]
+; GCN-NEXT:    s_swappc_b64 s[30:31], s[6:7]
 ; GCN-NEXT:    s_endpgm
 ;
 ; GISEL-LABEL: test_indirect_call_sgpr_ptr_arg:
 ; GISEL:         .amd_kernel_code_t
 ; GISEL-NEXT:     amd_code_version_major = 1
 ; GISEL-NEXT:     amd_code_version_minor = 2
 ; GISEL-NEXT:     amd_machine_kind = 1
 ; GISEL-NEXT:     amd_machine_version_major = 7
 ; GISEL-NEXT:     amd_machine_version_minor = 0
 ; GISEL-NEXT:     amd_machine_version_stepping = 0
 ; GISEL-NEXT:     kernel_code_entry_byte_offset = 256
 ; GISEL-NEXT:     kernel_code_prefetch_byte_size = 0
 ; GISEL-NEXT:     granulated_workitem_vgpr_count = 10
 ; GISEL-NEXT:     granulated_wavefront_sgpr_count = 8
 ; GISEL-NEXT:     priority = 0
 ; GISEL-NEXT:     float_mode = 240
 ; GISEL-NEXT:     priv = 0
 ; GISEL-NEXT:     enable_dx10_clamp = 1
 ; GISEL-NEXT:     debug_mode = 0
 ; GISEL-NEXT:     enable_ieee_mode = 1
 ; GISEL-NEXT:     enable_wgp_mode = 0
 ; GISEL-NEXT:     enable_mem_ordered = 0
 ; GISEL-NEXT:     enable_fwd_progress = 0
 ; GISEL-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
-; GISEL-NEXT:     user_sgpr_count = 14
+; GISEL-NEXT:     user_sgpr_count = 8
 ; GISEL-NEXT:     enable_trap_handler = 0
 ; GISEL-NEXT:     enable_sgpr_workgroup_id_x = 1
-; GISEL-NEXT:     enable_sgpr_workgroup_id_y = 1
-; GISEL-NEXT:     enable_sgpr_workgroup_id_z = 1
+; GISEL-NEXT:     enable_sgpr_workgroup_id_y = 0
+; GISEL-NEXT:     enable_sgpr_workgroup_id_z = 0
 ; GISEL-NEXT:     enable_sgpr_workgroup_info = 0
-; GISEL-NEXT:     enable_vgpr_workitem_id = 2
+; GISEL-NEXT:     enable_vgpr_workitem_id = 0
 ; GISEL-NEXT:     enable_exception_msb = 0
 ; GISEL-NEXT:     granulated_lds_size = 0
 ; GISEL-NEXT:     enable_exception = 0
 ; GISEL-NEXT:     enable_sgpr_private_segment_buffer = 1
-; GISEL-NEXT:     enable_sgpr_dispatch_ptr = 1
-; GISEL-NEXT:     enable_sgpr_queue_ptr = 1
+; GISEL-NEXT:     enable_sgpr_dispatch_ptr = 0
+; GISEL-NEXT:     enable_sgpr_queue_ptr = 0
 ; GISEL-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
-; GISEL-NEXT:     enable_sgpr_dispatch_id = 1
+; GISEL-NEXT:     enable_sgpr_dispatch_id = 0
 ; GISEL-NEXT:     enable_sgpr_flat_scratch_init = 1
 ; GISEL-NEXT:     enable_sgpr_private_segment_size = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
 ; GISEL-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
 ; GISEL-NEXT:     enable_wavefront_size32 = 0
 ; GISEL-NEXT:     enable_ordered_append_gds = 0
 ; GISEL-NEXT:     private_element_size = 1
 ; GISEL-NEXT:     is_ptr64 = 1
 ; GISEL-NEXT:     is_dynamic_callstack = 1
 ; GISEL-NEXT:     is_debug_enabled = 0
 ; GISEL-NEXT:     is_xnack_enabled = 0
 ; GISEL-NEXT:     workitem_private_segment_byte_size = 16384
 ; GISEL-NEXT:     workgroup_group_segment_byte_size = 0
 ; GISEL-NEXT:     gds_segment_byte_size = 0
-; GISEL-NEXT:     kernarg_segment_byte_size = 64
+; GISEL-NEXT:     kernarg_segment_byte_size = 4
 ; GISEL-NEXT:     workgroup_fbarrier_count = 0
 ; GISEL-NEXT:     wavefront_sgpr_count = 68
 ; GISEL-NEXT:     workitem_vgpr_count = 42
 ; GISEL-NEXT:     reserved_vgpr_first = 0
 ; GISEL-NEXT:     reserved_vgpr_count = 0
 ; GISEL-NEXT:     reserved_sgpr_first = 0
 ; GISEL-NEXT:     reserved_sgpr_count = 0
 ; GISEL-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GISEL-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GISEL-NEXT:     kernarg_segment_alignment = 4
 ; GISEL-NEXT:     group_segment_alignment = 4
 ; GISEL-NEXT:     private_segment_alignment = 4
 ; GISEL-NEXT:     wavefront_size = 6
 ; GISEL-NEXT:     call_convention = -1
 ; GISEL-NEXT:     runtime_loader_kernel_symbol = 0
 ; GISEL-NEXT:    .end_amd_kernel_code_t
 ; GISEL-NEXT:  ; %bb.0:
 ; GISEL-NEXT:    s_mov_b32 s32, 0
-; GISEL-NEXT:    s_mov_b32 flat_scratch_lo, s13
-; GISEL-NEXT:    s_add_i32 s12, s12, s17
-; GISEL-NEXT:    s_lshr_b32 flat_scratch_hi, s12, 8
-; GISEL-NEXT:    s_add_u32 s0, s0, s17
+; GISEL-NEXT:    s_mov_b32 flat_scratch_lo, s7
+; GISEL-NEXT:    s_add_i32 s6, s6, s9
+; GISEL-NEXT:    s_lshr_b32 flat_scratch_hi, s6, 8
+; GISEL-NEXT:    s_add_u32 s0, s0, s9
 ; GISEL-NEXT:    s_addc_u32 s1, s1, 0
-; GISEL-NEXT:    s_mov_b32 s13, s15
-; GISEL-NEXT:    s_mov_b32 s12, s14
-; GISEL-NEXT:    s_getpc_b64 s[14:15]
-; GISEL-NEXT:    s_add_u32 s14, s14, gv.fptr1@rel32@lo+4
-; GISEL-NEXT:    s_addc_u32 s15, s15, gv.fptr1@rel32@hi+12
-; GISEL-NEXT:    v_lshlrev_b32_e32 v1, 10, v1
-; GISEL-NEXT:    v_lshlrev_b32_e32 v2, 20, v2
-; GISEL-NEXT:    s_load_dwordx2 s[18:19], s[14:15], 0x0
-; GISEL-NEXT:    v_or_b32_e32 v0, v0, v1
-; GISEL-NEXT:    s_add_u32 s8, s8, 8
-; GISEL-NEXT:    s_addc_u32 s9, s9, 0
-; GISEL-NEXT:    v_or_b32_e32 v31, v0, v2
+; GISEL-NEXT:    v_mov_b32_e32 v31, v0
+; GISEL-NEXT:    s_getpc_b64 s[6:7]
+; GISEL-NEXT:    s_add_u32 s6, s6, gv.fptr1@rel32@lo+4
+; GISEL-NEXT:    s_addc_u32 s7, s7, gv.fptr1@rel32@hi+12
+; GISEL-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
+; GISEL-NEXT:    s_add_u32 s4, s4, 8
+; GISEL-NEXT:    s_addc_u32 s5, s5, 0
 ; GISEL-NEXT:    v_mov_b32_e32 v0, 0x7b
-; GISEL-NEXT:    s_mov_b32 s14, s16
+; GISEL-NEXT:    s_mov_b32 s12, s8
+; GISEL-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GISEL-NEXT:    s_waitcnt lgkmcnt(0)
-; GISEL-NEXT:    s_swappc_b64 s[30:31], s[18:19]
+; GISEL-NEXT:    s_swappc_b64 s[30:31], s[6:7]
 ; GISEL-NEXT:    s_endpgm
   %fptr = load ptr, ptr addrspace(4) @gv.fptr1
   call void %fptr(i32 123)
   ret void
 }
 
 define void @test_indirect_call_vgpr_ptr(ptr %fptr) {
 ; GCN-LABEL: test_indirect_call_vgpr_ptr:

llvm/test/CodeGen/AMDGPU/simple-indirect-call.ll

 ; ATTRIBUTOR_GCN-NEXT:    store ptr @indirect, ptr [[FPTR_CAST]], align 8
 ; ATTRIBUTOR_GCN-NEXT:    [[FP:%.*]] = load ptr, ptr [[FPTR_CAST]], align 8
 ; ATTRIBUTOR_GCN-NEXT:    call void [[FP]]()
 ; ATTRIBUTOR_GCN-NEXT:    ret void
 ;
 ; GFX9-LABEL: test_simple_indirect_call:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x4
-; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s12, s17
-; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s13, 0
-; GFX9-NEXT:    s_add_u32 s0, s0, s17
+; GFX9-NEXT:    s_add_u32 flat_scratch_lo, s6, s9
+; GFX9-NEXT:    s_addc_u32 flat_scratch_hi, s7, 0
+; GFX9-NEXT:    s_add_u32 s0, s0, s9
 ; GFX9-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 16
 ; GFX9-NEXT:    s_mul_i32 s4, s4, s5
 ; GFX9-NEXT:    v_mul_lo_u32 v0, s4, v0
 ; GFX9-NEXT:    s_getpc_b64 s[6:7]
 ; GFX9-NEXT:    s_add_u32 s6, s6, indirect@rel32@lo+4
 ; GFX9-NEXT:    s_addc_u32 s7, s7, indirect@rel32@hi+12
; GFX9-NEXT:    s_endpgm
   call void %fp()
   ret void
 }
 
 ;.
 ; AKF_GCN: attributes #[[ATTR0]] = { "amdgpu-calls" "amdgpu-stack-objects" }
 ;.
 ; ATTRIBUTOR_GCN: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
-; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "uniform-work-group-size"="false" }
+; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" }
 ;.