AMDGPU: Use module level register maximums for unknown callees

[arsenm]

Summary

Compute the theoretical register budget based on the IR function
signature/attributes, and use the global maximum register budgets for
unknown callees.

This should fix the kernel reported register usage in the presence of
indirect calls. The previous fix in
2b08f6af62afbf32e89a6a392dbafa92c62f7bdf was incorrect becauset it was
only taking the maximum in the known call graph, and missing something
that was either outside of it or codegened later.

This fixes a second case I discovered where calls to aliases also did
not work as expected. CallGraphAnalysis misses these, so functions
called through aliases were not codegened ahead of callers as
expected. CallGraphAnalysis should probably be fixed to understand
this case, and there's likely a bug with IPRA here. This fixes
numerous failures in the conformance test at -O0.

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

This is over-approximating the vgpr_count whenever an indirect call is involved, which is quite a performance hit.

Can we switch AMDGPUResourceUsageAnalysis to a ModulePass and run propagateIndirectCallRegisterUsage at the end, so that all functions with indirect calls will get the maximum VGPR count of all functions in the module?
(As opposed to max VGPR count of the SCC that is used currently, which I did not intend.)

[arsenm]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

I'd like to move switching to a module pass into a follow up patch. I'm a bit afraid of unintended side effects by switching to a module pass. We're already paying a compile time cost by using SCC codegen, and module passes will be worse

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

If the over-approximation goes in, we’ll have to revert it in our graphics branch, otherwise any benchmarks or optimization work that is going on would be meaningless (we had to locally revert D103636 for that reason).
So, I’d prefer it if we could fix this in one push, if you think that’s possible.

[arsenm]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

This is just completely broken as is, any benchmarks are working by accident

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

The current graphics use-case is not affected by this bug. We’re only compiling single functions per LLVM module and finding the maximum register usage and linking is done by the loader (PAL in this case).

[arsenm]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

So then it also wouldn't be impacted by this change?

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

The compiled modules contain only a single function, but the function contains indirect functions calls. So it hits the code path here that uses getMaxNumVGPRs(F) to approximate the register usage.
The VGPR usage for the compiled function should just stay as it is, even if it contains an indirect call, because there is no other function (in the module) that could be called.

I tested this patch on a pipeline and the reported VGPR usage goes from 140 VGPRs to 256 VGPRs, noticeably reducing the occupancy.

As said, I’d prefer if this does not go in without an accompanying patch that makes the VGPR usage accurate again.
If you think that’s unreasonable, I’ll unblock this patch and revert it in our graphics branch until the follow up patch.

[arsenm]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

I will push them both at the same time, but I'd like to defensively keep this and D117504 as separate commits in case something goes wrong

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

184

Thank you, I’ll have a look tomorrow.

[sebastian-ne]

Forgot to accept as amdgpu last time.

[arsenm]

935abab65cafb509f60e76bd7255dfe03befde85

llvm/lib/Target/AMDGPU/AMDGPUResourceUsageAnalysis.h

 #ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPURESOURCEUSAGEANALYSIS_H
 #define LLVM_LIB_TARGET_AMDGPU_AMDGPURESOURCEUSAGEANALYSIS_H
 
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 
 namespace llvm {
 
+class GCNTargetMachine;
 class GCNSubtarget;
 class MachineFunction;
 class TargetMachine;
 
 struct AMDGPUResourceUsageAnalysis : public CallGraphSCCPass {
   static char ID;
 
 public:
public:
 
   const SIFunctionResourceInfo &getResourceInfo(const Function *F) const {
     auto Info = CallGraphResourceInfo.find(F);
     assert(Info != CallGraphResourceInfo.end() &&
            "Failed to find resource info for function");
     return Info->getSecond();
   }
 
+  const SIFunctionResourceInfo &getWorstCaseResourceInfo(const Module &M);
+
 private:
-  SIFunctionResourceInfo analyzeResourceUsage(const MachineFunction &MF,
-                                              const TargetMachine &TM) const;
-  void propagateIndirectCallRegisterUsage();
+  void computeWorstCaseModuleRegisterUsage(const Module &M);
+
+  SIFunctionResourceInfo analyzeResourceUsage(const MachineFunction &MF);
 
+  const GCNTargetMachine *TM = nullptr;
   DenseMap<const Function *, SIFunctionResourceInfo> CallGraphResourceInfo;
+  Optional<SIFunctionResourceInfo> ModuleWorstCaseInfo;
 };
 } // namespace llvm
 #endif // LLVM_LIB_TARGET_AMDGPU_AMDGPURESOURCEUSAGEANALYSIS_H

llvm/lib/Target/AMDGPU/AMDGPUResourceUsageAnalysis.cpp

 /// hardware-entrypoints. Therefore the register usage of functions with
 /// indirect calls is estimated as the maximum of all non-entrypoint functions
 /// in the module.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPUResourceUsageAnalysis.h"
 #include "AMDGPU.h"
+#include "AMDGPUTargetMachine.h"
 #include "GCNSubtarget.h"
 #include "SIMachineFunctionInfo.h"
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/Target/TargetMachine.h"
 
int32_t AMDGPUResourceUsageAnalysis::SIFunctionResourceInfo::getTotalNumVGPRs(
   return getTotalNumVGPRs(ST, NumAGPR, NumVGPR);
 }
 
 bool AMDGPUResourceUsageAnalysis::runOnSCC(CallGraphSCC &SCC) {
   auto *TPC = getAnalysisIfAvailable<TargetPassConfig>();
   if (!TPC)
     return false;
 
-  const TargetMachine &TM = TPC->getTM<TargetMachine>();
-  bool HasIndirectCall = false;
+  TM = static_cast<const GCNTargetMachine *>(&TPC->getTM<TargetMachine>());
 
   for (CallGraphNode *I : SCC) {
     Function *F = I->getFunction();
     if (!F || F->isDeclaration())
       continue;
 
     MachineModuleInfo &MMI =
         getAnalysis<MachineModuleInfoWrapperPass>().getMMI();
     MachineFunction &MF = MMI.getOrCreateMachineFunction(*F);
 
     auto CI = CallGraphResourceInfo.insert(
         std::make_pair(&MF.getFunction(), SIFunctionResourceInfo()));
     SIFunctionResourceInfo &Info = CI.first->second;
     assert(CI.second && "should only be called once per function");
-    Info = analyzeResourceUsage(MF, TM);
-    HasIndirectCall |= Info.HasIndirectCall;
+    Info = analyzeResourceUsage(MF);
   }
 
-  if (HasIndirectCall)
-    propagateIndirectCallRegisterUsage();
-
   return false;
 }
 
 AMDGPUResourceUsageAnalysis::SIFunctionResourceInfo
-AMDGPUResourceUsageAnalysis::analyzeResourceUsage(
-    const MachineFunction &MF, const TargetMachine &TM) const {
+AMDGPUResourceUsageAnalysis::analyzeResourceUsage(const MachineFunction &MF) {
   SIFunctionResourceInfo Info;
 
   const SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();
   const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
   const MachineFrameInfo &FrameInfo = MF.getFrameInfo();
   const MachineRegisterInfo &MRI = MF.getRegInfo();
   const SIInstrInfo *TII = ST.getInstrInfo();
   const SIRegisterInfo &TRI = TII->getRegisterInfo();
for (const MachineInstr &MI : MBB) {
           }
         }
 
         if (IsIndirect || I == CallGraphResourceInfo.end()) {
           CalleeFrameSize =
               std::max(CalleeFrameSize,
                        static_cast<uint64_t>(AssumedStackSizeForExternalCall));
 
+          const SIFunctionResourceInfo &WorstCase =
+              getWorstCaseResourceInfo(*MF.getFunction().getParent());
+          MaxSGPR = std::max(WorstCase.NumExplicitSGPR - 1, MaxSGPR);
+          MaxVGPR = std::max(WorstCase.NumVGPR - 1, MaxVGPR);
+          MaxAGPR = std::max(WorstCase.NumAGPR - 1, MaxAGPR);
+
           // Register usage of indirect calls gets handled later
           Info.UsesVCC = true;
-          Info.UsesFlatScratch = ST.hasFlatAddressSpace();
+          Info.UsesFlatScratch |=
+              WorstCase.UsesFlatScratch && ST.hasFlatAddressSpace();
           Info.HasDynamicallySizedStack = true;
           Info.HasIndirectCall = true;
         } else {
           // We force CodeGen to run in SCC order, so the callee's register
           // usage etc. should be the cumulative usage of all callees.
           MaxSGPR = std::max(I->second.NumExplicitSGPR - 1, MaxSGPR);
           MaxVGPR = std::max(I->second.NumVGPR - 1, MaxVGPR);
           MaxAGPR = std::max(I->second.NumAGPR - 1, MaxAGPR);
AMDGPUResourceUsageAnalysis::analyzeResourceUsage(const MachineFunction &MF) {
   Info.NumExplicitSGPR = MaxSGPR + 1;
   Info.NumVGPR = MaxVGPR + 1;
   Info.NumAGPR = MaxAGPR + 1;
   Info.PrivateSegmentSize += CalleeFrameSize;
 
   return Info;
 }
 
-void AMDGPUResourceUsageAnalysis::propagateIndirectCallRegisterUsage() {
-  // Collect the maximum number of registers from non-hardware-entrypoints.
-  // All these functions are potential targets for indirect calls.
-  int32_t NonKernelMaxSGPRs = 0;
-  int32_t NonKernelMaxVGPRs = 0;
-  int32_t NonKernelMaxAGPRs = 0;
-
-  for (const auto &I : CallGraphResourceInfo) {
-    if (!AMDGPU::isEntryFunctionCC(I.getFirst()->getCallingConv())) {
-      auto &Info = I.getSecond();
-      NonKernelMaxSGPRs = std::max(NonKernelMaxSGPRs, Info.NumExplicitSGPR);
-      NonKernelMaxVGPRs = std::max(NonKernelMaxVGPRs, Info.NumVGPR);
-      NonKernelMaxAGPRs = std::max(NonKernelMaxAGPRs, Info.NumAGPR);
-    }
-  }
+const AMDGPUResourceUsageAnalysis::SIFunctionResourceInfo &
+AMDGPUResourceUsageAnalysis::getWorstCaseResourceInfo(const Module &M) {
+  if (ModuleWorstCaseInfo)
+    return *ModuleWorstCaseInfo;
+
+  computeWorstCaseModuleRegisterUsage(M);
+  return *ModuleWorstCaseInfo;
+}
+
+/// Find the worst case register usage for all callable functions in the module,
+/// assuming all reachable functions are defined in the current module.
+void AMDGPUResourceUsageAnalysis::computeWorstCaseModuleRegisterUsage(
+    const Module &M) {
+  assert(!ModuleWorstCaseInfo);
+  ModuleWorstCaseInfo = SIFunctionResourceInfo();
+  ModuleWorstCaseInfo->UsesVCC = true;
+  ModuleWorstCaseInfo->HasDynamicallySizedStack = true;
+  ModuleWorstCaseInfo->HasRecursion = true;
+  ModuleWorstCaseInfo->HasIndirectCall = true;
 
-  // Add register usage for functions with indirect calls.
-  // For calls to unknown functions, we assume the maximum register usage of
-  // all non-hardware-entrypoints in the current module.
-  for (auto &I : CallGraphResourceInfo) {
-    auto &Info = I.getSecond();
-    if (Info.HasIndirectCall) {
-      Info.NumExplicitSGPR = std::max(Info.NumExplicitSGPR, NonKernelMaxSGPRs);
-      Info.NumVGPR = std::max(Info.NumVGPR, NonKernelMaxVGPRs);
-      Info.NumAGPR = std::max(Info.NumAGPR, NonKernelMaxAGPRs);
+  for (const Function &F : M) {
+    if (F.isIntrinsic())
+      continue;
+
+    if (AMDGPU::isEntryFunctionCC(F.getCallingConv()))
+      continue;
+
+    const GCNSubtarget &ST = TM->getSubtarget<GCNSubtarget>(F);
+    const int32_t MaxVGPR = ST.getMaxNumVGPRs(F);
+    const int32_t MaxSGPR = ST.getMaxNumSGPRs(F);
+
+    ModuleWorstCaseInfo->NumVGPR =
+        std::max(ModuleWorstCaseInfo->NumVGPR, MaxVGPR);
+
+    if (ST.hasMAIInsts()) {
+      const int32_t MaxAGPR = ST.getMaxNumAGPRs(F);
+      ModuleWorstCaseInfo->NumAGPR =
+          std::max(ModuleWorstCaseInfo->NumAGPR, MaxAGPR);
     }
+
+    ModuleWorstCaseInfo->NumExplicitSGPR =
+        std::max(ModuleWorstCaseInfo->NumExplicitSGPR, MaxSGPR);
+
+    ModuleWorstCaseInfo->UsesFlatScratch |= ST.hasFlatAddressSpace();
   }
 }

llvm/lib/Target/AMDGPU/GCNSubtarget.h

public:
   /// requested using "amdgpu-num-vgpr" attribute attached to function \p F.
   ///
   /// \returns Value that meets number of waves per execution unit requirement
   /// if explicitly requested value cannot be converted to integer, violates
   /// subtarget's specifications, or does not meet number of waves per execution
   /// unit requirement.
   unsigned getMaxNumVGPRs(const Function &F) const;
 
+  unsigned getMaxNumAGPRs(const Function &F) const {
+    return getMaxNumVGPRs(F);
+  }
+
   /// \returns Maximum number of VGPRs that meets number of waves per execution
   /// unit requirement for function \p MF, or number of VGPRs explicitly
   /// requested using "amdgpu-num-vgpr" attribute attached to function \p MF.
   ///
   /// \returns Value that meets number of waves per execution unit requirement
   /// if explicitly requested value cannot be converted to integer, violates
   /// subtarget's specifications, or does not meet number of waves per execution
   /// unit requirement.

llvm/test/CodeGen/AMDGPU/agpr-register-count.ll

 bb:
   call void @func_32_agprs() #0
   ret void
 }
 
 declare void @undef_func()
 
 ; GCN-LABEL: {{^}}kernel_call_undef_func:
-; GFX908: .amdhsa_next_free_vgpr 32
-; GFX90A: .amdhsa_next_free_vgpr 64
-; GFX90A: .amdhsa_accum_offset 32
+; GFX908: .amdhsa_next_free_vgpr 128
+; GFX90A: .amdhsa_next_free_vgpr 512
+; GFX90A: .amdhsa_accum_offset 256
 ; GCN908: NumVgprs: 128
+; GCN908: NumAgprs: 128
 ; GCN90A: NumVgprs: 256
-; GCN:    NumAgprs: 32
-; GFX908: TotalNumVgprs: 32
-; GFX90A: TotalNumVgprs: 64
-; GFX908: VGPRBlocks: 7
-; GFX90A: VGPRBlocks: 7
-; GFX908: NumVGPRsForWavesPerEU: 32
-; GFX90A: NumVGPRsForWavesPerEU: 64
-; GFX90A: AccumOffset: 32
-; GFX908: Occupancy: 8
-; GFX90A: Occupancy: 8
-; GFX90A: COMPUTE_PGM_RSRC3_GFX90A:ACCUM_OFFSET: 7
+; GCN90A: NumAgprs: 256
+; GFX908: TotalNumVgprs: 128
+; GFX90A: TotalNumVgprs: 512
+; GFX908: VGPRBlocks: 31
+; GFX90A: VGPRBlocks: 63
+; GFX908: NumVGPRsForWavesPerEU: 128
+; GFX90A: NumVGPRsForWavesPerEU: 512
+; GFX90A: AccumOffset: 256
+; GFX908: Occupancy: 2
+; GFX90A: Occupancy: 1
+; GFX90A: COMPUTE_PGM_RSRC3_GFX90A:ACCUM_OFFSET: 63
 define amdgpu_kernel void @kernel_call_undef_func() #0 {
 bb:
   call void @undef_func()
   ret void
 }
 
 attributes #0 = { nounwind noinline "amdgpu-flat-work-group-size"="1,512" }

llvm/test/CodeGen/AMDGPU/amdpal-callable.ll

define amdgpu_gfx float @simple_lds_recurse(float %arg0) #0 {
   %res = call amdgpu_gfx float @simple_lds_recurse(float %val)
   ret float %res
 }
 
 attributes #0 = { nounwind }
 
 ; GCN: amdpal.pipelines:
 ; GCN-NEXT:  - .registers:
-; SDAG-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf01ca{{$}}
-; GISEL-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf01ce{{$}}
+; SDAG-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf03cf{{$}}
+; GISEL-NEXT:      0x2e12 (COMPUTE_PGM_RSRC1): 0xaf03cf{{$}}
 ; GCN-NEXT:      0x2e13 (COMPUTE_PGM_RSRC2): 0x8001{{$}}
 ; GCN-NEXT:    .shader_functions:
 ; GCN-NEXT:      dynamic_stack:
 ; GCN-NEXT:        .lds_size:       0{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x28{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
 ; SDAG-NEXT:        .vgpr_count:     0x2{{$}}
 ; GISEL-NEXT:        .vgpr_count:     0x3{{$}}

 ; GCN-NEXT:        .vgpr_count:     0x1{{$}}
 ; GCN-NEXT:      no_stack_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x26{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x2{{$}}
 ; GCN-NEXT:      no_stack_extern_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
-; GFX8-NEXT:        .sgpr_count:     0x28{{$}}
-; GFX9-NEXT:        .sgpr_count:     0x2c{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x6c{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
-; GCN-NEXT:        .vgpr_count:     0x29{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      no_stack_extern_call_many_args:
 ; GCN-NEXT:        .lds_size:       0{{$}}
-; GFX8-NEXT:        .sgpr_count:     0x28{{$}}
-; GFX9-NEXT:        .sgpr_count:     0x2c{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x6c{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x90{{$}}
-; SDAG-NEXT:        .vgpr_count:     0x2a{{$}}
-; GISEL-NEXT:        .vgpr_count:     0x34{{$}}
+; SDAG-NEXT:        .vgpr_count:     0x40{{$}}
+; GISEL-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      no_stack_indirect_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
-; GFX8-NEXT:        .sgpr_count:     0x28{{$}}
-; GFX9-NEXT:        .sgpr_count:     0x2c{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x6c{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
-; SDAG-NEXT:        .vgpr_count:     0x2a{{$}}
-; GISEL-NEXT:        .vgpr_count:     0x34{{$}}
+; SDAG-NEXT:        .vgpr_count:     0x40{{$}}
+; GISEL-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_lds:
 ; GCN-NEXT:        .lds_size:       0x100{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x20{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x1{{$}}
 ; GCN-NEXT:      simple_lds_recurse:
 ; GCN-NEXT:        .lds_size:       0x100{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x26{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x10{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x29{{$}}
 ; GCN-NEXT:      simple_stack:
 ; GCN-NEXT:        .lds_size:       0{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x21{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x14{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x2{{$}}
 ; GCN-NEXT:      simple_stack_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x26{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x3{{$}}
 ; GCN-NEXT:      simple_stack_extern_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
-; GFX8-NEXT:        .sgpr_count:     0x28{{$}}
-; GFX9-NEXT:        .sgpr_count:     0x2c{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x6c{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
-; GCN-NEXT:        .vgpr_count:     0x2a{{$}}
+; GCN-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_stack_indirect_call:
 ; GCN-NEXT:        .lds_size:       0{{$}}
-; GFX8-NEXT:        .sgpr_count:     0x28{{$}}
-; GFX9-NEXT:        .sgpr_count:     0x2c{{$}}
+; GFX8-NEXT:        .sgpr_count:     0x68{{$}}
+; GFX9-NEXT:        .sgpr_count:     0x6c{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
-; SDAG-NEXT:        .vgpr_count:     0x2b{{$}}
-; GISEL-NEXT:        .vgpr_count:     0x34{{$}}
+; SDAG-NEXT:        .vgpr_count:     0x40{{$}}
+; GISEL-NEXT:        .vgpr_count:     0x40{{$}}
 ; GCN-NEXT:      simple_stack_recurse:
 ; GCN-NEXT:        .lds_size:       0{{$}}
 ; GCN-NEXT:        .sgpr_count:     0x26{{$}}
 ; GCN-NEXT:        .stack_frame_size_in_bytes: 0x20{{$}}
 ; GCN-NEXT:        .vgpr_count:     0x2a{{$}}
 ; GCN-NEXT: ...

llvm/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size-vgpr-limit.ll

define amdgpu_kernel void @f512() #512 {
   call void @foo()
   call void @use256vgprs()
   ret void
 }
 attributes #512 = { nounwind "amdgpu-flat-work-group-size"="512,512" }
 
 ; GCN-LABEL: {{^}}f1024:
 ; GFX9: NumVgprs: 64
-; GFX90A: NumVgprs: 64
-; GFX90A: NumAgprs: 64
-; GFX90A: TotalNumVgprs: 128
+; GFX90A: NumVgprs: 128
+; GFX90A: NumAgprs: 128
+; GFX90A: TotalNumVgprs: 256
 ; GFX10WGP-WAVE32: NumVgprs: 128
 ; GFX10WGP-WAVE64: NumVgprs: 128
 ; GFX10CU-WAVE32: NumVgprs: 64
 ; GFX10CU-WAVE64: NumVgprs: 64
 define amdgpu_kernel void @f1024() #1024 {
   call void @foo()
   call void @use256vgprs()
   ret void
 }
 
 attributes #1024 = { nounwind "amdgpu-flat-work-group-size"="1024,1024" }
 
 declare void @foo()

llvm/test/CodeGen/AMDGPU/call-alias-register-usage-agpr.ll

+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx908 < %s | FileCheck -check-prefixes=ALL,GFX908 %s
+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a < %s | FileCheck -check-prefixes=ALL,GFX90A %s
+
+; CallGraphAnalysis, which CodeGenSCC order depends on, does not look
+; through aliases. If GlobalOpt is never run, we do not see direct
+; calls,
+
+@alias = hidden alias void (), void ()* @aliasee_default
+
+; ALL-LABEL: {{^}}kernel:
+; GFX908: .amdhsa_next_free_vgpr 64
+; GFX908-NEXT: .amdhsa_next_free_sgpr 102
+
+; GFX90A: .amdhsa_next_free_vgpr 256
+; GFX90A-NEXT: .amdhsa_next_free_sgpr 102
+; GFX90A-NEXT: .amdhsa_accum_offset 128
+define amdgpu_kernel void @kernel() #0 {
+bb:
+  call void @alias() #2
+  ret void
+}
+
+define internal void @aliasee_default() #1 {
+bb:
+  call void asm sideeffect "; clobber a26 ", "~{a26}"()
+  ret void
+}
+
+attributes #0 = { noinline norecurse nounwind optnone }
+attributes #1 = { noinline norecurse nounwind readnone willreturn }
+attributes #2 = { nounwind readnone willreturn }

llvm/test/CodeGen/AMDGPU/call-alias-register-usage0.ll

+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck %s
+
+; CallGraphAnalysis, which CodeGenSCC order depends on, does not look
+; through aliases. If GlobalOpt is never run, we do not see direct
+; calls,
+
+@alias0 = hidden alias void (), void ()* @aliasee_default_vgpr64_sgpr102
+
+; CHECK-LABEL: {{^}}kernel0:
+; CHECK: .amdhsa_next_free_vgpr 64
+; CHECK-NEXT: .amdhsa_next_free_sgpr 102
+define amdgpu_kernel void @kernel0() #0 {
+bb:
+  call void @alias0() #2
+  ret void
+}
+
+define internal void @aliasee_default_vgpr64_sgpr102() #1 {
+bb:
+  call void asm sideeffect "; clobber v52 ", "~{v52}"()
+  ret void
+}
+
+attributes #0 = { noinline norecurse nounwind optnone }
+attributes #1 = { noinline norecurse nounwind readnone willreturn }
+attributes #2 = { nounwind readnone willreturn }

llvm/test/CodeGen/AMDGPU/call-alias-register-usage1.ll

+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck %s
+
+; CallGraphAnalysis, which CodeGenSCC order depends on, does not look
+; through aliases. If GlobalOpt is never run, we do not see direct
+; calls,
+
+@alias1 = hidden alias void (), void ()* @aliasee_vgpr32_sgpr76
+
+; The parent kernel has a higher VGPR usage than the possible callees.
+
+; CHECK-LABEL: {{^}}kernel1:
+; CHECK: .amdhsa_next_free_vgpr 42
+; CHECK-NEXT: .amdhsa_next_free_sgpr 74
+define amdgpu_kernel void @kernel1() #0 {
+bb:
+  call void asm sideeffect "; clobber v40 ", "~{v40}"()
+  call void @alias1() #2
+  ret void
+}
+
+define internal void @aliasee_vgpr32_sgpr76() #1 {
+bb:
+  call void asm sideeffect "; clobber v26 ", "~{v26}"()
+  ret void
+}
+
+attributes #0 = { noinline norecurse nounwind optnone }
+attributes #1 = { noinline norecurse nounwind readnone willreturn "amdgpu-waves-per-eu"="8,10" }
+attributes #2 = { nounwind readnone willreturn }

llvm/test/CodeGen/AMDGPU/call-alias-register-usage2.ll

+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck %s
+
+; CallGraphAnalysis, which CodeGenSCC order depends on, does not look
+; through aliases. If GlobalOpt is never run, we do not see direct
+; calls,
+
+@alias2 = hidden alias void (), void()* @aliasee_vgpr64_sgpr102
+
+; CHECK-LABEL: {{^}}kernel2:
+; CHECK: .amdhsa_next_free_vgpr 64
+; CHECK-NEXT: .amdhsa_next_free_sgpr 102
+define amdgpu_kernel void @kernel2() #0 {
+bb:
+  call void @alias2() #2
+  ret void
+}
+
+define internal void @aliasee_vgpr64_sgpr102() #1 {
+bb:
+  call void asm sideeffect "; clobber v52 ", "~{v52}"()
+  ret void
+}
+
+attributes #0 = { noinline norecurse nounwind optnone }
+attributes #1 = { noinline norecurse nounwind readnone willreturn "amdgpu-waves-per-eu"="4,10" }
+attributes #2 = { nounwind readnone willreturn }

llvm/test/CodeGen/AMDGPU/call-alias-register-usage3.ll

+; RUN: llc -O0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck %s
+
+; CallGraphAnalysis, which CodeGenSCC order depends on, does not look
+; through aliases. If GlobalOpt is never run, we do not see direct
+; calls,
+
+@alias3 = hidden alias void (), void ()* @aliasee_vgpr256_sgpr102
+
+; CHECK-LABEL: {{^}}kernel3:
+; CHECK: .amdhsa_next_free_vgpr 256
+; CHECK-NEXT: .amdhsa_next_free_sgpr 102
+define amdgpu_kernel void @kernel3() #0 {
+bb:
+  call void @alias3() #2
+  ret void
+}
+
+define internal void @aliasee_vgpr256_sgpr102() #1 {
+bb:
+  call void asm sideeffect "; clobber v252 ", "~{v252}"()
+  ret void
+}
+
+attributes #0 = { noinline norecurse nounwind optnone }
+attributes #1 = { noinline norecurse nounwind readnone willreturn "amdgpu-flat-work-group-size"="1,256" "amdgpu-waves-per-eu"="1,1" }
+attributes #2 = { nounwind readnone willreturn }

llvm/test/CodeGen/AMDGPU/call-graph-register-usage.ll

 define amdgpu_kernel void @usage_direct_recursion(i32 %n) #0 {
   call void @direct_recursion_use_stack(i32 %n)
   ret void
 }
 
 ; Make sure there's no assert when a sgpr96 is used.
 ; GCN-LABEL: {{^}}count_use_sgpr96_external_call
 ; GCN: ; sgpr96 s[{{[0-9]+}}:{{[0-9]+}}]
-; CI: NumSgprs: 84
-; VI-NOBUG: NumSgprs: 86
+; CI: NumSgprs: 104
+; VI-NOBUG: NumSgprs: 108
 ; VI-BUG: NumSgprs: 96
-; GCN: NumVgprs: 50
+; GCN: NumVgprs: 64
 define amdgpu_kernel void @count_use_sgpr96_external_call()  {
 entry:
   tail call void asm sideeffect "; sgpr96 $0", "s"(<3 x i32> <i32 10, i32 11, i32 12>) #1
   call void @external()
   ret void
 }
 
 ; Make sure there's no assert when a sgpr160 is used.
 ; GCN-LABEL: {{^}}count_use_sgpr160_external_call
 ; GCN: ; sgpr160 s[{{[0-9]+}}:{{[0-9]+}}]
-; CI: NumSgprs: 84
-; VI-NOBUG: NumSgprs: 86
+; CI: NumSgprs: 104
+; VI-NOBUG: NumSgprs: 108
 ; VI-BUG: NumSgprs: 96
-; GCN: NumVgprs: 50
+; GCN: NumVgprs: 64
 define amdgpu_kernel void @count_use_sgpr160_external_call()  {
 entry:
   tail call void asm sideeffect "; sgpr160 $0", "s"(<5 x i32> <i32 10, i32 11, i32 12, i32 13, i32 14>) #1
   call void @external()
   ret void
 }
 
 ; Make sure there's no assert when a vgpr160 is used.
 ; GCN-LABEL: {{^}}count_use_vgpr160_external_call
 ; GCN: ; vgpr160 v[{{[0-9]+}}:{{[0-9]+}}]
-; CI: NumSgprs: 84
-; VI-NOBUG: NumSgprs: 86
+; CI: NumSgprs: 104
+; VI-NOBUG: NumSgprs: 108
 ; VI-BUG: NumSgprs: 96
-; GCN: NumVgprs: 50
+; GCN: NumVgprs: 64
 define amdgpu_kernel void @count_use_vgpr160_external_call()  {
 entry:
   tail call void asm sideeffect "; vgpr160 $0", "v"(<5 x i32> <i32 10, i32 11, i32 12, i32 13, i32 14>) #1
   call void @external()
   ret void
 }
 
 attributes #0 = { nounwind noinline norecurse }
 attributes #1 = { nounwind noinline norecurse }
 attributes #2 = { nounwind noinline }

llvm/test/CodeGen/AMDGPU/indirect-call.ll

 ; GCN-NEXT:     amd_code_version_major = 1
 ; GCN-NEXT:     amd_code_version_minor = 2
 ; GCN-NEXT:     amd_machine_kind = 1
 ; GCN-NEXT:     amd_machine_version_major = 7
 ; GCN-NEXT:     amd_machine_version_minor = 0
 ; GCN-NEXT:     amd_machine_version_stepping = 0
 ; GCN-NEXT:     kernel_code_entry_byte_offset = 256
 ; GCN-NEXT:     kernel_code_prefetch_byte_size = 0
-; GCN-NEXT:     granulated_workitem_vgpr_count = 7
-; GCN-NEXT:     granulated_wavefront_sgpr_count = 4
+; GCN-NEXT:     granulated_workitem_vgpr_count = 15
+; GCN-NEXT:     granulated_wavefront_sgpr_count = 12
 ; GCN-NEXT:     priority = 0
 ; GCN-NEXT:     float_mode = 240
 ; GCN-NEXT:     priv = 0
 ; GCN-NEXT:     enable_dx10_clamp = 1
 ; GCN-NEXT:     debug_mode = 0
 ; GCN-NEXT:     enable_ieee_mode = 1
 ; GCN-NEXT:     enable_wgp_mode = 0
 ; GCN-NEXT:     enable_mem_ordered = 0

 ; GCN-NEXT:     is_dynamic_callstack = 1
 ; GCN-NEXT:     is_debug_enabled = 0
 ; GCN-NEXT:     is_xnack_enabled = 0
 ; GCN-NEXT:     workitem_private_segment_byte_size = 16384
 ; GCN-NEXT:     workgroup_group_segment_byte_size = 0
 ; GCN-NEXT:     gds_segment_byte_size = 0
 ; GCN-NEXT:     kernarg_segment_byte_size = 64
 ; GCN-NEXT:     workgroup_fbarrier_count = 0
-; GCN-NEXT:     wavefront_sgpr_count = 37
-; GCN-NEXT:     workitem_vgpr_count = 32
+; GCN-NEXT:     wavefront_sgpr_count = 104
+; GCN-NEXT:     workitem_vgpr_count = 64
 ; GCN-NEXT:     reserved_vgpr_first = 0
 ; GCN-NEXT:     reserved_vgpr_count = 0
 ; GCN-NEXT:     reserved_sgpr_first = 0
 ; GCN-NEXT:     reserved_sgpr_count = 0
 ; GCN-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GCN-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GCN-NEXT:     kernarg_segment_alignment = 4
 ; GCN-NEXT:     group_segment_alignment = 4

 ; GISEL-NEXT:     amd_code_version_major = 1
 ; GISEL-NEXT:     amd_code_version_minor = 2
 ; GISEL-NEXT:     amd_machine_kind = 1
 ; GISEL-NEXT:     amd_machine_version_major = 7
 ; GISEL-NEXT:     amd_machine_version_minor = 0
 ; GISEL-NEXT:     amd_machine_version_stepping = 0
 ; GISEL-NEXT:     kernel_code_entry_byte_offset = 256
 ; GISEL-NEXT:     kernel_code_prefetch_byte_size = 0
-; GISEL-NEXT:     granulated_workitem_vgpr_count = 7
-; GISEL-NEXT:     granulated_wavefront_sgpr_count = 4
+; GISEL-NEXT:     granulated_workitem_vgpr_count = 15
+; GISEL-NEXT:     granulated_wavefront_sgpr_count = 12
 ; GISEL-NEXT:     priority = 0
 ; GISEL-NEXT:     float_mode = 240
 ; GISEL-NEXT:     priv = 0
 ; GISEL-NEXT:     enable_dx10_clamp = 1
 ; GISEL-NEXT:     debug_mode = 0
 ; GISEL-NEXT:     enable_ieee_mode = 1
 ; GISEL-NEXT:     enable_wgp_mode = 0
 ; GISEL-NEXT:     enable_mem_ordered = 0

 ; GISEL-NEXT:     is_dynamic_callstack = 1
 ; GISEL-NEXT:     is_debug_enabled = 0
 ; GISEL-NEXT:     is_xnack_enabled = 0
 ; GISEL-NEXT:     workitem_private_segment_byte_size = 16384
 ; GISEL-NEXT:     workgroup_group_segment_byte_size = 0
 ; GISEL-NEXT:     gds_segment_byte_size = 0
 ; GISEL-NEXT:     kernarg_segment_byte_size = 64
 ; GISEL-NEXT:     workgroup_fbarrier_count = 0
-; GISEL-NEXT:     wavefront_sgpr_count = 37
-; GISEL-NEXT:     workitem_vgpr_count = 32
+; GISEL-NEXT:     wavefront_sgpr_count = 104
+; GISEL-NEXT:     workitem_vgpr_count = 64
 ; GISEL-NEXT:     reserved_vgpr_first = 0
 ; GISEL-NEXT:     reserved_vgpr_count = 0
 ; GISEL-NEXT:     reserved_sgpr_first = 0
 ; GISEL-NEXT:     reserved_sgpr_count = 0
 ; GISEL-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GISEL-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GISEL-NEXT:     kernarg_segment_alignment = 4
 ; GISEL-NEXT:     group_segment_alignment = 4

 ; GCN-NEXT:     amd_code_version_major = 1
 ; GCN-NEXT:     amd_code_version_minor = 2
 ; GCN-NEXT:     amd_machine_kind = 1
 ; GCN-NEXT:     amd_machine_version_major = 7
 ; GCN-NEXT:     amd_machine_version_minor = 0
 ; GCN-NEXT:     amd_machine_version_stepping = 0
 ; GCN-NEXT:     kernel_code_entry_byte_offset = 256
 ; GCN-NEXT:     kernel_code_prefetch_byte_size = 0
-; GCN-NEXT:     granulated_workitem_vgpr_count = 7
-; GCN-NEXT:     granulated_wavefront_sgpr_count = 4
+; GCN-NEXT:     granulated_workitem_vgpr_count = 15
+; GCN-NEXT:     granulated_wavefront_sgpr_count = 12
 ; GCN-NEXT:     priority = 0
 ; GCN-NEXT:     float_mode = 240
 ; GCN-NEXT:     priv = 0
 ; GCN-NEXT:     enable_dx10_clamp = 1
 ; GCN-NEXT:     debug_mode = 0
 ; GCN-NEXT:     enable_ieee_mode = 1
 ; GCN-NEXT:     enable_wgp_mode = 0
 ; GCN-NEXT:     enable_mem_ordered = 0

 ; GCN-NEXT:     is_dynamic_callstack = 1
 ; GCN-NEXT:     is_debug_enabled = 0
 ; GCN-NEXT:     is_xnack_enabled = 0
 ; GCN-NEXT:     workitem_private_segment_byte_size = 16384
 ; GCN-NEXT:     workgroup_group_segment_byte_size = 0
 ; GCN-NEXT:     gds_segment_byte_size = 0
 ; GCN-NEXT:     kernarg_segment_byte_size = 64
 ; GCN-NEXT:     workgroup_fbarrier_count = 0
-; GCN-NEXT:     wavefront_sgpr_count = 37
-; GCN-NEXT:     workitem_vgpr_count = 32
+; GCN-NEXT:     wavefront_sgpr_count = 104
+; GCN-NEXT:     workitem_vgpr_count = 64
 ; GCN-NEXT:     reserved_vgpr_first = 0
 ; GCN-NEXT:     reserved_vgpr_count = 0
 ; GCN-NEXT:     reserved_sgpr_first = 0
 ; GCN-NEXT:     reserved_sgpr_count = 0
 ; GCN-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GCN-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GCN-NEXT:     kernarg_segment_alignment = 4
 ; GCN-NEXT:     group_segment_alignment = 4

 ; GISEL-NEXT:     amd_code_version_major = 1
 ; GISEL-NEXT:     amd_code_version_minor = 2
 ; GISEL-NEXT:     amd_machine_kind = 1
 ; GISEL-NEXT:     amd_machine_version_major = 7
 ; GISEL-NEXT:     amd_machine_version_minor = 0
 ; GISEL-NEXT:     amd_machine_version_stepping = 0
 ; GISEL-NEXT:     kernel_code_entry_byte_offset = 256
 ; GISEL-NEXT:     kernel_code_prefetch_byte_size = 0
-; GISEL-NEXT:     granulated_workitem_vgpr_count = 7
-; GISEL-NEXT:     granulated_wavefront_sgpr_count = 4
+; GISEL-NEXT:     granulated_workitem_vgpr_count = 15
+; GISEL-NEXT:     granulated_wavefront_sgpr_count = 12
 ; GISEL-NEXT:     priority = 0
 ; GISEL-NEXT:     float_mode = 240
 ; GISEL-NEXT:     priv = 0
 ; GISEL-NEXT:     enable_dx10_clamp = 1
 ; GISEL-NEXT:     debug_mode = 0
 ; GISEL-NEXT:     enable_ieee_mode = 1
 ; GISEL-NEXT:     enable_wgp_mode = 0
 ; GISEL-NEXT:     enable_mem_ordered = 0

 ; GISEL-NEXT:     is_dynamic_callstack = 1
 ; GISEL-NEXT:     is_debug_enabled = 0
 ; GISEL-NEXT:     is_xnack_enabled = 0
 ; GISEL-NEXT:     workitem_private_segment_byte_size = 16384
 ; GISEL-NEXT:     workgroup_group_segment_byte_size = 0
 ; GISEL-NEXT:     gds_segment_byte_size = 0
 ; GISEL-NEXT:     kernarg_segment_byte_size = 64
 ; GISEL-NEXT:     workgroup_fbarrier_count = 0
-; GISEL-NEXT:     wavefront_sgpr_count = 37
-; GISEL-NEXT:     workitem_vgpr_count = 32
+; GISEL-NEXT:     wavefront_sgpr_count = 104
+; GISEL-NEXT:     workitem_vgpr_count = 64
 ; GISEL-NEXT:     reserved_vgpr_first = 0
 ; GISEL-NEXT:     reserved_vgpr_count = 0
 ; GISEL-NEXT:     reserved_sgpr_first = 0
 ; GISEL-NEXT:     reserved_sgpr_count = 0
 ; GISEL-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
 ; GISEL-NEXT:     debug_private_segment_buffer_sgpr = 0
 ; GISEL-NEXT:     kernarg_segment_alignment = 4
 ; GISEL-NEXT:     group_segment_alignment = 4