[AMDGPU] Support emitting GOT relocations for function calls

[scott.linder]

Summary

Currently we make this decision based on addrspace, with the rationale being that Private (and so Generic/Flat) can't meaningfully support relocations. Until recently functions could not have a non-default (i.e. non-Generic/Flat) addrspace at all, meaning we never emitted GOT relocations for functions.

The approach taken in this patch is to explicitly support GOT for functions, regardless of addrspace. We might instead consider whether addrspace is the correct way to make the decision in the first place. In the long term we may also wish to change the default addrspace of our functions.

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

3488–3492

Looking for a specific load opcode looks concerning. Where is this coming from? I would think we need a pseudo at this point

[scott.linder]

lib/Target/AMDGPU/SIISelLowering.cpp

3488–3492

SITargetLowering::LowerGlobalAddress adds the load after creating the SI_PC_ADD_REL_OFFSET pseudo. Would it be better to add a new pseudo for just the load, or create a pseudo to include the existing PC relative calculation and the load, i.e. SI_LOAD_PC_ADD_REL_OFFSET? I don't think the load needs to be bundled to enforce an ordering or anything.

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

3488–3492

I'm worried about selecting a different opcode, and the possibility of some transform somewhere obscuring the address.

I think I looked at this before, and other targets use different call opcodes based on the address type. I think I did it this way because unlike other targets, we don't embed the call address in the actual call instruction. I think using the second pseudo would be closer to this, though I still don't like relying on getVRegDef here.

[scott.linder]

lib/Target/AMDGPU/SIISelLowering.cpp

3488–3492

I started to implement another pseudo, but as you mention getVRegDef doesn't seem right here. Is there a reason why the Global isn't carried through all of ISel, rather than being "recovered" later?

[scott.linder]

I've tried to implement carrying a copy of the function global through the DAG from the beginning, rather than recovering it after legalizing the GlobalAddress. I don't know if this is a reasonable approach, but I figured I would post something and see what you think.

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

3496

What happened to TCRETURN? I'm pretty sure we need this

test/CodeGen/AMDGPU/byval-frame-setup.ll

17

Can you commit the test changes adding hidden first?

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

3998

Can you add a comment here

[scott.linder]

Address feedback

lib/Target/AMDGPU/SIISelLowering.cpp

3496

After removing the need to insert the original global here there was nothing to do for TCRETURN in the custom inserter, so I just deleted the _ISEL variant and used tablegen to legalize ISD::TC_RETURN directly to SI_TCRETURN.

[arsenm]

test/CodeGen/AMDGPU/function-call-relocs.ll

49

Can you add a case with a tail call that needs the GOT just to be sure?

[scott.linder]

Add test for tail call through GOT

[arsenm]

LGTM

lib/Target/AMDGPU/AMDGPUInstrInfo.td

 def AMDGPUIfBreakOp : SDTypeProfile<1, 2,
   [SDTCisVT<0, i64>, SDTCisVT<1, i1>, SDTCisVT<2, i64>]
 >;
 
 def AMDGPUAddeSubeOp : SDTypeProfile<2, 3,
   [SDTCisSameAs<0, 2>, SDTCisSameAs<0, 3>, SDTCisVT<0, i32>, SDTCisVT<1, i1>, SDTCisVT<4, i1>]
 >;
 
-def SDT_AMDGPUTCRET : SDTypeProfile<0, 2, [SDTCisPtrTy<0>]>;
-
 //===----------------------------------------------------------------------===//
 // AMDGPU DAG Nodes
 //
 
 def AMDGPUif : SDNode<"AMDGPUISD::IF", AMDGPUIfOp, [SDNPHasChain]>;
 def AMDGPUelse : SDNode<"AMDGPUISD::ELSE", AMDGPUElseOp, [SDNPHasChain]>;
 def AMDGPUloop : SDNode<"AMDGPUISD::LOOP", AMDGPULoopOp, [SDNPHasChain]>;
 
 def callseq_start : SDNode<"ISD::CALLSEQ_START",
   SDCallSeqStart<[ SDTCisVT<0, i32>, SDTCisVT<1, i32> ]>,
   [SDNPHasChain, SDNPOutGlue]
 >;
 
 def callseq_end : SDNode<"ISD::CALLSEQ_END",
  SDCallSeqEnd<[ SDTCisVT<0, i32>, SDTCisVT<1, i32> ]>,
   [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue]
 >;
 
 def AMDGPUcall : SDNode<"AMDGPUISD::CALL",
   SDTypeProfile<0, -1, [SDTCisPtrTy<0>]>,
   [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
   SDNPVariadic]
 >;
 
-def AMDGPUtc_return: SDNode<"AMDGPUISD::TC_RETURN", SDT_AMDGPUTCRET,
+def AMDGPUtc_return: SDNode<"AMDGPUISD::TC_RETURN",
+  SDTypeProfile<0, 3, [SDTCisPtrTy<0>]>,
   [SDNPHasChain, SDNPOptInGlue, SDNPVariadic]
 >;
 
 def AMDGPUtrap : SDNode<"AMDGPUISD::TRAP",
   SDTypeProfile<0, -1, [SDTCisVT<0, i16>]>,
     [SDNPHasChain, SDNPVariadic, SDNPSideEffect, SDNPInGlue]
 >;
 

lib/Target/AMDGPU/SIISelLowering.cpp

Chain = DAG.getCALLSEQ_END(Chain,
                                DAG.getTargetConstant(0, DL, MVT::i32),
                                InFlag, DL);
     InFlag = Chain.getValue(1);
   }
 
   std::vector<SDValue> Ops;
   Ops.push_back(Chain);
   Ops.push_back(Callee);
+  // Add a redundant copy of the callee global which will not be legalized, as
+  // we need direct access to the callee later.
+  GlobalAddressSDNode *GSD = cast<GlobalAddressSDNode>(Callee);
+  const GlobalValue *GV = GSD->getGlobal();
+  Ops.push_back(DAG.getTargetGlobalAddress(GV, DL, MVT::i64));
 
   if (IsTailCall) {
     // Each tail call may have to adjust the stack by a different amount, so
     // this information must travel along with the operation for eventual
     // consumption by emitEpilogue.
     Ops.push_back(DAG.getTargetConstant(FPDiff, DL, MVT::i32));
 
     Ops.push_back(PhysReturnAddrReg);
case AMDGPU::ADJCALLSTACKDOWN: {
     // Add an implicit use of the frame offset reg to prevent the restore copy
     // inserted after the call from being reorderd after stack operations in the
     // the caller's frame.
     MIB.addReg(Info->getStackPtrOffsetReg(), RegState::ImplicitDefine)
         .addReg(Info->getStackPtrOffsetReg(), RegState::Implicit)
         .addReg(Info->getFrameOffsetReg(), RegState::Implicit);
     return BB;
   }
-  case AMDGPU::SI_CALL_ISEL:
-  case AMDGPU::SI_TCRETURN_ISEL: {
+  case AMDGPU::SI_CALL_ISEL: {
     const SIInstrInfo *TII = getSubtarget()->getInstrInfo();
     const DebugLoc &DL = MI.getDebugLoc();
-    unsigned ReturnAddrReg = TII->getRegisterInfo().getReturnAddressReg(*MF);
 
-    MachineRegisterInfo &MRI = MF->getRegInfo();
-    unsigned GlobalAddrReg = MI.getOperand(0).getReg();
-    MachineInstr *PCRel = MRI.getVRegDef(GlobalAddrReg);
-    assert(PCRel->getOpcode() == AMDGPU::SI_PC_ADD_REL_OFFSET);
-
-    const GlobalValue *G = PCRel->getOperand(1).getGlobal();
+    unsigned ReturnAddrReg = TII->getRegisterInfo().getReturnAddressReg(*MF);
 
     MachineInstrBuilder MIB;
-    if (MI.getOpcode() == AMDGPU::SI_CALL_ISEL) {
-      MIB = BuildMI(*BB, MI, DL, TII->get(AMDGPU::SI_CALL), ReturnAddrReg)
-        .add(MI.getOperand(0))
-        .addGlobalAddress(G);
-    } else {
-      MIB = BuildMI(*BB, MI, DL, TII->get(AMDGPU::SI_TCRETURN))
-        .add(MI.getOperand(0))
-        .addGlobalAddress(G);
-
-      // There is an additional imm operand for tcreturn, but it should be in the
-      // right place already.
-    }
+    MIB = BuildMI(*BB, MI, DL, TII->get(AMDGPU::SI_CALL), ReturnAddrReg);
 
-    for (unsigned I = 1, E = MI.getNumOperands(); I != E; ++I)
+    for (unsigned I = 0, E = MI.getNumOperands(); I != E; ++I)
       MIB.add(MI.getOperand(I));
 
     MIB.cloneMemRefs(MI);
     MI.eraseFromParent();
     return BB;
   }
   default:
     return AMDGPUTargetLowering::EmitInstrWithCustomInserter(MI, BB);
   }

 bool SITargetLowering::shouldEmitFixup(const GlobalValue *GV) const {
   const Triple &TT = getTargetMachine().getTargetTriple();
   return (GV->getType()->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS ||
           GV->getType()->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS_32BIT) &&
          AMDGPU::shouldEmitConstantsToTextSection(TT);
 }
 
 bool SITargetLowering::shouldEmitGOTReloc(const GlobalValue *GV) const {
-  return (GV->getType()->getAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS ||
+  return (GV->getValueType()->isFunctionTy() ||
+          GV->getType()->getAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS ||
           GV->getType()->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS ||
           GV->getType()->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS_32BIT) &&
          !shouldEmitFixup(GV) &&
          !getTargetMachine().shouldAssumeDSOLocal(*GV->getParent(), GV);
 }
 
 bool SITargetLowering::shouldEmitPCReloc(const GlobalValue *GV) const {
   return !shouldEmitFixup(GV) && !shouldEmitGOTReloc(GV);

lib/Target/AMDGPU/SIInstructions.td

def SI_RETURN : SPseudoInstSI <
   let SchedRW = [WriteBranch];
 }
 
 // Return for returning function calls without output register.
 //
 // This version is only needed so we can fill in the output regiter in
 // the custom inserter.
 def SI_CALL_ISEL : SPseudoInstSI <
-  (outs), (ins SSrc_b64:$src0), [(AMDGPUcall i64:$src0)]> {
+  (outs), (ins SSrc_b64:$src0, unknown:$callee),
+  [(AMDGPUcall i64:$src0, tglobaladdr:$callee)]> {
   let Size = 4;
   let isCall = 1;
   let SchedRW = [WriteBranch];
   let usesCustomInserter = 1;
 }
 
 // Wrapper around s_swappc_b64 with extra $callee parameter to track
 // the called function after regalloc.
 def SI_CALL : SPseudoInstSI <
   (outs SReg_64:$dst), (ins SSrc_b64:$src0, unknown:$callee)> {
   let Size = 4;
   let isCall = 1;
   let UseNamedOperandTable = 1;
   let SchedRW = [WriteBranch];
 }
 
 // Tail call handling pseudo
-def SI_TCRETURN_ISEL : SPseudoInstSI<(outs),
-  (ins SSrc_b64:$src0, i32imm:$fpdiff),
-  [(AMDGPUtc_return i64:$src0, i32:$fpdiff)]> {
-  let isCall = 1;
-  let isTerminator = 1;
-  let isReturn = 1;
-  let isBarrier = 1;
-  let SchedRW = [WriteBranch];
-  let usesCustomInserter = 1;
-}
-
-def SI_TCRETURN : SPseudoInstSI <
-  (outs),
-  (ins SSrc_b64:$src0, unknown:$callee, i32imm:$fpdiff)> {
+def SI_TCRETURN : SPseudoInstSI <(outs),
+  (ins SSrc_b64:$src0, unknown:$callee, i32imm:$fpdiff),
+  [(AMDGPUtc_return i64:$src0, tglobaladdr:$callee, i32:$fpdiff)]> {
   let Size = 4;
   let isCall = 1;
   let isTerminator = 1;
   let isReturn = 1;
   let isBarrier = 1;
   let UseNamedOperandTable = 1;
   let SchedRW = [WriteBranch];
 }

test/CodeGen/AMDGPU/byval-frame-setup.ll

 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=fiji -enable-ipra=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=hawaii -enable-ipra=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,CI %s
 
 %struct.ByValStruct = type { [4 x i32] }
 
 ; GCN-LABEL: {{^}}void_func_byval_struct:
 ; GCN: s_mov_b32 s5, s32
 ; GCN: buffer_load_dword [[LOAD0:v[0-9]+]], off, s[0:3], s5 offset:4{{$}}
 ; GCN-NOT: s32
 ; GCN: buffer_store_dword [[LOAD0]], off, s[0:3], s5 offset:4{{$}}
 ; GCN-NOT: s32
 
 ; GCN: buffer_load_dword [[LOAD1:v[0-9]+]], off, s[0:3], s5 offset:20{{$}}
 ; GCN-NOT: s32
 ; GCN: buffer_store_dword [[LOAD1]], off, s[0:3], s5 offset:20{{$}}
 ; GCN-NOT: s32
-define void @void_func_byval_struct(%struct.ByValStruct addrspace(5)* byval noalias nocapture align 4 %arg0, %struct.ByValStruct addrspace(5)* byval noalias nocapture align 4 %arg1) #1 {
+define hidden void @void_func_byval_struct(%struct.ByValStruct addrspace(5)* byval noalias nocapture align 4 %arg0, %struct.ByValStruct addrspace(5)* byval noalias nocapture align 4 %arg1) #1 {
 entry:
   %arrayidx = getelementptr inbounds %struct.ByValStruct, %struct.ByValStruct addrspace(5)* %arg0, i32 0, i32 0, i32 0
   %tmp = load volatile i32, i32 addrspace(5)* %arrayidx, align 4
   %add = add nsw i32 %tmp, 1
   store volatile i32 %add, i32 addrspace(5)* %arrayidx, align 4
   %arrayidx2 = getelementptr inbounds %struct.ByValStruct, %struct.ByValStruct addrspace(5)* %arg1, i32 0, i32 0, i32 0
   %tmp1 = load volatile i32, i32 addrspace(5)* %arrayidx2, align 4
   %add3 = add nsw i32 %tmp1, 2

 ; GCN-NOT: s32
 ; GCN: buffer_store_dword [[LOAD0]], off, s[0:3], s5 offset:8{{$}}
 ; GCN-NOT: s32
 
 ; GCN: buffer_load_dword [[LOAD1:v[0-9]+]], off, s[0:3], s5 offset:24{{$}}
 ; GCN-NOT: s32
 ; GCN: buffer_store_dword [[LOAD1]], off, s[0:3], s5 offset:24{{$}}
 ; GCN-NOT: s32
-define void @void_func_byval_struct_align8(%struct.ByValStruct addrspace(5)* byval noalias nocapture align 8 %arg0, %struct.ByValStruct addrspace(5)* byval noalias nocapture align 8 %arg1) #1 {
+define hidden void @void_func_byval_struct_align8(%struct.ByValStruct addrspace(5)* byval noalias nocapture align 8 %arg0, %struct.ByValStruct addrspace(5)* byval noalias nocapture align 8 %arg1) #1 {
 entry:
   %arrayidx = getelementptr inbounds %struct.ByValStruct, %struct.ByValStruct addrspace(5)* %arg0, i32 0, i32 0, i32 0
   %tmp = load volatile i32, i32 addrspace(5)* %arrayidx, align 8
   %add = add nsw i32 %tmp, 1
   store volatile i32 %add, i32 addrspace(5)* %arrayidx, align 8
   %arrayidx2 = getelementptr inbounds %struct.ByValStruct, %struct.ByValStruct addrspace(5)* %arg1, i32 0, i32 0, i32 0
   %tmp1 = load volatile i32, i32 addrspace(5)* %arrayidx2, align 8
   %add3 = add nsw i32 %tmp1, 2

test/CodeGen/AMDGPU/call-argument-types.ll

 ; RUN: llc -march=amdgcn -mcpu=fiji -mattr=-flat-for-global -amdgpu-scalarize-global-loads=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI,MESA %s
 ; RUN: llc -march=amdgcn -mcpu=hawaii -amdgpu-scalarize-global-loads=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,CI,MESA %s
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-scalarize-global-loads=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX9,VI,MESA %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=fiji -mattr=-flat-for-global -amdgpu-scalarize-global-loads=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI,HSA %s
 
-declare void @external_void_func_i1(i1) #0
-declare void @external_void_func_i1_signext(i1 signext) #0
-declare void @external_void_func_i1_zeroext(i1 zeroext) #0
-
-declare void @external_void_func_i8(i8) #0
-declare void @external_void_func_i8_signext(i8 signext) #0
-declare void @external_void_func_i8_zeroext(i8 zeroext) #0
-
-declare void @external_void_func_i16(i16) #0
-declare void @external_void_func_i16_signext(i16 signext) #0
-declare void @external_void_func_i16_zeroext(i16 zeroext) #0
-
-declare void @external_void_func_i32(i32) #0
-declare void @external_void_func_i64(i64) #0
-declare void @external_void_func_v2i64(<2 x i64>) #0
-declare void @external_void_func_v3i64(<3 x i64>) #0
-declare void @external_void_func_v4i64(<4 x i64>) #0
-
-declare void @external_void_func_f16(half) #0
-declare void @external_void_func_f32(float) #0
-declare void @external_void_func_f64(double) #0
-declare void @external_void_func_v2f32(<2 x float>) #0
-declare void @external_void_func_v2f64(<2 x double>) #0
-declare void @external_void_func_v3f64(<3 x double>) #0
-
-declare void @external_void_func_v2i16(<2 x i16>) #0
-declare void @external_void_func_v2f16(<2 x half>) #0
-declare void @external_void_func_v3i16(<3 x i16>) #0
-declare void @external_void_func_v3f16(<3 x half>) #0
-declare void @external_void_func_v4i16(<4 x i16>) #0
-declare void @external_void_func_v4f16(<4 x half>) #0
-
-declare void @external_void_func_v2i32(<2 x i32>) #0
-declare void @external_void_func_v3i32(<3 x i32>) #0
-declare void @external_void_func_v3i32_i32(<3 x i32>, i32) #0
-declare void @external_void_func_v4i32(<4 x i32>) #0
-declare void @external_void_func_v8i32(<8 x i32>) #0
-declare void @external_void_func_v16i32(<16 x i32>) #0
-declare void @external_void_func_v32i32(<32 x i32>) #0
-declare void @external_void_func_v32i32_i32(<32 x i32>, i32) #0
+declare hidden void @external_void_func_i1(i1) #0
+declare hidden void @external_void_func_i1_signext(i1 signext) #0
+declare hidden void @external_void_func_i1_zeroext(i1 zeroext) #0
+
+declare hidden void @external_void_func_i8(i8) #0
+declare hidden void @external_void_func_i8_signext(i8 signext) #0
+declare hidden void @external_void_func_i8_zeroext(i8 zeroext) #0
+
+declare hidden void @external_void_func_i16(i16) #0
+declare hidden void @external_void_func_i16_signext(i16 signext) #0
+declare hidden void @external_void_func_i16_zeroext(i16 zeroext) #0
+
+declare hidden void @external_void_func_i32(i32) #0
+declare hidden void @external_void_func_i64(i64) #0
+declare hidden void @external_void_func_v2i64(<2 x i64>) #0
+declare hidden void @external_void_func_v3i64(<3 x i64>) #0
+declare hidden void @external_void_func_v4i64(<4 x i64>) #0
+
+declare hidden void @external_void_func_f16(half) #0
+declare hidden void @external_void_func_f32(float) #0
+declare hidden void @external_void_func_f64(double) #0
+declare hidden void @external_void_func_v2f32(<2 x float>) #0
+declare hidden void @external_void_func_v2f64(<2 x double>) #0
+declare hidden void @external_void_func_v3f64(<3 x double>) #0
+
+declare hidden void @external_void_func_v2i16(<2 x i16>) #0
+declare hidden void @external_void_func_v2f16(<2 x half>) #0
+declare hidden void @external_void_func_v3i16(<3 x i16>) #0
+declare hidden void @external_void_func_v3f16(<3 x half>) #0
+declare hidden void @external_void_func_v4i16(<4 x i16>) #0
+declare hidden void @external_void_func_v4f16(<4 x half>) #0
+
+declare hidden void @external_void_func_v2i32(<2 x i32>) #0
+declare hidden void @external_void_func_v3i32(<3 x i32>) #0
+declare hidden void @external_void_func_v3i32_i32(<3 x i32>, i32) #0
+declare hidden void @external_void_func_v4i32(<4 x i32>) #0
+declare hidden void @external_void_func_v8i32(<8 x i32>) #0
+declare hidden void @external_void_func_v16i32(<16 x i32>) #0
+declare hidden void @external_void_func_v32i32(<32 x i32>) #0
+declare hidden void @external_void_func_v32i32_i32(<32 x i32>, i32) #0
 
 ; return value and argument
-declare i32 @external_i32_func_i32(i32) #0
+declare hidden i32 @external_i32_func_i32(i32) #0
 
 ; Structs
-declare void @external_void_func_struct_i8_i32({ i8, i32 }) #0
-declare void @external_void_func_byval_struct_i8_i32({ i8, i32 } addrspace(5)* byval) #0
-declare void @external_void_func_sret_struct_i8_i32_byval_struct_i8_i32({ i8, i32 } addrspace(5)* sret, { i8, i32 } addrspace(5)* byval) #0
+declare hidden void @external_void_func_struct_i8_i32({ i8, i32 }) #0
+declare hidden void @external_void_func_byval_struct_i8_i32({ i8, i32 } addrspace(5)* byval) #0
+declare hidden void @external_void_func_sret_struct_i8_i32_byval_struct_i8_i32({ i8, i32 } addrspace(5)* sret, { i8, i32 } addrspace(5)* byval) #0
 
-declare void @external_void_func_v16i8(<16 x i8>) #0
+declare hidden void @external_void_func_v16i8(<16 x i8>) #0
 
 
 ; FIXME: Should be passing -1
 ; GCN-LABEL: {{^}}test_call_external_void_func_i1_imm:
 ; MESA: s_mov_b32 s36, SCRATCH_RSRC_DWORD
 
 ; MESA-DAG: s_mov_b64 s[0:1], s[36:37]
 

 ; GCN: buffer_load_dword v32, off, s[0:3], s5 offset:16 ; 4-byte Folded Reload
 ; GCN: s_setpc_b64
 define void @tail_call_stack_passed_arg_alignment_v32i32_f64(<32 x i32> %val, double %tmp) #0 {
 entry:
   tail call void @stack_passed_f64_arg(<32 x i32> %val, double %tmp)
   ret void
 }
 
-declare void @byval_align16_f64_arg(<32 x i32>, double addrspace(5)* byval align 16) #0
-declare void @stack_passed_f64_arg(<32 x i32>, double) #0
+declare hidden void @byval_align16_f64_arg(<32 x i32>, double addrspace(5)* byval align 16) #0
+declare hidden void @stack_passed_f64_arg(<32 x i32>, double) #0
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readnone }
 attributes #2 = { nounwind noinline }

test/CodeGen/AMDGPU/call-constexpr.ll

define amdgpu_kernel void @test_bitcast_argument_and_return_types() #0 {
   store volatile float %op, float addrspace(1)* undef
   ret void
 }
 
 ; GCN-LABEL: {{^}}use_workitem_id_x:
 ; GCN: s_waitcnt
 ; GCN-NEXT: v_add_i32_e32 v0, vcc, v1, v0
 ; GCN-NEXT: s_setpc_b64
-define i32 @use_workitem_id_x(i32 %arg0) #0 {
+define hidden i32 @use_workitem_id_x(i32 %arg0) #0 {
   %id = call i32 @llvm.amdgcn.workitem.id.x()
   %op = add i32 %id, %arg0
   ret i32 %op
 }
 
 ; GCN-LABEL: {{^}}test_bitcast_use_workitem_id_x:
 ; GCN: v_mov_b32_e32 v1, v0
 ; GCN: s_getpc_b64
continue:
   %op = fadd float %val, 1.0
   store volatile float %op, float addrspace(1)* undef
   ret void
 }
 
 ; Callees appears last in source file to test that we still lower their
 ; arguments before we lower any calls to them.
 
-define i32 @ret_i32_noinline() #0 {
+define hidden i32 @ret_i32_noinline() #0 {
   ret i32 4
 }
 
-define i32 @ret_i32_alwaysinline() #1 {
+define hidden i32 @ret_i32_alwaysinline() #1 {
   ret i32 4
 }
 
-define i32 @ident_i32(i32 %i) #0 {
+define hidden i32 @ident_i32(i32 %i) #0 {
   ret i32 %i
 }
 
 declare i32 @llvm.amdgcn.workitem.id.x() #2
 
 attributes #0 = { nounwind noinline }
 attributes #1 = { alwaysinline nounwind }
 attributes #2 = { nounwind readnone speculatable }

test/CodeGen/AMDGPU/call-preserved-registers.ll

 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=fiji -enable-ipra=0 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=hawaii -enable-ipra=0 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -enable-ipra=0 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
-declare void @external_void_func_void() #0
+declare hidden void @external_void_func_void() #0
 
 ; GCN-LABEL: {{^}}test_kernel_call_external_void_func_void_clobber_s30_s31_call_external_void_func_void:
 ; GCN: s_mov_b32 s33, s7
 ; GCN: s_mov_b32 s4, s33
 ; GCN-NEXT: s_getpc_b64 s[34:35]
 ; GCN-NEXT: s_add_u32 s34, s34,
 ; GCN-NEXT: s_addc_u32 s35, s35,
 ; GCN-NEXT: s_mov_b32 s32, s33
define void @void_func_void_clobber_s30_s31() #2 {
   ret void
 }
 
 ; GCN-LABEL: {{^}}void_func_void_clobber_vcc:
 ; GCN: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT: ;;#ASMSTART
 ; GCN-NEXT: ;;#ASMEND
 ; GCN-NEXT: s_setpc_b64 s[30:31]
-define void @void_func_void_clobber_vcc() #2 {
+define hidden void @void_func_void_clobber_vcc() #2 {
   call void asm sideeffect "", "~{VCC}"() #0
   ret void
 }
 
 ; GCN-LABEL: {{^}}test_call_void_func_void_clobber_vcc:
 ; GCN: s_getpc_b64
 ; GCN-NEXT: s_add_u32
 ; GCN-NEXT: s_addc_u32

 
 ; GCN-LABEL: {{^}}void_func_void_clobber_s33:
 ; GCN: v_writelane_b32 v0, s33, 0
 ; GCN-NEXT: #ASMSTART
 ; GCN-NEXT: ; clobber
 ; GCN-NEXT: #ASMEND
 ; GCN-NEXT: v_readlane_b32 s33, v0, 0
 ; GCN-NEXT: s_setpc_b64
-define void @void_func_void_clobber_s33() #2 {
+define hidden void @void_func_void_clobber_s33() #2 {
   call void asm sideeffect "; clobber", "~{s33}"() #0
   ret void
 }
 
 ; GCN-LABEL: {{^}}test_call_void_func_void_clobber_s33:
 ; GCN: s_mov_b32 s33, s7
 ; GCN: s_mov_b32 s4, s33
 ; GCN-NEXT: s_getpc_b64

test/CodeGen/AMDGPU/cross-block-use-is-not-abi-copy.ll

bb1:
   %extract0 = extractelement <4 x i32> %val0, i32 0
   %extract1 = extractelement <4 x half> %val1, i32 0
   %ins0 = insertvalue { i32, half } undef, i32 %extract0, 0
   %ins1 = insertvalue { i32, half } %ins0, half %extract1, 1
   ret { i32, half } %ins1
 }
 
 
-declare <2 x float> @func_v2f32() #0
-declare <3 x float> @func_v3f32() #0
-declare <4 x float> @func_v4f32() #0
-declare <4 x half> @func_v4f16() #0
+declare hidden <2 x float> @func_v2f32() #0
+declare hidden <3 x float> @func_v3f32() #0
+declare hidden <4 x float> @func_v4f32() #0
+declare hidden <4 x half> @func_v4f16() #0
 
-declare { <4 x i32>, <4 x half> } @func_struct() #0
+declare hidden { <4 x i32>, <4 x half> } @func_struct() #0
 
 attributes #0 = { nounwind}

test/CodeGen/AMDGPU/function-call-relocs.ll

+; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji < %s | FileCheck %s
+
+define void @func(i32 addrspace(1)* %out) #0 {
+  store i32 1, i32 addrspace(1)* %out
+  ret void
+}
+
+define protected void @protected_func(i32 addrspace(1)* %out) #0 {
+  store i32 1, i32 addrspace(1)* %out
+  ret void
+}
+
+define hidden void @hidden_func(i32 addrspace(1)* %out) #0 {
+  store i32 1, i32 addrspace(1)* %out
+  ret void
+}
+
+; CHECK-LABEL: call_func:
+; CHECK: s_getpc_b64 s{{\[}}[[PC_LO:[0-9]+]]:[[PC_HI:[0-9]+]]{{\]}}
+; CHECK: s_add_u32 s[[GOT_ADDR_LO:[0-9]+]], s[[PC_LO]], func@gotpcrel32@lo+4
+; CHECK: s_addc_u32 s[[GOT_ADDR_HI:[0-9]+]], s[[PC_HI]], func@gotpcrel32@hi+4
+; CHECK: s_load_dwordx2 s{{\[}}[[ADDR_LO:[0-9]+]]:[[ADDR_HI:[0-9]+]]{{\]}}, s{{\[}}[[GOT_ADDR_LO]]:[[GOT_ADDR_HI]]{{\]}}, 0x0
+; CHECK: s_swappc_b64 s{{\[}}{{[0-9]+:[0-9]+}}{{\]}}, s{{\[}}[[ADDR_LO]]:[[ADDR_HI]]{{\]}}
+define amdgpu_kernel void @call_func(i32 addrspace(1)* %out) {
+  call void @func(i32 addrspace(1)* %out)
+  ret void
+}
+
+; CHECK-LABEL: call_protected_func:
+; CHECK: s_getpc_b64 s{{\[}}[[PC_LO:[0-9]+]]:[[PC_HI:[0-9]+]]{{\]}}
+; CHECK: s_add_u32 s[[ADDR_LO:[0-9]+]], s[[PC_LO]], protected_func@rel32@lo+4
+; CHECK: s_addc_u32 s[[ADDR_HI:[0-9]+]], s[[PC_HI]], protected_func@rel32@hi+4
+; CHECK: s_swappc_b64 s{{\[}}{{[0-9]+:[0-9]+}}{{\]}}, s{{\[}}[[ADDR_LO]]:[[ADDR_HI]]{{\]}}
+define amdgpu_kernel void @call_protected_func(i32 addrspace(1)* %out) {
+  call void @protected_func(i32 addrspace(1)* %out)
+  ret void
+}
+
+; CHECK-LABEL: call_hidden_func:
+; CHECK: s_getpc_b64 s{{\[}}[[PC_LO:[0-9]+]]:[[PC_HI:[0-9]+]]{{\]}}
+; CHECK: s_add_u32 s[[ADDR_LO:[0-9]+]], s[[PC_LO]], hidden_func@rel32@lo+4
+; CHECK: s_addc_u32 s[[ADDR_HI:[0-9]+]], s[[PC_HI]], hidden_func@rel32@hi+4
+; CHECK: s_swappc_b64 s{{\[}}{{[0-9]+:[0-9]+}}{{\]}}, s{{\[}}[[ADDR_LO]]:[[ADDR_HI]]{{\]}}
+define amdgpu_kernel void @call_hidden_func(i32 addrspace(1)* %out) {
+  call void @hidden_func(i32 addrspace(1)* %out)
+  ret void
+}
+
+attributes #0 = { noinline }

test/CodeGen/AMDGPU/ipra.ll

 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs -enable-ipra -amdgpu-sroa=0 < %s | FileCheck -check-prefix=GCN %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs -amdgpu-sroa=0 < %s | FileCheck -check-prefix=GCN %s
 
 ; Kernels are not called, so there is no call preserved mask.
 ; GCN-LABEL: {{^}}kernel:
 ; GCN: flat_store_dword
 define amdgpu_kernel void @kernel(i32 addrspace(1)* %out) #0 {
 entry:
   store i32 0, i32 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}func:
 ; GCN: ; NumVgprs: 8
-define void @func() #1 {
+define hidden void @func() #1 {
   call void asm sideeffect "", "~{v0},~{v1},~{v2},~{v3},~{v4},~{v5},~{v6},~{v7}"() #0
   ret void
 }
 
 ; GCN-LABEL: {{^}}kernel_call:
 ; GCN-NOT: buffer_store
 ; GCN-NOT: buffer_load
 ; GCN-NOT: readlane

test/CodeGen/AMDGPU/mem-builtins.ll

 ; RUN: not llc -march=r600 < %s 2>&1 | FileCheck -check-prefix=ERROR %s
 ; RUN: llc -march=amdgcn < %s | FileCheck -check-prefix=GCN %s
 
-declare i32 @memcmp(i8 addrspace(1)* readonly nocapture, i8 addrspace(1)* readonly nocapture, i64) #0
-declare i8 addrspace(1)* @memchr(i8 addrspace(1)* readonly nocapture, i32, i64) #1
-declare i8* @strcpy(i8* nocapture, i8* readonly nocapture) #0
-declare i32 @strlen(i8* nocapture) #1
-declare i32 @strnlen(i8* nocapture, i32) #1
-declare i32 @strcmp(i8* nocapture, i8* nocapture) #1
+declare hidden i32 @memcmp(i8 addrspace(1)* readonly nocapture, i8 addrspace(1)* readonly nocapture, i64) #0
+declare hidden i8 addrspace(1)* @memchr(i8 addrspace(1)* readonly nocapture, i32, i64) #1
+declare hidden i8* @strcpy(i8* nocapture, i8* readonly nocapture) #0
+declare hidden i32 @strlen(i8* nocapture) #1
+declare hidden i32 @strnlen(i8* nocapture, i32) #1
+declare hidden i32 @strcmp(i8* nocapture, i8* nocapture) #1
 
 
 ; ERROR: error: <unknown>:0:0: in function test_memcmp void (i8 addrspace(1)*, i8 addrspace(1)*, i32*): unsupported call to function memcmp
 
 ; GCN: s_add_u32 s{{[0-9]+}}, s{{[0-9]+}}, memcmp@rel32@lo+4
 ; GCN: s_addc_u32 s{{[0-9]+}}, s{{[0-9]+}}, memcmp@rel32@hi+4
 define amdgpu_kernel void @test_memcmp(i8 addrspace(1)* %x, i8 addrspace(1)* %y, i32* nocapture %p) #0 {
 entry:

test/CodeGen/AMDGPU/sibling-call.ll

define fastcc i32 @i32_fastcc_i32_i32_stack_object(i32 %arg0, i32 %arg1) #1 {
   %alloca = alloca [16 x i32], align 4, addrspace(5)
   %gep = getelementptr inbounds [16 x i32], [16 x i32] addrspace(5)* %alloca, i32 0, i32 5
   store volatile i32 9, i32 addrspace(5)* %gep
   %add0 = add i32 %arg0, %arg1
   ret i32 %add0
 }
 
 ; GCN-LABEL: {{^}}sibling_call_i32_fastcc_i32_i32:
-define fastcc i32 @sibling_call_i32_fastcc_i32_i32(i32 %a, i32 %b, i32 %c) #1 {
+define hidden fastcc i32 @sibling_call_i32_fastcc_i32_i32(i32 %a, i32 %b, i32 %c) #1 {
 entry:
   %ret = tail call fastcc i32 @i32_fastcc_i32_i32(i32 %a, i32 %b)
   ret i32 %ret
 }
 
 ; GCN-LABEL: {{^}}sibling_call_i32_fastcc_i32_i32_stack_object:
 ; GCN: v_mov_b32_e32 [[NINE:v[0-9]+]], 9
 ; GCN: buffer_store_dword [[NINE]], off, s[0:3], s5 offset:24

 ; GCN-NEXT: s_mov_b32 s5, s32
 ; GCN-NEXT: buffer_load_dword v1, off, s[0:3], s5 offset:4
 ; GCN-NEXT: s_waitcnt vmcnt(0)
 
 ; CIVI-NEXT: v_add_{{i|u}}32_e32 v0, vcc, v0, v1
 ; GFX9-NEXT: v_add_u32_e32 v0, v0, v1
 
 ; GCN-NEXT: s_setpc_b64 s[30:31]
-define fastcc i32 @i32_fastcc_i32_byval_i32(i32 %arg0, i32 addrspace(5)* byval align 4 %arg1) #1 {
+define hidden fastcc i32 @i32_fastcc_i32_byval_i32(i32 %arg0, i32 addrspace(5)* byval align 4 %arg1) #1 {
   %arg1.load = load i32, i32 addrspace(5)* %arg1, align 4
   %add0 = add i32 %arg0, %arg1.load
   ret i32 %add0
 }
 
 ; Tail call disallowed with byval in parent.
 ; GCN-LABEL: {{^}}sibling_call_i32_fastcc_i32_byval_i32_byval_parent:
 ; GCN-NOT: v_writelane_b32 v{{[0-9]+}}, s32