[PowerPC] Add Support for indirect calls on AIX.

[sfertile]

Summary

Extends the desciptor-based indirect call support for 32-bit codegen, and enables indirect calls for AIX.

In-depth Description:
In a function descriptor based ABI, a function pointer points at a descriptor structure as opposed to the function's entry point. The
descriptor takes the form of 3 pointers: 1 for the function's entry point, 1 for the TOC anchor of the module containing the function
definition, and 1 for the environment pointer.

struct FunctionDescriptor {
  void *EntryPoint;
  void *TOCAnchor;
  void *EnvironmentPointer;
};

An indirect call has several steps of loading the the information from the descriptor into the proper registers for setting up the call. Namely it has to:

1. Save the caller's TOC pointer into the TOC save slot in the linkage area, and then load the callee's TOC pointer into the TOC register (GPR 2 on AIX).
2. Load the function descriptor's entry point into the count register.
3. Load the environment pointer into the environment pointer register (GPR 11 on AIX).
4. Perform the call by branching on count register.
5. Restore the caller's TOC pointer after returning from the indirect call.

A couple important caveats to the above:

• There is no way to directly load a value from memory into the count register. Instead we populate the count register by loading the entry point address into a gpr and then moving the gpr to the count register.
• The TOC restore has to come immediately after the branch on count register instruction (i.e., the 1st instruction executed after we return from the call). This is an implementation limitation. We could, in theory, schedule the restore elsewhere as long as no uses of the TOC pointer fall in between the call and the restore; however, to keep it simple, we insert a pseudo instruction that represents both the indirect branch instruction and the load instruction that restores the caller's TOC from the linkage area. As they flow through the compiler as a single pseudo instruction, nothing can be inserted between them and the caller's TOC is then valid at any use.

[sfertile]

• Change targeted cpu in the test from pwr7 to pwr4.
• Add testing of object file generation.
• Change CHECK-NEXT directive to CHECK-DAG directive for instructions that can be scheduled differently.
• Use a file check variable instead of hard-coded register for scratch register used to move the callee address into the count register.

[sfertile]

Rebased.

[hubert.reinterpretcast]

@sfertile, the current diff is missing the tests.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

3155

Nit: Add "is" before "not" here and on the next change.

5238

Nit: Add "is" before "not". Add a period.

5270

Nit: Add a comma after "ABIs".

llvm/lib/Target/PowerPC/PPCSubtarget.h

359

Nit: Capitalize "should". Add "the" before "target". Same for the other cases below.

377

Nit: Add "a" before "TOC".

[sfertile]

Addressed comments and added back missing tests.

[hubert.reinterpretcast]

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

34

Just noting here that the test does not actually enforce that r2 is saved before being written to. Similarly for the other tests.

35

Just noting here that the test does not actually enforce that the input register is populated before this instruction. Similarly for the other tests.

73

This only parameterizes the scratch register in the human readable form. Two of the hex digits are sensitive to the scratch register value. Similarly for the other test.

75

Same comment. r4 is hardcoded.

96

Just noting that the test does not enforce that the function pointer has been moved to the scratch register before it is clobbered. Similarly for the other tests.

133

Scratch register is not parameterized here and below.

[ZarkoCA]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

5237

I'm getting confused by this assert. Will it assert if a function has a nest parameter and it's not AIX?
Can it be rewritten to be:

assert((hasNest && Subtarget.isAIXABI()) && "Nest parameter is not supported on AIX.");

Or am i misunderstanding something?

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

2

I forget whether PWR4 supports altivec,. Wondering if you need -mattr=-altivec here?

I remember there was a discussion about different altivec codegen behaviour when you specify any -mcpu vs leaving it.

[hubert.reinterpretcast]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

5237

Can it be rewritten to be:

assert((hasNest && Subtarget.isAIXABI()) && "Nest parameter is not supported on AIX.");

That formulation will trigger the assert unless it is AIX and hasNest is true.

[ZarkoCA]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

5237

You're right, I blanked on the the way the assert works. I see now it's meant to enforce that a function doesn't have nest or if it does it's not on AIX. Please ignore my comment, Sean.

[sfertile]

• Removed all instruction encodings from the test, other then those that are expanded from the newly added pseudo instruction.
• parameterized the scratch register used in the 64-bit asm.

[sfertile]

In D70724#1768155, @hubert.reinterpretcast wrote:

Just noting here that the test does not actually enforce that r2 is saved before being written to. Similarly for the other tests.

and

In D70724#1768155, @hubert.reinterpretcast wrote:

Just noting here that the test does not actually enforce that the input register is populated before this instruction. Similarly for the other tests.

The problem is I have 2 different pairs of instructions that are order dependent with respect to each other, and several other instructions that can be interspersed. I'm not sure if its best to use CHECK-NEXT and verify the order at the cost of becoming fragile to the changes in the scheduler, or if its best to use 'CHECK-DAG` and expect that lnt and bootstrap to fail in the case the scheduler ever violated the dependency. I'm not aware of a way to write the test that both verifies those dependencies and keeps the flexibility for the independent instructions.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

5237

I can Demorgan it if you prefer: !(hasNest && Subtarget.isAIXABI())).

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

2

I need the attribute or I hit an error in CallLowering_AIX. The check for the error is overly general (just checking if the subtarget supports altivec, and not for any particular use of the altivec functionality). I would like to remove the error but that would need to be done in a seperate patch from this one.

73

Good catch, I completely missed that. The only encoding I'm actually interested in testing are the encodings of the 2 instructions expanded from the new pseudo. I've removed the other encodings from the test.

[ZarkoCA]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

5237

It's fine the way it is. I think it's more informative as to what triggers it in your original form.

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

2

I agree this is something we should discuss. This error to may need to be added to lot of testcases in the future and, in turn, potentially removed when altivec is supported. While also potentially having to rewrite the tests.

[hubert.reinterpretcast]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6923

Nit: Indentation does not match clang-format:

const unsigned TOCSaveOffset =
    Subtarget.getFrameLowering()->getTOCSaveOffset();

6932

Nit: Indentation does not match clang-format:

Chain = DAG.getStore(
    Val.getValue(1), dl, Val, AddPtr,
    MachinePointerInfo::getStack(DAG.getMachineFunction(), TOCSaveOffset));

llvm/lib/Target/PowerPC/PPCInstrFormats.td

1532

Nit: Missing space before xo1.

1533

Please refer to XLForm_2_ext. There should be no parameter taken for bh.

1539

Refer to DForm_1. This should be bits<21> Addr.

1545

Nit: This comment is a label, not a sentence. Do not use a period, but do capitalize.

1552

Same nit about labels versus sentences.

1552

This is incorrect for D-form. The D field has 16 bits.

llvm/lib/Target/PowerPC/PPCInstrInfo.td

1652

Should this be memri (accepting non-aligned operands)?

1653

Should this be iaddr (i.e., D-form)?

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

146

Seems like it might be less duplication if the grouping was --check-prefixes=ASMOBJ32,ASM32 and --check-prefixes=ASMOBJ32,OBJ32.

[sfertile]

• Fixed formatting in several places.
• Fixed the added Instruction format and Ins of the new pseudo op. I had implemented as if the load were a DS form instruction instead of a D form instruction.
• Collapsed the asm and obj lit testing checks together for the instructions where we don't care about the encoding (and the checks end up being the same between asm and binary output)

[hubert.reinterpretcast]

Some minor comments; otherwise, I think this is fine, but I don't know if others believe there are still comments to be addressed.

llvm/lib/Target/PowerPC/PPCInstrFormats.td

1539

The naming still refers to the "DS" field for the DS-form instruction.

1545

BH comprises bits 19 and 20, not bits 16 through 18.

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

116

Should we have an OBJ32-LABEL as well?

[sfertile]

Rebased, combined the ASM32 and OBJ32 checks that were the same into 1 ASMOBJ32 check label, moved comment on tabelgen InstrFormat member to the bh bits field.

[sfertile]

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

146

Good Call.

[hubert.reinterpretcast]

llvm/lib/Target/PowerPC/PPCInstrFormats.td

1545

Do we want a comment here to indicate that this is a "reserved" field?

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

20

Missing reference to CHECKOBJ.

[ZarkoCA]

Nit: It may be better to rename the test aix-indirect-call.ll to be consistent.

I think this patch looks good, I'll leave for @hubert.reinterpretcast to approve after his latest round comments.

[hubert.reinterpretcast]

In D70724#1780372, @ZarkoCA wrote:

Nit: It may be better to rename the test aix-indirect-call.ll to be consistent.

I agree.

I think this patch looks good, I'll leave for @hubert.reinterpretcast to approve after his latest round comments.

It seems there are only minor comments left that can be fixed on the commit. This patch LGTM.

[hubert.reinterpretcast]

Committed as rG93faa237da8d. @sfertile, typo in the commit message: s/Differtential/Differential/;

llvm/lib/Target/PowerPC/P9InstrResources.td

def : InstRW<[P9_BR_2C, DISP_BR_1C],
   BA,
   BC,
   BCC,
   BCCA,
   BCL,
   BCLalways,
   BCLn,
   BCTRL8_LDinto_toc,
+  BCTRL_LWZinto_toc,
   BCn,
   CTRL_DEP
 )>;
 
 // Five Cycle Branch with a 2 Cycle ALU Op
 // Operations must be done consecutively and not in parallel.
 def : InstRW<[P9_BROpAndALUOp_7C, IP_EXEC_1C, DISP_BR_1C, DISP_1C],
       (instrs

llvm/lib/Target/PowerPC/PPCFrameLowering.cpp

if (CR2Spilled || CR3Spilled || CR4Spilled) {
     restoreCRs(Subtarget.isPPC64(), is31, CR2Spilled, CR3Spilled, CR4Spilled,
                MBB, I, CSI, CSIIndex);
   }
 
   return true;
 }
 
 unsigned PPCFrameLowering::getTOCSaveOffset() const {
-  if (Subtarget.isAIXABI())
-    // TOC save/restore is normally handled by the linker.
-    // Indirect calls should hit this limitation.
-    report_fatal_error("TOC save is not implemented on AIX yet.");
   return TOCSaveOffset;
 }
 
 unsigned PPCFrameLowering::getFramePointerSaveOffset() const {
   if (Subtarget.isAIXABI())
     report_fatal_error("FramePointer is not implemented on AIX yet.");
   return FramePointerSaveOffset;
 }

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
       /// MTCTR instruction.
       MTCTR,
 
       /// CHAIN,FLAG = BCTRL(CHAIN, INFLAG) - Directly corresponds to a
       /// BCTRL instruction.
       BCTRL,
 
       /// CHAIN,FLAG = BCTRL(CHAIN, ADDR, INFLAG) - The combination of a bctrl
-      /// instruction and the TOC reload required on SVR4 PPC64.
+      /// instruction and the TOC reload required on 64-bit ELF, 32-bit AIX
+      /// and 64-bit AIX.
       BCTRL_LOAD_TOC,
 
       /// Return with a flag operand, matched by 'blr'
       RET_FLAG,
 
       /// R32 = MFOCRF(CRREG, INFLAG) - Represents the MFOCRF instruction.
       /// This copies the bits corresponding to the specified CRREG into the
       /// resultant GPR.  Bits corresponding to other CR regs are undefined.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

SDValue PPCTargetLowering::LowerVACOPY(SDValue Op, SelectionDAG &DAG) const {
   return DAG.getMemcpy(Op.getOperand(0), Op,
                        Op.getOperand(1), Op.getOperand(2),
                        DAG.getConstant(12, SDLoc(Op), MVT::i32), 8, false, true,
                        false, MachinePointerInfo(), MachinePointerInfo());
 }
 
 SDValue PPCTargetLowering::LowerADJUST_TRAMPOLINE(SDValue Op,
                                                   SelectionDAG &DAG) const {
+  if (Subtarget.isAIXABI())
+    report_fatal_error("ADJUST_TRAMPOLINE operation not supported on AIX.");
+
   return Op.getOperand(0);
 }
 
 SDValue PPCTargetLowering::LowerINIT_TRAMPOLINE(SDValue Op,
                                                 SelectionDAG &DAG) const {
+  if (Subtarget.isAIXABI())
+    report_fatal_error("INIT_TRAMPOLINE operation not supported on AIX.");
+
   SDValue Chain = Op.getOperand(0);
   SDValue Trmp = Op.getOperand(1); // trampoline
   SDValue FPtr = Op.getOperand(2); // nested function
   SDValue Nest = Op.getOperand(3); // 'nest' parameter value
   SDLoc dl(Op);
 
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   bool isPPC64 = (PtrVT == MVT::i64);
static void prepareDescriptorIndirectCall(SelectionDAG &DAG, SDValue &Callee,
   SDValue LDChain = getOutputChainFromCallSeq(CallSeqStart);
   auto MMOFlags = Subtarget.hasInvariantFunctionDescriptors()
                       ? (MachineMemOperand::MODereferenceable |
                          MachineMemOperand::MOInvariant)
                       : MachineMemOperand::MONone;
 
   MachinePointerInfo MPI(CS ? CS.getCalledValue() : nullptr);
 
+  // Registers used in building the DAG.
+  const MCRegister EnvPtrReg = Subtarget.getEnvironmentPointerRegister();
+  const MCRegister TOCReg = Subtarget.getTOCPointerRegister();
+
+  // Offsets of descriptor members.
+  const unsigned TOCAnchorOffset = Subtarget.descriptorTOCAnchorOffset();
+  const unsigned EnvPtrOffset = Subtarget.descriptorEnvironmentPointerOffset();
+
+  const MVT RegVT = Subtarget.isPPC64() ? MVT::i64 : MVT::i32;
+  const unsigned Alignment = Subtarget.isPPC64() ? 8 : 4;
+
   // One load for the functions entry point address.
-  SDValue LoadFuncPtr = DAG.getLoad(MVT::i64, dl, LDChain, Callee, MPI,
-                                    /* Alignment = */ 8, MMOFlags);
+  SDValue LoadFuncPtr = DAG.getLoad(RegVT, dl, LDChain, Callee, MPI,
+                                    Alignment, MMOFlags);
 
   // One for loading the TOC anchor for the module that contains the called
   // function.
-  SDValue TOCOff = DAG.getIntPtrConstant(8, dl);
-  SDValue AddTOC = DAG.getNode(ISD::ADD, dl, MVT::i64, Callee, TOCOff);
+  SDValue TOCOff = DAG.getIntPtrConstant(TOCAnchorOffset, dl);
+  SDValue AddTOC = DAG.getNode(ISD::ADD, dl, RegVT, Callee, TOCOff);
   SDValue TOCPtr =
-      DAG.getLoad(MVT::i64, dl, LDChain, AddTOC, MPI.getWithOffset(8),
-                  /* Alignment = */ 8, MMOFlags);
+      DAG.getLoad(RegVT, dl, LDChain, AddTOC,
+                  MPI.getWithOffset(TOCAnchorOffset), Alignment, MMOFlags);
 
   // One for loading the environment pointer.
-  SDValue PtrOff = DAG.getIntPtrConstant(16, dl);
-  SDValue AddPtr = DAG.getNode(ISD::ADD, dl, MVT::i64, Callee, PtrOff);
+  SDValue PtrOff = DAG.getIntPtrConstant(EnvPtrOffset, dl);
+  SDValue AddPtr = DAG.getNode(ISD::ADD, dl, RegVT, Callee, PtrOff);
   SDValue LoadEnvPtr =
-      DAG.getLoad(MVT::i64, dl, LDChain, AddPtr, MPI.getWithOffset(16),
-                  /* Alignment = */ 8, MMOFlags);
+      DAG.getLoad(RegVT, dl, LDChain, AddPtr,
+                  MPI.getWithOffset(EnvPtrOffset), Alignment, MMOFlags);
+
 
   // Then copy the newly loaded TOC anchor to the TOC pointer.
-  SDValue TOCVal = DAG.getCopyToReg(Chain, dl, PPC::X2, TOCPtr, Glue);
+  SDValue TOCVal = DAG.getCopyToReg(Chain, dl, TOCReg, TOCPtr, Glue);
   Chain = TOCVal.getValue(0);
   Glue = TOCVal.getValue(1);
 
   // If the function call has an explicit 'nest' parameter, it takes the
   // place of the environment pointer.
+  assert((!hasNest || !Subtarget.isAIXABI()) &&
+         "Nest parameter not supported on AIX");
   if (!hasNest) {
-    SDValue EnvVal = DAG.getCopyToReg(Chain, dl, PPC::X11, LoadEnvPtr, Glue);
+    SDValue EnvVal = DAG.getCopyToReg(Chain, dl, EnvPtrReg, LoadEnvPtr, Glue);
     Chain = EnvVal.getValue(0);
     Glue = EnvVal.getValue(1);
   }
 
   // The rest of the indirect call sequence is the same as the non-descriptor
   // DAG.
   prepareIndirectCall(DAG, LoadFuncPtr, Glue, Chain, dl);
 }
 
 static void
 buildCallOperands(SmallVectorImpl<SDValue> &Ops, CallingConv::ID CallConv,
                   const SDLoc &dl, bool isTailCall, bool isVarArg,
                   bool isPatchPoint, bool hasNest, SelectionDAG &DAG,
                   SmallVector<std::pair<unsigned, SDValue>, 8> &RegsToPass,
                   SDValue Glue, SDValue Chain, SDValue &Callee, int SPDiff,
                   const PPCSubtarget &Subtarget, bool isIndirect) {
   const bool IsPPC64 = Subtarget.isPPC64();
   // MVT for a general purpose register.
-  const MVT RegVT = IsPPC64 ?  MVT::i64 : MVT::i32;
+  const MVT RegVT = IsPPC64 ? MVT::i64 : MVT::i32;
 
   // First operand is always the chain.
   Ops.push_back(Chain);
 
   // If it's a direct call pass the callee as the second operand.
   if (!isIndirect)
     Ops.push_back(Callee);
   else {
     assert(!isPatchPoint && "Patch point call are not indirect.");
-    if (Subtarget.isAIXABI())
-      report_fatal_error("Indirect call on AIX is not implemented.");
 
-    // For 64-bit ELF we have saved the TOC pointer to the linkage area on the
-    // stack (this would have  been done in `LowerCall_64SVR4`). The call
-    // instruction is a pseudo instruction that represents both the indirect
-    // branch and a load that restores the TOC pointer from the linkage area.
-    // The operand for the TOC restore is an add of the TOC save offset to the
-    // stack pointer. This must be the second operand: after the chain input but
-    // before any other variadic arguments.
-    if (Subtarget.is64BitELFABI()) {
-      SDValue StackPtr = DAG.getRegister(PPC::X1, MVT::i64);
+    // For the TOC based ABIs we have saved the TOC pointer to the linkage area
+    // on the stack (this would have been done in `LowerCall_64SVR4` or
+    // `LowerCall_AIX`). The call instruction is a pseudo instruction that
+    // represents both the indirect branch and a load that restores the TOC
+    // pointer from the linkage area. The operand for the TOC restore is an add
+    // of the TOC save offset to the stack pointer. This must be the second
+    // operand: after the chain input but before any other variadic arguments.
+    if (Subtarget.is64BitELFABI() || Subtarget.isAIXABI()) {
+      const MCRegister StackPtrReg = Subtarget.getStackPointerRegister();
+
+      SDValue StackPtr = DAG.getRegister(StackPtrReg, RegVT);
       unsigned TOCSaveOffset = Subtarget.getFrameLowering()->getTOCSaveOffset();
       SDValue TOCOff = DAG.getIntPtrConstant(TOCSaveOffset, dl);
-      SDValue AddTOC = DAG.getNode(ISD::ADD, dl, MVT::i64, StackPtr, TOCOff);
+      SDValue AddTOC = DAG.getNode(ISD::ADD, dl, RegVT, StackPtr, TOCOff);
       Ops.push_back(AddTOC);
     }
 
     // Add the register used for the environment pointer.
     if (Subtarget.usesFunctionDescriptors() && !hasNest)
-      Ops.push_back(DAG.getRegister(PPC::X11, MVT::i64));
+      Ops.push_back(DAG.getRegister(Subtarget.getEnvironmentPointerRegister(),
+                                    RegVT));
+
 
     // Add CTR register as callee so a bctr can be emitted later.
     if (isTailCall)
       Ops.push_back(DAG.getRegister(IsPPC64 ? PPC::CTR8 : PPC::CTR, RegVT));
   }
 
   // If this is a tail call add stack pointer delta.
   if (isTailCall)
     Ops.push_back(DAG.getConstant(SPDiff, dl, MVT::i32));
 
   // Add argument registers to the end of the list so that they are known live
   // into the call.
   for (unsigned i = 0, e = RegsToPass.size(); i != e; ++i)
     Ops.push_back(DAG.getRegister(RegsToPass[i].first,
                                   RegsToPass[i].second.getValueType()));
 
   // We cannot add R2/X2 as an operand here for PATCHPOINT, because there is
   // no way to mark dependencies as implicit here.
   // We will add the R2/X2 dependency in EmitInstrWithCustomInserter.
   if ((Subtarget.is64BitELFABI() || Subtarget.isAIXABI()) && !isPatchPoint)
-    Ops.push_back(DAG.getRegister(IsPPC64 ? PPC::X2 : PPC::R2, RegVT));
+    Ops.push_back(DAG.getRegister(Subtarget.getTOCPointerRegister(), RegVT));
 
   // Add implicit use of CR bit 6 for 32-bit SVR4 vararg calls
   if (isVarArg && Subtarget.is32BitELFABI())
     Ops.push_back(DAG.getRegister(PPC::CR1EQ, MVT::i32));
 
   // Add a register mask operand representing the call-preserved registers.
   const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
   const uint32_t *Mask =
SDValue PPCTargetLowering::LowerCall_AIX(
 
   assert((CallConv == CallingConv::C ||
           CallConv == CallingConv::Cold ||
           CallConv == CallingConv::Fast) && "Unexpected calling convention!");
 
   if (isVarArg || isPatchPoint)
     report_fatal_error("This call type is unimplemented on AIX.");
 
-  if (!isFunctionGlobalAddress(Callee) && !isa<ExternalSymbolSDNode>(Callee))
-    report_fatal_error("Handling of indirect call is unimplemented!");
-
   const PPCSubtarget& Subtarget =
       static_cast<const PPCSubtarget&>(DAG.getSubtarget());
   if (Subtarget.hasQPX())
     report_fatal_error("QPX is not supported on AIX.");
   if (Subtarget.hasAltivec())
     report_fatal_error("Altivec support is unimplemented on AIX.");
 
   MachineFunction &MF = DAG.getMachineFunction();
for (CCValAssign &VA : ArgLocs) {
     if (VA.isRegLoc())
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), Arg));
 
     if (VA.isMemLoc())
       report_fatal_error("Handling of placing parameters on the stack is "
                          "unimplemented!");
   }
 
+  // For indirect calls, we need to save the TOC base to the stack for
+  // restoration after the call.
+  if (!isTailCall && !isPatchPoint &&
+      !isFunctionGlobalAddress(Callee) && !isa<ExternalSymbolSDNode>(Callee)) {
+    const MCRegister TOCBaseReg = Subtarget.getTOCPointerRegister();
+    const MCRegister StackPtrReg = Subtarget.getStackPointerRegister();
+    const MVT PtrVT = Subtarget.isPPC64() ? MVT::i64 : MVT::i32;
+    const unsigned TOCSaveOffset =
+                   Subtarget.getFrameLowering()->getTOCSaveOffset();
+
+    setUsesTOCBasePtr(DAG);
+    SDValue Val = DAG.getCopyFromReg(Chain, dl, TOCBaseReg, PtrVT);
+    SDValue PtrOff = DAG.getIntPtrConstant(TOCSaveOffset, dl);
+    SDValue StackPtr = DAG.getRegister(StackPtrReg, PtrVT);
+    SDValue AddPtr = DAG.getNode(ISD::ADD, dl, PtrVT, StackPtr, PtrOff);
+    Chain = DAG.getStore(Val.getValue(1), dl, Val, AddPtr,
+                         MachinePointerInfo::getStack(
+                                DAG.getMachineFunction(), TOCSaveOffset));
+  }
+
   // Build a sequence of copy-to-reg nodes chained together with token chain
   // and flag operands which copy the outgoing args into the appropriate regs.
   SDValue InFlag;
   for (auto Reg : RegsToPass) {
     Chain = DAG.getCopyToReg(Chain, dl, Reg.first, Reg.second, InFlag);
     InFlag = Chain.getValue(1);
   }
 

llvm/lib/Target/PowerPC/PPCInstrFormats.td

class XLForm_2_ext_and_DSForm_1<bits<6> opcode1, bits<10> xo1,
                                 InstrItinClass itin, list<dag> pattern>
   : XLForm_2_and_DSForm_1<opcode1, xo1, lk, opcode2, xo2,
                           OOL, IOL, asmstr, itin, pattern> {
   let BO = bo;
   let BI = bi;
   let BH = 0;
 }
 
+class XLForm_2_ext_and_DForm_1<bits<6> opcode1, bits<10>xo1, bits<5> bo,
+                               bits<5> bi, bits<2> bh, bit lk, bits<6> opcode2,
+                               dag OOL, dag IOL, string asmstr,
+                               InstrItinClass itin, list<dag> pattern>
+  : I2<opcode1, opcode2, OOL, IOL, asmstr, itin> {
+
+  bits<5>  RST;
+  bits<19> DS_RA;
+
+  let Pattern = pattern;
+
+  let Inst{6-10} = bo;
+  let Inst{11-15} = bi;
+  let Inst{16-18} = 0;  // unused.
+  let Inst{19-20} = bh;
+  let Inst{21-30} = xo1;
+  let Inst{31} = lk;
+
+  let Inst{38-42} = RST;
+  let Inst{43-47} = DS_RA{18-14};  // Register #
+  let Inst{48-61} = DS_RA{13-0};   // Displacement.
+}
+
 // 1.7.8 XFX-Form
 class XFXForm_1<bits<6> opcode, bits<10> xo, dag OOL, dag IOL, string asmstr,
                 InstrItinClass itin>
          : I<opcode, OOL, IOL, asmstr, itin> {
   bits<5>  RT;
   bits<10> SPR;
 
   let Inst{6-10}  = RT;

llvm/lib/Target/PowerPC/PPCInstrInfo.td

def TCRETURNai :PPCEmitTimePseudo<(outs), (ins abscalltarget:$func, i32imm:$offset),
                  "#TC_RETURNa $func $offset",
                  [(PPCtc_return (i32 imm:$func), imm:$offset)]>;
 
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, Uses = [RM] in
 def TCRETURNri : PPCEmitTimePseudo<(outs), (ins CTRRC:$dst, i32imm:$offset),
                  "#TC_RETURNr $dst $offset",
                  []>;
 
+let isCall = 1, PPC970_Unit = 7, isCodeGenOnly = 1,
+    Defs = [LR, R2], Uses = [CTR, RM], RST = 2 in {
+  def BCTRL_LWZinto_toc:
+    XLForm_2_ext_and_DForm_1<19, 528, 20, 0, 0, 1, 32, (outs),
+     (ins memrix:$src), "bctrl\n\tlwz 2, $src", IIC_BrB,
+     [(PPCbctrl_load_toc iaddrX4:$src)]>, Requires<[In32BitMode]>;
+
+}
+
 
 let isCodeGenOnly = 1 in {
 
 let isTerminator = 1, isBarrier = 1, PPC970_Unit = 7, isBranch = 1,
     isIndirectBranch = 1, isCall = 1, isReturn = 1, Uses = [CTR, RM]  in
 def TAILBCTR : XLForm_2_ext<19, 528, 20, 0, 0, (outs), (ins), "bctr", IIC_BrB,
                             []>, Requires<[In32BitMode]>;
 

llvm/lib/Target/PowerPC/PPCSubtarget.h

public:
 
   /// True if the ABI is descriptor based.
   bool usesFunctionDescriptors() const {
     // Both 32-bit and 64-bit AIX are descriptor based. For ELF only the 64-bit
     // v1 ABI uses descriptors.
     return isAIXABI() || (is64BitELFABI() && !isELFv2ABI());
   }
 
+  unsigned descriptorTOCAnchorOffset() const {
+    assert(usesFunctionDescriptors() &&
+           "should only be called when target uses descriptors.");
+    return IsPPC64 ? 8 : 4;
+  }
+
+  unsigned descriptorEnvironmentPointerOffset() const {
+    assert(usesFunctionDescriptors() &&
+           "should only be called when target uses descriptors.");
+    return IsPPC64 ? 16 : 8;
+  }
+
+  MCRegister getEnvironmentPointerRegister() const {
+    assert(usesFunctionDescriptors() &&
+           "should only be called when target uses descriptors.");
+     return IsPPC64 ? PPC::X11 : PPC::R11;
+  }
+
+  MCRegister getTOCPointerRegister() const {
+    assert((is64BitELFABI() || isAIXABI()) &&
+           "should only be called when target is TOC based ABI.");
+    return IsPPC64 ? PPC::X2 : PPC::R2;
+  }
+
+  MCRegister getStackPointerRegister() const {
+    return IsPPC64 ? PPC::X1 : PPC::R1;
+  }
+
   bool isXRaySupported() const override { return IsPPC64 && IsLittleEndian; }
 };
 } // End llvm namespace
 
 #endif

llvm/test/CodeGen/PowerPC/aix-trampoline.ll

+; RUN: not llc -mtriple powerpc-ibm-aix-xcoff < %s 2>&1 | FileCheck %s
+; RUN: not llc -mtriple powerpc64-ibm-aix-xcoff < %s 2>&1 | FileCheck %s
+
+; CHECK: LLVM ERROR: INIT_TRAMPOLINE operation not supported on AIX.
+
+define void @create_trampoline(i8* %buffer, i8* %nval) nounwind {
+entry:
+  call void @llvm.init.trampoline(i8* %buffer, i8* bitcast (i32 (i32)* @nested to i8*) , i8* %nval)
+  ret void
+}
+
+declare i32 @nested(i32);
+
+declare void @llvm.init.trampoline(i8*, i8*, i8*) nounwind

llvm/test/CodeGen/PowerPC/aix_indirect_call.ll

+; RUN: llc -verify-machineinstrs -mcpu=pwr4 -mattr=-altivec \
+; RUN:  -mtriple powerpc-ibm-aix-xcoff -stop-after=machine-cp < %s | \
+; RUN: FileCheck --check-prefixes=CHECKMIR,MIR32 %s
+
+; RUN: llc -verify-machineinstrs -mcpu=pwr4 -mattr=-altivec \
+; RUN:  -mtriple powerpc-ibm-aix-xcoff < %s | \
+; RUN: FileCheck --check-prefixes=CHECKASM,ASM32 %s
+
+; RUN: llc -verify-machineinstrs -mcpu=pwr4 -mattr=-altivec \
+; RUN:  -mtriple powerpc64-ibm-aix-xcoff -stop-after=machine-cp < %s | \
+; RUN: FileCheck --check-prefixes=CHECKMIR,MIR64 %s
+
+; RUN: llc -verify-machineinstrs -mcpu=pwr4 -mattr=-altivec \
+; RUN:  -mtriple powerpc64-ibm-aix-xcoff < %s | \
+; RUN: FileCheck --check-prefixes=CHECKASM,ASM64 %s
+
+; RUN: llc -verify-machineinstrs -mcpu=pwr4 -mattr=-altivec \
+; RUN: -mtriple powerpc-ibm-aix-xcoff  -filetype=obj < %s -o %t
+; RUN: llvm-objdump -d %t | FileCheck --check-prefixes=OBJ32 %s
+
+define signext i32 @callThroughPtr(i32 ()* nocapture) {
+  %2 = tail call signext i32 %0()
+  ret i32 %2
+}
+
+; CHECKMIR:   name:            callThroughPtr
+
+; MIR32:      liveins: $r3
+; MIR32:      ADJCALLSTACKDOWN 56, 0, implicit-def dead $r1, implicit $r1
+; MIR32-DAG:  STW $r2, 20, $r1
+; MIR32-DAG:  renamable $r11 = LWZ 8, renamable $r3 :: (dereferenceable invariant load 4 from %ir.0 + 8)
+; MIR32-DAG:  renamable $[[REG:r[0-9]+]] = LWZ 0, renamable $r3 :: (dereferenceable invariant load 4 from %ir.0)
+; MIR32-DAG:  $r2 = LWZ 4, killed renamable $r3 :: (dereferenceable invariant load 4 from %ir.0 + 4)
+; MIR32-DAG:  MTCTR killed renamable $[[REG]], implicit-def $ctr
+; MIR32-NEXT: BCTRL_LWZinto_toc 20, $r1, csr_aix32, implicit-def dead $lr, implicit-def dead $r2, implicit $ctr, implicit $rm, implicit $r11, implicit $r2, implicit-def $r1, implicit-def $r3
+; MIR32-NEXT: ADJCALLSTACKUP 56, 0, implicit-def dead $r1, implicit $r1
+
+; MIR64:      liveins: $x3
+; MIR64:      ADJCALLSTACKDOWN 112, 0, implicit-def dead $r1, implicit $r1
+; MIR64-DAG:  STD $x2, 40, $x1 :: (store 8 into stack + 40)
+; MIR64-DAG:  renamable $x11 = LD 16, renamable $x3 :: (dereferenceable invariant load 8 from %ir.0 + 16)
+; MIR64-DAG:  renamable $[[REG:x[0-9]+]] = LD 0, renamable $x3 :: (dereferenceable invariant load 8 from %ir.0)
+; MIR64-DAG:  $x2 = LD 8, killed renamable $x3 :: (dereferenceable invariant load 8 from %ir.0 + 8)
+; MIR64-DAG:  MTCTR8 killed renamable $[[REG]], implicit-def $ctr8
+; MIR64-NEXT: BCTRL8_LDinto_toc 40, $x1, csr_aix64, implicit-def dead $lr8, implicit-def dead $x2, implicit $ctr8, implicit $rm, implicit $x11, implicit $x2, implicit-def $r1, implicit-def $x3
+; MIR64-NEXT: ADJCALLSTACKUP 112, 0, implicit-def dead $r1, implicit $r1
+
+; CHECKASM-LABEL: .callThroughPtr:
+
+; ASM32:         stwu 1, -64(1)
+; ASM32-DAG:     lwz [[REG:[0-9]+]], 0(3)
+; ASM32-DAG:     stw 2, 20(1)
+; ASM32-DAG:     mtctr [[REG]]
+; ASM32-DAG:     lwz 11, 8(3)
+; ASM32-DAG:     lwz 2, 4(3)
+; ASM32-NEXT:    bctrl
+; ASM32-NEXT:    lwz 2, 20(1)
+; ASM32-NEXT:    addi 1, 1, 64
+
+; ASM64:            stdu 1, -112(1)
+; ASM64-DAG:        ld [[REG:[0-9]+]], 0(3)
+; ASM64-DAG:        std 2, 40(1)
+; ASM64-DAG:        mtctr [[REG]]
+; ASM64-DAG:        ld 11, 16(3)
+; ASM64-DAG:        ld 2, 8(3)
+; ASM64-NEXT:       bctrl
+; ASM64-NEXT:       ld 2, 40(1)
+; ASM64-NEXT:       addi 1, 1, 112
+
+; OBJ32-LABEL: .text:
+; OBJ32:         94 21 ff c0  stwu 1, -64(1)
+; OBJ32-DAG:     80 83 00 00  lwz [[REG:[0-9]+]], 0(3)
+; OBJ32-DAG:     90 41 00 14  stw 2, 20(1)
+; OBJ32-DAG:     7c 89 03 a6  mtctr [[REG]]
+; OBJ32-DAG:     81 63 00 08  lwz 11, 8(3)
+; OBJ32-DAG:     80 43 00 04  lwz 2, 4(3)
+; OBJ32-NEXT:    4e 80 04 21  bctrl
+; OBJ32-NEXT:    80 41 00 14  lwz 2, 20(1)
+; OBJ32-NEXT:    38 21 00 40  addi 1, 1, 64
+
+define void @callThroughPtrWithArgs(void (i32, i16, i64)* nocapture) {
+  tail call void %0(i32 signext 1, i16 zeroext 2, i64 3)
+  ret void
+}
+
+; CHECKMIR:   name:            callThroughPtrWithArgs
+
+; MIR32:      liveins: $r3
+; MIR32:      ADJCALLSTACKDOWN 56, 0, implicit-def dead $r1, implicit $r1
+; MIR32-DAG:  renamable $[[REG:r[0-9]+]] = LWZ 0, renamable $r3 :: (dereferenceable invariant load 4 from %ir.0)
+; MIR32-DAG:  MTCTR killed renamable $[[REG]], implicit-def $ctr
+; MIR32-DAG:  STW $r2, 20, $r1 :: (store 4 into stack + 20)
+; MIR32-DAG:  renamable $r11 = LWZ 8, renamable $r3 :: (dereferenceable invariant load 4 from %ir.0 + 8)
+; MIR32-DAG:  $r2 = LWZ 4, killed renamable $r3 :: (dereferenceable invariant load 4 from %ir.0 + 4)
+; MIR32-DAG:  $r3 = LI 1
+; MIR32-DAG:  $r4 = LI 2
+; MIR32-DAG:  $r5 = LI 0
+; MIR32-DAG:  $r6 = LI 3
+; MIR32-NEXT: BCTRL_LWZinto_toc 20, $r1, csr_aix32, implicit-def dead $lr, implicit-def dead $r2, implicit $ctr, implicit $rm, implicit $r11, implicit $r3, implicit $r4, implicit $r5, implicit $r6, implicit $r2, implicit-def $r1
+; MIR32-NEXT: ADJCALLSTACKUP 56, 0, implicit-def dead $r1, implicit $r1
+
+; MIR64:      liveins: $x3
+; MIR64:      ADJCALLSTACKDOWN 112, 0, implicit-def dead $r1, implicit $r1
+; MIR64-DAG:  renamable $[[REG:x[0-9]+]] = LD 0, renamable $x3 :: (dereferenceable invariant load 8 from %ir.0)
+; MIR64-DAG:  MTCTR8 killed renamable $[[REG]], implicit-def $ctr8
+; MIR64-DAG:  STD $x2, 40, $x1 :: (store 8 into stack + 40)
+; MIR64-DAG:  renamable $x11 = LD 16, renamable $x3 :: (dereferenceable invariant load 8 from %ir.0 + 16)
+; MIR64-DAG:  $x2 = LD 8, killed renamable $x3 :: (dereferenceable invariant load 8 from %ir.0 + 8)
+; MIR64-DAG:  $x3 = LI8 1
+; MIR64-DAG:  $x4 = LI8 2
+; MIR64-DAG:  $x5 = LI8 3
+; MIR64-NEXT: BCTRL8_LDinto_toc 40, $x1, csr_aix64, implicit-def dead $lr8, implicit-def dead $x2, implicit $ctr8, implicit $rm, implicit $x11, implicit $x3, implicit $x4, implicit $x5, implicit $x2, implicit-def $r1
+; MIR64-NEXT: ADJCALLSTACKUP 112, 0, implicit-def dead $r1, implicit $r1
+
+; CHECKASM-LABEL: .callThroughPtrWithArgs:
+
+; ASM32:      stwu 1, -64(1)
+; ASM32-DAG:  lwz [[REG:[0-9]+]], 0(3)
+; ASM32-DAG:  li 5, 0
+; ASM32-DAG:  li 6, 3
+; ASM32-DAG:  stw 2, 20(1)
+; ASM32-DAG:  mtctr [[REG]]
+; ASM32-DAG:  li 4, 2
+; ASM32-DAG:  lwz 11, 8(3)
+; ASM32-DAG:  lwz 2, 4(3)
+; ASM32-DAG:  li 3, 1
+; ASM32-NEXT: bctrl
+; ASM32-NEXT: lwz 2, 20(1)
+; ASM32-NEXT: addi 1, 1, 64
+
+; ASM64:            stdu 1, -112(1)
+; ASM64-DAG:        ld 4, 0(3)
+; ASM64-DAG:        li 5, 3
+; ASM64-DAG:        std 2, 40(1)
+; ASM64-DAG:        mtctr 4
+; ASM64-DAG:        li 4, 2
+; ASM64-DAG:        ld 11, 16(3)
+; ASM64-DAG:        ld 2, 8(3)
+; ASM64-DAG:        li 3, 1
+; ASM64-NEXT:       bctrl
+; ASM64-NEXT:       ld 2, 40(1)
+; ASM64-NEXT:       addi 1, 1, 112
+
+; OBJ32-LABEL: .callThroughPtrWithArgs:
+; OBJ32:         94 21 ff c0  stwu 1, -64(1)
+; OBJ32-DAG:     80 83 00 00  lwz [[REG:[0-9]+]], 0(3)
+; OBJ32-DAG:     38 a0 00 00  li 5, 0
+; OBJ32-DAG:     38 c0 00 03  li 6, 3
+; OBJ32-DAG:     90 41 00 14  stw 2, 20(1)
+; OBJ32-DAG:     7c 89 03 a6  mtctr [[REG]]
+; OBJ32-DAG:     38 80 00 02  li 4, 2
+; OBJ32-DAG:     81 63 00 08  lwz 11, 8(3)
+; OBJ32-DAG:     80 43 00 04  lwz 2, 4(3)
+; OBJ32-DAG:     38 60 00 01  li 3, 1
+; OBJ32-NEXT:    4e 80 04 21  bctrl
+; OBJ32-NEXT:    80 41 00 14  lwz 2, 20(1)
+; OBJ32-NEXT:    38 21 00 40  addi 1, 1, 64