[XCOFF][AIX] Add Global Variables Directly to TOC for 32 bit AIX

[sidbav]

Summary

This patch implements the backend implementation of adding global variables directly to the table of contents, rather than adding the address of the variable to the table of contents.

Currently, this patch will look for the "toc-data" attribute on symbols in LLVM IR, and then add those symbols to the toc. This patch assumes that the symbols which have the toc-data attribute can be added to the TOC with no issues (strictly aligned, pointer sized).

ATM, this is implemented for 32 bit AIX.

[sfertile]

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

2156

I think we need to explain why we return early here. Maybe change , return
to
'it needs to be emitted later, when we emit the .toc section.'

2273–2274

Need to update this comment. if there are no functions in the module, and no toc-data definitions

2309

Please address the clang-tidy warning.

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

5603

Since all we need is true/false for if the global has the toc-data attribute I think we should factor all this casting into an appropriately named helper function. Then we can change the replaceWithLWZtoc lambda to take an opcode argument and reuse it for transforming the node.

A helper along the lines of

bool hasTocDataAttr(SDValue Val) {
   GlobalAddressSDNode *GA = dyn_cast<GlobalAddressSDNode>(Val);
   if (!GA)
     return false;

   const GlobalVariable *GV = dyn_cast_or_null<GlobalVariable>(GA->getGlobal());
  if (!GV)
    return false;

  if (!GV->hasAttribute("toc-data"))
    return false;


  // TODO assert on alignment and size ...?

  return true;
}

5607–5609

Rename this and add a new parameter auto replaceWith = [this, &dl](unsigned Opcode, SDNode *TocEntry).
Then change the hardcoded LWZtoc to the opcode argument, and move where we do the toc-data transformation down to existing if (isAIXABI && CModel == CodeModel::Small) block on line 5584.

llvm/test/CodeGen/PowerPC/basic-toc-data-2.ll

12 ↗

(On Diff #340064)

There is trailing whitespace on this line.

[sfertile]

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

2207

No need to convert from GlobalObject --> GlobalValue --> GlobalVariable. Just try to dyn_cast from GlobalObject -> GlobalVariable

2252

Same: no need to convert the GlobalObject to a GlobalValue before tying to cast to a GlobalVariable.

[sidbav]

Addressing Review Comments

[sfertile]

Thanks for the quick updates. Patch is getting really close I think.

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

2206

Real minor nit: missing the period at the end of the comment.

2250

ditto.

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

866

Missing the period.

869

Real minor nit: Probably useful to use 'load address' instead of 'la', and missing the period at the end of the sentance.

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

449

I'm not sure of the limitations of the option in XL that enables toc-data. For example if it can it be used to put a vector with an alignment/size of 16 in the TOC, or structures, arrays, etc; For now we don't have any testing coverage for these things so I think we should add some defensive errors here.

• If the alignment is greater then the alignment of a toc-entry it should be an error.
• If the size of the type is larger then a toc-entry it should be an error.
• For now, if the GlobalVariable is an array type or aggregate type we should error.

[sfertile]

llvm/test/CodeGen/PowerPC/toc-data.ll

4

Add a run step which compiles to assembly, and the associated checks.

[sidbav]

Addressing Sean's comments

[sfertile]

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

468

Minor nit: since most of these limitations are because this feature is a work in progress, we should change the asserts to report_fatal_error. The assert that the type is sized can stay an assert because we expect that to stay a limitation, and because I would expect the user facing option implementation to have already checked that and emitted some kind of error from the front-end.

484

I would split out 'hasCommonLinkage() into its own assert with the string saying something along the lines of tentative definitions cannot have mapping class XMC_TD`.

The local linkage limitation I believe we hit when compiling spec right? In that case maybe change the message to 'local linkage globals cannot have mapping class XMC_TD`. If we didn't, then make sure to test with the system assembler first to make sure it is invalid to define a local linkage global in the .toc.

llvm/test/CodeGen/PowerPC/basic-toc-data-2.ll

1 ↗

(On Diff #340958)

Minor nit: name this file basic-toc-data-extern.ll

2 ↗

(On Diff #340958)

Since this patch is only going to add assembly support for this feature, we need to also compile this test to an object file and make sure we fail gracefully. Add a run-step:

; RUN: not --crash llc -filetype=obj -mtriple powerpc-ibm-aix-xcoff  \
; RUN:                 -verify-machineinstrs < %s 2>&1 | \
; RUN:   FileCheck %s --check-prefix=OBJ

with a check: ; OBJ: LLVM ERROR: toc-data not yet supported when writing object files.

And in llvm/lib/MC/XCOFFObjectWriter.cpp add

@@ -439,6 +441,10 @@ void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
       TargetObjectWriter->getRelocTypeAndSignSize(Target, Fixup, IsPCRel);
 
   const MCSectionXCOFF *SymASec = getContainingCsect(cast<MCSymbolXCOFF>(SymA));
+
+  if (SymASec->isCsect() && SymASec->getMappingClass() == XCOFF::XMC_TD)
+    report_fatal_error("toc-data not yet supported when writing object files.");
+
   assert(SectionMap.find(SymASec) != SectionMap.end() &&
          "Expected containing csect to exist in map.");

to emit the error.

llvm/test/CodeGen/PowerPC/basic-toc-data.ll

1 ↗

(On Diff #340958)

minor nit: name this file basic-toc-data-def.ll

2 ↗

(On Diff #340958)

We need to similarly fail gracefully for this test when editing an object file, and add test coverage that we do so.

Run-step:

; RUN: not --crash llc -filetype=obj -mtriple powerpc-ibm-aix-xcoff  \
; RUN:                 -verify-machineinstrs < %s 2>&1 | \
; RUN:   FileCheck %s --check-prefix=OBJ

and to get the error emitted, add a switch case for XMC_TD: in XCOFFObjectWriter::getCsectGroup which emits the error.

llvm/test/CodeGen/PowerPC/toc-data.ll

22

minor nit: indent the instructions 2 spaces past the label, ditto for the other checks below.

[sfertile]

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

484

I did some poking, and the problem is that clang use .lcomm to emit local linkage for internal/private linkage globals. I believe we will be able to support local linkage toc-data by skipping the lcomm directive like

.csect i[TD],2
 .align 2
 .vbyte 4, <val>

So make the local/private linkage a fatal error instead of an assert (since its going to be a temporary limitation)

llvm/test/CodeGen/PowerPC/toc-data.ll

74

minor nit: name should be read_i32_local_linkage()

Also please move this into its own file, and put the toc-data attribute on ilocal and check for an error message.

[sidbav]

replacing some of the asserts with report_fatal_error, renaming some of the lit tests, and create a new one for local linkage

[sfertile]

Great work Sidharth, a couple minor nits but otherwise the patch LGTM.

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

456

Minor nit: reword to "GlobalVariables with an alignment requirement stricter then 4-bytes not supported by the toc data transformation"

465

minor nit: 'size must be know' -> 'size must be known'.

489

minor nit: pare down to just "Tentative definitions cannot have mapping class XMC_TD.".

[sidbav]

Addressed the all of the remaining comments in the commit.

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

TargetLoweringObjectFileXCOFF::getTargetSymbol(const GlobalValue *GV,
   // a declaration, a function descriptor, or a common symbol.
   // If a GV represents a GlobalVariable and -fdata-sections is enabled, we
   // also return a qualname so that a label symbol could be avoided.
   // It is inherently ambiguous when the GO represents the address of a
   // function, as the GO could either represent a function descriptor or a
   // function entry point. We choose to always return a function descriptor
   // here.
   if (const GlobalObject *GO = dyn_cast<GlobalObject>(GV)) {
+    if (const GlobalVariable *GVar = dyn_cast<GlobalVariable>(GV))
+      if (GVar->hasAttribute("toc-data"))
+        return cast<MCSectionXCOFF>(
+                   SectionForGlobal(GVar, SectionKind::getData(), TM))
+            ->getQualNameSymbol();
+
     if (GO->isDeclarationForLinker())
       return cast<MCSectionXCOFF>(getSectionForExternalReference(GO, TM))
           ->getQualNameSymbol();
 
     SectionKind GOKind = getKindForGlobal(GO, TM);
     if (GOKind.isText())
       return cast<MCSectionXCOFF>(
                  getSectionForFunctionDescriptor(cast<Function>(GO), TM))

 }
 
 MCSection *TargetLoweringObjectFileXCOFF::getExplicitSectionGlobal(
     const GlobalObject *GO, SectionKind Kind, const TargetMachine &TM) const {
   if (!GO->hasSection())
     report_fatal_error("#pragma clang section is not yet supported");
 
   StringRef SectionName = GO->getSection();
+
+  // Handle the XCOFF::TD case first, then deal with the rest.
+  if (const GlobalVariable *GVar = dyn_cast<GlobalVariable>(GO))
+    if (GVar->hasAttribute("toc-data"))
+      return getContext().getXCOFFSection(
+          SectionName, Kind,
+          XCOFF::CsectProperties(/*MappingClass*/ XCOFF::XMC_TD, XCOFF::XTY_SD),
+          /* MultiSymbolsAllowed*/ true);
+
   XCOFF::StorageMappingClass MappingClass;
   if (Kind.isText())
     MappingClass = XCOFF::XMC_PR;
   else if (Kind.isData() || Kind.isReadOnlyWithRel() || Kind.isBSS())
     MappingClass = XCOFF::XMC_RW;
   else if (Kind.isReadOnly())
     MappingClass = XCOFF::XMC_RO;
   else
MCSection *TargetLoweringObjectFileXCOFF::getSectionForExternalReference(
   // Externals go into a csect of type ER.
   return getContext().getXCOFFSection(
       Name, SectionKind::getMetadata(),
       XCOFF::CsectProperties(SMC, XCOFF::XTY_ER));
 }
 
 MCSection *TargetLoweringObjectFileXCOFF::SelectSectionForGlobal(
     const GlobalObject *GO, SectionKind Kind, const TargetMachine &TM) const {
+  // Handle the XCOFF::TD case first, then deal with the rest.
+  if (const GlobalVariable *GVar = dyn_cast<GlobalVariable>(GO))
+    if (GVar->hasAttribute("toc-data")) {
+      SmallString<128> Name;
+      getNameWithPrefix(Name, GO, TM);
+      return getContext().getXCOFFSection(
+          Name, Kind, XCOFF::CsectProperties(XCOFF::XMC_TD, XCOFF::XTY_SD),
+          /* MultiSymbolsAllowed*/ true);
+    }
+
   // Common symbols go into a csect with matching name which will get mapped
   // into the .bss section.
   // Zero-initialized local TLS symbols go into a csect with matching name which
   // will get mapped into the .tbss section.
   if (Kind.isBSSLocal() || GO->hasCommonLinkage() || Kind.isThreadBSSLocal()) {
     SmallString<128> Name;
     getNameWithPrefix(Name, GO, TM);
     XCOFF::StorageMappingClass SMC = Kind.isBSSLocal() ? XCOFF::XMC_BS

llvm/lib/MC/MCSectionXCOFF.cpp

if (getKind().isThreadData()) {
     printCsectDirective(OS);
     return;
   }
 
   if (getKind().isData()) {
     switch (getMappingClass()) {
     case XCOFF::XMC_RW:
     case XCOFF::XMC_DS:
+    case XCOFF::XMC_TD:
       printCsectDirective(OS);
       break;
     case XCOFF::XMC_TC:
     case XCOFF::XMC_TE:
       break;
     case XCOFF::XMC_TC0:
       OS << "\t.toc\n";
       break;
     default:
       report_fatal_error(
           "Unhandled storage-mapping class for .data csect.");
     }
     return;
   }
 
+  if (isCsect() && getMappingClass() == XCOFF::XMC_TD) {
+    assert((getKind().isBSSExtern() || getKind().isBSSLocal()) &&
+           "Unexepected section kind for toc-data");
+    printCsectDirective(OS);
+    return;
+  }
   // Common csect type (uninitialized storage) does not have to print csect
   // directive for section switching.
   if (isCsect() && getCSectType() == XCOFF::XTY_CM) {
     assert((getMappingClass() == XCOFF::XMC_RW ||
             getMappingClass() == XCOFF::XMC_BS ||
             getMappingClass() == XCOFF::XMC_UL) &&
            "Generated a storage-mapping class for a common/bss/tbss csect we "
            "don't "

llvm/lib/MC/XCOFFObjectWriter.cpp

case XCOFF::XMC_TC0:
     return TOCCsects;
   case XCOFF::XMC_TC:
   case XCOFF::XMC_TE:
     assert(XCOFF::XTY_SD == MCSec->getCSectType() &&
            "Only an initialized csect can contain TC entry.");
     assert(!TOCCsects.empty() &&
            "We should at least have a TOC-base in this CsectGroup.");
     return TOCCsects;
+  case XCOFF::XMC_TD:
+    report_fatal_error("toc-data not yet supported when writing object files.");
   default:
     report_fatal_error("Unhandled mapping of csect to section.");
   }
 }
 
 static MCSectionXCOFF *getContainingCsect(const MCSymbolXCOFF *XSym) {
   if (XSym->isDefined())
     return cast<MCSectionXCOFF>(XSym->getFragment()->getParent());
bool IsPCRel = Backend.getFixupKindInfo(Fixup.getKind()).Flags &
                  MCFixupKindInfo::FKF_IsPCRel;
 
   uint8_t Type;
   uint8_t SignAndSize;
   std::tie(Type, SignAndSize) =
       TargetObjectWriter->getRelocTypeAndSignSize(Target, Fixup, IsPCRel);
 
   const MCSectionXCOFF *SymASec = getContainingCsect(cast<MCSymbolXCOFF>(SymA));
+
+  if (SymASec->isCsect() && SymASec->getMappingClass() == XCOFF::XMC_TD)
+    report_fatal_error("toc-data not yet supported when writing object files.");
+
   assert(SectionMap.find(SymASec) != SectionMap.end() &&
          "Expected containing csect to exist in map.");
 
   const uint32_t Index = getIndex(SymA, SymASec);
   if (Type == XCOFF::RelocationType::R_POS)
     // The FixedValue should be symbol's virtual address in this object file
     // plus any constant value that we might get.
     FixedValue = getVirtualAddress(SymA, SymASec) + Target.getConstant();

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

private:
 
   // Record a list of GlobalAlias associated with a GlobalObject.
   // This is used for AIX's extra-label-at-definition aliasing strategy.
   DenseMap<const GlobalObject *, SmallVector<const GlobalAlias *, 1>>
       GOAliasMap;
 
   void emitTracebackTable();
 
+  SmallVector<const GlobalVariable *, 8> TOCDataGlobalVars;
+
+  void emitGlobalVariableHelper(const GlobalVariable *);
+
 public:
   PPCAIXAsmPrinter(TargetMachine &TM, std::unique_ptr<MCStreamer> Streamer)
       : PPCAsmPrinter(TM, std::move(Streamer)) {
     if (MAI->isLittleEndian())
       report_fatal_error(
           "cannot create AIX PPC Assembly Printer for a little-endian target");
   }
 
case PPC::LWZtoc: {
     // '.LTOC' to manifest the toc-relative offset.
     const MCExpr *PB = MCSymbolRefExpr::create(
         OutContext.getOrCreateSymbol(Twine(".LTOC")), OutContext);
     Exp = MCBinaryExpr::createSub(Exp, PB, OutContext);
     TmpInst.getOperand(1) = MCOperand::createExpr(Exp);
     EmitToStreamer(*OutStreamer, TmpInst);
     return;
   }
+  case PPC::ADDItoc: {
+    assert(IsAIX && TM.getCodeModel() == CodeModel::Small &&
+           "Operand only valid in AIX 32 bit mode");
+
+    // Transform %rN = ADDItoc @op1, %r2.
+    LowerPPCMachineInstrToMCInst(MI, TmpInst, *this);
+
+    // Change the opcode to load address.
+    TmpInst.setOpcode(PPC::LA);
+
+    const MachineOperand &MO = MI->getOperand(1);
+    assert(MO.isGlobal() && "Invalid operand for ADDItoc.");
+
+    // Map the operand to its corresponding MCSymbol.
+    const MCSymbol *const MOSymbol = getMCSymbolForTOCPseudoMO(MO, *this);
+
+    const MCExpr *Exp =
+        MCSymbolRefExpr::create(MOSymbol, MCSymbolRefExpr::VK_None, OutContext);
+
+    TmpInst.getOperand(1) = TmpInst.getOperand(2);
+    TmpInst.getOperand(2) = MCOperand::createExpr(Exp);
+    EmitToStreamer(*OutStreamer, TmpInst);
+    return;
+  }
   case PPC::LDtocJTI:
   case PPC::LDtocCPT:
   case PPC::LDtocBA:
   case PPC::LDtoc: {
     // Transform %x3 = LDtoc @min1, %x2
     LowerPPCMachineInstrToMCInst(MI, TmpInst, *this);
 
     // Change the opcode to LD.
return StringSwitch<bool>(GV->getName())
       .Default(false);
 }
 
 void PPCAIXAsmPrinter::emitGlobalVariable(const GlobalVariable *GV) {
   // Special LLVM global arrays have been handled at the initialization.
   if (isSpecialLLVMGlobalArrayToSkip(GV) || isSpecialLLVMGlobalArrayForStaticInit(GV))
     return;
 
+  // If the Global Variable has the toc-data attribute, it needs to be emitted
+  // when we emit the .toc section.
+  if (GV->hasAttribute("toc-data")) {
+    TOCDataGlobalVars.push_back(GV);
+    return;
+  }
+
+  emitGlobalVariableHelper(GV);
+}
+
+void PPCAIXAsmPrinter::emitGlobalVariableHelper(const GlobalVariable *GV) {
   assert(!GV->getName().startswith("llvm.") &&
          "Unhandled intrinsic global variable.");
 
   if (GV->hasComdat())
     report_fatal_error("COMDAT not yet supported by AIX.");
 
   MCSymbolXCOFF *GVSym = cast<MCSymbolXCOFF>(getSymbol(GV));
 
void PPCAIXAsmPrinter::emitFunctionEntryLabel() {
   // Emit aliasing label for function entry point label.
   llvm::for_each(
       GOAliasMap[&MF->getFunction()], [this](const GlobalAlias *Alias) {
         OutStreamer->emitLabel(
             getObjFileLowering().getFunctionEntryPointSymbol(Alias, TM));
       });
 }
 
 void PPCAIXAsmPrinter::emitEndOfAsmFile(Module &M) {
-  // If there are no functions in this module, we will never need to reference
-  // the TOC base.
-  if (M.empty())
+  // If there are no functions and there are no toc-data definitions in this
+  // module, we will never need to reference the TOC base.
+  if (M.empty() && TOCDataGlobalVars.empty())
     return;
 
   // Switch to section to emit TOC base.
   OutStreamer->SwitchSection(getObjFileLowering().getTOCBaseSection());
 
   PPCTargetStreamer *TS =
       static_cast<PPCTargetStreamer *>(OutStreamer->getTargetStreamer());
 
if (I.first.second == MCSymbolRefExpr::VariantKind::VK_PPC_AIX_TLSGDM) {
           getObjFileLowering().getSectionForTOCEntry(I.first.first, TM));
     }
     OutStreamer->SwitchSection(TCEntry);
 
     OutStreamer->emitLabel(I.second);
     if (TS != nullptr)
       TS->emitTCEntry(*I.first.first, I.first.second);
   }
+
+  for (const auto *GV : TOCDataGlobalVars)
+    emitGlobalVariableHelper(GV);
 }
 
 bool PPCAIXAsmPrinter::doInitialization(Module &M) {
   const bool Result = PPCAsmPrinter::doInitialization(M);
 
   auto setCsectAlignment = [this](const GlobalObject *GO) {
     // Declarations have 0 alignment which is set by default.
     if (GO->isDeclarationForLinker())

llvm/lib/Target/PowerPC/PPCISelDAGToDAG.cpp

if (PPCLowering->getPointerTy(CurDAG->getDataLayout()) == MVT::i32) {
       BuildMI(FirstMBB, MBBI, dl, TII.get(PPC::MFLR8), GlobalBaseReg);
     }
   }
   return CurDAG->getRegister(GlobalBaseReg,
                              PPCLowering->getPointerTy(CurDAG->getDataLayout()))
       .getNode();
 }
 
+// Check if a SDValue has the toc-data attribute.
+static bool hasTocDataAttr(SDValue Val, unsigned PointerSize) {
+  GlobalAddressSDNode *GA = dyn_cast<GlobalAddressSDNode>(Val);
+  if (!GA)
+    return false;
+
+  const GlobalVariable *GV = dyn_cast_or_null<GlobalVariable>(GA->getGlobal());
+  if (!GV)
+    return false;
+
+  if (!GV->hasAttribute("toc-data"))
+    return false;
+
+  // TODO: These asserts should be updated as more support for the toc data
+  // transformation is added (64 bit, struct support, etc.).
+
+  assert(PointerSize == 4 && "Only 32 Bit Codegen is currently supported by "
+                             "the toc data transformation.");
+
+  assert(PointerSize >= GV->getAlign().valueOrOne().value() &&
+         "GlobalVariables with an alignment requirement stricter then 4-bytes "
+         "not supported by the toc data transformation.");
+
+  Type *PtrType = GV->getType();
+  assert(PtrType->isPointerTy() &&
+         "GlobalVariables always have pointer type!.");
+
+  Type *GVType = dyn_cast<PointerType>(PtrType)->getElementType();
+
+  assert(GVType->isSized() && "A GlobalVariable's size must be known to be "
+                              "supported by the toc data transformation.");
+
+  if (GVType->isVectorTy())
+    report_fatal_error("A GlobalVariable of Vector type is not currently "
+                       "supported by the toc data transformation.");
+
+  if (GVType->isArrayTy())
+    report_fatal_error("A GlobalVariable of Array type is not currently "
+                       "supported by the toc data transformation.");
+
+  if (GVType->isStructTy())
+    report_fatal_error("A GlobalVariable of Struct type is not currently "
+                       "supported by the toc data transformation.");
+
+  assert(GVType->getPrimitiveSizeInBits() <= PointerSize * 8 &&
+         "A GlobalVariable with size larger than 32 bits is not currently "
+         "supported by the toc data transformation.");
+
+  if (GV->hasLocalLinkage() || GV->hasPrivateLinkage())
+    report_fatal_error("A GlobalVariable with private or local linkage is not "
+                       "currently supported by the toc data transformation.");
+
+  assert(!GV->hasCommonLinkage() &&
+         "Tentative definitions cannot have the mapping class XMC_TD.");
+
+  return true;
+}
+
 /// isInt32Immediate - This method tests to see if the node is a 32-bit constant
 /// operand. If so Imm will receive the 32-bit value.
 static bool isInt32Immediate(SDNode *N, unsigned &Imm) {
   if (N->getOpcode() == ISD::Constant && N->getValueType(0) == MVT::i32) {
     Imm = cast<ConstantSDNode>(N)->getZExtValue();
     return true;
   }
   return false;
assert(!(CModel == CodeModel::Tiny || CModel == CodeModel::Kernel) &&
            "PowerPC doesn't support tiny or kernel code models.");
 
     if (isAIXABI && CModel == CodeModel::Medium)
       report_fatal_error("Medium code model is not supported on AIX.");
 
     // For 64-bit small code model, we allow SelectCodeCommon to handle this,
     // selecting one of LDtoc, LDtocJTI, LDtocCPT, and LDtocBA.
     if (isPPC64 && CModel == CodeModel::Small)
       break;
 
     // Handle 32-bit small code model.
     if (!isPPC64) {
-      // Transforms the ISD::TOC_ENTRY node to a PPCISD::LWZtoc.
-      auto replaceWithLWZtoc = [this, &dl](SDNode *TocEntry) {
+      // Transforms the ISD::TOC_ENTRY node to passed in Opcode, either
+      // PPC::ADDItoc, or PPC::LWZtoc
+      auto replaceWith = [this, &dl](unsigned OpCode, SDNode *TocEntry) {
         SDValue GA = TocEntry->getOperand(0);
         SDValue TocBase = TocEntry->getOperand(1);
-        SDNode *MN = CurDAG->getMachineNode(PPC::LWZtoc, dl, MVT::i32, GA,
-                                            TocBase);
+        SDNode *MN = CurDAG->getMachineNode(OpCode, dl, MVT::i32, GA, TocBase);
         transferMemOperands(TocEntry, MN);
         ReplaceNode(TocEntry, MN);
       };
 
       if (isELFABI) {
         assert(TM.isPositionIndependent() &&
                "32-bit ELF can only have TOC entries in position independent"
                " code.");
         // 32-bit ELF always uses a small code model toc access.
-        replaceWithLWZtoc(N);
+        replaceWith(PPC::LWZtoc, N);
         return;
       }
 
       if (isAIXABI && CModel == CodeModel::Small) {
-        replaceWithLWZtoc(N);
+        if (hasTocDataAttr(N->getOperand(0),
+                           CurDAG->getDataLayout().getPointerSize()))
+          replaceWith(PPC::ADDItoc, N);
+        else
+          replaceWith(PPC::LWZtoc, N);
+
         return;
       }
     }
 
     assert(CModel != CodeModel::Small && "All small code models handled.");
 
     assert((isPPC64 || (isAIXABI && !isPPC64)) && "We are dealing with 64-bit"
            " ELF/AIX or 32-bit AIX in the following.");

llvm/lib/Target/PowerPC/PPCInstrInfo.td

 def LWZtocL : PPCEmitTimePseudo<(outs gprc:$rD), (ins tocentry32:$disp, gprc_nor0:$reg),
                     "#LWZtocL",
                     [(set i32:$rD,
                       (PPCtoc_entry tglobaladdr:$disp, i32:$reg))]>;
 def ADDIStocHA : PPCEmitTimePseudo<(outs gprc:$rD), (ins gprc_nor0:$reg, tocentry32:$disp),
                        "#ADDIStocHA",
                        [(set i32:$rD,
                          (PPCtoc_entry i32:$reg, tglobaladdr:$disp))]>;
+// Local Data Transform
+def ADDItoc : PPCEmitTimePseudo<(outs gprc:$rD), (ins tocentry32:$disp, gprc:$reg),
+                   "#ADDItoc",
+                   [(set i32:$rD,
+                     (PPCtoc_entry tglobaladdr:$disp, i32:$reg))]>;
 
 // Get Global (GOT) Base Register offset, from the word immediately preceding
 // the function label.
 def UpdateGBR : PPCEmitTimePseudo<(outs gprc:$rD, gprc:$rT), (ins gprc:$rI), "#UpdateGBR", []>;
 
 // Pseudo-instruction marked for deletion. When deleting the instruction would
 // cause iterator invalidation in MIR transformation passes, this pseudo can be
 // used instead. It will be removed unconditionally at pre-emit time (prior to

llvm/test/CodeGen/PowerPC/basic-toc-data-def.ll

+; RUN: llc -mtriple powerpc-ibm-aix-xcoff  -verify-machineinstrs < %s | FileCheck %s
+; RUN: not --crash llc -filetype=obj -mtriple powerpc-ibm-aix-xcoff  \
+; RUN:                 -verify-machineinstrs < %s 2>&1 | \
+; RUN:   FileCheck %s --check-prefix=OBJ
+
+@i = global i32 55, align 4 #0
+
+attributes #0 = { "toc-data" }
+; CHECK:            .toc
+; CHECK-NEXT:       .csect i[TD],2
+; CHECK-NEXT:       .globl i[TD]
+; CHECK-NEXT:       .align 2
+; CHECK-NEXT:       .vbyte 4, 55
+
+; OBJ: LLVM ERROR:  toc-data not yet supported when writing object files.

llvm/test/CodeGen/PowerPC/basic-toc-data-extern.ll

+; RUN: llc -mtriple powerpc-ibm-aix-xcoff -verify-machineinstrs < %s | FileCheck %s
+; RUN: not --crash llc -filetype=obj -mtriple powerpc-ibm-aix-xcoff  \
+; RUN:                 -verify-machineinstrs < %s 2>&1 | \
+; RUN:   FileCheck %s --check-prefix=OBJ
+
+@i = external global i32, align 4  #0
+
+; Function Attrs: noinline nounwind optnone
+define i32* @get() {
+  entry:
+    ret i32* @i
+}
+
+; CHECK:        la 3, i[TD](2)
+; CHECK:        .toc
+; CHECK-NEXT:   .extern i[TD]
+
+; OBJ: LLVM ERROR:  toc-data not yet supported when writing object files.
+
+attributes #0 = { "toc-data" }

llvm/test/CodeGen/PowerPC/basic-toc-data-local-linkage.ll

+; RUN: not --crash llc  -mtriple powerpc-ibm-aix-xcoff  -verify-machineinstrs \
+; RUN:     < %s 2>&1 | FileCheck %s
+
+@ilocal = internal global i32 0, align 4 #0
+
+define dso_local i32 @read_i32_local_linkage() {
+  entry:
+    %0 = load i32, i32* @ilocal, align 4
+    ret i32 %0
+}
+
+; CHECK: LLVM ERROR: A GlobalVariable with private or local linkage is not currently supported by the toc data transformation.
+
+attributes #0 = { "toc-data" }

llvm/test/CodeGen/PowerPC/toc-data.ll

+; RUN: llc -mtriple powerpc-ibm-aix-xcoff -verify-machineinstrs < %s \
+; RUN:     -stop-before=ppc-ctr-loops-verify | FileCheck %s
+; RUN: llc -mtriple powerpc-ibm-aix-xcoff -verify-machineinstrs < %s | FileCheck %s --check-prefix TEST
+
+@i = dso_local global i32 0, align 4 #0
+@d = dso_local local_unnamed_addr global double 3.141590e+00, align 8
+@f = dso_local local_unnamed_addr global float 0x4005BE76C0000000, align 4 #0
+@ll = dso_local local_unnamed_addr global i64 55, align 8
+@ilocal = internal global i32 0, align 4
+
+define dso_local void @write_int(i32 signext %in) {
+  entry:
+    store i32 %in, i32* @i, align 4
+    ret void
+}
+; CHECK: name:            write_int
+; CHECK:      %[[SCRATCH:[0-9]+]]:gprc_and_gprc_nor0 = ADDItoc @i, $r2
+; CHECK-NEXT: STW %{{[0-9]+}}, 0, killed %[[SCRATCH]] :: (store 4 into @i)
+
+; TEST:         .write_int:
+; TEST:           la 4, i[TD](2)
+; TEST-NEXT:      stw 3, 0(4)
+
+define dso_local i64 @read_ll() {
+  entry:
+    %0 = load i64, i64* @ll, align 8
+    ret i64 %0
+}
+; CHECK: name:            read_ll
+; CHECK: LWZtoc @ll, $r2 :: (load 4 from got)
+
+; TEST:       .read_ll:
+; TEST:         lwz 4, L..C0(2)
+; TEST-NEXT:    lwz 3, 0(4)
+; TEST-NEXT:    lwz 4, 4(4)
+
+define dso_local float @read_float() {
+  entry:
+    %0 = load float, float* @f, align 4
+    ret float %0
+}
+; CHECK: name:            read_float
+; CHECK: %[[SCRATCH:[0-9]+]]:gprc_and_gprc_nor0 = ADDItoc @f, $r2
+; CHECK: %{{[0-9]+}}:f4rc = LFS 0, killed %[[SCRATCH]] :: (dereferenceable load 4 from @f)
+
+; TEST:       .read_float:
+; TEST:         la 3, f[TD](2)
+; TEST-NEXT:    lfs 1, 0(3)
+
+define dso_local void @write_double(double %in) {
+  entry:
+    store double %in, double* @d, align 8
+    ret void
+}
+; CHECK: name:            write_double
+; CHECK: LWZtoc @d, $r2 :: (load 4 from got)
+
+; TEST:       .write_double
+; TEST:         lwz 3, L..C1(2)
+; TEST-NEXT:    stfd 1, 0(3)
+
+define dso_local nonnull i32* @addr() {
+  entry:
+    ret i32* @i
+}
+; CHECK: name:            addr
+; CHECK:       %[[SCRATCH:[0-9]+]]:gprc = ADDItoc @i, $r2
+; CHECK-NEXT:  $r3 = COPY %[[SCRATCH]]
+
+; TEST:       .addr
+; TEST:         la 3, i[TD](2)
+
+
+attributes #0 = { "toc-data" }