[PowerPC] Implement R_PPC64_REL24_NOTOC local calls. callee has a TOC

[NeHuang]

Summary

The PC Relative code now allows for calls that are marked with the relocation R_PPC64_REL24_NOTOC. This indicates that the caller does not have a valid TOC pointer in R2 and does not require R2 to be restored after the call. This patch is added to support local calls to callees that require a TOC

[MaskRay]

lld/ELF/Arch/PPC64.cpp

1054

misaligned

lld/ELF/Target.h

216

Make it imperative, i.e.

// Write a prefixed instruction, which is a 4-byte prefix followed by a 4-byte instruction (regardless of endianness).

"As a result, we need to shift the pieces around on little endian machines." is not needed.

lld/test/ELF/ppc64-pcrel-call-to-toc.s

19

##

23

-NEXT:

[NeHuang]

Addressed comments from MaskRay.

[MaskRay]

lld/ELF/Thunks.cpp

1012

What if (s.stOther >> 5) == 1?

lld/test/ELF/ppc64-pcrel-call-to-toc.s

19

The test is created to check that can be omitted. local is not accurate.

1. When a function without TOC accesses a function using TOC, an r12 setup stub is inserted.

[sfertile]

lld/ELF/Thunks.cpp

283

minor nit:

// When the caller does not use TOC and does not preserve R2 makes a local call to a callee that requires a TOC pointer ...

-->

// When a caller that does not maintain a toc-pointer  performs a local call to a callee which requires a toc-pointer ...

852

fatal error if the offset is to large to encode in the instruction.

856

Comment is wrong. destination.getVA() - getThunkTargetSym()->getVA() is the pc-relative offset to the function (func@pcrel), not the offset to the functions plt entry.

862

Good name. We could shorten to '__gep_setup' as I think anyone looking at a PPC64 binary is likely to understand gep is short for global entry point.

[sfertile]

lld/ELF/Thunks.cpp

856

Sorry, I missed that the comment also says the instruction is a pld which would be correct if we are loading the symbols address from the plt, but this thunk type is for functions not in the plt. Do you intend to load the address of the function out of a table? or do you meant to calculate the address relative to the program counter?

[NeHuang]

lld/ELF/Thunks.cpp

852

Good point. Added fatal error checking if the offset fits in 34 bits.

856

Good catch! Updated it accordingly.

1012

For (s.stOther >> 5) < 2 cases (0, 1), they do not need r12 setup stub but could need a long branch thunk.

[NeHuang]

Thanks Sean and MaskRay for the review!

• Check if the offset computed in PPC64R12SetupStub::writeTo fits in 34 bits and report fatal error if it is too large.
• Use __gep_setup in the name for r12 set up stub and update test case accordingly.
• Address nits in the comments.

[NeHuang]

lld/ELF/Thunks.cpp

856

No worries. Right, should load the symbol address from the plt instead so that we can use pld to load the callee's global entry point to r12. Will update it accordingly.

[nemanjai]

Seems fine to me, but of course, wait to hear from Sean/MaskRay.

lld/ELF/Thunks.cpp

855

Would anyone object to naming these in an enum?
Perhaps something like:

enum PPCInstrMasks : uint64_t {
  PLD_NO_DISP = 0x04100000E5800000,
  MTCTR_R12 = 0x7D8903A6,
  ...
};

in llvm/include/llvm/Object/ELF.h?

864

I do not object to the name, but if there is precedent in the GNU linker, we should match it for consistency and familiarity.

[NeHuang]

• Added a fix to use paddi instead of pld to put the callee global entry point to r12
• Added enum for PPCInstrMasks in llvm/include/llvm/Object/ELF.h e.g. PADDI_R12_NO_DISP and MTCTR_R12
• Updated the offset computation for r12 set up stub and test case.

[NeHuang]

lld/ELF/Thunks.cpp

855

Thanks for the advice. enum added in ELF.h

864

GNU linker ld uses long_branch.callee as the name, which also used for r2 save stub as mentioned in https://reviews.llvm.org/D82950 . I would go with __gep_steup_ as it represents the thunk type.

[NeHuang]

Re-based with ToT master branch to merge in other protocol implementations.

[stefanp]

I only had a couple of comments.
Overall I think this looks good.

lld/ELF/Arch/PPC64.cpp

113

nit: Why was this moved up and the comment deleted?
I assume it was something that the rebase did...

lld/ELF/Thunks.cpp

873

I feel like there may be an easier way to do this without all of the shifts:

uint64_t paddi = PADDI_R12_NO_DISP |
                 (((offset >> 16) & 0x3ffff) << 32) |
                 (offset & 0xffff);
	​
writePrefixedInstruction(buf + 0, paddi); // paddi r12, 0, func@pcrel, 1

That way you won't have to break the instruction apart and then put it back together.

[NeHuang]

lld/ELF/Arch/PPC64.cpp

113

The comment was moved to (line 216) in lld/ELF/Target.h when I added the function declaration there.

lld/ELF/Thunks.cpp

873

Good point! will do.

[NeHuang]

Thanks Stefan.

• Modified paddi hex computation with less shifts.
• Fixed nits for clang-format.

[MaskRay]

lld/test/ELF/ppc64-pcrel-call-to-toc.s

3

Do you mind adopting https://reviews.llvm.org/D83834#change-qrppZgZj6Chf ?

[stefanp]

I only had one question.
Otherwise, LGTM.

However, please wait for approval from sfertile and/or MaskRay before committing.

lld/ELF/Arch/PPC64.cpp

113

Ok, see that the comment was moved.
Why was the function moved up?

[NeHuang]

lld/ELF/Arch/PPC64.cpp

113

I moved it up to group the functions declared in lld/ELF/Target.h (following declaration order) Also separated from the static functions below.

[NeHuang]

lld/test/ELF/ppc64-pcrel-call-to-toc.s

3

Thanks MaskRay for the input! The new style looks better. Howvever, seems your patch has not landed it yet. : )
I tried applying your changes locally to use extract in this case and confirmed it passed.
To avoid introducing dependency, I prefer keeping the old style in this patch and starting using the new style once your patch committed.

[MaskRay]

Thanks!

[sfertile]

LGTM.

lld/ELF/Arch/PPC64.cpp

unsigned elf::getPPC64GlobalEntryToLocalEntryOffset(uint8_t stOther) {
   return 0;
 }
 
 bool elf::isPPC64SmallCodeModelTocReloc(RelType type) {
   // The only small code model relocations that access the .toc section.
   return type == R_PPC64_TOC16 || type == R_PPC64_TOC16_DS;
 }
 
+void elf::writePrefixedInstruction(uint8_t *loc, uint64_t insn) {
+  insn = config->isLE ? insn << 32 | insn >> 32 : insn;
+  write64(loc, insn);
+}
+
 static bool addOptional(StringRef name, uint64_t value,
                         std::vector<Defined *> &defined) {
   Symbol *sym = symtab->find(name);
   if (!sym || sym->isDefined())
     return false;
   sym->resolve(Defined{/*file=*/nullptr, saver.save(name), STB_GLOBAL,
                        STV_HIDDEN, STT_FUNC, value,
                        /*size=*/0, /*section=*/nullptr});

 static void writeFromHalf16(uint8_t *loc, uint32_t insn) {
   write32(config->isLE ? loc : loc - 2, insn);
 }
 
 static uint32_t readFromHalf16(const uint8_t *loc) {
   return read32(config->isLE ? loc : loc - 2);
 }
 
-// The prefixed instruction is always a 4 byte prefix followed by a 4 byte
-// instruction. Therefore, the prefix is always in lower memory than the
-// instruction (regardless of endianness).
-// As a result, we need to shift the pieces around on little endian machines.
-static void writePrefixedInstruction(uint8_t *loc, uint64_t insn) {
-  insn = config->isLE ? insn << 32 | insn >> 32 : insn;
-  write64(loc, insn);
-}
-
 static uint64_t readPrefixedInstruction(const uint8_t *loc) {
   uint64_t fullInstr = read64(loc);
   return config->isLE ? (fullInstr << 32 | fullInstr >> 32) : fullInstr;
 }
 
 PPC64::PPC64() {
   copyRel = R_PPC64_COPY;
   gotRel = R_PPC64_GLOB_DAT;
bool PPC64::needsThunk(RelExpr expr, RelType type, const InputFile *file,
   if (type == R_PPC64_REL24_NOTOC && s.isInPlt())
     fatal("unimplemented feature: external function call with the reltype"
           " R_PPC64_REL24_NOTOC");
 
   // If a function is in the Plt it needs to be called with a call-stub.
   if (s.isInPlt())
     return true;
 
-  // FIXME: Remove the fatal error once the call protocol is implemented.
-  if (type == R_PPC64_REL24_NOTOC && (s.stOther >> 5) > 1)
-    fatal("unimplemented feature: local function call with the reltype"
-          " R_PPC64_REL24_NOTOC and the callee needs toc-pointer setup");
-
   // This check looks at the st_other bits of the callee with relocation
   // R_PPC64_REL14 or R_PPC64_REL24. If the value is 1, then the callee
   // clobbers the TOC and we need an R2 save stub.
   if (type != R_PPC64_REL24_NOTOC && (s.stOther >> 5) == 1)
     return true;
 
+  if (type == R_PPC64_REL24_NOTOC && (s.stOther >> 5) > 1)
+    return true;
+
   // If a symbol is a weak undefined and we are compiling an executable
   // it doesn't need a range-extending thunk since it can't be called.
   if (s.isUndefWeak() && !config->shared)
     return false;
 
   // If the offset exceeds the range of the branch type then it will need
   // a range-extending thunk.
   // See the comment in getRelocTargetVA() about R_PPC64_CALL.

lld/ELF/Target.h

 // This function will return the offset (in bytes) from the global entry-point
 // to the local entry-point.
 unsigned getPPC64GlobalEntryToLocalEntryOffset(uint8_t stOther);
 
 // Returns true if a relocation is a small code model relocation that accesses
 // the .toc section.
 bool isPPC64SmallCodeModelTocReloc(RelType type);
 
+// Write a prefixed instruction, which is a 4-byte prefix followed by a 4-byte
+// instruction (regardless of endianness). Therefore, the prefix is always in
+// lower memory than the instruction.
+void writePrefixedInstruction(uint8_t *loc, uint64_t insn);
+
 void addPPC64SaveRestore();
 uint64_t getPPC64TocBase();
 uint64_t getAArch64Page(uint64_t expr);
 
 extern const TargetInfo *target;
 TargetInfo *getTarget();
 
 template <class ELFT> bool isMipsPIC(const Defined *sym);

lld/ELF/Thunks.cpp

 public:
   PPC64PltCallStub(Symbol &dest) : Thunk(dest, 0) {}
   uint32_t size() override { return 20; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
 // PPC64 R2 Save Stub
 // When the caller requires a valid R2 TOC pointer but the callee does not
 // require a TOC pointer and the callee cannot guarantee that it doesn't
 // clobber R2 then we need to save R2. This stub:
 // 1) Saves the TOC pointer to the stack.
 // 2) Tail calls the callee.
 class PPC64R2SaveStub final : public Thunk {
 public:
   PPC64R2SaveStub(Symbol &dest) : Thunk(dest, 0) {}
   uint32_t size() override { return 8; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
+// PPC64 R12 Setup Stub
+// When a caller that does not maintain a toc-pointer performs a local call to
+// a callee which requires a toc-pointer then we need this stub to place the
+// callee's global entry point into r12 without a save of R2.
+class PPC64R12SetupStub final : public Thunk {
+public:
+  PPC64R12SetupStub(Symbol &dest) : Thunk(dest, 0) {}
+  uint32_t size() override { return 16; }
+  void writeTo(uint8_t *buf) override;
+  void addSymbols(ThunkSection &isec) override;
+};
+
 // A bl instruction uses a signed 24 bit offset, with an implicit 4 byte
 // alignment. This gives a possible 26 bits of 'reach'. If the call offset is
 // larger then that we need to emit a long-branch thunk. The target address
 // of the callee is stored in a table to be accessed TOC-relative. Since the
 // call must be local (a non-local call will have a PltCallStub instead) the
 // table stores the address of the callee's local entry point. For
 // position-independent code a corresponding relative dynamic relocation is
 // used.

 void PPC64PltCallStub::addSymbols(ThunkSection &isec) {
   Defined *s = addSymbol(saver.save("__plt_" + destination.getName()), STT_FUNC,
                          0, isec);
   s->needsTocRestore = true;
   s->file = destination.file;
 }
 
 void PPC64R2SaveStub::writeTo(uint8_t *buf) {
   int64_t offset = destination.getVA() - (getThunkTargetSym()->getVA() + 4);
   // The branch offset needs to fit in 26 bits.
   if (!isInt<26>(offset))
     fatal("R2 save stub branch offset is too large: " + Twine(offset));
   write32(buf + 0, 0xf8410018);                         // std  r2,24(r1)
   write32(buf + 4, 0x48000000 | (offset & 0x03fffffc)); // b    <offset>
 }
 
 void PPC64R2SaveStub::addSymbols(ThunkSection &isec) {
   Defined *s = addSymbol(saver.save("__toc_save_" + destination.getName()),
                          STT_FUNC, 0, isec);
   s->needsTocRestore = true;
 }
 
+void PPC64R12SetupStub::writeTo(uint8_t *buf) {
+  int64_t offset = destination.getVA() - getThunkTargetSym()->getVA();
+  if (!isInt<34>(offset))
+    fatal("offset must fit in 34 bits to encode in the instruction");
+  uint64_t prefix = (PADDI_R12_NO_DISP >> 32) | ((offset >> 16) & 0x3ffff);
+  uint32_t suffix = (PADDI_R12_NO_DISP & 0xffffffff) | (offset & 0xffff);
+
+  writePrefixedInstruction(buf + 0, (prefix << 32) | suffix); // paddi r12, 0, func@pcrel, 1
+  write32(buf + 8, MTCTR_R12);                                // mtctr r12
+  write32(buf + 12, BCTR);                                    // bctr
+}
+
+void PPC64R12SetupStub::addSymbols(ThunkSection &isec) {
+  addSymbol(saver.save("__gep_setup_" + destination.getName()),
+                       STT_FUNC, 0, isec);
+}
+
 void PPC64LongBranchThunk::writeTo(uint8_t *buf) {
   int64_t offset = in.ppc64LongBranchTarget->getEntryVA(&destination, addend) -
                    getPPC64TocBase();
   writePPC64LoadAndBranch(buf, offset);
 }
 
 void PPC64LongBranchThunk::addSymbols(ThunkSection &isec) {
   addSymbol(saver.save("__long_branch_" + destination.getName()), STT_FUNC, 0,
assert((rel.type == R_PPC_LOCAL24PC || rel.type == R_PPC_REL24 ||
           rel.type == R_PPC_PLTREL24) &&
          "unexpected relocation type for thunk");
   if (s.isInPlt())
     return make<PPC32PltCallStub>(isec, rel, s);
   return make<PPC32LongThunk>(s, rel.addend);
 }
 
 static Thunk *addThunkPPC64(RelType type, Symbol &s, int64_t a) {
-  assert((type == R_PPC64_REL14 || type == R_PPC64_REL24) &&
+  assert((type == R_PPC64_REL14 || type == R_PPC64_REL24 ||
+          type == R_PPC64_REL24_NOTOC) &&
          "unexpected relocation type for thunk");
   if (s.isInPlt())
     return make<PPC64PltCallStub>(s);
 
   // This check looks at the st_other bits of the callee. If the value is 1
   // then the callee clobbers the TOC and we need an R2 save stub.
   if ((s.stOther >> 5) == 1)
     return make<PPC64R2SaveStub>(s);
 
+  if (type == R_PPC64_REL24_NOTOC && (s.stOther >> 5) > 1)
+    return make<PPC64R12SetupStub>(s);
+
   if (config->picThunk)
     return make<PPC64PILongBranchThunk>(s, a);
 
   return make<PPC64PDLongBranchThunk>(s, a);
 }
 
 Thunk *elf::addThunk(const InputSection &isec, Relocation &rel) {
   Symbol &s = *rel.sym;

lld/test/ELF/ppc64-pcrel-call-to-toc.s

+# REQUIRES: ppc
+# RUN: echo 'SECTIONS { \
+# RUN:   .text_func   0x10010000 : { *(.text_func) } \
+# RUN:   .text_callee 0x10020000 : { *(.text_callee) } \
+# RUN:   .text_caller 0x10030000 : { *(.text_caller) } \
+# RUN:   }' > %t.script
+
+# RUN: llvm-mc -filetype=obj -triple=powerpc64le %s -o %t.o
+# RUN: ld.lld -T %t.script %t.o -o %t
+# RUN: llvm-readelf -s %t | FileCheck %s --check-prefix=SYMBOL
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %t | FileCheck %s
+
+# RUN: llvm-mc -filetype=obj -triple=powerpc64 %s -o %t.o
+# RUN: ld.lld -T %t.script %t.o -o %t
+# RUN: llvm-readelf -s %t | FileCheck %s --check-prefix=SYMBOL
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %t | FileCheck %s
+
+## When a function without TOC accesses a function using TOC, an r12 setup stub
+## is inserted
+
+# SYMBOL:      1: 0000000010020000 0 NOTYPE LOCAL DEFAULT [<other: 0x60>] 2 callee
+# SYMBOL-NEXT: 2: 0000000010030000 0 NOTYPE LOCAL DEFAULT [<other: 0x20>] 3 caller
+# SYMBOL-NEXT: 3: 0000000010010000 0 NOTYPE LOCAL DEFAULT 1 func
+# SYMBOL:      6: 000000001003000c 16 FUNC LOCAL DEFAULT 3 __gep_setup_callee
+
+# CHECK-LABEL: <func>:
+# CHECK-NEXT:  blr
+
+# CHECK-LABEL: <callee>:
+# CHECK:       bl 0x10010000
+# CHECK-NEXT:  addis 4, 2, -1
+# CHECK-NEXT:  lwz 4, 32744(4)
+# CHECK-NEXT:  blr
+
+# CHECK-LABEL: <caller>:
+# CHECK-NEXT:  bl 0x1003000c
+# CHECK-NEXT:  blr
+
+# CHECK-LABEL: <__gep_setup_callee>:
+# CHECK-NEXT:  paddi 12, 0, -65548, 1
+# CHECK-NEXT:  mtctr 12
+# CHECK-NEXT:  bctr
+
+.section .text_func, "ax", %progbits
+func:
+  blr
+
+.section .text_callee, "ax", %progbits
+callee:
+.Lfunc_gep1:
+  addis 2, 12, .TOC.-.Lfunc_gep1@ha
+  addi 2, 2, .TOC.-.Lfunc_gep1@l
+.Lfunc_lep1:
+  .localentry callee, .Lfunc_lep1-.Lfunc_gep1
+  bl func
+  addis 4, 2, global@toc@ha
+  lwz 4, global@toc@l(4)
+  blr
+
+.section .text_caller, "ax", %progbits
+caller:
+  .localentry caller, 1
+  bl callee@notoc
+  blr
+global:
+  .long 0
+  .size global, 4

llvm/include/llvm/Object/ELF.h

getElfArchType(StringRef Object) {
   return std::make_pair((uint8_t)Object[ELF::EI_CLASS],
                         (uint8_t)Object[ELF::EI_DATA]);
 }
 
 static inline Error createError(const Twine &Err) {
   return make_error<StringError>(Err, object_error::parse_failed);
 }
 
+enum PPCInstrMasks : uint64_t {
+  PADDI_R12_NO_DISP = 0x0610000039800000,
+  MTCTR_R12 = 0x7D8903A6,
+  BCTR = 0x4E800420,
+};
+
 template <class ELFT> class ELFFile;
 
 template <class ELFT>
 std::string getSecIndexForError(const ELFFile<ELFT> *Obj,
                                 const typename ELFT::Shdr *Sec) {
   auto TableOrErr = Obj->sections();
   if (TableOrErr)
     return "[index " + std::to_string(Sec - &TableOrErr->front()) + "]";