[ELF][AArch64] Support R_AARCH64_{CALL26,JUMP26} range extension thunks with addends

[MaskRay]

Summary

Fixes AArch64 part of PR40438

The current range extension thunk framework does not handle a relocation
relative to a STT_SECTION symbol with a non-zero addend, which may be
used by jumps/calls to local functions on some RELA targets (AArch64,
powerpc ELFv1, powerpc64 ELFv2, etc). See PR40438 and the following
code for examples:

// clang -target $target a.cc
// .text.cold may be placed in a separate output section.
// The distance between bar in .text.cold and foo in .text may be larger than 128MiB.
static void foo() {}
__attribute__((section(".text.cold"))) static int bar() { foo(); return
0; }
__attribute__((used)) static int dummy = bar();

This patch makes such thunks with addends work for AArch64. The target
independent part can be reused by PPC in the future.

On REL targets (ARM, MIPS), jumps/calls are not represented as
STT_SECTION + non-zero addend (see
MCELFObjectTargetWriter::needsRelocateWithSymbol), so they don't need
this feature, but we need to make sure this patch does not affect them.

[peter.smith]

Will look at the code in detail this afternoon.

On REL targets (ARM, MIPS), jumps/calls are not represented as STT_SECTION + non-zero addend (see MCELFObjectTargetWriter::needsRelocateWithSymbol), so they don't need this feature, but we need to make sure this patch does not affect them.

On Arm using a STT_FUNC symbol is required to get an interworking thunk (LLD isn't strictly compliant here, see appendix) so using a branch to STT_SECTION symbol + offset may not work reliably unless the caller knows the destination execution state and can do the interworking at the call site. I've seen MIPS use the bottom bit of the symbol STO_MIPS_MICROMIPS but I don't know the details.

It is possible to write in assembler

bl sym + 8

Where the +8 will be added to the PC-bias correction (-8 in Arm and -4 in Thumb) to give an immediate of 0 in Arm or +4 in Thumb. So this is at least possible to encounter in Thumb. As far as I know compilers won't generate this though. This could be supported on Arm at least by reading the implicit addend rather than assuming it is either -8 in Arm or -4 in Thumb as we do now.

On ARM, the compensation of implicit addends is done in ARM::needsThunk. This makes it tricky not to affect ARM. I feel that the code can be generalized (see config->isRela introduced in the patch) if we reorganize the code (ARM::needsThunk).

Can you elaborate? My understanding is that needsThunk() and inBranchRange() assume all Arm branches to have an immediate (which becomes the relocation addend) of -8 and Thumb branches to have -4. This cancels the PC-bias of 8-bytes in Arm and 4-bytes in Thumb.

This is my current thinking. If the addend from the branch immediate, whether read from the instruction, or assumed to be -8 for an Arm branch or -4 for a Thumb branch, is passed in to needsThunk() I think that we could do:

src = branchAddr +  (Is Thumb branch then + 4, else +8) 
dst = (expr == R_PLT_PC) ? s.getPltVA(addend) : s.getVA(addend);

The code sequence:

// PC at Src is 2 instructions ahead, immediate of branch is signed
if (src > dst)
  range -= 2 * instrSize;
else
  range += instrSize;

if ((dst & 0x1) == 0)
  // Destination is ARM, if ARM caller then Src is already 4-byte aligned.
  // If Thumb Caller (BLX) the Src address has bottom 2 bits cleared to ensure
  // destination will be 4 byte aligned.
  src &= ~0x3;
else
  // Bit 0 == 1 denotes Thumb state, it is not part of the range
  dst &= ~0x1;

I think should be unaffected. This is accounting for the implicit PC-bias and the alignment requirements when calculating what immediates are encodable.

Appendix/Aside:
Strictly speaking Arm/Thumb interworking is only done for STT_FUNC symbols. LLD does this for all symbols as we have to choose between a BL or BLX in relocateOne which doesn't have access to Symbol to check the type. There are also some corner case rules to do with how the thumb bit is processed in relocations that we don't follow. Strictly, for a STT_FUNC the value of the symbol should strip out the Thumb bit before adding the addend, the Thumb bit is then orred in. For example some relocations are defined as ((S + A) | T) - P. This can matter when the addend is 1 modulo 2, which in practice never happens outside of assembly when someone is trying to manually interwork. I would like to fix this but it will mean adding Symbol, or at least the result of S.isFunc() as a parameter to relocateOne(). If that doesn't sound too objectionable I'd be happy to work on it.

[peter.smith]

I'm comfortable with the detailed code-changes. I initially thought that using a nested pair might make the code too difficult to read, but it is clear at the construction of the pair which field is which. I'd definitely want to avoid referring the elements of the pair with first.first or first.second, but as we only use it as a key it doesn't matter.

[MaskRay]

Make code more general

Rebase on D70690

[MaskRay]

Can you elaborate? My understanding is that needsThunk() and inBranchRange() assume all Arm branches to have an immediate (which becomes the relocation addend) of -8 and Thumb branches to have -4. This cancels the PC-bias of 8-bytes in Arm and 4-bytes in Thumb.

This is my current thinking. If the addend from the branch immediate, whether read from the instruction, or assumed to be -8 for an Arm branch or -4 for a Thumb branch, is passed in to needsThunk() I think that we could do:

src = branchAddr + (Is Thumb branch then + 4, else +8)
dst = (expr == R_PLT_PC) ? s.getPltVA(addend) : s.getVA(addend);

Created D70690 as a prerequisite change to make this patch affect ARM even less.

The update in test/ELF/arm-thumb-mix-range-thunk-os.s is related to the following diff:

if (auto *d = dyn_cast<Defined>(rel.sym))
  if (!d->isInPlt() && d->section)
    thunkVec = &thunkedSymbolsBySectionAndAddend[{
        {d->section->repl, d->value}, rel.addend}];  // If I use 0 instead of rel.addend,
if (!thunkVec)
  thunkVec = &thunkedSymbols[{rel.sym, rel.addend}]; // If I use 0 instead of rel.addend,  test/ELF/arm-thumb-mix-range-thunk-os.s will not need an update.

The R_ARM_THM_CALL in tfunc33 causes creation of a Thumb thunk. The R_ARM_CALL in tfunc34 have different implicit addends, so making rel.addend part of the key will create an ARM thunk, rather than reuse the existing Thumb thunk. If reusing the thunk is the intended behavior, I'll change it to

// This ternary expression makes the code less general.
int64_t addend = config->emachine == EM_ARM ? 0 : rel.addend;

if (auto *d = dyn_cast<Defined>(rel.sym))
  if (!d->isInPlt() && d->section)
    thunkVec = &thunkedSymbolsBySectionAndAddend[{
        {d->section->repl, d->value}, addend}];
if (!thunkVec)
  thunkVec = &thunkedSymbols[{rel.sym, addend}];

can keep the test unchanged.

I would like to fix this but it will mean adding Symbol, or at least the result of S.isFunc() as a parameter to relocateOne(). If that doesn't sound too objectionable I'd be happy to work on it.

I support that we add Symbol to the parameter list of relocateOne. This can improve reportRangeError messages as well. Right now we don't report which symbol has the problem. With the Symbol information the message can be more specific.

[peter.smith]

In D70637#1759352, @MaskRay wrote:

Can you elaborate? My understanding is that needsThunk() and inBranchRange() assume all Arm branches to have an immediate (which becomes the relocation addend) of -8 and Thumb branches to have -4. This cancels the PC-bias of 8-bytes in Arm and 4-bytes in Thumb.

This is my current thinking. If the addend from the branch immediate, whether read from the instruction, or assumed to be -8 for an Arm branch or -4 for a Thumb branch, is passed in to needsThunk() I think that we could do:

src = branchAddr + (Is Thumb branch then + 4, else +8)
dst = (expr == R_PLT_PC) ? s.getPltVA(addend) : s.getVA(addend);

Created D70690 as a prerequisite change to make this patch affect ARM even less.

The update in test/ELF/arm-thumb-mix-range-thunk-os.s is related to the following diff:

if (auto *d = dyn_cast<Defined>(rel.sym))
  if (!d->isInPlt() && d->section)
    thunkVec = &thunkedSymbolsBySectionAndAddend[{
        {d->section->repl, d->value}, rel.addend}];  // If I use 0 instead of rel.addend,
if (!thunkVec)
  thunkVec = &thunkedSymbols[{rel.sym, rel.addend}]; // If I use 0 instead of rel.addend,  test/ELF/arm-thumb-mix-range-thunk-os.s will not need an update.

The R_ARM_THM_CALL in tfunc33 causes creation of a Thumb thunk. The R_ARM_CALL in tfunc34 have different implicit addends, so making rel.addend part of the key will create an ARM thunk, rather than reuse the existing Thumb thunk. If reusing the thunk is the intended behavior, I'll change it to

// This ternary expression makes the code less general.
int64_t addend = config->emachine == EM_ARM ? 0 : rel.addend;

if (auto *d = dyn_cast<Defined>(rel.sym))
  if (!d->isInPlt() && d->section)
    thunkVec = &thunkedSymbolsBySectionAndAddend[{
        {d->section->repl, d->value}, addend}];
if (!thunkVec)
  thunkVec = &thunkedSymbols[{rel.sym, addend}];

can keep the test unchanged.

Yes it is intentional, an Arm or Thumb BL instruction can be changed into a BLX instruction that changes state, so it is possible for both an Arm and Thumb relocation to share the same Thunk.

Instead of

// This ternary expression makes the code less general.
int64_t addend = config->emachine == EM_ARM ? 0 : rel.addend;

May I suggest

int64_t addend = rel.addend + getPCBias(type, rel.addend);

This could be virtual function for Target, but could just be static function that in effect cancels -4 or -8 to 0 on Arm, but still accounts for non -4 or -8 addends. The body would be something like:

  if (Config->EMachine != EM_ARM)
    return 0;
  switch (type) {
  case R_ARM_THM_JUMP19:
  case R_ARM_THM_JUMP24:
  case R_ARM_THM_CALL:
    return 4;
  default:
    return 8;
  }
}

This is more code, but it is a bit more general.

I would like to fix this but it will mean adding Symbol, or at least the result of S.isFunc() as a parameter to relocateOne(). If that doesn't sound too objectionable I'd be happy to work on it.

I support that we add Symbol to the parameter list of relocateOne. This can improve reportRangeError messages as well. Right now we don't report which symbol has the problem. With the Symbol information the message can be more specific.

[MaskRay]

Rebase after D70690 update (adopt peter.smith's suggestion that using getPCBias instead of inBranchRangeBeforePCBias)

[peter.smith]

I'm happy with this change. It would be worth waiting a bit to see if there are any comments from other reviewers.

lld/ELF/Relocations.cpp

1822

Typo: addened -> addend

[MaskRay]

Fix a typo in the comment

lld/ELF/Arch/AArch64.cpp

public:
   RelExpr getRelExpr(RelType type, const Symbol &s,
                      const uint8_t *loc) const override;
   RelType getDynRel(RelType type) const override;
   void writeGotPlt(uint8_t *buf, const Symbol &s) const override;
   void writePltHeader(uint8_t *buf) const override;
   void writePlt(uint8_t *buf, uint64_t gotPltEntryAddr, uint64_t pltEntryAddr,
                 int32_t index, unsigned relOff) const override;
   bool needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                  uint64_t branchAddr, const Symbol &s) const override;
+                  uint64_t branchAddr, const Symbol &s,
+                  int64_t a) const override;
   uint32_t getThunkSectionSpacing() const override;
   bool inBranchRange(RelType type, uint64_t src, uint64_t dst) const override;
   bool usesOnlyLowPageBits(RelType type) const override;
   void relocateOne(uint8_t *loc, RelType type, uint64_t val) const override;
   RelExpr adjustRelaxExpr(RelType type, const uint8_t *data,
                           RelExpr expr) const override;
   void relaxTlsGdToLe(uint8_t *loc, RelType type, uint64_t val) const override;
   void relaxTlsGdToIe(uint8_t *loc, RelType type, uint64_t val) const override;
void AArch64::writePlt(uint8_t *buf, uint64_t gotPltEntryAddr,
 
   relocateOne(buf, R_AARCH64_ADR_PREL_PG_HI21,
               getAArch64Page(gotPltEntryAddr) - getAArch64Page(pltEntryAddr));
   relocateOne(buf + 4, R_AARCH64_LDST64_ABS_LO12_NC, gotPltEntryAddr);
   relocateOne(buf + 8, R_AARCH64_ADD_ABS_LO12_NC, gotPltEntryAddr);
 }
 
 bool AArch64::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                         uint64_t branchAddr, const Symbol &s) const {
+                         uint64_t branchAddr, const Symbol &s,
+                         int64_t a) const {
   // ELF for the ARM 64-bit architecture, section Call and Jump relocations
   // only permits range extension thunks for R_AARCH64_CALL26 and
   // R_AARCH64_JUMP26 relocation types.
   if (type != R_AARCH64_CALL26 && type != R_AARCH64_JUMP26)
     return false;
-  uint64_t dst = (expr == R_PLT_PC) ? s.getPltVA() : s.getVA();
+  uint64_t dst = expr == R_PLT_PC ? s.getPltVA() : s.getVA(a);
   return !inBranchRange(type, branchAddr, dst);
 }
 
 uint32_t AArch64::getThunkSectionSpacing() const {
   // See comment in Arch/ARM.cpp for a more detailed explanation of
   // getThunkSectionSpacing(). For AArch64 the only branches we are permitted to
   // Thunk have a range of +/- 128 MiB
   return (128 * 1024 * 1024) - 0x30000;

lld/ELF/Arch/ARM.cpp

public:
   void writeGotPlt(uint8_t *buf, const Symbol &s) const override;
   void writeIgotPlt(uint8_t *buf, const Symbol &s) const override;
   void writePltHeader(uint8_t *buf) const override;
   void writePlt(uint8_t *buf, uint64_t gotPltEntryAddr, uint64_t pltEntryAddr,
                 int32_t index, unsigned relOff) const override;
   void addPltSymbols(InputSection &isec, uint64_t off) const override;
   void addPltHeaderSymbols(InputSection &isd) const override;
   bool needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                  uint64_t branchAddr, const Symbol &s) const override;
+                  uint64_t branchAddr, const Symbol &s,
+                  int64_t a) const override;
   uint32_t getThunkSectionSpacing() const override;
   bool inBranchRange(RelType type, uint64_t src, uint64_t dst) const override;
   void relocateOne(uint8_t *loc, RelType type, uint64_t val) const override;
 };
 } // namespace
 
 ARM::ARM() {
   copyRel = R_ARM_COPY;

 }
 
 void ARM::addPltSymbols(InputSection &isec, uint64_t off) const {
   addSyntheticLocal("$a", STT_NOTYPE, off, 0, isec);
   addSyntheticLocal("$d", STT_NOTYPE, off + 12, 0, isec);
 }
 
 bool ARM::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                     uint64_t branchAddr, const Symbol &s) const {
+                     uint64_t branchAddr, const Symbol &s, int64_t /*a*/) const {
   // If S is an undefined weak symbol and does not have a PLT entry then it
   // will be resolved as a branch to the next instruction.
   if (s.isUndefWeak() && !s.isInPlt())
     return false;
   // A state change from ARM to Thumb and vice versa must go through an
   // interworking thunk if the relocation type is not R_ARM_CALL or
   // R_ARM_THM_CALL.
   switch (type) {

lld/ELF/Arch/Mips.cpp

RelExpr getRelExpr(RelType type, const Symbol &s,
                      const uint8_t *loc) const override;
   int64_t getImplicitAddend(const uint8_t *buf, RelType type) const override;
   RelType getDynRel(RelType type) const override;
   void writeGotPlt(uint8_t *buf, const Symbol &s) const override;
   void writePltHeader(uint8_t *buf) const override;
   void writePlt(uint8_t *buf, uint64_t gotPltEntryAddr, uint64_t pltEntryAddr,
                 int32_t index, unsigned relOff) const override;
   bool needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                  uint64_t branchAddr, const Symbol &s) const override;
+                  uint64_t branchAddr, const Symbol &s,
+                  int64_t a) const override;
   void relocateOne(uint8_t *loc, RelType type, uint64_t val) const override;
   bool usesOnlyLowPageBits(RelType type) const override;
 };
 } // namespace
 
 template <class ELFT> MIPS<ELFT>::MIPS() {
   gotPltHeaderEntriesNum = 2;
   defaultMaxPageSize = 65536;
void MIPS<ELFT>::writePlt(uint8_t *buf, uint64_t gotPltEntryAddr,
   write32(buf + 12, addInst);   // [d]addiu $24, $15, %lo(.got.plt entry)
   writeValue(buf, gotPltEntryAddr + 0x8000, 16, 16);
   writeValue(buf + 4, gotPltEntryAddr, 16, 0);
   writeValue(buf + 12, gotPltEntryAddr, 16, 0);
 }
 
 template <class ELFT>
 bool MIPS<ELFT>::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                            uint64_t branchAddr, const Symbol &s) const {
+                            uint64_t branchAddr, const Symbol &s,
+                            int64_t /*a*/) const {
   // Any MIPS PIC code function is invoked with its address in register $t9.
   // So if we have a branch instruction from non-PIC code to the PIC one
   // we cannot make the jump directly and need to create a small stubs
   // to save the target function address.
   // See page 3-38 ftp://www.linux-mips.org/pub/linux/mips/doc/ABI/mipsabi.pdf
   if (type != R_MIPS_26 && type != R_MIPS_PC26_S2 &&
       type != R_MICROMIPS_26_S1 && type != R_MICROMIPS_PC26_S1)
     return false;

lld/ELF/Arch/PPC.cpp

void writePltHeader(uint8_t *buf) const override {
     llvm_unreachable("should call writePPC32GlinkSection() instead");
   }
   void writePlt(uint8_t *buf, uint64_t gotPltEntryAddr, uint64_t pltEntryAddr,
     int32_t index, unsigned relOff) const override {
     llvm_unreachable("should call writePPC32GlinkSection() instead");
   }
   void writeGotPlt(uint8_t *buf, const Symbol &s) const override;
   bool needsThunk(RelExpr expr, RelType relocType, const InputFile *file,
-                  uint64_t branchAddr, const Symbol &s) const override;
+                  uint64_t branchAddr, const Symbol &s,
+                  int64_t a) const override;
   uint32_t getThunkSectionSpacing() const override;
   bool inBranchRange(RelType type, uint64_t src, uint64_t dst) const override;
   void relocateOne(uint8_t *loc, RelType type, uint64_t val) const override;
   RelExpr adjustRelaxExpr(RelType type, const uint8_t *data,
                           RelExpr expr) const override;
   int getTlsGdRelaxSkip(RelType type) const override;
   void relaxTlsGdToIe(uint8_t *loc, RelType type, uint64_t val) const override;
   void relaxTlsGdToLe(uint8_t *loc, RelType type, uint64_t val) const override;

 }
 
 void PPC::writeGotPlt(uint8_t *buf, const Symbol &s) const {
   // Address of the symbol resolver stub in .glink .
   write32(buf, in.plt->getVA() + 4 * s.pltIndex);
 }
 
 bool PPC::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                     uint64_t branchAddr, const Symbol &s) const {
+                     uint64_t branchAddr, const Symbol &s, int64_t /*a*/) const {
   if (type != R_PPC_REL24 && type != R_PPC_PLTREL24)
     return false;
   if (s.isInPlt())
     return true;
   if (s.isUndefWeak())
     return false;
   return !(expr == R_PC && PPC::inBranchRange(type, branchAddr, s.getVA()));
 }

lld/ELF/Arch/PPC64.cpp

RelExpr getRelExpr(RelType type, const Symbol &s,
                      const uint8_t *loc) const override;
   RelType getDynRel(RelType type) const override;
   void writePltHeader(uint8_t *buf) const override;
   void writePlt(uint8_t *buf, uint64_t gotPltEntryAddr, uint64_t pltEntryAddr,
                 int32_t index, unsigned relOff) const override;
   void relocateOne(uint8_t *loc, RelType type, uint64_t val) const override;
   void writeGotHeader(uint8_t *buf) const override;
   bool needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                  uint64_t branchAddr, const Symbol &s) const override;
+                  uint64_t branchAddr, const Symbol &s,
+                  int64_t a) const override;
   uint32_t getThunkSectionSpacing() const override;
   bool inBranchRange(RelType type, uint64_t src, uint64_t dst) const override;
   RelExpr adjustRelaxExpr(RelType type, const uint8_t *data,
                           RelExpr expr) const override;
   void relaxGot(uint8_t *loc, RelType type, uint64_t val) const override;
   void relaxTlsGdToIe(uint8_t *loc, RelType type, uint64_t val) const override;
   void relaxTlsGdToLe(uint8_t *loc, RelType type, uint64_t val) const override;
   void relaxTlsLdToLe(uint8_t *loc, RelType type, uint64_t val) const override;
case R_PPC64_DTPREL64:
     write64(loc, val - dynamicThreadPointerOffset);
     break;
   default:
     llvm_unreachable("unknown relocation");
   }
 }
 
 bool PPC64::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                       uint64_t branchAddr, const Symbol &s) const {
+                       uint64_t branchAddr, const Symbol &s, int64_t /*a*/) const {
   if (type != R_PPC64_REL14 && type != R_PPC64_REL24)
     return false;
 
   // If a function is in the Plt it needs to be called with a call-stub.
   if (s.isInPlt())
     return true;
 
   // If a symbol is a weak undefined and we are compiling an executable

lld/ELF/Relocations.h

private:
   std::pair<Thunk *, bool> getThunk(InputSection *isec, Relocation &rel,
                                     uint64_t src);
 
   ThunkSection *addThunkSection(OutputSection *os, InputSectionDescription *,
                                 uint64_t off);
 
   bool normalizeExistingThunk(Relocation &rel, uint64_t src);
 
-  // Record all the available Thunks for a Symbol
-  llvm::DenseMap<std::pair<SectionBase *, uint64_t>, std::vector<Thunk *>>
-      thunkedSymbolsBySection;
-  llvm::DenseMap<Symbol *, std::vector<Thunk *>> thunkedSymbols;
+  // Record all the available Thunks for a (Symbol, addend) pair, where Symbol
+  // is represented as a (section, offset) pair. There may be multiple
+  // relocations sharing the same (section, offset + addend) pair. We may revert
+  // a relocation back to its original non-Thunk target, and restore the
+  // original addend, so we cannot fold offset + addend. A nested pair is used
+  // because DenseMapInfo is not specialized for std::tuple.
+  llvm::DenseMap<std::pair<std::pair<SectionBase *, uint64_t>, int64_t>,
+                 std::vector<Thunk *>>
+      thunkedSymbolsBySectionAndAddend;
+  llvm::DenseMap<std::pair<Symbol *, int64_t>, std::vector<Thunk *>>
+      thunkedSymbols;
 
   // Find a Thunk from the Thunks symbol definition, we can use this to find
   // the Thunk from a relocation to the Thunks symbol definition.
   llvm::DenseMap<Symbol *, Thunk *> thunks;
 
   // Track InputSections that have an inline ThunkSection placed in front
   // an inline ThunkSection may have control fall through to the section below
   // so we need to make sure that there is only one of them.

lld/ELF/Relocations.cpp

static int64_t getPCBias(RelType type) {
   default:
     return 8;
   }
 }
 
 std::pair<Thunk *, bool> ThunkCreator::getThunk(InputSection *isec,
                                                 Relocation &rel, uint64_t src) {
   std::vector<Thunk *> *thunkVec = nullptr;
+  int64_t addend = rel.addend + getPCBias(rel.type);
 
-  // We use (section, offset) pair to find the thunk position if possible so
-  // that we create only one thunk for aliased symbols or ICFed sections.
+  // We use a ((section, offset), addend) pair to find the thunk position if
+  // possible so that we create only one thunk for aliased symbols or ICFed
+  // sections. There may be multiple relocations sharing the same (section,
+  // offset + addend) pair. We may revert the relocation back to its original
+  // non-Thunk target, so we cannot fold offset + addend.
   if (auto *d = dyn_cast<Defined>(rel.sym))
     if (!d->isInPlt() && d->section)
-      thunkVec = &thunkedSymbolsBySection[{d->section->repl, d->value}];
+      thunkVec = &thunkedSymbolsBySectionAndAddend[{
+          {d->section->repl, d->value}, addend}];
   if (!thunkVec)
-    thunkVec = &thunkedSymbols[rel.sym];
+    thunkVec = &thunkedSymbols[{rel.sym, addend}];
 
   // Check existing Thunks for Sym to see if they can be reused
   for (Thunk *t : *thunkVec)
     if (isThunkSectionCompatible(isec, t->getThunkTargetSym()->section) &&
         t->isCompatibleWith(*isec, rel) &&
         target->inBranchRange(rel.type, src,
                               t->getThunkTargetSym()->getVA(rel.addend) +
                                   getPCBias(rel.type)))

 // was originally to a Thunk, but is no longer in range we revert the
 // relocation back to its original non-Thunk target.
 bool ThunkCreator::normalizeExistingThunk(Relocation &rel, uint64_t src) {
   if (Thunk *t = thunks.lookup(rel.sym)) {
     if (target->inBranchRange(rel.type, src,
                               rel.sym->getVA(rel.addend) + getPCBias(rel.type)))
       return true;
     rel.sym = &t->destination;
+    // TODO Restore addend on all targets.
+    if (config->emachine == EM_AARCH64)
+      rel.addend = t->addend;
     if (rel.sym->isInPlt())
       rel.expr = toPlt(rel.expr);
   }
   return false;
 }
 
 // Process all relocations from the InputSections that have been assigned
 // to InputSectionDescriptions and redirect through Thunks if needed. The
forEachInputSectionDescription(
 
             // If we are a relocation to an existing Thunk, check if it is
             // still in range. If not then Rel will be altered to point to its
             // original target so another Thunk can be generated.
             if (pass > 0 && normalizeExistingThunk(rel, src))
               continue;
 
             if (!target->needsThunk(rel.expr, rel.type, isec->file, src,
-                                    *rel.sym))
+                                    *rel.sym, rel.addend))
               continue;
 
             Thunk *t;
             bool isNew;
             std::tie(t, isNew) = getThunk(isec, rel, src);
 
             if (isNew) {
               // Find or create a ThunkSection for the new Thunk
               ThunkSection *ts;
               if (auto *tis = t->getTargetInputSection())
                 ts = getISThunkSec(tis);
               else
                 ts = getISDThunkSec(os, isec, isd, rel.type, src);
               ts->addThunk(t);
               thunks[t->getThunkTargetSym()] = t;
             }
 
             // Redirect relocation to Thunk, we never go via the PLT to a Thunk
             rel.sym = t->getThunkTargetSym();
             rel.expr = fromPlt(rel.expr);
 
+            // On AArch64, a jump/call relocation may be encoded as STT_SECTION
+            // + non-zero addend, clear the addend after redirection.
+            //
             // The addend of R_PPC_PLTREL24 should be ignored after changing to
             // R_PC.
-            if (config->emachine == EM_PPC && rel.type == R_PPC_PLTREL24)
+            if (config->emachine == EM_AARCH64 ||
+                (config->emachine == EM_PPC && rel.type == R_PPC_PLTREL24))
               rel.addend = 0;
           }
 
         for (auto &p : isd->thunkSections)
           addressesChanged |= p.first->assignOffsets();
       });
 
   for (auto &p : thunkedSections)

lld/ELF/Target.h

public:
   // bits will always have the same value at runtime and we don't have to emit
   // a dynamic relocation.
   virtual bool usesOnlyLowPageBits(RelType type) const;
 
   // Decide whether a Thunk is needed for the relocation from File
   // targeting S.
   virtual bool needsThunk(RelExpr expr, RelType relocType,
                           const InputFile *file, uint64_t branchAddr,
-                          const Symbol &s) const;
+                          const Symbol &s, int64_t a) const;
 
   // On systems with range extensions we place collections of Thunks at
   // regular spacings that enable the majority of branches reach the Thunks.
   // a value of 0 means range extension thunks are not supported.
   virtual uint32_t getThunkSectionSpacing() const { return 0; }
 
   // The function with a prologue starting at Loc was compiled with
   // -fsplit-stack and it calls a function compiled without. Adjust the prologue

lld/ELF/Target.cpp

 
 int64_t TargetInfo::getImplicitAddend(const uint8_t *buf, RelType type) const {
   return 0;
 }
 
 bool TargetInfo::usesOnlyLowPageBits(RelType type) const { return false; }
 
 bool TargetInfo::needsThunk(RelExpr expr, RelType type, const InputFile *file,
-                            uint64_t branchAddr, const Symbol &s) const {
+                            uint64_t branchAddr, const Symbol &s,
+                            int64_t a) const {
   return false;
 }
 
 bool TargetInfo::adjustPrologueForCrossSplitStack(uint8_t *loc, uint8_t *end,
                                                   uint8_t stOther) const {
   llvm_unreachable("Target doesn't support split stacks.");
 }
 

lld/ELF/Thunks.h

 // after executing transfers control to the callee. Typical uses of Thunks
 // include transferring control from non-pi to pi and changing state on
 // targets like ARM.
 //
 // Thunks can be created for Defined, Shared and Undefined Symbols.
 // Thunks are assigned to synthetic ThunkSections
 class Thunk {
 public:
-  Thunk(Symbol &destination);
+  Thunk(Symbol &destination, int64_t addend);
   virtual ~Thunk();
 
   virtual uint32_t size() = 0;
   virtual void writeTo(uint8_t *buf) = 0;
 
   // All Thunks must define at least one symbol, known as the thunk target
   // symbol, so that we can redirect relocations to it. The thunk may define
   // additional symbols, but these are never targets for relocations.
public:
   // with it.
   virtual bool isCompatibleWith(const InputSection &,
                                 const Relocation &) const {
     return true;
   }
 
   Defined *getThunkTargetSym() const { return syms[0]; }
 
-  // The alignment requirement for this Thunk, defaults to the size of the
-  // typical code section alignment.
   Symbol &destination;
+  int64_t addend;
   llvm::SmallVector<Defined *, 3> syms;
   uint64_t offset = 0;
+  // The alignment requirement for this Thunk, defaults to the size of the
+  // typical code section alignment.
   uint32_t alignment = 4;
 };
 
 // For a Relocation to symbol S create a Thunk to be added to a synthetic
 // ThunkSection.
 Thunk *addThunk(const InputSection &isec, Relocation &rel);
 
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/Thunks.cpp

 namespace lld {
 namespace elf {
 
 namespace {
 
 // AArch64 long range Thunks
 class AArch64ABSLongThunk final : public Thunk {
 public:
-  AArch64ABSLongThunk(Symbol &dest) : Thunk(dest) {}
+  AArch64ABSLongThunk(Symbol &dest, int64_t addend) : Thunk(dest, addend) {}
   uint32_t size() override { return 16; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
 class AArch64ADRPThunk final : public Thunk {
 public:
-  AArch64ADRPThunk(Symbol &dest) : Thunk(dest) {}
+  AArch64ADRPThunk(Symbol &dest, int64_t addend) : Thunk(dest, addend) {}
   uint32_t size() override { return 12; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
 // Base class for ARM thunks.
 //
 // An ARM thunk may be either short or long. A short thunk is simply a branch
 // (B) instruction, and it may be used to call ARM functions when the distance
 // from the thunk to the target is less than 32MB. Long thunks can branch to any
 // virtual address and can switch between ARM and Thumb, and they are
 // implemented in the derived classes. This class tries to create a short thunk
 // if the target is in range, otherwise it creates a long thunk.
 class ARMThunk : public Thunk {
 public:
-  ARMThunk(Symbol &dest) : Thunk(dest) {}
+  ARMThunk(Symbol &dest) : Thunk(dest, 0) {}
 
   bool getMayUseShortThunk();
   uint32_t size() override { return getMayUseShortThunk() ? 4 : sizeLong(); }
   void writeTo(uint8_t *buf) override;
   bool isCompatibleWith(const InputSection &isec,
                         const Relocation &rel) const override;
 
   // Returns the size of a long thunk.

 };
 
 // Base class for Thumb-2 thunks.
 //
 // This class is similar to ARMThunk, but it uses the Thumb-2 B.W instruction
 // which has a range of 16MB.
 class ThumbThunk : public Thunk {
 public:
-  ThumbThunk(Symbol &dest) : Thunk(dest) { alignment = 2; }
+  ThumbThunk(Symbol &dest) : Thunk(dest, 0) { alignment = 2; }
 
   bool getMayUseShortThunk();
   uint32_t size() override { return getMayUseShortThunk() ? 4 : sizeLong(); }
   void writeTo(uint8_t *buf) override;
   bool isCompatibleWith(const InputSection &isec,
                         const Relocation &rel) const override;
 
   // Returns the size of a long thunk.
public:
   uint32_t sizeLong() override { return 16; }
   void writeLong(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
 // MIPS LA25 thunk
 class MipsThunk final : public Thunk {
 public:
-  MipsThunk(Symbol &dest) : Thunk(dest) {}
+  MipsThunk(Symbol &dest) : Thunk(dest, 0) {}
 
   uint32_t size() override { return 16; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
   InputSection *getTargetInputSection() const override;
 };
 
 // microMIPS R2-R5 LA25 thunk
 class MicroMipsThunk final : public Thunk {
 public:
-  MicroMipsThunk(Symbol &dest) : Thunk(dest) {}
+  MicroMipsThunk(Symbol &dest) : Thunk(dest, 0) {}
 
   uint32_t size() override { return 14; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
   InputSection *getTargetInputSection() const override;
 };
 
 // microMIPS R6 LA25 thunk
 class MicroMipsR6Thunk final : public Thunk {
 public:
-  MicroMipsR6Thunk(Symbol &dest) : Thunk(dest) {}
+  MicroMipsR6Thunk(Symbol &dest) : Thunk(dest, 0) {}
 
   uint32_t size() override { return 12; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
   InputSection *getTargetInputSection() const override;
 };
 
 class PPC32PltCallStub final : public Thunk {
 public:
-  PPC32PltCallStub(const InputSection &isec, const Relocation &rel, Symbol &dest)
-      : Thunk(dest), addend(rel.type == R_PPC_PLTREL24 ? rel.addend : 0),
+  // For R_PPC_PLTREL24, Thunk::addend records the addend which will be used to
+  // decide the offsets in the call stub.
+  PPC32PltCallStub(const InputSection &isec, const Relocation &rel,
+                   Symbol &dest)
+      : Thunk(dest, rel.type == R_PPC_PLTREL24 ? rel.addend : 0),
         file(isec.file) {}
   uint32_t size() override { return 16; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
   bool isCompatibleWith(const InputSection &isec, const Relocation &rel) const override;
 
 private:
-  // For R_PPC_PLTREL24, this records the addend, which will be used to decide
-  // the offsets in the call stub.
-  uint32_t addend;
-
   // Records the call site of the call stub.
   const InputFile *file;
 };
 
 // PPC64 Plt call stubs.
 // Any call site that needs to call through a plt entry needs a call stub in
 // the .text section. The call stub is responsible for:
 // 1) Saving the toc-pointer to the stack.
 // 2) Loading the target functions address from the procedure linkage table into
 //    r12 for use by the target functions global entry point, and into the count
 //    register.
 // 3) Transferring control to the target function through an indirect branch.
 class PPC64PltCallStub final : public Thunk {
 public:
-  PPC64PltCallStub(Symbol &dest) : Thunk(dest) {}
+  PPC64PltCallStub(Symbol &dest) : Thunk(dest, 0) {}
   uint32_t size() override { return 20; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 };
 
 // A bl instruction uses a signed 24 bit offset, with an implicit 4 byte
 // alignment. This gives a possible 26 bits of 'reach'. If the call offset is
 // larger then that we need to emit a long-branch thunk. The target address
 // of the callee is stored in a table to be accessed TOC-relative. Since the
 // call must be local (a non-local call will have a PltCallStub instead) the
 // table stores the address of the callee's local entry point. For
 // position-independent code a corresponding relative dynamic relocation is
 // used.
 class PPC64LongBranchThunk : public Thunk {
 public:
   uint32_t size() override { return 16; }
   void writeTo(uint8_t *buf) override;
   void addSymbols(ThunkSection &isec) override;
 
 protected:
-  PPC64LongBranchThunk(Symbol &dest) : Thunk(dest) {}
+  PPC64LongBranchThunk(Symbol &dest) : Thunk(dest, 0) {}
 };
 
 class PPC64PILongBranchThunk final : public PPC64LongBranchThunk {
 public:
   PPC64PILongBranchThunk(Symbol &dest) : PPC64LongBranchThunk(dest) {
     assert(!dest.isPreemptible);
     if (dest.isInPPC64Branchlt())
       return;

 void Thunk::setOffset(uint64_t newOffset) {
   for (Defined *d : syms)
     d->value = d->value - offset + newOffset;
   offset = newOffset;
 }
 
 // AArch64 long range Thunks
 
-static uint64_t getAArch64ThunkDestVA(const Symbol &s) {
-  uint64_t v = s.isInPlt() ? s.getPltVA() : s.getVA();
+static uint64_t getAArch64ThunkDestVA(const Symbol &s, int64_t a) {
+  uint64_t v = s.isInPlt() ? s.getPltVA() : s.getVA(a);
   return v;
 }
 
 void AArch64ABSLongThunk::writeTo(uint8_t *buf) {
   const uint8_t data[] = {
     0x50, 0x00, 0x00, 0x58, //     ldr x16, L0
     0x00, 0x02, 0x1f, 0xd6, //     br  x16
     0x00, 0x00, 0x00, 0x00, // L0: .xword S
     0x00, 0x00, 0x00, 0x00,
   };
-  uint64_t s = getAArch64ThunkDestVA(destination);
+  uint64_t s = getAArch64ThunkDestVA(destination, addend);
   memcpy(buf, data, sizeof(data));
   target->relocateOne(buf + 8, R_AARCH64_ABS64, s);
 }
 
 void AArch64ABSLongThunk::addSymbols(ThunkSection &isec) {
   addSymbol(saver.save("__AArch64AbsLongThunk_" + destination.getName()),
             STT_FUNC, 0, isec);
   addSymbol("$x", STT_NOTYPE, 0, isec);
   addSymbol("$d", STT_NOTYPE, 8, isec);
 }
 
 // This Thunk has a maximum range of 4Gb, this is sufficient for all programs
 // using the small code model, including pc-relative ones. At time of writing
 // clang and gcc do not support the large code model for position independent
 // code so it is safe to use this for position independent thunks without
 // worrying about the destination being more than 4Gb away.
 void AArch64ADRPThunk::writeTo(uint8_t *buf) {
   const uint8_t data[] = {
       0x10, 0x00, 0x00, 0x90, // adrp x16, Dest R_AARCH64_ADR_PREL_PG_HI21(Dest)
       0x10, 0x02, 0x00, 0x91, // add  x16, x16, R_AARCH64_ADD_ABS_LO12_NC(Dest)
       0x00, 0x02, 0x1f, 0xd6, // br   x16
   };
-  uint64_t s = getAArch64ThunkDestVA(destination);
+  uint64_t s = getAArch64ThunkDestVA(destination, addend);
   uint64_t p = getThunkTargetSym()->getVA();
   memcpy(buf, data, sizeof(data));
   target->relocateOne(buf, R_AARCH64_ADR_PREL_PG_HI21,
                       getAArch64Page(s) - getAArch64Page(p));
   target->relocateOne(buf + 4, R_AARCH64_ADD_ABS_LO12_NC, s);
 }
 
 void AArch64ADRPThunk::addSymbols(ThunkSection &isec) {
void PPC64LongBranchThunk::writeTo(uint8_t *buf) {
   writePPCLoadAndBranch(buf, offset);
 }
 
 void PPC64LongBranchThunk::addSymbols(ThunkSection &isec) {
   addSymbol(saver.save("__long_branch_" + destination.getName()), STT_FUNC, 0,
             isec);
 }
 
-Thunk::Thunk(Symbol &d) : destination(d), offset(0) {}
+Thunk::Thunk(Symbol &d, int64_t a) : destination(d), addend(a), offset(0) {}
 
 Thunk::~Thunk() = default;
 
-static Thunk *addThunkAArch64(RelType type, Symbol &s) {
+static Thunk *addThunkAArch64(RelType type, Symbol &s, int64_t a) {
   if (type != R_AARCH64_CALL26 && type != R_AARCH64_JUMP26)
     fatal("unrecognized relocation type");
   if (config->picThunk)
-    return make<AArch64ADRPThunk>(s);
-  return make<AArch64ABSLongThunk>(s);
+    return make<AArch64ADRPThunk>(s, a);
+  return make<AArch64ABSLongThunk>(s, a);
 }
 
 // Creates a thunk for Thumb-ARM interworking.
 // Arm Architectures v5 and v6 do not support Thumb2 technology. This means
 // - MOVT and MOVW instructions cannot be used
 // - Only Thumb relocation that can generate a Thunk is a BL, this can always
 //   be transformed into a BLX
 static Thunk *addThunkPreArmv7(RelType reloc, Symbol &s) {

 static Thunk *addThunkMips(RelType type, Symbol &s) {
   if ((s.stOther & STO_MIPS_MICROMIPS) && isMipsR6())
     return make<MicroMipsR6Thunk>(s);
   if (s.stOther & STO_MIPS_MICROMIPS)
     return make<MicroMipsThunk>(s);
   return make<MipsThunk>(s);
 }
 
-static Thunk *addThunkPPC32(const InputSection &isec, const Relocation &rel, Symbol &s) {
+static Thunk *addThunkPPC32(const InputSection &isec, const Relocation &rel,
+                            Symbol &s) {
   assert((rel.type == R_PPC_REL24 || rel.type == R_PPC_PLTREL24) &&
          "unexpected relocation type for thunk");
   return make<PPC32PltCallStub>(isec, rel, s);
 }
 
 static Thunk *addThunkPPC64(RelType type, Symbol &s) {
   assert(type == R_PPC64_REL24 && "unexpected relocation type for thunk");
   if (s.isInPlt())
     return make<PPC64PltCallStub>(s);
 
   if (config->picThunk)
     return make<PPC64PILongBranchThunk>(s);
 
   return make<PPC64PDLongBranchThunk>(s);
 }
 
 Thunk *addThunk(const InputSection &isec, Relocation &rel) {
   Symbol &s = *rel.sym;
+  int64_t a = rel.addend;
 
   if (config->emachine == EM_AARCH64)
-    return addThunkAArch64(rel.type, s);
+    return addThunkAArch64(rel.type, s, a);
 
   if (config->emachine == EM_ARM)
     return addThunkArm(rel.type, s);
 
   if (config->emachine == EM_MIPS)
     return addThunkMips(rel.type, s);
 
   if (config->emachine == EM_PPC)

lld/test/ELF/aarch64-thunk-pi.s

  .section .text_low, "ax", %progbits
  .globl low_target
  .type low_target, %function
 low_target:
  // Need thunk to high_target@plt
  bl high_target
  ret
 // CHECK: low_target:
-// CHECK-NEXT:       d8:       bl      #0x10 <__AArch64ADRPThunk_high_target>
+// CHECK-NEXT:       d8:       bl      #0x18 <__AArch64ADRPThunk_high_target>
 // CHECK-NEXT:                 ret
 
  .hidden low_target2
  .globl low_target2
  .type low_target2, %function
 low_target2:
- // Need thunk to high_target
+ // Need thunk to high_target2
  bl high_target2
+ // .text_high+8 = high_target2
+ bl .text_high+8
  ret
 // CHECK: low_target2:
-// CHECK-NEXT:       e0:       bl      #0x14 <__AArch64ADRPThunk_high_target2>
+// CHECK-NEXT:       e0:       bl      #0x1c <__AArch64ADRPThunk_high_target2>
+// CHECK-NEXT:       e4:       bl      #0x24 <__AArch64ADRPThunk_>
 // CHECK-NEXT:                 ret
 
 // Expect range extension thunks for .text_low
 // adrp calculation is (PC + signed immediate) & (!0xfff)
 // CHECK: __AArch64ADRPThunk_high_target:
-// CHECK-NEXT:       e8:       adrp    x16, #0x10000000
+// CHECK-NEXT:       f0:       adrp    x16, #0x10000000
 // CHECK-NEXT:                 add     x16, x16, #0x40
 // CHECK-NEXT:                 br      x16
 // CHECK: __AArch64ADRPThunk_high_target2:
-// CHECK-NEXT:       f4:       adrp    x16, #0x10000000
+// CHECK-NEXT:       fc:       adrp    x16, #0x10000000
+// CHECK-NEXT:                 add     x16, x16, #0x8
+// CHECK-NEXT:                 br      x16
+/// Identical to the previous one, but for the target .text_high+8.
+// CHECK: __AArch64ADRPThunk_:
+// CHECK-NEXT:      108:       adrp    x16, #0x10000000
 // CHECK-NEXT:                 add     x16, x16, #0x8
 // CHECK-NEXT:                 br      x16
 
 
  .section .text_high, "ax", %progbits
  .globl high_target
  .type high_target, %function
 high_target:

lld/test/ELF/aarch64-thunk-script.s

 // Check that we have the out of branch range calculation right. The immediate
 // field is signed so we have a slightly higher negative displacement.
  .section .text_low, "ax", %progbits
  .globl _start
  .type _start, %function
 _start:
  // Need thunk to high_target@plt
  bl high_target
+ // Need thunk to .text_high+4
+ bl .text_high+4
  ret
 
  .section .text_high, "ax", %progbits
  .globl high_target
  .type high_target, %function
 high_target:
  // No Thunk needed as we are within signed immediate range
  bl _start
  ret
 
 // CHECK: Disassembly of section .text_low:
 // CHECK-EMPTY:
 // CHECK-NEXT: _start:
-// CHECK-NEXT:     2000:       bl      #0x8 <__AArch64AbsLongThunk_high_target>
+// CHECK-NEXT:     2000:       bl      #0x10 <__AArch64AbsLongThunk_high_target>
+// CHECK-NEXT:     2004:       bl      #0x1c <__AArch64AbsLongThunk_>
 // CHECK-NEXT:                 ret
 // CHECK: __AArch64AbsLongThunk_high_target:
-// CHECK-NEXT:     2008:       ldr     x16, #0x8
+// CHECK-NEXT:     2010:       ldr     x16, #0x8
 // CHECK-NEXT:                 br      x16
 // CHECK: $d:
-// CHECK-NEXT:     2010:       00 20 00 08     .word   0x08002000
-// CHECK-NEXT:     2014:       00 00 00 00     .word   0x00000000
+// CHECK-NEXT:     2018:       00 20 00 08     .word   0x08002000
+// CHECK-NEXT:     201c:       00 00 00 00     .word   0x00000000
+// CHECK:      __AArch64AbsLongThunk_:
+// CHECK-NEXT:     2020:       ldr x16, #0x8
+// CHECK-NEXT:     2024:       br x16
+// CHECK:      $d:
+// CHECK-NEXT:     2028:       04 20 00 08     .word   0x08002004
+// CHECK-NEXT:     202c:       00 00 00 00     .word   0x00000000
 // CHECK: Disassembly of section .text_high:
 // CHECK-EMPTY:
 // CHECK-NEXT: high_target:
 // CHECK-NEXT:  8002000:       bl      #-0x8000000 <_start>
 // CHECK-NEXT:                 ret