ELF: Allow GOT relocs pointing to non-preemptable ifunc to resolve to an IRELATIVE where possible.

[pcc]

Summary

Non-GOT non-PLT relocations to non-preemptible ifuncs result in the
creation of a canonical PLT, which now takes the identity of the IFUNC
in the symbol table. This (a) ensures address consistency inside and
outside the module, and (b) fixes a bug where some of these relocations
end up pointing to the resolver.

Fixes (at least) PR40474 and PR40501.

[peter.smith]

Thanks for sharing the WIP. I think the approach will work and should match the behaviour of ld.bfd more closely. I've been trying to think of alternatives to redirecting all the symbols for the non GOT generating case. Off the top of my head:

• Keep the _PLT RelExpr variants and instead of changing the symbol alter the Expr for the non-plt non-got generating reference. This has the disadvantage of needing _PLT variants for any RelType that might fall into that category.
• Still create a new symbol for the non-plt, non-got generating reference, but instead of replacing the existing symbol, just change Sym for the affected relocations, leaving the others untouched. I think this would mean you wouldn't need to create DirectSym but it would mean having to store the symbol somewhere so that subsequent non-plt, non-got generating references could use it.

Not sure if either of those are better at this point though.

lld/ELF/InputSection.cpp

613 ↗

(On Diff #184016)

I think that the only use for the _PLT RelType is for redirection to ifunc PLT entries, you might be able to get rid of them from Relocations.h.

lld/ELF/Relocations.cpp

1029 ↗

(On Diff #184016)

I'd be tempted to reverse the condition to put the non-ifunc or preemptible case first; it is much simpler and more commonly encountered.

1033 ↗

(On Diff #184016)

As this comes out of WIP, I think it would be good to expand the comment to cover ifunc handling in general as there isn't a good external reference for them. The decisions below may seem a bit arbitrary without prior knowledge.

1048 ↗

(On Diff #184016)

Just to check I understand. This is the case where we have a non-generating address taking reference of the ifunc symbol that we need to redirect to the PLT entry for the ifunc symbol. Instead of changing the RelExpr to a _PLT form so that the PLT address of the ifunc is returned, we create a new non-ifunc symbol at the offset of the PLT entry for the ifunc symbol. This symbol inherits the PltIndex and GotIndex so that all RelExpr expressions resolve to the same address. As this symbol replaces the original one a new symbol (DirectSym) above needs to be created just for the purposes of the IRELATIVE relocation.

I think that this will have the property of changing the address of the symbol in the static and the dynamic symbol table (if the symbol is exported or a DSO references it). The former might be a little confusing but is otherwise harmless. Would be interesting to see what the dynamic loader will report to the shared library in that case. I note that ld.bfd gives an error message to recompile with PIC if the symbol is address taken using a non-got generating reference and exported; I think it would be worth following suit and giving an error.

1051 ↗

(On Diff #184016)

One advantage of the indirection is that we don't end up with a dynamic (ish) relocation such as IRELATIVE in the .text section for the case:

extern void f2();
void f() {
   void (*fp)(void) = fff; 
}

[pcc]

Address TODOs and review comments

[pcc]

PTAL, this should be ready for review now.

lld/ELF/InputSection.cpp

613 ↗

(On Diff #184016)

It looks like R_PLT is still used for R_MICROMIPS_26_S1. I was able to remove R_AARCH64_PLT_PAGE_PC, though, by noting that the only remaining caller of toPlt is now in the thunk creation code which I believe operates on relocations that were relaxed by fromPlt.

By the way, if I comment out that call to toPlt, all the tests still pass. We might want to add a test for that code.

lld/ELF/Relocations.cpp

1033 ↗

(On Diff #184016)

I wrote a long comment here which is basically a brain dump of everything that I've learned about ifuncs while working on PR40474.

1048 ↗

(On Diff #184016)

Just to check I understand. This is the case where we have a non-generating address taking reference of the ifunc symbol that we need to redirect to the PLT entry for the ifunc symbol. Instead of changing the RelExpr to a _PLT form so that the PLT address of the ifunc is returned, we create a new non-ifunc symbol at the offset of the PLT entry for the ifunc symbol.

Correct.

This symbol inherits the PltIndex and GotIndex so that all RelExpr expressions resolve to the same address.

Only all PLT-generating RelExprs resolve to the same address. The symbol would never have received a GotIndex because in the non-preemptible ifunc case we always just create a PLT if the relocation is GOT-generating. If a GOT is required after creating the canonical PLT (or if it was required before creating the PLT, which is now kept track of using the field GotInIgot), a GOT entry is created that points to the canonical PLT instead of the ifunc itself, to ensure address consistency between GOT and direct relocations. This handles the same case as what used to be handled by R_AARCH64_GOT_PAGE_PC_PLT.

As this symbol replaces the original one a new symbol (DirectSym) above needs to be created just for the purposes of the IRELATIVE relocation.

I think that this will have the property of changing the address of the symbol in the static and the dynamic symbol table (if the symbol is exported or a DSO references it).

Correct.

The former might be a little confusing but is otherwise harmless.

While updating the tests I realised that in fact the objdump output will normally look a little nicer if the ifunc is redirected because normally an ifunc is declared as an alias of a resolver, which means that the resolver's disassembly will be labelled with the resolver's symbol name and the plt will be labelled with the ifunc's symbol name.

Would be interesting to see what the dynamic loader will report to the shared library in that case.

Since I change the symbol type to STT_FUNC it should just be handled like a regular symbol, which means that the shared library will see the address of the plt. This is what I observed with your repro in PR40501 if I change it to take the address of fff in code rather than in an initializer.

I note that ld.bfd gives an error message to recompile with PIC if the symbol is address taken using a non-got generating reference and exported; I think it would be worth following suit and giving an error.

That might just be because it's difficult to teach ld.bfd to redirect a dynamic symbol. We handle that case just fine so I'm inclined not to do that unless there is a need to.

1051 ↗

(On Diff #184016)

In my implementation I've added a check that the section is writable before producing the IRELATIVE. This is similar to the check before producing a RELATIVE near the top of processRelocAux.

[peter.smith]

Thanks for the update. I'll be able to take a look at this in detail next week.

lld/ELF/InputSection.cpp

613 ↗

(On Diff #184016)

I'll add a test case next week. It may be possible to simplify toPlt as thunks should only use a subset of the expressions.

[peter.smith]

I've suggested a test case that fails when toPlt() is removed in D57743. Aiming to have comments on this review by the end of the day.

[peter.smith]

I've made a few suggestions on the comments, otherwise looks good to me. I think it is better to keep the complexity in one place and document it than spreading it out. It would be good to get some input from George and Rui, or someone else that hasn't been deep diving on ifuncs recently to see if they had any problems following the code. I've run this on some address comparison tests I had lying around for AArch64 and it looks good on these. I'll run these on Arm tomorrow, as that puts the .igot.plt in the .got and will comment if I find any problems, but I'm not expecting any.

I seem to remember FreeBsd making heavy use of ifuncs for some configuration, may be worth adding emaste as a reviewer to see if he can arrange for someone from that project to take a look and maybe run some tests.

lld/ELF/Relocations.cpp

1115 ↗

(On Diff #184460)

May I suggest highlighting the use of DirectSym as it wasn't immediately apparent what canonical meant in the context, although with the comment above explaining canonical this might be less of a problem. Maybe something like:
"Create a copy of the symbol to use as the target of the IRELATIVE relocation in the igotPlt. This is in case we make the PLT canonical later, which would overwrite the original symbol."

1142 ↗

(On Diff #184460)

Maybe worth expanding last part to "create a GOT entry pointing to the PLT entry for Sym."

1160 ↗

(On Diff #184460)

I initially missed the Sym.GotInIgot = false; and thought about what might happen if there was a GOT generating reference after canonicalisation. Perhaps something like:
"We previously encountered a GOT generating reference that we redirected to the Igot. Now that the PLT entry is canonical we must clear the redirection to the Igot and add a GOT entry. As we've changed the symbol type to STT_FUNC future got generating references will naturally use this GOT entry."

lld/test/ELF/gnu-ifunc-canon.s

2 ↗

(On Diff #184460)

I found it difficult to track the inputs from the outputs. For example it is easy to lose the type of reference from %t1.o, %t2.o and %t3.o. For example %t1.o has a RO pc-relative reference, %t2.o has a RO absolute reference and %t3.o has a RW absolute + offset reference. Maybe worth using %t-ro-pcrel.o etc.

Alternatively a comment above each ld.lld invocation with a description of the inputs, for example non-got-generating reference, expect canonical PLT.

I've not had a chance to check through each case yet. Will do that tomorrow and will post if I find any problems.

[pcc]

Thanks for the review. I've tested this using statically linked executables on Android on both AArch64 (with ifuncs in hwasan) and ARM (with ifuncs in libc.a).

@emaste do you know how I can test this on FreeBSD?

lld/test/ELF/gnu-ifunc-canon.s

2 ↗

(On Diff #184460)

I've renamed the files.

[pcc]

• Address review comments

[peter.smith]

Thanks for the update. My Arm tests were fine as well. This all looks good to me, I don't have any more comments at the moment.

[pcc]

I tested this patch by linking FreeBSD userspace with it and running their test suite [1], and there were no regressions. However, it appears that only a handful of userspace functions are using ifunc. The kernel uses more, but I couldn't test this on the kernel because they use a custom flag -z ifunc-noplt to link their kernel and this flag isn't implemented in upstream lld. In order to test this on the kernel the patch implementing the flag would need to be rebased on top of this change, and that's probably beyond the limit of what's reasonable to test here.

[1] https://www.freebsd.org/cgi/man.cgi?query=tests&apropos=0&sektion=7&manpath=FreeBSD+11-current&format=html

[ruiu]

Ed,

Did -z ifunc-noplt work well for FreeBSD kernel? If it works well, maybe we should merge it to the mainline lld tree?

[emaste]

In D57371#1390001, @ruiu wrote:

Ed,

Did -z ifunc-noplt work well for FreeBSD kernel? If it works well, maybe we should merge it to the mainline lld tree?

Yes it works well for us - the plan was for @markj to rebase and upstream it but it got preempted by other work; we'll bump the priority in light of this patch. Mark do you think we could test with this patch + -z ifunc-noplt in the near future?

[markj]

In D57371#1390027, @emaste wrote:

In D57371#1390001, @ruiu wrote:

Ed,

Did -z ifunc-noplt work well for FreeBSD kernel? If it works well, maybe we should merge it to the mainline lld tree?

Yes it works well for us - the plan was for @markj to rebase and upstream it but it got preempted by other work; we'll bump the priority in light of this patch. Mark do you think we could test with this patch + -z ifunc-noplt in the near future?

Yes, I will submit a patch to phabricator in the next couple of weeks. In the meantime I'll try and get this patch tested with -z ifunc-noplt before Monday.

[markj]

In D57371#1389410, @pcc wrote:

I tested this patch by linking FreeBSD userspace with it and running their test suite [1], and there were no regressions. However, it appears that only a handful of userspace functions are using ifunc. The kernel uses more, but I couldn't test this on the kernel because they use a custom flag -z ifunc-noplt to link their kernel and this flag isn't implemented in upstream lld. In order to test this on the kernel the patch implementing the flag would need to be rebased on top of this change, and that's probably beyond the limit of what's reasonable to test here.

For the purpose of testing it is possible to remove this flag without ill effect.

I built lld from git and built a kernel with this diff applied and -zifunc-noplt removed from the kernel; didn't have any problems booting. I ported the -zifunc-noplt patch and was able to boot with that flag enabled too. Note that the kernel doesn't have any non-preemptible ifuncs, though.

[pcc]

In D57371#1392254, @markj wrote:

In D57371#1389410, @pcc wrote:

I tested this patch by linking FreeBSD userspace with it and running their test suite [1], and there were no regressions. However, it appears that only a handful of userspace functions are using ifunc. The kernel uses more, but I couldn't test this on the kernel because they use a custom flag -z ifunc-noplt to link their kernel and this flag isn't implemented in upstream lld. In order to test this on the kernel the patch implementing the flag would need to be rebased on top of this change, and that's probably beyond the limit of what's reasonable to test here.

For the purpose of testing it is possible to remove this flag without ill effect.

I built lld from git and built a kernel with this diff applied and -zifunc-noplt removed from the kernel; didn't have any problems booting. I ported the -zifunc-noplt patch and was able to boot with that flag enabled too.

Good to hear, thanks.

Note that the kernel doesn't have any non-preemptible ifuncs, though.

Isn't the FreeBSD kernel an ET_EXEC? That should mean that all of its symbols are non-preemptible, right?

[markj]

In D57371#1392265, @pcc wrote:

In D57371#1392254, @markj wrote:

Note that the kernel doesn't have any non-preemptible ifuncs, though.

Isn't the FreeBSD kernel an ET_EXEC? That should mean that all of its symbols are non-preemptible, right?

Indeed, you are right.

[ruiu]

LGTM

Even though I believe what this patch does is correct, it seems really complicated to me, but that's not a fault of this patch. This part of lld code is mind-boggling. At this point, it is perhaps easier to rewrite it entirely than incrementally improving the code quality of this file. I don't think I would do this soon, but that's probably what we should keep in mind.

[pcc]

In D57371#1396776, @ruiu wrote:

LGTM

Even though I believe what this patch does is correct, it seems really complicated to me, but that's not a fault of this patch. This part of lld code is mind-boggling. At this point, it is perhaps easier to rewrite it entirely than incrementally improving the code quality of this file. I don't think I would do this soon, but that's probably what we should keep in mind.

Agreed that this needs a rewrite at some point. My biggest concern was that this patch causes relocations to be processed using different code paths depending on the order in which they are encountered (hence why the test checks both orders); fixing that will probably require a fundamental change here.

lld/ELF/Relocations.cpp

1209 ↗

(On Diff #185470)

Reading through the patch again I noticed that this ought to be R.Type not R.Sym->Type. I'll fix it in the commit but I can't seem to construct a test case exposing the change in behaviour.

ELF/InputSection.cpp

static uint64_t getRelocTargetVA(const InputFile *File, RelType Type, int64_t A,
   case R_RELAX_TLS_LD_TO_LE_ABS:
   case R_RELAX_GOT_PC_NOPIC:
     return Sym.getVA(A);
   case R_ADDEND:
     return A;
   case R_ARM_SBREL:
     return Sym.getVA(A) - getARMStaticBase(Sym);
   case R_GOT:
-  case R_GOT_PLT:
   case R_RELAX_TLS_GD_TO_IE_ABS:
     return Sym.getGotVA() + A;
   case R_GOTONLY_PC:
     return In.Got->getVA() + A - P;
   case R_GOTONLY_PC_FROM_END:
     return In.Got->getVA() + A - P + In.Got->getSize();
   case R_GOTREL:
     return Sym.getVA(A) - In.Got->getVA();
   case R_GOTREL_FROM_END:
     return Sym.getVA(A) - In.Got->getVA() - In.Got->getSize();
   case R_GOT_FROM_END:
   case R_RELAX_TLS_GD_TO_IE_END:
     return Sym.getGotOffset() + A - In.Got->getSize();
   case R_TLSLD_GOT_OFF:
   case R_GOT_OFF:
   case R_RELAX_TLS_GD_TO_IE_GOT_OFF:
     return Sym.getGotOffset() + A;
   case R_AARCH64_GOT_PAGE_PC:
-  case R_AARCH64_GOT_PAGE_PC_PLT:
   case R_AARCH64_RELAX_TLS_GD_TO_IE_PAGE_PC:
     return getAArch64Page(Sym.getGotVA() + A) - getAArch64Page(P);
   case R_GOT_PC:
   case R_RELAX_TLS_GD_TO_IE:
     return Sym.getGotVA() + A - P;
   case R_HEXAGON_GOT:
     return Sym.getGotVA() - In.GotPlt->getVA();
   case R_MIPS_GOTREL:
return In.MipsGot->getVA() + In.MipsGot->getGlobalDynOffset(File, Sym) -
            In.MipsGot->getGp(File);
   case R_MIPS_TLSLD:
     return In.MipsGot->getVA() + In.MipsGot->getTlsIndexOffset(File) -
            In.MipsGot->getGp(File);
   case R_AARCH64_PAGE_PC: {
     uint64_t Val = Sym.isUndefWeak() ? P + A : Sym.getVA(A);
     return getAArch64Page(Val) - getAArch64Page(P);
   }
-  case R_AARCH64_PLT_PAGE_PC: {
-    uint64_t Val = Sym.isUndefWeak() ? P + A : Sym.getPltVA() + A;
-    return getAArch64Page(Val) - getAArch64Page(P);
-  }
   case R_RISCV_PC_INDIRECT: {
     if (const Relocation *HiRel = getRISCVPCRelHi20(&Sym, A))
       return getRelocTargetVA(File, HiRel->Type, HiRel->Addend, Sym.getVA(),
                               *HiRel->Sym, HiRel->Expr);
     return 0;
   }
   case R_PC: {
     uint64_t Dest;

ELF/Relocations.h

 // List of target-independent relocation types. Relocations read
 // from files are converted to these types so that the main code
 // doesn't have to know about architecture-specific details.
 enum RelExpr {
   R_INVALID,
   R_ABS,
   R_ADDEND,
   R_AARCH64_GOT_PAGE_PC,
-  // The expression is used for IFUNC support. Describes PC-relative
-  // address of the memory page of GOT entry. This entry is used for
-  // a redirection to IPLT.
-  R_AARCH64_GOT_PAGE_PC_PLT,
   R_AARCH64_RELAX_TLS_GD_TO_IE_PAGE_PC,
   R_AARCH64_PAGE_PC,
-  R_AARCH64_PLT_PAGE_PC,
   R_AARCH64_TLSDESC_PAGE,
   R_ARM_SBREL,
   R_GOT,
-  // The expression is used for IFUNC support. Evaluates to GOT entry,
-  // containing redirection to the IPLT.
-  R_GOT_PLT,
   R_GOTONLY_PC,
   R_GOTONLY_PC_FROM_END,
   R_GOTREL,
   R_GOTREL_FROM_END,
   R_GOT_FROM_END,
   R_GOT_OFF,
   R_GOT_PC,
   R_HEXAGON_GOT,
struct RelocationOffsetComparator {
 
   bool operator()(uint64_t Val, const Relocation &Rel) {
     return Val < Rel.Offset;
   }
 };
 
 template <class ELFT> void scanRelocations(InputSectionBase &);
 
+void addIRelativeRelocs();
+
 class ThunkSection;
 class Thunk;
 struct InputSectionDescription;
 
 class ThunkCreator {
 public:
   // Return true if Thunks have been added to OutputSections
   bool createThunks(ArrayRef<OutputSection *> OutputSections);

ELF/Relocations.cpp

 }
 
 static bool isAbsoluteValue(const Symbol &Sym) {
   return isAbsolute(Sym) || Sym.isTls();
 }
 
 // Returns true if Expr refers a PLT entry.
 static bool needsPlt(RelExpr Expr) {
-  return isRelExprOneOf<R_PLT_PC, R_PPC_CALL_PLT, R_PLT, R_AARCH64_PLT_PAGE_PC,
-                        R_GOT_PLT, R_AARCH64_GOT_PAGE_PC_PLT>(Expr);
+  return isRelExprOneOf<R_PLT_PC, R_PPC_CALL_PLT, R_PLT>(Expr);
 }
 
 // Returns true if Expr refers a GOT entry. Note that this function
 // returns false for TLS variables even though they need GOT, because
 // TLS variables uses GOT differently than the regular variables.
 static bool needsGot(RelExpr Expr) {
   return isRelExprOneOf<R_GOT, R_GOT_OFF, R_HEXAGON_GOT, R_MIPS_GOT_LOCAL_PAGE,
                         R_MIPS_GOT_OFF, R_MIPS_GOT_OFF32, R_AARCH64_GOT_PAGE_PC,
-                        R_AARCH64_GOT_PAGE_PC_PLT, R_GOT_PC, R_GOT_FROM_END,
-                        R_GOT_PLT>(Expr);
+                        R_GOT_PC, R_GOT_FROM_END>(Expr);
 }
 
 // True if this expression is of the form Sym - X, where X is a position in the
 // file (PC, or GOT for example).
 static bool isRelExpr(RelExpr Expr) {
   return isRelExprOneOf<R_PC, R_GOTREL, R_GOTREL_FROM_END, R_MIPS_GOTREL,
                         R_PPC_CALL, R_PPC_CALL_PLT, R_AARCH64_PAGE_PC,
-                        R_AARCH64_PLT_PAGE_PC, R_RELAX_GOT_PC>(Expr);
+                        R_RELAX_GOT_PC>(Expr);
 }
 
 // Returns true if a given relocation can be computed at link-time.
 //
 // For instance, we know the offset from a relocation to its target at
 // link-time if the relocation is PC-relative and refers a
 // non-interposable function in the same executable. This function
 // will return true for such relocation.
 //
 // If this function returns false, that means we need to emit a
 // dynamic relocation so that the relocation will be fixed at load-time.
 static bool isStaticLinkTimeConstant(RelExpr E, RelType Type, const Symbol &Sym,
                                      InputSectionBase &S, uint64_t RelOff) {
   // These expressions always compute a constant
   if (isRelExprOneOf<R_GOT_FROM_END, R_GOT_OFF, R_HEXAGON_GOT, R_TLSLD_GOT_OFF,
                      R_MIPS_GOT_LOCAL_PAGE, R_MIPS_GOTREL, R_MIPS_GOT_OFF,
                      R_MIPS_GOT_OFF32, R_MIPS_GOT_GP_PC, R_MIPS_TLSGD,
-                     R_AARCH64_GOT_PAGE_PC, R_AARCH64_GOT_PAGE_PC_PLT, R_GOT_PC,
-                     R_GOTONLY_PC, R_GOTONLY_PC_FROM_END, R_PLT_PC, R_TLSGD_GOT,
+                     R_AARCH64_GOT_PAGE_PC, R_GOT_PC, R_GOTONLY_PC,
+                     R_GOTONLY_PC_FROM_END, R_PLT_PC, R_TLSGD_GOT,
                      R_TLSGD_GOT_FROM_END, R_TLSGD_PC, R_PPC_CALL_PLT,
                      R_TLSDESC_CALL, R_AARCH64_TLSDESC_PAGE, R_HINT,
                      R_TLSLD_HINT, R_TLSIE_HINT>(E))
     return true;
 
   // These never do, except if the entire file is position dependent or if
   // only the low bits are used.
-  if (E == R_GOT || E == R_GOT_PLT || E == R_PLT || E == R_TLSDESC)
+  if (E == R_GOT || E == R_PLT || E == R_TLSDESC)
     return Target->usesOnlyLowPageBits(Type) || !Config->Pic;
 
   if (Sym.IsPreemptible)
     return false;
   if (!Config->Pic)
     return true;
 
   // The size of a non preemptible symbol is a constant.

 }
 
 static RelExpr toPlt(RelExpr Expr) {
   switch (Expr) {
   case R_PPC_CALL:
     return R_PPC_CALL_PLT;
   case R_PC:
     return R_PLT_PC;
-  case R_AARCH64_PAGE_PC:
-    return R_AARCH64_PLT_PAGE_PC;
-  case R_AARCH64_GOT_PAGE_PC:
-    return R_AARCH64_GOT_PAGE_PC_PLT;
   case R_ABS:
     return R_PLT;
-  case R_GOT:
-    return R_GOT_PLT;
   default:
     return Expr;
   }
 }
 
 static RelExpr fromPlt(RelExpr Expr) {
   // We decided not to use a plt. Optimize a reference to the plt to a
   // reference to the symbol itself.
static void addPltEntry(PltSection *Plt, GotPltSection *GotPlt,
   GotPlt->addEntry(Sym);
   Rel->addReloc(
       {Type, GotPlt, Sym.getGotPltOffset(), !Sym.IsPreemptible, &Sym, 0});
 }
 
 template <class ELFT> static void addGotEntry(Symbol &Sym) {
   In.Got->addEntry(Sym);
 
-  RelExpr Expr;
-  if (Sym.isTls())
-    Expr = R_TLS;
-  else if (Sym.isGnuIFunc())
-    Expr = R_PLT;
-  else
-    Expr = R_ABS;
-
+  RelExpr Expr = Sym.isTls() ? R_TLS : R_ABS;
   uint64_t Off = Sym.getGotOffset();
 
   // If a GOT slot value can be calculated at link-time, which is now,
   // we can just fill that out.
   //
   // (We don't actually write a value to a GOT slot right now, but we
   // add a static relocation to a Relocations vector so that
   // InputSection::relocate will do the work for us. We may be able
if (Sym.isFunc()) {
     Sec.Relocations.push_back({Expr, Type, Offset, Addend, &Sym});
     return;
   }
 
   errorOrWarn("symbol '" + toString(Sym) + "' has no type" +
               getLocation(Sec, Sym, Offset));
 }
 
+struct IRelativeReloc {
+  RelType Type;
+  InputSectionBase *Sec;
+  uint64_t Offset;
+  Symbol *Sym;
+};
+
+static std::vector<IRelativeReloc> IRelativeRelocs;
+
 template <class ELFT, class RelTy>
 static void scanReloc(InputSectionBase &Sec, OffsetGetter &GetOffset, RelTy *&I,
                       RelTy *End) {
   const RelTy &Rel = *I;
   Symbol &Sym = Sec.getFile<ELFT>()->getRelocTargetSym(Rel);
   RelType Type;
 
   // Deal with MIPS oddity.
static void scanReloc(InputSectionBase &Sec, OffsetGetter &GetOffset, RelTy *&I,
   // lld allocates got entries to symbols on demand. Since we don't try to sort
   // the got entries in any way, we don't have to track which objects have
   // got-based small code model relocs. The .toc sections get placed after the
   // end of the linker allocated .got section and we do sort those so sections
   // addressed with small code model relocations come first.
   if (Config->EMachine == EM_PPC64 && isPPC64SmallCodeModelTocReloc(Type))
     Sec.File->PPC64SmallCodeModelTocRelocs = true;
 
-  // Strengthen or relax relocations.
-  //
-  // GNU ifunc symbols must be accessed via PLT because their addresses
-  // are determined by runtime.
-  //
-  // On the other hand, if we know that a PLT entry will be resolved within
-  // the same ELF module, we can skip PLT access and directly jump to the
-  // destination function. For example, if we are linking a main exectuable,
-  // all dynamic symbols that can be resolved within the executable will
-  // actually be resolved that way at runtime, because the main exectuable
-  // is always at the beginning of a search list. We can leverage that fact.
-  if (Sym.isGnuIFunc()) {
-    if (!Config->ZText && Config->WarnIfuncTextrel) {
-      warn("using ifunc symbols when text relocations are allowed may produce "
-           "a binary that will segfault, if the object file is linked with "
-           "old version of glibc (glibc 2.28 and earlier). If this applies to "
-           "you, consider recompiling the object files without -fPIC and "
-           "without -Wl,-z,notext option. Use -no-warn-ifunc-textrel to "
-           "turn off this warning." +
-           getLocation(Sec, Sym, Offset));
-    }
-    Expr = toPlt(Expr);
-  } else if (!Sym.IsPreemptible && Expr == R_GOT_PC && !isAbsoluteValue(Sym)) {
+  if (Sym.isGnuIFunc() && !Config->ZText && Config->WarnIfuncTextrel) {
+
+  // Relax relocations.
+  //
+  // If we know that a PLT entry will be resolved within the same ELF module, we
+  // can skip PLT access and directly jump to the destination function. For
+  // example, if we are linking a main exectuable, all dynamic symbols that can
+  // be resolved within the executable will actually be resolved that way at
+  // runtime, because the main exectuable is always at the beginning of a search
+  // list. We can leverage that fact.
+  if (!Sym.IsPreemptible && !Sym.isGnuIFunc()) {
+    if (Expr == R_GOT_PC && !isAbsoluteValue(Sym))
       Expr = Target->adjustRelaxExpr(Type, RelocatedAddr, Expr);
-  } else if (!Sym.IsPreemptible) {
+    else
       Expr = fromPlt(Expr);
   }
 
   // This relocation does not require got entry, but it is relative to got and
   // needs it to be created. Here we request for that.
   if (isRelExprOneOf<R_GOTONLY_PC, R_GOTONLY_PC_FROM_END, R_GOTREL,
                      R_GOTREL_FROM_END, R_PPC_TOC>(Expr))
     In.Got->HasGotOffRel = true;
 
   // Read an addend.
   int64_t Addend = computeAddend<ELFT>(Rel, End, Sec, Expr, Sym.isLocal());
 
   // Process some TLS relocations, including relaxing TLS relocations.
   // Note that this function does not handle all TLS relocations.
   if (unsigned Processed =
           handleTlsRelocation<ELFT>(Type, Sym, Sec, Offset, Addend, Expr)) {
     I += (Processed - 1);
     return;
   }
 
+  // Non-preemptible ifuncs require special handling. First, handle the usual
+  // case where the symbol isn't one of these.
+  if (!Sym.isGnuIFunc() || Sym.IsPreemptible) {
     // If a relocation needs PLT, we create PLT and GOTPLT slots for the symbol.
-  if (needsPlt(Expr) && !Sym.isInPlt()) {
-    if (Sym.isGnuIFunc() && !Sym.IsPreemptible)
-      addPltEntry<ELFT>(In.Iplt, In.IgotPlt, In.RelaIplt, Target->IRelativeRel,
-                        Sym);
-    else
+    if (needsPlt(Expr) && !Sym.isInPlt())
       addPltEntry<ELFT>(In.Plt, In.GotPlt, In.RelaPlt, Target->PltRel, Sym);
-  }
 
     // Create a GOT slot if a relocation needs GOT.
     if (needsGot(Expr)) {
       if (Config->EMachine == EM_MIPS) {
         // MIPS ABI has special rules to process GOT entries and doesn't
         // require relocation entries for them. A special case is TLS
         // relocations. In that case dynamic loader applies dynamic
         // relocations to initialize TLS GOT entries.
         // See "Global Offset Table" in Chapter 5 in the following document
         // for detailed description:
         // ftp://www.linux-mips.org/pub/linux/mips/doc/ABI/mipsabi.pdf
         In.MipsGot->addEntry(*Sec.File, Sym, Addend, Expr);
       } else if (!Sym.isInGot()) {
         addGotEntry<ELFT>(Sym);
       }
     }
+  } else {
+    // Handle a reference to a non-preemptible ifunc. These are special in a
+    // few ways:
+    //
+    // - Unlike most non-preemptible symbols, non-preemptible ifuncs do not have
+    //   a fixed value. But assuming that all references to the ifunc are
+    //   GOT-generating or PLT-generating, the handling of an ifunc is
+    //   relatively straightforward. We create a PLT entry in Iplt, which is
+    //   usually at the end of .plt, which makes an indirect call using a
+    //   matching GOT entry in IgotPlt, which is usually at the end of .got.plt.
+    //   The GOT entry is relocated using an IRELATIVE relocation in RelaIplt,
+    //   which is usually at the end of .rela.plt. Unlike most relocations in
+    //   .rela.plt, which may be evaluated lazily without -z now, dynamic
+    //   loaders evaluate IRELATIVE relocs eagerly, which means that for
+    //   IRELATIVE relocs only, GOT-generating relocations can point directly to
+    //   .got.plt without requiring a separate GOT entry.
+    //
+    // - Despite the fact that an ifunc does not have a fixed value, compilers
+    //   that are not passed -fPIC will assume that they do, and will emit
+    //   direct (non-GOT-generating, non-PLT-generating) relocations to the
+    //   symbol. This means that if a direct relocation to the symbol is
+    //   seen, the linker must set a value for the symbol, and this value must
+    //   be consistent no matter what type of reference is made to the symbol.
+    //   This can be done by creating a PLT entry for the symbol in the way
+    //   described above and making it canonical, that is, making all references
+    //   point to the PLT entry instead of the resolver. In lld we also store
+    //   the address of the PLT entry in the dynamic symbol table, which means
+    //   that the symbol will also have the same value in other modules.
+    //   Because the value loaded from the GOT needs to be consistent with
+    //   the value computed using a direct relocation, a non-preemptible ifunc
+    //   may end up with two GOT entries, one in .got.plt that points to the
+    //   address returned by the resolver and is used only by the PLT entry,
+    //   and another in .got that points to the PLT entry and is used by
+    //   GOT-generating relocations.
+    //
+    // - The fact that these symbols do not have a fixed value makes them an
+    //   exception to the general rule that a statically linked executable does
+    //   not require any form of dynamic relocation. To handle these relocations
+    //   correctly, the IRELATIVE relocations are stored in an array which a
+    //   statically linked executable's startup code must enumerate using the
+    //   linker-defined symbols __rela?_iplt_{start,end}.
+    //
+    // - An absolute relocation to a non-preemptible ifunc (such as a global
+    //   variable containing a pointer to the ifunc) needs to be relocated in
+    //   the exact same way as a GOT entry, so we can avoid needing to make the
+    //   PLT entry canonical by translating such relocations into IRELATIVE
+    //   relocations in the RelaIplt.
+    if (!Sym.isInPlt()) {
+      // Create PLT and GOTPLT slots for the symbol.
+      Sym.IsInIplt = true;
+
+      // Create a copy of the symbol to use as the target of the IRELATIVE
+      // relocation in the IgotPlt. This is in case we make the PLT canonical
+      // later, which would overwrite the original symbol.
+      //
+      // FIXME: Creating a copy of the symbol here is a bit of a hack. All
+      // that's really needed to create the IRELATIVE is the section and value,
+      // so ideally we should just need to copy those.
+      auto *DirectSym = make<Defined>(cast<Defined>(Sym));
+      addPltEntry<ELFT>(In.Iplt, In.IgotPlt, In.RelaIplt, Target->IRelativeRel,
+                        *DirectSym);
+      Sym.PltIndex = DirectSym->PltIndex;
+    }
+    if (Expr == R_ABS && Addend == 0 && (Sec.Flags & SHF_WRITE)) {
+      // We might be able to represent this as an IRELATIVE. But we don't know
+      // yet whether some later relocation will make the symbol point to a
+      // canonical PLT, which would make this either a dynamic RELATIVE (PIC) or
+      // static (non-PIC) relocation. So we keep a record of the information
+      // required to process the relocation, and after scanRelocs() has been
+      // called on all relocations, the relocation is resolved by
+      // addIRelativeRelocs().
+      IRelativeRelocs.push_back({Type, &Sec, Offset, &Sym});
+      return;
+    }
+    if (needsGot(Expr)) {
+      // Redirect GOT accesses to point to the Igot.
+      //
+      // This field is also used to keep track of whether we ever needed a GOT
+      // entry. If we did and we make the PLT canonical later, we'll need to
+      // create a GOT entry pointing to the PLT entry for Sym.
+      Sym.GotInIgot = true;
+    } else if (!needsPlt(Expr)) {
+      // Make the ifunc's PLT entry canonical by changing the value of its
+      // symbol to redirect all references to point to it.
+      unsigned EntryOffset = Sym.PltIndex * Target->PltEntrySize;
+      if (Config->ZRetpolineplt)
+        EntryOffset += Target->PltHeaderSize;
+
+      auto &D = cast<Defined>(Sym);
+      D.Section = In.Iplt;
+      D.Value = EntryOffset;
+      D.Size = 0;
+      // It's important to set the symbol type here so that dynamic loaders
+      // don't try to call the PLT as if it were an ifunc resolver.
+      D.Type = STT_FUNC;
+
+      if (Sym.GotInIgot) {
+        // We previously encountered a GOT generating reference that we
+        // redirected to the Igot. Now that the PLT entry is canonical we must
+        // clear the redirection to the Igot and add a GOT entry. As we've
+        // changed the symbol type to STT_FUNC future GOT generating references
+        // will naturally use this GOT entry.
+        //
+        // We don't need to worry about creating a MIPS GOT here because ifuncs
+        // aren't a thing on MIPS.
+        Sym.GotInIgot = false;
+        addGotEntry<ELFT>(Sym);
+      }
+    }
+  }
 
   processRelocAux<ELFT>(Sec, Expr, Type, Offset, Sym, Rel, Addend);
 }
 
 template <class ELFT, class RelTy>
 static void scanRelocs(InputSectionBase &Sec, ArrayRef<RelTy> Rels) {
   OffsetGetter GetOffset(Sec);
 

 
 template <class ELFT> void elf::scanRelocations(InputSectionBase &S) {
   if (S.AreRelocsRela)
     scanRelocs<ELFT>(S, S.relas<ELFT>());
   else
     scanRelocs<ELFT>(S, S.rels<ELFT>());
 }
 
+// Figure out which representation to use for any absolute relocs to
+// non-preemptible ifuncs that we visited during scanRelocs().
+void elf::addIRelativeRelocs() {
+  for (IRelativeReloc &R : IRelativeRelocs) {
+    if (R.Sym->Type == STT_GNU_IFUNC)
+      In.RelaIplt->addReloc(
+          {Target->IRelativeRel, R.Sec, R.Offset, true, R.Sym, 0});
+    else if (Config->Pic)
+      addRelativeReloc(R.Sec, R.Offset, R.Sym, 0, R_ABS, R.Type);
+    else
+      R.Sec->Relocations.push_back({R_ABS, R.Type, R.Offset, 0, R.Sym});
+  }
+  IRelativeRelocs.clear();
+}
+
 static bool mergeCmp(const InputSection *A, const InputSection *B) {
   // std::merge requires a strict weak ordering.
   if (A->OutSecOff < B->OutSecOff)
     return true;
 
   if (A->OutSecOff == B->OutSecOff) {
     auto *TA = dyn_cast<ThunkSection>(A);
     auto *TB = dyn_cast<ThunkSection>(B);

ELF/Symbols.h

public:
   uint64_t getSize() const;
   OutputSection *getOutputSection() const;
 
 protected:
   Symbol(Kind K, InputFile *File, StringRefZ Name, uint8_t Binding,
          uint8_t StOther, uint8_t Type)
       : File(File), NameData(Name.Data), NameSize(Name.Size), Binding(Binding),
         Type(Type), StOther(StOther), SymbolKind(K), NeedsPltAddr(false),
-        IsInIplt(false), IsInIgot(false), IsPreemptible(false),
+        IsInIplt(false), GotInIgot(false), IsPreemptible(false),
         Used(!Config->GcSections), NeedsTocRestore(false),
         ScriptDefined(false) {}
 
 public:
   // True the symbol should point to its PLT entry.
   // For SharedSymbol only.
   unsigned NeedsPltAddr : 1;
 
-  // True if this symbol is in the Iplt sub-section of the Plt.
+  // True if this symbol is in the Iplt sub-section of the Plt and the Igot
+  // sub-section of the .got.plt or .got.
   unsigned IsInIplt : 1;
 
-  // True if this symbol is in the Igot sub-section of the .got.plt or .got.
-  unsigned IsInIgot : 1;
+  // True if this symbol needs a GOT entry and its GOT entry is actually in
+  // Igot. This will be true only for certain non-preemptible ifuncs.
+  unsigned GotInIgot : 1;
 
   // True if this symbol is preemptible at load time.
   unsigned IsPreemptible : 1;
 
   // True if an undefined or shared symbol is used from a live section.
   unsigned Used : 1;
 
   // True if a call to this symbol needs to be followed by a restore of the

ELF/Symbols.cpp

static uint64_t getSymVA(const Symbol &Sym, int64_t &Addend) {
   llvm_unreachable("invalid symbol kind");
 }
 
 uint64_t Symbol::getVA(int64_t Addend) const {
   uint64_t OutVA = getSymVA(*this, Addend);
   return OutVA + Addend;
 }
 
-uint64_t Symbol::getGotVA() const { return In.Got->getVA() + getGotOffset(); }
+uint64_t Symbol::getGotVA() const {
+  if (GotInIgot)
+    return In.IgotPlt->getVA() + getGotPltOffset();
+  return In.Got->getVA() + getGotOffset();
+}
 
 uint64_t Symbol::getGotOffset() const {
   return GotIndex * Target->GotEntrySize;
 }
 
 uint64_t Symbol::getGotPltVA() const {
-  if (this->IsInIgot)
+  if (IsInIplt)
     return In.IgotPlt->getVA() + getGotPltOffset();
   return In.GotPlt->getVA() + getGotPltOffset();
 }
 
 uint64_t Symbol::getGotPltOffset() const {
-  if (IsInIgot)
+  if (IsInIplt)
     return PltIndex * Target->GotPltEntrySize;
   return (PltIndex + Target->GotPltHeaderEntriesNum) * Target->GotPltEntrySize;
 }
 
 uint64_t Symbol::getPPC64LongBranchOffset() const {
   assert(PPC64BranchltIndex != 0xffff);
   return PPC64BranchltIndex * Target->GotPltEntrySize;
 }

ELF/SyntheticSections.cpp

 // On PowerPC64 the GotPltSection type is SHT_NOBITS so we have to follow suit
 // with the IgotPltSection.
 IgotPltSection::IgotPltSection()
     : SyntheticSection(SHF_ALLOC | SHF_WRITE,
                        Config->EMachine == EM_PPC64 ? SHT_NOBITS : SHT_PROGBITS,
                        Target->GotPltEntrySize, getIgotPltName()) {}
 
 void IgotPltSection::addEntry(Symbol &Sym) {
-  Sym.IsInIgot = true;
   assert(Sym.PltIndex == Entries.size());
   Entries.push_back(&Sym);
 }
 
 size_t IgotPltSection::getSize() const {
   return Entries.size() * Target->GotPltEntrySize;
 }
 
for (auto &I : Entries) {
     Target->writePlt(Buf + Off, Got, Plt, B->PltIndex, RelOff);
     Off += Target->PltEntrySize;
   }
 }
 
 template <class ELFT> void PltSection::addEntry(Symbol &Sym) {
   Sym.PltIndex = Entries.size();
   RelocationBaseSection *PltRelocSection = In.RelaPlt;
-  if (IsIplt) {
+  if (IsIplt)
     PltRelocSection = In.RelaIplt;
-    Sym.IsInIplt = true;
-  }
   unsigned RelOff =
       static_cast<RelocationSection<ELFT> *>(PltRelocSection)->getRelocOffset();
   Entries.push_back(std::make_pair(&Sym, RelOff));
 }
 
 size_t PltSection::getSize() const {
   return HeaderSize + Entries.size() * Target->PltEntrySize;
 }

ELF/Writer.cpp

for (Symbol *S : Symtab->getSymbols())
     if (!S->IsPreemptible)
       S->IsPreemptible = computeIsPreemptible(*S);
 
   // Scan relocations. This must be done after every symbol is declared so that
   // we can correctly decide if a dynamic relocation is needed.
   if (!Config->Relocatable)
     forEachRelSec(scanRelocations<ELFT>);
 
+  addIRelativeRelocs();
+
   if (In.Plt && !In.Plt->empty())
     In.Plt->addSymbols();
   if (In.Iplt && !In.Iplt->empty())
     In.Iplt->addSymbols();
 
   if (!Config->AllowShlibUndefined) {
     // Error on undefined symbols in a shared object, if all of its DT_NEEDED
     // entires are seen. These cases would otherwise lead to runtime errors

test/ELF/Inputs/gnu-ifunc-canon-ro-abs.s

+.rodata
+.8byte ifunc

test/ELF/Inputs/gnu-ifunc-canon-ro-pcrel.s

+.rodata
+.4byte ifunc - .

test/ELF/Inputs/gnu-ifunc-canon-rw-addend.s

+.data
+.8byte ifunc + 1

test/ELF/aarch64-gnu-ifunc-address-pie.s

 # REQUIRES: aarch64
 # RUN: llvm-mc -filetype=obj -triple=aarch64-none-linux-gnu %s -o %t.o
 # RUN: ld.lld -pie %t.o -o %tout
 # RUN: llvm-objdump -D %tout | FileCheck %s
 # RUN: llvm-readobj -r %tout | FileCheck %s -check-prefix=CHECK-RELOCS
 
 # Test that when we take the address of a preemptible ifunc using -fpie, we can
 # handle the case when the ifunc is in the same translation unit as the address
 # taker. In this case the compiler knows that ifunc is not defined in a shared
 # library so it can use a non got generating relative reference.
 .text
 .globl myfunc
 .type myfunc,@gnu_indirect_function
 myfunc:
+.globl myfunc_resolver
+.type myfunc_resolver,@function
+myfunc_resolver:
  ret
 
 .text
 .globl main
 .type main,@function
 main:
  adrp x8, myfunc
  add  x8, x8, :lo12: myfunc
  ret
 
-# CHECK: 0000000000010000 myfunc:
+# CHECK: 0000000000010000 myfunc_resolver:
 # CHECK-NEXT:    10000: c0 03 5f d6   ret
 # CHECK: 0000000000010004 main:
 # CHECK-NEXT:    10004: 08 00 00 90   adrp  x8, #0
 # x8 = 0x10000
 # CHECK-NEXT:    10008: 08 41 00 91   add x8, x8, #16
 # x8 = 0x10010 = .plt for myfunc
 # CHECK-NEXT:    1000c: c0 03 5f d6   ret
 # CHECK-NEXT: Disassembly of section .plt:
-# CHECK-NEXT: 0000000000010010 .plt:
+# CHECK-NEXT: 0000000000010010 myfunc:
 # CHECK-NEXT:    10010: 90 00 00 90   adrp  x16, #65536
 # CHECK-NEXT:    10014: 11 02 40 f9   ldr x17, [x16]
 # CHECK-NEXT:    10018: 10 02 00 91   add x16, x16, #0
 # CHECK-NEXT:    1001c: 20 02 1f d6   br  x17
 
 # CHECK-RELOCS: Relocations [
 # CHECK-RELOCS-NEXT:   Section {{.*}} .rela.plt {
 # CHECK-RELOCS-NEXT:     0x20000 R_AARCH64_IRELATIVE - 0x10000
 # CHECK-RELOCS-NEXT:   }
 # CHECK-RELOCS-NEXT: ]

test/ELF/aarch64-gnu-ifunc-address.s

 .text
 .globl main
 .type main,@function
 main:
  adrp x8, :got:myfunc
  ldr  x8, [x8, :got_lo12:myfunc]
  ret
 # CHECK:   0000000000010004 main:
-# x8 = 0x30000
-# CHECK-NEXT:    10004: 08 01 00 90     adrp    x8, #131072
-# x8 = 0x300e0 = .got entry for myfunc with R_AARCH64_GLOB_DAT
-# CHECK-NEXT:    10008: 08 71 40 f9     ldr     x8, [x8, #224]
+# x8 = 0x20000
+# CHECK-NEXT:    10004: 88 00 00 90     adrp    x8, #65536
+# x8 = 0x200a0 = .got entry for myfunc with R_AARCH64_GLOB_DAT
+# CHECK-NEXT:    10008: 08 51 40 f9     ldr     x8, [x8, #160]
 # CHECK-NEXT:    1000c: c0 03 5f d6     ret
 
 # CHECK: Disassembly of section .got:
-# CHECK-NEXT: 00000000000300e0 .got:
+# CHECK-NEXT: 00000000000200a0 .got:
 
 # CHECK-RELOCS: Relocations [
 # CHECK-RELOCS-NEXT:   Section {{.*}} .rela.dyn {
-# CHECK-RELOCS-NEXT:     0x300E0 R_AARCH64_GLOB_DAT myfunc 0x0
-# CHECK-RELOCS-NEXT:   }
-# CHECK-RELOCS-NEXT:   Section {{.*}} .rela.plt {
-# CHECK-RELOCS-NEXT:     0x20018 R_AARCH64_JUMP_SLOT myfunc 0x0
+# CHECK-RELOCS-NEXT:     0x200A0 R_AARCH64_GLOB_DAT myfunc 0x0
 # CHECK-RELOCS-NEXT:   }
 # CHECK-RELOCS-NEXT: ]

test/ELF/aarch64-gnu-ifunc2.s

 # REQUIRES: aarch64
 # RUN: llvm-mc -filetype=obj -triple=aarch64-none-linux-gnu %s -o %t.o
 # RUN: ld.lld -static %t.o -o %tout
 # RUN: llvm-objdump -D %tout | FileCheck %s
 # RUN: llvm-readobj -r %tout | FileCheck %s --check-prefix=RELOC
 
 # CHECK:      Disassembly of section .text:
 # CHECK-NEXT: myfunc:
 # CHECK-NEXT:   210000:
 
 # CHECK:      main:
-# adrp x8, 0x230000, 0x230000 == address in .got
-# CHECK-NEXT:   210004: {{.*}} adrp    x8, #131072
+# adrp x8, 0x220000, 0x220000 == address in .got.plt
+# CHECK-NEXT:   210004: {{.*}} adrp    x8, #65536
 # CHECK-NEXT:   210008: {{.*}} ldr     x8, [x8]
 # CHECK-NEXT:   21000c: {{.*}} ret
 
 # CHECK:      Disassembly of section .plt:
 # CHECK-NEXT: .plt:
 # adrp x16, 0x220000, 0x220000 == address in .got.plt
 # CHECK-NEXT:   210010: {{.*}} adrp    x16, #65536
 # CHECK-NEXT:   210014: {{.*}} ldr     x17, [x16]
 # CHECK-NEXT:   210018: {{.*}} add     x16, x16, #0
 # CHECK-NEXT:   21001c: {{.*}} br      x17
 
 # CHECK:      Disassembly of section .got.plt:
 # CHECK-NEXT: .got.plt:
 # CHECK-NEXT:   220000:
 
-# CHECK:      Disassembly of section .got:
-# CHECK-NEXT: .got:
-# 0x210010 == address in .plt
-# CHECK-NEXT:   230000: 10 00 21 00
-
 # RELOC:      Relocations [
 # RELOC-NEXT:   Section {{.*}} .rela.plt {
 # RELOC-NEXT:     0x220000 R_AARCH64_IRELATIVE - 0x210000
 # RELOC-NEXT:   }
 # RELOC-NEXT: ]
 
 .text
 .globl myfunc

test/ELF/aarch64-gnu-ifunc3.s

 # REQUIRES: aarch64
 # RUN: llvm-mc -filetype=obj -triple=aarch64-none-linux-gnu %s -o %t.o
 # RUN: ld.lld -static %t.o -o %tout
 # RUN: llvm-objdump -D %tout | FileCheck %s
 # RUN: llvm-readobj -r %tout | FileCheck %s --check-prefix=RELOC
 
 # The address of myfunc is the address of the PLT entry for myfunc.
 # The adrp to myfunc should generate a PLT entry and a got entry with an
 # irelative relocation.
 .text
 .globl myfunc
 .type myfunc,@gnu_indirect_function
 myfunc:
+.globl myfunc_resolver
+.type myfunc_resolver,@function
+myfunc_resolver:
  ret
 
 .text
 .globl _start
 .type _start,@function
 _start:
  adrp x8, myfunc
  add  x8, x8, :lo12:myfunc
  ret
 
 # CHECK: Disassembly of section .text:
-# CHECK-NEXT: myfunc:
+# CHECK-NEXT: myfunc_resolver:
 # CHECK-NEXT:   210000: c0 03 5f d6   ret
 # CHECK: _start:
 # adrp x8, 0x210000 + 0x10 from add == .plt entry
 # CHECK-NEXT:   210004: 08 00 00 90   adrp  x8, #0
 # CHECK-NEXT:   210008: 08 41 00 91   add x8, x8, #16
 # CHECK-NEXT:   21000c: c0 03 5f d6   ret
 # CHECK-NEXT: Disassembly of section .plt:
-# CHECK-NEXT: .plt:
+# CHECK-NEXT: myfunc:
 # adrp x16, 0x220000, 0x220000 == address in .got.plt
 # CHECK-NEXT:   210010: 90 00 00 90   adrp  x16, #65536
 # CHECK-NEXT:   210014: 11 02 40 f9   ldr x17, [x16]
 # CHECK-NEXT:   210018: 10 02 00 91   add x16, x16, #0
 # CHECK-NEXT:   21001c: 20 02 1f d6   br  x17
 # CHECK-NEXT: Disassembly of section .got.plt:
 # CHECK-NEXT: .got.plt:
 # 0x210010 == address in .plt
 # CHECK-NEXT:   220000: 10 00 21 00
 # CHECK-NEXT:   220004: 00 00 00 00
 
 # RELOC: Relocations [
 # RELOC-NEXT:  Section (1) .rela.plt {
 # RELOC-NEXT:    0x220000 R_AARCH64_IRELATIVE - 0x210000
 # RELOC-NEXT:  }
 # RELOC-NEXT: ]

test/ELF/gnu-ifunc-canon.s

+// REQUIRES: x86
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %S/Inputs/gnu-ifunc-canon-ro-pcrel.s -o %t-ro-pcrel.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %S/Inputs/gnu-ifunc-canon-ro-abs.s -o %t-ro-abs.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %S/Inputs/gnu-ifunc-canon-rw-addend.s -o %t-rw-addend.o
+// RUN: ld.lld %t.o -o %t1
+// RUN: llvm-readobj -relocs %t1 | FileCheck --check-prefix=IREL2 %s
+// RUN: ld.lld %t.o %t-ro-pcrel.o -o %t2
+// RUN: llvm-readobj -relocs %t2 | FileCheck --check-prefix=IREL1 %s
+// RUN: ld.lld %t.o %t-ro-abs.o -o %t3
+// RUN: llvm-readobj -relocs %t3 | FileCheck --check-prefix=IREL1 %s
+// RUN: ld.lld %t.o %t-rw-addend.o -o %t4
+// RUN: llvm-readobj -relocs %t4 | FileCheck --check-prefix=IREL1 %s
+// RUN: llvm-objdump -s %t4 | FileCheck --check-prefix=DUMP %s
+// RUN: ld.lld %t.o %t-rw-addend.o -o %t4a -z retpolineplt
+// RUN: llvm-readobj -relocs %t4a | FileCheck --check-prefix=IREL1 %s
+// RUN: llvm-objdump -s %t4a | FileCheck --check-prefix=DUMP2 %s
+// RUN: ld.lld %t-ro-pcrel.o %t.o -o %t5
+// RUN: llvm-readobj -relocs %t5 | FileCheck --check-prefix=IREL1 %s
+// RUN: ld.lld %t-ro-abs.o %t.o -o %t6
+// RUN: llvm-readobj -relocs %t6 | FileCheck --check-prefix=IREL1 %s
+// RUN: ld.lld %t-rw-addend.o %t.o -o %t7
+// RUN: llvm-readobj -relocs %t7 | FileCheck --check-prefix=IREL1 %s
+// RUN: ld.lld %t.o -o %t8 -pie
+// RUN: llvm-readobj -relocs %t8 | FileCheck --check-prefix=IREL2 %s
+// RUN: ld.lld %t.o %t-ro-pcrel.o -o %t9 -pie
+// RUN: llvm-readobj -relocs %t9 | FileCheck --check-prefix=IREL1-REL2 %s
+// RUN: ld.lld %t.o %t-rw-addend.o -o %t10 -pie
+// RUN: llvm-readobj -relocs %t10 | FileCheck --check-prefix=IREL1-REL3 %s
+// RUN: ld.lld %t-ro-pcrel.o %t.o -o %t11 -pie
+// RUN: llvm-readobj -relocs %t11 | FileCheck --check-prefix=IREL1-REL2 %s
+// RUN: ld.lld %t-rw-addend.o %t.o -o %t12 -pie
+// RUN: llvm-readobj -relocs %t12 | FileCheck --check-prefix=IREL1-REL3 %s
+
+// Two relocs, one for the GOT and the other for .data.
+// IREL2-NOT: R_X86_64_
+// IREL2: .rela.plt
+// IREL2-NEXT: R_X86_64_IRELATIVE
+// IREL2-NEXT: R_X86_64_IRELATIVE
+// IREL2-NOT: R_X86_64_
+
+// One reloc for the canonical PLT.
+// IREL1-NOT: R_X86_64_
+// IREL1: .rela.plt
+// IREL1-NEXT: R_X86_64_IRELATIVE
+// IREL1-NOT: R_X86_64_
+
+// One reloc for the canonical PLT and two RELATIVE relocations pointing to it,
+// one in the GOT and one in .data.
+// IREL1-REL2-NOT: R_X86_64_
+// IREL1-REL2: .rela.dyn
+// IREL1-REL2-NEXT: R_X86_64_RELATIVE
+// IREL1-REL2-NEXT: R_X86_64_RELATIVE
+// IREL1-REL2: .rela.plt
+// IREL1-REL2-NEXT: R_X86_64_IRELATIVE
+// IREL1-REL2-NOT: R_X86_64_
+
+// One reloc for the canonical PLT and three RELATIVE relocations pointing to it,
+// one in the GOT and two in .data.
+// IREL1-REL3-NOT: R_X86_64_
+// IREL1-REL3: .rela.dyn
+// IREL1-REL3-NEXT: R_X86_64_RELATIVE
+// IREL1-REL3-NEXT: R_X86_64_RELATIVE
+// IREL1-REL3-NEXT: R_X86_64_RELATIVE
+// IREL1-REL3: .rela.plt
+// IREL1-REL3-NEXT: R_X86_64_IRELATIVE
+// IREL1-REL3-NOT: R_X86_64_
+
+// Make sure the static relocations look right, both with and without headers.
+// DUMP: Contents of section .plt:
+// DUMP-NEXT: 201010
+// DUMP: Contents of section .data:
+// DUMP-NEXT: 202000 10102000 00000000 11102000 00000000
+// DUMP: Contents of section .got:
+// DUMP-NEXT: 203000 10102000 00000000
+
+// DUMP2: Contents of section .plt:
+// DUMP2-NEXT: 201010
+// DUMP2: Contents of section .data:
+// DUMP2-NEXT: 202000 40102000 00000000 41102000 00000000
+// DUMP2: Contents of section .got:
+// DUMP2-NEXT: 203000 40102000 00000000
+
+lea ifunc@gotpcrel(%rip), %rbx
+
+.type ifunc STT_GNU_IFUNC
+.globl ifunc
+ifunc:
+ret
+
+.data
+.8byte ifunc

test/ELF/gnu-ifunc-i386.s

 // CHECK-NEXT:   Size: 0
 // CHECK-NEXT:   Binding: Global
 // CHECK-NEXT:   Type: None
 // CHECK-NEXT:   Other: 0
 // CHECK-NEXT:   Section: .text
 // CHECK-NEXT: }
 // CHECK-NEXT: Symbol {
 // CHECK-NEXT:   Name: bar
+// CHECK-NEXT:   Value: 0x401030
+// CHECK-NEXT:   Size: 0
+// CHECK-NEXT:   Binding: Global
+// CHECK-NEXT:   Type: Function
+// CHECK-NEXT:   Other: 0
+// CHECK-NEXT:   Section: .plt
+// CHECK-NEXT: }
+// CHECK-NEXT: Symbol {
+// CHECK-NEXT:   Name: bar_resolver
 // CHECK-NEXT:   Value: 0x401001
 // CHECK-NEXT:   Size: 0
 // CHECK-NEXT:   Binding: Global
-// CHECK-NEXT:   Type: GNU_IFunc
+// CHECK-NEXT:   Type: Function
 // CHECK-NEXT:   Other: 0
 // CHECK-NEXT:   Section: .text
 // CHECK-NEXT: }
 // CHECK-NEXT: Symbol {
 // CHECK-NEXT:   Name: foo
+// CHECK-NEXT:   Value: 0x401020
+// CHECK-NEXT:   Size: 0
+// CHECK-NEXT:   Binding: Global
+// CHECK-NEXT:   Type: Function
+// CHECK-NEXT:   Other: 0
+// CHECK-NEXT:   Section: .plt
+// CHECK-NEXT: }
+// CHECK-NEXT: Symbol {
+// CHECK-NEXT:   Name: foo_resolver
 // CHECK-NEXT:   Value: 0x401000
 // CHECK-NEXT:   Size: 0
 // CHECK-NEXT:   Binding: Global
-// CHECK-NEXT:   Type: GNU_IFunc
+// CHECK-NEXT:   Type: Function
 // CHECK-NEXT:   Other: 0
 // CHECK-NEXT:   Section: .text
 // CHECK-NEXT: }
 // CHECK-NEXT:]
 
 // DISASM: Disassembly of section .text:
-// DISASM-NEXT: foo:
+// DISASM-NEXT: foo_resolver:
 // DISASM-NEXT:    401000: c3 retl
-// DISASM: bar:
+// DISASM: bar_resolver:
 // DISASM-NEXT:    401001: c3 retl
 // DISASM:      _start:
 // DISASM-NEXT:    401002: e8 19 00 00 00 calll 25
 // DISASM-NEXT:    401007: e8 24 00 00 00 calll 36
 // DISASM-NEXT:    40100c: ba d4 00 40 00 movl $4194516, %edx
 // DISASM-NEXT:    401011: ba e4 00 40 00 movl $4194532, %edx
 // DISASM-NEXT: Disassembly of section .plt:
-// DISASM-NEXT: .plt:
+// DISASM-NEXT: foo:
 // DISASM-NEXT:    401020: ff 25 00 20 40 00 jmpl *4202496
 // DISASM-NEXT:    401026: 68 10 00 00 00 pushl $16
 // DISASM-NEXT:    40102b: e9 e0 ff ff ff jmp -32 <_start+0xe>
+// DISASM:      bar:
 // DISASM-NEXT:    401030: ff 25 04 20 40 00 jmpl *4202500
 // DISASM-NEXT:    401036: 68 18 00 00 00 pushl $24
 // DISASM-NEXT:    40103b: e9 d0 ff ff ff jmp -48 <_start+0xe>
 
 .text
 .type foo STT_GNU_IFUNC
 .globl foo
 foo:
+.type foo_resolver STT_FUNC
+.globl foo_resolver
+foo_resolver:
  ret
 
 .type bar STT_GNU_IFUNC
 .globl bar
 bar:
+.type bar_resolver STT_FUNC
+.globl bar_resolver
+bar_resolver:
  ret
 
 .globl _start
 _start:
  call foo
  call bar
  movl $__rel_iplt_start,%edx
  movl $__rel_iplt_end,%edx