[LLD][PowerPC] Implement GOT to PC-Rel relaxation

[nemanjai]

Summary

This patch implements the handling for the R_PPC64_PCREL_OPT relocation as well as the GOT relocation for the associated R_PPC64_GOT_PCREL34 relocation.

On Power10 targets with PC-Relative addressing, the linker can relax GOT-relative accesses to PC-Relative under some conditions. Since the sequence consists of a prefixed load, followed by a non-prefixed access (load or store), the linker needs to replace the first instruction (as the replacement instruction will be prefixed). The compiler communicates to the linker that this optimization is safe by placing the two aforementioned relocations on the GOT load (of the address).
The linker then does two things:

1. Convert the load from the got into a PC-Relative add to compute the address relative to the PC
2. Find the instruction referred to by the second relocation (R_PPC64_PCREL_OPT) and replace the first with the PC-Relative version of it

It is important to synchronize the mapping from legacy memory instructions to their PC-Relative form. Hence, this patch adds a file to be included by both the compiler and the linker so they're always in agreement.

[nemanjai]

lld/ELF/Arch/PPC64.cpp

42

Perhaps this entire enum should just be absorbed into the new one to eliminate confusion.

[nemanjai]

Add test cases for all the different instructions - thanks for your help producing these @NeHuang.
Run clang-format on the patch since the pre-commit build flagged a bunch of issues.

[stefanp]

Overall I think this looks good. I had a handful of comments but nothing major.
I wish we had a better way of doing the conversion between the old forms and the PCRel forms but I can't think of a better way.

[stefanp]

lld/ELF/Arch/PPC64.cpp

42

My two cents:
I would agree except that the file PPCLegacyToPCRelMap.def is meant specifically for PCRel (as is in the title of the file) and the update forms have nothing to do with that really...

We should clean these up but not with the PC Rel instructions.

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

72 ↗

(On Diff #280944)

nit:
You should probably add

DFLOADf32 = 48,  //  lfs
DFLOADf64 = 50,  //  lfd

The names DFLOADf32 and DFLOADf64 are internal names we have that won't mean anything to anyone that is not familiar with the compiler. The lfs and lfd are in the ISA.

196 ↗

(On Diff #280944)

I would almost prefer to have this function in the compiler in PPCPreEmitPeephole.cpp.
The function was in the compiler in the first place and it is only used in that one file.

If we do it that way we can get rid of the macro PPC_LEGACY_TO_PREFIXED_COMPILER.

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

50 ↗

(On Diff #280944)

nit:
I get the feeling that clang-format went a little crazy on this file and removed the indentation. :)
Should probably be a separate NFC patch.

[nemanjai]

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

72 ↗

(On Diff #280944)

For sure. Will do.

196 ↗

(On Diff #280944)

I don't think it would be a good thing to get rid of that macro. I have added the #error in this file to require that one of the two be defined so that this file cannot be included accidentally somewhere else and cause namespace pollution.

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

50 ↗

(On Diff #280944)

Yeah, I have no idea what clang-format's problem is - why is it modifying lines that I never touched in this patch? I don't mind pre-committing an NFC patch from running clang-format on this file.

[NeHuang]

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

36 ↗

(On Diff #280944)

nit: // Loads.

74 ↗

(On Diff #280944)

nit: // Stores.

123 ↗

(On Diff #280944)

Do we also need to add a FIXME here for PPC::LXVP and PPC::STXVP since they are not supported yet?

[NeHuang]

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

50 ↗

(On Diff #280944)

+1, It makes sense to have a separate NFC patch for clang-format changes. Phabricator complains those clang-format issues in https://reviews.llvm.org/D84360?id=279937

[MaskRay]

I haven't looked into the functional part of the patch. Just made some style issues and spot a few bugs due to incorrect -1lu uses.

lld/ELF/Arch/PPC64.cpp

66

return encoding >> 26;

658

Nit: uint64_t insn = readPrefixedInstruction(loc) & ~0xff000000fc000000; and move the comment below here

662

This file is not consistent on upper/lower case hexadecimals but for newer code we use lower case to be consistent with other files. The suffix lu is redundant (in any case not proper on 32-bit systems where unsigned long is 32-bit).

681

-1lu is wrong on a 32-bit platform: 4294967295.

pcRelInsn == UINT64_C(-1)

682

Consider using more errorOrWarn which can downgrade to warnings if --noinhibit-exec is implemented. error is used by code which can't really proceed. fatal is fatal error which should call exit immediately.

691

The two variables are only used once. Just write (insn & 0x0003ffff0000ffff) | pcRelInsn

lld/ELF/InputSection.cpp

1003

This is wrong on 32-bit platforms.

lld/test/ELF/Inputs/ppc64-got-to-pcrel-relaxation-def.s

5

.globl a, b, c, d can declare multiple variables

32

Are these .size and .type directives used? Dropping them can make the file much smaller

lld/test/ELF/ppc64-got-to-pcrel-relaxation.s

20

Some newer lld tests make sure that instructions are indented more than their containing labels

# CHECK-S-LABEL: <check_LBZ_STB>:
# CHECK-S-NEXT:    plbz 10

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

21 ↗

(On Diff #280944)

The parentheses are redundant

[MaskRay]

lld/test/ELF/ppc64-got-to-pcrel-relaxation.s

5

If a linked shared object is used to link another file, the shared object needs to specify --soname=t2 or another fixed string. Otherwise the DT_SONAME field of the other file is dependent on the full path of %t2.so, which can vary on different test machines.

[MaskRay]

lld/ELF/InputSection.cpp

1029

This assumes that relocation types on other architectures (which are represented by LLD's R_RELAX_GOT_PC or R_RELAX_GOT_PC_NOPIC) do not use the value R_PPC64_GOT_PCREL34. This is a subtle assumption we want to avoid. I think we need a new RelExpr: R_PPC64_RELAX_GOT_PCREL34.

lld/test/ELF/ppc64-got-to-pcrel-relaxation.s

169

Can these unrelated instructions be removed? We just need prefixed instructions and for the .reloc case, one instruction before .reloc

[nemanjai]

lld/ELF/Arch/PPC64.cpp

662

I can correct all the inconsistencies in an NFC patch after this lands. I will make sure I don't add any upper case ones.

lld/test/ELF/ppc64-got-to-pcrel-relaxation.s

169

I could remove them. I am reluctant to do that because the asm doesn't really make sense if we remove them. Although I realize this doesn't make a difference when linking, I just find it less visually jarring to have asm that is clearly incorrect (i.e. storing an undefined register in this case). And removing them doesn't save that many lines.

Of course, if you insist, I will remove them.

[nemanjai]

Fix up usage of unsigned long literal suffixes.
Add new rel-expr.
Fix up minor nits.
Shorten the test cases.

[nemanjai]

Remove erroneous unrelated update.

[sfertile]

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

1 ↗

(On Diff #282080)

This file does way more then I expect from a .def file. Are there other examples of a .def file defining so much data and code? If there isn't this needs to be moved to a header and potentially a source file since we don't want to change what a .def file represents in LLVM. If there is I still think we are better moving this to a header + source if it stay close to its current implementation using maps.

Alternatively if we split this up back to LLVM and LLD implementations it will make a clean .def fiie in lld.

My suggestion is to keep the switch in PPCPreEmitPeephole.cpp the way it is. I understand wanting to not have more then 1 place to modify when adding a new relaxable instruction is really desirable, but you have an error in LLD if we find the relocation on an unrecognized instruction to catch the mismatches. Combining both llc and lld implementations makes this much more complicated then it needs to be (and like I mentioned previously, inappropriate foe a .def file).

The LLD implementation I think we could turn into a proper .def file by working with the full encodings instead of primary opcodes and using a macro along the lines of:

MACRO(NONPCREL_ENCODING, PCREL_EQUIVALENT, OPCODE_MASK)

Then we define the enums in Arch/PPC64.cpp, and have 2 functions with switches over the NONPCREL_ENCODINGS: 1 that maps an encoding to its pc-relative equivalent encoding and one to map it to the operand mask. Instead of mixing in the bits for the non-primary opcode we just zero out the non-opcode bits, then use the 2 helper functions to implement getPCRelativeForm almost exactly the same as it is now.

As a bonus we can re-implement isInstructionUpdateForm as a macro in the .def file as well for consistency.

[nemanjai]

Need to account for a displacement on the access instruction and add some tests along the lines of

extern int Ext[40];
int test(int a) {
  return a + Ext[10];
}

lld/ELF/Arch/PPC64.cpp

687

As it turns out, the access instruction may actually have a displacement which needs to be added to the computed offset.
I will update this to account for that.

llvm/include/llvm/Target/PPCLegacyToPCRelMap.def

1 ↗

(On Diff #282080)

I agree with this and will rewrite this portion. This tight coupling between the compiler and the linker is quite unnecessary since there are at least two compilers on the platform and at least 3 linkers.

[nemanjai]

• Move implementation completely into LLD
• Add option to disable the optimization
• Handle computation of the total offset when the access instruction has a non-zero displacement
• Add tests with the new option as well as a test with non-zero displacement on the access instructions

[MaskRay]

lld/ELF/Options.td

407

Is this an option binutils will support as well?

Note that on x86, --no-relax can control whether R_X86_64_GOTPCRELX is optimized.
I hope PowerPC can adopt the same option.

There is a related thread https://sourceware.org/pipermail/binutils/2020-June/111571.html before I knew the PCREL GOT optimization activities on PowerPC.

[nemanjai]

lld/ELF/Options.td

407

I have asked Alan about support for this option in the GNU linkers. I'll get back to you when he responds.

I personally am not in favour of reducing the granularity by having all of this controlled by --no-relax because if we encounter bugs (or mismatches between compiler version and linker version) it is good to be able to turn off only the minimum set of optimizations.
A real example:
In Power9, we got new D-Form instructions which then ended up with relocations on them. The version of the linker in binutils on the various distros did not have support for those. Upgrading binutils was not an option for many customers, so they had to add --no-toc-optimize to their builds. If there was only --no-relax and that implied a bunch of other optimizations are turned off, such customers would be unnecessarily penalized in terms of performance.

Of course, the integration between LLD and Clang/LLVM is much more close so this particular issue is less likely to occur, but I think the argument for having finer granularity is still valid.

[sfertile]

lld/ELF/Arch/PPC64.cpp

405

Instead of shifting the primary opcode down to the least significant bits, and then shifting the secondary opcodes into the more significant bits, why don't we just use the significant opcode bits as the enum value.

For example: LDU 0xE8000001, LD = E8000000, LDU = 7C000000. It won't simplify this function since you still need the primary opcode to know how to mask out the encoding bits, but we could follow this patch with an NFC patch where we convert all the other uses of
DFormOpcd and XFormOpcode enums to use PPCInsn and get rid of the other 2 enums. If we do take this root I would suggest having separate enums for the 32-bit instructions and the prefixed instructions.

lld/ELF/Relocations.h

99

Should we drop the 'REL34' in the Expr? Its not really important how many bits it represents, and I think the REL is implied by 'PC' in R_RELAX_GOT_PC.

Also, should it be added to isRelExpr in Relocations.cpp?

[sfertile]

lld/ELF/Arch/PPC64.cpp

535

minor nit: The prefixed instruction in the sequence we are going to optimize should always be a paddi, since we relaxed the instruction referenced by the R_PPC64_GOT_PCREL34 from pld to paddi. My understanding could be wrong though, I expect we can only see the optimization relocation on a got-addressing sequence or a masked load/store sequence (which we don't support yet, but would also start with paddi).

[nemanjai]

lld/ELF/Arch/PPC64.cpp

405

While I agree that it is kind of odd to essentially reverse the positions of the primary and extended opcodes, I don't really follow what the details of your proposed update are.

How in your proposal does LDU go from 0xe8000001 to 0x7c000000?
I think that any proposal that puts the primary and secondary opcode in the same field is problematic.

How about keeping the bits where they are for DS and DQ form instructions that have extended opcodes. So lets take primary opcode 58 (LD, LDU, LWA):

LD  = 0xe8000000
LDU = 0xe8000001
LWA = 0xe8000002

Then in this function, all DS-Form (PO in { 57, 58, 61, 62 }) would be converted to a PPCInsn simply as encoding & 0xfc000003.

For the two versions of DQ-Form, we would have:
PO == 56 (only LQ) compute as encoding & 0xfc000000
PO == 61 compute as encoding & 0xfc000007
PO == 6 compute as encoding & 0xfc00000f

How does that sound?

535

Sorry, I should probably not use an extended mnemonic here.
The pla 3, 1000 is just paddi 3, 0, 1000, 1

lld/ELF/Relocations.h

99

Should we drop the 'REL34' in the Expr? Its not really important how many bits it represents, and I think the REL is implied by 'PC' in R_RELAX_GOT_PC.

I am definitely OK with this.

Also, should it be added to isRelExpr in Relocations.cpp?

This is just an omission on my part as I didn't know that existed. I'll add it.

[nemanjai]

lld/ELF/Relocations.h

99

Just to be clear, the name you are suggesting is R_PPC64_RELAX_GOT_PC, correct? I don't imagine you would like me to drop PPC64 since this is specific to this target.

[sfertile]

lld/ELF/Arch/PPC64.cpp

405

Sorry, I should have proofread my comment. The third one was suppsoed to be LDUX instead of `LDU again.

Your suggestion sounds perfect.

535

Thanks, I'm not that familiar with the new prefixed instructions yet so I didn't realize that was a paddi.

lld/ELF/Relocations.h

99

Yep, thats perfect.

[nemanjai]

Cleaned up the definitions of instructions and addressed other minor comments.

[nemanjai]

It is incorrect to relax this relocation if it is on a paddi. Bail out from the optimization if the relocation is on anything but a pld.

[sfertile]

lld/ELF/Arch/PPC64.cpp

72

This helper should return true for any 4-byte load/store instruction that is optimizable as part of a pc-rel access sequence? If so a better name would be isPCRelOptimizable or something along those lines.

lld/ELF/Arch/PPCInsns.def

11 ↗

(On Diff #284974)

This still isn't really the form of a .def file.

1. The enum values are supposed to be defined elsewhere, and the def file is included places where those definitions are visible.
2. I though we agreed to use the encodings directly as the enum values in our previous discussion. So LBZ would be 0x88000000 not 34.
3. We should have 2 distinct enum types, 1 for the non-prefixed instructions (which use an underlying 4 byte integral type) and the 1 for the prefixed instructions which uses an 8-byte type.

Then we should include the 4-byte instruction, the prefixed instruction it maps to, and the operand mask in the macro:
PCREL_OPT(Instruction, Prefixed_Equivalent, Operand_Mask)

Ex:

PCREL_OPT(LBZ, PLBZ, OPC_AND_RST)

And the helpers are implemented by defining the macro. For example to implement getPCRelativeForm:

switch(arg)
{
default:
  return NOINSN;
#define PCREL_OPT(Encoding, Pcrel_Encoding, Op_mask) case Encoding: return Pcrel_Encoding;
#include PPCInsns.def
#undef PCREL_OPT
}

Similarly:
For checkPPCInsn

switch(Encoding) 
{
default return false;
 #define PCREL_OPT(Encoding, Pcrel_Encoding, Op_mask) case Encoding: return true;
#include PPCInsns.def
#undef PCREL_OPT
}

ditto for getInsnMask

[nemanjai]

lld/ELF/Arch/PPC64.cpp

72

Not really. This is meant to check whether the underlying value is one of the enumerators or some value outside of the enumeration.
For example:
checkPPCinsn(static_cast<PPCInsn>(0x6100000A4000000) == true since it is PLWA
while
checkPPCinsn(static_cast<PPCInsn>(0x6100000A400ABCD) == false because it is not the value of any enumerator.

lld/ELF/Arch/PPCInsns.def

11 ↗

(On Diff #284974)

1. I don't think this is true. There are plenty of examples where the enumerators are specified in a .def file. For example include/llvm/IR/Instruction.def. I prefer to leave the defs in the .def file.
2. This is just a bit of miscommunication. I thought we were only talking about instructions for which we need the extended opcode to disambiguate.
3. I am not sure how this improves anything, but sure - since I am indifferent and you have a preference, I'll go with your preference.

[nemanjai]

Split the instruction enums into "Legacy" and "Prefixed" and use the .def file only for functions that access the enums as it simplifies the preprocessor logic.

[stefanp]

This patch makes sense to me.
I have a question about the access instruction but I don't think we should worry about it now.
LGTM!

lld/ELF/Arch/PPC64.cpp

536

Is it possible to have:

paddi 3, 0, 1000, 1
plwz 3, 20(2)

Where both instructions are prefixed.
I assume probably not (clang won't generate it?) but I figured it is worth asking.

685

I guess if the access instruction is already a prefixed instruction there won't be an equivalent PC Relative form of it and we will produce an error here.

[NeHuang]

Overall the patch looks good to me. I only have two questions and a clang-format nit which can be resolved before committing the code.

lld/ELF/Arch/PPC64.cpp

662

nit: clang-format, this can be resolved before committing the changes.

1311

I have two questions regarding void PPC64::relaxGot(uint8_t *loc, const Relocation &rel, uint64_t val) in lld/ELF/Arch/PCC64cpp line 619, where had similar check added as below:

case R_PPC64_GOT_PCREL34: {
    // Clear the first 8 bits of the prefix and the first 6 bits of the
...
    if ((insn & 0xfc000000) != 0xe4000000)
      error ("expected a 'pld' for got-indirect to pc-relative relaxing");
...

1. Did it catch the case that paddi instruction with R_PPC64_GOT_PCREL34 emitted from compiler side when it tried to optimize it?
2. After adding the new check here, does it make sense to remove the same check in PPC64::relaxGot?

[nemanjai]

lld/ELF/Arch/PPC64.cpp

536

At this point, it should not be possible (plus it would be a bad thing if it happened since we'd only look at a part of the instruction and part of the displacement).
We bail out before the call to this if we don't have a prefixed version of the instruction (which you already observed below).

662

Not sure what you mean, here. I ran clang-format on the patch and the pre-commit check didn't complain about the formatting.

685

Right, if the access instruction is already prefixed, we'd be looking at the prefix and not the instruction.

1311

1. This check was initially missing so we didn't know that we were mis-optimizing this.
2. I suppose it could be removed, but it is there defensively because if we ever get something other than a pld there, it is guaranteed that the resulting executable will be broken. I chose to go with an error there rather than an assert because it is crucial to fail in such a situation even if we build without asserts.

[NeHuang]

lld/ELF/Arch/PPC64.cpp

662

Yeah, the clang-format pre-check warning is gone after I refresh the page. It could be an outdated message.

[sfertile]

I believe we can have the optimization relocation on a code sequence like:

paddi ra, sym@pcrel
load/store rt, offset(ra)

If so we should have a test showing we relocate it properly even if we don't optimize it yet.

[nemanjai]

In D84360#2215915, @sfertile wrote:

I believe we can have the optimization relocation on a code sequence like:

paddi ra, sym@pcrel
load/store rt, offset(ra)

If so we should have a test showing we relocate it properly even if we don't optimize it yet.

But that's what check_LD_STD_W_PADDI does. Do you mean that I should add checks for the actual values?

[sfertile]

In D84360#2215944, @nemanjai wrote:

But that's what check_LD_STD_W_PADDI does. Do you mean that I should add checks for the actual values?

check_LD_STD_W_PADDI uses paddi referencing the got-entry of the symbol (sym@got@pcrel), My suggestion is for referencing a symbol pcrel (sym@pcrel) combined with a load/store with offset.

lld/ELF/Arch/PPC64.cpp

enum XFormOpcd {
   LDX = 21,
   STBX = 215,
   STHX = 407,
   STWX = 151,
   STDX = 149,
   ADD = 266,
 };
 
 enum DFormOpcd {
   LBZ = 34,
   LBZU = 35,
   LHZ = 40,
   LHZU = 41,
   LHAU = 43,
   LWZ = 32,
   LWZU = 33,
   LFSU = 49,
   LD = 58,
   LFDU = 51,
   STB = 38,
   STBU = 39,
   STH = 44,
   STHU = 45,
   STW = 36,
   STWU = 37,
   STFSU = 53,
   STFDU = 55,
   STD = 62,
   ADDI = 14
 };
 
+enum class PPCInsn : uint64_t {
+  NOINSN = 0,
+  PREFIX_MLS = 0x0610000000000000,
+  PREFIX_8LS = 0x0410000000000000,
+  // Loads.
+  LBZ = 34,
+  PLBZ = PREFIX_MLS, // Prefix only.
+  LHZ = 40,
+  PLHZ = PREFIX_MLS, // Prefix only.
+  LWZ = 32,
+  PLWZ = PREFIX_MLS, // Prefix only.
+  LHA = 42,
+  PLHA = PREFIX_MLS,              // Prefix only.
+  LWA = 58 | 0x80000000,          // (Encoding & 0x3) << 30.
+  PLWA = PREFIX_8LS | 0xa4000000, // Prefix | Primary opc.
+  LD = 58 | 0x0,                  // (Encoding & 0x3) << 30.
+  PLD = PREFIX_8LS | 0xe4000000,  // Prefix | Primary opc.
+  LFS = 48,
+  PLFS = PREFIX_MLS,                // Prefix only.
+  LXSSP = 57 | 0xc0000000,          // (Encoding & 0x3) << 30.
+  PLXSSP = PREFIX_8LS | 0xac000000, // Prefix | Primary opc.
+  LFD = 50,
+  PLFD = PREFIX_MLS,               // Prefix only.
+  LXSD = 57 | 0x80000000,          // (Encoding & 0x3) << 30.
+  PLXSD = PREFIX_8LS | 0xa8000000, // Prefix | Primary opc.
+  LXV = 61 | 0x20000000,           // (Encoding & 0x7) << 29.
+  PLXV = PREFIX_8LS | 0xc8000000,  // Prefix | Primary opc.
+  LXVP = 6 | 0x0,                  // (Encoding & 0xf) << 28.
+  PLXVP = PREFIX_8LS | 0xe8000000, // Prefix | Primary opc.
+
+  // Stores.
+  STB = 38,
+  PSTB = PREFIX_MLS, // Prefix only.
+  STH = 44,
+  PSTH = PREFIX_MLS, // Prefix only.
+  STW = 36,
+  PSTW = PREFIX_MLS, // Prefix only.
+  STD = 62,
+  PSTD = PREFIX_8LS | 0xf4000000, // Prefix | Primary opc.
+  STFS = 52,
+  PSTFS = PREFIX_MLS,                // Prefix only.
+  STXSSP = 61 | 0xc0000000,          // (Encoding & 0x3) << 30.
+  PSTXSSP = PREFIX_8LS | 0xbc000000, // Prefix | Primary opc.
+  STFD = 54,
+  PSTFD = PREFIX_MLS,               // Prefix only.
+  STXSD = 61 | 0x80000000,          // (Encoding & 0x3) << 30.
+  PSTXSD = PREFIX_8LS | 0xb8000000, // Prefix | Primary opc.
+  STXV = 61 | 0xa0000000,           // (Encoding & 0x7) << 29.
+  PSTXV = PREFIX_8LS | 0xd8000000,  // Prefix | Primary opc.
+  STXVP = 6 | 0x10000000,           // (Encoding & 0xf) << 28.
+  PSTXVP = PREFIX_8LS | 0xf8000000, // Prefix | Primary opc.
+};
+
+// Masks to apply to legacy instructions when converting them to prefixed,
+// pc-relative versions. For the most part, the primary opcode is shared
+// between the legacy instruction and the suffix of its prefixed version.
+// However, there are some instances where that isn't the case (DS-Form and
+// DQ-form instructions).
+enum class LegacyToPrefixMask : uint64_t {
+  OPC_AND_RST = 0xffe00000, // Primary opc (0-5) and R[ST] (6-10).
+  ONLY_RST = 0x3e00000,     // [RS]T (6-10).
+  ST_STX28_TO5 =
+      0x8000000003e00000, // S/T (6-10) - The [S/T]X bit moves from 28 to 5.
+};
+
 uint64_t elf::getPPC64TocBase() {
   // The TOC consists of sections .got, .toc, .tocbss, .plt in that order. The
   // TOC starts where the first of these sections starts. We always create a
   // .got when we see a relocation that uses it, so for us the start is always
   // the .got.
   uint64_t tocVA = in.got->getVA();
 
   // Per the ppc64-elf-linux ABI, The TOC base is TOC value plus 0x8000

 static uint16_t ha(uint64_t v) { return (v + 0x8000) >> 16; }
 static uint16_t higher(uint64_t v) { return v >> 32; }
 static uint16_t highera(uint64_t v) { return (v + 0x8000) >> 32; }
 static uint16_t highest(uint64_t v) { return v >> 48; }
 static uint16_t highesta(uint64_t v) { return (v + 0x8000) >> 48; }
 
 // Extracts the 'PO' field of an instruction encoding.
 static uint8_t getPrimaryOpCode(uint32_t encoding) { return (encoding >> 26); }
+static PPCInsn getPPCInsn(uint32_t encoding) {
+  uint64_t opc = getPrimaryOpCode(encoding);
+
+  // If the primary opcode is shared between multiple instructions, we need to
+  // fix it up to match the actual instruction we are after.
+
+  // For DQ-Form vector instrs, the two least significant bits are 01 and the
+  // field that differentiates them is 3 bits wide.
+  if (opc == 61 && (encoding & 0x3) == 0x1) // LXV/STXV.
+    opc |= (encoding & 0x7) << 29;
+
+  // For DS-Form instrs, there are 3 different primary opcodes and the two
+  // least significant bits differentiate instrs that share a PO.
+  else if (opc == 58 || opc == 57 || opc == 61)
+    opc |= (encoding & 0x3) << 30;
+
+  // Paired loads and stores from ISA3.1 use the 4 least significant bits to
+  // differentiate.
+  else if (opc == 6)
+    opc |= (encoding & 0xf) << 28;
+  return static_cast<PPCInsn>(opc);
+}
 
 static bool isDQFormInstruction(uint32_t encoding) {
   switch (getPrimaryOpCode(encoding)) {
   default:
     return false;
+  case 6: // Power10 paired loads/stores (lxvp, stxvp).
   case 56:
     // The only instruction with a primary opcode of 56 is `lq`.
     return true;
   case 61:
     // There are both DS and DQ instruction forms with this primary opcode.
     // Namely `lxv` and `stxv` are the DQ-forms that use it.
     // The DS 'XO' bits being set to 01 is restricted to DQ form.
     return (encoding & 3) == 0x1;
   }
 }
 
+static bool isDSFormInstruction(PPCInsn insn) {
+  switch (insn) {
+  default:
+    return false;
+  case PPCInsn::LWA:
+  case PPCInsn::LD:
+  case PPCInsn::LXSD:
+  case PPCInsn::LXSSP:
+  case PPCInsn::STD:
+  case PPCInsn::STXSD:
+  case PPCInsn::STXSSP:
+    return true;
+  }
+}
+
+static LegacyToPrefixMask getInsnMask(PPCInsn insn) {
+  if (isDSFormInstruction(insn) || insn == PPCInsn::LXVP ||
+      insn == PPCInsn::STXVP)
+    return LegacyToPrefixMask::ONLY_RST;
+  if (insn == PPCInsn::LXV || insn == PPCInsn::STXV)
+    return LegacyToPrefixMask::ST_STX28_TO5;
+  return LegacyToPrefixMask::OPC_AND_RST;
+}
+
+static PPCInsn getPCRelativeForm(PPCInsn insn) {
+  // If we are saving the opcode and the source/target register from the
+  // original instruction, the pc-relative form is just the prefix.
+  if (getInsnMask(insn) == LegacyToPrefixMask::OPC_AND_RST)
+    return PPCInsn::PREFIX_MLS;
+
+  switch (insn) {
+  default:
+    return PPCInsn::NOINSN;
+  case PPCInsn::LWA:
+    return PPCInsn::PLWA;
+  case PPCInsn::LD:
+    return PPCInsn::PLD;
+  case PPCInsn::LXSD:
+    return PPCInsn::PLXSD;
+  case PPCInsn::LXSSP:
+    return PPCInsn::PLXSSP;
+  case PPCInsn::LXV:
+    return PPCInsn::PLXV;
+  case PPCInsn::LXVP:
+    return PPCInsn::PLXVP;
+  case PPCInsn::STD:
+    return PPCInsn::PSTD;
+  case PPCInsn::STXSD:
+    return PPCInsn::PSTXSD;
+  case PPCInsn::STXSSP:
+    return PPCInsn::PSTXSSP;
+  case PPCInsn::STXV:
+    return PPCInsn::PSTXV;
+  case PPCInsn::STXVP:
+    return PPCInsn::PSTXVP;
+  }
+}
+static uint64_t getPCRelativeForm(uint32_t encoding) {
+  PPCInsn origInsn = getPPCInsn(encoding);
+  PPCInsn pcrelInsn = getPCRelativeForm(origInsn);
+  if (pcrelInsn == PPCInsn::NOINSN)
+    return UINT64_C(-1);
+  LegacyToPrefixMask origInsnMask = getInsnMask(origInsn);
+  uint64_t pcrelEncoding =
+      (uint64_t)pcrelInsn | (encoding & (uint64_t)origInsnMask);
+
+  // If the mask requires moving bit 28 to bit 5, do that now.
+  if (origInsnMask == LegacyToPrefixMask::ST_STX28_TO5)
+    pcrelEncoding |= (encoding & 0x8) << 23;
+  return pcrelEncoding;
+}
+
 static bool isInstructionUpdateForm(uint32_t encoding) {
   switch (getPrimaryOpCode(encoding)) {
   default:
     return false;
   case LBZU:
   case LHAU:
   case LHZU:
   case LWZU:
   case LFSU:
   case LFDU:
   case STBU:
   case STHU:
   case STWU:
   case STFSU:
   case STFDU:
     return true;
     // LWA has the same opcode as LD, and the DS bits is what differentiates
     // between LD/LDU/LWA
   case LD:
   case STD:
     return (encoding & 3) == 1;
   }
 }
 
+// Compute the total displacement between the prefixed instruction that gets
+// to the start of the data and the load/store instruction that has the offset
+// into the data structure.
+// For example:
+// pla 3, 1000
+// lwz 3, 20(3)
+// Should add up to 1020 for total displacement.
+static int64_t getTotalDisp(uint64_t prefixedInsn, uint32_t accessInsn) {
+  int64_t disp34 = llvm::SignExtend64(
+      ((prefixedInsn & 0x3ffff00000000) >> 16) | (prefixedInsn & 0xffff), 34);
+  int32_t disp16 = llvm::SignExtend32(accessInsn & 0xffff, 16);
+  // For DS and DQ form instructions, we need to mask out the XO bits.
+  if (isDQFormInstruction(accessInsn))
+    disp16 &= ~0xf;
+  else if (isDSFormInstruction(getPPCInsn(accessInsn)))
+    disp16 &= ~0x3;
+  return disp34 + disp16;
+}
+
 // There are a number of places when we either want to read or write an
 // instruction when handling a half16 relocation type. On big-endian the buffer
 // pointer is pointing into the middle of the word we want to extract, and on
 // little-endian it is pointing to the start of the word. These 2 helpers are to
 // simplify reading and writing in that context.
 static void writeFromHalf16(uint8_t *loc, uint32_t insn) {
   write32(config->isLE ? loc : loc - 2, insn);
 }
case R_PPC64_TOC16_LO_DS: {
     // "addi reg, 2, var@toc".
     uint32_t insn = readFromHalf16(loc);
     if (getPrimaryOpCode(insn) != LD)
       error("expected a 'ld' for got-indirect to toc-relative relaxing");
     writeFromHalf16(loc, (insn & 0x03ffffff) | 0x38000000);
     relocateNoSym(loc, R_PPC64_TOC16_LO, val);
     break;
   }
+  case R_PPC64_GOT_PCREL34: {
+    // Clear the first 8 bits of the prefix and the first 6 bits of the
+    // instruction (the primary opcode).
+    uint64_t insn = readPrefixedInstruction(loc) & ~0xff000000fc000000;
+
+    // Replace the cleared bits with the values for PADDI (0x600000038000000);
+    insn |= 0x600000038000000;
+    writePrefixedInstruction(loc, insn);
+    relocate(loc, rel, val);
+    break;
+  }
+  case R_PPC64_PCREL_OPT: {
+    // We can only relax this if the R_PPC64_GOT_PCREL34 at this offset can
+    // be relaxed. The eligibility for the relaxation needs to be determined
+    // on that relocation since this one does not relocate a symbol.
+    uint64_t insn = readPrefixedInstruction(loc);
+    uint32_t accessInsn = read32(loc + rel.addend);
+    uint64_t pcRelInsn = getPCRelativeForm(accessInsn);
+
+    // This error is not necessary for correctness but is emitted for now
+    // to ensure we don't miss these opportunities in real code. It can be
+    // removed at a later date.
+    if (pcRelInsn == UINT64_C(-1)) {
+      errorOrWarn(
+          "unrecognized instruction for R_PPC64_PCREL_OPT relaxation: 0x" +
+          Twine::utohexstr(accessInsn));
+      break;
+    }
+
+    int64_t totalDisp = getTotalDisp(insn, accessInsn);
+    if (!isInt<34>(totalDisp))
+      break; // Displacement doesn't fit.
+    // Convert the PADDI to the prefixed version of accessInsn and convert
+    // accessInsn to a nop.
+    writePrefixedInstruction(loc, pcRelInsn |
+                                      ((totalDisp & 0x3ffff0000) << 16) |
+                                      (totalDisp & 0xffff));
+    write32(loc + rel.addend, 0x60000000); // nop accessInsn.
+    break;
+  }
   default:
     llvm_unreachable("unexpected relocation type");
   }
 }
 
 void PPC64::relaxTlsGdToLe(uint8_t *loc, const Relocation &rel,
                            uint64_t val) const {
   // Reference: 3.7.4.2 of the 64-bit ELF V2 abi supplement.
RelExpr PPC64::getRelExpr(RelType type, const Symbol &s,
   case R_PPC64_GOT16_LO_DS:
     return R_GOT_OFF;
   case R_PPC64_TOC16:
   case R_PPC64_TOC16_DS:
   case R_PPC64_TOC16_HI:
   case R_PPC64_TOC16_LO:
     return R_GOTREL;
   case R_PPC64_GOT_PCREL34:
+  case R_PPC64_PCREL_OPT:
     return R_GOT_PC;
   case R_PPC64_TOC16_HA:
   case R_PPC64_TOC16_LO_DS:
     return config->tocOptimize ? R_PPC64_RELAX_TOC : R_GOTREL;
   case R_PPC64_TOC:
     return R_PPC64_TOCBASE;
   case R_PPC64_REL14:
   case R_PPC64_REL24:
case R_PPC64_GOT_PCREL34: {
     const uint64_t fullMask = 0x0003ffff0000ffff;
     checkInt(loc, val, 34, rel);
 
     uint64_t instr = readPrefixedInstruction(loc) & ~fullMask;
     writePrefixedInstruction(loc, instr | ((val & si0Mask) << 16) |
                              (val & si1Mask));
     break;
   }
+  // If we encounter a PCREL_OPT relocation that we won't optimize.
+  case R_PPC64_PCREL_OPT:
+    break;
   default:
     llvm_unreachable("unknown relocation");
   }
 }
 
 bool PPC64::needsThunk(RelExpr expr, RelType type, const InputFile *file,
                        uint64_t branchAddr, const Symbol &s, int64_t a) const {
   if (type != R_PPC64_REL14 && type != R_PPC64_REL24 &&
if (type == R_PPC64_REL14)
     return isInt<16>(offset);
   if (type == R_PPC64_REL24 || type == R_PPC64_REL24_NOTOC)
     return isInt<26>(offset);
   llvm_unreachable("unsupported relocation type used in branch");
 }
 
 RelExpr PPC64::adjustRelaxExpr(RelType type, const uint8_t *data,
                                RelExpr expr) const {
+  if ((type == R_PPC64_GOT_PCREL34 || type == R_PPC64_PCREL_OPT) &&
+      config->pcRelGotOptimize)
+    return R_PPC64_RELAX_GOT_PCREL34;
   if (expr == R_RELAX_TLS_GD_TO_IE)
     return R_RELAX_TLS_GD_TO_IE_GOT_OFF;
   if (expr == R_RELAX_TLS_LD_TO_LE)
     return R_RELAX_TLS_LD_TO_LE_ABS;
   return expr;
 }
 
 // Reference: 3.7.4.1 of the 64-bit ELF V2 abi supplement.
 // The general dynamic code sequence for a global `x` uses 4 instructions.
 // Instruction                    Relocation                Symbol
 // addis r3, r2, x@got@tlsgd@ha   R_PPC64_GOT_TLSGD16_HA      x

lld/ELF/Config.h

struct Configuration {
   bool isStatic = false;
   bool sysvHash = false;
   bool target1Rel;
   bool trace;
   bool thinLTOEmitImportsFiles;
   bool thinLTOIndexOnly;
   bool timeTraceEnabled;
   bool tocOptimize;
+  bool pcRelGotOptimize;
   bool undefinedVersion;
   bool unique;
   bool useAndroidRelrTags = false;
   bool warnBackrefs;
   std::vector<llvm::GlobPattern> warnBackrefsExclude;
   bool warnCommon;
   bool warnIfuncTextrel;
   bool warnMissingEntry;

lld/ELF/Driver.cpp

if (config->fixCortexA53Errata843419 && config->emachine != EM_AARCH64)
     error("--fix-cortex-a53-843419 is only supported on AArch64 targets");
 
   if (config->fixCortexA8 && config->emachine != EM_ARM)
     error("--fix-cortex-a8 is only supported on ARM targets");
 
   if (config->tocOptimize && config->emachine != EM_PPC64)
     error("--toc-optimize is only supported on the PowerPC64 target");
 
+  if (config->pcRelGotOptimize && config->emachine != EM_PPC64)
+    error("--pcrel-got-optimize is only supported on the PowerPC64 target");
+
   if (config->pie && config->shared)
     error("-shared and -pie may not be used together");
 
   if (!config->shared && !config->filterList.empty())
     error("-F may not be used without -shared");
 
   if (!config->shared && !config->auxiliaryList.empty())
     error("-f may not be used without -shared");
static void setConfigs(opt::InputArgList &args) {
   // We default to not writing the addends when using RELA relocations since
   // any standard conforming tool can find it in r_addend.
   config->writeAddends = args.hasFlag(OPT_apply_dynamic_relocs,
                                       OPT_no_apply_dynamic_relocs, false) ||
                          !config->isRela;
 
   config->tocOptimize =
       args.hasFlag(OPT_toc_optimize, OPT_no_toc_optimize, m == EM_PPC64);
+  config->pcRelGotOptimize = args.hasFlag(
+      OPT_pcrel_got_optimize, OPT_no_pcrel_got_optimize, m == EM_PPC64);
 }
 
 // Returns a value of "-format" option.
 static bool isFormatBinary(StringRef s) {
   if (s == "binary")
     return true;
   if (s == "elf" || s == "default")
     return false;

lld/ELF/InputSection.cpp

case R_PPC64_CALL: {
     // the callee. For local calls the caller and callee share the same
     // TOC base and so the TOC pointer initialization code should be skipped by
     // branching to the local entry point.
     return symVA - p + getPPC64GlobalEntryToLocalEntryOffset(sym.stOther);
   }
   case R_PPC64_TOCBASE:
     return getPPC64TocBase() + a;
   case R_RELAX_GOT_PC:
+  case R_PPC64_RELAX_GOT_PCREL34:
     return sym.getVA(a) - p;
   case R_RELAX_TLS_GD_TO_LE:
   case R_RELAX_TLS_IE_TO_LE:
   case R_RELAX_TLS_LD_TO_LE:
   case R_TLS:
     // It is not very clear what to return if the symbol is undefined. With
     // --noinhibit-exec, even a non-weak undefined reference may reach here.
     // Just return A, which matches R_ABS, and the behavior of some dynamic
else if (sec->areRelocsRela)
     sec->relocateNonAlloc<ELFT>(buf, sec->template relas<ELFT>());
   else
     sec->relocateNonAlloc<ELFT>(buf, sec->template rels<ELFT>());
 }
 
 void InputSectionBase::relocateAlloc(uint8_t *buf, uint8_t *bufEnd) {
   assert(flags & SHF_ALLOC);
   const unsigned bits = config->wordsize * 8;
+  uint64_t lastPPCRelaxedRelocOff = UINT64_C(-1);
 
   for (const Relocation &rel : relocations) {
     if (rel.expr == R_NONE)
       continue;
     uint64_t offset = rel.offset;
     if (auto *sec = dyn_cast<InputSection>(this))
       offset += sec->outSecOff;
     uint8_t *bufLoc = buf + offset;
     RelType type = rel.type;
 
     uint64_t addrLoc = getOutputSection()->addr + offset;
     RelExpr expr = rel.expr;
     uint64_t targetVA = SignExtend64(
         getRelocTargetVA(file, type, rel.addend, addrLoc, *rel.sym, expr),
         bits);
 
     switch (expr) {
     case R_RELAX_GOT_PC:
     case R_RELAX_GOT_PC_NOPIC:
       target->relaxGot(bufLoc, rel, targetVA);
       break;
+    case R_PPC64_RELAX_GOT_PCREL34: {
+      // The R_PPC64_PCREL_OPT relocation must appear immediately after
+      // R_PPC64_GOT_PCREL34 in the relocations table at the same offset.
+      // We can only relax R_PPC64_PCREL_OPT if we have also relaxed
+      // the associated R_PPC64_GOT_PCREL34 since only the latter has an
+      // associated symbol. So save the offset when relaxing R_PPC64_GOT_PCREL34
+      // and only relax the other if the saved offset matches.
+      if (type == R_PPC64_GOT_PCREL34)
+        lastPPCRelaxedRelocOff = offset;
+      if (type == R_PPC64_PCREL_OPT && offset != lastPPCRelaxedRelocOff)
+        break;
+      target->relaxGot(bufLoc, rel, targetVA);
+      break;
+    }
     case R_PPC64_RELAX_TOC:
       // rel.sym refers to the STT_SECTION symbol associated to the .toc input
       // section. If an R_PPC64_TOC16_LO (.toc + addend) references the TOC
       // entry, there may be R_PPC64_TOC16_HA not paired with
       // R_PPC64_TOC16_LO_DS. Don't relax. This loses some relaxation
       // opportunities but is safe.
       if (ppc64noTocRelax.count({rel.sym, rel.addend}) ||
           !tryRelaxPPC64TocIndirection(rel, bufLoc))

lld/ELF/Options.td

 
 defm time_trace_granularity: Eq<"time-trace-granularity",
   "Minimum time granularity (in microseconds) traced by time profiler">;
 
 defm toc_optimize : B<"toc-optimize",
     "(PowerPC64) Enable TOC related optimizations (default)",
     "(PowerPC64) Disable TOC related optimizations">;
 
+defm pcrel_got_optimize : B<"pcrel-got-optimize",
+    "(PowerPC64) Enable PCRel GOT related optimizations (default)",
+    "(PowerPC64) Disable PCRel GOT related optimizations">;
+
 def trace: F<"trace">, HelpText<"Print the names of the input files">;
 
 defm trace_symbol: Eq<"trace-symbol", "Trace references to symbols">;
 
 defm undefined: Eq<"undefined", "Force undefined symbol during linking">,
   MetaVarName<"<symbol>">;
 
 defm undefined_glob: Eq<"undefined-glob", "Force undefined symbol during linking">,

lld/ELF/Relocations.h

enum RelExpr {
   R_MIPS_GOT_OFF32,
   R_MIPS_TLSGD,
   R_MIPS_TLSLD,
   R_PPC32_PLTREL,
   R_PPC64_CALL,
   R_PPC64_CALL_PLT,
   R_PPC64_RELAX_TOC,
   R_PPC64_TOCBASE,
+  R_PPC64_RELAX_GOT_PCREL34,
   R_RISCV_ADD,
   R_RISCV_PC_INDIRECT,
 };
 
 // Architecture-neutral representation of relocation.
 struct Relocation {
   RelExpr expr;
   RelType type;

lld/test/ELF/Inputs/ppc64-got-to-pcrel-relaxation-def.s

+  .section  ".text"
+  .comm storeVal_vector,8,8
+  .comm useVal_vector,8,8
+  .globl storeVal_longlong, useAddr_longlong, useVal_longlong, storeVal_sshort
+  .globl useAddr_sshort, useVal_sshort, storeVal_sint, useAddr_sint, useVal_sint
+  .globl storeVal_double, useAddr_double, useVal_double, storeVal_float
+  .globl useAddr_float, useVal_float, storeVal_uint, storeVal_uint
+  .globl useVal_uint, storeVal_ushort, useAddr_ushort, useVal_ushort
+  .globl storeVal, useAddr, useVal
+  .section  ".data"
+  .align 3
+  .type storeVal_longlong, @object
+  .size storeVal_longlong, 8
+storeVal_longlong:
+  .quad 18
+useAddr_longlong:
+  .quad 17
+useVal_longlong:
+  .quad 16
+storeVal_sshort:
+  .short  -15
+useAddr_sshort:
+  .short  -14
+useVal_sshort:
+  .short  -13
+  .zero 2
+storeVal_sint:
+  .long -12
+useAddr_sint:
+  .long -11
+useVal_sint:
+  .long -10
+  .zero 4
+storeVal_double:
+  .long 858993459
+  .long 1076966195
+useAddr_double:
+  .long -1717986918
+  .long -1070589543
+useVal_double:
+  .long 0
+  .long 1076756480
+storeVal_float:
+  .long 1045220557
+useAddr_float:
+  .long -1050568294
+useVal_float:
+  .long 1095761920
+storeVal_uint:
+  .long 12
+useAddr_uint:
+  .long 11
+useVal_uint:
+  .long 10
+storeVal_ushort:
+  .short  1
+useAddr_ushort:
+  .short  10
+useVal_ushort:
+  .short  5
+storeVal:
+  .byte -1
+useAddr:
+  .byte 10
+useVal:
+  .byte 5

lld/test/ELF/ppc64-got-to-pcrel-relaxation.s

+# REQUIRES: ppc
+# RUN: llvm-mc -filetype=obj -triple=powerpc64le %s -o %t1.o
+# RUN: llvm-mc -filetype=obj -triple=powerpc64le %p/Inputs/ppc64-got-to-pcrel-relaxation-def.s -o %t2.o
+# RUN: ld.lld --shared %t2.o -o %t2.so --soname=t2
+# RUN: ld.lld %t1.o %t2.o -o %t
+# RUN: ld.lld %t1.o %t2.so -o %ts
+# RUN: ld.lld %t1.o %t2.o -o %tn --no-pcrel-got-optimize
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %t | FileCheck %s --check-prefix=CHECK-S
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %ts | FileCheck %s --check-prefix=CHECK-D
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %tn | FileCheck %s --check-prefix=CHECK-D
+
+# RUN: llvm-mc -filetype=obj -triple=powerpc64 %s -o %t1.o
+# RUN: llvm-mc -filetype=obj -triple=powerpc64 %p/Inputs/ppc64-got-to-pcrel-relaxation-def.s -o %t2.o
+# RUN: ld.lld --shared %t2.o -o %t2.so --soname=t2
+# RUN: ld.lld %t1.o %t2.o -o %t
+# RUN: ld.lld %t1.o %t2.so -o %ts
+# RUN: ld.lld %t1.o %t2.o -o %tn --no-pcrel-got-optimize
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %t | FileCheck %s --check-prefix=CHECK-S
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %ts | FileCheck %s --check-prefix=CHECK-D
+# RUN: llvm-objdump -d --no-show-raw-insn --mcpu=pwr10 %tn | FileCheck %s --check-prefix=CHECK-D
+
+# CHECK-S-LABEL: <check_LBZ_STB>:
+# CHECK-S-NEXT:    plbz 10
+# CHECK-S-NEXT:    paddi 9
+# CHECK-S-NEXT:    li 3, 0
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    rldicl 9, 9, 9, 60
+# CHECK-S-NEXT:    add 9, 9, 10
+# CHECK-S-NEXT:    pstb 9
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LBZ_STB>:
+# CHECK-D-NEXT:    pld 8
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    li 3, 0
+# CHECK-D-NEXT:    lbz 10, 0(8)
+# CHECK-D-NEXT:    rldicl 9, 9, 9, 60
+# CHECK-D-NEXT:    add 9, 9, 10
+# CHECK-D-NEXT:    pld 10
+# CHECK-D-NEXT:    stb 9, 0(10)
+# CHECK-D-NEXT:    blr
+check_LBZ_STB:
+  pld 8,useVal@got@pcrel(0),1
+.Lpcrel1:
+  pld 9,useAddr@got@pcrel(0),1
+  li 3,0
+  .reloc .Lpcrel1-8,R_PPC64_PCREL_OPT,.-(.Lpcrel1-8)
+  lbz 10,0(8)
+  rldicl 9,9,9,60
+  add 9,9,10
+  pld 10,storeVal@got@pcrel(0),1
+.Lpcrel2:
+  .reloc .Lpcrel2-8,R_PPC64_PCREL_OPT,.-(.Lpcrel2-8)
+  stb 9,0(10)
+  blr
+
+# CHECK-S-LABEL: <check_LHZ_STH>:
+# CHECK-S-NEXT:    plhz 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    psth 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LHZ_STH>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lhz 3, 0(9)
+# CHECK-D-NEXT:    nop
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    sth 3, 0(9)
+# CHECK-D-NEXT:    blr
+check_LHZ_STH:
+  pld 9,useVal_ushort@got@pcrel(0),1
+.Lpcrel3:
+  .reloc .Lpcrel3-8,R_PPC64_PCREL_OPT,.-(.Lpcrel3-8)
+  lhz 3,0(9)
+  pld 9,storeVal_ushort@got@pcrel(0),1
+.Lpcrel4:
+  .reloc .Lpcrel4-8,R_PPC64_PCREL_OPT,.-(.Lpcrel4-8)
+  sth 3,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LWZ_STW>:
+# CHECK-S-NEXT:    plwz 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstw 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LWZ_STW>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lwz 3, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stw 3, 0(9)
+# CHECK-D-NEXT:    blr
+check_LWZ_STW:
+  pld 9,useVal_uint@got@pcrel(0),1
+.Lpcrel5:
+  .reloc .Lpcrel5-8,R_PPC64_PCREL_OPT,.-(.Lpcrel5-8)
+  lwz 3,0(9)
+  pld 9,storeVal_uint@got@pcrel(0),1
+.Lpcrel6:
+  .reloc .Lpcrel6-8,R_PPC64_PCREL_OPT,.-(.Lpcrel6-8)
+  stw 3,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LFS_STFS>:
+# CHECK-S-NEXT:    plfs 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstfs 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LFS_STFS>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lfs 1, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stfs 1, 0(9)
+# CHECK-D-NEXT:    blr
+check_LFS_STFS:
+  pld 9,useVal_float@got@pcrel(0),1
+.Lpcrel7:
+  .reloc .Lpcrel7-8,R_PPC64_PCREL_OPT,.-(.Lpcrel7-8)
+  lfs 1,0(9)
+  pld 9,storeVal_float@got@pcrel(0),1
+.Lpcrel8:
+  .reloc .Lpcrel8-8,R_PPC64_PCREL_OPT,.-(.Lpcrel8-8)
+  stfs 1,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LFD_STFD>:
+# CHECK-S-NEXT:    plfd 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstfd 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LFD_STFD>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lfd 1, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stfd 1, 0(9)
+# CHECK-D-NEXT:    blr
+check_LFD_STFD:
+  pld 9,useVal_double@got@pcrel(0),1
+.Lpcrel9:
+  .reloc .Lpcrel9-8,R_PPC64_PCREL_OPT,.-(.Lpcrel9-8)
+  lfd 1,0(9)
+  pld 9,storeVal_double@got@pcrel(0),1
+.Lpcrel10:
+  .reloc .Lpcrel10-8,R_PPC64_PCREL_OPT,.-(.Lpcrel10-8)
+  stfd 1,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LWA_STW>:
+# CHECK-S-NEXT:    mr 9, 3
+# CHECK-S-NEXT:    plwa 3
+# CHECK-S-NEXT:    pstw 9
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LWA_STW>:
+# CHECK-D-NEXT:    mr 9, 3
+# CHECK-D-NEXT:    pld 8
+# CHECK-D-NEXT:    pld 10
+# CHECK-D-NEXT:    lwa 3, 0(8)
+# CHECK-D-NEXT:    stw 9, 0(10)
+# CHECK-D-NEXT:    blr
+check_LWA_STW:
+  mr 9,3
+  pld 8,useVal_sint@got@pcrel(0),1
+.Lpcrel11:
+  pld 10,storeVal_sint@got@pcrel(0),1
+.Lpcrel12:
+  .reloc .Lpcrel11-8,R_PPC64_PCREL_OPT,.-(.Lpcrel11-8)
+  lwa 3,0(8)
+  .reloc .Lpcrel12-8,R_PPC64_PCREL_OPT,.-(.Lpcrel12-8)
+  stw 9,0(10)
+  blr
+
+# CHECK-S-LABEL: <check_LHA_STH>:
+# CHECK-S-NEXT:    mr 9, 3
+# CHECK-S-NEXT:    plha 3
+# CHECK-S-NEXT:    psth 9
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LHA_STH>:
+# CHECK-D-NEXT:    mr 9, 3
+# CHECK-D-NEXT:    pld 8
+# CHECK-D-NEXT:    pld 10
+# CHECK-D-NEXT:    lha 3, 0(8)
+# CHECK-D-NEXT:    sth 9, 0(10)
+# CHECK-D-NEXT:    blr
+check_LHA_STH:
+  mr 9,3
+  pld 8,useVal_sshort@got@pcrel(0),1
+.Lpcrel13:
+  pld 10,storeVal_sshort@got@pcrel(0),1
+.Lpcrel14:
+  .reloc .Lpcrel13-8,R_PPC64_PCREL_OPT,.-(.Lpcrel13-8)
+  lha 3,0(8)
+  .reloc .Lpcrel14-8,R_PPC64_PCREL_OPT,.-(.Lpcrel14-8)
+  sth 9,0(10)
+  blr
+
+# CHECK-S-LABEL: <check_LD_STD>:
+# CHECK-S-NEXT:    pld 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstd 3
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LD_STD>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    ld 3, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    std 3, 0(9)
+# CHECK-D-NEXT:    blr
+check_LD_STD:
+  pld 9,useVal_longlong@got@pcrel(0),1
+.Lpcrel15:
+  .reloc .Lpcrel15-8,R_PPC64_PCREL_OPT,.-(.Lpcrel15-8)
+  ld 3,0(9)
+  pld 9,storeVal_longlong@got@pcrel(0),1
+.Lpcrel16:
+  .reloc .Lpcrel16-8,R_PPC64_PCREL_OPT,.-(.Lpcrel16-8)
+  std 3,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LXV_STXV>:
+# CHECK-S-NEXT:    plxv 34
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstxv 34
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LXV_STXV>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lxv 34, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stxv 34, 0(9)
+# CHECK-D-NEXT:    blr
+check_LXV_STXV:
+  pld 9,useVal_vector@got@pcrel(0),1
+.Lpcrel17:
+  .reloc .Lpcrel17-8,R_PPC64_PCREL_OPT,.-(.Lpcrel17-8)
+  lxv 34,0(9)
+  pld 9,storeVal_vector@got@pcrel(0),1
+.Lpcrel18:
+  .reloc .Lpcrel18-8,R_PPC64_PCREL_OPT,.-(.Lpcrel18-8)
+  stxv 34,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LXSSP_STXSSP>:
+# CHECK-S-NEXT:    plxssp 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstxssp 1
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LXSSP_STXSSP>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lxssp 1, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stxssp 1, 0(9)
+# CHECK-D-NEXT:    blr
+check_LXSSP_STXSSP:
+  pld 9,useVal_float@got@pcrel(0),1
+.Lpcrel19:
+  .reloc .Lpcrel19-8,R_PPC64_PCREL_OPT,.-(.Lpcrel19-8)
+  lxssp 1,0(9)
+  pld 9,storeVal_float@got@pcrel(0),1
+.Lpcrel20:
+  .reloc .Lpcrel20-8,R_PPC64_PCREL_OPT,.-(.Lpcrel20-8)
+  stxssp 1,0(9)
+  blr
+
+# CHECK-S-LABEL: <check_LXSD_STXSD>:
+# CHECK-S-NEXT:    plxsd 1, [[#ADDR1:]]
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstxsd 1, [[#ADDR2:]]
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LXSD_STXSD>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lxsd 1, 0(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stxsd 1, 0(9)
+# CHECK-D-NEXT:    blr
+check_LXSD_STXSD:
+  pld 9,useVal_double@got@pcrel(0),1
+.Lpcrel21:
+  .reloc .Lpcrel21-8,R_PPC64_PCREL_OPT,.-(.Lpcrel21-8)
+  lxsd 1,0(9)
+  pld 9,storeVal_double@got@pcrel(0),1
+.Lpcrel22:
+  .reloc .Lpcrel22-8,R_PPC64_PCREL_OPT,.-(.Lpcrel22-8)
+  stxsd 1,0(9)
+  blr
+
+# The respective displacements are computed relative to the PC which advanced
+# by 28 bytes in this function. Since the displacements in the two access
+# instructions are 8 and 32 so the displacements are those computed above minus
+# 20 and plus 4 (+8 - 28 and +32 - 28) respectively.
+# CHECK-S-LABEL: <check_LXSD_STXSD_aggr>:
+# CHECK-S-NEXT:    plxsd 1, [[#ADDR1-20]]
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    pstxsd 1, [[#ADDR2+4]]
+# CHECK-S-NEXT:    nop
+# CHECK-S-NEXT:    blr
+
+# CHECK-D-LABEL: <check_LXSD_STXSD_aggr>:
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    lxsd 1, 8(9)
+# CHECK-D-NEXT:    pld 9
+# CHECK-D-NEXT:    stxsd 1, 32(9)
+# CHECK-D-NEXT:    blr
+check_LXSD_STXSD_aggr:
+  pld 9,useVal_double@got@pcrel(0),1
+.Lpcrel23:
+  .reloc .Lpcrel23-8,R_PPC64_PCREL_OPT,.-(.Lpcrel23-8)
+  lxsd 1,8(9)
+  pld 9,storeVal_double@got@pcrel(0),1
+.Lpcrel24:
+  .reloc .Lpcrel24-8,R_PPC64_PCREL_OPT,.-(.Lpcrel24-8)
+  stxsd 1,32(9)
+  blr