[PowerPC] Implement Load VSX Vector and Sign Extend and Zero Extend

[Conanap]

Summary

Implemented Builtins for Load VSX Vector and sign extend and zero extend along with appropriate test cases. The new instructions are used for ZEXT but SEXT will continue to use the previously generated ASM.

[amyk]

I have a few comments from the last time we looked at this together.
Just also FYI that the backend tests will be apart of the p10-vsx-builtins.ll file introduced in https://reviews.llvm.org/D82431.

clang/lib/Headers/altivec.h

16514

Nit: space after this comment.

16517

I realized that I think we are supposed to use unaligned_vec_si128 and unaligned_vec_ui128 instead of vector signed __int128 and vector unsigned __int128 when we return (for here, and other places).

16535

Nit: space after this comment.

16881

Remove unrelated whitespace change.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13767

Move this comment above the combine function since this comment was intended to describe the function.

13774

Can omit the { } since we just have a single return underneath.

13781

Can omit the { } since we just have a single return underneath.

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

583

Can remove this comment.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

7

Please update this comment to be more descriptive. Maybe something like,

These test cases tests that zero extending loads utilize the Load VSX Vector Rightmost (lxvr[b|h|w|d]) instructions in Power10.

9

Please remove the comments/checks above the functions.

[Conanap]

Addressed Amy's comments regarding documentation of the changes.

[Conanap]

Fixed return signature for the open coded functions in altivec.h

[anil9]

clang/test/CodeGen/builtins-ppc-p10vector.c

17

nit: It seems that most pull requests follow an ordering like first signed declaration and then unsigned, declaration, this one follows too , except the above two lines. And should the above declarations of chars, be along with these lines ?

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

56

I am not too familiar with the builtins but I never saw a check outside of the two braces in the test cases before, is it not posible to include it inside the test cases ?

[Conanap]

Reordered some declarations in test cases and removed unecessary extra CHECKs.

[lei]

Please move encoding tests to ppc64-encoding-ISA31.[txt|s].

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13792

nit: check ValidLDType first:

if (! ValidLDType || (LD->getValueType(0) != MVT::i128) ||
    (LD->getExtensionType() != ISD::ZEXTLOAD) ||)

llvm/lib/Target/PowerPC/PPCISelLowering.h

528

nit: This shouldn't matter to code gen, but it would be nice to have this defined closer to other similar instructions. Maybe after LXVD2X?

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

225

nit: indentation

[Conanap]

Addressed Anil's comments with regards to the test cases

[Conanap]

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

56

these were extra checks that I should've removed; thanks for catching it

[Conanap]

Fixed some formatting stuff.

[Conanap]

Addressed Lei's comments

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.h

487

missing ,?

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

220

We should put all such def at the top of this file under:

// PowerPC specific type constraints.

224

Put these under the type constraints section and doc with:

// PowerPC specific DAG Nodes.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

4

remve extra line. Makes the comment harder to read.

11

You don't need the local_unnamed_addr #0

[Conanap]

Fixed a missing comma; ensured it builds.

[Conanap]

fixed a missing comma and ensure that it compiles

[Conanap]

Moved some code to the top of the file as per Lei's request

[Conanap]

Addressed Lei's comments

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13791

nit: don't need () aroud !ValidLDType

13797

I don't think we need to explicitly say this sine everything we do here is for pattern matching to appropriate instructions in the backend...

13798

Can just merge this into the next line and remove this tmp value.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

7

Please add tests for BE.

[Conanap]

Now depends on D83364.

[Conanap]

Also removed unnecessary brackets and comments.

[amyk]

Please update this patch to remove the instruction defs and MC tests. Also, you can update the patch to put your backend llc tests in the file I've introduced in: https://reviews.llvm.org/D82467

clang/test/CodeGen/builtins-ppc-p10vector.c

6

unintended change?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13796

You did not assign anything here?

[Conanap]

Removed unecessary comments and unintended changes.

[Conanap]

Removed duplicate test code found in the instructions and MC Test implementation of VSX Vector store and load with sign extend or zero extend; re-added a test file that was omitted in the last diff update.

[amyk]

I think the patterns for the load instructions may have accidentally been deleted. Please add them back to the patch. Thank you.

[Conanap]

Restored accidentally deleted pattern, removed duplicate tests, moved new tests to another pre-existing file instead.

[stefanp]

The title of the patch mentions both zero extend and sign extend.
However, it seems that we only have instructions for the zero extend case. Is that right?
I see both types of tests in:
test/CodeGen/builtins-ppc-p10vector.c
But I only see codegen tests for the zreo extend version.
test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

I bring it up because I want to make sure that this is intentional.

clang/lib/Headers/altivec.h

16516

I'm a little confused about this.
Your function signature says we return vector unsigned __int128 but the return statement casts to unaligned_vec_si128. Is that how this is supposed to look?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13792

It looks like you are doing this for zero extend only.
What about ISD::EXTLOAD? In that case the upper bits are undefined anyway so we can just assume a zero extend right?

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

41 ↗

(On Diff #277952)

nit:
Was this line auto-added?
Usually this comment is added at the top of the file by the script. I don't think it is required here.

[NeHuang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13785

nit: if we are loading either a byte....

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

13

nit: one space is enough.

[Conanap]

Return signature fix, added recognition for ISD:EXTLOAD, some code clean up.

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13795

nit: indentation.

13798

nit: indentation

[Conanap]

Addressed formatting comments

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13785

I'm not too sure what you mean, would you be able to elaborate?
The comment is:

This transformation is only valid if the we are loading either a byte,
halfword, word, or doubleword.

Thanks!

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13785

Extra the in your sentence:
This transformation is only valid if the we are loading -> This transformation is only valid if we are loading

[Conanap]

Included sext test case.

[amyk]

Could we also elaborate in the description on how we are utilizing the new load instructions for zero extend case but not the sign extend case?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13795

Over 80 characters?

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

5 ↗

(On Diff #281035)

This line actually is for -O0 and not a BE RUN line. I believe the most up to date version of the test includes the LE, BE and O0 line.

[Conanap]

Some updates on formatting and updated to match an updated test file.

[NeHuang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13799

please clang-format the changed files before updating the patch.

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

4 ↗

(On Diff #281974)

Why do we need different check prefixes for LE and BE? They produce the same asm. It seems we only need CHECK-O0.

[Conanap]

Clang formatted relevant lines, combined LE and BE tests as they produced the same ASM.

[amyk]

clang/test/CodeGen/builtins-ppc-p10vector.c

92

It would be good to be a little more specific with the CHECK lines.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13869

This line here and below looks like unintended changes from clang-format I am guessing? I am OK with them being removed during commit.

[amyk]

I realized I didn't put a comment on this earlier but this overall LGTM, but I think it would be good to see if @nemanjai has any additional comments on this patch.

[nemanjai]

LGTM aside from a couple of minor nits.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

13772

This line is redundant. Please remove it.

13791

The second condition was already checked on line 13604. You can omit it here.

clang/lib/Headers/altivec.h

 vec_xl_be(signed long long  __offset, unsigned __int128 *__ptr) {
   return vec_xl(__offset, __ptr);
 }
 #endif
 #else
   #define vec_xl_be vec_xl
 #endif
 
+#if defined(__POWER10_VECTOR) && defined(__VSX)
+
+/* vect_xl_sext */
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed char *__pointer) {
+  return (unaligned_vec_si128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed short *__pointer) {
+  return (unaligned_vec_si128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed int *__pointer) {
+  return (unaligned_vec_si128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed long long *__pointer) {
+  return (unaligned_vec_si128)*(__pointer + __offset);
+}
+
+/* vec_xl_zext */
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned char *__pointer) {
+  return (unaligned_vec_ui128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned short *__pointer) {
+  return (unaligned_vec_ui128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned int *__pointer) {
+  return (unaligned_vec_ui128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned long long *__pointer) {
+  return (unaligned_vec_ui128)*(__pointer + __offset);
+}
+
+#endif
+
 /* vec_xst */
 
 static inline __ATTRS_o_ai void vec_xst(vector signed char __vec,
                                         signed long long __offset,
                                         signed char *__ptr) {
   *(unaligned_vec_schar *)(__ptr + __offset) = __vec;
 }
 

 static __inline__ vector unsigned char __ATTRS_o_ai
 vec_clrr(vector unsigned char __a, unsigned int __n) {
 #ifdef __LITTLE_ENDIAN__
   return __builtin_altivec_vclrlb((vector signed char)__a, __n);
 #else
   return __builtin_altivec_vclrrb((vector signed char)__a, __n);
 #endif
 }
 #endif /* __POWER10_VECTOR__ */
 
 #undef __ATTRS_o_ai
 
 #endif /* __ALTIVEC_H */

clang/test/CodeGen/builtins-ppc-p10vector.c

 // REQUIRES: powerpc-registered-target
 // RUN: %clang_cc1 -target-feature +vsx -target-feature +altivec \
 // RUN:   -target-cpu pwr10 -triple powerpc64le-unknown-unknown -emit-llvm %s \
 // RUN:   -o - | FileCheck %s
 
 #include <altivec.h>
 
 vector signed char vsca;
 vector unsigned char vuca;
 vector unsigned short vusa;
 vector unsigned int vuia;
+signed char *ca;
+unsigned char *uca;
+
 vector unsigned long long vulla, vullb;
-unsigned int uia;
+unsigned int uia, *uiap;
+signed int *ia;
+signed short *sia;
+unsigned short *usia;
+signed long long *llia, llib;
+unsigned long long *ullia;
 
 vector unsigned long long test_vpdepd(void) {
   // CHECK: @llvm.ppc.altivec.vpdepd(<2 x i64>
   // CHECK-NEXT: ret <2 x i64>
   return vec_pdep(vulla, vullb);
 }
 
 vector unsigned long long test_vpextd(void) {

 
 vector unsigned char test_vec_clrr_uc(void) {
   // CHECK-BE: @llvm.ppc.altivec.vclrrb(<16 x i8>
   // CHECK-BE-NEXT: ret <16 x i8>
   // CHECK-LE: @llvm.ppc.altivec.vclrlb(<16 x i8>
   // CHECK-LE-NEXT: ret <16 x i8>
   return vec_clrr(vuca, uia);
 }
+
+vector signed __int128 test_vec_xl_sext_i8(void) {
+  // CHECK: load i8
+  // CHECK: sext i8
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llib, ca);
+}
+
+vector signed __int128 test_vec_xl_sext_i16(void) {
+  // CHECK: load i16
+  // CHECK: sext i16
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llib, sia);
+}
+
+vector signed __int128 test_vec_xl_sext_i32(void) {
+  // CHECK: load i32
+  // CHECK: sext i32
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llib, ia);
+}
+
+vector signed __int128 test_vec_xl_sext_i64(void) {
+  // CHECK: load i64
+  // CHECK: sext i64
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llib, llia);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i8(void) {
+  // CHECK: load i8
+  // CHECK: zext i8
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llib, uca);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i16(void) {
+  // CHECK: load i16
+  // CHECK: zext i16
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llib, usia);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i32(void) {
+  // CHECK: load i32
+  // CHECK: zext i32
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llib, uiap);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i64(void) {
+  // CHECK: load i64
+  // CHECK: zext i64
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llib, ullia);
+}

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
     /// Maps directly to an lxvd2x instruction that will be followed by
     /// an xxswapd.
     LXVD2X,
 
     /// VSRC, CHAIN = LOAD_VEC_BE CHAIN, Ptr - Occurs only for little endian.
     /// Maps directly to one of lxvd2x/lxvw4x/lxvh8x/lxvb16x depending on
     /// the vector type to load vector in big-endian element order.
     LOAD_VEC_BE,
 
     /// VSRC, CHAIN = LD_VSX_LH CHAIN, Ptr - This is a floating-point load of a
     /// v2f32 value into the lower half of a VSR register.
     LD_VSX_LH,
 
     /// VSRC, CHAIN = LD_SPLAT, CHAIN, Ptr - a splatting load memory
     /// instructions such as LXVDSX, LXVWSX.
     LD_SPLAT,
 
enum NodeType : unsigned {
     /// except they ensure that the compare input is zero-extended for
     /// sub-word versions because the atomic loads zero-extend.
     ATOMIC_CMP_SWAP_8,
     ATOMIC_CMP_SWAP_16,
 
     /// GPRC = TOC_ENTRY GA, TOC
     /// Loads the entry for GA from the TOC, where the TOC base is given by
     /// the last operand.
-    TOC_ENTRY
+    TOC_ENTRY,
+
+    /// LXVRZX - Load VSX Vector Rightmost and Zero Extend
+    /// This node represents v1i128 BUILD_VECTOR of a zero extending load
+    /// instruction from <byte, halfword, word, or doubleword> to i128.
+    /// Allows utilization of the Load VSX Vector Rightmost Instructions.
+    LXVRZX
   };
 
   } // end namespace PPCISD
 
   /// Define some predicates that are used for node matching.
   namespace PPC {
 
     /// isVPKUHUMShuffleMask - Return true if this is the shuffle mask for a

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::BUILD_SPE64:     return "PPCISD::BUILD_SPE64";
   case PPCISD::EXTRACT_SPE:     return "PPCISD::EXTRACT_SPE";
   case PPCISD::EXTSWSLI:        return "PPCISD::EXTSWSLI";
   case PPCISD::LD_VSX_LH:       return "PPCISD::LD_VSX_LH";
   case PPCISD::FP_EXTEND_HALF:  return "PPCISD::FP_EXTEND_HALF";
   case PPCISD::MAT_PCREL_ADDR:  return "PPCISD::MAT_PCREL_ADDR";
   case PPCISD::LD_SPLAT:        return "PPCISD::LD_SPLAT";
   case PPCISD::FNMSUB:          return "PPCISD::FNMSUB";
+  case PPCISD::LXVRZX:          return "PPCISD::LXVRZX";
   }
   return nullptr;
 }
 
 EVT PPCTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &C,
                                           EVT VT) const {
   if (!VT.isVector())
     return Subtarget.useCRBits() ? MVT::i1 : MVT::i32;
static SDValue combineBVOfVecSExt(SDNode *N, SelectionDAG &DAG) {
   if (Elems != CorrectElems) {
     return addShuffleForVecExtend(N, DAG, Input, Elems, CorrectElems);
   }
 
   // Regular lowering will catch cases where a shuffle is not needed.
   return SDValue();
 }
 
+// Look for the pattern of a load from a narrow width to i128, feeding
+// into a BUILD_VECTOR of v1i128. Replace this sequence with a PPCISD node
+// (LXVRZX). This node represents a zero extending load that will be matched
+// to the Load VSX Vector Rightmost instructions.
+static SDValue combineBVZEXTLOAD(SDNode *N, SelectionDAG &DAG) {
+  SDLoc dl(N);
+  
+  // This combine is only eligible for a BUILD_VECTOR of v1i128.
+  // Other return types are not valid for the LXVRZX replacement.
+  if (N->getValueType(0) != MVT::v1i128)
+    return SDValue();
+
+  SDValue Operand = N->getOperand(0);
+  // Proceed with the transformation if the operand to the BUILD_VECTOR
+  // is a load instruction.
+  if (Operand.getOpcode() != ISD::LOAD)
+    return SDValue();
+
+  LoadSDNode *LD = dyn_cast<LoadSDNode>(Operand);
+  EVT MemoryType = LD->getMemoryVT();
+
+  // This transformation is only valid if the we are loading either a byte,
+  // halfword, word, or doubleword.
+  bool ValidLDType = MemoryType == MVT::i8 || MemoryType == MVT::i16 ||
+                     MemoryType == MVT::i32 || MemoryType == MVT::i64;
+
+  // Ensure that the load from the narrow width is being zero extended to i128.
+  if ((LD->getValueType(0) != MVT::i128) ||
+      (LD->getExtensionType() != ISD::ZEXTLOAD) || (!ValidLDType))
+    return SDValue();
+
+  // The width of the narrow type becomes an operand of the LXVRZX node
+  // we are creating in order to pattern match to the appropriate instruction
+  // in the backend.
+  SDValue Width = DAG.getIntPtrConstant(MemoryType.getScalarSizeInBits(), dl);
+  SDValue LoadOps[] = {LD->getChain(), LD->getBasePtr(), Width};
+
+  return DAG.getMemIntrinsicNode(PPCISD::LXVRZX, dl,
+                                 DAG.getVTList(MVT::v1i128, MVT::Other),
+                                 LoadOps, MemoryType, LD->getMemOperand());
+}
+
 SDValue PPCTargetLowering::DAGCombineBuildVector(SDNode *N,
                                                  DAGCombinerInfo &DCI) const {
   assert(N->getOpcode() == ISD::BUILD_VECTOR &&
          "Should be called with a BUILD_VECTOR node");
 
   SelectionDAG &DAG = DCI.DAG;
   SDLoc dl(N);
 
SDValue PPCTargetLowering::DAGCombineBuildVector(SDNode *N,
   // input types (i.e. it can't handle things like v4i16) so do not run before
   // legalization.
   if (Subtarget.hasP9Altivec() && !DCI.isBeforeLegalize()) {
     Reduced = combineBVOfVecSExt(N, DAG);
     if (Reduced)
       return Reduced;
   }
 
+  // On Power10, the Load VSX Vector Rightmost instructions can be utilized
+  // if this is a BUILD_VECTOR of v1i128, and if the operand to the BUILD_VECTOR
+  // is a load from <valid narrow width> to i128.
+  if (Subtarget.isISA3_1()) {
+    SDValue BVOfZLoad = combineBVZEXTLOAD(N, DAG);
+    if (BVOfZLoad)
+      return BVOfZLoad;
+  }
 
   if (N->getValueType(0) != MVT::v2f64)
     return SDValue();
 
   // Looking for:
   // (build_vector ([su]int_to_fp (extractelt 0)), [su]int_to_fp (extractelt 1))
   if (FirstInput.getOpcode() != ISD::SINT_TO_FP &&
       FirstInput.getOpcode() != ISD::UINT_TO_FP)
     return SDValue();
   if (N->getOperand(1).getOpcode() != ISD::SINT_TO_FP &&
       N->getOperand(1).getOpcode() != ISD::UINT_TO_FP)
     return SDValue();
   if (FirstInput.getOpcode() != N->getOperand(1).getOpcode())
     return SDValue();
 
   SDValue Ext1 = FirstInput.getOperand(0);
   SDValue Ext2 = N->getOperand(1).getOperand(0);
   if(Ext1.getOpcode() != ISD::EXTRACT_VECTOR_ELT ||
      Ext2.getOpcode() != ISD::EXTRACT_VECTOR_ELT)
     return SDValue();
 
   ConstantSDNode *Ext1Op = dyn_cast<ConstantSDNode>(Ext1.getOperand(1));
   ConstantSDNode *Ext2Op = dyn_cast<ConstantSDNode>(Ext2.getOperand(1));
   if (!Ext1Op || !Ext2Op)
     return SDValue();
   if (Ext1.getOperand(0).getValueType() != MVT::v4i32 ||
       Ext1.getOperand(0) != Ext2.getOperand(0))

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

 // PC Relative flag (for instructions that use the address of the prefix for
 // address computations).
 class isPCRel { bit PCRel = 1; }
 
 // Top-level class for prefixed instructions.
 class PI<bits<6> pref, bits<6> opcode, dag OOL, dag IOL, string asmstr,
          InstrItinClass itin> : Instruction {
   field bits<64> Inst;
   field bits<64> SoftFail = 0;
   bit PCRel = 0; // Default value, set by isPCRel.
   let Size = 8;
 
   let Namespace = "PPC";
   let OutOperandList = OOL;
   let InOperandList = IOL;
   let AsmString = asmstr;
   let Itinerary = itin;
   let Inst{0-5} = pref;
   let Inst{32-37} = opcode;
 
   bits<1> PPC970_First = 0;
multiclass 8LS_DForm_R_SI34_XT6_RA5_p<bits<5> opcode, dag OOL, dag IOL,
   def pc : 8LS_DForm_R_SI34_XT6_RA5<opcode, OOL, PCRel_IOL,
                                     !strconcat(asmstr, ", 1"), itin, []>,
                                     isPCRel;
 }
 
 def PrefixInstrs : Predicate<"PPCSubTarget->hasPrefixInstrs()">;
 def IsISA3_1 : Predicate<"PPCSubTarget->isISA3_1()">;
 
+let mayLoad = 1, mayStore = 0, Predicates = [IsISA3_1] in {
+  // The XFormMemOp flag is set on the instruction format.
+  def LXVRBX : X_XT6_RA5_RB5<31, 13, "lxvrbx", vsrc, []>;
+  def LXVRHX : X_XT6_RA5_RB5<31, 45, "lxvrhx", vsrc, []>;
+  def LXVRWX : X_XT6_RA5_RB5<31, 77, "lxvrwx", vsrc, []>;
+  def LXVRDX : X_XT6_RA5_RB5<31, 109, "lxvrdx", vsrc, []>;
+}
+
+def SDT_PPCLXVRZX : SDTypeProfile<1, 2, [
+  SDTCisVT<0, v1i128>, SDTCisPtrTy<1>, SDTCisPtrTy<2>
+]>;
+
+def PPClxvrzx : SDNode<"PPCISD::LXVRZX", SDT_PPCLXVRZX,
+                    [SDNPHasChain, SDNPMayLoad]>;
+
 let Predicates = [PrefixInstrs] in {
   let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
     defm PADDI8 :
       MLS_DForm_R_SI34_RTA5_p<14, (outs g8rc:$RT), (ins g8rc:$RA, s34imm:$SI),
                               (ins immZero:$RA, s34imm:$SI),
                               "paddi $RT, $RA, $SI", IIC_LdStLFD>;
     let isReMaterializable = 1, isAsCheapAsAMove = 1, isMoveImm = 1 in {
       def PLI8 : MLS_DForm_SI34_RT5<14, (outs g8rc:$RT),
def : Pat<(PPCstore_scal_int_from_vsr
               (f64 (PPCcv_fp_to_uint_in_vsr f64:$src)),
               (PPCmatpcreladdr pcreladdr:$dst), 8),
             (PSTXSDpc (XSCVDPUXDS f64:$src), $dst, 0)>;
   def : Pat<(PPCstore_scal_int_from_vsr
               (f64 (PPCcv_fp_to_uint_in_vsr f128:$src)),
               (PPCmatpcreladdr pcreladdr:$dst), 8),
             (PSTXSDpc (COPY_TO_REGCLASS (XSCVQPUDZ f128:$src), VFRC), $dst, 0)>;
 
     // If the PPCmatpcreladdr node is not caught by any other pattern it should be
   // caught here and turned into a paddi instruction to materialize the address.
   def : Pat<(PPCmatpcreladdr pcreladdr:$addr), (PADDI8pc 0, $addr)>;
 }
 
 let Predicates = [IsISA3_1] in {
    def VPDEPD : VXForm_1<1485, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                          "vpdepd $vD, $vA, $vB", IIC_VecGeneral,
                          [(set v2i64:$vD,
let Predicates = [IsISA3_1] in {
   def : Pat<(v16i8 (int_ppc_vsx_xxgenpcvbm v16i8:$VRB, imm:$IMM)),
             (v16i8 (COPY_TO_REGCLASS (XXGENPCVBM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v8i16 (int_ppc_vsx_xxgenpcvhm v8i16:$VRB, imm:$IMM)),
             (v8i16 (COPY_TO_REGCLASS (XXGENPCVHM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v4i32 (int_ppc_vsx_xxgenpcvwm v4i32:$VRB, imm:$IMM)),
             (v4i32 (COPY_TO_REGCLASS (XXGENPCVWM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v2i64 (int_ppc_vsx_xxgenpcvdm v2i64:$VRB, imm:$IMM)),
             (v2i64 (COPY_TO_REGCLASS (XXGENPCVDM $VRB, imm:$IMM), VRRC))>;
+
+  // Utilize the appropriate Load VSX Vector Rightmost instruction depending
+  // on the width of PPClxvrzx.
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 8)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRBX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 16)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRHX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 32)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRWX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 64)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRDX xoaddr:$src), VRRC))>;
 }

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; These test cases tests that zero extending loads utilize the Load VSX Vector Rightmost
+
+; (lxvr[b|h|w|d]) instructions in Power10.
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:   -mcpu=pwr10 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
+; RUN:   FileCheck %s
+
+; CHECK: lxvrbx
+; Function Attrs: norecurse nounwind readonly
+define dso_local <1 x i128> @vec_xl_zext(i64 %__offset, i8* nocapture readonly %__pointer) local_unnamed_addr #0 {
+; CHECK-LABEL: vec_xl_zext:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lxvrbx v2, r4, r3
+; CHECK-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i8, i8* %__pointer, i64 %__offset
+  %0 = load i8, i8* %add.ptr, align 1
+  %conv = zext i8 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+; CHECK: lxvrhx
+; Function Attrs: norecurse nounwind readonly
+define dso_local <1 x i128> @vec_xl_zext_short(i64 %__offset, i16* nocapture readonly %__pointer) local_unnamed_addr #0 {
+; CHECK-LABEL: vec_xl_zext_short:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    sldi r3, r3, 1
+; CHECK-NEXT:    lxvrhx v2, r4, r3
+; CHECK-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i16, i16* %__pointer, i64 %__offset
+  %0 = load i16, i16* %add.ptr, align 2
+  %conv = zext i16 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+; CHECK: lxvrwx
+; Function Attrs: norecurse nounwind readonly
+define dso_local <1 x i128> @vec_xl_zext_word(i64 %__offset, i32* nocapture readonly %__pointer) local_unnamed_addr #0 {
+; CHECK-LABEL: vec_xl_zext_word:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    sldi r3, r3, 2
+; CHECK-NEXT:    lxvrwx v2, r4, r3
+; CHECK-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i32, i32* %__pointer, i64 %__offset
+  %0 = load i32, i32* %add.ptr, align 4
+  %conv = zext i32 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+; CHECK: lxvrdx
+; Function Attrs: norecurse nounwind readonly
+define dso_local <1 x i128> @vec_xl_zext_dw(i64 %__offset, i64* nocapture readonly %__pointer) local_unnamed_addr #0 {
+; CHECK-LABEL: vec_xl_zext_dw:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    sldi r3, r3, 3
+; CHECK-NEXT:    lxvrdx v2, r4, r3
+; CHECK-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i64, i64* %__pointer, i64 %__offset
+  %0 = load i64, i64* %add.ptr, align 8
+  %conv = zext i64 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+

llvm/test/MC/Disassembler/PowerPC/p10insts.txt

 # CHECK xxgenpcvdm 0, 1, 2
 0xf0 0x02 0x0f 0x6a
 
 # CHECK: vclrlb 1, 4, 3
 0x10 0x24 0x19 0x8d
 
 # CHECK: vclrrb 1, 4, 3
 0x10 0x24 0x19 0xcd
+
+# CHECK: lxvrbx 32, 1, 2
+0x7c 0x01 0x10 0x1b
+
+# CHECK: lxvrhx 33, 1, 2
+0x7c 0x21 0x10 0x5b
+
+# CHECK: lxvrdx 34, 1, 2
+0x7c 0x41 0x10 0xdb
+
+# CHECK: lxvrwx 35, 1, 2
+0x7c 0x61 0x10 0x9b

llvm/test/MC/PowerPC/p10.s

 # CHECK-LE: xxgenpcvdm 0, 1, 2                    # encoding: [0x6a,0x0f,0x02,0xf0]
             xxgenpcvdm 0, 1, 2
 # CHECK-BE: vclrlb 1, 4, 3                        # encoding: [0x10,0x24,0x19,0x8d]
 # CHECK-LE: vclrlb 1, 4, 3                        # encoding: [0x8d,0x19,0x24,0x10]
             vclrlb 1, 4, 3
 # CHECK-BE: vclrrb 1, 4, 3                        # encoding: [0x10,0x24,0x19,0xcd]
 # CHECK-LE: vclrrb 1, 4, 3                        # encoding: [0xcd,0x19,0x24,0x10]
             vclrrb 1, 4, 3
+# CHECK-BE: lxvrbx 32, 1, 2                       # encoding: [0x7c,0x01,0x10,0x1b]
+# CHECK-LE: lxvrbx 32, 1, 2                       # encoding: [0x1b,0x10,0x01,0x7c]
+            lxvrbx 32, 1, 2
+# CHECK-BE: lxvrhx 33, 1, 2                       # encoding: [0x7c,0x21,0x10,0x5b]
+# CHECK-LE: lxvrhx 33, 1, 2                       # encoding: [0x5b,0x10,0x21,0x7c]
+            lxvrhx 33, 1, 2
+# CHECK-BE: lxvrdx 34, 1, 2                       # encoding: [0x7c,0x41,0x10,0xdb]
+# CHECK-LE: lxvrdx 34, 1, 2                       # encoding: [0xdb,0x10,0x41,0x7c]
+            lxvrdx 34, 1, 2
+# CHECK-BE: lxvrwx 35, 1, 2                       # encoding: [0x7c,0x61,0x10,0x9b]
+# CHECK-LE: lxvrwx 35, 1, 2                       # encoding: [0x9b,0x10,0x61,0x7c]
+            lxvrwx 35, 1, 2