[PowerPC] Implement Load VSX Vector and Sign Extend and Zero Extend

[Conanap]

Summary

Implemented Builtins for Load VSX Vector and sign extend and zero extend along with appropriate test cases. The new instructions are used for ZEXT but SEXT will continue to use the previously generated ASM.

[amyk]

I have a few comments from the last time we looked at this together.
Just also FYI that the backend tests will be apart of the p10-vsx-builtins.ll file introduced in https://reviews.llvm.org/D82431.

clang/lib/Headers/altivec.h

16514

Nit: space after this comment.

16517

I realized that I think we are supposed to use unaligned_vec_si128 and unaligned_vec_ui128 instead of vector signed __int128 and vector unsigned __int128 when we return (for here, and other places).

16535

Nit: space after this comment.

16917

Remove unrelated whitespace change.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14169

Move this comment above the combine function since this comment was intended to describe the function.

14176

Can omit the { } since we just have a single return underneath.

14183

Can omit the { } since we just have a single return underneath.

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

1107

Can remove this comment.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

6 ↗

(On Diff #273159)

Please update this comment to be more descriptive. Maybe something like,

These test cases tests that zero extending loads utilize the Load VSX Vector Rightmost (lxvr[b|h|w|d]) instructions in Power10.

8 ↗

(On Diff #273159)

Please remove the comments/checks above the functions.

[Conanap]

Addressed Amy's comments regarding documentation of the changes.

[Conanap]

Fixed return signature for the open coded functions in altivec.h

[anil9]

clang/test/CodeGen/builtins-ppc-p10vector.c

17

nit: It seems that most pull requests follow an ordering like first signed declaration and then unsigned, declaration, this one follows too , except the above two lines. And should the above declarations of chars, be along with these lines ?

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

56 ↗

(On Diff #273421)

I am not too familiar with the builtins but I never saw a check outside of the two braces in the test cases before, is it not posible to include it inside the test cases ?

[Conanap]

Reordered some declarations in test cases and removed unecessary extra CHECKs.

[lei]

Please move encoding tests to ppc64-encoding-ISA31.[txt|s].

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14194

nit: check ValidLDType first:

if (! ValidLDType || (LD->getValueType(0) != MVT::i128) ||
    (LD->getExtensionType() != ISD::ZEXTLOAD) ||)

llvm/lib/Target/PowerPC/PPCISelLowering.h

547

nit: This shouldn't matter to code gen, but it would be nice to have this defined closer to other similar instructions. Maybe after LXVD2X?

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

480

nit: indentation

[Conanap]

Addressed Anil's comments with regards to the test cases

[Conanap]

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

56 ↗

(On Diff #273421)

these were extra checks that I should've removed; thanks for catching it

[Conanap]

Fixed some formatting stuff.

[Conanap]

Addressed Lei's comments

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.h

500

missing ,?

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

475

We should put all such def at the top of this file under:

// PowerPC specific type constraints.

479

Put these under the type constraints section and doc with:

// PowerPC specific DAG Nodes.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

3 ↗

(On Diff #275124)

remve extra line. Makes the comment harder to read.

10 ↗

(On Diff #275124)

You don't need the local_unnamed_addr #0

[Conanap]

Fixed a missing comma; ensured it builds.

[Conanap]

fixed a missing comma and ensure that it compiles

[Conanap]

Moved some code to the top of the file as per Lei's request

[Conanap]

Addressed Lei's comments

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14193

nit: don't need () aroud !ValidLDType

14199

I don't think we need to explicitly say this sine everything we do here is for pattern matching to appropriate instructions in the backend...

14200

Can just merge this into the next line and remove this tmp value.

llvm/test/CodeGen/PowerPC/p10-vsx-builtins.ll

6 ↗

(On Diff #275423)

Please add tests for BE.

[Conanap]

Now depends on D83364.

[Conanap]

Also removed unnecessary brackets and comments.

[amyk]

Please update this patch to remove the instruction defs and MC tests. Also, you can update the patch to put your backend llc tests in the file I've introduced in: https://reviews.llvm.org/D82467

clang/test/CodeGen/builtins-ppc-p10vector.c

14

unintended change?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14198

You did not assign anything here?

[Conanap]

Removed unecessary comments and unintended changes.

[Conanap]

Removed duplicate test code found in the instructions and MC Test implementation of VSX Vector store and load with sign extend or zero extend; re-added a test file that was omitted in the last diff update.

[amyk]

I think the patterns for the load instructions may have accidentally been deleted. Please add them back to the patch. Thank you.

[Conanap]

Restored accidentally deleted pattern, removed duplicate tests, moved new tests to another pre-existing file instead.

[stefanp]

The title of the patch mentions both zero extend and sign extend.
However, it seems that we only have instructions for the zero extend case. Is that right?
I see both types of tests in:
test/CodeGen/builtins-ppc-p10vector.c
But I only see codegen tests for the zreo extend version.
test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

I bring it up because I want to make sure that this is intentional.

clang/lib/Headers/altivec.h

16516

I'm a little confused about this.
Your function signature says we return vector unsigned __int128 but the return statement casts to unaligned_vec_si128. Is that how this is supposed to look?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14194

It looks like you are doing this for zero extend only.
What about ISD::EXTLOAD? In that case the upper bits are undefined anyway so we can just assume a zero extend right?

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

57

nit:
Was this line auto-added?
Usually this comment is added at the top of the file by the script. I don't think it is required here.

[NeHuang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14187

nit: if we are loading either a byte....

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

29

nit: one space is enough.

[Conanap]

Return signature fix, added recognition for ISD:EXTLOAD, some code clean up.

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14197

nit: indentation.

14200

nit: indentation

[Conanap]

Addressed formatting comments

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14187

I'm not too sure what you mean, would you be able to elaborate?
The comment is:

This transformation is only valid if the we are loading either a byte,
halfword, word, or doubleword.

Thanks!

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14187

Extra the in your sentence:
This transformation is only valid if the we are loading -> This transformation is only valid if we are loading

[Conanap]

Included sext test case.

[amyk]

Could we also elaborate in the description on how we are utilizing the new load instructions for zero extend case but not the sign extend case?

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14197

Over 80 characters?

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

5

This line actually is for -O0 and not a BE RUN line. I believe the most up to date version of the test includes the LE, BE and O0 line.

[Conanap]

Some updates on formatting and updated to match an updated test file.

[NeHuang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14201

please clang-format the changed files before updating the patch.

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

3–4

Why do we need different check prefixes for LE and BE? They produce the same asm. It seems we only need CHECK-O0.

[Conanap]

Clang formatted relevant lines, combined LE and BE tests as they produced the same ASM.

[amyk]

clang/test/CodeGen/builtins-ppc-p10vector.c

602

It would be good to be a little more specific with the CHECK lines.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14267

This line here and below looks like unintended changes from clang-format I am guessing? I am OK with them being removed during commit.

[amyk]

I realized I didn't put a comment on this earlier but this overall LGTM, but I think it would be good to see if @nemanjai has any additional comments on this patch.

[nemanjai]

LGTM aside from a couple of minor nits.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

14174

This line is redundant. Please remove it.

14193

The second condition was already checked on line 13604. You can omit it here.

clang/lib/Headers/altivec.h

 vec_xl_be(signed long long  __offset, unsigned __int128 *__ptr) {
   return vec_xl(__offset, __ptr);
 }
 #endif
 #else
   #define vec_xl_be vec_xl
 #endif
 
+#if defined(__POWER10_VECTOR__) && defined(__VSX__)
+
+/* vect_xl_sext */
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed char *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed short *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed int *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_sext(signed long long __offset, signed long long *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+/* vec_xl_zext */
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned char *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned short *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned int *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+static __inline__ vector unsigned __int128 __ATTRS_o_ai
+vec_xl_zext(signed long long __offset, unsigned long long *__pointer) {
+  return (vector unsigned __int128)*(__pointer + __offset);
+}
+
+#endif
+
 /* vec_xst */
 
 static inline __ATTRS_o_ai void vec_xst(vector signed char __vec,
                                         signed long long __offset,
                                         signed char *__ptr) {
   *(unaligned_vec_schar *)(__ptr + __offset) = __vec;
 }
 

 static __inline__ vector unsigned char __ATTRS_o_ai
 vec_clrr(vector unsigned char __a, unsigned int __n) {
 #ifdef __LITTLE_ENDIAN__
   return __builtin_altivec_vclrlb((vector signed char)__a, __n);
 #else
   return __builtin_altivec_vclrrb((vector signed char)__a, __n);
 #endif
 }
 
 /* vec_cntlzm */
 
 static __inline__ vector unsigned long long __ATTRS_o_ai
 vec_cntlzm(vector unsigned long long __a, vector unsigned long long __b) {
   return __builtin_altivec_vclzdm(__a, __b);
 }
 
 /* vec_cnttzm */

clang/test/CodeGen/builtins-ppc-p10vector.c

 // REQUIRES: powerpc-registered-target
 // RUN: %clang_cc1 -target-feature +vsx -target-feature +altivec \
 // RUN:   -target-cpu pwr10 -triple powerpc64le-unknown-unknown -emit-llvm %s \
 // RUN:   -o - | FileCheck %s
 
 // RUN: %clang_cc1 -target-feature +vsx -target-feature +altivec \
 // RUN:   -target-cpu pwr10 -triple powerpc64-unknown-unknown -emit-llvm %s \
 // RUN:   -o - | FileCheck %s -check-prefix=CHECK-BE
 
 // RUN: %clang_cc1 -target-feature +vsx -target-feature +altivec \
 // RUN:   -target-cpu pwr10 -triple powerpc64le-unknown-unknown -emit-llvm %s \
 // RUN:   -o - | FileCheck %s -check-prefix=CHECK-LE
 
 #include <altivec.h>
 
 vector signed char vsca, vscb;
 vector unsigned char vuca, vucb, vucc;
 vector signed short vssa, vssb;
 vector unsigned short vusa, vusb, vusc;
 vector signed int vsia, vsib;
 vector unsigned int vuia, vuib, vuic;
 vector signed long long vslla, vsllb;
 vector unsigned long long vulla, vullb, vullc;
 vector unsigned __int128 vui128a, vui128b, vui128c;
 vector float vfa, vfb;
 vector double vda, vdb;
-unsigned int uia, uib;
-unsigned char uca;
-unsigned short usa;
-unsigned long long ulla;
+signed int *iap;
+unsigned int uia, uib, *uiap;
+signed char *cap;
+unsigned char uca, *ucap;
+signed short *sap;
+unsigned short usa, *usap;
+signed long long *llap, llb;
+unsigned long long ulla, *ullap;
 
 vector unsigned long long test_vpdepd(void) {
   // CHECK: @llvm.ppc.altivec.vpdepd(<2 x i64>
   // CHECK-NEXT: ret <2 x i64>
   return vec_pdep(vulla, vullb);
 }
 
 vector unsigned long long test_vpextd(void) {
int test_vec_test_lsbb_all_ones(void) {
   return vec_test_lsbb_all_ones(vuca);
 }
 
 int test_vec_test_lsbb_all_zeros(void) {
   // CHECK: @llvm.ppc.vsx.xvtlsbb(<16 x i8> %{{.+}}, i32 0
   // CHECK-NEXT: ret i32
   return vec_test_lsbb_all_zeros(vuca);
 }
+
+vector signed __int128 test_vec_xl_sext_i8(void) {
+  // CHECK: load i8
+  // CHECK: sext i8
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llb, cap);
+}
+
+vector signed __int128 test_vec_xl_sext_i16(void) {
+  // CHECK: load i16
+  // CHECK: sext i16
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llb, sap);
+}
+
+vector signed __int128 test_vec_xl_sext_i32(void) {
+  // CHECK: load i32
+  // CHECK: sext i32
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llb, iap);
+}
+
+vector signed __int128 test_vec_xl_sext_i64(void) {
+  // CHECK: load i64
+  // CHECK: sext i64
+  // CHECK: ret <1 x i128>
+  return vec_xl_sext(llb, llap);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i8(void) {
+  // CHECK: load i8
+  // CHECK: zext i8
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llb, ucap);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i16(void) {
+  // CHECK: load i16
+  // CHECK: zext i16
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llb, usap);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i32(void) {
+  // CHECK: load i32
+  // CHECK: zext i32
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llb, uiap);
+}
+
+vector unsigned __int128 test_vec_xl_zext_i64(void) {
+  // CHECK: load i64
+  // CHECK: zext i64
+  // CHECK: ret <1 x i128>
+  return vec_xl_zext(llb, ullap);
+}
+

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
     /// followed by a byte-width for the store.
     STXSIX,
 
     /// VSRC, CHAIN = LXVD2X_LE CHAIN, Ptr - Occurs only for little endian.
     /// Maps directly to an lxvd2x instruction that will be followed by
     /// an xxswapd.
     LXVD2X,
 
+    /// LXVRZX - Load VSX Vector Rightmost and Zero Extend
+    /// This node represents v1i128 BUILD_VECTOR of a zero extending load
+    /// instruction from <byte, halfword, word, or doubleword> to i128.
+    /// Allows utilization of the Load VSX Vector Rightmost Instructions.
+    LXVRZX,
+
     /// VSRC, CHAIN = LOAD_VEC_BE CHAIN, Ptr - Occurs only for little endian.
     /// Maps directly to one of lxvd2x/lxvw4x/lxvh8x/lxvb16x depending on
     /// the vector type to load vector in big-endian element order.
     LOAD_VEC_BE,
 
     /// VSRC, CHAIN = LD_VSX_LH CHAIN, Ptr - This is a floating-point load of a
     /// v2f32 value into the lower half of a VSR register.
     LD_VSX_LH,
enum NodeType : unsigned {
     /// Loads the entry for GA from the TOC, where the TOC base is given by
     /// the last operand.
     TOC_ENTRY
   };
 
   } // end namespace PPCISD
 
   /// Define some predicates that are used for node matching.
   namespace PPC {
 
     /// isVPKUHUMShuffleMask - Return true if this is the shuffle mask for a
     /// VPKUHUM instruction.
     bool isVPKUHUMShuffleMask(ShuffleVectorSDNode *N, unsigned ShuffleKind,
                               SelectionDAG &DAG);
 
     /// isVPKUWUMShuffleMask - Return true if this is the shuffle mask for a
     /// VPKUWUM instruction.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::BUILD_SPE64:     return "PPCISD::BUILD_SPE64";
   case PPCISD::EXTRACT_SPE:     return "PPCISD::EXTRACT_SPE";
   case PPCISD::EXTSWSLI:        return "PPCISD::EXTSWSLI";
   case PPCISD::LD_VSX_LH:       return "PPCISD::LD_VSX_LH";
   case PPCISD::FP_EXTEND_HALF:  return "PPCISD::FP_EXTEND_HALF";
   case PPCISD::MAT_PCREL_ADDR:  return "PPCISD::MAT_PCREL_ADDR";
   case PPCISD::LD_SPLAT:        return "PPCISD::LD_SPLAT";
   case PPCISD::FNMSUB:          return "PPCISD::FNMSUB";
+  case PPCISD::LXVRZX:          return "PPCISD::LXVRZX";
   }
   return nullptr;
 }
 
 EVT PPCTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &C,
                                           EVT VT) const {
   if (!VT.isVector())
     return Subtarget.useCRBits() ? MVT::i1 : MVT::i32;
static SDValue combineBVOfVecSExt(SDNode *N, SelectionDAG &DAG) {
   if (Elems != CorrectElems) {
     return addShuffleForVecExtend(N, DAG, Input, Elems, CorrectElems);
   }
 
   // Regular lowering will catch cases where a shuffle is not needed.
   return SDValue();
 }
 
+// Look for the pattern of a load from a narrow width to i128, feeding
+// into a BUILD_VECTOR of v1i128. Replace this sequence with a PPCISD node
+// (LXVRZX). This node represents a zero extending load that will be matched
+// to the Load VSX Vector Rightmost instructions.
+static SDValue combineBVZEXTLOAD(SDNode *N, SelectionDAG &DAG) {
+  SDLoc dl(N);
+
+  // This combine is only eligible for a BUILD_VECTOR of v1i128.
+  // Other return types are not valid for the LXVRZX replacement.
+  if (N->getValueType(0) != MVT::v1i128)
+    return SDValue();
+
+  SDValue Operand = N->getOperand(0);
+  // Proceed with the transformation if the operand to the BUILD_VECTOR
+  // is a load instruction.
+  if (Operand.getOpcode() != ISD::LOAD)
+    return SDValue();
+
+  LoadSDNode *LD = dyn_cast<LoadSDNode>(Operand);
+  EVT MemoryType = LD->getMemoryVT();
+
+  // This transformation is only valid if the we are loading either a byte,
+  // halfword, word, or doubleword.
+  bool ValidLDType = MemoryType == MVT::i8 || MemoryType == MVT::i16 ||
+                     MemoryType == MVT::i32 || MemoryType == MVT::i64;
+
+  // Ensure that the load from the narrow width is being zero extended to i128.
+  if (!ValidLDType || (LD->getValueType(0) != MVT::i128) ||
+      (LD->getExtensionType() != ISD::ZEXTLOAD && LD->getExtensionType() != ISD::EXTLOAD))
+    return SDValue();
+
+ SDValue LoadOps[] = {LD->getChain(), LD->getBasePtr(), DAG.getIntPtrConstant(MemoryType.getScalarSizeInBits(), dl)};
+
+ return DAG.getMemIntrinsicNode(PPCISD::LXVRZX, dl,
+                                DAG.getVTList(MVT::v1i128, MVT::Other),
+                                LoadOps, MemoryType, LD->getMemOperand());
+}
+
 SDValue PPCTargetLowering::DAGCombineBuildVector(SDNode *N,
                                                  DAGCombinerInfo &DCI) const {
   assert(N->getOpcode() == ISD::BUILD_VECTOR &&
          "Should be called with a BUILD_VECTOR node");
 
   SelectionDAG &DAG = DCI.DAG;
   SDLoc dl(N);
 
SDValue PPCTargetLowering::DAGCombineBuildVector(SDNode *N,
   // input types (i.e. it can't handle things like v4i16) so do not run before
   // legalization.
   if (Subtarget.hasP9Altivec() && !DCI.isBeforeLegalize()) {
     Reduced = combineBVOfVecSExt(N, DAG);
     if (Reduced)
       return Reduced;
   }
 
+  // On Power10, the Load VSX Vector Rightmost instructions can be utilized
+  // if this is a BUILD_VECTOR of v1i128, and if the operand to the BUILD_VECTOR
+  // is a load from <valid narrow width> to i128.
+  if (Subtarget.isISA3_1()) {
+    SDValue BVOfZLoad = combineBVZEXTLOAD(N, DAG);
+    if (BVOfZLoad)
+      return BVOfZLoad;
+  }
 
   if (N->getValueType(0) != MVT::v2f64)
     return SDValue();
 
   // Looking for:
   // (build_vector ([su]int_to_fp (extractelt 0)), [su]int_to_fp (extractelt 1))
   if (FirstInput.getOpcode() != ISD::SINT_TO_FP &&
       FirstInput.getOpcode() != ISD::UINT_TO_FP)
     return SDValue();
   if (N->getOperand(1).getOpcode() != ISD::SINT_TO_FP &&
       N->getOperand(1).getOpcode() != ISD::UINT_TO_FP)
     return SDValue();
   if (FirstInput.getOpcode() != N->getOperand(1).getOpcode())
     return SDValue();
 
   SDValue Ext1 = FirstInput.getOperand(0);
   SDValue Ext2 = N->getOperand(1).getOperand(0);
   if(Ext1.getOpcode() != ISD::EXTRACT_VECTOR_ELT ||
      Ext2.getOpcode() != ISD::EXTRACT_VECTOR_ELT)
     return SDValue();
 
   ConstantSDNode *Ext1Op = dyn_cast<ConstantSDNode>(Ext1.getOperand(1));
   ConstantSDNode *Ext2Op = dyn_cast<ConstantSDNode>(Ext2.getOperand(1));
   if (!Ext1Op || !Ext2Op)
     return SDValue();
   if (Ext1.getOperand(0).getValueType() != MVT::v4i32 ||
       Ext1.getOperand(0) != Ext2.getOperand(0))

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

 def PPCxxsplti32dx : SDNode<"PPCISD::XXSPLTI32DX", SDT_PPCSplat32, []>;
 
 //===----------------------------------------------------------------------===//
 
 // PC Relative flag (for instructions that use the address of the prefix for
 // address computations).
 class isPCRel { bit PCRel = 1; }
 
+// PowerPC specific type constraints.
+def SDT_PPCLXVRZX : SDTypeProfile<1, 2, [
+  SDTCisVT<0, v1i128>, SDTCisPtrTy<1>, SDTCisPtrTy<2>
+]>;
+
+// PPC Specific DAG Nodes.
+def PPClxvrzx : SDNode<"PPCISD::LXVRZX", SDT_PPCLXVRZX,
+                       [SDNPHasChain, SDNPMayLoad]>;
+
 // Top-level class for prefixed instructions.
 class PI<bits<6> pref, bits<6> opcode, dag OOL, dag IOL, string asmstr,
          InstrItinClass itin> : Instruction {
   field bits<64> Inst;
   field bits<64> SoftFail = 0;
   bit PCRel = 0; // Default value, set by isPCRel.
   let Size = 8;
 

 let Predicates = [PrefixInstrs] in {
   let Interpretation64Bit = 1, isCodeGenOnly = 1 in {
     defm PADDI8 :
       MLS_DForm_R_SI34_RTA5_p<14, (outs g8rc:$RT), (ins g8rc:$RA, s34imm:$SI),
                               (ins immZero:$RA, s34imm:$SI),
                               "paddi $RT, $RA, $SI", IIC_LdStLFD>;
     let isReMaterializable = 1, isAsCheapAsAMove = 1, isMoveImm = 1 in {
       def PLI8 : MLS_DForm_SI34_RT5<14, (outs g8rc:$RT),
                                     (ins s34imm:$SI),
                                     "pli $RT, $SI", IIC_IntSimple, []>;
     }
   }
   defm PADDI :
     MLS_DForm_R_SI34_RTA5_p<14, (outs gprc:$RT), (ins gprc:$RA, s34imm:$SI),
                             (ins immZero:$RA, s34imm:$SI),
                             "paddi $RT, $RA, $SI", IIC_LdStLFD>;
   let isReMaterializable = 1, isAsCheapAsAMove = 1, isMoveImm = 1 in {
     def PLI : MLS_DForm_SI34_RT5<14, (outs gprc:$RT),
                                  (ins s34imm:$SI),
                                  "pli $RT, $SI", IIC_IntSimple, []>;
   }
 
def : Pat<(v16i8 (int_ppc_vsx_xxgenpcvbm v16i8:$VRB, imm:$IMM)),
             (v16i8 (COPY_TO_REGCLASS (XXGENPCVBM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v8i16 (int_ppc_vsx_xxgenpcvhm v8i16:$VRB, imm:$IMM)),
             (v8i16 (COPY_TO_REGCLASS (XXGENPCVHM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v4i32 (int_ppc_vsx_xxgenpcvwm v4i32:$VRB, imm:$IMM)),
             (v4i32 (COPY_TO_REGCLASS (XXGENPCVWM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(v2i64 (int_ppc_vsx_xxgenpcvdm v2i64:$VRB, imm:$IMM)),
             (v2i64 (COPY_TO_REGCLASS (XXGENPCVDM $VRB, imm:$IMM), VRRC))>;
   def : Pat<(i32 (int_ppc_vsx_xvtlsbb v16i8:$XB, 1)),
             (EXTRACT_SUBREG (XVTLSBB (COPY_TO_REGCLASS $XB, VSRC)), sub_lt)>;
   def : Pat<(i32 (int_ppc_vsx_xvtlsbb v16i8:$XB, 0)),
             (EXTRACT_SUBREG (XVTLSBB (COPY_TO_REGCLASS $XB, VSRC)), sub_eq)>;
+
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 8)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRBX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 16)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRHX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 32)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRWX xoaddr:$src), VRRC))>;
+  def : Pat <(v1i128 (PPClxvrzx xoaddr:$src, 64)),
+             (v1i128 (COPY_TO_REGCLASS (LXVRDX xoaddr:$src), VRRC))>;
 }
 
 let AddedComplexity = 400, Predicates = [PrefixInstrs] in {
  def : Pat<(v4i32 (build_vector i32immNonAllOneNonZero:$A,
                                 i32immNonAllOneNonZero:$A,
                                 i32immNonAllOneNonZero:$A,
                                 i32immNonAllOneNonZero:$A)),
            (v4i32 (XXSPLTIW imm:$A))>;

llvm/test/CodeGen/PowerPC/builtins-ppc-p10vsx.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
 ; RUN:   -mcpu=pwr10 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
-; RUN:   FileCheck %s
-; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -O0 \
+; RUN:   FileCheck %s --check-prefix=CHECK-LE
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu -O0 \
 ; RUN:   -mcpu=pwr10 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
-; RUN:   FileCheck %s
+; RUN:   FileCheck %s --check-prefix=CHECK-BE
 
 ; These test cases aims to test the builtins for the Power10 VSX vector
 ; instructions introduced in ISA 3.1.
 
 declare i32 @llvm.ppc.vsx.xvtlsbb(<16 x i8>, i32)
 
 define signext i32 @test_vec_test_lsbb_all_ones(<16 x i8> %vuca) {
-; CHECK-LABEL: test_vec_test_lsbb_all_ones:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    xvtlsbb cr0, v2
-; CHECK-NEXT:    mfocrf r3, 128
-; CHECK-NEXT:    srwi r3, r3, 31
-; CHECK-NEXT:    extsw r3, r3
-; CHECK-NEXT:    blr
+; CHECK-LE-LABEL: test_vec_test_lsbb_all_ones:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xvtlsbb cr0, v2
+; CHECK-LE-NEXT:    mfocrf r3, 128
+; CHECK-LE-NEXT:    srwi r3, r3, 31
+; CHECK-LE-NEXT:    extsw r3, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_vec_test_lsbb_all_ones:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xvtlsbb cr0, v2
+; CHECK-BE-NEXT:    mfocrf r3, 128
+; CHECK-BE-NEXT:    srwi r3, r3, 31
+; CHECK-BE-NEXT:    extsw r3, r3
+; CHECK-BE-NEXT:    blr
 entry:
   %0 = tail call i32 @llvm.ppc.vsx.xvtlsbb(<16 x i8> %vuca, i32 1)
   ret i32 %0
 }
 
 define signext i32 @test_vec_test_lsbb_all_zeros(<16 x i8> %vuca) {
-; CHECK-LABEL: test_vec_test_lsbb_all_zeros:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    xvtlsbb cr0, v2
-; CHECK-NEXT:    mfocrf r3, 128
-; CHECK-NEXT:    rlwinm r3, r3, 3, 31, 31
-; CHECK-NEXT:    extsw r3, r3
-; CHECK-NEXT:    blr
+; CHECK-LE-LABEL: test_vec_test_lsbb_all_zeros:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xvtlsbb cr0, v2
+; CHECK-LE-NEXT:    mfocrf r3, 128
+; CHECK-LE-NEXT:    rlwinm r3, r3, 3, 31, 31
+; CHECK-LE-NEXT:    extsw r3, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_vec_test_lsbb_all_zeros:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xvtlsbb cr0, v2
+; CHECK-BE-NEXT:    mfocrf r3, 128
+; CHECK-BE-NEXT:    rlwinm r3, r3, 3, 31, 31
+; CHECK-BE-NEXT:    extsw r3, r3
+; CHECK-BE-NEXT:    blr
 entry:
   %0 = tail call i32 @llvm.ppc.vsx.xvtlsbb(<16 x i8> %vuca, i32 0)
   ret i32 %0
 }
+
+; These test cases tests that zero extending loads utilize the Load VSX Vector Rightmost
+; (lxvr[b|h|w|d]x) instructions in Power10.
+
+define dso_local <1 x i128> @vec_xl_zext(i64 %__offset, i8* nocapture readonly %__pointer) {
+; CHECK-LE-LABEL: vec_xl_zext:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    lxvrbx v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_zext:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    lxvrbx vs0, r4, r3
+; CHECK-BE-NEXT:    xxlor v2, vs0, vs0
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i8, i8* %__pointer, i64 %__offset
+  %0 = load i8, i8* %add.ptr, align 1
+  %conv = zext i8 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_zext_short(i64 %__offset, i16* nocapture readonly %__pointer) {
+; CHECK-LE-LABEL: vec_xl_zext_short:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 1
+; CHECK-LE-NEXT:    lxvrhx v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_zext_short:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 1
+; CHECK-BE-NEXT:    lxvrhx vs0, r4, r3
+; CHECK-BE-NEXT:    xxlor v2, vs0, vs0
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i16, i16* %__pointer, i64 %__offset
+  %0 = load i16, i16* %add.ptr, align 2
+  %conv = zext i16 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_zext_word(i64 %__offset, i32* nocapture readonly %__pointer) {
+; CHECK-LE-LABEL: vec_xl_zext_word:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 2
+; CHECK-LE-NEXT:    lxvrwx v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_zext_word:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 2
+; CHECK-BE-NEXT:    lxvrwx vs0, r4, r3
+; CHECK-BE-NEXT:    xxlor v2, vs0, vs0
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i32, i32* %__pointer, i64 %__offset
+  %0 = load i32, i32* %add.ptr, align 4
+  %conv = zext i32 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_zext_dw(i64 %__offset, i64* nocapture readonly %__pointer) {
+; CHECK-LE-LABEL: vec_xl_zext_dw:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 3
+; CHECK-LE-NEXT:    lxvrdx v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_zext_dw:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 3
+; CHECK-BE-NEXT:    lxvrdx vs0, r4, r3
+; CHECK-BE-NEXT:    xxlor v2, vs0, vs0
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i64, i64* %__pointer, i64 %__offset
+  %0 = load i64, i64* %add.ptr, align 8
+  %conv = zext i64 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_sext_b(i64 %offset, i8* %p) {
+; CHECK-LE-LABEL: vec_xl_sext_b:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    lbzx r3, r4, r3
+; CHECK-LE-NEXT:    extsb r3, r3
+; CHECK-LE-NEXT:    sradi r4, r3, 63
+; CHECK-LE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_sext_b:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    lbzx r3, r4, r3
+; CHECK-BE-NEXT:    extsb r3, r3
+; CHECK-BE-NEXT:    sradi r4, r3, 63
+; CHECK-BE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i8, i8* %p, i64 %offset
+  %0 = load i8, i8* %add.ptr, align 1
+  %conv = sext i8 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_sext_h(i64 %offset, i16* %p) {
+; CHECK-LE-LABEL: vec_xl_sext_h:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 1
+; CHECK-LE-NEXT:    lhax r3, r4, r3
+; CHECK-LE-NEXT:    sradi r4, r3, 63
+; CHECK-LE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_sext_h:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 1
+; CHECK-BE-NEXT:    lhax r3, r4, r3
+; CHECK-BE-NEXT:    sradi r4, r3, 63
+; CHECK-BE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i16, i16* %p, i64 %offset
+  %0 = load i16, i16* %add.ptr, align 2
+  %conv = sext i16 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_sext_w(i64 %offset, i32* %p) {
+; CHECK-LE-LABEL: vec_xl_sext_w:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 2
+; CHECK-LE-NEXT:    lwax r3, r4, r3
+; CHECK-LE-NEXT:    sradi r4, r3, 63
+; CHECK-LE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_sext_w:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 2
+; CHECK-BE-NEXT:    lwax r3, r4, r3
+; CHECK-BE-NEXT:    sradi r4, r3, 63
+; CHECK-BE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i32, i32* %p, i64 %offset
+  %0 = load i32, i32* %add.ptr, align 4
+  %conv = sext i32 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}
+
+define dso_local <1 x i128> @vec_xl_sext_d(i64 %offset, i64* %p) {
+; CHECK-LE-LABEL: vec_xl_sext_d:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    sldi r3, r3, 3
+; CHECK-LE-NEXT:    ldx r3, r4, r3
+; CHECK-LE-NEXT:    sradi r4, r3, 63
+; CHECK-LE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: vec_xl_sext_d:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    sldi r3, r3, 3
+; CHECK-BE-NEXT:    ldx r3, r4, r3
+; CHECK-BE-NEXT:    sradi r4, r3, 63
+; CHECK-BE-NEXT:    mtvsrdd v2, r4, r3
+; CHECK-BE-NEXT:    blr
+entry:
+  %add.ptr = getelementptr inbounds i64, i64* %p, i64 %offset
+  %0 = load i64, i64* %add.ptr, align 8
+  %conv = sext i64 %0 to i128
+  %splat.splatinsert = insertelement <1 x i128> undef, i128 %conv, i32 0
+  ret <1 x i128> %splat.splatinsert
+}