[PowerPC][Power10] Exploit the xxsplti32dx instruction when lowering VECTOR_SHUFFLE.

[amyk]

Summary

This patch aims to exploit the xxsplti32dx XT, IX, IMM32 instruction when lowering VECTOR_SHUFFLEs.

We implement lowerToXXSPLTI32DX when lowering vector shuffles to check if:

• Element size is 4 bytes
• The RHS is a constant vector (and constant splat of 4-bytes)
• The shuffle mask is a suitable mask for the XXSPLTI32DX instruction where it is one of the 32 masks:

<0, 4-7, 2, 4-7>
<4-7, 1, 4-7, 3>

[lei]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9792

dl->DL

9825

no need for the tmp SplatSize

if ((SplatBitSize / 8) > 4)

9839

There see to be alot of extra, unnecessary () here... since all these are && I think alot of these can be removed.

9841

same.

9844

I think you are missing:

else
  return SDValue();

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

65

nit: indentation?

[lei]

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

62

nit: Maybe

PowerPC ISA 3.1 specific type constraints.

65

nvm. This is how it's been done else where..

67

nit:

// ISA 3.1 specific PPCISD nodes.

[amyk]

Address the following review comments:

• formatting in PPCInstrPrefix.td
• Remove extra ( )
• Indentation in PPCISelLowering
• Remove unnecessary variables
• Add extra case to return SDValue()

[amyk]

Address comment of adding more specific ISA 3.1 comments to PPCInstrPrefix.td.

[lei]

LGTM

[anil9]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9827

semantics the -> semantics of the

llvm/lib/Target/PowerPC/PPCISelLowering.h

105

nit : the other ones seem to have a extra line with ///

1278

nit :
/// otherwise return the default SDValue. ???

[nemanjai]

Pretty close to the way it should be but the indices do need to flip so I have to request changes.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9798

This comment is incorrect. The canonical type is v16i8.

9801

Is there a reason we don't just define these this way above?
i.e. SDValue LHS = peekThroughBitcasts(SVN->getOperand(0));

9817

You do not use IsBVNConstSplat anywhere except in the condition. You can just put the call in the condition
i.e. if (!BVN->isConstantSplat(...) || SplatBitSize > 32)

9824

This can be folded into the above condition. I think it is reasonable to expect the reader to understand that 32 bits is 4 bytes (on PPC) so we don't need to divide by 8.

9827

// Check that the shuffle mask matches the semantics of XXSPLTI32DX.
// The instruction splats a constant C into two words of the source vector
// producing { C, Unchanged, C, Unchanged } or { Unchanged, C, Unchanged, C }.
// Thus we check that the shuffle mask is the equivalent  of
// <0, [4-7], 2, [4-7]> or <[4-7], 1, [4-7], 3> respectively.
// Note: the check above of isNByteElemShuffleMask() ensures that the bytes
// within each word are consecutive, so we only need to check the first byte.

9839

We are after type legalization here, can you please use legal types (i.e. no MVT::i1).

9839

This is backwards. On LE, the rightmost element is element zero. In this path, the constant goes into the most significant word of each doubleword. So your Index needs to flip in both places.

llvm/lib/Target/PowerPC/PPCISelLowering.h

1278

All lowering and combine functions return a default constructed SDValue when unsuccessful. There is no reason to call that out specifically.

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

754

Can we please use i32 rather than i1 as the latter could lead to issues (with using CRBIT registers which we really don't want to do).

llvm/test/CodeGen/PowerPC/p10-splatImm32.ll

22

The result of this shuffle is:
{ a[0], 566, a[2], 566 }
Which produces a vector register:

LE: [ 566  | a[2] | 566  | a[0] ] => xxsplti32dx vs34, 0, 566
BE: [ a[0] | 566  | a[2] | 566  ] => xxsplti32dx vs34, 1, 566

So it is backwards - similarly to all the test cases.

[amyk]

Addressed comments from Nemanja:

• various variable changes
• update comment/documentation
• corrected the index for the xxsplti32dx instruction for LE/BE
• Updated the instruction to use u1imm instead of i1imm so the index in assembly can be 0/1, and this allows us the index to be i32 in the pattern.

[nemanjai]

The remaining updates are straightforward so feel free to address my comments on the commit. LGTM otherwise.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

9800

Forgot to remove these?

9840

If the splat is smaller than 32 bits, you need to replicate it.

// If the splat is narrower than 32-bits, we need to get the 32-bit value
// for XXSPLTI32DX.
unsigned SplatVal = APSplatValue.getZExtValue();
for (; SplatBitSize < 32; SplatBitSize <<= 1)
  SplatVal |= (SplatVal << SplatBitSize);

and then use SplatVal below when creating the XXSPLTI32DX node.
We also need a test case for this. Something like:

vector int test(vector int a) {
  unsigned Val = 0xABABABAB;
  a[0] = Val;
  a[2] = Val;
  return a;
}

This should give you a SplatBitSize == 8 and APSplatValue == 0xAB.

[amyk]

Address review comments from Nemanja:

• update comments, variables
• consider the case when the splat is smaller than 32-bits (and add associated test case)

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
     ///
     XXSPLT,
 
     /// XXSPLTI_SP_TO_DP - The PPC VSX splat instructions for immediates for
     /// converting immediate single precision numbers to double precision
     /// vector or scalar.
     XXSPLTI_SP_TO_DP,
 
+    /// XXSPLTI32DX - The PPC XXSPLTI32DX instruction.
+    XXSPLTI32DX,
+
     /// VECINSERT - The PPC vector insert instruction
     ///
     VECINSERT,
 
     /// VECSHL - The PPC vector shift left instruction
     ///
     VECSHL,
 
private:
     /// from one vector into the other.
     SDValue lowerToVINSERTH(ShuffleVectorSDNode *N, SelectionDAG &DAG) const;
 
     /// lowerToVINSERTB - Return the SDValue if this VECTOR_SHUFFLE can be
     /// handled by the VINSERTB instruction introduced in ISA 3.0. This is
     /// essentially v16i8 vector version of VINSERTH.
     SDValue lowerToVINSERTB(ShuffleVectorSDNode *N, SelectionDAG &DAG) const;
 
+    /// lowerToXXSPLTI32DX - Return the SDValue if this VECTOR_SHUFFLE can be
+    /// handled by the XXSPLTI32DX instruction introduced in ISA 3.1.
+    SDValue lowerToXXSPLTI32DX(ShuffleVectorSDNode *N, SelectionDAG &DAG) const;
+
     // Return whether the call instruction can potentially be optimized to a
     // tail call. This will cause the optimizers to attempt to move, or
     // duplicate return instructions to help enable tail call optimizations.
     bool mayBeEmittedAsTailCall(const CallInst *CI) const override;
     bool hasBitPreservingFPLogic(EVT VT) const override;
     bool isMaskAndCmp0FoldingBeneficial(const Instruction &AndI) const override;
   }; // end class PPCTargetLowering
 

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

case PPCISD::FP_TO_SINT_IN_VSR:
                                 return "PPCISD::FP_TO_SINT_IN_VSR";
   case PPCISD::FRE:             return "PPCISD::FRE";
   case PPCISD::FRSQRTE:         return "PPCISD::FRSQRTE";
   case PPCISD::STFIWX:          return "PPCISD::STFIWX";
   case PPCISD::VPERM:           return "PPCISD::VPERM";
   case PPCISD::XXSPLT:          return "PPCISD::XXSPLT";
   case PPCISD::XXSPLTI_SP_TO_DP:
     return "PPCISD::XXSPLTI_SP_TO_DP";
+  case PPCISD::XXSPLTI32DX:
+    return "PPCISD::XXSPLTI32DX";
   case PPCISD::VECINSERT:       return "PPCISD::VECINSERT";
   case PPCISD::XXPERMDI:        return "PPCISD::XXPERMDI";
   case PPCISD::VECSHL:          return "PPCISD::VECSHL";
   case PPCISD::CMPB:            return "PPCISD::CMPB";
   case PPCISD::Hi:              return "PPCISD::Hi";
   case PPCISD::Lo:              return "PPCISD::Lo";
   case PPCISD::TOC_ENTRY:       return "PPCISD::TOC_ENTRY";
   case PPCISD::ATOMIC_CMP_SWAP_8: return "PPCISD::ATOMIC_CMP_SWAP_8";
if (ShiftElts) {
     return DAG.getNode(ISD::BITCAST, dl, MVT::v16i8, Ins);
   }
   SDValue Conv2 = DAG.getNode(ISD::BITCAST, dl, MVT::v8i16, V2);
   SDValue Ins = DAG.getNode(PPCISD::VECINSERT, dl, MVT::v8i16, Conv1, Conv2,
                             DAG.getConstant(InsertAtByte, dl, MVT::i32));
   return DAG.getNode(ISD::BITCAST, dl, MVT::v16i8, Ins);
 }
 
+/// lowerToXXSPLTI32DX - Return the SDValue if this VECTOR_SHUFFLE can be
+/// handled by the XXSPLTI32DX instruction introduced in ISA 3.1, otherwise
+/// return the default SDValue.
+SDValue PPCTargetLowering::lowerToXXSPLTI32DX(ShuffleVectorSDNode *SVN,
+                                              SelectionDAG &DAG) const {
+  SDValue LHS = SVN->getOperand(0);
+  SDValue RHS = SVN->getOperand(1);
+  auto ShuffleMask = SVN->getMask();
+  SDValue VecShuffle(SVN, 0);
+  SDLoc DL(SVN);
+
+  // Check that we have a four byte shuffle.
+  if (!isNByteElemShuffleMask(SVN, 4, 1))
+    return SDValue();
+
+  // The LHS and RHS may be bitcasts to v8i16 as we canonicalize shuffles
+  // to v8i16. Peek through the bitcasts to get the actual operands,
+  // and canonicalize the RHS being a BUILD_VECTOR when lowering to xxsplti32dx.
+  LHS = peekThroughBitcasts(LHS);
+  RHS = peekThroughBitcasts(RHS);
+  if (RHS->getOpcode() != ISD::BUILD_VECTOR) {
+    std::swap(LHS, RHS);
+    VecShuffle = DAG.getCommutedVectorShuffle(*SVN);
+    ShuffleMask = cast<ShuffleVectorSDNode>(VecShuffle)->getMask();
+  }
+
+  // Ensure that the RHS is a vector of constants.
+  BuildVectorSDNode *BVN = dyn_cast<BuildVectorSDNode>(RHS.getNode());
+  if (!BVN)
+    return SDValue();
+
+  APInt APSplatValue, APSplatUndef;
+  unsigned SplatBitSize;
+  bool HasAnyUndefs;
+  bool IsBVNConstSplat =
+      BVN->isConstantSplat(APSplatValue, APSplatUndef, SplatBitSize,
+                           HasAnyUndefs, 0, !Subtarget.isLittleEndian());
+  if (!IsBVNConstSplat)
+    return SDValue();
+
+  // Check if RHS is a splat of 4-bytes (or smaller).
+  if ((SplatBitSize / 8) > 4)
+    return SDValue();
+
+  // Check that the shuffle mask matches the semantics the XXSPLTI32DX.
+  // XXSPLTI32DX can insert 4 byte chunks from the constant splat C into:
+  // { C, Unchanged, C, Unchanged } or { Unchanged, C, Unchanged, C }
+  // Thus we check if the shuffle is any of the following 32 masks
+  // (represented as a 4-byte element shuffle in this comment):
+  // Case 1: <0, 4-7, 2, 4-7> and
+  // Case 2: <4-7, 1, 4-7, 3>
+  SDValue Index;
+  bool IsLE = Subtarget.isLittleEndian();
+  if ((ShuffleMask[0] == 0 && ShuffleMask[8] == 8) &&
+      (ShuffleMask[4] % 4 == 0 && ShuffleMask[12] % 4 == 0 &&
+       ShuffleMask[4] > 15 && ShuffleMask[12] > 15)) // Case 1.
+    Index = DAG.getTargetConstant(IsLE ? 1 : 0, DL, MVT::i1);
+  else if ((ShuffleMask[4] == 4 && ShuffleMask[12] == 12) && // Case 2.
+           (ShuffleMask[0] % 4 == 0 && ShuffleMask[8] % 4 == 0 &&
+            ShuffleMask[0] > 15 && ShuffleMask[8] > 15))
+    Index = DAG.getTargetConstant(IsLE ? 0 : 1, DL, MVT::i1);
+  else
+    return SDValue();
+
+  SDValue SplatNode = DAG.getNode(
+      PPCISD::XXSPLTI32DX, DL, MVT::v2i64, DAG.getBitcast(MVT::v2i64, LHS),
+      Index, DAG.getTargetConstant(APSplatValue.getZExtValue(), DL, MVT::i32));
+  return DAG.getNode(ISD::BITCAST, DL, MVT::v16i8, SplatNode);
+}
+
 /// LowerROTL - Custom lowering for ROTL(v1i128) to vector_shuffle(v16i8).
 /// We lower ROTL(v1i128) to vector_shuffle(v16i8) only if shift amount is
 /// a multiple of 8. Otherwise convert it to a scalar rotation(i128)
 /// i.e (or (shl x, C1), (srl x, 128-C1)).
 SDValue PPCTargetLowering::LowerROTL(SDValue Op, SelectionDAG &DAG) const {
   assert(Op.getOpcode() == ISD::ROTL && "Should only be called for ISD::ROTL");
   assert(Op.getValueType() == MVT::v1i128 &&
          "Only set v1i128 as custom, other type shouldn't reach here!");
if (ShiftElts) {
                                 DAG.getConstant(InsertAtByte, dl, MVT::i32));
       return DAG.getNode(ISD::BITCAST, dl, MVT::v16i8, Ins);
     }
     SDValue Ins = DAG.getNode(PPCISD::VECINSERT, dl, MVT::v4i32, Conv1, Conv2,
                               DAG.getConstant(InsertAtByte, dl, MVT::i32));
     return DAG.getNode(ISD::BITCAST, dl, MVT::v16i8, Ins);
   }
 
+  if (Subtarget.hasPrefixInstrs()) {
+    SDValue SplatInsertNode;
+    if ((SplatInsertNode = lowerToXXSPLTI32DX(SVOp, DAG)))
+      return SplatInsertNode;
+  }
+
   if (Subtarget.hasP9Altivec()) {
     SDValue NewISDNode;
     if ((NewISDNode = lowerToVINSERTH(SVOp, DAG)))
       return NewISDNode;
 
     if ((NewISDNode = lowerToVINSERTB(SVOp, DAG)))
       return NewISDNode;
   }

llvm/lib/Target/PowerPC/PPCInstrPrefix.td

+//===----------------------------------------------------------------------===//
+// PowerPC specific type constraints.
+//
+
+def SDT_PPCSplat32 : SDTypeProfile<1, 3, [ SDTCisVT<0, v2i64>,
+  SDTCisVec<1>, SDTCisInt<2>, SDTCisInt<3>
+]>;
+
+//===----------------------------------------------------------------------===//
+// P10 specific PPCISD nodes.
+//
+
+def PPCxxsplti32dx : SDNode<"PPCISD::XXSPLTI32DX", SDT_PPCSplat32, []>;
+
+//===----------------------------------------------------------------------===//
+
 // PC Relative flag (for instructions that use the address of the prefix for
 // address computations).
 class isPCRel { bit PCRel = 1; }
 
 // Top-level class for prefixed instructions.
 class PI<bits<6> pref, bits<6> opcode, dag OOL, dag IOL, string asmstr,
          InstrItinClass itin> : Instruction {
   field bits<64> Inst;
class PI<bits<6> pref, bits<6> opcode, dag OOL, dag IOL, string asmstr,
   // and one for 32-bit arguments, this bit breaks the degeneracy between
   // the two forms and allows TableGen to generate mapping tables.
   bit Interpretation64Bit = 0;
 
   // Fields used for relation models.
   string BaseName = "";
 }
 
 class MLS_DForm_R_SI34_RTA5_MEM<bits<6> opcode, dag OOL, dag IOL, string asmstr,
                                 InstrItinClass itin, list<dag> pattern>
   : PI<1, opcode, OOL, IOL, asmstr, itin> {
   bits<5> FRS;
   bits<39> D_RA;
 
   let Pattern = pattern;
 
   // The prefix.
   let Inst{6-7} = 2;
   let Inst{8-10} = 0;
   let Inst{11} = PCRel;
   let Inst{12-13} = 0;
   let Inst{14-31} = D_RA{33-16}; // d0
let Predicates = [PrefixInstrs] in {
   def XXSPLTIDP : 8RR_DForm_IMM32_XT6<32, 2, (outs vsrc:$XT),
                                       (ins i32imm:$IMM32),
                                       "xxspltidp $XT, $IMM32", IIC_VecGeneral,
                                       [(set v2f64:$XT,
                                             (PPCxxspltidp i32:$IMM32))]>;
   def XXSPLTI32DX :
       8RR_DForm_IMM32_XT6_IX<32, 0, (outs vsrc:$XT),
                              (ins vsrc:$XTi, i1imm:$IX, i32imm:$IMM32),
-                             "xxsplti32dx $XT, $IX, $IMM32", IIC_VecGeneral, []>,
+                             "xxsplti32dx $XT, $IX, $IMM32", IIC_VecGeneral,
+                             [(set v2i64:$XT,
+                                   (PPCxxsplti32dx v2i64:$XTi, i1:$IX,
+                                                   i32:$IMM32))]>,
                              RegConstraint<"$XTi = $XT">, NoEncode<"$XTi">;
   def XXPERMX :
     8RR_XX4Form_IMM3_XTABC6<34, 0, (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB,
                             vsrc:$XC, u3imm:$UIM),
                             "xxpermx $XT, $XA, $XB, $XC, $UIM",
                             IIC_VecPerm, []>;
   def XXBLENDVB :
     8RR_XX4Form_XTABC6<33, 0, (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB,

llvm/test/CodeGen/PowerPC/p10-splatImm32.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -O2 \
+; RUN:     -ppc-asm-full-reg-names -mcpu=pwr10 < %s | \
+; RUN:     FileCheck --check-prefix=CHECK-LE %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu -O2 \
+; RUN:     -ppc-asm-full-reg-names -mcpu=pwr10 < %s | \
+; RUN:     FileCheck --check-prefix=CHECK-BE %s
+
+; Function Attrs: norecurse nounwind readnone
+define  <4 x i32> @test_xxsplti32dx_1(<4 x i32> %a) {
+; CHECK-LE-LABEL: test_xxsplti32dx_1:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, -1, 566
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_1:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, 0, 566
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins1 = shufflevector <4 x i32> %a, <4 x i32> <i32 undef, i32 566, i32 undef, i32 566>, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x i32> %vecins1
+}
+
+; Function Attrs: norecurse nounwind readnone
+define  <4 x i32> @test_xxsplti32dx_2(<4 x i32> %a) {
+; CHECK-LE-LABEL: test_xxsplti32dx_2:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, 0, 33
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_2:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, -1, 33
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins1 = shufflevector <4 x i32> <i32 33, i32 undef, i32 33, i32 undef>, <4 x i32> %a, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x i32> %vecins1
+}
+
+; Function Attrs: norecurse nounwind readnone
+define  <4 x i32> @test_xxsplti32dx_3(<4 x i32> %a) {
+; CHECK-LE-LABEL: test_xxsplti32dx_3:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, -1, 12
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_3:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, 0, 12
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins1 = shufflevector <4 x i32> %a, <4 x i32> <i32 undef, i32 12, i32 undef, i32 12>, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x i32> %vecins1
+}
+
+; Function Attrs: norecurse nounwind readnone
+define  <4 x i32> @test_xxsplti32dx_4(<4 x i32> %a) {
+; CHECK-LE-LABEL: test_xxsplti32dx_4:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, 0, -683
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_4:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, -1, -683
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins1 = shufflevector <4 x i32> <i32 -683, i32 undef, i32 -683, i32 undef>, <4 x i32> %a, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x i32> %vecins1
+}
+
+; Function Attrs: nounwind
+define  <4 x float> @test_xxsplti32dx_5(<4 x float> %vfa) {
+; CHECK-LE-LABEL: test_xxsplti32dx_5:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, -1, 1065353216
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_5:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, 0, 1065353216
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins3.i = shufflevector <4 x float> %vfa, <4 x float> <float undef, float 1.000000e+00, float undef, float 1.000000e+00>, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x float> %vecins3.i
+}
+
+; Function Attrs: nounwind
+define  <4 x float> @test_xxsplti32dx_6(<4 x float> %vfa) {
+; CHECK-LE-LABEL: test_xxsplti32dx_6:
+; CHECK-LE:       # %bb.0: # %entry
+; CHECK-LE-NEXT:    xxsplti32dx vs34, 0, 1073741824
+; CHECK-LE-NEXT:    blr
+;
+; CHECK-BE-LABEL: test_xxsplti32dx_6:
+; CHECK-BE:       # %bb.0: # %entry
+; CHECK-BE-NEXT:    xxsplti32dx vs34, -1, 1073741824
+; CHECK-BE-NEXT:    blr
+entry:
+  %vecins3.i = shufflevector <4 x float> <float 2.000000e+00, float undef, float 2.000000e+00, float undef>, <4 x float> %vfa, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+  ret <4 x float> %vecins3.i
+}