[PowerPC] Avoid scalarization of vector truncate

[RolandF]

Summary

The PowerPC code generator currently scalarizes vector truncates that would fit in a vector register, resulting in vector extracts, scalar operations, and vector merges. This patch custom lowers a vector truncate that would fit in a register to a vector shuffle instead.

[RolandF]

Fix comment.

[nemanjai]

LGTM. I've had a pretty close look at it and am going to approve it. However, I realized that I was the only reviewer on it so I'll ask you to give it a couple of days to give at least @hfinkel and @jsji (but others as well) a chance to have a look if they are interested.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6920

This may be misleading to the reader as it suggests that the byte order within an element in the register is different on little endian systems. It might be clearer to use LLVM-like notation for vectors and write these as:

BE:
<MSB1| LSB1, MSB2|LSB2,   uu,   uu,   uu,   uu,   uu,   uu> to
<LSB1, LSB2,      u, u, u, u, u, u, u, u, u, u, u, u, u, u>

LE:
<  uu,   uu,   uu,   uu,   uu,   uu, MSB2|LSB2, MSB1| LSB1> to
<u, u, u, u, u, u, u, u, u, u, u, u,      u, u, LSB2, LSB1>

[jsji]

LGTM. Thanks for exploiting this! Some minor comments.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

645

Maybe add a comment here about why only these 5 target VT are supported?

6928

Comment a little misleading here?
v4f64 is also a legal PPC QPX vector. Maybe we should limit it to be "legal Altivec vector" here?

6949

Maybe we can just use WideNumElts + 1 here?

llvm/test/CodeGen/PowerPC/vec-trunc.ll

3

Why we require pwr9 here? I think this should apply to pwr8 and below as well?

17

Can we add this new testcase in a NFC patch first, then show ONLY the difference caused by this opt here?

It will be great for others to see that this patch reduced the number of instructions from 33 to 1!

19

Maybe we should either add the check to xxswapd in BE ?
Or else why not just remove all CHECK-BE? As they are all identical to CHECK?

[nemanjai]

llvm/test/CodeGen/PowerPC/vec-trunc.ll

17

I think if we're doing this, it would probably be nice to see the entire codegen. Just produce the checks using utils/update_llc_checks.py.

[RolandF]

llvm/test/CodeGen/PowerPC/vec-trunc.ll

19

LowerTRUNCATEVector has an if statement checking LE or BE, so even though we want the result to be the same in the end we have to test both LE and BE to test the whole function.

[jsji]

llvm/test/CodeGen/PowerPC/vec-trunc.ll

19

Yes, we need to test both BE/LE, but no need to have CHECK-BE/CHECK-LE prefixes if the results are the same?

[RolandF]

llvm/test/CodeGen/PowerPC/vec-trunc.ll

19

Oh, okay, I see what you mean now. I could have saved a bunch of check lines.

[RolandF]

Update diff to show test changes and respond to comments.

llvm/lib/Target/PowerPC/PPCISelLowering.h

private:
 
     bool directMoveIsProfitable(const SDValue &Op) const;
     SDValue LowerINT_TO_FPDirectMove(SDValue Op, SelectionDAG &DAG,
                                      const SDLoc &dl) const;
 
     SDValue LowerINT_TO_FPVector(SDValue Op, SelectionDAG &DAG,
                                  const SDLoc &dl) const;
 
+    SDValue LowerTRUNCATEVector(SDValue Op, SelectionDAG &DAG) const;
+
     SDValue getFramePointerFrameIndex(SelectionDAG & DAG) const;
     SDValue getReturnAddrFrameIndex(SelectionDAG & DAG) const;
 
     bool
     IsEligibleForTailCallOptimization(SDValue Callee,
                                       CallingConv::ID CalleeCC,
                                       bool isVarArg,
                                       const SmallVectorImpl<ISD::InputArg> &Ins,

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

 static cl::opt<bool> EnableQuadPrecision("enable-ppc-quad-precision",
 cl::desc("enable quad precision float support on ppc"), cl::Hidden);
 
 STATISTIC(NumTailCalls, "Number of tail calls");
 STATISTIC(NumSiblingCalls, "Number of sibling calls");
 
 static bool isNByteElemShuffleMask(ShuffleVectorSDNode *, unsigned, int);
 
+static SDValue widenVec(SelectionDAG &DAG, SDValue Vec, const SDLoc &dl);
+
 // FIXME: Remove this once the bug has been fixed!
 extern cl::opt<bool> ANDIGlueBug;
 
 PPCTargetLowering::PPCTargetLowering(const PPCTargetMachine &TM,
                                      const PPCSubtarget &STI)
     : TargetLowering(TM), Subtarget(STI) {
   // Use _setjmp/_longjmp instead of setjmp/longjmp.
   setUseUnderscoreSetJmp(true);
for (MVT VT : MVT::vector_valuetypes()) {
         setLoadExtAction(ISD::EXTLOAD, VT, InnerVT, Expand);
       }
     }
 
     // We can custom expand all VECTOR_SHUFFLEs to VPERM, others we can handle
     // with merges, splats, etc.
     setOperationAction(ISD::VECTOR_SHUFFLE, MVT::v16i8, Custom);
 
+    setOperationAction(ISD::TRUNCATE, MVT::v8i8, Custom);
+    setOperationAction(ISD::TRUNCATE, MVT::v4i8, Custom);
+    setOperationAction(ISD::TRUNCATE, MVT::v2i8, Custom);
+    setOperationAction(ISD::TRUNCATE, MVT::v4i16, Custom);
+    setOperationAction(ISD::TRUNCATE, MVT::v2i16, Custom);
+
     setOperationAction(ISD::AND   , MVT::v4i32, Legal);
     setOperationAction(ISD::OR    , MVT::v4i32, Legal);
     setOperationAction(ISD::XOR   , MVT::v4i32, Legal);
     setOperationAction(ISD::LOAD  , MVT::v4i32, Legal);
     setOperationAction(ISD::SELECT, MVT::v4i32,
                        Subtarget.useCRBits() ? Legal : Expand);
     setOperationAction(ISD::STORE , MVT::v4i32, Legal);
     setOperationAction(ISD::FP_TO_SINT, MVT::v4i32, Legal);
SDValue PPCTargetLowering::LowerTRUNCATE(SDValue Op, SelectionDAG &DAG) const {
   assert(Op.getValueType() == MVT::i1 &&
          "Custom lowering only for i1 results");
 
   SDLoc DL(Op);
   return DAG.getNode(PPCISD::ANDIo_1_GT_BIT, DL, MVT::i1,
                      Op.getOperand(0));
 }
 
+SDValue PPCTargetLowering::LowerTRUNCATEVector(SDValue Op,
+                                               SelectionDAG &DAG) const {
+
+  // Implements a vector truncate that fits in a vector register as a shuffle.
+  // We want to legalize vector truncates down to where the source fits in
+  // a vector register (and target is therefore smaller than vector register
+  // size).  At that point legalization will try to custom lower the sub-legal
+  // result and get here - where we can contain the truncate as a single target
+  // operation.
+
+  // For example a trunc <2 x i16> to <2 x i8> could be visualized as follows:
+  //   [MSB1 LSB1][MSB2][MSB2 LSB2] to [LSB1][LSB2]
+  //
+  // We will implement it for big-endian ordering as this (where x denotes
+  // undefined):
+  //   [MSB1 LSB1][MSB2 LSB2][x x][x x][x x][x x][x x][x x] to
+  //   [LSB1][LSB2][x][x][x][x][x][x][x][x][x][x][x][x][x][x]
+  // 
+  // The same operation in little-endian ordering will be:
+  //   [LSB1 MSB1][LSB2 MSB2][x x][x x][x x][x x][x x][x x] to
+  //   [LSB1][LSB2][x][x][x][x][x][x][x][x][x][x][x][x][x][x]
+
+  assert(Op.getValueType().isVector() && "Vector type expected.");
+
+  SDLoc DL(Op);
+  SDValue N1 = Op.getOperand(0);
+  unsigned SrcSize = N1.getValueType().getSizeInBits();
+  assert(SrcSize <= 128 && "Source must be no wider than a legal PPC vector");
+  SDValue WideSrc = SrcSize == 128 ? N1 : widenVec(DAG, N1, DL);
+
+  EVT TrgVT = Op.getValueType();
+  unsigned TrgNumElts = TrgVT.getVectorNumElements();
+  EVT EltVT = TrgVT.getVectorElementType();
+  unsigned WideNumElts = 128 / EltVT.getSizeInBits();
+  EVT WideVT = EVT::getVectorVT(*DAG.getContext(), EltVT, WideNumElts);
+
+  // First list the elements we want to keep.
+  unsigned SizeMult = SrcSize / TrgVT.getSizeInBits();
+  SmallVector<int, 16> ShuffV;
+  if (Subtarget.isLittleEndian())
+    for (unsigned i = 0; i < TrgNumElts; ++i)
+      ShuffV.push_back(i * SizeMult);
+  else
+    for (unsigned i = 1; i <= TrgNumElts; ++i)
+      ShuffV.push_back(i * SizeMult - 1);
+
+  // Populate the remaining elements with undefs.
+  for (unsigned i = TrgNumElts; i < WideNumElts; ++i)
+    ShuffV.push_back(i + WideNumElts);
+
+  SDValue Conv = DAG.getNode(ISD::BITCAST, DL, WideVT, WideSrc);
+  return DAG.getVectorShuffle(WideVT, DL, Conv, DAG.getUNDEF(WideVT), ShuffV);
+}
+
 /// LowerSELECT_CC - Lower floating point select_cc's into fsel instruction when
 /// possible.
 SDValue PPCTargetLowering::LowerSELECT_CC(SDValue Op, SelectionDAG &DAG) const {
   // Not FP? Not a fsel.
   if (!Op.getOperand(0).getValueType().isFloatingPoint() ||
       !Op.getOperand(2).getValueType().isFloatingPoint())
     return Op;
 
void PPCTargetLowering::ReplaceNodeResults(SDNode *N,
   }
   case ISD::FP_TO_SINT:
   case ISD::FP_TO_UINT:
     // LowerFP_TO_INT() can only handle f32 and f64.
     if (N->getOperand(0).getValueType() == MVT::ppcf128)
       return;
     Results.push_back(LowerFP_TO_INT(SDValue(N, 0), DAG, dl));
     return;
+  case ISD::TRUNCATE: {
+    EVT TrgVT = N->getValueType(0);
+    if (TrgVT.isVector() &&
+        isOperationCustom(N->getOpcode(), TrgVT) &&
+        N->getOperand(0).getValueType().getSizeInBits() <= 128)
+      Results.push_back(LowerTRUNCATEVector(SDValue(N, 0), DAG));
+    return;
+  }
   case ISD::BITCAST:
     // Don't handle bitcast here.
     return;
   }
 }
 
 //===----------------------------------------------------------------------===//
 //  Other Lowering Code

llvm/test/CodeGen/PowerPC/vec-trunc.ll

+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN:     -mcpu=pwr9 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
+; RUN: FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu \
+; RUN:     -mcpu=pwr9 -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr < %s | \
+; RUN: FileCheck %s --check-prefix=CHECK-BE
+
+define void @test8i8(<8 x i8>* nocapture %Sink, <8 x i16>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <8 x i16>, <8 x i16>* %SrcPtr, align 16
+  %1 = trunc <8 x i16> %0 to <8 x i8>
+  store <8 x i8> %1, <8 x i8>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test8i8
+; CHECK: vpkuhum
+; CHECK-BE-LABEL: @test8i8
+; CHECK-BE: vpkuhum
+
+define void @test4i8(<4 x i8>* nocapture %Sink, <4 x i16>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <4 x i16>, <4 x i16>* %SrcPtr, align 16
+  %1 = trunc <4 x i16> %0 to <4 x i8>
+  store <4 x i8> %1, <4 x i8>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test4i8
+; CHECK: vpkuhum
+; CHECK-BE-LABEL: @test4i8
+; CHECK-BE: vpkuhum
+
+define void @test4i8w(<4 x i8>* nocapture %Sink, <4 x i32>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <4 x i32>, <4 x i32>* %SrcPtr, align 16
+  %1 = trunc <4 x i32> %0 to <4 x i8>
+  store <4 x i8> %1, <4 x i8>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test4i8w
+; CHECK: vperm
+; CHECK-BE-LABEL: @test4i8w
+; CHECK-BE: vperm
+
+define void @test2i8(<2 x i8>* nocapture %Sink, <2 x i16>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <2 x i16>, <2 x i16>* %SrcPtr, align 16
+  %1 = trunc <2 x i16> %0 to <2 x i8>
+  store <2 x i8> %1, <2 x i8>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test2i8
+; CHECK: vpkuhum
+; CHECK-BE-LABEL: @test2i8
+; CHECK-BE: vpkuhum
+
+define void @test4i16(<4 x i16>* nocapture %Sink, <4 x i32>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <4 x i32>, <4 x i32>* %SrcPtr, align 16
+  %1 = trunc <4 x i32> %0 to <4 x i16>
+  store <4 x i16> %1, <4 x i16>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test4i16
+; CHECK: vpkuwum
+; CHECK-BE-LABEL: @test4i16
+; CHECK-BE: vpkuwum
+
+define void @test2i16(<2 x i16>* nocapture %Sink, <2 x i32>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <2 x i32>, <2 x i32>* %SrcPtr, align 16
+  %1 = trunc <2 x i32> %0 to <2 x i16>
+  store <2 x i16> %1, <2 x i16>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test2i16
+; CHECK: vpkuwum
+; CHECK-BE-LABEL: @test2i16
+; CHECK-BE: vpkuwum
+
+define void @test2i16d(<2 x i16>* nocapture %Sink, <2 x i64>* nocapture readonly %SrcPtr) {
+entry:
+  %0 = load <2 x i64>, <2 x i64>* %SrcPtr, align 16
+  %1 = trunc <2 x i64> %0 to <2 x i16>
+  store <2 x i16> %1, <2 x i16>* %Sink, align 16
+  ret void
+}
+; CHECK-LABEL: @test2i16d
+; CHECK: vperm
+; CHECK-BE-LABEL: @test2i16d
+; CHECK-BE: vperm