Scalar to vector conversions using direct moves

[nemanjai]

Summary

We currently always build vectors from scalar non-constant values using stack operations which involves the load-hit-store hazard. This patch is the first in a series that will allow operations such as scalar to vector, extract vector element, etc. to be done using direct move instructions rather than memory operations.
Since this is just the first patch, the code in some cases is not yet optimal, but all scalar to vector operations involve fewer memory operations.

[hfinkel]

lib/Target/PowerPC/PPCISelLowering.cpp

7504

Just return the result of the DAG.getNode(...) call. Then you don't even need the { } around the body of the if.

That having been said, it does not seem like you have to do this at all. Just declare the relevant SCALAR_TO_VECTOR as Legal, and pattern match it in the usual way in the .td file.

lib/Target/PowerPC/PPCISelLowering.h

135

Why are you adding these? What generates them?

lib/Target/PowerPC/PPCInstrVSX.td

1198

Given that I don't see anything that actually generates this ISD nodes, I'm suspicions that you don't actually have tests for these.

1236

Maybe we should name this BE_LOW_BYTE, or something like that?

[nemanjai]

lib/Target/PowerPC/PPCISelLowering.cpp

7504

I agree with the first point and can do as you suggested.
However, I felt that this way is a little cleaner than doing it all in the .td file. If I try to use PPCmtvsrdVec in an output pattern, tblgen gives me the message:

Cannot use 'PPCmtvsrdVec' in an output pattern!

Of course, I could do away with that SDAG node altogether and provide the same patterns for scalar_to_vector as I did for the new nodes. I just felt that the node that explicitly names the move to VSR to be more descriptive and allow for a different implementation of scalar_to_vector should a need for that arise in the future.
And again, I will implement it however you would prefer.

lib/Target/PowerPC/PPCISelLowering.h

135

Thank you for catching this. It turns out that this is one of those events where I have tried a number of approaches to get the results I want until I settled on the approach that I plan to stick with. However, these SDAG nodes were missed in the re-re-re-factoring :).
The idea was to emit these SDAG nodes when we need scalar to vector conversions so that they can be matched, but since they will only match one instruction, they're kind of pointless.
I will remove them in the next review.

lib/Target/PowerPC/PPCInstrVSX.td

1198

Yes, sorry about this. See above. I will remove the SD nodes and the patterns from these instructions.

1236

The BE version of this move will put the value into the most significant byte in the VSR. Similarly with the halfword, word and doubleword versions. The LE versions will place it into the corresponding least significant element.
Maybe BE_MSB? BE_HI_BYTE? And correspondingly for the half, word and double?

1247

The naming is different with LE since what is happening is fundamentally different. The move is the same for the byte, halfword and word. The doubleword version is the same move as for BE. That's why the LE versions have the "MoveWord" portion, then the "CopyWord" portion (for the regclass copy) and finaly the LE_BHW to signify that this is a byte/half/word correctly aligned for the shift to element 0 on LE.
I can put a comment to this end in the code.

[nemanjai]

Removed the custom code that isn't necessary.
Added a test case specifically for scalar_to_vector nodes.
Renamed the DAG's in PPCInstrVSX.td.
Reconciled the predicate for the v4f32 version (it was predicated on HasDirectMove in the .td file and on hasP8Vector() in the c++ code).

[wschmidt]

A few issues and questions...

lib/Target/PowerPC/PPCInstrVSX.td

1198

The comment at line 1195 is wrong. These are scalar conversions between single and double precision, right?

1245

BE_DWORD_0? This isn't an LE pattern, so it needs a BE designator, correct? Even though it's used in an LE pattern, it is LE doubleword 1 there, so naming it BE_DWORD_0 is more expressive.

1252

I find LE_BHW and LE_DBL to be pretty incomprehensible names. For consistency and understanding, I would name them LE_WORD_0 and LE_DWORD_0. Also, LE_CPW is arguably LE_WORD_1 and LE_CPD is LE_DWORD_1. This gives the more understandable:

dag LE_DWORD_1 = (v2i64 (COPY_TO_REGCLASS BE_DWORD_0, VSRC));

reinforcing that LE doubleword 1 is the same as BE doubleword 0.

I don't much care what you call LE_MVW so long as it is readable and implies moving an integer to a vector register. Longer meaningful names beat brief incomprehensible ones. Of course that's still better than long incomprehensible ones. ;)

test/CodeGen/PowerPC/p8-scalar_vector_conversions.ll

74 ↗

(On Diff #30861)

I was confused by this at first, but now I get it. In all of these tests, the code generation that you are checking for is just for the %splat.splatinsert calculation, right? This is the part that translates into a scalar_to_vector node. I assume that the generated code follows this up with code to splat element 0 to the entire result register, correct?

[wschmidt]

lib/Target/PowerPC/PPCInstrVSX.td

1198

Withdrawn per online discussion.

[nemanjai]

lib/Target/PowerPC/PPCInstrVSX.td

1245

Fair enough, the value needs to be shifted/swapped for LE, so I'll rename it to BE_DWORD_0. And the subsequent swap will put it in LE doubleword 0.

1252

OK, I will rename them as you suggest:
LE_BHW -> LE_WORD_0
LE_CPW -> LE_WORD_1
LE_DBL -> LE_DWORD_0
LE_CPD -> LE_DWORD_1

test/CodeGen/PowerPC/p8-scalar_vector_conversions.ll

74 ↗

(On Diff #30861)

Yes, the subsequent instruction will be a splat (VSX if we have one, VMX otherwise). I can add those to the check patterns but, I can't verify the correct register (because of the "off-by-32 relationship" between the VSX and VMX registers). I was actually hoping to use the FileCheck capability of checking line numbers (ranges), but that actually looks for the word line I think. So I can add the CHECK: vsplat[bhw] and just confirm the right element in the splat instruction if you'd like.

[wschmidt]

test/CodeGen/PowerPC/p8-scalar_vector_conversions.ll

74 ↗

(On Diff #30861)

No, that's ok. Just add a comment at the beginning of the file indicating that you are just checking the code generated for the insertelement. Alternatively, you can make this more obvious by putting the CHECK comment lines right after the insertelement in each test variant.

[nemanjai]

@[e-mail] Hi Hal, if you get the chance, can you please have a look and let me know if you're ok with the updates? Thank you.

[hfinkel]

LGTM.

[nemanjai]

Committed revision 244921.

lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
       MFVSR,
 
       /// Direct move from a GPR to a VSX register (algebraic)
       MTVSRA,
 
       /// Direct move from a GPR to a VSX register (zero)
       MTVSRZ,
 
+      /// Direct move from a GPR to a VSX register (keeping a vector type)
+      MTVSRDVEC,
+
+      /// Conversions between SP 64-bit values and SP 32-bit values
+      SP_TO_VEC_SP,
+      VEC_SP_TO_SP,
+
       // FIXME: Remove these once the ANDI glue bug is fixed:
       /// i1 = ANDIo_1_[EQ|GT]_BIT(i32 or i64 x) - Represents the result of the
       /// eq or gt bit of CR0 after executing andi. x, 1. This is used to
       /// implement truncation of i32 or i64 to i1.
       ANDIo_1_EQ_BIT, ANDIo_1_GT_BIT,
 
       // READ_TIME_BASE - A read of the 64-bit time-base register on a 32-bit
       // target (returns (Lo, Hi)). It takes a chain operand.

lib/Target/PowerPC/PPCISelLowering.cpp

if (Subtarget.hasAltivec()) {
     // Altivec does not contain unordered floating-point compare instructions
     setCondCodeAction(ISD::SETUO, MVT::v4f32, Expand);
     setCondCodeAction(ISD::SETUEQ, MVT::v4f32, Expand);
     setCondCodeAction(ISD::SETO,   MVT::v4f32, Expand);
     setCondCodeAction(ISD::SETONE, MVT::v4f32, Expand);
 
     if (Subtarget.hasVSX()) {
       setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v2f64, Legal);
+      if (Subtarget.hasP8Vector())
+        setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v4f32, Legal);
+      if (Subtarget.hasDirectMove()) {
+        setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v16i8, Custom);
+        setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v8i16, Custom);
+        setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v4i32, Custom);
+        setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v2i64, Custom);
+      }
       setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v2f64, Legal);
 
       setOperationAction(ISD::FFLOOR, MVT::v2f64, Legal);
       setOperationAction(ISD::FCEIL, MVT::v2f64, Legal);
       setOperationAction(ISD::FTRUNC, MVT::v2f64, Legal);
       setOperationAction(ISD::FNEARBYINT, MVT::v2f64, Legal);
       setOperationAction(ISD::FROUND, MVT::v2f64, Legal);
 
const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::RET_FLAG:        return "PPCISD::RET_FLAG";
   case PPCISD::READ_TIME_BASE:  return "PPCISD::READ_TIME_BASE";
   case PPCISD::EH_SJLJ_SETJMP:  return "PPCISD::EH_SJLJ_SETJMP";
   case PPCISD::EH_SJLJ_LONGJMP: return "PPCISD::EH_SJLJ_LONGJMP";
   case PPCISD::MFOCRF:          return "PPCISD::MFOCRF";
   case PPCISD::MFVSR:           return "PPCISD::MFVSR";
   case PPCISD::MTVSRA:          return "PPCISD::MTVSRA";
   case PPCISD::MTVSRZ:          return "PPCISD::MTVSRZ";
+  case PPCISD::MTVSRDVEC:       return "PPCISD::MTVSRDVEC";
+  case PPCISD::SP_TO_VEC_SP:    return "PPCISD::SP_TO_VEC_SP";
+  case PPCISD::VEC_SP_TO_SP:    return "PPCISD::VEC_SP_TO_SP";
   case PPCISD::ANDIo_1_EQ_BIT:  return "PPCISD::ANDIo_1_EQ_BIT";
   case PPCISD::ANDIo_1_GT_BIT:  return "PPCISD::ANDIo_1_GT_BIT";
   case PPCISD::VCMP:            return "PPCISD::VCMP";
   case PPCISD::VCMPo:           return "PPCISD::VCMPo";
   case PPCISD::LBRX:            return "PPCISD::LBRX";
   case PPCISD::STBRX:           return "PPCISD::STBRX";
   case PPCISD::LFIWAX:          return "PPCISD::LFIWAX";
   case PPCISD::LFIWZX:          return "PPCISD::LFIWZX";
SDValue PPCTargetLowering::LowerSIGN_EXTEND_INREG(SDValue Op,
   }
 
   return SDValue();
 }
 
 SDValue PPCTargetLowering::LowerSCALAR_TO_VECTOR(SDValue Op,
                                                    SelectionDAG &DAG) const {
   SDLoc dl(Op);
+  // With direct moves, we can build integral vectors without store/load
+  if (Subtarget.hasDirectMove()) {
+    SDValue Move = DAG.getNode(PPCISD::MTVSRDVEC, dl, Op.getValueType(),
+                               Op.getOperand(0));
+    return Move;
+  }
+
   // Create a stack slot that is 16-byte aligned.
   MachineFrameInfo *FrameInfo = DAG.getMachineFunction().getFrameInfo();
   int FrameIdx = FrameInfo->CreateStackObject(16, 16, false);
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   SDValue FIdx = DAG.getFrameIndex(FrameIdx, PtrVT);
 
   // Store the input value into Value#0 of the stack slot.
   SDValue Store = DAG.getStore(DAG.getEntryNode(), dl,
PPCTargetLowering::getScratchRegisters(CallingConv::ID) const {
 
   return ScratchRegs;
 }
 
 bool
 PPCTargetLowering::shouldExpandBuildVectorWithShuffles(
                      EVT VT , unsigned DefinedValues) const {
   if (VT == MVT::v2i64)
-    return false;
+    return Subtarget.hasDirectMove(); // Don't need stack ops with direct moves
 
   if (Subtarget.hasQPX()) {
     if (VT == MVT::v4f32 || VT == MVT::v4f64 || VT == MVT::v4i1)
       return true;
   }
 
   return TargetLowering::shouldExpandBuildVectorWithShuffles(VT, DefinedValues);
 }

lib/Target/PowerPC/PPCInstrVSX.td

 def PPClxvd2x  : SDNode<"PPCISD::LXVD2X", SDT_PPClxvd2x,
                         [SDNPHasChain, SDNPMayLoad]>;
 def PPCstxvd2x : SDNode<"PPCISD::STXVD2X", SDT_PPCstxvd2x,
                         [SDNPHasChain, SDNPMayStore]>;
 def PPCxxswapd : SDNode<"PPCISD::XXSWAPD", SDT_PPCxxswapd, [SDNPHasChain]>;
 def PPCmfvsr : SDNode<"PPCISD::MFVSR", SDTUnaryOp, []>;
 def PPCmtvsra : SDNode<"PPCISD::MTVSRA", SDTUnaryOp, []>;
 def PPCmtvsrz : SDNode<"PPCISD::MTVSRZ", SDTUnaryOp, []>;
+def PPCSPtoVecSP : SDNode<"PPCISD::SP_TO_VEC_SP", SDTUnaryOp, []>;
+def PPCVecSPtoSP : SDNode<"PPCISD::VEC_SP_TO_SP", SDTUnaryOp, []>;
+def PPCmtvsrdVec : SDNode<"PPCISD::MTVSRDVEC", SDTUnaryOp, []>;
 
 multiclass XX3Form_Rcr<bits<6> opcode, bits<7> xo, dag OOL, dag IOL,
                     string asmbase, string asmstr, InstrItinClass itin,
                     list<dag> pattern> {
   let BaseName = asmbase in {
     def NAME : XX3Form_Rc<opcode, xo, OOL, IOL,
                        !strconcat(asmbase, !strconcat(" ", asmstr)), itin,
                        pattern>;
let AddedComplexity = 400 in { // Prefer VSX patterns over non-VSX patterns.
   let IsVSXFMAAlt = 1 in
   def XSNMSUBMSP : XX3Form<60, 153,
                           (outs vssrc:$XT),
                           (ins vssrc:$XTi, vssrc:$XA, vssrc:$XB),
                           "xsnmsubmsp $XT, $XA, $XB", IIC_VecFP, []>,
                           RegConstraint<"$XTi = $XT">, NoEncode<"$XTi">,
                           AltVSXFMARel;
   }
+
+  // Single Precision Conversions (FP <-> INT)
+  def XSCVSXDSP : XX2Form<60, 312,
+                      (outs vssrc:$XT), (ins vsfrc:$XB),
+                      "xscvsxdsp $XT, $XB", IIC_VecFP,
+                      [(set f32:$XT, (PPCfcfids f64:$XB))]>;
+  def XSCVUXDSP : XX2Form<60, 296,
+                      (outs vssrc:$XT), (ins vsfrc:$XB),
+                      "xscvuxdsp $XT, $XB", IIC_VecFP,
+                      [(set f32:$XT, (PPCfcfidus f64:$XB))]>;
+
+  // Conversions between vector and scalar single precision
+  def XSCVDPSPN : XX2Form<60, 267, (outs vsrc:$XT), (ins vssrc:$XB),
+                          "xscvdpspn $XT, $XB", IIC_VecFP,
+                          [(set v4f32:$XT, (PPCSPtoVecSP f32:$XB))]>;
+  def XSCVSPDPN : XX2Form<60, 331, (outs vssrc:$XT), (ins vsrc:$XB),
+                          "xscvspdpn $XT, $XB", IIC_VecFP,
+                          [(set f32:$XT, (PPCVecSPtoSP v4f32:$XB))]>;
+
 } // AddedComplexity = 400
 } // HasP8Vector
 
 let Predicates = [HasDirectMove, HasVSX] in {
   // VSX direct move instructions
   def MFVSRD : XX1_RS6_RD5_XO<31, 51, (outs g8rc:$rA), (ins vsfrc:$XT),
                               "mfvsrd $rA, $XT", IIC_VecGeneral,
                               [(set i64:$rA, (PPCmfvsr f64:$XT))]>,
       Requires<[In64BitMode]>;
   def MFVSRWZ : XX1_RS6_RD5_XO<31, 115, (outs gprc:$rA), (ins vsfrc:$XT),
                                "mfvsrwz $rA, $XT", IIC_VecGeneral,
                                [(set i32:$rA, (PPCmfvsr f64:$XT))]>;
   def MTVSRD : XX1_RS6_RD5_XO<31, 179, (outs vsfrc:$XT), (ins g8rc:$rA),
                               "mtvsrd $XT, $rA", IIC_VecGeneral,
                               [(set f64:$XT, (PPCmtvsra i64:$rA))]>,
       Requires<[In64BitMode]>;
   def MTVSRWA : XX1_RS6_RD5_XO<31, 211, (outs vsfrc:$XT), (ins gprc:$rA),
                                "mtvsrwa $XT, $rA", IIC_VecGeneral,
                                [(set f64:$XT, (PPCmtvsra i32:$rA))]>;
   def MTVSRWZ : XX1_RS6_RD5_XO<31, 243, (outs vsfrc:$XT), (ins gprc:$rA),
                                "mtvsrwz $XT, $rA", IIC_VecGeneral,
                                [(set f64:$XT, (PPCmtvsrz i32:$rA))]>;
 } // HasDirectMove, HasVSX
+
+/*  Direct moves of various size entities from GPR's into VSR's. Each lines
+    the value up into element 0 (both BE and LE). Namely, entities smaller than
+    a doubleword are shifted left and moved for BE. For LE, they're moved, then
+    swapped to go into the least significant element of the VSR.
+*/
+def Moves {
+  dag BE_BYTE = (MTVSRD
+                  (RLDICR
+                    (INSERT_SUBREG (i64 (IMPLICIT_DEF)), $A, sub_32), 56, 7));
+  dag BE_HALF = (MTVSRD
+                  (RLDICR
+                    (INSERT_SUBREG (i64 (IMPLICIT_DEF)), $A, sub_32), 48, 15));
+  dag BE_WORD = (MTVSRD
+                  (RLDICR
+                    (INSERT_SUBREG (i64 (IMPLICIT_DEF)), $A, sub_32), 32, 31));
+  dag DBLWORD = (MTVSRD $A);
+
+  dag LE_MVW = (MTVSRD (INSERT_SUBREG (i64 (IMPLICIT_DEF)), $A, sub_32));
+  dag LE_CPW = (v2i64 (COPY_TO_REGCLASS LE_MVW, VSRC));
+  dag LE_BHW = (XXPERMDI LE_CPW, LE_CPW, 2);
+  dag LE_CPD = (v2i64 (COPY_TO_REGCLASS DBLWORD, VSRC));
+  dag LE_DBL = (XXPERMDI LE_CPD, LE_CPD, 2);
+}
+
+let Predicates = [IsBigEndian, HasDirectMove] in {
+  def : Pat<(v4f32 (scalar_to_vector f32:$A)),
+            (v4f32 (XSCVDPSPN $A))>;
+  def : Pat<(v16i8 (PPCmtvsrdVec i32:$A)),
+            (v16i8 (COPY_TO_REGCLASS Moves.BE_BYTE, VSRC))>;
+  def : Pat<(v8i16 (PPCmtvsrdVec i32:$A)),
+            (v8i16 (COPY_TO_REGCLASS Moves.BE_HALF, VSRC))>;
+  def : Pat<(v4i32 (PPCmtvsrdVec i32:$A)),
+            (v4i32 (COPY_TO_REGCLASS Moves.BE_WORD, VSRC))>;
+  def : Pat<(v2i64 (PPCmtvsrdVec i64:$A)),
+            (v2i64 (COPY_TO_REGCLASS Moves.DBLWORD, VSRC))>;
+} // IsBigEndian
+
+let Predicates = [IsLittleEndian, HasDirectMove] in {
+  def : Pat<(v4f32 (scalar_to_vector f32:$A)),
+            (v4f32 (XXSLDWI (XSCVDPSPN $A), (XSCVDPSPN $A), 1))>;
+  def : Pat<(v16i8 (PPCmtvsrdVec i32:$A)),
+            (v16i8 (COPY_TO_REGCLASS Moves.LE_BHW, VSRC))>;
+  def : Pat<(v8i16 (PPCmtvsrdVec i32:$A)),
+            (v8i16 (COPY_TO_REGCLASS Moves.LE_BHW, VSRC))>;
+  def : Pat<(v4i32 (PPCmtvsrdVec i32:$A)),
+            (v4i32 (COPY_TO_REGCLASS Moves.LE_BHW, VSRC))>;
+  def : Pat<(v2i64 (PPCmtvsrdVec i64:$A)),
+            (v2i64 Moves.LE_DBL)>;
+} // IsLittleEndian
+

lib/Target/PowerPC/PPCVSXCopy.cpp

struct PPCVSXCopy : public MachineFunctionPass {
     bool IsVRReg(unsigned Reg, MachineRegisterInfo &MRI) {
       return IsRegInClass(Reg, &PPC::VRRCRegClass, MRI);
     }
 
     bool IsF8Reg(unsigned Reg, MachineRegisterInfo &MRI) {
       return IsRegInClass(Reg, &PPC::F8RCRegClass, MRI);
     }
 
+    bool IsVSFReg(unsigned Reg, MachineRegisterInfo &MRI) {
+      return IsRegInClass(Reg, &PPC::VSFRCRegClass, MRI);
+    }
+
 protected:
     bool processBlock(MachineBasicBlock &MBB) {
       bool Changed = false;
 
       MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
       for (MachineBasicBlock::iterator I = MBB.begin(), IE = MBB.end();
            I != IE; ++I) {
         MachineInstr *MI = I;
         if (!MI->isFullCopy())
           continue;
 
         MachineOperand &DstMO = MI->getOperand(0);
         MachineOperand &SrcMO = MI->getOperand(1);
 
         if ( IsVSReg(DstMO.getReg(), MRI) &&
             !IsVSReg(SrcMO.getReg(), MRI)) {
           // This is a copy *to* a VSX register from a non-VSX register.
           Changed = true;
 
           const TargetRegisterClass *SrcRC =
             IsVRReg(SrcMO.getReg(), MRI) ? &PPC::VSHRCRegClass :
                                            &PPC::VSLRCRegClass;
           assert((IsF8Reg(SrcMO.getReg(), MRI) ||
-                  IsVRReg(SrcMO.getReg(), MRI)) &&
+                  IsVRReg(SrcMO.getReg(), MRI) ||
+                  IsVSFReg(SrcMO.getReg(), MRI)) &&
                  "Unknown source for a VSX copy");
 
           unsigned NewVReg = MRI.createVirtualRegister(SrcRC);
           BuildMI(MBB, MI, MI->getDebugLoc(),
                   TII->get(TargetOpcode::SUBREG_TO_REG), NewVReg)
             .addImm(1) // add 1, not 0, because there is no implicit clearing
                        // of the high bits.
             .addOperand(SrcMO)

test/CodeGen/PowerPC/fp-int-conversions-direct-moves.ll

 entry:
   %arg.addr = alloca i8, align 1
   store i8 %arg, i8* %arg.addr, align 1
   %0 = load i8, i8* %arg.addr, align 1
   %conv = uitofp i8 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z6testfcc
 ; CHECK: mtvsrwz [[MOVEREG01:[0-9]+]], 3
-; FIXME: Once we have XSCVUXDSP implemented, this will change
-; CHECK: fcfidus 1, [[MOVEREG01]]
+; CHECK: xscvuxdsp 1, [[MOVEREG01]]
 }
 
 ; Function Attrs: nounwind
 define zeroext i8 @_Z6testcdd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i8, align 1
   store i8 %arg, i8* %arg.addr, align 1
   %0 = load i8, i8* %arg.addr, align 1
   %conv = uitofp i8 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z7testfuch
 ; CHECK: mtvsrwz [[MOVEREG03:[0-9]+]], 3
-; FIXME: Once we have XSCVUXDSP implemented, this will change
-; CHECK: fcfidus 1, [[MOVEREG03]]
+; CHECK: xscvuxdsp 1, [[MOVEREG03]]
 }
 
 ; Function Attrs: nounwind
 define zeroext i8 @_Z7testucdd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i16, align 2
   store i16 %arg, i16* %arg.addr, align 2
   %0 = load i16, i16* %arg.addr, align 2
   %conv = sitofp i16 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z6testfss
 ; CHECK: mtvsrwa [[MOVEREG05:[0-9]+]], 3
-; FIXME: Once we have XSCVSXDSP implemented, this will change
-; CHECK: fcfids 1, [[MOVEREG05]]
+; CHECK: xscvsxdsp 1, [[MOVEREG05]]
 }
 
 ; Function Attrs: nounwind
 define signext i16 @_Z6testsdd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i16, align 2
   store i16 %arg, i16* %arg.addr, align 2
   %0 = load i16, i16* %arg.addr, align 2
   %conv = uitofp i16 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z7testfust
 ; CHECK: mtvsrwz [[MOVEREG07:[0-9]+]], 3
-; FIXME: Once we have XSCVUXDSP implemented, this will change
-; CHECK: fcfidus 1, [[MOVEREG07]]
+; CHECK: xscvuxdsp 1, [[MOVEREG07]]
 }
 
 ; Function Attrs: nounwind
 define zeroext i16 @_Z7testusdd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i32, align 4
   store i32 %arg, i32* %arg.addr, align 4
   %0 = load i32, i32* %arg.addr, align 4
   %conv = sitofp i32 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z6testfii
 ; CHECK: mtvsrwa [[MOVEREG09:[0-9]+]], 3
-; FIXME: Once we have XSCVSXDSP implemented, this will change
-; CHECK: fcfids 1, [[MOVEREG09]]
+; CHECK: xscvsxdsp 1, [[MOVEREG09]]
 }
 
 ; Function Attrs: nounwind
 define signext i32 @_Z6testidd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i32, align 4
   store i32 %arg, i32* %arg.addr, align 4
   %0 = load i32, i32* %arg.addr, align 4
   %conv = uitofp i32 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z7testfuij
 ; CHECK: mtvsrwz [[MOVEREG11:[0-9]+]], 3
-; FIXME: Once we have XSCVUXDSP implemented, this will change
-; CHECK: fcfidus 1, [[MOVEREG11]]
+; CHECK: xscvuxdsp 1, [[MOVEREG11]]
 }
 
 ; Function Attrs: nounwind
 define zeroext i32 @_Z7testuidd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i64, align 8
   store i64 %arg, i64* %arg.addr, align 8
   %0 = load i64, i64* %arg.addr, align 8
   %conv = sitofp i64 %0 to float
   ret float %conv
 ; CHECK-LABEL:@_Z7testfllx
 ; CHECK: mtvsrd [[MOVEREG13:[0-9]+]], 3
-; FIXME: Once we have XSCVSXDSP implemented, this will change
-; CHECK: fcfids 1, [[MOVEREG13]]
+; CHECK: xscvsxdsp 1, [[MOVEREG13]]
 }
 
 ; Function Attrs: nounwind
 define i64 @_Z7testlldd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

 entry:
   %arg.addr = alloca i64, align 8
   store i64 %arg, i64* %arg.addr, align 8
   %0 = load i64, i64* %arg.addr, align 8
   %conv = uitofp i64 %0 to float
   ret float %conv
 ; CHECK-LABEL: @_Z8testfully
 ; CHECK: mtvsrd [[MOVEREG15:[0-9]+]], 3
-; FIXME: Once we have XSCVUXDSP implemented, this will change
-; CHECK: fcfidus 1, [[MOVEREG15]]
+; CHECK: xscvuxdsp 1, [[MOVEREG15]]
 }
 
 ; Function Attrs: nounwind
 define i64 @_Z8testulldd(double %arg) {
 entry:
   %arg.addr = alloca double, align 8
   store double %arg, double* %arg.addr, align 8
   %0 = load double, double* %arg.addr, align 8

test/CodeGen/PowerPC/vsx.ll

 ; CHECK-FISL-DAG: addi [[R2:[0-9]+]], 1, -16
 ; CHECK-FISL-DAG: addi [[R3:[0-9]+]], 3, 2
 ; CHECK-FISL-DAG: std [[R1]], -8(1)
 ; CHECK-FISL-DAG: std [[R3]], -16(1)
 ; CHECK-FISL-DAG: lxvd2x 0, 0, [[R2]]
 ; CHECK-FISL: blr
 
 ; CHECK-LE-LABEL: @test80
-; CHECK-LE-DAG: addi [[R1:[0-9]+]], 1, -16
+; CHECK-LE-DAG: mtvsrd [[R1:[0-9]+]], 3
 ; CHECK-LE-DAG: addi [[R2:[0-9]+]], {{[0-9]+}}, .LCPI
-; CHECK-LE-DAG: lxvd2x [[V1:[0-9]+]], 0, [[R1]]
+; CHECK-LE-DAG: xxswapd [[V1:[0-9]+]], [[R1]]
 ; CHECK-LE-DAG: lxvd2x [[V2:[0-9]+]], 0, [[R2]]
-; CHECK-LE-DAG: xxswapd 34, [[V1]]
+; CHECK-LE-DAG: xxspltd 34, [[V1]]
 ; CHECK-LE-DAG: xxswapd 35, [[V2]]
 ; CHECK-LE: vaddudm 2, 2, 3
 ; CHECK-LE: blr
 }
 
 define <2 x double> @test81(<4 x float> %b) {
   %w = bitcast <4 x float> %b to <2 x double>
   ret <2 x double> %w

test/CodeGen/PowerPC/vsx_scalar_ld_st.ll

 entry:
   %ff = alloca float, align 4
   %0 = load i32, i32* @i, align 4
   %conv = sitofp i32 %0 to float
   store volatile float %conv, float* %ff, align 4
   ret void
 ; CHECK-LABEL: @intToFlt
 ; CHECK: lxsiwax [[REGLD2:[0-9]+]],
-; FIXME: the below will change when the VSX form is implemented
-; CHECK: fcfids {{[0-9]}}, [[REGLD2]]
+; CHECK: xscvsxdsp {{[0-9]}}, [[REGLD2]]
 }
 
 ; Function Attrs: nounwind
 define void @dblToUInt() #0 {
 entry:
   %uiui = alloca i32, align 4
   %0 = load double, double* @d, align 8
   %conv = fptoui double %0 to i32

 entry:
   %ff = alloca float, align 4
   %0 = load i32, i32* @ui, align 4
   %conv = uitofp i32 %0 to float
   store volatile float %conv, float* %ff, align 4
   ret void
 ; CHECK-LABEL: @uIntToFlt
 ; CHECK: lxsiwzx [[REGLD4:[0-9]+]],
-; FIXME: the below will change when the VSX form is implemented
-; CHECK: fcfidus {{[0-9]+}}, [[REGLD4]]
+; CHECK: xscvuxdsp {{[0-9]+}}, [[REGLD4]]
 }
 
 ; Function Attrs: nounwind
 define void @dblToFloat() #0 {
 entry:
   %ff = alloca float, align 4
   %0 = load double, double* @d, align 8
   %conv = fptrunc double %0 to float

test/MC/Disassembler/PowerPC/vsx.txt

 0xf3 0x1f 0xd9 0x1c
 
 # CHECK: xscpsgndp 7, 63, 27
 0xf0 0xff 0xdd 0x84
 
 # CHECK: xscvdpsp 7, 27
 0xf0 0xe0 0xdc 0x24
 
+# CHECK: xscvdpspn 7, 27
+0xf0 0xe0 0xdc 0x2c
+
 # CHECK: xscvdpsxds 7, 27
 0xf0 0xe0 0xdd 0x60
 
 # CHECK: xscvdpsxws 7, 27
 0xf0 0xe0 0xd9 0x60
 
 # CHECK: xscvdpuxds 7, 27
 0xf0 0xe0 0xdd 0x20
 
 # CHECK: xscvdpuxws 7, 27
 0xf0 0xe0 0xd9 0x20
 
 # CHECK: xscvspdp 7, 27
 0xf0 0xe0 0xdd 0x24
 
+# CHECK: xscvspdpn 7, 27
+0xf0 0xe0 0xdd 0x2c
+
+# CHECK: xscvsxdsp 7, 27
+0xf0 0xe0 0xdc 0xe0
+
 # CHECK: xscvsxddp 7, 27
 0xf0 0xe0 0xdd 0xe0
 
+# CHECK: xscvuxdsp 7, 27
+0xf0 0xe0 0xdc 0xa0
+
 # CHECK: xscvuxddp 7, 27
 0xf0 0xe0 0xdd 0xa0
 
 # CHECK: xsdivsp 7, 63, 27
 0xf0 0xff 0xd8 0xc4
 
 # CHECK: xsdivdp 7, 63, 27
 0xf0 0xff 0xd9 0xc4

test/MC/PowerPC/vsx.s

 # CHECK-LE: xscmpudp 6, 63, 27                 # encoding: [0x1c,0xd9,0x1f,0xf3]
             xscmpudp 6, 63, 27
 # CHECK-BE: xscpsgndp 7, 63, 27                # encoding: [0xf0,0xff,0xdd,0x84]
 # CHECK-LE: xscpsgndp 7, 63, 27                # encoding: [0x84,0xdd,0xff,0xf0]
             xscpsgndp 7, 63, 27
 # CHECK-BE: xscvdpsp 7, 27                     # encoding: [0xf0,0xe0,0xdc,0x24]
 # CHECK-LE: xscvdpsp 7, 27                     # encoding: [0x24,0xdc,0xe0,0xf0]
             xscvdpsp 7, 27
+# CHECK-BE: xscvdpspn 7, 27                    # encoding: [0xf0,0xe0,0xdc,0x2c]
+# CHECK-LE: xscvdpspn 7, 27                    # encoding: [0x2c,0xdc,0xe0,0xf0]
+            xscvdpspn 7, 27
 # CHECK-BE: xscvdpsxds 7, 27                   # encoding: [0xf0,0xe0,0xdd,0x60]
 # CHECK-LE: xscvdpsxds 7, 27                   # encoding: [0x60,0xdd,0xe0,0xf0]
             xscvdpsxds 7, 27
 # CHECK-BE: xscvdpsxws 7, 27                   # encoding: [0xf0,0xe0,0xd9,0x60]
 # CHECK-LE: xscvdpsxws 7, 27                   # encoding: [0x60,0xd9,0xe0,0xf0]
             xscvdpsxws 7, 27
 # CHECK-BE: xscvdpuxds 7, 27                   # encoding: [0xf0,0xe0,0xdd,0x20]
 # CHECK-LE: xscvdpuxds 7, 27                   # encoding: [0x20,0xdd,0xe0,0xf0]
             xscvdpuxds 7, 27
 # CHECK-BE: xscvdpuxws 7, 27                   # encoding: [0xf0,0xe0,0xd9,0x20]
 # CHECK-LE: xscvdpuxws 7, 27                   # encoding: [0x20,0xd9,0xe0,0xf0]
             xscvdpuxws 7, 27
 # CHECK-BE: xscvspdp 7, 27                     # encoding: [0xf0,0xe0,0xdd,0x24]
 # CHECK-LE: xscvspdp 7, 27                     # encoding: [0x24,0xdd,0xe0,0xf0]
             xscvspdp 7, 27
+# CHECK-BE: xscvspdpn 7, 27                    # encoding: [0xf0,0xe0,0xdd,0x2c]
+# CHECK-LE: xscvspdpn 7, 27                    # encoding: [0x2c,0xdd,0xe0,0xf0]
+            xscvspdpn 7, 27
+# CHECK-BE: xscvsxdsp 7, 27                    # encoding: [0xf0,0xe0,0xdc,0xe0]
+# CHECK-LE: xscvsxdsp 7, 27                    # encoding: [0xe0,0xdc,0xe0,0xf0]
+            xscvsxdsp 7, 27
 # CHECK-BE: xscvsxddp 7, 27                    # encoding: [0xf0,0xe0,0xdd,0xe0]
 # CHECK-LE: xscvsxddp 7, 27                    # encoding: [0xe0,0xdd,0xe0,0xf0]
             xscvsxddp 7, 27
+# CHECK-BE: xscvuxdsp 7, 27                    # encoding: [0xf0,0xe0,0xdc,0xa0]
+# CHECK-LE: xscvuxdsp 7, 27                    # encoding: [0xa0,0xdc,0xe0,0xf0]
+            xscvuxdsp 7, 27
 # CHECK-BE: xscvuxddp 7, 27                    # encoding: [0xf0,0xe0,0xdd,0xa0]
 # CHECK-LE: xscvuxddp 7, 27                    # encoding: [0xa0,0xdd,0xe0,0xf0]
             xscvuxddp 7, 27
 # CHECK-BE: xsdivsp 7, 63, 27                  # encoding: [0xf0,0xff,0xd8,0xc4]
 # CHECK-LE: xsdivsp 7, 63, 27                  # encoding: [0xc4,0xd8,0xff,0xf0]
             xsdivsp 7, 63, 27
 # CHECK-BE: xsdivdp 7, 63, 27                  # encoding: [0xf0,0xff,0xd9,0xc4]
 # CHECK-LE: xsdivdp 7, 63, 27                  # encoding: [0xc4,0xd9,0xff,0xf0]