[PowerPC] Use fcti[dw] instructions in additional cases

[vddvss]

Summary

rG0f0330a78709 added support for lowering lrint and llrint intrinsics to
fcti[dw] instructions on POWER8 and later processors. This patch extends
support for lowering these intrinsics on processors that support these
instruction.

This uses the existing fp-to-int infrastructure, and cleans up some of the logic
in the logic for that.

[nemanjai]

Please refer to https://reviews.llvm.org/D69949 that is about to be committed. I haven't looked at this patch but it seems related/overlapping. If there is an unforeseen issue with the other patch, please comment there.

[jhibbits]

The conflict I see is the other patch uses VSX for the operations, but this uses instructions that have been around as far back as the PPC970.

[nemanjai]

In D71287#1797468, @jhibbits wrote:

The conflict I see is the other patch uses VSX for the operations, but this uses instructions that have been around as far back as the PPC970.

Fair enough. I keep forgetting that recent server CPUs are not the only things that matter in the PPC world. I think we should handle lrint and llrint here and handle the rest of the nodes in the other patch. In fact, I think both patches can go in as they don't really conflict each other (save for some test case changes). So I imagine you'll just need to rebase this once I commit the other one.

[nemanjai]

I have pushed the other patch now. Please rebase this patch against ToT.

[vddvss]

Updated differential. This moves the actions to near where D69949 had them, guards them with TM.Options.UnsafeFPMath (I can't believe I didn't do that in the first patch), and changes tests accordingly.

That said, the fc[tf]i* instructions go back to at least ISA 2.02, but the PPC.td file only lists POWER7 and later as having fpcvt instructions, so right now this patch would only add the instructions for POWER7, which has fpcvt but not direct move. I think determining what processors support the instructions would require a bit of archaeology, so I'm wondering if we should even bother with this?

[qiucf]

As I see, this patch tries to extend use of fcti* instructions into subtargets without direct-move. If so, this looks reasonable, but needs change/rebase since D81537 did refactor in this part conflicting with this.

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
       /// and f64 value containing the FP representation of the integer that
       /// was temporarily in the f64 operand.
       FCFID,
 
       /// Newer FCFID[US] integer-to-floating-point conversion instructions for
       /// unsigned integers and single-precision outputs.
       FCFIDU, FCFIDS, FCFIDUS,
 
+      /// FCTI[DW] - The FCTID and FCTIW instructions, taking an f32 or f64
+      /// operand, producing an f64 value containing the integer representation
+      /// of that FP value, rounded using the current rounding mode.
+      FCTID, FCTIW,
+
       /// FCTI[D,W]Z - The FCTIDZ and FCTIWZ instructions, taking an f32 or f64
       /// operand, producing an f64 value containing the integer representation
-      /// of that FP value.
+      /// of that FP value, rounding it toward zero.
       FCTIDZ, FCTIWZ,
 
       /// Newer FCTI[D,W]UZ floating-point-to-integer conversion instructions for
       /// unsigned integers with round toward zero.
       FCTIDUZ, FCTIWUZ,
 
       /// Floating-point-to-interger conversion instructions
       FP_TO_UINT_IN_VSR, FP_TO_SINT_IN_VSR,
private:
     }
 
     bool canReuseLoadAddress(SDValue Op, EVT MemVT, ReuseLoadInfo &RLI,
                              SelectionDAG &DAG,
                              ISD::LoadExtType ET = ISD::NON_EXTLOAD) const;
     void spliceIntoChain(SDValue ResChain, SDValue NewResChain,
                          SelectionDAG &DAG) const;
 
+    SDValue getPPCISDNodeForFP_TO_INT(SDValue Op, SelectionDAG &DAG,
+                                      const SDLoc &dl) const;
     void LowerFP_TO_INTForReuse(SDValue Op, ReuseLoadInfo &RLI,
                                 SelectionDAG &DAG, const SDLoc &dl) const;
     SDValue LowerFP_TO_INTDirectMove(SDValue Op, SelectionDAG &DAG,
                                      const SDLoc &dl) const;
 
     bool directMoveIsProfitable(const SDValue &Op) const;
     SDValue LowerINT_TO_FPDirectMove(SDValue Op, SelectionDAG &DAG,
                                      const SDLoc &dl) const;

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

if (TM.Options.UnsafeFPMath) {
       setOperationAction(ISD::LLROUND, MVT::f64, Legal);
       setOperationAction(ISD::LLROUND, MVT::f32, Legal);
     }
   } else {
     setOperationAction(ISD::BITCAST, MVT::f32, Expand);
     setOperationAction(ISD::BITCAST, MVT::i32, Expand);
     setOperationAction(ISD::BITCAST, MVT::i64, Expand);
     setOperationAction(ISD::BITCAST, MVT::f64, Expand);
+
+    if (Subtarget.hasFPCVT() && TM.Options.UnsafeFPMath) {
+      if (isPPC64) {
+        setOperationAction(ISD::LLRINT, MVT::f32, Custom);
+        setOperationAction(ISD::LLRINT, MVT::f64, Custom);
+      }
+
+      setOperationAction(ISD::LRINT, MVT::f32, Custom);
+      setOperationAction(ISD::LRINT, MVT::f64, Custom);
+    }
   }
 
   // We cannot sextinreg(i1).  Expand to shifts.
   setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i1, Expand);
 
   // NOTE: EH_SJLJ_SETJMP/_LONGJMP supported here is NOT intended to support
   // SjLj exception handling but a light-weight setjmp/longjmp replacement to
   // support continuation, user-level threading, and etc.. As a result, no
const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::FIRST_NUMBER:    break;
   case PPCISD::FSEL:            return "PPCISD::FSEL";
   case PPCISD::XSMAXCDP:        return "PPCISD::XSMAXCDP";
   case PPCISD::XSMINCDP:        return "PPCISD::XSMINCDP";
   case PPCISD::FCFID:           return "PPCISD::FCFID";
   case PPCISD::FCFIDU:          return "PPCISD::FCFIDU";
   case PPCISD::FCFIDS:          return "PPCISD::FCFIDS";
   case PPCISD::FCFIDUS:         return "PPCISD::FCFIDUS";
+  case PPCISD::FCTID:           return "PPCISD::FCTID";
+  case PPCISD::FCTIW:           return "PPCISD::FCTIW";
   case PPCISD::FCTIDZ:          return "PPCISD::FCTIDZ";
   case PPCISD::FCTIWZ:          return "PPCISD::FCTIWZ";
   case PPCISD::FCTIDUZ:         return "PPCISD::FCTIDUZ";
   case PPCISD::FCTIWUZ:         return "PPCISD::FCTIWUZ";
   case PPCISD::FP_TO_UINT_IN_VSR:
                                 return "PPCISD::FP_TO_UINT_IN_VSR,";
   case PPCISD::FP_TO_SINT_IN_VSR:
                                 return "PPCISD::FP_TO_SINT_IN_VSR";
case ISD::SETLE:
     Cmp = DAG.getNode(ISD::FSUB, dl, CmpVT, RHS, LHS, Flags);
     if (Cmp.getValueType() == MVT::f32)   // Comparison is always 64-bits
       Cmp = DAG.getNode(ISD::FP_EXTEND, dl, MVT::f64, Cmp);
     return DAG.getNode(PPCISD::FSEL, dl, ResVT, Cmp, TV, FV);
   }
   return Op;
 }
 
-void PPCTargetLowering::LowerFP_TO_INTForReuse(SDValue Op, ReuseLoadInfo &RLI,
+SDValue PPCTargetLowering::getPPCISDNodeForFP_TO_INT(SDValue Op,
                                                      SelectionDAG &DAG,
                                                      const SDLoc &dl) const {
   assert(Op.getOperand(0).getValueType().isFloatingPoint());
   SDValue Src = Op.getOperand(0);
+
   if (Src.getValueType() == MVT::f32)
     Src = DAG.getNode(ISD::FP_EXTEND, dl, MVT::f64, Src);
 
-  SDValue Tmp;
-  switch (Op.getSimpleValueType().SimpleTy) {
-  default: llvm_unreachable("Unhandled FP_TO_INT type in custom expander!");
-  case MVT::i32:
-    Tmp = DAG.getNode(
-        Op.getOpcode() == ISD::FP_TO_SINT
-            ? PPCISD::FCTIWZ
-            : (Subtarget.hasFPCVT() ? PPCISD::FCTIWUZ : PPCISD::FCTIDZ),
-        dl, MVT::f64, Src);
+  MVT::SimpleValueType Ty = Op.getSimpleValueType().SimpleTy;
+  assert((Ty == MVT::i32 || Ty == MVT::i64) &&
+      "SimpleValueType for operand must be 32- or 64-bit integer.");
+
+  unsigned Opc = Op.getOpcode();
+  unsigned PPCOpc;
+
+  switch (Opc) {
+  default:
+    llvm_unreachable("Got invalid ISD node in FP_TO_INT custom expander!");
+  case ISD::FP_TO_SINT:
+    PPCOpc = Ty == MVT::i32 ? PPCISD::FCTIWZ : PPCISD::FCTIDZ;
     break;
-  case MVT::i64:
-    assert((Op.getOpcode() == ISD::FP_TO_SINT || Subtarget.hasFPCVT()) &&
-           "i64 FP_TO_UINT is supported only with FPCVT");
-    Tmp = DAG.getNode(Op.getOpcode()==ISD::FP_TO_SINT ? PPCISD::FCTIDZ :
-                                                        PPCISD::FCTIDUZ,
-                      dl, MVT::f64, Src);
+  case ISD::FP_TO_UINT:
+    if (Subtarget.hasFPCVT())
+      PPCOpc = Ty == MVT::i32 ? PPCISD::FCTIWUZ : PPCISD::FCTIDUZ;
+    else if (Ty == MVT::i32)
+      PPCOpc = PPCISD::FCTIDZ;
+    else
+      llvm_unreachable("i64 FP_TO_UINT is supported only with FPCVT");
+    break;
+  case ISD::LRINT:
+    assert(Subtarget.hasFPCVT() &&
+           "Custom expander for LRINT only supported with FPCVT");
+    PPCOpc = Ty == MVT::i32 ? PPCISD::FCTIW : PPCISD::FCTID;
     break;
-  }
+  case ISD::LLRINT:
+    assert(Subtarget.hasFPCVT() && Subtarget.has64BitSupport() &&
+           "Custom expander for LLRINT only supported on 64-bit with FPCVT");
+    PPCOpc = PPCISD::FCTID;
+    break;
+  }
+
+  return DAG.getNode(PPCOpc, dl, MVT::f64, Src);
 
+void PPCTargetLowering::LowerFP_TO_INTForReuse(SDValue Op, ReuseLoadInfo &RLI,
+                                               SelectionDAG &DAG,
+                                               const SDLoc &dl) const {
+  SDValue Tmp = getPPCISDNodeForFP_TO_INT(Op, DAG, dl);
+
   // Convert the FP value to an int value through memory.
   bool i32Stack = Op.getValueType() == MVT::i32 && Subtarget.hasSTFIWX() &&
     (Op.getOpcode() == ISD::FP_TO_SINT || Subtarget.hasFPCVT());
   SDValue FIPtr = DAG.CreateStackTemporary(i32Stack ? MVT::i32 : MVT::f64);
   int FI = cast<FrameIndexSDNode>(FIPtr)->getIndex();
   MachinePointerInfo MPI =
       MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI);
 

 }
 
 /// Custom lowers floating point to integer conversions to use
 /// the direct move instructions available in ISA 2.07 to avoid the
 /// need for load/store combinations.
 SDValue PPCTargetLowering::LowerFP_TO_INTDirectMove(SDValue Op,
                                                     SelectionDAG &DAG,
                                                     const SDLoc &dl) const {
-  assert(Op.getOperand(0).getValueType().isFloatingPoint());
-  SDValue Src = Op.getOperand(0);
-
-  if (Src.getValueType() == MVT::f32)
-    Src = DAG.getNode(ISD::FP_EXTEND, dl, MVT::f64, Src);
-
-  SDValue Tmp;
-  switch (Op.getSimpleValueType().SimpleTy) {
-  default: llvm_unreachable("Unhandled FP_TO_INT type in custom expander!");
-  case MVT::i32:
-    Tmp = DAG.getNode(
-        Op.getOpcode() == ISD::FP_TO_SINT
-            ? PPCISD::FCTIWZ
-            : (Subtarget.hasFPCVT() ? PPCISD::FCTIWUZ : PPCISD::FCTIDZ),
-        dl, MVT::f64, Src);
-    Tmp = DAG.getNode(PPCISD::MFVSR, dl, MVT::i32, Tmp);
-    break;
-  case MVT::i64:
-    assert((Op.getOpcode() == ISD::FP_TO_SINT || Subtarget.hasFPCVT()) &&
-           "i64 FP_TO_UINT is supported only with FPCVT");
-    Tmp = DAG.getNode(Op.getOpcode()==ISD::FP_TO_SINT ? PPCISD::FCTIDZ :
-                                                        PPCISD::FCTIDUZ,
-                      dl, MVT::f64, Src);
-    Tmp = DAG.getNode(PPCISD::MFVSR, dl, MVT::i64, Tmp);
-    break;
-  }
-  return Tmp;
+  SDValue Tmp = getPPCISDNodeForFP_TO_INT(Op, DAG, dl);
+  return DAG.getNode(PPCISD::MFVSR, dl, Op.getSimpleValueType().SimpleTy, Tmp);
 }
 
 SDValue PPCTargetLowering::LowerFP_TO_INT(SDValue Op, SelectionDAG &DAG,
                                           const SDLoc &dl) const {
 
   // FP to INT conversions are legal for f128.
   if (EnableQuadPrecision && (Op->getOperand(0).getValueType() == MVT::f128))
     return Op;
SDValue PPCTargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::EH_DWARF_CFA:       return LowerEH_DWARF_CFA(Op, DAG);
   case ISD::EH_SJLJ_SETJMP:     return lowerEH_SJLJ_SETJMP(Op, DAG);
   case ISD::EH_SJLJ_LONGJMP:    return lowerEH_SJLJ_LONGJMP(Op, DAG);
 
   case ISD::LOAD:               return LowerLOAD(Op, DAG);
   case ISD::STORE:              return LowerSTORE(Op, DAG);
   case ISD::TRUNCATE:           return LowerTRUNCATE(Op, DAG);
   case ISD::SELECT_CC:          return LowerSELECT_CC(Op, DAG);
+  case ISD::LRINT:
+  case ISD::LLRINT:
   case ISD::FP_TO_UINT:
   case ISD::FP_TO_SINT:         return LowerFP_TO_INT(Op, DAG, SDLoc(Op));
   case ISD::UINT_TO_FP:
   case ISD::SINT_TO_FP:         return LowerINT_TO_FP(Op, DAG);
   case ISD::FLT_ROUNDS_:        return LowerFLT_ROUNDS_(Op, DAG);
 
   // Lower 64-bit shifts.
   case ISD::SHL_PARTS:          return LowerSHL_PARTS(Op, DAG);

llvm/lib/Target/PowerPC/PPCInstr64Bit.td

 
 let PPC970_Unit = 3, hasSideEffects = 0,
     Uses = [RM] in {  // FPU Operations.
 defm FCFID  : XForm_26r<63, 846, (outs f8rc:$frD), (ins f8rc:$frB),
                         "fcfid", "$frD, $frB", IIC_FPGeneral,
                         [(set f64:$frD, (PPCfcfid f64:$frB))]>, isPPC64;
 defm FCTID  : XForm_26r<63, 814, (outs f8rc:$frD), (ins f8rc:$frB),
                         "fctid", "$frD, $frB", IIC_FPGeneral,
-                        []>, isPPC64;
+                        [(set f64:$frD, (PPCfctid f64:$frB))]>, isPPC64;
 defm FCTIDU : XForm_26r<63, 942, (outs f8rc:$frD), (ins f8rc:$frB),
                         "fctidu", "$frD, $frB", IIC_FPGeneral,
                         []>, isPPC64;
 defm FCTIDZ : XForm_26r<63, 815, (outs f8rc:$frD), (ins f8rc:$frB),
                         "fctidz", "$frD, $frB", IIC_FPGeneral,
                         [(set f64:$frD, (PPCfctidz f64:$frB))]>, isPPC64;
 
 defm FCFIDU  : XForm_26r<63, 974, (outs f8rc:$frD), (ins f8rc:$frB),

llvm/lib/Target/PowerPC/PPCInstrInfo.td

 
 def PPCfre    : SDNode<"PPCISD::FRE",     SDTFPUnaryOp, []>;
 def PPCfrsqrte: SDNode<"PPCISD::FRSQRTE", SDTFPUnaryOp, []>;
 
 def PPCfcfid  : SDNode<"PPCISD::FCFID",   SDTFPUnaryOp, []>;
 def PPCfcfidu : SDNode<"PPCISD::FCFIDU",  SDTFPUnaryOp, []>;
 def PPCfcfids : SDNode<"PPCISD::FCFIDS",  SDTFPRoundOp, []>;
 def PPCfcfidus: SDNode<"PPCISD::FCFIDUS", SDTFPRoundOp, []>;
+def PPCfctid  : SDNode<"PPCISD::FCTID",  SDTFPUnaryOp, []>;
+def PPCfctiw  : SDNode<"PPCISD::FCTIW",  SDTFPUnaryOp, []>;
 def PPCfctidz : SDNode<"PPCISD::FCTIDZ", SDTFPUnaryOp, []>;
 def PPCfctiwz : SDNode<"PPCISD::FCTIWZ", SDTFPUnaryOp, []>;
 def PPCfctiduz: SDNode<"PPCISD::FCTIDUZ",SDTFPUnaryOp, []>;
 def PPCfctiwuz: SDNode<"PPCISD::FCTIWUZ",SDTFPUnaryOp, []>;
 
 def PPCcv_fp_to_uint_in_vsr:
     SDNode<"PPCISD::FP_TO_UINT_IN_VSR", SDT_PPCcv_fp_to_int, []>;
 def PPCcv_fp_to_sint_in_vsr:
def FTDIV: XForm_17<63, 128, (outs crrc:$crD), (ins f8rc:$fA, f8rc:$fB),
                       "ftdiv $crD, $fA, $fB", IIC_FPCompare>;
 def FTSQRT: XForm_17a<63, 160, (outs crrc:$crD), (ins f8rc:$fB),
                       "ftsqrt $crD, $fB", IIC_FPCompare>;
 
 let Uses = [RM] in {
   let hasSideEffects = 0 in {
   defm FCTIW  : XForm_26r<63, 14, (outs f8rc:$frD), (ins f8rc:$frB),
                           "fctiw", "$frD, $frB", IIC_FPGeneral,
-                          []>;
+                          [(set f64:$frD, (PPCfctiw f64:$frB))]>;
   defm FCTIWU  : XForm_26r<63, 142, (outs f8rc:$frD), (ins f8rc:$frB),
                           "fctiwu", "$frD, $frB", IIC_FPGeneral,
                           []>;
   defm FCTIWZ : XForm_26r<63, 15, (outs f8rc:$frD), (ins f8rc:$frB),
                           "fctiwz", "$frD, $frB", IIC_FPGeneral,
                           [(set f64:$frD, (PPCfctiwz f64:$frB))]>;
 
   defm FRSP   : XForm_26r<63, 12, (outs f4rc:$frD), (ins f8rc:$frB),

llvm/test/CodeGen/PowerPC/llrint-conv.ll

-; RUN: llc < %s -mtriple=powerpc64le | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le \
+; RUN: --enable-unsafe-fp-math < %s | FileCheck --check-prefix=CHECK-FAST %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64 --enable-unsafe-fp-math \
+; RUN: -mcpu=pwr7 < %s | FileCheck --check-prefix=CHECK-FAST-PWR7 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64 --enable-unsafe-fp-math \
+; RUN:     -mattr=-fpcvt,-direct-move < %s \
+; RUN: | FileCheck --check-prefix=CHECK-FAST-NOFPCVT %s
 
 ; CHECK-LABEL:  testmsws:
 ; CHECK:              bl      llrintf
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      llrintf
 define signext i32 @testmsws(float %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.f32(float %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsxs:
 ; CHECK:              bl      llrintf
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      llrintf
 define i64 @testmsxs(float %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.f32(float %x)
   ret i64 %0
 }
 
 ; CHECK-LABEL: testmswd:
 ; CHECK:              bl      llrint
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      llrint
 define signext i32 @testmswd(double %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.f64(double %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsxd:
 ; CHECK:              bl      llrint
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      llrint
 define i64 @testmsxd(double %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.f64(double %x)
   ret i64 %0
 }
 
 ; CHECK-LABEL: testmswl:
 ; CHECK:              bl      llrintl
+; CHECK-FAST:         bl      llrintl
+; CHECK-FAST-PWR7:    bl      llrintl
+; CHECK-FAST-NOFPCVT: bl      llrintl
 define signext i32 @testmswl(ppc_fp128 %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.ppcf128(ppc_fp128 %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsll:
 ; CHECK:              bl      llrintl
+; CHECK-FAST:         bl      llrintl
+; CHECK-FAST-PWR7:    bl      llrintl
+; CHECK-FAST-NOFPCVT: bl      llrintl
 define i64 @testmsll(ppc_fp128 %x) {
 entry:
   %0 = tail call i64 @llvm.llrint.ppcf128(ppc_fp128 %x)
   ret i64 %0
 }
 
 declare i64 @llvm.llrint.f32(float) nounwind readnone
 declare i64 @llvm.llrint.f64(double) nounwind readnone
 declare i64 @llvm.llrint.ppcf128(ppc_fp128) nounwind readnone

llvm/test/CodeGen/PowerPC/lrint-conv.ll

-; RUN: llc < %s -mtriple=powerpc64le | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le \
+; RUN: --enable-unsafe-fp-math < %s | FileCheck --check-prefix=CHECK-FAST %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64 --enable-unsafe-fp-math \
+; RUN: -mcpu=pwr7 < %s | FileCheck --check-prefix=CHECK-FAST-PWR7 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64 --enable-unsafe-fp-math \
+; RUN:     -mattr=-fpcvt,-direct-move < %s \
+; RUN: | FileCheck --check-prefix=CHECK-FAST-NOFPCVT %s
 
 ; CHECK-LABEL: testmsws:
 ; CHECK:              bl      lrintf
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      lrintf
 define signext i32 @testmsws(float %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.f32(float %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsxs:
 ; CHECK:              bl      lrintf
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      lrintf
 define i64 @testmsxs(float %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.f32(float %x)
   ret i64 %0
 }
 
 ; CHECK-LABEL: testmswd:
 ; CHECK:              bl      lrint
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      lrint
 define signext i32 @testmswd(double %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.f64(double %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsxd:
 ; CHECK:              bl      lrint
+; CHECK-FAST:         fctid 0, 1
+; CHECK-FAST-PWR7:    fctid 0, 1
+; CHECK-FAST-NOFPCVT: bl      lrint
 define i64 @testmsxd(double %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.f64(double %x)
   ret i64 %0
 }
 
 ; CHECK-LABEL: testmswl:
 ; CHECK:              bl      lrintl
+; CHECK-FAST:         bl      lrintl
+; CHECK-FAST-PWR7:    bl      lrintl
+; CHECK-FAST-NOFPCVT: bl      lrintl
 define signext i32 @testmswl(ppc_fp128 %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.ppcf128(ppc_fp128 %x)
   %conv = trunc i64 %0 to i32
   ret i32 %conv
 }
 
 ; CHECK-LABEL: testmsll:
 ; CHECK:              bl      lrintl
+; CHECK-FAST:         bl      lrintl
+; CHECK-FAST-PWR7:    bl      lrintl
+; CHECK-FAST-NOFPCVT: bl      lrintl
 define i64 @testmsll(ppc_fp128 %x) {
 entry:
   %0 = tail call i64 @llvm.lrint.i64.ppcf128(ppc_fp128 %x)
   ret i64 %0
 }
 
 declare i64 @llvm.lrint.i64.f32(float) nounwind readnone
 declare i64 @llvm.lrint.i64.f64(double) nounwind readnone
 declare i64 @llvm.lrint.i64.ppcf128(ppc_fp128) nounwind readnone