[ARM] Generate 8.1-m CSINC, CSNEG and CSINV instructions.

[dmgreen]

Summary

Arm 8.1-M adds a number of related CSEL instructions, including CSINC, CSNEG and CSINV. These choose between two values given the content in CPSR and a condition, performing an increment, negation or inverse of the false value.

This adds some selection for them, either from constant values or patterns. It doesn't include CSEL directly, which is currently not always making code better. It is still useful, but we will have to check more carefully where it should and shouldn't be used.

Code by Ranjeet Singh and Simon Tatham, with some modifications from me.

[efriedma]

Could you give a brief description of why this is better than just using an "it" block?

llvm/test/CodeGen/Thumb2/csel.ll

21 ↗

(On Diff #216172)

There's an opportunity to save another two bytes converting mvn->movs.

llvm/test/CodeGen/Thumb2/mve-vcmpfr.ll

769 ↗

(On Diff #216172)

This is cset, right? Why aren't we printing the alias?

[dmgreen]

Sorry for the delay. I found a problem which was proving difficult to reduce, but I'm now pretty sure that is just exposing some problem in loloops.

Could you give a brief description of why this is better than just using an "it" block?

The tests shows this comes up more than I had expected. In general, we are replacing a IT; MVN/ADD/RSB with a single CSINV/CSINC/CSNEG, which can also use zr and includes a "free move". For codesize the IT pair will be either 4 bytes or 6, depending on the instruction needed and what registers are used. The CS* will always be 4, and includes that free move. For cycle times, we are talking about cortex-m here so I would expect the IT may or may not be free depending on if the surrounding instructions can be dual issued. The CS* should always take a single cycle I would presume. So generally smaller or the same, which is what my other testing also shows.

llvm/test/CodeGen/Thumb2/mve-vcmpfr.ll

769 ↗

(On Diff #216172)

It looks like this is because the instruction has an implicit CPSR use, which the Alias isn't accounting for. It's only checking for 4 operands, but during codegen the MI instruction will have 5.

I'll split this into a separate patch. It's not a big change but it's probably better split out from here.

[dmgreen]

llvm/test/CodeGen/Thumb2/csel.ll

21 ↗

(On Diff #216172)

D66701 for this one.

[dmgreen]

Rebase and some extra csel tests

[dmgreen]

Ping

[samparker]

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

46 ↗

(On Diff #217016)

Freaky way to control code gen! Do we really need this if code size is better and performance is on par or better..?

llvm/lib/Target/ARM/ARMISelLowering.cpp

4829 ↗

(On Diff #217016)

Reduce the nesting here.

4853 ↗

(On Diff #217016)

It's not obvious to me why CSINC is excluded here, could you add a comment?

4854 ↗

(On Diff #217016)

But FalseVal = TrueVal?

[dmgreen]

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

46 ↗

(On Diff #217016)

I think you are right. I've removed the option.

llvm/lib/Target/ARM/ARMISelLowering.cpp

4854 ↗

(On Diff #217016)

It does it the other way around in the future (D66701). I've moved that change here, and I think the swap makes it obvious that we are inverting the two conditions, even if we are about to throw one away.

[samparker]

Cheers. Would you mind adding/changing a couple of tests in csel.ll so that we test conditions other than sgt?

[dmgreen]

More tests!

[samparker]

Cheers! LGTM.

llvm/trunk/lib/Target/ARM/ARMBaseInstrInfo.cpp

for (unsigned IO = 0, EO = Instr.getNumOperands();
       bool IsInstrVSel = true;
       switch (Instr.getOpcode()) {
       default:
         IsInstrVSel = false;
         CC = (ARMCC::CondCodes)Instr.getOperand(IO - 1).getImm();
         break;
       case ARM::VSELEQD:
       case ARM::VSELEQS:
+      case ARM::VSELEQH:
         CC = ARMCC::EQ;
         break;
       case ARM::VSELGTD:
       case ARM::VSELGTS:
+      case ARM::VSELGTH:
         CC = ARMCC::GT;
         break;
       case ARM::VSELGED:
       case ARM::VSELGES:
+      case ARM::VSELGEH:
         CC = ARMCC::GE;
         break;
-      case ARM::VSELVSS:
       case ARM::VSELVSD:
+      case ARM::VSELVSS:
+      case ARM::VSELVSH:
         CC = ARMCC::VS;
         break;
       }
 
       if (SubAdd) {
         // If we have SUB(r1, r2) and CMP(r2, r1), the condition code based
         // on CMP needs to be updated to be based on SUB.
         // If we have ADD(r1, r2, X) and CMP(r1, r2), the condition code also

llvm/trunk/lib/Target/ARM/ARMISelLowering.h

enum NodeType : unsigned {
 
       // Vector bitwise select
       VBSL,
 
       // Pseudo-instruction representing a memory copy using ldm/stm
       // instructions.
       MEMCPY,
 
+      // V8.1MMainline condition select
+      CSINV, // Conditional select invert.
+      CSNEG, // Conditional select negate.
+      CSINC, // Conditional select increment.
+
       // Vector load N-element structure to all lanes:
       VLD1DUP = ISD::FIRST_TARGET_MEMORY_OPCODE,
       VLD2DUP,
       VLD3DUP,
       VLD4DUP,
 
       // NEON loads with post-increment base updates:
       VLD1_UPD,

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

const char *ARMTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case ARMISD::VST3_UPD:      return "ARMISD::VST3_UPD";
   case ARMISD::VST4_UPD:      return "ARMISD::VST4_UPD";
   case ARMISD::VST2LN_UPD:    return "ARMISD::VST2LN_UPD";
   case ARMISD::VST3LN_UPD:    return "ARMISD::VST3LN_UPD";
   case ARMISD::VST4LN_UPD:    return "ARMISD::VST4LN_UPD";
   case ARMISD::WLS:           return "ARMISD::WLS";
   case ARMISD::LE:            return "ARMISD::LE";
   case ARMISD::LOOP_DEC:      return "ARMISD::LOOP_DEC";
+  case ARMISD::CSINV:         return "ARMISD::CSINV";
+  case ARMISD::CSNEG:         return "ARMISD::CSNEG";
+  case ARMISD::CSINC:         return "ARMISD::CSINC";
   }
   return nullptr;
 }
 
 EVT ARMTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &,
                                           EVT VT) const {
   if (!VT.isVector())
     return getPointerTy(DL);
if (isNullConstant(LowerSatConstant)) {
       return DAG.getNode(ISD::OR, dl, VT, SatValue, ShiftV);
   }
 
   SDValue LHS = Op.getOperand(0);
   SDValue RHS = Op.getOperand(1);
   ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(4))->get();
   SDValue TrueVal = Op.getOperand(2);
   SDValue FalseVal = Op.getOperand(3);
+  ConstantSDNode *CFVal = dyn_cast<ConstantSDNode>(FalseVal);
+  ConstantSDNode *CTVal = dyn_cast<ConstantSDNode>(TrueVal);
+
+  if (Subtarget->hasV8_1MMainlineOps() && CFVal && CTVal &&
+      LHS.getValueType() == MVT::i32 && RHS.getValueType() == MVT::i32) {
+    unsigned TVal = CTVal->getZExtValue();
+    unsigned FVal = CFVal->getZExtValue();
+    unsigned Opcode = 0;
+
+    if (TVal == ~FVal) {
+      Opcode = ARMISD::CSINV;
+    } else if (TVal == ~FVal + 1) {
+      Opcode = ARMISD::CSNEG;
+    } else if (TVal + 1 == FVal) {
+      Opcode = ARMISD::CSINC;
+    } else if (TVal == FVal + 1) {
+      Opcode = ARMISD::CSINC;
+      std::swap(TrueVal, FalseVal);
+      std::swap(TVal, FVal);
+      CC = ISD::getSetCCInverse(CC, true);
+    }
+
+    if (Opcode) {
+      // Attempt to use ZR checking TVal is 0, possibly inverting the condition
+      // to get there. CSINC not is invertable like the other two (~(~a) == a,
+      // -(-a) == a, but (a+1)+1 != a).
+      if (FVal == 0 && Opcode != ARMISD::CSINC) {
+        std::swap(TrueVal, FalseVal);
+        std::swap(TVal, FVal);
+        CC = ISD::getSetCCInverse(CC, true);
+      }
+      if (TVal == 0)
+        TrueVal = DAG.getRegister(ARM::ZR, MVT::i32);
+
+      // Drops F's value because we can get it by inverting/negating TVal.
+      FalseVal = TrueVal;
+
+      SDValue ARMcc;
+      SDValue Cmp = getARMCmp(LHS, RHS, CC, ARMcc, DAG, dl);
+      EVT VT = TrueVal.getValueType();
+      return DAG.getNode(Opcode, dl, VT, TrueVal, FalseVal, ARMcc, Cmp);
+    }
+  }
 
   if (isUnsupportedFloatingType(LHS.getValueType())) {
     DAG.getTargetLoweringInfo().softenSetCCOperands(
         DAG, LHS.getValueType(), LHS, RHS, CC, dl, LHS, RHS);
 
     // If softenSetCCOperands only returned one value, we should compare it to
     // zero.
     if (!RHS.getNode()) {

llvm/trunk/lib/Target/ARM/ARMInstrFormats.td

 // Same as cc_out except it defaults to setting CPSR.
 def s_cc_out : OptionalDefOperand<OtherVT, (ops CCR), (ops (i32 CPSR))> {
   let EncoderMethod = "getCCOutOpValue";
   let PrintMethod = "printSBitModifierOperand";
   let ParserMatchClass = CCOutOperand;
   let DecoderMethod = "DecodeCCOutOperand";
 }
 
+// Transform to generate the inverse of a condition code during ISel
+def inv_cond_XFORM : SDNodeXForm<imm, [{
+  ARMCC::CondCodes CC = static_cast<ARMCC::CondCodes>(N->getZExtValue());
+  return CurDAG->getTargetConstant(ARMCC::getOppositeCondition(CC), SDLoc(N),
+                                   MVT::i32);
+}]>;
+
 // VPT predicate
 
 def VPTPredNOperand : AsmOperandClass {
   let Name = "VPTPredN";
   let PredicateMethod = "isVPTPred";
 }
 def VPTPredROperand : AsmOperandClass {
   let Name = "VPTPredR";

llvm/trunk/lib/Target/ARM/ARMInstrInfo.td

 def SDT_ARMLoLoop : SDTypeProfile<0, 2, [SDTCisVT<0, i32>,
                                          SDTCisVT<1, OtherVT>]>;
 
 def ARMSmlald        : SDNode<"ARMISD::SMLALD", SDT_LongMac>;
 def ARMSmlaldx       : SDNode<"ARMISD::SMLALDX", SDT_LongMac>;
 def ARMSmlsld        : SDNode<"ARMISD::SMLSLD", SDT_LongMac>;
 def ARMSmlsldx       : SDNode<"ARMISD::SMLSLDX", SDT_LongMac>;
 
+def SDT_ARMCSel      : SDTypeProfile<1, 3,
+                                   [SDTCisSameAs<0, 1>,
+                                    SDTCisSameAs<0, 2>,
+                                    SDTCisInt<3>,
+                                    SDTCisVT<3, i32>]>;
+
+def ARMcsinv         : SDNode<"ARMISD::CSINV", SDT_ARMCSel, [SDNPOptInGlue]>;
+def ARMcsneg         : SDNode<"ARMISD::CSNEG", SDT_ARMCSel, [SDNPOptInGlue]>;
+def ARMcsinc         : SDNode<"ARMISD::CSINC", SDT_ARMCSel, [SDNPOptInGlue]>;
+
 def SDT_MulHSR       : SDTypeProfile<1, 3, [SDTCisVT<0,i32>,
                                             SDTCisSameAs<0, 1>,
                                             SDTCisSameAs<0, 2>,
                                             SDTCisSameAs<0, 3>]>;
 
 def ARMsmmlar      : SDNode<"ARMISD::SMMLAR", SDT_MulHSR>;
 def ARMsmmlsr      : SDNode<"ARMISD::SMMLSR", SDT_MulHSR>;
 

llvm/trunk/lib/Target/ARM/ARMInstrThumb2.td

class CS<string iname, bits<4> opcode, list<dag> pattern=[]>
   let Uses = [CPSR];
 }
 
 def t2CSEL  : CS<"csel",  0b1000>;
 def t2CSINC : CS<"csinc", 0b1001>;
 def t2CSINV : CS<"csinv", 0b1010>;
 def t2CSNEG : CS<"csneg", 0b1011>;
 
+let Predicates = [HasV8_1MMainline] in {
+  def : T2Pat<(ARMcsinc GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm),
+              (t2CSINC GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm)>;
+  def : T2Pat<(ARMcsinv GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm),
+              (t2CSINV GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm)>;
+  def : T2Pat<(ARMcsneg GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm),
+              (t2CSNEG GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm)>;
+
+  multiclass ModifiedV8_1CSEL<Instruction Insn, dag modvalue> {
+    def : T2Pat<(ARMcmov modvalue, GPRwithZR:$tval, cmovpred:$imm),
+                (Insn GPRwithZR:$tval, GPRwithZR:$fval, imm0_31:$imm)>;
+    def : T2Pat<(ARMcmov GPRwithZR:$tval, modvalue, cmovpred:$imm),
+                (Insn GPRwithZR:$tval, GPRwithZR:$fval,
+                         (i32 (inv_cond_XFORM imm:$imm)))>;
+  }
+  defm : ModifiedV8_1CSEL<t2CSINC, (add rGPR:$fval, 1)>;
+  defm : ModifiedV8_1CSEL<t2CSINV, (xor rGPR:$fval, -1)>;
+  defm : ModifiedV8_1CSEL<t2CSNEG, (sub 0, rGPR:$fval)>;
+}
 
 // CS aliases.
 let Predicates = [HasV8_1MMainline] in {
   def : InstAlias<"csetm\t$Rd, $fcond",
                  (t2CSINV rGPR:$Rd, ZR, ZR, pred_noal_inv:$fcond)>;
 
   def : InstAlias<"cset\t$Rd, $fcond",
                  (t2CSINC rGPR:$Rd, ZR, ZR, pred_noal_inv:$fcond)>;

llvm/trunk/test/CodeGen/Thumb2/csel.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi %s -verify-machineinstrs -o - | FileCheck %s
 
 define i32 @csinc_const_65(i32 %a) {
 ; CHECK-LABEL: csinc_const_65:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    movs r1, #5
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt r1, #6
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinc r0, r1, r1, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 6, i32 5
   ret i32 %spec.select
 }
 
 define i32 @csinc_const_56(i32 %a) {
 ; CHECK-LABEL: csinc_const_56:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r1, #6
+; CHECK-NEXT:    movs r1, #5
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt r1, #5
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinc r0, r1, r1, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 5, i32 6
   ret i32 %spec.select
 }
 
 define i32 @csinc_const_zext(i32 %a) {
 ; CHECK-LABEL: csinc_const_zext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r1, #0
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt r1, #1
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinc r0, zr, zr, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = zext i1 %cmp to i32
   ret i32 %spec.select
 }
 
 define i32 @csinv_const_56(i32 %a) {
 ; CHECK-LABEL: csinv_const_56:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r1, #5
+; CHECK-NEXT:    mvn r1, #5
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    mvngt r1, #5
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinv r0, r1, r1, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 -6, i32 5
   ret i32 %spec.select
 }
 
 define i32 @csinv_const_65(i32 %a) {
 ; CHECK-LABEL: csinv_const_65:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mvn r1, #5
+; CHECK-NEXT:    movs r1, #5
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt r1, #5
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinv r0, r1, r1, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 5, i32 -6
   ret i32 %spec.select
 }
 
 define i32 @csinv_const_sext(i32 %a) {
 ; CHECK-LABEL: csinv_const_sext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r1, #0
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r1, #-1
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinv r0, zr, zr, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = sext i1 %cmp to i32
   ret i32 %spec.select
 }
 
 define i32 @csneg_const(i32 %a) {
 ; CHECK-LABEL: csneg_const:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov.w r1, #-1
+; CHECK-NEXT:    movs r1, #1
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt r1, #1
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csneg r0, r1, r1, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 1, i32 -1
   ret i32 %spec.select
 }
 
 define i32 @csneg_const_r(i32 %a) {
 ; CHECK-LABEL: csneg_const_r:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r1, #1
+; CHECK-NEXT:    mov.w r1, #-1
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r1, #-1
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csneg r0, r1, r1, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %spec.select = select i1 %cmp, i32 -1, i32 1
   ret i32 %spec.select
 }
 
 define i32 @csel_var(i32 %a, i32 %b, i32 %c) {
entry:
   %spec.select = select i1 %cmp, i32 %b, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csinc_var(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csinc_var:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it le
-; CHECK-NEXT:    addle r1, r2, #1
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinc r0, r1, r2, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %cplus1 = add nsw i32 %c, 1
   %spec.select = select i1 %cmp, i32 %b, i32 %cplus1
   ret i32 %spec.select
 }
 
 define i32 @csinc_swap_var(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csinc_swap_var:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    addgt r2, r1, #1
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csinc r0, r2, r1, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %bplus1 = add nsw i32 %b, 1
   %spec.select = select i1 %cmp, i32 %bplus1, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csinv_var(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csinv_var:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it le
-; CHECK-NEXT:    mvnle r1, r2
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csinv r0, r1, r2, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %cinv = xor i32 %c, -1
   %spec.select = select i1 %cmp, i32 %b, i32 %cinv
   ret i32 %spec.select
 }
 
 define i32 @csinv_swap_var(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csinv_swap_var:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    mvngt r2, r1
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csinv r0, r2, r1, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %binv = xor i32 %b, -1
   %spec.select = select i1 %cmp, i32 %binv, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_var(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_var:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it le
-; CHECK-NEXT:    rsble r1, r2, #0
-; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    csneg r0, r1, r2, gt
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %cneg = sub i32 0, %c
   %spec.select = select i1 %cmp, i32 %b, i32 %cneg
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_sgt(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_sgt:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    rsbgt r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_sge(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_sge:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #44
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    rsbgt r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sge i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_sle(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_sle:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #46
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    rsblt r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, ge
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sle i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_slt(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_slt:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    rsblt r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, ge
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp slt i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_ugt(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_ugt:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it hi
-; CHECK-NEXT:    rsbhi r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, ls
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp ugt i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_uge(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_uge:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #44
-; CHECK-NEXT:    it hi
-; CHECK-NEXT:    rsbhi r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, ls
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp uge i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_ule(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_ule:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #46
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    rsblo r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, hs
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp ule i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_ult(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_ult:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    rsblo r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, hs
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp ult i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_ne(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_ne:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it eq
-; CHECK-NEXT:    rsbeq r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, ne
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp eq i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csneg_swap_var_eq(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: csneg_swap_var_eq:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, #45
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    rsbne r2, r1, #0
-; CHECK-NEXT:    mov r0, r2
+; CHECK-NEXT:    csneg r0, r2, r1, eq
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp ne i32 %a, 45
   %bneg = sub i32 0, %b
   %spec.select = select i1 %cmp, i32 %bneg, i32 %c
   ret i32 %spec.select
 }
 
 define i32 @csinc_inplace(i32 %a, i32 %b) {
 ; CHECK-LABEL: csinc_inplace:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r1, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    addgt r0, #1
+; CHECK-NEXT:    csinc r0, r0, r0, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %b, 45
   %inc = zext i1 %cmp to i32
   %spec.select = add nsw i32 %inc, %a
   ret i32 %spec.select
 }
 
 define i32 @csinv_inplace(i32 %a, i32 %b) {
 ; CHECK-LABEL: csinv_inplace:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r1, #45
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    mvngt r0, r0
+; CHECK-NEXT:    csinv r0, r0, r0, le
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp = icmp sgt i32 %b, 45
   %sub = sext i1 %cmp to i32
   %xor = xor i32 %sub, %a
   ret i32 %xor
 }
 

llvm/trunk/test/CodeGen/Thumb2/mve-abs.ll

entry:
   %1 = sub nsw <4 x i32> zeroinitializer, %s1
   %2 = select <4 x i1> %0, <4 x i32> %1, <4 x i32> %s1
   ret <4 x i32> %2
 }
 
 define arm_aapcs_vfpcc <2 x i64> @abs_v2i64(<2 x i64> %s1) {
 ; CHECK-LABEL: abs_v2i64:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
-; CHECK-NEXT:    vmov r12, s2
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    vmov r4, s0
-; CHECK-NEXT:    rsbs.w r3, r12, #0
-; CHECK-NEXT:    sbc.w lr, r2, r0
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vmov q1, q0
+; CHECK-NEXT:    mov.w r12, #0
+; CHECK-NEXT:    vmov lr, s4
+; CHECK-NEXT:    vmov r0, s5
+; CHECK-NEXT:    rsbs.w r3, lr, #0
+; CHECK-NEXT:    sbc.w r2, r12, r0
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it mi
-; CHECK-NEXT:    movmi r1, #1
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it eq
-; CHECK-NEXT:    moveq lr, r0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    rsbs r5, r4, #0
-; CHECK-NEXT:    sbc.w r6, r2, r0
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it mi
-; CHECK-NEXT:    movmi r2, #1
-; CHECK-NEXT:    cmp r2, #0
+; CHECK-NEXT:    csinc r1, zr, zr, pl
+; CHECK-NEXT:    ands r1, r1, #1
 ; CHECK-NEXT:    itt eq
-; CHECK-NEXT:    moveq r6, r0
-; CHECK-NEXT:    moveq r5, r4
-; CHECK-NEXT:    vmov.32 q0[0], r5
-; CHECK-NEXT:    vmov.32 q0[1], r6
-; CHECK-NEXT:    cmp r1, #0
+; CHECK-NEXT:    moveq r2, r0
+; CHECK-NEXT:    moveq r3, lr
+; CHECK-NEXT:    vmov lr, s6
+; CHECK-NEXT:    vmov.32 q0[0], r3
+; CHECK-NEXT:    vmov r0, s7
+; CHECK-NEXT:    vmov.32 q0[1], r2
+; CHECK-NEXT:    rsbs.w r2, lr, #0
+; CHECK-NEXT:    sbc.w r3, r12, r0
+; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    csinc r1, zr, zr, pl
+; CHECK-NEXT:    ands r1, r1, #1
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    moveq r2, lr
+; CHECK-NEXT:    vmov.32 q0[2], r2
 ; CHECK-NEXT:    it eq
-; CHECK-NEXT:    moveq r3, r12
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], lr
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    moveq r3, r0
+; CHECK-NEXT:    vmov.32 q0[3], r3
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = icmp slt <2 x i64> %s1, zeroinitializer
   %1 = sub nsw <2 x i64> zeroinitializer, %s1
   %2 = select <2 x i1> %0, <2 x i64> %1, <2 x i64> %s1
   ret <2 x i64> %2
 }

llvm/trunk/test/CodeGen/Thumb2/mve-fmath.ll

 ; CHECK-NEXT:    vmovx.f16 s4, s7
 ; CHECK-NEXT:    vstr.16 s6, [sp, #12]
 ; CHECK-NEXT:    vstr.16 s8, [sp, #8]
 ; CHECK-NEXT:    vstr.16 s7, [sp, #4]
 ; CHECK-NEXT:    vstr.16 s4, [sp]
 ; CHECK-NEXT:    ldrb.w r0, [sp, #29]
 ; CHECK-NEXT:    vabs.f16 s4, s0
 ; CHECK-NEXT:    vneg.f16 s6, s4
+; CHECK-NEXT:    ldrb.w r1, [sp, #25]
+; CHECK-NEXT:    tst.w r0, #128
 ; CHECK-NEXT:    vabs.f16 s8, s1
-; CHECK-NEXT:    ands r0, r0, #128
+; CHECK-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-NEXT:    vneg.f16 s10, s8
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s4, s4, s6
-; CHECK-NEXT:    ldrb.w r1, [sp, #25]
+; CHECK-NEXT:    tst.w r1, #128
 ; CHECK-NEXT:    vmov r0, s4
 ; CHECK-NEXT:    vmovx.f16 s4, s0
-; CHECK-NEXT:    ands r1, r1, #128
+; CHECK-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-NEXT:    vabs.f16 s4, s4
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r1, #1
 ; CHECK-NEXT:    vneg.f16 s6, s4
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    vmovx.f16 s0, s3
+; CHECK-NEXT:    lsls r1, r1, #31
 ; CHECK-NEXT:    vseleq.f16 s4, s4, s6
-; CHECK-NEXT:    vabs.f16 s0, s0
+; CHECK-NEXT:    vmovx.f16 s0, s3
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vmov.16 q1[0], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #21]
 ; CHECK-NEXT:    vmov.16 q1[1], r1
-; CHECK-NEXT:    ands r0, r0, #128
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    vabs.f16 s0, s0
+; CHECK-NEXT:    tst.w r0, #128
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s8, s8, s10
 ; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vmovx.f16 s8, s1
 ; CHECK-NEXT:    vmov.16 q1[2], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #17]
 ; CHECK-NEXT:    vabs.f16 s8, s8
-; CHECK-NEXT:    ands r0, r0, #128
+; CHECK-NEXT:    tst.w r0, #128
 ; CHECK-NEXT:    vneg.f16 s10, s8
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s8, s8, s10
 ; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vabs.f16 s8, s2
 ; CHECK-NEXT:    vmov.16 q1[3], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #13]
 ; CHECK-NEXT:    vneg.f16 s10, s8
-; CHECK-NEXT:    ands r0, r0, #128
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    tst.w r0, #128
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s8, s8, s10
 ; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vmovx.f16 s8, s2
 ; CHECK-NEXT:    vmov.16 q1[4], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #9]
 ; CHECK-NEXT:    vabs.f16 s8, s8
 ; CHECK-NEXT:    vneg.f16 s2, s0
-; CHECK-NEXT:    ands r0, r0, #128
+; CHECK-NEXT:    tst.w r0, #128
 ; CHECK-NEXT:    vneg.f16 s10, s8
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s8, s8, s10
 ; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vabs.f16 s8, s3
 ; CHECK-NEXT:    vmov.16 q1[5], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #5]
 ; CHECK-NEXT:    vneg.f16 s10, s8
-; CHECK-NEXT:    ands r0, r0, #128
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    tst.w r0, #128
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s8, s8, s10
 ; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vmov.16 q1[6], r0
 ; CHECK-NEXT:    ldrb.w r0, [sp, #1]
-; CHECK-NEXT:    ands r0, r0, #128
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    tst.w r0, #128
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    lsls r0, r0, #31
 ; CHECK-NEXT:    vseleq.f16 s0, s0, s2
 ; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    vmov.16 q1[7], r0
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    add sp, #32
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = call fast <8 x half> @llvm.copysign.v8f16(<8 x half> %src1, <8 x half> %src2)

llvm/trunk/test/CodeGen/Thumb2/mve-minmax.ll

 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r2, s4
 ; CHECK-NEXT:    sbcs.w r1, r1, r12
 ; CHECK-NEXT:    vmov r12, s5
 ; CHECK-NEXT:    mov.w r1, #0
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    subs r2, r3, r2
 ; CHECK-NEXT:    sbcs.w r2, lr, r12
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov.32 q2[2], r1
 ; CHECK-NEXT:    vmov.32 q2[3], r1
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    pop {r7, pc}

 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r2, s4
 ; CHECK-NEXT:    sbcs.w r1, r1, r12
 ; CHECK-NEXT:    vmov r12, s5
 ; CHECK-NEXT:    mov.w r1, #0
 ; CHECK-NEXT:    it lo
 ; CHECK-NEXT:    movlo r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    subs r2, r3, r2
 ; CHECK-NEXT:    sbcs.w r2, lr, r12
 ; CHECK-NEXT:    it lo
 ; CHECK-NEXT:    movlo r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov.32 q2[2], r1
 ; CHECK-NEXT:    vmov.32 q2[3], r1
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    pop {r7, pc}

 ; CHECK-NEXT:    vmov r3, s4
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    sbcs.w r1, r1, r12
 ; CHECK-NEXT:    vmov r12, s1
 ; CHECK-NEXT:    mov.w r1, #0
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    subs r2, r3, r2
 ; CHECK-NEXT:    sbcs.w r2, lr, r12
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov.32 q2[2], r1
 ; CHECK-NEXT:    vmov.32 q2[3], r1
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    pop {r7, pc}

 ; CHECK-NEXT:    vmov r3, s4
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    sbcs.w r1, r1, r12
 ; CHECK-NEXT:    vmov r12, s1
 ; CHECK-NEXT:    mov.w r1, #0
 ; CHECK-NEXT:    it lo
 ; CHECK-NEXT:    movlo r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    subs r2, r3, r2
 ; CHECK-NEXT:    sbcs.w r2, lr, r12
 ; CHECK-NEXT:    it lo
 ; CHECK-NEXT:    movlo r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov.32 q2[2], r1
 ; CHECK-NEXT:    vmov.32 q2[3], r1
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    pop {r7, pc}

 ; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    vmov q4, q1
 ; CHECK-NEXT:    vmov q5, q0
 ; CHECK-NEXT:    vmov r0, r1, d9
 ; CHECK-NEXT:    vmov r2, r3, d11
 ; CHECK-NEXT:    bl __aeabi_dcmpgt
-; CHECK-NEXT:    mov r4, r0
-; CHECK-NEXT:    vmov r0, r1, d8
+; CHECK-NEXT:    vmov r12, r1, d8
+; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    vmov r2, r3, d10
-; CHECK-NEXT:    cmp r4, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r4, #1
-; CHECK-NEXT:    cmp r4, #0
 ; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r4, #-1
+; CHECK-NEXT:    movne r0, #1
+; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    csinv r4, zr, zr, eq
+; CHECK-NEXT:    mov r0, r12
 ; CHECK-NEXT:    bl __aeabi_dcmpgt
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    it ne
 ; CHECK-NEXT:    movne r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q0[0], r0
 ; CHECK-NEXT:    vmov.32 q0[1], r0
 ; CHECK-NEXT:    vmov.32 q0[2], r4
 ; CHECK-NEXT:    vmov.32 q0[3], r4
 ; CHECK-NEXT:    vbic q1, q5, q0
 ; CHECK-NEXT:    vand q0, q4, q0
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-NEXT:    pop {r4, pc}
 entry:
   %cmp = fcmp fast ogt <2 x double> %src2, %src1
   %0 = select <2 x i1> %cmp, <2 x double> %src2, <2 x double> %src1
   ret <2 x double> %0
 }

llvm/trunk/test/CodeGen/Thumb2/mve-pred-and.ll

 define arm_aapcs_vfpcc <2 x i64> @cmpeqz_v2i1(<2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: cmpeqz_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vorr q2, q0, q1
 ; CHECK-NEXT:    vmov r0, s9
 ; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s10
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s11
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q1, q1, q3
 ; CHECK-NEXT:    vand q0, q0, q3
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer

 ; CHECK-NEXT:    vmov r1, s5
 ; CHECK-NEXT:    vmov r2, s4
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s6
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s7
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s11
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s10
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vand q2, q2, q3
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer
   %c2 = icmp eq <2 x i64> %b, %c
   %o = and <2 x i1> %c1, %c2
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <2 x i64> @cmpeqr_v2i1(<2 x i64> %a, <2 x i64> %b, i64 %c) {
 ; CHECK-LABEL: cmpeqr_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r2, s5
 ; CHECK-NEXT:    vmov r3, s4
 ; CHECK-NEXT:    eors r2, r1
 ; CHECK-NEXT:    eors r3, r0
 ; CHECK-NEXT:    orrs r2, r3
-; CHECK-NEXT:    clz r2, r2
-; CHECK-NEXT:    lsrs r2, r2, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r2, #-1
+; CHECK-NEXT:    csinc r2, zr, zr, ne
+; CHECK-NEXT:    tst.w r2, #1
+; CHECK-NEXT:    csinv r2, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r2
 ; CHECK-NEXT:    vmov.32 q2[1], r2
 ; CHECK-NEXT:    vmov r2, s7
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s6
 ; CHECK-NEXT:    eors r0, r2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer
   %i = insertelement <2 x i64> undef, i64 %c, i32 0
   %sp = shufflevector <2 x i64> %i, <2 x i64> undef, <2 x i32> zeroinitializer
   %c2 = icmp eq <2 x i64> %b, %sp
   %o = and <2 x i1> %c1, %c2
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }

llvm/trunk/test/CodeGen/Thumb2/mve-pred-bitcast.ll

 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
+; CHECK-NEXT:    csinc r0, zr, zr, ne
 ; CHECK-NEXT:    orrs r1, r2
-; CHECK-NEXT:    clz r1, r1
-; CHECK-NEXT:    lsrs r1, r1, #5
+; CHECK-NEXT:    csinc r1, zr, zr, ne
+; CHECK-NEXT:    ands r1, r1, #1
 ; CHECK-NEXT:    it ne
 ; CHECK-NEXT:    mvnne r1, #1
 ; CHECK-NEXT:    bfi r1, r0, #0, #1
 ; CHECK-NEXT:    and r0, r1, #3
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp eq <2 x i64> %a, zeroinitializer
   %b = bitcast <2 x i1> %c to i2
   ret i2 %b
 }

llvm/trunk/test/CodeGen/Thumb2/mve-pred-build-var.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 
 define arm_aapcs_vfpcc <4 x i32> @build_var0_v4i1(i32 %s, i32 %t, <4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: build_var0_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #4
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #0, #4
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <4 x i1> zeroinitializer, i1 %c, i64 0
   %r = select <4 x i1> %vc, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %r
 }
 
 define arm_aapcs_vfpcc <4 x i32> @build_var3_v4i1(i32 %s, i32 %t, <4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: build_var3_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #12, #4
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #12, #4
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <4 x i1> zeroinitializer, i1 %c, i64 3
   %r = select <4 x i1> %vc, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %r
 }
 
 define arm_aapcs_vfpcc <4 x i32> @build_varN_v4i1(i32 %s, i32 %t, <4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: build_varN_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #4
-; CHECK-NEXT:    bfi r2, r0, #4, #4
-; CHECK-NEXT:    bfi r2, r0, #8, #4
-; CHECK-NEXT:    bfi r2, r0, #12, #4
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
+; CHECK-NEXT:    bfi r1, r0, #0, #4
+; CHECK-NEXT:    bfi r1, r0, #4, #4
+; CHECK-NEXT:    bfi r1, r0, #8, #4
+; CHECK-NEXT:    bfi r1, r0, #12, #4
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc1 = insertelement <4 x i1> undef, i1 %c, i64 0
   %vc4 = shufflevector <4 x i1> %vc1, <4 x i1> undef, <4 x i32> zeroinitializer
   %r = select <4 x i1> %vc4, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %r
 }
 
 
 define arm_aapcs_vfpcc <8 x i16> @build_var0_v8i1(i32 %s, i32 %t, <8 x i16> %a, <8 x i16> %b) {
 ; CHECK-LABEL: build_var0_v8i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #2
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #0, #2
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <8 x i1> zeroinitializer, i1 %c, i64 0
   %r = select <8 x i1> %vc, <8 x i16> %a, <8 x i16> %b
   ret <8 x i16> %r
 }
 
 define arm_aapcs_vfpcc <8 x i16> @build_var3_v8i1(i32 %s, i32 %t, <8 x i16> %a, <8 x i16> %b) {
 ; CHECK-LABEL: build_var3_v8i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #6, #2
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #6, #2
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <8 x i1> zeroinitializer, i1 %c, i64 3
   %r = select <8 x i1> %vc, <8 x i16> %a, <8 x i16> %b
   ret <8 x i16> %r
 }
 
 define arm_aapcs_vfpcc <8 x i16> @build_varN_v8i1(i32 %s, i32 %t, <8 x i16> %a, <8 x i16> %b) {
 ; CHECK-LABEL: build_varN_v8i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #2
-; CHECK-NEXT:    bfi r2, r0, #2, #2
-; CHECK-NEXT:    bfi r2, r0, #4, #2
-; CHECK-NEXT:    bfi r2, r0, #6, #2
-; CHECK-NEXT:    bfi r2, r0, #8, #2
-; CHECK-NEXT:    bfi r2, r0, #10, #2
-; CHECK-NEXT:    bfi r2, r0, #12, #2
-; CHECK-NEXT:    bfi r2, r0, #14, #2
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
+; CHECK-NEXT:    bfi r1, r0, #0, #2
+; CHECK-NEXT:    bfi r1, r0, #2, #2
+; CHECK-NEXT:    bfi r1, r0, #4, #2
+; CHECK-NEXT:    bfi r1, r0, #6, #2
+; CHECK-NEXT:    bfi r1, r0, #8, #2
+; CHECK-NEXT:    bfi r1, r0, #10, #2
+; CHECK-NEXT:    bfi r1, r0, #12, #2
+; CHECK-NEXT:    bfi r1, r0, #14, #2
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc1 = insertelement <8 x i1> undef, i1 %c, i64 0
   %vc4 = shufflevector <8 x i1> %vc1, <8 x i1> undef, <8 x i32> zeroinitializer
   %r = select <8 x i1> %vc4, <8 x i16> %a, <8 x i16> %b
   ret <8 x i16> %r
 }
 
 
 define arm_aapcs_vfpcc <16 x i8> @build_var0_v16i1(i32 %s, i32 %t, <16 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: build_var0_v16i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #1
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #0, #1
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <16 x i1> zeroinitializer, i1 %c, i64 0
   %r = select <16 x i1> %vc, <16 x i8> %a, <16 x i8> %b
   ret <16 x i8> %r
 }
 
 define arm_aapcs_vfpcc <16 x i8> @build_var3_v16i1(i32 %s, i32 %t, <16 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: build_var3_v16i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #3, #1
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    bfi r1, r0, #3, #1
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <16 x i1> zeroinitializer, i1 %c, i64 3
   %r = select <16 x i1> %vc, <16 x i8> %a, <16 x i8> %b
   ret <16 x i8> %r
 }
 
 define arm_aapcs_vfpcc <16 x i8> @build_varN_v16i1(i32 %s, i32 %t, <16 x i8> %a, <16 x i8> %b) {
 ; CHECK-LABEL: build_varN_v16i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    mov.w r0, #0
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r0, #1
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    rsbs r0, r0, #0
-; CHECK-NEXT:    bfi r2, r0, #0, #1
-; CHECK-NEXT:    bfi r2, r0, #1, #1
-; CHECK-NEXT:    bfi r2, r0, #2, #1
-; CHECK-NEXT:    bfi r2, r0, #3, #1
-; CHECK-NEXT:    bfi r2, r0, #4, #1
-; CHECK-NEXT:    bfi r2, r0, #5, #1
-; CHECK-NEXT:    bfi r2, r0, #6, #1
-; CHECK-NEXT:    bfi r2, r0, #7, #1
-; CHECK-NEXT:    bfi r2, r0, #8, #1
-; CHECK-NEXT:    bfi r2, r0, #9, #1
-; CHECK-NEXT:    bfi r2, r0, #10, #1
-; CHECK-NEXT:    bfi r2, r0, #11, #1
-; CHECK-NEXT:    bfi r2, r0, #12, #1
-; CHECK-NEXT:    bfi r2, r0, #13, #1
-; CHECK-NEXT:    bfi r2, r0, #14, #1
-; CHECK-NEXT:    bfi r2, r0, #15, #1
-; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
+; CHECK-NEXT:    bfi r1, r0, #0, #1
+; CHECK-NEXT:    bfi r1, r0, #1, #1
+; CHECK-NEXT:    bfi r1, r0, #2, #1
+; CHECK-NEXT:    bfi r1, r0, #3, #1
+; CHECK-NEXT:    bfi r1, r0, #4, #1
+; CHECK-NEXT:    bfi r1, r0, #5, #1
+; CHECK-NEXT:    bfi r1, r0, #6, #1
+; CHECK-NEXT:    bfi r1, r0, #7, #1
+; CHECK-NEXT:    bfi r1, r0, #8, #1
+; CHECK-NEXT:    bfi r1, r0, #9, #1
+; CHECK-NEXT:    bfi r1, r0, #10, #1
+; CHECK-NEXT:    bfi r1, r0, #11, #1
+; CHECK-NEXT:    bfi r1, r0, #12, #1
+; CHECK-NEXT:    bfi r1, r0, #13, #1
+; CHECK-NEXT:    bfi r1, r0, #14, #1
+; CHECK-NEXT:    bfi r1, r0, #15, #1
+; CHECK-NEXT:    vmsr p0, r1
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc1 = insertelement <16 x i1> undef, i1 %c, i64 0
   %vc4 = shufflevector <16 x i1> %vc1, <16 x i1> undef, <16 x i32> zeroinitializer
   %r = select <16 x i1> %vc4, <16 x i8> %a, <16 x i8> %b
   ret <16 x i8> %r
 }
 
 
 define arm_aapcs_vfpcc <2 x i64> @build_var0_v2i1(i32 %s, i32 %t, <2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: build_var0_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r2, #0
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r2, #1
-; CHECK-NEXT:    rsbs r0, r2, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vmov s8, r0
 ; CHECK-NEXT:    vldr s10, .LCPI9_0
 ; CHECK-NEXT:    vmov.f32 s9, s8
 ; CHECK-NEXT:    vmov.f32 s11, s10
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI9_0:
 ; CHECK-NEXT:    .long 0 @ float 0
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <2 x i1> zeroinitializer, i1 %c, i64 0
   %r = select <2 x i1> %vc, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %r
 }
 
 define arm_aapcs_vfpcc <2 x i64> @build_var1_v2i1(i32 %s, i32 %t, <2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: build_var1_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r2, #0
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r2, #1
-; CHECK-NEXT:    rsbs r0, r2, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vmov s10, r0
 ; CHECK-NEXT:    vldr s8, .LCPI10_0
 ; CHECK-NEXT:    vmov.f32 s9, s8
 ; CHECK-NEXT:    vmov.f32 s11, s10
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI10_0:
 ; CHECK-NEXT:    .long 0 @ float 0
 entry:
   %c = icmp ult i32 %s, %t
   %vc = insertelement <2 x i1> zeroinitializer, i1 %c, i64 1
   %r = select <2 x i1> %vc, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %r
 }
 
 define arm_aapcs_vfpcc <2 x i64> @build_varN_v2i1(i32 %s, i32 %t, <2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: build_varN_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    movs r2, #0
 ; CHECK-NEXT:    cmp r0, r1
-; CHECK-NEXT:    it lo
-; CHECK-NEXT:    movlo r2, #1
-; CHECK-NEXT:    rsbs r0, r2, #0
+; CHECK-NEXT:    csinc r0, zr, zr, hs
+; CHECK-NEXT:    and r0, r0, #1
+; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vdup.32 q2, r0
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp ult i32 %s, %t
   %vc1 = insertelement <2 x i1> undef, i1 %c, i64 0
   %vc4 = shufflevector <2 x i1> %vc1, <2 x i1> undef, <2 x i32> zeroinitializer
   %r = select <2 x i1> %vc4, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %r
 }

llvm/trunk/test/CodeGen/Thumb2/mve-pred-ext.ll

 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov r1, s1
 ; CHECK-NEXT:    sbcs.w r0, r2, r0
 ; CHECK-NEXT:    mov.w r0, #0
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    sbcs.w r1, r2, r1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r2, #1
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r2, #-1
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    vmov.32 q0[1], r2
+; CHECK-NEXT:    csinv r1, zr, zr, eq
+; CHECK-NEXT:    vmov.32 q0[0], r1
+; CHECK-NEXT:    vmov.32 q0[1], r1
 ; CHECK-NEXT:    vmov.32 q0[2], r0
 ; CHECK-NEXT:    vmov.32 q0[3], r0
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp sgt <2 x i64> %src, zeroinitializer
   %0 = sext <2 x i1> %c to <2 x i64>
   ret <2 x i64> %0
 }

 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    sbcs.w r1, r0, r1
 ; CHECK-NEXT:    mov.w r1, #0
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    sbcs.w r2, r0, r2
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #1
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q0[0], r0
 ; CHECK-NEXT:    vmov.32 q0[2], r1
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI7_0:
 ; CHECK-NEXT:    .long 1 @ 0x1

llvm/trunk/test/CodeGen/Thumb2/mve-pred-loadstore.ll

 define arm_aapcs_vfpcc void @store_v2i1(<2 x i1> *%dst, <2 x i64> %a) {
 ; CHECK-LE-LABEL: store_v2i1:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vmov r1, s1
 ; CHECK-LE-NEXT:    vmov r2, s0
 ; CHECK-LE-NEXT:    vmov r3, s2
 ; CHECK-LE-NEXT:    orrs r1, r2
 ; CHECK-LE-NEXT:    vmov r2, s3
-; CHECK-LE-NEXT:    clz r1, r1
-; CHECK-LE-NEXT:    lsrs r1, r1, #5
+; CHECK-LE-NEXT:    csinc r1, zr, zr, ne
 ; CHECK-LE-NEXT:    orrs r2, r3
-; CHECK-LE-NEXT:    clz r2, r2
-; CHECK-LE-NEXT:    lsrs r2, r2, #5
+; CHECK-LE-NEXT:    csinc r2, zr, zr, ne
+; CHECK-LE-NEXT:    ands r2, r2, #1
 ; CHECK-LE-NEXT:    it ne
 ; CHECK-LE-NEXT:    mvnne r2, #1
 ; CHECK-LE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-LE-NEXT:    and r1, r2, #3
 ; CHECK-LE-NEXT:    strb r1, [r0]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: store_v2i1:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vmov r1, s6
 ; CHECK-BE-NEXT:    vmov r2, s7
 ; CHECK-BE-NEXT:    vmov r3, s5
 ; CHECK-BE-NEXT:    orrs r1, r2
 ; CHECK-BE-NEXT:    vmov r2, s4
-; CHECK-BE-NEXT:    clz r1, r1
-; CHECK-BE-NEXT:    lsrs r1, r1, #5
+; CHECK-BE-NEXT:    csinc r1, zr, zr, ne
 ; CHECK-BE-NEXT:    orrs r2, r3
-; CHECK-BE-NEXT:    clz r2, r2
-; CHECK-BE-NEXT:    lsrs r2, r2, #5
+; CHECK-BE-NEXT:    csinc r2, zr, zr, ne
+; CHECK-BE-NEXT:    ands r2, r2, #1
 ; CHECK-BE-NEXT:    it ne
 ; CHECK-BE-NEXT:    mvnne r2, #1
 ; CHECK-BE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-BE-NEXT:    and r1, r2, #3
 ; CHECK-BE-NEXT:    strb r1, [r0]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp eq <2 x i64> %a, zeroinitializer
   store <2 x i1> %c, <2 x i1>* %dst
   ret void
 }

llvm/trunk/test/CodeGen/Thumb2/mve-pred-not.ll

 
 define arm_aapcs_vfpcc <2 x i64> @cmpeqz_v2i1(<2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: cmpeqz_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vbic q0, q0, q2
 ; CHECK-NEXT:    vand q1, q1, q2
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer
   %o = xor <2 x i1> %c1, <i1 -1, i1 -1>
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <2 x i64> @cmpeq_v2i1(<2 x i64> %a, <2 x i64> %b, <2 x i64> %c) {
 ; CHECK-LABEL: cmpeq_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vbic q0, q0, q2
 ; CHECK-NEXT:    vand q1, q1, q2
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer

llvm/trunk/test/CodeGen/Thumb2/mve-pred-or.ll

 
 define arm_aapcs_vfpcc <2 x i64> @cmpeqz_v2i1(<2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: cmpeqz_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s5
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s6
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s7
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vorr q2, q3, q2
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:

 ; CHECK-NEXT:    vmov r1, s5
 ; CHECK-NEXT:    vmov r2, s4
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s6
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s7
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s11
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s10
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vorr q2, q2, q3
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer
   %c2 = icmp eq <2 x i64> %b, %c
   %o = or <2 x i1> %c1, %c2
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 

llvm/trunk/test/CodeGen/Thumb2/mve-pred-xor.ll

 
 define arm_aapcs_vfpcc <2 x i64> @cmpeqz_v2i1(<2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: cmpeqz_v2i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s5
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s6
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s7
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    veor q2, q3, q2
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:

 ; CHECK-NEXT:    vmov r1, s5
 ; CHECK-NEXT:    vmov r2, s4
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s6
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s7
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s11
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s10
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[0], r0
 ; CHECK-NEXT:    vmov.32 q2[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q2[2], r0
 ; CHECK-NEXT:    vmov.32 q2[3], r0
 ; CHECK-NEXT:    veor q2, q2, q3
 ; CHECK-NEXT:    vbic q1, q1, q2
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vorr q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp eq <2 x i64> %a, zeroinitializer
   %c2 = icmp eq <2 x i64> %b, %c
   %o = xor <2 x i1> %c1, %c2
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 

llvm/trunk/test/CodeGen/Thumb2/mve-vcmp.ll

 ; CHECK-NEXT:    vmov r1, s1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q4[0], r0
 ; CHECK-NEXT:    vmov.32 q4[1], r0
 ; CHECK-NEXT:    vmov r0, s7
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s6
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q4[2], r0
 ; CHECK-NEXT:    vmov.32 q4[3], r0
 ; CHECK-NEXT:    vbic q0, q3, q4
 ; CHECK-NEXT:    vand q1, q2, q4
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    vpop {d8, d9}
 ; CHECK-NEXT:    bx lr
 entry:

 ; CHECK-NEXT:    vmov r1, s1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q4[0], r0
 ; CHECK-NEXT:    vmov.32 q4[1], r0
 ; CHECK-NEXT:    vmov r0, s7
 ; CHECK-NEXT:    eors r0, r1
 ; CHECK-NEXT:    vmov r1, s6
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q4[2], r0
 ; CHECK-NEXT:    vmov.32 q4[3], r0
 ; CHECK-NEXT:    vbic q0, q3, q4
 ; CHECK-NEXT:    vand q1, q2, q4
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    vpop {d8, d9}
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp eq <2 x i64> %src, %srcb
   %s = select <2 x i1> %c, <2 x i32> %a, <2 x i32> %b
   ret <2 x i32> %s
 }
 
 define arm_aapcs_vfpcc <2 x i32> @vcmp_multi_v2i32(<2 x i64> %a, <2 x i32> %b, <2 x i32> %c) {
 ; CHECK-LABEL: vcmp_multi_v2i32:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov lr, s10
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    subs r1, r0, r2
-; CHECK-NEXT:    asr.w r12, r0, #31
+; CHECK-NEXT:    vmov lr, s0
+; CHECK-NEXT:    subs.w r1, lr, r2
+; CHECK-NEXT:    asr.w r12, lr, #31
 ; CHECK-NEXT:    sbcs.w r1, r12, r2, asr #31
 ; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    vmov r2, s10
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r1
 ; CHECK-NEXT:    vmov.32 q3[1], r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    subs.w r2, r1, lr
+; CHECK-NEXT:    subs r0, r1, r2
 ; CHECK-NEXT:    asr.w r12, r1, #31
-; CHECK-NEXT:    sbcs.w r2, r12, lr, asr #31
+; CHECK-NEXT:    sbcs.w r0, r12, r2, asr #31
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r3, #1
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r3, #-1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    cmp.w lr, #0
+; CHECK-NEXT:    vmov.32 q3[2], r0
+; CHECK-NEXT:    vmov.32 q3[3], r0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    vmov.32 q4[0], r0
 ; CHECK-NEXT:    vmov.32 q4[1], r0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    vmov.32 q4[2], r0
+; CHECK-NEXT:    vmov.32 q4[3], r0
 ; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r1, #1
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
-; CHECK-NEXT:    vmov.32 q4[2], r1
-; CHECK-NEXT:    vmov.32 q3[2], r3
-; CHECK-NEXT:    vmov.32 q4[3], r1
-; CHECK-NEXT:    vmov.32 q3[3], r3
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q5[0], r0
 ; CHECK-NEXT:    vmov.32 q5[1], r0
 ; CHECK-NEXT:    vmov r0, s6
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q5[2], r0
 ; CHECK-NEXT:    vmov.32 q5[3], r0
 ; CHECK-NEXT:    vand q1, q5, q4
 ; CHECK-NEXT:    vand q1, q3, q1
 ; CHECK-NEXT:    vbic q0, q0, q1
 ; CHECK-NEXT:    vand q1, q2, q1
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}

llvm/trunk/test/CodeGen/Thumb2/mve-vcmpf.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve,+fullfp16 -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVE
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVEFP
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oeq_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 eq, q0, q1
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp oeq <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_one_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_one_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, q1
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp one <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ogt_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, q1
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ogt <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oge_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, q1
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp oge <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_olt_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q1, q0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp olt <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ole_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q1, q0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ole <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ueq_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, q1
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ueq <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_une_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_une_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s7
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ne, q0, q1
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp une <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ugt_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q1, q0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ugt <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uge_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q1, q0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp uge <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ult_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, q1
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ult <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ule_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, q1
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ule <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ord_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 lt, q0, q1
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ord <4 x float> %src, %src2
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uno_v4f32(<4 x float> %src, <4 x float> %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s5
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s6
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s7
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 lt, q0, q1

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
+; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s6
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    mov.w r0, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s7
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
+; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s6
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    mov.w r0, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s7
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmp.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s4
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s8
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s12
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s5
-; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s12, s8
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov.16 q4[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s22, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q4[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s13, s9
-; CHECK-MVE-NEXT:    vmovx.f16 s4, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s5
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s13
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s6
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s2
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
+; CHECK-MVE-NEXT:    vseleq.f16 s20, s14, s10
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s6
 ; CHECK-MVE-NEXT:    vcmpe.f16 s22, s20
 ; CHECK-MVE-NEXT:    vmov.16 q4[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmovx.f16 s22, s14
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s22, s20
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s7
 ; CHECK-MVE-NEXT:    vmov r1, s20
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov.16 q4[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s4
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s20, s15, s11
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s11
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s15
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s20
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q4[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q4[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q4
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

llvm/trunk/test/CodeGen/Thumb2/mve-vcmpfr.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve,+fullfp16 -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVE
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVEFP
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oeq_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 eq, q0, r0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp oeq <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_one_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_one_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, r0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp one <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ogt_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, r0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ogt <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oge_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, r0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp oge <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_olt_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q1, q0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp olt <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ole_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q1, q0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ole <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ueq_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, r0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ueq <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_une_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_une_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmp.f32 s3, s4
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ne, q0, r0
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp une <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ugt_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q1, q0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ugt <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uge_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q1, q0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp uge <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ult_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, r0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ult <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ule_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, r0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ule <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ord_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 lt, q0, r0
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q2, q3
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %i = insertelement <4 x float> undef, float %src2, i32 0
   %sp = shufflevector <4 x float> %i, <4 x float> undef, <4 x i32> zeroinitializer
   %c = fcmp ord <4 x float> %src, %sp
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uno_v4f32(<4 x float> %src, float %src2, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s4
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s4
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s4
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s4
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s15, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s14, s10
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s13, s9
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s12, s8
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov r0, s4
 ; CHECK-MVEFP-NEXT:    vdup.32 q1, r0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q1, q0

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmp.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s16
-; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s16
-; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
+; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
-; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s16
-; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s16
-; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
+; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r0, s18
+; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
-; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmp.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vmov r0, s12
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
+; CHECK-MVE-NEXT:    vmov r0, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s1, s16
-; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r2, s12
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmovx.f16 s20, s9
+; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s5
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s20, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s10, s6
-; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s18, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s6
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s16
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s20, s18
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r0, s18
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s18, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov r0, s18
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    cmp r1, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9, d10}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

llvm/trunk/test/CodeGen/Thumb2/mve-vcmpfz.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve,+fullfp16 -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVE
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVEFP
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oeq_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 eq, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp oeq <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_one_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_one_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov.i32 q3, #0x0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q3, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, q3
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp one <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ogt_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ogt <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_oge_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp oge <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_olt_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 lt, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp olt <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ole_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ole <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ueq_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
-; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r3, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
 ; CHECK-MVE-NEXT:    mov.w r0, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov.i32 q3, #0x0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q3, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 le, q0, q3
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ueq <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_une_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_une_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmp.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmp.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmp.f32 s3, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ne, q0, zr
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp une <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ugt_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q0, zr
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ugt <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uge_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 lt, q0, zr
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp uge <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ult_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 ge, q0, zr
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ult <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ule_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, #0
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vcmp.f32 gt, q0, zr
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ule <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_ord_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s1
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s3
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s2
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s2
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s3
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov.i32 q3, #0x0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q3, q0
 ; CHECK-MVEFP-NEXT:    vpst
 ; CHECK-MVEFP-NEXT:    vcmpt.f32 lt, q0, q3
 ; CHECK-MVEFP-NEXT:    vpnot
 ; CHECK-MVEFP-NEXT:    vpsel q0, q1, q2
 ; CHECK-MVEFP-NEXT:    bx lr
 entry:
   %c = fcmp ord <4 x float> %src, zeroinitializer
   %s = select <4 x i1> %c, <4 x float> %a, <4 x float> %b
   ret <4 x float> %s
 }
 
 define arm_aapcs_vfpcc <4 x float> @vcmp_uno_v4f32(<4 x float> %src, <4 x float> %a, <4 x float> %b) {
 ; CHECK-MVE-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVE:       @ %bb.0: @ %entry
-; CHECK-MVE-NEXT:    vcmpe.f32 s1, s1
+; CHECK-MVE-NEXT:    vcmpe.f32 s0, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s0, s0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    vcmpe.f32 s1, s1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r2, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s3, s3
+; CHECK-MVE-NEXT:    vcmpe.f32 s2, s2
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r3, #0
-; CHECK-MVE-NEXT:    vcmpe.f32 s2, s2
+; CHECK-MVE-NEXT:    vcmpe.f32 s3, s3
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r3, #1
 ; CHECK-MVE-NEXT:    cmp r3, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r3, #1
+; CHECK-MVE-NEXT:    csinc r3, zr, zr, eq
 ; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    cmp r3, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s3, s11, s7
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    lsls r0, r3, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s2, s10, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r0, r2, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s1, s9, s5
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    lsls r0, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f32 s0, s8, s4
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v4f32:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry
 ; CHECK-MVEFP-NEXT:    vmov.i32 q3, #0x0
 ; CHECK-MVEFP-NEXT:    vcmp.f32 le, q3, q0
 ; CHECK-MVEFP-NEXT:    vpst

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmp.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oeq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
+; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
-; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_one_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it gt
 ; CHECK-MVE-NEXT:    movgt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ogt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ge
 ; CHECK-MVE-NEXT:    movge r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_oge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it mi
 ; CHECK-MVE-NEXT:    movmi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_olt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ls
 ; CHECK-MVE-NEXT:    movls r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ole_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vcmp.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r2, #1
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
+; CHECK-MVE-NEXT:    mov.w r0, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
+; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmov r1, s16
+; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r1, #1
-; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it eq
 ; CHECK-MVE-NEXT:    moveq r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ueq_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmp.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmp.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmp.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmp.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmp.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it ne
 ; CHECK-MVE-NEXT:    movne r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_une_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it hi
 ; CHECK-MVE-NEXT:    movhi r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ugt_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it pl
 ; CHECK-MVE-NEXT:    movpl r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uge_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it lt
 ; CHECK-MVE-NEXT:    movlt r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ult_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, #0
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, #0
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, #0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it le
 ; CHECK-MVE-NEXT:    movle r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ule_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s12
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, s1
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s2
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s3
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vc
 ; CHECK-MVE-NEXT:    movvc r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_ord_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s0
 ; CHECK-MVE-NEXT:    movs r1, #0
 ; CHECK-MVE-NEXT:    vcmpe.f16 s12, s12
 ; CHECK-MVE-NEXT:    vmovx.f16 s12, s4
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s14, s8
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    mov.w r2, #0
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s14, s12
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r2, #1
 ; CHECK-MVE-NEXT:    cmp r2, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r2, #1
-; CHECK-MVE-NEXT:    cmp r2, #0
+; CHECK-MVE-NEXT:    csinc r2, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s12
+; CHECK-MVE-NEXT:    lsls r2, r2, #31
+; CHECK-MVE-NEXT:    vcmpe.f16 s1, s1
 ; CHECK-MVE-NEXT:    vseleq.f16 s12, s8, s4
+; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r2, s12
-; CHECK-MVE-NEXT:    vcmpe.f16 s1, s1
+; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov.16 q3[0], r2
-; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov.16 q3[1], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    movs r0, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s9, s5
-; CHECK-MVE-NEXT:    vmovx.f16 s18, s9
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s1
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[2], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s5
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s2, s2
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s18, s10
 ; CHECK-MVE-NEXT:    vmov.16 q3[3], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s10, s6
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s3
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s2
 ; CHECK-MVE-NEXT:    vcmpe.f16 s16, s16
 ; CHECK-MVE-NEXT:    vmov.16 q3[4], r1
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmovx.f16 s16, s6
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
 ; CHECK-MVE-NEXT:    vcmpe.f16 s3, s3
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s18, s16
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    vmov r1, s16
 ; CHECK-MVE-NEXT:    vcmpe.f16 s0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[5], r1
 ; CHECK-MVE-NEXT:    mov.w r1, #0
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r1, #1
 ; CHECK-MVE-NEXT:    cmp r1, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r1, #1
-; CHECK-MVE-NEXT:    cmp r1, #0
+; CHECK-MVE-NEXT:    csinc r1, zr, zr, eq
+; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
+; CHECK-MVE-NEXT:    lsls r1, r1, #31
+; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
 ; CHECK-MVE-NEXT:    vseleq.f16 s16, s11, s7
-; CHECK-MVE-NEXT:    movs r0, #0
 ; CHECK-MVE-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-MVE-NEXT:    it vs
 ; CHECK-MVE-NEXT:    movvs r0, #1
 ; CHECK-MVE-NEXT:    cmp r0, #0
-; CHECK-MVE-NEXT:    it ne
-; CHECK-MVE-NEXT:    movne r0, #1
-; CHECK-MVE-NEXT:    vmovx.f16 s0, s7
-; CHECK-MVE-NEXT:    vmovx.f16 s2, s11
-; CHECK-MVE-NEXT:    cmp r0, #0
+; CHECK-MVE-NEXT:    csinc r0, zr, zr, eq
 ; CHECK-MVE-NEXT:    vmov r1, s16
-; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
+; CHECK-MVE-NEXT:    lsls r0, r0, #31
 ; CHECK-MVE-NEXT:    vmov.16 q3[6], r1
+; CHECK-MVE-NEXT:    vseleq.f16 s0, s2, s0
 ; CHECK-MVE-NEXT:    vmov r0, s0
 ; CHECK-MVE-NEXT:    vmov.16 q3[7], r0
 ; CHECK-MVE-NEXT:    vmov q0, q3
 ; CHECK-MVE-NEXT:    vpop {d8, d9}
 ; CHECK-MVE-NEXT:    bx lr
 ;
 ; CHECK-MVEFP-LABEL: vcmp_uno_v8f16:
 ; CHECK-MVEFP:       @ %bb.0: @ %entry

llvm/trunk/test/CodeGen/Thumb2/mve-vcmpr.ll

 define arm_aapcs_vfpcc <2 x i64> @vcmp_eq_v2i64(<2 x i64> %src, i64 %src2, <2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: vcmp_eq_v2i64:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    eors r2, r1
 ; CHECK-NEXT:    eors r3, r0
 ; CHECK-NEXT:    orrs r2, r3
-; CHECK-NEXT:    clz r2, r2
-; CHECK-NEXT:    lsrs r2, r2, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r2, #-1
+; CHECK-NEXT:    csinc r2, zr, zr, ne
+; CHECK-NEXT:    tst.w r2, #1
+; CHECK-NEXT:    csinv r2, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r2
 ; CHECK-NEXT:    vmov.32 q3[1], r2
 ; CHECK-NEXT:    vmov r2, s3
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    eors r0, r2
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
 ; CHECK-NEXT:    vand q1, q1, q3
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %i = insertelement <2 x i64> undef, i64 %src2, i32 0
   %sp = shufflevector <2 x i64> %i, <2 x i64> undef, <2 x i32> zeroinitializer
   %c = icmp eq <2 x i64> %src, %sp
   %s = select <2 x i1> %c, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <2 x i32> @vcmp_eq_v2i32(<2 x i64> %src, i64 %src2, <2 x i32> %a, <2 x i32> %b) {
 ; CHECK-LABEL: vcmp_eq_v2i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    eors r2, r1
 ; CHECK-NEXT:    eors r3, r0
 ; CHECK-NEXT:    orrs r2, r3
-; CHECK-NEXT:    clz r2, r2
-; CHECK-NEXT:    lsrs r2, r2, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r2, #-1
+; CHECK-NEXT:    csinc r2, zr, zr, ne
+; CHECK-NEXT:    tst.w r2, #1
+; CHECK-NEXT:    csinv r2, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r2
 ; CHECK-NEXT:    vmov.32 q3[1], r2
 ; CHECK-NEXT:    vmov r2, s3
 ; CHECK-NEXT:    eors r1, r2
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    eors r0, r2
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
 ; CHECK-NEXT:    vand q1, q1, q3
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %i = insertelement <2 x i64> undef, i64 %src2, i32 0

 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov lr, s10
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    subs r1, r0, r2
-; CHECK-NEXT:    asr.w r12, r0, #31
+; CHECK-NEXT:    vmov lr, s0
+; CHECK-NEXT:    subs.w r1, lr, r2
+; CHECK-NEXT:    asr.w r12, lr, #31
 ; CHECK-NEXT:    sbcs.w r1, r12, r2, asr #31
 ; CHECK-NEXT:    mov.w r1, #0
+; CHECK-NEXT:    vmov r2, s10
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #1
 ; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
+; CHECK-NEXT:    csinv r1, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r1
 ; CHECK-NEXT:    vmov.32 q3[1], r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    subs.w r2, r1, lr
+; CHECK-NEXT:    subs r0, r1, r2
 ; CHECK-NEXT:    asr.w r12, r1, #31
-; CHECK-NEXT:    sbcs.w r2, r12, lr, asr #31
+; CHECK-NEXT:    sbcs.w r0, r12, r2, asr #31
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r3, #1
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r3, #-1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    cmp.w lr, #0
+; CHECK-NEXT:    vmov.32 q3[2], r0
+; CHECK-NEXT:    vmov.32 q3[3], r0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    vmov.32 q4[0], r0
 ; CHECK-NEXT:    vmov.32 q4[1], r0
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
+; CHECK-NEXT:    vmov.32 q4[2], r0
+; CHECK-NEXT:    vmov.32 q4[3], r0
 ; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r1, #1
-; CHECK-NEXT:    cmp r1, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r1, #-1
-; CHECK-NEXT:    vmov.32 q4[2], r1
-; CHECK-NEXT:    vmov.32 q3[2], r3
-; CHECK-NEXT:    vmov.32 q4[3], r1
-; CHECK-NEXT:    vmov.32 q3[3], r3
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q5[0], r0
 ; CHECK-NEXT:    vmov.32 q5[1], r0
 ; CHECK-NEXT:    vmov r0, s6
 ; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne r0, #1
-; CHECK-NEXT:    cmp r0, #0
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, eq
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q5[2], r0
 ; CHECK-NEXT:    vmov.32 q5[3], r0
 ; CHECK-NEXT:    vand q1, q5, q4
 ; CHECK-NEXT:    vand q1, q3, q1
 ; CHECK-NEXT:    vbic q0, q0, q1
 ; CHECK-NEXT:    vand q1, q2, q1
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}

llvm/trunk/test/CodeGen/Thumb2/mve-vcmpz.ll

 
 define arm_aapcs_vfpcc <2 x i64> @vcmp_eqz_v2i64(<2 x i64> %src, <2 x i64> %a, <2 x i64> %b) {
 ; CHECK-LABEL: vcmp_eqz_v2i64:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
 ; CHECK-NEXT:    vand q1, q1, q3
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp eq <2 x i64> %src, zeroinitializer
   %s = select <2 x i1> %c, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <2 x i32> @vcmp_eqz_v2i32(<2 x i64> %src, <2 x i32> %a, <2 x i32> %b) {
 ; CHECK-LABEL: vcmp_eqz_v2i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov r0, s1
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    orrs r0, r1
 ; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[0], r0
 ; CHECK-NEXT:    vmov.32 q3[1], r0
 ; CHECK-NEXT:    vmov r0, s3
 ; CHECK-NEXT:    orrs r0, r1
-; CHECK-NEXT:    clz r0, r0
-; CHECK-NEXT:    lsrs r0, r0, #5
-; CHECK-NEXT:    it ne
-; CHECK-NEXT:    movne.w r0, #-1
+; CHECK-NEXT:    csinc r0, zr, zr, ne
+; CHECK-NEXT:    tst.w r0, #1
+; CHECK-NEXT:    csinv r0, zr, zr, eq
 ; CHECK-NEXT:    vmov.32 q3[2], r0
 ; CHECK-NEXT:    vmov.32 q3[3], r0
 ; CHECK-NEXT:    vbic q0, q2, q3
 ; CHECK-NEXT:    vand q1, q1, q3
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %c = icmp eq <2 x i64> %src, zeroinitializer
   %s = select <2 x i1> %c, <2 x i32> %a, <2 x i32> %b
   ret <2 x i32> %s
 }