[ARM] MVE VCVT lowering for f32->f16 truncs

[dmgreen]

Summary

This adds code to lower f32 to f16 fp_trunc's using a pair of MVE VCVT instructions. Due to v4f16 not being legal, fp_round are often split up fairly early. So this reconstructs the vcvt's from a buildvector of fp_rounds from two vector inputs. Something like:

BUILDVECTOR(FP_ROUND(EXTRACT_ELT(X, 0),
           FP_ROUND(EXTRACT_ELT(Y, 0),
           FP_ROUND(EXTRACT_ELT(X, 1),
           FP_ROUND(EXTRACT_ELT(Y, 1), ...)

It adds a VCVTN node to handle this, which like VMOVN or VQMOVN lowers into the top/bottom lanes of an MVE instruction.

[efriedma]

Due to v4f16 not being legal

This is a MVE-specific thing? hmm.

Dealing with mixed types is a recurring problem with target-independent vector handling; we should probably try to extend the approach currently used by SIGN_EXTEND_VECTOR_INREG to other cast opcodes, so we don't have to keep repeating exactly the same hacks for every target with vector registers.

[dmgreen]

In D81139#2073931, @efriedma wrote:

Due to v4f16 not being legal

This is a MVE-specific thing? hmm.

Dealing with mixed types is a recurring problem with target-independent vector handling; we should probably try to extend the approach currently used by SIGN_EXTEND_VECTOR_INREG to other cast opcodes, so we don't have to keep repeating exactly the same hacks for every target with vector registers.

Can you explain what kind of thing you are thinking of? A <4 x half> under MVE is very different to a <4 x i16> under MVE, as we currently do type promotion. And the "bottom half" of a v8i/f16 register is always a bit difficult to deal with.

If you have a better idea, I'm interested to hear it. I was currently planning to add something (probably pre-isel so it can look across BB boundaries) that took blobs of vector instructions surrounded by sext/zext/trunc (and now fpext/fptrunc) and turned them into something more MVE-y, by inserting shuffles like the tests above to allow us to use T/B instructions.

So I was thinking this was an MVE special.

[efriedma]

x86 has similar issues with SSE2 floating-point. I think maybe some other targets have related issues.

The key function that's making everything a complete mess is DAGTypeLegalizer::WidenVecRes_Convert. Ideally, it would have some target-independent vector operation it could generate (compare to DAGTypeLegalizer::WidenVecOp_EXTEND). But barring that, it probably makes sense to custom-legalize it. Pattern-matching the build_vector isn't broken, exactly, but it involves a lot of DAG nodes.

[dmgreen]

Hmm. shuffle_trunc1 comes from something like this:

    t17: v4f32 = vector_shuffle<0,4,1,5> t3, t6
    t18: v4f32 = vector_shuffle<2,6,3,7> t3, t6
  t19: v8f32 = concat_vectors t17, t18
t12: v8f16 = fp_round t19, TargetConstant:i32<0>

Which then gets split into two halves because of the v8f32 and the two halves look like this in WidenVecRes_Convert:

  t17: v4f32 = vector_shuffle<0,4,1,5> t3, t6
t20: v4f16 = fp_round t17, TargetConstant:i32<0>

Which need to be concat back together. The two halves of the BUILD_VECTOR are combined and that is what we end up lowering.

shuffle_trunc3 is that but twice as wide, starting with:

    t32: v8f32 = vector_shuffle<0,8,1,9,2,10,3,11> t13, t14
    t33: v8f32 = vector_shuffle<4,12,5,13,6,14,7,15> t13, t14
  t34: v16f32 = concat_vectors t32, t33
t20: v16f16 = fp_round t34, TargetConstant:i32<0>

We end up with 4 BuildVectors that are combined back together into 2.

shuffle_trunc5 is this before we start legalizing the types:

    t8: v4f16 = fp_round t3, TargetConstant:i32<0>
  t11: v8f16 = concat_vectors t8, undef:v4f16
    t9: v4f16 = fp_round t6, TargetConstant:i32<0>
  t12: v8f16 = concat_vectors t9, undef:v4f16
t13: v8f16 = vector_shuffle<0,8,1,9,2,10,3,11> t11, t12

So it's hard to see the shuffle from the fp_round. Again though, it creates BuildVectors, BuildVectors simplify, we lower from the BuildVectors. Perhaps that's a bit of a stranger case with the v4f16 vectors. But unfortunately they are likely to come up from the vectorizer at the moment.

shuffle_trunc7 is the same thing but double the width:

      t13: v8f32 = concat_vectors t3, t6
    t16: v8f16 = fp_round t13, TargetConstant:i32<0>
  t19: v16f16 = concat_vectors t16, undef:v8f16
      t14: v8f32 = concat_vectors t9, t12
    t17: v8f16 = fp_round t14, TargetConstant:i32<0>
  t20: v16f16 = concat_vectors t17, undef:v8f16
t21: v16f16 = vector_shuffle<0,16,1,17,2,18,3,19,4,20,5,21,6,22,7,23> t19, t20

I guess I'm still having trouble seeing what we would reliably latch onto here.

I do have some old code that was using a dagcombine on a fptrunc(shufflevector), but that didn't handle all these cases and doing this from a buildvector seemed much simpler. It is the way that we lower all shuffles in the arm backend (like vext and vmovn) after all. The only difference here is that we have an fptrunc in the mix too.

[efriedma]

LGTM

I wasn't really paying attention to how the interleaving works... I guess this is probably the least terrible solution here. (I still think we should make some target-independent enhancements in this area, but maybe there isn't enough common ground in this particular case.)

[dmgreen]

Thanks. I can agree with "least terrible" :)

I have another patch to do the same thing with FP_EXT. I'll put that up soon and we can see whether that one is OK too.

llvm/lib/Target/ARM/ARMISelLowering.h

enum NodeType : unsigned {
       VTBL1,        // 1-register shuffle with mask
       VTBL2,        // 2-register shuffle with mask
       VMOVN,        // MVE vmovn
 
       // MVE Saturating truncates
       VQMOVNs,      // Vector (V) Saturating (Q) Move and Narrow (N), signed (s)
       VQMOVNu,      // Vector (V) Saturating (Q) Move and Narrow (N), unsigned (u)
 
+      // MVE float <> half converts
+      VCVTN,        // MVE vcvt f32 -> f16, truncating into either the bottom or top lanes
+
       // Vector multiply long:
       VMULLs,       // ...signed
       VMULLu,       // ...unsigned
 
       // MVE reductions
       VADDVs,       // sign- or zero-extend the elements of a vector to i32,
       VADDVu,       //   add them all together, and return an i32 of their sum
       VADDLVs,      // sign- or zero-extend elements to i64 and sum, returning

llvm/lib/Target/ARM/ARMISelLowering.cpp

const char *ARMTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case ARMISD::VZIP:          return "ARMISD::VZIP";
   case ARMISD::VUZP:          return "ARMISD::VUZP";
   case ARMISD::VTRN:          return "ARMISD::VTRN";
   case ARMISD::VTBL1:         return "ARMISD::VTBL1";
   case ARMISD::VTBL2:         return "ARMISD::VTBL2";
   case ARMISD::VMOVN:         return "ARMISD::VMOVN";
   case ARMISD::VQMOVNs:       return "ARMISD::VQMOVNs";
   case ARMISD::VQMOVNu:       return "ARMISD::VQMOVNu";
+  case ARMISD::VCVTN:         return "ARMISD::VCVTN";
   case ARMISD::VMULLs:        return "ARMISD::VMULLs";
   case ARMISD::VMULLu:        return "ARMISD::VMULLu";
   case ARMISD::VADDVs:        return "ARMISD::VADDVs";
   case ARMISD::VADDVu:        return "ARMISD::VADDVu";
   case ARMISD::VADDLVs:       return "ARMISD::VADDLVs";
   case ARMISD::VADDLVu:       return "ARMISD::VADDLVu";
   case ARMISD::VADDLVAs:      return "ARMISD::VADDLVAs";
   case ARMISD::VADDLVAu:      return "ARMISD::VADDLVAu";
if (M[i] >= 0 && M[i] != (int)i)
       return false;
     if (M[i+1] >= 0 && M[i+1] != (int)(NumElts + i + Offset))
       return false;
   }
 
   return true;
 }
 
+// Reconstruct an MVE VCVT from a BuildVector of scalar fptrunc, all extract
+// from a pair of inputs. For example:
+// BUILDVECTOR(FP_ROUND(EXTRACT_ELT(X, 0),
+//             FP_ROUND(EXTRACT_ELT(Y, 0),
+//             FP_ROUND(EXTRACT_ELT(X, 1),
+//             FP_ROUND(EXTRACT_ELT(Y, 1), ...)
+static SDValue LowerBuildVectorOfFPTrunc(SDValue BV, SelectionDAG &DAG,
+                                         const ARMSubtarget *ST) {
+  assert(BV.getOpcode() == ISD::BUILD_VECTOR && "Unknown opcode!");
+  if (!ST->hasMVEFloatOps())
+    return SDValue();
+
+  SDLoc dl(BV);
+  EVT VT = BV.getValueType();
+  if (VT != MVT::v8f16)
+    return SDValue();
+
+  // We are looking for a buildvector of fptrunc elements, where all the
+  // elements are interleavingly extracted from two sources. Check the first two
+  // items are valid enough and extract some info from them (they are checked
+  // properly in the loop below).
+  if (BV.getOperand(0).getOpcode() != ISD::FP_ROUND ||
+      BV.getOperand(0).getOperand(0).getOpcode() != ISD::EXTRACT_VECTOR_ELT ||
+      BV.getOperand(0).getOperand(0).getConstantOperandVal(1) != 0)
+    return SDValue();
+  if (BV.getOperand(1).getOpcode() != ISD::FP_ROUND ||
+      BV.getOperand(1).getOperand(0).getOpcode() != ISD::EXTRACT_VECTOR_ELT ||
+      BV.getOperand(1).getOperand(0).getConstantOperandVal(1) != 0)
+    return SDValue();
+  SDValue Op0 = BV.getOperand(0).getOperand(0).getOperand(0);
+  SDValue Op1 = BV.getOperand(1).getOperand(0).getOperand(0);
+  if (Op0.getValueType() != MVT::v4f32 || Op1.getValueType() != MVT::v4f32)
+    return SDValue();
+
+  // Check all the values in the BuildVector line up with our expectations.
+  for (int i = 1; i < 4; i++) {
+    auto Check = [](SDValue Trunc, SDValue Op, int Idx) {
+      return Trunc.getOpcode() == ISD::FP_ROUND &&
+             Trunc.getOperand(0).getOpcode() == ISD::EXTRACT_VECTOR_ELT &&
+             Trunc.getOperand(0).getOperand(0) == Op &&
+             Trunc.getOperand(0).getConstantOperandVal(1) == Idx;
+    };
+    if (!Check(BV.getOperand(i * 2 + 0), Op0, i))
+      return SDValue();
+    if (!Check(BV.getOperand(i * 2 + 1), Op1, i))
+      return SDValue();
+  }
+
+  SDValue N1 = DAG.getNode(ARMISD::VCVTN, dl, VT, DAG.getUNDEF(VT), Op0,
+                           DAG.getConstant(0, dl, MVT::i32));
+  return DAG.getNode(ARMISD::VCVTN, dl, VT, N1, Op1,
+                     DAG.getConstant(1, dl, MVT::i32));
+}
+
 // If N is an integer constant that can be moved into a register in one
 // instruction, return an SDValue of such a constant (will become a MOV
 // instruction).  Otherwise return null.
 static SDValue IsSingleInstrConstant(SDValue N, SelectionDAG &DAG,
                                      const ARMSubtarget *ST, const SDLoc &dl) {
   uint64_t Val;
   if (!isa<ConstantSDNode>(N))
     return SDValue();
SDValue ARMTargetLowering::LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG,
   }
 
   // If all elements are constants and the case above didn't get hit, fall back
   // to the default expansion, which will generate a load from the constant
   // pool.
   if (isConstant)
     return SDValue();
 
-  // Empirical tests suggest this is rarely worth it for vectors of length <= 2.
-  if (NumElts >= 4) {
-    SDValue shuffle = ReconstructShuffle(Op, DAG);
-    if (shuffle != SDValue())
+  // Reconstruct the BUILDVECTOR to one of the legal shuffles (such as vext and
+  // vmovn). Empirical tests suggest this is rarely worth it for vectors of
+  // length <= 2.
+  if (NumElts >= 4)
+    if (SDValue shuffle = ReconstructShuffle(Op, DAG))
       return shuffle;
-  }
+
+  // Attempt to turn a buildvector of scalar fptrunc's back into VCVT's
+  if (SDValue VCVT = LowerBuildVectorOfFPTrunc(Op, DAG, Subtarget))
+    return VCVT;
 
   if (ST->hasNEON() && VT.is128BitVector() && VT != MVT::v2f64 && VT != MVT::v4f32) {
     // If we haven't found an efficient lowering, try splitting a 128-bit vector
     // into two 64-bit vectors; we might discover a better way to lower it.
     SmallVector<SDValue, 64> Ops(Op->op_begin(), Op->op_begin() + NumElts);
     EVT ExtVT = VT.getVectorElementType();
     EVT HVT = EVT::getVectorVT(*DAG.getContext(), ExtVT, NumElts / 2);
     SDValue Lower =

llvm/lib/Target/ARM/ARMInstrMVE.td

class MVE_VCVT_ff<string iname, string suffix, bit op, bit T,
   let Inst{12} = T;
   let Inst{8-7} = 0b00;
   let Inst{0} = 0b1;
 
   let Predicates = [HasMVEFloat];
   let retainsPreviousHalfElement = 1;
 }
 
+def MVEvcvtn       : SDNode<"ARMISD::VCVTN", SDTARMVMOVNQ>;
+
 multiclass MVE_VCVT_f2h_m<string iname, int half> {
   def "": MVE_VCVT_ff<iname, "f16.f32", 0b0, half,
                       (ins MQPR:$Qd_src), vpred_n, "$Qd = $Qd_src">;
   defvar Inst = !cast<Instruction>(NAME);
 
   let Predicates = [HasMVEFloat] in {
     def : Pat<(v8f16 (int_arm_mve_vcvt_narrow
                          (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm), (i32 half))),
               (v8f16 (Inst (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm)))>;
     def : Pat<(v8f16 (int_arm_mve_vcvt_narrow_predicated
                          (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm), (i32 half),
                          (v4i1 VCCR:$mask))),
               (v8f16 (Inst (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm),
                            ARMVCCThen, (v4i1 VCCR:$mask)))>;
+
+    def : Pat<(v8f16 (MVEvcvtn (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm), (i32 half))),
+              (v8f16 (Inst (v8f16 MQPR:$Qd_src), (v4f32 MQPR:$Qm)))>;
   }
 }
 
 multiclass MVE_VCVT_h2f_m<string iname, int half> {
   def "": MVE_VCVT_ff<iname, "f32.f16", 0b1, half, (ins), vpred_r, "">;
   defvar Inst = !cast<Instruction>(NAME);
 
   let Predicates = [HasMVEFloat] in {

llvm/test/CodeGen/Thumb2/mve-fp16convertloops.ll

 ; CHECK-NEXT:    .vsave {d8, d9}
 ; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adr r2, .LCPI9_0
 ; CHECK-NEXT:    mov.w lr, #128
 ; CHECK-NEXT:    vldrw.u32 q0, [r2]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrh.u16 q2, [r0], #16
-; CHECK-NEXT:    vcvtb.f32.f16 s7, s11
-; CHECK-NEXT:    vmovx.f16 s13, s11
-; CHECK-NEXT:    vcvtb.f32.f16 s6, s10
-; CHECK-NEXT:    vmovx.f16 s14, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s5, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s4, s8
-; CHECK-NEXT:    vmovx.f16 s8, s10
-; CHECK-NEXT:    vmul.f32 q1, q1, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s4
-; CHECK-NEXT:    vcvtb.f32.f16 s19, s13
-; CHECK-NEXT:    vmov r2, s12
-; CHECK-NEXT:    vmovx.f16 s12, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s18, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s17, s12
-; CHECK-NEXT:    vcvtb.f32.f16 s16, s14
-; CHECK-NEXT:    vmul.f32 q2, q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov r3, s12
-; CHECK-NEXT:    vmov.16 q3[0], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmov.16 q3[1], r3
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s9
-; CHECK-NEXT:    vmov.16 q3[2], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q3[3], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s10
-; CHECK-NEXT:    vmov.16 q3[4], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q3[5], r2
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s11
-; CHECK-NEXT:    vmov.16 q3[6], r2
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmov.16 q3[7], r2
-; CHECK-NEXT:    vstrb.8 q3, [r1], #16
+; CHECK-NEXT:    vldrh.u16 q1, [r0], #16
+; CHECK-NEXT:    vmovx.f16 s8, s7
+; CHECK-NEXT:    vmovx.f16 s14, s6
+; CHECK-NEXT:    vcvtb.f32.f16 s11, s8
+; CHECK-NEXT:    vmovx.f16 s13, s5
+; CHECK-NEXT:    vcvtb.f32.f16 s10, s14
+; CHECK-NEXT:    vmovx.f16 s12, s4
+; CHECK-NEXT:    vcvtb.f32.f16 s9, s13
+; CHECK-NEXT:    vcvtb.f32.f16 s19, s7
+; CHECK-NEXT:    vcvtb.f32.f16 s18, s6
+; CHECK-NEXT:    vcvtb.f32.f16 s17, s5
+; CHECK-NEXT:    vcvtb.f32.f16 s16, s4
+; CHECK-NEXT:    vcvtb.f32.f16 s8, s12
+; CHECK-NEXT:    vmul.f32 q1, q4, q0
+; CHECK-NEXT:    vmul.f32 q2, q2, q0
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q2
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    le lr, .LBB9_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    vpop {d8, d9}
 ; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI9_0:
 ; CHECK-NEXT:    .long 0x40066666 @ float 2.0999999

 ; CHECK-NEXT:    .vsave {d8, d9}
 ; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adr r2, .LCPI10_0
 ; CHECK-NEXT:    mov.w lr, #128
 ; CHECK-NEXT:    vldrw.u32 q0, [r2]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB10_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrh.u16 q2, [r0]
-; CHECK-NEXT:    vcvtb.f32.f16 s7, s11
-; CHECK-NEXT:    vmovx.f16 s13, s11
-; CHECK-NEXT:    vcvtb.f32.f16 s6, s10
-; CHECK-NEXT:    vmovx.f16 s14, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s5, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s4, s8
-; CHECK-NEXT:    vmovx.f16 s8, s10
-; CHECK-NEXT:    vmul.f32 q1, q1, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s4
-; CHECK-NEXT:    vcvtb.f32.f16 s19, s13
-; CHECK-NEXT:    vmov r2, s12
-; CHECK-NEXT:    vmovx.f16 s12, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s18, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s17, s12
-; CHECK-NEXT:    vcvtb.f32.f16 s16, s14
-; CHECK-NEXT:    vmul.f32 q2, q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov r3, s12
-; CHECK-NEXT:    vmov.16 q3[0], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmov.16 q3[1], r3
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s9
-; CHECK-NEXT:    vmov.16 q3[2], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q3[3], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s10
-; CHECK-NEXT:    vmov.16 q3[4], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q3[5], r2
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s11
-; CHECK-NEXT:    vmov.16 q3[6], r2
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldrh.u16 q2, [r0, #16]!
-; CHECK-NEXT:    vmov.16 q3[7], r2
-; CHECK-NEXT:    vstrh.16 q3, [r1]
-; CHECK-NEXT:    vmovx.f16 s12, s11
-; CHECK-NEXT:    vmovx.f16 s14, s10
+; CHECK-NEXT:    vldrh.u16 q1, [r0]
+; CHECK-NEXT:    vmovx.f16 s8, s7
+; CHECK-NEXT:    vmovx.f16 s14, s6
+; CHECK-NEXT:    vcvtb.f32.f16 s11, s8
+; CHECK-NEXT:    vmovx.f16 s13, s5
+; CHECK-NEXT:    vcvtb.f32.f16 s10, s14
+; CHECK-NEXT:    vmovx.f16 s12, s4
+; CHECK-NEXT:    vcvtb.f32.f16 s9, s13
+; CHECK-NEXT:    vcvtb.f32.f16 s19, s7
+; CHECK-NEXT:    vcvtb.f32.f16 s18, s6
+; CHECK-NEXT:    vcvtb.f32.f16 s17, s5
+; CHECK-NEXT:    vcvtb.f32.f16 s16, s4
+; CHECK-NEXT:    vcvtb.f32.f16 s8, s12
+; CHECK-NEXT:    vmul.f32 q1, q4, q0
+; CHECK-NEXT:    vmul.f32 q2, q2, q0
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q2
+; CHECK-NEXT:    vstrh.16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q1, [r0, #16]!
+; CHECK-NEXT:    vmovx.f16 s12, s7
+; CHECK-NEXT:    vmovx.f16 s14, s6
 ; CHECK-NEXT:    vcvtb.f32.f16 s19, s12
-; CHECK-NEXT:    vmovx.f16 s4, s9
+; CHECK-NEXT:    vmovx.f16 s8, s5
 ; CHECK-NEXT:    vcvtb.f32.f16 s18, s14
-; CHECK-NEXT:    vmovx.f16 s6, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s17, s4
-; CHECK-NEXT:    vcvtb.f32.f16 s16, s6
+; CHECK-NEXT:    vmovx.f16 s10, s4
+; CHECK-NEXT:    vcvtb.f32.f16 s17, s8
+; CHECK-NEXT:    vcvtb.f32.f16 s16, s10
+; CHECK-NEXT:    vcvtb.f32.f16 s11, s7
+; CHECK-NEXT:    vcvtb.f32.f16 s10, s6
+; CHECK-NEXT:    vcvtb.f32.f16 s9, s5
+; CHECK-NEXT:    vcvtb.f32.f16 s8, s4
 ; CHECK-NEXT:    vmul.f32 q1, q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s4
-; CHECK-NEXT:    vmov r2, s12
-; CHECK-NEXT:    vcvtb.f32.f16 s15, s11
-; CHECK-NEXT:    vcvtb.f32.f16 s14, s10
-; CHECK-NEXT:    vcvtb.f32.f16 s13, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s12, s8
-; CHECK-NEXT:    vmul.f32 q3, q3, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s13
-; CHECK-NEXT:    vmov r3, s8
-; CHECK-NEXT:    vmov.16 q2[0], r3
-; CHECK-NEXT:    vmov.16 q2[1], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov.16 q2[2], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s14
-; CHECK-NEXT:    vmov.16 q2[3], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q2[4], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s15
-; CHECK-NEXT:    vmov.16 q2[5], r2
-; CHECK-NEXT:    vmov r2, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[6], r2
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmov.16 q2[7], r2
+; CHECK-NEXT:    vmul.f32 q2, q2, q0
+; CHECK-NEXT:    vcvtb.f16.f32 q2, q2
+; CHECK-NEXT:    vcvtt.f16.f32 q2, q1
 ; CHECK-NEXT:    vstrb.8 q2, [r1, #16]!
 ; CHECK-NEXT:    le lr, .LBB10_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    vpop {d8, d9}
 ; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI10_0:

llvm/test/CodeGen/Thumb2/mve-vcvt16.ll

entry:
   %out = fptrunc <8 x float> %src1 to <8 x half>
   ret <8 x half> %out
 }
 
 
 define arm_aapcs_vfpcc <8 x half> @shuffle_trunc1(<4 x float> %src1, <4 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov q2, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s4
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s9
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s5
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s10
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s6
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <4 x float> %src1, <4 x float> %src2, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
   %out = fptrunc <8 x float> %strided.vec to <8 x half>
   ret <8 x half> %out
 }
 
 define arm_aapcs_vfpcc <8 x half> @shuffle_trunc2(<4 x float> %src1, <4 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s0
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s5
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s1
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s6
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s2
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s3
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q0, q2
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q0
+; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <4 x float> %src1, <4 x float> %src2, <8 x i32> <i32 4, i32 0, i32 5, i32 1, i32 6, i32 2, i32 7, i32 3>
   %out = fptrunc <8 x float> %strided.vec to <8 x half>
   ret <8 x half> %out
 }
 
 define arm_aapcs_vfpcc <16 x half> @shuffle_trunc3(<8 x float> %src1, <8 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .vsave {d8, d9, d10}
-; CHECK-NEXT:    vpush {d8, d9, d10}
-; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s16
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s17
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s9
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s18
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s10
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s19
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s12
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s13
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s14
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s15
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q1, q2
-; CHECK-NEXT:    vpop {d8, d9, d10}
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q2
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q3
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <8 x float> %src1, <8 x float> %src2, <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
   %out = fptrunc <16 x float> %strided.vec to <16 x half>
   ret <16 x half> %out
 }
 
 define arm_aapcs_vfpcc <16 x half> @shuffle_trunc4(<8 x float> %src1, <8 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .vsave {d8, d9, d10}
-; CHECK-NEXT:    vpush {d8, d9, d10}
-; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s16
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s9
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s17
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s10
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s18
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s19
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s12
-; CHECK-NEXT:    vmov.16 q0[7], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s13
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s14
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s15
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q1, q2
-; CHECK-NEXT:    vpop {d8, d9, d10}
+; CHECK-NEXT:    vcvtb.f16.f32 q2, q2
+; CHECK-NEXT:    vcvtb.f16.f32 q3, q3
+; CHECK-NEXT:    vcvtt.f16.f32 q2, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q3, q1
+; CHECK-NEXT:    vmov q0, q2
+; CHECK-NEXT:    vmov q1, q3
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <8 x float> %src1, <8 x float> %src2, <16 x i32> <i32 8, i32 0, i32 9, i32 1, i32 10, i32 2, i32 11, i32 3, i32 12, i32 4, i32 13, i32 5, i32 14, i32 6, i32 15, i32 7>
   %out = fptrunc <16 x float> %strided.vec to <16 x half>
   ret <16 x half> %out
 }
 
 define arm_aapcs_vfpcc <8 x half> @shuffle_trunc5(<4 x float> %src1, <4 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc5:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov q2, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s4
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s9
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s5
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s10
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s6
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %out1 = fptrunc <4 x float> %src1 to <4 x half>
   %out2 = fptrunc <4 x float> %src2 to <4 x half>
   %s = shufflevector <4 x half> %out1, <4 x half> %out2, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
   ret <8 x half> %s
 }
 
 define arm_aapcs_vfpcc <8 x half> @shuffle_trunc6(<4 x float> %src1, <4 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc6:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s0
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s5
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s1
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s6
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s2
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s3
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q0, q2
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q0
+; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %out1 = fptrunc <4 x float> %src1 to <4 x half>
   %out2 = fptrunc <4 x float> %src2 to <4 x half>
   %s = shufflevector <4 x half> %out1, <4 x half> %out2, <8 x i32> <i32 4, i32 0, i32 5, i32 1, i32 6, i32 2, i32 7, i32 3>
   ret <8 x half> %s
 }
 
 define arm_aapcs_vfpcc <16 x half> @shuffle_trunc7(<8 x float> %src1, <8 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc7:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .vsave {d8, d9, d10}
-; CHECK-NEXT:    vpush {d8, d9, d10}
-; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s16
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s17
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s9
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s18
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s10
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s19
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s12
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s13
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s14
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s15
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q1, q2
-; CHECK-NEXT:    vpop {d8, d9, d10}
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q2
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q3
 ; CHECK-NEXT:    bx lr
 entry:
   %out1 = fptrunc <8 x float> %src1 to <8 x half>
   %out2 = fptrunc <8 x float> %src2 to <8 x half>
   %s = shufflevector <8 x half> %out1, <8 x half> %out2, <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
   ret <16 x half> %s
 }
 
 define arm_aapcs_vfpcc <16 x half> @shuffle_trunc8(<8 x float> %src1, <8 x float> %src2) {
 ; CHECK-LABEL: shuffle_trunc8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .vsave {d8, d9, d10}
-; CHECK-NEXT:    vpush {d8, d9, d10}
-; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s8
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s16
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s9
-; CHECK-NEXT:    vmov.16 q0[1], r1
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s17
-; CHECK-NEXT:    vmov.16 q0[2], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s10
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s18
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s11
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s19
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s12
-; CHECK-NEXT:    vmov.16 q0[7], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vmov.16 q2[0], r0
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s13
-; CHECK-NEXT:    vmov.16 q2[1], r1
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s5
-; CHECK-NEXT:    vmov.16 q2[2], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s14
-; CHECK-NEXT:    vmov.16 q2[3], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s6
-; CHECK-NEXT:    vmov.16 q2[4], r0
-; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s15
-; CHECK-NEXT:    vmov.16 q2[5], r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q2[6], r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q2[7], r0
-; CHECK-NEXT:    vmov q1, q2
-; CHECK-NEXT:    vpop {d8, d9, d10}
+; CHECK-NEXT:    vcvtb.f16.f32 q2, q2
+; CHECK-NEXT:    vcvtb.f16.f32 q3, q3
+; CHECK-NEXT:    vcvtt.f16.f32 q2, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q3, q1
+; CHECK-NEXT:    vmov q0, q2
+; CHECK-NEXT:    vmov q1, q3
 ; CHECK-NEXT:    bx lr
 entry:
   %out1 = fptrunc <8 x float> %src1 to <8 x half>
   %out2 = fptrunc <8 x float> %src2 to <8 x half>
   %s = shufflevector <8 x half> %out1, <8 x half> %out2, <16 x i32> <i32 8, i32 0, i32 9, i32 1, i32 10, i32 2, i32 11, i32 3, i32 12, i32 4, i32 13, i32 5, i32 14, i32 6, i32 15, i32 7>
   ret <16 x half> %s
 }
 
entry:
   %e = fptrunc <16 x float> %val to <16 x half>
   store <16 x half> %e, <16 x half>* %src, align 4
   ret void
 }
 
 define arm_aapcs_vfpcc void @store_shuffletrunc_8(<8 x half>* %src, <4 x float> %val1, <4 x float> %val2) {
 ; CHECK-LABEL: store_shuffletrunc_8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s0
-; CHECK-NEXT:    vmov r1, s8
-; CHECK-NEXT:    vcvtb.f16.f32 s8, s4
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov.16 q2[0], r1
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s1
-; CHECK-NEXT:    vmov.16 q2[1], r2
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s5
-; CHECK-NEXT:    vmov.16 q2[2], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s2
-; CHECK-NEXT:    vmov.16 q2[3], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s6
-; CHECK-NEXT:    vmov.16 q2[4], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s3
-; CHECK-NEXT:    vmov.16 q2[5], r1
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s7
-; CHECK-NEXT:    vmov.16 q2[6], r1
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q2[7], r1
-; CHECK-NEXT:    vstrw.32 q2, [r0]
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q1
+; CHECK-NEXT:    vstrw.32 q0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <4 x float> %val1, <4 x float> %val2, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
   %out = fptrunc <8 x float> %strided.vec to <8 x half>
   store <8 x half> %out, <8 x half>* %src, align 4
   ret void
 }
 
 define arm_aapcs_vfpcc void @store_shuffletrunc_16(<16 x half>* %src, <8 x float> %val1, <8 x float> %val2) {
 ; CHECK-LABEL: store_shuffletrunc_16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .vsave {d8, d9, d10}
-; CHECK-NEXT:    vpush {d8, d9, d10}
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s4
-; CHECK-NEXT:    vmov r1, s16
-; CHECK-NEXT:    vcvtb.f16.f32 s16, s12
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmov.16 q4[0], r1
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s5
-; CHECK-NEXT:    vmov.16 q4[1], r2
-; CHECK-NEXT:    vmov r1, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s13
-; CHECK-NEXT:    vmov.16 q4[2], r1
-; CHECK-NEXT:    vmov r1, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s6
-; CHECK-NEXT:    vmov.16 q4[3], r1
-; CHECK-NEXT:    vmov r1, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s20, s14
-; CHECK-NEXT:    vmov.16 q4[4], r1
-; CHECK-NEXT:    vmov r1, s20
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s7
-; CHECK-NEXT:    vmov.16 q4[5], r1
-; CHECK-NEXT:    vmov r1, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s15
-; CHECK-NEXT:    vmov.16 q4[6], r1
-; CHECK-NEXT:    vmov r1, s4
-; CHECK-NEXT:    vmov.16 q4[7], r1
-; CHECK-NEXT:    vstrw.32 q4, [r0, #16]
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s8
-; CHECK-NEXT:    vmov r1, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s4, s0
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s1
-; CHECK-NEXT:    vmov.16 q1[0], r2
-; CHECK-NEXT:    vmov.16 q1[1], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s9
-; CHECK-NEXT:    vmov.16 q1[2], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s2
-; CHECK-NEXT:    vmov.16 q1[3], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s12, s10
-; CHECK-NEXT:    vmov.16 q1[4], r1
-; CHECK-NEXT:    vmov r1, s12
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s3
-; CHECK-NEXT:    vmov.16 q1[5], r1
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vcvtb.f16.f32 s0, s11
-; CHECK-NEXT:    vmov.16 q1[6], r1
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vmov.16 q1[7], r1
-; CHECK-NEXT:    vstrw.32 q1, [r0]
-; CHECK-NEXT:    vpop {d8, d9, d10}
+; CHECK-NEXT:    vcvtb.f16.f32 q1, q1
+; CHECK-NEXT:    vcvtb.f16.f32 q0, q0
+; CHECK-NEXT:    vcvtt.f16.f32 q1, q3
+; CHECK-NEXT:    vcvtt.f16.f32 q0, q2
+; CHECK-NEXT:    vstrw.32 q1, [r0, #16]
+; CHECK-NEXT:    vstrw.32 q0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %strided.vec = shufflevector <8 x float> %val1, <8 x float> %val2, <16 x i32> <i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11, i32 4, i32 12, i32 5, i32 13, i32 6, i32 14, i32 7, i32 15>
   %out = fptrunc <16 x float> %strided.vec to <16 x half>
   store <16 x half> %out, <16 x half>* %src, align 4
   ret void
 }