[ARM][MVE] Enable extending masked loads

[samparker]

Summary

Allow us to generate sext/zext masked loads which can access v4i8, v8i8 and v4i16 memory to produce v4i32, v8i16 and v4i32 respectively.

My little (endian) brain only really works in that mode, so I'm dubious about the big endian support here.

[dmgreen]

Nice one! Looks very useful. I _think_ bigendian should be fine here, so long as we don't use the wrong type.

There are a lot of other masked load tests in mve-masked-load.ll. I think we should add the same for widening loads and narrowing stores. There should be tests for things like align1 and different passthru values. We might want extra tests for odd types too, if we are making them legal through isLegalMaskedLoad.

lib/Target/ARM/ARMISelLowering.cpp

13760

How come this isn't in target independent code? I would expect this combine not to be MVE specific, so long as it's legal. I'm not sure if there are ways currently to check if a "widening masked load" is legal or not, in the same way as there are for normal loads.

lib/Target/ARM/ARMInstrMVE.td

5019

I think these (and perhaps the ones above, tbh) should maybe need "let ScalarMemoryVT = i8;". To ensure they are extending from the correct types?

lib/Target/ARM/ARMTargetTransformInfo.cpp

494

I think changes here might mean we need to handle stores too, or (temporarily) split the two out from one another.

506

What does this mean for a v2i32, or other weird types (for us)?

test/CodeGen/Thumb2/mve-masked-ldst.ll

112

This looks odd to me, with the vpsel. There is legalising code in LowerMLOAD, which might be doing something wrong.

191

This is wrong at the moment? Same for all the other masked stores?

[samparker]

Thanks for those points, I'll add loads more tests.

lib/Target/ARM/ARMISelLowering.cpp

13760

good point.

lib/Target/ARM/ARMInstrMVE.td

5019

I'll give it a go.

test/CodeGen/Thumb2/mve-masked-ldst.ll

112

Is the vpsel not just handling the predicate on the store?

191

Yes? I hadn't looked at stores but it looks like these should now be vstrb.16.

[samparker]

• Moved the combine into generic dagcombine.
• Now checking memory alignment to decide legality.
• Not allowing v2 vectors.
• Masked load patterns are now explicitly either aligned or unaligned.
• Added more tests.

[dmgreen]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283 ↗

(On Diff #223172)

Is it true that whenever you have a legal extending load, you will also have the equivalent legal extending masked load? (For MVE we do, but is that true for all archs?)

Do we need to add an extra set of flags for this? Or is isVectorLoadExtDesirable good enough to handle these cases when there is an asymmetry?

lib/Target/ARM/ARMInstrMVE.td

5203

t2addrmode_imm7<0> -> t2addrmode_imm7<1>, for a VLDRH. Same below.

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

If this is coming from codegen, can the alignment here be 0? I think in ISel it is always set (and clang will always set it), but it may not be guaranteed in llvm in general.

test/CodeGen/Thumb2/mve-masked-load.ll

551 ↗

(On Diff #223172)

I don't think this vpsel should be here (it's not wrong, just inefficient, the instruction will already to this setting off predicated lanes to 0).

I'm guessing that the LowerMLOAD is creating a zero vector (that is potentially the wrong type?), so when it is called on the newly created maskedload it doesn't recognise it as 0 and we end up with the vselect being added too.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283 ↗

(On Diff #223172)

Yes, we can't expect that it's true for everything. I don't understand why the APIs generally like to pass lots of arguments instead of just passing, say the load that you'd want to inspect... So hopefully both these calls will cover all cases and I'd like to avoid adding another flag. That or I could just change isLoadExtLegal to take the LoadSDNode, but I've assumed these calls are designed like they are for reason...

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

I can't see anything in the spec for any guarantees of these intrinsics, but for normal loads, it becomes defined by the target ABI. It's always safe for us to use a i8* accessor, so I don't see 0 being a problem here.

test/CodeGen/Thumb2/mve-masked-load.ll

551 ↗

(On Diff #223172)

I'll have a look.

[samparker]

Now handling the a bitcast passthru value in LowerMLOAD. Corrected the half load addr values.

[samparker]

I had missed the shift value on the input patterns.

[dmgreen]

Nice. I think this is looking good, just some details to sort out, like what to do about the target independent parts.

We will presumably want to add the pre and post inc to these in the future too, which will probably bring up the same kinds of questions.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283 ↗

(On Diff #223172)

They refer back to the LoadExtActions, which are set by setLoadExtAction in ISel. We may need more flags on there to specify the difference between the masked loads and the normal loads.

lib/Target/ARM/ARMISelLowering.cpp

8887

This is creating a zero vector of size VT, which is the size of what the masked loads returns. Should it instead be the size of the memory being loaded (because the extend happens to the passthru as well)? What happens if that isn't a legal value type?

lib/Target/ARM/ARMInstrMVE.td

5196

There likely needs to be an anyext too. Can (or is it beneficial for) these be merged into the MVEExtLoad multiclass below?

5203

Edit: You beat me to it. Can you add some tests?

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

Yeah. Alignment of 0 means ABI alignment, which means 8, not unaligned.

I think it may be better to just check this alignment is always the case, getting rid of that weird "use i8's to load unaligned masked loads" thing. That was probably a bad idea, more trouble than it's worth.

I think what will happen here at the moment is that the Vectorizer will call isLegalMaskedLoad with an scalar type and an alignment (which, lets say is unaligned). That alignment won't be checked so the masked loads and stores will be created. Then when we get to the backend the legalizer will call this with a vector type and we'll hit this check, expanding out the masked load into a that very inefficient bunch of code. Which is probably something that we want to avoid.

test/CodeGen/Thumb2/mve-masked-load.ll

903 ↗

(On Diff #223220)

Nice :)

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887

Well, surely the result VT of the masked load has to match the VT of the passthru input. passthru is not about what memory is accessed, but what is written to the destination register. VOVIMM will also generate the same zero value for all full width vector types so for vector widths less than 128-bits, the higher elements will be zeroed and that makes sense. For vectors wider than 128-bits, I think something would have gone before here. I'll add some tests for both these cases.

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

Hmmm, okay. I also can't see removing unaligned support having a big negative effect. Sounds like I need to add some vectorization tests too, unless we already have them?

[samparker]

lib/Target/ARM/ARMInstrMVE.td

5196

As much as I don't like copy-paste, I do appreciate being able to read the code! I think adding to that multiclass is more hassle than it's worth :)

[dmgreen]

lib/Target/ARM/ARMISelLowering.cpp

8887

Hmmm. Yeah OK. I see. The PassThru is explicitly extended in tryToFoldExtOfMaskedLoad?

That makes sense, and the tests look OK. (There's one that is both sext and zext the same value, but that looks correct for where it is used).

Test for masked loads/stores longer than 128 bits sounds like a good idea. We should ideally be able to deal with longer vector by splitting them just fine.

lib/Target/ARM/ARMInstrMVE.td

5196

Ha, Fair. I will agree with you there that sometimes more code is simpler.

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

There was one added to the vectoriser tests, but not for alignment checks as far as I remember.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887

At some point, I was extending passthru... but it seems that is no longer the case! Our VMOVIMM is probably keeping us correct and if I extend it in dag combine, hopefully we won't need the bitcast handling here anymore.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887

Ah, no. I was just being blind, passthru is extended.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283 ↗

(On Diff #223172)

I had a look and I don't see how we could add an extra flag here. isLoadExtLegal will return true if the operation has been marked as legal, but for both targets (arm, x86) the MLOAD and MSTORE operations are set as custom. So I think having both calls will be necessary, with isVectorLoadExtDesirable enabling the fine grained control that we need.

[samparker]

• Rebased so we're now using MaybeAlign.
• Removed codegen support for unaligned masked loads.
• Added anyext PatFrags.
• Added tests for wider than 128-bit vectors.
• Added loop vectorize tests for unaligned accesses.

[samparker]

@craig.topper This patch currently causes an isel failure for pr35443.ll when an v4i8 masked load is being zero extended into an v4i64. I know nothing about AVX, could you please advise whether this operation is supported or how to address the issue? Thanks.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9308 ↗

(On Diff #224835)

Is this function missing a one use check?

9311 ↗

(On Diff #224835)

Would a masked load ever not be a vector type?

9318 ↗

(On Diff #224835)

What if the masked load is already an extending load?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9324 ↗

(On Diff #224835)

Should we disable this if the load is an expanding load? X86 supports expanding loads, but not extending expanding loads. Or is the expectation that I should block that in X86's implementation of isVectorLoadExtDesirable?

[craig.topper]

In D68337#1707938, @samparker wrote:

@craig.topper This patch currently causes an isel failure for pr35443.ll when an v4i8 masked load is being zero extended into an v4i64. I know nothing about AVX, could you please advise whether this operation is supported or how to address the issue? Thanks.

Its supported. We need to add isel patterns for all the extend flavors. Or we need to block masked loads in isVectorLoadExtDesirable on X86 for now.

[samparker]

Thanks @craig.topper. I'll add the necessary changes into the X86 backend.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9324 ↗

(On Diff #224835)

Good point... I think it makes sense for this part to be as generic as possible and leave it to the backends. We'll have to do the same too.

[samparker]

• Addressed comments in the dag combiner.
• Changed x86 backend so that extending masked loads are not desirable.
• Changed arm backend so that expanding extending masked loads are not desirable.
• Added more tests.

[craig.topper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9312 ↗

(On Diff #225007)

Check the extension type is NON_EXT explicitly. Don't rely on it being encoding 0.

9328 ↗

(On Diff #225007)

I don't think this line is needed. Returning NewLoad should take care of it. The line that replaces SDValue(Ld, 1) is needed though.

[dmgreen]

lib/Target/ARM/ARMISelLowering.cpp

14707

MVE doesn't support expanding loads, so it would be surprising if we did see one here. Having the check is good though.

lib/Target/ARM/ARMTargetTransformInfo.cpp

501

How rare do you think loads with no explicit alignment to be? I think they don't come up from clang, but is it worth leaving them till later?

As far as I understand, if the alignment on the load was missing (value is 0), it is treated as the abi/pref alignment from the datalayout. So will have an alignment of 8 or 16, so will be aligned.

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

501

ok.

[samparker]

Addressed comments.

[dmgreen]

LGTM. If Craig is happy with the rest.

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

We still probably want to stop i64's and other types.

Maybe do it like "(EltWidth == 32 && (!Alignment || Alignment >= 4)) || ..."

[craig.topper]

I'm happy.

lib/Target/ARM/ARMISelLowering.cpp

 
   SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other,
                                  SDValue(NewLoad1.getNode(), 1),
                                  SDValue(NewLoad2.getNode(), 1));
   DAG.ReplaceAllUsesOfValueWith(SDValue(LD, 1), NewChain);
   return DAG.getNode(ISD::CONCAT_VECTORS, DL, ToVT, NewLoad1, NewLoad2);
 }
 
+static SDValue CombineExtendingMaskedLoad(SDNode *Ext, SelectionDAG &DAG) {
+  MaskedLoadSDNode *Ld = dyn_cast<MaskedLoadSDNode>(Ext->getOperand(0));
+  if (!Ld)
+    return SDValue();
+
+  EVT VT = Ext->getValueType(0);
+  SDLoc dl(Ld);
+  SDValue PassThru = DAG.getNode(Ext->getOpcode(), dl, VT, Ld->getPassThru());
+  ISD::LoadExtType ExtTy = Ext->getOpcode() == ISD::SIGN_EXTEND ?
+    ISD::SEXTLOAD : ISD::ZEXTLOAD;
+
+  SDValue NewLoad = DAG.getMaskedLoad(VT, dl, Ld->getChain(),
+                                      Ld->getBasePtr(), Ld->getMask(),
+                                      PassThru, Ld->getMemoryVT(),
+                                      Ld->getMemOperand(), ExtTy,
+                                      Ld->isExpandingLoad());
+  DAG.ReplaceAllUsesOfValueWith(SDValue(Ext, 0), SDValue(NewLoad.getNode(), 0));
+  DAG.ReplaceAllUsesOfValueWith(SDValue(Ld, 1), SDValue(NewLoad.getNode(), 1));
+  return NewLoad;
+}
+
 /// PerformExtendCombine - Target-specific DAG combining for ISD::SIGN_EXTEND,
 /// ISD::ZERO_EXTEND, and ISD::ANY_EXTEND.
 static SDValue PerformExtendCombine(SDNode *N, SelectionDAG &DAG,
                                     const ARMSubtarget *ST) {
   SDValue N0 = N->getOperand(0);
 
   // Check for sign- and zero-extensions of vector extract operations of 8-
   // and 16-bit vector elements.  NEON supports these directly.  They are

       case ISD::ANY_EXTEND:
         Opc = ARMISD::VGETLANEu;
         break;
       }
       return DAG.getNode(Opc, SDLoc(N), VT, Vec, Lane);
     }
   }
 
-  if (ST->hasMVEIntegerOps())
+  if (ST->hasMVEIntegerOps()) {
     if (SDValue NewLoad = PerformSplittingToWideningLoad(N, DAG))
       return NewLoad;
+    if (SDValue NewLoad = CombineExtendingMaskedLoad(N, DAG))
+      return NewLoad;
+  }
 
   return SDValue();
 }
 
 static const APInt *isPowerOf2Constant(SDValue V) {
   ConstantSDNode *C = dyn_cast<ConstantSDNode>(V);
   if (!C)
     return nullptr;

   case ARMISD::UMLAL:   return PerformUMLALCombine(N, DCI.DAG, Subtarget);
   case ISD::ADD:        return PerformADDCombine(N, DCI, Subtarget);
   case ISD::SUB:        return PerformSUBCombine(N, DCI);
   case ISD::MUL:        return PerformMULCombine(N, DCI, Subtarget);
   case ISD::OR:         return PerformORCombine(N, DCI, Subtarget);
   case ISD::XOR:        return PerformXORCombine(N, DCI, Subtarget);
   case ISD::AND:        return PerformANDCombine(N, DCI, Subtarget);
   case ISD::BRCOND:
   case ISD::BR_CC:      return PerformHWLoopCombine(N, DCI, Subtarget);

lib/Target/ARM/ARMInstrMVE.td

 def alignedmaskedstore32 : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                                    (masked_st node:$val, node:$ptr, node:$pred), [{
   return cast<MaskedStoreSDNode>(N)->getAlignment() >= 4;
 }]>;
 def alignedmaskedstore16 : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                                    (masked_st node:$val, node:$ptr, node:$pred), [{
   return cast<MaskedStoreSDNode>(N)->getAlignment() >= 2;
 }]>;
+def sextmaskedload8 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                              (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  return Ld->getExtensionType() == ISD::SEXTLOAD &&
+         Ld->getAlignment() == 1;
+}]>;
+
+def sextmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                               (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  return Ld->getExtensionType() == ISD::SEXTLOAD &&
+         Ld->getAlignment() >= 2;
+}]>;
+
+def zextmaskedload8 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                              (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  return Ld->getExtensionType() == ISD::ZEXTLOAD &&
+         Ld->getAlignment() == 1;
+}]>;
+
+def zextmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                               (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  return Ld->getExtensionType() == ISD::ZEXTLOAD &&
+         Ld->getAlignment() >= 2;
+}]>;
 def maskedstore : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                           (masked_st node:$val, node:$ptr, node:$pred)>;
 
 let Predicates = [HasMVEInt, IsLE] in {
   // Stores
   defm : MVE_vector_store<MVE_VSTRBU8, byte_alignedstore, 0>;
   defm : MVE_vector_store<MVE_VSTRHU16, hword_alignedstore, 1>;
   defm : MVE_vector_store<MVE_VSTRWU32, alignedstore32, 2>;

   def : MVE_vector_maskedstore_typed<v4i32, MVE_VSTRWU32, alignedmaskedstore32, 2>;
   def : MVE_vector_maskedstore_typed<v4f32, MVE_VSTRWU32, alignedmaskedstore32, 2>;
   // Aligned masked loads
   def : MVE_vector_maskedload_typed<v16i8, MVE_VLDRBU8, maskedload, 0>;
   def : MVE_vector_maskedload_typed<v8i16, MVE_VLDRHU16, alignedmaskedload16, 1>;
   def : MVE_vector_maskedload_typed<v8f16, MVE_VLDRHU16, alignedmaskedload16, 1>;
   def : MVE_vector_maskedload_typed<v4i32, MVE_VLDRWU32, alignedmaskedload32, 2>;
   def : MVE_vector_maskedload_typed<v4f32, MVE_VLDRWU32, alignedmaskedload32, 2>;
+  // Extending masked loads.
+  def : Pat<(v8i16 (sextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
+            (v8i16 (MVE_VLDRBS16 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (sextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRBS32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (sextmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRHS32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v8i16 (zextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
+            (v8i16 (MVE_VLDRBU16 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (zextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRBU32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (zextmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRHU32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
 }
 
 // Widening/Narrowing Loads/Stores
 
 let MinAlignment = 2 in {
   def truncstorevi16_align2 : PatFrag<(ops node:$val, node:$ptr),
                                       (truncstorevi16 node:$val, node:$ptr)>;
   def post_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),
                                         (post_truncstvi16 node:$val, node:$base, node:$offset)>;
   def pre_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),
                                        (pre_truncstvi16 node:$val, node:$base, node:$offset)>;
 }
 
 let Predicates = [HasMVEInt] in {
   def : Pat<(truncstorevi8 (v8i16 MQPR:$val), taddrmode_imm7<0>:$addr),
             (MVE_VSTRB16 MQPR:$val, taddrmode_imm7<0>:$addr)>;
   def : Pat<(truncstorevi8 (v4i32 MQPR:$val), taddrmode_imm7<0>:$addr),
             (MVE_VSTRB32 MQPR:$val, taddrmode_imm7<0>:$addr)>;

lib/Target/ARM/ARMTargetTransformInfo.cpp

if (ST->hasNEON()) {
 
     // In many cases the address computation is not merged into the instruction
     // addressing mode.
     return 1;
   }
   return BaseT::getAddressComputationCost(Ty, SE, Ptr);
 }
 
 bool ARMTTIImpl::isLegalMaskedLoad(Type *DataTy) {
   if (!EnableMaskedLoadStores || !ST->hasMVEIntegerOps())
     return false;
 
-  if (DataTy->isVectorTy()) {
-    // We don't yet support narrowing or widening masked loads/stores. Expand
-    // them for the moment.
+  if (auto *VecTy = dyn_cast<VectorType>(DataTy)) {
     unsigned VecWidth = DataTy->getPrimitiveSizeInBits();
-    if (VecWidth != 128)
+
+    // We don't support extending fp types.
+    if (VecTy->getElementType()->isFloatingPointTy() && VecWidth != 128)
+      return false;
+
+    // sext/zext integers.
+    if (VecWidth > 128)
       return false;
   }
 
   unsigned EltWidth = DataTy->getScalarSizeInBits();
   return EltWidth == 32 || EltWidth == 16 || EltWidth == 8;
 }
 
 int ARMTTIImpl::getMemcpyCost(const Instruction *I) {
   const MemCpyInst *MI = dyn_cast<MemCpyInst>(I);
   assert(MI && "MemcpyInst expected");
   ConstantInt *C = dyn_cast<ConstantInt>(MI->getLength());
 
   // To model the cost of a library call, we assume 1 for the call, and

test/CodeGen/Thumb2/LowOverheadLoops/mve-tail-data-types.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve -disable-mve-tail-predication=false -enable-arm-maskedldst=true %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_char(i8 zeroext %a, i8* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_char:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI0_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q4, #0x0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI0_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q3, q4
-; CHECK-NEXT:    vadd.i32 q4, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    add.w r3, r1, r0
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r4, [r3]
-; CHECK-NEXT:    vmovne.32 q4[0], r4
-; CHECK-NEXT:    lsls r4, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r4
-; CHECK-NEXT:    lsls r4, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r4
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r2, [r3, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r2
-; CHECK-NEXT:    vand q5, q4, q2
-; CHECK-NEXT:    vmov q4, q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmla.u32 q4, q5, r12
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r3, r1, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrbt.u32 q4, [r3]
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q4, q3
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    vpop {d8, d9}
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_short(i16 signext %a, i16* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_short:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vpush {d8, d9}
-; CHECK-NEXT:    sub sp, #8
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI1_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q3, #0x0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI1_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q2, q3
-; CHECK-NEXT:    vadd.i32 q3, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q3
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r3
-; CHECK-NEXT:    lsls r3, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r3
-; CHECK-NEXT:    lsls r3, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r3
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r2, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r2
-; CHECK-NEXT:    vmovlb.s16 q4, q3
-; CHECK-NEXT:    vmov q3, q2
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrht.s32 q4, [r1]
 ; CHECK-NEXT:    adds r1, #8
-; CHECK-NEXT:    vmla.u32 q3, q4, r12
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q3, q2
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
 ; CHECK-NEXT:    vpop {d8, d9}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI1_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_uchar(i8 zeroext %a, i8* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_uchar:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI2_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q4, #0x0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI2_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q3, q4
-; CHECK-NEXT:    vadd.i32 q4, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    add.w r3, r1, r0
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r4, [r3]
-; CHECK-NEXT:    vmovne.32 q4[0], r4
-; CHECK-NEXT:    lsls r4, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r4
-; CHECK-NEXT:    lsls r4, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r4
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r2, [r3, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r2
-; CHECK-NEXT:    vand q5, q4, q2
-; CHECK-NEXT:    vmov q4, q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmla.u32 q4, q5, r12
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r3, r1, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrbt.u32 q4, [r3]
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q4, q3
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    vpop {d8, d9}
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI2_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_ushort(i16 signext %a, i16* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_ushort:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vpush {d8, d9}
-; CHECK-NEXT:    sub sp, #8
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI3_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q3, #0x0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI3_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q2, q3
-; CHECK-NEXT:    vadd.i32 q3, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q3
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r3
-; CHECK-NEXT:    lsls r3, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r3
-; CHECK-NEXT:    lsls r3, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r3
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r2, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r2
-; CHECK-NEXT:    vmovlb.u16 q4, q3
-; CHECK-NEXT:    vmov q3, q2
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrht.u32 q4, [r1]
 ; CHECK-NEXT:    adds r1, #8
-; CHECK-NEXT:    vmla.u32 q3, q4, r12
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q3, q2
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
 ; CHECK-NEXT:    vpop {d8, d9}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI3_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %7, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_char(i8* nocapture readonly %a, i8* nocapture readonly %b, i8 zeroext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_char:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #72]
-; CHECK-NEXT:    cmp.w r12, #0
+; CHECK-NEXT:    ldr r7, [sp, #28]
+; CHECK-NEXT:    cmp r7, #0
 ; CHECK-NEXT:    beq.w .LBB5_12
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.lr.ph
-; CHECK-NEXT:    add.w r5, r3, r12, lsl #2
-; CHECK-NEXT:    add.w r6, r1, r12
-; CHECK-NEXT:    cmp r5, r1
-; CHECK-NEXT:    add.w r4, r0, r12
-; CHECK-NEXT:    cset r7, hi
-; CHECK-NEXT:    cmp r6, r3
-; CHECK-NEXT:    cset r6, hi
-; CHECK-NEXT:    cmp r5, r0
+; CHECK-NEXT:    add.w r4, r3, r7, lsl #2
+; CHECK-NEXT:    adds r5, r1, r7
+; CHECK-NEXT:    cmp r4, r1
+; CHECK-NEXT:    add.w r6, r0, r7
+; CHECK-NEXT:    cset r12, hi
+; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    cset r5, hi
-; CHECK-NEXT:    cmp r4, r3
+; CHECK-NEXT:    cmp r4, r0
 ; CHECK-NEXT:    cset r4, hi
-; CHECK-NEXT:    ands r5, r4
-; CHECK-NEXT:    lsls r5, r5, #31
+; CHECK-NEXT:    cmp r6, r3
+; CHECK-NEXT:    cset r6, hi
+; CHECK-NEXT:    ands r6, r4
+; CHECK-NEXT:    lsls r6, r6, #31
 ; CHECK-NEXT:    itt eq
-; CHECK-NEXT:    andeq r7, r6
-; CHECK-NEXT:    lslseq.w r7, r7, #31
+; CHECK-NEXT:    andeq.w r6, r5, r12
+; CHECK-NEXT:    lslseq.w r6, r6, #31
 ; CHECK-NEXT:    beq .LBB5_4
 ; CHECK-NEXT:  @ %bb.2: @ %for.body.preheader
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    and lr, r12, #3
-; CHECK-NEXT:    cmp r4, #3
-; CHECK-NEXT:    bhs.w .LBB5_6
+; CHECK-NEXT:    subs r6, r7, #1
+; CHECK-NEXT:    and lr, r7, #3
+; CHECK-NEXT:    cmp r6, #3
+; CHECK-NEXT:    bhs .LBB5_6
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    b .LBB5_9
 ; CHECK-NEXT:  .LBB5_4: @ %vector.ph
-; CHECK-NEXT:    add.w r7, r12, #3
-; CHECK-NEXT:    adr r5, .LCPI5_0
-; CHECK-NEXT:    bic r7, r7, #3
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    subs r7, #4
-; CHECK-NEXT:    movs r6, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
-; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
-; CHECK-NEXT:    movs r4, #0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vmov.i32 q3, #0xff
+; CHECK-NEXT:    adds r6, r7, #3
+; CHECK-NEXT:    movs r5, #1
+; CHECK-NEXT:    bic r6, r6, #3
+; CHECK-NEXT:    subs r7, #1
+; CHECK-NEXT:    subs r6, #4
+; CHECK-NEXT:    vdup.32 q0, r7
+; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    add.w lr, r5, r6, lsr #2
+; CHECK-NEXT:    adr r6, .LCPI5_0
+; CHECK-NEXT:    vldrw.u32 q1, [r6]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB5_5: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q4, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q5
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r0, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q4[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vand q4, q4, q2
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r1, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q5[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q5[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q5[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q5[3], r5
-; CHECK-NEXT:    vand q5, q5, q3
-; CHECK-NEXT:    vctp.32 r12
-; CHECK-NEXT:    vmul.i32 q4, q5, q4
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vadd.i32 q4, q4, r2
+; CHECK-NEXT:    vadd.i32 q2, q1, r7
+; CHECK-NEXT:    adds r4, r0, r7
+; CHECK-NEXT:    vpt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrbt.u32 q2, [r4]
+; CHECK-NEXT:    adds r4, r1, r7
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q4, [r3]
+; CHECK-NEXT:    vldrbt.u32 q3, [r4]
+; CHECK-NEXT:    vmul.i32 q2, q3, q2
+; CHECK-NEXT:    vadd.i32 q2, q2, r2
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
+; CHECK-NEXT:    adds r7, #4
 ; CHECK-NEXT:    le lr, .LBB5_5
 ; CHECK-NEXT:    b .LBB5_12
 ; CHECK-NEXT:  .LBB5_6: @ %for.body.preheader.new
-; CHECK-NEXT:    sub.w r12, lr, r12
+; CHECK-NEXT:    sub.w r12, lr, r7
 ; CHECK-NEXT:    subs r4, r1, #3
 ; CHECK-NEXT:    subs r5, r0, #3
 ; CHECK-NEXT:    sub.w r7, r3, #16
 ; CHECK-NEXT:    mov.w r9, #0
 ; CHECK-NEXT:  .LBB5_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb.w r8, [r5, #3]
 ; CHECK-NEXT:    sub.w r9, r9, #4

 ; CHECK-NEXT:  .LBB5_11: @ %for.body.epil
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r7, [r0, #1]!
 ; CHECK-NEXT:    ldrb r6, [r1, #1]!
 ; CHECK-NEXT:    smlabb r7, r6, r7, r2
 ; CHECK-NEXT:    str r7, [r3, #4]!
 ; CHECK-NEXT:    le lr, .LBB5_11
 ; CHECK-NEXT:  .LBB5_12: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.13:
 ; CHECK-NEXT:  .LCPI5_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
for.body:                                         ; preds = %for.body, %for.body.preheader.new
   %niter.nsub.3 = add i32 %niter, -4
   %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
   br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_short(i16* nocapture readonly %a, i16* nocapture readonly %b, i16 signext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_short:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #28]
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    ldr.w r12, [sp, #8]
 ; CHECK-NEXT:    cmp.w r12, #0
-; CHECK-NEXT:    beq.w .LBB6_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
-; CHECK-NEXT:    add.w r5, r12, #3
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    popeq {r4, pc}
+; CHECK-NEXT:    add.w lr, r12, #3
 ; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    bic r5, r5, #3
-; CHECK-NEXT:    subs r5, #4
-; CHECK-NEXT:    add.w lr, r4, r5, lsr #2
-; CHECK-NEXT:    adr r5, .LCPI6_0
+; CHECK-NEXT:    bic lr, lr, #3
+; CHECK-NEXT:    sub.w lr, lr, #4
+; CHECK-NEXT:    add.w lr, r4, lr, lsr #2
 ; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
 ; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    movs r4, #0
+; CHECK-NEXT:    adr r4, .LCPI6_0
+; CHECK-NEXT:    vldrw.u32 q1, [r4]
+; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB6_2: @ %vector.body
+; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q2, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vcmp.u32 cs, q0, q2
-; CHECK-NEXT:    @ implicit-def: $q2
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r0]
-; CHECK-NEXT:    vmovne.32 q2[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #2]
-; CHECK-NEXT:    vmovmi.32 q2[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #4]
-; CHECK-NEXT:    vmovmi.32 q2[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r0, #6]
-; CHECK-NEXT:    vmovmi.32 q2[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vmovlb.s16 q2, q2
+; CHECK-NEXT:    vadd.i32 q2, q1, r12
+; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    vptt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrht.s32 q2, [r0]
+; CHECK-NEXT:    vldrht.s32 q3, [r1]
 ; CHECK-NEXT:    adds r0, #8
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r5
-; CHECK-NEXT:    vmovlb.s16 q3, q3
-; CHECK-NEXT:    vctp.32 r12
 ; CHECK-NEXT:    vmul.i32 q2, q3, q2
 ; CHECK-NEXT:    adds r1, #8
 ; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
-; CHECK-NEXT:    le lr, .LBB6_2
-; CHECK-NEXT:  .LBB6_3: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    le lr, .LBB6_1
+; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
-; CHECK-NEXT:  @ %bb.4:
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI6_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %vector.ph

 for.cond.cleanup:                                 ; preds = %vector.body, %entry
   ret void
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_uchar(i8* nocapture readonly %a, i8* nocapture readonly %b, i8 zeroext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_uchar:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #72]
-; CHECK-NEXT:    cmp.w r12, #0
+; CHECK-NEXT:    ldr r7, [sp, #28]
+; CHECK-NEXT:    cmp r7, #0
 ; CHECK-NEXT:    beq.w .LBB7_12
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.lr.ph
-; CHECK-NEXT:    add.w r5, r3, r12, lsl #2
-; CHECK-NEXT:    add.w r6, r1, r12
-; CHECK-NEXT:    cmp r5, r1
-; CHECK-NEXT:    add.w r4, r0, r12
-; CHECK-NEXT:    cset r7, hi
-; CHECK-NEXT:    cmp r6, r3
-; CHECK-NEXT:    cset r6, hi
-; CHECK-NEXT:    cmp r5, r0
+; CHECK-NEXT:    add.w r4, r3, r7, lsl #2
+; CHECK-NEXT:    adds r5, r1, r7
+; CHECK-NEXT:    cmp r4, r1
+; CHECK-NEXT:    add.w r6, r0, r7
+; CHECK-NEXT:    cset r12, hi
+; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    cset r5, hi
-; CHECK-NEXT:    cmp r4, r3
+; CHECK-NEXT:    cmp r4, r0
 ; CHECK-NEXT:    cset r4, hi
-; CHECK-NEXT:    ands r5, r4
-; CHECK-NEXT:    lsls r5, r5, #31
+; CHECK-NEXT:    cmp r6, r3
+; CHECK-NEXT:    cset r6, hi
+; CHECK-NEXT:    ands r6, r4
+; CHECK-NEXT:    lsls r6, r6, #31
 ; CHECK-NEXT:    itt eq
-; CHECK-NEXT:    andeq r7, r6
-; CHECK-NEXT:    lslseq.w r7, r7, #31
+; CHECK-NEXT:    andeq.w r6, r5, r12
+; CHECK-NEXT:    lslseq.w r6, r6, #31
 ; CHECK-NEXT:    beq .LBB7_4
 ; CHECK-NEXT:  @ %bb.2: @ %for.body.preheader
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    and lr, r12, #3
-; CHECK-NEXT:    cmp r4, #3
-; CHECK-NEXT:    bhs.w .LBB7_6
+; CHECK-NEXT:    subs r6, r7, #1
+; CHECK-NEXT:    and lr, r7, #3
+; CHECK-NEXT:    cmp r6, #3
+; CHECK-NEXT:    bhs .LBB7_6
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    b .LBB7_9
 ; CHECK-NEXT:  .LBB7_4: @ %vector.ph
-; CHECK-NEXT:    add.w r7, r12, #3
-; CHECK-NEXT:    adr r5, .LCPI7_0
-; CHECK-NEXT:    bic r7, r7, #3
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    subs r7, #4
-; CHECK-NEXT:    movs r6, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
-; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
-; CHECK-NEXT:    movs r4, #0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vmov.i32 q3, #0xff
+; CHECK-NEXT:    adds r6, r7, #3
+; CHECK-NEXT:    movs r5, #1
+; CHECK-NEXT:    bic r6, r6, #3
+; CHECK-NEXT:    subs r7, #1
+; CHECK-NEXT:    subs r6, #4
+; CHECK-NEXT:    vdup.32 q0, r7
+; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    add.w lr, r5, r6, lsr #2
+; CHECK-NEXT:    adr r6, .LCPI7_0
+; CHECK-NEXT:    vldrw.u32 q1, [r6]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB7_5: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q4, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q5
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r0, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q4[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vand q4, q4, q2
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r1, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q5[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q5[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q5[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q5[3], r5
-; CHECK-NEXT:    vand q5, q5, q3
-; CHECK-NEXT:    vctp.32 r12
-; CHECK-NEXT:    vmul.i32 q4, q5, q4
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vadd.i32 q4, q4, r2
+; CHECK-NEXT:    vadd.i32 q2, q1, r7
+; CHECK-NEXT:    adds r4, r0, r7
+; CHECK-NEXT:    vpt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrbt.u32 q2, [r4]
+; CHECK-NEXT:    adds r4, r1, r7
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrbt.u32 q3, [r4]
+; CHECK-NEXT:    vmul.i32 q2, q3, q2
+; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q4, [r3]
+; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
+; CHECK-NEXT:    adds r7, #4
 ; CHECK-NEXT:    le lr, .LBB7_5
 ; CHECK-NEXT:    b .LBB7_12
 ; CHECK-NEXT:  .LBB7_6: @ %for.body.preheader.new
-; CHECK-NEXT:    sub.w r12, lr, r12
+; CHECK-NEXT:    sub.w r12, lr, r7
 ; CHECK-NEXT:    subs r4, r1, #3
 ; CHECK-NEXT:    subs r5, r0, #3
 ; CHECK-NEXT:    sub.w r7, r3, #16
 ; CHECK-NEXT:    mov.w r9, #0
 ; CHECK-NEXT:  .LBB7_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb.w r8, [r5, #3]
 ; CHECK-NEXT:    sub.w r9, r9, #4

 ; CHECK-NEXT:  .LBB7_11: @ %for.body.epil
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r7, [r0, #1]!
 ; CHECK-NEXT:    ldrb r6, [r1, #1]!
 ; CHECK-NEXT:    smlabb r7, r6, r7, r2
 ; CHECK-NEXT:    str r7, [r3, #4]!
 ; CHECK-NEXT:    le lr, .LBB7_11
 ; CHECK-NEXT:  .LBB7_12: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.13:
 ; CHECK-NEXT:  .LCPI7_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
for.body:                                         ; preds = %for.body, %for.body.preheader.new
   %niter.nsub.3 = add i32 %niter, -4
   %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
   br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_ushort(i16* nocapture readonly %a, i16* nocapture readonly %b, i16 signext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_ushort:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #28]
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    ldr.w r12, [sp, #8]
 ; CHECK-NEXT:    cmp.w r12, #0
-; CHECK-NEXT:    beq.w .LBB8_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
-; CHECK-NEXT:    add.w r5, r12, #3
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    popeq {r4, pc}
+; CHECK-NEXT:    add.w lr, r12, #3
 ; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    bic r5, r5, #3
-; CHECK-NEXT:    subs r5, #4
-; CHECK-NEXT:    add.w lr, r4, r5, lsr #2
-; CHECK-NEXT:    adr r5, .LCPI8_0
+; CHECK-NEXT:    bic lr, lr, #3
+; CHECK-NEXT:    sub.w lr, lr, #4
+; CHECK-NEXT:    add.w lr, r4, lr, lsr #2
 ; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
 ; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    movs r4, #0
+; CHECK-NEXT:    adr r4, .LCPI8_0
+; CHECK-NEXT:    vldrw.u32 q1, [r4]
+; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB8_2: @ %vector.body
+; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q2, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vcmp.u32 cs, q0, q2
-; CHECK-NEXT:    @ implicit-def: $q2
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r0]
-; CHECK-NEXT:    vmovne.32 q2[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #2]
-; CHECK-NEXT:    vmovmi.32 q2[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #4]
-; CHECK-NEXT:    vmovmi.32 q2[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r0, #6]
-; CHECK-NEXT:    vmovmi.32 q2[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vmovlb.u16 q2, q2
+; CHECK-NEXT:    vadd.i32 q2, q1, r12
+; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    vptt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrht.u32 q2, [r0]
+; CHECK-NEXT:    vldrht.u32 q3, [r1]
 ; CHECK-NEXT:    adds r0, #8
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r5
-; CHECK-NEXT:    vmovlb.u16 q3, q3
-; CHECK-NEXT:    vctp.32 r12
 ; CHECK-NEXT:    vmul.i32 q2, q3, q2
 ; CHECK-NEXT:    adds r1, #8
 ; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
-; CHECK-NEXT:    le lr, .LBB8_2
-; CHECK-NEXT:  .LBB8_3: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    le lr, .LBB8_1
+; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
-; CHECK-NEXT:  @ %bb.4:
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI8_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %vector.ph

test/CodeGen/Thumb2/mve-masked-ldst.ll

entry:
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   call void @llvm.masked.store.v4i32(<4 x i32> %2, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v4i32_v4i8(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i8> *%src) {
 ; CHECK-LABEL: foo_sext_v4i32_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #3]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrbt.s32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 1, <4 x i1> %1, <4 x i8> undef)
   %3 = sext <4 x i8> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v4i32_v4i16(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i16> *%src) {
 ; CHECK-LABEL: foo_sext_v4i32_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r1, [r2, #6]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrht.s32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i16> @llvm.masked.load.v4i16(<4 x i16>* %src, i32 2, <4 x i1> %1, <4 x i16> undef)
   %3 = sext <4 x i16> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v4i32_v4i8(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i8> *%src) {
 ; CHECK-LABEL: foo_zext_v4i32_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #3]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vand q0, q0, q1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrbt.u32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 1, <4 x i1> %1, <4 x i8> undef)
   %3 = zext <4 x i8> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v4i32_v4i16(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i16> *%src) {
 ; CHECK-LABEL: foo_zext_v4i32_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r1, [r2, #6]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrht.u32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i16> @llvm.masked.load.v4i16(<4 x i16>* %src, i32 2, <4 x i1> %1, <4 x i16> undef)
   %3 = zext <4 x i16> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %src, i32 2, <8 x i1> %1, <8 x i16> undef)
   call void @llvm.masked.store.v8i16(<8 x i16> %2, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v8i16_v8i8(<8 x i16> *%dest, <8 x i16> *%mask, <8 x i8> *%src) {
-; CHECK-LABEL: foo_sext_v8i16_v8i8:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r3, lr, #1
-; CHECK-NEXT:    ubfx r1, lr, #2, #1
-; CHECK-NEXT:    rsb.w r12, r3, #0
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r12, #0, #1
-; CHECK-NEXT:    bfi r3, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, lr, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, lr, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #3, #1
-; CHECK-NEXT:    ubfx r1, lr, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #4, #1
-; CHECK-NEXT:    ubfx r1, lr, #10, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #5, #1
-; CHECK-NEXT:    ubfx r1, lr, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #6, #1
-; CHECK-NEXT:    ubfx r1, lr, #14, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r3, r3, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.16 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.16 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r3
-; CHECK-NEXT:    lsls r3, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #3]
-; CHECK-NEXT:    vmovmi.16 q0[3], r3
-; CHECK-NEXT:    lsls r3, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.16 q0[4], r3
-; CHECK-NEXT:    lsls r3, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #5]
-; CHECK-NEXT:    vmovmi.16 q0[5], r3
-; CHECK-NEXT:    lsls r3, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #6]
-; CHECK-NEXT:    vmovmi.16 q0[6], r3
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #7]
-; CHECK-NEXT:    vmovmi.16 q0[7], r1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-LE-LABEL: foo_sext_v8i16_v8i8:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-LE-NEXT:    vldrbt.s16 q1, [r2]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_sext_v8i16_v8i8:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-BE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-BE-NEXT:    vrev32.16 q0, q0
+; CHECK-BE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.s16 q1, [r2]
+; CHECK-BE-NEXT:    vpsel q0, q1, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 1, <8 x i1> %1, <8 x i8> undef)
   %3 = sext <8 x i8> %2 to <8 x i16>
   call void @llvm.masked.store.v8i16(<8 x i16> %3, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v8i16_v8i8(<8 x i16> *%dest, <8 x i16> *%mask, <8 x i8> *%src) {
-; CHECK-LABEL: foo_zext_v8i16_v8i8:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r3, lr, #1
-; CHECK-NEXT:    ubfx r1, lr, #2, #1
-; CHECK-NEXT:    rsb.w r12, r3, #0
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r12, #0, #1
-; CHECK-NEXT:    bfi r3, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, lr, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, lr, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #3, #1
-; CHECK-NEXT:    ubfx r1, lr, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #4, #1
-; CHECK-NEXT:    ubfx r1, lr, #10, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #5, #1
-; CHECK-NEXT:    ubfx r1, lr, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #6, #1
-; CHECK-NEXT:    ubfx r1, lr, #14, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r3, r3, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.16 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.16 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r3
-; CHECK-NEXT:    lsls r3, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #3]
-; CHECK-NEXT:    vmovmi.16 q0[3], r3
-; CHECK-NEXT:    lsls r3, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.16 q0[4], r3
-; CHECK-NEXT:    lsls r3, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #5]
-; CHECK-NEXT:    vmovmi.16 q0[5], r3
-; CHECK-NEXT:    lsls r3, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #6]
-; CHECK-NEXT:    vmovmi.16 q0[6], r3
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #7]
-; CHECK-NEXT:    vmovmi.16 q0[7], r1
-; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-LE-LABEL: foo_zext_v8i16_v8i8:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-LE-NEXT:    vldrbt.u16 q1, [r2]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_zext_v8i16_v8i8:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-BE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-BE-NEXT:    vrev32.16 q0, q0
+; CHECK-BE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.u16 q1, [r2]
+; CHECK-BE-NEXT:    vpsel q0, q1, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 1, <8 x i1> %1, <8 x i8> undef)
   %3 = zext <8 x i8> %2 to <8 x i16>
   call void @llvm.masked.store.v8i16(<8 x i16> %3, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
entry:
   %0 = load <16 x i8>, <16 x i8>* %mask, align 1
   %1 = icmp sgt <16 x i8> %0, zeroinitializer
   %2 = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %src, i32 1, <16 x i1> %1, <16 x i8> undef)
   call void @llvm.masked.store.v16i8(<16 x i8> %2, <16 x i8>* %dest, i32 1, <16 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v8i8_v8i16(<8 x i8> *%dest, <8 x i16> *%mask, <8 x i16> *%src) {
-; CHECK-LABEL: foo_trunc_v8i8_v8i16:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vpt.s16 gt, q0, zr
-; CHECK-NEXT:    vldrht.u16 q0, [r2]
-; CHECK-NEXT:    vmrs r1, p0
-; CHECK-NEXT:    and r2, r1, #1
-; CHECK-NEXT:    rsbs r3, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r1, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r1, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, r1, #6, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    ubfx r3, r1, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #4, #1
-; CHECK-NEXT:    ubfx r3, r1, #10, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #5, #1
-; CHECK-NEXT:    ubfx r3, r1, #12, #1
-; CHECK-NEXT:    ubfx r1, r1, #14, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r2
-; CHECK-NEXT:    lsls r2, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne.u16 r2, q0[0]
-; CHECK-NEXT:    strbne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[1]
-; CHECK-NEXT:    strbmi r2, [r0, #1]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[2]
-; CHECK-NEXT:    strbmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r2, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[3]
-; CHECK-NEXT:    strbmi r2, [r0, #3]
-; CHECK-NEXT:    lsls r2, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[4]
-; CHECK-NEXT:    strbmi r2, [r0, #4]
-; CHECK-NEXT:    lsls r2, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[5]
-; CHECK-NEXT:    strbmi r2, [r0, #5]
-; CHECK-NEXT:    lsls r2, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[6]
-; CHECK-NEXT:    strbmi r2, [r0, #6]
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r1, q0[7]
-; CHECK-NEXT:    strbmi r1, [r0, #7]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v8i8_v8i16:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.u16 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v8i8_v8i16:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrht.u16 q0, [r2]
+; CHECK-BE-NEXT:    vrev16.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %src, i32 2, <8 x i1> %1, <8 x i16> undef)
   %3 = trunc <8 x i16> %2 to <8 x i8>
   call void @llvm.masked.store.v8i8(<8 x i8> %3, <8 x i8>* %dest, i32 1, <8 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v4i8_v4i32(<4 x i8> *%dest, <4 x i32> *%mask, <4 x i32> *%src) {
-; CHECK-LABEL: foo_trunc_v4i8_v4i32:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vpt.s32 gt, q0, zr
-; CHECK-NEXT:    vldrwt.u32 q0, [r2]
-; CHECK-NEXT:    vmrs r2, p0
-; CHECK-NEXT:    and r1, r2, #1
-; CHECK-NEXT:    rsbs r3, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    bfi r1, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r2, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r2, #8, #1
-; CHECK-NEXT:    ubfx r2, r2, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r1, r2, #3, #1
-; CHECK-NEXT:    lsls r2, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne r2, s0
-; CHECK-NEXT:    strbne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s1
-; CHECK-NEXT:    strbmi r2, [r0, #1]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s2
-; CHECK-NEXT:    strbmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    strbmi r1, [r0, #3]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v4i8_v4i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v4i8_v4i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-BE-NEXT:    vrev32.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   %3 = trunc <4 x i32> %2 to <4 x i8>
   call void @llvm.masked.store.v4i8(<4 x i8> %3, <4 x i8>* %dest, i32 1, <4 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v4i16_v4i32(<4 x i16> *%dest, <4 x i32> *%mask, <4 x i32> *%src) {
-; CHECK-LABEL: foo_trunc_v4i16_v4i32:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vpt.s32 gt, q0, zr
-; CHECK-NEXT:    vldrwt.u32 q0, [r2]
-; CHECK-NEXT:    vmrs r2, p0
-; CHECK-NEXT:    and r1, r2, #1
-; CHECK-NEXT:    rsbs r3, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    bfi r1, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r2, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r2, #8, #1
-; CHECK-NEXT:    ubfx r2, r2, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r1, r2, #3, #1
-; CHECK-NEXT:    lsls r2, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne r2, s0
-; CHECK-NEXT:    strhne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s1
-; CHECK-NEXT:    strhmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s2
-; CHECK-NEXT:    strhmi r2, [r0, #4]
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    strhmi r1, [r0, #6]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v4i16_v4i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v4i16_v4i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-BE-NEXT:    vrev32.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   %3 = trunc <4 x i32> %2 to <4 x i16>
   call void @llvm.masked.store.v4i16(<4 x i16> %3, <4 x i16>* %dest, i32 2, <4 x i1> %1)
   ret void
 }