[ARM][MVE] Enable extending masked loads

[samparker]

Summary

Allow us to generate sext/zext masked loads which can access v4i8, v8i8 and v4i16 memory to produce v4i32, v8i16 and v4i32 respectively.

My little (endian) brain only really works in that mode, so I'm dubious about the big endian support here.

[dmgreen]

Nice one! Looks very useful. I _think_ bigendian should be fine here, so long as we don't use the wrong type.

There are a lot of other masked load tests in mve-masked-load.ll. I think we should add the same for widening loads and narrowing stores. There should be tests for things like align1 and different passthru values. We might want extra tests for odd types too, if we are making them legal through isLegalMaskedLoad.

lib/Target/ARM/ARMISelLowering.cpp

13760 ↗

(On Diff #222830)

How come this isn't in target independent code? I would expect this combine not to be MVE specific, so long as it's legal. I'm not sure if there are ways currently to check if a "widening masked load" is legal or not, in the same way as there are for normal loads.

lib/Target/ARM/ARMInstrMVE.td

5019

I think these (and perhaps the ones above, tbh) should maybe need "let ScalarMemoryVT = i8;". To ensure they are extending from the correct types?

lib/Target/ARM/ARMTargetTransformInfo.cpp

494

I think changes here might mean we need to handle stores too, or (temporarily) split the two out from one another.

511

What does this mean for a v2i32, or other weird types (for us)?

test/CodeGen/Thumb2/mve-masked-ldst.ll

742

This looks odd to me, with the vpsel. There is legalising code in LowerMLOAD, which might be doing something wrong.

821

This is wrong at the moment? Same for all the other masked stores?

[samparker]

Thanks for those points, I'll add loads more tests.

lib/Target/ARM/ARMISelLowering.cpp

13760 ↗

(On Diff #222830)

good point.

lib/Target/ARM/ARMInstrMVE.td

5019

I'll give it a go.

test/CodeGen/Thumb2/mve-masked-ldst.ll

742

Is the vpsel not just handling the predicate on the store?

821

Yes? I hadn't looked at stores but it looks like these should now be vstrb.16.

[samparker]

• Moved the combine into generic dagcombine.
• Now checking memory alignment to decide legality.
• Not allowing v2 vectors.
• Masked load patterns are now explicitly either aligned or unaligned.
• Added more tests.

[dmgreen]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283

Is it true that whenever you have a legal extending load, you will also have the equivalent legal extending masked load? (For MVE we do, but is that true for all archs?)

Do we need to add an extra set of flags for this? Or is isVectorLoadExtDesirable good enough to handle these cases when there is an asymmetry?

lib/Target/ARM/ARMInstrMVE.td

5203

t2addrmode_imm7<0> -> t2addrmode_imm7<1>, for a VLDRH. Same below.

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

If this is coming from codegen, can the alignment here be 0? I think in ISel it is always set (and clang will always set it), but it may not be guaranteed in llvm in general.

test/CodeGen/Thumb2/mve-masked-load.ll

551

I don't think this vpsel should be here (it's not wrong, just inefficient, the instruction will already to this setting off predicated lanes to 0).

I'm guessing that the LowerMLOAD is creating a zero vector (that is potentially the wrong type?), so when it is called on the newly created maskedload it doesn't recognise it as 0 and we end up with the vselect being added too.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283

Yes, we can't expect that it's true for everything. I don't understand why the APIs generally like to pass lots of arguments instead of just passing, say the load that you'd want to inspect... So hopefully both these calls will cover all cases and I'd like to avoid adding another flag. That or I could just change isLoadExtLegal to take the LoadSDNode, but I've assumed these calls are designed like they are for reason...

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

I can't see anything in the spec for any guarantees of these intrinsics, but for normal loads, it becomes defined by the target ABI. It's always safe for us to use a i8* accessor, so I don't see 0 being a problem here.

test/CodeGen/Thumb2/mve-masked-load.ll

551

I'll have a look.

[samparker]

Now handling the a bitcast passthru value in LowerMLOAD. Corrected the half load addr values.

[samparker]

I had missed the shift value on the input patterns.

[dmgreen]

Nice. I think this is looking good, just some details to sort out, like what to do about the target independent parts.

We will presumably want to add the pre and post inc to these in the future too, which will probably bring up the same kinds of questions.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283

They refer back to the LoadExtActions, which are set by setLoadExtAction in ISel. We may need more flags on there to specify the difference between the masked loads and the normal loads.

lib/Target/ARM/ARMISelLowering.cpp

8887 ↗

(On Diff #223220)

This is creating a zero vector of size VT, which is the size of what the masked loads returns. Should it instead be the size of the memory being loaded (because the extend happens to the passthru as well)? What happens if that isn't a legal value type?

lib/Target/ARM/ARMInstrMVE.td

5196

There likely needs to be an anyext too. Can (or is it beneficial for) these be merged into the MVEExtLoad multiclass below?

5203

Edit: You beat me to it. Can you add some tests?

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

Yeah. Alignment of 0 means ABI alignment, which means 8, not unaligned.

I think it may be better to just check this alignment is always the case, getting rid of that weird "use i8's to load unaligned masked loads" thing. That was probably a bad idea, more trouble than it's worth.

I think what will happen here at the moment is that the Vectorizer will call isLegalMaskedLoad with an scalar type and an alignment (which, lets say is unaligned). That alignment won't be checked so the masked loads and stores will be created. Then when we get to the backend the legalizer will call this with a vector type and we'll hit this check, expanding out the masked load into a that very inefficient bunch of code. Which is probably something that we want to avoid.

test/CodeGen/Thumb2/mve-masked-load.ll

929

Nice :)

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887 ↗

(On Diff #223220)

Well, surely the result VT of the masked load has to match the VT of the passthru input. passthru is not about what memory is accessed, but what is written to the destination register. VOVIMM will also generate the same zero value for all full width vector types so for vector widths less than 128-bits, the higher elements will be zeroed and that makes sense. For vectors wider than 128-bits, I think something would have gone before here. I'll add some tests for both these cases.

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

Hmmm, okay. I also can't see removing unaligned support having a big negative effect. Sounds like I need to add some vectorization tests too, unless we already have them?

[samparker]

lib/Target/ARM/ARMInstrMVE.td

5196

As much as I don't like copy-paste, I do appreciate being able to read the code! I think adding to that multiclass is more hassle than it's worth :)

[dmgreen]

lib/Target/ARM/ARMISelLowering.cpp

8887 ↗

(On Diff #223220)

Hmmm. Yeah OK. I see. The PassThru is explicitly extended in tryToFoldExtOfMaskedLoad?

That makes sense, and the tests look OK. (There's one that is both sext and zext the same value, but that looks correct for where it is used).

Test for masked loads/stores longer than 128 bits sounds like a good idea. We should ideally be able to deal with longer vector by splitting them just fine.

lib/Target/ARM/ARMInstrMVE.td

5196

Ha, Fair. I will agree with you there that sometimes more code is simpler.

lib/Target/ARM/ARMTargetTransformInfo.cpp

511

There was one added to the vectoriser tests, but not for alignment checks as far as I remember.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887 ↗

(On Diff #223220)

At some point, I was extending passthru... but it seems that is no longer the case! Our VMOVIMM is probably keeping us correct and if I extend it in dag combine, hopefully we won't need the bitcast handling here anymore.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

8887 ↗

(On Diff #223220)

Ah, no. I was just being blind, passthru is extended.

[samparker]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9283

I had a look and I don't see how we could add an extra flag here. isLoadExtLegal will return true if the operation has been marked as legal, but for both targets (arm, x86) the MLOAD and MSTORE operations are set as custom. So I think having both calls will be necessary, with isVectorLoadExtDesirable enabling the fine grained control that we need.

[samparker]

• Rebased so we're now using MaybeAlign.
• Removed codegen support for unaligned masked loads.
• Added anyext PatFrags.
• Added tests for wider than 128-bit vectors.
• Added loop vectorize tests for unaligned accesses.

[samparker]

@craig.topper This patch currently causes an isel failure for pr35443.ll when an v4i8 masked load is being zero extended into an v4i64. I know nothing about AVX, could you please advise whether this operation is supported or how to address the issue? Thanks.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9308 ↗

(On Diff #224835)

Is this function missing a one use check?

9311 ↗

(On Diff #224835)

Would a masked load ever not be a vector type?

9318 ↗

(On Diff #224835)

What if the masked load is already an extending load?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9324 ↗

(On Diff #224835)

Should we disable this if the load is an expanding load? X86 supports expanding loads, but not extending expanding loads. Or is the expectation that I should block that in X86's implementation of isVectorLoadExtDesirable?

[craig.topper]

In D68337#1707938, @samparker wrote:

@craig.topper This patch currently causes an isel failure for pr35443.ll when an v4i8 masked load is being zero extended into an v4i64. I know nothing about AVX, could you please advise whether this operation is supported or how to address the issue? Thanks.

Its supported. We need to add isel patterns for all the extend flavors. Or we need to block masked loads in isVectorLoadExtDesirable on X86 for now.

[samparker]

Thanks @craig.topper. I'll add the necessary changes into the X86 backend.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9324 ↗

(On Diff #224835)

Good point... I think it makes sense for this part to be as generic as possible and leave it to the backends. We'll have to do the same too.

[samparker]

• Addressed comments in the dag combiner.
• Changed x86 backend so that extending masked loads are not desirable.
• Changed arm backend so that expanding extending masked loads are not desirable.
• Added more tests.

[craig.topper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9312

Check the extension type is NON_EXT explicitly. Don't rely on it being encoding 0.

9328

I don't think this line is needed. Returning NewLoad should take care of it. The line that replaces SDValue(Ld, 1) is needed though.

[dmgreen]

lib/Target/ARM/ARMISelLowering.cpp

14707 ↗

(On Diff #225007)

MVE doesn't support expanding loads, so it would be surprising if we did see one here. Having the check is good though.

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

How rare do you think loads with no explicit alignment to be? I think they don't come up from clang, but is it worth leaving them till later?

As far as I understand, if the alignment on the load was missing (value is 0), it is treated as the abi/pref alignment from the datalayout. So will have an alignment of 8 or 16, so will be aligned.

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

506

ok.

[samparker]

Addressed comments.

[dmgreen]

LGTM. If Craig is happy with the rest.

lib/Target/ARM/ARMTargetTransformInfo.cpp

521

We still probably want to stop i64's and other types.

Maybe do it like "(EltWidth == 32 && (!Alignment || Alignment >= 4)) || ..."

[craig.topper]

I'm happy.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

   } else {
     SDValue Trunc =
         DAG.getNode(ISD::TRUNCATE, SDLoc(N0), N0.getValueType(), ExtLoad);
     Combiner.CombineTo(LN0, Trunc, ExtLoad.getValue(1));
   }
   return SDValue(N, 0); // Return N so it doesn't get rechecked!
 }
 
+static SDValue tryToFoldExtOfMaskedLoad(SelectionDAG &DAG,
+                                        const TargetLowering &TLI, EVT VT,
+                                        SDNode *N, SDValue N0,
+                                        ISD::LoadExtType ExtLoadType,
+                                        ISD::NodeType ExtOpc) {
+  if (!TLI.isLoadExtLegal(ExtLoadType, VT, N0.getValueType()))
+    return SDValue();
+
+  if (VT.isVector() && !TLI.isVectorLoadExtDesirable(SDValue(N, 0)))
+    return SDValue();
+
+  MaskedLoadSDNode *Ld = dyn_cast<MaskedLoadSDNode>(N0);
+  if (!Ld)
+    return SDValue();
+
+  SDLoc dl(Ld);
+  SDValue PassThru = DAG.getNode(ExtOpc, dl, VT, Ld->getPassThru());
+  SDValue NewLoad = DAG.getMaskedLoad(VT, dl, Ld->getChain(),
+                                      Ld->getBasePtr(), Ld->getMask(),
+                                      PassThru, Ld->getMemoryVT(),
+                                      Ld->getMemOperand(), ExtLoadType,
+                                      Ld->isExpandingLoad());
+  DAG.ReplaceAllUsesOfValueWith(SDValue(N, 0), SDValue(NewLoad.getNode(), 0));
+  DAG.ReplaceAllUsesOfValueWith(SDValue(Ld, 1), SDValue(NewLoad.getNode(), 1));
+  return NewLoad;
+}
+
 static SDValue foldExtendedSignBitTest(SDNode *N, SelectionDAG &DAG,
                                        bool LegalOperations) {
   assert((N->getOpcode() == ISD::SIGN_EXTEND ||
           N->getOpcode() == ISD::ZERO_EXTEND) && "Expected sext or zext");
 
   SDValue SetCC = N->getOperand(0);
   if (LegalOperations || SetCC.getOpcode() != ISD::SETCC ||
       !SetCC.hasOneUse() || SetCC.getValueType() != MVT::i1)
     return SDValue();
 
   SDValue X = SetCC.getOperand(0);
   SDValue Ones = SetCC.getOperand(1);
   ISD::CondCode CC = cast<CondCodeSDNode>(SetCC.getOperand(2))->get();
   EVT VT = N->getValueType(0);
   EVT XVT = X.getValueType();
   // setge X, C is canonicalized to setgt, so we do not need to match that
   // pattern. The setlt sibling is folded in SimplifySelectCC() because it does
   // not require the 'not' op.
   if (CC == ISD::SETGT && isAllOnesConstant(Ones) && VT == XVT) {
     // Invert and smear/shift the sign bit:
     // sext i1 (setgt iN X, -1) --> sra (not X), (N - 1)
     // zext i1 (setgt iN X, -1) --> srl (not X), (N - 1)
     SDLoc DL(N);
     SDValue NotX = DAG.getNOT(DL, X, VT);
     SDValue ShiftAmount = DAG.getConstant(VT.getSizeInBits() - 1, DL, VT);
     auto ShiftOpcode = N->getOpcode() == ISD::SIGN_EXTEND ? ISD::SRA : ISD::SRL;
     return DAG.getNode(ShiftOpcode, DL, VT, NotX, ShiftAmount);
   }
   return SDValue();
 }
 
 SDValue DAGCombiner::visitSIGN_EXTEND(SDNode *N) {

   }
 
   // Try to simplify (sext (load x)).
   if (SDValue foldedExt =
           tryToFoldExtOfLoad(DAG, *this, TLI, VT, LegalOperations, N, N0,
                              ISD::SEXTLOAD, ISD::SIGN_EXTEND))
     return foldedExt;
 
+  if (SDValue foldedExt =
+      tryToFoldExtOfMaskedLoad(DAG, TLI, VT, N, N0, ISD::SEXTLOAD,
+                               ISD::SIGN_EXTEND))
+    return foldedExt;
+
   // fold (sext (load x)) to multiple smaller sextloads.
   // Only on illegal but splittable vectors.
   if (SDValue ExtLoad = CombineExtLoad(N))
     return ExtLoad;
 
   // Try to simplify (sext (sextload x)).
   if (SDValue foldedExt = tryToFoldExtOfExtload(
           DAG, *this, TLI, VT, LegalOperations, N, N0, ISD::SEXTLOAD))

   }
 
   // Try to simplify (zext (load x)).
   if (SDValue foldedExt =
           tryToFoldExtOfLoad(DAG, *this, TLI, VT, LegalOperations, N, N0,
                              ISD::ZEXTLOAD, ISD::ZERO_EXTEND))
     return foldedExt;
 
+  if (SDValue foldedExt =
+      tryToFoldExtOfMaskedLoad(DAG, TLI, VT, N, N0, ISD::ZEXTLOAD,
+                               ISD::ZERO_EXTEND))
+    return foldedExt;
+
   // fold (zext (load x)) to multiple smaller zextloads.
   // Only on illegal but splittable vectors.
   if (SDValue ExtLoad = CombineExtLoad(N))
     return ExtLoad;
 
   // fold (zext (and/or/xor (load x), cst)) ->
   //      (and/or/xor (zextload x), (zext cst))
   // Unless (and (load x) cst) will match as a zextload already and has

lib/Target/ARM/ARMInstrMVE.td

                                   (pre_store node:$val, node:$ptr, node:$offset), [{
   return cast<StoreSDNode>(N)->getAlignment() >= 2;
 }]>;
 def aligned16_post_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
                                    (post_store node:$val, node:$ptr, node:$offset), [{
   return cast<StoreSDNode>(N)->getAlignment() >= 2;
 }]>;
 
-def alignedmaskedload32 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
-                                  (masked_ld node:$ptr, node:$pred, node:$passthru), [{
-  return cast<MaskedLoadSDNode>(N)->getAlignment() >= 4;
+def maskedload8 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  return Ld->getMemoryVT().getScalarType() == MVT::i8;
+}]>;
+def sextmaskedload8 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                              (maskedload8 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getExtensionType() == ISD::SEXTLOAD;
+}]>;
+def zextmaskedload8 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                              (maskedload8 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getExtensionType() == ISD::ZEXTLOAD;
+}]>;
+def maskedload16: PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                          (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  EVT ScalarVT = Ld->getMemoryVT().getScalarType();
+  return ScalarVT == MVT::i16 || ScalarVT == MVT::f16;
 }]>;
 def alignedmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
-                                  (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+                                  (maskedload16 node:$ptr, node:$pred, node:$passthru), [{
   return cast<MaskedLoadSDNode>(N)->getAlignment() >= 2;
 }]>;
-def maskedload : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
-                         (masked_ld node:$ptr, node:$pred, node:$passthru)>;
+def unalignedmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                                    (maskedload16 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getAlignment() < 2;
+}]>;
+def sextmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                               (alignedmaskedload16 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getExtensionType() == ISD::SEXTLOAD;
+}]>;
+def zextmaskedload16 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                               (alignedmaskedload16 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getExtensionType() == ISD::ZEXTLOAD;
+}]>;
+def maskedload32: PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                          (masked_ld node:$ptr, node:$pred, node:$passthru), [{
+  auto *Ld = cast<MaskedLoadSDNode>(N);
+  EVT ScalarVT = Ld->getMemoryVT().getScalarType();
+  return ScalarVT == MVT::i32 || ScalarVT == MVT::f32;
+}]>;
+def alignedmaskedload32 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                                  (maskedload32 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getAlignment() >= 4;
+}]>;
+def unalignedmaskedload32 : PatFrag<(ops node:$ptr, node:$pred, node:$passthru),
+                                    (maskedload32 node:$ptr, node:$pred, node:$passthru), [{
+  return cast<MaskedLoadSDNode>(N)->getAlignment() < 4;
+}]>;
 
 def alignedmaskedstore32 : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                                    (masked_st node:$val, node:$ptr, node:$pred), [{
   return cast<MaskedStoreSDNode>(N)->getAlignment() >= 4;
 }]>;
 def alignedmaskedstore16 : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                                    (masked_st node:$val, node:$ptr, node:$pred), [{
   return cast<MaskedStoreSDNode>(N)->getAlignment() >= 2;
 }]>;
+
 def maskedstore : PatFrag<(ops node:$val, node:$ptr, node:$pred),
                           (masked_st node:$val, node:$ptr, node:$pred)>;
 
 let Predicates = [HasMVEInt, IsLE] in {
   // Stores
   defm : MVE_vector_store<MVE_VSTRBU8, byte_alignedstore, 0>;
   defm : MVE_vector_store<MVE_VSTRHU16, hword_alignedstore, 1>;
   defm : MVE_vector_store<MVE_VSTRWU32, alignedstore32, 2>;

   def : Pat<(maskedstore (v4f32 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 MQPR:$val, t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   def : Pat<(maskedstore (v8i16 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 MQPR:$val, t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   def : Pat<(maskedstore (v8f16 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 MQPR:$val, t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
 
   // Unaligned masked loads
-  def : Pat<(v4i32 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v4i32 NEONimmAllZerosV))),
+  def : Pat<(v4i32 (unalignedmaskedload32 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
             (v4i32 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
-  def : Pat<(v4f32 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v4f32 NEONimmAllZerosV))),
+  def : Pat<(v4f32 (unalignedmaskedload32 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4f32 NEONimmAllZerosV))),
             (v4f32 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
-  def : Pat<(v8i16 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v8i16 NEONimmAllZerosV))),
+  def : Pat<(v8i16 (unalignedmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
             (v8i16 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
-  def : Pat<(v8f16 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v8f16 NEONimmAllZerosV))),
+  def : Pat<(v8f16 (unalignedmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8f16 NEONimmAllZerosV))),
             (v8f16 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
 }
 
 let Predicates = [HasMVEInt, IsBE] in {
   // Aligned Stores
   def : MVE_vector_store_typed<v16i8, MVE_VSTRBU8, store, 0>;
   def : MVE_vector_store_typed<v8i16, MVE_VSTRHU16, alignedstore16, 1>;
   def : MVE_vector_store_typed<v8f16, MVE_VSTRHU16, alignedstore16, 1>;

             (MVE_VSTRBU8 (MVE_VREV32_8 MQPR:$val), t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   def : Pat<(maskedstore (v4f32 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 (MVE_VREV32_8 MQPR:$val), t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   def : Pat<(maskedstore (v8i16 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 (MVE_VREV16_8 MQPR:$val), t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   def : Pat<(maskedstore (v8f16 MQPR:$val), t2addrmode_imm7<0>:$addr, VCCR:$pred),
             (MVE_VSTRBU8 (MVE_VREV16_8 MQPR:$val), t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)>;
   // Unaligned masked loads
-  def : Pat<(v4i32 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v4i32 NEONimmAllZerosV))),
+  def : Pat<(v4i32 (unalignedmaskedload32 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
             (v4i32 (MVE_VREV32_8 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)))>;
-  def : Pat<(v4f32 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v4f32 NEONimmAllZerosV))),
+  def : Pat<(v4f32 (unalignedmaskedload32 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4f32 NEONimmAllZerosV))),
             (v4f32 (MVE_VREV32_8 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)))>;
-  def : Pat<(v8i16 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v8i16 NEONimmAllZerosV))),
+  def : Pat<(v8i16 (unalignedmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
             (v8i16 (MVE_VREV16_8 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)))>;
-  def : Pat<(v8f16 (maskedload t2addrmode_imm7<0>:$addr, VCCR:$pred, (v8f16 NEONimmAllZerosV))),
+  def : Pat<(v8f16 (unalignedmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8f16 NEONimmAllZerosV))),
             (v8f16 (MVE_VREV16_8 (MVE_VLDRBU8 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred)))>;
 }
 
 let Predicates = [HasMVEInt] in {
   // Aligned masked store, shared between LE and BE
   def : MVE_vector_maskedstore_typed<v16i8, MVE_VSTRBU8, maskedstore, 0>;
   def : MVE_vector_maskedstore_typed<v8i16, MVE_VSTRHU16, alignedmaskedstore16, 1>;
   def : MVE_vector_maskedstore_typed<v8f16, MVE_VSTRHU16, alignedmaskedstore16, 1>;
   def : MVE_vector_maskedstore_typed<v4i32, MVE_VSTRWU32, alignedmaskedstore32, 2>;
   def : MVE_vector_maskedstore_typed<v4f32, MVE_VSTRWU32, alignedmaskedstore32, 2>;
   // Aligned masked loads
-  def : MVE_vector_maskedload_typed<v16i8, MVE_VLDRBU8, maskedload, 0>;
+  def : MVE_vector_maskedload_typed<v16i8, MVE_VLDRBU8, maskedload8, 0>;
   def : MVE_vector_maskedload_typed<v8i16, MVE_VLDRHU16, alignedmaskedload16, 1>;
   def : MVE_vector_maskedload_typed<v8f16, MVE_VLDRHU16, alignedmaskedload16, 1>;
   def : MVE_vector_maskedload_typed<v4i32, MVE_VLDRWU32, alignedmaskedload32, 2>;
   def : MVE_vector_maskedload_typed<v4f32, MVE_VLDRWU32, alignedmaskedload32, 2>;
+  // Extending masked loads.
+  def : Pat<(v8i16 (sextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
+            (v8i16 (MVE_VLDRBS16 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (sextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRBS32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (sextmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRHS32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v8i16 (zextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v8i16 NEONimmAllZerosV))),
+            (v8i16 (MVE_VLDRBU16 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (zextmaskedload8 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRBU32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
+  def : Pat<(v4i32 (zextmaskedload16 t2addrmode_imm7<0>:$addr, VCCR:$pred,
+                    (v4i32 NEONimmAllZerosV))),
+            (v4i32 (MVE_VLDRHU32 t2addrmode_imm7<0>:$addr, (i32 1), VCCR:$pred))>;
 }
 
+
 // Widening/Narrowing Loads/Stores
 
 let MinAlignment = 2 in {
   def truncstorevi16_align2 : PatFrag<(ops node:$val, node:$ptr),
                                       (truncstorevi16 node:$val, node:$ptr)>;
   def post_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),
                                         (post_truncstvi16 node:$val, node:$base, node:$offset)>;
   def pre_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),

lib/Target/ARM/ARMTargetTransformInfo.cpp

if (ST->hasNEON()) {
 
     // In many cases the address computation is not merged into the instruction
     // addressing mode.
     return 1;
   }
   return BaseT::getAddressComputationCost(Ty, SE, Ptr);
 }
 
 bool ARMTTIImpl::isLegalMaskedLoad(Type *DataTy, unsigned Alignment) {
   if (!EnableMaskedLoadStores || !ST->hasMVEIntegerOps())
     return false;
 
-  if (DataTy->isVectorTy()) {
-    // We don't yet support narrowing or widening masked loads/stores. Expand
-    // them for the moment.
+  unsigned EltWidth = DataTy->getScalarSizeInBits();
+  if (auto *VecTy = dyn_cast<VectorType>(DataTy)) {
+    // Don't support v2i1 yet.
+    if (VecTy->getNumElements() == 2)
+      return false;
+
     unsigned VecWidth = DataTy->getPrimitiveSizeInBits();
-    if (VecWidth != 128)
+    if (VecWidth != 128) {
+      // We don't support extending fp types.
+      if (VecTy->getElementType()->isFloatingPointTy())
+        return false;
+
+      // Only support extending integers if the memory is aligned.
+      if ((EltWidth == 16 && Alignment < 2) ||
+          (EltWidth == 32 && Alignment < 4))
+        return false;
+    }
+
+    if (VecWidth > 128)
       return false;
   }
 
-  unsigned EltWidth = DataTy->getScalarSizeInBits();
   return EltWidth == 32 || EltWidth == 16 || EltWidth == 8;
 }
 
 int ARMTTIImpl::getMemcpyCost(const Instruction *I) {
   const MemCpyInst *MI = dyn_cast<MemCpyInst>(I);
   assert(MI && "MemcpyInst expected");
   ConstantInt *C = dyn_cast<ConstantInt>(MI->getLength());
 
   // To model the cost of a library call, we assume 1 for the call, and
   // 3 for the argument setup.

test/CodeGen/Thumb2/LowOverheadLoops/mve-tail-data-types.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve -disable-mve-tail-predication=false -enable-arm-maskedldst=true %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_char(i8 zeroext %a, i8* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_char:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI0_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q4, #0x0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI0_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q3, q4
-; CHECK-NEXT:    vadd.i32 q4, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    add.w r3, r1, r0
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r4, [r3]
-; CHECK-NEXT:    vmovne.32 q4[0], r4
-; CHECK-NEXT:    lsls r4, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r4
-; CHECK-NEXT:    lsls r4, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r4
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r2, [r3, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r2
-; CHECK-NEXT:    vand q5, q4, q2
-; CHECK-NEXT:    vmov q4, q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmla.u32 q4, q5, r12
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r3, r1, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrbt.u32 q4, [r3]
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q4, q3
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    vpop {d8, d9}
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_short(i16 signext %a, i16* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_short:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vpush {d8, d9}
-; CHECK-NEXT:    sub sp, #8
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI1_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q3, #0x0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI1_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q2, q3
-; CHECK-NEXT:    vadd.i32 q3, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q3
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r3
-; CHECK-NEXT:    lsls r3, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r3
-; CHECK-NEXT:    lsls r3, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r3
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r2, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r2
-; CHECK-NEXT:    vmovlb.s16 q4, q3
-; CHECK-NEXT:    vmov q3, q2
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrht.s32 q4, [r1]
 ; CHECK-NEXT:    adds r1, #8
-; CHECK-NEXT:    vmla.u32 q3, q4, r12
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q3, q2
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
 ; CHECK-NEXT:    vpop {d8, d9}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI1_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_uchar(i8 zeroext %a, i8* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_uchar:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI2_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q4, #0x0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI2_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q3, q4
-; CHECK-NEXT:    vadd.i32 q4, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    add.w r3, r1, r0
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r4, [r3]
-; CHECK-NEXT:    vmovne.32 q4[0], r4
-; CHECK-NEXT:    lsls r4, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r4
-; CHECK-NEXT:    lsls r4, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r4, [r3, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r4
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r2, [r3, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r2
-; CHECK-NEXT:    vand q5, q4, q2
-; CHECK-NEXT:    vmov q4, q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmla.u32 q4, q5, r12
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r3, r1, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrbt.u32 q4, [r3]
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q4, q3
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    vpop {d8, d9}
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI2_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %8, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc i32 @test_acc_scalar_ushort(i16 signext %a, i16* nocapture readonly %b, i32 %N) {
 ; CHECK-LABEL: test_acc_scalar_ushort:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    mov r12, r0
-; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    it eq
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
 ; CHECK-NEXT:    bxeq lr
-; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vpush {d8, d9}
-; CHECK-NEXT:    sub sp, #8
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    bic r3, r3, #3
-; CHECK-NEXT:    vdup.32 q0, r2
-; CHECK-NEXT:    sub.w lr, r3, #4
-; CHECK-NEXT:    adr r2, .LCPI3_0
+; CHECK-NEXT:    vdup.32 q1, r2
+; CHECK-NEXT:    sub.w r12, r3, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r2]
-; CHECK-NEXT:    add.w lr, r3, lr, lsr #2
-; CHECK-NEXT:    vmov.i32 q3, #0x0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI3_0
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov q2, q3
-; CHECK-NEXT:    vadd.i32 q3, q1, r0
-; CHECK-NEXT:    vcmp.u32 cs, q0, q3
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r0, #4
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r4, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r4, #0, #1
-; CHECK-NEXT:    ubfx r4, r3, #4, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #1, #1
-; CHECK-NEXT:    ubfx r4, r3, #8, #1
-; CHECK-NEXT:    ubfx r3, r3, #12, #1
-; CHECK-NEXT:    rsbs r4, r4, #0
-; CHECK-NEXT:    bfi r2, r4, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r3
-; CHECK-NEXT:    lsls r3, r2, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r3
-; CHECK-NEXT:    lsls r3, r2, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r3
-; CHECK-NEXT:    lsls r2, r2, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r2, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r2
-; CHECK-NEXT:    vmovlb.u16 q4, q3
-; CHECK-NEXT:    vmov q3, q2
+; CHECK-NEXT:    vadd.i32 q4, q2, r2
+; CHECK-NEXT:    adds r2, #4
+; CHECK-NEXT:    vpt.u32 cs, q1, q4
+; CHECK-NEXT:    vldrht.u32 q4, [r1]
 ; CHECK-NEXT:    adds r1, #8
-; CHECK-NEXT:    vmla.u32 q3, q4, r12
+; CHECK-NEXT:    vmov q3, q0
+; CHECK-NEXT:    vmla.u32 q0, q4, r0
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %middle.block
-; CHECK-NEXT:    vpsel q0, q3, q2
+; CHECK-NEXT:    vpsel q0, q0, q3
 ; CHECK-NEXT:    vaddv.u32 r0, q0
-; CHECK-NEXT:    add sp, #8
 ; CHECK-NEXT:    vpop {d8, d9}
-; CHECK-NEXT:    pop.w {r4, lr}
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI3_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %7, %middle.block ]
   ret i32 %res.0.lcssa
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_char(i8* nocapture readonly %a, i8* nocapture readonly %b, i8 zeroext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_char:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #72]
-; CHECK-NEXT:    cmp.w r12, #0
+; CHECK-NEXT:    ldr r7, [sp, #28]
+; CHECK-NEXT:    cmp r7, #0
 ; CHECK-NEXT:    beq.w .LBB5_12
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.lr.ph
-; CHECK-NEXT:    add.w r5, r3, r12, lsl #2
-; CHECK-NEXT:    add.w r6, r1, r12
-; CHECK-NEXT:    cmp r5, r1
-; CHECK-NEXT:    add.w r4, r0, r12
-; CHECK-NEXT:    cset r7, hi
-; CHECK-NEXT:    cmp r6, r3
-; CHECK-NEXT:    cset r6, hi
-; CHECK-NEXT:    cmp r5, r0
+; CHECK-NEXT:    add.w r4, r3, r7, lsl #2
+; CHECK-NEXT:    adds r5, r1, r7
+; CHECK-NEXT:    cmp r4, r1
+; CHECK-NEXT:    add.w r6, r0, r7
+; CHECK-NEXT:    cset r12, hi
+; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    cset r5, hi
-; CHECK-NEXT:    cmp r4, r3
+; CHECK-NEXT:    cmp r4, r0
 ; CHECK-NEXT:    cset r4, hi
-; CHECK-NEXT:    ands r5, r4
-; CHECK-NEXT:    lsls r5, r5, #31
+; CHECK-NEXT:    cmp r6, r3
+; CHECK-NEXT:    cset r6, hi
+; CHECK-NEXT:    ands r6, r4
+; CHECK-NEXT:    lsls r6, r6, #31
 ; CHECK-NEXT:    itt eq
-; CHECK-NEXT:    andeq r7, r6
-; CHECK-NEXT:    lslseq.w r7, r7, #31
+; CHECK-NEXT:    andeq.w r6, r5, r12
+; CHECK-NEXT:    lslseq.w r6, r6, #31
 ; CHECK-NEXT:    beq .LBB5_4
 ; CHECK-NEXT:  @ %bb.2: @ %for.body.preheader
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    and lr, r12, #3
-; CHECK-NEXT:    cmp r4, #3
-; CHECK-NEXT:    bhs.w .LBB5_6
+; CHECK-NEXT:    subs r6, r7, #1
+; CHECK-NEXT:    and lr, r7, #3
+; CHECK-NEXT:    cmp r6, #3
+; CHECK-NEXT:    bhs .LBB5_6
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    b .LBB5_9
 ; CHECK-NEXT:  .LBB5_4: @ %vector.ph
-; CHECK-NEXT:    add.w r7, r12, #3
-; CHECK-NEXT:    adr r5, .LCPI5_0
-; CHECK-NEXT:    bic r7, r7, #3
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    subs r7, #4
-; CHECK-NEXT:    movs r6, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
-; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
-; CHECK-NEXT:    movs r4, #0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vmov.i32 q3, #0xff
+; CHECK-NEXT:    adds r6, r7, #3
+; CHECK-NEXT:    movs r5, #1
+; CHECK-NEXT:    bic r6, r6, #3
+; CHECK-NEXT:    subs r7, #1
+; CHECK-NEXT:    subs r6, #4
+; CHECK-NEXT:    vdup.32 q0, r7
+; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    add.w lr, r5, r6, lsr #2
+; CHECK-NEXT:    adr r6, .LCPI5_0
+; CHECK-NEXT:    vldrw.u32 q1, [r6]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB5_5: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q4, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q5
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r0, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q4[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vand q4, q4, q2
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r1, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q5[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q5[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q5[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q5[3], r5
-; CHECK-NEXT:    vand q5, q5, q3
-; CHECK-NEXT:    vctp.32 r12
-; CHECK-NEXT:    vmul.i32 q4, q5, q4
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vadd.i32 q4, q4, r2
+; CHECK-NEXT:    vadd.i32 q2, q1, r7
+; CHECK-NEXT:    adds r4, r0, r7
+; CHECK-NEXT:    vpt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrbt.u32 q2, [r4]
+; CHECK-NEXT:    adds r4, r1, r7
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q4, [r3]
+; CHECK-NEXT:    vldrbt.u32 q3, [r4]
+; CHECK-NEXT:    vmul.i32 q2, q3, q2
+; CHECK-NEXT:    vadd.i32 q2, q2, r2
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
+; CHECK-NEXT:    adds r7, #4
 ; CHECK-NEXT:    le lr, .LBB5_5
 ; CHECK-NEXT:    b .LBB5_12
 ; CHECK-NEXT:  .LBB5_6: @ %for.body.preheader.new
-; CHECK-NEXT:    sub.w r12, lr, r12
+; CHECK-NEXT:    sub.w r12, lr, r7
 ; CHECK-NEXT:    subs r4, r1, #3
 ; CHECK-NEXT:    subs r5, r0, #3
 ; CHECK-NEXT:    sub.w r7, r3, #16
 ; CHECK-NEXT:    mov.w r9, #0
 ; CHECK-NEXT:  .LBB5_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb.w r8, [r5, #3]
 ; CHECK-NEXT:    sub.w r9, r9, #4

 ; CHECK-NEXT:  .LBB5_11: @ %for.body.epil
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r7, [r0, #1]!
 ; CHECK-NEXT:    ldrb r6, [r1, #1]!
 ; CHECK-NEXT:    smlabb r7, r6, r7, r2
 ; CHECK-NEXT:    str r7, [r3, #4]!
 ; CHECK-NEXT:    le lr, .LBB5_11
 ; CHECK-NEXT:  .LBB5_12: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.13:
 ; CHECK-NEXT:  .LCPI5_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
for.body:                                         ; preds = %for.body, %for.body.preheader.new
   %niter.nsub.3 = add i32 %niter, -4
   %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
   br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_short(i16* nocapture readonly %a, i16* nocapture readonly %b, i16 signext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_short:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #28]
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    ldr.w r12, [sp, #8]
 ; CHECK-NEXT:    cmp.w r12, #0
-; CHECK-NEXT:    beq.w .LBB6_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
-; CHECK-NEXT:    add.w r5, r12, #3
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    popeq {r4, pc}
+; CHECK-NEXT:    add.w lr, r12, #3
 ; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    bic r5, r5, #3
-; CHECK-NEXT:    subs r5, #4
-; CHECK-NEXT:    add.w lr, r4, r5, lsr #2
-; CHECK-NEXT:    adr r5, .LCPI6_0
+; CHECK-NEXT:    bic lr, lr, #3
+; CHECK-NEXT:    sub.w lr, lr, #4
+; CHECK-NEXT:    add.w lr, r4, lr, lsr #2
 ; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
 ; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    movs r4, #0
+; CHECK-NEXT:    adr r4, .LCPI6_0
+; CHECK-NEXT:    vldrw.u32 q1, [r4]
+; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB6_2: @ %vector.body
+; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q2, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vcmp.u32 cs, q0, q2
-; CHECK-NEXT:    @ implicit-def: $q2
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r0]
-; CHECK-NEXT:    vmovne.32 q2[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #2]
-; CHECK-NEXT:    vmovmi.32 q2[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #4]
-; CHECK-NEXT:    vmovmi.32 q2[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r0, #6]
-; CHECK-NEXT:    vmovmi.32 q2[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vmovlb.s16 q2, q2
+; CHECK-NEXT:    vadd.i32 q2, q1, r12
+; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    vptt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrht.s32 q2, [r0]
+; CHECK-NEXT:    vldrht.s32 q3, [r1]
 ; CHECK-NEXT:    adds r0, #8
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r5
-; CHECK-NEXT:    vmovlb.s16 q3, q3
-; CHECK-NEXT:    vctp.32 r12
 ; CHECK-NEXT:    vmul.i32 q2, q3, q2
 ; CHECK-NEXT:    adds r1, #8
 ; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
-; CHECK-NEXT:    le lr, .LBB6_2
-; CHECK-NEXT:  .LBB6_3: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    le lr, .LBB6_1
+; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
-; CHECK-NEXT:  @ %bb.4:
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI6_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %vector.ph

 for.cond.cleanup:                                 ; preds = %vector.body, %entry
   ret void
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_uchar(i8* nocapture readonly %a, i8* nocapture readonly %b, i8 zeroext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_uchar:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #72]
-; CHECK-NEXT:    cmp.w r12, #0
+; CHECK-NEXT:    ldr r7, [sp, #28]
+; CHECK-NEXT:    cmp r7, #0
 ; CHECK-NEXT:    beq.w .LBB7_12
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.lr.ph
-; CHECK-NEXT:    add.w r5, r3, r12, lsl #2
-; CHECK-NEXT:    add.w r6, r1, r12
-; CHECK-NEXT:    cmp r5, r1
-; CHECK-NEXT:    add.w r4, r0, r12
-; CHECK-NEXT:    cset r7, hi
-; CHECK-NEXT:    cmp r6, r3
-; CHECK-NEXT:    cset r6, hi
-; CHECK-NEXT:    cmp r5, r0
+; CHECK-NEXT:    add.w r4, r3, r7, lsl #2
+; CHECK-NEXT:    adds r5, r1, r7
+; CHECK-NEXT:    cmp r4, r1
+; CHECK-NEXT:    add.w r6, r0, r7
+; CHECK-NEXT:    cset r12, hi
+; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    cset r5, hi
-; CHECK-NEXT:    cmp r4, r3
+; CHECK-NEXT:    cmp r4, r0
 ; CHECK-NEXT:    cset r4, hi
-; CHECK-NEXT:    ands r5, r4
-; CHECK-NEXT:    lsls r5, r5, #31
+; CHECK-NEXT:    cmp r6, r3
+; CHECK-NEXT:    cset r6, hi
+; CHECK-NEXT:    ands r6, r4
+; CHECK-NEXT:    lsls r6, r6, #31
 ; CHECK-NEXT:    itt eq
-; CHECK-NEXT:    andeq r7, r6
-; CHECK-NEXT:    lslseq.w r7, r7, #31
+; CHECK-NEXT:    andeq.w r6, r5, r12
+; CHECK-NEXT:    lslseq.w r6, r6, #31
 ; CHECK-NEXT:    beq .LBB7_4
 ; CHECK-NEXT:  @ %bb.2: @ %for.body.preheader
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    and lr, r12, #3
-; CHECK-NEXT:    cmp r4, #3
-; CHECK-NEXT:    bhs.w .LBB7_6
+; CHECK-NEXT:    subs r6, r7, #1
+; CHECK-NEXT:    and lr, r7, #3
+; CHECK-NEXT:    cmp r6, #3
+; CHECK-NEXT:    bhs .LBB7_6
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    b .LBB7_9
 ; CHECK-NEXT:  .LBB7_4: @ %vector.ph
-; CHECK-NEXT:    add.w r7, r12, #3
-; CHECK-NEXT:    adr r5, .LCPI7_0
-; CHECK-NEXT:    bic r7, r7, #3
-; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    subs r7, #4
-; CHECK-NEXT:    movs r6, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
-; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
-; CHECK-NEXT:    movs r4, #0
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vmov.i32 q3, #0xff
+; CHECK-NEXT:    adds r6, r7, #3
+; CHECK-NEXT:    movs r5, #1
+; CHECK-NEXT:    bic r6, r6, #3
+; CHECK-NEXT:    subs r7, #1
+; CHECK-NEXT:    subs r6, #4
+; CHECK-NEXT:    vdup.32 q0, r7
+; CHECK-NEXT:    movs r7, #0
+; CHECK-NEXT:    add.w lr, r5, r6, lsr #2
+; CHECK-NEXT:    adr r6, .LCPI7_0
+; CHECK-NEXT:    vldrw.u32 q1, [r6]
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB7_5: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q4, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q5
-; CHECK-NEXT:    vcmp.u32 cs, q0, q4
-; CHECK-NEXT:    @ implicit-def: $q4
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r0, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q4[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q4[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q4[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q4[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vand q4, q4, q2
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    add.w r6, r1, r4
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r7, [r6]
-; CHECK-NEXT:    vmovne.32 q5[0], r7
-; CHECK-NEXT:    lsls r7, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #1]
-; CHECK-NEXT:    vmovmi.32 q5[1], r7
-; CHECK-NEXT:    lsls r7, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r7, [r6, #2]
-; CHECK-NEXT:    vmovmi.32 q5[2], r7
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r5, [r6, #3]
-; CHECK-NEXT:    vmovmi.32 q5[3], r5
-; CHECK-NEXT:    vand q5, q5, q3
-; CHECK-NEXT:    vctp.32 r12
-; CHECK-NEXT:    vmul.i32 q4, q5, q4
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vadd.i32 q4, q4, r2
+; CHECK-NEXT:    vadd.i32 q2, q1, r7
+; CHECK-NEXT:    adds r4, r0, r7
+; CHECK-NEXT:    vpt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrbt.u32 q2, [r4]
+; CHECK-NEXT:    adds r4, r1, r7
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrbt.u32 q3, [r4]
+; CHECK-NEXT:    vmul.i32 q2, q3, q2
+; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q4, [r3]
+; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
+; CHECK-NEXT:    adds r7, #4
 ; CHECK-NEXT:    le lr, .LBB7_5
 ; CHECK-NEXT:    b .LBB7_12
 ; CHECK-NEXT:  .LBB7_6: @ %for.body.preheader.new
-; CHECK-NEXT:    sub.w r12, lr, r12
+; CHECK-NEXT:    sub.w r12, lr, r7
 ; CHECK-NEXT:    subs r4, r1, #3
 ; CHECK-NEXT:    subs r5, r0, #3
 ; CHECK-NEXT:    sub.w r7, r3, #16
 ; CHECK-NEXT:    mov.w r9, #0
 ; CHECK-NEXT:  .LBB7_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb.w r8, [r5, #3]
 ; CHECK-NEXT:    sub.w r9, r9, #4

 ; CHECK-NEXT:  .LBB7_11: @ %for.body.epil
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r7, [r0, #1]!
 ; CHECK-NEXT:    ldrb r6, [r1, #1]!
 ; CHECK-NEXT:    smlabb r7, r6, r7, r2
 ; CHECK-NEXT:    str r7, [r3, #4]!
 ; CHECK-NEXT:    le lr, .LBB7_11
 ; CHECK-NEXT:  .LBB7_12: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11}
-; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.13:
 ; CHECK-NEXT:  .LCPI7_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
for.body:                                         ; preds = %for.body, %for.body.preheader.new
   %niter.nsub.3 = add i32 %niter, -4
   %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
   br i1 %niter.ncmp.3, label %for.cond.cleanup.loopexit.unr-lcssa, label %for.body
 }
 
 define arm_aapcs_vfpcc void @test_vec_mul_scalar_add_ushort(i16* nocapture readonly %a, i16* nocapture readonly %b, i16 signext %c, i32* nocapture %res, i32 %N) {
 ; CHECK-LABEL: test_vec_mul_scalar_add_ushort:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldr.w r12, [sp, #28]
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    ldr.w r12, [sp, #8]
 ; CHECK-NEXT:    cmp.w r12, #0
-; CHECK-NEXT:    beq.w .LBB8_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
-; CHECK-NEXT:    add.w r5, r12, #3
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    popeq {r4, pc}
+; CHECK-NEXT:    add.w lr, r12, #3
 ; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    bic r5, r5, #3
-; CHECK-NEXT:    subs r5, #4
-; CHECK-NEXT:    add.w lr, r4, r5, lsr #2
-; CHECK-NEXT:    adr r5, .LCPI8_0
+; CHECK-NEXT:    bic lr, lr, #3
+; CHECK-NEXT:    sub.w lr, lr, #4
+; CHECK-NEXT:    add.w lr, r4, lr, lsr #2
 ; CHECK-NEXT:    sub.w r4, r12, #1
-; CHECK-NEXT:    vldrw.u32 q1, [r5]
 ; CHECK-NEXT:    vdup.32 q0, r4
-; CHECK-NEXT:    movs r4, #0
+; CHECK-NEXT:    adr r4, .LCPI8_0
+; CHECK-NEXT:    vldrw.u32 q1, [r4]
+; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB8_2: @ %vector.body
+; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vadd.i32 q2, q1, r4
-; CHECK-NEXT:    @ implicit-def: $q3
-; CHECK-NEXT:    adds r4, #4
-; CHECK-NEXT:    vcmp.u32 cs, q0, q2
-; CHECK-NEXT:    @ implicit-def: $q2
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r0]
-; CHECK-NEXT:    vmovne.32 q2[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #2]
-; CHECK-NEXT:    vmovmi.32 q2[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r0, #4]
-; CHECK-NEXT:    vmovmi.32 q2[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r0, #6]
-; CHECK-NEXT:    vmovmi.32 q2[3], r5
-; CHECK-NEXT:    vmrs r6, p0
-; CHECK-NEXT:    vmovlb.u16 q2, q2
+; CHECK-NEXT:    vadd.i32 q2, q1, r12
+; CHECK-NEXT:    add.w r12, r12, #4
+; CHECK-NEXT:    vptt.u32 cs, q0, q2
+; CHECK-NEXT:    vldrht.u32 q2, [r0]
+; CHECK-NEXT:    vldrht.u32 q3, [r1]
 ; CHECK-NEXT:    adds r0, #8
-; CHECK-NEXT:    and r5, r6, #1
-; CHECK-NEXT:    rsbs r7, r5, #0
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    bfi r5, r7, #0, #1
-; CHECK-NEXT:    ubfx r7, r6, #4, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #1, #1
-; CHECK-NEXT:    ubfx r7, r6, #8, #1
-; CHECK-NEXT:    ubfx r6, r6, #12, #1
-; CHECK-NEXT:    rsbs r7, r7, #0
-; CHECK-NEXT:    bfi r5, r7, #2, #1
-; CHECK-NEXT:    rsbs r6, r6, #0
-; CHECK-NEXT:    bfi r5, r6, #3, #1
-; CHECK-NEXT:    lsls r6, r5, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r6, [r1]
-; CHECK-NEXT:    vmovne.32 q3[0], r6
-; CHECK-NEXT:    lsls r6, r5, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #2]
-; CHECK-NEXT:    vmovmi.32 q3[1], r6
-; CHECK-NEXT:    lsls r6, r5, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r6, [r1, #4]
-; CHECK-NEXT:    vmovmi.32 q3[2], r6
-; CHECK-NEXT:    lsls r5, r5, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r5, [r1, #6]
-; CHECK-NEXT:    vmovmi.32 q3[3], r5
-; CHECK-NEXT:    vmovlb.u16 q3, q3
-; CHECK-NEXT:    vctp.32 r12
 ; CHECK-NEXT:    vmul.i32 q2, q3, q2
 ; CHECK-NEXT:    adds r1, #8
 ; CHECK-NEXT:    vadd.i32 q2, q2, r2
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q2, [r3]
 ; CHECK-NEXT:    adds r3, #16
-; CHECK-NEXT:    sub.w r12, r12, #4
-; CHECK-NEXT:    le lr, .LBB8_2
-; CHECK-NEXT:  .LBB8_3: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    le lr, .LBB8_1
+; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
-; CHECK-NEXT:  @ %bb.4:
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI8_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %vector.ph

test/CodeGen/Thumb2/mve-masked-ldst.ll

entry:
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   call void @llvm.masked.store.v4i32(<4 x i32> %2, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v4i32_v4i8(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i8> *%src) {
 ; CHECK-LABEL: foo_sext_v4i32_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #3]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrbt.s32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 1, <4 x i1> %1, <4 x i8> undef)
   %3 = sext <4 x i8> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v4i32_v4i16(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i16> *%src) {
 ; CHECK-LABEL: foo_sext_v4i32_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r1, [r2, #6]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrht.s32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i16> @llvm.masked.load.v4i16(<4 x i16>* %src, i32 2, <4 x i1> %1, <4 x i16> undef)
   %3 = sext <4 x i16> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v4i32_v4i8(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i8> *%src) {
 ; CHECK-LABEL: foo_zext_v4i32_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #3]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vand q0, q0, q1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrbt.u32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 1, <4 x i1> %1, <4 x i8> undef)
   %3 = zext <4 x i8> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v4i32_v4i16(<4 x i32> *%dest, <4 x i32> *%mask, <4 x i16> *%src) {
 ; CHECK-LABEL: foo_zext_v4i32_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vcmp.s32 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r1, lr, #1
-; CHECK-NEXT:    ubfx r3, lr, #4, #1
-; CHECK-NEXT:    rsb.w r12, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r12, #0, #1
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, lr, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, lr, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #3, #1
-; CHECK-NEXT:    lsls r3, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrhne r3, [r2]
-; CHECK-NEXT:    vmovne.32 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.32 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.32 q0[2], r3
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrhmi r1, [r2, #6]
-; CHECK-NEXT:    vmovmi.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-NEXT:    vldrht.u32 q0, [r2]
 ; CHECK-NEXT:    vstrwt.32 q0, [r0]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i16> @llvm.masked.load.v4i16(<4 x i16>* %src, i32 2, <4 x i1> %1, <4 x i16> undef)
   %3 = zext <4 x i16> %2 to <4 x i32>
   call void @llvm.masked.store.v4i32(<4 x i32> %3, <4 x i32>* %dest, i32 4, <4 x i1> %1)
   ret void
 }
 
+define void @foo_sext_v2i64_v2i32(<2 x i64> *%dest, <2 x i32> *%mask, <2 x i32> *%src) {
+; CHECK-LE-LABEL: foo_sext_v2i64_v2i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-LE-NEXT:    push {r4, r5, r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    ldrd lr, r12, [r1]
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    movs r4, #0
+; CHECK-LE-NEXT:    rsbs.w r3, lr, #0
+; CHECK-LE-NEXT:    vmov.32 q0[0], lr
+; CHECK-LE-NEXT:    sbcs.w r3, r1, lr, asr #31
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w lr, #1
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    cmp r1, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r1, #1
+; CHECK-LE-NEXT:    bfi r1, lr, #0, #1
+; CHECK-LE-NEXT:    vmov.32 q0[2], r12
+; CHECK-LE-NEXT:    and r3, r1, #3
+; CHECK-LE-NEXT:    lsls r1, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrne r1, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q1[0], r1
+; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q1[2], r1
+; CHECK-LE-NEXT:    vmov r2, s0
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov r1, s6
+; CHECK-LE-NEXT:    vmov.32 q1[0], r3
+; CHECK-LE-NEXT:    rsbs r5, r2, #0
+; CHECK-LE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-LE-NEXT:    vmov r2, s2
+; CHECK-LE-NEXT:    asr.w lr, r3, #31
+; CHECK-LE-NEXT:    vmov.32 q1[1], lr
+; CHECK-LE-NEXT:    asr.w r12, r1, #31
+; CHECK-LE-NEXT:    vmov.32 q1[2], r1
+; CHECK-LE-NEXT:    mov.w r1, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    vmov.32 q1[3], r12
+; CHECK-LE-NEXT:    rsbs r3, r2, #0
+; CHECK-LE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r4, #1
+; CHECK-LE-NEXT:    cmp r4, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r4, #1
+; CHECK-LE-NEXT:    bfi r4, r1, #0, #1
+; CHECK-LE-NEXT:    and r1, r4, #3
+; CHECK-LE-NEXT:    lsls r2, r4, #31
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    vstrne d2, [r0]
+; CHECK-LE-NEXT:    lsls r1, r1, #30
+; CHECK-LE-NEXT:    it mi
+; CHECK-LE-NEXT:    vstrmi d3, [r0, #8]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r4, r5, r7, pc}
+;
+; CHECK-BE-LABEL: foo_sext_v2i64_v2i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    ldrd r12, lr, [r1]
+; CHECK-BE-NEXT:    rsbs.w r1, lr, #0
+; CHECK-BE-NEXT:    mov.w r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, lr, asr #31
+; CHECK-BE-NEXT:    vmov.32 q0[1], r12
+; CHECK-BE-NEXT:    @ implicit-def: $q2
+; CHECK-BE-NEXT:    vmov.32 q0[3], lr
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w lr, #1
+; CHECK-BE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, r12, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r3, #1
+; CHECK-BE-NEXT:    cmp r3, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r3, #1
+; CHECK-BE-NEXT:    bfi r3, lr, #0, #1
+; CHECK-BE-NEXT:    and r1, r3, #3
+; CHECK-BE-NEXT:    lsls r3, r3, #31
+; CHECK-BE-NEXT:    beq .LBB5_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    ldr r3, [r2]
+; CHECK-BE-NEXT:    vmov.32 q1[1], r3
+; CHECK-BE-NEXT:    vrev64.32 q2, q1
+; CHECK-BE-NEXT:  .LBB5_2: @ %else
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    bpl .LBB5_4
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    ldr r1, [r2, #4]
+; CHECK-BE-NEXT:    vrev64.32 q0, q2
+; CHECK-BE-NEXT:    vmov.32 q0[3], r1
+; CHECK-BE-NEXT:    vrev64.32 q2, q0
+; CHECK-BE-NEXT:  .LBB5_4: @ %else2
+; CHECK-BE-NEXT:    vrev64.32 q0, q2
+; CHECK-BE-NEXT:    vrev64.32 q2, q1
+; CHECK-BE-NEXT:    vmov r2, s11
+; CHECK-BE-NEXT:    movs r4, #0
+; CHECK-BE-NEXT:    vmov r3, s1
+; CHECK-BE-NEXT:    vmov r1, s3
+; CHECK-BE-NEXT:    rsbs r5, r2, #0
+; CHECK-BE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-BE-NEXT:    vmov r2, s9
+; CHECK-BE-NEXT:    asr.w lr, r3, #31
+; CHECK-BE-NEXT:    vmov.32 q1[0], lr
+; CHECK-BE-NEXT:    asr.w r12, r1, #31
+; CHECK-BE-NEXT:    vmov.32 q1[1], r3
+; CHECK-BE-NEXT:    vmov.32 q1[2], r12
+; CHECK-BE-NEXT:    vmov.32 q1[3], r1
+; CHECK-BE-NEXT:    mov.w r1, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r1, #1
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    rsbs r3, r2, #0
+; CHECK-BE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r4, #1
+; CHECK-BE-NEXT:    cmp r4, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r4, #1
+; CHECK-BE-NEXT:    bfi r4, r1, #0, #1
+; CHECK-BE-NEXT:    and r1, r4, #3
+; CHECK-BE-NEXT:    lsls r2, r4, #31
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    vstrne d0, [r0]
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    it mi
+; CHECK-BE-NEXT:    vstrmi d1, [r0, #8]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+entry:
+  %0 = load <2 x i32>, <2 x i32>* %mask, align 4
+  %1 = icmp sgt <2 x i32> %0, zeroinitializer
+  %2 = call <2 x i32> @llvm.masked.load.v2i32(<2 x i32>* %src, i32 4, <2 x i1> %1, <2 x i32> undef)
+  %3 = sext <2 x i32> %2 to <2 x i64>
+  call void @llvm.masked.store.v2i64(<2 x i64> %3, <2 x i64>* %dest, i32 8, <2 x i1> %1)
+  ret void
+}
+
+define void @foo_sext_v2i64_v2i32_unaligned(<2 x i64> *%dest, <2 x i32> *%mask, <2 x i32> *%src) {
+; CHECK-LE-LABEL: foo_sext_v2i64_v2i32_unaligned:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-LE-NEXT:    push {r4, r5, r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    ldrd lr, r12, [r1]
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    movs r4, #0
+; CHECK-LE-NEXT:    rsbs.w r3, lr, #0
+; CHECK-LE-NEXT:    vmov.32 q0[0], lr
+; CHECK-LE-NEXT:    sbcs.w r3, r1, lr, asr #31
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w lr, #1
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    cmp r1, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r1, #1
+; CHECK-LE-NEXT:    bfi r1, lr, #0, #1
+; CHECK-LE-NEXT:    vmov.32 q0[2], r12
+; CHECK-LE-NEXT:    and r3, r1, #3
+; CHECK-LE-NEXT:    lsls r1, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrne r1, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q1[0], r1
+; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q1[2], r1
+; CHECK-LE-NEXT:    vmov r2, s0
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov r1, s6
+; CHECK-LE-NEXT:    vmov.32 q1[0], r3
+; CHECK-LE-NEXT:    rsbs r5, r2, #0
+; CHECK-LE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-LE-NEXT:    vmov r2, s2
+; CHECK-LE-NEXT:    asr.w lr, r3, #31
+; CHECK-LE-NEXT:    vmov.32 q1[1], lr
+; CHECK-LE-NEXT:    asr.w r12, r1, #31
+; CHECK-LE-NEXT:    vmov.32 q1[2], r1
+; CHECK-LE-NEXT:    mov.w r1, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    vmov.32 q1[3], r12
+; CHECK-LE-NEXT:    rsbs r3, r2, #0
+; CHECK-LE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r4, #1
+; CHECK-LE-NEXT:    cmp r4, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r4, #1
+; CHECK-LE-NEXT:    bfi r4, r1, #0, #1
+; CHECK-LE-NEXT:    and r1, r4, #3
+; CHECK-LE-NEXT:    lsls r2, r4, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    vmovne r2, r3, d2
+; CHECK-LE-NEXT:    strdne r2, r3, [r0]
+; CHECK-LE-NEXT:    lsls r1, r1, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    vmovmi r1, r2, d3
+; CHECK-LE-NEXT:    strdmi r1, r2, [r0, #8]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r4, r5, r7, pc}
+;
+; CHECK-BE-LABEL: foo_sext_v2i64_v2i32_unaligned:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    ldrd r12, lr, [r1]
+; CHECK-BE-NEXT:    rsbs.w r1, lr, #0
+; CHECK-BE-NEXT:    mov.w r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, lr, asr #31
+; CHECK-BE-NEXT:    vmov.32 q0[1], r12
+; CHECK-BE-NEXT:    @ implicit-def: $q2
+; CHECK-BE-NEXT:    vmov.32 q0[3], lr
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w lr, #1
+; CHECK-BE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, r12, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r3, #1
+; CHECK-BE-NEXT:    cmp r3, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r3, #1
+; CHECK-BE-NEXT:    bfi r3, lr, #0, #1
+; CHECK-BE-NEXT:    and r1, r3, #3
+; CHECK-BE-NEXT:    lsls r3, r3, #31
+; CHECK-BE-NEXT:    beq .LBB6_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    ldr r3, [r2]
+; CHECK-BE-NEXT:    vmov.32 q1[1], r3
+; CHECK-BE-NEXT:    vrev64.32 q2, q1
+; CHECK-BE-NEXT:  .LBB6_2: @ %else
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    bpl .LBB6_4
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    ldr r1, [r2, #4]
+; CHECK-BE-NEXT:    vrev64.32 q0, q2
+; CHECK-BE-NEXT:    vmov.32 q0[3], r1
+; CHECK-BE-NEXT:    vrev64.32 q2, q0
+; CHECK-BE-NEXT:  .LBB6_4: @ %else2
+; CHECK-BE-NEXT:    vrev64.32 q0, q2
+; CHECK-BE-NEXT:    vrev64.32 q2, q1
+; CHECK-BE-NEXT:    vmov r2, s11
+; CHECK-BE-NEXT:    movs r4, #0
+; CHECK-BE-NEXT:    vmov r3, s1
+; CHECK-BE-NEXT:    vmov r1, s3
+; CHECK-BE-NEXT:    rsbs r5, r2, #0
+; CHECK-BE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-BE-NEXT:    vmov r2, s9
+; CHECK-BE-NEXT:    asr.w lr, r3, #31
+; CHECK-BE-NEXT:    vmov.32 q1[0], lr
+; CHECK-BE-NEXT:    asr.w r12, r1, #31
+; CHECK-BE-NEXT:    vmov.32 q1[1], r3
+; CHECK-BE-NEXT:    vmov.32 q1[2], r12
+; CHECK-BE-NEXT:    vmov.32 q1[3], r1
+; CHECK-BE-NEXT:    mov.w r1, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r1, #1
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    rsbs r3, r2, #0
+; CHECK-BE-NEXT:    sbcs.w r2, r4, r2, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r4, #1
+; CHECK-BE-NEXT:    cmp r4, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r4, #1
+; CHECK-BE-NEXT:    bfi r4, r1, #0, #1
+; CHECK-BE-NEXT:    and r1, r4, #3
+; CHECK-BE-NEXT:    lsls r2, r4, #31
+; CHECK-BE-NEXT:    itt ne
+; CHECK-BE-NEXT:    vmovne r2, r3, d0
+; CHECK-BE-NEXT:    strdne r3, r2, [r0]
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    vmovmi r1, r2, d1
+; CHECK-BE-NEXT:    strdmi r2, r1, [r0, #8]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+entry:
+  %0 = load <2 x i32>, <2 x i32>* %mask, align 4
+  %1 = icmp sgt <2 x i32> %0, zeroinitializer
+  %2 = call <2 x i32> @llvm.masked.load.v2i32(<2 x i32>* %src, i32 2, <2 x i1> %1, <2 x i32> undef)
+  %3 = sext <2 x i32> %2 to <2 x i64>
+  call void @llvm.masked.store.v2i64(<2 x i64> %3, <2 x i64>* %dest, i32 4, <2 x i1> %1)
+  ret void
+}
+
+define void @foo_zext_v2i64_v2i32(<2 x i64> *%dest, <2 x i32> *%mask, <2 x i32> *%src) {
+; CHECK-LE-LABEL: foo_zext_v2i64_v2i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r7, lr}
+; CHECK-LE-NEXT:    push {r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    ldrd lr, r12, [r1]
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    rsbs.w r3, lr, #0
+; CHECK-LE-NEXT:    vmov.32 q0[0], lr
+; CHECK-LE-NEXT:    sbcs.w r3, r1, lr, asr #31
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w lr, #1
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    cmp r1, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r1, #1
+; CHECK-LE-NEXT:    bfi r1, lr, #0, #1
+; CHECK-LE-NEXT:    vmov.32 q0[2], r12
+; CHECK-LE-NEXT:    and r3, r1, #3
+; CHECK-LE-NEXT:    adr.w r12, .LCPI7_0
+; CHECK-LE-NEXT:    lsls r1, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrne r1, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q1[0], r1
+; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    vmov r3, s0
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q1[2], r1
+; CHECK-LE-NEXT:    movs r2, #0
+; CHECK-LE-NEXT:    vldrw.u32 q2, [r12]
+; CHECK-LE-NEXT:    mov.w r12, #0
+; CHECK-LE-NEXT:    vand q1, q1, q2
+; CHECK-LE-NEXT:    rsbs r1, r3, #0
+; CHECK-LE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-LE-NEXT:    vmov r3, s2
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w r12, #1
+; CHECK-LE-NEXT:    rsbs r1, r3, #0
+; CHECK-LE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r2, #1
+; CHECK-LE-NEXT:    cmp r2, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r2, #1
+; CHECK-LE-NEXT:    bfi r2, r12, #0, #1
+; CHECK-LE-NEXT:    and r1, r2, #3
+; CHECK-LE-NEXT:    lsls r2, r2, #31
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    vstrne d2, [r0]
+; CHECK-LE-NEXT:    lsls r1, r1, #30
+; CHECK-LE-NEXT:    it mi
+; CHECK-LE-NEXT:    vstrmi d3, [r0, #8]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r7, pc}
+; CHECK-LE-NEXT:    .p2align 4
+; CHECK-LE-NEXT:  @ %bb.1:
+; CHECK-LE-NEXT:  .LCPI7_0:
+; CHECK-LE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-LE-NEXT:    .long 0 @ 0x0
+; CHECK-LE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-LE-NEXT:    .long 0 @ 0x0
+;
+; CHECK-BE-LABEL: foo_zext_v2i64_v2i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r7, lr}
+; CHECK-BE-NEXT:    push {r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    ldrd r12, lr, [r1]
+; CHECK-BE-NEXT:    rsbs.w r1, lr, #0
+; CHECK-BE-NEXT:    mov.w r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, lr, asr #31
+; CHECK-BE-NEXT:    vmov.32 q0[1], r12
+; CHECK-BE-NEXT:    @ implicit-def: $q1
+; CHECK-BE-NEXT:    vmov.32 q0[3], lr
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w lr, #1
+; CHECK-BE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, r12, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r3, #1
+; CHECK-BE-NEXT:    cmp r3, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r3, #1
+; CHECK-BE-NEXT:    bfi r3, lr, #0, #1
+; CHECK-BE-NEXT:    and r1, r3, #3
+; CHECK-BE-NEXT:    lsls r3, r3, #31
+; CHECK-BE-NEXT:    beq .LBB7_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    ldr r3, [r2]
+; CHECK-BE-NEXT:    vmov.32 q2[1], r3
+; CHECK-BE-NEXT:    vrev64.32 q1, q2
+; CHECK-BE-NEXT:  .LBB7_2: @ %else
+; CHECK-BE-NEXT:    vrev64.32 q2, q0
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    bpl .LBB7_4
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    ldr r1, [r2, #4]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    vmov.32 q0[3], r1
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:  .LBB7_4: @ %else2
+; CHECK-BE-NEXT:    vrev64.32 q3, q2
+; CHECK-BE-NEXT:    movs r2, #0
+; CHECK-BE-NEXT:    vmov r3, s15
+; CHECK-BE-NEXT:    adr.w r12, .LCPI7_0
+; CHECK-BE-NEXT:    vldrb.u8 q0, [r12]
+; CHECK-BE-NEXT:    mov.w r12, #0
+; CHECK-BE-NEXT:    vrev64.8 q2, q0
+; CHECK-BE-NEXT:    vand q0, q1, q2
+; CHECK-BE-NEXT:    rsbs r1, r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-BE-NEXT:    vmov r3, s13
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w r12, #1
+; CHECK-BE-NEXT:    rsbs r1, r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r2, #1
+; CHECK-BE-NEXT:    cmp r2, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r2, #1
+; CHECK-BE-NEXT:    bfi r2, r12, #0, #1
+; CHECK-BE-NEXT:    and r1, r2, #3
+; CHECK-BE-NEXT:    lsls r2, r2, #31
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    vstrne d0, [r0]
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    it mi
+; CHECK-BE-NEXT:    vstrmi d1, [r0, #8]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r7, pc}
+; CHECK-BE-NEXT:    .p2align 4
+; CHECK-BE-NEXT:  @ %bb.5:
+; CHECK-BE-NEXT:  .LCPI7_0:
+; CHECK-BE-NEXT:    .long 0 @ 0x0
+; CHECK-BE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-BE-NEXT:    .long 0 @ 0x0
+; CHECK-BE-NEXT:    .long 4294967295 @ 0xffffffff
+entry:
+  %0 = load <2 x i32>, <2 x i32>* %mask, align 4
+  %1 = icmp sgt <2 x i32> %0, zeroinitializer
+  %2 = call <2 x i32> @llvm.masked.load.v2i32(<2 x i32>* %src, i32 4, <2 x i1> %1, <2 x i32> undef)
+  %3 = zext <2 x i32> %2 to <2 x i64>
+  call void @llvm.masked.store.v2i64(<2 x i64> %3, <2 x i64>* %dest, i32 8, <2 x i1> %1)
+  ret void
+}
+
+define void @foo_zext_v2i64_v2i32_unaligned(<2 x i64> *%dest, <2 x i32> *%mask, <2 x i32> *%src) {
+; CHECK-LE-LABEL: foo_zext_v2i64_v2i32_unaligned:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r7, lr}
+; CHECK-LE-NEXT:    push {r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    ldrd lr, r12, [r1]
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    rsbs.w r3, lr, #0
+; CHECK-LE-NEXT:    vmov.32 q0[0], lr
+; CHECK-LE-NEXT:    sbcs.w r3, r1, lr, asr #31
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w lr, #1
+; CHECK-LE-NEXT:    rsbs.w r3, r12, #0
+; CHECK-LE-NEXT:    sbcs.w r3, r1, r12, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r1, #1
+; CHECK-LE-NEXT:    cmp r1, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r1, #1
+; CHECK-LE-NEXT:    bfi r1, lr, #0, #1
+; CHECK-LE-NEXT:    vmov.32 q0[2], r12
+; CHECK-LE-NEXT:    and r3, r1, #3
+; CHECK-LE-NEXT:    adr.w r12, .LCPI8_0
+; CHECK-LE-NEXT:    lsls r1, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrne r1, [r2]
+; CHECK-LE-NEXT:    vmovne.32 q1[0], r1
+; CHECK-LE-NEXT:    lsls r1, r3, #30
+; CHECK-LE-NEXT:    vmov r3, s0
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrmi r1, [r2, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q1[2], r1
+; CHECK-LE-NEXT:    movs r2, #0
+; CHECK-LE-NEXT:    vldrw.u32 q2, [r12]
+; CHECK-LE-NEXT:    mov.w r12, #0
+; CHECK-LE-NEXT:    vand q1, q1, q2
+; CHECK-LE-NEXT:    rsbs r1, r3, #0
+; CHECK-LE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-LE-NEXT:    vmov r3, s2
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt.w r12, #1
+; CHECK-LE-NEXT:    rsbs r1, r3, #0
+; CHECK-LE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-LE-NEXT:    it lt
+; CHECK-LE-NEXT:    movlt r2, #1
+; CHECK-LE-NEXT:    cmp r2, #0
+; CHECK-LE-NEXT:    it ne
+; CHECK-LE-NEXT:    mvnne r2, #1
+; CHECK-LE-NEXT:    bfi r2, r12, #0, #1
+; CHECK-LE-NEXT:    and r1, r2, #3
+; CHECK-LE-NEXT:    lsls r2, r2, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    vmovne r2, r3, d2
+; CHECK-LE-NEXT:    strdne r2, r3, [r0]
+; CHECK-LE-NEXT:    lsls r1, r1, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    vmovmi r1, r2, d3
+; CHECK-LE-NEXT:    strdmi r1, r2, [r0, #8]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r7, pc}
+; CHECK-LE-NEXT:    .p2align 4
+; CHECK-LE-NEXT:  @ %bb.1:
+; CHECK-LE-NEXT:  .LCPI8_0:
+; CHECK-LE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-LE-NEXT:    .long 0 @ 0x0
+; CHECK-LE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-LE-NEXT:    .long 0 @ 0x0
+;
+; CHECK-BE-LABEL: foo_zext_v2i64_v2i32_unaligned:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r7, lr}
+; CHECK-BE-NEXT:    push {r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    ldrd r12, lr, [r1]
+; CHECK-BE-NEXT:    rsbs.w r1, lr, #0
+; CHECK-BE-NEXT:    mov.w r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, lr, asr #31
+; CHECK-BE-NEXT:    vmov.32 q0[1], r12
+; CHECK-BE-NEXT:    @ implicit-def: $q1
+; CHECK-BE-NEXT:    vmov.32 q0[3], lr
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w lr, #1
+; CHECK-BE-NEXT:    rsbs.w r1, r12, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r3, r12, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r3, #1
+; CHECK-BE-NEXT:    cmp r3, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r3, #1
+; CHECK-BE-NEXT:    bfi r3, lr, #0, #1
+; CHECK-BE-NEXT:    and r1, r3, #3
+; CHECK-BE-NEXT:    lsls r3, r3, #31
+; CHECK-BE-NEXT:    beq .LBB8_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    ldr r3, [r2]
+; CHECK-BE-NEXT:    vmov.32 q2[1], r3
+; CHECK-BE-NEXT:    vrev64.32 q1, q2
+; CHECK-BE-NEXT:  .LBB8_2: @ %else
+; CHECK-BE-NEXT:    vrev64.32 q2, q0
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    bpl .LBB8_4
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    ldr r1, [r2, #4]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    vmov.32 q0[3], r1
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:  .LBB8_4: @ %else2
+; CHECK-BE-NEXT:    vrev64.32 q3, q2
+; CHECK-BE-NEXT:    movs r2, #0
+; CHECK-BE-NEXT:    vmov r3, s15
+; CHECK-BE-NEXT:    adr.w r12, .LCPI8_0
+; CHECK-BE-NEXT:    vldrb.u8 q0, [r12]
+; CHECK-BE-NEXT:    mov.w r12, #0
+; CHECK-BE-NEXT:    vrev64.8 q2, q0
+; CHECK-BE-NEXT:    vand q0, q1, q2
+; CHECK-BE-NEXT:    rsbs r1, r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-BE-NEXT:    vmov r3, s13
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt.w r12, #1
+; CHECK-BE-NEXT:    rsbs r1, r3, #0
+; CHECK-BE-NEXT:    sbcs.w r1, r2, r3, asr #31
+; CHECK-BE-NEXT:    it lt
+; CHECK-BE-NEXT:    movlt r2, #1
+; CHECK-BE-NEXT:    cmp r2, #0
+; CHECK-BE-NEXT:    it ne
+; CHECK-BE-NEXT:    mvnne r2, #1
+; CHECK-BE-NEXT:    bfi r2, r12, #0, #1
+; CHECK-BE-NEXT:    and r1, r2, #3
+; CHECK-BE-NEXT:    lsls r2, r2, #31
+; CHECK-BE-NEXT:    itt ne
+; CHECK-BE-NEXT:    vmovne r2, r3, d0
+; CHECK-BE-NEXT:    strdne r3, r2, [r0]
+; CHECK-BE-NEXT:    lsls r1, r1, #30
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    vmovmi r1, r2, d1
+; CHECK-BE-NEXT:    strdmi r2, r1, [r0, #8]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r7, pc}
+; CHECK-BE-NEXT:    .p2align 4
+; CHECK-BE-NEXT:  @ %bb.5:
+; CHECK-BE-NEXT:  .LCPI8_0:
+; CHECK-BE-NEXT:    .long 0 @ 0x0
+; CHECK-BE-NEXT:    .long 4294967295 @ 0xffffffff
+; CHECK-BE-NEXT:    .long 0 @ 0x0
+; CHECK-BE-NEXT:    .long 4294967295 @ 0xffffffff
+entry:
+  %0 = load <2 x i32>, <2 x i32>* %mask, align 4
+  %1 = icmp sgt <2 x i32> %0, zeroinitializer
+  %2 = call <2 x i32> @llvm.masked.load.v2i32(<2 x i32>* %src, i32 2, <2 x i1> %1, <2 x i32> undef)
+  %3 = zext <2 x i32> %2 to <2 x i64>
+  call void @llvm.masked.store.v2i64(<2 x i64> %3, <2 x i64>* %dest, i32 4, <2 x i1> %1)
+  ret void
+}
+
 define void @foo_v8i16_v8i16(<8 x i16> *%dest, <8 x i16> *%mask, <8 x i16> *%src) {
 ; CHECK-LABEL: foo_v8i16_v8i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
 ; CHECK-NEXT:    vptt.s16 gt, q0, zr
 ; CHECK-NEXT:    vldrht.u16 q0, [r2]
 ; CHECK-NEXT:    vstrht.16 q0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %src, i32 2, <8 x i1> %1, <8 x i16> undef)
   call void @llvm.masked.store.v8i16(<8 x i16> %2, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
 
 define void @foo_sext_v8i16_v8i8(<8 x i16> *%dest, <8 x i16> *%mask, <8 x i8> *%src) {
-; CHECK-LABEL: foo_sext_v8i16_v8i8:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r3, lr, #1
-; CHECK-NEXT:    ubfx r1, lr, #2, #1
-; CHECK-NEXT:    rsb.w r12, r3, #0
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r12, #0, #1
-; CHECK-NEXT:    bfi r3, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, lr, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, lr, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #3, #1
-; CHECK-NEXT:    ubfx r1, lr, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #4, #1
-; CHECK-NEXT:    ubfx r1, lr, #10, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #5, #1
-; CHECK-NEXT:    ubfx r1, lr, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #6, #1
-; CHECK-NEXT:    ubfx r1, lr, #14, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r3, r3, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.16 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.16 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r3
-; CHECK-NEXT:    lsls r3, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #3]
-; CHECK-NEXT:    vmovmi.16 q0[3], r3
-; CHECK-NEXT:    lsls r3, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.16 q0[4], r3
-; CHECK-NEXT:    lsls r3, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #5]
-; CHECK-NEXT:    vmovmi.16 q0[5], r3
-; CHECK-NEXT:    lsls r3, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #6]
-; CHECK-NEXT:    vmovmi.16 q0[6], r3
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #7]
-; CHECK-NEXT:    vmovmi.16 q0[7], r1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-LE-LABEL: foo_sext_v8i16_v8i8:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-LE-NEXT:    vldrbt.s16 q1, [r2]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_sext_v8i16_v8i8:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-BE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-BE-NEXT:    vrev32.16 q0, q0
+; CHECK-BE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.s16 q1, [r2]
+; CHECK-BE-NEXT:    vpsel q0, q1, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 1, <8 x i1> %1, <8 x i8> undef)
   %3 = sext <8 x i8> %2 to <8 x i16>
   call void @llvm.masked.store.v8i16(<8 x i16> %3, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
 
 define void @foo_zext_v8i16_v8i8(<8 x i16> *%dest, <8 x i16> *%mask, <8 x i8> *%src) {
-; CHECK-LABEL: foo_zext_v8i16_v8i8:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    @ implicit-def: $q0
-; CHECK-NEXT:    vmrs lr, p0
-; CHECK-NEXT:    and r3, lr, #1
-; CHECK-NEXT:    ubfx r1, lr, #2, #1
-; CHECK-NEXT:    rsb.w r12, r3, #0
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r12, #0, #1
-; CHECK-NEXT:    bfi r3, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, lr, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, lr, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #3, #1
-; CHECK-NEXT:    ubfx r1, lr, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #4, #1
-; CHECK-NEXT:    ubfx r1, lr, #10, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #5, #1
-; CHECK-NEXT:    ubfx r1, lr, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #6, #1
-; CHECK-NEXT:    ubfx r1, lr, #14, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r3, r3, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrbne r3, [r2]
-; CHECK-NEXT:    vmovne.16 q0[0], r3
-; CHECK-NEXT:    lsls r3, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #1]
-; CHECK-NEXT:    vmovmi.16 q0[1], r3
-; CHECK-NEXT:    lsls r3, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r3
-; CHECK-NEXT:    lsls r3, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #3]
-; CHECK-NEXT:    vmovmi.16 q0[3], r3
-; CHECK-NEXT:    lsls r3, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #4]
-; CHECK-NEXT:    vmovmi.16 q0[4], r3
-; CHECK-NEXT:    lsls r3, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #5]
-; CHECK-NEXT:    vmovmi.16 q0[5], r3
-; CHECK-NEXT:    lsls r3, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r3, [r2, #6]
-; CHECK-NEXT:    vmovmi.16 q0[6], r3
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    ldrbmi r1, [r2, #7]
-; CHECK-NEXT:    vmovmi.16 q0[7], r1
-; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-LE-LABEL: foo_zext_v8i16_v8i8:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-LE-NEXT:    vldrbt.u16 q1, [r2]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_zext_v8i16_v8i8:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-BE-NEXT:    vmov.i32 q0, #0x0
+; CHECK-BE-NEXT:    vrev32.16 q0, q0
+; CHECK-BE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.u16 q1, [r2]
+; CHECK-BE-NEXT:    vpsel q0, q1, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrht.16 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 1, <8 x i1> %1, <8 x i8> undef)
   %3 = zext <8 x i8> %2 to <8 x i16>
   call void @llvm.masked.store.v8i16(<8 x i16> %3, <8 x i16>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
entry:
   %0 = load <16 x i8>, <16 x i8>* %mask, align 1
   %1 = icmp sgt <16 x i8> %0, zeroinitializer
   %2 = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %src, i32 1, <16 x i1> %1, <16 x i8> undef)
   call void @llvm.masked.store.v16i8(<16 x i8> %2, <16 x i8>* %dest, i32 1, <16 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v8i8_v8i16(<8 x i8> *%dest, <8 x i16> *%mask, <8 x i16> *%src) {
-; CHECK-LABEL: foo_trunc_v8i8_v8i16:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vpt.s16 gt, q0, zr
-; CHECK-NEXT:    vldrht.u16 q0, [r2]
-; CHECK-NEXT:    vmrs r1, p0
-; CHECK-NEXT:    and r2, r1, #1
-; CHECK-NEXT:    rsbs r3, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r1, #2, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r1, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #2, #1
-; CHECK-NEXT:    ubfx r3, r1, #6, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #3, #1
-; CHECK-NEXT:    ubfx r3, r1, #8, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #4, #1
-; CHECK-NEXT:    ubfx r3, r1, #10, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #5, #1
-; CHECK-NEXT:    ubfx r3, r1, #12, #1
-; CHECK-NEXT:    ubfx r1, r1, #14, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r2, r3, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r2
-; CHECK-NEXT:    lsls r2, r2, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne.u16 r2, q0[0]
-; CHECK-NEXT:    strbne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[1]
-; CHECK-NEXT:    strbmi r2, [r0, #1]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[2]
-; CHECK-NEXT:    strbmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r2, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[3]
-; CHECK-NEXT:    strbmi r2, [r0, #3]
-; CHECK-NEXT:    lsls r2, r1, #27
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[4]
-; CHECK-NEXT:    strbmi r2, [r0, #4]
-; CHECK-NEXT:    lsls r2, r1, #26
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[5]
-; CHECK-NEXT:    strbmi r2, [r0, #5]
-; CHECK-NEXT:    lsls r2, r1, #25
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r2, q0[6]
-; CHECK-NEXT:    strbmi r2, [r0, #6]
-; CHECK-NEXT:    lsls r1, r1, #24
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi.u16 r1, q0[7]
-; CHECK-NEXT:    strbmi r1, [r0, #7]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v8i8_v8i16:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrh.u16 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.u16 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v8i8_v8i16:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrh.u16 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrht.u16 q0, [r2]
+; CHECK-BE-NEXT:    vrev16.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %src, i32 2, <8 x i1> %1, <8 x i16> undef)
   %3 = trunc <8 x i16> %2 to <8 x i8>
   call void @llvm.masked.store.v8i8(<8 x i8> %3, <8 x i8>* %dest, i32 1, <8 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v4i8_v4i32(<4 x i8> *%dest, <4 x i32> *%mask, <4 x i32> *%src) {
-; CHECK-LABEL: foo_trunc_v4i8_v4i32:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vpt.s32 gt, q0, zr
-; CHECK-NEXT:    vldrwt.u32 q0, [r2]
-; CHECK-NEXT:    vmrs r2, p0
-; CHECK-NEXT:    and r1, r2, #1
-; CHECK-NEXT:    rsbs r3, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    bfi r1, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r2, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r2, #8, #1
-; CHECK-NEXT:    ubfx r2, r2, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r1, r2, #3, #1
-; CHECK-NEXT:    lsls r2, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne r2, s0
-; CHECK-NEXT:    strbne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s1
-; CHECK-NEXT:    strbmi r2, [r0, #1]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s2
-; CHECK-NEXT:    strbmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    strbmi r1, [r0, #3]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v4i8_v4i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v4i8_v4i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-BE-NEXT:    vrev32.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   %3 = trunc <4 x i32> %2 to <4 x i8>
   call void @llvm.masked.store.v4i8(<4 x i8> %3, <4 x i8>* %dest, i32 1, <4 x i1> %1)
   ret void
 }
 
 define void @foo_trunc_v4i16_v4i32(<4 x i16> *%dest, <4 x i32> *%mask, <4 x i32> *%src) {
-; CHECK-LABEL: foo_trunc_v4i16_v4i32:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vpt.s32 gt, q0, zr
-; CHECK-NEXT:    vldrwt.u32 q0, [r2]
-; CHECK-NEXT:    vmrs r2, p0
-; CHECK-NEXT:    and r1, r2, #1
-; CHECK-NEXT:    rsbs r3, r1, #0
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    bfi r1, r3, #0, #1
-; CHECK-NEXT:    ubfx r3, r2, #4, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #1, #1
-; CHECK-NEXT:    ubfx r3, r2, #8, #1
-; CHECK-NEXT:    ubfx r2, r2, #12, #1
-; CHECK-NEXT:    rsbs r3, r3, #0
-; CHECK-NEXT:    bfi r1, r3, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r1, r2, #3, #1
-; CHECK-NEXT:    lsls r2, r1, #31
-; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    vmovne r2, s0
-; CHECK-NEXT:    strhne r2, [r0]
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s1
-; CHECK-NEXT:    strhmi r2, [r0, #2]
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r2, s2
-; CHECK-NEXT:    strhmi r2, [r0, #4]
-; CHECK-NEXT:    lsls r1, r1, #28
-; CHECK-NEXT:    itt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    strhmi r1, [r0, #6]
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    bx lr
+; CHECK-LE-LABEL: foo_trunc_v4i16_v4i32:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-LE-NEXT:    vptt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-LE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: foo_trunc_v4i16_v4i32:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrwt.u32 q0, [r2]
+; CHECK-BE-NEXT:    vrev32.8 q0, q0
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    bx lr
 entry:
   %0 = load <4 x i32>, <4 x i32>* %mask, align 4
   %1 = icmp sgt <4 x i32> %0, zeroinitializer
   %2 = call <4 x i32> @llvm.masked.load.v4i32(<4 x i32>* %src, i32 4, <4 x i1> %1, <4 x i32> undef)
   %3 = trunc <4 x i32> %2 to <4 x i16>
   call void @llvm.masked.store.v4i16(<4 x i16> %3, <4 x i16>* %dest, i32 2, <4 x i1> %1)
   ret void
 }

 entry:
   %0 = load <8 x i16>, <8 x i16>* %mask, align 2
   %1 = icmp sgt <8 x i16> %0, zeroinitializer
   %2 = call <8 x half> @llvm.masked.load.v8f16(<8 x half>* %src, i32 2, <8 x i1> %1, <8 x half> undef)
   call void @llvm.masked.store.v8f16(<8 x half> %2, <8 x half>* %dest, i32 2, <8 x i1> %1)
   ret void
 }
 
+define void @foo_v4f32_v4f16(<4 x float> *%dest, <4 x i16> *%mask, <4 x half> *%src) {
+; CHECK-LE-LABEL: foo_v4f32_v4f16:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r7, lr}
+; CHECK-LE-NEXT:    push {r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    vldrh.s32 q0, [r1]
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vmrs r3, p0
+; CHECK-LE-NEXT:    and r1, r3, #1
+; CHECK-LE-NEXT:    rsb.w r12, r1, #0
+; CHECK-LE-NEXT:    ubfx r1, r3, #4, #1
+; CHECK-LE-NEXT:    bfi lr, r12, #0, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #1, #1
+; CHECK-LE-NEXT:    ubfx r1, r3, #8, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #2, #1
+; CHECK-LE-NEXT:    ubfx r1, r3, #12, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #3, #1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #31
+; CHECK-LE-NEXT:    beq .LBB18_2
+; CHECK-LE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-LE-NEXT:    vldr.16 s4, [r2]
+; CHECK-LE-NEXT:  .LBB18_2: @ %else
+; CHECK-LE-NEXT:    lsls.w r1, lr, #30
+; CHECK-LE-NEXT:    bpl .LBB18_6
+; CHECK-LE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-LE-NEXT:    vldr.16 s0, [r2, #2]
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmovx.f16 s4, s5
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q0[0], r3
+; CHECK-LE-NEXT:    vmov.16 q0[1], r1
+; CHECK-LE-NEXT:    vmov r1, s5
+; CHECK-LE-NEXT:    vmov.16 q0[2], r1
+; CHECK-LE-NEXT:    vmov r1, s4
+; CHECK-LE-NEXT:    vmov.16 q0[3], r1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #29
+; CHECK-LE-NEXT:    bmi .LBB18_7
+; CHECK-LE-NEXT:  .LBB18_4:
+; CHECK-LE-NEXT:    vmov q1, q0
+; CHECK-LE-NEXT:    lsls.w r1, lr, #28
+; CHECK-LE-NEXT:    bmi .LBB18_8
+; CHECK-LE-NEXT:  .LBB18_5:
+; CHECK-LE-NEXT:    vmov q0, q1
+; CHECK-LE-NEXT:    b .LBB18_9
+; CHECK-LE-NEXT:  .LBB18_6:
+; CHECK-LE-NEXT:    vmov q0, q1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #29
+; CHECK-LE-NEXT:    bpl .LBB18_4
+; CHECK-LE-NEXT:  .LBB18_7: @ %cond.load4
+; CHECK-LE-NEXT:    vmovx.f16 s4, s0
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vldr.16 s8, [r2, #4]
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov.16 q1[0], r1
+; CHECK-LE-NEXT:    vmovx.f16 s0, s1
+; CHECK-LE-NEXT:    vmov.16 q1[1], r3
+; CHECK-LE-NEXT:    vmov r1, s8
+; CHECK-LE-NEXT:    vmov.16 q1[2], r1
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q1[3], r1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #28
+; CHECK-LE-NEXT:    bpl .LBB18_5
+; CHECK-LE-NEXT:  .LBB18_8: @ %cond.load7
+; CHECK-LE-NEXT:    vmovx.f16 s0, s4
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q0[0], r3
+; CHECK-LE-NEXT:    vmov.16 q0[1], r1
+; CHECK-LE-NEXT:    vmov r1, s5
+; CHECK-LE-NEXT:    vldr.16 s4, [r2, #6]
+; CHECK-LE-NEXT:    vmov.16 q0[2], r1
+; CHECK-LE-NEXT:    vmov r1, s4
+; CHECK-LE-NEXT:    vmov.16 q0[3], r1
+; CHECK-LE-NEXT:  .LBB18_9: @ %else8
+; CHECK-LE-NEXT:    vmovx.f16 s6, s1
+; CHECK-LE-NEXT:    vmovx.f16 s4, s0
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s11, s6
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s10, s1
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s9, s4
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s8, s0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrbt.8 q2, [r0]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r7, pc}
+;
+; CHECK-BE-LABEL: foo_v4f32_v4f16:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r7, lr}
+; CHECK-BE-NEXT:    push {r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    vldrh.s32 q0, [r1]
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    @ implicit-def: $q1
+; CHECK-BE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vmrs r3, p0
+; CHECK-BE-NEXT:    and r1, r3, #1
+; CHECK-BE-NEXT:    rsb.w r12, r1, #0
+; CHECK-BE-NEXT:    ubfx r1, r3, #4, #1
+; CHECK-BE-NEXT:    bfi lr, r12, #0, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #1, #1
+; CHECK-BE-NEXT:    ubfx r1, r3, #8, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #2, #1
+; CHECK-BE-NEXT:    ubfx r1, r3, #12, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #3, #1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #31
+; CHECK-BE-NEXT:    beq .LBB18_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    vldr.16 s4, [r2]
+; CHECK-BE-NEXT:  .LBB18_2: @ %else
+; CHECK-BE-NEXT:    lsls.w r1, lr, #30
+; CHECK-BE-NEXT:    bpl .LBB18_6
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    vldr.16 s0, [r2, #2]
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmovx.f16 s4, s5
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q0[0], r3
+; CHECK-BE-NEXT:    vmov.16 q0[1], r1
+; CHECK-BE-NEXT:    vmov r1, s5
+; CHECK-BE-NEXT:    vmov.16 q0[2], r1
+; CHECK-BE-NEXT:    vmov r1, s4
+; CHECK-BE-NEXT:    vmov.16 q0[3], r1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #29
+; CHECK-BE-NEXT:    bmi .LBB18_7
+; CHECK-BE-NEXT:  .LBB18_4:
+; CHECK-BE-NEXT:    vmov q1, q0
+; CHECK-BE-NEXT:    lsls.w r1, lr, #28
+; CHECK-BE-NEXT:    bmi .LBB18_8
+; CHECK-BE-NEXT:  .LBB18_5:
+; CHECK-BE-NEXT:    vmov q0, q1
+; CHECK-BE-NEXT:    b .LBB18_9
+; CHECK-BE-NEXT:  .LBB18_6:
+; CHECK-BE-NEXT:    vmov q0, q1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #29
+; CHECK-BE-NEXT:    bpl .LBB18_4
+; CHECK-BE-NEXT:  .LBB18_7: @ %cond.load4
+; CHECK-BE-NEXT:    vmovx.f16 s4, s0
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vldr.16 s8, [r2, #4]
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmov.16 q1[0], r1
+; CHECK-BE-NEXT:    vmovx.f16 s0, s1
+; CHECK-BE-NEXT:    vmov.16 q1[1], r3
+; CHECK-BE-NEXT:    vmov r1, s8
+; CHECK-BE-NEXT:    vmov.16 q1[2], r1
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q1[3], r1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #28
+; CHECK-BE-NEXT:    bpl .LBB18_5
+; CHECK-BE-NEXT:  .LBB18_8: @ %cond.load7
+; CHECK-BE-NEXT:    vmovx.f16 s0, s4
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q0[0], r3
+; CHECK-BE-NEXT:    vmov.16 q0[1], r1
+; CHECK-BE-NEXT:    vmov r1, s5
+; CHECK-BE-NEXT:    vldr.16 s4, [r2, #6]
+; CHECK-BE-NEXT:    vmov.16 q0[2], r1
+; CHECK-BE-NEXT:    vmov r1, s4
+; CHECK-BE-NEXT:    vmov.16 q0[3], r1
+; CHECK-BE-NEXT:  .LBB18_9: @ %else8
+; CHECK-BE-NEXT:    vmovx.f16 s6, s1
+; CHECK-BE-NEXT:    vmovx.f16 s4, s0
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s11, s6
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s10, s1
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s9, s4
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s8, s0
+; CHECK-BE-NEXT:    vrev32.8 q0, q2
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r7, pc}
+entry:
+  %0 = load <4 x i16>, <4 x i16>* %mask, align 2
+  %1 = icmp sgt <4 x i16> %0, zeroinitializer
+  %2 = call <4 x half> @llvm.masked.load.v4f16(<4 x half>* %src, i32 2, <4 x i1> %1, <4 x half> undef)
+  %3 = fpext <4 x half> %2 to <4 x float>
+  call void @llvm.masked.store.v4f32(<4 x float> %3, <4 x float>* %dest, i32 2, <4 x i1> %1)
+  ret void
+}
+
+define void @foo_v4f32_v4f16_unaligned(<4 x float> *%dest, <4 x i16> *%mask, <4 x half> *%src) {
+; CHECK-LE-LABEL: foo_v4f32_v4f16_unaligned:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .save {r7, lr}
+; CHECK-LE-NEXT:    push {r7, lr}
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    vldrh.s32 q0, [r1]
+; CHECK-LE-NEXT:    mov.w lr, #0
+; CHECK-LE-NEXT:    @ implicit-def: $q1
+; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vmrs r3, p0
+; CHECK-LE-NEXT:    and r1, r3, #1
+; CHECK-LE-NEXT:    rsb.w r12, r1, #0
+; CHECK-LE-NEXT:    ubfx r1, r3, #4, #1
+; CHECK-LE-NEXT:    bfi lr, r12, #0, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #1, #1
+; CHECK-LE-NEXT:    ubfx r1, r3, #8, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #2, #1
+; CHECK-LE-NEXT:    ubfx r1, r3, #12, #1
+; CHECK-LE-NEXT:    rsbs r1, r1, #0
+; CHECK-LE-NEXT:    bfi lr, r1, #3, #1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #31
+; CHECK-LE-NEXT:    beq .LBB19_2
+; CHECK-LE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-LE-NEXT:    vldr.16 s4, [r2]
+; CHECK-LE-NEXT:  .LBB19_2: @ %else
+; CHECK-LE-NEXT:    lsls.w r1, lr, #30
+; CHECK-LE-NEXT:    bpl .LBB19_6
+; CHECK-LE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-LE-NEXT:    vldr.16 s0, [r2, #2]
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmovx.f16 s4, s5
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q0[0], r3
+; CHECK-LE-NEXT:    vmov.16 q0[1], r1
+; CHECK-LE-NEXT:    vmov r1, s5
+; CHECK-LE-NEXT:    vmov.16 q0[2], r1
+; CHECK-LE-NEXT:    vmov r1, s4
+; CHECK-LE-NEXT:    vmov.16 q0[3], r1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #29
+; CHECK-LE-NEXT:    bmi .LBB19_7
+; CHECK-LE-NEXT:  .LBB19_4:
+; CHECK-LE-NEXT:    vmov q1, q0
+; CHECK-LE-NEXT:    lsls.w r1, lr, #28
+; CHECK-LE-NEXT:    bmi .LBB19_8
+; CHECK-LE-NEXT:  .LBB19_5:
+; CHECK-LE-NEXT:    vmov q0, q1
+; CHECK-LE-NEXT:    b .LBB19_9
+; CHECK-LE-NEXT:  .LBB19_6:
+; CHECK-LE-NEXT:    vmov q0, q1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #29
+; CHECK-LE-NEXT:    bpl .LBB19_4
+; CHECK-LE-NEXT:  .LBB19_7: @ %cond.load4
+; CHECK-LE-NEXT:    vmovx.f16 s4, s0
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vldr.16 s8, [r2, #4]
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov.16 q1[0], r1
+; CHECK-LE-NEXT:    vmovx.f16 s0, s1
+; CHECK-LE-NEXT:    vmov.16 q1[1], r3
+; CHECK-LE-NEXT:    vmov r1, s8
+; CHECK-LE-NEXT:    vmov.16 q1[2], r1
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q1[3], r1
+; CHECK-LE-NEXT:    lsls.w r1, lr, #28
+; CHECK-LE-NEXT:    bpl .LBB19_5
+; CHECK-LE-NEXT:  .LBB19_8: @ %cond.load7
+; CHECK-LE-NEXT:    vmovx.f16 s0, s4
+; CHECK-LE-NEXT:    vmov r3, s4
+; CHECK-LE-NEXT:    vmov r1, s0
+; CHECK-LE-NEXT:    vmov.16 q0[0], r3
+; CHECK-LE-NEXT:    vmov.16 q0[1], r1
+; CHECK-LE-NEXT:    vmov r1, s5
+; CHECK-LE-NEXT:    vldr.16 s4, [r2, #6]
+; CHECK-LE-NEXT:    vmov.16 q0[2], r1
+; CHECK-LE-NEXT:    vmov r1, s4
+; CHECK-LE-NEXT:    vmov.16 q0[3], r1
+; CHECK-LE-NEXT:  .LBB19_9: @ %else8
+; CHECK-LE-NEXT:    vmovx.f16 s6, s1
+; CHECK-LE-NEXT:    vmovx.f16 s4, s0
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s11, s6
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s10, s1
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s9, s4
+; CHECK-LE-NEXT:    vcvtb.f32.f16 s8, s0
+; CHECK-LE-NEXT:    vpst
+; CHECK-LE-NEXT:    vstrbt.8 q2, [r0]
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    pop {r7, pc}
+;
+; CHECK-BE-LABEL: foo_v4f32_v4f16_unaligned:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .save {r7, lr}
+; CHECK-BE-NEXT:    push {r7, lr}
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    vldrh.s32 q0, [r1]
+; CHECK-BE-NEXT:    mov.w lr, #0
+; CHECK-BE-NEXT:    @ implicit-def: $q1
+; CHECK-BE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vmrs r3, p0
+; CHECK-BE-NEXT:    and r1, r3, #1
+; CHECK-BE-NEXT:    rsb.w r12, r1, #0
+; CHECK-BE-NEXT:    ubfx r1, r3, #4, #1
+; CHECK-BE-NEXT:    bfi lr, r12, #0, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #1, #1
+; CHECK-BE-NEXT:    ubfx r1, r3, #8, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #2, #1
+; CHECK-BE-NEXT:    ubfx r1, r3, #12, #1
+; CHECK-BE-NEXT:    rsbs r1, r1, #0
+; CHECK-BE-NEXT:    bfi lr, r1, #3, #1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #31
+; CHECK-BE-NEXT:    beq .LBB19_2
+; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
+; CHECK-BE-NEXT:    vldr.16 s4, [r2]
+; CHECK-BE-NEXT:  .LBB19_2: @ %else
+; CHECK-BE-NEXT:    lsls.w r1, lr, #30
+; CHECK-BE-NEXT:    bpl .LBB19_6
+; CHECK-BE-NEXT:  @ %bb.3: @ %cond.load1
+; CHECK-BE-NEXT:    vldr.16 s0, [r2, #2]
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmovx.f16 s4, s5
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q0[0], r3
+; CHECK-BE-NEXT:    vmov.16 q0[1], r1
+; CHECK-BE-NEXT:    vmov r1, s5
+; CHECK-BE-NEXT:    vmov.16 q0[2], r1
+; CHECK-BE-NEXT:    vmov r1, s4
+; CHECK-BE-NEXT:    vmov.16 q0[3], r1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #29
+; CHECK-BE-NEXT:    bmi .LBB19_7
+; CHECK-BE-NEXT:  .LBB19_4:
+; CHECK-BE-NEXT:    vmov q1, q0
+; CHECK-BE-NEXT:    lsls.w r1, lr, #28
+; CHECK-BE-NEXT:    bmi .LBB19_8
+; CHECK-BE-NEXT:  .LBB19_5:
+; CHECK-BE-NEXT:    vmov q0, q1
+; CHECK-BE-NEXT:    b .LBB19_9
+; CHECK-BE-NEXT:  .LBB19_6:
+; CHECK-BE-NEXT:    vmov q0, q1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #29
+; CHECK-BE-NEXT:    bpl .LBB19_4
+; CHECK-BE-NEXT:  .LBB19_7: @ %cond.load4
+; CHECK-BE-NEXT:    vmovx.f16 s4, s0
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vldr.16 s8, [r2, #4]
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmov.16 q1[0], r1
+; CHECK-BE-NEXT:    vmovx.f16 s0, s1
+; CHECK-BE-NEXT:    vmov.16 q1[1], r3
+; CHECK-BE-NEXT:    vmov r1, s8
+; CHECK-BE-NEXT:    vmov.16 q1[2], r1
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q1[3], r1
+; CHECK-BE-NEXT:    lsls.w r1, lr, #28
+; CHECK-BE-NEXT:    bpl .LBB19_5
+; CHECK-BE-NEXT:  .LBB19_8: @ %cond.load7
+; CHECK-BE-NEXT:    vmovx.f16 s0, s4
+; CHECK-BE-NEXT:    vmov r3, s4
+; CHECK-BE-NEXT:    vmov r1, s0
+; CHECK-BE-NEXT:    vmov.16 q0[0], r3
+; CHECK-BE-NEXT:    vmov.16 q0[1], r1
+; CHECK-BE-NEXT:    vmov r1, s5
+; CHECK-BE-NEXT:    vldr.16 s4, [r2, #6]
+; CHECK-BE-NEXT:    vmov.16 q0[2], r1
+; CHECK-BE-NEXT:    vmov r1, s4
+; CHECK-BE-NEXT:    vmov.16 q0[3], r1
+; CHECK-BE-NEXT:  .LBB19_9: @ %else8
+; CHECK-BE-NEXT:    vmovx.f16 s6, s1
+; CHECK-BE-NEXT:    vmovx.f16 s4, s0
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s11, s6
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s10, s1
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s9, s4
+; CHECK-BE-NEXT:    vcvtb.f32.f16 s8, s0
+; CHECK-BE-NEXT:    vrev32.8 q0, q2
+; CHECK-BE-NEXT:    vpst
+; CHECK-BE-NEXT:    vstrbt.8 q0, [r0]
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    pop {r7, pc}
+entry:
+  %0 = load <4 x i16>, <4 x i16>* %mask, align 2
+  %1 = icmp sgt <4 x i16> %0, zeroinitializer
+  %2 = call <4 x half> @llvm.masked.load.v4f16(<4 x half>* %src, i32 2, <4 x i1> %1, <4 x half> undef)
+  %3 = fpext <4 x half> %2 to <4 x float>
+  call void @llvm.masked.store.v4f32(<4 x float> %3, <4 x float>* %dest, i32 1, <4 x i1> %1)
+  ret void
+}
+
 declare void @llvm.masked.store.v4i32(<4 x i32>, <4 x i32>*, i32, <4 x i1>)
 declare void @llvm.masked.store.v8i16(<8 x i16>, <8 x i16>*, i32, <8 x i1>)
 declare void @llvm.masked.store.v16i8(<16 x i8>, <16 x i8>*, i32, <16 x i1>)
 declare void @llvm.masked.store.v8f16(<8 x half>, <8 x half>*, i32, <8 x i1>)
 declare void @llvm.masked.store.v4f32(<4 x float>, <4 x float>*, i32, <4 x i1>)
 declare <16 x i8> @llvm.masked.load.v16i8(<16 x i8>*, i32, <16 x i1>, <16 x i8>)
 declare <8 x i16> @llvm.masked.load.v8i16(<8 x i16>*, i32, <8 x i1>, <8 x i16>)
+declare <2 x i32> @llvm.masked.load.v2i32(<2 x i32>*, i32, <2 x i1>, <2 x i32>)
 declare <4 x i32> @llvm.masked.load.v4i32(<4 x i32>*, i32, <4 x i1>, <4 x i32>)
 declare <4 x float> @llvm.masked.load.v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
+declare <4 x half> @llvm.masked.load.v4f16(<4 x half>*, i32, <4 x i1>, <4 x half>)
 declare <8 x half> @llvm.masked.load.v8f16(<8 x half>*, i32, <8 x i1>, <8 x half>)
 
 declare void @llvm.masked.store.v8i8(<8 x i8>, <8 x i8>*, i32, <8 x i1>)
 declare void @llvm.masked.store.v4i8(<4 x i8>, <4 x i8>*, i32, <4 x i1>)
 declare void @llvm.masked.store.v4i16(<4 x i16>, <4 x i16>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v2i64(<2 x i64>, <2 x i64>*, i32, <2 x i1>)
 declare <4 x i16> @llvm.masked.load.v4i16(<4 x i16>*, i32, <4 x i1>, <4 x i16>)
 declare <4 x i8> @llvm.masked.load.v4i8(<4 x i8>*, i32, <4 x i1>, <4 x i8>)
 declare <8 x i8> @llvm.masked.load.v8i8(<8 x i8>*, i32, <8 x i1>, <8 x i8>)

test/CodeGen/Thumb2/mve-masked-load.ll

 ; CHECK-BE-NEXT:    vrev64.32 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %l = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %dest, i32 4, <4 x i1> %c, <4 x i32> %a)
   ret <4 x i32> %l
 }
 
+define arm_aapcs_vfpcc <4 x i32> @zext16_masked_v4i32_align2_zero(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: zext16_masked_v4i32_align2_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.u32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext16_masked_v4i32_align2_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrht.u32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> zeroinitializer)
+  %ext = zext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext16_masked_v4i32_align2_undef(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: zext16_masked_v4i32_align2_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.u32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext16_masked_v4i32_align2_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrht.u32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> undef)
+  %ext = zext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext16_masked_v4i32_align1_undef(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: zext16_masked_v4i32_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    @ implicit-def: $q0
+; CHECK-LE-NEXT:    vmrs r2, p0
+; CHECK-LE-NEXT:    and r1, r2, #1
+; CHECK-LE-NEXT:    rsbs r3, r1, #0
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    ubfx r3, r2, #4, #1
+; CHECK-LE-NEXT:    rsbs r3, r3, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    ubfx r3, r2, #8, #1
+; CHECK-LE-NEXT:    ubfx r2, r2, #12, #1
+; CHECK-LE-NEXT:    rsbs r3, r3, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #2, #1
+; CHECK-LE-NEXT:    rsbs r2, r2, #0
+; CHECK-LE-NEXT:    bfi r1, r2, #3, #1
+; CHECK-LE-NEXT:    lsls r2, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrhne r2, [r0]
+; CHECK-LE-NEXT:    vmovne.32 q0[0], r2
+; CHECK-LE-NEXT:    lsls r2, r1, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r2, [r0, #2]
+; CHECK-LE-NEXT:    vmovmi.32 q0[1], r2
+; CHECK-LE-NEXT:    lsls r2, r1, #29
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r2, [r0, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q0[2], r2
+; CHECK-LE-NEXT:    lsls r1, r1, #28
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r0, [r0, #6]
+; CHECK-LE-NEXT:    vmovmi.32 q0[3], r0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext16_masked_v4i32_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    @ implicit-def: $q0
+; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vmrs r2, p0
+; CHECK-BE-NEXT:    and r1, r2, #1
+; CHECK-BE-NEXT:    rsbs r3, r1, #0
+; CHECK-BE-NEXT:    movs r1, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-BE-NEXT:    ubfx r3, r2, #4, #1
+; CHECK-BE-NEXT:    rsbs r3, r3, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-BE-NEXT:    ubfx r3, r2, #8, #1
+; CHECK-BE-NEXT:    ubfx r2, r2, #12, #1
+; CHECK-BE-NEXT:    rsbs r3, r3, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #2, #1
+; CHECK-BE-NEXT:    rsbs r2, r2, #0
+; CHECK-BE-NEXT:    bfi r1, r2, #3, #1
+; CHECK-BE-NEXT:    lsls r2, r1, #31
+; CHECK-BE-NEXT:    itt ne
+; CHECK-BE-NEXT:    ldrhne r2, [r0]
+; CHECK-BE-NEXT:    vmovne.32 q0[0], r2
+; CHECK-BE-NEXT:    lsls r2, r1, #30
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r2, [r0, #2]
+; CHECK-BE-NEXT:    vmovmi.32 q0[1], r2
+; CHECK-BE-NEXT:    lsls r2, r1, #29
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r2, [r0, #4]
+; CHECK-BE-NEXT:    vmovmi.32 q0[2], r2
+; CHECK-BE-NEXT:    lsls r1, r1, #28
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r0, [r0, #6]
+; CHECK-BE-NEXT:    vmovmi.32 q0[3], r0
+; CHECK-BE-NEXT:    vmovlb.s16 q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 1, <4 x i1> %c, <4 x i16> undef)
+  %ext = sext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext16_masked_v4i32_align2_other(<4 x i16> *%dest, <4 x i16> %a) {
+; CHECK-LE-LABEL: zext16_masked_v4i32_align2_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.u16 q1, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.u32 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext16_masked_v4i32_align2_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.u16 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q1, q1
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrht.u32 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i16> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> %a)
+  %ext = zext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext16_masked_v4i32_align2_zero(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: sext16_masked_v4i32_align2_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.s32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext16_masked_v4i32_align2_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrht.s32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> zeroinitializer)
+  %sext = sext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %sext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext16_masked_v4i32_align2_undef(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: sext16_masked_v4i32_align2_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.s32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext16_masked_v4i32_align2_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrht.s32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> undef)
+  %sext = sext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %sext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext16_masked_v4i32_align1_undef(<4 x i16> *%dest, <4 x i32> %a) {
+; CHECK-LE-LABEL: sext16_masked_v4i32_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    .pad #4
+; CHECK-LE-NEXT:    sub sp, #4
+; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    @ implicit-def: $q0
+; CHECK-LE-NEXT:    vmrs r2, p0
+; CHECK-LE-NEXT:    and r1, r2, #1
+; CHECK-LE-NEXT:    rsbs r3, r1, #0
+; CHECK-LE-NEXT:    movs r1, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-LE-NEXT:    ubfx r3, r2, #4, #1
+; CHECK-LE-NEXT:    rsbs r3, r3, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-LE-NEXT:    ubfx r3, r2, #8, #1
+; CHECK-LE-NEXT:    ubfx r2, r2, #12, #1
+; CHECK-LE-NEXT:    rsbs r3, r3, #0
+; CHECK-LE-NEXT:    bfi r1, r3, #2, #1
+; CHECK-LE-NEXT:    rsbs r2, r2, #0
+; CHECK-LE-NEXT:    bfi r1, r2, #3, #1
+; CHECK-LE-NEXT:    lsls r2, r1, #31
+; CHECK-LE-NEXT:    itt ne
+; CHECK-LE-NEXT:    ldrhne r2, [r0]
+; CHECK-LE-NEXT:    vmovne.32 q0[0], r2
+; CHECK-LE-NEXT:    lsls r2, r1, #30
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r2, [r0, #2]
+; CHECK-LE-NEXT:    vmovmi.32 q0[1], r2
+; CHECK-LE-NEXT:    lsls r2, r1, #29
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r2, [r0, #4]
+; CHECK-LE-NEXT:    vmovmi.32 q0[2], r2
+; CHECK-LE-NEXT:    lsls r1, r1, #28
+; CHECK-LE-NEXT:    itt mi
+; CHECK-LE-NEXT:    ldrhmi r0, [r0, #6]
+; CHECK-LE-NEXT:    vmovmi.32 q0[3], r0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    add sp, #4
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext16_masked_v4i32_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    .pad #4
+; CHECK-BE-NEXT:    sub sp, #4
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    @ implicit-def: $q0
+; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vmrs r2, p0
+; CHECK-BE-NEXT:    and r1, r2, #1
+; CHECK-BE-NEXT:    rsbs r3, r1, #0
+; CHECK-BE-NEXT:    movs r1, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #0, #1
+; CHECK-BE-NEXT:    ubfx r3, r2, #4, #1
+; CHECK-BE-NEXT:    rsbs r3, r3, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #1, #1
+; CHECK-BE-NEXT:    ubfx r3, r2, #8, #1
+; CHECK-BE-NEXT:    ubfx r2, r2, #12, #1
+; CHECK-BE-NEXT:    rsbs r3, r3, #0
+; CHECK-BE-NEXT:    bfi r1, r3, #2, #1
+; CHECK-BE-NEXT:    rsbs r2, r2, #0
+; CHECK-BE-NEXT:    bfi r1, r2, #3, #1
+; CHECK-BE-NEXT:    lsls r2, r1, #31
+; CHECK-BE-NEXT:    itt ne
+; CHECK-BE-NEXT:    ldrhne r2, [r0]
+; CHECK-BE-NEXT:    vmovne.32 q0[0], r2
+; CHECK-BE-NEXT:    lsls r2, r1, #30
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r2, [r0, #2]
+; CHECK-BE-NEXT:    vmovmi.32 q0[1], r2
+; CHECK-BE-NEXT:    lsls r2, r1, #29
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r2, [r0, #4]
+; CHECK-BE-NEXT:    vmovmi.32 q0[2], r2
+; CHECK-BE-NEXT:    lsls r1, r1, #28
+; CHECK-BE-NEXT:    itt mi
+; CHECK-BE-NEXT:    ldrhmi r0, [r0, #6]
+; CHECK-BE-NEXT:    vmovmi.32 q0[3], r0
+; CHECK-BE-NEXT:    vmovlb.s16 q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    add sp, #4
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i32> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 1, <4 x i1> %c, <4 x i16> undef)
+  %sext = sext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %sext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext16_masked_v4i32_align2_other(<4 x i16> *%dest, <4 x i16> %a) {
+; CHECK-LE-LABEL: sext16_masked_v4i32_align2_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrht.s32 q1, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext16_masked_v4i32_align2_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q1
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrht.s32 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i16> %a, zeroinitializer
+  %l = call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %dest, i32 2, <4 x i1> %c, <4 x i16> %a)
+  %sext = sext <4 x i16> %l to <4 x i32>
+  ret <4 x i32> %sext
+}
+
 define arm_aapcs_vfpcc i8* @masked_v4i32_preinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4i32_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0, #4]
 ; CHECK-LE-NEXT:    adds r0, #4
 ; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
entry:
   %0 = bitcast i8* %x to <4 x i32>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %0, i32 4, <4 x i1> %c, <4 x i32> undef)
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
-
-
 define arm_aapcs_vfpcc <8 x i16> @masked_v8i16_align4_zero(<8 x i16> *%dest, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr

 ; CHECK-BE-NEXT:    vrev64.16 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %l = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %dest, i32 2, <8 x i1> %c, <8 x i16> %a)
   ret <8 x i16> %l
 }
 
+define arm_aapcs_vfpcc <8 x i16> @sext8_masked_v8i16_align1_zero(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v8i16_align1_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s16 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v8i16_align1_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-BE-NEXT:    vrev64.16 q2, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q2
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s16 q0, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q0, q1
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> zeroinitializer)
+  %ext = sext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
+define arm_aapcs_vfpcc <8 x i16> @sext8_masked_v8i16_align1_undef(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v8i16_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s16 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v8i16_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-BE-NEXT:    vrev64.16 q2, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q2
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s16 q0, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q0, q1
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> undef)
+  %ext = sext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
+define arm_aapcs_vfpcc <8 x i16> @sext8_masked_v8i16_align1_other(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v8i16_align1_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s16 q1, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v8i16_align1_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.16 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s16 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> %a)
+  %ext = sext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext8_masked_v4i32_align1_zero(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v4i32_align1_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v4i32_align1_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> zeroinitializer)
+  %ext = sext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext8_masked_v4i32_align1_undef(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v4i32_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v4i32_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> undef)
+  %ext = sext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @sext8_masked_v4i32_align1_other(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: sext8_masked_v4i32_align1_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.s32 q1, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q1, q0
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: sext8_masked_v4i32_align1_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.s32 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> %a)
+  %ext = sext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext8_masked_v4i32_align1_zero(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v4i32_align1_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v4i32_align1_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.u32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> zeroinitializer)
+  %ext = zext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext8_masked_v4i32_align1_undef(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v4i32_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u32 q0, [r0]
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v4i32_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.32 q1, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-BE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.u32 q1, [r0]
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> undef)
+  %ext = zext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <4 x i32> @zext8_masked_v4i32_align1_other(<4 x i8> *%dest, <4 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v4i32_align1_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmov.i32 q1, #0xff
+; CHECK-LE-NEXT:    vand q1, q0, q1
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmovlb.s16 q0, q0
+; CHECK-LE-NEXT:    vpt.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u32 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v4i32_align1_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vmov.i32 q1, #0xff
+; CHECK-BE-NEXT:    vrev64.32 q2, q0
+; CHECK-BE-NEXT:    vand q0, q2, q1
+; CHECK-BE-NEXT:    vmovlb.s8 q1, q2
+; CHECK-BE-NEXT:    vmovlb.s16 q1, q1
+; CHECK-BE-NEXT:    vpt.s32 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.u32 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.32 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <4 x i8> %a, zeroinitializer
+  %l = call <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>* %dest, i32 1, <4 x i1> %c, <4 x i8> %a)
+  %ext = zext <4 x i8> %l to <4 x i32>
+  ret <4 x i32> %ext
+}
+
+define arm_aapcs_vfpcc <8 x i16> @zext8_masked_v8i16_align1_zero(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v8i16_align1_zero:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u16 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v8i16_align1_zero:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-BE-NEXT:    vrev64.16 q2, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q2
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.u16 q0, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q0, q1
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> zeroinitializer)
+  %ext = zext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
+define arm_aapcs_vfpcc <8 x i16> @zext8_masked_v8i16_align1_undef(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v8i16_align1_undef:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u16 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v8i16_align1_undef:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
+; CHECK-BE-NEXT:    vrev64.16 q2, q0
+; CHECK-BE-NEXT:    vmovlb.s8 q0, q2
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-BE-NEXT:    vldrbt.u16 q0, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q0, q1
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> undef)
+  %ext = zext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
+define arm_aapcs_vfpcc <8 x i16> @zext8_masked_v8i16_align1_other(<8 x i8> *%dest, <8 x i8> %a) {
+; CHECK-LE-LABEL: zext8_masked_v8i16_align1_other:
+; CHECK-LE:       @ %bb.0: @ %entry
+; CHECK-LE-NEXT:    vmovlb.u8 q1, q0
+; CHECK-LE-NEXT:    vmovlb.s8 q0, q0
+; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vldrbt.u16 q0, [r0]
+; CHECK-LE-NEXT:    vpsel q0, q0, q1
+; CHECK-LE-NEXT:    bx lr
+;
+; CHECK-BE-LABEL: zext8_masked_v8i16_align1_other:
+; CHECK-BE:       @ %bb.0: @ %entry
+; CHECK-BE-NEXT:    vrev64.16 q1, q0
+; CHECK-BE-NEXT:    vmovlb.u8 q0, q1
+; CHECK-BE-NEXT:    vmovlb.s8 q1, q1
+; CHECK-BE-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-BE-NEXT:    vldrbt.u16 q1, [r0]
+; CHECK-BE-NEXT:    vpsel q1, q1, q0
+; CHECK-BE-NEXT:    vrev64.16 q0, q1
+; CHECK-BE-NEXT:    bx lr
+entry:
+  %c = icmp sgt <8 x i8> %a, zeroinitializer
+  %l = call <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>* %dest, i32 1, <8 x i1> %c, <8 x i8> %a)
+  %ext = zext <8 x i8> %l to <8 x i16>
+  ret <8 x i16> %ext
+}
+
 define i8* @masked_v8i16_preinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vldr d1, [sp]
 ; CHECK-LE-NEXT:    vmov d0, r2, r3
 ; CHECK-LE-NEXT:    vpt.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0, #4]
 ; CHECK-LE-NEXT:    adds r0, #4
entry:
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <16 x i8> @masked_v16i8_align4_zero(<16 x i8> *%dest, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpt.s8 gt, q0, zr
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.8 q2, q0

 ; CHECK-LE-NEXT:    it lt
 ; CHECK-LE-NEXT:    movlt r2, #1
 ; CHECK-LE-NEXT:    cmp r2, #0
 ; CHECK-LE-NEXT:    it ne
 ; CHECK-LE-NEXT:    mvnne r2, #1
 ; CHECK-LE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-LE-NEXT:    and r1, r2, #3
 ; CHECK-LE-NEXT:    lsls r2, r2, #31
-; CHECK-LE-NEXT:    beq .LBB29_2
+; CHECK-LE-NEXT:    beq .LBB49_2
 ; CHECK-LE-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-LE-NEXT:    vldr d1, .LCPI29_0
+; CHECK-LE-NEXT:    vldr d1, .LCPI49_0
 ; CHECK-LE-NEXT:    vldr d0, [r0]
-; CHECK-LE-NEXT:    b .LBB29_3
-; CHECK-LE-NEXT:  .LBB29_2:
+; CHECK-LE-NEXT:    b .LBB49_3
+; CHECK-LE-NEXT:  .LBB49_2:
 ; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
-; CHECK-LE-NEXT:  .LBB29_3: @ %else
+; CHECK-LE-NEXT:  .LBB49_3: @ %else
 ; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    it mi
 ; CHECK-LE-NEXT:    vldrmi d1, [r0, #8]
 ; CHECK-LE-NEXT:    add sp, #4
 ; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:    .p2align 3
 ; CHECK-LE-NEXT:  @ %bb.4:
-; CHECK-LE-NEXT:  .LCPI29_0:
+; CHECK-LE-NEXT:  .LCPI49_0:
 ; CHECK-LE-NEXT:    .long 0 @ double 0
 ; CHECK-LE-NEXT:    .long 0
 ;
 ; CHECK-BE-LABEL: masked_v2i64_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    .pad #4
 ; CHECK-BE-NEXT:    sub sp, #4
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0

 ; CHECK-BE-NEXT:    it lt
 ; CHECK-BE-NEXT:    movlt r2, #1
 ; CHECK-BE-NEXT:    cmp r2, #0
 ; CHECK-BE-NEXT:    it ne
 ; CHECK-BE-NEXT:    mvnne r2, #1
 ; CHECK-BE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-BE-NEXT:    and r1, r2, #3
 ; CHECK-BE-NEXT:    lsls r2, r2, #31
-; CHECK-BE-NEXT:    beq .LBB29_2
+; CHECK-BE-NEXT:    beq .LBB49_2
 ; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-BE-NEXT:    vldr d1, .LCPI29_0
+; CHECK-BE-NEXT:    vldr d1, .LCPI49_0
 ; CHECK-BE-NEXT:    vldr d0, [r0]
-; CHECK-BE-NEXT:    b .LBB29_3
-; CHECK-BE-NEXT:  .LBB29_2:
+; CHECK-BE-NEXT:    b .LBB49_3
+; CHECK-BE-NEXT:  .LBB49_2:
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.32 q0, q1
-; CHECK-BE-NEXT:  .LBB29_3: @ %else
+; CHECK-BE-NEXT:  .LBB49_3: @ %else
 ; CHECK-BE-NEXT:    lsls r1, r1, #30
 ; CHECK-BE-NEXT:    it mi
 ; CHECK-BE-NEXT:    vldrmi d1, [r0, #8]
 ; CHECK-BE-NEXT:    add sp, #4
 ; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:    .p2align 3
 ; CHECK-BE-NEXT:  @ %bb.4:
-; CHECK-BE-NEXT:  .LCPI29_0:
+; CHECK-BE-NEXT:  .LCPI49_0:
 ; CHECK-BE-NEXT:    .long 0 @ double 0
 ; CHECK-BE-NEXT:    .long 0
 entry:
   %c = icmp sgt <2 x i64> %a, zeroinitializer
   %l = call <2 x i64> @llvm.masked.load.v2i64.p0v2i64(<2 x i64>* %dest, i32 8, <2 x i1> %c, <2 x i64> zeroinitializer)
   ret <2 x i64> %l
 }
 

 ; CHECK-LE-NEXT:    it lt
 ; CHECK-LE-NEXT:    movlt r2, #1
 ; CHECK-LE-NEXT:    cmp r2, #0
 ; CHECK-LE-NEXT:    it ne
 ; CHECK-LE-NEXT:    mvnne r2, #1
 ; CHECK-LE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-LE-NEXT:    and r1, r2, #3
 ; CHECK-LE-NEXT:    lsls r2, r2, #31
-; CHECK-LE-NEXT:    beq .LBB30_2
+; CHECK-LE-NEXT:    beq .LBB50_2
 ; CHECK-LE-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-LE-NEXT:    vldr d1, .LCPI30_0
+; CHECK-LE-NEXT:    vldr d1, .LCPI50_0
 ; CHECK-LE-NEXT:    vldr d0, [r0]
-; CHECK-LE-NEXT:    b .LBB30_3
-; CHECK-LE-NEXT:  .LBB30_2:
+; CHECK-LE-NEXT:    b .LBB50_3
+; CHECK-LE-NEXT:  .LBB50_2:
 ; CHECK-LE-NEXT:    vmov.i32 q0, #0x0
-; CHECK-LE-NEXT:  .LBB30_3: @ %else
+; CHECK-LE-NEXT:  .LBB50_3: @ %else
 ; CHECK-LE-NEXT:    lsls r1, r1, #30
 ; CHECK-LE-NEXT:    it mi
 ; CHECK-LE-NEXT:    vldrmi d1, [r0, #8]
 ; CHECK-LE-NEXT:    add sp, #4
 ; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:    .p2align 3
 ; CHECK-LE-NEXT:  @ %bb.4:
-; CHECK-LE-NEXT:  .LCPI30_0:
+; CHECK-LE-NEXT:  .LCPI50_0:
 ; CHECK-LE-NEXT:    .long 0 @ double 0
 ; CHECK-LE-NEXT:    .long 0
 ;
 ; CHECK-BE-LABEL: masked_v2f64_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    .pad #4
 ; CHECK-BE-NEXT:    sub sp, #4
 ; CHECK-BE-NEXT:    vrev64.32 q0, q1

 ; CHECK-BE-NEXT:    it lt
 ; CHECK-BE-NEXT:    movlt r2, #1
 ; CHECK-BE-NEXT:    cmp r2, #0
 ; CHECK-BE-NEXT:    it ne
 ; CHECK-BE-NEXT:    mvnne r2, #1
 ; CHECK-BE-NEXT:    bfi r2, r1, #0, #1
 ; CHECK-BE-NEXT:    and r1, r2, #3
 ; CHECK-BE-NEXT:    lsls r2, r2, #31
-; CHECK-BE-NEXT:    beq .LBB30_2
+; CHECK-BE-NEXT:    beq .LBB50_2
 ; CHECK-BE-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-BE-NEXT:    vldr d1, .LCPI30_0
+; CHECK-BE-NEXT:    vldr d1, .LCPI50_0
 ; CHECK-BE-NEXT:    vldr d0, [r0]
-; CHECK-BE-NEXT:    b .LBB30_3
-; CHECK-BE-NEXT:  .LBB30_2:
+; CHECK-BE-NEXT:    b .LBB50_3
+; CHECK-BE-NEXT:  .LBB50_2:
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.32 q0, q1
-; CHECK-BE-NEXT:  .LBB30_3: @ %else
+; CHECK-BE-NEXT:  .LBB50_3: @ %else
 ; CHECK-BE-NEXT:    lsls r1, r1, #30
 ; CHECK-BE-NEXT:    it mi
 ; CHECK-BE-NEXT:    vldrmi d1, [r0, #8]
 ; CHECK-BE-NEXT:    add sp, #4
 ; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:    .p2align 3
 ; CHECK-BE-NEXT:  @ %bb.4:
-; CHECK-BE-NEXT:  .LCPI30_0:
+; CHECK-BE-NEXT:  .LCPI50_0:
 ; CHECK-BE-NEXT:    .long 0 @ double 0
 ; CHECK-BE-NEXT:    .long 0
 entry:
   %c = icmp sgt <2 x i64> %b, zeroinitializer
   %l = call <2 x double> @llvm.masked.load.v2f64.p0v2f64(<2 x double>* %dest, i32 8, <2 x i1> %c, <2 x double> zeroinitializer)
   ret <2 x double> %l
 }
 
+declare <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>*, i32, <4 x i1>, <4 x i16>)
 declare <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>*, i32, <4 x i1>, <4 x i32>)
 declare <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>*, i32, <8 x i1>, <8 x i16>)
+declare <4 x i8> @llvm.masked.load.v4i8.p0v4i8(<4 x i8>*, i32, <4 x i1>, <4 x i8>)
+declare <8 x i8> @llvm.masked.load.v8i8.p0v8i8(<8 x i8>*, i32, <8 x i1>, <8 x i8>)
 declare <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>*, i32, <16 x i1>, <16 x i8>)
 declare <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
 declare <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>*, i32, <8 x i1>, <8 x half>)
 declare <2 x i64> @llvm.masked.load.v2i64.p0v2i64(<2 x i64>*, i32, <2 x i1>, <2 x i64>)
 declare <2 x double> @llvm.masked.load.v2f64.p0v2f64(<2 x double>*, i32, <2 x i1>, <2 x double>)