[ARM] Add support for MVE pre and post inc loads and stores.

[dmgreen]

Summary

This adds pre- and post- increment and decrements for MVE loads and stores. It uses the builtin pre and post load/store detection, unlike Neon. Loads are selected with the code in tryT2IndexedLoad, stores are selected with tablegen patterns. The immediates have a +/-7bit range, multiplied by the size of the element.

[SjoerdMeijer]

llvm/include/llvm/Target/TargetSelectionDAG.td

1114 ↗

(On Diff #206735)

nit: indent off by 1

1156 ↗

(On Diff #206735)

here

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

151 ↗

(On Diff #206735)

and here

153 ↗

(On Diff #206735)

and here

that's it, just nitpicking! :-)

llvm/lib/Target/ARM/ARMISelLowering.cpp

332 ↗

(On Diff #206735)

perhaps we could have a little bit of fun with c++ and do something like this:

for (unsigned I : ISD::pre_inc_ins())

where pre_inc_ins() creates an iterator range, similar to what happens in MachineValueType.h

[dmgreen]

The original version wasn't getting alignment correct in some cases. For little endian which instruction we choose has more to do with alignment and offset than type being loaded. I've rewritten parts of this to, at least for LE, get this hopefully more correct. I've tried to add BE too, but not added any tests for that yet. We still have a task to sort out BE properly.

[SjoerdMeijer]

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

1610 ↗

(On Diff #209840)

bail here for BE?

1612 ↗

(On Diff #209840)

Why the >= comparisons for the alignment? Is that right?

1631 ↗

(On Diff #209840)

I am wondering if this is logically correct, the (IsLE || ... in particular.
Perhaps we don't need the IsLE check here, if we bail for BE earlier.

[dmgreen]

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

1610 ↗

(On Diff #209840)

We still need to select instructions for BE.

Granted I can't guarantee this will work correctly there yet, I have to go through and make sure all BE code works.

1612 ↗

(On Diff #209840)

The instruction (VLDRH) only supports alignments >= 2. Anything more than 2 is fine though (presuming it's a power of 2).

1631 ↗

(On Diff #209840)

The idea is that so long as we are LE, all the VLDRX instructions load data into the same lanes, so any can be used. The VLDRB.u8 will load the same data into the same place as a VLDRW.u32, just with a lower alignment constraint and a lower immediate range.

In BE though, they will reverse the values as they are loaded into the lanes, so the types do actually become important.

[SjoerdMeijer]

Looks reasonable

llvm/include/llvm/Target/TargetSelectionDAG.td

1114 ↗

(On Diff #206735)

same nit if I'm not mistaken

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

1610 ↗

(On Diff #209840)

Okay, got it. Given you're still working on BE, this looks okay.

1612 ↗

(On Diff #209840)

I was actually thinking about that, can we assume it's a power of 2?

1631 ↗

(On Diff #209840)

okay, got it, cheers.

[dmgreen]

llvm/include/llvm/Target/TargetSelectionDAG.td

1114 ↗

(On Diff #206735)

Sorry, I did mean to get to this! I need to rebase this over the bigendian code too. Let me do that now.

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

1612 ↗

(On Diff #209840)

I believe alignments are always powers of 2. They can be 0 in IR, but then the abi alignment will be used by this point (which is 8 for these vectors, IIRC).

[dmgreen]

Formatting and rebased over D65580 and D65583.

llvm/trunk/include/llvm/Target/TargetSelectionDAG.td

def pre_truncsti32 : PatFrag<(ops node:$val, node:$base, node:$offset),
   let IsStore = 1;
   let MemoryVT = i32;
 }
 def pre_truncstf32 : PatFrag<(ops node:$val, node:$base, node:$offset),
                              (pre_truncst node:$val, node:$base, node:$offset)> {
   let IsStore = 1;
   let MemoryVT = f32;
 }
+def pre_truncstvi8 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                             (pre_truncst node:$val, node:$base, node:$offset)> {
+  let IsStore = 1;
+  let ScalarMemoryVT = i8;
+}
+def pre_truncstvi16 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                              (pre_truncst node:$val, node:$base, node:$offset)> {
+  let IsStore = 1;
+  let ScalarMemoryVT = i16;
+}
 
 def post_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
                          (istore node:$val, node:$ptr, node:$offset), [{
   ISD::MemIndexedMode AM = cast<StoreSDNode>(N)->getAddressingMode();
   return AM == ISD::POST_INC || AM == ISD::POST_DEC;
 }]>;
 
 def post_truncst : PatFrag<(ops node:$val, node:$base, node:$offset),
def post_truncsti32 : PatFrag<(ops node:$val, node:$base, node:$offset),
   let IsStore = 1;
   let MemoryVT = i32;
 }
 def post_truncstf32 : PatFrag<(ops node:$val, node:$base, node:$offset),
                               (post_truncst node:$val, node:$base, node:$offset)> {
   let IsStore = 1;
   let MemoryVT = f32;
 }
+def post_truncstvi8 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                              (post_truncst node:$val, node:$base, node:$offset)> {
+  let IsStore = 1;
+  let ScalarMemoryVT = i8;
+}
+def post_truncstvi16 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                               (post_truncst node:$val, node:$base, node:$offset)> {
+  let IsStore = 1;
+  let ScalarMemoryVT = i16;
+}
 
 def nonvolatile_load : PatFrag<(ops node:$ptr),
                                (load node:$ptr), [{
   return !cast<LoadSDNode>(N)->isVolatile();
 }]>;
 def nonvolatile_store : PatFrag<(ops node:$val, node:$ptr),
                                 (store node:$val, node:$ptr), [{
   return !cast<StoreSDNode>(N)->isVolatile();

llvm/trunk/lib/Target/ARM/ARMISelDAGToDAG.cpp

public:
   bool SelectThumbAddrModeSP(SDValue N, SDValue &Base, SDValue &OffImm);
 
   // Thumb 2 Addressing Modes:
   bool SelectT2AddrModeImm12(SDValue N, SDValue &Base, SDValue &OffImm);
   bool SelectT2AddrModeImm8(SDValue N, SDValue &Base,
                             SDValue &OffImm);
   bool SelectT2AddrModeImm8Offset(SDNode *Op, SDValue N,
                                  SDValue &OffImm);
-  template<unsigned Shift>
-  bool SelectT2AddrModeImm7(SDValue N, SDValue &Base,
-                            SDValue &OffImm);
+  template <unsigned Shift>
+  bool SelectT2AddrModeImm7Offset(SDNode *Op, SDValue N, SDValue &OffImm);
+  bool SelectT2AddrModeImm7Offset(SDNode *Op, SDValue N, SDValue &OffImm,
+                                  unsigned Shift);
+  template <unsigned Shift>
+  bool SelectT2AddrModeImm7(SDValue N, SDValue &Base, SDValue &OffImm);
   bool SelectT2AddrModeSoReg(SDValue N, SDValue &Base,
                              SDValue &OffReg, SDValue &ShImm);
   bool SelectT2AddrModeExclusive(SDValue N, SDValue &Base, SDValue &OffImm);
 
   inline bool is_so_imm(unsigned Imm) const {
     return ARM_AM::getSOImmVal(Imm) != -1;
   }
 

 
 private:
   void transferMemOperands(SDNode *Src, SDNode *Dst);
 
   /// Indexed (pre/post inc/dec) load matching code for ARM.
   bool tryARMIndexedLoad(SDNode *N);
   bool tryT1IndexedLoad(SDNode *N);
   bool tryT2IndexedLoad(SDNode *N);
+  bool tryMVEIndexedLoad(SDNode *N);
 
   /// SelectVLD - Select NEON load intrinsics.  NumVecs should be
   /// 1, 2, 3 or 4.  The opcode arrays specify the instructions used for
   /// loads of D registers and even subregs and odd subregs of Q registers.
   /// For NumVecs <= 2, QOpcodes1 is not used.
   void SelectVLD(SDNode *N, bool isUpdating, unsigned NumVecs,
                  const uint16_t *DOpcodes, const uint16_t *QOpcodes0,
                  const uint16_t *QOpcodes1);
bool ARMDAGToDAGISel::SelectT2AddrModeImm7(SDValue N,
   }
 
   // Base only.
   Base = N;
   OffImm  = CurDAG->getTargetConstant(0, SDLoc(N), MVT::i32);
   return true;
 }
 
+template <unsigned Shift>
+bool ARMDAGToDAGISel::SelectT2AddrModeImm7Offset(SDNode *Op, SDValue N,
+                                                 SDValue &OffImm) {
+  return SelectT2AddrModeImm7Offset(Op, N, OffImm, Shift);
+}
+
+bool ARMDAGToDAGISel::SelectT2AddrModeImm7Offset(SDNode *Op, SDValue N,
+                                                 SDValue &OffImm,
+                                                 unsigned Shift) {
+  unsigned Opcode = Op->getOpcode();
+  ISD::MemIndexedMode AM = (Opcode == ISD::LOAD)
+                               ? cast<LoadSDNode>(Op)->getAddressingMode()
+                               : cast<StoreSDNode>(Op)->getAddressingMode();
+  int RHSC;
+  if (isScaledConstantInRange(N, 1 << Shift, 0, 0x80, RHSC)) { // 7 bits.
+    OffImm =
+        ((AM == ISD::PRE_INC) || (AM == ISD::POST_INC))
+            ? CurDAG->getTargetConstant(RHSC * (1 << Shift), SDLoc(N), MVT::i32)
+            : CurDAG->getTargetConstant(-RHSC * (1 << Shift), SDLoc(N),
+                                        MVT::i32);
+    return true;
+  }
+  return false;
+}
+
 bool ARMDAGToDAGISel::SelectT2AddrModeSoReg(SDValue N,
                                             SDValue &Base,
                                             SDValue &OffReg, SDValue &ShImm) {
   // (R - imm8) should be handled by t2LDRi8. The rest are handled by t2LDRi12.
   if (N.getOpcode() != ISD::ADD && !CurDAG->isBaseWithConstantOffset(N))
     return false;
 
   // Leave (R + imm12) for t2LDRi12, (R - imm8) for t2LDRi8.
if (Match) {
     transferMemOperands(N, New);
     ReplaceNode(N, New);
     return true;
   }
 
   return false;
 }
 
+bool ARMDAGToDAGISel::tryMVEIndexedLoad(SDNode *N) {
+  LoadSDNode *LD = cast<LoadSDNode>(N);
+  ISD::MemIndexedMode AM = LD->getAddressingMode();
+  if (AM == ISD::UNINDEXED)
+    return false;
+  EVT LoadedVT = LD->getMemoryVT();
+  if (!LoadedVT.isVector())
+    return false;
+  bool isSExtLd = LD->getExtensionType() == ISD::SEXTLOAD;
+  SDValue Offset;
+  bool isPre = (AM == ISD::PRE_INC) || (AM == ISD::PRE_DEC);
+  unsigned Opcode = 0;
+  unsigned Align = LD->getAlignment();
+  bool IsLE = Subtarget->isLittle();
+
+  if (Align >= 2 && LoadedVT == MVT::v4i16 &&
+      SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 1)) {
+    if (isSExtLd)
+      Opcode = isPre ? ARM::MVE_VLDRHS32_pre : ARM::MVE_VLDRHS32_post;
+    else
+      Opcode = isPre ? ARM::MVE_VLDRHU32_pre : ARM::MVE_VLDRHU32_post;
+  } else if (LoadedVT == MVT::v8i8 &&
+             SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 0)) {
+    if (isSExtLd)
+      Opcode = isPre ? ARM::MVE_VLDRBS16_pre : ARM::MVE_VLDRBS16_post;
+    else
+      Opcode = isPre ? ARM::MVE_VLDRBU16_pre : ARM::MVE_VLDRBU16_post;
+  } else if (LoadedVT == MVT::v4i8 &&
+             SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 0)) {
+    if (isSExtLd)
+      Opcode = isPre ? ARM::MVE_VLDRBS32_pre : ARM::MVE_VLDRBS32_post;
+    else
+      Opcode = isPre ? ARM::MVE_VLDRBU32_pre : ARM::MVE_VLDRBU32_post;
+  } else if (Align >= 4 &&
+             (IsLE || LoadedVT == MVT::v4i32 || LoadedVT == MVT::v4f32) &&
+             SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 2))
+    Opcode = isPre ? ARM::MVE_VLDRWU32_pre : ARM::MVE_VLDRWU32_post;
+  else if (Align >= 2 &&
+           (IsLE || LoadedVT == MVT::v8i16 || LoadedVT == MVT::v8f16) &&
+           SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 1))
+    Opcode = isPre ? ARM::MVE_VLDRHU16_pre : ARM::MVE_VLDRHU16_post;
+  else if ((IsLE || LoadedVT == MVT::v16i8) &&
+           SelectT2AddrModeImm7Offset(N, LD->getOffset(), Offset, 0))
+    Opcode = isPre ? ARM::MVE_VLDRBU8_pre : ARM::MVE_VLDRBU8_post;
+  else
+    return false;
+
+  SDValue Chain = LD->getChain();
+  SDValue Base = LD->getBasePtr();
+  SDValue Ops[] = {Base, Offset,
+                   CurDAG->getTargetConstant(ARMVCC::None, SDLoc(N), MVT::i32),
+                   CurDAG->getRegister(0, MVT::i32), Chain};
+  SDNode *New = CurDAG->getMachineNode(Opcode, SDLoc(N), LD->getValueType(0),
+                                       MVT::i32, MVT::Other, Ops);
+  transferMemOperands(N, New);
+  ReplaceUses(SDValue(N, 0), SDValue(New, 1));
+  ReplaceUses(SDValue(N, 1), SDValue(New, 0));
+  ReplaceUses(SDValue(N, 2), SDValue(New, 2));
+  CurDAG->RemoveDeadNode(N);
+  return true;
+}
+
 /// Form a GPRPair pseudo register from a pair of GPR regs.
 SDNode *ARMDAGToDAGISel::createGPRPairNode(EVT VT, SDValue V0, SDValue V1) {
   SDLoc dl(V0.getNode());
   SDValue RegClass =
     CurDAG->getTargetConstant(ARM::GPRPairRegClassID, dl, MVT::i32);
   SDValue SubReg0 = CurDAG->getTargetConstant(ARM::gsub_0, dl, MVT::i32);
   SDValue SubReg1 = CurDAG->getTargetConstant(ARM::gsub_1, dl, MVT::i32);
   const SDValue Ops[] = { RegClass, V0, SubReg0, V1, SubReg1 };
case ARMISD::SUBE: {
     unsigned Opc = Subtarget->isThumb2() ? ARM::t2SMMLS : ARM::SMMLS;
     SDValue Ops[] = { SmulLoHi.getOperand(0), SmulLoHi.getOperand(1),
                       N->getOperand(0), getAL(CurDAG, dl),
                       CurDAG->getRegister(0, MVT::i32) };
     ReplaceNode(N, CurDAG->getMachineNode(Opc, dl, MVT::i32, Ops));
     return;
   }
   case ISD::LOAD: {
+    if (Subtarget->hasMVEIntegerOps() && tryMVEIndexedLoad(N))
+      return;
     if (Subtarget->isThumb() && Subtarget->hasThumb2()) {
       if (tryT2IndexedLoad(N))
         return;
     } else if (Subtarget->isThumb()) {
       if (tryT1IndexedLoad(N))
         return;
     } else if (tryARMIndexedLoad(N))
       return;

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

for (auto VT : IntTypes) {
     setOperationAction(ISD::CTPOP, VT, Expand);
 
     if (!HasMVEFP) {
       setOperationAction(ISD::SINT_TO_FP, VT, Expand);
       setOperationAction(ISD::UINT_TO_FP, VT, Expand);
       setOperationAction(ISD::FP_TO_SINT, VT, Expand);
       setOperationAction(ISD::FP_TO_UINT, VT, Expand);
     }
+
+    // Pre and Post inc are supported on loads and stores
+    for (unsigned im = (unsigned)ISD::PRE_INC;
+         im != (unsigned)ISD::LAST_INDEXED_MODE; ++im) {
+      setIndexedLoadAction(im, VT, Legal);
+      setIndexedStoreAction(im, VT, Legal);
+    }
   }
 
   const MVT FloatTypes[] = { MVT::v8f16, MVT::v4f32 };
   for (auto VT : FloatTypes) {
     addRegisterClass(VT, &ARM::QPRRegClass);
     if (!HasMVEFP)
       setAllExpand(VT);
 
     // These are legal or custom whether we have MVE.fp or not
     setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT, VT.getVectorElementType(), Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
     setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
     setOperationAction(ISD::BUILD_VECTOR, VT.getVectorElementType(), Custom);
     setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Legal);
     setOperationAction(ISD::SETCC, VT, Custom);
 
+    // Pre and Post inc are supported on loads and stores
+    for (unsigned im = (unsigned)ISD::PRE_INC;
+         im != (unsigned)ISD::LAST_INDEXED_MODE; ++im) {
+      setIndexedLoadAction(im, VT, Legal);
+      setIndexedStoreAction(im, VT, Legal);
+    }
+
     if (HasMVEFP) {
       setOperationAction(ISD::FMINNUM, VT, Legal);
       setOperationAction(ISD::FMAXNUM, VT, Legal);
       setOperationAction(ISD::FROUND, VT, Legal);
 
       // No native support for these.
       setOperationAction(ISD::FDIV, VT, Expand);
       setOperationAction(ISD::FREM, VT, Expand);
void ARMTargetLowering::addMVEVectorTypes(bool HasMVEFP) {
   addAllExtLoads(MVT::v4i32, MVT::v4i16, Legal);
   addAllExtLoads(MVT::v4i32, MVT::v4i8, Legal);
 
   // Some truncating stores are legal too.
   setTruncStoreAction(MVT::v4i32, MVT::v4i16, Legal);
   setTruncStoreAction(MVT::v4i32, MVT::v4i8,  Legal);
   setTruncStoreAction(MVT::v8i16, MVT::v8i8,  Legal);
 
+  // Pre and Post inc on these are legal, given the correct extends
+  for (unsigned im = (unsigned)ISD::PRE_INC;
+       im != (unsigned)ISD::LAST_INDEXED_MODE; ++im) {
+    setIndexedLoadAction(im, MVT::v8i8, Legal);
+    setIndexedStoreAction(im, MVT::v8i8, Legal);
+    setIndexedLoadAction(im, MVT::v4i8, Legal);
+    setIndexedStoreAction(im, MVT::v4i8, Legal);
+    setIndexedLoadAction(im, MVT::v4i16, Legal);
+    setIndexedStoreAction(im, MVT::v4i16, Legal);
+  }
+
   // Predicate types
   const MVT pTypes[] = {MVT::v16i1, MVT::v8i1, MVT::v4i1};
   for (auto VT : pTypes) {
     addRegisterClass(VT, &ARM::VCCRRegClass);
     setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);
     setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
     setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
if (RHSC < 0 && RHSC > -0x100) { // 8 bits.
       Offset = DAG.getConstant(RHSC, SDLoc(Ptr), RHS->getValueType(0));
       return true;
     }
   }
 
   return false;
 }
 
+static bool getMVEIndexedAddressParts(SDNode *Ptr, EVT VT, unsigned Align,
+                                      bool isSEXTLoad, bool isLE, SDValue &Base,
+                                      SDValue &Offset, bool &isInc,
+                                      SelectionDAG &DAG) {
+  if (Ptr->getOpcode() != ISD::ADD && Ptr->getOpcode() != ISD::SUB)
+    return false;
+  if (!isa<ConstantSDNode>(Ptr->getOperand(1)))
+    return false;
+
+  ConstantSDNode *RHS = cast<ConstantSDNode>(Ptr->getOperand(1));
+  int RHSC = (int)RHS->getZExtValue();
+
+  auto IsInRange = [&](int RHSC, int Limit, int Scale) {
+    if (RHSC < 0 && RHSC > -Limit * Scale && RHSC % Scale == 0) {
+      assert(Ptr->getOpcode() == ISD::ADD);
+      isInc = false;
+      Offset = DAG.getConstant(-RHSC, SDLoc(Ptr), RHS->getValueType(0));
+      return true;
+    } else if (RHSC > 0 && RHSC < Limit * Scale && RHSC % Scale == 0) {
+      isInc = Ptr->getOpcode() == ISD::ADD;
+      Offset = DAG.getConstant(RHSC, SDLoc(Ptr), RHS->getValueType(0));
+      return true;
+    }
+    return false;
+  };
+
+  // Try to find a matching instruction based on s/zext, Alignment, Offset and
+  // (in BE) type.
+  Base = Ptr->getOperand(0);
+  if (VT == MVT::v4i16) {
+    if (Align >= 2 && IsInRange(RHSC, 0x80, 2))
+      return true;
+  } else if (VT == MVT::v4i8 || VT == MVT::v8i8) {
+    if (IsInRange(RHSC, 0x80, 1))
+      return true;
+  } else if (Align >= 4 && (isLE || VT == MVT::v4i32 || VT == MVT::v4f32) &&
+             IsInRange(RHSC, 0x80, 4))
+    return true;
+  else if (Align >= 2 && (isLE || VT == MVT::v8i16 || VT == MVT::v8f16) &&
+           IsInRange(RHSC, 0x80, 2))
+    return true;
+  else if ((isLE || VT == MVT::v16i8) && IsInRange(RHSC, 0x80, 1))
+    return true;
+  return false;
+}
+
 /// getPreIndexedAddressParts - returns true by value, base pointer and
 /// offset pointer and addressing mode by reference if the node's address
 /// can be legally represented as pre-indexed load / store address.
 bool
 ARMTargetLowering::getPreIndexedAddressParts(SDNode *N, SDValue &Base,
                                              SDValue &Offset,
                                              ISD::MemIndexedMode &AM,
                                              SelectionDAG &DAG) const {
   if (Subtarget->isThumb1Only())
     return false;
 
   EVT VT;
   SDValue Ptr;
+  unsigned Align;
   bool isSEXTLoad = false;
   if (LoadSDNode *LD = dyn_cast<LoadSDNode>(N)) {
     Ptr = LD->getBasePtr();
-    VT  = LD->getMemoryVT();
+    VT = LD->getMemoryVT();
+    Align = LD->getAlignment();
     isSEXTLoad = LD->getExtensionType() == ISD::SEXTLOAD;
   } else if (StoreSDNode *ST = dyn_cast<StoreSDNode>(N)) {
     Ptr = ST->getBasePtr();
-    VT  = ST->getMemoryVT();
+    VT = ST->getMemoryVT();
+    Align = ST->getAlignment();
   } else
     return false;
 
   bool isInc;
   bool isLegal = false;
+  if (VT.isVector())
+    isLegal = Subtarget->hasMVEIntegerOps() &&
+              getMVEIndexedAddressParts(Ptr.getNode(), VT, Align, isSEXTLoad,
+                                        Subtarget->isLittle(), Base, Offset,
+                                        isInc, DAG);
+  else {
     if (Subtarget->isThumb2())
       isLegal = getT2IndexedAddressParts(Ptr.getNode(), VT, isSEXTLoad, Base,
                                          Offset, isInc, DAG);
     else
       isLegal = getARMIndexedAddressParts(Ptr.getNode(), VT, isSEXTLoad, Base,
                                           Offset, isInc, DAG);
+  }
   if (!isLegal)
     return false;
 
   AM = isInc ? ISD::PRE_INC : ISD::PRE_DEC;
   return true;
 }
 
 /// getPostIndexedAddressParts - returns true by value, base pointer and
 /// offset pointer and addressing mode by reference if this node can be
 /// combined with a load / store to form a post-indexed load / store.
 bool ARMTargetLowering::getPostIndexedAddressParts(SDNode *N, SDNode *Op,
                                                    SDValue &Base,
                                                    SDValue &Offset,
                                                    ISD::MemIndexedMode &AM,
                                                    SelectionDAG &DAG) const {
   EVT VT;
   SDValue Ptr;
+  unsigned Align;
   bool isSEXTLoad = false, isNonExt;
   if (LoadSDNode *LD = dyn_cast<LoadSDNode>(N)) {
-    VT  = LD->getMemoryVT();
+    VT = LD->getMemoryVT();
     Ptr = LD->getBasePtr();
+    Align = LD->getAlignment();
     isSEXTLoad = LD->getExtensionType() == ISD::SEXTLOAD;
     isNonExt = LD->getExtensionType() == ISD::NON_EXTLOAD;
   } else if (StoreSDNode *ST = dyn_cast<StoreSDNode>(N)) {
-    VT  = ST->getMemoryVT();
+    VT = ST->getMemoryVT();
     Ptr = ST->getBasePtr();
+    Align = ST->getAlignment();
     isNonExt = !ST->isTruncatingStore();
   } else
     return false;
 
   if (Subtarget->isThumb1Only()) {
     // Thumb-1 can do a limited post-inc load or store as an updating LDM. It
     // must be non-extending/truncating, i32, with an offset of 4.
     assert(Op->getValueType(0) == MVT::i32 && "Non-i32 post-inc op?!");
     if (Op->getOpcode() != ISD::ADD || !isNonExt)
       return false;
     auto *RHS = dyn_cast<ConstantSDNode>(Op->getOperand(1));
     if (!RHS || RHS->getZExtValue() != 4)
       return false;
 
     Offset = Op->getOperand(1);
     Base = Op->getOperand(0);
     AM = ISD::POST_INC;
     return true;
   }
 
   bool isInc;
   bool isLegal = false;
+  if (VT.isVector())
+    isLegal = Subtarget->hasMVEIntegerOps() &&
+              getMVEIndexedAddressParts(Op, VT, Align, isSEXTLoad,
+                                        Subtarget->isLittle(), Base, Offset,
+                                        isInc, DAG);
+  else {
     if (Subtarget->isThumb2())
       isLegal = getT2IndexedAddressParts(Op, VT, isSEXTLoad, Base, Offset,
                                          isInc, DAG);
     else
       isLegal = getARMIndexedAddressParts(Op, VT, isSEXTLoad, Base, Offset,
                                           isInc, DAG);
+  }
   if (!isLegal)
     return false;
 
   if (Ptr != Base) {
     // Swap base ptr and offset to catch more post-index load / store when
     // it's legal. In Thumb2 mode, offset must be an immediate.
     if (Ptr == Offset && Op->getOpcode() == ISD::ADD &&
         !Subtarget->isThumb2())

llvm/trunk/lib/Target/ARM/ARMInstrMVE.td

 }
 
 class t2am_imm7shiftOffsetAsmOperand<int shift>
   : AsmOperandClass { let Name = "Imm7Shift"#shift; }
 def t2am_imm7shift0OffsetAsmOperand : t2am_imm7shiftOffsetAsmOperand<0>;
 def t2am_imm7shift1OffsetAsmOperand : t2am_imm7shiftOffsetAsmOperand<1>;
 def t2am_imm7shift2OffsetAsmOperand : t2am_imm7shiftOffsetAsmOperand<2>;
 
-class t2am_imm7_offset<int shift> : MemOperand {
+class t2am_imm7_offset<int shift> : MemOperand,
+      ComplexPattern<i32, 1, "SelectT2AddrModeImm7Offset<"#shift#">",
+                     [], [SDNPWantRoot]> {
   // They are printed the same way as the imm8 version
   let PrintMethod = "printT2AddrModeImm8OffsetOperand";
   let ParserMatchClass =
     !cast<AsmOperandClass>("t2am_imm7shift"#shift#"OffsetAsmOperand");
   let EncoderMethod = "getT2ScaledImmOpValue<7,"#shift#">";
   let DecoderMethod = "DecodeT2Imm7<"#shift#">";
 }
 
multiclass MVE_unpred_vector_load<Instruction RegImmInst, PatFrag LoadKind,
   def : MVE_unpred_vector_load_typed<v8i16, RegImmInst, LoadKind, shift>;
   def : MVE_unpred_vector_load_typed<v8f16, RegImmInst, LoadKind, shift>;
   def : MVE_unpred_vector_load_typed<v4i32, RegImmInst, LoadKind, shift>;
   def : MVE_unpred_vector_load_typed<v4f32, RegImmInst, LoadKind, shift>;
   def : MVE_unpred_vector_load_typed<v2i64, RegImmInst, LoadKind, shift>;
   def : MVE_unpred_vector_load_typed<v2f64, RegImmInst, LoadKind, shift>;
 }
 
+class MVE_unpred_vector_offset_store_typed<ValueType Ty, Instruction Opcode,
+                                           PatFrag StoreKind, int shift>
+      : Pat<(StoreKind (Ty MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<shift>:$addr),
+            (Opcode MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<shift>:$addr)>;
+
+multiclass MVE_unpred_vector_offset_store<Instruction RegImmInst, PatFrag StoreKind,
+                                          int shift> {
+  def : MVE_unpred_vector_offset_store_typed<v16i8, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v8i16, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v8f16, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v4i32, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v4f32, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v2i64, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_offset_store_typed<v2f64, RegImmInst, StoreKind, shift>;
+}
+def aligned32_pre_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
+                                  (pre_store node:$val, node:$ptr, node:$offset), [{
+  return cast<StoreSDNode>(N)->getAlignment() >= 4;
+}]>;
+def aligned32_post_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
+                                  (post_store node:$val, node:$ptr, node:$offset), [{
+  return cast<StoreSDNode>(N)->getAlignment() >= 4;
+}]>;
+def aligned16_pre_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
+                                  (pre_store node:$val, node:$ptr, node:$offset), [{
+  return cast<StoreSDNode>(N)->getAlignment() == 2;
+}]>;
+def aligned16_post_store : PatFrag<(ops node:$val, node:$ptr, node:$offset),
+                                  (post_store node:$val, node:$ptr, node:$offset), [{
+  return cast<StoreSDNode>(N)->getAlignment() == 2;
+}]>;
+
 let Predicates = [HasMVEInt, IsLE] in {
   defm : MVE_unpred_vector_store<MVE_VSTRBU8, byte_alignedstore, 0>;
   defm : MVE_unpred_vector_store<MVE_VSTRHU16, hword_alignedstore, 1>;
   defm : MVE_unpred_vector_store<MVE_VSTRWU32, alignedstore32, 2>;
 
   defm : MVE_unpred_vector_load<MVE_VLDRBU8, byte_alignedload, 0>;
   defm : MVE_unpred_vector_load<MVE_VLDRHU16, hword_alignedload, 1>;
   defm : MVE_unpred_vector_load<MVE_VLDRWU32, alignedload32, 2>;
+
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRBU8_pre, pre_store, 0>;
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRBU8_post, post_store, 0>;
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRHU16_pre, aligned16_pre_store, 1>;
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRHU16_post, aligned16_post_store, 1>;
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRWU32_pre, aligned32_pre_store, 2>;
+  defm : MVE_unpred_vector_offset_store<MVE_VSTRWU32_post, aligned32_post_store, 2>;
 }
 
 let Predicates = [HasMVEInt, IsBE] in {
   def : MVE_unpred_vector_store_typed<v16i8, MVE_VSTRBU8, store, 0>;
   def : MVE_unpred_vector_store_typed<v8i16, MVE_VSTRHU16, alignedstore16, 1>;
   def : MVE_unpred_vector_store_typed<v8f16, MVE_VSTRHU16, alignedstore16, 1>;
   def : MVE_unpred_vector_store_typed<v4i32, MVE_VSTRWU32, alignedstore32, 2>;
   def : MVE_unpred_vector_store_typed<v4f32, MVE_VSTRWU32, alignedstore32, 2>;
let Predicates = [HasMVEInt, IsBE] in {
   def : Pat<(store (v4i32 MQPR:$val), t2addrmode_imm7<0>:$addr),
             (MVE_VSTRBU8 (MVE_VREV32_8 MQPR:$val), t2addrmode_imm7<0>:$addr)>;
   def : Pat<(store (v4f32 MQPR:$val), t2addrmode_imm7<0>:$addr),
             (MVE_VSTRBU8 (MVE_VREV32_8 MQPR:$val), t2addrmode_imm7<0>:$addr)>;
   def : Pat<(store (v8i16 MQPR:$val), t2addrmode_imm7<0>:$addr),
             (MVE_VSTRBU8 (MVE_VREV16_8 MQPR:$val), t2addrmode_imm7<0>:$addr)>;
   def : Pat<(store (v8f16 MQPR:$val), t2addrmode_imm7<0>:$addr),
             (MVE_VSTRBU8 (MVE_VREV16_8 MQPR:$val), t2addrmode_imm7<0>:$addr)>;
+
+  def : MVE_unpred_vector_offset_store_typed<v16i8, MVE_VSTRBU8_pre, pre_store, 0>;
+  def : MVE_unpred_vector_offset_store_typed<v16i8, MVE_VSTRBU8_post, post_store, 0>;
+  def : MVE_unpred_vector_offset_store_typed<v8i16, MVE_VSTRHU16_pre, aligned16_pre_store, 1>;
+  def : MVE_unpred_vector_offset_store_typed<v8i16, MVE_VSTRHU16_post, aligned16_post_store, 1>;
+  def : MVE_unpred_vector_offset_store_typed<v8f16, MVE_VSTRHU16_pre, aligned16_pre_store, 1>;
+  def : MVE_unpred_vector_offset_store_typed<v8f16, MVE_VSTRHU16_post, aligned16_post_store, 1>;
+  def : MVE_unpred_vector_offset_store_typed<v4i32, MVE_VSTRWU32_pre, aligned32_pre_store, 2>;
+  def : MVE_unpred_vector_offset_store_typed<v4i32, MVE_VSTRWU32_post, aligned32_post_store, 2>;
+  def : MVE_unpred_vector_offset_store_typed<v4f32, MVE_VSTRWU32_pre, aligned32_pre_store, 2>;
+  def : MVE_unpred_vector_offset_store_typed<v4f32, MVE_VSTRWU32_post, aligned32_post_store, 2>;
 }
 
 let Predicates = [HasMVEInt] in {
   def  : Pat<(v16i1 (load t2addrmode_imm7<2>:$addr)),
              (v16i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
   def  : Pat<(v8i1 (load t2addrmode_imm7<2>:$addr)),
              (v8i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
   def  : Pat<(v4i1 (load t2addrmode_imm7<2>:$addr)),
              (v4i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
 }
 
 
 // Widening/Narrowing Loads/Stores
 
 let MinAlignment = 2 in {
   def truncstorevi16_align2 : PatFrag<(ops node:$val, node:$ptr),
                                       (truncstorevi16 node:$val, node:$ptr)>;
+  def post_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                                        (post_truncstvi16 node:$val, node:$base, node:$offset)>;
+  def pre_truncstvi16_align2 : PatFrag<(ops node:$val, node:$base, node:$offset),
+                                       (pre_truncstvi16 node:$val, node:$base, node:$offset)>;
 }
 
 let Predicates = [HasMVEInt] in {
   def : Pat<(truncstorevi8  (v8i16 MQPR:$val), t2addrmode_imm7<0>:$addr),
              (MVE_VSTRB16 MQPR:$val, t2addrmode_imm7<0>:$addr)>;
   def : Pat<(truncstorevi8  (v4i32 MQPR:$val), t2addrmode_imm7<0>:$addr),
              (MVE_VSTRB32 MQPR:$val, t2addrmode_imm7<0>:$addr)>;
   def : Pat<(truncstorevi16_align2 (v4i32 MQPR:$val), t2addrmode_imm7<1>:$addr),
              (MVE_VSTRH32 MQPR:$val, t2addrmode_imm7<1>:$addr)>;
+
+  def : Pat<(post_truncstvi8 (v8i16 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<0>:$addr),
+          (MVE_VSTRB16_post MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<0>:$addr)>;
+  def : Pat<(post_truncstvi8 (v4i32 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<0>:$addr),
+          (MVE_VSTRB32_post MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<0>:$addr)>;
+  def : Pat<(post_truncstvi16_align2 (v4i32 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<1>:$addr),
+          (MVE_VSTRH32_post MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<1>:$addr)>;
+
+  def : Pat<(pre_truncstvi8 (v8i16 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<0>:$addr),
+          (MVE_VSTRB16_pre MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<0>:$addr)>;
+  def : Pat<(pre_truncstvi8 (v4i32 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<0>:$addr),
+          (MVE_VSTRB32_pre MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<0>:$addr)>;
+  def : Pat<(pre_truncstvi16_align2 (v4i32 MQPR:$Rt), tGPR:$Rn, t2am_imm7_offset<1>:$addr),
+          (MVE_VSTRH32_pre MQPR:$Rt, tGPR:$Rn, t2am_imm7_offset<1>:$addr)>;
 }
 
 
 let MinAlignment = 2 in {
   def extloadvi16_align2  : PatFrag<(ops node:$ptr), (extloadvi16 node:$ptr)>;
   def sextloadvi16_align2 : PatFrag<(ops node:$ptr), (sextloadvi16 node:$ptr)>;
   def zextloadvi16_align2 : PatFrag<(ops node:$ptr), (zextloadvi16 node:$ptr)>;
 }

llvm/trunk/test/CodeGen/Thumb2/mve-ldst-postinc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 define i8* @ldrwu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_m4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_m4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #-4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 -4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_508(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    add.w r0, r0, #508
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #508
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
entry:
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_m508(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_m508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    sub.w r0, r0, #508
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #-508
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 -508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
entry:
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhu32_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.u32 q0, [r0], #2
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhu32_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #254
+; CHECK-NEXT:    vldrh.u32 q0, [r0], #254
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhs32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.s32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhs32_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.s32 q0, [r0], #2
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhs32_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #254
+; CHECK-NEXT:    vldrh.s32 q0, [r0], #254
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhu16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #4
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #2
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #254
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #254
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
entry:
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbu32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u32 q0, [r0], #3
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbu32_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vldrb.u32 q0, [r0], #127
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrbs32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.s32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbs32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.s32 q0, [r0], #3
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbs32_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vldrb.s32 q0, [r0], #127
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrbu16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u16 q0, [r0], #4
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbu16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u16 q0, [r0], #3
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbu16_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vldrb.u16 q0, [r0], #127
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
entry:
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbs16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.s16 q0, [r0], #4
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbs16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s16 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.s16 q0, [r0], #3
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbs16_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s16 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vldrb.s16 q0, [r0], #127
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
entry:
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbu8_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #4
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrbu8_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrbu8_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #127
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
entry:
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrwf32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwf32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #4
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwf16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwf16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #4
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrwi32_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwi32_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 1
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrhi16_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhi16_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 1
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
entry:
   %3 = sext <4 x i16> %1 to <4 x i32>
   store <4 x i32> %3, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrf32_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrf32_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 1
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrf16_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrf16_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0], #3
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 1
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 
 
 
 
 define i8* @strw32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_m4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_m4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #-4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 -4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_508(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_508:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    add.w r0, r0, #508
+; CHECK-NEXT:    vstrw.32 q0, [r0], #508
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
entry:
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_m508(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_m508:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    sub.w r0, r0, #508
+; CHECK-NEXT:    vstrw.32 q0, [r0], #-508
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 -508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strh32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrh.32 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %y to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
entry:
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh32_2(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_2:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vstrh.32 q0, [r0], #2
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 2
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %y to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh32_254(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_254:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #254
+; CHECK-NEXT:    vstrh.32 q0, [r0], #254
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 254
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %y to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strh16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_2(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_2:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vstrb.8 q0, [r0], #2
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 2
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_254(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_254:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #254
+; CHECK-NEXT:    vstrh.16 q0, [r0], #254
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 254
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.32 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %y to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb32_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.32 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %y to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb32_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_127:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vstrb.32 q0, [r0], #127
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %y to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.16 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %y to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb16_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.16 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %y to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb16_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_127:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vstrb.16 q0, [r0], #127
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %y to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb8_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb8_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb8_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_127:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #127
+; CHECK-NEXT:    vstrb.8 q0, [r0], #127
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
entry:
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strf16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0], #4
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strwi32_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strwi32_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strhi16_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strhi16_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 1
   ret i8* %z
entry:
   store <4 x i16> %3, <4 x i16>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf32_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf32_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf16_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf16_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0], #3
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 1
   ret i8* %z
 }

llvm/trunk/test/CodeGen/Thumb2/mve-ldst-preinc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 define i8* @ldrwu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #3
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_m4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_m4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0, #-4]
-; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0, #-4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 -4
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_508(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    add.w r0, r0, #508
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, #508]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 508
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
entry:
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwu32_m508(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwu32_m508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    sub.w r0, r0, #508
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, #-508]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 -508
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
entry:
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhu32_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r0, #2]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.u32 q0, [r0, #2]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhu32_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu32_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #254
-; CHECK-NEXT:    vldrh.u32 q0, [r0]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, #254]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = zext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhs32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.s32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhs32_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r0, #2]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.s32 q0, [r0, #2]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrhs32_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhs32_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #254
-; CHECK-NEXT:    vldrh.s32 q0, [r0]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, #254]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %z to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = sext <4 x i16> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrhu16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u16 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #3
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_2(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0, #2]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vldrh.u16 q0, [r0, #2]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 2
   %0 = bitcast i8* %z to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrhu16_254(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhu16_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #254
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, #254]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 254
   %0 = bitcast i8* %z to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
entry:
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbu32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbu32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u32 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbu32_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu32_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
-; CHECK-NEXT:    vldrb.u32 q0, [r0]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, #127]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = zext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrbs32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.s32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbs32_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.s32 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 define i8* @ldrbs32_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs32_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
-; CHECK-NEXT:    vldrb.s32 q0, [r0]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, #127]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %z to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = sext <4 x i8> %1 to <4 x i32>
   %3 = bitcast i8* %y to <4 x i32>*
entry:
   store <4 x i32> %2, <4 x i32>* %3, align 4
   ret i8* %z
 }
 
 
 define i8* @ldrbu16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u16 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbu16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u16 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u16 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbu16_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu16_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
-; CHECK-NEXT:    vldrb.u16 q0, [r0]
+; CHECK-NEXT:    vldrb.u16 q0, [r0, #127]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = zext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
entry:
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbs16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.s16 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbs16_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s16 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.s16 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 define i8* @ldrbs16_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbs16_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
-; CHECK-NEXT:    vldrb.s16 q0, [r0]
+; CHECK-NEXT:    vldrb.s16 q0, [r0, #127]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %z to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = sext <8 x i8> %1 to <8 x i16>
   %3 = bitcast i8* %y to <8 x i16>*
entry:
   store <8 x i16> %2, <8 x i16>* %3, align 2
   ret i8* %z
 }
 
 
 define i8* @ldrbu8_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrbu8_3(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrbu8_127(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrbu8_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
-; CHECK-NEXT:    vldrb.u8 q0, [r0]
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #127]!
 ; CHECK-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 127
   %0 = bitcast i8* %z to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
entry:
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @ldrwf32_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwf32_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrw.u32 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrwf16_4(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwf16_4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vldrh.u16 q0, [r0, #4]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 define i8* @ldrwi32_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrwi32_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 1
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrhi16_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrhi16_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 1
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
entry:
   %3 = sext <4 x i16> %1 to <4 x i32>
   store <4 x i32> %3, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrf32_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrf32_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 1
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @ldrf16_align1(i8* %x, i8* %y) {
 ; CHECK-LABEL: ldrf16_align1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vldrb.u8 q0, [r0, #3]!
 ; CHECK-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 3
   %0 = bitcast i8* %z to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 1
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 
 
 
 
 define i8* @strw32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #3
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_m4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_m4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0, #-4]
-; CHECK-NEXT:    subs r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #-4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 -4
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_508(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    add.w r0, r0, #508
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
+; CHECK-NEXT:    vstrw.32 q0, [r0, #508]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
entry:
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strw32_m508(i8* %y, i8* %x) {
 ; CHECK-LABEL: strw32_m508:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    sub.w r0, r0, #508
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0]
+; CHECK-NEXT:    vstrw.32 q0, [r0, #-508]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 -508
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strh32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrh.32 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %z to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
entry:
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh32_2(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_2:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0, #2]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vstrh.32 q0, [r0, #2]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 2
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %z to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh32_254(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh32_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #254
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vstrh.32 q0, [r0]
+; CHECK-NEXT:    vstrh.32 q0, [r0, #254]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 254
   %0 = bitcast i8* %x to <4 x i16>*
   %1 = load <4 x i16>, <4 x i16>* %0, align 2
   %2 = bitcast i8* %z to <4 x i16>*
   store <4 x i16> %1, <4 x i16>* %2, align 2
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strh16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %z to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_3:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #3
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %z to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_2(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_2:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0, #2]
-; CHECK-NEXT:    adds r0, #2
+; CHECK-NEXT:    vstrb.8 q0, [r0, #2]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 2
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %z to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strh16_254(i8* %y, i8* %x) {
 ; CHECK-LABEL: strh16_254:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #254
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0]
+; CHECK-NEXT:    vstrh.16 q0, [r0, #254]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 254
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %z to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 2
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.32 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %z to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb32_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.32 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %z to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb32_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb32_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.32 q0, [r0]
+; CHECK-NEXT:    vstrb.32 q0, [r0, #127]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <4 x i8>*
   %1 = load <4 x i8>, <4 x i8>* %0, align 1
   %2 = bitcast i8* %z to <4 x i8>*
   store <4 x i8> %1, <4 x i8>* %2, align 1
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.16 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %z to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb16_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.16 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %z to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb16_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb16_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
 ; CHECK-NEXT:    vldrb.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.16 q0, [r0]
+; CHECK-NEXT:    vstrb.16 q0, [r0, #127]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <8 x i8>*
   %1 = load <8 x i8>, <8 x i8>* %0, align 1
   %2 = bitcast i8* %z to <8 x i8>*
   store <8 x i8> %1, <8 x i8>* %2, align 1
   ret i8* %z
entry:
   ret i8* %z
 }
 
 
 define i8* @strb8_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %z to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb8_3(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %z to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strb8_127(i8* %y, i8* %x) {
 ; CHECK-LABEL: strb8_127:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    adds r0, #127
 ; CHECK-NEXT:    vldrb.u8 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0]
+; CHECK-NEXT:    vstrb.8 q0, [r0, #127]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 127
   %0 = bitcast i8* %x to <16 x i8>*
   %1 = load <16 x i8>, <16 x i8>* %0, align 1
   %2 = bitcast i8* %z to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
entry:
   store <16 x i8> %1, <16 x i8>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf32_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf32_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrw.32 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %z to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define i8* @strf16_4(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf16_4:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrh.16 q0, [r0, #4]
-; CHECK-NEXT:    adds r0, #4
+; CHECK-NEXT:    vstrb.8 q0, [r0, #4]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 4
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %z to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret i8* %z
 }
 
 define i8* @strwi32_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strwi32_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x i32>*
   %1 = load <4 x i32>, <4 x i32>* %0, align 4
   %2 = bitcast i8* %z to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strhi16_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strhi16_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x i16>*
   %1 = load <8 x i16>, <8 x i16>* %0, align 2
   %2 = bitcast i8* %z to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 1
   ret i8* %z
entry:
   store <4 x i16> %3, <4 x i16>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf32_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf32_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <4 x float>*
   %1 = load <4 x float>, <4 x float>* %0, align 4
   %2 = bitcast i8* %z to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 1
   ret i8* %z
 }
 
 define i8* @strf16_align1(i8* %y, i8* %x) {
 ; CHECK-LABEL: strf16_align1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    vstrb.8 q0, [r0, #3]
-; CHECK-NEXT:    adds r0, #3
+; CHECK-NEXT:    vstrb.8 q0, [r0, #3]!
 ; CHECK-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %y, i32 3
   %0 = bitcast i8* %x to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast i8* %z to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 1
   ret i8* %z
 }

llvm/trunk/test/CodeGen/Thumb2/mve-ldst-regimm.ll

entry:
   %2 = bitcast i32* %arrayidx3 to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret void
 }
 
 define hidden void @fwd_float16_t(%struct.s_float16_t* noalias nocapture %v) local_unnamed_addr #0 {
 ; CHECK-LABEL: fwd_float16_t:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    vstrh.16 q0, [r0, #16]
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #16
+; CHECK-NEXT:    vstrh.16 q0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %arrayidx3 = getelementptr inbounds %struct.s_float16_t, %struct.s_float16_t* %v, i32 0, i32 1, i32 0
   %0 = bitcast %struct.s_float16_t* %v to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %2 = bitcast half* %arrayidx3 to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret void
for.end:
   %2 = bitcast i32* %arrayidx3 to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret void
 }
 
 define hidden void @bwd_float16_t(%struct.s_float16_t* noalias nocapture %v) local_unnamed_addr #0 {
 ; CHECK-LABEL: bwd_float16_t:
 ; CHECK:       @ %bb.0: @ %for.end
-; CHECK-NEXT:    vldrh.u16 q0, [r0]
-; CHECK-NEXT:    vstrh.16 q0, [r0, #-16]
+; CHECK-NEXT:    vldrh.u16 q0, [r0], #-16
+; CHECK-NEXT:    vstrh.16 q0, [r0]
 ; CHECK-NEXT:    bx lr
 for.end:
   %0 = bitcast %struct.s_float16_t* %v to <8 x half>*
   %1 = load <8 x half>, <8 x half>* %0, align 2
   %arrayidx3 = getelementptr inbounds %struct.s_float16_t, %struct.s_float16_t* %v, i32 -1, i32 1, i32 0
   %2 = bitcast half* %arrayidx3 to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 2
   ret void