[ARM] Transforming memset to Tail predicated Loop

[malharJ]

Summary

This patch converts llvm.memcpy intrinsic into Tail Predicated
Hardware loops for a target that supports the Arm M-profile
Vector Extension (MVE).

The llvm.memset intrinsic is converted to a TP loop for both
constant and non-constant input sizes (of llvm.memset).

Depends on D99723

[dmgreen]

Can you base this on top of D99723? Some of the code may be able to be shared, even if there will be differences.

llvm/lib/Target/ARM/ARMISelLowering.cpp

11088

It might be worth creating a dup in EmitTargetCodeForMemset and having this use the vector value it produced.

[malharJ]

Moved vdup creation to inside EmitTargetCodeForMemset()

[tschuett]

For comparison:
https://github.com/llvm/llvm-project/blob/main/libc/src/string/memory_utils/memset_utils.h

[malharJ]

• Combined test files of this patch and parent patch (D99723)
• Corrected some minor formatting errors

[malharJ]

llvm/test/CodeGen/Thumb2/mve-phireg.ll

213

@dmgreen ,

This seems to be happening because of a 16-byte spill ( line 169 ).

And it's being generated after I updated the code to create the VDUP in
EmitTargetCodeForMemset() instead of during the MIR level transform.

I'm not sure why the spill is happening (not really familiar with RA), but do you
think it's worth investigating ? It doesnt seem to happen in the simpler tests.

[dmgreen]

llvm/lib/Target/ARM/ARMISelLowering.cpp

11078

This and genTPLoopBodyMemcpy are very similar. Is it possible to combine them more?

11079

Formatting.

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

21

One option for both is probably fine.

265

OptSize?

267

It's best to create a shuffle vector or build vector, not a ARMISD::VDUP directly. That may optimize better in places.

Is the input always an i8?

268

Does it need Src.getValue(0)?

llvm/test/CodeGen/Thumb2/mve-phireg.ll

213

Yeah, this test might be difficult like that, it's designed to spill a lot. It's probably fine so long as we don't see it happening in other places.

[malharJ]

Addressed review comments:

• replaced vdup node with buildSplatVector
  • looks like it gets converted to vdup.8 in most cases, apart from the memclr case where it become vmov.i32 <vec_reg> #0x0
• combined TP loop body generation functions into one
• added conditions for generating inline TP loop for memset, similar to memcpy case
• minor formatting changes

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

267

I've made this update with a build vector. Have 2 minor queries:

• Why would shuffle vector be appropriate here (given that all we want to create is a vector of constants) ?

• Even though I'm not utilising vdup directly, just to understand better, why would the input need to be i8 to generate a v16i8 vector ... can the vector not be generated by a i32 source register ?

[dmgreen]

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

132

Maybe shouldGenerateInlineTPLoop would be a more descriptive name?

144

for -> For, probably with a full stop on the previous line.

149

Make sure you clang-format the patch.

267

Do you mean "Why use a shuffle as opposed to a ARMISD::VDUP"? Normal codegen will usually start off by producing a shuffle, which will then be optimized and eventually turned into a VDUP. Because it works that way around there are not a lot of optimizations on VDUP directly, as can be seen with the constant becoming a VMOVimm. We could theoretically add them, but its added complexity and it's easier to just use the optimizations that are already present by starting from a shuffle.

The input should be able to be an i32, but the types would need to be correct.

Do we know the type of the value is always an i8?

[malharJ]

Minor formatting updates to address review comments.

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

267

Ok. Thanks for clarifying about the shuffle vector.

Do we know the type of the value is always an i8?

I have a truncate operation on line 298 which ensures the input is i8.

[dmgreen]

llvm/lib/Target/ARM/ARMISelLowering.h

296

.. for tail predicated loops.

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

267

Yep, but if the value isn't an i8 it will discard some bits it should not. Something like a @llvm.memset.p0i8.i32 or @llvm.memset.p0i32.i32, if they are valid.

Is it possible to add an assert at least?

[malharJ]

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

267

So having a look at the language ref for llvm.memset suggests that the Src value is always an i8 (It may get zero extended before reaching here but that's not a problem) ..

[malharJ]

Rebased on top of latest parent patch to get the fix for clang-format issues.
Minor formatting update.

[dmgreen]

llvm/lib/Target/ARM/ARMISelLowering.cpp

11235

Why has this changed from false to true?

What happens if CPSR is live across the MVE_MEMCPYLOOPINST? WhileLoopStart and LoopEnd both clobber that physical register, incase they get reverted to subs; bne. Do we need to add the same clobber to these new MVE mem loop instructions?

[malharJ]

Rebased patch to get latest updates to parent patch.

llvm/lib/Target/ARM/ARMISelLowering.cpp

11235

Im a bit unclear on this, so I've still left this as UpdateLiveIns=true in the latest patchset.

Inside splitAt(), I can see this relevant piece of code where:

• LiveRegs.addLiveOuts() will add the live outs of the original block.
• The for-loop will add the liveins for any physical register uses by the instructions in the newly split block.

LivePhysRegs LiveRegs;
if (UpdateLiveIns) {
    ...
    LiveRegs.addLiveOuts(*this);
    for (auto I = rbegin(), E = Prev.getReverse(); I != E; ++I)
      LiveRegs.stepBackward(*I);
}

...

  if (UpdateLiveIns)
    addLiveIns(*SplitBB, LiveRegs);

The condition uses I != E (where E is the memcpy/set pseudo) ... so I suppose any liveness information
from it is not added ? and in case it gets reverted and clobbers CPSR in a later pass, shouldn't that take care of updating the liveness information of successive blocks ?

[dmgreen]

OK. Sounds good. This looks good to me. Lets get this in, it's off by default and we can make adjustments as needed in-tree whilst getting some extra testing.

LGTM. Thanks

llvm/lib/Target/ARM/ARMISelLowering.h

 class TargetLibraryInfo;
 class TargetMachine;
 class TargetRegisterInfo;
 class VectorType;
 
   namespace ARMISD {
 
     // ARM Specific DAG Nodes
     enum NodeType : unsigned {
       // Start the numbering where the builtin ops and target ops leave off.
       FIRST_NUMBER = ISD::BUILTIN_OP_END,
 
       Wrapper,      // Wrapper - A wrapper node for TargetConstantPool,
                     // TargetExternalSymbol, and TargetGlobalAddress.
       WrapperPIC,   // WrapperPIC - A wrapper node for TargetGlobalAddress in
                     // PIC mode.
       WrapperJT,    // WrapperJT - A wrapper node for TargetJumpTable
enum NodeType : unsigned {
 
       // Pseudo vector bitwise select
       VBSP,
 
       // Pseudo-instruction representing a memory copy using ldm/stm
       // instructions.
       MEMCPY,
 
+      // Node representing a memset pseudo-instruction
+      MEMSETLOOP,
+
       // V8.1MMainline condition select
       CSINV, // Conditional select invert.
       CSNEG, // Conditional select negate.
       CSINC, // Conditional select increment.
 
       // Vector load N-element structure to all lanes:
       VLD1DUP = ISD::FIRST_TARGET_MEMORY_OPCODE,
       VLD2DUP,

llvm/lib/Target/ARM/ARMISelLowering.cpp

const char *ARMTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case ARMISD::VST4LN_UPD:    return "ARMISD::VST4LN_UPD";
   case ARMISD::WLS:           return "ARMISD::WLS";
   case ARMISD::WLSSETUP:      return "ARMISD::WLSSETUP";
   case ARMISD::LE:            return "ARMISD::LE";
   case ARMISD::LOOP_DEC:      return "ARMISD::LOOP_DEC";
   case ARMISD::CSINV:         return "ARMISD::CSINV";
   case ARMISD::CSNEG:         return "ARMISD::CSNEG";
   case ARMISD::CSINC:         return "ARMISD::CSINC";
+  case ARMISD::MEMSETLOOP:    return "ARMISD::MEMSETLOOP";
   }
   return nullptr;
 }
 
 EVT ARMTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &,
                                           EVT VT) const {
   if (!VT.isVector())
     return getPointerTy(DL);
static bool checkAndUpdateCPSRKill(MachineBasicBlock::iterator SelectItr,
   }
 
   // We found a def, or hit the end of the basic block and CPSR wasn't live
   // out. SelectMI should have a kill flag on CPSR.
   SelectItr->addRegisterKilled(ARM::CPSR, TRI);
   return true;
 }
 
+/// Adds logic in loop entry MBB to calculate loop iteration count and adds
+/// t2WhileLoopSetup and t2WhileLoopStart to generate WLS loop
+static std::pair<Register, Register> genTPEntry(MachineBasicBlock *TpEntry,
+                           MachineBasicBlock *TpLoopBody, MachineBasicBlock *TpExit,
+                           Register OpSrcReg, Register OpSizeReg,
+                           const TargetInstrInfo *TII, DebugLoc Dl,
+                           MachineRegisterInfo &MRI) {
+
+  // duplicate scalar value as a vector register
+  Register UndefReg = MRI.createVirtualRegister(&ARM::MQPRRegClass);
+  BuildMI(TpEntry, Dl, TII->get(TargetOpcode::IMPLICIT_DEF), UndefReg);
+
+  Register VdupReg = MRI.createVirtualRegister(&ARM::MQPRRegClass);
+  BuildMI(TpEntry, Dl, TII->get(ARM::MVE_VDUP8))
+      .addDef(VdupReg)
+      .addUse(OpSrcReg)
+      .addImm(ARMVCC::None)
+      .addReg(0)
+      .addReg(UndefReg);
+
+  // Calculates loop iteration count = ceil(n/16)/16 = ((n + 15)&(-16)) / 16.
+  Register AddDestReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  BuildMI(TpEntry, Dl, TII->get(ARM::t2ADDri), AddDestReg)
+      .addUse(OpSizeReg)
+      .addImm(15)
+      .add(predOps(ARMCC::AL))
+      .addReg(0);
+
+  Register BicDestReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  BuildMI(TpEntry, Dl, TII->get(ARM::t2BICri), BicDestReg)
+      .addUse(AddDestReg, RegState::Kill)
+      .addImm(16)
+      .add(predOps(ARMCC::AL))
+      .addReg(0);
+
+  Register LsrDestReg = MRI.createVirtualRegister(&ARM::GPRlrRegClass);
+  BuildMI(TpEntry, Dl, TII->get(ARM::t2LSRri), LsrDestReg)
+      .addUse(BicDestReg, RegState::Kill)
+      .addImm(4)
+      .add(predOps(ARMCC::AL))
+      .addReg(0);
+
+  Register TotalIterationsReg = MRI.createVirtualRegister(&ARM::GPRlrRegClass);
+  BuildMI(TpEntry, Dl, TII->get(ARM::t2WhileLoopSetup), TotalIterationsReg)
+      .addUse(LsrDestReg, RegState::Kill);
+
+  BuildMI(TpEntry, Dl, TII->get(ARM::t2WhileLoopStart))
+      .addUse(TotalIterationsReg)
+      .addMBB(TpExit);
+
+  return std::make_pair(TotalIterationsReg, VdupReg);
+}
+
+/// Adds logic in the loopBody MBB to generate MVE_VCTP, t2DoLoopDec and
+/// t2DoLoopEnd. These are used by later passes to generate tail predicated
+/// loops.
+static void genTPLoopBody(MachineBasicBlock *TpLoopBody,
+                          MachineBasicBlock *TpEntry, MachineBasicBlock *TpExit,
+                          const TargetInstrInfo *TII, DebugLoc Dl,
+                          MachineRegisterInfo &MRI, Register OpSrcReg,
+                          Register OpDestReg, Register ElementCountReg,
+                          Register TotalIterationsReg) {
+
+  // First insert 3 PHI nodes for: Current pointer to Dest array, loop
+  // iteration counter, predication counter Current position in the src array
+
+  // Current position in the dest array
+  Register DestPhiReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  Register CurrDestReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::PHI), DestPhiReg)
+      .addUse(OpDestReg)
+      .addMBB(TpEntry)
+      .addUse(CurrDestReg)
+      .addMBB(TpLoopBody);
+
+  // Current loop counter
+  Register LoopCounterPhiReg = MRI.createVirtualRegister(&ARM::GPRlrRegClass);
+  Register RemainingLoopIterationsReg =
+      MRI.createVirtualRegister(&ARM::GPRlrRegClass);
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::PHI), LoopCounterPhiReg)
+      .addUse(TotalIterationsReg)
+      .addMBB(TpEntry)
+      .addUse(RemainingLoopIterationsReg)
+      .addMBB(TpLoopBody);
+
+  // Predication counter
+  Register PredCounterPhiReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  Register RemainingElementsReg = MRI.createVirtualRegister(&ARM::rGPRRegClass);
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::PHI), PredCounterPhiReg)
+      .addUse(ElementCountReg)
+      .addMBB(TpEntry)
+      .addUse(RemainingElementsReg)
+      .addMBB(TpLoopBody);
+
+  // Pass predication counter to VCTP
+  Register VccrReg = MRI.createVirtualRegister(&ARM::VCCRRegClass);
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::MVE_VCTP8), VccrReg)
+      .addUse(PredCounterPhiReg)
+      .addImm(ARMVCC::None)
+      .addReg(0);
+
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::t2SUBri), RemainingElementsReg)
+      .addUse(PredCounterPhiReg)
+      .addImm(16)
+      .add(predOps(ARMCC::AL))
+      .addReg(0);
+
+  // VSTRB, predicated using VPR
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::MVE_VSTRBU8_post))
+      .addDef(CurrDestReg)
+      .addUse(OpSrcReg)
+      .addReg(DestPhiReg)
+      .addImm(16)
+      .addImm(ARMVCC::Then)
+      .addUse(VccrReg);
+
+  // Add the pseudoInstrs for decrementing the loop counter and marking the
+  // end:t2DoLoopDec and t2DoLoopEnd
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::t2LoopDec), RemainingLoopIterationsReg)
+      .addUse(LoopCounterPhiReg)
+      .addImm(1);
+
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::t2LoopEnd))
+      .addUse(RemainingLoopIterationsReg)
+      .addMBB(TpLoopBody);
+
+  BuildMI(TpLoopBody, Dl, TII->get(ARM::t2B))
+      .addMBB(TpExit)
+      .add(predOps(ARMCC::AL));
+}
+
 MachineBasicBlock *
 ARMTargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
                                                MachineBasicBlock *BB) const {
   const TargetInstrInfo *TII = Subtarget->getInstrInfo();
   DebugLoc dl = MI.getDebugLoc();
   bool isThumb2 = Subtarget->isThumb2();
   switch (MI.getOpcode()) {
   default: {
BuildMI(*BB, MI, dl, TII->get(ARM::tLDMIA_UPD))
         .add(MI.getOperand(3))  // PredImm
         .add(MI.getOperand(4))  // PredReg
         .add(MI.getOperand(0))  // Rt
         .cloneMemRefs(MI);
     MI.eraseFromParent();
     return BB;
   }
 
+  case ARM::MVE_MEMSETLOOPINST: {
+
+    // Transformation below expands MVE_MEMSETLOOPINST Pseudo instruction
+    // into a Tail Predicated (TP) Loop. It adds the instructions to calculate
+    // the iteration count =ceil(size_in_bytes/16)) in the TP entry block and
+    // adds the relevant instructions in the TP loop Body for generation of a
+    // WLSTP loop.
+
+    // Below is relevant portion of the CFG after the transformation.
+    // The Machine Basic Blocks are shown along with branch conditions (in
+    // brackets). Note that TP entry/exit MBBs depict the entry/exit of this
+    // portion of the CFG and may not necessarily be the entry/exit of the
+    // function.
+
+    //             (Relevant) CFG after transformation:
+    //               TP entry MBB
+    //                   |
+    //          |-----------------|
+    //       (n <= 0)          (n > 0)
+    //          |                 |
+    //          |         TP loop Body MBB
+    //          \                |
+    //            \             /
+    //              TP exit MBB
+
+    MachineFunction *MF = BB->getParent();
+    MachineRegisterInfo &MRI = MF->getRegInfo();
+
+    Register OpDestReg = MI.getOperand(0).getReg();
+    Register OpSrcReg = MI.getOperand(1).getReg();
+    Register OpSizeReg = MI.getOperand(2).getReg();
+
+    // Allocate the required MBBs and add to parent function.
+    MachineBasicBlock *TpEntry = BB;
+    MachineBasicBlock *TpLoopBody = MF->CreateMachineBasicBlock();
+    MachineBasicBlock *TpExit;
+
+    MF->push_back(TpLoopBody);
+
+    // If any instructions are present in the current block after
+    // MVE_MEMSETLOOPINST, move them into the exit block. This is required since
+    // a terminator(t2WhileLoopStart) will be placed at that site. If no
+    // instructions are present after MVE_MEMSETLOOPINST, then fallthrough is
+    // the exit.
+    TpExit = BB->splitAt(MI);
+    if (TpExit == BB) {
+      assert(BB->canFallThrough() &&
+             "exit Block must be Fallthrough of the block containing memcpy");
+      TpExit = BB->getFallThrough();
+    }
+
+    // Add logic for iteration count and setting up the source value
+    Register TotalIterationsReg, SrcVecReg;
+    std::tie(TotalIterationsReg, SrcVecReg) = genTPEntry(TpEntry, TpLoopBody, TpExit, OpSrcReg, OpSizeReg, TII, dl, MRI);
+
+    // Add the vectorized (and predicated) loads/store instructions
+    genTPLoopBody(TpLoopBody, TpEntry, TpExit, TII, dl, MRI, SrcVecReg,
+                  OpDestReg, OpSizeReg, TotalIterationsReg);
+
+    // Connect the blocks
+    TpEntry->addSuccessor(TpLoopBody);
+    TpLoopBody->addSuccessor(TpLoopBody);
+    TpLoopBody->addSuccessor(TpExit);
+
+    // Reorder for better readability of generated MIR
+    TpLoopBody->moveAfter(TpEntry);
+    TpExit->moveAfter(TpLoopBody);
+
+    // Finally, remove the memcpy Psuedo Instruction
+    MI.eraseFromParent();
+
+    // Return the exit block as it may contain other instructions requiring a
+    // custom inserter
+    return TpExit;
+  }
+
   // The Thumb2 pre-indexed stores have the same MI operands, they just
   // define them differently in the .td files from the isel patterns, so
   // they need pseudos.
   case ARM::t2STR_preidx:
     MI.setDesc(TII->get(ARM::t2STR_PRE));
     return BB;
   case ARM::t2STRB_preidx:
     MI.setDesc(TII->get(ARM::t2STRB_PRE));

llvm/lib/Target/ARM/ARMInstrMVE.td

class MVE_WLSTP<string asm, bits<2> size>
   bits<11> label;
   let Inst{13} = 0b0;
   let Inst{11} = label{0};
   let Inst{10-1} = label{10-1};
   let isBranch = 1;
   let isTerminator = 1;
 }
 
+def SDT_MVEMEMSETLOOPNODE
+    : SDTypeProfile<0, 3, [SDTCisPtrTy<0>, SDTCisVT<1, i32>, SDTCisVT<2, i32>]>;
+def MVE_MEMSETLOOPNODE : SDNode<"ARMISD::MEMSETLOOP", SDT_MVEMEMSETLOOPNODE,
+                                [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
+
+let usesCustomInserter = 1, hasNoSchedulingInfo = 1 in {
+  def MVE_MEMSETLOOPINST : PseudoInst<(outs),
+        (ins rGPR:$dst, rGPR:$src, rGPR:$sz),
+        NoItinerary,
+        [(MVE_MEMSETLOOPNODE rGPR:$dst, rGPR:$src, rGPR:$sz)]>;
+}
+
 def MVE_DLSTP_8  : MVE_DLSTP<"dlstp.8",  0b00>;
 def MVE_DLSTP_16 : MVE_DLSTP<"dlstp.16", 0b01>;
 def MVE_DLSTP_32 : MVE_DLSTP<"dlstp.32", 0b10>;
 def MVE_DLSTP_64 : MVE_DLSTP<"dlstp.64", 0b11>;
 
 def MVE_WLSTP_8  : MVE_WLSTP<"wlstp.8",  0b00>;
 def MVE_WLSTP_16 : MVE_WLSTP<"wlstp.16", 0b01>;
 def MVE_WLSTP_32 : MVE_WLSTP<"wlstp.32", 0b10>;

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

 
 #include "ARMTargetMachine.h"
 #include "llvm/CodeGen/SelectionDAG.h"
 #include "llvm/IR/DerivedTypes.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "arm-selectiondag-info"
 
+static cl::opt<cl::boolOrDefault>
+    EnableMemsetTPLoop("arm-memset-tploop", cl::Hidden,
+                       cl::desc("Enable/disable conversion of llvm.memset to "
+                                "Tail predicated loops (WLSTP)"));
+
 // Emit, if possible, a specialized version of the given Libcall. Typically this
 // means selecting the appropriately aligned version, but we also convert memset
 // of 0 into memclr.
 SDValue ARMSelectionDAGInfo::EmitSpecializedLibcall(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, unsigned Align, RTLIB::Libcall LC) const {
   const ARMSubtarget &Subtarget =
       DAG.getMachineFunction().getSubtarget<ARMSubtarget>();
CLI.setDebugLoc(dl)
                                 TLI->getPointerTy(DAG.getDataLayout())),
           std::move(Args))
       .setDiscardResult();
   std::pair<SDValue,SDValue> CallResult = TLI->LowerCallTo(CLI);
 
   return CallResult.second;
 }
 
 SDValue ARMSelectionDAGInfo::EmitTargetCodeForMemcpy(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   const ARMSubtarget &Subtarget =
       DAG.getMachineFunction().getSubtarget<ARMSubtarget>();
   // Do repeated 4-byte loads and stores. To be improved.
   // This requires 4-byte alignment.
   if (Alignment < Align(4))
     return SDValue();
   // This requires the copy size to be a constant, preferably
   // within a subtarget-specific limit.
   ConstantSDNode *ConstantSize = dyn_cast<ConstantSDNode>(Size);
   if (!ConstantSize)
     return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                   Alignment.value(), RTLIB::MEMCPY);
   uint64_t SizeVal = ConstantSize->getZExtValue();
   if (!AlwaysInline && SizeVal > Subtarget.getMaxInlineSizeThreshold())
     return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                   Alignment.value(), RTLIB::MEMCPY);
 
   unsigned BytesLeft = SizeVal & 3;
   unsigned NumMemOps = SizeVal >> 2;
   unsigned EmittedNumMemOps = 0;
   EVT VT = MVT::i32;
   unsigned VTSize = 4;
SDValue ARMSelectionDAGInfo::EmitTargetCodeForMemmove(
   return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                 Alignment.value(), RTLIB::MEMMOVE);
 }
 
 SDValue ARMSelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, Align Alignment, bool isVolatile,
     MachinePointerInfo DstPtrInfo) const {
+
+  const ARMSubtarget &Subtarget =
+      DAG.getMachineFunction().getSubtarget<ARMSubtarget>();
+
+  // Generate TP loop for llvm.memset for both compile-time constant/variable input size.
+  if (Subtarget.hasMVEIntegerOps())
+    if ((EnableMemsetTPLoop == cl::BOU_TRUE) || (EnableMemsetTPLoop == cl::BOU_UNSET &&
+        !DAG.getMachineFunction().getFunction().hasOptNone())) {
+      return DAG.getNode(ARMISD::MEMSETLOOP, dl, MVT::Other, Chain, Dst,
+                         DAG.getZExtOrTrunc(Src, dl, MVT::i32),
+                         DAG.getZExtOrTrunc(Size, dl, MVT::i32));
+  }
+
   return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                 Alignment.value(), RTLIB::MEMSET);
 }

llvm/test/CodeGen/Thumb2/LowOverheadLoops/memcall.ll

for.body:                                         ; preds = %entry, %for.body
   %inc = add nuw nsw i32 %i.011, 1
   %exitcond.not = icmp eq i32 %inc, %n
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
 define void @test_memset(i32* nocapture %x, i32 %n, i32 %m) {
 ; CHECK-LABEL: test_memset:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r1, #1
-; CHECK-NEXT:    blt .LBB1_3
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
-; CHECK-NEXT:    mov r4, r2
-; CHECK-NEXT:    mov r5, r1
-; CHECK-NEXT:    mov r6, r0
-; CHECK-NEXT:    lsls r7, r2, #2
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r4, pc}
+; CHECK-NEXT:  .LBB1_1: @ %for.body.preheader
+; CHECK-NEXT:    movs r3, #0
+; CHECK-NEXT:    lsl.w r12, r2, #2
+; CHECK-NEXT:    vdup.8 q0, r3
+; CHECK-NEXT:    b .LBB1_2
 ; CHECK-NEXT:  .LBB1_2: @ %for.body
-; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    mov r0, r6
-; CHECK-NEXT:    mov r1, r4
-; CHECK-NEXT:    bl __aeabi_memclr4
-; CHECK-NEXT:    add r6, r7
-; CHECK-NEXT:    subs r5, #1
-; CHECK-NEXT:    bne .LBB1_2
-; CHECK-NEXT:  .LBB1_3: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    @ =>This Loop Header: Depth=1
+; CHECK-NEXT:    @ Child Loop BB1_4 Depth 2
+; CHECK-NEXT:    mov r4, r0
+; CHECK-NEXT:    mov r3, r2
+; CHECK-NEXT:    wlstp.8 lr, r3, .LBB1_3
+; CHECK-NEXT:    b .LBB1_4
+; CHECK-NEXT:  .LBB1_3: @ %for.body
+; CHECK-NEXT:    @ in Loop: Header=BB1_2 Depth=1
+; CHECK-NEXT:    add r0, r12
+; CHECK-NEXT:    subs r1, #1
+; CHECK-NEXT:    beq .LBB1_5
+; CHECK-NEXT:    b .LBB1_2
+; CHECK-NEXT:  .LBB1_4: @ Parent Loop BB1_2 Depth=1
+; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
+; CHECK-NEXT:    vstrb.8 q0, [r4], #16
+; CHECK-NEXT:    letp lr, .LBB1_4
+; CHECK-NEXT:    b .LBB1_3
+; CHECK-NEXT:  .LBB1_5: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r4, pc}
 entry:
   %cmp5 = icmp sgt i32 %n, 0
   br i1 %cmp5, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   ret void
 
 for.body:                                         ; preds = %entry, %for.body

llvm/test/CodeGen/Thumb2/mve-gather-scatter-optimisation.ll

 
 define dso_local void @arm_mat_mult_q15(i16* noalias nocapture readonly %A, i16* noalias nocapture readonly %B, i16* noalias nocapture %C, i32 %n, i32 %m, i32 %l) local_unnamed_addr #0 {
 ; CHECK-LABEL: arm_mat_mult_q15:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13}
-; CHECK-NEXT:    .pad #24
-; CHECK-NEXT:    sub sp, #24
-; CHECK-NEXT:    str r0, [sp, #20] @ 4-byte Spill
+; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
+; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
+; CHECK-NEXT:    .pad #32
+; CHECK-NEXT:    sub sp, #32
+; CHECK-NEXT:    strd r0, r2, [sp, #24] @ 8-byte Folded Spill
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    str r3, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT:    str r3, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:    mov r0, r3
 ; CHECK-NEXT:    itt ne
-; CHECK-NEXT:    ldrne r0, [sp, #112]
+; CHECK-NEXT:    ldrne r0, [sp, #136]
 ; CHECK-NEXT:    cmpne r0, #0
 ; CHECK-NEXT:    bne .LBB10_2
 ; CHECK-NEXT:  .LBB10_1: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #24
-; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
+; CHECK-NEXT:    add sp, #32
+; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ; CHECK-NEXT:  .LBB10_2: @ %for.cond1.preheader.us.preheader
-; CHECK-NEXT:    ldr.w r9, [sp, #116]
-; CHECK-NEXT:    mov r6, r1
-; CHECK-NEXT:    movs r1, #1
-; CHECK-NEXT:    mov r11, r2
-; CHECK-NEXT:    bic r10, r9, #3
-; CHECK-NEXT:    mov.w r8, #0
-; CHECK-NEXT:    sub.w r0, r10, #4
-; CHECK-NEXT:    add.w r0, r1, r0, lsr #2
-; CHECK-NEXT:    ldr r1, [sp, #112]
-; CHECK-NEXT:    str r0, [sp, #12] @ 4-byte Spill
-; CHECK-NEXT:    lsl.w r0, r9, #1
-; CHECK-NEXT:    str r0, [sp] @ 4-byte Spill
-; CHECK-NEXT:    adr r0, .LCPI10_0
-; CHECK-NEXT:    vdup.32 q4, r1
-; CHECK-NEXT:    vldrw.u32 q5, [r0]
-; CHECK-NEXT:    lsls r4, r1, #1
-; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
-; CHECK-NEXT:    vshl.i32 q6, q4, #2
-; CHECK-NEXT:    movs r1, #0
+; CHECK-NEXT:    ldr.w r12, [sp, #140]
+; CHECK-NEXT:    movs r7, #1
+; CHECK-NEXT:    mov.w r11, #0
+; CHECK-NEXT:    bic r2, r12, #3
+; CHECK-NEXT:    vdup.8 q0, r11
+; CHECK-NEXT:    subs r3, r2, #4
+; CHECK-NEXT:    add.w r0, r7, r3, lsr #2
+; CHECK-NEXT:    ldr r7, [sp, #136]
+; CHECK-NEXT:    adr r3, .LCPI10_0
 ; CHECK-NEXT:    str r0, [sp, #16] @ 4-byte Spill
-; CHECK-NEXT:    b .LBB10_5
-; CHECK-NEXT:  .LBB10_3: @ %for.cond5.preheader.us73.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB10_5 Depth=1
-; CHECK-NEXT:    add.w r0, r11, r12, lsl #1
-; CHECK-NEXT:    mov r1, r4
-; CHECK-NEXT:    bl __aeabi_memclr
-; CHECK-NEXT:  .LBB10_4: @ %for.cond1.for.cond.cleanup3_crit_edge.us
-; CHECK-NEXT:    @ in Loop: Header=BB10_5 Depth=1
-; CHECK-NEXT:    ldr r0, [sp] @ 4-byte Reload
-; CHECK-NEXT:    add r8, r9
-; CHECK-NEXT:    ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT:    add r1, r0
-; CHECK-NEXT:    str r1, [sp, #16] @ 4-byte Spill
-; CHECK-NEXT:    ldr r1, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:    adds r1, #1
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    beq .LBB10_1
-; CHECK-NEXT:  .LBB10_5: @ %for.cond1.preheader.us
+; CHECK-NEXT:    lsl.w r0, r12, #1
+; CHECK-NEXT:    vdup.32 q1, r7
+; CHECK-NEXT:    vldrw.u32 q2, [r3]
+; CHECK-NEXT:    str r0, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT:    ldr r0, [sp, #24] @ 4-byte Reload
+; CHECK-NEXT:    lsls r6, r7, #1
+; CHECK-NEXT:    vshl.i32 q3, q1, #2
+; CHECK-NEXT:    movs r3, #0
+; CHECK-NEXT:    str r0, [sp, #20] @ 4-byte Spill
+; CHECK-NEXT:    b .LBB10_3
+; CHECK-NEXT:  .LBB10_3: @ %for.cond1.preheader.us
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
-; CHECK-NEXT:    @ Child Loop BB10_8 Depth 2
-; CHECK-NEXT:    @ Child Loop BB10_11 Depth 3
-; CHECK-NEXT:    @ Child Loop BB10_14 Depth 3
-; CHECK-NEXT:    ldr r0, [sp, #112]
-; CHECK-NEXT:    cmp.w r9, #0
-; CHECK-NEXT:    str r1, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT:    mul r12, r1, r0
-; CHECK-NEXT:    beq .LBB10_3
-; CHECK-NEXT:  @ %bb.6: @ %for.cond5.preheader.us.us.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB10_5 Depth=1
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    b .LBB10_8
-; CHECK-NEXT:  .LBB10_7: @ %for.cond5.for.cond.cleanup7_crit_edge.us.us
-; CHECK-NEXT:    @ in Loop: Header=BB10_8 Depth=2
-; CHECK-NEXT:    ldr r0, [sp, #112]
-; CHECK-NEXT:    add.w r3, r1, r12
-; CHECK-NEXT:    adds r1, #1
-; CHECK-NEXT:    cmp r1, r0
-; CHECK-NEXT:    strh.w r2, [r11, r3, lsl #1]
-; CHECK-NEXT:    beq .LBB10_4
-; CHECK-NEXT:  .LBB10_8: @ %for.cond5.preheader.us.us
-; CHECK-NEXT:    @ Parent Loop BB10_5 Depth=1
+; CHECK-NEXT:    @ Child Loop BB10_6 Depth 2
+; CHECK-NEXT:    @ Child Loop BB10_9 Depth 3
+; CHECK-NEXT:    @ Child Loop BB10_12 Depth 3
+; CHECK-NEXT:    @ Child Loop BB10_15 Depth 2
+; CHECK-NEXT:    mul r5, r3, r7
+; CHECK-NEXT:    cmp.w r12, #0
+; CHECK-NEXT:    str r3, [sp, #12] @ 4-byte Spill
+; CHECK-NEXT:    beq .LBB10_13
+; CHECK-NEXT:  @ %bb.4: @ %for.cond5.preheader.us.us.preheader
+; CHECK-NEXT:    @ in Loop: Header=BB10_3 Depth=1
+; CHECK-NEXT:    mov.w r8, #0
+; CHECK-NEXT:    b .LBB10_6
+; CHECK-NEXT:  .LBB10_5: @ %for.cond5.for.cond.cleanup7_crit_edge.us.us
+; CHECK-NEXT:    @ in Loop: Header=BB10_6 Depth=2
+; CHECK-NEXT:    ldr r3, [sp, #28] @ 4-byte Reload
+; CHECK-NEXT:    add.w r0, r8, r5
+; CHECK-NEXT:    add.w r8, r8, #1
+; CHECK-NEXT:    cmp r8, r7
+; CHECK-NEXT:    strh.w r10, [r3, r0, lsl #1]
+; CHECK-NEXT:    beq .LBB10_14
+; CHECK-NEXT:  .LBB10_6: @ %for.cond5.preheader.us.us
+; CHECK-NEXT:    @ Parent Loop BB10_3 Depth=1
 ; CHECK-NEXT:    @ => This Loop Header: Depth=2
-; CHECK-NEXT:    @ Child Loop BB10_11 Depth 3
-; CHECK-NEXT:    @ Child Loop BB10_14 Depth 3
-; CHECK-NEXT:    cmp.w r9, #3
-; CHECK-NEXT:    bhi .LBB10_10
-; CHECK-NEXT:  @ %bb.9: @ in Loop: Header=BB10_8 Depth=2
-; CHECK-NEXT:    movs r7, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    b .LBB10_13
-; CHECK-NEXT:  .LBB10_10: @ %vector.ph
-; CHECK-NEXT:    @ in Loop: Header=BB10_8 Depth=2
-; CHECK-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT:    vmov q1, q4
-; CHECK-NEXT:    vmov.i32 q0, #0x0
-; CHECK-NEXT:    vmlas.u32 q1, q5, r1
+; CHECK-NEXT:    @ Child Loop BB10_9 Depth 3
+; CHECK-NEXT:    @ Child Loop BB10_12 Depth 3
+; CHECK-NEXT:    cmp.w r12, #3
+; CHECK-NEXT:    bhi .LBB10_8
+; CHECK-NEXT:  @ %bb.7: @ in Loop: Header=BB10_6 Depth=2
+; CHECK-NEXT:    movs r4, #0
+; CHECK-NEXT:    mov.w r10, #0
+; CHECK-NEXT:    b .LBB10_11
+; CHECK-NEXT:  .LBB10_8: @ %vector.ph
+; CHECK-NEXT:    @ in Loop: Header=BB10_6 Depth=2
+; CHECK-NEXT:    ldr r0, [sp, #16] @ 4-byte Reload
+; CHECK-NEXT:    vmov q5, q1
+; CHECK-NEXT:    vmov.i32 q4, #0x0
+; CHECK-NEXT:    vmlas.u32 q5, q2, r8
 ; CHECK-NEXT:    dls lr, r0
-; CHECK-NEXT:    ldr r2, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT:  .LBB10_11: @ %vector.body
-; CHECK-NEXT:    @ Parent Loop BB10_5 Depth=1
-; CHECK-NEXT:    @ Parent Loop BB10_8 Depth=2
+; CHECK-NEXT:    ldr r3, [sp, #20] @ 4-byte Reload
+; CHECK-NEXT:  .LBB10_9: @ %vector.body
+; CHECK-NEXT:    @ Parent Loop BB10_3 Depth=1
+; CHECK-NEXT:    @ Parent Loop BB10_6 Depth=2
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=3
-; CHECK-NEXT:    vadd.i32 q2, q1, q6
-; CHECK-NEXT:    vldrh.s32 q3, [r6, q1, uxtw #1]
-; CHECK-NEXT:    vldrh.s32 q1, [r2], #8
-; CHECK-NEXT:    vmul.i32 q1, q3, q1
-; CHECK-NEXT:    vadd.i32 q0, q1, q0
-; CHECK-NEXT:    vmov q1, q2
-; CHECK-NEXT:    le lr, .LBB10_11
-; CHECK-NEXT:  @ %bb.12: @ %middle.block
-; CHECK-NEXT:    @ in Loop: Header=BB10_8 Depth=2
-; CHECK-NEXT:    vaddv.u32 r2, q0
-; CHECK-NEXT:    cmp r10, r9
-; CHECK-NEXT:    mov r7, r10
-; CHECK-NEXT:    beq .LBB10_7
-; CHECK-NEXT:  .LBB10_13: @ %for.body8.us.us.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB10_8 Depth=2
-; CHECK-NEXT:    ldr r0, [sp, #112]
-; CHECK-NEXT:    add.w r5, r8, r7
-; CHECK-NEXT:    sub.w lr, r9, r7
-; CHECK-NEXT:    mla r3, r0, r7, r1
-; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
-; CHECK-NEXT:    add.w r5, r0, r5, lsl #1
-; CHECK-NEXT:    add.w r3, r6, r3, lsl #1
-; CHECK-NEXT:  .LBB10_14: @ %for.body8.us.us
-; CHECK-NEXT:    @ Parent Loop BB10_5 Depth=1
-; CHECK-NEXT:    @ Parent Loop BB10_8 Depth=2
+; CHECK-NEXT:    vadd.i32 q6, q5, q3
+; CHECK-NEXT:    vldrh.s32 q7, [r1, q5, uxtw #1]
+; CHECK-NEXT:    vldrh.s32 q5, [r3], #8
+; CHECK-NEXT:    vmul.i32 q5, q7, q5
+; CHECK-NEXT:    vadd.i32 q4, q5, q4
+; CHECK-NEXT:    vmov q5, q6
+; CHECK-NEXT:    le lr, .LBB10_9
+; CHECK-NEXT:  @ %bb.10: @ %middle.block
+; CHECK-NEXT:    @ in Loop: Header=BB10_6 Depth=2
+; CHECK-NEXT:    vaddv.u32 r10, q4
+; CHECK-NEXT:    cmp r2, r12
+; CHECK-NEXT:    mov r4, r2
+; CHECK-NEXT:    beq .LBB10_5
+; CHECK-NEXT:  .LBB10_11: @ %for.body8.us.us.preheader
+; CHECK-NEXT:    @ in Loop: Header=BB10_6 Depth=2
+; CHECK-NEXT:    mla r3, r7, r4, r8
+; CHECK-NEXT:    add.w r0, r11, r4
+; CHECK-NEXT:    ldr r7, [sp, #24] @ 4-byte Reload
+; CHECK-NEXT:    sub.w lr, r12, r4
+; CHECK-NEXT:    add.w r9, r7, r0, lsl #1
+; CHECK-NEXT:    ldr r7, [sp, #136]
+; CHECK-NEXT:    add.w r3, r1, r3, lsl #1
+; CHECK-NEXT:  .LBB10_12: @ %for.body8.us.us
+; CHECK-NEXT:    @ Parent Loop BB10_3 Depth=1
+; CHECK-NEXT:    @ Parent Loop BB10_6 Depth=2
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=3
-; CHECK-NEXT:    ldrsh.w r0, [r3]
-; CHECK-NEXT:    add r3, r4
-; CHECK-NEXT:    ldrsh r7, [r5], #2
-; CHECK-NEXT:    smlabb r2, r0, r7, r2
-; CHECK-NEXT:    le lr, .LBB10_14
-; CHECK-NEXT:    b .LBB10_7
+; CHECK-NEXT:    ldrsh.w r4, [r3]
+; CHECK-NEXT:    add r3, r6
+; CHECK-NEXT:    ldrsh r0, [r9], #2
+; CHECK-NEXT:    smlabb r10, r4, r0, r10
+; CHECK-NEXT:    le lr, .LBB10_12
+; CHECK-NEXT:    b .LBB10_5
+; CHECK-NEXT:  .LBB10_13: @ %for.cond5.preheader.us73.preheader
+; CHECK-NEXT:    @ in Loop: Header=BB10_3 Depth=1
+; CHECK-NEXT:    ldr r0, [sp, #28] @ 4-byte Reload
+; CHECK-NEXT:    add.w r3, r0, r5, lsl #1
+; CHECK-NEXT:    mov r5, r6
+; CHECK-NEXT:    wlstp.8 lr, r5, .LBB10_14
+; CHECK-NEXT:    b .LBB10_15
+; CHECK-NEXT:  .LBB10_14: @ %for.cond1.for.cond.cleanup3_crit_edge.us
+; CHECK-NEXT:    @ in Loop: Header=BB10_3 Depth=1
+; CHECK-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
+; CHECK-NEXT:    add r11, r12
+; CHECK-NEXT:    ldr r3, [sp, #20] @ 4-byte Reload
+; CHECK-NEXT:    add r3, r0
+; CHECK-NEXT:    str r3, [sp, #20] @ 4-byte Spill
+; CHECK-NEXT:    ldr r3, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-NEXT:    adds r3, #1
+; CHECK-NEXT:    cmp r3, r0
+; CHECK-NEXT:    beq.w .LBB10_1
+; CHECK-NEXT:    b .LBB10_3
+; CHECK-NEXT:  .LBB10_15: @ Parent Loop BB10_3 Depth=1
+; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
+; CHECK-NEXT:    vstrb.8 q0, [r3], #16
+; CHECK-NEXT:    letp lr, .LBB10_15
+; CHECK-NEXT:    b .LBB10_14
 ; CHECK-NEXT:    .p2align 4
-; CHECK-NEXT:  @ %bb.15:
+; CHECK-NEXT:  @ %bb.16:
 ; CHECK-NEXT:  .LCPI10_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:
   %cmp48 = icmp eq i32 %n, 0
   br i1 %cmp48, label %for.cond.cleanup, label %for.cond1.preheader.lr.ph

llvm/test/CodeGen/Thumb2/mve-phireg.ll

 @a = external dso_local global i32, align 4
 @b = dso_local local_unnamed_addr global i32 ptrtoint (i32* @a to i32), align 4
 @c = dso_local global i32 2, align 4
 @d = dso_local global i32 2, align 4
 
 define dso_local i32 @e() #0 {
 ; CHECK-LABEL: e:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, lr}
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, lr}
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11}
+; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
+; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #392
 ; CHECK-NEXT:    sub sp, #392
-; CHECK-NEXT:    movw r9, :lower16:.L_MergedGlobals
-; CHECK-NEXT:    vldr s0, .LCPI1_0
-; CHECK-NEXT:    movt r9, :upper16:.L_MergedGlobals
-; CHECK-NEXT:    vldr s3, .LCPI1_1
-; CHECK-NEXT:    mov r7, r9
-; CHECK-NEXT:    mov r5, r9
-; CHECK-NEXT:    ldr r0, [r7, #4]!
-; CHECK-NEXT:    movw r4, :lower16:e
-; CHECK-NEXT:    ldr r1, [r5, #8]!
-; CHECK-NEXT:    movt r4, :upper16:e
-; CHECK-NEXT:    vmov r6, s3
-; CHECK-NEXT:    vdup.32 q4, r7
-; CHECK-NEXT:    vmov s1, r7
-; CHECK-NEXT:    vmov q1[2], q1[0], r5, r5
-; CHECK-NEXT:    vmov s9, r4
-; CHECK-NEXT:    vmov q1[3], q1[1], r6, r4
-; CHECK-NEXT:    vmov.f32 s2, s1
-; CHECK-NEXT:    vmov q3, q4
-; CHECK-NEXT:    vmov.f32 s8, s0
-; CHECK-NEXT:    vmov q5, q4
-; CHECK-NEXT:    vmov.f32 s10, s1
-; CHECK-NEXT:    vstrw.32 q1, [sp, #76]
-; CHECK-NEXT:    vmov q1[2], q1[0], r7, r6
-; CHECK-NEXT:    mov.w r8, #4
-; CHECK-NEXT:    mov.w r10, #0
-; CHECK-NEXT:    vmov q1[3], q1[1], r7, r4
-; CHECK-NEXT:    vmov.32 q3[0], r4
-; CHECK-NEXT:    vmov.32 q5[1], r4
-; CHECK-NEXT:    str r1, [r0]
-; CHECK-NEXT:    vmov.f32 s11, s3
+; CHECK-NEXT:    movw r8, :lower16:.L_MergedGlobals
+; CHECK-NEXT:    vldr s8, .LCPI1_0
+; CHECK-NEXT:    movt r8, :upper16:.L_MergedGlobals
+; CHECK-NEXT:    vldr s11, .LCPI1_1
+; CHECK-NEXT:    mov r6, r8
+; CHECK-NEXT:    mov r3, r8
+; CHECK-NEXT:    ldr r7, [r6, #8]!
+; CHECK-NEXT:    movw r2, :lower16:e
+; CHECK-NEXT:    ldr r4, [r3, #4]!
+; CHECK-NEXT:    vmov r5, s11
+; CHECK-NEXT:    movt r2, :upper16:e
+; CHECK-NEXT:    vmov q0[2], q0[0], r6, r6
+; CHECK-NEXT:    vdup.32 q6, r3
+; CHECK-NEXT:    vmov q0[3], q0[1], r5, r2
+; CHECK-NEXT:    vmov s9, r3
+; CHECK-NEXT:    vmov q5[2], q5[0], r3, r5
+; CHECK-NEXT:    vmov q7, q6
+; CHECK-NEXT:    vstrw.32 q0, [sp, #76]
+; CHECK-NEXT:    vmov q0[2], q0[0], r6, r3
+; CHECK-NEXT:    vmov q3, q6
 ; CHECK-NEXT:    movs r1, #64
-; CHECK-NEXT:    strh.w r8, [sp, #390]
-; CHECK-NEXT:    strd r0, r10, [sp, #24]
-; CHECK-NEXT:    vstrw.32 q0, [sp, #44]
-; CHECK-NEXT:    str r0, [r0]
-; CHECK-NEXT:    vstrw.32 q2, [r0]
+; CHECK-NEXT:    vmov s17, r2
+; CHECK-NEXT:    vmov q0[3], q0[1], r2, r6
+; CHECK-NEXT:    vmov q5[3], q5[1], r3, r2
+; CHECK-NEXT:    vmov.32 q7[0], r2
+; CHECK-NEXT:    vmov.32 q6[1], r2
+; CHECK-NEXT:    vmov.f32 s10, s9
+; CHECK-NEXT:    vmov.f32 s16, s8
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    vmov q1[2], q1[0], r3, r3
+; CHECK-NEXT:    vmov.f32 s18, s9
+; CHECK-NEXT:    mov.w r12, #4
+; CHECK-NEXT:    vmov.f32 s19, s11
+; CHECK-NEXT:    vstrw.32 q2, [sp, #44]
+; CHECK-NEXT:    vmov q1[3], q1[1], r6, r5
+; CHECK-NEXT:    vmov.32 q3[0], r0
+; CHECK-NEXT:    vdup.8 q2, r0
+; CHECK-NEXT:    @ implicit-def: $r2
+; CHECK-NEXT:    str r0, [sp, #28]
+; CHECK-NEXT:    vstrw.32 q4, [r0]
+; CHECK-NEXT:    strh.w r12, [sp, #390]
+; CHECK-NEXT:    vstrw.32 q6, [r0]
+; CHECK-NEXT:    str r4, [sp, #24]
+; CHECK-NEXT:    vstrw.32 q7, [r0]
+; CHECK-NEXT:    str r7, [r0]
 ; CHECK-NEXT:    vstrw.32 q5, [r0]
-; CHECK-NEXT:    vstrw.32 q3, [r0]
-; CHECK-NEXT:    vstrw.32 q1, [r0]
-; CHECK-NEXT:    bl __aeabi_memclr4
-; CHECK-NEXT:    vmov q0[2], q0[0], r5, r7
-; CHECK-NEXT:    vmov q1[2], q1[0], r7, r7
-; CHECK-NEXT:    vmov q0[3], q0[1], r4, r5
-; CHECK-NEXT:    vmov q1[3], q1[1], r5, r6
-; CHECK-NEXT:    vmov.32 q4[0], r10
+; CHECK-NEXT:    str r4, [r0]
+; CHECK-NEXT:    wlstp.8 lr, r1, .LBB1_2
+; CHECK-NEXT:  .LBB1_1: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    letp lr, .LBB1_1
+; CHECK-NEXT:  .LBB1_2: @ %entry
 ; CHECK-NEXT:    vstrw.32 q0, [r0]
-; CHECK-NEXT:    str.w r10, [r9]
-; CHECK-NEXT:    vstrw.32 q4, [r0]
+; CHECK-NEXT:    str.w r0, [r8]
+; CHECK-NEXT:    vstrw.32 q3, [r0]
 ; CHECK-NEXT:    vstrw.32 q1, [r0]
-; CHECK-NEXT:    str.w r8, [sp, #308]
-; CHECK-NEXT:  .LBB1_1: @ %for.cond
+; CHECK-NEXT:    str.w r12, [sp, #308]
+; CHECK-NEXT:  .LBB1_3: @ %for.cond
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    b .LBB1_1
+; CHECK-NEXT:    b .LBB1_3
 ; CHECK-NEXT:    .p2align 2
-; CHECK-NEXT:  @ %bb.2:
+; CHECK-NEXT:  @ %bb.4:
 ; CHECK-NEXT:  .LCPI1_0:
 ; CHECK-NEXT:    .long 0x00000004 @ float 5.60519386E-45
 ; CHECK-NEXT:  .LCPI1_1:
 ; CHECK-NEXT:    .long 0x00000000 @ float 0
 entry:
   %f = alloca i16, align 2
   %g = alloca [3 x [8 x [4 x i16*]]], align 4
   store i16 4, i16* %f, align 2

llvm/test/CodeGen/Thumb2/mve_tp_loop.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O1 -mtriple=arm-arm-none-eabi -mcpu=cortex-m55 --verify-machineinstrs %s -o - | FileCheck %s
+
+declare void @llvm.memset.p0i8.i32(i8* nocapture writeonly, i8, i32, i1 immarg)
+;declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+
+; Check that WLSTP loop is generated for simplest case of align = 1
+; TODO: might need to change this based on results of benchmarking
+define void @test1(i8* %X, i8 zeroext %c, i32 %n) {
+; CHECK-LABEL: test1:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vdup.8 q0, r1
+; CHECK-NEXT:    wlstp.8 lr, r2, .LBB0_2
+; CHECK-NEXT:  .LBB0_1: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q0, [r0], #16
+; CHECK-NEXT:    letp lr, .LBB0_1
+; CHECK-NEXT:  .LBB0_2: @ %entry
+; CHECK-NEXT:    pop {r7, pc}
+entry:
+  call void @llvm.memset.p0i8.i32(i8* align 1 %X, i8 %c, i32 %n, i1 false)
+  ret void
+}
+
+
+; Check that WLSTP loop is generated for alignment >= 4
+define void @test2(i32* %X, i8 zeroext %c, i32 %n) {
+; CHECK-LABEL: test2:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    vdup.8 q0, r1
+; CHECK-NEXT:    wlstp.8 lr, r2, .LBB1_2
+; CHECK-NEXT:  .LBB1_1: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q0, [r0], #16
+; CHECK-NEXT:    letp lr, .LBB1_1
+; CHECK-NEXT:  .LBB1_2: @ %entry
+; CHECK-NEXT:    pop {r7, pc}
+entry:
+  %0 = bitcast i32* %X to i8*
+  call void @llvm.memset.p0i8.i32(i8* align 4 %0, i8 %c, i32 %n, i1 false)
+  ret void
+}
+
+
+; Checks that transform correctly handles input with some arithmetic on input arguments.
+; void test3(int* X, char c, int n)
+; {
+;     memset(X+2, c, (n*2)+10);
+; }
+
+define void @test3(i32* %X, i8 zeroext %c, i32 %n) {
+; CHECK-LABEL: test3:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    movs r3, #10
+; CHECK-NEXT:    add.w r2, r3, r2, lsl #1
+; CHECK-NEXT:    adds r0, #8
+; CHECK-NEXT:    vdup.8 q0, r1
+; CHECK-NEXT:    wlstp.8 lr, r2, .LBB2_2
+; CHECK-NEXT:  .LBB2_1: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q0, [r0], #16
+; CHECK-NEXT:    letp lr, .LBB2_1
+; CHECK-NEXT:  .LBB2_2: @ %entry
+; CHECK-NEXT:    pop {r7, pc}
+entry:
+  %add.ptr = getelementptr inbounds i32, i32* %X, i32 2
+  %0 = bitcast i32* %add.ptr to i8*
+  %mul = shl nsw i32 %n, 1
+  %add = add nsw i32 %mul, 10
+  call void @llvm.memset.p0i8.i32(i8* nonnull align 4 %0, i8 %c, i32 %add, i1 false)
+  ret void
+}
+
+
+
+
+; Checks that transform handles for-loops (that get implicitly converted to memset)
+; void test4(int* X, char Y, int n){
+;     for(int i = 0; i < n; ++i){
+;         X[i] = c;
+;     }
+; }
+
+define void @test4(i8* nocapture %X, i8 zeroext %c, i32 %n) {
+; CHECK-LABEL: test4:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r7, pc}
+; CHECK-NEXT:  .LBB3_1: @ %for.body.preheader
+; CHECK-NEXT:    vdup.8 q0, r1
+; CHECK-NEXT:    wlstp.8 lr, r2, .LBB3_3
+; CHECK-NEXT:  .LBB3_2: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q0, [r0], #16
+; CHECK-NEXT:    letp lr, .LBB3_2
+; CHECK-NEXT:  .LBB3_3: @ %for.cond.cleanup
+; CHECK-NEXT:    pop {r7, pc}
+entry:
+  %cmp4 = icmp sgt i32 %n, 0
+  br i1 %cmp4, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:                               ; preds = %entry
+  call void @llvm.memset.p0i8.i32(i8* align 4 %X, i8 %c, i32 %n, i1 false)
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.body.preheader, %entry
+  ret void
+}
+
+; Checks that transform handles case with 0 as src value. No difference is expected.
+define void @test5(i32* %X, i8 zeroext %c, i32 %n) {
+; CHECK-LABEL: test5:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    movs r1, #0
+; CHECK-NEXT:    vdup.8 q0, r1
+; CHECK-NEXT:    wlstp.8 lr, r2, .LBB4_2
+; CHECK-NEXT:  .LBB4_1: @ =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    vstrb.8 q0, [r0], #16
+; CHECK-NEXT:    letp lr, .LBB4_1
+; CHECK-NEXT:  .LBB4_2: @ %entry
+; CHECK-NEXT:    pop {r7, pc}
+entry:
+  %0 = bitcast i32* %X to i8*
+  call void @llvm.memset.p0i8.i32(i8* align 4 %0, i8 0, i32 %n, i1 false)
+  ret void
+}

llvm/test/CodeGen/Thumb2/mve_tp_loop.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -O1 -mtriple=arm-arm-none-eabi -mcpu=cortex-m55 -simplify-mir -run-pass=finalize-isel %s -o - | FileCheck %s
+--- |
+  ; ModuleID = '<stdin>'
+  source_filename = "/Users/maljaj01/llvm_work/llvm-project/test.c"
+  target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
+  target triple = "arm-arm-none-eabi"
+
+  ; Function Attrs: nofree nounwind willreturn writeonly
+  define dso_local void @test1(i32* nocapture %X, i8 zeroext %c, i32 %n) local_unnamed_addr #0 {
+  entry:
+    %0 = bitcast i32* %X to i8*
+    tail call void @llvm.memset.p0i8.i32(i8* align 4 %0, i8 %c, i32 %n, i1 false)
+    ret void
+  }
+
+  ; Function Attrs: argmemonly nofree nosync nounwind willreturn writeonly
+  declare void @llvm.memset.p0i8.i32(i8* nocapture writeonly, i8, i32, i1 immarg) #1
+
+  ; Function Attrs: nofree norecurse nounwind writeonly
+  define dso_local void @test2(i8* nocapture %X, i8 zeroext %c, i32 %n) local_unnamed_addr #2 {
+  entry:
+    %cmp4 = icmp sgt i32 %n, 0
+    br i1 %cmp4, label %for.body.preheader, label %for.cond.cleanup
+
+  for.body.preheader:                               ; preds = %entry
+    call void @llvm.memset.p0i8.i32(i8* align 1 %X, i8 %c, i32 %n, i1 false)
+    br label %for.cond.cleanup
+
+  for.cond.cleanup:                                 ; preds = %for.body.preheader, %entry
+    ret void
+  }
+
+  attributes #0 = { nofree nounwind willreturn writeonly "frame-pointer"="all" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="cortex-m55" "target-features"="+armv8.1-m.main,+dsp,+fp-armv8d16,+fp-armv8d16sp,+fp16,+fp64,+fullfp16,+hwdiv,+lob,+mve,+mve.fp,+ras,+strict-align,+thumb-mode,+vfp2,+vfp2sp,+vfp3d16,+vfp3d16sp,+vfp4d16,+vfp4d16sp,-aes,-bf16,-cdecp0,-cdecp1,-cdecp2,-cdecp3,-cdecp4,-cdecp5,-cdecp6,-cdecp7,-crc,-crypto,-dotprod,-fp16fml,-hwdiv-arm,-i8mm,-sb,-sha2" }
+  attributes #1 = { argmemonly nofree nosync nounwind willreturn writeonly "target-cpu"="cortex-m55" }
+  attributes #2 = { nofree norecurse nounwind writeonly "frame-pointer"="all" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="cortex-m55" "target-features"="+armv8.1-m.main,+dsp,+fp-armv8d16,+fp-armv8d16sp,+fp16,+fp64,+fullfp16,+hwdiv,+lob,+mve,+mve.fp,+ras,+strict-align,+thumb-mode,+vfp2,+vfp2sp,+vfp3d16,+vfp3d16sp,+vfp4d16,+vfp4d16sp,-aes,-bf16,-cdecp0,-cdecp1,-cdecp2,-cdecp3,-cdecp4,-cdecp5,-cdecp6,-cdecp7,-crc,-crypto,-dotprod,-fp16fml,-hwdiv-arm,-i8mm,-sb,-sha2" }
+
+  !llvm.module.flags = !{!0, !1}
+  !llvm.ident = !{!2}
+
+  !0 = !{i32 1, !"wchar_size", i32 4}
+  !1 = !{i32 1, !"min_enum_size", i32 4}
+  !2 = !{!"clang version 13.0.0 (https://github.com/llvm/llvm-project.git 2a2720a2dec4ad4fdc7ae58939448e51824a12c4)"}
+
+...
+---
+name:            test1
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    liveins: $r0, $r1, $r2
+
+    ; CHECK-LABEL: name: test1
+    ; CHECK: liveins: $r0, $r1, $r2
+    ; CHECK: [[COPY:%[0-9]+]]:rgpr = COPY $r2
+    ; CHECK: [[COPY1:%[0-9]+]]:rgpr = COPY $r1
+    ; CHECK: [[COPY2:%[0-9]+]]:rgpr = COPY $r0
+    ; CHECK: [[DEF:%[0-9]+]]:mqpr = IMPLICIT_DEF
+    ; CHECK: [[MVE_VDUP8_:%[0-9]+]]:mqpr = MVE_VDUP8 [[COPY1]], 0, $noreg, [[DEF]]
+    ; CHECK: [[t2ADDri:%[0-9]+]]:rgpr = t2ADDri [[COPY]], 15, 14 /* CC::al */, $noreg, $noreg
+    ; CHECK: [[t2BICri:%[0-9]+]]:rgpr = t2BICri killed [[t2ADDri]], 16, 14 /* CC::al */, $noreg, $noreg
+    ; CHECK: [[t2LSRri:%[0-9]+]]:gprlr = t2LSRri killed [[t2BICri]], 4, 14 /* CC::al */, $noreg, $noreg
+    ; CHECK: [[t2WhileLoopSetup:%[0-9]+]]:gprlr = t2WhileLoopSetup killed [[t2LSRri]]
+    ; CHECK: t2WhileLoopStart [[t2WhileLoopSetup]], %bb.2, implicit-def $cpsr
+    ; CHECK: .1:
+    ; CHECK: [[PHI:%[0-9]+]]:rgpr = PHI [[COPY2]], %bb.0, %10, %bb.1
+    ; CHECK: [[PHI1:%[0-9]+]]:gprlr = PHI [[t2WhileLoopSetup]], %bb.0, %12, %bb.1
+    ; CHECK: [[PHI2:%[0-9]+]]:rgpr = PHI [[COPY]], %bb.0, %14, %bb.1
+    ; CHECK: [[MVE_VCTP8_:%[0-9]+]]:vccr = MVE_VCTP8 [[PHI2]], 0, $noreg
+    ; CHECK: [[t2SUBri:%[0-9]+]]:rgpr = t2SUBri [[PHI2]], 16, 14 /* CC::al */, $noreg, $noreg
+    ; CHECK: [[MVE_VSTRBU8_post:%[0-9]+]]:rgpr = MVE_VSTRBU8_post [[MVE_VDUP8_]], [[PHI]], 16, 1, [[MVE_VCTP8_]]
+    ; CHECK: [[t2LoopDec:%[0-9]+]]:gprlr = t2LoopDec [[PHI1]], 1
+    ; CHECK: t2LoopEnd [[t2LoopDec]], %bb.1, implicit-def $cpsr
+    ; CHECK: t2B %bb.2, 14 /* CC::al */, $noreg
+    ; CHECK: .2.entry:
+    ; CHECK: tBX_RET 14 /* CC::al */, $noreg
+    %2:rgpr = COPY $r2
+    %1:rgpr = COPY $r1
+    %0:rgpr = COPY $r0
+    MVE_MEMSETLOOPINST %0, %1, %2
+    tBX_RET 14 /* CC::al */, $noreg
+
+...
+---
+name:            test2
+alignment:       2
+tracksRegLiveness: true
+body:             |
+  ; CHECK-LABEL: name: test2
+  ; CHECK: bb.0.entry:
+  ; CHECK:   successors: %bb.1(0x50000000), %bb.2(0x30000000)
+  ; CHECK:   liveins: $r0, $r1, $r2
+  ; CHECK:   [[COPY:%[0-9]+]]:rgpr = COPY $r2
+  ; CHECK:   [[COPY1:%[0-9]+]]:gpr = COPY $r1
+  ; CHECK:   [[COPY2:%[0-9]+]]:rgpr = COPY $r0
+  ; CHECK:   [[COPY3:%[0-9]+]]:rgpr = COPY [[COPY1]]
+  ; CHECK:   t2CMPri [[COPY]], 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
+  ; CHECK:   t2Bcc %bb.2, 11 /* CC::lt */, $cpsr
+  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
+  ; CHECK: bb.1.for.body.preheader:
+  ; CHECK:   successors: %bb.2(0x80000000), %bb.3(0x00000000)
+  ; CHECK:   [[DEF:%[0-9]+]]:mqpr = IMPLICIT_DEF
+  ; CHECK:   [[MVE_VDUP8_:%[0-9]+]]:mqpr = MVE_VDUP8 [[COPY3]], 0, $noreg, [[DEF]]
+  ; CHECK:   [[t2ADDri:%[0-9]+]]:rgpr = t2ADDri [[COPY]], 15, 14 /* CC::al */, $noreg, $noreg
+  ; CHECK:   [[t2BICri:%[0-9]+]]:rgpr = t2BICri killed [[t2ADDri]], 16, 14 /* CC::al */, $noreg, $noreg
+  ; CHECK:   [[t2LSRri:%[0-9]+]]:gprlr = t2LSRri killed [[t2BICri]], 4, 14 /* CC::al */, $noreg, $noreg
+  ; CHECK:   [[t2WhileLoopSetup:%[0-9]+]]:gprlr = t2WhileLoopSetup killed [[t2LSRri]]
+  ; CHECK:   t2WhileLoopStart [[t2WhileLoopSetup]], %bb.2, implicit-def $cpsr
+  ; CHECK: bb.3:
+  ; CHECK:   [[PHI:%[0-9]+]]:rgpr = PHI [[COPY2]], %bb.1, %11, %bb.3
+  ; CHECK:   [[PHI1:%[0-9]+]]:gprlr = PHI [[t2WhileLoopSetup]], %bb.1, %13, %bb.3
+  ; CHECK:   [[PHI2:%[0-9]+]]:rgpr = PHI [[COPY]], %bb.1, %15, %bb.3
+  ; CHECK:   [[MVE_VCTP8_:%[0-9]+]]:vccr = MVE_VCTP8 [[PHI2]], 0, $noreg
+  ; CHECK:   [[t2SUBri:%[0-9]+]]:rgpr = t2SUBri [[PHI2]], 16, 14 /* CC::al */, $noreg, $noreg
+  ; CHECK:   [[MVE_VSTRBU8_post:%[0-9]+]]:rgpr = MVE_VSTRBU8_post [[MVE_VDUP8_]], [[PHI]], 16, 1, [[MVE_VCTP8_]]
+  ; CHECK:   [[t2LoopDec:%[0-9]+]]:gprlr = t2LoopDec [[PHI1]], 1
+  ; CHECK:   t2LoopEnd [[t2LoopDec]], %bb.3, implicit-def $cpsr
+  ; CHECK:   t2B %bb.2, 14 /* CC::al */, $noreg
+  ; CHECK: bb.2.for.cond.cleanup:
+  ; CHECK:   tBX_RET 14 /* CC::al */, $noreg
+  bb.0.entry:
+    successors: %bb.1(0x50000000), %bb.2(0x30000000)
+    liveins: $r0, $r1, $r2
+
+    %2:rgpr = COPY $r2
+    %1:gpr = COPY $r1
+    %0:rgpr = COPY $r0
+    %3:rgpr = COPY %1
+    t2CMPri %2, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
+    t2Bcc %bb.2, 11 /* CC::lt */, $cpsr
+    t2B %bb.1, 14 /* CC::al */, $noreg
+
+  bb.1.for.body.preheader:
+    MVE_MEMSETLOOPINST %0, %3, %2
+
+  bb.2.for.cond.cleanup:
+    tBX_RET 14 /* CC::al */, $noreg
+
+...