[RISCV] Implement getOptimalMemOpType for memcpy/memset lowering

[reames]

Summary

This patch implements the getOptimalMemOpType callback which is used by the generic mem* lowering in SelectionDAG to pick the widest type used. This patch only changes the behavior when vector instructions are available, as the default is reasonable for scalar.

Without this change, we were emitting either XLEN sized stores (for aligned operations) or byte sized stores (for unaligned operations.) Interestingly, the final codegen was nowhere near as bad as that would seem to imply. Generic load combining and store merging kicked in, and frequently (but not always) produced pretty reasonable vector code.

The primary effects of this change are:

1. Enable the use of vector operations for memset of non-constant. Our generic store merging logic doesn't know how to merge a broadcast store, and thus we were seeing the generic (and awful) byte expansion lowering for unaligned memset.
2. Enable the generic misaligned overlap trick where we write to some of the same bytes twice. The alternative is to either a) use an increasing small sequence of stores for the tail or b) use VL to restrict the vector store. The later is not implemented at this time, so the former is what previously happened. Interestingly, I'm not sure that changing VL (as opposed to the overlap trick) is even obviously profitable here.

One thing I intentionally left out of this was lowering for operations with size less than min-VLENB. I've got some thoughts there, but I'm not sure exactly where we're going to settle yet, and my first attempt seems to require some changes to generic code which seemed worth separating.

[reames]

After offline discussion with @craig.topper, we decided that the overlap at LMUL8 is a bit too aggressive here. Going to rework this patch series.

[kito-cheng]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

17002–17009

Just a minor optimization thought here, we might use i16 or i32 (on RV64) if alignment is OK?

[reames]

Changes:

• Restrict to LMUL1 - This causes a minor regression in bzero lowering due to a store merge limitation I will address separately.
• Handle NoImplicitFloat like other targets.
• Use the largest type allowed by alignment.
• Use ELEN not XLEN for deciding preferred element size. Most of our rv32+vector configurations support e64 and we can exploit that.

[craig.topper]

LGTM

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

17018

Did you mean to repeat this comment block from earlier?

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
 
   /// Returns true if the target allows unaligned memory accesses of the
   /// specified type.
   bool allowsMisalignedMemoryAccesses(
       EVT VT, unsigned AddrSpace = 0, Align Alignment = Align(1),
       MachineMemOperand::Flags Flags = MachineMemOperand::MONone,
       unsigned *Fast = nullptr) const override;
 
+  EVT getOptimalMemOpType(const MemOp &Op,
+                          const AttributeList &FuncAttributes) const override;
+
   bool splitValueIntoRegisterParts(
       SelectionDAG & DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
       unsigned NumParts, MVT PartVT, std::optional<CallingConv::ID> CC)
       const override;
 
   SDValue joinRegisterPartsIntoValue(
       SelectionDAG & DAG, const SDLoc &DL, const SDValue *Parts,
       unsigned NumParts, MVT PartVT, EVT ValueVT,

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

bool RISCVTargetLowering::allowsMisalignedMemoryAccesses(
   // e8 element type access.  Given this, we effectively support all unmasked
   // misaligned accesses.  TODO: Work through the codegen implications of
   // allowing such accesses to be formed, and considered fast.
   if (Fast)
     *Fast = Subtarget.enableUnalignedVectorMem();
   return Subtarget.enableUnalignedVectorMem();
 }
 
+
+EVT RISCVTargetLowering::getOptimalMemOpType(const MemOp &Op,
+                                             const AttributeList &FuncAttributes) const {
+  if (!Subtarget.hasVInstructions())
+    return MVT::Other;
+
+  // Round down to the next smallest LMUL register class.  This is the largest
+  // single load/store we can perform.  TODO: For non-lmul sized operations
+  // smaller than LMUL8, can we do better by using VL?
+  unsigned VLen = Subtarget.getRealMinVLen()/8;
+  unsigned Size = NextPowerOf2(Op.size() & ~(VLen - 1))/2;
+  Size = std::min(Size, 8*VLen);
+  if (Size == 0)
+    // TODO: Figure out short memops.  For the moment, do the default thing
+    // which ends up using scalar sequences.
+    return MVT::Other;
+
+  // Prefer i8 for non-zero memset as it allows us to avoid materializing
+  // a large scalar constant and instead use vmv.v.x/i to do the
+  // broadcast.  For everything else, prefer XLenVT to minimize VL and thus
+  // maximize the chance we can encode the size in the vsetvli.
+  MVT PreferredVT =
+    (Op.isMemset() && !Op.isZeroMemset()) ? MVT::i8 : Subtarget.getXLenVT();
+
+  // Do we have sufficient alignment for our preferred VT?  If not, revert
+  // to byte aligned accesses.
+  if (PreferredVT != MVT::i8 && !Subtarget.enableUnalignedVectorMem()) {
+    if (Op.isFixedDstAlign() &&
+        PreferredVT.getStoreSize() > Op.getDstAlign().value())
+      PreferredVT = MVT::i8;
+    if (Op.isMemcpy() &&
+        PreferredVT.getStoreSize() > Op.getSrcAlign().value())
+      PreferredVT = MVT::i8;
+  }
+  return MVT::getVectorVT(PreferredVT, Size/PreferredVT.getStoreSize());
+}
+
 bool RISCVTargetLowering::splitValueIntoRegisterParts(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
     unsigned NumParts, MVT PartVT, std::optional<CallingConv::ID> CC) const {
   bool IsABIRegCopy = CC.has_value();
   EVT ValueVT = Val.getValueType();
   if (IsABIRegCopy && (ValueVT == MVT::f16 || ValueVT == MVT::bf16) &&
       PartVT == MVT::f32) {
     // Cast the [b]f16 to i16, extend to i32, pad with ones to make a float
     // nan, and cast to f32.
     Val = DAG.getNode(ISD::BITCAST, DL, MVT::i16, Val);
     Val = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i32, Val);
     Val = DAG.getNode(ISD::OR, DL, MVT::i32, Val,
                       DAG.getConstant(0xFFFF0000, DL, MVT::i32));

llvm/test/CodeGen/RISCV/rvv/memcpy-inline.ll

 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dest, ptr %src, i64 16, i1 false)
   ret void
 }
 
 define void @unaligned_memcpy31(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-LABEL: unaligned_memcpy31:
 ; RV32:       # %bb.0: # %entry
-; RV32-NEXT:    lbu a2, 30(a1)
-; RV32-NEXT:    sb a2, 30(a0)
 ; RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
 ; RV32-NEXT:    vle8.v v8, (a1)
 ; RV32-NEXT:    vse8.v v8, (a0)
-; RV32-NEXT:    addi a2, a1, 28
-; RV32-NEXT:    vsetivli zero, 2, e8, mf8, ta, ma
-; RV32-NEXT:    vle8.v v8, (a2)
-; RV32-NEXT:    addi a2, a0, 28
-; RV32-NEXT:    vse8.v v8, (a2)
-; RV32-NEXT:    addi a2, a1, 24
-; RV32-NEXT:    vsetivli zero, 4, e8, mf4, ta, ma
-; RV32-NEXT:    vle8.v v8, (a2)
-; RV32-NEXT:    addi a2, a0, 24
-; RV32-NEXT:    vse8.v v8, (a2)
-; RV32-NEXT:    addi a1, a1, 16
-; RV32-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; RV32-NEXT:    addi a1, a1, 15
 ; RV32-NEXT:    vle8.v v8, (a1)
-; RV32-NEXT:    addi a0, a0, 16
+; RV32-NEXT:    addi a0, a0, 15
 ; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: unaligned_memcpy31:
 ; RV64:       # %bb.0: # %entry
-; RV64-NEXT:    lbu a2, 30(a1)
-; RV64-NEXT:    sb a2, 30(a0)
 ; RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
 ; RV64-NEXT:    vle8.v v8, (a1)
 ; RV64-NEXT:    vse8.v v8, (a0)
-; RV64-NEXT:    addi a2, a1, 28
-; RV64-NEXT:    vsetivli zero, 2, e8, mf8, ta, ma
-; RV64-NEXT:    vle8.v v8, (a2)
-; RV64-NEXT:    addi a2, a0, 28
-; RV64-NEXT:    vse8.v v8, (a2)
-; RV64-NEXT:    addi a2, a1, 24
-; RV64-NEXT:    vsetivli zero, 4, e8, mf4, ta, ma
-; RV64-NEXT:    vle8.v v8, (a2)
-; RV64-NEXT:    addi a2, a0, 24
-; RV64-NEXT:    vse8.v v8, (a2)
-; RV64-NEXT:    addi a1, a1, 16
-; RV64-NEXT:    vsetivli zero, 8, e8, mf2, ta, ma
+; RV64-NEXT:    addi a1, a1, 15
 ; RV64-NEXT:    vle8.v v8, (a1)
-; RV64-NEXT:    addi a0, a0, 16
+; RV64-NEXT:    addi a0, a0, 15
 ; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: unaligned_memcpy31:
 ; RV32-FAST:       # %bb.0: # %entry
-; RV32-FAST-NEXT:    lw a2, 27(a1)
-; RV32-FAST-NEXT:    sw a2, 27(a0)
-; RV32-FAST-NEXT:    lw a2, 24(a1)
-; RV32-FAST-NEXT:    sw a2, 24(a0)
 ; RV32-FAST-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
-; RV32-FAST-NEXT:    addi a1, a1, 16
-; RV32-FAST-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; RV32-FAST-NEXT:    addi a1, a1, 15
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
-; RV32-FAST-NEXT:    addi a0, a0, 16
+; RV32-FAST-NEXT:    addi a0, a0, 15
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: unaligned_memcpy31:
 ; RV64-FAST:       # %bb.0: # %entry
-; RV64-FAST-NEXT:    ld a2, 23(a1)
-; RV64-FAST-NEXT:    sd a2, 23(a0)
-; RV64-FAST-NEXT:    ld a2, 16(a1)
-; RV64-FAST-NEXT:    sd a2, 16(a0)
 ; RV64-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
+; RV64-FAST-NEXT:    addi a1, a1, 15
+; RV64-FAST-NEXT:    vle64.v v8, (a1)
+; RV64-FAST-NEXT:    addi a0, a0, 15
+; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dest, ptr %src, i64 31, i1 false)
   ret void
 }
 
 define void @unaligned_memcpy32(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-LABEL: unaligned_memcpy32:

 
 define void @unaligned_memcpy96(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-LABEL: unaligned_memcpy96:
 ; RV32:       # %bb.0: # %entry
 ; RV32-NEXT:    li a2, 64
 ; RV32-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; RV32-NEXT:    vle8.v v8, (a1)
 ; RV32-NEXT:    vse8.v v8, (a0)
-; RV32-NEXT:    addi a1, a1, 64
-; RV32-NEXT:    li a2, 32
-; RV32-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV32-NEXT:    addi a1, a1, 32
 ; RV32-NEXT:    vle8.v v8, (a1)
-; RV32-NEXT:    addi a0, a0, 64
+; RV32-NEXT:    addi a0, a0, 32
 ; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: unaligned_memcpy96:
 ; RV64:       # %bb.0: # %entry
 ; RV64-NEXT:    li a2, 64
 ; RV64-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
 ; RV64-NEXT:    vle8.v v8, (a1)
 ; RV64-NEXT:    vse8.v v8, (a0)
-; RV64-NEXT:    addi a1, a1, 64
-; RV64-NEXT:    li a2, 32
-; RV64-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV64-NEXT:    addi a1, a1, 32
 ; RV64-NEXT:    vle8.v v8, (a1)
-; RV64-NEXT:    addi a0, a0, 64
+; RV64-NEXT:    addi a0, a0, 32
 ; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: unaligned_memcpy96:
 ; RV32-FAST:       # %bb.0: # %entry
 ; RV32-FAST-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
-; RV32-FAST-NEXT:    addi a1, a1, 64
-; RV32-FAST-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; RV32-FAST-NEXT:    addi a1, a1, 32
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
-; RV32-FAST-NEXT:    addi a0, a0, 64
+; RV32-FAST-NEXT:    addi a0, a0, 32
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: unaligned_memcpy96:
 ; RV64-FAST:       # %bb.0: # %entry
 ; RV64-FAST-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
-; RV64-FAST-NEXT:    addi a1, a1, 64
-; RV64-FAST-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; RV64-FAST-NEXT:    addi a1, a1, 32
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
-; RV64-FAST-NEXT:    addi a0, a0, 64
+; RV64-FAST-NEXT:    addi a0, a0, 32
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dest, ptr %src, i64 96, i1 false)
   ret void
 }
 
 define void @unaligned_memcpy128(ptr nocapture %dest, ptr %src) nounwind {

 
 define void @unaligned_memcpy196(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-LABEL: unaligned_memcpy196:
 ; RV32:       # %bb.0: # %entry
 ; RV32-NEXT:    li a2, 128
 ; RV32-NEXT:    vsetvli zero, a2, e8, m8, ta, ma
 ; RV32-NEXT:    vle8.v v8, (a1)
 ; RV32-NEXT:    vse8.v v8, (a0)
-; RV32-NEXT:    addi a2, a1, 128
-; RV32-NEXT:    li a3, 64
-; RV32-NEXT:    vsetvli zero, a3, e8, m4, ta, ma
-; RV32-NEXT:    vle8.v v8, (a2)
-; RV32-NEXT:    addi a2, a0, 128
-; RV32-NEXT:    vse8.v v8, (a2)
-; RV32-NEXT:    addi a1, a1, 192
-; RV32-NEXT:    vsetivli zero, 4, e8, mf4, ta, ma
+; RV32-NEXT:    addi a1, a1, 68
 ; RV32-NEXT:    vle8.v v8, (a1)
-; RV32-NEXT:    addi a0, a0, 192
+; RV32-NEXT:    addi a0, a0, 68
 ; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: unaligned_memcpy196:
 ; RV64:       # %bb.0: # %entry
 ; RV64-NEXT:    li a2, 128
 ; RV64-NEXT:    vsetvli zero, a2, e8, m8, ta, ma
 ; RV64-NEXT:    vle8.v v8, (a1)
 ; RV64-NEXT:    vse8.v v8, (a0)
-; RV64-NEXT:    addi a2, a1, 128
-; RV64-NEXT:    li a3, 64
-; RV64-NEXT:    vsetvli zero, a3, e8, m4, ta, ma
-; RV64-NEXT:    vle8.v v8, (a2)
-; RV64-NEXT:    addi a2, a0, 128
-; RV64-NEXT:    vse8.v v8, (a2)
-; RV64-NEXT:    addi a1, a1, 192
-; RV64-NEXT:    vsetivli zero, 4, e8, mf4, ta, ma
+; RV64-NEXT:    addi a1, a1, 68
 ; RV64-NEXT:    vle8.v v8, (a1)
-; RV64-NEXT:    addi a0, a0, 192
+; RV64-NEXT:    addi a0, a0, 68
 ; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: unaligned_memcpy196:
 ; RV32-FAST:       # %bb.0: # %entry
 ; RV32-FAST-NEXT:    li a2, 32
 ; RV32-FAST-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
-; RV32-FAST-NEXT:    lw a2, 192(a1)
-; RV32-FAST-NEXT:    sw a2, 192(a0)
-; RV32-FAST-NEXT:    addi a1, a1, 128
-; RV32-FAST-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; RV32-FAST-NEXT:    addi a1, a1, 68
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
-; RV32-FAST-NEXT:    addi a0, a0, 128
+; RV32-FAST-NEXT:    addi a0, a0, 68
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: unaligned_memcpy196:
 ; RV64-FAST:       # %bb.0: # %entry
-; RV64-FAST-NEXT:    lw a2, 192(a1)
-; RV64-FAST-NEXT:    sw a2, 192(a0)
 ; RV64-FAST-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
-; RV64-FAST-NEXT:    addi a1, a1, 128
-; RV64-FAST-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; RV64-FAST-NEXT:    addi a1, a1, 68
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
-; RV64-FAST-NEXT:    addi a0, a0, 128
+; RV64-FAST-NEXT:    addi a0, a0, 68
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dest, ptr %src, i64 196, i1 false)
   ret void
 }
 
 define void @unaligned_memcpy256(ptr nocapture %dest, ptr %src) nounwind {

 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr align 8 %dest, ptr align 8 %src, i64 16, i1 false)
   ret void
 }
 
 define void @aligned_memcpy31(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-LABEL: aligned_memcpy31:
 ; RV32:       # %bb.0: # %entry
-; RV32-NEXT:    lbu a2, 30(a1)
-; RV32-NEXT:    sb a2, 30(a0)
-; RV32-NEXT:    lh a2, 28(a1)
-; RV32-NEXT:    sh a2, 28(a0)
-; RV32-NEXT:    lw a2, 24(a1)
-; RV32-NEXT:    sw a2, 24(a0)
 ; RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-NEXT:    vle32.v v8, (a1)
 ; RV32-NEXT:    vse32.v v8, (a0)
-; RV32-NEXT:    addi a1, a1, 16
-; RV32-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
-; RV32-NEXT:    vle32.v v8, (a1)
-; RV32-NEXT:    addi a0, a0, 16
-; RV32-NEXT:    vse32.v v8, (a0)
+; RV32-NEXT:    addi a1, a1, 15
+; RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV32-NEXT:    vle8.v v8, (a1)
+; RV32-NEXT:    addi a0, a0, 15
+; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: aligned_memcpy31:
 ; RV64:       # %bb.0: # %entry
-; RV64-NEXT:    lbu a2, 30(a1)
-; RV64-NEXT:    sb a2, 30(a0)
-; RV64-NEXT:    lh a2, 28(a1)
-; RV64-NEXT:    sh a2, 28(a0)
-; RV64-NEXT:    lw a2, 24(a1)
-; RV64-NEXT:    sw a2, 24(a0)
-; RV64-NEXT:    ld a2, 16(a1)
-; RV64-NEXT:    sd a2, 16(a0)
 ; RV64-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV64-NEXT:    vle64.v v8, (a1)
 ; RV64-NEXT:    vse64.v v8, (a0)
+; RV64-NEXT:    addi a1, a1, 15
+; RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-NEXT:    vle8.v v8, (a1)
+; RV64-NEXT:    addi a0, a0, 15
+; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: aligned_memcpy31:
 ; RV32-FAST:       # %bb.0: # %entry
-; RV32-FAST-NEXT:    lw a2, 27(a1)
-; RV32-FAST-NEXT:    sw a2, 27(a0)
-; RV32-FAST-NEXT:    lw a2, 24(a1)
-; RV32-FAST-NEXT:    sw a2, 24(a0)
 ; RV32-FAST-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
-; RV32-FAST-NEXT:    addi a1, a1, 16
-; RV32-FAST-NEXT:    vsetivli zero, 2, e32, mf2, ta, ma
+; RV32-FAST-NEXT:    addi a1, a1, 15
 ; RV32-FAST-NEXT:    vle32.v v8, (a1)
-; RV32-FAST-NEXT:    addi a0, a0, 16
+; RV32-FAST-NEXT:    addi a0, a0, 15
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: aligned_memcpy31:
 ; RV64-FAST:       # %bb.0: # %entry
-; RV64-FAST-NEXT:    ld a2, 23(a1)
-; RV64-FAST-NEXT:    sd a2, 23(a0)
-; RV64-FAST-NEXT:    ld a2, 16(a1)
-; RV64-FAST-NEXT:    sd a2, 16(a0)
 ; RV64-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
+; RV64-FAST-NEXT:    addi a1, a1, 15
+; RV64-FAST-NEXT:    vle64.v v8, (a1)
+; RV64-FAST-NEXT:    addi a0, a0, 15
+; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr align 8 %dest, ptr align 8 %src, i64 31, i1 false)
   ret void
 }
 
 define void @aligned_memcpy32(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-BOTH-LABEL: aligned_memcpy32:

 }
 
 define void @aligned_memcpy96(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-BOTH-LABEL: aligned_memcpy96:
 ; RV32-BOTH:       # %bb.0: # %entry
 ; RV32-BOTH-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; RV32-BOTH-NEXT:    vle32.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
-; RV32-BOTH-NEXT:    addi a1, a1, 64
-; RV32-BOTH-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
+; RV32-BOTH-NEXT:    addi a1, a1, 32
 ; RV32-BOTH-NEXT:    vle32.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a0, a0, 64
+; RV32-BOTH-NEXT:    addi a0, a0, 32
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_memcpy96:
 ; RV64-BOTH:       # %bb.0: # %entry
 ; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-BOTH-NEXT:    vle64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
-; RV64-BOTH-NEXT:    addi a1, a1, 64
-; RV64-BOTH-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
+; RV64-BOTH-NEXT:    addi a1, a1, 32
 ; RV64-BOTH-NEXT:    vle64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a0, a0, 64
+; RV64-BOTH-NEXT:    addi a0, a0, 32
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr align 8 %dest, ptr align 8 %src, i64 96, i1 false)
   ret void
 }
 
 define void @aligned_memcpy128(ptr nocapture %dest, ptr %src) nounwind {

 
 define void @aligned_memcpy196(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-BOTH-LABEL: aligned_memcpy196:
 ; RV32-BOTH:       # %bb.0: # %entry
 ; RV32-BOTH-NEXT:    li a2, 32
 ; RV32-BOTH-NEXT:    vsetvli zero, a2, e32, m8, ta, ma
 ; RV32-BOTH-NEXT:    vle32.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
-; RV32-BOTH-NEXT:    lw a2, 192(a1)
-; RV32-BOTH-NEXT:    sw a2, 192(a0)
-; RV32-BOTH-NEXT:    addi a1, a1, 128
-; RV32-BOTH-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
+; RV32-BOTH-NEXT:    addi a1, a1, 68
 ; RV32-BOTH-NEXT:    vle32.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a0, a0, 128
+; RV32-BOTH-NEXT:    addi a0, a0, 68
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
-; RV64-BOTH-LABEL: aligned_memcpy196:
-; RV64-BOTH:       # %bb.0: # %entry
-; RV64-BOTH-NEXT:    lw a2, 192(a1)
-; RV64-BOTH-NEXT:    sw a2, 192(a0)
-; RV64-BOTH-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
-; RV64-BOTH-NEXT:    vle64.v v8, (a1)
-; RV64-BOTH-NEXT:    vse64.v v8, (a0)
-; RV64-BOTH-NEXT:    addi a1, a1, 128
-; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
-; RV64-BOTH-NEXT:    vle64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a0, a0, 128
-; RV64-BOTH-NEXT:    vse64.v v8, (a0)
-; RV64-BOTH-NEXT:    ret
+; RV64-LABEL: aligned_memcpy196:
+; RV64:       # %bb.0: # %entry
+; RV64-NEXT:    addi a2, a1, 68
+; RV64-NEXT:    li a3, 128
+; RV64-NEXT:    vsetvli zero, a3, e8, m8, ta, ma
+; RV64-NEXT:    vle8.v v8, (a2)
+; RV64-NEXT:    addi a2, a0, 68
+; RV64-NEXT:    vse8.v v8, (a2)
+; RV64-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
+; RV64-NEXT:    vle64.v v8, (a1)
+; RV64-NEXT:    vse64.v v8, (a0)
+; RV64-NEXT:    ret
+;
+; RV64-FAST-LABEL: aligned_memcpy196:
+; RV64-FAST:       # %bb.0: # %entry
+; RV64-FAST-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
+; RV64-FAST-NEXT:    vle64.v v8, (a1)
+; RV64-FAST-NEXT:    vse64.v v8, (a0)
+; RV64-FAST-NEXT:    addi a1, a1, 68
+; RV64-FAST-NEXT:    vle64.v v8, (a1)
+; RV64-FAST-NEXT:    addi a0, a0, 68
+; RV64-FAST-NEXT:    vse64.v v8, (a0)
+; RV64-FAST-NEXT:    ret
 entry:
   tail call void @llvm.memcpy.inline.p0.p0.i64(ptr align 8 %dest, ptr align 8 %src, i64 196, i1 false)
   ret void
 }
 
 define void @aligned_memcpy256(ptr nocapture %dest, ptr %src) nounwind {
 ; RV32-BOTH-LABEL: aligned_memcpy256:
 ; RV32-BOTH:       # %bb.0: # %entry

 ; RV32-BOTH:       # %bb.0: # %entry
 ; RV32-BOTH-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-BOTH-NEXT:    vle32.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-LABEL: memcpy16_align4:
 ; RV64:       # %bb.0: # %entry
-; RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
-; RV64-NEXT:    vle32.v v8, (a1)
-; RV64-NEXT:    vse32.v v8, (a0)
+; RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-NEXT:    vle8.v v8, (a1)
+; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: memcpy16_align4:
 ; RV64-FAST:       # %bb.0: # %entry
 ; RV64-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV64-FAST-NEXT:    vle64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret

llvm/test/CodeGen/RISCV/rvv/memset-inline.ll

 ; RV64-FAST-NEXT:    mul a1, a1, a2
 ; RV64-FAST-NEXT:    sd a1, 0(a0)
 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 %value, i64 8, i1 0)
   ret void
 }
 
 define void @memset_16(ptr %a, i8 %value) nounwind {
-; RV32-LABEL: memset_16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    sb a1, 15(a0)
-; RV32-NEXT:    sb a1, 14(a0)
-; RV32-NEXT:    sb a1, 13(a0)
-; RV32-NEXT:    sb a1, 12(a0)
-; RV32-NEXT:    sb a1, 11(a0)
-; RV32-NEXT:    sb a1, 10(a0)
-; RV32-NEXT:    sb a1, 9(a0)
-; RV32-NEXT:    sb a1, 8(a0)
-; RV32-NEXT:    sb a1, 7(a0)
-; RV32-NEXT:    sb a1, 6(a0)
-; RV32-NEXT:    sb a1, 5(a0)
-; RV32-NEXT:    sb a1, 4(a0)
-; RV32-NEXT:    sb a1, 3(a0)
-; RV32-NEXT:    sb a1, 2(a0)
-; RV32-NEXT:    sb a1, 1(a0)
-; RV32-NEXT:    sb a1, 0(a0)
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: memset_16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    sb a1, 15(a0)
-; RV64-NEXT:    sb a1, 14(a0)
-; RV64-NEXT:    sb a1, 13(a0)
-; RV64-NEXT:    sb a1, 12(a0)
-; RV64-NEXT:    sb a1, 11(a0)
-; RV64-NEXT:    sb a1, 10(a0)
-; RV64-NEXT:    sb a1, 9(a0)
-; RV64-NEXT:    sb a1, 8(a0)
-; RV64-NEXT:    sb a1, 7(a0)
-; RV64-NEXT:    sb a1, 6(a0)
-; RV64-NEXT:    sb a1, 5(a0)
-; RV64-NEXT:    sb a1, 4(a0)
-; RV64-NEXT:    sb a1, 3(a0)
-; RV64-NEXT:    sb a1, 2(a0)
-; RV64-NEXT:    sb a1, 1(a0)
-; RV64-NEXT:    sb a1, 0(a0)
-; RV64-NEXT:    ret
-;
-; RV32-FAST-LABEL: memset_16:
-; RV32-FAST:       # %bb.0:
-; RV32-FAST-NEXT:    andi a1, a1, 255
-; RV32-FAST-NEXT:    lui a2, 4112
-; RV32-FAST-NEXT:    addi a2, a2, 257
-; RV32-FAST-NEXT:    mul a1, a1, a2
-; RV32-FAST-NEXT:    sw a1, 12(a0)
-; RV32-FAST-NEXT:    sw a1, 8(a0)
-; RV32-FAST-NEXT:    sw a1, 4(a0)
-; RV32-FAST-NEXT:    sw a1, 0(a0)
-; RV32-FAST-NEXT:    ret
+; RV32-BOTH-LABEL: memset_16:
+; RV32-BOTH:       # %bb.0:
+; RV32-BOTH-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
+; RV32-BOTH-NEXT:    ret
 ;
-; RV64-FAST-LABEL: memset_16:
-; RV64-FAST:       # %bb.0:
-; RV64-FAST-NEXT:    andi a1, a1, 255
-; RV64-FAST-NEXT:    lui a2, 4112
-; RV64-FAST-NEXT:    addiw a2, a2, 257
-; RV64-FAST-NEXT:    slli a3, a2, 32
-; RV64-FAST-NEXT:    add a2, a2, a3
-; RV64-FAST-NEXT:    mul a1, a1, a2
-; RV64-FAST-NEXT:    sd a1, 8(a0)
-; RV64-FAST-NEXT:    sd a1, 0(a0)
-; RV64-FAST-NEXT:    ret
+; RV64-BOTH-LABEL: memset_16:
+; RV64-BOTH:       # %bb.0:
+; RV64-BOTH-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
+; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 %value, i64 16, i1 0)
   ret void
 }
 
 define void @memset_32(ptr %a, i8 %value) nounwind {
-; RV32-LABEL: memset_32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    sb a1, 31(a0)
-; RV32-NEXT:    sb a1, 30(a0)
-; RV32-NEXT:    sb a1, 29(a0)
-; RV32-NEXT:    sb a1, 28(a0)
-; RV32-NEXT:    sb a1, 27(a0)
-; RV32-NEXT:    sb a1, 26(a0)
-; RV32-NEXT:    sb a1, 25(a0)
-; RV32-NEXT:    sb a1, 24(a0)
-; RV32-NEXT:    sb a1, 23(a0)
-; RV32-NEXT:    sb a1, 22(a0)
-; RV32-NEXT:    sb a1, 21(a0)
-; RV32-NEXT:    sb a1, 20(a0)
-; RV32-NEXT:    sb a1, 19(a0)
-; RV32-NEXT:    sb a1, 18(a0)
-; RV32-NEXT:    sb a1, 17(a0)
-; RV32-NEXT:    sb a1, 16(a0)
-; RV32-NEXT:    sb a1, 15(a0)
-; RV32-NEXT:    sb a1, 14(a0)
-; RV32-NEXT:    sb a1, 13(a0)
-; RV32-NEXT:    sb a1, 12(a0)
-; RV32-NEXT:    sb a1, 11(a0)
-; RV32-NEXT:    sb a1, 10(a0)
-; RV32-NEXT:    sb a1, 9(a0)
-; RV32-NEXT:    sb a1, 8(a0)
-; RV32-NEXT:    sb a1, 7(a0)
-; RV32-NEXT:    sb a1, 6(a0)
-; RV32-NEXT:    sb a1, 5(a0)
-; RV32-NEXT:    sb a1, 4(a0)
-; RV32-NEXT:    sb a1, 3(a0)
-; RV32-NEXT:    sb a1, 2(a0)
-; RV32-NEXT:    sb a1, 1(a0)
-; RV32-NEXT:    sb a1, 0(a0)
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: memset_32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    sb a1, 31(a0)
-; RV64-NEXT:    sb a1, 30(a0)
-; RV64-NEXT:    sb a1, 29(a0)
-; RV64-NEXT:    sb a1, 28(a0)
-; RV64-NEXT:    sb a1, 27(a0)
-; RV64-NEXT:    sb a1, 26(a0)
-; RV64-NEXT:    sb a1, 25(a0)
-; RV64-NEXT:    sb a1, 24(a0)
-; RV64-NEXT:    sb a1, 23(a0)
-; RV64-NEXT:    sb a1, 22(a0)
-; RV64-NEXT:    sb a1, 21(a0)
-; RV64-NEXT:    sb a1, 20(a0)
-; RV64-NEXT:    sb a1, 19(a0)
-; RV64-NEXT:    sb a1, 18(a0)
-; RV64-NEXT:    sb a1, 17(a0)
-; RV64-NEXT:    sb a1, 16(a0)
-; RV64-NEXT:    sb a1, 15(a0)
-; RV64-NEXT:    sb a1, 14(a0)
-; RV64-NEXT:    sb a1, 13(a0)
-; RV64-NEXT:    sb a1, 12(a0)
-; RV64-NEXT:    sb a1, 11(a0)
-; RV64-NEXT:    sb a1, 10(a0)
-; RV64-NEXT:    sb a1, 9(a0)
-; RV64-NEXT:    sb a1, 8(a0)
-; RV64-NEXT:    sb a1, 7(a0)
-; RV64-NEXT:    sb a1, 6(a0)
-; RV64-NEXT:    sb a1, 5(a0)
-; RV64-NEXT:    sb a1, 4(a0)
-; RV64-NEXT:    sb a1, 3(a0)
-; RV64-NEXT:    sb a1, 2(a0)
-; RV64-NEXT:    sb a1, 1(a0)
-; RV64-NEXT:    sb a1, 0(a0)
-; RV64-NEXT:    ret
-;
-; RV32-FAST-LABEL: memset_32:
-; RV32-FAST:       # %bb.0:
-; RV32-FAST-NEXT:    andi a1, a1, 255
-; RV32-FAST-NEXT:    lui a2, 4112
-; RV32-FAST-NEXT:    addi a2, a2, 257
-; RV32-FAST-NEXT:    mul a1, a1, a2
-; RV32-FAST-NEXT:    sw a1, 28(a0)
-; RV32-FAST-NEXT:    sw a1, 24(a0)
-; RV32-FAST-NEXT:    sw a1, 20(a0)
-; RV32-FAST-NEXT:    sw a1, 16(a0)
-; RV32-FAST-NEXT:    sw a1, 12(a0)
-; RV32-FAST-NEXT:    sw a1, 8(a0)
-; RV32-FAST-NEXT:    sw a1, 4(a0)
-; RV32-FAST-NEXT:    sw a1, 0(a0)
-; RV32-FAST-NEXT:    ret
+; RV32-BOTH-LABEL: memset_32:
+; RV32-BOTH:       # %bb.0:
+; RV32-BOTH-NEXT:    li a2, 32
+; RV32-BOTH-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
+; RV32-BOTH-NEXT:    ret
 ;
-; RV64-FAST-LABEL: memset_32:
-; RV64-FAST:       # %bb.0:
-; RV64-FAST-NEXT:    andi a1, a1, 255
-; RV64-FAST-NEXT:    lui a2, 4112
-; RV64-FAST-NEXT:    addiw a2, a2, 257
-; RV64-FAST-NEXT:    slli a3, a2, 32
-; RV64-FAST-NEXT:    add a2, a2, a3
-; RV64-FAST-NEXT:    mul a1, a1, a2
-; RV64-FAST-NEXT:    sd a1, 24(a0)
-; RV64-FAST-NEXT:    sd a1, 16(a0)
-; RV64-FAST-NEXT:    sd a1, 8(a0)
-; RV64-FAST-NEXT:    sd a1, 0(a0)
-; RV64-FAST-NEXT:    ret
+; RV64-BOTH-LABEL: memset_32:
+; RV64-BOTH:       # %bb.0:
+; RV64-BOTH-NEXT:    li a2, 32
+; RV64-BOTH-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
+; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 %value, i64 32, i1 0)
   ret void
 }
 
 define void @memset_64(ptr %a, i8 %value) nounwind {
-; RV32-LABEL: memset_64:
-; RV32:       # %bb.0:
-; RV32-NEXT:    sb a1, 63(a0)
-; RV32-NEXT:    sb a1, 62(a0)
-; RV32-NEXT:    sb a1, 61(a0)
-; RV32-NEXT:    sb a1, 60(a0)
-; RV32-NEXT:    sb a1, 59(a0)
-; RV32-NEXT:    sb a1, 58(a0)
-; RV32-NEXT:    sb a1, 57(a0)
-; RV32-NEXT:    sb a1, 56(a0)
-; RV32-NEXT:    sb a1, 55(a0)
-; RV32-NEXT:    sb a1, 54(a0)
-; RV32-NEXT:    sb a1, 53(a0)
-; RV32-NEXT:    sb a1, 52(a0)
-; RV32-NEXT:    sb a1, 51(a0)
-; RV32-NEXT:    sb a1, 50(a0)
-; RV32-NEXT:    sb a1, 49(a0)
-; RV32-NEXT:    sb a1, 48(a0)
-; RV32-NEXT:    sb a1, 47(a0)
-; RV32-NEXT:    sb a1, 46(a0)
-; RV32-NEXT:    sb a1, 45(a0)
-; RV32-NEXT:    sb a1, 44(a0)
-; RV32-NEXT:    sb a1, 43(a0)
-; RV32-NEXT:    sb a1, 42(a0)
-; RV32-NEXT:    sb a1, 41(a0)
-; RV32-NEXT:    sb a1, 40(a0)
-; RV32-NEXT:    sb a1, 39(a0)
-; RV32-NEXT:    sb a1, 38(a0)
-; RV32-NEXT:    sb a1, 37(a0)
-; RV32-NEXT:    sb a1, 36(a0)
-; RV32-NEXT:    sb a1, 35(a0)
-; RV32-NEXT:    sb a1, 34(a0)
-; RV32-NEXT:    sb a1, 33(a0)
-; RV32-NEXT:    sb a1, 32(a0)
-; RV32-NEXT:    sb a1, 31(a0)
-; RV32-NEXT:    sb a1, 30(a0)
-; RV32-NEXT:    sb a1, 29(a0)
-; RV32-NEXT:    sb a1, 28(a0)
-; RV32-NEXT:    sb a1, 27(a0)
-; RV32-NEXT:    sb a1, 26(a0)
-; RV32-NEXT:    sb a1, 25(a0)
-; RV32-NEXT:    sb a1, 24(a0)
-; RV32-NEXT:    sb a1, 23(a0)
-; RV32-NEXT:    sb a1, 22(a0)
-; RV32-NEXT:    sb a1, 21(a0)
-; RV32-NEXT:    sb a1, 20(a0)
-; RV32-NEXT:    sb a1, 19(a0)
-; RV32-NEXT:    sb a1, 18(a0)
-; RV32-NEXT:    sb a1, 17(a0)
-; RV32-NEXT:    sb a1, 16(a0)
-; RV32-NEXT:    sb a1, 15(a0)
-; RV32-NEXT:    sb a1, 14(a0)
-; RV32-NEXT:    sb a1, 13(a0)
-; RV32-NEXT:    sb a1, 12(a0)
-; RV32-NEXT:    sb a1, 11(a0)
-; RV32-NEXT:    sb a1, 10(a0)
-; RV32-NEXT:    sb a1, 9(a0)
-; RV32-NEXT:    sb a1, 8(a0)
-; RV32-NEXT:    sb a1, 7(a0)
-; RV32-NEXT:    sb a1, 6(a0)
-; RV32-NEXT:    sb a1, 5(a0)
-; RV32-NEXT:    sb a1, 4(a0)
-; RV32-NEXT:    sb a1, 3(a0)
-; RV32-NEXT:    sb a1, 2(a0)
-; RV32-NEXT:    sb a1, 1(a0)
-; RV32-NEXT:    sb a1, 0(a0)
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: memset_64:
-; RV64:       # %bb.0:
-; RV64-NEXT:    sb a1, 63(a0)
-; RV64-NEXT:    sb a1, 62(a0)
-; RV64-NEXT:    sb a1, 61(a0)
-; RV64-NEXT:    sb a1, 60(a0)
-; RV64-NEXT:    sb a1, 59(a0)
-; RV64-NEXT:    sb a1, 58(a0)
-; RV64-NEXT:    sb a1, 57(a0)
-; RV64-NEXT:    sb a1, 56(a0)
-; RV64-NEXT:    sb a1, 55(a0)
-; RV64-NEXT:    sb a1, 54(a0)
-; RV64-NEXT:    sb a1, 53(a0)
-; RV64-NEXT:    sb a1, 52(a0)
-; RV64-NEXT:    sb a1, 51(a0)
-; RV64-NEXT:    sb a1, 50(a0)
-; RV64-NEXT:    sb a1, 49(a0)
-; RV64-NEXT:    sb a1, 48(a0)
-; RV64-NEXT:    sb a1, 47(a0)
-; RV64-NEXT:    sb a1, 46(a0)
-; RV64-NEXT:    sb a1, 45(a0)
-; RV64-NEXT:    sb a1, 44(a0)
-; RV64-NEXT:    sb a1, 43(a0)
-; RV64-NEXT:    sb a1, 42(a0)
-; RV64-NEXT:    sb a1, 41(a0)
-; RV64-NEXT:    sb a1, 40(a0)
-; RV64-NEXT:    sb a1, 39(a0)
-; RV64-NEXT:    sb a1, 38(a0)
-; RV64-NEXT:    sb a1, 37(a0)
-; RV64-NEXT:    sb a1, 36(a0)
-; RV64-NEXT:    sb a1, 35(a0)
-; RV64-NEXT:    sb a1, 34(a0)
-; RV64-NEXT:    sb a1, 33(a0)
-; RV64-NEXT:    sb a1, 32(a0)
-; RV64-NEXT:    sb a1, 31(a0)
-; RV64-NEXT:    sb a1, 30(a0)
-; RV64-NEXT:    sb a1, 29(a0)
-; RV64-NEXT:    sb a1, 28(a0)
-; RV64-NEXT:    sb a1, 27(a0)
-; RV64-NEXT:    sb a1, 26(a0)
-; RV64-NEXT:    sb a1, 25(a0)
-; RV64-NEXT:    sb a1, 24(a0)
-; RV64-NEXT:    sb a1, 23(a0)
-; RV64-NEXT:    sb a1, 22(a0)
-; RV64-NEXT:    sb a1, 21(a0)
-; RV64-NEXT:    sb a1, 20(a0)
-; RV64-NEXT:    sb a1, 19(a0)
-; RV64-NEXT:    sb a1, 18(a0)
-; RV64-NEXT:    sb a1, 17(a0)
-; RV64-NEXT:    sb a1, 16(a0)
-; RV64-NEXT:    sb a1, 15(a0)
-; RV64-NEXT:    sb a1, 14(a0)
-; RV64-NEXT:    sb a1, 13(a0)
-; RV64-NEXT:    sb a1, 12(a0)
-; RV64-NEXT:    sb a1, 11(a0)
-; RV64-NEXT:    sb a1, 10(a0)
-; RV64-NEXT:    sb a1, 9(a0)
-; RV64-NEXT:    sb a1, 8(a0)
-; RV64-NEXT:    sb a1, 7(a0)
-; RV64-NEXT:    sb a1, 6(a0)
-; RV64-NEXT:    sb a1, 5(a0)
-; RV64-NEXT:    sb a1, 4(a0)
-; RV64-NEXT:    sb a1, 3(a0)
-; RV64-NEXT:    sb a1, 2(a0)
-; RV64-NEXT:    sb a1, 1(a0)
-; RV64-NEXT:    sb a1, 0(a0)
-; RV64-NEXT:    ret
-;
-; RV32-FAST-LABEL: memset_64:
-; RV32-FAST:       # %bb.0:
-; RV32-FAST-NEXT:    andi a1, a1, 255
-; RV32-FAST-NEXT:    lui a2, 4112
-; RV32-FAST-NEXT:    addi a2, a2, 257
-; RV32-FAST-NEXT:    mul a1, a1, a2
-; RV32-FAST-NEXT:    sw a1, 60(a0)
-; RV32-FAST-NEXT:    sw a1, 56(a0)
-; RV32-FAST-NEXT:    sw a1, 52(a0)
-; RV32-FAST-NEXT:    sw a1, 48(a0)
-; RV32-FAST-NEXT:    sw a1, 44(a0)
-; RV32-FAST-NEXT:    sw a1, 40(a0)
-; RV32-FAST-NEXT:    sw a1, 36(a0)
-; RV32-FAST-NEXT:    sw a1, 32(a0)
-; RV32-FAST-NEXT:    sw a1, 28(a0)
-; RV32-FAST-NEXT:    sw a1, 24(a0)
-; RV32-FAST-NEXT:    sw a1, 20(a0)
-; RV32-FAST-NEXT:    sw a1, 16(a0)
-; RV32-FAST-NEXT:    sw a1, 12(a0)
-; RV32-FAST-NEXT:    sw a1, 8(a0)
-; RV32-FAST-NEXT:    sw a1, 4(a0)
-; RV32-FAST-NEXT:    sw a1, 0(a0)
-; RV32-FAST-NEXT:    ret
+; RV32-BOTH-LABEL: memset_64:
+; RV32-BOTH:       # %bb.0:
+; RV32-BOTH-NEXT:    li a2, 64
+; RV32-BOTH-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
+; RV32-BOTH-NEXT:    ret
 ;
-; RV64-FAST-LABEL: memset_64:
-; RV64-FAST:       # %bb.0:
-; RV64-FAST-NEXT:    andi a1, a1, 255
-; RV64-FAST-NEXT:    lui a2, 4112
-; RV64-FAST-NEXT:    addiw a2, a2, 257
-; RV64-FAST-NEXT:    slli a3, a2, 32
-; RV64-FAST-NEXT:    add a2, a2, a3
-; RV64-FAST-NEXT:    mul a1, a1, a2
-; RV64-FAST-NEXT:    sd a1, 56(a0)
-; RV64-FAST-NEXT:    sd a1, 48(a0)
-; RV64-FAST-NEXT:    sd a1, 40(a0)
-; RV64-FAST-NEXT:    sd a1, 32(a0)
-; RV64-FAST-NEXT:    sd a1, 24(a0)
-; RV64-FAST-NEXT:    sd a1, 16(a0)
-; RV64-FAST-NEXT:    sd a1, 8(a0)
-; RV64-FAST-NEXT:    sd a1, 0(a0)
-; RV64-FAST-NEXT:    ret
+; RV64-BOTH-LABEL: memset_64:
+; RV64-BOTH:       # %bb.0:
+; RV64-BOTH-NEXT:    li a2, 64
+; RV64-BOTH-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
+; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 %value, i64 64, i1 0)
   ret void
 }
 
 ; /////////////////////////////////////////////////////////////////////////////
 
 define void @aligned_memset_2(ptr align 2 %a, i8 %value) nounwind {
 ; RV32-BOTH-LABEL: aligned_memset_2:

 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 8 %a, i8 %value, i64 8, i1 0)
   ret void
 }
 
 define void @aligned_memset_16(ptr align 16 %a, i8 %value) nounwind {
 ; RV32-BOTH-LABEL: aligned_memset_16:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    andi a1, a1, 255
-; RV32-BOTH-NEXT:    lui a2, 4112
-; RV32-BOTH-NEXT:    addi a2, a2, 257
-; RV32-BOTH-NEXT:    mul a1, a1, a2
-; RV32-BOTH-NEXT:    sw a1, 12(a0)
-; RV32-BOTH-NEXT:    sw a1, 8(a0)
-; RV32-BOTH-NEXT:    sw a1, 4(a0)
-; RV32-BOTH-NEXT:    sw a1, 0(a0)
+; RV32-BOTH-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_memset_16:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    andi a1, a1, 255
-; RV64-BOTH-NEXT:    lui a2, 4112
-; RV64-BOTH-NEXT:    addiw a2, a2, 257
-; RV64-BOTH-NEXT:    slli a3, a2, 32
-; RV64-BOTH-NEXT:    add a2, a2, a3
-; RV64-BOTH-NEXT:    mul a1, a1, a2
-; RV64-BOTH-NEXT:    sd a1, 8(a0)
-; RV64-BOTH-NEXT:    sd a1, 0(a0)
+; RV64-BOTH-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 16 %a, i8 %value, i64 16, i1 0)
   ret void
 }
 
 define void @aligned_memset_32(ptr align 32 %a, i8 %value) nounwind {
 ; RV32-BOTH-LABEL: aligned_memset_32:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    andi a1, a1, 255
-; RV32-BOTH-NEXT:    lui a2, 4112
-; RV32-BOTH-NEXT:    addi a2, a2, 257
-; RV32-BOTH-NEXT:    mul a1, a1, a2
-; RV32-BOTH-NEXT:    sw a1, 28(a0)
-; RV32-BOTH-NEXT:    sw a1, 24(a0)
-; RV32-BOTH-NEXT:    sw a1, 20(a0)
-; RV32-BOTH-NEXT:    sw a1, 16(a0)
-; RV32-BOTH-NEXT:    sw a1, 12(a0)
-; RV32-BOTH-NEXT:    sw a1, 8(a0)
-; RV32-BOTH-NEXT:    sw a1, 4(a0)
-; RV32-BOTH-NEXT:    sw a1, 0(a0)
+; RV32-BOTH-NEXT:    li a2, 32
+; RV32-BOTH-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_memset_32:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    andi a1, a1, 255
-; RV64-BOTH-NEXT:    lui a2, 4112
-; RV64-BOTH-NEXT:    addiw a2, a2, 257
-; RV64-BOTH-NEXT:    slli a3, a2, 32
-; RV64-BOTH-NEXT:    add a2, a2, a3
-; RV64-BOTH-NEXT:    mul a1, a1, a2
-; RV64-BOTH-NEXT:    sd a1, 24(a0)
-; RV64-BOTH-NEXT:    sd a1, 16(a0)
-; RV64-BOTH-NEXT:    sd a1, 8(a0)
-; RV64-BOTH-NEXT:    sd a1, 0(a0)
+; RV64-BOTH-NEXT:    li a2, 32
+; RV64-BOTH-NEXT:    vsetvli zero, a2, e8, m2, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 32 %a, i8 %value, i64 32, i1 0)
   ret void
 }
 
 define void @aligned_memset_64(ptr align 64 %a, i8 %value) nounwind {
 ; RV32-BOTH-LABEL: aligned_memset_64:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    andi a1, a1, 255
-; RV32-BOTH-NEXT:    lui a2, 4112
-; RV32-BOTH-NEXT:    addi a2, a2, 257
-; RV32-BOTH-NEXT:    mul a1, a1, a2
-; RV32-BOTH-NEXT:    sw a1, 60(a0)
-; RV32-BOTH-NEXT:    sw a1, 56(a0)
-; RV32-BOTH-NEXT:    sw a1, 52(a0)
-; RV32-BOTH-NEXT:    sw a1, 48(a0)
-; RV32-BOTH-NEXT:    sw a1, 44(a0)
-; RV32-BOTH-NEXT:    sw a1, 40(a0)
-; RV32-BOTH-NEXT:    sw a1, 36(a0)
-; RV32-BOTH-NEXT:    sw a1, 32(a0)
-; RV32-BOTH-NEXT:    sw a1, 28(a0)
-; RV32-BOTH-NEXT:    sw a1, 24(a0)
-; RV32-BOTH-NEXT:    sw a1, 20(a0)
-; RV32-BOTH-NEXT:    sw a1, 16(a0)
-; RV32-BOTH-NEXT:    sw a1, 12(a0)
-; RV32-BOTH-NEXT:    sw a1, 8(a0)
-; RV32-BOTH-NEXT:    sw a1, 4(a0)
-; RV32-BOTH-NEXT:    sw a1, 0(a0)
+; RV32-BOTH-NEXT:    li a2, 64
+; RV32-BOTH-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.x v8, a1
+; RV32-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_memset_64:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    andi a1, a1, 255
-; RV64-BOTH-NEXT:    lui a2, 4112
-; RV64-BOTH-NEXT:    addiw a2, a2, 257
-; RV64-BOTH-NEXT:    slli a3, a2, 32
-; RV64-BOTH-NEXT:    add a2, a2, a3
-; RV64-BOTH-NEXT:    mul a1, a1, a2
-; RV64-BOTH-NEXT:    sd a1, 56(a0)
-; RV64-BOTH-NEXT:    sd a1, 48(a0)
-; RV64-BOTH-NEXT:    sd a1, 40(a0)
-; RV64-BOTH-NEXT:    sd a1, 32(a0)
-; RV64-BOTH-NEXT:    sd a1, 24(a0)
-; RV64-BOTH-NEXT:    sd a1, 16(a0)
-; RV64-BOTH-NEXT:    sd a1, 8(a0)
-; RV64-BOTH-NEXT:    sd a1, 0(a0)
+; RV64-BOTH-NEXT:    li a2, 64
+; RV64-BOTH-NEXT:    vsetvli zero, a2, e8, m4, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.x v8, a1
+; RV64-BOTH-NEXT:    vse8.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 %value, i64 64, i1 0)
   ret void
 }
 
 ; /////////////////////////////////////////////////////////////////////////////
 
 define void @bzero_1(ptr %a) nounwind {

 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 0, i64 8, i1 0)
   ret void
 }
 
 define void @bzero_16(ptr %a) nounwind {
 ; RV32-LABEL: bzero_16:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    sb zero, 15(a0)
-; RV32-NEXT:    sb zero, 14(a0)
-; RV32-NEXT:    sb zero, 13(a0)
-; RV32-NEXT:    sb zero, 12(a0)
-; RV32-NEXT:    sb zero, 11(a0)
-; RV32-NEXT:    sb zero, 10(a0)
-; RV32-NEXT:    sb zero, 9(a0)
-; RV32-NEXT:    sb zero, 8(a0)
-; RV32-NEXT:    sb zero, 7(a0)
-; RV32-NEXT:    sb zero, 6(a0)
-; RV32-NEXT:    sb zero, 5(a0)
-; RV32-NEXT:    sb zero, 4(a0)
-; RV32-NEXT:    sb zero, 3(a0)
-; RV32-NEXT:    sb zero, 2(a0)
-; RV32-NEXT:    sb zero, 1(a0)
-; RV32-NEXT:    sb zero, 0(a0)
+; RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV32-NEXT:    vmv.v.i v8, 0
+; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: bzero_16:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    sb zero, 15(a0)
-; RV64-NEXT:    sb zero, 14(a0)
-; RV64-NEXT:    sb zero, 13(a0)
-; RV64-NEXT:    sb zero, 12(a0)
-; RV64-NEXT:    sb zero, 11(a0)
-; RV64-NEXT:    sb zero, 10(a0)
-; RV64-NEXT:    sb zero, 9(a0)
-; RV64-NEXT:    sb zero, 8(a0)
-; RV64-NEXT:    sb zero, 7(a0)
-; RV64-NEXT:    sb zero, 6(a0)
-; RV64-NEXT:    sb zero, 5(a0)
-; RV64-NEXT:    sb zero, 4(a0)
-; RV64-NEXT:    sb zero, 3(a0)
-; RV64-NEXT:    sb zero, 2(a0)
-; RV64-NEXT:    sb zero, 1(a0)
-; RV64-NEXT:    sb zero, 0(a0)
+; RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-NEXT:    vmv.v.i v8, 0
+; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: bzero_16:
 ; RV32-FAST:       # %bb.0:
 ; RV32-FAST-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-FAST-NEXT:    vmv.v.i v8, 0
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: bzero_16:
 ; RV64-FAST:       # %bb.0:
-; RV64-FAST-NEXT:    sd zero, 8(a0)
-; RV64-FAST-NEXT:    sd zero, 0(a0)
+; RV64-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-FAST-NEXT:    vmv.v.i v8, 0
+; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 0, i64 16, i1 0)
   ret void
 }
 
 define void @bzero_32(ptr %a) nounwind {
 ; RV32-LABEL: bzero_32:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    sb zero, 31(a0)
-; RV32-NEXT:    sb zero, 30(a0)
-; RV32-NEXT:    sb zero, 29(a0)
-; RV32-NEXT:    sb zero, 28(a0)
-; RV32-NEXT:    sb zero, 27(a0)
-; RV32-NEXT:    sb zero, 26(a0)
-; RV32-NEXT:    sb zero, 25(a0)
-; RV32-NEXT:    sb zero, 24(a0)
-; RV32-NEXT:    sb zero, 23(a0)
-; RV32-NEXT:    sb zero, 22(a0)
-; RV32-NEXT:    sb zero, 21(a0)
-; RV32-NEXT:    sb zero, 20(a0)
-; RV32-NEXT:    sb zero, 19(a0)
-; RV32-NEXT:    sb zero, 18(a0)
-; RV32-NEXT:    sb zero, 17(a0)
-; RV32-NEXT:    sb zero, 16(a0)
-; RV32-NEXT:    sb zero, 15(a0)
-; RV32-NEXT:    sb zero, 14(a0)
-; RV32-NEXT:    sb zero, 13(a0)
-; RV32-NEXT:    sb zero, 12(a0)
-; RV32-NEXT:    sb zero, 11(a0)
-; RV32-NEXT:    sb zero, 10(a0)
-; RV32-NEXT:    sb zero, 9(a0)
-; RV32-NEXT:    sb zero, 8(a0)
-; RV32-NEXT:    sb zero, 7(a0)
-; RV32-NEXT:    sb zero, 6(a0)
-; RV32-NEXT:    sb zero, 5(a0)
-; RV32-NEXT:    sb zero, 4(a0)
-; RV32-NEXT:    sb zero, 3(a0)
-; RV32-NEXT:    sb zero, 2(a0)
-; RV32-NEXT:    sb zero, 1(a0)
-; RV32-NEXT:    sb zero, 0(a0)
+; RV32-NEXT:    li a1, 32
+; RV32-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; RV32-NEXT:    vmv.v.i v8, 0
+; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: bzero_32:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    sb zero, 31(a0)
-; RV64-NEXT:    sb zero, 30(a0)
-; RV64-NEXT:    sb zero, 29(a0)
-; RV64-NEXT:    sb zero, 28(a0)
-; RV64-NEXT:    sb zero, 27(a0)
-; RV64-NEXT:    sb zero, 26(a0)
-; RV64-NEXT:    sb zero, 25(a0)
-; RV64-NEXT:    sb zero, 24(a0)
-; RV64-NEXT:    sb zero, 23(a0)
-; RV64-NEXT:    sb zero, 22(a0)
-; RV64-NEXT:    sb zero, 21(a0)
-; RV64-NEXT:    sb zero, 20(a0)
-; RV64-NEXT:    sb zero, 19(a0)
-; RV64-NEXT:    sb zero, 18(a0)
-; RV64-NEXT:    sb zero, 17(a0)
-; RV64-NEXT:    sb zero, 16(a0)
-; RV64-NEXT:    sb zero, 15(a0)
-; RV64-NEXT:    sb zero, 14(a0)
-; RV64-NEXT:    sb zero, 13(a0)
-; RV64-NEXT:    sb zero, 12(a0)
-; RV64-NEXT:    sb zero, 11(a0)
-; RV64-NEXT:    sb zero, 10(a0)
-; RV64-NEXT:    sb zero, 9(a0)
-; RV64-NEXT:    sb zero, 8(a0)
-; RV64-NEXT:    sb zero, 7(a0)
-; RV64-NEXT:    sb zero, 6(a0)
-; RV64-NEXT:    sb zero, 5(a0)
-; RV64-NEXT:    sb zero, 4(a0)
-; RV64-NEXT:    sb zero, 3(a0)
-; RV64-NEXT:    sb zero, 2(a0)
-; RV64-NEXT:    sb zero, 1(a0)
-; RV64-NEXT:    sb zero, 0(a0)
+; RV64-NEXT:    li a1, 32
+; RV64-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; RV64-NEXT:    vmv.v.i v8, 0
+; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: bzero_32:
 ; RV32-FAST:       # %bb.0:
 ; RV32-FAST-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
 ; RV32-FAST-NEXT:    vmv.v.i v8, 0
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: bzero_32:
 ; RV64-FAST:       # %bb.0:
 ; RV64-FAST-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
 ; RV64-FAST-NEXT:    vmv.v.i v8, 0
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 0, i64 32, i1 0)
   ret void
 }
 
 define void @bzero_64(ptr %a) nounwind {
 ; RV32-LABEL: bzero_64:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    sb zero, 63(a0)
-; RV32-NEXT:    sb zero, 62(a0)
-; RV32-NEXT:    sb zero, 61(a0)
-; RV32-NEXT:    sb zero, 60(a0)
-; RV32-NEXT:    sb zero, 59(a0)
-; RV32-NEXT:    sb zero, 58(a0)
-; RV32-NEXT:    sb zero, 57(a0)
-; RV32-NEXT:    sb zero, 56(a0)
-; RV32-NEXT:    sb zero, 55(a0)
-; RV32-NEXT:    sb zero, 54(a0)
-; RV32-NEXT:    sb zero, 53(a0)
-; RV32-NEXT:    sb zero, 52(a0)
-; RV32-NEXT:    sb zero, 51(a0)
-; RV32-NEXT:    sb zero, 50(a0)
-; RV32-NEXT:    sb zero, 49(a0)
-; RV32-NEXT:    sb zero, 48(a0)
-; RV32-NEXT:    sb zero, 47(a0)
-; RV32-NEXT:    sb zero, 46(a0)
-; RV32-NEXT:    sb zero, 45(a0)
-; RV32-NEXT:    sb zero, 44(a0)
-; RV32-NEXT:    sb zero, 43(a0)
-; RV32-NEXT:    sb zero, 42(a0)
-; RV32-NEXT:    sb zero, 41(a0)
-; RV32-NEXT:    sb zero, 40(a0)
-; RV32-NEXT:    sb zero, 39(a0)
-; RV32-NEXT:    sb zero, 38(a0)
-; RV32-NEXT:    sb zero, 37(a0)
-; RV32-NEXT:    sb zero, 36(a0)
-; RV32-NEXT:    sb zero, 35(a0)
-; RV32-NEXT:    sb zero, 34(a0)
-; RV32-NEXT:    sb zero, 33(a0)
-; RV32-NEXT:    sb zero, 32(a0)
-; RV32-NEXT:    sb zero, 31(a0)
-; RV32-NEXT:    sb zero, 30(a0)
-; RV32-NEXT:    sb zero, 29(a0)
-; RV32-NEXT:    sb zero, 28(a0)
-; RV32-NEXT:    sb zero, 27(a0)
-; RV32-NEXT:    sb zero, 26(a0)
-; RV32-NEXT:    sb zero, 25(a0)
-; RV32-NEXT:    sb zero, 24(a0)
-; RV32-NEXT:    sb zero, 23(a0)
-; RV32-NEXT:    sb zero, 22(a0)
-; RV32-NEXT:    sb zero, 21(a0)
-; RV32-NEXT:    sb zero, 20(a0)
-; RV32-NEXT:    sb zero, 19(a0)
-; RV32-NEXT:    sb zero, 18(a0)
-; RV32-NEXT:    sb zero, 17(a0)
-; RV32-NEXT:    sb zero, 16(a0)
-; RV32-NEXT:    sb zero, 15(a0)
-; RV32-NEXT:    sb zero, 14(a0)
-; RV32-NEXT:    sb zero, 13(a0)
-; RV32-NEXT:    sb zero, 12(a0)
-; RV32-NEXT:    sb zero, 11(a0)
-; RV32-NEXT:    sb zero, 10(a0)
-; RV32-NEXT:    sb zero, 9(a0)
-; RV32-NEXT:    sb zero, 8(a0)
-; RV32-NEXT:    sb zero, 7(a0)
-; RV32-NEXT:    sb zero, 6(a0)
-; RV32-NEXT:    sb zero, 5(a0)
-; RV32-NEXT:    sb zero, 4(a0)
-; RV32-NEXT:    sb zero, 3(a0)
-; RV32-NEXT:    sb zero, 2(a0)
-; RV32-NEXT:    sb zero, 1(a0)
-; RV32-NEXT:    sb zero, 0(a0)
+; RV32-NEXT:    li a1, 64
+; RV32-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
+; RV32-NEXT:    vmv.v.i v8, 0
+; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: bzero_64:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    sb zero, 63(a0)
-; RV64-NEXT:    sb zero, 62(a0)
-; RV64-NEXT:    sb zero, 61(a0)
-; RV64-NEXT:    sb zero, 60(a0)
-; RV64-NEXT:    sb zero, 59(a0)
-; RV64-NEXT:    sb zero, 58(a0)
-; RV64-NEXT:    sb zero, 57(a0)
-; RV64-NEXT:    sb zero, 56(a0)
-; RV64-NEXT:    sb zero, 55(a0)
-; RV64-NEXT:    sb zero, 54(a0)
-; RV64-NEXT:    sb zero, 53(a0)
-; RV64-NEXT:    sb zero, 52(a0)
-; RV64-NEXT:    sb zero, 51(a0)
-; RV64-NEXT:    sb zero, 50(a0)
-; RV64-NEXT:    sb zero, 49(a0)
-; RV64-NEXT:    sb zero, 48(a0)
-; RV64-NEXT:    sb zero, 47(a0)
-; RV64-NEXT:    sb zero, 46(a0)
-; RV64-NEXT:    sb zero, 45(a0)
-; RV64-NEXT:    sb zero, 44(a0)
-; RV64-NEXT:    sb zero, 43(a0)
-; RV64-NEXT:    sb zero, 42(a0)
-; RV64-NEXT:    sb zero, 41(a0)
-; RV64-NEXT:    sb zero, 40(a0)
-; RV64-NEXT:    sb zero, 39(a0)
-; RV64-NEXT:    sb zero, 38(a0)
-; RV64-NEXT:    sb zero, 37(a0)
-; RV64-NEXT:    sb zero, 36(a0)
-; RV64-NEXT:    sb zero, 35(a0)
-; RV64-NEXT:    sb zero, 34(a0)
-; RV64-NEXT:    sb zero, 33(a0)
-; RV64-NEXT:    sb zero, 32(a0)
-; RV64-NEXT:    sb zero, 31(a0)
-; RV64-NEXT:    sb zero, 30(a0)
-; RV64-NEXT:    sb zero, 29(a0)
-; RV64-NEXT:    sb zero, 28(a0)
-; RV64-NEXT:    sb zero, 27(a0)
-; RV64-NEXT:    sb zero, 26(a0)
-; RV64-NEXT:    sb zero, 25(a0)
-; RV64-NEXT:    sb zero, 24(a0)
-; RV64-NEXT:    sb zero, 23(a0)
-; RV64-NEXT:    sb zero, 22(a0)
-; RV64-NEXT:    sb zero, 21(a0)
-; RV64-NEXT:    sb zero, 20(a0)
-; RV64-NEXT:    sb zero, 19(a0)
-; RV64-NEXT:    sb zero, 18(a0)
-; RV64-NEXT:    sb zero, 17(a0)
-; RV64-NEXT:    sb zero, 16(a0)
-; RV64-NEXT:    sb zero, 15(a0)
-; RV64-NEXT:    sb zero, 14(a0)
-; RV64-NEXT:    sb zero, 13(a0)
-; RV64-NEXT:    sb zero, 12(a0)
-; RV64-NEXT:    sb zero, 11(a0)
-; RV64-NEXT:    sb zero, 10(a0)
-; RV64-NEXT:    sb zero, 9(a0)
-; RV64-NEXT:    sb zero, 8(a0)
-; RV64-NEXT:    sb zero, 7(a0)
-; RV64-NEXT:    sb zero, 6(a0)
-; RV64-NEXT:    sb zero, 5(a0)
-; RV64-NEXT:    sb zero, 4(a0)
-; RV64-NEXT:    sb zero, 3(a0)
-; RV64-NEXT:    sb zero, 2(a0)
-; RV64-NEXT:    sb zero, 1(a0)
-; RV64-NEXT:    sb zero, 0(a0)
+; RV64-NEXT:    li a1, 64
+; RV64-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
+; RV64-NEXT:    vmv.v.i v8, 0
+; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: bzero_64:
 ; RV32-FAST:       # %bb.0:
 ; RV32-FAST-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; RV32-FAST-NEXT:    vmv.v.i v8, 0
 ; RV32-FAST-NEXT:    vse32.v v8, (a0)
 ; RV32-FAST-NEXT:    ret

 ; RV32-BOTH:       # %bb.0:
 ; RV32-BOTH-NEXT:    vsetivli zero, 4, e32, m1, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_16:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    sd zero, 8(a0)
-; RV64-BOTH-NEXT:    sd zero, 0(a0)
+; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.i v8, 0
+; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 16 %a, i8 0, i64 16, i1 0)
   ret void
 }
 
 define void @aligned_bzero_32(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_32:
 ; RV32-BOTH:       # %bb.0:

 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 66, i1 0)
   ret void
 }
 
 define void @aligned_bzero_96(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_96:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    addi a1, a0, 64
-; RV32-BOTH-NEXT:    vsetivli zero, 8, e32, m2, ta, ma
-; RV32-BOTH-NEXT:    vmv.v.i v8, 0
-; RV32-BOTH-NEXT:    vse32.v v8, (a1)
+; RV32-BOTH-NEXT:    addi a1, a0, 32
 ; RV32-BOTH-NEXT:    vsetivli zero, 16, e32, m4, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
+; RV32-BOTH-NEXT:    vse32.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse32.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_96:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    addi a1, a0, 64
-; RV64-BOTH-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
-; RV64-BOTH-NEXT:    vmv.v.i v8, 0
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
+; RV64-BOTH-NEXT:    addi a1, a0, 32
 ; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-BOTH-NEXT:    vmv.v.i v8, 0
+; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 96, i1 0)
   ret void
 }
 
 define void @aligned_bzero_128(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_128:

llvm/test/CodeGen/RISCV/rvv/rvv-out-arguments.ll

 ; CHECK-NEXT:    sd ra, 104(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    sd s0, 96(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    sd s1, 88(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    addi s0, sp, 112
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    sub sp, sp, a0
 ; CHECK-NEXT:    sw zero, -36(s0)
-; CHECK-NEXT:    sd zero, -48(s0)
-; CHECK-NEXT:    sd zero, -56(s0)
-; CHECK-NEXT:    vsetivli a0, 4, e32, m8, ta, ma
-; CHECK-NEXT:    sd a0, -64(s0)
-; CHECK-NEXT:    ld a0, -64(s0)
-; CHECK-NEXT:    addi a1, s0, -56
-; CHECK-NEXT:    vsetvli zero, a0, e32, m8, ta, ma
-; CHECK-NEXT:    vle32.v v8, (a1)
+; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vmv.v.i v8, 0
+; CHECK-NEXT:    addi a0, s0, -64
+; CHECK-NEXT:    vse64.v v8, (a0)
+; CHECK-NEXT:    vsetivli a1, 4, e32, m8, ta, ma
+; CHECK-NEXT:    sd a1, -72(s0)
+; CHECK-NEXT:    ld a1, -72(s0)
+; CHECK-NEXT:    vsetvli zero, a1, e32, m8, ta, ma
+; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    csrr s1, vlenb
 ; CHECK-NEXT:    slli s1, s1, 3
 ; CHECK-NEXT:    sub s1, s0, s1
 ; CHECK-NEXT:    addi s1, s1, -112
 ; CHECK-NEXT:    vs8r.v v8, (s1)
 ; CHECK-NEXT:    li a0, 1
-; CHECK-NEXT:    sw a0, -68(s0)
-; CHECK-NEXT:    sw a0, -72(s0)
 ; CHECK-NEXT:    sw a0, -76(s0)
 ; CHECK-NEXT:    sw a0, -80(s0)
 ; CHECK-NEXT:    sw a0, -84(s0)
 ; CHECK-NEXT:    sw a0, -88(s0)
 ; CHECK-NEXT:    sw a0, -92(s0)
 ; CHECK-NEXT:    sw a0, -96(s0)
 ; CHECK-NEXT:    sw a0, -100(s0)
 ; CHECK-NEXT:    sw a0, -104(s0)
-; CHECK-NEXT:    lw a0, -68(s0)
-; CHECK-NEXT:    lw a1, -72(s0)
+; CHECK-NEXT:    sw a0, -108(s0)
+; CHECK-NEXT:    sw a0, -112(s0)
+; CHECK-NEXT:    lw a0, -76(s0)
+; CHECK-NEXT:    lw a1, -80(s0)
 ; CHECK-NEXT:    vl8re32.v v8, (s1)
-; CHECK-NEXT:    lw a2, -76(s0)
-; CHECK-NEXT:    lw a3, -80(s0)
-; CHECK-NEXT:    lw a4, -84(s0)
-; CHECK-NEXT:    lw a5, -88(s0)
-; CHECK-NEXT:    lw a6, -92(s0)
-; CHECK-NEXT:    lw a7, -96(s0)
-; CHECK-NEXT:    lw t0, -100(s0)
-; CHECK-NEXT:    lw t1, -104(s0)
+; CHECK-NEXT:    lw a2, -84(s0)
+; CHECK-NEXT:    lw a3, -88(s0)
+; CHECK-NEXT:    lw a4, -92(s0)
+; CHECK-NEXT:    lw a5, -96(s0)
+; CHECK-NEXT:    lw a6, -100(s0)
+; CHECK-NEXT:    lw a7, -104(s0)
+; CHECK-NEXT:    lw t0, -108(s0)
+; CHECK-NEXT:    lw t1, -112(s0)
 ; CHECK-NEXT:    addi sp, sp, -16
 ; CHECK-NEXT:    sd t1, 8(sp)
 ; CHECK-NEXT:    sd t0, 0(sp)
 ; CHECK-NEXT:    call lots_args
 ; CHECK-NEXT:    addi sp, sp, 16
-; CHECK-NEXT:    lw a0, -68(s0)
-; CHECK-NEXT:    lw a1, -72(s0)
+; CHECK-NEXT:    lw a0, -76(s0)
+; CHECK-NEXT:    lw a1, -80(s0)
 ; CHECK-NEXT:    vl8re32.v v8, (s1)
-; CHECK-NEXT:    lw a2, -76(s0)
-; CHECK-NEXT:    lw a3, -80(s0)
-; CHECK-NEXT:    lw a4, -84(s0)
-; CHECK-NEXT:    lw a5, -88(s0)
-; CHECK-NEXT:    lw a6, -92(s0)
-; CHECK-NEXT:    lw a7, -96(s0)
-; CHECK-NEXT:    lw t0, -100(s0)
-; CHECK-NEXT:    lw t1, -104(s0)
+; CHECK-NEXT:    lw a2, -84(s0)
+; CHECK-NEXT:    lw a3, -88(s0)
+; CHECK-NEXT:    lw a4, -92(s0)
+; CHECK-NEXT:    lw a5, -96(s0)
+; CHECK-NEXT:    lw a6, -100(s0)
+; CHECK-NEXT:    lw a7, -104(s0)
+; CHECK-NEXT:    lw t0, -108(s0)
+; CHECK-NEXT:    lw t1, -112(s0)
 ; CHECK-NEXT:    addi sp, sp, -16
 ; CHECK-NEXT:    sd t1, 8(sp)
 ; CHECK-NEXT:    sd t0, 0(sp)
 ; CHECK-NEXT:    call lots_args
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    li a0, 0
 ; CHECK-NEXT:    addi sp, s0, -112
 ; CHECK-NEXT:    ld ra, 104(sp) # 8-byte Folded Reload

llvm/test/CodeGen/RISCV/rvv/wrong-chain-fixed-load.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv64 -mattr=+v -riscv-v-vector-bits-min=128 < %s \
 ; RUN:   | FileCheck %s
 
 @c = global [7 x i64] [i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7], align 8
 
 define void @do.memmove() nounwind {
 ; CHECK-LABEL: do.memmove:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    lui a0, %hi(c)
 ; CHECK-NEXT:    addi a0, a0, %lo(c)
-; CHECK-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; CHECK-NEXT:    vsetivli zero, 4, e64, m2, ta, ma
 ; CHECK-NEXT:    vle64.v v8, (a0)
-; CHECK-NEXT:    addi a1, a0, 16
-; CHECK-NEXT:    vle64.v v9, (a1)
-; CHECK-NEXT:    addi a1, a0, 8
-; CHECK-NEXT:    vse64.v v8, (a1)
-; CHECK-NEXT:    addi a0, a0, 24
-; CHECK-NEXT:    vse64.v v9, (a0)
+; CHECK-NEXT:    addi a0, a0, 8
+; CHECK-NEXT:    vse64.v v8, (a0)
 ; CHECK-NEXT:    ret
 entry:
   ; this thing is "__builtin_memmove(&c[1], &c[0], sizeof(c[0]) * 4);"
   tail call void @llvm.memmove.p0.p0.i64(
         ptr noundef nonnull align 8 dereferenceable(32) getelementptr inbounds ([7 x i64], ptr @c, i64 0, i64 1),
         ptr noundef nonnull align 8 dereferenceable(32) @c, i64 32, i1 false)
   ret void
 }
 
 ; Function Attrs: argmemonly mustprogress nofree nounwind willreturn
 declare void @llvm.memmove.p0.p0.i64(ptr nocapture writeonly, ptr nocapture readonly, i64, i1 immarg) #1
 
 attributes #1 = { argmemonly mustprogress nofree nounwind willreturn }