[RISCV] Use vmv.v.[v|i] if we know COPY is under the same vl and vtype.

[HsiangKai]

Summary

If we know the source operand of COPY is defined by a vector instruction
with tail agnostic and the same LMUL and there is no vsetvli between
COPY and the define instruction to change the vl and vtype, we could use
vmv.v.v or vmv.v.i to copy vector registers to get better performance than
the whole vector register move instructions.

If the source of COPY is from vmv.v.i, we could use vmv.v.i for the
COPY.

This patch only considers all these instructions within one basic block.

Case 1:

bb.0:
  ...
  VSETVLI          # The first VSETVLI before COPY and VOP.
  ...              # Use this VSETVLI to check LMUL and tail agnostic.
  ...
  vy = VOP va, vb  # Define vy.
  ...              # There is no vsetvli between VOP and COPY.
  vx = COPY vy

Case 2:

bb.0:
  ...
  VSETVLI          # The first VSETVLI before VOP.
  ...              # Use this VSETVLI to check LMUL and tail agnostic.
  ...
  vy = VOP va, vb  # Define vy.
  ...              # There is no vsetvli to change vl between VOP and COPY.
  ...
  VSETVLI          # This vsetvli will not change vl.
  ...
  VSETVLI          # The first VSETVLI before COPY.
  ...              # This VSETVLI does not change vl and vtype.
  ...
  vx = COPY vy

Co-Authored-by: Zakk Chen <[e-mail]>

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

168

What if the defining instruction something that doesn't obey vsetvli for example an earlier COPY that became a whole register move. Or a whole register load which could be a spill reload or just a VLMAX load.

Or a reduction which only writes element 0 regardless of the tail policy. I think we currently force tail agnostic on the vsetvli for those, but I might update it to keep whatever tail policy the previous instruction used.

185

If the SEW and LMul are equal, isn't the vsetvli insertion pass usually going to use x0, x0 for VSetVLIForCopy?

205

Is this equivalent to "return LMul == SetLMul;"?

210

What about this

vsetvli
v0 = vectorop
v1 = vleff
copy v2, v0

The vleff may change VL, but if you replace the copy you need the VL from before the vsetvli. So I think you need to stop if you encountered a vleff.

393

Is this supposed to be in this patch. It's not mentioned in the description.

[HsiangKai]

Consider vleff and whole register load/store.

[HsiangKai]

Consider vsetvli x0, x0, vtype.

[frasercrmck]

My browser's really chugging on this huge patch so my input has to be brief. Maybe we could hide the test changes for now?

Regarding vmv.v.i, is there not already some support for rematerialization (e.g. of immediates)? Or is it non-trivial due to having to manage VSETVLIs?

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

193

Can we use MBBI->modifiesRegister(RISCV::VL)?

398–410

I think we need to add the VL and VTYPE implicit uses?

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

168

Looks like I was wrong about reductions. The behavior changed after 0.9 and our implementation is incorrectly forcing tail agnostic for those.

[HsiangKai]

• Remove most of test updates.
• Add implicit-use arguments for vmv.v.v or vmv.v.i.

[HsiangKai]

In D103510#2798999, @frasercrmck wrote:

My browser's really chugging on this huge patch so my input has to be brief. Maybe we could hide the test changes for now?

Regarding vmv.v.i, is there not already some support for rematerialization (e.g. of immediates)? Or is it non-trivial due to having to manage VSETVLIs?

There seems no support for rematerialization for vmv.v.i. Do you mean to set isReMaterializable = 1, isAsCheapAsAMove = 1 to vmv.v.i pseudo instructions?

[craig.topper]

In D103510#2800595, @HsiangKai wrote:

In D103510#2798999, @frasercrmck wrote:

My browser's really chugging on this huge patch so my input has to be brief. Maybe we could hide the test changes for now?

Regarding vmv.v.i, is there not already some support for rematerialization (e.g. of immediates)? Or is it non-trivial due to having to manage VSETVLIs?

There seems no support for rematerialization for vmv.v.i. Do you mean to set isReMaterializable = 1, isAsCheapAsAMove = 1 to vmv.v.i pseudo instructions?

It still won’t work due to the implicit VL and VTYPE operands from vsetvli insertion. If we move vsetvli insertion later after coalescing, rematerialization would work when avl is a 5 bit immediate. Otherwise AVL being a register also disables rematerialization.

[HsiangKai]

• Add more strict constraint for the conversion.
• Add comments.

[HsiangKai]

It may have more implicit arguments in the end of the producing instruction. Such as,

$v25 = PseudoVNCLIP_WV_M1 killed renamable $v8m2, killed renamable $v10, $noreg, 3, implicit-def dead $vxsat, implicit $vxrm, implicit $vl, implicit $vtype

[HsiangKai]

Add VL, VTYPE directly.

MIB.addReg(RISCV::VL, RegState::Implicit);
MIB.addReg(RISCV::VTYPE, RegState::Implicit);

[HsiangKai]

Update to the latest downstream version.

[HsiangKai]

When producing instruction is a widening reduction instruction with LMUL = 1,
we could not convert whole register move to vmv. The result of reduction
instructions is LMUL = 1 with widening SEW. @khchen has found the bug and help
to fix it. This update includes his fix.

[HsiangKai]

Ping.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

436

Do we have test cases for the tuple classes?

[HsiangKai]

In the current implementation, the size of registers of producing and copying must be the same. For Zvlsseg, these sizes must be different. So, we have no need to consider it. I removed the checking in the update.

[HsiangKai]

Revert the unnecessary change.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

418

Use RISCVVType::decodeVLMUL

425

Write an assert for this?

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

129

Does LMul need to be passed by value?

129

Oops I meant to say does it need to be passed by reference

[craig.topper]

This test case triggers a conversion for a tuple class

dead $x0 = PseudoVSETVLI killed renamable $x11, 64, implicit-def $vl, implicit-def $vtype
renamable $v0_v1 = PseudoVLSEG2E8_V_M1 killed renamable $x10, $noreg, 3, implicit $vl, implicit $vtype
$v2_v3 = COPY $v0_v1                                                         
PseudoVSSEG2E8_V_M1 killed renamable $v2_v3, killed renamable $x12, $noreg, 3, implicit $vl, implicit $vtype
PseudoRET

[HsiangKai]

In D103510#3056401, @craig.topper wrote:

This test case triggers a conversion for a tuple class

dead $x0 = PseudoVSETVLI killed renamable $x11, 64, implicit-def $vl, implicit-def $vtype
renamable $v0_v1 = PseudoVLSEG2E8_V_M1 killed renamable $x10, $noreg, 3, implicit $vl, implicit $vtype
$v2_v3 = COPY $v0_v1                                                         
PseudoVSSEG2E8_V_M1 killed renamable $v2_v3, killed renamable $x12, $noreg, 3, implicit $vl, implicit $vtype
PseudoRET

Thanks. I added the test case and added back the checking for zvlsseg instructions.

[HsiangKai]

Add test cases for zvlsseg register classes and add back the checkings for zvlsseg cases.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

360

Is fractional lmul possible here?

420

Is a fractional lmul even possible?

[HsiangKai]

Add assertion for fractional lmul.

[HsiangKai]

Ping.

[frasercrmck]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

204

Do we need to worry about implicit defs?

[frasercrmck]

Sorry for being sluggish on the review. I'm okay with the idea and the approach but I wouldn't have caught the various issues which seem to have cropped up during review, like those with widening reductions or zvlsseg tuples, etc. I think that's why I'm hesitant to give it an official approval. I'm not sure how to take it forward though. Presumably you're using this pass by default in your downstream fork?

I can say that I have taken this patch and run some of our testing with it. Everything seemed fine, though I realise that's not formal verification.

[HsiangKai]

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

204

The function only handles vector COPY. Is there an instruction to have an implicit define for vector registers?

[HsiangKai]

In D103510#3077278, @frasercrmck wrote:

Sorry for being sluggish on the review. I'm okay with the idea and the approach but I wouldn't have caught the various issues which seem to have cropped up during review, like those with widening reductions or zvlsseg tuples, etc. I think that's why I'm hesitant to give it an official approval. I'm not sure how to take it forward though. Presumably you're using this pass by default in your downstream fork?

I can say that I have taken this patch and run some of our testing with it. Everything seemed fine, though I realise that's not formal verification.

Sorry for replying late. Yes, we enable the conversion by default in our downstream. We have a random test generator to run thousands and thousands tests on it. We found several bugs in my initial version and I have applied these fixes on this patch. You could see these cases from the test case in this patch. How about to turn it off by default on the upstream?

[frasercrmck]

In D103510#3080823, @HsiangKai wrote:

In D103510#3077278, @frasercrmck wrote:

Sorry for being sluggish on the review. I'm okay with the idea and the approach but I wouldn't have caught the various issues which seem to have cropped up during review, like those with widening reductions or zvlsseg tuples, etc. I think that's why I'm hesitant to give it an official approval. I'm not sure how to take it forward though. Presumably you're using this pass by default in your downstream fork?

I can say that I have taken this patch and run some of our testing with it. Everything seemed fine, though I realise that's not formal verification.

Sorry for replying late. Yes, we enable the conversion by default in our downstream. We have a random test generator to run thousands and thousands tests on it. We found several bugs in my initial version and I have applied these fixes on this patch. You could see these cases from the test case in this patch. How about to turn it off by default on the upstream?

No I think that's fine. I've given it another look over and it LGTM aside from my last comment. Thanks for your patience!

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

204

I'm not aware of one, but I still think it's more theoretically sound to check all defs in the instruction. I think we agree that if there was such an implicit def we'd need to handle it? Therefore at lesat an assert that we're not expecting that case.

[HsiangKai]

Check implicit defined operands.

llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h

enum {
   // Pseudos.
   HasVLOpShift = HasSEWOpShift + 1,
   HasVLOpMask = 1 << HasVLOpShift,
 
   // Does this instruction have a vector policy operand. It will be the last
   // explicit operand. Used by RVV Pseudos.
   HasVecPolicyOpShift = HasVLOpShift + 1,
   HasVecPolicyOpMask = 1 << HasVecPolicyOpShift,
+
+  // Is this instruction a vector widening reduction instruction. Used by RVV
+  // Pseudos.
+  IsRVVWideningReductionShift = HasVecPolicyOpShift + 1,
+  IsRVVWideningReductionMask = 1 << IsRVVWideningReductionShift,
 };
 
 // Match with the definitions in RISCVInstrFormatsV.td
 enum VConstraintType {
   NoConstraint = 0,
   VS2Constraint = 0b001,
   VS1Constraint = 0b010,
   VMConstraint = 0b100,

 /// \returns true if there is a VL operand for the instruction.
 static inline bool hasVLOp(uint64_t TSFlags) {
   return TSFlags & HasVLOpMask;
 }
 /// \returns true if there is a vector policy operand for this instruction.
 static inline bool hasVecPolicyOp(uint64_t TSFlags) {
   return TSFlags & HasVecPolicyOpMask;
 }
+/// \returns true if it is a vector widening reduction instruction.
+static inline bool isRVVWideningReduction(uint64_t TSFlags) {
+  return TSFlags & IsRVVWideningReductionMask;
+}
 
 // RISC-V Specific Machine Operand Flags
 enum {
   MO_None = 0,
   MO_CALL = 1,
   MO_PLT = 2,
   MO_LO = 3,
   MO_HI = 4,

llvm/lib/Target/RISCV/RISCVInstrFormats.td

class RVInst<dag outs, dag ins, string opcodestr, string argstr,
   bit HasSEWOp = 0;
   let TSFlags{14} = HasSEWOp;
 
   bit HasVLOp = 0;
   let TSFlags{15} = HasVLOp;
 
   bit HasVecPolicyOp = 0;
   let TSFlags{16} = HasVecPolicyOp;
+
+  bit IsRVVWideningReduction = 0;
+  let TSFlags{17} =  IsRVVWideningReduction;
 }
 
 // Pseudo instructions
 class Pseudo<dag outs, dag ins, list<dag> pattern, string opcodestr = "", string argstr = "">
     : RVInst<outs, ins, opcodestr, argstr, pattern, InstFormatPseudo>,
       Sched<[]> {
   let isPseudo = 1;
   let isCodeGenOnly = 1;

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

 using namespace llvm;
 
 #define GEN_CHECK_COMPRESS_INSTR
 #include "RISCVGenCompressInstEmitter.inc"
 
 #define GET_INSTRINFO_CTOR_DTOR
 #include "RISCVGenInstrInfo.inc"
 
+static cl::opt<bool> PreferWholeRegisterMove(
+    "riscv-prefer-whole-register-move", cl::init(false), cl::Hidden,
+    cl::desc("Prefer whole register move for vector registers."));
+
 namespace llvm {
 namespace RISCVVPseudosTable {
 
 using namespace RISCV;
 
 #define GET_RISCVVPseudosTable_IMPL
 #include "RISCVGenSearchableTables.inc"
 

 
 static bool forwardCopyWillClobberTuple(unsigned DstReg, unsigned SrcReg,
                                         unsigned NumRegs) {
   // We really want the positive remainder mod 32 here, that happens to be
   // easily obtainable with a mask.
   return ((DstReg - SrcReg) & 0x1f) < NumRegs;
 }
 
+static bool isConvertibleToVMV_V_V(const RISCVSubtarget &STI,
+                                   const MachineBasicBlock &MBB,
+                                   MachineBasicBlock::const_iterator MBBI,
+                                   MachineBasicBlock::const_iterator &DefMBBI,
+                                   RISCVII::VLMUL &LMul) {
+  if (PreferWholeRegisterMove)
+    return false;
+
+  assert(MBBI->getOpcode() == TargetOpcode::COPY &&
+         "Unexpected COPY instruction.");
+  Register SrcReg = MBBI->getOperand(1).getReg();
+  const TargetRegisterInfo *TRI = STI.getRegisterInfo();
+
+  bool FoundDef = false;
+  bool FirstVSetVLI = false;
+  unsigned FirstSEW = 0;
+  while (MBBI != MBB.begin()) {
+    --MBBI;
+    if (MBBI->isMetaInstruction())
+      continue;
+
+    if (MBBI->getOpcode() == RISCV::PseudoVSETVLI ||
+        MBBI->getOpcode() == RISCV::PseudoVSETVLIX0 ||
+        MBBI->getOpcode() == RISCV::PseudoVSETIVLI) {
+      // There is a vsetvli between COPY and source define instruction.
+      // vy = def_vop ...  (producing instruction)
+      // ...
+      // vsetvli
+      // ...
+      // vx = COPY vy
+      if (!FoundDef) {
+        if (!FirstVSetVLI) {
+          FirstVSetVLI = true;
+          unsigned FirstVType = MBBI->getOperand(2).getImm();
+          RISCVII::VLMUL FirstLMul = RISCVVType::getVLMUL(FirstVType);
+          FirstSEW = RISCVVType::getSEW(FirstVType);
+          // The first encountered vsetvli must have the same lmul as the
+          // register class of COPY.
+          if (FirstLMul != LMul)
+            return false;
+        }
+        // Only permit `vsetvli x0, x0, vtype` between COPY and the source
+        // define instruction.
+        if (MBBI->getOperand(0).getReg() != RISCV::X0)
+          return false;
+        if (MBBI->getOperand(1).isImm())
+          return false;
+        if (MBBI->getOperand(1).getReg() != RISCV::X0)
+          return false;
+        continue;
+      }
+
+      // MBBI is the first vsetvli before the producing instruction.
+      unsigned VType = MBBI->getOperand(2).getImm();
+      // If there is a vsetvli between COPY and the producing instruction.
+      if (FirstVSetVLI) {
+        // If SEW is different, return false.
+        if (RISCVVType::getSEW(VType) != FirstSEW)
+          return false;
+      }
+
+      // If the vsetvli is tail undisturbed, keep the whole register move.
+      if (!RISCVVType::isTailAgnostic(VType))
+        return false;
+
+      // The checking is conservative. We only have register classes for
+      // LMUL = 1/2/4/8. We should be able to convert vmv1r.v to vmv.v.v
+      // for fractional LMUL operations. However, we could not use the vsetvli
+      // lmul for widening operations. The result of widening operation is
+      // 2 x LMUL.
+      return LMul == RISCVVType::getVLMUL(VType);
+    } else if (MBBI->isInlineAsm() || MBBI->isCall()) {
+      return false;
+    } else if (MBBI->getNumDefs()) {
+      // Check all the instructions which will change VL.
+      // For example, vleff has implicit def VL.
+      if (MBBI->modifiesRegister(RISCV::VL))
+        return false;
+
+      for (const MachineOperand &MO : MBBI->defs()) {
+        if (!FoundDef && TRI->isSubRegisterEq(MO.getReg(), SrcReg)) {
+          // We only permit the source of COPY has the same LMUL as the defined
+          // operand.
+          // There are cases we need to keep the whole register copy if the LMUL
+          // is different.
+          // For example,
+          // $x0 = PseudoVSETIVLI 4, 73   // vsetivli zero, 4, e16,m2,ta,m
+          // $v28m4 = PseudoVWADD_VV_M2 $v26m2, $v8m2
+          // # The COPY may be created by vlmul_trunc intrinsic.
+          // $v26m2 = COPY renamable $v28m2, implicit killed $v28m4
+          //
+          // After widening, the valid value will be 4 x e32 elements. If we
+          // convert the COPY to vmv.v.v, it will only copy 4 x e16 elements.
+          // FIXME: The COPY of subregister of Zvlsseg register will not be able
+          // to convert to vmv.v.[v|i] under the constraint.
+          if (MO.getReg() != SrcReg)
+            return false;
+
+          // In widening reduction instructions with LMUL_1 input vector case,
+          // only checking the LMUL is insufficient due to reduction result is
+          // always LMUL_1.
+          // For example,
+          // $x11 = PseudoVSETIVLI 1, 64 // vsetivli a1, 1, e8, m1, ta, mu
+          // $v8m1 = PseudoVWREDSUM_VS_M1 $v26, $v27
+          // $v26 = COPY killed renamable $v8
+          // After widening, The valid value will be 1 x e16 elements. If we
+          // convert the COPY to vmv.v.v, it will only copy 1 x e8 elements.
+          uint64_t TSFlags = MBBI->getDesc().TSFlags;
+          if (RISCVII::isRVVWideningReduction(TSFlags))
+            return false;
+
+          // Found the definition.
+          FoundDef = true;
+          DefMBBI = MBBI;
+          // If the producing instruction does not depend on vsetvli, do not
+          // convert COPY to vmv.v.v. For example, VL1R_V or PseudoVRELOAD.
+          if (!RISCVII::hasSEWOp(TSFlags))
+            return false;
+          break;
+        }
+      }
+    }
+  }
+
+  return false;
+}
+
 void RISCVInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator MBBI,
                                  const DebugLoc &DL, MCRegister DstReg,
                                  MCRegister SrcReg, bool KillSrc) const {
   if (RISCV::GPRRegClass.contains(DstReg, SrcReg)) {
     BuildMI(MBB, MBBI, DL, get(RISCV::ADDI), DstReg)
         .addReg(SrcReg, getKillRegState(KillSrc))
         .addImm(0);
     return;
   }
 
   // FPR->FPR copies and VR->VR copies.
   unsigned Opc;
   bool IsScalableVector = true;
   unsigned NF = 1;
-  unsigned LMul = 1;
+  RISCVII::VLMUL LMul = RISCVII::LMUL_1;
   unsigned SubRegIdx = RISCV::sub_vrm1_0;
   if (RISCV::FPR16RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::FSGNJ_H;
     IsScalableVector = false;
   } else if (RISCV::FPR32RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::FSGNJ_S;
     IsScalableVector = false;
   } else if (RISCV::FPR64RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::FSGNJ_D;
     IsScalableVector = false;
   } else if (RISCV::VRRegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRM2RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV2R_V;
+    LMul = RISCVII::LMUL_2;
   } else if (RISCV::VRM4RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV4R_V;
+    LMul = RISCVII::LMUL_4;
   } else if (RISCV::VRM8RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV8R_V;
+    LMul = RISCVII::LMUL_8;
   } else if (RISCV::VRN2M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 2;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN2M2RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV2R_V;
     SubRegIdx = RISCV::sub_vrm2_0;
     NF = 2;
-    LMul = 2;
+    LMul = RISCVII::LMUL_2;
   } else if (RISCV::VRN2M4RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV4R_V;
     SubRegIdx = RISCV::sub_vrm4_0;
     NF = 2;
-    LMul = 4;
+    LMul = RISCVII::LMUL_4;
   } else if (RISCV::VRN3M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 3;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN3M2RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV2R_V;
     SubRegIdx = RISCV::sub_vrm2_0;
     NF = 3;
-    LMul = 2;
+    LMul = RISCVII::LMUL_2;
   } else if (RISCV::VRN4M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 4;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN4M2RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV2R_V;
     SubRegIdx = RISCV::sub_vrm2_0;
     NF = 4;
-    LMul = 2;
+    LMul = RISCVII::LMUL_2;
   } else if (RISCV::VRN5M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 5;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN6M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 6;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN7M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 7;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else if (RISCV::VRN8M1RegClass.contains(DstReg, SrcReg)) {
     Opc = RISCV::PseudoVMV1R_V;
     SubRegIdx = RISCV::sub_vrm1_0;
     NF = 8;
-    LMul = 1;
+    LMul = RISCVII::LMUL_1;
   } else {
     llvm_unreachable("Impossible reg-to-reg copy");
   }
 
   if (IsScalableVector) {
+    bool UseVMV_V_V = false;
+    MachineBasicBlock::const_iterator DefMBBI;
+    unsigned DefExplicitOpNum;
+    unsigned VIOpc;
+    if (isConvertibleToVMV_V_V(STI, MBB, MBBI, DefMBBI, LMul)) {
+      UseVMV_V_V = true;
+      DefExplicitOpNum = DefMBBI->getNumExplicitOperands();
+      switch (LMul) {
+      default:
+        llvm_unreachable("Impossible LMUL for vector register copy.");
+      case RISCVII::LMUL_F8:
+        Opc = RISCV::PseudoVMV_V_V_MF8;
+        VIOpc = RISCV::PseudoVMV_V_I_MF8;
+        break;
+      case RISCVII::LMUL_F4:
+        Opc = RISCV::PseudoVMV_V_V_MF4;
+        VIOpc = RISCV::PseudoVMV_V_I_MF4;
+        break;
+      case RISCVII::LMUL_F2:
+        Opc = RISCV::PseudoVMV_V_V_MF2;
+        VIOpc = RISCV::PseudoVMV_V_I_MF2;
+        break;
+      case RISCVII::LMUL_1:
+        Opc = RISCV::PseudoVMV_V_V_M1;
+        VIOpc = RISCV::PseudoVMV_V_I_M1;
+        break;
+      case RISCVII::LMUL_2:
+        Opc = RISCV::PseudoVMV_V_V_M2;
+        VIOpc = RISCV::PseudoVMV_V_I_M2;
+        break;
+      case RISCVII::LMUL_4:
+        Opc = RISCV::PseudoVMV_V_V_M4;
+        VIOpc = RISCV::PseudoVMV_V_I_M4;
+        break;
+      case RISCVII::LMUL_8:
+        Opc = RISCV::PseudoVMV_V_V_M8;
+        VIOpc = RISCV::PseudoVMV_V_I_M8;
+        break;
+      }
+    }
+
+    bool UseVMV_V_I = false;
+    if (UseVMV_V_V && (DefMBBI->getOpcode() == VIOpc)) {
+      UseVMV_V_I = true;
+      Opc = VIOpc;
+    }
+
     if (NF == 1) {
-      BuildMI(MBB, MBBI, DL, get(Opc), DstReg)
-          .addReg(SrcReg, getKillRegState(KillSrc));
+      auto MIB = BuildMI(MBB, MBBI, DL, get(Opc), DstReg);
+      if (UseVMV_V_I)
+        MIB = MIB.add(DefMBBI->getOperand(1));
+      else
+        MIB = MIB.addReg(SrcReg, getKillRegState(KillSrc));
+      if (UseVMV_V_V) {
+        // The last two arguments of vector instructions are
+        // AVL, SEW. We also need to append the implicit-use vl and vtype.
+        MIB.add(DefMBBI->getOperand(DefExplicitOpNum - 2)); // AVL
+        MIB.add(DefMBBI->getOperand(DefExplicitOpNum - 1)); // SEW
+        MIB.addReg(RISCV::VL, RegState::Implicit);
+        MIB.addReg(RISCV::VTYPE, RegState::Implicit);
+      }
     } else {
       const TargetRegisterInfo *TRI = STI.getRegisterInfo();
 
       int I = 0, End = NF, Incr = 1;
       unsigned SrcEncoding = TRI->getEncodingValue(SrcReg);
       unsigned DstEncoding = TRI->getEncodingValue(DstReg);
-      if (forwardCopyWillClobberTuple(DstEncoding, SrcEncoding, NF * LMul)) {
+      unsigned LMulBits = static_cast<unsigned>(LMul);
+      unsigned LMulVal = 1 << (LMulBits < 4 ? LMulBits : 0);
+      if (forwardCopyWillClobberTuple(DstEncoding, SrcEncoding, NF * LMulVal)) {
         I = NF - 1;
         End = -1;
         Incr = -1;
       }
 
+      // For Zvlsseg, UseVMV_V_V is always false.
       for (; I != End; I += Incr) {
         BuildMI(MBB, MBBI, DL, get(Opc), TRI->getSubReg(DstReg, SubRegIdx + I))
             .addReg(TRI->getSubReg(SrcReg, SubRegIdx + I),
                     getKillRegState(KillSrc));
       }
     }
   } else {
     BuildMI(MBB, MBBI, DL, get(Opc), DstReg)
         .addReg(SrcReg, getKillRegState(KillSrc))
         .addReg(SrcReg, getKillRegState(KillSrc));
   }
 }
 
 void RISCVInstrInfo::storeRegToStackSlot(MachineBasicBlock &MBB,
                                          MachineBasicBlock::iterator I,
                                          Register SrcReg, bool IsKill, int FI,
                                          const TargetRegisterClass *RC,
                                          const TargetRegisterInfo *TRI) const {
   DebugLoc DL;

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

 defm PseudoVREDMINU    : VPseudoReductionV_VS;
 defm PseudoVREDMIN     : VPseudoReductionV_VS;
 defm PseudoVREDMAXU    : VPseudoReductionV_VS;
 defm PseudoVREDMAX     : VPseudoReductionV_VS;
 
 //===----------------------------------------------------------------------===//
 // 15.2. Vector Widening Integer Reduction Instructions
 //===----------------------------------------------------------------------===//
+let IsRVVWideningReduction = 1 in {
 defm PseudoVWREDSUMU   : VPseudoReductionV_VS;
 defm PseudoVWREDSUM    : VPseudoReductionV_VS;
+}
 } // Predicates = [HasStdExtV]
 
 let Predicates = [HasStdExtV, HasStdExtF] in {
 //===----------------------------------------------------------------------===//
 // 15.3. Vector Single-Width Floating-Point Reduction Instructions
 //===----------------------------------------------------------------------===//
 defm PseudoVFREDOSUM   : VPseudoReductionV_VS;
 defm PseudoVFREDSUM    : VPseudoReductionV_VS;
 defm PseudoVFREDMIN    : VPseudoReductionV_VS;
 defm PseudoVFREDMAX    : VPseudoReductionV_VS;
 
 //===----------------------------------------------------------------------===//
 // 15.4. Vector Widening Floating-Point Reduction Instructions
 //===----------------------------------------------------------------------===//
+let IsRVVWideningReduction = 1 in {
 defm PseudoVFWREDSUM   : VPseudoReductionV_VS;
 defm PseudoVFWREDOSUM  : VPseudoReductionV_VS;
+}
 
 } // Predicates = [HasStdExtV, HasStdExtF]
 
 //===----------------------------------------------------------------------===//
 // 16. Vector Mask Instructions
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//

llvm/test/CodeGen/RISCV/rvv/calling-conv-fastcc.ll

 ; RV32-NEXT:    addi t4, sp, 16
 ; RV32-NEXT:    addi t6, zero, 8
 ; RV32-NEXT:    csrr a0, vlenb
 ; RV32-NEXT:    slli a0, a0, 4
 ; RV32-NEXT:    add a0, sp, a0
 ; RV32-NEXT:    addi a0, a0, 16
 ; RV32-NEXT:    vs8r.v v8, (a0)
 ; RV32-NEXT:    mv a0, zero
-; RV32-NEXT:    vmv8r.v v16, v8
+; RV32-NEXT:    vmv.v.i v16, 0
 ; RV32-NEXT:    call vector_arg_indirect_stack@plt
 ; RV32-NEXT:    csrr a0, vlenb
 ; RV32-NEXT:    slli a0, a0, 5
 ; RV32-NEXT:    add sp, sp, a0
 ; RV32-NEXT:    lw ra, 28(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    addi sp, sp, 32
 ; RV32-NEXT:    ret
 ;

 ; RV64-NEXT:    addi t4, sp, 24
 ; RV64-NEXT:    addi t6, zero, 8
 ; RV64-NEXT:    csrr a0, vlenb
 ; RV64-NEXT:    slli a0, a0, 4
 ; RV64-NEXT:    add a0, sp, a0
 ; RV64-NEXT:    addi a0, a0, 24
 ; RV64-NEXT:    vs8r.v v8, (a0)
 ; RV64-NEXT:    mv a0, zero
-; RV64-NEXT:    vmv8r.v v16, v8
+; RV64-NEXT:    vmv.v.i v16, 0
 ; RV64-NEXT:    call vector_arg_indirect_stack@plt
 ; RV64-NEXT:    csrr a0, vlenb
 ; RV64-NEXT:    slli a0, a0, 5
 ; RV64-NEXT:    add sp, sp, a0
 ; RV64-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    addi sp, sp, 32
 ; RV64-NEXT:    ret
   %s = call fastcc <vscale x 32 x i32> @vector_arg_indirect_stack(i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, <vscale x 32 x i32> zeroinitializer, <vscale x 32 x i32> zeroinitializer, <vscale x 32 x i32> zeroinitializer, i32 8)
   ret <vscale x 32 x i32> %s
 }

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-calling-conv.ll

 ; LMULMAX8-NEXT:    andi sp, sp, -128
 ; LMULMAX8-NEXT:    addi a2, zero, 32
 ; LMULMAX8-NEXT:    vsetvli zero, a2, e32, m8, ta, mu
 ; LMULMAX8-NEXT:    vle32.v v24, (a0)
 ; LMULMAX8-NEXT:    addi a0, sp, 128
 ; LMULMAX8-NEXT:    addi a2, zero, 42
 ; LMULMAX8-NEXT:    addi a3, sp, 128
 ; LMULMAX8-NEXT:    vse32.v v8, (a3)
-; LMULMAX8-NEXT:    vmv8r.v v8, v24
+; LMULMAX8-NEXT:    vmv.v.v v8, v24
 ; LMULMAX8-NEXT:    call ext3@plt
 ; LMULMAX8-NEXT:    addi sp, s0, -384
 ; LMULMAX8-NEXT:    ld s0, 368(sp) # 8-byte Folded Reload
 ; LMULMAX8-NEXT:    ld ra, 376(sp) # 8-byte Folded Reload
 ; LMULMAX8-NEXT:    addi sp, sp, 384
 ; LMULMAX8-NEXT:    ret
 ;
 ; LMULMAX4-LABEL: ret_v32i32_call_v32i32_v32i32_v32i32_i32:

 ; LMULMAX4-NEXT:    addi a0, a0, 64
 ; LMULMAX4-NEXT:    vle32.v v24, (a0)
 ; LMULMAX4-NEXT:    addi a0, sp, 192
 ; LMULMAX4-NEXT:    vse32.v v12, (a0)
 ; LMULMAX4-NEXT:    addi a0, sp, 128
 ; LMULMAX4-NEXT:    addi a3, zero, 42
 ; LMULMAX4-NEXT:    addi a1, sp, 128
 ; LMULMAX4-NEXT:    vse32.v v8, (a1)
-; LMULMAX4-NEXT:    vmv4r.v v8, v28
-; LMULMAX4-NEXT:    vmv4r.v v12, v24
+; LMULMAX4-NEXT:    vmv.v.v v8, v28
+; LMULMAX4-NEXT:    vmv.v.v v12, v24
 ; LMULMAX4-NEXT:    call ext3@plt
 ; LMULMAX4-NEXT:    addi sp, s0, -384
 ; LMULMAX4-NEXT:    ld s0, 368(sp) # 8-byte Folded Reload
 ; LMULMAX4-NEXT:    ld ra, 376(sp) # 8-byte Folded Reload
 ; LMULMAX4-NEXT:    addi sp, sp, 384
 ; LMULMAX4-NEXT:    ret
 ;
 ; LMULMAX2-LABEL: ret_v32i32_call_v32i32_v32i32_v32i32_i32:

 ; LMULMAX2-NEXT:    addi a0, sp, 192
 ; LMULMAX2-NEXT:    vse32.v v12, (a0)
 ; LMULMAX2-NEXT:    addi a0, sp, 160
 ; LMULMAX2-NEXT:    vse32.v v10, (a0)
 ; LMULMAX2-NEXT:    addi a0, sp, 128
 ; LMULMAX2-NEXT:    addi a5, zero, 42
 ; LMULMAX2-NEXT:    addi a1, sp, 128
 ; LMULMAX2-NEXT:    vse32.v v8, (a1)
-; LMULMAX2-NEXT:    vmv2r.v v8, v26
-; LMULMAX2-NEXT:    vmv2r.v v10, v28
-; LMULMAX2-NEXT:    vmv2r.v v12, v30
-; LMULMAX2-NEXT:    vmv2r.v v14, v24
+; LMULMAX2-NEXT:    vmv.v.v v8, v26
+; LMULMAX2-NEXT:    vmv.v.v v10, v28
+; LMULMAX2-NEXT:    vmv.v.v v12, v30
+; LMULMAX2-NEXT:    vmv.v.v v14, v24
 ; LMULMAX2-NEXT:    call ext3@plt
 ; LMULMAX2-NEXT:    addi sp, s0, -384
 ; LMULMAX2-NEXT:    ld s0, 368(sp) # 8-byte Folded Reload
 ; LMULMAX2-NEXT:    ld ra, 376(sp) # 8-byte Folded Reload
 ; LMULMAX2-NEXT:    addi sp, sp, 384
 ; LMULMAX2-NEXT:    ret
 ;
 ; LMULMAX1-LABEL: ret_v32i32_call_v32i32_v32i32_v32i32_i32:

 ; LMULMAX1-NEXT:    addi a1, sp, 144
 ; LMULMAX1-NEXT:    vse32.v v9, (a1)
 ; LMULMAX1-NEXT:    addi a1, zero, 42
 ; LMULMAX1-NEXT:    sd a1, 8(sp)
 ; LMULMAX1-NEXT:    sd a0, 0(sp)
 ; LMULMAX1-NEXT:    addi a0, sp, 128
 ; LMULMAX1-NEXT:    addi a1, sp, 128
 ; LMULMAX1-NEXT:    vse32.v v8, (a1)
-; LMULMAX1-NEXT:    vmv1r.v v8, v25
-; LMULMAX1-NEXT:    vmv1r.v v9, v26
-; LMULMAX1-NEXT:    vmv1r.v v10, v27
-; LMULMAX1-NEXT:    vmv1r.v v11, v28
-; LMULMAX1-NEXT:    vmv1r.v v12, v29
-; LMULMAX1-NEXT:    vmv1r.v v13, v30
-; LMULMAX1-NEXT:    vmv1r.v v14, v31
-; LMULMAX1-NEXT:    vmv1r.v v15, v24
+; LMULMAX1-NEXT:    vmv.v.v v8, v25
+; LMULMAX1-NEXT:    vmv.v.v v9, v26
+; LMULMAX1-NEXT:    vmv.v.v v10, v27
+; LMULMAX1-NEXT:    vmv.v.v v11, v28
+; LMULMAX1-NEXT:    vmv.v.v v12, v29
+; LMULMAX1-NEXT:    vmv.v.v v13, v30
+; LMULMAX1-NEXT:    vmv.v.v v14, v31
+; LMULMAX1-NEXT:    vmv.v.v v15, v24
 ; LMULMAX1-NEXT:    call ext3@plt
 ; LMULMAX1-NEXT:    addi sp, s0, -384
 ; LMULMAX1-NEXT:    ld s0, 368(sp) # 8-byte Folded Reload
 ; LMULMAX1-NEXT:    ld ra, 376(sp) # 8-byte Folded Reload
 ; LMULMAX1-NEXT:    addi sp, sp, 384
 ; LMULMAX1-NEXT:    ret
   %t = call <32 x i32> @ext3(<32 x i32> %z, <32 x i32> %y, <32 x i32> %x, i32 %w, i32 42)
   ret <32 x i32> %t

 ; LMULMAX2-NEXT:    vse32.v v18, (a0)
 ; LMULMAX2-NEXT:    addi a0, sp, 128
 ; LMULMAX2-NEXT:    addi a1, sp, 128
 ; LMULMAX2-NEXT:    vse32.v v16, (a1)
 ; LMULMAX2-NEXT:    vmv1r.v v9, v8
 ; LMULMAX2-NEXT:    vmv1r.v v10, v8
 ; LMULMAX2-NEXT:    vmv1r.v v11, v8
 ; LMULMAX2-NEXT:    vmv1r.v v12, v8
-; LMULMAX2-NEXT:    vmv2r.v v22, v14
+; LMULMAX2-NEXT:    vmv.v.v v22, v14
 ; LMULMAX2-NEXT:    call split_vector_args@plt
 ; LMULMAX2-NEXT:    addi sp, s0, -256
 ; LMULMAX2-NEXT:    ld s0, 240(sp) # 8-byte Folded Reload
 ; LMULMAX2-NEXT:    ld ra, 248(sp) # 8-byte Folded Reload
 ; LMULMAX2-NEXT:    addi sp, sp, 256
 ; LMULMAX2-NEXT:    ret
 ;
 ; LMULMAX1-LABEL: call_split_vector_args:

 ; LMULMAX1-NEXT:    vse32.v v17, (a0)
 ; LMULMAX1-NEXT:    addi a0, sp, 128
 ; LMULMAX1-NEXT:    addi a1, sp, 128
 ; LMULMAX1-NEXT:    vse32.v v16, (a1)
 ; LMULMAX1-NEXT:    vmv1r.v v9, v8
 ; LMULMAX1-NEXT:    vmv1r.v v10, v8
 ; LMULMAX1-NEXT:    vmv1r.v v11, v8
 ; LMULMAX1-NEXT:    vmv1r.v v12, v8
-; LMULMAX1-NEXT:    vmv1r.v v21, v13
-; LMULMAX1-NEXT:    vmv1r.v v22, v14
-; LMULMAX1-NEXT:    vmv1r.v v23, v15
+; LMULMAX1-NEXT:    vmv.v.v v21, v13
+; LMULMAX1-NEXT:    vmv.v.v v22, v14
+; LMULMAX1-NEXT:    vmv.v.v v23, v15
 ; LMULMAX1-NEXT:    call split_vector_args@plt
 ; LMULMAX1-NEXT:    addi sp, s0, -256
 ; LMULMAX1-NEXT:    ld s0, 240(sp) # 8-byte Folded Reload
 ; LMULMAX1-NEXT:    ld ra, 248(sp) # 8-byte Folded Reload
 ; LMULMAX1-NEXT:    addi sp, sp, 256
 ; LMULMAX1-NEXT:    ret
   %a = load <2 x i32>, <2 x i32>* %pa
   %b = load <32 x i32>, <32 x i32>* %pb

 ; LMULMAX8-NEXT:    addi a2, zero, 2
 ; LMULMAX8-NEXT:    addi a3, zero, 3
 ; LMULMAX8-NEXT:    addi a4, zero, 4
 ; LMULMAX8-NEXT:    addi a5, zero, 5
 ; LMULMAX8-NEXT:    addi a6, zero, 6
 ; LMULMAX8-NEXT:    addi a7, zero, 7
 ; LMULMAX8-NEXT:    sd a0, 128(sp)
 ; LMULMAX8-NEXT:    mv a0, zero
-; LMULMAX8-NEXT:    vmv8r.v v16, v8
+; LMULMAX8-NEXT:    vmv.v.i v16, 0
 ; LMULMAX8-NEXT:    call vector_arg_via_stack@plt
 ; LMULMAX8-NEXT:    ld ra, 136(sp) # 8-byte Folded Reload
 ; LMULMAX8-NEXT:    addi sp, sp, 144
 ; LMULMAX8-NEXT:    ret
 ;
 ; LMULMAX4-LABEL: pass_vector_arg_via_stack:
 ; LMULMAX4:       # %bb.0:
 ; LMULMAX4-NEXT:    addi sp, sp, -144

 ; LMULMAX4-NEXT:    addi a2, zero, 2
 ; LMULMAX4-NEXT:    addi a3, zero, 3
 ; LMULMAX4-NEXT:    addi a4, zero, 4
 ; LMULMAX4-NEXT:    addi a5, zero, 5
 ; LMULMAX4-NEXT:    addi a6, zero, 6
 ; LMULMAX4-NEXT:    addi a7, zero, 7
 ; LMULMAX4-NEXT:    vse32.v v8, (a0)
 ; LMULMAX4-NEXT:    mv a0, zero
-; LMULMAX4-NEXT:    vmv4r.v v12, v8
-; LMULMAX4-NEXT:    vmv4r.v v16, v8
-; LMULMAX4-NEXT:    vmv4r.v v20, v8
+; LMULMAX4-NEXT:    vmv.v.i v12, 0
+; LMULMAX4-NEXT:    vmv.v.i v16, 0
+; LMULMAX4-NEXT:    vmv.v.i v20, 0
 ; LMULMAX4-NEXT:    call vector_arg_via_stack@plt
 ; LMULMAX4-NEXT:    ld ra, 136(sp) # 8-byte Folded Reload
 ; LMULMAX4-NEXT:    addi sp, sp, 144
 ; LMULMAX4-NEXT:    ret
 ;
 ; LMULMAX2-LABEL: pass_vector_arg_via_stack:
 ; LMULMAX2:       # %bb.0:
 ; LMULMAX2-NEXT:    addi sp, sp, -144

 ; LMULMAX2-NEXT:    addi a2, zero, 2
 ; LMULMAX2-NEXT:    addi a3, zero, 3
 ; LMULMAX2-NEXT:    addi a4, zero, 4
 ; LMULMAX2-NEXT:    addi a5, zero, 5
 ; LMULMAX2-NEXT:    addi a6, zero, 6
 ; LMULMAX2-NEXT:    addi a7, zero, 7
 ; LMULMAX2-NEXT:    vse32.v v8, (a0)
 ; LMULMAX2-NEXT:    mv a0, zero
-; LMULMAX2-NEXT:    vmv2r.v v10, v8
-; LMULMAX2-NEXT:    vmv2r.v v12, v8
-; LMULMAX2-NEXT:    vmv2r.v v14, v8
-; LMULMAX2-NEXT:    vmv2r.v v16, v8
-; LMULMAX2-NEXT:    vmv2r.v v18, v8
-; LMULMAX2-NEXT:    vmv2r.v v20, v8
-; LMULMAX2-NEXT:    vmv2r.v v22, v8
+; LMULMAX2-NEXT:    vmv.v.i v10, 0
+; LMULMAX2-NEXT:    vmv.v.i v12, 0
+; LMULMAX2-NEXT:    vmv.v.i v14, 0
+; LMULMAX2-NEXT:    vmv.v.i v16, 0
+; LMULMAX2-NEXT:    vmv.v.i v18, 0
+; LMULMAX2-NEXT:    vmv.v.i v20, 0
+; LMULMAX2-NEXT:    vmv.v.i v22, 0
 ; LMULMAX2-NEXT:    call vector_arg_via_stack@plt
 ; LMULMAX2-NEXT:    ld ra, 136(sp) # 8-byte Folded Reload
 ; LMULMAX2-NEXT:    addi sp, sp, 144
 ; LMULMAX2-NEXT:    ret
 ;
 ; LMULMAX1-LABEL: pass_vector_arg_via_stack:
 ; LMULMAX1:       # %bb.0:
 ; LMULMAX1-NEXT:    addi sp, sp, -144

 ; LMULMAX1-NEXT:    addi a2, zero, 2
 ; LMULMAX1-NEXT:    addi a3, zero, 3
 ; LMULMAX1-NEXT:    addi a4, zero, 4
 ; LMULMAX1-NEXT:    addi a5, zero, 5
 ; LMULMAX1-NEXT:    addi a6, zero, 6
 ; LMULMAX1-NEXT:    addi a7, zero, 7
 ; LMULMAX1-NEXT:    vse32.v v8, (a0)
 ; LMULMAX1-NEXT:    mv a0, zero
-; LMULMAX1-NEXT:    vmv1r.v v9, v8
-; LMULMAX1-NEXT:    vmv1r.v v10, v8
-; LMULMAX1-NEXT:    vmv1r.v v11, v8
-; LMULMAX1-NEXT:    vmv1r.v v12, v8
-; LMULMAX1-NEXT:    vmv1r.v v13, v8
-; LMULMAX1-NEXT:    vmv1r.v v14, v8
-; LMULMAX1-NEXT:    vmv1r.v v15, v8
-; LMULMAX1-NEXT:    vmv1r.v v16, v8
-; LMULMAX1-NEXT:    vmv1r.v v17, v8
-; LMULMAX1-NEXT:    vmv1r.v v18, v8
-; LMULMAX1-NEXT:    vmv1r.v v19, v8
-; LMULMAX1-NEXT:    vmv1r.v v20, v8
-; LMULMAX1-NEXT:    vmv1r.v v21, v8
-; LMULMAX1-NEXT:    vmv1r.v v22, v8
-; LMULMAX1-NEXT:    vmv1r.v v23, v8
+; LMULMAX1-NEXT:    vmv.v.i v9, 0
+; LMULMAX1-NEXT:    vmv.v.i v10, 0
+; LMULMAX1-NEXT:    vmv.v.i v11, 0
+; LMULMAX1-NEXT:    vmv.v.i v12, 0
+; LMULMAX1-NEXT:    vmv.v.i v13, 0
+; LMULMAX1-NEXT:    vmv.v.i v14, 0
+; LMULMAX1-NEXT:    vmv.v.i v15, 0
+; LMULMAX1-NEXT:    vmv.v.i v16, 0
+; LMULMAX1-NEXT:    vmv.v.i v17, 0
+; LMULMAX1-NEXT:    vmv.v.i v18, 0
+; LMULMAX1-NEXT:    vmv.v.i v19, 0
+; LMULMAX1-NEXT:    vmv.v.i v20, 0
+; LMULMAX1-NEXT:    vmv.v.i v21, 0
+; LMULMAX1-NEXT:    vmv.v.i v22, 0
+; LMULMAX1-NEXT:    vmv.v.i v23, 0
 ; LMULMAX1-NEXT:    call vector_arg_via_stack@plt
 ; LMULMAX1-NEXT:    ld ra, 136(sp) # 8-byte Folded Reload
 ; LMULMAX1-NEXT:    addi sp, sp, 144
 ; LMULMAX1-NEXT:    ret
   %s = call <32 x i32> @vector_arg_via_stack(i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, <32 x i32> zeroinitializer, <32 x i32> zeroinitializer, <32 x i32> zeroinitializer, i32 8)
   ret <32 x i32> %s
 }
 

llvm/test/CodeGen/RISCV/rvv/vmv-copy.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -verify-machineinstrs -mtriple riscv64 -run-pass=postrapseudos %s -o - | FileCheck %s
+
+...
+---
+name: copy_different_lmul
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 82 = e32,m4
+    ; CHECK-LABEL: name: copy_different_lmul
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m2 = PseudoVMV2R_V $v28m2
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $v12m2 = COPY $v28m2
+...
+---
+name: copy_convert_to_vmv_v_v
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 82 = e32,m4
+    ; CHECK-LABEL: name: copy_convert_to_vmv_v_v
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV_V_V_M4 $v28m4, $noreg, 5, implicit $vl, implicit $vtype
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_convert_to_vmv_v_i
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14
+    ; 82 = e32,m4
+    ; CHECK-LABEL: name: copy_convert_to_vmv_v_i
+    ; CHECK: liveins: $x14
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVMV_V_I_M4 0, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV_V_I_M4 0, $noreg, 5, implicit $vl, implicit $vtype
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVMV_V_I_M4 0, $noreg, 5, implicit $vl, implicit $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_from_whole_load_store
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 82 = e32,m4
+    ; CHECK-LABEL: name: copy_from_whole_load_store
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = VL4RE32_V $x16
+    ; CHECK-NEXT: $v12m4 = PseudoVMV4R_V $v28m4
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = VL4RE32_V $x16
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_with_vleff
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 82 = e32,m4
+    ; CHECK-LABEL: name: copy_with_vleff
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVMV_V_I_M4 0, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v4m4 = PseudoVLE32FF_V_M4 $x16, $noreg, 5, implicit-def $vl
+    ; CHECK-NEXT: $v12m4 = PseudoVMV4R_V $v28m4
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVMV_V_I_M4 0, $noreg, 5, implicit $vl, implicit $vtype
+    $v4m4 = PseudoVLE32FF_V_M4 $x16, $noreg, 5, implicit-def $vl
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_with_vsetvl_x0_x0_1
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16, $x17, $x18
+    ; 82 = e32,m4
+    ; 73 = e16,m2
+    ; CHECK-LABEL: name: copy_with_vsetvl_x0_x0_1
+    ; CHECK: liveins: $x14, $x16, $x17, $x18
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x17, 73, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x0 = PseudoVSETVLIX0 $x0, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v4m4 = PseudoVLE32_V_M4 killed $x18, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV4R_V $v28m4
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $x15 = PseudoVSETVLI $x17, 73, implicit-def $vl, implicit-def $vtype
+    $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    $x0 = PseudoVSETVLIX0 $x0, 82, implicit-def $vl, implicit-def $vtype
+    $v4m4 = PseudoVLE32_V_M4 killed $x18, $noreg, 5, implicit $vl, implicit $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_with_vsetvl_x0_x0_2
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16, $x17, $x18
+    ; 82 = e32,m4
+    ; 73 = e16,m2
+    ; CHECK-LABEL: name: copy_with_vsetvl_x0_x0_2
+    ; CHECK: liveins: $x14, $x16, $x17, $x18
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x0 = PseudoVSETVLIX0 $x0, 73, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x0 = PseudoVSETVLIX0 $x0, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v4m4 = PseudoVLE32_V_M4 killed $x18, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV_V_V_M4 $v28m4, $noreg, 5, implicit $vl, implicit $vtype
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $x0 = PseudoVSETVLIX0 $x0, 73, implicit-def $vl, implicit-def $vtype
+    $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    $x0 = PseudoVSETVLIX0 $x0, 82, implicit-def $vl, implicit-def $vtype
+    $v4m4 = PseudoVLE32_V_M4 killed $x18, $noreg, 5, implicit $vl, implicit $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_with_vsetvl_x0_x0_3
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16, $x17, $x18
+    ; 82 = e32,m4
+    ; 73 = e16,m2
+    ; CHECK-LABEL: name: copy_with_vsetvl_x0_x0_3
+    ; CHECK: liveins: $x14, $x16, $x17, $x18
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x0 = PseudoVSETVLIX0 $x0, 73, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV4R_V $v28m4
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $x0 = PseudoVSETVLIX0 $x0, 73, implicit-def $vl, implicit-def $vtype
+    $v0m2 = PseudoVLE32_V_M2 $x18, $noreg, 4, implicit $vl, implicit $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_subregister
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x16, $x17
+    ; 73 = e16,m2
+    ; CHECK-LABEL: name: copy_subregister
+    ; CHECK: liveins: $x16, $x17
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETIVLI 4, 73, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v26m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v8m2 = PseudoVLE16_V_M2 killed $x17, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: early-clobber $v28m4 = PseudoVWADD_VV_M2 $v26m2, $v8m2, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v12m2 = PseudoVMV2R_V $v28m2
+    $x15 = PseudoVSETIVLI 4, 73, implicit-def $vl, implicit-def $vtype
+    $v26m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
+    $v8m2 = PseudoVLE16_V_M2 killed $x17, $noreg, 4, implicit $vl, implicit $vtype
+    $v28m4 = PseudoVWADD_VV_M2 $v26m2, $v8m2, $noreg, 4, implicit $vl, implicit $vtype
+    $v12m2 = COPY $v28m2
+...
+---
+name: copy_from_reload
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16, $x17
+    ; 73 = e16,m2
+    ; CHECK-LABEL: name: copy_from_reload
+    ; CHECK: liveins: $x14, $x16, $x17
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 73, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v2m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x12 = PseudoReadVLENB
+    ; CHECK-NEXT: $x12 = SLLI $x12, 1
+    ; CHECK-NEXT: $v2m2_v4m2, $x17 = PseudoVRELOAD2_M2 killed $x17, killed $x12
+    ; CHECK-NEXT: $v12m2 = PseudoVMV2R_V $v2m2
+    $x15 = PseudoVSETVLI $x14, 73, implicit-def $vl, implicit-def $vtype
+    $v2m2 = PseudoVLE16_V_M2 killed $x16, $noreg, 4, implicit $vl, implicit $vtype
+    $x12 = PseudoReadVLENB
+    $x12 = SLLI $x12, 1
+    $v2m2_v4m2, $x17 = PseudoVRELOAD2_M2 killed $x17, killed $x12
+    $v12m2 = COPY $v2m2
+...
+---
+name: copy_with_different_vlmax
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 82 = e32,m4
+    ; 74 = e16,m4
+    ; CHECK-LABEL: name: copy_with_different_vlmax
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $x0 = PseudoVSETVLIX0 $x0, 74, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v12m4 = PseudoVMV4R_V $v28m4
+    $x15 = PseudoVSETVLI $x14, 82, implicit-def $vl, implicit-def $vtype
+    $v28m4 = PseudoVLE32_V_M4 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $x0 = PseudoVSETVLIX0 $x0, 74, implicit-def $vl, implicit-def $vtype
+    $v12m4 = COPY $v28m4
+...
+---
+name: copy_with_widening_reduction
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x10, $v8, $v26, $v27
+    ; CHECK-LABEL: name: copy_with_widening_reduction
+    ; CHECK: liveins: $x10, $v8, $v26, $v27
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x11 = PseudoVSETIVLI 1, 64, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v8 = PseudoVWREDSUM_VS_M1 killed renamable $v8, killed renamable $v26, killed renamable $v27, 1, 3, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v26 = PseudoVMV1R_V killed $v8
+    ; CHECK-NEXT: $x10 = PseudoVSETVLI killed renamable $x10, 75, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v8m8 = PseudoVRELOAD_M8 killed $x10
+    $x11 = PseudoVSETIVLI 1, 64, implicit-def $vl, implicit-def $vtype
+    $v8 = PseudoVWREDSUM_VS_M1 killed renamable $v8, killed renamable $v26, killed renamable $v27, 1, 3, implicit $vl, implicit $vtype
+    $v26 = COPY killed renamable $v8
+    $x10 = PseudoVSETVLI killed renamable $x10, 75, implicit-def $vl, implicit-def $vtype
+    $v8m8 = PseudoVRELOAD_M8 killed $x10
+...
+---
+name: copy_zvlsseg_reg
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x14, $x16
+    ; 80 = e32,m1
+    ; CHECK-LABEL: name: copy_zvlsseg_reg
+    ; CHECK: liveins: $x14, $x16
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $x15 = PseudoVSETVLI $x14, 80, implicit-def $vl, implicit-def $vtype
+    ; CHECK-NEXT: $v8_v9 = PseudoVLSEG2E32_V_M1 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    ; CHECK-NEXT: $v10 = PseudoVMV1R_V $v8
+    $x15 = PseudoVSETVLI $x14, 80, implicit-def $vl, implicit-def $vtype
+    $v8_v9 = PseudoVLSEG2E32_V_M1 killed $x16, $noreg, 5, implicit $vl, implicit $vtype
+    $v10 = COPY $v8
+...