[RISCV] Hoist vsetvli with vreg operand out of loops

[reames]

Summary

This is an extension to the PRE line of work. The goal is to be able to handle large constant AVLs which are too large to be encoded in a five bit immediate and thus need to be materialized in register. Handling other register cases is a nice to have.

This patch removes a predicate from the correctness logic for the PRE transform, and replaces it with an alternate one. The code being removed was phrased as "having a fixed result"; it's being replaced with a standard (if handrolled to avoid needing dominators) availability check.

When I wrote the original code, I was concerned about three cases. Thinking about each further, I think I was simply being too conservative from the beginning.

First, we might be in a loop where the VL is potentially changing. The availability check handles this as the value would not be available in the preheader in this case. PREing into a rarer path in the loop is fine, as by assumption the final VL state is the same along all paths.

Second, vsetvli is undefined in certain cases - specifically the vsetvli x0, x0, vtype case. If we moved it above control flow, could we introduce a fault? I believe the answer to be yes, but the key point here is that we're *not* moving an existing instruction. We're moving a state transition. If VL actually changes, the insert code would not use the x0, x0 form at the new point. (And couldn't at the original either since VL could change on at least one incoming path.)

To say this point differently, we don't need the standard "anticipation" check from PRE as the data flow and insertion logic covers that bit for us. By construction, there isn't some unmodeled control dependence for the "instruction being moved". Or said differently, state transitions are always safe to speculate.

Third, the specification leaves the hardware significant flexibility to handle VLMAX < AVL < 2 x VLMAX. Originally, I think my concern had been that hoisting such cases might allow different VL values, but thinking about this further, the spec explicitly requires determinism for each AVL value. Given that, we'd only run into trouble if we considered two different AVLs (which we'd somehow falsely proven to produce the same VL) along different paths. Doing this would be an unrelated bug in the dataflow, and thus we don't have to account for it here.

Skeptical review to make sure I haven't missed something here is appreciated.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

1321

Does the end iterator of a basic block point a unique state such that this returning true means that DefMI is in UnavailablePred? Or could they both be at the end of different basic blocks?

[reames]

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

1321

They could be in different basic blocks, so there's an iterator comparison bug here. Thanks!

Will rev with a fix.

[reames]

Fix bug noted by reviewer.

[craig.topper]

LGTM

[reames]

Went to land this today - I've been intentionally spacing out changes since I have a lot of churn in this file - and encountered a new crash in a test failure. Need to investigate and probably re-review once cause is understood.

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

for (MachineInstr &MI : MBB) {
     // If this is something that updates VL/VTYPE that we don't know about,
     // set the state to unknown.
     if (MI.isCall() || MI.isInlineAsm() || MI.modifiesRegister(RISCV::VL) ||
         MI.modifiesRegister(RISCV::VTYPE))
       CurInfo = VSETVLIInfo::getUnknown();
   }
 }
 
-/// Return true if the VL value configured must be equal to the requested one.
-static bool hasFixedResult(const VSETVLIInfo &Info, const RISCVSubtarget &ST) {
-  if (!Info.hasAVLImm())
-    // VLMAX is always the same value.
-    // TODO: Could extend to other registers by looking at the associated vreg
-    // def placement.
-    return RISCV::X0 == Info.getAVLReg();
-
-  unsigned AVL = Info.getAVLImm();
-  unsigned SEW = Info.getSEW();
-  unsigned AVLInBits = AVL * SEW;
-
-  unsigned LMul;
-  bool Fractional;
-  std::tie(LMul, Fractional) = RISCVVType::decodeVLMUL(Info.getVLMUL());
-
-  if (Fractional)
-    return ST.getRealMinVLen() / LMul >= AVLInBits;
-  return ST.getRealMinVLen() * LMul >= AVLInBits;
-}
-
 /// Perform simple partial redundancy elimination of the VSETVLI instructions
 /// we're about to insert by looking for cases where we can PRE from the
 /// beginning of one block to the end of one of its predecessors.  Specifically,
 /// this is geared to catch the common case of a fixed length vsetvl in a single
 /// block loop when it could execute once in the preheader instead.
 void RISCVInsertVSETVLI::doPRE(MachineBasicBlock &MBB) {
-  const MachineFunction &MF = *MBB.getParent();
-  const RISCVSubtarget &ST = MF.getSubtarget<RISCVSubtarget>();
-
   if (!BlockInfo[MBB.getNumber()].Pred.isUnknown())
     return;
 
   MachineBasicBlock *UnavailablePred = nullptr;
   VSETVLIInfo AvailableInfo;
   for (MachineBasicBlock *P : MBB.predecessors()) {
     const VSETVLIInfo &PredInfo = BlockInfo[P->getNumber()].Exit;
     if (PredInfo.isUnknown()) {
void RISCVInsertVSETVLI::doPRE(MachineBasicBlock &MBB) {
   // phase 3.
   if (!UnavailablePred || !AvailableInfo.isValid())
     return;
 
   // Critical edge - TODO: consider splitting?
   if (UnavailablePred->succ_size() != 1)
     return;
 
-  // If VL can be less than AVL, then we can't reduce the frequency of exec.
-  if (!hasFixedResult(AvailableInfo, ST))
+  // Note there's an implicit assumption here that terminators never use
+  // or modify VL or VTYPE.  Also, fallthrough will return end().
+  auto InsertPt = UnavailablePred->getFirstInstrTerminator();
+
+  // Is the AVL value available in the predecessor?
+  if (!AvailableInfo.hasAVLImm() && RISCV::X0 != AvailableInfo.getAVLReg()) {
+    auto *DefBB = MRI->getVRegDef(AvailableInfo.getAVLReg())->getParent();
+    // Do a very cheap dominance check specialized for the case where the
+    // register is defined in the block we're moving into.  This is common
+    // for e.g. values hoisted into preheaders of loops.  We could do
+    // a general dominance check here instead if we though the compile
+    // time impact was worthwhile.
+    if (DefBB != UnavailablePred || DefBB->instr_end() != InsertPt)
       return;
+  }
 
   // Does it actually let us remove an implicit transition in MBB?
   bool Found = false;
   for (auto &MI : MBB) {
     if (isVectorConfigInstr(MI))
       return;
 
     const uint64_t TSFlags = MI.getDesc().TSFlags;
void RISCVInsertVSETVLI::doPRE(MachineBasicBlock &MBB) {
   // is critical for correctness of phase 3.
   auto OldInfo = BlockInfo[UnavailablePred->getNumber()].Exit;
   LLVM_DEBUG(dbgs() << "PRE VSETVLI from " << MBB.getName() << " to "
                     << UnavailablePred->getName() << " with state "
                     << AvailableInfo << "\n");
   BlockInfo[UnavailablePred->getNumber()].Exit = AvailableInfo;
   BlockInfo[MBB.getNumber()].Pred = AvailableInfo;
 
-  // Note there's an implicit assumption here that terminators never use
-  // or modify VL or VTYPE.  Also, fallthrough will return end().
-  auto InsertPt = UnavailablePred->getFirstInstrTerminator();
   insertVSETVLI(*UnavailablePred, InsertPt,
                 UnavailablePred->findDebugLoc(InsertPt),
                 AvailableInfo, OldInfo);
 }
 
 void RISCVInsertVSETVLI::doLocalPostpass(MachineBasicBlock &MBB) {
   MachineInstr *PrevMI = nullptr;
   bool UsedVL = false, UsedVTYPE = false;

llvm/test/CodeGen/RISCV/rvv/fixed-vector-strided-load-store-asm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=riscv64 -mattr=+m,+v -riscv-v-vector-bits-min=256 | FileCheck %s
 
 %struct.foo = type { i32, i32, i32, i32 }
 
 ; void gather(signed char * __restrict  A, signed char * __restrict B) {
 ;   for (int i = 0; i != 1024; ++i)
 ;       A[i] += B[i * 5];
 ; }
 define void @gather(i8* noalias nocapture %A, i8* noalias nocapture readonly %B) {
 ;
 ; CHECK-LABEL: gather:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 0
-; CHECK-NEXT:    li a3, 32
-; CHECK-NEXT:    li a4, 5
-; CHECK-NEXT:    li a5, 1024
+; CHECK-NEXT:    li a4, 32
+; CHECK-NEXT:    li a3, 5
+; CHECK-NEXT:    vsetvli zero, a4, e8, m1, ta, mu
+; CHECK-NEXT:    li a4, 1024
 ; CHECK-NEXT:  .LBB0_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e8, m1, ta, mu
-; CHECK-NEXT:    vlse8.v v8, (a1), a4
-; CHECK-NEXT:    add a6, a0, a2
-; CHECK-NEXT:    vle8.v v9, (a6)
+; CHECK-NEXT:    vlse8.v v8, (a1), a3
+; CHECK-NEXT:    add a5, a0, a2
+; CHECK-NEXT:    vle8.v v9, (a5)
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vse8.v v8, (a6)
+; CHECK-NEXT:    vse8.v v8, (a5)
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a1, a1, 160
-; CHECK-NEXT:    bne a2, a5, .LBB0_1
+; CHECK-NEXT:    bne a2, a4, .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %entry
   %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
   %vec.ind = phi <32 x i64> [ <i64 0, i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8, i64 9, i64 10, i64 11, i64 12, i64 13, i64 14, i64 15, i64 16, i64 17, i64 18, i64 19, i64 20, i64 21, i64 22, i64 23, i64 24, i64 25, i64 26, i64 27, i64 28, i64 29, i64 30, i64 31>, %entry ], [ %vec.ind.next, %vector.body ]

 }
 
 define void @gather_negative_stride(i8* noalias nocapture %A, i8* noalias nocapture readonly %B) {
 ;
 ; CHECK-LABEL: gather_negative_stride:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 0
 ; CHECK-NEXT:    addi a1, a1, 155
-; CHECK-NEXT:    li a3, 32
-; CHECK-NEXT:    li a4, -5
-; CHECK-NEXT:    li a5, 1024
+; CHECK-NEXT:    li a4, 32
+; CHECK-NEXT:    li a3, -5
+; CHECK-NEXT:    vsetvli zero, a4, e8, m1, ta, mu
+; CHECK-NEXT:    li a4, 1024
 ; CHECK-NEXT:  .LBB2_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e8, m1, ta, mu
-; CHECK-NEXT:    vlse8.v v8, (a1), a4
-; CHECK-NEXT:    add a6, a0, a2
-; CHECK-NEXT:    vle8.v v9, (a6)
+; CHECK-NEXT:    vlse8.v v8, (a1), a3
+; CHECK-NEXT:    add a5, a0, a2
+; CHECK-NEXT:    vle8.v v9, (a5)
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vse8.v v8, (a6)
+; CHECK-NEXT:    vse8.v v8, (a5)
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a1, a1, 160
-; CHECK-NEXT:    bne a2, a5, .LBB2_1
+; CHECK-NEXT:    bne a2, a4, .LBB2_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %entry
   %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
   %vec.ind = phi <32 x i64> [ <i64 31, i64 30, i64 29, i64 28, i64 27, i64 26, i64 25, i64 24, i64 23, i64 22, i64 21, i64 20, i64 19, i64 18, i64 17, i64 16, i64 15, i64 14, i64 13, i64 12, i64 11, i64 10, i64 9, i64 8, i64 7, i64 6, i64 5, i64 4, i64 3, i64 2, i64 1, i64 0>, %entry ], [ %vec.ind.next, %vector.body ]

 }
 
 define void @gather_zero_stride(i8* noalias nocapture %A, i8* noalias nocapture readonly %B) {
 ;
 ; CHECK-LABEL: gather_zero_stride:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 0
 ; CHECK-NEXT:    li a3, 32
-; CHECK-NEXT:    li a4, 1024
+; CHECK-NEXT:    vsetvli zero, a3, e8, m1, ta, mu
+; CHECK-NEXT:    li a3, 1024
 ; CHECK-NEXT:  .LBB3_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e8, m1, ta, mu
 ; CHECK-NEXT:    vlse8.v v8, (a1), zero
-; CHECK-NEXT:    add a5, a0, a2
-; CHECK-NEXT:    vle8.v v9, (a5)
+; CHECK-NEXT:    add a4, a0, a2
+; CHECK-NEXT:    vle8.v v9, (a4)
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vse8.v v8, (a5)
+; CHECK-NEXT:    vse8.v v8, (a4)
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a1, a1, 160
-; CHECK-NEXT:    bne a2, a4, .LBB3_1
+; CHECK-NEXT:    bne a2, a3, .LBB3_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %entry
   %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
   %vec.ind = phi <32 x i64> [ zeroinitializer, %entry ], [ %vec.ind.next, %vector.body ]

 ;  for (int i = 0; i < 1024; ++i)
 ;      A[i * 5] += B[i];
 ;}
 define void @scatter(i8* noalias nocapture %A, i8* noalias nocapture readonly %B) {
 ;
 ; CHECK-LABEL: scatter:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 0
-; CHECK-NEXT:    li a3, 32
-; CHECK-NEXT:    li a4, 5
-; CHECK-NEXT:    li a5, 1024
+; CHECK-NEXT:    li a4, 32
+; CHECK-NEXT:    li a3, 5
+; CHECK-NEXT:    vsetvli zero, a4, e8, m1, ta, mu
+; CHECK-NEXT:    li a4, 1024
 ; CHECK-NEXT:  .LBB4_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    add a6, a1, a2
-; CHECK-NEXT:    vsetvli zero, a3, e8, m1, ta, mu
-; CHECK-NEXT:    vle8.v v8, (a6)
-; CHECK-NEXT:    vlse8.v v9, (a0), a4
+; CHECK-NEXT:    add a5, a1, a2
+; CHECK-NEXT:    vle8.v v8, (a5)
+; CHECK-NEXT:    vlse8.v v9, (a0), a3
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsse8.v v8, (a0), a4
+; CHECK-NEXT:    vsse8.v v8, (a0), a3
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a0, a0, 160
-; CHECK-NEXT:    bne a2, a5, .LBB4_1
+; CHECK-NEXT:    bne a2, a4, .LBB4_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %entry
   %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
   %vec.ind = phi <32 x i64> [ <i64 0, i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8, i64 9, i64 10, i64 11, i64 12, i64 13, i64 14, i64 15, i64 16, i64 17, i64 18, i64 19, i64 20, i64 21, i64 22, i64 23, i64 24, i64 25, i64 26, i64 27, i64 28, i64 29, i64 30, i64 31>, %entry ], [ %vec.ind.next, %vector.body ]

 ; CHECK-NEXT:    srli a3, a3, 32
 ; CHECK-NEXT:    addi a4, a3, 1
 ; CHECK-NEXT:    andi a5, a4, -32
 ; CHECK-NEXT:    add a3, a5, a2
 ; CHECK-NEXT:    slli a6, a2, 2
 ; CHECK-NEXT:    add a6, a6, a2
 ; CHECK-NEXT:    add a2, a0, a2
 ; CHECK-NEXT:    add a6, a1, a6
-; CHECK-NEXT:    li a7, 32
-; CHECK-NEXT:    li t0, 5
-; CHECK-NEXT:    mv t1, a5
+; CHECK-NEXT:    li t0, 32
+; CHECK-NEXT:    li a7, 5
+; CHECK-NEXT:    vsetvli zero, t0, e8, m1, ta, mu
+; CHECK-NEXT:    mv t0, a5
 ; CHECK-NEXT:  .LBB12_3: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a7, e8, m1, ta, mu
-; CHECK-NEXT:    vlse8.v v8, (a6), t0
+; CHECK-NEXT:    vlse8.v v8, (a6), a7
 ; CHECK-NEXT:    vle8.v v9, (a2)
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
 ; CHECK-NEXT:    vse8.v v8, (a2)
-; CHECK-NEXT:    addi t1, t1, -32
+; CHECK-NEXT:    addi t0, t0, -32
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a6, a6, 160
-; CHECK-NEXT:    bnez t1, .LBB12_3
+; CHECK-NEXT:    bnez t0, .LBB12_3
 ; CHECK-NEXT:  # %bb.4:
 ; CHECK-NEXT:    beq a4, a5, .LBB12_7
 ; CHECK-NEXT:  .LBB12_5:
 ; CHECK-NEXT:    slli a2, a3, 2
 ; CHECK-NEXT:    add a2, a2, a3
 ; CHECK-NEXT:    add a1, a1, a2
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:  .LBB12_6: # =>This Inner Loop Header: Depth=1

llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll

for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_mul_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_mul_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB74_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vmul.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB74_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_add_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_add_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB75_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vadd.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB75_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_sub_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_sub_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB76_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vsub.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB76_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_rsub_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_rsub_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB77_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vrsub.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB77_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_and_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_and_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB78_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vand.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB78_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_or_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_or_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB79_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vor.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB79_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
for.cond.cleanup:                                 ; preds = %vector.body
   ret void
 }
 
 define void @sink_splat_xor_lmul8(i32* nocapture %a, i32 signext %x) {
 ; CHECK-LABEL: sink_splat_xor_lmul8:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:    li a3, 32
+; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:  .LBB80_1: # %vector.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a3, e32, m8, ta, mu
 ; CHECK-NEXT:    vle32.v v8, (a0)
 ; CHECK-NEXT:    vxor.vx v8, v8, a1
 ; CHECK-NEXT:    vse32.v v8, (a0)
 ; CHECK-NEXT:    addi a2, a2, -4
 ; CHECK-NEXT:    addi a0, a0, 16
 ; CHECK-NEXT:    bnez a2, .LBB80_1
 ; CHECK-NEXT:  # %bb.2: # %for.cond.cleanup
 ; CHECK-NEXT:    ret