[LSR][TTI][RISCV] Add isAllowTerminatingConditionFoldingAfterLSR into TTI and enable it for RISC-V

[reames]

Summary

Previously in D132443, the transformation was added and guarded by an option.
This commit attempts to create an TTI and enable it for the RISC-V backend.

[eopXD]

I ran through lit test under llvm/test/CodeGen/RISCV before submitting this patch, these two affected test case shows the benefit of the transformation.

[JojoR]

There is a error report as below if i apply this patch on branch main 8d4ebd1a7c9e1f47a4a610aeb41d1613f822ee20
error: Terminating value is not safe to expand, need to add it to predicate.

my compiler option is "-march=rv64gc -O2"
my test case is :

alac.i279 KBDownload

[eopXD]

@JojoR Thanks for the report. I was wondering that whether the assertion would hit. Will add a patch to fix this.

[JojoR]

I have a question about patch D132443, why do not you put optimization in pass "iv-users" ? I think it belongs to IV analyzation.

because that the pass "iv-users" do analyzation only without any transformation ?

[eopXD]

In D134893#3855128, @JojoR wrote:

I have a question about patch D132443, why do not you put optimization in pass "iv-users" ? I think it belongs to IV analyzation.

because that the pass "iv-users" do analyzation only without any transformation ?

IVUser is an analysis pass, and the LSR pass uses it before doing strength reduction.
This transformation is beneficial after LSR, not before it.

[JojoR]

In D134893#3855134, @eopXD wrote:

In D134893#3855128, @JojoR wrote:

I have a question about patch D132443, why do not you put optimization in pass "iv-users" ? I think it belongs to IV analyzation.

because that the pass "iv-users" do analyzation only without any transformation ?

IVUser is an analysis pass, and the LSR pass uses it before doing strength reduction.
This transformation is beneficial after LSR, not before it.

Got it, thanks :)

[JojoR]

@eopXD I saw the patch D132443 only support eq/neq condterm, and I want to add gt/lt,
any suggestion for me ? or you have plan to implement that ?

[eopXD]

In D134893#3855180, @JojoR wrote:

@eopXD I saw the patch D132443 only support eq/neq condterm, and I want to add gt/lt,
any suggestion for me ? or you have plan to implement that ?

I will submit proceeding patches for more support, thank you.

[eopXD]

In D134893#3848868, @JojoR wrote:

There is a error report as below if i apply this patch on branch main 8d4ebd1a7c9e1f47a4a610aeb41d1613f822ee20
error: Terminating value is not safe to expand, need to add it to predicate.

my compiler option is "-march=rv64gc -O2"
my test case is :

alac.i279 KBDownload

Is it possible for you to have a reduced test case on this? It would help to land the revision to fix this. Thank you.

[eopXD]

Rebase to latest main

[MarkGoncharovAl]

I think we should start discussion again, because this pass has changed.

@eopXD
@JojoR

1. Implemented : https://reviews.llvm.org/D145929

I saw the patch D132443 only support eq/neq condterm, and I want to add gt/lt,
any suggestion for me

2. Implemented : https://reviews.llvm.org/D136415

There is a error report as below if i apply this patch on branch main
error: Terminating value is not safe to expand, need to add it to predicate.

So, may we should rebase this MR and apply changes?

[eopXD]

I agree with @fhahn that such hook to a middle-end optimization like LSR is not a great approach. I have generally improving LSR term-fold and default enabling it as an item to my queue. I have the RVV intrinsics to be dealt with first but this will be the first thing I'll revisit once I am done there.

If there is no further comments I will drop this patch later this week.

[reames]

lsr-term-fold is still wildly unsound. I have patches out to address this, but we should wait until that is done discussing enabling it (by default or by target).

[reames]

Ok, at this point all the known soundness problems are fixed in tree. I was simply fixing ones obvious on inspection, and have not done any testing of this mechanism beyond the LIT tests themselves.

Before we move to discussing whether lsr-term-fold makes sense to enable by default, and on what basis, I think we need to have a discussion about validation and testing. Whoever is going to drive this patch forward needs to describe their testing, and validate that after all the bug fixes this still triggers enough to be worthwhile.

I will note that I am generally skeptical of this being enabled on a per-target basis. There needs to be a compelling argument as why this shouldn't be enabled more broadly. One of the major advantages of enabling this globally is increasing testing, and thus smoking out bugs more quickly. Given existing problems with testing, I think that advantage is one I'm very reluctant to give up. I also think that all targets should benefit from the transform as current framed, so I don't see any reason not to enable it. (I'm open to counter arguments here; they just need to be made.)

[reames]

In D134893#4210164, @reames wrote:

I will note that I am generally skeptical of this being enabled on a per-target basis.

To circle back around here, I spent some time looking at the impact of this on other targets. In short, this heuristic really only makes sense on RISCV at the moment. Given this, my prior objection to a transform specific target hook no longer applies.

I do want to see some discussion of the testing this patch has been put through before we move forward with enabling it though. I'd encourage a rebase and a summary comment describing any testing which has been done.

[eopXD]

Rebase to latest main.

[eopXD]

Resolve other test case failures.

[reames]

Reverse ping. Any progress on that testing summary?

[reames]

Reverse ping. I applied this locally, and collected dynamic instruction counts for SPEC on a rv64gcv config. The results didn't reveal any surprises or correctness issues. Dynamic instruction count was very mildly improved overall (0.42% geomean improvement), with no regression larger than 0.25%.

I'd like to see this landed. Can you rebase for a final LGTM?

llvm/include/llvm/Analysis/TargetTransformInfo.h

619

Please change to: shouldFoldTerminatingConditionAfterLSR

[reames]

A reworked version of this has been landed in e947f953370abe8ffc8713b8f3250a3ec39599fe.

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
   bool isLSRCostLess(const TargetTransformInfo::LSRCost &C1,
                      const TargetTransformInfo::LSRCost &C2) const;
 
   /// Return true if LSR major cost is number of registers. Targets which
   /// implement their own isLSRCostLess and unset number of registers as major
   /// cost should return false, otherwise return true.
   bool isNumRegsMajorCostOfLSR() const;
 
+  /// Return true if LSR attempts to replace primary IV with other IV.
+  bool isAllowTerminatingConditionFoldingAfterLSR() const;
+
   /// \returns true if LSR should not optimize a chain that includes \p I.
   bool isProfitableLSRChainElement(Instruction *I) const;
 
   /// Return true if the target can fuse a compare and branch.
   /// Loop-strength-reduction (LSR) uses that knowledge to adjust its cost
   /// calculation for the instructions in a loop.
   bool canMacroFuseCmp() const;
 
public:
   virtual bool isLegalICmpImmediate(int64_t Imm) = 0;
   virtual bool isLegalAddressingMode(Type *Ty, GlobalValue *BaseGV,
                                      int64_t BaseOffset, bool HasBaseReg,
                                      int64_t Scale, unsigned AddrSpace,
                                      Instruction *I) = 0;
   virtual bool isLSRCostLess(const TargetTransformInfo::LSRCost &C1,
                              const TargetTransformInfo::LSRCost &C2) = 0;
   virtual bool isNumRegsMajorCostOfLSR() = 0;
+  virtual bool isAllowTerminatingConditionFoldingAfterLSR() const = 0;
   virtual bool isProfitableLSRChainElement(Instruction *I) = 0;
   virtual bool canMacroFuseCmp() = 0;
   virtual bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE,
                           LoopInfo *LI, DominatorTree *DT, AssumptionCache *AC,
                           TargetLibraryInfo *LibInfo) = 0;
   virtual AddressingModeKind
     getPreferredAddressingMode(const Loop *L, ScalarEvolution *SE) const = 0;
   virtual bool isLegalMaskedStore(Type *DataType, Align Alignment) = 0;
public:
   }
   bool isLSRCostLess(const TargetTransformInfo::LSRCost &C1,
                      const TargetTransformInfo::LSRCost &C2) override {
     return Impl.isLSRCostLess(C1, C2);
   }
   bool isNumRegsMajorCostOfLSR() override {
     return Impl.isNumRegsMajorCostOfLSR();
   }
+  bool isAllowTerminatingConditionFoldingAfterLSR() const override {
+    return Impl.isAllowTerminatingConditionFoldingAfterLSR();
+  }
   bool isProfitableLSRChainElement(Instruction *I) override {
     return Impl.isProfitableLSRChainElement(I);
   }
   bool canMacroFuseCmp() override { return Impl.canMacroFuseCmp(); }
   bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
                   DominatorTree *DT, AssumptionCache *AC,
                   TargetLibraryInfo *LibInfo) override {
     return Impl.canSaveCmp(L, BI, SE, LI, DT, AC, LibInfo);

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

bool isLSRCostLess(const TTI::LSRCost &C1, const TTI::LSRCost &C2) const {
     return std::tie(C1.NumRegs, C1.AddRecCost, C1.NumIVMuls, C1.NumBaseAdds,
                     C1.ScaleCost, C1.ImmCost, C1.SetupCost) <
            std::tie(C2.NumRegs, C2.AddRecCost, C2.NumIVMuls, C2.NumBaseAdds,
                     C2.ScaleCost, C2.ImmCost, C2.SetupCost);
   }
 
   bool isNumRegsMajorCostOfLSR() const { return true; }
 
+  bool isAllowTerminatingConditionFoldingAfterLSR() const { return false; }
+
   bool isProfitableLSRChainElement(Instruction *I) const { return false; }
 
   bool canMacroFuseCmp() const { return false; }
 
   bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
                   DominatorTree *DT, AssumptionCache *AC,
                   TargetLibraryInfo *LibInfo) const {
     return false;

llvm/include/llvm/CodeGen/BasicTTIImpl.h

public:
   bool isLSRCostLess(TTI::LSRCost C1, TTI::LSRCost C2) {
     return TargetTransformInfoImplBase::isLSRCostLess(C1, C2);
   }
 
   bool isNumRegsMajorCostOfLSR() {
     return TargetTransformInfoImplBase::isNumRegsMajorCostOfLSR();
   }
 
+  bool isAllowTerminatingConditionFoldingAfterLSR() const {
+    return TargetTransformInfoImplBase::
+        isAllowTerminatingConditionFoldingAfterLSR();
+  }
+
   bool isProfitableLSRChainElement(Instruction *I) {
     return TargetTransformInfoImplBase::isProfitableLSRChainElement(I);
   }
 
   InstructionCost getScalingFactorCost(Type *Ty, GlobalValue *BaseGV,
                                        int64_t BaseOffset, bool HasBaseReg,
                                        int64_t Scale, unsigned AddrSpace) {
     TargetLoweringBase::AddrMode AM;

llvm/lib/Analysis/TargetTransformInfo.cpp

bool TargetTransformInfo::isLSRCostLess(const LSRCost &C1,
                                         const LSRCost &C2) const {
   return TTIImpl->isLSRCostLess(C1, C2);
 }
 
 bool TargetTransformInfo::isNumRegsMajorCostOfLSR() const {
   return TTIImpl->isNumRegsMajorCostOfLSR();
 }
 
+bool TargetTransformInfo::isAllowTerminatingConditionFoldingAfterLSR() const {
+  return TTIImpl->isAllowTerminatingConditionFoldingAfterLSR();
+}
+
 bool TargetTransformInfo::isProfitableLSRChainElement(Instruction *I) const {
   return TTIImpl->isProfitableLSRChainElement(I);
 }
 
 bool TargetTransformInfo::canMacroFuseCmp() const {
   return TTIImpl->canMacroFuseCmp();
 }
 

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

case RISCVRegisterClass::GPRRC:
       return "RISCV::GPRRC";
     case RISCVRegisterClass::FPRRC:
       return "RISCV::FPRRC";
     case RISCVRegisterClass::VRRC:
       return "RISCV::VRRC";
     }
     llvm_unreachable("unknown register class");
   }
+
+  bool isAllowTerminatingConditionFoldingAfterLSR() const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVTARGETTRANSFORMINFO_H

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

 unsigned RISCVTTIImpl::getMaximumVF(unsigned ElemWidth, unsigned Opcode) const {
   // This interface is currently only used by SLP.  Returning 1 (which is the
   // default value for SLPMaxVF) disables SLP. We currently have a cost modeling
   // problem w/ constant materialization which causes SLP to perform majorly
   // unprofitable transformations.
   // TODO: Figure out constant materialization cost modeling and remove.
   return SLPMaxVF;
 }
+
+bool RISCVTTIImpl::isAllowTerminatingConditionFoldingAfterLSR() const {
+  return true;
+}

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

if (L->isRecursivelyLCSSAForm(DT, LI) && L->getExitBlock()) {
     if (Rewrites) {
       Changed = true;
       RecursivelyDeleteTriviallyDeadInstructionsPermissive(DeadInsts, &TLI,
                                                            MSSAU.get());
       DeleteDeadPHIs(L->getHeader(), &TLI, MSSAU.get());
     }
   }
 
-  if (AllowTerminatingConditionFoldingAfterLSR) {
+  if (AllowTerminatingConditionFoldingAfterLSR ||
+      TTI.isAllowTerminatingConditionFoldingAfterLSR()) {
     auto CanFoldTerminatingCondition = canFoldTermCondOfLoop(L, SE, DT, LI);
     if (CanFoldTerminatingCondition) {
       BasicBlock *LoopPreheader = L->getLoopPreheader();
       BasicBlock *LoopLatch = L->getLoopLatch();
 
       PHINode *ToFold = CanFoldTerminatingCondition->first;
       PHINode *ToHelpFold = CanFoldTerminatingCondition->second;
 

llvm/test/CodeGen/RISCV/rvv/fixed-vector-strided-load-store-asm.ll

 ; CHECK-NEXT:    srli a3, a3, 32
 ; CHECK-NEXT:    addi a4, a3, 1
 ; CHECK-NEXT:    andi a5, a4, -32
 ; CHECK-NEXT:    add a3, a5, a2
 ; CHECK-NEXT:    slli a6, a2, 2
 ; CHECK-NEXT:    add a6, a6, a2
 ; CHECK-NEXT:    add a2, a0, a2
 ; CHECK-NEXT:    add a6, a1, a6
-; CHECK-NEXT:    li a7, 32
-; CHECK-NEXT:    li t0, 5
-; CHECK-NEXT:    mv t1, a5
+; CHECK-NEXT:    add a7, a3, a0
+; CHECK-NEXT:    li t0, 32
+; CHECK-NEXT:    li t1, 5
 ; CHECK-NEXT:  .LBB12_3: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vsetvli zero, a7, e8, m1, ta, ma
-; CHECK-NEXT:    vlse8.v v8, (a6), t0
+; CHECK-NEXT:    vsetvli zero, t0, e8, m1, ta, ma
+; CHECK-NEXT:    vlse8.v v8, (a6), t1
 ; CHECK-NEXT:    vle8.v v9, (a2)
 ; CHECK-NEXT:    vadd.vv v8, v9, v8
 ; CHECK-NEXT:    vse8.v v8, (a2)
-; CHECK-NEXT:    addi t1, t1, -32
 ; CHECK-NEXT:    addi a2, a2, 32
 ; CHECK-NEXT:    addi a6, a6, 160
-; CHECK-NEXT:    bnez t1, .LBB12_3
+; CHECK-NEXT:    bne a2, a7, .LBB12_3
 ; CHECK-NEXT:  # %bb.4:
 ; CHECK-NEXT:    beq a4, a5, .LBB12_7
 ; CHECK-NEXT:  .LBB12_5:
 ; CHECK-NEXT:    slli a2, a3, 2
 ; CHECK-NEXT:    add a2, a2, a3
 ; CHECK-NEXT:    add a1, a1, a2
 ; CHECK-NEXT:    li a2, 1024
 ; CHECK-NEXT:  .LBB12_6: # =>This Inner Loop Header: Depth=1

llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll

 ; CHECK-NEXT:    vmul.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB7_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB7_7
 ; CHECK-NEXT:  .LBB7_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB7_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    mulw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB7_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    mulw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB7_6
 ; CHECK-NEXT:  .LBB7_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vadd.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB8_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB8_7
 ; CHECK-NEXT:  .LBB8_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB8_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    addw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB8_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    addw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB8_6
 ; CHECK-NEXT:  .LBB8_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vsub.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB9_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB9_7
 ; CHECK-NEXT:  .LBB9_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB9_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    addw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB9_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    addw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB9_6
 ; CHECK-NEXT:  .LBB9_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vrsub.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB10_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB10_7
 ; CHECK-NEXT:  .LBB10_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB10_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    subw a2, a1, a2
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB10_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    subw a3, a1, a3
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB10_6
 ; CHECK-NEXT:  .LBB10_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vand.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB11_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB11_7
 ; CHECK-NEXT:  .LBB11_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB11_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    and a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB11_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    and a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB11_6
 ; CHECK-NEXT:  .LBB11_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vor.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB12_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB12_7
 ; CHECK-NEXT:  .LBB12_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB12_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    or a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB12_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    or a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB12_6
 ; CHECK-NEXT:  .LBB12_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vxor.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB13_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB13_7
 ; CHECK-NEXT:  .LBB13_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB13_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    xor a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB13_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    xor a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB13_6
 ; CHECK-NEXT:  .LBB13_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vsll.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB17_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB17_7
 ; CHECK-NEXT:  .LBB17_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB17_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    sllw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB17_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    sllw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB17_6
 ; CHECK-NEXT:  .LBB17_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vsrl.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB18_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB18_7
 ; CHECK-NEXT:  .LBB18_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB18_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    srlw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB18_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    srlw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB18_6
 ; CHECK-NEXT:  .LBB18_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vsra.vi v8, v8, 2
 ; CHECK-NEXT:    vs2r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a2
 ; CHECK-NEXT:    add a5, a5, a4
 ; CHECK-NEXT:    bnez a6, .LBB19_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a3, .LBB19_7
 ; CHECK-NEXT:  .LBB19_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB19_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a1, 0(a0)
-; CHECK-NEXT:    srli a1, a1, 2
-; CHECK-NEXT:    sw a1, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB19_6
+; CHECK-NEXT:    lw a2, 0(a1)
+; CHECK-NEXT:    srli a2, a2, 2
+; CHECK-NEXT:    sw a2, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB19_6
 ; CHECK-NEXT:  .LBB19_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfmul.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB26_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB26_7
 ; CHECK-NEXT:  .LBB26_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB26_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fmul.s ft0, ft0, fa0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB26_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB26_6
 ; CHECK-NEXT:  .LBB26_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfdiv.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB27_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB27_7
 ; CHECK-NEXT:  .LBB27_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB27_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fdiv.s ft0, ft0, fa0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB27_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB27_6
 ; CHECK-NEXT:  .LBB27_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfrdiv.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB28_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB28_7
 ; CHECK-NEXT:  .LBB28_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB28_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fdiv.s ft0, fa0, ft0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB28_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB28_6
 ; CHECK-NEXT:  .LBB28_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfadd.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB29_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB29_7
 ; CHECK-NEXT:  .LBB29_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB29_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fadd.s ft0, ft0, fa0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB29_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB29_6
 ; CHECK-NEXT:  .LBB29_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfsub.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB30_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB30_7
 ; CHECK-NEXT:  .LBB30_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB30_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fsub.s ft0, ft0, fa0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB30_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB30_6
 ; CHECK-NEXT:  .LBB30_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfrsub.vf v8, v8, fa0
 ; CHECK-NEXT:    vs1r.v v8, (a5)
 ; CHECK-NEXT:    sub a6, a6, a3
 ; CHECK-NEXT:    add a5, a5, a2
 ; CHECK-NEXT:    bnez a6, .LBB31_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB31_7
 ; CHECK-NEXT:  .LBB31_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a1, -1024
 ; CHECK-NEXT:    slli a1, a1, 2
-; CHECK-NEXT:    add a0, a0, a1
+; CHECK-NEXT:    add a1, a0, a1
+; CHECK-NEXT:    lui a2, 1
+; CHECK-NEXT:    add a0, a0, a2
 ; CHECK-NEXT:  .LBB31_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    flw ft0, 0(a0)
+; CHECK-NEXT:    flw ft0, 0(a1)
 ; CHECK-NEXT:    fsub.s ft0, fa0, ft0
-; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB31_6
+; CHECK-NEXT:    fsw ft0, 0(a1)
+; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    bne a1, a0, .LBB31_6
 ; CHECK-NEXT:  .LBB31_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfmacc.vf v9, fa0, v8
 ; CHECK-NEXT:    vs1r.v v9, (t0)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a2
 ; CHECK-NEXT:    bnez a7, .LBB34_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a5, .LBB34_7
 ; CHECK-NEXT:  .LBB34_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a4, -1024
 ; CHECK-NEXT:    slli a3, a4, 2
-; CHECK-NEXT:    add a1, a1, a3
+; CHECK-NEXT:    add a2, a1, a3
 ; CHECK-NEXT:    add a0, a0, a3
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a1, a1, a3
 ; CHECK-NEXT:  .LBB34_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    flw ft0, 0(a0)
-; CHECK-NEXT:    flw ft1, 0(a1)
+; CHECK-NEXT:    flw ft1, 0(a2)
 ; CHECK-NEXT:    fmadd.s ft0, ft0, fa0, ft1
 ; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    addi a2, a2, 4
 ; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB34_6
+; CHECK-NEXT:    bne a2, a1, .LBB34_6
 ; CHECK-NEXT:  .LBB34_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vfmacc.vf v9, fa0, v8
 ; CHECK-NEXT:    vs1r.v v9, (t0)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a2
 ; CHECK-NEXT:    bnez a7, .LBB35_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a5, .LBB35_7
 ; CHECK-NEXT:  .LBB35_5: # %for.body.preheader
-; CHECK-NEXT:    addi a2, a4, -1024
 ; CHECK-NEXT:    slli a3, a4, 2
-; CHECK-NEXT:    add a1, a1, a3
+; CHECK-NEXT:    add a2, a1, a3
 ; CHECK-NEXT:    add a0, a0, a3
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a1, a1, a3
 ; CHECK-NEXT:  .LBB35_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    flw ft0, 0(a0)
-; CHECK-NEXT:    flw ft1, 0(a1)
+; CHECK-NEXT:    flw ft1, 0(a2)
 ; CHECK-NEXT:    fmadd.s ft0, fa0, ft0, ft1
 ; CHECK-NEXT:    fsw ft0, 0(a0)
-; CHECK-NEXT:    addi a2, a2, 1
-; CHECK-NEXT:    addi a1, a1, 4
+; CHECK-NEXT:    addi a2, a2, 4
 ; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a2, .LBB35_6
+; CHECK-NEXT:    bne a2, a1, .LBB35_6
 ; CHECK-NEXT:  .LBB35_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 1
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vdivu.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB42_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB42_7
 ; CHECK-NEXT:  .LBB42_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB42_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    divuw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB42_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    divuw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB42_6
 ; CHECK-NEXT:  .LBB42_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vdiv.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB43_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB43_7
 ; CHECK-NEXT:  .LBB43_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB43_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    divw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB43_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    divw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB43_6
 ; CHECK-NEXT:  .LBB43_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vremu.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB44_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB44_7
 ; CHECK-NEXT:  .LBB44_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB44_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    remuw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB44_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    remuw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB44_6
 ; CHECK-NEXT:  .LBB44_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph
 

 ; CHECK-NEXT:    vrem.vx v8, v8, a1
 ; CHECK-NEXT:    vs2r.v v8, (a6)
 ; CHECK-NEXT:    sub a7, a7, a3
 ; CHECK-NEXT:    add a6, a6, a5
 ; CHECK-NEXT:    bnez a7, .LBB45_3
 ; CHECK-NEXT:  # %bb.4: # %middle.block
 ; CHECK-NEXT:    beqz a4, .LBB45_7
 ; CHECK-NEXT:  .LBB45_5: # %for.body.preheader
-; CHECK-NEXT:    addi a3, a2, -1024
 ; CHECK-NEXT:    slli a2, a2, 2
-; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    add a2, a0, a2
+; CHECK-NEXT:    lui a3, 1
+; CHECK-NEXT:    add a0, a0, a3
 ; CHECK-NEXT:  .LBB45_6: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    lw a2, 0(a0)
-; CHECK-NEXT:    remw a2, a2, a1
-; CHECK-NEXT:    sw a2, 0(a0)
-; CHECK-NEXT:    addi a3, a3, 1
-; CHECK-NEXT:    addi a0, a0, 4
-; CHECK-NEXT:    bnez a3, .LBB45_6
+; CHECK-NEXT:    lw a3, 0(a2)
+; CHECK-NEXT:    remw a3, a3, a1
+; CHECK-NEXT:    sw a3, 0(a2)
+; CHECK-NEXT:    addi a2, a2, 4
+; CHECK-NEXT:    bne a2, a0, .LBB45_6
 ; CHECK-NEXT:  .LBB45_7: # %for.cond.cleanup
 ; CHECK-NEXT:    ret
 entry:
   %0 = call i64 @llvm.vscale.i64()
   %1 = shl i64 %0, 2
   %min.iters.check = icmp ugt i64 %1, 1024
   br i1 %min.iters.check, label %for.body.preheader, label %vector.ph