[RISCV] Add a prepass to vsetvli insertion to propagate VLMAX vsetvli to the instructions.

[craig.topper]

Summary

If the VL value for a vector instruction comes from a vsetvli for
VLMAX and the SEW/LMUL ratio matches, we can replace the VL on the
instruction with X0. This allows all the vsetvlis generated for the
instruction to use VLMAX vsetvlis. Or it can prevent vsetvli
instructions from being generated in loops.

This is based on a test case rogfer01 sent me a while back and the code
found here https://repo.hca.bsc.es/gitlab/rferrer/llvm-epi/-/raw/EPI/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp.
We could propagate any AVL, but looking at some of the test changes for that
I'm not sure it makes sense. So I've restricted to just the VLMAX case.

[craig.topper]

Ping

[frasercrmck]

Not sure how I missed this, sorry

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

927

Is there a reason not to propagate right away here and save us a boolean variable?

934

Good question. We aren't required to set dead, are we? From what I recall it can't be set if it's not true but can be left unset if it is true?

llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll

586

Could pre-commit this test so better show the effect?

[craig.topper]

Address feedback
Rebase on top of pre-committed test.

[reames]

Any chance we can get more tests here?

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

900

Pulling out a helper function to match this would help readability for me.

918

Can you use an early continue to reduce nesting here?

[craig.topper]

More early outs.
Helper function for determining if it is a VLMAX vsetvli.
Restoring the comparison that the vsetvli and instruction have the same SEW/LMUL ratio.

Clearly I need more tests, the last version was missing a critical check.

[reames]

Taking another look at this triggered by our offline chat.

I can't help but think the forward direction here is a bit awkward. It would seem more natural to visit all the instructions with VL operands, and then check for the VReg which defines it whether we could prove it be equivalent to VLMAX. Doing this either as a pre-pass, or maybe even inside getInfoForVSETVLI does seem to be worthwhile as it could eliminate inserted vsetvlis. On the other hand, a post-pass could be easier to test (via explicit vsetvli tests), and might catch some intrinsic cases.

In short, I don't have strong opinions on pre-vs-post, but might encourage working back from the consumer instruction instead of forward.

Hm, just a thought, if we are working backwards in a pre-pass this starts looking like a dag combine? Would we benefit from doing this there instead?

[craig.topper]

In D122769#3498032, @reames wrote:

Taking another look at this triggered by our offline chat.

I can't help but think the forward direction here is a bit awkward. It would seem more natural to visit all the instructions with VL operands, and then check for the VReg which defines it whether we could prove it be equivalent to VLMAX. Doing this either as a pre-pass, or maybe even inside getInfoForVSETVLI does seem to be worthwhile as it could eliminate inserted vsetvlis. On the other hand, a post-pass could be easier to test (via explicit vsetvli tests), and might catch some intrinsic cases.

In short, I don't have strong opinions on pre-vs-post, but might encourage working back from the consumer instruction instead of forward.

Makes sense. I'll make that change

Hm, just a thought, if we are working backwards in a pre-pass this starts looking like a dag combine? Would we benefit from doing this there instead?

DAGCombine can't do cross basic block.

[reames]

I played around with this a bit last night and realized you can achieve the same effect with a small change inside insertVSETVLI. I also implemented a simple version of the post-pass, and getting that to cover your case of interest turns out to be difficult as you also need the cross block reasoning.

@@ -682,6 +705,16 @@ void RISCVInsertVSETVLI::insertVSETVLI(MachineBasicBlock &MBB,
     return;
   }
 
+  
+  if (AVLReg.isVirtual()) {
+    MachineInstr &DefMI = *MRI->getVRegDef(AVLReg);
+    if (isVLMaxVSETVLI(DefMI)) {
+      VSETVLIInfo DefInfo = getInfoForVSETVLI(DefMI);
+      if (Info.hasSameVLMAX(DefInfo))
+        AVLReg = RISCV::X0;
+    }
+  }
   if (AVLReg.isVirtual())
     MRI->constrainRegClass(AVLReg, &RISCV::GPRNoX0RegClass);

[craig.topper]

I think D125812 covers this

llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp

 //===- RISCVInsertVSETVLI.cpp - Insert VSETVLI instructions ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
public:
   StringRef getPassName() const override { return RISCV_INSERT_VSETVLI_NAME; }
 
 private:
   bool needVSETVLI(const VSETVLIInfo &Require, const VSETVLIInfo &CurInfo);
   bool needVSETVLIPHI(const VSETVLIInfo &Require, const MachineBasicBlock &MBB);
   void insertVSETVLI(MachineBasicBlock &MBB, MachineInstr &MI,
                      const VSETVLIInfo &Info, const VSETVLIInfo &PrevInfo);
 
+  void forwardPropagateVLMAX(MachineBasicBlock &MBB);
   bool computeVLVTYPEChanges(const MachineBasicBlock &MBB);
   void computeIncomingVLVTYPE(const MachineBasicBlock &MBB);
   void emitVSETVLIs(MachineBasicBlock &MBB);
 };
 
 } // end anonymous namespace
 
 char RISCVInsertVSETVLI::ID = 0;
 
 INITIALIZE_PASS(RISCVInsertVSETVLI, DEBUG_TYPE, RISCV_INSERT_VSETVLI_NAME,
                 false, false)
 
 static bool isVectorConfigInstr(const MachineInstr &MI) {
   return MI.getOpcode() == RISCV::PseudoVSETVLI ||
          MI.getOpcode() == RISCV::PseudoVSETVLIX0 ||
          MI.getOpcode() == RISCV::PseudoVSETIVLI;
 }
 
+static bool isVLMaxVSETVLI(const MachineInstr &MI) {
+  return MI.getOpcode() == RISCV::PseudoVSETVLIX0 &&
+         MI.getOperand(0).getReg() != RISCV::X0;
+}
+
 static MachineInstr *elideCopies(MachineInstr *MI,
                                  const MachineRegisterInfo *MRI) {
   while (true) {
     if (!MI->isFullCopy())
       return MI;
     if (!Register::isVirtualRegister(MI->getOperand(1).getReg()))
       return nullptr;
     MI = MRI->getVRegDef(MI->getOperand(1).getReg());
bool canSkipVSETVLIForLoadStore(const MachineInstr &MI,
   case RISCV::PseudoVSSE64_V_M8_MASK:
     EEW = 64;
     break;
   }
 
   return CurInfo.isCompatibleWithLoadStoreEEW(EEW, Require);
 }
 
+// Forward propagate VLMAX vsetvlis.
+void RISCVInsertVSETVLI::forwardPropagateVLMAX(MachineBasicBlock &MBB) {
+  for (const MachineInstr &MI : MBB) {
+    // Look for VLMAX vsetvlis.
+    if (!isVLMaxVSETVLI(MI))
+      continue;
+
+    VSETVLIInfo VI = getInfoForVSETVLI(MI);
+    const MachineOperand &DestVLOp = MI.getOperand(0);
+    Register DestVLReg = DestVLOp.getReg();
+    // Walk through all uses of this VL.
+    for (MachineRegisterInfo::use_nodbg_iterator
+             UI = MRI->use_nodbg_begin(DestVLReg),
+             UIEnd = MRI->use_nodbg_end();
+         UI != UIEnd;) {
+      MachineOperand &Use(*UI++);
+      assert(Use.getParent() != nullptr && "Expected parent instruction");
+      const MachineInstr &UseMI = *Use.getParent();
+      const unsigned UseIndex = UseMI.getOperandNo(&Use);
+
+      uint64_t TSFlags = UseMI.getDesc().TSFlags;
+      if (!RISCVII::hasSEWOp(TSFlags) || !RISCVII::hasVLOp(TSFlags))
+        continue;
+
+      unsigned NumOperands = UseMI.getNumExplicitOperands();
+      if (RISCVII::hasVecPolicyOp(TSFlags))
+        --NumOperands;
+
+      // NumOperands - 2 == VLOpIndex
+      if (UseIndex != (NumOperands - 2))
+        continue;
+
+      VSETVLIInfo UseInfo = computeInfoForInstr(UseMI, TSFlags, MRI);
+      if (!UseInfo.hasSameVLMAX(VI))
+        continue;
+
+      Use.setReg(RISCV::X0);
+      // TODO: Should we update the dead flag or remove the instruction if
+      // we propagated to all users?
+    }
+  }
+}
+
 bool RISCVInsertVSETVLI::computeVLVTYPEChanges(const MachineBasicBlock &MBB) {
   bool HadVectorOp = false;
 
   BlockData &BBInfo = BlockInfo[MBB.getNumber()];
   for (const MachineInstr &MI : MBB) {
     // If this is an explicit VSETVLI or VSETIVLI, update our state.
     if (isVectorConfigInstr(MI)) {
       HadVectorOp = true;
if (!ST.hasVInstructions())
     return false;
 
   TII = ST.getInstrInfo();
   MRI = &MF.getRegInfo();
 
   assert(BlockInfo.empty() && "Expect empty block infos");
   BlockInfo.resize(MF.getNumBlockIDs());
 
+  // Phase 0 - propagate VLMAX vsetvlis.
+  for (MachineBasicBlock &MBB : MF)
+    forwardPropagateVLMAX(MBB);
+
   bool HaveVectorOp = false;
 
   // Phase 1 - determine how VL/VTYPE are affected by the each block.
   for (const MachineBasicBlock &MBB : MF)
     HaveVectorOp |= computeVLVTYPEChanges(MBB);
 
   // If we didn't find any instructions that need VSETVLI, we're done.
   if (HaveVectorOp) {

llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll

 
 if2.end:
   %g = phi <vscale x 2 x i32> [ %d, %if.end ], [ %f, %if2 ]
   %h = call <vscale x 2 x i32> @llvm.riscv.vadd.nxv2i32(<vscale x 2 x i32> undef, <vscale x 2 x i32> %g, <vscale x 2 x i32> %w, i64 %vl)
   ret <vscale x 2 x i32> %h
 }
 declare <vscale x 2 x i32> @llvm.riscv.vwadd.w.nxv2i32.nxv2i16(<vscale x 2 x i32>, <vscale x 2 x i32>, <vscale x 2 x i16>, i64)
 
 ; We should only need 1 vsetvli for this code.
 define void @vlmax(i64 %N, double* %c, double* %a, double* %b) {
 ; CHECK-LABEL: vlmax:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    vsetvli a6, zero, e64, m1, ta, mu
 ; CHECK-NEXT:    blez a0, .LBB11_3
 ; CHECK-NEXT:  # %bb.1: # %for.body.preheader
 ; CHECK-NEXT:    li a5, 0
-; CHECK-NEXT:    li t1, 0
+; CHECK-NEXT:    li t0, 0
 ; CHECK-NEXT:    slli a7, a6, 3
 ; CHECK-NEXT:  .LBB11_2: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    add t0, a2, a5
-; CHECK-NEXT:    vsetvli zero, a6, e64, m1, ta, mu
-; CHECK-NEXT:    vle64.v v8, (t0)
+; CHECK-NEXT:    add a4, a2, a5
+; CHECK-NEXT:    vle64.v v8, (a4)
 ; CHECK-NEXT:    add a4, a3, a5
 ; CHECK-NEXT:    vle64.v v9, (a4)
 ; CHECK-NEXT:    vfadd.vv v8, v8, v9
 ; CHECK-NEXT:    add a4, a1, a5
 ; CHECK-NEXT:    vse64.v v8, (a4)
-; CHECK-NEXT:    add t1, t1, a6
+; CHECK-NEXT:    add t0, t0, a6
 ; CHECK-NEXT:    add a5, a5, a7
-; CHECK-NEXT:    blt t1, a0, .LBB11_2
+; CHECK-NEXT:    blt t0, a0, .LBB11_2
 ; CHECK-NEXT:  .LBB11_3: # %for.end
 ; CHECK-NEXT:    ret
 entry:
   %0 = tail call i64 @llvm.riscv.vsetvlimax.i64(i64 3, i64 0)
   %cmp13 = icmp sgt i64 %N, 0
   br i1 %cmp13, label %for.body, label %for.end
 
 for.body:                                         ; preds = %entry, %for.body