[RISCV] Lower experimental_get_vector_length intrinsic to vsetvli for some cases.

[craig.topper]

Summary

This patch lowers to vsetvli when the AVL is i32 or XLenVT and
the VF is a power of 2 in the range [1, 64]. VLEN=32 is not supported
as we don't have a valid type mapping for that. VF=1 is not supported
with Zve32* only.

The element width is used to set the SEW for the vsetvli if possible.
Otherwise we use SEW=8.

[asb]

I think you meant to mark this as dependent on D149916.

[craig.topper]

Rebase on parent patch

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1184

For fixed vectors, we should be able to statically compute this when the ElementWidth * VF is less than VLEN right? It's fine to do that in a different patch, just want to make sure I'm not missing something.

1187

Why not i16? I don't see anything in the implementation which couldn't be handled via an extend?

6526

The "if possible" bit is bugging me here. Isn't this a hard requirement? VLMAX should be a function of SEW and LMUL. Given the result here depends on VLMAX, don't we have a hard requirement on having a recognized SEW/LMUL combination?

6535

encodeSEW

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1184

The ElementWidth doesn't have any real meaning. The vectorizer already picked a VF. The get_vector_length is asking, given that VF, how many elements fit in that type. The width of the elements or VLEN doesn't change that answer.

I guess we could return a lower number but that means we'd never use the entire type the vectorizer is emitting. That seems like we a cost model issue if we actually want to use less than the type the vectorizer picks.

1187

No strong reason. Might need to add setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i16, Custom) to get the legalization to be called.

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1184

I see your point for fixed length vectors. Essentially, we can let the vectorizer pick one via cost modeling, and then split as needed. So here, we can report the number of elements even if splitting might be required.

[craig.topper]

Rebase after removing element width from the intrinsic.

[reames]

LGTM w/comment addressed.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

6522

This depends very heavily on the fact that VLMAX for 1 x i8 and VLMAX for 1 x i64 are the same. It took me a while to convince myself that this was actually true. The former maps to a fractional lmul, the later maps to m1. And as a result, the element width is irrelevant.

Can you add a comment which explains this in a bit more detail? If this is already well explained elsewhere, you can simply point to that comment.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

6522

A lot of this needs to be rewritten using RVVBitsPerBlock. I'll update.

[craig.topper]

Use RVVBitsPerBlock to define the math.
Add explanatory comments.

[craig.topper]

Requesting review to make sure the new comments are sufficient.

[reames]

Ok, yeah, that's a lot easier to follow. Thanks!

LGTM

llvm/lib/Target/RISCV/RISCVISelLowering.h

 //===-- RISCVISelLowering.h - RISC-V DAG Lowering Interface -----*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
private:
   void validateCCReservedRegs(
       const SmallVectorImpl<std::pair<llvm::Register, llvm::SDValue>> &Regs,
       MachineFunction &MF) const;
 
   bool useRVVForFixedLengthVectorVT(MVT VT) const;
 
   MVT getVPExplicitVectorLengthTy() const override;
 
+  bool shouldExpandGetVectorLength(EVT TripCountVT, unsigned VF,
+                                   bool IsScalable) const override;
+
   /// RVV code generation for fixed length vectors does not lower all
   /// BUILD_VECTORs. This makes BUILD_VECTOR legalisation a source of stores to
   /// merge. However, merging them creates a BUILD_VECTOR that is just as
   /// illegal as the original, thus leading to an infinite legalisation loop.
   /// NOTE: Once BUILD_VECTOR can be custom lowered for all legal vector types,
   /// this override can be removed.
   bool mergeStoresAfterLegalization(EVT VT) const override;
 

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

 //===-- RISCVISelLowering.cpp - RISC-V DAG Lowering Implementation  -------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (Subtarget.hasVInstructions() &&
     return EVT::getVectorVT(Context, MVT::i1, VT.getVectorElementCount());
   return VT.changeVectorElementTypeToInteger();
 }
 
 MVT RISCVTargetLowering::getVPExplicitVectorLengthTy() const {
   return Subtarget.getXLenVT();
 }
 
+// Return false if we can lower get_vector_length to a vsetvli intrinsic.
+bool RISCVTargetLowering::shouldExpandGetVectorLength(EVT TripCountVT,
+                                                      unsigned VF,
+                                                      bool IsScalable) const {
+  if (!Subtarget.hasVInstructions())
+    return true;
+
+  if (!IsScalable)
+    return true;
+
+  if (TripCountVT != MVT::i32 && TripCountVT != Subtarget.getXLenVT())
+    return true;
+
+  // Don't allow VF=1 if those types are't legal.
+  if (VF < RISCV::RVVBitsPerBlock / Subtarget.getELEN())
+    return true;
+
+  // VLEN=32 support is incomplete.
+  if (Subtarget.getRealMinVLen() < RISCV::RVVBitsPerBlock)
+    return true;
+
+  // We can only use vsetvli for power of 2 in the range [1, 64].
+  return VF > 64 || !isPowerOf2_32(VF);
+}
+
 bool RISCVTargetLowering::getTgtMemIntrinsic(IntrinsicInfo &Info,
                                              const CallInst &I,
                                              MachineFunction &MF,
                                              unsigned Intrinsic) const {
   auto &DL = I.getModule()->getDataLayout();
 
   auto SetRVVLoadStoreInfo = [&](unsigned PtrOp, bool IsStore,
                                  bool IsUnitStrided) {
static SDValue lowerVectorIntrinsicScalars(SDValue Op, SelectionDAG &DAG,
 
   // We need to convert the scalar to a splat vector.
   SDValue VL = getVLOperand(Op);
   assert(VL.getValueType() == XLenVT);
   ScalarOp = splatSplitI64WithVL(DL, VT, SDValue(), ScalarOp, VL, DAG);
   return DAG.getNode(Op->getOpcode(), DL, Op->getVTList(), Operands);
 }
 
+static SDValue lowerGetVectorLength(SDNode *N, SelectionDAG &DAG,
+                                    const RISCVSubtarget &Subtarget) {
+  MVT XLenVT = Subtarget.getXLenVT();
+
+  unsigned ElementWidth = N->getConstantOperandVal(2);
+  // Sanitize the ElementWidth.
+  if (ElementWidth < 8 || ElementWidth > Subtarget.getELEN() ||
+      !isPowerOf2_64(ElementWidth))
+    ElementWidth = 8;
+  unsigned VF = N->getConstantOperandVal(3);
+  assert(VF >= 1 && VF <= 64 && isPowerOf2_32(VF) && "Unexpected VF");
+  // We want to use an SEW and LMUL that matches the ElementWidth if possible.
+  unsigned Factor = ElementWidth / 8;
+  if (VF * Factor <= 64)
+    VF = VF * Factor;
+  else
+    ElementWidth = 8;
+
+  unsigned LMulExp = Log2_32(VF);
+  unsigned LMulVal = LMulExp >= 3 ? LMulExp - 3 : LMulExp + 5;
+  unsigned SewVal = Log2_32(ElementWidth) - 3;
+
+  SDLoc DL(N);
+
+  SDValue LMul = DAG.getTargetConstant(LMulVal, DL, XLenVT);
+  SDValue Sew = DAG.getTargetConstant(SewVal, DL, XLenVT);
+
+  SDValue AVL = DAG.getNode(ISD::ZERO_EXTEND, DL, XLenVT, N->getOperand(1));
+
+  SDValue ID = DAG.getTargetConstant(Intrinsic::riscv_vsetvli, DL, XLenVT);
+  return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, XLenVT, ID, AVL, Sew, LMul);
+}
+
 SDValue RISCVTargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
                                                      SelectionDAG &DAG) const {
   unsigned IntNo = Op.getConstantOperandVal(0);
   SDLoc DL(Op);
   MVT XLenVT = Subtarget.getXLenVT();
 
   switch (IntNo) {
   default:
case Intrinsic::riscv_brev8: {
     return DAG.getNode(Opc, DL, XLenVT, Op.getOperand(1));
   }
   case Intrinsic::riscv_zip:
   case Intrinsic::riscv_unzip: {
     unsigned Opc =
         IntNo == Intrinsic::riscv_zip ? RISCVISD::ZIP : RISCVISD::UNZIP;
     return DAG.getNode(Opc, DL, XLenVT, Op.getOperand(1));
   }
+  case Intrinsic::experimental_get_vector_length:
+    return lowerGetVectorLength(Op.getNode(), DAG, Subtarget);
   case Intrinsic::riscv_vmv_x_s:
     assert(Op.getValueType() == XLenVT && "Unexpected VT!");
     return DAG.getNode(RISCVISD::VMV_X_S, DL, Op.getValueType(),
                        Op.getOperand(1));
   case Intrinsic::riscv_vfmv_f_s:
     return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, Op.getValueType(),
                        Op.getOperand(1), DAG.getConstant(0, DL, XLenVT));
   case Intrinsic::riscv_vmv_v_x:
case ISD::EXTRACT_VECTOR_ELT: {
     break;
   }
   case ISD::INTRINSIC_WO_CHAIN: {
     unsigned IntNo = cast<ConstantSDNode>(N->getOperand(0))->getZExtValue();
     switch (IntNo) {
     default:
       llvm_unreachable(
           "Don't know how to custom type legalize this intrinsic!");
+    case Intrinsic::experimental_get_vector_length: {
+      SDValue Res = lowerGetVectorLength(N, DAG, Subtarget);
+      Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i32, Res));
+      return;
+    }
     case Intrinsic::riscv_orc_b: {
       SDValue NewOp =
           DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, N->getOperand(1));
       SDValue Res = DAG.getNode(RISCVISD::ORC_B, DL, MVT::i64, NewOp);
       Results.push_back(DAG.getNode(ISD::TRUNCATE, DL, MVT::i32, Res));
       return;
     }
     case Intrinsic::riscv_vmv_x_s: {

llvm/test/CodeGen/RISCV/rvv/get_vector_length.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
-; RUN: sed 's/iXLen/i32/g' %s | llc -mtriple=riscv32 -mattr=+v -verify-machineinstrs | FileCheck %s --check-prefixes=CHECK,RV32
-; RUN: sed 's/iXLen/i32/g' %s | llc -mtriple=riscv64 -mattr=+v -verify-machineinstrs | FileCheck %s --check-prefixes=CHECK,RV64
+; RUN: sed 's/iXLen/i32/g' %s | llc -mtriple=riscv32 -mattr=+m,+v -verify-machineinstrs | FileCheck %s --check-prefixes=CHECK,RV32
+; RUN: sed 's/iXLen/i32/g' %s | llc -mtriple=riscv64 -mattr=+m,+v -verify-machineinstrs | FileCheck %s --check-prefixes=CHECK,RV64
 
 declare i32 @llvm.experimental.get.vector.length.i16(i16, i32, i32, i1)
 declare i32 @llvm.experimental.get.vector.length.i32(i32, i32, i32, i1)
 declare i32 @llvm.experimental.get.vector.length.i64(i64, i32, i32, i1)
 
 define i32 @vector_length_i16(i16 zeroext %tc) {
 ; CHECK-LABEL: vector_length_i16:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    csrr a1, vlenb
 ; CHECK-NEXT:    srli a1, a1, 2
 ; CHECK-NEXT:    bltu a0, a1, .LBB0_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a0, a1
 ; CHECK-NEXT:  .LBB0_2:
 ; CHECK-NEXT:    ret
   %a = call i32 @llvm.experimental.get.vector.length.i16(i16 %tc, i32 8, i32 2, i1 true)
   ret i32 %a
 }
 
 define i32 @vector_length_i32(i32 zeroext %tc) {
-; RV32-LABEL: vector_length_i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    csrr a1, vlenb
-; RV32-NEXT:    srli a1, a1, 2
-; RV32-NEXT:    bltu a0, a1, .LBB1_2
-; RV32-NEXT:  # %bb.1:
-; RV32-NEXT:    mv a0, a1
-; RV32-NEXT:  .LBB1_2:
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: vector_length_i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    sext.w a0, a0
-; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    srli a1, a1, 2
-; RV64-NEXT:    bltu a0, a1, .LBB1_2
-; RV64-NEXT:  # %bb.1:
-; RV64-NEXT:    mv a0, a1
-; RV64-NEXT:  .LBB1_2:
-; RV64-NEXT:    ret
+; CHECK-LABEL: vector_length_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, mf4, ta, ma
+; CHECK-NEXT:    ret
   %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 8, i32 2, i1 true)
   ret i32 %a
 }
 
 define i32 @vector_length_XLen(iXLen zeroext %tc) {
-; RV32-LABEL: vector_length_XLen:
-; RV32:       # %bb.0:
-; RV32-NEXT:    csrr a1, vlenb
-; RV32-NEXT:    srli a1, a1, 2
-; RV32-NEXT:    bltu a0, a1, .LBB2_2
-; RV32-NEXT:  # %bb.1:
-; RV32-NEXT:    mv a0, a1
-; RV32-NEXT:  .LBB2_2:
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: vector_length_XLen:
-; RV64:       # %bb.0:
-; RV64-NEXT:    sext.w a0, a0
-; RV64-NEXT:    csrr a1, vlenb
-; RV64-NEXT:    srli a1, a1, 2
-; RV64-NEXT:    bltu a0, a1, .LBB2_2
-; RV64-NEXT:  # %bb.1:
-; RV64-NEXT:    mv a0, a1
-; RV64-NEXT:  .LBB2_2:
-; RV64-NEXT:    ret
+; CHECK-LABEL: vector_length_XLen:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, mf4, ta, ma
+; CHECK-NEXT:    ret
   %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 8, i32 2, i1 true)
   ret i32 %a
 }
 
 define i32 @vector_length_i16_fixed(i16 zeroext %tc) {
 ; CHECK-LABEL: vector_length_i16_fixed:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    li a1, 2

 ; RV64-NEXT:    bltu a0, a1, .LBB5_2
 ; RV64-NEXT:  # %bb.1:
 ; RV64-NEXT:    li a0, 2
 ; RV64-NEXT:  .LBB5_2:
 ; RV64-NEXT:    ret
   %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 8, i32 2, i1 false)
   ret i32 %a
 }
+
+define i32 @vector_length_e8vf64_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e8vf64_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 8, i32 64, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e8vf64_XLen(iXLen zeroext %tc) {
+; CHECK-LABEL: vector_length_e8vf64_XLen:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 8, i32 64, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e16vf2_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf2_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e16, mf2, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 16, i32 2, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e16vf2_XLen(iXLen zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf2_XLen:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e16, mf2, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 16, i32 2, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e16vf32_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf32_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e16, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 16, i32 32, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e16vf32_XLen(iXLen zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf32_XLen:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e16, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 16, i32 32, i1 true)
+  ret i32 %a
+}
+
+;; Element width ignored because VF is too large.
+define i32 @vector_length_e16vf64_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf64_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 16, i32 64, i1 true)
+  ret i32 %a
+}
+
+;; Element width ignored because VF is too large.
+define i32 @vector_length_e16vf64_XLen(iXLen zeroext %tc) {
+; CHECK-LABEL: vector_length_e16vf64_XLen:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, m8, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.iXLen(iXLen %tc, i32 16, i32 64, i1 true)
+  ret i32 %a
+}
+
+;; Element width ignored
+define i32 @vector_length_e7vf2_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e7vf2_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, mf4, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 7, i32 2, i1 true)
+  ret i32 %a
+}
+
+;; Element width ignored
+define i32 @vector_length_e128vf2_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e128vf2_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, mf4, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 128, i32 2, i1 true)
+  ret i32 %a
+}
+
+;; Element width ignored
+define i32 @vector_length_e9vf2_i32(i32 zeroext %tc) {
+; CHECK-LABEL: vector_length_e9vf2_i32:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli a0, a0, e8, mf4, ta, ma
+; CHECK-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 9, i32 2, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e8vf128_i32(i32 zeroext %tc) {
+; RV32-LABEL: vector_length_e8vf128_i32:
+; RV32:       # %bb.0:
+; RV32-NEXT:    csrr a1, vlenb
+; RV32-NEXT:    slli a1, a1, 4
+; RV32-NEXT:    bltu a0, a1, .LBB17_2
+; RV32-NEXT:  # %bb.1:
+; RV32-NEXT:    mv a0, a1
+; RV32-NEXT:  .LBB17_2:
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: vector_length_e8vf128_i32:
+; RV64:       # %bb.0:
+; RV64-NEXT:    sext.w a0, a0
+; RV64-NEXT:    csrr a1, vlenb
+; RV64-NEXT:    slli a1, a1, 4
+; RV64-NEXT:    bltu a0, a1, .LBB17_2
+; RV64-NEXT:  # %bb.1:
+; RV64-NEXT:    mv a0, a1
+; RV64-NEXT:  .LBB17_2:
+; RV64-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 8, i32 128, i1 true)
+  ret i32 %a
+}
+
+define i32 @vector_length_e8vf3_i32(i32 zeroext %tc) {
+; RV32-LABEL: vector_length_e8vf3_i32:
+; RV32:       # %bb.0:
+; RV32-NEXT:    csrr a1, vlenb
+; RV32-NEXT:    srli a1, a1, 3
+; RV32-NEXT:    slli a2, a1, 1
+; RV32-NEXT:    add a1, a2, a1
+; RV32-NEXT:    bltu a0, a1, .LBB18_2
+; RV32-NEXT:  # %bb.1:
+; RV32-NEXT:    mv a0, a1
+; RV32-NEXT:  .LBB18_2:
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: vector_length_e8vf3_i32:
+; RV64:       # %bb.0:
+; RV64-NEXT:    sext.w a0, a0
+; RV64-NEXT:    csrr a1, vlenb
+; RV64-NEXT:    srli a1, a1, 3
+; RV64-NEXT:    slli a2, a1, 1
+; RV64-NEXT:    add a1, a2, a1
+; RV64-NEXT:    bltu a0, a1, .LBB18_2
+; RV64-NEXT:  # %bb.1:
+; RV64-NEXT:    mv a0, a1
+; RV64-NEXT:  .LBB18_2:
+; RV64-NEXT:    ret
+  %a = call i32 @llvm.experimental.get.vector.length.i32(i32 %tc, i32 9, i32 3, i1 true)
+  ret i32 %a
+}