[RISCV] Initial support of LoopVectorizer for RISC-V Vector.

[HsiangKai]

Summary

Define an option -riscv-vector-bits-max to specify the maximum vector
bits for vectorizer. Loop vectorizer will use the value to check if it
is safe to use the whole vector registers to vectorize the loop.

It is not the optimum solution for loop vectorizing for scalable vector.
It assumed the whole vector registers will be used to vectorize the code.
If it is possible, we should configure vl to do vectorize instead of
using whole vector registers.

[HsiangKai]

Rebase.

[craig.topper]

llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h

330

I'm not sure this should be in this file. This file belongs to the MC layer, but this isn't an MC layer property or a property of the V extension. It's a property of our CodeGen implementation.

I'm not sure where a better place to put it is.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

4576

You can probably just the element VTs and then check that Align == ElemVT.getStoreSize() rather than spelling out all of the alignments.

What is considered misaligned for scalable vectors? Should we be checking the alignement is >= the element size?

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

55

Is this used with scalable vectors?

AArch64 seems to base their return here for SVE on a command line, but I expected them to require specific scalable vector types in IR for the backend to work.

llvm/test/Transforms/LoopVectorize/RISCV/scalable-vf-hint.ll

1

Is this directory new? If so it needs a lit.local.cfg to mark that all tests in it require the RISCV target to be compiled

[rogfer01]

Hi Kai, are we OK with having a test that goes from IR to assembly in the Transforms component?

I'd expect here a vectorized IR test. Then we can add tests those inputs Codegen so they generate sensible RVV instructions.

[frasercrmck]

In D95659#2530358, @rogfer01 wrote:

Hi Kai, are we OK with having a test that goes from IR to assembly in the Transforms component?

I'd expect here a vectorized IR test. Then we can add tests those inputs Codegen so they generate sensible RVV instructions.

I'd prefer this too.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

4576

Yeah, I agree. For RVV we just need to check that the vectors are at least aligned to the element size, don't we? I reckon Align >= EltVT.getStoreSize() is sufficient.

[vkmr]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

20

Minor nit: Reword the description for more clarity - may be something like "Maximum vector register size in bits"?

105

Nit: Use call to getRegisterBitWidth() here instead of RISCVVType::RVVBitsPerBlock. (Or implement and use getMinVectorRegisterBitWidth())

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

If I understand correctly, the assumption behind this code is that a single vector register is of size vscale x RVVBitsPerBlock and ignore the idea (for now?) of having register groups, i.e LMUL>1.
Unless we are ignoring register grouping for now, from Loop Vectorizer's perspective it would make sense to view the register group size as the real register size, specially for computing a feasible VF based on register usage.
Since the documentation of getRegisterBitWidth() defines it to be "The width of the largest scalar or vector register type", it might be more accurate to use getMinVectorRegisterBitWidth() to return RISCVVType::RVVBitsPerBlock and getRegisterBitWidth() to return getMinVectorRegisterBitWidth() * MAX_LMUL. (I am not considering fractional LMUL here.)

[craig.topper]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

105

getRegisterWidth() is likely going to be updated to be similar to AArch64 and be controlled by a command line option for minimum width. So it won't be the right thing.

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

Returning a non-zero value seems to at least partially enable the vectorizer to generate fixed vectors which isn't supported by the backend yet. It looks like something else stopped it in my testing, but it at least queried the cost model. Not sure what stopped it.

I do plan to support fixed vectors in the RVV backend, but it will probably be a couple weeks away. The register width here will probably need to be a command line controlled value like AArch64. And it should be at least 128 bits per the 0.10 spec. So I don't think its connected to RVVBitsPerBlock.

[vkmr]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

Perhaps I misunderstood something, my concern here is more about how to encapsulate the idea of register grouping for scalable vectors in the TTI methods to query register widths. Having a command line option to control register width would still only reflect the width of a single register, right? Perhaps, we can add another command line option to specify a max group multiplier (essentially the Maximum LMUL value).

IIRC, the TTI method getMinVectorRegisterBitWidth() in addition to getRegisterBitWidth() was introduced to handle similar concerns with NEON. With scalable vectors, things are a little more complicated.

[craig.topper]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

I don't think I understand how this interface works for scalable vector vectorization. AArch64 has it connected to a command line which means it can be larger than 128 bits. But I thought the backend needed specific types like <vscale x 4 x i32>. Does this interface effect the fixed portion of the scalable type for scalable vector vectorization?

[HsiangKai]

llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h

330

I will move it to RISCVISelLowering.h.

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

I think I didn't dig into how the callback is used. I remove it in this patch. We could add it back after we have clear idea how to do it.

[HsiangKai]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

Craig is right. getRegisterBitWidth() is not related to scalable vector vectorization. It is reasonable to remove it in this patch.

[vkmr]

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

52–55

Does this interface effect the fixed portion of the scalable type for scalable vector vectorization?

Yes, the auto vectorizer uses this interface to compute the VF (for scalable vectors this is the fixed part of the VF) with the most optimal cost.

Craig is right. getRegisterBitWidth() is not related to scalable vector vectorization. It is reasonable to remove it in this patch.

Agreed.

[craig.topper]

LGTM

[frasercrmck]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

4564

/// .... If true, it also returns
/// whether the unaligned memory access is "fast" in the last argument by
/// reference.

I think this suggests that setting Fast is part of the contract. It could theoretically read an uninitialized variable if we don't set it but return true.

[HsiangKai]

Address @frasercrmck's comments.

[frasercrmck]

LGTM

llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h

 }
 
 inline static bool isTailAgnostic(unsigned VType) { return VType & 0x40; }
 
 inline static bool isMaskAgnostic(unsigned VType) { return VType & 0x80; }
 
 void printVType(unsigned VType, raw_ostream &OS);
 
+static constexpr unsigned RVVBitsPerBlock = 64;
+
 } // namespace RISCVVType
 
 } // namespace llvm
 
 #endif

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
   TargetLowering::AtomicExpansionKind
   shouldExpandAtomicCmpXchgInIR(AtomicCmpXchgInst *CI) const override;
   Value *emitMaskedAtomicCmpXchgIntrinsic(IRBuilder<> &Builder,
                                           AtomicCmpXchgInst *CI,
                                           Value *AlignedAddr, Value *CmpVal,
                                           Value *NewVal, Value *Mask,
                                           AtomicOrdering Ord) const override;
 
+  /// Returns true if the target allows unaligned memory accesses of the
+  /// specified type.
+  bool allowsMisalignedMemoryAccesses(
+      EVT VT, unsigned AddrSpace = 0, unsigned Align = 1,
+      MachineMemOperand::Flags Flags = MachineMemOperand::MONone,
+      bool *Fast = nullptr) const override;
+
 private:
   void analyzeInputArgs(MachineFunction &MF, CCState &CCInfo,
                         const SmallVectorImpl<ISD::InputArg> &Ins,
                         bool IsRet) const;
   void analyzeOutputArgs(MachineFunction &MF, CCState &CCInfo,
                          const SmallVectorImpl<ISD::OutputArg> &Outs,
                          bool IsRet, CallLoweringInfo *CLI) const;
 

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if (auto *ConstNode = dyn_cast<ConstantSDNode>(C.getNode())) {
         return true;
       }
     }
   }
 
   return false;
 }
 
+bool RISCVTargetLowering::allowsMisalignedMemoryAccesses(
+    EVT VT, unsigned AddrSpace, unsigned Align, MachineMemOperand::Flags Flags,
+    bool *Fast) const {
+  if (!VT.isScalableVector())
+    return false;
+
+  EVT ElemVT = VT.getVectorElementType();
+  if ((ElemVT == MVT::i8 && Align == 1) ||
+      (ElemVT == MVT::i16 && Align == 2) ||
+      (ElemVT == MVT::i32 && Align == 4) ||
+      (ElemVT == MVT::i64 && Align == 8) ||
+      (ElemVT == MVT::f16 && Align == 2) ||
+      (ElemVT == MVT::f32 && Align == 4) ||
+      (ElemVT == MVT::f64 && Align == 8))
+    return true;
+
+  return false;
+}
+
 #define GET_REGISTER_MATCHER
 #include "RISCVGenAsmMatcher.inc"
 
 Register
 RISCVTargetLowering::getRegisterByName(const char *RegName, LLT VT,
                                        const MachineFunction &MF) const {
   Register Reg = MatchRegisterAltName(RegName);
   if (Reg == RISCV::NoRegister)

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

explicit RISCVTTIImpl(const RISCVTargetMachine *TM, const Function &F)
         TLI(ST->getTargetLowering()) {}
 
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
   int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
                         Type *Ty, TTI::TargetCostKind CostKind,
                         Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
+
+  bool supportsScalableVectors() const { return ST->hasStdExtV(); }
+  unsigned getRegisterBitWidth(bool Vector) const {
+    if (Vector) {
+      if (ST->hasStdExtV())
+        return RISCVVType::RVVBitsPerBlock;
+      return 0;
+    }
+    if (ST->is64Bit())
+      return 64;
+    return 32;
+  }
+  Optional<unsigned> getMaxVScale() const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVTARGETTRANSFORMINFO_H

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

 #include "MCTargetDesc/RISCVMatInt.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/BasicTTIImpl.h"
 #include "llvm/CodeGen/TargetLowering.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "riscvtti"
 
+static cl::opt<unsigned> MaxVectorLen(
+  "riscv-vector-bits-max",
+  cl::desc("The assumed maximum vector bits."),
+  cl::init(0), cl::Hidden);
+
 int RISCVTTIImpl::getIntImmCost(const APInt &Imm, Type *Ty,
                                 TTI::TargetCostKind CostKind) {
   assert(Ty->isIntegerTy() &&
          "getIntImmCost can only estimate cost of materialising integers");
 
   // We have a Zero register, so 0 is always free.
   if (Imm == 0)
     return TTI::TCC_Free;

 }
 
 int RISCVTTIImpl::getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                       const APInt &Imm, Type *Ty,
                                       TTI::TargetCostKind CostKind) {
   // Prevent hoisting in unknown cases.
   return TTI::TCC_Free;
 }
+
+Optional<unsigned> RISCVTTIImpl::getMaxVScale() const {
+  if (ST->hasStdExtV())
+    return MaxVectorLen / RISCVVType::RVVBitsPerBlock;
+  return BaseT::getMaxVScale();
+}

llvm/test/Transforms/LoopVectorize/RISCV/scalable-vf-hint.ll

+; RUN: opt -mtriple=riscv64 -mattr=+m,+experimental-v -loop-vectorize \
+; RUN:   -riscv-vector-bits-max=512 -S < %s 2>&1 \
+; RUN:   | llc -mattr=+m,+experimental-v | FileCheck %s
+
+; void test(int *a, int *b, int N) {
+;   #pragma clang loop vectorize(enable) vectorize_width(2, scalable)
+;   for (int i=0; i<N; ++i) {
+;     a[i + 64] = a[i] + b[i];
+;   }
+; }
+;
+; CHECK: vsetvli a3, zero, e32,m1,ta,mu
+; CHECK: vle32.v v25, (a2)
+; CHECK: add     a3, a1, a5
+; CHECK: vle32.v v26, (a3)
+; CHECK: vadd.vv v25, v26, v25
+; CHECK: addi    a2, a2, 256
+; CHECK: add     a4, a4, a6
+; CHECK: vse32.v v25, (a2)
+define void @test(i32* %a, i32* %b) {
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i32, i32* %b, i64 %iv
+  %1 = load i32, i32* %arrayidx2, align 4
+  %add = add nsw i32 %1, %0
+  %2 = add nuw nsw i64 %iv, 64
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %2
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !6
+
+exit:
+  ret void
+}
+
+!6 = !{!6, !7, !8}
+!7 = !{!"llvm.loop.vectorize.width", i32 2}
+!8 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}