[RISCV] Don't use zero-stride vector load if there's no optimized u-arch

[• pcwang-thead]

Summary

For vector strided instructions, as the RVV spec says:

When rs2=x0, then an implementation is allowed, but not required, to
perform fewer memory operations than the number of active elements, and
may perform different numbers of memory operations across different
dynamic executions of the same static instruction.

So compiler shouldn't assume that fewer memory operations will be
performed when rs2=x0.

We add a target feature to specify whether u-arch supports optimized
zero-stride vector load. And we do vector splat optimization iff this
feature is supported.

This feature is enabled by default since most designs implement this
optimization.

[reames]

I have no problem with adding a feature to disable this optimization, but there's two problems with this patch.

First, a bunch of existing CPUs do support this optimization. Your change doesn't update any of them, so this is strictly a regression.

Second, most designs I'm aware of do implement this optimization. As such, I really think this patch gets the default wrong. We should allow disabling optimizations for code targeting designs which don't optimize this case, but if we're targeting a generic riscv64 vector core, assuming this optimization should be the default.

Also, are you are of a specific design which doesn't optimize this case? If not, the complexity doesn't seem worthwhile.

[• pcwang-thead]

Enbale this feature by default.

[• pcwang-thead]

In D137699#3919799, @reames wrote:

I have no problem with adding a feature to disable this optimization, but there's two problems with this patch.

First, a bunch of existing CPUs do support this optimization. Your change doesn't update any of them, so this is strictly a regression.

Second, most designs I'm aware of do implement this optimization. As such, I really think this patch gets the default wrong. We should allow disabling optimizations for code targeting designs which don't optimize this case, but if we're targeting a generic riscv64 vector core, assuming this optimization should be the default.

Thanks! I do agree with you and I have made it default now.

Also, are you are of a specific design which doesn't optimize this case? If not, the complexity doesn't seem worthwhile.

Yes, some of our taped-out low-end products didn't do this optimization.

[craig.topper]

llvm/lib/Target/RISCV/RISCV.td

455

FeatureNoOptimizedZeroStrideLoad -> TuneNoOptimizedZeroStrideLoad.

[• pcwang-thead]

Rename FeatureNoOptimizedZeroStrideLoad to TuneNoOptimizedZeroStrideLoad

[reames]

LGTM

If you're interested in optimizing for such a target, I'd suggest a follow up. We should probably be canonicalizing in the other direction (i.e. replace a zero stride load with a load and splat). A zero stride load can probably be matched during gather lowering.

[• pcwang-thead]

In D137699#3921845, @reames wrote:

LGTM

If you're interested in optimizing for such a target, I'd suggest a follow up. We should probably be canonicalizing in the other direction (i.e. replace a zero stride load with a load and splat). A zero stride load can probably be matched during gather lowering.

Thanks! I will have a try.

llvm/lib/Target/RISCV/RISCV.td

 def FeatureSaveRestore : SubtargetFeature<"save-restore", "EnableSaveRestore",
                                           "true", "Enable save/restore.">;
 
 def FeatureUnalignedScalarMem
    : SubtargetFeature<"unaligned-scalar-mem", "EnableUnalignedScalarMem",
                       "true", "Has reasonably performant unaligned scalar "
                       "loads and stores">;
 
+def TuneNoOptimizedZeroStrideLoad
+   : SubtargetFeature<"no-optimized-zero-stride-load", "HasOptimizedZeroStrideLoad",
+                      "false", "Hasn't optimized (perform fewer memory operations)"
+                      "zero-stride vector load">;
+
 def TuneLUIADDIFusion
     : SubtargetFeature<"lui-addi-fusion", "HasLUIADDIFusion",
                        "true", "Enable LUI+ADDI macrofusion">;
 
 def TuneNoDefaultUnroll
     : SubtargetFeature<"no-default-unroll", "EnableDefaultUnroll", "false",
                        "Disable default unroll preference.">;
 

llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp

case ISD::EXTRACT_SUBVECTOR: {
     SDValue Extract = CurDAG->getTargetExtractSubreg(SubRegIdx, DL, VT, V);
     ReplaceNode(Node, Extract.getNode());
     return;
   }
   case RISCVISD::VMV_S_X_VL:
   case RISCVISD::VFMV_S_F_VL:
   case RISCVISD::VMV_V_X_VL:
   case RISCVISD::VFMV_V_F_VL: {
+    // Only if we have optimized zero-stride vector load.
+    if (!Subtarget->hasOptimizedZeroStrideLoad())
+      break;
+
     // Try to match splat of a scalar load to a strided load with stride of x0.
     bool IsScalarMove = Node->getOpcode() == RISCVISD::VMV_S_X_VL ||
                         Node->getOpcode() == RISCVISD::VFMV_S_F_VL;
     if (!Node->getOperand(0).isUndef())
       break;
     SDValue Src = Node->getOperand(1);
     auto *Ld = dyn_cast<LoadSDNode>(Src);
     if (!Ld)

llvm/lib/Target/RISCV/RISCVSubtarget.h

private:
   bool EnableLinkerRelax = false;
   bool EnableRVCHintInstrs = true;
   bool EnableDefaultUnroll = true;
   bool EnableSaveRestore = false;
   bool EnableUnalignedScalarMem = false;
   bool HasShortForwardBranchOpt = false;
   bool HasLUIADDIFusion = false;
   bool HasForcedAtomics = false;
+  bool HasOptimizedZeroStrideLoad = true;
   unsigned XLen = 32;
   unsigned ZvlLen = 0;
   MVT XLenVT = MVT::i32;
   uint8_t MaxInterleaveFactor = 2;
   RISCVABI::ABI TargetABI = RISCVABI::ABI_Unknown;
   std::bitset<RISCV::NUM_TARGET_REGS> UserReservedRegister;
   RISCVFrameLowering FrameLowering;
   RISCVInstrInfo InstrInfo;
public:
   bool enableLinkerRelax() const { return EnableLinkerRelax; }
   bool enableRVCHintInstrs() const { return EnableRVCHintInstrs; }
   bool enableDefaultUnroll() const { return EnableDefaultUnroll; }
   bool enableSaveRestore() const { return EnableSaveRestore; }
   bool hasShortForwardBranchOpt() const { return HasShortForwardBranchOpt; }
   bool enableUnalignedScalarMem() const { return EnableUnalignedScalarMem; }
   bool hasLUIADDIFusion() const { return HasLUIADDIFusion; }
   bool hasForcedAtomics() const { return HasForcedAtomics; }
+  bool hasOptimizedZeroStrideLoad() const { return HasOptimizedZeroStrideLoad; }
   MVT getXLenVT() const { return XLenVT; }
   unsigned getXLen() const { return XLen; }
   unsigned getFLen() const {
     if (HasStdExtD)
       return 64;
 
     if (HasStdExtF)
       return 32;

llvm/test/CodeGen/RISCV/rvv/vsplats-fp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -mattr=+f,+d,+zfh,+experimental-zvfh,+v -target-abi ilp32d -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck %s --check-prefixes=CHECK,OPTIMIZED
 ; RUN: llc -mtriple=riscv64 -mattr=+f,+d,+zfh,+experimental-zvfh,+v -target-abi lp64d -verify-machineinstrs < %s \
-; RUN:   | FileCheck %s
+; RUN:   | FileCheck %s --check-prefixes=CHECK,OPTIMIZED
+; RUN: llc -mtriple=riscv32 -mattr=+f,+d,+zfh,+experimental-zvfh,+v,+no-optimized-zero-stride-load -target-abi ilp32d -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefixes=CHECK,NOT-OPTIMIZED
+; RUN: llc -mtriple=riscv64 -mattr=+f,+d,+zfh,+experimental-zvfh,+v,+no-optimized-zero-stride-load -target-abi lp64d -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s --check-prefixes=CHECK,NOT-OPTIMIZED
 
 define <vscale x 8 x half> @vsplat_nxv8f16(half %f) {
 ; CHECK-LABEL: vsplat_nxv8f16:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli a0, zero, e16, m2, ta, ma
 ; CHECK-NEXT:    vfmv.v.f v8, fa0
 ; CHECK-NEXT:    ret
   %head = insertelement <vscale x 8 x half> poison, half %f, i32 0

 ; CHECK-NEXT:    ret
   %head = insertelement <vscale x 8 x double> poison, double zeroinitializer, i32 0
   %splat = shufflevector <vscale x 8 x double> %head, <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x double> %splat
 }
 
 ; Test that we fold this to a vlse with 0 stride.
 define <vscale x 8 x float> @vsplat_load_nxv8f32(float* %ptr) {
-; CHECK-LABEL: vsplat_load_nxv8f32:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a1, zero, e32, m4, ta, ma
-; CHECK-NEXT:    vlse32.v v8, (a0), zero
-; CHECK-NEXT:    ret
+; OPTIMIZED-LABEL: vsplat_load_nxv8f32:
+; OPTIMIZED:       # %bb.0:
+; OPTIMIZED-NEXT:    vsetvli a1, zero, e32, m4, ta, ma
+; OPTIMIZED-NEXT:    vlse32.v v8, (a0), zero
+; OPTIMIZED-NEXT:    ret
+;
+; NOT-OPTIMIZED-LABEL: vsplat_load_nxv8f32:
+; NOT-OPTIMIZED:       # %bb.0:
+; NOT-OPTIMIZED-NEXT:    flw ft0, 0(a0)
+; NOT-OPTIMIZED-NEXT:    vsetvli a0, zero, e32, m4, ta, ma
+; NOT-OPTIMIZED-NEXT:    vfmv.v.f v8, ft0
+; NOT-OPTIMIZED-NEXT:    ret
   %f = load float, float* %ptr
   %head = insertelement <vscale x 8 x float> poison, float %f, i32 0
   %splat = shufflevector <vscale x 8 x float> %head, <vscale x 8 x float> poison, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x float> %splat
 }