[RISCV] Support interleaved load lowering

[luke957]

Summary

Lower interleaved load to segment load intrinsic.

[craig.topper]

Don't you need to add the InterleavedAccess pass to RISCVTargetMachine?

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1231

These intrinsics only have SelectionDAG support for scalable vector types. And they take VL as an operand.

[luke957]

Update

[luke957]

In D107210#2918081, @craig.topper wrote:

Don't you need to add the InterleavedAccess pass to RISCVTargetMachine?

Yeah, we need to do this. I have updated the pacth.

[luke957]

ping

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1289

Please address these clang-format warnings

[craig.topper]

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

6

Why are we only testing one factor?

[luke957]

Update

[luke957]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1289

Fix format.

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

6

Add test case for factor 3 and 4.

[luke957]

ping

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1201

Does this need to check if floating point elements are supported?

1225

Please fix this clang-tidy warning

1227

This is not the correct way to convert a fixed vector type to a scalable vector type. For large vectors this will create a type that maps to something larger than LMUL=8. We need to map fixed vector types to scalable vector types using the logic from RISCVTargetLowering::getContainerForFixedLengthVector which takes into account a user provided VLEN via -riscv-v-vector-bits-min command line option.

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

9

Why is the VL 1? The vector type has 16 elements, the interleave factor is 2. So the VL should be 8.

[craig.topper]

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

18

Please make sure the tests can't be optimized to nothing. These shuffles aren't used so are allowed to be deleted.

[luke957]

Clang tidy

[luke957]

Update

[luke957]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1201

Em, is it enough to check bit size only? I think floating point elements will not produce new bit size.

1225

Sorry, I should clang tidy every diff before uploading.

1227

Em, I'm a little confused here. I understand RISCVTargetLowering::getContainerForFixedLengthVector is used in backend when SDAG is built as this method operates on MVT types. So if we just use opt in the middle-end, is this still needed? Further more, if something larger than LMUL=8 appears, could it be legalized by the backend?

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

9

Fix this. It's my mistake.

18

Could we guarantee this by carefully using the RUN instructions in the beginning of the test case?

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1201

You're creating a target specific intrinsic, like Intrinsic::riscv_vlseg2. If the floating point type isn't legal, the backend will error when trying to select the intrinsic. If its not legal and this code doesn't touch it, then the backend will be able to scalarize the loads and shuffles.

1227

All conversions from a fixed vector type to a scalable vector must follow the VLEN passed on the command line in -riscv-v-vector-bits-min. The backend should not be splitting a scalable type if the fixed vector type was supposed to be legal type.

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

18

Technically yes, but I wanted to see how this codegened in SelectionDAG and everything got deleted. So I'd rather have a more robust test.

[luke957]

Update

[luke957]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1201

Thanks. I understand.

1227

Yeah, you are right. It takes me some time to understand that. I think we need to make sure the target specific intrinsic is legal according to the VLEN passed on the command line in -riscv-v-vector-bits-min, so either fixed length vector or scalable vector need legality check (or conversion) for they might exceed the limit of max(LMUL)==8. It seems to be a little bit complicated. Or maybe we could fix this in another patch?

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

18

I understand. Address comment.

[luke957]

Gently ping.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

1227

This needs to be handled in this patch. I don't want to think through issues that can arise by not honoring the VLEN correctly.

It might be easier to add fixed vector instrinsics for segment load/store and handle the conversion from fixed vector to scalable vector inside of SelectionDAG where we already have helper functions for the conversion. This is what I did for the RISCVGatherScatterLowering pass which uses masked_strided_load and masked_strided_store intrinsics.

[luke957]

rebase and update

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
   bool getTgtMemIntrinsic(IntrinsicInfo &Info, const CallInst &I,
                           MachineFunction &MF,
                           unsigned Intrinsic) const override;
   bool isLegalAddressingMode(const DataLayout &DL, const AddrMode &AM, Type *Ty,
                              unsigned AS,
                              Instruction *I = nullptr) const override;
   bool isLegalICmpImmediate(int64_t Imm) const override;
   bool isLegalAddImmediate(int64_t Imm) const override;
+
+  unsigned getMaxSupportedInterleaveFactor() const override { return 4; }
+  bool isLegalInterleavedAccessType(VectorType *VecTy,
+                                    const DataLayout &DL) const;
+  bool lowerInterleavedLoad(LoadInst *LI,
+                            ArrayRef<ShuffleVectorInst *> Shuffles,
+                            ArrayRef<unsigned> Indices,
+                            unsigned Factor) const override;
+
   bool isTruncateFree(Type *SrcTy, Type *DstTy) const override;
   bool isTruncateFree(EVT SrcVT, EVT DstVT) const override;
   bool isZExtFree(SDValue Val, EVT VT2) const override;
   bool isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT) const override;
   bool isCheapToSpeculateCttz() const override;
   bool isCheapToSpeculateCtlz() const override;
   bool hasAndNotCompare(SDValue Y) const override;
   bool shouldSinkOperands(Instruction *I,

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

 #include "RISCV.h"
 #include "RISCVMachineFunctionInfo.h"
 #include "RISCVRegisterInfo.h"
 #include "RISCVSubtarget.h"
 #include "RISCVTargetMachine.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/MemoryLocation.h"
+#include "llvm/Analysis/VectorUtils.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/TargetLoweringObjectFileImpl.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/DiagnosticInfo.h"

 bool RISCVTargetLowering::isLegalICmpImmediate(int64_t Imm) const {
   return isInt<12>(Imm);
 }
 
 bool RISCVTargetLowering::isLegalAddImmediate(int64_t Imm) const {
   return isInt<12>(Imm);
 }
 
+bool RISCVTargetLowering::isLegalInterleavedAccessType(
+    VectorType *VecTy, const DataLayout &DL) const {
+
+  unsigned ElSize = DL.getTypeSizeInBits(VecTy->getElementType());
+
+  unsigned NumElements =
+      isa<ScalableVectorType>(VecTy)
+          ? cast<ScalableVectorType>(VecTy)->getMinNumElements()
+          : cast<FixedVectorType>(VecTy)->getNumElements();
+  // Ensure the number of vector elements is greater than 1.
+  if (NumElements < 2)
+    return false;
+
+  // Ensure the element type is legal.
+  if (ElSize != 8 && ElSize != 16 && ElSize != 32 && ElSize != 64)
+    return false;
+
+  if (VecTy->getElementType()->isHalfTy())
+    return Subtarget.hasStdExtZfh();
+  if (VecTy->getElementType()->isFloatTy())
+    return Subtarget.hasStdExtF();
+  if (VecTy->getElementType()->isDoubleTy())
+    return Subtarget.hasStdExtD();
+
+  return true;
+}
+
+bool RISCVTargetLowering::lowerInterleavedLoad(
+    LoadInst *LI, ArrayRef<ShuffleVectorInst *> Shuffles,
+    ArrayRef<unsigned> Indices, unsigned Factor) const {
+  assert(Factor >= 2 && Factor <= getMaxSupportedInterleaveFactor() &&
+         "Invalid interleave factor");
+  assert(!Shuffles.empty() && "Empty shufflevector input");
+  assert(Shuffles.size() == Indices.size() &&
+         "Unmatched number of shufflevectors and indices");
+
+  const DataLayout &DL = LI->getModule()->getDataLayout();
+
+  VectorType *VTy = Shuffles[0]->getType();
+
+  // Skip if we do not have StdExtV and skip illegal vector types.
+  if (!Subtarget.hasStdExtV() || !isLegalInterleavedAccessType(VTy, DL))
+    return false;
+
+  auto *FVTy = cast<FixedVectorType>(VTy);
+  // FIXME: Support large vectors.
+  if (DL.getTypeSizeInBits(FVTy->getElementType()) * FVTy->getNumElements() >
+      Subtarget.getMinRVVVectorSizeInBits() * 8)
+    return false;
+
+  // A pointer vector can not be the return type of the vlsegN intrinsics. Need
+  // to load integer vectors first and then convert to pointer vectors.
+  Type *EltTy = FVTy->getElementType();
+  if (EltTy->isPointerTy())
+    FVTy =
+        FixedVectorType::get(DL.getIntPtrType(EltTy), FVTy->getNumElements());
+
+  IRBuilder<> Builder(LI);
+
+  // The base address of the load.
+  Value *BaseAddr = LI->getPointerOperand();
+
+  Type *PtrTy =
+      FVTy->getElementType()->getPointerTo(LI->getPointerAddressSpace());
+  Type *IntTy;
+  Triple T = getTargetMachine().getTargetTriple();
+  if (T.isArch64Bit())
+    IntTy = IntegerType::get(FVTy->getContext(), 64);
+  else {
+    assert(T.isArch32Bit() && "only RV32 and RV64 are currently supported");
+    IntTy = IntegerType::get(FVTy->getContext(), 32);
+  }
+
+  Type *Tys[] = {FVTy, PtrTy, IntTy};
+  static const Intrinsic::ID VlsegInts[3] = {Intrinsic::riscv_seg2_load,
+                                             Intrinsic::riscv_seg3_load,
+                                             Intrinsic::riscv_seg4_load};
+  Function *VlsegNFunc =
+      Intrinsic::getDeclaration(LI->getModule(), VlsegInts[Factor - 2], Tys);
+
+  DenseMap<ShuffleVectorInst *, SmallVector<Value *, 4>> SubVecs;
+
+  Value *VL =
+      ConstantInt::get(IntTy, cast<FixedVectorType>(VTy)->getNumElements());
+  CallInst *VlsegN = Builder.CreateCall(
+      VlsegNFunc, {Builder.CreateBitCast(BaseAddr, PtrTy), VL}, "vlsegN");
+
+  // Extract and store the sub-vectors returned by the load intrinsic.
+  for (unsigned I = 0; I < Shuffles.size(); I++) {
+    ShuffleVectorInst *SVI = Shuffles[I];
+    unsigned Index = Indices[I];
+
+    Value *SubVec = Builder.CreateExtractValue(VlsegN, Index);
+
+    // Convert the integer vector to pointer vector if the element is pointer.
+    if (EltTy->isPointerTy())
+      SubVec = Builder.CreateIntToPtr(
+          SubVec, FixedVectorType::get(SVI->getType()->getElementType(),
+                                       FVTy->getNumElements()));
+    SubVecs[SVI].push_back(SubVec);
+  }
+
+  // Replace uses of the shufflevector instructions with the sub-vectors
+  // returned by the load intrinsic. If a shufflevector instruction is
+  // associated with more than one sub-vector, those sub-vectors will be
+  // concatenated into a single wide vector.
+  for (ShuffleVectorInst *SVI : Shuffles) {
+    auto &SubVec = SubVecs[SVI];
+    auto *WideVec =
+        SubVec.size() > 1 ? concatenateVectors(Builder, SubVec) : SubVec[0];
+    SVI->replaceAllUsesWith(WideVec);
+  }
+
+  return true;
+}
+
 // On RV32, 64-bit integers are split into their high and low parts and held
 // in two different registers, so the trunc is free since the low register can
 // just be used.
 bool RISCVTargetLowering::isTruncateFree(Type *SrcTy, Type *DstTy) const {
   if (Subtarget.is64Bit() || !SrcTy->isIntegerTy() || !DstTy->isIntegerTy())
     return false;
   unsigned SrcBits = SrcTy->getPrimitiveSizeInBits();
   unsigned DestBits = DstTy->getPrimitiveSizeInBits();

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

 }
 
 void RISCVPassConfig::addIRPasses() {
   addPass(createAtomicExpandPass());
 
   addPass(createRISCVGatherScatterLoweringPass());
 
   TargetPassConfig::addIRPasses();
+  addPass(createInterleavedAccessPass());
 }
 
 bool RISCVPassConfig::addInstSelector() {
   addPass(createRISCVISelDag(getRISCVTargetMachine()));
 
   return false;
 }
 

llvm/test/CodeGen/RISCV/O0-pipeline.ll

 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
+; CHECK-NEXT:       Interleaved Access Pass
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       Constant Hoisting
 ; CHECK-NEXT:       Replace intrinsics with calls to vector library
 ; CHECK-NEXT:       Partially inline calls to library functions
 ; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       TLS Variable Hoist
+; CHECK-NEXT:       Interleaved Access Pass
 ; CHECK-NEXT:       CodeGen Prepare
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results

llvm/test/Transforms/InterleavedAccess/RISCV/interleaved-accesses.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple riscv64-linux-gnu -interleaved-access -mattr=+v -riscv-v-vector-bits-min=128 \
+; RUN:     -S < %s | FileCheck %s
+
+define <8 x i8> @load_factor2(<16 x i8>* %ptr) {
+; CHECK-LABEL: @load_factor2(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <16 x i8>* [[PTR:%.*]] to i8*
+; CHECK-NEXT:    [[VLSEGN:%.*]] = call { <8 x i8>, <8 x i8> } @llvm.riscv.seg2.load.v8i8.p0i8.i64(i8* [[TMP1]], i64 8)
+; CHECK-NEXT:    [[TMP2:%.*]] = extractvalue { <8 x i8>, <8 x i8> } [[VLSEGN]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = extractvalue { <8 x i8>, <8 x i8> } [[VLSEGN]], 0
+; CHECK-NEXT:    ret <8 x i8> [[TMP2]]
+;
+  %interleaved.vec = load <16 x i8>, <16 x i8>* %ptr, align 4
+  %v0 = shufflevector <16 x i8> %interleaved.vec, <16 x i8> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
+  %v1 = shufflevector <16 x i8> %interleaved.vec, <16 x i8> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
+  ret <8 x i8> %v1
+}
+
+define <4 x i32> @load_factor3(<12 x i32>* %ptr) {
+; CHECK-LABEL: @load_factor3(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <12 x i32>* [[PTR:%.*]] to i32*
+; CHECK-NEXT:    [[VLSEGN:%.*]] = call { <4 x i32>, <4 x i32>, <4 x i32> } @llvm.riscv.seg3.load.v4i32.p0i32.i64(i32* [[TMP1]], i64 4)
+; CHECK-NEXT:    [[TMP2:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 2
+; CHECK-NEXT:    [[TMP3:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 1
+; CHECK-NEXT:    [[TMP4:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 0
+; CHECK-NEXT:    ret <4 x i32> [[TMP2]]
+;
+  %interleaved.vec = load <12 x i32>, <12 x i32>* %ptr, align 4
+  %v0 = shufflevector <12 x i32> %interleaved.vec, <12 x i32> poison, <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+  %v1 = shufflevector <12 x i32> %interleaved.vec, <12 x i32> poison, <4 x i32> <i32 1, i32 4, i32 7, i32 10>
+  %v2 = shufflevector <12 x i32> %interleaved.vec, <12 x i32> poison, <4 x i32> <i32 2, i32 5, i32 8, i32 11>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_factor4(<16 x i32>* %ptr) {
+; CHECK-LABEL: @load_factor4(
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <16 x i32>* [[PTR:%.*]] to i32*
+; CHECK-NEXT:    [[VLSEGN:%.*]] = call { <4 x i32>, <4 x i32>, <4 x i32>, <4 x i32> } @llvm.riscv.seg4.load.v4i32.p0i32.i64(i32* [[TMP1]], i64 4)
+; CHECK-NEXT:    [[TMP2:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 3
+; CHECK-NEXT:    [[TMP3:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 1
+; CHECK-NEXT:    [[TMP5:%.*]] = extractvalue { <4 x i32>, <4 x i32>, <4 x i32>, <4 x i32> } [[VLSEGN]], 0
+; CHECK-NEXT:    ret <4 x i32> [[TMP2]]
+;
+  %interleaved.vec = load <16 x i32>, <16 x i32>* %ptr, align 4
+  %v0 = shufflevector <16 x i32> %interleaved.vec, <16 x i32> poison, <4 x i32> <i32 0, i32 4, i32 8, i32 12>
+  %v1 = shufflevector <16 x i32> %interleaved.vec, <16 x i32> poison, <4 x i32> <i32 1, i32 5, i32 9, i32 13>
+  %v2 = shufflevector <16 x i32> %interleaved.vec, <16 x i32> poison, <4 x i32> <i32 2, i32 6, i32 10, i32 14>
+  %v3 = shufflevector <16 x i32> %interleaved.vec, <16 x i32> poison, <4 x i32> <i32 3, i32 7, i32 11, i32 15>
+  ret <4 x i32> %v3
+}

llvm/test/Transforms/InterleavedAccess/RISCV/lit.local.cfg

+if not 'RISCV' in config.root.targets:
+  config.unsupported = True