[SelectionDAG] Support integer promotion for VP_LOAD and VP_STORE

[frasercrmck]

Summary

This patch adds support for integer promotion for VP_LOAD and VP_STORE
by legalizing to extending and truncating forms of each, respectively.

[craig.topper]

Tests?

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1972

Just write the word "or" here instead of the ||.

1974

PromoteTargetBoolean would be incorrect for EVL wouldn't it? EVL should always ZERO_EXTEND right?

1992

Same question as above.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

769

Should we choose the extend type from N if it was already an extending VP load? Similar to PromoteIntRes_LOAD.

[hussainjk]

Addressing review comments.

[hussainjk]

In D109377#2987331, @craig.topper wrote:

Tests?

Hm how test? Selection doesn't work at this point.

[craig.topper]

In D109377#2987666, @hussainjk wrote:

In D109377#2987331, @craig.topper wrote:

Tests?

Hm how test? Selection doesn't work at this point.

Didn't D108999 add Selection for RISCV?

[frasercrmck]

In D109377#2987751, @craig.topper wrote:

In D109377#2987666, @hussainjk wrote:

Hm how test? Selection doesn't work at this point.

Didn't D108999 add Selection for RISCV?

Yes you should be able to test this with RISCV. Anything that's not i1/i8/i16/i32 but is less than i64 will need promoting. Also see D108288 for examples.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1973

I don't think we expect EVL to need promoting - do we? It should always be a legal target-specific type, reported by TLI.getVPExplicitVectorLengthTy(). I don't know how well we're asserting on that right now.

[hussainjk]

Fixing some miner logic issues.

[hussainjk]

Adding tests using RISCV lowering

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1976

GetPromotedInteger -> ZExtPromotedInteger. We need to force the upper bits to 0.

1977

5 -> 6 if I've counted the number of operands correctly.

1985

Should we assert that the store isn't indexed since this call would be incorrect if it was?

1996

GetPromotedInteger -> ZExtPromotedInteger.

[hussainjk]

Added more legalization to this patch rather than putting it into another patch. Added another form of getLoadVP that was needed by the legalization. Expanded tests to cover vector splitting.

[hussainjk]

Addressing review comments.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1985

Why does this use isIndexed() but load uses N->getAddressingMode() == ISD::UNINDEXED. Are the interfaces for checking indexed different for loads and stores?

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7700 ↗

(On Diff #371782)

Why should this be different than the assert in getLoad?

[hussainjk]

More assertion fixes.

[frasercrmck]

llvm/test/CodeGen/Generic/fixed-vector-vp-mem-legalization.ll

2 ↗

(On Diff #371791)

We shouldn't put RISC-V tests in Generic unless we know the target is built. So either this needs to use REQUIRES: or we put it in CodeGen/RISCV. I prefer the latter as that's where I've seen all other target-specific legalization tests go.

[hussainjk]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7700 ↗

(On Diff #371782)

These assertions fail on cases that masked loads do not.

[hussainjk]

Addressing review comments.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7700 ↗

(On Diff #371782)

But why? Seems weird to have an extending load that doesn't extend.

[frasercrmck]

@hussainjk are you still working on this or are you taking over, @bmahjour? I might be able to find time if you're unavailable.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1994

Again, I'm not sure whether we ever need to promote the length - it's always a target-specific legal integer type, isn't it?

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2774 ↗

(On Diff #371966)

return Lo would simplify this flow a bit

4344 ↗

(On Diff #371966)

true here is FillWithZeroes - I don't think we need to do this for VP widening, since we're passing the original length along, right?

5223 ↗

(On Diff #371966)

same comment about FillWithZeroes

[bmahjour]

@hussainjk are you still working on this or are you taking over, @bmahjour? I might be able to find time if you're unavailable.

@frasercrmck, Hussain is not working on this any more. Please feel free to commandeer this revision. Thanks for your help!

[frasercrmck]

• rebase

[frasercrmck]

restore full patch

[frasercrmck]

• rebase
• drop widening and splitting (committed separately in D117235)

Note that while the legalized nodes are correct, the generated code we're
testing is *not*; RISCV ignores extending and truncating operations.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
   case ISD::CTLZ:        Res = PromoteIntRes_CTLZ(N); break;
   case ISD::PARITY:
   case ISD::CTPOP:       Res = PromoteIntRes_CTPOP_PARITY(N); break;
   case ISD::CTTZ_ZERO_UNDEF:
   case ISD::CTTZ:        Res = PromoteIntRes_CTTZ(N); break;
   case ISD::EXTRACT_VECTOR_ELT:
                          Res = PromoteIntRes_EXTRACT_VECTOR_ELT(N); break;
   case ISD::LOAD:        Res = PromoteIntRes_LOAD(cast<LoadSDNode>(N)); break;
+  case ISD::VP_LOAD:
+    Res = PromoteIntRes_VP_LOAD(cast<VPLoadSDNode>(N));
+    break;
   case ISD::MLOAD:       Res = PromoteIntRes_MLOAD(cast<MaskedLoadSDNode>(N));
     break;
   case ISD::MGATHER:     Res = PromoteIntRes_MGATHER(cast<MaskedGatherSDNode>(N));
     break;
   case ISD::SELECT:
   case ISD::VSELECT:
   case ISD::VP_SELECT:
     Res = PromoteIntRes_Select(N);
SDValue Res = DAG.getExtLoad(ExtType, dl, NVT, N->getChain(), N->getBasePtr(),
                                N->getMemoryVT(), N->getMemOperand());
 
   // Legalize the chain result - switch anything that used the old chain to
   // use the new one.
   ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
   return Res;
 }
 
+SDValue DAGTypeLegalizer::PromoteIntRes_VP_LOAD(VPLoadSDNode *N) {
+  assert(!N->isIndexed() && "Indexed vp_load during type legalization!");
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  ISD::LoadExtType ExtType = (N->getExtensionType() == ISD::NON_EXTLOAD)
+                                 ? ISD::EXTLOAD
+                                 : N->getExtensionType();
+  SDLoc dl(N);
+  SDValue Res =
+      DAG.getLoadVP(N->getAddressingMode(), ExtType, NVT, dl, N->getChain(),
+                    N->getBasePtr(), N->getOffset(), N->getMask(),
+                    N->getVectorLength(), N->getMemoryVT(), N->getMemOperand());
+  // Legalize the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
+  return Res;
+}
+
 SDValue DAGTypeLegalizer::PromoteIntRes_MLOAD(MaskedLoadSDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   SDValue ExtPassThru = GetPromotedInteger(N->getPassThru());
 
   ISD::LoadExtType ExtType = N->getExtensionType();
   if (ExtType == ISD::NON_EXTLOAD)
     ExtType = ISD::EXTLOAD;
 
bool DAGTypeLegalizer::PromoteIntegerOperand(SDNode *N, unsigned OpNo) {
   case ISD::SELECT:       Res = PromoteIntOp_SELECT(N, OpNo); break;
   case ISD::SELECT_CC:    Res = PromoteIntOp_SELECT_CC(N, OpNo); break;
   case ISD::SETCC:        Res = PromoteIntOp_SETCC(N, OpNo); break;
   case ISD::SIGN_EXTEND:  Res = PromoteIntOp_SIGN_EXTEND(N); break;
   case ISD::SINT_TO_FP:   Res = PromoteIntOp_SINT_TO_FP(N); break;
   case ISD::STRICT_SINT_TO_FP: Res = PromoteIntOp_STRICT_SINT_TO_FP(N); break;
   case ISD::STORE:        Res = PromoteIntOp_STORE(cast<StoreSDNode>(N),
                                                    OpNo); break;
+  case ISD::VP_STORE:
+    Res = PromoteIntOp_VP_STORE(cast<VPStoreSDNode>(N), OpNo);
+    break;
   case ISD::MSTORE:       Res = PromoteIntOp_MSTORE(cast<MaskedStoreSDNode>(N),
                                                     OpNo); break;
+  case ISD::VP_LOAD:
+    Res = PromoteIntOp_VP_LOAD(cast<VPLoadSDNode>(N), OpNo);
+    break;
   case ISD::MLOAD:        Res = PromoteIntOp_MLOAD(cast<MaskedLoadSDNode>(N),
                                                     OpNo); break;
   case ISD::MGATHER:  Res = PromoteIntOp_MGATHER(cast<MaskedGatherSDNode>(N),
                                                  OpNo); break;
   case ISD::MSCATTER: Res = PromoteIntOp_MSCATTER(cast<MaskedScatterSDNode>(N),
                                                   OpNo); break;
   case ISD::TRUNCATE:     Res = PromoteIntOp_TRUNCATE(N); break;
   case ISD::FP16_TO_FP:
SDValue DAGTypeLegalizer::PromoteIntOp_STORE(StoreSDNode *N, unsigned OpNo){
 
   SDValue Val = GetPromotedInteger(N->getValue());  // Get promoted value.
 
   // Truncate the value and store the result.
   return DAG.getTruncStore(Ch, dl, Val, Ptr,
                            N->getMemoryVT(), N->getMemOperand());
 }
 
+SDValue DAGTypeLegalizer::PromoteIntOp_VP_STORE(VPStoreSDNode *N,
+                                                unsigned OpNo) {
+  SDValue DataOp = N->getValue();
+  SDValue Operand = N->getOperand(OpNo);
+
+  if (OpNo >= 4) {
+    // The Mask or EVL. Update in place.
+    EVT DataVT = DataOp.getValueType();
+    SDValue PromotedOperand = OpNo == 4 ? PromoteTargetBoolean(Operand, DataVT)
+                                        : ZExtPromotedInteger(Operand);
+    SmallVector<SDValue, 6> NewOps(N->op_begin(), N->op_end());
+    NewOps[OpNo] = PromotedOperand;
+    return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);
+  }
+
+  assert(OpNo == 1 && "Unexpected operand for promotion");
+  DataOp = GetPromotedInteger(DataOp);
+
+  assert(!N->isIndexed() && "expecting unindexed vp_store!");
+
+  return DAG.getTruncStoreVP(N->getChain(), SDLoc(N), DataOp, N->getBasePtr(),
+                             N->getMask(), N->getVectorLength(),
+                             N->getMemoryVT(), N->getMemOperand(),
+                             N->isCompressingStore());
+}
+
+SDValue DAGTypeLegalizer::PromoteIntOp_VP_LOAD(VPLoadSDNode *N, unsigned OpNo) {
+  assert(OpNo >= 3 && "Only know how to promote the mask or length!");
+  EVT DataVT = N->getValueType(0);
+  SDValue Operand = N->getOperand(OpNo);
+  SDValue PromotedOperand = OpNo == 3 ? PromoteTargetBoolean(Operand, DataVT)
+                                      : ZExtPromotedInteger(Operand);
+  SmallVector<SDValue, 5> NewOps(N->op_begin(), N->op_end());
+  NewOps[OpNo] = PromotedOperand;
+  SDNode *Res = DAG.UpdateNodeOperands(N, NewOps);
+  if (Res == N)
+    return SDValue(Res, 0);
+
+  // Update triggered CSE, do our own replacement since caller can't.
+  ReplaceValueWith(SDValue(N, 0), SDValue(Res, 0));
+  ReplaceValueWith(SDValue(N, 1), SDValue(Res, 1));
+  return SDValue();
+}
+
 SDValue DAGTypeLegalizer::PromoteIntOp_MSTORE(MaskedStoreSDNode *N,
                                               unsigned OpNo) {
   SDValue DataOp = N->getValue();
   SDValue Mask = N->getMask();
 
   if (OpNo == 4) {
     // The Mask. Update in place.
     EVT DataVT = DataOp.getValueType();

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue PromoteIntRes_CTTZ(SDNode *N);
   SDValue PromoteIntRes_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue PromoteIntRes_FP_TO_XINT(SDNode *N);
   SDValue PromoteIntRes_FP_TO_XINT_SAT(SDNode *N);
   SDValue PromoteIntRes_FP_TO_FP16(SDNode *N);
   SDValue PromoteIntRes_FREEZE(SDNode *N);
   SDValue PromoteIntRes_INT_EXTEND(SDNode *N);
   SDValue PromoteIntRes_LOAD(LoadSDNode *N);
+  SDValue PromoteIntRes_VP_LOAD(VPLoadSDNode *N);
   SDValue PromoteIntRes_MLOAD(MaskedLoadSDNode *N);
   SDValue PromoteIntRes_MGATHER(MaskedGatherSDNode *N);
   SDValue PromoteIntRes_Overflow(SDNode *N);
   SDValue PromoteIntRes_SADDSUBO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_Select(SDNode *N);
   SDValue PromoteIntRes_SELECT_CC(SDNode *N);
   SDValue PromoteIntRes_SETCC(SDNode *N);
   SDValue PromoteIntRes_SHL(SDNode *N);
private:
   SDValue PromoteIntOp_SIGN_EXTEND(SDNode *N);
   SDValue PromoteIntOp_SINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_STRICT_SINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_STORE(StoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_TRUNCATE(SDNode *N);
   SDValue PromoteIntOp_UINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_STRICT_UINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_ZERO_EXTEND(SDNode *N);
+  SDValue PromoteIntOp_VP_STORE(VPStoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);
+  SDValue PromoteIntOp_VP_LOAD(VPLoadSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MSCATTER(MaskedScatterSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MGATHER(MaskedGatherSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_ADDSUBCARRY(SDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_FRAMERETURNADDR(SDNode *N);
   SDValue PromoteIntOp_PREFETCH(SDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_FIX(SDNode *N);
   SDValue PromoteIntOp_FPOWI(SDNode *N);

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpload.ll

 ; CHECK-NEXT:    vle8.v v8, (a0)
 ; CHECK-NEXT:    ret
   %a = insertelement <4 x i1> poison, i1 true, i32 0
   %b = shufflevector <4 x i1> %a, <4 x i1> poison, <4 x i32> zeroinitializer
   %load = call <4 x i8> @llvm.vp.load.v4i8.p0v4i8(<4 x i8>* %ptr, <4 x i1> %b, i32 %evl)
   ret <4 x i8> %load
 }
 
+declare <8 x i7> @llvm.vp.load.v8i7.p0v8i7(<8 x i7>*, <8 x i1>, i32)
+
+define <8 x i7> @vpload_v8i7(<8 x i7>* %ptr, <8 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_v8i7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
+; CHECK-NEXT:    vle8.v v8, (a0), v0.t
+; CHECK-NEXT:    ret
+  %load = call <8 x i7> @llvm.vp.load.v8i7.p0v8i7(<8 x i7>* %ptr, <8 x i1> %m, i32 %evl)
+  ret <8 x i7> %load
+}
+
 declare <8 x i8> @llvm.vp.load.v8i8.p0v8i8(<8 x i8>*, <8 x i1>, i32)
 
 define <8 x i8> @vpload_v8i8(<8 x i8>* %ptr, <8 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpload_v8i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vle8.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret

 declare <32 x double> @llvm.vp.load.v32f64.p0v32f64(<32 x double>*, <32 x i1>, i32)
 
 define <32 x double> @vpload_v32f64(<32 x double>* %ptr, <32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpload_v32f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addi a3, a1, -16
 ; CHECK-NEXT:    vmv1r.v v8, v0
 ; CHECK-NEXT:    li a2, 0
-; CHECK-NEXT:    bltu a1, a3, .LBB31_2
+; CHECK-NEXT:    bltu a1, a3, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a2, a3
-; CHECK-NEXT:  .LBB31_2:
+; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, mu
 ; CHECK-NEXT:    vslidedown.vi v0, v8, 2
 ; CHECK-NEXT:    addi a3, a0, 128
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v16, (a3), v0.t
 ; CHECK-NEXT:    li a2, 16
-; CHECK-NEXT:    bltu a1, a2, .LBB31_4
+; CHECK-NEXT:    bltu a1, a2, .LBB32_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    li a1, 16
-; CHECK-NEXT:  .LBB31_4:
+; CHECK-NEXT:  .LBB32_4:
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v8
 ; CHECK-NEXT:    vle64.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret
   %load = call <32 x double> @llvm.vp.load.v32f64.p0v32f64(<32 x double>* %ptr, <32 x i1> %m, i32 %evl)
   ret <32 x double> %load
 }
 
 declare <33 x double> @llvm.vp.load.v33f64.p0v33f64(<33 x double>*, <33 x i1>, i32)
 
 ; Widen to v64f64 then split into 4 x v16f64, of which 1 is empty.
 
 define <33 x double> @vpload_v33f64(<33 x double>* %ptr, <33 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpload_v33f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addi a4, a2, -32
 ; CHECK-NEXT:    vmv1r.v v8, v0
 ; CHECK-NEXT:    li a3, 0
 ; CHECK-NEXT:    li a5, 0
-; CHECK-NEXT:    bltu a2, a4, .LBB32_2
+; CHECK-NEXT:    bltu a2, a4, .LBB33_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a5, a4
-; CHECK-NEXT:  .LBB32_2:
+; CHECK-NEXT:  .LBB33_2:
 ; CHECK-NEXT:    li a4, 16
-; CHECK-NEXT:    bltu a5, a4, .LBB32_4
+; CHECK-NEXT:    bltu a5, a4, .LBB33_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    li a5, 16
-; CHECK-NEXT:  .LBB32_4:
+; CHECK-NEXT:  .LBB33_4:
 ; CHECK-NEXT:    vsetivli zero, 4, e8, mf2, ta, mu
 ; CHECK-NEXT:    vslidedown.vi v0, v8, 4
 ; CHECK-NEXT:    addi a6, a1, 256
 ; CHECK-NEXT:    vsetvli zero, a5, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v16, (a6), v0.t
 ; CHECK-NEXT:    li a5, 32
-; CHECK-NEXT:    bltu a2, a5, .LBB32_6
+; CHECK-NEXT:    bltu a2, a5, .LBB33_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    li a2, 32
-; CHECK-NEXT:  .LBB32_6:
+; CHECK-NEXT:  .LBB33_6:
 ; CHECK-NEXT:    addi a5, a2, -16
-; CHECK-NEXT:    bltu a2, a5, .LBB32_8
+; CHECK-NEXT:    bltu a2, a5, .LBB33_8
 ; CHECK-NEXT:  # %bb.7:
 ; CHECK-NEXT:    mv a3, a5
-; CHECK-NEXT:  .LBB32_8:
+; CHECK-NEXT:  .LBB33_8:
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, mu
 ; CHECK-NEXT:    vslidedown.vi v0, v8, 2
 ; CHECK-NEXT:    addi a5, a1, 128
 ; CHECK-NEXT:    vsetvli zero, a3, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v24, (a5), v0.t
-; CHECK-NEXT:    bltu a2, a4, .LBB32_10
+; CHECK-NEXT:    bltu a2, a4, .LBB33_10
 ; CHECK-NEXT:  # %bb.9:
 ; CHECK-NEXT:    li a2, 16
-; CHECK-NEXT:  .LBB32_10:
+; CHECK-NEXT:  .LBB33_10:
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v8
 ; CHECK-NEXT:    vle64.v v8, (a1), v0.t
 ; CHECK-NEXT:    vsetivli zero, 16, e64, m8, ta, mu
 ; CHECK-NEXT:    vse64.v v8, (a0)
 ; CHECK-NEXT:    addi a1, a0, 128
 ; CHECK-NEXT:    vse64.v v24, (a1)
 ; CHECK-NEXT:    vfmv.f.s ft0, v16
 ; CHECK-NEXT:    fsd ft0, 256(a0)
 ; CHECK-NEXT:    ret
   %load = call <33 x double> @llvm.vp.load.v33f64.p0v33f64(<33 x double>* %ptr, <33 x i1> %m, i32 %evl)
   ret <33 x double> %load
 }

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpstore.ll

 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, mu
 ; CHECK-NEXT:    vse8.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret
   call void @llvm.vp.store.v4i8.p0v4i8(<4 x i8> %val, <4 x i8>* %ptr, <4 x i1> %m, i32 %evl)
   ret void
 }
 
+declare void @llvm.vp.store.v8i7.p0v8i7(<8 x i7>, <8 x i7>*, <8 x i1>, i32)
+
+define void @vpstore_v8i7(<8 x i7> %val, <8 x i7>* %ptr, <8 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_v8i7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
+; CHECK-NEXT:    vse8.v v8, (a0), v0.t
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.v8i7.p0v8i7(<8 x i7> %val, <8 x i7>* %ptr, <8 x i1> %m, i32 %evl)
+  ret void
+}
+
 declare void @llvm.vp.store.v8i8.p0v8i8(<8 x i8>, <8 x i8>*, <8 x i1>, i32)
 
 define void @vpstore_v8i8(<8 x i8> %val, <8 x i8>* %ptr, <8 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpstore_v8i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vse8.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret

 
 declare void @llvm.vp.store.v32f64.p0v32f64(<32 x double>, <32 x double>*, <32 x i1>, i32)
 
 define void @vpstore_v32f64(<32 x double> %val, <32 x double>* %ptr, <32 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpstore_v32f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    li a2, 16
 ; CHECK-NEXT:    mv a3, a1
-; CHECK-NEXT:    bltu a1, a2, .LBB23_2
+; CHECK-NEXT:    bltu a1, a2, .LBB24_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    li a3, 16
-; CHECK-NEXT:  .LBB23_2:
+; CHECK-NEXT:  .LBB24_2:
 ; CHECK-NEXT:    li a2, 0
 ; CHECK-NEXT:    vsetvli zero, a3, e64, m8, ta, mu
 ; CHECK-NEXT:    addi a3, a1, -16
 ; CHECK-NEXT:    vse64.v v8, (a0), v0.t
-; CHECK-NEXT:    bltu a1, a3, .LBB23_4
+; CHECK-NEXT:    bltu a1, a3, .LBB24_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    mv a2, a3
-; CHECK-NEXT:  .LBB23_4:
+; CHECK-NEXT:  .LBB24_4:
 ; CHECK-NEXT:    vsetivli zero, 2, e8, mf4, ta, mu
 ; CHECK-NEXT:    vslidedown.vi v0, v0, 2
 ; CHECK-NEXT:    addi a0, a0, 128
 ; CHECK-NEXT:    vsetvli zero, a2, e64, m8, ta, mu
 ; CHECK-NEXT:    vse64.v v16, (a0), v0.t
 ; CHECK-NEXT:    ret
   call void @llvm.vp.store.v32f64.p0v32f64(<32 x double> %val, <32 x double>* %ptr, <32 x i1> %m, i32 %evl)
   ret void
 }

llvm/test/CodeGen/RISCV/rvv/vpload.ll

 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vle8.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret
   %load = call <vscale x 3 x i8> @llvm.vp.load.nxv3i8.p0nxv3i8(<vscale x 3 x i8>* %ptr, <vscale x 3 x i1> %m, i32 %evl)
   ret <vscale x 3 x i8> %load
 }
 
+declare <vscale x 4 x i6> @llvm.vp.load.nxv4i6.p0nxv4i6(<vscale x 4 x i6>*, <vscale x 4 x i1>, i32)
+
+define <vscale x 4 x i6> @vpload_nxv4i6(<vscale x 4 x i6>* %ptr, <vscale x 4 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv4i6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
+; CHECK-NEXT:    vle8.v v8, (a0), v0.t
+; CHECK-NEXT:    ret
+  %load = call <vscale x 4 x i6> @llvm.vp.load.nxv4i6.p0nxv4i6(<vscale x 4 x i6>* %ptr, <vscale x 4 x i1> %m, i32 %evl)
+  ret <vscale x 4 x i6> %load
+}
+
 declare <vscale x 4 x i8> @llvm.vp.load.nxv4i8.p0nxv4i8(<vscale x 4 x i8>*, <vscale x 4 x i1>, i32)
 
 define <vscale x 4 x i8> @vpload_nxv4i8(<vscale x 4 x i8>* %ptr, <vscale x 4 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpload_nxv4i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, mu
 ; CHECK-NEXT:    vle8.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret

 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmv1r.v v8, v0
 ; CHECK-NEXT:    li a3, 0
 ; CHECK-NEXT:    csrr a2, vlenb
 ; CHECK-NEXT:    srli a5, a2, 3
 ; CHECK-NEXT:    vsetvli a4, zero, e8, mf4, ta, mu
 ; CHECK-NEXT:    sub a4, a1, a2
 ; CHECK-NEXT:    vslidedown.vx v0, v0, a5
-; CHECK-NEXT:    bltu a1, a4, .LBB37_2
+; CHECK-NEXT:    bltu a1, a4, .LBB38_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a3, a4
-; CHECK-NEXT:  .LBB37_2:
+; CHECK-NEXT:  .LBB38_2:
 ; CHECK-NEXT:    slli a4, a2, 3
 ; CHECK-NEXT:    add a4, a0, a4
 ; CHECK-NEXT:    vsetvli zero, a3, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v16, (a4), v0.t
-; CHECK-NEXT:    bltu a1, a2, .LBB37_4
+; CHECK-NEXT:    bltu a1, a2, .LBB38_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    mv a1, a2
-; CHECK-NEXT:  .LBB37_4:
+; CHECK-NEXT:  .LBB38_4:
 ; CHECK-NEXT:    vsetvli zero, a1, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v8
 ; CHECK-NEXT:    vle64.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret
   %load = call <vscale x 16 x double> @llvm.vp.load.nxv16f64.p0nxv16f64(<vscale x 16 x double>* %ptr, <vscale x 16 x i1> %m, i32 %evl)
   ret <vscale x 16 x double> %load
 }
 

 
 define <vscale x 16 x double> @vpload_nxv17f64(<vscale x 17 x double>* %ptr, <vscale x 1 x double>* %out, <vscale x 17 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpload_nxv17f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    csrr a3, vlenb
 ; CHECK-NEXT:    slli a5, a3, 1
 ; CHECK-NEXT:    vmv1r.v v8, v0
 ; CHECK-NEXT:    mv a4, a2
-; CHECK-NEXT:    bltu a2, a5, .LBB38_2
+; CHECK-NEXT:    bltu a2, a5, .LBB39_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a4, a5
-; CHECK-NEXT:  .LBB38_2:
+; CHECK-NEXT:  .LBB39_2:
 ; CHECK-NEXT:    sub a7, a4, a3
 ; CHECK-NEXT:    li a6, 0
-; CHECK-NEXT:    bltu a4, a7, .LBB38_4
+; CHECK-NEXT:    bltu a4, a7, .LBB39_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    mv a6, a7
-; CHECK-NEXT:  .LBB38_4:
+; CHECK-NEXT:  .LBB39_4:
 ; CHECK-NEXT:    li a7, 0
 ; CHECK-NEXT:    srli t0, a3, 3
 ; CHECK-NEXT:    vsetvli t1, zero, e8, mf4, ta, mu
 ; CHECK-NEXT:    vslidedown.vx v0, v8, t0
 ; CHECK-NEXT:    slli t0, a3, 3
 ; CHECK-NEXT:    add t0, a0, t0
 ; CHECK-NEXT:    vsetvli zero, a6, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v16, (t0), v0.t
 ; CHECK-NEXT:    srli a6, a3, 2
 ; CHECK-NEXT:    sub t0, a2, a5
 ; CHECK-NEXT:    slli a5, a3, 4
-; CHECK-NEXT:    bltu a2, t0, .LBB38_6
+; CHECK-NEXT:    bltu a2, t0, .LBB39_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    mv a7, t0
-; CHECK-NEXT:  .LBB38_6:
+; CHECK-NEXT:  .LBB39_6:
 ; CHECK-NEXT:    vsetvli a2, zero, e8, mf2, ta, mu
 ; CHECK-NEXT:    vslidedown.vx v0, v8, a6
 ; CHECK-NEXT:    add a2, a0, a5
-; CHECK-NEXT:    bltu a7, a3, .LBB38_8
+; CHECK-NEXT:    bltu a7, a3, .LBB39_8
 ; CHECK-NEXT:  # %bb.7:
 ; CHECK-NEXT:    mv a7, a3
-; CHECK-NEXT:  .LBB38_8:
+; CHECK-NEXT:  .LBB39_8:
 ; CHECK-NEXT:    vsetvli zero, a7, e64, m8, ta, mu
 ; CHECK-NEXT:    vle64.v v24, (a2), v0.t
-; CHECK-NEXT:    bltu a4, a3, .LBB38_10
+; CHECK-NEXT:    bltu a4, a3, .LBB39_10
 ; CHECK-NEXT:  # %bb.9:
 ; CHECK-NEXT:    mv a4, a3
-; CHECK-NEXT:  .LBB38_10:
+; CHECK-NEXT:  .LBB39_10:
 ; CHECK-NEXT:    vsetvli zero, a4, e64, m8, ta, mu
 ; CHECK-NEXT:    vmv1r.v v0, v8
 ; CHECK-NEXT:    vle64.v v8, (a0), v0.t
 ; CHECK-NEXT:    vs1r.v v24, (a1)
 ; CHECK-NEXT:    ret
   %load = call <vscale x 17 x double> @llvm.vp.load.nxv17f64.p0nxv17f64(<vscale x 17 x double>* %ptr, <vscale x 17 x i1> %m, i32 %evl)
   %lo = call <vscale x 16 x double> @llvm.experimental.vector.extract.nxv16f64(<vscale x 17 x double> %load, i64 0)
   %hi = call <vscale x 1 x double> @llvm.experimental.vector.extract.nxv1f64(<vscale x 17 x double> %load, i64 16)
   store <vscale x 1 x double> %hi, <vscale x 1 x double>* %out
   ret <vscale x 16 x double> %lo
 }

llvm/test/CodeGen/RISCV/rvv/vpstore.ll

 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m1, ta, mu
 ; CHECK-NEXT:    vse16.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret
   call void @llvm.vp.store.nxv4i16.p0nxv4i16(<vscale x 4 x i16> %val, <vscale x 4 x i16>* %ptr, <vscale x 4 x i1> %m, i32 %evl)
   ret void
 }
 
+declare void @llvm.vp.store.nxv8i12.p0nxv8i12(<vscale x 8 x i12>, <vscale x 8 x i12>*, <vscale x 8 x i1>, i32)
+
+define void @vpstore_nxv8i12(<vscale x 8 x i12> %val, <vscale x 8 x i12>* %ptr, <vscale x 8 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv8i12:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
+; CHECK-NEXT:    vse16.v v8, (a0), v0.t
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv8i12.p0nxv8i12(<vscale x 8 x i12> %val, <vscale x 8 x i12>* %ptr, <vscale x 8 x i1> %m, i32 %evl)
+  ret void
+}
+
 declare void @llvm.vp.store.nxv8i16.p0nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>*, <vscale x 8 x i1>, i32)
 
 define void @vpstore_nxv8i16(<vscale x 8 x i16> %val, <vscale x 8 x i16>* %ptr, <vscale x 8 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpstore_nxv8i16:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli zero, a1, e16, m2, ta, mu
 ; CHECK-NEXT:    vse16.v v8, (a0), v0.t
 ; CHECK-NEXT:    ret

 
 declare void @llvm.vp.store.nxv16f64.p0nxv16f64(<vscale x 16 x double>, <vscale x 16 x double>*, <vscale x 16 x i1>, i32)
 
 define void @vpstore_nxv16f64(<vscale x 16 x double> %val, <vscale x 16 x double>* %ptr, <vscale x 16 x i1> %m, i32 zeroext %evl) {
 ; CHECK-LABEL: vpstore_nxv16f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    csrr a2, vlenb
 ; CHECK-NEXT:    mv a3, a1
-; CHECK-NEXT:    bltu a1, a2, .LBB30_2
+; CHECK-NEXT:    bltu a1, a2, .LBB31_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a3, a2
-; CHECK-NEXT:  .LBB30_2:
+; CHECK-NEXT:  .LBB31_2:
 ; CHECK-NEXT:    li a4, 0
 ; CHECK-NEXT:    vsetvli zero, a3, e64, m8, ta, mu
 ; CHECK-NEXT:    vse64.v v8, (a0), v0.t
 ; CHECK-NEXT:    srli a5, a2, 3
 ; CHECK-NEXT:    vsetvli a3, zero, e8, mf4, ta, mu
 ; CHECK-NEXT:    sub a3, a1, a2
 ; CHECK-NEXT:    vslidedown.vx v0, v0, a5
-; CHECK-NEXT:    bltu a1, a3, .LBB30_4
+; CHECK-NEXT:    bltu a1, a3, .LBB31_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    mv a4, a3
-; CHECK-NEXT:  .LBB30_4:
+; CHECK-NEXT:  .LBB31_4:
 ; CHECK-NEXT:    slli a1, a2, 3
 ; CHECK-NEXT:    add a0, a0, a1
 ; CHECK-NEXT:    vsetvli zero, a4, e64, m8, ta, mu
 ; CHECK-NEXT:    vse64.v v16, (a0), v0.t
 ; CHECK-NEXT:    ret
   call void @llvm.vp.store.nxv16f64.p0nxv16f64(<vscale x 16 x double> %val, <vscale x 16 x double>* %ptr, <vscale x 16 x i1> %m, i32 %evl)
   ret void
 }

 ; CHECK-NEXT:    slli a3, a3, 3
 ; CHECK-NEXT:    sub sp, sp, a3
 ; CHECK-NEXT:    csrr a3, vlenb
 ; CHECK-NEXT:    slli a4, a3, 1
 ; CHECK-NEXT:    vmv1r.v v24, v0
 ; CHECK-NEXT:    addi a5, sp, 16
 ; CHECK-NEXT:    vs8r.v v16, (a5) # Unknown-size Folded Spill
 ; CHECK-NEXT:    mv a5, a2
-; CHECK-NEXT:    bltu a2, a4, .LBB31_2
+; CHECK-NEXT:    bltu a2, a4, .LBB32_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    mv a5, a4
-; CHECK-NEXT:  .LBB31_2:
+; CHECK-NEXT:  .LBB32_2:
 ; CHECK-NEXT:    mv a7, a5
-; CHECK-NEXT:    bltu a5, a3, .LBB31_4
+; CHECK-NEXT:    bltu a5, a3, .LBB32_4
 ; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    mv a7, a3
-; CHECK-NEXT:  .LBB31_4:
+; CHECK-NEXT:  .LBB32_4:
 ; CHECK-NEXT:    li a6, 0
 ; CHECK-NEXT:    vl8re64.v v16, (a0)
 ; CHECK-NEXT:    vsetvli zero, a7, e64, m8, ta, mu
 ; CHECK-NEXT:    sub a0, a5, a3
 ; CHECK-NEXT:    vmv1r.v v0, v24
 ; CHECK-NEXT:    vse64.v v8, (a1), v0.t
-; CHECK-NEXT:    bltu a5, a0, .LBB31_6
+; CHECK-NEXT:    bltu a5, a0, .LBB32_6
 ; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    mv a6, a0
-; CHECK-NEXT:  .LBB31_6:
+; CHECK-NEXT:  .LBB32_6:
 ; CHECK-NEXT:    li a0, 0
 ; CHECK-NEXT:    srli a5, a3, 3
 ; CHECK-NEXT:    vsetvli a7, zero, e8, mf4, ta, mu
 ; CHECK-NEXT:    vslidedown.vx v0, v24, a5
 ; CHECK-NEXT:    slli a5, a3, 3
 ; CHECK-NEXT:    add a5, a1, a5
 ; CHECK-NEXT:    vsetvli zero, a6, e64, m8, ta, mu
 ; CHECK-NEXT:    addi a6, sp, 16
 ; CHECK-NEXT:    vl8re8.v v8, (a6) # Unknown-size Folded Reload
 ; CHECK-NEXT:    vse64.v v8, (a5), v0.t
 ; CHECK-NEXT:    srli a5, a3, 2
 ; CHECK-NEXT:    sub a6, a2, a4
 ; CHECK-NEXT:    slli a4, a3, 4
-; CHECK-NEXT:    bltu a2, a6, .LBB31_8
+; CHECK-NEXT:    bltu a2, a6, .LBB32_8
 ; CHECK-NEXT:  # %bb.7:
 ; CHECK-NEXT:    mv a0, a6
-; CHECK-NEXT:  .LBB31_8:
+; CHECK-NEXT:  .LBB32_8:
 ; CHECK-NEXT:    vsetvli a2, zero, e8, mf2, ta, mu
 ; CHECK-NEXT:    vslidedown.vx v0, v24, a5
 ; CHECK-NEXT:    add a1, a1, a4
-; CHECK-NEXT:    bltu a0, a3, .LBB31_10
+; CHECK-NEXT:    bltu a0, a3, .LBB32_10
 ; CHECK-NEXT:  # %bb.9:
 ; CHECK-NEXT:    mv a0, a3
-; CHECK-NEXT:  .LBB31_10:
+; CHECK-NEXT:  .LBB32_10:
 ; CHECK-NEXT:    vsetvli zero, a0, e64, m8, ta, mu
 ; CHECK-NEXT:    vse64.v v16, (a1), v0.t
 ; CHECK-NEXT:    csrr a0, vlenb
 ; CHECK-NEXT:    slli a0, a0, 3
 ; CHECK-NEXT:    add sp, sp, a0
 ; CHECK-NEXT:    addi sp, sp, 16
 ; CHECK-NEXT:    ret
   call void @llvm.vp.store.nxv17f64.p0nxv17f64(<vscale x 17 x double> %val, <vscale x 17 x double>* %ptr, <vscale x 17 x i1> %m, i32 %evl)
   ret void
 }