[VP] Vector predicated vector splice intrinsic

[vkmr]

Summary

This patch introduces the vector-predicated version of the experimental_vector_splice intrinsic [1] at the IR level. It considers the active vector length for both vectors and and uses a vector mask to disable certain lanes in the result.

[1] https://reviews.llvm.org/D94708

[simoll]

As discussed on the call, maybe rename llvm.experimental.vp.vector.splice to llvm.experimental.vp.splice.. the "vector" is implied

[craig.topper]

Are evl1 and evl2 likely to be the same value? Do you have an example when they would be different?

[vkmr]

In D103898#2809074, @craig.topper wrote:

Are evl1 and evl2 likely to be the same value? Do you have an example when they would be different?

For a first order reduction, using a tail-folded loop with VP, if the EVL is computed using target-specific instructions, there is no guarantee that the EVL for the previous iteration would be the same as the EVL for current iteration or even that it would be equal to the runtime VF. For a more specific example, for RISC-V, the constraints on the vset{i}vl{i} instruction allow the EVL for the last non-tail iteration to be less than the runtime VF and unequal to the EVL for the tail iteration.

[simoll]

This patch needs to implement VPIntrinsic::getDeclarationForParams.

[vkmr]

Addressed @simoll's comment. Took the liberty to clang-format part of the test for memory ops.

[vkmr]

In D103898#2812522, @simoll wrote:

This patch needs to implement VPIntrinsic::getDeclarationForParams.

for vp_splice, the implementation will be the same as the default case so I did not implement it explicitly. Did I miss anything?

[vkmr]

In D103898#2807243, @simoll wrote:

As discussed on the call, maybe rename llvm.experimental.vp.vector.splice to llvm.experimental.vp.splice.. the "vector" is implied

Done!

[simoll]

In D103898#2854635, @vkmr wrote:

In D103898#2812522, @simoll wrote:

This patch needs to implement VPIntrinsic::getDeclarationForParams.

for vp_splice, the implementation will be the same as the default case so I did not implement it explicitly. Did I miss anything?

You didn't. LGTM with nit.

llvm/docs/LangRef.rst

19568

I think it's not totally obvious that %imm has to be an immediate. Eg, you could add immarg in the intrinsic declarations (Syntax section) and mention it again in the Arguments section.

[simoll]

llvm/include/llvm/IR/VPIntrinsics.def

262

+1 what lint said

[vkmr]

Addressed comments.

[simoll]

LGTM

[frasercrmck]

llvm/docs/LangRef.rst

19581

typo: selectng

19584

I think undef is generally back-ticked in this document

[vkmr]

addressed typos found by @frasercrmck.

[vkmr]

Can someone please commit it on my behalf? I don't have commit access.

[vkmr]

@simoll I believe there is a way to commit without changing the author, such that both phabricator and github show something like "authored by <author> and committed by <committer>".

[simoll]

In D103898#3058405, @vkmr wrote:

@simoll I believe there is a way to commit without changing the author, such that both phabricator and github show something like "authored by <author> and committed by <committer>".

That's what i normally do. There were some issues with arcanist and i had to commit manually.

llvm/docs/LangRef.rst

 """""""""
 
 .. code-block:: llvm
 
       %active.lane.mask = call <4 x i1> @llvm.get.active.lane.mask.v4i1.i64(i64 %elem0, i64 429)
       %wide.masked.load = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %3, i32 4, <4 x i1> %active.lane.mask, <4 x i32> undef)
 
 
+.. _int_experimental_vp_splice:
+
+'``llvm.experimental.vp.splice``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <2 x double> @llvm.experimental.vp.splice.v2f64(<2 x double> %vec1, <2 x double> %vec2, i32 %imm, <2 x i1> %mask, i32 %evl1, i32 %evl2)
+      declare <vscale x 4 x i32> @llvm.experimental.vp.splice.nxv4i32(<vscale x 4 x i32> %vec1, <vscale x 4 x i32> %vec2, i32 %imm, <2 x i1> %mask i32 %evl1, i32 %evl2)
+
+Overview:
+"""""""""
+
+The '``llvm.experimental.vp.splice.*``' intrinsic is the vector length
+predicated version of the '``llvm.experimental.vector.splice.*``' intrinsic.
+
+Arguments:
+""""""""""
+
+The result and the first two arguments ``vec1`` and ``vec2`` are vectors with
+the same type.  The third argument ``imm`` is an immediate signed integer that
+indicates the offset index.  The fourth argument ``mask`` is a vector mask and
+has the same number of elements as the result.  The last two arguments ``evl1``
+and ``evl2`` are unsigned integers indicating the explicit vector lengths of
+``vec1`` and ``vec2`` respectively.  ``imm``, ``evl1`` and ``evl2`` should
+respect the following constraints: ``-evl1 <= imm < evl1``, ``0 <= evl1 <= VL``
+and ``0 <= evl2 <= VL``, where ``VL`` is the runtime vector factor. If these
+constraints are not satisfied the intrinsic has undefined behaviour.
+
+Semantics:
+""""""""""
+
+Effectively, this intrinsic concatenates ``vec1[0..evl1-1]`` and
+``vec2[0..evl2-1]`` and creates the result vector by selecting the elements in a
+window of size ``evl2``, starting at index ``imm`` (for a positive immediate) of
+the concatenated vector. Elements in the result vector beyond ``evl2`` are
+``undef``.  If ``imm`` is negative the starting index is ``evl1 + imm``.  The result
+vector of active vector length ``evl2`` contains ``evl1 - imm`` (``-imm`` for
+negative ``imm``) elements from indices ``[imm..evl1 - 1]``
+(``[evl1 + imm..evl1 -1]`` for negative ``imm``) of ``vec1`` followed by the
+first ``evl2 - (evl1 - imm)`` (``evl2 + imm`` for negative ``imm``) elements of
+``vec2``. If ``evl1 - imm`` (``-imm``) >= ``evl2``, only the first ``evl2``
+elements are considered and the remaining are ``undef``.  The lanes in the result
+vector disabled by ``mask`` are ``undef``.
+
+Examples:
+"""""""""
+
+.. code-block:: text
+
+ llvm.experimental.vp.splice(<A,B,C,D>, <E,F,G,H>, 1, 2, 3)  ==> <B, E, F, undef> ; index
+ llvm.experimental.vp.splice(<A,B,C,D>, <E,F,G,H>, -2, 3, 2) ==> <B, C, undef, undef> ; trailing elements
+
+
 .. _int_mload_mstore:
 
 Masked Vector Load and Store Intrinsics
 ---------------------------------------
 
 LLVM provides intrinsics for predicated vector load and store operations. The predicate is specified by a mask operand, which holds one bit per vector element, switching the associated vector lane on or off. The memory addresses corresponding to the "off" lanes are not accessed. When all bits of the mask are on, the intrinsic is identical to a regular vector load or store. When all bits are off, no memory is accessed.
 
 .. _int_mload:

llvm/include/llvm/IR/Intrinsics.td

def int_vp_reduce_fmin : DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
                                      llvm_i32_ty]>;
 }
 
 def int_get_active_lane_mask:
   DefaultAttrsIntrinsic<[llvm_anyvector_ty],
             [llvm_anyint_ty, LLVMMatchType<1>],
             [IntrNoMem, IntrNoSync, IntrWillReturn]>;
 
+def int_experimental_vp_splice:
+  DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+            [LLVMMatchType<0>,
+             LLVMMatchType<0>,
+             llvm_i32_ty,
+             LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+             llvm_i32_ty, llvm_i32_ty],
+            [IntrNoMem, ImmArg<ArgIndex<2>>]>;
+
 //===-------------------------- Masked Intrinsics -------------------------===//
 //
 def int_masked_load:
   DefaultAttrsIntrinsic<[llvm_anyvector_ty],
             [LLVMAnyPointerType<LLVMMatchType<0>>, llvm_i32_ty,
              LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>, LLVMMatchType<0>],
             [IntrReadMem, IntrArgMemOnly, IntrWillReturn, ImmArg<ArgIndex<1>>]>;
 

llvm/include/llvm/IR/VPIntrinsics.def

 HANDLE_VP_IS_MEMOP(vp_gather, 0, None)
 END_REGISTER_VP(vp_gather, VP_GATHER)
 
 ///// } Memory Operations
 
 ///// Reductions {
 
 // Specialized helper macro for VP reductions (%start, %x, %mask, %evl).
 #ifdef HELPER_REGISTER_REDUCTION_VP
 #error "The internal helper macro HELPER_REGISTER_REDUCTION_VP is already defined!"
 #endif
 #define HELPER_REGISTER_REDUCTION_VP(VPINTRIN, SDOPC, INTRIN) \
 BEGIN_REGISTER_VP(VPINTRIN, 2, 3, SDOPC, -1) \
 HANDLE_VP_TO_INTRIN(INTRIN) \
 HANDLE_VP_REDUCTION(VPINTRIN, 0, 1) \
 PROPERTY_VP_REDUCTION_SDNODE(SDOPC) \
 END_REGISTER_VP(VPINTRIN, SDOPC)

 ///// Shuffles {
 
 // llvm.vp.select(mask,on_true,on_false,vlen)
 BEGIN_REGISTER_VP_INTRINSIC(vp_select, 0, 3)
 // BEGIN_REGISTER_VP_SDNODE(VP_SELECT, -1, vp_select, 0, 4)
 // END_REGISTER_CASES(vp_select, VP_SELECT)
 END_REGISTER_VP_INTRINSIC(vp_select)
 
+BEGIN_REGISTER_VP(experimental_vp_splice, 3, 5,
+                  EXPERIMENTAL_VP_SPLICE, -1)
+END_REGISTER_VP(experimental_vp_splice, EXPERIMENTAL_VP_SPLICE)
+
 ///// } Shuffles
 
 #undef BEGIN_REGISTER_VP
 #undef BEGIN_REGISTER_VP_INTRINSIC
 #undef BEGIN_REGISTER_VP_SDNODE
 #undef END_REGISTER_VP
 #undef END_REGISTER_VP_INTRINSIC
 #undef END_REGISTER_VP_SDNODE
 #undef HANDLE_VP_TO_OPC
 #undef HANDLE_VP_TO_CONSTRAINEDFP
 #undef HANDLE_VP_TO_INTRIN
 #undef HANDLE_VP_IS_MEMOP
 #undef HANDLE_VP_REDUCTION
 #undef PROPERTY_VP_BINARYOP_SDNODE
 #undef PROPERTY_VP_REDUCTION_SDNODE

llvm/test/Verifier/vp-intrinsics.ll

define void @test_vp_reduction(i32 %x, <8 x i32> %vi, <8 x float> %vf, float %f, <8 x i1> %m, i32 %n) {
   %r8 = call i32 @llvm.vp.reduce.umin.v8i32(i32 %x, <8 x i32> %vi, <8 x i1> %m, i32 %n)
   %r9 = call float @llvm.vp.reduce.fmin.v8f32(float %f, <8 x float> %vf, <8 x i1> %m, i32 %n)
   %rA = call float @llvm.vp.reduce.fmax.v8f32(float %f, <8 x float> %vf, <8 x i1> %m, i32 %n)
   %rB = call float @llvm.vp.reduce.fadd.v8f32(float %f, <8 x float> %vf, <8 x i1> %m, i32 %n)
   %rC = call float @llvm.vp.reduce.fmul.v8f32(float %f, <8 x float> %vf, <8 x i1> %m, i32 %n)
   ret void
 }
 
+define void @test_vp_splice0(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %l0, i32 %l1) {
+  %r0 = call <8 x i32> @llvm.experimental.vp.splice.v8i32(<8 x i32> %i0, <8 x i32> %i1, i32 2, <8 x i1> %m, i32 %l0, i32 %l1)
+  ret void
+}
+
+define void @test_vp_splice1(<vscale x 8 x i32> %i0, <vscale x 8 x i32> %i1, <vscale x 8 x i1> %m, i32 %l0, i32 %l1) {
+  %r0 = call <vscale x 8 x i32> @llvm.experimental.vp.splice.nxv8i32(<vscale x 8 x i32> %i0, <vscale x 8 x i32> %i1, i32 -1, <vscale x 8 x i1> %m, i32 %l0, i32 %l1)
+  ret void
+}
+
 ; integer arith
 declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)

 declare i32 @llvm.vp.reduce.smax.v8i32(i32, <8 x i32>, <8 x i1>, i32)
 declare i32 @llvm.vp.reduce.smin.v8i32(i32, <8 x i32>, <8 x i1>, i32)
 declare i32 @llvm.vp.reduce.umax.v8i32(i32, <8 x i32>, <8 x i1>, i32)
 declare i32 @llvm.vp.reduce.umin.v8i32(i32, <8 x i32>, <8 x i1>, i32)
 declare float @llvm.vp.reduce.fmin.v8f32(float, <8 x float>, <8 x i1>, i32)
 declare float @llvm.vp.reduce.fmax.v8f32(float, <8 x float>, <8 x i1>, i32)
 declare float @llvm.vp.reduce.fadd.v8f32(float, <8 x float>, <8 x i1>, i32)
 declare float @llvm.vp.reduce.fmul.v8f32(float, <8 x float>, <8 x i1>, i32)
+; shuffles
+declare <8 x i32> @llvm.experimental.vp.splice.v8i32(<8 x i32>, <8 x i32>, i32, <8 x i1>, i32, i32)
+declare <vscale x 8 x i32> @llvm.experimental.vp.splice.nxv8i32(<vscale x 8 x i32>, <vscale x 8 x i32>, i32, <vscale x 8 x i1>, i32, i32)

llvm/unittests/IR/VPIntrinsicTest.cpp

for (const char *ReductionOpcode : ReductionIntOpcodes)
           << ".v8i32(i32, <8 x i32>, <8 x i1>, i32) ";
 
     for (const char *ReductionOpcode : ReductionFPOpcodes)
       Str << " declare float @llvm.vp.reduce." << ReductionOpcode
           << ".v8f32(float, <8 x float>, <8 x i1>, i32) ";
 
     Str << " declare <8 x i32> @llvm.vp.select.v8i32(<8 x i1>, <8 x i32>, <8 x "
            "i32>, i32)";
+    Str << " declare <8 x i32> @llvm.experimental.vp.splice.v8i32(<8 x "
+           "i32>, <8 x i32>, i32, <8 x i1>, i32, i32) ";
+
     return parseAssemblyString(Str.str(), Err, C);
   }
 };
 
 /// Check that the property scopes include/llvm/IR/VPIntrinsics.def are closed.
 TEST_F(VPIntrinsicTest, VPIntrinsicsDefScopes) {
   Optional<Intrinsic::ID> ScopeVPID;
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...)                                 \